x86/resctrl telemetry monitoring
Subsystems (5)
Tony Luck
maintainerv3 cover x86/resctrl telemetry monitoring
Previous version here:
https://lore.kernel.org/all/[email protected]/
This series is based on James Morse's "fs/resctrl/" snapshot.
General changes since v2:
------------------------
Fixed several violations of the FS/ARCH interface. As far as possible the patches are now divided into patches for ARCH code v.s. patches for FS code.
All event names/numbers are defined by FS code.
Changed the names of new bits in FS code to PERF_PKG.
Changed names of detailed event descriptions from using "CWF" (short for Clearwater Forest) to names using the guid value. E.g. NUM_RMIDS_CWF -> NUM_RMIDS_0x26696143
Fixed typos from v2, and improved many commit messages and code comments.
Roadmap to patches in the series:
--------------------------------
First patch provides a simpler implementation of James' "Expand the width of dom_id by replacing mon_data_bits" patch. James will merge this into the next version of his resctrl fs series. So this patch will be dropped.
Second patch is a cleaned up version of something I posted before to improve handling multiple MBM events.
https://lore.kernel.org/all/[email protected]/
It turns out to be a useful base to add other monitor events.
Third patch is preparation for more monitor events changing how and when they are initialized. The "when" moves the initialization to first mount of resctrl ready for adding perf events (which are enumerated much later than normal resctrl initialization).
Fourth sets up the Kconfig elements for PERF events.
Fifth patch is the fake OOBMSM interface, providing the intel_pmt_get_regions_by_feature() and intel_pmt_put_feature_group() that will eventually be supplied directly by OOBMSM code.
Sixth patch is in response to Reinette's review of the V2 series where I had deleted some rdt_mon_domain type checking which could lead to problems with the additional domain type. I extended the type checking from simple "ctrl" vs. "mon" to also check the resource type.
7-9 refactor some x86 code to make adding a new rdt_resource easier
10 Implemented a suggestion from James on coding so that events that can be read from any CPU avoid a cross-processor interrupt while keeping the same calling sequence between FS and ARCH code. The attribute is available independently for each event.
11 Adds support for ARCH code to describe which format to display monitor values from choices listed in an FS enum. Two choices
implemented: 64-bit unsigned decimal number, and fixed-point with 18 binary digits displayed as floating point.
12 Add FS hook for ARCH code to set event counter attributes.
13 Add ARCH hook called by FS on each mount.
14-16 X86 enumeration of perf events using calls to OOBMSM
17 Add lookup table from FS event numbers to ARCH structures needed to process event read requests
18 Hook into resctrl_arch_rmid_read() for RDT_RESOURCE_PERF_PKG to read counters for its events. Code to read the events sums over all aggregators in a package.
19 Cross check three enumerated sources of number of RMIDS for perf events. This part needs further discussion on how to pass information to users. Existing FS/ARCH interfaces doesn't have obvious way to add additional info/ files.
20 Add new RDT_RESOURCE_PERF_PKG resource and package scope define
21 Domain creation/deletion for RDT_RESOURCE_PERF_PKG
22 Type define for resctrl files for RDT_RESOURCE_PERF_PKG
23 Add new events to typedef resctrl_event_id and all_events[].
24 Enable all enumerated telemetry events and the RDT_RESOURCE_PERF_PKG resource.
25 Add details for the telemetry events on Clearwater Forest systems.
26 Documentation.
Background
Telemetry features are being implemented in conjunction with the IA32_PQR_ASSOC.RMID value on each logical CPU. This is used to send counts for various events to a collector in a nearby OOMMSM device to be accumulated with counts for each <RMID, event> pair received from other CPUs. Cores send event counts when the RMID value changes, or after each 2ms elapsed time.
Each OOBMSM device may implement multiple event collectors with each servicing a subset of the logical CPUs on a package. In the initial hardware implementation, there are two categories of events:
The counters are arranged in groups in MMIO space of the OOBMSM device. E.g. for the energy counters the layout is:
Offset: Counter0x00 core energy for RMID 00x08 core activity for RMID 00x10 core energy for RMID 10x18 core activity for RMID 1
1) Energy - Two counterscore_energy: This is an estimate of Joules consumed by each core. It is calculated based on the types of instructions executed, not from a power meter. This counter is useful to understand how much energy a workload is consuming.
activity: This measures "accumulated dynamic capacitance". Users who want to optimize energy consumption for a workload may use this rather than core_energy because it provides consistent results independent of any frequency or voltage changes that may occur during the runtime of the application (e.g. entry/exit from turbo mode).
2) Performance - Seven counters
These are similar events to those available via the Linux "perf" tool, but collected in a way with mush lower overhead (no need to collect data on every context switch).
stalls_llc_hit - Counts the total number of unhalted core clock cycles when the core is stalled due to a demand load miss which hit in the LLC
c1_res - Counts the total C1 residency across all cores. The underlying counter increments on 100MHz clock ticks
unhalted_core_cycles - Counts the total number of unhalted core clock cycles
stalls_llc_miss - Counts the total number of unhalted core clock cycles when the core is stalled due to a demand load miss which missed all the local caches
c6_res - Counts the total C6 residency. The underlying counter increments on crystal clock (25MHz) ticks
unhalted_ref_cycles - Counts the total number of unhalted reference clock (TSC) cycles
uops_retired - Counts the total number of uops retired
Enumeration
The only CPUID based enumeration for this feature is the legacy CPUID(eax=7,ecx=0).ebx{12} that indicates the presence of the IA32_PQR_ASSOC MSR and the RMID field within it.
The OOBMSM driver discovers which features are present via PCIe VSEC capabilities. Each feature is tagged with a unique identifier. These identifiers indicate which XML description file from
https://github.com/intel/Intel-PMT describes which event counters are available and their layout within the MMIO BAR space of the OOBMSM device.
Resctrl User Interface
Because there may be multiple OOBMSM collection agents per processor package, resctrl accumulates event counts from all agents on a package and presents a single value to users. This will provide a consistent user interface on future platforms that vary the number of collectors, or the mappings from logical CPUs to collectors.
Users will see the legacy monitoring files in the "L3" directories and the telemetry files in "PKG" directories (with each file providing the aggregated value from all OOBMSM collectors on that package).
$ tree /sys/fs/resctrl/mon_data/
/sys/fs/resctrl/mon_data/
├── mon_L3_00
│ ├── llc_occupancy
│ ├── mbm_local_bytes
│ └── mbm_total_bytes
├── mon_L3_01
│ ├── llc_occupancy
│ ├── mbm_local_bytes
│ └── mbm_total_bytes
├── mon_PKG_00
│ ├── activity
│ ├── c1_res
│ ├── c6_res
│ ├── core_energy
│ ├── stalls_llc_hit
│ ├── stalls_llc_miss
│ ├── unhalted_core_cycles
│ ├── unhalted_ref_cycles
│ └── uops_retired
└── mon_PKG_01
├── activity
├── c1_res
├── c6_res
├── core_energy
├── stalls_llc_hit
├── stalls_llc_miss
├── unhalted_core_cycles
├── unhalted_ref_cycles
└── uops_retiredResctrl Implementation
The OOBMSM driver exposes a function "intel_pmt_get_regions_by_feature()" that returns an array of structures describing the per-RMID groups it found from the VSEC enumeration. Linux looks at the unique identifiers for each group and enables resctrl for all groups with known unique identifiers.
The memory map for the counters for each <RMID, event> pair is described by the XML file. This is too unwieldy to use in the Linux kernel, so a simplified representation is built into the resctrl code. Note that the counters are in MMIO space instead of accessed using the IA32_QM_EVTSEL and IA32_QM_CTR MSRs. This means there is no need for cross-processor calls to read counters from a CPU in a specific domain. The counters can be read from any CPU.
High level description of code changes:
1) New scope RESCTRL_PACKAGE
2) New struct rdt_resource RDT_RESOURCE_INTEL_PMT
3) Refactor monitor code paths to split existing L3 paths from new ones. In some cases this ends up with:
switch (r->rid) {
case RDT_RESOURCE_L3:
helper for L3
break;
case RDT_RESOURCE_INTEL_PMT:
helper for PKG
break;
}4) New source code file "intel_pmt.c" for the code to enumerate, configure, and report event counts.
With only one platform providing this feature, it's tricky to tell exactly where it is going to go. I've made the event definitions platform specific (based on the unique ID from the VSEC enumeration). It seems possible/likely that the list of events may change from generation to generation.
I've picked names for events based on the descriptions in the XML file.
Tony Luck (26):
fs/resctrl: Simplify allocation of mon_data structures fs-x86/resctrl: Prepare for more monitor events fs/resctrl: Change how events are initialized
fs/resctrl: Set up Kconfig options for telemetry events x86/rectrl: Fake OOBMSM interface
fs-x86/rectrl: Improve domain type checking
x86/resctrl: Move L3 initialization out of domain_add_cpu_mon() x86/resctrl: Refactor domain_remove_cpu_mon() ready for new domain
types
x86/resctrl: Change generic monitor functions to use struct
rdt_domain_hdr
fs/resctrl: Improve handling for events that can be read from any CPU fs/resctrl: Add support for additional monitor event display formats fs/resctrl: Add hook for architecture code to set monitor event
attributes
fs/resctrl: Add an architectural hook called for each mount x86/resctrl: Add first part of telemetry event enumeration x86/resctrl: Second stage of telemetry event enumeration x86/resctrl: Third phase of telemetry event enumeration x86/resctrl: Build a lookup table for each resctrl event id x86/resctrl: Add code to read core telemetry events x86/resctrl: Sanity check telemetry RMID values x86/resctrl: Add and initialize rdt_resource for package scope core
monitor
fs-x86/resctrl: Handle RDT_RESOURCE_PERF_PKG in domain create/delete fs/resctrl: Add type define for PERF_PKG files
fs/resctrl: Add new telemetry event id and structures x86/resctrl: Final steps to enable RDT_RESOURCE_PERF_PKG fs-x86/resctrl: Add detailed descriptions for Clearwater Forest events x86/resctrl: Update Documentation for package events
Documentation/filesystems/resctrl.rst | 38 +- include/linux/resctrl.h | 42 +- include/linux/resctrl_types.h | 36 +- arch/x86/include/asm/resctrl.h | 8 +- .../cpu/resctrl/fake_intel_aet_features.h | 73 ++++ arch/x86/kernel/cpu/resctrl/internal.h | 19 +- fs/resctrl/internal.h | 23 +- arch/x86/kernel/cpu/resctrl/core.c | 199 ++++++--- .../cpu/resctrl/fake_intel_aet_features.c | 87 ++++ arch/x86/kernel/cpu/resctrl/intel_aet.c | 377 ++++++++++++++++++ arch/x86/kernel/cpu/resctrl/monitor.c | 39 +- fs/resctrl/ctrlmondata.c | 76 ++-- fs/resctrl/monitor.c | 164 ++++++-- fs/resctrl/rdtgroup.c | 266 ++++++------ arch/x86/Kconfig | 1 + arch/x86/kernel/cpu/resctrl/Makefile | 2 + drivers/platform/x86/intel/pmt/Kconfig | 6 + 17 files changed, 1125 insertions(+), 331 deletions(-) create mode 100644 arch/x86/kernel/cpu/resctrl/fake_intel_aet_features.h create mode 100644 arch/x86/kernel/cpu/resctrl/fake_intel_aet_features.c create mode 100644 arch/x86/kernel/cpu/resctrl/intel_aet.c
base-commit: 89679ca4122a5a6622530490ef4855dd39baa54e
Signature · 1 line
2.48.1
Tony Luck
maintainerPATCH v3 1/26 fs/resctrl: Simplify allocation of mon_data structures
Instead of making a special case to allocate and attach these structures to kernfs files in the default control group, simply allocate a structure when a new combination of <rid, domain, mevt, do_sum> is needed and reuse existing structures when possible.
Free all structures when resctrl filesystem is unmounted.
Partial revert of commit fa563b5171e9 ("x86/resctrl: Expand the width of dom_id by replacing mon_data_bits")
--- fs/resctrl/internal.h | 2 + fs/resctrl/rdtgroup.c | 138 ++++++++++++------------------------------ 2 files changed, 40 insertions(+), 100 deletions(-)
fs/resctrl/internal.h +2 -0
@@ -83,6 +83,7 @@ struct mon_evt {/** * struct mon_data - Monitoring details for each event file. +* @list: List of all allocated structures. * @rid: Resource id associated with the event file. * @evtid: Event id associated with the event file. * @sum: Set when event must be summed across multiple
@@ -96,6 +97,7 @@ struct mon_evt {* rdtgroup_mutex. */ struct mon_data { +struct list_head list;unsigned int rid;enum resctrl_event_id evtid;unsigned int sum;
fs/resctrl/rdtgroup.c +38 -100
197 lines not on this page · raw message
Signature · 1 line
2.48.1
Reinette Chatre
maintainerRe: [PATCH v3 01/26] fs/resctrl: Simplify allocation of mon_data structures
Hi Tony,
On 4/7/25 4:40 PM, Tony Luck wrote:
...
fs/resctrl/rdtgroup.c static struct file_system_type rdt_fs_type = { · line 2950
Quoted 6 lines
fs/resctrl/rdtgroup.c +3
-0+list_for_each_entry_safe (priv, tmp, &kn_priv_list, list) { +kfree (priv); +list_del (&priv->list); } }
Did not look through this patch in detail considering its other version in the arch/fs split but this caught my eye. Order should be switched.
Reinette
Tony Luck
maintainerPATCH v3 2/26 fs-x86/resctrl: Prepare for more monitor events
In reply to Tony LuckThere's a rule in computer programming that objects appear zero, once, or many times. So code accordingly.
There are two MBM events and resctrl is coded with a lot of
if (local)
do one thing
if (total)
do a different thing
Simplify the code by coding for many events using loops on which are enabled.
Make rdt_mon_features a bitmap to allow for expansion.
Move resctrl_is_mbm_event() to <asm/resctrl.h> as it gets used by core.c
--- include/linux/resctrl.h | 6 +-- include/linux/resctrl_types.h | 8 +++ arch/x86/include/asm/resctrl.h | 8 +-- arch/x86/kernel/cpu/resctrl/internal.h | 6 +-- fs/resctrl/internal.h | 4 ++ arch/x86/kernel/cpu/resctrl/core.c | 45 +++++++++-------- arch/x86/kernel/cpu/resctrl/monitor.c | 33 ++++++------ fs/resctrl/ctrlmondata.c | 41 ++++----------- fs/resctrl/monitor.c | 70 ++++++++++++++++---------- fs/resctrl/rdtgroup.c | 47 ++++++++--------- 10 files changed, 133 insertions(+), 135 deletions(-)
include/linux/resctrl.h +2 -4
@@ -151,8 +151,7 @@ struct rdt_ctrl_domain {* @hdr: common header for different domain types * @ci: cache info for this domain * @rmid_busy_llc: bitmap of which limbo RMIDs are above threshold -* @mbm_total: saved state for MBM total bandwidth -* @mbm_local: saved state for MBM local bandwidth +* @mbm_states: saved state for each QOS MBM event * @mbm_over: worker to periodically read MBM h/w counters * @cqm_limbo: worker to periodically read CQM h/w counters * @mbm_work_cpu: worker CPU for MBM h/w counters
@@ -162,8 +161,7 @@ struct rdt_mon_domain {struct rdt_domain_hdr hdr;struct cacheinfo *ci;unsigned long *rmid_busy_llc; -struct mbm_state *mbm_total; -struct mbm_state *mbm_local; +struct mbm_state *mbm_states[QOS_NUM_MBM_EVENTS ];struct delayed_work mbm_over;struct delayed_work cqm_limbo;int mbm_work_cpu;
include/linux/resctrl_types.h +8 -0
@@ -55,5 +55,13 @@ enum resctrl_event_id { };#define QOS_NUM_EVENTS (QOS_L3_MBM_LOCAL_EVENT_ID + 1) +#define QOS_NUM_MBM_EVENTS (QOS_L3_MBM_LOCAL_EVENT_ID - QOS_L3_MBM_TOTAL_EVENT_ID + 1) +#define MBM_EVENT_IDX (evt) ((evt) - QOS_L3_MBM_TOTAL_EVENT_ID) + +static inline bool resctrl_is_mbm_event (int e) +{ +return (e >=QOS_L3_MBM_TOTAL_EVENT_ID && + e <=QOS_L3_MBM_LOCAL_EVENT_ID ); +}#endif /* __LINUX_RESCTRL_TYPES_H */
arch/x86/include/asm/resctrl.h +4 -4
@@ -42,7 +42,7 @@ DECLARE_PER_CPU(struct resctrl_pqr_state, pqr_state);extern bool rdt_alloc_capable;extern bool rdt_mon_capable; -extern unsigned int rdt_mon_features; +extern DECLARE_BITMAP(rdt_mon_features, QOS_NUM_EVENTS);DECLARE_STATIC_KEY_FALSE (rdt_enable_key);DECLARE_STATIC_KEY_FALSE (rdt_alloc_enable_key);
@@ -84,17 +84,17 @@ static inline void resctrl_arch_disable_mon(void)static inline bool resctrl_arch_is_llc_occupancy_enabled (void ) { -return (rdt_mon_features & (1 <<QOS_L3_OCCUP_EVENT_ID )); +return test_bit (QOS_L3_OCCUP_EVENT_ID , rdt_mon_features); }static inline bool resctrl_arch_is_mbm_total_enabled (void ) { -return (rdt_mon_features & (1 <<QOS_L3_MBM_TOTAL_EVENT_ID )); +return test_bit (QOS_L3_MBM_TOTAL_EVENT_ID , rdt_mon_features); }static inline bool resctrl_arch_is_mbm_local_enabled (void ) { -return (rdt_mon_features & (1 <<QOS_L3_MBM_LOCAL_EVENT_ID )); +return test_bit (QOS_L3_MBM_LOCAL_EVENT_ID , rdt_mon_features); }/*
arch/x86/kernel/cpu/resctrl/internal.h +2 -4
@@ -59,15 +59,13 @@ struct rdt_hw_ctrl_domain {* struct rdt_hw_mon_domain - Arch private attributes of a set of CPUs that share * a resource for a monitor function * @d_resctrl: Properties exposed to the resctrl file system -* @arch_mbm_total: arch private state for MBM total bandwidth -* @arch_mbm_local: arch private state for MBM local bandwidth +* @arch_mbm_states: arch private state for each MBM event * * Members of this structure are accessed via helpers that provide abstraction. */ struct rdt_hw_mon_domain {struct rdt_mon_domain d_resctrl; -struct arch_mbm_state *arch_mbm_total; -struct arch_mbm_state *arch_mbm_local; +struct arch_mbm_state *arch_mbm_states[QOS_NUM_MBM_EVENTS ]; };static inline struct rdt_hw_ctrl_domain *resctrl_to_arch_ctrl_dom (struct rdt_ctrl_domain *r)
fs/resctrl/internal.h +4 -0
@@ -386,6 +386,10 @@ bool closid_allocated(unsigned int closid);int resctrl_find_cleanest_closid (void ); +int rdt_lookup_evtid_by_name (char *name); + +char *rdt_event_name (enum resctrl_event_id evt); +#ifdef CONFIG_RESCTRL_FS_PSEUDO_LOCK int rdtgroup_locksetup_enter (struct rdtgroup *rdtgrp);
arch/x86/kernel/cpu/resctrl/core.c +23 -22
69 lines not on this page · raw message
arch/x86/kernel/cpu/resctrl/monitor.c +15 -18
@@ -36,7 +36,7 @@ bool rdt_mon_capable;/* * Global to indicate which monitoring events are enabled. */ -unsigned int rdt_mon_features; +DECLARE_BITMAP (rdt_mon_features,QOS_NUM_EVENTS );#define CF (cf) ((unsigned long)(1048576 * (cf) + 0.5))
@@ -168,19 +168,14 @@ static struct arch_mbm_state *get_arch_mbm_state(struct rdt_hw_mon_domain *hw_dou32 rmid, enum resctrl_event_id eventid) { - switch (eventid) { - case QOS_L3_OCCUP_EVENT_ID: + struct arch_mbm_state *state; + +if (!resctrl_is_mbm_event (eventid))return NULL ; -case QOS_L3_MBM_TOTAL_EVENT_ID : -return &hw_dom->arch_mbm_total[rmid]; -case QOS_L3_MBM_LOCAL_EVENT_ID : -return &hw_dom->arch_mbm_local[rmid]; - } -/* Never expect to get here */ -WARN_ON_ONCE (1 ); + state = hw_dom->arch_mbm_states[MBM_EVENT_IDX (eventid)]; -return NULL ; +return state ? &state[rmid] :NULL ; } void resctrl_arch_reset_rmid(struct rdt_resource *r, struct rdt_mon_domain *d,
@@ -209,14 +204,16 @@ void resctrl_arch_reset_rmid(struct rdt_resource *r, struct rdt_mon_domain *d,void resctrl_arch_reset_rmid_all (struct rdt_resource *r,struct rdt_mon_domain *d) {struct rdt_hw_mon_domain *hw_dom =resctrl_to_arch_mon_dom (d); +int evt, idx; + +for_each_set_bit (evt, rdt_mon_features,QOS_NUM_EVENTS ) { + idx =MBM_EVENT_IDX (evt); +if (!hw_dom->arch_mbm_states[idx]) +continue ; +memset (hw_dom->arch_mbm_states[idx],0 , +sizeof (struct arch_mbm_state ) * r->num_rmid); + } -if (resctrl_arch_is_mbm_total_enabled ()) -memset (hw_dom->arch_mbm_total,0 , -sizeof (*hw_dom->arch_mbm_total) * r->num_rmid); - -if (resctrl_arch_is_mbm_local_enabled ()) -memset (hw_dom->arch_mbm_local,0 , -sizeof (*hw_dom->arch_mbm_local) * r->num_rmid); }static u64 mbm_overflow_count (u64 prev_msr,u64 cur_msr,unsigned int width)
fs/resctrl/ctrlmondata.c +9 -32
64 lines not on this page · raw message
fs/resctrl/monitor.c +44 -26
105 lines not on this page · raw message
fs/resctrl/rdtgroup.c +22 -25
74 lines not on this page · raw message
Signature · 1 line
2.48.1
Reinette Chatre
maintainerRe: [PATCH v3 02/26] fs-x86/resctrl: Prepare for more monitor events
Hi Tony,
Quoted 10 lines · Tony Luck
There's a rule in computer programming that objects appear zero, once, or many times. So code accordingly.
There are two MBM events and resctrl is coded with a lot of
if (local)
do one thing
if (total)
do a different thing
first change:
Quoted 2 lines
Simplify the code by coding for many events using loops on which are enabled.
Please elaborate on how the primary change is the change in data structure and that is what enables loops to be used.
second change:
Quoted 2 lines
Make
rdt_mon_featuresa bitmap to allow for expansion.
... and then a third change: Introduce rdt_lookup_evtid_by_name() and rdt_event_name().
I recognize three logical changes. Could you please split this patch?
Quoted 2 lines
Move
resctrl_is_mbm_event()to <asm/resctrl.h> as it gets used bycore.c
What the patch actually does is move resctrl_is_mbm_event() to include/linux/resctrl_types.h that is in itself unexpected considering what resctrl_types.h is intended to be used for. See details in changelog of commit
f16adbaf9272 ("x86/resctrl: Move resctrl types to a separate header")
for details on how resctrl_types.h is intended to be used.
Quoted 15 lines
Signed-off-by Tony Luck <[email protected]>--- include/linux/resctrl.h | 6 +-- include/linux/resctrl_types.h | 8 +++ arch/x86/include/asm/resctrl.h | 8 +-- arch/x86/kernel/cpu/resctrl/internal.h | 6 +-- fs/resctrl/internal.h | 4 ++ arch/x86/kernel/cpu/resctrl/core.c | 45 +++++++++-------- arch/x86/kernel/cpu/resctrl/monitor.c | 33 ++++++------ fs/resctrl/ctrlmondata.c | 41 ++++----------- fs/resctrl/monitor.c | 70 ++++++++++++++++---------- fs/resctrl/rdtgroup.c | 47 ++++++++--------- 10 files changed, 133 insertions(+), 135 deletions(-)
...
Quoted 6 lines
include/linux/resctrl_types.h +0
-0@@ -55,5 +55,13 @@ enum resctrl_event_id { };
include/linux/resctrl_types.h enum resctrl_event_id { · line 65
Quoted 10 lines
include/linux/resctrl_types.h +8
-0#define QOS_NUM_EVENTS (QOS_L3_MBM_LOCAL_EVENT_ID + 1) +#define QOS_NUM_MBM_EVENTS (QOS_L3_MBM_LOCAL_EVENT_ID - QOS_L3_MBM_TOTAL_EVENT_ID + 1) +#define MBM_EVENT_IDX (evt) ((evt) - QOS_L3_MBM_TOTAL_EVENT_ID) + +static inline bool resctrl_is_mbm_event (int e) +{ +return (e >=QOS_L3_MBM_TOTAL_EVENT_ID && + e <=QOS_L3_MBM_LOCAL_EVENT_ID ); +}
include/linux/resctrl.h should be a better fit.
Quoted 2 lines
include/linux/resctrl_types.h +0
-0#endif /* __LINUX_RESCTRL_TYPES_H */
...
Quoted 8 lines
static int
get_domain_id_from_scope(int cpu, enum resctrl_scope scope)include/linux/resctrl_types.h +1
-1@@ -864,13 +865,13 @@ static __init bool get_rdt_mon_resources(void)struct rdt_resource *r = &rdt_resources_all[RDT_RESOURCE_L3 ].r_resctrl;if (rdt_cpu_has (X86_FEATURE_CQM_OCCUP_LLC )) - rdt_mon_features |= (1 <<QOS_L3_OCCUP_EVENT_ID ); +__set_bit (QOS_L3_OCCUP_EVENT_ID , rdt_mon_features);
include/linux/resctrl_types.h static __init bool get_rdt_mon_resources(void) · line 875
Quoted 10 lines
include/linux/resctrl_types.h +3
-3if (rdt_cpu_has (X86_FEATURE_CQM_MBM_TOTAL )) - rdt_mon_features |= (1 <<QOS_L3_MBM_TOTAL_EVENT_ID ); +__set_bit (QOS_L3_MBM_TOTAL_EVENT_ID , rdt_mon_features);if (rdt_cpu_has (X86_FEATURE_CQM_MBM_LOCAL )) - rdt_mon_features |= (1 <<QOS_L3_MBM_LOCAL_EVENT_ID ); +__set_bit (QOS_L3_MBM_LOCAL_EVENT_ID , rdt_mon_features); -if (!rdt_mon_features) +if (find_first_bit (rdt_mon_features,QOS_NUM_EVENTS ) ==QOS_NUM_EVENTS )return false;
Could you please use bitmap_empty() instead? It does the same, but makes it obvious what is being tested for.
Quoted 41 lines
include/linux/resctrl_types.h +0
-0return !rdt_get_mon_l3_config (r);arch/x86/kernel/cpu/resctrl/monitor.c +6
-11@@ -36,7 +36,7 @@ bool rdt_mon_capable;/* * Global to indicate which monitoring events are enabled. */ -unsigned int rdt_mon_features; +DECLARE_BITMAP (rdt_mon_features,QOS_NUM_EVENTS );#define CF (cf) ((unsigned long)(1048576 * (cf) + 0.5))@@ -168,19 +168,14 @@ static struct arch_mbm_state *get_arch_mbm_state(struct rdt_hw_mon_domain *hw_dou32 rmid, enum resctrl_event_id eventid) { - switch (eventid) { - case QOS_L3_OCCUP_EVENT_ID: + struct arch_mbm_state *state; + +if (!resctrl_is_mbm_event (eventid))return NULL ; -case QOS_L3_MBM_TOTAL_EVENT_ID : -return &hw_dom->arch_mbm_total[rmid]; -case QOS_L3_MBM_LOCAL_EVENT_ID : -return &hw_dom->arch_mbm_local[rmid]; - } -/* Never expect to get here */ -WARN_ON_ONCE (1 ); + state = hw_dom->arch_mbm_states[MBM_EVENT_IDX (eventid)]; -return NULL ; +return state ? &state[rmid] :NULL ; } void resctrl_arch_reset_rmid(struct rdt_resource *r, struct rdt_mon_domain *d,
arch/x86/kernel/cpu/resctrl/monitor.c void resctrl_arch_reset_rmid(struct rdt_resource *r, struct rdt_mon_domain *d, · line 212
Quoted 9 lines
arch/x86/kernel/cpu/resctrl/monitor.c +6
-0@@ -209,14 +204,16 @@ void resctrl_arch_reset_rmid(struct rdt_resource *r, struct rdt_mon_domain *d,void resctrl_arch_reset_rmid_all (struct rdt_resource *r,struct rdt_mon_domain *d) {struct rdt_hw_mon_domain *hw_dom =resctrl_to_arch_mon_dom (d); +int evt, idx; + +for_each_set_bit (evt, rdt_mon_features,QOS_NUM_EVENTS ) { + idx =MBM_EVENT_IDX (evt); +if (!hw_dom->arch_mbm_states[idx]) +continue ;
This does not look safe. Missing a resctrl_is_mbm_event() check?
Quoted 13 lines
arch/x86/kernel/cpu/resctrl/monitor.c +3
-7+memset (hw_dom->arch_mbm_states[idx],0 , +sizeof (struct arch_mbm_state ) * r->num_rmid); + } -if (resctrl_arch_is_mbm_total_enabled ()) -memset (hw_dom->arch_mbm_total,0 , -sizeof (*hw_dom->arch_mbm_total) * r->num_rmid); - -if (resctrl_arch_is_mbm_local_enabled ()) -memset (hw_dom->arch_mbm_local,0 , -sizeof (*hw_dom->arch_mbm_local) * r->num_rmid); }
...
Quoted 27 lines
fs/resctrl/monitor.c +6
-7@@ -347,15 +347,14 @@ static struct mbm_state *get_mbm_state(struct rdt_mon_domain *d, u32 closid, u32 rmid, enum resctrl_event_id evtid) { u32 idx = resctrl_arch_rmid_idx_encode(closid, rmid); +struct mbm_state *states; -switch (evtid) { -case QOS_L3_MBM_TOTAL_EVENT_ID : -return &d->mbm_total[idx]; -case QOS_L3_MBM_LOCAL_EVENT_ID : -return &d->mbm_local[idx]; -default : +if (!resctrl_is_mbm_event (evtid))return NULL ; - } + + states = d->mbm_states[MBM_EVENT_IDX (evtid)]; + +return states ? &states[idx] :NULL ; }static int __mon_event_count (u32 closid,u32 rmid,struct rmid_read *rr)
fs/resctrl/monitor.c static void dom_data_exit(struct rdt_resource *r) · line 845
Quoted 7 lines
fs/resctrl/monitor.c +1
-3@@ -843,20 +842,40 @@ static void dom_data_exit(struct rdt_resource *r)mutex_unlock (&rdtgroup_mutex); } -static struct mon_evt llc_occupancy_event = { - .name ="llc_occupancy" , - .evtid =QOS_L3_OCCUP_EVENT_ID , +static struct mon_evt all_events[QOS_NUM_EVENTS ] = {
"all_events" is very generic for a global. How about "mon_event_all" (placing the "_all" at end to match, for example, resctrl_schema_all).
Quoted 11 lines
fs/resctrl/monitor.c +11
-0+ [QOS_L3_OCCUP_EVENT_ID] = { + .name = "llc_occupancy", + .evtid = QOS_L3_OCCUP_EVENT_ID, + }, + [QOS_L3_MBM_TOTAL_EVENT_ID] = { + .name = "mbm_total_bytes", + .evtid = QOS_L3_MBM_TOTAL_EVENT_ID, + }, + [QOS_L3_MBM_LOCAL_EVENT_ID] = { + .name = "mbm_local_bytes", + .evtid = QOS_L3_MBM_LOCAL_EVENT_ID,
fs/resctrl/monitor.c static void dom_data_exit(struct rdt_resource *r) · line 862
Quoted 10 lines
fs/resctrl/monitor.c +4
-4+ }, }; -static struct mon_evt mbm_total_event = { - .name ="mbm_total_bytes" , - .evtid =QOS_L3_MBM_TOTAL_EVENT_ID , -}; +int rdt_lookup_evtid_by_name (char *name) +{ +int evt;
Since this is resctrl fs code, please replace "rdt" with "resctrl".
fs/resctrl/monitor.c static void dom_data_exit(struct rdt_resource *r) · line 867
Quoted 9 lines
fs/resctrl/monitor.c +4
-4-static struct mon_evt mbm_local_event = { - .name ="mbm_local_bytes" , - .evtid =QOS_L3_MBM_LOCAL_EVENT_ID , -}; +for_each_set_bit (evt, rdt_mon_features,QOS_NUM_EVENTS ) { +if (!strcmp (name, all_events[evt].name)) +return evt; + }
* This is resctrl fs code. rdt_mon_features should be private to x86 so resctrl fs should not be accessing it directly. Perhaps there can be a new arch helper that can be used by resctrl to query if event is enabled? Similar to resctrl_arch_is_llc_occupancy_enabled() and friend but where the event ID is parameter and arch code can use rdt_mon_features.
* While the function name is "lookup_evtid_by_name" this function does not just look up the event id by name but also ensures that the event is enabled. The caller then uses "lookup_evtid_by_name" as a proxy for "is this event enabled". I think the code will be easier to understand if the functions do not have such hidden "features". resctrl fs could first use new arch helper to determine if
event is enabled and then use a fs helper that reads name directly from the event array.
* For a function returning event id the return type is expected to be enum resctrl_event_id.
fs/resctrl/monitor.c static void dom_data_exit(struct rdt_resource *r) · line 878
Quoted 11 lines
fs/resctrl/monitor.c +11
-0+ +return -EINVAL ; +} + +char *rdt_event_name (enum resctrl_event_id evt) +{ +if (!test_bit (evt, rdt_mon_features)) +return "unknown" ; + +return all_events[evt].name; +}
Same comments as rdt_lookup_evtid_by_name()
Also please watch for rdt_mon_features in rest of resctrl fs changes.
Reinette
Tony Luck
maintainerPATCH v3 3/26 fs/resctrl: Change how events are initialized
In reply to Tony LuckNew monitor events break some assumptions:
1) New events can be in resources other than L3.
2) Enumeration of events may not be complete during early boot.
Prepare for events in other resources.
Delay building the event lists until first mount of the resctrl file system.
--- fs/resctrl/internal.h | 3 +++ fs/resctrl/monitor.c | 30 +++++++++++++++++++----------- fs/resctrl/rdtgroup.c | 2 ++ 3 files changed, 24 insertions(+), 11 deletions(-)
fs/resctrl/internal.h +3 -0
@@ -76,6 +76,7 @@ static inline struct rdt_fs_context *rdt_fc2context(struct fs_context *fc)*/ struct mon_evt {enum resctrl_event_id evtid; +enum resctrl_res_level rid;char *name;bool configurable;struct list_head list;
@@ -390,6 +391,8 @@ int rdt_lookup_evtid_by_name(char *name);char *rdt_event_name (enum resctrl_event_id evt); +void resctrl_init_mon_events (void ); +#ifdef CONFIG_RESCTRL_FS_PSEUDO_LOCK int rdtgroup_locksetup_enter (struct rdtgroup *rdtgrp);
fs/resctrl/monitor.c +19 -11
@@ -846,14 +846,17 @@ static struct mon_evt all_events[QOS_NUM_EVENTS] = { [QOS_L3_OCCUP_EVENT_ID ] = { .name ="llc_occupancy" , .evtid =QOS_L3_OCCUP_EVENT_ID , + .rid =RDT_RESOURCE_L3 , }, [QOS_L3_MBM_TOTAL_EVENT_ID ] = { .name ="mbm_total_bytes" , .evtid =QOS_L3_MBM_TOTAL_EVENT_ID , + .rid =RDT_RESOURCE_L3 , }, [QOS_L3_MBM_LOCAL_EVENT_ID ] = { .name ="mbm_local_bytes" , .evtid =QOS_L3_MBM_LOCAL_EVENT_ID , + .rid =RDT_RESOURCE_L3 , }, };
@@ -878,22 +881,29 @@ char *rdt_event_name(enum resctrl_event_id evt) }/* -* Initialize the event list for the resource. +* Initialize the event list for all mon_capable resources. * -* Note that MBM events are also part of RDT_RESOURCE_L3 resource -* because as per the SDM the total and local memory bandwidth -* are enumerated as part of L3 monitoring. -* -* mon_put_default_kn_priv_all() also assumes monitor events are only supported -* on the L3 resource. +* Called on each mount of the resctrl file system when all +* events have been enumerated. Only needs to build the per-resource +* event lists once. */ -static void l3_mon_evt_init (struct rdt_resource *r) +void resctrl_init_mon_events (void ) { +struct rdt_resource *r; +static bool only_once;int evt; -INIT_LIST_HEAD (&r->evt_list); +if (only_once) +return ; + only_once = true; + +for_each_mon_capable_rdt_resource (r) +INIT_LIST_HEAD (&r->evt_list);for_each_set_bit (evt, rdt_mon_features,QOS_NUM_EVENTS ) { + r =resctrl_arch_get_resource (all_events[evt].rid); +if (!r->mon_capable) +continue ;list_add_tail (&all_events[evt].list, &r->evt_list); } }
@@ -922,8 +932,6 @@ int resctrl_mon_resource_init(void)if (ret)return ret; -l3_mon_evt_init (r); -if (resctrl_arch_is_evt_configurable (QOS_L3_MBM_TOTAL_EVENT_ID )) { all_events[QOS_L3_MBM_TOTAL_EVENT_ID ].configurable = true; resctrl_file_fflags_init("mbm_total_bytes_config",
fs/resctrl/rdtgroup.c +2 -0
@@ -2574,6 +2574,8 @@ static int rdt_get_tree(struct fs_context *fc) goto out; } +resctrl_init_mon_events (); + ret =rdtgroup_setup_root (ctx);if (ret) goto out;
Signature · 1 line
2.48.1
Reinette Chatre
maintainerRe: [PATCH v3 03/26] fs/resctrl: Change how events are initialized
Hi Tony,
Quoted 1 line · Tony Luck
New monitor events break some assumptions:
This changelog jumps into a problem without any context.
Please follow changelog guidance from maintainer-tip.rst. Specifically, "A good structure is to explain the context, the problem and the solution in separate paragraphs and this order."
Quoted 6 lines
1) New events can be in resources other than L3.
2) Enumeration of events may not be complete during early boot.Prepare for events in other resources.
Please include what this preparation involves.
Quoted 3 lines
Delay building the event lists until first mount of the resctrl file system.
Please include in context what is meant by "event lists".
But ... previous patch reminded reader about all the event state that is allocated during domain online, which usually happens *before* mount of resctrl. This work thus goes from "use enumeration of events during boot to allocate necessary event state" in one patch to "enumeration of events are not complete during boot so build event lists on resctrl mount" in the next patch. This is a big contradiction to me.
I think it is clear that not all events can be treated equally but this implementation pretends to treat them equally when convenient (this patch) and relies on code flow assumptions (previous patch that only allocated state for L3 events during domain online) for things to "work out" in the end.
Quoted 13 lines
Signed-off-by Tony Luck <[email protected]>--- fs/resctrl/internal.h | 3 +++ fs/resctrl/monitor.c | 30 +++++++++++++++++++----------- fs/resctrl/rdtgroup.c | 2 ++ 3 files changed, 24 insertions(+), 11 deletions(-)fs/resctrl/internal.h +0
-0
fs/resctrl/internal.h static inline struct rdt_fs_context *rdt_fc2context(struct fs_context *fc) · line 79
Quoted 4 lines
fs/resctrl/internal.h +1
-0@@ -76,6 +76,7 @@ static inline struct rdt_fs_context *rdt_fc2context(struct fs_context *fc)*/ struct mon_evt {enum resctrl_event_id evtid; +enum resctrl_res_level rid;
This structure has some kernel-doc that is not visible in this hunk but also needs a change when adding a new member.
Quoted 57 lines
fs/resctrl/internal.h +2
-0char *name;bool configurable;struct list_head list;@@ -390,6 +391,8 @@ int rdt_lookup_evtid_by_name(char *name);char *rdt_event_name (enum resctrl_event_id evt); +void resctrl_init_mon_events (void ); +#ifdef CONFIG_RESCTRL_FS_PSEUDO_LOCK int rdtgroup_locksetup_enter (struct rdtgroup *rdtgrp);fs/resctrl/monitor.c +10
-8@@ -846,14 +846,17 @@ static struct mon_evt all_events[QOS_NUM_EVENTS] = { [QOS_L3_OCCUP_EVENT_ID ] = { .name ="llc_occupancy" , .evtid =QOS_L3_OCCUP_EVENT_ID , + .rid =RDT_RESOURCE_L3 , }, [QOS_L3_MBM_TOTAL_EVENT_ID ] = { .name ="mbm_total_bytes" , .evtid =QOS_L3_MBM_TOTAL_EVENT_ID , + .rid =RDT_RESOURCE_L3 , }, [QOS_L3_MBM_LOCAL_EVENT_ID ] = { .name ="mbm_local_bytes" , .evtid =QOS_L3_MBM_LOCAL_EVENT_ID , + .rid =RDT_RESOURCE_L3 , }, };@@ -878,22 +881,29 @@ char *rdt_event_name(enum resctrl_event_id evt) }/* -* Initialize the event list for the resource. +* Initialize the event list for all mon_capable resources. * -* Note that MBM events are also part of RDT_RESOURCE_L3 resource -* because as per the SDM the total and local memory bandwidth -* are enumerated as part of L3 monitoring. -* -* mon_put_default_kn_priv_all() also assumes monitor events are only supported -* on the L3 resource. +* Called on each mount of the resctrl file system when all +* events have been enumerated. Only needs to build the per-resource +* event lists once. */ -static void l3_mon_evt_init (struct rdt_resource *r) +void resctrl_init_mon_events (void ) { +struct rdt_resource *r; +static bool only_once;int evt;
fs/resctrl/monitor.c char *rdt_event_name(enum resctrl_event_id evt) · line 903
Quoted 10 lines
fs/resctrl/monitor.c +6
-1-INIT_LIST_HEAD (&r->evt_list); +if (only_once) +return ; + only_once = true; + +for_each_mon_capable_rdt_resource (r) +INIT_LIST_HEAD (&r->evt_list);for_each_set_bit (evt, rdt_mon_features,QOS_NUM_EVENTS ) {
This is fs code so this needs to be done without peeking into rdt_mon_features.
Quoted 20 lines
fs/resctrl/monitor.c +3
-2+ r =resctrl_arch_get_resource (all_events[evt].rid); +if (!r->mon_capable) +continue ;list_add_tail (&all_events[evt].list, &r->evt_list); } }@@ -922,8 +932,6 @@ int resctrl_mon_resource_init(void)if (ret)return ret; -l3_mon_evt_init (r); -if (resctrl_arch_is_evt_configurable (QOS_L3_MBM_TOTAL_EVENT_ID )) { all_events[QOS_L3_MBM_TOTAL_EVENT_ID ].configurable = true; resctrl_file_fflags_init("mbm_total_bytes_config",fs/resctrl/rdtgroup.c +0
-0
fs/resctrl/rdtgroup.c static int rdt_get_tree(struct fs_context *fc) · line 2581
Quoted 8 lines
fs/resctrl/rdtgroup.c +2
-0@@ -2574,6 +2574,8 @@ static int rdt_get_tree(struct fs_context *fc) goto out; } +resctrl_init_mon_events (); + ret =rdtgroup_setup_root (ctx);if (ret) goto out;
Reinette
Tony Luck
maintainerPATCH v3 4/26 fs/resctrl: Set up Kconfig options for telemetry events
In reply to Tony LuckEnumeration support is provided by the Intel PMT subsystem.
arch/x86 selects this option based on:
X86_64: Counter registers are in MMIO space. There is no readq() function on 32-bit. Emulation is possible with readl(), but there are races. Running 32-bit kernels on systems that support this feature seems pointless.
CPU_SUP_INTEL: It is an Intel specific feature.
--- arch/x86/Kconfig | 1 + drivers/platform/x86/intel/pmt/Kconfig | 6 ++++++ 2 files changed, 7 insertions(+)
arch/x86/Kconfig +1 -0
@@ -504,6 +504,7 @@ config X86_CPU_RESCTRL bool "x86 CPU resource control support" depends on X86 && (CPU_SUP_INTEL || CPU_SUP_AMD) depends on MISC_FILESYSTEMS + select INTEL_AET_RESCTRL if (X86_64 && CPU_SUP_INTEL) select ARCH_HAS_CPU_RESCTRL select RESCTRL_FS select RESCTRL_FS_PSEUDO_LOCK
drivers/platform/x86/intel/pmt/Kconfig +6 -0
@@ -38,3 +38,9 @@ config INTEL_PMT_CRASHLOG To compile this driver as a module, choose M here: the module will be called intel_pmt_crashlog. + +config INTEL_AET_RESCTRL + bool + help + Architecture config should "select" this option to enable + support for RMID telemtry events in the resctrl file system.
Signature · 1 line
2.48.1
Reinette Chatre
maintainerRe: [PATCH v3 04/26] fs/resctrl: Set up Kconfig options for telemetry events
Hi Tony,
Quoted 1 line · Tony Luck
Enumeration support is provided by the Intel PMT subsystem.
Needs context before jumping in with solution.
Quoted 33 lines
arch/x86selects this option based on:
X86_64: Counter registers are in MMIO space. There is noreadq()function on 32-bit. Emulation is possible withreadl(), but there are races. Running 32-bit kernels on systems that support this feature seems pointless.
CPU_SUP_INTEL: It is an Intel specific feature.Signed-off-by Tony Luck <[email protected]>--- arch/x86/Kconfig | 1 + drivers/platform/x86/intel/pmt/Kconfig | 6 ++++++ 2 files changed, 7 insertions(+)arch/x86/Kconfig +1
-0@@ -504,6 +504,7 @@ config X86_CPU_RESCTRL bool "x86 CPU resource control support" depends on X86 && (CPU_SUP_INTEL || CPU_SUP_AMD) depends on MISC_FILESYSTEMS + select INTEL_AET_RESCTRL if (X86_64 && CPU_SUP_INTEL) select ARCH_HAS_CPU_RESCTRL select RESCTRL_FS select RESCTRL_FS_PSEUDO_LOCKdrivers/platform/x86/intel/pmt/Kconfig +0
-0
drivers/platform/x86/intel/pmt/Kconfig config INTEL_PMT_CRASHLOG · line 43
Quoted 6 lines
drivers/platform/x86/intel/pmt/Kconfig +3
-0@@ -38,3 +38,9 @@ config INTEL_PMT_CRASHLOG To compile this driver as a module, choose M here: the module will be called intel_pmt_crashlog. + +config INTEL_AET_RESCTRL + bool
Usually these settings come with some "depends on" ... no dependencies needed?
drivers/platform/x86/intel/pmt/Kconfig config INTEL_PMT_CRASHLOG · line 46
Quoted 3 lines
drivers/platform/x86/intel/pmt/Kconfig +3
-0+ help + Architecture config should "select" this option to enable + support for RMID telemtry events in the resctrl file system.
telemtry -> telemetry
Please define what is meant with "RMID telemetry events" before this statement.
Reinette
Tony Luck
maintainerPATCH v3 5/26 x86/rectrl: Fake OOBMSM interface
In reply to Tony LuckReal version is coming soon ... this is here so the remaining parts will build (and run ... assuming a 2 socket system that supports RDT monitoring ... only missing part is that the event counters just report fixed values).
Just for ease of testing and RFC discussion.
--- .../cpu/resctrl/fake_intel_aet_features.h | 73 ++++++++++++++++ .../cpu/resctrl/fake_intel_aet_features.c | 87 +++++++++++++++++++ arch/x86/kernel/cpu/resctrl/Makefile | 1 + 3 files changed, 161 insertions(+) create mode 100644 arch/x86/kernel/cpu/resctrl/fake_intel_aet_features.h create mode 100644 arch/x86/kernel/cpu/resctrl/fake_intel_aet_features.c
arch/x86/kernel/cpu/resctrl/fake_intel_aet_features.h new file +73 -0
@@ -0,0 +1,73 @@ +/* SPDX-License-Identifier: GPL-2.0 */ + +/* Bits stolen from OOBMSM VSEC discovery code */ + +enum pmt_feature_id { +FEATURE_INVALID =0x0 , +FEATURE_PER_CORE_PERF_TELEM =0x1 , +FEATURE_PER_CORE_ENV_TELEM =0x2 , +FEATURE_PER_RMID_PERF_TELEM =0x3 , +FEATURE_ACCEL_TELEM =0x4 , +FEATURE_UNCORE_TELEM =0x5 , +FEATURE_CRASH_LOG =0x6 , +FEATURE_PETE_LOG =0x7 , +FEATURE_TPMI_CTRL =0x8 , +FEATURE_RESERVED =0x9 , +FEATURE_TRACING =0xA , +FEATURE_PER_RMID_ENERGY_TELEM =0xB , +FEATURE_MAX =0xB , +}; + +/** +* struct oobmsm_plat_info - Platform information for a device instance +* @cdie_mask: Mask of all compute dies in the partition +* @package_id: CPU Package id +* @partition: Package partition id when multiple VSEC PCI devices per package +* @segment: PCI segment ID +* @bus_number: PCI bus number +* @device_number: PCI device number +* @function_number: PCI function number +* +* Structure to store platform data for a OOBMSM device instance. +*/ +struct oobmsm_plat_info { +u16 cdie_mask; +u8 package_id; +u8 partition; +u8 segment; +u8 bus_number; +u8 device_number; +u8 function_number; +}; + +enum oobmsm_supplier_type { +OOBMSM_SUP_PLAT_INFO , +OOBMSM_SUP_DISC_INFO , +OOBMSM_SUP_S3M_SIMICS , +OOBMSM_SUP_TYPE_MAX +}; + +struct oobmsm_mapping_supplier { +struct device *supplier_dev[OOBMSM_SUP_TYPE_MAX ]; +struct oobmsm_plat_info plat_info; +unsigned long features; +}; + +struct telemetry_region { +struct oobmsm_plat_info plat_info; +void __iomem *addr; +size_t size; +u32 guid; +u32 num_rmids; +}; + +struct pmt_feature_group { +enum pmt_feature_id id; +int count; +struct kref kref; +struct telemetry_region regions[]; +}; + +struct pmt_feature_group *intel_pmt_get_regions_by_feature (enum pmt_feature_id id); + +void intel_pmt_put_feature_group (struct pmt_feature_group *feature_group);
arch/x86/kernel/cpu/resctrl/fake_intel_aet_features.c new file +87 -0
87 lines not on this page · raw message
arch/x86/kernel/cpu/resctrl/Makefile +1 -0
@@ -1,6 +1,7 @@ # SPDX-License-Identifier: GPL-2.0 obj-$(CONFIG_X86_CPU_RESCTRL) += core.o rdtgroup.o monitor.o obj-$(CONFIG_X86_CPU_RESCTRL) += ctrlmondata.o +obj-$(CONFIG_INTEL_AET_RESCTRL) += fake_intel_aet_features.o obj-$(CONFIG_RESCTRL_FS_PSEUDO_LOCK) += pseudo_lock.o # To allow define_trace.h's recursive include:
Signature · 1 line
2.48.1
Reinette Chatre
maintainerRe: [PATCH v3 05/26] x86/rectrl: Fake OOBMSM interface
Hi Tony:
(deja vu ... lets try this comment again ... x86/rectrl -> x86/resctrl)
On 4/7/25 4:40 PM, Tony Luck wrote:
...
Quoted 11 lines
arch/x86/kernel/cpu/resctrl/fake_intel_aet_features.c new file +5
-0@@ -0,0 +1,87 @@ +// SPDX-License-Identifier: GPL-2.0-only +#include <linux/cleanup.h> +#include <linux/minmax.h> +#include <linux/slab.h> +#include "fake_intel_aet_features.h"
arch/x86/kernel/cpu/resctrl/fake_intel_aet_features.c line 15
Quoted 10 lines
arch/x86/kernel/cpu/resctrl/fake_intel_aet_features.c new file +10
-0+#include <linux/intel_vsec.h> +#include <linux/resctrl.h> + +#include "internal.h" + +/* Amount of memory for each fake MMIO space */ +#define ENERGY_QWORDS ((576 * 2) + 3) +#define ENERGY_SIZE (ENERGY_QWORDS * 8) +#define PERF_QWORDS ((576 * 7) + 3) +#define PERF_SIZE (PERF_QWORDS * 8)
Could you please add explanations for the magic numbers? For example, why are both energy and perf using 576?
Quoted 54 lines
arch/x86/kernel/cpu/resctrl/fake_intel_aet_features.c new file +54
-0+ +static long pg[4 *ENERGY_QWORDS +2 *PERF_QWORDS ]; + +/* +* Fill the fake MMIO space with all different values, +* all with BIT(63) set to indicate valid entries. +*/ +static int __init fill (void ) +{ +u64 val =0 ; + +for (int i =0 ; i <sizeof (pg); i +=sizeof (val)) { + pg[i /sizeof (val)] =BIT_ULL (63 ) + val; + val++; + } +return 0 ; +} +device_initcall (fill); + +#define PKG_REGION (_entry, _guid, _addr, _size, _pkg, _num_rmids) \ + [_entry] = { .guid = _guid, .addr = (void __iomem *)_addr, \ + .num_rmids = _num_rmids, \ + .size = _size, .plat_info = { .package_id = _pkg }} + +/* +* Set up a fake return for call to: +* intel_pmt_get_regions_by_feature(FEATURE_PER_RMID_ENERGY_TELEM); +* Pretend there are two aggregators on each of the sockets to test +* the code that sums over multiple aggregators. +*/ +static struct pmt_feature_group fake_energy = { + .count =4 , + .regions = { +PKG_REGION (0 ,0x26696143 , &pg[0 *ENERGY_QWORDS ],ENERGY_SIZE ,0 ,64 ), +PKG_REGION (1 ,0x26696143 , &pg[1 *ENERGY_QWORDS ],ENERGY_SIZE ,0 ,64 ), +PKG_REGION (2 ,0x26696143 , &pg[2 *ENERGY_QWORDS ],ENERGY_SIZE ,1 ,64 ), +PKG_REGION (3 ,0x26696143 , &pg[3 *ENERGY_QWORDS ],ENERGY_SIZE ,1 ,64 ) + } +}; + +/* +* Fake return for: +* intel_pmt_get_regions_by_feature(FEATURE_PER_RMID_PERF_TELEM); +*/ +static struct pmt_feature_group fake_perf = { + .count =2 , + .regions = { +PKG_REGION (0 ,0x26557651 , &pg[4 *ENERGY_QWORDS +0 *PERF_QWORDS ],PERF_SIZE ,0 ,576 ), +PKG_REGION (1 ,0x26557651 , &pg[4 *ENERGY_QWORDS +1 *PERF_QWORDS ],PERF_SIZE ,1 ,576 ) + } +}; + +struct pmt_feature_group * +intel_pmt_get_regions_by_feature (enum pmt_feature_id id)
arch/x86/kernel/cpu/resctrl/fake_intel_aet_features.c line 79
Quoted 10 lines
arch/x86/kernel/cpu/resctrl/fake_intel_aet_features.c new file +10
-0+{ +switch (id) { +case FEATURE_PER_RMID_ENERGY_TELEM : +return &fake_energy; +case FEATURE_PER_RMID_PERF_TELEM : +return &fake_perf; +default : +return ERR_PTR (-ENOENT ); + } +return ERR_PTR (-ENOENT );
Not reachable.
Quoted 13 lines
arch/x86/kernel/cpu/resctrl/fake_intel_aet_features.c new file +8
-0+} + +/* +* Nothing needed for the "put" function. +*/ +void intel_pmt_put_feature_group (struct pmt_feature_group *feature_group) +{ +}arch/x86/kernel/cpu/resctrl/Makefile +0
-0
arch/x86/kernel/cpu/resctrl/Makefile line 7
Quoted 7 lines
arch/x86/kernel/cpu/resctrl/Makefile +1
-0@@ -1,6 +1,7 @@ # SPDX-License-Identifier: GPL-2.0 obj-$(CONFIG_X86_CPU_RESCTRL) += core.o rdtgroup.o monitor.o obj-$(CONFIG_X86_CPU_RESCTRL) += ctrlmondata.o +obj-$(CONFIG_INTEL_AET_RESCTRL) += fake_intel_aet_features.o obj-$(CONFIG_RESCTRL_FS_PSEUDO_LOCK) += pseudo_lock.o # To allow define_trace.h's recursive include:
Reinette
Tony Luck
maintainerPATCH v3 6/26 fs-x86/rectrl: Improve domain type checking
In reply to Tony LuckThe rdt_domain_hdr structure is used in both control and monitor domain structures to provide common methods for operations such as adding a CPU to a domain, removing a CPU from a domain, accessing the mask of all CPUs in a domain.
The "type" field provides a simple check whether a domain is a control or monitor domain so that programming errors operating on domains will be quickly caught.
To prepare for additional domain types that depend on the rdt_resource to which they are connected add the resource id into the type.
--- include/linux/resctrl.h | 19 ++++++++++--------- arch/x86/kernel/cpu/resctrl/core.c | 12 ++++++------ fs/resctrl/ctrlmondata.c | 2 +- 3 files changed, 17 insertions(+), 16 deletions(-)
include/linux/resctrl.h +10 -9
@@ -111,11 +111,6 @@ struct resctrl_staged_config {bool have_new_ctrl; }; -enum resctrl_domain_type { -RESCTRL_CTRL_DOMAIN , -RESCTRL_MON_DOMAIN , -}; -/** * struct rdt_domain_hdr - common header for different domain types * @list: all instances of this resource
@@ -124,12 +119,18 @@ enum resctrl_domain_type {* @cpu_mask: which CPUs share this resource */ struct rdt_domain_hdr { -struct list_head list; -int id; -enum resctrl_domain_type type; -struct cpumask cpu_mask; +struct list_head list; +int id; +u32 type; +struct cpumask cpu_mask; }; +/* Bitfields in rdt_domain_hdr.type */ +#define DOMTYPE_RID GENMASK(8, 0) +#define DOMTYPE_CTRL BIT(9) +#define DOMTYPE_MON BIT(10) +#define DOMTYPE (rid, type) (((rid) & DOMTYPE_RID) | (type)) +/** * struct rdt_ctrl_domain - group of CPUs sharing a resctrl control resource * @hdr: common header for different domain types
arch/x86/kernel/cpu/resctrl/core.c +6 -6
@@ -457,7 +457,7 @@ static void domain_add_cpu_ctrl(int cpu, struct rdt_resource *r) hdr =resctrl_find_domain (&r->ctrl_domains, id, &add_pos);if (hdr) { -if (WARN_ON_ONCE (hdr->type !=RESCTRL_CTRL_DOMAIN )) +if (WARN_ON_ONCE (hdr->type !=DOMTYPE (r->rid,DOMTYPE_CTRL )))return ; d = container_of(hdr, struct rdt_ctrl_domain, hdr);
@@ -473,7 +473,7 @@ static void domain_add_cpu_ctrl(int cpu, struct rdt_resource *r) d = &hw_dom->d_resctrl; d->hdr.id = id; - d->hdr.type =RESCTRL_CTRL_DOMAIN ; + d->hdr.type =DOMTYPE (r->rid,DOMTYPE_CTRL );cpumask_set_cpu (cpu, &d->hdr.cpu_mask);rdt_domain_reconfigure_cdp (r);
@@ -512,7 +512,7 @@ static void domain_add_cpu_mon(int cpu, struct rdt_resource *r) hdr =resctrl_find_domain (&r->mon_domains, id, &add_pos);if (hdr) { -if (WARN_ON_ONCE (hdr->type !=RESCTRL_MON_DOMAIN )) +if (WARN_ON_ONCE (hdr->type !=DOMTYPE (r->rid,DOMTYPE_MON )))return ; d = container_of(hdr, struct rdt_mon_domain, hdr);
@@ -526,7 +526,7 @@ static void domain_add_cpu_mon(int cpu, struct rdt_resource *r) d = &hw_dom->d_resctrl; d->hdr.id = id; - d->hdr.type =RESCTRL_MON_DOMAIN ; + d->hdr.type =DOMTYPE (r->rid,DOMTYPE_MON ); d->ci =get_cpu_cacheinfo_level (cpu,RESCTRL_L3_CACHE );if (!d->ci) {pr_warn_once ("Can't find L3 cache for CPU:%d resource %s\n" , cpu, r->name);
@@ -582,7 +582,7 @@ static void domain_remove_cpu_ctrl(int cpu, struct rdt_resource *r)return ; } -if (WARN_ON_ONCE (hdr->type !=RESCTRL_CTRL_DOMAIN )) +if (WARN_ON_ONCE (hdr->type !=DOMTYPE (r->rid,DOMTYPE_CTRL )))return ; d = container_of(hdr, struct rdt_ctrl_domain, hdr);
@@ -628,7 +628,7 @@ static void domain_remove_cpu_mon(int cpu, struct rdt_resource *r)return ; } -if (WARN_ON_ONCE (hdr->type !=RESCTRL_MON_DOMAIN )) +if (WARN_ON_ONCE (hdr->type !=DOMTYPE (r->rid,DOMTYPE_MON )))return ; d = container_of(hdr, struct rdt_mon_domain, hdr);
fs/resctrl/ctrlmondata.c +1 -1
@@ -614,7 +614,7 @@ int rdtgroup_mondata_show(struct seq_file *m, void *arg)* the resource to find the domain with "domid". */ hdr =resctrl_find_domain (&r->mon_domains, domid,NULL ); -if (!hdr ||WARN_ON_ONCE (hdr->type !=RESCTRL_MON_DOMAIN )) { +if (!hdr ||WARN_ON_ONCE (hdr->type !=DOMTYPE (r->rid,DOMTYPE_MON ))) { ret = -ENOENT ; goto out; }
Signature · 1 line
2.48.1
Reinette Chatre
maintainerRe: [PATCH v3 06/26] fs-x86/rectrl: Improve domain type checking
Hi Tony,
Quoted 45 lines · Tony Luck
The
rdt_domain_hdrstructure is used in both control and monitor domain structures to provide common methods for operations such as adding a CPU to a domain, removing a CPU from a domain, accessing the mask of all CPUs in a domain.The "type" field provides a simple check whether a domain is a control or monitor domain so that programming errors operating on domains will be quickly caught.
To prepare for additional domain types that depend on the
rdt_resourceto which they are connected add the resource id into the type.Signed-off-by Tony Luck <[email protected]>--- include/linux/resctrl.h | 19 ++++++++++--------- arch/x86/kernel/cpu/resctrl/core.c | 12 ++++++------ fs/resctrl/ctrlmondata.c | 2 +- 3 files changed, 17 insertions(+), 16 deletions(-)include/linux/resctrl.h +2
-9@@ -111,11 +111,6 @@ struct resctrl_staged_config {bool have_new_ctrl; }; -enum resctrl_domain_type { -RESCTRL_CTRL_DOMAIN , -RESCTRL_MON_DOMAIN , -}; -/** * struct rdt_domain_hdr - common header for different domain types * @list: all instances of this resource @@ -124,12 +119,18 @@ enum resctrl_domain_type {* @cpu_mask: which CPUs share this resource */ struct rdt_domain_hdr { -struct list_head list; -int id; -enum resctrl_domain_type type; -struct cpumask cpu_mask; +struct list_head list; +int id;
include/linux/resctrl.h enum resctrl_domain_type { · line 133
Quoted 10 lines
include/linux/resctrl.h +8
-0+u32 type; +struct cpumask cpu_mask; }; +/* Bitfields in rdt_domain_hdr.type */ +#define DOMTYPE_RID GENMASK(8, 0) +#define DOMTYPE_CTRL BIT(9) +#define DOMTYPE_MON BIT(10) +#define DOMTYPE (rid, type) (((rid) & DOMTYPE_RID) | (type)) +
This seems unnecessarily complicated to me. Why not just add the resource id to the domain header?
Reinette
Tony Luck
maintainerPATCH v3 7/26 x86/resctrl: Move L3 initialization out of domain_add_cpu_mon()
In reply to Tony LuckTo prepare for additional types of monitoring domains, move all the L3 specific initialization into a helper function.
--- arch/x86/kernel/cpu/resctrl/core.c | 57 ++++++++++++++++++------------ 1 file changed, 35 insertions(+), 22 deletions(-)
arch/x86/kernel/cpu/resctrl/core.c +35 -22
@@ -493,33 +493,12 @@ static void domain_add_cpu_ctrl(int cpu, struct rdt_resource *r) } } -static void domain_add_cpu_mon (int cpu,struct rdt_resource *r) +static void setup_l3_mon_domain (int cpu,int id,struct rdt_resource *r,struct list_head *add_pos) { -int id =get_domain_id_from_scope (cpu, r->mon_scope); -struct list_head *add_pos =NULL ;struct rdt_hw_mon_domain *hw_dom; -struct rdt_domain_hdr *hdr;struct rdt_mon_domain *d;int err; -lockdep_assert_held (&domain_list_lock); - -if (id <0 ) { -pr_warn_once ("Can't find monitor domain id for CPU:%d scope:%d for resource %s\n" , - cpu, r->mon_scope, r->name); -return ; - } - - hdr =resctrl_find_domain (&r->mon_domains, id, &add_pos); -if (hdr) { -if (WARN_ON_ONCE (hdr->type !=DOMTYPE (r->rid,DOMTYPE_MON ))) -return ; - d = container_of(hdr, struct rdt_mon_domain, hdr); - -cpumask_set_cpu (cpu, &d->hdr.cpu_mask); -return ; - } - hw_dom =kzalloc_node (sizeof (*hw_dom),GFP_KERNEL ,cpu_to_node (cpu));if (!hw_dom)return ;
@@ -552,6 +531,40 @@ static void domain_add_cpu_mon(int cpu, struct rdt_resource *r) } } +static void domain_add_cpu_mon (int cpu,struct rdt_resource *r) +{ +int id =get_domain_id_from_scope (cpu, r->mon_scope); +struct list_head *add_pos =NULL ; +struct rdt_domain_hdr *hdr; +struct rdt_mon_domain *d; + +lockdep_assert_held (&domain_list_lock); + +if (id <0 ) { +pr_warn_once ("Can't find monitor domain id for CPU:%d scope:%d for resource %s\n" , + cpu, r->mon_scope, r->name); +return ; + } + + hdr =resctrl_find_domain (&r->mon_domains, id, &add_pos); +if (hdr) { +if (WARN_ON_ONCE (hdr->type !=DOMTYPE (r->rid,DOMTYPE_MON ))) +return ; + d = container_of(hdr, struct rdt_mon_domain, hdr); + +cpumask_set_cpu (cpu, &d->hdr.cpu_mask); +return ; + } + +switch (r->rid) { +case RDT_RESOURCE_L3 : +setup_l3_mon_domain (cpu, id, r, add_pos); +break ; +default : +WARN_ON_ONCE (1 ); + } +} +static void domain_add_cpu (int cpu,struct rdt_resource *r) {if (r->alloc_capable)
Signature · 1 line
2.48.1
Reinette Chatre
maintainerRe: [PATCH v3 07/26] x86/resctrl: Move L3 initialization out of domain_add_cpu_mon()
Hi Tony,
Quoted 34 lines · Tony Luck
To prepare for additional types of monitoring domains, move all the L3 specific initialization into a helper function.
Signed-off-by Tony Luck <[email protected]>--- arch/x86/kernel/cpu/resctrl/core.c | 57 ++++++++++++++++++------------ 1 file changed, 35 insertions(+), 22 deletions(-)arch/x86/kernel/cpu/resctrl/core.c +1
-12@@ -493,33 +493,12 @@ static void domain_add_cpu_ctrl(int cpu, struct rdt_resource *r) } } -static void domain_add_cpu_mon (int cpu,struct rdt_resource *r) +static void setup_l3_mon_domain (int cpu,int id,struct rdt_resource *r,struct list_head *add_pos) { -int id =get_domain_id_from_scope (cpu, r->mon_scope); -struct list_head *add_pos =NULL ;struct rdt_hw_mon_domain *hw_dom; -struct rdt_domain_hdr *hdr;struct rdt_mon_domain *d;int err; -lockdep_assert_held (&domain_list_lock); - -if (id <0 ) { -pr_warn_once ("Can't find monitor domain id for CPU:%d scope:%d for resource %s\n" , - cpu, r->mon_scope, r->name); -return ; - } -
arch/x86/kernel/cpu/resctrl/core.c static void domain_add_cpu_ctrl(int cpu, struct rdt_resource *r) · line 522
Quoted 10 lines
arch/x86/kernel/cpu/resctrl/core.c +0
-10- hdr =resctrl_find_domain (&r->mon_domains, id, &add_pos); -if (hdr) { -if (WARN_ON_ONCE (hdr->type !=DOMTYPE (r->rid,DOMTYPE_MON ))) -return ; - d = container_of(hdr, struct rdt_mon_domain, hdr); - -cpumask_set_cpu (cpu, &d->hdr.cpu_mask); -return ; - } -
Many functions called by this new "setup_l3_mon_domain()" are specific to L3 resource but since L3 resource has so far been the only one supported the naming has been generic. Now that this function is made resource specific I think it will help make the code clear if the L3 specific functions called by it are also renamed. For example, mon_domain_free() can be renamed to free_l3_mon_domain() to match the "setup_l3_mon_domain()" introduced here. Also arch_mon_domain_online() -> arch_l3_mon_domain_online().
Seems like resctrl_online_mon_domain() is only temporarily specific to L3 in this series (would be helpful if such details are explained in changelog) ... to start it could include a check that ensures it is only called on L3 resource and then it will help to clarify what this patch does and how following work builds on it.
arch/x86/kernel/cpu/resctrl/core.c static void domain_add_cpu_mon(int cpu, struct rdt_resource *r) · line 533
Quoted 7 lines
arch/x86/kernel/cpu/resctrl/core.c +0
-0hw_dom =kzalloc_node (sizeof (*hw_dom),GFP_KERNEL ,cpu_to_node (cpu));if (!hw_dom)return ;@@ -552,6 +531,40 @@ static void domain_add_cpu_mon(int cpu, struct rdt_resource *r) } }
Reinette
Luck, Tony
maintainerRe: [PATCH v3 07/26] x86/resctrl: Move L3 initialization out of domain_add_cpu_mon()
Quoted 46 lines · Reinette Chatre
Hi Tony,
Quoted 43 lines · Tony Luck
To prepare for additional types of monitoring domains, move all the L3 specific initialization into a helper function.
Signed-off-by Tony Luck <[email protected]>--- arch/x86/kernel/cpu/resctrl/core.c | 57 ++++++++++++++++++------------ 1 file changed, 35 insertions(+), 22 deletions(-)arch/x86/kernel/cpu/resctrl/core.c +1
-21@@ -493,33 +493,12 @@ static void domain_add_cpu_ctrl(int cpu, struct rdt_resource *r) } } -static void domain_add_cpu_mon (int cpu,struct rdt_resource *r) +static void setup_l3_mon_domain (int cpu,int id,struct rdt_resource *r,struct list_head *add_pos) { -int id =get_domain_id_from_scope (cpu, r->mon_scope); -struct list_head *add_pos =NULL ;struct rdt_hw_mon_domain *hw_dom; -struct rdt_domain_hdr *hdr;struct rdt_mon_domain *d;int err; -lockdep_assert_held (&domain_list_lock); - -if (id <0 ) { -pr_warn_once ("Can't find monitor domain id for CPU:%d scope:%d for resource %s\n" , - cpu, r->mon_scope, r->name); -return ; - } - - hdr =resctrl_find_domain (&r->mon_domains, id, &add_pos); -if (hdr) { -if (WARN_ON_ONCE (hdr->type !=DOMTYPE (r->rid,DOMTYPE_MON ))) -return ; - d = container_of(hdr, struct rdt_mon_domain, hdr); - -cpumask_set_cpu (cpu, &d->hdr.cpu_mask); -return ; - }
Quoted 8 lines
arch/x86/kernel/cpu/resctrl/core.c static void domain_add_cpu_ctrl(int cpu, struct rdt_resource *r) · line 522
Quoted 1 line
arch/x86/kernel/cpu/resctrl/core.c +0
-1-Many functions called by this new "
setup_l3_mon_domain()" are specific to L3 resource but since L3 resource has so far been the only one supported the naming has been generic. Now that this function is made resource specific I think it will help make the code clear if the L3 specific functions called by it are also renamed. For example,mon_domain_free()can be renamed tofree_l3_mon_domain()to match the "setup_l3_mon_domain()" introduced here. Alsoarch_mon_domain_online()->arch_l3_mon_domain_online().
What about "struct rdt_mon_domain"? It is now specific to L3. Should I change that to rdt_l3_mon_domain" as well (60 lines affected)?
Ditto for rdt_hw_mon_domain (but only 12 lines for this one).
Quoted 14 lines
Seems like
resctrl_online_mon_domain()is only temporarily specific to L3 in this series (would be helpful if such details are explained in changelog) ... to start it could include a check that ensures it is only called on L3 resource and then it will help to clarify what this patch does and how following work builds on it.arch/x86/kernel/cpu/resctrl/core.c static void domain_add_cpu_mon(int cpu, struct rdt_resource *r) · line 533
Quoted 7 lines
arch/x86/kernel/cpu/resctrl/core.c +0
-0hw_dom =kzalloc_node (sizeof (*hw_dom),GFP_KERNEL ,cpu_to_node (cpu));if (!hw_dom)return ;@@ -552,6 +531,40 @@ static void domain_add_cpu_mon(int cpu, struct rdt_resource *r) } }Reinette
Reinette Chatre
maintainerRe: [PATCH v3 07/26] x86/resctrl: Move L3 initialization out of domain_add_cpu_mon()
Hi Tony,
Quoted 56 lines · Tony
Quoted 46 lines · Reinette Chatre
Hi Tony,
Quoted 43 lines · Tony Luck
To prepare for additional types of monitoring domains, move all the L3 specific initialization into a helper function.
Signed-off-by Tony Luck <[email protected]>--- arch/x86/kernel/cpu/resctrl/core.c | 57 ++++++++++++++++++------------ 1 file changed, 35 insertions(+), 22 deletions(-)arch/x86/kernel/cpu/resctrl/core.c +1
-21@@ -493,33 +493,12 @@ static void domain_add_cpu_ctrl(int cpu, struct rdt_resource *r) } } -static void domain_add_cpu_mon (int cpu,struct rdt_resource *r) +static void setup_l3_mon_domain (int cpu,int id,struct rdt_resource *r,struct list_head *add_pos) { -int id =get_domain_id_from_scope (cpu, r->mon_scope); -struct list_head *add_pos =NULL ;struct rdt_hw_mon_domain *hw_dom; -struct rdt_domain_hdr *hdr;struct rdt_mon_domain *d;int err; -lockdep_assert_held (&domain_list_lock); - -if (id <0 ) { -pr_warn_once ("Can't find monitor domain id for CPU:%d scope:%d for resource %s\n" , - cpu, r->mon_scope, r->name); -return ; - } - - hdr =resctrl_find_domain (&r->mon_domains, id, &add_pos); -if (hdr) { -if (WARN_ON_ONCE (hdr->type !=DOMTYPE (r->rid,DOMTYPE_MON ))) -return ; - d = container_of(hdr, struct rdt_mon_domain, hdr); - -cpumask_set_cpu (cpu, &d->hdr.cpu_mask); -return ; - }Quoted 8 lines
arch/x86/kernel/cpu/resctrl/core.c static void domain_add_cpu_ctrl(int cpu, struct rdt_resource *r) · line 522
Quoted 1 line
arch/x86/kernel/cpu/resctrl/core.c +0
-1-Many functions called by this new "
setup_l3_mon_domain()" are specific to L3 resource but since L3 resource has so far been the only one supported the naming has been generic. Now that this function is made resource specific I think it will help make the code clear if the L3 specific functions called by it are also renamed. For example,mon_domain_free()can be renamed tofree_l3_mon_domain()to match the "setup_l3_mon_domain()" introduced here. Alsoarch_mon_domain_online()->arch_l3_mon_domain_online().
Quoted 4 lines
What about "struct
rdt_mon_domain"? It is now specific to L3. Should I change that tordt_l3_mon_domain" as well (60 lines affected)?Ditto for
rdt_hw_mon_domain(but only 12 lines for this one).
Thank you for considering this. My vote would be "yes" for both. I think it will help understand and maintain the code if naming helps to make obvious what data/code applies to. Monitoring has been synonymous with L3 monitoring for so long that there may be many instances of this implicit assumption.
Reinette
Tony Luck
maintainerPATCH v3 8/26 x86/resctrl: Refactor domain_remove_cpu_mon() ready for new domain types
In reply to Tony LuckDifferent types of domains require different actions when the last CPU in the domain is removed.
Refactor to make it easy to add new actions for new types of domains.
--- arch/x86/kernel/cpu/resctrl/core.c | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-)
arch/x86/kernel/cpu/resctrl/core.c +8 -6
@@ -644,17 +644,19 @@ static void domain_remove_cpu_mon(int cpu, struct rdt_resource *r)if (WARN_ON_ONCE (hdr->type !=DOMTYPE (r->rid,DOMTYPE_MON )))return ; - d = container_of(hdr, struct rdt_mon_domain, hdr); - hw_dom =resctrl_to_arch_mon_dom (d); +cpumask_clear_cpu (cpu, &hdr->cpu_mask); +if (!cpumask_empty (&hdr->cpu_mask)) +return ; -cpumask_clear_cpu (cpu, &d->hdr.cpu_mask); -if (cpumask_empty (&d->hdr.cpu_mask)) { +switch (r->rid) { +case RDT_RESOURCE_L3 : + d = container_of(hdr, struct rdt_mon_domain, hdr); + hw_dom =resctrl_to_arch_mon_dom (d);resctrl_offline_mon_domain (r, d);list_del_rcu (&d->hdr.list);synchronize_rcu ();mon_domain_free (hw_dom); - -return ; +break ; } }
Signature · 1 line
2.48.1
Reinette Chatre
maintainerRe: [PATCH v3 08/26] x86/resctrl: Refactor domain_remove_cpu_mon() ready for new domain types
Hi Tony,
Quoted 1 line · Tony Luck
Different types of domains require different actions when the last
Please expand/explain what is meant with "different types of domains".
Quoted 8 lines
CPU in the domain is removed.
Refactor to make it easy to add new actions for new types of domains.
Signed-off-by Tony Luck <[email protected]>---
Reinette
Tony Luck
maintainerPATCH v3 9/26 x86/resctrl: Change generic monitor functions to use struct rdt_domain_hdr
In reply to Tony LuckFunctions that don't need the internal details of the rdt_mon_domain can operate on just the rdt_domain_hdr.
Add sanity checks where container_of() is used to find the surrounding domain structure that hdr has the expected type.
--- include/linux/resctrl.h | 2 +- arch/x86/kernel/cpu/resctrl/core.c | 2 +- fs/resctrl/rdtgroup.c | 66 +++++++++++++++++++++--------- 3 files changed, 48 insertions(+), 22 deletions(-)
include/linux/resctrl.h +1 -1
@@ -412,7 +412,7 @@ int resctrl_arch_update_one(struct rdt_resource *r, struct rdt_ctrl_domain *d,u32 resctrl_arch_get_config (struct rdt_resource *r,struct rdt_ctrl_domain *d,u32 closid,enum resctrl_conf_type type);int resctrl_online_ctrl_domain (struct rdt_resource *r,struct rdt_ctrl_domain *d); -int resctrl_online_mon_domain (struct rdt_resource *r,struct rdt_mon_domain *d); +int resctrl_online_mon_domain (struct rdt_resource *r,struct rdt_domain_hdr *hdr);void resctrl_offline_ctrl_domain (struct rdt_resource *r,struct rdt_ctrl_domain *d);void resctrl_offline_mon_domain (struct rdt_resource *r,struct rdt_mon_domain *d);void resctrl_online_cpu (unsigned int cpu);
arch/x86/kernel/cpu/resctrl/core.c +1 -1
@@ -523,7 +523,7 @@ static void setup_l3_mon_domain(int cpu, int id, struct rdt_resource *r, structlist_add_tail_rcu (&d->hdr.list, add_pos); - err =resctrl_online_mon_domain (r, d); + err =resctrl_online_mon_domain (r, &d->hdr);if (err) {list_del_rcu (&d->hdr.list);synchronize_rcu ();
fs/resctrl/rdtgroup.c +46 -20
177 lines not on this page · raw message
Signature · 1 line
2.48.1
Reinette Chatre
maintainerRe: [PATCH v3 09/26] x86/resctrl: Change generic monitor functions to use struct rdt_domain_hdr
Hi Tony,
fs/resctrl/rdtgroup.c int resctrl_online_ctrl_domain(struct rdt_resource *r, struct rdt_ctrl_domain *d · line 4107
Quoted 13 lines · Tony Luck
fs/resctrl/rdtgroup.c +4
-1@@ -4074,10 +4097,13 @@ int resctrl_online_ctrl_domain(struct rdt_resource *r, struct rdt_ctrl_domain *dreturn err; } -int resctrl_online_mon_domain (struct rdt_resource *r,struct rdt_mon_domain *d) +int resctrl_online_mon_domain (struct rdt_resource *r,struct rdt_domain_hdr *hdr) { +struct rdt_mon_domain *d;int err; +WARN_ON_ONCE (hdr->type !=DOMTYPE (r->rid,DOMTYPE_MON )); + d = container_of(hdr, struct rdt_mon_domain, hdr);mutex_lock (&rdtgroup_mutex);
It is a bit unexpected to see code added outside of the mutex_lock(). This looks fine since it is accessing domain list from the hotplug handlers but since it is fs code that is jumped to by arch code and fs code relies on arch code for the locking I'd like to suggest a:
lockdep_assert_cpus_held();
Reinette
Tony Luck
maintainerPATCH v3 10/26 fs/resctrl: Improve handling for events that can be read from any CPU
In reply to Tony LuckAdd a flag to each instance of struct mon_event to indicate that there is no need for cross-processor interrupts to read this event from a CPU in a specific rdt_mon_domain.
The flag is copied to struct mon_data for ease of access when a user reads an event file invoking rdtgroup_mondata_show().
Copied again into struct rmid_read in mon_event_read() for use by sanity checks in __mon_event_count().
When the flag is set allow choice from cpu_online_mask. This makes the smp_call*() functions default to the current CPU.
--- fs/resctrl/internal.h | 8 +++++++- fs/resctrl/ctrlmondata.c | 10 +++++++--- fs/resctrl/monitor.c | 4 ++-- fs/resctrl/rdtgroup.c | 1 + 4 files changed, 17 insertions(+), 6 deletions(-)
fs/resctrl/internal.h +7 -1
@@ -72,6 +72,7 @@ static inline struct rdt_fs_context *rdt_fc2context(struct fs_context *fc)* @evtid: event id * @name: name of the event * @configurable: true if the event is configurable +* @any_cpu: true if this event can be read from any CPU * @list: entry in &rdt_resource->evt_list */ struct mon_evt {
@@ -79,6 +80,7 @@ struct mon_evt {enum resctrl_res_level rid;char *name;bool configurable; +bool any_cpu;struct list_head list; };
@@ -93,6 +95,7 @@ struct mon_evt {* the event file belongs. When @sum is one this * is the id of the L3 cache that all domains to be * summed share. +* @any_cpu: true if this event can be read from any CPU * * Stored in the kernfs kn->priv field, readers and writers must hold * rdtgroup_mutex.
@@ -103,6 +106,7 @@ struct mon_data {enum resctrl_event_id evtid;unsigned int sum;unsigned int domid; +bool any_cpu; };/**
@@ -115,6 +119,7 @@ struct mon_data {* domains in @r sharing L3 @ci.id * @evtid: Which monitor event to read. * @first: Initialize MBM counter when true. +* @any_cpu: When true read can be executed on any CPU. * @ci: Cacheinfo for L3. Only set when @d is NULL. Used when summing domains. * @err: Error encountered when reading counter. * @val: Returned value of event counter. If @rgrp is a parent resource group,
@@ -129,6 +134,7 @@ struct rmid_read {struct rdt_mon_domain *d;enum resctrl_event_id evtid;bool first; +bool any_cpu;struct cacheinfo *ci;int err;u64 val;
@@ -358,7 +364,7 @@ int rdtgroup_mondata_show(struct seq_file *m, void *arg);void mon_event_read (struct rmid_read *rr,struct rdt_resource *r,struct rdt_mon_domain *d,struct rdtgroup *rdtgrp, -cpumask_t *cpumask,int evtid,int first); +const cpumask_t *cpumask,int evtid,int first);int resctrl_mon_resource_init (void );
fs/resctrl/ctrlmondata.c +7 -3
@@ -525,7 +525,7 @@ struct rdt_domain_hdr *resctrl_find_domain(struct list_head *h, int id,void mon_event_read (struct rmid_read *rr,struct rdt_resource *r,struct rdt_mon_domain *d,struct rdtgroup *rdtgrp, -cpumask_t *cpumask,int evtid,int first) +const cpumask_t *cpumask,int evtid,int first) {int cpu;
@@ -571,6 +571,7 @@ int rdtgroup_mondata_show(struct seq_file *m, void *arg)u32 resid, evtid, domid;struct rdtgroup *rdtgrp;struct rdt_resource *r; +const cpumask_t *mask;struct mon_data *md;int ret =0 ;
@@ -589,6 +590,7 @@ int rdtgroup_mondata_show(struct seq_file *m, void *arg) resid = md->rid; domid = md->domid; evtid = md->evtid; + rr.any_cpu = md->any_cpu; r =resctrl_arch_get_resource (resid);if (md->sum) {
@@ -601,8 +603,9 @@ int rdtgroup_mondata_show(struct seq_file *m, void *arg)list_for_each_entry (d, &r->mon_domains, hdr.list) {if (d->ci->id == domid) { rr.ci = d->ci; + mask = md->any_cpu ? cpu_online_mask : &d->ci->shared_cpu_map;mon_event_read (&rr, r,NULL , rdtgrp, - &d->ci->shared_cpu_map, evtid, false); + mask, evtid, false); goto checkresult; } }
@@ -619,7 +622,8 @@ int rdtgroup_mondata_show(struct seq_file *m, void *arg) goto out; } d = container_of(hdr, struct rdt_mon_domain, hdr); -mon_event_read (&rr, r, d, rdtgrp, &d->hdr.cpu_mask, evtid, false); + mask = md->any_cpu ? cpu_online_mask : &d->hdr.cpu_mask; +mon_event_read (&rr, r, d, rdtgrp, mask, evtid, false); } checkresult:
fs/resctrl/monitor.c +2 -2
@@ -375,7 +375,7 @@ static int __mon_event_count(u32 closid, u32 rmid, struct rmid_read *rr)if (rr->d) {/* Reading a single domain, must be on a CPU in that domain. */ -if (!cpumask_test_cpu (cpu, &rr->d->hdr.cpu_mask)) +if (!rr->any_cpu && !cpumask_test_cpu (cpu, &rr->d->hdr.cpu_mask))return -EINVAL ; rr->err =resctrl_arch_rmid_read (rr->r, rr->d, closid, rmid, rr->evtid, &tval, rr->arch_mon_ctx);
@@ -388,7 +388,7 @@ static int __mon_event_count(u32 closid, u32 rmid, struct rmid_read *rr) }/* Summing domains that share a cache, must be on a CPU for that cache. */ -if (!cpumask_test_cpu (cpu, &rr->ci->shared_cpu_map)) +if (!rr->any_cpu && !cpumask_test_cpu (cpu, &rr->ci->shared_cpu_map))return -EINVAL ;/*
fs/resctrl/rdtgroup.c +1 -0
@@ -2926,6 +2926,7 @@ static struct mon_data *mon_get_kn_priv(int rid, int domid, struct mon_evt *mevt priv->domid = domid; priv->sum = do_sum; priv->evtid = mevt->evtid; + priv->any_cpu = mevt->any_cpu;list_add_tail (&priv->list, &kn_priv_list);return priv;
Signature · 1 line
2.48.1
Reinette Chatre
maintainerRe: [PATCH v3 10/26] fs/resctrl: Improve handling for events that can be read from any CPU
Hi Tony,
Quoted 5 lines · Tony Luck
Add a flag to each instance of struct
mon_eventto indicate that there is no need for cross-processor interrupts to read this event from a CPU in a specificrdt_mon_domain.The flag is copied to struct
mon_datafor ease of access when a user
Copy the flag ...
Quoted 7 lines
reads an event file invoking
rdtgroup_mondata_show().Copied again into struct
rmid_readinmon_event_read()for use by sanity checks in__mon_event_count().When the flag is set allow choice from
cpu_online_mask. This makes the smp_call*() functions default to the current CPU.
Please use imperative tone.
Quoted 55 lines
Suggested-by James Morse <[email protected]>Signed-off-by Tony Luck <[email protected]>--- fs/resctrl/internal.h | 8 +++++++- fs/resctrl/ctrlmondata.c | 10 +++++++--- fs/resctrl/monitor.c | 4 ++-- fs/resctrl/rdtgroup.c | 1 + 4 files changed, 17 insertions(+), 6 deletions(-)fs/resctrl/internal.h +5
-0@@ -72,6 +72,7 @@ static inline struct rdt_fs_context *rdt_fc2context(struct fs_context *fc)* @evtid: event id * @name: name of the event * @configurable: true if the event is configurable +* @any_cpu: true if this event can be read from any CPU * @list: entry in &rdt_resource->evt_list */ struct mon_evt {@@ -79,6 +80,7 @@ struct mon_evt {enum resctrl_res_level rid;char *name;bool configurable; +bool any_cpu;struct list_head list; };@@ -93,6 +95,7 @@ struct mon_evt {* the event file belongs. When @sum is one this * is the id of the L3 cache that all domains to be * summed share. +* @any_cpu: true if this event can be read from any CPU * * Stored in the kernfs kn->priv field, readers and writers must hold * rdtgroup_mutex. @@ -103,6 +106,7 @@ struct mon_data {enum resctrl_event_id evtid;unsigned int sum;unsigned int domid; +bool any_cpu; };/** @@ -115,6 +119,7 @@ struct mon_data {* domains in @r sharing L3 @ci.id * @evtid: Which monitor event to read. * @first: Initialize MBM counter when true. +* @any_cpu: When true read can be executed on any CPU. * @ci: Cacheinfo for L3. Only set when @d is NULL. Used when summing domains. * @err: Error encountered when reading counter. * @val: Returned value of event counter. If @rgrp is a parent resource group,
fs/resctrl/internal.h struct rmid_read { · line 140
Quoted 7 lines
fs/resctrl/internal.h +1
-0@@ -129,6 +134,7 @@ struct rmid_read {struct rdt_mon_domain *d;enum resctrl_event_id evtid;bool first; +bool any_cpu;struct cacheinfo *ci;int err;u64 val;
Duplicating the same property across three structures does not look right. It looks to me that struct mon_evt should be the "source of truth" for any event and that these other structures can point to it instead of copying the data?
Quoted 50 lines
fs/resctrl/internal.h +1
-1@@ -358,7 +364,7 @@ int rdtgroup_mondata_show(struct seq_file *m, void *arg);void mon_event_read (struct rmid_read *rr,struct rdt_resource *r,struct rdt_mon_domain *d,struct rdtgroup *rdtgrp, -cpumask_t *cpumask,int evtid,int first); +const cpumask_t *cpumask,int evtid,int first);int resctrl_mon_resource_init (void );fs/resctrl/ctrlmondata.c +5
-2@@ -525,7 +525,7 @@ struct rdt_domain_hdr *resctrl_find_domain(struct list_head *h, int id,void mon_event_read (struct rmid_read *rr,struct rdt_resource *r,struct rdt_mon_domain *d,struct rdtgroup *rdtgrp, -cpumask_t *cpumask,int evtid,int first) +const cpumask_t *cpumask,int evtid,int first) {int cpu;@@ -571,6 +571,7 @@ int rdtgroup_mondata_show(struct seq_file *m, void *arg)u32 resid, evtid, domid;struct rdtgroup *rdtgrp;struct rdt_resource *r; +const cpumask_t *mask;struct mon_data *md;int ret =0 ;@@ -589,6 +590,7 @@ int rdtgroup_mondata_show(struct seq_file *m, void *arg) resid = md->rid; domid = md->domid; evtid = md->evtid; + rr.any_cpu = md->any_cpu; r =resctrl_arch_get_resource (resid);if (md->sum) {@@ -601,8 +603,9 @@ int rdtgroup_mondata_show(struct seq_file *m, void *arg)list_for_each_entry (d, &r->mon_domains, hdr.list) {if (d->ci->id == domid) { rr.ci = d->ci; + mask = md->any_cpu ? cpu_online_mask : &d->ci->shared_cpu_map;mon_event_read (&rr, r,NULL , rdtgrp, - &d->ci->shared_cpu_map, evtid, false); + mask, evtid, false); goto checkresult; } }
fs/resctrl/ctrlmondata.c int rdtgroup_mondata_show(struct seq_file *m, void *arg) · line 626
Quoted 6 lines
fs/resctrl/ctrlmondata.c +2
-1@@ -619,7 +622,8 @@ int rdtgroup_mondata_show(struct seq_file *m, void *arg) goto out; } d = container_of(hdr, struct rdt_mon_domain, hdr); -mon_event_read (&rr, r, d, rdtgrp, &d->hdr.cpu_mask, evtid, false); + mask = md->any_cpu ? cpu_online_mask : &d->hdr.cpu_mask; +mon_event_read (&rr, r, d, rdtgrp, mask, evtid, false);
I do not think this accomplishes the goal of this patch. Looking at mon_event_read() it calls cpumask_any_housekeeping(cpumask, RESCTRL_PICK_ANY_CPU) before any of the smp_*() calls.
cpumask_any_housekeeping()
{
...
if (exclude_cpu == RESCTRL_PICK_ANY_CPU)
cpu = cpumask_any(mask);
...
}cpumask_any() is just cpumask_first() so it will pick the first CPU in the online mask that may not be the current CPU.
fwiw ... there are some optimizations planned in this area that I have not yet studied:
https://lore.kernel.org/lkml/[email protected]/
Reinette
Luck, Tony
maintainerRe: [PATCH v3 10/26] fs/resctrl: Improve handling for events that can be read from any CPU
Quoted 19 lines · Reinette Chatre
fs/resctrl/ctrlmondata.c int rdtgroup_mondata_show(struct seq_file *m, void *arg) · line 626
Quoted 7 lines
fs/resctrl/ctrlmondata.c +2
-1@@ -619,7 +622,8 @@ int rdtgroup_mondata_show(struct seq_file *m, void *arg) goto out; } d = container_of(hdr, struct rdt_mon_domain, hdr); -mon_event_read (&rr, r, d, rdtgrp, &d->hdr.cpu_mask, evtid, false); + mask = md->any_cpu ? cpu_online_mask : &d->hdr.cpu_mask; +mon_event_read (&rr, r, d, rdtgrp, mask, evtid, false);I do not think this accomplishes the goal of this patch. Looking at
mon_event_read()it callscpumask_any_housekeeping(cpumask, RESCTRL_PICK_ANY_CPU)before any of the smp_*() calls.cpumask_any_housekeeping() { ... if (exclude_cpu == RESCTRL_PICK_ANY_CPU) cpu = cpumask_any(mask); ... }
Quoted 5 lines
cpumask_any()is justcpumask_first()so it will pick the first CPU in the online mask that may not be the current CPU.fwiw ... there are some optimizations planned in this area that I have not yet studied:
https://lore.kernel.org/lkml/[email protected]/
I remember Peter complaining[1] about extra context switches when cpumask_any_housekeeping() was introduced, but it seems that the discussion died with no fix applied.
The blocking problem is that ARM may not be able to read a counter on a tick_nohz CPU because it may need to sleep.
Do we need more options for events:
1) Must be read on a CPU in the right domain // Legacy
2) Can be read from any CPU // My addtion
3) Must be read on a "housekeeping" CPU // James' code in upstream
4) Cannot be read on a tick_nohz CPU // Could be combined with 1 or 2?
Quoted 1 line
Reinette
[1] https://lore.kernel.org/all/[email protected]/
Quoted 1 line
Reinette Chatre
maintainerRe: [PATCH v3 10/26] fs/resctrl: Improve handling for events that can be read from any CPU
Hi Tony,
Quoted 23 lines · Tony
Quoted 22 lines · Reinette Chatre
fs/resctrl/ctrlmondata.c int rdtgroup_mondata_show(struct seq_file *m, void *arg) · line 626
Quoted 7 lines
fs/resctrl/ctrlmondata.c +2
-1@@ -619,7 +622,8 @@ int rdtgroup_mondata_show(struct seq_file *m, void *arg) goto out; } d = container_of(hdr, struct rdt_mon_domain, hdr); -mon_event_read (&rr, r, d, rdtgrp, &d->hdr.cpu_mask, evtid, false); + mask = md->any_cpu ? cpu_online_mask : &d->hdr.cpu_mask; +mon_event_read (&rr, r, d, rdtgrp, mask, evtid, false);I do not think this accomplishes the goal of this patch. Looking at
mon_event_read()it callscpumask_any_housekeeping(cpumask, RESCTRL_PICK_ANY_CPU)before any of the smp_*() calls.cpumask_any_housekeeping() { ... if (exclude_cpu == RESCTRL_PICK_ANY_CPU) cpu = cpumask_any(mask); ... }
cpumask_any()is justcpumask_first()so it will pick the first CPU in the online mask that may not be the current CPU.
Quoted 6 lines
Quoted 2 lines
fwiw ... there are some optimizations planned in this area that I have not yet studied:
https://lore.kernel.org/lkml/[email protected]/I remember Peter complaining[1] about extra context switches when
cpumask_any_housekeeping()was introduced, but it seems that the discussion died with no fix applied.
The initial complaint was indeed that reading individual events is slower.
The issue is that the intended use case read from many files at frequent intervals and thus becomes vulnerable to any changes in this area that really is already a slow path (reading from a file ... taking a mutex ...).
Instead of working on shaving cycles off this path the discussion transitioned to resctrl providing better support for the underlying use case. I understood that this is being experimented with [2] and last I heard it looks promising.
Quoted 10 lines
The blocking problem is that ARM may not be able to read a counter on a
tick_nohzCPU because it may need to sleep.Do we need more options for events:
1) Must be read on a CPU in the right domain // Legacy
2) Can be read from any CPU // My addtion
3) Must be read on a "housekeeping" CPU // James' code in upstream
4) Cannot be read on atick_nohzCPU // Could be combined with 1 or 2?
I do not see needing additional complexity here. I think it will be simpler
to just replace use of cpumask_any_housekeeping() in mon_event_read() with
open code that supports the particular usage. As I understand it is prohibited for all CPUs to be in tick_nohz_full_mask so it looks to me as though the existing "if (tick_nohz_full_cpu(cpu))" should never be true (since no CPU is being excluded). Also, since mon_event_read() has no need to exclude CPUs, just a cpumask_andnot() should suffice to determine what remains of given mask after accounting for all the NO_HZ CPUs if tick_nohz_full_enabled().
Reinette
Quoted 5 lines
Quoted 2 lines
Reinette
[1] https://lore.kernel.org/all/[email protected]/
Quoted 1 line
[2] https://lore.kernel.org/lkml/CALPaoCgpnVORZfbKVLXDFUZvv8jhpShHPzB3cwdLTZQH1o9ULw@mail.gmail.com/
Luck, Tony
maintainerRE: [PATCH v3 10/26] fs/resctrl: Improve handling for events that can be read from any CPU
Quoted 32 lines
Quoted 9 lines
Quoted 5 lines
cpumask_any()is justcpumask_first()so it will pick the first CPU in the online mask that may not be the current CPU.fwiw ... there are some optimizations planned in this area that I have not yet studied:
https://lore.kernel.org/lkml/[email protected]/I remember Peter complaining[1] about extra context switches when
cpumask_any_housekeeping()was introduced, but it seems that the discussion died with no fix applied.The initial complaint was indeed that reading individual events is slower.
The issue is that the intended use case read from many files at frequent intervals and thus becomes vulnerable to any changes in this area that really is already a slow path (reading from a file ... taking a mutex ...).
Instead of working on shaving cycles off this path the discussion transitioned to resctrl providing better support for the underlying use case. I understood that this is being experimented with [2] and last I heard it looks promising.
Quoted 10 lines
The blocking problem is that ARM may not be able to read a counter on a
tick_nohzCPU because it may need to sleep.Do we need more options for events:
1) Must be read on a CPU in the right domain // Legacy
2) Can be read from any CPU // My addtion
3) Must be read on a "housekeeping" CPU // James' code in upstream
4) Cannot be read on atick_nohzCPU // Could be combined with 1 or 2?
Quoted 8 lines
I do not see needing additional complexity here. I think it will be simpler
to just replace use ofcpumask_any_housekeeping()inmon_event_read()with
open code that supports the particular usage. As I understand it is prohibited for all CPUs to be intick_nohz_full_maskso it looks to me as though the existing "if (tick_nohz_full_cpu(cpu))" should never be true (since no CPU is being excluded). Also, sincemon_event_read()has no need to exclude CPUs, just acpumask_andnot()should suffice to determine what remains of given mask after accounting for all theNO_HZCPUs iftick_nohz_full_enabled().
Maybe there isn’t much complexity to make this "read one counter" better on systems where reading from any CPU is possible. Taking your advice from the earlier review the filesystem code can set a flag in the mon_evt structure. struct mon_data and struct rmid_read can change from holding the event id to holding a pointer to the mon_evt (as the source of truth).
Then mon_event_read() can just have a simple direct call to mon_event_count() just before the call to cpumask_any_housekeeping() like this:
if (evt->any_cpu) {
mon_event_count(rr);
goto done;
}The "goto done" jumps to the resctrl_arch_mon_ctx_free(r, evt->evtid, rr->arch_mon_ctx); at the end of mon_event_read()
Folks can still pursue the bulk read of many counters (though I expect you might want one file per domain, rather than a single file to report everything).
Quoted 10 lines
Reinette
Quoted 5 lines
Quoted 2 lines
Reinette
[1] https://lore.kernel.org/all/[email protected]/
Quoted 1 line
[2] https://lore.kernel.org/lkml/CALPaoCgpnVORZfbKVLXDFUZvv8jhpShHPzB3cwdLTZQH1o9ULw@mail.gmail.com/
-Tony
Reinette Chatre
maintainerRe: [PATCH v3 10/26] fs/resctrl: Improve handling for events that can be read from any CPU
Hi Tony,
Quoted 50 lines · Tony
Quoted 32 lines
Quoted 9 lines
Quoted 5 lines
cpumask_any()is justcpumask_first()so it will pick the first CPU in the online mask that may not be the current CPU.fwiw ... there are some optimizations planned in this area that I have not yet studied:
https://lore.kernel.org/lkml/[email protected]/I remember Peter complaining[1] about extra context switches when
cpumask_any_housekeeping()was introduced, but it seems that the discussion died with no fix applied.The initial complaint was indeed that reading individual events is slower.
The issue is that the intended use case read from many files at frequent intervals and thus becomes vulnerable to any changes in this area that really is already a slow path (reading from a file ... taking a mutex ...).
Instead of working on shaving cycles off this path the discussion transitioned to resctrl providing better support for the underlying use case. I understood that this is being experimented with [2] and last I heard it looks promising.
Quoted 10 lines
The blocking problem is that ARM may not be able to read a counter on a
tick_nohzCPU because it may need to sleep.Do we need more options for events:
1) Must be read on a CPU in the right domain // Legacy
2) Can be read from any CPU // My addtion
3) Must be read on a "housekeeping" CPU // James' code in upstream
4) Cannot be read on atick_nohzCPU // Could be combined with 1 or 2?Quoted 8 lines
I do not see needing additional complexity here. I think it will be simpler
to just replace use ofcpumask_any_housekeeping()inmon_event_read()with
open code that supports the particular usage. As I understand it is prohibited for all CPUs to be intick_nohz_full_maskso it looks to me as though the existing "if (tick_nohz_full_cpu(cpu))" should never be true (since no CPU is being excluded). Also, sincemon_event_read()has no need to exclude CPUs, just acpumask_andnot()should suffice to determine what remains of given mask after accounting for all theNO_HZCPUs iftick_nohz_full_enabled().Maybe there isn’t much complexity to make this "read one counter" better on systems where reading from any CPU is possible. Taking your advice from the earlier review the filesystem code can set a flag in the
mon_evtstructure. structmon_dataand structrmid_readcan change from holding the event id to holding a pointer to themon_evt(as the source of truth).Then
mon_event_read()can just have a simple direct call tomon_event_count()just before the call tocpumask_any_housekeeping()like this:
Quoted 7 lines
if (evt->any_cpu) { mon_event_count(rr); goto done; }The "goto done" jumps to the
resctrl_arch_mon_ctx_free(r, evt->evtid, rr->arch_mon_ctx); at the end ofmon_event_read()
Thanks, this looks great.
Quoted 3 lines
Folks can still pursue the bulk read of many counters (though I expect you might want one file per domain, rather than a single file to report everything).
I will have to re-read that thread to refresh myself on the discussion but scanning the thread I did find a summary of points (end of [3]) and this was there.
Quoted 5 lines
Quoted 5 lines
Reinette
Quoted 1 line
Quoted 7 lines
Quoted 6 lines
Quoted 4 lines
Quoted 1 line
Reinette
[1] https://lore.kernel.org/all/[email protected]/
Quoted 1 line
[2] https://lore.kernel.org/lkml/CALPaoCgpnVORZfbKVLXDFUZvv8jhpShHPzB3cwdLTZQH1o9ULw@mail.gmail.com/
[3] https://lore.kernel.org/lkml/[email protected]/
Reinette
Re: [PATCH v3 10/26] fs/resctrl: Improve handling for events that can be read from any CPU
In reply to Reinette ChatreHi Reinette,
Quoted 36 lines · Reinette Chatre
Hi Tony,
Quoted 23 lines · Tony
Quoted 22 lines · Reinette Chatre
fs/resctrl/ctrlmondata.c int rdtgroup_mondata_show(struct seq_file *m, void *arg) · line 626
Quoted 7 lines
fs/resctrl/ctrlmondata.c +2
-1@@ -619,7 +622,8 @@ int rdtgroup_mondata_show(struct seq_file *m, void *arg) goto out; } d = container_of(hdr, struct rdt_mon_domain, hdr); -mon_event_read (&rr, r, d, rdtgrp, &d->hdr.cpu_mask, evtid, false); + mask = md->any_cpu ? cpu_online_mask : &d->hdr.cpu_mask; +mon_event_read (&rr, r, d, rdtgrp, mask, evtid, false);I do not think this accomplishes the goal of this patch. Looking at
mon_event_read()it callscpumask_any_housekeeping(cpumask, RESCTRL_PICK_ANY_CPU)before any of the smp_*() calls.cpumask_any_housekeeping() { ... if (exclude_cpu == RESCTRL_PICK_ANY_CPU) cpu = cpumask_any(mask); ... }
cpumask_any()is justcpumask_first()so it will pick the first CPU in the online mask that may not be the current CPU.Quoted 6 lines
Quoted 2 lines
fwiw ... there are some optimizations planned in this area that I have not yet studied:
https://lore.kernel.org/lkml/[email protected]/I remember Peter complaining[1] about extra context switches when
cpumask_any_housekeeping()was introduced, but it seems that the discussion died with no fix applied.The initial complaint was indeed that reading individual events is slower.
Quoted 12 lines
The issue is that the intended use case read from many files at frequent intervals and thus becomes vulnerable to any changes in this area that really is already a slow path (reading from a file ... taking a mutex ...).
Instead of working on shaving cycles off this path the discussion transitioned to resctrl providing better support for the underlying use case. I understood that this is being experimented with [2] and last I heard it looks promising.
Quoted 3 lines
The blocking problem is that ARM may not be able to read a counter on a
tick_nohzCPU because it may need to sleep.
If I hadn't already turned my attention to optimizing bulk counter reads, I might have mentioned that the change Tony referred to is broken on MPAM implementations because the MPAM resctrl_arch_rmid_read() cannot wait for its internal mutex with preemption disabled.
Quoted 8 lines
Quoted 7 lines
Do we need more options for events:
1) Must be read on a CPU in the right domain // Legacy
2) Can be read from any CPU // My addtion
3) Must be read on a "housekeeping" CPU // James' code in upstream
4) Cannot be read on atick_nohzCPU // Could be combined with 1 or 2?
Quoted 8 lines
I do not see needing additional complexity here. I think it will be simpler
to just replace use ofcpumask_any_housekeeping()inmon_event_read()with
open code that supports the particular usage. As I understand it is prohibited for all CPUs to be intick_nohz_full_maskso it looks to me as though the existing "if (tick_nohz_full_cpu(cpu))" should never be true (since no CPU is being excluded). Also, sincemon_event_read()has no need to exclude CPUs, just acpumask_andnot()should suffice to determine what remains of given mask after accounting for all theNO_HZCPUs iftick_nohz_full_enabled().
Can you clarify what you mean by "all CPUs"? It's not difficult for all CPUs in an L3 domain to be in tick_nohz_full_mask on AMD implementations, where there are many small L3 domains (~8 CPUs each) in a socket.
Google makes use of isolation along this domain boundary on AMD platforms in some products and these users prefer to read counters using IPIs because they are concerned about introducing context switches to the isolated part of the system. In these configurations, there is typically only one RMID in that domain, so few of these IPIs are needed. (Note that these are different users from the ones I had described before who spawn large numbers of containers not limited to any domains and want to read the MBM counters for all the RMIDs on all the domains frequently.)
Thanks,
-Peter
Reinette Chatre
maintainerRe: [PATCH v3 10/26] fs/resctrl: Improve handling for events that can be read from any CPU
Hi Peter,
Quoted 76 lines · Peter Newman
Hi Reinette,
Quoted 36 lines · Reinette Chatre
Hi Tony,
Quoted 23 lines · Tony
Quoted 22 lines · Reinette Chatre
fs/resctrl/ctrlmondata.c int rdtgroup_mondata_show(struct seq_file *m, void *arg) · line 626
Quoted 7 lines
fs/resctrl/ctrlmondata.c +2
-1@@ -619,7 +622,8 @@ int rdtgroup_mondata_show(struct seq_file *m, void *arg) goto out; } d = container_of(hdr, struct rdt_mon_domain, hdr); -mon_event_read (&rr, r, d, rdtgrp, &d->hdr.cpu_mask, evtid, false); + mask = md->any_cpu ? cpu_online_mask : &d->hdr.cpu_mask; +mon_event_read (&rr, r, d, rdtgrp, mask, evtid, false);I do not think this accomplishes the goal of this patch. Looking at
mon_event_read()it callscpumask_any_housekeeping(cpumask, RESCTRL_PICK_ANY_CPU)before any of the smp_*() calls.cpumask_any_housekeeping() { ... if (exclude_cpu == RESCTRL_PICK_ANY_CPU) cpu = cpumask_any(mask); ... }
cpumask_any()is justcpumask_first()so it will pick the first CPU in the online mask that may not be the current CPU.Quoted 6 lines
Quoted 2 lines
fwiw ... there are some optimizations planned in this area that I have not yet studied:
https://lore.kernel.org/lkml/[email protected]/I remember Peter complaining[1] about extra context switches when
cpumask_any_housekeeping()was introduced, but it seems that the discussion died with no fix applied.The initial complaint was indeed that reading individual events is slower.
Quoted 12 lines
The issue is that the intended use case read from many files at frequent intervals and thus becomes vulnerable to any changes in this area that really is already a slow path (reading from a file ... taking a mutex ...).
Instead of working on shaving cycles off this path the discussion transitioned to resctrl providing better support for the underlying use case. I understood that this is being experimented with [2] and last I heard it looks promising.
Quoted 3 lines
The blocking problem is that ARM may not be able to read a counter on a
tick_nohzCPU because it may need to sleep.If I hadn't already turned my attention to optimizing bulk counter reads, I might have mentioned that the change Tony referred to is broken on MPAM implementations because the MPAM
resctrl_arch_rmid_read()cannot wait for its internal mutex with preemption disabled.Quoted 8 lines
Quoted 7 lines
Do we need more options for events:
1) Must be read on a CPU in the right domain // Legacy
2) Can be read from any CPU // My addtion
3) Must be read on a "housekeeping" CPU // James' code in upstream
4) Cannot be read on atick_nohzCPU // Could be combined with 1 or 2?Quoted 8 lines
I do not see needing additional complexity here. I think it will be simpler
to just replace use ofcpumask_any_housekeeping()inmon_event_read()with
open code that supports the particular usage. As I understand it is prohibited for all CPUs to be intick_nohz_full_maskso it looks to me as though the existing "if (tick_nohz_full_cpu(cpu))" should never be true (since no CPU is being excluded). Also, sincemon_event_read()has no need to exclude CPUs, just acpumask_andnot()should suffice to determine what remains of given mask after accounting for all theNO_HZCPUs iftick_nohz_full_enabled().
Quoted 1 line
Can you clarify what you mean by "all CPUs"? It's not difficult for
I mentioned this in the context of this patch that adds support for events that can be ready from *any* CPU. The CPU reading the event data need not be in the domain for which data is being read so all CPUs on the system are available to the flow supporting these events. Since all CPUs on the system cannot be in tick_nohz_full_mask there will always be a CPU available to read this type of event that can be read from any CPU.
I made it way too complicated with this though. Tony proposed something much better and simpler [1].
Quoted 4 lines
all CPUs in an L3 domain to be in
tick_nohz_full_maskon AMD implementations, where there are many small L3 domains (~8 CPUs each) in a socket.
Quoted 10 lines
Google makes use of isolation along this domain boundary on AMD platforms in some products and these users prefer to read counters using IPIs because they are concerned about introducing context switches to the isolated part of the system. In these configurations, there is typically only one RMID in that domain, so few of these IPIs are needed. (Note that these are different users from the ones I had described before who spawn large numbers of containers not limited to any domains and want to read the MBM counters for all the RMIDs on all the domains frequently.)
Thank you for this insight. There is no change planned for reading event counters for those events that need to be read from their domain. Tony's recent proposal [1] moves the handling of these new style of events to a separate branch.
Reinette
Tony Luck
maintainerPATCH v3 11/26 fs/resctrl: Add support for additional monitor event display formats
In reply to Tony LuckAdd a type field to both the mon_evt and mon_data structures.
Legacy monitor events are still all displayed as an unsigned decimal 64-bit integer.
Add an additional format of fixed-point with 18 binary places displayed as a floating point value with six decimal places.
--- include/linux/resctrl_types.h | 8 ++++++++ fs/resctrl/internal.h | 4 ++++ fs/resctrl/ctrlmondata.c | 23 ++++++++++++++++++++++- fs/resctrl/monitor.c | 3 +++ fs/resctrl/rdtgroup.c | 1 + 5 files changed, 38 insertions(+), 1 deletion(-)
include/linux/resctrl_types.h +8 -0
@@ -58,6 +58,14 @@ enum resctrl_event_id {#define QOS_NUM_MBM_EVENTS (QOS_L3_MBM_LOCAL_EVENT_ID - QOS_L3_MBM_TOTAL_EVENT_ID + 1)#define MBM_EVENT_IDX (evt) ((evt) - QOS_L3_MBM_TOTAL_EVENT_ID) +/* +* Event value display types +*/ +enum resctrl_event_type { +EVT_TYPE_U64 , +EVT_TYPE_U46_18 , +}; +static inline bool resctrl_is_mbm_event (int e) { return (e >= QOS_L3_MBM_TOTAL_EVENT_ID &&
fs/resctrl/internal.h +4 -0
@@ -71,6 +71,7 @@ static inline struct rdt_fs_context *rdt_fc2context(struct fs_context *fc)* struct mon_evt - Entry in the event list of a resource * @evtid: event id * @name: name of the event +* @type: format for display to user * @configurable: true if the event is configurable * @any_cpu: true if this event can be read from any CPU * @list: entry in &rdt_resource->evt_list
@@ -79,6 +80,7 @@ struct mon_evt {enum resctrl_event_id evtid;enum resctrl_res_level rid;char *name; +enum resctrl_event_type type;bool configurable;bool any_cpu;struct list_head list;
@@ -89,6 +91,7 @@ struct mon_evt {* @list: List of all allocated structures. * @rid: Resource id associated with the event file. * @evtid: Event id associated with the event file. +* @type: Format for display to user * @sum: Set when event must be summed across multiple * domains. * @domid: When @sum is zero this is the domain to which
@@ -104,6 +107,7 @@ struct mon_data {struct list_head list;unsigned int rid;enum resctrl_event_id evtid; +enum resctrl_event_type type;unsigned int sum;unsigned int domid;bool any_cpu;
fs/resctrl/ctrlmondata.c +22 -1
@@ -562,6 +562,27 @@ void mon_event_read(struct rmid_read *rr, struct rdt_resource *r,resctrl_arch_mon_ctx_free (r, evtid, rr->arch_mon_ctx); } +#define NUM_FRAC_BITS 18 +#define FRAC_MASK GENMASK(NUM_FRAC_BITS - 1, 0) + +static void show_value (struct seq_file *m,enum resctrl_event_type type,u64 val) +{ +u64 frac; + +switch (type) { +case EVT_TYPE_U64 : +seq_printf (m,"%llu\n" , val); +break ; +case EVT_TYPE_U46_18 : + frac = val &FRAC_MASK ; + frac = frac *1000000 ; + frac +=1ul << (NUM_FRAC_BITS -1 ); + frac >>=NUM_FRAC_BITS ; +seq_printf (m,"%llu.%06llu\n" , val >>NUM_FRAC_BITS , frac); +break ; + } +} +int rdtgroup_mondata_show (struct seq_file *m,void *arg) {struct kernfs_open_file *of = m->private;
@@ -633,7 +654,7 @@ int rdtgroup_mondata_show(struct seq_file *m, void *arg)else if (rr.err == -EINVAL )seq_puts (m,"Unavailable\n" );else -seq_printf (m,"%llu\n" , rr.val); +show_value (m, md->type, rr.val); out:rdtgroup_kn_unlock (of->kn);
fs/resctrl/monitor.c +3 -0
@@ -847,16 +847,19 @@ static struct mon_evt all_events[QOS_NUM_EVENTS] = { .name = "llc_occupancy", .evtid = QOS_L3_OCCUP_EVENT_ID, .rid = RDT_RESOURCE_L3, + .type = EVT_TYPE_U64, }, [QOS_L3_MBM_TOTAL_EVENT_ID] = { .name = "mbm_total_bytes", .evtid = QOS_L3_MBM_TOTAL_EVENT_ID, .rid = RDT_RESOURCE_L3, + .type = EVT_TYPE_U64, }, [QOS_L3_MBM_LOCAL_EVENT_ID] = { .name = "mbm_local_bytes", .evtid = QOS_L3_MBM_LOCAL_EVENT_ID, .rid = RDT_RESOURCE_L3, + .type = EVT_TYPE_U64, }, };
fs/resctrl/rdtgroup.c +1 -0
@@ -2927,6 +2927,7 @@ static struct mon_data *mon_get_kn_priv(int rid, int domid, struct mon_evt *mevt priv->sum = do_sum; priv->evtid = mevt->evtid; priv->any_cpu = mevt->any_cpu; + priv->type = mevt->type;list_add_tail (&priv->list, &kn_priv_list);return priv;
Signature · 1 line
2.48.1
Reinette Chatre
maintainerRe: [PATCH v3 11/26] fs/resctrl: Add support for additional monitor event display formats
Hi Tony,
Please add context.
Quoted 1 line · Tony Luck
Add a type field to both the
mon_evtandmon_datastructures.
This is getting too much? How about mon_data pointing to mon_evt?
include/linux/resctrl_types.h
Quoted 20 lines
Legacy monitor events are still all displayed as an unsigned decimal 64-bit integer.
Add an additional format of fixed-point with 18 binary places displayed as a floating point value with six decimal places.
Signed-off-by Tony Luck <[email protected]>--- include/linux/resctrl_types.h | 8 ++++++++ fs/resctrl/internal.h | 4 ++++ fs/resctrl/ctrlmondata.c | 23 ++++++++++++++++++++++- fs/resctrl/monitor.c | 3 +++ fs/resctrl/rdtgroup.c | 1 + 5 files changed, 38 insertions(+), 1 deletion(-)include/linux/resctrl_types.h +0
-0
Why is this exposed to architecture? There are no users in arch code. changelog gives no hints on how to interpret this.
include/linux/resctrl_types.h enum resctrl_event_id { · line 62
Quoted 6 lines
include/linux/resctrl_types.h +2
-0@@ -58,6 +58,14 @@ enum resctrl_event_id {#define QOS_NUM_MBM_EVENTS (QOS_L3_MBM_LOCAL_EVENT_ID - QOS_L3_MBM_TOTAL_EVENT_ID + 1)#define MBM_EVENT_IDX (evt) ((evt) - QOS_L3_MBM_TOTAL_EVENT_ID) +/* +* Event value display types
How about "Event value display formats"?
include/linux/resctrl_types.h enum resctrl_event_id { · line 64
Quoted 2 lines
include/linux/resctrl_types.h +2
-0+*/ +enum resctrl_event_type {
resctrl_event_format/resctrl_event_fmt?
include/linux/resctrl_types.h enum resctrl_event_id { · line 65
Quoted 1 line
include/linux/resctrl_types.h +1
-0+ EVT_TYPE_U64,
EVT_FMT_U64/EVT_FORMAT_U64?
Quoted 47 lines
include/linux/resctrl_types.h +3
-0+ EVT_TYPE_U46_18, +}; +static inline bool resctrl_is_mbm_event (int e) { return (e >= QOS_L3_MBM_TOTAL_EVENT_ID &&fs/resctrl/internal.h +4
-0@@ -71,6 +71,7 @@ static inline struct rdt_fs_context *rdt_fc2context(struct fs_context *fc)* struct mon_evt - Entry in the event list of a resource * @evtid: event id * @name: name of the event +* @type: format for display to user * @configurable: true if the event is configurable * @any_cpu: true if this event can be read from any CPU * @list: entry in &rdt_resource->evt_list @@ -79,6 +80,7 @@ struct mon_evt {enum resctrl_event_id evtid;enum resctrl_res_level rid;char *name; +enum resctrl_event_type type;bool configurable;bool any_cpu;struct list_head list;@@ -89,6 +91,7 @@ struct mon_evt {* @list: List of all allocated structures. * @rid: Resource id associated with the event file. * @evtid: Event id associated with the event file. +* @type: Format for display to user * @sum: Set when event must be summed across multiple * domains. * @domid: When @sum is zero this is the domain to which @@ -104,6 +107,7 @@ struct mon_data {struct list_head list;unsigned int rid;enum resctrl_event_id evtid; +enum resctrl_event_type type;unsigned int sum;unsigned int domid;bool any_cpu;fs/resctrl/ctrlmondata.c +0
-0
fs/resctrl/ctrlmondata.c void mon_event_read(struct rmid_read *rr, struct rdt_resource *r, · line 568
Quoted 7 lines
fs/resctrl/ctrlmondata.c +4
-0@@ -562,6 +562,27 @@ void mon_event_read(struct rmid_read *rr, struct rdt_resource *r,resctrl_arch_mon_ctx_free (r, evtid, rr->arch_mon_ctx); } +#define NUM_FRAC_BITS 18 +#define FRAC_MASK GENMASK(NUM_FRAC_BITS - 1, 0) + +static void show_value (struct seq_file *m,enum resctrl_event_type type,u64 val)
show_value() is a bit vague ... print_event_value() ?
fs/resctrl/ctrlmondata.c void mon_event_read(struct rmid_read *rr, struct rdt_resource *r, · line 579
Quoted 11 lines
fs/resctrl/ctrlmondata.c +11
-0+{ +u64 frac; + +switch (type) { +case EVT_TYPE_U64 : +seq_printf (m,"%llu\n" , val); +break ; +case EVT_TYPE_U46_18 : + frac = val &FRAC_MASK ; + frac = frac *1000000 ; + frac +=1ul << (NUM_FRAC_BITS -1 );
Could you please help me understand why above line is needed? Seems like shift below will just undo it?
Quoted 47 lines
fs/resctrl/ctrlmondata.c +7
-1+ frac >>=NUM_FRAC_BITS ; +seq_printf (m,"%llu.%06llu\n" , val >>NUM_FRAC_BITS , frac); +break ; + } +} +int rdtgroup_mondata_show (struct seq_file *m,void *arg) {struct kernfs_open_file *of = m->private;@@ -633,7 +654,7 @@ int rdtgroup_mondata_show(struct seq_file *m, void *arg)else if (rr.err == -EINVAL )seq_puts (m,"Unavailable\n" );else -seq_printf (m,"%llu\n" , rr.val); +show_value (m, md->type, rr.val); out:rdtgroup_kn_unlock (of->kn);fs/resctrl/monitor.c +3
-0@@ -847,16 +847,19 @@ static struct mon_evt all_events[QOS_NUM_EVENTS] = { .name = "llc_occupancy", .evtid = QOS_L3_OCCUP_EVENT_ID, .rid = RDT_RESOURCE_L3, + .type = EVT_TYPE_U64, }, [QOS_L3_MBM_TOTAL_EVENT_ID] = { .name = "mbm_total_bytes", .evtid = QOS_L3_MBM_TOTAL_EVENT_ID, .rid = RDT_RESOURCE_L3, + .type = EVT_TYPE_U64, }, [QOS_L3_MBM_LOCAL_EVENT_ID] = { .name = "mbm_local_bytes", .evtid = QOS_L3_MBM_LOCAL_EVENT_ID, .rid = RDT_RESOURCE_L3, + .type = EVT_TYPE_U64, }, };fs/resctrl/rdtgroup.c +0
-0
fs/resctrl/rdtgroup.c static struct mon_data *mon_get_kn_priv(int rid, int domid, struct mon_evt *mevt · line 2933
Quoted 7 lines
fs/resctrl/rdtgroup.c +1
-0@@ -2927,6 +2927,7 @@ static struct mon_data *mon_get_kn_priv(int rid, int domid, struct mon_evt *mevt priv->sum = do_sum; priv->evtid = mevt->evtid; priv->any_cpu = mevt->any_cpu; + priv->type = mevt->type;list_add_tail (&priv->list, &kn_priv_list);return priv;
Reinette
Luck, Tony
maintainerRe: [PATCH v3 11/26] fs/resctrl: Add support for additional monitor event display formats
Quoted 8 lines · Reinette Chatre
fs/resctrl/ctrlmondata.c void mon_event_read(struct rmid_read *rr, struct rdt_resource *r, · line 579
Quoted 4 lines
fs/resctrl/ctrlmondata.c +4
-0+case EVT_TYPE_U46_18 : + frac = val &FRAC_MASK ; + frac = frac *1000000 ; + frac +=1ul << (NUM_FRAC_BITS -1 );Could you please help me understand why above line is needed? Seems like shift below will just undo it?
Quoted 4 lines
fs/resctrl/ctrlmondata.c void mon_event_read(struct rmid_read *rr, struct rdt_resource *r, · line 583
Quoted 4 lines
fs/resctrl/ctrlmondata.c +4
-0+ frac >>=NUM_FRAC_BITS ; +seq_printf (m,"%llu.%06llu\n" , val >>NUM_FRAC_BITS , frac); +break ; + }
The extra addtion is to round the value to nearest decimal supported value.
E.g. take the case where val == 1 This is a 1 in the 18th binary place, so the precise decimal representation is:
1 / 2^18 = 0.000003814697265625
rounding to six decimal places should give: 0.000004
If you run the above code without the small addition you get:
frac = val & FRAC_MASK; // frac == 1 frac = frac * 1000000; // frac == 1000000 frac >>= NUM_FRAC_BITS; // frac == 3
So the output with be the truncated, not rounded, 0.000003
The addition will have a "carry bit" in to the upper bits. That isn't lost when shifting right. The value added is as if there was a "1" in the 19th binary place.
-Tony
Reinette Chatre
maintainerRe: [PATCH v3 11/26] fs/resctrl: Add support for additional monitor event display formats
Hi Tony,
Quoted 30 lines · Tony
Quoted 8 lines · Reinette Chatre
fs/resctrl/ctrlmondata.c void mon_event_read(struct rmid_read *rr, struct rdt_resource *r, · line 579
Quoted 4 lines
fs/resctrl/ctrlmondata.c +4
-0+case EVT_TYPE_U46_18 : + frac = val &FRAC_MASK ; + frac = frac *1000000 ; + frac +=1ul << (NUM_FRAC_BITS -1 );Could you please help me understand why above line is needed? Seems like shift below will just undo it?
Quoted 4 lines
fs/resctrl/ctrlmondata.c void mon_event_read(struct rmid_read *rr, struct rdt_resource *r, · line 583
Quoted 4 lines
fs/resctrl/ctrlmondata.c +4
-0+ frac >>=NUM_FRAC_BITS ; +seq_printf (m,"%llu.%06llu\n" , val >>NUM_FRAC_BITS , frac); +break ; + }The extra addtion is to round the value to nearest decimal supported value.
E.g. take the case where val == 1 This is a 1 in the 18th binary place, so the precise decimal representation is:
1 / 2^18 = 0.000003814697265625rounding to six decimal places should give: 0.000004
If you run the above code without the small addition you get:
frac = val & FRAC_MASK; // frac == 1 frac = frac * 1000000; // frac == 1000000 frac >>= NUM_FRAC_BITS; // frac == 3
Quoted 6 lines
So the output with be the truncated, not rounded, 0.000003
The addition will have a "carry bit" in to the upper bits. That isn't lost when shifting right. The value added is as if there was a "1" in the 19th binary place.
Thank you very much.
Reinette
Tony Luck
maintainerPATCH v3 12/26 fs/resctrl: Add hook for architecture code to set monitor event attributes
In reply to Tony LuckArchitecture code knows whether an event can be read from any CPU, or from a CPU on a specific domain. It also knows what format to use when printing each event value.
Add a hook to set mon_event.any_cpu and mon_event.type.
--- include/linux/resctrl.h | 3 +++ fs/resctrl/monitor.c | 12 ++++++++++++ 2 files changed, 15 insertions(+)
include/linux/resctrl.h +3 -0
@@ -418,6 +418,9 @@ void resctrl_offline_mon_domain(struct rdt_resource *r, struct rdt_mon_domain *dvoid resctrl_online_cpu (unsigned int cpu);void resctrl_offline_cpu (unsigned int cpu); +int resctrl_set_event_attributes (enum resctrl_event_id evt, +enum resctrl_event_type type,bool any_cpu); +/** * resctrl_arch_rmid_read() - Read the eventid counter corresponding to rmid * for this resource and domain.
fs/resctrl/monitor.c +12 -0
@@ -863,6 +863,18 @@ static struct mon_evt all_events[QOS_NUM_EVENTS] = { }, }; +int resctrl_set_event_attributes (enum resctrl_event_id evt, +enum resctrl_event_type type,bool any_cpu) +{ +if (evt >=QOS_NUM_EVENTS ) +return -ENOENT ; + + all_events[evt].type = type; + all_events[evt].any_cpu = any_cpu; + +return 0 ; +} +int rdt_lookup_evtid_by_name (char *name) {int evt;
Signature · 1 line
2.48.1
Reinette Chatre
maintainerRe: [PATCH v3 12/26] fs/resctrl: Add hook for architecture code to set monitor event attributes
Hi Tony,
Quoted 5 lines · Tony Luck
Architecture code knows whether an event can be read from any CPU, or from a CPU on a specific domain. It also knows what format to use when printing each event value.
Add a hook to set mon_event.any_cpu and mon_event.type.
If the architecture modifies the output format then the values exposed to user space will look different between architectures. User space will need to know how to parse the data. We do not want user space to need to know which architecture it is running on to determine how to interact with user space so this makes me think that this change needs to be accompanied with a change that exposes the event format to user space.
Quoted 26 lines
Signed-off-by Tony Luck <[email protected]>--- include/linux/resctrl.h | 3 +++ fs/resctrl/monitor.c | 12 ++++++++++++ 2 files changed, 15 insertions(+)include/linux/resctrl.h +3
-0@@ -418,6 +418,9 @@ void resctrl_offline_mon_domain(struct rdt_resource *r, struct rdt_mon_domain *dvoid resctrl_online_cpu (unsigned int cpu);void resctrl_offline_cpu (unsigned int cpu); +int resctrl_set_event_attributes (enum resctrl_event_id evt, +enum resctrl_event_type type,bool any_cpu); +/** * resctrl_arch_rmid_read() - Read the eventid counter corresponding to rmid * for this resource and domain. fs/resctrl/monitor.c +0
-0
fs/resctrl/monitor.c static struct mon_evt all_events[QOS_NUM_EVENTS] = { · line 867
Quoted 5 lines
fs/resctrl/monitor.c +2
-0@@ -863,6 +863,18 @@ static struct mon_evt all_events[QOS_NUM_EVENTS] = { }, }; +int resctrl_set_event_attributes (enum resctrl_event_id evt, +enum resctrl_event_type type,bool any_cpu)
So this is not actually a hook (in the resctrl_arch sense) but a direct interface for arch code to change resctrl fs internals ... this needs to be done with care.
fs/resctrl/monitor.c static struct mon_evt all_events[QOS_NUM_EVENTS] = { · line 880
Quoted 13 lines
fs/resctrl/monitor.c +10
-0+{ +if (evt >=QOS_NUM_EVENTS ) +return -ENOENT ; + + all_events[evt].type = type; + all_events[evt].any_cpu = any_cpu; + +return 0 ; +} +int rdt_lookup_evtid_by_name (char *name) {int evt;
Reinette
Luck, Tony
maintainerRe: [PATCH v3 12/26] fs/resctrl: Add hook for architecture code to set monitor event attributes
Quoted 7 lines · Reinette Chatre
Hi Tony,
Quoted 4 lines · Tony Luck
Architecture code knows whether an event can be read from any CPU, or from a CPU on a specific domain. It also knows what format to use when printing each event value.
Quoted 8 lines
Quoted 1 line
Add a hook to set mon_event.any_cpu and mon_event.type.
If the architecture modifies the output format then the values exposed to user space will look different between architectures. User space will need to know how to parse the data. We do not want user space to need to know which architecture it is running on to determine how to interact with user space so this makes me think that this change needs to be accompanied with a change that exposes the event format to user space.
Would it be enough to include this in Documentation? I.e. add specific entries for "core_energy" to say that it is reported as a floating point value with unit Joules, and "activity" is reported as a floating point value with unit of Farads.
Alternatively the filesystem code could convert the fixed point values to integer values of micro-Joules and micro-Farads. Then the filesystem code can print with "%llu" just like every other event. Would still need the Documentation entries to explain the units. This has the limitation that some future implementation that measures in greater precision would be forced to round to nearest micro-{Joule,Farad}.
File system code controls the formating options available, so options for architecture code to break the user interface are limited.
-Tony
Reinette Chatre
maintainerRe: [PATCH v3 12/26] fs/resctrl: Add hook for architecture code to set monitor event attributes
Hi Tony,
Quoted 17 lines · Tony
Quoted 7 lines · Reinette Chatre
Hi Tony,
Quoted 4 lines · Tony Luck
Architecture code knows whether an event can be read from any CPU, or from a CPU on a specific domain. It also knows what format to use when printing each event value.
Quoted 8 lines
Quoted 1 line
Add a hook to set mon_event.any_cpu and mon_event.type.
If the architecture modifies the output format then the values exposed to user space will look different between architectures. User space will need to know how to parse the data. We do not want user space to need to know which architecture it is running on to determine how to interact with user space so this makes me think that this change needs to be accompanied with a change that exposes the event format to user space.
Quoted 4 lines
Would it be enough to include this in Documentation? I.e. add specific entries for "
core_energy" to say that it is reported as a floating point value with unit Joules, and "activity" is reported as a floating point value with unit of Farads.
If it is always a floating point, yes, but this patch enables an architecture to modify output to be different.
Quoted 7 lines
Alternatively the filesystem code could convert the fixed point values to integer values of micro-Joules and micro-Farads. Then the filesystem code can print with "%llu" just like every other event. Would still need the Documentation entries to explain the units. This has the limitation that some future implementation that measures in greater precision would be forced to round to nearest micro-{Joule,Farad}.
I do not think we are talking about the same issue.
Quoted 3 lines
File system code controls the formating options available, so options for architecture code to break the user interface are limited.
File system gives away the control with this patch, no?
This patch enables the architecture to change the output format. On one architecture the event may thus be presented as floating point while on another architecture the event will not be floating point.
Reinette
Tony Luck
maintainerPATCH v3 13/26 fs/resctrl: Add an architectural hook called for each mount
In reply to Tony LuckEnumeration of Intel telemetry events is not complete when the resctrl "late_init" code is executed.
Add a hook at the beginning of the mount code that will be used to check for telemetry events and initialize if any are found.
The hook is called on every mount. But expectations are that most actions (like enumeration) will only need to be performed on the first call.
--- include/linux/resctrl.h | 3 +++ arch/x86/kernel/cpu/resctrl/core.c | 9 +++++++++ fs/resctrl/rdtgroup.c | 2 ++ 3 files changed, 14 insertions(+)
include/linux/resctrl.h +3 -0
@@ -421,6 +421,9 @@ void resctrl_offline_cpu(unsigned int cpu);int resctrl_set_event_attributes (enum resctrl_event_id evt,enum resctrl_event_type type,bool any_cpu); +/* Architecture hook called for each file system mount */ +void resctrl_arch_mount (void ); +/** * resctrl_arch_rmid_read() - Read the eventid counter corresponding to rmid * for this resource and domain.
arch/x86/kernel/cpu/resctrl/core.c +9 -0
@@ -711,6 +711,15 @@ static int resctrl_arch_offline_cpu(unsigned int cpu)return 0 ; } +void resctrl_arch_mount (void ) +{ +static bool only_once; + +if (only_once) +return ; + only_once = true; +} +enum {RDT_FLAG_CMT ,RDT_FLAG_MBM_TOTAL ,
fs/resctrl/rdtgroup.c +2 -0
@@ -2564,6 +2564,8 @@ static int rdt_get_tree(struct fs_context *fc)struct rdt_resource *r;int ret; +resctrl_arch_mount (); +cpus_read_lock ();mutex_lock (&rdtgroup_mutex);/*
Signature · 1 line
2.48.1
Reinette Chatre
maintainerRe: [PATCH v3 13/26] fs/resctrl: Add an architectural hook called for each mount
Hi Tony,
Quoted 22 lines · Tony Luck
Enumeration of Intel telemetry events is not complete when the resctrl "
late_init" code is executed.Add a hook at the beginning of the mount code that will be used to check for telemetry events and initialize if any are found.
The hook is called on every mount. But expectations are that most actions (like enumeration) will only need to be performed on the first call.
Signed-off-by Tony Luck <[email protected]>--- include/linux/resctrl.h | 3 +++ arch/x86/kernel/cpu/resctrl/core.c | 9 +++++++++ fs/resctrl/rdtgroup.c | 2 ++ 3 files changed, 14 insertions(+)include/linux/resctrl.h +0
-0
include/linux/resctrl.h void resctrl_offline_cpu(unsigned int cpu); · line 424
Quoted 4 lines
include/linux/resctrl.h +1
-0@@ -421,6 +421,9 @@ void resctrl_offline_cpu(unsigned int cpu);int resctrl_set_event_attributes (enum resctrl_event_id evt,enum resctrl_event_type type,bool any_cpu); +/* Architecture hook called for each file system mount */
Please add some description of what architecture could use it for as well as more specific detail of when it is called during mount. I think it is important to highlight and make it part of agreement that resctrl fs calls this on mount before any resctrl fs actions. Considering this, perhaps resctrl_arch_pre_mount()?
It is also worth highlighting in the API doc that fs does not actually call resctrl_arch_mount() on every mount but every mount *attempt* (resctrl may already be mounted) so it is up for arch to maintain any needed state.
(Also see later comment about locking)
Quoted 30 lines
include/linux/resctrl.h +2
-0+void resctrl_arch_mount (void ); +/** * resctrl_arch_rmid_read() - Read the eventid counter corresponding to rmid * for this resource and domain. arch/x86/kernel/cpu/resctrl/core.c +9
-0@@ -711,6 +711,15 @@ static int resctrl_arch_offline_cpu(unsigned int cpu)return 0 ; } +void resctrl_arch_mount (void ) +{ +static bool only_once; + +if (only_once) +return ; + only_once = true; +} +enum {RDT_FLAG_CMT ,RDT_FLAG_MBM_TOTAL ,fs/resctrl/rdtgroup.c +0
-0
fs/resctrl/rdtgroup.c static int rdt_get_tree(struct fs_context *fc) · line 2571
Quoted 8 lines
fs/resctrl/rdtgroup.c +2
-0@@ -2564,6 +2564,8 @@ static int rdt_get_tree(struct fs_context *fc)struct rdt_resource *r;int ret; +resctrl_arch_mount (); +cpus_read_lock ();mutex_lock (&rdtgroup_mutex);/*
Could you please elaborate on the locking requirements here? Worth a mention of these expectations in changelog also. That it is called without any locks held and arch is responsible for all locking should be documented as part of API in include/linux/resctrl.h
Reinette
Tony Luck
maintainerPATCH v3 14/26 x86/resctrl: Add first part of telemetry event enumeration
In reply to Tony LuckThe OOBMSM driver provides an interface to discover any RMID based events for "energy" and "perf" classes.
Hold onto references to any pmt_feature_groups that resctrl uses until resctrl exit.
--- arch/x86/kernel/cpu/resctrl/internal.h | 8 ++++ arch/x86/kernel/cpu/resctrl/core.c | 5 ++ arch/x86/kernel/cpu/resctrl/intel_aet.c | 62 +++++++++++++++++++++++++ arch/x86/kernel/cpu/resctrl/Makefile | 1 + 4 files changed, 76 insertions(+) create mode 100644 arch/x86/kernel/cpu/resctrl/intel_aet.c
arch/x86/kernel/cpu/resctrl/internal.h +8 -0
@@ -172,4 +172,12 @@ void __init intel_rdt_mbm_apply_quirk(void);void rdt_domain_reconfigure_cdp (struct rdt_resource *r); +#ifdef CONFIG_INTEL_AET_RESCTRL +bool intel_aet_get_events (void ); +void __exit intel_aet_exit(void); +#else +static inline bool intel_aet_get_events (void ) {return false; } +static inline void intel_aet_exit (void ) { }; +#endif +#endif /* _ASM_X86_RESCTRL_INTERNAL_H */
arch/x86/kernel/cpu/resctrl/core.c +5 -0
@@ -718,6 +718,9 @@ void resctrl_arch_mount(void)if (only_once)return ; only_once = true; + +if (!intel_aet_get_events ()) +return ; }enum {
@@ -1063,6 +1066,8 @@ late_initcall(resctrl_arch_late_init); static void __exit resctrl_arch_exit(void) { +intel_aet_exit (); +cpuhp_remove_state (rdt_online);resctrl_exit ();
arch/x86/kernel/cpu/resctrl/intel_aet.c new file +62 -0
62 lines not on this page · raw message
arch/x86/kernel/cpu/resctrl/Makefile +1 -0
@@ -1,6 +1,7 @@ # SPDX-License-Identifier: GPL-2.0 obj-$(CONFIG_X86_CPU_RESCTRL) += core.o rdtgroup.o monitor.o obj-$(CONFIG_X86_CPU_RESCTRL) += ctrlmondata.o +obj-$(CONFIG_INTEL_AET_RESCTRL) += intel_aet.o obj-$(CONFIG_INTEL_AET_RESCTRL) += fake_intel_aet_features.o obj-$(CONFIG_RESCTRL_FS_PSEUDO_LOCK) += pseudo_lock.o
Signature · 1 line
2.48.1
Reinette Chatre
maintainerRe: [PATCH v3 14/26] x86/resctrl: Add first part of telemetry event enumeration
Hi Tony,
The following three patches progress as follows:
x86/resctrl: Add first part ...
x86/resctrl: Second stage ...
x86/resctrl: Third phase ...
Could you please make the language consistent?
Quoted 2 lines · Tony Luck
The OOBMSM driver provides an interface to discover any RMID based events for "energy" and "perf" classes.
Please add context about what an RMID based event is.
Quoted 2 lines
Hold onto references to any
pmt_feature_groupsthat resctrl
Please add context about what a "pmt_feature_groups" (intended to be pmt_feature_group?) is.
Quoted 33 lines
uses until resctrl exit.
Signed-off-by Tony Luck <[email protected]>--- arch/x86/kernel/cpu/resctrl/internal.h | 8 ++++ arch/x86/kernel/cpu/resctrl/core.c | 5 ++ arch/x86/kernel/cpu/resctrl/intel_aet.c | 62 +++++++++++++++++++++++++ arch/x86/kernel/cpu/resctrl/Makefile | 1 + 4 files changed, 76 insertions(+) create mode 100644 arch/x86/kernel/cpu/resctrl/intel_aet.carch/x86/kernel/cpu/resctrl/internal.h +8
-0@@ -172,4 +172,12 @@ void __init intel_rdt_mbm_apply_quirk(void);void rdt_domain_reconfigure_cdp (struct rdt_resource *r); +#ifdef CONFIG_INTEL_AET_RESCTRL +bool intel_aet_get_events (void ); +void __exit intel_aet_exit(void); +#else +static inline bool intel_aet_get_events (void ) {return false; } +static inline void intel_aet_exit (void ) { }; +#endif +#endif /* _ASM_X86_RESCTRL_INTERNAL_H */ arch/x86/kernel/cpu/resctrl/core.c +0
-0
arch/x86/kernel/cpu/resctrl/core.c void resctrl_arch_mount(void) · line 722
Quoted 5 lines
arch/x86/kernel/cpu/resctrl/core.c +2
-0@@ -718,6 +718,9 @@ void resctrl_arch_mount(void)if (only_once)return ; only_once = true; + +if (!intel_aet_get_events ())
hmmm ... keep in mind that this is called without any locking and thus there may be risk of parallel calls?
Please document how/if this is relying on some fs features to ensure there are not two instances running at same time.
Quoted 25 lines
arch/x86/kernel/cpu/resctrl/core.c +3
-0+return ; }enum {@@ -1063,6 +1066,8 @@ late_initcall(resctrl_arch_late_init); static void __exit resctrl_arch_exit(void) { +intel_aet_exit (); +cpuhp_remove_state (rdt_online);resctrl_exit ();arch/x86/kernel/cpu/resctrl/intel_aet.c new file +6
-0@@ -0,0 +1,62 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* +* Resource Director Technology(RDT) +* - Intel Application Energy Telemetry +* +* Copyright (C) 2025 Intel Corporation
arch/x86/kernel/cpu/resctrl/intel_aet.c line 16
Quoted 10 lines
arch/x86/kernel/cpu/resctrl/intel_aet.c new file +10
-0+* +* Author: +* Tony Luck <[email protected]> +*/ + +#define pr_fmt (fmt) "resctrl: " fmt + +#include <linux/cpu.h> +#include <linux/cleanup.h> +#include "fake_intel_aet_features.h"
This include can be marked as "Temporary" to highlight that it will not stay.
Please separate headers into blocks and sort alphabetically.
arch/x86/kernel/cpu/resctrl/intel_aet.c line 19
Quoted 3 lines
arch/x86/kernel/cpu/resctrl/intel_aet.c new file +3
-0+#include <linux/intel_vsec.h> +#include <linux/resctrl.h> +#include <linux/slab.h>
Are all these headers used in code below?
Quoted 22 lines
arch/x86/kernel/cpu/resctrl/intel_aet.c new file +22
-0+ +#include "internal.h" + +static struct pmt_feature_group *feat_energy; +static struct pmt_feature_group *feat_perf; + +DEFINE_FREE(intel_pmt_put_feature_group, struct pmt_feature_group *, \ + if (!IS_ERR_OR_NULL(_T)) \ +intel_pmt_put_feature_group (_T)) + +/* +* Ask OOBMSM discovery driver for all the RMID based telemetry groups +* that it supports. + */ +bool intel_aet_get_events (void ) +{ + struct pmt_feature_group *p1 __free(intel_pmt_put_feature_group) = NULL; + struct pmt_feature_group *p2 __free(intel_pmt_put_feature_group) = NULL; +bool use_p1, use_p2; + + p1 =intel_pmt_get_regions_by_feature (FEATURE_PER_RMID_ENERGY_TELEM ); + p2 =intel_pmt_get_regions_by_feature (FEATURE_PER_RMID_PERF_TELEM );
arch/x86/kernel/cpu/resctrl/intel_aet.c line 51
Quoted 10 lines
arch/x86/kernel/cpu/resctrl/intel_aet.c new file +10
-0+ use_p1 = !IS_ERR_OR_NULL (p1); + use_p2 = !IS_ERR_OR_NULL (p2); + +if (!use_p1 && !use_p2) +return false; + +if (use_p1) + feat_energy =no_free_ptr (p1); +if (use_p2) + feat_perf =no_free_ptr (p2);
This reminds me of something I read recently .... "There's a rule in computer programming that objects appear zero, once, or many times. So code accordingly." Not expecting a change .... just finding it amusing.
Quoted 16 lines
arch/x86/kernel/cpu/resctrl/intel_aet.c new file +11
-0+ +return true; +} + +void __exit intel_aet_exit(void) +{ +if (feat_energy) +intel_pmt_put_feature_group (feat_energy); +if (feat_perf) +intel_pmt_put_feature_group (feat_perf); +}arch/x86/kernel/cpu/resctrl/Makefile +0
-0
arch/x86/kernel/cpu/resctrl/Makefile line 7
Quoted 7 lines
arch/x86/kernel/cpu/resctrl/Makefile +1
-0@@ -1,6 +1,7 @@ # SPDX-License-Identifier: GPL-2.0 obj-$(CONFIG_X86_CPU_RESCTRL) += core.o rdtgroup.o monitor.o obj-$(CONFIG_X86_CPU_RESCTRL) += ctrlmondata.o +obj-$(CONFIG_INTEL_AET_RESCTRL) += intel_aet.o obj-$(CONFIG_INTEL_AET_RESCTRL) += fake_intel_aet_features.o obj-$(CONFIG_RESCTRL_FS_PSEUDO_LOCK) += pseudo_lock.o
Reinette
Tony Luck
maintainerPATCH v3 15/26 x86/resctrl: Second stage of telemetry event enumeration
In reply to Tony LuckScan the telemetry_region structures looking for recognised guid values. Count how many are found in each package.
Note that telemetry support depends on at least one of the original RDT monitoring features being enabled (so that the CPU hotplug notifiers for resctrl are running).
--- arch/x86/kernel/cpu/resctrl/intel_aet.c | 112 +++++++++++++++++++++++- 1 file changed, 110 insertions(+), 2 deletions(-)
arch/x86/kernel/cpu/resctrl/intel_aet.c +110 -2
@@ -23,10 +23,100 @@static struct pmt_feature_group *feat_energy;static struct pmt_feature_group *feat_perf; +/* Per-package event groups active on this machine */ +static struct pkg_info { +int count; +struct telemetry_region *regions; +} *pkg_info; + +/** +* struct pmt_event - Telemetry event. +* @evtid: Resctrl event id +* @evt_offset: MMIO offset of counter +* @type: Type for format user display of event value +*/ +struct pmt_event { +enum resctrl_event_id evtid; +int evt_offset; +enum resctrl_event_type type; +}; + +/** +* struct telem_entry - Summarized form from XML telemetry description +* @name: Name for this group of events +* @guid: Unique ID for this group +* @size: Size of MMIO mapped counter registers +* @num_rmids: Number of RMIDS supported +* @overflow_counter_off: Offset of overflow count +* @last_overflow_tstamp_off: Offset of overflow timestamp +* @last_update_tstamp_off: Offset of last update timestamp +* @active: Marks this group as active on this system +* @num_events: Size of @evts array +* @evts: Telemetry events in this group +*/ +struct telem_entry { +char *name; +int guid; +int size; +int num_rmids; +int overflow_counter_off; +int last_overflow_tstamp_off; +int last_update_tstamp_off; +bool active; +int num_events; +struct pmt_event evts[]; +}; + +/* All known telemetry event groups */ +static struct telem_entry *telem_entry[] = { +NULL +}; + +/* +* Scan a feature group looking for guids recognized +* and update the per-package counts of known groups. +*/ +static bool count_events (struct pkg_info *pkg,int max_pkgs,struct pmt_feature_group *p) +{ +struct telem_entry **tentry; +bool found = false; + +if (IS_ERR_OR_NULL (p)) +return false; + +for (int i =0 ; i < p->count; i++) { +struct telemetry_region *tr = &p->regions[i]; + +for (tentry = telem_entry; *tentry; tentry++) { +if (tr->guid == (*tentry)->guid) { +if (tr->plat_info.package_id > max_pkgs) { +pr_warn_once ("Bad package %d\n" , tr->plat_info.package_id); +continue ; + } +if (tr->size > (*tentry)->size) { +pr_warn_once ("MMIO region for guid 0x%x too small\n" , tr->guid); +continue ; + } + found = true; + (*tentry)->active = true; + pkg[tr->plat_info.package_id].count++; +break ; + } + } + } + +return found; +} + DEFINE_FREE(intel_pmt_put_feature_group, struct pmt_feature_group *, \ if (!IS_ERR_OR_NULL(_T)) \ intel_pmt_put_feature_group(_T)) +DEFINE_FREE(free_pkg_info, struct pkg_info *, \ + if (_T) \ + for (int i = 0; i < topology_max_packages(); i++) \ + kfree(_T[i].regions); \ +kfree (_T))/* * Ask OOBMSM discovery driver for all the RMID based telemetry groups * that it supports.
@@ -35,20 +125,32 @@ bool intel_aet_get_events(void) { struct pmt_feature_group *p1 __free(intel_pmt_put_feature_group) = NULL; struct pmt_feature_group *p2 __free(intel_pmt_put_feature_group) = NULL; + struct pkg_info *pkg __free(free_pkg_info) = NULL; +int num_pkgs =topology_max_packages ();bool use_p1, use_p2; + pkg =kcalloc (num_pkgs,sizeof (*pkg_info),GFP_KERNEL ); +if (!pkg) +return false; + p1 =intel_pmt_get_regions_by_feature (FEATURE_PER_RMID_ENERGY_TELEM ); p2 =intel_pmt_get_regions_by_feature (FEATURE_PER_RMID_PERF_TELEM ); - use_p1 = !IS_ERR_OR_NULL (p1); - use_p2 = !IS_ERR_OR_NULL (p2); + use_p1 =count_events (pkg, num_pkgs, p1); + use_p2 =count_events (pkg, num_pkgs, p2);if (!use_p1 && !use_p2)return false; +if (!resctrl_arch_mon_capable ()) { +pr_info ("Telemetry available but monitor support disabled\n" ); +return false; + } +if (use_p1) feat_energy =no_free_ptr (p1);if (use_p2) feat_perf =no_free_ptr (p2); + pkg_info =no_free_ptr (pkg);return true; }
@@ -59,4 +161,10 @@ void __exit intel_aet_exit(void)intel_pmt_put_feature_group (feat_energy);if (feat_perf)intel_pmt_put_feature_group (feat_perf); + +if (pkg_info) { +for (int i =0 ; i <topology_max_packages (); i++) +kfree (pkg_info[i].regions); + } +kfree (pkg_info); }
Signature · 1 line
2.48.1
Reinette Chatre
maintainerRe: [PATCH v3 15/26] x86/resctrl: Second stage of telemetry event enumeration
Hi Tony,
Quoted 1 line · Tony Luck
Scan the
telemetry_regionstructures looking for recognised guid
Please add context before description of what patch does.
Also, please pick British or American English and stick with it.
Quoted 17 lines
values. Count how many are found in each package.
Note that telemetry support depends on at least one of the original RDT monitoring features being enabled (so that the CPU hotplug notifiers for resctrl are running).
Signed-off-by Tony Luck <[email protected]>--- arch/x86/kernel/cpu/resctrl/intel_aet.c | 112 +++++++++++++++++++++++- 1 file changed, 110 insertions(+), 2 deletions(-)arch/x86/kernel/cpu/resctrl/intel_aet.c +0
-0@@ -23,10 +23,100 @@static struct pmt_feature_group *feat_energy;
arch/x86/kernel/cpu/resctrl/intel_aet.c line 33
Quoted 10 lines
arch/x86/kernel/cpu/resctrl/intel_aet.c +8
-0static struct pmt_feature_group *feat_perf; +/* Per-package event groups active on this machine */ +static struct pkg_info { +int count; +struct telemetry_region *regions; +} *pkg_info; + +/** +* struct pmt_event - Telemetry event.
Why does it need "pmt" prefix? Can it be "telem_event" to match telem_entry?
arch/x86/kernel/cpu/resctrl/intel_aet.c line 36
Quoted 3 lines
arch/x86/kernel/cpu/resctrl/intel_aet.c +3
-0+* @evtid: Resctrl event id +* @evt_offset: MMIO offset of counter +* @type: Type for format user display of event value
I cannot make sense of "Type for format user display of event value"
arch/x86/kernel/cpu/resctrl/intel_aet.c line 45
Quoted 9 lines
arch/x86/kernel/cpu/resctrl/intel_aet.c +9
-0+*/ +struct pmt_event { +enum resctrl_event_id evtid; +int evt_offset; +enum resctrl_event_type type; +}; + +/** +* struct telem_entry - Summarized form from XML telemetry description
Copying from v2 review:
"It is not clear to me how useful it is to document that this is "Summarized form from XML telemetry description". Either more detail should be added to help reader understand what XML is being talked about or the description should be a summary of what this data structure represents."
arch/x86/kernel/cpu/resctrl/intel_aet.c line 50
Quoted 5 lines
arch/x86/kernel/cpu/resctrl/intel_aet.c +5
-0+* @name: Name for this group of events +* @guid: Unique ID for this group +* @size: Size of MMIO mapped counter registers +* @num_rmids: Number of RMIDS supported +* @overflow_counter_off: Offset of overflow count
Description just rewrites member name and changes "counter" to "count". Could description have more details about what is represented by this? What overflowed?
arch/x86/kernel/cpu/resctrl/intel_aet.c line 51
Quoted 1 line
arch/x86/kernel/cpu/resctrl/intel_aet.c +1
-0+* @last_overflow_tstamp_off: Offset of overflow timestamp
What overflowed at this timestamp?
arch/x86/kernel/cpu/resctrl/intel_aet.c line 52
Quoted 1 line
arch/x86/kernel/cpu/resctrl/intel_aet.c +1
-0+* @last_update_tstamp_off: Offset of last update timestamp
What was updated at this timestamp?
arch/x86/kernel/cpu/resctrl/intel_aet.c line 53
Quoted 1 line
arch/x86/kernel/cpu/resctrl/intel_aet.c +1
-0+* @active: Marks this group as active on this system
What does it mean when a group is "active"?
arch/x86/kernel/cpu/resctrl/intel_aet.c line 54
Quoted 1 line
arch/x86/kernel/cpu/resctrl/intel_aet.c +1
-0+* @num_events: Size of @evts array
Would __counted_by() be useful?
arch/x86/kernel/cpu/resctrl/intel_aet.c line 64
Quoted 10 lines
arch/x86/kernel/cpu/resctrl/intel_aet.c +10
-0+* @evts: Telemetry events in this group +*/ +struct telem_entry { +char *name; +int guid; +int size; +int num_rmids; +int overflow_counter_off; +int last_overflow_tstamp_off; +int last_update_tstamp_off;
Most of types are "int" ... I do not expect many of these types to be negative so I would like to check if int is most appropriate for all? Usually size_t is used for size and off_t/loff_t is available for offsets.
arch/x86/kernel/cpu/resctrl/intel_aet.c line 67
Quoted 3 lines
arch/x86/kernel/cpu/resctrl/intel_aet.c +3
-0+bool active; +int num_events; +struct pmt_event evts[];
(missing tab)
arch/x86/kernel/cpu/resctrl/intel_aet.c line 70
Quoted 3 lines
arch/x86/kernel/cpu/resctrl/intel_aet.c +3
-0+}; + +/* All known telemetry event groups */
This is more useful by not being "All known Summarized form from XML telemetry description".
arch/x86/kernel/cpu/resctrl/intel_aet.c line 76
Quoted 6 lines
arch/x86/kernel/cpu/resctrl/intel_aet.c +6
-0+static struct telem_entry *telem_entry[] = { +NULL +}; + +/* +* Scan a feature group looking for guids recognized
Switch from British to American English in same patch.
Quoted 6 lines
arch/x86/kernel/cpu/resctrl/intel_aet.c +6
-0+* and update the per-package counts of known groups. +*/ +static bool count_events (struct pkg_info *pkg,int max_pkgs,struct pmt_feature_group *p) +{ +struct telem_entry **tentry; +bool found = false;
arch/x86/kernel/cpu/resctrl/intel_aet.c line 92
Quoted 10 lines
arch/x86/kernel/cpu/resctrl/intel_aet.c +10
-0+ +if (IS_ERR_OR_NULL (p)) +return false; + +for (int i =0 ; i < p->count; i++) { +struct telemetry_region *tr = &p->regions[i]; + +for (tentry = telem_entry; *tentry; tentry++) { +if (tr->guid == (*tentry)->guid) { +if (tr->plat_info.package_id > max_pkgs) {
Should this be >=?
Quoted 31 lines
arch/x86/kernel/cpu/resctrl/intel_aet.c +23
-0+pr_warn_once ("Bad package %d\n" , tr->plat_info.package_id); +continue ; + } +if (tr->size > (*tentry)->size) { +pr_warn_once ("MMIO region for guid 0x%x too small\n" , tr->guid); +continue ; + } + found = true; + (*tentry)->active = true; + pkg[tr->plat_info.package_id].count++; +break ; + } + } + } + +return found; +} + DEFINE_FREE(intel_pmt_put_feature_group, struct pmt_feature_group *, \ if (!IS_ERR_OR_NULL(_T)) \ intel_pmt_put_feature_group(_T)) +DEFINE_FREE(free_pkg_info, struct pkg_info *, \ + if (_T) \ + for (int i = 0; i < topology_max_packages(); i++) \ + kfree(_T[i].regions); \ +kfree (_T))/* * Ask OOBMSM discovery driver for all the RMID based telemetry groups * that it supports.
arch/x86/kernel/cpu/resctrl/intel_aet.c bool intel_aet_get_events(void) · line 132
Quoted 8 lines
arch/x86/kernel/cpu/resctrl/intel_aet.c +3
-0@@ -35,20 +125,32 @@ bool intel_aet_get_events(void) { struct pmt_feature_group *p1 __free(intel_pmt_put_feature_group) = NULL; struct pmt_feature_group *p2 __free(intel_pmt_put_feature_group) = NULL; + struct pkg_info *pkg __free(free_pkg_info) = NULL; +int num_pkgs =topology_max_packages ();bool use_p1, use_p2; + pkg =kcalloc (num_pkgs,sizeof (*pkg_info),GFP_KERNEL );
sizeof(*pkg)?
Reinette
Tony Luck
maintainerPATCH v3 16/26 x86/resctrl: Third phase of telemetry event enumeration
In reply to Tony LuckAllocate per-package arrays for the known telemetry_regions and initialize with pointers to the structures acquired from the intel_pmt_get_regions_by_feature() calls.
--- arch/x86/kernel/cpu/resctrl/intel_aet.c | 38 +++++++++++++++++++++++++ 1 file changed, 38 insertions(+)
arch/x86/kernel/cpu/resctrl/intel_aet.c +38 -0
@@ -108,6 +108,30 @@ static bool count_events(struct pkg_info *pkg, int max_pkgs, struct pmt_feature_return found; } +/* +* Copy the pointers to telemetry regions associated with a given package +* and with known guids over to the pkg_info structure for that package. +*/ +static int setup (struct pkg_info *pkg,int pkgnum,struct pmt_feature_group *p,int slot) +{ +struct telem_entry **tentry; + +for (int i =0 ; i < p->count; i++) { +for (tentry = telem_entry; *tentry; tentry++) { +if (!(*tentry)->active) +continue ; +if (pkgnum != p->regions[i].plat_info.package_id) +continue ; +if (p->regions[i].guid != (*tentry)->guid) +continue ; + + pkg[pkgnum].regions[slot++] = p->regions[i]; + } + } + +return slot; +} + DEFINE_FREE(intel_pmt_put_feature_group, struct pmt_feature_group *, \ if (!IS_ERR_OR_NULL(_T)) \intel_pmt_put_feature_group (_T))
@@ -128,6 +152,7 @@ bool intel_aet_get_events(void) struct pkg_info *pkg __free(free_pkg_info) = NULL;int num_pkgs =topology_max_packages ();bool use_p1, use_p2; +int slot; pkg =kcalloc (num_pkgs,sizeof (*pkg_info),GFP_KERNEL );if (!pkg)
@@ -146,6 +171,19 @@ bool intel_aet_get_events(void)return false; } +for (int i =0 ; i < num_pkgs; i++) { +if (!pkg[i].count) +continue ; + pkg[i].regions =kmalloc_array (pkg[i].count,sizeof (*pkg[i].regions),GFP_KERNEL ); +if (!pkg[i].regions) +return false; + slot =0 ; +if (use_p1) + slot =setup (pkg, i, p1, slot); +if (use_p2) + slot =setup (pkg, i, p2, slot); + } +if (use_p1) feat_energy =no_free_ptr (p1);if (use_p2)
Signature · 1 line
2.48.1
Reinette Chatre
maintainerRe: [PATCH v3 16/26] x86/resctrl: Third phase of telemetry event enumeration
Hi Tony,
Quoted 3 lines · Tony Luck
Allocate per-package arrays for the known
telemetry_regionsand initialize with pointers to the structures acquired from theintel_pmt_get_regions_by_feature()calls.
Why?
Quoted 11 lines
Signed-off-by Tony Luck <[email protected]>--- arch/x86/kernel/cpu/resctrl/intel_aet.c | 38 +++++++++++++++++++++++++ 1 file changed, 38 insertions(+)arch/x86/kernel/cpu/resctrl/intel_aet.c +0
-0
arch/x86/kernel/cpu/resctrl/intel_aet.c static bool count_events(struct pkg_info *pkg, int max_pkgs, struct pmt_feature_ · line 115
Quoted 8 lines
arch/x86/kernel/cpu/resctrl/intel_aet.c +5
-0@@ -108,6 +108,30 @@ static bool count_events(struct pkg_info *pkg, int max_pkgs, struct pmt_feature_return found; } +/* +* Copy the pointers to telemetry regions associated with a given package +* and with known guids over to the pkg_info structure for that package. +*/ +static int setup (struct pkg_info *pkg,int pkgnum,struct pmt_feature_group *p,int slot)
setup() is very generic. I do not have a suggestion since it is not yet clear what this does (more below).
arch/x86/kernel/cpu/resctrl/intel_aet.c static bool count_events(struct pkg_info *pkg, int max_pkgs, struct pmt_feature_ · line 128
Quoted 13 lines
arch/x86/kernel/cpu/resctrl/intel_aet.c +13
-0+{ +struct telem_entry **tentry; + +for (int i =0 ; i < p->count; i++) { +for (tentry = telem_entry; *tentry; tentry++) { +if (!(*tentry)->active) +continue ; +if (pkgnum != p->regions[i].plat_info.package_id) +continue ; +if (p->regions[i].guid != (*tentry)->guid) +continue ; + + pkg[pkgnum].regions[slot++] = p->regions[i];
(please fix spacing)
hmmm .. this looks like one structure copied to another, not the pointer copy the comment mentions.
Quoted 18 lines
arch/x86/kernel/cpu/resctrl/intel_aet.c +7
-0+ } + } + +return slot; +} + DEFINE_FREE(intel_pmt_put_feature_group, struct pmt_feature_group *, \ if (!IS_ERR_OR_NULL(_T)) \intel_pmt_put_feature_group (_T))@@ -128,6 +152,7 @@ bool intel_aet_get_events(void) struct pkg_info *pkg __free(free_pkg_info) = NULL;int num_pkgs =topology_max_packages ();bool use_p1, use_p2; +int slot; pkg =kcalloc (num_pkgs,sizeof (*pkg_info),GFP_KERNEL );if (!pkg)
arch/x86/kernel/cpu/resctrl/intel_aet.c bool intel_aet_get_events(void) · line 177
Quoted 7 lines
arch/x86/kernel/cpu/resctrl/intel_aet.c +4
-0@@ -146,6 +171,19 @@ bool intel_aet_get_events(void)return false; } +for (int i =0 ; i < num_pkgs; i++) { +if (!pkg[i].count) +continue ; + pkg[i].regions =kmalloc_array (pkg[i].count,sizeof (*pkg[i].regions),GFP_KERNEL );
As I understand it sizeof(*pkg[i].regions) would be the size of a struct telemetry_region that the code in setup() initializes by copying the data from the struct pmt_feature_group.
The changelog and comments creates impression that resctrl's initialization consists of adding pointers to the data in struct pmt_feature_group but as I read it the data is copied instead. Am I reading it wrong?
arch/x86/kernel/cpu/resctrl/intel_aet.c bool intel_aet_get_events(void) · line 189
Quoted 12 lines
arch/x86/kernel/cpu/resctrl/intel_aet.c +9
-0+if (!pkg[i].regions) +return false; + slot =0 ; +if (use_p1) + slot =setup (pkg, i, p1, slot); +if (use_p2) + slot =setup (pkg, i, p2, slot); + } +if (use_p1) feat_energy =no_free_ptr (p1);if (use_p2)
Reinette
Tony Luck
maintainerPATCH v3 17/26 x86/resctrl: Build a lookup table for each resctrl event id
In reply to Tony LuckUser requests to read events arrive from the file system layer with a domain id, RMID, and resctrl event id. Responding to those requests needs information from various structures.
Build a quick lookup table indexed by resctrl event id.
--- arch/x86/kernel/cpu/resctrl/intel_aet.c | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+)
arch/x86/kernel/cpu/resctrl/intel_aet.c +18 -0
@@ -67,6 +67,12 @@ struct telem_entry {struct pmt_event evts[]; }; +/* Lookup table to get from resctrl event id to useful structures */ +static struct evtinfo { +struct telem_entry *telem_entry; +struct pmt_event *pmt_event; +} evtinfo[QOS_NUM_EVENTS ]; +/* All known telemetry event groups */ static struct telem_entry *telem_entry[] = {NULL
@@ -151,6 +157,7 @@ bool intel_aet_get_events(void) struct pmt_feature_group *p2 __free(intel_pmt_put_feature_group) = NULL; struct pkg_info *pkg __free(free_pkg_info) = NULL;int num_pkgs =topology_max_packages (); +struct telem_entry **tentry;bool use_p1, use_p2;int slot;
@@ -184,6 +191,17 @@ bool intel_aet_get_events(void) slot =setup (pkg, i, p2, slot); } +for (tentry = telem_entry; *tentry; tentry++) { +if (!(*tentry)->active) +continue ; +for (int i =0 ; i < (*tentry)->num_events; i++) { +enum resctrl_event_id evtid = (*tentry)->evts[i].evtid; + + evtinfo[evtid].telem_entry = *tentry; + evtinfo[evtid].pmt_event = &(*tentry)->evts[i]; + } + } +if (use_p1) feat_energy =no_free_ptr (p1);if (use_p2)
Signature · 1 line
2.48.1
Reinette Chatre
maintainerRe: [PATCH v3 17/26] x86/resctrl: Build a lookup table for each resctrl event id
Hi Tony,
Quoted 5 lines · Tony Luck
User requests to read events arrive from the file system layer with a domain id, RMID, and resctrl event id. Responding to those requests needs information from various structures.
Build a quick lookup table indexed by resctrl event id.
Why is a second lookup table needed? What about the lookup table ("all_events") that patch #2 introduced?
Perhaps struct mon_evt could get a "void *priv" or related member that points to event's private data that varies by type of event?
Reinette
Tony Luck
maintainerPATCH v3 18/26 x86/resctrl: Add code to read core telemetry events
In reply to Tony LuckThe new telemetry events will be part of a new resctrl resource. Add the RDT_RESOURCE_PERF_PKG to enum resctrl_res_level.
Add hook resctrl_arch_rmid_read() to pass reads on this resource to the telemetry code.
There may be multiple devices tracking each package, so scan all of them and add up counters.
--- include/linux/resctrl_types.h | 1 + arch/x86/kernel/cpu/resctrl/internal.h | 5 +++ arch/x86/kernel/cpu/resctrl/intel_aet.c | 58 +++++++++++++++++++++++++ arch/x86/kernel/cpu/resctrl/monitor.c | 6 +++ 4 files changed, 70 insertions(+)
include/linux/resctrl_types.h +1 -0
@@ -39,6 +39,7 @@ enum resctrl_res_level {RDT_RESOURCE_L2 ,RDT_RESOURCE_MBA ,RDT_RESOURCE_SMBA , +RDT_RESOURCE_PERF_PKG ,/* Must be the last */ RDT_NUM_RESOURCES,
arch/x86/kernel/cpu/resctrl/internal.h +5 -0
@@ -175,9 +175,14 @@ void rdt_domain_reconfigure_cdp(struct rdt_resource *r);#ifdef CONFIG_INTEL_AET_RESCTRL bool intel_aet_get_events (void ); void __exit intel_aet_exit(void); +int intel_aet_read_event (int domid,int rmid,int evtid,u64 *val);#else static inline bool intel_aet_get_events (void ) {return false; }static inline void intel_aet_exit (void ) { }; +static inline int intel_aet_read_event (int domid,int rmid,int evtid,u64 *val) +{ +return -EINVAL ; +}#endif #endif /* _ASM_X86_RESCTRL_INTERNAL_H */
arch/x86/kernel/cpu/resctrl/intel_aet.c +58 -0
67 lines not on this page · raw message
arch/x86/kernel/cpu/resctrl/monitor.c +6 -0
@@ -236,6 +236,12 @@ int resctrl_arch_rmid_read(struct rdt_resource *r, struct rdt_mon_domain *d,u32 prmid;int ret; +if (r->rid ==RDT_RESOURCE_PERF_PKG ) { + ret =intel_aet_read_event (d->hdr.id, rmid, eventid, val); + +return ret ? ret :0 ; + } +resctrl_arch_rmid_read_context_check (); prmid =logical_rmid_to_physical_rmid (cpu, rmid);
Signature · 1 line
2.48.1
Reinette Chatre
maintainerRe: [PATCH v3 18/26] x86/resctrl: Add code to read core telemetry events
Hi Tony,
(deja vu ... "Add code to" can be dropped)
Quoted 2 lines · Tony Luck
The new telemetry events will be part of a new resctrl resource. Add the
RDT_RESOURCE_PERF_PKGto enumresctrl_res_level.
Please follow tip changelog structure custom throughout this series.
Quoted 32 lines
Add hook
resctrl_arch_rmid_read()to pass reads on this resource to the telemetry code.There may be multiple devices tracking each package, so scan all of them and add up counters.
Signed-off-by Tony Luck <[email protected]>--- include/linux/resctrl_types.h | 1 + arch/x86/kernel/cpu/resctrl/internal.h | 5 +++ arch/x86/kernel/cpu/resctrl/intel_aet.c | 58 +++++++++++++++++++++++++ arch/x86/kernel/cpu/resctrl/monitor.c | 6 +++ 4 files changed, 70 insertions(+)include/linux/resctrl_types.h +1
-0@@ -39,6 +39,7 @@ enum resctrl_res_level {RDT_RESOURCE_L2 ,RDT_RESOURCE_MBA ,RDT_RESOURCE_SMBA , +RDT_RESOURCE_PERF_PKG ,/* Must be the last */ RDT_NUM_RESOURCES,arch/x86/kernel/cpu/resctrl/internal.h +0
-0
arch/x86/kernel/cpu/resctrl/internal.h void rdt_domain_reconfigure_cdp(struct rdt_resource *r); · line 178
Quoted 4 lines
arch/x86/kernel/cpu/resctrl/internal.h +1
-0@@ -175,9 +175,14 @@ void rdt_domain_reconfigure_cdp(struct rdt_resource *r);#ifdef CONFIG_INTEL_AET_RESCTRL bool intel_aet_get_events (void ); void __exit intel_aet_exit(void); +int intel_aet_read_event (int domid,int rmid,int evtid,u64 *val);
This can use enum resctrl_event_id for evtid?
Quoted 15 lines
arch/x86/kernel/cpu/resctrl/internal.h +4
-0#else static inline bool intel_aet_get_events (void ) {return false; }static inline void intel_aet_exit (void ) { }; +static inline int intel_aet_read_event (int domid,int rmid,int evtid,u64 *val) +{ +return -EINVAL ; +}#endif #endif /* _ASM_X86_RESCTRL_INTERNAL_H */ arch/x86/kernel/cpu/resctrl/intel_aet.c +0
-0
arch/x86/kernel/cpu/resctrl/intel_aet.c static struct evtinfo { · line 80
Quoted 8 lines
arch/x86/kernel/cpu/resctrl/intel_aet.c +5
-0@@ -73,6 +73,12 @@ static struct evtinfo {struct pmt_event *pmt_event; } evtinfo[QOS_NUM_EVENTS ]; +#define EVT_NUM_RMIDS (evtid) (evtinfo[evtid].telem_entry->num_rmids) +#define EVT_NUM_EVENTS (evtid) (evtinfo[evtid].telem_entry->num_events) +#define EVT_GUID (evtid) (evtinfo[evtid].telem_entry->guid) + +#define EVT_OFFSET (evtid) (evtinfo[evtid].pmt_event->evt_offset)
Please open code these or use functions if you need to.
Quoted 14 lines
arch/x86/kernel/cpu/resctrl/intel_aet.c +7
-0+/* All known telemetry event groups */ static struct telem_entry *telem_entry[] = {NULL @@ -224,3 +230,55 @@ void __exit intel_aet_exit(void) }kfree (pkg_info); } + +#define VALID_BIT BIT_ULL(63) +#define DATA_BITS GENMASK_ULL(62, 0) + +/* +* Walk the array of telemetry groups on a specific package.
arch/x86/kernel/cpu/resctrl/intel_aet.c void __exit intel_aet_exit(void) · line 248
Quoted 10 lines
arch/x86/kernel/cpu/resctrl/intel_aet.c +10
-0+* Read and sum values for a specific counter (described by +* guid and offset). +* Return failure (~0x0ull) if any counter isn't valid. +*/ +static u64 scan_pmt_devs (int package,int guid,int offset) +{ +u64 rval, val; +int ndev =0 ; + + rval =0 ;
This can be done as part of definition.
Quoted 41 lines
arch/x86/kernel/cpu/resctrl/intel_aet.c +36
-0+ +for (int i =0 ; i < pkg_info[package].count; i++) { +if (pkg_info[package].regions[i].guid != guid) +continue ; + ndev++; + val =readq (pkg_info[package].regions[i].addr + offset); + +if (!(val &VALID_BIT )) +return ~0ull ; + rval += val &DATA_BITS ; + } + +return ndev ? rval : ~0ull ; +} + +/* +* Read counter for an event on a domain (summing all aggregators +* on the domain). +*/ +int intel_aet_read_event (int domid,int rmid,int evtid,u64 *val) +{ +u64 evtcount; +int offset; + +if (rmid >=EVT_NUM_RMIDS (evtid)) +return -ENOENT ; + + offset = rmid *EVT_NUM_EVENTS (evtid) *sizeof (u64); + offset +=EVT_OFFSET (evtid); + evtcount =scan_pmt_devs (domid,EVT_GUID (evtid), offset); + +if (evtcount != ~0ull || *val ==0 ) + *val += evtcount; + +return evtcount != ~0ull ?0 : -EINVAL ; +}arch/x86/kernel/cpu/resctrl/monitor.c +0
-0
arch/x86/kernel/cpu/resctrl/monitor.c int resctrl_arch_rmid_read(struct rdt_resource *r, struct rdt_mon_domain *d, · line 243
Quoted 8 lines
arch/x86/kernel/cpu/resctrl/monitor.c +5
-0@@ -236,6 +236,12 @@ int resctrl_arch_rmid_read(struct rdt_resource *r, struct rdt_mon_domain *d,u32 prmid;int ret; +if (r->rid ==RDT_RESOURCE_PERF_PKG ) { + ret =intel_aet_read_event (d->hdr.id, rmid, eventid, val); + +return ret ? ret :0 ; + }
Not sure if I am missing something at this stage but it looks like, since resctrl_arch_rmid_read() can now return ENOENT, and rmid_read::err obtain value of ENOENT, that there may be an
issue when this error is returned since rdtgroup_mondata_show()'s "checkresult" does not have handling for ENOENT and will attempt to print data to user space.
arch/x86/kernel/cpu/resctrl/monitor.c int resctrl_arch_rmid_read(struct rdt_resource *r, struct rdt_mon_domain *d, · line 245
Quoted 2 lines
arch/x86/kernel/cpu/resctrl/monitor.c +1
-0+resctrl_arch_rmid_read_context_check ();
Please keep this context check at top of function.
arch/x86/kernel/cpu/resctrl/monitor.c int resctrl_arch_rmid_read(struct rdt_resource *r, struct rdt_mon_domain *d, · line 247
Quoted 2 lines
arch/x86/kernel/cpu/resctrl/monitor.c +0
-0prmid =logical_rmid_to_physical_rmid (cpu, rmid);
Reinette
Tony Luck
maintainerPATCH v3 19/26 x86/resctrl: Sanity check telemetry RMID values
In reply to Tony LuckThere are three values of interest:
1) The number of RMIDs supported by the CPU core. This is enumerated by CPUID leaf 0xF. Linux saves the value in boot_cpu_data.x86_cache_max_rmid.
2) The number of counter registers in each telemetry region. This is described in the XML file for the region. Linux hard codes it into the struct telem_entry..num_rmids field.
3) The maximum number of RMIDs that can be tracked simultaneously for a telemetry region. This is provided in the structures received from the intel_pmt_get_regions_by_feature() calls.
Print appropriate warnings if these values do not match.
TODO: Need a better UI. The number of implemented counters can be different per telemetry region.
--- arch/x86/kernel/cpu/resctrl/intel_aet.c | 31 +++++++++++++++++++++++++ 1 file changed, 31 insertions(+)
arch/x86/kernel/cpu/resctrl/intel_aet.c +31 -0
@@ -13,6 +13,7 @@#include <linux/cpu.h> #include <linux/cleanup.h> +#include <linux/minmax.h> #include "fake_intel_aet_features.h" #include <linux/intel_vsec.h> #include <linux/resctrl.h>
@@ -51,6 +52,7 @@ struct pmt_event {* @last_overflow_tstamp_off: Offset of overflow timestamp * @last_update_tstamp_off: Offset of last update timestamp * @active: Marks this group as active on this system +* @rmid_warned: Set to stop multiple rmid sanity warnings * @num_events: Size of @evts array * @evts: Telemetry events in this group */
@@ -63,6 +65,7 @@ struct telem_entry {int last_overflow_tstamp_off;int last_update_tstamp_off;bool active; +bool rmid_warned;int num_events;struct pmt_event evts[]; };
@@ -84,6 +87,33 @@ static struct telem_entry *telem_entry[] = { NULL }; +static void rmid_sanity_check (struct telemetry_region *tr,struct telem_entry *tentry) +{ +struct rdt_resource *r = &rdt_resources_all[RDT_RESOURCE_PERF_PKG ].r_resctrl; +int system_rmids = boot_cpu_data.x86_cache_max_rmid +1 ; + +if (tentry->rmid_warned) +return ; + +if (tentry->num_rmids != system_rmids) { +pr_info ("Telemetry region %s has %d RMIDs system supports %d\n" , + tentry->name, tentry->num_rmids, system_rmids); + tentry->rmid_warned = true; + } + +if (tr->num_rmids < tentry->num_rmids) { +pr_info ("Telemetry region %s only supports %d simultaneous RMIDS\n" , + tentry->name, tr->num_rmids); + tentry->rmid_warned = true; + } + +/* info/PKG_PERF_MON/num_rmids reports number of guaranteed counters */ +if (!r->num_rmid) + r->num_rmid = tr->num_rmids; +else + r->num_rmid =min ((u32 )r->num_rmid, tr->num_rmids); +} +/* * Scan a feature group looking for guids recognized * and update the per-package counts of known groups.
@@ -109,6 +139,7 @@ static bool count_events(struct pkg_info *pkg, int max_pkgs, struct pmt_feature_pr_warn_once ("MMIO region for guid 0x%x too small\n" , tr->guid);continue ; } +rmid_sanity_check (tr, *tentry); found = true; (*tentry)->active = true; pkg[tr->plat_info.package_id].count++;
Signature · 1 line
2.48.1
Reinette Chatre
maintainerRe: [PATCH v3 19/26] x86/resctrl: Sanity check telemetry RMID values
Hi Tony,
Quoted 6 lines · Tony Luck
There are three values of interest:
1) The number of RMIDs supported by the CPU core. This is enumerated by CPUID leaf0xF. Linux saves the value in boot_cpu_data.x86_cache_max_rmid.
2) The number of counter registers in each telemetry region. This is described in the XML file for the region. Linux hard codes it into the struct telem_entry..num_rmids field.
Syntax telem_entry::num_rmids can be used for a member.
Quoted 3 lines
3) The maximum number of RMIDs that can be tracked simultaneously for a telemetry region. This is provided in the structures received from the
intel_pmt_get_regions_by_feature()calls.
Is (2) and (3) not required to be the same? If not, how does resctrl know which counter/RMID is being tracked?
Quoted 2 lines
Print appropriate warnings if these values do not match.
As mentioned in cover letter I do not think that just printing a warning is sufficient. It really becomes a trial-and-error guessing game for user space to know which monitor group supports telemetry events.
Quoted 14 lines
TODO: Need a better UI. The number of implemented counters can be different per telemetry region.
Signed-off-by Tony Luck <[email protected]>--- arch/x86/kernel/cpu/resctrl/intel_aet.c | 31 +++++++++++++++++++++++++ 1 file changed, 31 insertions(+)arch/x86/kernel/cpu/resctrl/intel_aet.c +0
-0
arch/x86/kernel/cpu/resctrl/intel_aet.c line 16
Quoted 4 lines
arch/x86/kernel/cpu/resctrl/intel_aet.c +1
-0@@ -13,6 +13,7 @@#include <linux/cpu.h> #include <linux/cleanup.h> +#include <linux/minmax.h>
Please sort includes alphabetically.
arch/x86/kernel/cpu/resctrl/intel_aet.c struct pmt_event { · line 55
Quoted 8 lines
arch/x86/kernel/cpu/resctrl/intel_aet.c +1
-0#include "fake_intel_aet_features.h" #include <linux/intel_vsec.h> #include <linux/resctrl.h> @@ -51,6 +52,7 @@ struct pmt_event {* @last_overflow_tstamp_off: Offset of overflow timestamp * @last_update_tstamp_off: Offset of last update timestamp * @active: Marks this group as active on this system +* @rmid_warned: Set to stop multiple rmid sanity warnings
rmid -> RMID.
I find the description unclear on how to interact with this member. How about something like:
True if user space have been warned about number of RMIDs used by different resources not matching.
Quoted 12 lines
arch/x86/kernel/cpu/resctrl/intel_aet.c +1
-0* @num_events: Size of @evts array * @evts: Telemetry events in this group */ @@ -63,6 +65,7 @@ struct telem_entry {int last_overflow_tstamp_off;int last_update_tstamp_off;bool active; +bool rmid_warned;int num_events;struct pmt_event evts[]; };
arch/x86/kernel/cpu/resctrl/intel_aet.c static struct telem_entry *telem_entry[] = { · line 93
Quoted 7 lines
arch/x86/kernel/cpu/resctrl/intel_aet.c +4
-0@@ -84,6 +87,33 @@ static struct telem_entry *telem_entry[] = { NULL }; +static void rmid_sanity_check (struct telemetry_region *tr,struct telem_entry *tentry) +{ +struct rdt_resource *r = &rdt_resources_all[RDT_RESOURCE_PERF_PKG ].r_resctrl; +int system_rmids = boot_cpu_data.x86_cache_max_rmid +1 ;
It is not clear what "system_rmids" should represent here. Is it, as changelog states, maximum supported by CPU core, or is it maximum supported by L3 resource, which is the maximum number of monitor groups that can be created.
We see in rdt_get_mon_l3_config() that:
r->num_rmid = (boot_cpu_data.x86_cache_max_rmid + 1) / snc_nodes_per_l3_cache;
This makes me wonder how this feature behaves on SNC systems?
arch/x86/kernel/cpu/resctrl/intel_aet.c static struct telem_entry *telem_entry[] = { · line 99
Quoted 6 lines
arch/x86/kernel/cpu/resctrl/intel_aet.c +6
-0+ +if (tentry->rmid_warned) +return ; + +if (tentry->num_rmids != system_rmids) { + pr_info("Telemetry region %s has %d RMIDs system supports %d\n",
Is pr_info() intended to be pr_warn()?
The message self could do with a comma?
arch/x86/kernel/cpu/resctrl/intel_aet.c static struct telem_entry *telem_entry[] = { · line 102
Quoted 3 lines
arch/x86/kernel/cpu/resctrl/intel_aet.c +3
-0+ tentry->name, tentry->num_rmids, system_rmids); + tentry->rmid_warned = true; + }
Could you please add comments about consequences of when this is encountered?
arch/x86/kernel/cpu/resctrl/intel_aet.c static struct telem_entry *telem_entry[] = { · line 108
Quoted 6 lines
arch/x86/kernel/cpu/resctrl/intel_aet.c +6
-0+ +if (tr->num_rmids < tentry->num_rmids) { +pr_info ("Telemetry region %s only supports %d simultaneous RMIDS\n" , + tentry->name, tr->num_rmids); + tentry->rmid_warned = true; + }
I am still trying to get used to all the data structures. From what I can tell, the offset of counter is obtained from struct telem_entry. If struct telem_entry thus thinks there are more RMIDs than what the region supports, would this not cause memory reads to exceed what region supports?
Could you please add comments about consequences of when this is encountered?
arch/x86/kernel/cpu/resctrl/intel_aet.c static struct telem_entry *telem_entry[] = { · line 115
Quoted 7 lines
arch/x86/kernel/cpu/resctrl/intel_aet.c +7
-0+ +/* info/PKG_PERF_MON/num_rmids reports number of guaranteed counters */ +if (!r->num_rmid) + r->num_rmid = tr->num_rmids; +else + r->num_rmid =min ((u32 )r->num_rmid, tr->num_rmids); +}
As I mentioned in response to previous version it may be possible to move resctrl_mon_resource_init() to rdt_get_tree() to be done after these RMID counts are discovered. When doing so it is possible to size the available RMIDs used on system to be supported by all resources.
arch/x86/kernel/cpu/resctrl/intel_aet.c static bool count_events(struct pkg_info *pkg, int max_pkgs, struct pmt_feature_ · line 145
Quoted 12 lines
arch/x86/kernel/cpu/resctrl/intel_aet.c +2
-0+/* * Scan a feature group looking for guids recognized * and update the per-package counts of known groups. @@ -109,6 +139,7 @@ static bool count_events(struct pkg_info *pkg, int max_pkgs, struct pmt_feature_pr_warn_once ("MMIO region for guid 0x%x too small\n" , tr->guid);continue ; } +rmid_sanity_check (tr, *tentry); found = true; (*tentry)->active = true; pkg[tr->plat_info.package_id].count++;
Reinette
Tony Luck
maintainerPATCH v3 20/26 x86/resctrl: Add and initialize rdt_resource for package scope core monitor
In reply to Tony LuckCounts for each Intel telemetry event are periodically sent to one or more aggregators on each package where accumulated totals are made available in MMIO registers.
Add a new resource for monitoring these events with code to build domains at the package granularity.
--- include/linux/resctrl.h | 1 + arch/x86/kernel/cpu/resctrl/core.c | 11 +++++++++++ 2 files changed, 12 insertions(+)
include/linux/resctrl.h +1 -0
@@ -232,6 +232,7 @@ enum resctrl_scope {RESCTRL_L2_CACHE =2 ,RESCTRL_L3_CACHE =3 ,RESCTRL_L3_NODE , + RESCTRL_PACKAGE, };/**
arch/x86/kernel/cpu/resctrl/core.c +11 -0
@@ -99,6 +99,15 @@ struct rdt_hw_resource rdt_resources_all[RDT_NUM_RESOURCES] = { .schema_fmt = RESCTRL_SCHEMA_RANGE, }, }, + [RDT_RESOURCE_PERF_PKG ] = + { + .r_resctrl = { + .rid =RDT_RESOURCE_PERF_PKG , + .name ="PERF_PKG" , + .mon_scope =RESCTRL_PACKAGE , + .mon_domains =mon_domain_init (RDT_RESOURCE_PERF_PKG ), + }, + }, };u32 resctrl_arch_system_num_rmid_idx (void )
@@ -431,6 +440,8 @@ static int get_domain_id_from_scope(int cpu, enum resctrl_scope scope)return get_cpu_cacheinfo_id (cpu, scope);case RESCTRL_L3_NODE :return cpu_to_node (cpu); +case RESCTRL_PACKAGE : +return topology_physical_package_id (cpu);default :break ; }
Signature · 1 line
2.48.1
Tony Luck
maintainerPATCH v3 21/26 fs-x86/resctrl: Handle RDT_RESOURCE_PERF_PKG in domain create/delete
In reply to Tony LuckAdd a new rdt_perf_pkg_mon_domain structure. This only consists of the common rdt_domain_hdr as there is no need for any per-domain data structures.
Use as much as possible of the existing domain setup and tear down infrastructure. In many cases the RDT_RESOURCE_PERF_PKG uses the same functions but just skips over the pieces it does not need.
--- include/linux/resctrl.h | 8 ++++++++ arch/x86/kernel/cpu/resctrl/core.c | 32 ++++++++++++++++++++++++++++++ fs/resctrl/rdtgroup.c | 11 ++++++++-- 3 files changed, 49 insertions(+), 2 deletions(-)
include/linux/resctrl.h +8 -0
@@ -169,6 +169,14 @@ struct rdt_mon_domain {int cqm_work_cpu; }; +/** +* struct rdt_perf_pkg_mon_domain - CPUs sharing an Intel-PMT-scoped resctrl monitor resource +* @hdr: common header for different domain types +*/ +struct rdt_perf_pkg_mon_domain { +struct rdt_domain_hdr hdr; +}; +/** * struct resctrl_cache - Cache allocation related data * @cbm_len: Length of the cache bit mask
arch/x86/kernel/cpu/resctrl/core.c +32 -0
@@ -542,6 +542,29 @@ static void setup_l3_mon_domain(int cpu, int id, struct rdt_resource *r, struct } } +static void setup_intel_aet_mon_domain (int cpu,int id,struct rdt_resource *r, +struct list_head *add_pos) +{ +struct rdt_perf_pkg_mon_domain *d; +int err; + + d =kzalloc_node (sizeof (*d),GFP_KERNEL ,cpu_to_node (cpu)); +if (!d) +return ; + + d->hdr.id = id; + d->hdr.type =DOMTYPE (r->rid,DOMTYPE_MON ); +cpumask_set_cpu (cpu, &d->hdr.cpu_mask); +list_add_tail_rcu (&d->hdr.list, add_pos); + + err =resctrl_online_mon_domain (r, &d->hdr); +if (err) { +list_del_rcu (&d->hdr.list); +synchronize_rcu (); +kfree (d); + } +} +static void domain_add_cpu_mon (int cpu,struct rdt_resource *r) {int id =get_domain_id_from_scope (cpu, r->mon_scope);
@@ -571,6 +594,9 @@ static void domain_add_cpu_mon(int cpu, struct rdt_resource *r)case RDT_RESOURCE_L3 :setup_l3_mon_domain (cpu, id, r, add_pos);break ; +case RDT_RESOURCE_PERF_PKG : +setup_intel_aet_mon_domain (cpu, id, r, add_pos); +break ;default :WARN_ON_ONCE (1 ); }
@@ -668,6 +694,12 @@ static void domain_remove_cpu_mon(int cpu, struct rdt_resource *r)synchronize_rcu ();mon_domain_free (hw_dom);break ; +case RDT_RESOURCE_PERF_PKG : +resctrl_offline_mon_domain (r, d); +list_del_rcu (&hdr->list); +synchronize_rcu (); +kfree (container_of (hdr ,struct rdt_perf_pkg_mon_domain ,hdr )); +break ; } }
fs/resctrl/rdtgroup.c +9 -2
@@ -4020,6 +4020,9 @@ void resctrl_offline_mon_domain(struct rdt_resource *r, struct rdt_mon_domain *dif (resctrl_mounted &&resctrl_arch_mon_capable ())rmdir_mondata_subdir_allrdtgrp (r, &d->hdr); +if (r->rid ==RDT_RESOURCE_PERF_PKG ) + goto done; +if (resctrl_is_mbm_enabled ())cancel_delayed_work (&d->mbm_over);if (resctrl_arch_is_llc_occupancy_enabled () &&has_busy_rmid (d)) {
@@ -4036,7 +4039,7 @@ void resctrl_offline_mon_domain(struct rdt_resource *r, struct rdt_mon_domain *d }domain_destroy_mon_state (d); - +done:mutex_unlock (&rdtgroup_mutex); }
@@ -4104,12 +4107,15 @@ int resctrl_online_ctrl_domain(struct rdt_resource *r, struct rdt_ctrl_domain *dint resctrl_online_mon_domain (struct rdt_resource *r,struct rdt_domain_hdr *hdr) {struct rdt_mon_domain *d; -int err; +int err =0 ;WARN_ON_ONCE (hdr->type !=DOMTYPE (r->rid,DOMTYPE_MON )); d = container_of(hdr, struct rdt_mon_domain, hdr);mutex_lock (&rdtgroup_mutex); +if (r->rid ==RDT_RESOURCE_PERF_PKG ) + goto do_mkdir; + err =domain_setup_mon_state (r, d);if (err) goto out_unlock;
@@ -4123,6 +4129,7 @@ int resctrl_online_mon_domain(struct rdt_resource *r, struct rdt_domain_hdr *hdrif (resctrl_arch_is_llc_occupancy_enabled ())INIT_DELAYED_WORK (&d->cqm_limbo, cqm_handle_limbo); +do_mkdir:/* * If the filesystem is not mounted then only the default resource group * exists. Creation of its directories is deferred until mount time
Signature · 1 line
2.48.1
Reinette Chatre
maintainerRe: [PATCH v3 21/26] fs-x86/resctrl: Handle RDT_RESOURCE_PERF_PKG in domain create/delete
Hi Tony,
Quoted 20 lines · Tony Luck
Add a new
rdt_perf_pkg_mon_domainstructure. This only consists of the commonrdt_domain_hdras there is no need for any per-domain data structures.Use as much as possible of the existing domain setup and tear down infrastructure. In many cases the
RDT_RESOURCE_PERF_PKGuses the same functions but just skips over the pieces it does not need.Signed-off-by Tony Luck <[email protected]>--- include/linux/resctrl.h | 8 ++++++++ arch/x86/kernel/cpu/resctrl/core.c | 32 ++++++++++++++++++++++++++++++ fs/resctrl/rdtgroup.c | 11 ++++++++-- 3 files changed, 49 insertions(+), 2 deletions(-)include/linux/resctrl.h +0
-0
include/linux/resctrl.h struct rdt_mon_domain { · line 173
Quoted 5 lines
include/linux/resctrl.h +2
-0@@ -169,6 +169,14 @@ struct rdt_mon_domain {int cqm_work_cpu; }; +/** +* struct rdt_perf_pkg_mon_domain - CPUs sharing an Intel-PMT-scoped resctrl monitor resource
It is a red flag when architecture specific things ("rdt" and "Intel-PMT-scoped") land in include/linux/resctrl.h
Quoted 54 lines
include/linux/resctrl.h +6
-0+* @hdr: common header for different domain types +*/ +struct rdt_perf_pkg_mon_domain { +struct rdt_domain_hdr hdr; +}; +/** * struct resctrl_cache - Cache allocation related data * @cbm_len: Length of the cache bit mask arch/x86/kernel/cpu/resctrl/core.c +26
-0@@ -542,6 +542,29 @@ static void setup_l3_mon_domain(int cpu, int id, struct rdt_resource *r, struct } } +static void setup_intel_aet_mon_domain (int cpu,int id,struct rdt_resource *r, +struct list_head *add_pos) +{ +struct rdt_perf_pkg_mon_domain *d; +int err; + + d =kzalloc_node (sizeof (*d),GFP_KERNEL ,cpu_to_node (cpu)); +if (!d) +return ; + + d->hdr.id = id; + d->hdr.type =DOMTYPE (r->rid,DOMTYPE_MON ); +cpumask_set_cpu (cpu, &d->hdr.cpu_mask); +list_add_tail_rcu (&d->hdr.list, add_pos); + + err =resctrl_online_mon_domain (r, &d->hdr); +if (err) { +list_del_rcu (&d->hdr.list); +synchronize_rcu (); +kfree (d); + } +} +static void domain_add_cpu_mon (int cpu,struct rdt_resource *r) {int id =get_domain_id_from_scope (cpu, r->mon_scope);@@ -571,6 +594,9 @@ static void domain_add_cpu_mon(int cpu, struct rdt_resource *r)case RDT_RESOURCE_L3 :setup_l3_mon_domain (cpu, id, r, add_pos);break ; +case RDT_RESOURCE_PERF_PKG : +setup_intel_aet_mon_domain (cpu, id, r, add_pos); +break ;default :WARN_ON_ONCE (1 ); }
arch/x86/kernel/cpu/resctrl/core.c static void domain_remove_cpu_mon(int cpu, struct rdt_resource *r) · line 698
Quoted 5 lines
arch/x86/kernel/cpu/resctrl/core.c +2
-0@@ -668,6 +694,12 @@ static void domain_remove_cpu_mon(int cpu, struct rdt_resource *r)synchronize_rcu ();mon_domain_free (hw_dom);break ; +case RDT_RESOURCE_PERF_PKG : +resctrl_offline_mon_domain (r, d);
Something should have complained about using an uninitialized variable here (d).
Quoted 12 lines
arch/x86/kernel/cpu/resctrl/core.c +4
-0+list_del_rcu (&hdr->list); +synchronize_rcu (); +kfree (container_of(hdr ,struct rdt_perf_pkg_mon_domain ,hdr )); +break ; } }fs/resctrl/rdtgroup.c +0
-0
fs/resctrl/rdtgroup.c void resctrl_offline_mon_domain(struct rdt_resource *r, struct rdt_mon_domain *d · line 4024
Quoted 5 lines
fs/resctrl/rdtgroup.c +2
-0@@ -4020,6 +4020,9 @@ void resctrl_offline_mon_domain(struct rdt_resource *r, struct rdt_mon_domain *dif (resctrl_mounted &&resctrl_arch_mon_capable ())rmdir_mondata_subdir_allrdtgrp (r, &d->hdr); +if (r->rid ==RDT_RESOURCE_PERF_PKG ) + goto done;
Could you please change this test to
if (r->rid != RDT_RESOURCE_L3)
this makes it clear about what the code that follows supports as opposed to one resource that it does not support (and possibly cause issue in future when adding another monitoring resource).
Quoted 16 lines
fs/resctrl/rdtgroup.c +2
-1+if (resctrl_is_mbm_enabled ())cancel_delayed_work (&d->mbm_over);if (resctrl_arch_is_llc_occupancy_enabled () &&has_busy_rmid (d)) {@@ -4036,7 +4039,7 @@ void resctrl_offline_mon_domain(struct rdt_resource *r, struct rdt_mon_domain *d }domain_destroy_mon_state (d); - +done:mutex_unlock (&rdtgroup_mutex); }@@ -4104,12 +4107,15 @@ int resctrl_online_ctrl_domain(struct rdt_resource *r, struct rdt_ctrl_domain *dint resctrl_online_mon_domain (struct rdt_resource *r,struct rdt_domain_hdr *hdr) {
fs/resctrl/rdtgroup.c int resctrl_online_ctrl_domain(struct rdt_resource *r, struct rdt_ctrl_domain *d · line 4117
Quoted 10 lines
fs/resctrl/rdtgroup.c +3
-1struct rdt_mon_domain *d; -int err; +int err =0 ;WARN_ON_ONCE (hdr->type !=DOMTYPE (r->rid,DOMTYPE_MON )); d = container_of(hdr, struct rdt_mon_domain, hdr);mutex_lock (&rdtgroup_mutex); +if (r->rid ==RDT_RESOURCE_PERF_PKG ) + goto do_mkdir;
same
fs/resctrl/rdtgroup.c int resctrl_online_mon_domain(struct rdt_resource *r, struct rdt_domain_hdr *hdr · line 4135
Quoted 12 lines
fs/resctrl/rdtgroup.c +2
-0+ err =domain_setup_mon_state (r, d);if (err) goto out_unlock;@@ -4123,6 +4129,7 @@ int resctrl_online_mon_domain(struct rdt_resource *r, struct rdt_domain_hdr *hdrif (resctrl_arch_is_llc_occupancy_enabled ())INIT_DELAYED_WORK (&d->cqm_limbo, cqm_handle_limbo); +do_mkdir:/* * If the filesystem is not mounted then only the default resource group * exists. Creation of its directories is deferred until mount time
Reinette
Tony Luck
maintainerPATCH v3 22/26 fs/resctrl: Add type define for PERF_PKG files
In reply to Tony LuckCreation of the default info file for monitor resources requires an RFTYPE_RES_ define and mapping from the resource id.
Add the define and case in fflags_from_resource().
--- fs/resctrl/internal.h | 2 ++ fs/resctrl/rdtgroup.c | 2 ++ 2 files changed, 4 insertions(+)
fs/resctrl/internal.h +2 -0
@@ -253,6 +253,8 @@ struct rdtgroup {#define RFTYPE_DEBUG BIT(10) +#define RFTYPE_RES_PERF_PKG BIT(11) +#define RFTYPE_CTRL_INFO (RFTYPE_INFO | RFTYPE_CTRL)#define RFTYPE_MON_INFO (RFTYPE_INFO | RFTYPE_MON)
fs/resctrl/rdtgroup.c +2 -0
@@ -2175,6 +2175,8 @@ static unsigned long fflags_from_resource(struct rdt_resource *r)case RDT_RESOURCE_MBA :case RDT_RESOURCE_SMBA :return RFTYPE_RES_MB ; +case RDT_RESOURCE_PERF_PKG : +return RFTYPE_RES_PERF_PKG ; }return WARN_ON_ONCE (1 );
Signature · 1 line
2.48.1
Tony Luck
maintainerPATCH v3 23/26 fs/resctrl: Add new telemetry event id and structures
In reply to Tony LuckNine new events for energy and perf monitoring per-RMID.
--- include/linux/resctrl_types.h | 19 ++++++++++++--- fs/resctrl/monitor.c | 45 +++++++++++++++++++++++++++++++++++ 2 files changed, 61 insertions(+), 3 deletions(-)
include/linux/resctrl_types.h +16 -3
@@ -46,16 +46,29 @@ enum resctrl_res_level { };/* -* Event IDs, the values match those used to program IA32_QM_EVTSEL before -* reading IA32_QM_CTR on RDT systems. +* Event IDs */ enum resctrl_event_id { +/* Legacy events. Values must match X86 IA32_QM_EVTSEL usage */ QOS_L3_OCCUP_EVENT_ID =0x01 ,QOS_L3_MBM_TOTAL_EVENT_ID =0x02 ,QOS_L3_MBM_LOCAL_EVENT_ID =0x03 , + +/* Intel Telemetry Events */ +PMT_EVENT_ENERGY , +PMT_EVENT_ACTIVITY , +PMT_EVENT_STALLS_LLC_HIT , +PMT_EVENT_C1_RES , +PMT_EVENT_UNHALTED_CORE_CYCLES , +PMT_EVENT_STALLS_LLC_MISS , +PMT_EVENT_AUTO_C6_RES , +PMT_EVENT_UNHALTED_REF_CYCLES , +PMT_EVENT_UOPS_RETIRED , + +/* Must be the last */ +QOS_NUM_EVENTS }; -#define QOS_NUM_EVENTS (QOS_L3_MBM_LOCAL_EVENT_ID + 1)#define QOS_NUM_MBM_EVENTS (QOS_L3_MBM_LOCAL_EVENT_ID - QOS_L3_MBM_TOTAL_EVENT_ID + 1)#define MBM_EVENT_IDX (evt) ((evt) - QOS_L3_MBM_TOTAL_EVENT_ID)
fs/resctrl/monitor.c +45 -0
51 lines not on this page · raw message
Signature · 1 line
2.48.1
Tony Luck
maintainerPATCH v3 24/26 x86/resctrl: Final steps to enable RDT_RESOURCE_PERF_PKG
In reply to Tony LuckFor each of the discovered telemetry events:
Mark as enabled in the rdt_mon_features bitmap.
Set the value display type.
Mark that the event can be read from any CPU.
Because the resource was not marked as enabled during early initialization no domain discovery and allocation was done. Do that in the architecture first mount hook.
--- arch/x86/kernel/cpu/resctrl/core.c | 14 ++++++++++++++ arch/x86/kernel/cpu/resctrl/intel_aet.c | 5 +++++ 2 files changed, 19 insertions(+)
arch/x86/kernel/cpu/resctrl/core.c +14 -0
@@ -756,7 +756,9 @@ static int resctrl_arch_offline_cpu(unsigned int cpu)void resctrl_arch_mount (void ) { +struct rdt_resource *r = &rdt_resources_all[RDT_RESOURCE_PERF_PKG ].r_resctrl;static bool only_once; +int cpu;if (only_once)return ;
@@ -764,6 +766,18 @@ void resctrl_arch_mount(void)if (!intel_aet_get_events ())return ; + +/* +* Late discovery of telemetry events means the domains for the +* resource were not built. Do that now. +*/ +cpus_read_lock (); +mutex_lock (&domain_list_lock); + r->mon_capable = true; +for_each_online_cpu (cpu) +domain_add_cpu_mon (cpu, r); +mutex_unlock (&domain_list_lock); +cpus_read_unlock (); }enum {
arch/x86/kernel/cpu/resctrl/intel_aet.c +5 -0
@@ -233,9 +233,14 @@ bool intel_aet_get_events(void)continue ;for (int i =0 ; i < (*tentry)->num_events; i++) {enum resctrl_event_id evtid = (*tentry)->evts[i].evtid; +enum resctrl_event_type type; evtinfo[evtid].telem_entry = *tentry; evtinfo[evtid].pmt_event = &(*tentry)->evts[i]; + +__set_bit (evtid, rdt_mon_features); + type = (*tentry)->evts[i].type; +resctrl_set_event_attributes (evtid, type, true); } }
Signature · 1 line
2.48.1
Tony Luck
maintainerPATCH v3 25/26 fs-x86/resctrl: Add detailed descriptions for Clearwater Forest events
In reply to Tony LuckThere are two event groups one for energy reporting and another for "perf" events.
See the XML description files in https://github.com/intel/Intel-PMT
in the xml/CWF/OOBMSM/{RMID-ENERGY,RMID-PERF}/ for the detailed descriptions that were used to derive these descriptions.
--- arch/x86/kernel/cpu/resctrl/intel_aet.c | 57 +++++++++++++++++++++++++ 1 file changed, 57 insertions(+)
arch/x86/kernel/cpu/resctrl/intel_aet.c +57 -0
@@ -42,6 +42,8 @@ struct pmt_event {enum resctrl_event_type type; }; +#define EVT (id, offset, _type) { .evtid = id, .evt_offset = offset, .type = _type } +/** * struct telem_entry - Summarized form from XML telemetry description * @name: Name for this group of events
@@ -82,8 +84,63 @@ static struct evtinfo {#define EVT_OFFSET (evtid) (evtinfo[evtid].pmt_event->evt_offset) +/* +* https://github.com/intel/Intel-PMT +* xml/CWF/OOBMSM/RMID-ENERGY *.xml +*/ +#define NUM_RMIDS_0x26696143 576 +#define GUID_0x26696143 0x26696143 +#define NUM_EVENTS_0x26696143 2 +#define EVT_BYTES_0x26696143 (NUM_RMIDS_0x26696143 * NUM_EVENTS_0x26696143 * sizeof(u64)) + +static struct telem_entry energy_0x26696143 = { + .name ="energy" , + .guid = GUID_0x26696143, + .size = EVT_BYTES_0x26696143 +sizeof (u64) *3 , + .num_rmids = NUM_RMIDS_0x26696143, + .overflow_counter_off = EVT_BYTES_0x26696143 +sizeof (u64) *0 , + .last_overflow_tstamp_off = EVT_BYTES_0x26696143 +sizeof (u64) *1 , + .last_update_tstamp_off = EVT_BYTES_0x26696143 +sizeof (u64) *2 , + .num_events = NUM_EVENTS_0x26696143, + .evts = { +EVT (PMT_EVENT_ENERGY ,0x0 ,EVT_TYPE_U46_18 ), +EVT (PMT_EVENT_ACTIVITY ,0x8 ,EVT_TYPE_U46_18 ), + } +}; + +/* +* https://github.com/intel/Intel-PMT +* xml/CWF/OOBMSM/RMID-PERF *.xml +*/ +#define NUM_RMIDS_0x26557651 576 +#define GUID_0x26557651 0x26557651 +#define NUM_EVENTS_0x26557651 7 +#define EVT_BYTES_0x26557651 (NUM_RMIDS_0x26557651 * NUM_EVENTS_0x26557651 * sizeof(u64)) + +static struct telem_entry perf_0x26557651 = { + .name ="perf" , + .guid = GUID_0x26557651, + .size = EVT_BYTES_0x26557651 +sizeof (u64) *3 , + .num_rmids = NUM_RMIDS_0x26557651, + .overflow_counter_off = EVT_BYTES_0x26557651 +sizeof (u64) *0 , + .last_overflow_tstamp_off = EVT_BYTES_0x26557651 +sizeof (u64) *1 , + .last_update_tstamp_off = EVT_BYTES_0x26557651 +sizeof (u64) *2 , + .num_events = NUM_EVENTS_0x26557651, + .evts = { +EVT (PMT_EVENT_STALLS_LLC_HIT ,0x0 ,EVT_TYPE_U64 ), +EVT (PMT_EVENT_C1_RES ,0x8 ,EVT_TYPE_U64 ), +EVT (PMT_EVENT_UNHALTED_CORE_CYCLES ,0x10 ,EVT_TYPE_U64 ), +EVT (PMT_EVENT_STALLS_LLC_MISS ,0x18 ,EVT_TYPE_U64 ), +EVT (PMT_EVENT_AUTO_C6_RES ,0x20 ,EVT_TYPE_U64 ), +EVT (PMT_EVENT_UNHALTED_REF_CYCLES ,0x28 ,EVT_TYPE_U64 ), +EVT (PMT_EVENT_UOPS_RETIRED ,0x30 ,EVT_TYPE_U64 ), + } +}; +/* All known telemetry event groups */ static struct telem_entry *telem_entry[] = { + &energy_0x26696143, + &perf_0x26557651,NULL };
Signature · 1 line
2.48.1
Reinette Chatre
maintainerRe: [PATCH v3 25/26] fs-x86/resctrl: Add detailed descriptions for Clearwater Forest events
Hi Tony,
Quoted 2 lines · Tony Luck
There are two event groups one for energy reporting and another for "perf" events.
Please add context.
Quoted 6 lines
See the XML description files in https://github.com/intel/Intel-PMT
in the xml/CWF/OOBMSM/{RMID-ENERGY,RMID-PERF}/ for the detailed descriptions that were used to derive these descriptions.Signed-off-by Tony Luck <[email protected]>
The url in text can be a "Link:" here.
Quoted 18 lines
--- arch/x86/kernel/cpu/resctrl/intel_aet.c | 57 +++++++++++++++++++++++++ 1 file changed, 57 insertions(+)arch/x86/kernel/cpu/resctrl/intel_aet.c +2
-0@@ -42,6 +42,8 @@ struct pmt_event {enum resctrl_event_type type; }; +#define EVT (id, offset, _type) { .evtid = id, .evt_offset = offset, .type = _type } +/** * struct telem_entry - Summarized form from XML telemetry description * @name: Name for this group of events
arch/x86/kernel/cpu/resctrl/intel_aet.c static struct evtinfo { · line 89
Quoted 6 lines
arch/x86/kernel/cpu/resctrl/intel_aet.c +3
-0@@ -82,8 +84,63 @@ static struct evtinfo {#define EVT_OFFSET (evtid) (evtinfo[evtid].pmt_event->evt_offset) +/* +* https://github.com/intel/Intel-PMT +* xml/CWF/OOBMSM/RMID-ENERGY *.xml
This can be one line.
Quoted 14 lines
arch/x86/kernel/cpu/resctrl/intel_aet.c +14
-0+*/ +#define NUM_RMIDS_0x26696143 576 +#define GUID_0x26696143 0x26696143 +#define NUM_EVENTS_0x26696143 2 +#define EVT_BYTES_0x26696143 (NUM_RMIDS_0x26696143 * NUM_EVENTS_0x26696143 * sizeof(u64)) + +static struct telem_entry energy_0x26696143 = { + .name ="energy" , + .guid = GUID_0x26696143, + .size = EVT_BYTES_0x26696143 +sizeof (u64) *3 , + .num_rmids = NUM_RMIDS_0x26696143, + .overflow_counter_off = EVT_BYTES_0x26696143 +sizeof (u64) *0 , + .last_overflow_tstamp_off = EVT_BYTES_0x26696143 +sizeof (u64) *1 , + .last_update_tstamp_off = EVT_BYTES_0x26696143 +sizeof (u64) *2 ,
arch/x86/kernel/cpu/resctrl/intel_aet.c static struct evtinfo { · line 113
Quoted 10 lines
arch/x86/kernel/cpu/resctrl/intel_aet.c +10
-0+ .num_events = NUM_EVENTS_0x26696143, + .evts = { + EVT(PMT_EVENT_ENERGY, 0x0, EVT_TYPE_U46_18), + EVT(PMT_EVENT_ACTIVITY, 0x8, EVT_TYPE_U46_18), + } +}; + +/* +* https://github.com/intel/Intel-PMT +* xml/CWF/OOBMSM/RMID-PERF *.xml
This can be one line.
Quoted 21 lines
arch/x86/kernel/cpu/resctrl/intel_aet.c +21
-0+*/ +#define NUM_RMIDS_0x26557651 576 +#define GUID_0x26557651 0x26557651 +#define NUM_EVENTS_0x26557651 7 +#define EVT_BYTES_0x26557651 (NUM_RMIDS_0x26557651 * NUM_EVENTS_0x26557651 * sizeof(u64)) + +static struct telem_entry perf_0x26557651 = { + .name ="perf" , + .guid = GUID_0x26557651, + .size = EVT_BYTES_0x26557651 +sizeof (u64) *3 , + .num_rmids = NUM_RMIDS_0x26557651, + .overflow_counter_off = EVT_BYTES_0x26557651 +sizeof (u64) *0 , + .last_overflow_tstamp_off = EVT_BYTES_0x26557651 +sizeof (u64) *1 , + .last_update_tstamp_off = EVT_BYTES_0x26557651 +sizeof (u64) *2 , + .num_events = NUM_EVENTS_0x26557651, + .evts = { +EVT (PMT_EVENT_STALLS_LLC_HIT ,0x0 ,EVT_TYPE_U64 ), +EVT (PMT_EVENT_C1_RES ,0x8 ,EVT_TYPE_U64 ), +EVT (PMT_EVENT_UNHALTED_CORE_CYCLES ,0x10 ,EVT_TYPE_U64 ), +EVT (PMT_EVENT_STALLS_LLC_MISS ,0x18 ,EVT_TYPE_U64 ), +EVT (PMT_EVENT_AUTO_C6_RES ,0x20 ,EVT_TYPE_U64 ),
arch/x86/kernel/cpu/resctrl/intel_aet.c static struct evtinfo { · line 144
Quoted 10 lines
arch/x86/kernel/cpu/resctrl/intel_aet.c +7
-0+EVT (PMT_EVENT_UNHALTED_REF_CYCLES ,0x28 ,EVT_TYPE_U64 ), + EVT(PMT_EVENT_UOPS_RETIRED, 0x30, EVT_TYPE_U64), + } +}; +/* All known telemetry event groups */ static struct telem_entry *telem_entry[] = { + &energy_0x26696143, + &perf_0x26557651, NULL
Looks like a change from previous design to use telem_entry::num_events instead of NULL entry. The NULL entry can thus be removed?
arch/x86/kernel/cpu/resctrl/intel_aet.c static struct evtinfo { · line 146
Quoted 2 lines
arch/x86/kernel/cpu/resctrl/intel_aet.c +0
-0};
Reinette
Tony Luck
maintainerPATCH v3 26/26 x86/resctrl: Update Documentation for package events
In reply to Tony LuckEach "mon_data" directory is now divided between L3 events and package events.
The "info/PERF_PKG_MON" directory contains parameters for perf events.
--- Documentation/filesystems/resctrl.rst | 38 ++++++++++++++++++++------- 1 file changed, 28 insertions(+), 10 deletions(-)
Documentation/filesystems/resctrl.rst +28 -10
@@ -167,7 +167,7 @@ with respect to allocation: bandwidth percentages are directly applied to the threads running on the core -If RDT monitoring is available there will be an "L3_MON" directory +If RDT L3 monitoring is available there will be an "L3_MON" directory with the following files: "num_rmids":
@@ -261,6 +261,17 @@ with the following files: bytes) at which a previously used LLC_occupancy counter can be considered for re-use. +If RDT PERF monitoring is available there will be an "L3_PERF_PKG" directory +with the following files: + +"num_rmids": + The guaranteed number of hardware countes supporting RMIDs. + If more "CTRL_MON" + "MON" groups than this number are created, + the system may report that counters are "unavailable" when read. + +"mon_features": + Lists the perf monitoring events that are enabled on this system. + Finally, in the top level of the "info" directory there is a file named "last_cmd_status". This is reset with every "command" issued via the file system (making new directories or writing to any of the
@@ -366,15 +377,22 @@ When control is enabled all CTRL_MON groups will also contain: When monitoring is enabled all MON groups will also contain: "mon_data": - This contains a set of files organized by L3 domain and by - RDT event. E.g. on a system with two L3 domains there will - be subdirectories "mon_L3_00" and "mon_L3_01". Each of these - directories have one file per event (e.g. "llc_occupancy", - "mbm_total_bytes", and "mbm_local_bytes"). In a MON group these - files provide a read out of the current value of the event for - all tasks in the group. In CTRL_MON groups these files provide - the sum for all tasks in the CTRL_MON group and all tasks in - MON groups. Please see example section for more details on usage. + This contains a set of directories, one for each instance + of an L3 cache, or of a processor package. The L3 cache + directories are named "mon_L3_00", "mon_L3_01" etc. The + package directories "mon_PERF_PKG_00", "mon_PERF_PKG_01" etc. + + Within each directory there is one file per event. In + the L3 directories: "llc_occupancy", "mbm_total_bytes", + and "mbm_local_bytes". In the PERF_PKG directories: "core_energy", + "activity", etc. + + In a MON group these files provide a read out of the current + value of the event for all tasks in the group. In CTRL_MON groups + these files provide the sum for all tasks in the CTRL_MON group + and all tasks in MON groups. Please see example section for more + details on usage. + On systems with Sub-NUMA Cluster (SNC) enabled there are extra directories for each node (located within the "mon_L3_XX" directory for the L3 cache they occupy). These are named "mon_sub_L3_YY"
Signature · 1 line
2.48.1
Reinette Chatre
maintainerRe: [PATCH v3 26/26] x86/resctrl: Update Documentation for package events
Hi Tony,
Quoted 15 lines · Tony Luck
Each "
mon_data" directory is now divided between L3 events and package events.The "info/PERF_PKG_MON" directory contains parameters for perf events.
Signed-off-by Tony Luck <[email protected]>--- Documentation/filesystems/resctrl.rst | 38 ++++++++++++++++++++------- 1 file changed, 28 insertions(+), 10 deletions(-)Documentation/filesystems/resctrl.rst +0
-0
Documentation/filesystems/resctrl.rst with respect to allocation: · line 170
Quoted 5 lines
Documentation/filesystems/resctrl.rst +1
-1@@ -167,7 +167,7 @@ with respect to allocation: bandwidth percentages are directly applied to the threads running on the core -If RDT monitoring is available there will be an "L3_MON" directory +If RDT L3 monitoring is available there will be an "L3_MON" directory
I think "RDT" can just be dropped.
Documentation/filesystems/resctrl.rst with the following files: · line 264
Quoted 8 lines
Documentation/filesystems/resctrl.rst +1
-0with the following files: "num_rmids":@@ -261,6 +261,17 @@ with the following files: bytes) at which a previously used LLC_occupancy counter can be considered for re-use. +If RDT PERF monitoring is available there will be an "L3_PERF_PKG" directory
"L3_PERF_PKG" -> "PERF_PKG_MON" ?
I understand that the existing L3 documentation contains this term but I do not see a reason why the documentation should make this new monitoring Intel/RDT specific.
Also, I do not think user can be expected to know what "perf monitoring" is.
Documentation/filesystems/resctrl.rst with the following files: · line 268
Quoted 4 lines
Documentation/filesystems/resctrl.rst +4
-0+with the following files: + +"num_rmids": + The guaranteed number of hardware countes supporting RMIDs.
countes -> counters?
The use of "hardware counters" is a bit unexpected ... the series did not mention this or I must have missed this.
Documentation/filesystems/resctrl.rst with the following files: · line 270
Quoted 2 lines
Documentation/filesystems/resctrl.rst +2
-0+ If more "CTRL_MON" + "MON" groups than this number are created, + the system may report that counters are "unavailable" when read.
To be precise it is "Unavailable" ... but I do not think that is a good interface.
Documentation/filesystems/resctrl.rst with the following files: · line 273
Quoted 3 lines
Documentation/filesystems/resctrl.rst +3
-0+ +"mon_features": + Lists the perf monitoring events that are enabled on this system.
"PERF" (all caps) at top and "perf" lower case here?
Quoted 15 lines
Documentation/filesystems/resctrl.rst +1
-7+ Finally, in the top level of the "info" directory there is a file named "last_cmd_status". This is reset with every "command" issued via the file system (making new directories or writing to any of the@@ -366,15 +377,22 @@ When control is enabled all CTRL_MON groups will also contain: When monitoring is enabled all MON groups will also contain: "mon_data": - This contains a set of files organized by L3 domain and by - RDT event. E.g. on a system with two L3 domains there will - be subdirectories "mon_L3_00" and "mon_L3_01". Each of these - directories have one file per event (e.g. "llc_occupancy", - "mbm_total_bytes", and "mbm_local_bytes"). In a MON group these - files provide a read out of the current value of the event for - all tasks in the group. In CTRL_MON groups these files provide
Documentation/filesystems/resctrl.rst When control is enabled all CTRL_MON groups will also contain: · line 387
Quoted 10 lines
Documentation/filesystems/resctrl.rst +8
-2- the sum for all tasks in the CTRL_MON group and all tasks in - MON groups. Please see example section for more details on usage. + This contains a set of directories, one for each instance + of an L3 cache, or of a processor package. The L3 cache + directories are named "mon_L3_00", "mon_L3_01" etc. The + package directories "mon_PERF_PKG_00", "mon_PERF_PKG_01" etc. + + Within each directory there is one file per event. In + the L3 directories: "llc_occupancy", "mbm_total_bytes", + and "mbm_local_bytes". In the PERF_PKG directories: "core_energy",
stray tab here
Documentation/filesystems/resctrl.rst When control is enabled all CTRL_MON groups will also contain: · line 398
Quoted 11 lines
Documentation/filesystems/resctrl.rst +8
-0+ "activity", etc. + + In a MON group these files provide a read out of the current + value of the event for all tasks in the group. In CTRL_MON groups + these files provide the sum for all tasks in the CTRL_MON group + and all tasks in MON groups. Please see example section for more + details on usage. + On systems with Sub-NUMA Cluster (SNC) enabled there are extra directories for each node (located within the "mon_L3_XX" directory for the L3 cache they occupy). These are named "mon_sub_L3_YY"
Reinette
Reinette Chatre
maintainerHi Tony,
Quoted 4 lines · Tony Luck
Previous version here:
https://lore.kernel.org/all/[email protected]/This series is based on James Morse's "
fs/resctrl/" snapshot.
Would be helpful to provide link to snapshot used to avoid any uncertainty about what base to use.
Quoted 10 lines
Background
Telemetry features are being implemented in conjunction with the IA32_PQR_ASSOC.RMID value on each logical CPU. This is used to send counts for various events to a collector in a nearby OOMMSM device to be accumulated with counts for each <RMID, event> pair received from other CPUs. Cores send event counts when the RMID value changes, or after each 2ms elapsed time.
Quoted 4 lines
Each OOBMSM device may implement multiple event collectors with each servicing a subset of the logical CPUs on a package. In the initial hardware implementation, there are two categories of events:
(missing the two categories of events)
Quoted 21 lines
The counters are arranged in groups in MMIO space of the OOBMSM device. E.g. for the energy counters the layout is:
Offset: Counter
0x00core energy for RMID 00x08core activity for RMID 00x10core energy for RMID 10x18core activity for RMID 11) Energy - Two counters
core_energy: This is an estimate of Joules consumed by each core. It is calculated based on the types of instructions executed, not from a power meter. This counter is useful to understand how much energy a workload is consuming.activity: This measures "accumulated dynamic capacitance". Users who want to optimize energy consumption for a workload may use this rather than
core_energybecause it provides consistent results independent of any frequency or voltage changes that may occur during the runtime of the application (e.g. entry/exit from turbo mode).
Quoted 3 lines
2) Performance - Seven counters
These are similar events to those available via the Linux "perf" tool, but collected in a way with mush lower overhead (no need to collect data
"mush" -> "much"
Quoted 37 lines
on every context switch).
stalls_llc_hit- Counts the total number of unhalted core clock cycles when the core is stalled due to a demand load miss which hit in the LLC
c1_res- Counts the total C1 residency across all cores. The underlying counter increments on 100MHz clock ticks
unhalted_core_cycles- Counts the total number of unhalted core clock cycles
stalls_llc_miss- Counts the total number of unhalted core clock cycles when the core is stalled due to a demand load miss which missed all the local caches
c6_res- Counts the total C6 residency. The underlying counter increments on crystal clock (25MHz) ticks
unhalted_ref_cycles- Counts the total number of unhalted reference clock (TSC) cycles
uops_retired- Counts the total number of uops retiredEnumeration
The only CPUID based enumeration for this feature is the legacy CPUID(eax=7,ecx=0).ebx{12} that indicates the presence of the
IA32_PQR_ASSOCMSR and the RMID field within it.The OOBMSM driver discovers which features are present via PCIe VSEC capabilities. Each feature is tagged with a unique identifier. These identifiers indicate which XML description file from
https://github.com/intel/Intel-PMT describes which event counters are available and their layout within the MMIO BAR space of the OOBMSM device.Resctrl User Interface
Quoted 8 lines
Because there may be multiple OOBMSM collection agents per processor package, resctrl accumulates event counts from all agents on a package and presents a single value to users. This will provide a consistent user interface on future platforms that vary the number of collectors, or the mappings from logical CPUs to collectors.
Users will see the legacy monitoring files in the "L3" directories and the telemetry files in "PKG" directories (with each file
Now PERF_PKG?
Quoted 34 lines
providing the aggregated value from all OOBMSM collectors on that package).
$ tree /sys/fs/resctrl/mon_data/ /sys/fs/resctrl/mon_data/ ├── mon_L3_00 │ ├── llc_occupancy │ ├── mbm_local_bytes │ └── mbm_total_bytes ├── mon_L3_01 │ ├── llc_occupancy │ ├── mbm_local_bytes │ └── mbm_total_bytes ├── mon_PKG_00 │ ├── activity │ ├── c1_res │ ├── c6_res │ ├── core_energy │ ├── stalls_llc_hit │ ├── stalls_llc_miss │ ├── unhalted_core_cycles │ ├── unhalted_ref_cycles │ └── uops_retired └── mon_PKG_01 ├── activity ├── c1_res ├── c6_res ├── core_energy ├── stalls_llc_hit ├── stalls_llc_miss ├── unhalted_core_cycles ├── unhalted_ref_cycles └── uops_retired
Quoted 3 lines
Resctrl Implementation
The OOBMSM driver exposes a function "
intel_pmt_get_regions_by_feature()"
(nit: no need to use "a function" if using ())
Quoted 26 lines
that returns an array of structures describing the per-RMID groups it found from the VSEC enumeration. Linux looks at the unique identifiers for each group and enables resctrl for all groups with known unique identifiers.
The memory map for the counters for each <RMID, event> pair is described by the XML file. This is too unwieldy to use in the Linux kernel, so a simplified representation is built into the resctrl code. Note that the counters are in MMIO space instead of accessed using the
IA32_QM_EVTSELandIA32_QM_CTRMSRs. This means there is no need for cross-processor calls to read counters from a CPU in a specific domain. The counters can be read from any CPU.High level description of code changes:
1) New scope
RESCTRL_PACKAGE
2) New structrdt_resourceRDT_RESOURCE_INTEL_PMT
3) Refactor monitor code paths to split existing L3 paths from new ones. In some cases this ends up with:switch (r->rid) { case RDT_RESOURCE_L3: helper for L3 break; case RDT_RESOURCE_INTEL_PMT: helper for PKG break; }
Quoted 1 line
4) New source code file "
intel_pmt.c" for the code to enumerate, configure, and report event counts.
Needs an update to match new version of this work.
Quoted 8 lines
With only one platform providing this feature, it's tricky to tell exactly where it is going to go. I've made the event definitions platform specific (based on the unique ID from the VSEC enumeration). It seems possible/likely that the list of events may change from generation to generation.
I've picked names for events based on the descriptions in the XML file.
One aspect that is only hinted to in the final documentation patch is how users are expected to use this feature. As I understand the number of monitor groups supported by resctrl is still guided by the number of RMIDs supported by L3 monitoring. This work hints that the telemetry feature may not match that number of RMIDs and a monitor group may thus exist but when a user attempts to ready any of these perf files it will return "unavailable".
The series attempts to address it by placing the number of RMIDs available for this feature in a "num_rmids" file, but since the RMID assigned to a monitor group is not exposed to user space (unless debugging enabled) the user does not know if a monitor group will support this feature or not. This seems awkward to me. Why not limit the number of monitor groups that can be created to the minimum number of RMIDs across these resources like what is done for CLOSid?
Reinette
Luck, Tony
maintainerQuoted 8 lines · Reinette Chatre
One aspect that is only hinted to in the final documentation patch is how users are expected to use this feature. As I understand the number of monitor groups supported by resctrl is still guided by the number of RMIDs supported by L3 monitoring. This work hints that the telemetry feature may not match that number of RMIDs and a monitor group may thus exist but when a user attempts to ready any of these perf files it will return "unavailable".
Quoted 6 lines
The series attempts to address it by placing the number of RMIDs available for this feature in a "
num_rmids" file, but since the RMID assigned to a monitor group is not exposed to user space (unless debugging enabled) the user does not know if a monitor group will support this feature or not. This seems awkward to me. Why not limit the number of monitor groups that can be created to the minimum number of RMIDs across these resources like what is done for CLOSid?
Reinette,
The mismatch between number of RMIDs supported by different components is a thorny one, and may keep repeating since it feels like systems are composed of a bunch of lego-like bricks snapped together from a box of parts available to the h/w architect.
In this case we have three meanings for "number of RMIDs":
1) The number for legacy features enumerated by CPUID leaf 0xF.
2) The number of registers in MMIO space for each event. This is enumerated in the XML files and is the value I placed into telem_entry::num_rmids.
3) The number of "h/w counters" (this isn't a strictly accurate description of how things work, but serves as a useful analogy that does describe the limitations) feeding to those MMIO registers. This is enumerated in telemetry_region::num_rmids returned from the call to intel_pmt_get_regions_by_feature()
If "1" is the smallest of these values, the OS will be limited in which values can be written to the IA32_PQR_ASSOC MSR. Existing code will do the right thing by limiting RMID allocation to this value.
If "2" is greater than "1", then the extra MMIO registers will sit unused.
If "2" is less than "1" my v3 returns the (problematic) -ENOENT This can't happen in the CPU that debuts this feature, but the check is there to prevent running past the end of the MMIO space in case this does occur some day. I'll fix error path in next version to make sure this end up with "Unavailable".
If "3" is less than "2" then the system will attach "h/w counters" to MMIO registers in a "most recently used" algorithm. So if the number of active RMIDs in some time interval is less than "3" the user will get good values. But if the number of active RMIDs rises above "3" then the user will see "Unavailable" returns as "h/w counters" are reassigned to different RMIDs (making the feature really hard to use).
In the debut CPU the "energy" feature has sufficient "energy" counters to avoid this. But not enough "perf" counters. I've pushed and the next CPU with the feature will have enough "h/w counters".
My proposal for v4:
Add new options to the "rdt=" kernel boot parameter for "energy" and "perf".
Treat the case where there are not enough "h/w counters" as an erratum and do not enable the feature. User can override with "rdt=perf" if they want the counters for some special case where they limit the number of simultaneous active RMIDs.
User can use "rdt=!energy,!perf" if they don't want to see the clutter of all the new files in each mon_data directory.
I'll maybe look at moving resctrl_mon_resource_init() to rdt_get_tree() and add a "take min of all RMID limits". But since this is a "can't happen" scenario I may skip this if it starts to get complicated.
Which leaves what should be in info/PERF_PKG_MON/num_rmids? It's possible that some CPU implementation will have different MMIO register counts for "perf" and "energy". It's more than possible that number of "h/w counters" will be different. But they share the same info file. My v3 code reports the minimum of the number of "h/w counters" which is the most conservative option. It tells the user not to make more mon_data directories than this if they want usable counters across *both* perf and energy. Though they will actually keep getting good "energy" results even if then go past this limit.
-Tony
Quoted 1 line
Reinette Chatre
maintainerHi Tony,
Quoted 16 lines · Tony
Quoted 8 lines · Reinette Chatre
One aspect that is only hinted to in the final documentation patch is how users are expected to use this feature. As I understand the number of monitor groups supported by resctrl is still guided by the number of RMIDs supported by L3 monitoring. This work hints that the telemetry feature may not match that number of RMIDs and a monitor group may thus exist but when a user attempts to ready any of these perf files it will return "unavailable".
Quoted 6 lines
The series attempts to address it by placing the number of RMIDs available for this feature in a "
num_rmids" file, but since the RMID assigned to a monitor group is not exposed to user space (unless debugging enabled) the user does not know if a monitor group will support this feature or not. This seems awkward to me. Why not limit the number of monitor groups that can be created to the minimum number of RMIDs across these resources like what is done for CLOSid?
Quoted 6 lines
Reinette,
The mismatch between number of RMIDs supported by different components is a thorny one, and may keep repeating since it feels like systems are composed of a bunch of lego-like bricks snapped together from a box of parts available to the h/w architect.
With resctrl needing to support multiple architectures' way of doing things, needing to support variety within an architecture just seems like another step.
Quoted 5 lines
In this case we have three meanings for "number of RMIDs":
1) The number for legacy features enumerated by CPUID leaf
0xF.
Quoted 8 lines
2) The number of registers in MMIO space for each event. This is enumerated in the XML files and is the value I placed into
telem_entry::num_rmids.3) The number of "h/w counters" (this isn't a strictly accurate description of how things work, but serves as a useful analogy that does describe the limitations) feeding to those MMIO registers. This is enumerated in
telemetry_region::num_rmidsreturned from the call tointel_pmt_get_regions_by_feature()
Thank you for explaining this. This was not clear to me.
Quoted 8 lines
If "1" is the smallest of these values, the OS will be limited in which values can be written to the
IA32_PQR_ASSOCMSR. Existing code will do the right thing by limiting RMID allocation to this value.If "2" is greater than "1", then the extra MMIO registers will sit unused.
This is also an issue with this implementation, no? resctrl will not allow creating more monitor groups than "1".
Quoted 5 lines
If "2" is less than "1" my v3 returns the (problematic)
-ENOENTThis can't happen in the CPU that debuts this feature, but the check is there to prevent running past the end of the MMIO space in case this does occur some day. I'll fix error path in next version to make sure this end up with "Unavailable".
This is a concern since this means the interface becomes a "try and see" for user space. As I understand a later statement the idea is that "2" should be used by user space to know how many "mon_groups" directories should be created to get telemetry support. To me this looks to be a space that will create a lot of confusion. The moment user space creates "2" + 1 "mon_groups" directories it becomes a guessing game of what any new monitor group actually supports. After crossing that threshold I do not see a good way for going back since if user space removes one "mon_data" directory it does get back to "2" but then needs to rely on resctrl internals or debugging to know for sure what the new monitor group supports.
Quoted 7 lines
If "3" is less than "2" then the system will attach "h/w counters" to MMIO registers in a "most recently used" algorithm. So if the number of active RMIDs in some time interval is less than "3" the user will get good values. But if the number of active RMIDs rises above "3" then the user will see "Unavailable" returns as "h/w counters" are reassigned to different RMIDs (making the feature really hard to use).
Could the next step be for the architecture to allow user space to specify which hardware counters need to be assigned? With a new user interface being created for such capability it may be worthwhile to consider how it could be used/adapted for this feature. [1]
Quoted 7 lines
In the debut CPU the "energy" feature has sufficient "energy" counters to avoid this. But not enough "perf" counters. I've pushed and the next CPU with the feature will have enough "h/w counters".
My proposal for v4:
Quoted 7 lines
Add new options to the "rdt=" kernel boot parameter for "energy" and "perf".
Treat the case where there are not enough "h/w counters" as an erratum and do not enable the feature. User can override with "rdt=perf" if they want the counters for some special case where they limit the number of simultaneous active RMIDs.
This only seems to address the "3" is less than "2" issue. It is not so obvious to me that it should be treated as an erratum. Although, I could not tell from your description how obvious this issue will be to user space. For example, is it clear that if user space
gets *any* value then it is "good" and "Unavailable" means ... "Unavailable", or could a returned value mean "this is partial data that was collected during timeframe with hardware counter re-assigned at some point"?
Quoted 7 lines
User can use "rdt=!energy,!perf" if they don't want to see the clutter of all the new files in each
mon_datadirectory.I'll maybe look at moving
resctrl_mon_resource_init()tordt_get_tree()and add a "take min of all RMID limits". But since this is a "can't happen" scenario I may skip this if it starts to get complicated.
I do not think that the "2" is less than "1" scenario should be ignored for reasons stated above and in review of this version.
What if we enhance resctrl's RMID assignment (setting aside for a moment PMG assignment) to be directed by user space?
Below is an idea of an interface that can give user space control over what monitor groups are monitoring. This is very likely not the ideal interface but I would like to present it as a start for better ideas.
For example, monitor groups are by default created with most abundant (and thus supporting fewest features on fewest resources) RMID.
The user is then presented with a new file (within each monitor group) that lists all available features and which one(s) are active. For example, let's consider hypothetical example where PERF_PKG perf has x RMID, PERF_PKG energy has y RMID, and L3_MON has z RMID, with x < y < z. By default when user space creates a monitor group resctrl will pick "abundant" RMID from range y + 1 to z that only supports L3 monitoring:
# cat /sys/fs/resctrl/mon_groups/m1/new_file_mon_resource_and_features [L3] PERF_PKG:energy PERF_PKG:perf
In above case there will be *no* mon_PERF_PKG_XX directories in /sys/fs/resctrl/mon_groups/m1/mon_data.
*If* user space wants perf/energy telemetry for this monitor group then they can enable needed feature with clear understanding that it is disruptive to all ongoing monitoring since a new RMID will be assigned. For example, if user wants PERF_PKG:energy and PERF_PKG:perf then user can do so with:
# echo PERF_PKG:perf > /sys/fs/resctrl/mon_groups/m1/new_file_mon_resource_and_features # cat /sys/fs/resctrl/mon_groups/m1/new_file_mon_resource_and_features [L3] [PERF_PKG:energy] [PERF_PKG:perf]
After the above all energy and perf files will appear in new mon_PERF_PKG_XX directories.
User space can then have full control of what is monitored by which monitoring group. If no RMIDs are available in a particular pool then user space can get an "out of space" error and be the one to decide how it should be managed.
This also could be a way in which the "2" is larger than "1" scenario can be addressed.
Quoted 10 lines
Which leaves what should be in info/PERF_PKG_MON/num_rmids? It's possible that some CPU implementation will have different MMIO register counts for "perf" and "energy". It's more than possible that number of "h/w counters" will be different. But they share the same info file. My v3 code reports the minimum of the number of "h/w counters" which is the most conservative option. It tells the user not to make more
mon_datadirectories than this if they want usable counters across *both* perf and energy. Though they will actually keep getting good "energy" results even if then go past this limit.
num_rmids is a source of complications. It does not have a good equivalent for MPAM and there has been a few attempts at proposing alternatives that may be worth keeping in mind while making changes here:
https://lore.kernel.org/all/[email protected]/
https://lore.kernel.org/lkml/[email protected]/
Quoted 3 lines
-Tony
Reinette
[1] https://lore.kernel.org/lkml/[email protected]/
ps. I needed to go back an re-read the original cover-letter a couple of times, while doing so I noticed one typo in the Background section: OOMMSM -> OOBMSM.
Luck, Tony
maintainerQuoted 51 lines · Reinette Chatre
Hi Tony,
Quoted 16 lines · Tony
Quoted 8 lines · Reinette Chatre
One aspect that is only hinted to in the final documentation patch is how users are expected to use this feature. As I understand the number of monitor groups supported by resctrl is still guided by the number of RMIDs supported by L3 monitoring. This work hints that the telemetry feature may not match that number of RMIDs and a monitor group may thus exist but when a user attempts to ready any of these perf files it will return "unavailable".
Quoted 6 lines
The series attempts to address it by placing the number of RMIDs available for this feature in a "
num_rmids" file, but since the RMID assigned to a monitor group is not exposed to user space (unless debugging enabled) the user does not know if a monitor group will support this feature or not. This seems awkward to me. Why not limit the number of monitor groups that can be created to the minimum number of RMIDs across these resources like what is done for CLOSid?Quoted 6 lines
Reinette,
The mismatch between number of RMIDs supported by different components is a thorny one, and may keep repeating since it feels like systems are composed of a bunch of lego-like bricks snapped together from a box of parts available to the h/w architect.
With resctrl needing to support multiple architectures' way of doing things, needing to support variety within an architecture just seems like another step.
Quoted 5 lines
In this case we have three meanings for "number of RMIDs":
1) The number for legacy features enumerated by CPUID leaf
0xF.Quoted 8 lines
2) The number of registers in MMIO space for each event. This is enumerated in the XML files and is the value I placed into
telem_entry::num_rmids.3) The number of "h/w counters" (this isn't a strictly accurate description of how things work, but serves as a useful analogy that does describe the limitations) feeding to those MMIO registers. This is enumerated in
telemetry_region::num_rmidsreturned from the call tointel_pmt_get_regions_by_feature()Thank you for explaining this. This was not clear to me.
Quoted 6 lines
If "1" is the smallest of these values, the OS will be limited in which values can be written to the
IA32_PQR_ASSOCMSR. Existing code will do the right thing by limiting RMID allocation to this value.
Quoted 5 lines
Quoted 2 lines
If "2" is greater than "1", then the extra MMIO registers will sit unused.
This is also an issue with this implementation, no? resctrl will not allow creating more monitor groups than "1".
On Intel there is no point in creating more groups than "1" allows. You can't make use of any RMID above that limit because you will get a #GP fault trying to write to the IA32_PQR_ASSOC MSR.
You could read the extra MMIO registers provided by "2", but they will always be zero since no execution occurred with an RMID in the range "1" ... "2".
The "2" is greater than "1" may be relatively common since the h/w for the telemetry counters is common for SKUs with different numbers of cores, and thus different values of "1". So low core count systems will see more telemetry counters than they can actually make use of. I will make sure not to print a message for this case.
Quoted 6 lines
Quoted 5 lines
If "2" is less than "1" my v3 returns the (problematic)
-ENOENTThis can't happen in the CPU that debuts this feature, but the check is there to prevent running past the end of the MMIO space in case this does occur some day. I'll fix error path in next version to make sure this end up with "Unavailable".
Quoted 11 lines
This is a concern since this means the interface becomes a "try and see" for user space. As I understand a later statement the idea is that "2" should be used by user space to know how many "
mon_groups" directories should be created to get telemetry support. To me this looks to be a space that will create a lot of confusion. The moment user space creates "2" + 1 "mon_groups" directories it becomes a guessing game of what any new monitor group actually supports. After crossing that threshold I do not see a good way for going back since if user space removes one "mon_data" directory it does get back to "2" but then needs to rely on resctrl internals or debugging to know for sure what the new monitor group supports.
But I assert that it is a "can't happen" concern. "2" will be >= "1". See below. I will look at addressing this, unless it gets crazy complex because of the different enumeration timeline. Delaying calculation of number of RMIDs until rdt_get_tree() as you have suggested may be the right thing to do.
"3" is the real problem
Quoted 28 lines
Quoted 7 lines
If "3" is less than "2" then the system will attach "h/w counters" to MMIO registers in a "most recently used" algorithm. So if the number of active RMIDs in some time interval is less than "3" the user will get good values. But if the number of active RMIDs rises above "3" then the user will see "Unavailable" returns as "h/w counters" are reassigned to different RMIDs (making the feature really hard to use).
Could the next step be for the architecture to allow user space to specify which hardware counters need to be assigned? With a new user interface being created for such capability it may be worthwhile to consider how it could be used/adapted for this feature. [1]
Quoted 7 lines
In the debut CPU the "energy" feature has sufficient "energy" counters to avoid this. But not enough "perf" counters. I've pushed and the next CPU with the feature will have enough "h/w counters".
My proposal for v4:
Quoted 7 lines
Add new options to the "rdt=" kernel boot parameter for "energy" and "perf".
Treat the case where there are not enough "h/w counters" as an erratum and do not enable the feature. User can override with "rdt=perf" if they want the counters for some special case where they limit the number of simultaneous active RMIDs.
Quoted 7 lines
This only seems to address the "3" is less than "2" issue. It is not so obvious to me that it should be treated as an erratum. Although, I could not tell from your description how obvious this issue will be to user space. For example, is it clear that if user space
gets *any* value then it is "good" and "Unavailable" means ... "Unavailable", or could a returned value mean "this is partial data that was collected during timeframe with hardware counter re-assigned at some point"?
When running jobs with more distinct RMIDs than "3" users are at the mercy of the h/w replacement algorithm. Resctrl use cases for monitoring are all "read an event counter; wait for some time; re-read the event counter; compute the rate". With "h/w counter" reassignment the second read may get "Unavailable", or worse the "h/w counter" may have been taken, and the returned so a value will be provided to the user, but it won't provide the count of events since the first read.
That's why I consider this an erratum. There's just false hope that you can get a pair of meaningful event counts and no sure indication that you didn't get garbage.
Quoted 8 lines
Quoted 7 lines
User can use "rdt=!energy,!perf" if they don't want to see the clutter of all the new files in each
mon_datadirectory.I'll maybe look at moving
resctrl_mon_resource_init()tordt_get_tree()and add a "take min of all RMID limits". But since this is a "can't happen" scenario I may skip this if it starts to get complicated.
Quoted 5 lines
I do not think that the "2" is less than "1" scenario should be ignored for reasons stated above and in review of this version.
What if we enhance resctrl's RMID assignment (setting aside for a moment PMG assignment) to be directed by user space?
I'll take a look at reducing user reported num_rmids to the minimum of the "1" and "2" values.
Quoted 5 lines
Below is an idea of an interface that can give user space control over what monitor groups are monitoring. This is very likely not the ideal interface but I would like to present it as a start for better ideas.
Quoted 8 lines
For example, monitor groups are by default created with most abundant (and thus supporting fewest features on fewest resources) RMID.
The user is then presented with a new file (within each monitor group) that lists all available features and which one(s) are active. For example, let's consider hypothetical example wherePERF_PKGperf has x RMID,PERF_PKGenergy has y RMID, andL3_MONhas z RMID, with x < y < z. By default when user space creates a monitor group resctrl will pick "abundant" RMID from range y + 1 to z that only supports L3 monitoring:
There is no way for s/w to control the reallocation of "h/w counters" when "3" is too small. So there is no set of RMIDs that support many events vs. fewer events. AMD is solving this similar problem with their scheme to pin h/w counters to specific RMIDs. I discussed such an option for the "3" case, but it wasn't practical to apply to the upcoming CPU that has this problem. The long term solution is to ensure that "3" is always large enough that all RMIDs have equal monitoring capabilities.
Quoted 51 lines
# cat /sys/fs/resctrl/mon_groups/m1/new_file_mon_resource_and_features [L3] PERF_PKG:energy PERF_PKG:perfIn above case there will be *no*
mon_PERF_PKG_XXdirectories in/sys/fs/resctrl/mon_groups/m1/mon_data.*If* user space wants perf/energy telemetry for this monitor group then they can enable needed feature with clear understanding that it is disruptive to all ongoing monitoring since a new RMID will be assigned. For example, if user wants PERF_PKG:energy and PERF_PKG:perf then user can do so with:
# echo PERF_PKG:perf > /sys/fs/resctrl/mon_groups/m1/new_file_mon_resource_and_features # cat /sys/fs/resctrl/mon_groups/m1/new_file_mon_resource_and_features [L3] [PERF_PKG:energy] [PERF_PKG:perf]After the above all energy and perf files will appear in new
mon_PERF_PKG_XXdirectories.User space can then have full control of what is monitored by which monitoring group. If no RMIDs are available in a particular pool then user space can get an "out of space" error and be the one to decide how it should be managed.
This also could be a way in which the "2" is larger than "1" scenario can be addressed.
Quoted 10 lines
Which leaves what should be in info/PERF_PKG_MON/num_rmids? It's possible that some CPU implementation will have different MMIO register counts for "perf" and "energy". It's more than possible that number of "h/w counters" will be different. But they share the same info file. My v3 code reports the minimum of the number of "h/w counters" which is the most conservative option. It tells the user not to make more
mon_datadirectories than this if they want usable counters across *both* perf and energy. Though they will actually keep getting good "energy" results even if then go past this limit.
num_rmidsis a source of complications. It does not have a good equivalent for MPAM and there has been a few attempts at proposing alternatives that may be worth keeping in mind while making changes here:
https://lore.kernel.org/all/[email protected]/
https://lore.kernel.org/lkml/[email protected]/Quoted 3 lines
-Tony
Quoted 7 lines
Reinette
[1] https://lore.kernel.org/lkml/[email protected]/
ps. I needed to go back an re-read the original cover-letter a couple of times, while doing so I noticed one typo in the Background section: OOMMSM -> OOBMSM.
Noted. Will fix.
-Tony
Reinette Chatre
maintainerHi Tony,
Quoted 66 lines · Tony
Quoted 51 lines · Reinette Chatre
Hi Tony,
Quoted 16 lines · Tony
Quoted 8 lines · Reinette Chatre
One aspect that is only hinted to in the final documentation patch is how users are expected to use this feature. As I understand the number of monitor groups supported by resctrl is still guided by the number of RMIDs supported by L3 monitoring. This work hints that the telemetry feature may not match that number of RMIDs and a monitor group may thus exist but when a user attempts to ready any of these perf files it will return "unavailable".
Quoted 6 lines
The series attempts to address it by placing the number of RMIDs available for this feature in a "
num_rmids" file, but since the RMID assigned to a monitor group is not exposed to user space (unless debugging enabled) the user does not know if a monitor group will support this feature or not. This seems awkward to me. Why not limit the number of monitor groups that can be created to the minimum number of RMIDs across these resources like what is done for CLOSid?Quoted 6 lines
Reinette,
The mismatch between number of RMIDs supported by different components is a thorny one, and may keep repeating since it feels like systems are composed of a bunch of lego-like bricks snapped together from a box of parts available to the h/w architect.
With resctrl needing to support multiple architectures' way of doing things, needing to support variety within an architecture just seems like another step.
Quoted 5 lines
In this case we have three meanings for "number of RMIDs":
1) The number for legacy features enumerated by CPUID leaf
0xF.Quoted 8 lines
2) The number of registers in MMIO space for each event. This is enumerated in the XML files and is the value I placed into
telem_entry::num_rmids.3) The number of "h/w counters" (this isn't a strictly accurate description of how things work, but serves as a useful analogy that does describe the limitations) feeding to those MMIO registers. This is enumerated in
telemetry_region::num_rmidsreturned from the call tointel_pmt_get_regions_by_feature()Thank you for explaining this. This was not clear to me.
Quoted 6 lines
If "1" is the smallest of these values, the OS will be limited in which values can be written to the
IA32_PQR_ASSOCMSR. Existing code will do the right thing by limiting RMID allocation to this value.Quoted 5 lines
Quoted 2 lines
If "2" is greater than "1", then the extra MMIO registers will sit unused.
This is also an issue with this implementation, no? resctrl will not allow creating more monitor groups than "1".
On Intel there is no point in creating more groups than "1" allows. You can't make use of any RMID above that limit because you will get a #GP fault trying to write to the
IA32_PQR_ASSOCMSR.You could read the extra MMIO registers provided by "2", but they will always be zero since no execution occurred with an RMID in the range "1" ... "2".
Quoted 5 lines
The "2" is greater than "1" may be relatively common since the h/w for the telemetry counters is common for SKUs with different numbers of cores, and thus different values of "1". So low core count systems will see more telemetry counters than they can actually make use of. I will make sure not to print a message for this case.
I see, thank you.
Quoted 19 lines
Quoted 7 lines
Quoted 6 lines
If "2" is less than "1" my v3 returns the (problematic)
-ENOENTThis can't happen in the CPU that debuts this feature, but the check is there to prevent running past the end of the MMIO space in case this does occur some day. I'll fix error path in next version to make sure this end up with "Unavailable".Quoted 11 lines
This is a concern since this means the interface becomes a "try and see" for user space. As I understand a later statement the idea is that "2" should be used by user space to know how many "
mon_groups" directories should be created to get telemetry support. To me this looks to be a space that will create a lot of confusion. The moment user space creates "2" + 1 "mon_groups" directories it becomes a guessing game of what any new monitor group actually supports. After crossing that threshold I do not see a good way for going back since if user space removes one "mon_data" directory it does get back to "2" but then needs to rely on resctrl internals or debugging to know for sure what the new monitor group supports.
Quoted 5 lines
But I assert that it is a "can't happen" concern. "2" will be >= "1". See below. I will look at addressing this, unless it gets crazy complex because of the different enumeration timeline. Delaying calculation of number of RMIDs until
rdt_get_tree()as you have suggested may be the right thing to do.
As you explain it, it does not sound as though calculating how many RMIDs can be supported is required to be done in rdt_get_tree(), but doing so would make the implementation more robust since doing so does not rely on assumptions about what hardware can and will support.
Quoted 30 lines
"3" is the real problem
Quoted 27 lines
Quoted 7 lines
If "3" is less than "2" then the system will attach "h/w counters" to MMIO registers in a "most recently used" algorithm. So if the number of active RMIDs in some time interval is less than "3" the user will get good values. But if the number of active RMIDs rises above "3" then the user will see "Unavailable" returns as "h/w counters" are reassigned to different RMIDs (making the feature really hard to use).
Could the next step be for the architecture to allow user space to specify which hardware counters need to be assigned? With a new user interface being created for such capability it may be worthwhile to consider how it could be used/adapted for this feature. [1]
Quoted 14 lines
In the debut CPU the "energy" feature has sufficient "energy" counters to avoid this. But not enough "perf" counters. I've pushed and the next CPU with the feature will have enough "h/w counters".
My proposal for v4:
Add new options to the "rdt=" kernel boot parameter for "energy" and "perf".
Treat the case where there are not enough "h/w counters" as an erratum and do not enable the feature. User can override with "rdt=perf" if they want the counters for some special case where they limit the number of simultaneous active RMIDs.
I get this now. This will require rework of the kernel command line parsing support since current implementation is so closely integrated with the X86_FEATURE_* flags (and is perhaps an unexpected architecture specific portion of resctrl).
What if "rdt=perf" means that "3" is also included in the computation of how many monitor groups are supported? That would help users to not need to limit the number of simultaneous active RMIDs.
Quoted 29 lines
Quoted 8 lines
This only seems to address the "3" is less than "2" issue. It is not so obvious to me that it should be treated as an erratum. Although, I could not tell from your description how obvious this issue will be to user space. For example, is it clear that if user space
gets *any* value then it is "good" and "Unavailable" means ... "Unavailable", or could a returned value mean "this is partial data that was collected during timeframe with hardware counter re-assigned at some point"?When running jobs with more distinct RMIDs than "3" users are at the mercy of the h/w replacement algorithm. Resctrl use cases for monitoring are all "read an event counter; wait for some time; re-read the event counter; compute the rate". With "h/w counter" reassignment the second read may get "Unavailable", or worse the "h/w counter" may have been taken, and the returned so a value will be provided to the user, but it won't provide the count of events since the first read.
That's why I consider this an erratum. There's just false hope that you can get a pair of meaningful event counts and no sure indication that you didn't get garbage.
Quoted 8 lines
Quoted 7 lines
User can use "rdt=!energy,!perf" if they don't want to see the clutter of all the new files in each
mon_datadirectory.I'll maybe look at moving
resctrl_mon_resource_init()tordt_get_tree()and add a "take min of all RMID limits". But since this is a "can't happen" scenario I may skip this if it starts to get complicated.
Quoted 8 lines
Quoted 5 lines
I do not think that the "2" is less than "1" scenario should be ignored for reasons stated above and in review of this version.
What if we enhance resctrl's RMID assignment (setting aside for a moment PMG assignment) to be directed by user space?
I'll take a look at reducing user reported
num_rmidsto the minimum of the "1" and "2" values.
When comparing to "num_closids" the expectation may be that "num_rmids" would be accurate for particular resource with understanding that the minimum among all resources guides the number of monitor groups. This seems close enough to existing interface to not use this as moment to move to a new "num_mon_hw_id" or such that works for MPAM also.
Quoted 15 lines
Quoted 6 lines
Below is an idea of an interface that can give user space control over what monitor groups are monitoring. This is very likely not the ideal interface but I would like to present it as a start for better ideas.
Quoted 8 lines
For example, monitor groups are by default created with most abundant (and thus supporting fewest features on fewest resources) RMID.
The user is then presented with a new file (within each monitor group) that lists all available features and which one(s) are active. For example, let's consider hypothetical example wherePERF_PKGperf has x RMID,PERF_PKGenergy has y RMID, andL3_MONhas z RMID, with x < y < z. By default when user space creates a monitor group resctrl will pick "abundant" RMID from range y + 1 to z that only supports L3 monitoring:
Quoted 7 lines
There is no way for s/w to control the reallocation of "h/w counters" when "3" is too small. So there is no set of RMIDs that support many events vs. fewer events. AMD is solving this similar problem with their scheme to pin h/w counters to specific RMIDs. I discussed such an option for the "3" case, but it wasn't practical to apply to the upcoming CPU that has this problem. The long term solution is to ensure that "3" is always large enough that all RMIDs have equal monitoring capabilities.
ack.
Reinette
Reinette Chatre
maintainerHi Tony,
Just noticed ... could you please include x86 maintainers (X86 ARCHITECTURE (32-BIT AND 64-BIT) in MAINTAINERS) in your submission?
Thank you
Reinette