x86,fs/resctrl telemetry monitoring
Subsystems (3)
Tony Luck
maintainerv11 cover x86,fs/resctrl telemetry monitoring
Patches based on tip tree x86/cache branch
Top commit of x86/cache at time of my snapshot was:dd86b69d20fb ("fs/resctrl: Fix counter auto-assignment on mkdir with mbm_event enabled")
The combination of above components plus this series is here:git://git.kernel.org/pub/scm/linux/kernel/git/aegl/linux.git rdt-aet-v11
Top commit:74134daf5662 ("fs/resctrl: Some kerneldoc updates")
Changes since v10 was posted here:
Global: Rewrote several commit messages based on suggestions and guidance from Boris[1]. Dropped RB tags where commit message rewritten.
V10 patch references in form "xx/28 {subject}. V11 numbering changes after patch 18. V11 patch numbers as 4 digits (e.g. 0019).
=== 00/28 x86,fs/resctrl: telemetry monitoring ===
Background section in cover letter unchanged.
=== 01/28 x86,fs/resctrl: Improve domain type checking ===
New commit message, no code changes.
=== 02/28 x86/resctrl: Move L3 initialization into new helper function ===
New commit message (suggested by Boris).
Put the call to resctrl_arch_mbm_cntr_assign_set_one() that was lost in a merge error back ... do it inside "case RDT_RESOURCE_L3:" as it needs to be done on every CPU that comes online.
=== 03/28 x86,fs/resctrl: Refactor domain_remove_cpu_mon() ready for new domain types ===
Commit message had stale descriptions of code changes that have been moved to later patches in the series. Drop spurious pieces.
=== 04/28 x86/resctrl: Clean up domain_remove_cpu_ctrl() ===
RB tag from Reinette applied from earlier review. No changes.
=== 05/28 x86,fs/resctrl: Use struct rdt_domain_hdr instead of struct rdt_mon_domain ===
Strip out all changes unrelated to rdt_domain_hdr. Rewrite commit message based on Boris suggestion.
Fix "Subject" to match change since change split into two patches. Fix unbalanced domain_header_is_valid() / container_of() call.
=== 06/28 x86,fs/resctrl: Use struct rdt_domain_hdr when reading counters ===
Strip out all changes unrelated to rdt_domain_hdr. Rewrite commit message using Boris suggested text.
=== 07/28 x86,fs/resctrl: Rename struct rdt_mon_domain and rdt_hw_mon_domain ===
Simplify the commit message.
=== 08/28 x86,fs/resctrl: Rename some L3 specific functions ===
No change
=== 09/28 fs/resctrl: Make event details accessible to functions when reading events ===
Simplify commit comment.
=== 10/28 x86,fs/resctrl: Handle events that can be read from any CPU ===
v10 had moved part of this patch earlier, but that jumbled extra changes into the rdt_domain_hdr refactoring. Brought those parts back into this patch.
Simplify commit comment.
=== 11/28 x86,fs/resctrl: Support binary fixed point event counters ===
Applied conditionsal RB tag from Reinette (after fixing "nit: "0.5." -> "0.5") in v10 from v9 review. No other changes.
=== 12/28 x86,fs/resctrl: Add an architectural hook called for each mount ===
RB tag from Reinette applied from earlier review. No changes.
=== 13/28 x86,fs/resctrl: Add and initialize rdt_resource for package scope monitor ===
RB tag from Reinette applied from earlier review. No changes.
=== 14/28 x86/resctrl: Discover hardware telemetry events ===
No change
=== 15/28 x86,fs/resctrl: Fill in details of events for guid 0x26696143 and 0x26557651 ===
No change
=== 16/28 x86,fs/resctrl: Add architectural event pointer ===
No change
=== 17/28 x86/resctrl: Find and enable usable telemetry events ===
No change
=== 18/28 x86/resctrl: Read telemetry events ===
0018 Add new patch "fs/resctrl: Refactor L3 specific parts of __mon_event_count() with
changes deferred from rdt_domain_hdr patch.
0019 No changes from v10 patch 18
=== 19/28 x86/resctrl: Handle domain creation/deletion for RDT_RESOURCE_PERF_PKG ===
0020 Add new patch "fs/resctrl: Refactor Sub-NUMA Cluster (SNC) in mkdir/rmdir code flow"
with changes deferred from rdt_domain_hdr patch.
0021 Simplfy commit message. No code change from v10 patch 19
=== 20/28 x86/resctrl: Add energy/perf choices to rdt boot option ===
0022 No change from v10 patch 20
=== 21/28 x86/resctrl: Handle number of RMIDs supported by telemetry resources ===
0023 No change from v10 patch 21
=== 22/28 fs/resctrl: Move allocation/free of closid_num_dirty_rmid[] ===
0024 No change from v10 patch 22
=== 23/28 fs,x86/resctrl: Compute number of RMIDs as minimum across resources ===
0025 No change from v10 patch 23
=== 24/28 fs/resctrl: Move RMID initialization to first mount ===
0026 No change from v10 patch 24
=== 25/28 x86/resctrl: Enable RDT_RESOURCE_PERF_PKG ===
0027 No change from v10 patch 25
=== 26/28 fs/resctrl: Provide interface to create architecture specific debugfs area ===
0028 No change from v10 patch 26
=== 27/28 x86/resctrl: Add debugfs files to show telemetry aggregator status ===
0029 No change from v10 patch 27
=== 28/28 x86,fs/resctrl: Update Documentation for package events ===
0030 No change from v10 patch 28
0031 Add new patch "fs/resctrl: Update kerneldoc for rmid_read::sum and mon_get_kn_priv()
(suggested by Reinette in v7, but bundled in with other changes). Plus fix "Return:" for functions touched in this series.
Background
----------
On Intel systems that support per-RMID telemetry monitoring each logical processor keeps a local count for various events. When the IA32_PQR_ASSOC.RMID value for the logical processor changes (or when a two millisecond counter expires) these event counts are transmitted to an event aggregator on the same package as the processor together with the current RMID value. The event counters are reset to zero to begin counting again.
Each aggregator takes the incoming event counts and adds them to cumulative counts for each event for each RMID. Note that there can be multiple aggregators on each package with no architectural association between logical processors and an aggregator.
All of these aggregated counters can be read by an operating system from the MMIO space of the Out Of Band Management Service Module (OOBMSM) device(s) on a system. Any counter can be read from any logical processor.
Intel publishes details for each processor generation showing which events are counted by each logical processor and the offsets for each accumulated counter value within the MMIO space in XML files here:
https://github.com/intel/Intel-PMT.
For example there are two energy related telemetry events for the Clearwater Forest family of processors and the MMIO space looks like this:
Offset RMID Event ------ ---- ----- 0x0000 0 core_energy 0x0008 0 activity 0x0010 1 core_energy 0x0018 1 activity ... 0x23F0 575 core_energy 0x23F8 575 activity
In addition the XML file provides the units (Joules for core_energy, Farads for activity) and the type of data (fixed-point binary with bit 63 used to indicate the data is valid, and the low 18 bits as a binary fraction).
Finally, each XML file provides a 32-bit unique id (or guid) that is used as an index to find the correct XML description file for each telemetry implementation.
The INTEL_PMT_TELEMETRY driver provides intel_pmt_get_regions_by_feature() to enumerate the aggregator instances (also referred to as "telemetry regions" in this series) on a platform. It provides:
1) guid - so resctrl can determine which events are supported
2) MMIO base address of counters
3) package id
Resctrl accumulates counts from all aggregators on a package in order to provide a consistent user interface across processor generations.
Directory structure for the telemetry events looks like this:
$ tree /sys/fs/resctrl/mon_data/
/sys/fs/resctrl/mon_data/
mon_data
├── mon_PERF_PKG_00
│ ├── activity
│ └── core_energy
└── mon_PERF_PKG_01
├── activity
└── core_energyReading the "core_energy" file from some resctrl mon_data directory shows the cumulative energy (in Joules) used by all tasks that ran with the RMID associated with that directory on a given package. Note that "core_energy" reports only energy consumed by CPU cores (data processing units, L1/L2 caches, etc.). It does not include energy used in the "uncore" (L3 cache, on package devices, etc.), or used by memory or I/O devices.
Tony Luck (31):
x86,fs/resctrl: Improve domain type checking
x86/resctrl: Move L3 initialization into new helper function x86,fs/resctrl: Refactor domain_remove_cpu_mon() ready for new domain
types
x86/resctrl: Clean up domain_remove_cpu_ctrl() x86,fs/resctrl: Refactor domain create/remove using struct
rdt_domain_hdr
x86,fs/resctrl: Use struct rdt_domain_hdr when reading counters x86,fs/resctrl: Rename struct rdt_mon_domain and rdt_hw_mon_domain x86,fs/resctrl: Rename some L3 specific functions fs/resctrl: Make event details accessible to functions when reading
events
x86,fs/resctrl: Handle events that can be read from any CPU x86,fs/resctrl: Support binary fixed point event counters x86,fs/resctrl: Add an architectural hook called for each mount x86,fs/resctrl: Add and initialize rdt_resource for package scope
monitor
x86/resctrl: Discover hardware telemetry events x86,fs/resctrl: Fill in details of events for guid 0x26696143 and
0x26557651
x86,fs/resctrl: Add architectural event pointer x86/resctrl: Find and enable usable telemetry events fs/resctrl: Refactor L3 specific parts of __mon_event_count() x86/resctrl: Read telemetry events
fs/resctrl: Refactor Sub-NUMA Cluster (SNC) in mkdir/rmdir code flow x86/resctrl: Handle domain creation/deletion for RDT_RESOURCE_PERF_PKG x86/resctrl: Add energy/perf choices to rdt boot option x86/resctrl: Handle number of RMIDs supported by telemetry resources fs/resctrl: Move allocation/free of closid_num_dirty_rmid[] fs,x86/resctrl: Compute number of RMIDs as minimum across resources fs/resctrl: Move RMID initialization to first mount x86/resctrl: Enable RDT_RESOURCE_PERF_PKG
fs/resctrl: Provide interface to create architecture specific debugfs
area
x86/resctrl: Add debugfs files to show telemetry aggregator status x86,fs/resctrl: Update Documentation for package events fs/resctrl: Some kerneldoc updates
.../admin-guide/kernel-parameters.txt | 2 +- Documentation/filesystems/resctrl.rst | 100 ++++- include/linux/resctrl.h | 69 ++- include/linux/resctrl_types.h | 11 + arch/x86/kernel/cpu/resctrl/internal.h | 50 ++- fs/resctrl/internal.h | 68 ++- arch/x86/kernel/cpu/resctrl/core.c | 270 ++++++++---- arch/x86/kernel/cpu/resctrl/intel_aet.c | 417 ++++++++++++++++++ arch/x86/kernel/cpu/resctrl/monitor.c | 52 ++- fs/resctrl/ctrlmondata.c | 123 +++++- fs/resctrl/monitor.c | 343 ++++++++------ fs/resctrl/rdtgroup.c | 200 ++++++--- arch/x86/Kconfig | 13 + arch/x86/kernel/cpu/resctrl/Makefile | 1 + 14 files changed, 1352 insertions(+), 367 deletions(-) create mode 100644 arch/x86/kernel/cpu/resctrl/intel_aet.c
base-commit: dd86b69d20fb9fa7e941ed01ff05f1e662fcc3ff
Signature · 1 line
2.51.0
Tony Luck
maintainerPATCH v11 1/31 x86,fs/resctrl: Improve domain type checking
Each resctrl resource has a list of domain structures. These all begin with a common rdt_domain_hdr.
Improve type checking of these headers by adding the resource id. Add domain_header_is_valid() before each call to container_of() to ensure the domain is the expected type.
--- include/linux/resctrl.h | 9 +++++++++ arch/x86/kernel/cpu/resctrl/core.c | 10 ++++++---- fs/resctrl/ctrlmondata.c | 2 +- 3 files changed, 16 insertions(+), 5 deletions(-)
include/linux/resctrl.h +9 -0
@@ -131,15 +131,24 @@ enum resctrl_domain_type {* @list: all instances of this resource * @id: unique id for this instance * @type: type of this instance +* @rid: resource id for this instance * @cpu_mask: which CPUs share this resource */ struct rdt_domain_hdr {struct list_head list;int id;enum resctrl_domain_type type; +enum resctrl_res_level rid;struct cpumask cpu_mask; }; +static inline bool domain_header_is_valid (struct rdt_domain_hdr *hdr, +enum resctrl_domain_type type, +enum resctrl_res_level rid) +{ +return !WARN_ON_ONCE (hdr->type != type || hdr->rid != rid); +} +/** * struct rdt_ctrl_domain - group of CPUs sharing a resctrl control resource * @hdr: common header for different domain types
arch/x86/kernel/cpu/resctrl/core.c +6 -4
@@ -459,7 +459,7 @@ static void domain_add_cpu_ctrl(int cpu, struct rdt_resource *r) hdr =resctrl_find_domain (&r->ctrl_domains, id, &add_pos);if (hdr) { -if (WARN_ON_ONCE (hdr->type !=RESCTRL_CTRL_DOMAIN )) +if (!domain_header_is_valid (hdr,RESCTRL_CTRL_DOMAIN , r->rid))return ; d = container_of(hdr, struct rdt_ctrl_domain, hdr);
@@ -476,6 +476,7 @@ static void domain_add_cpu_ctrl(int cpu, struct rdt_resource *r) d = &hw_dom->d_resctrl; d->hdr.id = id; d->hdr.type =RESCTRL_CTRL_DOMAIN ; + d->hdr.rid = r->rid;cpumask_set_cpu (cpu, &d->hdr.cpu_mask);rdt_domain_reconfigure_cdp (r);
@@ -515,7 +516,7 @@ static void domain_add_cpu_mon(int cpu, struct rdt_resource *r) hdr =resctrl_find_domain (&r->mon_domains, id, &add_pos);if (hdr) { -if (WARN_ON_ONCE (hdr->type !=RESCTRL_MON_DOMAIN )) +if (!domain_header_is_valid (hdr,RESCTRL_MON_DOMAIN , r->rid))return ; d = container_of(hdr, struct rdt_mon_domain, hdr);
@@ -533,6 +534,7 @@ static void domain_add_cpu_mon(int cpu, struct rdt_resource *r) d = &hw_dom->d_resctrl; d->hdr.id = id; d->hdr.type =RESCTRL_MON_DOMAIN ; + d->hdr.rid = r->rid; ci =get_cpu_cacheinfo_level (cpu,RESCTRL_L3_CACHE );if (!ci) {pr_warn_once ("Can't find L3 cache for CPU:%d resource %s\n" , cpu, r->name);
@@ -593,7 +595,7 @@ static void domain_remove_cpu_ctrl(int cpu, struct rdt_resource *r)return ; } -if (WARN_ON_ONCE (hdr->type !=RESCTRL_CTRL_DOMAIN )) +if (!domain_header_is_valid (hdr,RESCTRL_CTRL_DOMAIN , r->rid))return ; d = container_of(hdr, struct rdt_ctrl_domain, hdr);
@@ -639,7 +641,7 @@ static void domain_remove_cpu_mon(int cpu, struct rdt_resource *r)return ; } -if (WARN_ON_ONCE (hdr->type !=RESCTRL_MON_DOMAIN )) +if (!domain_header_is_valid (hdr,RESCTRL_MON_DOMAIN , r->rid))return ; d = container_of(hdr, struct rdt_mon_domain, hdr);
fs/resctrl/ctrlmondata.c +1 -1
@@ -649,7 +649,7 @@ int rdtgroup_mondata_show(struct seq_file *m, void *arg)* the resource to find the domain with "domid". */ hdr =resctrl_find_domain (&r->mon_domains, domid,NULL ); -if (!hdr ||WARN_ON_ONCE (hdr->type !=RESCTRL_MON_DOMAIN )) { +if (!hdr || !domain_header_is_valid (hdr,RESCTRL_MON_DOMAIN , resid)) { ret = -ENOENT ; goto out; }
Signature · 1 line
2.51.0
Reinette Chatre
maintainerRe: [PATCH v11 01/31] x86,fs/resctrl: Improve domain type checking
Hi Tony,
Quoted 6 lines · Tony Luck
Each resctrl resource has a list of domain structures. These all begin with a common
rdt_domain_hdr.Improve type checking of these headers by adding the resource id. Add
domain_header_is_valid()before each call tocontainer_of()to ensure the domain is the expected type.
Apart from the "short and sweet" part of the guidance [1] followed in the changelog above the same guidance concluded with summary (in bold) that the *why* of the patch should be clear. This changelog is missing the "why".
Here is an attempt to address that, please feel free to rewrite and improve:
Every resctrl resource has a list of domain structures. struct rdt_ctrl_domain and struct rdt_mon_domain both begin with struct rdt_domain_hdr with rdt_domain_hdr::type used in validity checks before accessing the domain of a particular type.
Add the resource id to struct rdt_domain_hdr in preparation for a new monitoring domain structure that will be associated with a new monitoring resource. Improve existing domain validity checks with a new helper domain_header_is_valid() that checks both domain type and resource id. domain_header_is_valid() should be used before every call to container_of() that accesses a domain structure.
[1] https://lore.kernel.org/all/20250923100956.GAaNJx9BYhXKkfNJ71@fat_crate.local/
Quoted 3 lines
Signed-off-by Tony Luck <[email protected]>---
Patch looks good to me.
Reinette
Tony Luck
maintainerPATCH v11 2/31 x86/resctrl: Move L3 initialization into new helper function
In reply to Tony LuckCarve out the resource monitoring domain init code into a separate helper in order to be able to initialize new types of monitoring domains besides the usual L3 ones.
--- arch/x86/kernel/cpu/resctrl/core.c | 64 ++++++++++++++++-------------- 1 file changed, 34 insertions(+), 30 deletions(-)
arch/x86/kernel/cpu/resctrl/core.c +34 -30
@@ -496,37 +496,13 @@ static void domain_add_cpu_ctrl(int cpu, struct rdt_resource *r) } } -static void domain_add_cpu_mon (int cpu,struct rdt_resource *r) +static void l3_mon_domain_setup (int cpu,int id,struct rdt_resource *r,struct list_head *add_pos) { -int id =get_domain_id_from_scope (cpu, r->mon_scope); -struct list_head *add_pos =NULL ;struct rdt_hw_mon_domain *hw_dom; -struct rdt_domain_hdr *hdr;struct rdt_mon_domain *d;struct cacheinfo *ci;int err; -lockdep_assert_held (&domain_list_lock); - -if (id <0 ) { -pr_warn_once ("Can't find monitor domain id for CPU:%d scope:%d for resource %s\n" , - cpu, r->mon_scope, r->name); -return ; - } - - hdr =resctrl_find_domain (&r->mon_domains, id, &add_pos); -if (hdr) { -if (!domain_header_is_valid (hdr,RESCTRL_MON_DOMAIN , r->rid)) -return ; - d = container_of(hdr, struct rdt_mon_domain, hdr); - -cpumask_set_cpu (cpu, &d->hdr.cpu_mask); -/* Update the mbm_assign_mode state for the CPU if supported */ -if (r->mon.mbm_cntr_assignable) -resctrl_arch_mbm_cntr_assign_set_one (r); -return ; - } - hw_dom =kzalloc_node (sizeof (*hw_dom),GFP_KERNEL ,cpu_to_node (cpu));if (!hw_dom)return ;
@@ -534,7 +510,7 @@ static void domain_add_cpu_mon(int cpu, struct rdt_resource *r) d = &hw_dom->d_resctrl; d->hdr.id = id; d->hdr.type =RESCTRL_MON_DOMAIN ; - d->hdr.rid = r->rid; + d->hdr.rid =RDT_RESOURCE_L3 ; ci =get_cpu_cacheinfo_level (cpu,RESCTRL_L3_CACHE );if (!ci) {pr_warn_once ("Can't find L3 cache for CPU:%d resource %s\n" , cpu, r->name);
@@ -544,10 +520,6 @@ static void domain_add_cpu_mon(int cpu, struct rdt_resource *r) d->ci_id = ci->id;cpumask_set_cpu (cpu, &d->hdr.cpu_mask); -/* Update the mbm_assign_mode state for the CPU if supported */ -if (r->mon.mbm_cntr_assignable) -resctrl_arch_mbm_cntr_assign_set_one (r); -arch_mon_domain_online (r, d);if (arch_domain_mbm_alloc (r->mon.num_rmid, hw_dom)) {
@@ -565,6 +537,38 @@ static void domain_add_cpu_mon(int cpu, struct rdt_resource *r) } } +static void domain_add_cpu_mon (int cpu,struct rdt_resource *r) +{ +int id =get_domain_id_from_scope (cpu, r->mon_scope); +struct list_head *add_pos =NULL ; +struct rdt_domain_hdr *hdr; + +lockdep_assert_held (&domain_list_lock); + +if (id <0 ) { +pr_warn_once ("Can't find monitor domain id for CPU:%d scope:%d for resource %s\n" , + cpu, r->mon_scope, r->name); +return ; + } + + hdr =resctrl_find_domain (&r->mon_domains, id, &add_pos); +if (hdr) +cpumask_set_cpu (cpu, &hdr->cpu_mask); + +switch (r->rid) { +case RDT_RESOURCE_L3 : +/* Update the mbm_assign_mode state for the CPU if supported */ +if (r->mon.mbm_cntr_assignable) +resctrl_arch_mbm_cntr_assign_set_one (r); +if (!hdr) +l3_mon_domain_setup (cpu, id, r, add_pos); +break ; +default : +pr_warn_once ("Unknown resource rid=%d\n" , r->rid); +break ; + } +} +static void domain_add_cpu (int cpu,struct rdt_resource *r) {if (r->alloc_capable)
Signature · 1 line
2.51.0
Re: [PATCH v11 02/31] x86/resctrl: Move L3 initialization into new helper function
Hi Tony,
Quoted 6 lines · Tony Luck
Carve out the resource monitoring domain init code into a separate helper in order to be able to initialize new types of monitoring domains besides the usual L3 ones.
Signed-off-by Tony Luck <[email protected]>---
Reinette
Tony Luck
maintainerPATCH v11 3/31 x86,fs/resctrl: Refactor domain_remove_cpu_mon() ready for new domain types
In reply to Tony LuckNew telemetry events will be associated with a new package scoped resource with new domain structures.
Refactor domain_remove_cpu_mon() so all the L3 processing is separate from general actions of clearing the CPU bit in the mask.
--- arch/x86/kernel/cpu/resctrl/core.c | 21 +++++++++++++-------- 1 file changed, 13 insertions(+), 8 deletions(-)
arch/x86/kernel/cpu/resctrl/core.c +13 -8
@@ -645,20 +645,25 @@ static void domain_remove_cpu_mon(int cpu, struct rdt_resource *r)return ; } -if (!domain_header_is_valid (hdr,RESCTRL_MON_DOMAIN , r->rid)) +cpumask_clear_cpu (cpu, &hdr->cpu_mask); +if (!cpumask_empty (&hdr->cpu_mask))return ; - d = container_of(hdr, struct rdt_mon_domain, hdr); - hw_dom =resctrl_to_arch_mon_dom (d); +switch (r->rid) { +case RDT_RESOURCE_L3 : +if (!domain_header_is_valid (hdr,RESCTRL_MON_DOMAIN ,RDT_RESOURCE_L3 )) +return ; -cpumask_clear_cpu (cpu, &d->hdr.cpu_mask); -if (cpumask_empty (&d->hdr.cpu_mask)) { + d = container_of(hdr, struct rdt_mon_domain, hdr); + hw_dom =resctrl_to_arch_mon_dom (d);resctrl_offline_mon_domain (r, d); -list_del_rcu (&d->hdr.list); +list_del_rcu (&hdr->list);synchronize_rcu ();mon_domain_free (hw_dom); - -return ; +break ; +default : +pr_warn_once ("Unknown resource rid=%d\n" , r->rid); +break ; } }
Signature · 1 line
2.51.0
Reinette Chatre
maintainerRe: [PATCH v11 03/31] x86,fs/resctrl: Refactor domain_remove_cpu_mon() ready for new domain types
Hi Tony,
Note that subject prefix indicates resctrl fs code is changed in this patch but it only changes arch code.
Quoted 4 lines · Tony Luck
New telemetry events will be associated with a new package scoped resource with new domain structures.
Refactor
domain_remove_cpu_mon()so all the L3 processing is separate
"L3 processing" -> "L3 domain processing" ?
Quoted 1 line
from general actions of clearing the CPU bit in the mask.
"general actions" -> "general domain actions" ?
Quoted 13 lines
Signed-off-by Tony Luck <[email protected]>--- arch/x86/kernel/cpu/resctrl/core.c | 21 +++++++++++++-------- 1 file changed, 13 insertions(+), 8 deletions(-)arch/x86/kernel/cpu/resctrl/core.c +0
-0@@ -645,20 +645,25 @@ static void domain_remove_cpu_mon(int cpu, struct rdt_resource *r)return ; }
arch/x86/kernel/cpu/resctrl/core.c static void domain_remove_cpu_mon(int cpu, struct rdt_resource *r) · line 653
Quoted 10 lines
arch/x86/kernel/cpu/resctrl/core.c +4
-3-if (!domain_header_is_valid (hdr,RESCTRL_MON_DOMAIN , r->rid)) +cpumask_clear_cpu (cpu, &hdr->cpu_mask); +if (!cpumask_empty (&hdr->cpu_mask))return ; - d = container_of(hdr, struct rdt_mon_domain, hdr); - hw_dom =resctrl_to_arch_mon_dom (d); +switch (r->rid) { +case RDT_RESOURCE_L3 :
This function evolves to where its local declarations contain a mix of domain structures of different resources. I think it will make this function easier to understand if the resource specific structures are declared local to the code block dedicated to that resource.
case RDT_RESOURCE_L3: {
struct rdt_hw_mon_domain *hw_dom;
struct rdt_mon_domain *d;
...
break;
}Quoted 11 lines
arch/x86/kernel/cpu/resctrl/core.c +5
-3+if (!domain_header_is_valid (hdr,RESCTRL_MON_DOMAIN ,RDT_RESOURCE_L3 )) +return ; -cpumask_clear_cpu (cpu, &d->hdr.cpu_mask); -if (cpumask_empty (&d->hdr.cpu_mask)) { + d = container_of(hdr, struct rdt_mon_domain, hdr); + hw_dom =resctrl_to_arch_mon_dom (d);resctrl_offline_mon_domain (r, d); -list_del_rcu (&d->hdr.list); +list_del_rcu (&hdr->list);synchronize_rcu ();
arch/x86/kernel/cpu/resctrl/core.c static void domain_remove_cpu_mon(int cpu, struct rdt_resource *r) · line 669
Quoted 10 lines
arch/x86/kernel/cpu/resctrl/core.c +4
-2mon_domain_free (hw_dom); - -return ; +break ; +default : +pr_warn_once ("Unknown resource rid=%d\n" , r->rid); +break ; } }
Reinette
Tony Luck
maintainerPATCH v11 4/31 x86/resctrl: Clean up domain_remove_cpu_ctrl()
In reply to Tony LuckFor symmetry with domain_remove_cpu_mon() refactor domain_remove_cpu_ctrl() to take an early return when removing a CPU does not empty the domain.
--- arch/x86/kernel/cpu/resctrl/core.c | 29 ++++++++++++++--------------- 1 file changed, 14 insertions(+), 15 deletions(-)
arch/x86/kernel/cpu/resctrl/core.c +14 -15
@@ -599,28 +599,27 @@ static void domain_remove_cpu_ctrl(int cpu, struct rdt_resource *r)return ; } +cpumask_clear_cpu (cpu, &hdr->cpu_mask); +if (!cpumask_empty (&hdr->cpu_mask)) +return ; +if (!domain_header_is_valid (hdr,RESCTRL_CTRL_DOMAIN , r->rid))return ; d = container_of(hdr, struct rdt_ctrl_domain, hdr); hw_dom =resctrl_to_arch_ctrl_dom (d); -cpumask_clear_cpu (cpu, &d->hdr.cpu_mask); -if (cpumask_empty (&d->hdr.cpu_mask)) { -resctrl_offline_ctrl_domain (r, d); -list_del_rcu (&d->hdr.list); -synchronize_rcu (); - -/* -* rdt_ctrl_domain "d" is going to be freed below, so clear -* its pointer from pseudo_lock_region struct. -*/ -if (d->plr) - d->plr->d =NULL ; -ctrl_domain_free (hw_dom); +resctrl_offline_ctrl_domain (r, d); +list_del_rcu (&hdr->list); +synchronize_rcu (); -return ; - } +/* +* rdt_ctrl_domain "d" is going to be freed below, so clear +* its pointer from pseudo_lock_region struct. +*/ +if (d->plr) + d->plr->d =NULL ; +ctrl_domain_free (hw_dom); }static void domain_remove_cpu_mon (int cpu,struct rdt_resource *r)
Signature · 1 line
2.51.0
Reinette Chatre
maintainerRe: [PATCH v11 04/31] x86/resctrl: Clean up domain_remove_cpu_ctrl()
Hi Tony,
Quoted 3 lines · Tony Luck
For symmetry with
domain_remove_cpu_mon()refactordomain_remove_cpu_ctrl()to take an early return when removing a CPU does not empty the domain.
These changelog lines are getting shorter and shorter. I do not know if you noticed but almost all the ABMC changelogs were reformatted during merge to use closer to 80 characters per line, sometimes more. You can avoid that extra churn by ensuring the changelogs make use of 80 columns. There was a brief exchange about this at
https://lore.kernel.org/lkml/20250916105447.GCaMlB976WLxHHeNMD@fat_crate.local/
Quoted 3 lines
Signed-off-by Tony Luck <[email protected]>Reviewed-by Reinette Chatre <[email protected]>
Reinette
Tony Luck
maintainerPATCH v11 5/31 x86,fs/resctrl: Refactor domain create/remove using struct rdt_domain_hdr
In reply to Tony LuckUp until now, all monitoring events were associated with the L3 resource and it made sense to use the L3 specific "struct rdt_mon_domain *" arguments to functions manipulating domains.
To simplify enabling of enumeration of domains for events in other resources change the calling convention to pass the generic struct rdt_domain_hdr and use that to find the domain specific structure where needed.
--- include/linux/resctrl.h | 4 +- fs/resctrl/internal.h | 2 +- arch/x86/kernel/cpu/resctrl/core.c | 4 +- fs/resctrl/ctrlmondata.c | 15 ++++--- fs/resctrl/rdtgroup.c | 65 ++++++++++++++++++++---------- 5 files changed, 58 insertions(+), 32 deletions(-)
include/linux/resctrl.h +2 -2
@@ -504,9 +504,9 @@ int resctrl_arch_update_one(struct rdt_resource *r, struct rdt_ctrl_domain *d,u32 resctrl_arch_get_config (struct rdt_resource *r,struct rdt_ctrl_domain *d,u32 closid,enum resctrl_conf_type type);int resctrl_online_ctrl_domain (struct rdt_resource *r,struct rdt_ctrl_domain *d); -int resctrl_online_mon_domain (struct rdt_resource *r,struct rdt_mon_domain *d); +int resctrl_online_mon_domain (struct rdt_resource *r,struct rdt_domain_hdr *hdr);void resctrl_offline_ctrl_domain (struct rdt_resource *r,struct rdt_ctrl_domain *d); -void resctrl_offline_mon_domain (struct rdt_resource *r,struct rdt_mon_domain *d); +void resctrl_offline_mon_domain (struct rdt_resource *r,struct rdt_domain_hdr *hdr);void resctrl_online_cpu (unsigned int cpu);void resctrl_offline_cpu (unsigned int cpu);
fs/resctrl/internal.h +1 -1
@@ -362,7 +362,7 @@ void mon_event_count(void *info);int rdtgroup_mondata_show (struct seq_file *m,void *arg);void mon_event_read (struct rmid_read *rr,struct rdt_resource *r, -struct rdt_mon_domain *d,struct rdtgroup *rdtgrp, +struct rdt_domain_hdr *hdr,struct rdtgroup *rdtgrp,cpumask_t *cpumask,int evtid,int first);int resctrl_mon_resource_init (void );
arch/x86/kernel/cpu/resctrl/core.c +2 -2
@@ -529,7 +529,7 @@ static void l3_mon_domain_setup(int cpu, int id, struct rdt_resource *r, structlist_add_tail_rcu (&d->hdr.list, add_pos); - err =resctrl_online_mon_domain (r, d); + err =resctrl_online_mon_domain (r, &d->hdr);if (err) {list_del_rcu (&d->hdr.list);synchronize_rcu ();
@@ -655,7 +655,7 @@ static void domain_remove_cpu_mon(int cpu, struct rdt_resource *r) d = container_of(hdr, struct rdt_mon_domain, hdr); hw_dom =resctrl_to_arch_mon_dom (d); -resctrl_offline_mon_domain (r, d); +resctrl_offline_mon_domain (r, hdr);list_del_rcu (&hdr->list);synchronize_rcu ();mon_domain_free (hw_dom);
fs/resctrl/ctrlmondata.c +10 -5
@@ -547,11 +547,16 @@ struct rdt_domain_hdr *resctrl_find_domain(struct list_head *h, int id, }void mon_event_read (struct rmid_read *rr,struct rdt_resource *r, -struct rdt_mon_domain *d,struct rdtgroup *rdtgrp, +struct rdt_domain_hdr *hdr,struct rdtgroup *rdtgrp,cpumask_t *cpumask,int evtid,int first) { +struct rdt_mon_domain *d;int cpu; +if (!domain_header_is_valid (hdr,RESCTRL_MON_DOMAIN ,RDT_RESOURCE_L3 )) +return ; + d = container_of(hdr, struct rdt_mon_domain, hdr); +/* When picking a CPU from cpu_mask, ensure it can't race with cpuhp */ lockdep_assert_cpus_held ();
@@ -598,7 +603,6 @@ int rdtgroup_mondata_show(struct seq_file *m, void *arg)enum resctrl_event_id evtid;struct rdt_domain_hdr *hdr;struct rmid_read rr = {0 }; -struct rdt_mon_domain *d;struct rdtgroup *rdtgrp;int domid, cpu, ret =0 ;struct rdt_resource *r;
@@ -623,6 +627,8 @@ int rdtgroup_mondata_show(struct seq_file *m, void *arg) r =resctrl_arch_get_resource (resid);if (md->sum) { +struct rdt_mon_domain *d; +/* * This file requires summing across all domains that share * the L3 cache id that was provided in the "domid" field of the
@@ -649,12 +655,11 @@ int rdtgroup_mondata_show(struct seq_file *m, void *arg)* the resource to find the domain with "domid". */ hdr =resctrl_find_domain (&r->mon_domains, domid,NULL ); -if (!hdr || !domain_header_is_valid (hdr,RESCTRL_MON_DOMAIN , resid)) { +if (!hdr) { ret = -ENOENT ; goto out; } - d = container_of(hdr, struct rdt_mon_domain, hdr); -mon_event_read (&rr, r, d, rdtgrp, &d->hdr.cpu_mask, evtid, false); +mon_event_read (&rr, r, hdr, rdtgrp, &hdr->cpu_mask, evtid, false); } checkresult:
fs/resctrl/rdtgroup.c +43 -22
199 lines not on this page · raw message
Signature · 1 line
2.51.0
Reinette Chatre
maintainerRe: [PATCH v11 05/31] x86,fs/resctrl: Refactor domain create/remove using struct rdt_domain_hdr
Hi Tony,
Quoted 5 lines · Tony Luck
Up until now, all monitoring events were associated with the L3 resource and it made sense to use the L3 specific "struct
rdt_mon_domain*" arguments to functions manipulating domains.To simplify enabling of enumeration of domains for events in other
What does "enabling of enumeration of domains" mean?
Quoted 3 lines
resources change the calling convention to pass the generic struct
rdt_domain_hdrand use that to find the domain specific structure where needed.
I think it will be helpful to highlight that this is a stepping stone that highlights what domain management code is L3 specific and thus in need of further refactoring to support new domain types vs. what is generic.
Quoted 3 lines
Signed-off-by Tony Luck <[email protected]>---
...
Quoted 6 lines
fs/resctrl/ctrlmondata.c +0
-0@@ -547,11 +547,16 @@ struct rdt_domain_hdr *resctrl_find_domain(struct list_head *h, int id, }
fs/resctrl/ctrlmondata.c struct rdt_domain_hdr *resctrl_find_domain(struct list_head *h, int id, · line 556
Quoted 10 lines
fs/resctrl/ctrlmondata.c +3
-1void mon_event_read (struct rmid_read *rr,struct rdt_resource *r, -struct rdt_mon_domain *d,struct rdtgroup *rdtgrp, +struct rdt_domain_hdr *hdr,struct rdtgroup *rdtgrp,cpumask_t *cpumask,int evtid,int first) { +struct rdt_mon_domain *d;int cpu; +if (!domain_header_is_valid (hdr,RESCTRL_MON_DOMAIN ,RDT_RESOURCE_L3 ))
hdr can be NULL here so this is not safe. I understand this is removed in the next patch but it is difficult to reason about the code if the steps are not solid.
Quoted 72 lines
fs/resctrl/ctrlmondata.c +7
-4+return ; + d = container_of(hdr, struct rdt_mon_domain, hdr); +/* When picking a CPU from cpu_mask, ensure it can't race with cpuhp */ lockdep_assert_cpus_held ();@@ -598,7 +603,6 @@ int rdtgroup_mondata_show(struct seq_file *m, void *arg)enum resctrl_event_id evtid;struct rdt_domain_hdr *hdr;struct rmid_read rr = {0 }; -struct rdt_mon_domain *d;struct rdtgroup *rdtgrp;int domid, cpu, ret =0 ;struct rdt_resource *r;@@ -623,6 +627,8 @@ int rdtgroup_mondata_show(struct seq_file *m, void *arg) r =resctrl_arch_get_resource (resid);if (md->sum) { +struct rdt_mon_domain *d; +/* * This file requires summing across all domains that share * the L3 cache id that was provided in the "domid" field of the @@ -649,12 +655,11 @@ int rdtgroup_mondata_show(struct seq_file *m, void *arg)* the resource to find the domain with "domid". */ hdr =resctrl_find_domain (&r->mon_domains, domid,NULL ); -if (!hdr || !domain_header_is_valid (hdr,RESCTRL_MON_DOMAIN , resid)) { +if (!hdr) { ret = -ENOENT ; goto out; } - d = container_of(hdr, struct rdt_mon_domain, hdr); -mon_event_read (&rr, r, d, rdtgrp, &d->hdr.cpu_mask, evtid, false); +mon_event_read (&rr, r, hdr, rdtgrp, &hdr->cpu_mask, evtid, false); } checkresult:fs/resctrl/rdtgroup.c +8
-3@@ -3164,13 +3164,18 @@ static void mon_rmdir_one_subdir(struct kernfs_node *pkn, char *name, char *subn* when last domain being summed is removed. */ static void rmdir_mondata_subdir_allrdtgrp (struct rdt_resource *r, -struct rdt_mon_domain *d) +struct rdt_domain_hdr *hdr) {struct rdtgroup *prgrp, *crgrp; +struct rdt_mon_domain *d;char subname[32 ];bool snc_mode;char name[32 ]; +if (!domain_header_is_valid (hdr,RESCTRL_MON_DOMAIN ,RDT_RESOURCE_L3 )) +return ; + + d = container_of(hdr, struct rdt_mon_domain, hdr); snc_mode = r->mon_scope ==RESCTRL_L3_NODE ;sprintf (name,"mon_%s_%02d" , r->name, snc_mode ? d->ci_id : d->hdr.id);if (snc_mode)@@ -3184,19 +3189,18 @@ static void rmdir_mondata_subdir_allrdtgrp(struct rdt_resource *r, } } -static int mon_add_all_files (struct kernfs_node *kn,struct rdt_mon_domain *d, +static int mon_add_all_files (struct kernfs_node *kn,struct rdt_domain_hdr *hdr,struct rdt_resource *r,struct rdtgroup *prgrp, -bool do_sum) +int domid,bool do_sum) {
fs/resctrl/rdtgroup.c static void rmdir_mondata_subdir_allrdtgrp(struct rdt_resource *r, · line 3199
Quoted 10 lines
fs/resctrl/rdtgroup.c +1
-2struct rmid_read rr = {0 };struct mon_data *priv;struct mon_evt *mevt; -int ret, domid; +int ret;for_each_mon_event (mevt) {if (mevt->rid != r->rid || !mevt->enabled)continue ; - domid = do_sum ? d->ci_id : d->hdr.id;
Looks like an unrelated change. Would this not be more appropriate for "fs/resctrl: Refactor Sub-NUMA Cluster (SNC) in mkdir/rmdir code flow"?
Quoted 35 lines
fs/resctrl/rdtgroup.c +7
-2priv =mon_get_kn_priv (r->rid, domid, mevt, do_sum);if (WARN_ON_ONCE (!priv))return -EINVAL ;@@ -3206,23 +3210,28 @@ static int mon_add_all_files(struct kernfs_node *kn, struct rdt_mon_domain *d,return ret;if (!do_sum &&resctrl_is_mbm_event (mevt->evtid)) -mon_event_read (&rr, r, d, prgrp, &d->hdr.cpu_mask, mevt->evtid, true); +mon_event_read (&rr, r, hdr, prgrp, &hdr->cpu_mask, mevt->evtid, true); }return 0 ; }static int mkdir_mondata_subdir (struct kernfs_node *parent_kn, -struct rdt_mon_domain *d, +struct rdt_domain_hdr *hdr,struct rdt_resource *r,struct rdtgroup *prgrp) {struct kernfs_node *kn, *ckn; +struct rdt_mon_domain *d;char name[32 ];bool snc_mode;int ret =0 ;lockdep_assert_held (&rdtgroup_mutex); +if (!domain_header_is_valid (hdr,RESCTRL_MON_DOMAIN ,RDT_RESOURCE_L3 )) +return -EINVAL ; + + d = container_of(hdr, struct rdt_mon_domain, hdr); snc_mode = r->mon_scope ==RESCTRL_L3_NODE ;sprintf (name,"mon_%s_%02d" , r->name, snc_mode ? d->ci_id : d->hdr.id); kn =kernfs_find_and_get (parent_kn, name);
fs/resctrl/rdtgroup.c static int mkdir_mondata_subdir(struct kernfs_node *parent_kn, · line 3252
Quoted 5 lines
fs/resctrl/rdtgroup.c +1
-1@@ -3240,13 +3249,13 @@ static int mkdir_mondata_subdir(struct kernfs_node *parent_kn, ret =rdtgroup_kn_set_ugid (kn);if (ret) goto out_destroy; - ret =mon_add_all_files (kn, d, r, prgrp, snc_mode); + ret =mon_add_all_files (kn, hdr, r, prgrp, hdr->id, snc_mode);
This does not seem right ... looks like this aims to do some of the SNC enabling but the domain id is always set to the domain of the node and does not distinguish between the L3 id and node id?
Quoted 74 lines
fs/resctrl/rdtgroup.c +11
-9if (ret) goto out_destroy; }if (snc_mode) { -sprintf (name,"mon_sub_%s_%02d" , r->name, d->hdr.id); +sprintf (name,"mon_sub_%s_%02d" , r->name, hdr->id); ckn =kernfs_create_dir (kn, name, parent_kn->mode, prgrp);if (IS_ERR (ckn)) { ret = -EINVAL ;@@ -3257,7 +3266,7 @@ static int mkdir_mondata_subdir(struct kernfs_node *parent_kn,if (ret) goto out_destroy; - ret =mon_add_all_files (ckn, d, r, prgrp, false); + ret =mon_add_all_files (ckn, hdr, r, prgrp, hdr->id, false);if (ret) goto out_destroy; }@@ -3275,7 +3284,7 @@ static int mkdir_mondata_subdir(struct kernfs_node *parent_kn,* and "monitor" groups with given domain id. */ static void mkdir_mondata_subdir_allrdtgrp (struct rdt_resource *r, -struct rdt_mon_domain *d) +struct rdt_domain_hdr *hdr) {struct kernfs_node *parent_kn;struct rdtgroup *prgrp, *crgrp;@@ -3283,12 +3292,12 @@ static void mkdir_mondata_subdir_allrdtgrp(struct rdt_resource *r,list_for_each_entry (prgrp, &rdt_all_groups, rdtgroup_list) { parent_kn = prgrp->mon.mon_data_kn; -mkdir_mondata_subdir (parent_kn, d, r, prgrp); +mkdir_mondata_subdir (parent_kn, hdr, r, prgrp); head = &prgrp->mon.crdtgrp_list;list_for_each_entry (crgrp, head, mon.crdtgrp_list) { parent_kn = crgrp->mon.mon_data_kn; -mkdir_mondata_subdir (parent_kn, d, r, crgrp); +mkdir_mondata_subdir (parent_kn, hdr, r, crgrp); } } }@@ -3297,14 +3306,14 @@ static int mkdir_mondata_subdir_alldom(struct kernfs_node *parent_kn,struct rdt_resource *r, struct rdtgroup *prgrp) { - struct rdt_mon_domain *dom; + struct rdt_domain_hdr *hdr;int ret;/* Walking r->domains, ensure it can't race with cpuhp */ lockdep_assert_cpus_held (); -list_for_each_entry (dom, &r->mon_domains, hdr.list) { - ret =mkdir_mondata_subdir (parent_kn, dom, r, prgrp); +list_for_each_entry (hdr, &r->mon_domains, list) { + ret =mkdir_mondata_subdir (parent_kn, hdr, r, prgrp);if (ret)return ret; }@@ -4187,8 +4196,10 @@ void resctrl_offline_ctrl_domain(struct rdt_resource *r, struct rdt_ctrl_domainmutex_unlock (&rdtgroup_mutex); } -void resctrl_offline_mon_domain (struct rdt_resource *r,struct rdt_mon_domain *d) +void resctrl_offline_mon_domain (struct rdt_resource *r,struct rdt_domain_hdr *hdr) { +struct rdt_mon_domain *d; +mutex_lock (&rdtgroup_mutex);/*
fs/resctrl/rdtgroup.c void resctrl_offline_mon_domain(struct rdt_resource *r, struct rdt_mon_domain *d · line 4214
Quoted 9 lines
fs/resctrl/rdtgroup.c +4
-1@@ -4196,8 +4207,12 @@ void resctrl_offline_mon_domain(struct rdt_resource *r, struct rdt_mon_domain *d* per domain monitor data directories. */ if (resctrl_mounted &&resctrl_arch_mon_capable ()) -rmdir_mondata_subdir_allrdtgrp (r, d); +rmdir_mondata_subdir_allrdtgrp (r, hdr); +if (!domain_header_is_valid (hdr,RESCTRL_MON_DOMAIN ,RDT_RESOURCE_L3 )) + goto out_unlock; +
One logical change per patch please.
While all other L3 specific functions modified to receive hdr as parameter are changed to use container_of() at beginning of function to highlight that the functions are L3 specific ... resctrl_offline_mon_domain() is changed differently. Looks like this changes the flow to sneak in some PERF_PKG enabling for convenience and thus makes this patch harder to understand. Splitting resctrl_offline_mon_domain() to handle different domain types seems more appropriate for "x86/resctrl: Handle domain creation/deletion for RDT_RESOURCE_PERF_PKG" where it should be clear what changes are made to support PERF_PKG. In this patch, in this stage of series, the entire function can be L3 specific.
Quoted 34 lines
fs/resctrl/rdtgroup.c +9
-3+ d = container_of(hdr, struct rdt_mon_domain, hdr);if (resctrl_is_mbm_enabled ())cancel_delayed_work (&d->mbm_over);if (resctrl_is_mon_event_enabled (QOS_L3_OCCUP_EVENT_ID ) &&has_busy_rmid (d)) {@@ -4214,7 +4229,7 @@ void resctrl_offline_mon_domain(struct rdt_resource *r, struct rdt_mon_domain *d }domain_destroy_mon_state (d); - +out_unlock:mutex_unlock (&rdtgroup_mutex); }@@ -4287,12 +4302,17 @@ int resctrl_online_ctrl_domain(struct rdt_resource *r, struct rdt_ctrl_domain *dreturn err; } -int resctrl_online_mon_domain (struct rdt_resource *r,struct rdt_mon_domain *d) +int resctrl_online_mon_domain (struct rdt_resource *r,struct rdt_domain_hdr *hdr) { -int err; +struct rdt_mon_domain *d; +int err = -EINVAL ;mutex_lock (&rdtgroup_mutex); +if (!domain_header_is_valid (hdr,RESCTRL_MON_DOMAIN ,RDT_RESOURCE_L3 )) + goto out_unlock; + + d = container_of(hdr, struct rdt_mon_domain, hdr); err =domain_setup_mon_state (r, d);if (err) goto out_unlock;
fs/resctrl/rdtgroup.c int resctrl_online_mon_domain(struct rdt_resource *r, struct rdt_mon_domain *d) · line 4329
Quoted 4 lines
fs/resctrl/rdtgroup.c +1
-0@@ -4306,6 +4326,7 @@ int resctrl_online_mon_domain(struct rdt_resource *r, struct rdt_mon_domain *d)if (resctrl_is_mon_event_enabled (QOS_L3_OCCUP_EVENT_ID ))INIT_DELAYED_WORK (&d->cqm_limbo, cqm_handle_limbo); + err =0 ;
Considering the earlier exit on "if (err)", err can be expected to be 0 here?
fs/resctrl/rdtgroup.c int resctrl_online_mon_domain(struct rdt_resource *r, struct rdt_mon_domain *d) · line 4340
Quoted 12 lines
fs/resctrl/rdtgroup.c +1
-1/* * If the filesystem is not mounted then only the default resource group * exists. Creation of its directories is deferred until mount time @@ -4313,7 +4334,7 @@ int resctrl_online_mon_domain(struct rdt_resource *r, struct rdt_mon_domain *d)* If resctrl is mounted, add per domain monitor data directories. */ if (resctrl_mounted &&resctrl_arch_mon_capable ()) -mkdir_mondata_subdir_allrdtgrp (r, d); +mkdir_mondata_subdir_allrdtgrp (r, hdr); out_unlock:mutex_unlock (&rdtgroup_mutex);
Reinette
Luck, Tony
maintainerRe: [PATCH v11 05/31] x86,fs/resctrl: Refactor domain create/remove using struct rdt_domain_hdr
Quoted 10 lines · Reinette Chatre
Hi Tony,
Quoted 5 lines · Tony Luck
Up until now, all monitoring events were associated with the L3 resource and it made sense to use the L3 specific "struct
rdt_mon_domain*" arguments to functions manipulating domains.To simplify enabling of enumeration of domains for events in other
What does "enabling of enumeration of domains" mean?
Is this better?
To prepare for events in resources other than L3, change the calling convention to pass the generic struct rdt_domain_hdr and use that to find the domain specific structure where needed.
Quoted 8 lines
Quoted 4 lines
resources change the calling convention to pass the generic struct
rdt_domain_hdrand use that to find the domain specific structure where needed.I think it will be helpful to highlight that this is a stepping stone that highlights what domain management code is L3 specific and thus in need of further refactoring to support new domain types vs. what is generic.
Above re-wording cover this.
Quoted 22 lines
Quoted 4 lines
Signed-off-by Tony Luck <[email protected]>---
...
Quoted 15 lines
fs/resctrl/ctrlmondata.c +2
-1@@ -547,11 +547,16 @@ struct rdt_domain_hdr *resctrl_find_domain(struct list_head *h, int id, }void mon_event_read (struct rmid_read *rr,struct rdt_resource *r, -struct rdt_mon_domain *d,struct rdtgroup *rdtgrp, +struct rdt_domain_hdr *hdr,struct rdtgroup *rdtgrp,cpumask_t *cpumask,int evtid,int first) { +struct rdt_mon_domain *d;int cpu;
Quoted 4 lines
fs/resctrl/ctrlmondata.c struct rdt_domain_hdr *resctrl_find_domain(struct list_head *h, int id, · line 556
Quoted 1 line
fs/resctrl/ctrlmondata.c +1
-0+if (!domain_header_is_valid (hdr,RESCTRL_MON_DOMAIN ,RDT_RESOURCE_L3 ))hdr can be NULL here so this is not safe. I understand this is removed in the next patch but it is difficult to reason about the code if the steps are not solid.
Will fix.
Quoted 79 lines
Quoted 79 lines
fs/resctrl/ctrlmondata.c +7
-4+return ; + d = container_of(hdr, struct rdt_mon_domain, hdr); +/* When picking a CPU from cpu_mask, ensure it can't race with cpuhp */ lockdep_assert_cpus_held ();@@ -598,7 +603,6 @@ int rdtgroup_mondata_show(struct seq_file *m, void *arg)enum resctrl_event_id evtid;struct rdt_domain_hdr *hdr;struct rmid_read rr = {0 }; -struct rdt_mon_domain *d;struct rdtgroup *rdtgrp;int domid, cpu, ret =0 ;struct rdt_resource *r;@@ -623,6 +627,8 @@ int rdtgroup_mondata_show(struct seq_file *m, void *arg) r =resctrl_arch_get_resource (resid);if (md->sum) { +struct rdt_mon_domain *d; +/* * This file requires summing across all domains that share * the L3 cache id that was provided in the "domid" field of the @@ -649,12 +655,11 @@ int rdtgroup_mondata_show(struct seq_file *m, void *arg)* the resource to find the domain with "domid". */ hdr =resctrl_find_domain (&r->mon_domains, domid,NULL ); -if (!hdr || !domain_header_is_valid (hdr,RESCTRL_MON_DOMAIN , resid)) { +if (!hdr) { ret = -ENOENT ; goto out; } - d = container_of(hdr, struct rdt_mon_domain, hdr); -mon_event_read (&rr, r, d, rdtgrp, &d->hdr.cpu_mask, evtid, false); +mon_event_read (&rr, r, hdr, rdtgrp, &hdr->cpu_mask, evtid, false); } checkresult:fs/resctrl/rdtgroup.c +9
-4@@ -3164,13 +3164,18 @@ static void mon_rmdir_one_subdir(struct kernfs_node *pkn, char *name, char *subn* when last domain being summed is removed. */ static void rmdir_mondata_subdir_allrdtgrp (struct rdt_resource *r, -struct rdt_mon_domain *d) +struct rdt_domain_hdr *hdr) {struct rdtgroup *prgrp, *crgrp; +struct rdt_mon_domain *d;char subname[32 ];bool snc_mode;char name[32 ]; +if (!domain_header_is_valid (hdr,RESCTRL_MON_DOMAIN ,RDT_RESOURCE_L3 )) +return ; + + d = container_of(hdr, struct rdt_mon_domain, hdr); snc_mode = r->mon_scope ==RESCTRL_L3_NODE ;sprintf (name,"mon_%s_%02d" , r->name, snc_mode ? d->ci_id : d->hdr.id);if (snc_mode)@@ -3184,19 +3189,18 @@ static void rmdir_mondata_subdir_allrdtgrp(struct rdt_resource *r, } } -static int mon_add_all_files (struct kernfs_node *kn,struct rdt_mon_domain *d, +static int mon_add_all_files (struct kernfs_node *kn,struct rdt_domain_hdr *hdr,struct rdt_resource *r,struct rdtgroup *prgrp, -bool do_sum) +int domid,bool do_sum) {struct rmid_read rr = {0 };struct mon_data *priv;struct mon_evt *mevt; -int ret, domid; +int ret;
Quoted 7 lines
fs/resctrl/rdtgroup.c static void rmdir_mondata_subdir_allrdtgrp(struct rdt_resource *r, · line 3199
Quoted 4 lines
fs/resctrl/rdtgroup.c +0
-1for_each_mon_event (mevt) {if (mevt->rid != r->rid || !mevt->enabled)continue ; - domid = do_sum ? d->ci_id : d->hdr.id;Looks like an unrelated change. Would this not be more appropriate for "fs/resctrl: Refactor Sub-NUMA Cluster (SNC) in mkdir/rmdir code flow"?
Agreed. I'll cut this out and move to the later patch.
Quoted 37 lines
Quoted 37 lines
fs/resctrl/rdtgroup.c +7
-2priv =mon_get_kn_priv (r->rid, domid, mevt, do_sum);if (WARN_ON_ONCE (!priv))return -EINVAL ;@@ -3206,23 +3210,28 @@ static int mon_add_all_files(struct kernfs_node *kn, struct rdt_mon_domain *d,return ret;if (!do_sum &&resctrl_is_mbm_event (mevt->evtid)) -mon_event_read (&rr, r, d, prgrp, &d->hdr.cpu_mask, mevt->evtid, true); +mon_event_read (&rr, r, hdr, prgrp, &hdr->cpu_mask, mevt->evtid, true); }return 0 ; }static int mkdir_mondata_subdir (struct kernfs_node *parent_kn, -struct rdt_mon_domain *d, +struct rdt_domain_hdr *hdr,struct rdt_resource *r,struct rdtgroup *prgrp) {struct kernfs_node *kn, *ckn; +struct rdt_mon_domain *d;char name[32 ];bool snc_mode;int ret =0 ;lockdep_assert_held (&rdtgroup_mutex); +if (!domain_header_is_valid (hdr,RESCTRL_MON_DOMAIN ,RDT_RESOURCE_L3 )) +return -EINVAL ; + + d = container_of(hdr, struct rdt_mon_domain, hdr); snc_mode = r->mon_scope ==RESCTRL_L3_NODE ;sprintf (name,"mon_%s_%02d" , r->name, snc_mode ? d->ci_id : d->hdr.id); kn =kernfs_find_and_get (parent_kn, name);@@ -3240,13 +3249,13 @@ static int mkdir_mondata_subdir(struct kernfs_node *parent_kn, ret =rdtgroup_kn_set_ugid (kn);
Quoted 8 lines
fs/resctrl/rdtgroup.c static int mkdir_mondata_subdir(struct kernfs_node *parent_kn, · line 3252
Quoted 4 lines
fs/resctrl/rdtgroup.c +1
-1if (ret) goto out_destroy; - ret =mon_add_all_files (kn, d, r, prgrp, snc_mode); + ret =mon_add_all_files (kn, hdr, r, prgrp, hdr->id, snc_mode);This does not seem right ... looks like this aims to do some of the SNC enabling but the domain id is always set to the domain of the node and does not distinguish between the L3 id and node id?
Also move to later patch (and get it right)
Quoted 87 lines
Quoted 75 lines
fs/resctrl/rdtgroup.c +11
-9if (ret) goto out_destroy; }if (snc_mode) { -sprintf (name,"mon_sub_%s_%02d" , r->name, d->hdr.id); +sprintf (name,"mon_sub_%s_%02d" , r->name, hdr->id); ckn =kernfs_create_dir (kn, name, parent_kn->mode, prgrp);if (IS_ERR (ckn)) { ret = -EINVAL ;@@ -3257,7 +3266,7 @@ static int mkdir_mondata_subdir(struct kernfs_node *parent_kn,if (ret) goto out_destroy; - ret =mon_add_all_files (ckn, d, r, prgrp, false); + ret =mon_add_all_files (ckn, hdr, r, prgrp, hdr->id, false);if (ret) goto out_destroy; }@@ -3275,7 +3284,7 @@ static int mkdir_mondata_subdir(struct kernfs_node *parent_kn,* and "monitor" groups with given domain id. */ static void mkdir_mondata_subdir_allrdtgrp (struct rdt_resource *r, -struct rdt_mon_domain *d) +struct rdt_domain_hdr *hdr) {struct kernfs_node *parent_kn;struct rdtgroup *prgrp, *crgrp;@@ -3283,12 +3292,12 @@ static void mkdir_mondata_subdir_allrdtgrp(struct rdt_resource *r,list_for_each_entry (prgrp, &rdt_all_groups, rdtgroup_list) { parent_kn = prgrp->mon.mon_data_kn; -mkdir_mondata_subdir (parent_kn, d, r, prgrp); +mkdir_mondata_subdir (parent_kn, hdr, r, prgrp); head = &prgrp->mon.crdtgrp_list;list_for_each_entry (crgrp, head, mon.crdtgrp_list) { parent_kn = crgrp->mon.mon_data_kn; -mkdir_mondata_subdir (parent_kn, d, r, crgrp); +mkdir_mondata_subdir (parent_kn, hdr, r, crgrp); } } }@@ -3297,14 +3306,14 @@ static int mkdir_mondata_subdir_alldom(struct kernfs_node *parent_kn,struct rdt_resource *r, struct rdtgroup *prgrp) { - struct rdt_mon_domain *dom; + struct rdt_domain_hdr *hdr;int ret;/* Walking r->domains, ensure it can't race with cpuhp */ lockdep_assert_cpus_held (); -list_for_each_entry (dom, &r->mon_domains, hdr.list) { - ret =mkdir_mondata_subdir (parent_kn, dom, r, prgrp); +list_for_each_entry (hdr, &r->mon_domains, list) { + ret =mkdir_mondata_subdir (parent_kn, hdr, r, prgrp);if (ret)return ret; }@@ -4187,8 +4196,10 @@ void resctrl_offline_ctrl_domain(struct rdt_resource *r, struct rdt_ctrl_domainmutex_unlock (&rdtgroup_mutex); } -void resctrl_offline_mon_domain (struct rdt_resource *r,struct rdt_mon_domain *d) +void resctrl_offline_mon_domain (struct rdt_resource *r,struct rdt_domain_hdr *hdr) { +struct rdt_mon_domain *d; +mutex_lock (&rdtgroup_mutex);/* fs/resctrl/rdtgroup.c void resctrl_offline_mon_domain(struct rdt_resource *r, struct rdt_mon_domain *d · line 4214
Quoted 9 lines
fs/resctrl/rdtgroup.c +4
-1@@ -4196,8 +4207,12 @@ void resctrl_offline_mon_domain(struct rdt_resource *r, struct rdt_mon_domain *d* per domain monitor data directories. */ if (resctrl_mounted &&resctrl_arch_mon_capable ()) -rmdir_mondata_subdir_allrdtgrp (r, d); +rmdir_mondata_subdir_allrdtgrp (r, hdr); +if (!domain_header_is_valid (hdr,RESCTRL_MON_DOMAIN ,RDT_RESOURCE_L3 )) + goto out_unlock; +One logical change per patch please.
Quoted 8 lines
While all other L3 specific functions modified to receive hdr as parameter are changed to use
container_of()at beginning of function to highlight that the functions are L3 specific ...resctrl_offline_mon_domain()is changed differently. Looks like this changes the flow to sneak in somePERF_PKGenabling for convenience and thus makes this patch harder to understand. Splittingresctrl_offline_mon_domain()to handle different domain types seems more appropriate for "x86/resctrl: Handle domain creation/deletion forRDT_RESOURCE_PERF_PKG" where it should be clear what changes are made to supportPERF_PKG. In this patch, in this stage of series, the entire function can be L3 specific.
Will move to later patch and make the offline/online patches have same style.
Quoted 38 lines
Quoted 38 lines
fs/resctrl/rdtgroup.c +9
-3+ d = container_of(hdr, struct rdt_mon_domain, hdr);if (resctrl_is_mbm_enabled ())cancel_delayed_work (&d->mbm_over);if (resctrl_is_mon_event_enabled (QOS_L3_OCCUP_EVENT_ID ) &&has_busy_rmid (d)) {@@ -4214,7 +4229,7 @@ void resctrl_offline_mon_domain(struct rdt_resource *r, struct rdt_mon_domain *d }domain_destroy_mon_state (d); - +out_unlock:mutex_unlock (&rdtgroup_mutex); }@@ -4287,12 +4302,17 @@ int resctrl_online_ctrl_domain(struct rdt_resource *r, struct rdt_ctrl_domain *dreturn err; } -int resctrl_online_mon_domain (struct rdt_resource *r,struct rdt_mon_domain *d) +int resctrl_online_mon_domain (struct rdt_resource *r,struct rdt_domain_hdr *hdr) { -int err; +struct rdt_mon_domain *d; +int err = -EINVAL ;mutex_lock (&rdtgroup_mutex); +if (!domain_header_is_valid (hdr,RESCTRL_MON_DOMAIN ,RDT_RESOURCE_L3 )) + goto out_unlock; + + d = container_of(hdr, struct rdt_mon_domain, hdr); err =domain_setup_mon_state (r, d);if (err) goto out_unlock;@@ -4306,6 +4326,7 @@ int resctrl_online_mon_domain(struct rdt_resource *r, struct rdt_mon_domain *d)if (resctrl_is_mon_event_enabled (QOS_L3_OCCUP_EVENT_ID ))INIT_DELAYED_WORK (&d->cqm_limbo, cqm_handle_limbo);
Quoted 3 lines
fs/resctrl/rdtgroup.c int resctrl_online_mon_domain(struct rdt_resource *r, struct rdt_mon_domain *d) · line 4329
Quoted 1 line
fs/resctrl/rdtgroup.c +1
-0+ err =0 ;Considering the earlier exit on "if (err)", err can be expected to be 0 here?
Yes. Dropped this superfluous assignment.
Quoted 11 lines
Quoted 11 lines
fs/resctrl/rdtgroup.c +1
-1/* * If the filesystem is not mounted then only the default resource group * exists. Creation of its directories is deferred until mount time @@ -4313,7 +4334,7 @@ int resctrl_online_mon_domain(struct rdt_resource *r, struct rdt_mon_domain *d)* If resctrl is mounted, add per domain monitor data directories. */ if (resctrl_mounted &&resctrl_arch_mon_capable ()) -mkdir_mondata_subdir_allrdtgrp (r, d); +mkdir_mondata_subdir_allrdtgrp (r, hdr);
Quoted 4 lines
fs/resctrl/rdtgroup.c int resctrl_online_mon_domain(struct rdt_resource *r, struct rdt_mon_domain *d) · line 4340
Quoted 2 lines
fs/resctrl/rdtgroup.c +0
-0out_unlock:mutex_unlock (&rdtgroup_mutex);Reinette
-Tony
Reinette Chatre
maintainerRe: [PATCH v11 05/31] x86,fs/resctrl: Refactor domain create/remove using struct rdt_domain_hdr
Hi Tony,
Quoted 10 lines · Tony
Quoted 9 lines · Reinette Chatre
Hi Tony,
Quoted 5 lines · Tony Luck
Up until now, all monitoring events were associated with the L3 resource and it made sense to use the L3 specific "struct
rdt_mon_domain*" arguments to functions manipulating domains.To simplify enabling of enumeration of domains for events in other
Quoted 7 lines
Quoted 1 line
What does "enabling of enumeration of domains" mean?
Is this better?
To prepare for events in resources other than L3, change the calling convention to pass the generic struct
rdt_domain_hdrand use that to find the domain specific structure where needed.
I interpret above as a solution that is unrelated to the problem because the problem is stated as "prepare for events in *resources*" while the solution changes how *domain* structures are accessed.
Here is an attempt to make the problem and solution clear, please feel free to change:
Up until now, all monitoring events were associated with the L3 resource and it made sense to use the L3 specific "struct rdt_mon_domain *" argument to functions operating on domains.
Telemetry events will be tied to a new resource with its instances represented by a new domain structure that, just like struct rdt_mon_domain,
starts with the generic struct rdt_domain_hdr.
Prepare to support domains belonging to different resources by changing the calling convention of functions operating on domains. Pass the generic header and use that to find the domain specific structure where needed.
sidenote: I changed "manipulating" to "operating on" even though Boris wrote it since a few of the functions changed in this patch do not manipulate the domains but instead use them as reference.
Reinette
Tony Luck
maintainerPATCH v11 6/31 x86,fs/resctrl: Use struct rdt_domain_hdr when reading counters
In reply to Tony LuckUse a generic struct rdt_domain_hdr representing a generic domain header in struct rmid_read in order to support other telemetry events' domains besides an L3 one. Adjust the code interacting with it to the new struct layout.
--- include/linux/resctrl.h | 8 +++---- fs/resctrl/internal.h | 18 +++++++------- arch/x86/kernel/cpu/resctrl/monitor.c | 17 +++++++++++--- fs/resctrl/ctrlmondata.c | 7 +----- fs/resctrl/monitor.c | 34 +++++++++++++++++---------- 5 files changed, 50 insertions(+), 34 deletions(-)
include/linux/resctrl.h +4 -4
@@ -514,7 +514,7 @@ void resctrl_offline_cpu(unsigned int cpu);* resctrl_arch_rmid_read() - Read the eventid counter corresponding to rmid * for this resource and domain. * @r: resource that the counter should be read from. -* @d: domain that the counter should be read from. +* @hdr: Header of domain that the counter should be read from. * @closid: closid that matches the rmid. Depending on the architecture, the * counter may match traffic of both @closid and @rmid, or @rmid * only.
@@ -535,7 +535,7 @@ void resctrl_offline_cpu(unsigned int cpu);* Return: * 0 on success, or -EIO, -EINVAL etc on error. */ -int resctrl_arch_rmid_read (struct rdt_resource *r,struct rdt_mon_domain *d, +int resctrl_arch_rmid_read (struct rdt_resource *r,struct rdt_domain_hdr *hdr,u32 closid,u32 rmid,enum resctrl_event_id eventid,u64 *val,void *arch_mon_ctx);
@@ -630,7 +630,7 @@ void resctrl_arch_config_cntr(struct rdt_resource *r, struct rdt_mon_domain *d,* assigned to the RMID, event pair for this resource * and domain. * @r: Resource that the counter should be read from. -* @d: Domain that the counter should be read from. +* @hdr: Header of domain that the counter should be read from. * @closid: CLOSID that matches the RMID. * @rmid: The RMID to which @cntr_id is assigned. * @cntr_id: The counter to read.
@@ -644,7 +644,7 @@ void resctrl_arch_config_cntr(struct rdt_resource *r, struct rdt_mon_domain *d,* Return: * 0 on success, or -EIO, -EINVAL etc on error. */ -int resctrl_arch_cntr_read (struct rdt_resource *r,struct rdt_mon_domain *d, +int resctrl_arch_cntr_read (struct rdt_resource *r,struct rdt_domain_hdr *hdr,u32 closid,u32 rmid,int cntr_id,enum resctrl_event_id eventid,u64 *val);
fs/resctrl/internal.h +10 -8
@@ -106,24 +106,26 @@ struct mon_data {* resource group then its event count is summed with the count from all * its child resource groups. * @r: Resource describing the properties of the event being read. -* @d: Domain that the counter should be read from. If NULL then sum all -* domains in @r sharing L3 @ci.id +* @hdr: Header of domain that the counter should be read from. If NULL then +* sum all domains in @r sharing L3 @ci.id * @evtid: Which monitor event to read. * @first: Initialize MBM counter when true. -* @ci: Cacheinfo for L3. Only set when @d is NULL. Used when summing domains. +* @ci: Cacheinfo for L3. Only set when @hdr is NULL. Used when summing +* domains. * @is_mbm_cntr: true if "mbm_event" counter assignment mode is enabled and it * is an MBM event. * @err: Error encountered when reading counter. -* @val: Returned value of event counter. If @rgrp is a parent resource group, -* @val includes the sum of event counts from its child resource groups. -* If @d is NULL, @val includes the sum of all domains in @r sharing @ci.id, -* (summed across child resource groups if @rgrp is a parent resource group). +* @val: Returned value of event counter. If @rgrp is a parent resource +* group, @val includes the sum of event counts from its child +* resource groups. If @hdr is NULL, @val includes the sum of all +* domains in @r sharing @ci.id, (summed across child resource groups +* if @rgrp is a parent resource group). * @arch_mon_ctx: Hardware monitor allocated for this read request (MPAM only). */ struct rmid_read {struct rdtgroup *rgrp;struct rdt_resource *r; -struct rdt_mon_domain *d; +struct rdt_domain_hdr *hdr;enum resctrl_event_id evtid;bool first;struct cacheinfo *ci;
arch/x86/kernel/cpu/resctrl/monitor.c +14 -3
@@ -238,17 +238,23 @@ static u64 get_corrected_val(struct rdt_resource *r, struct rdt_mon_domain *d,return chunks * hw_res->mon_scale; } -int resctrl_arch_rmid_read (struct rdt_resource *r,struct rdt_mon_domain *d, +int resctrl_arch_rmid_read (struct rdt_resource *r,struct rdt_domain_hdr *hdr,u32 unused,u32 rmid,enum resctrl_event_id eventid,u64 *val,void *ignored) { -int cpu =cpumask_any (&d->hdr.cpu_mask); +struct rdt_mon_domain *d;u64 msr_val;u32 prmid; +int cpu;int ret;resctrl_arch_rmid_read_context_check (); +if (!domain_header_is_valid (hdr,RESCTRL_MON_DOMAIN ,RDT_RESOURCE_L3 )) +return -EINVAL ; + + d = container_of(hdr, struct rdt_mon_domain, hdr); + cpu =cpumask_any (&hdr->cpu_mask); prmid =logical_rmid_to_physical_rmid (cpu, rmid); ret =__rmid_read_phys (prmid, eventid, &msr_val);if (ret)
@@ -312,13 +318,18 @@ void resctrl_arch_reset_cntr(struct rdt_resource *r, struct rdt_mon_domain *d, } } -int resctrl_arch_cntr_read (struct rdt_resource *r,struct rdt_mon_domain *d, +int resctrl_arch_cntr_read (struct rdt_resource *r,struct rdt_domain_hdr *hdr,u32 unused,u32 rmid,int cntr_id,enum resctrl_event_id eventid,u64 *val) { +struct rdt_mon_domain *d;u64 msr_val;int ret; +if (!domain_header_is_valid (hdr,RESCTRL_MON_DOMAIN ,RDT_RESOURCE_L3 )) +return -EINVAL ; + + d = container_of(hdr, struct rdt_mon_domain, hdr); ret =__cntr_id_read (cntr_id, &msr_val);if (ret)return ret;
fs/resctrl/ctrlmondata.c +1 -6
@@ -550,13 +550,8 @@ void mon_event_read(struct rmid_read *rr, struct rdt_resource *r, struct rdt_domain_hdr *hdr, struct rdtgroup *rdtgrp, cpumask_t *cpumask, int evtid, int first) { - struct rdt_mon_domain *d; int cpu; -if (!domain_header_is_valid (hdr,RESCTRL_MON_DOMAIN ,RDT_RESOURCE_L3 )) -return ; - d = container_of(hdr, struct rdt_mon_domain, hdr); -/* When picking a CPU from cpu_mask, ensure it can't race with cpuhp */ lockdep_assert_cpus_held ();
@@ -566,7 +561,7 @@ void mon_event_read(struct rmid_read *rr, struct rdt_resource *r, rr->rgrp = rdtgrp; rr->evtid = evtid; rr->r = r; - rr->d = d; + rr->hdr = hdr; rr->first = first;if (resctrl_arch_mbm_cntr_assign_enabled (r) &&resctrl_is_mbm_event (evtid)) {
fs/resctrl/monitor.c +21 -13
@@ -159,7 +159,7 @@ void __check_limbo(struct rdt_mon_domain *d, bool force_free)break ; entry =__rmid_entry (idx); -if (resctrl_arch_rmid_read (r, d, entry->closid, entry->rmid, +if (resctrl_arch_rmid_read (r, &d->hdr, entry->closid, entry->rmid,QOS_L3_OCCUP_EVENT_ID , &val, arch_mon_ctx)) { rmid_dirty = true;
@@ -424,8 +424,12 @@ static int __mon_event_count(struct rdtgroup *rdtgrp, struct rmid_read *rr)int err, ret;u64 tval =0 ; +if (!domain_header_is_valid (rr->hdr,RESCTRL_MON_DOMAIN ,RDT_RESOURCE_L3 )) +return -EINVAL ; + d = container_of(rr->hdr, struct rdt_mon_domain, hdr); +if (rr->is_mbm_cntr) { - cntr_id =mbm_cntr_get (rr->r, rr->d, rdtgrp, rr->evtid); + cntr_id =mbm_cntr_get (rr->r, d, rdtgrp, rr->evtid);if (cntr_id <0 ) { rr->err = -ENOENT ;return -EINVAL ;
@@ -434,24 +438,24 @@ static int __mon_event_count(struct rdtgroup *rdtgrp, struct rmid_read *rr)if (rr->first) {if (rr->is_mbm_cntr) -resctrl_arch_reset_cntr (rr->r, rr->d, closid, rmid, cntr_id, rr->evtid); +resctrl_arch_reset_cntr (rr->r, d, closid, rmid, cntr_id, rr->evtid);else -resctrl_arch_reset_rmid (rr->r, rr->d, closid, rmid, rr->evtid); - m =get_mbm_state (rr->d, closid, rmid, rr->evtid); +resctrl_arch_reset_rmid (rr->r, d, closid, rmid, rr->evtid); + m =get_mbm_state (d, closid, rmid, rr->evtid);if (m)memset (m,0 ,sizeof (struct mbm_state ));return 0 ; } -if (rr->d) { +if (rr->hdr) {/* Reading a single domain, must be on a CPU in that domain. */ -if (!cpumask_test_cpu (cpu, &rr->d->hdr.cpu_mask)) +if (!cpumask_test_cpu (cpu, &rr->hdr->cpu_mask))return -EINVAL ;if (rr->is_mbm_cntr) - rr->err =resctrl_arch_cntr_read (rr->r, rr->d, closid, rmid, cntr_id, + rr->err =resctrl_arch_cntr_read (rr->r, rr->hdr, closid, rmid, cntr_id, rr->evtid, &tval);else - rr->err =resctrl_arch_rmid_read (rr->r, rr->d, closid, rmid, + rr->err =resctrl_arch_rmid_read (rr->r, rr->hdr, closid, rmid, rr->evtid, &tval, rr->arch_mon_ctx);if (rr->err)return rr->err;
@@ -477,10 +481,10 @@ static int __mon_event_count(struct rdtgroup *rdtgrp, struct rmid_read *rr)if (d->ci_id != rr->ci->id)continue ;if (rr->is_mbm_cntr) - err =resctrl_arch_cntr_read (rr->r, d, closid, rmid, cntr_id, + err =resctrl_arch_cntr_read (rr->r, &d->hdr, closid, rmid, cntr_id, rr->evtid, &tval);else - err =resctrl_arch_rmid_read (rr->r, d, closid, rmid, + err =resctrl_arch_rmid_read (rr->r, &d->hdr, closid, rmid, rr->evtid, &tval, rr->arch_mon_ctx);if (!err) { rr->val += tval;
@@ -511,9 +515,13 @@ static void mbm_bw_count(struct rdtgroup *rdtgrp, struct rmid_read *rr)u64 cur_bw, bytes, cur_bytes;u32 closid = rdtgrp->closid;u32 rmid = rdtgrp->mon.rmid; +struct rdt_mon_domain *d;struct mbm_state *m; - m =get_mbm_state (rr->d, closid, rmid, rr->evtid); +if (!domain_header_is_valid (rr->hdr,RESCTRL_MON_DOMAIN ,RDT_RESOURCE_L3 )) +return ; + d = container_of(rr->hdr, struct rdt_mon_domain, hdr); + m =get_mbm_state (d, closid, rmid, rr->evtid);if (WARN_ON_ONCE (!m))return ;
@@ -686,7 +694,7 @@ static void mbm_update_one_event(struct rdt_resource *r, struct rdt_mon_domain *struct rmid_read rr = {0 }; rr.r = r; - rr.d = d; + rr.hdr = &d->hdr; rr.evtid = evtid;if (resctrl_arch_mbm_cntr_assign_enabled (r)) { rr.is_mbm_cntr = true;
Signature · 1 line
2.51.0
Reinette Chatre
maintainerRe: [PATCH v11 06/31] x86,fs/resctrl: Use struct rdt_domain_hdr when reading counters
Hi Tony,
Quoted 4 lines · Tony Luck
Use a generic struct
rdt_domain_hdrrepresenting a generic domain header in structrmid_readin order to support other telemetry events' domains besides an L3 one. Adjust the code interacting with it to the new struct layout.
I'd propose a small amend to be more specific and not assume reader knows what rmid_read is used for:
struct rmid_read contains data passed around to read event counts. Use the generic domain header struct rdt_domain_hdr in struct rmid_read in order to support other telemetry events' domains besides an L3 one. Adjust the code interacting with it to the new struct layout.
Quoted 5 lines
fs/resctrl/ctrlmondata.c +0
-0
fs/resctrl/ctrlmondata.c void mon_event_read(struct rmid_read *rr, struct rdt_resource *r, · line 559
Quoted 10 lines
fs/resctrl/ctrlmondata.c +0
-5@@ -550,13 +550,8 @@ void mon_event_read(struct rmid_read *rr, struct rdt_resource *r, struct rdt_domain_hdr *hdr, struct rdtgroup *rdtgrp, cpumask_t *cpumask, int evtid, int first) { - struct rdt_mon_domain *d; int cpu; -if (!domain_header_is_valid (hdr,RESCTRL_MON_DOMAIN ,RDT_RESOURCE_L3 )) -return ; - d = container_of(hdr, struct rdt_mon_domain, hdr); -
Problematic snippet removed here ...
Quoted 26 lines
fs/resctrl/ctrlmondata.c +1
-1/* When picking a CPU from cpu_mask, ensure it can't race with cpuhp */ lockdep_assert_cpus_held ();@@ -566,7 +561,7 @@ void mon_event_read(struct rmid_read *rr, struct rdt_resource *r, rr->rgrp = rdtgrp; rr->evtid = evtid; rr->r = r; - rr->d = d; + rr->hdr = hdr; rr->first = first;if (resctrl_arch_mbm_cntr_assign_enabled (r) &&resctrl_is_mbm_event (evtid)) {fs/resctrl/monitor.c +1
-1@@ -159,7 +159,7 @@ void __check_limbo(struct rdt_mon_domain *d, bool force_free)break ; entry =__rmid_entry (idx); -if (resctrl_arch_rmid_read (r, d, entry->closid, entry->rmid, +if (resctrl_arch_rmid_read (r, &d->hdr, entry->closid, entry->rmid,QOS_L3_OCCUP_EVENT_ID , &val, arch_mon_ctx)) { rmid_dirty = true;
fs/resctrl/monitor.c static int __mon_event_count(struct rdtgroup *rdtgrp, struct rmid_read *rr) · line 430
Quoted 7 lines
fs/resctrl/monitor.c +4
-0@@ -424,8 +424,12 @@ static int __mon_event_count(struct rdtgroup *rdtgrp, struct rmid_read *rr)int err, ret;u64 tval =0 ; +if (!domain_header_is_valid (rr->hdr,RESCTRL_MON_DOMAIN ,RDT_RESOURCE_L3 )) +return -EINVAL ; + d = container_of(rr->hdr, struct rdt_mon_domain, hdr); +
... but now the problem is moved to __mon_event_count() where rr->hdr can be NULL and the domain_header_is_valid() check is referencing NULL pointer when SNC is enabled? Am I missing something here? Does this work on SNC?
Reinette
Luck, Tony
maintainerRe: [PATCH v11 06/31] x86,fs/resctrl: Use struct rdt_domain_hdr when reading counters
Quoted 8 lines · Reinette Chatre
Hi Tony,
Quoted 4 lines · Tony Luck
Use a generic struct
rdt_domain_hdrrepresenting a generic domain header in structrmid_readin order to support other telemetry events' domains besides an L3 one. Adjust the code interacting with it to the new struct layout.
Quoted 7 lines
I'd propose a small amend to be more specific and not assume reader knows what
rmid_readis used for:struct
rmid_readcontains data passed around to read event counts. Use the generic domain header structrdt_domain_hdrin structrmid_readin order to support other telemetry events' domains besides an L3 one. Adjust the code interacting with it to the new struct layout.
Looks good. Thanks.
Quoted 13 lines
Quoted 13 lines
fs/resctrl/ctrlmondata.c +0
-1@@ -550,13 +550,8 @@ void mon_event_read(struct rmid_read *rr, struct rdt_resource *r, struct rdt_domain_hdr *hdr, struct rdtgroup *rdtgrp, cpumask_t *cpumask, int evtid, int first) { - struct rdt_mon_domain *d; int cpu;
Quoted 7 lines
fs/resctrl/ctrlmondata.c void mon_event_read(struct rmid_read *rr, struct rdt_resource *r, · line 559
Quoted 4 lines
fs/resctrl/ctrlmondata.c +0
-4-if (!domain_header_is_valid (hdr,RESCTRL_MON_DOMAIN ,RDT_RESOURCE_L3 )) -return ; - d = container_of(hdr, struct rdt_mon_domain, hdr); -Problematic snippet removed here ...
Yup.
Quoted 29 lines
Quoted 29 lines
fs/resctrl/ctrlmondata.c +1
-1/* When picking a CPU from cpu_mask, ensure it can't race with cpuhp */ lockdep_assert_cpus_held ();@@ -566,7 +561,7 @@ void mon_event_read(struct rmid_read *rr, struct rdt_resource *r, rr->rgrp = rdtgrp; rr->evtid = evtid; rr->r = r; - rr->d = d; + rr->hdr = hdr; rr->first = first;if (resctrl_arch_mbm_cntr_assign_enabled (r) &&resctrl_is_mbm_event (evtid)) {fs/resctrl/monitor.c +1
-1@@ -159,7 +159,7 @@ void __check_limbo(struct rdt_mon_domain *d, bool force_free)break ; entry =__rmid_entry (idx); -if (resctrl_arch_rmid_read (r, d, entry->closid, entry->rmid, +if (resctrl_arch_rmid_read (r, &d->hdr, entry->closid, entry->rmid,QOS_L3_OCCUP_EVENT_ID , &val, arch_mon_ctx)) { rmid_dirty = true;@@ -424,8 +424,12 @@ static int __mon_event_count(struct rdtgroup *rdtgrp, struct rmid_read *rr)int err, ret;u64 tval =0 ;
Quoted 8 lines
fs/resctrl/monitor.c static int __mon_event_count(struct rdtgroup *rdtgrp, struct rmid_read *rr) · line 430
Quoted 4 lines
fs/resctrl/monitor.c +4
-0+if (!domain_header_is_valid (rr->hdr,RESCTRL_MON_DOMAIN ,RDT_RESOURCE_L3 )) +return -EINVAL ; + d = container_of(rr->hdr, struct rdt_mon_domain, hdr); +... but now the problem is moved to
__mon_event_count()whererr->hdrcan be NULL and thedomain_header_is_valid()check is referencing NULL pointer when SNC is enabled? Am I missing something here? Does this work on SNC?
You are right. This likely breaks SNC. I'll add a check for "!hdr" and move this inside the "if (rr->is_mbm_cntr)" with a duplicate inside the "if (rr->first)". This duplication will be cleaned up with a later patch to refactor __mon_event_count().
Quoted 3 lines
Reinette
-Tony
Quoted 1 line
Tony Luck
maintainerPATCH v11 7/31 x86,fs/resctrl: Rename struct rdt_mon_domain and rdt_hw_mon_domain
In reply to Tony LuckThe upcoming telemetry event monitoring are not tied to the L3 resource and will have a new domain structures.
Rename the L3 resource specific domain data structures to include "l3_" in their names to avoid confusion between the different resource specific domain structures:rdt_mon_domain -> rdt_l3_mon_domainrdt_hw_mon_domain -> rdt_hw_l3_mon_domain
No functional change.
--- include/linux/resctrl.h | 20 ++++---- arch/x86/kernel/cpu/resctrl/internal.h | 16 +++--- fs/resctrl/internal.h | 8 +-- arch/x86/kernel/cpu/resctrl/core.c | 14 +++--- arch/x86/kernel/cpu/resctrl/monitor.c | 36 +++++++------- fs/resctrl/ctrlmondata.c | 2 +- fs/resctrl/monitor.c | 68 +++++++++++++------------- fs/resctrl/rdtgroup.c | 36 +++++++------- 8 files changed, 100 insertions(+), 100 deletions(-)
include/linux/resctrl.h +10 -10
@@ -178,7 +178,7 @@ struct mbm_cntr_cfg { };/** -* struct rdt_mon_domain - group of CPUs sharing a resctrl monitor resource +* struct rdt_l3_mon_domain - group of CPUs sharing a resctrl monitor resource * @hdr: common header for different domain types * @ci_id: cache info id for this domain * @rmid_busy_llc: bitmap of which limbo RMIDs are above threshold
@@ -192,7 +192,7 @@ struct mbm_cntr_cfg {* @cntr_cfg: array of assignable counters' configuration (indexed * by counter ID) */ -struct rdt_mon_domain { +struct rdt_l3_mon_domain {struct rdt_domain_hdr hdr;unsigned int ci_id;unsigned long *rmid_busy_llc;
@@ -364,10 +364,10 @@ struct resctrl_cpu_defaults { };struct resctrl_mon_config_info { -struct rdt_resource *r; -struct rdt_mon_domain *d; -u32 evtid; -u32 mon_config; +struct rdt_resource *r; +struct rdt_l3_mon_domain *d; +u32 evtid; +u32 mon_config; };/**
@@ -582,7 +582,7 @@ struct rdt_domain_hdr *resctrl_find_domain(struct list_head *h, int id,* * This can be called from any CPU. */ -void resctrl_arch_reset_rmid (struct rdt_resource *r,struct rdt_mon_domain *d, +void resctrl_arch_reset_rmid (struct rdt_resource *r,struct rdt_l3_mon_domain *d,u32 closid,u32 rmid,enum resctrl_event_id eventid);
@@ -595,7 +595,7 @@ void resctrl_arch_reset_rmid(struct rdt_resource *r, struct rdt_mon_domain *d,* * This can be called from any CPU. */ -void resctrl_arch_reset_rmid_all (struct rdt_resource *r,struct rdt_mon_domain *d); +void resctrl_arch_reset_rmid_all (struct rdt_resource *r,struct rdt_l3_mon_domain *d);/** * resctrl_arch_reset_all_ctrls() - Reset the control for each CLOSID to its
@@ -621,7 +621,7 @@ void resctrl_arch_reset_all_ctrls(struct rdt_resource *r);* * This can be called from any CPU. */ -void resctrl_arch_config_cntr (struct rdt_resource *r,struct rdt_mon_domain *d, +void resctrl_arch_config_cntr (struct rdt_resource *r,struct rdt_l3_mon_domain *d,enum resctrl_event_id evtid,u32 rmid,u32 closid,u32 cntr_id,bool assign);
@@ -659,7 +659,7 @@ int resctrl_arch_cntr_read(struct rdt_resource *r, struct rdt_domain_hdr *hdr,* * This can be called from any CPU. */ -void resctrl_arch_reset_cntr (struct rdt_resource *r,struct rdt_mon_domain *d, +void resctrl_arch_reset_cntr (struct rdt_resource *r,struct rdt_l3_mon_domain *d,u32 closid,u32 rmid,int cntr_id,enum resctrl_event_id eventid);
arch/x86/kernel/cpu/resctrl/internal.h +8 -8
@@ -60,17 +60,17 @@ struct rdt_hw_ctrl_domain { };/** -* struct rdt_hw_mon_domain - Arch private attributes of a set of CPUs that share -* a resource for a monitor function -* @d_resctrl: Properties exposed to the resctrl file system +* struct rdt_hw_l3_mon_domain - Arch private attributes of a set of CPUs that share +* a resource for a monitor function +* @d_resctrl: Properties exposed to the resctrl file system * @arch_mbm_states: Per-event pointer to the MBM event's saved state. * An MBM event's state is an array of struct arch_mbm_state * indexed by RMID on x86. * * Members of this structure are accessed via helpers that provide abstraction. */ -struct rdt_hw_mon_domain { -struct rdt_mon_domain d_resctrl; +struct rdt_hw_l3_mon_domain { +struct rdt_l3_mon_domain d_resctrl;struct arch_mbm_state *arch_mbm_states[QOS_NUM_L3_MBM_EVENTS ]; };
@@ -79,9 +79,9 @@ static inline struct rdt_hw_ctrl_domain *resctrl_to_arch_ctrl_dom(struct rdt_ctr return container_of(r, struct rdt_hw_ctrl_domain, d_resctrl); } -static inline struct rdt_hw_mon_domain *resctrl_to_arch_mon_dom (struct rdt_mon_domain *r) +static inline struct rdt_hw_l3_mon_domain *resctrl_to_arch_mon_dom (struct rdt_l3_mon_domain *r) { - return container_of(r, struct rdt_hw_mon_domain, d_resctrl); + return container_of(r, struct rdt_hw_l3_mon_domain, d_resctrl); }/**
@@ -135,7 +135,7 @@ static inline struct rdt_hw_resource *resctrl_to_arch_res(struct rdt_resource *rextern struct rdt_hw_resource rdt_resources_all[]; -void arch_mon_domain_online (struct rdt_resource *r,struct rdt_mon_domain *d); +void arch_mon_domain_online (struct rdt_resource *r,struct rdt_l3_mon_domain *d);/* CPUID.(EAX=10H, ECX=ResID=1).EAX */ union cpuid_0x10_1_eax {
fs/resctrl/internal.h +4 -4
@@ -369,7 +369,7 @@ void mon_event_read(struct rmid_read *rr, struct rdt_resource *r,int resctrl_mon_resource_init (void ); -void mbm_setup_overflow_handler (struct rdt_mon_domain *dom, +void mbm_setup_overflow_handler (struct rdt_l3_mon_domain *dom,unsigned long delay_ms,int exclude_cpu);
@@ -377,14 +377,14 @@ void mbm_handle_overflow(struct work_struct *work);bool is_mba_sc (struct rdt_resource *r); -void cqm_setup_limbo_handler (struct rdt_mon_domain *dom,unsigned long delay_ms, +void cqm_setup_limbo_handler (struct rdt_l3_mon_domain *dom,unsigned long delay_ms,int exclude_cpu);void cqm_handle_limbo (struct work_struct *work); -bool has_busy_rmid (struct rdt_mon_domain *d); +bool has_busy_rmid (struct rdt_l3_mon_domain *d); -void __check_limbo (struct rdt_mon_domain *d,bool force_free); +void __check_limbo (struct rdt_l3_mon_domain *d,bool force_free);void resctrl_file_fflags_init (const char *config,unsigned long fflags);
arch/x86/kernel/cpu/resctrl/core.c +7 -7
@@ -363,7 +363,7 @@ static void ctrl_domain_free(struct rdt_hw_ctrl_domain *hw_dom)kfree (hw_dom); } -static void mon_domain_free (struct rdt_hw_mon_domain *hw_dom) +static void mon_domain_free (struct rdt_hw_l3_mon_domain *hw_dom) {int idx;
@@ -400,7 +400,7 @@ static int domain_setup_ctrlval(struct rdt_resource *r, struct rdt_ctrl_domain ** @num_rmid: The size of the MBM counter array * @hw_dom: The domain that owns the allocated arrays */ -static int arch_domain_mbm_alloc (u32 num_rmid,struct rdt_hw_mon_domain *hw_dom) +static int arch_domain_mbm_alloc (u32 num_rmid,struct rdt_hw_l3_mon_domain *hw_dom) {size_t tsize =sizeof (*hw_dom->arch_mbm_states[0 ]);enum resctrl_event_id eventid;
@@ -498,8 +498,8 @@ static void domain_add_cpu_ctrl(int cpu, struct rdt_resource *r)static void l3_mon_domain_setup (int cpu,int id,struct rdt_resource *r,struct list_head *add_pos) { -struct rdt_hw_mon_domain *hw_dom; -struct rdt_mon_domain *d; +struct rdt_hw_l3_mon_domain *hw_dom; +struct rdt_l3_mon_domain *d;struct cacheinfo *ci;int err;
@@ -625,9 +625,9 @@ static void domain_remove_cpu_ctrl(int cpu, struct rdt_resource *r)static void domain_remove_cpu_mon (int cpu,struct rdt_resource *r) {int id =get_domain_id_from_scope (cpu, r->mon_scope); -struct rdt_hw_mon_domain *hw_dom; +struct rdt_hw_l3_mon_domain *hw_dom; +struct rdt_l3_mon_domain *d;struct rdt_domain_hdr *hdr; -struct rdt_mon_domain *d;lockdep_assert_held (&domain_list_lock);
@@ -653,7 +653,7 @@ static void domain_remove_cpu_mon(int cpu, struct rdt_resource *r)if (!domain_header_is_valid (hdr,RESCTRL_MON_DOMAIN ,RDT_RESOURCE_L3 ))return ; - d = container_of(hdr, struct rdt_mon_domain, hdr); + d = container_of(hdr, struct rdt_l3_mon_domain, hdr); hw_dom =resctrl_to_arch_mon_dom (d);resctrl_offline_mon_domain (r, hdr);list_del_rcu (&hdr->list);
arch/x86/kernel/cpu/resctrl/monitor.c +18 -18
@@ -109,7 +109,7 @@ static inline u64 get_corrected_mbm_count(u32 rmid, unsigned long val)* * In RMID sharing mode there are fewer "logical RMID" values available * to accumulate data ("physical RMIDs" are divided evenly between SNC -* nodes that share an L3 cache). Linux creates an rdt_mon_domain for +* nodes that share an L3 cache). Linux creates an rdt_l3_mon_domain for * each SNC node. * * The value loaded into IA32_PQR_ASSOC is the "logical RMID".
@@ -157,7 +157,7 @@ static int __rmid_read_phys(u32 prmid, enum resctrl_event_id eventid, u64 *val)return 0 ; } -static struct arch_mbm_state *get_arch_mbm_state (struct rdt_hw_mon_domain *hw_dom, +static struct arch_mbm_state *get_arch_mbm_state (struct rdt_hw_l3_mon_domain *hw_dom,u32 rmid,enum resctrl_event_id eventid) {
@@ -171,11 +171,11 @@ static struct arch_mbm_state *get_arch_mbm_state(struct rdt_hw_mon_domain *hw_doreturn state ? &state[rmid] :NULL ; } -void resctrl_arch_reset_rmid (struct rdt_resource *r,struct rdt_mon_domain *d, +void resctrl_arch_reset_rmid (struct rdt_resource *r,struct rdt_l3_mon_domain *d,u32 unused,u32 rmid,enum resctrl_event_id eventid) { -struct rdt_hw_mon_domain *hw_dom =resctrl_to_arch_mon_dom (d); +struct rdt_hw_l3_mon_domain *hw_dom =resctrl_to_arch_mon_dom (d);int cpu =cpumask_any (&d->hdr.cpu_mask);struct arch_mbm_state *am;u32 prmid;
@@ -194,9 +194,9 @@ void resctrl_arch_reset_rmid(struct rdt_resource *r, struct rdt_mon_domain *d,* Assumes that hardware counters are also reset and thus that there is * no need to record initial non-zero counts. */ -void resctrl_arch_reset_rmid_all (struct rdt_resource *r,struct rdt_mon_domain *d) +void resctrl_arch_reset_rmid_all (struct rdt_resource *r,struct rdt_l3_mon_domain *d) { -struct rdt_hw_mon_domain *hw_dom =resctrl_to_arch_mon_dom (d); +struct rdt_hw_l3_mon_domain *hw_dom =resctrl_to_arch_mon_dom (d);enum resctrl_event_id eventid;int idx;
@@ -217,10 +217,10 @@ static u64 mbm_overflow_count(u64 prev_msr, u64 cur_msr, unsigned int width)return chunks >> shift; } -static u64 get_corrected_val (struct rdt_resource *r,struct rdt_mon_domain *d, +static u64 get_corrected_val (struct rdt_resource *r,struct rdt_l3_mon_domain *d,u32 rmid,enum resctrl_event_id eventid,u64 msr_val) { -struct rdt_hw_mon_domain *hw_dom =resctrl_to_arch_mon_dom (d); +struct rdt_hw_l3_mon_domain *hw_dom =resctrl_to_arch_mon_dom (d);struct rdt_hw_resource *hw_res =resctrl_to_arch_res (r);struct arch_mbm_state *am;u64 chunks;
@@ -242,7 +242,7 @@ int resctrl_arch_rmid_read(struct rdt_resource *r, struct rdt_domain_hdr *hdr, u32 unused, u32 rmid, enum resctrl_event_id eventid, u64 *val, void *ignored) { - struct rdt_mon_domain *d; + struct rdt_l3_mon_domain *d;u64 msr_val;u32 prmid;int cpu;
@@ -253,7 +253,7 @@ int resctrl_arch_rmid_read(struct rdt_resource *r, struct rdt_domain_hdr *hdr,if (!domain_header_is_valid (hdr,RESCTRL_MON_DOMAIN ,RDT_RESOURCE_L3 ))return -EINVAL ; - d = container_of(hdr, struct rdt_mon_domain, hdr); + d = container_of(hdr, struct rdt_l3_mon_domain, hdr); cpu =cpumask_any (&hdr->cpu_mask); prmid =logical_rmid_to_physical_rmid (cpu, rmid); ret =__rmid_read_phys (prmid, eventid, &msr_val);
@@ -302,11 +302,11 @@ static int __cntr_id_read(u32 cntr_id, u64 *val)return 0 ; } -void resctrl_arch_reset_cntr (struct rdt_resource *r,struct rdt_mon_domain *d, +void resctrl_arch_reset_cntr (struct rdt_resource *r,struct rdt_l3_mon_domain *d,u32 unused,u32 rmid,int cntr_id,enum resctrl_event_id eventid) { -struct rdt_hw_mon_domain *hw_dom =resctrl_to_arch_mon_dom (d); +struct rdt_hw_l3_mon_domain *hw_dom =resctrl_to_arch_mon_dom (d);struct arch_mbm_state *am; am =get_arch_mbm_state (hw_dom, rmid, eventid);
@@ -322,14 +322,14 @@ int resctrl_arch_cntr_read(struct rdt_resource *r, struct rdt_domain_hdr *hdr, u32 unused, u32 rmid, int cntr_id, enum resctrl_event_id eventid, u64 *val) { - struct rdt_mon_domain *d; + struct rdt_l3_mon_domain *d;u64 msr_val;int ret;if (!domain_header_is_valid (hdr,RESCTRL_MON_DOMAIN ,RDT_RESOURCE_L3 ))return -EINVAL ; - d = container_of(hdr, struct rdt_mon_domain, hdr); + d = container_of(hdr, struct rdt_l3_mon_domain, hdr); ret =__cntr_id_read (cntr_id, &msr_val);if (ret)return ret;
@@ -353,7 +353,7 @@ int resctrl_arch_cntr_read(struct rdt_resource *r, struct rdt_domain_hdr *hdr,* must adjust RMID counter numbers based on SNC node. See * logical_rmid_to_physical_rmid() for code that does this. */ -void arch_mon_domain_online (struct rdt_resource *r,struct rdt_mon_domain *d) +void arch_mon_domain_online (struct rdt_resource *r,struct rdt_l3_mon_domain *d) {if (snc_nodes_per_l3_cache >1 )msr_clear_bit (MSR_RMID_SNC_CONFIG ,0 );
@@ -505,7 +505,7 @@ static void resctrl_abmc_set_one_amd(void *arg)*/ static void _resctrl_abmc_enable (struct rdt_resource *r,bool enable) { -struct rdt_mon_domain *d; +struct rdt_l3_mon_domain *d;lockdep_assert_cpus_held ();
@@ -544,11 +544,11 @@ static void resctrl_abmc_config_one_amd(void *info)/* * Send an IPI to the domain to assign the counter to RMID, event pair. */ -void resctrl_arch_config_cntr (struct rdt_resource *r,struct rdt_mon_domain *d, +void resctrl_arch_config_cntr (struct rdt_resource *r,struct rdt_l3_mon_domain *d,enum resctrl_event_id evtid,u32 rmid,u32 closid,u32 cntr_id,bool assign) { -struct rdt_hw_mon_domain *hw_dom =resctrl_to_arch_mon_dom (d); +struct rdt_hw_l3_mon_domain *hw_dom =resctrl_to_arch_mon_dom (d);union l3_qos_abmc_cfg abmc_cfg = {0 };struct arch_mbm_state *am;
fs/resctrl/ctrlmondata.c +1 -1
@@ -622,7 +622,7 @@ int rdtgroup_mondata_show(struct seq_file *m, void *arg) r =resctrl_arch_get_resource (resid);if (md->sum) { -struct rdt_mon_domain *d; +struct rdt_l3_mon_domain *d;/* * This file requires summing across all domains that share
fs/resctrl/monitor.c +34 -34
268 lines not on this page · raw message
fs/resctrl/rdtgroup.c +18 -18
@@ -1618,7 +1618,7 @@ static void mondata_config_read(struct resctrl_mon_config_info *mon_info)static int mbm_config_show (struct seq_file *s,struct rdt_resource *r,u32 evtid) {struct resctrl_mon_config_info mon_info; -struct rdt_mon_domain *dom; +struct rdt_l3_mon_domain *dom;bool sep = false;cpus_read_lock ();
@@ -1666,7 +1666,7 @@ static int mbm_local_bytes_config_show(struct kernfs_open_file *of, }static void mbm_config_write_domain (struct rdt_resource *r, -struct rdt_mon_domain *d,u32 evtid,u32 val) +struct rdt_l3_mon_domain *d,u32 evtid,u32 val) {struct resctrl_mon_config_info mon_info = {0 };
@@ -1707,8 +1707,8 @@ static void mbm_config_write_domain(struct rdt_resource *r,static int mon_config_write (struct rdt_resource *r,char *tok,u32 evtid) {char *dom_str =NULL , *id_str; +struct rdt_l3_mon_domain *d;unsigned long dom_id, val; -struct rdt_mon_domain *d;/* Walking r->domains, ensure it can't race with cpuhp */ lockdep_assert_cpus_held ();
@@ -2716,7 +2716,7 @@ static int rdt_get_tree(struct fs_context *fc) {struct rdt_fs_context *ctx =rdt_fc2context (fc);unsigned long flags =RFTYPE_CTRL_BASE ; -struct rdt_mon_domain *dom; +struct rdt_l3_mon_domain *dom;struct rdt_resource *r;int ret;
@@ -3167,7 +3167,7 @@ static void rmdir_mondata_subdir_allrdtgrp(struct rdt_resource *r, struct rdt_domain_hdr *hdr) {struct rdtgroup *prgrp, *crgrp; -struct rdt_mon_domain *d; +struct rdt_l3_mon_domain *d;char subname[32 ];bool snc_mode;char name[32 ];
@@ -3175,7 +3175,7 @@ static void rmdir_mondata_subdir_allrdtgrp(struct rdt_resource *r,if (!domain_header_is_valid (hdr,RESCTRL_MON_DOMAIN ,RDT_RESOURCE_L3 ))return ; - d = container_of(hdr, struct rdt_mon_domain, hdr); + d = container_of(hdr, struct rdt_l3_mon_domain, hdr); snc_mode = r->mon_scope ==RESCTRL_L3_NODE ;sprintf (name,"mon_%s_%02d" , r->name, snc_mode ? d->ci_id : d->hdr.id);if (snc_mode)
@@ -3221,7 +3221,7 @@ static int mkdir_mondata_subdir(struct kernfs_node *parent_kn, struct rdt_resource *r, struct rdtgroup *prgrp) { struct kernfs_node *kn, *ckn; -struct rdt_mon_domain *d; +struct rdt_l3_mon_domain *d;char name[32 ];bool snc_mode;int ret =0 ;
@@ -3231,7 +3231,7 @@ static int mkdir_mondata_subdir(struct kernfs_node *parent_kn,if (!domain_header_is_valid (hdr,RESCTRL_MON_DOMAIN ,RDT_RESOURCE_L3 ))return -EINVAL ; - d = container_of(hdr, struct rdt_mon_domain, hdr); + d = container_of(hdr, struct rdt_l3_mon_domain, hdr); snc_mode = r->mon_scope ==RESCTRL_L3_NODE ;sprintf (name,"mon_%s_%02d" , r->name, snc_mode ? d->ci_id : d->hdr.id); kn =kernfs_find_and_get (parent_kn, name);
@@ -4174,7 +4174,7 @@ static void rdtgroup_setup_default(void)mutex_unlock (&rdtgroup_mutex); } -static void domain_destroy_mon_state (struct rdt_mon_domain *d) +static void domain_destroy_mon_state (struct rdt_l3_mon_domain *d) {int idx;
@@ -4198,7 +4198,7 @@ void resctrl_offline_ctrl_domain(struct rdt_resource *r, struct rdt_ctrl_domainvoid resctrl_offline_mon_domain (struct rdt_resource *r,struct rdt_domain_hdr *hdr) { -struct rdt_mon_domain *d; +struct rdt_l3_mon_domain *d;mutex_lock (&rdtgroup_mutex);
@@ -4212,7 +4212,7 @@ void resctrl_offline_mon_domain(struct rdt_resource *r, struct rdt_domain_hdr *hif (!domain_header_is_valid (hdr,RESCTRL_MON_DOMAIN ,RDT_RESOURCE_L3 )) goto out_unlock; - d = container_of(hdr, struct rdt_mon_domain, hdr); + d = container_of(hdr, struct rdt_l3_mon_domain, hdr);if (resctrl_is_mbm_enabled ())cancel_delayed_work (&d->mbm_over);if (resctrl_is_mon_event_enabled (QOS_L3_OCCUP_EVENT_ID ) &&has_busy_rmid (d)) {
@@ -4246,7 +4246,7 @@ void resctrl_offline_mon_domain(struct rdt_resource *r, struct rdt_domain_hdr *h* * Returns 0 for success, or -ENOMEM. */ -static int domain_setup_mon_state (struct rdt_resource *r,struct rdt_mon_domain *d) +static int domain_setup_mon_state (struct rdt_resource *r,struct rdt_l3_mon_domain *d) {u32 idx_limit =resctrl_arch_system_num_rmid_idx ();size_t tsize =sizeof (*d->mbm_states[0 ]);
@@ -4304,7 +4304,7 @@ int resctrl_online_ctrl_domain(struct rdt_resource *r, struct rdt_ctrl_domain *dint resctrl_online_mon_domain (struct rdt_resource *r,struct rdt_domain_hdr *hdr) { -struct rdt_mon_domain *d; +struct rdt_l3_mon_domain *d;int err = -EINVAL ;mutex_lock (&rdtgroup_mutex);
@@ -4312,7 +4312,7 @@ int resctrl_online_mon_domain(struct rdt_resource *r, struct rdt_domain_hdr *hdrif (!domain_header_is_valid (hdr,RESCTRL_MON_DOMAIN ,RDT_RESOURCE_L3 )) goto out_unlock; - d = container_of(hdr, struct rdt_mon_domain, hdr); + d = container_of(hdr, struct rdt_l3_mon_domain, hdr); err =domain_setup_mon_state (r, d);if (err) goto out_unlock;
@@ -4360,10 +4360,10 @@ static void clear_childcpus(struct rdtgroup *r, unsigned int cpu) } } -static struct rdt_mon_domain *get_mon_domain_from_cpu (int cpu, -struct rdt_resource *r) +static struct rdt_l3_mon_domain *get_mon_domain_from_cpu (int cpu, +struct rdt_resource *r) { -struct rdt_mon_domain *d; +struct rdt_l3_mon_domain *d;lockdep_assert_cpus_held ();
@@ -4379,7 +4379,7 @@ static struct rdt_mon_domain *get_mon_domain_from_cpu(int cpu,void resctrl_offline_cpu (unsigned int cpu) {struct rdt_resource *l3 =resctrl_arch_get_resource (RDT_RESOURCE_L3 ); -struct rdt_mon_domain *d; +struct rdt_l3_mon_domain *d;struct rdtgroup *rdtgrp;mutex_lock (&rdtgroup_mutex);
Signature · 1 line
2.51.0
Re: [PATCH v11 07/31] x86,fs/resctrl: Rename struct rdt_mon_domain and rdt_hw_mon_domain
Hi Tony,
Quoted 9 lines · Tony Luck
The upcoming telemetry event monitoring are not tied to the L3 resource and will have a new domain structures.
Rename the L3 resource specific domain data structures to include "
l3_" in their names to avoid confusion between the different resource specific domain structures:rdt_mon_domain->rdt_l3_mon_domainrdt_hw_mon_domain->rdt_hw_l3_mon_domain
Quoted 4 lines
No functional change.
Signed-off-by Tony Luck <[email protected]>---
Reinette
Tony Luck
maintainerPATCH v11 8/31 x86,fs/resctrl: Rename some L3 specific functions
In reply to Tony LuckWith the arrival of monitor events tied to new domains associated with a different resource it would be clearer if the L3 resource specific functions are more accurately named.
Rename three groups of functions:
Functions that allocate/free architecture per-RMID MBM state information:arch_domain_mbm_alloc() -> l3_mon_domain_mbm_alloc()mon_domain_free() -> l3_mon_domain_free()
Functions that allocate/free filesystem per-RMID MBM state information:domain_setup_mon_state() -> domain_setup_l3_mon_state()domain_destroy_mon_state() -> domain_destroy_l3_mon_state()
Initialization/exit:rdt_get_mon_l3_config() -> rdt_get_l3_mon_config()resctrl_mon_resource_init() -> resctrl_l3_mon_resource_init()resctrl_mon_resource_exit() -> resctrl_l3_mon_resource_exit()
--- arch/x86/kernel/cpu/resctrl/internal.h | 2 +- fs/resctrl/internal.h | 6 +++--- arch/x86/kernel/cpu/resctrl/core.c | 18 +++++++++--------- arch/x86/kernel/cpu/resctrl/monitor.c | 2 +- fs/resctrl/monitor.c | 6 +++--- fs/resctrl/rdtgroup.c | 22 +++++++++++----------- 6 files changed, 28 insertions(+), 28 deletions(-)
arch/x86/kernel/cpu/resctrl/internal.h +1 -1
@@ -208,7 +208,7 @@ union l3_qos_abmc_cfg {void rdt_ctrl_update (void *arg); -int rdt_get_mon_l3_config (struct rdt_resource *r); +int rdt_get_l3_mon_config (struct rdt_resource *r);bool rdt_cpu_has (int flag);
fs/resctrl/internal.h +3 -3
@@ -357,7 +357,9 @@ int alloc_rmid(u32 closid);void free_rmid (u32 closid,u32 rmid); -void resctrl_mon_resource_exit (void ); +int resctrl_l3_mon_resource_init (void ); + +void resctrl_l3_mon_resource_exit (void );void mon_event_count (void *info);
@@ -367,8 +369,6 @@ void mon_event_read(struct rmid_read *rr, struct rdt_resource *r, struct rdt_domain_hdr *hdr, struct rdtgroup *rdtgrp, cpumask_t *cpumask, int evtid, int first); -int resctrl_mon_resource_init (void ); -void mbm_setup_overflow_handler (struct rdt_l3_mon_domain *dom,unsigned long delay_ms,int exclude_cpu);
arch/x86/kernel/cpu/resctrl/core.c +9 -9
@@ -363,7 +363,7 @@ static void ctrl_domain_free(struct rdt_hw_ctrl_domain *hw_dom)kfree (hw_dom); } -static void mon_domain_free (struct rdt_hw_l3_mon_domain *hw_dom) +static void l3_mon_domain_free (struct rdt_hw_l3_mon_domain *hw_dom) {int idx;
@@ -396,11 +396,11 @@ static int domain_setup_ctrlval(struct rdt_resource *r, struct rdt_ctrl_domain * }/** -* arch_domain_mbm_alloc() - Allocate arch private storage for the MBM counters +* l3_mon_domain_mbm_alloc() - Allocate arch private storage for the MBM counters * @num_rmid: The size of the MBM counter array * @hw_dom: The domain that owns the allocated arrays */ -static int arch_domain_mbm_alloc (u32 num_rmid,struct rdt_hw_l3_mon_domain *hw_dom) +static int l3_mon_domain_mbm_alloc (u32 num_rmid,struct rdt_hw_l3_mon_domain *hw_dom) {size_t tsize =sizeof (*hw_dom->arch_mbm_states[0 ]);enum resctrl_event_id eventid;
@@ -514,7 +514,7 @@ static void l3_mon_domain_setup(int cpu, int id, struct rdt_resource *r, struct ci =get_cpu_cacheinfo_level (cpu,RESCTRL_L3_CACHE );if (!ci) {pr_warn_once ("Can't find L3 cache for CPU:%d resource %s\n" , cpu, r->name); -mon_domain_free (hw_dom); +l3_mon_domain_free (hw_dom);return ; } d->ci_id = ci->id;
@@ -522,8 +522,8 @@ static void l3_mon_domain_setup(int cpu, int id, struct rdt_resource *r, structarch_mon_domain_online (r, d); -if (arch_domain_mbm_alloc (r->mon.num_rmid, hw_dom)) { -mon_domain_free (hw_dom); +if (l3_mon_domain_mbm_alloc (r->mon.num_rmid, hw_dom)) { +l3_mon_domain_free (hw_dom);return ; }
@@ -533,7 +533,7 @@ static void l3_mon_domain_setup(int cpu, int id, struct rdt_resource *r, structif (err) {list_del_rcu (&d->hdr.list);synchronize_rcu (); -mon_domain_free (hw_dom); +l3_mon_domain_free (hw_dom); } }
@@ -658,7 +658,7 @@ static void domain_remove_cpu_mon(int cpu, struct rdt_resource *r)resctrl_offline_mon_domain (r, hdr);list_del_rcu (&hdr->list);synchronize_rcu (); -mon_domain_free (hw_dom); +l3_mon_domain_free (hw_dom);break ;default :pr_warn_once ("Unknown resource rid=%d\n" , r->rid);
@@ -906,7 +906,7 @@ static __init bool get_rdt_mon_resources(void)if (!ret)return false; -return !rdt_get_mon_l3_config (r); +return !rdt_get_l3_mon_config (r); }static __init void __check_quirks_intel (void )
arch/x86/kernel/cpu/resctrl/monitor.c +1 -1
@@ -422,7 +422,7 @@ static __init int snc_get_config(void)return ret; } -int __init rdt_get_mon_l3_config (struct rdt_resource *r) +int __init rdt_get_l3_mon_config (struct rdt_resource *r) {unsigned int mbm_offset = boot_cpu_data.x86_cache_mbm_width_offset;struct rdt_hw_resource *hw_res =resctrl_to_arch_res (r);
fs/resctrl/monitor.c +3 -3
@@ -1750,7 +1750,7 @@ ssize_t mbm_L3_assignments_write(struct kernfs_open_file *of, char *buf, }/** -* resctrl_mon_resource_init() - Initialise global monitoring structures. +* resctrl_l3_mon_resource_init() - Initialise global monitoring structures. * * Allocate and initialise global monitor resources that do not belong to a * specific domain. i.e. the rmid_ptrs[] used for the limbo and free lists.
@@ -1761,7 +1761,7 @@ ssize_t mbm_L3_assignments_write(struct kernfs_open_file *of, char *buf,* * Returns 0 for success, or -ENOMEM. */ -int resctrl_mon_resource_init (void ) +int resctrl_l3_mon_resource_init (void ) {struct rdt_resource *r =resctrl_arch_get_resource (RDT_RESOURCE_L3 );int ret;
@@ -1813,7 +1813,7 @@ int resctrl_mon_resource_init(void)return 0 ; } -void resctrl_mon_resource_exit (void ) +void resctrl_l3_mon_resource_exit (void ) {struct rdt_resource *r =resctrl_arch_get_resource (RDT_RESOURCE_L3 );
fs/resctrl/rdtgroup.c +11 -11
@@ -4174,7 +4174,7 @@ static void rdtgroup_setup_default(void)mutex_unlock (&rdtgroup_mutex); } -static void domain_destroy_mon_state (struct rdt_l3_mon_domain *d) +static void domain_destroy_l3_mon_state (struct rdt_l3_mon_domain *d) {int idx;
@@ -4228,13 +4228,13 @@ void resctrl_offline_mon_domain(struct rdt_resource *r, struct rdt_domain_hdr *hcancel_delayed_work (&d->cqm_limbo); } -domain_destroy_mon_state (d); +domain_destroy_l3_mon_state (d); out_unlock:mutex_unlock (&rdtgroup_mutex); }/** -* domain_setup_mon_state() - Initialise domain monitoring structures. +* domain_setup_l3_mon_state() - Initialise domain monitoring structures. * @r: The resource for the newly online domain. * @d: The newly online domain. *
@@ -4242,11 +4242,11 @@ void resctrl_offline_mon_domain(struct rdt_resource *r, struct rdt_domain_hdr *h* Called when the first CPU of a domain comes online, regardless of whether * the filesystem is mounted. * During boot this may be called before global allocations have been made by -* resctrl_mon_resource_init(). +* resctrl_l3_mon_resource_init(). * * Returns 0 for success, or -ENOMEM. */ -static int domain_setup_mon_state (struct rdt_resource *r,struct rdt_l3_mon_domain *d) +static int domain_setup_l3_mon_state (struct rdt_resource *r,struct rdt_l3_mon_domain *d) {u32 idx_limit =resctrl_arch_system_num_rmid_idx ();size_t tsize =sizeof (*d->mbm_states[0 ]);
@@ -4313,7 +4313,7 @@ int resctrl_online_mon_domain(struct rdt_resource *r, struct rdt_domain_hdr *hdr goto out_unlock; d = container_of(hdr, struct rdt_l3_mon_domain, hdr); - err =domain_setup_mon_state (r, d); + err =domain_setup_l3_mon_state (r, d);if (err) goto out_unlock;
@@ -4429,13 +4429,13 @@ int resctrl_init(void)thread_throttle_mode_init (); - ret =resctrl_mon_resource_init (); + ret =resctrl_l3_mon_resource_init ();if (ret)return ret; ret =sysfs_create_mount_point (fs_kobj,"resctrl" );if (ret) { -resctrl_mon_resource_exit (); +resctrl_l3_mon_resource_exit ();return ret; }
@@ -4470,7 +4470,7 @@ int resctrl_init(void) cleanup_mountpoint:sysfs_remove_mount_point (fs_kobj,"resctrl" ); -resctrl_mon_resource_exit (); +resctrl_l3_mon_resource_exit ();return ret; }
@@ -4506,7 +4506,7 @@ static bool resctrl_online_domains_exist(void)* When called by the architecture code, all CPUs and resctrl domains must be * offline. This ensures the limbo and overflow handlers are not scheduled to * run, meaning the data structures they access can be freed by -* resctrl_mon_resource_exit(). +* resctrl_l3_mon_resource_exit(). * * After resctrl_exit() returns, the architecture code should return an * error from all resctrl_arch_ functions that can do this.
@@ -4533,5 +4533,5 @@ void resctrl_exit(void)* it can be used to umount resctrl. */ -resctrl_mon_resource_exit (); +resctrl_l3_mon_resource_exit (); }
Signature · 1 line
2.51.0
Re: [PATCH v11 08/31] x86,fs/resctrl: Rename some L3 specific functions
Hi Tony,
Quoted 14 lines · Tony Luck
With the arrival of monitor events tied to new domains associated with a different resource it would be clearer if the L3 resource specific functions are more accurately named.
Rename three groups of functions:
Functions that allocate/free architecture per-RMID MBM state information:
arch_domain_mbm_alloc()->l3_mon_domain_mbm_alloc()mon_domain_free()->l3_mon_domain_free()Functions that allocate/free filesystem per-RMID MBM state information:
domain_setup_mon_state()->domain_setup_l3_mon_state()domain_destroy_mon_state()->domain_destroy_l3_mon_state()
Quoted 7 lines
Initialization/exit:
rdt_get_mon_l3_config()->rdt_get_l3_mon_config()resctrl_mon_resource_init()->resctrl_l3_mon_resource_init()resctrl_mon_resource_exit()->resctrl_l3_mon_resource_exit()Signed-off-by Tony Luck <[email protected]>---
Reinette
Tony Luck
maintainerPATCH v11 9/31 fs/resctrl: Make event details accessible to functions when reading events
In reply to Tony LuckReading monitoring data from MMIO requires more context to be able to read the correct memory location. struct mon_evt is the appropriate place for this event specific context.
Prepare for addition of extra fields to mon_evt by changing the calling conventions to pass a pointer to the mon_evt structure instead of just the event id.
--- fs/resctrl/internal.h | 10 +++++----- fs/resctrl/ctrlmondata.c | 18 +++++++++--------- fs/resctrl/monitor.c | 24 ++++++++++++------------ fs/resctrl/rdtgroup.c | 6 +++--- 4 files changed, 29 insertions(+), 29 deletions(-)
fs/resctrl/internal.h +5 -5
@@ -81,7 +81,7 @@ extern struct mon_evt mon_event_all[QOS_NUM_EVENTS];* struct mon_data - Monitoring details for each event file. * @list: Member of the global @mon_data_kn_priv_list list. * @rid: Resource id associated with the event file. -* @evtid: Event id associated with the event file. +* @evt: Event structure associated with the event file. * @sum: Set when event must be summed across multiple * domains. * @domid: When @sum is zero this is the domain to which
@@ -95,7 +95,7 @@ extern struct mon_evt mon_event_all[QOS_NUM_EVENTS];struct mon_data {struct list_head list;enum resctrl_res_level rid; -enum resctrl_event_id evtid; +struct mon_evt *evt;int domid;bool sum; };
@@ -108,7 +108,7 @@ struct mon_data {* @r: Resource describing the properties of the event being read. * @hdr: Header of domain that the counter should be read from. If NULL then * sum all domains in @r sharing L3 @ci.id -* @evtid: Which monitor event to read. +* @evt: Which monitor event to read. * @first: Initialize MBM counter when true. * @ci: Cacheinfo for L3. Only set when @hdr is NULL. Used when summing * domains.
@@ -126,7 +126,7 @@ struct rmid_read {struct rdtgroup *rgrp;struct rdt_resource *r;struct rdt_domain_hdr *hdr; -enum resctrl_event_id evtid; +struct mon_evt *evt;bool first;struct cacheinfo *ci;bool is_mbm_cntr;
@@ -367,7 +367,7 @@ int rdtgroup_mondata_show(struct seq_file *m, void *arg);void mon_event_read (struct rmid_read *rr,struct rdt_resource *r,struct rdt_domain_hdr *hdr,struct rdtgroup *rdtgrp, -cpumask_t *cpumask,int evtid,int first); +cpumask_t *cpumask,struct mon_evt *evt,int first); void mbm_setup_overflow_handler(struct rdt_l3_mon_domain *dom, unsigned long delay_ms,
fs/resctrl/ctrlmondata.c +9 -9
@@ -548,7 +548,7 @@ struct rdt_domain_hdr *resctrl_find_domain(struct list_head *h, int id,void mon_event_read (struct rmid_read *rr,struct rdt_resource *r,struct rdt_domain_hdr *hdr,struct rdtgroup *rdtgrp, -cpumask_t *cpumask,int evtid,int first) +cpumask_t *cpumask,struct mon_evt *evt,int first) {int cpu;
@@ -559,15 +559,15 @@ void mon_event_read(struct rmid_read *rr, struct rdt_resource *r,* Setup the parameters to pass to mon_event_count() to read the data. */ rr->rgrp = rdtgrp; - rr->evtid = evtid; + rr->evt = evt; rr->r = r; rr->hdr = hdr; rr->first = first;if (resctrl_arch_mbm_cntr_assign_enabled (r) && -resctrl_is_mbm_event (evtid)) { +resctrl_is_mbm_event (evt->evtid)) { rr->is_mbm_cntr = true; }else { - rr->arch_mon_ctx =resctrl_arch_mon_ctx_alloc (r, evtid); + rr->arch_mon_ctx =resctrl_arch_mon_ctx_alloc (r, evt->evtid);if (IS_ERR (rr->arch_mon_ctx)) { rr->err = -EINVAL ;return ;
@@ -588,20 +588,20 @@ void mon_event_read(struct rmid_read *rr, struct rdt_resource *r,smp_call_on_cpu (cpu, smp_mon_event_count, rr, false);if (rr->arch_mon_ctx) -resctrl_arch_mon_ctx_free (r, evtid, rr->arch_mon_ctx); +resctrl_arch_mon_ctx_free (r, evt->evtid, rr->arch_mon_ctx); }int rdtgroup_mondata_show (struct seq_file *m,void *arg) {struct kernfs_open_file *of = m->private;enum resctrl_res_level resid; -enum resctrl_event_id evtid;struct rdt_domain_hdr *hdr;struct rmid_read rr = {0 };struct rdtgroup *rdtgrp;int domid, cpu, ret =0 ;struct rdt_resource *r;struct cacheinfo *ci; +struct mon_evt *evt;struct mon_data *md; rdtgrp =rdtgroup_kn_lock_live (of->kn);
@@ -618,7 +618,7 @@ int rdtgroup_mondata_show(struct seq_file *m, void *arg) resid = md->rid; domid = md->domid; - evtid = md->evtid; + evt = md->evt; r =resctrl_arch_get_resource (resid);if (md->sum) {
@@ -638,7 +638,7 @@ int rdtgroup_mondata_show(struct seq_file *m, void *arg)continue ; rr.ci = ci;mon_event_read (&rr, r,NULL , rdtgrp, - &ci->shared_cpu_map, evtid, false); + &ci->shared_cpu_map, evt, false); goto checkresult; } }
@@ -654,7 +654,7 @@ int rdtgroup_mondata_show(struct seq_file *m, void *arg) ret = -ENOENT ; goto out; } -mon_event_read (&rr, r, hdr, rdtgrp, &hdr->cpu_mask, evtid, false); +mon_event_read (&rr, r, hdr, rdtgrp, &hdr->cpu_mask, evt, false); } checkresult:
fs/resctrl/monitor.c +12 -12
@@ -429,7 +429,7 @@ static int __mon_event_count(struct rdtgroup *rdtgrp, struct rmid_read *rr) d = container_of(rr->hdr, struct rdt_l3_mon_domain, hdr);if (rr->is_mbm_cntr) { - cntr_id =mbm_cntr_get (rr->r, d, rdtgrp, rr->evtid); + cntr_id =mbm_cntr_get (rr->r, d, rdtgrp, rr->evt->evtid);if (cntr_id <0 ) { rr->err = -ENOENT ;return -EINVAL ;
@@ -438,10 +438,10 @@ static int __mon_event_count(struct rdtgroup *rdtgrp, struct rmid_read *rr)if (rr->first) {if (rr->is_mbm_cntr) -resctrl_arch_reset_cntr (rr->r, d, closid, rmid, cntr_id, rr->evtid); +resctrl_arch_reset_cntr (rr->r, d, closid, rmid, cntr_id, rr->evt->evtid);else -resctrl_arch_reset_rmid (rr->r, d, closid, rmid, rr->evtid); - m =get_mbm_state (d, closid, rmid, rr->evtid); +resctrl_arch_reset_rmid (rr->r, d, closid, rmid, rr->evt->evtid); + m =get_mbm_state (d, closid, rmid, rr->evt->evtid);if (m)memset (m,0 ,sizeof (struct mbm_state ));return 0 ;
@@ -453,10 +453,10 @@ static int __mon_event_count(struct rdtgroup *rdtgrp, struct rmid_read *rr)return -EINVAL ;if (rr->is_mbm_cntr) rr->err =resctrl_arch_cntr_read (rr->r, rr->hdr, closid, rmid, cntr_id, - rr->evtid, &tval); + rr->evt->evtid, &tval);else rr->err =resctrl_arch_rmid_read (rr->r, rr->hdr, closid, rmid, - rr->evtid, &tval, rr->arch_mon_ctx); + rr->evt->evtid, &tval, rr->arch_mon_ctx);if (rr->err)return rr->err;
@@ -482,10 +482,10 @@ static int __mon_event_count(struct rdtgroup *rdtgrp, struct rmid_read *rr)continue ;if (rr->is_mbm_cntr) err =resctrl_arch_cntr_read (rr->r, &d->hdr, closid, rmid, cntr_id, - rr->evtid, &tval); + rr->evt->evtid, &tval);else err =resctrl_arch_rmid_read (rr->r, &d->hdr, closid, rmid, - rr->evtid, &tval, rr->arch_mon_ctx); + rr->evt->evtid, &tval, rr->arch_mon_ctx);if (!err) { rr->val += tval; ret =0 ;
@@ -521,7 +521,7 @@ static void mbm_bw_count(struct rdtgroup *rdtgrp, struct rmid_read *rr)if (!domain_header_is_valid (rr->hdr,RESCTRL_MON_DOMAIN ,RDT_RESOURCE_L3 ))return ; d = container_of(rr->hdr, struct rdt_l3_mon_domain, hdr); - m =get_mbm_state (d, closid, rmid, rr->evtid); + m =get_mbm_state (d, closid, rmid, rr->evt->evtid);if (WARN_ON_ONCE (!m))return ;
@@ -695,11 +695,11 @@ static void mbm_update_one_event(struct rdt_resource *r, struct rdt_l3_mon_domai rr.r = r; rr.hdr = &d->hdr; - rr.evtid = evtid; + rr.evt = &mon_event_all[evtid];if (resctrl_arch_mbm_cntr_assign_enabled (r)) { rr.is_mbm_cntr = true; }else { - rr.arch_mon_ctx =resctrl_arch_mon_ctx_alloc (rr.r, rr.evtid); + rr.arch_mon_ctx =resctrl_arch_mon_ctx_alloc (rr.r, evtid);if (IS_ERR (rr.arch_mon_ctx)) {pr_warn_ratelimited ("Failed to allocate monitor context: %ld" ,PTR_ERR (rr.arch_mon_ctx));
@@ -717,7 +717,7 @@ static void mbm_update_one_event(struct rdt_resource *r, struct rdt_l3_mon_domaimbm_bw_count (rdtgrp, &rr);if (rr.arch_mon_ctx) -resctrl_arch_mon_ctx_free (rr.r, rr.evtid, rr.arch_mon_ctx); +resctrl_arch_mon_ctx_free (rr.r, evtid, rr.arch_mon_ctx); } static void mbm_update(struct rdt_resource *r, struct rdt_l3_mon_domain *d,
fs/resctrl/rdtgroup.c +3 -3
@@ -3038,7 +3038,7 @@ static struct mon_data *mon_get_kn_priv(enum resctrl_res_level rid, int domid,list_for_each_entry (priv, &mon_data_kn_priv_list, list) {if (priv->rid == rid && priv->domid == domid && - priv->sum == do_sum && priv->evtid == mevt->evtid) + priv->sum == do_sum && priv->evt == mevt)return priv; }
@@ -3049,7 +3049,7 @@ static struct mon_data *mon_get_kn_priv(enum resctrl_res_level rid, int domid, priv->rid = rid; priv->domid = domid; priv->sum = do_sum; - priv->evtid = mevt->evtid; + priv->evt = mevt;list_add_tail (&priv->list, &mon_data_kn_priv_list);return priv;
@@ -3210,7 +3210,7 @@ static int mon_add_all_files(struct kernfs_node *kn, struct rdt_domain_hdr *hdr,return ret;if (!do_sum &&resctrl_is_mbm_event (mevt->evtid)) -mon_event_read (&rr, r, hdr, prgrp, &hdr->cpu_mask, mevt->evtid, true); +mon_event_read (&rr, r, hdr, prgrp, &hdr->cpu_mask, mevt, true); }return 0 ;
Signature · 1 line
2.51.0
Reinette Chatre
maintainerRe: [PATCH v11 09/31] fs/resctrl: Make event details accessible to functions when reading events
Hi Tony,
Quoted 1 line · Tony Luck
Reading monitoring data from MMIO requires more context to be able to
To tie this together with the "instead of just the event id" used later I'd propose:
Reading monitoring event data from MMIO requires more context than the event id to be able to ...
Quoted 4 lines
read the correct memory location. struct
mon_evtis the appropriate place for this event specific context.Prepare for addition of extra fields to
mon_evtby changing the calling
"mon_evt" -> "struct mon_evt"
Quoted 5 lines
conventions to pass a pointer to the
mon_evtstructure instead of just the event id.Signed-off-by Tony Luck <[email protected]>---
| Reviewed-by: Reinette Chatre <[email protected]>
Reinette
Tony Luck
maintainerPATCH v11 10/31 x86,fs/resctrl: Handle events that can be read from any CPU
In reply to Tony Luckresctrl assumes that monitor events can only be read from a CPU in the cpumask_t set of each domain. This is true for x86 events accessed with an MSR interface, but may not be true for other access methods such as MMIO.
Add a flag to struct mon_evt, settable by architecture code, to indicate there are no restrictions on which CPU can read that event.
Bypass all the smp_call*() code for events that can be read on any CPU and call mon_event_count() directly from mon_event_read().
Simplify CPU checking in __mon_event_count() with a helper.
--- include/linux/resctrl.h | 2 +- fs/resctrl/internal.h | 2 ++ arch/x86/kernel/cpu/resctrl/core.c | 6 ++--- fs/resctrl/ctrlmondata.c | 6 +++++ fs/resctrl/monitor.c | 43 ++++++++++++++++++++++-------- 5 files changed, 44 insertions(+), 15 deletions(-)
include/linux/resctrl.h +1 -1
@@ -409,7 +409,7 @@ u32 resctrl_arch_get_num_closid(struct rdt_resource *r);u32 resctrl_arch_system_num_rmid_idx (void );int resctrl_arch_update_domains (struct rdt_resource *r,u32 closid); -void resctrl_enable_mon_event (enum resctrl_event_id eventid); +void resctrl_enable_mon_event (enum resctrl_event_id eventid,bool any_cpu);bool resctrl_is_mon_event_enabled (enum resctrl_event_id eventid);
fs/resctrl/internal.h +2 -0
@@ -61,6 +61,7 @@ static inline struct rdt_fs_context *rdt_fc2context(struct fs_context *fc)* READS_TO_REMOTE_MEM) being tracked by @evtid. * Only valid if @evtid is an MBM event. * @configurable: true if the event is configurable +* @any_cpu: true if the event can be read from any CPU * @enabled: true if the event is enabled */ struct mon_evt {
@@ -69,6 +70,7 @@ struct mon_evt {char *name;u32 evt_cfg;bool configurable; +bool any_cpu;bool enabled; };
arch/x86/kernel/cpu/resctrl/core.c +3 -3
@@ -889,15 +889,15 @@ static __init bool get_rdt_mon_resources(void)bool ret = false;if (rdt_cpu_has (X86_FEATURE_CQM_OCCUP_LLC )) { -resctrl_enable_mon_event (QOS_L3_OCCUP_EVENT_ID ); +resctrl_enable_mon_event (QOS_L3_OCCUP_EVENT_ID , false); ret = true; }if (rdt_cpu_has (X86_FEATURE_CQM_MBM_TOTAL )) { -resctrl_enable_mon_event (QOS_L3_MBM_TOTAL_EVENT_ID ); +resctrl_enable_mon_event (QOS_L3_MBM_TOTAL_EVENT_ID , false); ret = true; }if (rdt_cpu_has (X86_FEATURE_CQM_MBM_LOCAL )) { -resctrl_enable_mon_event (QOS_L3_MBM_LOCAL_EVENT_ID ); +resctrl_enable_mon_event (QOS_L3_MBM_LOCAL_EVENT_ID , false); ret = true; }if (rdt_cpu_has (X86_FEATURE_ABMC ))
fs/resctrl/ctrlmondata.c +6 -0
@@ -574,6 +574,11 @@ void mon_event_read(struct rmid_read *rr, struct rdt_resource *r, } } +if (evt->any_cpu) { +mon_event_count (rr); + goto out_ctx_free; + } + cpu =cpumask_any_housekeeping (cpumask,RESCTRL_PICK_ANY_CPU );/*
@@ -587,6 +592,7 @@ void mon_event_read(struct rmid_read *rr, struct rdt_resource *r,else smp_call_on_cpu (cpu, smp_mon_event_count, rr, false); +out_ctx_free:if (rr->arch_mon_ctx)resctrl_arch_mon_ctx_free (r, evt->evtid, rr->arch_mon_ctx); }
fs/resctrl/monitor.c +32 -11
88 lines not on this page · raw message
Signature · 1 line
2.51.0
Reinette Chatre
maintainerRe: [PATCH v11 10/31] x86,fs/resctrl: Handle events that can be read from any CPU
Hi Tony,
Quoted 6 lines · Tony Luck
resctrl assumes that monitor events can only be read from a CPU in the
cpumask_tset of each domain. This is true for x86 events accessed with an MSR interface, but may not be true for other access methods such as MMIO.Add a flag to struct
mon_evt, settable by architecture code, to indicate
"Add a flag to struct mon_evt" -> "Add flag mon_evt::any_cpu"
Quoted 1 line
there are no restrictions on which CPU can read that event.
Or rather:
Introduce and use flag mon_evt::any_cpu, settable by architecture, that indicates there are no restrictions on which CPU can read that event.
Quoted 3 lines
Bypass all the smp_call*() code for events that can be read on any CPU and call
mon_event_count()directly frommon_event_read().
Above (from "Bypass ...") can be dropped since it is clear from patch.
Quoted 2 lines
Simplify CPU checking in
__mon_event_count()with a helper.
Above can be seen from patch but when trying to do so it is not clear why this helper is needed and indicates that this is missing "why". Proposal:
Refactor the CPU checking to avoid always calling smp_processor_id() now that events can be read from preemptible context.
Quoted 3 lines
Signed-off-by Tony Luck <[email protected]>---
Reinette
Tony Luck
maintainerPATCH v11 11/31 x86,fs/resctrl: Support binary fixed point event counters
In reply to Tony Luckresctrl assumes that all monitor events can be displayed as unsigned decimal integers.
Hardware architecture counters may provide some telemetry events with greater precision where the event is not a simple count, but is a measurement of some sort (e.g. Joules for energy consumed).
Add a new argument to resctrl_enable_mon_event() for architecture code to inform the file system that the value for a counter is a fixed-point value with a specific number of binary places.
Only allow architecture to use floating point format on events that the file system has marked with mon_evt::is_floating_point.
Display fixed point values with values rounded to an appropriate number of decimal places for the precision of the number of binary places provided. Add one extra decimal place for every three additional binary places, except for low precision binary values where exact representation is possible:
1 binary place is 0.0 or 0.5 => 1 decimal place
2 binary places is 0.0, 0.25, 0.5, 0.75 => 2 decimal places
3 binary places is 0.0, 0.125, etc. => 3 decimal places
--- include/linux/resctrl.h | 3 +- fs/resctrl/internal.h | 8 +++ arch/x86/kernel/cpu/resctrl/core.c | 6 +-- fs/resctrl/ctrlmondata.c | 84 ++++++++++++++++++++++++++++++ fs/resctrl/monitor.c | 14 +++-- 5 files changed, 107 insertions(+), 8 deletions(-)
include/linux/resctrl.h +2 -1
@@ -409,7 +409,8 @@ u32 resctrl_arch_get_num_closid(struct rdt_resource *r);u32 resctrl_arch_system_num_rmid_idx (void );int resctrl_arch_update_domains (struct rdt_resource *r,u32 closid); -void resctrl_enable_mon_event (enum resctrl_event_id eventid,bool any_cpu); +void resctrl_enable_mon_event (enum resctrl_event_id eventid,bool any_cpu, +unsigned int binary_bits);bool resctrl_is_mon_event_enabled (enum resctrl_event_id eventid);
fs/resctrl/internal.h +8 -0
@@ -62,6 +62,9 @@ static inline struct rdt_fs_context *rdt_fc2context(struct fs_context *fc)* Only valid if @evtid is an MBM event. * @configurable: true if the event is configurable * @any_cpu: true if the event can be read from any CPU +* @is_floating_point: event values are displayed in floating point format +* @binary_bits: number of fixed-point binary bits from architecture, +* only valid if @is_floating_point is true * @enabled: true if the event is enabled */ struct mon_evt {
@@ -71,6 +74,8 @@ struct mon_evt {u32 evt_cfg;bool configurable;bool any_cpu; +bool is_floating_point; +unsigned int binary_bits;bool enabled; };
@@ -79,6 +84,9 @@ extern struct mon_evt mon_event_all[QOS_NUM_EVENTS];#define for_each_mon_event (mevt) for (mevt = &mon_event_all[QOS_FIRST_EVENT]; \ mevt < &mon_event_all[QOS_NUM_EVENTS]; mevt++) +/* Limit for mon_evt::binary_bits */ +#define MAX_BINARY_BITS 27 +/** * struct mon_data - Monitoring details for each event file. * @list: Member of the global @mon_data_kn_priv_list list.
arch/x86/kernel/cpu/resctrl/core.c +3 -3
@@ -889,15 +889,15 @@ static __init bool get_rdt_mon_resources(void)bool ret = false;if (rdt_cpu_has (X86_FEATURE_CQM_OCCUP_LLC )) { -resctrl_enable_mon_event (QOS_L3_OCCUP_EVENT_ID , false); +resctrl_enable_mon_event (QOS_L3_OCCUP_EVENT_ID , false,0 ); ret = true; }if (rdt_cpu_has (X86_FEATURE_CQM_MBM_TOTAL )) { -resctrl_enable_mon_event (QOS_L3_MBM_TOTAL_EVENT_ID , false); +resctrl_enable_mon_event (QOS_L3_MBM_TOTAL_EVENT_ID , false,0 ); ret = true; }if (rdt_cpu_has (X86_FEATURE_CQM_MBM_LOCAL )) { -resctrl_enable_mon_event (QOS_L3_MBM_LOCAL_EVENT_ID , false); +resctrl_enable_mon_event (QOS_L3_MBM_LOCAL_EVENT_ID , false,0 ); ret = true; }if (rdt_cpu_has (X86_FEATURE_ABMC ))
fs/resctrl/ctrlmondata.c +84 -0
102 lines not on this page · raw message
fs/resctrl/monitor.c +10 -4
@@ -977,16 +977,22 @@ struct mon_evt mon_event_all[QOS_NUM_EVENTS] = { }, }; -void resctrl_enable_mon_event (enum resctrl_event_id eventid,bool any_cpu) +void resctrl_enable_mon_event (enum resctrl_event_id eventid,bool any_cpu,unsigned int binary_bits) { -if (WARN_ON_ONCE (eventid <QOS_FIRST_EVENT || eventid >=QOS_NUM_EVENTS )) +if (WARN_ON_ONCE (eventid <QOS_FIRST_EVENT || eventid >=QOS_NUM_EVENTS || + binary_bits >MAX_BINARY_BITS ))return ;if (mon_event_all[eventid].enabled) {pr_warn ("Duplicate enable for event %d\n" , eventid);return ; } +if (binary_bits && !mon_event_all[eventid].is_floating_point) { +pr_warn ("Event %d may not be floating point\n" , eventid); +return ; + } mon_event_all[eventid].any_cpu = any_cpu; + mon_event_all[eventid].binary_bits = binary_bits; mon_event_all[eventid].enabled = true; }
@@ -1812,9 +1818,9 @@ int resctrl_l3_mon_resource_init(void)if (r->mon.mbm_cntr_assignable) {if (!resctrl_is_mon_event_enabled (QOS_L3_MBM_TOTAL_EVENT_ID )) -resctrl_enable_mon_event (QOS_L3_MBM_TOTAL_EVENT_ID , false); +resctrl_enable_mon_event (QOS_L3_MBM_TOTAL_EVENT_ID , false,0 );if (!resctrl_is_mon_event_enabled (QOS_L3_MBM_LOCAL_EVENT_ID )) -resctrl_enable_mon_event (QOS_L3_MBM_LOCAL_EVENT_ID , false); +resctrl_enable_mon_event (QOS_L3_MBM_LOCAL_EVENT_ID , false,0 ); mon_event_all[QOS_L3_MBM_TOTAL_EVENT_ID ].evt_cfg = r->mon.mbm_cfg_mask; mon_event_all[QOS_L3_MBM_LOCAL_EVENT_ID].evt_cfg = r->mon.mbm_cfg_mask & (READS_TO_LOCAL_MEM |
Signature · 1 line
2.51.0
Tony Luck
maintainerPATCH v11 12/31 x86,fs/resctrl: Add an architectural hook called for each mount
In reply to Tony LuckEnumeration of Intel telemetry events is an asynchronous process involving several mutually dependent drivers added as auxiliary devices during the device_initcall() phase of Linux boot. The process finishes after the probe functions of these drivers completes. But this happens after resctrl_arch_late_init() is executed.
Tracing the enumeration process shows that it does complete a full seven seconds before the earliest possible mount of the resctrl file system (when included in /etc/fstab for automatic mount by systemd).
Add a hook at the beginning of the mount code that will be used to check for telemetry events and initialize if any are found.
Call the hook on every attempted mount. Expectations are that most actions (like enumeration) will only need to be performed on the first call.
resctrl filesystem calls the hook with no locks held. Architecture code is responsible for any required locking.
--- include/linux/resctrl.h | 6 ++++++ arch/x86/kernel/cpu/resctrl/core.c | 9 +++++++++ fs/resctrl/rdtgroup.c | 2 ++ 3 files changed, 17 insertions(+)
include/linux/resctrl.h +6 -0
@@ -511,6 +511,12 @@ void resctrl_offline_mon_domain(struct rdt_resource *r, struct rdt_domain_hdr *hvoid resctrl_online_cpu (unsigned int cpu);void resctrl_offline_cpu (unsigned int cpu); +/* +* Architecture hook called at beginning of each file system mount attempt. +* No locks are held. +*/ +void resctrl_arch_pre_mount (void ); +/** * resctrl_arch_rmid_read() - Read the eventid counter corresponding to rmid * for this resource and domain.
arch/x86/kernel/cpu/resctrl/core.c +9 -0
@@ -717,6 +717,15 @@ static int resctrl_arch_offline_cpu(unsigned int cpu)return 0 ; } +void resctrl_arch_pre_mount (void ) +{ +static atomic_t only_once =ATOMIC_INIT (0 ); +int old =0 ; + +if (!atomic_try_cmpxchg (&only_once, &old,1 )) +return ; +} +enum {RDT_FLAG_CMT ,RDT_FLAG_MBM_TOTAL ,
fs/resctrl/rdtgroup.c +2 -0
@@ -2720,6 +2720,8 @@ static int rdt_get_tree(struct fs_context *fc)struct rdt_resource *r;int ret; +resctrl_arch_pre_mount (); +cpus_read_lock ();mutex_lock (&rdtgroup_mutex);/*
Signature · 1 line
2.51.0
Tony Luck
maintainerPATCH v11 13/31 x86,fs/resctrl: Add and initialize rdt_resource for package scope monitor
In reply to Tony LuckAdd a new PERF_PKG resource and introduce package level scope for monitoring telemetry events so that CPU hot plug notifiers can build domains at the package granularity.
Use the physical package ID available via topology_physical_package_id() to identify the monitoring domains with package level scope. This enables user space to use:
/sys/devices/system/cpu/cpuX/topology/physical_package_id
to identify the monitoring domain a CPU is associated with.
--- include/linux/resctrl.h | 2 ++ fs/resctrl/internal.h | 2 ++ arch/x86/kernel/cpu/resctrl/core.c | 10 ++++++++++ fs/resctrl/rdtgroup.c | 2 ++ 4 files changed, 16 insertions(+)
include/linux/resctrl.h +2 -0
@@ -53,6 +53,7 @@ enum resctrl_res_level {RDT_RESOURCE_L2 ,RDT_RESOURCE_MBA ,RDT_RESOURCE_SMBA , +RDT_RESOURCE_PERF_PKG ,/* Must be the last */ RDT_NUM_RESOURCES,
@@ -267,6 +268,7 @@ enum resctrl_scope {RESCTRL_L2_CACHE =2 ,RESCTRL_L3_CACHE =3 ,RESCTRL_L3_NODE , + RESCTRL_PACKAGE, };/**
fs/resctrl/internal.h +2 -0
@@ -255,6 +255,8 @@ struct rdtgroup {#define RFTYPE_ASSIGN_CONFIG BIT(11) +#define RFTYPE_RES_PERF_PKG BIT(11) +#define RFTYPE_CTRL_INFO (RFTYPE_INFO | RFTYPE_CTRL)#define RFTYPE_MON_INFO (RFTYPE_INFO | RFTYPE_MON)
arch/x86/kernel/cpu/resctrl/core.c +10 -0
@@ -100,6 +100,14 @@ struct rdt_hw_resource rdt_resources_all[RDT_NUM_RESOURCES] = { .schema_fmt = RESCTRL_SCHEMA_RANGE, }, }, + [RDT_RESOURCE_PERF_PKG ] = + { + .r_resctrl = { + .name ="PERF_PKG" , + .mon_scope =RESCTRL_PACKAGE , + .mon_domains =mon_domain_init (RDT_RESOURCE_PERF_PKG ), + }, + }, };u32 resctrl_arch_system_num_rmid_idx (void )
@@ -433,6 +441,8 @@ static int get_domain_id_from_scope(int cpu, enum resctrl_scope scope)return get_cpu_cacheinfo_id (cpu, scope);case RESCTRL_L3_NODE :return cpu_to_node (cpu); +case RESCTRL_PACKAGE : +return topology_physical_package_id (cpu);default :break ; }
fs/resctrl/rdtgroup.c +2 -0
@@ -2330,6 +2330,8 @@ static unsigned long fflags_from_resource(struct rdt_resource *r)case RDT_RESOURCE_MBA :case RDT_RESOURCE_SMBA :return RFTYPE_RES_MB ; +case RDT_RESOURCE_PERF_PKG : +return RFTYPE_RES_PERF_PKG ; }return WARN_ON_ONCE (1 );
Signature · 1 line
2.51.0
Tony Luck
maintainerPATCH v11 14/31 x86/resctrl: Discover hardware telemetry events
In reply to Tony LuckEach CPU collects data for telemetry events that it sends to the nearest telemetry event aggregator either when the value of IA32_PQR_ASSOC.RMID changes, or when a two millisecond timer expires.
The telemetry event aggregators maintain per-RMID per-event counts of the total seen for all the CPUs. There may be more than one set of telemetry event aggregators per package.
There are separate sets of aggregators for each type of event, but all aggregators for a given type are symmetric keeping counts for the same set of events for the CPUs that provide data to them.
Each telemetry event aggregator is responsible for a specific group of events. E.g. on the Intel Clearwater Forest CPU there are two types of aggregators. One type tracks a pair of energy related events. The other type tracks a subset of "perf" type events.
The event counts are made available to Linux in a region of MMIO space for each aggregator. All details about the layout of counters in each aggregator MMIO region are described in XML files published by Intel and made available in a GitHub repository [1].
The key to matching a specific telemetry aggregator to the XML file that describes the MMIO layout is a 32-bit value. The Linux telemetry subsystem refers to this as a "guid" while the XML files call it a "uniqueid".
Each XML file provides the following information:
1) Which telemetry events are included in the group.
2) The order in which the event counters appear for each RMID.
3) The value type of each event counter (integer or fixed-point).
4) The number of RMIDs supported.
5) Which additional aggregator status registers are included.
6) The total size of the MMIO region for an aggregator.
The INTEL_PMT_TELEMETRY driver enumerates support for telemetry events. This driver provides intel_pmt_get_regions_by_feature() to list all available telemetry event aggregators. The list includes the "guid", the base address in MMIO space for the region where the event counters are exposed, and the package id where the all the CPUs that report to this aggregator are located.
Add a new Kconfig option CONFIG_X86_CPU_RESCTRL_INTEL_AET for the Intel specific parts of telemetry code. This depends on the INTEL_PMT_TELEMETRY and INTEL_TPMI drivers being built-in to the kernel for enumeration of telemetry features.
Use INTEL_PMT_TELEMETRY's intel_pmt_get_regions_by_feature() with each per-RMID telemetry feature id to obtain a private copy of struct pmt_feature_group that contains all discovered/enumerated telemetry aggregator data for all event groups (known and unknown to resctrl) of that feature id. Further processing on this structure will enable all supported events in resctrl. Return the structure to INTEL_PMT_TELEMETRY at resctrl exit time.
---
Note that checkpatch complains about this:
DEFINE_FREE(intel_pmt_put_feature_group, struct pmt_feature_group *,
if (!IS_ERR_OR_NULL(_T)) intel_pmt_put_feature_group(_T))
with:
CHECK: Alignment should match open parenthesis
But if the alignment is fixed, it then complains:
WARNING: Statements should start on a tabstop
--- arch/x86/kernel/cpu/resctrl/internal.h | 8 ++ arch/x86/kernel/cpu/resctrl/core.c | 5 + arch/x86/kernel/cpu/resctrl/intel_aet.c | 144 ++++++++++++++++++++++++ arch/x86/Kconfig | 13 +++ arch/x86/kernel/cpu/resctrl/Makefile | 1 + 5 files changed, 171 insertions(+) create mode 100644 arch/x86/kernel/cpu/resctrl/intel_aet.c
arch/x86/kernel/cpu/resctrl/internal.h +8 -0
@@ -217,4 +217,12 @@ void __init intel_rdt_mbm_apply_quirk(void);void rdt_domain_reconfigure_cdp (struct rdt_resource *r);void resctrl_arch_mbm_cntr_assign_set_one (struct rdt_resource *r); +#ifdef CONFIG_X86_CPU_RESCTRL_INTEL_AET +bool intel_aet_get_events (void ); +void __exit intel_aet_exit(void); +#else +static inline bool intel_aet_get_events (void ) {return false; } +static inline void __exit intel_aet_exit(void) { } +#endif +#endif /* _ASM_X86_RESCTRL_INTERNAL_H */
arch/x86/kernel/cpu/resctrl/core.c +5 -0
@@ -734,6 +734,9 @@ void resctrl_arch_pre_mount(void)if (!atomic_try_cmpxchg (&only_once, &old,1 ))return ; + +if (!intel_aet_get_events ()) +return ; }enum {
@@ -1091,6 +1094,8 @@ late_initcall(resctrl_arch_late_init); static void __exit resctrl_arch_exit(void) { +intel_aet_exit (); +cpuhp_remove_state (rdt_online);resctrl_exit ();
arch/x86/kernel/cpu/resctrl/intel_aet.c new file +144 -0
144 lines not on this page · raw message
arch/x86/Kconfig +13 -0
@@ -525,6 +525,19 @@ config X86_CPU_RESCTRL Say N if unsure. +config X86_CPU_RESCTRL_INTEL_AET + bool "Intel Application Energy Telemetry" + depends on X86_CPU_RESCTRL && CPU_SUP_INTEL && INTEL_PMT_TELEMETRY=y && INTEL_TPMI=y + help + Enable per-RMID telemetry events in resctrl. + + Intel feature that collects per-RMID execution data + about energy consumption, measure of frequency independent + activity and other performance metrics. Data is aggregated + per package. + + Say N if unsure. + config X86_FRED bool "Flexible Return and Event Delivery" depends on X86_64
arch/x86/kernel/cpu/resctrl/Makefile +1 -0
@@ -1,6 +1,7 @@ # SPDX-License-Identifier: GPL-2.0 obj-$(CONFIG_X86_CPU_RESCTRL) += core.o rdtgroup.o monitor.o obj-$(CONFIG_X86_CPU_RESCTRL) += ctrlmondata.o +obj-$(CONFIG_X86_CPU_RESCTRL_INTEL_AET) += intel_aet.o obj-$(CONFIG_RESCTRL_FS_PSEUDO_LOCK) += pseudo_lock.o # To allow define_trace.h's recursive include:
Signature · 1 line
2.51.0
Reinette Chatre
maintainerRe: [PATCH v11 14/31] x86/resctrl: Discover hardware telemetry events
Hi Tony,
Quoted 2 lines · Tony Luck
Each CPU collects data for telemetry events that it sends to the nearest telemetry event aggregator either when the value of IA32_PQR_ASSOC.RMID
Please note that one of the "Touchups" done during merge of [1] was to use full names for registers in descriptions. Considering this, "IA32_PQR_ASSOC.RMID" -> "MSR_IA32_PQR_ASSOC.RMID
(also please make same change in cover letter)
Quoted 2 lines
changes, or when a two millisecond timer expires.
...
Quoted 18 lines
arch/x86/kernel/cpu/resctrl/intel_aet.c new file +18
-0+ +/** +* struct event_group - All information about a group of telemetry events. +* @pfg: Points to the aggregated telemetry space information +* returned by the intel_pmt_get_regions_by_feature() +* call to the INTEL_PMT_TELEMETRY driver that contains +* data for all telemetry regions of a specific type. +* Valid if the system supports the event group. +* NULL otherwise. +* @guid: Unique number per XML description file. +*/ +struct event_group { +/* Data fields for additional structures to manage this group. */ +struct pmt_feature_group *pfg; + +/* Remaining fields initialized from XML file. */ +u32 guid; +};
...
arch/x86/kernel/cpu/resctrl/intel_aet.c line 89
Quoted 7 lines
arch/x86/kernel/cpu/resctrl/intel_aet.c new file +7
-0+ +/* +* Make a request to the INTEL_PMT_TELEMETRY driver for a copy of the +* pmt_feature_group for a specific feature. If there is one, the returned +* structure has an array of telemetry_region structures. Each describes +* one telemetry aggregator. +* Try to use every telemetry aggregator with a known guid.
The guid is associated with struct event_group and every telemetry region has its own guid. It is not clear to me why the guid is not associated with pmt_feature_group. To me this implies that a pmt_feature_group my contain telemetry regions that have different guid.
This is not fully apparent in this patch but as this code evolves I do not think the scenario where telemetry regions have different supported (by resctrl) guid is handled by this enumeration.
If I understand correctly, all telemetry regions of a given pmt_feature_group will be matched against a single supported guid at a time and all telemetry regions with that guid will be considered usable and any other considered unusable without further processing of that pmt_feature_group. If there are more than one matching guid supported by resctrl then only events of the first one will be enumerated?
Quoted 13 lines
+ */
arch/x86/kernel/cpu/resctrl/intel_aet.c new file +12
-0+static bool get_pmt_feature (enum pmt_feature_id feature,struct event_group **evgs, +unsigned int num_evg) +{ + struct pmt_feature_group *p __free(intel_pmt_put_feature_group) = NULL; +struct event_group **peg; +bool ret; + + p =intel_pmt_get_regions_by_feature (feature); + +if (IS_ERR_OR_NULL (p)) +return false; +
arch/x86/kernel/cpu/resctrl/intel_aet.c line 112
Quoted 10 lines
arch/x86/kernel/cpu/resctrl/intel_aet.c new file +10
-0+for (peg = evgs; peg < &evgs[num_evg]; peg++) { + ret =enable_events (*peg, p); +if (ret) { + (*peg)->pfg =no_free_ptr (p); +return true; + } + } + +return false; +}
Reinette
[1] https://lore.kernel.org/all/175793566119.709179.8448328033383658699.tip-bot2@tip-bot2/
Luck, Tony
maintainerRe: [PATCH v11 14/31] x86/resctrl: Discover hardware telemetry events
Quoted 11 lines · Reinette Chatre
Hi Tony,
Quoted 2 lines · Tony Luck
Each CPU collects data for telemetry events that it sends to the nearest telemetry event aggregator either when the value of IA32_PQR_ASSOC.RMID
Please note that one of the "Touchups" done during merge of [1] was to use full names for registers in descriptions. Considering this, "IA32_PQR_ASSOC.RMID" -> "MSR_IA32_PQR_ASSOC.RMID
(also please make same change in cover letter)
Will do.
Quoted 65 lines
Quoted 3 lines
changes, or when a two millisecond timer expires.
...
Quoted 18 lines
arch/x86/kernel/cpu/resctrl/intel_aet.c new file +18
-0+ +/** +* struct event_group - All information about a group of telemetry events. +* @pfg: Points to the aggregated telemetry space information +* returned by the intel_pmt_get_regions_by_feature() +* call to the INTEL_PMT_TELEMETRY driver that contains +* data for all telemetry regions of a specific type. +* Valid if the system supports the event group. +* NULL otherwise. +* @guid: Unique number per XML description file. +*/ +struct event_group { +/* Data fields for additional structures to manage this group. */ +struct pmt_feature_group *pfg; + +/* Remaining fields initialized from XML file. */ +u32 guid; +};...
arch/x86/kernel/cpu/resctrl/intel_aet.c line 89
Quoted 7 lines
arch/x86/kernel/cpu/resctrl/intel_aet.c new file +7
-0+ +/* +* Make a request to the INTEL_PMT_TELEMETRY driver for a copy of the +* pmt_feature_group for a specific feature. If there is one, the returned +* structure has an array of telemetry_region structures. Each describes +* one telemetry aggregator. +* Try to use every telemetry aggregator with a known guid. The guid is associated with struct
event_groupand every telemetry region has its own guid. It is not clear to me why the guid is not associated withpmt_feature_group. To me this implies that apmt_feature_groupmy contain telemetry regions that have different guid.This is not fully apparent in this patch but as this code evolves I do not think the scenario where telemetry regions have different supported (by resctrl) guid is handled by this enumeration.
If I understand correctly, all telemetry regions of a givenpmt_feature_groupwill be matched against a single supported guid at a time and all telemetry regions with that guid will be considered usable and any other considered unusable without further processing of thatpmt_feature_group. If there are more than one matching guid supported by resctrl then only events of the first one will be enumerated?Quoted 15 lines
+ */
arch/x86/kernel/cpu/resctrl/intel_aet.c new file +14
-0+static bool get_pmt_feature (enum pmt_feature_id feature,struct event_group **evgs, +unsigned int num_evg) +{ + struct pmt_feature_group *p __free(intel_pmt_put_feature_group) = NULL; +struct event_group **peg; +bool ret; + + p =intel_pmt_get_regions_by_feature (feature); + +if (IS_ERR_OR_NULL (p)) +return false; + +for (peg = evgs; peg < &evgs[num_evg]; peg++) { + ret =enable_events (*peg, p);
Quoted 8 lines
arch/x86/kernel/cpu/resctrl/intel_aet.c line 112
Quoted 8 lines
arch/x86/kernel/cpu/resctrl/intel_aet.c new file +8
-0+if (ret) { + (*peg)->pfg =no_free_ptr (p); +return true; + } + } + +return false; +}
Perhaps David wants to cope with a future system that supports multiple guids?
You are right that my code will not handle this. It will just enable the first recognised guid and ignore any others.
How about this. Take an extra reference on any pmt_feature_group structures that include a known guid (to keep the accounting right when intel_aet_exit() is called). This simplifies the function so I don't need the __free() handler that confuses checkpatch.pl :-)
/*
* Make a request to the INTEL_PMT_TELEMETRY driver for a copy of the
* pmt_feature_group for a specific feature. If there is one, the returned
* structure has an array of telemetry_region structures, each element of
* the array describes one telemetry aggregator.
* A single pmt_feature_group may include multiple different guids.
* Try to use every telemetry aggregator with a known guid.
*/
static bool get_pmt_feature(enum pmt_feature_id feature, struct event_group **evgs,
unsigned int num_evg)
{
struct pmt_feature_group *p = intel_pmt_get_regions_by_feature(feature);
struct event_group **peg;
bool ret = false;
if (IS_ERR_OR_NULL(p))
return false;
for (peg = evgs; peg < &evgs[num_evg]; peg++) {
if (enable_events(*peg, p)) {
kref_get(&p->kref);
(*peg)->pfg = no_free_ptr(p);
ret = true;
}
}
intel_pmt_put_feature_group(p);
return ret;
}Quoted 4 lines
Reinette
[1] https://lore.kernel.org/all/175793566119.709179.8448328033383658699.tip-bot2@tip-bot2/
Reinette Chatre
maintainerRe: [PATCH v11 14/31] x86/resctrl: Discover hardware telemetry events
Hi Tony,
Quoted 72 lines · Tony
Quoted 40 lines · Reinette Chatre
Hi Tony,
arch/x86/kernel/cpu/resctrl/intel_aet.c line 89
Quoted 7 lines · Tony Luck
arch/x86/kernel/cpu/resctrl/intel_aet.c new file +7
-0+ +/* +* Make a request to the INTEL_PMT_TELEMETRY driver for a copy of the +* pmt_feature_group for a specific feature. If there is one, the returned +* structure has an array of telemetry_region structures. Each describes +* one telemetry aggregator. +* Try to use every telemetry aggregator with a known guid. The guid is associated with struct
event_groupand every telemetry region has its own guid. It is not clear to me why the guid is not associated withpmt_feature_group. To me this implies that apmt_feature_groupmy contain telemetry regions that have different guid.This is not fully apparent in this patch but as this code evolves I do not think the scenario where telemetry regions have different supported (by resctrl) guid is handled by this enumeration.
If I understand correctly, all telemetry regions of a givenpmt_feature_groupwill be matched against a single supported guid at a time and all telemetry regions with that guid will be considered usable and any other considered unusable without further processing of thatpmt_feature_group. If there are more than one matching guid supported by resctrl then only events of the first one will be enumerated?Quoted 15 lines
+ */
arch/x86/kernel/cpu/resctrl/intel_aet.c new file +14
-0+static bool get_pmt_feature (enum pmt_feature_id feature,struct event_group **evgs, +unsigned int num_evg) +{ + struct pmt_feature_group *p __free(intel_pmt_put_feature_group) = NULL; +struct event_group **peg; +bool ret; + + p =intel_pmt_get_regions_by_feature (feature); + +if (IS_ERR_OR_NULL (p)) +return false; + +for (peg = evgs; peg < &evgs[num_evg]; peg++) { + ret =enable_events (*peg, p);Quoted 8 lines
arch/x86/kernel/cpu/resctrl/intel_aet.c line 112
Quoted 8 lines
arch/x86/kernel/cpu/resctrl/intel_aet.c new file +8
-0+if (ret) { + (*peg)->pfg =no_free_ptr (p); +return true; + } + } + +return false; +}Perhaps David wants to cope with a future system that supports multiple guids?
You are right that my code will not handle this. It will just enable the first recognised guid and ignore any others.
How about this. Take an extra reference on any
pmt_feature_groupstructures that include a known guid (to keep the accounting right whenintel_aet_exit()is called). This simplifies the function so I don't need the__free()handler that confuses checkpatch.pl :-)/*
* Make a request to theINTEL_PMT_TELEMETRYdriver for a copy of the
*pmt_feature_groupfor a specific feature. If there is one, the returned
* structure has an array oftelemetry_regionstructures, each element of
* the array describes one telemetry aggregator.
* A singlepmt_feature_groupmay include multiple different guids.
* Try to use every telemetry aggregator with a known guid.
*/
static bool get_pmt_feature(enumpmt_feature_idfeature, structevent_group**evgs,
unsigned intnum_evg)
Quoted 11 lines
{
struct pmt_feature_group *p = intel_pmt_get_regions_by_feature(feature);
struct event_group **peg;
bool ret = false;
if (IS_ERR_OR_NULL(p))
return false;
for (peg = evgs; peg < &evgs[num_evg]; peg++) {
if (enable_events(*peg, p)) {
kref_get(&p->kref);This is not clear to me ... would enable_events() still mark all telemetry_regions that do not match the event_group's guid as unusable? It seems to me that if more than one even_group refers to the same pmt_feature_group then the first one to match will "win" and make the other event_group's telemetry regions unusable.
Quoted 9 lines
(*peg)->pfg = no_free_ptr(p); ret = true; } } intel_pmt_put_feature_group(p); return ret; }
Reinette
Luck, Tony
maintainerRe: [PATCH v11 14/31] x86/resctrl: Discover hardware telemetry events
Quoted 83 lines · Reinette Chatre
Hi Tony,
Quoted 80 lines · Tony
Quoted 40 lines · Reinette Chatre
Hi Tony,
arch/x86/kernel/cpu/resctrl/intel_aet.c line 89
Quoted 7 lines · Tony Luck
arch/x86/kernel/cpu/resctrl/intel_aet.c new file +7
-0+ +/* +* Make a request to the INTEL_PMT_TELEMETRY driver for a copy of the +* pmt_feature_group for a specific feature. If there is one, the returned +* structure has an array of telemetry_region structures. Each describes +* one telemetry aggregator. +* Try to use every telemetry aggregator with a known guid. The guid is associated with struct
event_groupand every telemetry region has its own guid. It is not clear to me why the guid is not associated withpmt_feature_group. To me this implies that apmt_feature_groupmy contain telemetry regions that have different guid.This is not fully apparent in this patch but as this code evolves I do not think the scenario where telemetry regions have different supported (by resctrl) guid is handled by this enumeration.
If I understand correctly, all telemetry regions of a givenpmt_feature_groupwill be matched against a single supported guid at a time and all telemetry regions with that guid will be considered usable and any other considered unusable without further processing of thatpmt_feature_group. If there are more than one matching guid supported by resctrl then only events of the first one will be enumerated?Quoted 15 lines
+ */
arch/x86/kernel/cpu/resctrl/intel_aet.c new file +14
-0+static bool get_pmt_feature (enum pmt_feature_id feature,struct event_group **evgs, +unsigned int num_evg) +{ + struct pmt_feature_group *p __free(intel_pmt_put_feature_group) = NULL; +struct event_group **peg; +bool ret; + + p =intel_pmt_get_regions_by_feature (feature); + +if (IS_ERR_OR_NULL (p)) +return false; + +for (peg = evgs; peg < &evgs[num_evg]; peg++) { + ret =enable_events (*peg, p);Quoted 8 lines
arch/x86/kernel/cpu/resctrl/intel_aet.c line 112
Quoted 8 lines
arch/x86/kernel/cpu/resctrl/intel_aet.c new file +8
-0+if (ret) { + (*peg)->pfg =no_free_ptr (p); +return true; + } + } + +return false; +}Perhaps David wants to cope with a future system that supports multiple guids?
You are right that my code will not handle this. It will just enable the first recognised guid and ignore any others.
How about this. Take an extra reference on any
pmt_feature_groupstructures that include a known guid (to keep the accounting right whenintel_aet_exit()is called). This simplifies the function so I don't need the__free()handler that confuses checkpatch.pl :-)/*
* Make a request to theINTEL_PMT_TELEMETRYdriver for a copy of the
*pmt_feature_groupfor a specific feature. If there is one, the returned
* structure has an array oftelemetry_regionstructures, each element of
* the array describes one telemetry aggregator.
* A singlepmt_feature_groupmay include multiple different guids.
* Try to use every telemetry aggregator with a known guid.
*/
static bool get_pmt_feature(enumpmt_feature_idfeature, structevent_group**evgs,
unsigned intnum_evg){ struct pmt_feature_group *p = intel_pmt_get_regions_by_feature(feature); struct event_group **peg; bool ret = false; if (IS_ERR_OR_NULL(p)) return false;
Quoted 8 lines
Quoted 3 lines
for (peg = evgs; peg < &evgs[num_evg]; peg++) { if (enable_events(*peg, p)) { kref_get(&p->kref);This is not clear to me ... would
enable_events()still mark alltelemetry_regionsthat do not match the event_group's guid as unusable? It seems to me that if more than oneeven_grouprefers to the samepmt_feature_groupthen the first one to match will "win" and make the other event_group's telemetry regions unusable.
Extra context needed. Sorry.
I'm changing enable_events() to only mark telemetry_regions regions as unusable if they have a bad package id, or the MMIO size doesn't match. I.e. they truly are bad.
Mis-match on guid will skip then while associating with a specific event_gruoup, but leave them as usable.
This means that intel_aet_read_event() now has to check the guid as well as !addr.
An alternative approach would be to ask the PMT code for separate copies of the pmt_feature_group to attach to each event_group. I didn't like this, do you think it would be better?
Quoted 7 lines
Quoted 7 lines
(*peg)->pfg = no_free_ptr(p); ret = true; } } intel_pmt_put_feature_group(p);
Quoted 6 lines
Quoted 3 lines
return ret; }Reinette
-Tony
Luck, Tony
maintainerRe: [PATCH v11 14/31] x86/resctrl: Discover hardware telemetry events
Quoted 30 lines · Tony
Quoted 11 lines · Reinette Chatre
Hi Tony,
Quoted 10 lines
static bool get_pmt_feature(enum
pmt_feature_idfeature, structevent_group**evgs,
unsigned intnum_evg){ struct pmt_feature_group *p = intel_pmt_get_regions_by_feature(feature); struct event_group **peg; bool ret = false; if (IS_ERR_OR_NULL(p)) return false;Quoted 8 lines
Quoted 3 lines
for (peg = evgs; peg < &evgs[num_evg]; peg++) { if (enable_events(*peg, p)) { kref_get(&p->kref);This is not clear to me ... would
enable_events()still mark alltelemetry_regionsthat do not match the event_group's guid as unusable? It seems to me that if more than oneeven_grouprefers to the samepmt_feature_groupthen the first one to match will "win" and make the other event_group's telemetry regions unusable.Extra context needed. Sorry.
I'm changing
enable_events()to only marktelemetry_regionsregions as unusable if they have a bad package id, or the MMIO size doesn't match. I.e. they truly are bad.Mis-match on guid will skip then while associating with a specific
event_gruoup, but leave them as usable.
Quoted 6 lines
This means that
intel_aet_read_event()now has to check the guid as well as !addr.An alternative approach would be to ask the PMT code for separate copies of the
pmt_feature_groupto attach to eachevent_group. I didn't like this, do you think it would be better?
Working through more patches in the series, I've come to the one that adjusts the number of RMIDs. The alternative approach of having a separate copy of the pmt_feature_group is suddently looking more attractive.
So the code would become:
static bool get_pmt_feature(enum pmt_feature_id feature, struct event_group **evgs,
unsigned int num_evg)
{
struct pmt_feature_group *p;
struct event_group **peg;
bool ret = false;
for (peg = evgs; peg < &evgs[num_evg]; peg++) {
p = intel_pmt_get_regions_by_feature(feature);
if (IS_ERR_OR_NULL(p))
return false;
if (enable_events(*peg, p)) {
(*peg)->pfg = p;
ret = true;
} else {
intel_pmt_put_feature_group(p);
}
}
intel_pmt_put_feature_group(p);
return ret;
}Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
(*peg)->pfg = no_free_ptr(p); ret = true; } } intel_pmt_put_feature_group(p);
Quoted 8 lines
Quoted 6 lines
Quoted 3 lines
return ret; }Reinette
-Tony
-Tony
Reinette Chatre
maintainerRe: [PATCH v11 14/31] x86/resctrl: Discover hardware telemetry events
Hi Tony,
Quoted 34 lines · Tony
Quoted 33 lines · Tony
Quoted 11 lines · Reinette Chatre
Hi Tony,
Quoted 10 lines
static bool get_pmt_feature(enum
pmt_feature_idfeature, structevent_group**evgs,
unsigned intnum_evg){ struct pmt_feature_group *p = intel_pmt_get_regions_by_feature(feature); struct event_group **peg; bool ret = false; if (IS_ERR_OR_NULL(p)) return false;Quoted 8 lines
Quoted 3 lines
for (peg = evgs; peg < &evgs[num_evg]; peg++) { if (enable_events(*peg, p)) { kref_get(&p->kref);This is not clear to me ... would
enable_events()still mark alltelemetry_regionsthat do not match the event_group's guid as unusable? It seems to me that if more than oneeven_grouprefers to the samepmt_feature_groupthen the first one to match will "win" and make the other event_group's telemetry regions unusable.Extra context needed. Sorry.
I'm changing
enable_events()to only marktelemetry_regionsregions as unusable if they have a bad package id, or the MMIO size doesn't match. I.e. they truly are bad.Mis-match on guid will skip then while associating with a specific
event_gruoup, but leave them as usable.This means that
intel_aet_read_event()now has to check the guid as well as !addr.
Quoted 6 lines
Quoted 3 lines
An alternative approach would be to ask the PMT code for separate copies of the
pmt_feature_groupto attach to eachevent_group. I didn't like this, do you think it would be better?Working through more patches in the series, I've come to the one that adjusts the number of RMIDs. The alternative approach of
I see, with the number of RMIDs a property of the event group self this seems reasonable. While there is duplication of pmt_feature_group I am not able to tell if this is a big issue since I am not clear on how/if systems will be built this way.
Quoted 18 lines
having a separate copy of the
pmt_feature_groupis suddently looking more attractive.So the code would become:
static bool get_pmt_feature(enum
pmt_feature_idfeature, structevent_group**evgs,
unsigned intnum_evg){ struct pmt_feature_group *p; struct event_group **peg; bool ret = false; for (peg = evgs; peg < &evgs[num_evg]; peg++) { p = intel_pmt_get_regions_by_feature(feature); if (IS_ERR_OR_NULL(p)) return false;
Quoted 8 lines
if (enable_events(*peg, p)) {
(*peg)->pfg = p;
ret = true;
} else {
intel_pmt_put_feature_group(p);
}
}
intel_pmt_put_feature_group(p);I am not able to tell why this "put" is needed? I assume the "put" of a pmt_feature_group assigned to an event_group will still be done in intel_aet_exit()?
Reinette
Luck, Tony
maintainerRE: [PATCH v11 14/31] x86/resctrl: Discover hardware telemetry events
Quoted 21 lines
Quoted 12 lines
static bool get_pmt_feature(enum
pmt_feature_idfeature, structevent_group**evgs,
unsigned intnum_evg){ struct pmt_feature_group *p; struct event_group **peg; bool ret = false; for (peg = evgs; peg < &evgs[num_evg]; peg++) { p = intel_pmt_get_regions_by_feature(feature); if (IS_ERR_OR_NULL(p)) return false;Quoted 8 lines
if (enable_events(*peg, p)) { (*peg)->pfg = p; ret = true; } else { intel_pmt_put_feature_group(p); } } intel_pmt_put_feature_group(p);
Quoted 3 lines
I am not able to tell why this "put" is needed? I assume the "put" of a
pmt_feature_groupassigned to anevent_groupwill still be done inintel_aet_exit()?
Reinette
That "put" was left over from the previous version. You are right it isn't needed. The "put" will be done in intel_aet_exit()
-Tony
Tony Luck
maintainerPATCH v11 15/31 x86,fs/resctrl: Fill in details of events for guid 0x26696143 and 0x26557651
In reply to Tony LuckThe Intel Clearwater Forest CPU supports two RMID-based PMT feature groups documented in the xml/CWF/OOBMSM/RMID-ENERGY/cwf_aggregator.xml and xml/CWF/OOBMSM/RMID-PERF/cwf_aggregator.xml files in the Intel PMT GIT repository [1].
The counter offsets in MMIO space are arranged in groups for each RMID.
E.g the "energy" counters for guid 0x26696143 are arranged like this:
MMIO offset:0x0000 Counter for RMID 0 PMT_EVENT_ENERGY
MMIO offset:0x0008 Counter for RMID 0 PMT_EVENT_ACTIVITY
MMIO offset:0x0010 Counter for RMID 1 PMT_EVENT_ENERGY
MMIO offset:0x0018 Counter for RMID 1 PMT_EVENT_ACTIVITY
...
MMIO offset:0x23F0 Counter for RMID 575 PMT_EVENT_ENERGY
MMIO offset:0x23F8 Counter for RMID 575 PMT_EVENT_ACTIVITYAfter all counters there are three status registers that provide indications of how many times an aggregator was unable to process event counts, the time stamp for the most recent loss of data, and the time stamp of the most recent successful update.
MMIO offset:0x2400 AGG_DATA_LOSS_COUNT MMIO offset:0x2408 AGG_DATA_LOSS_TIMESTAMP MMIO offset:0x2410 LAST_UPDATE_TIMESTAMP
Define these events in the file system code and add the events to the event_group structures.
PMT_EVENT_ENERGY and PMT_EVENT_ACTIVITY are produced in fixed point format. File system code must output as floating point values.
--- include/linux/resctrl_types.h | 11 +++++++ arch/x86/kernel/cpu/resctrl/intel_aet.c | 43 +++++++++++++++++++++++++ fs/resctrl/monitor.c | 35 +++++++++++--------- 3 files changed, 74 insertions(+), 15 deletions(-)
include/linux/resctrl_types.h +11 -0
@@ -50,6 +50,17 @@ enum resctrl_event_id {QOS_L3_MBM_TOTAL_EVENT_ID =0x02 ,QOS_L3_MBM_LOCAL_EVENT_ID =0x03 , +/* Intel Telemetry Events */ +PMT_EVENT_ENERGY , +PMT_EVENT_ACTIVITY , +PMT_EVENT_STALLS_LLC_HIT , +PMT_EVENT_C1_RES , +PMT_EVENT_UNHALTED_CORE_CYCLES , +PMT_EVENT_STALLS_LLC_MISS , +PMT_EVENT_AUTO_C6_RES , +PMT_EVENT_UNHALTED_REF_CYCLES , +PMT_EVENT_UOPS_RETIRED , +/* Must be the last */ QOS_NUM_EVENTS, };
arch/x86/kernel/cpu/resctrl/intel_aet.c +43 -0
86 lines not on this page · raw message
fs/resctrl/monitor.c +20 -15
@@ -954,27 +954,32 @@ static void dom_data_exit(struct rdt_resource *r)mutex_unlock (&rdtgroup_mutex); } +#define MON_EVENT (_eventid, _name, _res, _fp) \ + [_eventid] = { \ + .name = _name, \ + .evtid = _eventid, \ + .rid = _res, \ + .is_floating_point = _fp, \ +} +/* * All available events. Architecture code marks the ones that * are supported by a system using resctrl_enable_mon_event() * to set .enabled. */ struct mon_evt mon_event_all[QOS_NUM_EVENTS ] = { - [QOS_L3_OCCUP_EVENT_ID ] = { - .name ="llc_occupancy" , - .evtid =QOS_L3_OCCUP_EVENT_ID , - .rid =RDT_RESOURCE_L3 , - }, - [QOS_L3_MBM_TOTAL_EVENT_ID ] = { - .name ="mbm_total_bytes" , - .evtid =QOS_L3_MBM_TOTAL_EVENT_ID , - .rid =RDT_RESOURCE_L3 , - }, - [QOS_L3_MBM_LOCAL_EVENT_ID ] = { - .name ="mbm_local_bytes" , - .evtid =QOS_L3_MBM_LOCAL_EVENT_ID , - .rid =RDT_RESOURCE_L3 , - }, +MON_EVENT (QOS_L3_OCCUP_EVENT_ID ,"llc_occupancy" ,RDT_RESOURCE_L3 , false), +MON_EVENT (QOS_L3_MBM_TOTAL_EVENT_ID ,"mbm_total_bytes" ,RDT_RESOURCE_L3 , false), +MON_EVENT (QOS_L3_MBM_LOCAL_EVENT_ID ,"mbm_local_bytes" ,RDT_RESOURCE_L3 , false), +MON_EVENT (PMT_EVENT_ENERGY ,"core_energy" ,RDT_RESOURCE_PERF_PKG , true), +MON_EVENT (PMT_EVENT_ACTIVITY ,"activity" ,RDT_RESOURCE_PERF_PKG , true), +MON_EVENT (PMT_EVENT_STALLS_LLC_HIT ,"stalls_llc_hit" ,RDT_RESOURCE_PERF_PKG , false), +MON_EVENT (PMT_EVENT_C1_RES ,"c1_res" ,RDT_RESOURCE_PERF_PKG , false), +MON_EVENT (PMT_EVENT_UNHALTED_CORE_CYCLES ,"unhalted_core_cycles" ,RDT_RESOURCE_PERF_PKG , false), +MON_EVENT (PMT_EVENT_STALLS_LLC_MISS ,"stalls_llc_miss" ,RDT_RESOURCE_PERF_PKG , false), +MON_EVENT (PMT_EVENT_AUTO_C6_RES ,"c6_res" ,RDT_RESOURCE_PERF_PKG , false), +MON_EVENT (PMT_EVENT_UNHALTED_REF_CYCLES ,"unhalted_ref_cycles" ,RDT_RESOURCE_PERF_PKG , false), +MON_EVENT (PMT_EVENT_UOPS_RETIRED ,"uops_retired" ,RDT_RESOURCE_PERF_PKG , false), };void resctrl_enable_mon_event (enum resctrl_event_id eventid,bool any_cpu,unsigned int binary_bits)
Signature · 1 line
2.51.0
Tony Luck
maintainerPATCH v11 16/31 x86,fs/resctrl: Add architectural event pointer
In reply to Tony LuckThe resctrl file system layer passes the domain, RMID, and event id to resctrl_arch_rmid_read() to fetch an event counter.
Fetching a telemetry event counter requires additional information that is private to the architecture, for example, the offset into MMIO space from where counter should be read.
Add mon_evt::arch_priv void pointer. Architecture code can initialize this when marking each event enabled.
File system code passes this pointer to resctrl_arch_rmid_read().
--- include/linux/resctrl.h | 7 +++++-- fs/resctrl/internal.h | 4 ++++ arch/x86/kernel/cpu/resctrl/core.c | 6 +++--- arch/x86/kernel/cpu/resctrl/monitor.c | 2 +- fs/resctrl/monitor.c | 18 ++++++++++++------ 5 files changed, 25 insertions(+), 12 deletions(-)
include/linux/resctrl.h +5 -2
@@ -412,7 +412,7 @@ u32 resctrl_arch_system_num_rmid_idx(void);int resctrl_arch_update_domains (struct rdt_resource *r,u32 closid);void resctrl_enable_mon_event (enum resctrl_event_id eventid,bool any_cpu, -unsigned int binary_bits); +unsigned int binary_bits,void *arch_priv);bool resctrl_is_mon_event_enabled (enum resctrl_event_id eventid);
@@ -529,6 +529,9 @@ void resctrl_arch_pre_mount(void);* only. * @rmid: rmid of the counter to read. * @eventid: eventid to read, e.g. L3 occupancy. +* @arch_priv: Architecture private data for this event. +* The @arch_priv provided by the architecture via +* resctrl_enable_mon_event(). * @val: result of the counter read in bytes. * @arch_mon_ctx: An architecture specific value from * resctrl_arch_mon_ctx_alloc(), for MPAM this identifies
@@ -546,7 +549,7 @@ void resctrl_arch_pre_mount(void);*/ int resctrl_arch_rmid_read (struct rdt_resource *r,struct rdt_domain_hdr *hdr,u32 closid,u32 rmid,enum resctrl_event_id eventid, -u64 *val,void *arch_mon_ctx); +void *arch_priv,u64 *val,void *arch_mon_ctx);/** * resctrl_arch_rmid_read_context_check() - warn about invalid contexts
fs/resctrl/internal.h +4 -0
@@ -66,6 +66,9 @@ static inline struct rdt_fs_context *rdt_fc2context(struct fs_context *fc)* @binary_bits: number of fixed-point binary bits from architecture, * only valid if @is_floating_point is true * @enabled: true if the event is enabled +* @arch_priv: Architecture private data for this event. +* The @arch_priv provided by the architecture via +* resctrl_enable_mon_event(). */ struct mon_evt {enum resctrl_event_id evtid;
@@ -77,6 +80,7 @@ struct mon_evt {bool is_floating_point;unsigned int binary_bits;bool enabled; +void *arch_priv; };extern struct mon_evt mon_event_all[QOS_NUM_EVENTS ];
arch/x86/kernel/cpu/resctrl/core.c +3 -3
@@ -911,15 +911,15 @@ static __init bool get_rdt_mon_resources(void)bool ret = false;if (rdt_cpu_has (X86_FEATURE_CQM_OCCUP_LLC )) { -resctrl_enable_mon_event (QOS_L3_OCCUP_EVENT_ID , false,0 ); +resctrl_enable_mon_event (QOS_L3_OCCUP_EVENT_ID , false,0 ,NULL ); ret = true; }if (rdt_cpu_has (X86_FEATURE_CQM_MBM_TOTAL )) { -resctrl_enable_mon_event (QOS_L3_MBM_TOTAL_EVENT_ID , false,0 ); +resctrl_enable_mon_event (QOS_L3_MBM_TOTAL_EVENT_ID , false,0 ,NULL ); ret = true; }if (rdt_cpu_has (X86_FEATURE_CQM_MBM_LOCAL )) { -resctrl_enable_mon_event (QOS_L3_MBM_LOCAL_EVENT_ID , false,0 ); +resctrl_enable_mon_event (QOS_L3_MBM_LOCAL_EVENT_ID , false,0 ,NULL ); ret = true; }if (rdt_cpu_has (X86_FEATURE_ABMC ))
arch/x86/kernel/cpu/resctrl/monitor.c +1 -1
@@ -240,7 +240,7 @@ static u64 get_corrected_val(struct rdt_resource *r, struct rdt_l3_mon_domain *dint resctrl_arch_rmid_read (struct rdt_resource *r,struct rdt_domain_hdr *hdr,u32 unused,u32 rmid,enum resctrl_event_id eventid, -u64 *val,void *ignored) +void *arch_priv,u64 *val,void *ignored) {struct rdt_l3_mon_domain *d;u64 msr_val;
fs/resctrl/monitor.c +12 -6
@@ -137,9 +137,11 @@ void __check_limbo(struct rdt_l3_mon_domain *d, bool force_free)struct rmid_entry *entry;u32 idx, cur_idx =1 ;void *arch_mon_ctx; +void *arch_priv;bool rmid_dirty;u64 val =0 ; + arch_priv = mon_event_all[QOS_L3_OCCUP_EVENT_ID ].arch_priv; arch_mon_ctx =resctrl_arch_mon_ctx_alloc (r,QOS_L3_OCCUP_EVENT_ID );if (IS_ERR (arch_mon_ctx)) { pr_warn_ratelimited("Failed to allocate monitor context: %ld",
@@ -160,7 +162,7 @@ void __check_limbo(struct rdt_l3_mon_domain *d, bool force_free) entry =__rmid_entry (idx);if (resctrl_arch_rmid_read (r, &d->hdr, entry->closid, entry->rmid, -QOS_L3_OCCUP_EVENT_ID , &val, +QOS_L3_OCCUP_EVENT_ID , arch_priv, &val, arch_mon_ctx)) { rmid_dirty = true; }else {
@@ -480,7 +482,8 @@ static int __mon_event_count(struct rdtgroup *rdtgrp, struct rmid_read *rr) rr->evt->evtid, &tval); else rr->err = resctrl_arch_rmid_read(rr->r, rr->hdr, closid, rmid, - rr->evt->evtid, &tval, rr->arch_mon_ctx); + rr->evt->evtid, rr->evt->arch_priv, + &tval, rr->arch_mon_ctx);if (rr->err)return rr->err;
@@ -505,7 +508,8 @@ static int __mon_event_count(struct rdtgroup *rdtgrp, struct rmid_read *rr) rr->evt->evtid, &tval); else err =resctrl_arch_rmid_read (rr->r, &d->hdr, closid, rmid, - rr->evt->evtid, &tval, rr->arch_mon_ctx); + rr->evt->evtid, rr->evt->arch_priv, + &tval, rr->arch_mon_ctx);if (!err) { rr->val += tval; ret =0 ;
@@ -982,7 +986,8 @@ struct mon_evt mon_event_all[QOS_NUM_EVENTS] = { MON_EVENT(PMT_EVENT_UOPS_RETIRED, "uops_retired", RDT_RESOURCE_PERF_PKG, false), }; -void resctrl_enable_mon_event (enum resctrl_event_id eventid,bool any_cpu,unsigned int binary_bits) +void resctrl_enable_mon_event (enum resctrl_event_id eventid,bool any_cpu, +unsigned int binary_bits,void *arch_priv) {if (WARN_ON_ONCE (eventid <QOS_FIRST_EVENT || eventid >=QOS_NUM_EVENTS || binary_bits >MAX_BINARY_BITS ))
@@ -998,6 +1003,7 @@ void resctrl_enable_mon_event(enum resctrl_event_id eventid, bool any_cpu, unsig mon_event_all[eventid].any_cpu = any_cpu; mon_event_all[eventid].binary_bits = binary_bits; + mon_event_all[eventid].arch_priv = arch_priv; mon_event_all[eventid].enabled = true; }
@@ -1823,9 +1829,9 @@ int resctrl_l3_mon_resource_init(void)if (r->mon.mbm_cntr_assignable) {if (!resctrl_is_mon_event_enabled (QOS_L3_MBM_TOTAL_EVENT_ID )) -resctrl_enable_mon_event (QOS_L3_MBM_TOTAL_EVENT_ID , false,0 ); +resctrl_enable_mon_event (QOS_L3_MBM_TOTAL_EVENT_ID , false,0 ,NULL );if (!resctrl_is_mon_event_enabled (QOS_L3_MBM_LOCAL_EVENT_ID )) -resctrl_enable_mon_event (QOS_L3_MBM_LOCAL_EVENT_ID , false,0 ); +resctrl_enable_mon_event (QOS_L3_MBM_LOCAL_EVENT_ID , false,0 ,NULL ); mon_event_all[QOS_L3_MBM_TOTAL_EVENT_ID ].evt_cfg = r->mon.mbm_cfg_mask; mon_event_all[QOS_L3_MBM_LOCAL_EVENT_ID].evt_cfg = r->mon.mbm_cfg_mask & (READS_TO_LOCAL_MEM |
Signature · 1 line
2.51.0
Reinette Chatre
maintainerRe: [PATCH v11 16/31] x86,fs/resctrl: Add architectural event pointer
Hi Tony,
Quoted 11 lines · Tony Luck
The resctrl file system layer passes the domain, RMID, and event id to
resctrl_arch_rmid_read()to fetch an event counter.Fetching a telemetry event counter requires additional information that is private to the architecture, for example, the offset into MMIO space from where counter should be read.
Add
mon_evt::arch_privvoid pointer. Architecture code can initialize this when marking each event enabled.File system code passes this pointer to
resctrl_arch_rmid_read().
A suggestion to avoid describing code that can be seen from the patch:
The resctrl file system layer passes the domain, RMID, and event id to the architecture to fetch an event counter.
Fetching a telemetry event counter requires additional information that is private to the architecture, for example, the offset into MMIO space from where the counter should be read.
Add mon_evt::arch_priv that architecture can use for any private data related to the event. resctrl filesystem initializes mon_evt::arch_priv when the architecture enables the event and passes it back to architecture when needing to fetch an event counter.
Quoted 4 lines
Suggested-by Reinette Chatre <[email protected]>Signed-off-by Tony Luck <[email protected]>---
Reinette
Tony Luck
maintainerPATCH v11 17/31 x86/resctrl: Find and enable usable telemetry events
In reply to Tony LuckThe INTEL_PMT_TELEMETRY driver provides telemetry region structures of the types requested by resctrl.
Scan these structures to discover which pass sanity checks to derive a list of valid regions:
1) They have guid known to resctrl.
2) They have a valid package ID.
3) The enumerated size of the MMIO region matches the expected value from the XML description file.
4) At least one region passes the above checks.
For each valid region enable all the events in the associated event_group::evts[].
Pass a pointer to the pmt_event structure of the event within the struct event_group that resctrl stores in mon_evt::arch_priv. resctrl passes this pointer back when asking to read the event data which enables the data to be found in MMIO.
--- arch/x86/kernel/cpu/resctrl/intel_aet.c | 38 +++++++++++++++++++++++-- 1 file changed, 36 insertions(+), 2 deletions(-)
arch/x86/kernel/cpu/resctrl/intel_aet.c +36 -2
@@ -20,9 +20,11 @@#include <linux/intel_pmt_features.h> #include <linux/intel_vsec.h> #include <linux/overflow.h> +#include <linux/printk.h> #include <linux/resctrl.h> #include <linux/resctrl_types.h> #include <linux/stddef.h> +#include <linux/topology.h> #include <linux/types.h> #include "internal.h"
@@ -114,12 +116,44 @@ static struct event_group *known_perf_event_groups[] = {for (_peg = (_grp); _peg < &_grp[ARRAY_SIZE (_grp)]; _peg++) \if ((*_peg)->pfg) -/* Stub for now */ -static bool enable_events(struct event_group *e, struct pmt_feature_group *p) +static bool skip_telem_region(struct telemetry_region *tr, struct event_group *e) { +if (tr->guid != e->guid) +return true; +if (tr->plat_info.package_id >=topology_max_packages ()) { +pr_warn ("Bad package %u in guid 0x%x\n" , tr->plat_info.package_id, + tr->guid); +return true; + } +if (tr->size != e->mmio_size) { +pr_warn ("MMIO space wrong size (%zu bytes) for guid 0x%x. Expected %zu bytes.\n" , + tr->size, e->guid, e->mmio_size); +return true; + } +return false; } +static bool enable_events (struct event_group *e,struct pmt_feature_group *p) +{ +bool usable_events = false; + +for (int i =0 ; i < p->count; i++) { +if (skip_telem_region (&p->regions[i], e)) +continue ; + usable_events = true; + } + +if (!usable_events) +return false; + +for (int j =0 ; j < e->num_events; j++) +resctrl_enable_mon_event (e->evts[j].id, true, + e->evts[j].bin_bits, &e->evts[j]); + +return true; +} + DEFINE_FREE(intel_pmt_put_feature_group, struct pmt_feature_group *, if (!IS_ERR_OR_NULL(_T))intel_pmt_put_feature_group (_T))
Signature · 1 line
2.51.0
Reinette Chatre
maintainerRe: [PATCH v11 17/31] x86/resctrl: Find and enable usable telemetry events
Hi Tony,
Quoted 5 lines · Tony Luck
The
INTEL_PMT_TELEMETRYdriver provides telemetry region structures of the types requested by resctrl.Scan these structures to discover which pass sanity checks to derive a list of valid regions:
The "to derive a list of valid regions" does not align with the
"At least one region passes the above checks" requirement. If this is about valid (usable?) regions then I think (4) should be dropped. If this is instead about valid events then above should be reworded to say that instead.
Quoted 7 lines
1) They have guid known to resctrl.
2) They have a valid package ID.
3) The enumerated size of the MMIO region matches the expected value from the XML description file.
4) At least one region passes the above checks.
Everything below is clear by looking at the patch. It can also be seen from patch that enabling is done only once if there is *any* valid region instead of "for each valid region". One thing that may be useful to add is "why" all events can be enabled. If I understand correctly it can be something like:
Enable events that usable telemetry regions are responsible for.
Quoted 34 lines
For each valid region enable all the events in the associated event_group::evts[].
Pass a pointer to the
pmt_eventstructure of the event within the structevent_groupthat resctrl stores inmon_evt::arch_priv. resctrl passes this pointer back when asking to read the event data which enables the data to be found in MMIO.Signed-off-by Tony Luck <[email protected]>--- arch/x86/kernel/cpu/resctrl/intel_aet.c | 38 +++++++++++++++++++++++-- 1 file changed, 36 insertions(+), 2 deletions(-)arch/x86/kernel/cpu/resctrl/intel_aet.c +2
-1@@ -20,9 +20,11 @@#include <linux/intel_pmt_features.h> #include <linux/intel_vsec.h> #include <linux/overflow.h> +#include <linux/printk.h> #include <linux/resctrl.h> #include <linux/resctrl_types.h> #include <linux/stddef.h> +#include <linux/topology.h> #include <linux/types.h> #include "internal.h" @@ -114,12 +116,44 @@ static struct event_group *known_perf_event_groups[] = {for (_peg = (_grp); _peg < &_grp[ARRAY_SIZE (_grp)]; _peg++) \if ((*_peg)->pfg) -/* Stub for now */
arch/x86/kernel/cpu/resctrl/intel_aet.c static struct event_group *known_perf_event_groups[] = { · line 127
Quoted 10 lines
arch/x86/kernel/cpu/resctrl/intel_aet.c +8
-1-static bool enable_events (struct event_group *e,struct pmt_feature_group *p) +static bool skip_telem_region (struct telemetry_region *tr,struct event_group *e) { +if (tr->guid != e->guid) +return true; +if (tr->plat_info.package_id >=topology_max_packages ()) { +pr_warn ("Bad package %u in guid 0x%x\n" , tr->plat_info.package_id, + tr->guid); +return true; + }
I have not encountered any mention of the possibility that packages may differ in which telemetry region types they support. For example, could it be possible for package A to have usable regions of the PERF type but package B doesn't? From what I can tell INTEL_PMT_TELEMETRY supports layouts where this can be possible. If I understand correctly this implementation will create event files for these domains but when the user attempts to read the data it will fail. Can this work add some snippet about possibility of this scenario and if/how it is supported?
Quoted 7 lines
arch/x86/kernel/cpu/resctrl/intel_aet.c +6
-0+if (tr->size != e->mmio_size) { +pr_warn ("MMIO space wrong size (%zu bytes) for guid 0x%x. Expected %zu bytes.\n" , + tr->size, e->guid, e->mmio_size); +return true; + } +return false;
arch/x86/kernel/cpu/resctrl/intel_aet.c static struct event_group *known_perf_event_groups[] = { · line 144
Quoted 10 lines
arch/x86/kernel/cpu/resctrl/intel_aet.c +8
-0} +static bool enable_events (struct event_group *e,struct pmt_feature_group *p) +{ +bool usable_events = false; + +for (int i =0 ; i < p->count; i++) { +if (skip_telem_region (&p->regions[i], e)) +continue ; + usable_events = true;
A previous concern [1] was why this loop does not break out at this point. I think it will help to make this clear if marking a telemetry region as unusable (mark_telem_region_unusable()) is done in this patch. Doing so makes the "usable" and "unusable" distinction in one patch while making clear that the loop needs to complete.
Quoted 5 lines
arch/x86/kernel/cpu/resctrl/intel_aet.c +5
-0+ } + +if (!usable_events) +return false; +
arch/x86/kernel/cpu/resctrl/intel_aet.c static struct event_group *known_perf_event_groups[] = { · line 159
Quoted 10 lines
arch/x86/kernel/cpu/resctrl/intel_aet.c +7
-0+for (int j =0 ; j < e->num_events; j++) +resctrl_enable_mon_event (e->evts[j].id, true, + e->evts[j].bin_bits, &e->evts[j]); + +return true; +} + DEFINE_FREE(intel_pmt_put_feature_group, struct pmt_feature_group *, if (!IS_ERR_OR_NULL(_T))intel_pmt_put_feature_group (_T))
Reinette
Luck, Tony
maintainerRe: [PATCH v11 17/31] x86/resctrl: Find and enable usable telemetry events
Quoted 6 lines · Reinette Chatre
Hi Tony,
Quoted 3 lines · Tony Luck
The
INTEL_PMT_TELEMETRYdriver provides telemetry region structures of the types requested by resctrl.
Quoted 7 lines
Quoted 2 lines
Scan these structures to discover which pass sanity checks to derive a list of valid regions:
The "to derive a list of valid regions" does not align with the
"At least one region passes the above checks" requirement. If this is about valid (usable?) regions then I think (4) should be dropped. If this is instead about valid events then above should be reworded to say that instead.
Will drop "4".
Quoted 9 lines
Quoted 8 lines
1) They have guid known to resctrl.
2) They have a valid package ID.
3) The enumerated size of the MMIO region matches the expected value from the XML description file.
4) At least one region passes the above checks.
Quoted 6 lines
Everything below is clear by looking at the patch. It can also be seen from patch that enabling is done only once if there is *any* valid region instead of "for each valid region". One thing that may be useful to add is "why" all events can be enabled. If I understand correctly it can be something like:
Enable events that usable telemetry regions are responsible for.
Looks better. I will use this.
Quoted 46 lines
Quoted 35 lines
For each valid region enable all the events in the associated event_group::evts[].
Pass a pointer to the
pmt_eventstructure of the event within the structevent_groupthat resctrl stores inmon_evt::arch_priv. resctrl passes this pointer back when asking to read the event data which enables the data to be found in MMIO.Signed-off-by Tony Luck <[email protected]>--- arch/x86/kernel/cpu/resctrl/intel_aet.c | 38 +++++++++++++++++++++++-- 1 file changed, 36 insertions(+), 2 deletions(-)arch/x86/kernel/cpu/resctrl/intel_aet.c +2
-1@@ -20,9 +20,11 @@#include <linux/intel_pmt_features.h> #include <linux/intel_vsec.h> #include <linux/overflow.h> +#include <linux/printk.h> #include <linux/resctrl.h> #include <linux/resctrl_types.h> #include <linux/stddef.h> +#include <linux/topology.h> #include <linux/types.h> #include "internal.h" @@ -114,12 +116,44 @@ static struct event_group *known_perf_event_groups[] = {for (_peg = (_grp); _peg < &_grp[ARRAY_SIZE (_grp)]; _peg++) \if ((*_peg)->pfg) -/* Stub for now */ arch/x86/kernel/cpu/resctrl/intel_aet.c static struct event_group *known_perf_event_groups[] = { · line 127
Quoted 10 lines
arch/x86/kernel/cpu/resctrl/intel_aet.c +8
-1-static bool enable_events (struct event_group *e,struct pmt_feature_group *p) +static bool skip_telem_region (struct telemetry_region *tr,struct event_group *e) { +if (tr->guid != e->guid) +return true; +if (tr->plat_info.package_id >=topology_max_packages ()) { +pr_warn ("Bad package %u in guid 0x%x\n" , tr->plat_info.package_id, + tr->guid); +return true; + }
Quoted 7 lines
I have not encountered any mention of the possibility that packages may differ in which telemetry region types they support. For example, could it be possible for package A to have usable regions of the PERF type but package B doesn't? From what I can tell
INTEL_PMT_TELEMETRYsupports layouts where this can be possible. If I understand correctly this implementation will create event files for these domains but when the user attempts to read the data it will fail. Can this work add some snippet about possibility of this scenario and if/how it is supported?
Yes, this is architecturally possible. But I do not expect that systems will be built that do this. You are right that such a system will create files that always return "Unavailable" when read.
Is it sufficient to document this in the commit message?
I don't feel that it would be worthwhile to suppress creation of these files for a "can't happen" situation. I'm not sure that doing so would be significantly better from a user interface perspective. Users would get slightly more notice (-ENOENT when trying to open the file). But the code would require architecture calls from file system code to check which files need to be created separately for each domain.
Quoted 10 lines
Quoted 10 lines
arch/x86/kernel/cpu/resctrl/intel_aet.c +6
-0+if (tr->size != e->mmio_size) { +pr_warn ("MMIO space wrong size (%zu bytes) for guid 0x%x. Expected %zu bytes.\n" , + tr->size, e->guid, e->mmio_size); +return true; + } +return false; }
Quoted 13 lines
arch/x86/kernel/cpu/resctrl/intel_aet.c static struct event_group *known_perf_event_groups[] = { · line 144
Quoted 8 lines
arch/x86/kernel/cpu/resctrl/intel_aet.c +8
-0+static bool enable_events (struct event_group *e,struct pmt_feature_group *p) +{ +bool usable_events = false; + +for (int i =0 ; i < p->count; i++) { +if (skip_telem_region (&p->regions[i], e)) +continue ; + usable_events = true;A previous concern [1] was why this loop does not break out at this point. I think it will help to make this clear if marking a telemetry region as unusable (
mark_telem_region_unusable()) is done in this patch. Doing so makes the "usable" and "unusable" distinction in one patch while making clear that the loop needs to complete.
Ok. I'll pull mark_telem_region_unusable() into this patch.
Quoted 18 lines
Quoted 5 lines
arch/x86/kernel/cpu/resctrl/intel_aet.c +5
-0+ } + +if (!usable_events) +return false; +arch/x86/kernel/cpu/resctrl/intel_aet.c static struct event_group *known_perf_event_groups[] = { · line 159
Quoted 10 lines
arch/x86/kernel/cpu/resctrl/intel_aet.c +7
-0+for (int j =0 ; j < e->num_events; j++) +resctrl_enable_mon_event (e->evts[j].id, true, + e->evts[j].bin_bits, &e->evts[j]); + +return true; +} + DEFINE_FREE(intel_pmt_put_feature_group, struct pmt_feature_group *, if (!IS_ERR_OR_NULL(_T))intel_pmt_put_feature_group (_T))Reinette
Quoted 1 line
-Tony
Reinette Chatre
maintainerRe: [PATCH v11 17/31] x86/resctrl: Find and enable usable telemetry events
Hi Tony,
Quoted 30 lines · Tony
Quoted 17 lines · Reinette Chatre
Quoted 5 lines · Tony Luck
arch/x86/kernel/cpu/resctrl/intel_aet.c +0
-1@@ -114,12 +116,44 @@ static struct event_group *known_perf_event_groups[] = {for (_peg = (_grp); _peg < &_grp[ARRAY_SIZE (_grp)]; _peg++) \if ((*_peg)->pfg) -/* Stub for now */ arch/x86/kernel/cpu/resctrl/intel_aet.c static struct event_group *known_perf_event_groups[] = { · line 127
Quoted 10 lines
arch/x86/kernel/cpu/resctrl/intel_aet.c +8
-1-static bool enable_events (struct event_group *e,struct pmt_feature_group *p) +static bool skip_telem_region (struct telemetry_region *tr,struct event_group *e) { +if (tr->guid != e->guid) +return true; +if (tr->plat_info.package_id >=topology_max_packages ()) { +pr_warn ("Bad package %u in guid 0x%x\n" , tr->plat_info.package_id, + tr->guid); +return true; + }Quoted 7 lines
I have not encountered any mention of the possibility that packages may differ in which telemetry region types they support. For example, could it be possible for package A to have usable regions of the PERF type but package B doesn't? From what I can tell
INTEL_PMT_TELEMETRYsupports layouts where this can be possible. If I understand correctly this implementation will create event files for these domains but when the user attempts to read the data it will fail. Can this work add some snippet about possibility of this scenario and if/how it is supported?Yes, this is architecturally possible. But I do not expect that systems will be built that do this. You are right that such a system will create files that always return "Unavailable" when read.
Quoted 8 lines
Is it sufficient to document this in the commit message?
I don't feel that it would be worthwhile to suppress creation of these files for a "can't happen" situation. I'm not sure that doing so would be significantly better from a user interface perspective. Users would get slightly more notice (
-ENOENTwhen trying to open the file). But the code would require architecture calls from file system code to check which files need to be created separately for each domain.
I think it is sufficient to document this in the commit message to help create confidence in robustness in support of different scenarios. I have not encountered such a system but could this scenario be similar to one where a two socket system supports MBM but only one socket has memory populated? I do not know what reading the counter MSR will return in this case though.
Quoted 25 lines
Quoted 11 lines
Quoted 11 lines
arch/x86/kernel/cpu/resctrl/intel_aet.c +6
-0+if (tr->size != e->mmio_size) { +pr_warn ("MMIO space wrong size (%zu bytes) for guid 0x%x. Expected %zu bytes.\n" , + tr->size, e->guid, e->mmio_size); +return true; + } +return false; }Quoted 13 lines
arch/x86/kernel/cpu/resctrl/intel_aet.c static struct event_group *known_perf_event_groups[] = { · line 144
Quoted 8 lines
arch/x86/kernel/cpu/resctrl/intel_aet.c +8
-0+static bool enable_events (struct event_group *e,struct pmt_feature_group *p) +{ +bool usable_events = false; + +for (int i =0 ; i < p->count; i++) { +if (skip_telem_region (&p->regions[i], e)) +continue ; + usable_events = true;A previous concern [1] was why this loop does not break out at this point. I think it will help to make this clear if marking a telemetry region as unusable (
mark_telem_region_unusable()) is done in this patch. Doing so makes the "usable" and "unusable" distinction in one patch while making clear that the loop needs to complete.
Quoted 1 line
Ok. I'll pull
mark_telem_region_unusable()into this patch.
Thank you.
Reinette
Luck, Tony
maintainerRE: [PATCH v11 17/31] x86/resctrl: Find and enable usable telemetry events
Quoted 7 lines
Quoted 6 lines
I don't feel that it would be worthwhile to suppress creation of these files for a "can't happen" situation. I'm not sure that doing so would be significantly better from a user interface perspective. Users would get slightly more notice (
-ENOENTwhen trying to open the file). But the code would require architecture calls from file system code to check which files need to be created separately for each domain.
Quoted 5 lines
I think it is sufficient to document this in the commit message to help create confidence in robustness in support of different scenarios. I have not encountered such a system but could this scenario be similar to one where a two socket system supports MBM but only one socket has memory populated? I do not know what reading the counter MSR will return in this case though.
The counting h/w is likely unaware of whether DIMMs slots are populated or not. My guess would be that in this case the counters would read as zero forever, rather than "unavailable".
CXL supports hot-add memory. So whether a node has memory could change at runtime on a system with CXL support.
-Tony
Tony Luck
maintainerPATCH v11 18/31 fs/resctrl: Refactor L3 specific parts of __mon_event_count()
In reply to Tony LuckThe "MBM counter assignment" and "reset counter on first read" features are only applicable to the RDT_RESOURCE_L3 resource.
Add a check for the RDT_RESOURCE_L3 resource.
--- fs/resctrl/monitor.c | 38 ++++++++++++++++++++------------------ 1 file changed, 20 insertions(+), 18 deletions(-)
fs/resctrl/monitor.c +20 -18
@@ -453,27 +453,29 @@ static int __mon_event_count(struct rdtgroup *rdtgrp, struct rmid_read *rr)if (!cpu_on_correct_domain (rr))return -EINVAL ; -if (!domain_header_is_valid (rr->hdr,RESCTRL_MON_DOMAIN ,RDT_RESOURCE_L3 )) -return -EINVAL ; - d = container_of(rr->hdr, struct rdt_l3_mon_domain, hdr); - -if (rr->is_mbm_cntr) { - cntr_id =mbm_cntr_get (rr->r, d, rdtgrp, rr->evt->evtid); -if (cntr_id <0 ) { - rr->err = -ENOENT ; +if (rr->r->rid ==RDT_RESOURCE_L3 ) { +if (!domain_header_is_valid (rr->hdr,RESCTRL_MON_DOMAIN ,RDT_RESOURCE_L3 ))return -EINVAL ; + d = container_of(rr->hdr, struct rdt_l3_mon_domain, hdr); + +if (rr->is_mbm_cntr) { + cntr_id =mbm_cntr_get (rr->r, d, rdtgrp, rr->evt->evtid); +if (cntr_id <0 ) { + rr->err = -ENOENT ; +return -EINVAL ; + } } - } -if (rr->first) { -if (rr->is_mbm_cntr) -resctrl_arch_reset_cntr (rr->r, d, closid, rmid, cntr_id, rr->evt->evtid); -else -resctrl_arch_reset_rmid (rr->r, d, closid, rmid, rr->evt->evtid); - m =get_mbm_state (d, closid, rmid, rr->evt->evtid); -if (m) -memset (m,0 ,sizeof (struct mbm_state )); -return 0 ; +if (rr->first) { +if (rr->is_mbm_cntr) +resctrl_arch_reset_cntr (rr->r, d, closid, rmid, cntr_id, rr->evt->evtid); +else +resctrl_arch_reset_rmid (rr->r, d, closid, rmid, rr->evt->evtid); + m =get_mbm_state (d, closid, rmid, rr->evt->evtid); +if (m) +memset (m,0 ,sizeof (struct mbm_state )); +return 0 ; + } }if (rr->hdr) {
Signature · 1 line
2.51.0
Reinette Chatre
maintainerRe: [PATCH v11 18/31] fs/resctrl: Refactor L3 specific parts of __mon_event_count()
Hi Tony,
Quoted 4 lines · Tony Luck
The "MBM counter assignment" and "reset counter on first read" features are only applicable to the
RDT_RESOURCE_L3resource.Add a check for the
RDT_RESOURCE_L3resource.
Why?
"Add a check for the RDT_RESOURCE_L3 resource" can be seen from code.
Quoted 15 lines
Signed-off-by Tony Luck <[email protected]>--- fs/resctrl/monitor.c | 38 ++++++++++++++++++++------------------ 1 file changed, 20 insertions(+), 18 deletions(-)fs/resctrl/monitor.c +0
-1@@ -453,27 +453,29 @@ static int __mon_event_count(struct rdtgroup *rdtgrp, struct rmid_read *rr)if (!cpu_on_correct_domain (rr))return -EINVAL ; -if (!domain_header_is_valid (rr->hdr,RESCTRL_MON_DOMAIN ,RDT_RESOURCE_L3 ))
fs/resctrl/monitor.c static int __mon_event_count(struct rdtgroup *rdtgrp, struct rmid_read *rr) · line 458
Quoted 10 lines
fs/resctrl/monitor.c +2
-7-return -EINVAL ; - d = container_of(rr->hdr, struct rdt_l3_mon_domain, hdr); - -if (rr->is_mbm_cntr) { - cntr_id =mbm_cntr_get (rr->r, d, rdtgrp, rr->evt->evtid); -if (cntr_id <0 ) { - rr->err = -ENOENT ; +if (rr->r->rid ==RDT_RESOURCE_L3 ) { +if (!domain_header_is_valid (rr->hdr,RESCTRL_MON_DOMAIN ,RDT_RESOURCE_L3 ))return -EINVAL ;
This snippet keeps moving around but it remains problematic from what I can tell since rr->hdr can still be NULL here.
Quoted 21 lines
fs/resctrl/monitor.c +9
-10+ d = container_of(rr->hdr, struct rdt_l3_mon_domain, hdr); + +if (rr->is_mbm_cntr) { + cntr_id =mbm_cntr_get (rr->r, d, rdtgrp, rr->evt->evtid); +if (cntr_id <0 ) { + rr->err = -ENOENT ; +return -EINVAL ; + } } - } -if (rr->first) { -if (rr->is_mbm_cntr) -resctrl_arch_reset_cntr (rr->r, d, closid, rmid, cntr_id, rr->evt->evtid); -else -resctrl_arch_reset_rmid (rr->r, d, closid, rmid, rr->evt->evtid); - m =get_mbm_state (d, closid, rmid, rr->evt->evtid); -if (m) -memset (m,0 ,sizeof (struct mbm_state )); -return 0 ; +if (rr->first) {
fs/resctrl/monitor.c static int __mon_event_count(struct rdtgroup *rdtgrp, struct rmid_read *rr) · line 479
Quoted 10 lines
fs/resctrl/monitor.c +9
-0+if (rr->is_mbm_cntr) +resctrl_arch_reset_cntr (rr->r, d, closid, rmid, cntr_id, rr->evt->evtid); +else +resctrl_arch_reset_rmid (rr->r, d, closid, rmid, rr->evt->evtid); + m =get_mbm_state (d, closid, rmid, rr->evt->evtid); +if (m) +memset (m,0 ,sizeof (struct mbm_state )); +return 0 ; + } }
__mon_event_count() is now unreasonably complicated. One motivation from changelog is because "MBM counter assignment is only applicable to RDT_RESOURCE_L3" but then the change proceeds to only add a RDT_RESOURCE_L3 check in some parts of __mon_event_count() that deals with counter assignment and leaves other parts to rely on rmid_read properties. It also fails to mention or make explicit that all the special SNC handling is only applicable to L3 and leaves that code to seemingly be generic and reader needs to be very familiar with the code to know differently. I find SNC already a challenge and find this to increase that complexity unnecessarily.
PERF_PKG only needs to reach a handful of lines in __mon_event_count() to read a counter. I thus think the best way forward is to split __mon_event_count() and move all the L3 code into its own function.
Reinette
Tony Luck
maintainerPATCH v11 19/31 x86/resctrl: Read telemetry events
In reply to Tony LuckTelemetry events are enabled during the first mount of the resctrl file system.
Mark telemetry regions that did not pass the sanity checks by clearing their MMIO address fields so that they will not be used when reading events.
Introduce intel_aet_read_event() to read telemetry events for resource RDT_RESOURCE_PERF_PKG. There may be multiple aggregators tracking each package, so scan all of them and add up all counters. Aggregators may return an invalid data indication if they have received no records for a given RMID. Return success to the user if one or more aggregators provide valid data.
Resctrl now uses readq() so depends on X86_64. Update Kconfig.
--- arch/x86/kernel/cpu/resctrl/internal.h | 7 +++ arch/x86/kernel/cpu/resctrl/intel_aet.c | 65 ++++++++++++++++++++++++- arch/x86/kernel/cpu/resctrl/monitor.c | 3 ++ arch/x86/Kconfig | 2 +- 4 files changed, 75 insertions(+), 2 deletions(-)
arch/x86/kernel/cpu/resctrl/internal.h +7 -0
@@ -220,9 +220,16 @@ void resctrl_arch_mbm_cntr_assign_set_one(struct rdt_resource *r);#ifdef CONFIG_X86_CPU_RESCTRL_INTEL_AET bool intel_aet_get_events (void ); void __exit intel_aet_exit(void); +int intel_aet_read_event (int domid,u32 rmid,enum resctrl_event_id evtid, +void *arch_priv,u64 *val);#else static inline bool intel_aet_get_events (void ) {return false; } static inline void __exit intel_aet_exit(void) { } +static inline int intel_aet_read_event (int domid,u32 rmid,enum resctrl_event_id evtid, +void *arch_priv,u64 *val) +{ +return -EINVAL ; +}#endif #endif /* _ASM_X86_RESCTRL_INTERNAL_H */
arch/x86/kernel/cpu/resctrl/intel_aet.c +64 -1
93 lines not on this page · raw message
arch/x86/kernel/cpu/resctrl/monitor.c +3 -0
@@ -250,6 +250,9 @@ int resctrl_arch_rmid_read(struct rdt_resource *r, struct rdt_domain_hdr *hdr,resctrl_arch_rmid_read_context_check (); +if (r->rid ==RDT_RESOURCE_PERF_PKG ) +return intel_aet_read_event (hdr->id, rmid, eventid, arch_priv, val); +if (!domain_header_is_valid (hdr,RESCTRL_MON_DOMAIN ,RDT_RESOURCE_L3 ))return -EINVAL ;
arch/x86/Kconfig +1 -1
@@ -527,7 +527,7 @@ config X86_CPU_RESCTRL config X86_CPU_RESCTRL_INTEL_AET bool "Intel Application Energy Telemetry" - depends on X86_CPU_RESCTRL && CPU_SUP_INTEL && INTEL_PMT_TELEMETRY=y && INTEL_TPMI=y + depends on X86_64 && X86_CPU_RESCTRL && CPU_SUP_INTEL && INTEL_PMT_TELEMETRY=y && INTEL_TPMI=y help Enable per-RMID telemetry events in resctrl.
Signature · 1 line
2.51.0
Tony Luck
maintainerPATCH v11 20/31 fs/resctrl: Refactor Sub-NUMA Cluster (SNC) in mkdir/rmdir code flow
In reply to Tony LuckSNC is only present in the RDT_RESOURCE_L3 domain.
Refactor code that makes and removes directories under "mon_data" to special case the L3 resource.
--- fs/resctrl/rdtgroup.c | 50 +++++++++++++++++++++++++++---------------- 1 file changed, 32 insertions(+), 18 deletions(-)
fs/resctrl/rdtgroup.c +32 -18
@@ -3155,6 +3155,7 @@ static void mon_rmdir_one_subdir(struct kernfs_node *pkn, char *name, char *subnreturn ;kernfs_put (kn); +/* Subdirectories are only present on SNC enabled systems */ if (kn->dir.subdirs <=1 )kernfs_remove (kn);else
@@ -3171,19 +3172,24 @@ static void rmdir_mondata_subdir_allrdtgrp(struct rdt_resource *r, struct rdt_domain_hdr *hdr) {struct rdtgroup *prgrp, *crgrp; -struct rdt_l3_mon_domain *d; +int domid = hdr->id;char subname[32 ]; -bool snc_mode;char name[32 ]; -if (!domain_header_is_valid (hdr,RESCTRL_MON_DOMAIN ,RDT_RESOURCE_L3 )) -return ; +if (r->rid ==RDT_RESOURCE_L3 ) { +struct rdt_l3_mon_domain *d; - d = container_of(hdr, struct rdt_l3_mon_domain, hdr); - snc_mode = r->mon_scope ==RESCTRL_L3_NODE ; -sprintf (name,"mon_%s_%02d" , r->name, snc_mode ? d->ci_id : d->hdr.id); -if (snc_mode) -sprintf (subname,"mon_sub_%s_%02d" , r->name, d->hdr.id); +if (!domain_header_is_valid (hdr,RESCTRL_MON_DOMAIN ,RDT_RESOURCE_L3 )) +return ; + + d = container_of(hdr, struct rdt_l3_mon_domain, hdr); +/* SNC mode? */ +if (r->mon_scope ==RESCTRL_L3_NODE ) { + domid = d->ci_id; +sprintf (subname,"mon_sub_%s_%02d" , r->name, hdr->id); + } + } +sprintf (name,"mon_%s_%02d" , r->name, domid);list_for_each_entry (prgrp, &rdt_all_groups, rdtgroup_list) {mon_rmdir_one_subdir (prgrp->mon.mon_data_kn, name, subname);
@@ -3213,7 +3219,7 @@ static int mon_add_all_files(struct kernfs_node *kn, struct rdt_domain_hdr *hdr,if (ret)return ret; -if (!do_sum &&resctrl_is_mbm_event (mevt->evtid)) +if (r->rid ==RDT_RESOURCE_L3 && !do_sum &&resctrl_is_mbm_event (mevt->evtid))mon_event_read (&rr, r, hdr, prgrp, &hdr->cpu_mask, mevt, true); }
@@ -3225,19 +3231,27 @@ static int mkdir_mondata_subdir(struct kernfs_node *parent_kn, struct rdt_resource *r, struct rdtgroup *prgrp) { struct kernfs_node *kn, *ckn; -struct rdt_l3_mon_domain *d; +bool snc_mode = false; +int domid = hdr->id;char name[32 ]; -bool snc_mode;int ret =0 ;lockdep_assert_held (&rdtgroup_mutex); -if (!domain_header_is_valid (hdr,RESCTRL_MON_DOMAIN ,RDT_RESOURCE_L3 )) -return -EINVAL ; +if (r->rid ==RDT_RESOURCE_L3 ) { + snc_mode = r->mon_scope ==RESCTRL_L3_NODE ; +if (snc_mode) { +struct rdt_l3_mon_domain *d; + +if (!domain_header_is_valid (hdr,RESCTRL_MON_DOMAIN ,RDT_RESOURCE_L3 )) +return -EINVAL ; + + d = container_of(hdr, struct rdt_l3_mon_domain, hdr); + domid = d->ci_id; + } + } +sprintf (name,"mon_%s_%02d" , r->name, domid); - d = container_of(hdr, struct rdt_l3_mon_domain, hdr); - snc_mode = r->mon_scope ==RESCTRL_L3_NODE ; -sprintf (name,"mon_%s_%02d" , r->name, snc_mode ? d->ci_id : d->hdr.id); kn =kernfs_find_and_get (parent_kn, name);if (kn) {/*
@@ -3253,7 +3267,7 @@ static int mkdir_mondata_subdir(struct kernfs_node *parent_kn, ret =rdtgroup_kn_set_ugid (kn);if (ret) goto out_destroy; - ret =mon_add_all_files (kn, hdr, r, prgrp, hdr->id, snc_mode); + ret =mon_add_all_files (kn, hdr, r, prgrp, domid, snc_mode);if (ret) goto out_destroy; }
Signature · 1 line
2.51.0
Reinette Chatre
maintainerRe: [PATCH v11 20/31] fs/resctrl: Refactor Sub-NUMA Cluster (SNC) in mkdir/rmdir code flow
Hi Tony,
Quoted 3 lines · Tony Luck
SNC is only present in the
RDT_RESOURCE_L3domain.Refactor code that makes and removes directories under "
mon_data" to
"makes and removes directories" -> "makes and removes SNC directories"?
Quoted 1 line
special case the L3 resource.
Why?
Quoted 97 lines
Signed-off-by Tony Luck <[email protected]>--- fs/resctrl/rdtgroup.c | 50 +++++++++++++++++++++++++++---------------- 1 file changed, 32 insertions(+), 18 deletions(-)fs/resctrl/rdtgroup.c +31
-17@@ -3155,6 +3155,7 @@ static void mon_rmdir_one_subdir(struct kernfs_node *pkn, char *name, char *subnreturn ;kernfs_put (kn); +/* Subdirectories are only present on SNC enabled systems */ if (kn->dir.subdirs <=1 )kernfs_remove (kn);else @@ -3171,19 +3172,24 @@ static void rmdir_mondata_subdir_allrdtgrp(struct rdt_resource *r, struct rdt_domain_hdr *hdr) {struct rdtgroup *prgrp, *crgrp; -struct rdt_l3_mon_domain *d; +int domid = hdr->id;char subname[32 ]; -bool snc_mode;char name[32 ]; -if (!domain_header_is_valid (hdr,RESCTRL_MON_DOMAIN ,RDT_RESOURCE_L3 )) -return ; +if (r->rid ==RDT_RESOURCE_L3 ) { +struct rdt_l3_mon_domain *d; - d = container_of(hdr, struct rdt_l3_mon_domain, hdr); - snc_mode = r->mon_scope ==RESCTRL_L3_NODE ; -sprintf (name,"mon_%s_%02d" , r->name, snc_mode ? d->ci_id : d->hdr.id); -if (snc_mode) -sprintf (subname,"mon_sub_%s_%02d" , r->name, d->hdr.id); +if (!domain_header_is_valid (hdr,RESCTRL_MON_DOMAIN ,RDT_RESOURCE_L3 )) +return ; + + d = container_of(hdr, struct rdt_l3_mon_domain, hdr); +/* SNC mode? */ +if (r->mon_scope ==RESCTRL_L3_NODE ) { + domid = d->ci_id; +sprintf (subname,"mon_sub_%s_%02d" , r->name, hdr->id); + } + } +sprintf (name,"mon_%s_%02d" , r->name, domid);list_for_each_entry (prgrp, &rdt_all_groups, rdtgroup_list) {mon_rmdir_one_subdir (prgrp->mon.mon_data_kn, name, subname);@@ -3213,7 +3219,7 @@ static int mon_add_all_files(struct kernfs_node *kn, struct rdt_domain_hdr *hdr,if (ret)return ret; -if (!do_sum &&resctrl_is_mbm_event (mevt->evtid)) +if (r->rid ==RDT_RESOURCE_L3 && !do_sum &&resctrl_is_mbm_event (mevt->evtid))mon_event_read (&rr, r, hdr, prgrp, &hdr->cpu_mask, mevt, true); }@@ -3225,19 +3231,27 @@ static int mkdir_mondata_subdir(struct kernfs_node *parent_kn, struct rdt_resource *r, struct rdtgroup *prgrp) { struct kernfs_node *kn, *ckn; -struct rdt_l3_mon_domain *d; +bool snc_mode = false; +int domid = hdr->id;char name[32 ]; -bool snc_mode;int ret =0 ;lockdep_assert_held (&rdtgroup_mutex); -if (!domain_header_is_valid (hdr,RESCTRL_MON_DOMAIN ,RDT_RESOURCE_L3 )) -return -EINVAL ; +if (r->rid ==RDT_RESOURCE_L3 ) { + snc_mode = r->mon_scope ==RESCTRL_L3_NODE ; +if (snc_mode) { +struct rdt_l3_mon_domain *d; + +if (!domain_header_is_valid (hdr,RESCTRL_MON_DOMAIN ,RDT_RESOURCE_L3 )) +return -EINVAL ; + + d = container_of(hdr, struct rdt_l3_mon_domain, hdr); + domid = d->ci_id; + } + } +sprintf (name,"mon_%s_%02d" , r->name, domid); - d = container_of(hdr, struct rdt_l3_mon_domain, hdr); - snc_mode = r->mon_scope ==RESCTRL_L3_NODE ; -sprintf (name,"mon_%s_%02d" , r->name, snc_mode ? d->ci_id : d->hdr.id); kn =kernfs_find_and_get (parent_kn, name);if (kn) {/*
fs/resctrl/rdtgroup.c static int mkdir_mondata_subdir(struct kernfs_node *parent_kn, · line 3273
Quoted 8 lines
fs/resctrl/rdtgroup.c +1
-1@@ -3253,7 +3267,7 @@ static int mkdir_mondata_subdir(struct kernfs_node *parent_kn, ret =rdtgroup_kn_set_ugid (kn);if (ret) goto out_destroy; - ret =mon_add_all_files (kn, hdr, r, prgrp, hdr->id, snc_mode); + ret =mon_add_all_files (kn, hdr, r, prgrp, domid, snc_mode);if (ret) goto out_destroy; }
mkdir_mondata_subdir(), similar to __mon_event_count(), is now unreasonably complicated. Just like in that earlier change this inconsistently adds RDT_RESOURCE_L3 checks, not to separate L3 code but instead to benefit PERF_PKG enabling to reach the handful of lines needed by it.
Here too I think the best way forward is to split mkdir_mondata_subdir().
rmdir_mondata_subdir_allrdtgrp() may also do with a split ... most of the code within it is dedicated to SNC and mon_rmdir_one_subdir() only exists because of SNC ... any other usage can just call kernfs_remove_by_name(), no?
SNC is already complicated enabling and I think that PERF_PKG trying to wedge itself into that is just too confusing. I expect separating this should simplify this a lot.
Reinette
Luck, Tony
maintainerRe: [PATCH v11 20/31] fs/resctrl: Refactor Sub-NUMA Cluster (SNC) in mkdir/rmdir code flow
Quoted 125 lines · Reinette Chatre
Hi Tony,
Quoted 3 lines · Tony Luck
SNC is only present in the
RDT_RESOURCE_L3domain.Refactor code that makes and removes directories under "
mon_data" to"makes and removes directories" -> "makes and removes SNC directories"?
Quoted 1 line
special case the L3 resource.
Why?
Quoted 97 lines
Signed-off-by Tony Luck <[email protected]>--- fs/resctrl/rdtgroup.c | 50 +++++++++++++++++++++++++++---------------- 1 file changed, 32 insertions(+), 18 deletions(-)fs/resctrl/rdtgroup.c +31
-17@@ -3155,6 +3155,7 @@ static void mon_rmdir_one_subdir(struct kernfs_node *pkn, char *name, char *subnreturn ;kernfs_put (kn); +/* Subdirectories are only present on SNC enabled systems */ if (kn->dir.subdirs <=1 )kernfs_remove (kn);else @@ -3171,19 +3172,24 @@ static void rmdir_mondata_subdir_allrdtgrp(struct rdt_resource *r, struct rdt_domain_hdr *hdr) {struct rdtgroup *prgrp, *crgrp; -struct rdt_l3_mon_domain *d; +int domid = hdr->id;char subname[32 ]; -bool snc_mode;char name[32 ]; -if (!domain_header_is_valid (hdr,RESCTRL_MON_DOMAIN ,RDT_RESOURCE_L3 )) -return ; +if (r->rid ==RDT_RESOURCE_L3 ) { +struct rdt_l3_mon_domain *d; - d = container_of(hdr, struct rdt_l3_mon_domain, hdr); - snc_mode = r->mon_scope ==RESCTRL_L3_NODE ; -sprintf (name,"mon_%s_%02d" , r->name, snc_mode ? d->ci_id : d->hdr.id); -if (snc_mode) -sprintf (subname,"mon_sub_%s_%02d" , r->name, d->hdr.id); +if (!domain_header_is_valid (hdr,RESCTRL_MON_DOMAIN ,RDT_RESOURCE_L3 )) +return ; + + d = container_of(hdr, struct rdt_l3_mon_domain, hdr); +/* SNC mode? */ +if (r->mon_scope ==RESCTRL_L3_NODE ) { + domid = d->ci_id; +sprintf (subname,"mon_sub_%s_%02d" , r->name, hdr->id); + } + } +sprintf (name,"mon_%s_%02d" , r->name, domid);list_for_each_entry (prgrp, &rdt_all_groups, rdtgroup_list) {mon_rmdir_one_subdir (prgrp->mon.mon_data_kn, name, subname);@@ -3213,7 +3219,7 @@ static int mon_add_all_files(struct kernfs_node *kn, struct rdt_domain_hdr *hdr,if (ret)return ret; -if (!do_sum &&resctrl_is_mbm_event (mevt->evtid)) +if (r->rid ==RDT_RESOURCE_L3 && !do_sum &&resctrl_is_mbm_event (mevt->evtid))mon_event_read (&rr, r, hdr, prgrp, &hdr->cpu_mask, mevt, true); }@@ -3225,19 +3231,27 @@ static int mkdir_mondata_subdir(struct kernfs_node *parent_kn, struct rdt_resource *r, struct rdtgroup *prgrp) { struct kernfs_node *kn, *ckn; -struct rdt_l3_mon_domain *d; +bool snc_mode = false; +int domid = hdr->id;char name[32 ]; -bool snc_mode;int ret =0 ;lockdep_assert_held (&rdtgroup_mutex); -if (!domain_header_is_valid (hdr,RESCTRL_MON_DOMAIN ,RDT_RESOURCE_L3 )) -return -EINVAL ; +if (r->rid ==RDT_RESOURCE_L3 ) { + snc_mode = r->mon_scope ==RESCTRL_L3_NODE ; +if (snc_mode) { +struct rdt_l3_mon_domain *d; + +if (!domain_header_is_valid (hdr,RESCTRL_MON_DOMAIN ,RDT_RESOURCE_L3 )) +return -EINVAL ; + + d = container_of(hdr, struct rdt_l3_mon_domain, hdr); + domid = d->ci_id; + } + } +sprintf (name,"mon_%s_%02d" , r->name, domid); - d = container_of(hdr, struct rdt_l3_mon_domain, hdr); - snc_mode = r->mon_scope ==RESCTRL_L3_NODE ; -sprintf (name,"mon_%s_%02d" , r->name, snc_mode ? d->ci_id : d->hdr.id); kn =kernfs_find_and_get (parent_kn, name);if (kn) {/* fs/resctrl/rdtgroup.c static int mkdir_mondata_subdir(struct kernfs_node *parent_kn, · line 3273
Quoted 8 lines
fs/resctrl/rdtgroup.c +1
-1@@ -3253,7 +3267,7 @@ static int mkdir_mondata_subdir(struct kernfs_node *parent_kn, ret =rdtgroup_kn_set_ugid (kn);if (ret) goto out_destroy; - ret =mon_add_all_files (kn, hdr, r, prgrp, hdr->id, snc_mode); + ret =mon_add_all_files (kn, hdr, r, prgrp, domid, snc_mode);if (ret) goto out_destroy; }
mkdir_mondata_subdir(), similar to__mon_event_count(), is now unreasonably complicated. Just like in that earlier change this inconsistently addsRDT_RESOURCE_L3checks, not to separate L3 code but instead to benefitPERF_PKGenabling to reach the handful of lines needed by it.
Here too I think the best way forward is to splitmkdir_mondata_subdir().
Quoted 7 lines
rmdir_mondata_subdir_allrdtgrp()may also do with a split ... most of the code within it is dedicated to SNC andmon_rmdir_one_subdir()only exists because of SNC ... any other usage can just callkernfs_remove_by_name(), no?SNC is already complicated enabling and I think that
PERF_PKGtrying to wedge itself into that is just too confusing. I expect separating this should simplify this a lot.
Ok. Splitting these makes sense. I'm terrible at naming. So I tentatively have:
static int mkdir_mondata_subdir(struct kernfs_node *parent_kn,
struct rdt_domain_hdr *hdr,
struct rdt_resource *r, struct rdtgroup *prgrp)
{
lockdep_assert_held(&rdtgroup_mutex);
if (r->mon_scope == RESCTRL_L3_NODE)
return mkdir_mondata_subdir_snc(parent_kn, hdr, r, prgrp);
else
return mkdir_mondata_subdir_normal(parent_kn, hdr, r, prgrp);
}and:
static void rmdir_mondata_subdir_allrdtgrp(struct rdt_resource *r,
struct rdt_domain_hdr *hdr)
{
if (r->mon_scope == RESCTRL_L3_NODE)
rmdir_mondata_subdir_allrdtgrp_snc(r, hdr);
else
rmdir_mondata_subdir_allrdtgrp_normal(r, hdr);
}Better suggestions gratefully accepted.
Quoted 3 lines
Reinette
Reinette Chatre
maintainerRe: [PATCH v11 20/31] fs/resctrl: Refactor Sub-NUMA Cluster (SNC) in mkdir/rmdir code flow
Hi Tony,
Quoted 46 lines · Tony
Quoted 18 lines · Reinette Chatre
fs/resctrl/rdtgroup.c static int mkdir_mondata_subdir(struct kernfs_node *parent_kn, · line 3273
Quoted 9 lines · Tony Luck
fs/resctrl/rdtgroup.c +1
-1@@ -3253,7 +3267,7 @@ static int mkdir_mondata_subdir(struct kernfs_node *parent_kn, ret =rdtgroup_kn_set_ugid (kn);if (ret) goto out_destroy; - ret =mon_add_all_files (kn, hdr, r, prgrp, hdr->id, snc_mode); + ret =mon_add_all_files (kn, hdr, r, prgrp, domid, snc_mode);if (ret) goto out_destroy; }
mkdir_mondata_subdir(), similar to__mon_event_count(), is now unreasonably complicated. Just like in that earlier change this inconsistently addsRDT_RESOURCE_L3checks, not to separate L3 code but instead to benefitPERF_PKGenabling to reach the handful of lines needed by it.
Here too I think the best way forward is to splitmkdir_mondata_subdir().Quoted 7 lines
rmdir_mondata_subdir_allrdtgrp()may also do with a split ... most of the code within it is dedicated to SNC andmon_rmdir_one_subdir()only exists because of SNC ... any other usage can just callkernfs_remove_by_name(), no?SNC is already complicated enabling and I think that
PERF_PKGtrying to wedge itself into that is just too confusing. I expect separating this should simplify this a lot.Ok. Splitting these makes sense. I'm terrible at naming. So I tentatively have:
static int mkdir_mondata_subdir(struct
kernfs_node*parent_kn,
structrdt_domain_hdr*hdr,
structrdt_resource*r, struct rdtgroup *prgrp){ lockdep_assert_held(&rdtgroup_mutex); if (r->mon_scope == RESCTRL_L3_NODE) return mkdir_mondata_subdir_snc(parent_kn, hdr, r, prgrp); else return mkdir_mondata_subdir_normal(parent_kn, hdr, r, prgrp); }and:
static void rmdir_mondata_subdir_allrdtgrp(struct
rdt_resource*r,
structrdt_domain_hdr*hdr)
Quoted 8 lines
{ if (r->mon_scope == RESCTRL_L3_NODE) rmdir_mondata_subdir_allrdtgrp_snc(r, hdr); else rmdir_mondata_subdir_allrdtgrp_normal(r, hdr); }Better suggestions gratefully accepted.
It is not quite obvious to me how it all will turn out from here with the addition of support for PERF_PKG. By just considering the above I think that it helps to match the naming pattern with partners,
for example rmdir_mondata_subdir_allrdtgrp() as you have that matches mkdir_mondata_subdir_allrdtgrp() that is not listed here. The problem is that the new rmdir_mondata_subdir_allrdtgrp() is L3 specific while mkdir_mondata_subdir_allrdtgrp() remains generic. I thus think that it may make the code easier to follow if the L3 specific functions have _l3_ in the name as you established in patch #8. So perhaps above should be rmdir_l3_mondata_subdir_allrdtgrp() instead and then there may be a new rmdir_mondata_subdir_allrdtgrp() that will be the new generic function that calls the resource specific ones?
This could be extended to the new mkdir_mondata_subdir() above where it is named mkdir_l3_mondata_subdir() called by generic mkdir_mondata_subdir()?
Reinette
Luck, Tony
maintainerRe: [PATCH v11 20/31] fs/resctrl: Refactor Sub-NUMA Cluster (SNC) in mkdir/rmdir code flow
Quoted 71 lines · Reinette Chatre
Hi Tony,
Quoted 46 lines · Tony
Quoted 18 lines · Reinette Chatre
fs/resctrl/rdtgroup.c static int mkdir_mondata_subdir(struct kernfs_node *parent_kn, · line 3273
Quoted 9 lines · Tony Luck
fs/resctrl/rdtgroup.c +1
-1@@ -3253,7 +3267,7 @@ static int mkdir_mondata_subdir(struct kernfs_node *parent_kn, ret =rdtgroup_kn_set_ugid (kn);if (ret) goto out_destroy; - ret =mon_add_all_files (kn, hdr, r, prgrp, hdr->id, snc_mode); + ret =mon_add_all_files (kn, hdr, r, prgrp, domid, snc_mode);if (ret) goto out_destroy; }
mkdir_mondata_subdir(), similar to__mon_event_count(), is now unreasonably complicated. Just like in that earlier change this inconsistently addsRDT_RESOURCE_L3checks, not to separate L3 code but instead to benefitPERF_PKGenabling to reach the handful of lines needed by it.
Here too I think the best way forward is to splitmkdir_mondata_subdir().Quoted 7 lines
rmdir_mondata_subdir_allrdtgrp()may also do with a split ... most of the code within it is dedicated to SNC andmon_rmdir_one_subdir()only exists because of SNC ... any other usage can just callkernfs_remove_by_name(), no?SNC is already complicated enabling and I think that
PERF_PKGtrying to wedge itself into that is just too confusing. I expect separating this should simplify this a lot.Ok. Splitting these makes sense. I'm terrible at naming. So I tentatively have:
static int mkdir_mondata_subdir(struct
kernfs_node*parent_kn,
structrdt_domain_hdr*hdr,
structrdt_resource*r, struct rdtgroup *prgrp){ lockdep_assert_held(&rdtgroup_mutex); if (r->mon_scope == RESCTRL_L3_NODE) return mkdir_mondata_subdir_snc(parent_kn, hdr, r, prgrp); else return mkdir_mondata_subdir_normal(parent_kn, hdr, r, prgrp); }and:
static void rmdir_mondata_subdir_allrdtgrp(struct
rdt_resource*r,
structrdt_domain_hdr*hdr)Quoted 8 lines
{ if (r->mon_scope == RESCTRL_L3_NODE) rmdir_mondata_subdir_allrdtgrp_snc(r, hdr); else rmdir_mondata_subdir_allrdtgrp_normal(r, hdr); }Better suggestions gratefully accepted.
It is not quite obvious to me how it all will turn out from here with the addition of support for
PERF_PKG. By just considering the above I think that it helps to match the naming pattern with partners,
for examplermdir_mondata_subdir_allrdtgrp()as you have that matchesmkdir_mondata_subdir_allrdtgrp()that is not listed here. The problem is that the newrmdir_mondata_subdir_allrdtgrp()is L3 specific whilemkdir_mondata_subdir_allrdtgrp()remains generic. I thus think that it may make the code easier to follow if the L3 specific functions have _l3_ in the name as you established in patch #8. So perhaps above should bermdir_l3_mondata_subdir_allrdtgrp()instead and then there may be a newrmdir_mondata_subdir_allrdtgrp()that will be the new generic function that calls the resource specific ones?
Quoted 2 lines
This could be extended to the new
mkdir_mondata_subdir()above where it is namedmkdir_l3_mondata_subdir()called by genericmkdir_mondata_subdir()?
Reinette
Making and removing the mon_data directories is the same for non-SNC L3 and PERF_PKG. The only "l3" connection is that SNC only occurs on L3.
So maybe my refactor should look like:
static int mkdir_mondata_subdir(struct kernfs_node *parent_kn,
struct rdt_domain_hdr *hdr,
struct rdt_resource *r, struct rdtgroup *prgrp)
{
lockdep_assert_held(&rdtgroup_mutex);
if (r->mon_scope == RESCTRL_L3_NODE)
return mkdir_mondata_subdir_snc(parent_kn, hdr, r, prgrp);
... pruned version of original code without SNC bits ...
}and:
static void rmdir_mondata_subdir_allrdtgrp(struct rdt_resource *r,
struct rdt_domain_hdr *hdr)
{
if (r->mon_scope == RESCTRL_L3_NODE) {
rmdir_mondata_subdir_allrdtgrp_snc(r, hdr);
return;
}
... pruned version of original code without SNC bits ...
}-Tony
Reinette Chatre
maintainerRe: [PATCH v11 20/31] fs/resctrl: Refactor Sub-NUMA Cluster (SNC) in mkdir/rmdir code flow
Hi Tony,
Quoted 72 lines · Tony
Quoted 71 lines · Reinette Chatre
Hi Tony,
Quoted 46 lines · Tony
Quoted 18 lines · Reinette Chatre
fs/resctrl/rdtgroup.c static int mkdir_mondata_subdir(struct kernfs_node *parent_kn, · line 3273
Quoted 9 lines · Tony Luck
fs/resctrl/rdtgroup.c +1
-1@@ -3253,7 +3267,7 @@ static int mkdir_mondata_subdir(struct kernfs_node *parent_kn, ret =rdtgroup_kn_set_ugid (kn);if (ret) goto out_destroy; - ret =mon_add_all_files (kn, hdr, r, prgrp, hdr->id, snc_mode); + ret =mon_add_all_files (kn, hdr, r, prgrp, domid, snc_mode);if (ret) goto out_destroy; }
mkdir_mondata_subdir(), similar to__mon_event_count(), is now unreasonably complicated. Just like in that earlier change this inconsistently addsRDT_RESOURCE_L3checks, not to separate L3 code but instead to benefitPERF_PKGenabling to reach the handful of lines needed by it.
Here too I think the best way forward is to splitmkdir_mondata_subdir().Quoted 7 lines
rmdir_mondata_subdir_allrdtgrp()may also do with a split ... most of the code within it is dedicated to SNC andmon_rmdir_one_subdir()only exists because of SNC ... any other usage can just callkernfs_remove_by_name(), no?SNC is already complicated enabling and I think that
PERF_PKGtrying to wedge itself into that is just too confusing. I expect separating this should simplify this a lot.Ok. Splitting these makes sense. I'm terrible at naming. So I tentatively have:
static int mkdir_mondata_subdir(struct
kernfs_node*parent_kn,
structrdt_domain_hdr*hdr,
structrdt_resource*r, struct rdtgroup *prgrp){ lockdep_assert_held(&rdtgroup_mutex); if (r->mon_scope == RESCTRL_L3_NODE) return mkdir_mondata_subdir_snc(parent_kn, hdr, r, prgrp); else return mkdir_mondata_subdir_normal(parent_kn, hdr, r, prgrp); }and:
static void rmdir_mondata_subdir_allrdtgrp(struct
rdt_resource*r,
structrdt_domain_hdr*hdr)Quoted 8 lines
{ if (r->mon_scope == RESCTRL_L3_NODE) rmdir_mondata_subdir_allrdtgrp_snc(r, hdr); else rmdir_mondata_subdir_allrdtgrp_normal(r, hdr); }Better suggestions gratefully accepted.
It is not quite obvious to me how it all will turn out from here with the addition of support for
PERF_PKG. By just considering the above I think that it helps to match the naming pattern with partners,
for examplermdir_mondata_subdir_allrdtgrp()as you have that matchesmkdir_mondata_subdir_allrdtgrp()that is not listed here. The problem is that the newrmdir_mondata_subdir_allrdtgrp()is L3 specific whilemkdir_mondata_subdir_allrdtgrp()remains generic. I thus think that it may make the code easier to follow if the L3 specific functions have _l3_ in the name as you established in patch #8. So perhaps above should bermdir_l3_mondata_subdir_allrdtgrp()instead and then there may be a newrmdir_mondata_subdir_allrdtgrp()that will be the new generic function that calls the resource specific ones?
Quoted 7 lines
Quoted 2 lines
This could be extended to the new
mkdir_mondata_subdir()above where it is namedmkdir_l3_mondata_subdir()called by genericmkdir_mondata_subdir()?Reinette
Making and removing the
mon_datadirectories is the same for non-SNC L3 andPERF_PKG. The only "l3" connection is that SNC only occurs on L3.
Thank you for clarifying. I was not able to keep this flow in my head.
Quoted 20 lines
So maybe my refactor should look like:
static int mkdir_mondata_subdir(struct
kernfs_node*parent_kn,
structrdt_domain_hdr*hdr,
structrdt_resource*r, struct rdtgroup *prgrp){ lockdep_assert_held(&rdtgroup_mutex); if (r->mon_scope == RESCTRL_L3_NODE) return mkdir_mondata_subdir_snc(parent_kn, hdr, r, prgrp); ... pruned version of original code without SNC bits ... }and:
static void rmdir_mondata_subdir_allrdtgrp(struct
rdt_resource*r,
structrdt_domain_hdr*hdr)
Quoted 8 lines
{
if (r->mon_scope == RESCTRL_L3_NODE) {
rmdir_mondata_subdir_allrdtgrp_snc(r, hdr);
return;
}
... pruned version of original code without SNC bits ...
}Indeed, this will keep the functions generic in the sense that it operates on all resource types. This looks good since I think once the SNC code is taken what remains should be easy to follow.
I think it may also help to (in addition to the mon_scope check) add a RDT_RESOURCE_L3 check before the SNC code to keep the pattern that SNC only applies to the L3 resource.
Reinette
Luck, Tony
maintainerRE: [PATCH v11 20/31] fs/resctrl: Refactor Sub-NUMA Cluster (SNC) in mkdir/rmdir code flow
Quoted 22 lines
Quoted 22 lines
static int mkdir_mondata_subdir(struct
kernfs_node*parent_kn,
structrdt_domain_hdr*hdr,
structrdt_resource*r, struct rdtgroup *prgrp){ lockdep_assert_held(&rdtgroup_mutex); if (r->mon_scope == RESCTRL_L3_NODE) return mkdir_mondata_subdir_snc(parent_kn, hdr, r, prgrp); ... pruned version of original code without SNC bits ... }and:
static void rmdir_mondata_subdir_allrdtgrp(struct
rdt_resource*r,
structrdt_domain_hdr*hdr){ if (r->mon_scope == RESCTRL_L3_NODE) { rmdir_mondata_subdir_allrdtgrp_snc(r, hdr); return; }
Quoted 8 lines
Quoted 2 lines
... pruned version of original code without SNC bits ...
}Indeed, this will keep the functions generic in the sense that it operates on all resource types. This looks good since I think once the SNC code is taken what remains should be easy to follow.
I think it may also help to (in addition to themon_scopecheck) add aRDT_RESOURCE_L3check before the SNC code to keep the pattern that SNC only applies to the L3 resource.
Reinette,
The SNC versions to make and remove directories need to get the rdt_l3_mon_domain from hdr. So they both begin with the standard:
if (!domain_header_is_valid(hdr, RESCTRL_MON_DOMAIN, RDT_RESOURCE_L3))
return -EINVAL;
d = container_of(hdr, struct rdt_l3_mon_domain, hdr);[though rmdir function returns void, so doesn't have that "-EINVAL".]
-Tony
Reinette Chatre
maintainerRe: [PATCH v11 20/31] fs/resctrl: Refactor Sub-NUMA Cluster (SNC) in mkdir/rmdir code flow
Hi Tony,
Quoted 36 lines · Tony
Quoted 22 lines
Quoted 22 lines
static int mkdir_mondata_subdir(struct
kernfs_node*parent_kn,
structrdt_domain_hdr*hdr,
structrdt_resource*r, struct rdtgroup *prgrp){ lockdep_assert_held(&rdtgroup_mutex); if (r->mon_scope == RESCTRL_L3_NODE) return mkdir_mondata_subdir_snc(parent_kn, hdr, r, prgrp); ... pruned version of original code without SNC bits ... }and:
static void rmdir_mondata_subdir_allrdtgrp(struct
rdt_resource*r,
structrdt_domain_hdr*hdr){ if (r->mon_scope == RESCTRL_L3_NODE) { rmdir_mondata_subdir_allrdtgrp_snc(r, hdr); return; }Quoted 8 lines
Quoted 2 lines
... pruned version of original code without SNC bits ...
}Indeed, this will keep the functions generic in the sense that it operates on all resource types. This looks good since I think once the SNC code is taken what remains should be easy to follow.
I think it may also help to (in addition to themon_scopecheck) add aRDT_RESOURCE_L3check before the SNC code to keep the pattern that SNC only applies to the L3 resource.Reinette,
The SNC versions to make and remove directories need to get the
rdt_l3_mon_domainfrom hdr. So they both begin with the standard:
Quoted 6 lines
if (!domain_header_is_valid(hdr, RESCTRL_MON_DOMAIN, RDT_RESOURCE_L3)) return -EINVAL; d = container_of(hdr, struct rdt_l3_mon_domain, hdr);[though rmdir function returns void, so doesn't have that "
-EINVAL".]
Understood. This is not about correctness but making the code easier to understand. What I am aiming for is consistency in the code where the pattern
in existing flows use the resource ID as check to direct code flow to resource specific code. In the above flow it uses the monitoring scope. This works of course, but it is an implicit check because the L3 resource is the only one that currently supports the "node" scope and does so when SNC is enabled.
My preference is for the code to be consistent in patterns used and find doing so makes the code easier to read and understand.
Reinette
Luck, Tony
maintainerRe: [PATCH v11 20/31] fs/resctrl: Refactor Sub-NUMA Cluster (SNC) in mkdir/rmdir code flow
Quoted 9 lines · Reinette Chatre
Understood. This is not about correctness but making the code easier to understand. What I am aiming for is consistency in the code where the pattern
in existing flows use the resource ID as check to direct code flow to resource specific code. In the above flow it uses the monitoring scope. This works of course, but it is an implicit check because the L3 resource is the only one that currently supports the "node" scope and does so when SNC is enabled.
My preference is for the code to be consistent in patterns used and find doing so makes the code easier to read and understand.
Reinette,
Should I address this "only one that currently" issue now? Maybe by adding a bool "rdt_resource:snc_mode" so the implicit
if (r->mon_scope == RESCTRL_L3_NODE)
changes to:
if (r->snc_mode)
There are only two places where SNC mode is checked in this way. The others rely on seeing that mon_data::sum is set, or that rr->hdr is NULL. So it seems like a very small improvement.
If we ever add a node scoped resource that isn't related to SNC, it would be needed at that point. But I'm not sure why hardware would ever do that.
-Tony
Reinette Chatre
maintainerRe: [PATCH v11 20/31] fs/resctrl: Refactor Sub-NUMA Cluster (SNC) in mkdir/rmdir code flow
Hi Tony,
Quoted 17 lines · Tony
Quoted 9 lines · Reinette Chatre
Understood. This is not about correctness but making the code easier to understand. What I am aiming for is consistency in the code where the pattern
in existing flows use the resource ID as check to direct code flow to resource specific code. In the above flow it uses the monitoring scope. This works of course, but it is an implicit check because the L3 resource is the only one that currently supports the "node" scope and does so when SNC is enabled.
My preference is for the code to be consistent in patterns used and find doing so makes the code easier to read and understand.Reinette,
Should I address this "only one that currently" issue now? Maybe by adding a bool "rdt_resource:snc_mode" so the implicit
if (r->mon_scope == RESCTRL_L3_NODE)
Quoted 7 lines
changes to:
if (r->snc_mode)There are only two places where SNC mode is checked in this way. The others rely on seeing that
mon_data::sumis set, or thatrr->hdris NULL. So it seems like a very small improvement.
This is not about SNC mode or not but instead about this code being L3 resource specific.
I see the mon_data::sum and rr->hdr checks as supporting a separate feature that was introduced to support SNC - it should not be used as a check for SNC support even though it currently implies this due to SNC being the only user. Could we not, hypothetically, even use these properties in the region aware MBM work?
Quoted 3 lines
If we ever add a node scoped resource that isn't related to SNC, it would be needed at that point. But I'm not sure why hardware would ever do that.
Right. This is not about just what is needed to enable this feature but about making the code easy to follow for those that attempt to understand, debug, and/or build on top.
Reinette
Luck, Tony
maintainerRE: [PATCH v11 20/31] fs/resctrl: Refactor Sub-NUMA Cluster (SNC) in mkdir/rmdir code flow
Quoted 13 lines
Quoted 3 lines
There are only two places where SNC mode is checked in this way. The others rely on seeing that
mon_data::sumis set, or thatrr->hdris NULL. So it seems like a very small improvement.This is not about SNC mode or not but instead about this code being L3 resource specific.
I see the
mon_data::sumandrr->hdrchecks as supporting a separate feature that was introduced to support SNC - it should not be used as a check for SNC support even though it currently implies this due to SNC being the only user. Could we not, hypothetically, even use these properties in the region aware MBM work?
Quoted 7 lines
Quoted 3 lines
If we ever add a node scoped resource that isn't related to SNC, it would be needed at that point. But I'm not sure why hardware would ever do that.
Right. This is not about just what is needed to enable this feature but about making the code easy to follow for those that attempt to understand, debug, and/or build on top.
Reinette,
Region aware MBM work will need to sum things to support legacy resctrl "mbm_total_bytes". But while SNC sums across domains that share the same cache id inside the same resource, we may be summing across different resources (assuming we go with separate resources per region) or summing across regions within a domain (if we bundle the regions into a new struct rdt_region_mon_domain).
So __mon_event_count() will need to get additional refactoring and helper functions, and struct mon_data an additional field to say that this other "sum" function must be used.
-Tony
Reinette Chatre
maintainerRe: [PATCH v11 20/31] fs/resctrl: Refactor Sub-NUMA Cluster (SNC) in mkdir/rmdir code flow
Hi Tony,
Quoted 30 lines · Tony
Quoted 13 lines
Quoted 3 lines
There are only two places where SNC mode is checked in this way. The others rely on seeing that
mon_data::sumis set, or thatrr->hdris NULL. So it seems like a very small improvement.This is not about SNC mode or not but instead about this code being L3 resource specific.
I see the
mon_data::sumandrr->hdrchecks as supporting a separate feature that was introduced to support SNC - it should not be used as a check for SNC support even though it currently implies this due to SNC being the only user. Could we not, hypothetically, even use these properties in the region aware MBM work?Quoted 7 lines
Quoted 3 lines
If we ever add a node scoped resource that isn't related to SNC, it would be needed at that point. But I'm not sure why hardware would ever do that.
Right. This is not about just what is needed to enable this feature but about making the code easy to follow for those that attempt to understand, debug, and/or build on top.
Reinette,
Region aware MBM work will need to sum things to support legacy resctrl "
mbm_total_bytes". But while SNC sums across domains that share the same cache id inside the same resource, we may be summing across different resources (assuming we go with separate resources per region) or summing across regions within a domain (if we bundle the regions into a new structrdt_region_mon_domain).
Quoted 4 lines
So
__mon_event_count()will need to get additional refactoring and helper functions, and structmon_dataan additional field to say that this other "sum" function must be used.
I did not mean to imply that this can be supported without refactoring. It does seem as though you agree that mon_data::sum may be used for something other than SNC and thus that using mon_data::sum as a check for SNC is not ideal.
Reinette
Luck, Tony
maintainerRE: [PATCH v11 20/31] fs/resctrl: Refactor Sub-NUMA Cluster (SNC) in mkdir/rmdir code flow
Quoted 3 lines
I did not mean to imply that this can be supported without refactoring. It does seem as though you agree that
mon_data::summay be used for something other than SNC and thus that usingmon_data::sumas a check for SNC is not ideal.
Reinette,
Yes, we are in agreement about non-SNC future usage.
Is it sufficient that I plant some WARN_ON_ONCE() in places where the code assumes that mon_data::sum is only used by RDT_RESOURCE_L3 or for SNC?
Such code can be fixed by future patches that want to use mon_data::sum for other things.
-Tony
Reinette Chatre
maintainerRe: [PATCH v11 20/31] fs/resctrl: Refactor Sub-NUMA Cluster (SNC) in mkdir/rmdir code flow
Hi Tony,
Quoted 11 lines · Tony
Quoted 3 lines
I did not mean to imply that this can be supported without refactoring. It does seem as though you agree that
mon_data::summay be used for something other than SNC and thus that usingmon_data::sumas a check for SNC is not ideal.Reinette,
Yes, we are in agreement about non-SNC future usage.
Is it sufficient that I plant some
WARN_ON_ONCE()in places where the code assumes thatmon_data::sumis only used byRDT_RESOURCE_L3or for SNC?
From what I understand this series does this already? I think this only applies to rdtgroup_mondata_show() that does below ("L3 specific" comments added by me just for this example) in this series:
rdtgroup_mondata_show()
{
...
if (md->sum) {
struct rdt_l3_mon_domain *d;
if (WARN_ON_ONCE(resid != RDT_RESOURCE_L3)) {
...
}
list_for_each_entry(d, &r->mon_domains, hdr.list) {
if (d->ci_id == domid) { /* L3 specific field */
...
/* L3 specific */
ci = get_cpu_cacheinfo_level(cpu, RESCTRL_L3_CACHE);
}
}
...
}This seems reasonable since the flow is different from the typical "check resource" followed by a domain_header_is_valid() that a refactor to support another resource would probably do as you state below.
Quoted 3 lines
Such code can be fixed by future patches that want to use
mon_data::sumfor other things.
This discussion digressed a bit. The discussion started with a request to add a check for the L3 resource before calling rmdir_mondata_subdir_allrdtgrp_snc(). I see this as something like:
if (r->rid == RDT_RESOURCE_L3 && r->mon_scope == RESCTRL_L3_NODE) {
rmdir_mondata_subdir_allrdtgrp_snc(r, hdr);
...
}I understand that rmdir_mondata_subdir_allrdtgrp_snc() may look something like below but I still find the flow easier to follow if a resource check is done before calling rmdir_mondata_subdir_allrdtgrp_snc().
rmdir_mondata_subdir_allrdtgrp_snc(r, hdr)
{
if (!domain_header_is_valid(hdr, RESCTRL_MON_DOMAIN, RDT_RESOURCE_L3))
return -EINVAL;
d = container_of(hdr, struct rdt_l3_mon_domain, hdr);
...
}Reinette
Luck, Tony
maintainerRe: [PATCH v11 20/31] fs/resctrl: Refactor Sub-NUMA Cluster (SNC) in mkdir/rmdir code flow
Quoted 19 lines · Reinette Chatre
Hi Tony,
Quoted 11 lines · Tony
Quoted 3 lines
I did not mean to imply that this can be supported without refactoring. It does seem as though you agree that
mon_data::summay be used for something other than SNC and thus that usingmon_data::sumas a check for SNC is not ideal.Reinette,
Yes, we are in agreement about non-SNC future usage.
Is it sufficient that I plant some
WARN_ON_ONCE()in places where the code assumes thatmon_data::sumis only used byRDT_RESOURCE_L3or for SNC?From what I understand this series does this already? I think this only applies to
rdtgroup_mondata_show()that does below ("L3 specific" comments added by me just for this example) in this series:
Quoted 7 lines
rdtgroup_mondata_show()
{
...
if (md->sum) {
struct rdt_l3_mon_domain *d;
if (WARN_ON_ONCE(resid != RDT_RESOURCE_L3)) {Exactly what I now have.
Quoted 1 line
...
My "..." is:
return -EINVAL;
Quoted 12 lines
}
list_for_each_entry(d, &r->mon_domains, hdr.list) {
if (d->ci_id == domid) { /* L3 specific field */
...
/* L3 specific */
ci = get_cpu_cacheinfo_level(cpu, RESCTRL_L3_CACHE);
}
}
...
}Quoted 3 lines
This seems reasonable since the flow is different from the typical "check resource" followed by a
domain_header_is_valid()that a refactor to support another resource would probably do as you state below.
I looked around to see if there were any other places that needed this, but they all have checks for RDT_RESOURCE_L3 by the end of the series. I've added a check in __mon_event_count() in patch 13 that gets deleted in patch 18 when the L3 code is split out into a separate function.
Quoted 8 lines
Quoted 3 lines
Such code can be fixed by future patches that want to use
mon_data::sumfor other things.This discussion digressed a bit. The discussion started with a request to add a check for the L3 resource before calling
rmdir_mondata_subdir_allrdtgrp_snc(). I see this as something like:if (r->rid == RDT_RESOURCE_L3 && r->mon_scope == RESCTRL_L3_NODE) {
I'll add this. Same is needed in mkdir_mondata_subdir().
Quoted 18 lines
rmdir_mondata_subdir_allrdtgrp_snc(r, hdr); ... }I understand that
rmdir_mondata_subdir_allrdtgrp_snc()may look something like below but I still find the flow easier to follow if a resource check is done before callingrmdir_mondata_subdir_allrdtgrp_snc().rmdir_mondata_subdir_allrdtgrp_snc(r, hdr) { if (!domain_header_is_valid(hdr, RESCTRL_MON_DOMAIN, RDT_RESOURCE_L3)) return -EINVAL; d = container_of(hdr, struct rdt_l3_mon_domain, hdr); ... }
Quoted 1 line
Reinette
-Tony
Reinette Chatre
maintainerRe: [PATCH v11 20/31] fs/resctrl: Refactor Sub-NUMA Cluster (SNC) in mkdir/rmdir code flow
Hi Tony,
Quoted 44 lines · Tony
Quoted 19 lines · Reinette Chatre
Hi Tony,
Quoted 11 lines · Tony
Quoted 3 lines
I did not mean to imply that this can be supported without refactoring. It does seem as though you agree that
mon_data::summay be used for something other than SNC and thus that usingmon_data::sumas a check for SNC is not ideal.Reinette,
Yes, we are in agreement about non-SNC future usage.
Is it sufficient that I plant some
WARN_ON_ONCE()in places where the code assumes thatmon_data::sumis only used byRDT_RESOURCE_L3or for SNC?From what I understand this series does this already? I think this only applies to
rdtgroup_mondata_show()that does below ("L3 specific" comments added by me just for this example) in this series:Quoted 7 lines
rdtgroup_mondata_show() { ... if (md->sum) { struct rdt_l3_mon_domain *d; if (WARN_ON_ONCE(resid != RDT_RESOURCE_L3)) {Exactly what I now have.
Quoted 1 line
...My "..." is:
return -EINVAL;Quoted 12 lines
} list_for_each_entry(d, &r->mon_domains, hdr.list) { if (d->ci_id == domid) { /* L3 specific field */ ... /* L3 specific */ ci = get_cpu_cacheinfo_level(cpu, RESCTRL_L3_CACHE); } } ... }
Quoted 6 lines
Quoted 3 lines
This seems reasonable since the flow is different from the typical "check resource" followed by a
domain_header_is_valid()that a refactor to support another resource would probably do as you state below.I looked around to see if there were any other places that needed this, but they all have checks for
RDT_RESOURCE_L3by the end of the series.
Thank you for checking. This seems like a good pattern to use consistently.
Quoted 7 lines
I've added a check in
__mon_event_count()in patch 13 that gets deleted in patch 18 when the L3 code is split out into a separate function.Quoted 4 lines
Quoted 3 lines
Such code can be fixed by future patches that want to use
mon_data::sumfor other things.
Quoted 7 lines
Quoted 4 lines
This discussion digressed a bit. The discussion started with a request to add a check for the L3 resource before calling
rmdir_mondata_subdir_allrdtgrp_snc(). I see this as something like:if (r->rid == RDT_RESOURCE_L3 && r->mon_scope == RESCTRL_L3_NODE) {I'll add this. Same is needed in
mkdir_mondata_subdir().
Thank you very much.
Reinette
Tony Luck
maintainerPATCH v11 21/31 x86/resctrl: Handle domain creation/deletion for RDT_RESOURCE_PERF_PKG
In reply to Tony LuckThe L3 resource has several requirements for domains. There are per-domain structures that hold the 64-bit values of counters, and elements to keep track of the overflow and limbo threads.
None of these are needed for the PERF_PKG resource. The hardware counters are wide enough that they do not wrap around for decades.
Define a new rdt_perf_pkg_mon_domain structure which just consists of the standard rdt_domain_hdr to keep track of domain id and CPU mask.
Support the PERF_PKG resource in the CPU online/offline handlers.
Add WARN checks to code that sums domains for Sub-NUMA cluster to confirm the resource ID is RDT_RESOURCE_L3.
--- arch/x86/kernel/cpu/resctrl/internal.h | 13 +++++++++++ arch/x86/kernel/cpu/resctrl/core.c | 15 +++++++++++++ arch/x86/kernel/cpu/resctrl/intel_aet.c | 29 +++++++++++++++++++++++++ fs/resctrl/ctrlmondata.c | 5 +++++ fs/resctrl/rdtgroup.c | 10 +++++++++ 5 files changed, 72 insertions(+)
arch/x86/kernel/cpu/resctrl/internal.h +13 -0
@@ -84,6 +84,14 @@ static inline struct rdt_hw_l3_mon_domain *resctrl_to_arch_mon_dom(struct rdt_l3 return container_of(r, struct rdt_hw_l3_mon_domain, d_resctrl); } +/** +* struct rdt_perf_pkg_mon_domain - CPUs sharing an package scoped resctrl monitor resource +* @hdr: common header for different domain types +*/ +struct rdt_perf_pkg_mon_domain { +struct rdt_domain_hdr hdr; +}; +/** * struct msr_param - set a range of MSRs from a domain * @res: The resource to use
@@ -222,6 +230,8 @@ bool intel_aet_get_events(void); void __exit intel_aet_exit(void);int intel_aet_read_event (int domid,u32 rmid,enum resctrl_event_id evtid,void *arch_priv,u64 *val); +void intel_aet_mon_domain_setup (int cpu,int id,struct rdt_resource *r, +struct list_head *add_pos);#else static inline bool intel_aet_get_events (void ) {return false; } static inline void __exit intel_aet_exit(void) { }
@@ -230,6 +240,9 @@ static inline int intel_aet_read_event(int domid, u32 rmid, enum resctrl_event_i {return -EINVAL ; } + +static inline void intel_aet_mon_domain_setup (int cpu,int id,struct rdt_resource *r, +struct list_head *add_pos) { }#endif #endif /* _ASM_X86_RESCTRL_INTERNAL_H */
arch/x86/kernel/cpu/resctrl/core.c +15 -0
@@ -573,6 +573,10 @@ static void domain_add_cpu_mon(int cpu, struct rdt_resource *r)if (!hdr)l3_mon_domain_setup (cpu, id, r, add_pos);break ; +case RDT_RESOURCE_PERF_PKG : +if (!hdr) +intel_aet_mon_domain_setup (cpu, id, r, add_pos); +break ;default :pr_warn_once ("Unknown resource rid=%d\n" , r->rid);break ;
@@ -635,6 +639,7 @@ static void domain_remove_cpu_ctrl(int cpu, struct rdt_resource *r)static void domain_remove_cpu_mon (int cpu,struct rdt_resource *r) {int id =get_domain_id_from_scope (cpu, r->mon_scope); +struct rdt_perf_pkg_mon_domain *pkgd;struct rdt_hw_l3_mon_domain *hw_dom;struct rdt_l3_mon_domain *d;struct rdt_domain_hdr *hdr;
@@ -670,6 +675,16 @@ static void domain_remove_cpu_mon(int cpu, struct rdt_resource *r)synchronize_rcu ();l3_mon_domain_free (hw_dom);break ; +case RDT_RESOURCE_PERF_PKG : +if (!domain_header_is_valid (hdr,RESCTRL_MON_DOMAIN ,RDT_RESOURCE_PERF_PKG )) +return ; + + pkgd = container_of(hdr, struct rdt_perf_pkg_mon_domain, hdr); +resctrl_offline_mon_domain (r, hdr); +list_del_rcu (&hdr->list); +synchronize_rcu (); +kfree (pkgd); +break ;default :pr_warn_once ("Unknown resource rid=%d\n" , r->rid);break ;
arch/x86/kernel/cpu/resctrl/intel_aet.c +29 -0
@@ -17,16 +17,21 @@#include <linux/compiler_types.h> #include <linux/container_of.h> #include <linux/cpu.h> +#include <linux/cpumask.h> #include <linux/err.h> #include <linux/errno.h> +#include <linux/gfp_types.h> #include <linux/init.h> #include <linux/intel_pmt_features.h> #include <linux/intel_vsec.h> #include <linux/io.h> #include <linux/overflow.h> #include <linux/printk.h> +#include <linux/rculist.h> +#include <linux/rcupdate.h> #include <linux/resctrl.h> #include <linux/resctrl_types.h> +#include <linux/slab.h> #include <linux/stddef.h> #include <linux/topology.h> #include <linux/types.h>
@@ -282,3 +287,27 @@ int intel_aet_read_event(int domid, u32 rmid, enum resctrl_event_id eventid,return valid ?0 : -EINVAL ; } + +void intel_aet_mon_domain_setup (int cpu,int id,struct rdt_resource *r, +struct list_head *add_pos) +{ +struct rdt_perf_pkg_mon_domain *d; +int err; + + d =kzalloc_node (sizeof (*d),GFP_KERNEL ,cpu_to_node (cpu)); +if (!d) +return ; + + d->hdr.id = id; + d->hdr.type =RESCTRL_MON_DOMAIN ; + d->hdr.rid = r->rid; +cpumask_set_cpu (cpu, &d->hdr.cpu_mask); +list_add_tail_rcu (&d->hdr.list, add_pos); + + err =resctrl_online_mon_domain (r, &d->hdr); +if (err) { +list_del_rcu (&d->hdr.list); +synchronize_rcu (); +kfree (d); + } +}
fs/resctrl/ctrlmondata.c +5 -0
@@ -712,6 +712,11 @@ int rdtgroup_mondata_show(struct seq_file *m, void *arg)if (md->sum) {struct rdt_l3_mon_domain *d; +if (WARN_ON_ONCE (resid !=RDT_RESOURCE_L3 )) { + ret = -EINVAL ; + goto out; + } +/* * This file requires summing across all domains that share * the L3 cache id that was provided in the "domid" field of the
fs/resctrl/rdtgroup.c +10 -0
@@ -3040,6 +3040,9 @@ static struct mon_data *mon_get_kn_priv(enum resctrl_res_level rid, int domid,lockdep_assert_held (&rdtgroup_mutex); +if (WARN_ON_ONCE (do_sum && rid !=RDT_RESOURCE_L3 )) +return NULL ; +list_for_each_entry (priv, &mon_data_kn_priv_list, list) {if (priv->rid == rid && priv->domid == domid && priv->sum == do_sum && priv->evt == mevt)
@@ -4227,6 +4230,9 @@ void resctrl_offline_mon_domain(struct rdt_resource *r, struct rdt_domain_hdr *hif (resctrl_mounted &&resctrl_arch_mon_capable ())rmdir_mondata_subdir_allrdtgrp (r, hdr); +if (r->rid !=RDT_RESOURCE_L3 ) + goto out_unlock; +if (!domain_header_is_valid (hdr,RESCTRL_MON_DOMAIN ,RDT_RESOURCE_L3 )) goto out_unlock;
@@ -4327,6 +4333,9 @@ int resctrl_online_mon_domain(struct rdt_resource *r, struct rdt_domain_hdr *hdrmutex_lock (&rdtgroup_mutex); +if (r->rid !=RDT_RESOURCE_L3 ) + goto mkdir; +if (!domain_header_is_valid (hdr,RESCTRL_MON_DOMAIN ,RDT_RESOURCE_L3 )) goto out_unlock;
@@ -4344,6 +4353,7 @@ int resctrl_online_mon_domain(struct rdt_resource *r, struct rdt_domain_hdr *hdrif (resctrl_is_mon_event_enabled (QOS_L3_OCCUP_EVENT_ID ))INIT_DELAYED_WORK (&d->cqm_limbo, cqm_handle_limbo); +mkdir: err =0 ;/* * If the filesystem is not mounted then only the default resource group
Signature · 1 line
2.51.0
Reinette Chatre
maintainerRe: [PATCH v11 21/31] x86/resctrl: Handle domain creation/deletion for RDT_RESOURCE_PERF_PKG
Hi Tony,
(subject prefix missing the resctrl fs changes)
Quoted 11 lines · Tony Luck
The L3 resource has several requirements for domains. There are per-domain structures that hold the 64-bit values of counters, and elements to keep track of the overflow and limbo threads.
None of these are needed for the
PERF_PKGresource. The hardware counters are wide enough that they do not wrap around for decades.Define a new
rdt_perf_pkg_mon_domainstructure which just consists of the standardrdt_domain_hdrto keep track of domain id and CPU mask.Support the
PERF_PKGresource in the CPU online/offline handlers.
Above can be seen from the patch. Would be helpful to highlight what this support involves since it is not obvious from changes that just adds gotos.
Quoted 3 lines
Add WARN checks to code that sums domains for Sub-NUMA cluster to confirm the resource ID is
RDT_RESOURCE_L3.
Above is clear from patch. Is there a "why" since it is not clear what these changes that are related to counter reading have to do with domain creation and deletion topic of this patch.
Quoted 64 lines
Signed-off-by Tony Luck <[email protected]>--- arch/x86/kernel/cpu/resctrl/internal.h | 13 +++++++++++ arch/x86/kernel/cpu/resctrl/core.c | 15 +++++++++++++ arch/x86/kernel/cpu/resctrl/intel_aet.c | 29 +++++++++++++++++++++++++ fs/resctrl/ctrlmondata.c | 5 +++++ fs/resctrl/rdtgroup.c | 10 +++++++++ 5 files changed, 72 insertions(+)arch/x86/kernel/cpu/resctrl/internal.h +13
-0@@ -84,6 +84,14 @@ static inline struct rdt_hw_l3_mon_domain *resctrl_to_arch_mon_dom(struct rdt_l3 return container_of(r, struct rdt_hw_l3_mon_domain, d_resctrl); } +/** +* struct rdt_perf_pkg_mon_domain - CPUs sharing an package scoped resctrl monitor resource +* @hdr: common header for different domain types +*/ +struct rdt_perf_pkg_mon_domain { +struct rdt_domain_hdr hdr; +}; +/** * struct msr_param - set a range of MSRs from a domain * @res: The resource to use @@ -222,6 +230,8 @@ bool intel_aet_get_events(void); void __exit intel_aet_exit(void);int intel_aet_read_event (int domid,u32 rmid,enum resctrl_event_id evtid,void *arch_priv,u64 *val); +void intel_aet_mon_domain_setup (int cpu,int id,struct rdt_resource *r, +struct list_head *add_pos);#else static inline bool intel_aet_get_events (void ) {return false; } static inline void __exit intel_aet_exit(void) { }@@ -230,6 +240,9 @@ static inline int intel_aet_read_event(int domid, u32 rmid, enum resctrl_event_i {return -EINVAL ; } + +static inline void intel_aet_mon_domain_setup (int cpu,int id,struct rdt_resource *r, +struct list_head *add_pos) { }#endif #endif /* _ASM_X86_RESCTRL_INTERNAL_H */ arch/x86/kernel/cpu/resctrl/core.c +4
-0@@ -573,6 +573,10 @@ static void domain_add_cpu_mon(int cpu, struct rdt_resource *r)if (!hdr)l3_mon_domain_setup (cpu, id, r, add_pos);break ; +case RDT_RESOURCE_PERF_PKG : +if (!hdr) +intel_aet_mon_domain_setup (cpu, id, r, add_pos); +break ;default :pr_warn_once ("Unknown resource rid=%d\n" , r->rid);break ;
arch/x86/kernel/cpu/resctrl/core.c static void domain_remove_cpu_ctrl(int cpu, struct rdt_resource *r) · line 642
Quoted 4 lines
arch/x86/kernel/cpu/resctrl/core.c +1
-0@@ -635,6 +639,7 @@ static void domain_remove_cpu_ctrl(int cpu, struct rdt_resource *r)static void domain_remove_cpu_mon (int cpu,struct rdt_resource *r) {int id =get_domain_id_from_scope (cpu, r->mon_scope); +struct rdt_perf_pkg_mon_domain *pkgd;
Please move this declaration to resource specific case statement block.
Quoted 54 lines
arch/x86/kernel/cpu/resctrl/core.c +10
-0struct rdt_hw_l3_mon_domain *hw_dom;struct rdt_l3_mon_domain *d;struct rdt_domain_hdr *hdr;@@ -670,6 +675,16 @@ static void domain_remove_cpu_mon(int cpu, struct rdt_resource *r)synchronize_rcu ();l3_mon_domain_free (hw_dom);break ; +case RDT_RESOURCE_PERF_PKG : +if (!domain_header_is_valid (hdr,RESCTRL_MON_DOMAIN ,RDT_RESOURCE_PERF_PKG )) +return ; + + pkgd = container_of(hdr, struct rdt_perf_pkg_mon_domain, hdr); +resctrl_offline_mon_domain (r, hdr); +list_del_rcu (&hdr->list); +synchronize_rcu (); +kfree (pkgd); +break ;default :pr_warn_once ("Unknown resource rid=%d\n" , r->rid);break ;arch/x86/kernel/cpu/resctrl/intel_aet.c +9
-0@@ -17,16 +17,21 @@#include <linux/compiler_types.h> #include <linux/container_of.h> #include <linux/cpu.h> +#include <linux/cpumask.h> #include <linux/err.h> #include <linux/errno.h> +#include <linux/gfp_types.h> #include <linux/init.h> #include <linux/intel_pmt_features.h> #include <linux/intel_vsec.h> #include <linux/io.h> #include <linux/overflow.h> #include <linux/printk.h> +#include <linux/rculist.h> +#include <linux/rcupdate.h> #include <linux/resctrl.h> #include <linux/resctrl_types.h> +#include <linux/slab.h> #include <linux/stddef.h> #include <linux/topology.h> #include <linux/types.h> @@ -282,3 +287,27 @@ int intel_aet_read_event(int domid, u32 rmid, enum resctrl_event_id eventid,return valid ?0 : -EINVAL ; } + +void intel_aet_mon_domain_setup (int cpu,int id,struct rdt_resource *r, +struct list_head *add_pos) +{
arch/x86/kernel/cpu/resctrl/intel_aet.c int intel_aet_read_event(int domid, u32 rmid, enum resctrl_event_id eventid, · line 303
Quoted 10 lines
arch/x86/kernel/cpu/resctrl/intel_aet.c +10
-0+struct rdt_perf_pkg_mon_domain *d; +int err; + + d =kzalloc_node (sizeof (*d),GFP_KERNEL ,cpu_to_node (cpu)); +if (!d) +return ; + + d->hdr.id = id; + d->hdr.type =RESCTRL_MON_DOMAIN ; + d->hdr.rid = r->rid;
In this series l3_mon_domain_setup() received an update to hardcode the resource. Can this do the same to be consistent?
Reinette
Tony Luck
maintainerPATCH v11 22/31 x86/resctrl: Add energy/perf choices to rdt boot option
In reply to Tony LuckLegacy resctrl features are enumerated by X86_FEATURE_* flags. These may be overridden by quirks to disable features in the case of errata. Users can use kernel command line options to either disable a feature, or to force enable a feature that was disabled by a quirk.
Provide similar functionality for hardware features that do not have an X86_FEATURE_* flag. Unlike other features that are tied to X86_FEATURE_* flags, these must be queried by name. Add rdt_is_feature_enabled() to check whether quirks or kernel command line have disabled a feature.
Users may force a feature to be disabled. E.g. "rdt=!perf" will ensure that none of the perf telemetry events are enabled.
Resctrl architecture code may disable a feature that does not provide full functionality. Users may override that decision. E.g. "rdt=energy" will enable any available energy telemetry events even if they do not provide full functionality.
--- .../admin-guide/kernel-parameters.txt | 2 +- arch/x86/kernel/cpu/resctrl/internal.h | 2 ++ arch/x86/kernel/cpu/resctrl/core.c | 29 +++++++++++++++++++ 3 files changed, 32 insertions(+), 1 deletion(-)
Documentation/admin-guide/kernel-parameters.txt +1 -1
@@ -6155,7 +6155,7 @@ rdt= [HW,X86,RDT] Turn on/off individual RDT features. List is: cmt, mbmtotal, mbmlocal, l3cat, l3cdp, l2cat, l2cdp, - mba, smba, bmec, abmc. + mba, smba, bmec, abmc, energy, perf. E.g. to turn on cmt and turn off mba use: rdt=cmt,!mba
arch/x86/kernel/cpu/resctrl/internal.h +2 -0
@@ -225,6 +225,8 @@ void __init intel_rdt_mbm_apply_quirk(void);void rdt_domain_reconfigure_cdp (struct rdt_resource *r);void resctrl_arch_mbm_cntr_assign_set_one (struct rdt_resource *r); +bool rdt_is_feature_enabled (char *name); +#ifdef CONFIG_X86_CPU_RESCTRL_INTEL_AET bool intel_aet_get_events (void ); void __exit intel_aet_exit(void);
arch/x86/kernel/cpu/resctrl/core.c +29 -0
@@ -766,6 +766,8 @@ enum {RDT_FLAG_SMBA ,RDT_FLAG_BMEC ,RDT_FLAG_ABMC , +RDT_FLAG_ENERGY , + RDT_FLAG_PERF, };#define RDT_OPT (idx, n, f) \
@@ -792,6 +794,8 @@ static struct rdt_options rdt_options[] __ro_after_init = {RDT_OPT (RDT_FLAG_SMBA ,"smba" ,X86_FEATURE_SMBA ),RDT_OPT (RDT_FLAG_BMEC ,"bmec" ,X86_FEATURE_BMEC ),RDT_OPT (RDT_FLAG_ABMC ,"abmc" ,X86_FEATURE_ABMC ), +RDT_OPT (RDT_FLAG_ENERGY ,"energy" ,0 ), + RDT_OPT(RDT_FLAG_PERF, "perf", 0), };#define NUM_RDT_OPTIONS ARRAY_SIZE(rdt_options)
@@ -841,6 +845,31 @@ bool rdt_cpu_has(int flag)return ret; } +/* +* Hardware features that do not have X86_FEATURE_* bits. There is no +* "hardware does not support this at all" case. Assume that the caller +* has already determined that hardware support is present and just needs +* to check if the feature has been disabled by a quirk that has not been +* overridden by a command line option. +*/ +bool rdt_is_feature_enabled (char *name) +{ +struct rdt_options *o; +bool ret = true; + +for (o = rdt_options; o < &rdt_options[NUM_RDT_OPTIONS ]; o++) { +if (!strcmp (name, o->name)) { +if (o->force_off) + ret = false; +if (o->force_on) + ret = true; +break ; + } + } + +return ret; +} +bool resctrl_arch_is_evt_configurable (enum resctrl_event_id evt) {if (!rdt_cpu_has (X86_FEATURE_BMEC ))
Signature · 1 line
2.51.0
Tony Luck
maintainerPATCH v11 23/31 x86/resctrl: Handle number of RMIDs supported by telemetry resources
In reply to Tony LuckThere are now three meanings for "number of RMIDs":
1) The number for legacy features enumerated by CPUID leaf 0xF. This is the maximum number of distinct values that can be loaded into the IA32_PQR_ASSOC MSR. Note that systems with Sub-NUMA Cluster mode enabled will force scaling down the CPUID enumerated value by the number of SNC nodes per L3-cache.
2) The number of registers in MMIO space for each event. This is enumerated in the XML files and is the value initialized into event_group::num_rmids.
3) The number of "hardware counters" (this isn't a strictly accurate description of how things work, but serves as a useful analogy that does describe the limitations) feeding to those MMIO registers. This is enumerated in telemetry_region::num_rmids returned from the call to intel_pmt_get_regions_by_feature()
Event groups with insufficient "hardware counters" to track all RMIDs are difficult for users to use, since the system may reassign "hardware counters" at any time. This means that users cannot reliably collect two consecutive event counts to compute the rate at which events are occurring.
Introduce rdt_set_feature_disabled() to mark any under-resourced event groups (those with telemetry_region::num_rmids < event_group::num_rmids) as unusable. Note that the rdt_options[] structure must now be writable at run-time. The request to disable will be overridden if the user explicitly requests to enable using the "rdt=" Linux boot argument. This will result in the available number of monitoring resource groups being limited by the under-resourced event groups.
Scan all enabled event groups and assign the RDT_RESOURCE_PERF_PKG resource "num_rmids" value to the smallest of these values as this value will be used later to compare against the number of RMIDs supported by other resources to determine how many monitoring resource groups are supported.
N.B. Change type of rdt_resource::num_rmid to u32 to match type of event_group::num_rmids so that min(r->num_rmid, e->num_rmids) won't complain about mixing signed and unsigned types. Print r->num_rmid as unsigned value in rdt_num_rmids_show().
--- include/linux/resctrl.h | 2 +- arch/x86/kernel/cpu/resctrl/internal.h | 2 ++ arch/x86/kernel/cpu/resctrl/core.c | 18 +++++++++- arch/x86/kernel/cpu/resctrl/intel_aet.c | 48 +++++++++++++++++++++++++ fs/resctrl/rdtgroup.c | 2 +- 5 files changed, 69 insertions(+), 3 deletions(-)
include/linux/resctrl.h +1 -1
@@ -292,7 +292,7 @@ enum resctrl_schema_fmt {* events of monitor groups created via mkdir. */ struct resctrl_mon { -int num_rmid; +u32 num_rmid;unsigned int mbm_cfg_mask;int num_mbm_cntrs;bool mbm_cntr_assignable;
arch/x86/kernel/cpu/resctrl/internal.h +2 -0
@@ -227,6 +227,8 @@ void resctrl_arch_mbm_cntr_assign_set_one(struct rdt_resource *r);bool rdt_is_feature_enabled (char *name); +void rdt_set_feature_disabled (char *name); +#ifdef CONFIG_X86_CPU_RESCTRL_INTEL_AET bool intel_aet_get_events (void ); void __exit intel_aet_exit(void);
arch/x86/kernel/cpu/resctrl/core.c +17 -1
@@ -782,7 +782,7 @@ struct rdt_options {bool force_off, force_on; }; -static struct rdt_options rdt_options[]__ro_after_init = { +static struct rdt_options rdt_options[] = {RDT_OPT (RDT_FLAG_CMT ,"cmt" ,X86_FEATURE_CQM_OCCUP_LLC ),RDT_OPT (RDT_FLAG_MBM_TOTAL ,"mbmtotal" ,X86_FEATURE_CQM_MBM_TOTAL ),RDT_OPT (RDT_FLAG_MBM_LOCAL ,"mbmlocal" ,X86_FEATURE_CQM_MBM_LOCAL ),
@@ -845,6 +845,22 @@ bool rdt_cpu_has(int flag)return ret; } +/* +* Can be called during feature enumeration if sanity check of +* a feature's parameters indicates problems with the feature. +*/ +void rdt_set_feature_disabled (char *name) +{ +struct rdt_options *o; + +for (o = rdt_options; o < &rdt_options[NUM_RDT_OPTIONS ]; o++) { +if (!strcmp (name, o->name)) { + o->force_off = true; +return ; + } + } +} +/* * Hardware features that do not have X86_FEATURE_* bits. There is no * "hardware does not support this at all" case. Assume that the caller
arch/x86/kernel/cpu/resctrl/intel_aet.c +48 -0
111 lines not on this page · raw message
fs/resctrl/rdtgroup.c +1 -1
@@ -1135,7 +1135,7 @@ static int rdt_num_rmids_show(struct kernfs_open_file *of, {struct rdt_resource *r =rdt_kn_parent_priv (of->kn); -seq_printf (seq,"%d\n" , r->mon.num_rmid); +seq_printf (seq,"%u\n" , r->mon.num_rmid);return 0 ; }
Signature · 1 line
2.51.0
Reinette Chatre
maintainerRe: [PATCH v11 23/31] x86/resctrl: Handle number of RMIDs supported by telemetry resources
Hi Tony,
(nit in subject ... "resources" -> "resource" ... with caveat that the term "telemetry resource" is not used much at all in this series)
Quoted 5 lines · Tony Luck
There are now three meanings for "number of RMIDs":
1) The number for legacy features enumerated by CPUID leaf
0xF. This is the maximum number of distinct values that can be loaded into theIA32_PQR_ASSOCMSR. Note that systems with Sub-NUMA Cluster mode enabled
"the IA32_PQR_ASSOC MSR" -> "MSR_IA32_PQR_ASSOC"
Quoted 13 lines
will force scaling down the CPUID enumerated value by the number of SNC nodes per L3-cache.
2) The number of registers in MMIO space for each event. This is enumerated in the XML files and is the value initialized into
event_group::num_rmids.3) The number of "hardware counters" (this isn't a strictly accurate description of how things work, but serves as a useful analogy that does describe the limitations) feeding to those MMIO registers. This is enumerated in
telemetry_region::num_rmidsreturned from the call tointel_pmt_get_regions_by_feature()
Quoted 8 lines
Event groups with insufficient "hardware counters" to track all RMIDs are difficult for users to use, since the system may reassign "hardware counters" at any time. This means that users cannot reliably collect two consecutive event counts to compute the rate at which events are occurring.
Introduce
rdt_set_feature_disabled()to mark any under-resourced event groups (those withtelemetry_region::num_rmids<event_group::num_rmids)
Would it be more accurate to say
"(those with telemetry_region::num_rmids < event_group::num_rmids for any
of the event group's telemetry regions)"
Quoted 2 lines
as unusable. Note that the rdt_options[] structure must now be writable at run-time. The request to disable will be overridden if the user
"Override the request ..."?
Quoted 3 lines
explicitly requests to enable using the "rdt=" Linux boot argument. This will result in the available number of monitoring resource groups being limited by the under-resourced event groups.
needs imperative ... how about something like (for text starting with "The request to disable ..."):
Limit an event group's number of possible monitor resource groups to the lowest number of "hardware counters" if the user explicitly requests to enable an under-resourced event group.
...
Quoted 28 lines
arch/x86/kernel/cpu/resctrl/intel_aet.c +20
-0@@ -156,21 +168,57 @@ static void mark_telem_region_unusable(struct telemetry_region *tr) tr->addr =NULL ; } +static bool all_regions_have_sufficient_rmid (struct event_group *e,struct pmt_feature_group *p) +{ +struct telemetry_region *tr; + +for (int i =0 ; i < p->count; i++) { + tr = &p->regions[i]; +if (skip_telem_region (tr, e)) +continue ; + +if (tr->num_rmids < e->num_rmids) +return false; + } + +return true; +} +static bool enable_events (struct event_group *e,struct pmt_feature_group *p) { +struct rdt_resource *r = &rdt_resources_all[RDT_RESOURCE_PERF_PKG ].r_resctrl;bool usable_events = false; +/* Disable feature if insufficient RMIDs */ +if (!all_regions_have_sufficient_rmid (e, p)) +rdt_set_feature_disabled (e->name);
arch/x86/kernel/cpu/resctrl/intel_aet.c static void mark_telem_region_unusable(struct telemetry_region *tr) · line 204
Quoted 10 lines
arch/x86/kernel/cpu/resctrl/intel_aet.c +5
-0+ +/* User can override above disable from kernel command line */ +if (!rdt_is_feature_enabled (e->name)) +return false; +for (int i =0 ; i < p->count; i++) {if (skip_telem_region (&p->regions[i], e)) {mark_telem_region_unusable (&p->regions[i]);continue ; }
It is unexpected to me that skip_telem_region() needs to be run twice with second time marking regions as unusable. I think it will be simpler to just run skip_telem_region() once to determine which telemetry regions are unusable, mark them as such at that time, and from that point forward just interact with the usable telemetry regions?
arch/x86/kernel/cpu/resctrl/intel_aet.c static void mark_telem_region_unusable(struct telemetry_region *tr) · line 208
Quoted 4 lines
arch/x86/kernel/cpu/resctrl/intel_aet.c +4
-0+ +/* +* e->num_rmids only adjusted lower if user forces an unusable +* region to be usable
In this function usable/unusable regions have a distinct meaning that is different from what this comment intends since insufficient rmid does not make a region "unusable" per skip_telem_region(). Perhaps something like:
e->num_rmids only adjusted lower if user (via rdt= kernel parameter) forces an event group with insufficient RMID to be enabled.
Reinette
Tony Luck
maintainerPATCH v11 24/31 fs/resctrl: Move allocation/free of closid_num_dirty_rmid[]
In reply to Tony Luckclosid_num_dirty_rmid[] is allocated in dom_data_init() during resctrl initialization and freed by dom_data_exit() during resctrl exit giving it the same life cycle as rmid_ptrs[].
Move closid_num_dirty_rmid[] allocaction/free out to resctrl_l3_mon_resource_init() and resctrl_l3_mon_resource_exit() in preparation for rmid_ptrs[] to be allocated on resctrl mount in support of the new telemetry events.
Keep the rdtgroup_mutex protection around the allocation/free of closid_num_dirty_rmid[] as ARM needs this to guarantee memory ordering.
--- fs/resctrl/monitor.c | 77 ++++++++++++++++++++++++++++---------------- 1 file changed, 49 insertions(+), 28 deletions(-)
fs/resctrl/monitor.c +49 -28
@@ -883,36 +883,14 @@ void mbm_setup_overflow_handler(struct rdt_l3_mon_domain *dom, unsigned long delstatic int dom_data_init (struct rdt_resource *r) {u32 idx_limit =resctrl_arch_system_num_rmid_idx (); -u32 num_closid =resctrl_arch_get_num_closid (r);struct rmid_entry *entry =NULL ;int err =0 , i;u32 idx;mutex_lock (&rdtgroup_mutex); -if (IS_ENABLED (CONFIG_RESCTRL_RMID_DEPENDS_ON_CLOSID )) { -u32 *tmp; - -/* -* If the architecture hasn't provided a sanitised value here, -* this may result in larger arrays than necessary. Resctrl will -* use a smaller system wide value based on the resources in -* use. -*/ - tmp =kcalloc (num_closid,sizeof (*tmp),GFP_KERNEL ); -if (!tmp) { - err = -ENOMEM ; - goto out_unlock; - } - - closid_num_dirty_rmid = tmp; - } rmid_ptrs =kcalloc (idx_limit,sizeof (struct rmid_entry ),GFP_KERNEL );if (!rmid_ptrs) { -if (IS_ENABLED (CONFIG_RESCTRL_RMID_DEPENDS_ON_CLOSID )) { -kfree (closid_num_dirty_rmid); - closid_num_dirty_rmid =NULL ; - } err = -ENOMEM ; goto out_unlock; }
@@ -948,11 +926,6 @@ static void dom_data_exit(struct rdt_resource *r)if (!r->mon_capable) goto out_unlock; -if (IS_ENABLED (CONFIG_RESCTRL_RMID_DEPENDS_ON_CLOSID )) { -kfree (closid_num_dirty_rmid); - closid_num_dirty_rmid =NULL ; - } -kfree (rmid_ptrs); rmid_ptrs =NULL ;
@@ -1789,6 +1762,43 @@ ssize_t mbm_L3_assignments_write(struct kernfs_open_file *of, char *buf,return ret ?: nbytes; } +static int closid_num_dirty_rmid_alloc (struct rdt_resource *r) +{ +if (IS_ENABLED (CONFIG_RESCTRL_RMID_DEPENDS_ON_CLOSID )) { +u32 num_closid =resctrl_arch_get_num_closid (r); +u32 *tmp; + +/* For ARM memory ordering access to closid_num_dirty_rmid */ +mutex_lock (&rdtgroup_mutex); + +/* +* If the architecture hasn't provided a sanitised value here, +* this may result in larger arrays than necessary. Resctrl will +* use a smaller system wide value based on the resources in +* use. +*/ + tmp =kcalloc (num_closid,sizeof (*tmp),GFP_KERNEL ); +if (!tmp) +return -ENOMEM ; + + closid_num_dirty_rmid = tmp; + +mutex_unlock (&rdtgroup_mutex); + } + +return 0 ; +} + +static void closid_num_dirty_rmid_free (void ) +{ +if (IS_ENABLED (CONFIG_RESCTRL_RMID_DEPENDS_ON_CLOSID )) { +mutex_lock (&rdtgroup_mutex); +kfree (closid_num_dirty_rmid); + closid_num_dirty_rmid =NULL ; +mutex_unlock (&rdtgroup_mutex); + } +} +/** * resctrl_l3_mon_resource_init() - Initialise global monitoring structures. *
@@ -1809,10 +1819,16 @@ int resctrl_l3_mon_resource_init(void)if (!r->mon_capable)return 0 ; - ret =dom_data_init (r); + ret =closid_num_dirty_rmid_alloc (r);if (ret)return ret; + ret =dom_data_init (r); +if (ret) { +closid_num_dirty_rmid_free (); +return ret; + } +if (resctrl_arch_is_evt_configurable (QOS_L3_MBM_TOTAL_EVENT_ID )) { mon_event_all[QOS_L3_MBM_TOTAL_EVENT_ID ].configurable = true; resctrl_file_fflags_init("mbm_total_bytes_config",
@@ -1857,5 +1873,10 @@ void resctrl_l3_mon_resource_exit(void) {struct rdt_resource *r =resctrl_arch_get_resource (RDT_RESOURCE_L3 ); +if (!r->mon_capable) +return ; + +closid_num_dirty_rmid_free (); +dom_data_exit (r); }
Signature · 1 line
2.51.0
Reinette Chatre
maintainerRe: [PATCH v11 24/31] fs/resctrl: Move allocation/free of closid_num_dirty_rmid[]
Hi Tony,
Quoted 4 lines · Tony Luck
closid_num_dirty_rmid[] is allocated in
dom_data_init()during resctrl initialization and freed bydom_data_exit()during resctrl exit giving it the same life cycle as rmid_ptrs[].
Quoted 8 lines
Move closid_num_dirty_rmid[] allocaction/free out to
resctrl_l3_mon_resource_init()andresctrl_l3_mon_resource_exit()in preparation for rmid_ptrs[] to be allocated on resctrl mount in support of the new telemetry events.Keep the
rdtgroup_mutexprotection around the allocation/free of closid_num_dirty_rmid[] as ARM needs this to guarantee memory ordering.
I think this is heavy on describing the code that we were asked to avoid. I amended the changelog below in an attempt to address this, please feel free to improve:
closid_num_dirty_rmid[] and rmid_ptrs[] are allocated together
during resctrl initialization and freed together during resctrl exit.
Telemetry events are enumerated on resctrl mount so only at resctrl
mount will the number of RMID supported by all monitoring resources
and needed as size for rmid_ptrs[] be known.
Separate closid_num_dirty_rmid[] and rmid_ptrs[] allocation and free
in preparation for rmid_ptrs[] to be allocated on resctrl mount.
Keep the rdtgroup_mutex protection around the allocation and free of
closid_num_dirty_rmid[] as ARM needs this to guarantee memory
ordering. Quoted 3 lines
Signed-off-by Tony Luck <[email protected]>---
Patch looks good to me.
Reinette
Tony Luck
maintainerPATCH v11 25/31 fs,x86/resctrl: Compute number of RMIDs as minimum across resources
In reply to Tony Luckresctrl assumes that only the L3 resource supports monitor events, so it simply takes the rdt_resource::num_rmid from RDT_RESOURCE_L3 as the system's number of RMIDs.
The addition of telemetry events in a different resource breaks that assumption.
Compute the number of available RMIDs as the minimum value across all mon_capable resources (analogous to how the number of CLOSIDs is computed across alloc_capable resources).
Note that mount time enumeration of the telemetry resource means that this number can be reduced. If this happens, then some memory will be wasted as the allocations for rdt_l3_mon_domain::mbm_states[] and rdt_l3_mon_domain::rmid_busy_llc created during resctrl initialization will be larger than needed.
--- arch/x86/kernel/cpu/resctrl/core.c | 15 +++++++++++++-- fs/resctrl/rdtgroup.c | 6 ++++++ 2 files changed, 19 insertions(+), 2 deletions(-)
arch/x86/kernel/cpu/resctrl/core.c +13 -2
@@ -110,12 +110,23 @@ struct rdt_hw_resource rdt_resources_all[RDT_NUM_RESOURCES] = { }, }; +/** +* resctrl_arch_system_num_rmid_idx - Compute number of supported RMIDs +* (minimum across all mon_capable resource) +* +* Return: Number of supported RMIDs at time of call. Note that mount time +* enumeration of resources may reduce the number. +*/ u32 resctrl_arch_system_num_rmid_idx (void ) { -struct rdt_resource *r = &rdt_resources_all[RDT_RESOURCE_L3 ].r_resctrl; +u32 num_rmids =U32_MAX ; +struct rdt_resource *r; + +for_each_mon_capable_rdt_resource (r) + num_rmids =min (num_rmids, r->mon.num_rmid);/* RMID are independent numbers for x86. num_rmid_idx == num_rmid */ -return r->mon.num_rmid; +return num_rmids ==U32_MAX ?0 : num_rmids; }struct rdt_resource *resctrl_arch_get_resource (enum resctrl_res_level l)
fs/resctrl/rdtgroup.c +6 -0
@@ -4268,6 +4268,12 @@ void resctrl_offline_mon_domain(struct rdt_resource *r, struct rdt_domain_hdr *h* During boot this may be called before global allocations have been made by * resctrl_l3_mon_resource_init(). * +* Called during CPU online that may run as soon as CPU online callbacks +* are set up during resctrl initialization. The number of supported RMIDs +* may be reduced if additional mon_capable resources are enumerated +* at mount time. This means the rdt_l3_mon_domain::mbm_states[] and +* rdt_l3_mon_domain::rmid_busy_llc allocations may be larger than needed. +* * Returns 0 for success, or -ENOMEM. */ static int domain_setup_l3_mon_state (struct rdt_resource *r,struct rdt_l3_mon_domain *d)
Signature · 1 line
2.51.0
Reinette Chatre
maintainerRe: [PATCH v11 25/31] fs,x86/resctrl: Compute number of RMIDs as minimum across resources
Hi Tony,
Please use a consistent subject prefix (x86,fs/resctrl) when patch changes both arch and fs code.
Quoted 11 lines · Tony Luck
resctrl assumes that only the L3 resource supports monitor events, so it simply takes the
rdt_resource::num_rmidfromRDT_RESOURCE_L3as the system's number of RMIDs.The addition of telemetry events in a different resource breaks that assumption.
Compute the number of available RMIDs as the minimum value across all
mon_capableresources (analogous to how the number of CLOSIDs is computed acrossalloc_capableresources).
Quoted 8 lines
Note that mount time enumeration of the telemetry resource means that this number can be reduced. If this happens, then some memory will be wasted as the allocations for rdt_l3_mon_domain::mbm_states[] and
rdt_l3_mon_domain::rmid_busy_llccreated during resctrl initialization will be larger than needed.Signed-off-by Tony Luck <[email protected]>---
| Reviewed-by: Reinette Chatre <[email protected]>
Reinette
Tony Luck
maintainerPATCH v11 26/31 fs/resctrl: Move RMID initialization to first mount
In reply to Tony LuckL3 monitor features are enumerated during resctrl initialization and rmid_ptrs[] that tracks all RMIDs and depends on the number of supported RMIDs is allocated during this time.
Telemetry monitor features are enumerated during first resctrl mount and may support a different number of RMIDs compared to L3 monitor features.
Delay allocation and initialization of rmid_ptrs[] until first mount. Since the number of RMIDs cannot change on later mounts, keep the same set of rmid_ptrs[] until resctrl_exit(). This is required because the limbo handler keeps running after resctrl is unmounted and may likely need to access rmid_ptrs[] as it keeps tracking busy RMIDs after unmount.
Rename routines to match what they now do:dom_data_init() -> setup_rmid_lru_list()dom_data_exit() -> free_rmid_lru_list()
--- fs/resctrl/internal.h | 4 ++++ fs/resctrl/monitor.c | 50 +++++++++++++++++++------------------------ fs/resctrl/rdtgroup.c | 5 +++++ 3 files changed, 31 insertions(+), 28 deletions(-)
fs/resctrl/internal.h +4 -0
@@ -369,6 +369,10 @@ int closids_supported(void);void closid_free (int closid); +int setup_rmid_lru_list (void ); + +void free_rmid_lru_list (void ); +int alloc_rmid (u32 closid);void free_rmid (u32 closid,u32 rmid);
fs/resctrl/monitor.c +22 -28
97 lines not on this page · raw message
fs/resctrl/rdtgroup.c +5 -0
@@ -2734,6 +2734,10 @@ static int rdt_get_tree(struct fs_context *fc) goto out; } + ret =setup_rmid_lru_list (); +if (ret) + goto out; + ret =rdtgroup_setup_root (ctx);if (ret) goto out;
@@ -4568,4 +4572,5 @@ void resctrl_exit(void)*/ resctrl_l3_mon_resource_exit (); +free_rmid_lru_list (); }
Signature · 1 line
2.51.0
Reinette Chatre
maintainerRe: [PATCH v11 26/31] fs/resctrl: Move RMID initialization to first mount
Hi Tony,
Quoted 44 lines · Tony Luck
L3 monitor features are enumerated during resctrl initialization and rmid_ptrs[] that tracks all RMIDs and depends on the number of supported RMIDs is allocated during this time.
Telemetry monitor features are enumerated during first resctrl mount and may support a different number of RMIDs compared to L3 monitor features.
Delay allocation and initialization of rmid_ptrs[] until first mount. Since the number of RMIDs cannot change on later mounts, keep the same set of rmid_ptrs[] until
resctrl_exit(). This is required because the limbo handler keeps running after resctrl is unmounted and may likely need to access rmid_ptrs[] as it keeps tracking busy RMIDs after unmount.Rename routines to match what they now do:
dom_data_init()->setup_rmid_lru_list()dom_data_exit()->free_rmid_lru_list()Signed-off-by Tony Luck <[email protected]>--- fs/resctrl/internal.h | 4 ++++ fs/resctrl/monitor.c | 50 +++++++++++++++++++------------------------ fs/resctrl/rdtgroup.c | 5 +++++ 3 files changed, 31 insertions(+), 28 deletions(-)fs/resctrl/internal.h +4
-0@@ -369,6 +369,10 @@ int closids_supported(void);void closid_free (int closid); +int setup_rmid_lru_list (void ); + +void free_rmid_lru_list (void ); +int alloc_rmid (u32 closid);void free_rmid (u32 closid,u32 rmid);fs/resctrl/monitor.c +0
-0
fs/resctrl/monitor.c void mbm_setup_overflow_handler(struct rdt_l3_mon_domain *dom, unsigned long del · line 885
Quoted 7 lines
fs/resctrl/monitor.c +1
-1@@ -880,20 +880,27 @@ void mbm_setup_overflow_handler(struct rdt_l3_mon_domain *dom, unsigned long delschedule_delayed_work_on (cpu, &dom->mbm_over, delay); } -static int dom_data_init (struct rdt_resource *r) +int setup_rmid_lru_list (void ) {u32 idx_limit =resctrl_arch_system_num_rmid_idx ();
Should this be done after the resctrl_arch_mon_capable() check? Seems unnecessary to determine the number of RMID on a system that does not support monitoring.
fs/resctrl/monitor.c void mbm_setup_overflow_handler(struct rdt_l3_mon_domain *dom, unsigned long del · line 896
Quoted 13 lines
fs/resctrl/monitor.c +8
-2struct rmid_entry *entry =NULL ; -int err =0 , i;u32 idx; +int i; -mutex_lock (&rdtgroup_mutex); +if (!resctrl_arch_mon_capable ()) +return 0 ; + +/* +* Called on every mount, but the number of RMIDs cannot change +* after the first mount, so keep using the same set of rmid_ptrs[] +* until resctrl_exit().
Could you please add the motivation that limbo handler accesses rmid_ptrs[] after unmount? This seems a much stronger motivation about why this is not freed on unmount and thus valuable for anyone that wants to refactor this later.
Quoted 4 lines
fs/resctrl/monitor.c +3
-0+*/ +if (rmid_ptrs) +return 0 ;
fs/resctrl/monitor.c void mbm_setup_overflow_handler(struct rdt_l3_mon_domain *dom, unsigned long del · line 906
Quoted 10 lines
fs/resctrl/monitor.c +2
-4rmid_ptrs =kcalloc (idx_limit,sizeof (struct rmid_entry ),GFP_KERNEL ); -if (!rmid_ptrs) { - err = -ENOMEM ; - goto out_unlock; - } +if (!rmid_ptrs) +return -ENOMEM ;for (i =0 ; i < idx_limit; i++) { entry = &rmid_ptrs[i];
Reinette
Tony Luck
maintainerPATCH v11 27/31 x86/resctrl: Enable RDT_RESOURCE_PERF_PKG
In reply to Tony LuckMark the RDT_RESOURCE_PERF_PKG resource as mon_capable and set the global rdt_mon_capable flag.
Call domain_add_cpu_mon() for each online CPU to allocate all domains for the RDT_RESOURCE_PERF_PKG since they were not created during resctrl initialization because of the enumeration delay until first mount.
--- arch/x86/kernel/cpu/resctrl/core.c | 17 ++++++++++++++++- arch/x86/kernel/cpu/resctrl/intel_aet.c | 5 +++++ 2 files changed, 21 insertions(+), 1 deletion(-)
arch/x86/kernel/cpu/resctrl/core.c +16 -1
@@ -755,14 +755,29 @@ static int resctrl_arch_offline_cpu(unsigned int cpu)void resctrl_arch_pre_mount (void ) { +struct rdt_resource *r = &rdt_resources_all[RDT_RESOURCE_PERF_PKG ].r_resctrl;static atomic_t only_once =ATOMIC_INIT (0 ); -int old =0 ; +int cpu, old =0 ;if (!atomic_try_cmpxchg (&only_once, &old,1 ))return ;if (!intel_aet_get_events ())return ; + +if (!r->mon_capable) +return ; + +/* +* Late discovery of telemetry events means the domains for the +* resource were not built. Do that now. +*/ +cpus_read_lock (); +mutex_lock (&domain_list_lock); +for_each_online_cpu (cpu) +domain_add_cpu_mon (cpu, r); +mutex_unlock (&domain_list_lock); +cpus_read_unlock (); }enum {
arch/x86/kernel/cpu/resctrl/intel_aet.c +5 -0
@@ -214,6 +214,9 @@ static bool enable_events(struct event_group *e, struct pmt_feature_group *p)if (!usable_events)return false; + r->mon_capable = true; + rdt_mon_capable = true; +if (r->mon.num_rmid) r->mon.num_rmid =min (r->mon.num_rmid, e->num_rmids);else
@@ -223,6 +226,8 @@ static bool enable_events(struct event_group *e, struct pmt_feature_group *p)resctrl_enable_mon_event (e->evts[j].id, true, e->evts[j].bin_bits, &e->evts[j]); +pr_info ("%s %s monitoring detected\n" , r->name, e->name); +return true; }
Signature · 1 line
2.51.0
Reinette Chatre
maintainerRe: [PATCH v11 27/31] x86/resctrl: Enable RDT_RESOURCE_PERF_PKG
Hi Tony,
Quoted 2 lines · Tony Luck
Mark the
RDT_RESOURCE_PERF_PKGresource asmon_capableand set the globalrdt_mon_capableflag.
Above is clear from patch.
Quoted 4 lines
Call
domain_add_cpu_mon()for each online CPU to allocate all domains for theRDT_RESOURCE_PERF_PKGsince they were not created during resctrl initialization because of the enumeration delay until first mount.
Attempt at alternative:
Since telemetry events are enumerated on resctrl mount the RDT_RESOURCE_PERF_PKG resource is not considered "monitoring capable" during early resctrl initialization. This means that the domain list for RDT_RESOURCE_PERF_PKG is not built when the CPU hot plug notifiers are registered and run for the first time right after resctrl initialization.
Mark the RDT_RESOURCE_PERF_PKG as "monitoring capable" upon successful telemetry event enumeration to ensure future CPU hotplug events include this resource and initialize its domain list for CPUs that are already online.
Quoted 18 lines
Signed-off-by Tony Luck <[email protected]>--- arch/x86/kernel/cpu/resctrl/core.c | 17 ++++++++++++++++- arch/x86/kernel/cpu/resctrl/intel_aet.c | 5 +++++ 2 files changed, 21 insertions(+), 1 deletion(-)arch/x86/kernel/cpu/resctrl/core.c +1
-1@@ -755,14 +755,29 @@ static int resctrl_arch_offline_cpu(unsigned int cpu)void resctrl_arch_pre_mount (void ) { +struct rdt_resource *r = &rdt_resources_all[RDT_RESOURCE_PERF_PKG ].r_resctrl;static atomic_t only_once =ATOMIC_INIT (0 ); -int old =0 ;
arch/x86/kernel/cpu/resctrl/core.c static int resctrl_arch_offline_cpu(unsigned int cpu) · line 769
Quoted 10 lines
arch/x86/kernel/cpu/resctrl/core.c +4
-0+int cpu, old =0 ;if (!atomic_try_cmpxchg (&only_once, &old,1 ))return ;if (!intel_aet_get_events ())return ; + +if (!r->mon_capable) +return ;
Is this necessary? Can r->mon_capable be false if intel_aet_get_events() fails?
arch/x86/kernel/cpu/resctrl/core.c static int resctrl_arch_offline_cpu(unsigned int cpu) · line 775
Quoted 6 lines
arch/x86/kernel/cpu/resctrl/core.c +6
-0+ +/* +* Late discovery of telemetry events means the domains for the +* resource were not built. Do that now. +*/ +cpus_read_lock ();
hmmm ... until this point CPUs can come and go. This means that from the moment r->mon_capable is set resctrl_arch_online_cpu() may run and thus domain_add_cpu_mon() could be called twice for PERF_PKG? If all the second run does is set (again) a bit in the cpumask then that *may* be ok (but should be documented) but the flow does not seem safe to end up like that (more below)
Quoted 23 lines
arch/x86/kernel/cpu/resctrl/core.c +5
-0+mutex_lock (&domain_list_lock); +for_each_online_cpu (cpu) +domain_add_cpu_mon (cpu, r); +mutex_unlock (&domain_list_lock); +cpus_read_unlock (); }enum {arch/x86/kernel/cpu/resctrl/intel_aet.c +3
-0@@ -214,6 +214,9 @@ static bool enable_events(struct event_group *e, struct pmt_feature_group *p)if (!usable_events)return false; + r->mon_capable = true; + rdt_mon_capable = true; +if (r->mon.num_rmid) r->mon.num_rmid =min (r->mon.num_rmid, e->num_rmids);else
arch/x86/kernel/cpu/resctrl/intel_aet.c static bool enable_events(struct event_group *e, struct pmt_feature_group *p) · line 227
Quoted 2 lines
arch/x86/kernel/cpu/resctrl/intel_aet.c +0
-0@@ -223,6 +226,8 @@ static bool enable_events(struct event_group *e, struct pmt_feature_group *p)resctrl_enable_mon_event (e->evts[j].id, true, e->evts[j].bin_bits, &e->evts[j]);
I notice that the mon_capable flags are set *before* the events are enabled. If the first CPU of a package comes online between setting the flag and enabling the events then the initial domain creation will not be correct?
What if the mon_capable flags are set in resctrl_arch_pre_mount() after a successful intel_aet_get_events()? Perhaps with CPU hotplug lock held? From what I can tell doing so will impact the debugfs flow since that depends on the resource being mon_capable. Would there be a problem with delaying the debugfs setup until after domain list is built?
arch/x86/kernel/cpu/resctrl/intel_aet.c static bool enable_events(struct event_group *e, struct pmt_feature_group *p) · line 233
Quoted 6 lines
arch/x86/kernel/cpu/resctrl/intel_aet.c +2
-0+pr_info ("%s %s monitoring detected\n" , r->name, e->name); +return true; }
Reinette
Tony Luck
maintainerPATCH v11 28/31 fs/resctrl: Provide interface to create architecture specific debugfs area
In reply to Tony LuckAll files below /sys/fs/resctrl are considered user ABI.
This leaves no place for architectures to provide additional interfaces.
Add resctrl_debugfs_mon_info_arch_mkdir() which creates a directory in the debugfs file system for a monitoring resource. Naming follows the layout of the main resctrl hierarchy:
/sys/kernel/debug/resctrl/info/{resource}_MON/{arch}The {arch} last level directory name matches the output of the user level "uname -m" command.
Architecture code may use this directory for debug information, or for minor tuning of features. It must not be used for basic feature enabling as debugfs may not be configured/mounted on production systems.
--- include/linux/resctrl.h | 10 ++++++++++ fs/resctrl/rdtgroup.c | 29 +++++++++++++++++++++++++++++ 2 files changed, 39 insertions(+)
include/linux/resctrl.h +10 -0
@@ -678,6 +678,16 @@ void resctrl_arch_reset_cntr(struct rdt_resource *r, struct rdt_l3_mon_domain *dextern unsigned int resctrl_rmid_realloc_threshold;extern unsigned int resctrl_rmid_realloc_limit; +/** +* resctrl_debugfs_mon_info_arch_mkdir() - Create a debugfs info directory. +* Removed by resctrl_exit(). +* @r: Resource (must be mon_capable). +* +* Return: NULL if resource is not monitoring capable, +* dentry pointer on success, or ERR_PTR(-ERROR) on failure. +*/ +struct dentry *resctrl_debugfs_mon_info_arch_mkdir (struct rdt_resource *r); +int resctrl_init (void );void resctrl_exit (void );
fs/resctrl/rdtgroup.c +29 -0
@@ -24,6 +24,7 @@#include <linux/sched/task.h> #include <linux/slab.h> #include <linux/user_namespace.h> +#include <linux/utsname.h> #include <uapi/linux/magic.h>
@@ -75,6 +76,8 @@ static void rdtgroup_destroy_root(void);struct dentry *debugfs_resctrl; +static struct dentry *debugfs_resctrl_info; +/* * Memory bandwidth monitoring event to use for the default CTRL_MON group * and each new CTRL_MON group created by the user. Only relevant when
@@ -4513,6 +4516,31 @@ int resctrl_init(void)return ret; } +/* +* Create /sys/kernel/debug/resctrl/info/{r->name}_MON/{arch} directory +* by request for architecture to use for debugging or minor tuning. +* Basic functionality of features must not be controlled by files +* added to this directory as debugfs may not be configured/mounted +* on production systems. +*/ +struct dentry *resctrl_debugfs_mon_info_arch_mkdir (struct rdt_resource *r) +{ +struct dentry *moninfodir; +char name[32 ]; + +if (!r->mon_capable) +return NULL ; + +if (!debugfs_resctrl_info) + debugfs_resctrl_info =debugfs_create_dir ("info" , debugfs_resctrl); + +sprintf (name,"%s_MON" , r->name); + + moninfodir =debugfs_create_dir (name, debugfs_resctrl_info); + +return debugfs_create_dir (utsname ()->machine, moninfodir); +} +static bool resctrl_online_domains_exist (void ) {struct rdt_resource *r;
@@ -4564,6 +4592,7 @@ void resctrl_exit(void)debugfs_remove_recursive (debugfs_resctrl); debugfs_resctrl =NULL ; + debugfs_resctrl_info =NULL ;unregister_filesystem (&rdt_fs_type);/*
Signature · 1 line
2.51.0
Tony Luck
maintainerPATCH v11 29/31 x86/resctrl: Add debugfs files to show telemetry aggregator status
In reply to Tony LuckEach telemetry aggregator provides three status registers at the top end of MMIO space after all the per-RMID per-event counters:
data_loss_count: This counts the number of times that this aggregator failed to accumulate a counter value supplied by a CPU core.
data_loss_timestamp: This is a "timestamp" from a free running 25MHz uncore timer indicating when the most recent data loss occurred.
last_update_timestamp: Another 25MHz timestamp indicating when the most recent counter update was successfully applied.
Create files in /sys/kernel/debug/resctrl/info/PERF_PKG_MON/x86_64/ to display the value of each of these status registers for each aggregator in each enabled event group. The prefix for each file name describes the type of aggregator, which package it is located on, and an opaque instance number to provide a unique file name when there are multiple aggregators on a package.
The suffix is one of the three strings listed above. An example name is:
energy_pkg0_agg2_data_loss_count
These files are removed along with all other debugfs entries by the call to debugfs_remove_recursive() in resctrl_exit().
--- arch/x86/kernel/cpu/resctrl/intel_aet.c | 51 +++++++++++++++++++++++++ 1 file changed, 51 insertions(+)
arch/x86/kernel/cpu/resctrl/intel_aet.c +51 -0
@@ -18,8 +18,11 @@#include <linux/container_of.h> #include <linux/cpu.h> #include <linux/cpumask.h> +#include <linux/debugfs.h> +#include <linux/dcache.h> #include <linux/err.h> #include <linux/errno.h> +#include <linux/fs.h> #include <linux/gfp_types.h> #include <linux/init.h> #include <linux/intel_pmt_features.h>
@@ -33,6 +36,7 @@#include <linux/resctrl.h> #include <linux/resctrl_types.h> #include <linux/slab.h> +#include <linux/sprintf.h> #include <linux/stddef.h> #include <linux/topology.h> #include <linux/types.h>
@@ -184,9 +188,50 @@ static bool all_regions_have_sufficient_rmid(struct event_group *e, struct pmt_freturn true; } +static int status_read (void *priv,u64 *val) +{ +void __iomem *info = (void __iomem *)priv; + + *val =readq (info); + +return 0 ; +} + +DEFINE_SIMPLE_ATTRIBUTE (status_fops, status_read,NULL ,"%llu\n" ); + +static void make_status_files (struct dentry *dir,struct event_group *e,int pkg, +int instance,void *info_end) +{ +char name[64 ]; + +sprintf (name,"%s_pkg%d_agg%d_data_loss_count" , e->name, pkg, instance); +debugfs_create_file (name,0400 , dir, info_end -24 , &status_fops); + +sprintf (name,"%s_pkg%d_agg%d_data_loss_timestamp" , e->name, pkg, instance); +debugfs_create_file (name,0400 , dir, info_end -16 , &status_fops); + +sprintf (name,"%s_pkg%d_agg%d_last_update_timestamp" , e->name, pkg, instance); +debugfs_create_file (name,0400 , dir, info_end -8 , &status_fops); +} + +static void create_debug_event_status_files (struct dentry *dir,struct event_group *e, +struct pmt_feature_group *p) +{ +void *info_end; + +for (int i =0 ; i < p->count; i++) { +if (!p->regions[i].addr) +continue ; + info_end = (void __force *)p->regions[i].addr + e->mmio_size; +make_status_files (dir, e, p->regions[i].plat_info.package_id, + i, info_end); + } +} +static bool enable_events (struct event_group *e,struct pmt_feature_group *p) {struct rdt_resource *r = &rdt_resources_all[RDT_RESOURCE_PERF_PKG ].r_resctrl; +static struct dentry *infodir;bool usable_events = false;/* Disable feature if insufficient RMIDs */
@@ -226,6 +271,12 @@ static bool enable_events(struct event_group *e, struct pmt_feature_group *p)resctrl_enable_mon_event (e->evts[j].id, true, e->evts[j].bin_bits, &e->evts[j]); +if (!infodir) + infodir =resctrl_debugfs_mon_info_arch_mkdir (r); + +if (!IS_ERR_OR_NULL (infodir)) +create_debug_event_status_files (infodir, e, p); +pr_info ("%s %s monitoring detected\n" , r->name, e->name);return true;
Signature · 1 line
2.51.0
Reinette Chatre
maintainerRe: [PATCH v11 29/31] x86/resctrl: Add debugfs files to show telemetry aggregator status
Hi Tony,
arch/x86/kernel/cpu/resctrl/intel_aet.c static bool all_regions_have_sufficient_rmid(struct event_group *e, struct pmt_f · line 207
Quoted 7 lines · Tony Luck
arch/x86/kernel/cpu/resctrl/intel_aet.c +7
-0+ +static void make_status_files (struct dentry *dir,struct event_group *e,int pkg, +int instance,void *info_end) +{ +char name[64 ]; + +sprintf (name,"%s_pkg%d_agg%d_data_loss_count" , e->name, pkg, instance);
Please keep the type used for package id consistent throughout the series.
Reinette
Tony Luck
maintainerPATCH v11 30/31 x86,fs/resctrl: Update Documentation for package events
In reply to Tony LuckUpdate resctrl filesystem documentation with the details about the resctrl files that support telemetry events.
--- Documentation/filesystems/resctrl.rst | 100 ++++++++++++++++++++++---- 1 file changed, 87 insertions(+), 13 deletions(-)
Documentation/filesystems/resctrl.rst +87 -13
@@ -168,13 +168,12 @@ with respect to allocation: bandwidth percentages are directly applied to the threads running on the core -If RDT monitoring is available there will be an "L3_MON" directory +If L3 monitoring is available there will be an "L3_MON" directory with the following files: "num_rmids": - The number of RMIDs available. This is the - upper bound for how many "CTRL_MON" + "MON" - groups can be created. + The number of RMIDs supported by hardware for + L3 monitoring events. "mon_features": Lists the monitoring events if
@@ -400,6 +399,19 @@ with the following files: bytes) at which a previously used LLC_occupancy counter can be considered for re-use. +If telemetry monitoring is available there will be an "PERF_PKG_MON" directory +with the following files: + +"num_rmids": + The number of RMIDs supported by hardware for + telemetry monitoring events. + +"mon_features": + Lists the telemetry monitoring events that are enabled on this system. + +The upper bound for how many "CTRL_MON" + "MON" can be created +is the smaller of the L3_MON and PERF_PKG_MON "num_rmids" values. + Finally, in the top level of the "info" directory there is a file named "last_cmd_status". This is reset with every "command" issued via the file system (making new directories or writing to any of the
@@ -505,15 +517,40 @@ When control is enabled all CTRL_MON groups will also contain: When monitoring is enabled all MON groups will also contain: "mon_data": - This contains a set of files organized by L3 domain and by - RDT event. E.g. on a system with two L3 domains there will - be subdirectories "mon_L3_00" and "mon_L3_01". Each of these - directories have one file per event (e.g. "llc_occupancy", - "mbm_total_bytes", and "mbm_local_bytes"). In a MON group these - files provide a read out of the current value of the event for - all tasks in the group. In CTRL_MON groups these files provide - the sum for all tasks in the CTRL_MON group and all tasks in - MON groups. Please see example section for more details on usage. + This contains directories for each monitor domain. One set for + each instance of an L3 cache, another set for each processor + package. The L3 cache directories are named "mon_L3_00", + "mon_L3_01" etc. The package directories "mon_PERF_PKG_00", + "mon_PERF_PKG_01" etc. + + Within each directory there is one file per event. For + example the L3 directories may contain "llc_occupancy", "mbm_total_bytes", + and "mbm_local_bytes". The PERF_PKG directories may contain "core_energy", + "activity", etc. The info/`*`/mon_features files provide the full + list of event/file names. + + "core energy" reports a floating point number for the energy (in Joules) + consumed by cores (registers, arithmetic units, TLB and L1/L2 caches) + during execution of instructions summed across all logical CPUs on a + package for the current RMID. + + "activity" also reports a floating point value (in Farads). + This provides an estimate of work done independent of the + frequency that the CPUs used for execution. + + Note that these two counters only measure energy/activity + in the "core" of the CPU (arithmetic units, TLB, L1 and L2 + caches, etc.). They do not include L3 cache, memory, I/O + devices etc. + + All other events report decimal integer values. + + In a MON group these files provide a read out of the current + value of the event for all tasks in the group. In CTRL_MON groups + these files provide the sum for all tasks in the CTRL_MON group + and all tasks in MON groups. Please see example section for more + details on usage. + On systems with Sub-NUMA Cluster (SNC) enabled there are extra directories for each node (located within the "mon_L3_XX" directory for the L3 cache they occupy). These are named "mon_sub_L3_YY"
@@ -1506,6 +1543,43 @@ Example with C:: resctrl_release_lock(fd); } +Debugfs +======= +In addition to the use of debugfs for tracing of pseudo-locking +performance, architecture code may create debugfs directories +associated with monitoring features for a specific resource. + +The full pathname for these is in the form: + + /sys/kernel/debug/resctrl/info/{resource_name}_MON/{arch}/ + +The presence, names, and format of these files may vary +between architectures even if the same resource is present. + +PERF_PKG_MON/x86_64 +------------------- +Three files are present per telemetry aggregator instance +that show status. The prefix of +each file name describes the type ("energy" or "perf") which +processor package it belongs to, and the instance number of +the aggregator. For example: "energy_pkg1_agg2". + +The suffix describes which data is reported in the file and +is one of: + +data_loss_count: + This counts the number of times that this aggregator + failed to accumulate a counter value supplied by a CPU. + +data_loss_timestamp: + This is a "timestamp" from a free running 25MHz uncore + timer indicating when the most recent data loss occurred. + +last_update_timestamp: + Another 25MHz timestamp indicating when the + most recent counter update was successfully applied. + + Examples for RDT Monitoring along with allocation usage ======================================================= Reading monitored data
Signature · 1 line
2.51.0
Reinette Chatre
maintainerRe: [PATCH v11 30/31] x86,fs/resctrl: Update Documentation for package events
Hi Tony,
Two nits in subject:
"Documentation" -> "documentation"
"package events" -> "telemetry events"?
(this is the one and only instance of "package event" in this series and does not match changelog that follows)
Quoted 30 lines · Tony Luck
Update resctrl filesystem documentation with the details about the resctrl files that support telemetry events.
Signed-off-by Tony Luck <[email protected]>--- Documentation/filesystems/resctrl.rst | 100 ++++++++++++++++++++++---- 1 file changed, 87 insertions(+), 13 deletions(-)Documentation/filesystems/resctrl.rst +3
-4@@ -168,13 +168,12 @@ with respect to allocation: bandwidth percentages are directly applied to the threads running on the core -If RDT monitoring is available there will be an "L3_MON" directory +If L3 monitoring is available there will be an "L3_MON" directory with the following files: "num_rmids": - The number of RMIDs available. This is the - upper bound for how many "CTRL_MON" + "MON" - groups can be created. + The number of RMIDs supported by hardware for + L3 monitoring events. "mon_features": Lists the monitoring events if
Documentation/filesystems/resctrl.rst with the following files: · line 407
Quoted 9 lines
Documentation/filesystems/resctrl.rst +6
-0@@ -400,6 +399,19 @@ with the following files: bytes) at which a previously used LLC_occupancy counter can be considered for re-use. +If telemetry monitoring is available there will be an "PERF_PKG_MON" directory +with the following files: + +"num_rmids": + The number of RMIDs supported by hardware for + telemetry monitoring events.
There may be some additional detail about how num_rmids is determined that could be valuable to user space since from what I understand user space seems to have some control over this number in addition to it being "supported by hardware".
For example, if the PERF event group has more RMID than the ENERGY event group and the user needs to do significant monitoring of PERF then it may be useful to know that by disabling ENERGY it could be possible to increase the number of RMIDs in order to do that monitoring.
Additionally, from patch #23 we learned that "supported by hardware" can have different meanings ... it could be the number of RMIDs "supported" or it could mean the number of RMIDs that can be reliably "counted". A user force-enabling an under resourced event group will thus encounter a num_rmids that does not match the (XML) spec.
Quoted 16 lines
Documentation/filesystems/resctrl.rst +7
-2+ +"mon_features": + Lists the telemetry monitoring events that are enabled on this system. + +The upper bound for how many "CTRL_MON" + "MON" can be created +is the smaller of the L3_MON and PERF_PKG_MON "num_rmids" values. + Finally, in the top level of the "info" directory there is a file named "last_cmd_status". This is reset with every "command" issued via the file system (making new directories or writing to any of the@@ -505,15 +517,40 @@ When control is enabled all CTRL_MON groups will also contain: When monitoring is enabled all MON groups will also contain: "mon_data": - This contains a set of files organized by L3 domain and by - RDT event. E.g. on a system with two L3 domains there will
Documentation/filesystems/resctrl.rst When control is enabled all CTRL_MON groups will also contain: · line 522
Quoted 10 lines
Documentation/filesystems/resctrl.rst +3
-7- be subdirectories "mon_L3_00" and "mon_L3_01". Each of these - directories have one file per event (e.g. "llc_occupancy", - "mbm_total_bytes", and "mbm_local_bytes"). In a MON group these - files provide a read out of the current value of the event for - all tasks in the group. In CTRL_MON groups these files provide - the sum for all tasks in the CTRL_MON group and all tasks in - MON groups. Please see example section for more details on usage. + This contains directories for each monitor domain. One set for + each instance of an L3 cache, another set for each processor + package. The L3 cache directories are named "mon_L3_00",
I still do not understand the "set" terminology. There is just one directory per domain, no? For example, "This contains a directory for each monitoring domain of a monitoring capable resource. One directory for each instance of an L3 cache if L3 monitoring is available, another directory for each processor package if telemetry monitoring is available."
Quoted 9 lines
Documentation/filesystems/resctrl.rst +9
-0+ "mon_L3_01" etc. The package directories "mon_PERF_PKG_00", + "mon_PERF_PKG_01" etc. + + Within each directory there is one file per event. For + example the L3 directories may contain "llc_occupancy", "mbm_total_bytes", + and "mbm_local_bytes". The PERF_PKG directories may contain "core_energy", + "activity", etc. The info/`*`/mon_features files provide the full + list of event/file names. +
Documentation/filesystems/resctrl.rst When control is enabled all CTRL_MON groups will also contain: · line 541
Quoted 10 lines
Documentation/filesystems/resctrl.rst +10
-0+ "core energy" reports a floating point number for the energy (in Joules) + consumed by cores (registers, arithmetic units, TLB and L1/L2 caches) + during execution of instructions summed across all logical CPUs on a + package for the current RMID. + + "activity" also reports a floating point value (in Farads). + This provides an estimate of work done independent of the + frequency that the CPUs used for execution. + + Note that these two counters only measure energy/activity
To help be specific:
""core energy" and "activity" only measure ..."
Documentation/filesystems/resctrl.rst When control is enabled all CTRL_MON groups will also contain: · line 553
Quoted 12 lines
Documentation/filesystems/resctrl.rst +12
-0+ in the "core" of the CPU (arithmetic units, TLB, L1 and L2 + caches, etc.). They do not include L3 cache, memory, I/O + devices etc. + + All other events report decimal integer values. + + In a MON group these files provide a read out of the current + value of the event for all tasks in the group. In CTRL_MON groups + these files provide the sum for all tasks in the CTRL_MON group + and all tasks in MON groups. Please see example section for more + details on usage. +
Please have this text line length be consistent with surrounding text.
Quoted 14 lines
Documentation/filesystems/resctrl.rst +7
-0On systems with Sub-NUMA Cluster (SNC) enabled there are extra directories for each node (located within the "mon_L3_XX" directory for the L3 cache they occupy). These are named "mon_sub_L3_YY"@@ -1506,6 +1543,43 @@ Example with C:: resctrl_release_lock(fd); } +Debugfs +======= +In addition to the use of debugfs for tracing of pseudo-locking +performance, architecture code may create debugfs directories +associated with monitoring features for a specific resource. + +The full pathname for these is in the form:
Documentation/filesystems/resctrl.rst Example with C:: · line 1562
Quoted 10 lines
Documentation/filesystems/resctrl.rst +10
-0+ + /sys/kernel/debug/resctrl/info/{resource_name}_MON/{arch}/ + +The presence, names, and format of these files may vary +between architectures even if the same resource is present. + +PERF_PKG_MON/x86_64 +------------------- +Three files are present per telemetry aggregator instance +that show status. The prefix of
Please be consistent with line length and do not trim lines so short.
Quoted 13 lines
Documentation/filesystems/resctrl.rst +13
-0+each file name describes the type ("energy" or "perf") which +processor package it belongs to, and the instance number of +the aggregator. For example: "energy_pkg1_agg2". + +The suffix describes which data is reported in the file and +is one of: + +data_loss_count: + This counts the number of times that this aggregator + failed to accumulate a counter value supplied by a CPU. + +data_loss_timestamp: + This is a "timestamp" from a free running 25MHz uncore
Documentation/filesystems/resctrl.rst Example with C:: · line 1585
Quoted 10 lines
Documentation/filesystems/resctrl.rst +7
-0+ timer indicating when the most recent data loss occurred. + +last_update_timestamp: + Another 25MHz timestamp indicating when the + most recent counter update was successfully applied. + + Examples for RDT Monitoring along with allocation usage ======================================================= Reading monitored data
Reinette
Tony Luck
maintainerPATCH v11 31/31 fs/resctrl: Some kerneldoc updates
In reply to Tony Luckresctrl event monitoring on Sub-NUMA Cluster (SNC) systems sums the counts for events across all nodes sharing an L3 cache.
Update the kerneldoc for rmid_read::sum and the do_sum argument to mon_get_kn_priv() to say these are only used on the RDT_RESOURCE_L3 resource.
Add Return: value description for l3_mon_domain_mbm_alloc(), resctrl_l3_mon_resource_init(), and domain_setup_l3_mon_state()
--- fs/resctrl/internal.h | 4 ++-- arch/x86/kernel/cpu/resctrl/core.c | 2 ++ fs/resctrl/monitor.c | 2 +- fs/resctrl/rdtgroup.c | 5 +++-- 4 files changed, 8 insertions(+), 5 deletions(-)
fs/resctrl/internal.h +2 -2
@@ -96,8 +96,8 @@ extern struct mon_evt mon_event_all[QOS_NUM_EVENTS];* @list: Member of the global @mon_data_kn_priv_list list. * @rid: Resource id associated with the event file. * @evt: Event structure associated with the event file. -* @sum: Set when event must be summed across multiple -* domains. +* @sum: Set for RDT_RESOURCE_L3 when event must be summed +* across multiple domains. * @domid: When @sum is zero this is the domain to which * the event file belongs. When @sum is one this * is the id of the L3 cache that all domains to be
arch/x86/kernel/cpu/resctrl/core.c +2 -0
@@ -418,6 +418,8 @@ static int domain_setup_ctrlval(struct rdt_resource *r, struct rdt_ctrl_domain ** l3_mon_domain_mbm_alloc() - Allocate arch private storage for the MBM counters * @num_rmid: The size of the MBM counter array * @hw_dom: The domain that owns the allocated arrays +* +* Return: %0 for success; Error code otherwise. */ static int l3_mon_domain_mbm_alloc (u32 num_rmid,struct rdt_hw_l3_mon_domain *hw_dom) {
fs/resctrl/monitor.c +1 -1
@@ -1811,7 +1811,7 @@ static void closid_num_dirty_rmid_free(void)* Resctrl's cpuhp callbacks may be called before this point to bring a domain * online. * -* Returns 0 for success, or -ENOMEM. +* Return: %0 for success; Error code otherwise. */ int resctrl_l3_mon_resource_init (void ) {
fs/resctrl/rdtgroup.c +3 -2
@@ -3037,7 +3037,8 @@ static void rmdir_all_sub(void)* @rid: The resource id for the event file being created. * @domid: The domain id for the event file being created. * @mevt: The type of event file being created. -* @do_sum: Whether SNC summing monitors are being created. +* @do_sum: Whether SNC summing monitors are being created. Only set +* when @rid == RDT_RESOURCE_L3. */ static struct mon_data *mon_get_kn_priv(enum resctrl_res_level rid, int domid, struct mon_evt *mevt,
@@ -4281,7 +4282,7 @@ void resctrl_offline_mon_domain(struct rdt_resource *r, struct rdt_domain_hdr *h* at mount time. This means the rdt_l3_mon_domain::mbm_states[] and * rdt_l3_mon_domain::rmid_busy_llc allocations may be larger than needed. * -* Returns 0 for success, or -ENOMEM. +* Return: %0 for success; Error code otherwise. */ static int domain_setup_l3_mon_state (struct rdt_resource *r,struct rdt_l3_mon_domain *d) {
Signature · 1 line
2.51.0
Reinette Chatre
maintainerRe: [PATCH v11 31/31] fs/resctrl: Some kerneldoc updates
Hi Tony,
Quoted 6 lines · Tony Luck
resctrl event monitoring on Sub-NUMA Cluster (SNC) systems sums the counts for events across all nodes sharing an L3 cache.
Update the kerneldoc for
rmid_read::sumand thedo_sumargument tomon_get_kn_priv()to say these are only used on theRDT_RESOURCE_L3resource.
This is clear from the patch. Why is this needed as part of telemetry event enabling? Perhaps this can be combined with the unrelated SNC warnings found in "x86/resctrl: Handle domain creation/deletion for RDT_RESOURCE_PERF_PKG" to be a patch dedicated to addressing SNC topics related to telemetry events?
Quoted 3 lines
Add Return: value description for
l3_mon_domain_mbm_alloc(),resctrl_l3_mon_resource_init(), anddomain_setup_l3_mon_state()
Appreciate the cleanups but please have series start with cleanups instead of end.
Quoted 3 lines
Signed-off-by Tony Luck <[email protected]>---
Reinette
Luck, Tony
maintainerRe: [PATCH v11 31/31] fs/resctrl: Some kerneldoc updates
Quoted 10 lines · Reinette Chatre
Hi Tony,
Quoted 6 lines · Tony Luck
resctrl event monitoring on Sub-NUMA Cluster (SNC) systems sums the counts for events across all nodes sharing an L3 cache.
Update the kerneldoc for
rmid_read::sumand thedo_sumargument tomon_get_kn_priv()to say these are only used on theRDT_RESOURCE_L3resource.
Quoted 5 lines
This is clear from the patch. Why is this needed as part of telemetry event enabling? Perhaps this can be combined with the unrelated SNC warnings found in "x86/resctrl: Handle domain creation/deletion for
RDT_RESOURCE_PERF_PKG" to be a patch dedicated to addressing SNC topics related to telemetry events?
I will add an SNC cleanup patch to the series and make these changes there.
Quoted 6 lines
Quoted 4 lines
Add Return: value description for
l3_mon_domain_mbm_alloc(),resctrl_l3_mon_resource_init(), anddomain_setup_l3_mon_state()Appreciate the cleanups but please have series start with cleanups instead of end.
Can I bundle these cleanups with patch 8 that renames these functions?
Quoted 6 lines
Quoted 4 lines
Signed-off-by Tony Luck <[email protected]>---
Reinette
Reinette Chatre
maintainerRe: [PATCH v11 31/31] fs/resctrl: Some kerneldoc updates
Hi Tony,
Quoted 11 lines · Tony
Quoted 10 lines · Reinette Chatre
Hi Tony,
Quoted 6 lines · Tony Luck
resctrl event monitoring on Sub-NUMA Cluster (SNC) systems sums the counts for events across all nodes sharing an L3 cache.
Update the kerneldoc for
rmid_read::sumand thedo_sumargument tomon_get_kn_priv()to say these are only used on theRDT_RESOURCE_L3resource.
Quoted 7 lines
Quoted 5 lines
This is clear from the patch. Why is this needed as part of telemetry event enabling? Perhaps this can be combined with the unrelated SNC warnings found in "x86/resctrl: Handle domain creation/deletion for
RDT_RESOURCE_PERF_PKG" to be a patch dedicated to addressing SNC topics related to telemetry events?I will add an SNC cleanup patch to the series and make these changes there.
Thank you very much.
Quoted 8 lines
Quoted 6 lines
Quoted 4 lines
Add Return: value description for
l3_mon_domain_mbm_alloc(),resctrl_l3_mon_resource_init(), anddomain_setup_l3_mon_state()Appreciate the cleanups but please have series start with cleanups instead of end.
Can I bundle these cleanups with patch 8 that renames these functions?
Good question. It is ok with me. I am not aware of concerns with doing something like this.
Reinette