Skip to contentSkip to conversation

x86/resctrl: Merge the CDP resources

PATCH Superseded James Morse 3 participants 52 messages Oct 30, 2020 – Mar 12, 2021

Subsystems (2)

v1 of 7 Superseded Latest is v7 →

lore
Share & tools
lore

James Morse

author reviewer
#0 lore raw

cover x86/resctrl: Merge the CDP resources

Hi folks,

This series re-folds the resctrl code so the CDP resources (L3CODE et al) behaviour is all contained in the filesystem parts, with a minimum amount of arch specific code.

Arm have some CPU support for dividing caches into portions, and applying bandwidth limits at various points in the SoC. The collective term for these features is MPAM: Memory Partitioning and Monitoring.

MPAM is similar enough to Intel RDT, that it should use the defacto linux
interface: resctrl. This filesystem currently lives under arch/x86, and is tightly coupled to the architecture.
Ultimately, my plan is to split the existing resctrl code up to have an arch<->fs abstraction, then move all the bits out to fs/resctrl. From there MPAM can be wired up.

x86 might have two resources with cache controls, (L2 and L3) but has extra copies for CDP: L{2,3}{CODE,DATA}, which are marked as enabled if CDP is enabled for the corresponding cache.

MPAM has an equivalent feature to CDP, but its a property of the CPU, not the cache. Resctrl needs to have x86's odd/even behaviour, as that its the ABI, but this isn't how the MPAM hardware works. It is entirely possible that an in-kernel user of MPAM would not be using CDP, whereas resctrl is.
Pretending L3CODE and L3DATA are entirely separate resources is a neat trick, but doing this is specific to x86.
Doing this leaves the arch code in control of various parts of the filesystem ABI: the resources names, and the way the schemata are parsed. Allowing this stuff to vary between architectures is bad for user space.

This series collapses the CODE/DATA resources, moving all the user-visible resctrl ABI into the filesystem code. CDP becomes the type of configuration being applied to a cache. This is done by adding a struct resctrl_schema to the parts of resctrl that will move to fs. This holds the arch-code resource that is in use for this schema, along with other properties like the name, and whether the configuration being applied is CODE/DATA/BOTH.

This lets us fold the extra resources out of the arch code so that they don't need to be duplicated if the equivalent feature to CDP is missing, or implemented in a different way.

The first two patches split the resource and domain structs to have an arch specific 'hw' portion, and the rest that is visible to resctrl. Future series massage the resctrl code so there are no accesses to 'hw' structures in the parts of resctrl that will move to fs, providing helpers where necessary.

Since anyone last looked at this, the CDP property has been made per-resource instead of global. MPAM will need to make this global in the arch code, as CODE/DATA closid are based on how the CPU tags traffic, not how the cache interprets it. resctrl sets CDP enabled on a resource, but reads it back on each one.
The attempt to keep closids as-used-by-resctrl and closids as-written-to-hw appart has been dropped.
There are two copies of num_closid. The version private to the arch code is the value discovered from hardware. resctrl has its own version, which it may write to, which is exposed to user-space. This lets resctrl do its odd/even thing, even if thats not how the hardware works.

This series adds temporary scaffolding, which it removes a few patches later. This is to allow things like the ctrlval arrays and resources to be merged separately, which should make is easier to bisect. These things are marked temporary, and should all be gone by the end of the series.

This series is a little rough around the monitors, would a fake struct resctrl_schema for the monitors simplify things, or be a source of bugs?

This series is based on v5.10-rc1, and can be retrieved from:
git://git.kernel.org/pub/scm/linux/kernel/git/morse/linux.git mpam/resctrl_merge_cdp/v1

Parts were previously posted as an RFC here:
https://lore.kernel.org/lkml/[email protected]/

Thanks,

James Morse (24):
x86/resctrl: Split struct rdt_resource
x86/resctrl: Split struct rdt_domain
x86/resctrl: Add resctrl_arch_get_num_closid() x86/resctrl: Add a separate schema list for resctrl x86/resctrl: Pass the schema in resdir's private pointer x86/resctrl: Store the effective num_closid in the schema x86/resctrl: Label the resources with their configuration type x86/resctrl: Walk the resctrl schema list instead of an arch list x86/resctrl: Change rdt_resource to resctrl_schema in
pseudo_lock_region
x86/resctrl: Move the schema names into struct resctrl_schema x86/resctrl: Group staged configuration into a separate struct x86/resctrl: Add closid to the staged config x86/resctrl: Allow different CODE/DATA configurations to be staged x86/resctrl: Make update_domains() learn the affected closids x86/resctrl: Add a helper to read a closid's configuration x86/resctrl: Add a helper to read/set the CDP configuration x86/resctrl: Use cdp_enabled in rdt_domain_reconfigure_cdp() x86/resctrl: Pass configuration type to resctrl_arch_get_config() x86/resctrl: Make ctrlval arrays the same size x86/resctrl: Apply offset correction when config is staged x86/resctrl: Calculate the index from the configuration type x86/resctrl: Merge the ctrlval arrays
x86/resctrl: Remove rdt_cdp_peer_get()
x86/resctrl: Merge the CDP resources

 arch/x86/kernel/cpu/resctrl/core.c        | 290 ++++++--------
 arch/x86/kernel/cpu/resctrl/ctrlmondata.c | 174 ++++++---
 arch/x86/kernel/cpu/resctrl/internal.h    | 215 +++-------
 arch/x86/kernel/cpu/resctrl/monitor.c     |  44 ++-
 arch/x86/kernel/cpu/resctrl/pseudo_lock.c |  12 +-
 arch/x86/kernel/cpu/resctrl/rdtgroup.c    | 454 ++++++++++++----------
 include/linux/resctrl.h                   | 189 +++++++++
 7 files changed, 775 insertions(+), 603 deletions(-)
Signature · 1 line
2.28.0
author reviewer
#1 lore raw

PATCH 1/24 x86/resctrl: Split struct rdt_resource

resctrl is the defacto Linux ABI for SoC resource partitioning features. To support it on another architecture, it needs to be abstracted from Intel RDT, and moved it to /fs/.

Start by splitting struct rdt_resource, (the name is kept to keep the noise down), and add some type-trickery to keep the foreach helpers working.

Move everything that that is particular to resctrl into a new header file, keeping the x86 hardware accessors where they are. resctrl code paths touching a 'hw' struct indicates where an abstraction is needed.

Splitting rdt_domain up in a similar way happens in the next patch. No change in behaviour, this patch just moves types around.

---
 arch/x86/kernel/cpu/resctrl/core.c        | 258 ++++++++++++----------
 arch/x86/kernel/cpu/resctrl/ctrlmondata.c |  14 +-
 arch/x86/kernel/cpu/resctrl/internal.h    | 138 +++---------
 arch/x86/kernel/cpu/resctrl/monitor.c     |  32 +--
 arch/x86/kernel/cpu/resctrl/pseudo_lock.c |   4 +-
 arch/x86/kernel/cpu/resctrl/rdtgroup.c    |  69 +++---
 include/linux/resctrl.h                   | 117 ++++++++++
 7 files changed, 362 insertions(+), 270 deletions(-)
arch/x86/kernel/cpu/resctrl/core.c +147 -111
index e5f4ee8f4c3b..470661f2eb68 100644
@@ -57,120 +57,134 @@ static void
 mba_wrmsr_amd(struct rdt_domain *d, struct msr_param *m,
 	      struct rdt_resource *r);
 
-#define domain_init(id) LIST_HEAD_INIT(rdt_resources_all[id].domains)
+#define domain_init(id) LIST_HEAD_INIT(rdt_resources_all[id].resctrl.domains)
 
-struct rdt_resource rdt_resources_all[] = {
+struct rdt_hw_resource rdt_resources_all[] = {
 	[RDT_RESOURCE_L3] =
 	{
-		.rid			= RDT_RESOURCE_L3,
-		.name			= "L3",
-		.domains		= domain_init(RDT_RESOURCE_L3),
+		.resctrl = {
+			.rid			= RDT_RESOURCE_L3,
+			.name			= "L3",
+			.cache_level		= 3,
+			.cache = {
+				.min_cbm_bits	= 1,
+				.cbm_idx_mult	= 1,
+				.cbm_idx_offset	= 0,
+			},
+			.domains		= domain_init(RDT_RESOURCE_L3),
+			.parse_ctrlval		= parse_cbm,
+			.format_str		= "%d=%0*x",
+			.fflags			= RFTYPE_RES_CACHE,
+		},
 		.msr_base		= MSR_IA32_L3_CBM_BASE,
 		.msr_update		= cat_wrmsr,
-		.cache_level		= 3,
-		.cache = {
-			.min_cbm_bits	= 1,
-			.cbm_idx_mult	= 1,
-			.cbm_idx_offset	= 0,
-		},
-		.parse_ctrlval		= parse_cbm,
-		.format_str		= "%d=%0*x",
-		.fflags			= RFTYPE_RES_CACHE,
 	},
 	[RDT_RESOURCE_L3DATA] =
 	{
-		.rid			= RDT_RESOURCE_L3DATA,
-		.name			= "L3DATA",
-		.domains		= domain_init(RDT_RESOURCE_L3DATA),
+		.resctrl = {
+			.rid			= RDT_RESOURCE_L3DATA,
+			.name			= "L3DATA",
+			.cache_level		= 3,
+			.cache = {
+				.min_cbm_bits	= 1,
+				.cbm_idx_mult	= 2,
+				.cbm_idx_offset	= 0,
+			},
+			.domains		= domain_init(RDT_RESOURCE_L3DATA),
+			.parse_ctrlval		= parse_cbm,
+			.format_str		= "%d=%0*x",
+			.fflags			= RFTYPE_RES_CACHE,
+		},
 		.msr_base		= MSR_IA32_L3_CBM_BASE,
 		.msr_update		= cat_wrmsr,
-		.cache_level		= 3,
-		.cache = {
-			.min_cbm_bits	= 1,
-			.cbm_idx_mult	= 2,
-			.cbm_idx_offset	= 0,
-		},
-		.parse_ctrlval		= parse_cbm,
-		.format_str		= "%d=%0*x",
-		.fflags			= RFTYPE_RES_CACHE,
 	},
 	[RDT_RESOURCE_L3CODE] =
 	{
-		.rid			= RDT_RESOURCE_L3CODE,
-		.name			= "L3CODE",
-		.domains		= domain_init(RDT_RESOURCE_L3CODE),
+		.resctrl = {
+			.rid			= RDT_RESOURCE_L3CODE,
+			.name			= "L3CODE",
+			.cache_level		= 3,
+			.cache = {
+				.min_cbm_bits	= 1,
+				.cbm_idx_mult	= 2,
+				.cbm_idx_offset	= 1,
+			},
+			.domains		= domain_init(RDT_RESOURCE_L3CODE),
+			.parse_ctrlval		= parse_cbm,
+			.format_str		= "%d=%0*x",
+			.fflags			= RFTYPE_RES_CACHE,
+		},
 		.msr_base		= MSR_IA32_L3_CBM_BASE,
 		.msr_update		= cat_wrmsr,
-		.cache_level		= 3,
-		.cache = {
-			.min_cbm_bits	= 1,
-			.cbm_idx_mult	= 2,
-			.cbm_idx_offset	= 1,
-		},
-		.parse_ctrlval		= parse_cbm,
-		.format_str		= "%d=%0*x",
-		.fflags			= RFTYPE_RES_CACHE,
 	},
 	[RDT_RESOURCE_L2] =
 	{
-		.rid			= RDT_RESOURCE_L2,
-		.name			= "L2",
-		.domains		= domain_init(RDT_RESOURCE_L2),
+		.resctrl = {
+			.rid			= RDT_RESOURCE_L2,
+			.name			= "L2",
+			.cache_level		= 2,
+			.cache = {
+				.min_cbm_bits	= 1,
+				.cbm_idx_mult	= 1,
+				.cbm_idx_offset	= 0,
+			},
+			.domains		= domain_init(RDT_RESOURCE_L2),
+			.parse_ctrlval		= parse_cbm,
+			.format_str		= "%d=%0*x",
+			.fflags			= RFTYPE_RES_CACHE,
+		},
 		.msr_base		= MSR_IA32_L2_CBM_BASE,
 		.msr_update		= cat_wrmsr,
-		.cache_level		= 2,
-		.cache = {
-			.min_cbm_bits	= 1,
-			.cbm_idx_mult	= 1,
-			.cbm_idx_offset	= 0,
-		},
-		.parse_ctrlval		= parse_cbm,
-		.format_str		= "%d=%0*x",
-		.fflags			= RFTYPE_RES_CACHE,
 	},
 	[RDT_RESOURCE_L2DATA] =
 	{
-		.rid			= RDT_RESOURCE_L2DATA,
-		.name			= "L2DATA",
-		.domains		= domain_init(RDT_RESOURCE_L2DATA),
+		.resctrl = {
+			.rid			= RDT_RESOURCE_L2DATA,
+			.name			= "L2DATA",
+			.cache_level		= 2,
+			.cache = {
+				.min_cbm_bits	= 1,
+				.cbm_idx_mult	= 2,
+				.cbm_idx_offset	= 0,
+			},
+			.domains		= domain_init(RDT_RESOURCE_L2DATA),
+			.parse_ctrlval		= parse_cbm,
+			.format_str		= "%d=%0*x",
+			.fflags			= RFTYPE_RES_CACHE,
+		},
 		.msr_base		= MSR_IA32_L2_CBM_BASE,
 		.msr_update		= cat_wrmsr,
-		.cache_level		= 2,
-		.cache = {
-			.min_cbm_bits	= 1,
-			.cbm_idx_mult	= 2,
-			.cbm_idx_offset	= 0,
-		},
-		.parse_ctrlval		= parse_cbm,
-		.format_str		= "%d=%0*x",
-		.fflags			= RFTYPE_RES_CACHE,
 	},
 	[RDT_RESOURCE_L2CODE] =
 	{
-		.rid			= RDT_RESOURCE_L2CODE,
-		.name			= "L2CODE",
-		.domains		= domain_init(RDT_RESOURCE_L2CODE),
+		.resctrl = {
+			.rid			= RDT_RESOURCE_L2CODE,
+			.name			= "L2CODE",
+			.cache_level		= 2,
+			.cache = {
+				.min_cbm_bits	= 1,
+				.cbm_idx_mult	= 2,
+				.cbm_idx_offset	= 1,
+			},
+			.domains		= domain_init(RDT_RESOURCE_L2CODE),
+			.parse_ctrlval		= parse_cbm,
+			.format_str		= "%d=%0*x",
+			.fflags			= RFTYPE_RES_CACHE,
+		},
 		.msr_base		= MSR_IA32_L2_CBM_BASE,
 		.msr_update		= cat_wrmsr,
-		.cache_level		= 2,
-		.cache = {
-			.min_cbm_bits	= 1,
-			.cbm_idx_mult	= 2,
-			.cbm_idx_offset	= 1,
-		},
-		.parse_ctrlval		= parse_cbm,
-		.format_str		= "%d=%0*x",
-		.fflags			= RFTYPE_RES_CACHE,
 	},
 	[RDT_RESOURCE_MBA] =
 	{
-		.rid			= RDT_RESOURCE_MBA,
-		.name			= "MB",
-		.domains		= domain_init(RDT_RESOURCE_MBA),
-		.cache_level		= 3,
-		.parse_ctrlval		= parse_bw,
-		.format_str		= "%d=%*u",
-		.fflags			= RFTYPE_RES_MB,
+		.resctrl = {
+			.rid			= RDT_RESOURCE_MBA,
+			.name			= "MB",
+			.cache_level		= 3,
+			.domains		= domain_init(RDT_RESOURCE_MBA),
+			.parse_ctrlval		= parse_bw,
+			.format_str		= "%d=%*u",
+			.fflags			= RFTYPE_RES_MB,
+		},
 	},
 };
 
@@ -199,7 +213,8 @@ static unsigned int cbm_idx(struct rdt_resource *r, unsigned int closid)
  */
 static inline void cache_alloc_hsw_probe(void)
 {
-	struct rdt_resource *r  = &rdt_resources_all[RDT_RESOURCE_L3];
+	struct rdt_hw_resource *hw_res = &rdt_resources_all[RDT_RESOURCE_L3];
+	struct rdt_resource *r  = &hw_res->resctrl;
 	u32 l, h, max_cbm = BIT_MASK(20) - 1;
 
 	if (wrmsr_safe(MSR_IA32_L3_CBM_BASE, max_cbm, 0))
@@ -211,7 +226,7 @@ static inline void cache_alloc_hsw_probe(void)
 	if (l != max_cbm)
 		return;
 
-	r->num_closid = 4;
+	hw_res->num_closid = 4;
 	r->default_ctrl = max_cbm;
 	r->cache.cbm_len = 20;
 	r->cache.shareable_bits = 0xc0000;
@@ -225,7 +240,7 @@ static inline void cache_alloc_hsw_probe(void)
 bool is_mba_sc(struct rdt_resource *r)
 {
 	if (!r)
-		return rdt_resources_all[RDT_RESOURCE_MBA].membw.mba_sc;
+		return rdt_resources_all[RDT_RESOURCE_MBA].resctrl.membw.mba_sc;
 
 	return r->membw.mba_sc;
 }
@@ -253,12 +268,13 @@ static inline bool rdt_get_mb_table(struct rdt_resource *r)
 
 static bool __get_mem_config_intel(struct rdt_resource *r)
 {
+	struct rdt_hw_resource *hw_res = resctrl_to_arch_res(r);
 	union cpuid_0x10_3_eax eax;
 	union cpuid_0x10_x_edx edx;
 	u32 ebx, ecx, max_delay;
 
 	cpuid_count(0x00000010, 3, &eax.full, &ebx, &ecx, &edx.full);
-	r->num_closid = edx.split.cos_max + 1;
+	hw_res->num_closid = edx.split.cos_max + 1;
 	max_delay = eax.split.max_delay + 1;
 	r->default_ctrl = MAX_MBA_BW;
 	r->membw.arch_needs_linear = true;
@@ -287,12 +303,13 @@ static bool __get_mem_config_intel(struct rdt_resource *r)
 
 static bool __rdt_get_mem_config_amd(struct rdt_resource *r)
 {
+	struct rdt_hw_resource *hw_res = resctrl_to_arch_res(r);
 	union cpuid_0x10_3_eax eax;
 	union cpuid_0x10_x_edx edx;
 	u32 ebx, ecx;
 
 	cpuid_count(0x80000020, 1, &eax.full, &ebx, &ecx, &edx.full);
-	r->num_closid = edx.split.cos_max + 1;
+	hw_res->num_closid = edx.split.cos_max + 1;
 	r->default_ctrl = MAX_MBA_BW_AMD;
 
 	/* AMD does not use delay */
@@ -317,12 +334,13 @@ static bool __rdt_get_mem_config_amd(struct rdt_resource *r)
 
 static void rdt_get_cache_alloc_cfg(int idx, struct rdt_resource *r)
 {
+	struct rdt_hw_resource *hw_res = resctrl_to_arch_res(r);
 	union cpuid_0x10_1_eax eax;
 	union cpuid_0x10_x_edx edx;
 	u32 ebx, ecx;
 
 	cpuid_count(0x00000010, idx, &eax.full, &ebx, &ecx, &edx.full);
-	r->num_closid = edx.split.cos_max + 1;
+	hw_res->num_closid = edx.split.cos_max + 1;
 	r->cache.cbm_len = eax.split.cbm_len + 1;
 	r->default_ctrl = BIT_MASK(eax.split.cbm_len + 1) - 1;
 	r->cache.shareable_bits = ebx & r->default_ctrl;
@@ -333,10 +351,12 @@ static void rdt_get_cache_alloc_cfg(int idx, struct rdt_resource *r)
 
 static void rdt_get_cdp_config(int level, int type)
 {
-	struct rdt_resource *r_l = &rdt_resources_all[level];
-	struct rdt_resource *r = &rdt_resources_all[type];
+	struct rdt_resource *r_l = &rdt_resources_all[level].resctrl;
+	struct rdt_hw_resource *hw_res_l = resctrl_to_arch_res(r_l);
+	struct rdt_resource *r = &rdt_resources_all[type].resctrl;
+	struct rdt_hw_resource *hw_res = resctrl_to_arch_res(r);
 
-	r->num_closid = r_l->num_closid / 2;
+	hw_res->num_closid = hw_res_l->num_closid / 2;
 	r->cache.cbm_len = r_l->cache.cbm_len;
 	r->default_ctrl = r_l->default_ctrl;
 	r->cache.shareable_bits = r_l->cache.shareable_bits;
@@ -364,10 +384,11 @@ static void rdt_get_cdp_l2_config(void)
 static void
 mba_wrmsr_amd(struct rdt_domain *d, struct msr_param *m, struct rdt_resource *r)
 {
+	struct rdt_hw_resource *hw_res = resctrl_to_arch_res(r);
 	unsigned int i;
 
 	for (i = m->low; i < m->high; i++)
-		wrmsrl(r->msr_base + i, d->ctrl_val[i]);
+		wrmsrl(hw_res->msr_base + i, d->ctrl_val[i]);
 }
 
 /*
@@ -388,20 +409,22 @@ static void
 mba_wrmsr_intel(struct rdt_domain *d, struct msr_param *m,
 		struct rdt_resource *r)
 {
+	struct rdt_hw_resource *hw_res = resctrl_to_arch_res(r);
 	unsigned int i;
 
 	/*  Write the delay values for mba. */
 	for (i = m->low; i < m->high; i++)
-		wrmsrl(r->msr_base + i, delay_bw_map(d->ctrl_val[i], r));
+		wrmsrl(hw_res->msr_base + i, delay_bw_map(d->ctrl_val[i], r));
 }
 
 static void
 cat_wrmsr(struct rdt_domain *d, struct msr_param *m, struct rdt_resource *r)
 {
+	struct rdt_hw_resource *hw_res = resctrl_to_arch_res(r);
 	unsigned int i;
 
 	for (i = m->low; i < m->high; i++)
-		wrmsrl(r->msr_base + cbm_idx(r, i), d->ctrl_val[i]);
+		wrmsrl(hw_res->msr_base + cbm_idx(r, i), d->ctrl_val[i]);
 }
 
 struct rdt_domain *get_domain_from_cpu(int cpu, struct rdt_resource *r)
@@ -420,13 +443,14 @@ struct rdt_domain *get_domain_from_cpu(int cpu, struct rdt_resource *r)
 void rdt_ctrl_update(void *arg)
 {
 	struct msr_param *m = arg;
+	struct rdt_hw_resource *hw_res = resctrl_to_arch_res(m->res);
 	struct rdt_resource *r = m->res;
 	int cpu = smp_processor_id();
 	struct rdt_domain *d;
 
 	d = get_domain_from_cpu(cpu, r);
 	if (d) {
-		r->msr_update(d, m, r);
+		hw_res->msr_update(d, m, r);
 		return;
 	}
 	pr_warn_once("cpu %d not found in any domain for resource %s\n",
@@ -468,6 +492,7 @@ struct rdt_domain *rdt_find_domain(struct rdt_resource *r, int id,
 
 void setup_default_ctrlval(struct rdt_resource *r, u32 *dc, u32 *dm)
 {
+	struct rdt_hw_resource *hw_res = resctrl_to_arch_res(r);
 	int i;
 
 	/*
@@ -476,7 +501,7 @@ void setup_default_ctrlval(struct rdt_resource *r, u32 *dc, u32 *dm)
 	 * For Memory Allocation: Set b/w requested to 100%
 	 * and the bandwidth in MBps to U32_MAX
 	 */
-	for (i = 0; i < r->num_closid; i++, dc++, dm++) {
+	for (i = 0; i < hw_res->num_closid; i++, dc++, dm++) {
 		*dc = r->default_ctrl;
 		*dm = MBA_MAX_MBPS;
 	}
@@ -484,14 +509,15 @@ void setup_default_ctrlval(struct rdt_resource *r, u32 *dc, u32 *dm)
 
 static int domain_setup_ctrlval(struct rdt_resource *r, struct rdt_domain *d)
 {
+	struct rdt_hw_resource *hw_res = resctrl_to_arch_res(r);
 	struct msr_param m;
 	u32 *dc, *dm;
 
-	dc = kmalloc_array(r->num_closid, sizeof(*d->ctrl_val), GFP_KERNEL);
+	dc = kmalloc_array(hw_res->num_closid, sizeof(*d->ctrl_val), GFP_KERNEL);
 	if (!dc)
 		return -ENOMEM;
 
-	dm = kmalloc_array(r->num_closid, sizeof(*d->mbps_val), GFP_KERNEL);
+	dm = kmalloc_array(hw_res->num_closid, sizeof(*d->mbps_val), GFP_KERNEL);
 	if (!dm) {
 		kfree(dc);
 		return -ENOMEM;
@@ -502,8 +528,8 @@ static int domain_setup_ctrlval(struct rdt_resource *r, struct rdt_domain *d)
 	setup_default_ctrlval(r, dc, dm);
 
 	m.low = 0;
-	m.high = r->num_closid;
-	r->msr_update(d, &m, r);
+	m.high = hw_res->num_closid;
+	hw_res->msr_update(d, &m, r);
 	return 0;
 }
 
@@ -653,7 +679,7 @@ static void domain_remove_cpu(int cpu, struct rdt_resource *r)
 		return;
 	}
 
-	if (r == &rdt_resources_all[RDT_RESOURCE_L3]) {
+	if (r == &rdt_resources_all[RDT_RESOURCE_L3].resctrl) {
 		if (is_mbm_enabled() && cpu == d->mbm_work_cpu) {
 			cancel_delayed_work(&d->mbm_over);
 			mbm_setup_overflow_handler(d, 0);
@@ -829,9 +855,9 @@ static __init bool get_mem_config(void)
 		return false;
 
 	if (boot_cpu_data.x86_vendor == X86_VENDOR_INTEL)
-		return __get_mem_config_intel(&rdt_resources_all[RDT_RESOURCE_MBA]);
+		return __get_mem_config_intel(&rdt_resources_all[RDT_RESOURCE_MBA].resctrl);
 	else if (boot_cpu_data.x86_vendor == X86_VENDOR_AMD)
-		return __rdt_get_mem_config_amd(&rdt_resources_all[RDT_RESOURCE_MBA]);
+		return __rdt_get_mem_config_amd(&rdt_resources_all[RDT_RESOURCE_MBA].resctrl);
 
 	return false;
 }
@@ -847,14 +873,14 @@ static __init bool get_rdt_alloc_resources(void)
 		return false;
 
 	if (rdt_cpu_has(X86_FEATURE_CAT_L3)) {
-		rdt_get_cache_alloc_cfg(1, &rdt_resources_all[RDT_RESOURCE_L3]);
+		rdt_get_cache_alloc_cfg(1, &rdt_resources_all[RDT_RESOURCE_L3].resctrl);
 		if (rdt_cpu_has(X86_FEATURE_CDP_L3))
 			rdt_get_cdp_l3_config();
 		ret = true;
 	}
 	if (rdt_cpu_has(X86_FEATURE_CAT_L2)) {
 		/* CPUID 0x10.2 fields are same format at 0x10.1 */
-		rdt_get_cache_alloc_cfg(2, &rdt_resources_all[RDT_RESOURCE_L2]);
+		rdt_get_cache_alloc_cfg(2, &rdt_resources_all[RDT_RESOURCE_L2].resctrl);
 		if (rdt_cpu_has(X86_FEATURE_CDP_L2))
 			rdt_get_cdp_l2_config();
 		ret = true;
@@ -878,7 +904,7 @@ static __init bool get_rdt_mon_resources(void)
 	if (!rdt_mon_features)
 		return false;
 
-	return !rdt_get_mon_l3_config(&rdt_resources_all[RDT_RESOURCE_L3]);
+	return !rdt_get_mon_l3_config(&rdt_resources_all[RDT_RESOURCE_L3].resctrl);
 }
 
 static __init void __check_quirks_intel(void)
@@ -912,9 +938,14 @@ static __init bool get_rdt_resources(void)
 
 static __init void rdt_init_res_defs_intel(void)
 {
+	struct rdt_hw_resource *hw_res;
 	struct rdt_resource *r;
+	int i;
+
+	for (i = 0; i < RDT_NUM_RESOURCES; i++) {
+		hw_res = &rdt_resources_all[i];
+		r = &rdt_resources_all[i].resctrl;
 
-	for_each_rdt_resource(r) {
 		if (r->rid == RDT_RESOURCE_L3 ||
 		    r->rid == RDT_RESOURCE_L3DATA ||
 		    r->rid == RDT_RESOURCE_L3CODE ||
@@ -924,17 +955,22 @@ static __init void rdt_init_res_defs_intel(void)
 			r->cache.arch_has_sparse_bitmaps = false;
 			r->cache.arch_has_empty_bitmaps = false;
 		} else if (r->rid == RDT_RESOURCE_MBA) {
-			r->msr_base = MSR_IA32_MBA_THRTL_BASE;
-			r->msr_update = mba_wrmsr_intel;
+			hw_res->msr_base = MSR_IA32_MBA_THRTL_BASE;
+			hw_res->msr_update = mba_wrmsr_intel;
 		}
 	}
 }
 
 static __init void rdt_init_res_defs_amd(void)
 {
+	struct rdt_hw_resource *hw_res;
 	struct rdt_resource *r;
+	int i;
+
+	for (i = 0; i < RDT_NUM_RESOURCES; i++) {
+		hw_res = &rdt_resources_all[i];
+		r = &rdt_resources_all[i].resctrl;
 
-	for_each_rdt_resource(r) {
 		if (r->rid == RDT_RESOURCE_L3 ||
 		    r->rid == RDT_RESOURCE_L3DATA ||
 		    r->rid == RDT_RESOURCE_L3CODE ||
@@ -944,8 +980,8 @@ static __init void rdt_init_res_defs_amd(void)
 			r->cache.arch_has_sparse_bitmaps = true;
 			r->cache.arch_has_empty_bitmaps = true;
 		} else if (r->rid == RDT_RESOURCE_MBA) {
-			r->msr_base = MSR_IA32_MBA_BW_BASE;
-			r->msr_update = mba_wrmsr_amd;
+			hw_res->msr_base = MSR_IA32_MBA_BW_BASE;
+			hw_res->msr_update = mba_wrmsr_amd;
 		}
 	}
 }
arch/x86/kernel/cpu/resctrl/ctrlmondata.c +10 -4
index c877642e8a14..ab6e584c9d2d 100644
@@ -284,10 +284,12 @@ int update_domains(struct rdt_resource *r, int closid)
 static int rdtgroup_parse_resource(char *resname, char *tok,
 				   struct rdtgroup *rdtgrp)
 {
+	struct rdt_hw_resource *hw_res;
 	struct rdt_resource *r;
 
 	for_each_alloc_enabled_rdt_resource(r) {
-		if (!strcmp(resname, r->name) && rdtgrp->closid < r->num_closid)
+		hw_res = resctrl_to_arch_res(r);
+		if (!strcmp(resname, r->name) && rdtgrp->closid < hw_res->num_closid)
 			return parse_line(tok, r, rdtgrp);
 	}
 	rdt_last_cmd_printf("Unknown or unsupported resource name '%s'\n", resname);
@@ -394,6 +396,7 @@ static void show_doms(struct seq_file *s, struct rdt_resource *r, int closid)
 int rdtgroup_schemata_show(struct kernfs_open_file *of,
 			   struct seq_file *s, void *v)
 {
+	struct rdt_hw_resource *hw_res;
 	struct rdtgroup *rdtgrp;
 	struct rdt_resource *r;
 	int ret = 0;
@@ -418,7 +421,8 @@ int rdtgroup_schemata_show(struct kernfs_open_file *of,
 		} else {
 			closid = rdtgrp->closid;
 			for_each_alloc_enabled_rdt_resource(r) {
-				if (closid < r->num_closid)
+				hw_res = resctrl_to_arch_res(r);
+				if (closid < hw_res->num_closid)
 					show_doms(s, r, closid);
 			}
 		}
@@ -449,6 +453,7 @@ void mon_event_read(struct rmid_read *rr, struct rdt_resource *r,
 int rdtgroup_mondata_show(struct seq_file *m, void *arg)
 {
 	struct kernfs_open_file *of = m->private;
+	struct rdt_hw_resource *hw_res;
 	u32 resid, evtid, domid;
 	struct rdtgroup *rdtgrp;
 	struct rdt_resource *r;
@@ -468,7 +473,8 @@ int rdtgroup_mondata_show(struct seq_file *m, void *arg)
 	domid = md.u.domid;
 	evtid = md.u.evtid;
 
-	r = &rdt_resources_all[resid];
+	hw_res = &rdt_resources_all[resid];
+	r = &hw_res->resctrl;
 	d = rdt_find_domain(r, domid, NULL);
 	if (IS_ERR_OR_NULL(d)) {
 		ret = -ENOENT;
@@ -482,7 +488,7 @@ int rdtgroup_mondata_show(struct seq_file *m, void *arg)
 	else if (rr.val & RMID_VAL_UNAVAIL)
 		seq_puts(m, "Unavailable\n");
 	else
-		seq_printf(m, "%llu\n", rr.val * r->mon_scale);
+		seq_printf(m, "%llu\n", rr.val * hw_res->mon_scale);
 
 out:
 	rdtgroup_kn_unlock(of->kn);
arch/x86/kernel/cpu/resctrl/internal.h +29 -109

195 lines not on this page · raw message

arch/x86/kernel/cpu/resctrl/monitor.c +18 -14
index 54dffe574e67..d0afd76d3578 100644
@@ -111,7 +111,7 @@ void __check_limbo(struct rdt_domain *d, bool force_free)
 	struct rdt_resource *r;
 	u32 crmid = 1, nrmid;
 
-	r = &rdt_resources_all[RDT_RESOURCE_L3];
+	r = &rdt_resources_all[RDT_RESOURCE_L3].resctrl;
 
 	/*
 	 * Skip RMID 0 and start from RMID 1 and check all the RMIDs that
@@ -169,7 +169,7 @@ static void add_rmid_to_limbo(struct rmid_entry *entry)
 	int cpu;
 	u64 val;
 
-	r = &rdt_resources_all[RDT_RESOURCE_L3];
+	r = &rdt_resources_all[RDT_RESOURCE_L3].resctrl;
 
 	entry->busy = 0;
 	cpu = get_cpu();
@@ -224,6 +224,7 @@ static u64 mbm_overflow_count(u64 prev_msr, u64 cur_msr, unsigned int width)
 
 static int __mon_event_count(u32 rmid, struct rmid_read *rr)
 {
+	struct rdt_hw_resource *hw_res = resctrl_to_arch_res(rr->r);
 	struct mbm_state *m;
 	u64 chunks, tval;
 
@@ -256,7 +257,7 @@ static int __mon_event_count(u32 rmid, struct rmid_read *rr)
 		return 0;
 	}
 
-	chunks = mbm_overflow_count(m->prev_msr, tval, rr->r->mbm_width);
+	chunks = mbm_overflow_count(m->prev_msr, tval, hw_res->mbm_width);
 	m->chunks += chunks;
 	m->prev_msr = tval;
 
@@ -270,7 +271,7 @@ static int __mon_event_count(u32 rmid, struct rmid_read *rr)
  */
 static void mbm_bw_count(u32 rmid, struct rmid_read *rr)
 {
-	struct rdt_resource *r = &rdt_resources_all[RDT_RESOURCE_L3];
+	struct rdt_hw_resource *hw_res = resctrl_to_arch_res(rr->r);
 	struct mbm_state *m = &rr->d->mbm_local[rmid];
 	u64 tval, cur_bw, chunks;
 
@@ -278,9 +279,9 @@ static void mbm_bw_count(u32 rmid, struct rmid_read *rr)
 	if (tval & (RMID_VAL_ERROR | RMID_VAL_UNAVAIL))
 		return;
 
-	chunks = mbm_overflow_count(m->prev_bw_msr, tval, rr->r->mbm_width);
+	chunks = mbm_overflow_count(m->prev_bw_msr, tval, hw_res->mbm_width);
 	m->chunks += chunks;
-	cur_bw = (chunks * r->mon_scale) >> 20;
+	cur_bw = (chunks * hw_res->mon_scale) >> 20;
 
 	if (m->delta_comp)
 		m->delta_bw = abs(cur_bw - m->prev_bw);
@@ -353,6 +354,7 @@ static void update_mba_bw(struct rdtgroup *rgrp, struct rdt_domain *dom_mbm)
 {
 	u32 closid, rmid, cur_msr, cur_msr_val, new_msr_val;
 	struct mbm_state *pmbm_data, *cmbm_data;
+	struct rdt_hw_resource *hw_r_mba;
 	u32 cur_bw, delta_bw, user_bw;
 	struct rdt_resource *r_mba;
 	struct rdt_domain *dom_mba;
@@ -362,7 +364,8 @@ static void update_mba_bw(struct rdtgroup *rgrp, struct rdt_domain *dom_mbm)
 	if (!is_mbm_local_enabled())
 		return;
 
-	r_mba = &rdt_resources_all[RDT_RESOURCE_MBA];
+	hw_r_mba = &rdt_resources_all[RDT_RESOURCE_MBA];
+	r_mba = &hw_r_mba->resctrl;
 	closid = rgrp->closid;
 	rmid = rgrp->mon.rmid;
 	pmbm_data = &dom_mbm->mbm_local[rmid];
@@ -411,7 +414,7 @@ static void update_mba_bw(struct rdtgroup *rgrp, struct rdt_domain *dom_mbm)
 		return;
 	}
 
-	cur_msr = r_mba->msr_base + closid;
+	cur_msr = hw_r_mba->msr_base + closid;
 	wrmsrl(cur_msr, delay_bw_map(new_msr_val, r_mba));
 	dom_mba->ctrl_val[closid] = new_msr_val;
 
@@ -476,7 +479,7 @@ void cqm_handle_limbo(struct work_struct *work)
 
 	mutex_lock(&rdtgroup_mutex);
 
-	r = &rdt_resources_all[RDT_RESOURCE_L3];
+	r = &rdt_resources_all[RDT_RESOURCE_L3].resctrl;
 	d = container_of(work, struct rdt_domain, cqm_limbo.work);
 
 	__check_limbo(d, false);
@@ -512,7 +515,7 @@ void mbm_handle_overflow(struct work_struct *work)
 	if (!static_branch_likely(&rdt_mon_enable_key))
 		goto out_unlock;
 
-	r = &rdt_resources_all[RDT_RESOURCE_L3];
+	r = &rdt_resources_all[RDT_RESOURCE_L3].resctrl;
 	d = container_of(work, struct rdt_domain, mbm_over.work);
 
 	list_for_each_entry(prgrp, &rdt_all_groups, rdtgroup_list) {
@@ -609,15 +612,16 @@ static void l3_mon_evt_init(struct rdt_resource *r)
 int rdt_get_mon_l3_config(struct rdt_resource *r)
 {
 	unsigned int mbm_offset = boot_cpu_data.x86_cache_mbm_width_offset;
+	struct rdt_hw_resource *hw_res = resctrl_to_arch_res(r);
 	unsigned int cl_size = boot_cpu_data.x86_cache_size;
 	int ret;
 
-	r->mon_scale = boot_cpu_data.x86_cache_occ_scale;
+	hw_res->mon_scale = boot_cpu_data.x86_cache_occ_scale;
 	r->num_rmid = boot_cpu_data.x86_cache_max_rmid + 1;
-	r->mbm_width = MBM_CNTR_WIDTH_BASE;
+	hw_res->mbm_width = MBM_CNTR_WIDTH_BASE;
 
 	if (mbm_offset > 0 && mbm_offset <= MBM_CNTR_WIDTH_OFFSET_MAX)
-		r->mbm_width += mbm_offset;
+		hw_res->mbm_width += mbm_offset;
 	else if (mbm_offset > MBM_CNTR_WIDTH_OFFSET_MAX)
 		pr_warn("Ignoring impossible MBM counter offset\n");
 
@@ -631,7 +635,7 @@ int rdt_get_mon_l3_config(struct rdt_resource *r)
 	resctrl_cqm_threshold = cl_size * 1024 / r->num_rmid;
 
 	/* h/w works in units of "boot_cpu_data.x86_cache_occ_scale" */
-	resctrl_cqm_threshold /= r->mon_scale;
+	resctrl_cqm_threshold /= hw_res->mon_scale;
 
 	ret = dom_data_init(r);
 	if (ret)
arch/x86/kernel/cpu/resctrl/pseudo_lock.c +2 -2
index 0daf2f1cf7a8..d58f2ffa65e0 100644
@@ -684,8 +684,8 @@ int rdtgroup_locksetup_enter(struct rdtgroup *rdtgrp)
 	 *   resource, the portion of cache used by it should be made
 	 *   unavailable to all future allocations from both resources.
 	 */
-	if (rdt_resources_all[RDT_RESOURCE_L3DATA].alloc_enabled ||
-	    rdt_resources_all[RDT_RESOURCE_L2DATA].alloc_enabled) {
+	if (rdt_resources_all[RDT_RESOURCE_L3DATA].resctrl.alloc_enabled ||
+	    rdt_resources_all[RDT_RESOURCE_L2DATA].resctrl.alloc_enabled) {
 		rdt_last_cmd_puts("CDP enabled\n");
 		return -EINVAL;
 	}
arch/x86/kernel/cpu/resctrl/rdtgroup.c +39 -30

199 lines not on this page · raw message

include/linux/resctrl.h +117 -0

127 lines not on this page · raw message

Signature · 1 line
2.28.0
maintainer
#2 lore raw

Re: [PATCH 01/24] x86/resctrl: Split struct rdt_resource

Hi James,

Quoted 3 lines · James Morse

resctrl is the defacto Linux ABI for SoC resource partitioning features. To support it on another architecture, it needs to be abstracted from Intel RDT, and moved it to /fs/.

Current support for AMD PQoS should also be considered.

s/and moved it to/and moved to/?

Quoted 5 lines

Start by splitting struct rdt_resource, (the name is kept to keep the noise down), and add some type-trickery to keep the foreach helpers working.

Move everything that that is particular to resctrl into a new header

s/that that/that/

Quoted 5 lines

file, keeping the x86 hardware accessors where they are. resctrl code paths touching a 'hw' struct indicates where an abstraction is needed.

Splitting rdt_domain up in a similar way happens in the next patch. No change in behaviour, this patch just moves types around.

Please remove the "this patch" term.

Quoted 16 lines
---
  arch/x86/kernel/cpu/resctrl/core.c        | 258 ++++++++++++----------
  arch/x86/kernel/cpu/resctrl/ctrlmondata.c |  14 +-
  arch/x86/kernel/cpu/resctrl/internal.h    | 138 +++---------
  arch/x86/kernel/cpu/resctrl/monitor.c     |  32 +--
  arch/x86/kernel/cpu/resctrl/pseudo_lock.c |   4 +-
  arch/x86/kernel/cpu/resctrl/rdtgroup.c    |  69 +++---
  include/linux/resctrl.h                   | 117 ++++++++++
  7 files changed, 362 insertions(+), 270 deletions(-)
arch/x86/kernel/cpu/resctrl/core.c +0 -0
index e5f4ee8f4c3b..470661f2eb68 100644

...

Quoted 6 lines
arch/x86/kernel/cpu/resctrl/core.c +1 -0
@@ -912,9 +938,14 @@ static __init bool get_rdt_resources(void)
 
  static __init void rdt_init_res_defs_intel(void)
  {
+	struct rdt_hw_resource *hw_res;
  	struct rdt_resource *r;

arch/x86/kernel/cpu/resctrl/core.c static __init bool get_rdt_resources(void) · line 951

Quoted 10 lines
arch/x86/kernel/cpu/resctrl/core.c +5 -1
+	int i;
+
+	for (i = 0; i < RDT_NUM_RESOURCES; i++) {
+		hw_res = &rdt_resources_all[i];
+		r = &rdt_resources_all[i].resctrl;
 
-	for_each_rdt_resource(r) {
  		if (r->rid == RDT_RESOURCE_L3 ||
  		    r->rid == RDT_RESOURCE_L3DATA ||
  		    r->rid == RDT_RESOURCE_L3CODE ||

Could using for_each_rdt_resource() remain with the additional assignment of hw_res? Similar to the later usage of for_each_alloc_enabled_rdt_resource()?

Quoted 16 lines
arch/x86/kernel/cpu/resctrl/core.c +3 -2
@@ -924,17 +955,22 @@ static __init void rdt_init_res_defs_intel(void)
  			r->cache.arch_has_sparse_bitmaps = false;
  			r->cache.arch_has_empty_bitmaps = false;
  		} else if (r->rid == RDT_RESOURCE_MBA) {
-			r->msr_base = MSR_IA32_MBA_THRTL_BASE;
-			r->msr_update = mba_wrmsr_intel;
+			hw_res->msr_base = MSR_IA32_MBA_THRTL_BASE;
+			hw_res->msr_update = mba_wrmsr_intel;
  		}
  	}
  }
 
  static __init void rdt_init_res_defs_amd(void)
  {
+	struct rdt_hw_resource *hw_res;
  	struct rdt_resource *r;

arch/x86/kernel/cpu/resctrl/core.c static __init void rdt_init_res_defs_intel(void) · line 976

Quoted 10 lines
arch/x86/kernel/cpu/resctrl/core.c +5 -1
+	int i;
+
+	for (i = 0; i < RDT_NUM_RESOURCES; i++) {
+		hw_res = &rdt_resources_all[i];
+		r = &rdt_resources_all[i].resctrl;
 
-	for_each_rdt_resource(r) {
  		if (r->rid == RDT_RESOURCE_L3 ||
  		    r->rid == RDT_RESOURCE_L3DATA ||
  		    r->rid == RDT_RESOURCE_L3CODE ||

Same here (keep using for_each_rdt_resource()?).

Quoted 18 lines
arch/x86/kernel/cpu/resctrl/core.c +2 -2
@@ -944,8 +980,8 @@ static __init void rdt_init_res_defs_amd(void)
  			r->cache.arch_has_sparse_bitmaps = true;
  			r->cache.arch_has_empty_bitmaps = true;
  		} else if (r->rid == RDT_RESOURCE_MBA) {
-			r->msr_base = MSR_IA32_MBA_BW_BASE;
-			r->msr_update = mba_wrmsr_amd;
+			hw_res->msr_base = MSR_IA32_MBA_BW_BASE;
+			hw_res->msr_update = mba_wrmsr_amd;
  		}
  	}
  }
arch/x86/kernel/cpu/resctrl/ctrlmondata.c +0 -0
index c877642e8a14..ab6e584c9d2d 100644
@@ -284,10 +284,12 @@ int update_domains(struct rdt_resource *r, int closid)
  static int rdtgroup_parse_resource(char *resname, char *tok,
  				   struct rdtgroup *rdtgrp)

arch/x86/kernel/cpu/resctrl/ctrlmondata.c int update_domains(struct rdt_resource *r, int closid) · line 294

Quoted 10 lines
arch/x86/kernel/cpu/resctrl/ctrlmondata.c +3 -1
  {
+	struct rdt_hw_resource *hw_res;
  	struct rdt_resource *r;
 
  	for_each_alloc_enabled_rdt_resource(r) {
-		if (!strcmp(resname, r->name) && rdtgrp->closid < r->num_closid)
+		hw_res = resctrl_to_arch_res(r);
+		if (!strcmp(resname, r->name) && rdtgrp->closid < hw_res->num_closid)
  			return parse_line(tok, r, rdtgrp);
  	}

This is the usage of for_each_alloc_enabled_rdt_resource() I refer to earlier.

Quoted 1 line
arch/x86/kernel/cpu/resctrl/ctrlmondata.c +0 -0
  	rdt_last_cmd_printf("Unknown or unsupported resource name '%s'\n", resname);

...

Quoted 4 lines
arch/x86/kernel/cpu/resctrl/internal.h +0 -0
index 80fa997fae60..bcae86138cad 100644

...

Quoted 5 lines
arch/x86/kernel/cpu/resctrl/internal.h +0 -1
@@ -438,56 +381,29 @@ struct rdt_parse_data {
  };
 
  /**
- * struct rdt_resource - attributes of an RDT resource

arch/x86/kernel/cpu/resctrl/internal.h struct rdt_parse_data { · line 384

Quoted 10 lines
arch/x86/kernel/cpu/resctrl/internal.h +1 -9
- * @rid:		The index of the resource
- * @alloc_enabled:	Is allocation enabled on this machine
- * @mon_enabled:	Is monitoring enabled for this feature
- * @alloc_capable:	Is allocation available on this machine
- * @mon_capable:	Is monitor feature available on this machine
- * @name:		Name to use in "schemata" file
- * @num_closid:		Number of CLOSIDs available
- * @cache_level:	Which cache level defines scope of this resource
- * @default_ctrl:	Specifies default cache cbm or memory B/W percent.
+ * struct rdt_hw_resource - hw attributes of an RDT resource

Perhaps rather "hw attributes of a resctrl resource" to help with the transition away from being specific to Intel RDT.

Quoted 13 lines
arch/x86/kernel/cpu/resctrl/internal.h +1 -8
+ * @num_closid:		Number of CLOSIDs available.
   * @msr_base:		Base MSR address for CBMs
   * @msr_update:		Function pointer to update QOS MSRs
- * @data_width:		Character width of data when displaying
- * @domains:		All domains for this resource
- * @cache:		Cache allocation related data
- * @format_str:		Per resource format string to show domain value
- * @parse_ctrlval:	Per resource function pointer to parse control values
- * @evt_list:		List of monitoring events
- * @num_rmid:		Number of RMIDs available
   * @mon_scale:		cqm counter * mon_scale = occupancy in bytes
- * @fflags:		flags to choose base and info files
   */

arch/x86/kernel/cpu/resctrl/internal.h struct rdt_parse_data { · line 392

Quoted 10 lines
arch/x86/kernel/cpu/resctrl/internal.h +3 -7
-struct rdt_resource {
-	int			rid;
-	bool			alloc_enabled;
-	bool			mon_enabled;
-	bool			alloc_capable;
-	bool			mon_capable;
-	char			*name;
+struct rdt_hw_resource {
+	struct rdt_resource     resctrl;
+

Please use tabs and spaces similar to the existing format.

arch/x86/kernel/cpu/resctrl/internal.h struct rdt_parse_data { · line 394

Quoted 4 lines
arch/x86/kernel/cpu/resctrl/internal.h +1 -2
  	int			num_closid;
-	int			cache_level;
-	u32			default_ctrl;
+

Please remove these empty lines.

Quoted 26 lines
arch/x86/kernel/cpu/resctrl/internal.h +5 -11
  	unsigned int		msr_base;
  	void (*msr_update)	(struct rdt_domain *d, struct msr_param *m,
  				 struct rdt_resource *r);
-	int			data_width;
-	struct list_head	domains;
-	struct rdt_cache	cache;
-	struct rdt_membw	membw;
-	const char		*format_str;
-	int (*parse_ctrlval)(struct rdt_parse_data *data,
-			     struct rdt_resource *r,
-			     struct rdt_domain *d);
-	struct list_head	evt_list;
-	int			num_rmid;
  	unsigned int		mon_scale;
  	unsigned int		mbm_width;
-	unsigned long		fflags;
  };
 
+static inline struct rdt_hw_resource *resctrl_to_arch_res(struct rdt_resource *r)
+{
+	return container_of(r, struct rdt_hw_resource, resctrl);
+}
+
  int parse_cbm(struct rdt_parse_data *data, struct rdt_resource *r,
  	      struct rdt_domain *d);
  int parse_bw(struct rdt_parse_data *data, struct rdt_resource *r,

...

Quoted 70 lines
include/linux/resctrl.h +55 -0
index 9b05af9b3e28..b2c2b7386d28 100644
@@ -2,6 +2,8 @@
  #ifndef _RESCTRL_H
  #define _RESCTRL_H
 
+#include <linux/kernel.h>
+#include <linux/list.h>
  #include <linux/pid.h>
 
  #ifdef CONFIG_PROC_CPU_RESCTRL
@@ -13,4 +15,119 @@ int proc_resctrl_show(struct seq_file *m,
 
  #endif
 
+struct rdt_domain;
+
+/**
+ * struct resctrl_cache - Cache allocation related data
+ * @cbm_len:		Length of the cache bit mask
+ * @min_cbm_bits:	Minimum number of consecutive bits to be set
+ * @cbm_idx_mult:	Multiplier of CBM index
+ * @cbm_idx_offset:	Offset of CBM index. CBM index is computed by:
+ *			closid * cbm_idx_multi + cbm_idx_offset
+ *			in a cache bit mask
+ * @shareable_bits:	Bitmask of shareable resource with other
+ *			executing entities
+ * @arch_has_sparse_bitmaps:	True if a bitmap like f00f is valid.
+ * @arch_has_empty_bitmaps:	True if the '0' bitmap is valid.
+ */
+struct resctrl_cache {
+	u32		cbm_len;
+	u32		min_cbm_bits;
+	unsigned int	cbm_idx_mult;	// TODO remove this
+	unsigned int	cbm_idx_offset; // TODO remove this
+	u32		shareable_bits;
+	bool		arch_has_sparse_bitmaps;
+	bool		arch_has_empty_bitmaps;
+};
+
+/**
+ * enum membw_throttle_mode - System's memory bandwidth throttling mode
+ * @THREAD_THROTTLE_UNDEFINED:	Not relevant to the system
+ * @THREAD_THROTTLE_MAX:	Memory bandwidth is throttled at the core
+ *				always using smallest bandwidth percentage
+ *				assigned to threads, aka "max throttling"
+ * @THREAD_THROTTLE_PER_THREAD:	Memory bandwidth is throttled at the thread
+ */
+enum membw_throttle_mode {
+	THREAD_THROTTLE_UNDEFINED = 0,
+	THREAD_THROTTLE_MAX,
+	THREAD_THROTTLE_PER_THREAD,
+};
+
+/**
+ * struct resctrl_membw - Memory bandwidth allocation related data
+ * @min_bw:		Minimum memory bandwidth percentage user can request
+ * @bw_gran:		Granularity at which the memory bandwidth is allocated
+ * @delay_linear:	True if memory B/W delay is in linear scale
+ * @arch_needs_linear:	True if we can't configure non-linear resources
+ * @throttle_mode:	Bandwidth throttling mode when threads request
+ *			different memory bandwidths
+ * @mba_sc:		True if MBA software controller(mba_sc) is enabled
+ * @mb_map:		Mapping of memory B/W percentage to memory B/W delay
+ */
+struct resctrl_membw {
+	u32				min_bw;
+	u32				bw_gran;

include/linux/resctrl.h int proc_resctrl_show(struct seq_file *m, · line 80

Quoted 10 lines
include/linux/resctrl.h +10 -0
+	u32				delay_linear;
+	bool				arch_needs_linear;
+	enum membw_throttle_mode	throttle_mode;
+	bool				mba_sc;
+	u32				*mb_map;
+};
+
+struct rdt_parse_data;
+
+/**

Missing a kernel-doc description here.

Quoted 13 lines
include/linux/resctrl.h +13 -0
+ * @rid:		The index of the resource
+ * @alloc_enabled:	Is allocation enabled on this machine
+ * @mon_enabled:	Is monitoring enabled for this feature
+ * @alloc_capable:	Is allocation available on this machine
+ * @mon_capable:	Is monitor feature available on this machine
+ *
+ * @num_rmid:		Number of RMIDs available.
+ *
+ * @cache_level:	Which cache level defines scope of this resource
+ *
+ * @cache:		If the component has cache controls, their properties.
+ * @membw:		If the component has bandwidth controls, their properties.
+ *

include/linux/resctrl.h int proc_resctrl_show(struct seq_file *m, · line 103

Quoted 10 lines
include/linux/resctrl.h +10 -0
+ * @domains:		All domains for this resource
+ *
+ * @name:		Name to use in "schemata" file.
+ * @data_width:		Character width of data when displaying.
+ * @default_ctrl:	Specifies default cache cbm or memory B/W percent.
+ * @format_str:		Per resource format string to show domain value
+ * @parse_ctrlval:	Per resource function pointer to parse control values
+ *
+ * @evt_list:		List of monitoring events
+ * @fflags:		flags to choose base and info files

Please remove the empty lines in the above. It hints at some grouping that is unclear.

Quoted 7 lines
include/linux/resctrl.h +7 -0
+ */
+struct rdt_resource {
+	int			rid;
+	bool			alloc_enabled;
+	bool			mon_enabled;
+	bool			alloc_capable;
+	bool			mon_capable;

include/linux/resctrl.h int proc_resctrl_show(struct seq_file *m, · line 120

Quoted 10 lines
include/linux/resctrl.h +10 -0
+
+	int			num_rmid;
+
+	int			cache_level;
+
+	struct resctrl_cache	cache;
+	struct resctrl_membw	membw;
+
+	struct list_head	domains;
+

Please remove empty lines.

Quoted 4 lines
include/linux/resctrl.h +3 -0
+	char			*name;
+	int			data_width;
+	u32			default_ctrl;

include/linux/resctrl.h int proc_resctrl_show(struct seq_file *m, · line 133

Quoted 10 lines
include/linux/resctrl.h +9 -0
+	const char		*format_str;
+	int			(*parse_ctrlval)(struct rdt_parse_data *data,
+						 struct rdt_resource *r,
+						 struct rdt_domain *d);
+	struct list_head	evt_list;
+	unsigned long		fflags;
+
+};
+
  #endif /* _RESCTRL_H */

Reinette

author reviewer
#3 lore raw

Re: [PATCH 01/24] x86/resctrl: Split struct rdt_resource

Hi Reinette,

Quoted 20 lines · Reinette Chatre
Quoted 2 lines · James Morse

Splitting rdt_domain up in a similar way happens in the next patch. No change in behaviour, this patch just moves types around.

Please remove the "this patch" term.

Quoted 14 lines
arch/x86/kernel/cpu/resctrl/core.c +5 -0
index e5f4ee8f4c3b..470661f2eb68 100644
@@ -912,9 +938,14 @@ static __init bool get_rdt_resources(void)
     static __init void rdt_init_res_defs_intel(void)
   {
+    struct rdt_hw_resource *hw_res;
       struct rdt_resource *r;
+    int i;
+
+    for (i = 0; i < RDT_NUM_RESOURCES; i++) {
+        hw_res = &rdt_resources_all[i];
Quoted 8 lines
Quoted 5 lines
arch/x86/kernel/cpu/resctrl/core.c +1 -0
+        r = &rdt_resources_all[i].resctrl;
   -    for_each_rdt_resource(r) {

          if (r->rid == RDT_RESOURCE_L3 ||

              r->rid == RDT_RESOURCE_L3DATA ||
              r->rid == RDT_RESOURCE_L3CODE ||

Could using for_each_rdt_resource() remain with the additional assignment of hw_res? Similar to the later usage of for_each_alloc_enabled_rdt_resource()?

Sure. I didn't do it here because of the back-to-back container_of(), even though the array is defined in the same file. But the compiler will remove all of that.

Quoted 11 lines
Quoted 11 lines
arch/x86/kernel/cpu/resctrl/ctrlmondata.c +1 -0
index c877642e8a14..ab6e584c9d2d 100644
@@ -284,10 +284,12 @@ int update_domains(struct rdt_resource *r, int closid)
   static int rdtgroup_parse_resource(char *resname, char *tok,
                      struct rdtgroup *rdtgrp)
   {
+    struct rdt_hw_resource *hw_res;
       struct rdt_resource *r;
Quoted 8 lines

arch/x86/kernel/cpu/resctrl/ctrlmondata.c int update_domains(struct rdt_resource *r, int closid) · line 294

Quoted 6 lines
arch/x86/kernel/cpu/resctrl/ctrlmondata.c +2 -1
         for_each_alloc_enabled_rdt_resource(r) {
-        if (!strcmp(resname, r->name) && rdtgrp->closid < r->num_closid)
+        hw_res = resctrl_to_arch_res(r);
+        if (!strcmp(resname, r->name) && rdtgrp->closid < hw_res->num_closid)
               return parse_line(tok, r, rdtgrp);
       }

This is the usage of for_each_alloc_enabled_rdt_resource() I refer to earlier.

(if it helps, the difference in treatment is because this one is to do with the filesystem interface, the others were clearly arch code)

Thanks,

James

author reviewer
#4 lore raw

PATCH 2/24 x86/resctrl: Split struct rdt_domain

In reply to James Morse

resctrl is the defacto Linux ABI for SoC resource partitioning features. To support it on another architecture, it needs to be abstracted from Intel RDT, and moved it to /fs/.

Split struct rdt_domain up too. Move everything that that is particular to resctrl into a new header file. resctrl code paths touching a 'hw' struct indicates where an abstraction is needed.

No change in behaviour, this patch just moves types around.

---
 arch/x86/kernel/cpu/resctrl/core.c        | 32 +++++++++++-------
 arch/x86/kernel/cpu/resctrl/ctrlmondata.c | 10 ++++--
 arch/x86/kernel/cpu/resctrl/internal.h    | 40 +++++------------------
 arch/x86/kernel/cpu/resctrl/monitor.c     |  8 +++--
 arch/x86/kernel/cpu/resctrl/rdtgroup.c    | 29 ++++++++++------
 include/linux/resctrl.h                   | 35 +++++++++++++++++++-
 6 files changed, 94 insertions(+), 60 deletions(-)
arch/x86/kernel/cpu/resctrl/core.c +20 -12
index 470661f2eb68..97040a54cc9a 100644
@@ -385,10 +385,11 @@ static void
 mba_wrmsr_amd(struct rdt_domain *d, struct msr_param *m, struct rdt_resource *r)
 {
 	struct rdt_hw_resource *hw_res = resctrl_to_arch_res(r);
+	struct rdt_hw_domain *hw_dom = resctrl_to_arch_dom(d);
 	unsigned int i;
 
 	for (i = m->low; i < m->high; i++)
-		wrmsrl(hw_res->msr_base + i, d->ctrl_val[i]);
+		wrmsrl(hw_res->msr_base + i, hw_dom->ctrl_val[i]);
 }
 
 /*
@@ -410,21 +411,23 @@ mba_wrmsr_intel(struct rdt_domain *d, struct msr_param *m,
 		struct rdt_resource *r)
 {
 	struct rdt_hw_resource *hw_res = resctrl_to_arch_res(r);
+	struct rdt_hw_domain *hw_dom = resctrl_to_arch_dom(d);
 	unsigned int i;
 
 	/*  Write the delay values for mba. */
 	for (i = m->low; i < m->high; i++)
-		wrmsrl(hw_res->msr_base + i, delay_bw_map(d->ctrl_val[i], r));
+		wrmsrl(hw_res->msr_base + i, delay_bw_map(hw_dom->ctrl_val[i], r));
 }
 
 static void
 cat_wrmsr(struct rdt_domain *d, struct msr_param *m, struct rdt_resource *r)
 {
 	struct rdt_hw_resource *hw_res = resctrl_to_arch_res(r);
+	struct rdt_hw_domain *hw_dom = resctrl_to_arch_dom(d);
 	unsigned int i;
 
 	for (i = m->low; i < m->high; i++)
-		wrmsrl(hw_res->msr_base + cbm_idx(r, i), d->ctrl_val[i]);
+		wrmsrl(hw_res->msr_base + cbm_idx(r, i), hw_dom->ctrl_val[i]);
 }
 
 struct rdt_domain *get_domain_from_cpu(int cpu, struct rdt_resource *r)
@@ -510,21 +513,22 @@ void setup_default_ctrlval(struct rdt_resource *r, u32 *dc, u32 *dm)
 static int domain_setup_ctrlval(struct rdt_resource *r, struct rdt_domain *d)
 {
 	struct rdt_hw_resource *hw_res = resctrl_to_arch_res(r);
+	struct rdt_hw_domain *hw_dom = resctrl_to_arch_dom(d);
 	struct msr_param m;
 	u32 *dc, *dm;
 
-	dc = kmalloc_array(hw_res->num_closid, sizeof(*d->ctrl_val), GFP_KERNEL);
+	dc = kmalloc_array(hw_res->num_closid, sizeof(*hw_dom->ctrl_val), GFP_KERNEL);
 	if (!dc)
 		return -ENOMEM;
 
-	dm = kmalloc_array(hw_res->num_closid, sizeof(*d->mbps_val), GFP_KERNEL);
+	dm = kmalloc_array(hw_res->num_closid, sizeof(*hw_dom->mbps_val), GFP_KERNEL);
 	if (!dm) {
 		kfree(dc);
 		return -ENOMEM;
 	}
 
-	d->ctrl_val = dc;
-	d->mbps_val = dm;
+	hw_dom->ctrl_val = dc;
+	hw_dom->mbps_val = dm;
 	setup_default_ctrlval(r, dc, dm);
 
 	m.low = 0;
@@ -586,6 +590,7 @@ static void domain_add_cpu(int cpu, struct rdt_resource *r)
 {
 	int id = get_cpu_cacheinfo_id(cpu, r->cache_level);
 	struct list_head *add_pos = NULL;
+	struct rdt_hw_domain *hw_dom;
 	struct rdt_domain *d;
 
 	d = rdt_find_domain(r, id, &add_pos);
@@ -599,10 +604,11 @@ static void domain_add_cpu(int cpu, struct rdt_resource *r)
 		return;
 	}
 
-	d = kzalloc_node(sizeof(*d), GFP_KERNEL, cpu_to_node(cpu));
-	if (!d)
+	hw_dom = kzalloc_node(sizeof(*hw_dom), GFP_KERNEL, cpu_to_node(cpu));
+	if (!hw_dom)
 		return;
 
+	d = &hw_dom->resctrl;
 	d->id = id;
 	cpumask_set_cpu(cpu, &d->cpu_mask);
 
@@ -631,6 +637,7 @@ static void domain_add_cpu(int cpu, struct rdt_resource *r)
 static void domain_remove_cpu(int cpu, struct rdt_resource *r)
 {
 	int id = get_cpu_cacheinfo_id(cpu, r->cache_level);
+	struct rdt_hw_domain *hw_dom;
 	struct rdt_domain *d;
 
 	d = rdt_find_domain(r, id, NULL);
@@ -638,6 +645,7 @@ static void domain_remove_cpu(int cpu, struct rdt_resource *r)
 		pr_warn("Couldn't find cache id for CPU %d\n", cpu);
 		return;
 	}
+	hw_dom = resctrl_to_arch_dom(d);
 
 	cpumask_clear_cpu(cpu, &d->cpu_mask);
 	if (cpumask_empty(&d->cpu_mask)) {
@@ -670,12 +678,12 @@ static void domain_remove_cpu(int cpu, struct rdt_resource *r)
 		if (d->plr)
 			d->plr->d = NULL;
 
-		kfree(d->ctrl_val);
-		kfree(d->mbps_val);
+		kfree(hw_dom->ctrl_val);
+		kfree(hw_dom->mbps_val);
 		bitmap_free(d->rmid_busy_llc);
 		kfree(d->mbm_total);
 		kfree(d->mbm_local);
-		kfree(d);
+		kfree(hw_dom);
 		return;
 	}
 
arch/x86/kernel/cpu/resctrl/ctrlmondata.c +7 -3
index ab6e584c9d2d..2e7466659af3 100644
@@ -238,6 +238,7 @@ static int parse_line(char *line, struct rdt_resource *r,
 
 int update_domains(struct rdt_resource *r, int closid)
 {
+	struct rdt_hw_domain *hw_dom;
 	struct msr_param msr_param;
 	cpumask_var_t cpu_mask;
 	struct rdt_domain *d;
@@ -254,7 +255,8 @@ int update_domains(struct rdt_resource *r, int closid)
 
 	mba_sc = is_mba_sc(r);
 	list_for_each_entry(d, &r->domains, list) {
-		dc = !mba_sc ? d->ctrl_val : d->mbps_val;
+		hw_dom = resctrl_to_arch_dom(d);
+		dc = !mba_sc ? hw_dom->ctrl_val : hw_dom->mbps_val;
 		if (d->have_new_ctrl && d->new_ctrl != dc[closid]) {
 			cpumask_set_cpu(cpumask_any(&d->cpu_mask), cpu_mask);
 			dc[closid] = d->new_ctrl;
@@ -375,17 +377,19 @@ ssize_t rdtgroup_schemata_write(struct kernfs_open_file *of,
 
 static void show_doms(struct seq_file *s, struct rdt_resource *r, int closid)
 {
+	struct rdt_hw_domain *hw_dom;
 	struct rdt_domain *dom;
 	bool sep = false;
 	u32 ctrl_val;
 
 	seq_printf(s, "%*s:", max_name_width, r->name);
 	list_for_each_entry(dom, &r->domains, list) {
+		hw_dom = resctrl_to_arch_dom(dom);
 		if (sep)
 			seq_puts(s, ";");
 
-		ctrl_val = (!is_mba_sc(r) ? dom->ctrl_val[closid] :
-			    dom->mbps_val[closid]);
+		ctrl_val = (!is_mba_sc(r) ? hw_dom->ctrl_val[closid] :
+			    hw_dom->mbps_val[closid]);
 		seq_printf(s, r->format_str, dom->id, max_data_width,
 			   ctrl_val);
 		sep = true;
arch/x86/kernel/cpu/resctrl/internal.h +9 -31
index bcae86138cad..f7aab9245259 100644
@@ -299,44 +299,22 @@ struct mbm_state {
 };
 
 /**
- * struct rdt_domain - group of cpus sharing an RDT resource
- * @list:	all instances of this resource
- * @id:		unique id for this instance
- * @cpu_mask:	which cpus share this resource
- * @rmid_busy_llc:
- *		bitmap of which limbo RMIDs are above threshold
- * @mbm_total:	saved state for MBM total bandwidth
- * @mbm_local:	saved state for MBM local bandwidth
- * @mbm_over:	worker to periodically read MBM h/w counters
- * @cqm_limbo:	worker to periodically read CQM h/w counters
- * @mbm_work_cpu:
- *		worker cpu for MBM h/w counters
- * @cqm_work_cpu:
- *		worker cpu for CQM h/w counters
+ * struct rdt_hw_domain - group of cpus sharing an RDT resource
+ * @resctrl:    Properties exposed to the resctrl file system
  * @ctrl_val:	array of cache or mem ctrl values (indexed by CLOSID)
  * @mbps_val:	When mba_sc is enabled, this holds the bandwidth in MBps
- * @new_ctrl:	new ctrl value to be loaded
- * @have_new_ctrl: did user provide new_ctrl for this domain
- * @plr:	pseudo-locked region (if any) associated with domain
  */
-struct rdt_domain {
-	struct list_head		list;
-	int				id;
-	struct cpumask			cpu_mask;
-	unsigned long			*rmid_busy_llc;
-	struct mbm_state		*mbm_total;
-	struct mbm_state		*mbm_local;
-	struct delayed_work		mbm_over;
-	struct delayed_work		cqm_limbo;
-	int				mbm_work_cpu;
-	int				cqm_work_cpu;
+struct rdt_hw_domain {
+	struct rdt_domain		resctrl;
 	u32				*ctrl_val;
 	u32				*mbps_val;
-	u32				new_ctrl;
-	bool				have_new_ctrl;
-	struct pseudo_lock_region	*plr;
 };
 
+static inline struct rdt_hw_domain *resctrl_to_arch_dom(struct rdt_domain *r)
+{
+	return container_of(r, struct rdt_hw_domain, resctrl);
+}
+
 /**
  * struct msr_param - set a range of MSRs from a domain
  * @res:       The resource to use
arch/x86/kernel/cpu/resctrl/monitor.c +5 -3
index d0afd76d3578..8b7d7ebfcd4b 100644
@@ -355,6 +355,7 @@ static void update_mba_bw(struct rdtgroup *rgrp, struct rdt_domain *dom_mbm)
 	u32 closid, rmid, cur_msr, cur_msr_val, new_msr_val;
 	struct mbm_state *pmbm_data, *cmbm_data;
 	struct rdt_hw_resource *hw_r_mba;
+	struct rdt_hw_domain *hw_dom_mba;
 	u32 cur_bw, delta_bw, user_bw;
 	struct rdt_resource *r_mba;
 	struct rdt_domain *dom_mba;
@@ -375,11 +376,12 @@ static void update_mba_bw(struct rdtgroup *rgrp, struct rdt_domain *dom_mbm)
 		pr_warn_once("Failure to get domain for MBA update\n");
 		return;
 	}
+	hw_dom_mba = resctrl_to_arch_dom(dom_mba);
 
 	cur_bw = pmbm_data->prev_bw;
-	user_bw = dom_mba->mbps_val[closid];
+	user_bw = hw_dom_mba->mbps_val[closid];
 	delta_bw = pmbm_data->delta_bw;
-	cur_msr_val = dom_mba->ctrl_val[closid];
+	cur_msr_val = hw_dom_mba->ctrl_val[closid];
 
 	/*
 	 * For Ctrl groups read data from child monitor groups.
@@ -416,7 +418,7 @@ static void update_mba_bw(struct rdtgroup *rgrp, struct rdt_domain *dom_mbm)
 
 	cur_msr = hw_r_mba->msr_base + closid;
 	wrmsrl(cur_msr, delay_bw_map(new_msr_val, r_mba));
-	dom_mba->ctrl_val[closid] = new_msr_val;
+	hw_dom_mba->ctrl_val[closid] = new_msr_val;
 
 	/*
 	 * Delta values are updated dynamically package wise for each
arch/x86/kernel/cpu/resctrl/rdtgroup.c +19 -10
index e0a35a23b945..b55861ff4e34 100644
@@ -916,7 +916,7 @@ static int rdt_bit_usage_show(struct kernfs_open_file *of,
 	list_for_each_entry(dom, &r->domains, list) {
 		if (sep)
 			seq_putc(seq, ';');
-		ctrl = dom->ctrl_val;
+		ctrl = resctrl_to_arch_dom(dom)->ctrl_val;
 		sw_shareable = 0;
 		exclusive = 0;
 		seq_printf(seq, "%d=", dom->id);
@@ -1195,7 +1195,7 @@ static bool __rdtgroup_cbm_overlaps(struct rdt_resource *r, struct rdt_domain *d
 	}
 
 	/* Check for overlap with other resource groups */
-	ctrl = d->ctrl_val;
+	ctrl = resctrl_to_arch_dom(d)->ctrl_val;
 	for (i = 0; i < closids_supported(); i++, ctrl++) {
 		ctrl_b = *ctrl;
 		mode = rdtgroup_mode_by_closid(i);
@@ -1264,6 +1264,7 @@ bool rdtgroup_cbm_overlaps(struct rdt_resource *r, struct rdt_domain *d,
  */
 static bool rdtgroup_mode_test_exclusive(struct rdtgroup *rdtgrp)
 {
+	struct rdt_hw_domain *hw_dom;
 	int closid = rdtgrp->closid;
 	struct rdt_resource *r;
 	bool has_cache = false;
@@ -1274,7 +1275,8 @@ static bool rdtgroup_mode_test_exclusive(struct rdtgroup *rdtgrp)
 			continue;
 		has_cache = true;
 		list_for_each_entry(d, &r->domains, list) {
-			if (rdtgroup_cbm_overlaps(r, d, d->ctrl_val[closid],
+			hw_dom = resctrl_to_arch_dom(d);
+			if (rdtgroup_cbm_overlaps(r, d, hw_dom->ctrl_val[closid],
 						  rdtgrp->closid, false)) {
 				rdt_last_cmd_puts("Schemata overlaps\n");
 				return false;
@@ -1406,6 +1408,7 @@ unsigned int rdtgroup_cbm_to_size(struct rdt_resource *r,
 static int rdtgroup_size_show(struct kernfs_open_file *of,
 			      struct seq_file *s, void *v)
 {
+	struct rdt_hw_domain *hw_dom;
 	struct rdtgroup *rdtgrp;
 	struct rdt_resource *r;
 	struct rdt_domain *d;
@@ -1440,14 +1443,15 @@ static int rdtgroup_size_show(struct kernfs_open_file *of,
 		sep = false;
 		seq_printf(s, "%*s:", max_name_width, r->name);
 		list_for_each_entry(d, &r->domains, list) {
+			hw_dom = resctrl_to_arch_dom(d);
 			if (sep)
 				seq_putc(s, ';');
 			if (rdtgrp->mode == RDT_MODE_PSEUDO_LOCKSETUP) {
 				size = 0;
 			} else {
 				ctrl = (!is_mba_sc(r) ?
-						d->ctrl_val[rdtgrp->closid] :
-						d->mbps_val[rdtgrp->closid]);
+						hw_dom->ctrl_val[rdtgrp->closid] :
+						hw_dom->mbps_val[rdtgrp->closid]);
 				if (r->rid == RDT_RESOURCE_MBA)
 					size = ctrl;
 				else
@@ -1951,6 +1955,7 @@ void rdt_domain_reconfigure_cdp(struct rdt_resource *r)
 static int set_mba_sc(bool mba_sc)
 {
 	struct rdt_resource *r = &rdt_resources_all[RDT_RESOURCE_MBA].resctrl;
+	struct rdt_hw_domain *hw_dom;
 	struct rdt_domain *d;
 
 	if (!is_mbm_enabled() || !is_mba_linear() ||
@@ -1958,8 +1963,10 @@ static int set_mba_sc(bool mba_sc)
 		return -EINVAL;
 
 	r->membw.mba_sc = mba_sc;
-	list_for_each_entry(d, &r->domains, list)
-		setup_default_ctrlval(r, d->ctrl_val, d->mbps_val);
+	list_for_each_entry(d, &r->domains, list) {
+		hw_dom = resctrl_to_arch_dom(d);
+		setup_default_ctrlval(r, hw_dom->ctrl_val, hw_dom->mbps_val);
+	}
 
 	return 0;
 }
@@ -2279,6 +2286,7 @@ static int rdt_init_fs_context(struct fs_context *fc)
 static int reset_all_ctrls(struct rdt_resource *r)
 {
 	struct rdt_hw_resource *hw_res = resctrl_to_arch_res(r);
+	struct rdt_hw_domain *hw_dom;
 	struct msr_param msr_param;
 	cpumask_var_t cpu_mask;
 	struct rdt_domain *d;
@@ -2297,10 +2305,11 @@ static int reset_all_ctrls(struct rdt_resource *r)
 	 * from each domain to update the MSRs below.
 	 */
 	list_for_each_entry(d, &r->domains, list) {
+		hw_dom = resctrl_to_arch_dom(d);
 		cpumask_set_cpu(cpumask_any(&d->cpu_mask), cpu_mask);
 
 		for (i = 0; i < hw_res->num_closid; i++)
-			d->ctrl_val[i] = r->default_ctrl;
+			hw_dom->ctrl_val[i] = r->default_ctrl;
 	}
 	cpu = get_cpu();
 	/* Update CBM on this cpu if it's in cpu_mask. */
@@ -2688,7 +2697,7 @@ static int __init_one_rdt_domain(struct rdt_domain *d, struct rdt_resource *r,
 	d->have_new_ctrl = false;
 	d->new_ctrl = r->cache.shareable_bits;
 	used_b = r->cache.shareable_bits;
-	ctrl = d->ctrl_val;
+	ctrl = resctrl_to_arch_dom(d)->ctrl_val;
 	for (i = 0; i < closids_supported(); i++, ctrl++) {
 		if (closid_allocated(i) && i != closid) {
 			mode = rdtgroup_mode_by_closid(i);
@@ -2705,7 +2714,7 @@ static int __init_one_rdt_domain(struct rdt_domain *d, struct rdt_resource *r,
 			 * with an exclusive group.
 			 */
 			if (d_cdp)
-				peer_ctl = d_cdp->ctrl_val[i];
+				peer_ctl = resctrl_to_arch_dom(d_cdp)->ctrl_val[i];
 			else
 				peer_ctl = 0;
 			used_b |= *ctrl | peer_ctl;
include/linux/resctrl.h +34 -1
index b2c2b7386d28..f5af59b8f2a9 100644
@@ -15,7 +15,40 @@ int proc_resctrl_show(struct seq_file *m,
 
 #endif
 
-struct rdt_domain;
+/**
+ * struct rdt_domain - group of cpus sharing an RDT resource
+ * @list:		all instances of this resource
+ * @id:			unique id for this instance
+ * @cpu_mask:		which cpus share this resource
+ * @new_ctrl:		new ctrl value to be loaded
+ * @have_new_ctrl:	did user provide new_ctrl for this domain
+ * @rmid_busy_llc:	bitmap of which limbo RMIDs are above threshold
+ * @mbm_total:		saved state for MBM total bandwidth
+ * @mbm_local:		saved state for MBM local bandwidth
+ * @mbm_over:		worker to periodically read MBM h/w counters
+ * @cqm_limbo:		worker to periodically read CQM h/w counters
+ * @mbm_work_cpu:	worker cpu for MBM h/w counters
+ * @cqm_work_cpu:	worker cpu for CQM h/w counters
+ * @plr:		pseudo-locked region (if any) associated with domain
+ */
+struct rdt_domain {
+	struct list_head		list;
+	int				id;
+	struct cpumask			cpu_mask;
+
+	u32				new_ctrl;
+	bool				have_new_ctrl;
+
+	unsigned long			*rmid_busy_llc;
+	struct mbm_state		*mbm_total;
+	struct mbm_state		*mbm_local;
+	struct delayed_work		mbm_over;
+	struct delayed_work		cqm_limbo;
+	int				mbm_work_cpu;
+	int				cqm_work_cpu;
+
+	struct pseudo_lock_region	*plr;
+};
 
 /**
  * struct resctrl_cache - Cache allocation related data
Signature · 1 line
2.28.0
maintainer
#5 lore raw

Re: [PATCH 02/24] x86/resctrl: Split struct rdt_domain

Hi James,

Quoted 3 lines · James Morse

resctrl is the defacto Linux ABI for SoC resource partitioning features. To support it on another architecture, it needs to be abstracted from Intel RDT, and moved it to /fs/.

Same comments as previous patch regarding "Intel RDT" and "moved it to"

Quoted 2 lines

Split struct rdt_domain up too. Move everything that that is particular

s/that that/that/

Quoted 4 lines

to resctrl into a new header file. resctrl code paths touching a 'hw' struct indicates where an abstraction is needed.

No change in behaviour, this patch just moves types around.

Please remove the "this patch" term.

Quoted 11 lines
---
  arch/x86/kernel/cpu/resctrl/core.c        | 32 +++++++++++-------
  arch/x86/kernel/cpu/resctrl/ctrlmondata.c | 10 ++++--
  arch/x86/kernel/cpu/resctrl/internal.h    | 40 +++++------------------
  arch/x86/kernel/cpu/resctrl/monitor.c     |  8 +++--
  arch/x86/kernel/cpu/resctrl/rdtgroup.c    | 29 ++++++++++------
  include/linux/resctrl.h                   | 35 +++++++++++++++++++-
  6 files changed, 94 insertions(+), 60 deletions(-)

...

Quoted 13 lines
arch/x86/kernel/cpu/resctrl/internal.h +0 -5
index bcae86138cad..f7aab9245259 100644
@@ -299,44 +299,22 @@ struct mbm_state {
  };
 
  /**
- * struct rdt_domain - group of cpus sharing an RDT resource
- * @list:	all instances of this resource
- * @id:		unique id for this instance
- * @cpu_mask:	which cpus share this resource
- * @rmid_busy_llc:

arch/x86/kernel/cpu/resctrl/internal.h struct mbm_state { · line 302

Quoted 10 lines
arch/x86/kernel/cpu/resctrl/internal.h +1 -9
- *		bitmap of which limbo RMIDs are above threshold
- * @mbm_total:	saved state for MBM total bandwidth
- * @mbm_local:	saved state for MBM local bandwidth
- * @mbm_over:	worker to periodically read MBM h/w counters
- * @cqm_limbo:	worker to periodically read CQM h/w counters
- * @mbm_work_cpu:
- *		worker cpu for MBM h/w counters
- * @cqm_work_cpu:
- *		worker cpu for CQM h/w counters
+ * struct rdt_hw_domain - group of cpus sharing an RDT resource

s/RDT/resctrl/?

Even so, looking ahead, struct rdt_hw_domain and struct rdt_domain are receiving duplicate descriptions that needs to be adjusted.

Also, CPU is preferred instead of cpu. I understand that in some cases you copy existing usage and I am actually not sure if this would be insisted upon when going to the next level of review. Since these are comments changing it does seem like a good time to use preferred term.

arch/x86/kernel/cpu/resctrl/internal.h struct mbm_state { · line 303

Quoted 1 line
arch/x86/kernel/cpu/resctrl/internal.h +1 -0
+ * @resctrl:    Properties exposed to the resctrl file system

Please use tabs and spaces consistently.

Quoted 34 lines
arch/x86/kernel/cpu/resctrl/internal.h +7 -17
   * @ctrl_val:	array of cache or mem ctrl values (indexed by CLOSID)
   * @mbps_val:	When mba_sc is enabled, this holds the bandwidth in MBps
- * @new_ctrl:	new ctrl value to be loaded
- * @have_new_ctrl: did user provide new_ctrl for this domain
- * @plr:	pseudo-locked region (if any) associated with domain
   */
-struct rdt_domain {
-	struct list_head		list;
-	int				id;
-	struct cpumask			cpu_mask;
-	unsigned long			*rmid_busy_llc;
-	struct mbm_state		*mbm_total;
-	struct mbm_state		*mbm_local;
-	struct delayed_work		mbm_over;
-	struct delayed_work		cqm_limbo;
-	int				mbm_work_cpu;
-	int				cqm_work_cpu;
+struct rdt_hw_domain {
+	struct rdt_domain		resctrl;
  	u32				*ctrl_val;
  	u32				*mbps_val;
-	u32				new_ctrl;
-	bool				have_new_ctrl;
-	struct pseudo_lock_region	*plr;
  };
 
+static inline struct rdt_hw_domain *resctrl_to_arch_dom(struct rdt_domain *r)
+{
+	return container_of(r, struct rdt_hw_domain, resctrl);
+}
+
  /**
   * struct msr_param - set a range of MSRs from a domain
   * @res:       The resource to use

...

include/linux/resctrl.h int proc_resctrl_show(struct seq_file *m, · line 19

Quoted 11 lines
include/linux/resctrl.h +2 -1
index b2c2b7386d28..f5af59b8f2a9 100644
@@ -15,7 +15,40 @@ int proc_resctrl_show(struct seq_file *m,
 
  #endif
 
-struct rdt_domain;
+/**
+ * struct rdt_domain - group of cpus sharing an RDT resource

Duplicate description here (same as rdt_hw_domain). Please use CPUs instead and resctrl instead of RDT.

Quoted 4 lines
include/linux/resctrl.h +4 -0
+ * @list:		all instances of this resource
+ * @id:			unique id for this instance
+ * @cpu_mask:		which cpus share this resource
+ * @new_ctrl:		new ctrl value to be loaded

include/linux/resctrl.h int proc_resctrl_show(struct seq_file *m, · line 33

Quoted 10 lines
include/linux/resctrl.h +10 -0
+ * @have_new_ctrl:	did user provide new_ctrl for this domain
+ * @rmid_busy_llc:	bitmap of which limbo RMIDs are above threshold
+ * @mbm_total:		saved state for MBM total bandwidth
+ * @mbm_local:		saved state for MBM local bandwidth
+ * @mbm_over:		worker to periodically read MBM h/w counters
+ * @cqm_limbo:		worker to periodically read CQM h/w counters
+ * @mbm_work_cpu:	worker cpu for MBM h/w counters
+ * @cqm_work_cpu:	worker cpu for CQM h/w counters
+ * @plr:		pseudo-locked region (if any) associated with domain
+ */

The above usages of cpu is what I considered earlier. I know that the x86 area prefers CPU but I also understand that you are just copying existing comments. I do not know if the x86 maintainers would require a change to CPU at this time but it seems like a good time to transition.

Quoted 9 lines
include/linux/resctrl.h +9 -0
+struct rdt_domain {
+	struct list_head		list;
+	int				id;
+	struct cpumask			cpu_mask;
+
+	u32				new_ctrl;
+	bool				have_new_ctrl;
+
+	unsigned long			*rmid_busy_llc;

include/linux/resctrl.h int proc_resctrl_show(struct seq_file *m, · line 52

Quoted 10 lines
include/linux/resctrl.h +9 -0
+	struct mbm_state		*mbm_total;
+	struct mbm_state		*mbm_local;
+	struct delayed_work		mbm_over;
+	struct delayed_work		cqm_limbo;
+	int				mbm_work_cpu;
+	int				cqm_work_cpu;
+
+	struct pseudo_lock_region	*plr;
+};
 

Please remove the empty lines.

include/linux/resctrl.h int proc_resctrl_show(struct seq_file *m, · line 54

Quoted 3 lines
include/linux/resctrl.h +0 -0
  /**
   * struct resctrl_cache - Cache allocation related data

Reinette

author reviewer
#6 lore raw

PATCH 3/24 x86/resctrl: Add resctrl_arch_get_num_closid()

In reply to James Morse

resctrl chooses whether to enable CDP, once it does, half the number of closid are available. MPAM doesn't behave like this, an in-kernel user of MPAM could be 'using CDP' while resctrl is not.

To move the 'half the closids' behaviour to be part of the core code, each schema would have a num_closids. This may be different from the single resources num_closid if CDP is in use.

Add a helper to read the resource's num_closid, this should return the number of closid that the resource supports, regardless of whether CDP is in use.

For now return the hw_res->num_closid, which is already adjusted for CDP. Once the CODE/DATA/BOTH resources are merged, resctrl can make the adjustment when copying the value to the schema's num_closid.

---
 arch/x86/kernel/cpu/resctrl/core.c        |  5 +++++
 arch/x86/kernel/cpu/resctrl/ctrlmondata.c |  9 +++------
 arch/x86/kernel/cpu/resctrl/rdtgroup.c    | 14 +++++---------
 include/linux/resctrl.h                   |  3 +++
 4 files changed, 16 insertions(+), 15 deletions(-)
arch/x86/kernel/cpu/resctrl/core.c +5 -0
index 97040a54cc9a..5d5b566c4359 100644
@@ -443,6 +443,11 @@ struct rdt_domain *get_domain_from_cpu(int cpu, struct rdt_resource *r)
 	return NULL;
 }
 
+u32 resctrl_arch_get_num_closid(struct rdt_resource *r)
+{
+	return resctrl_to_arch_res(r)->num_closid;
+}
+
 void rdt_ctrl_update(void *arg)
 {
 	struct msr_param *m = arg;
arch/x86/kernel/cpu/resctrl/ctrlmondata.c +3 -6
index 2e7466659af3..14ea6a40993f 100644
@@ -286,12 +286,11 @@ int update_domains(struct rdt_resource *r, int closid)
 static int rdtgroup_parse_resource(char *resname, char *tok,
 				   struct rdtgroup *rdtgrp)
 {
-	struct rdt_hw_resource *hw_res;
 	struct rdt_resource *r;
 
 	for_each_alloc_enabled_rdt_resource(r) {
-		hw_res = resctrl_to_arch_res(r);
-		if (!strcmp(resname, r->name) && rdtgrp->closid < hw_res->num_closid)
+		if (!strcmp(resname, r->name) &&
+		     rdtgrp->closid < resctrl_arch_get_num_closid(r))
 			return parse_line(tok, r, rdtgrp);
 	}
 	rdt_last_cmd_printf("Unknown or unsupported resource name '%s'\n", resname);
@@ -400,7 +399,6 @@ static void show_doms(struct seq_file *s, struct rdt_resource *r, int closid)
 int rdtgroup_schemata_show(struct kernfs_open_file *of,
 			   struct seq_file *s, void *v)
 {
-	struct rdt_hw_resource *hw_res;
 	struct rdtgroup *rdtgrp;
 	struct rdt_resource *r;
 	int ret = 0;
@@ -425,8 +423,7 @@ int rdtgroup_schemata_show(struct kernfs_open_file *of,
 		} else {
 			closid = rdtgrp->closid;
 			for_each_alloc_enabled_rdt_resource(r) {
-				hw_res = resctrl_to_arch_res(r);
-				if (closid < hw_res->num_closid)
+				if (closid < resctrl_arch_get_num_closid(r))
 					show_doms(s, r, closid);
 			}
 		}
arch/x86/kernel/cpu/resctrl/rdtgroup.c +5 -9
index b55861ff4e34..df10135f021e 100644
@@ -100,15 +100,13 @@ int closids_supported(void)
 
 static void closid_init(void)
 {
-	struct rdt_hw_resource *hw_res;
+	u32 rdt_min_closid = 32;
 	struct rdt_resource *r;
-	int rdt_min_closid = 32;
 
 	/* Compute rdt_min_closid across all resources */
-	for_each_alloc_enabled_rdt_resource(r) {
-		hw_res = resctrl_to_arch_res(r);
-		rdt_min_closid = min(rdt_min_closid, hw_res->num_closid);
-	}
+	for_each_alloc_enabled_rdt_resource(r)
+		rdt_min_closid = min(rdt_min_closid,
+				     resctrl_arch_get_num_closid(r));
 
 	closid_free_map = BIT_MASK(rdt_min_closid) - 1;
 
@@ -847,10 +845,8 @@ static int rdt_num_closids_show(struct kernfs_open_file *of,
 				struct seq_file *seq, void *v)
 {
 	struct rdt_resource *r = of->kn->parent->priv;
-	struct rdt_hw_resource *hw_res;
 
-	hw_res = resctrl_to_arch_res(r);
-	seq_printf(seq, "%d\n", hw_res->num_closid);
+	seq_printf(seq, "%d\n", resctrl_arch_get_num_closid(r));
 	return 0;
 }
 
include/linux/resctrl.h +3 -0
index f5af59b8f2a9..dfb0f32b73a1 100644
@@ -163,4 +163,7 @@ struct rdt_resource {
 
 };
 
+/* The number of closid supported by this resource regardless of CDP */
+u32 resctrl_arch_get_num_closid(struct rdt_resource *r);
+
 #endif /* _RESCTRL_H */
Signature · 1 line
2.28.0
maintainer
#7 lore raw

Re: [PATCH 03/24] x86/resctrl: Add resctrl_arch_get_num_closid()

Hi James,

Quoted 29 lines · James Morse

resctrl chooses whether to enable CDP, once it does, half the number of closid are available. MPAM doesn't behave like this, an in-kernel user of MPAM could be 'using CDP' while resctrl is not.

To move the 'half the closids' behaviour to be part of the core code, each schema would have a num_closids. This may be different from the single resources num_closid if CDP is in use.

Add a helper to read the resource's num_closid, this should return the number of closid that the resource supports, regardless of whether CDP is in use.

For now return the hw_res->num_closid, which is already adjusted for CDP. Once the CODE/DATA/BOTH resources are merged, resctrl can make the adjustment when copying the value to the schema's num_closid.

---
  arch/x86/kernel/cpu/resctrl/core.c        |  5 +++++
  arch/x86/kernel/cpu/resctrl/ctrlmondata.c |  9 +++------
  arch/x86/kernel/cpu/resctrl/rdtgroup.c    | 14 +++++---------
  include/linux/resctrl.h                   |  3 +++
  4 files changed, 16 insertions(+), 15 deletions(-)
arch/x86/kernel/cpu/resctrl/core.c +0 -0
index 97040a54cc9a..5d5b566c4359 100644

arch/x86/kernel/cpu/resctrl/core.c struct rdt_domain *get_domain_from_cpu(int cpu, struct rdt_resource *r) · line 450

Quoted 8 lines
arch/x86/kernel/cpu/resctrl/core.c +5 -0
@@ -443,6 +443,11 @@ struct rdt_domain *get_domain_from_cpu(int cpu, struct rdt_resource *r)
  	return NULL;
  }
 
+u32 resctrl_arch_get_num_closid(struct rdt_resource *r)
+{
+	return resctrl_to_arch_res(r)->num_closid;
+}
+

Helper returns the value but also changes the type. Could you please add motivation for this in a comment?

Quoted 9 lines
arch/x86/kernel/cpu/resctrl/core.c +0 -0
  void rdt_ctrl_update(void *arg)
  {
  	struct msr_param *m = arg;
arch/x86/kernel/cpu/resctrl/ctrlmondata.c +0 -0
index 2e7466659af3..14ea6a40993f 100644
@@ -286,12 +286,11 @@ int update_domains(struct rdt_resource *r, int closid)
  static int rdtgroup_parse_resource(char *resname, char *tok,

arch/x86/kernel/cpu/resctrl/ctrlmondata.c int update_domains(struct rdt_resource *r, int closid) · line 293

Quoted 10 lines
arch/x86/kernel/cpu/resctrl/ctrlmondata.c +2 -3
  				   struct rdtgroup *rdtgrp)
  {
-	struct rdt_hw_resource *hw_res;
  	struct rdt_resource *r;
 
  	for_each_alloc_enabled_rdt_resource(r) {
-		hw_res = resctrl_to_arch_res(r);
-		if (!strcmp(resname, r->name) && rdtgrp->closid < hw_res->num_closid)
+		if (!strcmp(resname, r->name) &&
+		     rdtgrp->closid < resctrl_arch_get_num_closid(r))

Please let alignment match open parenthesis.

Quoted 46 lines
arch/x86/kernel/cpu/resctrl/ctrlmondata.c +1 -3
  			return parse_line(tok, r, rdtgrp);
  	}
  	rdt_last_cmd_printf("Unknown or unsupported resource name '%s'\n", resname);
@@ -400,7 +399,6 @@ static void show_doms(struct seq_file *s, struct rdt_resource *r, int closid)
  int rdtgroup_schemata_show(struct kernfs_open_file *of,
  			   struct seq_file *s, void *v)
  {
-	struct rdt_hw_resource *hw_res;
  	struct rdtgroup *rdtgrp;
  	struct rdt_resource *r;
  	int ret = 0;
@@ -425,8 +423,7 @@ int rdtgroup_schemata_show(struct kernfs_open_file *of,
  		} else {
  			closid = rdtgrp->closid;
  			for_each_alloc_enabled_rdt_resource(r) {
-				hw_res = resctrl_to_arch_res(r);
-				if (closid < hw_res->num_closid)
+				if (closid < resctrl_arch_get_num_closid(r))
  					show_doms(s, r, closid);
  			}
  		}
arch/x86/kernel/cpu/resctrl/rdtgroup.c +4 -6
index b55861ff4e34..df10135f021e 100644
@@ -100,15 +100,13 @@ int closids_supported(void)
 
  static void closid_init(void)
  {
-	struct rdt_hw_resource *hw_res;
+	u32 rdt_min_closid = 32;
  	struct rdt_resource *r;
-	int rdt_min_closid = 32;
 
  	/* Compute rdt_min_closid across all resources */
-	for_each_alloc_enabled_rdt_resource(r) {
-		hw_res = resctrl_to_arch_res(r);
-		rdt_min_closid = min(rdt_min_closid, hw_res->num_closid);
-	}
+	for_each_alloc_enabled_rdt_resource(r)
+		rdt_min_closid = min(rdt_min_closid,
+				     resctrl_arch_get_num_closid(r));
 
  	closid_free_map = BIT_MASK(rdt_min_closid) - 1;
 

arch/x86/kernel/cpu/resctrl/rdtgroup.c static int rdt_num_closids_show(struct kernfs_open_file *of, · line 850

Quoted 9 lines
arch/x86/kernel/cpu/resctrl/rdtgroup.c +1 -3
@@ -847,10 +845,8 @@ static int rdt_num_closids_show(struct kernfs_open_file *of,
  				struct seq_file *seq, void *v)
  {
  	struct rdt_resource *r = of->kn->parent->priv;
-	struct rdt_hw_resource *hw_res;
 
-	hw_res = resctrl_to_arch_res(r);
-	seq_printf(seq, "%d\n", hw_res->num_closid);
+	seq_printf(seq, "%d\n", resctrl_arch_get_num_closid(r));
  	return 0;

Now that this type is changed this will need to be %u?

Quoted 8 lines
arch/x86/kernel/cpu/resctrl/rdtgroup.c +0 -0
  }
 
include/linux/resctrl.h +0 -0
index f5af59b8f2a9..dfb0f32b73a1 100644

include/linux/resctrl.h struct rdt_resource { · line 169

Quoted 7 lines
include/linux/resctrl.h +3 -0
@@ -163,4 +163,7 @@ struct rdt_resource {
 
  };
 
+/* The number of closid supported by this resource regardless of CDP */
+u32 resctrl_arch_get_num_closid(struct rdt_resource *r);
+
  #endif /* _RESCTRL_H */

The purpose of this change is unclear and introducing confusion. It introduces a helper that returns the num_closid associated with a resource but it does not use the helper in all the cases where this value is needed. After this change some code uses this new helper while other code continue to access the value directly.

Could you please elaborate in the commit message why this helper is not used everywhere and how the places that were changed were chosen? Seems like the places that _don't_ need the (eventual) value of resctrl_arch_get_num_closid() are changed (so that it is easier to move to the schema's num_closid later) while the places that actually do need the helper are not changed?

Reinette

author reviewer
#8 lore raw

Re: [PATCH 03/24] x86/resctrl: Add resctrl_arch_get_num_closid()

Hi Reinette,

Quoted 18 lines · Reinette Chatre
Quoted 15 lines · James Morse

resctrl chooses whether to enable CDP, once it does, half the number of closid are available. MPAM doesn't behave like this, an in-kernel user of MPAM could be 'using CDP' while resctrl is not.

To move the 'half the closids' behaviour to be part of the core code, each schema would have a num_closids. This may be different from the single resources num_closid if CDP is in use.

Add a helper to read the resource's num_closid, this should return the number of closid that the resource supports, regardless of whether CDP is in use.

For now return the hw_res->num_closid, which is already adjusted for CDP. Once the CODE/DATA/BOTH resources are merged, resctrl can make the adjustment when copying the value to the schema's num_closid.

Quoted 16 lines

arch/x86/kernel/cpu/resctrl/core.c struct rdt_domain *get_domain_from_cpu(int cpu, struct rdt_resource · line 450

Quoted 13 lines
arch/x86/kernel/cpu/resctrl/core.c +4 -0
index 97040a54cc9a..5d5b566c4359 100644
@@ -443,6 +443,11 @@ struct rdt_domain *get_domain_from_cpu(int cpu, struct rdt_resource
 *r)
       return NULL;
   }
   +u32 resctrl_arch_get_num_closid(struct rdt_resource *r)
+{
+    return resctrl_to_arch_res(r)->num_closid;
+}
+

Helper returns the value but also changes the type. Could you please add motivation for this in a comment?

That was just sloppiness. The values from the MPAM driver are unsigned, and this saved casts.

I do think any cross-architecture bits should use types like u32 to avoid nasty surprises. (the value in struct rdtgroup is already unsigned)

I'll do this better, and fix the commit message.

Quoted 28 lines
Quoted 28 lines
arch/x86/kernel/cpu/resctrl/core.c +0 -0
   void rdt_ctrl_update(void *arg)
   {

      struct msr_param *m = arg;

arch/x86/kernel/cpu/resctrl/rdtgroup.c +4 -6
index b55861ff4e34..df10135f021e 100644
@@ -100,15 +100,13 @@ int closids_supported(void)
     static void closid_init(void)
   {
-    struct rdt_hw_resource *hw_res;
+    u32 rdt_min_closid = 32;
       struct rdt_resource *r;
-    int rdt_min_closid = 32;
         /* Compute rdt_min_closid across all resources */
-    for_each_alloc_enabled_rdt_resource(r) {
-        hw_res = resctrl_to_arch_res(r);
-        rdt_min_closid = min(rdt_min_closid, hw_res->num_closid);
-    }
+    for_each_alloc_enabled_rdt_resource(r)
+        rdt_min_closid = min(rdt_min_closid,
+                     resctrl_arch_get_num_closid(r));
         closid_free_map = BIT_MASK(rdt_min_closid) - 1;
   @@ -847,10 +845,8 @@ static int rdt_num_closids_show(struct kernfs_open_file *of,
                   struct seq_file *seq, void *v)
   {
Quoted 8 lines
Quoted 6 lines
arch/x86/kernel/cpu/resctrl/rdtgroup.c +0 -1
       struct rdt_resource *r = of->kn->parent->priv;
-    struct rdt_hw_resource *hw_res;

  -    hw_res = resctrl_to_arch_res(r);
-    seq_printf(seq, "%d\n", hw_res->num_closid);
+    seq_printf(seq, "%d\n", resctrl_arch_get_num_closid(r)); return 0;

Now that this type is changed this will need to be %u?

Oops,

Quoted 8 lines
Quoted 8 lines

  diff --git a/include/linux/resctrl.h b/include/linux/resctrl.h index f5af59b8f2a9..dfb0f32b73a1 100644

include/linux/resctrl.h +0 -0
@@ -163,4 +163,7 @@ struct rdt_resource {
     };

  +/* The number of closid supported by this resource regardless of CDP */ +u32 resctrl_arch_get_num_closid(struct rdt_resource *r);

Quoted 8 lines
Quoted 2 lines

+
  #endif /* _RESCTRL_H */

The purpose of this change is unclear and introducing confusion. It introduces a helper that returns the num_closid associated with a resource but it does not use the helper in all the cases where this value is needed. After this change some code uses this new helper while other code continue to access the value directly.

This is the split between the bits that are fs/resctrl and architecture-specific emerging. Those functions that legitimately use struct rdt_hw_resource directly don't need the helper. e.g. reset_all_ctrls() remains part of the architecture-specific code.

The aim is none of the code in fs/resctrl would ever touch those structs, it would always use the helper.

I agree the commit message is over-focussed on why you can't reach in and grab a value.

Quoted 2 lines

Could you please elaborate in the commit message why this helper is not used everywhere and how the places that were changed were chosen?

These three are obviously part of the filesystem interface:
rdt_num_closids_show()
rdtgroup_parse_resource()
rdtgroup_schemata_show()

closid_init() initialises resctrl's bitmaps, and remains part of the filesystem code.

reset_all_ctrls() is setting up a structure for modifying the hardware, its part of the architecture code, the maximum closid should be the maximum value the hardware has, regardless of what was described to resctrl.

Everything in core.c is architecture specifc.

I'll include this list in the commit message.

Quoted 4 lines

Seems like the places that _don't_ need
the (eventual) value of resctrl_arch_get_num_closid() are changed (so that it is easier to move to the schema's num_closid later) while the places that actually do need the helper are not changed?

I don't follow this. Yes most of these become a read of the num_closid from the schema once the CDP correction moves to be part of the filesystem bits of resctrl. Doing this now removes the arch-specific struct from functions like rdtgroup_parse_resource(), taking one step closer to a distinction between the arch-specific and filesystem parts of the code.

Yes some lines get changed a second time later in the series, but for a different reason. I've juggled the patch order, which should avoid this.

Hopefully the list above will have improved the commit message for v2.

Thanks!

James

author reviewer
#9 lore raw

PATCH 4/24 x86/resctrl: Add a separate schema list for resctrl

In reply to James Morse

To support multiple architectures, the resctrl code needs to be split into a 'fs' specific part in core code, and an arch-specific backend.

It should be difficult for the arch-specific backends to diverge, supporting slightly different ABIs for user-space. For example, generating, parsing and validating the schema configuration values should be done in what becomes the core code to prevent divergence. Today, the schema emerge from which entries in the rdt_resources_all array the arch code has chosen to enable.

Start by creating a struct resctrl_schema, which will eventually hold the name and pending configuration values for resctrl.

---
 arch/x86/kernel/cpu/resctrl/internal.h |  1 +
 arch/x86/kernel/cpu/resctrl/rdtgroup.c | 43 +++++++++++++++++++++++++-
 include/linux/resctrl.h                |  9 ++++++
 3 files changed, 52 insertions(+), 1 deletion(-)
arch/x86/kernel/cpu/resctrl/internal.h +1 -0
index f7aab9245259..682e84aebd14 100644
@@ -106,6 +106,7 @@ extern unsigned int resctrl_cqm_threshold;
 extern bool rdt_alloc_capable;
 extern bool rdt_mon_capable;
 extern unsigned int rdt_mon_features;
+extern struct list_head resctrl_all_schema;
 
 enum rdt_group_type {
 	RDTCTRL_GROUP = 0,
arch/x86/kernel/cpu/resctrl/rdtgroup.c +42 -1

77 lines not on this page · raw message

include/linux/resctrl.h +9 -0
index dfb0f32b73a1..de6cbc725753 100644
@@ -163,6 +163,15 @@ struct rdt_resource {
 
 };
 
+/**
+ * @list:	Member of resctrl's schema list
+ * @res:	The rdt_resource for this entry
+ */
+struct resctrl_schema {
+	struct list_head		list;
+	struct rdt_resource		*res;
+};
+
 /* The number of closid supported by this resource regardless of CDP */
 u32 resctrl_arch_get_num_closid(struct rdt_resource *r);
 
Signature · 1 line
2.28.0
maintainer
#10 lore raw

Re: [PATCH 04/24] x86/resctrl: Add a separate schema list for resctrl

Hi James,

Quoted 10 lines · James Morse

To support multiple architectures, the resctrl code needs to be split into a 'fs' specific part in core code, and an arch-specific backend.

It should be difficult for the arch-specific backends to diverge, supporting slightly different ABIs for user-space. For example, generating, parsing and validating the schema configuration values should be done in what becomes the core code to prevent divergence. Today, the schema emerge from which entries in the rdt_resources_all array the arch code has chosen to enable.

Quoted 2 lines

Start by creating a struct resctrl_schema, which will eventually hold the name and pending configuration values for resctrl.

Looking ahead I am not able to identify the "pending configuration values" that will eventually be held in resctrl_schema. With entire series applied there is the name, type, num_closid, and pointer to the resource.

Quoted 13 lines
---
  arch/x86/kernel/cpu/resctrl/internal.h |  1 +
  arch/x86/kernel/cpu/resctrl/rdtgroup.c | 43 +++++++++++++++++++++++++-
  include/linux/resctrl.h                |  9 ++++++
  3 files changed, 52 insertions(+), 1 deletion(-)
arch/x86/kernel/cpu/resctrl/internal.h +0 -0
index f7aab9245259..682e84aebd14 100644

arch/x86/kernel/cpu/resctrl/internal.h extern unsigned int resctrl_cqm_threshold; · line 109

Quoted 4 lines
arch/x86/kernel/cpu/resctrl/internal.h +1 -0
@@ -106,6 +106,7 @@ extern unsigned int resctrl_cqm_threshold;
  extern bool rdt_alloc_capable;
  extern bool rdt_mon_capable;
  extern unsigned int rdt_mon_features;
+extern struct list_head resctrl_all_schema;

Considering the relationship with rdt_resources_all it seems more consistent to call it resctrl_schema_all?

Quoted 18 lines
arch/x86/kernel/cpu/resctrl/internal.h +0 -0
  enum rdt_group_type {
  	RDTCTRL_GROUP = 0,
arch/x86/kernel/cpu/resctrl/rdtgroup.c +3 -0
index df10135f021e..f79a5e548138 100644
@@ -39,6 +39,9 @@ static struct kernfs_root *rdt_root;
  struct rdtgroup rdtgroup_default;
  LIST_HEAD(rdt_all_groups);
 
+/* list of entries for the schemata file */
+LIST_HEAD(resctrl_all_schema);
+
  /* Kernel fs node for "info" directory under root */
  static struct kernfs_node *kn_info;
 

arch/x86/kernel/cpu/resctrl/rdtgroup.c static int rdt_enable_ctx(struct rdt_fs_context *ctx) · line 2124

Quoted 5 lines
arch/x86/kernel/cpu/resctrl/rdtgroup.c +2 -0
@@ -2117,6 +2120,35 @@ static int rdt_enable_ctx(struct rdt_fs_context *ctx)
  	return ret;
  }
 
+static int create_schemata_list(void)
+{

Could you please use the new namespace (schema_list_) as a prefix instead of suffix? That will make it easier to search.

Quoted 30 lines
arch/x86/kernel/cpu/resctrl/rdtgroup.c +27 -0
+	struct resctrl_schema *s;
+	struct rdt_resource *r;
+
+	for_each_alloc_enabled_rdt_resource(r) {
+		s = kzalloc(sizeof(*s), GFP_KERNEL);
+		if (!s)
+			return -ENOMEM;
+
+		s->res = r;
+
+		INIT_LIST_HEAD(&s->list);
+		list_add(&s->list, &resctrl_all_schema);
+	}
+
+	return 0;
+}
+
+static void destroy_schemata_list(void)
+{
+	struct resctrl_schema *s, *tmp;
+
+	list_for_each_entry_safe(s, tmp, &resctrl_all_schema, list) {
+		list_del(&s->list);
+		kfree(s);
+	}
+}
+
  static int rdt_get_tree(struct fs_context *fc)
  {
  	struct rdt_fs_context *ctx = rdt_fc2context(fc);

...

include/linux/resctrl.h struct rdt_resource { · line 166

Quoted 9 lines
include/linux/resctrl.h +1 -0
index dfb0f32b73a1..de6cbc725753 100644
@@ -163,6 +163,15 @@ struct rdt_resource {
 
  };
 
+/**

Missing a kernel-doc description here

include/linux/resctrl.h struct rdt_resource { · line 177

Quoted 12 lines
include/linux/resctrl.h +8 -0
+ * @list:	Member of resctrl's schema list
+ * @res:	The rdt_resource for this entry
+ */
+struct resctrl_schema {
+	struct list_head		list;
+	struct rdt_resource		*res;
+};
+
  /* The number of closid supported by this resource regardless of CDP */
  u32 resctrl_arch_get_num_closid(struct rdt_resource *r);
 

Reinette

author reviewer
#11 lore raw

Re: [PATCH 04/24] x86/resctrl: Add a separate schema list for resctrl

Hi Reinette

Quoted 11 lines · Reinette Chatre
Quoted 9 lines · James Morse

To support multiple architectures, the resctrl code needs to be split into a 'fs' specific part in core code, and an arch-specific backend.

It should be difficult for the arch-specific backends to diverge, supporting slightly different ABIs for user-space. For example, generating, parsing and validating the schema configuration values should be done in what becomes the core code to prevent divergence. Today, the schema emerge from which entries in the rdt_resources_all array the arch code has chosen to enable.

Quoted 6 lines
Quoted 2 lines

Start by creating a struct resctrl_schema, which will eventually hold the name and pending configuration values for resctrl.

Looking ahead I am not able to identify the "pending configuration values" that will eventually be held in resctrl_schema. With entire series applied there is the name, type, num_closid, and pointer to the resource.

Looks like a vestige from an earlier version:
This referred to the staged configuration in a much earlier version, but this has to stay in the domain structure as its per-domain.

This should refer to the type that is used as an index into the domain's staged configuration array.

Thanks,

James

author reviewer
#12 lore raw

PATCH 5/24 x86/resctrl: Pass the schema in resdir's private pointer

In reply to James Morse

Moving properties that resctrl exposes to user-space into the core 'fs' code, (e.g. the name of the schema), means some of the functions that back the filesystem need the schema struct, but currently take the resource.

Once the CDP resources are merged, the resource doesn't reflect the right level of information.

For the info dirs that represent a control, the information needed is in the schema, as this is how the resource is being used. For the monitors, its the resource as L3CODE_MON doesn't make sense, and would monitor data too.

This difference means the type of the private pointers varies between control and monitor info dirs.

If the flags are RF_MON_INFO, its a struct rdt_resource. If the flags are RF_CTRL_INFO, its a struct resctrl_schema. Nothing in res_common_files[] has both flags.

---
Fake schema for monitors may simplify this if anyone thinks that is preferable.

---
 arch/x86/kernel/cpu/resctrl/rdtgroup.c | 37 +++++++++++++++++---------
 1 file changed, 24 insertions(+), 13 deletions(-)
arch/x86/kernel/cpu/resctrl/rdtgroup.c +24 -13
index f79a5e548138..cb16454a6b0e 100644
@@ -847,7 +847,8 @@ static int rdt_last_cmd_status_show(struct kernfs_open_file *of,
 static int rdt_num_closids_show(struct kernfs_open_file *of,
 				struct seq_file *seq, void *v)
 {
-	struct rdt_resource *r = of->kn->parent->priv;
+	struct resctrl_schema *s = of->kn->parent->priv;
+	struct rdt_resource *r = s->res;
 
 	seq_printf(seq, "%d\n", resctrl_arch_get_num_closid(r));
 	return 0;
@@ -856,7 +857,8 @@ static int rdt_num_closids_show(struct kernfs_open_file *of,
 static int rdt_default_ctrl_show(struct kernfs_open_file *of,
 			     struct seq_file *seq, void *v)
 {
-	struct rdt_resource *r = of->kn->parent->priv;
+	struct resctrl_schema *s = of->kn->parent->priv;
+	struct rdt_resource *r = s->res;
 
 	seq_printf(seq, "%x\n", r->default_ctrl);
 	return 0;
@@ -865,7 +867,8 @@ static int rdt_default_ctrl_show(struct kernfs_open_file *of,
 static int rdt_min_cbm_bits_show(struct kernfs_open_file *of,
 			     struct seq_file *seq, void *v)
 {
-	struct rdt_resource *r = of->kn->parent->priv;
+	struct resctrl_schema *s = of->kn->parent->priv;
+	struct rdt_resource *r = s->res;
 
 	seq_printf(seq, "%u\n", r->cache.min_cbm_bits);
 	return 0;
@@ -874,7 +877,8 @@ static int rdt_min_cbm_bits_show(struct kernfs_open_file *of,
 static int rdt_shareable_bits_show(struct kernfs_open_file *of,
 				   struct seq_file *seq, void *v)
 {
-	struct rdt_resource *r = of->kn->parent->priv;
+	struct resctrl_schema *s = of->kn->parent->priv;
+	struct rdt_resource *r = s->res;
 
 	seq_printf(seq, "%x\n", r->cache.shareable_bits);
 	return 0;
@@ -897,13 +901,14 @@ static int rdt_shareable_bits_show(struct kernfs_open_file *of,
 static int rdt_bit_usage_show(struct kernfs_open_file *of,
 			      struct seq_file *seq, void *v)
 {
-	struct rdt_resource *r = of->kn->parent->priv;
+	struct resctrl_schema *s = of->kn->parent->priv;
 	/*
 	 * Use unsigned long even though only 32 bits are used to ensure
 	 * test_bit() is used safely.
 	 */
 	unsigned long sw_shareable = 0, hw_shareable = 0;
 	unsigned long exclusive = 0, pseudo_locked = 0;
+	struct rdt_resource *r = s->res;
 	struct rdt_domain *dom;
 	int i, hwb, swb, excl, psl;
 	enum rdtgrp_mode mode;
@@ -975,7 +980,8 @@ static int rdt_bit_usage_show(struct kernfs_open_file *of,
 static int rdt_min_bw_show(struct kernfs_open_file *of,
 			     struct seq_file *seq, void *v)
 {
-	struct rdt_resource *r = of->kn->parent->priv;
+	struct resctrl_schema *s = of->kn->parent->priv;
+	struct rdt_resource *r = s->res;
 
 	seq_printf(seq, "%u\n", r->membw.min_bw);
 	return 0;
@@ -1006,7 +1012,8 @@ static int rdt_mon_features_show(struct kernfs_open_file *of,
 static int rdt_bw_gran_show(struct kernfs_open_file *of,
 			     struct seq_file *seq, void *v)
 {
-	struct rdt_resource *r = of->kn->parent->priv;
+	struct resctrl_schema *s = of->kn->parent->priv;
+	struct rdt_resource *r = s->res;
 
 	seq_printf(seq, "%u\n", r->membw.bw_gran);
 	return 0;
@@ -1015,7 +1022,8 @@ static int rdt_bw_gran_show(struct kernfs_open_file *of,
 static int rdt_delay_linear_show(struct kernfs_open_file *of,
 			     struct seq_file *seq, void *v)
 {
-	struct rdt_resource *r = of->kn->parent->priv;
+	struct resctrl_schema *s = of->kn->parent->priv;
+	struct rdt_resource *r = s->res;
 
 	seq_printf(seq, "%u\n", r->membw.delay_linear);
 	return 0;
@@ -1036,7 +1044,8 @@ static int max_threshold_occ_show(struct kernfs_open_file *of,
 static int rdt_thread_throttle_mode_show(struct kernfs_open_file *of,
 					 struct seq_file *seq, void *v)
 {
-	struct rdt_resource *r = of->kn->parent->priv;
+	struct resctrl_schema *s = of->kn->parent->priv;
+	struct rdt_resource *r = s->res;
 
 	if (r->membw.throttle_mode == THREAD_THROTTLE_PER_THREAD)
 		seq_puts(seq, "per-thread\n");
@@ -1769,14 +1778,14 @@ int rdtgroup_kn_mode_restore(struct rdtgroup *r, const char *name,
 	return ret;
 }
 
-static int rdtgroup_mkdir_info_resdir(struct rdt_resource *r, char *name,
+static int rdtgroup_mkdir_info_resdir(void *priv, char *name,
 				      unsigned long fflags)
 {
 	struct kernfs_node *kn_subdir;
 	int ret;
 
 	kn_subdir = kernfs_create_dir(kn_info, name,
-				      kn_info->mode, r);
+				      kn_info->mode, priv);
 	if (IS_ERR(kn_subdir))
 		return PTR_ERR(kn_subdir);
 
@@ -1794,6 +1803,7 @@ static int rdtgroup_mkdir_info_resdir(struct rdt_resource *r, char *name,
 
 static int rdtgroup_create_info_dir(struct kernfs_node *parent_kn)
 {
+	struct resctrl_schema *s;
 	struct rdt_resource *r;
 	unsigned long fflags;
 	char name[32];
@@ -1809,9 +1819,10 @@ static int rdtgroup_create_info_dir(struct kernfs_node *parent_kn)
 	if (ret)
 		goto out_destroy;
 
-	for_each_alloc_enabled_rdt_resource(r) {
+	list_for_each_entry(s, &resctrl_all_schema, list) {
+		r = s->res;
 		fflags =  r->fflags | RF_CTRL_INFO;
-		ret = rdtgroup_mkdir_info_resdir(r, r->name, fflags);
+		ret = rdtgroup_mkdir_info_resdir(s, r->name, fflags);
 		if (ret)
 			goto out_destroy;
 	}
Signature · 1 line
2.28.0
maintainer
#13 lore raw

Re: [PATCH 05/24] x86/resctrl: Pass the schema in resdir's private pointer

Hi James,

It is not clear what "resdir" mentioned in subject line refers to. Could it be changed to "info dir"?

Quoted 4 lines · James Morse

Moving properties that resctrl exposes to user-space into the core 'fs' code, (e.g. the name of the schema), means some of the functions that back the filesystem need the schema struct, but currently take the resource.

I think a simple addition would help to parse the above ...

" ... need the schema struct (to where the properties are moved), ..."

Quoted 29 lines

Once the CDP resources are merged, the resource doesn't reflect the right level of information.

For the info dirs that represent a control, the information needed is in the schema, as this is how the resource is being used. For the monitors, its the resource as L3CODE_MON doesn't make sense, and would monitor data too.

This difference means the type of the private pointers varies between control and monitor info dirs.

If the flags are RF_MON_INFO, its a struct rdt_resource. If the flags are RF_CTRL_INFO, its a struct resctrl_schema. Nothing in res_common_files[] has both flags.

---
Fake schema for monitors may simplify this if anyone thinks that is preferable.

---
  arch/x86/kernel/cpu/resctrl/rdtgroup.c | 37 +++++++++++++++++---------
  1 file changed, 24 insertions(+), 13 deletions(-)
arch/x86/kernel/cpu/resctrl/rdtgroup.c +0 -0
index f79a5e548138..cb16454a6b0e 100644

...

Quoted 12 lines
arch/x86/kernel/cpu/resctrl/rdtgroup.c +1 -0
@@ -1794,6 +1803,7 @@ static int rdtgroup_mkdir_info_resdir(struct rdt_resource *r, char *name,
 
  static int rdtgroup_create_info_dir(struct kernfs_node *parent_kn)
  {
+	struct resctrl_schema *s;
  	struct rdt_resource *r;
  	unsigned long fflags;
  	char name[32];
@@ -1809,9 +1819,10 @@ static int rdtgroup_create_info_dir(struct kernfs_node *parent_kn)
  	if (ret)
  		goto out_destroy;

arch/x86/kernel/cpu/resctrl/rdtgroup.c static int rdtgroup_create_info_dir(struct kernfs_node *parent_kn) · line 1828

Quoted 10 lines
arch/x86/kernel/cpu/resctrl/rdtgroup.c +3 -2
 
-	for_each_alloc_enabled_rdt_resource(r) {
+	list_for_each_entry(s, &resctrl_all_schema, list) {
+		r = s->res;
  		fflags =  r->fflags | RF_CTRL_INFO;
-		ret = rdtgroup_mkdir_info_resdir(r, r->name, fflags);
+		ret = rdtgroup_mkdir_info_resdir(s, r->name, fflags);
  		if (ret)
  			goto out_destroy;
  	}

I think it would be helpful to add a comment here to compensate for the symmetry that is removed ("for_each_alloc_enabled_rdt_resource()" followed by a "for_each_mon_enabled_rdt_resource()").

Reinette

author reviewer
#14 lore raw

Re: [PATCH 05/24] x86/resctrl: Pass the schema in resdir's private pointer

Hi Reinette,

Quoted 1 line · Reinette Chatre

It is not clear what "resdir" mentioned in subject line refers to.

rdtgroup_mkdir_info_resdir(), it looks I picked the wrong bit to identify it. ('info' in a name usually conveys no information at all!)

Quoted 1 line

Could it be changed to "info dir"?

Sure,

Quoted 42 lines · James Morse
Quoted 4 lines

Moving properties that resctrl exposes to user-space into the core 'fs' code, (e.g. the name of the schema), means some of the functions that back the filesystem need the schema struct, but currently take the resource.

I think a simple addition would help to parse the above ...

" ... need the schema struct (to where the properties are moved), ..."

Quoted 32 lines

Once the CDP resources are merged, the resource doesn't reflect the right level of information.

For the info dirs that represent a control, the information needed is in the schema, as this is how the resource is being used. For the monitors, its the resource as L3CODE_MON doesn't make sense, and would monitor data too.

This difference means the type of the private pointers varies between control and monitor info dirs.

If the flags are RF_MON_INFO, its a struct rdt_resource. If the flags are RF_CTRL_INFO, its a struct resctrl_schema. Nothing in res_common_files[] has both flags.

arch/x86/kernel/cpu/resctrl/rdtgroup.c +3 -1
@@ -1794,6 +1803,7 @@ static int rdtgroup_mkdir_info_resdir(struct rdt_resource *r, char
 *name,
     static int rdtgroup_create_info_dir(struct kernfs_node *parent_kn)
   {
+    struct resctrl_schema *s;
       struct rdt_resource *r;
       unsigned long fflags;
       char name[32];
@@ -1809,9 +1819,10 @@ static int rdtgroup_create_info_dir(struct kernfs_node *parent_kn)
       if (ret)
           goto out_destroy;
   -    for_each_alloc_enabled_rdt_resource(r) {
+    list_for_each_entry(s, &resctrl_all_schema, list) {
+        r = s->res;
           fflags =  r->fflags | RF_CTRL_INFO;
-        ret = rdtgroup_mkdir_info_resdir(r, r->name, fflags);
Quoted 8 lines

arch/x86/kernel/cpu/resctrl/rdtgroup.c static int rdtgroup_create_info_dir(struct kernfs_node *parent_kn) · line 1828

Quoted 4 lines
arch/x86/kernel/cpu/resctrl/rdtgroup.c +1 -0
+        ret = rdtgroup_mkdir_info_resdir(s, r->name, fflags);
           if (ret)
               goto out_destroy;
       }

I think it would be helpful to add a comment here to compensate for the symmetry that is removed ("for_each_alloc_enabled_rdt_resource()" followed by a "for_each_mon_enabled_rdt_resource()").

Sure, the thing to convey is the first loop is for 'alloc_enabled' controls.

Thanks,

James

author reviewer
#15 lore raw

PATCH 6/24 x86/resctrl: Store the effective num_closid in the schema

In reply to James Morse

resctrl_schema holds properties that vary with the style of configuration that resctrl applies to a resource.

Once the arch code has a single resource per cache that can be configured, resctrl will need to keep track of the num_closid itself.

Add num_closid to resctrl_schema. Change callers like rdtgroup_schemata_show() to walk the schema instead.

---
 arch/x86/kernel/cpu/resctrl/ctrlmondata.c | 13 ++++++++-----
 arch/x86/kernel/cpu/resctrl/rdtgroup.c    | 11 +++++------
 include/linux/resctrl.h                   |  2 ++
 3 files changed, 15 insertions(+), 11 deletions(-)
arch/x86/kernel/cpu/resctrl/ctrlmondata.c +8 -5
index 14ea6a40993f..8ac104c634fe 100644
@@ -286,11 +286,12 @@ int update_domains(struct rdt_resource *r, int closid)
 static int rdtgroup_parse_resource(char *resname, char *tok,
 				   struct rdtgroup *rdtgrp)
 {
+	struct resctrl_schema *s;
 	struct rdt_resource *r;
 
-	for_each_alloc_enabled_rdt_resource(r) {
-		if (!strcmp(resname, r->name) &&
-		     rdtgrp->closid < resctrl_arch_get_num_closid(r))
+	list_for_each_entry(s, &resctrl_all_schema, list) {
+		r = s->res;
+		if (!strcmp(resname, r->name) && rdtgrp->closid < s->num_closid)
 			return parse_line(tok, r, rdtgrp);
 	}
 	rdt_last_cmd_printf("Unknown or unsupported resource name '%s'\n", resname);
@@ -399,6 +400,7 @@ static void show_doms(struct seq_file *s, struct rdt_resource *r, int closid)
 int rdtgroup_schemata_show(struct kernfs_open_file *of,
 			   struct seq_file *s, void *v)
 {
+	struct resctrl_schema *schema;
 	struct rdtgroup *rdtgrp;
 	struct rdt_resource *r;
 	int ret = 0;
@@ -422,8 +424,9 @@ int rdtgroup_schemata_show(struct kernfs_open_file *of,
 			}
 		} else {
 			closid = rdtgrp->closid;
-			for_each_alloc_enabled_rdt_resource(r) {
-				if (closid < resctrl_arch_get_num_closid(r))
+			list_for_each_entry(schema, &resctrl_all_schema, list) {
+				r = schema->res;
+				if (closid < schema->num_closid)
 					show_doms(s, r, closid);
 			}
 		}
arch/x86/kernel/cpu/resctrl/rdtgroup.c +5 -6
index cb16454a6b0e..1bd785b1920c 100644
@@ -103,13 +103,12 @@ int closids_supported(void)
 
 static void closid_init(void)
 {
+	struct resctrl_schema *s;
 	u32 rdt_min_closid = 32;
-	struct rdt_resource *r;
 
 	/* Compute rdt_min_closid across all resources */
-	for_each_alloc_enabled_rdt_resource(r)
-		rdt_min_closid = min(rdt_min_closid,
-				     resctrl_arch_get_num_closid(r));
+	list_for_each_entry(s, &resctrl_all_schema, list)
+		rdt_min_closid = min(rdt_min_closid, s->num_closid);
 
 	closid_free_map = BIT_MASK(rdt_min_closid) - 1;
 
@@ -848,9 +847,8 @@ static int rdt_num_closids_show(struct kernfs_open_file *of,
 				struct seq_file *seq, void *v)
 {
 	struct resctrl_schema *s = of->kn->parent->priv;
-	struct rdt_resource *r = s->res;
 
-	seq_printf(seq, "%d\n", resctrl_arch_get_num_closid(r));
+	seq_printf(seq, "%d\n", s->num_closid);
 	return 0;
 }
 
@@ -2142,6 +2140,7 @@ static int create_schemata_list(void)
 			return -ENOMEM;
 
 		s->res = r;
+		s->num_closid = resctrl_arch_get_num_closid(r);
 
 		INIT_LIST_HEAD(&s->list);
 		list_add(&s->list, &resctrl_all_schema);
include/linux/resctrl.h +2 -0
index de6cbc725753..b32152968bca 100644
@@ -166,10 +166,12 @@ struct rdt_resource {
 /**
  * @list:	Member of resctrl's schema list
  * @res:	The rdt_resource for this entry
+ * @num_closid	Number of CLOSIDs available for this resource
  */
 struct resctrl_schema {
 	struct list_head		list;
 	struct rdt_resource		*res;
+	u32				num_closid;
 };
 
 /* The number of closid supported by this resource regardless of CDP */
Signature · 1 line
2.28.0
maintainer
#16 lore raw

Re: [PATCH 06/24] x86/resctrl: Store the effective num_closid in the schema

Hi James,

Quoted 8 lines · James Morse

resctrl_schema holds properties that vary with the style of configuration that resctrl applies to a resource.

Once the arch code has a single resource per cache that can be configured, resctrl will need to keep track of the num_closid itself.

Add num_closid to resctrl_schema. Change callers like rdtgroup_schemata_show() to walk the schema instead.

This is a significant patch in that it introduces a second num_closid available for code to use. Even so, the commit message is treating it quite nonchalantly ... essentially stating that "here is a new closid and change some code to use it".

Could you please elaborate how the callers needing to "walk the schema instead" were chosen?

This seems almost a revert of the earlier patch that introduced the helper and I wonder if it may not make this easier to understand if these areas do not receive the temporary change to use that helper.

Quoted 8 lines
---
  arch/x86/kernel/cpu/resctrl/ctrlmondata.c | 13 ++++++++-----
  arch/x86/kernel/cpu/resctrl/rdtgroup.c    | 11 +++++------
  include/linux/resctrl.h                   |  2 ++
  3 files changed, 15 insertions(+), 11 deletions(-)

...

include/linux/resctrl.h struct rdt_resource { · line 169

Quoted 9 lines
include/linux/resctrl.h +1 -0
index de6cbc725753..b32152968bca 100644
@@ -166,10 +166,12 @@ struct rdt_resource {
  /**
   * @list:	Member of resctrl's schema list
   * @res:	The rdt_resource for this entry
+ * @num_closid	Number of CLOSIDs available for this resource

Missing a ":"?

include/linux/resctrl.h struct rdt_resource { · line 177

Quoted 9 lines
include/linux/resctrl.h +1 -0
   */
  struct resctrl_schema {
  	struct list_head		list;
  	struct rdt_resource		*res;
+	u32				num_closid;
  };
 
  /* The number of closid supported by this resource regardless of CDP */

Reinette

author reviewer
#17 lore raw

Re: [PATCH 06/24] x86/resctrl: Store the effective num_closid in the schema

Hi Reinette,

Quoted 7 lines · Reinette Chatre
Quoted 6 lines · James Morse

resctrl_schema holds properties that vary with the style of configuration that resctrl applies to a resource.

Once the arch code has a single resource per cache that can be configured, resctrl will need to keep track of the num_closid itself.

Quoted 6 lines
Quoted 2 lines

Add num_closid to resctrl_schema. Change callers like rdtgroup_schemata_show() to walk the schema instead.

This is a significant patch in that it introduces a second num_closid available for code to use. Even so, the commit message is treating it quite nonchalantly ... essentially stating that "here is a new closid and change some code to use it".

The difference already exists, the number of closid that resctrl exposes to user-space may be different from what the hardware supports. Currently the arch code does a bait-and-switch with the L3CODE/L3DATA resources, but this is specific to x86, it shouldn't be required for another architecture to copy it if its not necessary to support CDP.

I'll expand the commit message with this.

An earlier version tried to use different types for the 'hw' number of closid, and the version used by resctrl, but I figured it was too noisy.

Quoted 1 line

Could you please elaborate how the callers needing to "walk the schema instead" were chosen?

These all want the num_closid value that is exposed to user-space:
rdtgroup_parse_resource()
rdtgroup_schemata_show()
rdt_num_closids_show()
closid_init()

If resctrl is in control of that, it should come from the schema instead of being pulled straight out of the architecture code.

Quoted 3 lines

This seems almost a revert of the earlier patch that introduced the helper and I wonder if it may not make this easier to understand if these areas do not receive the temporary change to use that helper.

Its a trade-off between churn for a self contained change (i.e. all the 'fs' bits, regardless of if they are around later), and keeping the patches that are to do with the schema together.

I don't think its a good idea to have "these are left alone as they will be changed differently later" as that is liable to break (or just get weird) as the series is restructured to fix it.

It will probably be fewer lines of change to do it the other way round. v2 does this, hopefully that is easier on the eye!

Thanks,

James

author reviewer
#18 lore raw

PATCH 7/24 x86/resctrl: Label the resources with their configuration type

In reply to James Morse

Before the name for the schema can be generated, the type of the configuration being applied to the resource needs to be known. Label all the entries in rdt_resources_all[], and copy that value in to struct resctrl_schema.

Subsequent patches will generate the schema names in what will become the fs code. Eventually the fs code will generate pairs of CODE/DATA if the platform supports CDP for this resource.

---
 arch/x86/kernel/cpu/resctrl/core.c     | 7 +++++++
 arch/x86/kernel/cpu/resctrl/internal.h | 1 +
 arch/x86/kernel/cpu/resctrl/rdtgroup.c | 1 +
 include/linux/resctrl.h                | 8 ++++++++
 4 files changed, 17 insertions(+)
arch/x86/kernel/cpu/resctrl/core.c +7 -0
index 5d5b566c4359..1ed5e04031e6 100644
@@ -62,6 +62,7 @@ mba_wrmsr_amd(struct rdt_domain *d, struct msr_param *m,
 struct rdt_hw_resource rdt_resources_all[] = {
 	[RDT_RESOURCE_L3] =
 	{
+		.conf_type			= CDP_BOTH,
 		.resctrl = {
 			.rid			= RDT_RESOURCE_L3,
 			.name			= "L3",
@@ -81,6 +82,7 @@ struct rdt_hw_resource rdt_resources_all[] = {
 	},
 	[RDT_RESOURCE_L3DATA] =
 	{
+		.conf_type			= CDP_DATA,
 		.resctrl = {
 			.rid			= RDT_RESOURCE_L3DATA,
 			.name			= "L3DATA",
@@ -100,6 +102,7 @@ struct rdt_hw_resource rdt_resources_all[] = {
 	},
 	[RDT_RESOURCE_L3CODE] =
 	{
+		.conf_type			= CDP_CODE,
 		.resctrl = {
 			.rid			= RDT_RESOURCE_L3CODE,
 			.name			= "L3CODE",
@@ -119,6 +122,7 @@ struct rdt_hw_resource rdt_resources_all[] = {
 	},
 	[RDT_RESOURCE_L2] =
 	{
+		.conf_type			= CDP_BOTH,
 		.resctrl = {
 			.rid			= RDT_RESOURCE_L2,
 			.name			= "L2",
@@ -138,6 +142,7 @@ struct rdt_hw_resource rdt_resources_all[] = {
 	},
 	[RDT_RESOURCE_L2DATA] =
 	{
+		.conf_type			= CDP_DATA,
 		.resctrl = {
 			.rid			= RDT_RESOURCE_L2DATA,
 			.name			= "L2DATA",
@@ -157,6 +162,7 @@ struct rdt_hw_resource rdt_resources_all[] = {
 	},
 	[RDT_RESOURCE_L2CODE] =
 	{
+		.conf_type			= CDP_CODE,
 		.resctrl = {
 			.rid			= RDT_RESOURCE_L2CODE,
 			.name			= "L2CODE",
@@ -176,6 +182,7 @@ struct rdt_hw_resource rdt_resources_all[] = {
 	},
 	[RDT_RESOURCE_MBA] =
 	{
+		.conf_type			= CDP_BOTH,
 		.resctrl = {
 			.rid			= RDT_RESOURCE_MBA,
 			.name			= "MB",
arch/x86/kernel/cpu/resctrl/internal.h +1 -0
index 682e84aebd14..6c87a81946b1 100644
@@ -367,6 +367,7 @@ struct rdt_parse_data {
  * @mon_scale:		cqm counter * mon_scale = occupancy in bytes
  */
 struct rdt_hw_resource {
+	enum resctrl_conf_type	conf_type;
 	struct rdt_resource     resctrl;
 
 	int			num_closid;
arch/x86/kernel/cpu/resctrl/rdtgroup.c +1 -0
index 1bd785b1920c..628e5eb4d7a9 100644
@@ -2141,6 +2141,7 @@ static int create_schemata_list(void)
 
 		s->res = r;
 		s->num_closid = resctrl_arch_get_num_closid(r);
+		s->conf_type = resctrl_to_arch_res(r)->conf_type;
 
 		INIT_LIST_HEAD(&s->list);
 		list_add(&s->list, &resctrl_all_schema);
include/linux/resctrl.h +8 -0
index b32152968bca..20d8b6dd4af4 100644
@@ -15,6 +15,12 @@ int proc_resctrl_show(struct seq_file *m,
 
 #endif
 
+enum resctrl_conf_type {
+	CDP_BOTH,
+	CDP_CODE,
+	CDP_DATA,
+};
+
 /**
  * struct rdt_domain - group of cpus sharing an RDT resource
  * @list:		all instances of this resource
@@ -165,11 +171,13 @@ struct rdt_resource {
 
 /**
  * @list:	Member of resctrl's schema list
+ * @cdp_type:	Whether this entry is for code/data/both
  * @res:	The rdt_resource for this entry
  * @num_closid	Number of CLOSIDs available for this resource
  */
 struct resctrl_schema {
 	struct list_head		list;
+	enum resctrl_conf_type		conf_type;
 	struct rdt_resource		*res;
 	u32				num_closid;
 };
Signature · 1 line
2.28.0
maintainer
#19 lore raw

Re: [PATCH 07/24] x86/resctrl: Label the resources with their configuration type

Hi James,

Quoted 3 lines · James Morse

Before the name for the schema can be generated, the type of the configuration being applied to the resource needs to be known. Label all the entries in rdt_resources_all[], and copy that value in to struct

s/in to/into/ or s/in to/to/ ?

Quoted 2 lines

resctrl_schema.

This commit message does not explain why it is needed to copy this value.

Quoted 3 lines

Subsequent patches will generate the schema names in what will become the fs code. Eventually the fs code will generate pairs of CODE/DATA if the platform supports CDP for this resource.

Explaining how the copy is a step towards accomplishing this goal would be very helpful.

Quoted 16 lines
---
  arch/x86/kernel/cpu/resctrl/core.c     | 7 +++++++
  arch/x86/kernel/cpu/resctrl/internal.h | 1 +
  arch/x86/kernel/cpu/resctrl/rdtgroup.c | 1 +
  include/linux/resctrl.h                | 8 ++++++++
  4 files changed, 17 insertions(+)
arch/x86/kernel/cpu/resctrl/internal.h +0 -0
index 682e84aebd14..6c87a81946b1 100644

arch/x86/kernel/cpu/resctrl/internal.h struct rdt_parse_data { · line 372

Quoted 6 lines
arch/x86/kernel/cpu/resctrl/internal.h +1 -0
@@ -367,6 +367,7 @@ struct rdt_parse_data {
   * @mon_scale:		cqm counter * mon_scale = occupancy in bytes
   */
  struct rdt_hw_resource {
+	enum resctrl_conf_type	conf_type;
  	struct rdt_resource     resctrl;
 

Missing an accompanying kernel-doc entry?

arch/x86/kernel/cpu/resctrl/rdtgroup.c static int create_schemata_list(void) · line 2143

Quoted 9 lines
arch/x86/kernel/cpu/resctrl/internal.h +0 -0
  	int			num_closid;
arch/x86/kernel/cpu/resctrl/rdtgroup.c +0 -0
index 1bd785b1920c..628e5eb4d7a9 100644
@@ -2141,6 +2141,7 @@ static int create_schemata_list(void)
 
  		s->res = r;
  		s->num_closid = resctrl_arch_get_num_closid(r);

Above seems to be last user of this helper remaining ... why is helper needed instead of something similar to below?

Quoted 22 lines
arch/x86/kernel/cpu/resctrl/rdtgroup.c +1 -0
+		s->conf_type = resctrl_to_arch_res(r)->conf_type;
 
  		INIT_LIST_HEAD(&s->list);
  		list_add(&s->list, &resctrl_all_schema);
include/linux/resctrl.h +6 -0
index b32152968bca..20d8b6dd4af4 100644
@@ -15,6 +15,12 @@ int proc_resctrl_show(struct seq_file *m,
 
  #endif
 
+enum resctrl_conf_type {
+	CDP_BOTH,
+	CDP_CODE,
+	CDP_DATA,
+};
+
  /**
   * struct rdt_domain - group of cpus sharing an RDT resource
   * @list:		all instances of this resource

include/linux/resctrl.h struct rdt_resource { · line 174

Quoted 4 lines
include/linux/resctrl.h +1 -0
@@ -165,11 +171,13 @@ struct rdt_resource {
 
  /**
   * @list:	Member of resctrl's schema list
+ * @cdp_type:	Whether this entry is for code/data/both

Typo? This is fixed in the next patch so that hunk could be merged here.

include/linux/resctrl.h struct rdt_resource { · line 183

Quoted 10 lines
include/linux/resctrl.h +1 -0
   * @res:	The rdt_resource for this entry
   * @num_closid	Number of CLOSIDs available for this resource
   */
  struct resctrl_schema {
  	struct list_head		list;
+	enum resctrl_conf_type		conf_type;
  	struct rdt_resource		*res;
  	u32				num_closid;
  };

Reinette

author reviewer
#20 lore raw

Re: [PATCH 07/24] x86/resctrl: Label the resources with their configuration type

Hi Reinette,

Quoted 10 lines · Reinette Chatre
Quoted 3 lines · James Morse

Before the name for the schema can be generated, the type of the configuration being applied to the resource needs to be known. Label all the entries in rdt_resources_all[], and copy that value in to struct

s/in to/into/ or s/in to/to/ ?

Quoted 2 lines

resctrl_schema.

Quoted 7 lines

This commit message does not explain why it is needed to copy this value.

Quoted 3 lines

Subsequent patches will generate the schema names in what will become the fs code. Eventually the fs code will generate pairs of CODE/DATA if the platform supports CDP for this resource.

Explaining how the copy is a step towards accomplishing this goal would be very helpful.

(I've added text about what this is used for, and why it can't assign the values it wants at this point in the series)

Quoted 11 lines

arch/x86/kernel/cpu/resctrl/rdtgroup.c static int create_schemata_list(void) · line 2142

Quoted 8 lines
arch/x86/kernel/cpu/resctrl/rdtgroup.c +0 -0
index 1bd785b1920c..628e5eb4d7a9 100644
@@ -2141,6 +2141,7 @@ static int create_schemata_list(void)
             s->res = r;
           s->num_closid = resctrl_arch_get_num_closid(r);

Above seems to be last user of this helper remaining ... why is helper needed instead of something similar to below?

Great question, resctrl_to_arch_res(r)->num_closid? That won't work for MPAM, or would at least force all architectures to copy x86's arch-specific structure.

schemata_list_create() needs to be part of the filesystem code after the split, but it can't touch the hw structure like the conf_type is doing here.

This is mentioned in the commit message of the first two patches:
| resctrl code paths touching a 'hw' struct indicates where an abstraction is needed.

I evidently need to make that clearer.

Quoted 1 line

arch/x86/kernel/cpu/resctrl/rdtgroup.c static int create_schemata_list(void) · line 2144

Quoted 1 line
arch/x86/kernel/cpu/resctrl/rdtgroup.c +1 -0
+        s->conf_type = resctrl_to_arch_res(r)->conf_type;

I'll to do this temporarily, as by then end of the series schemata_list_create() chooses the value, so this disappears.

Thanks,

James

author reviewer
#21 lore raw

PATCH 8/24 x86/resctrl: Walk the resctrl schema list instead of an arch list

In reply to James Morse

Now that resctrl has its own list of resources it is using, walk that list instead of the architectures list. This means resctrl has somewhere to keep schema properties with the resource that is using them.

Most users of for_each_alloc_enabled_rdt_resource() are per-schema, and also want a schema property, like the conf_type. Switch these to walk the schema list. Schema were only created for alloc_enabled resources so these two lists are currently equivalent.

---
 arch/x86/kernel/cpu/resctrl/ctrlmondata.c | 38 ++++++++++++++---------
 arch/x86/kernel/cpu/resctrl/internal.h    |  6 ++--
 arch/x86/kernel/cpu/resctrl/rdtgroup.c    | 34 +++++++++++++-------
 include/linux/resctrl.h                   |  5 +--
 4 files changed, 53 insertions(+), 30 deletions(-)
arch/x86/kernel/cpu/resctrl/ctrlmondata.c +24 -14
index 8ac104c634fe..d3f9d142f58a 100644
@@ -57,9 +57,10 @@ static bool bw_validate(char *buf, unsigned long *data, struct rdt_resource *r)
 	return true;
 }
 
-int parse_bw(struct rdt_parse_data *data, struct rdt_resource *r,
+int parse_bw(struct rdt_parse_data *data, struct resctrl_schema *s,
 	     struct rdt_domain *d)
 {
+	struct rdt_resource *r = s->res;
 	unsigned long bw_val;
 
 	if (d->have_new_ctrl) {
@@ -125,10 +126,11 @@ static bool cbm_validate(char *buf, u32 *data, struct rdt_resource *r)
  * Read one cache bit mask (hex). Check that it is valid for the current
  * resource type.
  */
-int parse_cbm(struct rdt_parse_data *data, struct rdt_resource *r,
+int parse_cbm(struct rdt_parse_data *data, struct resctrl_schema *s,
 	      struct rdt_domain *d)
 {
 	struct rdtgroup *rdtgrp = data->rdtgrp;
+	struct rdt_resource *r = s->res;
 	u32 cbm_val;
 
 	if (d->have_new_ctrl) {
@@ -160,12 +162,12 @@ int parse_cbm(struct rdt_parse_data *data, struct rdt_resource *r,
 	 * The CBM may not overlap with the CBM of another closid if
 	 * either is exclusive.
 	 */
-	if (rdtgroup_cbm_overlaps(r, d, cbm_val, rdtgrp->closid, true)) {
+	if (rdtgroup_cbm_overlaps(s, d, cbm_val, rdtgrp->closid, true)) {
 		rdt_last_cmd_puts("Overlaps with exclusive group\n");
 		return -EINVAL;
 	}
 
-	if (rdtgroup_cbm_overlaps(r, d, cbm_val, rdtgrp->closid, false)) {
+	if (rdtgroup_cbm_overlaps(s, d, cbm_val, rdtgrp->closid, false)) {
 		if (rdtgrp->mode == RDT_MODE_EXCLUSIVE ||
 		    rdtgrp->mode == RDT_MODE_PSEUDO_LOCKSETUP) {
 			rdt_last_cmd_puts("Overlaps with other group\n");
@@ -185,9 +187,10 @@ int parse_cbm(struct rdt_parse_data *data, struct rdt_resource *r,
  * separated by ";". The "id" is in decimal, and must match one of
  * the "id"s for this resource.
  */
-static int parse_line(char *line, struct rdt_resource *r,
+static int parse_line(char *line, struct resctrl_schema *s,
 		      struct rdtgroup *rdtgrp)
 {
+	struct rdt_resource *r = s->res;
 	struct rdt_parse_data data;
 	char *dom = NULL, *id;
 	struct rdt_domain *d;
@@ -213,7 +216,8 @@ static int parse_line(char *line, struct rdt_resource *r,
 		if (d->id == dom_id) {
 			data.buf = dom;
 			data.rdtgrp = rdtgrp;
-			if (r->parse_ctrlval(&data, r, d))
+
+			if (r->parse_ctrlval(&data, s, d))
 				return -EINVAL;
 			if (rdtgrp->mode ==  RDT_MODE_PSEUDO_LOCKSETUP) {
 				/*
@@ -289,10 +293,12 @@ static int rdtgroup_parse_resource(char *resname, char *tok,
 	struct resctrl_schema *s;
 	struct rdt_resource *r;
 
+	lockdep_assert_held(&rdtgroup_mutex);
+
 	list_for_each_entry(s, &resctrl_all_schema, list) {
 		r = s->res;
 		if (!strcmp(resname, r->name) && rdtgrp->closid < s->num_closid)
-			return parse_line(tok, r, rdtgrp);
+			return parse_line(tok, s, rdtgrp);
 	}
 	rdt_last_cmd_printf("Unknown or unsupported resource name '%s'\n", resname);
 	return -EINVAL;
@@ -301,6 +307,7 @@ static int rdtgroup_parse_resource(char *resname, char *tok,
 ssize_t rdtgroup_schemata_write(struct kernfs_open_file *of,
 				char *buf, size_t nbytes, loff_t off)
 {
+	struct resctrl_schema *s;
 	struct rdtgroup *rdtgrp;
 	struct rdt_domain *dom;
 	struct rdt_resource *r;
@@ -331,8 +338,8 @@ ssize_t rdtgroup_schemata_write(struct kernfs_open_file *of,
 		goto out;
 	}
 
-	for_each_alloc_enabled_rdt_resource(r) {
-		list_for_each_entry(dom, &r->domains, list)
+	list_for_each_entry(s, &resctrl_all_schema, list) {
+		list_for_each_entry(dom, &s->res->domains, list)
 			dom->have_new_ctrl = false;
 	}
 
@@ -353,7 +360,8 @@ ssize_t rdtgroup_schemata_write(struct kernfs_open_file *of,
 			goto out;
 	}
 
-	for_each_alloc_enabled_rdt_resource(r) {
+	list_for_each_entry(s, &resctrl_all_schema, list) {
+		r = s->res;
 		ret = update_domains(r, rdtgrp->closid);
 		if (ret)
 			goto out;
@@ -375,8 +383,9 @@ ssize_t rdtgroup_schemata_write(struct kernfs_open_file *of,
 	return ret ?: nbytes;
 }
 
-static void show_doms(struct seq_file *s, struct rdt_resource *r, int closid)
+static void show_doms(struct seq_file *s, struct resctrl_schema *schema, int closid)
 {
+	struct rdt_resource *r = schema->res;
 	struct rdt_hw_domain *hw_dom;
 	struct rdt_domain *dom;
 	bool sep = false;
@@ -409,8 +418,10 @@ int rdtgroup_schemata_show(struct kernfs_open_file *of,
 	rdtgrp = rdtgroup_kn_lock_live(of->kn);
 	if (rdtgrp) {
 		if (rdtgrp->mode == RDT_MODE_PSEUDO_LOCKSETUP) {
-			for_each_alloc_enabled_rdt_resource(r)
+			list_for_each_entry(schema, &resctrl_all_schema, list) {
+				r = schema->res;
 				seq_printf(s, "%s:uninitialized\n", r->name);
+			}
 		} else if (rdtgrp->mode == RDT_MODE_PSEUDO_LOCKED) {
 			if (!rdtgrp->plr->d) {
 				rdt_last_cmd_clear();
@@ -425,9 +436,8 @@ int rdtgroup_schemata_show(struct kernfs_open_file *of,
 		} else {
 			closid = rdtgrp->closid;
 			list_for_each_entry(schema, &resctrl_all_schema, list) {
-				r = schema->res;
 				if (closid < schema->num_closid)
-					show_doms(s, r, closid);
+					show_doms(s, schema, closid);
 			}
 		}
 	} else {
arch/x86/kernel/cpu/resctrl/internal.h +3 -3
index 6c87a81946b1..1e1f2493a87f 100644
@@ -384,9 +384,9 @@ static inline struct rdt_hw_resource *resctrl_to_arch_res(struct rdt_resource *r
 	return container_of(r, struct rdt_hw_resource, resctrl);
 }
 
-int parse_cbm(struct rdt_parse_data *data, struct rdt_resource *r,
+int parse_cbm(struct rdt_parse_data *data, struct resctrl_schema *s,
 	      struct rdt_domain *d);
-int parse_bw(struct rdt_parse_data *data, struct rdt_resource *r,
+int parse_bw(struct rdt_parse_data *data, struct resctrl_schema *s,
 	     struct rdt_domain *d);
 
 extern struct mutex rdtgroup_mutex;
@@ -483,7 +483,7 @@ ssize_t rdtgroup_schemata_write(struct kernfs_open_file *of,
 				char *buf, size_t nbytes, loff_t off);
 int rdtgroup_schemata_show(struct kernfs_open_file *of,
 			   struct seq_file *s, void *v);
-bool rdtgroup_cbm_overlaps(struct rdt_resource *r, struct rdt_domain *d,
+bool rdtgroup_cbm_overlaps(struct resctrl_schema *s, struct rdt_domain *d,
 			   unsigned long cbm, int closid, bool exclusive);
 unsigned int rdtgroup_cbm_to_size(struct rdt_resource *r, struct rdt_domain *d,
 				  unsigned long cbm);
arch/x86/kernel/cpu/resctrl/rdtgroup.c +23 -11
index 628e5eb4d7a9..592a517afd6a 100644
@@ -1223,7 +1223,7 @@ static bool __rdtgroup_cbm_overlaps(struct rdt_resource *r, struct rdt_domain *d
 
 /**
  * rdtgroup_cbm_overlaps - Does CBM overlap with other use of hardware
- * @r: Resource to which domain instance @d belongs.
+ * @s: Schema for the resource to which domain instance @d belongs.
  * @d: The domain instance for which @closid is being tested.
  * @cbm: Capacity bitmask being tested.
  * @closid: Intended closid for @cbm.
@@ -1241,9 +1241,10 @@ static bool __rdtgroup_cbm_overlaps(struct rdt_resource *r, struct rdt_domain *d
  *
  * Return: true if CBM overlap detected, false if there is no overlap
  */
-bool rdtgroup_cbm_overlaps(struct rdt_resource *r, struct rdt_domain *d,
+bool rdtgroup_cbm_overlaps(struct resctrl_schema *s, struct rdt_domain *d,
 			   unsigned long cbm, int closid, bool exclusive)
 {
+	struct rdt_resource *r = s->res;
 	struct rdt_resource *r_cdp;
 	struct rdt_domain *d_cdp;
 
@@ -1272,17 +1273,20 @@ static bool rdtgroup_mode_test_exclusive(struct rdtgroup *rdtgrp)
 {
 	struct rdt_hw_domain *hw_dom;
 	int closid = rdtgrp->closid;
+	struct resctrl_schema *s;
 	struct rdt_resource *r;
 	bool has_cache = false;
 	struct rdt_domain *d;
 
-	for_each_alloc_enabled_rdt_resource(r) {
+	list_for_each_entry(s, &resctrl_all_schema, list) {
+		r = s->res;
 		if (r->rid == RDT_RESOURCE_MBA)
 			continue;
 		has_cache = true;
 		list_for_each_entry(d, &r->domains, list) {
 			hw_dom = resctrl_to_arch_dom(d);
-			if (rdtgroup_cbm_overlaps(r, d, hw_dom->ctrl_val[closid],
+			if (rdtgroup_cbm_overlaps(s, d,
+						  hw_dom->ctrl_val[closid],
 						  rdtgrp->closid, false)) {
 				rdt_last_cmd_puts("Schemata overlaps\n");
 				return false;
@@ -1414,6 +1418,7 @@ unsigned int rdtgroup_cbm_to_size(struct rdt_resource *r,
 static int rdtgroup_size_show(struct kernfs_open_file *of,
 			      struct seq_file *s, void *v)
 {
+	struct resctrl_schema *schema;
 	struct rdt_hw_domain *hw_dom;
 	struct rdtgroup *rdtgrp;
 	struct rdt_resource *r;
@@ -1445,7 +1450,9 @@ static int rdtgroup_size_show(struct kernfs_open_file *of,
 		goto out;
 	}
 
-	for_each_alloc_enabled_rdt_resource(r) {
+	list_for_each_entry(schema, &resctrl_all_schema, list) {
+		r = schema->res;
+
 		sep = false;
 		seq_printf(s, "%*s:", max_name_width, r->name);
 		list_for_each_entry(d, &r->domains, list) {
@@ -2730,11 +2737,12 @@ static u32 cbm_ensure_valid(u32 _val, struct rdt_resource *r)
  * Set the RDT domain up to start off with all usable allocations. That is,
  * all shareable and unused bits. All-zero CBM is invalid.
  */
-static int __init_one_rdt_domain(struct rdt_domain *d, struct rdt_resource *r,
+static int __init_one_rdt_domain(struct rdt_domain *d, struct resctrl_schema *s,
 				 u32 closid)
 {
 	struct rdt_resource *r_cdp = NULL;
 	struct rdt_domain *d_cdp = NULL;
+	struct rdt_resource *r = s->res;
 	u32 used_b = 0, unused_b = 0;
 	unsigned long tmp_cbm;
 	enum rdtgrp_mode mode;
@@ -2804,13 +2812,13 @@ static int __init_one_rdt_domain(struct rdt_domain *d, struct rdt_resource *r,
  * If there are no more shareable bits available on any domain then
  * the entire allocation will fail.
  */
-static int rdtgroup_init_cat(struct rdt_resource *r, u32 closid)
+static int rdtgroup_init_cat(struct resctrl_schema *s, u32 closid)
 {
 	struct rdt_domain *d;
 	int ret;
 
-	list_for_each_entry(d, &r->domains, list) {
-		ret = __init_one_rdt_domain(d, r, closid);
+	list_for_each_entry(d, &s->res->domains, list) {
+		ret = __init_one_rdt_domain(d, s, closid);
 		if (ret < 0)
 			return ret;
 	}
@@ -2832,14 +2840,18 @@ static void rdtgroup_init_mba(struct rdt_resource *r)
 /* Initialize the RDT group's allocations. */
 static int rdtgroup_init_alloc(struct rdtgroup *rdtgrp)
 {
+	struct resctrl_schema *s;
 	struct rdt_resource *r;
 	int ret;
 
-	for_each_alloc_enabled_rdt_resource(r) {
+	lockdep_assert_held(&rdtgroup_mutex);
+
+	list_for_each_entry(s, &resctrl_all_schema, list) {
+		r = s->res;
 		if (r->rid == RDT_RESOURCE_MBA) {
 			rdtgroup_init_mba(r);
 		} else {
-			ret = rdtgroup_init_cat(r, rdtgrp->closid);
+			ret = rdtgroup_init_cat(s, rdtgrp->closid);
 			if (ret < 0)
 				return ret;
 		}
include/linux/resctrl.h +3 -2
index 20d8b6dd4af4..8a12f4128209 100644
@@ -115,6 +115,7 @@ struct resctrl_membw {
 };
 
 struct rdt_parse_data;
+struct resctrl_schema;
 
 /**
  * @rid:		The index of the resource
@@ -162,7 +163,7 @@ struct rdt_resource {
 	u32			default_ctrl;
 	const char		*format_str;
 	int			(*parse_ctrlval)(struct rdt_parse_data *data,
-						 struct rdt_resource *r,
+						 struct resctrl_schema *s,
 						 struct rdt_domain *d);
 	struct list_head	evt_list;
 	unsigned long		fflags;
@@ -171,7 +172,7 @@ struct rdt_resource {
 
 /**
  * @list:	Member of resctrl's schema list
- * @cdp_type:	Whether this entry is for code/data/both
+ * @conf_type:	Type of configuration, e.g. code/data/both
  * @res:	The rdt_resource for this entry
  * @num_closid	Number of CLOSIDs available for this resource
  */
Signature · 1 line
2.28.0
maintainer
#22 lore raw

Re: [PATCH 08/24] x86/resctrl: Walk the resctrl schema list instead of an arch list

Hi James,

Quoted 9 lines · James Morse

Now that resctrl has its own list of resources it is using, walk that list instead of the architectures list. This means resctrl has somewhere to keep schema properties with the resource that is using them.

Most users of for_each_alloc_enabled_rdt_resource() are per-schema, and also want a schema property, like the conf_type. Switch these to walk the schema list. Schema were only created for alloc_enabled resources so these two lists are currently equivalent.

From what I understand based on this description the patch will essentially change instances of for_each_alloc_enabled_rdt_resource() to walking the schema list ....

Quoted 14 lines
---
  arch/x86/kernel/cpu/resctrl/ctrlmondata.c | 38 ++++++++++++++---------
  arch/x86/kernel/cpu/resctrl/internal.h    |  6 ++--
  arch/x86/kernel/cpu/resctrl/rdtgroup.c    | 34 +++++++++++++-------
  include/linux/resctrl.h                   |  5 +--
  4 files changed, 53 insertions(+), 30 deletions(-)
arch/x86/kernel/cpu/resctrl/ctrlmondata.c +0 -0
index 8ac104c634fe..d3f9d142f58a 100644
@@ -57,9 +57,10 @@ static bool bw_validate(char *buf, unsigned long *data, struct rdt_resource *r)
  	return true;

arch/x86/kernel/cpu/resctrl/ctrlmondata.c static bool bw_validate(char *buf, unsigned long *data, struct rdt_resource *r) · line 66

Quoted 10 lines
arch/x86/kernel/cpu/resctrl/ctrlmondata.c +2 -1
  }
 
-int parse_bw(struct rdt_parse_data *data, struct rdt_resource *r,
+int parse_bw(struct rdt_parse_data *data, struct resctrl_schema *s,
  	     struct rdt_domain *d)
  {
+	struct rdt_resource *r = s->res;
  	unsigned long bw_val;
 
  	if (d->have_new_ctrl) {

... this change and also the ones to parse_cbm() and rdtgroup_cbm_overlaps() are not clear to me because it seems they replace the rdt_resource parameter with resctrl_schema, but all in turn use that to access rdt_resource again. That seems unnecessary?

arch/x86/kernel/cpu/resctrl/ctrlmondata.c static bool cbm_validate(char *buf, u32 *data, struct rdt_resource *r) · line 136

Quoted 13 lines
arch/x86/kernel/cpu/resctrl/ctrlmondata.c +2 -1
@@ -125,10 +126,11 @@ static bool cbm_validate(char *buf, u32 *data, struct rdt_resource *r)
   * Read one cache bit mask (hex). Check that it is valid for the current
   * resource type.
   */
-int parse_cbm(struct rdt_parse_data *data, struct rdt_resource *r,
+int parse_cbm(struct rdt_parse_data *data, struct resctrl_schema *s,
  	      struct rdt_domain *d)
  {
  	struct rdtgroup *rdtgrp = data->rdtgrp;
+	struct rdt_resource *r = s->res;
  	u32 cbm_val;
 
  	if (d->have_new_ctrl) {

Really needed?

Quoted 5 lines
arch/x86/kernel/cpu/resctrl/ctrlmondata.c +0 -1
@@ -160,12 +162,12 @@ int parse_cbm(struct rdt_parse_data *data, struct rdt_resource *r,
  	 * The CBM may not overlap with the CBM of another closid if
  	 * either is exclusive.
  	 */
-	if (rdtgroup_cbm_overlaps(r, d, cbm_val, rdtgrp->closid, true)) {

arch/x86/kernel/cpu/resctrl/ctrlmondata.c int parse_cbm(struct rdt_parse_data *data, struct rdt_resource *r, · line 173

Quoted 10 lines
arch/x86/kernel/cpu/resctrl/ctrlmondata.c +2 -1
+	if (rdtgroup_cbm_overlaps(s, d, cbm_val, rdtgrp->closid, true)) {
  		rdt_last_cmd_puts("Overlaps with exclusive group\n");
  		return -EINVAL;
  	}
 
-	if (rdtgroup_cbm_overlaps(r, d, cbm_val, rdtgrp->closid, false)) {
+	if (rdtgroup_cbm_overlaps(s, d, cbm_val, rdtgrp->closid, false)) {
  		if (rdtgrp->mode == RDT_MODE_EXCLUSIVE ||
  		    rdtgrp->mode == RDT_MODE_PSEUDO_LOCKSETUP) {
  			rdt_last_cmd_puts("Overlaps with other group\n");

Needed?

Quoted 13 lines
arch/x86/kernel/cpu/resctrl/ctrlmondata.c +2 -1
@@ -185,9 +187,10 @@ int parse_cbm(struct rdt_parse_data *data, struct rdt_resource *r,
   * separated by ";". The "id" is in decimal, and must match one of
   * the "id"s for this resource.
   */
-static int parse_line(char *line, struct rdt_resource *r,
+static int parse_line(char *line, struct resctrl_schema *s,
  		      struct rdtgroup *rdtgrp)
  {
+	struct rdt_resource *r = s->res;
  	struct rdt_parse_data data;
  	char *dom = NULL, *id;
  	struct rdt_domain *d;

arch/x86/kernel/cpu/resctrl/ctrlmondata.c static int parse_line(char *line, struct rdt_resource *r, · line 223

Quoted 9 lines
arch/x86/kernel/cpu/resctrl/ctrlmondata.c +2 -1
@@ -213,7 +216,8 @@ static int parse_line(char *line, struct rdt_resource *r,
  		if (d->id == dom_id) {
  			data.buf = dom;
  			data.rdtgrp = rdtgrp;
-			if (r->parse_ctrlval(&data, r, d))
+
+			if (r->parse_ctrlval(&data, s, d))
  				return -EINVAL;
  			if (rdtgrp->mode ==  RDT_MODE_PSEUDO_LOCKSETUP) {
 

needed?

			/*

arch/x86/kernel/cpu/resctrl/ctrlmondata.c static int rdtgroup_parse_resource(char *resname, char *tok, · line 297

Quoted 6 lines
arch/x86/kernel/cpu/resctrl/ctrlmondata.c +2 -0
@@ -289,10 +293,12 @@ static int rdtgroup_parse_resource(char *resname, char *tok,
  	struct resctrl_schema *s;
  	struct rdt_resource *r;
 
+	lockdep_assert_held(&rdtgroup_mutex);
+

It is not clear how this addition fits into patch.

arch/x86/kernel/cpu/resctrl/ctrlmondata.c static int rdtgroup_parse_resource(char *resname, char *tok, · line 302

Quoted 6 lines
arch/x86/kernel/cpu/resctrl/ctrlmondata.c +1 -1
  	list_for_each_entry(s, &resctrl_all_schema, list) {
  		r = s->res;
  		if (!strcmp(resname, r->name) && rdtgrp->closid < s->num_closid)
-			return parse_line(tok, r, rdtgrp);
+			return parse_line(tok, s, rdtgrp);
  	}

needed? (similar comments to other changes in this patch but I will stop here)

Quoted 2 lines
arch/x86/kernel/cpu/resctrl/ctrlmondata.c +0 -0
  	rdt_last_cmd_printf("Unknown or unsupported resource name '%s'\n", resname);
  	return -EINVAL;

...

Quoted 4 lines
arch/x86/kernel/cpu/resctrl/rdtgroup.c +0 -0
index 628e5eb4d7a9..592a517afd6a 100644

...

arch/x86/kernel/cpu/resctrl/rdtgroup.c static void rdtgroup_init_mba(struct rdt_resource *r) · line 2848

Quoted 11 lines
arch/x86/kernel/cpu/resctrl/rdtgroup.c +3 -1
@@ -2832,14 +2840,18 @@ static void rdtgroup_init_mba(struct rdt_resource *r)
  /* Initialize the RDT group's allocations. */
  static int rdtgroup_init_alloc(struct rdtgroup *rdtgrp)
  {
+	struct resctrl_schema *s;
  	struct rdt_resource *r;
  	int ret;
 
-	for_each_alloc_enabled_rdt_resource(r) {
+	lockdep_assert_held(&rdtgroup_mutex);
+

Another addition that does not match patch description.

Quoted 14 lines
arch/x86/kernel/cpu/resctrl/rdtgroup.c +3 -1
+	list_for_each_entry(s, &resctrl_all_schema, list) {
+		r = s->res;
  		if (r->rid == RDT_RESOURCE_MBA) {
  			rdtgroup_init_mba(r);
  		} else {
-			ret = rdtgroup_init_cat(r, rdtgrp->closid);
+			ret = rdtgroup_init_cat(s, rdtgrp->closid);
  			if (ret < 0)
  				return ret;
  		}
include/linux/resctrl.h +0 -0
index 20d8b6dd4af4..8a12f4128209 100644

...

include/linux/resctrl.h struct rdt_resource { · line 178

Quoted 10 lines
include/linux/resctrl.h +1 -1
@@ -171,7 +172,7 @@ struct rdt_resource {
 
  /**
   * @list:	Member of resctrl's schema list
- * @cdp_type:	Whether this entry is for code/data/both
+ * @conf_type:	Type of configuration, e.g. code/data/both
   * @res:	The rdt_resource for this entry
   * @num_closid	Number of CLOSIDs available for this resource
   */

This hunk can be merged with previous patch.

Reinette

author reviewer
#23 lore raw

Re: [PATCH 08/24] x86/resctrl: Walk the resctrl schema list instead of an arch list

Hi Reinette,

Quoted 26 lines · Reinette Chatre
Quoted 9 lines · James Morse

Now that resctrl has its own list of resources it is using, walk that list instead of the architectures list. This means resctrl has somewhere to keep schema properties with the resource that is using them.

Most users of for_each_alloc_enabled_rdt_resource() are per-schema, and also want a schema property, like the conf_type. Switch these to walk the schema list. Schema were only created for alloc_enabled resources so these two lists are currently equivalent.

From what I understand based on this description the patch will essentially change instances of for_each_alloc_enabled_rdt_resource() to walking the schema list ....

Quoted 12 lines
arch/x86/kernel/cpu/resctrl/ctrlmondata.c +1 -0
index 8ac104c634fe..d3f9d142f58a 100644
@@ -57,9 +57,10 @@ static bool bw_validate(char *buf, unsigned long *data, struct
 rdt_resource *r)
       return true;
   }
   -int parse_bw(struct rdt_parse_data *data, struct rdt_resource *r,
+int parse_bw(struct rdt_parse_data *data, struct resctrl_schema *s,
            struct rdt_domain *d)
Quoted 8 lines

arch/x86/kernel/cpu/resctrl/ctrlmondata.c static bool bw_validate(char *buf, unsigned long *data, struct rdt_resource *r) · line 66

Quoted 4 lines
arch/x86/kernel/cpu/resctrl/ctrlmondata.c +1 -0
   {
+    struct rdt_resource *r = s->res;
       unsigned long bw_val;
         if (d->have_new_ctrl) {

... this change and also the ones to parse_cbm() and rdtgroup_cbm_overlaps() are not clear to me because it seems they replace the rdt_resource parameter with resctrl_schema, but all in turn use that to access rdt_resource again. That seems unnecessary?

I previously restructured the series to do the schema stuff first, as I thought it would make it easier to follow. It looks like this patch has picked up other stuff - I 'll split this up so those changes get their own commit message.

By the end of the series, the rdt_resource isn't unique, the same 'L3' is provided to L3CODE and L3DATA. The things that make these different are stored in the schema. In both these cases the configuration is read/written, where that should go depends on the type of the schema, which lives in the schema struct.

Thanks,

James

author reviewer
#24 lore raw

PATCH 9/24 x86/resctrl: Change rdt_resource to resctrl_schema in pseudo_lock_region

In reply to James Morse

struct pseudo_lock_region points to the rdt_resource. Once the resources are merged, this won't be unique. The resource name is moving into the schema, so that eventually resctrl can generate it.

Change pseudo_lock_region's rdt_resource pointer for a schema pointer.

---
 arch/x86/kernel/cpu/resctrl/ctrlmondata.c | 4 ++--
 arch/x86/kernel/cpu/resctrl/internal.h    | 6 +++---
 arch/x86/kernel/cpu/resctrl/pseudo_lock.c | 8 ++++----
 arch/x86/kernel/cpu/resctrl/rdtgroup.c    | 4 ++--
 4 files changed, 11 insertions(+), 11 deletions(-)
arch/x86/kernel/cpu/resctrl/ctrlmondata.c +2 -2
index d3f9d142f58a..a65ff53394ed 100644
@@ -228,7 +228,7 @@ static int parse_line(char *line, struct resctrl_schema *s,
 				 * the required initialization for single
 				 * region and return.
 				 */
-				rdtgrp->plr->r = r;
+				rdtgrp->plr->s = s;
 				rdtgrp->plr->d = d;
 				rdtgrp->plr->cbm = d->new_ctrl;
 				d->plr = rdtgrp->plr;
@@ -429,7 +429,7 @@ int rdtgroup_schemata_show(struct kernfs_open_file *of,
 				ret = -ENODEV;
 			} else {
 				seq_printf(s, "%s:%d=%x\n",
-					   rdtgrp->plr->r->name,
+					   rdtgrp->plr->s->res->name,
 					   rdtgrp->plr->d->id,
 					   rdtgrp->plr->cbm);
 			}
arch/x86/kernel/cpu/resctrl/internal.h +3 -3
index 1e1f2493a87f..27671a654f8b 100644
@@ -158,8 +158,8 @@ struct mongroup {
 
 /**
  * struct pseudo_lock_region - pseudo-lock region information
- * @r:			RDT resource to which this pseudo-locked region
- *			belongs
+ * @s:			Resctrl schema for the resource to which this
+ *			pseudo-locked region belongs
  * @d:			RDT domain to which this pseudo-locked region
  *			belongs
  * @cbm:		bitmask of the pseudo-locked region
@@ -179,7 +179,7 @@ struct mongroup {
  * @pm_reqs:		Power management QoS requests related to this region
  */
 struct pseudo_lock_region {
-	struct rdt_resource	*r;
+	struct resctrl_schema	*s;
 	struct rdt_domain	*d;
 	u32			cbm;
 	wait_queue_head_t	lock_thread_wq;
arch/x86/kernel/cpu/resctrl/pseudo_lock.c +4 -4
index d58f2ffa65e0..d9d9861f244f 100644
@@ -246,7 +246,7 @@ static void pseudo_lock_region_clear(struct pseudo_lock_region *plr)
 	plr->line_size = 0;
 	kfree(plr->kmem);
 	plr->kmem = NULL;
-	plr->r = NULL;
+	plr->s = NULL;
 	if (plr->d)
 		plr->d->plr = NULL;
 	plr->d = NULL;
@@ -290,10 +290,10 @@ static int pseudo_lock_region_init(struct pseudo_lock_region *plr)
 
 	ci = get_cpu_cacheinfo(plr->cpu);
 
-	plr->size = rdtgroup_cbm_to_size(plr->r, plr->d, plr->cbm);
+	plr->size = rdtgroup_cbm_to_size(plr->s->res, plr->d, plr->cbm);
 
 	for (i = 0; i < ci->num_leaves; i++) {
-		if (ci->info_list[i].level == plr->r->cache_level) {
+		if (ci->info_list[i].level == plr->s->res->cache_level) {
 			plr->line_size = ci->info_list[i].coherency_line_size;
 			return 0;
 		}
@@ -796,7 +796,7 @@ bool rdtgroup_cbm_overlaps_pseudo_locked(struct rdt_domain *d, unsigned long cbm
 	unsigned long cbm_b;
 
 	if (d->plr) {
-		cbm_len = d->plr->r->cache.cbm_len;
+		cbm_len = d->plr->s->res->cache.cbm_len;
 		cbm_b = d->plr->cbm;
 		if (bitmap_intersects(&cbm, &cbm_b, cbm_len))
 			return true;
arch/x86/kernel/cpu/resctrl/rdtgroup.c +2 -2
index 592a517afd6a..311a3890bc53 100644
@@ -1441,8 +1441,8 @@ static int rdtgroup_size_show(struct kernfs_open_file *of,
 			ret = -ENODEV;
 		} else {
 			seq_printf(s, "%*s:", max_name_width,
-				   rdtgrp->plr->r->name);
-			size = rdtgroup_cbm_to_size(rdtgrp->plr->r,
+				   rdtgrp->plr->s->res->name);
+			size = rdtgroup_cbm_to_size(rdtgrp->plr->s->res,
 						    rdtgrp->plr->d,
 						    rdtgrp->plr->cbm);
 			seq_printf(s, "%d=%u\n", rdtgrp->plr->d->id, size);
Signature · 1 line
2.28.0
author reviewer
#25 lore raw

PATCH 10/24 x86/resctrl: Move the schema names into struct resctrl_schema

In reply to James Morse

Move the names used for the schemata file out of the resource and into struct resctrl_schema. This allows one resource to have two different names, based on the other schema properties.

This patch copies the names, eventually resctrl will generate them.

Remove the arch code's max_name_width, this is now resctrl's problem.

---
 arch/x86/kernel/cpu/resctrl/core.c        |  9 ++-------
 arch/x86/kernel/cpu/resctrl/ctrlmondata.c | 10 +++-------
 arch/x86/kernel/cpu/resctrl/internal.h    |  2 +-
 arch/x86/kernel/cpu/resctrl/rdtgroup.c    | 17 ++++++++++++-----
 include/linux/resctrl.h                   |  7 +++++++
 5 files changed, 25 insertions(+), 20 deletions(-)
arch/x86/kernel/cpu/resctrl/core.c +2 -7
index 1ed5e04031e6..cda071009fed 100644
@@ -37,10 +37,10 @@ DEFINE_MUTEX(rdtgroup_mutex);
 DEFINE_PER_CPU(struct resctrl_pqr_state, pqr_state);
 
 /*
- * Used to store the max resource name width and max resource data width
+ * Used to store the max resource data width
  * to display the schemata in a tabular format
  */
-int max_name_width, max_data_width;
+int max_data_width;
 
 /*
  * Global boolean for rdt_alloc which is true if any
@@ -776,13 +776,8 @@ static int resctrl_offline_cpu(unsigned int cpu)
 static __init void rdt_init_padding(void)
 {
 	struct rdt_resource *r;
-	int cl;
 
 	for_each_alloc_capable_rdt_resource(r) {
-		cl = strlen(r->name);
-		if (cl > max_name_width)
-			max_name_width = cl;
-
 		if (r->data_width > max_data_width)
 			max_data_width = r->data_width;
 	}
arch/x86/kernel/cpu/resctrl/ctrlmondata.c +3 -7
index a65ff53394ed..28d69c78c29e 100644
@@ -291,13 +291,11 @@ static int rdtgroup_parse_resource(char *resname, char *tok,
 				   struct rdtgroup *rdtgrp)
 {
 	struct resctrl_schema *s;
-	struct rdt_resource *r;
 
 	lockdep_assert_held(&rdtgroup_mutex);
 
 	list_for_each_entry(s, &resctrl_all_schema, list) {
-		r = s->res;
-		if (!strcmp(resname, r->name) && rdtgrp->closid < s->num_closid)
+		if (!strcmp(resname, s->name) && rdtgrp->closid < s->num_closid)
 			return parse_line(tok, s, rdtgrp);
 	}
 	rdt_last_cmd_printf("Unknown or unsupported resource name '%s'\n", resname);
@@ -391,7 +389,7 @@ static void show_doms(struct seq_file *s, struct resctrl_schema *schema, int clo
 	bool sep = false;
 	u32 ctrl_val;
 
-	seq_printf(s, "%*s:", max_name_width, r->name);
+	seq_printf(s, "%*s:", RESCTRL_NAME_LEN, schema->name);
 	list_for_each_entry(dom, &r->domains, list) {
 		hw_dom = resctrl_to_arch_dom(dom);
 		if (sep)
@@ -411,7 +409,6 @@ int rdtgroup_schemata_show(struct kernfs_open_file *of,
 {
 	struct resctrl_schema *schema;
 	struct rdtgroup *rdtgrp;
-	struct rdt_resource *r;
 	int ret = 0;
 	u32 closid;
 
@@ -419,8 +416,7 @@ int rdtgroup_schemata_show(struct kernfs_open_file *of,
 	if (rdtgrp) {
 		if (rdtgrp->mode == RDT_MODE_PSEUDO_LOCKSETUP) {
 			list_for_each_entry(schema, &resctrl_all_schema, list) {
-				r = schema->res;
-				seq_printf(s, "%s:uninitialized\n", r->name);
+				seq_printf(s, "%s:uninitialized\n", schema->name);
 			}
 		} else if (rdtgrp->mode == RDT_MODE_PSEUDO_LOCKED) {
 			if (!rdtgrp->plr->d) {
arch/x86/kernel/cpu/resctrl/internal.h +1 -1
index 27671a654f8b..5294ae0c3ed9 100644
@@ -248,7 +248,7 @@ struct rdtgroup {
 /* List of all resource groups */
 extern struct list_head rdt_all_groups;
 
-extern int max_name_width, max_data_width;
+extern int max_data_width;
 
 int __init rdtgroup_init(void);
 void __exit rdtgroup_exit(void);
arch/x86/kernel/cpu/resctrl/rdtgroup.c +12 -5
index 311a3890bc53..48f4d6783647 100644
@@ -1440,8 +1440,8 @@ static int rdtgroup_size_show(struct kernfs_open_file *of,
 			rdt_last_cmd_puts("Cache domain offline\n");
 			ret = -ENODEV;
 		} else {
-			seq_printf(s, "%*s:", max_name_width,
-				   rdtgrp->plr->s->res->name);
+			seq_printf(s, "%*s:", RESCTRL_NAME_LEN,
+				   rdtgrp->plr->s->name);
 			size = rdtgroup_cbm_to_size(rdtgrp->plr->s->res,
 						    rdtgrp->plr->d,
 						    rdtgrp->plr->cbm);
@@ -1454,7 +1454,7 @@ static int rdtgroup_size_show(struct kernfs_open_file *of,
 		r = schema->res;
 
 		sep = false;
-		seq_printf(s, "%*s:", max_name_width, r->name);
+		seq_printf(s, "%*s:", RESCTRL_NAME_LEN, schema->name);
 		list_for_each_entry(d, &r->domains, list) {
 			hw_dom = resctrl_to_arch_dom(d);
 			if (sep)
@@ -1827,7 +1827,7 @@ static int rdtgroup_create_info_dir(struct kernfs_node *parent_kn)
 	list_for_each_entry(s, &resctrl_all_schema, list) {
 		r = s->res;
 		fflags =  r->fflags | RF_CTRL_INFO;
-		ret = rdtgroup_mkdir_info_resdir(s, r->name, fflags);
+		ret = rdtgroup_mkdir_info_resdir(s, s->name, fflags);
 		if (ret)
 			goto out_destroy;
 	}
@@ -2140,6 +2140,7 @@ static int create_schemata_list(void)
 {
 	struct resctrl_schema *s;
 	struct rdt_resource *r;
+	int ret;
 
 	for_each_alloc_enabled_rdt_resource(r) {
 		s = kzalloc(sizeof(*s), GFP_KERNEL);
@@ -2150,6 +2151,12 @@ static int create_schemata_list(void)
 		s->num_closid = resctrl_arch_get_num_closid(r);
 		s->conf_type = resctrl_to_arch_res(r)->conf_type;
 
+		ret = snprintf(s->name, sizeof(s->name), r->name);
+		if (ret >= sizeof(s->name)) {
+			kfree(s);
+			return -EINVAL;
+		}
+
 		INIT_LIST_HEAD(&s->list);
 		list_add(&s->list, &resctrl_all_schema);
 	}
@@ -2794,7 +2801,7 @@ static int __init_one_rdt_domain(struct rdt_domain *d, struct resctrl_schema *s,
 	 */
 	tmp_cbm = d->new_ctrl;
 	if (bitmap_weight(&tmp_cbm, r->cache.cbm_len) < r->cache.min_cbm_bits) {
-		rdt_last_cmd_printf("No space on %s:%d\n", r->name, d->id);
+		rdt_last_cmd_printf("No space on %s:%d\n", s->name, d->id);
 		return -ENOSPC;
 	}
 	d->have_new_ctrl = true;
include/linux/resctrl.h +7 -0
index 8a12f4128209..9f71f0238239 100644
@@ -15,6 +15,11 @@ int proc_resctrl_show(struct seq_file *m,
 
 #endif
 
+/*
+ * The longest name we expect in the schemata file:
+ */
+#define RESCTRL_NAME_LEN	7
+
 enum resctrl_conf_type {
 	CDP_BOTH,
 	CDP_CODE,
@@ -172,12 +177,14 @@ struct rdt_resource {
 
 /**
  * @list:	Member of resctrl's schema list
+ * @names:	Name to use in "schemata" file
  * @conf_type:	Type of configuration, e.g. code/data/both
  * @res:	The rdt_resource for this entry
  * @num_closid	Number of CLOSIDs available for this resource
  */
 struct resctrl_schema {
 	struct list_head		list;
+	char				name[RESCTRL_NAME_LEN];
 	enum resctrl_conf_type		conf_type;
 	struct rdt_resource		*res;
 	u32				num_closid;
Signature · 1 line
2.28.0
#26 lore raw

Re: [PATCH 10/24] x86/resctrl: Move the schema names into struct resctrl_schema

Hi James,

Quoted 17 lines · James Morse

Move the names used for the schemata file out of the resource and into struct resctrl_schema. This allows one resource to have two different names, based on the other schema properties.

This patch copies the names, eventually resctrl will generate them.

Remove the arch code's max_name_width, this is now resctrl's problem.

---
 arch/x86/kernel/cpu/resctrl/core.c        |  9 ++-------
 arch/x86/kernel/cpu/resctrl/ctrlmondata.c | 10 +++-------
 arch/x86/kernel/cpu/resctrl/internal.h    |  2 +-
 arch/x86/kernel/cpu/resctrl/rdtgroup.c    | 17 ++++++++++++-----
 include/linux/resctrl.h                   |  7 +++++++
 5 files changed, 25 insertions(+), 20 deletions(-)

...

Quoted 55 lines
arch/x86/kernel/cpu/resctrl/internal.h +1 -1
index 27671a654f8b..5294ae0c3ed9 100644
@@ -248,7 +248,7 @@ struct rdtgroup {
 /* List of all resource groups */
 extern struct list_head rdt_all_groups;
 
-extern int max_name_width, max_data_width;
+extern int max_data_width;
 
 int __init rdtgroup_init(void);
 void __exit rdtgroup_exit(void);
arch/x86/kernel/cpu/resctrl/rdtgroup.c +5 -4
index 311a3890bc53..48f4d6783647 100644
@@ -1440,8 +1440,8 @@ static int rdtgroup_size_show(struct kernfs_open_file *of,
 			rdt_last_cmd_puts("Cache domain offline\n");
 			ret = -ENODEV;
 		} else {
-			seq_printf(s, "%*s:", max_name_width,
-				   rdtgrp->plr->s->res->name);
+			seq_printf(s, "%*s:", RESCTRL_NAME_LEN,
+				   rdtgrp->plr->s->name);
 			size = rdtgroup_cbm_to_size(rdtgrp->plr->s->res,
 						    rdtgrp->plr->d,
 						    rdtgrp->plr->cbm);
@@ -1454,7 +1454,7 @@ static int rdtgroup_size_show(struct kernfs_open_file *of,
 		r = schema->res;
 
 		sep = false;
-		seq_printf(s, "%*s:", max_name_width, r->name);
+		seq_printf(s, "%*s:", RESCTRL_NAME_LEN, schema->name);
 		list_for_each_entry(d, &r->domains, list) {
 			hw_dom = resctrl_to_arch_dom(d);
 			if (sep)
@@ -1827,7 +1827,7 @@ static int rdtgroup_create_info_dir(struct kernfs_node *parent_kn)
 	list_for_each_entry(s, &resctrl_all_schema, list) {
 		r = s->res;
 		fflags =  r->fflags | RF_CTRL_INFO;
-		ret = rdtgroup_mkdir_info_resdir(s, r->name, fflags);
+		ret = rdtgroup_mkdir_info_resdir(s, s->name, fflags);
 		if (ret)
 			goto out_destroy;
 	}
@@ -2140,6 +2140,7 @@ static int create_schemata_list(void)
 {
 	struct resctrl_schema *s;
 	struct rdt_resource *r;
+	int ret;
 
 	for_each_alloc_enabled_rdt_resource(r) {
 		s = kzalloc(sizeof(*s), GFP_KERNEL);

arch/x86/kernel/cpu/resctrl/rdtgroup.c static int create_schemata_list(void) · line 2159

Quoted 9 lines
arch/x86/kernel/cpu/resctrl/rdtgroup.c +6 -0
@@ -2150,6 +2151,12 @@ static int create_schemata_list(void)
 		s->num_closid = resctrl_arch_get_num_closid(r);
 		s->conf_type = resctrl_to_arch_res(r)->conf_type;
 
+		ret = snprintf(s->name, sizeof(s->name), r->name);
+		if (ret >= sizeof(s->name)) {
+			kfree(s);
+			return -EINVAL;
+		}
+

How about:

+ ret = strscpy(s->name, r->name, sizeof(s->name));
+ if (ret < 0)) {
+ kfree(s);
+ return -EINVAL;
+ }

So that there isn't a non-constant format specifier that'll trip Coverity+friends up later?

Thanks,

Jamie

author reviewer
#27 lore raw

Re: [PATCH 10/24] x86/resctrl: Move the schema names into struct resctrl_schema

Hi Jamie,

Thanks for taking a look,

Quoted 27 lines · Jamie Iles
Quoted 14 lines · James Morse

Move the names used for the schemata file out of the resource and into struct resctrl_schema. This allows one resource to have two different names, based on the other schema properties.

This patch copies the names, eventually resctrl will generate them.

Remove the arch code's max_name_width, this is now resctrl's problem.

arch/x86/kernel/cpu/resctrl/rdtgroup.c +0 -0
index 311a3890bc53..48f4d6783647 100644

arch/x86/kernel/cpu/resctrl/rdtgroup.c static int create_schemata_list(void) · line 2159

Quoted 9 lines
arch/x86/kernel/cpu/resctrl/rdtgroup.c +6 -0
@@ -2150,6 +2151,12 @@ static int create_schemata_list(void)
 		s->num_closid = resctrl_arch_get_num_closid(r);
 		s->conf_type = resctrl_to_arch_res(r)->conf_type;
 
+		ret = snprintf(s->name, sizeof(s->name), r->name);
+		if (ret >= sizeof(s->name)) {
+			kfree(s);
+			return -EINVAL;
+		}
+

How about:

arch/x86/kernel/cpu/resctrl/rdtgroup.c static int create_schemata_list(void) · line 2158

Quoted 5 lines
arch/x86/kernel/cpu/resctrl/rdtgroup.c +5 -0
+		ret = strscpy(s->name, r->name, sizeof(s->name));
+		if (ret < 0)) {
+			kfree(s);
+			return -EINVAL;
+		}

Never heard of it ... yup, that looks better. Thanks! (I thought I knew not to write that bug!)

Quoted 2 lines

So that there isn't a non-constant format specifier that'll trip Coverity+friends up later?

Heh, its gone by the last patch.
Fixed locally.

Thanks,

James

maintainer
#28 lore raw

Re: [PATCH 10/24] x86/resctrl: Move the schema names into struct resctrl_schema

In reply to James Morse

Hi James,

Quoted 5 lines · James Morse

Move the names used for the schemata file out of the resource and into struct resctrl_schema. This allows one resource to have two different names, based on the other schema properties.

This patch copies the names, eventually resctrl will generate them.

Please remove "This patch".

Quoted 34 lines

Remove the arch code's max_name_width, this is now resctrl's problem.

---
  arch/x86/kernel/cpu/resctrl/core.c        |  9 ++-------
  arch/x86/kernel/cpu/resctrl/ctrlmondata.c | 10 +++-------
  arch/x86/kernel/cpu/resctrl/internal.h    |  2 +-
  arch/x86/kernel/cpu/resctrl/rdtgroup.c    | 17 ++++++++++++-----
  include/linux/resctrl.h                   |  7 +++++++
  5 files changed, 25 insertions(+), 20 deletions(-)
arch/x86/kernel/cpu/resctrl/core.c +2 -2
index 1ed5e04031e6..cda071009fed 100644
@@ -37,10 +37,10 @@ DEFINE_MUTEX(rdtgroup_mutex);
  DEFINE_PER_CPU(struct resctrl_pqr_state, pqr_state);
 
  /*
- * Used to store the max resource name width and max resource data width
+ * Used to store the max resource data width
   * to display the schemata in a tabular format
   */
-int max_name_width, max_data_width;
+int max_data_width;
 
  /*
   * Global boolean for rdt_alloc which is true if any
@@ -776,13 +776,8 @@ static int resctrl_offline_cpu(unsigned int cpu)
  static __init void rdt_init_padding(void)
  {
  	struct rdt_resource *r;

arch/x86/kernel/cpu/resctrl/core.c static int resctrl_offline_cpu(unsigned int cpu) · line 783

Quoted 10 lines
arch/x86/kernel/cpu/resctrl/core.c +0 -5
-	int cl;
 
  	for_each_alloc_capable_rdt_resource(r) {
-		cl = strlen(r->name);
-		if (cl > max_name_width)
-			max_name_width = cl;
-
  		if (r->data_width > max_data_width)
  			max_data_width = r->data_width;
  	}

The original code determines the maximum width based on resources supported by the platform.

Quoted 4 lines
arch/x86/kernel/cpu/resctrl/ctrlmondata.c +0 -0
index a65ff53394ed..28d69c78c29e 100644

...

arch/x86/kernel/cpu/resctrl/ctrlmondata.c static void show_doms(struct seq_file *s, struct resctrl_schema *schema, int clo · line 392

Quoted 6 lines
arch/x86/kernel/cpu/resctrl/ctrlmondata.c +1 -1
@@ -391,7 +389,7 @@ static void show_doms(struct seq_file *s, struct resctrl_schema *schema, int clo
  	bool sep = false;
  	u32 ctrl_val;
 
-	seq_printf(s, "%*s:", max_name_width, r->name);
+	seq_printf(s, "%*s:", RESCTRL_NAME_LEN, schema->name);

From what I understand this changes what some users will see. In the original code the "max_name_width" is computed based on the maximum length of resources supported. Systems that only support MBA would thus show a schemata of:

MB:0=100;1=100

I expect the above change would change the output to:

    MB:0=100;1=100
Quoted 3 lines
arch/x86/kernel/cpu/resctrl/ctrlmondata.c +0 -0
  	list_for_each_entry(dom, &r->domains, list) {
  		hw_dom = resctrl_to_arch_dom(dom);
  		if (sep)

...

Quoted 17 lines
include/linux/resctrl.h +5 -0
index 8a12f4128209..9f71f0238239 100644
@@ -15,6 +15,11 @@ int proc_resctrl_show(struct seq_file *m,
 
  #endif
 
+/*
+ * The longest name we expect in the schemata file:
+ */
+#define RESCTRL_NAME_LEN	7
+
  enum resctrl_conf_type {
  	CDP_BOTH,
  	CDP_CODE,

include/linux/resctrl.h struct rdt_resource { · line 180

Quoted 4 lines
include/linux/resctrl.h +1 -0
@@ -172,12 +177,14 @@ struct rdt_resource {
 
  /**
   * @list:	Member of resctrl's schema list
+ * @names:	Name to use in "schemata" file

s/names/name/?

include/linux/resctrl.h struct rdt_resource { · line 190

Quoted 11 lines
include/linux/resctrl.h +1 -0
   * @conf_type:	Type of configuration, e.g. code/data/both
   * @res:	The rdt_resource for this entry
   * @num_closid	Number of CLOSIDs available for this resource
   */
  struct resctrl_schema {
  	struct list_head		list;
+	char				name[RESCTRL_NAME_LEN];
  	enum resctrl_conf_type		conf_type;
  	struct rdt_resource		*res;
  	u32				num_closid;

Reinette

author reviewer
#29 lore raw

Re: [PATCH 10/24] x86/resctrl: Move the schema names into struct resctrl_schema

Hi Reinette,

Quoted 32 lines · Reinette Chatre
Quoted 5 lines · James Morse

Move the names used for the schemata file out of the resource and into struct resctrl_schema. This allows one resource to have two different names, based on the other schema properties.

This patch copies the names, eventually resctrl will generate them.

Please remove "This patch".

Quoted 9 lines

Remove the arch code's max_name_width, this is now resctrl's problem.

arch/x86/kernel/cpu/resctrl/ctrlmondata.c +0 -0
index a65ff53394ed..28d69c78c29e 100644

...

arch/x86/kernel/cpu/resctrl/ctrlmondata.c static void show_doms(struct seq_file *s, struct resctrl_schema · line 393

Quoted 6 lines
arch/x86/kernel/cpu/resctrl/ctrlmondata.c +1 -0
@@ -391,7 +389,7 @@ static void show_doms(struct seq_file *s, struct resctrl_schema
 *schema, int clo
       bool sep = false;
       u32 ctrl_val;
   -    seq_printf(s, "%*s:", max_name_width, r->name);
+    seq_printf(s, "%*s:", RESCTRL_NAME_LEN, schema->name);

From what I understand this changes what some users will see. In the original code the "max_name_width" is computed based on the maximum length of resources supported. Systems that only support MBA would thus show a schemata of:

Quoted 4 lines

MB:0=100;1=100

I expect the above change would change the output to:
    MB:0=100;1=100

Aha! Despite the comment - I've totally miss-understood what this code is doing.

Thanks!

James

author reviewer
#30 lore raw

PATCH 11/24 x86/resctrl: Group staged configuration into a separate struct

In reply to James Morse

Arm's MPAM may have surprisingly large bitmaps for its cache portions as the architecture allows up to 4K portions. The size exposed via resctrl may not be the same, some scaling may occur.

The values written to hardware may be unlike the values received from resctrl, e.g. MBA percentages may be backed by a bitmap, or a maximum value that isn't a percentage.

Today resctrl's ctrlval arrays are written to directly by the resctrl filesystem code. e.g. apply_config(). This is a problem if scaling or conversion is needed by the architecture.

The arch code should own the ctrlval array (to allow scaling and conversion), and should only need a single copy of the array for the values currently applied in hardware.

Move the new_ctrl bitmap value and flag into a struct for staged configuration changes. This is created as an array to allow one per type of configuration. Today there is only one element in the array, but eventually resctrl will use the array slots for CODE/DATA/BOTH to detect a duplicate schema being written.

---
 arch/x86/kernel/cpu/resctrl/ctrlmondata.c | 49 ++++++++++++++++-------
 arch/x86/kernel/cpu/resctrl/rdtgroup.c    | 22 +++++-----
 include/linux/resctrl.h                   | 17 +++++---
 3 files changed, 60 insertions(+), 28 deletions(-)
arch/x86/kernel/cpu/resctrl/ctrlmondata.c +35 -14
index 28d69c78c29e..0c95ed83eb05 100644
@@ -60,18 +60,19 @@ static bool bw_validate(char *buf, unsigned long *data, struct rdt_resource *r)
 int parse_bw(struct rdt_parse_data *data, struct resctrl_schema *s,
 	     struct rdt_domain *d)
 {
+	struct resctrl_staged_config *cfg = &d->staged_config[0];
 	struct rdt_resource *r = s->res;
 	unsigned long bw_val;
 
-	if (d->have_new_ctrl) {
+	if (cfg->have_new_ctrl) {
 		rdt_last_cmd_printf("Duplicate domain %d\n", d->id);
 		return -EINVAL;
 	}
 
 	if (!bw_validate(data->buf, &bw_val, r))
 		return -EINVAL;
-	d->new_ctrl = bw_val;
-	d->have_new_ctrl = true;
+	cfg->new_ctrl = bw_val;
+	cfg->have_new_ctrl = true;
 
 	return 0;
 }
@@ -129,11 +130,12 @@ static bool cbm_validate(char *buf, u32 *data, struct rdt_resource *r)
 int parse_cbm(struct rdt_parse_data *data, struct resctrl_schema *s,
 	      struct rdt_domain *d)
 {
+	struct resctrl_staged_config *cfg = &d->staged_config[0];
 	struct rdtgroup *rdtgrp = data->rdtgrp;
 	struct rdt_resource *r = s->res;
 	u32 cbm_val;
 
-	if (d->have_new_ctrl) {
+	if (cfg->have_new_ctrl) {
 		rdt_last_cmd_printf("Duplicate domain %d\n", d->id);
 		return -EINVAL;
 	}
@@ -175,8 +177,8 @@ int parse_cbm(struct rdt_parse_data *data, struct resctrl_schema *s,
 		}
 	}
 
-	d->new_ctrl = cbm_val;
-	d->have_new_ctrl = true;
+	cfg->new_ctrl = cbm_val;
+	cfg->have_new_ctrl = true;
 
 	return 0;
 }
@@ -190,6 +192,7 @@ int parse_cbm(struct rdt_parse_data *data, struct resctrl_schema *s,
 static int parse_line(char *line, struct resctrl_schema *s,
 		      struct rdtgroup *rdtgrp)
 {
+	struct resctrl_staged_config *cfg;
 	struct rdt_resource *r = s->res;
 	struct rdt_parse_data data;
 	char *dom = NULL, *id;
@@ -220,6 +223,7 @@ static int parse_line(char *line, struct resctrl_schema *s,
 			if (r->parse_ctrlval(&data, s, d))
 				return -EINVAL;
 			if (rdtgrp->mode ==  RDT_MODE_PSEUDO_LOCKSETUP) {
+				cfg = &d->staged_config[0];
 				/*
 				 * In pseudo-locking setup mode and just
 				 * parsed a valid CBM that should be
@@ -230,7 +234,7 @@ static int parse_line(char *line, struct resctrl_schema *s,
 				 */
 				rdtgrp->plr->s = s;
 				rdtgrp->plr->d = d;
-				rdtgrp->plr->cbm = d->new_ctrl;
+				rdtgrp->plr->cbm = cfg->new_ctrl;
 				d->plr = rdtgrp->plr;
 				return 0;
 			}
@@ -240,15 +244,30 @@ static int parse_line(char *line, struct resctrl_schema *s,
 	return -EINVAL;
 }
 
+static void apply_config(struct rdt_hw_domain *hw_dom,
+			 struct resctrl_staged_config *cfg, int closid,
+			 cpumask_var_t cpu_mask, bool mba_sc)
+{
+	struct rdt_domain *dom = &hw_dom->resctrl;
+	u32 *dc = mba_sc ? hw_dom->mbps_val : hw_dom->ctrl_val;
+
+	if (cfg->new_ctrl != dc[closid]) {
+		cpumask_set_cpu(cpumask_any(&dom->cpu_mask), cpu_mask);
+		dc[closid] = cfg->new_ctrl;
+	}
+
+	cfg->have_new_ctrl = false;
+}
+
 int update_domains(struct rdt_resource *r, int closid)
 {
+	struct resctrl_staged_config *cfg;
 	struct rdt_hw_domain *hw_dom;
 	struct msr_param msr_param;
 	cpumask_var_t cpu_mask;
 	struct rdt_domain *d;
 	bool mba_sc;
-	u32 *dc;
-	int cpu;
+	int cpu, i;
 
 	if (!zalloc_cpumask_var(&cpu_mask, GFP_KERNEL))
 		return -ENOMEM;
@@ -260,10 +279,12 @@ int update_domains(struct rdt_resource *r, int closid)
 	mba_sc = is_mba_sc(r);
 	list_for_each_entry(d, &r->domains, list) {
 		hw_dom = resctrl_to_arch_dom(d);
-		dc = !mba_sc ? hw_dom->ctrl_val : hw_dom->mbps_val;
-		if (d->have_new_ctrl && d->new_ctrl != dc[closid]) {
-			cpumask_set_cpu(cpumask_any(&d->cpu_mask), cpu_mask);
-			dc[closid] = d->new_ctrl;
+		for (i = 0; i < ARRAY_SIZE(d->staged_config); i++) {
+			cfg = &hw_dom->resctrl.staged_config[i];
+			if (!cfg->have_new_ctrl)
+				continue;
+
+			apply_config(hw_dom, cfg, closid, cpu_mask, mba_sc);
 		}
 	}
 
@@ -338,7 +359,7 @@ ssize_t rdtgroup_schemata_write(struct kernfs_open_file *of,
 
 	list_for_each_entry(s, &resctrl_all_schema, list) {
 		list_for_each_entry(dom, &s->res->domains, list)
-			dom->have_new_ctrl = false;
+			memset(dom->staged_config, 0, sizeof(dom->staged_config));
 	}
 
 	while ((tok = strsep(&buf, "\n")) != NULL) {
arch/x86/kernel/cpu/resctrl/rdtgroup.c +13 -9
index 48f4d6783647..c307170ee45f 100644
@@ -2748,6 +2748,7 @@ static int __init_one_rdt_domain(struct rdt_domain *d, struct resctrl_schema *s,
 				 u32 closid)
 {
 	struct rdt_resource *r_cdp = NULL;
+	struct resctrl_staged_config *cfg;
 	struct rdt_domain *d_cdp = NULL;
 	struct rdt_resource *r = s->res;
 	u32 used_b = 0, unused_b = 0;
@@ -2757,8 +2758,9 @@ static int __init_one_rdt_domain(struct rdt_domain *d, struct resctrl_schema *s,
 	int i;
 
 	rdt_cdp_peer_get(r, d, &r_cdp, &d_cdp);
-	d->have_new_ctrl = false;
-	d->new_ctrl = r->cache.shareable_bits;
+	cfg = &d->staged_config[0];
+	cfg->have_new_ctrl = false;
+	cfg->new_ctrl = r->cache.shareable_bits;
 	used_b = r->cache.shareable_bits;
 	ctrl = resctrl_to_arch_dom(d)->ctrl_val;
 	for (i = 0; i < closids_supported(); i++, ctrl++) {
@@ -2782,29 +2784,29 @@ static int __init_one_rdt_domain(struct rdt_domain *d, struct resctrl_schema *s,
 				peer_ctl = 0;
 			used_b |= *ctrl | peer_ctl;
 			if (mode == RDT_MODE_SHAREABLE)
-				d->new_ctrl |= *ctrl | peer_ctl;
+				cfg->new_ctrl |= *ctrl | peer_ctl;
 		}
 	}
 	if (d->plr && d->plr->cbm > 0)
 		used_b |= d->plr->cbm;
 	unused_b = used_b ^ (BIT_MASK(r->cache.cbm_len) - 1);
 	unused_b &= BIT_MASK(r->cache.cbm_len) - 1;
-	d->new_ctrl |= unused_b;
+	cfg->new_ctrl |= unused_b;
 	/*
 	 * Force the initial CBM to be valid, user can
 	 * modify the CBM based on system availability.
 	 */
-	d->new_ctrl = cbm_ensure_valid(d->new_ctrl, r);
+	cfg->new_ctrl = cbm_ensure_valid(cfg->new_ctrl, r);
 	/*
 	 * Assign the u32 CBM to an unsigned long to ensure that
 	 * bitmap_weight() does not access out-of-bound memory.
 	 */
-	tmp_cbm = d->new_ctrl;
+	tmp_cbm = cfg->new_ctrl;
 	if (bitmap_weight(&tmp_cbm, r->cache.cbm_len) < r->cache.min_cbm_bits) {
 		rdt_last_cmd_printf("No space on %s:%d\n", s->name, d->id);
 		return -ENOSPC;
 	}
-	d->have_new_ctrl = true;
+	cfg->have_new_ctrl = true;
 
 	return 0;
 }
@@ -2836,11 +2838,13 @@ static int rdtgroup_init_cat(struct resctrl_schema *s, u32 closid)
 /* Initialize MBA resource with default values. */
 static void rdtgroup_init_mba(struct rdt_resource *r)
 {
+	struct resctrl_staged_config *cfg;
 	struct rdt_domain *d;
 
 	list_for_each_entry(d, &r->domains, list) {
-		d->new_ctrl = is_mba_sc(r) ? MBA_MAX_MBPS : r->default_ctrl;
-		d->have_new_ctrl = true;
+		cfg = &d->staged_config[0];
+		cfg->new_ctrl = is_mba_sc(r) ? MBA_MAX_MBPS : r->default_ctrl;
+		cfg->have_new_ctrl = true;
 	}
 }
 
include/linux/resctrl.h +12 -5
index 9f71f0238239..f1164bbb66c5 100644
@@ -26,13 +26,21 @@ enum resctrl_conf_type {
 	CDP_DATA,
 };
 
+/**
+ * struct resctrl_staged_config - parsed configuration to be applied
+ * @new_ctrl:		new ctrl value to be loaded
+ * @have_new_ctrl:	did user provide new_ctrl for this domain
+ */
+struct resctrl_staged_config {
+	u32			new_ctrl;
+	bool			have_new_ctrl;
+};
+
 /**
  * struct rdt_domain - group of cpus sharing an RDT resource
  * @list:		all instances of this resource
  * @id:			unique id for this instance
  * @cpu_mask:		which cpus share this resource
- * @new_ctrl:		new ctrl value to be loaded
- * @have_new_ctrl:	did user provide new_ctrl for this domain
  * @rmid_busy_llc:	bitmap of which limbo RMIDs are above threshold
  * @mbm_total:		saved state for MBM total bandwidth
  * @mbm_local:		saved state for MBM local bandwidth
@@ -41,15 +49,13 @@ enum resctrl_conf_type {
  * @mbm_work_cpu:	worker cpu for MBM h/w counters
  * @cqm_work_cpu:	worker cpu for CQM h/w counters
  * @plr:		pseudo-locked region (if any) associated with domain
+ * @staged_config:	parsed configuration to be applied
  */
 struct rdt_domain {
 	struct list_head		list;
 	int				id;
 	struct cpumask			cpu_mask;
 
-	u32				new_ctrl;
-	bool				have_new_ctrl;
-
 	unsigned long			*rmid_busy_llc;
 	struct mbm_state		*mbm_total;
 	struct mbm_state		*mbm_local;
@@ -59,6 +65,7 @@ struct rdt_domain {
 	int				cqm_work_cpu;
 
 	struct pseudo_lock_region	*plr;
+	struct resctrl_staged_config	staged_config[1];
 };
 
 /**
Signature · 1 line
2.28.0
maintainer
#31 lore raw

Re: [PATCH 11/24] x86/resctrl: Group staged configuration into a separate struct

Hi James,

Quoted 10 lines · James Morse

Arm's MPAM may have surprisingly large bitmaps for its cache portions as the architecture allows up to 4K portions. The size exposed via resctrl may not be the same, some scaling may occur.

The values written to hardware may be unlike the values received from resctrl, e.g. MBA percentages may be backed by a bitmap, or a maximum value that isn't a percentage.

Today resctrl's ctrlval arrays are written to directly by the

If using a cryptic word like "ctrlval" it would be easier to understand what is meant if it matches the variable in the code, "ctrl_val".

Quoted 1 line

resctrl filesystem code. e.g. apply_config(). This is a problem

This sentence starts with "Today" implying what code does before this change but the example function, apply_config() is introduced in this patch.

Quoted 5 lines

if scaling or conversion is needed by the architecture.

The arch code should own the ctrlval array (to allow scaling and conversion), and should only need a single copy of the array for the values currently applied in hardware.

ok, but that is the case, no?

Quoted 3 lines

Move the new_ctrl bitmap value and flag into a struct for staged configuration changes. This is created as an array to allow one per type

This is a bit cryptic as the reader may not know while reading this commit message what "new_ctrl" is or where it is currently hosted.

Quoted 15 lines

of configuration. Today there is only one element in the array, but eventually resctrl will use the array slots for CODE/DATA/BOTH to detect a duplicate schema being written.

---
  arch/x86/kernel/cpu/resctrl/ctrlmondata.c | 49 ++++++++++++++++-------
  arch/x86/kernel/cpu/resctrl/rdtgroup.c    | 22 +++++-----
  include/linux/resctrl.h                   | 17 +++++---
  3 files changed, 60 insertions(+), 28 deletions(-)
arch/x86/kernel/cpu/resctrl/ctrlmondata.c +0 -0
index 28d69c78c29e..0c95ed83eb05 100644

...

Quoted 7 lines
arch/x86/kernel/cpu/resctrl/ctrlmondata.c +3 -0
@@ -240,15 +244,30 @@ static int parse_line(char *line, struct resctrl_schema *s,
  	return -EINVAL;
  }
 
+static void apply_config(struct rdt_hw_domain *hw_dom,
+			 struct resctrl_staged_config *cfg, int closid,
+			 cpumask_var_t cpu_mask, bool mba_sc)

arch/x86/kernel/cpu/resctrl/ctrlmondata.c static int parse_line(char *line, struct resctrl_schema *s, · line 259

Quoted 10 lines
arch/x86/kernel/cpu/resctrl/ctrlmondata.c +10 -0
+{
+	struct rdt_domain *dom = &hw_dom->resctrl;
+	u32 *dc = mba_sc ? hw_dom->mbps_val : hw_dom->ctrl_val;
+
+	if (cfg->new_ctrl != dc[closid]) {
+		cpumask_set_cpu(cpumask_any(&dom->cpu_mask), cpu_mask);
+		dc[closid] = cfg->new_ctrl;
+	}
+
+	cfg->have_new_ctrl = false;

Why is this necessary?

Quoted 17 lines
arch/x86/kernel/cpu/resctrl/ctrlmondata.c +4 -2
+}
+
  int update_domains(struct rdt_resource *r, int closid)
  {
+	struct resctrl_staged_config *cfg;
  	struct rdt_hw_domain *hw_dom;
  	struct msr_param msr_param;
  	cpumask_var_t cpu_mask;
  	struct rdt_domain *d;
  	bool mba_sc;
-	u32 *dc;
-	int cpu;
+	int cpu, i;
 
  	if (!zalloc_cpumask_var(&cpu_mask, GFP_KERNEL))
  		return -ENOMEM;

arch/x86/kernel/cpu/resctrl/ctrlmondata.c int update_domains(struct rdt_resource *r, int closid) · line 282

Quoted 8 lines
arch/x86/kernel/cpu/resctrl/ctrlmondata.c +1 -4
@@ -260,10 +279,12 @@ int update_domains(struct rdt_resource *r, int closid)
  	mba_sc = is_mba_sc(r);
  	list_for_each_entry(d, &r->domains, list) {
  		hw_dom = resctrl_to_arch_dom(d);
-		dc = !mba_sc ? hw_dom->ctrl_val : hw_dom->mbps_val;
-		if (d->have_new_ctrl && d->new_ctrl != dc[closid]) {
-			cpumask_set_cpu(cpumask_any(&d->cpu_mask), cpu_mask);
-			dc[closid] = d->new_ctrl;
+		for (i = 0; i < ARRAY_SIZE(d->staged_config); i++) {

I understand it may make later patches easier but it seems too early to introduce this loop because apply_config() does not seem to be ready for it yet (it would just keep overwriting a closid's config).

Quoted 17 lines
arch/x86/kernel/cpu/resctrl/ctrlmondata.c +6 -1
+			cfg = &hw_dom->resctrl.staged_config[i];
+			if (!cfg->have_new_ctrl)
+				continue;
+
+			apply_config(hw_dom, cfg, closid, cpu_mask, mba_sc);
  		}
  	}
 
@@ -338,7 +359,7 @@ ssize_t rdtgroup_schemata_write(struct kernfs_open_file *of,
 
  	list_for_each_entry(s, &resctrl_all_schema, list) {
  		list_for_each_entry(dom, &s->res->domains, list)
-			dom->have_new_ctrl = false;
+			memset(dom->staged_config, 0, sizeof(dom->staged_config));
  	}
 
  	while ((tok = strsep(&buf, "\n")) != NULL) {

...

include/linux/resctrl.h enum resctrl_conf_type { · line 32

Quoted 12 lines
include/linux/resctrl.h +4 -0
index 9f71f0238239..f1164bbb66c5 100644
@@ -26,13 +26,21 @@ enum resctrl_conf_type {
  	CDP_DATA,
  };
 
+/**
+ * struct resctrl_staged_config - parsed configuration to be applied
+ * @new_ctrl:		new ctrl value to be loaded
+ * @have_new_ctrl:	did user provide new_ctrl for this domain

The "for this domain" in this description is no longer appropriate after the copy.

Quoted 35 lines
include/linux/resctrl.h +7 -5
+ */
+struct resctrl_staged_config {
+	u32			new_ctrl;
+	bool			have_new_ctrl;
+};
+
  /**
   * struct rdt_domain - group of cpus sharing an RDT resource
   * @list:		all instances of this resource
   * @id:			unique id for this instance
   * @cpu_mask:		which cpus share this resource
- * @new_ctrl:		new ctrl value to be loaded
- * @have_new_ctrl:	did user provide new_ctrl for this domain
   * @rmid_busy_llc:	bitmap of which limbo RMIDs are above threshold
   * @mbm_total:		saved state for MBM total bandwidth
   * @mbm_local:		saved state for MBM local bandwidth
@@ -41,15 +49,13 @@ enum resctrl_conf_type {
   * @mbm_work_cpu:	worker cpu for MBM h/w counters
   * @cqm_work_cpu:	worker cpu for CQM h/w counters
   * @plr:		pseudo-locked region (if any) associated with domain
+ * @staged_config:	parsed configuration to be applied
   */
  struct rdt_domain {
  	struct list_head		list;
  	int				id;
  	struct cpumask			cpu_mask;
 
-	u32				new_ctrl;
-	bool				have_new_ctrl;
-
  	unsigned long			*rmid_busy_llc;
  	struct mbm_state		*mbm_total;
  	struct mbm_state		*mbm_local;

include/linux/resctrl.h struct rdt_domain { · line 71

Quoted 7 lines
include/linux/resctrl.h +1 -0
@@ -59,6 +65,7 @@ struct rdt_domain {
  	int				cqm_work_cpu;
 
  	struct pseudo_lock_region	*plr;
+	struct resctrl_staged_config	staged_config[1];
  };
 
  /**

Reinette

author reviewer
#32 lore raw

Re: [PATCH 11/24] x86/resctrl: Group staged configuration into a separate struct

Hi Reinette,

Quoted 6 lines · Reinette Chatre
Quoted 5 lines · James Morse

Arm's MPAM may have surprisingly large bitmaps for its cache portions as the architecture allows up to 4K portions. The size exposed via resctrl may not be the same, some scaling may occur.

Quoted 8 lines
Quoted 5 lines

The values written to hardware may be unlike the values received from resctrl, e.g. MBA percentages may be backed by a bitmap, or a maximum value that isn't a percentage.

Today resctrl's ctrlval arrays are written to directly by the

If using a cryptic word like "ctrlval" it would be easier to understand what is meant if it matches the variable in the code, "ctrl_val".

I thought the non-underscore version was the preferred version, e.g:
setup_default_ctrlval(), domain_setup_ctrlval() and parse_ctrlval. I'll switch to the underscore version for things other than functions if you think its clearer.

Quoted 4 lines
Quoted 1 line

resctrl filesystem code. e.g. apply_config(). This is a problem

This sentence starts with "Today" implying what code does before this change but the example function, apply_config() is introduced in this patch.

I don't follow the problem here, 'today' refers to what the code does before the patch is applied. "Before this patch" would make me unpopular, I'll try 'previously'.

Quoted 7 lines
Quoted 5 lines

if scaling or conversion is needed by the architecture.

The arch code should own the ctrlval array (to allow scaling and conversion), and should only need a single copy of the array for the values currently applied in hardware.

ok, but that is the case, no?

Its true for x86. But whether its true for MPAM depends on which side of the divide this thing lands as the value from user-space may be different to what gets written to hardware. If the filesystem code owned the list of values, there would need to be two copies to allow MPAM to restore the values in hardware when CPUs come online.

(in particular, the MBA percentage control can be emulated with MPAMs bitmap or fractional min/max, the driver has to choose at startup).

I'll try and bundle this as a clearer explanation into the commit message.

Quoted 5 lines
Quoted 2 lines

Move the new_ctrl bitmap value and flag into a struct for staged configuration changes. This is created as an array to allow one per type

This is a bit cryptic as the reader may not know while reading this commit message what "new_ctrl" is or where it is currently hosted.

Sure, I'll add more explanation of the current code.

Quoted 23 lines
Quoted 10 lines

of configuration. Today there is only one element in the array, but eventually resctrl will use the array slots for CODE/DATA/BOTH to detect a duplicate schema being written.

arch/x86/kernel/cpu/resctrl/ctrlmondata.c +0 -0
index 28d69c78c29e..0c95ed83eb05 100644

...

Quoted 10 lines
arch/x86/kernel/cpu/resctrl/ctrlmondata.c +6 -0
@@ -240,15 +244,30 @@ static int parse_line(char *line, struct resctrl_schema *s,
       return -EINVAL;
   }
   +static void apply_config(struct rdt_hw_domain *hw_dom,
+             struct resctrl_staged_config *cfg, int closid,
+             cpumask_var_t cpu_mask, bool mba_sc)
+{
+    struct rdt_domain *dom = &hw_dom->resctrl;
+    u32 *dc = mba_sc ? hw_dom->mbps_val : hw_dom->ctrl_val;
+
Quoted 8 lines

arch/x86/kernel/cpu/resctrl/ctrlmondata.c static int parse_line(char *line, struct resctrl_schema *s, · line 259

Quoted 6 lines
arch/x86/kernel/cpu/resctrl/ctrlmondata.c +6 -0
+    if (cfg->new_ctrl != dc[closid]) {
+        cpumask_set_cpu(cpumask_any(&dom->cpu_mask), cpu_mask);
+        dc[closid] = cfg->new_ctrl;
+    }
+
+    cfg->have_new_ctrl = false;

Why is this necessary?

(hmm, its ended up in the wrong patch, but:) This was to ensure that once the resources are merged, all the work for applying configuration changes is done by the first IPI, ensuring if update_domains() is called for a second schema with the same resource, it finds no new work to do.
But without this, the empty_bitmap check would still catch it. I'll remove it.

Quoted 20 lines
Quoted 20 lines
arch/x86/kernel/cpu/resctrl/ctrlmondata.c +4 -3
+}
+
   int update_domains(struct rdt_resource *r, int closid)
   {
+    struct resctrl_staged_config *cfg;
       struct rdt_hw_domain *hw_dom;
       struct msr_param msr_param;
       cpumask_var_t cpu_mask;
       struct rdt_domain *d;
       bool mba_sc;
-    u32 *dc;
-    int cpu;
+    int cpu, i;
         if (!zalloc_cpumask_var(&cpu_mask, GFP_KERNEL))
           return -ENOMEM;
@@ -260,10 +279,12 @@ int update_domains(struct rdt_resource *r, int closid)
       mba_sc = is_mba_sc(r);
       list_for_each_entry(d, &r->domains, list) {
           hw_dom = resctrl_to_arch_dom(d);
-        dc = !mba_sc ? hw_dom->ctrl_val : hw_dom->mbps_val;
Quoted 8 lines

arch/x86/kernel/cpu/resctrl/ctrlmondata.c int update_domains(struct rdt_resource *r, int closid) · line 282

Quoted 4 lines
arch/x86/kernel/cpu/resctrl/ctrlmondata.c +1 -3
-        if (d->have_new_ctrl && d->new_ctrl != dc[closid]) {
-            cpumask_set_cpu(cpumask_any(&d->cpu_mask), cpu_mask);
-            dc[closid] = d->new_ctrl;
+        for (i = 0; i < ARRAY_SIZE(d->staged_config); i++) {

I understand it may make later patches easier but it seems too early to introduce this loop because apply_config() does not seem to be ready for it yet (it would just keep overwriting a closid's config).

Grouping these values is needed because they become a set, one per type of configuration. This ended up here because its looping over the set, even its only got one entry at the moment.

Sure, I'll move it into a later patch. Presumably you want the array indexing to move too.

Quoted 17 lines
Quoted 14 lines
arch/x86/kernel/cpu/resctrl/ctrlmondata.c +5 -0
+            cfg = &hw_dom->resctrl.staged_config[i];
+            if (!cfg->have_new_ctrl)
+                continue;
+
+            apply_config(hw_dom, cfg, closid, cpu_mask, mba_sc);
           }
       }
   @@ -338,7 +359,7 @@ ssize_t rdtgroup_schemata_write(struct kernfs_open_file *of,

        list_for_each_entry(s, &resctrl_all_schema, list) { list_for_each_entry(dom, &s->res->domains, list)
-            dom->have_new_ctrl = false;
+            memset(dom->staged_config, 0, sizeof(dom->staged_config)); }
        while ((tok = strsep(&buf, "\n")) != NULL) {

...

Quoted 13 lines
Quoted 11 lines
include/linux/resctrl.h +0 -0
index 9f71f0238239..f1164bbb66c5 100644
@@ -26,13 +26,21 @@ enum resctrl_conf_type {
       CDP_DATA,
   };

  +/**
+ * struct resctrl_staged_config - parsed configuration to be applied
+ * @new_ctrl:        new ctrl value to be loaded
+ * @have_new_ctrl:    did user provide new_ctrl for this domain

The "for this domain" in this description is no longer appropriate after the copy.

This structure is still embedded in struct rdt_domain, but I'll rephrase it.

Thanks,

James

Quoted 11 lines
Quoted 11 lines

+ */

include/linux/resctrl.h +5 -0
+struct resctrl_staged_config {
+    u32            new_ctrl;
+    bool            have_new_ctrl;
+};
@@ -59,6 +65,7 @@ struct rdt_domain {
       int                cqm_work_cpu;
         struct pseudo_lock_region    *plr;
+    struct resctrl_staged_config    staged_config[1];
   };
author reviewer
#33 lore raw

PATCH 12/24 x86/resctrl: Add closid to the staged config

In reply to James Morse

Once the L2/L2CODE/L2DATA resources are merged, there may be two configurations staged for one resource when CDP is enabled. The closid should always be passed with the type of configuration to the arch code.

Because update_domains() will eventually apply a set of configurations, it should take the closid from the same place, so they pair up.

Move the closid to be a staged parameter.

---
 arch/x86/kernel/cpu/resctrl/ctrlmondata.c | 10 ++++++----
 arch/x86/kernel/cpu/resctrl/rdtgroup.c    |  6 ++++--
 include/linux/resctrl.h                   |  2 ++
 3 files changed, 12 insertions(+), 6 deletions(-)
arch/x86/kernel/cpu/resctrl/ctrlmondata.c +6 -4
index 0c95ed83eb05..b107c0202cfb 100644
@@ -72,6 +72,7 @@ int parse_bw(struct rdt_parse_data *data, struct resctrl_schema *s,
 	if (!bw_validate(data->buf, &bw_val, r))
 		return -EINVAL;
 	cfg->new_ctrl = bw_val;
+	cfg->closid = data->rdtgrp->closid;
 	cfg->have_new_ctrl = true;
 
 	return 0;
@@ -178,6 +179,7 @@ int parse_cbm(struct rdt_parse_data *data, struct resctrl_schema *s,
 	}
 
 	cfg->new_ctrl = cbm_val;
+	cfg->closid = data->rdtgrp->closid;
 	cfg->have_new_ctrl = true;
 
 	return 0;
@@ -245,15 +247,15 @@ static int parse_line(char *line, struct resctrl_schema *s,
 }
 
 static void apply_config(struct rdt_hw_domain *hw_dom,
-			 struct resctrl_staged_config *cfg, int closid,
+			 struct resctrl_staged_config *cfg,
 			 cpumask_var_t cpu_mask, bool mba_sc)
 {
 	struct rdt_domain *dom = &hw_dom->resctrl;
 	u32 *dc = mba_sc ? hw_dom->mbps_val : hw_dom->ctrl_val;
 
-	if (cfg->new_ctrl != dc[closid]) {
+	if (cfg->new_ctrl != dc[cfg->closid]) {
 		cpumask_set_cpu(cpumask_any(&dom->cpu_mask), cpu_mask);
-		dc[closid] = cfg->new_ctrl;
+		dc[cfg->closid] = cfg->new_ctrl;
 	}
 
 	cfg->have_new_ctrl = false;
@@ -284,7 +286,7 @@ int update_domains(struct rdt_resource *r, int closid)
 			if (!cfg->have_new_ctrl)
 				continue;
 
-			apply_config(hw_dom, cfg, closid, cpu_mask, mba_sc);
+			apply_config(hw_dom, cfg, cpu_mask, mba_sc);
 		}
 	}
 
arch/x86/kernel/cpu/resctrl/rdtgroup.c +4 -2
index c307170ee45f..1092631ac0b3 100644
@@ -2806,6 +2806,7 @@ static int __init_one_rdt_domain(struct rdt_domain *d, struct resctrl_schema *s,
 		rdt_last_cmd_printf("No space on %s:%d\n", s->name, d->id);
 		return -ENOSPC;
 	}
+	cfg->closid = closid;
 	cfg->have_new_ctrl = true;
 
 	return 0;
@@ -2836,7 +2837,7 @@ static int rdtgroup_init_cat(struct resctrl_schema *s, u32 closid)
 }
 
 /* Initialize MBA resource with default values. */
-static void rdtgroup_init_mba(struct rdt_resource *r)
+static void rdtgroup_init_mba(struct rdt_resource *r, u32 closid)
 {
 	struct resctrl_staged_config *cfg;
 	struct rdt_domain *d;
@@ -2844,6 +2845,7 @@ static void rdtgroup_init_mba(struct rdt_resource *r)
 	list_for_each_entry(d, &r->domains, list) {
 		cfg = &d->staged_config[0];
 		cfg->new_ctrl = is_mba_sc(r) ? MBA_MAX_MBPS : r->default_ctrl;
+		cfg->closid = closid;
 		cfg->have_new_ctrl = true;
 	}
 }
@@ -2860,7 +2862,7 @@ static int rdtgroup_init_alloc(struct rdtgroup *rdtgrp)
 	list_for_each_entry(s, &resctrl_all_schema, list) {
 		r = s->res;
 		if (r->rid == RDT_RESOURCE_MBA) {
-			rdtgroup_init_mba(r);
+			rdtgroup_init_mba(r, rdtgrp->closid);
 		} else {
 			ret = rdtgroup_init_cat(s, rdtgrp->closid);
 			if (ret < 0)
include/linux/resctrl.h +2 -0
index f1164bbb66c5..695247c08ba3 100644
@@ -28,10 +28,12 @@ enum resctrl_conf_type {
 
 /**
  * struct resctrl_staged_config - parsed configuration to be applied
+ * @closid:		The closid the new configuration applies to
  * @new_ctrl:		new ctrl value to be loaded
  * @have_new_ctrl:	did user provide new_ctrl for this domain
  */
 struct resctrl_staged_config {
+	u32			closid;
 	u32			new_ctrl;
 	bool			have_new_ctrl;
 };
Signature · 1 line
2.28.0
maintainer
#34 lore raw

Re: [PATCH 12/24] x86/resctrl: Add closid to the staged config

Hi James,

Quoted 9 lines · James Morse

Once the L2/L2CODE/L2DATA resources are merged, there may be two configurations staged for one resource when CDP is enabled. The closid should always be passed with the type of configuration to the arch code.

Because update_domains() will eventually apply a set of configurations, it should take the closid from the same place, so they pair up.

Move the closid to be a staged parameter.

Move implies that it is taken from one location and added to another. This seems like a copy instead?

Quoted 21 lines
---
  arch/x86/kernel/cpu/resctrl/ctrlmondata.c | 10 ++++++----
  arch/x86/kernel/cpu/resctrl/rdtgroup.c    |  6 ++++--
  include/linux/resctrl.h                   |  2 ++
  3 files changed, 12 insertions(+), 6 deletions(-)
arch/x86/kernel/cpu/resctrl/ctrlmondata.c +1 -0
index 0c95ed83eb05..b107c0202cfb 100644
@@ -72,6 +72,7 @@ int parse_bw(struct rdt_parse_data *data, struct resctrl_schema *s,
  	if (!bw_validate(data->buf, &bw_val, r))
  		return -EINVAL;
  	cfg->new_ctrl = bw_val;
+	cfg->closid = data->rdtgrp->closid;
  	cfg->have_new_ctrl = true;
 
  	return 0;

arch/x86/kernel/cpu/resctrl/ctrlmondata.c int parse_cbm(struct rdt_parse_data *data, struct resctrl_schema *s, · line 184

Quoted 6 lines
arch/x86/kernel/cpu/resctrl/ctrlmondata.c +1 -0
@@ -178,6 +179,7 @@ int parse_cbm(struct rdt_parse_data *data, struct resctrl_schema *s,
  	}
 
  	cfg->new_ctrl = cbm_val;
+	cfg->closid = data->rdtgrp->closid;
  	cfg->have_new_ctrl = true;
 

rdtgrp is already available so it could just be:
cfg->closid = rdtgrp->closid?

Quoted 21 lines
arch/x86/kernel/cpu/resctrl/ctrlmondata.c +3 -3
  	return 0;
@@ -245,15 +247,15 @@ static int parse_line(char *line, struct resctrl_schema *s,
  }
 
  static void apply_config(struct rdt_hw_domain *hw_dom,
-			 struct resctrl_staged_config *cfg, int closid,
+			 struct resctrl_staged_config *cfg,
  			 cpumask_var_t cpu_mask, bool mba_sc)
  {
  	struct rdt_domain *dom = &hw_dom->resctrl;
  	u32 *dc = mba_sc ? hw_dom->mbps_val : hw_dom->ctrl_val;
 
-	if (cfg->new_ctrl != dc[closid]) {
+	if (cfg->new_ctrl != dc[cfg->closid]) {
  		cpumask_set_cpu(cpumask_any(&dom->cpu_mask), cpu_mask);
-		dc[closid] = cfg->new_ctrl;
+		dc[cfg->closid] = cfg->new_ctrl;
  	}
 
  	cfg->have_new_ctrl = false;

arch/x86/kernel/cpu/resctrl/ctrlmondata.c int update_domains(struct rdt_resource *r, int closid) · line 291

Quoted 7 lines
arch/x86/kernel/cpu/resctrl/ctrlmondata.c +1 -1
@@ -284,7 +286,7 @@ int update_domains(struct rdt_resource *r, int closid)
  			if (!cfg->have_new_ctrl)
  				continue;
 
-			apply_config(hw_dom, cfg, closid, cpu_mask, mba_sc);
+			apply_config(hw_dom, cfg, cpu_mask, mba_sc);
  		}
  	}

It is not clear to me that storing the closid in the staged config is necessary. A closid is associated with a resource group so when the user writes to the schemata file all configurations would be (from the resource group perspective) for the same closid. This is the value provided here to update domains. Looking ahead in this series this closid is later used to compute the index (get_config_index()) that would use as input cfg->closid, but that cfg->closid would be identical for all resources/staged configs and then the new index would be computed based on the resource type. Having the closid in the staged config thus does not seem to be necessary, it could remain as this function parameter and be used for all staged configs?

include/linux/resctrl.h enum resctrl_conf_type { · line 31

Quoted 10 lines
include/linux/resctrl.h +1 -0
index f1164bbb66c5..695247c08ba3 100644
@@ -28,10 +28,12 @@ enum resctrl_conf_type {
 
  /**
   * struct resctrl_staged_config - parsed configuration to be applied
+ * @closid:		The closid the new configuration applies to

Please be consistent on how descriptions are started with or without capital letters.

include/linux/resctrl.h enum resctrl_conf_type { · line 39

Quoted 9 lines
include/linux/resctrl.h +1 -0
   * @new_ctrl:		new ctrl value to be loaded
   * @have_new_ctrl:	did user provide new_ctrl for this domain
   */
  struct resctrl_staged_config {
+	u32			closid;
  	u32			new_ctrl;
  	bool			have_new_ctrl;
  };

Reinette

author reviewer
#35 lore raw

Re: [PATCH 12/24] x86/resctrl: Add closid to the staged config

Hi Reinette,

Quoted 6 lines · Reinette Chatre
Quoted 5 lines · James Morse

Once the L2/L2CODE/L2DATA resources are merged, there may be two configurations staged for one resource when CDP is enabled. The closid should always be passed with the type of configuration to the arch code.

Quoted 7 lines
Quoted 4 lines

Because update_domains() will eventually apply a set of configurations, it should take the closid from the same place, so they pair up.

Move the closid to be a staged parameter.

Move implies that it is taken from one location and added to another. This seems like a copy instead?

I'll rephrase it.

Quoted 12 lines
Quoted 12 lines
arch/x86/kernel/cpu/resctrl/ctrlmondata.c +1 -0
index 0c95ed83eb05..b107c0202cfb 100644
@@ -72,6 +72,7 @@ int parse_bw(struct rdt_parse_data *data, struct resctrl_schema *s,
       if (!bw_validate(data->buf, &bw_val, r))
           return -EINVAL;
       cfg->new_ctrl = bw_val;
+    cfg->closid = data->rdtgrp->closid;
       cfg->have_new_ctrl = true;
         return 0;
Quoted 8 lines

arch/x86/kernel/cpu/resctrl/ctrlmondata.c int parse_cbm(struct rdt_parse_data *data, struct resctrl_schema *s, · line 182

Quoted 5 lines
arch/x86/kernel/cpu/resctrl/ctrlmondata.c +1 -0
@@ -178,6 +179,7 @@ int parse_cbm(struct rdt_parse_data *data, struct resctrl_schema *s,
       }
         cfg->new_ctrl = cbm_val;
+    cfg->closid = data->rdtgrp->closid;
       cfg->have_new_ctrl = true;

rdtgrp is already available so it could just be:
cfg->closid = rdtgrp->closid?

Yes, this is just trying to be identical to the earlier version. I'll change it.

Quoted 26 lines
Quoted 18 lines
arch/x86/kernel/cpu/resctrl/ctrlmondata.c +3 -2
       return 0;
@@ -245,15 +247,15 @@ static int parse_line(char *line, struct resctrl_schema *s,
   }
     static void apply_config(struct rdt_hw_domain *hw_dom,
-             struct resctrl_staged_config *cfg, int closid,
+             struct resctrl_staged_config *cfg,
                cpumask_var_t cpu_mask, bool mba_sc)
   {
       struct rdt_domain *dom = &hw_dom->resctrl;
       u32 *dc = mba_sc ? hw_dom->mbps_val : hw_dom->ctrl_val;
   -    if (cfg->new_ctrl != dc[closid]) {
+    if (cfg->new_ctrl != dc[cfg->closid]) {
           cpumask_set_cpu(cpumask_any(&dom->cpu_mask), cpu_mask);
-        dc[closid] = cfg->new_ctrl;
+        dc[cfg->closid] = cfg->new_ctrl;
       }
         cfg->have_new_ctrl = false;

arch/x86/kernel/cpu/resctrl/ctrlmondata.c int update_domains(struct rdt_resource *r, int closid) · line 291

Quoted 6 lines
arch/x86/kernel/cpu/resctrl/ctrlmondata.c +1 -0
@@ -284,7 +286,7 @@ int update_domains(struct rdt_resource *r, int closid)
               if (!cfg->have_new_ctrl)
                   continue;
   -            apply_config(hw_dom, cfg, closid, cpu_mask, mba_sc);
+            apply_config(hw_dom, cfg, cpu_mask, mba_sc);
           }
       }

It is not clear to me that storing the closid in the staged config is necessary. A closid

Quoted 8 lines

is associated with a resource group so when the user writes to the schemata file all configurations would be (from the resource group perspective) for the same closid. This is the value provided here to update domains. Looking ahead in this series this closid is later used to compute the index (get_config_index()) that would use as input cfg->closid, but that cfg->closid would be identical for all resources/staged configs and then the new index would be computed based on the resource type. Having the closid in the staged config thus does not seem to be necessary, it could remain as this function parameter and be used for all staged configs?

Yes, when emulating CDP with MPAM, these can be two unrelated numbers, but it looks like I lost track of the 'its always going to be the same value' for each invocation through the filesystem.

An earlier version tried to push separate code/data closid values further into the resctrl code, but I thought it was getting too confusing.
The x86 CDP behaviour is either 'configuration is twice the size' or 'an odd even pair of closid' depending on how you look at it. For MPAM the equivalent feature is an (arbitary!) pair of closid/partid.

Yes its not necessary for resctrl, I'll see how much of this I can unpick!

Thanks,

James

author reviewer
#36 lore raw

PATCH 13/24 x86/resctrl: Allow different CODE/DATA configurations to be staged

In reply to James Morse

Now that the configuration is staged via an array, allow resctrl to stage more than configuration at a time for a single resource and closid.

To detect that the same schema is being specified twice when the schemata file is written, the same slot in the staged_configuration array must be used for each schema. Use the conf_type enum directly as an index.

---
 arch/x86/kernel/cpu/resctrl/ctrlmondata.c | 16 ++++++++++------
 arch/x86/kernel/cpu/resctrl/rdtgroup.c    |  5 +++--
 include/linux/resctrl.h                   |  4 +++-
 3 files changed, 16 insertions(+), 9 deletions(-)
arch/x86/kernel/cpu/resctrl/ctrlmondata.c +10 -6
index b107c0202cfb..f7152c7fdc1b 100644
@@ -60,10 +60,11 @@ static bool bw_validate(char *buf, unsigned long *data, struct rdt_resource *r)
 int parse_bw(struct rdt_parse_data *data, struct resctrl_schema *s,
 	     struct rdt_domain *d)
 {
-	struct resctrl_staged_config *cfg = &d->staged_config[0];
+	struct resctrl_staged_config *cfg;
 	struct rdt_resource *r = s->res;
 	unsigned long bw_val;
 
+	cfg = &d->staged_config[s->conf_type];
 	if (cfg->have_new_ctrl) {
 		rdt_last_cmd_printf("Duplicate domain %d\n", d->id);
 		return -EINVAL;
@@ -131,11 +132,12 @@ static bool cbm_validate(char *buf, u32 *data, struct rdt_resource *r)
 int parse_cbm(struct rdt_parse_data *data, struct resctrl_schema *s,
 	      struct rdt_domain *d)
 {
-	struct resctrl_staged_config *cfg = &d->staged_config[0];
 	struct rdtgroup *rdtgrp = data->rdtgrp;
+	struct resctrl_staged_config *cfg;
 	struct rdt_resource *r = s->res;
 	u32 cbm_val;
 
+	cfg = &d->staged_config[s->conf_type];
 	if (cfg->have_new_ctrl) {
 		rdt_last_cmd_printf("Duplicate domain %d\n", d->id);
 		return -EINVAL;
@@ -194,6 +196,7 @@ int parse_cbm(struct rdt_parse_data *data, struct resctrl_schema *s,
 static int parse_line(char *line, struct resctrl_schema *s,
 		      struct rdtgroup *rdtgrp)
 {
+	enum resctrl_conf_type t = s->conf_type;
 	struct resctrl_staged_config *cfg;
 	struct rdt_resource *r = s->res;
 	struct rdt_parse_data data;
@@ -225,7 +228,7 @@ static int parse_line(char *line, struct resctrl_schema *s,
 			if (r->parse_ctrlval(&data, s, d))
 				return -EINVAL;
 			if (rdtgrp->mode ==  RDT_MODE_PSEUDO_LOCKSETUP) {
-				cfg = &d->staged_config[0];
+				cfg = &d->staged_config[t];
 				/*
 				 * In pseudo-locking setup mode and just
 				 * parsed a valid CBM that should be
@@ -266,10 +269,11 @@ int update_domains(struct rdt_resource *r, int closid)
 	struct resctrl_staged_config *cfg;
 	struct rdt_hw_domain *hw_dom;
 	struct msr_param msr_param;
+	enum resctrl_conf_type t;
 	cpumask_var_t cpu_mask;
 	struct rdt_domain *d;
 	bool mba_sc;
-	int cpu, i;
+	int cpu;
 
 	if (!zalloc_cpumask_var(&cpu_mask, GFP_KERNEL))
 		return -ENOMEM;
@@ -281,8 +285,8 @@ int update_domains(struct rdt_resource *r, int closid)
 	mba_sc = is_mba_sc(r);
 	list_for_each_entry(d, &r->domains, list) {
 		hw_dom = resctrl_to_arch_dom(d);
-		for (i = 0; i < ARRAY_SIZE(d->staged_config); i++) {
-			cfg = &hw_dom->resctrl.staged_config[i];
+		for (t = 0; t < ARRAY_SIZE(d->staged_config); t++) {
+			cfg = &hw_dom->resctrl.staged_config[t];
 			if (!cfg->have_new_ctrl)
 				continue;
 
arch/x86/kernel/cpu/resctrl/rdtgroup.c +3 -2
index 1092631ac0b3..5eb14dc9c579 100644
@@ -2747,6 +2747,7 @@ static u32 cbm_ensure_valid(u32 _val, struct rdt_resource *r)
 static int __init_one_rdt_domain(struct rdt_domain *d, struct resctrl_schema *s,
 				 u32 closid)
 {
+	enum resctrl_conf_type t = s-> conf_type;
 	struct rdt_resource *r_cdp = NULL;
 	struct resctrl_staged_config *cfg;
 	struct rdt_domain *d_cdp = NULL;
@@ -2758,7 +2759,7 @@ static int __init_one_rdt_domain(struct rdt_domain *d, struct resctrl_schema *s,
 	int i;
 
 	rdt_cdp_peer_get(r, d, &r_cdp, &d_cdp);
-	cfg = &d->staged_config[0];
+	cfg = &d->staged_config[t];
 	cfg->have_new_ctrl = false;
 	cfg->new_ctrl = r->cache.shareable_bits;
 	used_b = r->cache.shareable_bits;
@@ -2843,7 +2844,7 @@ static void rdtgroup_init_mba(struct rdt_resource *r, u32 closid)
 	struct rdt_domain *d;
 
 	list_for_each_entry(d, &r->domains, list) {
-		cfg = &d->staged_config[0];
+		cfg = &d->staged_config[CDP_BOTH];
 		cfg->new_ctrl = is_mba_sc(r) ? MBA_MAX_MBPS : r->default_ctrl;
 		cfg->closid = closid;
 		cfg->have_new_ctrl = true;
include/linux/resctrl.h +3 -1
index 695247c08ba3..e33d6dfce8a1 100644
@@ -25,6 +25,8 @@ enum resctrl_conf_type {
 	CDP_CODE,
 	CDP_DATA,
 };
+#define NUM_CDP_TYPES	CDP_DATA + 1
+
 
 /**
  * struct resctrl_staged_config - parsed configuration to be applied
@@ -67,7 +69,7 @@ struct rdt_domain {
 	int				cqm_work_cpu;
 
 	struct pseudo_lock_region	*plr;
-	struct resctrl_staged_config	staged_config[1];
+	struct resctrl_staged_config	staged_config[NUM_CDP_TYPES];
 };
 
 /**
Signature · 1 line
2.28.0
maintainer
#37 lore raw

Re: [PATCH 13/24] x86/resctrl: Allow different CODE/DATA configurations to be staged

Hi James,

Quoted 2 lines · James Morse

Now that the configuration is staged via an array, allow resctrl to stage more than configuration at a time for a single resource and

s/more than/more than one/ ?

Quoted 17 lines

closid.

To detect that the same schema is being specified twice when the schemata file is written, the same slot in the staged_configuration array must be used for each schema. Use the conf_type enum directly as an index.

---
  arch/x86/kernel/cpu/resctrl/ctrlmondata.c | 16 ++++++++++------
  arch/x86/kernel/cpu/resctrl/rdtgroup.c    |  5 +++--
  include/linux/resctrl.h                   |  4 +++-
  3 files changed, 16 insertions(+), 9 deletions(-)
arch/x86/kernel/cpu/resctrl/ctrlmondata.c +0 -0
index b107c0202cfb..f7152c7fdc1b 100644

...

Quoted 14 lines
arch/x86/kernel/cpu/resctrl/ctrlmondata.c +2 -1
@@ -266,10 +269,11 @@ int update_domains(struct rdt_resource *r, int closid)
  	struct resctrl_staged_config *cfg;
  	struct rdt_hw_domain *hw_dom;
  	struct msr_param msr_param;
+	enum resctrl_conf_type t;
  	cpumask_var_t cpu_mask;
  	struct rdt_domain *d;
  	bool mba_sc;
-	int cpu, i;
+	int cpu;
 
  	if (!zalloc_cpumask_var(&cpu_mask, GFP_KERNEL))
  		return -ENOMEM;

arch/x86/kernel/cpu/resctrl/ctrlmondata.c int update_domains(struct rdt_resource *r, int closid) · line 288

Quoted 6 lines
arch/x86/kernel/cpu/resctrl/ctrlmondata.c +1 -2
@@ -281,8 +285,8 @@ int update_domains(struct rdt_resource *r, int closid)
  	mba_sc = is_mba_sc(r);
  	list_for_each_entry(d, &r->domains, list) {
  		hw_dom = resctrl_to_arch_dom(d);
-		for (i = 0; i < ARRAY_SIZE(d->staged_config); i++) {
-			cfg = &hw_dom->resctrl.staged_config[i];
+		for (t = 0; t < ARRAY_SIZE(d->staged_config); t++) {

Since t is now an enum, it may be easier to read if NUM_CDP_TYPES is used instead of ARRAY_SIZE?

arch/x86/kernel/cpu/resctrl/rdtgroup.c static u32 cbm_ensure_valid(u32 _val, struct rdt_resource *r) · line 2750

Quoted 13 lines
arch/x86/kernel/cpu/resctrl/ctrlmondata.c +1 -0
+			cfg = &hw_dom->resctrl.staged_config[t];
  			if (!cfg->have_new_ctrl)
  				continue;
 
arch/x86/kernel/cpu/resctrl/rdtgroup.c +1 -0
index 1092631ac0b3..5eb14dc9c579 100644
@@ -2747,6 +2747,7 @@ static u32 cbm_ensure_valid(u32 _val, struct rdt_resource *r)
  static int __init_one_rdt_domain(struct rdt_domain *d, struct resctrl_schema *s,
  				 u32 closid)
  {
+	enum resctrl_conf_type t = s-> conf_type;

Extra space above in "s-> conf_type".

Quoted 26 lines
arch/x86/kernel/cpu/resctrl/rdtgroup.c +2 -2
  	struct rdt_resource *r_cdp = NULL;
  	struct resctrl_staged_config *cfg;
  	struct rdt_domain *d_cdp = NULL;
@@ -2758,7 +2759,7 @@ static int __init_one_rdt_domain(struct rdt_domain *d, struct resctrl_schema *s,
  	int i;
 
  	rdt_cdp_peer_get(r, d, &r_cdp, &d_cdp);
-	cfg = &d->staged_config[0];
+	cfg = &d->staged_config[t];
  	cfg->have_new_ctrl = false;
  	cfg->new_ctrl = r->cache.shareable_bits;
  	used_b = r->cache.shareable_bits;
@@ -2843,7 +2844,7 @@ static void rdtgroup_init_mba(struct rdt_resource *r, u32 closid)
  	struct rdt_domain *d;
 
  	list_for_each_entry(d, &r->domains, list) {
-		cfg = &d->staged_config[0];
+		cfg = &d->staged_config[CDP_BOTH];
  		cfg->new_ctrl = is_mba_sc(r) ? MBA_MAX_MBPS : r->default_ctrl;
  		cfg->closid = closid;
  		cfg->have_new_ctrl = true;
include/linux/resctrl.h +0 -0
index 695247c08ba3..e33d6dfce8a1 100644

include/linux/resctrl.h enum resctrl_conf_type { · line 29

Quoted 5 lines
include/linux/resctrl.h +2 -0
@@ -25,6 +25,8 @@ enum resctrl_conf_type {
  	CDP_CODE,
  	CDP_DATA,
  };
+#define NUM_CDP_TYPES	CDP_DATA + 1
+

There is also the option of making NUM_CDP_TYPES the last member of resctrl_conf_type ... and changing its name to CDP_NUM_TYPES to be consistent with similar usages (RDT_NUM_GROUP, RDT_NUM_MODES, RDT_NUM_RESOURCES)? The problem then is that you need to handle it in the switch statement. If you choose to keep this way then please change the name to be consistent with the other enums, add an empty line after the enum to make checkpatch happy, and add parenthesis around its value.

include/linux/resctrl.h struct rdt_domain { · line 75

Quoted 13 lines
include/linux/resctrl.h +1 -1
  /**
   * struct resctrl_staged_config - parsed configuration to be applied
@@ -67,7 +69,7 @@ struct rdt_domain {
  	int				cqm_work_cpu;
 
  	struct pseudo_lock_region	*plr;
-	struct resctrl_staged_config	staged_config[1];
+	struct resctrl_staged_config	staged_config[NUM_CDP_TYPES];
  };
 
  /**

Reinette

author reviewer
#38 lore raw

PATCH 14/24 x86/resctrl: Make update_domains() learn the affected closids

In reply to James Morse

Now that the closid is present in the staged configuration, update_domains() can learn which low/high values it should update, instead of being explicitly told. This paves the way for multiple configuration changes being staged, affecting different indexes in the ctrlval array.

Remove the single passed in closid, and update msr_param as each staged config is applied.

Once the L2/L2CODE/L2DATA resources are merged this will allow update_domains() to be called once for the single resource, even when CDP is in use. This results in both CODE and DATA configurations being applied and the two consecutive closids being updated with a single smp_call_function_many().

This keeps the CDP odd/even behaviour inside the arch code for resctrl, so that architectures that don't do this don't need to emulate it.

As update_domains() applies the staged configuration to the hw_dom's configuration array, and updates the hardware, make it part of the arch code interface.

---
 arch/x86/kernel/cpu/resctrl/ctrlmondata.c | 40 +++++++++++++++++------
 arch/x86/kernel/cpu/resctrl/internal.h    |  6 ++--
 arch/x86/kernel/cpu/resctrl/rdtgroup.c    |  2 +-
 include/linux/resctrl.h                   |  1 +
 4 files changed, 35 insertions(+), 14 deletions(-)
arch/x86/kernel/cpu/resctrl/ctrlmondata.c +30 -10
index f7152c7fdc1b..91864c2e5795 100644
@@ -249,37 +249,44 @@ static int parse_line(char *line, struct resctrl_schema *s,
 	return -EINVAL;
 }
 
-static void apply_config(struct rdt_hw_domain *hw_dom,
+/*
+ * Merge the staged config with the domains configuration array.
+ * Return true if changes were made.
+ */
+static bool apply_config(struct rdt_hw_domain *hw_dom,
 			 struct resctrl_staged_config *cfg,
-			 cpumask_var_t cpu_mask, bool mba_sc)
+			 cpumask_var_t cpu_mask, u32 idx, bool mba_sc)
 {
 	struct rdt_domain *dom = &hw_dom->resctrl;
 	u32 *dc = mba_sc ? hw_dom->mbps_val : hw_dom->ctrl_val;
 
-	if (cfg->new_ctrl != dc[cfg->closid]) {
+	cfg->have_new_ctrl = false;
+	if (cfg->new_ctrl != dc[idx]) {
 		cpumask_set_cpu(cpumask_any(&dom->cpu_mask), cpu_mask);
-		dc[cfg->closid] = cfg->new_ctrl;
+		dc[idx] = cfg->new_ctrl;
+
+		return true;
 	}
 
-	cfg->have_new_ctrl = false;
+	return false;
 }
 
-int update_domains(struct rdt_resource *r, int closid)
+int resctrl_arch_update_domains(struct rdt_resource *r)
 {
 	struct resctrl_staged_config *cfg;
 	struct rdt_hw_domain *hw_dom;
+	bool msr_param_init = false;
 	struct msr_param msr_param;
 	enum resctrl_conf_type t;
 	cpumask_var_t cpu_mask;
 	struct rdt_domain *d;
 	bool mba_sc;
 	int cpu;
+	u32 idx;
 
 	if (!zalloc_cpumask_var(&cpu_mask, GFP_KERNEL))
 		return -ENOMEM;
 
-	msr_param.low = closid;
-	msr_param.high = msr_param.low + 1;
 	msr_param.res = r;
 
 	mba_sc = is_mba_sc(r);
@@ -290,10 +297,23 @@ int update_domains(struct rdt_resource *r, int closid)
 			if (!cfg->have_new_ctrl)
 				continue;
 
-			apply_config(hw_dom, cfg, cpu_mask, mba_sc);
+			idx = cfg->closid;
+			if (!apply_config(hw_dom, cfg, cpu_mask, idx, mba_sc))
+				continue;
+
+			if (!msr_param_init) {
+				msr_param.low = idx;
+				msr_param.high = idx;
+				msr_param_init = true;
+			} else {
+				msr_param.low = min(msr_param.low, idx);
+				msr_param.high = max(msr_param.high, idx);
+			}
 		}
 	}
 
+	msr_param.high += 1;
+
 	/*
 	 * Avoid writing the control msr with control values when
 	 * MBA software controller is enabled
@@ -387,7 +407,7 @@ ssize_t rdtgroup_schemata_write(struct kernfs_open_file *of,
 
 	list_for_each_entry(s, &resctrl_all_schema, list) {
 		r = s->res;
-		ret = update_domains(r, rdtgrp->closid);
+		ret = resctrl_arch_update_domains(r);
 		if (ret)
 			goto out;
 	}
arch/x86/kernel/cpu/resctrl/internal.h +3 -3
index 5294ae0c3ed9..e86550d888cc 100644
@@ -324,8 +324,8 @@ static inline struct rdt_hw_domain *resctrl_to_arch_dom(struct rdt_domain *r)
  */
 struct msr_param {
 	struct rdt_resource	*res;
-	int			low;
-	int			high;
+	u32			low;
+	u32			high;
 };
 
 static inline bool is_llc_occupancy_enabled(void)
@@ -498,7 +498,7 @@ void rdt_pseudo_lock_release(void);
 int rdtgroup_pseudo_lock_create(struct rdtgroup *rdtgrp);
 void rdtgroup_pseudo_lock_remove(struct rdtgroup *rdtgrp);
 struct rdt_domain *get_domain_from_cpu(int cpu, struct rdt_resource *r);
-int update_domains(struct rdt_resource *r, int closid);
+int update_domains(struct rdt_resource *r);
 int closids_supported(void);
 void closid_free(int closid);
 int alloc_rmid(void);
arch/x86/kernel/cpu/resctrl/rdtgroup.c +1 -1
index 5eb14dc9c579..c6689cad1ce7 100644
@@ -2870,7 +2870,7 @@ static int rdtgroup_init_alloc(struct rdtgroup *rdtgrp)
 				return ret;
 		}
 
-		ret = update_domains(r, rdtgrp->closid);
+		ret = resctrl_arch_update_domains(r);
 		if (ret < 0) {
 			rdt_last_cmd_puts("Failed to initialize allocations\n");
 			return ret;
include/linux/resctrl.h +1 -0
index e33d6dfce8a1..2b3828df13cf 100644
@@ -203,5 +203,6 @@ struct resctrl_schema {
 
 /* The number of closid supported by this resource regardless of CDP */
 u32 resctrl_arch_get_num_closid(struct rdt_resource *r);
+int resctrl_arch_update_domains(struct rdt_resource *r);
 
 #endif /* _RESCTRL_H */
Signature · 1 line
2.28.0
author reviewer
#39 lore raw

PATCH 15/24 x86/resctrl: Add a helper to read a closid's configuration

In reply to James Morse

The hardware configuration may look completely different to the values resctrl gets from user-space. The staged configuration and resctrl_arch_update_domains() allow the architecture to convert or translate these values.
(e.g. Arm's MPAM may back MBA's percentage control using the
'BWPBM' bitmap)

Resctrl shouldn't read or write these values directly. As a step towards taking direct access away, add a helper to read the current configuration.

This will allow another architecture to scale the bitmaps if necessary, and possibly use controls that don't take the user-space control format at all.

---
 arch/x86/kernel/cpu/resctrl/ctrlmondata.c | 16 ++++++---
 arch/x86/kernel/cpu/resctrl/monitor.c     |  6 +++-
 arch/x86/kernel/cpu/resctrl/rdtgroup.c    | 43 ++++++++++-------------
 include/linux/resctrl.h                   |  2 ++
 4 files changed, 37 insertions(+), 30 deletions(-)
arch/x86/kernel/cpu/resctrl/ctrlmondata.c +12 -4
index 91864c2e5795..0cf2f24e5c3b 100644
@@ -428,22 +428,30 @@ ssize_t rdtgroup_schemata_write(struct kernfs_open_file *of,
 	return ret ?: nbytes;
 }
 
+void resctrl_arch_get_config(struct rdt_resource *r, struct rdt_domain *d,
+			     u32 closid, u32 *value)
+{
+	struct rdt_hw_domain *hw_dom = resctrl_to_arch_dom(d);
+
+	if (!is_mba_sc(r))
+		*value = hw_dom->ctrl_val[closid];
+	else
+		*value = hw_dom->mbps_val[closid];
+}
+
 static void show_doms(struct seq_file *s, struct resctrl_schema *schema, int closid)
 {
 	struct rdt_resource *r = schema->res;
-	struct rdt_hw_domain *hw_dom;
 	struct rdt_domain *dom;
 	bool sep = false;
 	u32 ctrl_val;
 
 	seq_printf(s, "%*s:", RESCTRL_NAME_LEN, schema->name);
 	list_for_each_entry(dom, &r->domains, list) {
-		hw_dom = resctrl_to_arch_dom(dom);
 		if (sep)
 			seq_puts(s, ";");
 
-		ctrl_val = (!is_mba_sc(r) ? hw_dom->ctrl_val[closid] :
-			    hw_dom->mbps_val[closid]);
+		resctrl_arch_get_config(r, dom, closid, &ctrl_val);
 		seq_printf(s, r->format_str, dom->id, max_data_width,
 			   ctrl_val);
 		sep = true;
arch/x86/kernel/cpu/resctrl/monitor.c +5 -1
index 8b7d7ebfcd4b..6a62f1323b27 100644
@@ -379,8 +379,12 @@ static void update_mba_bw(struct rdtgroup *rgrp, struct rdt_domain *dom_mbm)
 	hw_dom_mba = resctrl_to_arch_dom(dom_mba);
 
 	cur_bw = pmbm_data->prev_bw;
-	user_bw = hw_dom_mba->mbps_val[closid];
+	resctrl_arch_get_config(r_mba, dom_mba, closid, &user_bw);
 	delta_bw = pmbm_data->delta_bw;
+	/*
+	 * resctrl_arch_get_config() chooses the mbps/ctrl value to return
+	 * based on is_mba_sc(). For now, reach into the hw_dom.
+	 */
 	cur_msr_val = hw_dom_mba->ctrl_val[closid];
 
 	/*
arch/x86/kernel/cpu/resctrl/rdtgroup.c +18 -25

134 lines not on this page · raw message

include/linux/resctrl.h +2 -0
index 2b3828df13cf..b870c2f3c3c9 100644
@@ -204,5 +204,7 @@ struct resctrl_schema {
 /* The number of closid supported by this resource regardless of CDP */
 u32 resctrl_arch_get_num_closid(struct rdt_resource *r);
 int resctrl_arch_update_domains(struct rdt_resource *r);
+void resctrl_arch_get_config(struct rdt_resource *r, struct rdt_domain *d,
+			     u32 closid, u32 *value);
 
 #endif /* _RESCTRL_H */
Signature · 1 line
2.28.0
author reviewer
#40 lore raw

PATCH 16/24 x86/resctrl: Add a helper to read/set the CDP configuration

In reply to James Morse

Currently whether CDP is enabled is described in the alloc_enabled and alloc_capable flags, which are set differently between the L3 and L3CODE+L3DATA resources.

To merge these resources, to give us one configuration, the CDP state of the resource needs tracking explicitly. Add cdp_capable as something visible to resctrl, and cdp_enabled as something the arch code manages.

resctrl_arch_set_cdp_enabled() lets resctrl enable or disable CDP on a resource. resctrl_arch_get_cdp_enabled() lets it read the current state.

With Arm's MPAM, separate code and data closids is a part of the CPU configuration. Enabling CDP for one resource means all resources see the different closid values.

---
It may be possible for MPAM to apply the same 'L3' configuration to the two closid that are in use, giving the illusion that CDP is enabled for some resources, but disabled for others ... but this will complicate monitoring.

---
 arch/x86/kernel/cpu/resctrl/core.c        |  4 ++
 arch/x86/kernel/cpu/resctrl/internal.h    | 11 +++-
 arch/x86/kernel/cpu/resctrl/pseudo_lock.c |  4 +-
 arch/x86/kernel/cpu/resctrl/rdtgroup.c    | 67 +++++++++++++----------
 4 files changed, 55 insertions(+), 31 deletions(-)
arch/x86/kernel/cpu/resctrl/core.c +4 -0
index cda071009fed..7e98869ba006 100644
@@ -369,11 +369,15 @@ static void rdt_get_cdp_config(int level, int type)
 	r->cache.shareable_bits = r_l->cache.shareable_bits;
 	r->data_width = (r->cache.cbm_len + 3) / 4;
 	r->alloc_capable = true;
+	hw_res_l->cdp_capable = true;
+	hw_res->cdp_capable = true;
 	/*
 	 * By default, CDP is disabled. CDP can be enabled by mount parameter
 	 * "cdp" during resctrl file system mount time.
 	 */
 	r->alloc_enabled = false;
+	hw_res_l->cdp_enabled = false;
+	hw_res->cdp_enabled = false;
 }
 
 static void rdt_get_cdp_l3_config(void)
arch/x86/kernel/cpu/resctrl/internal.h +10 -1
index e86550d888cc..f039fd9f4f4f 100644
@@ -365,6 +365,8 @@ struct rdt_parse_data {
  * @msr_base:		Base MSR address for CBMs
  * @msr_update:		Function pointer to update QOS MSRs
  * @mon_scale:		cqm counter * mon_scale = occupancy in bytes
+ * @cdp_capable:	Is the CDP feature available on this resource
+ * @cdp_enabled:	CDP state of this resource
  */
 struct rdt_hw_resource {
 	enum resctrl_conf_type	conf_type;
@@ -377,6 +379,8 @@ struct rdt_hw_resource {
 				 struct rdt_resource *r);
 	unsigned int		mon_scale;
 	unsigned int		mbm_width;
+	bool			cdp_capable;
+	bool			cdp_enabled;
 };
 
 static inline struct rdt_hw_resource *resctrl_to_arch_res(struct rdt_resource *r)
@@ -397,7 +401,7 @@ DECLARE_STATIC_KEY_FALSE(rdt_alloc_enable_key);
 
 extern struct dentry *debugfs_resctrl;
 
-enum {
+enum resctrl_res_level {
 	RDT_RESOURCE_L3,
 	RDT_RESOURCE_L3DATA,
 	RDT_RESOURCE_L3CODE,
@@ -418,6 +422,11 @@ static inline struct rdt_resource *resctrl_inc(struct rdt_resource *res)
 	return &hw_res->resctrl;
 }
 
+static inline bool resctrl_arch_get_cdp_enabled(enum resctrl_res_level l)
+{
+	return rdt_resources_all[l].cdp_enabled;
+}
+
 #define for_each_rdt_resource(r)					      \
 	for (r = &rdt_resources_all[0].resctrl;				      \
 	     r < &rdt_resources_all[RDT_NUM_RESOURCES].resctrl;		      \
arch/x86/kernel/cpu/resctrl/pseudo_lock.c +2 -2
index d9d9861f244f..f126d442a65f 100644
@@ -684,8 +684,8 @@ int rdtgroup_locksetup_enter(struct rdtgroup *rdtgrp)
 	 *   resource, the portion of cache used by it should be made
 	 *   unavailable to all future allocations from both resources.
 	 */
-	if (rdt_resources_all[RDT_RESOURCE_L3DATA].resctrl.alloc_enabled ||
-	    rdt_resources_all[RDT_RESOURCE_L2DATA].resctrl.alloc_enabled) {
+	if (resctrl_arch_get_cdp_enabled(RDT_RESOURCE_L3) ||
+	    resctrl_arch_get_cdp_enabled(RDT_RESOURCE_L2)) {
 		rdt_last_cmd_puts("CDP enabled\n");
 		return -EINVAL;
 	}
arch/x86/kernel/cpu/resctrl/rdtgroup.c +39 -28

110 lines not on this page · raw message

Signature · 1 line
2.28.0
author reviewer
#41 lore raw

PATCH 17/24 x86/resctrl: Use cdp_enabled in rdt_domain_reconfigure_cdp()

In reply to James Morse

rdt_domain_reconfigure_cdp() infers whether CDP is enabled by checking the alloc_capable and alloc_enabled flags of the data resources.

Now that there is an explicit cdp_enabled, use that.

---
 arch/x86/kernel/cpu/resctrl/rdtgroup.c | 8 +++++---
 1 file changed, 5 insertions(+), 3 deletions(-)
arch/x86/kernel/cpu/resctrl/rdtgroup.c +5 -3
index 6e150560c3c1..eeedafa7d5e7 100644
@@ -1944,14 +1944,16 @@ static int set_cache_qos_cfg(int level, bool enable)
 /* Restore the qos cfg state when a domain comes online */
 void rdt_domain_reconfigure_cdp(struct rdt_resource *r)
 {
-	if (!r->alloc_capable)
+	struct rdt_hw_resource *hw_res = resctrl_to_arch_res(r);
+
+	if (!hw_res->cdp_capable)
 		return;
 
 	if (r == &rdt_resources_all[RDT_RESOURCE_L2DATA].resctrl)
-		l2_qos_cfg_update(&r->alloc_enabled);
+		l2_qos_cfg_update(&hw_res->cdp_enabled);
 
 	if (r == &rdt_resources_all[RDT_RESOURCE_L3DATA].resctrl)
-		l3_qos_cfg_update(&r->alloc_enabled);
+		l3_qos_cfg_update(&hw_res->cdp_enabled);
 }
 
 /*
Signature · 1 line
2.28.0
author reviewer
#42 lore raw

PATCH 18/24 x86/resctrl: Pass configuration type to resctrl_arch_get_config()

In reply to James Morse

Once the configuration arrays are merged, the get_config() helper needs to be told whether the CODE, DATA or BOTH configuration is being retrieved.

Pass this information from the schema into resctrl_arch_get_config().

Nothing uses this yet, but it will later be used to map the closid to the index in the configuration array.

---
 arch/x86/kernel/cpu/resctrl/ctrlmondata.c |  5 ++--
 arch/x86/kernel/cpu/resctrl/monitor.c     |  2 +-
 arch/x86/kernel/cpu/resctrl/rdtgroup.c    | 35 +++++++++++++++--------
 include/linux/resctrl.h                   |  3 +-
 4 files changed, 29 insertions(+), 16 deletions(-)
arch/x86/kernel/cpu/resctrl/ctrlmondata.c +3 -2
index 0cf2f24e5c3b..f6b4049c67c2 100644
@@ -429,7 +429,7 @@ ssize_t rdtgroup_schemata_write(struct kernfs_open_file *of,
 }
 
 void resctrl_arch_get_config(struct rdt_resource *r, struct rdt_domain *d,
-			     u32 closid, u32 *value)
+			     u32 closid, enum resctrl_conf_type type, u32 *value)
 {
 	struct rdt_hw_domain *hw_dom = resctrl_to_arch_dom(d);
 
@@ -451,7 +451,8 @@ static void show_doms(struct seq_file *s, struct resctrl_schema *schema, int clo
 		if (sep)
 			seq_puts(s, ";");
 
-		resctrl_arch_get_config(r, dom, closid, &ctrl_val);
+		resctrl_arch_get_config(r, dom, closid, schema->conf_type,
+					&ctrl_val);
 		seq_printf(s, r->format_str, dom->id, max_data_width,
 			   ctrl_val);
 		sep = true;
arch/x86/kernel/cpu/resctrl/monitor.c +1 -1
index 6a62f1323b27..ab6630b466d5 100644
@@ -379,7 +379,7 @@ static void update_mba_bw(struct rdtgroup *rgrp, struct rdt_domain *dom_mbm)
 	hw_dom_mba = resctrl_to_arch_dom(dom_mba);
 
 	cur_bw = pmbm_data->prev_bw;
-	resctrl_arch_get_config(r_mba, dom_mba, closid, &user_bw);
+	resctrl_arch_get_config(r_mba, dom_mba, closid, CDP_BOTH, &user_bw);
 	delta_bw = pmbm_data->delta_bw;
 	/*
 	 * resctrl_arch_get_config() chooses the mbps/ctrl value to return
arch/x86/kernel/cpu/resctrl/rdtgroup.c +23 -12
index eeedafa7d5e7..cb9ca56ce2e6 100644
@@ -924,7 +924,8 @@ static int rdt_bit_usage_show(struct kernfs_open_file *of,
 		for (i = 0; i < closids_supported(); i++) {
 			if (!closid_allocated(i))
 				continue;
-			resctrl_arch_get_config(r, dom, i, &ctrl_val);
+			resctrl_arch_get_config(r, dom, i, s->conf_type,
+						&ctrl_val);
 			mode = rdtgroup_mode_by_closid(i);
 			switch (mode) {
 			case RDT_MODE_SHAREABLE:
@@ -1101,6 +1102,7 @@ static int rdtgroup_mode_show(struct kernfs_open_file *of,
  *         Used to return the result.
  * @d_cdp: RDT domain that shares hardware with @d (RDT domain peer)
  *         Used to return the result.
+ * @peer_type: The CDP configuration type of the peer resource.
  *
  * RDT resources are managed independently and by extension the RDT domains
  * (RDT resource instances) are managed independently also. The Code and
@@ -1118,7 +1120,8 @@ static int rdtgroup_mode_show(struct kernfs_open_file *of,
  */
 static int rdt_cdp_peer_get(struct rdt_resource *r, struct rdt_domain *d,
 			    struct rdt_resource **r_cdp,
-			    struct rdt_domain **d_cdp)
+			    struct rdt_domain **d_cdp,
+			    enum resctrl_conf_type *peer_type)
 {
 	struct rdt_resource *_r_cdp = NULL;
 	struct rdt_domain *_d_cdp = NULL;
@@ -1127,15 +1130,19 @@ static int rdt_cdp_peer_get(struct rdt_resource *r, struct rdt_domain *d,
 	switch (r->rid) {
 	case RDT_RESOURCE_L3DATA:
 		_r_cdp = &rdt_resources_all[RDT_RESOURCE_L3CODE].resctrl;
+		*peer_type = CDP_CODE;
 		break;
 	case RDT_RESOURCE_L3CODE:
 		_r_cdp =  &rdt_resources_all[RDT_RESOURCE_L3DATA].resctrl;
+		*peer_type = CDP_DATA;
 		break;
 	case RDT_RESOURCE_L2DATA:
 		_r_cdp =  &rdt_resources_all[RDT_RESOURCE_L2CODE].resctrl;
+		*peer_type = CDP_CODE;
 		break;
 	case RDT_RESOURCE_L2CODE:
 		_r_cdp =  &rdt_resources_all[RDT_RESOURCE_L2DATA].resctrl;
+		*peer_type = CDP_DATA;
 		break;
 	default:
 		ret = -ENOENT;
@@ -1186,7 +1193,8 @@ static int rdt_cdp_peer_get(struct rdt_resource *r, struct rdt_domain *d,
  * Return: false if CBM does not overlap, true if it does.
  */
 static bool __rdtgroup_cbm_overlaps(struct rdt_resource *r, struct rdt_domain *d,
-				    unsigned long cbm, int closid, bool exclusive)
+				    unsigned long cbm, int closid,
+				    enum resctrl_conf_type type, bool exclusive)
 {
 	enum rdtgrp_mode mode;
 	unsigned long ctrl_b;
@@ -1201,7 +1209,7 @@ static bool __rdtgroup_cbm_overlaps(struct rdt_resource *r, struct rdt_domain *d
 
 	/* Check for overlap with other resource groups */
 	for (i = 0; i < closids_supported(); i++) {
-		resctrl_arch_get_config(r, d, i, (u32 *)&ctrl_b);
+		resctrl_arch_get_config(r, d, i, type, (u32 *)&ctrl_b);
 		mode = rdtgroup_mode_by_closid(i);
 		if (closid_allocated(i) && i != closid &&
 		    mode != RDT_MODE_PSEUDO_LOCKSETUP) {
@@ -1242,17 +1250,19 @@ static bool __rdtgroup_cbm_overlaps(struct rdt_resource *r, struct rdt_domain *d
 bool rdtgroup_cbm_overlaps(struct resctrl_schema *s, struct rdt_domain *d,
 			   unsigned long cbm, int closid, bool exclusive)
 {
+	enum resctrl_conf_type peer_type;
 	struct rdt_resource *r = s->res;
 	struct rdt_resource *r_cdp;
 	struct rdt_domain *d_cdp;
 
-	if (__rdtgroup_cbm_overlaps(r, d, cbm, closid, exclusive))
+	if (__rdtgroup_cbm_overlaps(r, d, cbm, closid, s->conf_type,
+				    exclusive))
 		return true;
 
-	if (rdt_cdp_peer_get(r, d, &r_cdp, &d_cdp) < 0)
+	if (rdt_cdp_peer_get(r, d, &r_cdp, &d_cdp, &peer_type) < 0)
 		return false;
 
-	return  __rdtgroup_cbm_overlaps(r_cdp, d_cdp, cbm, closid, exclusive);
+	return  __rdtgroup_cbm_overlaps(r_cdp, d_cdp, cbm, closid, peer_type, exclusive);
 }
 
 /**
@@ -1282,7 +1292,7 @@ static bool rdtgroup_mode_test_exclusive(struct rdtgroup *rdtgrp)
 			continue;
 		has_cache = true;
 		list_for_each_entry(d, &r->domains, list) {
-			resctrl_arch_get_config(r, d, closid, &ctrl);
+			resctrl_arch_get_config(r, d, closid, s->conf_type, &ctrl);
 			if (rdtgroup_cbm_overlaps(s, d, ctrl, closid, false)) {
 				rdt_last_cmd_puts("Schemata overlaps\n");
 				return false;
@@ -1457,7 +1467,7 @@ static int rdtgroup_size_show(struct kernfs_open_file *of,
 				size = 0;
 			} else {
 				resctrl_arch_get_config(r, d, rdtgrp->closid,
-							&ctrl);
+							schema->conf_type, &ctrl);
 				if (r->rid == RDT_RESOURCE_MBA)
 					size = ctrl;
 				else
@@ -2756,6 +2766,7 @@ static int __init_one_rdt_domain(struct rdt_domain *d, struct resctrl_schema *s,
 	enum resctrl_conf_type t = s-> conf_type;
 	struct rdt_resource *r_cdp = NULL;
 	struct resctrl_staged_config *cfg;
+	enum resctrl_conf_type peer_type;
 	struct rdt_domain *d_cdp = NULL;
 	struct rdt_resource *r = s->res;
 	u32 used_b = 0, unused_b = 0;
@@ -2764,7 +2775,7 @@ static int __init_one_rdt_domain(struct rdt_domain *d, struct resctrl_schema *s,
 	enum rdtgrp_mode mode;
 	int i;
 
-	rdt_cdp_peer_get(r, d, &r_cdp, &d_cdp);
+	rdt_cdp_peer_get(r, d, &r_cdp, &d_cdp, &peer_type);
 	cfg = &d->staged_config[t];
 	cfg->have_new_ctrl = false;
 	cfg->new_ctrl = r->cache.shareable_bits;
@@ -2785,10 +2796,10 @@ static int __init_one_rdt_domain(struct rdt_domain *d, struct resctrl_schema *s,
 			 * with an exclusive group.
 			 */
 			if (d_cdp)
-				resctrl_arch_get_config(r_cdp, d_cdp, i, &peer_ctl);
+				resctrl_arch_get_config(r_cdp, d_cdp, i, peer_type, &peer_ctl);
 			else
 				peer_ctl = 0;
-			resctrl_arch_get_config(r, d, i, &ctrl_val);
+			resctrl_arch_get_config(r, d, i, s->conf_type, &ctrl_val);
 			used_b |= ctrl_val | peer_ctl;
 			if (mode == RDT_MODE_SHAREABLE)
 				cfg->new_ctrl |= ctrl_val | peer_ctl;
include/linux/resctrl.h +2 -1
index b870c2f3c3c9..df670f204abf 100644
@@ -205,6 +205,7 @@ struct resctrl_schema {
 u32 resctrl_arch_get_num_closid(struct rdt_resource *r);
 int resctrl_arch_update_domains(struct rdt_resource *r);
 void resctrl_arch_get_config(struct rdt_resource *r, struct rdt_domain *d,
-			     u32 closid, u32 *value);
+			     u32 closid, enum resctrl_conf_type type,
+			     u32 *value);
 
 #endif /* _RESCTRL_H */
Signature · 1 line
2.28.0
author reviewer
#43 lore raw

PATCH 19/24 x86/resctrl: Make ctrlval arrays the same size

In reply to James Morse

The CODE and DATA resources have their own ctrlval arrays which are half the size because num_closid was already adjusted.

Prior to having one ctrlval array for the resource, move the num_closid correction into resctrl, so that the ctrlval arrays are all the same size.

A shortlived quirk of this is that the caches are reset twice, once for CODE once for DATA.

---
 arch/x86/kernel/cpu/resctrl/core.c     | 10 +++++++++-
 arch/x86/kernel/cpu/resctrl/rdtgroup.c | 10 ++++++++++
 2 files changed, 19 insertions(+), 1 deletion(-)
arch/x86/kernel/cpu/resctrl/core.c +9 -1
index 7e98869ba006..b2fda4cd88ba 100644
@@ -363,7 +363,7 @@ static void rdt_get_cdp_config(int level, int type)
 	struct rdt_resource *r = &rdt_resources_all[type].resctrl;
 	struct rdt_hw_resource *hw_res = resctrl_to_arch_res(r);
 
-	hw_res->num_closid = hw_res_l->num_closid / 2;
+	hw_res->num_closid = hw_res_l->num_closid;
 	r->cache.cbm_len = r_l->cache.cbm_len;
 	r->default_ctrl = r_l->default_ctrl;
 	r->cache.shareable_bits = r_l->cache.shareable_bits;
@@ -549,6 +549,14 @@ static int domain_setup_ctrlval(struct rdt_resource *r, struct rdt_domain *d)
 
 	m.low = 0;
 	m.high = hw_res->num_closid;
+
+	/*
+	 * temporary: the array is full-size, but cat_wrmsr() still re-maps
+	 * the index.
+	 */
+	if (hw_res->conf_type != CDP_BOTH)
+		m.high /= 2;
+
 	hw_res->msr_update(d, &m, r);
 	return 0;
 }
arch/x86/kernel/cpu/resctrl/rdtgroup.c +10 -0
index cb9ca56ce2e6..4fa6c386d751 100644
@@ -2165,6 +2165,9 @@ static int create_schemata_list(void)
 
 		s->res = r;
 		s->num_closid = resctrl_arch_get_num_closid(r);
+		if (resctrl_arch_get_cdp_enabled(r->rid))
+			s->num_closid /= 2;
+
 		s->conf_type = resctrl_to_arch_res(r)->conf_type;
 
 		ret = snprintf(s->name, sizeof(s->name), r->name);
@@ -2376,6 +2379,13 @@ static int reset_all_ctrls(struct rdt_resource *r)
 	msr_param.low = 0;
 	msr_param.high = hw_res->num_closid;
 
+	/*
+	 * temporary: the array is full-sized, but cat_wrmsr() still re-maps
+	 * the index.
+	 */
+	if (hw_res->cdp_enabled)
+		msr_param.high /= 2;
+
 	/*
 	 * Disable resource control for this resource by setting all
 	 * CBMs in all domains to the maximum mask value. Pick one CPU
Signature · 1 line
2.28.0
author reviewer
#44 lore raw

PATCH 20/24 x86/resctrl: Apply offset correction when config is staged

In reply to James Morse

When resctrl comes to write the CAT MSR values, it applies an adjustment based on the style of the resource. CODE and DATA resources have their closid mapped into an odd/even range.

Previously the ctrlval array was increased to be the same size regardless of CODE/DATA/BOTH. Move the arithmetic into apply_config() so that odd/even slots in the ctrlval array are used.

This makes it possible to merge the resources.

In future, the arithmetic will be based on the style of the configuration, not the resource.

---
 arch/x86/kernel/cpu/resctrl/core.c        | 15 +--------------
 arch/x86/kernel/cpu/resctrl/ctrlmondata.c | 15 ++++++++++++---
 arch/x86/kernel/cpu/resctrl/rdtgroup.c    |  7 -------
 3 files changed, 13 insertions(+), 24 deletions(-)
arch/x86/kernel/cpu/resctrl/core.c +1 -14
index b2fda4cd88ba..79b17ece4528 100644
@@ -195,11 +195,6 @@ struct rdt_hw_resource rdt_resources_all[] = {
 	},
 };
 
-static unsigned int cbm_idx(struct rdt_resource *r, unsigned int closid)
-{
-	return closid * r->cache.cbm_idx_mult + r->cache.cbm_idx_offset;
-}
-
 /*
  * cache_alloc_hsw_probe() - Have to probe for Intel haswell server CPUs
  * as they do not have CPUID enumeration support for Cache allocation.
@@ -438,7 +433,7 @@ cat_wrmsr(struct rdt_domain *d, struct msr_param *m, struct rdt_resource *r)
 	unsigned int i;
 
 	for (i = m->low; i < m->high; i++)
-		wrmsrl(hw_res->msr_base + cbm_idx(r, i), hw_dom->ctrl_val[i]);
+		wrmsrl(hw_res->msr_base + i, hw_dom->ctrl_val[i]);
 }
 
 struct rdt_domain *get_domain_from_cpu(int cpu, struct rdt_resource *r)
@@ -549,14 +544,6 @@ static int domain_setup_ctrlval(struct rdt_resource *r, struct rdt_domain *d)
 
 	m.low = 0;
 	m.high = hw_res->num_closid;
-
-	/*
-	 * temporary: the array is full-size, but cat_wrmsr() still re-maps
-	 * the index.
-	 */
-	if (hw_res->conf_type != CDP_BOTH)
-		m.high /= 2;
-
 	hw_res->msr_update(d, &m, r);
 	return 0;
 }
arch/x86/kernel/cpu/resctrl/ctrlmondata.c +12 -3
index f6b4049c67c2..28a251cf3c60 100644
@@ -249,6 +249,14 @@ static int parse_line(char *line, struct resctrl_schema *s,
 	return -EINVAL;
 }
 
+static unsigned int cbm_idx(struct rdt_resource *r, unsigned int closid)
+{
+	if (r->rid == RDT_RESOURCE_MBA)
+		return closid;
+
+	return closid * r->cache.cbm_idx_mult + r->cache.cbm_idx_offset;
+}
+
 /*
  * Merge the staged config with the domains configuration array.
  * Return true if changes were made.
@@ -297,7 +305,7 @@ int resctrl_arch_update_domains(struct rdt_resource *r)
 			if (!cfg->have_new_ctrl)
 				continue;
 
-			idx = cfg->closid;
+			idx = cbm_idx(r, cfg->closid);
 			if (!apply_config(hw_dom, cfg, cpu_mask, idx, mba_sc))
 				continue;
 
@@ -432,11 +440,12 @@ void resctrl_arch_get_config(struct rdt_resource *r, struct rdt_domain *d,
 			     u32 closid, enum resctrl_conf_type type, u32 *value)
 {
 	struct rdt_hw_domain *hw_dom = resctrl_to_arch_dom(d);
+	u32 idx = cbm_idx(r, closid);
 
 	if (!is_mba_sc(r))
-		*value = hw_dom->ctrl_val[closid];
+		*value = hw_dom->ctrl_val[idx];
 	else
-		*value = hw_dom->mbps_val[closid];
+		*value = hw_dom->mbps_val[idx];
 }
 
 static void show_doms(struct seq_file *s, struct resctrl_schema *schema, int closid)
arch/x86/kernel/cpu/resctrl/rdtgroup.c +0 -7
index 4fa6c386d751..162e415d5d09 100644
@@ -2379,13 +2379,6 @@ static int reset_all_ctrls(struct rdt_resource *r)
 	msr_param.low = 0;
 	msr_param.high = hw_res->num_closid;
 
-	/*
-	 * temporary: the array is full-sized, but cat_wrmsr() still re-maps
-	 * the index.
-	 */
-	if (hw_res->cdp_enabled)
-		msr_param.high /= 2;
-
 	/*
 	 * Disable resource control for this resource by setting all
 	 * CBMs in all domains to the maximum mask value. Pick one CPU
Signature · 1 line
2.28.0
author reviewer
#45 lore raw

PATCH 21/24 x86/resctrl: Calculate the index from the configuration type

In reply to James Morse

resctrl uses cbm_idx() to map a closid to an index in the configuration array. This is based on whether this is a CODE, DATA or BOTH resource.

To merge the resources, resctrl needs to make this decision based on something else, as there will only be one resource. Decide based on the staged configuration type. This makes the static mult and offset parameters set by the arch code redundant.

---
 arch/x86/kernel/cpu/resctrl/core.c        | 12 ------------
 arch/x86/kernel/cpu/resctrl/ctrlmondata.c | 17 +++++++++++------
 include/linux/resctrl.h                   |  6 ------
 3 files changed, 11 insertions(+), 24 deletions(-)
arch/x86/kernel/cpu/resctrl/core.c +0 -12
index 79b17ece4528..e2f5ea129be2 100644
@@ -69,8 +69,6 @@ struct rdt_hw_resource rdt_resources_all[] = {
 			.cache_level		= 3,
 			.cache = {
 				.min_cbm_bits	= 1,
-				.cbm_idx_mult	= 1,
-				.cbm_idx_offset	= 0,
 			},
 			.domains		= domain_init(RDT_RESOURCE_L3),
 			.parse_ctrlval		= parse_cbm,
@@ -89,8 +87,6 @@ struct rdt_hw_resource rdt_resources_all[] = {
 			.cache_level		= 3,
 			.cache = {
 				.min_cbm_bits	= 1,
-				.cbm_idx_mult	= 2,
-				.cbm_idx_offset	= 0,
 			},
 			.domains		= domain_init(RDT_RESOURCE_L3DATA),
 			.parse_ctrlval		= parse_cbm,
@@ -109,8 +105,6 @@ struct rdt_hw_resource rdt_resources_all[] = {
 			.cache_level		= 3,
 			.cache = {
 				.min_cbm_bits	= 1,
-				.cbm_idx_mult	= 2,
-				.cbm_idx_offset	= 1,
 			},
 			.domains		= domain_init(RDT_RESOURCE_L3CODE),
 			.parse_ctrlval		= parse_cbm,
@@ -129,8 +123,6 @@ struct rdt_hw_resource rdt_resources_all[] = {
 			.cache_level		= 2,
 			.cache = {
 				.min_cbm_bits	= 1,
-				.cbm_idx_mult	= 1,
-				.cbm_idx_offset	= 0,
 			},
 			.domains		= domain_init(RDT_RESOURCE_L2),
 			.parse_ctrlval		= parse_cbm,
@@ -149,8 +141,6 @@ struct rdt_hw_resource rdt_resources_all[] = {
 			.cache_level		= 2,
 			.cache = {
 				.min_cbm_bits	= 1,
-				.cbm_idx_mult	= 2,
-				.cbm_idx_offset	= 0,
 			},
 			.domains		= domain_init(RDT_RESOURCE_L2DATA),
 			.parse_ctrlval		= parse_cbm,
@@ -169,8 +159,6 @@ struct rdt_hw_resource rdt_resources_all[] = {
 			.cache_level		= 2,
 			.cache = {
 				.min_cbm_bits	= 1,
-				.cbm_idx_mult	= 2,
-				.cbm_idx_offset	= 1,
 			},
 			.domains		= domain_init(RDT_RESOURCE_L2CODE),
 			.parse_ctrlval		= parse_cbm,
arch/x86/kernel/cpu/resctrl/ctrlmondata.c +11 -6
index 28a251cf3c60..cb91dcd0f329 100644
@@ -249,12 +249,17 @@ static int parse_line(char *line, struct resctrl_schema *s,
 	return -EINVAL;
 }
 
-static unsigned int cbm_idx(struct rdt_resource *r, unsigned int closid)
+static u32 get_config_index(u32 closid, enum resctrl_conf_type type)
 {
-	if (r->rid == RDT_RESOURCE_MBA)
+	switch (type) {
+	default:
+	case CDP_BOTH:
 		return closid;
-
-	return closid * r->cache.cbm_idx_mult + r->cache.cbm_idx_offset;
+	case CDP_CODE:
+		return (closid * 2) + 1;
+	case CDP_DATA:
+		return (closid * 2);
+	}
 }
 
 /*
@@ -305,7 +310,7 @@ int resctrl_arch_update_domains(struct rdt_resource *r)
 			if (!cfg->have_new_ctrl)
 				continue;
 
-			idx = cbm_idx(r, cfg->closid);
+			idx = get_config_index(cfg->closid, t);
 			if (!apply_config(hw_dom, cfg, cpu_mask, idx, mba_sc))
 				continue;
 
@@ -440,7 +445,7 @@ void resctrl_arch_get_config(struct rdt_resource *r, struct rdt_domain *d,
 			     u32 closid, enum resctrl_conf_type type, u32 *value)
 {
 	struct rdt_hw_domain *hw_dom = resctrl_to_arch_dom(d);
-	u32 idx = cbm_idx(r, closid);
+	u32 idx = get_config_index(closid, type);
 
 	if (!is_mba_sc(r))
 		*value = hw_dom->ctrl_val[idx];
include/linux/resctrl.h +0 -6
index df670f204abf..e1f390b9ece1 100644
@@ -76,10 +76,6 @@ struct rdt_domain {
  * struct resctrl_cache - Cache allocation related data
  * @cbm_len:		Length of the cache bit mask
  * @min_cbm_bits:	Minimum number of consecutive bits to be set
- * @cbm_idx_mult:	Multiplier of CBM index
- * @cbm_idx_offset:	Offset of CBM index. CBM index is computed by:
- *			closid * cbm_idx_multi + cbm_idx_offset
- *			in a cache bit mask
  * @shareable_bits:	Bitmask of shareable resource with other
  *			executing entities
  * @arch_has_sparse_bitmaps:	True if a bitmap like f00f is valid.
@@ -88,8 +84,6 @@ struct rdt_domain {
 struct resctrl_cache {
 	u32		cbm_len;
 	u32		min_cbm_bits;
-	unsigned int	cbm_idx_mult;	// TODO remove this
-	unsigned int	cbm_idx_offset; // TODO remove this
 	u32		shareable_bits;
 	bool		arch_has_sparse_bitmaps;
 	bool		arch_has_empty_bitmaps;
Signature · 1 line
2.28.0
author reviewer
#46 lore raw

PATCH 22/24 x86/resctrl: Merge the ctrlval arrays

In reply to James Morse

Now that the CODE/DATA resources don't use overlapping slots in the ctrlval arrays, they can be merged. This allows the cdp_peer configuration to be read from any resource's domain, instead of searching for the matching flavour.

Add a helper to allocate the ctrlval array, that returns the value on the L2 or L3 resource if it already exists. This gets removed once the resources are merged, and there really is only one ctrlval array.

---
 arch/x86/kernel/cpu/resctrl/core.c | 79 +++++++++++++++++++++++++++---
 1 file changed, 72 insertions(+), 7 deletions(-)
arch/x86/kernel/cpu/resctrl/core.c +72 -7
index e2f5ea129be2..01d010977367 100644
@@ -509,6 +509,72 @@ void setup_default_ctrlval(struct rdt_resource *r, u32 *dc, u32 *dm)
 	}
 }
 
+/*
+ * temporary.
+ * This relies on L2 or L3 being allocated before their CODE/DATA aliases
+ */
+static u32 *alloc_ctrlval_array(struct rdt_resource *r, struct rdt_domain *d,
+				bool mba_sc)
+{
+	/* these are for the underlying hardware, they may not match r/d */
+	struct rdt_domain *underlying_domain;
+	struct rdt_hw_resource *hw_res;
+	struct rdt_hw_domain *hw_dom;
+	bool remapped;
+
+	switch (r->rid) {
+	case RDT_RESOURCE_L3DATA:
+	case RDT_RESOURCE_L3CODE:
+		hw_res = &rdt_resources_all[RDT_RESOURCE_L3];
+		remapped = true;
+		break;
+	case RDT_RESOURCE_L2DATA:
+	case RDT_RESOURCE_L2CODE:
+		hw_res = &rdt_resources_all[RDT_RESOURCE_L2];
+		remapped = true;
+		break;
+	default:
+		hw_res = resctrl_to_arch_res(r);
+		remapped = false;
+	}
+
+	/*
+	 * If we changed the resource, we need to search for the underlying
+	 * domain. Doing this for all resources would make it tricky to add the
+	 * first resource, as domains aren't added to a resource list until
+	 * after the ctrlval arrays have been allocated.
+	 */
+	if (remapped)
+		underlying_domain = rdt_find_domain(&hw_res->resctrl, d->id,
+						    NULL);
+	else
+		underlying_domain = d;
+	hw_dom = resctrl_to_arch_dom(underlying_domain);
+
+	if (mba_sc) {
+		if (hw_dom->mbps_val)
+			return hw_dom->mbps_val;
+		return kmalloc_array(hw_res->num_closid,
+				     sizeof(*hw_dom->mbps_val), GFP_KERNEL);
+	} else {
+		if (hw_dom->ctrl_val)
+			return hw_dom->ctrl_val;
+		return kmalloc_array(hw_res->num_closid,
+				     sizeof(*hw_dom->ctrl_val), GFP_KERNEL);
+	}
+}
+
+/* Only kfree() for L2/L3, not the CODE/DATA aliases */
+static void free_ctrlval_arrays(struct rdt_resource *r, struct rdt_domain *d)
+{
+	struct rdt_hw_domain *hw_dom = resctrl_to_arch_dom(d);
+
+	if (r->rid == RDT_RESOURCE_L2 || r->rid == RDT_RESOURCE_L3) {
+		kfree(hw_dom->ctrl_val);
+		kfree(hw_dom->mbps_val);
+	}
+}
+
 static int domain_setup_ctrlval(struct rdt_resource *r, struct rdt_domain *d)
 {
 	struct rdt_hw_resource *hw_res = resctrl_to_arch_res(r);
@@ -516,18 +582,18 @@ static int domain_setup_ctrlval(struct rdt_resource *r, struct rdt_domain *d)
 	struct msr_param m;
 	u32 *dc, *dm;
 
-	dc = kmalloc_array(hw_res->num_closid, sizeof(*hw_dom->ctrl_val), GFP_KERNEL);
+	dc = alloc_ctrlval_array(r, d, false);
 	if (!dc)
 		return -ENOMEM;
+	hw_dom->ctrl_val = dc;
 
-	dm = kmalloc_array(hw_res->num_closid, sizeof(*hw_dom->mbps_val), GFP_KERNEL);
+	dm = alloc_ctrlval_array(r, d, true);
 	if (!dm) {
-		kfree(dc);
+		free_ctrlval_arrays(r, d);
 		return -ENOMEM;
 	}
-
-	hw_dom->ctrl_val = dc;
 	hw_dom->mbps_val = dm;
+
 	setup_default_ctrlval(r, dc, dm);
 
 	m.low = 0;
@@ -677,8 +743,7 @@ static void domain_remove_cpu(int cpu, struct rdt_resource *r)
 		if (d->plr)
 			d->plr->d = NULL;
 
-		kfree(hw_dom->ctrl_val);
-		kfree(hw_dom->mbps_val);
+		free_ctrlval_arrays(r, d);
 		bitmap_free(d->rmid_busy_llc);
 		kfree(d->mbm_total);
 		kfree(d->mbm_local);
Signature · 1 line
2.28.0
author reviewer
#47 lore raw

PATCH 23/24 x86/resctrl: Remove rdt_cdp_peer_get()

In reply to James Morse

Now that the configuration can be read from either resource, as they share the ctrlval array, rdt_cdp_peer_get() is not needed to to map the resource and search for the corresponding domain.

Replace it with a helper to return the 'other' CODE/DATA type, and use the existing get-config helper.

---
 arch/x86/kernel/cpu/resctrl/rdtgroup.c | 99 ++++----------------------
 1 file changed, 14 insertions(+), 85 deletions(-)
arch/x86/kernel/cpu/resctrl/rdtgroup.c +14 -85
index 162e415d5d09..0d561679f7e8 100644
@@ -1094,82 +1094,17 @@ static int rdtgroup_mode_show(struct kernfs_open_file *of,
 	return 0;
 }
 
-/**
- * rdt_cdp_peer_get - Retrieve CDP peer if it exists
- * @r: RDT resource to which RDT domain @d belongs
- * @d: Cache instance for which a CDP peer is requested
- * @r_cdp: RDT resource that shares hardware with @r (RDT resource peer)
- *         Used to return the result.
- * @d_cdp: RDT domain that shares hardware with @d (RDT domain peer)
- *         Used to return the result.
- * @peer_type: The CDP configuration type of the peer resource.
- *
- * RDT resources are managed independently and by extension the RDT domains
- * (RDT resource instances) are managed independently also. The Code and
- * Data Prioritization (CDP) RDT resources, while managed independently,
- * could refer to the same underlying hardware. For example,
- * RDT_RESOURCE_L2CODE and RDT_RESOURCE_L2DATA both refer to the L2 cache.
- *
- * When provided with an RDT resource @r and an instance of that RDT
- * resource @d rdt_cdp_peer_get() will return if there is a peer RDT
- * resource and the exact instance that shares the same hardware.
- *
- * Return: 0 if a CDP peer was found, <0 on error or if no CDP peer exists.
- *         If a CDP peer was found, @r_cdp will point to the peer RDT resource
- *         and @d_cdp will point to the peer RDT domain.
- */
-static int rdt_cdp_peer_get(struct rdt_resource *r, struct rdt_domain *d,
-			    struct rdt_resource **r_cdp,
-			    struct rdt_domain **d_cdp,
-			    enum resctrl_conf_type *peer_type)
+static enum resctrl_conf_type resctrl_peer_type(enum resctrl_conf_type my_type)
 {
-	struct rdt_resource *_r_cdp = NULL;
-	struct rdt_domain *_d_cdp = NULL;
-	int ret = 0;
-
-	switch (r->rid) {
-	case RDT_RESOURCE_L3DATA:
-		_r_cdp = &rdt_resources_all[RDT_RESOURCE_L3CODE].resctrl;
-		*peer_type = CDP_CODE;
-		break;
-	case RDT_RESOURCE_L3CODE:
-		_r_cdp =  &rdt_resources_all[RDT_RESOURCE_L3DATA].resctrl;
-		*peer_type = CDP_DATA;
-		break;
-	case RDT_RESOURCE_L2DATA:
-		_r_cdp =  &rdt_resources_all[RDT_RESOURCE_L2CODE].resctrl;
-		*peer_type = CDP_CODE;
-		break;
-	case RDT_RESOURCE_L2CODE:
-		_r_cdp =  &rdt_resources_all[RDT_RESOURCE_L2DATA].resctrl;
-		*peer_type = CDP_DATA;
-		break;
+	switch (my_type) {
+	case CDP_CODE:
+		return CDP_DATA;
+	case CDP_DATA:
+		return CDP_CODE;
 	default:
-		ret = -ENOENT;
-		goto out;
-	}
-
-	/*
-	 * When a new CPU comes online and CDP is enabled then the new
-	 * RDT domains (if any) associated with both CDP RDT resources
-	 * are added in the same CPU online routine while the
-	 * rdtgroup_mutex is held. It should thus not happen for one
-	 * RDT domain to exist and be associated with its RDT CDP
-	 * resource but there is no RDT domain associated with the
-	 * peer RDT CDP resource. Hence the WARN.
-	 */
-	_d_cdp = rdt_find_domain(_r_cdp, d->id, NULL);
-	if (WARN_ON(IS_ERR_OR_NULL(_d_cdp))) {
-		_r_cdp = NULL;
-		_d_cdp = NULL;
-		ret = -EINVAL;
+	case CDP_BOTH:
+		return CDP_BOTH;
 	}
-
-out:
-	*r_cdp = _r_cdp;
-	*d_cdp = _d_cdp;
-
-	return ret;
 }
 
 /**
@@ -1250,19 +1185,16 @@ static bool __rdtgroup_cbm_overlaps(struct rdt_resource *r, struct rdt_domain *d
 bool rdtgroup_cbm_overlaps(struct resctrl_schema *s, struct rdt_domain *d,
 			   unsigned long cbm, int closid, bool exclusive)
 {
-	enum resctrl_conf_type peer_type;
+	enum resctrl_conf_type peer_type = resctrl_peer_type(s->conf_type);
 	struct rdt_resource *r = s->res;
-	struct rdt_resource *r_cdp;
-	struct rdt_domain *d_cdp;
 
 	if (__rdtgroup_cbm_overlaps(r, d, cbm, closid, s->conf_type,
 				    exclusive))
 		return true;
 
-	if (rdt_cdp_peer_get(r, d, &r_cdp, &d_cdp, &peer_type) < 0)
+	if (!resctrl_arch_get_cdp_enabled(r->rid))
 		return false;
-
-	return  __rdtgroup_cbm_overlaps(r_cdp, d_cdp, cbm, closid, peer_type, exclusive);
+	return  __rdtgroup_cbm_overlaps(r, d, cbm, closid, peer_type, exclusive);
 }
 
 /**
@@ -2766,11 +2698,9 @@ static u32 cbm_ensure_valid(u32 _val, struct rdt_resource *r)
 static int __init_one_rdt_domain(struct rdt_domain *d, struct resctrl_schema *s,
 				 u32 closid)
 {
+	enum resctrl_conf_type peer_type = resctrl_peer_type(s->conf_type);
 	enum resctrl_conf_type t = s-> conf_type;
-	struct rdt_resource *r_cdp = NULL;
 	struct resctrl_staged_config *cfg;
-	enum resctrl_conf_type peer_type;
-	struct rdt_domain *d_cdp = NULL;
 	struct rdt_resource *r = s->res;
 	u32 used_b = 0, unused_b = 0;
 	u32 peer_ctl, ctrl_val;
@@ -2778,7 +2708,6 @@ static int __init_one_rdt_domain(struct rdt_domain *d, struct resctrl_schema *s,
 	enum rdtgrp_mode mode;
 	int i;
 
-	rdt_cdp_peer_get(r, d, &r_cdp, &d_cdp, &peer_type);
 	cfg = &d->staged_config[t];
 	cfg->have_new_ctrl = false;
 	cfg->new_ctrl = r->cache.shareable_bits;
@@ -2798,8 +2727,8 @@ static int __init_one_rdt_domain(struct rdt_domain *d, struct resctrl_schema *s,
 			 * usage to ensure there is no overlap
 			 * with an exclusive group.
 			 */
-			if (d_cdp)
-				resctrl_arch_get_config(r_cdp, d_cdp, i, peer_type, &peer_ctl);
+			if (resctrl_arch_get_cdp_enabled(r->rid))
+				resctrl_arch_get_config(r, d, i, peer_type, &peer_ctl);
 			else
 				peer_ctl = 0;
 			resctrl_arch_get_config(r, d, i, s->conf_type, &ctrl_val);
Signature · 1 line
2.28.0
author reviewer
#48 lore raw

PATCH 24/24 x86/resctrl: Merge the CDP resources

In reply to James Morse

Now that resctrl uses the schema's configuration type as the source of CODE/DATA configuration styles, and there is only one configuration array between the three views of the resource, remove the CODE and DATA aliases.

This means the arch code only needs to describe the hardware to resctrl, which will then create the separate CODE/DATA schema for its ABI.

Add a helper to add schema with a the CDP suffix if CDP is enabled.

---
 arch/x86/kernel/cpu/resctrl/core.c     | 193 ++-----------------------
 arch/x86/kernel/cpu/resctrl/internal.h |   4 -
 arch/x86/kernel/cpu/resctrl/rdtgroup.c | 113 ++++++++-------
 3 files changed, 72 insertions(+), 238 deletions(-)
arch/x86/kernel/cpu/resctrl/core.c +13 -180
index 01d010977367..57d4131fdd80 100644
@@ -78,42 +78,6 @@ struct rdt_hw_resource rdt_resources_all[] = {
 		.msr_base		= MSR_IA32_L3_CBM_BASE,
 		.msr_update		= cat_wrmsr,
 	},
-	[RDT_RESOURCE_L3DATA] =
-	{
-		.conf_type			= CDP_DATA,
-		.resctrl = {
-			.rid			= RDT_RESOURCE_L3DATA,
-			.name			= "L3DATA",
-			.cache_level		= 3,
-			.cache = {
-				.min_cbm_bits	= 1,
-			},
-			.domains		= domain_init(RDT_RESOURCE_L3DATA),
-			.parse_ctrlval		= parse_cbm,
-			.format_str		= "%d=%0*x",
-			.fflags			= RFTYPE_RES_CACHE,
-		},
-		.msr_base		= MSR_IA32_L3_CBM_BASE,
-		.msr_update		= cat_wrmsr,
-	},
-	[RDT_RESOURCE_L3CODE] =
-	{
-		.conf_type			= CDP_CODE,
-		.resctrl = {
-			.rid			= RDT_RESOURCE_L3CODE,
-			.name			= "L3CODE",
-			.cache_level		= 3,
-			.cache = {
-				.min_cbm_bits	= 1,
-			},
-			.domains		= domain_init(RDT_RESOURCE_L3CODE),
-			.parse_ctrlval		= parse_cbm,
-			.format_str		= "%d=%0*x",
-			.fflags			= RFTYPE_RES_CACHE,
-		},
-		.msr_base		= MSR_IA32_L3_CBM_BASE,
-		.msr_update		= cat_wrmsr,
-	},
 	[RDT_RESOURCE_L2] =
 	{
 		.conf_type			= CDP_BOTH,
@@ -132,42 +96,6 @@ struct rdt_hw_resource rdt_resources_all[] = {
 		.msr_base		= MSR_IA32_L2_CBM_BASE,
 		.msr_update		= cat_wrmsr,
 	},
-	[RDT_RESOURCE_L2DATA] =
-	{
-		.conf_type			= CDP_DATA,
-		.resctrl = {
-			.rid			= RDT_RESOURCE_L2DATA,
-			.name			= "L2DATA",
-			.cache_level		= 2,
-			.cache = {
-				.min_cbm_bits	= 1,
-			},
-			.domains		= domain_init(RDT_RESOURCE_L2DATA),
-			.parse_ctrlval		= parse_cbm,
-			.format_str		= "%d=%0*x",
-			.fflags			= RFTYPE_RES_CACHE,
-		},
-		.msr_base		= MSR_IA32_L2_CBM_BASE,
-		.msr_update		= cat_wrmsr,
-	},
-	[RDT_RESOURCE_L2CODE] =
-	{
-		.conf_type			= CDP_CODE,
-		.resctrl = {
-			.rid			= RDT_RESOURCE_L2CODE,
-			.name			= "L2CODE",
-			.cache_level		= 2,
-			.cache = {
-				.min_cbm_bits	= 1,
-			},
-			.domains		= domain_init(RDT_RESOURCE_L2CODE),
-			.parse_ctrlval		= parse_cbm,
-			.format_str		= "%d=%0*x",
-			.fflags			= RFTYPE_RES_CACHE,
-		},
-		.msr_base		= MSR_IA32_L2_CBM_BASE,
-		.msr_update		= cat_wrmsr,
-	},
 	[RDT_RESOURCE_MBA] =
 	{
 		.conf_type			= CDP_BOTH,
@@ -339,40 +267,16 @@ static void rdt_get_cache_alloc_cfg(int idx, struct rdt_resource *r)
 	r->alloc_enabled = true;
 }
 
-static void rdt_get_cdp_config(int level, int type)
-{
-	struct rdt_resource *r_l = &rdt_resources_all[level].resctrl;
-	struct rdt_hw_resource *hw_res_l = resctrl_to_arch_res(r_l);
-	struct rdt_resource *r = &rdt_resources_all[type].resctrl;
-	struct rdt_hw_resource *hw_res = resctrl_to_arch_res(r);
-
-	hw_res->num_closid = hw_res_l->num_closid;
-	r->cache.cbm_len = r_l->cache.cbm_len;
-	r->default_ctrl = r_l->default_ctrl;
-	r->cache.shareable_bits = r_l->cache.shareable_bits;
-	r->data_width = (r->cache.cbm_len + 3) / 4;
-	r->alloc_capable = true;
-	hw_res_l->cdp_capable = true;
-	hw_res->cdp_capable = true;
-	/*
-	 * By default, CDP is disabled. CDP can be enabled by mount parameter
-	 * "cdp" during resctrl file system mount time.
-	 */
-	r->alloc_enabled = false;
-	hw_res_l->cdp_enabled = false;
-	hw_res->cdp_enabled = false;
-}
-
 static void rdt_get_cdp_l3_config(void)
 {
-	rdt_get_cdp_config(RDT_RESOURCE_L3, RDT_RESOURCE_L3DATA);
-	rdt_get_cdp_config(RDT_RESOURCE_L3, RDT_RESOURCE_L3CODE);
+	rdt_resources_all[RDT_RESOURCE_L3].cdp_capable = true;
+	rdt_resources_all[RDT_RESOURCE_L3].cdp_enabled = false;
 }
 
 static void rdt_get_cdp_l2_config(void)
 {
-	rdt_get_cdp_config(RDT_RESOURCE_L2, RDT_RESOURCE_L2DATA);
-	rdt_get_cdp_config(RDT_RESOURCE_L2, RDT_RESOURCE_L2CODE);
+	rdt_resources_all[RDT_RESOURCE_L2].cdp_capable = true;
+	rdt_resources_all[RDT_RESOURCE_L2].cdp_enabled = false;
 }
 
 static void
@@ -509,72 +413,6 @@ void setup_default_ctrlval(struct rdt_resource *r, u32 *dc, u32 *dm)
 	}
 }
 
-/*
- * temporary.
- * This relies on L2 or L3 being allocated before their CODE/DATA aliases
- */
-static u32 *alloc_ctrlval_array(struct rdt_resource *r, struct rdt_domain *d,
-				bool mba_sc)
-{
-	/* these are for the underlying hardware, they may not match r/d */
-	struct rdt_domain *underlying_domain;
-	struct rdt_hw_resource *hw_res;
-	struct rdt_hw_domain *hw_dom;
-	bool remapped;
-
-	switch (r->rid) {
-	case RDT_RESOURCE_L3DATA:
-	case RDT_RESOURCE_L3CODE:
-		hw_res = &rdt_resources_all[RDT_RESOURCE_L3];
-		remapped = true;
-		break;
-	case RDT_RESOURCE_L2DATA:
-	case RDT_RESOURCE_L2CODE:
-		hw_res = &rdt_resources_all[RDT_RESOURCE_L2];
-		remapped = true;
-		break;
-	default:
-		hw_res = resctrl_to_arch_res(r);
-		remapped = false;
-	}
-
-	/*
-	 * If we changed the resource, we need to search for the underlying
-	 * domain. Doing this for all resources would make it tricky to add the
-	 * first resource, as domains aren't added to a resource list until
-	 * after the ctrlval arrays have been allocated.
-	 */
-	if (remapped)
-		underlying_domain = rdt_find_domain(&hw_res->resctrl, d->id,
-						    NULL);
-	else
-		underlying_domain = d;
-	hw_dom = resctrl_to_arch_dom(underlying_domain);
-
-	if (mba_sc) {
-		if (hw_dom->mbps_val)
-			return hw_dom->mbps_val;
-		return kmalloc_array(hw_res->num_closid,
-				     sizeof(*hw_dom->mbps_val), GFP_KERNEL);
-	} else {
-		if (hw_dom->ctrl_val)
-			return hw_dom->ctrl_val;
-		return kmalloc_array(hw_res->num_closid,
-				     sizeof(*hw_dom->ctrl_val), GFP_KERNEL);
-	}
-}
-
-/* Only kfree() for L2/L3, not the CODE/DATA aliases */
-static void free_ctrlval_arrays(struct rdt_resource *r, struct rdt_domain *d)
-{
-	struct rdt_hw_domain *hw_dom = resctrl_to_arch_dom(d);
-
-	if (r->rid == RDT_RESOURCE_L2 || r->rid == RDT_RESOURCE_L3) {
-		kfree(hw_dom->ctrl_val);
-		kfree(hw_dom->mbps_val);
-	}
-}
-
 static int domain_setup_ctrlval(struct rdt_resource *r, struct rdt_domain *d)
 {
 	struct rdt_hw_resource *hw_res = resctrl_to_arch_res(r);
@@ -582,14 +420,16 @@ static int domain_setup_ctrlval(struct rdt_resource *r, struct rdt_domain *d)
 	struct msr_param m;
 	u32 *dc, *dm;
 
-	dc = alloc_ctrlval_array(r, d, false);
+	dc = kmalloc_array(hw_res->num_closid, sizeof(*hw_dom->ctrl_val),
+			   GFP_KERNEL);
 	if (!dc)
 		return -ENOMEM;
 	hw_dom->ctrl_val = dc;
 
-	dm = alloc_ctrlval_array(r, d, true);
+	dm = kmalloc_array(hw_res->num_closid, sizeof(*hw_dom->mbps_val),
+			   GFP_KERNEL);
 	if (!dm) {
-		free_ctrlval_arrays(r, d);
+		kfree(dc);
 		return -ENOMEM;
 	}
 	hw_dom->mbps_val = dm;
@@ -743,7 +583,8 @@ static void domain_remove_cpu(int cpu, struct rdt_resource *r)
 		if (d->plr)
 			d->plr->d = NULL;
 
-		free_ctrlval_arrays(r, d);
+		kfree(hw_dom->ctrl_val);
+		kfree(hw_dom->mbps_val);
 		bitmap_free(d->rmid_busy_llc);
 		kfree(d->mbm_total);
 		kfree(d->mbm_local);
@@ -1014,11 +855,7 @@ static __init void rdt_init_res_defs_intel(void)
 		r = &rdt_resources_all[i].resctrl;
 
 		if (r->rid == RDT_RESOURCE_L3 ||
-		    r->rid == RDT_RESOURCE_L3DATA ||
-		    r->rid == RDT_RESOURCE_L3CODE ||
-		    r->rid == RDT_RESOURCE_L2 ||
-		    r->rid == RDT_RESOURCE_L2DATA ||
-		    r->rid == RDT_RESOURCE_L2CODE) {
+		    r->rid == RDT_RESOURCE_L2) {
 			r->cache.arch_has_sparse_bitmaps = false;
 			r->cache.arch_has_empty_bitmaps = false;
 		} else if (r->rid == RDT_RESOURCE_MBA) {
@@ -1039,11 +876,7 @@ static __init void rdt_init_res_defs_amd(void)
 		r = &rdt_resources_all[i].resctrl;
 
 		if (r->rid == RDT_RESOURCE_L3 ||
-		    r->rid == RDT_RESOURCE_L3DATA ||
-		    r->rid == RDT_RESOURCE_L3CODE ||
-		    r->rid == RDT_RESOURCE_L2 ||
-		    r->rid == RDT_RESOURCE_L2DATA ||
-		    r->rid == RDT_RESOURCE_L2CODE) {
+		    r->rid == RDT_RESOURCE_L2) {
 			r->cache.arch_has_sparse_bitmaps = true;
 			r->cache.arch_has_empty_bitmaps = true;
 		} else if (r->rid == RDT_RESOURCE_MBA) {
arch/x86/kernel/cpu/resctrl/internal.h +0 -4
index f039fd9f4f4f..4f04c9cac6f5 100644
@@ -403,11 +403,7 @@ extern struct dentry *debugfs_resctrl;
 
 enum resctrl_res_level {
 	RDT_RESOURCE_L3,
-	RDT_RESOURCE_L3DATA,
-	RDT_RESOURCE_L3CODE,
 	RDT_RESOURCE_L2,
-	RDT_RESOURCE_L2DATA,
-	RDT_RESOURCE_L2CODE,
 	RDT_RESOURCE_MBA,
 
 	/* Must be the last */
arch/x86/kernel/cpu/resctrl/rdtgroup.c +59 -54

174 lines not on this page · raw message

Signature · 1 line
2.28.0
Reviewed-by on the v1 series
#49 lore raw
In reply to James Morse

Hi James,

Quoted 73 lines · James Morse

Hi folks,

This series re-folds the resctrl code so the CDP resources (L3CODE et al) behaviour is all contained in the filesystem parts, with a minimum amount of arch specific code.

Arm have some CPU support for dividing caches into portions, and applying bandwidth limits at various points in the SoC. The collective term for these features is MPAM: Memory Partitioning and Monitoring.

MPAM is similar enough to Intel RDT, that it should use the defacto linux
interface: resctrl. This filesystem currently lives under arch/x86, and is tightly coupled to the architecture.
Ultimately, my plan is to split the existing resctrl code up to have an arch<->fs abstraction, then move all the bits out to fs/resctrl. From there MPAM can be wired up.

x86 might have two resources with cache controls, (L2 and L3) but has extra copies for CDP: L{2,3}{CODE,DATA}, which are marked as enabled if CDP is enabled for the corresponding cache.

MPAM has an equivalent feature to CDP, but its a property of the CPU, not the cache. Resctrl needs to have x86's odd/even behaviour, as that its the ABI, but this isn't how the MPAM hardware works. It is entirely possible that an in-kernel user of MPAM would not be using CDP, whereas resctrl is.
Pretending L3CODE and L3DATA are entirely separate resources is a neat trick, but doing this is specific to x86.
Doing this leaves the arch code in control of various parts of the filesystem ABI: the resources names, and the way the schemata are parsed. Allowing this stuff to vary between architectures is bad for user space.

This series collapses the CODE/DATA resources, moving all the user-visible resctrl ABI into the filesystem code. CDP becomes the type of configuration being applied to a cache. This is done by adding a struct resctrl_schema to the parts of resctrl that will move to fs. This holds the arch-code resource that is in use for this schema, along with other properties like the name, and whether the configuration being applied is CODE/DATA/BOTH.

This lets us fold the extra resources out of the arch code so that they don't need to be duplicated if the equivalent feature to CDP is missing, or implemented in a different way.

The first two patches split the resource and domain structs to have an arch specific 'hw' portion, and the rest that is visible to resctrl. Future series massage the resctrl code so there are no accesses to 'hw' structures in the parts of resctrl that will move to fs, providing helpers where necessary.

Since anyone last looked at this, the CDP property has been made per-resource instead of global. MPAM will need to make this global in the arch code, as CODE/DATA closid are based on how the CPU tags traffic, not how the cache interprets it. resctrl sets CDP enabled on a resource, but reads it back on each one.
The attempt to keep closids as-used-by-resctrl and closids as-written-to-hw appart has been dropped.
There are two copies of num_closid. The version private to the arch code is the value discovered from hardware. resctrl has its own version, which it may write to, which is exposed to user-space. This lets resctrl do its odd/even thing, even if thats not how the hardware works.

This series adds temporary scaffolding, which it removes a few patches later. This is to allow things like the ctrlval arrays and resources to be merged separately, which should make is easier to bisect. These things are marked temporary, and should all be gone by the end of the series.

This series is a little rough around the monitors, would a fake struct resctrl_schema for the monitors simplify things, or be a source of bugs?

Quoted 5 lines

This series is based on v5.10-rc1, and can be retrieved from:
git://git.kernel.org/pub/scm/linux/kernel/git/morse/linux.git mpam/resctrl_merge_cdp/v1

Parts were previously posted as an RFC here:
https://lore.kernel.org/lkml/[email protected]/

Jamie

maintainer
#50 lore raw
In reply to James Morse

Hi James,

Quoted 5 lines · James Morse

MPAM has an equivalent feature to CDP, but its a property of the CPU, not the cache. Resctrl needs to have x86's odd/even behaviour, as that its the ABI, but this isn't how the MPAM hardware works. It is entirely possible that an in-kernel user of MPAM would not be using CDP, whereas resctrl is.

The above seems to distinguish between "in-kernel user of MPAM" and resctrl (now obtaining support for MPAM). Could you please provide more details on the "in-kernel user of MPAM" and elaborate on how these two usages are expected to interact with MPAM concurrently?

Thank you

Reinette

author reviewer
#51 lore raw

Hi Reinette,

Quoted 10 lines · Reinette Chatre
Quoted 5 lines · James Morse

MPAM has an equivalent feature to CDP, but its a property of the CPU, not the cache. Resctrl needs to have x86's odd/even behaviour, as that its the ABI, but this isn't how the MPAM hardware works. It is entirely possible that an in-kernel user of MPAM would not be using CDP, whereas resctrl is.

The above seems to distinguish between "in-kernel user of MPAM" and resctrl (now obtaining support for MPAM). Could you please provide more details on the "in-kernel user of MPAM" and elaborate on how these two usages are expected to interact with MPAM concurrently?

This is a badly phrased reference to all the bits of MPAM that are left on the floor after the resctrl support is plumbed up.

Currently none of the software exists, but MPAM also has support for: virtualisation, the interrupt-controller (GIC) and the IO-MMU. None of these things are exposed via resctrl, so they either need new schema (which must also work for x86), or handling 'invisibly' in the kernel.

Virtualisation is probably the easiest example: With MPAM, the guest may be 'using CDP' whereas the host is not, or vice-versa.
The guest will never be allowed to access the MMIO configuration directly, it will be managed via the host's driver. Now the host's driver has to handle CDP-on and CDP-off configurations.
Keeping the odd/even CDP stuff in resctrl means the arch-code/driver doesn't need to know or care about this stuff if the hardware doesn't.

If the interrupt-controller or IO-MMU consume closid/rmid, then I'd describe them as in-kernel users (as the kernel owns their configuration). These would never use CDP as they don't fetch instructions.

How do I envision these things working concurrently?
(a) closid/rmid can be reserved before resctrl is mounted, or
(b) allocated by user-space and handed back to the kernel (e.g. virtualisation).

The ctrlval values move to belong to the arch-code/driver, so if 'something' changes the configuration behind resctrls back, the new schema values are immediately visible via the corresponding schema file in case (b). In case (a), resctrl would never look at those closid, but it wouldn't matter if it did.

(the counter-example is mba_sc, which may need to convert the current ctrlval back to a mbps_val if its being managed by something other than resctrl)

Thanks,

James