Skip to contentSkip to conversation

x86/resctrl: Merge the CDP resources

PATCH Superseded James Morse 3 participants 42 messages Mar 12, 2021 – Apr 8, 2021

Subsystems (2)

v2 of 7 Superseded Latest is v7 →

lore
Share & tools
lore

James Morse

author reviewer
#0 lore raw

v2 cover x86/resctrl: Merge the CDP resources

Hi folks,

Thanks to Reinette and Jamie for the comments on v1. Major changes in v2 are to keep the closid in resctrl_arch_update_domains(), eliminating one patch, splitting another that was making two sorts of change, and to re-order the first few patches. See each patches changelog for more.
----

This series re-folds the resctrl code so the CDP resources (L3CODE et al) behaviour is all contained in the filesystem parts, with a minimum amount of arch specific code.

Arm have some CPU support for dividing caches into portions, and applying bandwidth limits at various points in the SoC. The collective term for these features is MPAM: Memory Partitioning and Monitoring.

MPAM is similar enough to Intel RDT, that it should use the defacto linux
interface: resctrl. This filesystem currently lives under arch/x86, and is tightly coupled to the architecture.
Ultimately, my plan is to split the existing resctrl code up to have an arch<->fs abstraction, then move all the bits out to fs/resctrl. From there MPAM can be wired up.

x86 might have two resources with cache controls, (L2 and L3) but has extra copies for CDP: L{2,3}{CODE,DATA}, which are marked as enabled if CDP is enabled for the corresponding cache.

MPAM has an equivalent feature to CDP, but its a property of the CPU, not the cache. Resctrl needs to have x86's odd/even behaviour, as that its the ABI, but this isn't how the MPAM hardware works. It is entirely possible that an in-kernel user of MPAM would not be using CDP, whereas resctrl is.
Pretending L3CODE and L3DATA are entirely separate resources is a neat trick, but doing this is specific to x86.
Doing this leaves the arch code in control of various parts of the filesystem ABI: the resources names, and the way the schemata are parsed. Allowing this stuff to vary between architectures is bad for user space.

This series collapses the CODE/DATA resources, moving all the user-visible resctrl ABI into what becomes the filesystem code. CDP becomes the type of configuration being applied to a cache. This is done by adding a struct resctrl_schema to the parts of resctrl that will move to fs. This holds the arch-code resource that is in use for this schema, along with other properties like the name, and whether the configuration being applied is CODE/DATA/BOTH.

This lets us fold the extra resources out of the arch code so that they don't need to be duplicated if the equivalent feature to CDP is missing, or implemented in a different way.

The first two patches split the resource and domain structs to have an arch specific 'hw' portion, and the rest that is visible to resctrl. Future series massage the resctrl code so there are no accesses to 'hw' structures in the parts of resctrl that will move to fs, providing helpers where necessary.

This series adds temporary scaffolding, which it removes a few patches later. This is to allow things like the ctrlval arrays and resources to be merged separately, which should make is easier to bisect. These things are marked temporary, and should all be gone by the end of the series.

This series is a little rough around the monitors, would a fake struct resctrl_schema for the monitors simplify things, or be a source of bugs?

This series is based on v5.12-rc2, and can be retrieved from:
git://git.kernel.org/pub/scm/linux/kernel/git/morse/linux.git mpam/resctrl_merge_cdp/v2

v1 was posted here:
https://lore.kernel.org/lkml/[email protected]/

Parts were previously posted as an RFC here:
https://lore.kernel.org/lkml/[email protected]/

Thanks,

James Morse (24):
x86/resctrl: Split struct rdt_resource
x86/resctrl: Split struct rdt_domain
x86/resctrl: Add a separate schema list for resctrl x86/resctrl: Pass the schema in info dir's private pointer x86/resctrl: Label the resources with their configuration type x86/resctrl: Walk the resctrl schema list instead of an arch list x86/resctrl: Store the effective num_closid in the schema x86/resctrl: Add resctrl_arch_get_num_closid() x86/resctrl: Pass the schema to resctrl filesystem functions x86/resctrl: Swizzle rdt_resource and resctrl_schema in
pseudo_lock_region
x86/resctrl: Move the schemata names into struct resctrl_schema x86/resctrl: Group staged configuration into a separate struct x86/resctrl: Allow different CODE/DATA configurations to be staged x86/resctrl: Rename update_domains() resctrl_arch_update_domains() x86/resctrl: Add a helper to read a closid's configuration x86/resctrl: Add a helper to read/set the CDP configuration x86/resctrl: Use cdp_enabled in rdt_domain_reconfigure_cdp() x86/resctrl: Pass configuration type to resctrl_arch_get_config() x86/resctrl: Make ctrlval arrays the same size x86/resctrl: Apply offset correction when config is staged x86/resctrl: Calculate the index from the configuration type x86/resctrl: Merge the ctrl_val arrays
x86/resctrl: Remove rdt_cdp_peer_get()
x86/resctrl: Merge the CDP resources

 arch/x86/kernel/cpu/resctrl/core.c        | 272 +++++--------
 arch/x86/kernel/cpu/resctrl/ctrlmondata.c | 164 +++++---
 arch/x86/kernel/cpu/resctrl/internal.h    | 218 +++--------
 arch/x86/kernel/cpu/resctrl/monitor.c     |  44 ++-
 arch/x86/kernel/cpu/resctrl/pseudo_lock.c |  12 +-
 arch/x86/kernel/cpu/resctrl/rdtgroup.c    | 447 ++++++++++++----------
 include/linux/resctrl.h                   | 176 +++++++++
 7 files changed, 741 insertions(+), 592 deletions(-)
Signature · 1 line
2.30.0
author reviewer
#1 lore raw

PATCH v2 1/24 x86/resctrl: Split struct rdt_resource

resctrl is the defacto Linux ABI for SoC resource partitioning features. To support it on another architecture, it needs to be abstracted from the features provided by Intel RDT and AMD PQoS, and moved to /fs/.

Start by splitting struct rdt_resource, (the name is kept to keep the noise down), and add some type-trickery to keep the foreach helpers working.

Move everything that is particular to resctrl into a new header file, keeping the x86 hardware accessors where they are. resctrl code paths touching a 'hw' struct indicates where an abstraction is needed.

Splitting this structure only moves types around, and should not lead to any change in behaviour.

Splitting rdt_domain up in a similar way happens in the next patch.

---
Changes since v1:
* Tabs space and other whitespace
* Restored for_each_rdt_resource() calls in arch code

---
 arch/x86/kernel/cpu/resctrl/core.c        | 250 ++++++++++++----------
 arch/x86/kernel/cpu/resctrl/ctrlmondata.c |  14 +-
 arch/x86/kernel/cpu/resctrl/internal.h    | 139 +++---------
 arch/x86/kernel/cpu/resctrl/monitor.c     |  32 +--
 arch/x86/kernel/cpu/resctrl/pseudo_lock.c |   4 +-
 arch/x86/kernel/cpu/resctrl/rdtgroup.c    |  68 +++---
 include/linux/resctrl.h                   | 111 ++++++++++
 7 files changed, 347 insertions(+), 271 deletions(-)
arch/x86/kernel/cpu/resctrl/core.c +141 -109
index 698bb26aeb6e..6cd84d54086f 100644
@@ -57,120 +57,134 @@ static void
 mba_wrmsr_amd(struct rdt_domain *d, struct msr_param *m,
 	      struct rdt_resource *r);
 
-#define domain_init(id) LIST_HEAD_INIT(rdt_resources_all[id].domains)
+#define domain_init(id) LIST_HEAD_INIT(rdt_resources_all[id].resctrl.domains)
 
-struct rdt_resource rdt_resources_all[] = {
+struct rdt_hw_resource rdt_resources_all[] = {
 	[RDT_RESOURCE_L3] =
 	{
-		.rid			= RDT_RESOURCE_L3,
-		.name			= "L3",
-		.domains		= domain_init(RDT_RESOURCE_L3),
+		.resctrl = {
+			.rid			= RDT_RESOURCE_L3,
+			.name			= "L3",
+			.cache_level		= 3,
+			.cache = {
+				.min_cbm_bits	= 1,
+				.cbm_idx_mult	= 1,
+				.cbm_idx_offset	= 0,
+			},
+			.domains		= domain_init(RDT_RESOURCE_L3),
+			.parse_ctrlval		= parse_cbm,
+			.format_str		= "%d=%0*x",
+			.fflags			= RFTYPE_RES_CACHE,
+		},
 		.msr_base		= MSR_IA32_L3_CBM_BASE,
 		.msr_update		= cat_wrmsr,
-		.cache_level		= 3,
-		.cache = {
-			.min_cbm_bits	= 1,
-			.cbm_idx_mult	= 1,
-			.cbm_idx_offset	= 0,
-		},
-		.parse_ctrlval		= parse_cbm,
-		.format_str		= "%d=%0*x",
-		.fflags			= RFTYPE_RES_CACHE,
 	},
 	[RDT_RESOURCE_L3DATA] =
 	{
-		.rid			= RDT_RESOURCE_L3DATA,
-		.name			= "L3DATA",
-		.domains		= domain_init(RDT_RESOURCE_L3DATA),
+		.resctrl = {
+			.rid			= RDT_RESOURCE_L3DATA,
+			.name			= "L3DATA",
+			.cache_level		= 3,
+			.cache = {
+				.min_cbm_bits	= 1,
+				.cbm_idx_mult	= 2,
+				.cbm_idx_offset	= 0,
+			},
+			.domains		= domain_init(RDT_RESOURCE_L3DATA),
+			.parse_ctrlval		= parse_cbm,
+			.format_str		= "%d=%0*x",
+			.fflags			= RFTYPE_RES_CACHE,
+		},
 		.msr_base		= MSR_IA32_L3_CBM_BASE,
 		.msr_update		= cat_wrmsr,
-		.cache_level		= 3,
-		.cache = {
-			.min_cbm_bits	= 1,
-			.cbm_idx_mult	= 2,
-			.cbm_idx_offset	= 0,
-		},
-		.parse_ctrlval		= parse_cbm,
-		.format_str		= "%d=%0*x",
-		.fflags			= RFTYPE_RES_CACHE,
 	},
 	[RDT_RESOURCE_L3CODE] =
 	{
-		.rid			= RDT_RESOURCE_L3CODE,
-		.name			= "L3CODE",
-		.domains		= domain_init(RDT_RESOURCE_L3CODE),
+		.resctrl = {
+			.rid			= RDT_RESOURCE_L3CODE,
+			.name			= "L3CODE",
+			.cache_level		= 3,
+			.cache = {
+				.min_cbm_bits	= 1,
+				.cbm_idx_mult	= 2,
+				.cbm_idx_offset	= 1,
+			},
+			.domains		= domain_init(RDT_RESOURCE_L3CODE),
+			.parse_ctrlval		= parse_cbm,
+			.format_str		= "%d=%0*x",
+			.fflags			= RFTYPE_RES_CACHE,
+		},
 		.msr_base		= MSR_IA32_L3_CBM_BASE,
 		.msr_update		= cat_wrmsr,
-		.cache_level		= 3,
-		.cache = {
-			.min_cbm_bits	= 1,
-			.cbm_idx_mult	= 2,
-			.cbm_idx_offset	= 1,
-		},
-		.parse_ctrlval		= parse_cbm,
-		.format_str		= "%d=%0*x",
-		.fflags			= RFTYPE_RES_CACHE,
 	},
 	[RDT_RESOURCE_L2] =
 	{
-		.rid			= RDT_RESOURCE_L2,
-		.name			= "L2",
-		.domains		= domain_init(RDT_RESOURCE_L2),
+		.resctrl = {
+			.rid			= RDT_RESOURCE_L2,
+			.name			= "L2",
+			.cache_level		= 2,
+			.cache = {
+				.min_cbm_bits	= 1,
+				.cbm_idx_mult	= 1,
+				.cbm_idx_offset	= 0,
+			},
+			.domains		= domain_init(RDT_RESOURCE_L2),
+			.parse_ctrlval		= parse_cbm,
+			.format_str		= "%d=%0*x",
+			.fflags			= RFTYPE_RES_CACHE,
+		},
 		.msr_base		= MSR_IA32_L2_CBM_BASE,
 		.msr_update		= cat_wrmsr,
-		.cache_level		= 2,
-		.cache = {
-			.min_cbm_bits	= 1,
-			.cbm_idx_mult	= 1,
-			.cbm_idx_offset	= 0,
-		},
-		.parse_ctrlval		= parse_cbm,
-		.format_str		= "%d=%0*x",
-		.fflags			= RFTYPE_RES_CACHE,
 	},
 	[RDT_RESOURCE_L2DATA] =
 	{
-		.rid			= RDT_RESOURCE_L2DATA,
-		.name			= "L2DATA",
-		.domains		= domain_init(RDT_RESOURCE_L2DATA),
+		.resctrl = {
+			.rid			= RDT_RESOURCE_L2DATA,
+			.name			= "L2DATA",
+			.cache_level		= 2,
+			.cache = {
+				.min_cbm_bits	= 1,
+				.cbm_idx_mult	= 2,
+				.cbm_idx_offset	= 0,
+			},
+			.domains		= domain_init(RDT_RESOURCE_L2DATA),
+			.parse_ctrlval		= parse_cbm,
+			.format_str		= "%d=%0*x",
+			.fflags			= RFTYPE_RES_CACHE,
+		},
 		.msr_base		= MSR_IA32_L2_CBM_BASE,
 		.msr_update		= cat_wrmsr,
-		.cache_level		= 2,
-		.cache = {
-			.min_cbm_bits	= 1,
-			.cbm_idx_mult	= 2,
-			.cbm_idx_offset	= 0,
-		},
-		.parse_ctrlval		= parse_cbm,
-		.format_str		= "%d=%0*x",
-		.fflags			= RFTYPE_RES_CACHE,
 	},
 	[RDT_RESOURCE_L2CODE] =
 	{
-		.rid			= RDT_RESOURCE_L2CODE,
-		.name			= "L2CODE",
-		.domains		= domain_init(RDT_RESOURCE_L2CODE),
+		.resctrl = {
+			.rid			= RDT_RESOURCE_L2CODE,
+			.name			= "L2CODE",
+			.cache_level		= 2,
+			.cache = {
+				.min_cbm_bits	= 1,
+				.cbm_idx_mult	= 2,
+				.cbm_idx_offset	= 1,
+			},
+			.domains		= domain_init(RDT_RESOURCE_L2CODE),
+			.parse_ctrlval		= parse_cbm,
+			.format_str		= "%d=%0*x",
+			.fflags			= RFTYPE_RES_CACHE,
+		},
 		.msr_base		= MSR_IA32_L2_CBM_BASE,
 		.msr_update		= cat_wrmsr,
-		.cache_level		= 2,
-		.cache = {
-			.min_cbm_bits	= 1,
-			.cbm_idx_mult	= 2,
-			.cbm_idx_offset	= 1,
-		},
-		.parse_ctrlval		= parse_cbm,
-		.format_str		= "%d=%0*x",
-		.fflags			= RFTYPE_RES_CACHE,
 	},
 	[RDT_RESOURCE_MBA] =
 	{
-		.rid			= RDT_RESOURCE_MBA,
-		.name			= "MB",
-		.domains		= domain_init(RDT_RESOURCE_MBA),
-		.cache_level		= 3,
-		.parse_ctrlval		= parse_bw,
-		.format_str		= "%d=%*u",
-		.fflags			= RFTYPE_RES_MB,
+		.resctrl = {
+			.rid			= RDT_RESOURCE_MBA,
+			.name			= "MB",
+			.cache_level		= 3,
+			.domains		= domain_init(RDT_RESOURCE_MBA),
+			.parse_ctrlval		= parse_bw,
+			.format_str		= "%d=%*u",
+			.fflags			= RFTYPE_RES_MB,
+		},
 	},
 };
 
@@ -199,7 +213,8 @@ static unsigned int cbm_idx(struct rdt_resource *r, unsigned int closid)
  */
 static inline void cache_alloc_hsw_probe(void)
 {
-	struct rdt_resource *r  = &rdt_resources_all[RDT_RESOURCE_L3];
+	struct rdt_hw_resource *hw_res = &rdt_resources_all[RDT_RESOURCE_L3];
+	struct rdt_resource *r  = &hw_res->resctrl;
 	u32 l, h, max_cbm = BIT_MASK(20) - 1;
 
 	if (wrmsr_safe(MSR_IA32_L3_CBM_BASE, max_cbm, 0))
@@ -211,7 +226,7 @@ static inline void cache_alloc_hsw_probe(void)
 	if (l != max_cbm)
 		return;
 
-	r->num_closid = 4;
+	hw_res->num_closid = 4;
 	r->default_ctrl = max_cbm;
 	r->cache.cbm_len = 20;
 	r->cache.shareable_bits = 0xc0000;
@@ -225,7 +240,7 @@ static inline void cache_alloc_hsw_probe(void)
 bool is_mba_sc(struct rdt_resource *r)
 {
 	if (!r)
-		return rdt_resources_all[RDT_RESOURCE_MBA].membw.mba_sc;
+		return rdt_resources_all[RDT_RESOURCE_MBA].resctrl.membw.mba_sc;
 
 	return r->membw.mba_sc;
 }
@@ -253,12 +268,13 @@ static inline bool rdt_get_mb_table(struct rdt_resource *r)
 
 static bool __get_mem_config_intel(struct rdt_resource *r)
 {
+	struct rdt_hw_resource *hw_res = resctrl_to_arch_res(r);
 	union cpuid_0x10_3_eax eax;
 	union cpuid_0x10_x_edx edx;
 	u32 ebx, ecx, max_delay;
 
 	cpuid_count(0x00000010, 3, &eax.full, &ebx, &ecx, &edx.full);
-	r->num_closid = edx.split.cos_max + 1;
+	hw_res->num_closid = edx.split.cos_max + 1;
 	max_delay = eax.split.max_delay + 1;
 	r->default_ctrl = MAX_MBA_BW;
 	r->membw.arch_needs_linear = true;
@@ -287,12 +303,13 @@ static bool __get_mem_config_intel(struct rdt_resource *r)
 
 static bool __rdt_get_mem_config_amd(struct rdt_resource *r)
 {
+	struct rdt_hw_resource *hw_res = resctrl_to_arch_res(r);
 	union cpuid_0x10_3_eax eax;
 	union cpuid_0x10_x_edx edx;
 	u32 ebx, ecx;
 
 	cpuid_count(0x80000020, 1, &eax.full, &ebx, &ecx, &edx.full);
-	r->num_closid = edx.split.cos_max + 1;
+	hw_res->num_closid = edx.split.cos_max + 1;
 	r->default_ctrl = MAX_MBA_BW_AMD;
 
 	/* AMD does not use delay */
@@ -317,12 +334,13 @@ static bool __rdt_get_mem_config_amd(struct rdt_resource *r)
 
 static void rdt_get_cache_alloc_cfg(int idx, struct rdt_resource *r)
 {
+	struct rdt_hw_resource *hw_res = resctrl_to_arch_res(r);
 	union cpuid_0x10_1_eax eax;
 	union cpuid_0x10_x_edx edx;
 	u32 ebx, ecx;
 
 	cpuid_count(0x00000010, idx, &eax.full, &ebx, &ecx, &edx.full);
-	r->num_closid = edx.split.cos_max + 1;
+	hw_res->num_closid = edx.split.cos_max + 1;
 	r->cache.cbm_len = eax.split.cbm_len + 1;
 	r->default_ctrl = BIT_MASK(eax.split.cbm_len + 1) - 1;
 	r->cache.shareable_bits = ebx & r->default_ctrl;
@@ -333,10 +351,12 @@ static void rdt_get_cache_alloc_cfg(int idx, struct rdt_resource *r)
 
 static void rdt_get_cdp_config(int level, int type)
 {
-	struct rdt_resource *r_l = &rdt_resources_all[level];
-	struct rdt_resource *r = &rdt_resources_all[type];
+	struct rdt_resource *r_l = &rdt_resources_all[level].resctrl;
+	struct rdt_hw_resource *hw_res_l = resctrl_to_arch_res(r_l);
+	struct rdt_resource *r = &rdt_resources_all[type].resctrl;
+	struct rdt_hw_resource *hw_res = resctrl_to_arch_res(r);
 
-	r->num_closid = r_l->num_closid / 2;
+	hw_res->num_closid = hw_res_l->num_closid / 2;
 	r->cache.cbm_len = r_l->cache.cbm_len;
 	r->default_ctrl = r_l->default_ctrl;
 	r->cache.shareable_bits = r_l->cache.shareable_bits;
@@ -365,9 +385,10 @@ static void
 mba_wrmsr_amd(struct rdt_domain *d, struct msr_param *m, struct rdt_resource *r)
 {
 	unsigned int i;
+	struct rdt_hw_resource *hw_res = resctrl_to_arch_res(r);
 
 	for (i = m->low; i < m->high; i++)
-		wrmsrl(r->msr_base + i, d->ctrl_val[i]);
+		wrmsrl(hw_res->msr_base + i, d->ctrl_val[i]);
 }
 
 /*
@@ -389,19 +410,21 @@ mba_wrmsr_intel(struct rdt_domain *d, struct msr_param *m,
 		struct rdt_resource *r)
 {
 	unsigned int i;
+	struct rdt_hw_resource *hw_res = resctrl_to_arch_res(r);
 
 	/*  Write the delay values for mba. */
 	for (i = m->low; i < m->high; i++)
-		wrmsrl(r->msr_base + i, delay_bw_map(d->ctrl_val[i], r));
+		wrmsrl(hw_res->msr_base + i, delay_bw_map(d->ctrl_val[i], r));
 }
 
 static void
 cat_wrmsr(struct rdt_domain *d, struct msr_param *m, struct rdt_resource *r)
 {
 	unsigned int i;
+	struct rdt_hw_resource *hw_res = resctrl_to_arch_res(r);
 
 	for (i = m->low; i < m->high; i++)
-		wrmsrl(r->msr_base + cbm_idx(r, i), d->ctrl_val[i]);
+		wrmsrl(hw_res->msr_base + cbm_idx(r, i), d->ctrl_val[i]);
 }
 
 struct rdt_domain *get_domain_from_cpu(int cpu, struct rdt_resource *r)
@@ -420,13 +443,14 @@ struct rdt_domain *get_domain_from_cpu(int cpu, struct rdt_resource *r)
 void rdt_ctrl_update(void *arg)
 {
 	struct msr_param *m = arg;
+	struct rdt_hw_resource *hw_res = resctrl_to_arch_res(m->res);
 	struct rdt_resource *r = m->res;
 	int cpu = smp_processor_id();
 	struct rdt_domain *d;
 
 	d = get_domain_from_cpu(cpu, r);
 	if (d) {
-		r->msr_update(d, m, r);
+		hw_res->msr_update(d, m, r);
 		return;
 	}
 	pr_warn_once("cpu %d not found in any domain for resource %s\n",
@@ -468,6 +492,7 @@ struct rdt_domain *rdt_find_domain(struct rdt_resource *r, int id,
 
 void setup_default_ctrlval(struct rdt_resource *r, u32 *dc, u32 *dm)
 {
+	struct rdt_hw_resource *hw_res = resctrl_to_arch_res(r);
 	int i;
 
 	/*
@@ -476,7 +501,7 @@ void setup_default_ctrlval(struct rdt_resource *r, u32 *dc, u32 *dm)
 	 * For Memory Allocation: Set b/w requested to 100%
 	 * and the bandwidth in MBps to U32_MAX
 	 */
-	for (i = 0; i < r->num_closid; i++, dc++, dm++) {
+	for (i = 0; i < hw_res->num_closid; i++, dc++, dm++) {
 		*dc = r->default_ctrl;
 		*dm = MBA_MAX_MBPS;
 	}
@@ -484,14 +509,15 @@ void setup_default_ctrlval(struct rdt_resource *r, u32 *dc, u32 *dm)
 
 static int domain_setup_ctrlval(struct rdt_resource *r, struct rdt_domain *d)
 {
+	struct rdt_hw_resource *hw_res = resctrl_to_arch_res(r);
 	struct msr_param m;
 	u32 *dc, *dm;
 
-	dc = kmalloc_array(r->num_closid, sizeof(*d->ctrl_val), GFP_KERNEL);
+	dc = kmalloc_array(hw_res->num_closid, sizeof(*d->ctrl_val), GFP_KERNEL);
 	if (!dc)
 		return -ENOMEM;
 
-	dm = kmalloc_array(r->num_closid, sizeof(*d->mbps_val), GFP_KERNEL);
+	dm = kmalloc_array(hw_res->num_closid, sizeof(*d->mbps_val), GFP_KERNEL);
 	if (!dm) {
 		kfree(dc);
 		return -ENOMEM;
@@ -502,8 +528,8 @@ static int domain_setup_ctrlval(struct rdt_resource *r, struct rdt_domain *d)
 	setup_default_ctrlval(r, dc, dm);
 
 	m.low = 0;
-	m.high = r->num_closid;
-	r->msr_update(d, &m, r);
+	m.high = hw_res->num_closid;
+	hw_res->msr_update(d, &m, r);
 	return 0;
 }
 
@@ -655,7 +681,7 @@ static void domain_remove_cpu(int cpu, struct rdt_resource *r)
 		return;
 	}
 
-	if (r == &rdt_resources_all[RDT_RESOURCE_L3]) {
+	if (r == &rdt_resources_all[RDT_RESOURCE_L3].resctrl) {
 		if (is_mbm_enabled() && cpu == d->mbm_work_cpu) {
 			cancel_delayed_work(&d->mbm_over);
 			mbm_setup_overflow_handler(d, 0);
@@ -831,9 +857,9 @@ static __init bool get_mem_config(void)
 		return false;
 
 	if (boot_cpu_data.x86_vendor == X86_VENDOR_INTEL)
-		return __get_mem_config_intel(&rdt_resources_all[RDT_RESOURCE_MBA]);
+		return __get_mem_config_intel(&rdt_resources_all[RDT_RESOURCE_MBA].resctrl);
 	else if (boot_cpu_data.x86_vendor == X86_VENDOR_AMD)
-		return __rdt_get_mem_config_amd(&rdt_resources_all[RDT_RESOURCE_MBA]);
+		return __rdt_get_mem_config_amd(&rdt_resources_all[RDT_RESOURCE_MBA].resctrl);
 
 	return false;
 }
@@ -849,14 +875,14 @@ static __init bool get_rdt_alloc_resources(void)
 		return false;
 
 	if (rdt_cpu_has(X86_FEATURE_CAT_L3)) {
-		rdt_get_cache_alloc_cfg(1, &rdt_resources_all[RDT_RESOURCE_L3]);
+		rdt_get_cache_alloc_cfg(1, &rdt_resources_all[RDT_RESOURCE_L3].resctrl);
 		if (rdt_cpu_has(X86_FEATURE_CDP_L3))
 			rdt_get_cdp_l3_config();
 		ret = true;
 	}
 	if (rdt_cpu_has(X86_FEATURE_CAT_L2)) {
 		/* CPUID 0x10.2 fields are same format at 0x10.1 */
-		rdt_get_cache_alloc_cfg(2, &rdt_resources_all[RDT_RESOURCE_L2]);
+		rdt_get_cache_alloc_cfg(2, &rdt_resources_all[RDT_RESOURCE_L2].resctrl);
 		if (rdt_cpu_has(X86_FEATURE_CDP_L2))
 			rdt_get_cdp_l2_config();
 		ret = true;
@@ -880,7 +906,7 @@ static __init bool get_rdt_mon_resources(void)
 	if (!rdt_mon_features)
 		return false;
 
-	return !rdt_get_mon_l3_config(&rdt_resources_all[RDT_RESOURCE_L3]);
+	return !rdt_get_mon_l3_config(&rdt_resources_all[RDT_RESOURCE_L3].resctrl);
 }
 
 static __init void __check_quirks_intel(void)
@@ -918,9 +944,12 @@ static __init bool get_rdt_resources(void)
 
 static __init void rdt_init_res_defs_intel(void)
 {
+	struct rdt_hw_resource *hw_res;
 	struct rdt_resource *r;
 
 	for_each_rdt_resource(r) {
+		hw_res = resctrl_to_arch_res(r);
+
 		if (r->rid == RDT_RESOURCE_L3 ||
 		    r->rid == RDT_RESOURCE_L3DATA ||
 		    r->rid == RDT_RESOURCE_L3CODE ||
@@ -931,17 +960,20 @@ static __init void rdt_init_res_defs_intel(void)
 			r->cache.arch_has_empty_bitmaps = false;
 			r->cache.arch_has_per_cpu_cfg = false;
 		} else if (r->rid == RDT_RESOURCE_MBA) {
-			r->msr_base = MSR_IA32_MBA_THRTL_BASE;
-			r->msr_update = mba_wrmsr_intel;
+			hw_res->msr_base = MSR_IA32_MBA_THRTL_BASE;
+			hw_res->msr_update = mba_wrmsr_intel;
 		}
 	}
 }
 
 static __init void rdt_init_res_defs_amd(void)
 {
+	struct rdt_hw_resource *hw_res;
 	struct rdt_resource *r;
 
 	for_each_rdt_resource(r) {
+		hw_res = resctrl_to_arch_res(r);
+
 		if (r->rid == RDT_RESOURCE_L3 ||
 		    r->rid == RDT_RESOURCE_L3DATA ||
 		    r->rid == RDT_RESOURCE_L3CODE ||
@@ -952,8 +984,8 @@ static __init void rdt_init_res_defs_amd(void)
 			r->cache.arch_has_empty_bitmaps = true;
 			r->cache.arch_has_per_cpu_cfg = true;
 		} else if (r->rid == RDT_RESOURCE_MBA) {
-			r->msr_base = MSR_IA32_MBA_BW_BASE;
-			r->msr_update = mba_wrmsr_amd;
+			hw_res->msr_base = MSR_IA32_MBA_BW_BASE;
+			hw_res->msr_update = mba_wrmsr_amd;
 		}
 	}
 }
arch/x86/kernel/cpu/resctrl/ctrlmondata.c +10 -4
index c877642e8a14..ab6e584c9d2d 100644
@@ -284,10 +284,12 @@ int update_domains(struct rdt_resource *r, int closid)
 static int rdtgroup_parse_resource(char *resname, char *tok,
 				   struct rdtgroup *rdtgrp)
 {
+	struct rdt_hw_resource *hw_res;
 	struct rdt_resource *r;
 
 	for_each_alloc_enabled_rdt_resource(r) {
-		if (!strcmp(resname, r->name) && rdtgrp->closid < r->num_closid)
+		hw_res = resctrl_to_arch_res(r);
+		if (!strcmp(resname, r->name) && rdtgrp->closid < hw_res->num_closid)
 			return parse_line(tok, r, rdtgrp);
 	}
 	rdt_last_cmd_printf("Unknown or unsupported resource name '%s'\n", resname);
@@ -394,6 +396,7 @@ static void show_doms(struct seq_file *s, struct rdt_resource *r, int closid)
 int rdtgroup_schemata_show(struct kernfs_open_file *of,
 			   struct seq_file *s, void *v)
 {
+	struct rdt_hw_resource *hw_res;
 	struct rdtgroup *rdtgrp;
 	struct rdt_resource *r;
 	int ret = 0;
@@ -418,7 +421,8 @@ int rdtgroup_schemata_show(struct kernfs_open_file *of,
 		} else {
 			closid = rdtgrp->closid;
 			for_each_alloc_enabled_rdt_resource(r) {
-				if (closid < r->num_closid)
+				hw_res = resctrl_to_arch_res(r);
+				if (closid < hw_res->num_closid)
 					show_doms(s, r, closid);
 			}
 		}
@@ -449,6 +453,7 @@ void mon_event_read(struct rmid_read *rr, struct rdt_resource *r,
 int rdtgroup_mondata_show(struct seq_file *m, void *arg)
 {
 	struct kernfs_open_file *of = m->private;
+	struct rdt_hw_resource *hw_res;
 	u32 resid, evtid, domid;
 	struct rdtgroup *rdtgrp;
 	struct rdt_resource *r;
@@ -468,7 +473,8 @@ int rdtgroup_mondata_show(struct seq_file *m, void *arg)
 	domid = md.u.domid;
 	evtid = md.u.evtid;
 
-	r = &rdt_resources_all[resid];
+	hw_res = &rdt_resources_all[resid];
+	r = &hw_res->resctrl;
 	d = rdt_find_domain(r, domid, NULL);
 	if (IS_ERR_OR_NULL(d)) {
 		ret = -ENOENT;
@@ -482,7 +488,7 @@ int rdtgroup_mondata_show(struct seq_file *m, void *arg)
 	else if (rr.val & RMID_VAL_UNAVAIL)
 		seq_puts(m, "Unavailable\n");
 	else
-		seq_printf(m, "%llu\n", rr.val * r->mon_scale);
+		seq_printf(m, "%llu\n", rr.val * hw_res->mon_scale);
 
 out:
 	rdtgroup_kn_unlock(of->kn);
arch/x86/kernel/cpu/resctrl/internal.h +27 -112

196 lines not on this page · raw message

arch/x86/kernel/cpu/resctrl/monitor.c +18 -14
index 7ac31210e452..cd0abe007ab8 100644
@@ -174,7 +174,7 @@ void __check_limbo(struct rdt_domain *d, bool force_free)
 	struct rdt_resource *r;
 	u32 crmid = 1, nrmid;
 
-	r = &rdt_resources_all[RDT_RESOURCE_L3];
+	r = &rdt_resources_all[RDT_RESOURCE_L3].resctrl;
 
 	/*
 	 * Skip RMID 0 and start from RMID 1 and check all the RMIDs that
@@ -232,7 +232,7 @@ static void add_rmid_to_limbo(struct rmid_entry *entry)
 	int cpu;
 	u64 val;
 
-	r = &rdt_resources_all[RDT_RESOURCE_L3];
+	r = &rdt_resources_all[RDT_RESOURCE_L3].resctrl;
 
 	entry->busy = 0;
 	cpu = get_cpu();
@@ -287,6 +287,7 @@ static u64 mbm_overflow_count(u64 prev_msr, u64 cur_msr, unsigned int width)
 
 static int __mon_event_count(u32 rmid, struct rmid_read *rr)
 {
+	struct rdt_hw_resource *hw_res = resctrl_to_arch_res(rr->r);
 	struct mbm_state *m;
 	u64 chunks, tval;
 
@@ -319,7 +320,7 @@ static int __mon_event_count(u32 rmid, struct rmid_read *rr)
 		return 0;
 	}
 
-	chunks = mbm_overflow_count(m->prev_msr, tval, rr->r->mbm_width);
+	chunks = mbm_overflow_count(m->prev_msr, tval, hw_res->mbm_width);
 	m->chunks += chunks;
 	m->prev_msr = tval;
 
@@ -334,7 +335,7 @@ static int __mon_event_count(u32 rmid, struct rmid_read *rr)
  */
 static void mbm_bw_count(u32 rmid, struct rmid_read *rr)
 {
-	struct rdt_resource *r = &rdt_resources_all[RDT_RESOURCE_L3];
+	struct rdt_hw_resource *hw_res = resctrl_to_arch_res(rr->r);
 	struct mbm_state *m = &rr->d->mbm_local[rmid];
 	u64 tval, cur_bw, chunks;
 
@@ -342,8 +343,8 @@ static void mbm_bw_count(u32 rmid, struct rmid_read *rr)
 	if (tval & (RMID_VAL_ERROR | RMID_VAL_UNAVAIL))
 		return;
 
-	chunks = mbm_overflow_count(m->prev_bw_msr, tval, rr->r->mbm_width);
-	cur_bw = (get_corrected_mbm_count(rmid, chunks) * r->mon_scale) >> 20;
+	chunks = mbm_overflow_count(m->prev_bw_msr, tval, hw_res->mbm_width);
+	cur_bw = (get_corrected_mbm_count(rmid, chunks) * hw_res->mon_scale) >> 20;
 
 	if (m->delta_comp)
 		m->delta_bw = abs(cur_bw - m->prev_bw);
@@ -416,6 +417,7 @@ static void update_mba_bw(struct rdtgroup *rgrp, struct rdt_domain *dom_mbm)
 {
 	u32 closid, rmid, cur_msr, cur_msr_val, new_msr_val;
 	struct mbm_state *pmbm_data, *cmbm_data;
+	struct rdt_hw_resource *hw_r_mba;
 	u32 cur_bw, delta_bw, user_bw;
 	struct rdt_resource *r_mba;
 	struct rdt_domain *dom_mba;
@@ -425,7 +427,8 @@ static void update_mba_bw(struct rdtgroup *rgrp, struct rdt_domain *dom_mbm)
 	if (!is_mbm_local_enabled())
 		return;
 
-	r_mba = &rdt_resources_all[RDT_RESOURCE_MBA];
+	hw_r_mba = &rdt_resources_all[RDT_RESOURCE_MBA];
+	r_mba = &hw_r_mba->resctrl;
 	closid = rgrp->closid;
 	rmid = rgrp->mon.rmid;
 	pmbm_data = &dom_mbm->mbm_local[rmid];
@@ -474,7 +477,7 @@ static void update_mba_bw(struct rdtgroup *rgrp, struct rdt_domain *dom_mbm)
 		return;
 	}
 
-	cur_msr = r_mba->msr_base + closid;
+	cur_msr = hw_r_mba->msr_base + closid;
 	wrmsrl(cur_msr, delay_bw_map(new_msr_val, r_mba));
 	dom_mba->ctrl_val[closid] = new_msr_val;
 
@@ -538,7 +541,7 @@ void cqm_handle_limbo(struct work_struct *work)
 
 	mutex_lock(&rdtgroup_mutex);
 
-	r = &rdt_resources_all[RDT_RESOURCE_L3];
+	r = &rdt_resources_all[RDT_RESOURCE_L3].resctrl;
 	d = container_of(work, struct rdt_domain, cqm_limbo.work);
 
 	__check_limbo(d, false);
@@ -574,7 +577,7 @@ void mbm_handle_overflow(struct work_struct *work)
 	if (!static_branch_likely(&rdt_mon_enable_key))
 		goto out_unlock;
 
-	r = &rdt_resources_all[RDT_RESOURCE_L3];
+	r = &rdt_resources_all[RDT_RESOURCE_L3].resctrl;
 	d = container_of(work, struct rdt_domain, mbm_over.work);
 
 	list_for_each_entry(prgrp, &rdt_all_groups, rdtgroup_list) {
@@ -671,15 +674,16 @@ static void l3_mon_evt_init(struct rdt_resource *r)
 int rdt_get_mon_l3_config(struct rdt_resource *r)
 {
 	unsigned int mbm_offset = boot_cpu_data.x86_cache_mbm_width_offset;
+	struct rdt_hw_resource *hw_res = resctrl_to_arch_res(r);
 	unsigned int cl_size = boot_cpu_data.x86_cache_size;
 	int ret;
 
-	r->mon_scale = boot_cpu_data.x86_cache_occ_scale;
+	hw_res->mon_scale = boot_cpu_data.x86_cache_occ_scale;
 	r->num_rmid = boot_cpu_data.x86_cache_max_rmid + 1;
-	r->mbm_width = MBM_CNTR_WIDTH_BASE;
+	hw_res->mbm_width = MBM_CNTR_WIDTH_BASE;
 
 	if (mbm_offset > 0 && mbm_offset <= MBM_CNTR_WIDTH_OFFSET_MAX)
-		r->mbm_width += mbm_offset;
+		hw_res->mbm_width += mbm_offset;
 	else if (mbm_offset > MBM_CNTR_WIDTH_OFFSET_MAX)
 		pr_warn("Ignoring impossible MBM counter offset\n");
 
@@ -693,7 +697,7 @@ int rdt_get_mon_l3_config(struct rdt_resource *r)
 	resctrl_cqm_threshold = cl_size * 1024 / r->num_rmid;
 
 	/* h/w works in units of "boot_cpu_data.x86_cache_occ_scale" */
-	resctrl_cqm_threshold /= r->mon_scale;
+	resctrl_cqm_threshold /= hw_res->mon_scale;
 
 	ret = dom_data_init(r);
 	if (ret)
arch/x86/kernel/cpu/resctrl/pseudo_lock.c +2 -2
index e916646adc69..f8c5af759c0d 100644
@@ -684,8 +684,8 @@ int rdtgroup_locksetup_enter(struct rdtgroup *rdtgrp)
 	 *   resource, the portion of cache used by it should be made
 	 *   unavailable to all future allocations from both resources.
 	 */
-	if (rdt_resources_all[RDT_RESOURCE_L3DATA].alloc_enabled ||
-	    rdt_resources_all[RDT_RESOURCE_L2DATA].alloc_enabled) {
+	if (rdt_resources_all[RDT_RESOURCE_L3DATA].resctrl.alloc_enabled ||
+	    rdt_resources_all[RDT_RESOURCE_L2DATA].resctrl.alloc_enabled) {
 		rdt_last_cmd_puts("CDP enabled\n");
 		return -EINVAL;
 	}
arch/x86/kernel/cpu/resctrl/rdtgroup.c +38 -30

198 lines not on this page · raw message

include/linux/resctrl.h +111 -0

121 lines not on this page · raw message

Signature · 1 line
2.30.0
maintainer
#2 lore raw

Re: [PATCH v2 01/24] x86/resctrl: Split struct rdt_resource

Hi James,

A significant time has passed since the first version and with that a lot of my context lost.

Quoted 6 lines · James Morse

resctrl is the defacto Linux ABI for SoC resource partitioning features. To support it on another architecture, it needs to be abstracted from the features provided by Intel RDT and AMD PQoS, and moved to /fs/.

Start by splitting struct rdt_resource, (the name is kept to keep the noise down), and add some type-trickery to keep the foreach helpers working.

Could you please replace "add some type-trickery" with a description of the changes(tricks?) referred to? Comments in the code would be helpful also ... helping to avoid frowning at what at first glance seems like an out-of-bounds access.

Quoted 4 lines

Move everything that is particular to resctrl into a new header file, keeping the x86 hardware accessors where they are. resctrl code paths touching a 'hw' struct indicates where an abstraction is needed.

This establishes the significance of this patch. Here the rdt_resource struct is split up and it is this split that guides the subsequent abstraction. Considering this I find that this description does not explain the resulting split sufficiently.

Specifically, after reading the above summary I expect fs information in rdt_resource and hw information in rdt_hw_resource but that does not seem to be the case. For example, num_rmid is a property obtained from hardware but is found in rdt_resource while other hardware properties initialized at the same time are found in rdt_hw_resource. It is interesting to look at when the hardware is discovered (for example, functions like cache_alloc_hsw_probe(), __get_mem_config_intel(), __rdt_get_mem_config_amd(), rdt_get_cache_alloc_cfg()). Note how some of the discovered values end up in rdt_resource and some in rdt_hw_resource. I was expecting these properties discovered from hardware to be in rdt_hw_resource.

It is also not clear to me how these structures are intended to be used for related hardware properties. For example, rdt_resource keeps the properties alloc_capable/alloc_enabled/mon_capable/mon_enabled - but in this series companion properties of cdp_capable/cdp_enabled are introduced and placed in rdt_hw_resource. That seems contradicting to me.

Since this change is so foundational it would be very helpful if the resulting split could be explained in more detail.

Thank you

Reinette

author reviewer
#3 lore raw

Re: [PATCH v2 01/24] x86/resctrl: Split struct rdt_resource

Hi Reinette,

Quoted 11 lines · Reinette Chatre
Quoted 6 lines · James Morse

resctrl is the defacto Linux ABI for SoC resource partitioning features. To support it on another architecture, it needs to be abstracted from the features provided by Intel RDT and AMD PQoS, and moved to /fs/.

Start by splitting struct rdt_resource, (the name is kept to keep the noise down), and add some type-trickery to keep the foreach helpers working.

Could you please replace "add some type-trickery" with a description of the changes(tricks?) referred to? Comments in the code would be helpful also ... helping to avoid frowning at what at first glance seems like an out-of-bounds access.

Sure, this paragraph is rephrased:

| Start by splitting struct rdt_resource, into an arch specific 'hw'
| struct, which contains the common resctrl structure that would be used
| by any architecture.
|
| The foreach helpers are most commonly used by the filesystem code,
| and should return the common resctrl structure. for_each_rdt_resource()
| is changed to walk the common structure in its parent arch specific
| structure.

and a comment above for_each_rdt_resource():

| /*
|  * To return the common struct rdt_resource, which is contained in struct
|  * rdt_hw_resource, walk the resctrl member of struct rdt_hw_resource.
|  * This makes the limit the resctrl member past the end of the array.
|  */
Quoted 17 lines
Quoted 3 lines

Move everything that is particular to resctrl into a new header file, keeping the x86 hardware accessors where they are. resctrl code paths touching a 'hw' struct indicates where an abstraction is needed.

This establishes the significance of this patch. Here the rdt_resource struct is split up and it is this split that guides the subsequent abstraction. Considering this I find that this description does not explain the resulting split sufficiently.

Specifically, after reading the above summary I expect fs information in rdt_resource and hw information in rdt_hw_resource but that does not seem to be the case. For example, num_rmid is a property obtained from hardware but is found in rdt_resource while other hardware properties initialized at the same time are found in rdt_hw_resource. It is interesting to look at when the hardware is discovered (for example, functions like cache_alloc_hsw_probe(), __get_mem_config_intel(), __rdt_get_mem_config_amd(), rdt_get_cache_alloc_cfg()). Note how some of the discovered values end up in rdt_resource and some in rdt_hw_resource.

Quoted 2 lines

I was expecting these properties discovered from hardware to be in rdt_hw_resource.

Not all values discovered from the hardware are private to the architecture. They only need to be private if there is some further abstraction involved.

There is a trade-off here. Everything could be accessed via helpers, but I think that would result in a lot of boiler plate.

On your specific example: the resctrl filesystem code allocates from num_rmid. Its meaning doesn't change. num_closid on the other hand changes depending on whether CDP is in use.

Putting num_closid in resctrl's struct rdt_resource would work, but the value is wrong once CDP is enabled. This would be annoying to debug, hiding the hardware value and providing it via a helper avoids this, as by the end of the series there is only one
consumer: schemata_list_create().

For MPAM, the helper would return arm64's version of rdt_min_closid as there is only one 'num_closid' for the system, regardless of the resource. The driver has to duplicate the logic in closid_init() to find the minimum common value of all the resources, as not all the resources are exposed to resctrl, and an out-of-range closid value triggers an error interrupt.

Quoted 4 lines

It is also not clear to me how these structures are intended to be used for related hardware properties. For example, rdt_resource keeps the properties alloc_capable/alloc_enabled/mon_capable/mon_enabled - but in this series companion properties of cdp_capable/cdp_enabled are introduced and placed in rdt_hw_resource.

There needs to be further abstraction around cdp_enabled. For Arm's MPAM CDP is emulated by providing different closid for data-access and instruction-fetch. This is done in the equivalent to IA32_PQR_ASSOC, so it affects all the resources.

For MPAM all resources would be cdp_capable, so the field doesn't need to exist. cdp_enabled has to be used via a helper, as its a global property for all the tasks that resctrl is in control of, not a per-resource field.

(this is the reason the previous version tried to make the CDP state global, on the assumption it would never appear on both L2 and L3 for x86 systems)

(The next patch after these removes alloc_enabled, as it no longer means anything once the resources are merged. I didn't post it to try and keep the series small)

Quoted 4 lines

That seems contradicting to me.

Since this change is so foundational it would be very helpful if the resulting split could be explained in more detail.

Sure. I'll add a paragraph on where I think extra abstraction is needed for the members of struct rdt_hw_resource. The two not described above are mon_scale and mbm_width.

Currently rephrased as:

| Move as much of the structure as possible into the common structure
| in the core code's header file. The x86 hardware accessors remain
| part of the architecture private code, as do num_closid, mon_scale
| and mbm_width.
| mon_scale and mbm_width are used to detect overflow of the hardware
| counters, and convert them from their native size to bytes. Any
| cross-architecture abstraction should be in terms of bytes, making
| these properties private.
| The hardware's num_closid is kept in the private structure to force
| the filesystem code to use a helper to access it. MPAM would return a
| single value for the system, regardless of the resource. Using the
| helper prevents this field from being confused with the version of
| num_closid that is being exposed to user-space (added in a later patch).

Thanks,

James

maintainer
#4 lore raw

Re: [PATCH v2 01/24] x86/resctrl: Split struct rdt_resource

Hi James,

Quoted 23 lines · James Morse
Quoted 11 lines · Reinette Chatre
Quoted 6 lines · James Morse

resctrl is the defacto Linux ABI for SoC resource partitioning features. To support it on another architecture, it needs to be abstracted from the features provided by Intel RDT and AMD PQoS, and moved to /fs/.

Start by splitting struct rdt_resource, (the name is kept to keep the noise down), and add some type-trickery to keep the foreach helpers working.

Could you please replace "add some type-trickery" with a description of the changes(tricks?) referred to? Comments in the code would be helpful also ... helping to avoid frowning at what at first glance seems like an out-of-bounds access.

Sure, this paragraph is rephrased:

| Start by splitting struct rdt_resource, into an arch specific 'hw'
| struct, which contains the common resctrl structure that would be used
| by any architecture.
|
| The foreach helpers are most commonly used by the filesystem code,
| and should return the common resctrl structure. for_each_rdt_resource()
| is changed to walk the common structure in its parent arch specific
| structure.
Quoted 6 lines

and a comment above for_each_rdt_resource():

| /*
|  * To return the common struct rdt_resource, which is contained in struct
|  * rdt_hw_resource, walk the resctrl member of struct rdt_hw_resource.
|  * This makes the limit the resctrl member past the end of the array.
|  */

Thank you.

Quoted 17 lines
Quoted 16 lines
Quoted 3 lines

Move everything that is particular to resctrl into a new header file, keeping the x86 hardware accessors where they are. resctrl code paths touching a 'hw' struct indicates where an abstraction is needed.

This establishes the significance of this patch. Here the rdt_resource struct is split up and it is this split that guides the subsequent abstraction. Considering this I find that this description does not explain the resulting split sufficiently.

Specifically, after reading the above summary I expect fs information in rdt_resource and hw information in rdt_hw_resource but that does not seem to be the case. For example, num_rmid is a property obtained from hardware but is found in rdt_resource while other hardware properties initialized at the same time are found in rdt_hw_resource. It is interesting to look at when the hardware is discovered (for example, functions like cache_alloc_hsw_probe(), __get_mem_config_intel(), __rdt_get_mem_config_amd(), rdt_get_cache_alloc_cfg()). Note how some of the discovered values end up in rdt_resource and some in rdt_hw_resource.

Quoted 5 lines
Quoted 2 lines

I was expecting these properties discovered from hardware to be in rdt_hw_resource.

Not all values discovered from the hardware are private to the architecture. They only need to be private if there is some further abstraction involved.

ok, but rdt_hw_resource is described as "hw attributes of a resctrl resource" so this can be very confusing if rdt_hw_resource does _not_ actually contain (all of) the hw attributes of a resctrl resource.

Could you please expand the kernel doc for rdt_hw_resource to explain that, apart from @resctrl (that I just noticed is missing a description), it contains attributes needing abstraction for different architectures as opposed to the actual hardware attributes?

Quoted 3 lines

There is a trade-off here. Everything could be accessed via helpers, but I think that would result in a lot of boiler plate.

I see.

Quoted 20 lines

On your specific example: the resctrl filesystem code allocates from num_rmid. Its meaning doesn't change. num_closid on the other hand changes depending on whether CDP is in use.

Putting num_closid in resctrl's struct rdt_resource would work, but the value is wrong once CDP is enabled. This would be annoying to debug, hiding the hardware value and providing it via a helper avoids this, as by the end of the series there is only one
consumer: schemata_list_create().

For MPAM, the helper would return arm64's version of rdt_min_closid as there is only one 'num_closid' for the system, regardless of the resource. The driver has to duplicate the logic in closid_init() to find the minimum common value of all the resources, as not all the resources are exposed to resctrl, and an out-of-range closid value triggers an error interrupt.

Quoted 4 lines

It is also not clear to me how these structures are intended to be used for related hardware properties. For example, rdt_resource keeps the properties alloc_capable/alloc_enabled/mon_capable/mon_enabled - but in this series companion properties of cdp_capable/cdp_enabled are introduced and placed in rdt_hw_resource.

Quoted 5 lines

There needs to be further abstraction around cdp_enabled. For Arm's MPAM CDP is emulated by providing different closid for data-access and instruction-fetch. This is done in the equivalent to IA32_PQR_ASSOC, so it affects all the resources.

For MPAM all resources would be cdp_capable, so the field doesn't need to exist.

Will it be removed?

Quoted 18 lines

cdp_enabled has to be used via a helper, as its a global property for all the tasks that resctrl is in control of, not a per-resource field.

(this is the reason the previous version tried to make the CDP state global, on the assumption it would never appear on both L2 and L3 for x86 systems)

(The next patch after these removes alloc_enabled, as it no longer means anything once the resources are merged. I didn't post it to try and keep the series small)

Quoted 4 lines

That seems contradicting to me.

Since this change is so foundational it would be very helpful if the resulting split could be explained in more detail.

Sure. I'll add a paragraph on where I think extra abstraction is needed for the members of struct rdt_hw_resource. The two not described above are mon_scale and mbm_width.

Currently rephrased as:

Quoted 14 lines
| Move as much of the structure as possible into the common structure
| in the core code's header file. The x86 hardware accessors remain
| part of the architecture private code, as do num_closid, mon_scale
| and mbm_width.
| mon_scale and mbm_width are used to detect overflow of the hardware
| counters, and convert them from their native size to bytes. Any
| cross-architecture abstraction should be in terms of bytes, making
| these properties private.
| The hardware's num_closid is kept in the private structure to force
| the filesystem code to use a helper to access it. MPAM would return a
| single value for the system, regardless of the resource. Using the
| helper prevents this field from being confused with the version of
| num_closid that is being exposed to user-space (added in a later patch).

This is very helpful. Thank you. I also think that adding a similar per-property summary to the kernel-doc of rt_hw_resource would be very helpful.

Reinette

author reviewer
#5 lore raw

Re: [PATCH v2 01/24] x86/resctrl: Split struct rdt_resource

Hi Reinette,

Quoted 30 lines · Reinette Chatre
Quoted 29 lines · James Morse
Quoted 27 lines · Reinette Chatre
Quoted 10 lines · James Morse

resctrl is the defacto Linux ABI for SoC resource partitioning features. To support it on another architecture, it needs to be abstracted from the features provided by Intel RDT and AMD PQoS, and moved to /fs/.

Start by splitting struct rdt_resource, (the name is kept to keep the noise down), and add some type-trickery to keep the foreach helpers working.

Move everything that is particular to resctrl into a new header file, keeping the x86 hardware accessors where they are. resctrl code paths touching a 'hw' struct indicates where an abstraction is needed.

This establishes the significance of this patch. Here the rdt_resource struct is split up and it is this split that guides the subsequent abstraction. Considering this I find that this description does not explain the resulting split sufficiently.

Specifically, after reading the above summary I expect fs information in rdt_resource and hw information in rdt_hw_resource but that does not seem to be the case. For example, num_rmid is a property obtained from hardware but is found in rdt_resource while other hardware properties initialized at the same time are found in rdt_hw_resource. It is interesting to look at when the hardware is discovered (for example, functions like cache_alloc_hsw_probe(), __get_mem_config_intel(), __rdt_get_mem_config_amd(), rdt_get_cache_alloc_cfg()). Note how some of the discovered values end up in rdt_resource and some in rdt_hw_resource.

I was expecting these properties discovered from hardware to be in rdt_hw_resource.

Quoted 6 lines
Quoted 2 lines

Not all values discovered from the hardware are private to the architecture. They only need to be private if there is some further abstraction involved.

ok, but rdt_hw_resource is described as "hw attributes of a resctrl resource" so this can be very confusing if rdt_hw_resource does _not_ actually contain (all of) the hw attributes of a resctrl resource.

Aha, right. I'm bad at naming things. This started as untangling the hardware (cough:
arch) specific bits, but some things have migrated back the other way.

Do you think either of arch_rdt_resource or rdt_priv_resource are clearer?

Quoted 2 lines

Could you please expand the kernel doc for rdt_hw_resource to explain that, apart from @resctrl (that I just noticed is missing a description),

I'll add one for mbm_width too,

Quoted 2 lines

it contains attributes needing
abstraction for different architectures as opposed to the actual hardware attributes?

|/**
| * struct rdt_hw_resource - arch private attributes of a resctrl resource
| * @resctrl:		Attributes of the resource used directly by resctrl.
| * @num_closid:	Number of CLOSIDs available.
| * @msr_base:		Base MSR address for CBMs
| * @msr_update:	Function pointer to update QOS MSRs
| * @mon_scale:		cqm counter * mon_scale = occupancy in bytes
| * @mbm_width:		Monitor width, to detect and correct for overflow.
| *
| * Members of this structure are either private to the architecture
| * e.g. mbm_width, or accessed via helpers that provide abstraction. e.g.
| * msr_update and msr_base.
| */
Quoted 20 lines
Quoted 20 lines

On your specific example: the resctrl filesystem code allocates from num_rmid. Its meaning doesn't change. num_closid on the other hand changes depending on whether CDP is in use.

Putting num_closid in resctrl's struct rdt_resource would work, but the value is wrong once CDP is enabled. This would be annoying to debug, hiding the hardware value and providing it via a helper avoids this, as by the end of the series there is only one
consumer: schemata_list_create().

For MPAM, the helper would return arm64's version of rdt_min_closid as there is only one 'num_closid' for the system, regardless of the resource. The driver has to duplicate the logic in closid_init() to find the minimum common value of all the resources, as not all the resources are exposed to resctrl, and an out-of-range closid value triggers an error interrupt.

Quoted 4 lines

It is also not clear to me how these structures are intended to be used for related hardware properties. For example, rdt_resource keeps the properties alloc_capable/alloc_enabled/mon_capable/mon_enabled - but in this series companion properties of cdp_capable/cdp_enabled are introduced and placed in rdt_hw_resource.

Quoted 7 lines
Quoted 5 lines

There needs to be further abstraction around cdp_enabled. For Arm's MPAM CDP is emulated by providing different closid for data-access and instruction-fetch. This is done in the equivalent to IA32_PQR_ASSOC, so it affects all the resources.

For MPAM all resources would be cdp_capable, so the field doesn't need to exist.

Will it be removed?

It wouldn't exist in the MPAM version of rdt_hw_resource.

It is needed for Intel's RDT to ensure CDP can be supported and enabled per-resource, which is how I read your no 'force enabling of CDP on all cache levels' comment from:
https://lore.kernel.org/lkml/[email protected]/

If you don't think per-resources tracking is needed, did I read that wrong? (it only 'forced' CDP on for the L2 if it had been enabled for L3. My understanding is no SoC today has both)

Quoted 32 lines
Quoted 18 lines

cdp_enabled has to be used via a helper, as its a global property for all the tasks that resctrl is in control of, not a per-resource field.

(this is the reason the previous version tried to make the CDP state global, on the assumption it would never appear on both L2 and L3 for x86 systems)

(The next patch after these removes alloc_enabled, as it no longer means anything once the resources are merged. I didn't post it to try and keep the series small)

Quoted 4 lines

That seems contradicting to me.

Since this change is so foundational it would be very helpful if the resulting split could be explained in more detail.

Sure. I'll add a paragraph on where I think extra abstraction is needed for the members of struct rdt_hw_resource. The two not described above are mon_scale and mbm_width.

Currently rephrased as:

Quoted 13 lines
| Move as much of the structure as possible into the common structure
| in the core code's header file. The x86 hardware accessors remain
| part of the architecture private code, as do num_closid, mon_scale
| and mbm_width.
| mon_scale and mbm_width are used to detect overflow of the hardware
| counters, and convert them from their native size to bytes. Any
| cross-architecture abstraction should be in terms of bytes, making
| these properties private.
| The hardware's num_closid is kept in the private structure to force
| the filesystem code to use a helper to access it. MPAM would return a
| single value for the system, regardless of the resource. Using the
| helper prevents this field from being confused with the version of
| num_closid that is being exposed to user-space (added in a later patch).
Quoted 2 lines

This is very helpful. Thank you. I also think that adding a similar per-property summary to the kernel-doc of rt_hw_resource would be very helpful.

The problem is 'any cross-architecture abstraction' refers to patches that will show up quite a bit later. I think this is fine for the motivation in the commit message as the information is only relevant at the point of the change, but its decidedly weird to refer to MPAM in the x86's header files.

I'll add the reason behind num_closid being odd to the patch that adds the helper.

Thanks,

James

maintainer
#6 lore raw

Re: [PATCH v2 01/24] x86/resctrl: Split struct rdt_resource

Hi James,

Quoted 34 lines · James Morse
Quoted 32 lines · Reinette Chatre
Quoted 31 lines · James Morse
Quoted 25 lines · Reinette Chatre
Quoted 10 lines · James Morse

resctrl is the defacto Linux ABI for SoC resource partitioning features. To support it on another architecture, it needs to be abstracted from the features provided by Intel RDT and AMD PQoS, and moved to /fs/.

Start by splitting struct rdt_resource, (the name is kept to keep the noise down), and add some type-trickery to keep the foreach helpers working.

Move everything that is particular to resctrl into a new header file, keeping the x86 hardware accessors where they are. resctrl code paths touching a 'hw' struct indicates where an abstraction is needed.

This establishes the significance of this patch. Here the rdt_resource struct is split up and it is this split that guides the subsequent abstraction. Considering this I find that this description does not explain the resulting split sufficiently.

Specifically, after reading the above summary I expect fs information in rdt_resource and hw information in rdt_hw_resource but that does not seem to be the case. For example, num_rmid is a property obtained from hardware but is found in rdt_resource while other hardware properties initialized at the same time are found in rdt_hw_resource. It is interesting to look at when the hardware is discovered (for example, functions like cache_alloc_hsw_probe(), __get_mem_config_intel(), __rdt_get_mem_config_amd(), rdt_get_cache_alloc_cfg()). Note how some of the discovered values end up in rdt_resource and some in rdt_hw_resource.

Quoted 2 lines

I was expecting these properties discovered from hardware to be in rdt_hw_resource.

Not all values discovered from the hardware are private to the architecture. They only need to be private if there is some further abstraction involved.

Quoted 6 lines
Quoted 3 lines

ok, but rdt_hw_resource is described as "hw attributes of a resctrl resource" so this can be very confusing if rdt_hw_resource does _not_ actually contain (all of) the hw attributes of a resctrl resource.

Aha, right. I'm bad at naming things. This started as untangling the hardware (cough:
arch) specific bits, but some things have migrated back the other way.

It was the description that really tripped me. I'm ok with the current naming if the description is clear and usage consistent.

Quoted 12 lines

Do you think either of arch_rdt_resource or rdt_priv_resource are clearer?

Quoted 2 lines

Could you please expand the kernel doc for rdt_hw_resource to explain that, apart from @resctrl (that I just noticed is missing a description),

I'll add one for mbm_width too,

Quoted 2 lines

it contains attributes needing
abstraction for different architectures as opposed to the actual hardware attributes?

Quoted 14 lines
|/**
| * struct rdt_hw_resource - arch private attributes of a resctrl resource
| * @resctrl:		Attributes of the resource used directly by resctrl.
| * @num_closid:	Number of CLOSIDs available.
| * @msr_base:		Base MSR address for CBMs
| * @msr_update:	Function pointer to update QOS MSRs
| * @mon_scale:		cqm counter * mon_scale = occupancy in bytes
| * @mbm_width:		Monitor width, to detect and correct for overflow.
| *
| * Members of this structure are either private to the architecture
| * e.g. mbm_width, or accessed via helpers that provide abstraction. e.g.
| * msr_update and msr_base.
| */

As I commented in patch 7, where num_closid is stored in the schema, I think the descriptions can be improved to help understand the differences between the two num_closid instances. The two num_closid descriptions (found in struct resctrl_schema and struct rdt_hw_resource) should be complimentary to help somebody have clear understanding of their difference. Currently they are mostly a copy of the same description not helping to understand what the difference is. Perhaps something here like "Number of CLOSIDs supported by hardware/architecture"? Please feel free to improve.

The rest looks good, thank you.

Quoted 30 lines
Quoted 20 lines
Quoted 20 lines

On your specific example: the resctrl filesystem code allocates from num_rmid. Its meaning doesn't change. num_closid on the other hand changes depending on whether CDP is in use.

Putting num_closid in resctrl's struct rdt_resource would work, but the value is wrong once CDP is enabled. This would be annoying to debug, hiding the hardware value and providing it via a helper avoids this, as by the end of the series there is only one
consumer: schemata_list_create().

For MPAM, the helper would return arm64's version of rdt_min_closid as there is only one 'num_closid' for the system, regardless of the resource. The driver has to duplicate the logic in closid_init() to find the minimum common value of all the resources, as not all the resources are exposed to resctrl, and an out-of-range closid value triggers an error interrupt.

Quoted 4 lines

It is also not clear to me how these structures are intended to be used for related hardware properties. For example, rdt_resource keeps the properties alloc_capable/alloc_enabled/mon_capable/mon_enabled - but in this series companion properties of cdp_capable/cdp_enabled are introduced and placed in rdt_hw_resource.

Quoted 7 lines
Quoted 5 lines

There needs to be further abstraction around cdp_enabled. For Arm's MPAM CDP is emulated by providing different closid for data-access and instruction-fetch. This is done in the equivalent to IA32_PQR_ASSOC, so it affects all the resources.

For MPAM all resources would be cdp_capable, so the field doesn't need to exist.

Will it be removed?

It wouldn't exist in the MPAM version of rdt_hw_resource.

Quoted 7 lines

It is needed for Intel's RDT to ensure CDP can be supported and enabled per-resource, which is how I read your no 'force enabling of CDP on all cache levels' comment from:
https://lore.kernel.org/lkml/[email protected]/

If you don't think per-resources tracking is needed, did I read that wrong? (it only 'forced' CDP on for the L2 if it had been enabled for L3. My understanding is no SoC today has both)

Thank you for clarifying. No, you did not read my earlier comments wrong. Intel RDT does support and enable CDP per-resource.

Quoted 32 lines
Quoted 31 lines
Quoted 31 lines

cdp_enabled has to be used via a helper, as its a global property for all the tasks that resctrl is in control of, not a per-resource field.

(this is the reason the previous version tried to make the CDP state global, on the assumption it would never appear on both L2 and L3 for x86 systems)

(The next patch after these removes alloc_enabled, as it no longer means anything once the resources are merged. I didn't post it to try and keep the series small)

Quoted 4 lines

That seems contradicting to me.

Since this change is so foundational it would be very helpful if the resulting split could be explained in more detail.

Sure. I'll add a paragraph on where I think extra abstraction is needed for the members of struct rdt_hw_resource. The two not described above are mon_scale and mbm_width.

Currently rephrased as:

| Move as much of the structure as possible into the common structure
| in the core code's header file. The x86 hardware accessors remain
| part of the architecture private code, as do num_closid, mon_scale
| and mbm_width.
| mon_scale and mbm_width are used to detect overflow of the hardware
| counters, and convert them from their native size to bytes. Any
| cross-architecture abstraction should be in terms of bytes, making
| these properties private.
| The hardware's num_closid is kept in the private structure to force
| the filesystem code to use a helper to access it. MPAM would return a
| single value for the system, regardless of the resource. Using the
| helper prevents this field from being confused with the version of
| num_closid that is being exposed to user-space (added in a later patch).
Quoted 7 lines
Quoted 2 lines

This is very helpful. Thank you. I also think that adding a similar per-property summary to the kernel-doc of rt_hw_resource would be very helpful.

The problem is 'any cross-architecture abstraction' refers to patches that will show up quite a bit later. I think this is fine for the motivation in the commit message as the information is only relevant at the point of the change, but its decidedly weird to refer to MPAM in the x86's header files.

ok, I think I see where you are going ... MPAM is not expected to have mon_scale and mbm_width?

Quoted 2 lines

I'll add the reason behind num_closid being odd to the patch that adds the helper.

Thank you

Reinette

author reviewer
#7 lore raw

PATCH v2 2/24 x86/resctrl: Split struct rdt_domain

In reply to James Morse

resctrl is the defacto Linux ABI for SoC resource partitioning features. To support it on another architecture, it needs to be abstracted from the features provided by Intel RDT and AMD PQoS, and moved to /fs/.

Split struct rdt_domain up too. Move everything that that is particular to resctrl into a new header file. resctrl code paths touching a 'hw' struct indicates where an abstraction is needed.

Splitting this structure only moves types around, and should not lead to any change in behaviour.

---
Changes since v1:
* Tabs space and other whitespace
* cpu becomes CPU in all comments touched

---
 arch/x86/kernel/cpu/resctrl/core.c        | 32 +++++++++++-------
 arch/x86/kernel/cpu/resctrl/ctrlmondata.c | 10 ++++--
 arch/x86/kernel/cpu/resctrl/internal.h    | 40 +++++------------------
 arch/x86/kernel/cpu/resctrl/monitor.c     |  8 +++--
 arch/x86/kernel/cpu/resctrl/rdtgroup.c    | 29 ++++++++++------
 include/linux/resctrl.h                   | 32 +++++++++++++++++-
 6 files changed, 91 insertions(+), 60 deletions(-)
arch/x86/kernel/cpu/resctrl/core.c +20 -12
index 6cd84d54086f..ca43a7491fda 100644
@@ -385,10 +385,11 @@ static void
 mba_wrmsr_amd(struct rdt_domain *d, struct msr_param *m, struct rdt_resource *r)
 {
 	unsigned int i;
+	struct rdt_hw_domain *hw_dom = resctrl_to_arch_dom(d);
 	struct rdt_hw_resource *hw_res = resctrl_to_arch_res(r);
 
 	for (i = m->low; i < m->high; i++)
-		wrmsrl(hw_res->msr_base + i, d->ctrl_val[i]);
+		wrmsrl(hw_res->msr_base + i, hw_dom->ctrl_val[i]);
 }
 
 /*
@@ -410,21 +411,23 @@ mba_wrmsr_intel(struct rdt_domain *d, struct msr_param *m,
 		struct rdt_resource *r)
 {
 	unsigned int i;
+	struct rdt_hw_domain *hw_dom = resctrl_to_arch_dom(d);
 	struct rdt_hw_resource *hw_res = resctrl_to_arch_res(r);
 
 	/*  Write the delay values for mba. */
 	for (i = m->low; i < m->high; i++)
-		wrmsrl(hw_res->msr_base + i, delay_bw_map(d->ctrl_val[i], r));
+		wrmsrl(hw_res->msr_base + i, delay_bw_map(hw_dom->ctrl_val[i], r));
 }
 
 static void
 cat_wrmsr(struct rdt_domain *d, struct msr_param *m, struct rdt_resource *r)
 {
 	unsigned int i;
+	struct rdt_hw_domain *hw_dom = resctrl_to_arch_dom(d);
 	struct rdt_hw_resource *hw_res = resctrl_to_arch_res(r);
 
 	for (i = m->low; i < m->high; i++)
-		wrmsrl(hw_res->msr_base + cbm_idx(r, i), d->ctrl_val[i]);
+		wrmsrl(hw_res->msr_base + cbm_idx(r, i), hw_dom->ctrl_val[i]);
 }
 
 struct rdt_domain *get_domain_from_cpu(int cpu, struct rdt_resource *r)
@@ -510,21 +513,22 @@ void setup_default_ctrlval(struct rdt_resource *r, u32 *dc, u32 *dm)
 static int domain_setup_ctrlval(struct rdt_resource *r, struct rdt_domain *d)
 {
 	struct rdt_hw_resource *hw_res = resctrl_to_arch_res(r);
+	struct rdt_hw_domain *hw_dom = resctrl_to_arch_dom(d);
 	struct msr_param m;
 	u32 *dc, *dm;
 
-	dc = kmalloc_array(hw_res->num_closid, sizeof(*d->ctrl_val), GFP_KERNEL);
+	dc = kmalloc_array(hw_res->num_closid, sizeof(*hw_dom->ctrl_val), GFP_KERNEL);
 	if (!dc)
 		return -ENOMEM;
 
-	dm = kmalloc_array(hw_res->num_closid, sizeof(*d->mbps_val), GFP_KERNEL);
+	dm = kmalloc_array(hw_res->num_closid, sizeof(*hw_dom->mbps_val), GFP_KERNEL);
 	if (!dm) {
 		kfree(dc);
 		return -ENOMEM;
 	}
 
-	d->ctrl_val = dc;
-	d->mbps_val = dm;
+	hw_dom->ctrl_val = dc;
+	hw_dom->mbps_val = dm;
 	setup_default_ctrlval(r, dc, dm);
 
 	m.low = 0;
@@ -586,6 +590,7 @@ static void domain_add_cpu(int cpu, struct rdt_resource *r)
 {
 	int id = get_cpu_cacheinfo_id(cpu, r->cache_level);
 	struct list_head *add_pos = NULL;
+	struct rdt_hw_domain *hw_dom;
 	struct rdt_domain *d;
 
 	d = rdt_find_domain(r, id, &add_pos);
@@ -601,10 +606,11 @@ static void domain_add_cpu(int cpu, struct rdt_resource *r)
 		return;
 	}
 
-	d = kzalloc_node(sizeof(*d), GFP_KERNEL, cpu_to_node(cpu));
-	if (!d)
+	hw_dom = kzalloc_node(sizeof(*hw_dom), GFP_KERNEL, cpu_to_node(cpu));
+	if (!hw_dom)
 		return;
 
+	d = &hw_dom->resctrl;
 	d->id = id;
 	cpumask_set_cpu(cpu, &d->cpu_mask);
 
@@ -633,6 +639,7 @@ static void domain_add_cpu(int cpu, struct rdt_resource *r)
 static void domain_remove_cpu(int cpu, struct rdt_resource *r)
 {
 	int id = get_cpu_cacheinfo_id(cpu, r->cache_level);
+	struct rdt_hw_domain *hw_dom;
 	struct rdt_domain *d;
 
 	d = rdt_find_domain(r, id, NULL);
@@ -640,6 +647,7 @@ static void domain_remove_cpu(int cpu, struct rdt_resource *r)
 		pr_warn("Couldn't find cache id for CPU %d\n", cpu);
 		return;
 	}
+	hw_dom = resctrl_to_arch_dom(d);
 
 	cpumask_clear_cpu(cpu, &d->cpu_mask);
 	if (cpumask_empty(&d->cpu_mask)) {
@@ -672,12 +680,12 @@ static void domain_remove_cpu(int cpu, struct rdt_resource *r)
 		if (d->plr)
 			d->plr->d = NULL;
 
-		kfree(d->ctrl_val);
-		kfree(d->mbps_val);
+		kfree(hw_dom->ctrl_val);
+		kfree(hw_dom->mbps_val);
 		bitmap_free(d->rmid_busy_llc);
 		kfree(d->mbm_total);
 		kfree(d->mbm_local);
-		kfree(d);
+		kfree(hw_dom);
 		return;
 	}
 
arch/x86/kernel/cpu/resctrl/ctrlmondata.c +7 -3
index ab6e584c9d2d..2e7466659af3 100644
@@ -238,6 +238,7 @@ static int parse_line(char *line, struct rdt_resource *r,
 
 int update_domains(struct rdt_resource *r, int closid)
 {
+	struct rdt_hw_domain *hw_dom;
 	struct msr_param msr_param;
 	cpumask_var_t cpu_mask;
 	struct rdt_domain *d;
@@ -254,7 +255,8 @@ int update_domains(struct rdt_resource *r, int closid)
 
 	mba_sc = is_mba_sc(r);
 	list_for_each_entry(d, &r->domains, list) {
-		dc = !mba_sc ? d->ctrl_val : d->mbps_val;
+		hw_dom = resctrl_to_arch_dom(d);
+		dc = !mba_sc ? hw_dom->ctrl_val : hw_dom->mbps_val;
 		if (d->have_new_ctrl && d->new_ctrl != dc[closid]) {
 			cpumask_set_cpu(cpumask_any(&d->cpu_mask), cpu_mask);
 			dc[closid] = d->new_ctrl;
@@ -375,17 +377,19 @@ ssize_t rdtgroup_schemata_write(struct kernfs_open_file *of,
 
 static void show_doms(struct seq_file *s, struct rdt_resource *r, int closid)
 {
+	struct rdt_hw_domain *hw_dom;
 	struct rdt_domain *dom;
 	bool sep = false;
 	u32 ctrl_val;
 
 	seq_printf(s, "%*s:", max_name_width, r->name);
 	list_for_each_entry(dom, &r->domains, list) {
+		hw_dom = resctrl_to_arch_dom(dom);
 		if (sep)
 			seq_puts(s, ";");
 
-		ctrl_val = (!is_mba_sc(r) ? dom->ctrl_val[closid] :
-			    dom->mbps_val[closid]);
+		ctrl_val = (!is_mba_sc(r) ? hw_dom->ctrl_val[closid] :
+			    hw_dom->mbps_val[closid]);
 		seq_printf(s, r->format_str, dom->id, max_data_width,
 			   ctrl_val);
 		sep = true;
arch/x86/kernel/cpu/resctrl/internal.h +9 -31
index 4562a5e3cd35..d3e47fd51e3a 100644
@@ -299,44 +299,22 @@ struct mbm_state {
 };
 
 /**
- * struct rdt_domain - group of cpus sharing an RDT resource
- * @list:	all instances of this resource
- * @id:		unique id for this instance
- * @cpu_mask:	which cpus share this resource
- * @rmid_busy_llc:
- *		bitmap of which limbo RMIDs are above threshold
- * @mbm_total:	saved state for MBM total bandwidth
- * @mbm_local:	saved state for MBM local bandwidth
- * @mbm_over:	worker to periodically read MBM h/w counters
- * @cqm_limbo:	worker to periodically read CQM h/w counters
- * @mbm_work_cpu:
- *		worker cpu for MBM h/w counters
- * @cqm_work_cpu:
- *		worker cpu for CQM h/w counters
+ * struct rdt_hw_domain - hw attributes of a group of CPUs that share a resource
+ * @resctrl:	Properties exposed to the resctrl file system
  * @ctrl_val:	array of cache or mem ctrl values (indexed by CLOSID)
  * @mbps_val:	When mba_sc is enabled, this holds the bandwidth in MBps
- * @new_ctrl:	new ctrl value to be loaded
- * @have_new_ctrl: did user provide new_ctrl for this domain
- * @plr:	pseudo-locked region (if any) associated with domain
  */
-struct rdt_domain {
-	struct list_head		list;
-	int				id;
-	struct cpumask			cpu_mask;
-	unsigned long			*rmid_busy_llc;
-	struct mbm_state		*mbm_total;
-	struct mbm_state		*mbm_local;
-	struct delayed_work		mbm_over;
-	struct delayed_work		cqm_limbo;
-	int				mbm_work_cpu;
-	int				cqm_work_cpu;
+struct rdt_hw_domain {
+	struct rdt_domain		resctrl;
 	u32				*ctrl_val;
 	u32				*mbps_val;
-	u32				new_ctrl;
-	bool				have_new_ctrl;
-	struct pseudo_lock_region	*plr;
 };
 
+static inline struct rdt_hw_domain *resctrl_to_arch_dom(struct rdt_domain *r)
+{
+	return container_of(r, struct rdt_hw_domain, resctrl);
+}
+
 /**
  * struct msr_param - set a range of MSRs from a domain
  * @res:       The resource to use
arch/x86/kernel/cpu/resctrl/monitor.c +5 -3
index cd0abe007ab8..6f700ff1b64c 100644
@@ -418,6 +418,7 @@ static void update_mba_bw(struct rdtgroup *rgrp, struct rdt_domain *dom_mbm)
 	u32 closid, rmid, cur_msr, cur_msr_val, new_msr_val;
 	struct mbm_state *pmbm_data, *cmbm_data;
 	struct rdt_hw_resource *hw_r_mba;
+	struct rdt_hw_domain *hw_dom_mba;
 	u32 cur_bw, delta_bw, user_bw;
 	struct rdt_resource *r_mba;
 	struct rdt_domain *dom_mba;
@@ -438,11 +439,12 @@ static void update_mba_bw(struct rdtgroup *rgrp, struct rdt_domain *dom_mbm)
 		pr_warn_once("Failure to get domain for MBA update\n");
 		return;
 	}
+	hw_dom_mba = resctrl_to_arch_dom(dom_mba);
 
 	cur_bw = pmbm_data->prev_bw;
-	user_bw = dom_mba->mbps_val[closid];
+	user_bw = hw_dom_mba->mbps_val[closid];
 	delta_bw = pmbm_data->delta_bw;
-	cur_msr_val = dom_mba->ctrl_val[closid];
+	cur_msr_val = hw_dom_mba->ctrl_val[closid];
 
 	/*
 	 * For Ctrl groups read data from child monitor groups.
@@ -479,7 +481,7 @@ static void update_mba_bw(struct rdtgroup *rgrp, struct rdt_domain *dom_mbm)
 
 	cur_msr = hw_r_mba->msr_base + closid;
 	wrmsrl(cur_msr, delay_bw_map(new_msr_val, r_mba));
-	dom_mba->ctrl_val[closid] = new_msr_val;
+	hw_dom_mba->ctrl_val[closid] = new_msr_val;
 
 	/*
 	 * Delta values are updated dynamically package wise for each
arch/x86/kernel/cpu/resctrl/rdtgroup.c +19 -10
index d8db58d5f9ce..8c29304d3e01 100644
@@ -915,7 +915,7 @@ static int rdt_bit_usage_show(struct kernfs_open_file *of,
 	list_for_each_entry(dom, &r->domains, list) {
 		if (sep)
 			seq_putc(seq, ';');
-		ctrl = dom->ctrl_val;
+		ctrl = resctrl_to_arch_dom(dom)->ctrl_val;
 		sw_shareable = 0;
 		exclusive = 0;
 		seq_printf(seq, "%d=", dom->id);
@@ -1193,7 +1193,7 @@ static bool __rdtgroup_cbm_overlaps(struct rdt_resource *r, struct rdt_domain *d
 	}
 
 	/* Check for overlap with other resource groups */
-	ctrl = d->ctrl_val;
+	ctrl = resctrl_to_arch_dom(d)->ctrl_val;
 	for (i = 0; i < closids_supported(); i++, ctrl++) {
 		ctrl_b = *ctrl;
 		mode = rdtgroup_mode_by_closid(i);
@@ -1262,6 +1262,7 @@ bool rdtgroup_cbm_overlaps(struct rdt_resource *r, struct rdt_domain *d,
  */
 static bool rdtgroup_mode_test_exclusive(struct rdtgroup *rdtgrp)
 {
+	struct rdt_hw_domain *hw_dom;
 	int closid = rdtgrp->closid;
 	struct rdt_resource *r;
 	bool has_cache = false;
@@ -1272,7 +1273,8 @@ static bool rdtgroup_mode_test_exclusive(struct rdtgroup *rdtgrp)
 			continue;
 		has_cache = true;
 		list_for_each_entry(d, &r->domains, list) {
-			if (rdtgroup_cbm_overlaps(r, d, d->ctrl_val[closid],
+			hw_dom = resctrl_to_arch_dom(d);
+			if (rdtgroup_cbm_overlaps(r, d, hw_dom->ctrl_val[closid],
 						  rdtgrp->closid, false)) {
 				rdt_last_cmd_puts("Schemata overlaps\n");
 				return false;
@@ -1404,6 +1406,7 @@ unsigned int rdtgroup_cbm_to_size(struct rdt_resource *r,
 static int rdtgroup_size_show(struct kernfs_open_file *of,
 			      struct seq_file *s, void *v)
 {
+	struct rdt_hw_domain *hw_dom;
 	struct rdtgroup *rdtgrp;
 	struct rdt_resource *r;
 	struct rdt_domain *d;
@@ -1438,14 +1441,15 @@ static int rdtgroup_size_show(struct kernfs_open_file *of,
 		sep = false;
 		seq_printf(s, "%*s:", max_name_width, r->name);
 		list_for_each_entry(d, &r->domains, list) {
+			hw_dom = resctrl_to_arch_dom(d);
 			if (sep)
 				seq_putc(s, ';');
 			if (rdtgrp->mode == RDT_MODE_PSEUDO_LOCKSETUP) {
 				size = 0;
 			} else {
 				ctrl = (!is_mba_sc(r) ?
-						d->ctrl_val[rdtgrp->closid] :
-						d->mbps_val[rdtgrp->closid]);
+						hw_dom->ctrl_val[rdtgrp->closid] :
+						hw_dom->mbps_val[rdtgrp->closid]);
 				if (r->rid == RDT_RESOURCE_MBA)
 					size = ctrl;
 				else
@@ -1940,6 +1944,7 @@ void rdt_domain_reconfigure_cdp(struct rdt_resource *r)
 static int set_mba_sc(bool mba_sc)
 {
 	struct rdt_resource *r = &rdt_resources_all[RDT_RESOURCE_MBA].resctrl;
+	struct rdt_hw_domain *hw_dom;
 	struct rdt_domain *d;
 
 	if (!is_mbm_enabled() || !is_mba_linear() ||
@@ -1947,8 +1952,10 @@ static int set_mba_sc(bool mba_sc)
 		return -EINVAL;
 
 	r->membw.mba_sc = mba_sc;
-	list_for_each_entry(d, &r->domains, list)
-		setup_default_ctrlval(r, d->ctrl_val, d->mbps_val);
+	list_for_each_entry(d, &r->domains, list) {
+		hw_dom = resctrl_to_arch_dom(d);
+		setup_default_ctrlval(r, hw_dom->ctrl_val, hw_dom->mbps_val);
+	}
 
 	return 0;
 }
@@ -2265,6 +2272,7 @@ static int rdt_init_fs_context(struct fs_context *fc)
 static int reset_all_ctrls(struct rdt_resource *r)
 {
 	struct rdt_hw_resource *hw_res = resctrl_to_arch_res(r);
+	struct rdt_hw_domain *hw_dom;
 	struct msr_param msr_param;
 	cpumask_var_t cpu_mask;
 	struct rdt_domain *d;
@@ -2283,10 +2291,11 @@ static int reset_all_ctrls(struct rdt_resource *r)
 	 * from each domain to update the MSRs below.
 	 */
 	list_for_each_entry(d, &r->domains, list) {
+		hw_dom = resctrl_to_arch_dom(d);
 		cpumask_set_cpu(cpumask_any(&d->cpu_mask), cpu_mask);
 
 		for (i = 0; i < hw_res->num_closid; i++)
-			d->ctrl_val[i] = r->default_ctrl;
+			hw_dom->ctrl_val[i] = r->default_ctrl;
 	}
 	cpu = get_cpu();
 	/* Update CBM on this cpu if it's in cpu_mask. */
@@ -2665,7 +2674,7 @@ static int __init_one_rdt_domain(struct rdt_domain *d, struct rdt_resource *r,
 	d->have_new_ctrl = false;
 	d->new_ctrl = r->cache.shareable_bits;
 	used_b = r->cache.shareable_bits;
-	ctrl = d->ctrl_val;
+	ctrl = resctrl_to_arch_dom(d)->ctrl_val;
 	for (i = 0; i < closids_supported(); i++, ctrl++) {
 		if (closid_allocated(i) && i != closid) {
 			mode = rdtgroup_mode_by_closid(i);
@@ -2682,7 +2691,7 @@ static int __init_one_rdt_domain(struct rdt_domain *d, struct rdt_resource *r,
 			 * with an exclusive group.
 			 */
 			if (d_cdp)
-				peer_ctl = d_cdp->ctrl_val[i];
+				peer_ctl = resctrl_to_arch_dom(d_cdp)->ctrl_val[i];
 			else
 				peer_ctl = 0;
 			used_b |= *ctrl | peer_ctl;
include/linux/resctrl.h +31 -1
index ee92df9c7252..be6f5df78e31 100644
@@ -15,7 +15,37 @@ int proc_resctrl_show(struct seq_file *m,
 
 #endif
 
-struct rdt_domain;
+/**
+ * struct rdt_domain - group of CPUs sharing a resctrl resource
+ * @list:		all instances of this resource
+ * @id:			unique id for this instance
+ * @cpu_mask:		which CPUs share this resource
+ * @new_ctrl:		new ctrl value to be loaded
+ * @have_new_ctrl:	did user provide new_ctrl for this domain
+ * @rmid_busy_llc:	bitmap of which limbo RMIDs are above threshold
+ * @mbm_total:		saved state for MBM total bandwidth
+ * @mbm_local:		saved state for MBM local bandwidth
+ * @mbm_over:		worker to periodically read MBM h/w counters
+ * @cqm_limbo:		worker to periodically read CQM h/w counters
+ * @mbm_work_cpu:	worker CPU for MBM h/w counters
+ * @cqm_work_cpu:	worker CPU for CQM h/w counters
+ * @plr:		pseudo-locked region (if any) associated with domain
+ */
+struct rdt_domain {
+	struct list_head		list;
+	int				id;
+	struct cpumask			cpu_mask;
+	u32				new_ctrl;
+	bool				have_new_ctrl;
+	unsigned long			*rmid_busy_llc;
+	struct mbm_state		*mbm_total;
+	struct mbm_state		*mbm_local;
+	struct delayed_work		mbm_over;
+	struct delayed_work		cqm_limbo;
+	int				mbm_work_cpu;
+	int				cqm_work_cpu;
+	struct pseudo_lock_region	*plr;
+};
 
 /**
  * struct resctrl_cache - Cache allocation related data
Signature · 1 line
2.30.0
maintainer
#8 lore raw

Re: [PATCH v2 02/24] x86/resctrl: Split struct rdt_domain

Hi James,

Quoted 5 lines · James Morse

resctrl is the defacto Linux ABI for SoC resource partitioning features. To support it on another architecture, it needs to be abstracted from the features provided by Intel RDT and AMD PQoS, and moved to /fs/.

Split struct rdt_domain up too. Move everything that that is particular

s/that that/that/

Quoted 2 lines

to resctrl into a new header file. resctrl code paths touching a 'hw' struct indicates where an abstraction is needed.

Similar to previous patch it would help to explain how this split was chosen. For example, why are the CPUs to which a resource is associated not considered a hardware property?

Thank you

Reinette

author reviewer
#9 lore raw

Re: [PATCH v2 02/24] x86/resctrl: Split struct rdt_domain

Hi Reinette,

Quoted 7 lines · Reinette Chatre
Quoted 5 lines · James Morse

resctrl is the defacto Linux ABI for SoC resource partitioning features. To support it on another architecture, it needs to be abstracted from the features provided by Intel RDT and AMD PQoS, and moved to /fs/.

Split struct rdt_domain up too. Move everything that that is particular

Quoted 7 lines

s/that that/that/

Quoted 2 lines

to resctrl into a new header file. resctrl code paths touching a 'hw' struct indicates where an abstraction is needed.

Similar to previous patch it would help to explain how this split was chosen. For example, why are the CPUs to which a resource is associated not considered a hardware property?

Similarly, because the meaning of those CPUs doesn't differ or change between architectures.

I've expanded the middle paragraph in the commit message to explain why the arch specific things are arch specific:

| Continue by splitting struct rdt_domain, into an architecture private
| 'hw' struct, which contains the common resctrl structure that would be
| used by any architecture.
|
| The hardware values in ctrl_val and mbps_val need to be accessed via
| helpers to allow another architecture to convert these into a different
| format if necessary.
|
| After this split, filesystem code code paths touching a 'hw' struct
| indicates where an abstraction is needed.

and similarly changed the kernel doc comment.

Let me know if you prefer some other struct name.

Thanks,

James

maintainer
#10 lore raw

Re: [PATCH v2 02/24] x86/resctrl: Split struct rdt_domain

Hi James,

Quoted 10 lines · James Morse
Quoted 9 lines · Reinette Chatre
Quoted 5 lines · James Morse

resctrl is the defacto Linux ABI for SoC resource partitioning features. To support it on another architecture, it needs to be abstracted from the features provided by Intel RDT and AMD PQoS, and moved to /fs/.

Split struct rdt_domain up too. Move everything that that is particular

s/that that/that/

Quoted 7 lines
Quoted 5 lines
Quoted 2 lines

to resctrl into a new header file. resctrl code paths touching a 'hw' struct indicates where an abstraction is needed.

Similar to previous patch it would help to explain how this split was chosen. For example, why are the CPUs to which a resource is associated not considered a hardware property?

Similarly, because the meaning of those CPUs doesn't differ or change between architectures.

Got it. This seems to be a repeat of the discussion about patch 1. Please note that the description of rdt_hw_domain in this patch reads "hw attributes of a group of CPUs that share a resource". This can be understood to mean that the struct contains attributes discovered from hardware.

Quoted 13 lines

I've expanded the middle paragraph in the commit message to explain why the arch specific things are arch specific:

| Continue by splitting struct rdt_domain, into an architecture private
| 'hw' struct, which contains the common resctrl structure that would be
| used by any architecture.
|
| The hardware values in ctrl_val and mbps_val need to be accessed via
| helpers to allow another architecture to convert these into a different
| format if necessary.
|
| After this split, filesystem code code paths touching a 'hw' struct
| indicates where an abstraction is needed.
Quoted 1 line

and similarly changed the kernel doc comment.

Thank you. I assume this includes changing the "hw attributes of a group of CPUs that share a resource" I mention above.

Quoted 3 lines

Let me know if you prefer some other struct name.

I am ok with current naming. Other folks may have better ideas.

Thank you

Reinette

author reviewer
#11 lore raw

PATCH v2 3/24 x86/resctrl: Add a separate schema list for resctrl

In reply to James Morse

To support multiple architectures, the resctrl code needs to be split into a 'fs' specific part in core code, and an arch-specific backend.

It should be difficult for the arch-specific backends to diverge, supporting slightly different ABIs for user-space. For example, generating, parsing and validating the schema configuration values should be done in what becomes the core code to prevent divergence. Today, the schema emerge from which entries in the rdt_resources_all array the arch code has chosen to enable.

Start by creating a struct resctrl_schema, which will eventually hold the name and type of configuration values for resctrl.

---
Changes since v1:
* Renamed resctrl_all_schema list
* Used schemata_list as a prefix to make these easier to search for
* Added kerneldoc string
* Removed 'pending configuration' reference in commit message

---
 arch/x86/kernel/cpu/resctrl/internal.h |  1 +
 arch/x86/kernel/cpu/resctrl/rdtgroup.c | 43 +++++++++++++++++++++++++-
 include/linux/resctrl.h                |  9 ++++++
 3 files changed, 52 insertions(+), 1 deletion(-)
arch/x86/kernel/cpu/resctrl/internal.h +1 -0
index d3e47fd51e3a..8a9da490134b 100644
@@ -106,6 +106,7 @@ extern unsigned int resctrl_cqm_threshold;
 extern bool rdt_alloc_capable;
 extern bool rdt_mon_capable;
 extern unsigned int rdt_mon_features;
+extern struct list_head resctrl_schema_all;
 
 enum rdt_group_type {
 	RDTCTRL_GROUP = 0,
arch/x86/kernel/cpu/resctrl/rdtgroup.c +42 -1

77 lines not on this page · raw message

include/linux/resctrl.h +9 -0
index be6f5df78e31..092ff0c13b9b 100644
@@ -154,4 +154,13 @@ struct rdt_resource {
 
 };
 
+/**
+ * struct resctrl_schema - configuration abilities of a resource presented to user-space
+ * @list:	Member of resctrl's schema list
+ * @res:	The rdt_resource for this entry
+ */
+struct resctrl_schema {
+	struct list_head		list;
+	struct rdt_resource		*res;
+};
 #endif /* _RESCTRL_H */
Signature · 1 line
2.30.0
maintainer
#12 lore raw

Re: [PATCH v2 03/24] x86/resctrl: Add a separate schema list for resctrl

Hi James,

Quoted 8 lines · James Morse

To support multiple architectures, the resctrl code needs to be split into a 'fs' specific part in core code, and an arch-specific backend.

It should be difficult for the arch-specific backends to diverge, supporting slightly different ABIs for user-space. For example, generating, parsing and validating the schema configuration values should be done in what becomes the core code to prevent divergence. Today, the schema emerge from which entries in the rdt_resources_all

rdt_resources_all -> rdt_resources_all[]

Quoted 1 line

array the arch code has chosen to enable.

Throughout this series the commit messages do not have a consistent format. On top of this some commit messages uses terms like "Today" or "Previously" to document the context of the change ... sometimes in the middle of a commit message like here after the solution has been presented. In a long series like this it does make things increasingly harder to follow. There is an established commit message format in the x86 area that makes communicating changes much easier to do. Quoting Thomas ([1]) "A good structure is to explain the context, the problem and the solution in separate paragraphs and this order." Following this format makes changes much easier to communicate in a commit message and would definitely help this series during the next level of review by the contributors to [1].

[1] https://lore.kernel.org/lkml/[email protected]/

Quoted 2 lines

Start by creating a struct resctrl_schema, which will eventually hold

This term "eventually" shows up a lot in the commit messages of this series and causing some trouble because sometimes what it refers to is done in this series but sometimes what it refers to is _not_ done in this series. For example, the changes mentioned here are indeed made in this series but the changes mentioned in patch 6 as "eventually" are not. This concerns me about how many gaps are created by these changes.

Quoted 1 line

the name and type of configuration values for resctrl.

The future members are mentioned but the one introduced here and why is not.

...

Quoted 15 lines
arch/x86/kernel/cpu/resctrl/rdtgroup.c +3 -0
index 8c29304d3e01..73a695e7096d 100644
@@ -39,6 +39,9 @@ static struct kernfs_root *rdt_root;
  struct rdtgroup rdtgroup_default;
  LIST_HEAD(rdt_all_groups);
 
+/* list of entries for the schemata file */
+LIST_HEAD(resctrl_schema_all);
+
  /* Kernel fs node for "info" directory under root */
  static struct kernfs_node *kn_info;
 

arch/x86/kernel/cpu/resctrl/rdtgroup.c static int rdt_enable_ctx(struct rdt_fs_context *ctx) · line 2118

Quoted 7 lines
arch/x86/kernel/cpu/resctrl/rdtgroup.c +4 -0
@@ -2109,6 +2112,35 @@ static int rdt_enable_ctx(struct rdt_fs_context *ctx)
  	return ret;
  }
 
+static int schemata_list_create(void)
+{
+	struct rdt_resource *r;
+	struct resctrl_schema *s;

Please maintain reverse fir tree format.

Quoted 14 lines
arch/x86/kernel/cpu/resctrl/rdtgroup.c +14 -0
+
+	for_each_alloc_enabled_rdt_resource(r) {
+		s = kzalloc(sizeof(*s), GFP_KERNEL);
+		if (!s)
+			return -ENOMEM;
+
+		s->res = r;
+
+		INIT_LIST_HEAD(&s->list);
+		list_add(&s->list, &resctrl_schema_all);
+	}
+
+	return 0;
+}

...

include/linux/resctrl.h struct rdt_resource { · line 160

Quoted 12 lines
include/linux/resctrl.h +4 -0
index be6f5df78e31..092ff0c13b9b 100644
@@ -154,4 +154,13 @@ struct rdt_resource {
 
  };
 
+/**
+ * struct resctrl_schema - configuration abilities of a resource presented to user-space
+ * @list:	Member of resctrl's schema list
+ * @res:	The rdt_resource for this entry

Could this description be improved? It merely states what can be seen from the code.

Thank you

Reinette

author reviewer
#13 lore raw

PATCH v2 4/24 x86/resctrl: Pass the schema in info dir's private pointer

In reply to James Morse

Moving properties that resctrl exposes to user-space into the core 'fs' code, (e.g. the name of the schema), means some of the functions that back the filesystem need the schema struct (to where the properties are moved), but currently take the resource.

Once the CDP resources are merged, the resource doesn't reflect the right level of information.

For the info dirs that represent a control, the information needed is in the schema, as this is how the resource is being used. For the monitors, its the resource as L3CODE_MON doesn't make sense, and would monitor data too.

This difference means the type of the private pointers varies between control and monitor info dirs.

If the flags are RF_MON_INFO, its a struct rdt_resource. If the flags are RF_CTRL_INFO, its a struct resctrl_schema. Nothing in res_common_files[] has both flags.

---
Changes since v1:
* Added comment above removed for_each_alloc_enabled_rdt_resource() to hint at symmetry.

---
 arch/x86/kernel/cpu/resctrl/rdtgroup.c | 38 +++++++++++++++++---------
 1 file changed, 25 insertions(+), 13 deletions(-)
arch/x86/kernel/cpu/resctrl/rdtgroup.c +25 -13
index 73a695e7096d..92b94d85c689 100644
@@ -848,7 +848,8 @@ static int rdt_last_cmd_status_show(struct kernfs_open_file *of,
 static int rdt_num_closids_show(struct kernfs_open_file *of,
 				struct seq_file *seq, void *v)
 {
-	struct rdt_resource *r = of->kn->parent->priv;
+	struct resctrl_schema *s = of->kn->parent->priv;
+	struct rdt_resource *r = s->res;
 	struct rdt_hw_resource *hw_res;
 
 	hw_res = resctrl_to_arch_res(r);
@@ -859,7 +860,8 @@ static int rdt_num_closids_show(struct kernfs_open_file *of,
 static int rdt_default_ctrl_show(struct kernfs_open_file *of,
 			     struct seq_file *seq, void *v)
 {
-	struct rdt_resource *r = of->kn->parent->priv;
+	struct resctrl_schema *s = of->kn->parent->priv;
+	struct rdt_resource *r = s->res;
 
 	seq_printf(seq, "%x\n", r->default_ctrl);
 	return 0;
@@ -868,7 +870,8 @@ static int rdt_default_ctrl_show(struct kernfs_open_file *of,
 static int rdt_min_cbm_bits_show(struct kernfs_open_file *of,
 			     struct seq_file *seq, void *v)
 {
-	struct rdt_resource *r = of->kn->parent->priv;
+	struct resctrl_schema *s = of->kn->parent->priv;
+	struct rdt_resource *r = s->res;
 
 	seq_printf(seq, "%u\n", r->cache.min_cbm_bits);
 	return 0;
@@ -877,7 +880,8 @@ static int rdt_min_cbm_bits_show(struct kernfs_open_file *of,
 static int rdt_shareable_bits_show(struct kernfs_open_file *of,
 				   struct seq_file *seq, void *v)
 {
-	struct rdt_resource *r = of->kn->parent->priv;
+	struct resctrl_schema *s = of->kn->parent->priv;
+	struct rdt_resource *r = s->res;
 
 	seq_printf(seq, "%x\n", r->cache.shareable_bits);
 	return 0;
@@ -900,13 +904,14 @@ static int rdt_shareable_bits_show(struct kernfs_open_file *of,
 static int rdt_bit_usage_show(struct kernfs_open_file *of,
 			      struct seq_file *seq, void *v)
 {
-	struct rdt_resource *r = of->kn->parent->priv;
+	struct resctrl_schema *s = of->kn->parent->priv;
 	/*
 	 * Use unsigned long even though only 32 bits are used to ensure
 	 * test_bit() is used safely.
 	 */
 	unsigned long sw_shareable = 0, hw_shareable = 0;
 	unsigned long exclusive = 0, pseudo_locked = 0;
+	struct rdt_resource *r = s->res;
 	struct rdt_domain *dom;
 	int i, hwb, swb, excl, psl;
 	enum rdtgrp_mode mode;
@@ -978,7 +983,8 @@ static int rdt_bit_usage_show(struct kernfs_open_file *of,
 static int rdt_min_bw_show(struct kernfs_open_file *of,
 			     struct seq_file *seq, void *v)
 {
-	struct rdt_resource *r = of->kn->parent->priv;
+	struct resctrl_schema *s = of->kn->parent->priv;
+	struct rdt_resource *r = s->res;
 
 	seq_printf(seq, "%u\n", r->membw.min_bw);
 	return 0;
@@ -1009,7 +1015,8 @@ static int rdt_mon_features_show(struct kernfs_open_file *of,
 static int rdt_bw_gran_show(struct kernfs_open_file *of,
 			     struct seq_file *seq, void *v)
 {
-	struct rdt_resource *r = of->kn->parent->priv;
+	struct resctrl_schema *s = of->kn->parent->priv;
+	struct rdt_resource *r = s->res;
 
 	seq_printf(seq, "%u\n", r->membw.bw_gran);
 	return 0;
@@ -1018,7 +1025,8 @@ static int rdt_bw_gran_show(struct kernfs_open_file *of,
 static int rdt_delay_linear_show(struct kernfs_open_file *of,
 			     struct seq_file *seq, void *v)
 {
-	struct rdt_resource *r = of->kn->parent->priv;
+	struct resctrl_schema *s = of->kn->parent->priv;
+	struct rdt_resource *r = s->res;
 
 	seq_printf(seq, "%u\n", r->membw.delay_linear);
 	return 0;
@@ -1038,7 +1046,8 @@ static int max_threshold_occ_show(struct kernfs_open_file *of,
 static int rdt_thread_throttle_mode_show(struct kernfs_open_file *of,
 					 struct seq_file *seq, void *v)
 {
-	struct rdt_resource *r = of->kn->parent->priv;
+	struct resctrl_schema *s = of->kn->parent->priv;
+	struct rdt_resource *r = s->res;
 
 	if (r->membw.throttle_mode == THREAD_THROTTLE_PER_THREAD)
 		seq_puts(seq, "per-thread\n");
@@ -1771,14 +1780,14 @@ int rdtgroup_kn_mode_restore(struct rdtgroup *r, const char *name,
 	return ret;
 }
 
-static int rdtgroup_mkdir_info_resdir(struct rdt_resource *r, char *name,
+static int rdtgroup_mkdir_info_resdir(void *priv, char *name,
 				      unsigned long fflags)
 {
 	struct kernfs_node *kn_subdir;
 	int ret;
 
 	kn_subdir = kernfs_create_dir(kn_info, name,
-				      kn_info->mode, r);
+				      kn_info->mode, priv);
 	if (IS_ERR(kn_subdir))
 		return PTR_ERR(kn_subdir);
 
@@ -1795,6 +1804,7 @@ static int rdtgroup_mkdir_info_resdir(struct rdt_resource *r, char *name,
 
 static int rdtgroup_create_info_dir(struct kernfs_node *parent_kn)
 {
+	struct resctrl_schema *s;
 	struct rdt_resource *r;
 	unsigned long fflags;
 	char name[32];
@@ -1809,9 +1819,11 @@ static int rdtgroup_create_info_dir(struct kernfs_node *parent_kn)
 	if (ret)
 		goto out_destroy;
 
-	for_each_alloc_enabled_rdt_resource(r) {
+	/* loop over enabled controls, these are all alloc_enabled */
+	list_for_each_entry(s, &resctrl_schema_all, list) {
+		r = s->res;
 		fflags =  r->fflags | RF_CTRL_INFO;
-		ret = rdtgroup_mkdir_info_resdir(r, r->name, fflags);
+		ret = rdtgroup_mkdir_info_resdir(s, r->name, fflags);
 		if (ret)
 			goto out_destroy;
 	}
Signature · 1 line
2.30.0
author reviewer
#14 lore raw

PATCH v2 5/24 x86/resctrl: Label the resources with their configuration type

In reply to James Morse

The names of resources are used for the schemata name presented to user-space. These should be part of the filesystem code that is common to all architectures, as otherwise different architectures could accidentally support different schemata.

resctrl should be able to generate 'L3, L3CODE, L3DATA' from the architectures description of a cache at level 3 that supports CDP, creating two separate struct resctrl_schema for the CDP case that share the same resource, but differ in name and configuration type.

The configuration type is needed in struct resctrl_schema to generate the name, and as an index into the array of per-domain staged configurations that is added by a later patch.

Currently the resources are different for these types, the type is currently encoded in the name, (and cbm_idx_offset).

Label all the entries in rdt_resources_all[], and copy that value to struct resctrl_schema.

Copying the value ensures there is no mismatch, but allows the filesystem parts of resctrl to be modified to use the schema. Once the resources are merged, the filesystem code can assign this value based on the schema being created.

---
Changes since v1:
* {cdp,conf}_type typo
* Added kerneldoc comment

---
 arch/x86/kernel/cpu/resctrl/core.c     | 7 +++++++
 arch/x86/kernel/cpu/resctrl/internal.h | 2 ++
 arch/x86/kernel/cpu/resctrl/rdtgroup.c | 1 +
 include/linux/resctrl.h                | 8 ++++++++
 4 files changed, 18 insertions(+)
arch/x86/kernel/cpu/resctrl/core.c +7 -0
index ca43a7491fda..048c82e3baca 100644
@@ -62,6 +62,7 @@ mba_wrmsr_amd(struct rdt_domain *d, struct msr_param *m,
 struct rdt_hw_resource rdt_resources_all[] = {
 	[RDT_RESOURCE_L3] =
 	{
+		.conf_type			= CDP_BOTH,
 		.resctrl = {
 			.rid			= RDT_RESOURCE_L3,
 			.name			= "L3",
@@ -81,6 +82,7 @@ struct rdt_hw_resource rdt_resources_all[] = {
 	},
 	[RDT_RESOURCE_L3DATA] =
 	{
+		.conf_type			= CDP_DATA,
 		.resctrl = {
 			.rid			= RDT_RESOURCE_L3DATA,
 			.name			= "L3DATA",
@@ -100,6 +102,7 @@ struct rdt_hw_resource rdt_resources_all[] = {
 	},
 	[RDT_RESOURCE_L3CODE] =
 	{
+		.conf_type			= CDP_CODE,
 		.resctrl = {
 			.rid			= RDT_RESOURCE_L3CODE,
 			.name			= "L3CODE",
@@ -119,6 +122,7 @@ struct rdt_hw_resource rdt_resources_all[] = {
 	},
 	[RDT_RESOURCE_L2] =
 	{
+		.conf_type			= CDP_BOTH,
 		.resctrl = {
 			.rid			= RDT_RESOURCE_L2,
 			.name			= "L2",
@@ -138,6 +142,7 @@ struct rdt_hw_resource rdt_resources_all[] = {
 	},
 	[RDT_RESOURCE_L2DATA] =
 	{
+		.conf_type			= CDP_DATA,
 		.resctrl = {
 			.rid			= RDT_RESOURCE_L2DATA,
 			.name			= "L2DATA",
@@ -157,6 +162,7 @@ struct rdt_hw_resource rdt_resources_all[] = {
 	},
 	[RDT_RESOURCE_L2CODE] =
 	{
+		.conf_type			= CDP_CODE,
 		.resctrl = {
 			.rid			= RDT_RESOURCE_L2CODE,
 			.name			= "L2CODE",
@@ -176,6 +182,7 @@ struct rdt_hw_resource rdt_resources_all[] = {
 	},
 	[RDT_RESOURCE_MBA] =
 	{
+		.conf_type			= CDP_BOTH,
 		.resctrl = {
 			.rid			= RDT_RESOURCE_MBA,
 			.name			= "MB",
arch/x86/kernel/cpu/resctrl/internal.h +2 -0
index 8a9da490134b..57484d2f6214 100644
@@ -361,12 +361,14 @@ struct rdt_parse_data {
 
 /**
  * struct rdt_hw_resource - hw attributes of a resctrl resource
+ * @conf_type:		The type that should be used when configuring. temporary
  * @num_closid:		Number of CLOSIDs available.
  * @msr_base:		Base MSR address for CBMs
  * @msr_update:		Function pointer to update QOS MSRs
  * @mon_scale:		cqm counter * mon_scale = occupancy in bytes
  */
 struct rdt_hw_resource {
+	enum resctrl_conf_type	conf_type;
 	struct rdt_resource	resctrl;
 	int			num_closid;
 	unsigned int		msr_base;
arch/x86/kernel/cpu/resctrl/rdtgroup.c +1 -0
index 92b94d85c689..b5702238797b 100644
@@ -2135,6 +2135,7 @@ static int schemata_list_create(void)
 			return -ENOMEM;
 
 		s->res = r;
+		s->conf_type = resctrl_to_arch_res(r)->conf_type;
 
 		INIT_LIST_HEAD(&s->list);
 		list_add(&s->list, &resctrl_schema_all);
include/linux/resctrl.h +8 -0
index 092ff0c13b9b..c6f749f54765 100644
@@ -15,6 +15,12 @@ int proc_resctrl_show(struct seq_file *m,
 
 #endif
 
+enum resctrl_conf_type {
+	CDP_BOTH,
+	CDP_CODE,
+	CDP_DATA,
+};
+
 /**
  * struct rdt_domain - group of CPUs sharing a resctrl resource
  * @list:		all instances of this resource
@@ -157,10 +163,12 @@ struct rdt_resource {
 /**
  * struct resctrl_schema - configuration abilities of a resource presented to user-space
  * @list:	Member of resctrl's schema list
+ * @conf_type:	Whether this entry is for code/data/both
  * @res:	The rdt_resource for this entry
  */
 struct resctrl_schema {
 	struct list_head		list;
+	enum resctrl_conf_type		conf_type;
 	struct rdt_resource		*res;
 };
 #endif /* _RESCTRL_H */
Signature · 1 line
2.30.0
maintainer
#15 lore raw

Re: [PATCH v2 05/24] x86/resctrl: Label the resources with their configuration type

Hi James,

arch/x86/kernel/cpu/resctrl/internal.h struct rdt_parse_data { · line 364

Quoted 9 lines · James Morse
arch/x86/kernel/cpu/resctrl/internal.h +1 -0
index 8a9da490134b..57484d2f6214 100644
@@ -361,12 +361,14 @@ struct rdt_parse_data {
 
  /**
   * struct rdt_hw_resource - hw attributes of a resctrl resource
+ * @conf_type:		The type that should be used when configuring. temporary

This is annotated with "temporary" but still remains after entire series is applied.

Reinette

author reviewer
#16 lore raw

PATCH v2 6/24 x86/resctrl: Walk the resctrl schema list instead of an arch list

In reply to James Morse

Once the arch code is abstracted from the resctrl filesystem code the separate schema for CDP are created by the filesystem code. This means the same resource is used for different schema, or types of configuration.

Helpers like rdtgroup_cbm_overlaps() need the resctrl_schema to retrieve the configuration (or configurations). Before these helpers can be changed to take the schema instead of the resource, their callers must have the schema on hand.

Change the users of for_each_alloc_enabled_rdt_resource() to walk the schema instead. Schema were only created for alloc_enabled resources so these two lists are currently equivalent.

schemata_list_create() and rdt_kill_sb() are ignored. The first creates the schema list, and will eventually loop over the resource indexes using an arch helper to retrieve the resource. rdt_kill_sb() will eventually make use of an arch 'reset everything' helper.

After the filesystem code is moved, rdtgroup_pseudo_locked_in_hierarchy() remains part of the x86 specific hooks to support psuedo lock. This code walks each domain, and still does this after the separate resources are merged.

---
Changes since v1:
* Expanded commit message
* Split from a larger patch

---
 arch/x86/kernel/cpu/resctrl/ctrlmondata.c | 23 +++++++++++++++--------
 arch/x86/kernel/cpu/resctrl/rdtgroup.c    | 18 ++++++++++++------
 2 files changed, 27 insertions(+), 14 deletions(-)
arch/x86/kernel/cpu/resctrl/ctrlmondata.c +15 -8
index 2e7466659af3..a6f9548a8a59 100644
@@ -287,10 +287,12 @@ static int rdtgroup_parse_resource(char *resname, char *tok,
 				   struct rdtgroup *rdtgrp)
 {
 	struct rdt_hw_resource *hw_res;
+	struct resctrl_schema *s;
 	struct rdt_resource *r;
 
-	for_each_alloc_enabled_rdt_resource(r) {
-		hw_res = resctrl_to_arch_res(r);
+	list_for_each_entry(s, &resctrl_schema_all, list) {
+		r = s->res;
+		hw_res = resctrl_to_arch_res(s->res);
 		if (!strcmp(resname, r->name) && rdtgrp->closid < hw_res->num_closid)
 			return parse_line(tok, r, rdtgrp);
 	}
@@ -301,6 +303,7 @@ static int rdtgroup_parse_resource(char *resname, char *tok,
 ssize_t rdtgroup_schemata_write(struct kernfs_open_file *of,
 				char *buf, size_t nbytes, loff_t off)
 {
+	struct resctrl_schema *s;
 	struct rdtgroup *rdtgrp;
 	struct rdt_domain *dom;
 	struct rdt_resource *r;
@@ -331,8 +334,8 @@ ssize_t rdtgroup_schemata_write(struct kernfs_open_file *of,
 		goto out;
 	}
 
-	for_each_alloc_enabled_rdt_resource(r) {
-		list_for_each_entry(dom, &r->domains, list)
+	list_for_each_entry(s, &resctrl_schema_all, list) {
+		list_for_each_entry(dom, &s->res->domains, list)
 			dom->have_new_ctrl = false;
 	}
 
@@ -353,7 +356,8 @@ ssize_t rdtgroup_schemata_write(struct kernfs_open_file *of,
 			goto out;
 	}
 
-	for_each_alloc_enabled_rdt_resource(r) {
+	list_for_each_entry(s, &resctrl_schema_all, list) {
+		r = s->res;
 		ret = update_domains(r, rdtgrp->closid);
 		if (ret)
 			goto out;
@@ -401,6 +405,7 @@ int rdtgroup_schemata_show(struct kernfs_open_file *of,
 			   struct seq_file *s, void *v)
 {
 	struct rdt_hw_resource *hw_res;
+	struct resctrl_schema *schema;
 	struct rdtgroup *rdtgrp;
 	struct rdt_resource *r;
 	int ret = 0;
@@ -409,8 +414,10 @@ int rdtgroup_schemata_show(struct kernfs_open_file *of,
 	rdtgrp = rdtgroup_kn_lock_live(of->kn);
 	if (rdtgrp) {
 		if (rdtgrp->mode == RDT_MODE_PSEUDO_LOCKSETUP) {
-			for_each_alloc_enabled_rdt_resource(r)
+			list_for_each_entry(schema, &resctrl_schema_all, list) {
+				r = schema->res;
 				seq_printf(s, "%s:uninitialized\n", r->name);
+			}
 		} else if (rdtgrp->mode == RDT_MODE_PSEUDO_LOCKED) {
 			if (!rdtgrp->plr->d) {
 				rdt_last_cmd_clear();
@@ -424,8 +431,8 @@ int rdtgroup_schemata_show(struct kernfs_open_file *of,
 			}
 		} else {
 			closid = rdtgrp->closid;
-			for_each_alloc_enabled_rdt_resource(r) {
-				hw_res = resctrl_to_arch_res(r);
+			list_for_each_entry(schema, &resctrl_schema_all, list) {
+				hw_res = resctrl_to_arch_res(schema->res);
 				if (closid < hw_res->num_closid)
 					show_doms(s, r, closid);
 			}
arch/x86/kernel/cpu/resctrl/rdtgroup.c +12 -6
index b5702238797b..4f25c3c6f6e8 100644
@@ -104,12 +104,12 @@ int closids_supported(void)
 static void closid_init(void)
 {
 	struct rdt_hw_resource *hw_res;
-	struct rdt_resource *r;
+	struct resctrl_schema *s;
 	int rdt_min_closid = 32;
 
 	/* Compute rdt_min_closid across all resources */
-	for_each_alloc_enabled_rdt_resource(r) {
-		hw_res = resctrl_to_arch_res(r);
+	list_for_each_entry(s, &resctrl_schema_all, list) {
+		hw_res = resctrl_to_arch_res(s->res);
 		rdt_min_closid = min(rdt_min_closid, hw_res->num_closid);
 	}
 
@@ -1276,11 +1276,13 @@ static bool rdtgroup_mode_test_exclusive(struct rdtgroup *rdtgrp)
 {
 	struct rdt_hw_domain *hw_dom;
 	int closid = rdtgrp->closid;
+	struct resctrl_schema *s;
 	struct rdt_resource *r;
 	bool has_cache = false;
 	struct rdt_domain *d;
 
-	for_each_alloc_enabled_rdt_resource(r) {
+	list_for_each_entry(s, &resctrl_schema_all, list) {
+		r = s->res;
 		if (r->rid == RDT_RESOURCE_MBA)
 			continue;
 		has_cache = true;
@@ -1418,6 +1420,7 @@ unsigned int rdtgroup_cbm_to_size(struct rdt_resource *r,
 static int rdtgroup_size_show(struct kernfs_open_file *of,
 			      struct seq_file *s, void *v)
 {
+	struct resctrl_schema *schema;
 	struct rdt_hw_domain *hw_dom;
 	struct rdtgroup *rdtgrp;
 	struct rdt_resource *r;
@@ -1449,7 +1452,8 @@ static int rdtgroup_size_show(struct kernfs_open_file *of,
 		goto out;
 	}
 
-	for_each_alloc_enabled_rdt_resource(r) {
+	list_for_each_entry(schema, &resctrl_schema_all, list) {
+		r = schema->res;
 		sep = false;
 		seq_printf(s, "%*s:", max_name_width, r->name);
 		list_for_each_entry(d, &r->domains, list) {
@@ -2815,10 +2819,12 @@ static void rdtgroup_init_mba(struct rdt_resource *r)
 /* Initialize the RDT group's allocations. */
 static int rdtgroup_init_alloc(struct rdtgroup *rdtgrp)
 {
+	struct resctrl_schema *s;
 	struct rdt_resource *r;
 	int ret;
 
-	for_each_alloc_enabled_rdt_resource(r) {
+	list_for_each_entry(s, &resctrl_schema_all, list) {
+		r = s->res;
 		if (r->rid == RDT_RESOURCE_MBA) {
 			rdtgroup_init_mba(r);
 		} else {
Signature · 1 line
2.30.0
maintainer
#17 lore raw

Re: [PATCH v2 06/24] x86/resctrl: Walk the resctrl schema list instead of an arch list

Hi James,

Quoted 5 lines · James Morse

Once the arch code is abstracted from the resctrl filesystem code the separate schema for CDP are created by the filesystem code. This means the same resource is used for different schema, or types of configuration.

Quoted 8 lines

Helpers like rdtgroup_cbm_overlaps() need the resctrl_schema to retrieve the configuration (or configurations). Before these helpers can be changed to take the schema instead of the resource, their callers must have the schema on hand.

Change the users of for_each_alloc_enabled_rdt_resource() to walk the schema instead. Schema were only created for alloc_enabled resources so these two lists are currently equivalent.

Currently equivalent? Does this mean that at some point they will not be equivalent and this change will be impacted?

Quoted 5 lines

schemata_list_create() and rdt_kill_sb() are ignored. The first creates the schema list, and will eventually loop over the resource indexes using an arch helper to retrieve the resource. rdt_kill_sb() will eventually make use of an arch 'reset everything' helper.

Please elaborate on what "eventually" means here. It does not seem to indicate this patch series so please clarify that and any impacts.

Quoted 3 lines

After the filesystem code is moved, rdtgroup_pseudo_locked_in_hierarchy() remains part of the x86 specific hooks to support psuedo lock. This code

psuedo -> pseudo

Thank you

Reinette

author reviewer
#18 lore raw

PATCH v2 7/24 x86/resctrl: Store the effective num_closid in the schema

In reply to James Morse

The resctrl_schema struct holds properties that vary with the style of configuration that resctrl applies to a resource.

There are already two values for the hardware's num_closid, depending on whether the architecture presents the L3 or L3CODE/L3DATA resources.

As the way CDP changes the number of control groups that resctrl can create is part of the user-space interface, it should be managed by the filesystem parts of resctrl. This allows the architecture code to only describe the value the hardware supports.

Add num_closid to resctrl_schema. This is the value seen by the filesystem, and when the CDP resources are merged, will be different to the value described by the arch code when CDP is enabled.

These functions operate on the num_closid value that is exposed to user-space:
rdtgroup_parse_resource()
rdtgroup_schemata_show()
rdt_num_closids_show()
closid_init()

These are changed to use the schema value instead.

schemata_list_create() sets this value, and reaches into the architecture specific structure to get the value. This will eventually be replaced with a helper.

---
Changes since v1:
* Added missing : in a comment.
* Expanded commit message.
* Reordered patches

---
 arch/x86/kernel/cpu/resctrl/ctrlmondata.c |  9 +++------
 arch/x86/kernel/cpu/resctrl/rdtgroup.c    | 13 ++++---------
 include/linux/resctrl.h                   |  2 ++
 3 files changed, 9 insertions(+), 15 deletions(-)
arch/x86/kernel/cpu/resctrl/ctrlmondata.c +3 -6
index a6f9548a8a59..fcd6ca73ac41 100644
@@ -286,14 +286,12 @@ int update_domains(struct rdt_resource *r, int closid)
 static int rdtgroup_parse_resource(char *resname, char *tok,
 				   struct rdtgroup *rdtgrp)
 {
-	struct rdt_hw_resource *hw_res;
 	struct resctrl_schema *s;
 	struct rdt_resource *r;
 
 	list_for_each_entry(s, &resctrl_schema_all, list) {
 		r = s->res;
-		hw_res = resctrl_to_arch_res(s->res);
-		if (!strcmp(resname, r->name) && rdtgrp->closid < hw_res->num_closid)
+		if (!strcmp(resname, r->name) && rdtgrp->closid < s->num_closid)
 			return parse_line(tok, r, rdtgrp);
 	}
 	rdt_last_cmd_printf("Unknown or unsupported resource name '%s'\n", resname);
@@ -404,7 +402,6 @@ static void show_doms(struct seq_file *s, struct rdt_resource *r, int closid)
 int rdtgroup_schemata_show(struct kernfs_open_file *of,
 			   struct seq_file *s, void *v)
 {
-	struct rdt_hw_resource *hw_res;
 	struct resctrl_schema *schema;
 	struct rdtgroup *rdtgrp;
 	struct rdt_resource *r;
@@ -432,8 +429,8 @@ int rdtgroup_schemata_show(struct kernfs_open_file *of,
 		} else {
 			closid = rdtgrp->closid;
 			list_for_each_entry(schema, &resctrl_schema_all, list) {
-				hw_res = resctrl_to_arch_res(schema->res);
-				if (closid < hw_res->num_closid)
+				r = schema->res;
+				if (closid < schema->num_closid)
 					show_doms(s, r, closid);
 			}
 		}
arch/x86/kernel/cpu/resctrl/rdtgroup.c +4 -9
index 4f25c3c6f6e8..1ff883f68ee1 100644
@@ -103,15 +103,12 @@ int closids_supported(void)
 
 static void closid_init(void)
 {
-	struct rdt_hw_resource *hw_res;
 	struct resctrl_schema *s;
 	int rdt_min_closid = 32;
 
 	/* Compute rdt_min_closid across all resources */
-	list_for_each_entry(s, &resctrl_schema_all, list) {
-		hw_res = resctrl_to_arch_res(s->res);
-		rdt_min_closid = min(rdt_min_closid, hw_res->num_closid);
-	}
+	list_for_each_entry(s, &resctrl_schema_all, list)
+		rdt_min_closid = min(rdt_min_closid, s->num_closid);
 
 	closid_free_map = BIT_MASK(rdt_min_closid) - 1;
 
@@ -849,11 +846,8 @@ static int rdt_num_closids_show(struct kernfs_open_file *of,
 				struct seq_file *seq, void *v)
 {
 	struct resctrl_schema *s = of->kn->parent->priv;
-	struct rdt_resource *r = s->res;
-	struct rdt_hw_resource *hw_res;
 
-	hw_res = resctrl_to_arch_res(r);
-	seq_printf(seq, "%d\n", hw_res->num_closid);
+	seq_printf(seq, "%u\n", s->num_closid);
 	return 0;
 }
 
@@ -2140,6 +2134,7 @@ static int schemata_list_create(void)
 
 		s->res = r;
 		s->conf_type = resctrl_to_arch_res(r)->conf_type;
+		s->num_closid = resctrl_to_arch_res(r)->num_closid;
 
 		INIT_LIST_HEAD(&s->list);
 		list_add(&s->list, &resctrl_schema_all);
include/linux/resctrl.h +2 -0
index c6f749f54765..0ff10468940b 100644
@@ -165,10 +165,12 @@ struct rdt_resource {
  * @list:	Member of resctrl's schema list
  * @conf_type:	Whether this entry is for code/data/both
  * @res:	The rdt_resource for this entry
+ * @num_closid:	Number of CLOSIDs available for this resource
  */
 struct resctrl_schema {
 	struct list_head		list;
 	enum resctrl_conf_type		conf_type;
 	struct rdt_resource		*res;
+	int				num_closid;
 };
 #endif /* _RESCTRL_H */
Signature · 1 line
2.30.0
maintainer
#19 lore raw

Re: [PATCH v2 07/24] x86/resctrl: Store the effective num_closid in the schema

Hi James,

Quoted 8 lines · James Morse

The resctrl_schema struct holds properties that vary with the style of configuration that resctrl applies to a resource.

There are already two values for the hardware's num_closid, depending on whether the architecture presents the L3 or L3CODE/L3DATA resources.

As the way CDP changes the number of control groups that resctrl can create is part of the user-space interface, it should be managed by the filesystem

s/create is part of/create as part of/ ?

Quoted 22 lines

parts of resctrl. This allows the architecture code to only describe the value the hardware supports.

Add num_closid to resctrl_schema. This is the value seen by the filesystem, and when the CDP resources are merged, will be different to the value described by the arch code when CDP is enabled.

These functions operate on the num_closid value that is exposed to user-space:
rdtgroup_parse_resource()
rdtgroup_schemata_show()
rdt_num_closids_show()
closid_init()

These are changed to use the schema value instead.

schemata_list_create() sets this value, and reaches into the architecture specific structure to get the value. This will eventually be replaced with a helper.

...

include/linux/resctrl.h struct rdt_resource { · line 168

Quoted 9 lines
include/linux/resctrl.h +1 -0
index c6f749f54765..0ff10468940b 100644
@@ -165,10 +165,12 @@ struct rdt_resource {
   * @list:	Member of resctrl's schema list
   * @conf_type:	Whether this entry is for code/data/both
   * @res:	The rdt_resource for this entry
+ * @num_closid:	Number of CLOSIDs available for this resource

It is not clear what "this resource" is. This description is essentially the same as its copy found in rdt_hw_resource. Could you please improve these descriptions to capture the differences between the two num_closid values?

Thank you

Reinette

author reviewer
#20 lore raw

PATCH v2 8/24 x86/resctrl: Add resctrl_arch_get_num_closid()

In reply to James Morse

resctrl chooses whether to enable CDP, once it does, half the number of closid are available. MPAM doesn't behave like this, an in-kernel user of MPAM could be 'using CDP' while resctrl is not. (e.g. a KVM guest)

To move the 'half the closids' behaviour to be part of the filesystem code, each schema would have a num_closids. This may be different from the single resources num_closid if CDP is in use. An abstraction between the hardware property, and the way resctrl is using it is needed.

Add a helper to read the resource's num_closid from the arch code. This should return the number of closid that the resource supports, regardless of whether CDP is in use.

This helper is used in the one remaining path that is specific to the
filesystem: schemata_list_create().

In contrast reset_all_ctrls() sets up a structure for modifying the hardware, it is part of the architecture code, the maximum closid should be the maximum value the hardware has, regardless of the way resctrl is using it. All the uses in core.c are naturally part of the architecture specific code.

For now return the hw_res->num_closid, which is already adjusted for CDP. Once the CODE/DATA/BOTH resources are merged, resctrl can make the adjustment when copying the value to the schema's num_closid.

Using a type with an obvious size for the architecture specific helper means changing the type of num_closid to u32, which matches the type already used by struct rdtgroup.

---
Changes since v1:
* Rewrote commit message
* Whitespace fixes
* num_closid becomes u32 in all occurences to reduce surprises

---
 arch/x86/kernel/cpu/resctrl/core.c     | 5 +++++
 arch/x86/kernel/cpu/resctrl/internal.h | 2 +-
 arch/x86/kernel/cpu/resctrl/rdtgroup.c | 4 ++--
 include/linux/resctrl.h                | 6 +++++-
 4 files changed, 13 insertions(+), 4 deletions(-)
arch/x86/kernel/cpu/resctrl/core.c +5 -0
index 048c82e3baca..078822bc58ae 100644
@@ -450,6 +450,11 @@ struct rdt_domain *get_domain_from_cpu(int cpu, struct rdt_resource *r)
 	return NULL;
 }
 
+u32 resctrl_arch_get_num_closid(struct rdt_resource *r)
+{
+	return resctrl_to_arch_res(r)->num_closid;
+}
+
 void rdt_ctrl_update(void *arg)
 {
 	struct msr_param *m = arg;
arch/x86/kernel/cpu/resctrl/internal.h +1 -1
index 57484d2f6214..51a6e5f2f035 100644
@@ -370,7 +370,7 @@ struct rdt_parse_data {
 struct rdt_hw_resource {
 	enum resctrl_conf_type	conf_type;
 	struct rdt_resource	resctrl;
-	int			num_closid;
+	u32			num_closid;
 	unsigned int		msr_base;
 	void (*msr_update)	(struct rdt_domain *d, struct msr_param *m,
 				 struct rdt_resource *r);
arch/x86/kernel/cpu/resctrl/rdtgroup.c +2 -2
index 1ff883f68ee1..a8d8499e6919 100644
@@ -104,7 +104,7 @@ int closids_supported(void)
 static void closid_init(void)
 {
 	struct resctrl_schema *s;
-	int rdt_min_closid = 32;
+	u32 rdt_min_closid = 32;
 
 	/* Compute rdt_min_closid across all resources */
 	list_for_each_entry(s, &resctrl_schema_all, list)
@@ -2134,7 +2134,7 @@ static int schemata_list_create(void)
 
 		s->res = r;
 		s->conf_type = resctrl_to_arch_res(r)->conf_type;
-		s->num_closid = resctrl_to_arch_res(r)->num_closid;
+		s->num_closid = resctrl_arch_get_num_closid(r);
 
 		INIT_LIST_HEAD(&s->list);
 		list_add(&s->list, &resctrl_schema_all);
include/linux/resctrl.h +5 -1
index 0ff10468940b..5bd48c1ed497 100644
@@ -171,6 +171,10 @@ struct resctrl_schema {
 	struct list_head		list;
 	enum resctrl_conf_type		conf_type;
 	struct rdt_resource		*res;
-	int				num_closid;
+	u32				num_closid;
 };
+
+/* The number of closid supported by this resource regardless of CDP */
+u32 resctrl_arch_get_num_closid(struct rdt_resource *r);
+
 #endif /* _RESCTRL_H */
Signature · 1 line
2.30.0
author reviewer
#21 lore raw

PATCH v2 9/24 x86/resctrl: Pass the schema to resctrl filesystem functions

In reply to James Morse

Once the arch code is abstracted from the resctrl filesystem code the separate schema for CDP are created by the filesystem code. This means the same resource is used for different schema, or types of configuration.

Helpers like rdtgroup_cbm_overlaps() will need the resctrl_schema to retrieve the configuration (or configurations) as this is where properties like the name, closid limit and CODE/DATA/BOTH type are stored.

Change these functions to take a struct schema instead of the struct rdt_resource.

All the modified functions eventually move to be part of the filesystem code.

---
Changes since v1:
* split from a larger patch

---
 arch/x86/kernel/cpu/resctrl/ctrlmondata.c | 23 +++++++++++++----------
 arch/x86/kernel/cpu/resctrl/internal.h    |  6 +++---
 arch/x86/kernel/cpu/resctrl/rdtgroup.c    | 19 +++++++++++--------
 include/linux/resctrl.h                   |  3 ++-
 4 files changed, 29 insertions(+), 22 deletions(-)
arch/x86/kernel/cpu/resctrl/ctrlmondata.c +13 -10
index fcd6ca73ac41..dbbdd9f275e9 100644
@@ -57,9 +57,10 @@ static bool bw_validate(char *buf, unsigned long *data, struct rdt_resource *r)
 	return true;
 }
 
-int parse_bw(struct rdt_parse_data *data, struct rdt_resource *r,
+int parse_bw(struct rdt_parse_data *data, struct resctrl_schema *s,
 	     struct rdt_domain *d)
 {
+	struct rdt_resource *r = s->res;
 	unsigned long bw_val;
 
 	if (d->have_new_ctrl) {
@@ -125,10 +126,11 @@ static bool cbm_validate(char *buf, u32 *data, struct rdt_resource *r)
  * Read one cache bit mask (hex). Check that it is valid for the current
  * resource type.
  */
-int parse_cbm(struct rdt_parse_data *data, struct rdt_resource *r,
+int parse_cbm(struct rdt_parse_data *data, struct resctrl_schema *s,
 	      struct rdt_domain *d)
 {
 	struct rdtgroup *rdtgrp = data->rdtgrp;
+	struct rdt_resource *r = s->res;
 	u32 cbm_val;
 
 	if (d->have_new_ctrl) {
@@ -160,12 +162,12 @@ int parse_cbm(struct rdt_parse_data *data, struct rdt_resource *r,
 	 * The CBM may not overlap with the CBM of another closid if
 	 * either is exclusive.
 	 */
-	if (rdtgroup_cbm_overlaps(r, d, cbm_val, rdtgrp->closid, true)) {
+	if (rdtgroup_cbm_overlaps(s, d, cbm_val, rdtgrp->closid, true)) {
 		rdt_last_cmd_puts("Overlaps with exclusive group\n");
 		return -EINVAL;
 	}
 
-	if (rdtgroup_cbm_overlaps(r, d, cbm_val, rdtgrp->closid, false)) {
+	if (rdtgroup_cbm_overlaps(s, d, cbm_val, rdtgrp->closid, false)) {
 		if (rdtgrp->mode == RDT_MODE_EXCLUSIVE ||
 		    rdtgrp->mode == RDT_MODE_PSEUDO_LOCKSETUP) {
 			rdt_last_cmd_puts("Overlaps with other group\n");
@@ -185,9 +187,10 @@ int parse_cbm(struct rdt_parse_data *data, struct rdt_resource *r,
  * separated by ";". The "id" is in decimal, and must match one of
  * the "id"s for this resource.
  */
-static int parse_line(char *line, struct rdt_resource *r,
+static int parse_line(char *line, struct resctrl_schema *s,
 		      struct rdtgroup *rdtgrp)
 {
+	struct rdt_resource *r = s->res;
 	struct rdt_parse_data data;
 	char *dom = NULL, *id;
 	struct rdt_domain *d;
@@ -213,7 +216,7 @@ static int parse_line(char *line, struct rdt_resource *r,
 		if (d->id == dom_id) {
 			data.buf = dom;
 			data.rdtgrp = rdtgrp;
-			if (r->parse_ctrlval(&data, r, d))
+			if (r->parse_ctrlval(&data, s, d))
 				return -EINVAL;
 			if (rdtgrp->mode ==  RDT_MODE_PSEUDO_LOCKSETUP) {
 				/*
@@ -292,7 +295,7 @@ static int rdtgroup_parse_resource(char *resname, char *tok,
 	list_for_each_entry(s, &resctrl_schema_all, list) {
 		r = s->res;
 		if (!strcmp(resname, r->name) && rdtgrp->closid < s->num_closid)
-			return parse_line(tok, r, rdtgrp);
+			return parse_line(tok, s, rdtgrp);
 	}
 	rdt_last_cmd_printf("Unknown or unsupported resource name '%s'\n", resname);
 	return -EINVAL;
@@ -377,8 +380,9 @@ ssize_t rdtgroup_schemata_write(struct kernfs_open_file *of,
 	return ret ?: nbytes;
 }
 
-static void show_doms(struct seq_file *s, struct rdt_resource *r, int closid)
+static void show_doms(struct seq_file *s, struct resctrl_schema *schema, int closid)
 {
+	struct rdt_resource *r = schema->res;
 	struct rdt_hw_domain *hw_dom;
 	struct rdt_domain *dom;
 	bool sep = false;
@@ -429,9 +433,8 @@ int rdtgroup_schemata_show(struct kernfs_open_file *of,
 		} else {
 			closid = rdtgrp->closid;
 			list_for_each_entry(schema, &resctrl_schema_all, list) {
-				r = schema->res;
 				if (closid < schema->num_closid)
-					show_doms(s, r, closid);
+					show_doms(s, schema, closid);
 			}
 		}
 	} else {
arch/x86/kernel/cpu/resctrl/internal.h +3 -3
index 51a6e5f2f035..97f1fff90147 100644
@@ -383,9 +383,9 @@ static inline struct rdt_hw_resource *resctrl_to_arch_res(struct rdt_resource *r
 	return container_of(r, struct rdt_hw_resource, resctrl);
 }
 
-int parse_cbm(struct rdt_parse_data *data, struct rdt_resource *r,
+int parse_cbm(struct rdt_parse_data *data, struct resctrl_schema *s,
 	      struct rdt_domain *d);
-int parse_bw(struct rdt_parse_data *data, struct rdt_resource *r,
+int parse_bw(struct rdt_parse_data *data, struct resctrl_schema *s,
 	     struct rdt_domain *d);
 
 extern struct mutex rdtgroup_mutex;
@@ -483,7 +483,7 @@ ssize_t rdtgroup_schemata_write(struct kernfs_open_file *of,
 				char *buf, size_t nbytes, loff_t off);
 int rdtgroup_schemata_show(struct kernfs_open_file *of,
 			   struct seq_file *s, void *v);
-bool rdtgroup_cbm_overlaps(struct rdt_resource *r, struct rdt_domain *d,
+bool rdtgroup_cbm_overlaps(struct resctrl_schema *s, struct rdt_domain *d,
 			   unsigned long cbm, int closid, bool exclusive);
 unsigned int rdtgroup_cbm_to_size(struct rdt_resource *r, struct rdt_domain *d,
 				  unsigned long cbm);
arch/x86/kernel/cpu/resctrl/rdtgroup.c +11 -8
index a8d8499e6919..368cb3a16cfd 100644
@@ -1221,7 +1221,7 @@ static bool __rdtgroup_cbm_overlaps(struct rdt_resource *r, struct rdt_domain *d
 
 /**
  * rdtgroup_cbm_overlaps - Does CBM overlap with other use of hardware
- * @r: Resource to which domain instance @d belongs.
+ * @s: Schema for the resource to which domain instance @d belongs.
  * @d: The domain instance for which @closid is being tested.
  * @cbm: Capacity bitmask being tested.
  * @closid: Intended closid for @cbm.
@@ -1239,9 +1239,10 @@ static bool __rdtgroup_cbm_overlaps(struct rdt_resource *r, struct rdt_domain *d
  *
  * Return: true if CBM overlap detected, false if there is no overlap
  */
-bool rdtgroup_cbm_overlaps(struct rdt_resource *r, struct rdt_domain *d,
+bool rdtgroup_cbm_overlaps(struct resctrl_schema *s, struct rdt_domain *d,
 			   unsigned long cbm, int closid, bool exclusive)
 {
+	struct rdt_resource *r = s->res;
 	struct rdt_resource *r_cdp;
 	struct rdt_domain *d_cdp;
 
@@ -1282,7 +1283,8 @@ static bool rdtgroup_mode_test_exclusive(struct rdtgroup *rdtgrp)
 		has_cache = true;
 		list_for_each_entry(d, &r->domains, list) {
 			hw_dom = resctrl_to_arch_dom(d);
-			if (rdtgroup_cbm_overlaps(r, d, hw_dom->ctrl_val[closid],
+			if (rdtgroup_cbm_overlaps(s, d,
+						  hw_dom->ctrl_val[closid],
 						  rdtgrp->closid, false)) {
 				rdt_last_cmd_puts("Schemata overlaps\n");
 				return false;
@@ -2712,11 +2714,12 @@ static u32 cbm_ensure_valid(u32 _val, struct rdt_resource *r)
  * Set the RDT domain up to start off with all usable allocations. That is,
  * all shareable and unused bits. All-zero CBM is invalid.
  */
-static int __init_one_rdt_domain(struct rdt_domain *d, struct rdt_resource *r,
+static int __init_one_rdt_domain(struct rdt_domain *d, struct resctrl_schema *s,
 				 u32 closid)
 {
 	struct rdt_resource *r_cdp = NULL;
 	struct rdt_domain *d_cdp = NULL;
+	struct rdt_resource *r = s->res;
 	u32 used_b = 0, unused_b = 0;
 	unsigned long tmp_cbm;
 	enum rdtgrp_mode mode;
@@ -2786,13 +2789,13 @@ static int __init_one_rdt_domain(struct rdt_domain *d, struct rdt_resource *r,
  * If there are no more shareable bits available on any domain then
  * the entire allocation will fail.
  */
-static int rdtgroup_init_cat(struct rdt_resource *r, u32 closid)
+static int rdtgroup_init_cat(struct resctrl_schema *s, u32 closid)
 {
 	struct rdt_domain *d;
 	int ret;
 
-	list_for_each_entry(d, &r->domains, list) {
-		ret = __init_one_rdt_domain(d, r, closid);
+	list_for_each_entry(d, &s->res->domains, list) {
+		ret = __init_one_rdt_domain(d, s, closid);
 		if (ret < 0)
 			return ret;
 	}
@@ -2823,7 +2826,7 @@ static int rdtgroup_init_alloc(struct rdtgroup *rdtgrp)
 		if (r->rid == RDT_RESOURCE_MBA) {
 			rdtgroup_init_mba(r);
 		} else {
-			ret = rdtgroup_init_cat(r, rdtgrp->closid);
+			ret = rdtgroup_init_cat(s, rdtgrp->closid);
 			if (ret < 0)
 				return ret;
 		}
include/linux/resctrl.h +2 -1
index 5bd48c1ed497..60650ec08eb2 100644
@@ -115,6 +115,7 @@ struct resctrl_membw {
 };
 
 struct rdt_parse_data;
+struct resctrl_schema;
 
 /**
  * struct rdt_resource - attributes of a resctrl resource
@@ -153,7 +154,7 @@ struct rdt_resource {
 	u32			default_ctrl;
 	const char		*format_str;
 	int			(*parse_ctrlval)(struct rdt_parse_data *data,
-						 struct rdt_resource *r,
+						 struct resctrl_schema *s,
 						 struct rdt_domain *d);
 	struct list_head	evt_list;
 	unsigned long		fflags;
Signature · 1 line
2.30.0
author reviewer
#22 lore raw

PATCH v2 10/24 x86/resctrl: Swizzle rdt_resource and resctrl_schema in pseudo_lock_region

In reply to James Morse

struct pseudo_lock_region points to the rdt_resource. Once the resources are merged, this won't be unique. The resource name is moving into the scheam, so that eventually resctrl can generate it.

Swap pseudo_lock_region's rdt_resource pointer for a schema pointer.

---
 arch/x86/kernel/cpu/resctrl/ctrlmondata.c | 4 ++--
 arch/x86/kernel/cpu/resctrl/internal.h    | 6 +++---
 arch/x86/kernel/cpu/resctrl/pseudo_lock.c | 8 ++++----
 arch/x86/kernel/cpu/resctrl/rdtgroup.c    | 4 ++--
 4 files changed, 11 insertions(+), 11 deletions(-)
arch/x86/kernel/cpu/resctrl/ctrlmondata.c +2 -2
index dbbdd9f275e9..4428ec499037 100644
@@ -227,7 +227,7 @@ static int parse_line(char *line, struct resctrl_schema *s,
 				 * the required initialization for single
 				 * region and return.
 				 */
-				rdtgrp->plr->r = r;
+				rdtgrp->plr->s = s;
 				rdtgrp->plr->d = d;
 				rdtgrp->plr->cbm = d->new_ctrl;
 				d->plr = rdtgrp->plr;
@@ -426,7 +426,7 @@ int rdtgroup_schemata_show(struct kernfs_open_file *of,
 				ret = -ENODEV;
 			} else {
 				seq_printf(s, "%s:%d=%x\n",
-					   rdtgrp->plr->r->name,
+					   rdtgrp->plr->s->res->name,
 					   rdtgrp->plr->d->id,
 					   rdtgrp->plr->cbm);
 			}
arch/x86/kernel/cpu/resctrl/internal.h +3 -3
index 97f1fff90147..34b24a89eb6d 100644
@@ -158,8 +158,8 @@ struct mongroup {
 
 /**
  * struct pseudo_lock_region - pseudo-lock region information
- * @r:			RDT resource to which this pseudo-locked region
- *			belongs
+ * @s:			Resctrl schema for the resource to which this
+ *			pseudo-locked region belongs
  * @d:			RDT domain to which this pseudo-locked region
  *			belongs
  * @cbm:		bitmask of the pseudo-locked region
@@ -179,7 +179,7 @@ struct mongroup {
  * @pm_reqs:		Power management QoS requests related to this region
  */
 struct pseudo_lock_region {
-	struct rdt_resource	*r;
+	struct resctrl_schema	*s;
 	struct rdt_domain	*d;
 	u32			cbm;
 	wait_queue_head_t	lock_thread_wq;
arch/x86/kernel/cpu/resctrl/pseudo_lock.c +4 -4
index f8c5af759c0d..0ec6e335468c 100644
@@ -246,7 +246,7 @@ static void pseudo_lock_region_clear(struct pseudo_lock_region *plr)
 	plr->line_size = 0;
 	kfree(plr->kmem);
 	plr->kmem = NULL;
-	plr->r = NULL;
+	plr->s = NULL;
 	if (plr->d)
 		plr->d->plr = NULL;
 	plr->d = NULL;
@@ -290,10 +290,10 @@ static int pseudo_lock_region_init(struct pseudo_lock_region *plr)
 
 	ci = get_cpu_cacheinfo(plr->cpu);
 
-	plr->size = rdtgroup_cbm_to_size(plr->r, plr->d, plr->cbm);
+	plr->size = rdtgroup_cbm_to_size(plr->s->res, plr->d, plr->cbm);
 
 	for (i = 0; i < ci->num_leaves; i++) {
-		if (ci->info_list[i].level == plr->r->cache_level) {
+		if (ci->info_list[i].level == plr->s->res->cache_level) {
 			plr->line_size = ci->info_list[i].coherency_line_size;
 			return 0;
 		}
@@ -796,7 +796,7 @@ bool rdtgroup_cbm_overlaps_pseudo_locked(struct rdt_domain *d, unsigned long cbm
 	unsigned long cbm_b;
 
 	if (d->plr) {
-		cbm_len = d->plr->r->cache.cbm_len;
+		cbm_len = d->plr->s->res->cache.cbm_len;
 		cbm_b = d->plr->cbm;
 		if (bitmap_intersects(&cbm, &cbm_b, cbm_len))
 			return true;
arch/x86/kernel/cpu/resctrl/rdtgroup.c +2 -2
index 368cb3a16cfd..1e59d3e9b861 100644
@@ -1439,8 +1439,8 @@ static int rdtgroup_size_show(struct kernfs_open_file *of,
 			ret = -ENODEV;
 		} else {
 			seq_printf(s, "%*s:", max_name_width,
-				   rdtgrp->plr->r->name);
-			size = rdtgroup_cbm_to_size(rdtgrp->plr->r,
+				   rdtgrp->plr->s->res->name);
+			size = rdtgroup_cbm_to_size(rdtgrp->plr->s->res,
 						    rdtgrp->plr->d,
 						    rdtgrp->plr->cbm);
 			seq_printf(s, "%d=%u\n", rdtgrp->plr->d->id, size);
Signature · 1 line
2.30.0
maintainer
#23 lore raw

Re: [PATCH v2 10/24] x86/resctrl: Swizzle rdt_resource and resctrl_schema in pseudo_lock_region

Hi James,

Quoted 3 lines · James Morse

struct pseudo_lock_region points to the rdt_resource. Once the resources are merged, this won't be unique. The resource name is moving into the scheam, so that eventually resctrl can generate

s/scheam/schema/?

Reinette

author reviewer
#24 lore raw

PATCH v2 11/24 x86/resctrl: Move the schemata names into struct resctrl_schema

In reply to James Morse

Move the names used for the schemata file out of the resource and into struct resctrl_schema. This allows one resource to have two different names, based on the other schema properties.

Currently the names are copied as they must match the resource, which is currently separate. Eventually resctrl will generate two names for one resource.

The filesystem code should calculate max_name_width for padding the schemata file, move this to live with the code that will generate the names.

---
Changes since v1;
* Don't hardcode max_name_width, that leads to bugs
* Move max_name_width to live with the code that will generate the name.
* Fixed name/names typo

---
 arch/x86/kernel/cpu/resctrl/core.c        |  5 -----
 arch/x86/kernel/cpu/resctrl/ctrlmondata.c | 10 +++-------
 arch/x86/kernel/cpu/resctrl/rdtgroup.c    | 19 +++++++++++++++----
 include/linux/resctrl.h                   |  2 ++
 4 files changed, 20 insertions(+), 16 deletions(-)
arch/x86/kernel/cpu/resctrl/core.c +0 -5
index 078822bc58ae..4898fb6f70a4 100644
@@ -778,13 +778,8 @@ static int resctrl_offline_cpu(unsigned int cpu)
 static __init void rdt_init_padding(void)
 {
 	struct rdt_resource *r;
-	int cl;
 
 	for_each_alloc_capable_rdt_resource(r) {
-		cl = strlen(r->name);
-		if (cl > max_name_width)
-			max_name_width = cl;
-
 		if (r->data_width > max_data_width)
 			max_data_width = r->data_width;
 	}
arch/x86/kernel/cpu/resctrl/ctrlmondata.c +3 -7
index 4428ec499037..57c2b0e121d2 100644
@@ -290,11 +290,9 @@ static int rdtgroup_parse_resource(char *resname, char *tok,
 				   struct rdtgroup *rdtgrp)
 {
 	struct resctrl_schema *s;
-	struct rdt_resource *r;
 
 	list_for_each_entry(s, &resctrl_schema_all, list) {
-		r = s->res;
-		if (!strcmp(resname, r->name) && rdtgrp->closid < s->num_closid)
+		if (!strcmp(resname, s->name) && rdtgrp->closid < s->num_closid)
 			return parse_line(tok, s, rdtgrp);
 	}
 	rdt_last_cmd_printf("Unknown or unsupported resource name '%s'\n", resname);
@@ -388,7 +386,7 @@ static void show_doms(struct seq_file *s, struct resctrl_schema *schema, int clo
 	bool sep = false;
 	u32 ctrl_val;
 
-	seq_printf(s, "%*s:", max_name_width, r->name);
+	seq_printf(s, "%*s:", max_name_width, schema->name);
 	list_for_each_entry(dom, &r->domains, list) {
 		hw_dom = resctrl_to_arch_dom(dom);
 		if (sep)
@@ -408,7 +406,6 @@ int rdtgroup_schemata_show(struct kernfs_open_file *of,
 {
 	struct resctrl_schema *schema;
 	struct rdtgroup *rdtgrp;
-	struct rdt_resource *r;
 	int ret = 0;
 	u32 closid;
 
@@ -416,8 +413,7 @@ int rdtgroup_schemata_show(struct kernfs_open_file *of,
 	if (rdtgrp) {
 		if (rdtgrp->mode == RDT_MODE_PSEUDO_LOCKSETUP) {
 			list_for_each_entry(schema, &resctrl_schema_all, list) {
-				r = schema->res;
-				seq_printf(s, "%s:uninitialized\n", r->name);
+				seq_printf(s, "%s:uninitialized\n", schema->name);
 			}
 		} else if (rdtgrp->mode == RDT_MODE_PSEUDO_LOCKED) {
 			if (!rdtgrp->plr->d) {
arch/x86/kernel/cpu/resctrl/rdtgroup.c +15 -4
index 1e59d3e9b861..1ff969d7de75 100644
@@ -1439,7 +1439,7 @@ static int rdtgroup_size_show(struct kernfs_open_file *of,
 			ret = -ENODEV;
 		} else {
 			seq_printf(s, "%*s:", max_name_width,
-				   rdtgrp->plr->s->res->name);
+				   rdtgrp->plr->s->name);
 			size = rdtgroup_cbm_to_size(rdtgrp->plr->s->res,
 						    rdtgrp->plr->d,
 						    rdtgrp->plr->cbm);
@@ -1451,7 +1451,7 @@ static int rdtgroup_size_show(struct kernfs_open_file *of,
 	list_for_each_entry(schema, &resctrl_schema_all, list) {
 		r = schema->res;
 		sep = false;
-		seq_printf(s, "%*s:", max_name_width, r->name);
+		seq_printf(s, "%*s:", max_name_width, schema->name);
 		list_for_each_entry(d, &r->domains, list) {
 			hw_dom = resctrl_to_arch_dom(d);
 			if (sep)
@@ -1823,7 +1823,7 @@ static int rdtgroup_create_info_dir(struct kernfs_node *parent_kn)
 	list_for_each_entry(s, &resctrl_schema_all, list) {
 		r = s->res;
 		fflags =  r->fflags | RF_CTRL_INFO;
-		ret = rdtgroup_mkdir_info_resdir(s, r->name, fflags);
+		ret = rdtgroup_mkdir_info_resdir(s, s->name, fflags);
 		if (ret)
 			goto out_destroy;
 	}
@@ -2128,6 +2128,7 @@ static int schemata_list_create(void)
 {
 	struct rdt_resource *r;
 	struct resctrl_schema *s;
+	int ret, cl;
 
 	for_each_alloc_enabled_rdt_resource(r) {
 		s = kzalloc(sizeof(*s), GFP_KERNEL);
@@ -2138,6 +2139,16 @@ static int schemata_list_create(void)
 		s->conf_type = resctrl_to_arch_res(r)->conf_type;
 		s->num_closid = resctrl_arch_get_num_closid(r);
 
+		ret = snprintf(s->name, sizeof(s->name), r->name);
+		if (ret >= sizeof(s->name)) {
+			kfree(s);
+			return -EINVAL;
+		}
+
+		cl = strlen(s->name);
+		if (cl > max_name_width)
+			max_name_width = cl;
+
 		INIT_LIST_HEAD(&s->list);
 		list_add(&s->list, &resctrl_schema_all);
 	}
@@ -2771,7 +2782,7 @@ static int __init_one_rdt_domain(struct rdt_domain *d, struct resctrl_schema *s,
 	 */
 	tmp_cbm = d->new_ctrl;
 	if (bitmap_weight(&tmp_cbm, r->cache.cbm_len) < r->cache.min_cbm_bits) {
-		rdt_last_cmd_printf("No space on %s:%d\n", r->name, d->id);
+		rdt_last_cmd_printf("No space on %s:%d\n", s->name, d->id);
 		return -ENOSPC;
 	}
 	d->have_new_ctrl = true;
include/linux/resctrl.h +2 -0
index 60650ec08eb2..384e232650a8 100644
@@ -164,12 +164,14 @@ struct rdt_resource {
 /**
  * struct resctrl_schema - configuration abilities of a resource presented to user-space
  * @list:	Member of resctrl's schema list
+ * @name:	Name to use in "schemata" file
  * @conf_type:	Whether this entry is for code/data/both
  * @res:	The rdt_resource for this entry
  * @num_closid:	Number of CLOSIDs available for this resource
  */
 struct resctrl_schema {
 	struct list_head		list;
+	char				name[8];
 	enum resctrl_conf_type		conf_type;
 	struct rdt_resource		*res;
 	u32				num_closid;
Signature · 1 line
2.30.0
author reviewer
#25 lore raw

PATCH v2 12/24 x86/resctrl: Group staged configuration into a separate struct

In reply to James Morse

Arm's MPAM may have surprisingly large bitmaps for its cache portions as the architecture allows up to 4K portions. The size exposed via resctrl may not be the same, some scaling may occur.

The values written to hardware may be unlike the values received from resctrl, e.g. MBA percentages may be backed by a bitmap, or a maximum value that isn't a percentage.

Previously the values to be used by hardware are written directly to the ctrl_val array by the resctrl filesystem code before rdt_ctrl_update() uses a function pointer to write the values to hardware. This is a problem if scaling or conversion is needed by the architecture as the hardware values will be different, possibly requiring a second copy of the array.

To avoid having two copies, the arch code should own the ctrl_val array (to allow scaling and conversion), and provide helpers to expose the values in the format resctrl expects. The existing update_domains() will be part of the architecture specific code.

Currently each domain has a have_new_ctrl flag, which indicates the domains new_ctrl value should be written into the ctrl_val array and the hardware updated if the value is different. For lack of a better term, this is called staging the configuration. The changes are applied by update_domains(), which takes a resource and a closid as arguments. The have_new_ctrl flag is used to detect duplicate schemas being specified by user-space. (e.g. "L3:0=0xff;0=0xff")

Once the resources are merged, staged configuration changes for CODE and DATA types of resource would be written to the same domain. The domain needs to hold a staged configuration for each type of configuration.

Move the new_ctrl bitmap value and flag into a struct. Eventually there will be an array of these, the values in the struct need to be specified per type configuration, and there can be one configuration of each type staged.

---
Changes since v1:
* Expanded commit message
* Removed explicit clearing of have_new_ctrl,
* Moved ARRAY_SIZE() trickery to a later patch
* Removed extra whitespace

---
 arch/x86/kernel/cpu/resctrl/ctrlmondata.c | 43 +++++++++++++++--------
 arch/x86/kernel/cpu/resctrl/rdtgroup.c    | 22 +++++++-----
 include/linux/resctrl.h                   | 16 ++++++---
 3 files changed, 54 insertions(+), 27 deletions(-)
arch/x86/kernel/cpu/resctrl/ctrlmondata.c +29 -14
index 57c2b0e121d2..a47a792fdcb3 100644
@@ -62,16 +62,17 @@ int parse_bw(struct rdt_parse_data *data, struct resctrl_schema *s,
 {
 	struct rdt_resource *r = s->res;
 	unsigned long bw_val;
+	struct resctrl_staged_config *cfg = &d->staged_config;
 
-	if (d->have_new_ctrl) {
+	if (cfg->have_new_ctrl) {
 		rdt_last_cmd_printf("Duplicate domain %d\n", d->id);
 		return -EINVAL;
 	}
 
 	if (!bw_validate(data->buf, &bw_val, r))
 		return -EINVAL;
-	d->new_ctrl = bw_val;
-	d->have_new_ctrl = true;
+	cfg->new_ctrl = bw_val;
+	cfg->have_new_ctrl = true;
 
 	return 0;
 }
@@ -129,11 +130,12 @@ static bool cbm_validate(char *buf, u32 *data, struct rdt_resource *r)
 int parse_cbm(struct rdt_parse_data *data, struct resctrl_schema *s,
 	      struct rdt_domain *d)
 {
+	struct resctrl_staged_config *cfg = &d->staged_config;
 	struct rdtgroup *rdtgrp = data->rdtgrp;
 	struct rdt_resource *r = s->res;
 	u32 cbm_val;
 
-	if (d->have_new_ctrl) {
+	if (cfg->have_new_ctrl) {
 		rdt_last_cmd_printf("Duplicate domain %d\n", d->id);
 		return -EINVAL;
 	}
@@ -175,8 +177,8 @@ int parse_cbm(struct rdt_parse_data *data, struct resctrl_schema *s,
 		}
 	}
 
-	d->new_ctrl = cbm_val;
-	d->have_new_ctrl = true;
+	cfg->new_ctrl = cbm_val;
+	cfg->have_new_ctrl = true;
 
 	return 0;
 }
@@ -190,6 +192,7 @@ int parse_cbm(struct rdt_parse_data *data, struct resctrl_schema *s,
 static int parse_line(char *line, struct resctrl_schema *s,
 		      struct rdtgroup *rdtgrp)
 {
+	struct resctrl_staged_config *cfg;
 	struct rdt_resource *r = s->res;
 	struct rdt_parse_data data;
 	char *dom = NULL, *id;
@@ -219,6 +222,7 @@ static int parse_line(char *line, struct resctrl_schema *s,
 			if (r->parse_ctrlval(&data, s, d))
 				return -EINVAL;
 			if (rdtgrp->mode ==  RDT_MODE_PSEUDO_LOCKSETUP) {
+				cfg = &d->staged_config;
 				/*
 				 * In pseudo-locking setup mode and just
 				 * parsed a valid CBM that should be
@@ -229,7 +233,7 @@ static int parse_line(char *line, struct resctrl_schema *s,
 				 */
 				rdtgrp->plr->s = s;
 				rdtgrp->plr->d = d;
-				rdtgrp->plr->cbm = d->new_ctrl;
+				rdtgrp->plr->cbm = cfg->new_ctrl;
 				d->plr = rdtgrp->plr;
 				return 0;
 			}
@@ -239,14 +243,27 @@ static int parse_line(char *line, struct resctrl_schema *s,
 	return -EINVAL;
 }
 
+static void apply_config(struct rdt_hw_domain *hw_dom,
+			 struct resctrl_staged_config *cfg, int closid,
+			 cpumask_var_t cpu_mask, bool mba_sc)
+{
+	struct rdt_domain *dom = &hw_dom->resctrl;
+	u32 *dc = !mba_sc ? hw_dom->ctrl_val : hw_dom->mbps_val;
+
+	if (cfg->new_ctrl != dc[closid]) {
+		cpumask_set_cpu(cpumask_any(&dom->cpu_mask), cpu_mask);
+		dc[closid] = cfg->new_ctrl;
+	}
+}
+
 int update_domains(struct rdt_resource *r, int closid)
 {
+	struct resctrl_staged_config *cfg;
 	struct rdt_hw_domain *hw_dom;
 	struct msr_param msr_param;
 	cpumask_var_t cpu_mask;
 	struct rdt_domain *d;
 	bool mba_sc;
-	u32 *dc;
 	int cpu;
 
 	if (!zalloc_cpumask_var(&cpu_mask, GFP_KERNEL))
@@ -259,11 +276,9 @@ int update_domains(struct rdt_resource *r, int closid)
 	mba_sc = is_mba_sc(r);
 	list_for_each_entry(d, &r->domains, list) {
 		hw_dom = resctrl_to_arch_dom(d);
-		dc = !mba_sc ? hw_dom->ctrl_val : hw_dom->mbps_val;
-		if (d->have_new_ctrl && d->new_ctrl != dc[closid]) {
-			cpumask_set_cpu(cpumask_any(&d->cpu_mask), cpu_mask);
-			dc[closid] = d->new_ctrl;
-		}
+		cfg = &hw_dom->resctrl.staged_config;
+		if (cfg->have_new_ctrl)
+			apply_config(hw_dom, cfg, closid, cpu_mask, mba_sc);
 	}
 
 	/*
@@ -335,7 +350,7 @@ ssize_t rdtgroup_schemata_write(struct kernfs_open_file *of,
 
 	list_for_each_entry(s, &resctrl_schema_all, list) {
 		list_for_each_entry(dom, &s->res->domains, list)
-			dom->have_new_ctrl = false;
+			memset(&dom->staged_config, 0, sizeof(dom->staged_config));
 	}
 
 	while ((tok = strsep(&buf, "\n")) != NULL) {
arch/x86/kernel/cpu/resctrl/rdtgroup.c +13 -9
index 1ff969d7de75..0867319cf5ec 100644
@@ -2729,6 +2729,7 @@ static int __init_one_rdt_domain(struct rdt_domain *d, struct resctrl_schema *s,
 				 u32 closid)
 {
 	struct rdt_resource *r_cdp = NULL;
+	struct resctrl_staged_config *cfg;
 	struct rdt_domain *d_cdp = NULL;
 	struct rdt_resource *r = s->res;
 	u32 used_b = 0, unused_b = 0;
@@ -2738,8 +2739,9 @@ static int __init_one_rdt_domain(struct rdt_domain *d, struct resctrl_schema *s,
 	int i;
 
 	rdt_cdp_peer_get(r, d, &r_cdp, &d_cdp);
-	d->have_new_ctrl = false;
-	d->new_ctrl = r->cache.shareable_bits;
+	cfg = &d->staged_config;
+	cfg->have_new_ctrl = false;
+	cfg->new_ctrl = r->cache.shareable_bits;
 	used_b = r->cache.shareable_bits;
 	ctrl = resctrl_to_arch_dom(d)->ctrl_val;
 	for (i = 0; i < closids_supported(); i++, ctrl++) {
@@ -2763,29 +2765,29 @@ static int __init_one_rdt_domain(struct rdt_domain *d, struct resctrl_schema *s,
 				peer_ctl = 0;
 			used_b |= *ctrl | peer_ctl;
 			if (mode == RDT_MODE_SHAREABLE)
-				d->new_ctrl |= *ctrl | peer_ctl;
+				cfg->new_ctrl |= *ctrl | peer_ctl;
 		}
 	}
 	if (d->plr && d->plr->cbm > 0)
 		used_b |= d->plr->cbm;
 	unused_b = used_b ^ (BIT_MASK(r->cache.cbm_len) - 1);
 	unused_b &= BIT_MASK(r->cache.cbm_len) - 1;
-	d->new_ctrl |= unused_b;
+	cfg->new_ctrl |= unused_b;
 	/*
 	 * Force the initial CBM to be valid, user can
 	 * modify the CBM based on system availability.
 	 */
-	d->new_ctrl = cbm_ensure_valid(d->new_ctrl, r);
+	cfg->new_ctrl = cbm_ensure_valid(cfg->new_ctrl, r);
 	/*
 	 * Assign the u32 CBM to an unsigned long to ensure that
 	 * bitmap_weight() does not access out-of-bound memory.
 	 */
-	tmp_cbm = d->new_ctrl;
+	tmp_cbm = cfg->new_ctrl;
 	if (bitmap_weight(&tmp_cbm, r->cache.cbm_len) < r->cache.min_cbm_bits) {
 		rdt_last_cmd_printf("No space on %s:%d\n", s->name, d->id);
 		return -ENOSPC;
 	}
-	d->have_new_ctrl = true;
+	cfg->have_new_ctrl = true;
 
 	return 0;
 }
@@ -2817,11 +2819,13 @@ static int rdtgroup_init_cat(struct resctrl_schema *s, u32 closid)
 /* Initialize MBA resource with default values. */
 static void rdtgroup_init_mba(struct rdt_resource *r)
 {
+	struct resctrl_staged_config *cfg;
 	struct rdt_domain *d;
 
 	list_for_each_entry(d, &r->domains, list) {
-		d->new_ctrl = is_mba_sc(r) ? MBA_MAX_MBPS : r->default_ctrl;
-		d->have_new_ctrl = true;
+		cfg = &d->staged_config;
+		cfg->new_ctrl = is_mba_sc(r) ? MBA_MAX_MBPS : r->default_ctrl;
+		cfg->have_new_ctrl = true;
 	}
 }
 
include/linux/resctrl.h +12 -4
index 384e232650a8..2d0f1f6e0998 100644
@@ -21,13 +21,21 @@ enum resctrl_conf_type {
 	CDP_DATA,
 };
 
+/**
+ * struct resctrl_staged_config - parsed configuration to be applied
+ * @new_ctrl:		new ctrl value to be loaded
+ * @have_new_ctrl:	whether the user provide new_ctrl is valid
+ */
+struct resctrl_staged_config {
+	u32			new_ctrl;
+	bool			have_new_ctrl;
+};
+
 /**
  * struct rdt_domain - group of CPUs sharing a resctrl resource
  * @list:		all instances of this resource
  * @id:			unique id for this instance
  * @cpu_mask:		which CPUs share this resource
- * @new_ctrl:		new ctrl value to be loaded
- * @have_new_ctrl:	did user provide new_ctrl for this domain
  * @rmid_busy_llc:	bitmap of which limbo RMIDs are above threshold
  * @mbm_total:		saved state for MBM total bandwidth
  * @mbm_local:		saved state for MBM local bandwidth
@@ -36,13 +44,12 @@ enum resctrl_conf_type {
  * @mbm_work_cpu:	worker CPU for MBM h/w counters
  * @cqm_work_cpu:	worker CPU for CQM h/w counters
  * @plr:		pseudo-locked region (if any) associated with domain
+ * @staged_config:	parsed configuration to be applied
  */
 struct rdt_domain {
 	struct list_head		list;
 	int				id;
 	struct cpumask			cpu_mask;
-	u32				new_ctrl;
-	bool				have_new_ctrl;
 	unsigned long			*rmid_busy_llc;
 	struct mbm_state		*mbm_total;
 	struct mbm_state		*mbm_local;
@@ -51,6 +58,7 @@ struct rdt_domain {
 	int				mbm_work_cpu;
 	int				cqm_work_cpu;
 	struct pseudo_lock_region	*plr;
+	struct resctrl_staged_config	staged_config;
 };
 
 /**
Signature · 1 line
2.30.0
author reviewer
#26 lore raw

PATCH v2 13/24 x86/resctrl: Allow different CODE/DATA configurations to be staged

In reply to James Morse

Now that the staged configuration is grouped as a single struct, convert it to an array to allow resctrl to stage more than one configuration at a time for a single resource and closid.

To detect that the same schema is being specified twice when the schemata file is written, the same slot in the staged_configuration array must be used for each schema. Use the conf_type enum directly as an index.

---
Changes since v1:
* Renamed max enum value CDP_NUM_TYPES
* Whitespace and parenthesis
* Missing word in the commit message

---
 arch/x86/kernel/cpu/resctrl/ctrlmondata.c | 20 ++++++++++++++------
 arch/x86/kernel/cpu/resctrl/rdtgroup.c    |  5 +++--
 include/linux/resctrl.h                   |  5 ++++-
 3 files changed, 21 insertions(+), 9 deletions(-)
arch/x86/kernel/cpu/resctrl/ctrlmondata.c +14 -6
index a47a792fdcb3..c46300bce210 100644
@@ -60,10 +60,11 @@ static bool bw_validate(char *buf, unsigned long *data, struct rdt_resource *r)
 int parse_bw(struct rdt_parse_data *data, struct resctrl_schema *s,
 	     struct rdt_domain *d)
 {
+	struct resctrl_staged_config *cfg;
 	struct rdt_resource *r = s->res;
 	unsigned long bw_val;
-	struct resctrl_staged_config *cfg = &d->staged_config;
 
+	cfg = &d->staged_config[s->conf_type];
 	if (cfg->have_new_ctrl) {
 		rdt_last_cmd_printf("Duplicate domain %d\n", d->id);
 		return -EINVAL;
@@ -130,11 +131,12 @@ static bool cbm_validate(char *buf, u32 *data, struct rdt_resource *r)
 int parse_cbm(struct rdt_parse_data *data, struct resctrl_schema *s,
 	      struct rdt_domain *d)
 {
-	struct resctrl_staged_config *cfg = &d->staged_config;
 	struct rdtgroup *rdtgrp = data->rdtgrp;
+	struct resctrl_staged_config *cfg;
 	struct rdt_resource *r = s->res;
 	u32 cbm_val;
 
+	cfg = &d->staged_config[s->conf_type];
 	if (cfg->have_new_ctrl) {
 		rdt_last_cmd_printf("Duplicate domain %d\n", d->id);
 		return -EINVAL;
@@ -192,6 +194,7 @@ int parse_cbm(struct rdt_parse_data *data, struct resctrl_schema *s,
 static int parse_line(char *line, struct resctrl_schema *s,
 		      struct rdtgroup *rdtgrp)
 {
+	enum resctrl_conf_type t = s->conf_type;
 	struct resctrl_staged_config *cfg;
 	struct rdt_resource *r = s->res;
 	struct rdt_parse_data data;
@@ -222,7 +225,7 @@ static int parse_line(char *line, struct resctrl_schema *s,
 			if (r->parse_ctrlval(&data, s, d))
 				return -EINVAL;
 			if (rdtgrp->mode ==  RDT_MODE_PSEUDO_LOCKSETUP) {
-				cfg = &d->staged_config;
+				cfg = &d->staged_config[t];
 				/*
 				 * In pseudo-locking setup mode and just
 				 * parsed a valid CBM that should be
@@ -261,6 +264,7 @@ int update_domains(struct rdt_resource *r, int closid)
 	struct resctrl_staged_config *cfg;
 	struct rdt_hw_domain *hw_dom;
 	struct msr_param msr_param;
+	enum resctrl_conf_type t;
 	cpumask_var_t cpu_mask;
 	struct rdt_domain *d;
 	bool mba_sc;
@@ -276,9 +280,13 @@ int update_domains(struct rdt_resource *r, int closid)
 	mba_sc = is_mba_sc(r);
 	list_for_each_entry(d, &r->domains, list) {
 		hw_dom = resctrl_to_arch_dom(d);
-		cfg = &hw_dom->resctrl.staged_config;
-		if (cfg->have_new_ctrl)
+		for (t = 0; t < CDP_NUM_TYPES; t++) {
+			cfg = &hw_dom->resctrl.staged_config[t];
+			if (!cfg->have_new_ctrl)
+				continue;
+
 			apply_config(hw_dom, cfg, closid, cpu_mask, mba_sc);
+		}
 	}
 
 	/*
@@ -350,7 +358,7 @@ ssize_t rdtgroup_schemata_write(struct kernfs_open_file *of,
 
 	list_for_each_entry(s, &resctrl_schema_all, list) {
 		list_for_each_entry(dom, &s->res->domains, list)
-			memset(&dom->staged_config, 0, sizeof(dom->staged_config));
+			memset(dom->staged_config, 0, sizeof(dom->staged_config));
 	}
 
 	while ((tok = strsep(&buf, "\n")) != NULL) {
arch/x86/kernel/cpu/resctrl/rdtgroup.c +3 -2
index 0867319cf5ec..f89cdb107297 100644
@@ -2728,6 +2728,7 @@ static u32 cbm_ensure_valid(u32 _val, struct rdt_resource *r)
 static int __init_one_rdt_domain(struct rdt_domain *d, struct resctrl_schema *s,
 				 u32 closid)
 {
+	enum resctrl_conf_type t = s->conf_type;
 	struct rdt_resource *r_cdp = NULL;
 	struct resctrl_staged_config *cfg;
 	struct rdt_domain *d_cdp = NULL;
@@ -2739,7 +2740,7 @@ static int __init_one_rdt_domain(struct rdt_domain *d, struct resctrl_schema *s,
 	int i;
 
 	rdt_cdp_peer_get(r, d, &r_cdp, &d_cdp);
-	cfg = &d->staged_config;
+	cfg = &d->staged_config[t];
 	cfg->have_new_ctrl = false;
 	cfg->new_ctrl = r->cache.shareable_bits;
 	used_b = r->cache.shareable_bits;
@@ -2823,7 +2824,7 @@ static void rdtgroup_init_mba(struct rdt_resource *r)
 	struct rdt_domain *d;
 
 	list_for_each_entry(d, &r->domains, list) {
-		cfg = &d->staged_config;
+		cfg = &d->staged_config[CDP_BOTH];
 		cfg->new_ctrl = is_mba_sc(r) ? MBA_MAX_MBPS : r->default_ctrl;
 		cfg->have_new_ctrl = true;
 	}
include/linux/resctrl.h +4 -1
index 2d0f1f6e0998..06654f9346a2 100644
@@ -21,6 +21,9 @@ enum resctrl_conf_type {
 	CDP_DATA,
 };
 
+#define CDP_NUM_TYPES	(CDP_DATA + 1)
+
+
 /**
  * struct resctrl_staged_config - parsed configuration to be applied
  * @new_ctrl:		new ctrl value to be loaded
@@ -58,7 +61,7 @@ struct rdt_domain {
 	int				mbm_work_cpu;
 	int				cqm_work_cpu;
 	struct pseudo_lock_region	*plr;
-	struct resctrl_staged_config	staged_config;
+	struct resctrl_staged_config	staged_config[CDP_NUM_TYPES];
 };
 
 /**
Signature · 1 line
2.30.0
author reviewer
#27 lore raw

PATCH v2 14/24 x86/resctrl: Rename update_domains() resctrl_arch_update_domains()

In reply to James Morse

update_domains() merges the staged configuration changes into the arch codes configuration array. Rename to make it clear its part of the arch code interface to resctrl.

---
Changes since v1:
* The closid is no longer staged as from resctrl its always going to be the same number even with CDP.

---
 arch/x86/kernel/cpu/resctrl/ctrlmondata.c | 4 ++--
 arch/x86/kernel/cpu/resctrl/internal.h    | 1 -
 arch/x86/kernel/cpu/resctrl/rdtgroup.c    | 2 +-
 include/linux/resctrl.h                   | 1 +
 4 files changed, 4 insertions(+), 4 deletions(-)
arch/x86/kernel/cpu/resctrl/ctrlmondata.c +2 -2
index c46300bce210..271f5d28412a 100644
@@ -259,7 +259,7 @@ static void apply_config(struct rdt_hw_domain *hw_dom,
 	}
 }
 
-int update_domains(struct rdt_resource *r, int closid)
+int resctrl_arch_update_domains(struct rdt_resource *r, u32 closid)
 {
 	struct resctrl_staged_config *cfg;
 	struct rdt_hw_domain *hw_dom;
@@ -380,7 +380,7 @@ ssize_t rdtgroup_schemata_write(struct kernfs_open_file *of,
 
 	list_for_each_entry(s, &resctrl_schema_all, list) {
 		r = s->res;
-		ret = update_domains(r, rdtgrp->closid);
+		ret = resctrl_arch_update_domains(r, rdtgrp->closid);
 		if (ret)
 			goto out;
 	}
arch/x86/kernel/cpu/resctrl/internal.h +0 -1
index 34b24a89eb6d..ff1d58dee66b 100644
@@ -498,7 +498,6 @@ void rdt_pseudo_lock_release(void);
 int rdtgroup_pseudo_lock_create(struct rdtgroup *rdtgrp);
 void rdtgroup_pseudo_lock_remove(struct rdtgroup *rdtgrp);
 struct rdt_domain *get_domain_from_cpu(int cpu, struct rdt_resource *r);
-int update_domains(struct rdt_resource *r, int closid);
 int closids_supported(void);
 void closid_free(int closid);
 int alloc_rmid(void);
arch/x86/kernel/cpu/resctrl/rdtgroup.c +1 -1
index f89cdb107297..a103ed4f29fa 100644
@@ -2847,7 +2847,7 @@ static int rdtgroup_init_alloc(struct rdtgroup *rdtgrp)
 				return ret;
 		}
 
-		ret = update_domains(r, rdtgrp->closid);
+		ret = resctrl_arch_update_domains(r, rdtgrp->closid);
 		if (ret < 0) {
 			rdt_last_cmd_puts("Failed to initialize allocations\n");
 			return ret;
include/linux/resctrl.h +1 -0
index 06654f9346a2..a6d2d7a25293 100644
@@ -190,5 +190,6 @@ struct resctrl_schema {
 
 /* The number of closid supported by this resource regardless of CDP */
 u32 resctrl_arch_get_num_closid(struct rdt_resource *r);
+int resctrl_arch_update_domains(struct rdt_resource *r, u32 closid);
 
 #endif /* _RESCTRL_H */
Signature · 1 line
2.30.0
author reviewer
#28 lore raw

PATCH v2 15/24 x86/resctrl: Add a helper to read a closid's configuration

In reply to James Morse

The hardware configuration may look completely different to the values resctrl gets from user-space. The staged configuration and resctrl_arch_update_domains() allow the architecture to convert or translate these values.
(e.g. Arm's MPAM may back MBA's percentage control using the
'BWPBM' bitmap)

Resctrl shouldn't read or write the ctrl_val[] values directly. As a step towards taking direct access away, add a helper to read the current configuration.

This will allow another architecture to scale the bitmaps if necessary, and possibly use controls that don't take the user-space control format at all.

---
 arch/x86/kernel/cpu/resctrl/ctrlmondata.c | 16 ++++++---
 arch/x86/kernel/cpu/resctrl/monitor.c     |  6 +++-
 arch/x86/kernel/cpu/resctrl/rdtgroup.c    | 43 ++++++++++-------------
 include/linux/resctrl.h                   |  2 ++
 4 files changed, 37 insertions(+), 30 deletions(-)
arch/x86/kernel/cpu/resctrl/ctrlmondata.c +12 -4
index 271f5d28412a..1cd54402b02a 100644
@@ -401,22 +401,30 @@ ssize_t rdtgroup_schemata_write(struct kernfs_open_file *of,
 	return ret ?: nbytes;
 }
 
+void resctrl_arch_get_config(struct rdt_resource *r, struct rdt_domain *d,
+			     u32 closid, u32 *value)
+{
+	struct rdt_hw_domain *hw_dom = resctrl_to_arch_dom(d);
+
+	if (!is_mba_sc(r))
+		*value = hw_dom->ctrl_val[closid];
+	else
+		*value = hw_dom->mbps_val[closid];
+}
+
 static void show_doms(struct seq_file *s, struct resctrl_schema *schema, int closid)
 {
 	struct rdt_resource *r = schema->res;
-	struct rdt_hw_domain *hw_dom;
 	struct rdt_domain *dom;
 	bool sep = false;
 	u32 ctrl_val;
 
 	seq_printf(s, "%*s:", max_name_width, schema->name);
 	list_for_each_entry(dom, &r->domains, list) {
-		hw_dom = resctrl_to_arch_dom(dom);
 		if (sep)
 			seq_puts(s, ";");
 
-		ctrl_val = (!is_mba_sc(r) ? hw_dom->ctrl_val[closid] :
-			    hw_dom->mbps_val[closid]);
+		resctrl_arch_get_config(r, dom, closid, &ctrl_val);
 		seq_printf(s, r->format_str, dom->id, max_data_width,
 			   ctrl_val);
 		sep = true;
arch/x86/kernel/cpu/resctrl/monitor.c +5 -1
index 6f700ff1b64c..c4d146658b44 100644
@@ -442,8 +442,12 @@ static void update_mba_bw(struct rdtgroup *rgrp, struct rdt_domain *dom_mbm)
 	hw_dom_mba = resctrl_to_arch_dom(dom_mba);
 
 	cur_bw = pmbm_data->prev_bw;
-	user_bw = hw_dom_mba->mbps_val[closid];
+	resctrl_arch_get_config(r_mba, dom_mba, closid, &user_bw);
 	delta_bw = pmbm_data->delta_bw;
+	/*
+	 * resctrl_arch_get_config() chooses the mbps/ctrl value to return
+	 * based on is_mba_sc(). For now, reach into the hw_dom.
+	 */
 	cur_msr_val = hw_dom_mba->ctrl_val[closid];
 
 	/*
arch/x86/kernel/cpu/resctrl/rdtgroup.c +18 -25

132 lines not on this page · raw message

include/linux/resctrl.h +2 -0
index a6d2d7a25293..dec27b03675e 100644
@@ -191,5 +191,7 @@ struct resctrl_schema {
 /* The number of closid supported by this resource regardless of CDP */
 u32 resctrl_arch_get_num_closid(struct rdt_resource *r);
 int resctrl_arch_update_domains(struct rdt_resource *r, u32 closid);
+void resctrl_arch_get_config(struct rdt_resource *r, struct rdt_domain *d,
+			     u32 closid, u32 *value);
 
 #endif /* _RESCTRL_H */
Signature · 1 line
2.30.0
author reviewer
#29 lore raw

PATCH v2 16/24 x86/resctrl: Add a helper to read/set the CDP configuration

In reply to James Morse

Previously whether CDP is enabled is described in the alloc_enabled and alloc_capable flags, which are set differently between the L3 and L3CODE+L3DATA resources.

To merge these resources, to give us one configuration, the CDP state of the resource needs tracking explicitly. Add cdp_capable and cdp_enabled as something the arch code manages.

resctrl_arch_set_cdp_enabled() lets resctrl enable or disable CDP on a resource. resctrl_arch_get_cdp_enabled() lets it read the current state.

With Arm's MPAM, separate code and data closids is a part of the CPU configuration. Enabling CDP for one resource means it is enable for all resources, as all resources would need to configure both interpretations of the closid value.

---
Changes since v1:
* Added prototype for resctrl_arch_set_cdp_enabled()
* s/Currently/Previously/
* rdt_get_cdp_config() accesses the array directly as most of the code here disappears once the resrouces are merged.

It isn't practical for MPAM to hide the CDP emulation by applying the same 'L3' configuration to the two closid that are in use, as this would then consume two monitors, which are likely to be in short supply.

---
 arch/x86/kernel/cpu/resctrl/core.c        |  4 ++
 arch/x86/kernel/cpu/resctrl/internal.h    | 13 ++++-
 arch/x86/kernel/cpu/resctrl/pseudo_lock.c |  4 +-
 arch/x86/kernel/cpu/resctrl/rdtgroup.c    | 67 +++++++++++++----------
 4 files changed, 57 insertions(+), 31 deletions(-)
arch/x86/kernel/cpu/resctrl/core.c +4 -0
index 4898fb6f70a4..4db28ce114bd 100644
@@ -374,6 +374,10 @@ static void rdt_get_cdp_config(int level, int type)
 	 * "cdp" during resctrl file system mount time.
 	 */
 	r->alloc_enabled = false;
+	rdt_resources_all[level].cdp_enabled = false;
+	rdt_resources_all[type].cdp_enabled = false;
+	rdt_resources_all[level].cdp_capable = true;
+	rdt_resources_all[type].cdp_capable = true;
 }
 
 static void rdt_get_cdp_l3_config(void)
arch/x86/kernel/cpu/resctrl/internal.h +12 -1
index ff1d58dee66b..9bafe3b32035 100644
@@ -366,6 +366,8 @@ struct rdt_parse_data {
  * @msr_base:		Base MSR address for CBMs
  * @msr_update:		Function pointer to update QOS MSRs
  * @mon_scale:		cqm counter * mon_scale = occupancy in bytes
+ * @cdp_enabled:	CDP state of this resource
+ * @cdp_capable:	Is the CDP feature available on this resource
  */
 struct rdt_hw_resource {
 	enum resctrl_conf_type	conf_type;
@@ -376,6 +378,8 @@ struct rdt_hw_resource {
 				 struct rdt_resource *r);
 	unsigned int		mon_scale;
 	unsigned int		mbm_width;
+	bool			cdp_enabled;
+	bool			cdp_capable;
 };
 
 static inline struct rdt_hw_resource *resctrl_to_arch_res(struct rdt_resource *r)
@@ -396,7 +400,7 @@ DECLARE_STATIC_KEY_FALSE(rdt_alloc_enable_key);
 
 extern struct dentry *debugfs_resctrl;
 
-enum {
+enum resctrl_res_level {
 	RDT_RESOURCE_L3,
 	RDT_RESOURCE_L3DATA,
 	RDT_RESOURCE_L3CODE,
@@ -417,6 +421,13 @@ static inline struct rdt_resource *resctrl_inc(struct rdt_resource *res)
 	return &hw_res->resctrl;
 }
 
+static inline bool resctrl_arch_get_cdp_enabled(enum resctrl_res_level l)
+{
+	return rdt_resources_all[l].cdp_enabled;
+}
+
+int resctrl_arch_set_cdp_enabled(enum resctrl_res_level l, bool enable);
+
 #define for_each_rdt_resource(r)					      \
 	for (r = &rdt_resources_all[0].resctrl;				      \
 	     r < &rdt_resources_all[RDT_NUM_RESOURCES].resctrl;		      \
arch/x86/kernel/cpu/resctrl/pseudo_lock.c +2 -2
index 0ec6e335468c..d4ab1909a7bc 100644
@@ -684,8 +684,8 @@ int rdtgroup_locksetup_enter(struct rdtgroup *rdtgrp)
 	 *   resource, the portion of cache used by it should be made
 	 *   unavailable to all future allocations from both resources.
 	 */
-	if (rdt_resources_all[RDT_RESOURCE_L3DATA].resctrl.alloc_enabled ||
-	    rdt_resources_all[RDT_RESOURCE_L2DATA].resctrl.alloc_enabled) {
+	if (resctrl_arch_get_cdp_enabled(RDT_RESOURCE_L3) ||
+	    resctrl_arch_get_cdp_enabled(RDT_RESOURCE_L2)) {
 		rdt_last_cmd_puts("CDP enabled\n");
 		return -EINVAL;
 	}
arch/x86/kernel/cpu/resctrl/rdtgroup.c +39 -28

110 lines not on this page · raw message

Signature · 1 line
2.30.0
author reviewer
#30 lore raw

PATCH v2 17/24 x86/resctrl: Use cdp_enabled in rdt_domain_reconfigure_cdp()

In reply to James Morse

rdt_domain_reconfigure_cdp() infers whether CDP is enabled by checking the alloc_capable and alloc_enabled flags of the data resources.

Now that we have an explicit cdp_enabled, use that.

---
 arch/x86/kernel/cpu/resctrl/rdtgroup.c | 8 +++++---
 1 file changed, 5 insertions(+), 3 deletions(-)
arch/x86/kernel/cpu/resctrl/rdtgroup.c +5 -3
index 55c17adf763c..378cad0020da 100644
@@ -1933,14 +1933,16 @@ static int set_cache_qos_cfg(int level, bool enable)
 /* Restore the qos cfg state when a domain comes online */
 void rdt_domain_reconfigure_cdp(struct rdt_resource *r)
 {
-	if (!r->alloc_capable)
+	struct rdt_hw_resource *hw_res = resctrl_to_arch_res(r);
+
+	if (!hw_res->cdp_capable)
 		return;
 
 	if (r == &rdt_resources_all[RDT_RESOURCE_L2DATA].resctrl)
-		l2_qos_cfg_update(&r->alloc_enabled);
+		l2_qos_cfg_update(&hw_res->cdp_enabled);
 
 	if (r == &rdt_resources_all[RDT_RESOURCE_L3DATA].resctrl)
-		l3_qos_cfg_update(&r->alloc_enabled);
+		l3_qos_cfg_update(&hw_res->cdp_enabled);
 }
 
 /*
Signature · 1 line
2.30.0
author reviewer
#31 lore raw

PATCH v2 18/24 x86/resctrl: Pass configuration type to resctrl_arch_get_config()

In reply to James Morse

Once the configuration arrays are merged, the get_config() helper needs to be told whether the CODE, DATA or BOTH configuration is being retrieved.

Pass this information from the schema into resctrl_arch_get_config().

Nothing uses this yet, but it will later be used to map the closid to the index in the configuration array.

---
 arch/x86/kernel/cpu/resctrl/ctrlmondata.c |  5 ++--
 arch/x86/kernel/cpu/resctrl/monitor.c     |  2 +-
 arch/x86/kernel/cpu/resctrl/rdtgroup.c    | 35 +++++++++++++++--------
 include/linux/resctrl.h                   |  3 +-
 4 files changed, 29 insertions(+), 16 deletions(-)
arch/x86/kernel/cpu/resctrl/ctrlmondata.c +3 -2
index 1cd54402b02a..72a8cf52de47 100644
@@ -402,7 +402,7 @@ ssize_t rdtgroup_schemata_write(struct kernfs_open_file *of,
 }
 
 void resctrl_arch_get_config(struct rdt_resource *r, struct rdt_domain *d,
-			     u32 closid, u32 *value)
+			     u32 closid, enum resctrl_conf_type type, u32 *value)
 {
 	struct rdt_hw_domain *hw_dom = resctrl_to_arch_dom(d);
 
@@ -424,7 +424,8 @@ static void show_doms(struct seq_file *s, struct resctrl_schema *schema, int clo
 		if (sep)
 			seq_puts(s, ";");
 
-		resctrl_arch_get_config(r, dom, closid, &ctrl_val);
+		resctrl_arch_get_config(r, dom, closid, schema->conf_type,
+					&ctrl_val);
 		seq_printf(s, r->format_str, dom->id, max_data_width,
 			   ctrl_val);
 		sep = true;
arch/x86/kernel/cpu/resctrl/monitor.c +1 -1
index c4d146658b44..7cab888f0a9c 100644
@@ -442,7 +442,7 @@ static void update_mba_bw(struct rdtgroup *rgrp, struct rdt_domain *dom_mbm)
 	hw_dom_mba = resctrl_to_arch_dom(dom_mba);
 
 	cur_bw = pmbm_data->prev_bw;
-	resctrl_arch_get_config(r_mba, dom_mba, closid, &user_bw);
+	resctrl_arch_get_config(r_mba, dom_mba, closid, CDP_BOTH, &user_bw);
 	delta_bw = pmbm_data->delta_bw;
 	/*
 	 * resctrl_arch_get_config() chooses the mbps/ctrl value to return
arch/x86/kernel/cpu/resctrl/rdtgroup.c +23 -12
index 378cad0020da..4ef8f90da043 100644
@@ -923,7 +923,8 @@ static int rdt_bit_usage_show(struct kernfs_open_file *of,
 		for (i = 0; i < closids_supported(); i++) {
 			if (!closid_allocated(i))
 				continue;
-			resctrl_arch_get_config(r, dom, i, &ctrl_val);
+			resctrl_arch_get_config(r, dom, i, s->conf_type,
+						&ctrl_val);
 			mode = rdtgroup_mode_by_closid(i);
 			switch (mode) {
 			case RDT_MODE_SHAREABLE:
@@ -1099,6 +1100,7 @@ static int rdtgroup_mode_show(struct kernfs_open_file *of,
  *         Used to return the result.
  * @d_cdp: RDT domain that shares hardware with @d (RDT domain peer)
  *         Used to return the result.
+ * @peer_type: The CDP configuration type of the peer resource.
  *
  * RDT resources are managed independently and by extension the RDT domains
  * (RDT resource instances) are managed independently also. The Code and
@@ -1116,7 +1118,8 @@ static int rdtgroup_mode_show(struct kernfs_open_file *of,
  */
 static int rdt_cdp_peer_get(struct rdt_resource *r, struct rdt_domain *d,
 			    struct rdt_resource **r_cdp,
-			    struct rdt_domain **d_cdp)
+			    struct rdt_domain **d_cdp,
+			    enum resctrl_conf_type *peer_type)
 {
 	struct rdt_resource *_r_cdp = NULL;
 	struct rdt_domain *_d_cdp = NULL;
@@ -1125,15 +1128,19 @@ static int rdt_cdp_peer_get(struct rdt_resource *r, struct rdt_domain *d,
 	switch (r->rid) {
 	case RDT_RESOURCE_L3DATA:
 		_r_cdp = &rdt_resources_all[RDT_RESOURCE_L3CODE].resctrl;
+		*peer_type = CDP_CODE;
 		break;
 	case RDT_RESOURCE_L3CODE:
 		_r_cdp =  &rdt_resources_all[RDT_RESOURCE_L3DATA].resctrl;
+		*peer_type = CDP_DATA;
 		break;
 	case RDT_RESOURCE_L2DATA:
 		_r_cdp =  &rdt_resources_all[RDT_RESOURCE_L2CODE].resctrl;
+		*peer_type = CDP_CODE;
 		break;
 	case RDT_RESOURCE_L2CODE:
 		_r_cdp =  &rdt_resources_all[RDT_RESOURCE_L2DATA].resctrl;
+		*peer_type = CDP_DATA;
 		break;
 	default:
 		ret = -ENOENT;
@@ -1184,7 +1191,8 @@ static int rdt_cdp_peer_get(struct rdt_resource *r, struct rdt_domain *d,
  * Return: false if CBM does not overlap, true if it does.
  */
 static bool __rdtgroup_cbm_overlaps(struct rdt_resource *r, struct rdt_domain *d,
-				    unsigned long cbm, int closid, bool exclusive)
+				    unsigned long cbm, int closid,
+				    enum resctrl_conf_type type, bool exclusive)
 {
 	enum rdtgrp_mode mode;
 	unsigned long ctrl_b;
@@ -1199,7 +1207,7 @@ static bool __rdtgroup_cbm_overlaps(struct rdt_resource *r, struct rdt_domain *d
 
 	/* Check for overlap with other resource groups */
 	for (i = 0; i < closids_supported(); i++) {
-		resctrl_arch_get_config(r, d, i, (u32 *)&ctrl_b);
+		resctrl_arch_get_config(r, d, i, type, (u32 *)&ctrl_b);
 		mode = rdtgroup_mode_by_closid(i);
 		if (closid_allocated(i) && i != closid &&
 		    mode != RDT_MODE_PSEUDO_LOCKSETUP) {
@@ -1240,17 +1248,19 @@ static bool __rdtgroup_cbm_overlaps(struct rdt_resource *r, struct rdt_domain *d
 bool rdtgroup_cbm_overlaps(struct resctrl_schema *s, struct rdt_domain *d,
 			   unsigned long cbm, int closid, bool exclusive)
 {
+	enum resctrl_conf_type peer_type;
 	struct rdt_resource *r = s->res;
 	struct rdt_resource *r_cdp;
 	struct rdt_domain *d_cdp;
 
-	if (__rdtgroup_cbm_overlaps(r, d, cbm, closid, exclusive))
+	if (__rdtgroup_cbm_overlaps(r, d, cbm, closid, s->conf_type,
+				    exclusive))
 		return true;
 
-	if (rdt_cdp_peer_get(r, d, &r_cdp, &d_cdp) < 0)
+	if (rdt_cdp_peer_get(r, d, &r_cdp, &d_cdp, &peer_type) < 0)
 		return false;
 
-	return  __rdtgroup_cbm_overlaps(r_cdp, d_cdp, cbm, closid, exclusive);
+	return  __rdtgroup_cbm_overlaps(r_cdp, d_cdp, cbm, closid, peer_type, exclusive);
 }
 
 /**
@@ -1280,7 +1290,7 @@ static bool rdtgroup_mode_test_exclusive(struct rdtgroup *rdtgrp)
 			continue;
 		has_cache = true;
 		list_for_each_entry(d, &r->domains, list) {
-			resctrl_arch_get_config(r, d, closid, &ctrl);
+			resctrl_arch_get_config(r, d, closid, s->conf_type, &ctrl);
 			if (rdtgroup_cbm_overlaps(s, d, ctrl, closid, false)) {
 				rdt_last_cmd_puts("Schemata overlaps\n");
 				return false;
@@ -1454,7 +1464,7 @@ static int rdtgroup_size_show(struct kernfs_open_file *of,
 				size = 0;
 			} else {
 				resctrl_arch_get_config(r, d, rdtgrp->closid,
-							&ctrl);
+							schema->conf_type, &ctrl);
 				if (r->rid == RDT_RESOURCE_MBA)
 					size = ctrl;
 				else
@@ -2737,6 +2747,7 @@ static int __init_one_rdt_domain(struct rdt_domain *d, struct resctrl_schema *s,
 	enum resctrl_conf_type t = s->conf_type;
 	struct rdt_resource *r_cdp = NULL;
 	struct resctrl_staged_config *cfg;
+	enum resctrl_conf_type peer_type;
 	struct rdt_domain *d_cdp = NULL;
 	struct rdt_resource *r = s->res;
 	u32 used_b = 0, unused_b = 0;
@@ -2745,7 +2756,7 @@ static int __init_one_rdt_domain(struct rdt_domain *d, struct resctrl_schema *s,
 	u32 peer_ctl, ctrl_val;
 	int i;
 
-	rdt_cdp_peer_get(r, d, &r_cdp, &d_cdp);
+	rdt_cdp_peer_get(r, d, &r_cdp, &d_cdp, &peer_type);
 	cfg = &d->staged_config[t];
 	cfg->have_new_ctrl = false;
 	cfg->new_ctrl = r->cache.shareable_bits;
@@ -2766,10 +2777,10 @@ static int __init_one_rdt_domain(struct rdt_domain *d, struct resctrl_schema *s,
 			 * with an exclusive group.
 			 */
 			if (d_cdp)
-				resctrl_arch_get_config(r_cdp, d_cdp, i, &peer_ctl);
+				resctrl_arch_get_config(r_cdp, d_cdp, i, peer_type, &peer_ctl);
 			else
 				peer_ctl = 0;
-			resctrl_arch_get_config(r, d, i, &ctrl_val);
+			resctrl_arch_get_config(r, d, i, s->conf_type, &ctrl_val);
 			used_b |= ctrl_val | peer_ctl;
 			if (mode == RDT_MODE_SHAREABLE)
 				cfg->new_ctrl |= ctrl_val | peer_ctl;
include/linux/resctrl.h +2 -1
index dec27b03675e..250e96c073db 100644
@@ -192,6 +192,7 @@ struct resctrl_schema {
 u32 resctrl_arch_get_num_closid(struct rdt_resource *r);
 int resctrl_arch_update_domains(struct rdt_resource *r, u32 closid);
 void resctrl_arch_get_config(struct rdt_resource *r, struct rdt_domain *d,
-			     u32 closid, u32 *value);
+			     u32 closid, enum resctrl_conf_type type,
+			     u32 *value);
 
 #endif /* _RESCTRL_H */
Signature · 1 line
2.30.0
author reviewer
#32 lore raw

PATCH v2 19/24 x86/resctrl: Make ctrlval arrays the same size

In reply to James Morse

The CODE and DATA resources have their own ctrlval arrays which are half the size because num_closid was already adjusted.

Prior to having one ctrlval array for the resource, move the num_closid correction into resctrl, so that the ctrlval arrays are all the same size.

A short-lived quirk of this is that the caches are reset twice, once for CODE once for DATA.

---
 arch/x86/kernel/cpu/resctrl/core.c     | 10 +++++++++-
 arch/x86/kernel/cpu/resctrl/rdtgroup.c |  9 +++++++++
 2 files changed, 18 insertions(+), 1 deletion(-)
arch/x86/kernel/cpu/resctrl/core.c +9 -1
index 4db28ce114bd..f1d0d64e5d97 100644
@@ -363,7 +363,7 @@ static void rdt_get_cdp_config(int level, int type)
 	struct rdt_resource *r = &rdt_resources_all[type].resctrl;
 	struct rdt_hw_resource *hw_res = resctrl_to_arch_res(r);
 
-	hw_res->num_closid = hw_res_l->num_closid / 2;
+	hw_res->num_closid = hw_res_l->num_closid;
 	r->cache.cbm_len = r_l->cache.cbm_len;
 	r->default_ctrl = r_l->default_ctrl;
 	r->cache.shareable_bits = r_l->cache.shareable_bits;
@@ -549,6 +549,14 @@ static int domain_setup_ctrlval(struct rdt_resource *r, struct rdt_domain *d)
 
 	m.low = 0;
 	m.high = hw_res->num_closid;
+
+	/*
+	 * temporary: the array is full-size, but cat_wrmsr() still re-maps
+	 * the index.
+	 */
+	if (hw_res->conf_type != CDP_BOTH)
+		m.high /= 2;
+
 	hw_res->msr_update(d, &m, r);
 	return 0;
 }
arch/x86/kernel/cpu/resctrl/rdtgroup.c +9 -0
index 4ef8f90da043..a3e5c8b1b0cb 100644
@@ -2154,6 +2154,8 @@ static int schemata_list_create(void)
 		s->res = r;
 		s->conf_type = resctrl_to_arch_res(r)->conf_type;
 		s->num_closid = resctrl_arch_get_num_closid(r);
+		if (resctrl_arch_get_cdp_enabled(r->rid))
+			s->num_closid /= 2;
 
 		ret = snprintf(s->name, sizeof(s->name), r->name);
 		if (ret >= sizeof(s->name)) {
@@ -2366,6 +2368,13 @@ static int reset_all_ctrls(struct rdt_resource *r)
 	msr_param.low = 0;
 	msr_param.high = hw_res->num_closid;
 
+	/*
+	 * temporary: the array is full-sized, but cat_wrmsr() still re-maps
+	 * the index.
+	 */
+	if (hw_res->cdp_enabled)
+		msr_param.high /= 2;
+
 	/*
 	 * Disable resource control for this resource by setting all
 	 * CBMs in all domains to the maximum mask value. Pick one CPU
Signature · 1 line
2.30.0
author reviewer
#33 lore raw

PATCH v2 20/24 x86/resctrl: Apply offset correction when config is staged

In reply to James Morse

When resctrl comes to write the CAT MSR values, it applies an adjustment based on the style of the resource. CODE and DATA resources have their closid mapped into an odd/even range. Previously this decision was based on the resource.

Move this logic to apply_config() so that in future it can be based on the style of the configuration, not the resource.

This makes it possible to merge the resources.

Once the resources are merged, there may be multiple configurations to apply for a single closid and resource. resctrl_arch_update_domains() should supply the low and high indexes based on the changes it has made to the ctrl_val array as this allows the hardware to be updated once for a set of changes.

---
Changes since v1:
* Removing the patch that moved the closid to the staged config means the min/max and return from apply_config() appears here.

---
 arch/x86/kernel/cpu/resctrl/core.c        | 15 +-------
 arch/x86/kernel/cpu/resctrl/ctrlmondata.c | 44 ++++++++++++++++++-----
 arch/x86/kernel/cpu/resctrl/internal.h    |  4 +--
 arch/x86/kernel/cpu/resctrl/rdtgroup.c    |  7 ----
 4 files changed, 38 insertions(+), 32 deletions(-)
arch/x86/kernel/cpu/resctrl/core.c +1 -14
index f1d0d64e5d97..cb3186bc248b 100644
@@ -195,11 +195,6 @@ struct rdt_hw_resource rdt_resources_all[] = {
 	},
 };
 
-static unsigned int cbm_idx(struct rdt_resource *r, unsigned int closid)
-{
-	return closid * r->cache.cbm_idx_mult + r->cache.cbm_idx_offset;
-}
-
 /*
  * cache_alloc_hsw_probe() - Have to probe for Intel haswell server CPUs
  * as they do not have CPUID enumeration support for Cache allocation.
@@ -438,7 +433,7 @@ cat_wrmsr(struct rdt_domain *d, struct msr_param *m, struct rdt_resource *r)
 	struct rdt_hw_resource *hw_res = resctrl_to_arch_res(r);
 
 	for (i = m->low; i < m->high; i++)
-		wrmsrl(hw_res->msr_base + cbm_idx(r, i), hw_dom->ctrl_val[i]);
+		wrmsrl(hw_res->msr_base + i, hw_dom->ctrl_val[i]);
 }
 
 struct rdt_domain *get_domain_from_cpu(int cpu, struct rdt_resource *r)
@@ -549,14 +544,6 @@ static int domain_setup_ctrlval(struct rdt_resource *r, struct rdt_domain *d)
 
 	m.low = 0;
 	m.high = hw_res->num_closid;
-
-	/*
-	 * temporary: the array is full-size, but cat_wrmsr() still re-maps
-	 * the index.
-	 */
-	if (hw_res->conf_type != CDP_BOTH)
-		m.high /= 2;
-
 	hw_res->msr_update(d, &m, r);
 	return 0;
 }
arch/x86/kernel/cpu/resctrl/ctrlmondata.c +35 -9

91 lines not on this page · raw message

arch/x86/kernel/cpu/resctrl/internal.h +2 -2
index 9bafe3b32035..ac36249f43ba 100644
@@ -324,8 +324,8 @@ static inline struct rdt_hw_domain *resctrl_to_arch_dom(struct rdt_domain *r)
  */
 struct msr_param {
 	struct rdt_resource	*res;
-	int			low;
-	int			high;
+	u32			low;
+	u32			high;
 };
 
 static inline bool is_llc_occupancy_enabled(void)
arch/x86/kernel/cpu/resctrl/rdtgroup.c +0 -7
index a3e5c8b1b0cb..801bff59db06 100644
@@ -2368,13 +2368,6 @@ static int reset_all_ctrls(struct rdt_resource *r)
 	msr_param.low = 0;
 	msr_param.high = hw_res->num_closid;
 
-	/*
-	 * temporary: the array is full-sized, but cat_wrmsr() still re-maps
-	 * the index.
-	 */
-	if (hw_res->cdp_enabled)
-		msr_param.high /= 2;
-
 	/*
 	 * Disable resource control for this resource by setting all
 	 * CBMs in all domains to the maximum mask value. Pick one CPU
Signature · 1 line
2.30.0
author reviewer
#34 lore raw

PATCH v2 21/24 x86/resctrl: Calculate the index from the configuration type

In reply to James Morse

resctrl uses cbm_idx() to map a closid to an index in the configuration array. This is based on a multiplier and offset that are held in the resource.

To merge the resources, the resctrl arch code needs to calculate the index from something else, as there will only be one resource.

Decide based on the staged configuration type. This makes the static mult and offset parameters redundant.

---
 arch/x86/kernel/cpu/resctrl/core.c        | 12 ------------
 arch/x86/kernel/cpu/resctrl/ctrlmondata.c | 17 +++++++++++------
 include/linux/resctrl.h                   |  6 ------
 3 files changed, 11 insertions(+), 24 deletions(-)
arch/x86/kernel/cpu/resctrl/core.c +0 -12
index cb3186bc248b..8d5c1e9eefa1 100644
@@ -69,8 +69,6 @@ struct rdt_hw_resource rdt_resources_all[] = {
 			.cache_level		= 3,
 			.cache = {
 				.min_cbm_bits	= 1,
-				.cbm_idx_mult	= 1,
-				.cbm_idx_offset	= 0,
 			},
 			.domains		= domain_init(RDT_RESOURCE_L3),
 			.parse_ctrlval		= parse_cbm,
@@ -89,8 +87,6 @@ struct rdt_hw_resource rdt_resources_all[] = {
 			.cache_level		= 3,
 			.cache = {
 				.min_cbm_bits	= 1,
-				.cbm_idx_mult	= 2,
-				.cbm_idx_offset	= 0,
 			},
 			.domains		= domain_init(RDT_RESOURCE_L3DATA),
 			.parse_ctrlval		= parse_cbm,
@@ -109,8 +105,6 @@ struct rdt_hw_resource rdt_resources_all[] = {
 			.cache_level		= 3,
 			.cache = {
 				.min_cbm_bits	= 1,
-				.cbm_idx_mult	= 2,
-				.cbm_idx_offset	= 1,
 			},
 			.domains		= domain_init(RDT_RESOURCE_L3CODE),
 			.parse_ctrlval		= parse_cbm,
@@ -129,8 +123,6 @@ struct rdt_hw_resource rdt_resources_all[] = {
 			.cache_level		= 2,
 			.cache = {
 				.min_cbm_bits	= 1,
-				.cbm_idx_mult	= 1,
-				.cbm_idx_offset	= 0,
 			},
 			.domains		= domain_init(RDT_RESOURCE_L2),
 			.parse_ctrlval		= parse_cbm,
@@ -149,8 +141,6 @@ struct rdt_hw_resource rdt_resources_all[] = {
 			.cache_level		= 2,
 			.cache = {
 				.min_cbm_bits	= 1,
-				.cbm_idx_mult	= 2,
-				.cbm_idx_offset	= 0,
 			},
 			.domains		= domain_init(RDT_RESOURCE_L2DATA),
 			.parse_ctrlval		= parse_cbm,
@@ -169,8 +159,6 @@ struct rdt_hw_resource rdt_resources_all[] = {
 			.cache_level		= 2,
 			.cache = {
 				.min_cbm_bits	= 1,
-				.cbm_idx_mult	= 2,
-				.cbm_idx_offset	= 1,
 			},
 			.domains		= domain_init(RDT_RESOURCE_L2CODE),
 			.parse_ctrlval		= parse_cbm,
arch/x86/kernel/cpu/resctrl/ctrlmondata.c +11 -6
index 12a898d42689..50266b524222 100644
@@ -246,12 +246,17 @@ static int parse_line(char *line, struct resctrl_schema *s,
 	return -EINVAL;
 }
 
-static unsigned int cbm_idx(struct rdt_resource *r, unsigned int closid)
+static u32 get_config_index(u32 closid, enum resctrl_conf_type type)
 {
-	if (r->rid == RDT_RESOURCE_MBA)
+	switch (type) {
+	default:
+	case CDP_BOTH:
 		return closid;
-
-	return closid * r->cache.cbm_idx_mult + r->cache.cbm_idx_offset;
+	case CDP_CODE:
+		return (closid * 2) + 1;
+	case CDP_DATA:
+		return (closid * 2);
+	}
 }
 
 static bool apply_config(struct rdt_hw_domain *hw_dom,
@@ -297,7 +302,7 @@ int resctrl_arch_update_domains(struct rdt_resource *r, u32 closid)
 			if (!cfg->have_new_ctrl)
 				continue;
 
-			idx = cbm_idx(r, closid);
+			idx = get_config_index(closid, t);
 			if (!apply_config(hw_dom, cfg, idx, cpu_mask, mba_sc))
 				continue;
 
@@ -430,7 +435,7 @@ void resctrl_arch_get_config(struct rdt_resource *r, struct rdt_domain *d,
 			     u32 closid, enum resctrl_conf_type type, u32 *value)
 {
 	struct rdt_hw_domain *hw_dom = resctrl_to_arch_dom(d);
-	u32 idx = cbm_idx(r, closid);
+	u32 idx = get_config_index(closid, type);
 
 	if (!is_mba_sc(r))
 		*value = hw_dom->ctrl_val[idx];
include/linux/resctrl.h +0 -6
index 250e96c073db..abe280f8a76b 100644
@@ -68,10 +68,6 @@ struct rdt_domain {
  * struct resctrl_cache - Cache allocation related data
  * @cbm_len:		Length of the cache bit mask
  * @min_cbm_bits:	Minimum number of consecutive bits to be set
- * @cbm_idx_mult:	Multiplier of CBM index
- * @cbm_idx_offset:	Offset of CBM index. CBM index is computed by:
- *			closid * cbm_idx_multi + cbm_idx_offset
- *			in a cache bit mask
  * @shareable_bits:	Bitmask of shareable resource with other
  *			executing entities
  * @arch_has_sparse_bitmaps:	True if a bitmap like f00f is valid.
@@ -82,8 +78,6 @@ struct rdt_domain {
 struct resctrl_cache {
 	unsigned int	cbm_len;
 	unsigned int	min_cbm_bits;
-	unsigned int	cbm_idx_mult;	// TODO remove this
-	unsigned int	cbm_idx_offset; // TODO remove this
 	unsigned int	shareable_bits;
 	bool		arch_has_sparse_bitmaps;
 	bool		arch_has_empty_bitmaps;
Signature · 1 line
2.30.0
author reviewer
#35 lore raw

PATCH v2 22/24 x86/resctrl: Merge the ctrl_val arrays

In reply to James Morse

Now that the CODE/DATA resources don't use overlapping slots in the ctrl_val arrays, they can be merged. This allows the cdp_peer configuration to be read from either resource's domain, instead of searching for the matching flavour.

Doing this before merging the resources temporarily complicates allocating and freeing the ctrl_val arrays. Add a helper to allocate the ctrl_val array, that returns the value on the L2 or L3 resource if it already exists. This gets removed once the resources are merged, and there really is only one ctrl_val array.

---
Changes since v1:
* Added underscores to ctrlval when its not in a function name
* Removed temporary free_ctrlval_arrays() function.
* Reduced churn in domain_setup_ctrlval().

Doing this before the resources are merged allows any bugs introduced to be bisected to a smaller change, and keeps the rdt_cdp_peer_get() change separate.

---
 arch/x86/kernel/cpu/resctrl/core.c | 66 ++++++++++++++++++++++++++++--
 1 file changed, 62 insertions(+), 4 deletions(-)
arch/x86/kernel/cpu/resctrl/core.c +62 -4
index 8d5c1e9eefa1..5021a726e87d 100644
@@ -509,6 +509,58 @@ void setup_default_ctrlval(struct rdt_resource *r, u32 *dc, u32 *dm)
 	}
 }
 
+static u32 *alloc_ctrlval_array(struct rdt_resource *r, struct rdt_domain *d,
+				bool mba_sc)
+{
+	/* these are for the underlying hardware, they may not match r/d */
+	struct rdt_domain *underlying_domain;
+	struct rdt_hw_resource *hw_res;
+	struct rdt_hw_domain *hw_dom;
+	bool remapped;
+
+	switch (r->rid) {
+	case RDT_RESOURCE_L3DATA:
+	case RDT_RESOURCE_L3CODE:
+		hw_res = &rdt_resources_all[RDT_RESOURCE_L3];
+		remapped = true;
+		break;
+	case RDT_RESOURCE_L2DATA:
+	case RDT_RESOURCE_L2CODE:
+		hw_res = &rdt_resources_all[RDT_RESOURCE_L2];
+		remapped = true;
+		break;
+	default:
+		hw_res = resctrl_to_arch_res(r);
+		remapped = false;
+	}
+
+
+	/*
+	 * If we changed the resource, we need to search for the underlying
+	 * domain. Doing this for all resources would make it tricky to add the
+	 * first resource, as domains aren't added to a resource list until
+	 * after the ctrlval arrays have been allocated.
+	 */
+	if (remapped)
+		underlying_domain = rdt_find_domain(&hw_res->resctrl, d->id,
+						    NULL);
+	else
+		underlying_domain = d;
+	hw_dom = resctrl_to_arch_dom(underlying_domain);
+
+	if (mba_sc) {
+		if (hw_dom->mbps_val)
+			return hw_dom->mbps_val;
+		return kmalloc_array(hw_res->num_closid,
+				     sizeof(*hw_dom->mbps_val), GFP_KERNEL);
+	} else {
+		if (hw_dom->ctrl_val)
+			return hw_dom->ctrl_val;
+		return kmalloc_array(hw_res->num_closid,
+				     sizeof(*hw_dom->ctrl_val), GFP_KERNEL);
+	}
+}
+
 static int domain_setup_ctrlval(struct rdt_resource *r, struct rdt_domain *d)
 {
 	struct rdt_hw_resource *hw_res = resctrl_to_arch_res(r);
@@ -516,11 +568,11 @@ static int domain_setup_ctrlval(struct rdt_resource *r, struct rdt_domain *d)
 	struct msr_param m;
 	u32 *dc, *dm;
 
-	dc = kmalloc_array(hw_res->num_closid, sizeof(*hw_dom->ctrl_val), GFP_KERNEL);
+	dc = alloc_ctrlval_array(r, d, false);
 	if (!dc)
 		return -ENOMEM;
 
-	dm = kmalloc_array(hw_res->num_closid, sizeof(*hw_dom->mbps_val), GFP_KERNEL);
+	dm = alloc_ctrlval_array(r, d, true);
 	if (!dm) {
 		kfree(dc);
 		return -ENOMEM;
@@ -679,8 +731,14 @@ static void domain_remove_cpu(int cpu, struct rdt_resource *r)
 		if (d->plr)
 			d->plr->d = NULL;
 
-		kfree(hw_dom->ctrl_val);
-		kfree(hw_dom->mbps_val);
+		/* temporary: these four don't have a unique ctrlval array */
+		if ((r->rid != RDT_RESOURCE_L3CODE) &&
+		    (r->rid != RDT_RESOURCE_L3DATA) &&
+		    (r->rid != RDT_RESOURCE_L2CODE) &&
+		    (r->rid != RDT_RESOURCE_L2DATA)) {
+			kfree(hw_dom->ctrl_val);
+			kfree(hw_dom->mbps_val);
+		}
 		bitmap_free(d->rmid_busy_llc);
 		kfree(d->mbm_total);
 		kfree(d->mbm_local);
Signature · 1 line
2.30.0
author reviewer
#36 lore raw

PATCH v2 23/24 x86/resctrl: Remove rdt_cdp_peer_get()

In reply to James Morse

Now that the configuration can be read from either CDP peer, rdt_cdp_peer_get() is not needed to to map the resource and search for the corresponding domain.

As __rdtgroup_cbm_overlaps() takes the configuration type from the schema, this can be replaced with a second call for the other configuration type if cdp is enabled.

Changes since v1:
* Expanded commit mesasge.

---
 arch/x86/kernel/cpu/resctrl/rdtgroup.c | 99 ++++----------------------
 1 file changed, 14 insertions(+), 85 deletions(-)
arch/x86/kernel/cpu/resctrl/rdtgroup.c +14 -85
index 801bff59db06..36e2905f4da6 100644
@@ -1092,82 +1092,17 @@ static int rdtgroup_mode_show(struct kernfs_open_file *of,
 	return 0;
 }
 
-/**
- * rdt_cdp_peer_get - Retrieve CDP peer if it exists
- * @r: RDT resource to which RDT domain @d belongs
- * @d: Cache instance for which a CDP peer is requested
- * @r_cdp: RDT resource that shares hardware with @r (RDT resource peer)
- *         Used to return the result.
- * @d_cdp: RDT domain that shares hardware with @d (RDT domain peer)
- *         Used to return the result.
- * @peer_type: The CDP configuration type of the peer resource.
- *
- * RDT resources are managed independently and by extension the RDT domains
- * (RDT resource instances) are managed independently also. The Code and
- * Data Prioritization (CDP) RDT resources, while managed independently,
- * could refer to the same underlying hardware. For example,
- * RDT_RESOURCE_L2CODE and RDT_RESOURCE_L2DATA both refer to the L2 cache.
- *
- * When provided with an RDT resource @r and an instance of that RDT
- * resource @d rdt_cdp_peer_get() will return if there is a peer RDT
- * resource and the exact instance that shares the same hardware.
- *
- * Return: 0 if a CDP peer was found, <0 on error or if no CDP peer exists.
- *         If a CDP peer was found, @r_cdp will point to the peer RDT resource
- *         and @d_cdp will point to the peer RDT domain.
- */
-static int rdt_cdp_peer_get(struct rdt_resource *r, struct rdt_domain *d,
-			    struct rdt_resource **r_cdp,
-			    struct rdt_domain **d_cdp,
-			    enum resctrl_conf_type *peer_type)
+static enum resctrl_conf_type resctrl_peer_type(enum resctrl_conf_type my_type)
 {
-	struct rdt_resource *_r_cdp = NULL;
-	struct rdt_domain *_d_cdp = NULL;
-	int ret = 0;
-
-	switch (r->rid) {
-	case RDT_RESOURCE_L3DATA:
-		_r_cdp = &rdt_resources_all[RDT_RESOURCE_L3CODE].resctrl;
-		*peer_type = CDP_CODE;
-		break;
-	case RDT_RESOURCE_L3CODE:
-		_r_cdp =  &rdt_resources_all[RDT_RESOURCE_L3DATA].resctrl;
-		*peer_type = CDP_DATA;
-		break;
-	case RDT_RESOURCE_L2DATA:
-		_r_cdp =  &rdt_resources_all[RDT_RESOURCE_L2CODE].resctrl;
-		*peer_type = CDP_CODE;
-		break;
-	case RDT_RESOURCE_L2CODE:
-		_r_cdp =  &rdt_resources_all[RDT_RESOURCE_L2DATA].resctrl;
-		*peer_type = CDP_DATA;
-		break;
+	switch (my_type) {
+	case CDP_CODE:
+		return CDP_DATA;
+	case CDP_DATA:
+		return CDP_CODE;
 	default:
-		ret = -ENOENT;
-		goto out;
-	}
-
-	/*
-	 * When a new CPU comes online and CDP is enabled then the new
-	 * RDT domains (if any) associated with both CDP RDT resources
-	 * are added in the same CPU online routine while the
-	 * rdtgroup_mutex is held. It should thus not happen for one
-	 * RDT domain to exist and be associated with its RDT CDP
-	 * resource but there is no RDT domain associated with the
-	 * peer RDT CDP resource. Hence the WARN.
-	 */
-	_d_cdp = rdt_find_domain(_r_cdp, d->id, NULL);
-	if (WARN_ON(IS_ERR_OR_NULL(_d_cdp))) {
-		_r_cdp = NULL;
-		_d_cdp = NULL;
-		ret = -EINVAL;
+	case CDP_BOTH:
+		return CDP_BOTH;
 	}
-
-out:
-	*r_cdp = _r_cdp;
-	*d_cdp = _d_cdp;
-
-	return ret;
 }
 
 /**
@@ -1248,19 +1183,16 @@ static bool __rdtgroup_cbm_overlaps(struct rdt_resource *r, struct rdt_domain *d
 bool rdtgroup_cbm_overlaps(struct resctrl_schema *s, struct rdt_domain *d,
 			   unsigned long cbm, int closid, bool exclusive)
 {
-	enum resctrl_conf_type peer_type;
+	enum resctrl_conf_type peer_type = resctrl_peer_type(s->conf_type);
 	struct rdt_resource *r = s->res;
-	struct rdt_resource *r_cdp;
-	struct rdt_domain *d_cdp;
 
 	if (__rdtgroup_cbm_overlaps(r, d, cbm, closid, s->conf_type,
 				    exclusive))
 		return true;
 
-	if (rdt_cdp_peer_get(r, d, &r_cdp, &d_cdp, &peer_type) < 0)
+	if (!resctrl_arch_get_cdp_enabled(r->rid))
 		return false;
-
-	return  __rdtgroup_cbm_overlaps(r_cdp, d_cdp, cbm, closid, peer_type, exclusive);
+	return  __rdtgroup_cbm_overlaps(r, d, cbm, closid, peer_type, exclusive);
 }
 
 /**
@@ -2746,11 +2678,9 @@ static u32 cbm_ensure_valid(u32 _val, struct rdt_resource *r)
 static int __init_one_rdt_domain(struct rdt_domain *d, struct resctrl_schema *s,
 				 u32 closid)
 {
+	enum resctrl_conf_type peer_type = resctrl_peer_type(s->conf_type);
 	enum resctrl_conf_type t = s->conf_type;
-	struct rdt_resource *r_cdp = NULL;
 	struct resctrl_staged_config *cfg;
-	enum resctrl_conf_type peer_type;
-	struct rdt_domain *d_cdp = NULL;
 	struct rdt_resource *r = s->res;
 	u32 used_b = 0, unused_b = 0;
 	unsigned long tmp_cbm;
@@ -2758,7 +2688,6 @@ static int __init_one_rdt_domain(struct rdt_domain *d, struct resctrl_schema *s,
 	u32 peer_ctl, ctrl_val;
 	int i;
 
-	rdt_cdp_peer_get(r, d, &r_cdp, &d_cdp, &peer_type);
 	cfg = &d->staged_config[t];
 	cfg->have_new_ctrl = false;
 	cfg->new_ctrl = r->cache.shareable_bits;
@@ -2778,8 +2707,8 @@ static int __init_one_rdt_domain(struct rdt_domain *d, struct resctrl_schema *s,
 			 * usage to ensure there is no overlap
 			 * with an exclusive group.
 			 */
-			if (d_cdp)
-				resctrl_arch_get_config(r_cdp, d_cdp, i, peer_type, &peer_ctl);
+			if (resctrl_arch_get_cdp_enabled(r->rid))
+				resctrl_arch_get_config(r, d, i, peer_type, &peer_ctl);
 			else
 				peer_ctl = 0;
 			resctrl_arch_get_config(r, d, i, s->conf_type, &ctrl_val);
Signature · 1 line
2.30.0
author reviewer
#37 lore raw

PATCH v2 24/24 x86/resctrl: Merge the CDP resources

In reply to James Morse

Now that resctrl uses the schema's configuration type is the source of CODE/DATA configuration styles, and there is only one configuration array between the three views of the resource, remove the CODE and DATA aliases.

This allows the alloc_ctrlval_array() and complications around free()ing the ctrl_val arrays to be removed.

To continue providing the CDP user-interface, the resctrl filesystem code creates two schema for one resource when CDP is enabled, and generates the names itself. This ensures the user interface is the same when another architecture emulates CDPs behaviour.

---
Changes since v1:
* rdt_get_cdp_config() is kept for its comment.

---
 arch/x86/kernel/cpu/resctrl/core.c     | 174 ++-----------------------
 arch/x86/kernel/cpu/resctrl/internal.h |   4 -
 arch/x86/kernel/cpu/resctrl/rdtgroup.c | 122 ++++++++---------
 3 files changed, 75 insertions(+), 225 deletions(-)
arch/x86/kernel/cpu/resctrl/core.c +11 -163
index 5021a726e87d..7c20d0469b3a 100644
@@ -78,42 +78,6 @@ struct rdt_hw_resource rdt_resources_all[] = {
 		.msr_base		= MSR_IA32_L3_CBM_BASE,
 		.msr_update		= cat_wrmsr,
 	},
-	[RDT_RESOURCE_L3DATA] =
-	{
-		.conf_type			= CDP_DATA,
-		.resctrl = {
-			.rid			= RDT_RESOURCE_L3DATA,
-			.name			= "L3DATA",
-			.cache_level		= 3,
-			.cache = {
-				.min_cbm_bits	= 1,
-			},
-			.domains		= domain_init(RDT_RESOURCE_L3DATA),
-			.parse_ctrlval		= parse_cbm,
-			.format_str		= "%d=%0*x",
-			.fflags			= RFTYPE_RES_CACHE,
-		},
-		.msr_base		= MSR_IA32_L3_CBM_BASE,
-		.msr_update		= cat_wrmsr,
-	},
-	[RDT_RESOURCE_L3CODE] =
-	{
-		.conf_type			= CDP_CODE,
-		.resctrl = {
-			.rid			= RDT_RESOURCE_L3CODE,
-			.name			= "L3CODE",
-			.cache_level		= 3,
-			.cache = {
-				.min_cbm_bits	= 1,
-			},
-			.domains		= domain_init(RDT_RESOURCE_L3CODE),
-			.parse_ctrlval		= parse_cbm,
-			.format_str		= "%d=%0*x",
-			.fflags			= RFTYPE_RES_CACHE,
-		},
-		.msr_base		= MSR_IA32_L3_CBM_BASE,
-		.msr_update		= cat_wrmsr,
-	},
 	[RDT_RESOURCE_L2] =
 	{
 		.conf_type			= CDP_BOTH,
@@ -132,42 +96,6 @@ struct rdt_hw_resource rdt_resources_all[] = {
 		.msr_base		= MSR_IA32_L2_CBM_BASE,
 		.msr_update		= cat_wrmsr,
 	},
-	[RDT_RESOURCE_L2DATA] =
-	{
-		.conf_type			= CDP_DATA,
-		.resctrl = {
-			.rid			= RDT_RESOURCE_L2DATA,
-			.name			= "L2DATA",
-			.cache_level		= 2,
-			.cache = {
-				.min_cbm_bits	= 1,
-			},
-			.domains		= domain_init(RDT_RESOURCE_L2DATA),
-			.parse_ctrlval		= parse_cbm,
-			.format_str		= "%d=%0*x",
-			.fflags			= RFTYPE_RES_CACHE,
-		},
-		.msr_base		= MSR_IA32_L2_CBM_BASE,
-		.msr_update		= cat_wrmsr,
-	},
-	[RDT_RESOURCE_L2CODE] =
-	{
-		.conf_type			= CDP_CODE,
-		.resctrl = {
-			.rid			= RDT_RESOURCE_L2CODE,
-			.name			= "L2CODE",
-			.cache_level		= 2,
-			.cache = {
-				.min_cbm_bits	= 1,
-			},
-			.domains		= domain_init(RDT_RESOURCE_L2CODE),
-			.parse_ctrlval		= parse_cbm,
-			.format_str		= "%d=%0*x",
-			.fflags			= RFTYPE_RES_CACHE,
-		},
-		.msr_base		= MSR_IA32_L2_CBM_BASE,
-		.msr_update		= cat_wrmsr,
-	},
 	[RDT_RESOURCE_MBA] =
 	{
 		.conf_type			= CDP_BOTH,
@@ -339,40 +267,24 @@ static void rdt_get_cache_alloc_cfg(int idx, struct rdt_resource *r)
 	r->alloc_enabled = true;
 }
 
-static void rdt_get_cdp_config(int level, int type)
+static void rdt_get_cdp_config(int level)
 {
-	struct rdt_resource *r_l = &rdt_resources_all[level].resctrl;
-	struct rdt_hw_resource *hw_res_l = resctrl_to_arch_res(r_l);
-	struct rdt_resource *r = &rdt_resources_all[type].resctrl;
-	struct rdt_hw_resource *hw_res = resctrl_to_arch_res(r);
-
-	hw_res->num_closid = hw_res_l->num_closid;
-	r->cache.cbm_len = r_l->cache.cbm_len;
-	r->default_ctrl = r_l->default_ctrl;
-	r->cache.shareable_bits = r_l->cache.shareable_bits;
-	r->data_width = (r->cache.cbm_len + 3) / 4;
-	r->alloc_capable = true;
 	/*
 	 * By default, CDP is disabled. CDP can be enabled by mount parameter
 	 * "cdp" during resctrl file system mount time.
 	 */
-	r->alloc_enabled = false;
 	rdt_resources_all[level].cdp_enabled = false;
-	rdt_resources_all[type].cdp_enabled = false;
 	rdt_resources_all[level].cdp_capable = true;
-	rdt_resources_all[type].cdp_capable = true;
 }
 
 static void rdt_get_cdp_l3_config(void)
 {
-	rdt_get_cdp_config(RDT_RESOURCE_L3, RDT_RESOURCE_L3DATA);
-	rdt_get_cdp_config(RDT_RESOURCE_L3, RDT_RESOURCE_L3CODE);
+	rdt_get_cdp_config(RDT_RESOURCE_L3);
 }
 
 static void rdt_get_cdp_l2_config(void)
 {
-	rdt_get_cdp_config(RDT_RESOURCE_L2, RDT_RESOURCE_L2DATA);
-	rdt_get_cdp_config(RDT_RESOURCE_L2, RDT_RESOURCE_L2CODE);
+	rdt_get_cdp_config(RDT_RESOURCE_L2);
 }
 
 static void
@@ -509,58 +421,6 @@ void setup_default_ctrlval(struct rdt_resource *r, u32 *dc, u32 *dm)
 	}
 }
 
-static u32 *alloc_ctrlval_array(struct rdt_resource *r, struct rdt_domain *d,
-				bool mba_sc)
-{
-	/* these are for the underlying hardware, they may not match r/d */
-	struct rdt_domain *underlying_domain;
-	struct rdt_hw_resource *hw_res;
-	struct rdt_hw_domain *hw_dom;
-	bool remapped;
-
-	switch (r->rid) {
-	case RDT_RESOURCE_L3DATA:
-	case RDT_RESOURCE_L3CODE:
-		hw_res = &rdt_resources_all[RDT_RESOURCE_L3];
-		remapped = true;
-		break;
-	case RDT_RESOURCE_L2DATA:
-	case RDT_RESOURCE_L2CODE:
-		hw_res = &rdt_resources_all[RDT_RESOURCE_L2];
-		remapped = true;
-		break;
-	default:
-		hw_res = resctrl_to_arch_res(r);
-		remapped = false;
-	}
-
-
-	/*
-	 * If we changed the resource, we need to search for the underlying
-	 * domain. Doing this for all resources would make it tricky to add the
-	 * first resource, as domains aren't added to a resource list until
-	 * after the ctrlval arrays have been allocated.
-	 */
-	if (remapped)
-		underlying_domain = rdt_find_domain(&hw_res->resctrl, d->id,
-						    NULL);
-	else
-		underlying_domain = d;
-	hw_dom = resctrl_to_arch_dom(underlying_domain);
-
-	if (mba_sc) {
-		if (hw_dom->mbps_val)
-			return hw_dom->mbps_val;
-		return kmalloc_array(hw_res->num_closid,
-				     sizeof(*hw_dom->mbps_val), GFP_KERNEL);
-	} else {
-		if (hw_dom->ctrl_val)
-			return hw_dom->ctrl_val;
-		return kmalloc_array(hw_res->num_closid,
-				     sizeof(*hw_dom->ctrl_val), GFP_KERNEL);
-	}
-}
-
 static int domain_setup_ctrlval(struct rdt_resource *r, struct rdt_domain *d)
 {
 	struct rdt_hw_resource *hw_res = resctrl_to_arch_res(r);
@@ -568,11 +428,13 @@ static int domain_setup_ctrlval(struct rdt_resource *r, struct rdt_domain *d)
 	struct msr_param m;
 	u32 *dc, *dm;
 
-	dc = alloc_ctrlval_array(r, d, false);
+	dc = kmalloc_array(hw_res->num_closid, sizeof(*hw_dom->ctrl_val),
+			   GFP_KERNEL);
 	if (!dc)
 		return -ENOMEM;
 
-	dm = alloc_ctrlval_array(r, d, true);
+	dm = kmalloc_array(hw_res->num_closid, sizeof(*hw_dom->mbps_val),
+			   GFP_KERNEL);
 	if (!dm) {
 		kfree(dc);
 		return -ENOMEM;
@@ -731,14 +593,8 @@ static void domain_remove_cpu(int cpu, struct rdt_resource *r)
 		if (d->plr)
 			d->plr->d = NULL;
 
-		/* temporary: these four don't have a unique ctrlval array */
-		if ((r->rid != RDT_RESOURCE_L3CODE) &&
-		    (r->rid != RDT_RESOURCE_L3DATA) &&
-		    (r->rid != RDT_RESOURCE_L2CODE) &&
-		    (r->rid != RDT_RESOURCE_L2DATA)) {
-			kfree(hw_dom->ctrl_val);
-			kfree(hw_dom->mbps_val);
-		}
+		kfree(hw_dom->ctrl_val);
+		kfree(hw_dom->mbps_val);
 		bitmap_free(d->rmid_busy_llc);
 		kfree(d->mbm_total);
 		kfree(d->mbm_local);
@@ -1011,11 +867,7 @@ static __init void rdt_init_res_defs_intel(void)
 		hw_res = resctrl_to_arch_res(r);
 
 		if (r->rid == RDT_RESOURCE_L3 ||
-		    r->rid == RDT_RESOURCE_L3DATA ||
-		    r->rid == RDT_RESOURCE_L3CODE ||
-		    r->rid == RDT_RESOURCE_L2 ||
-		    r->rid == RDT_RESOURCE_L2DATA ||
-		    r->rid == RDT_RESOURCE_L2CODE) {
+		    r->rid == RDT_RESOURCE_L2) {
 			r->cache.arch_has_sparse_bitmaps = false;
 			r->cache.arch_has_empty_bitmaps = false;
 			r->cache.arch_has_per_cpu_cfg = false;
@@ -1035,11 +887,7 @@ static __init void rdt_init_res_defs_amd(void)
 		hw_res = resctrl_to_arch_res(r);
 
 		if (r->rid == RDT_RESOURCE_L3 ||
-		    r->rid == RDT_RESOURCE_L3DATA ||
-		    r->rid == RDT_RESOURCE_L3CODE ||
-		    r->rid == RDT_RESOURCE_L2 ||
-		    r->rid == RDT_RESOURCE_L2DATA ||
-		    r->rid == RDT_RESOURCE_L2CODE) {
+		    r->rid == RDT_RESOURCE_L2) {
 			r->cache.arch_has_sparse_bitmaps = true;
 			r->cache.arch_has_empty_bitmaps = true;
 			r->cache.arch_has_per_cpu_cfg = true;
arch/x86/kernel/cpu/resctrl/internal.h +0 -4
index ac36249f43ba..e06bc5ccf6cd 100644
@@ -402,11 +402,7 @@ extern struct dentry *debugfs_resctrl;
 
 enum resctrl_res_level {
 	RDT_RESOURCE_L3,
-	RDT_RESOURCE_L3DATA,
-	RDT_RESOURCE_L3CODE,
 	RDT_RESOURCE_L2,
-	RDT_RESOURCE_L2DATA,
-	RDT_RESOURCE_L2CODE,
 	RDT_RESOURCE_MBA,
 
 	/* Must be the last */
arch/x86/kernel/cpu/resctrl/rdtgroup.c +64 -58

182 lines not on this page · raw message

Signature · 1 line
2.30.0
reviewer
#38 lore raw
In reply to James Morse

Hi James,
Thanks for the patches. Few comments below.

Quoted 38 lines · James Morse

Hi folks,

Thanks to Reinette and Jamie for the comments on v1. Major changes in v2 are to keep the closid in resctrl_arch_update_domains(), eliminating one patch, splitting another that was making two sorts of change, and to re-order the first few patches. See each patches changelog for more.
----

This series re-folds the resctrl code so the CDP resources (L3CODE et al) behaviour is all contained in the filesystem parts, with a minimum amount of arch specific code.

Arm have some CPU support for dividing caches into portions, and applying bandwidth limits at various points in the SoC. The collective term for these features is MPAM: Memory Partitioning and Monitoring.

MPAM is similar enough to Intel RDT, that it should use the defacto linux
interface: resctrl. This filesystem currently lives under arch/x86, and is tightly coupled to the architecture.
Ultimately, my plan is to split the existing resctrl code up to have an arch<->fs abstraction, then move all the bits out to fs/resctrl. From there MPAM can be wired up.

x86 might have two resources with cache controls, (L2 and L3) but has extra copies for CDP: L{2,3}{CODE,DATA}, which are marked as enabled if CDP is enabled for the corresponding cache.

MPAM has an equivalent feature to CDP, but its a property of the CPU, not the cache. Resctrl needs to have x86's odd/even behaviour, as that its the ABI, but this isn't how the MPAM hardware works. It is entirely possible that an in-kernel user of MPAM would not be using CDP, whereas resctrl is.
Pretending L3CODE and L3DATA are entirely separate resources is a neat trick, but doing this is specific to x86.
Doing this leaves the arch code in control of various parts of the filesystem ABI: the resources names, and the way the schemata are parsed. Allowing this stuff to vary between architectures is bad for user space.

Quoted 7 lines

This series collapses the CODE/DATA resources, moving all the user-visible resctrl ABI into what becomes the filesystem code. CDP becomes the type of configuration being applied to a cache. This is done by adding a struct resctrl_schema to the parts of resctrl that will move to fs. This holds the arch-code resource that is in use for this schema, along with other properties like the name, and whether the configuration being applied is CODE/DATA/BOTH.

I applied your patches on my AMD box. Seeing some difference in the behavior.

Before these patches.

# dmesg |grep -i resctrl
[   13.076973] resctrl: L3 allocation detected
[   13.087835] resctrl: L3DATA allocation detected
[   13.092886] resctrl: L3CODE allocation detected
[   13.097936] resctrl: MB allocation detected
[   13.102599] resctrl: L3 monitoring detected

After the patches.

# dmesg |grep -i resctrl
[   13.076973] resctrl: L3 allocation detected
[   13.097936] resctrl: MB allocation detected
[   13.102599] resctrl: L3 monitoring detected

You can see that L3DATA and L3CODE disappeared. I think we should keep the behavior same for x86(at least).

I am still not clear why we needed resctrl_conf_type

enum resctrl_conf_type {
CDP_BOTH,
CDP_CODE,
CDP_DATA,
};

Right now, I see all the resources are initialized as CDP_BOTH.

[RDT_RESOURCE_L3] =
{
.conf_type = CDP_BOTH,
[RDT_RESOURCE_L2] =
{
.conf_type = CDP_BOTH,
[RDT_RESOURCE_MBA] =
{
.conf_type = CDP_BOTH,

If all the resources are CDP_BOTH, then why we need separate CDP_CODE and CDP_DATA? Are these going to be different for ARM?

Also initializing RDT_RESOURCE_MBA as CDP_BOTH does not seem right. I dont think there will CDP support in MBA in future.

Quoted 69 lines

This lets us fold the extra resources out of the arch code so that they don't need to be duplicated if the equivalent feature to CDP is missing, or implemented in a different way.

The first two patches split the resource and domain structs to have an arch specific 'hw' portion, and the rest that is visible to resctrl. Future series massage the resctrl code so there are no accesses to 'hw' structures in the parts of resctrl that will move to fs, providing helpers where necessary.

This series adds temporary scaffolding, which it removes a few patches later. This is to allow things like the ctrlval arrays and resources to be merged separately, which should make is easier to bisect. These things are marked temporary, and should all be gone by the end of the series.

This series is a little rough around the monitors, would a fake struct resctrl_schema for the monitors simplify things, or be a source of bugs?

This series is based on v5.12-rc2, and can be retrieved from:
git://git.kernel.org/pub/scm/linux/kernel/git/morse/linux.git mpam/resctrl_merge_cdp/v2

v1 was posted here:
https://lore.kernel.org/lkml/[email protected]/

Parts were previously posted as an RFC here:
https://lore.kernel.org/lkml/[email protected]/

Thanks,

James Morse (24):
x86/resctrl: Split struct rdt_resource
x86/resctrl: Split struct rdt_domain
x86/resctrl: Add a separate schema list for resctrl x86/resctrl: Pass the schema in info dir's private pointer x86/resctrl: Label the resources with their configuration type x86/resctrl: Walk the resctrl schema list instead of an arch list x86/resctrl: Store the effective num_closid in the schema x86/resctrl: Add resctrl_arch_get_num_closid() x86/resctrl: Pass the schema to resctrl filesystem functions x86/resctrl: Swizzle rdt_resource and resctrl_schema in
pseudo_lock_region
x86/resctrl: Move the schemata names into struct resctrl_schema x86/resctrl: Group staged configuration into a separate struct x86/resctrl: Allow different CODE/DATA configurations to be staged x86/resctrl: Rename update_domains() resctrl_arch_update_domains() x86/resctrl: Add a helper to read a closid's configuration x86/resctrl: Add a helper to read/set the CDP configuration x86/resctrl: Use cdp_enabled in rdt_domain_reconfigure_cdp() x86/resctrl: Pass configuration type to resctrl_arch_get_config() x86/resctrl: Make ctrlval arrays the same size x86/resctrl: Apply offset correction when config is staged x86/resctrl: Calculate the index from the configuration type x86/resctrl: Merge the ctrl_val arrays
x86/resctrl: Remove rdt_cdp_peer_get()
x86/resctrl: Merge the CDP resources

 arch/x86/kernel/cpu/resctrl/core.c        | 272 +++++--------
 arch/x86/kernel/cpu/resctrl/ctrlmondata.c | 164 +++++---
 arch/x86/kernel/cpu/resctrl/internal.h    | 218 +++--------
 arch/x86/kernel/cpu/resctrl/monitor.c     |  44 ++-
 arch/x86/kernel/cpu/resctrl/pseudo_lock.c |  12 +-
 arch/x86/kernel/cpu/resctrl/rdtgroup.c    | 447 ++++++++++++----------
 include/linux/resctrl.h                   | 176 +++++++++
 7 files changed, 741 insertions(+), 592 deletions(-)
author reviewer
#39 lore raw

Hi Babu,

Quoted 40 lines · Babu Moger
Quoted 30 lines · James Morse

This series re-folds the resctrl code so the CDP resources (L3CODE et al) behaviour is all contained in the filesystem parts, with a minimum amount of arch specific code.

Arm have some CPU support for dividing caches into portions, and applying bandwidth limits at various points in the SoC. The collective term for these features is MPAM: Memory Partitioning and Monitoring.

MPAM is similar enough to Intel RDT, that it should use the defacto linux
interface: resctrl. This filesystem currently lives under arch/x86, and is tightly coupled to the architecture.
Ultimately, my plan is to split the existing resctrl code up to have an arch<->fs abstraction, then move all the bits out to fs/resctrl. From there MPAM can be wired up.

x86 might have two resources with cache controls, (L2 and L3) but has extra copies for CDP: L{2,3}{CODE,DATA}, which are marked as enabled if CDP is enabled for the corresponding cache.

MPAM has an equivalent feature to CDP, but its a property of the CPU, not the cache. Resctrl needs to have x86's odd/even behaviour, as that its the ABI, but this isn't how the MPAM hardware works. It is entirely possible that an in-kernel user of MPAM would not be using CDP, whereas resctrl is.
Pretending L3CODE and L3DATA are entirely separate resources is a neat trick, but doing this is specific to x86.
Doing this leaves the arch code in control of various parts of the filesystem ABI: the resources names, and the way the schemata are parsed. Allowing this stuff to vary between architectures is bad for user space.

Quoted 7 lines

This series collapses the CODE/DATA resources, moving all the user-visible resctrl ABI into what becomes the filesystem code. CDP becomes the type of configuration being applied to a cache. This is done by adding a struct resctrl_schema to the parts of resctrl that will move to fs. This holds the arch-code resource that is in use for this schema, along with other properties like the name, and whether the configuration being applied is CODE/DATA/BOTH.

Quoted 1 line

I applied your patches on my AMD box.

Great! Thanks for taking a look,

Quoted 1 line

Seeing some difference in the behavior.

Ooer,

Quoted 12 lines

Before these patches.

# dmesg |grep -i resctrl
[   13.076973] resctrl: L3 allocation detected
[   13.087835] resctrl: L3DATA allocation detected
[   13.092886] resctrl: L3CODE allocation detected
[   13.097936] resctrl: MB allocation detected
[   13.102599] resctrl: L3 monitoring detected

After the patches.

Quoted 7 lines
# dmesg |grep -i resctrl
[   13.076973] resctrl: L3 allocation detected
[   13.097936] resctrl: MB allocation detected
[   13.102599] resctrl: L3 monitoring detected

You can see that L3DATA and L3CODE disappeared. I think we should keep the behavior same for x86(at least).

This is the kernel log ... what user-space software is parsing that for an expected value? What happens if the resctrl strings have been overwritten by more kernel log?

I don't think user-space should be relying on this. I'd argue any user-space doing this is already broken. Is it just the kernel selftest's filter_dmesg()? It doesn't seem to do anything useful

Whether resctrl is support can be read from /proc/filesystems. CDP is probably a try-it-and-see. User-space could parse /proc/cpuinfo, but its probably not a good idea.

Its easy to fix, but it seems odd that the kernel has to print things for user-space to try and parse. (I'd like to point at the user-space software that depends on this)

Quoted 10 lines

I am still not clear why we needed resctrl_conf_type

enum resctrl_conf_type {
CDP_BOTH,
CDP_CODE,
CDP_DATA,
};

Right now, I see all the resources are initialized as CDP_BOTH.

Quoted 9 lines

[RDT_RESOURCE_L3] =
{
.conf_type = CDP_BOTH,
[RDT_RESOURCE_L2] =
{
.conf_type = CDP_BOTH,
[RDT_RESOURCE_MBA] =
{
.conf_type = CDP_BOTH,

Ah, those should have been removed in patch 24. Once all the resources are the same, the resource doesn't need to describe what kind it is.

Quoted 2 lines

If all the resources are CDP_BOTH, then why we need separate CDP_CODE and CDP_DATA?

The filesystem code for resctrl that will eventually move out of arch/x86 needs to be able to describe the type of configuration change being made back to the arch code. The enum gets used for that.

x86 needs this as it affects which MSRs the configuration value is written to.

Quoted 1 line

Are these going to be different for ARM?

Nope. Arm's MPAM ends up emulating CDP with the closid values that get applied to transactions.

Quoted 2 lines

Also initializing RDT_RESOURCE_MBA as CDP_BOTH does not seem right. I dont think there will CDP support in MBA in future.

Its not code or data, which makes it both. 'BOTH' is more of a 'do nothing special', there may be a better name, but I'm not very good at naming things. (any suggestions?)

Thanks,

James

reviewer
#40 lore raw
Quoted 76 lines · James Morse

Hi Babu,

Quoted 40 lines · Babu Moger
Quoted 30 lines · James Morse

This series re-folds the resctrl code so the CDP resources (L3CODE et al) behaviour is all contained in the filesystem parts, with a minimum amount of arch specific code.

Arm have some CPU support for dividing caches into portions, and applying bandwidth limits at various points in the SoC. The collective term for these features is MPAM: Memory Partitioning and Monitoring.

MPAM is similar enough to Intel RDT, that it should use the defacto linux
interface: resctrl. This filesystem currently lives under arch/x86, and is tightly coupled to the architecture.
Ultimately, my plan is to split the existing resctrl code up to have an arch<->fs abstraction, then move all the bits out to fs/resctrl. From there MPAM can be wired up.

x86 might have two resources with cache controls, (L2 and L3) but has extra copies for CDP: L{2,3}{CODE,DATA}, which are marked as enabled if CDP is enabled for the corresponding cache.

MPAM has an equivalent feature to CDP, but its a property of the CPU, not the cache. Resctrl needs to have x86's odd/even behaviour, as that its the ABI, but this isn't how the MPAM hardware works. It is entirely possible that an in-kernel user of MPAM would not be using CDP, whereas resctrl is.
Pretending L3CODE and L3DATA are entirely separate resources is a neat trick, but doing this is specific to x86.
Doing this leaves the arch code in control of various parts of the filesystem ABI: the resources names, and the way the schemata are parsed. Allowing this stuff to vary between architectures is bad for user space.

Quoted 7 lines

This series collapses the CODE/DATA resources, moving all the user-visible resctrl ABI into what becomes the filesystem code. CDP becomes the type of configuration being applied to a cache. This is done by adding a struct resctrl_schema to the parts of resctrl that will move to fs. This holds the arch-code resource that is in use for this schema, along with other properties like the name, and whether the configuration being applied is CODE/DATA/BOTH.

Quoted 1 line

I applied your patches on my AMD box.

Great! Thanks for taking a look,

Quoted 1 line

Seeing some difference in the behavior.

Ooer,

Quoted 12 lines

Before these patches.

# dmesg |grep -i resctrl
[   13.076973] resctrl: L3 allocation detected
[   13.087835] resctrl: L3DATA allocation detected
[   13.092886] resctrl: L3CODE allocation detected
[   13.097936] resctrl: MB allocation detected
[   13.102599] resctrl: L3 monitoring detected

After the patches.

Quoted 7 lines
# dmesg |grep -i resctrl
[   13.076973] resctrl: L3 allocation detected
[   13.097936] resctrl: MB allocation detected
[   13.102599] resctrl: L3 monitoring detected

You can see that L3DATA and L3CODE disappeared. I think we should keep the behavior same for x86(at least).

This is the kernel log ... what user-space software is parsing that for an expected value? What happens if the resctrl strings have been overwritten by more kernel log?

Quoted 6 lines

I don't think user-space should be relying on this. I'd argue any user-space doing this is already broken. Is it just the kernel selftest's filter_dmesg()? It doesn't seem to do anything useful

Whether resctrl is support can be read from /proc/filesystems. CDP is probably a try-it-and-see. User-space could parse /proc/cpuinfo, but its probably not a good idea.

Yes. Agree. Looking at the dmesg may no be right way to figure out all the support details. As a normal practice, I searched for these texts and noticed difference. That is why I felt it is best to keep those texts same as before.

Quoted 4 lines

Its easy to fix, but it seems odd that the kernel has to print things for user-space to try and parse. (I'd like to point at the user-space software that depends on this)

I dont think there is any software that parses the dmesg for these details. These are info messages for the developers.

Quoted 42 lines
Quoted 12 lines

I am still not clear why we needed resctrl_conf_type

enum resctrl_conf_type {
CDP_BOTH,
CDP_CODE,
CDP_DATA,
};

Right now, I see all the resources are initialized as CDP_BOTH.

Quoted 9 lines

[RDT_RESOURCE_L3] =
{
.conf_type = CDP_BOTH,
[RDT_RESOURCE_L2] =
{
.conf_type = CDP_BOTH,
[RDT_RESOURCE_MBA] =
{
.conf_type = CDP_BOTH,

Ah, those should have been removed in patch 24. Once all the resources are the same, the resource doesn't need to describe what kind it is.

Quoted 2 lines

If all the resources are CDP_BOTH, then why we need separate CDP_CODE and CDP_DATA?

The filesystem code for resctrl that will eventually move out of arch/x86 needs to be able to describe the type of configuration change being made back to the arch code. The enum gets used for that.

x86 needs this as it affects which MSRs the configuration value is written to.

Quoted 1 line

Are these going to be different for ARM?

Nope. Arm's MPAM ends up emulating CDP with the closid values that get applied to transactions.

Quoted 5 lines
Quoted 2 lines

Also initializing RDT_RESOURCE_MBA as CDP_BOTH does not seem right. I dont think there will CDP support in MBA in future.

Its not code or data, which makes it both. 'BOTH' is more of a 'do nothing special', there may be a better name, but I'm not very good at naming things. (any suggestions?)

Do you think CDP_NONE will make some sense?

Quoted 6 lines

Thanks,

James

author reviewer
#41 lore raw

Hi Babu,

Quoted 61 lines · Babu Moger
Quoted 48 lines · James Morse
Quoted 14 lines · Babu Moger
Quoted 11 lines · James Morse

This series re-folds the resctrl code so the CDP resources (L3CODE et al) behaviour is all contained in the filesystem parts, with a minimum amount of arch specific code.

This series collapses the CODE/DATA resources, moving all the user-visible resctrl ABI into what becomes the filesystem code. CDP becomes the type of configuration being applied to a cache. This is done by adding a struct resctrl_schema to the parts of resctrl that will move to fs. This holds the arch-code resource that is in use for this schema, along with other properties like the name, and whether the configuration being applied is CODE/DATA/BOTH.

Quoted 1 line

I applied your patches on my AMD box.

Great! Thanks for taking a look,

Quoted 1 line

Seeing some difference in the behavior.

Ooer,

Quoted 12 lines

Before these patches.

# dmesg |grep -i resctrl
[   13.076973] resctrl: L3 allocation detected
[   13.087835] resctrl: L3DATA allocation detected
[   13.092886] resctrl: L3CODE allocation detected
[   13.097936] resctrl: MB allocation detected
[   13.102599] resctrl: L3 monitoring detected

After the patches.

Quoted 7 lines
# dmesg |grep -i resctrl
[   13.076973] resctrl: L3 allocation detected
[   13.097936] resctrl: MB allocation detected
[   13.102599] resctrl: L3 monitoring detected

You can see that L3DATA and L3CODE disappeared. I think we should keep the behavior same for x86(at least).

This is the kernel log ... what user-space software is parsing that for an expected value? What happens if the resctrl strings have been overwritten by more kernel log?

Quoted 6 lines

I don't think user-space should be relying on this. I'd argue any user-space doing this is already broken. Is it just the kernel selftest's filter_dmesg()? It doesn't seem to do anything useful

Whether resctrl is support can be read from /proc/filesystems. CDP is probably a try-it-and-see. User-space could parse /proc/cpuinfo, but its probably not a good idea.

Yes. Agree. Looking at the dmesg may no be right way to figure out all the support details. As a normal practice, I searched for these texts and noticed difference. That is why I felt it is best to keep those texts same as before.

Quoted 5 lines
Quoted 2 lines

Its easy to fix, but it seems odd that the kernel has to print things for user-space to try and parse. (I'd like to point at the user-space software that depends on this)

I dont think there is any software that parses the dmesg for these details. These are info messages for the developers.

The kernel log changes all the time, messages at boot aren't something you can depend on seeing later. Unless there is some user-space software broken by this, I'm afraid I don't think its a good idea to add extra code to keep it the same.

Printing 'CDP supported by Lx' would be more useful to developers perusing the log. Even more useful would be exposing feature attributes via sysfs to say what resctrl supports without having to mount-it-and-see. This can then be used by user-space too. e.g.:
| cat /sys/fs/ext4/features/fast_commit

Quoted 40 lines
Quoted 40 lines
Quoted 10 lines

I am still not clear why we needed resctrl_conf_type

enum resctrl_conf_type {
CDP_BOTH,
CDP_CODE,
CDP_DATA,
};

Right now, I see all the resources are initialized as CDP_BOTH.

Quoted 9 lines

[RDT_RESOURCE_L3] =
{
.conf_type = CDP_BOTH,
[RDT_RESOURCE_L2] =
{
.conf_type = CDP_BOTH,
[RDT_RESOURCE_MBA] =
{
.conf_type = CDP_BOTH,

Ah, those should have been removed in patch 24. Once all the resources are the same, the resource doesn't need to describe what kind it is.

Quoted 2 lines

If all the resources are CDP_BOTH, then why we need separate CDP_CODE and CDP_DATA?

The filesystem code for resctrl that will eventually move out of arch/x86 needs to be able to describe the type of configuration change being made back to the arch code. The enum gets used for that.

x86 needs this as it affects which MSRs the configuration value is written to.

Quoted 1 line

Are these going to be different for ARM?

Nope. Arm's MPAM ends up emulating CDP with the closid values that get applied to transactions.

Quoted 7 lines
Quoted 5 lines
Quoted 2 lines

Also initializing RDT_RESOURCE_MBA as CDP_BOTH does not seem right. I dont think there will CDP support in MBA in future.

Its not code or data, which makes it both. 'BOTH' is more of a 'do nothing special', there may be a better name, but I'm not very good at naming things. (any suggestions?)

Do you think CDP_NONE will make some sense?

If you think that is clearer, sure.

Thanks,

James