Skip to contentSkip to conversation

x86/resctrl: Add support for Sub-NUMA cluster (SNC) systems

PATCH Superseded Tony Luck 12 participants 331 messages Jul 13, 2023 – Feb 13, 2024 Page 4 of 4

Subsystems (4)

v3 of 23 Superseded Latest is v23 →

lore
Share & tools
lore
reviewer
#303 lore raw

Re: [PATCH v15-RFC 5/8] x86/resctrl: Add "NODE" as an option for resource scope

In reply to Tony Luck

Hi Tony,

This patch probably needs to be merged with with patch 7.

Thanks

Quoted 73 lines · Tony Luck

Add RESCTRL_NODE to the enum, and to the helper function that looks up a domain id from a scope.

There are a couple of places where the scope must be a cache scope. Add some defensive WARN_ON checks to those.

---
 include/linux/resctrl.h                   | 1 +
 arch/x86/kernel/cpu/resctrl/core.c        | 3 +++
 arch/x86/kernel/cpu/resctrl/pseudo_lock.c | 4 ++++
 arch/x86/kernel/cpu/resctrl/rdtgroup.c    | 3 +++
 4 files changed, 11 insertions(+)
include/linux/resctrl.h +1 -0
index 2155dc15e636..e3cddf3f07f8 100644
@@ -147,6 +147,7 @@ struct resctrl_schema;
 enum resctrl_scope {
 	RESCTRL_L2_CACHE = 2,
 	RESCTRL_L3_CACHE = 3,
+	RESCTRL_NODE,
 };
 
 /**
arch/x86/kernel/cpu/resctrl/core.c +3 -0
index 59e6aa7abef5..b741cbf61843 100644
@@ -505,6 +505,9 @@ static int get_domain_id_from_scope(int cpu, enum resctrl_scope scope)
 	case RESCTRL_L2_CACHE:
 	case RESCTRL_L3_CACHE:
 		return get_cpu_cacheinfo_id(cpu, scope);
+	case RESCTRL_NODE:
+		return cpu_to_node(cpu);
+
 	default:
 		break;
 	}
arch/x86/kernel/cpu/resctrl/pseudo_lock.c +4 -0
index 6a72fb627aa5..2bafc73b51e2 100644
@@ -292,10 +292,14 @@ static void pseudo_lock_region_clear(struct pseudo_lock_region *plr)
  */
 static int pseudo_lock_region_init(struct pseudo_lock_region *plr)
 {
+	enum resctrl_scope scope = plr->s->res->scope;
 	struct cpu_cacheinfo *ci;
 	int ret;
 	int i;
 
+	if (WARN_ON_ONCE(scope != RESCTRL_L2_CACHE && scope != RESCTRL_L3_CACHE))
+		return -ENODEV;
+
 	/* Pick the first cpu we find that is associated with the cache. */
 	plr->cpu = cpumask_first(&plr->d->cpu_mask);
 
arch/x86/kernel/cpu/resctrl/rdtgroup.c +3 -0
index eff9d87547c9..770f2bf98462 100644
@@ -1413,6 +1413,9 @@ unsigned int rdtgroup_cbm_to_size(struct rdt_resource *r,
 	unsigned int size = 0;
 	int num_b, i;
 
+	if (WARN_ON_ONCE(r->scope != RESCTRL_L2_CACHE && r->scope != RESCTRL_L3_CACHE))
+		return size;
+
 	num_b = bitmap_weight(&cbm, r->cache.cbm_len);
 	ci = get_cpu_cacheinfo(cpumask_any(&d->cpu_mask));
 	for (i = 0; i < ci->num_leaves; i++) {
Signature · 2 lines
Thanks
Babu Moger
author maintainer
#304 lore raw

Re: [PATCH v15-RFC 5/8] x86/resctrl: Add "NODE" as an option for resource scope

Quoted 3 lines · Babu

Hi Tony,

This patch probably needs to be merged with with patch 7.

If it just added RESCTRL_NODE to the enum and the switch() I'd agree (as patch 7 is where RESCTRL_NODE first used). But this patch also adds the sanity checks on scope where it has to be a cache level.

Patch 7 is already on the big side (119 lines added to core.c).

If you really think this series needs to cut back the number of patches, I could move the sanity check pieces from here to patch 3 (where the enum is introduced) and just the RESCTRL_NODE bits to patch 7.

-Tony

reviewer
#305 lore raw

Re: [PATCH v15-RFC 5/8] x86/resctrl: Add "NODE" as an option for resource scope

Hi Tony,

Quoted 10 lines · Tony Luck
Quoted 3 lines · Babu

Hi Tony,

This patch probably needs to be merged with with patch 7.

If it just added RESCTRL_NODE to the enum and the switch() I'd agree (as patch 7 is where RESCTRL_NODE first used). But this patch also adds the sanity checks on scope where it has to be a cache level.

Quoted 6 lines

Patch 7 is already on the big side (119 lines added to core.c).

If you really think this series needs to cut back the number of patches, I could move the sanity check pieces from here to patch 3 (where the enum is introduced) and just the RESCTRL_NODE bits to patch 7.

No. You dont have to cut back on number of patches. I think it is easy to review if these changes are merged with patch 7.

Signature · 2 lines
Thanks
Babu Moger
author maintainer
#306 lore raw

PATCH v3 6/8 x86/resctrl: Introduce snc_nodes_per_l3_cache

In reply to Tony Luck

Intel Sub-NUMA Cluster (SNC) is a feature that subdivides the CPU cores and memory controllers on a socket into two or more groups. These are presented to the operating system as NUMA nodes.

This may enable some workloads to have slightly lower latency to memory as the memory controller(s) in an SNC node are electrically closer to the CPU cores on that SNC node. This cost may be offset by lower bandwidth since the memory accesses for each core can only be interleaved between the memory controllers on the same SNC node.

Resctrl monitoring on an Intel system depends upon attaching RMIDs to tasks to track L3 cache occupancy and memory bandwidth. There is an MSR that controls how the RMIDs are shared between SNC nodes.

The default mode divides them numerically. E.g. when there are two SNC nodes on a socket the lower number half of the RMIDs are given to the first node, the remainder to the second node. This would be difficult to use with the Linux resctrl interface as specific RMID values assigned to resctrl groups are not visible to users.

The other mode divides the RMIDs and renumbers the ones on the second SNC node to start from zero.

Even with this renumbering SNC mode requires several changes in resctrl behavior for correct operation.

Add a global integer "snc_nodes_per_l3_cache" that shows how many SNC nodes share each L3 cache. When "snc_nodes_per_l3_cache" is "1", SNC mode is either not implemented, or not enabled.

Update all places to take appropriate action when SNC mode is enabled:
1) The number of logical RMIDs per L3 cache available for use is the number of physical RMIDs divided by the number of SNC nodes.
2) Likewise the "mon_scale" value must be divided by the number of SNC nodes.
3) The RMID renumbering operates when using the value from the IA32_PQR_ASSOC MSR to count accesses by a task. When reading an RMID counter, adjust from the logical RMID to the physical RMID value for the SNC node that it wishes to read and load the adjusted value into the IA32_QM_EVTSEL MSR.
4) Divide the L3 cache between the SNC nodes. Divide the value reported in the resctrl "size" file by the number of SNC nodes because the effective amount of cache that can be allocated is reduced by that factor.
5) Disable the "-o mba_MBps" mount option in SNC mode because the monitoring is being done per SNC node, while the bandwidth allocation is still done at the L3 cache scope. Trying to use this feedback loop might result in contradictory changes to the throttling level coming from each of the SNC node bandwidth measurements.

---
 arch/x86/kernel/cpu/resctrl/internal.h |  2 ++
 arch/x86/kernel/cpu/resctrl/core.c     |  6 ++++++
 arch/x86/kernel/cpu/resctrl/monitor.c  | 16 +++++++++++++---
 arch/x86/kernel/cpu/resctrl/rdtgroup.c |  5 +++--
 4 files changed, 24 insertions(+), 5 deletions(-)
arch/x86/kernel/cpu/resctrl/internal.h +2 -0
index c6051bc70e96..d9c6dcf30922 100644
@@ -428,6 +428,8 @@ DECLARE_STATIC_KEY_FALSE(rdt_alloc_enable_key);
 
 extern struct dentry *debugfs_resctrl;
 
+extern unsigned int snc_nodes_per_l3_cache;
+
 enum resctrl_res_level {
 	RDT_RESOURCE_L3_MON,
 	RDT_RESOURCE_L3,
arch/x86/kernel/cpu/resctrl/core.c +6 -0
index b741cbf61843..dc886d2c9a33 100644
@@ -48,6 +48,12 @@ int max_name_width, max_data_width;
  */
 bool rdt_alloc_capable;
 
+/*
+ * Number of SNC nodes that share each L3 cache.  Default is 1 for
+ * systems that do not support SNC, or have SNC disabled.
+ */
+unsigned int snc_nodes_per_l3_cache = 1;
+
 static void
 mba_wrmsr_intel(struct rdt_domain *d, struct msr_param *m,
 		struct rdt_resource *r);
arch/x86/kernel/cpu/resctrl/monitor.c +13 -3
index 080cad0d7288..357919bbadbe 100644
@@ -148,8 +148,18 @@ static inline struct rmid_entry *__rmid_entry(u32 rmid)
 
 static int __rmid_read(u32 rmid, enum resctrl_event_id eventid, u64 *val)
 {
+	struct rdt_resource *r = &rdt_resources_all[RDT_RESOURCE_L3].r_resctrl;
+	int cpu = smp_processor_id();
+	int rmid_offset = 0;
 	u64 msr_val;
 
+	/*
+	 * When SNC mode is on, need to compute the offset to read the
+	 * physical RMID counter for the node to which this CPU belongs.
+	 */
+	if (snc_nodes_per_l3_cache > 1)
+		rmid_offset = (cpu_to_node(cpu) % snc_nodes_per_l3_cache) * r->num_rmid;
+
 	/*
 	 * As per the SDM, when IA32_QM_EVTSEL.EvtID (bits 7:0) is configured
 	 * with a valid event code for supported resource type and the bits
@@ -158,7 +168,7 @@ static int __rmid_read(u32 rmid, enum resctrl_event_id eventid, u64 *val)
 	 * IA32_QM_CTR.Error (bit 63) and IA32_QM_CTR.Unavailable (bit 62)
 	 * are error bits.
 	 */
-	wrmsr(MSR_IA32_QM_EVTSEL, eventid, rmid);
+	wrmsr(MSR_IA32_QM_EVTSEL, eventid, rmid + rmid_offset);
 	rdmsrl(MSR_IA32_QM_CTR, msr_val);
 
 	if (msr_val & RMID_VAL_ERROR)
@@ -757,8 +767,8 @@ int __init rdt_get_mon_l3_config(struct rdt_resource *r)
 	int ret;
 
 	resctrl_rmid_realloc_limit = boot_cpu_data.x86_cache_size * 1024;
-	hw_res->mon_scale = boot_cpu_data.x86_cache_occ_scale;
-	r->num_rmid = boot_cpu_data.x86_cache_max_rmid + 1;
+	hw_res->mon_scale = boot_cpu_data.x86_cache_occ_scale / snc_nodes_per_l3_cache;
+	r->num_rmid = (boot_cpu_data.x86_cache_max_rmid + 1) / snc_nodes_per_l3_cache;
 	hw_res->mbm_width = MBM_CNTR_WIDTH_BASE;
 
 	if (mbm_offset > 0 && mbm_offset <= MBM_CNTR_WIDTH_OFFSET_MAX)
arch/x86/kernel/cpu/resctrl/rdtgroup.c +3 -2
index 770f2bf98462..e639069f871a 100644
@@ -1425,7 +1425,7 @@ unsigned int rdtgroup_cbm_to_size(struct rdt_resource *r,
 		}
 	}
 
-	return size;
+	return size / snc_nodes_per_l3_cache;
 }
 
 /*
@@ -2293,7 +2293,8 @@ static bool supports_mba_mbps(void)
 	struct rdt_resource *r = &rdt_resources_all[RDT_RESOURCE_MBA].r_resctrl;
 
 	return (is_mbm_local_enabled() &&
-		r->alloc_capable && is_mba_linear());
+		r->alloc_capable && is_mba_linear() &&
+		snc_nodes_per_l3_cache == 1);
 }
 
 /*
Signature · 1 line
2.43.0
reviewer
#307 lore raw

Re: [PATCH v15-RFC 6/8] x86/resctrl: Introduce snc_nodes_per_l3_cache

Hi Tony,

Quoted 64 lines · Tony Luck

Intel Sub-NUMA Cluster (SNC) is a feature that subdivides the CPU cores and memory controllers on a socket into two or more groups. These are presented to the operating system as NUMA nodes.

This may enable some workloads to have slightly lower latency to memory as the memory controller(s) in an SNC node are electrically closer to the CPU cores on that SNC node. This cost may be offset by lower bandwidth since the memory accesses for each core can only be interleaved between the memory controllers on the same SNC node.

Resctrl monitoring on an Intel system depends upon attaching RMIDs to tasks to track L3 cache occupancy and memory bandwidth. There is an MSR that controls how the RMIDs are shared between SNC nodes.

The default mode divides them numerically. E.g. when there are two SNC nodes on a socket the lower number half of the RMIDs are given to the first node, the remainder to the second node. This would be difficult to use with the Linux resctrl interface as specific RMID values assigned to resctrl groups are not visible to users.

The other mode divides the RMIDs and renumbers the ones on the second SNC node to start from zero.

Even with this renumbering SNC mode requires several changes in resctrl behavior for correct operation.

Add a global integer "snc_nodes_per_l3_cache" that shows how many SNC nodes share each L3 cache. When "snc_nodes_per_l3_cache" is "1", SNC mode is either not implemented, or not enabled.

Update all places to take appropriate action when SNC mode is enabled:
1) The number of logical RMIDs per L3 cache available for use is the number of physical RMIDs divided by the number of SNC nodes.
2) Likewise the "mon_scale" value must be divided by the number of SNC nodes.
3) The RMID renumbering operates when using the value from the IA32_PQR_ASSOC MSR to count accesses by a task. When reading an RMID counter, adjust from the logical RMID to the physical RMID value for the SNC node that it wishes to read and load the adjusted value into the IA32_QM_EVTSEL MSR.
4) Divide the L3 cache between the SNC nodes. Divide the value reported in the resctrl "size" file by the number of SNC nodes because the effective amount of cache that can be allocated is reduced by that factor.
5) Disable the "-o mba_MBps" mount option in SNC mode because the monitoring is being done per SNC node, while the bandwidth allocation is still done at the L3 cache scope. Trying to use this feedback loop might result in contradictory changes to the throttling level coming from each of the SNC node bandwidth measurements.

---
 arch/x86/kernel/cpu/resctrl/internal.h |  2 ++
 arch/x86/kernel/cpu/resctrl/core.c     |  6 ++++++
 arch/x86/kernel/cpu/resctrl/monitor.c  | 16 +++++++++++++---
 arch/x86/kernel/cpu/resctrl/rdtgroup.c |  5 +++--
 4 files changed, 24 insertions(+), 5 deletions(-)
arch/x86/kernel/cpu/resctrl/internal.h +0 -0
index c6051bc70e96..d9c6dcf30922 100644

arch/x86/kernel/cpu/resctrl/internal.h DECLARE_STATIC_KEY_FALSE(rdt_alloc_enable_key); · line 431

Quoted 4 lines
arch/x86/kernel/cpu/resctrl/internal.h +1 -0
@@ -428,6 +428,8 @@ DECLARE_STATIC_KEY_FALSE(rdt_alloc_enable_key);
 
 extern struct dentry *debugfs_resctrl;
 
+extern unsigned int snc_nodes_per_l3_cache;

I feel this can be part of rdt_resource instead of global.

Quoted 26 lines
arch/x86/kernel/cpu/resctrl/internal.h +1 -0
+
 enum resctrl_res_level {
 	RDT_RESOURCE_L3_MON,
 	RDT_RESOURCE_L3,
arch/x86/kernel/cpu/resctrl/core.c +6 -0
index b741cbf61843..dc886d2c9a33 100644
@@ -48,6 +48,12 @@ int max_name_width, max_data_width;
  */
 bool rdt_alloc_capable;
 
+/*
+ * Number of SNC nodes that share each L3 cache.  Default is 1 for
+ * systems that do not support SNC, or have SNC disabled.
+ */
+unsigned int snc_nodes_per_l3_cache = 1;
+
 static void
 mba_wrmsr_intel(struct rdt_domain *d, struct msr_param *m,
 		struct rdt_resource *r);
arch/x86/kernel/cpu/resctrl/monitor.c +0 -0
index 080cad0d7288..357919bbadbe 100644

arch/x86/kernel/cpu/resctrl/monitor.c static inline struct rmid_entry *__rmid_entry(u32 rmid) · line 151

Quoted 4 lines
arch/x86/kernel/cpu/resctrl/monitor.c +1 -0
@@ -148,8 +148,18 @@ static inline struct rmid_entry *__rmid_entry(u32 rmid)
 
 static int __rmid_read(u32 rmid, enum resctrl_event_id eventid, u64 *val)
 {
+	struct rdt_resource *r = &rdt_resources_all[RDT_RESOURCE_L3].r_resctrl;

RDT_RESOURCE_L3_MON?

arch/x86/kernel/cpu/resctrl/monitor.c static inline struct rmid_entry *__rmid_entry(u32 rmid) · line 161

Quoted 10 lines
arch/x86/kernel/cpu/resctrl/monitor.c +8 -0
+	int cpu = smp_processor_id();
+	int rmid_offset = 0;
 	u64 msr_val;
 
+	/*
+	 * When SNC mode is on, need to compute the offset to read the
+	 * physical RMID counter for the node to which this CPU belongs.
+	 */
+	if (snc_nodes_per_l3_cache > 1)
+		rmid_offset = (cpu_to_node(cpu) % snc_nodes_per_l3_cache) * r->num_rmid;

Not sure if you have tested or not. r->num_rmid is initialized for the resource RDT_RESOURCE_L3_MON. For other resource it is always 0.

Quoted 47 lines
arch/x86/kernel/cpu/resctrl/monitor.c +4 -3
+
 	/*
 	 * As per the SDM, when IA32_QM_EVTSEL.EvtID (bits 7:0) is configured
 	 * with a valid event code for supported resource type and the bits
@@ -158,7 +168,7 @@ static int __rmid_read(u32 rmid, enum resctrl_event_id eventid, u64 *val)
 	 * IA32_QM_CTR.Error (bit 63) and IA32_QM_CTR.Unavailable (bit 62)
 	 * are error bits.
 	 */
-	wrmsr(MSR_IA32_QM_EVTSEL, eventid, rmid);
+	wrmsr(MSR_IA32_QM_EVTSEL, eventid, rmid + rmid_offset);
 	rdmsrl(MSR_IA32_QM_CTR, msr_val);
 
 	if (msr_val & RMID_VAL_ERROR)
@@ -757,8 +767,8 @@ int __init rdt_get_mon_l3_config(struct rdt_resource *r)
 	int ret;
 
 	resctrl_rmid_realloc_limit = boot_cpu_data.x86_cache_size * 1024;
-	hw_res->mon_scale = boot_cpu_data.x86_cache_occ_scale;
-	r->num_rmid = boot_cpu_data.x86_cache_max_rmid + 1;
+	hw_res->mon_scale = boot_cpu_data.x86_cache_occ_scale / snc_nodes_per_l3_cache;
+	r->num_rmid = (boot_cpu_data.x86_cache_max_rmid + 1) / snc_nodes_per_l3_cache;
 	hw_res->mbm_width = MBM_CNTR_WIDTH_BASE;
 
 	if (mbm_offset > 0 && mbm_offset <= MBM_CNTR_WIDTH_OFFSET_MAX)
arch/x86/kernel/cpu/resctrl/rdtgroup.c +3 -2
index 770f2bf98462..e639069f871a 100644
@@ -1425,7 +1425,7 @@ unsigned int rdtgroup_cbm_to_size(struct rdt_resource *r,
 		}
 	}
 
-	return size;
+	return size / snc_nodes_per_l3_cache;
 }
 
 /*
@@ -2293,7 +2293,8 @@ static bool supports_mba_mbps(void)
 	struct rdt_resource *r = &rdt_resources_all[RDT_RESOURCE_MBA].r_resctrl;
 
 	return (is_mbm_local_enabled() &&
-		r->alloc_capable && is_mba_linear());
+		r->alloc_capable && is_mba_linear() &&
+		snc_nodes_per_l3_cache == 1);
 }
 
 /*
Signature · 2 lines
Thanks
Babu Moger
author maintainer
#308 lore raw

Re: [PATCH v15-RFC 6/8] x86/resctrl: Introduce snc_nodes_per_l3_cache

Quoted 4 lines · Babu
Quoted 2 lines

+extern unsigned int snc_nodes_per_l3_cache;

I feel this can be part of rdt_resource instead of global.

Mixed emotions about that. It would be another field that appears in every instance of rdt_resource, but only used by the RDT_RESOURCE_L3_MON copy.

Quoted 28 lines
Quoted 28 lines
arch/x86/kernel/cpu/resctrl/internal.h +1 -0
+
 enum resctrl_res_level {
 	RDT_RESOURCE_L3_MON,
 	RDT_RESOURCE_L3,
arch/x86/kernel/cpu/resctrl/core.c +6 -0
index b741cbf61843..dc886d2c9a33 100644
@@ -48,6 +48,12 @@ int max_name_width, max_data_width;
  */
 bool rdt_alloc_capable;
 
+/*
+ * Number of SNC nodes that share each L3 cache.  Default is 1 for
+ * systems that do not support SNC, or have SNC disabled.
+ */
+unsigned int snc_nodes_per_l3_cache = 1;
+
 static void
 mba_wrmsr_intel(struct rdt_domain *d, struct msr_param *m,
 		struct rdt_resource *r);
arch/x86/kernel/cpu/resctrl/monitor.c +0 -0
index 080cad0d7288..357919bbadbe 100644
@@ -148,8 +148,18 @@ static inline struct rmid_entry *__rmid_entry(u32 rmid)
 
Quoted 5 lines

arch/x86/kernel/cpu/resctrl/monitor.c static inline struct rmid_entry *__rmid_entry(u32 rmid) · line 151

Quoted 3 lines
arch/x86/kernel/cpu/resctrl/monitor.c +1 -0
 static int __rmid_read(u32 rmid, enum resctrl_event_id eventid, u64 *val)
 {
+	struct rdt_resource *r = &rdt_resources_all[RDT_RESOURCE_L3].r_resctrl;

RDT_RESOURCE_L3_MON?

Second good catch.

Quoted 5 lines
Quoted 5 lines
arch/x86/kernel/cpu/resctrl/monitor.c +2 -0
+	int cpu = smp_processor_id();
+	int rmid_offset = 0;
 	u64 msr_val;
 
Quoted 9 lines

arch/x86/kernel/cpu/resctrl/monitor.c static inline struct rmid_entry *__rmid_entry(u32 rmid) · line 161

Quoted 6 lines
arch/x86/kernel/cpu/resctrl/monitor.c +6 -0
+	/*
+	 * When SNC mode is on, need to compute the offset to read the
+	 * physical RMID counter for the node to which this CPU belongs.
+	 */
+	if (snc_nodes_per_l3_cache > 1)
+		rmid_offset = (cpu_to_node(cpu) % snc_nodes_per_l3_cache) * r->num_rmid;

Not sure if you have tested or not. r->num_rmid is initialized for the resource RDT_RESOURCE_L3_MON. For other resource it is always 0.

I hadn't got time on the SNC machine to try this out. Thanks for catching this one, I'd have been scratching my head for a while to track the symptoms of this problem back to this mistake.

Thanks

-Tony

reviewer
#309 lore raw

Re: [PATCH v15-RFC 6/8] x86/resctrl: Introduce snc_nodes_per_l3_cache

Hi Tony,

Quoted 10 lines · Tony Luck
Quoted 4 lines · Babu
Quoted 2 lines

+extern unsigned int snc_nodes_per_l3_cache;

I feel this can be part of rdt_resource instead of global.

Mixed emotions about that. It would be another field that appears in every instance of rdt_resource, but only used by the RDT_RESOURCE_L3_MON copy.

That was the comment I got earlier for my patches(search for global).

https://lore.kernel.org/lkml/[email protected]/

Signature · 2 lines
Thanks
Babu Moger
author maintainer
#310 lore raw

PATCH v3 7/8 x86/resctrl: Sub NUMA Cluster detection and enable

In reply to Tony Luck

There isn't a simple hardware bit that indicates whether a CPU is running in Sub NUMA Cluster (SNC) mode. Infer the state by comparing the ratio of NUMA nodes to L3 cache instances.

When SNC mode is detected, reconfigure the RMID counters by updating the MSR_RMID_SNC_CONFIG MSR on each socket as CPUs are seen. Update the scope of the RDT_RESOURCE_L3_MON resource to NODE.

Clearing bit zero of the MSR divides the RMIDs and renumbers the ones on the second SNC node to start from zero.

---
 arch/x86/include/asm/msr-index.h   |   1 +
 arch/x86/kernel/cpu/resctrl/core.c | 119 +++++++++++++++++++++++++++++
 2 files changed, 120 insertions(+)
arch/x86/include/asm/msr-index.h +1 -0
index f1bd7b91b3c6..f6ba7d0397b8 100644
@@ -1119,6 +1119,7 @@
 #define MSR_IA32_QM_CTR			0xc8e
 #define MSR_IA32_PQR_ASSOC		0xc8f
 #define MSR_IA32_L3_CBM_BASE		0xc90
+#define MSR_RMID_SNC_CONFIG		0xca0
 #define MSR_IA32_L2_CBM_BASE		0xd10
 #define MSR_IA32_MBA_THRTL_BASE		0xd50
 
arch/x86/kernel/cpu/resctrl/core.c +119 -0

152 lines not on this page · raw message

Signature · 1 line
2.43.0
author maintainer
#311 lore raw

PATCH v3 8/8 x86/resctrl: Update documentation with Sub-NUMA cluster changes

In reply to Tony Luck

With Sub-NUMA Cluster mode enabled the scope of monitoring resources is per-NODE instead of per-L3 cache. Suffixes of directories with "L3" in their name refer to Sub-NUMA nodes instead of L3 cache ids.

Users should be aware that SNC mode also affects the amount of L3 cache available for allocation within each SNC node.

---
 Documentation/arch/x86/resctrl.rst | 25 +++++++++++++++++++++----
 1 file changed, 21 insertions(+), 4 deletions(-)
Documentation/arch/x86/resctrl.rst +21 -4
index a6279df64a9d..15f1cff6ee76 100644
@@ -366,10 +366,10 @@ When control is enabled all CTRL_MON groups will also contain:
 When monitoring is enabled all MON groups will also contain:
 
 "mon_data":
-	This contains a set of files organized by L3 domain and by
-	RDT event. E.g. on a system with two L3 domains there will
-	be subdirectories "mon_L3_00" and "mon_L3_01".	Each of these
-	directories have one file per event (e.g. "llc_occupancy",
+	This contains a set of files organized by L3 domain or by NUMA
+	node (depending on whether Sub-NUMA Cluster (SNC) mode is disabled
+	or enabled respectively) and by RDT event.  Each of these
+	directories has one file per event (e.g. "llc_occupancy",
 	"mbm_total_bytes", and "mbm_local_bytes"). In a MON group these
 	files provide a read out of the current value of the event for
 	all tasks in the group. In CTRL_MON groups these files provide
@@ -478,6 +478,23 @@ if non-contiguous 1s value is supported. On a system with a 20-bit mask
 each bit represents 5% of the capacity of the cache. You could partition
 the cache into four equal parts with masks: 0x1f, 0x3e0, 0x7c00, 0xf8000.
 
+Notes on Sub-NUMA Cluster mode
+==============================
+When SNC mode is enabled, Linux may load balance tasks between Sub-NUMA
+nodes much more readily than between regular NUMA nodes since the CPUs
+on Sub-NUMA nodes share the same L3 cache and the system may report
+the NUMA distance between Sub-NUMA nodes with a lower value than used
+for regular NUMA nodes.  Users who do not bind tasks to the CPUs of a
+specific Sub-NUMA node must read the "llc_occupancy", "mbm_total_bytes",
+and "mbm_local_bytes" for all Sub-NUMA nodes where the tasks may execute
+to get the full view of traffic for which the tasks were the source.
+
+The cache allocation feature still provides the same number of
+bits in a mask to control allocation into the L3 cache, but each
+of those ways has its capacity reduced because the cache is divided
+between the SNC nodes. The values reported in the resctrl
+"size" files are adjusted accordingly.
+
 Memory bandwidth Allocation and monitoring
 ==========================================
 
Signature · 1 line
2.43.0
maintainer
#312 lore raw

Re: [PATCH v15-RFC 0/8] Add support for Sub-NUMA cluster (SNC) systems

In reply to Tony Luck

Hi Tony,

Quoted 13 lines · Tony Luck

This is the re-worked version of this series that I promised to post yesterday. Check that e-mail for the arguments for this alternate approach.

https://lore.kernel.org/all/ZbhLRDvZrxBZDv2j@agluck-desk3/

Apologies to Drew Fustini who I'd somehow dropped from later versions of this series. Drew: you had made a comment at one point that having different scopes within a single resource may be useful on RISC-V. Version 14 included that, but it's gone here. Maybe multiple resctrl "struct resource" for a single h/w entity like L3 as I'm doing in this version could work for you too?

Quoted 4 lines

Patches 1-5 are almost completely rewritten based around the new idea to give CMT and MBM their own "resource" instead of sharing one with L3 CAT. This removes the need for separate domain lists, and thus most of the churn of the previous version of this series.

I do not see it as removing the need for separate domain lists but instead keeping the idea of separate domain lists but in this case duplicating the resource in order to host the second domain list. This solution also keeps the same structures for control and monitoring that previous version cleaned up [1].
To me this thus seems like a similar solution as v14 but with additional duplication due to an extra resource and without the cleanup.

Reinette

[1] https://lore.kernel.org/lkml/[email protected]/

#313 lore raw

Re: [PATCH v15-RFC 0/8] Add support for Sub-NUMA cluster (SNC) systems

In reply to Tony Luck
Quoted 4 lines · Tony Luck

This is the re-worked version of this series that I promised to post yesterday. Check that e-mail for the arguments for this alternate approach.

Quoted 8 lines

https://lore.kernel.org/all/ZbhLRDvZrxBZDv2j@agluck-desk3/

Apologies to Drew Fustini who I'd somehow dropped from later versions of this series. Drew: you had made a comment at one point that having different scopes within a single resource may be useful on RISC-V. Version 14 included that, but it's gone here. Maybe multiple resctrl "struct resource" for a single h/w entity like L3 as I'm doing in this version could work for you too?

Sorry for the latency.

The RISC-V CBQRI specification [1] describes a bandwidth controller register interface [2]. It allows a controller to implement both bandwidth allocation and bandwidth usage monitoring.

The proof-of-concept resctrl implementation [3] that I worked on created two domains for each memory controller in the example SoC. One domain would contain the MBA resource and the other would contain the L3 resource to represent MBM files like local_bytes:

  # cat /sys/fs/resctrl/schemata
  MB:4=  80;6=  80;8=  80
  L2:0=0fff;1=0fff
  L3:2=ffff;3=0000;5=0000;7=0000

Where:

Domain 0 is L2 cache controller 0 capacity allocation Domain 1 is L2 cache controller 1 capacity allocation Domain 2 is L3 cache controller capacity allocation

Domain 4 is Memory controller 0 bandwidth allocation Domain 6 is Memory controller 1 bandwidth allocation Domain 8 is Memory controller 2 bandwidth allocation

Domain 3 is Memory controller 0 bandwidth monitoring Domain 5 is Memory controller 1 bandwidth monitoring Domain 7 is Memory controller 2 bandwidth monitoring

I think this scheme is confusing but I wasn't able to find a better way to do it at the time.

Quoted 4 lines

Patches 1-5 are almost completely rewritten based around the new idea to give CMT and MBM their own "resource" instead of sharing one with L3 CAT. This removes the need for separate domain lists, and thus most of the churn of the previous version of this series.

Very interesting. Do you think I would be able to create MBM files for each memory controller without creating pointless L3 domains that show up in schemata?

Thanks,
Drew

[1] https://github.com/riscv-non-isa/riscv-cbqri/releases/tag/v1.0-rc1
[2] https://github.com/riscv-non-isa/riscv-cbqri/blob/main/qos_bandwidth.adoc
[3] https://lore.kernel.org/linux-riscv/[email protected]/

author maintainer
#314 lore raw

RE: [PATCH v15-RFC 0/8] Add support for Sub-NUMA cluster (SNC) systems

Quoted 8 lines
Quoted 4 lines

Patches 1-5 are almost completely rewritten based around the new idea to give CMT and MBM their own "resource" instead of sharing one with L3 CAT. This removes the need for separate domain lists, and thus most of the churn of the previous version of this series.

Very interesting. Do you think I would be able to create MBM files for each memory controller without creating pointless L3 domains that show up in schemata?

Entries only show up in the schemata file for resources that are "alloc_capable". So you should be able to make as many rdt_hw_resource structures as you need that are "mon_capable", but not "alloc_capable" ... though making more than one such resource may explore untested areas of the code since there has historically only been one mon_capable resource. It looks like the resource id from the "rid" field is passed through to the "show" functions for MBM and CQM.

This patch series splits the one resource that is marked as both mon_capable and alloc_capable into two. Maybe that's a useful cleanup, but maybe not a requirement for what you need.

-Tony

reviewer
#315 lore raw

Re: [PATCH v15-RFC 0/8] Add support for Sub-NUMA cluster (SNC) systems

In reply to Tony Luck

Hi Tony,

Quoted 3 lines · Tony Luck

This is the re-worked version of this series that I promised to post yesterday. Check that e-mail for the arguments for this alternate approach.

To be honest, I like this series more than the previous series. I always thought RDT_RESOURCE_L3_MON should have been a separate resource by itself.

You need to separate the domain lists for RDT_RESOURCE_L3 and RDT_RESOURCE_L3_MON if you are going this route. I didn't see that in this series. Also I have few other comments as well.

Thanks
Babu

author maintainer
#316 lore raw

Re: [PATCH v15-RFC 0/8] Add support for Sub-NUMA cluster (SNC) systems

Quoted 7 lines · Babu

Hi Tony,

Quoted 3 lines · Tony Luck

This is the re-worked version of this series that I promised to post yesterday. Check that e-mail for the arguments for this alternate approach.

Quoted 6 lines

To be honest, I like this series more than the previous series. I always thought RDT_RESOURCE_L3_MON should have been a separate resource by itself.

You need to separate the domain lists for RDT_RESOURCE_L3 and RDT_RESOURCE_L3_MON if you are going this route. I didn't see that in this series. Also I have few other comments as well.

They are separated. Each "struct rdt_resource" has its own domain list.

Or do you mean break up the struct rdt_domain into the control and monitor versions as was done in the previous series?

Quoted 4 lines

Thanks
Babu

reviewer
#317 lore raw

Re: [PATCH v15-RFC 0/8] Add support for Sub-NUMA cluster (SNC) systems

Quoted 8 lines · Tony Luck
Quoted 7 lines · Babu

Hi Tony,

Quoted 3 lines · Tony Luck

This is the re-worked version of this series that I promised to post yesterday. Check that e-mail for the arguments for this alternate approach.

Quoted 8 lines
Quoted 6 lines

To be honest, I like this series more than the previous series. I always thought RDT_RESOURCE_L3_MON should have been a separate resource by itself.

You need to separate the domain lists for RDT_RESOURCE_L3 and RDT_RESOURCE_L3_MON if you are going this route. I didn't see that in this series. Also I have few other comments as well.

They are separated. Each "struct rdt_resource" has its own domain list.

Yea. You are right.

Quoted 3 lines

Or do you mean break up the struct rdt_domain into the control and monitor versions as was done in the previous series?

No. Not required. Each resource has its own domain list. So, it is separated already as far as I can see.

Reinette seem to have some concerns about this series. But, I am fine with both these approaches. I feel this is more clean approach.

Signature · 2 lines
Thanks
Babu Moger
maintainer
#318 lore raw

Re: [PATCH v15-RFC 0/8] Add support for Sub-NUMA cluster (SNC) systems

Quoted 6 lines · Babu
Quoted 2 lines · Tony Luck
Quoted 1 line · Babu

On 1/30/24 16:20, Tony Luck wrote:

Reinette seem to have some concerns about this series. But, I am fine with both these approaches. I feel this is more clean approach.

I questioned the motivation but never received a response.

Reinette

author maintainer
#319 lore raw

RE: [PATCH v15-RFC 0/8] Add support for Sub-NUMA cluster (SNC) systems

Quoted 4 lines
Quoted 2 lines

Reinette seem to have some concerns about this series. But, I am fine with both these approaches. I feel this is more clean approach.

I questioned the motivation but never received a response.

Reinette,

Sorry. My motivation was to reduce the amount of code churn that was done in the by the previous incarnation.

   9 files changed, 629 insertions(+), 282 deletions(-)

Vast amounts of that just added "_mon" or "_ctrl" to structure or variable names.

-Tony

maintainer
#320 lore raw

Re: [PATCH v15-RFC 0/8] Add support for Sub-NUMA cluster (SNC) systems

Hi Tony,

Quoted 7 lines · Tony
Quoted 4 lines
Quoted 2 lines

Reinette seem to have some concerns about this series. But, I am fine with both these approaches. I feel this is more clean approach.

I questioned the motivation but never received a response.

Reinette,

Quoted 7 lines

Sorry. My motivation was to reduce the amount of code churn that was done in the by the previous incarnation.

   9 files changed, 629 insertions(+), 282 deletions(-)

Vast amounts of that just added "_mon" or "_ctrl" to structure or variable names.

I actually had specific points that this response also ignores. Let me repeat and highlight the same points:

1) You claim that this series "removes the need for separate domain lists" ... but then this series does just that (create a separate domain list), but in an obfuscated way (duplicate the resource to have the monitoring domain list in there).
2) You claim this series "reduces amount of code churn", but this is because this series keeps using the same original data structures for separate monitoring and control usages. The previous series made an effort to separate the structures for the different usages but this series does not. What makes it ok in this series to use the same data structures for different usages?

Additionally:

Regarding "Vast amounts of that just added "_mon" or "_ctrl" to structure or variable names." ... that is because the structures are actually split, no? It is not just renaming for unnecessary churn.

What is the benefit of keeping the data structures to be shared between monitor and control usages?
If there is a benefit to keeping these data structures, why not just address this aspect in previous solution?

Reinette

author maintainer
#321 lore raw

RE: [PATCH v15-RFC 0/8] Add support for Sub-NUMA cluster (SNC) systems

Quoted 7 lines

I actually had specific points that this response also ignores. Let me repeat and highlight the same points:

1) You claim that this series "removes the need for separate domain lists" ... but then this series does just that (create a separate domain list), but in an obfuscated way (duplicate the resource to have the monitoring domain list in there).

That was poorly worded on my part. I should have said "removes the need for separate domain lists within a single rdt_resource".

Adding an extra domain list to a resource may be the start of a slippery slope. What if there is some additional "L3"-like resctrl operation that acts at the socket level (Intel has made products with multiple L3 instances per socket before). Would you be OK add a third domain list to every struct rdt_resource to handle this? Or would it be simpler to just add a new rdt_resource structure with socket scoped domains?

Quoted 6 lines

2) You claim this series "reduces amount of code churn", but this is because this series keeps using the same original data structures for separate monitoring and control usages. The previous series made an effort to separate the structures for the different usages but this series does not. What makes it ok in this series to use the same data structures for different usages?

Legacy resctrl has been using the same rdt_domain structure for both usages since the dawn of time. So it has been OK up until now.

Quoted 5 lines

Additionally:

Regarding "Vast amounts of that just added "_mon" or "_ctrl" to structure or variable names." ... that is because the structures are actually split, no? It is not just renaming for unnecessary churn.

Perhaps not "unnecessary" churn. But certainly a lot of code change for what I perceive as very little real gain.

Quoted 2 lines

What is the benefit of keeping the data structures to be shared between monitor and control usages?

Benefit is no code changes. Cost is continuing to waste memory with structures that are slightly bigger than they need to be.

Quoted 2 lines

If there is a benefit to keeping these data structures, why not just address this aspect in previous solution?

The previous solution evolved to splitting these structures. But this happened incrementally (remember that at an early stage the monitor structures all got the "_mon" addition to their names, but the control structures kept the original names). Only when I got to the end of this process did I look at the magnitude of the change.

-Tony

maintainer
#322 lore raw

Re: [PATCH v15-RFC 0/8] Add support for Sub-NUMA cluster (SNC) systems

Hi Tony,

Quoted 11 lines · Tony
Quoted 7 lines

I actually had specific points that this response also ignores. Let me repeat and highlight the same points:

1) You claim that this series "removes the need for separate domain lists" ... but then this series does just that (create a separate domain list), but in an obfuscated way (duplicate the resource to have the monitoring domain list in there).

That was poorly worded on my part. I should have said "removes the need for separate domain lists within a single rdt_resource".

Quoted 6 lines

Adding an extra domain list to a resource may be the start of a slippery slope. What if there is some additional "L3"-like resctrl operation that acts at the socket level (Intel has made products with multiple L3 instances per socket before). Would you be OK add a third domain list to every struct rdt_resource to handle this? Or would it be simpler to just add a new rdt_resource structure with socket scoped domains?

This should not be about what is simplest to patch into current resctrl.

There is no need to support a new domain list for a new scope. The domain lists support the functionality: control or monitoring. If control has socket scope the existing implementation supports that.
If there is another operation supported by a resource apart from control or monitoring then we can consider how to support it when we know what it is. That would also be a great point to decide if the same data structure should just grow to support an operation that not all resources may support. That may depend on the amount of data needed to support this hypothetical operation.

Quoted 10 lines
Quoted 7 lines

2) You claim this series "reduces amount of code churn", but this is because this series keeps using the same original data structures for separate monitoring and control usages. The previous series made an effort to separate the structures for the different usages but this series does not. What makes it ok in this series to use the same data structures for different usages?

Legacy resctrl has been using the same rdt_domain structure for both usages since the dawn of time. So it has been OK up until now.

This is not the same.

Legacy resctrl uses the same data structure in the same list for both control and monitoring usages so it is fine to have both monitoring and control data in the data structure.

What you are doing in both solutions is to place the same data structure in separate lists for control and monitoring usages. In the one list only the control data is used, on the other only the monitoring data is used.

Quoted 8 lines
Quoted 5 lines

Additionally:

Regarding "Vast amounts of that just added "_mon" or "_ctrl" to structure or variable names." ... that is because the structures are actually split, no? It is not just renaming for unnecessary churn.

Perhaps not "unnecessary" churn. But certainly a lot of code change for what I perceive as very little real gain.

ok. There may be little gain wrt saving space. One complication with this single data structure is that its content may only be decided based on which list it is part of. It should be obvious to developers when which members are valid. Perhaps this can be addressed with clear documentation of the data structures.

Quoted 7 lines
Quoted 3 lines

What is the benefit of keeping the data structures to be shared between monitor and control usages?

Benefit is no code changes. Cost is continuing to waste memory with structures that are slightly bigger than they need to be.

Quoted 8 lines
Quoted 2 lines

If there is a benefit to keeping these data structures, why not just address this aspect in previous solution?

The previous solution evolved to splitting these structures. But this happened incrementally (remember that at an early stage the monitor structures all got the "_mon" addition to their names, but the control structures kept the original names). Only when I got to the end of this process did I look at the magnitude of the change.

Not answering my question.

Reinette

author maintainer
#323 lore raw

RE: [PATCH v15-RFC 0/8] Add support for Sub-NUMA cluster (SNC) systems

Quoted 11 lines
Quoted 11 lines
Quoted 7 lines

I actually had specific points that this response also ignores. Let me repeat and highlight the same points:

1) You claim that this series "removes the need for separate domain lists" ... but then this series does just that (create a separate domain list), but in an obfuscated way (duplicate the resource to have the monitoring domain list in there).

That was poorly worded on my part. I should have said "removes the need for separate domain lists within a single rdt_resource".

Quoted 8 lines
Quoted 6 lines

Adding an extra domain list to a resource may be the start of a slippery slope. What if there is some additional "L3"-like resctrl operation that acts at the socket level (Intel has made products with multiple L3 instances per socket before). Would you be OK add a third domain list to every struct rdt_resource to handle this? Or would it be simpler to just add a new rdt_resource structure with socket scoped domains?

This should not be about what is simplest to patch into current resctrl.

I wanted to offer this in case Boris also thought that the previous version was too much churn to support an obscure Intel-only (so far) feature.

But if you are going to Nack this new version on the grounds that it muddies the water about usage of the rdt_domain structure, then I will abandon it.

Quoted 56 lines

There is no need to support a new domain list for a new scope. The domain lists support the functionality: control or monitoring. If control has socket scope the existing implementation supports that.
If there is another operation supported by a resource apart from control or monitoring then we can consider how to support it when we know what it is. That would also be a great point to decide if the same data structure should just grow to support an operation that not all resources may support. That may depend on the amount of data needed to support this hypothetical operation.

Quoted 10 lines
Quoted 7 lines

2) You claim this series "reduces amount of code churn", but this is because this series keeps using the same original data structures for separate monitoring and control usages. The previous series made an effort to separate the structures for the different usages but this series does not. What makes it ok in this series to use the same data structures for different usages?

Legacy resctrl has been using the same rdt_domain structure for both usages since the dawn of time. So it has been OK up until now.

This is not the same.

Legacy resctrl uses the same data structure in the same list for both control and monitoring usages so it is fine to have both monitoring and control data in the data structure.

What you are doing in both solutions is to place the same data structure in separate lists for control and monitoring usages. In the one list only the control data is used, on the other only the monitoring data is used.

Quoted 8 lines
Quoted 5 lines

Additionally:

Regarding "Vast amounts of that just added "_mon" or "_ctrl" to structure or variable names." ... that is because the structures are actually split, no? It is not just renaming for unnecessary churn.

Perhaps not "unnecessary" churn. But certainly a lot of code change for what I perceive as very little real gain.

ok. There may be little gain wrt saving space. One complication with this single data structure is that its content may only be decided based on which list it is part of. It should be obvious to developers when which members are valid. Perhaps this can be addressed with clear documentation of the data structures.

Quoted 10 lines
Quoted 3 lines

What is the benefit of keeping the data structures to be shared between monitor and control usages?

Benefit is no code changes. Cost is continuing to waste memory with structures that are slightly bigger than they need to be.

Quoted 2 lines

If there is a benefit to keeping these data structures, why not just address this aspect in previous solution?

Quoted 7 lines
Quoted 5 lines

The previous solution evolved to splitting these structures. But this happened incrementally (remember that at an early stage the monitor structures all got the "_mon" addition to their names, but the control structures kept the original names). Only when I got to the end of this process did I look at the magnitude of the change.

Not answering my question.

I'm not exactly sure what "aspect" you thought could be addressed in the previous series. But the point is moot now. This diversion from the series has come to a dead end, and I hope that Boris will look at v14 (either before the next group of ARM patches, or after).

-Tony

maintainer
#324 lore raw

Re: [PATCH v15-RFC 0/8] Add support for Sub-NUMA cluster (SNC) systems

In reply to Moger, Babu

Hi Babu,

Quoted 2 lines · Babu

To be honest, I like this series more than the previous series. I always thought RDT_RESOURCE_L3_MON should have been a separate resource by itself.

Would you prefer that your "Reviewed-by" tag be removed from the previous series?

Reinette

author maintainer
#325 lore raw

RE: [PATCH v15-RFC 0/8] Add support for Sub-NUMA cluster (SNC) systems

Quoted 5 lines
Quoted 2 lines

To be honest, I like this series more than the previous series. I always thought RDT_RESOURCE_L3_MON should have been a separate resource by itself.

Would you prefer that your "Reviewed-by" tag be removed from the previous series?

I'm thinking that I could continue splitting things and break "struct rdt_resource" into separate "ctrl" and "mon" structures. Then we'd have a clean split from top to bottom.

Doing that would get rid of the rdt_resources_all[] array. Replacing with individual rdt_hw_ctrl_resource and rdt_hw_mon_resource declarations for each feature.

Features found on a system would be added to a list of ctrl or list of mon resources.

-Tony

maintainer
#326 lore raw

Re: [PATCH v15-RFC 0/8] Add support for Sub-NUMA cluster (SNC) systems

Hi Tony,

Quoted 8 lines · Tony
Quoted 5 lines
Quoted 2 lines

To be honest, I like this series more than the previous series. I always thought RDT_RESOURCE_L3_MON should have been a separate resource by itself.

Would you prefer that your "Reviewed-by" tag be removed from the previous series?

I'm thinking that I could continue splitting things and break "struct rdt_resource" into separate "ctrl" and "mon" structures. Then we'd have a clean split from top to bottom.

It is not obvious what you mean with "continue splitting things". Are you speaking about "continue splitting from v14" or "continue splitting from v15-RFC"?

I think that any solution needs to consider what makes sense for resctrl as a whole instead of how to support SNC with smallest patch possible.

There should not be any changes that makes resctrl harder to understand and maintain, as exemplified by confusion introduced by a simple thing as resource name choice [1].

Quoted 5 lines

Doing that would get rid of the rdt_resources_all[] array. Replacing with individual rdt_hw_ctrl_resource and rdt_hw_mon_resource declarations for each feature.

Features found on a system would be added to a list of ctrl or list of mon resources.

Could you please elaborate what is architecturally wrong with v14 and how this new proposal addresses that?

Reinette

[1] https://lore.kernel.org/lkml/ZcZyqs5hnQqZ5ZV0@agluck-desk3/

author maintainer
#327 lore raw

Re: [PATCH v15-RFC 0/8] Add support for Sub-NUMA cluster (SNC) systems

Quoted 6 lines · Reinette Chatre

Hi Tony,

Quoted 3 lines · Tony
Quoted 3 lines
Quoted 2 lines

To be honest, I like this series more than the previous series. I always thought RDT_RESOURCE_L3_MON should have been a separate resource by itself.

Quoted 8 lines
Quoted 5 lines
Quoted 2 lines

Would you prefer that your "Reviewed-by" tag be removed from the previous series?

I'm thinking that I could continue splitting things and break "struct rdt_resource" into separate "ctrl" and "mon" structures. Then we'd have a clean split from top to bottom.

It is not obvious what you mean with "continue splitting things". Are you speaking about "continue splitting from v14" or "continue splitting from v15-RFC"?

I'm speaking of some future potential changes. Not proposing to do this now.

Quoted 2 lines

I think that any solution needs to consider what makes sense for resctrl as a whole instead of how to support SNC with smallest patch possible.

I am officially abandoning my v15-RFC patches. I wasn't clear enough in my e-mail earlier today.

https://lore.kernel.org/all/SJ1PR11MB608378D1304224D9E8A9016FFC482@SJ1PR11MB6083.namprd11.prod.outlook.com/

Quoted 6 lines

There should not be any changes that makes resctrl harder to understand and maintain, as exemplified by confusion introduced by a simple thing as resource name choice [1].

Quoted 1 line
Quoted 7 lines
Quoted 4 lines

Doing that would get rid of the rdt_resources_all[] array. Replacing with individual rdt_hw_ctrl_resource and rdt_hw_mon_resource declarations for each feature.

Features found on a system would be added to a list of ctrl or list of mon resources.

Could you please elaborate what is architecturally wrong with v14 and how this new proposal addresses that?

There is nothing architecturally wrong with v14. I thought it was more complex than it needed to be. You have convinced me that my v15-RFC series, while simpler, is not a reasonable path for long-term resctrl maintainability.

Quoted 4 lines

Reinette

[1] https://lore.kernel.org/lkml/ZcZyqs5hnQqZ5ZV0@agluck-desk3/

-Tony

reviewer
#328 lore raw

Re: [PATCH v15-RFC 0/8] Add support for Sub-NUMA cluster (SNC) systems

Hello,

Quoted 30 lines · Tony Luck
Quoted 4 lines · Reinette Chatre
Quoted 3 lines · Tony
Quoted 3 lines
Quoted 2 lines

To be honest, I like this series more than the previous series. I always thought RDT_RESOURCE_L3_MON should have been a separate resource by itself.

Quoted 8 lines
Quoted 5 lines
Quoted 2 lines

Would you prefer that your "Reviewed-by" tag be removed from the previous series?

I'm thinking that I could continue splitting things and break "struct rdt_resource" into separate "ctrl" and "mon" structures. Then we'd have a clean split from top to bottom.

It is not obvious what you mean with "continue splitting things". Are you speaking about "continue splitting from v14" or "continue splitting from v15-RFC"?

I'm speaking of some future potential changes. Not proposing to do this now.

Quoted 13 lines

I think that any solution needs to consider what makes sense for resctrl as a whole instead of how to support SNC with smallest patch possible.

There should not be any changes that makes resctrl harder to understand and maintain, as exemplified by confusion introduced by a simple thing as resource name choice [1].

Quoted 5 lines

Doing that would get rid of the rdt_resources_all[] array. Replacing with individual rdt_hw_ctrl_resource and rdt_hw_mon_resource declarations for each feature.

Features found on a system would be added to a list of ctrl or list of mon resources.

Quoted 7 lines
Quoted 2 lines

Could you please elaborate what is architecturally wrong with v14 and how this new proposal addresses that?

There is nothing architecturally wrong with v14. I thought it was more complex than it needed to be. You have convinced me that my v15-RFC series, while simpler, is not a reasonable path for long-term resctrl maintainability.

I'm not sure if its helpful to describe a third approach at this point - but on the off chance its useful:
With SNC enable, the L3 monitors are unaffected, but the controls behave as if they were part of some other component in the system..

ACPI describes something called "memory side caches" [0] in the HMAT table, which are outside the CPU cache hierarchy, and are associated with a Proximity-Domain. I've heard that one of Arm's partners has built a system with MPAM controls on something like this. How would we support this - and would this be a better fit for the way SNC behaves?

I think this would be a new resource and schema, 'MSC'(?) with domain-ids using the NUMA nid. As these aren't CPU caches, they wouldn't appear in the same part of the sysfs hierarchy, and wouldn't necessarily have a cache-id.

For SNC systems, I think this would look like CMT on the L3, and CAT on the 'MSC'. Existing software wouldn't know to use the new schema, but equally wouldn't be surprised by the domain-ids being something other than the cache-id, and the controls and monitors not lining up.
Where its not quite right for SNC is sysfs may not describe a memory side cache, but one would be present in resctrl. I don't think that's a problem - unless these systems do also have a memory-side-cache that behaves differently. (where is the controls being applied at the 'near' side of the link - I don't think the difference matters)

I'm a little nervous that the SNC support looks strange if we ever add support for something like the above. Given its described in ACPI, I assume there are plenty of machines out there that look like this.

(Why aren't memory-side-caches a CPU cache? They live near the memory controller and cache based on the PA, not the CPU that issued the transaction)

Thanks,

James

[0]
https://uefi.org/specs/ACPI/6.5/05_ACPI_Software_Programming_Model.html#memory-side-cache-overview

author maintainer
#329 lore raw

RE: [PATCH v15-RFC 0/8] Add support for Sub-NUMA cluster (SNC) systems

Quoted 2 lines

With SNC enable, the L3 monitors are unaffected, but the controls behave as if they were part of some other component in the system.

I don't think of it like that. See attached picture of a single socket divided in two by SNC. [If the attachment is stripped off for those reading this via mailing lists, if you want the picture, just send me an e-mail.]

Everything in blue is node 0. Yellow for node 1.

The rectangles in the middle represent the L3 cache (12-way associative). When cores in node 0 access memory in node 0, it will be cached using the "top" half of the cache indices. Similarly for node 1 using the "bottom" half.

Here’s how each of the Intel L3 resctrl functions operate with SNC enabled:

CQM: Reports how much of your half of the L3 cache is occupied

MBM: Reports on memory traffic from your half of the cache to your memory controllers.

CAT: Still controls which ways of the cache are available for allocation (but each way has half the capacity.)

MBA: The same throttling levels applied to "blue" and "yellow" traffic (because there are only socket level controls).

Quoted 3 lines

I'm a little nervous that the SNC support looks strange if we ever add support for something like the above. Given its described in ACPI, I assume there are plenty of machines out there that look like this.

I'm also nervous as h/w designers find various ways to diverge from the old paradigm of

	socket scope == L3 cache scope == NUMA node scope

-Tony

reviewer
#330 lore raw

Re: [PATCH v15-RFC 0/8] Add support for Sub-NUMA cluster (SNC) systems

In reply to Reinette Chatre
Quoted 10 lines · Reinette Chatre

Hi Babu,

Quoted 2 lines · Babu

To be honest, I like this series more than the previous series. I always thought RDT_RESOURCE_L3_MON should have been a separate resource by itself.

Would you prefer that your "Reviewed-by" tag be removed from the previous series?

Sure. I will plan to review again the new series when Tony submits v16.

Signature · 2 lines
Thanks
Babu Moger