x86,fs/resctrl: Support L3 Smart Data Cache Injection Allocation Enforcement (SDCIAE)

Smart Data Cache Injection (SDCI) is a mechanism that enables direct insertion of data from I/O devices into the L3 cache. By directly caching data from I/O devices rather than first storing the I/O data in DRAM, SDCI reduces demands on DRAM bandwidth and reduces latency to the processor consuming the I/O data.

The SDCIAE (SDCI Allocation Enforcement) PQE feature allows system software to control the portion of the L3 cache used for SDCI.

When enabled, SDCIAE forces all SDCI lines to be placed into the L3 cache partitions identified by the highest-supported L3_MASK_n register, where n is the maximum supported CLOSID.

Add CPUID feature bit that can be used to configure SDCIAE.

The feature details are documented in APM listed below [1].
[1] AMD64 Architecture Programmer's Manual Volume 2: System Programming Publication # 24593 Revision 3.41 section 19.4.7 L3 Smart Data Cache Injection Allocation Enforcement (SDCIAE)

Link https://bugzilla.kernel.org/show_bug.cgi?id=206537

Signed-off-by Babu Moger <[email protected]>

---
v7: No changes. Fixed few conflicts in
arch/x86/include/asm/cpufeatures.h
arch/x86/kernel/cpu/scattered.c

v6: Resolved conflicts in cpufeatures.h.

v5: No changes.

v4: Resolved a minor conflict in cpufeatures.h.

v3: No changes.

v2: Added dependancy on X86_FEATURE_CAT_L3
Removed the "" in CPU feature definition.
Minor text changes.

---
 arch/x86/include/asm/cpufeatures.h | 1 +
 arch/x86/kernel/cpu/cpuid-deps.c   | 1 +
 arch/x86/kernel/cpu/scattered.c    | 1 +
 3 files changed, 3 insertions(+)

Smart Data Cache Injection (SDCI) is a mechanism that enables direct insertion of data from I/O devices into the L3 cache. By directly caching data from I/O devices rather than first storing the I/O data in DRAM, SDCI reduces demands on DRAM bandwidth and reduces latency to the processor consuming the I/O data.

The SDCIAE (SDCI Allocation Enforcement) PQE feature allows system software to control the portion of the L3 cache used for SDCI.

When enabled, SDCIAE forces all SDCI lines to be placed into the L3 cache partitions identified by the highest-supported L3_MASK_n register, where n is the maximum supported CLOSID.

Add CPUID feature bit that can be used to configure SDCIAE.

The feature details are documented in APM listed below [1].
[1] AMD64 Architecture Programmer's Manual Volume 2: System Programming Publication # 24593 Revision 3.41 section 19.4.7 L3 Smart Data Cache Injection Allocation Enforcement (SDCIAE)

Link https://bugzilla.kernel.org/show_bug.cgi?id=206537

Signed-off-by Babu Moger <[email protected]>

---

Add the command line options to enable or disable the new resctrl feature L3 Smart Data Cache Injection Allocation Enforcement (SDCIAE).

Signed-off-by Babu Moger <[email protected]>

---

---
 Documentation/admin-guide/kernel-parameters.txt | 2 +-
 arch/x86/kernel/cpu/resctrl/core.c              | 2 ++
 2 files changed, 3 insertions(+), 1 deletion(-)

Hi Babu,

Quoted 5 lines · Babu Moger

Add the command line options to enable or disable the new resctrl feature L3 Smart Data Cache Injection Allocation Enforcement (SDCIAE).

Signed-off-by Babu Moger <[email protected]>

---

...

Quoted 4 lines

---
 Documentation/admin-guide/kernel-parameters.txt | 2 +-
 arch/x86/kernel/cpu/resctrl/core.c              | 2 ++
 2 files changed, 3 insertions(+), 1 deletion(-)

Could you please also add associated update to Documentation/filesystems/resctrl.rst ? For reference, the similar ABMC change:
https://lore.kernel.org/lkml/00cd603997e3ee6a389f83aef066fe7313b1abaf.1752013061.git.babu.moger@amd.com/

Smart Data Cache Injection (SDCI) is a mechanism that enables direct insertion of data from I/O devices into the L3 cache. It can reduce the demands on DRAM bandwidth and reduces latency to the processor consuming the I/O data.

Introduce cache resource property "io_alloc_capable" that an architecture can set if a portion of the cache can be allocated for I/O traffic.

Set this property on x86 systems that support SDCIAE (L3 Smart Data Cache Injection Allocation Enforcement). This property is set only for the L3 cache resource on systems that support SDCIAE.

Signed-off-by Babu Moger <[email protected]>

---

"io_alloc" enables direct insertion of data from I/O devices into the cache.

On AMD systems, "io_alloc" feature is backed by L3 Smart Data Cache Injection Allocation Enforcement (SDCIAE). Change SDCIAE state by setting (to enable) or clearing (to disable) bit 1 of MSR L3_QOS_EXT_CFG on all logical processors within the cache domain.

Introduce architecture-specific call to enable and disable the feature.

The SDCIAE feature details are available in APM listed below [1].
[1] AMD64 Architecture Programmer's Manual Volume 2: System Programming Publication # 24593 Revision 3.41 section 19.4.7 L3 Smart Data Cache Injection Allocation Enforcement (SDCIAE)

Link https://bugzilla.kernel.org/show_bug.cgi?id=206537

Signed-off-by Babu Moger <[email protected]>

---

Hi Babu,

Quoted 18 lines · Babu Moger

"io_alloc" enables direct insertion of data from I/O devices into the cache.

On AMD systems, "io_alloc" feature is backed by L3 Smart Data Cache Injection Allocation Enforcement (SDCIAE). Change SDCIAE state by setting (to enable) or clearing (to disable) bit 1 of MSR L3_QOS_EXT_CFG on all logical processors within the cache domain.

Introduce architecture-specific call to enable and disable the feature.

The SDCIAE feature details are available in APM listed below [1].
[1] AMD64 Architecture Programmer's Manual Volume 2: System Programming Publication # 24593 Revision 3.41 section 19.4.7 L3 Smart Data Cache Injection Allocation Enforcement (SDCIAE)

Link https://bugzilla.kernel.org/show_bug.cgi?id=206537

Signed-off-by Babu Moger <[email protected]>

---

...

Quoted 1 line

arch/x86/kernel/cpu/resctrl/rdtgroup.c

Quoted 4 lines

arch/x86/kernel/cpu/resctrl/rdtgroup.c +0 -0

index 885026468440..c165ac333336 100644

The code looks good to me but rdtgroup.c is becoming bloated by being the default place for any resctrl changes. While I do know there are counter examples (and I admit I do not have a clear understanding of original intent) ctrlmondata.c is documented and supported by its content as the place for cache allocation code. Could you please move these changes to arch/x86/kernel/cpu/resctrl/ctrlmondata.c? I will try to highlight the other places where I believe the either fs/resctrl/ctrlmondata.c or arch/x86/kernel/cpu/resctrl/ctrlmondata.c is more appropriate but please as a high level consider where appropriate to move the IO alloc code to be with the rest of the cache allocation code. Looking at the series it will help keep a couple of existing static functions static and keep data structures local to cache allocation code.

When the io_alloc feature is enabled, a portion of the cache can be configured for shared use between hardware and software.

Update the bit_usage representation to reflect the io_alloc configuration.

Signed-off-by Babu Moger <[email protected]>

---
v7: New patch split from earlier patch #5.
Added resctrl_io_alloc_closid() to return max COSID.

---
 Documentation/filesystems/resctrl.rst | 20 ++++++++++-----
 fs/resctrl/rdtgroup.c                 | 37 +++++++++++++++++++++++++--
 2 files changed, 49 insertions(+), 8 deletions(-)

Documentation/filesystems/resctrl.rst +0 -4

index c7949dd44f2f..c3c412733632 100644
@@ -89,12 +89,20 @@ related to allocation:
 		must be set when writing a mask.
 
 "shareable_bits":
-		Bitmask of shareable resource with other executing
-		entities (e.g. I/O). User can use this when
-		setting up exclusive cache partitions. Note that
-		some platforms support devices that have their

fs/resctrl/rdtgroup.c +0 -0

index 77d08229d855..a2eea85aecc8 100644

fs/resctrl/rdtgroup.c +11 -2

+{
+	if (resctrl_arch_get_cdp_enabled(r->rid))
+		return resctrl_arch_get_num_closid(r) / 2  - 1;
+	else
+		return resctrl_arch_get_num_closid(r) - 1;
+}
+
 /*
  * rdt_bit_usage_show - Display current usage of resources
  *

@@ -1063,15 +1077,17 @@ static int rdt_bit_usage_show(struct kernfs_open_file *of,
 
 	cpus_read_lock();
 	mutex_lock(&rdtgroup_mutex);
-	hw_shareable = r->cache.shareable_bits;
 	list_for_each_entry(dom, &r->ctrl_domains, hdr.list) {
 		if (sep)
 			seq_putc(seq, ';');
+		hw_shareable = r->cache.shareable_bits;
 		sw_shareable = 0;
 		exclusive = 0;
 		seq_printf(seq, "%d=", dom->hdr.id);
 		for (i = 0; i < closids_supported(); i++) {
-			if (!closid_allocated(i))
+			if (!closid_allocated(i) ||
+			    (resctrl_arch_get_io_alloc_enabled(r) &&
+			     i == resctrl_io_alloc_closid(r)))
 				continue;
 			ctrl_val = resctrl_arch_get_config(r, dom, i,
 							   s->conf_type);

Hi Babu,

Quoted 4 lines · Babu Moger

When the io_alloc feature is enabled, a portion of the cache can be configured for shared use between hardware and software.

Update the bit_usage representation to reflect the io_alloc configuration.

This patch is early in the series but also relies on capabilities added later in the series. This cryptic changelog leaves a lot for the user to decipher. For example:
- How is the "io_alloc CLOSID" chosen? This is mentioned in cover letter but not here. Doing so here may help explain the hardcoding of CDP_CODE done in this patch (which seem unnecessary, more later, but a lot depends on changes that follow this patch).
- No mention that when io_alloc is enabled then an associated CLOSID will be allocated. This is done later in series but assumed to be known here where rdt_bit_usage_show() implicitly assumes that when io_alloc is enabled then the "io_alloc CLOSID" is supported AND allocated (otherwise array overrun?)

Quoted 29 lines

Signed-off-by Babu Moger <[email protected]>

---
v7: New patch split from earlier patch #5.
Added resctrl_io_alloc_closid() to return max COSID.

---
 Documentation/filesystems/resctrl.rst | 20 ++++++++++-----
 fs/resctrl/rdtgroup.c                 | 37 +++++++++++++++++++++++++--
 2 files changed, 49 insertions(+), 8 deletions(-)

Documentation/filesystems/resctrl.rst +4 -6

index c7949dd44f2f..c3c412733632 100644
@@ -89,12 +89,20 @@ related to allocation:
 		must be set when writing a mask.
 
 "shareable_bits":
-		Bitmask of shareable resource with other executing
-		entities (e.g. I/O). User can use this when
-		setting up exclusive cache partitions. Note that
-		some platforms support devices that have their
-		own settings for cache use which can over-ride
-		these bits.
+		Bitmask of shareable resource with other executing entities
+		(e.g. I/O). Applies to all instances of this resource. User
+		can use this when setting up exclusive cache partitions.
+		Note that some platforms support devices that have their

Documentation/filesystems/resctrl.rst related to allocation: · line 99

Quoted 4 lines

Documentation/filesystems/resctrl.rst +4 -0

+		own settings for cache use which can over-ride these bits.
+
+		When "io_alloc" feature is enabled, a portion of the cache
+		can be configured for shared use between hardware and software.

To help distinguish how "io_alloc" is different from "shareable_bits" this can be:
When "io_alloc" is enabled, a portion of each cache instance can be configured for shared use between hardware and software.

Please merge these two paragraphs.

Documentation/filesystems/resctrl.rst related to allocation: · line 104

Quoted 5 lines

Documentation/filesystems/resctrl.rst +4 -0

+		"bit_usage" should be used to see which portions of each cache
+		instance is configured for hardware use via "io_alloc" feature
+		because every cache instance can have its "io_alloc" bitmask
+		configured independently.

append " via "io_alloc_cbm"."?

(but io_alloc_cbm does not exist at this point ... another motivation for this patch to move later)

Documentation/filesystems/resctrl.rst related to allocation: · line 108

Quoted 4 lines

Documentation/filesystems/resctrl.rst +1 -0

+
 "bit_usage":
 		Annotated capacity bitmasks showing how all
 		instances of the resource are used. The legend is:

Please note that this "bit_usage" section contain several references to "shareable_bits" that should be updated to refer to "io_alloc_cbm" also.

With all these new terms introduced as common knowledge it is starting to look like this patch would be easier to consume later in the series.

Quoted 9 lines

fs/resctrl/rdtgroup.c +0 -0

index 77d08229d855..a2eea85aecc8 100644
@@ -1030,6 +1030,20 @@ static int rdt_shareable_bits_show(struct kernfs_open_file *of,
 	return 0;
 }
 

fs/resctrl/rdtgroup.c static int rdt_shareable_bits_show(struct kernfs_open_file *of, · line 1039

Quoted 7 lines

fs/resctrl/rdtgroup.c +7 -0

+/*
+ * resctrl_io_alloc_closid() - io_alloc feature routes I/O traffic using
+ * the highest available CLOSID. Retrieve the maximum CLOSID supported by the
+ * resource. Note that if Code Data Prioritization (CDP) is enabled, the number
+ * of available CLOSIDs is reduced by half.
+ */
+static u32 resctrl_io_alloc_closid(struct rdt_resource *r)

Please move to ctrlmondata.c.

Quoted 32 lines

fs/resctrl/rdtgroup.c +11 -2

+{
+	if (resctrl_arch_get_cdp_enabled(r->rid))
+		return resctrl_arch_get_num_closid(r) / 2  - 1;
+	else
+		return resctrl_arch_get_num_closid(r) - 1;
+}
+
 /*
  * rdt_bit_usage_show - Display current usage of resources
  *

@@ -1063,15 +1077,17 @@ static int rdt_bit_usage_show(struct kernfs_open_file *of,
 
 	cpus_read_lock();
 	mutex_lock(&rdtgroup_mutex);
-	hw_shareable = r->cache.shareable_bits;
 	list_for_each_entry(dom, &r->ctrl_domains, hdr.list) {
 		if (sep)
 			seq_putc(seq, ';');
+		hw_shareable = r->cache.shareable_bits;
 		sw_shareable = 0;
 		exclusive = 0;
 		seq_printf(seq, "%d=", dom->hdr.id);
 		for (i = 0; i < closids_supported(); i++) {
-			if (!closid_allocated(i))
+			if (!closid_allocated(i) ||
+			    (resctrl_arch_get_io_alloc_enabled(r) &&
+			     i == resctrl_io_alloc_closid(r)))
 				continue;
 			ctrl_val = resctrl_arch_get_config(r, dom, i,
 							   s->conf_type);

fs/resctrl/rdtgroup.c static int rdt_bit_usage_show(struct kernfs_open_file *of, · line 1123

Quoted 9 lines

fs/resctrl/rdtgroup.c +6 -0

@@ -1099,6 +1115,23 @@ static int rdt_bit_usage_show(struct kernfs_open_file *of,
 				break;
 			}
 		}
+
+		/*
+		 * When the "io_alloc" feature is enabled, a portion of the
+		 * cache is configured for shared use between hardware and software.
+		 */
+		if (resctrl_arch_get_io_alloc_enabled(r)) {

Here is undocumented implicit assumption that when io_alloc is enabled then the "io_alloc CLOSID" is allocated. This is also outside the closids_supported() loop which adds the other implicit assumption that if io_alloc is enabled then

its CLOSID is supported by resctrl fs. None of these concepts have been introduced so far and is not mentioned in changelog.
It is not obvious here that an io_alloc CLOSID must be supported (this is just something enforced by later patches) and also not obvious that an io_alloc CLOSID must be allocated from same "pool" as other CLOSIDs. Without a good changelog and context of later patches this is hard to understand.
These are motivations for this patch to move later in the series and then the changelog can just refer to these assumptions as fact, making it all easier to follow.

fs/resctrl/rdtgroup.c static int rdt_bit_usage_show(struct kernfs_open_file *of, · line 1131

Quoted 9 lines

fs/resctrl/rdtgroup.c +8 -0

+			if (resctrl_arch_get_cdp_enabled(r->rid))
+				ctrl_val = resctrl_arch_get_config(r, dom,
+								   resctrl_io_alloc_closid(r),
+								   CDP_CODE);
+			else
+				ctrl_val = resctrl_arch_get_config(r, dom,
+								   resctrl_io_alloc_closid(r),
+								   CDP_NONE);

This does not look necessary to me. Why not just:

		if (resctrl_arch_get_io_alloc_enabled(r)) {
			ctrl_val = resctrl_arch_get_config(r, dom,
							   resctrl_io_alloc_closid(r),
							   s->conf_type);
			hw_shareable |= ctrl_val;
		}

Since the later patches keep the CDP_CODE and CDP_DATA CBMs in sync it does not matter if the io_alloc CBM is obtained from CDP_CODE or CDP_DATA and just providing the schema's type to resctrl_arch_get_config() will have it retrieve the right CBM, no?

This may also be easier to understand/claim if this patch is later in the series.

fs/resctrl/rdtgroup.c static int rdt_bit_usage_show(struct kernfs_open_file *of, · line 1137

Quoted 8 lines

fs/resctrl/rdtgroup.c +3 -0

+			hw_shareable |= ctrl_val;
+		}
+
 		for (i = r->cache.cbm_len - 1; i >= 0; i--) {
 			pseudo_locked = dom->plr ? dom->plr->cbm : 0;
 			hwb = test_bit(i, &hw_shareable);

Reinette

"io_alloc" feature in resctrl allows direct insertion of data from I/O devices into the cache.

Introduce the 'io_alloc' resctrl file to indicate the support for the feature.

Signed-off-by Babu Moger <[email protected]>

---

---
 Documentation/filesystems/resctrl.rst | 25 +++++++++++++++++
 fs/resctrl/rdtgroup.c                 | 39 +++++++++++++++++++++++++++
 2 files changed, 64 insertions(+)

Documentation/filesystems/resctrl.rst +3 -0

index c3c412733632..354e6a00fa45 100644
@@ -143,6 +143,31 @@ related to allocation:
 			"1":
 			      Non-contiguous 1s value in CBM is supported.
 
+"io_alloc":
+		"io_alloc" enables system software to configure the portion of
+		the cache allocated for I/O traffic. File may only exist if the

fs/resctrl/rdtgroup.c +27 -0

+				 struct seq_file *seq, void *v)
+{
+	struct resctrl_schema *s = rdt_kn_parent_priv(of->kn);
+	struct rdt_resource *r = s->res;
+
+	mutex_lock(&rdtgroup_mutex);
+
+	if (r->cache.io_alloc_capable) {
+		if (resctrl_arch_get_io_alloc_enabled(r))
+			seq_puts(seq, "enabled\n");
+		else
+			seq_puts(seq, "disabled\n");
+	} else {
+		seq_puts(seq, "not supported\n");
+	}
+
+	mutex_unlock(&rdtgroup_mutex);
+
+	return 0;
+}
+
 /* rdtgroup information files for one cache resource. */
 static struct rftype res_common_files[] = {
 	{

@@ -1926,6 +1948,12 @@ static struct rftype res_common_files[] = {
 		.kf_ops		= &rdtgroup_kf_single_ops,
 		.seq_show	= rdt_thread_throttle_mode_show,
 	},
+	{
+		.name		= "io_alloc",
+		.mode		= 0444,
+		.kf_ops		= &rdtgroup_kf_single_ops,
+		.seq_show	= resctrl_io_alloc_show,
+	},
 	{
 		.name		= "max_threshold_occupancy",
 		.mode		= 0644,

fs/resctrl/rdtgroup.c +8 -0

+{
+	struct rdt_resource *r = resctrl_arch_get_resource(RDT_RESOURCE_L3);
+
+	if (r->cache.io_alloc_capable)
+		resctrl_file_fflags_init("io_alloc", RFTYPE_CTRL_INFO |
+					 RFTYPE_RES_CACHE);
+}
+
 void resctrl_file_fflags_init(const char *config, unsigned long fflags)
 {
 	struct rftype *rft;

Hi Babu,

Quoted 8 lines · Babu Moger

"io_alloc" feature in resctrl allows direct insertion of data from I/O devices into the cache.

Introduce the 'io_alloc' resctrl file to indicate the support for the feature.

Signed-off-by Babu Moger <[email protected]>

---

...

Quoted 13 lines

---
 Documentation/filesystems/resctrl.rst | 25 +++++++++++++++++
 fs/resctrl/rdtgroup.c                 | 39 +++++++++++++++++++++++++++
 2 files changed, 64 insertions(+)

Documentation/filesystems/resctrl.rst +0 -0

index c3c412733632..354e6a00fa45 100644
@@ -143,6 +143,31 @@ related to allocation:
 			"1":
 			      Non-contiguous 1s value in CBM is supported.
 

Documentation/filesystems/resctrl.rst related to allocation: · line 158

Quoted 13 lines

Documentation/filesystems/resctrl.rst +13 -0

+"io_alloc":
+		"io_alloc" enables system software to configure the portion of
+		the cache allocated for I/O traffic. File may only exist if the
+		system supports this feature on some of its cache resources.
+
+			"disabled":
+			      Portions of cache used for allocation of I/O traffic
+			      cannot be configured.
+			"enabled":
+			      Portions of cache used for allocation of I/O traffic
+			      can be configured using "io_alloc_cbm".
+			"not supported":
+			      Support not available on the system.

"Support not available on the system." -> "Support not available for this resource."?

Documentation/filesystems/resctrl.rst related to allocation: · line 163

Quoted 6 lines

Documentation/filesystems/resctrl.rst +5 -0

+
+		The underlying implementation may reduce resources available to
+		general (CPU) cache allocation. See architecture specific notes
+		below. Depending on usage requirements the feature can be enabled
+		or disabled:

"disabled:" -> "disabled."?

Documentation/filesystems/resctrl.rst related to allocation: · line 167

Quoted 5 lines

Documentation/filesystems/resctrl.rst +4 -0

+
+		On AMD systems, the io_alloc feature is supported by the L3 Smart
+		Data Cache Injection Allocation Enforcement (SDCIAE). The CLOSID for
+		io_alloc is determined by the highest CLOSID supported by the resource.

"is determined by the" -> "is the"?

To make clear connection with previous paragraph you can append something like:
When io_alloc is enabled on an AMD system the highest CLOSID is dedicated to io_alloc and no longer available for general (CPU) cache allocation.

Documentation/filesystems/resctrl.rst related to allocation: · line 169

Quoted 3 lines

Documentation/filesystems/resctrl.rst +2 -0

+		When CDP is enabled, io_alloc routes I/O traffic using the highest
+		CLOSID allocated for the instruction cache (L3CODE).

To clear up what happens with L3DATA, what do you think of appending something like:
, making this CLOSID no longer available for general (CPU) cache allocation for both the L3CODE and L3DATA resources.

Quoted 12 lines

Documentation/filesystems/resctrl.rst +1 -0

+
 Memory bandwidth(MB) subdirectory contains the following files
 with respect to allocation:
 

fs/resctrl/rdtgroup.c +0 -0

index a2eea85aecc8..d7c4417b4516 100644
@@ -1836,6 +1836,28 @@ static ssize_t mbm_local_bytes_config_write(struct kernfs_open_file *of,
 	return ret ?: nbytes;
 }
 

fs/resctrl/rdtgroup.c static ssize_t mbm_local_bytes_config_write(struct kernfs_open_file *of, · line 1839

Quoted 1 line

fs/resctrl/rdtgroup.c +1 -0

+static int resctrl_io_alloc_show(struct kernfs_open_file *of,

Please move to ctrlmondata.c

Quoted 43 lines

fs/resctrl/rdtgroup.c +27 -0

+				 struct seq_file *seq, void *v)
+{
+	struct resctrl_schema *s = rdt_kn_parent_priv(of->kn);
+	struct rdt_resource *r = s->res;
+
+	mutex_lock(&rdtgroup_mutex);
+
+	if (r->cache.io_alloc_capable) {
+		if (resctrl_arch_get_io_alloc_enabled(r))
+			seq_puts(seq, "enabled\n");
+		else
+			seq_puts(seq, "disabled\n");
+	} else {
+		seq_puts(seq, "not supported\n");
+	}
+
+	mutex_unlock(&rdtgroup_mutex);
+
+	return 0;
+}
+
 /* rdtgroup information files for one cache resource. */
 static struct rftype res_common_files[] = {
 	{

@@ -1926,6 +1948,12 @@ static struct rftype res_common_files[] = {
 		.kf_ops		= &rdtgroup_kf_single_ops,
 		.seq_show	= rdt_thread_throttle_mode_show,
 	},
+	{
+		.name		= "io_alloc",
+		.mode		= 0444,
+		.kf_ops		= &rdtgroup_kf_single_ops,
+		.seq_show	= resctrl_io_alloc_show,
+	},
 	{
 		.name		= "max_threshold_occupancy",
 		.mode		= 0644,

@@ -2095,6 +2123,15 @@ static void thread_throttle_mode_init(void)
 				 RFTYPE_CTRL_INFO | RFTYPE_RES_MB);
 }
 

fs/resctrl/rdtgroup.c static void thread_throttle_mode_init(void) · line 2126

Quoted 1 line

fs/resctrl/rdtgroup.c +1 -0

+static void io_alloc_init(void)

This function's comment can benefit from a snippet that highlights that even though this operates on hardcoded L3 resource it results in this file being visible for *all* cache resources (eg. L2 cache also), whether they support io_alloc or not.

Quoted 13 lines

fs/resctrl/rdtgroup.c +8 -0

+{
+	struct rdt_resource *r = resctrl_arch_get_resource(RDT_RESOURCE_L3);
+
+	if (r->cache.io_alloc_capable)
+		resctrl_file_fflags_init("io_alloc", RFTYPE_CTRL_INFO |
+					 RFTYPE_RES_CACHE);
+}
+
 void resctrl_file_fflags_init(const char *config, unsigned long fflags)
 {
 	struct rftype *rft;

fs/resctrl/rdtgroup.c int resctrl_init(void) · line 4326

Quoted 8 lines

fs/resctrl/rdtgroup.c +2 -0

@@ -4282,6 +4319,8 @@ int resctrl_init(void)
 
 	thread_throttle_mode_init();
 
+	io_alloc_init();
+
 	ret = resctrl_mon_resource_init();
 	if (ret)
 		return ret;

Reinette

"io_alloc" feature in resctrl enables direct insertion of data from I/O devices into the cache.

Introduce user interface to enable/disable io_alloc feature.

On AMD systems, io_alloc feature is backed by SDCIAE (L3 Smart Data Cache Injection Allocation Enforcement). When enabled, SDCIAE directs all SDCI lines to be placed into the L3 cache partitions specified by the register corresponding to the highest CLOSID supported by the resource. With CDP enabled, io_alloc routes I/O traffic using the highest CLOSID assigned to the instruction cache (L3CODE).

Signed-off-by Babu Moger <[email protected]>

---

---
 Documentation/filesystems/resctrl.rst |   8 ++
 fs/resctrl/rdtgroup.c                 | 149 +++++++++++++++++++++++++-
 2 files changed, 156 insertions(+), 1 deletion(-)

Documentation/filesystems/resctrl.rst +8 -0

index 354e6a00fa45..189c1ccb92d6 100644
@@ -157,6 +157,14 @@ related to allocation:
 			"not supported":
 			      Support not available on the system.
 
+		The feature can be modified by writing to the interface, for example:
+
+		To enable:
+			# echo 1 > /sys/fs/resctrl/info/L3/io_alloc
+
+		To disable:
+			# echo 0 > /sys/fs/resctrl/info/L3/io_alloc
+
 		The underlying implementation may reduce resources available to
 		general (CPU) cache allocation. See architecture specific notes
 		below. Depending on usage requirements the feature can be enabled

fs/resctrl/rdtgroup.c +1 -0

index d7c4417b4516..06c854caa55c 100644
@@ -70,6 +70,7 @@ static struct seq_buf last_cmd_status;
 static char last_cmd_status_buf[512];
 
 static int rdtgroup_setup_root(struct rdt_fs_context *ctx);
+static int rdtgroup_init_cat(struct resctrl_schema *s, u32 closid);
 
 static void rdtgroup_destroy_root(void);
 

@@ -232,6 +233,16 @@ bool closid_allocated(unsigned int closid)
 	return !test_bit(closid, closid_free_map);
 }

fs/resctrl/rdtgroup.c +1 -0

+
 /**
  * rdtgroup_mode_by_closid - Return mode of resource group with closid
  * @closid: closid if the resource group

fs/resctrl/rdtgroup.c +4 -0

+{
+	return io_alloc_closid < closids_supported();
+}
+
 /*
  * resctrl_io_alloc_closid() - io_alloc feature routes I/O traffic using
  * the highest available CLOSID. Retrieve the maximum CLOSID supported by the

fs/resctrl/rdtgroup.c +52 -0

+{
+	struct resctrl_schema *schema;
+
+	list_for_each_entry(schema, &resctrl_schema_all, list) {
+		if (schema->conf_type == type)
+			return schema;
+	}
+
+	return NULL;
+}
+
+static ssize_t resctrl_io_alloc_write(struct kernfs_open_file *of, char *buf,
+				      size_t nbytes, loff_t off)
+{
+	struct resctrl_schema *s = rdt_kn_parent_priv(of->kn);
+	enum resctrl_conf_type peer_type;
+	struct rdt_resource *r = s->res;
+	struct resctrl_schema *peer_s;
+	char const *grp_name;
+	u32 io_alloc_closid;
+	bool enable;
+	int ret;
+
+	ret = kstrtobool(buf, &enable);
+	if (ret)
+		return ret;
+
+	cpus_read_lock();
+	mutex_lock(&rdtgroup_mutex);
+
+	rdt_last_cmd_clear();
+
+	if (!r->cache.io_alloc_capable) {
+		rdt_last_cmd_printf("io_alloc is not supported on %s\n", s->name);
+		ret = -ENODEV;
+		goto out_unlock;
+	}
+
+	io_alloc_closid = resctrl_io_alloc_closid(r);
+	if (!resctrl_io_alloc_closid_supported(io_alloc_closid)) {
+		rdt_last_cmd_printf("io_alloc CLOSID (ctrl_hw_id) %d is not available\n",
+				    io_alloc_closid);
+		ret = -EINVAL;
+		goto out_unlock;
+	}
+
+	/* If the feature is already up to date, no action is needed. */
+	if (resctrl_arch_get_io_alloc_enabled(r) == enable)
+		goto out_unlock;
+
+	if (enable) {
+		if (!resctrl_io_alloc_closid_alloc(io_alloc_closid)) {

fs/resctrl/rdtgroup.c +28 -0

+			peer_type = resctrl_peer_type(s->conf_type);
+			peer_s = resctrl_get_schema(peer_type);
+			if (peer_s)
+				ret = resctrl_io_alloc_init_cat(r, peer_s, io_alloc_closid);
+		}
+
+		if (!ret)
+			ret = resctrl_io_alloc_init_cat(r, s, io_alloc_closid);
+
+		if (ret) {
+			rdt_last_cmd_puts("Failed to initialize io_alloc allocations\n");
+			resctrl_io_alloc_closid_free(io_alloc_closid);
+			goto out_unlock;
+		}
+
+	} else {
+		resctrl_io_alloc_closid_free(io_alloc_closid);
+	}
+
+	ret = resctrl_arch_io_alloc_enable(r, enable);
+
+out_unlock:
+	mutex_unlock(&rdtgroup_mutex);
+	cpus_read_unlock();
+
+	return ret ?: nbytes;
+}
+
 /* rdtgroup information files for one cache resource. */
 static struct rftype res_common_files[] = {
 	{

@@ -1950,9 +2096,10 @@ static struct rftype res_common_files[] = {
 	},

Hi Babu,

Quoted 3 lines · Babu Moger

"io_alloc" feature in resctrl enables direct insertion of data from I/O devices into the cache.

Quoted 1 line

Introduce user interface to enable/disable io_alloc feature.

I think it is worth a mention *why* a user may want to disable this feature and why is not just always enabled. Here it can be highlighted that this feature may take resources (CLOSID) away from general (CPU) cache allocation and since this may be scarce enabling user to disable this feature supports different use cases.

Quoted 7 lines

On AMD systems, io_alloc feature is backed by SDCIAE (L3 Smart Data Cache Injection Allocation Enforcement). When enabled, SDCIAE directs all SDCI lines to be placed into the L3 cache partitions specified by the register corresponding to the highest CLOSID supported by the resource. With CDP enabled, io_alloc routes I/O traffic using the highest CLOSID assigned to the instruction cache (L3CODE).

This is a lot of architecture specific text for a resctrl fs patch ... I think you are trying to motivate the resctrl fs implementation. Similar motivation as proposed for cover letter can be used here to help explain the implementation choices.

Quoted 4 lines

Signed-off-by Babu Moger <[email protected]>

---

...

Quoted 39 lines

---
  Documentation/filesystems/resctrl.rst |   8 ++
  fs/resctrl/rdtgroup.c                 | 149 +++++++++++++++++++++++++-
  2 files changed, 156 insertions(+), 1 deletion(-)

Documentation/filesystems/resctrl.rst +8 -0

index 354e6a00fa45..189c1ccb92d6 100644
@@ -157,6 +157,14 @@ related to allocation:
  			"not supported":
  			      Support not available on the system.
 
+		The feature can be modified by writing to the interface, for example:
+
+		To enable:
+			# echo 1 > /sys/fs/resctrl/info/L3/io_alloc
+
+		To disable:
+			# echo 0 > /sys/fs/resctrl/info/L3/io_alloc
+
  		The underlying implementation may reduce resources available to
  		general (CPU) cache allocation. See architecture specific notes
  		below. Depending on usage requirements the feature can be enabled

fs/resctrl/rdtgroup.c +1 -0

index d7c4417b4516..06c854caa55c 100644
@@ -70,6 +70,7 @@ static struct seq_buf last_cmd_status;
  static char last_cmd_status_buf[512];
 
  static int rdtgroup_setup_root(struct rdt_fs_context *ctx);
+static int rdtgroup_init_cat(struct resctrl_schema *s, u32 closid);
 
  static void rdtgroup_destroy_root(void);
 

@@ -232,6 +233,16 @@ bool closid_allocated(unsigned int closid)
  	return !test_bit(closid, closid_free_map);
  }

fs/resctrl/rdtgroup.c bool closid_allocated(unsigned int closid) · line 244

Quoted 10 lines

fs/resctrl/rdtgroup.c +9 -0

 
+static bool resctrl_io_alloc_closid_alloc(u32 io_alloc_closid)
+{
+	return __test_and_clear_bit(io_alloc_closid, closid_free_map);
+}
+
+static void resctrl_io_alloc_closid_free(u32 io_alloc_closid)
+{
+	closid_free(io_alloc_closid);
+}

I do not think these should be helpers/wrappers with a separate namespace. It will make the code easier to understand if it is clear that the "io_alloc" CLOSID is allocated from the same "pool" as the CLOSID for control groups.

I would thus propose a specific, for example closid_alloc_fixed(u32 closid) helper, and just call closid_free() directly.

Quoted 8 lines

fs/resctrl/rdtgroup.c +1 -0

+
  /**
   * rdtgroup_mode_by_closid - Return mode of resource group with closid
   * @closid: closid if the resource group

@@ -1030,6 +1041,16 @@ static int rdt_shareable_bits_show(struct kernfs_open_file *of,
  	return 0;
  }
 

fs/resctrl/rdtgroup.c static int rdt_shareable_bits_show(struct kernfs_open_file *of, · line 1049

Quoted 6 lines

fs/resctrl/rdtgroup.c +6 -0

+/*
+ * resctrl_io_alloc_closid_supported() - io_alloc feature utilizes the
+ * highest CLOSID value to direct I/O traffic. Ensure that io_alloc_closid
+ * is in the supported range.
+ */
+static bool resctrl_io_alloc_closid_supported(u32 io_alloc_closid)

Please move to ctrlmondata.c

Quoted 11 lines

fs/resctrl/rdtgroup.c +4 -0

+{
+	return io_alloc_closid < closids_supported();
+}
+
  /*
   * resctrl_io_alloc_closid() - io_alloc feature routes I/O traffic using
   * the highest available CLOSID. Retrieve the maximum CLOSID supported by the

@@ -1858,6 +1879,131 @@ static int resctrl_io_alloc_show(struct kernfs_open_file *of,
  	return 0;
  }
 

fs/resctrl/rdtgroup.c static int resctrl_io_alloc_show(struct kernfs_open_file *of, · line 1886

Quoted 5 lines

fs/resctrl/rdtgroup.c +5 -0

+/*
+ * Initialize io_alloc CLOSID cache resource CBM with all usable (shared
+ * and unused) cache portions.
+ */
+static int resctrl_io_alloc_init_cat(struct rdt_resource *r,

Mixing two features (io_alloc and CAT) in the function name is confusing. How about resctrl_io_alloc_init_cbm() and move to ctrlmondata.c?

fs/resctrl/rdtgroup.c static int resctrl_io_alloc_show(struct kernfs_open_file *of, · line 1887

Quoted 2 lines

fs/resctrl/rdtgroup.c +1 -0

+				     struct resctrl_schema *s, u32 closid)

No need to provide resource as parameter, it can be determined from schema.

fs/resctrl/rdtgroup.c static int resctrl_io_alloc_show(struct kernfs_open_file *of, · line 1895

Quoted 9 lines

fs/resctrl/rdtgroup.c +8 -0

+{
+	int ret;
+
+	rdt_staged_configs_clear();
+
+	ret = rdtgroup_init_cat(s, closid);
+	if (ret < 0)
+		goto out;

More below, but I think this flow can be simplified by moving the logic handling CDP here. If CDP is enabled for the resource then a successful rdtgroup_init_cat() can just be followed by a snippet that copies the staged config of the CDP type to the staged config of its peer type. Their CBMs are supposed to be identical so there is no reason for all the rdtgroup_init_cat() processing to be repeated. resctrl_arch_update_domains() can handle updating both in a single call.

fs/resctrl/rdtgroup.c static int resctrl_io_alloc_show(struct kernfs_open_file *of, · line 1904

Quoted 10 lines

fs/resctrl/rdtgroup.c +9 -0

+
+	ret = resctrl_arch_update_domains(r, closid);
+
+out:
+	rdt_staged_configs_clear();
+	return ret;
+}
+
+static const char *rdtgroup_name_by_closid(int closid)

This seems generic enough and appropriate for rdtgroup.c

fs/resctrl/rdtgroup.c static int resctrl_io_alloc_show(struct kernfs_open_file *of, · line 1916

Quoted 12 lines

fs/resctrl/rdtgroup.c +12 -0

+{
+	struct rdtgroup *rdtgrp;
+
+	list_for_each_entry(rdtgrp, &rdt_all_groups, rdtgroup_list) {
+		if (rdtgrp->closid == closid)
+			return rdt_kn_name(rdtgrp->kn);
+	}
+
+	return NULL;
+}
+
+static struct resctrl_schema *resctrl_get_schema(enum resctrl_conf_type type)

This also seems generic enough and appropriate for rdtgroup.c

Quoted 57 lines

fs/resctrl/rdtgroup.c +57 -0

+{
+	struct resctrl_schema *schema;
+
+	list_for_each_entry(schema, &resctrl_schema_all, list) {
+		if (schema->conf_type == type)
+			return schema;
+	}
+
+	return NULL;
+}
+
+static ssize_t resctrl_io_alloc_write(struct kernfs_open_file *of, char *buf,
+				      size_t nbytes, loff_t off)
+{
+	struct resctrl_schema *s = rdt_kn_parent_priv(of->kn);
+	enum resctrl_conf_type peer_type;
+	struct rdt_resource *r = s->res;
+	struct resctrl_schema *peer_s;
+	char const *grp_name;
+	u32 io_alloc_closid;
+	bool enable;
+	int ret;
+
+	ret = kstrtobool(buf, &enable);
+	if (ret)
+		return ret;
+
+	cpus_read_lock();
+	mutex_lock(&rdtgroup_mutex);
+
+	rdt_last_cmd_clear();
+
+	if (!r->cache.io_alloc_capable) {
+		rdt_last_cmd_printf("io_alloc is not supported on %s\n", s->name);
+		ret = -ENODEV;
+		goto out_unlock;
+	}
+
+	io_alloc_closid = resctrl_io_alloc_closid(r);
+	if (!resctrl_io_alloc_closid_supported(io_alloc_closid)) {
+		rdt_last_cmd_printf("io_alloc CLOSID (ctrl_hw_id) %d is not available\n",
+				    io_alloc_closid);
+		ret = -EINVAL;
+		goto out_unlock;
+	}
+
+	/* If the feature is already up to date, no action is needed. */
+	if (resctrl_arch_get_io_alloc_enabled(r) == enable)
+		goto out_unlock;
+
+	if (enable) {
+		if (!resctrl_io_alloc_closid_alloc(io_alloc_closid)) {
+			grp_name = rdtgroup_name_by_closid(io_alloc_closid);
+			WARN_ON_ONCE(!grp_name);
+			rdt_last_cmd_printf("CLOSID (ctrl_hw_id) %d for io_alloc is used by %s group\n",
+					    io_alloc_closid, grp_name ? grp_name : "another");
+			ret = -ENOSPC;

fs/resctrl/rdtgroup.c static int resctrl_io_alloc_show(struct kernfs_open_file *of, · line 1978

Quoted 5 lines

fs/resctrl/rdtgroup.c +5 -0

+			goto out_unlock;
+		}
+
+		/* Initialize schema for both CDP_DATA and CDP_CODE when CDP is enabled */
+		if (resctrl_arch_get_cdp_enabled(r->rid)) {

I think this block can be removed to simplify the flow by moving the CDP handling to resctrl_io_alloc_init_cat().

Quoted 38 lines

fs/resctrl/rdtgroup.c +29 -1

+			peer_type = resctrl_peer_type(s->conf_type);
+			peer_s = resctrl_get_schema(peer_type);
+			if (peer_s)
+				ret = resctrl_io_alloc_init_cat(r, peer_s, io_alloc_closid);
+		}
+
+		if (!ret)
+			ret = resctrl_io_alloc_init_cat(r, s, io_alloc_closid);
+
+		if (ret) {
+			rdt_last_cmd_puts("Failed to initialize io_alloc allocations\n");
+			resctrl_io_alloc_closid_free(io_alloc_closid);
+			goto out_unlock;
+		}
+
+	} else {
+		resctrl_io_alloc_closid_free(io_alloc_closid);
+	}
+
+	ret = resctrl_arch_io_alloc_enable(r, enable);
+
+out_unlock:
+	mutex_unlock(&rdtgroup_mutex);
+	cpus_read_unlock();
+
+	return ret ?: nbytes;
+}
+
  /* rdtgroup information files for one cache resource. */
  static struct rftype res_common_files[] = {
  	{

@@ -1950,9 +2096,10 @@ static struct rftype res_common_files[] = {
  	},
  	{
  		.name		= "io_alloc",
-		.mode		= 0444,
+		.mode		= 0644,

fs/resctrl/rdtgroup.c static struct rftype res_common_files[] = { · line 2105

Quoted 6 lines

fs/resctrl/rdtgroup.c +1 -0

  		.kf_ops		= &rdtgroup_kf_single_ops,
  		.seq_show	= resctrl_io_alloc_show,
+		.write          = resctrl_io_alloc_write,
  	},
  	{
  		.name		= "max_threshold_occupancy",

Reinette

Hi Reinette,

Quoted 6 lines · Reinette Chatre

Hi Babu,

Quoted 3 lines · Babu Moger

"io_alloc" feature in resctrl enables direct insertion of data from I/O devices into the cache.

Quoted 7 lines

Quoted 1 line

Introduce user interface to enable/disable io_alloc feature.

I think it is worth a mention *why* a user may want to disable this feature and why is not just always enabled. Here it can be highlighted that this feature may take resources (CLOSID) away from general (CPU) cache allocation and since this may be scarce enabling user to disable this feature supports different use cases.

Sure.

Quoted 8 lines

Quoted 7 lines

On AMD systems, io_alloc feature is backed by SDCIAE (L3 Smart Data Cache Injection Allocation Enforcement). When enabled, SDCIAE directs all SDCI lines to be placed into the L3 cache partitions specified by the register corresponding to the highest CLOSID supported by the resource. With CDP enabled, io_alloc routes I/O traffic using the highest CLOSID assigned to the instruction cache (L3CODE).

Quoted 4 lines

This is a lot of architecture specific text for a resctrl fs patch ... I think you are trying to motivate the resctrl fs implementation. Similar motivation as proposed for cover letter can be used here to help explain the implementation choices.

Updated the whole changelog.

fs/resctrl: Add user interface to enable/disable io_alloc feature

"io_alloc" feature in resctrl enables direct insertion of data from I/O devices into the cache.

Introduce user interface to enable/disable io_alloc feature.

On AMD systems, when io_alloc is enabled, the highest CLOSID is reserved exclusively for I/O allocation traffic and is no longer available for general CPU cache allocation. This feature is disabled by default. Users

are encouraged to enable it only when running workloads that can benefit from this functionality.

Since CLOSIDs are managed by resctrl fs, it is least invasive to make the "io_alloc is supported by maximum supported CLOSID" part of the initial resctrl fs support for io_alloc. Take care not to expose this use of CLOSID for io_alloc to user space so that this is not required from other architectures that may support io_alloc differently in the future.

Hi Babu,

Quoted 36 lines · Babu

Hi Reinette,

Quoted 6 lines · Reinette Chatre

Hi Babu,

Quoted 3 lines · Babu Moger

"io_alloc" feature in resctrl enables direct insertion of data from I/O devices into the cache.

Quoted 7 lines

Quoted 1 line

Introduce user interface to enable/disable io_alloc feature.

I think it is worth a mention *why* a user may want to disable this feature and why is not just always enabled. Here it can be highlighted that this feature may take resources (CLOSID) away from general (CPU) cache allocation and since this may be scarce enabling user to disable this feature supports different use cases.

Sure.

Quoted 8 lines

Quoted 7 lines

On AMD systems, io_alloc feature is backed by SDCIAE (L3 Smart Data Cache Injection Allocation Enforcement). When enabled, SDCIAE directs all SDCI lines to be placed into the L3 cache partitions specified by the register corresponding to the highest CLOSID supported by the resource. With CDP enabled, io_alloc routes I/O traffic using the highest CLOSID assigned to the instruction cache (L3CODE).

Quoted 4 lines

This is a lot of architecture specific text for a resctrl fs patch ... I think you are trying to motivate the resctrl fs implementation. Similar motivation as proposed for cover letter can be used here to help explain the implementation choices.

Updated the whole changelog.

fs/resctrl: Add user interface to enable/disable io_alloc feature

Quoted 4 lines

"io_alloc" feature in resctrl enables direct insertion of data from I/O devices into the cache.

Introduce user interface to enable/disable io_alloc feature.

The solution should be at end of changelog after description of problem it solves.

Quoted 4 lines

On AMD systems, when io_alloc is enabled, the highest CLOSID is reserved exclusively for I/O allocation traffic and is no longer available for general CPU cache allocation. This feature is disabled by default. Users

Changelog should always be in imperative tone and problem and solution should be in separate paragraphs (above paragraph mixes problem and solution).

For example, "Disable "io_alloc" feature by default to ensure all resources are available for general CPU cache allocation. ..." Although I do not think this is accurate since this patch does not do this?

Quoted 10 lines

are encouraged to enable it only when running workloads that can benefit from this functionality.

Since CLOSIDs are managed by resctrl fs, it is least invasive to make the "io_alloc is supported by maximum supported CLOSID" part of the initial resctrl fs support for io_alloc. Take care not to expose this use of CLOSID for io_alloc to user space so that this is not required from other architectures that may support io_alloc differently in the future.

The changelog requirements I refer to are documented in "Changelog" section of Documentation/process/maintainer-tip.rst. I feel like this should be familiar to you by now.

The io_alloc feature in resctrl enables system software to configure the portion of the cache allocated for I/O traffic.

Add "io_alloc_cbm" resctrl file to display CBMs (Capacity Bit Mask) o fio_alloc feature.

The CBM interface file io_alloc_cbm will reside in the info directory

(e.g., /sys/fs/resctrl/info/L3/). Displaying the resource name is not necessary. Pass the resource name to show_doms() and print it only if the name is valid. For io_alloc, pass NULL to suppress printing the resource name.

When CDP is enabled, io_alloc routes traffic using the highest CLOSID associated with an L3CODE resource. However, CBMs can be accessed via either L3CODE or L3DATA resources.

Signed-off-by Babu Moger <[email protected]>

---

---
 Documentation/filesystems/resctrl.rst | 18 +++++++++++
 fs/resctrl/ctrlmondata.c              |  8 +++--
 fs/resctrl/internal.h                 |  2 ++
 fs/resctrl/rdtgroup.c                 | 43 ++++++++++++++++++++++++++-
 4 files changed, 67 insertions(+), 4 deletions(-)

Documentation/filesystems/resctrl.rst +0 -0

index 189c1ccb92d6..c22a9dd667cd 100644

Documentation/filesystems/resctrl.rst +15 -0

+
+		CBMs are displayed in the following format:
+
+			<cache_id0>=<cbm>;<cache_id1>=<cbm>;...
+
+		Example::
+
+			# cat /sys/fs/resctrl/info/L3/io_alloc_cbm
+			0=ffff;1=ffff
+
+		When CDP is enabled "io_alloc_cbm" associated with the DATA and CODE
+		resources may reflect the same values. For example, values read from and
+		written to /sys/fs/resctrl/info/L3DATA/io_alloc_cbm may be reflected by
+		/sys/fs/resctrl/info/L3CODE/io_alloc_cbm and vice versa.
+
 Memory bandwidth(MB) subdirectory contains the following files
 with respect to allocation:
 

fs/resctrl/ctrlmondata.c +0 -0

index d98e0d2de09f..e78828b0408a 100644

Hi Babu,

Quoted 5 lines · Babu Moger

The io_alloc feature in resctrl enables system software to configure the portion of the cache allocated for I/O traffic.

Add "io_alloc_cbm" resctrl file to display CBMs (Capacity Bit Mask) o fio_alloc feature.

"o fio_alloc" -> "of the io_alloc"

Quoted 2 lines

The CBM interface file io_alloc_cbm will reside in the info directory

"will reside" -> "resides"

Quoted 8 lines

(e.g., /sys/fs/resctrl/info/L3/). Displaying the resource name is not necessary. Pass the resource name to show_doms() and print it only if the name is valid. For io_alloc, pass NULL to suppress printing the resource name.

When CDP is enabled, io_alloc routes traffic using the highest CLOSID associated with an L3CODE resource. However, CBMs can be accessed via either L3CODE or L3DATA resources.

Seems like something is missing here since this is motivation for a behavior but no mention of the behavior being motivated. Also please use imperative tone.

Quoted 4 lines

Signed-off-by Babu Moger <[email protected]>

---

...

Quoted 15 lines

---
 Documentation/filesystems/resctrl.rst | 18 +++++++++++
 fs/resctrl/ctrlmondata.c              |  8 +++--
 fs/resctrl/internal.h                 |  2 ++
 fs/resctrl/rdtgroup.c                 | 43 ++++++++++++++++++++++++++-
 4 files changed, 67 insertions(+), 4 deletions(-)

Documentation/filesystems/resctrl.rst +0 -0

index 189c1ccb92d6..c22a9dd667cd 100644
@@ -176,6 +176,24 @@ related to allocation:
 		When CDP is enabled, io_alloc routes I/O traffic using the highest
 		CLOSID allocated for the instruction cache (L3CODE).
 

Documentation/filesystems/resctrl.rst related to allocation: · line 181

Quoted 3 lines

Documentation/filesystems/resctrl.rst +3 -0

+"io_alloc_cbm":
+		CBMs(Capacity Bit Masks) that describe the portions of cache instances
+		to which I/O traffic from supported I/O devices are routed.

"are routed" -> "are routed when "io_alloc" is enabled"?

Quoted 26 lines

Documentation/filesystems/resctrl.rst +15 -0

+
+		CBMs are displayed in the following format:
+
+			<cache_id0>=<cbm>;<cache_id1>=<cbm>;...
+
+		Example::
+
+			# cat /sys/fs/resctrl/info/L3/io_alloc_cbm
+			0=ffff;1=ffff
+
+		When CDP is enabled "io_alloc_cbm" associated with the DATA and CODE
+		resources may reflect the same values. For example, values read from and
+		written to /sys/fs/resctrl/info/L3DATA/io_alloc_cbm may be reflected by
+		/sys/fs/resctrl/info/L3CODE/io_alloc_cbm and vice versa.
+
 Memory bandwidth(MB) subdirectory contains the following files
 with respect to allocation:
 

fs/resctrl/ctrlmondata.c +0 -0

index d98e0d2de09f..e78828b0408a 100644
@@ -381,7 +381,8 @@ ssize_t rdtgroup_schemata_write(struct kernfs_open_file *of,
 	return ret ?: nbytes;
 }
 

fs/resctrl/ctrlmondata.c ssize_t rdtgroup_schemata_write(struct kernfs_open_file *of, · line 384

Quoted 2 lines

fs/resctrl/ctrlmondata.c +1 -1

-static void show_doms(struct seq_file *s, struct resctrl_schema *schema, int closid)
+void show_doms(struct seq_file *s, struct resctrl_schema *schema, char *resource_name,

show_doms() can remain static within ctrlmondata.c by moving resctrl_io_alloc_cbm_show() to ctrlmondata.c

The functions parse_cbm() and parse_bw() require mode and CLOSID to

validate the Capacity Bit Mask (CBM). It is passed through struct

rdtgroup in rdt_parse_data.

This can be simplified by passing the mode and closid directly, instead of

through the rdtgroup struct. Doing so also facilitates calling parse_cbm() to verify the CBM within the io_alloc feature, since io_alloc does not have rdtgroup context.

Signed-off-by Babu Moger <[email protected]>

---

---
 fs/resctrl/ctrlmondata.c | 29 +++++++++++++----------------
 fs/resctrl/internal.h    |  6 ++++++
 2 files changed, 19 insertions(+), 16 deletions(-)

fs/resctrl/ctrlmondata.c +0 -0

index e78828b0408a..5c16557fb7a8 100644

fs/resctrl/ctrlmondata.c +1 -1

@@ -171,7 +167,7 @@ static int parse_cbm(struct rdt_parse_data *data, struct resctrl_schema *s,
 	 * Cannot set up more than one pseudo-locked region in a cache
 	 * hierarchy.
 	 */
-	if (rdtgrp->mode == RDT_MODE_PSEUDO_LOCKSETUP &&
+	if (mode == RDT_MODE_PSEUDO_LOCKSETUP &&
 	    rdtgroup_pseudo_locked_in_hierarchy(d)) {
 		rdt_last_cmd_puts("Pseudo-locked region in hierarchy\n");
 		return -EINVAL;

Hi Babu,

Quoted 1 line · Babu Moger

The functions parse_cbm() and parse_bw() require mode and CLOSID to

No need to say "function" when using (). Also, drop parse_bw(), since it does not validate CBMs.

Quoted 2 lines

validate the Capacity Bit Mask (CBM). It is passed through struct

"passed through" -> "passed via"?

Quoted 1 line

rdtgroup in rdt_parse_data.

"rdt_parse_data" -> "struct rdt_parse_data"

Quoted 2 lines

This can be simplified by passing the mode and closid directly, instead of

closid -> CLOSID

Quoted 3 lines

through the rdtgroup struct. Doing so also facilitates calling parse_cbm() to verify the CBM within the io_alloc feature, since io_alloc does not have rdtgroup context.

Above notes that "simplification" is the primary motivation but I do not think this change qualifies as a "simplification". How about second paragraph changed to something like:

The io_alloc feature also uses CBMs to indicate which portions of cache are allocated for I/O traffic. The CBMs are provided by user space and need to be validated the same as CBMs provided for general (CPU) cache allocation. parse_cbm() cannot be used as-is since io_alloc does not have rdtgroup context.

Pass the mode and CLOSID directly to parse_cbm() via struct rdt_parse_data instead of through the rdtgroup struct to facilitate calling parse_cbm() to verify the CBM of the io_alloc feature.

(please feel free to improve)

Quoted 19 lines

Signed-off-by Babu Moger <[email protected]>

---

---
 fs/resctrl/ctrlmondata.c | 29 +++++++++++++----------------
 fs/resctrl/internal.h    |  6 ++++++
 2 files changed, 19 insertions(+), 16 deletions(-)

fs/resctrl/ctrlmondata.c +0 -0

index e78828b0408a..5c16557fb7a8 100644
@@ -23,11 +23,6 @@
 
 #include "internal.h"
 

fs/resctrl/ctrlmondata.c line 30

Quoted 5 lines

fs/resctrl/ctrlmondata.c +0 -5

-struct rdt_parse_data {
-	struct rdtgroup		*rdtgrp;
-	char			*buf;
-};
-

This patch is only about replacing rdtgroup with mode and CLOSID, there is no motivation for relocating the structure declaration. This looks to be a change needed by following patch but is another change that becomes unnecessary if the io_alloc code, specifically resctrl_io_alloc_cbm_write() and resctrl_io_alloc_parse_line() from next patch, are moved to ctrlmondata.c.

...

Quoted 13 lines

fs/resctrl/ctrlmondata.c +1 -1

@@ -171,7 +167,7 @@ static int parse_cbm(struct rdt_parse_data *data, struct resctrl_schema *s,
 	 * Cannot set up more than one pseudo-locked region in a cache
 	 * hierarchy.
 	 */
-	if (rdtgrp->mode == RDT_MODE_PSEUDO_LOCKSETUP &&
+	if (mode == RDT_MODE_PSEUDO_LOCKSETUP &&
 	    rdtgroup_pseudo_locked_in_hierarchy(d)) {
 		rdt_last_cmd_puts("Pseudo-locked region in hierarchy\n");
 		return -EINVAL;

@@ -180,9 +176,9 @@ static int parse_cbm(struct rdt_parse_data *data, struct resctrl_schema *s,
 	if (!cbm_validate(data->buf, &cbm_val, r))
 		return -EINVAL;
 

fs/resctrl/ctrlmondata.c static int parse_cbm(struct rdt_parse_data *data, struct resctrl_schema *s, · line 181

Quoted 6 lines

fs/resctrl/ctrlmondata.c +3 -3

-	if ((rdtgrp->mode == RDT_MODE_EXCLUSIVE ||
-	     rdtgrp->mode == RDT_MODE_SHAREABLE) &&
-	    rdtgroup_cbm_overlaps_pseudo_locked(d, cbm_val)) {
+	if ((mode == RDT_MODE_EXCLUSIVE ||
+	     mode == RDT_MODE_SHAREABLE) &&
+	     rdtgroup_cbm_overlaps_pseudo_locked(d, cbm_val)) {

Please fix alignment.

fs/resctrl/ctrlmondata.c static int parse_cbm(struct rdt_parse_data *data, struct resctrl_schema *s, · line 184

Quoted 4 lines

fs/resctrl/ctrlmondata.c +0 -0

 		rdt_last_cmd_puts("CBM overlaps with pseudo-locked region\n");
 		return -EINVAL;
 	}

Reinette

"io_alloc" feature enables direct insertion of data from I/O devices into the cache. By directly caching data from I/O devices rather than first storing the I/O data in DRAM, it reduces the demands on DRAM bandwidth and reduces latency to the processor consuming the I/O data.

Provide the interface to modify io_alloc CBMs (Capacity Bit Mask) when

feature is enabled. Update the CBMs for both CDP_DATA and CDP_CODE when CDP is enabled.

Signed-off-by Babu Moger <[email protected]>

---

---
 Documentation/filesystems/resctrl.rst |   8 ++
 fs/resctrl/ctrlmondata.c              |   4 +-
 fs/resctrl/internal.h                 |   2 +
 fs/resctrl/rdtgroup.c                 | 112 +++++++++++++++++++++++++-
 4 files changed, 123 insertions(+), 3 deletions(-)

Documentation/filesystems/resctrl.rst +8 -0

index c22a9dd667cd..b9c3ffdec27f 100644
@@ -189,6 +189,14 @@ related to allocation:
 			# cat /sys/fs/resctrl/info/L3/io_alloc_cbm
 			0=ffff;1=ffff
 
+		CBMs can be configured by writing to the interface.
+
+		Example::
+
+			# echo 1=FF > /sys/fs/resctrl/info/L3/io_alloc_cbm
+			# cat /sys/fs/resctrl/info/L3/io_alloc_cbm
+			0=ffff;1=00ff
+
 		When CDP is enabled "io_alloc_cbm" associated with the DATA and CODE
 		resources may reflect the same values. For example, values read from and
 		written to /sys/fs/resctrl/info/L3DATA/io_alloc_cbm may be reflected by

fs/resctrl/ctrlmondata.c +0 -0

index 5c16557fb7a8..23c72a5ac043 100644

fs/resctrl/ctrlmondata.c +0 -0

 {
 	enum rdtgrp_mode mode = data->mode;
 	struct resctrl_staged_config *cfg;

fs/resctrl/internal.h +2 -0

index ab76a1e2e679..011564e69ed5 100644
@@ -382,6 +382,8 @@ bool closid_allocated(unsigned int closid);
 int resctrl_find_cleanest_closid(void);
 void show_doms(struct seq_file *s, struct resctrl_schema *schema,
 	       char *name, int closid);
+int parse_cbm(struct rdt_parse_data *data, struct resctrl_schema *s,
+	      struct rdt_ctrl_domain *d);
 
 #ifdef CONFIG_RESCTRL_FS_PSEUDO_LOCK
 int rdtgroup_locksetup_enter(struct rdtgroup *rdtgrp);

fs/resctrl/rdtgroup.c +0 -0

index 008657e43656..1093ec5e385b 100644

fs/resctrl/rdtgroup.c +23 -0

+{
+	struct rdt_parse_data data;
+	struct rdt_ctrl_domain *d;
+	char *dom = NULL, *id;
+	unsigned long dom_id;
+
+next:
+	if (!line || line[0] == '\0')
+		return 0;
+
+	dom = strsep(&line, ";");
+	id = strsep(&dom, "=");
+	if (!dom || kstrtoul(id, 10, &dom_id)) {
+		rdt_last_cmd_puts("Missing '=' or non-numeric domain\n");
+		return -EINVAL;
+	}
+
+	dom = strim(dom);
+	list_for_each_entry(d, &r->ctrl_domains, hdr.list) {
+		if (d->hdr.id == dom_id) {
+			data.buf = dom;
+			data.mode = RDT_MODE_SHAREABLE;
+			data.closid = closid;

fs/resctrl/rdtgroup.c +10 -0

+{
+	struct resctrl_schema *s = rdt_kn_parent_priv(of->kn);
+	enum resctrl_conf_type peer_type;
+	struct rdt_resource *r = s->res;
+	struct resctrl_schema *peer_s;
+	u32 io_alloc_closid;
+	char *peer_buf;
+	int ret = 0;
+
+	/* Valid input requires a trailing newline */

fs/resctrl/rdtgroup.c +14 -0

+		goto out_unlock;
+	}
+
+	io_alloc_closid = resctrl_io_alloc_closid(r);
+
+	/*
+	 * When CDP is enabled, update the schema for both CDP_DATA and CDP_CODE.
+	 * Since the buffer is altered during parsing, create a copy to handle
+	 * peer schemata separately.
+	 */
+	if (resctrl_arch_get_cdp_enabled(r->rid)) {
+		peer_type = resctrl_peer_type(s->conf_type);
+		peer_s = resctrl_get_schema(peer_type);
+		peer_buf = kmalloc(nbytes, GFP_KERNEL);

fs/resctrl/rdtgroup.c +20 -0

+
+		kfree(peer_buf);
+	}
+
+	if (!ret)
+		ret = resctrl_io_alloc_parse_line(buf, r, s, io_alloc_closid);
+
+	if (ret)
+		goto out_unlock;
+
+	ret = resctrl_arch_update_domains(r, io_alloc_closid);
+
+out_unlock:
+	rdt_staged_configs_clear();
+	mutex_unlock(&rdtgroup_mutex);
+	cpus_read_unlock();
+
+	return ret ?: nbytes;
+}
+
 /* rdtgroup information files for one cache resource. */
 static struct rftype res_common_files[] = {
 	{

@@ -2135,9 +2244,10 @@ static struct rftype res_common_files[] = {
 	},

Hi Babu,

Quoted 5 lines · Babu Moger

"io_alloc" feature enables direct insertion of data from I/O devices into the cache. By directly caching data from I/O devices rather than first storing the I/O data in DRAM, it reduces the demands on DRAM bandwidth and reduces latency to the processor consuming the I/O data.

Quoted 1 line

Provide the interface to modify io_alloc CBMs (Capacity Bit Mask) when

"Provide the interface to modify io_alloc CBMs (Capacity Bit Mask) when feature is enabled." ->
"Enable users to modify io_alloc CBMs (Capacity Bit Masks) via the io_alloc_cbm resctrl file when io_alloc is enabled."?

Quoted 6 lines

feature is enabled. Update the CBMs for both CDP_DATA and CDP_CODE when CDP is enabled.

Signed-off-by Babu Moger <[email protected]>

---

...

Quoted 32 lines

---
 Documentation/filesystems/resctrl.rst |   8 ++
 fs/resctrl/ctrlmondata.c              |   4 +-
 fs/resctrl/internal.h                 |   2 +
 fs/resctrl/rdtgroup.c                 | 112 +++++++++++++++++++++++++-
 4 files changed, 123 insertions(+), 3 deletions(-)

Documentation/filesystems/resctrl.rst +8 -0

index c22a9dd667cd..b9c3ffdec27f 100644
@@ -189,6 +189,14 @@ related to allocation:
 			# cat /sys/fs/resctrl/info/L3/io_alloc_cbm
 			0=ffff;1=ffff
 
+		CBMs can be configured by writing to the interface.
+
+		Example::
+
+			# echo 1=FF > /sys/fs/resctrl/info/L3/io_alloc_cbm
+			# cat /sys/fs/resctrl/info/L3/io_alloc_cbm
+			0=ffff;1=00ff
+
 		When CDP is enabled "io_alloc_cbm" associated with the DATA and CODE
 		resources may reflect the same values. For example, values read from and
 		written to /sys/fs/resctrl/info/L3DATA/io_alloc_cbm may be reflected by

fs/resctrl/ctrlmondata.c +0 -0

index 5c16557fb7a8..23c72a5ac043 100644
@@ -148,8 +148,8 @@ static bool cbm_validate(char *buf, u32 *data, struct rdt_resource *r)
  * Read one cache bit mask (hex). Check that it is valid for the current

fs/resctrl/ctrlmondata.c static bool cbm_validate(char *buf, u32 *data, struct rdt_resource *r) · line 152

Quoted 6 lines

fs/resctrl/ctrlmondata.c +2 -2

  * resource type.
  */
-static int parse_cbm(struct rdt_parse_data *data, struct resctrl_schema *s,
-		     struct rdt_ctrl_domain *d)
+int parse_cbm(struct rdt_parse_data *data, struct resctrl_schema *s,
+	      struct rdt_ctrl_domain *d)

This can remain static by moving caller to ctrlmondata.c

Quoted 25 lines

fs/resctrl/ctrlmondata.c +0 -0

 {
 	enum rdtgrp_mode mode = data->mode;
 	struct resctrl_staged_config *cfg;

fs/resctrl/internal.h +2 -0

index ab76a1e2e679..011564e69ed5 100644
@@ -382,6 +382,8 @@ bool closid_allocated(unsigned int closid);
 int resctrl_find_cleanest_closid(void);
 void show_doms(struct seq_file *s, struct resctrl_schema *schema,
 	       char *name, int closid);
+int parse_cbm(struct rdt_parse_data *data, struct resctrl_schema *s,
+	      struct rdt_ctrl_domain *d);
 
 #ifdef CONFIG_RESCTRL_FS_PSEUDO_LOCK
 int rdtgroup_locksetup_enter(struct rdtgroup *rdtgrp);

fs/resctrl/rdtgroup.c +0 -0

index 008657e43656..1093ec5e385b 100644
@@ -2036,6 +2036,115 @@ static int resctrl_io_alloc_cbm_show(struct kernfs_open_file *of,
 	return ret;
 }
 

fs/resctrl/rdtgroup.c static int resctrl_io_alloc_cbm_show(struct kernfs_open_file *of, · line 2040

Quoted 2 lines

fs/resctrl/rdtgroup.c +2 -0

+static int resctrl_io_alloc_parse_line(char *line,  struct rdt_resource *r,
+				       struct resctrl_schema *s, u32 closid)

Please move to ctrlmondata.c

Quoted 28 lines

fs/resctrl/rdtgroup.c +27 -0

+{
+	struct rdt_parse_data data;
+	struct rdt_ctrl_domain *d;
+	char *dom = NULL, *id;
+	unsigned long dom_id;
+
+next:
+	if (!line || line[0] == '\0')
+		return 0;
+
+	dom = strsep(&line, ";");
+	id = strsep(&dom, "=");
+	if (!dom || kstrtoul(id, 10, &dom_id)) {
+		rdt_last_cmd_puts("Missing '=' or non-numeric domain\n");
+		return -EINVAL;
+	}
+
+	dom = strim(dom);
+	list_for_each_entry(d, &r->ctrl_domains, hdr.list) {
+		if (d->hdr.id == dom_id) {
+			data.buf = dom;
+			data.mode = RDT_MODE_SHAREABLE;
+			data.closid = closid;
+			if (parse_cbm(&data, s, d))
+				return -EINVAL;
+			goto next;
+		}

fs/resctrl/rdtgroup.c static int resctrl_io_alloc_cbm_show(struct kernfs_open_file *of, · line 2073

Quoted 6 lines

fs/resctrl/rdtgroup.c +6 -0

+	}
+	return -EINVAL;
+}
+
+static ssize_t resctrl_io_alloc_cbm_write(struct kernfs_open_file *of,
+					  char *buf, size_t nbytes, loff_t off)

Please move to ctrlmondata.c

Quoted 15 lines

fs/resctrl/rdtgroup.c +14 -0

+{
+	struct resctrl_schema *s = rdt_kn_parent_priv(of->kn);
+	enum resctrl_conf_type peer_type;
+	struct rdt_resource *r = s->res;
+	struct resctrl_schema *peer_s;
+	u32 io_alloc_closid;
+	char *peer_buf;
+	int ret = 0;
+
+	/* Valid input requires a trailing newline */
+	if (nbytes == 0 || buf[nbytes - 1] != '\n')
+		return -EINVAL;
+
+	buf[nbytes - 1] = '\0';

fs/resctrl/rdtgroup.c static int resctrl_io_alloc_cbm_show(struct kernfs_open_file *of, · line 2093

Quoted 6 lines

fs/resctrl/rdtgroup.c +6 -0

+
+	cpus_read_lock();
+	mutex_lock(&rdtgroup_mutex);
+
+	if (!r->cache.io_alloc_capable) {
+		rdt_last_cmd_printf("io_alloc is not supported on %s\n", s->name);

rdt_last_cmd_clear() is required before any write to the buffer.

fs/resctrl/rdtgroup.c static int resctrl_io_alloc_cbm_show(struct kernfs_open_file *of, · line 2103

Quoted 11 lines

fs/resctrl/rdtgroup.c +10 -0

+		ret = -EINVAL;
+		goto out_unlock;
+	}
+
+	rdt_last_cmd_clear();
+	rdt_staged_configs_clear();
+
+	if (!resctrl_arch_get_io_alloc_enabled(r)) {
+		rdt_last_cmd_printf("io_alloc is not enabled on %s\n", s->name);
+		ret = -EINVAL;

Inconsistent error codes when io_alloc is not enabled. resctrl_io_alloc_write() and resctrl_io_alloc_cbm_show() returns -ENODEV.

Quoted 21 lines

fs/resctrl/rdtgroup.c +20 -0

+		goto out_unlock;
+	}
+
+	io_alloc_closid = resctrl_io_alloc_closid(r);
+
+	/*
+	 * When CDP is enabled, update the schema for both CDP_DATA and CDP_CODE.
+	 * Since the buffer is altered during parsing, create a copy to handle
+	 * peer schemata separately.
+	 */
+	if (resctrl_arch_get_cdp_enabled(r->rid)) {
+		peer_type = resctrl_peer_type(s->conf_type);
+		peer_s = resctrl_get_schema(peer_type);
+		peer_buf = kmalloc(nbytes, GFP_KERNEL);
+		if (!peer_buf) {
+			rdt_last_cmd_puts("Out of Memory - io_alloc update failed\n");
+			ret = -ENOMEM;
+			goto out_unlock;
+		}
+

fs/resctrl/rdtgroup.c static int resctrl_io_alloc_cbm_show(struct kernfs_open_file *of, · line 2127

Quoted 4 lines

fs/resctrl/rdtgroup.c +4 -0

+		memcpy(peer_buf, buf, nbytes);
+
+		if (peer_s)
+			ret = resctrl_io_alloc_parse_line(peer_buf, r, peer_s, io_alloc_closid);

Similar to resctrl_io_alloc_write() this looks unnecessary. I think this can be simplified by moving the CDP check to resctrl_io_alloc_parse_line() where the provided CBM can be parsed *once* and the staged config of a CDP type just copied to the staged config of its peer type.

Quoted 26 lines

fs/resctrl/rdtgroup.c +20 -0

+
+		kfree(peer_buf);
+	}
+
+	if (!ret)
+		ret = resctrl_io_alloc_parse_line(buf, r, s, io_alloc_closid);
+
+	if (ret)
+		goto out_unlock;
+
+	ret = resctrl_arch_update_domains(r, io_alloc_closid);
+
+out_unlock:
+	rdt_staged_configs_clear();
+	mutex_unlock(&rdtgroup_mutex);
+	cpus_read_unlock();
+
+	return ret ?: nbytes;
+}
+
 /* rdtgroup information files for one cache resource. */
 static struct rftype res_common_files[] = {
 	{

@@ -2135,9 +2244,10 @@ static struct rftype res_common_files[] = {
 	},

fs/resctrl/rdtgroup.c static struct rftype res_common_files[] = { · line 2253

Quoted 10 lines

fs/resctrl/rdtgroup.c +2 -1

 	{
 		.name		= "io_alloc_cbm",
-		.mode		= 0444,
+		.mode		= 0644,
 		.kf_ops		= &rdtgroup_kf_single_ops,
 		.seq_show	= resctrl_io_alloc_cbm_show,
+		.write		= resctrl_io_alloc_cbm_write,
 	},
 	{
 		.name		= "max_threshold_occupancy",

Reinette

This series adds the support for L3 Smart Data Cache Injection Allocation Enforcement (SDCIAE) to resctrl infrastructure. It is referred to as "io_alloc" in resctrl subsystem.

Upcoming AMD hardware implements Smart Data Cache Injection (SDCI). Smart Data Cache Injection (SDCI) is a mechanism that enables direct insertion of data from I/O devices into the L3 cache. By directly caching data from I/O devices rather than first storing the I/O data in DRAM, SDCI reduces demands on DRAM bandwidth and reduces latency to the processor consuming the I/O data.

The SDCIAE (SDCI Allocation Enforcement) PQE feature allows system software to control the portion of the L3 cache used for SDCI devices.

When enabled, SDCIAE forces all SDCI lines to be placed into the L3 cache partitions identified by the highest-supported L3_MASK_n register, where n is the maximum supported CLOSID.

The feature details are documented in the APM listed below [1].
[1] AMD64 Architecture Programmer's Manual Volume 2: System Programming Publication # 24593 Revision 3.41 section 19.4.7 L3 Smart Data Cache Injection Allocation Enforcement (SDCIAE)

Link https://bugzilla.kernel.org/show_bug.cgi?id=206537

The feature requires linux support of TPH (TLP Processing Hints). The support is available in linux kernel after the commit 48d0fd2b903e3 ("PCI/TPH: Add TPH documentation")

The patches are based on top of commit (6.16.0-rc5) commit b4ec95e3bc3f ("Merge x86/microcode into tip/master")

# Linux Implementation

Feature adds following interface files when the resctrl "io_alloc" feature is supported on the resource:

/sys/fs/resctrl/info/L3/io_alloc: Report the feature status. Enable/disable the
feature by writing to the interface.

/sys/fs/resctrl/info/L3/io_alloc_cbm: List the Capacity Bit Masks (CBMs) available
for I/O devices when io_alloc feature is enabled. Configure the CBM by writing to the interface.

When CDP is enabled, these files will be created both in L3CODE and L3DATA.

# Examples:

a. Check if io_alloc feature is available

	# mount -t resctrl resctrl /sys/fs/resctrl/

	# cat /sys/fs/resctrl/info/L3/io_alloc
	disabled

b. Enable the io_alloc feature.

	# echo 1 > /sys/fs/resctrl/info/L3/io_alloc 
	# cat /sys/fs/resctrl/info/L3/io_alloc
	enabled

c. Check the CBM values for the io_alloc feature.

	# cat /sys/fs/resctrl/info/L3/io_alloc_cbm 
	0=ffff;1=ffff

d. Change the CBM value for the domain 1:

	# echo 1=FF > /sys/fs/resctrl/info/L3/io_alloc_cbm

	# cat /sys/fs/resctrl/info/L3/io_alloc_cbm 
	0=ffff;1=00ff

d. Disable io_alloc feature and exit.

	# echo 0 > /sys/fs/resctrl/info/L3/io_alloc 
	# cat /sys/fs/resctrl/info/L3/io_alloc
	disabled

	# umount /sys/fs/resctrl/

---

Hi Babu,

Quoted 12 lines · Babu Moger

This series adds the support for L3 Smart Data Cache Injection Allocation Enforcement (SDCIAE) to resctrl infrastructure. It is referred to as "io_alloc" in resctrl subsystem.

Upcoming AMD hardware implements Smart Data Cache Injection (SDCI). Smart Data Cache Injection (SDCI) is a mechanism that enables direct insertion of data from I/O devices into the L3 cache. By directly caching data from I/O devices rather than first storing the I/O data in DRAM, SDCI reduces demands on DRAM bandwidth and reduces latency to the processor consuming the I/O data.

Quoted 6 lines

The SDCIAE (SDCI Allocation Enforcement) PQE feature allows system software to control the portion of the L3 cache used for SDCI devices.

When enabled, SDCIAE forces all SDCI lines to be placed into the L3 cache partitions identified by the highest-supported L3_MASK_n register, where n is the maximum supported CLOSID.

Even though this CLOSID use is unique to AMD this implementation makes it part of resctrl fs's support of io_alloc. It is confusing to have architectural specific features be handled by resctrl fs so I think it will be useful to add a snippet here to help folks trying to decipher this work. Consider for example a snippet like:

Since CLOSIDs are managed by resctrl fs it is least invasive to make the "io_alloc is supported by maximum supported CLOSID" part of the initial resctrl fs support for io_alloc. Take care not to expose this use of CLOSID for io_alloc to user space so that this is not required from other architectures that may support io_alloc differently in the future.

Quoted 22 lines

The feature details are documented in the APM listed below [1].
[1] AMD64 Architecture Programmer's Manual Volume 2: System Programming Publication # 24593 Revision 3.41 section 19.4.7 L3 Smart Data Cache Injection Allocation Enforcement (SDCIAE)

Link https://bugzilla.kernel.org/show_bug.cgi?id=206537

The feature requires linux support of TPH (TLP Processing Hints). The support is available in linux kernel after the commit 48d0fd2b903e3 ("PCI/TPH: Add TPH documentation")

The patches are based on top of commit (6.16.0-rc5) commit b4ec95e3bc3f ("Merge x86/microcode into tip/master")

# Linux Implementation

Feature adds following interface files when the resctrl "io_alloc" feature is supported on the resource:

/sys/fs/resctrl/info/L3/io_alloc: Report the feature status. Enable/disable the
feature by writing to the interface.

Quoted 5 lines

/sys/fs/resctrl/info/L3/io_alloc_cbm: List the Capacity Bit Masks (CBMs) available
for I/O devices when io_alloc feature is enabled. Configure the CBM by writing to the interface.

When CDP is enabled, these files will be created both in L3CODE and L3DATA.

"will be" -> "are"

Quoted 15 lines

# Examples:

a. Check if io_alloc feature is available

	# mount -t resctrl resctrl /sys/fs/resctrl/

	# cat /sys/fs/resctrl/info/L3/io_alloc
	disabled

b. Enable the io_alloc feature.

	# echo 1 > /sys/fs/resctrl/info/L3/io_alloc 
	# cat /sys/fs/resctrl/info/L3/io_alloc
	enabled

Quoted 6 lines

c. Check the CBM values for the io_alloc feature.

	# cat /sys/fs/resctrl/info/L3/io_alloc_cbm 
	0=ffff;1=ffff

d. Change the CBM value for the domain 1:

"for the domain 1" -> "of domain 1"?

(nit: inconsistent use of "." vs. ":" in items)

Quoted 8 lines

	# echo 1=FF > /sys/fs/resctrl/info/L3/io_alloc_cbm

	# cat /sys/fs/resctrl/info/L3/io_alloc_cbm 
	0=ffff;1=00ff

d. Disable io_alloc feature and exit.

Quoted 7 lines

	# echo 0 > /sys/fs/resctrl/info/L3/io_alloc 
	# cat /sys/fs/resctrl/info/L3/io_alloc
	disabled

	# umount /sys/fs/resctrl/

---

Reinette