Support L3 Smart Data Cache Injection Allocation Enforcement (SDCIAE)

PATCH Superseded Babu Moger 5 participants 22 messages Apr 21, 2025 – May 13, 2025

Subsystems (3)

Documentation x86 RDT - resource allocation

lore

Share & tools

lore

Babu Moger

author reviewer

2025-04-21 22:44 UTC #0 lore raw

v4 cover Support L3 Smart Data Cache Injection Allocation Enforcement (SDCIAE)

This series adds the support for L3 Smart Data Cache Injection Allocation Enforcement (SDCIAE) to resctrl infrastructure. It is refered to as "io_alloc" in resctrl subsystem.

Upcoming AMD hardware implements Smart Data Cache Injection (SDCI). Smart Data Cache Injection (SDCI) is a mechanism that enables direct insertion of data from I/O devices into the L3 cache. By directly caching data from I/O devices rather than first storing the I/O data in DRAM, SDCI reduces demands on DRAM bandwidth and reduces latency to the processor consuming the I/O data.

The SDCIAE (SDCI Allocation Enforcement) PQE feature allows system software to control the portion of the L3 cache used for SDCI devices.

When enabled, SDCIAE forces all SDCI lines to be placed into the L3 cache partitions identified by the highest-supported L3_MASK_n register, where n is the maximum supported CLOSID.

The feature details are documented in the APM listed below [1].
[1] AMD64 Architecture Programmer's Manual Volume 2: System Programming Publication # 24593 Revision 3.41 section 19.4.7 L3 Smart Data Cache Injection Allocation Enforcement (SDCIAE)

Link https://bugzilla.kernel.org/show_bug.cgi?id=206537

The feature requires linux support of TPH (TLP Processing Hints). The support is available in linux kernel after the commit 48d0fd2b903e3 ("PCI/TPH: Add TPH documentation")

The patches are based on top of commit 84c319145cbad6 Merge branch into ("tip/master: 'x86/nmi'")

# Linux Implementation

Feature adds following interface files when the resctrl "io_alloc" feature is supported on L3 resource:

/sys/fs/resctrl/info/L3/io_alloc: Report the feature status. Enable/disable the
feature by writing to the interface.

/sys/fs/resctrl/info/L3/io_alloc_cbm: List the Capacity Bit Masks (CBMs) available
for I/O devices when io_alloc feature is enabled. Configure the CBM by writing to the interface.

# Examples:

a. Check if io_alloc feature is available
#mount -t resctrl resctrl /sys/fs/resctrl/

# cat /sys/fs/resctrl/info/L3/io_alloc
disabled

b. Enable the io_alloc feature.

	# echo 1 > /sys/fs/resctrl/info/L3/io_alloc 
	# cat /sys/fs/resctrl/info/L3/io_alloc
	enabled

c. Check the CBM values for the io_alloc feature.

	# cat /sys/fs/resctrl/info/L3/io_alloc_cbm 
	L3:0=ffff;1=ffff

d. Change the CBM value for the domain 1:

	# echo L3:1=FF > /sys/fs/resctrl/info/L3/io_alloc_cbm

	# cat /sys/fs/resctrl/info/L3/io_alloc_cbm 
	L3:0=ffff;1=00ff

d. Disable io_alloc feature and exit.

	# echo 0 > /sys/fs/resctrl/info/L3/io_alloc 
	# cat /sys/fs/resctrl/info/L3/io_alloc
	disabled

#umount /sys/fs/resctrl/

---
v4: The "io_alloc" interface will report "enabled/disabled/not supported"
instead of 0 or 1..

Updated resctrl_io_alloc_closid_get() to verify the max closid availability using closids_supported().

Updated the documentation for "shareable_bits" and "bit_usage".

NOTE: io_alloc is about specific CLOS. rdt_bit_usage_show() is not designed handle bit_usage for specific CLOS. Its about overall system. So, we cannot really tell the user which CLOS is shared across both hardware and software. This is something we need to discuss.

Introduced io_alloc_init() to initialize fflags.

Printed the group name when io_alloc enablement fails to help user.

Added rdtgroup_mutex before rdt_last_cmd_puts() in resctrl_io_alloc_cbm_show(). Returned -ENODEV when resource type is CDP_DATA.

Kept the resource name while printing the CBM (L3:0=ffff) that way we dont have to change show_doms() just for this feature and it is consistant across all the schemata display.

Added new patch to call parse_cbm() directly to avoid code duplication.

Checked all the series(v1-v3) again to verify if I missed any comment.

v3: Rewrote commit log for the last 3 patches. Changed the text to bit
more generic than the AMD specific feature. Added AMD feature specifics in the end.

Renamed the rdt_get_sdciae_alloc_cfg() to rdt_set_io_alloc_capable(). Renamed the _resctrl_io_alloc_enable() to _resctrl_sdciae_enable() as it is arch specific.

Changed the return to void in _resctrl_sdciae_enable() instead of int.

The number of CLOSIDs is determined based on the minimum supported across all resources (in closid_init). It needs to match the max supported on the resource. Added the check to verify if MAX CLOSID availability on the system.

Added CDP check to make sure io_alloc is configured in CDP_CODE. Highest CLOSID corresponds to CDP_CODE.

Added resctrl_io_alloc_closid_free() to free the io_alloc CLOSID.

Added errors in few cases when CLOSID allocation fails.
Fixes splat reported when info/L3/bit_usage is accesed when io_alloc is enabled.
https://lore.kernel.org/lkml/SJ1PR11MB60837B532254E7B23BC27E84FC052@SJ1PR11MB6083.namprd11.prod.outlook.com/

v2: Added dependancy on X86_FEATURE_CAT_L3
Removed the "" in CPU feature definition.

Changed sdciae_capable to io_alloc_capable to make it as generic feature. Moved io_alloc_capable field in struct resctrl_cache.

Changed the name of few arch functions similar to ABMC series. resctrl_arch_get_io_alloc_enabled() resctrl_arch_io_alloc_enable()

Renamed the feature to "io_alloc".
Added generic texts for the feature in commit log and resctrl.rst doc. Added resctrl_io_alloc_init_cat() to initialize io_alloc to default values when enabled.
Fixed io_alloc interface to show only on L3 resource.
Added the locks while processing io_alloc CBMs.

Previous versions:
v3: https://lore.kernel.org/lkml/[email protected]/
v2: https://lore.kernel.org/lkml/[email protected]/
v1: https://lore.kernel.org/lkml/[email protected]/

Babu Moger (8):
x86/cpufeatures: Add support for L3 Smart Data Cache Injection
Allocation Enforcement
x86/resctrl: Add SDCIAE feature in the command line options x86/resctrl: Detect io_alloc feature
x86/resctrl: Implement "io_alloc" enable/disable handlers x86/resctrl: Add user interface to enable/disable io_alloc feature x86/resctrl: Introduce interface to display io_alloc CBMs x86/resctrl: Modify rdt_parse_data to pass mode and CLOSID x86/resctrl: Introduce interface to modify io_alloc Capacity Bit Masks

 .../admin-guide/kernel-parameters.txt         |   2 +-
 Documentation/arch/x86/resctrl.rst            |  55 +++
 arch/x86/include/asm/cpufeatures.h            |   1 +
 arch/x86/include/asm/msr-index.h              |   1 +
 arch/x86/kernel/cpu/cpuid-deps.c              |   1 +
 arch/x86/kernel/cpu/resctrl/core.c            |   9 +
 arch/x86/kernel/cpu/resctrl/ctrlmondata.c     |  35 +-
 arch/x86/kernel/cpu/resctrl/internal.h        |  19 +
 arch/x86/kernel/cpu/resctrl/rdtgroup.c        | 372 +++++++++++++++++-
 arch/x86/kernel/cpu/scattered.c               |   1 +
 include/linux/resctrl.h                       |  12 +
 11 files changed, 487 insertions(+), 21 deletions(-)

Signature · 1 line

2.34.1

Babu Moger

author reviewer

2025-04-21 22:44 UTC #1 lore raw

PATCH v4 1/8 x86/cpufeatures: Add support for L3 Smart Data Cache Injection Allocation Enforcement

Smart Data Cache Injection (SDCI) is a mechanism that enables direct insertion of data from I/O devices into the L3 cache. By directly caching data from I/O devices rather than first storing the I/O data in DRAM, SDCI reduces demands on DRAM bandwidth and reduces latency to the processor consuming the I/O data.

The SDCIAE (SDCI Allocation Enforcement) PQE feature allows system software to control the portion of the L3 cache used for SDCI.

When enabled, SDCIAE forces all SDCI lines to be placed into the L3 cache partitions identified by the highest-supported L3_MASK_n register, where n is the maximum supported CLOSID.

Add CPUID feature bit that can be used to configure SDCIAE.

The feature details are documented in APM listed below [1].
[1] AMD64 Architecture Programmer's Manual Volume 2: System Programming Publication # 24593 Revision 3.41 section 19.4.7 L3 Smart Data Cache Injection Allocation Enforcement (SDCIAE)

Link https://bugzilla.kernel.org/show_bug.cgi?id=206537

Signed-off-by Babu Moger <[email protected]>

---
v4: Resolved a minor conflict in cpufeatures.h.

v3: No changes.

v2: Added dependancy on X86_FEATURE_CAT_L3
Removed the "" in CPU feature definition.
Minor text changes.

---
 arch/x86/include/asm/cpufeatures.h | 1 +
 arch/x86/kernel/cpu/cpuid-deps.c   | 1 +
 arch/x86/kernel/cpu/scattered.c    | 1 +
 3 files changed, 3 insertions(+)

arch/x86/include/asm/cpufeatures.h +1 -0

index 6c2c152d8a67..8dfbea91bef6 100644
@@ -481,6 +481,7 @@
 #define X86_FEATURE_AMD_HETEROGENEOUS_CORES (21*32 + 6) /* Heterogeneous Core Topology */
 #define X86_FEATURE_AMD_WORKLOAD_CLASS	(21*32 + 7) /* Workload Classification */
 #define X86_FEATURE_PREFER_YMM		(21*32 + 8) /* Avoid ZMM registers due to downclocking */
+#define X86_FEATURE_SDCIAE		(21*32 + 9) /* L3 Smart Data Cache Injection Allocation Enforcement */
 
 /*
  * BUG word(s)

arch/x86/kernel/cpu/cpuid-deps.c +1 -0

index 94c062cddfa4..24ff4a98d204 100644
@@ -71,6 +71,7 @@ static const struct cpuid_dep cpuid_deps[] = {
 	{ X86_FEATURE_CQM_MBM_LOCAL,		X86_FEATURE_CQM_LLC   },
 	{ X86_FEATURE_BMEC,			X86_FEATURE_CQM_MBM_TOTAL   },
 	{ X86_FEATURE_BMEC,			X86_FEATURE_CQM_MBM_LOCAL   },
+	{ X86_FEATURE_SDCIAE,			X86_FEATURE_CAT_L3    },
 	{ X86_FEATURE_AVX512_BF16,		X86_FEATURE_AVX512VL  },
 	{ X86_FEATURE_AVX512_FP16,		X86_FEATURE_AVX512BW  },
 	{ X86_FEATURE_ENQCMD,			X86_FEATURE_XSAVES    },

arch/x86/kernel/cpu/scattered.c +1 -0

index 16f3ca30626a..d18a7ce16388 100644
@@ -49,6 +49,7 @@ static const struct cpuid_bit cpuid_bits[] = {
 	{ X86_FEATURE_MBA,			CPUID_EBX,  6, 0x80000008, 0 },
 	{ X86_FEATURE_SMBA,			CPUID_EBX,  2, 0x80000020, 0 },
 	{ X86_FEATURE_BMEC,			CPUID_EBX,  3, 0x80000020, 0 },
+	{ X86_FEATURE_SDCIAE,			CPUID_EBX,  6, 0x80000020, 0 },
 	{ X86_FEATURE_AMD_WORKLOAD_CLASS,	CPUID_EAX, 22, 0x80000021, 0 },
 	{ X86_FEATURE_PERFMON_V2,		CPUID_EAX,  0, 0x80000022, 0 },
 	{ X86_FEATURE_AMD_LBR_V2,		CPUID_EAX,  1, 0x80000022, 0 },

Signature · 1 line

2.34.1

Babu Moger

author reviewer

2025-04-21 22:44 UTC #2 lore raw

PATCH v4 2/8 x86/resctrl: Add SDCIAE feature in the command line options

In reply to Babu Moger

Add the command line options to enable or disable the new resctrl feature L3 Smart Data Cache Injection Allocation Enforcement (SDCIAE).

Signed-off-by Babu Moger <[email protected]>

---
v4: No changes.

v3: No changes.

v2: No changes.

---
 Documentation/admin-guide/kernel-parameters.txt | 2 +-
 arch/x86/kernel/cpu/resctrl/core.c              | 2 ++
 2 files changed, 3 insertions(+), 1 deletion(-)

Documentation/admin-guide/kernel-parameters.txt +1 -1

index 76e538c77e31..5e5abc270f91 100644
@@ -5991,7 +5991,7 @@
 	rdt=		[HW,X86,RDT]
 			Turn on/off individual RDT features. List is:
 			cmt, mbmtotal, mbmlocal, l3cat, l3cdp, l2cat, l2cdp,
-			mba, smba, bmec.
+			mba, smba, bmec, sdciae.
 			E.g. to turn on cmt and turn off mba use:
 				rdt=cmt,!mba

arch/x86/kernel/cpu/resctrl/core.c +2 -0

index cf29681d01e0..422083dc4651 100644
@@ -723,6 +723,7 @@ enum {
 	RDT_FLAG_MBA,
 	RDT_FLAG_SMBA,
 	RDT_FLAG_BMEC,
+	RDT_FLAG_SDCIAE,
 };
 
 #define RDT_OPT(idx, n, f)	\

@@ -748,6 +749,7 @@ static struct rdt_options rdt_options[]  __initdata = {
 	RDT_OPT(RDT_FLAG_MBA,	    "mba",	X86_FEATURE_MBA),
 	RDT_OPT(RDT_FLAG_SMBA,	    "smba",	X86_FEATURE_SMBA),
 	RDT_OPT(RDT_FLAG_BMEC,	    "bmec",	X86_FEATURE_BMEC),
+	RDT_OPT(RDT_FLAG_SDCIAE,    "sdciae",	X86_FEATURE_SDCIAE),
 };
 #define NUM_RDT_OPTIONS ARRAY_SIZE(rdt_options)

Signature · 1 line

2.34.1

Babu Moger

author reviewer

2025-04-21 22:44 UTC #3 lore raw

PATCH v4 3/8 x86/resctrl: Detect io_alloc feature

In reply to Babu Moger

Data from I/O devices can be inserted directly into L3 cache. This reduces demands on DRAM bandwidth and reduces latency to the processor consuming the I/O data.

Introduce cache resource property "io_alloc_capable" that an architecture can set if a portion of the L3 cache can be allocated for I/O traffic.

Set this property on x86 systems that support SDCIAE (L3 Smart Data Cache Injection Allocation Enforcement).

Signed-off-by Babu Moger <[email protected]>

---
v4: Updated the commit message and code comment based on feedback.

v3: Rewrote commit log. Changed the text to bit generic than the AMD specific.
Renamed the rdt_get_sdciae_alloc_cfg() to rdt_set_io_alloc_capable(). Removed leftover comment from v2.

v2: Changed sdciae_capable to io_alloc_capable to make it generic feature.
Also moved the io_alloc_capable in struct resctrl_cache.

---
 arch/x86/kernel/cpu/resctrl/core.c | 7 +++++++
 include/linux/resctrl.h            | 3 +++
 2 files changed, 10 insertions(+)

arch/x86/kernel/cpu/resctrl/core.c +7 -0

index 422083dc4651..c478f591b7c1 100644
@@ -292,6 +292,11 @@ static void rdt_get_cdp_config(int level)
 	rdt_resources_all[level].r_resctrl.cdp_capable = true;
 }
 
+static void rdt_set_io_alloc_capable(struct rdt_resource *r)
+{
+	r->cache.io_alloc_capable = true;
+}
+
 static void rdt_get_cdp_l3_config(void)
 {
 	rdt_get_cdp_config(RDT_RESOURCE_L3);

@@ -858,6 +863,8 @@ static __init bool get_rdt_alloc_resources(void)
 		rdt_get_cache_alloc_cfg(1, r);
 		if (rdt_cpu_has(X86_FEATURE_CDP_L3))
 			rdt_get_cdp_l3_config();
+		if (rdt_cpu_has(X86_FEATURE_SDCIAE))
+			rdt_set_io_alloc_capable(r);
 		ret = true;
 	}
 	if (rdt_cpu_has(X86_FEATURE_CAT_L2)) {

include/linux/resctrl.h +3 -0

index 880351ca3dfc..dd09bb9a173b 100644
@@ -177,6 +177,8 @@ struct rdt_mon_domain {
  * @arch_has_sparse_bitmasks:	True if a bitmask like f00f is valid.
  * @arch_has_per_cpu_cfg:	True if QOS_CFG register for this cache
  *				level has CPU scope.
+ * @io_alloc_capable:	True if portion of the cache can be allocated
+ *			for I/O traffic.
  */
 struct resctrl_cache {
 	unsigned int	cbm_len;

@@ -184,6 +186,7 @@ struct resctrl_cache {
 	unsigned int	shareable_bits;
 	bool		arch_has_sparse_bitmasks;
 	bool		arch_has_per_cpu_cfg;
+	bool		io_alloc_capable;
 };
 
 /**

Signature · 1 line

2.34.1

Babu Moger

author reviewer

2025-04-21 22:44 UTC #4 lore raw

PATCH v4 4/8 x86/resctrl: Implement "io_alloc" enable/disable handlers

In reply to Babu Moger

"io_alloc" enables direct insertion of data from I/O devices into the L3 cache.

On AMD, "io_alloc" feature is backed by L3 Smart Data Cache Injection Allocation Enforcement (SDCIAE). Change SDCIAE state by setting (to enable) or clearing (to disable) bit 1 of MSR L3_QOS_EXT_CFG on all logical processors within the cache domain.

Introduce architecture-specific handlers to enable and disable the feature.

The SDCIAE feature details are available in APM listed below [1].
[1] AMD64 Architecture Programmer's Manual Volume 2: System Programming Publication # 24593 Revision 3.41 section 19.4.7 L3 Smart Data Cache Injection Allocation Enforcement (SDCIAE)

Link https://bugzilla.kernel.org/show_bug.cgi?id=206537

Signed-off-by Babu Moger <[email protected]>

---
v4: Updated the commit log to address the feedback.

v3: Passed the struct rdt_resource to resctrl_arch_get_io_alloc_enabled() instead of resource id.
Renamed the _resctrl_io_alloc_enable() to _resctrl_sdciae_enable() as it is arch specific. Changed the return to void in _resctrl_sdciae_enable() instead of int.
Added more context in commit log and fixed few typos.

v2: Renamed the functions to simplify the code.
Renamed sdciae_capable to io_alloc_capable.

Changed the name of few arch functions similar to ABMC series. resctrl_arch_get_io_alloc_enabled() resctrl_arch_io_alloc_enable()

---
 arch/x86/include/asm/msr-index.h       |  1 +
 arch/x86/kernel/cpu/resctrl/internal.h | 10 ++++++++
 arch/x86/kernel/cpu/resctrl/rdtgroup.c | 32 ++++++++++++++++++++++++++
 include/linux/resctrl.h                |  9 ++++++++
 4 files changed, 52 insertions(+)

arch/x86/include/asm/msr-index.h +1 -0

index e6134ef2263d..3970e0b16e47 100644
@@ -1203,6 +1203,7 @@
 /* - AMD: */
 #define MSR_IA32_MBA_BW_BASE		0xc0000200
 #define MSR_IA32_SMBA_BW_BASE		0xc0000280
+#define MSR_IA32_L3_QOS_EXT_CFG		0xc00003ff
 #define MSR_IA32_EVT_CFG_BASE		0xc0000400
 
 /* AMD-V MSRs */

arch/x86/kernel/cpu/resctrl/internal.h +10 -0

index eaae99602b61..6ead222904fe 100644
@@ -32,6 +32,9 @@
  */
 #define MBM_CNTR_WIDTH_OFFSET_MAX (62 - MBM_CNTR_WIDTH_BASE)
 
+/* Setting bit 1 in L3_QOS_EXT_CFG enables the SDCIAE feature. */
+#define SDCIAE_ENABLE_BIT		1
+
 /**
  * cpumask_any_housekeeping() - Choose any CPU in @mask, preferring those that
  *			        aren't marked nohz_full

@@ -381,6 +384,7 @@ struct msr_param {
  * @mon_scale:		cqm counter * mon_scale = occupancy in bytes
  * @mbm_width:		Monitor width, to detect and correct for overflow.
  * @cdp_enabled:	CDP state of this resource
+ * @sdciae_enabled:	SDCIAE feature is enabled
  *
  * Members of this structure are either private to the architecture
  * e.g. mbm_width, or accessed via helpers that provide abstraction. e.g.

@@ -394,6 +398,7 @@ struct rdt_hw_resource {
 	unsigned int		mon_scale;
 	unsigned int		mbm_width;
 	bool			cdp_enabled;
+	bool			sdciae_enabled;
 };
 
 static inline struct rdt_hw_resource *resctrl_to_arch_res(struct rdt_resource *r)

@@ -420,6 +425,11 @@ static inline bool resctrl_arch_get_cdp_enabled(enum resctrl_res_level l)
 
 int resctrl_arch_set_cdp_enabled(enum resctrl_res_level l, bool enable);
 
+static inline bool resctrl_arch_get_io_alloc_enabled(struct rdt_resource *r)
+{
+	return resctrl_to_arch_res(r)->sdciae_enabled;
+}
+
 void arch_mon_domain_online(struct rdt_resource *r, struct rdt_mon_domain *d);
 
 /* CPUID.(EAX=10H, ECX=ResID=1).EAX */

arch/x86/kernel/cpu/resctrl/rdtgroup.c +32 -0

index 93ec829015f1..85796a186374 100644
@@ -1845,6 +1845,38 @@ static ssize_t mbm_local_bytes_config_write(struct kernfs_open_file *of,
 	return ret ?: nbytes;
 }
 
+static void resctrl_sdciae_set_one_amd(void *arg)
+{
+	bool *enable = arg;
+
+	if (*enable)
+		msr_set_bit(MSR_IA32_L3_QOS_EXT_CFG, SDCIAE_ENABLE_BIT);
+	else
+		msr_clear_bit(MSR_IA32_L3_QOS_EXT_CFG, SDCIAE_ENABLE_BIT);
+}
+
+static void _resctrl_sdciae_enable(struct rdt_resource *r, bool enable)
+{
+	struct rdt_ctrl_domain *d;
+
+	/* Update L3_QOS_EXT_CFG MSR on all the CPUs in all domains */
+	list_for_each_entry(d, &r->ctrl_domains, hdr.list)
+		on_each_cpu_mask(&d->hdr.cpu_mask, resctrl_sdciae_set_one_amd, &enable, 1);
+}
+
+int resctrl_arch_io_alloc_enable(struct rdt_resource *r, bool enable)
+{
+	struct rdt_hw_resource *hw_res = resctrl_to_arch_res(r);
+
+	if (hw_res->r_resctrl.cache.io_alloc_capable &&
+	    hw_res->sdciae_enabled != enable) {
+		_resctrl_sdciae_enable(r, enable);
+		hw_res->sdciae_enabled = enable;
+	}
+
+	return 0;
+}
+
 /* rdtgroup information files for one cache resource. */
 static struct rftype res_common_files[] = {
 	{

include/linux/resctrl.h +9 -0

index dd09bb9a173b..92e242c13719 100644
@@ -514,6 +514,15 @@ void resctrl_arch_reset_rmid_all(struct rdt_resource *r, struct rdt_mon_domain *
  */
 void resctrl_arch_reset_all_ctrls(struct rdt_resource *r);
 
+/**
+ * resctrl_arch_io_alloc_enable() - Enable/disable io_alloc feature.
+ * @r:		The resctrl resource.
+ * @enable:	Enable (true) or disable (false) io_alloc on resource @r.
+ *
+ * This can be called from any CPU.
+ */
+int resctrl_arch_io_alloc_enable(struct rdt_resource *r, bool enable);
+
 extern unsigned int resctrl_rmid_realloc_threshold;
 extern unsigned int resctrl_rmid_realloc_limit;

Signature · 1 line

2.34.1

Babu Moger

author reviewer

2025-04-21 22:44 UTC #5 lore raw

PATCH v4 5/8 x86/resctrl: Add user interface to enable/disable io_alloc feature

In reply to Babu Moger

The io_alloc feature in resctrl is a mechanism that enables direct insertion of data from I/O devices into the L3 cache.

On AMD systems, io_alloc feature is backed by SDCIAE (L3 Smart Data Cache Injection Allocation Enforcement). When enabled, SDCIAE forces all SDCI lines to be placed into the L3 cache partitions identified by the highest-supported L3_MASK_n register as reported by CPUID Fn0000_0010_EDX_x1.MAX_COS. For example, if MAX_COS=15, SDCI lines will be allocated into the L3 cache partitions determined by the bitmask in the L3_MASK_15 register.

When CDP is enabled, io_alloc routes I/O traffic using the highest CLOSID allocated for the instruction cache (L3CODE).

Introduce user interface to enable/disable "io_alloc" feature.

Signed-off-by Babu Moger <[email protected]>

---
v4: Updated the change log.
Updated the user doc.
The "io_alloc" interface will report "enabled/disabled/not supported". Updated resctrl_io_alloc_closid_get() to verify the max closid availability. Updated the documentation for "shareable_bits" and "bit_usage". Introduced io_alloc_init() to initialize fflags.
Printed the group name when io_alloc enablement fails.

v3: Rewrote the change to make it generic.
Rewrote the documentation in resctrl.rst to be generic and added AMD feature details in the end.
Added the check to verify if MAX CLOSID availability on the system. Added CDP check to make sure io_alloc is configured in CDP_CODE. Added resctrl_io_alloc_closid_free() to free the io_alloc CLOSID. Added errors in few cases when CLOSID allocation fails. Fixes splat reported when info/L3/bit_usage is accesed when io_alloc is enabled.
https://lore.kernel.org/lkml/SJ1PR11MB60837B532254E7B23BC27E84FC052@SJ1PR11MB6083.namprd11.prod.outlook.com/

v2: Renamed the feature to "io_alloc".
Added generic texts for the feature in commit log and resctrl.rst doc. Added resctrl_io_alloc_init_cat() to initialize io_alloc to default values when enabled.
Fixed io_alloc show functinality to display only on L3 resource.

---
 Documentation/arch/x86/resctrl.rst     |  34 +++++
 arch/x86/kernel/cpu/resctrl/rdtgroup.c | 185 ++++++++++++++++++++++++-
 2 files changed, 218 insertions(+), 1 deletion(-)

Documentation/arch/x86/resctrl.rst +34 -0

index 6768fc1fad16..7672c5c52c1a 100644
@@ -95,6 +95,11 @@ related to allocation:
 		some platforms support devices that have their
 		own settings for cache use which can over-ride
 		these bits.
+
+		When the "io_alloc" feature is enabled, a portion of the cache
+		is reserved for shared use between hardware and software. Refer
+		to "bit_usage" to see which portion is allocated for this purpose.
+
 "bit_usage":
 		Annotated capacity bitmasks showing how all
 		instances of the resource are used. The legend is:

@@ -135,6 +140,35 @@ related to allocation:
 			"1":
 			      Non-contiguous 1s value in CBM is supported.
 
+"io_alloc":
+		The "io_alloc" enables system software to configure the portion
+		of the L3 cache allocated for I/O traffic.
+
+		The feature routes the I/O traffic via specific CLOSID reserved
+		for io_alloc feature. By configuring the CBM (Capacity Bit Mask)
+		for the CLOSID, users can control the L3 portions available for
+		I/0 traffic. The reserved CLOSID will be excluded for group creation.
+
+		The interface provides a means to query the status of feature support.
+
+		Example::
+
+			# cat /sys/fs/resctrl/info/L3/io_alloc
+			disabled
+
+		Feature can be enabled/disabled by writing to the interface.
+		Example::
+
+			# echo 1 > /sys/fs/resctrl/info/L3/io_alloc
+			# cat /sys/fs/resctrl/info/L3/io_alloc
+			enabled
+
+		On AMD systems, the io_alloc feature is supported by the L3 Smart
+		Data Cache Injection Allocation Enforcement (SDCIAE). The CLOSID for
+		io_alloc is determined by the highest CLOSID supported by the resource.
+		When CDP is enabled, io_alloc routes I/O traffic using the highest
+		CLOSID allocated for the instruction cache (L3CODE).
+
 Memory bandwidth(MB) subdirectory contains the following files
 with respect to allocation:

arch/x86/kernel/cpu/resctrl/rdtgroup.c +184 -1

235 lines not on this page · raw message

Signature · 1 line

2.34.1

kernel test robot

build bot

2025-05-07 09:31 UTC #6 lore raw

Re: [PATCH v4 5/8] x86/resctrl: Add user interface to enable/disable io_alloc feature

kernel test robot noticed the following build warnings:

Hi Babu,

kernel test robot noticed the following build warnings:

[auto build test WARNING on linus/master]
[also build test WARNING on v6.15-rc5]
[cannot apply to tip/x86/core next-20250507]
[If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]

url:    https://github.com/intel-lab-lkp/linux/commits/Babu-Moger/x86-cpufeatures-Add-support-for-L3-Smart-Data-Cache-Injection-Allocation-Enforcement/20250422-064804
base:   linus/master
patch link:    https://lore.kernel.org/r/963710cb374d55856053989e4edc61d7fd033662.1745275431.git.babu.moger%40amd.com
patch subject: [PATCH v4 5/8] x86/resctrl: Add user interface to enable/disable io_alloc feature
config: x86_64-randconfig-r131-20250426 (https://download.01.org/0day-ci/archive/20250507/[email protected]/config)
compiler: gcc-12 (Debian 12.2.0-14) 12.2.0
reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20250507/[email protected]/reproduce)

If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/

sparse warnings: (new ones prefixed by >>)

>> arch/x86/kernel/cpu/resctrl/rdtgroup.c:1975:42: sparse: sparse: incorrect type in return expression (different address spaces) @@     expected char const * @@     got char const [noderef] __rcu *name @@

arch/x86/kernel/cpu/resctrl/rdtgroup.c:1975:42: sparse: expected char const *
arch/x86/kernel/cpu/resctrl/rdtgroup.c:1975:42: sparse: got char const [noderef] __rcu *name

vim +1975 arch/x86/kernel/cpu/resctrl/rdtgroup.c

1968
1969 static const char *rdtgroup_name_by_closid(int closid)
1970 {
1971 struct rdtgroup *rdtgrp;
1972
1973 list_for_each_entry(rdtgrp, &rdt_all_groups, rdtgroup_list) {
1974 if (rdtgrp->closid == closid)

> 1975				return rdtgrp->kn->name;

1976 }
1977
1978 return NULL;
1979 }
1980

Signature · 2 lines

0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki

Moger, Babu

author reviewer

2025-05-13 20:20 UTC #7 lore raw

Re: [PATCH v4 5/8] x86/resctrl: Add user interface to enable/disable io_alloc feature

Quoted 38 lines · kernel test robot

Hi Babu,
kernel test robot noticed the following build warnings:
[auto build test WARNING on linus/master]
[also build test WARNING on v6.15-rc5]
[cannot apply to tip/x86/core next-20250507]
[If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]
url:    https://github.com/intel-lab-lkp/linux/commits/Babu-Moger/x86-cpufeatures-Add-support-for-L3-Smart-Data-Cache-Injection-Allocation-Enforcement/20250422-064804
base:   linus/master
patch link:    https://lore.kernel.org/r/963710cb374d55856053989e4edc61d7fd033662.1745275431.git.babu.moger%40amd.com
patch subject: [PATCH v4 5/8] x86/resctrl: Add user interface to enable/disable io_alloc feature
config: x86_64-randconfig-r131-20250426 (https://download.01.org/0day-ci/archive/20250507/[email protected]/config)
compiler: gcc-12 (Debian 12.2.0-14) 12.2.0
reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20250507/[email protected]/reproduce)
If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/
sparse warnings: (new ones prefixed by >>)
Quoted 1 line
Quoted 1 line
arch/x86/kernel/cpu/resctrl/rdtgroup.c:1975:42: sparse: sparse: incorrect type in return expression (different address spaces) @@ expected char const * @@ got char const [noderef] __rcu *name @@
arch/x86/kernel/cpu/resctrl/rdtgroup.c:1975:42: sparse: expected char const *
arch/x86/kernel/cpu/resctrl/rdtgroup.c:1975:42: sparse: got char const [noderef] __rcu *name
vim +1975 arch/x86/kernel/cpu/resctrl/rdtgroup.c
1968
1969 static const char *rdtgroup_name_by_closid(int closid)
1970 {
1971 struct rdtgroup *rdtgrp;
1972
1973 list_for_each_entry(rdtgrp, &rdt_all_groups, rdtgroup_list) {
1974 if (rdtgrp->closid == closid)

Quoted 7 lines

Quoted 1 line
1975 return rdtgrp->kn->name;
1976 }
1977
1978 return NULL;
1979 }
1980

Thanks for the report. Will take care in next revision.

Signature · 2 lines

Thanks
Babu Moger

Babu Moger

author reviewer

2025-04-21 22:45 UTC #8 lore raw

PATCH v4 6/8 x86/resctrl: Introduce interface to display io_alloc CBMs

In reply to Babu Moger

The io_alloc feature in resctrl enables system software to configure the portion of the L3 cache allocated for I/O traffic.

Add the interface to display CBMs (Capacity Bit Mask) of io_alloc feature.

When CDP is enabled, io_alloc routes traffic using the highest CLOSID used by a L3CODE resource. Add a check for the CDP resource type.

Signed-off-by Babu Moger <[email protected]>

---
v4: Updated the change log.
Added rdtgroup_mutex before rdt_last_cmd_puts(). Returned -ENODEV when resource type is CDP_DATA.
Kept the resource name while printing the CBM (L3:0=fff) that way I dont have to change show_doms() just for this feature and it is consistant across all the schemata display.

v3: Minor changes due to changes in resctrl_arch_get_io_alloc_enabled()
and resctrl_io_alloc_closid_get().
Added the check to verify CDP resource type.
Updated the commit log.

v2: Fixed to display only on L3 resources.
Added the locks while processing.
Rename the displat to io_alloc_cbm (from sdciae_cmd).

---
 arch/x86/kernel/cpu/resctrl/ctrlmondata.c |  2 +-
 arch/x86/kernel/cpu/resctrl/internal.h    |  1 +
 arch/x86/kernel/cpu/resctrl/rdtgroup.c    | 51 ++++++++++++++++++++++-
 3 files changed, 52 insertions(+), 2 deletions(-)

arch/x86/kernel/cpu/resctrl/ctrlmondata.c +1 -1

index 0a0ac5f6112e..d1a59b56a456 100644
@@ -454,7 +454,7 @@ u32 resctrl_arch_get_config(struct rdt_resource *r, struct rdt_ctrl_domain *d,
 	return hw_dom->ctrl_val[idx];
 }
 
-static void show_doms(struct seq_file *s, struct resctrl_schema *schema, int closid)
+void show_doms(struct seq_file *s, struct resctrl_schema *schema, int closid)
 {
 	struct rdt_resource *r = schema->res;
 	struct rdt_ctrl_domain *dom;

arch/x86/kernel/cpu/resctrl/internal.h +1 -0

index 6ead222904fe..2ac78650500a 100644
@@ -519,6 +519,7 @@ void resctrl_file_fflags_init(const char *config, unsigned long fflags);
 void rdt_staged_configs_clear(void);
 bool closid_allocated(unsigned int closid);
 int resctrl_find_cleanest_closid(void);
+void show_doms(struct seq_file *s, struct resctrl_schema *schema, int closid);
 
 #ifdef CONFIG_RESCTRL_FS_PSEUDO_LOCK
 int rdtgroup_locksetup_enter(struct rdtgroup *rdtgrp);

arch/x86/kernel/cpu/resctrl/rdtgroup.c +50 -1

71 lines not on this page · raw message

Signature · 1 line

2.34.1

Babu Moger

author reviewer

2025-04-21 22:45 UTC #9 lore raw

PATCH v4 7/8 x86/resctrl: Modify rdt_parse_data to pass mode and CLOSID

In reply to Babu Moger

The functions parse_cbm() and parse_bw() require mode and CLOSID to validate the Capacity Bit Mask (CBM). It is passed through struct rdtgroup in rdt_parse_data. Instead of passing them through struct rdtgroup, pass mode and closid directly.

This change enables parse_cbm() to be used for verifying CBM in io_alloc feature.

Signed-off-by Babu Moger <[email protected]>

---
v4: New patch to call parse_cbm() directly to avoid code duplication.

---
 arch/x86/kernel/cpu/resctrl/ctrlmondata.c | 29 ++++++++++-------------
 arch/x86/kernel/cpu/resctrl/internal.h    |  6 +++++
 2 files changed, 19 insertions(+), 16 deletions(-)

arch/x86/kernel/cpu/resctrl/ctrlmondata.c +13 ~~-16~~

index d1a59b56a456..e5d1e77e1995 100644
@@ -23,11 +23,6 @@
 
 #include "internal.h"
 
-struct rdt_parse_data {
-	struct rdtgroup		*rdtgrp;
-	char			*buf;
-};
-
 typedef int (ctrlval_parser_t)(struct rdt_parse_data *data,
 			       struct resctrl_schema *s,
 			       struct rdt_ctrl_domain *d);

@@ -77,8 +72,8 @@ static int parse_bw(struct rdt_parse_data *data, struct resctrl_schema *s,
 		    struct rdt_ctrl_domain *d)
 {
 	struct resctrl_staged_config *cfg;
-	u32 closid = data->rdtgrp->closid;
 	struct rdt_resource *r = s->res;
+	u32 closid = data->closid;
 	u32 bw_val;
 
 	cfg = &d->staged_config[s->conf_type];

@@ -156,9 +151,10 @@ static bool cbm_validate(char *buf, u32 *data, struct rdt_resource *r)
 static int parse_cbm(struct rdt_parse_data *data, struct resctrl_schema *s,
 		     struct rdt_ctrl_domain *d)
 {
-	struct rdtgroup *rdtgrp = data->rdtgrp;
+	enum rdtgrp_mode mode = data->mode;
 	struct resctrl_staged_config *cfg;
 	struct rdt_resource *r = s->res;
+	u32 closid = data->closid;
 	u32 cbm_val;
 
 	cfg = &d->staged_config[s->conf_type];

@@ -171,7 +167,7 @@ static int parse_cbm(struct rdt_parse_data *data, struct resctrl_schema *s,
 	 * Cannot set up more than one pseudo-locked region in a cache
 	 * hierarchy.
 	 */
-	if (rdtgrp->mode == RDT_MODE_PSEUDO_LOCKSETUP &&
+	if (mode == RDT_MODE_PSEUDO_LOCKSETUP &&
 	    rdtgroup_pseudo_locked_in_hierarchy(d)) {
 		rdt_last_cmd_puts("Pseudo-locked region in hierarchy\n");
 		return -EINVAL;

@@ -180,9 +176,9 @@ static int parse_cbm(struct rdt_parse_data *data, struct resctrl_schema *s,
 	if (!cbm_validate(data->buf, &cbm_val, r))
 		return -EINVAL;
 
-	if ((rdtgrp->mode == RDT_MODE_EXCLUSIVE ||
-	     rdtgrp->mode == RDT_MODE_SHAREABLE) &&
-	    rdtgroup_cbm_overlaps_pseudo_locked(d, cbm_val)) {
+	if ((mode == RDT_MODE_EXCLUSIVE ||
+	     mode == RDT_MODE_SHAREABLE) &&
+	     rdtgroup_cbm_overlaps_pseudo_locked(d, cbm_val)) {
 		rdt_last_cmd_puts("CBM overlaps with pseudo-locked region\n");
 		return -EINVAL;
 	}

@@ -191,14 +187,14 @@ static int parse_cbm(struct rdt_parse_data *data, struct resctrl_schema *s,
 	 * The CBM may not overlap with the CBM of another closid if
 	 * either is exclusive.
 	 */
-	if (rdtgroup_cbm_overlaps(s, d, cbm_val, rdtgrp->closid, true)) {
+	if (rdtgroup_cbm_overlaps(s, d, cbm_val, closid, true)) {
 		rdt_last_cmd_puts("Overlaps with exclusive group\n");
 		return -EINVAL;
 	}
 
-	if (rdtgroup_cbm_overlaps(s, d, cbm_val, rdtgrp->closid, false)) {
-		if (rdtgrp->mode == RDT_MODE_EXCLUSIVE ||
-		    rdtgrp->mode == RDT_MODE_PSEUDO_LOCKSETUP) {
+	if (rdtgroup_cbm_overlaps(s, d, cbm_val, closid, false)) {
+		if (mode == RDT_MODE_EXCLUSIVE ||
+		    mode == RDT_MODE_PSEUDO_LOCKSETUP) {
 			rdt_last_cmd_puts("Overlaps with other group\n");
 			return -EINVAL;
 		}

@@ -262,7 +258,8 @@ static int parse_line(char *line, struct resctrl_schema *s,
 	list_for_each_entry(d, &r->ctrl_domains, hdr.list) {
 		if (d->hdr.id == dom_id) {
 			data.buf = dom;
-			data.rdtgrp = rdtgrp;
+			data.closid = rdtgrp->closid;
+			data.mode = rdtgrp->mode;
 			if (parse_ctrlval(&data, s, d))
 				return -EINVAL;
 			if (rdtgrp->mode ==  RDT_MODE_PSEUDO_LOCKSETUP) {

arch/x86/kernel/cpu/resctrl/internal.h +6 -0

index 2ac78650500a..92246d2b91c8 100644
@@ -241,6 +241,12 @@ struct rdtgroup {
 	struct pseudo_lock_region	*plr;
 };
 
+struct rdt_parse_data {
+	u32			closid;
+	enum rdtgrp_mode	mode;
+	char			*buf;
+};
+
 /* rdtgroup.flags */
 #define	RDT_DELETED		1

Signature · 1 line

2.34.1

Babu Moger

author reviewer

2025-04-21 22:45 UTC #10 lore raw

PATCH v4 8/8 x86/resctrl: Introduce interface to modify io_alloc Capacity Bit Masks

In reply to Babu Moger

"io_alloc" feature is a mechanism that enables direct insertion of data from I/O devices into the L3 cache. By directly caching data from I/O devices rather than first storing the I/O data in DRAM, it reduces the demands on DRAM bandwidth and reduces latency to the processor consuming the I/O data.

io_alloc feature uses the highest CLOSID to route the traffic from I/O devices. Provide the interface to modify io_alloc CBMs (Capacity Bit Mask) when feature is enabled.

Signed-off-by Babu Moger <[email protected]>

---
v4: Removed resctrl_io_alloc_parse_cbm and called parse_cbm() directly.

v3: Minor changes due to changes in resctrl_arch_get_io_alloc_enabled()
and resctrl_io_alloc_closid_get().
Taken care of handling the CBM update when CDP is enabled. Updated the commit log to make it generic.

v2: Added more generic text in documentation.

---
 Documentation/arch/x86/resctrl.rst        |  21 +++++
 arch/x86/kernel/cpu/resctrl/ctrlmondata.c |   4 +-
 arch/x86/kernel/cpu/resctrl/internal.h    |   2 +
 arch/x86/kernel/cpu/resctrl/rdtgroup.c    | 108 +++++++++++++++++++++-
 4 files changed, 132 insertions(+), 3 deletions(-)

Documentation/arch/x86/resctrl.rst +21 -0

index 7672c5c52c1a..6fdea77a1675 100644
@@ -169,6 +169,27 @@ related to allocation:
 		When CDP is enabled, io_alloc routes I/O traffic using the highest
 		CLOSID allocated for the instruction cache (L3CODE).
 
+"io_alloc_cbm":
+		Capacity Bit Masks (CBMs) available to supported IO devices which
+		can directly insert cache lines in L3 which can help to reduce the
+		latency. CBM can be configured by writing to the interface in the
+		following format::
+
+			<resource_name>:<cache_id0>=<cbm>;<cache_id1>=<cbm>;...
+
+		Example::
+
+			# cat /sys/fs/resctrl/info/L3/io_alloc_cbm
+			L3:0=ffff;1=ffff
+
+			# echo L3:1=FF > /sys/fs/resctrl/info/L3/io_alloc_cbm
+			# cat /sys/fs/resctrl/info/L3/io_alloc_cbm
+			L3:0=ffff;1=00ff
+
+		When CDP is enabled, L3 control is divided into two separate resources:
+		L3CODE and L3DATA. However, the CBM can only be updated on the L3CODE
+		resource.
+
 Memory bandwidth(MB) subdirectory contains the following files
 with respect to allocation:

arch/x86/kernel/cpu/resctrl/ctrlmondata.c +2 -2

index e5d1e77e1995..315584415cc4 100644
@@ -148,8 +148,8 @@ static bool cbm_validate(char *buf, u32 *data, struct rdt_resource *r)
  * Read one cache bit mask (hex). Check that it is valid for the current
  * resource type.
  */
-static int parse_cbm(struct rdt_parse_data *data, struct resctrl_schema *s,
-		     struct rdt_ctrl_domain *d)
+int parse_cbm(struct rdt_parse_data *data, struct resctrl_schema *s,
+	      struct rdt_ctrl_domain *d)
 {
 	enum rdtgrp_mode mode = data->mode;
 	struct resctrl_staged_config *cfg;

arch/x86/kernel/cpu/resctrl/internal.h +2 -0

index 92246d2b91c8..1d3b60741a39 100644
@@ -526,6 +526,8 @@ void rdt_staged_configs_clear(void);
 bool closid_allocated(unsigned int closid);
 int resctrl_find_cleanest_closid(void);
 void show_doms(struct seq_file *s, struct resctrl_schema *schema, int closid);
+int parse_cbm(struct rdt_parse_data *data, struct resctrl_schema *s,
+	      struct rdt_ctrl_domain *d);
 
 #ifdef CONFIG_RESCTRL_FS_PSEUDO_LOCK
 int rdtgroup_locksetup_enter(struct rdtgroup *rdtgrp);

arch/x86/kernel/cpu/resctrl/rdtgroup.c +107 -1

122 lines not on this page · raw message

Signature · 1 line

2.34.1

Reinette Chatre

maintainer

2025-05-02 21:20 UTC #11 lore raw

In reply to Babu Moger

Hi Babu,

Quoted 37 lines · Babu Moger

# Linux Implementation
Feature adds following interface files when the resctrl "io_alloc" feature is supported on L3 resource:
/sys/fs/resctrl/info/L3/io_alloc: Report the feature status. Enable/disable the
feature by writing to the interface.
/sys/fs/resctrl/info/L3/io_alloc_cbm: List the Capacity Bit Masks (CBMs) available
for I/O devices when io_alloc feature is enabled. Configure the CBM by writing to the interface.
# Examples:
a. Check if io_alloc feature is available
#mount -t resctrl resctrl /sys/fs/resctrl/
# cat /sys/fs/resctrl/info/L3/io_alloc
disabled
b. Enable the io_alloc feature.
	# echo 1 > /sys/fs/resctrl/info/L3/io_alloc 
	# cat /sys/fs/resctrl/info/L3/io_alloc
	enabled
c. Check the CBM values for the io_alloc feature.
	# cat /sys/fs/resctrl/info/L3/io_alloc_cbm 
	L3:0=ffff;1=ffff
d. Change the CBM value for the domain 1:
	# echo L3:1=FF > /sys/fs/resctrl/info/L3/io_alloc_cbm
	# cat /sys/fs/resctrl/info/L3/io_alloc_cbm 
	L3:0=ffff;1=00ff

Quoted 8 lines

d. Disable io_alloc feature and exit.
	# echo 0 > /sys/fs/resctrl/info/L3/io_alloc 
	# cat /sys/fs/resctrl/info/L3/io_alloc
	disabled
#umount /sys/fs/resctrl/

From what I can tell the interface when CDP is enabled will look as follows:

 	# mount -o cdp -t resctrl resctrl /sys/fs/resctrl/
 	# cat /sys/fs/resctrl/info/L3CODE/io_alloc
 	disabled 
 	# cat /sys/fs/resctrl/info/L3DATA/io_alloc
 	not supported

"io_alloc" can thus be enabled for L3CODE but not for L3DATA. This is unexpected considering the feature is called "L3 Smart *Data* Cache Injection Allocation Enforcement".

I understand that the interface evolved into this because the "code" allocation of CDP uses the CLOSID required by SDCIAE but I think leaking implementation details like this to the user interface can cause confusion.

Since there is no distinction between code and data in these IO allocations, what do you think of connecting the io_alloc and io_alloc_cbm files within L3CODE and L3DATA so that the user can read/write from either with a read showing the same data and user able to write to either? For example,

 	# mount -o cdp -t resctrl resctrl /sys/fs/resctrl/
 	# cat /sys/fs/resctrl/info/L3CODE/io_alloc
 	disabled
 	# cat /sys/fs/resctrl/info/L3DATA/io_alloc
 	disabled
	# echo 1 > /sys/fs/resctrl/info/L3CODE/io_alloc
 	# cat /sys/fs/resctrl/info/L3CODE/io_alloc
 	enabled
 	# cat /sys/fs/resctrl/info/L3DATA/io_alloc
 	enabled
 	# cat /sys/fs/resctrl/info/L3DATA/io_alloc_cbm 
 	0=ffff;1=ffff
 	# cat /sys/fs/resctrl/info/L3CODE/io_alloc_cbm 
 	0=ffff;1=ffff
 	# echo 1=FF > /sys/fs/resctrl/info/L3DATA/io_alloc_cbm
 	# cat /sys/fs/resctrl/info/L3DATA/io_alloc_cbm 
 	0=ffff;1=00ff
 	# cat /sys/fs/resctrl/info/L3CODE/io_alloc_cbm 
 	0=ffff;1=00ff

(Note in above I removed the resource name from io_alloc_cbm to match what was discussed during previous version:
https://lore.kernel.org/lkml/[email protected]/ )

What do you think?

Quoted 4 lines

---
v4: The "io_alloc" interface will report "enabled/disabled/not supported"
instead of 0 or 1..

Quoted 8 lines

Updated resctrl_io_alloc_closid_get() to verify the max closid availability using closids_supported().
Updated the documentation for "shareable_bits" and "bit_usage".
NOTE: io_alloc is about specific CLOS. rdt_bit_usage_show() is not designed handle bit_usage for specific CLOS. Its about overall system. So, we cannot really tell the user which CLOS is shared across both hardware and software.

"bit_usage" is not about CLOS but how the resource is used. Per the doc:

"bit_usage":
Annotated capacity bitmasks showing how all instances of the resource are used.

The key here is the CBM, not CLOS. For each bit in the *CBM* "bit_usage" shows how that portion of the cache is used with the legend documented in Documentation/arch/x86/resctrl.rst.

Consider a system with the following allocations:

# cat /sys/fs/resctrl/schemata
L3:0=0ff0
# cat /sys/fs/resctrl/info/L3/io_alloc_cbm 
0=ff00

Then "bit_usage" will look like:

# cat /sys/fs/resctrl/info/L3/bit_usage
0=HHHHXXXXSSSS0000

"bit_usage" shows how the cache is being used. It shows that the portion of cache represented by first four bits of CBM is unused, portion of cache represented by bits 4 to 7 of CBM is only used by software, portion of cache represented by bits 8 to 11 of CBM is shared between software and hardware, portion of cache represented by bits 12 to 15 is only used by hardware.

Quoted 1 line

This is something we need to discuss.

Looking at implementation in patch #5 the "io_alloc_cbm" bits of CBM are presented as software bits, since "io_alloc_cbm" represents IO from devices it should be "hardware" bits (hw_shareable), no?

Reinette

Moger, Babu

2025-05-03 00:53 UTC #12 lore raw

Hi Reinette,

Thanks for quick turnaround.

Quoted 84 lines · Reinette Chatre

Hi Babu,
Quoted 39 lines · Babu Moger
# Linux Implementation
Feature adds following interface files when the resctrl "io_alloc" feature is supported on L3 resource:
/sys/fs/resctrl/info/L3/io_alloc: Report the feature status. Enable/disable the
feature by writing to the interface.
/sys/fs/resctrl/info/L3/io_alloc_cbm: List the Capacity Bit Masks (CBMs) available
for I/O devices when io_alloc feature is enabled. Configure the CBM by writing to the interface.
# Examples:
a. Check if io_alloc feature is available
#mount -t resctrl resctrl /sys/fs/resctrl/
# cat /sys/fs/resctrl/info/L3/io_alloc
disabled
b. Enable the io_alloc feature.
	# echo 1 > /sys/fs/resctrl/info/L3/io_alloc
	# cat /sys/fs/resctrl/info/L3/io_alloc
	enabled
c. Check the CBM values for the io_alloc feature.
	# cat /sys/fs/resctrl/info/L3/io_alloc_cbm
	L3:0=ffff;1=ffff
d. Change the CBM value for the domain 1:
	# echo L3:1=FF > /sys/fs/resctrl/info/L3/io_alloc_cbm
	# cat /sys/fs/resctrl/info/L3/io_alloc_cbm
	L3:0=ffff;1=00ff
d. Disable io_alloc feature and exit.
Quoted 8 lines
	# echo 0 > /sys/fs/resctrl/info/L3/io_alloc
	# cat /sys/fs/resctrl/info/L3/io_alloc
	disabled
#umount /sys/fs/resctrl/
From what I can tell the interface when CDP is enabled will look
as follows:
  	# mount -o cdp -t resctrl resctrl /sys/fs/resctrl/
  	# cat /sys/fs/resctrl/info/L3CODE/io_alloc
  	disabled
  	# cat /sys/fs/resctrl/info/L3DATA/io_alloc
  	not supported
"io_alloc" can thus be enabled for L3CODE but not for L3DATA. This is unexpected considering the feature is called "L3 Smart *Data* Cache Injection Allocation Enforcement".
I understand that the interface evolved into this because the "code" allocation of CDP uses the CLOSID required by SDCIAE but I think leaking implementation details like this to the user interface can cause confusion.
Since there is no distinction between code and data in these IO allocations, what do you think of connecting the io_alloc and io_alloc_cbm files within L3CODE and L3DATA so that the user can read/write from either with a read showing the same data and user able to write to either? For example,
  	# mount -o cdp -t resctrl resctrl /sys/fs/resctrl/
  	# cat /sys/fs/resctrl/info/L3CODE/io_alloc
  	disabled
  	# cat /sys/fs/resctrl/info/L3DATA/io_alloc
  	disabled
	# echo 1 > /sys/fs/resctrl/info/L3CODE/io_alloc
  	# cat /sys/fs/resctrl/info/L3CODE/io_alloc
  	enabled
  	# cat /sys/fs/resctrl/info/L3DATA/io_alloc
  	enabled
  	# cat /sys/fs/resctrl/info/L3DATA/io_alloc_cbm

Quoted 8 lines

  	0=ffff;1=ffff
  	# cat /sys/fs/resctrl/info/L3CODE/io_alloc_cbm
  	0=ffff;1=ffff
  	# echo 1=FF > /sys/fs/resctrl/info/L3DATA/io_alloc_cbm
  	# cat /sys/fs/resctrl/info/L3DATA/io_alloc_cbm
  	0=ffff;1=00ff
  	# cat /sys/fs/resctrl/info/L3CODE/io_alloc_cbm
  	0=ffff;1=00ff

I agree. There is no right or wrong here. It can be done this way like you mentioned above. But I am not sure if will clear the confusion.

We have already added the text in user doc (also spec says the same).

"On AMD systems, the io_alloc feature is supported by the L3 Smart Data Cache Injection Allocation Enforcement (SDCIAE). The CLOSID for io_alloc is determined by the highest CLOSID supported by the resource. When CDP is enabled, io_alloc routes I/O traffic using the highest CLOSID allocated for the instruction cache (L3CODE).

Dont you think this text might clear the confusion? We can add examples also if that makes it even more clear.

Quoted 5 lines

(Note in above I removed the resource name from io_alloc_cbm to match what was discussed during previous version:
https://lore.kernel.org/lkml/[email protected]/ ) What do you think?

Yes. I remember. "Kept the resource name while printing the CBM for io_alloc, so we dont have to change show_doms() just for this feature and it is consistant across all the schemata display.

I added the note in here.
https://lore.kernel.org/lkml/784fbc61e02e9a834473c3476ee196ef6a44e338.1745275431.git.babu.moger@amd.com/

I will change it if you feel strongly about it. We will have to change show_doms() to handle this.

Quoted 21 lines

Quoted 6 lines
---
v4: The "io_alloc" interface will report "enabled/disabled/not supported"
instead of 0 or 1..
Quoted 8 lines
Updated resctrl_io_alloc_closid_get() to verify the max closid availability using closids_supported().
Updated the documentation for "shareable_bits" and "bit_usage".
NOTE: io_alloc is about specific CLOS. rdt_bit_usage_show() is not designed handle bit_usage for specific CLOS. Its about overall system. So, we cannot really tell the user which CLOS is shared across both hardware and software.
"bit_usage" is not about CLOS but how the resource is used. Per the doc:
"bit_usage":
Annotated capacity bitmasks showing how all instances of the resource are used.

Quoted 7 lines

The key here is the CBM, not CLOS. For each bit in the *CBM* "bit_usage" shows how that portion of the cache is used with the legend documented in Documentation/arch/x86/resctrl.rst.
Consider a system with the following allocations:
# cat /sys/fs/resctrl/schemata
L3:0=0ff0

This is CLOS 0.

Quoted 2 lines

# cat /sys/fs/resctrl/info/L3/io_alloc_cbm
0=ff00

This is CLOS 15.

Quoted 5 lines

Then "bit_usage" will look like:
# cat /sys/fs/resctrl/info/L3/bit_usage
0=HHHHXXXXSSSS0000

It is confusing here. To make it clear we may have to print all the CLOSes in each domain.

# cat /sys/fs/resctrl/info/L3/bit_usage
DOM0=CLOS0:SSSSSSSSSSSSSSSS;... ;CLOS15=HHHHXXXXSSSS0000;
DOM1=CLOS0:SSSSSSSSSSSSSSSS;... ;CLOS15=HHHHXXXXSSSS0000

Quoted 6 lines

"bit_usage" shows how the cache is being used. It shows that the portion of cache represented by first four bits of CBM is unused, portion of cache represented by bits 4 to 7 of CBM is only used by software, portion of cache represented by bits 8 to 11 of CBM is shared between software and hardware, portion of cache represented by bits 12 to 15 is only used by hardware.

Quoted 6 lines

Quoted 1 line
This is something we need to discuss.
Looking at implementation in patch #5 the "io_alloc_cbm" bits of CBM are presented as software bits, since "io_alloc_cbm" represents IO from devices it should be "hardware" bits (hw_shareable), no?

Yes. It is. But logic is bit different there.

It loops thru all the CLOSes on the domain. So, it will print again like this below.

#cat bit_usage
0=HHHHXXXXSSSS0000

It tells the user that all the CLOSes in domain 0 has this sharing propery which is not correct.

To make it clear we really need to print every CLOS here. What do you think?

Thanks
Babu

Reinette Chatre

maintainer

2025-05-05 16:22 UTC #13 lore raw

Hi Babu,

Quoted 101 lines · Babu

Hi Reinette,
Thanks for quick turnaround.
Quoted 83 lines · Reinette Chatre
Hi Babu,
Quoted 39 lines · Babu Moger
# Linux Implementation
Feature adds following interface files when the resctrl "io_alloc" feature is supported on L3 resource:
/sys/fs/resctrl/info/L3/io_alloc: Report the feature status. Enable/disable the feature by writing to the interface.
/sys/fs/resctrl/info/L3/io_alloc_cbm: List the Capacity Bit Masks (CBMs) available for I/O devices when io_alloc feature is enabled. Configure the CBM by writing to the interface.
# Examples:
a. Check if io_alloc feature is available
#mount -t resctrl resctrl /sys/fs/resctrl/
    # cat /sys/fs/resctrl/info/L3/io_alloc
    disabled
b. Enable the io_alloc feature.
    # echo 1 > /sys/fs/resctrl/info/L3/io_alloc
    # cat /sys/fs/resctrl/info/L3/io_alloc
    enabled
c. Check the CBM values for the io_alloc feature.
    # cat /sys/fs/resctrl/info/L3/io_alloc_cbm
    L3:0=ffff;1=ffff
d. Change the CBM value for the domain 1:
    # echo L3:1=FF > /sys/fs/resctrl/info/L3/io_alloc_cbm
    # cat /sys/fs/resctrl/info/L3/io_alloc_cbm
    L3:0=ffff;1=00ff
d. Disable io_alloc feature and exit.
Quoted 8 lines
    # echo 0 > /sys/fs/resctrl/info/L3/io_alloc
    # cat /sys/fs/resctrl/info/L3/io_alloc
    disabled
#umount /sys/fs/resctrl/
From what I can tell the interface when CDP is enabled will look
as follows:
      # mount -o cdp -t resctrl resctrl /sys/fs/resctrl/
      # cat /sys/fs/resctrl/info/L3CODE/io_alloc
      disabled
      # cat /sys/fs/resctrl/info/L3DATA/io_alloc
      not supported
  "io_alloc" can thus be enabled for L3CODE but not for L3DATA.
This is unexpected considering the feature is called
"L3 Smart *Data* Cache Injection Allocation Enforcement".
I understand that the interface evolved into this because the "code" allocation of CDP uses the CLOSID required by SDCIAE but I think leaking implementation details like this to the user interface can cause confusion.
Since there is no distinction between code and data in these IO allocations, what do you think of connecting the io_alloc and io_alloc_cbm files within L3CODE and L3DATA so that the user can read/write from either with a read showing the same data and user able to write to either? For example,
      # mount -o cdp -t resctrl resctrl /sys/fs/resctrl/
      # cat /sys/fs/resctrl/info/L3CODE/io_alloc
      disabled
      # cat /sys/fs/resctrl/info/L3DATA/io_alloc
      disabled
    # echo 1 > /sys/fs/resctrl/info/L3CODE/io_alloc
      # cat /sys/fs/resctrl/info/L3CODE/io_alloc
      enabled
      # cat /sys/fs/resctrl/info/L3DATA/io_alloc
      enabled
      # cat /sys/fs/resctrl/info/L3DATA/io_alloc_cbm
Quoted 8 lines
      0=ffff;1=ffff
      # cat /sys/fs/resctrl/info/L3CODE/io_alloc_cbm
      0=ffff;1=ffff
      # echo 1=FF > /sys/fs/resctrl/info/L3DATA/io_alloc_cbm
      # cat /sys/fs/resctrl/info/L3DATA/io_alloc_cbm
      0=ffff;1=00ff
      # cat /sys/fs/resctrl/info/L3CODE/io_alloc_cbm
      0=ffff;1=00ff
I agree. There is no right or wrong here. It can be done this way like you mentioned above. But I am not sure if will clear the confusion.
We have already added the text in user doc (also spec says the same).

Quoted 7 lines

"On AMD systems, the io_alloc feature is supported by the L3 Smart Data Cache Injection Allocation Enforcement (SDCIAE). The CLOSID for io_alloc is determined by the highest CLOSID supported by the resource. When CDP is enabled, io_alloc routes I/O traffic using the highest CLOSID allocated for the instruction cache (L3CODE).
Dont you think this text might clear the confusion? We can add examples also if that makes it even more clear.

The user interface is not intended to be a mirror of the hardware interface. If it was, doing so is becoming increasingly difficult with multiple architectures with different hardware intefaces needing to use the same user interface for control. Remember, there are no "CLOSID" in MPAM and I do not know details of what RISC-V brings.

We should aim to have something as generic as possible that makes sense for user space. All the hardware interface details should be hidden as much as possible from user interface. When we expose the hardware interface details it becomes very difficult to support new use cases.

The only aspect of "closids" that has been exposed to user space thus far is the "num_closids" and in user documentation a CLOSid has been linked to the number of control groups. That is the only constraint we need to think about here. I have repeatedly asked for IO alloc connection with CLOSIDs to not be exposed to user space (yet user documentation and messages to user space keeps doing so in this series). Support for IO alloc in this way is unique to AMD. We do not want resctrl to be constrained like this if another architecture needs to support some form of IO alloc and does so in a different way.

I understand that IO alloc backed by CLOSID is forming part of resctrl fs in this implementation and that is ok for now. As long as we do not leak this to user space it gives use flexibility to change resctrl fs when/if we learn different architecture needs later.

Quoted 6 lines

Quoted 4 lines
(Note in above I removed the resource name from io_alloc_cbm to match what was discussed during previous version:
https://lore.kernel.org/lkml/[email protected]/ ) What do you think?
Yes. I remember. "Kept the resource name while printing the CBM for io_alloc, so we dont have to change show_doms() just for this feature and it is consistant across all the schemata display.

It almost sounds like you do not want to implement something because the code to support it does not exist?

Quoted 3 lines

I added the note in here.
https://lore.kernel.org/lkml/784fbc61e02e9a834473c3476ee196ef6a44e338.1745275431.git.babu.moger@amd.com/

You mention "I dont have to change show_doms() just for this feature and it is consistant across all the schemata display."
I am indeed seeing a pattern where one goal is to add changes by changing minimum amount of code. Please let this not be a goal but instead make it a goal to integrate changes into resctrl appropriately, not just pasted on top.

When it comes to the schemata display then it makes sense to add the resource name since the schemata file is within a resource group containing multiple resources and the schemata file thus needs to identify resources. Compare this to, for example, the "bit_usage" file that is unique to a resource and thus no need to identify the resource.

Quoted 2 lines

I will change it if you feel strongly about it. We will have to change show_doms() to handle this.

What is the problem with changing show_doms()?

Quoted 38 lines

Quoted 22 lines
Quoted 4 lines
---
v4: The "io_alloc" interface will report "enabled/disabled/not supported" instead of 0 or 1..
Quoted 8 lines
     Updated resctrl_io_alloc_closid_get() to verify the max closid availability using closids_supported().
     Updated the documentation for "shareable_bits" and "bit_usage".
     NOTE: io_alloc is about specific CLOS. rdt_bit_usage_show() is not designed handle bit_usage for specific CLOS. Its about overall system. So, we cannot really tell the user which CLOS is shared across both hardware and software.
"bit_usage" is not about CLOS but how the resource is used. Per the doc:
"bit_usage":
        Annotated capacity bitmasks showing how all instances of the resource are used.
Quoted 7 lines
The key here is the CBM, not CLOS. For each bit in the *CBM* "bit_usage" shows how that portion of the cache is used with the legend documented in Documentation/arch/x86/resctrl.rst.
Consider a system with the following allocations:
# cat /sys/fs/resctrl/schemata
L3:0=0ff0
This is CLOS 0.
Quoted 2 lines
# cat /sys/fs/resctrl/info/L3/io_alloc_cbm
0=ff00
This is CLOS 15.
Quoted 1 line

Quoted 6 lines

Quoted 4 lines
Then "bit_usage" will look like:
# cat /sys/fs/resctrl/info/L3/bit_usage
0=HHHHXXXXSSSS0000
It is confusing here. To make it clear we may have to print all the CLOSes in each domain.

Could you please elaborate how this is confusing?

Quoted 4 lines

# cat /sys/fs/resctrl/info/L3/bit_usage
DOM0=CLOS0:SSSSSSSSSSSSSSSS;... ;CLOS15=HHHHXXXXSSSS0000;
DOM1=CLOS0:SSSSSSSSSSSSSSSS;... ;CLOS15=HHHHXXXXSSSS0000

Please no. Not just does this change existing user interface it also breaks the goal of "bit_usage".

Please think of it from user perspective. If user wants to know, for example, "how is my L3 cache allocated" then the "bit_usage" file provides that summary.

Quoted 7 lines

Quoted 7 lines
"bit_usage" shows how the cache is being used. It shows that the portion of cache represented by first four bits of CBM is unused, portion of cache represented by bits 4 to 7 of CBM is only used by software, portion of cache represented by bits 8 to 11 of CBM is shared between software and hardware, portion of cache represented by bits 12 to 15 is only used by hardware.
Quoted 1 line
This is something we need to discuss.

Quoted 7 lines

Quoted 4 lines
Looking at implementation in patch #5 the "io_alloc_cbm" bits of CBM are presented as software bits, since "io_alloc_cbm" represents IO from devices it should be "hardware" bits (hw_shareable), no?
Yes. It is. But logic is bit different there.
It loops thru all the CLOSes on the domain. So, it will print again like this below.

This is what current code does, but the code can be changed, no? For example, rdt_bit_usage_show() does not need to treat the IO allocation like all the other resource groups but instead handle it separately. Below us some pseudo code that presents the idea, untested, not compiled.

	hw_shareable = r->cache.shareable_bits;

	for (i = 0; i < closids_supported(); i++) {
		if (!closid_allocated(i) || 
		    (resctrl_arch_get_io_alloc_enabled(r) && i == resctrl_io_alloc_closid_get(r, s)))
			continue;

		/*  Intitialize sw_shareable and exclusive */
	}

	if (resctrl_arch_get_io_alloc_enabled(r)) {
		 /*
		  * Sidenote: I do not think schemata parameter is needed for
		  * resctrl_io_alloc_closid_get()
		  */
		io_alloc_closid = resctrl_io_alloc_closid_get(r, s);
		if (resctrl_arch_get_cdp_enabled(r->rid))
			ctrl_val = resctrl_arch_get_config(r, dom, io_alloc_closid, CDP_CODE);
		else
			ctrl_val = resctrl_arch_get_config(r, dom, io_alloc_closid, CDP_NONE);
		hw_shareable |= ctrl_val;
	}

	for (i = r->cache.cbm_len - 1; i >= 0; i--) {
		/* Write annotated bitmask to user space */
	}

Quoted 7 lines

#cat bit_usage
0=HHHHXXXXSSSS0000
It tells the user that all the CLOSes in domain 0 has this sharing propery which is not correct.
To make it clear we really need to print every CLOS here. What do you think?

No. We cannot just change user space API like this. The way I see it the implementation can support existing user space API. I am sure the above can be improved but it presents an idea that we can use to start with.

Reinette

Luck, Tony

maintainer

2025-05-05 17:02 UTC #14 lore raw

Quoted 8 lines

The only aspect of "closids" that has been exposed to user space thus far is the "num_closids" and in user documentation a CLOSid has been linked to the number of control groups. That is the only constraint we need to think about here. I have repeatedly asked for IO alloc connection with CLOSIDs to not be exposed to user space (yet user documentation and messages to user space keeps doing so in this series). Support for IO alloc in this way is unique to AMD. We do not want resctrl to be constrained like this if another architecture needs to support some form of IO alloc and does so in a different way.

This isn't unique to AMD. Intel also ties CLOSid to control features associated with I/O (likewise with RMIDs for monitoring).

See the Intel RDT architecture specification[1] chapter 4.4:

" Non-CPU agent RDT uses the RMID and CLOS tags in the same way that they are used for CPU agents."

-Tony

[1] https://cdrdv2.intel.com/v1/dl/getContent/789566

Reinette Chatre

maintainer

2025-05-05 17:15 UTC #15 lore raw

Hi Tony,

Quoted 9 lines · Tony

Quoted 8 lines
The only aspect of "closids" that has been exposed to user space thus far is the "num_closids" and in user documentation a CLOSid has been linked to the number of control groups. That is the only constraint we need to think about here. I have repeatedly asked for IO alloc connection with CLOSIDs to not be exposed to user space (yet user documentation and messages to user space keeps doing so in this series). Support for IO alloc in this way is unique to AMD. We do not want resctrl to be constrained like this if another architecture needs to support some form of IO alloc and does so in a different way.

Quoted 6 lines

This isn't unique to AMD. Intel also ties CLOSid to control features associated with I/O (likewise with RMIDs for monitoring).
See the Intel RDT architecture specification[1] chapter 4.4:
" Non-CPU agent RDT uses the RMID and CLOS tags in the same way that they are used for CPU agents."

As I understand AMD uses a single specific (the highest CLOSid supported by L3) CLOS that is then reserved for IO allocation. While both Intel and AMD technically "uses CLOSid", it is done differently, no?

Specifically, is this documentation introduced in patch #5 accurate for Intel?
+ The feature routes the I/O traffic via specific CLOSID reserved
+ for io_alloc feature. By configuring the CBM (Capacity Bit Mask)
+ for the CLOSID, users can control the L3 portions available for
+ I/0 traffic. The reserved CLOSID will be excluded for group creation.

Reinette

Luck, Tony

maintainer

2025-05-05 17:27 UTC #16 lore raw

Review

Quoted 11 lines

Quoted 1 line
" Non-CPU agent RDT uses the RMID and CLOS tags in the same way that they are used for CPU agents."
As I understand AMD uses a single specific (the highest CLOSid supported by L3) CLOS that is then reserved for IO allocation. While both Intel and AMD technically "uses CLOSid", it is done differently, no?
Specifically, is this documentation introduced in patch #5 accurate for Intel?
(quoted diff) +4 -0
+             The feature routes the I/O traffic via specific CLOSID reserved
+             for io_alloc feature. By configuring the CBM (Capacity Bit Mask)
+             for the CLOSID, users can control the L3 portions available for
+             I/0 traffic. The reserved CLOSID will be excluded for group creation.

No. Intel doesn't reserve a single CLOS. It allows to assign RMIDs and CLOSids for I/O monitoring and control. Different IDs can be assigned to different groups of devices (the "grouping" is dependent on h/w routing to devices, not assignable by the OS).

I had some patches for this in my abandoned "resctrl2" implementation. No immediate plans to resurrect them since it became clear that the h/w implementation was model specific for just one generation.

-Tony

Reinette Chatre

maintainer

2025-05-05 17:40 UTC #17 lore raw

Hi Tony,

Quoted 12 lines · Tony

Review
Quoted 11 lines
Quoted 1 line
" Non-CPU agent RDT uses the RMID and CLOS tags in the same way that they are used for CPU agents."
As I understand AMD uses a single specific (the highest CLOSid supported by L3) CLOS that is then reserved for IO allocation. While both Intel and AMD technically "uses CLOSid", it is done differently, no?
Specifically, is this documentation introduced in patch #5 accurate for Intel?
(quoted diff) +4 -0
+             The feature routes the I/O traffic via specific CLOSID reserved
+             for io_alloc feature. By configuring the CBM (Capacity Bit Mask)
+             for the CLOSID, users can control the L3 portions available for
+             I/0 traffic. The reserved CLOSID will be excluded for group creation.

Quoted 4 lines

No. Intel doesn't reserve a single CLOS. It allows to assign RMIDs and CLOSids for I/O monitoring and control. Different IDs can be assigned to different groups of devices (the "grouping" is dependent on h/w routing to devices, not assignable by the OS).

How does this work with CDP on Intel? Can CDP be enabled for CPU agents while the "code" and "data" CLOSids be used for I/O control?

Reinette

Luck, Tony

maintainer

2025-05-05 17:50 UTC #18 lore raw

Quoted 7 lines

Quoted 4 lines
No. Intel doesn't reserve a single CLOS. It allows to assign RMIDs and CLOSids for I/O monitoring and control. Different IDs can be assigned to different groups of devices (the "grouping" is dependent on h/w routing to devices, not assignable by the OS).
How does this work with CDP on Intel? Can CDP be enabled for CPU agents while the "code" and "data" CLOSids be used for I/O control?

Reinette,

Good question. I'll have to check with h/w folks.

-Tony

Moger, Babu

author reviewer

2025-05-05 19:54 UTC #19 lore raw

In reply to Reinette Chatre

Hi Reinette,

Quoted 132 lines · Reinette Chatre

Hi Babu,
Quoted 101 lines · Babu
Hi Reinette,
Thanks for quick turnaround.
Quoted 83 lines · Reinette Chatre
Hi Babu,
Quoted 39 lines · Babu Moger
# Linux Implementation
Feature adds following interface files when the resctrl "io_alloc" feature is supported on L3 resource:
/sys/fs/resctrl/info/L3/io_alloc: Report the feature status. Enable/disable the feature by writing to the interface.
/sys/fs/resctrl/info/L3/io_alloc_cbm: List the Capacity Bit Masks (CBMs) available for I/O devices when io_alloc feature is enabled. Configure the CBM by writing to the interface.
# Examples:
a. Check if io_alloc feature is available
#mount -t resctrl resctrl /sys/fs/resctrl/
    # cat /sys/fs/resctrl/info/L3/io_alloc
    disabled
b. Enable the io_alloc feature.
    # echo 1 > /sys/fs/resctrl/info/L3/io_alloc
    # cat /sys/fs/resctrl/info/L3/io_alloc
    enabled
c. Check the CBM values for the io_alloc feature.
    # cat /sys/fs/resctrl/info/L3/io_alloc_cbm
    L3:0=ffff;1=ffff
d. Change the CBM value for the domain 1:
    # echo L3:1=FF > /sys/fs/resctrl/info/L3/io_alloc_cbm
    # cat /sys/fs/resctrl/info/L3/io_alloc_cbm
    L3:0=ffff;1=00ff
d. Disable io_alloc feature and exit.
Quoted 8 lines
    # echo 0 > /sys/fs/resctrl/info/L3/io_alloc
    # cat /sys/fs/resctrl/info/L3/io_alloc
    disabled
#umount /sys/fs/resctrl/
From what I can tell the interface when CDP is enabled will look
as follows:
      # mount -o cdp -t resctrl resctrl /sys/fs/resctrl/
      # cat /sys/fs/resctrl/info/L3CODE/io_alloc
      disabled
      # cat /sys/fs/resctrl/info/L3DATA/io_alloc
      not supported
  "io_alloc" can thus be enabled for L3CODE but not for L3DATA.
This is unexpected considering the feature is called
"L3 Smart *Data* Cache Injection Allocation Enforcement".
I understand that the interface evolved into this because the "code" allocation of CDP uses the CLOSID required by SDCIAE but I think leaking implementation details like this to the user interface can cause confusion.
Since there is no distinction between code and data in these IO allocations, what do you think of connecting the io_alloc and io_alloc_cbm files within L3CODE and L3DATA so that the user can read/write from either with a read showing the same data and user able to write to either? For example,
      # mount -o cdp -t resctrl resctrl /sys/fs/resctrl/
      # cat /sys/fs/resctrl/info/L3CODE/io_alloc
      disabled
      # cat /sys/fs/resctrl/info/L3DATA/io_alloc
      disabled
    # echo 1 > /sys/fs/resctrl/info/L3CODE/io_alloc
      # cat /sys/fs/resctrl/info/L3CODE/io_alloc
      enabled
      # cat /sys/fs/resctrl/info/L3DATA/io_alloc
      enabled
      # cat /sys/fs/resctrl/info/L3DATA/io_alloc_cbm
Quoted 8 lines
      0=ffff;1=ffff
      # cat /sys/fs/resctrl/info/L3CODE/io_alloc_cbm
      0=ffff;1=ffff
      # echo 1=FF > /sys/fs/resctrl/info/L3DATA/io_alloc_cbm
      # cat /sys/fs/resctrl/info/L3DATA/io_alloc_cbm
      0=ffff;1=00ff
      # cat /sys/fs/resctrl/info/L3CODE/io_alloc_cbm
      0=ffff;1=00ff
I agree. There is no right or wrong here. It can be done this way like you mentioned above. But I am not sure if will clear the confusion.
We have already added the text in user doc (also spec says the same).
Quoted 7 lines
"On AMD systems, the io_alloc feature is supported by the L3 Smart Data Cache Injection Allocation Enforcement (SDCIAE). The CLOSID for io_alloc is determined by the highest CLOSID supported by the resource. When CDP is enabled, io_alloc routes I/O traffic using the highest CLOSID allocated for the instruction cache (L3CODE).
Dont you think this text might clear the confusion? We can add examples also if that makes it even more clear.
The user interface is not intended to be a mirror of the hardware interface. If it was, doing so is becoming increasingly difficult with multiple architectures with different hardware intefaces needing to use the same user interface for control. Remember, there are no "CLOSID" in MPAM and I do not know details of what RISC-V brings.
We should aim to have something as generic as possible that makes sense for user space. All the hardware interface details should be hidden as much as possible from user interface. When we expose the hardware interface details it becomes very difficult to support new use cases.
The only aspect of "closids" that has been exposed to user space thus far is the "num_closids" and in user documentation a CLOSid has been linked to the number of control groups. That is the only constraint we need to think about here. I have repeatedly asked for IO alloc connection with CLOSIDs to not be exposed to user space (yet user documentation and messages to user space keeps doing so in this series). Support for IO alloc in this way is unique to AMD. We do not want resctrl to be constrained like this if another architecture needs to support some form of IO alloc and does so in a different way.

Quoted 4 lines

I understand that IO alloc backed by CLOSID is forming part of resctrl fs in this implementation and that is ok for now. As long as we do not leak this to user space it gives use flexibility to change resctrl fs when/if we learn different architecture needs later.

That makes sense. I’ll go ahead and adjust it as suggested.

Quoted 28 lines

Quoted 8 lines
Quoted 6 lines
(Note in above I removed the resource name from io_alloc_cbm to match what was discussed during previous version:
https://lore.kernel.org/lkml/[email protected]/ ) What do you think?
Yes. I remember. "Kept the resource name while printing the CBM for io_alloc, so we dont have to change show_doms() just for this feature and it is consistant across all the schemata display.
It almost sounds like you do not want to implement something because the code to support it does not exist?
Quoted 3 lines
I added the note in here.
https://lore.kernel.org/lkml/784fbc61e02e9a834473c3476ee196ef6a44e338.1745275431.git.babu.moger@amd.com/
You mention "I dont have to change show_doms() just for this feature and it is consistant across all the schemata display."
I am indeed seeing a pattern where one goal is to add changes by changing minimum amount of code. Please let this not be a goal but instead make it a goal to integrate changes into resctrl appropriately, not just pasted on top.
When it comes to the schemata display then it makes sense to add the resource name since the schemata file is within a resource group containing multiple resources and the schemata file thus needs to identify resources. Compare this to, for example, the "bit_usage" file that is unique to a resource and thus no need to identify the resource.
Quoted 1 line

Quoted 3 lines

Quoted 1 line
I will change it if you feel strongly about it. We will have to change show_doms() to handle this.
What is the problem with changing show_doms()?

There is no problem changing show_doms(). My intenstion was to keep the change as minimul as possible.

Sure. Will make the changes "not" to print the resource name for io_alloc_cbm.

Quoted 39 lines

Quoted 39 lines
Quoted 23 lines
Quoted 4 lines
---
v4: The "io_alloc" interface will report "enabled/disabled/not supported" instead of 0 or 1..
Quoted 8 lines
     Updated resctrl_io_alloc_closid_get() to verify the max closid availability using closids_supported().
     Updated the documentation for "shareable_bits" and "bit_usage".
     NOTE: io_alloc is about specific CLOS. rdt_bit_usage_show() is not designed handle bit_usage for specific CLOS. Its about overall system. So, we cannot really tell the user which CLOS is shared across both hardware and software.
"bit_usage" is not about CLOS but how the resource is used. Per the doc:
"bit_usage":
        Annotated capacity bitmasks showing how all instances of the resource are used.
Quoted 7 lines
The key here is the CBM, not CLOS. For each bit in the *CBM* "bit_usage" shows how that portion of the cache is used with the legend documented in Documentation/arch/x86/resctrl.rst.
Consider a system with the following allocations:
# cat /sys/fs/resctrl/schemata
L3:0=0ff0
This is CLOS 0.
Quoted 2 lines
# cat /sys/fs/resctrl/info/L3/io_alloc_cbm
0=ff00
This is CLOS 15.
Quoted 1 line

Quoted 8 lines

Quoted 6 lines
Quoted 4 lines
Then "bit_usage" will look like:
# cat /sys/fs/resctrl/info/L3/bit_usage
0=HHHHXXXXSSSS0000
It is confusing here. To make it clear we may have to print all the CLOSes in each domain.
Could you please elaborate how this is confusing?

# cat /sys/fs/resctrl/info/L3/bit_usage
0=HHHHXXXXSSSS0000

This may give the impression that the all CLOSes in all domains carries this property, but in reality, it applies only to one CLOS(15) within each domain.

Example below....

Quoted 31 lines

Quoted 5 lines
# cat /sys/fs/resctrl/info/L3/bit_usage
DOM0=CLOS0:SSSSSSSSSSSSSSSS;... ;CLOS15=HHHHXXXXSSSS0000;
DOM1=CLOS0:SSSSSSSSSSSSSSSS;... ;CLOS15=HHHHXXXXSSSS0000
Please no. Not just does this change existing user interface it also breaks the goal of "bit_usage".
Please think of it from user perspective. If user wants to know, for example, "how is my L3 cache allocated" then the "bit_usage" file provides that summary.
Quoted 7 lines
Quoted 7 lines
"bit_usage" shows how the cache is being used. It shows that the portion of cache represented by first four bits of CBM is unused, portion of cache represented by bits 4 to 7 of CBM is only used by software, portion of cache represented by bits 8 to 11 of CBM is shared between software and hardware, portion of cache represented by bits 12 to 15 is only used by hardware.
Quoted 1 line
This is something we need to discuss.
Quoted 7 lines
Quoted 4 lines
Looking at implementation in patch #5 the "io_alloc_cbm" bits of CBM are presented as software bits, since "io_alloc_cbm" represents IO from devices it should be "hardware" bits (hw_shareable), no?
Yes. It is. But logic is bit different there.
It loops thru all the CLOSes on the domain. So, it will print again like this below.
This is what current code does, but the code can be changed, no? For example, rdt_bit_usage_show() does not need to treat the IO allocation like all the other resource groups but instead handle it separately. Below us some pseudo code that presents the idea, untested, not compiled.

Quoted 14 lines

	hw_shareable = r->cache.shareable_bits;

	for (i = 0; i < closids_supported(); i++) {
		if (!closid_allocated(i) || 
		    (resctrl_arch_get_io_alloc_enabled(r) && i == resctrl_io_alloc_closid_get(r, s)))
			continue;

		/*  Intitialize sw_shareable and exclusive */
	}

	if (resctrl_arch_get_io_alloc_enabled(r)) {
		 /*
		  * Sidenote: I do not think schemata parameter is needed for
		  * resctrl_io_alloc_closid_get()

Sure. Got it.

Quoted 4 lines

		  */
		io_alloc_closid = resctrl_io_alloc_closid_get(r, s);
		if (resctrl_arch_get_cdp_enabled(r->rid))
			ctrl_val = resctrl_arch_get_config(r, dom, io_alloc_closid, CDP_CODE);

Quoted 9 lines

		else
			ctrl_val = resctrl_arch_get_config(r, dom, io_alloc_closid, CDP_NONE);
		hw_shareable |= ctrl_val;
	}

	for (i = r->cache.cbm_len - 1; i >= 0; i--) {
		/* Write annotated bitmask to user space */
	}

Here is the behaviour after these cahnges.

=== Before io_alloc enabled==============================

#cd /sys/fs/resctrl/L3/

# cat io_alloc
disabled

# cat shareable_bits
0  (This is always 0 for AMD)

# cat bit_usage
0=SSSSSSSSSSSSSSSS;1=SSSSSSSSSSSSSSSS;2=SSSSSSSSSSSSSSSS;3=SSSSSSSSSSSSSSSS

==== After io_alloc enabled=================================

# echo 1 > io_alloc

# cat io_alloc
enabled

# cat io_alloc_cbm
L3:0=ffff;1=ffff;2=ffff;3=ffff

#cat bit_usage 0=XXXXXXXXXXXXXXXX;1=XXXXXXXXXXXXXXXX;2=XXXXXXXXXXXXXXXX;3=XXXXXXXXXXXXXXXX

==== After changing io_alloc_cbm ============================

#echo "L3:0=ff00;1=ff00;2=ff00;3=ff00 > io_alloc_cbm

# cat io_alloc_cbm
L3:0=ff00;1=ff00;2=ff00;3=ff00

#cat bit_usage 0=XXXXXXXXSSSSSSSS;1=XXXXXXXXSSSSSSSS;2=XXXXXXXXSSSSSSSS;3=XXXXXXXXSSSSSSSS
=============================================================

My concern here is, this may imply that the property is present across all CLOSes in all the domains, while in fact, it only applies to a single CLOS(15) within each domain.

Thanks
Babu Moger

Reinette Chatre

maintainer

2025-05-05 21:13 UTC #20 lore raw

Hi Babu,

Quoted 4 lines · Babu

Quoted 3 lines · Reinette Chatre
Quoted 2 lines · Babu
Quoted 1 line · Reinette Chatre
On 4/21/25 3:43 PM, Babu Moger wrote:

...

Quoted 19 lines

Quoted 9 lines
Quoted 7 lines
Quoted 5 lines
Then "bit_usage" will look like:
# cat /sys/fs/resctrl/info/L3/bit_usage
0=HHHHXXXXSSSS0000
It is confusing here. To make it clear we may have to print all the CLOSes in each domain.
Could you please elaborate how this is confusing?
# cat /sys/fs/resctrl/info/L3/bit_usage
0=HHHHXXXXSSSS0000
This may give the impression that the all CLOSes in all domains carries this property, but in reality, it applies only to one CLOS(15) within each domain.
Example below....

...

Quoted 5 lines

Here is the behaviour after these cahnges.
=== Before io_alloc enabled==============================
#cd /sys/fs/resctrl/L3/

Quoted 8 lines

# cat io_alloc
disabled

# cat shareable_bits
0  (This is always 0 for AMD)

# cat bit_usage
0=SSSSSSSSSSSSSSSS;1=SSSSSSSSSSSSSSSS;2=SSSSSSSSSSSSSSSS;3=SSSSSSSSSSSSSSSS

Please note that the "S" in above does not have anything to do with "shareable_bits" at this point. The "S" indicates that all L3 instances are currently used by software and that sharing is allowed.

"bit_usage" gives insight to user space how all L3 instances are used.

If at this point a new resource group is created and it has an "exclusive" allocation then "bit_usage" will change to reflect that. For example, you can try this on the system you are testing on:

# echo 'L3:0=fff0;1=fff0;2=fff0;3=fff0' > /sys/fs/resctrl/schemata
# mkdir /sys/fs/resctrl/g1
# echo 'L3:0=f;1=f;2=f;3=f' > /sys/fs/resctrl/g1/schemata
# echo 'exclusive' > /sys/fs/resctrl/g1/mode

The above isolates a portion of all L3 instances for exclusive use by g1. After above changes:

# cat /sys/fs/resctrl/info/L3/bit_usage
0=SSSSSSSSSSSSEEEE;1=SSSSSSSSSSSSEEEE;2=SSSSSSSSSSSSEEEE;3=SSSSSSSSSSSSEEEE

Note that there is no "closid" or resource group information but instead, "bit_usage" shows to user space how each cache instance is being used across all resource groups and hardware (IO) allocations.

Quoted 5 lines

==== After io_alloc enabled=================================
# echo 1 > io_alloc

Quoted 8 lines

# cat io_alloc
enabled
# cat io_alloc_cbm
L3:0=ffff;1=ffff;2=ffff;3=ffff
#cat bit_usage 0=XXXXXXXXXXXXXXXX;1=XXXXXXXXXXXXXXXX;2=XXXXXXXXXXXXXXXX;3=XXXXXXXXXXXXXXXX

Looks accurate to me. It shows that both hardware and software can allocate into all portions of all caches.

Quoted 10 lines

==== After changing io_alloc_cbm ============================
#echo "L3:0=ff00;1=ff00;2=ff00;3=ff00 > io_alloc_cbm
# cat io_alloc_cbm
L3:0=ff00;1=ff00;2=ff00;3=ff00
#cat bit_usage 0=XXXXXXXXSSSSSSSS;1=XXXXXXXXSSSSSSSS;2=XXXXXXXXSSSSSSSS;3=XXXXXXXXSSSSSSSS

Looks accurate to me.

Quoted 5 lines

=============================================================
My concern here is, this may imply that the property is present across all CLOSes in all the domains, while in fact, it only applies to a single CLOS(15) within each domain.

If a user wants a resource group specific view then the schemata should be used. "bit_usage" presents the view from the cache instance perspective and reflects how each L3 cache instance is being used at that moment in time. It helps system administrator answer the question "how are the caches used at the moment"? "bit_usage" does so by presenting a summary of all allocations across all resource groups and any hardware allocations that may exist. This file helps user space to understand how the cache is being used without needing to correlate the CBMs of all resource groups and IO allocations. For example, "bit_usage" is to be used by system administrator to ensure cache is used optimally (for example, there are no unused portions). Also, a user may be investigating a performance issue in a particular resource group and "bit_usage" will help with that to see if the tasks in that resource group may be competing with IO.

Reinette

Moger, Babu

2025-05-05 22:29 UTC #21 lore raw

Hi Reinette,

Quoted 61 lines · Reinette Chatre

Hi Babu,
Quoted 4 lines · Babu
Quoted 3 lines · Reinette Chatre
Quoted 2 lines · Babu
Quoted 1 line · Reinette Chatre
On 4/21/25 3:43 PM, Babu Moger wrote:
...
Quoted 19 lines
Quoted 9 lines
Quoted 7 lines
Quoted 5 lines
Then "bit_usage" will look like:
# cat /sys/fs/resctrl/info/L3/bit_usage
0=HHHHXXXXSSSS0000
It is confusing here. To make it clear we may have to print all the CLOSes in each domain.
Could you please elaborate how this is confusing?
# cat /sys/fs/resctrl/info/L3/bit_usage
0=HHHHXXXXSSSS0000
This may give the impression that the all CLOSes in all domains carries this property, but in reality, it applies only to one CLOS(15) within each domain.
Example below....
...
Quoted 5 lines
Here is the behaviour after these cahnges.
=== Before io_alloc enabled==============================
#cd /sys/fs/resctrl/L3/
Quoted 8 lines
# cat io_alloc
disabled
# cat shareable_bits
0  (This is always 0 for AMD)
# cat bit_usage
0=SSSSSSSSSSSSSSSS;1=SSSSSSSSSSSSSSSS;2=SSSSSSSSSSSSSSSS;3=SSSSSSSSSSSSSSSS
Please note that the "S" in above does not have anything to do with "shareable_bits" at this point. The "S" indicates that all L3 instances are currently used by software and that sharing is allowed.
"bit_usage" gives insight to user space how all L3 instances are used.
If at this point a new resource group is created and it has an "exclusive" allocation then "bit_usage" will change to reflect that. For example, you can try this on the system you are testing on:
# echo 'L3:0=fff0;1=fff0;2=fff0;3=fff0' > /sys/fs/resctrl/schemata
# mkdir /sys/fs/resctrl/g1
# echo 'L3:0=f;1=f;2=f;3=f' > /sys/fs/resctrl/g1/schemata
# echo 'exclusive' > /sys/fs/resctrl/g1/mode

Quoted 5 lines

The above isolates a portion of all L3 instances for exclusive use by g1. After above changes:
# cat /sys/fs/resctrl/info/L3/bit_usage
0=SSSSSSSSSSSSEEEE;1=SSSSSSSSSSSSEEEE;2=SSSSSSSSSSSSEEEE;3=SSSSSSSSSSSSEEEE

Yes. I see the same output.

Quoted 3 lines

Note that there is no "closid" or resource group information but instead, "bit_usage" shows to user space how each cache instance is being used across all resource groups and hardware (IO) allocations.

Ok. Got it.

Quoted 37 lines

Quoted 6 lines
==== After io_alloc enabled=================================
# echo 1 > io_alloc
Quoted 8 lines
# cat io_alloc
enabled
# cat io_alloc_cbm
L3:0=ffff;1=ffff;2=ffff;3=ffff
#cat bit_usage 0=XXXXXXXXXXXXXXXX;1=XXXXXXXXXXXXXXXX;2=XXXXXXXXXXXXXXXX;3=XXXXXXXXXXXXXXXX
Looks accurate to me. It shows that both hardware and software can allocate into all portions of all caches.
Quoted 10 lines
==== After changing io_alloc_cbm ============================
#echo "L3:0=ff00;1=ff00;2=ff00;3=ff00 > io_alloc_cbm
# cat io_alloc_cbm
L3:0=ff00;1=ff00;2=ff00;3=ff00
#cat bit_usage 0=XXXXXXXXSSSSSSSS;1=XXXXXXXXSSSSSSSS;2=XXXXXXXXSSSSSSSS;3=XXXXXXXXSSSSSSSS
Looks accurate to me.
Quoted 5 lines
=============================================================
My concern here is, this may imply that the property is present across all CLOSes in all the domains, while in fact, it only applies to a single CLOS(15) within each domain.

Quoted 13 lines

If a user wants a resource group specific view then the schemata should be used. "bit_usage" presents the view from the cache instance perspective and reflects how each L3 cache instance is being used at that moment in time. It helps system administrator answer the question "how are the caches used at the moment"? "bit_usage" does so by presenting a summary of all allocations across all resource groups and any hardware allocations that may exist. This file helps user space to understand how the cache is being used without needing to correlate the CBMs of all resource groups and IO allocations. For example, "bit_usage" is to be used by system administrator to ensure cache is used optimally (for example, there are no unused portions). Also, a user may be investigating a performance issue in a particular resource group and "bit_usage" will help with that to see if the tasks in that resource group may be competing with IO.

Ok, "bit_usage" is a summary across all the groups. That is a good point. Thanks for the detailed explanation.

Will make those changes in next revision.

Thank you.
Babu