Skip to contentSkip to conversation

x86,fs/resctrl: Support for Global Bandwidth Enforcement and Priviledge Level Zero Association

RFC PATCH New Babu Moger 1 participant 2 messages Jan 21

Subsystems (4)
lore
Share & tools
lore

Babu Moger

author reviewer
#0 lore raw

RFC PATCH 1/1 x86,fs/resctrl: Support for Global Bandwidth Enforcement and Priviledge Level Zero Association

This patch series adds support for Global Bandwidth Enforcement (GLBE), Global Slow Bandwidth Enforcement (GLSBE) and Priviledge Level Zero Association (PLZA) in the x86 architecture's fs/resctrl subsystem. The changes include modifications to the resctrl filesystem to allow users to configure GLBE settings and associate CPUs and tasks with a separate CLOS when executiing in CPL0.

The feature documentation is not yet publicly available, but it is expected to be released in the next few weeks. In the meantime, a brief description of the features is provided below. Sharing this series as an RFC to gather initial feedback. Comments are welcome.

Global Bandwidth Enforcement (GLBE)

AMD Global Bandwidth Enforcement (GLBE) provides a mechanism for software to specify bandwidth limits for groups of threads that span multiple QOoS Domains. This collection of QOS Domains is referred to as the GLBE Control Domain. The GLBE ceiling is a bandwidth ceiling for L3 External Bandwidth competitively shared between all threads in a COS (Class of Service) across all QOS Domains within the GLBE Control Domain. This complements L3BE L3 External Bandwidth Enforcement (L3BE) which provides L3 eExternal Bandwidth control on a per QOS Domain granularity.

Global Slow Bandwidth Enforcement (GLSBE)

AMD PQoS Global Slow Bandwidth Enforcement (GLSBE) provides a mechanism for software to specify bandwidth limits for groups of threads that span multiple QOS Domains. GLSBE operates within the same GLBE Control Domains defined by GLBE. The GLSBE ceiling is a bandwidth ceiling for L3 External Bandwidth to Slow Memory competitively shared between all threads in a COS in all QOS Domains within the GLBE Control Domain. This complements L3SMBE which provides Slow Memory bandwidth control on a per QOS Domain granularity.

Privilege Level Zero Association (PLZA)

Privilege Level Zero Association (PLZA) allows the hardware to automatically associate execution in Privilege Level Zero (CPL=0) with a specific COS (Class of Service) and/or RMID (Resource Monitoring Identifier). The QoS feature set already has a mechanism to associate execution on each logical processor with an RMID or COS. PLZA allows the system to override this per-thread association for a thread that is executing with CPL=0.

The patches are based on top of commit (v6.19-rc5) Commit 0f61b1860cc3 (tag: v6.19-rc5, tip/tip/urgent) Linux 6.19-rc5

Changes include:
- Introduction of a new max_bandwidth file for each resctrl resource to expose the maximum supported bandwidth.
- Addition of new schemata GMB and GSMBA interfaces for configuring GLBE and GSLBE parameters.
- Modifications to associate resctrl groups with PLZA.
- Documentation updates to describe the new functionality.

Interface Changes:
1. A new max_bandwidth file has been added under each resource type directory (for example, /sys/fs/resctrl/info/GMB/max_bandwidth) to report the maximum bandwidth supported by the resource.

2. New resource types, GMB and GSMBA, have been introduced and are exposed through the schemata interface:

   # cat /sys/fs/resctrl/schemata
     GSMBA:0=4096;1=4096
      SMBA:0=8192;1=8192
       GMB:0=4096;1=4096
        MB:0=8192;1=8192
        L3:0=ffff;1=ffff

3. A new plza_capable file has been added under each resource type directory (for example, /sys/fs/resctrl/info/GMB/plza_capable) to indicate whether the resource supports the PLZA feature.

4. A new plza control file has been added to each resctrl group (for example, /sys/fs/resctrl/plza) to enable or disable PLZA association for the group. Writing 1 enables PLZA for the group, while writing 0 disables it.

Babu Moger (1):
x86,fs/resctrl: Add support for Global Bandwidth Enforcement (GLBE)

 Documentation/admin-guide/kernel-parameters.txt | 2 +-
 arch/x86/include/asm/cpufeatures.h              | 2 +-
 arch/x86/kernel/cpu/resctrl/core.c              | 2 ++
 arch/x86/kernel/cpu/scattered.c                 | 1 +
 4 files changed, 5 insertions(+), 2 deletions(-)
Signature · 1 line
2.34.1
author reviewer
#1 lore raw

RFC PATCH 1/1 x86,fs/resctrl: Add support for Global Bandwidth Enforcement (GLBE)

On AMD systems, the existing MBA feature allows the user to set a bandwidth limit for each QOS domain. However, multiple QOS domains share system memory bandwidth as a resource. In order to ensure that system memory bandwidth is not over-utilized, user must statically partition the available system bandwidth between the active QOS domains. This typically results in system memory being under-utilized since not all QOS domains are using their full bandwidth Allocation.

AMD PQoS Global Bandwidth Enforcement(GLBE) provides a mechanism for software to specify bandwidth limits for groups of threads that span multiple QoS Domains. This collection of QOS domains is referred to as GLBE control domain. The GLBE ceiling sets a maximum limit on a memory bandwidth in GLBE control domain. Bandwidth is shared by all threads in a Class of Service(COS) across every QoS domain managed by the GLBE control domain.

GLBE support is reported through CPUID.8000_0020_EBX_x0[GLBE] (bit 7). When this bit is set to 1, the platform supports GLBE.

Since the AMD Memory Bandwidth Enforcement feature is represented as MBA, the Global Bandwidth Enforcement feature will be shown as GMBA to maintain consistent naming.

Add GMBA support to resctrl and introduce a kernel parameter that allows enabling or disabling the feature at boot time.

---
 Documentation/admin-guide/kernel-parameters.txt | 2 +-
 arch/x86/include/asm/cpufeatures.h              | 2 +-
 arch/x86/kernel/cpu/resctrl/core.c              | 2 ++
 arch/x86/kernel/cpu/scattered.c                 | 1 +
 4 files changed, 5 insertions(+), 2 deletions(-)
Documentation/admin-guide/kernel-parameters.txt +1 -1
index abd77f39c783..e3058b3d47e9 100644
@@ -6325,7 +6325,7 @@ Kernel parameters
 	rdt=		[HW,X86,RDT]
 			Turn on/off individual RDT features. List is:
 			cmt, mbmtotal, mbmlocal, l3cat, l3cdp, l2cat, l2cdp,
-			mba, smba, bmec, abmc, sdciae, energy[:guid],
+			mba, gmba, smba, bmec, abmc, sdciae, energy[:guid],
 			perf[:guid].
 			E.g. to turn on cmt and turn off mba use:
 				rdt=cmt,!mba
arch/x86/include/asm/cpufeatures.h +1 -1
index c3b53beb1300..86d1339cd1bd 100644
@@ -505,7 +505,6 @@
 #define X86_FEATURE_ABMC		(21*32+15) /* Assignable Bandwidth Monitoring Counters */
 #define X86_FEATURE_MSR_IMM		(21*32+16) /* MSR immediate form instructions */
 #define X86_FEATURE_SGX_EUPDATESVN	(21*32+17) /* Support for ENCLS[EUPDATESVN] instruction */
-
 #define X86_FEATURE_SDCIAE		(21*32+18) /* L3 Smart Data Cache Injection Allocation Enforcement */
 #define X86_FEATURE_CLEAR_CPU_BUF_VM_MMIO (21*32+19) /*
 						      * Clear CPU buffers before VM-Enter if the vCPU
@@ -513,6 +512,7 @@
 						      * and purposes if CLEAR_CPU_BUF_VM is set).
 						      */
 #define X86_FEATURE_X2AVIC_EXT		(21*32+20) /* AMD SVM x2AVIC support for 4k vCPUs */
+#define X86_FEATURE_GMBA		(21*32+21) /* Global Memory Bandwidth Allocation */
 
 /*
  * BUG word(s)
arch/x86/kernel/cpu/resctrl/core.c +2 -0
index 9fcc06e9e72e..8b3457518ff4 100644
@@ -795,6 +795,7 @@ enum {
 	RDT_FLAG_L2_CAT,
 	RDT_FLAG_L2_CDP,
 	RDT_FLAG_MBA,
+	RDT_FLAG_GMBA,
 	RDT_FLAG_SMBA,
 	RDT_FLAG_BMEC,
 	RDT_FLAG_ABMC,
@@ -822,6 +823,7 @@ static struct rdt_options rdt_options[]  __ro_after_init = {
 	RDT_OPT(RDT_FLAG_L2_CAT,    "l2cat",	X86_FEATURE_CAT_L2),
 	RDT_OPT(RDT_FLAG_L2_CDP,    "l2cdp",	X86_FEATURE_CDP_L2),
 	RDT_OPT(RDT_FLAG_MBA,	    "mba",	X86_FEATURE_MBA),
+	RDT_OPT(RDT_FLAG_GMBA,	    "gmba",	X86_FEATURE_GMBA),
 	RDT_OPT(RDT_FLAG_SMBA,	    "smba",	X86_FEATURE_SMBA),
 	RDT_OPT(RDT_FLAG_BMEC,	    "bmec",	X86_FEATURE_BMEC),
 	RDT_OPT(RDT_FLAG_ABMC,	    "abmc",	X86_FEATURE_ABMC),
arch/x86/kernel/cpu/scattered.c +1 -0
index 42c7eac0c387..d081d167bac9 100644
@@ -59,6 +59,7 @@ static const struct cpuid_bit cpuid_bits[] = {
 	{ X86_FEATURE_BMEC,			CPUID_EBX,  3, 0x80000020, 0 },
 	{ X86_FEATURE_ABMC,			CPUID_EBX,  5, 0x80000020, 0 },
 	{ X86_FEATURE_SDCIAE,			CPUID_EBX,  6, 0x80000020, 0 },
+	{ X86_FEATURE_GMBA,			CPUID_EBX,  7, 0x80000020, 0 },
 	{ X86_FEATURE_TSA_SQ_NO,		CPUID_ECX,  1, 0x80000021, 0 },
 	{ X86_FEATURE_TSA_L1_NO,		CPUID_ECX,  2, 0x80000021, 0 },
 	{ X86_FEATURE_AMD_WORKLOAD_CLASS,	CPUID_EAX, 22, 0x80000021, 0 },
Signature · 1 line
2.34.1