x86/resctrl: Support for AMD Global (Slow) Memory Bandwidth Allocation
Subsystems (3)
Babu Moger
reviewerv2 cover x86/resctrl: Support for AMD Global (Slow) Memory Bandwidth Allocation
This series adds resctrl support for two new AMD memory-bandwidth allocation features:
- GMBA - Global Memory Bandwidth Allocation (hardware name: GLBE). Bounds DRAM bandwidth for groups of threads that span multiple L3 QoS domains, rather than being per-L3 like MBA.
- GSMBA - Global Slow Memory Bandwidth Allocation (hardware name:
GLSBE). The CXL.memory / slow-memory counterpart of GMBA, analogous to how SMBA relates to MBA.
Both features share a new "NPS-node" control domain: a set of QoS (L3) domains grouped together and aligned to the system's NPS (Nodes Per Socket) BIOS configuration. Although the control domain is NPS-scoped, the underlying bandwidth-limit MSRs (MSR_IA32_GMBA_BW_BASE 0xc0000600, MSR_IA32_GSMBA_BW_BASE 0xc0000680) are instantiated per L3. Programming a single control domain therefore requires writing the MSR on one CPU per L3 that the domain spans - a new pattern for resctrl. Patches 2/8 and 3/8 introduce that infrastructure so the new resources can reuse it.
The features are documented in:
AMD64 Zen6 Platform Quality of Service (PQOS) Extensions, Publication # 69193 Revision 1.00, Issue Date March 2026
available at https://bugzilla.kernel.org/show_bug.cgi?id=206537
Series overview
---------------
Patches 1-5 to enable GMBA:
1/8 x86,fs/resctrl: Add support for Global Bandwidth Enforcement (GLBE)
2/8 x86/resctrl: Add RESCTRL_NPS_NODE scope for AMD NPS-aligned domains
Add a new ctrl_scope value for resctrl resources whose control domain spans multiple L3s within an NPS node.
3/8 x86/resctrl: Update control MSRs per L3 for NPS-scoped resources
Add resctrl_arch_update_nps(): builds a cpumask with one CPU per distinct L3 in the domain, then issues rdt_ctrl_update() via smp_call_function_many() on that mask. Falls back to the full domain mask if the scratch masks cannot be built. Route resctrl_arch_update_domains() and resctrl_arch_reset_all_ctrls() through this helper when ctrl_scope == RESCTRL_NPS_NODE.
4/8 x86,fs/resctrl: Add the resource for Global Memory Bandwidth Allocation
Register RDT_RESOURCE_GMBA in rdt_resources_all[] with ctrl_scope=RESCTRL_NPS_NODE and schema_fmt=RANGE, add commands to discover feature details.
5/8 fs/resctrl: Add the documentation for Global Memory Bandwidth Allocation
Add examples in Documentation/filesystems/resctrl.rst.
Patches 6-8 to enable GSMBA in the same shape:
6/8 x86,fs/resctrl: Add support for Global Slow Memory Bandwidth Allocation
7/8 x86,fs/resctrl: Add the resource for Global Slow Memory Bandwidth Allocation
Register RDT_RESOURCE_GSMBA with ctrl_scope=RESCTRL_NPS_NODE.
8/8 fs/resctrl: Add the documentation for Global Slow Memory Bandwidth Allocation
Add examples in Documentation/filesystems/resctrl.rst.
Changes since v1
----------------
- Earlier sent RFC(v1) with Global Bandwidth Enforcement (GLBE) and Privilege Level Zero Association (PLZA). This series only handles Global Memory Bandwidth Allocation. Both the features are sent separately.
- Documentation
* Fixed grammar in the GMBA / GSMBA sections of resctrl.rst.
* Added examples to update GMBA and GSMBA in resctrl.rst documentation.
- Major changes are releated to RESCTRL_NPS_NODE scope handling.
- Commit messages
* Reworked the changelogs in all the patches.
Previous Revisions:
v1 : https://lore.kernel.org/lkml/[email protected]/
---
base-commit: 216fe4b3e06754e73c79a88b1df7e9806e41f29d
Babu Moger (8):
x86,fs/resctrl: Add support for Global Bandwidth Enforcement (GLBE) x86/resctrl: Add RESCTRL_NPS_NODE scope for AMD NPS-aligned domains x86/resctrl: Update control MSRs per L3 for NPS-scoped resources x86,fs/resctrl: Add the resource for Global Bandwidth Allocation fs/resctrl: Add the documentation for Global Memory Bandwidth
Allocation
x86,fs/resctrl: Add support for Global Slow Memory Bandwidth
Allocation
x86,fs/resctrl: Add the resource for Global Slow Memory Bandwidth
Allocation
fs/resctrl: Add the documentation for Global Slow Memory Bandwidth
Allocation
.../admin-guide/kernel-parameters.txt | 2 +- Documentation/filesystems/resctrl.rst | 81 ++++++++++++++ arch/x86/include/asm/cpufeatures.h | 3 +- arch/x86/include/asm/msr-index.h | 2 + arch/x86/kernel/cpu/resctrl/core.c | 101 +++++++++++++++++- arch/x86/kernel/cpu/resctrl/ctrlmondata.c | 90 +++++++++++++++- arch/x86/kernel/cpu/resctrl/internal.h | 1 + arch/x86/kernel/cpu/resctrl/rdtgroup.c | 15 ++- arch/x86/kernel/cpu/scattered.c | 2 + fs/resctrl/ctrlmondata.c | 5 +- fs/resctrl/rdtgroup.c | 25 ++++- include/linux/resctrl.h | 3 + 12 files changed, 313 insertions(+), 17 deletions(-)
Signature · 1 line
2.43.0
Babu Moger
reviewerPATCH v2 1/8 x86,fs/resctrl: Add support for Global Bandwidth Enforcement (GLBE)
On AMD systems, the existing MBA feature allows the user to set a bandwidth limit for each QoS domain. However, multiple QoS domains share system memory bandwidth as a resource. In order to ensure that system memory bandwidth is not over-utilized, the user must statically partition the available system bandwidth between the active QoS domains. Active domains are those that have at least one thread running in them. If a QoS domain is not active, its bandwidth allocation is not used by any thread and is effectively wasted. On the other hand, if a QoS domain is active but its bandwidth allocation is too low, threads running in that domain may not be able to utilize the full potential of the system memory bandwidth. This typically results in system memory being under-utilized since not all QoS domains are using their full bandwidth allocation.
AMD PQoS Global Bandwidth Enforcement (GLBE) provides a mechanism for software to specify bandwidth limits for groups of threads that span multiple QoS domains. This collection of QoS domains is referred to as a GLBE control domain. The GLBE ceiling sets a maximum limit on memory bandwidth in GLBE control domain. Bandwidth is shared by all threads in a Class of Service (COS) across every QoS domain managed by the GLBE control domain. The GLBE control domain is configurable using the BIOS setup utility.
By default the GLBE control domain is aligned to system NPS (Nodes Per Socket) configuration. NPS is a BIOS-level setting on AMD processors that determines how many NUMA (Non Uniform Memory Access) nodes each CPU socket is divided into. NPS allows administrators to tune memory locality, bandwidth, and latency for different workloads.
GLBE support is reported through CPUID.8000_0020_EBX_x0[GLBE] (bit 7). When this bit is set to 1, the platform supports GLBE.
Since the AMD Memory Bandwidth Enforcement feature is represented as MBA, the Global Bandwidth Enforcement feature will be shown as GMBA to maintain consistent naming.
Add GMBA support to resctrl and introduce a kernel parameter that allows enabling or disabling the feature at boot time.
The GLBE feature details are documented in:
AMD64 Zen6 Platform Quality of Service (PQOS) Extensions:
Publication # 69193 Revision: 1.00, Issue Date: March 2026
available at https://bugzilla.kernel.org/show_bug.cgi?id=206537
---
v2: Updated the commit message to include more details about active QoS domains.
Updated the commit message to clarify that the GLBE control domain is configurable using the BIOS setup utility is aligned to NPS. Added link to the PQOS spec.
--- Documentation/admin-guide/kernel-parameters.txt | 2 +- arch/x86/include/asm/cpufeatures.h | 2 +- arch/x86/kernel/cpu/resctrl/core.c | 2 ++ arch/x86/kernel/cpu/scattered.c | 1 + 4 files changed, 5 insertions(+), 2 deletions(-)
Documentation/admin-guide/kernel-parameters.txt +1 -1
@@ -6452,7 +6452,7 @@ Kernel parameters rdt= [HW,X86,RDT] Turn on/off individual RDT features. List is: cmt, mbmtotal, mbmlocal, l3cat, l3cdp, l2cat, l2cdp, - mba, smba, bmec, abmc, sdciae, energy[:guid], + mba, gmba, smba, bmec, abmc, sdciae, energy[:guid], perf[:guid]. E.g. to turn on cmt and turn off mba use: rdt=cmt,!mba
arch/x86/include/asm/cpufeatures.h +1 -1
@@ -508,7 +508,6 @@#define X86_FEATURE_ABMC (21*32+15)/* Assignable Bandwidth Monitoring Counters */ #define X86_FEATURE_MSR_IMM (21*32+16)/* MSR immediate form instructions */ #define X86_FEATURE_SGX_EUPDATESVN (21*32+17)/* Support for ENCLS[EUPDATESVN] instruction */ -#define X86_FEATURE_SDCIAE (21*32+18)/* L3 Smart Data Cache Injection Allocation Enforcement */ #define X86_FEATURE_CLEAR_CPU_BUF_VM_MMIO (21*32+19)/* * Clear CPU buffers before VM-Enter if the vCPU
@@ -516,6 +515,7 @@* and purposes if CLEAR_CPU_BUF_VM is set). */ #define X86_FEATURE_X2AVIC_EXT (21*32+20)/* AMD SVM x2AVIC support for 4k vCPUs */ +#define X86_FEATURE_GMBA (21*32+21)/* Global Memory Bandwidth Allocation */ /* * BUG word(s)
arch/x86/kernel/cpu/resctrl/core.c +2 -0
@@ -795,6 +795,7 @@ enum {RDT_FLAG_L2_CAT ,RDT_FLAG_L2_CDP ,RDT_FLAG_MBA , +RDT_FLAG_GMBA ,RDT_FLAG_SMBA ,RDT_FLAG_BMEC , RDT_FLAG_ABMC,
@@ -822,6 +823,7 @@ static struct rdt_options rdt_options[] __ro_after_init = {RDT_OPT (RDT_FLAG_L2_CAT ,"l2cat" ,X86_FEATURE_CAT_L2 ),RDT_OPT (RDT_FLAG_L2_CDP ,"l2cdp" ,X86_FEATURE_CDP_L2 ),RDT_OPT (RDT_FLAG_MBA ,"mba" ,X86_FEATURE_MBA ), +RDT_OPT (RDT_FLAG_GMBA ,"gmba" ,X86_FEATURE_GMBA ),RDT_OPT (RDT_FLAG_SMBA ,"smba" ,X86_FEATURE_SMBA ),RDT_OPT (RDT_FLAG_BMEC ,"bmec" ,X86_FEATURE_BMEC ), RDT_OPT(RDT_FLAG_ABMC, "abmc", X86_FEATURE_ABMC),
arch/x86/kernel/cpu/scattered.c +1 -0
@@ -60,6 +60,7 @@ static const struct cpuid_bit cpuid_bits[] = { { X86_FEATURE_BMEC, CPUID_EBX, 3, 0x80000020, 0 }, { X86_FEATURE_ABMC, CPUID_EBX, 5, 0x80000020, 0 }, { X86_FEATURE_SDCIAE, CPUID_EBX, 6, 0x80000020, 0 }, + { X86_FEATURE_GMBA, CPUID_EBX, 7, 0x80000020, 0 }, { X86_FEATURE_TSA_SQ_NO, CPUID_ECX, 1, 0x80000021, 0 }, { X86_FEATURE_TSA_L1_NO, CPUID_ECX, 2, 0x80000021, 0 }, { X86_FEATURE_AMD_WORKLOAD_CLASS, CPUID_EAX, 22, 0x80000021, 0 },
Signature · 1 line
2.43.0
Babu Moger
reviewerPATCH v2 2/8 x86/resctrl: Add RESCTRL_NPS_NODE scope for AMD NPS-aligned domains
In reply to Babu MogerGlobal Memory Bandwidth Allocation (GMBA) control domains on AMD follow the Nodes Per Socket (NPS) configuration. With NPS 4 there is a single domain per package; otherwise the domain aligns with the NUMA node.
By default, all QOS Domains in the system are included in a single GLBE Control Domain. However, BIOS options may establish several GLBE Control Domains within the system using NPS configuration. When configured, each NUMA node functions as an individual GLBE domain, except when the system is configured in NPS=4 mode in which case only a single GLBE domain exists which contains all QOS Domains in the system.
Add RESCTRL_NPS_NODE to enum resctrl_scope and implement get_domain_id_nps().
---
v2: Add RESCTRL_NPS_NODE scope to add support for NPS-aligned resource.
--- arch/x86/kernel/cpu/resctrl/core.c | 20 ++++++++++++++++++++ include/linux/resctrl.h | 1 + 2 files changed, 21 insertions(+)
arch/x86/kernel/cpu/resctrl/core.c +20 -0
@@ -451,6 +451,24 @@ static int l3_mon_domain_mbm_alloc(u32 num_rmid, struct rdt_hw_l3_mon_domain *hwreturn -ENOMEM ; } +/** +* get_domain_id_nps() - Domain id for %RESCTRL_NPS_NODE (AMD NPS / GMBA) +* @cpu: CPU to query. +* +* Global memory bandwidth allocation (GMBA) control domains on AMD follow +* the socket NPS layout. With NPS 4 there is a single control domain per +* package, so every CPU maps to domain id 0. For other NPS settings the +* domain matches the CPU's NUMA node. +* +* Return: 0 when NPS is 4, otherwise the NUMA node id for @cpu. +*/ +static int get_domain_id_nps (int cpu) +{ +if (topology_num_nodes_per_package () ==4 ) +return 0 ; +return cpu_to_node (cpu); +} +static int get_domain_id_from_scope (int cpu,enum resctrl_scope scope) {switch (scope) {
@@ -459,6 +477,8 @@ static int get_domain_id_from_scope(int cpu, enum resctrl_scope scope)return get_cpu_cacheinfo_id (cpu, scope);case RESCTRL_L3_NODE :return cpu_to_node (cpu); +case RESCTRL_NPS_NODE : +return get_domain_id_nps (cpu);case RESCTRL_PACKAGE :return topology_physical_package_id (cpu);default :
include/linux/resctrl.h +1 -0
@@ -271,6 +271,7 @@ enum resctrl_scope {RESCTRL_L2_CACHE =2 ,RESCTRL_L3_CACHE =3 ,RESCTRL_L3_NODE , +RESCTRL_NPS_NODE , RESCTRL_PACKAGE, };
Signature · 1 line
2.43.0
Re: [PATCH v2 2/8] x86/resctrl: Add RESCTRL_NPS_NODE scope for AMD NPS-aligned domains
Hi Babu,
Quoted 11 lines · Babu Moger
Global Memory Bandwidth Allocation (GMBA) control domains on AMD follow the Nodes Per Socket (NPS) configuration. With NPS 4 there is a single domain per package; otherwise the domain aligns with the NUMA node.
By default, all QOS Domains in the system are included in a single GLBE Control Domain. However, BIOS options may establish several GLBE Control Domains within the system using NPS configuration. When configured, each NUMA node functions as an individual GLBE domain, except when the system is configured in NPS=4 mode in which case only a single GLBE domain exists which contains all QOS Domains in the system.
The previous paragraph said NPS 4 is a single domain per package, not system-wide.
Quoted 25 lines
Add
RESCTRL_NPS_NODEto enumresctrl_scopeand implementget_domain_id_nps().Signed-off-by Babu Moger <[email protected]>---
v2: AddRESCTRL_NPS_NODEscope to add support for NPS-aligned resource.--- arch/x86/kernel/cpu/resctrl/core.c | 20 ++++++++++++++++++++ include/linux/resctrl.h | 1 + 2 files changed, 21 insertions(+)arch/x86/kernel/cpu/resctrl/core.c +5
-0@@ -451,6 +451,24 @@ static int l3_mon_domain_mbm_alloc(u32 num_rmid, struct rdt_hw_l3_mon_domain *hwreturn -ENOMEM ; } +/** +* get_domain_id_nps() - Domain id for %RESCTRL_NPS_NODE (AMD NPS / GMBA) +* @cpu: CPU to query. +* +* Global memory bandwidth allocation (GMBA) control domains on AMD follow
arch/x86/kernel/cpu/resctrl/core.c static int l3_mon_domain_mbm_alloc(u32 num_rmid, struct rdt_hw_l3_mon_domain *hw · line 468
Quoted 10 lines
arch/x86/kernel/cpu/resctrl/core.c +10
-0+* the socket NPS layout. With NPS 4 there is a single control domain per +* package, so every CPU maps to domain id 0. For other NPS settings the +* domain matches the CPU's NUMA node. +* +* Return: 0 when NPS is 4, otherwise the NUMA node id for @cpu. +*/ +static int get_domain_id_nps (int cpu) +{ +if (topology_num_nodes_per_package () ==4 ) +return 0 ;
ID is 0 regardless of the number of packages, unlike the RESCTRL_PACKAGE case below?
Quoted 30 lines
arch/x86/kernel/cpu/resctrl/core.c +5
-0+return cpu_to_node (cpu); +} +static int get_domain_id_from_scope (int cpu,enum resctrl_scope scope) {switch (scope) {@@ -459,6 +477,8 @@ static int get_domain_id_from_scope(int cpu, enum resctrl_scope scope)return get_cpu_cacheinfo_id (cpu, scope);case RESCTRL_L3_NODE :return cpu_to_node (cpu); +case RESCTRL_NPS_NODE : +return get_domain_id_nps (cpu);case RESCTRL_PACKAGE :return topology_physical_package_id (cpu);default :include/linux/resctrl.h +1
-1@@ -271,6 +271,7 @@ enum resctrl_scope {RESCTRL_L2_CACHE =2 ,RESCTRL_L3_CACHE =3 ,RESCTRL_L3_NODE , +RESCTRL_NPS_NODE , RESCTRL_PACKAGE, }; --2.43.0
Re: [PATCH v2 2/8] x86/resctrl: Add RESCTRL_NPS_NODE scope for AMD NPS-aligned domains
Hi Peter,
Quoted 7 lines · Peter Newman
Hi Babu,
Quoted 4 lines · Babu Moger
Global Memory Bandwidth Allocation (GMBA) control domains on AMD follow the Nodes Per Socket (NPS) configuration. With NPS 4 there is a single domain per package; otherwise the domain aligns with the NUMA node.
My mistake. It should be system wide. Confirmed with team also. Line from spec. Will fix it.
When configured, each NUMA node functions as an individual GLBE domain, except when the system is configured in NPS=4 mode in which case only a single GLBE domain exists which contains all QOS Domains in the system.
Quoted 31 lines
Quoted 7 lines
By default, all QOS Domains in the system are included in a single GLBE Control Domain. However, BIOS options may establish several GLBE Control Domains within the system using NPS configuration. When configured, each NUMA node functions as an individual GLBE domain, except when the system is configured in NPS=4 mode in which case only a single GLBE domain exists which contains all QOS Domains in the system.
The previous paragraph said NPS 4 is a single domain per package, not system-wide.
Quoted 20 lines
Add
RESCTRL_NPS_NODEto enumresctrl_scopeand implementget_domain_id_nps().Signed-off-by Babu Moger <[email protected]>---
v2: AddRESCTRL_NPS_NODEscope to add support for NPS-aligned resource.--- arch/x86/kernel/cpu/resctrl/core.c | 20 ++++++++++++++++++++ include/linux/resctrl.h | 1 + 2 files changed, 21 insertions(+)arch/x86/kernel/cpu/resctrl/core.c +0
-0@@ -451,6 +451,24 @@ static int l3_mon_domain_mbm_alloc(u32 num_rmid, struct rdt_hw_l3_mon_domain *hwreturn -ENOMEM ; }
Quoted 8 lines
arch/x86/kernel/cpu/resctrl/core.c static int l3_mon_domain_mbm_alloc(u32 num_rmid, struct rdt_hw_l3_mon_domain *hw · line 461
Quoted 8 lines
arch/x86/kernel/cpu/resctrl/core.c +8
-0+/** +* get_domain_id_nps() - Domain id for %RESCTRL_NPS_NODE (AMD NPS / GMBA) +* @cpu: CPU to query. +* +* Global memory bandwidth allocation (GMBA) control domains on AMD follow +* the socket NPS layout. With NPS 4 there is a single control domain per +* package, so every CPU maps to domain id 0. For other NPS settings the +* domain matches the CPU's NUMA node.
Need to fix this text also.
Quoted 10 lines
arch/x86/kernel/cpu/resctrl/core.c static int l3_mon_domain_mbm_alloc(u32 num_rmid, struct rdt_hw_l3_mon_domain *hw · line 468
Quoted 7 lines
arch/x86/kernel/cpu/resctrl/core.c +7
-0+* +* Return: 0 when NPS is 4, otherwise the NUMA node id for @cpu. +*/ +static int get_domain_id_nps (int cpu) +{ +if (topology_num_nodes_per_package () ==4 ) +return 0 ;ID is 0 regardless of the number of packages, unlike the
RESCTRL_PACKAGEcase below?
Code is correct. Will fix the comments and commit message.
Thanks
Babu
Babu Moger
reviewerPATCH v2 3/8 x86/resctrl: Update control MSRs per L3 for NPS-scoped resources
In reply to Babu MogerAMD Global Memory Bandwidth Allocation (GMBA) uses the new RESCTRL_NPS_NODE ctrl_scope. On these CPUs the control MSRs are implemented per L3 complex, so a single resctrl control domain can span several L3s and every L3 in the domain must have its MSRs programmed.
For non-NPS scopes this is already handled: resctrl_arch_update_domains() and resctrl_arch_reset_all_ctrls() use smp_call_function_any() on the domain CPU mask, which is sufficient because one CPU per domain is enough to program the MSRs. That is not sufficient for RESCTRL_NPS_NODE, where rdt_ctrl_update() must run on one CPU per distinct L3 within the domain.
Add two helpers in ctrlmondata.c:
- resctrl_get_l3_mask() builds a cpumask containing exactly one CPU per unique L3 cacheinfo id observed in the domain mask.
- resctrl_arch_update_nps() allocates the scratch masks, calls resctrl_get_l3_mask() and issues rdt_ctrl_update() via smp_call_function_many() on the per-L3 mask. If the scratch cpumask or L3-id bitmap allocation fails, or the per-L3 mask ends up empty, it falls back to smp_call_function_many() on the full domain mask. This is conservative (more IPIs than strictly needed) but guarantees every L3 in the domain is covered.
Route resctrl_arch_update_domains() and resctrl_arch_reset_all_ctrls() through resctrl_arch_update_nps() when ctrl_scope == RESCTRL_NPS_NODE, and keep the existing smp_call_function_any() path for every other scope. Existing L3/MBA/SMBA resources therefore see no functional change.
---
v2: New patch to handle NPS-scoped MSR Update.
--- arch/x86/kernel/cpu/resctrl/ctrlmondata.c | 90 ++++++++++++++++++++++- arch/x86/kernel/cpu/resctrl/internal.h | 1 + arch/x86/kernel/cpu/resctrl/rdtgroup.c | 15 +++- 3 files changed, 100 insertions(+), 6 deletions(-)
arch/x86/kernel/cpu/resctrl/ctrlmondata.c +88 -2
@@ -41,6 +41,86 @@ int resctrl_arch_update_one(struct rdt_resource *r, struct rdt_ctrl_domain *d,return 0 ; } +/** +* resctrl_get_l3_mask() - One CPU per distinct L3 within a resctrl domain +* @domain_mask: Full domain CPU mask (typically &d->hdr.cpu_mask). +* @l3_mask: Output mask. Cleared on entry, then populated with exactly +* one CPU per unique L3 cacheinfo id observed in @domain_mask. +* Always a subset of @domain_mask; may end up empty if no CPU +* in @domain_mask has a valid L3 id. +* +* For %RESCTRL_NPS_NODE controls (e.g. AMD GMBA) the control MSRs are +* instantiated per L3 complex, so a single IPI per resctrl domain is not +* sufficient. Callers are expected to run rdt_ctrl_update() on each CPU in +* @l3_mask to cover every L3 that participates in the domain +* (see resctrl_arch_update_nps()). +* +* Return: @l3_mask on success, %NULL if the scratch L3-id bitmap could not +* be allocated (in which case @l3_mask is left cleared). +*/ +static struct cpumask *resctrl_get_l3_mask (const struct cpumask *domain_mask, +struct cpumask *l3_mask) +{ +unsigned long *l3_dom_id; +int cpu, id; + +cpumask_clear (l3_mask); + l3_dom_id =bitmap_zalloc (nr_cpu_ids,GFP_KERNEL ); +if (!l3_dom_id) +return NULL ; + +for_each_cpu (cpu, domain_mask) { + id =get_cpu_cacheinfo_id (cpu,RESCTRL_L3_CACHE ); +if (id <0 || id >= nr_cpu_ids) +continue ; +if (test_bit (id, l3_dom_id)) +continue ; +set_bit (id, l3_dom_id); +cpumask_set_cpu (cpu, l3_mask); + } + +bitmap_free (l3_dom_id); +return l3_mask; +} + +/** +* resctrl_arch_update_nps() - Apply staged ctrl MSRs for NPS-scoped resources +* @mp: Parameters describing the MSR index range, resource and domain +* passed through to rdt_ctrl_update(). +* @d: Control domain whose CPUs must see the MSR update. +* +* %RESCTRL_NPS_NODE resources program control MSRs per L3 complex, so one +* IPI per resctrl domain is not enough when the domain spans multiple L3s. +* Build a per-L3 representative mask with resctrl_get_l3_mask() and issue +* rdt_ctrl_update() via smp_call_function_many() on that mask. +* +* If the temporary cpumask or the scratch L3-id bitmap cannot be allocated, +* or the resulting per-L3 mask is empty, fall back to invoking +* smp_call_function_many() on the full domain CPU mask. This is +* conservative (more IPIs than strictly needed) but guarantees every L3 in +* the domain is covered. +*/ +void resctrl_arch_update_nps (struct msr_param *mp,struct rdt_ctrl_domain *d) +{ +const struct cpumask *mask = &d->hdr.cpu_mask; +struct cpumask *new_mask; +cpumask_var_t l3_mask; +bool l3_alloc; + + l3_alloc =zalloc_cpumask_var (&l3_mask,GFP_KERNEL ); +if (l3_alloc) { + new_mask =resctrl_get_l3_mask (&d->hdr.cpu_mask, l3_mask); + +if (new_mask && !cpumask_empty (new_mask)) + mask = new_mask; + } + +smp_call_function_many (mask, rdt_ctrl_update, mp,1 ); + +if (l3_alloc) +free_cpumask_var (l3_mask); +} +int resctrl_arch_update_domains (struct rdt_resource *r,u32 closid) {struct resctrl_staged_config *cfg;
@@ -76,8 +156,14 @@ int resctrl_arch_update_domains(struct rdt_resource *r, u32 closid) msr_param.high =max (msr_param.high, idx +1 ); } } -if (msr_param.res) -smp_call_function_any (&d->hdr.cpu_mask, rdt_ctrl_update, &msr_param,1 ); + +if (msr_param.res) { +if (msr_param.res->ctrl_scope ==RESCTRL_NPS_NODE ) +resctrl_arch_update_nps (&msr_param, d); +else +smp_call_function_any (&d->hdr.cpu_mask, + rdt_ctrl_update, &msr_param,1 ); + } }return 0 ;
arch/x86/kernel/cpu/resctrl/internal.h +1 -0
@@ -223,6 +223,7 @@ union l3_qos_abmc_cfg { };void rdt_ctrl_update (void *arg); +void resctrl_arch_update_nps (struct msr_param *mp,struct rdt_ctrl_domain *d);int rdt_get_l3_mon_config (struct rdt_resource *r);
arch/x86/kernel/cpu/resctrl/rdtgroup.c +11 -4
@@ -245,9 +245,12 @@ void resctrl_arch_reset_all_ctrls(struct rdt_resource *r) msr_param.high = hw_res->num_closid;/* -* Disable resource control for this resource by setting all -* CBMs in all ctrl_domains to the maximum mask value. Pick one CPU -* from each domain to update the MSRs below. +* Disable resource control for this resource by setting all CBMs in +* all ctrl_domains to the maximum mask value. For non-NPS scopes pick +* one CPU from each domain to update the MSRs below; for +* %RESCTRL_NPS_NODE the MSRs are per-L3, so defer to +* resctrl_arch_update_nps() which issues the update on one CPU per +* distinct L3 in the domain. */ list_for_each_entry (d, &r->ctrl_domains, hdr.list) { hw_dom =resctrl_to_arch_ctrl_dom (d);
@@ -255,7 +258,11 @@ void resctrl_arch_reset_all_ctrls(struct rdt_resource *r)for (i =0 ; i < hw_res->num_closid; i++) hw_dom->ctrl_val[i] =resctrl_get_default_ctrl (r); msr_param.dom = d; -smp_call_function_any (&d->hdr.cpu_mask, rdt_ctrl_update, &msr_param,1 ); +if (msr_param.res->ctrl_scope ==RESCTRL_NPS_NODE ) +resctrl_arch_update_nps (&msr_param, d); +else +smp_call_function_any (&d->hdr.cpu_mask, rdt_ctrl_update, + &msr_param,1 ); }return ;
Signature · 1 line
2.43.0
Babu Moger
reviewerPATCH v2 4/8 x86,fs/resctrl: Add the resource for Global Bandwidth Allocation
In reply to Babu MogerAMD PQoS Global Bandwidth Enforcement (GLBE) feature provides a mechanism for software to specify bandwidth limits for groups of threads inside a GLBE control domain. GLBE control domain is a set of participating QoS domains that are grouped together for global bandwidth allocation.
Add the resource definition for GLBE in resctrl filesystem. Resource allows users to configure and manage the global memory bandwidth allocation settings for GLBE control domain. GLBE control domain is a set of participating QoS domains that are grouped together for global bandwidth allocation.
By default the GLBE control domain is aligned to system NPS (Nodes Per Socket) configuration. However, MSR_IA32_GMBA_BW_BASE must be programmed in L3 domains.
---
v2: Updated the change log about NPS scope.
--- arch/x86/include/asm/msr-index.h | 1 + arch/x86/kernel/cpu/resctrl/core.c | 46 ++++++++++++++++++++++++++++-- fs/resctrl/ctrlmondata.c | 5 ++-- fs/resctrl/rdtgroup.c | 13 +++++++-- include/linux/resctrl.h | 1 + 5 files changed, 59 insertions(+), 7 deletions(-)
arch/x86/include/asm/msr-index.h +1 -0
@@ -1293,6 +1293,7 @@#define MSR_IA32_L3_QOS_ABMC_CFG 0xc00003fd#define MSR_IA32_L3_QOS_EXT_CFG 0xc00003ff#define MSR_IA32_EVT_CFG_BASE 0xc0000400 +#define MSR_IA32_GMBA_BW_BASE 0xc0000600/* AMD-V MSRs */ #define MSR_VM_CR 0xc0010114
arch/x86/kernel/cpu/resctrl/core.c +43 -3
77 lines not on this page · raw message
fs/resctrl/ctrlmondata.c +3 -2
@@ -246,8 +246,9 @@ static int parse_line(char *line, struct resctrl_schema *s,return -EINVAL ;if (rdtgrp->mode ==RDT_MODE_PSEUDO_LOCKSETUP && - (r->rid ==RDT_RESOURCE_MBA || r->rid ==RDT_RESOURCE_SMBA )) { -rdt_last_cmd_puts ("Cannot pseudo-lock MBA resource\n" ); + (r->rid ==RDT_RESOURCE_MBA || r->rid ==RDT_RESOURCE_GMBA || + r->rid ==RDT_RESOURCE_SMBA )) { +rdt_last_cmd_puts ("Cannot pseudo-lock MBA/SMBA/GMBA resource\n" );return -EINVAL ; }
fs/resctrl/rdtgroup.c +11 -2
@@ -1412,7 +1412,8 @@ static bool rdtgroup_mode_test_exclusive(struct rdtgroup *rdtgrp)list_for_each_entry (s, &resctrl_schema_all, list) { r = s->res; -if (r->rid ==RDT_RESOURCE_MBA || r->rid ==RDT_RESOURCE_SMBA ) +if (r->rid ==RDT_RESOURCE_MBA || r->rid ==RDT_RESOURCE_GMBA || + r->rid ==RDT_RESOURCE_SMBA )continue ; has_cache = true;list_for_each_entry (d, &r->ctrl_domains, hdr.list) {
@@ -1617,6 +1618,7 @@ static int rdtgroup_size_show(struct kernfs_open_file *of, closid, type);if (r->rid ==RDT_RESOURCE_MBA || + r->rid ==RDT_RESOURCE_GMBA || r->rid ==RDT_RESOURCE_SMBA ) size = ctrl;else
@@ -2170,13 +2172,18 @@ static struct rftype *rdtgroup_get_rftype_by_name(const char *name)static void thread_throttle_mode_init (void ) {enum membw_throttle_mode throttle_mode =THREAD_THROTTLE_UNDEFINED ; -struct rdt_resource *r_mba, *r_smba; +struct rdt_resource *r_mba, *r_gmba, *r_smba; r_mba =resctrl_arch_get_resource (RDT_RESOURCE_MBA );if (r_mba->alloc_capable && r_mba->membw.throttle_mode !=THREAD_THROTTLE_UNDEFINED ) throttle_mode = r_mba->membw.throttle_mode; + r_gmba =resctrl_arch_get_resource (RDT_RESOURCE_GMBA ); +if (r_gmba->alloc_capable && + r_gmba->membw.throttle_mode !=THREAD_THROTTLE_UNDEFINED ) + throttle_mode = r_gmba->membw.throttle_mode; + r_smba =resctrl_arch_get_resource (RDT_RESOURCE_SMBA );if (r_smba->alloc_capable && r_smba->membw.throttle_mode !=THREAD_THROTTLE_UNDEFINED )
@@ -2396,6 +2403,7 @@ static unsigned long fflags_from_resource(struct rdt_resource *r)case RDT_RESOURCE_L2 :return RFTYPE_RES_CACHE ;case RDT_RESOURCE_MBA : +case RDT_RESOURCE_GMBA :case RDT_RESOURCE_SMBA :return RFTYPE_RES_MB ;case RDT_RESOURCE_PERF_PKG :
@@ -3649,6 +3657,7 @@ static int rdtgroup_init_alloc(struct rdtgroup *rdtgrp)list_for_each_entry (s, &resctrl_schema_all, list) { r = s->res;if (r->rid ==RDT_RESOURCE_MBA || + r->rid ==RDT_RESOURCE_GMBA || r->rid ==RDT_RESOURCE_SMBA ) {rdtgroup_init_mba (r, rdtgrp->closid);if (is_mba_sc (r))
include/linux/resctrl.h +1 -0
@@ -52,6 +52,7 @@ enum resctrl_res_level {RDT_RESOURCE_L3 ,RDT_RESOURCE_L2 ,RDT_RESOURCE_MBA , +RDT_RESOURCE_GMBA ,RDT_RESOURCE_SMBA , RDT_RESOURCE_PERF_PKG,
Signature · 1 line
2.43.0
Babu Moger
reviewerPATCH v2 5/8 fs/resctrl: Add the documentation for Global Memory Bandwidth Allocation
In reply to Babu MogerGMBA (added to resctrl by the earlier patches in this series) is a new DRAM bandwidth allocation resource that applies limits across multiple QoS (L3) domains rather than per-L3 like MBA. Each GMBA control domain is aligned to the system's NPS (Nodes Per Socket) configuration. In the schemata file the resource is exposed under the label "GMB" (not "GMBA", to match resctrl's short-label convention) with values in multiples of 1 GB/s.
Document GMBA in Documentation/filesystems/resctrl.rst:
- Add GMBA to the resctrl feature table at the top of the file.
- Add a "Global Memory Bandwidth Allocation (GMBA)" section describing the resource, its control domain and the schemata syntax/unit.
- Add a "Reading/writing the schemata file (on AMD systems) with GMBA feature" section with a self-consistent worked example.
---
v2: Describe GMBA's distinguishing characteristics (NPS-aligned control
domain, per-L3 span, DRAM target).
Added examples to update GMB schemata.
--- Documentation/filesystems/resctrl.rst | 39 +++++++++++++++++++++++++++ 1 file changed, 39 insertions(+)
Documentation/filesystems/resctrl.rst +39 -0
@@ -28,6 +28,7 @@ SMBA (Slow Memory Bandwidth Allocation) "" BMEC (Bandwidth Monitoring Event Configuration) "" ABMC (Assignable Bandwidth Monitoring Counters) "" SDCIAE (Smart Data Cache Injection Allocation Enforcement) "" +GMBA (Global Memory Bandwidth Allocation) "" =============================================================== ================================ Historically, new features were made visible by default in /proc/cpuinfo. This
@@ -964,6 +965,23 @@ Memory bandwidth domain is L3 cache. MB:<cache_id0>=bw_MiBps0;<cache_id1>=bw_MiBps1;... +Global Memory Bandwidth Allocation (GMBA) +----------------------------------------- + +AMD hardware supports Global Memory Bandwidth Allocation (GMBA). GMBA +provides a mechanism for software to specify bandwidth limits for groups +of threads that span multiple QoS (L3) domains. Each such collection of +QoS domains is called a GMBA control domain and is aligned to the system's +NPS (Nodes Per Socket) configuration. NPS is a BIOS-level setting on AMD +processors that selects how many NUMA (Non-Uniform Memory Access) nodes +each CPU socket is divided into. + +The bandwidth domain for GMBA is the GMBA control domain. GMBA is exposed +in the schemata file under the resource label ``GMB``, with values +expressed in multiples of 1 GB/s:: + + GMB:<domain_id0>=bw_GBps0;<domain_id1>=bw_GBps1;... + Slow Memory Bandwidth Allocation (SMBA) --------------------------------------- AMD hardware supports Slow Memory Bandwidth Allocation (SMBA).
@@ -1018,6 +1036,27 @@ For example, to allocate 2GB/s limit on the first cache id: MB:0=2048;1= 16;2=2048;3=2048 L3:0=ffff;1=ffff;2=ffff;3=ffff +Reading/writing the schemata file (on AMD systems) with GMBA feature +-------------------------------------------------------------------- +Reading the schemata file shows the current bandwidth limit on every +GMBA control domain. Values are in multiples of 1 GB/s. + +For example, to set an 8 GB/s limit on GMBA control domain 0, leaving +control domain 1 at its previous limit: + +:: + + # cat schemata + GMB:0=4096;1=4096 + MB:0=8192;1=8192;2=8192;3=8192 + L3:0=ffff;1=ffff;2=ffff;3=ffff + + # echo "GMB:0=8" > schemata + # cat schemata + GMB:0= 8;1=4096 + MB:0=8192;1=8192;2=8192;3=8192 + L3:0=ffff;1=ffff;2=ffff;3=ffff + Reading/writing the schemata file (on AMD systems) with SMBA feature -------------------------------------------------------------------- Reading and writing the schemata file is the same as without SMBA in
Signature · 1 line
2.43.0
Babu Moger
reviewerPATCH v2 6/8 x86,fs/resctrl: Add support for Global Slow Memory Bandwidth Allocation
In reply to Babu MogerAMD PQoS Global Slow Memory Bandwidth Enforcement (GLSBE) is an extension to GLBE that lets software specify slow-memory bandwidth limits for groups of threads that span multiple QoS domains. GLSBE operates within the same GLBE control domains defined by GLBE.
Like GLBE, the GLSBE control domain is by default aligned to the system's NPS (Nodes Per Socket) configuration, a BIOS-level setting on AMD processors that determines how many NUMA (Non-Uniform Memory Access) nodes each CPU socket is divided into.
Support for GLSBE is indicated by CPUID leaf 0x80000020, ECX=0, EBX bit 8. When this bit is set to 1, the platform supports GLSBE.
Because AMD Slow Memory Bandwidth Enforcement is exposed to resctrl as SMBA, Global Slow Memory Bandwidth Enforcement is exposed as GSMBA to keep the naming consistent.
Add following changes to support GLSBE:
- Add X86_FEATURE_GSMBA and its scattered-CPUID entry
(0x80000020 EBX[8]).
- Add RDT_FLAG_GSMBA and the corresponding rdt_options entry so the existing rdt= kernel parameter accepts "gsmba".
The GLSBE feature is documented in:
AMD64 Zen6 Platform Quality of Service (PQOS) Extensions, Publication # 69193 Revision 1.00, Issue Date March 2026
available at https://bugzilla.kernel.org/show_bug.cgi?id=206537
---
v2: Clarify that the GLSBE control domain is aligned to NPS and that NPS
is configured via the BIOS setup utility.
--- Documentation/admin-guide/kernel-parameters.txt | 2 +- arch/x86/include/asm/cpufeatures.h | 1 + arch/x86/kernel/cpu/resctrl/core.c | 2 ++ arch/x86/kernel/cpu/scattered.c | 1 + 4 files changed, 5 insertions(+), 1 deletion(-)
Documentation/admin-guide/kernel-parameters.txt +1 -1
@@ -6452,7 +6452,7 @@ Kernel parameters rdt= [HW,X86,RDT] Turn on/off individual RDT features. List is: cmt, mbmtotal, mbmlocal, l3cat, l3cdp, l2cat, l2cdp, - mba, gmba, smba, bmec, abmc, sdciae, energy[:guid], + mba, gmba, smba, gsmba, bmec, abmc, sdciae, energy[:guid], perf[:guid]. E.g. to turn on cmt and turn off mba use: rdt=cmt,!mba
arch/x86/include/asm/cpufeatures.h +1 -0
@@ -516,6 +516,7 @@*/ #define X86_FEATURE_X2AVIC_EXT (21*32+20)/* AMD SVM x2AVIC support for 4k vCPUs */ #define X86_FEATURE_GMBA (21*32+21)/* Global Memory Bandwidth Allocation */ +#define X86_FEATURE_GSMBA (21*32+22)/* Global Slow Memory Bandwidth Enforcement */ /* * BUG word(s)
arch/x86/kernel/cpu/resctrl/core.c +2 -0
@@ -838,6 +838,7 @@ enum {RDT_FLAG_MBA ,RDT_FLAG_GMBA ,RDT_FLAG_SMBA , +RDT_FLAG_GSMBA ,RDT_FLAG_BMEC ,RDT_FLAG_ABMC , RDT_FLAG_SDCIAE,
@@ -866,6 +867,7 @@ static struct rdt_options rdt_options[] __ro_after_init = {RDT_OPT (RDT_FLAG_MBA ,"mba" ,X86_FEATURE_MBA ),RDT_OPT (RDT_FLAG_GMBA ,"gmba" ,X86_FEATURE_GMBA ),RDT_OPT (RDT_FLAG_SMBA ,"smba" ,X86_FEATURE_SMBA ), +RDT_OPT (RDT_FLAG_GSMBA ,"gsmba" ,X86_FEATURE_GSMBA ),RDT_OPT (RDT_FLAG_BMEC ,"bmec" ,X86_FEATURE_BMEC ),RDT_OPT (RDT_FLAG_ABMC ,"abmc" ,X86_FEATURE_ABMC ), RDT_OPT(RDT_FLAG_SDCIAE, "sdciae", X86_FEATURE_SDCIAE),
arch/x86/kernel/cpu/scattered.c +1 -0
@@ -61,6 +61,7 @@ static const struct cpuid_bit cpuid_bits[] = { { X86_FEATURE_ABMC, CPUID_EBX, 5, 0x80000020, 0 }, { X86_FEATURE_SDCIAE, CPUID_EBX, 6, 0x80000020, 0 }, { X86_FEATURE_GMBA, CPUID_EBX, 7, 0x80000020, 0 }, + { X86_FEATURE_GSMBA, CPUID_EBX, 8, 0x80000020, 0 }, { X86_FEATURE_TSA_SQ_NO, CPUID_ECX, 1, 0x80000021, 0 }, { X86_FEATURE_TSA_L1_NO, CPUID_ECX, 2, 0x80000021, 0 }, { X86_FEATURE_AMD_WORKLOAD_CLASS, CPUID_EAX, 22, 0x80000021, 0 },
Signature · 1 line
2.43.0
Babu Moger
reviewerPATCH v2 7/8 x86,fs/resctrl: Add the resource for Global Slow Memory Bandwidth Allocation
In reply to Babu MogerAMD PQoS Global Slow Memory Bandwidth Enforcement (GLSBE) lets software specify slow-memory (CXL.memory) bandwidth limits for groups of threads inside a GLBE control domain. A GLBE control domain is a set of QoS (L3) domains that are grouped together for global bandwidth allocation; GLSBE reuses those same control domains.
Add the resource definition for GLSBE in the resctrl filesystem so administrators can configure and manage slow-memory bandwidth at GLSBE control domain granularity:
The GLSBE control domain is aligned to the system's NPS (Nodes Per Socket) configuration. Because RDT_RESOURCE_GSMBA uses the RESCTRL_NPS_NODE scope, updates routed through resctrl_arch_update_domains() already flow into resctrl_arch_update_nps(), which programs MSR_IA32_GSMBA_BW_BASE on one CPU per L3 within the domain.
The schemata identifier is named GSMBA (Global Slow Memory Bandwidth Allocation) to mirror SMBA; the architecturally documented name GLSBE is used only in CPUID references.
---
v2: Rewrite commit log to describe GSMBA and NPS scope.
--- arch/x86/include/asm/msr-index.h | 1 + arch/x86/kernel/cpu/resctrl/core.c | 31 ++++++++++++++++++++++++++++++ fs/resctrl/ctrlmondata.c | 4 ++-- fs/resctrl/rdtgroup.c | 16 +++++++++++---- include/linux/resctrl.h | 1 + 5 files changed, 47 insertions(+), 6 deletions(-)
arch/x86/include/asm/msr-index.h +1 -0
@@ -1294,6 +1294,7 @@#define MSR_IA32_L3_QOS_EXT_CFG 0xc00003ff#define MSR_IA32_EVT_CFG_BASE 0xc0000400#define MSR_IA32_GMBA_BW_BASE 0xc0000600 +#define MSR_IA32_GSMBA_BW_BASE 0xc0000680/* AMD-V MSRs */ #define MSR_VM_CR 0xc0010114
arch/x86/kernel/cpu/resctrl/core.c +31 -0
@@ -109,6 +109,15 @@ struct rdt_hw_resource rdt_resources_all[RDT_NUM_RESOURCES] = { .schema_fmt = RESCTRL_SCHEMA_RANGE, }, }, + [RDT_RESOURCE_GSMBA ] = + { + .r_resctrl = { + .name ="GSMBA" , + .ctrl_scope =RESCTRL_NPS_NODE , + .ctrl_domains =ctrl_domain_init (RDT_RESOURCE_GSMBA ), + .schema_fmt =RESCTRL_SCHEMA_RANGE , + }, + }, [RDT_RESOURCE_PERF_PKG ] = { .r_resctrl = {
@@ -261,6 +270,9 @@ static __init bool __rdt_get_mem_config_amd(struct rdt_resource *r)case RDT_RESOURCE_GMBA : subleaf =7 ;break ; +case RDT_RESOURCE_GSMBA : + subleaf =8 ; +break ;default :return false; }
@@ -979,6 +991,19 @@ static __init bool get_slow_mem_config(void)return false; } +static __init bool get_gslow_mem_config (void ) +{ +struct rdt_hw_resource *hw_res = &rdt_resources_all[RDT_RESOURCE_GSMBA ]; + +if (!rdt_cpu_has (X86_FEATURE_GSMBA )) +return false; + +if (boot_cpu_data.x86_vendor ==X86_VENDOR_AMD ) +return __rdt_get_mem_config_amd (&hw_res->r_resctrl); + +return false; +} +static __init bool get_rdt_alloc_resources (void ) {struct rdt_resource *r;
@@ -1017,6 +1042,9 @@ static __init bool get_rdt_alloc_resources(void)if (get_slow_mem_config ()) ret = true; +if (get_gslow_mem_config ()) + ret = true; +return ret; }
@@ -1120,6 +1148,9 @@ static __init void rdt_init_res_defs_amd(void) }else if (r->rid ==RDT_RESOURCE_SMBA ) { hw_res->msr_base =MSR_IA32_SMBA_BW_BASE ; hw_res->msr_update = mba_wrmsr_amd; + }else if (r->rid ==RDT_RESOURCE_GSMBA ) { + hw_res->msr_base =MSR_IA32_GSMBA_BW_BASE ; + hw_res->msr_update = mba_wrmsr_amd; } } }
fs/resctrl/ctrlmondata.c +2 -2
@@ -247,8 +247,8 @@ static int parse_line(char *line, struct resctrl_schema *s,if (rdtgrp->mode ==RDT_MODE_PSEUDO_LOCKSETUP && (r->rid ==RDT_RESOURCE_MBA || r->rid ==RDT_RESOURCE_GMBA || - r->rid ==RDT_RESOURCE_SMBA )) { -rdt_last_cmd_puts ("Cannot pseudo-lock MBA/SMBA/GMBA resource\n" ); + r->rid ==RDT_RESOURCE_SMBA || r->rid ==RDT_RESOURCE_GSMBA )) { +rdt_last_cmd_puts ("Cannot pseudo-lock MBA/SMBA/GMBA/GSMBA resource\n" );return -EINVAL ; }
fs/resctrl/rdtgroup.c +12 -4
@@ -1413,7 +1413,7 @@ static bool rdtgroup_mode_test_exclusive(struct rdtgroup *rdtgrp)list_for_each_entry (s, &resctrl_schema_all, list) { r = s->res;if (r->rid ==RDT_RESOURCE_MBA || r->rid ==RDT_RESOURCE_GMBA || - r->rid ==RDT_RESOURCE_SMBA ) + r->rid ==RDT_RESOURCE_SMBA || r->rid ==RDT_RESOURCE_GSMBA )continue ; has_cache = true;list_for_each_entry (d, &r->ctrl_domains, hdr.list) {
@@ -1619,7 +1619,8 @@ static int rdtgroup_size_show(struct kernfs_open_file *of, type);if (r->rid ==RDT_RESOURCE_MBA || r->rid ==RDT_RESOURCE_GMBA || - r->rid ==RDT_RESOURCE_SMBA ) + r->rid ==RDT_RESOURCE_SMBA || + r->rid ==RDT_RESOURCE_GSMBA ) size = ctrl;else size =rdtgroup_cbm_to_size (r, d, ctrl);
@@ -2172,7 +2173,7 @@ static struct rftype *rdtgroup_get_rftype_by_name(const char *name)static void thread_throttle_mode_init (void ) {enum membw_throttle_mode throttle_mode =THREAD_THROTTLE_UNDEFINED ; -struct rdt_resource *r_mba, *r_gmba, *r_smba; +struct rdt_resource *r_mba, *r_gmba, *r_smba, *r_gsmba; r_mba =resctrl_arch_get_resource (RDT_RESOURCE_MBA ); if (r_mba->alloc_capable &&
@@ -2189,6 +2190,11 @@ static void thread_throttle_mode_init(void) r_smba->membw.throttle_mode != THREAD_THROTTLE_UNDEFINED) throttle_mode = r_smba->membw.throttle_mode; + r_gsmba =resctrl_arch_get_resource (RDT_RESOURCE_GSMBA ); +if (r_gsmba->alloc_capable && + r_gsmba->membw.throttle_mode !=THREAD_THROTTLE_UNDEFINED ) + throttle_mode = r_gsmba->membw.throttle_mode; +if (throttle_mode ==THREAD_THROTTLE_UNDEFINED )return ;
@@ -2405,6 +2411,7 @@ static unsigned long fflags_from_resource(struct rdt_resource *r)case RDT_RESOURCE_MBA :case RDT_RESOURCE_GMBA :case RDT_RESOURCE_SMBA : +case RDT_RESOURCE_GSMBA :return RFTYPE_RES_MB ;case RDT_RESOURCE_PERF_PKG :return RFTYPE_RES_PERF_PKG ;
@@ -3658,7 +3665,8 @@ static int rdtgroup_init_alloc(struct rdtgroup *rdtgrp) r = s->res;if (r->rid ==RDT_RESOURCE_MBA || r->rid ==RDT_RESOURCE_GMBA || - r->rid ==RDT_RESOURCE_SMBA ) { + r->rid ==RDT_RESOURCE_SMBA || + r->rid ==RDT_RESOURCE_GSMBA ) {rdtgroup_init_mba (r, rdtgrp->closid);if (is_mba_sc (r))continue ;
include/linux/resctrl.h +1 -0
@@ -54,6 +54,7 @@ enum resctrl_res_level {RDT_RESOURCE_MBA ,RDT_RESOURCE_GMBA ,RDT_RESOURCE_SMBA , +RDT_RESOURCE_GSMBA , RDT_RESOURCE_PERF_PKG,/* Must be the last */
Signature · 1 line
2.43.0
Babu Moger
reviewerPATCH v2 8/8 fs/resctrl: Add the documentation for Global Slow Memory Bandwidth Allocation
In reply to Babu MogerAMD Global Slow Memory Bandwidth Allocation (GSMBA) is the slow-memory (CXL.memory) counterpart of GMBA. Like GMBA it applies bandwidth limits to groups of threads that span multiple QoS (L3) domains, using the same GMBA control domains that are aligned to the system's NPS (Nodes Per Socket) configuration. GSMBA is exposed in the schemata file under the resource label "GSMBA", with values in multiples of 1 GB/s.
Document GSMBA in Documentation/filesystems/resctrl.rst:
- Add GSMBA to the resctrl feature table at the top of the file.
- Add a "Global Slow Memory Bandwidth Allocation (GSMBA)" section describing the resource, its bandwidth domain and the schemata syntax and unit.
- Add a "Reading/writing the schemata file (on AMD systems) with GSMBA feature" section with a worked example.
---
v2: Updated the documentation with clarity and format issues.
Improved changelog.
--- Documentation/filesystems/resctrl.rst | 42 +++++++++++++++++++++++++++ 1 file changed, 42 insertions(+)
Documentation/filesystems/resctrl.rst +42 -0
@@ -29,6 +29,7 @@ BMEC (Bandwidth Monitoring Event Configuration) "" ABMC (Assignable Bandwidth Monitoring Counters) "" SDCIAE (Smart Data Cache Injection Allocation Enforcement) "" GMBA (Global Memory Bandwidth Allocation) "" +GSMBA (Global Slow Memory Bandwidth Allocation) "" =============================================================== ================================ Historically, new features were made visible by default in /proc/cpuinfo. This
@@ -1001,6 +1002,22 @@ is formatted as: SMBA:<cache_id0>=bandwidth0;<cache_id1>=bandwidth1;... +Global Slow Memory Bandwidth Allocation (GSMBA) +----------------------------------------------- + +AMD hardware supports Global Slow Memory Bandwidth Allocation (GSMBA). +GSMBA provides a mechanism for software to specify slow-memory bandwidth +limits for groups of threads that span multiple QoS (L3) domains. Each +such collection of QoS domains is called a GSMBA control domain. GSMBA +operates similarly to GMBA, but targets slow memory (CXL.memory) instead +of DRAM; it reuses the same NPS-aligned control domains as GMBA. + +The bandwidth domain for GSMBA is the GSMBA control domain. GSMBA is +exposed in the schemata file under the resource label ``GSMBA``, with +values expressed in multiples of 1 GB/s:: + + GSMBA:<domain_id0>=bw_GBps0;<domain_id1>=bw_GBps1;... + Reading/writing the schemata file --------------------------------- Reading the schemata file will show the state of all resources
@@ -1077,6 +1094,31 @@ For example, to allocate 8GB/s limit on the first cache id: MB:0=2048;1=2048;2=2048;3=2048 L3:0=ffff;1=ffff;2=ffff;3=ffff +Reading/writing the schemata file (on AMD systems) with GSMBA feature +--------------------------------------------------------------------- +Reading the schemata file shows the current bandwidth limit on every +GSMBA control domain. Values are in multiples of 1 GB/s. + +For example, to set an 8 GB/s limit on GSMBA control domain 0, leaving +control domain 1 at its previous limit: + +:: + + # cat schemata + GSMBA:0=4096;1=4096 + SMBA:0=8192;1=8192;2=8192;3=8192 + GMB:0=4096;1=4096 + MB:0=8192;1=8192;2=8192;3=8192 + L3:0=ffff;1=ffff;2=ffff;3=ffff + + # echo "GSMBA:0=8" > schemata + # cat schemata + GSMBA:0= 8;1=4096 + SMBA:0=8192;1=8192;2=8192;3=8192 + GMB:0=4096;1=4096 + MB:0=8192;1=8192;2=8192;3=8192 + L3:0=ffff;1=ffff;2=ffff;3=ffff + Cache Pseudo-Locking ==================== CAT enables a user to specify the amount of cache space that an
Signature · 1 line
2.43.0
Reinette Chatre
maintainerHi Babu,
Quoted 76 lines · Babu Moger
This series adds resctrl support for two new AMD memory-bandwidth allocation features:
- GMBA - Global Memory Bandwidth Allocation (hardware name: GLBE). Bounds DRAM bandwidth for groups of threads that span multiple L3 QoS domains, rather than being per-L3 like MBA.
- GSMBA - Global Slow Memory Bandwidth Allocation (hardware name:
GLSBE). The CXL.memory / slow-memory counterpart of GMBA, analogous to how SMBA relates to MBA.Both features share a new "NPS-node" control domain: a set of QoS (L3) domains grouped together and aligned to the system's NPS (Nodes Per Socket) BIOS configuration. Although the control domain is NPS-scoped, the underlying bandwidth-limit MSRs (
MSR_IA32_GMBA_BW_BASE0xc0000600,MSR_IA32_GSMBA_BW_BASE0xc0000680) are instantiated per L3. Programming a single control domain therefore requires writing the MSR on one CPU per L3 that the domain spans - a new pattern for resctrl. Patches 2/8 and 3/8 introduce that infrastructure so the new resources can reuse it.The features are documented in:
AMD64 Zen6 Platform Quality of Service (PQOS) Extensions, Publication # 69193 Revision 1.00, Issue Date March 2026
available at https://bugzilla.kernel.org/show_bug.cgi?id=206537
Series overview
---------------Patches 1-5 to enable GMBA:
1/8 x86,fs/resctrl: Add support for Global Bandwidth Enforcement (GLBE)
2/8 x86/resctrl: Add
RESCTRL_NPS_NODEscope for AMD NPS-aligned domains
Add a newctrl_scopevalue for resctrl resources whose control domain spans multiple L3s within an NPS node.3/8 x86/resctrl: Update control MSRs per L3 for NPS-scoped resources
Addresctrl_arch_update_nps(): builds a cpumask with one CPU per distinct L3 in the domain, then issuesrdt_ctrl_update()viasmp_call_function_many()on that mask. Falls back to the full domain mask if the scratch masks cannot be built. Routeresctrl_arch_update_domains()andresctrl_arch_reset_all_ctrls()through this helper whenctrl_scope==RESCTRL_NPS_NODE.4/8 x86,fs/resctrl: Add the resource for Global Memory Bandwidth Allocation
RegisterRDT_RESOURCE_GMBAin rdt_resources_all[] with ctrl_scope=RESCTRL_NPS_NODE and schema_fmt=RANGE, add commands to discover feature details.5/8 fs/resctrl: Add the documentation for Global Memory Bandwidth Allocation
Add examples inDocumentation/filesystems/resctrl.rst.Patches 6-8 to enable GSMBA in the same shape:
6/8 x86,fs/resctrl: Add support for Global Slow Memory Bandwidth Allocation
7/8 x86,fs/resctrl: Add the resource for Global Slow Memory Bandwidth Allocation
RegisterRDT_RESOURCE_GSMBAwith ctrl_scope=RESCTRL_NPS_NODE.8/8 fs/resctrl: Add the documentation for Global Slow Memory Bandwidth Allocation
Add examples inDocumentation/filesystems/resctrl.rst.Changes since v1
----------------
- Earlier sent RFC(v1) with Global Bandwidth Enforcement (GLBE) and Privilege Level Zero Association (PLZA). This series only handles Global Memory Bandwidth Allocation. Both the features are sent separately.- Documentation
* Fixed grammar in the GMBA / GSMBA sections ofresctrl.rst.
* Added examples to update GMBA and GSMBA inresctrl.rstdocumentation.
Quoted 7 lines
- Major changes are releated to
RESCTRL_NPS_NODEscope handling.- Commit messages
* Reworked the changelogs in all the patches.Previous Revisions:
v1 : https://lore.kernel.org/lkml/[email protected]/
What are your expectations from this submission? From what I can tell this ignores v1 feedback in several ways:
- It introduces two new resources, GMBA and GSMBA, when the previous discussion agreed that these are not actually new resources but instead new controls for the existing MBA/SMBA resources.
- It does not mention or attempt to address dependency on new resource schema descriptions [1] to support user space in understanding how to interact with the new GMBA/GSMBA controls but instead defers that to a snippet in the documentation that user space needs to parse to know this control operates at multiples of 1GB/s.
Apart from ignoring v1 feedback this new version appears to complicate user interface even more since now it is possible for there to be a single control that may operate at different scopes but from what I can tell there is nothing that helps user understand whether, for example, domain "0" means the whole system or a NUMA node?
We have discussed several times now how resctrl interface needs to be enhanced to support this and other upcoming features from Intel, RISC-V, Arm MPAM, and NVidia. It is thus unexpected that this submission ignores all the previous discussions.
Since there are so many dependencies on the new schema format support I am prioritizing this and created a PoC that I am currently refining and hope to share soon. We can collaborate on this to ensure that it provides a good foundation for the GMBA and GSMBA support.
Consider what I describe in [2] - even in that response I speculate that a "scope" may be needed and this seems to be case. I believe would help this "NPS = 4" scenario. Adding "scope" to what I shared in
[2] may look like:
info/
└── MB/
└── resource_schemata/
├── GMB/
│ ├── max:4096
│ ├── min:1
│ ├── resolution:1
│ ├── scale:1
│ ├── tolerance:0
│ ├── type:scalar linear
│ ├── scope:NODE
│ └── unit:GBps
└── MB/
├── max:8192
├── min:1
├── resolution:8
├── scale:1
├── tolerance:0
├── type:scalar linear
├── scope:L3
└── unit:GBpsWith a "scope" property of the control user space can know what the domain ID in the schemata file refers to. In above example the "GMB" control has "NODE" scope so user space knows that a domain ID refers to NUMA node. If the system is "NPS = 4" then the scope could be, for example, "SYSTEM" (for the lack of a better term) so that user space knows that "0" means entire system. What do you think?
Also note how the other control properties helps user understand what the schemata file control values mean. This is what I expected the GMBA/GSMBA enabling to look like ... and you seemed to agree [3] in v1 discussion. What changed?
Reinette
[1] https://lore.kernel.org/lkml/aPtfMFfLV1l%[email protected]/
[2] https://lore.kernel.org/lkml/[email protected]/
[3] https://lore.kernel.org/lkml/[email protected]/
Hi Reinette,
Quoted 96 lines · Reinette Chatre
Hi Babu,
Quoted 76 lines · Babu Moger
This series adds resctrl support for two new AMD memory-bandwidth allocation features:
- GMBA - Global Memory Bandwidth Allocation (hardware name: GLBE). Bounds DRAM bandwidth for groups of threads that span multiple L3 QoS domains, rather than being per-L3 like MBA.
- GSMBA - Global Slow Memory Bandwidth Allocation (hardware name:
GLSBE). The CXL.memory / slow-memory counterpart of GMBA, analogous to how SMBA relates to MBA.Both features share a new "NPS-node" control domain: a set of QoS (L3) domains grouped together and aligned to the system's NPS (Nodes Per Socket) BIOS configuration. Although the control domain is NPS-scoped, the underlying bandwidth-limit MSRs (
MSR_IA32_GMBA_BW_BASE0xc0000600,MSR_IA32_GSMBA_BW_BASE0xc0000680) are instantiated per L3. Programming a single control domain therefore requires writing the MSR on one CPU per L3 that the domain spans - a new pattern for resctrl. Patches 2/8 and 3/8 introduce that infrastructure so the new resources can reuse it.The features are documented in:
AMD64 Zen6 Platform Quality of Service (PQOS) Extensions, Publication # 69193 Revision 1.00, Issue Date March 2026
available at https://bugzilla.kernel.org/show_bug.cgi?id=206537
Series overview
---------------Patches 1-5 to enable GMBA:
1/8 x86,fs/resctrl: Add support for Global Bandwidth Enforcement (GLBE)
2/8 x86/resctrl: Add
RESCTRL_NPS_NODEscope for AMD NPS-aligned domains
Add a newctrl_scopevalue for resctrl resources whose control domain spans multiple L3s within an NPS node.3/8 x86/resctrl: Update control MSRs per L3 for NPS-scoped resources
Addresctrl_arch_update_nps(): builds a cpumask with one CPU per distinct L3 in the domain, then issuesrdt_ctrl_update()viasmp_call_function_many()on that mask. Falls back to the full domain mask if the scratch masks cannot be built. Routeresctrl_arch_update_domains()andresctrl_arch_reset_all_ctrls()through this helper whenctrl_scope==RESCTRL_NPS_NODE.4/8 x86,fs/resctrl: Add the resource for Global Memory Bandwidth Allocation
RegisterRDT_RESOURCE_GMBAin rdt_resources_all[] with ctrl_scope=RESCTRL_NPS_NODE and schema_fmt=RANGE, add commands to discover feature details.5/8 fs/resctrl: Add the documentation for Global Memory Bandwidth Allocation
Add examples inDocumentation/filesystems/resctrl.rst.Patches 6-8 to enable GSMBA in the same shape:
6/8 x86,fs/resctrl: Add support for Global Slow Memory Bandwidth Allocation
7/8 x86,fs/resctrl: Add the resource for Global Slow Memory Bandwidth Allocation
RegisterRDT_RESOURCE_GSMBAwith ctrl_scope=RESCTRL_NPS_NODE.8/8 fs/resctrl: Add the documentation for Global Slow Memory Bandwidth Allocation
Add examples inDocumentation/filesystems/resctrl.rst.Changes since v1
----------------
- Earlier sent RFC(v1) with Global Bandwidth Enforcement (GLBE) and Privilege Level Zero Association (PLZA). This series only handles Global Memory Bandwidth Allocation. Both the features are sent separately.- Documentation
* Fixed grammar in the GMBA / GSMBA sections ofresctrl.rst.
* Added examples to update GMBA and GSMBA inresctrl.rstdocumentation.Quoted 7 lines
- Major changes are releated to
RESCTRL_NPS_NODEscope handling.- Commit messages
* Reworked the changelogs in all the patches.Previous Revisions:
v1 : https://lore.kernel.org/lkml/[email protected]/What are your expectations from this submission? From what I can tell this ignores
v1 feedback in several ways:
- It introduces two new resources, GMBA and GSMBA, when the previous discussion agreed that these are not actually new resources but instead new controls for the existing MBA/SMBA resources.
- It does not mention or attempt to address dependency on new resource schema descriptions [1] to support user space in understanding how to interact with the new GMBA/GSMBA controls but instead defers that to a snippet in the documentation that user space needs to parse to know this control operates at multiples of 1GB/s.
Quoted 8 lines
Apart from ignoring v1 feedback this new version appears to complicate user interface even more since now it is possible for there to be a single control that may operate at different scopes but from what I can tell there is nothing that helps user understand whether, for example, domain "0" means the whole system or a NUMA node?
We have discussed several times now how resctrl interface needs to be enhanced to support this and other upcoming features from Intel, RISC-V, Arm MPAM, and NVidia. It is thus unexpected that this submission ignores all the previous discussions.
I think there may be some misunderstanding on this topic.
Yes, we discussed it earlier. It depends on other requirements (region-aware aspects), so I assumed it would be handled by someone with full context and addressed as a separate feature. I didn’t have complete visibility into all the requirements.
Quoted 4 lines
Since there are so many dependencies on the new schema format support I am prioritizing this and created a PoC that I am currently refining and hope to share soon. We can collaborate on this to ensure that it provides a good foundation for the GMBA and GSMBA support.
That is good to know. Let me know when you are ready.
Could you please share which parts of the feature (e.g., Part 1, Part 2, etc.) you are planning to cover in your PoC?
Quoted 27 lines
Consider what I describe in [2] - even in that response I speculate that a "scope" may be needed and this seems to be case. I believe would help this "NPS = 4" scenario. Adding "scope" to what I shared in
[2] may look like:info/
└── MB/ └── resource_schemata/ ├── GMB/ │ ├── max:4096 │ ├── min:1 │ ├── resolution:1 │ ├── scale:1 │ ├── tolerance:0 │ ├── type:scalar linear │ ├── scope:NODE │ └── unit:GBps └── MB/ ├── max:8192 ├── min:1 ├── resolution:8 ├── scale:1 ├── tolerance:0 ├── type:scalar linear ├── scope:L3 └── unit:GBps
Quoted 5 lines
With a "scope" property of the control user space can know what the domain ID in the schemata file refers to. In above example the "GMB" control has "NODE" scope so user space knows that a domain ID refers to NUMA node. If the system is "NPS = 4" then the scope could be, for example, "SYSTEM" (for the lack of a better term) so that user space knows that "0" means entire system. What do you think?
Yes. Sound good to me.
Quoted 4 lines
Also note how the other control properties helps user understand what the schemata file control values mean. This is what I expected the GMBA/GSMBA enabling to look like ... and you seemed to agree [3] in v1 discussion. What changed?
Again, I was under the impression that this would be handled as a separate feature and patch series.
Thanks
Babu
Reinette Chatre
maintainerHi Babu,
Quoted 104 lines · Babu
Hi Reinette,
Quoted 101 lines · Reinette Chatre
Hi Babu,
Quoted 76 lines · Babu Moger
This series adds resctrl support for two new AMD memory-bandwidth allocation features:
- GMBA - Global Memory Bandwidth Allocation (hardware name: GLBE). Bounds DRAM bandwidth for groups of threads that span multiple L3 QoS domains, rather than being per-L3 like MBA.
- GSMBA - Global Slow Memory Bandwidth Allocation (hardware name:
GLSBE). The CXL.memory / slow-memory counterpart of GMBA, analogous to how SMBA relates to MBA.Both features share a new "NPS-node" control domain: a set of QoS (L3) domains grouped together and aligned to the system's NPS (Nodes Per Socket) BIOS configuration. Although the control domain is NPS-scoped, the underlying bandwidth-limit MSRs (
MSR_IA32_GMBA_BW_BASE0xc0000600,MSR_IA32_GSMBA_BW_BASE0xc0000680) are instantiated per L3. Programming a single control domain therefore requires writing the MSR on one CPU per L3 that the domain spans - a new pattern for resctrl. Patches 2/8 and 3/8 introduce that infrastructure so the new resources can reuse it.The features are documented in:
AMD64 Zen6 Platform Quality of Service (PQOS) Extensions, Publication # 69193 Revision 1.00, Issue Date March 2026
available at https://bugzilla.kernel.org/show_bug.cgi?id=206537
Series overview
---------------Patches 1-5 to enable GMBA:
1/8 x86,fs/resctrl: Add support for Global Bandwidth Enforcement (GLBE)
2/8 x86/resctrl: Add
RESCTRL_NPS_NODEscope for AMD NPS-aligned domains Add a newctrl_scopevalue for resctrl resources whose control domain spans multiple L3s within an NPS node.3/8 x86/resctrl: Update control MSRs per L3 for NPS-scoped resources Add
resctrl_arch_update_nps(): builds a cpumask with one CPU per distinct L3 in the domain, then issuesrdt_ctrl_update()viasmp_call_function_many()on that mask. Falls back to the full domain mask if the scratch masks cannot be built. Routeresctrl_arch_update_domains()andresctrl_arch_reset_all_ctrls()through this helper whenctrl_scope==RESCTRL_NPS_NODE.4/8 x86,fs/resctrl: Add the resource for Global Memory Bandwidth Allocation Register
RDT_RESOURCE_GMBAin rdt_resources_all[] with ctrl_scope=RESCTRL_NPS_NODE and schema_fmt=RANGE, add commands to discover feature details.5/8 fs/resctrl: Add the documentation for Global Memory Bandwidth Allocation Add examples in
Documentation/filesystems/resctrl.rst.Patches 6-8 to enable GSMBA in the same shape:
6/8 x86,fs/resctrl: Add support for Global Slow Memory Bandwidth Allocation
7/8 x86,fs/resctrl: Add the resource for Global Slow Memory Bandwidth Allocation Register
RDT_RESOURCE_GSMBAwith ctrl_scope=RESCTRL_NPS_NODE.8/8 fs/resctrl: Add the documentation for Global Slow Memory Bandwidth Allocation Add examples in
Documentation/filesystems/resctrl.rst.Changes since v1
----------------
- Earlier sent RFC(v1) with Global Bandwidth Enforcement (GLBE) and Privilege Level Zero Association (PLZA). This series only handles Global Memory Bandwidth Allocation. Both the features are sent separately.- Documentation
* Fixed grammar in the GMBA / GSMBA sections ofresctrl.rst.
* Added examples to update GMBA and GSMBA inresctrl.rstdocumentation.Quoted 7 lines
- Major changes are releated to
RESCTRL_NPS_NODEscope handling.- Commit messages
* Reworked the changelogs in all the patches.Previous Revisions:
v1 : https://lore.kernel.org/lkml/[email protected]/What are your expectations from this submission? From what I can tell this ignores
v1 feedback in several ways:
- It introduces two new resources, GMBA and GSMBA, when the previous discussion agreed that these are not actually new resources but instead new controls for the existing MBA/SMBA resources.
- It does not mention or attempt to address dependency on new resource schema descriptions [1] to support user space in understanding how to interact with the new GMBA/GSMBA controls but instead defers that to a snippet in the documentation that user space needs to parse to know this control operates at multiples of 1GB/s.Apart from ignoring v1 feedback this new version appears to complicate user interface even more since now it is possible for there to be a single control that may operate at different scopes but from what I can tell there is nothing that helps user understand whether, for example, domain "0" means the whole system or a NUMA node?
Quoted 7 lines
Quoted 3 lines
We have discussed several times now how resctrl interface needs to be enhanced to support this and other upcoming features from Intel, RISC-V, Arm MPAM, and NVidia. It is thus unexpected that this submission ignores all the previous discussions.
I think there may be some misunderstanding on this topic.
Yes, we discussed it earlier. It depends on other requirements (region-aware aspects), so I assumed it would be handled by someone with full context and addressed as a separate feature. I didn’t have complete visibility into all the requirements.
Please read https://lore.kernel.org/lkml/[email protected]/ again.
You should have complete visibility into the foundation of this work since one of the primary goals is to address the resctrl interface breakage that came with the initial AMD support for MBA that resctrl has been living with until now.
With this series you completely disregard attempts to support users in understanding how to interact with the schemata file and instead introduce *another* obfuscated control. I will not support this.
Also, no, this does not depend on region-aware work. Needing to support multiple controls for a single resource is independent from region-aware.
Quoted 7 lines
Quoted 3 lines
Since there are so many dependencies on the new schema format support I am prioritizing this and created a PoC that I am currently refining and hope to share soon. We can collaborate on this to ensure that it provides a good foundation for the GMBA and GSMBA support.
That is good to know. Let me know when you are ready.
Could you please share which parts of the feature (e.g., Part 1, Part 2, etc.) you are planning to cover in your PoC?
All three parts mentioned in https://lore.kernel.org/lkml/[email protected]/
This does not address all the features discussed, for example it does not support emulated controls, but I hope it is enough of a foundation to build on.
Quoted 32 lines
Quoted 26 lines
Consider what I describe in [2] - even in that response I speculate that a "scope" may be needed and this seems to be case. I believe would help this "NPS = 4" scenario. Adding "scope" to what I shared in
[2] may look like:info/
└── MB/ └── resource_schemata/ ├── GMB/ │ ├── max:4096 │ ├── min:1 │ ├── resolution:1 │ ├── scale:1 │ ├── tolerance:0 │ ├── type:scalar linear │ ├── scope:NODE │ └── unit:GBps └── MB/ ├── max:8192 ├── min:1 ├── resolution:8 ├── scale:1 ├── tolerance:0 ├── type:scalar linear ├── scope:L3 └── unit:GBpsQuoted 5 lines
With a "scope" property of the control user space can know what the domain ID in the schemata file refers to. In above example the "GMB" control has "NODE" scope so user space knows that a domain ID refers to NUMA node. If the system is "NPS = 4" then the scope could be, for example, "SYSTEM" (for the lack of a better term) so that user space knows that "0" means entire system. What do you think?
Quoted 8 lines
Yes. Sound good to me.
Quoted 4 lines
Also note how the other control properties helps user understand what the schemata file control values mean. This is what I expected the GMBA/GSMBA enabling to look like ... and you seemed to agree [3] in v1 discussion. What changed?
Again, I was under the impression that this would be handled as a separate feature and patch series.
Right. And since GMBA/GSMBA depends on it this submission without resolving this this dependency first is unexpected.
Reinette
Hi Reinette,
Quoted 132 lines · Reinette Chatre
Hi Babu,
Quoted 104 lines · Babu
Hi Reinette,
Quoted 101 lines · Reinette Chatre
Hi Babu,
Quoted 76 lines · Babu Moger
This series adds resctrl support for two new AMD memory-bandwidth allocation features:
- GMBA - Global Memory Bandwidth Allocation (hardware name: GLBE). Bounds DRAM bandwidth for groups of threads that span multiple L3 QoS domains, rather than being per-L3 like MBA.
- GSMBA - Global Slow Memory Bandwidth Allocation (hardware name:
GLSBE). The CXL.memory / slow-memory counterpart of GMBA, analogous to how SMBA relates to MBA.Both features share a new "NPS-node" control domain: a set of QoS (L3) domains grouped together and aligned to the system's NPS (Nodes Per Socket) BIOS configuration. Although the control domain is NPS-scoped, the underlying bandwidth-limit MSRs (
MSR_IA32_GMBA_BW_BASE0xc0000600,MSR_IA32_GSMBA_BW_BASE0xc0000680) are instantiated per L3. Programming a single control domain therefore requires writing the MSR on one CPU per L3 that the domain spans - a new pattern for resctrl. Patches 2/8 and 3/8 introduce that infrastructure so the new resources can reuse it.The features are documented in:
AMD64 Zen6 Platform Quality of Service (PQOS) Extensions, Publication # 69193 Revision 1.00, Issue Date March 2026
available at https://bugzilla.kernel.org/show_bug.cgi?id=206537
Series overview
---------------Patches 1-5 to enable GMBA:
1/8 x86,fs/resctrl: Add support for Global Bandwidth Enforcement (GLBE)
2/8 x86/resctrl: Add
RESCTRL_NPS_NODEscope for AMD NPS-aligned domains Add a newctrl_scopevalue for resctrl resources whose control domain spans multiple L3s within an NPS node.3/8 x86/resctrl: Update control MSRs per L3 for NPS-scoped resources Add
resctrl_arch_update_nps(): builds a cpumask with one CPU per distinct L3 in the domain, then issuesrdt_ctrl_update()viasmp_call_function_many()on that mask. Falls back to the full domain mask if the scratch masks cannot be built. Routeresctrl_arch_update_domains()andresctrl_arch_reset_all_ctrls()through this helper whenctrl_scope==RESCTRL_NPS_NODE.4/8 x86,fs/resctrl: Add the resource for Global Memory Bandwidth Allocation Register
RDT_RESOURCE_GMBAin rdt_resources_all[] with ctrl_scope=RESCTRL_NPS_NODE and schema_fmt=RANGE, add commands to discover feature details.5/8 fs/resctrl: Add the documentation for Global Memory Bandwidth Allocation Add examples in
Documentation/filesystems/resctrl.rst.Patches 6-8 to enable GSMBA in the same shape:
6/8 x86,fs/resctrl: Add support for Global Slow Memory Bandwidth Allocation
7/8 x86,fs/resctrl: Add the resource for Global Slow Memory Bandwidth Allocation Register
RDT_RESOURCE_GSMBAwith ctrl_scope=RESCTRL_NPS_NODE.8/8 fs/resctrl: Add the documentation for Global Slow Memory Bandwidth Allocation Add examples in
Documentation/filesystems/resctrl.rst.Changes since v1
----------------
- Earlier sent RFC(v1) with Global Bandwidth Enforcement (GLBE) and Privilege Level Zero Association (PLZA). This series only handles Global Memory Bandwidth Allocation. Both the features are sent separately.- Documentation
* Fixed grammar in the GMBA / GSMBA sections ofresctrl.rst.
* Added examples to update GMBA and GSMBA inresctrl.rstdocumentation.Quoted 7 lines
- Major changes are releated to
RESCTRL_NPS_NODEscope handling.- Commit messages
* Reworked the changelogs in all the patches.Previous Revisions:
v1 : https://lore.kernel.org/lkml/[email protected]/What are your expectations from this submission? From what I can tell this ignores
v1 feedback in several ways:
- It introduces two new resources, GMBA and GSMBA, when the previous discussion agreed that these are not actually new resources but instead new controls for the existing MBA/SMBA resources.
- It does not mention or attempt to address dependency on new resource schema descriptions [1] to support user space in understanding how to interact with the new GMBA/GSMBA controls but instead defers that to a snippet in the documentation that user space needs to parse to know this control operates at multiples of 1GB/s.Apart from ignoring v1 feedback this new version appears to complicate user interface even more since now it is possible for there to be a single control that may operate at different scopes but from what I can tell there is nothing that helps user understand whether, for example, domain "0" means the whole system or a NUMA node?
Quoted 7 lines
Quoted 3 lines
We have discussed several times now how resctrl interface needs to be enhanced to support this and other upcoming features from Intel, RISC-V, Arm MPAM, and NVidia. It is thus unexpected that this submission ignores all the previous discussions.
I think there may be some misunderstanding on this topic.
Yes, we discussed it earlier. It depends on other requirements (region-aware aspects), so I assumed it would be handled by someone with full context and addressed as a separate feature. I didn’t have complete visibility into all the requirements.
Please read https://lore.kernel.org/lkml/[email protected]/ again.
You should have complete visibility into the foundation of this work since one of the primary goals is to address the resctrl interface breakage that came with the initial AMD support for MBA that resctrl has been living with until now.
With this series you completely disregard attempts to support users in understanding how to interact with the schemata file and instead introduce *another* obfuscated control. I will not support this.
Also, no, this does not depend on region-aware work. Needing to support multiple controls for a single resource is independent from region-aware.
Quoted 4 lines
Quoted 3 lines
Since there are so many dependencies on the new schema format support I am prioritizing this and created a PoC that I am currently refining and hope to share soon. We can collaborate on this to ensure that it provides a good foundation for the GMBA and GSMBA support.
Quoted 8 lines
Quoted 3 lines
That is good to know. Let me know when you are ready.
Could you please share which parts of the feature (e.g., Part 1, Part 2, etc.) you are planning to cover in your PoC?
All three parts mentioned in https://lore.kernel.org/lkml/[email protected]/
This does not address all the features discussed, for example it does not support emulated controls, but I hope it is enough of a foundation to build on.
Please share your code when you are ready. I can build GMB and GSMBA on top of your patches. Hopefully, I can reuse some of the code from this series.
Thanks
Babu
Hi Reinette,
Quoted 139 lines · Babu
Hi Reinette,
Quoted 136 lines · Reinette Chatre
Hi Babu,
Quoted 106 lines · Babu
Hi Reinette,
Quoted 103 lines · Reinette Chatre
Hi Babu,
Quoted 76 lines · Babu Moger
This series adds resctrl support for two new AMD memory-bandwidth allocation features:
- GMBA - Global Memory Bandwidth Allocation (hardware name: GLBE). Bounds DRAM bandwidth for groups of threads that span multiple L3 QoS domains, rather than being per-L3 like MBA.
- GSMBA - Global Slow Memory Bandwidth Allocation (hardware name:
GLSBE). The CXL.memory / slow-memory counterpart of GMBA, analogous to how SMBA relates to MBA.Both features share a new "NPS-node" control domain: a set of QoS (L3) domains grouped together and aligned to the system's NPS (Nodes Per Socket) BIOS configuration. Although the control domain is NPS-scoped, the underlying bandwidth-limit MSRs (
MSR_IA32_GMBA_BW_BASE0xc0000600,MSR_IA32_GSMBA_BW_BASE0xc0000680) are instantiated per L3. Programming a single control domain therefore requires writing the MSR on one CPU per L3 that the domain spans - a new pattern for resctrl. Patches 2/8 and 3/8 introduce that infrastructure so the new resources can reuse it.The features are documented in:
AMD64 Zen6 Platform Quality of Service (PQOS) Extensions, Publication # 69193 Revision 1.00, Issue Date March 2026
available at https://bugzilla.kernel.org/show_bug.cgi?id=206537
Series overview
---------------Patches 1-5 to enable GMBA:
1/8 x86,fs/resctrl: Add support for Global Bandwidth Enforcement (GLBE)
2/8 x86/resctrl: Add
RESCTRL_NPS_NODEscope for AMD NPS- aligned domains Add a newctrl_scopevalue for resctrl resources whose control domain spans multiple L3s within an NPS node.3/8 x86/resctrl: Update control MSRs per L3 for NPS-scoped resources Add
resctrl_arch_update_nps(): builds a cpumask with one CPU per distinct L3 in the domain, then issuesrdt_ctrl_update()viasmp_call_function_many()on that mask. Falls back to the full domain mask if the scratch masks cannot be built. Routeresctrl_arch_update_domains()andresctrl_arch_reset_all_ctrls()through this helper whenctrl_scope==RESCTRL_NPS_NODE.4/8 x86,fs/resctrl: Add the resource for Global Memory Bandwidth Allocation Register
RDT_RESOURCE_GMBAin rdt_resources_all[] with ctrl_scope=RESCTRL_NPS_NODE and schema_fmt=RANGE, add commands to discover feature details.5/8 fs/resctrl: Add the documentation for Global Memory Bandwidth Allocation Add examples in
Documentation/filesystems/resctrl.rst.Patches 6-8 to enable GSMBA in the same shape:
6/8 x86,fs/resctrl: Add support for Global Slow Memory Bandwidth Allocation
7/8 x86,fs/resctrl: Add the resource for Global Slow Memory Bandwidth Allocation Register
RDT_RESOURCE_GSMBAwith ctrl_scope=RESCTRL_NPS_NODE.8/8 fs/resctrl: Add the documentation for Global Slow Memory Bandwidth Allocation Add examples in
Documentation/filesystems/resctrl.rst.Changes since v1
----------------
- Earlier sent RFC(v1) with Global Bandwidth Enforcement (GLBE) and Privilege Level Zero Association (PLZA). This series only handles Global Memory Bandwidth Allocation. Both the features are sent separately.- Documentation
* Fixed grammar in the GMBA / GSMBA sections ofresctrl.rst.
* Added examples to update GMBA and GSMBA inresctrl.rstdocumentation.Quoted 7 lines
- Major changes are releated to
RESCTRL_NPS_NODEscope handling.- Commit messages
* Reworked the changelogs in all the patches.Previous Revisions:
v1 : https://lore.kernel.org/lkml/ [email protected]/What are your expectations from this submission? From what I can tell this ignores
v1 feedback in several ways:
- It introduces two new resources, GMBA and GSMBA, when the previous
discussion agreed that
these are not actually new resources but instead new controls for the existing MBA/SMBA resources.
- It does not mention or attempt to address dependency on new
resource schema descriptions [1]
to support user space in understanding how to interact with the new GMBA/GSMBA controls but instead defers that to a snippet in the documentation that user space needs to parse to know this control operates at multiples of 1GB/s.Apart from ignoring v1 feedback this new version appears to complicate user interface even more since now it is possible for there to be a single control that may operate at different scopes but from what I can tell there is nothing that helps user understand whether, for example, domain "0" means the whole system or a NUMA node?
Quoted 7 lines
Quoted 3 lines
We have discussed several times now how resctrl interface needs to be enhanced to support this and other upcoming features from Intel, RISC-V, Arm MPAM, and NVidia. It is thus unexpected that this submission ignores all the previous discussions.
I think there may be some misunderstanding on this topic.
Yes, we discussed it earlier. It depends on other requirements (region-aware aspects), so I assumed it would be handled by someone with full context and addressed as a separate feature. I didn’t have complete visibility into all the requirements.
Please read https://lore.kernel.org/lkml/06a237bd- [email protected]/ again.
You should have complete visibility into the foundation of this work since one of the primary goals is to address the resctrl interface breakage that came with the initial AMD support for MBA that resctrl has been living with until now.
With this series you completely disregard attempts to support users in understanding how to interact with the schemata file and instead introduce *another* obfuscated control. I will not support this.
Also, no, this does not depend on region-aware work. Needing to support multiple controls for a single resource is independent from region-aware.
Quoted 6 lines
Quoted 3 lines
Since there are so many dependencies on the new schema format support I am prioritizing this and created a PoC that I am currently refining and hope to share soon. We can collaborate on this to ensure that it provides a good foundation for the GMBA and GSMBA support.
That is good to know. Let me know when you are ready.
Quoted 8 lines
Quoted 6 lines
Quoted 1 line
Could you please share which parts of the feature (e.g., Part 1, Part 2, etc.) you are planning to cover in your PoC?
All three parts mentioned in https://lore.kernel.org/lkml/06a237bd- [email protected]/
This does not address all the features discussed, for example it does not support emulated controls, but I hope it is enough of a foundation to build on.
Please share your code when you are ready. I can build GMB and GSMBA on top of your patches. Hopefully, I can reuse some of the code from this series.
I didn’t see your acknowledgment on my previous note, so I wanted to follow up to ensure we’re aligned.
Just to confirm—are you planning to share your PoC?
My understanding is that I would build GMB/GSMBA on top of your patches. Please let me know if that’s correct.
There’s no urgency on the patches at this point; I mainly wanted to get some clarity on the plan.
Thanks,
Babu
Reinette Chatre
maintainerHi Babu,
Quoted 6 lines · Babu
Quoted 5 lines · Babu
Quoted 3 lines · Reinette Chatre
Quoted 3 lines
Quoted 3 lines
Since there are so many dependencies on the new schema format support I am prioritizing this and created a PoC that I am currently refining and hope to share soon. We can collaborate on this to ensure that it provides a good foundation for the GMBA and GSMBA support.
Above is comment from me indicating plans to share the PoC and goal to have it provide a foundation for GMBA and GSMBA.
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 4 lines
That is good to know. Let me know when you are ready.
Could you please share which parts of the feature (e.g., Part 1, Part 2, etc.) you are planning to cover in your PoC?
Quoted 8 lines
Quoted 6 lines
Quoted 4 lines
All three parts mentioned in https://lore.kernel.org/lkml/06a237bd- [email protected]/
This does not address all the features discussed, for example it does not support emulated controls, but I hope it is enough of a foundation to build on.
Please share your code when you are ready. I can build GMB and GSMBA on top of your patches. Hopefully, I can reuse some of the code from this series.
I didn’t see your acknowledgment on my previous note, so I wanted to follow up to ensure we’re aligned.
I did not think a response was necessary since it essentially rephrased my earlier comment and did not contain a question.
Quoted 1 line
Just to confirm—are you planning to share your PoC?
Yes. The fixes needed in existing resctrl code are taking higher priority though.
Quoted 2 lines
My understanding is that I would build GMB/GSMBA on top of your patches. Please let me know if that’s correct.
That is my understanding also.
Quoted 2 lines
There’s no urgency on the patches at this point; I mainly wanted to get some clarity on the plan.
Reinette
Hi Reinette,
Quoted 4 lines · Reinette Chatre
Hi Babu,
Quoted 1 line · Babu
On 5/1/2026 9:38 AM, Moger, Babu wrote:
Quoted 8 lines
Quoted 5 lines
Quoted 5 lines · Reinette Chatre
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Since there are so many dependencies on the new schema format support I am prioritizing this and created a PoC that I am currently refining and hope to share soon. We can collaborate on this to ensure that it provides a good foundation for the GMBA and GSMBA support.
Above is comment from me indicating plans to share the PoC and goal to have it provide a foundation for GMBA and GSMBA.
ok.
Quoted 13 lines
Quoted 13 lines
Quoted 12 lines
Quoted 10 lines
Quoted 5 lines
That is good to know. Let me know when you are ready.
Could you please share which parts of the feature (e.g., Part 1, Part 2, etc.) you are planning to cover in your PoC?
All three parts mentioned in https://lore.kernel.org/lkml/06a237bd- [email protected]/
This does not address all the features discussed, for example it does not support emulated controls, but I hope it is enough of a foundation to build on.
Please share your code when you are ready. I can build GMB and GSMBA on top of your patches. Hopefully, I can reuse some of the code from this series.
Quoted 9 lines
Quoted 1 line
I didn’t see your acknowledgment on my previous note, so I wanted to follow up to ensure we’re aligned.
I did not think a response was necessary since it essentially rephrased my earlier comment and did not contain a question.
Quoted 1 line
Just to confirm—are you planning to share your PoC?
Yes. The fixes needed in existing resctrl code are taking higher priority though.
Ok. Sure.
Quoted 4 lines
Quoted 2 lines
My understanding is that I would build GMB/GSMBA on top of your patches. Please let me know if that’s correct.
That is my understanding also.
Thanks for the confirmation.
-Babu