Skip to contentSkip to conversation

mpam,x86,fs/resctrl: Generic schema description Proof of Concept

RFC Reinette Chatre 9 participants 108 messages May 29 – Aug 7 Page 1 of 2

Subsystems (2)
lore
Share & tools
lore

Reinette Chatre

author maintainer
#0 lore raw

Hi Everybody,

It has been a while since we discussed the resctrl changes required to support hardware that has controls with fine granularity or hardware that has multiple controls per resource. For reference, the most recent email discussion can be found at [1] with a summary of discussions in last year's plumbers slides [2].

I created a PoC that I believe supports what folks have agreed to so far. I hope this can help us to restart the discussion with the goal that resctrl gains support for upcoming hardware that require these features.

Request regarding this PoC
==========================

Please consider this PoC as a "direction check" on the schema description and multiple control discussions held thus far.

Could folks working on enabling new hardware requiring this capability please consider if this is something you can build on and how it should be improved to support these upcoming capabilities?

Opens
=====

While the PoC aims to support what folks agreed on some opens remain:
- I attempted to make some MPAM supporting changes but these are all just compile tested. While MPAM should benefit from the new control properties I did not initialize them on MPAM and did not attempt refactor to separate out
the architecture specific control properties (more on what this means later). I did attempt some MPAM refactoring that duplicates the MPAM domain to the control domain and monitoring domain lists in support of there being multiple controls each with its own list of control domains but it is definitely not good design.
- No support for emulated controls (yet). The PoC is quite large already
but I think it can be used as a base for emulated controls for which the software controller could be a potential first customer. In this PoC mounting with software controller will still display the original controller's properties.
- One open that needs to be addressed as part of support for emulated controls is how best to display emulation relationship via resctrl hierarchy.
- No support for "read-modify-write" usage of schemata file. This is where we discussed (without agreement) on possibly introducing the "#" prefix to schemata file entries. This PoC does not support this prefix and the current assumption/expectation is that when user space changes a configuration only the new control values are written to schemata file. I thus do not have a plan to support this so please share opinions in this regard if you have some.
- Controls are independent for now. This means that, for example, if a resource supports a "MIN" and "MAX" control then this implementation would allow user to set the "maximum" control values to be less than the "minimum" control values.
- PoC supports the "bitmap" control but does not (yet) expose properties of a bitmap control to the new info/<resource>/resource_schemata directory.

Accessing PoC
=============

Please consider the PoC as a rough draft. It has only been compile tested for Arm and known to be incomplete in Arm support. To help with experimenting I only fully adapted the Intel MBA resource to demo two dummy additional MBA controls. All architectures should immediately benefit from the new schema descriptions and new info/MB/resource_schemata hierarchy.

I considered the patches self too many for email. Instead, the PoC can be found at:

git://git.kernel.org/pub/scm/linux/kernel/git/reinette/linux.git branch resctrl/controls_rfc_v1

The work is based on v7.1-rc2 that also includes the following series (two of which has since been queued) included:

"selftests/resctrl: Fixes and improvements focused on Intel platforms"
https://lore.kernel.org/lkml/[email protected]/

"x86,fs/resctrl: Improve resctrl quality and consistency"
https://lore.kernel.org/lkml/[email protected]/

"x86,fs/resctrl: Pave the way for MPAM counter assignment"
https://lore.kernel.org/lkml/[email protected]/

Primary resctrl fs data structure changes
=========================================

Introduces a control represented by struct resctrl_ctrl that looks as below. To make the changes easier to follow I kept some of the original names to help communicate where familiar data structures land.

What to notice about a control is that it has some common properties required from all controls (scope, type, etc.) and then depending on the type of control (RESCTRL_CTRL_BITMAP or RESCTRL_CTRL_SCALAR) there are type specific properties.

/**
* struct resctrl_ctrl - A resource control
* @entry: List entry of rdt_resource::controls
* @scope: Scope of the resource that this control allocates
* @domains: RCU list of all control domains
* @type: The control type that determines the properties of the control,
* format string for displaying control values to user space, and
* parser of control values provided by user space.
* @name: Name of the control. Appended to final resource name
* (rdt_resource_final::name) to create final schema entry.
* Specifically, "rdt_resource_final::name"_"resctrl_ctrl::name".
* For example, with resource name "MB" and control name "MAX" the
* schema entry will be "MB_MAX".
* @cache: Cache allocation control properties.
* @membw: Bandwidth control properties.
*/

struct resctrl_ctrl {
	struct list_head	entry;
	enum resctrl_scope	scope;
	struct list_head	domains;
	enum resctrl_ctrl_type	type;
	enum resctrl_ctrl_name	name;
	union {
		struct resctrl_cache	cache;
		struct resctrl_membw	membw;
	};
};

Two members summarize how this new structure fits into the rest of resctrl:
a) resctrl_ctrl::entry
Since a resource can support multiple controls there is a new list in struct rdt_resource named "controls" that contains the list of all controls supported by the resource.
b) resctrl_ctrl::domains
Instead of the list of control domains belonging to a resource they now belong to the control self. By doing so resctrl can support resource controls at different scope for the same resource. This is intended to support some upcoming MPAM and RISC-V usages.

Example architectural data structure changes
============================================

An architecture can use the new control by following a similar pattern to resource and domain use by architectures. Consider the following for x86 where a new architecture specific struct resctrl_hw_ctrl includes struct resctrl_ctrl and any architecture private data needed to support the control:

/*
* struct resctrl_hw_ctrl - Arch private properties of a resource control
* @r_ctrl: Control properties exposed to resctrl file system
* @msr_base: Base MSR address where control values should be programmed
* @msr_update: Function pointer to update control values
*/

struct resctrl_hw_ctrl {
	struct resctrl_ctrl	r_ctrl;
	unsigned int		msr_base;
	void			(*msr_update)(struct msr_param *m);
};

Structure of patch series
=========================

As a PoC the series is not perfectly structured but to help navigate this work on a high level the changes can be categorized as follows:

Patch 1 to 11:
With a vision of what a "control" is, remove unused/unnecessary members, make clear what is a *resource* property vs a *control* property, do some renaming to help with the PoC.

Patch 12:
Introduce struct resctrl_ctrl and re-arrange existing struct rdt_resource members to form part of new rdt_resource::ctrl

Patch 13 to 44:
A lot of wrangling to introduce struct resctrl_ctrl to all code that needs to work with a control and/or domain without assuming that the control is the one and only control embedded in the resource it belongs to. Essentially, a lot of changes passing the control around in addition to the resource/domain.

Patch 45:
Switch the single struct resctrl_ctrl member of struct rdt_resource to be a list of struct resctrl_ctrl.

Patch 47 to 49:
Introduce new info/<resource>/resource_schemata hierarchy to first only consist of properties already known to resctrl fs.

Patch 50 to 52:
Introduce the new control properties per [1], initialize them for x86, and expose them via info/<resource>/resource_schemata

Patch 53:
Let the new struct resctrl_hw_ctrl contain architecture's control properties.

Patch 54:
Teach resctrl fs about "MIN" and "MAX" controls.

Patch 55:
Sample of "MIN" and "MAX" memory bandwidth controls for x86.

Example interactions
====================

This series can be used on an x86 system where it will show two new dummy controls where it is possible to interact with the new controls.
For example:

# cat schemata 
        MB_MAX:0=100;1=100
        MB_MIN:0=100;1=100
        MB:0=100;1=100
        L3:0=fff;1=fff
# echo 'MB_MIN:0=50' > schemata 
# cat schemata 
        MB_MAX:0=100;1=100
        MB_MIN:0=50;1=100
        MB:0=100;1=100
        L3:0=fff;1=fff

Writing to the dummy control will call a dummy callback that just prints to the kernel log:
"resctrl: Updata temporary MIN control on domain 0 with user value 50"

Example output of info/MB/:
/sys/fs/resctrl/info/MB/thread_throttle_mode:max /sys/fs/resctrl/info/MB/num_closids:15 /sys/fs/resctrl/info/MB/delay_linear:1 /sys/fs/resctrl/info/MB/min_bandwidth:10 /sys/fs/resctrl/info/MB/resource_schemata/MB/resolution:100 /sys/fs/resctrl/info/MB/resource_schemata/MB/tolerance:5 /sys/fs/resctrl/info/MB/resource_schemata/MB/type:scalar /sys/fs/resctrl/info/MB/resource_schemata/MB/min:10 /sys/fs/resctrl/info/MB/resource_schemata/MB/scale:1 /sys/fs/resctrl/info/MB/resource_schemata/MB/scope:L3 /sys/fs/resctrl/info/MB/resource_schemata/MB/unit:all /sys/fs/resctrl/info/MB/resource_schemata/MB/max:100 /sys/fs/resctrl/info/MB/resource_schemata/MB_MIN/resolution:100 /sys/fs/resctrl/info/MB/resource_schemata/MB_MIN/tolerance:5 /sys/fs/resctrl/info/MB/resource_schemata/MB_MIN/type:scalar /sys/fs/resctrl/info/MB/resource_schemata/MB_MIN/min:10 /sys/fs/resctrl/info/MB/resource_schemata/MB_MIN/scale:1 /sys/fs/resctrl/info/MB/resource_schemata/MB_MIN/scope:L3 /sys/fs/resctrl/info/MB/resource_schemata/MB_MIN/unit:all /sys/fs/resctrl/info/MB/resource_schemata/MB_MIN/max:100 /sys/fs/resctrl/info/MB/resource_schemata/MB_MAX/resolution:100 /sys/fs/resctrl/info/MB/resource_schemata/MB_MAX/tolerance:5 /sys/fs/resctrl/info/MB/resource_schemata/MB_MAX/type:scalar /sys/fs/resctrl/info/MB/resource_schemata/MB_MAX/min:10 /sys/fs/resctrl/info/MB/resource_schemata/MB_MAX/scale:1 /sys/fs/resctrl/info/MB/resource_schemata/MB_MAX/scope:L3 /sys/fs/resctrl/info/MB/resource_schemata/MB_MAX/unit:all /sys/fs/resctrl/info/MB/resource_schemata/MB_MAX/max:100 /sys/fs/resctrl/info/MB/bandwidth_gran:10

Any feedback is appreciated.

Reinette

[1] https://lore.kernel.org/lkml/aPtfMFfLV1l%[email protected]/
[2] https://lpc.events/event/19/contributions/2093/attachments/1958/4172/resctrl%20Microconference%20LPC%202025%20Tokyo.pdf

reviewer
#1 lore raw

Hi Reinette,

For some reason, I couldn’t find your patch on lore.kernel.org:
https://lore.kernel.org/lkml/?q=Reinette+Chatre

I eventually located it here:
https://sashiko.dev/#/message/aab804b9-e8b5-40ad-a85b-af7033391243%40intel.com

Thanks for sharing the patches. I’m still reviewing them.

I was able to build and boot the kernel and can see the MIN and MAX controls. After moving your test code to __rdt_get_mem_config_amd().

Quoted 116 lines · Reinette Chatre

Hi Everybody,

It has been a while since we discussed the resctrl changes required to support hardware that has controls with fine granularity or hardware that has multiple controls per resource. For reference, the most recent email discussion can be found at [1] with a summary of discussions in last year's plumbers slides [2].

I created a PoC that I believe supports what folks have agreed to so far. I hope this can help us to restart the discussion with the goal that resctrl gains support for upcoming hardware that require these features.

Request regarding this PoC
==========================

Please consider this PoC as a "direction check" on the schema description and multiple control discussions held thus far.

Could folks working on enabling new hardware requiring this capability please consider if this is something you can build on and how it should be improved to support these upcoming capabilities?

Opens
=====

While the PoC aims to support what folks agreed on some opens remain:
- I attempted to make some MPAM supporting changes but these are all just compile tested. While MPAM should benefit from the new control properties I did not initialize them on MPAM and did not attempt refactor to separate out
the architecture specific control properties (more on what this means later). I did attempt some MPAM refactoring that duplicates the MPAM domain to the control domain and monitoring domain lists in support of there being multiple controls each with its own list of control domains but it is definitely not good design.
- No support for emulated controls (yet). The PoC is quite large already
but I think it can be used as a base for emulated controls for which the software controller could be a potential first customer. In this PoC mounting with software controller will still display the original controller's properties.
- One open that needs to be addressed as part of support for emulated controls is how best to display emulation relationship via resctrl hierarchy.
- No support for "read-modify-write" usage of schemata file. This is where we discussed (without agreement) on possibly introducing the "#" prefix to schemata file entries. This PoC does not support this prefix and the current assumption/expectation is that when user space changes a configuration only the new control values are written to schemata file. I thus do not have a plan to support this so please share opinions in this regard if you have some.
- Controls are independent for now. This means that, for example, if a resource supports a "MIN" and "MAX" control then this implementation would allow user to set the "maximum" control values to be less than the "minimum" control values.
- PoC supports the "bitmap" control but does not (yet) expose properties of a bitmap control to the new info/<resource>/resource_schemata directory.

Accessing PoC
=============

Please consider the PoC as a rough draft. It has only been compile tested for Arm and known to be incomplete in Arm support. To help with experimenting I only fully adapted the Intel MBA resource to demo two dummy additional MBA controls. All architectures should immediately benefit from the new schema descriptions and new info/MB/resource_schemata hierarchy.

I considered the patches self too many for email. Instead, the PoC can be found at:

git://git.kernel.org/pub/scm/linux/kernel/git/reinette/linux.git branch resctrl/controls_rfc_v1

The work is based on v7.1-rc2 that also includes the following series (two of which has since been queued) included:

"selftests/resctrl: Fixes and improvements focused on Intel platforms"
https://lore.kernel.org/lkml/[email protected]/

"x86,fs/resctrl: Improve resctrl quality and consistency"
https://lore.kernel.org/lkml/[email protected]/

"x86,fs/resctrl: Pave the way for MPAM counter assignment"
https://lore.kernel.org/lkml/[email protected]/

Primary resctrl fs data structure changes
=========================================

Introduces a control represented by struct resctrl_ctrl that looks as below. To make the changes easier to follow I kept some of the original names to help communicate where familiar data structures land.

What to notice about a control is that it has some common properties required from all controls (scope, type, etc.) and then depending on the type of control (RESCTRL_CTRL_BITMAP or RESCTRL_CTRL_SCALAR) there are type specific properties.

/**
* struct resctrl_ctrl - A resource control
* @entry: List entry of rdt_resource::controls
* @scope: Scope of the resource that this control allocates
* @domains: RCU list of all control domains
* @type: The control type that determines the properties of the control,
* format string for displaying control values to user space, and
* parser of control values provided by user space.
* @name: Name of the control. Appended to final resource name
* (rdt_resource_final::name) to create final schema entry.
* Specifically, "rdt_resource_final::name"_"resctrl_ctrl::name".
* For example, with resource name "MB" and control name "MAX" the
* schema entry will be "MB_MAX".
* @cache: Cache allocation control properties.
* @membw: Bandwidth control properties.
*/

struct resctrl_ctrl {
	struct list_head	entry;
	enum resctrl_scope	scope;
	struct list_head	domains;
	enum resctrl_ctrl_type	type;
	enum resctrl_ctrl_name	name;
	union {
		struct resctrl_cache	cache;
		struct resctrl_membw	membw;
	};
};
Quoted 11 lines

Two members summarize how this new structure fits into the rest of resctrl:
a) resctrl_ctrl::entry
Since a resource can support multiple controls there is a new list in struct rdt_resource named "controls" that contains the list of all controls supported by the resource.
b) resctrl_ctrl::domains
Instead of the list of control domains belonging to a resource they now belong to the control self. By doing so resctrl can support resource controls at different scope for the same resource. This is intended to support some upcoming MPAM and RISC-V usages.

I like the idea of supporting multiple controls for each resource.

With these patches, now we have one list containing all the controls.

However, in case of RDT_RESOURCE_L3, we have two lists "mon_domains" and "controls". mon_domains list deals with monitoring and control deals with control parts(multiple).

Have you thought about making the list("control") generic so that the control can be monitoring also. It will just one list containing multiple controls or monitor.

Quoted 54 lines

Example architectural data structure changes
============================================

An architecture can use the new control by following a similar pattern to resource and domain use by architectures. Consider the following for x86 where a new architecture specific struct resctrl_hw_ctrl includes struct resctrl_ctrl and any architecture private data needed to support the control:

/*
* struct resctrl_hw_ctrl - Arch private properties of a resource control
* @r_ctrl: Control properties exposed to resctrl file system
* @msr_base: Base MSR address where control values should be programmed
* @msr_update: Function pointer to update control values
*/

struct resctrl_hw_ctrl {
	struct resctrl_ctrl	r_ctrl;
	unsigned int		msr_base;
	void			(*msr_update)(struct msr_param *m);
};

Structure of patch series
=========================

As a PoC the series is not perfectly structured but to help navigate this work on a high level the changes can be categorized as follows:

Patch 1 to 11:
With a vision of what a "control" is, remove unused/unnecessary members, make clear what is a *resource* property vs a *control* property, do some renaming to help with the PoC.

Patch 12:
Introduce struct resctrl_ctrl and re-arrange existing struct rdt_resource members to form part of new rdt_resource::ctrl

Patch 13 to 44:
A lot of wrangling to introduce struct resctrl_ctrl to all code that needs to work with a control and/or domain without assuming that the control is the one and only control embedded in the resource it belongs to. Essentially, a lot of changes passing the control around in addition to the resource/domain.

Patch 45:
Switch the single struct resctrl_ctrl member of struct rdt_resource to be a list of struct resctrl_ctrl.

Patch 47 to 49:
Introduce new info/<resource>/resource_schemata hierarchy to first only consist of properties already known to resctrl fs.

Patch 50 to 52:
Introduce the new control properties per [1], initialize them for x86, and expose them via info/<resource>/resource_schemata

Quoted 8 lines

Patch 53:
Let the new struct resctrl_hw_ctrl contain architecture's control properties.

Patch 54:
Teach resctrl fs about "MIN" and "MAX" controls.

Patch 55:
Sample of "MIN" and "MAX" memory bandwidth controls for x86.

My assumption is that the MIN and MAX controls here are just examples, correct?

You only mentioned patch 55 as "NOT_FOR_INCLUSION". I assume patch 54 should also be marked as "NOT_FOR_INCLUSION"?

Thanks
Babu

Quoted 63 lines

Example interactions
====================

This series can be used on an x86 system where it will show two new dummy controls where it is possible to interact with the new controls.
For example:

# cat schemata
         MB_MAX:0=100;1=100
         MB_MIN:0=100;1=100
         MB:0=100;1=100
         L3:0=fff;1=fff
# echo 'MB_MIN:0=50' > schemata
# cat schemata
         MB_MAX:0=100;1=100
         MB_MIN:0=50;1=100
         MB:0=100;1=100
         L3:0=fff;1=fff

Writing to the dummy control will call a dummy callback that just prints to the kernel log:
"resctrl: Updata temporary MIN control on domain 0 with user value 50"

Example output of info/MB/:
/sys/fs/resctrl/info/MB/thread_throttle_mode:max /sys/fs/resctrl/info/MB/num_closids:15 /sys/fs/resctrl/info/MB/delay_linear:1 /sys/fs/resctrl/info/MB/min_bandwidth:10 /sys/fs/resctrl/info/MB/resource_schemata/MB/resolution:100 /sys/fs/resctrl/info/MB/resource_schemata/MB/tolerance:5 /sys/fs/resctrl/info/MB/resource_schemata/MB/type:scalar /sys/fs/resctrl/info/MB/resource_schemata/MB/min:10 /sys/fs/resctrl/info/MB/resource_schemata/MB/scale:1 /sys/fs/resctrl/info/MB/resource_schemata/MB/scope:L3 /sys/fs/resctrl/info/MB/resource_schemata/MB/unit:all /sys/fs/resctrl/info/MB/resource_schemata/MB/max:100 /sys/fs/resctrl/info/MB/resource_schemata/MB_MIN/resolution:100 /sys/fs/resctrl/info/MB/resource_schemata/MB_MIN/tolerance:5 /sys/fs/resctrl/info/MB/resource_schemata/MB_MIN/type:scalar /sys/fs/resctrl/info/MB/resource_schemata/MB_MIN/min:10 /sys/fs/resctrl/info/MB/resource_schemata/MB_MIN/scale:1 /sys/fs/resctrl/info/MB/resource_schemata/MB_MIN/scope:L3 /sys/fs/resctrl/info/MB/resource_schemata/MB_MIN/unit:all /sys/fs/resctrl/info/MB/resource_schemata/MB_MIN/max:100 /sys/fs/resctrl/info/MB/resource_schemata/MB_MAX/resolution:100 /sys/fs/resctrl/info/MB/resource_schemata/MB_MAX/tolerance:5 /sys/fs/resctrl/info/MB/resource_schemata/MB_MAX/type:scalar /sys/fs/resctrl/info/MB/resource_schemata/MB_MAX/min:10 /sys/fs/resctrl/info/MB/resource_schemata/MB_MAX/scale:1 /sys/fs/resctrl/info/MB/resource_schemata/MB_MAX/scope:L3 /sys/fs/resctrl/info/MB/resource_schemata/MB_MAX/unit:all /sys/fs/resctrl/info/MB/resource_schemata/MB_MAX/max:100 /sys/fs/resctrl/info/MB/bandwidth_gran:10

Any feedback is appreciated.

Reinette

[1] https://lore.kernel.org/lkml/aPtfMFfLV1l%[email protected]/
[2] https://lpc.events/event/19/contributions/2093/attachments/1958/4172/resctrl%20Microconference%20LPC%202025%20Tokyo.pdf

author maintainer
#2 lore raw

Hi Babu,

Quoted 4 lines · Babu Moger

Hi Reinette,

For some reason, I couldn’t find your patch on lore.kernel.org:
https://lore.kernel.org/lkml/?q=Reinette+Chatre

How about:
https://lore.kernel.org/lkml/[email protected]/

Quoted 7 lines

I eventually located it here:
https://sashiko.dev/#/message/aab804b9-e8b5-40ad-a85b-af7033391243%40intel.com

Thanks for sharing the patches. I’m still reviewing them.

I was able to build and boot the kernel and can see the MIN and MAX controls. After moving your test code to __rdt_get_mem_config_amd().

Thank you very much for trying it out.

Quoted 1 line · Reinette Chatre

...

Quoted 51 lines
Quoted 39 lines

Primary resctrl fs data structure changes
=========================================

Introduces a control represented by struct resctrl_ctrl that looks as below. To make the changes easier to follow I kept some of the original names to help communicate where familiar data structures land.

What to notice about a control is that it has some common properties required from all controls (scope, type, etc.) and then depending on the type of control (RESCTRL_CTRL_BITMAP or RESCTRL_CTRL_SCALAR) there are type specific properties.

/**
  * struct resctrl_ctrl - A resource control
  * @entry:    List entry of rdt_resource::controls
  * @scope:    Scope of the resource that this control allocates
  * @domains:    RCU list of all control domains
  * @type:    The control type that determines the properties of the control, *        format string for displaying control values to user space, and *        parser of control values provided by user space.
  * @name:    Name of the control. Appended to final resource name *        (rdt_resource_final::name) to create final schema entry. *        Specifically, "rdt_resource_final::name"_"resctrl_ctrl::name". *        For example, with resource name "MB" and control name "MAX" the *        schema entry will be "MB_MAX".
  * @cache:    Cache allocation control properties.
  * @membw:    Bandwidth control properties.
  */
struct resctrl_ctrl {
    struct list_head    entry;
    enum resctrl_scope    scope;
    struct list_head    domains;
    enum resctrl_ctrl_type    type;
    enum resctrl_ctrl_name    name;
    union {
        struct resctrl_cache    cache;
        struct resctrl_membw    membw;
    };
};

Quoted 11 lines

Two members summarize how this new structure fits into the rest of resctrl:
a) resctrl_ctrl::entry
    Since a resource can support multiple controls there is a new list in struct rdt_resource named "controls" that contains the list of all controls supported by the resource.
b) resctrl_ctrl::domains
    Instead of the list of control domains belonging to a resource they now belong to the control self. By doing so resctrl can support resource controls at different scope for the same resource. This is intended to support some upcoming MPAM and RISC-V usages.

Quoted 7 lines

I like the idea of supporting multiple controls for each resource.

With these patches, now we have one list containing all the controls.

However, in case of RDT_RESOURCE_L3, we have two lists "mon_domains" and "controls". mon_domains list deals with monitoring and control deals with control parts(multiple).

Have you thought about making the list("control") generic so that the control can be monitoring also. It will just one list containing multiple controls or monitor.

The control list adds an additional layer of abstraction just for control management, independent from monitoring. The mon_domains list is unchanged while each control now has its own ctrl_domains list.

Here is an attempt to visualize how a resource with two monitoring domains, and two controls, each with two control domains end up being managed:

+-------------------------+
| struct rdt_resource     |
+-------------------------+
| ...                     |
| controls (list_head)    |---------+
| mon_domains (list_head) |---+     |
| ...                     |   |     |
+-------------------------+   |     |
                              |     |
  +---------------------------+     |
  |                                 |
  v                                 v
+-----------------------------+   +-------------------------+
| struct rdt_l3_mon_domain #1 |   | struct resctrl_ctrl #A  |
+-----------------------------+   +-------------------------+
| rdt_domain_hdr              |-+ | entry (list_head)       |       +----------------------------+
| ...                         | | | domains (list_head)     |------>| struct rdt_ctrl_domain #A1 |
+-----------------------------+ | | ...                     |       +----------------------------+
                                | +-------------------------+       | rdt_domain_hdr             |---+
  +-----------------------------+   |                               | ...                        |   |
  | (next)                          | (next)                        +----------------------------+   |
  v                                 v                                                                |
+-----------------------------+   +-------------------------+       +----------------------------+<--+
| struct rdt_l3_mon_domain #2 |   | struct resctrl_ctrl #B  |       | struct rdt_ctrl_domain #A2 |
+-----------------------------+   +-------------------------+       +----------------------------+
| rdt_domain_hdr              |   | entry (list_head)       |       | rdt_domain_hdr             |
| ...                         |   | domains (list_head)     |----n  | ...                        |
+-----------------------------+   | ...                     |    |  +----------------------------+
                                  +-------------------------+    |
                                                                 |
                                    +----------------------------+
                                    |
                                    v
                                  +----------------------------+
                                  | struct rdt_ctrl_domain #B1 |
                                  +----------------------------+
                                  | rdt_domain_hdr             |---+
                                  | ...                        |   |
                                  +----------------------------+   |
                                                                   |
                                  +----------------------------+<--+
                                  | struct rdt_ctrl_domain #B2 |
                                  +----------------------------+
                                  | rdt_domain_hdr             |
                                  | ...                        |
                                  +----------------------------+

resctrl used to manage single domains that are capable of both monitoring and control but this was split to support scenario where monitoring and control of a resource are done at different scope. See
cd84f72b6a5c ("x86/resctrl: Prepare for different scope for control/monitor operations")

This feature further expands the difference between monitoring and control since now there can be multiple instances of a control domain (one per control) associated with a resource while monitoring still just supports one monitoring domain per resource.

I thus cannot see how this can be accomplished with a single list. Could you sketch out what you have in mind?

...

Quoted 10 lines
Quoted 6 lines

Patch 54:
Teach resctrl fs about "MIN" and "MAX" controls.

Patch 55:
Sample of "MIN" and "MAX" memory bandwidth controls for x86.

My assumption is that the MIN and MAX controls here are just examples, correct?

You only mentioned patch 55 as "NOT_FOR_INCLUSION". I assume patch 54 should also be marked as "NOT_FOR_INCLUSION"?

While patch 55 is the first and only "user" of patch 54 I believe that patch 54 (or some variant of it) will stay since we already know that both MPAM and Intel need to support min and max controls.

Reinette

#3 lore raw

Hi Reinette,

Quoted 10 lines · Reinette Chatre

Hi Babu,

Quoted 4 lines · Babu Moger

Hi Reinette,

For some reason, I couldn’t find your patch on lore.kernel.org:
https://lore.kernel.org/lkml/?q=Reinette+Chatre

How about:
https://lore.kernel.org/lkml/[email protected]/

Yes. It works. But, I used to find patches using the names. It did not work this time.

Quoted 133 lines
Quoted 8 lines

I eventually located it here:
https://sashiko.dev/#/message/aab804b9-e8b5-40ad-a85b-af7033391243%40intel.com

Thanks for sharing the patches. I’m still reviewing them.

I was able to build and boot the kernel and can see the MIN and MAX controls. After moving your test code to __rdt_get_mem_config_amd().

Thank you very much for trying it out.

Quoted 1 line · Reinette Chatre

...

Quoted 51 lines
Quoted 39 lines

Primary resctrl fs data structure changes
=========================================

Introduces a control represented by struct resctrl_ctrl that looks as below. To make the changes easier to follow I kept some of the original names to help communicate where familiar data structures land.

What to notice about a control is that it has some common properties required from all controls (scope, type, etc.) and then depending on the type of control (RESCTRL_CTRL_BITMAP or RESCTRL_CTRL_SCALAR) there are type specific properties.

/**
  * struct resctrl_ctrl - A resource control
  * @entry:    List entry of rdt_resource::controls
  * @scope:    Scope of the resource that this control allocates
  * @domains:    RCU list of all control domains
  * @type:    The control type that determines the properties of the control, *        format string for displaying control values to user space, and *        parser of control values provided by user space.
  * @name:    Name of the control. Appended to final resource name *        (rdt_resource_final::name) to create final schema entry. *        Specifically, "rdt_resource_final::name"_"resctrl_ctrl::name". *        For example, with resource name "MB" and control name "MAX" the *        schema entry will be "MB_MAX".
  * @cache:    Cache allocation control properties.
  * @membw:    Bandwidth control properties.
  */
struct resctrl_ctrl {
    struct list_head    entry;
    enum resctrl_scope    scope;
    struct list_head    domains;
    enum resctrl_ctrl_type    type;
    enum resctrl_ctrl_name    name;
    union {
        struct resctrl_cache    cache;
        struct resctrl_membw    membw;
    };
};

Quoted 11 lines

Two members summarize how this new structure fits into the rest of resctrl:
a) resctrl_ctrl::entry
    Since a resource can support multiple controls there is a new list in struct rdt_resource named "controls" that contains the list of all controls supported by the resource.
b) resctrl_ctrl::domains
    Instead of the list of control domains belonging to a resource they now belong to the control self. By doing so resctrl can support resource controls at different scope for the same resource. This is intended to support some upcoming MPAM and RISC-V usages.

Quoted 7 lines

I like the idea of supporting multiple controls for each resource.

With these patches, now we have one list containing all the controls.

However, in case of RDT_RESOURCE_L3, we have two lists "mon_domains" and "controls". mon_domains list deals with monitoring and control deals with control parts(multiple).

Have you thought about making the list("control") generic so that the control can be monitoring also. It will just one list containing multiple controls or monitor.

The control list adds an additional layer of abstraction just for control management, independent from monitoring. The mon_domains list is unchanged while each control now has its own ctrl_domains list.

Here is an attempt to visualize how a resource with two monitoring domains, and two controls, each with two control domains end up being managed:

+-------------------------+
| struct rdt_resource     |
+-------------------------+
| ...                     |
| controls (list_head)    |---------+
| mon_domains (list_head) |---+     |
| ...                     |   |     |
+-------------------------+   |     |
                               |     |
   +---------------------------+     |
   |                                 |
   v                                 v
+-----------------------------+   +-------------------------+
| struct rdt_l3_mon_domain #1 |   | struct resctrl_ctrl #A  |
+-----------------------------+   +-------------------------+
| rdt_domain_hdr              |-+ | entry (list_head)       |       +----------------------------+
| ...                         | | | domains (list_head)     |------>| struct rdt_ctrl_domain #A1 |
+-----------------------------+ | | ...                     |       +----------------------------+
                                 | +-------------------------+       | rdt_domain_hdr             |---+
   +-----------------------------+   |                               | ...                        |   |
   | (next)                          | (next)                        +----------------------------+   |
   v                                 v                                                                |
+-----------------------------+   +-------------------------+       +----------------------------+<--+
| struct rdt_l3_mon_domain #2 |   | struct resctrl_ctrl #B  |       | struct rdt_ctrl_domain #A2 |
+-----------------------------+   +-------------------------+       +----------------------------+
| rdt_domain_hdr              |   | entry (list_head)       |       | rdt_domain_hdr             |
| ...                         |   | domains (list_head)     |----n  | ...                        |
+-----------------------------+   | ...                     |    |  +----------------------------+
                                   +-------------------------+    |
                                                                  |
                                     +----------------------------+
                                     |
                                     v
                                   +----------------------------+
                                   | struct rdt_ctrl_domain #B1 |
                                   +----------------------------+
                                   | rdt_domain_hdr             |---+
                                   | ...                        |   |
                                   +----------------------------+   |
                                                                    |
                                   +----------------------------+<--+
                                   | struct rdt_ctrl_domain #B2 |
                                   +----------------------------+
                                   | rdt_domain_hdr             |
                                   | ...                        |
                                   +----------------------------+

resctrl used to manage single domains that are capable of both monitoring and control but this was split to support scenario where monitoring and control of a resource are done at different scope. See
cd84f72b6a5c ("x86/resctrl: Prepare for different scope for control/monitor operations")

Quoted 7 lines

This feature further expands the difference between monitoring and control since now there can be multiple instances of a control domain (one per control) associated with a resource while monitoring still just supports one monitoring domain per resource.

I thus cannot see how this can be accomplished with a single list. Could you sketch out what you have in mind?

Thanks for the sketch. It makes it much more clear.

mon_domains directly maintains a list of monitoring domains, whereas controls maintains a list of control objects, each of which owns its own domain list.

I was thinking, one possibility would be to make mon_domains follow a structure similar to controls, allowing both to be represented through a common top-level list.

However, the resulting abstraction may become fairly large and complex, with limited overlap between the monitoring and control data structures.

Quoted 9 lines

...

Quoted 7 lines
Quoted 6 lines

Patch 54:
Teach resctrl fs about "MIN" and "MAX" controls.

Patch 55:
Sample of "MIN" and "MAX" memory bandwidth controls for x86.

Quoted 6 lines
Quoted 3 lines

My assumption is that the MIN and MAX controls here are just examples, correct?

You only mentioned patch 55 as "NOT_FOR_INCLUSION". I assume patch 54 should also be marked as "NOT_FOR_INCLUSION"?

While patch 55 is the first and only "user" of patch 54 I believe that patch 54 (or some variant of it) will stay since we already know that both MPAM and Intel need to support min and max controls.

That is good to know.

Thanks
Babu

author maintainer
#4 lore raw

Hi Babu,

Quoted 8 lines · Babu

Hi Reinette,

Quoted 5 lines · Reinette Chatre

Hi Babu,

Quoted 2 lines · Babu Moger

Hi Reinette,

Quoted 7 lines
Quoted 5 lines
Quoted 2 lines

For some reason, I couldn’t find your patch on lore.kernel.org:
https://lore.kernel.org/lkml/?q=Reinette+Chatre

How about:
https://lore.kernel.org/lkml/[email protected]/

Yes. It works. But, I used to find patches using the names. It did not work this time.

Interesting. When I run your lore query the first email from this thread is among the results.

Quoted 142 lines
Quoted 134 lines
Quoted 9 lines

I eventually located it here:
https://sashiko.dev/#/message/aab804b9-e8b5-40ad-a85b-af7033391243%40intel.com

Thanks for sharing the patches. I’m still reviewing them.

I was able to build and boot the kernel and can see the MIN and MAX controls. After moving your test code to __rdt_get_mem_config_amd().

Thank you very much for trying it out.

Quoted 1 line · Reinette Chatre

...

Quoted 51 lines
Quoted 39 lines

Primary resctrl fs data structure changes
=========================================

Introduces a control represented by struct resctrl_ctrl that looks as below. To make the changes easier to follow I kept some of the original names to help communicate where familiar data structures land.

What to notice about a control is that it has some common properties required from all controls (scope, type, etc.) and then depending on the type of control (RESCTRL_CTRL_BITMAP or RESCTRL_CTRL_SCALAR) there are type specific properties.

/**
   * struct resctrl_ctrl - A resource control
   * @entry:    List entry of rdt_resource::controls
   * @scope:    Scope of the resource that this control allocates
   * @domains:    RCU list of all control domains
   * @type:    The control type that determines the properties of the control, *        format string for displaying control values to user space, and *        parser of control values provided by user space.
   * @name:    Name of the control. Appended to final resource name *        (rdt_resource_final::name) to create final schema entry. *        Specifically, "rdt_resource_final::name"_"resctrl_ctrl::name". *        For example, with resource name "MB" and control name "MAX" the *        schema entry will be "MB_MAX".
   * @cache:    Cache allocation control properties.
   * @membw:    Bandwidth control properties.
   */
struct resctrl_ctrl {
     struct list_head    entry;
     enum resctrl_scope    scope;
     struct list_head    domains;
     enum resctrl_ctrl_type    type;
     enum resctrl_ctrl_name    name;
     union {
         struct resctrl_cache    cache;
         struct resctrl_membw    membw;
     };
};

Quoted 11 lines

Two members summarize how this new structure fits into the rest of resctrl:
a) resctrl_ctrl::entry
     Since a resource can support multiple controls there is a new list in struct rdt_resource named "controls" that contains the list of all controls supported by the resource.
b) resctrl_ctrl::domains
     Instead of the list of control domains belonging to a resource they now belong to the control self. By doing so resctrl can support resource controls at different scope for the same resource. This is intended to support some upcoming MPAM and RISC-V usages.

Quoted 7 lines

I like the idea of supporting multiple controls for each resource.

With these patches, now we have one list containing all the controls.

However, in case of RDT_RESOURCE_L3, we have two lists "mon_domains" and "controls". mon_domains list deals with monitoring and control deals with control parts(multiple).

Have you thought about making the list("control") generic so that the control can be monitoring also. It will just one list containing multiple controls or monitor.

The control list adds an additional layer of abstraction just for control management, independent from monitoring. The mon_domains list is unchanged while each control now has its own ctrl_domains list.

Here is an attempt to visualize how a resource with two monitoring domains, and two controls, each with two control domains end up being managed:

+-------------------------+
| struct rdt_resource     |
+-------------------------+
| ...                     |
| controls (list_head)    |---------+
| mon_domains (list_head) |---+     |
| ...                     |   |     |
+-------------------------+   |     |
                               |     |
   +---------------------------+     |
   |                                 |
   v                                 v
+-----------------------------+   +-------------------------+
| struct rdt_l3_mon_domain #1 |   | struct resctrl_ctrl #A  |
+-----------------------------+   +-------------------------+
| rdt_domain_hdr              |-+ | entry (list_head)       |       +----------------------------+
| ...                         | | | domains (list_head)     |------>| struct rdt_ctrl_domain #A1 |
+-----------------------------+ | | ...                     |       +----------------------------+
                                 | +-------------------------+       | rdt_domain_hdr             |---+
   +-----------------------------+   |                               | ...                        |   |
   | (next)                          | (next)                        +----------------------------+   |
   v                                 v                                                                |
+-----------------------------+   +-------------------------+       +----------------------------+<--+
| struct rdt_l3_mon_domain #2 |   | struct resctrl_ctrl #B  |       | struct rdt_ctrl_domain #A2 |
+-----------------------------+   +-------------------------+       +----------------------------+
| rdt_domain_hdr              |   | entry (list_head)       |       | rdt_domain_hdr             |
| ...                         |   | domains (list_head)     |----n  | ...                        |
+-----------------------------+   | ...                     |    |  +----------------------------+
                                   +-------------------------+    |
                                                                  |
                                     +----------------------------+
                                     |
                                     v
                                   +----------------------------+
                                   | struct rdt_ctrl_domain #B1 |
                                   +----------------------------+
                                   | rdt_domain_hdr             |---+
                                   | ...                        |   |
                                   +----------------------------+   |
                                                                    |
                                   +----------------------------+<--+
                                   | struct rdt_ctrl_domain #B2 |
                                   +----------------------------+
                                   | rdt_domain_hdr             |
                                   | ...                        |
                                   +----------------------------+

resctrl used to manage single domains that are capable of both monitoring and control but this was split to support scenario where monitoring and control of a resource are done at different scope. See
cd84f72b6a5c ("x86/resctrl: Prepare for different scope for control/monitor operations")

Quoted 7 lines

This feature further expands the difference between monitoring and control since now there can be multiple instances of a control domain (one per control) associated with a resource while monitoring still just supports one monitoring domain per resource.

I thus cannot see how this can be accomplished with a single list. Could you sketch out what you have in mind?

Quoted 8 lines

Thanks for the sketch. It makes it much more clear.

mon_domains directly maintains a list of monitoring domains, whereas controls maintains a list of control objects, each of which owns its own domain list.

I was thinking, one possibility would be to make mon_domains follow a structure similar to controls, allowing both to be represented through a common top-level list.

However, the resulting abstraction may become fairly large and complex, with limited overlap between the monitoring and control data structures.

I am trying to envision this but it sounds to be as though each control has its own monitoring which is not clear to me. resctrl ultimately needs to support hardware features. Could you please share more about the hardware feature(s) this design intends to support?

Reinette

reviewer
#5 lore raw

Hi Reinette,

Quoted 168 lines · Reinette Chatre

Hi Babu,

Quoted 8 lines · Babu

Hi Reinette,

Quoted 5 lines · Reinette Chatre

Hi Babu,

Quoted 2 lines · Babu Moger

Hi Reinette,

Quoted 7 lines
Quoted 5 lines
Quoted 2 lines

For some reason, I couldn’t find your patch on lore.kernel.org:
https://lore.kernel.org/lkml/?q=Reinette+Chatre

How about:
https://lore.kernel.org/lkml/[email protected]/

Yes. It works. But, I used to find patches using the names. It did not work this time.

Interesting. When I run your lore query the first email from this thread is among the results.

Quoted 146 lines
Quoted 134 lines
Quoted 9 lines

I eventually located it here:
https://sashiko.dev/#/message/aab804b9-e8b5-40ad-a85b-af7033391243%40intel.com

Thanks for sharing the patches. I’m still reviewing them.

I was able to build and boot the kernel and can see the MIN and MAX controls. After moving your test code to __rdt_get_mem_config_amd().

Thank you very much for trying it out.

Quoted 1 line · Reinette Chatre

...

Quoted 51 lines
Quoted 39 lines

Primary resctrl fs data structure changes
=========================================

Introduces a control represented by struct resctrl_ctrl that looks as below. To make the changes easier to follow I kept some of the original names to help communicate where familiar data structures land.

What to notice about a control is that it has some common properties required from all controls (scope, type, etc.) and then depending on the type of control (RESCTRL_CTRL_BITMAP or RESCTRL_CTRL_SCALAR) there are type specific properties.

/**
   * struct resctrl_ctrl - A resource control
   * @entry:    List entry of rdt_resource::controls
   * @scope:    Scope of the resource that this control allocates
   * @domains:    RCU list of all control domains
   * @type:    The control type that determines the properties of the control, *        format string for displaying control values to user space, and *        parser of control values provided by user space.
   * @name:    Name of the control. Appended to final resource name *        (rdt_resource_final::name) to create final schema entry. *        Specifically, "rdt_resource_final::name"_"resctrl_ctrl::name". *        For example, with resource name "MB" and control name "MAX" the *        schema entry will be "MB_MAX".
   * @cache:    Cache allocation control properties.
   * @membw:    Bandwidth control properties.
   */
struct resctrl_ctrl {
     struct list_head    entry;
     enum resctrl_scope    scope;
     struct list_head    domains;
     enum resctrl_ctrl_type    type;
     enum resctrl_ctrl_name    name;
     union {
         struct resctrl_cache    cache;
         struct resctrl_membw    membw;
     };
};

Quoted 11 lines

Two members summarize how this new structure fits into the rest of resctrl:
a) resctrl_ctrl::entry
     Since a resource can support multiple controls there is a new list in struct rdt_resource named "controls" that contains the list of all controls supported by the resource.
b) resctrl_ctrl::domains
     Instead of the list of control domains belonging to a resource they now belong to the control self. By doing so resctrl can support resource controls at different scope for the same resource. This is intended to support some upcoming MPAM and RISC-V usages.

Quoted 7 lines

I like the idea of supporting multiple controls for each resource.

With these patches, now we have one list containing all the controls.

However, in case of RDT_RESOURCE_L3, we have two lists "mon_domains" and "controls". mon_domains list deals with monitoring and control deals with control parts(multiple).

Have you thought about making the list("control") generic so that the control can be monitoring also. It will just one list containing multiple controls or monitor.

The control list adds an additional layer of abstraction just for control management, independent from monitoring. The mon_domains list is unchanged while each control now has its own ctrl_domains list.

Here is an attempt to visualize how a resource with two monitoring domains, and two controls, each with two control domains end up being managed:

+-------------------------+
| struct rdt_resource     |
+-------------------------+
| ...                     |
| controls (list_head)    |---------+
| mon_domains (list_head) |---+     |
| ...                     |   |     |
+-------------------------+   |     |
                                |     |
    +---------------------------+     |
    |                                 |
    v                                 v
+-----------------------------+   +-------------------------+
| struct rdt_l3_mon_domain #1 |   | struct resctrl_ctrl #A  |
+-----------------------------+   +-------------------------+
| rdt_domain_hdr              |-+ | entry (list_head)       |       +----------------------------+
| ...                         | | | domains (list_head)     |------>| struct rdt_ctrl_domain #A1 |
+-----------------------------+ | | ...                     |       +----------------------------+
                                  | +-------------------------+       | rdt_domain_hdr             |---+
    +-----------------------------+   |                               | ...                        |   |
    | (next)                          | (next)                        +----------------------------+   |
    v                                 v                                                                |
+-----------------------------+   +-------------------------+       +----------------------------+<--+
| struct rdt_l3_mon_domain #2 |   | struct resctrl_ctrl #B  |       | struct rdt_ctrl_domain #A2 |
+-----------------------------+   +-------------------------+       +----------------------------+
| rdt_domain_hdr              |   | entry (list_head)       |       | rdt_domain_hdr             |
| ...                         |   | domains (list_head)     |----n  | ...                        |
+-----------------------------+   | ...                     |    |  +----------------------------+
                                    +-------------------------+    |
                                                                   |
                                      +----------------------------+
                                      |
                                      v
                                    +----------------------------+
                                    | struct rdt_ctrl_domain #B1 |
                                    +----------------------------+
                                    | rdt_domain_hdr             |---+
                                    | ...                        |   |
                                    +----------------------------+   |
                                                                     |
                                    +----------------------------+<--+
                                    | struct rdt_ctrl_domain #B2 |
                                    +----------------------------+
                                    | rdt_domain_hdr             |
                                    | ...                        |
                                    +----------------------------+

resctrl used to manage single domains that are capable of both monitoring and control but this was split to support scenario where monitoring and control of a resource are done at different scope. See
cd84f72b6a5c ("x86/resctrl: Prepare for different scope for control/monitor operations")

Quoted 7 lines

This feature further expands the difference between monitoring and control since now there can be multiple instances of a control domain (one per control) associated with a resource while monitoring still just supports one monitoring domain per resource.

I thus cannot see how this can be accomplished with a single list. Could you sketch out what you have in mind?

Thanks for the sketch. It makes it much more clear.

mon_domains directly maintains a list of monitoring domains, whereas controls maintains a list of control objects, each of which owns its own domain list.

Quoted 9 lines
Quoted 4 lines

I was thinking, one possibility would be to make mon_domains follow a structure similar to controls, allowing both to be represented through a common top-level list.

However, the resulting abstraction may become fairly large and complex, with limited overlap between the monitoring and control data structures.

I am trying to envision this but it sounds to be as though each control has its own monitoring which is not clear to me. resctrl ultimately needs to support hardware features. Could you please share more about the hardware feature(s) this design intends to support?

This is mostly my thinking. There is no new hardware feature that required this change. We can safely ignore this idea.

Thanks
Babu

#6 lore raw
In reply to Reinette Chatre

Hi Reinette,
On 5/30/2026 2:06 AM, Reinette Chatre wrote:

[ ... ]

Quoted 16 lines

/**
* struct resctrl_ctrl - A resource control
* @entry: List entry of rdt_resource::controls
* @scope: Scope of the resource that this control allocates
* @domains: RCU list of all control domains
* @type: The control type that determines the properties of the control,
* format string for displaying control values to user space, and
* parser of control values provided by user space.
* @name: Name of the control. Appended to final resource name
* (rdt_resource_final::name) to create final schema entry.
* Specifically, "rdt_resource_final::name"_"resctrl_ctrl::name".
* For example, with resource name "MB" and control name "MAX" the
* schema entry will be "MB_MAX".
* @cache: Cache allocation control properties.
* @membw: Bandwidth control properties.
*/

Quoted 12 lines
struct resctrl_ctrl {
	struct list_head	entry;
	enum resctrl_scope	scope;
	struct list_head	domains;
	enum resctrl_ctrl_type	type;
	enum resctrl_ctrl_name	name;
	union {
		struct resctrl_cache	cache;
		struct resctrl_membw	membw;
	};
};

Thanks for re-spinning this patch set.
Looking at commit (fs/resctrl: Introduce additional schema properties), the newly added properties appear to be implemented for the MBA resctrl_membw controller.
Would it make sense for these properties to be generic across all resctrl controllers, CAT included? Should they consequently be relocated into struct resctrl_ctrl if that approach is appropriate?

thanks,
Chenyu

author maintainer
#7 lore raw

Hi Chenyu,

Quoted 34 lines · Yu C

Hi Reinette,
On 5/30/2026 2:06 AM, Reinette Chatre wrote:

[ ... ]

Quoted 19 lines

/**
  * struct resctrl_ctrl - A resource control
  * @entry:    List entry of rdt_resource::controls
  * @scope:    Scope of the resource that this control allocates
  * @domains:    RCU list of all control domains
  * @type:    The control type that determines the properties of the control, *        format string for displaying control values to user space, and *        parser of control values provided by user space.
  * @name:    Name of the control. Appended to final resource name *        (rdt_resource_final::name) to create final schema entry. *        Specifically, "rdt_resource_final::name"_"resctrl_ctrl::name". *        For example, with resource name "MB" and control name "MAX" the *        schema entry will be "MB_MAX".
  * @cache:    Cache allocation control properties.
  * @membw:    Bandwidth control properties.
  */
struct resctrl_ctrl {
    struct list_head    entry;
    enum resctrl_scope    scope;

Quoted 9 lines

    struct list_head    domains;
    enum resctrl_ctrl_type    type;
    enum resctrl_ctrl_name    name;
    union {
        struct resctrl_cache    cache;
        struct resctrl_membw    membw;
    };
};

Quoted 4 lines

Thanks for re-spinning this patch set.
Looking at commit (fs/resctrl: Introduce additional schema properties), the newly added properties appear to be implemented for the MBA resctrl_membw controller.

Please do note that the "struct resctrl_membw" naming is temporary. I only kept current naming to help folks find where existing known structures land in this new design but the "struct resctrl_membw" name is not appropriate moving forward.

When user space interacts with the new controls the intention is that each control has a "type" that the user can find in the new "type" file and depending on that value the user will know what other files/properties can be expected.

In this PoC there are two types: "scalar" and "bitmap" and each is associated with a struct that contains the properties associated with that type.

Considering this, these structures could, for example, be renamed as (very long):
struct resctrl_membw -> struct resctrl_ctrl_scalar
struct resctrl_cache -> struct resctrl_ctrl_bitmap

Quoted 3 lines

Would it make sense for these properties to be generic across all resctrl controllers, CAT included? Should they consequently be relocated into struct resctrl_ctrl if that approach is appropriate?

If there are properties that are common across all controls of all types then they should be moved to struct resctrl_ctrl. Current examples are "type" and "scope" that user space can expect from every control. It is not obvious to me at this time what other properties we can designate as common in this way.

For bitmaps there is already the properties that are available in the top-level directory. These can just be replicated as properties into the resource_schemata, but perhaps using new names as suggested by Dave in
https://lore.kernel.org/lkml/aQOUAeVP9oc7RIn%[email protected]/

Reinette

#8 lore raw

Hi Reinette,

Quoted 2 lines · Reinette Chatre

Hi Chenyu,

[ ... ]

Quoted 14 lines
Quoted 5 lines

Thanks for re-spinning this patch set.
Looking at commit (fs/resctrl: Introduce additional schema properties), the newly added properties appear to be implemented for the MBA resctrl_membw controller.

Please do note that the "struct resctrl_membw" naming is temporary. I only kept current naming to help folks find where existing known structures land in this new design but the "struct resctrl_membw" name is not appropriate moving forward.

When user space interacts with the new controls the intention is that each control has a "type" that the user can find in the new "type" file and depending on that value the user will know what other files/properties can be expected.

Quoted 7 lines

In this PoC there are two types: "scalar" and "bitmap" and each is associated with a struct that contains the properties associated with that type.

Considering this, these structures could, for example, be renamed as (very long):
struct resctrl_membw -> struct resctrl_ctrl_scalar
struct resctrl_cache -> struct resctrl_ctrl_bitmap

OK, the type field serves as the primary key for querying other properties. Alongside the "type" entry, there also exists a "flag" property. I haven't spotted this field within resctrl_membw, though it appears in resctrl_cache via arch_has_sparse_bitmasks. Would it make sense to introduce a dedicated enum for this flag, or alternatively reuse the existing bw_delay_linear for MBA?

Quoted 9 lines
Quoted 3 lines

Would it make sense for these properties to be generic across all resctrl controllers, CAT included? Should they consequently be relocated into struct resctrl_ctrl if that approach is appropriate?

If there are properties that are common across all controls of all types then they should be moved to struct resctrl_ctrl. Current examples are "type" and "scope" that user space can expect from every control. It is not obvious to me at this time what other properties we can designate as common in this way.

Quoted 5 lines

For bitmaps there is already the properties that are available in the top-level directory. These can just be replicated as properties into the resource_schemata, but perhaps using new names as suggested by Dave in
https://lore.kernel.org/lkml/aQOUAeVP9oc7RIn%[email protected]/

Got it. Will further look into the code.

thanks,
Chenyu

author maintainer
#9 lore raw

Hi Chenyu,

Quoted 30 lines · Yu C

Hi Reinette,

Quoted 2 lines · Reinette Chatre

Hi Chenyu,

[ ... ]

Quoted 14 lines
Quoted 5 lines

Thanks for re-spinning this patch set.
Looking at commit (fs/resctrl: Introduce additional schema properties), the newly added properties appear to be implemented for the MBA resctrl_membw controller.

Please do note that the "struct resctrl_membw" naming is temporary. I only kept current naming to help folks find where existing known structures land in this new design but the "struct resctrl_membw" name is not appropriate moving forward.

When user space interacts with the new controls the intention is that each control has a "type" that the user can find in the new "type" file and depending on that value the user will know what other files/properties can be expected.

Quoted 7 lines

In this PoC there are two types: "scalar" and "bitmap" and each is associated with a struct that contains the properties associated with that type.

Considering this, these structures could, for example, be renamed as (very long):
struct resctrl_membw -> struct resctrl_ctrl_scalar
struct resctrl_cache -> struct resctrl_ctrl_bitmap

Quoted 5 lines

OK, the type field serves as the primary key for querying other properties. Alongside the "type" entry, there also exists a "flag" property. I haven't spotted this field within resctrl_membw, though it appears in resctrl_cache via arch_has_sparse_bitmasks. Would it make sense to introduce a dedicated enum for this flag, or alternatively reuse the existing bw_delay_linear for MBA?

I realize now my previous answer to you was incomplete. You are right that there is a plan to let the resctrl "type" file contain the schema type as well as optional flags. This plan is unchanged but at this time it is not obvious to me what flags this implementation should start with.

In the original proposal [1] "linear" was provided as example of a flag for the MB resource but as I worked through this PoC whether a control is linear or not seemed to fit better as a resource property. This is how I ended up with commit bdcd8ac6e946 ("mpam,x86,fs/resctrl: Make memory bandwidth delay a resource property")

For this specific property I expect that all controls would have the same value. This worked out well since resctrl already has the per-resource top level "delay_linear".

We can surely move this back to be a control property and have it be the first "flag" but at this time it seems to me that all MB controls would just have the same flag value?

Perhaps the safest alternative would be to keep it a resource property and just duplicate this as a flag value among all the controls? I think this is what you are suggesting to reuse the bw_delay_linear for MBA. This would not result in dedicated flags associated with controls but it may set the user interface up for most flexibility.

Reinette

[1] https://lore.kernel.org/lkml/aPtfMFfLV1l%[email protected]/

#10 lore raw

Hi Reinette,

On 6/5/2026 12:37 AM, Reinette Chatre wrote:

[ ... ]

Quoted 20 lines
Quoted 6 lines

OK, the type field serves as the primary key for querying other properties. Alongside the "type" entry, there also exists a "flag" property. I haven't spotted this field within resctrl_membw, though it appears in resctrl_cache via arch_has_sparse_bitmasks. Would it make sense to introduce a dedicated enum for this flag, or alternatively reuse the existing bw_delay_linear for MBA?

I realize now my previous answer to you was incomplete. You are right that there is a plan to let the resctrl "type" file contain the schema type as well as optional flags. This plan is unchanged but at this time it is not obvious to me what flags this implementation should start with.

In the original proposal [1] "linear" was provided as example of a flag for the MB resource but as I worked through this PoC whether a control is linear or not seemed to fit better as a resource property. This is how I ended up with commit bdcd8ac6e946 ("mpam,x86,fs/resctrl: Make memory bandwidth delay a resource property")

For this specific property I expect that all controls would have the same value. This worked out well since resctrl already has the per-resource top level "delay_linear".

Quoted 9 lines

We can surely move this back to be a control property and have it be the first "flag" but at this time it seems to me that all MB controls would just have the same flag value?

Perhaps the safest alternative would be to keep it a resource property and just duplicate this as a flag value among all the controls? I think this is what you are suggesting to reuse the bw_delay_linear for MBA. This would not result in dedicated flags associated with controls but it may set the user interface up for most flexibility.

Yes I think we can simply reuse the value of bw_delay_linear when displaying the "flag" field for each controller under the info directory.
BTW, would the L3 controller have similar case that, all L3 controllers of the same resource share the same value of "flag"? If yes, should we also move arch_has_sparse_bitmasks to rdt_resource? If yes, I'm not sure why we need "flag" in the controller.

thanks,
Chenyu

author maintainer
#11 lore raw

Hi Chenyu,

Quoted 36 lines · Yu C

Hi Reinette,

On 6/5/2026 12:37 AM, Reinette Chatre wrote:

[ ... ]

Quoted 20 lines
Quoted 6 lines

OK, the type field serves as the primary key for querying other properties. Alongside the "type" entry, there also exists a "flag" property. I haven't spotted this field within resctrl_membw, though it appears in resctrl_cache via arch_has_sparse_bitmasks. Would it make sense to introduce a dedicated enum for this flag, or alternatively reuse the existing bw_delay_linear for MBA?

I realize now my previous answer to you was incomplete. You are right that there is a plan to let the resctrl "type" file contain the schema type as well as optional flags. This plan is unchanged but at this time it is not obvious to me what flags this implementation should start with.

In the original proposal [1] "linear" was provided as example of a flag for the MB resource but as I worked through this PoC whether a control is linear or not seemed to fit better as a resource property. This is how I ended up with commit bdcd8ac6e946 ("mpam,x86,fs/resctrl: Make memory bandwidth delay a resource property")

For this specific property I expect that all controls would have the same value. This worked out well since resctrl already has the per-resource top level "delay_linear".

Quoted 9 lines

We can surely move this back to be a control property and have it be the first "flag" but at this time it seems to me that all MB controls would just have the same flag value?

Perhaps the safest alternative would be to keep it a resource property and just duplicate this as a flag value among all the controls? I think this is what you are suggesting to reuse the bw_delay_linear for MBA. This would not result in dedicated flags associated with controls but it may set the user interface up for most flexibility.

Quoted 5 lines

Yes I think we can simply reuse the value of bw_delay_linear when displaying the "flag" field for each controller under the info directory.
BTW, would the L3 controller have similar case that, all L3 controllers of the same resource share the same value of "flag"? If yes, should we also move arch_has_sparse_bitmasks to rdt_resource? If yes, I'm not sure why we need "flag" in the controller.

Whether a control of type bitmap allows sparse values or not does seem to be a valid property of the bitmap control to me. It is difficult to predict how this will land since I am not familiar with a resource that has multiple bitmap controllers.

I do think having a controller type support different flags in the interface exposed to the user does give resctrl most flexibility for the future. With the user interface having that capability the internal implementation can always adapt.

Reinette

maintainer
#12 lore raw
In reply to Reinette Chatre

Hi Reinette,

Quoted 10 lines · Reinette Chatre

Hi Everybody,

It has been a while since we discussed the resctrl changes required to support hardware that has controls with fine granularity or hardware that has multiple controls per resource. For reference, the most recent email discussion can be found at [1] with a summary of discussions in last year's plumbers slides [2].

I created a PoC that I believe supports what folks have agreed to so far. I hope this can help us to restart the discussion with the goal that resctrl gains support for upcoming hardware that require these features.

Thank you very much for doing this work. I believe this will be very useful for MPAM and other architectures.

Quoted 14 lines

Request regarding this PoC
==========================

Please consider this PoC as a "direction check" on the schema description and multiple control discussions held thus far.

Could folks working on enabling new hardware requiring this capability please consider if this is something you can build on and how it should be improved to support these upcoming capabilities?

Opens
=====

Quoted 9 lines

While the PoC aims to support what folks agreed on some opens remain:
- I attempted to make some MPAM supporting changes but these are all just compile tested. While MPAM should benefit from the new control properties I did not initialize them on MPAM and did not attempt refactor to separate out the architecture specific control properties (more on what this means later). I did attempt some MPAM refactoring that duplicates the MPAM domain to the control domain and monitoring domain lists in support of there being multiple controls each with its own list of control domains but it is definitely not good design.

I appreciate you including MPAM in this PoC. With this one line change I was able to boot an MPAM system and mount resctrl which appears to behave correctly. We can consider how best to do the code design later.

drivers/resctrl/mpam_resctrl.c +1 -0
@@ -1697,6 +1697,8 @@ int mpam_resctrl_setup(void)
 
        /* Initialise the resctrl structures from the classes */
        for_each_mpam_resctrl_control(res, rid) {
+               INIT_LIST_HEAD(&res->resctrl_res.controls); // list_empty needs

to work
+
if (!res->class)
continue; // dummy resource

I plumbed in support for the MB_MIN resource schema which also works under light testing. The only fs resctrl code change I needed was:

include/linux/resctrl.h +0 -0
@@ -483,6 +483,9 @@ static inline u32 resctrl_get_default_ctrlval(struct

resctrl_ctrl *ctrl)

        case RESCTRL_CTRL_BITMAP:
                return BIT_MASK(ctrl->cache.cbm_len) - 1;
        case RESCTRL_CTRL_SCALAR:
+               if (ctrl->name == RESCTRL_CTRL_NAME_MIN)
+                       return ctrl->membw.min_bw;
+
                return ctrl->membw.max_bw;
        }

At least on MPAM systems, we use a default of 0 for minimum bandwidth controls as the maximum bandwidth controls only take effect if their value is higher than the minimum bandwidth value. I have specialised this on the ctrl->name which breaks your ctrl->type based classification but that's fixable by just adding a default field to membw.

Quoted 6 lines

- No support for emulated controls (yet). The PoC is quite large already but I think it can be used as a base for emulated controls for which the software controller could be a potential first customer. In this PoC mounting with software controller will still display the original controller's properties.
- One open that needs to be addressed as part of support for emulated controls is how best to display emulation relationship via resctrl hierarchy.

What does emulated controls mean here? Is there some previous discussion you could point me at?

Quoted 6 lines

- No support for "read-modify-write" usage of schemata file. This is where we discussed (without agreement) on possibly introducing the "#" prefix to schemata file entries. This PoC does not support this prefix and the current assumption/expectation is that when user space changes a configuration only the new control values are written to schemata file. I thus do not have a plan to support this so please share opinions in this regard if you have some.

There is now less motivation from the MPAM side for this than when this was initially discussed. In pre-upstream versions of the MPAM patches a change in the MB resource control value would change both the mpam h/w mbw_min and mbw_max values but now (on non-broken h/w) we just change the mbw_max. (mbw_min kept at 0).

However, it would be useful not to be limited by percentages. In my quick experimentation with your patches I used a percentage value for MB_MIN but it would be best to move away from this. For new controls I think we can mandate that user space has to discover the resolution from the info directly but how can we retrofit this. For MPAM, MB and MB_MAX, would control the same things. Could we just add MB_MAX with a h/w friendly scale and then reflect changes in MB_MAX in MB and vica versa with MB taking precedent if both are set? Old software can continue setting MB can move to using MB_MAX and take advantage of the improved control. (I don't think we should expose the MPAM hardware value directly as it has confusion over whether all 1s is 100% or not and we'd like to have something generic and friendly to the user.)

Quoted 3 lines

- Controls are independent for now. This means that, for example, if a resource supports a "MIN" and "MAX" control then this implementation would allow user to set the "maximum" control values to be less than the "minimum" control values.

I think this is ok as long as adding support for new controls in resctrl doesn't change the existing behaviour. In MPAM we dodged this by introducing MB as only affecting the h/w mbw_max and not mbw_min (as mentioned above).

Quoted 68 lines

- PoC supports the "bitmap" control but does not (yet) expose properties of a bitmap control to the new info/<resource>/resource_schemata directory.

Accessing PoC
=============

Please consider the PoC as a rough draft. It has only been compile tested for Arm and known to be incomplete in Arm support. To help with experimenting I only fully adapted the Intel MBA resource to demo two dummy additional MBA controls. All architectures should immediately benefit from the new schema descriptions and new info/MB/resource_schemata hierarchy.

I considered the patches self too many for email. Instead, the PoC can be found at:

git://git.kernel.org/pub/scm/linux/kernel/git/reinette/linux.git branch resctrl/controls_rfc_v1

The work is based on v7.1-rc2 that also includes the following series (two of which has since been queued) included:

"selftests/resctrl: Fixes and improvements focused on Intel platforms"
https://lore.kernel.org/lkml/[email protected]/

"x86,fs/resctrl: Improve resctrl quality and consistency"
https://lore.kernel.org/lkml/[email protected]/

"x86,fs/resctrl: Pave the way for MPAM counter assignment"
https://lore.kernel.org/lkml/[email protected]/

Primary resctrl fs data structure changes
=========================================

Introduces a control represented by struct resctrl_ctrl that looks as below. To make the changes easier to follow I kept some of the original names to help communicate where familiar data structures land.

What to notice about a control is that it has some common properties required from all controls (scope, type, etc.) and then depending on the type of control (RESCTRL_CTRL_BITMAP or RESCTRL_CTRL_SCALAR) there are type specific properties.

/**
* struct resctrl_ctrl - A resource control
* @entry: List entry of rdt_resource::controls
* @scope: Scope of the resource that this control allocates
* @domains: RCU list of all control domains
* @type: The control type that determines the properties of the control,
* format string for displaying control values to user space, and
* parser of control values provided by user space.
* @name: Name of the control. Appended to final resource name
* (rdt_resource_final::name) to create final schema entry.
* Specifically, "rdt_resource_final::name"_"resctrl_ctrl::name".
* For example, with resource name "MB" and control name "MAX" the
* schema entry will be "MB_MAX".
* @cache: Cache allocation control properties.
* @membw: Bandwidth control properties.
*/

struct resctrl_ctrl {
	struct list_head	entry;
	enum resctrl_scope	scope;
	struct list_head	domains;
	enum resctrl_ctrl_type	type;
	enum resctrl_ctrl_name	name;
	union {
		struct resctrl_cache	cache;
		struct resctrl_membw	membw;
	};
};
Quoted 10 lines

Two members summarize how this new structure fits into the rest of resctrl:
a) resctrl_ctrl::entry
Since a resource can support multiple controls there is a new list in struct rdt_resource named "controls" that contains the list of all controls supported by the resource.
b) resctrl_ctrl::domains
Instead of the list of control domains belonging to a resource they now belong to the control self. By doing so resctrl can support resource controls at different scope for the same resource. This is intended to support some upcoming MPAM and RISC-V usages.

Please can you expand a bit on part b).

In an MPAM system we consider 3 resctrl resources, RDT_RESOURCE_L3, RDT_RESOURCE_L2 and RDT_RESOURCE_MBA which correspond to the L3 caches, L2 caches and memory bandwidth on egress from the L3 caches. The domain for each of these corresponds to the instance of the resource. That is, for RDT_RESOURCE_L2 there is a resource for each L2 instance, similarly for L3, and for RDT_RESOURCE_MBA there is a domain for each L3 cache. If we were to add suport for controls on a new cache level, say the L4, then I'd expect to add a new resource. For memory bandwidth, we'd like to be able to control b/w on the L2 egress (e.g. in a DSU). Wouldn't this too be a separate resource or would this be a new set of controls on the same resource?

New controls on the same resource
MB_MIN2
MB_MAX2
MB_PROP2
...

or
MB2_MIN
MB2_MAX
MB2_PROP

AFAIK, the DSU h/w just supports proportional bandwidth controls at the moment but we should consider what to do about the potential naming.

In the MPAM driver, we collect MSC into components (based on instances) and those into classes (components of the same type). Currently, a resource is mapped to a single class. (Two resources may map to the same class.)

I expect it is useful in the memory region and sub numa cases but I'd still expect the common case to be that the domains are the same within a control. Or am I missing something?

Quoted 25 lines

Example architectural data structure changes
============================================

An architecture can use the new control by following a similar pattern to resource and domain use by architectures. Consider the following for x86 where a new architecture specific struct resctrl_hw_ctrl includes struct resctrl_ctrl and any architecture private data needed to support the control:

/*
* struct resctrl_hw_ctrl - Arch private properties of a resource control
* @r_ctrl: Control properties exposed to resctrl file system
* @msr_base: Base MSR address where control values should be programmed
* @msr_update: Function pointer to update control values
*/

struct resctrl_hw_ctrl {
	struct resctrl_ctrl	r_ctrl;
	unsigned int		msr_base;
	void			(*msr_update)(struct msr_param *m);
};

Structure of patch series
=========================

Quoted 7 lines

As a PoC the series is not perfectly structured but to help navigate this work on a high level the changes can be categorized as follows:

Patch 1 to 11:
With a vision of what a "control" is, remove unused/unnecessary members, make clear what is a *resource* property vs a *control* property, do some renaming to help with the PoC.

A few of the changes are generic cleanup and could hopefully be dealt with before decisions on the larger PoC are made. I see:
fs/resctrl: Remove unused resctrl_membw::mb_map x86,fs/resctrl: Remove "arch_needs_linear"
Perhaps a few more.

Quoted 10 lines

Patch 12:
Introduce struct resctrl_ctrl and re-arrange existing struct rdt_resource members to form part of new rdt_resource::ctrl

Patch 13 to 44:
A lot of wrangling to introduce struct resctrl_ctrl to all code that needs to work with a control and/or domain without assuming that the control is the one and only control embedded in the resource it belongs to. Essentially, a lot of changes passing the control around in addition to the resource/domain.

You mention a few times in the commit message that you expect the cache resources to only have one control. On MPAM we have CMAX (and there looks to be a RISC-V equivalent) where the total number of bytes in the cache for a given closid is limited. The allocation must still respect the CPBM bitmap though. Looking at the code though I don't see much problem in adding this as an additional control. The assumption that these patches is making is not that there is only one control for cache resources but rather that cache portions are managed by the default cache resource control. Am I missing something or does that assessment make sense to you?

I have been looking at adding CMAX control to resctrl and will have a go at basing what I have so far on top of this series.

Quoted 67 lines

Patch 45:
Switch the single struct resctrl_ctrl member of struct rdt_resource to be a list of struct resctrl_ctrl.

Patch 47 to 49:
Introduce new info/<resource>/resource_schemata hierarchy to first only consist of properties already known to resctrl fs.

Patch 50 to 52:
Introduce the new control properties per [1], initialize them for x86, and expose them via info/<resource>/resource_schemata

Patch 53:
Let the new struct resctrl_hw_ctrl contain architecture's control properties.

Patch 54:
Teach resctrl fs about "MIN" and "MAX" controls.

Patch 55:
Sample of "MIN" and "MAX" memory bandwidth controls for x86.

Quoted 1 line

Example interactions

====================

This series can be used on an x86 system where it will show two new dummy controls where it is possible to interact with the new controls.
For example:

# cat schemata 
        MB_MAX:0=100;1=100
        MB_MIN:0=100;1=100
        MB:0=100;1=100
        L3:0=fff;1=fff
# echo 'MB_MIN:0=50' > schemata 
# cat schemata 
        MB_MAX:0=100;1=100
        MB_MIN:0=50;1=100
        MB:0=100;1=100
        L3:0=fff;1=fff

Writing to the dummy control will call a dummy callback that just prints to the kernel log:
"resctrl: Updata temporary MIN control on domain 0 with user value 50"

Example output of info/MB/:
/sys/fs/resctrl/info/MB/thread_throttle_mode:max /sys/fs/resctrl/info/MB/num_closids:15 /sys/fs/resctrl/info/MB/delay_linear:1 /sys/fs/resctrl/info/MB/min_bandwidth:10 /sys/fs/resctrl/info/MB/resource_schemata/MB/resolution:100 /sys/fs/resctrl/info/MB/resource_schemata/MB/tolerance:5 /sys/fs/resctrl/info/MB/resource_schemata/MB/type:scalar /sys/fs/resctrl/info/MB/resource_schemata/MB/min:10 /sys/fs/resctrl/info/MB/resource_schemata/MB/scale:1 /sys/fs/resctrl/info/MB/resource_schemata/MB/scope:L3 /sys/fs/resctrl/info/MB/resource_schemata/MB/unit:all /sys/fs/resctrl/info/MB/resource_schemata/MB/max:100 /sys/fs/resctrl/info/MB/resource_schemata/MB_MIN/resolution:100 /sys/fs/resctrl/info/MB/resource_schemata/MB_MIN/tolerance:5 /sys/fs/resctrl/info/MB/resource_schemata/MB_MIN/type:scalar /sys/fs/resctrl/info/MB/resource_schemata/MB_MIN/min:10 /sys/fs/resctrl/info/MB/resource_schemata/MB_MIN/scale:1 /sys/fs/resctrl/info/MB/resource_schemata/MB_MIN/scope:L3 /sys/fs/resctrl/info/MB/resource_schemata/MB_MIN/unit:all /sys/fs/resctrl/info/MB/resource_schemata/MB_MIN/max:100 /sys/fs/resctrl/info/MB/resource_schemata/MB_MAX/resolution:100

Quoted 9 lines

/sys/fs/resctrl/info/MB/resource_schemata/MB_MAX/tolerance:5 /sys/fs/resctrl/info/MB/resource_schemata/MB_MAX/type:scalar /sys/fs/resctrl/info/MB/resource_schemata/MB_MAX/min:10 /sys/fs/resctrl/info/MB/resource_schemata/MB_MAX/scale:1 /sys/fs/resctrl/info/MB/resource_schemata/MB_MAX/scope:L3 /sys/fs/resctrl/info/MB/resource_schemata/MB_MAX/unit:all /sys/fs/resctrl/info/MB/resource_schemata/MB_MAX/max:100 /sys/fs/resctrl/info/MB/bandwidth_gran:10

Quoted 1 line

Any feedback is appreciated.

Overall, this looks to be a big step in the right direction.

Thanks,

Ben

Quoted 5 lines

Reinette

[1] https://lore.kernel.org/lkml/aPtfMFfLV1l%[email protected]/
[2] https://lpc.events/event/19/contributions/2093/attachments/1958/4172/resctrl%20Microconference%20LPC%202025%20Tokyo.pdf

#13 lore raw
Quoted 10 lines · Ben Horgan

Hi Reinette,

Quoted 7 lines · Reinette Chatre

Hi Everybody,

It has been a while since we discussed the resctrl changes required to support hardware that has controls with fine granularity or hardware that has multiple controls per resource. For reference, the most recent email discussion can be found at [1] with a summary of discussions in last year's plumbers slides [2].

Quoted 6 lines
Quoted 3 lines

I created a PoC that I believe supports what folks have agreed to so far. I hope this can help us to restart the discussion with the goal that resctrl gains support for upcoming hardware that require these features.

Thank you very much for doing this work. I believe this will be very useful for MPAM and other architectures.

Yes, thanks to Reinette for working on the generic schema proof of concept. This will be helpful for supporting the RISC-V CBQRI (capacity and bandwidth QoS) spec.

Quoted 17 lines

I plumbed in support for the MB_MIN resource schema which also works under light testing. The only fs resctrl code change I needed was:

include/linux/resctrl.h +3 -0
@@ -483,6 +483,9 @@ static inline u32 resctrl_get_default_ctrlval(struct
 resctrl_ctrl *ctrl)
        case RESCTRL_CTRL_BITMAP:
                return BIT_MASK(ctrl->cache.cbm_len) - 1;
        case RESCTRL_CTRL_SCALAR:
+               if (ctrl->name == RESCTRL_CTRL_NAME_MIN)
+                       return ctrl->membw.min_bw;
+
                return ctrl->membw.max_bw;
        }
Quoted 5 lines

At least on MPAM systems, we use a default of 0 for minimum bandwidth controls as the maximum bandwidth controls only take effect if their value is higher than the minimum bandwidth value. I have specialised this on the ctrl->name which breaks your ctrl->type based classification but that's fixable by just adding a default field to membw.

This should be useful for RISC-V.

RESCTRL_CTRL_NAME_MIN maps well to CBQRI Rbwb (reserved bandwidth blocks). The sum of Rbwb across all control groups must be less than MRBWB (maximum number of reserved bandwidth blocks). As a result, MB_MIN needs to default to 1 so that the sum does not violate that rule. In my RFC series, I added default_to_min to resctrl_membw [1] but this solution looks cleaner.

Quoted 12 lines
Quoted 6 lines

- No support for "read-modify-write" usage of schemata file. This is where we discussed (without agreement) on possibly introducing the "#" prefix to schemata file entries. This PoC does not support this prefix and the current assumption/expectation is that when user space changes a configuration only the new control values are written to schemata file. I thus do not have a plan to support this so please share opinions in this regard if you have some.

There is now less motivation from the MPAM side for this than when this was initially discussed. In pre-upstream versions of the MPAM patches a change in the MB resource control value would change both the mpam h/w mbw_min and mbw_max values but now (on non-broken h/w) we just change the mbw_max. (mbw_min kept at 0).

Quoted 11 lines

However, it would be useful not to be limited by percentages. In my quick experimentation with your patches I used a percentage value for MB_MIN but it would be best to move away from this. For new controls I think we can mandate that user space has to discover the resolution from the info directly but how can we retrofit this. For MPAM, MB and MB_MAX, would control the same things. Could we just add MB_MAX with a h/w friendly scale and then reflect changes in MB_MAX in MB and vica versa with MB taking precedent if both are set? Old software can continue setting MB can move to using MB_MAX and take advantage of the improved control. (I don't think we should expose the MPAM hardware value directly as it has confusion over whether all 1s is 100% or not and we'd like to have something generic and friendly to the user.)

The facility for non-percentage value is import for RISC-V as CBQRI does not include percentage throttle. It has two controls for bandwidth:

- Rbwb: number of reserved bandwidth blocks [1, 2^13]
- Mweight: weighted share of the remaining bandwidth [0, 255]
- 0: disables work-conserving sharing
- 1..255: compete for the leftover pool
- It makes for it to default to max (255) so that there won't be any unused bandwidth

I think Mweight could be aligned with MPAM's proportional stride.

Here is the patch I created to add Mweight support:

fs/resctrl/ctrlmondata.c +1 -0
index d95ab8ad36e2..3537071e3ab0 100644
@@ -304,6 +304,7 @@ static const char * const resctrl_ctrl_name[] = {
 	[RESCTRL_CTRL_NAME_DEF]		= "",
 	[RESCTRL_CTRL_NAME_MIN]		= "MIN",
 	[RESCTRL_CTRL_NAME_MAX]		= "MAX",
+	[RESCTRL_CTRL_NAME_WGHT]	= "WGHT",
 };
 
 const char *resctrl_ctrl_name_str(enum resctrl_ctrl_name name)
include/linux/resctrl.h +3 -1
index 72fb7256270e..09efcef9ce66 100644
@@ -348,12 +348,14 @@ struct resctrl_mon {
  *				has the same name as the resource.
  * @RESCTRL_CTRL_NAME_MIN:	"MIN"
  * @RESCTRL_CTRL_NAME_MAX:	"MAX"
+ * @RESCTRL_CTRL_NAME_WGHT:	"WGHT"
  */
 enum resctrl_ctrl_name {
 	RESCTRL_CTRL_NAME_DEF,
 	RESCTRL_CTRL_NAME_MIN,
 	RESCTRL_CTRL_NAME_MAX,
-	RESCTRL_CTRL_NAME_LAST = RESCTRL_CTRL_NAME_MAX
+	RESCTRL_CTRL_NAME_WGHT,
+	RESCTRL_CTRL_NAME_LAST = RESCTRL_CTRL_NAME_WGHT
 };
 
Quoted 7 lines
Quoted 3 lines

- Controls are independent for now. This means that, for example, if a resource supports a "MIN" and "MAX" control then this implementation would allow user to set the "maximum" control values to be less than the "minimum" control values.

I think this is ok as long as adding support for new controls in resctrl doesn't change the existing behaviour. In MPAM we dodged this by introducing MB as only affecting the h/w mbw_max and not mbw_min (as mentioned above).

There is no equivalent to MB (percentage throttle) in RISC-V so I would want it to be valid to have MB_MIN (minimum reservation) without MB.

I rebased my RISC-V CBQRI v6 series on top of this proof of concept and was able to validate it works okay in Qemu:

MB_WGHT:72=255
MB_MIN:72=756

     L2:64=fff;65=fff
     L3:75=ffff

Thanks,
Drew

[1] https://lore.kernel.org/all/[email protected]/

maintainer
#14 lore raw

Hi Drew,

Quoted 81 lines · Drew Fustini
Quoted 10 lines · Ben Horgan

Hi Reinette,

Quoted 7 lines · Reinette Chatre

Hi Everybody,

It has been a while since we discussed the resctrl changes required to support hardware that has controls with fine granularity or hardware that has multiple controls per resource. For reference, the most recent email discussion can be found at [1] with a summary of discussions in last year's plumbers slides [2].

Quoted 6 lines
Quoted 3 lines

I created a PoC that I believe supports what folks have agreed to so far. I hope this can help us to restart the discussion with the goal that resctrl gains support for upcoming hardware that require these features.

Thank you very much for doing this work. I believe this will be very useful for MPAM and other architectures.

Yes, thanks to Reinette for working on the generic schema proof of concept. This will be helpful for supporting the RISC-V CBQRI (capacity and bandwidth QoS) spec.

Quoted 17 lines

I plumbed in support for the MB_MIN resource schema which also works under light testing. The only fs resctrl code change I needed was:

include/linux/resctrl.h +3 -0
@@ -483,6 +483,9 @@ static inline u32 resctrl_get_default_ctrlval(struct
 resctrl_ctrl *ctrl)
        case RESCTRL_CTRL_BITMAP:
                return BIT_MASK(ctrl->cache.cbm_len) - 1;
        case RESCTRL_CTRL_SCALAR:
+               if (ctrl->name == RESCTRL_CTRL_NAME_MIN)
+                       return ctrl->membw.min_bw;
+
                return ctrl->membw.max_bw;
        }
Quoted 5 lines

At least on MPAM systems, we use a default of 0 for minimum bandwidth controls as the maximum bandwidth controls only take effect if their value is higher than the minimum bandwidth value. I have specialised this on the ctrl->name which breaks your ctrl->type based classification but that's fixable by just adding a default field to membw.

This should be useful for RISC-V.

RESCTRL_CTRL_NAME_MIN maps well to CBQRI Rbwb (reserved bandwidth blocks). The sum of Rbwb across all control groups must be less than MRBWB (maximum number of reserved bandwidth blocks). As a result, MB_MIN needs to default to 1 so that the sum does not violate that rule. In my RFC series, I added default_to_min to resctrl_membw [1] but this solution looks cleaner.

Quoted 12 lines
Quoted 6 lines

- No support for "read-modify-write" usage of schemata file. This is where we discussed (without agreement) on possibly introducing the "#" prefix to schemata file entries. This PoC does not support this prefix and the current assumption/expectation is that when user space changes a configuration only the new control values are written to schemata file. I thus do not have a plan to support this so please share opinions in this regard if you have some.

There is now less motivation from the MPAM side for this than when this was initially discussed. In pre-upstream versions of the MPAM patches a change in the MB resource control value would change both the mpam h/w mbw_min and mbw_max values but now (on non-broken h/w) we just change the mbw_max. (mbw_min kept at 0).

Quoted 11 lines

However, it would be useful not to be limited by percentages. In my quick experimentation with your patches I used a percentage value for MB_MIN but it would be best to move away from this. For new controls I think we can mandate that user space has to discover the resolution from the info directly but how can we retrofit this. For MPAM, MB and MB_MAX, would control the same things. Could we just add MB_MAX with a h/w friendly scale and then reflect changes in MB_MAX in MB and vica versa with MB taking precedent if both are set? Old software can continue setting MB can move to using MB_MAX and take advantage of the improved control. (I don't think we should expose the MPAM hardware value directly as it has confusion over whether all 1s is 100% or not and we'd like to have something generic and friendly to the user.)

The facility for non-percentage value is import for RISC-V as CBQRI does not include percentage throttle. It has two controls for bandwidth:

Quoted 8 lines

- Rbwb: number of reserved bandwidth blocks [1, 2^13]
- Mweight: weighted share of the remaining bandwidth [0, 255]
- 0: disables work-conserving sharing
- 1..255: compete for the leftover pool
- It makes for it to default to max (255) so that there won't be any unused bandwidth

I think Mweight could be aligned with MPAM's proportional stride.

Yes, I hope so. There a few differences which would have to be considered.

MPAM doesn't have a concept of only applying the weights once reserved min bandwidth is consumed. The interaction with min bandwidth is currently unspecified. I don't think there are any designs where proportional bandwidth and min b/w are on the same component and so it's only a theoretical/future problem.

For MPAM proportional stride, the higher the stride the lower the weight. We'll have to make sure that whatever user configuration scale we provide works well for both. If two PARTIDs have stride 2x and a third x then the 2 PARTIDS with stride 2x together get the same bandidth as the third. Whereas, to get the same in RISC-V the two partid would have weight y and the third 2y.

It's not specified for MPAM exactly what happens when you disable proportional stride for a given PARTID.

The MPAM proportional control is work-conserving (the table in B.b RWKZBJ has been confirmed as a spec mistake) and only corresponds to the current contenders for bandwidth. From my reading of the CBQRI spec this is the same for RISC-V.

Thanks,

Ben

Quoted 57 lines

Here is the patch I created to add Mweight support:

fs/resctrl/ctrlmondata.c +1 -0
index d95ab8ad36e2..3537071e3ab0 100644
@@ -304,6 +304,7 @@ static const char * const resctrl_ctrl_name[] = {
 	[RESCTRL_CTRL_NAME_DEF]		= "",
 	[RESCTRL_CTRL_NAME_MIN]		= "MIN",
 	[RESCTRL_CTRL_NAME_MAX]		= "MAX",
+	[RESCTRL_CTRL_NAME_WGHT]	= "WGHT",
 };
 
 const char *resctrl_ctrl_name_str(enum resctrl_ctrl_name name)
include/linux/resctrl.h +3 -1
index 72fb7256270e..09efcef9ce66 100644
@@ -348,12 +348,14 @@ struct resctrl_mon {
  *				has the same name as the resource.
  * @RESCTRL_CTRL_NAME_MIN:	"MIN"
  * @RESCTRL_CTRL_NAME_MAX:	"MAX"
+ * @RESCTRL_CTRL_NAME_WGHT:	"WGHT"
  */
 enum resctrl_ctrl_name {
 	RESCTRL_CTRL_NAME_DEF,
 	RESCTRL_CTRL_NAME_MIN,
 	RESCTRL_CTRL_NAME_MAX,
-	RESCTRL_CTRL_NAME_LAST = RESCTRL_CTRL_NAME_MAX
+	RESCTRL_CTRL_NAME_WGHT,
+	RESCTRL_CTRL_NAME_LAST = RESCTRL_CTRL_NAME_WGHT
 };
 
Quoted 7 lines
Quoted 3 lines

- Controls are independent for now. This means that, for example, if a resource supports a "MIN" and "MAX" control then this implementation would allow user to set the "maximum" control values to be less than the "minimum" control values.

I think this is ok as long as adding support for new controls in resctrl doesn't change the existing behaviour. In MPAM we dodged this by introducing MB as only affecting the h/w mbw_max and not mbw_min (as mentioned above).

There is no equivalent to MB (percentage throttle) in RISC-V so I would want it to be valid to have MB_MIN (minimum reservation) without MB.

I rebased my RISC-V CBQRI v6 series on top of this proof of concept and was able to validate it works okay in Qemu:

MB_WGHT:72=255
MB_MIN:72=756

     L2:64=fff;65=fff
     L3:75=ffff

Thanks,
Drew

[1] https://lore.kernel.org/all/[email protected]/

#15 lore raw
Quoted 19 lines · Ben Horgan
Quoted 11 lines

The facility for non-percentage value is import for RISC-V as CBQRI does not include percentage throttle. It has two controls for bandwidth:

- Rbwb: number of reserved bandwidth blocks [1, 2^13]
- Mweight: weighted share of the remaining bandwidth [0, 255]
- 0: disables work-conserving sharing
- 1..255: compete for the leftover pool
- It makes for it to default to max (255) so that there won't be any unused bandwidth

I think Mweight could be aligned with MPAM's proportional stride.

Yes, I hope so. There a few differences which would have to be considered.

MPAM doesn't have a concept of only applying the weights once reserved min bandwidth is consumed. The interaction with min bandwidth is currently unspecified. I don't think there are any designs where proportional bandwidth and min b/w are on the same component and so it's only a theoretical/future problem.

Quoted 5 lines

For MPAM proportional stride, the higher the stride the lower the weight. We'll have to make sure that whatever user configuration scale we provide works well for both. If two PARTIDs have stride 2x and a third x then the 2 PARTIDS with stride 2x together get the same bandidth as the third. Whereas, to get the same in RISC-V the two partid would have weight y and the third 2y.

Thanks for explaining. I had been thinking it would be best to try to share a control for stride and weight, but now I am wondering if those should just be considered separate controls.

It seems like they are similar enough to be able to convert to a common scale but maybe that would be too confusing for the user. I guess it is a matter of how much userspace needs or wants to be aware of the difference between systems with MPAM and CBQRI.

Alternatively, it just occurred to me that Mweight could be mapped to MB. I think Mweight could be thought of in the context of throttling: all groups start with the max of 255 which can be represented as 100%.

Quoted 6 lines

It's not specified for MPAM exactly what happens when you disable proportional stride for a given PARTID.

The MPAM proportional control is work-conserving (the table in B.b RWKZBJ has been confirmed as a spec mistake) and only corresponds to the current contenders for bandwidth. From my reading of the CBQRI spec this is the same for RISC-V.

Yes, I think they are the same. The only exception is that Mweight of 0 means no shared badwidth and that the group is restricted to just its reserved bandwidth blocks (e.g. MB_MIN).

Thanks,
Drew

#16 lore raw

Hello Reinette, Ben, Drew,

Quoted 14 lines · Reinette Chatre
Quoted 13 lines
Quoted 7 lines

The ability to change scope is much needed for RISC-V. There are compromises in my RFC [1] as a result of trying to map everything to either L2 or L3 scope.

I would also like to see a non-cpu cache scope for monitoring too, but would that be better discussed outside the context of this proof of concept?

I also think it would be good for it to be clear that monitoring is based on
scope, not a resource. With the MB controls supporting different scope I do think that this would be a good next step. A previous musing from me on this topic can
be found (at the end of ) https://lore.kernel.org/lkml/[email protected]/

Quoted 6 lines
Quoted 1 line

I have not yet considered how this can be built on top of this PoC though.

Thanks for explaining. I like how how you show an example of mon_data/mon_NODE_00/mbm_total_bytes in that thread. I believe that sort of scheme would work well for RISc-V as a bandwidth controller implementing the CBQRI spec can be located anywhere within the system.

I have a few questions regarding the scope parameter and the name "mbm_total_bytes".

First, concerning the scope parameter, does the "NODE" specified in "scope" refer to a NUMA node? If so, wouldn't using "NUMA" directly be more explicit and user-friendly?
Could you please explain why "NODE" is used instead of "NUMA"?

Second, regarding the naming of "mbm_total_bytes",
the meaning of this name seems to differ from what is typically found under mon_data/mon_L3_<id>/mbm_total_bytes. To avoid confusion and better reflect its different nature,
how about considering an alternative name such as mbm_global_bytes or another more appropriate identifier.

Best regards,
Shaopeng TAN

author maintainer
#17 lore raw

Hi Shaopeng,

Quoted 18 lines · Shaopeng Tan (Fujitsu)

Hello Reinette, Ben, Drew,

Quoted 16 lines · Reinette Chatre
Quoted 14 lines
Quoted 7 lines

The ability to change scope is much needed for RISC-V. There are compromises in my RFC [1] as a result of trying to map everything to either L2 or L3 scope.

I would also like to see a non-cpu cache scope for monitoring too, but would that be better discussed outside the context of this proof of concept?

I also think it would be good for it to be clear that monitoring is based on
scope, not a resource. With the MB controls supporting different scope I do think that this would be a good next step. A previous musing from me on this topic can
be found (at the end of ) https://lore.kernel.org/lkml/[email protected]/

I have not yet considered how this can be built on top of this PoC though.

Quoted 8 lines
Quoted 4 lines

Thanks for explaining. I like how how you show an example of mon_data/mon_NODE_00/mbm_total_bytes in that thread. I believe that sort of scheme would work well for RISc-V as a bandwidth controller implementing the CBQRI spec can be located anywhere within the system.

I have a few questions regarding the scope parameter and the name "mbm_total_bytes".

First, concerning the scope parameter, does the "NODE" specified in "scope" refer to a NUMA node?

Yes.

Quoted 2 lines

If so, wouldn't using "NUMA" directly be more explicit and user-friendly? Could you please explain why "NODE" is used instead of "NUMA"?

By using "node" the idea is to create an interface that is familiar to user space. Since /sys/devices/system/node already exists to expose the NUMA layout to user space I expect that using similar terminology would make the resctrl interface easier to use since it would be clear that a "node" in /sys/devices/system/node would have a matching "node" in resctrl.

Quoted 3 lines

Second, regarding the naming of "mbm_total_bytes",
the meaning of this name seems to differ from what is typically found under mon_data/mon_L3_<id>/mbm_total_bytes.

Could you please elaborate how the meaning is different? The only information I have about the systems needing this is in Nvidia's portion of
https://lpc.events/event/19/contributions/2093/attachments/1958/4172/resctrl%20Microconference%20LPC%202025%20Tokyo.pdf

Quoted 2 lines

To avoid confusion and better reflect its different nature,
how about considering an alternative name such as mbm_global_bytes or another more appropriate identifier.

Reinette

#18 lore raw

Hello Reinette,

Quoted 39 lines · Shaopeng Tan (Fujitsu)
Quoted 18 lines

Hello Reinette, Ben, Drew,

Quoted 16 lines · Reinette Chatre
Quoted 14 lines
Quoted 7 lines

The ability to change scope is much needed for RISC-V. There are compromises in my RFC [1] as a result of trying to map everything to either L2 or L3 scope.

I would also like to see a non-cpu cache scope for monitoring too, but would that be better discussed outside the context of this proof of concept?

I also think it would be good for it to be clear that monitoring is based on
scope, not a resource. With the MB controls supporting different scope I do think that this would be a good next step. A previous musing from me on this topic can
be found (at the end of ) https://lore.kernel.org/lkml/[email protected]/

I have not yet considered how this can be built on top of this PoC though.

Quoted 8 lines
Quoted 4 lines

Thanks for explaining. I like how how you show an example of mon_data/mon_NODE_00/mbm_total_bytes in that thread. I believe that sort of scheme would work well for RISc-V as a bandwidth controller implementing the CBQRI spec can be located anywhere within the system.

I have a few questions regarding the scope parameter and the name "mbm_total_bytes".

First, concerning the scope parameter, does the "NODE" specified in "scope" refer to a NUMA node?

Yes.

Quoted 2 lines

If so, wouldn't using "NUMA" directly be more explicit and user-friendly? Could you please explain why "NODE" is used instead of "NUMA"?

By using "node" the idea is to create an interface that is familiar to user space. Since /sys/devices/system/node already exists to expose the NUMA layout to user space I expect that using similar terminology would make the resctrl interface easier to use since it would be clear that a "node" in /sys/devices/system/node would have a matching "node" in resctrl.

Quoted 1 line
Quoted 9 lines
Quoted 2 lines

Second, regarding the naming of "mbm_total_bytes",
the meaning of this name seems to differ from what is typically found under mon_data/mon_L3_<id>/mbm_total_bytes.

Could you please elaborate how the meaning is different? The only information I have about the systems needing this is in Nvidia's portion of
https://lpc.events/event/19/contributions/2093/attachments/1958/4172/resctrl%20Microconference%20LPC%202025%20Tokyo.pdf

Quoted 2 lines

To avoid confusion and better reflect its different nature,
how about considering an alternative name such as mbm_global_bytes or another more appropriate identifier.

Current MBM events in resctrl are per L3 to monitor MB between L3 and memory,
and mbm_total_bytes monitors the L3 total external bandwidth to the next level of the memory hierarchy. Is my understanding of this correct?

Based on NVIDIA's information, they even assume that CPU-less nodes will not feature an L3 cache. I believe NVIDIA's objective is to collect traffic data for each NODE/NUMA, without needing to distinguish which specific L3 cache the traffic originated from.

This aligns with the fact that, in ARM architectures (e.g., on NVIDIA's platforms or Fujitsu's MONAKA), the MPAM MSC (Memory System Component) can also be located on the memory controller. In such cases,it becomes impossible to distinguish which L3 cache the traffic came from.

Therefore, the proposed 'mbm_global_bytes' refers to account for traffic from all L3 cache within the global system.

Best regards
Shaopeng TAN

author maintainer
#19 lore raw

Hi Shaopeng Tan,

Quoted 10 lines · Shaopeng Tan (Fujitsu)
Quoted 10 lines · Shaopeng Tan (Fujitsu)
Quoted 3 lines

Second, regarding the naming of "mbm_total_bytes",
the meaning of this name seems to differ from what is typically found under mon_data/mon_L3_<id>/mbm_total_bytes.

Could you please elaborate how the meaning is different? The only information I have about the systems needing this is in Nvidia's portion of
https://lpc.events/event/19/contributions/2093/attachments/1958/4172/resctrl%20Microconference%20LPC%202025%20Tokyo.pdf

Quoted 6 lines
Quoted 2 lines
Quoted 2 lines

To avoid confusion and better reflect its different nature,
how about considering an alternative name such as mbm_global_bytes or another more appropriate identifier.

Current MBM events in resctrl are per L3 to monitor MB between L3 and memory,
and mbm_total_bytes monitors the L3 total external bandwidth to the next level of the memory hierarchy. Is my understanding of this correct?

It is my understanding also.

Quoted 4 lines

Based on NVIDIA's information, they even assume that CPU-less nodes will not feature an L3 cache. I believe NVIDIA's objective is to collect traffic data for each NODE/NUMA, without needing to distinguish which specific L3 cache the traffic originated from.

That is what associating monitoring with scope (instead of L3 resource), and supporting node scope, aims to address.

Quoted 4 lines

This aligns with the fact that, in ARM architectures (e.g., on NVIDIA's platforms or Fujitsu's MONAKA), the MPAM MSC (Memory System Component) can also be located on the memory controller. In such cases,it becomes impossible to distinguish which L3 cache the traffic came from.

This is not clear to me. Is the goal to expose the bandwidth at each memory controller?

Quoted 3 lines

Therefore, the proposed 'mbm_global_bytes' refers to account for traffic from all L3 cache within the global system.

As I see it the directory name would make the monitoring scope clear, whether it is at L3 scope or NUMA node scope. Within the directory there would be the individual files that should be interpreted based on the scope of the directory they are in. I thus see a "mbm_total_bytes" in a "L3 scope" directory to have different meaning from a "mbm_total_bytes" in a "NUMA node scope" directory.

Reinette

#20 lore raw

Hello Reinette,

Quoted 25 lines
Quoted 7 lines
Quoted 7 lines
Quoted 2 lines

Second, regarding the naming of "mbm_total_bytes",
the meaning of this name seems to differ from what is typically found under mon_data/mon_L3_<id>/mbm_total_bytes.

Could you please elaborate how the meaning is different? The only information I have about the systems needing this is in Nvidia's portion of
https://lpc.events/event/19/contributions/2093/attachments/1958/4172/resctrl%20Microconference%20LPC%202025%20Tokyo.pdf

Quoted 6 lines
Quoted 2 lines
Quoted 2 lines

To avoid confusion and better reflect its different nature,
how about considering an alternative name such as mbm_global_bytes or another more appropriate identifier.

Current MBM events in resctrl are per L3 to monitor MB between L3 and memory,
and mbm_total_bytes monitors the L3 total external bandwidth to the next level of the memory hierarchy. Is my understanding of this correct?

It is my understanding also.

Quoted 4 lines

Based on NVIDIA's information, they even assume that CPU-less nodes will not feature an L3 cache. I believe NVIDIA's objective is to collect traffic data for each NODE/NUMA, without needing to distinguish which specific L3 cache the traffic originated from.

That is what associating monitoring with scope (instead of L3 resource), and supporting node scope, aims to address.

Quoted 1 line
Quoted 5 lines
Quoted 3 lines

This aligns with the fact that, in ARM architectures (e.g., on NVIDIA's platforms or Fujitsu's MONAKA), the MPAM MSC (Memory System Component) can also be located on the memory controller. In such cases,it becomes impossible to distinguish which L3 cache the traffic came from.

This is not clear to me. Is the goal to expose the bandwidth at each memory controller?

The goal is to expose the bandwidth of each NUMA/NODE.
In this context, my intention was to explain why it is impossible to distinguish which L3 cache the traffic came from.

Quoted 9 lines
Quoted 3 lines

Therefore, the proposed 'mbm_global_bytes' refers to account for traffic from all L3 cache within the global system.

As I see it the directory name would make the monitoring scope clear, whether it is at L3 scope or NUMA node scope. Within the directory there would be the individual files that should be interpreted based on the scope of the directory they are in. I thus see a "mbm_total_bytes" in a "L3 scope" directory to have different meaning from a "mbm_total_bytes" in a "NUMA node scope" directory.

I understand.
Your suggestion is to clarify the monitoring scope (L3 or NUMA node) by the directory name, and that individual files within that directory would be interpreted based on its scope. If the scope is "L3", then "mbm_total_bytes" would mean collecting traffic data for each L3, regardless of which NUMA node the traffic is destined for.
If the scope is "NUMA node", then "mbm_total_bytes" would mean collecting traffic data for each node/NUMA, regardless of which L3 cache the traffic originates from.
I believe this approach will be clear as long as it's clearly documented. Thanks.

Also, regarding the scope for NUMA nodes,
since it's not possible to distinguish which L3 cache the traffic came from, it also appears that support for event_configs is not necessary. (There is no need to configure "non-local NUMA" or "local NUMA".)

Best regards,
Shaopeng TAN

author maintainer
#21 lore raw
In reply to Drew Fustini

Hi Drew,

Quoted 15 lines · Drew Fustini
Quoted 14 lines · Ben Horgan

Hi Reinette,

Quoted 10 lines · Reinette Chatre

Hi Everybody,

It has been a while since we discussed the resctrl changes required to support hardware that has controls with fine granularity or hardware that has multiple controls per resource. For reference, the most recent email discussion can be found at [1] with a summary of discussions in last year's plumbers slides [2].

I created a PoC that I believe supports what folks have agreed to so far. I hope this can help us to restart the discussion with the goal that resctrl gains support for upcoming hardware that require these features.

Quoted 6 lines
Quoted 2 lines

Thank you very much for doing this work. I believe this will be very useful for MPAM and other architectures.

Yes, thanks to Reinette for working on the generic schema proof of concept. This will be helpful for supporting the RISC-V CBQRI (capacity and bandwidth QoS) spec.

Thank you very much for considering this work.

Quoted 26 lines
Quoted 18 lines

I plumbed in support for the MB_MIN resource schema which also works under light testing. The only fs resctrl code change I needed was:

include/linux/resctrl.h +3 -0
@@ -483,6 +483,9 @@ static inline u32 resctrl_get_default_ctrlval(struct
 resctrl_ctrl *ctrl)
        case RESCTRL_CTRL_BITMAP:
                return BIT_MASK(ctrl->cache.cbm_len) - 1;
        case RESCTRL_CTRL_SCALAR:
+               if (ctrl->name == RESCTRL_CTRL_NAME_MIN)
+                       return ctrl->membw.min_bw;
+
                return ctrl->membw.max_bw;
        }
Quoted 5 lines

At least on MPAM systems, we use a default of 0 for minimum bandwidth controls as the maximum bandwidth controls only take effect if their value is higher than the minimum bandwidth value. I have specialised this on the ctrl->name which breaks your ctrl->type based classification but that's fixable by just adding a default field to membw.

This should be useful for RISC-V.

Quoted 6 lines

RESCTRL_CTRL_NAME_MIN maps well to CBQRI Rbwb (reserved bandwidth blocks). The sum of Rbwb across all control groups must be less than MRBWB (maximum number of reserved bandwidth blocks). As a result, MB_MIN needs to default to 1 so that the sum does not violate that rule. In my RFC series, I added default_to_min to resctrl_membw [1] but this solution looks cleaner.

As I mentioned in response to Ben [2] there seems to be a mismatch between architecture requirements here. resctrl uses the value returned by resctrl_get_default_ctrlval() as the control value that means "no throttling". For Intel this means min == max but this does not seem to be the case for MPAM and CBQRI. I am not familiar enough with either to have an alternative proposal here so I need to become familiar now. There is a bit of backlog on other resctl work right now so this will take me some time to sort out.

Quoted 81 lines
Quoted 13 lines
Quoted 7 lines

- No support for "read-modify-write" usage of schemata file. This is where we discussed (without agreement) on possibly introducing the "#" prefix to schemata file entries. This PoC does not support this prefix and the current assumption/expectation is that when user space changes a configuration only the new control values are written to schemata file. I thus do not have a plan to support this so please share opinions in this regard if you have some.

There is now less motivation from the MPAM side for this than when this was initially discussed. In pre-upstream versions of the MPAM patches a change in the MB resource control value would change both the mpam h/w mbw_min and mbw_max values but now (on non-broken h/w) we just change the mbw_max. (mbw_min kept at 0).

Quoted 11 lines

However, it would be useful not to be limited by percentages. In my quick experimentation with your patches I used a percentage value for MB_MIN but it would be best to move away from this. For new controls I think we can mandate that user space has to discover the resolution from the info directly but how can we retrofit this. For MPAM, MB and MB_MAX, would control the same things. Could we just add MB_MAX with a h/w friendly scale and then reflect changes in MB_MAX in MB and vica versa with MB taking precedent if both are set? Old software can continue setting MB can move to using MB_MAX and take advantage of the improved control. (I don't think we should expose the MPAM hardware value directly as it has confusion over whether all 1s is 100% or not and we'd like to have something generic and friendly to the user.)

The facility for non-percentage value is import for RISC-V as CBQRI does not include percentage throttle. It has two controls for bandwidth:

- Rbwb: number of reserved bandwidth blocks [1, 2^13]
- Mweight: weighted share of the remaining bandwidth [0, 255]
- 0: disables work-conserving sharing
- 1..255: compete for the leftover pool
- It makes for it to default to max (255) so that there won't be any unused bandwidth

I think Mweight could be aligned with MPAM's proportional stride.

Here is the patch I created to add Mweight support:

fs/resctrl/ctrlmondata.c +1 -0
index d95ab8ad36e2..3537071e3ab0 100644
@@ -304,6 +304,7 @@ static const char * const resctrl_ctrl_name[] = {
 	[RESCTRL_CTRL_NAME_DEF]		= "",
 	[RESCTRL_CTRL_NAME_MIN]		= "MIN",
 	[RESCTRL_CTRL_NAME_MAX]		= "MAX",
+	[RESCTRL_CTRL_NAME_WGHT]	= "WGHT",
 };
 
 const char *resctrl_ctrl_name_str(enum resctrl_ctrl_name name)
include/linux/resctrl.h +3 -1
index 72fb7256270e..09efcef9ce66 100644
@@ -348,12 +348,14 @@ struct resctrl_mon {
  *				has the same name as the resource.
  * @RESCTRL_CTRL_NAME_MIN:	"MIN"
  * @RESCTRL_CTRL_NAME_MAX:	"MAX"
+ * @RESCTRL_CTRL_NAME_WGHT:	"WGHT"
  */
 enum resctrl_ctrl_name {
 	RESCTRL_CTRL_NAME_DEF,
 	RESCTRL_CTRL_NAME_MIN,
 	RESCTRL_CTRL_NAME_MAX,
-	RESCTRL_CTRL_NAME_LAST = RESCTRL_CTRL_NAME_MAX
+	RESCTRL_CTRL_NAME_WGHT,
+	RESCTRL_CTRL_NAME_LAST = RESCTRL_CTRL_NAME_WGHT
 };
 
Quoted 7 lines
Quoted 3 lines

- Controls are independent for now. This means that, for example, if a resource supports a "MIN" and "MAX" control then this implementation would allow user to set the "maximum" control values to be less than the "minimum" control values.

I think this is ok as long as adding support for new controls in resctrl doesn't change the existing behaviour. In MPAM we dodged this by introducing MB as only affecting the h/w mbw_max and not mbw_min (as mentioned above).

There is no equivalent to MB (percentage throttle) in RISC-V so I would want it to be valid to have MB_MIN (minimum reservation) without MB.

Quoted 7 lines

I rebased my RISC-V CBQRI v6 series on top of this proof of concept and was able to validate it works okay in Qemu:

MB_WGHT:72=255
MB_MIN:72=756

     L2:64=fff;65=fff
     L3:75=ffff

Ideally any new support should not break existing user space and the existing user interface expects a MB entry in the schemata file when the MB resource exists. Is it possible to emulate the percentage based MB control with MB_WGHT or MB_MIN? This sounds similar as what is/was planned for MPAM [2].

Something that may be of interest is a proposal that Chenyu is refining to address an issue with the region-aware MBA support where there is no intuitive backward compatible interface. This was highlighted in the plumbers slides (see slide titled "Open: maintaining backward compatibility when region aware"). The current idea to deal with this is to introduce a "mode" associated with the resource controls. For example,

# cat /sys/fs/resctrl/info/MB/resource_schemata/mode 
[legacy] native

By default the "legacy" mode will be enabled and exposes the "MB" default control to user space via the schemata file. In support of this each new control has a new property file named "status" that can have value "enabled" or "disabled". Only "enabled" controls are present in the schemata file but all controls are always present in the resource_schemata directory. By writing to the "mode" file user space acknowledges familiarity with the new "resource_schemata" based interface and can change the status of a control and thus manage its visibility in the schemata file.
Could something like this work for CBQRI?

Reinette

Quoted 1 line

[1] https://lore.kernel.org/all/[email protected]/

[2] https://lore.kernel.org/lkml/[email protected]/

#22 lore raw
Quoted 32 lines · Reinette Chatre
Quoted 25 lines
Quoted 17 lines

I plumbed in support for the MB_MIN resource schema which also works under light testing. The only fs resctrl code change I needed was:

include/linux/resctrl.h +3 -0
@@ -483,6 +483,9 @@ static inline u32 resctrl_get_default_ctrlval(struct
 resctrl_ctrl *ctrl)
        case RESCTRL_CTRL_BITMAP:
                return BIT_MASK(ctrl->cache.cbm_len) - 1;
        case RESCTRL_CTRL_SCALAR:
+               if (ctrl->name == RESCTRL_CTRL_NAME_MIN)
+                       return ctrl->membw.min_bw;
+
                return ctrl->membw.max_bw;
        }
Quoted 5 lines

At least on MPAM systems, we use a default of 0 for minimum bandwidth controls as the maximum bandwidth controls only take effect if their value is higher than the minimum bandwidth value. I have specialised this on the ctrl->name which breaks your ctrl->type based classification but that's fixable by just adding a default field to membw.

This should be useful for RISC-V.

Quoted 6 lines

RESCTRL_CTRL_NAME_MIN maps well to CBQRI Rbwb (reserved bandwidth blocks). The sum of Rbwb across all control groups must be less than MRBWB (maximum number of reserved bandwidth blocks). As a result, MB_MIN needs to default to 1 so that the sum does not violate that rule. In my RFC series, I added default_to_min to resctrl_membw [1] but this solution looks cleaner.

Quoted 7 lines

As I mentioned in response to Ben [2] there seems to be a mismatch between architecture requirements here. resctrl uses the value returned by resctrl_get_default_ctrlval() as the control value that means "no throttling". For Intel this means min == max but this does not seem to be the case for MPAM and CBQRI. I am not familiar enough with either to have an alternative proposal here so I need to become familiar now. There is a bit of backlog on other resctl work right now so this will take me some time to sort out.

Thanks for pointing this out. In that case, it doesn't seem to match what I was thinking of for MB_MIN. The CBQRI reserved bandwidth blocks Rbwb) control can be thought of as a minimum amount of guranteed bandwidth for a control group. Each RCID (e.g. CLOSID) must be assigned at least 1 bandwidth block per the spec. Therefore, the membw.min_bw would need to be 1.

There is also a max bandwidth reservation across all control groups (RCIDs / CLOSIDs) so that there will be some amount of unreserved bandwidth. Mweight (1-255) controls how much of that unreserved bandwidth pool that a group can use. Mweight of 0 means no shared bandwidth. I think the membw.min_bw would need to 255 so that all groups get equal share of the unreserved pool.

It seems like that would be incorrect use of membw.min_bw in both cases?

Quoted 11 lines
Quoted 10 lines

There is no equivalent to MB (percentage throttle) in RISC-V so I would want it to be valid to have MB_MIN (minimum reservation) without MB.

I rebased my RISC-V CBQRI v6 series on top of this proof of concept and was able to validate it works okay in Qemu:

MB_WGHT:72=255
MB_MIN:72=756

     L2:64=fff;65=fff
     L3:75=ffff
Quoted 4 lines

Ideally any new support should not break existing user space and the existing user interface expects a MB entry in the schemata file when the MB resource exists. Is it possible to emulate the percentage based MB control with MB_WGHT or MB_MIN? This sounds similar as what is/was planned for MPAM [2].

Yes, I think that Mweight could be mapped to the MB concept of throttling. All groups could start with the max Mweight of 255 which could can be represented as 100%.

However, I'm not sure what to do about membw.min_bw. Mweight = 0 means it can not use any of the shared unreserved bandwidth pool. If resctrl_get_default_ctrlval() is designed to mean "no throttling", then it seems like the membw.min_bw would need to be 255. But that feels weird for the min_bw value to be equal to the max weight for unreserved bandwidth.

Quoted 9 lines

Something that may be of interest is a proposal that Chenyu is refining to address an issue with the region-aware MBA support where there is no intuitive backward compatible interface. This was highlighted in the plumbers slides (see slide titled "Open: maintaining backward compatibility when region aware"). The current idea to deal with this is to introduce a "mode" associated with the resource controls. For example,

# cat /sys/fs/resctrl/info/MB/resource_schemata/mode 
[legacy] native
Quoted 8 lines

By default the "legacy" mode will be enabled and exposes the "MB" default control to user space via the schemata file. In support of this each new control has a new property file named "status" that can have value "enabled" or "disabled". Only "enabled" controls are present in the schemata file but all controls are always present in the resource_schemata directory. By writing to the "mode" file user space acknowledges familiarity with the new "resource_schemata" based interface and can change the status of a control and thus manage its visibility in the schemata file.
Could something like this work for CBQRI?

Yes, I think that would work. There are no existing users of resctrl on RISC-V so I think having users opt into this resource_schemata interface would work, especially if that allows a truer represenation of the controls in the CBQRI spec.

Thanks,
Drew

#23 lore raw
Quoted 15 lines · Drew Fustini
Quoted 7 lines

As I mentioned in response to Ben [2] there seems to be a mismatch between architecture requirements here. resctrl uses the value returned by resctrl_get_default_ctrlval() as the control value that means "no throttling". For Intel this means min == max but this does not seem to be the case for MPAM and CBQRI. I am not familiar enough with either to have an alternative proposal here so I need to become familiar now. There is a bit of backlog on other resctl work right now so this will take me some time to sort out.

Thanks for pointing this out. In that case, it doesn't seem to match what I was thinking of for MB_MIN. The CBQRI reserved bandwidth blocks Rbwb) control can be thought of as a minimum amount of guranteed bandwidth for a control group. Each RCID (e.g. CLOSID) must be assigned at least 1 bandwidth block per the spec. Therefore, the membw.min_bw would need to be 1.

Quoted 6 lines

There is also a max bandwidth reservation across all control groups (RCIDs / CLOSIDs) so that there will be some amount of unreserved bandwidth. Mweight (1-255) controls how much of that unreserved bandwidth pool that a group can use. Mweight of 0 means no shared bandwidth. I think the membw.min_bw would need to 255 so that all groups get equal share of the unreserved pool.

Sorry, I wasn't thinking about this right. If Mweight is used for MB, then membw.max_bw would be 100 (MAX_MBA_BW) and membw.min_bw would be 0 which means no shared bandwidth.

Quoted 1 line

It seems like that would be incorrect use of membw.min_bw in both cases?

The issue is really just for Rbwb (reserved bandwidth) as that needs to default to the minimum of 1. What about introducing membw.reset_val which would be returned by resctrl_get_default_ctrl()?

MB could set membw.reset_val to be the same value as membw.max_bw.

Quoted 20 lines
Quoted 11 lines
Quoted 10 lines

There is no equivalent to MB (percentage throttle) in RISC-V so I would want it to be valid to have MB_MIN (minimum reservation) without MB.

I rebased my RISC-V CBQRI v6 series on top of this proof of concept and was able to validate it works okay in Qemu:

MB_WGHT:72=255
MB_MIN:72=756

     L2:64=fff;65=fff
     L3:75=ffff
Quoted 4 lines

Ideally any new support should not break existing user space and the existing user interface expects a MB entry in the schemata file when the MB resource exists. Is it possible to emulate the percentage based MB control with MB_WGHT or MB_MIN? This sounds similar as what is/was planned for MPAM [2].

Yes, I think that Mweight could be mapped to the MB concept of throttling. All groups could start with the max Mweight of 255 which could can be represented as 100%.

Quoted 6 lines

However, I'm not sure what to do about membw.min_bw. Mweight = 0 means it can not use any of the shared unreserved bandwidth pool. If resctrl_get_default_ctrlval() is designed to mean "no throttling", then it seems like the membw.min_bw would need to be 255. But that feels weird for the min_bw value to be equal to the max weight for unreserved bandwidth.

MB would have the typical membw.min_bw = 100, and resctrl_get_default_ctrl() would return 100. The controller would be programmed with Mweight 255 for 100%.

Thanks,
Drew

#24 lore raw
Quoted 62 lines · Drew Fustini
Quoted 15 lines · Drew Fustini
Quoted 7 lines

As I mentioned in response to Ben [2] there seems to be a mismatch between architecture requirements here. resctrl uses the value returned by resctrl_get_default_ctrlval() as the control value that means "no throttling". For Intel this means min == max but this does not seem to be the case for MPAM and CBQRI. I am not familiar enough with either to have an alternative proposal here so I need to become familiar now. There is a bit of backlog on other resctl work right now so this will take me some time to sort out.

Thanks for pointing this out. In that case, it doesn't seem to match what I was thinking of for MB_MIN. The CBQRI reserved bandwidth blocks Rbwb) control can be thought of as a minimum amount of guranteed bandwidth for a control group. Each RCID (e.g. CLOSID) must be assigned at least 1 bandwidth block per the spec. Therefore, the membw.min_bw would need to be 1.

Quoted 6 lines

There is also a max bandwidth reservation across all control groups (RCIDs / CLOSIDs) so that there will be some amount of unreserved bandwidth. Mweight (1-255) controls how much of that unreserved bandwidth pool that a group can use. Mweight of 0 means no shared bandwidth. I think the membw.min_bw would need to 255 so that all groups get equal share of the unreserved pool.

Sorry, I wasn't thinking about this right. If Mweight is used for MB, then membw.max_bw would be 100 (MAX_MBA_BW) and membw.min_bw would be 0 which means no shared bandwidth.

Quoted 1 line

It seems like that would be incorrect use of membw.min_bw in both cases?

The issue is really just for Rbwb (reserved bandwidth) as that needs to default to the minimum of 1. What about introducing membw.reset_val which would be returned by resctrl_get_default_ctrl()?

MB could set membw.reset_val to be the same value as membw.max_bw.

Quoted 20 lines
Quoted 11 lines
Quoted 10 lines

There is no equivalent to MB (percentage throttle) in RISC-V so I would want it to be valid to have MB_MIN (minimum reservation) without MB.

I rebased my RISC-V CBQRI v6 series on top of this proof of concept and was able to validate it works okay in Qemu:

MB_WGHT:72=255
MB_MIN:72=756

     L2:64=fff;65=fff
     L3:75=ffff
Quoted 4 lines

Ideally any new support should not break existing user space and the existing user interface expects a MB entry in the schemata file when the MB resource exists. Is it possible to emulate the percentage based MB control with MB_WGHT or MB_MIN? This sounds similar as what is/was planned for MPAM [2].

Yes, I think that Mweight could be mapped to the MB concept of throttling. All groups could start with the max Mweight of 255 which could can be represented as 100%.

Quoted 6 lines

However, I'm not sure what to do about membw.min_bw. Mweight = 0 means it can not use any of the shared unreserved bandwidth pool. If resctrl_get_default_ctrlval() is designed to mean "no throttling", then it seems like the membw.min_bw would need to be 255. But that feels weird for the min_bw value to be equal to the max weight for unreserved bandwidth.

Quoted 3 lines

MB would have the typical membw.min_bw = 100, and resctrl_get_default_ctrl() would return 100. The controller would be programmed with Mweight 255 for 100%.

Sorry - I meant MB would have membw.max_bw = 100 which would cause CBQRI bandwidth controller to be programmed with Mweight = 255.

Drew

author maintainer
#25 lore raw
In reply to Ben Horgan

Hi Ben,

Quoted 10 lines · Ben Horgan

Hi Reinette,

Quoted 7 lines · Reinette Chatre

Hi Everybody,

It has been a while since we discussed the resctrl changes required to support hardware that has controls with fine granularity or hardware that has multiple controls per resource. For reference, the most recent email discussion can be found at [1] with a summary of discussions in last year's plumbers slides [2].

Quoted 6 lines
Quoted 3 lines

I created a PoC that I believe supports what folks have agreed to so far. I hope this can help us to restart the discussion with the goal that resctrl gains support for upcoming hardware that require these features.

Thank you very much for doing this work. I believe this will be very useful for MPAM and other architectures.

Thank you very much for reviewing this.

Quoted 36 lines
Quoted 15 lines

Request regarding this PoC
==========================

Please consider this PoC as a "direction check" on the schema description and multiple control discussions held thus far.

Could folks working on enabling new hardware requiring this capability please consider if this is something you can build on and how it should be improved to support these upcoming capabilities?

Opens
=====

Quoted 9 lines

While the PoC aims to support what folks agreed on some opens remain:
- I attempted to make some MPAM supporting changes but these are all just compile tested. While MPAM should benefit from the new control properties I did not initialize them on MPAM and did not attempt refactor to separate out the architecture specific control properties (more on what this means later). I did attempt some MPAM refactoring that duplicates the MPAM domain to the control domain and monitoring domain lists in support of there being multiple controls each with its own list of control domains but it is definitely not good design.

I appreciate you including MPAM in this PoC. With this one line change I was able to boot an MPAM system and mount resctrl which appears to behave correctly. We can consider how best to do the code design later.

drivers/resctrl/mpam_resctrl.c +1 -0
@@ -1697,6 +1697,8 @@ int mpam_resctrl_setup(void)
 
        /* Initialise the resctrl structures from the classes */
        for_each_mpam_resctrl_control(res, rid) {
+               INIT_LIST_HEAD(&res->resctrl_res.controls); // list_empty needs
Quoted 5 lines

to work
+
if (!res->class)
continue; // dummy resource

Thank you very much. I picked this up but ended up moving it earlier to be next to the mon_domains list initialization. Doing so made it easier for me to follow the initialization. That ok?

Quoted 17 lines

I plumbed in support for the MB_MIN resource schema which also works under light testing. The only fs resctrl code change I needed was:

include/linux/resctrl.h +3 -0
@@ -483,6 +483,9 @@ static inline u32 resctrl_get_default_ctrlval(struct
 resctrl_ctrl *ctrl)
        case RESCTRL_CTRL_BITMAP:
                return BIT_MASK(ctrl->cache.cbm_len) - 1;
        case RESCTRL_CTRL_SCALAR:
+               if (ctrl->name == RESCTRL_CTRL_NAME_MIN)
+                       return ctrl->membw.min_bw;
+
                return ctrl->membw.max_bw;
        }
Quoted 5 lines

At least on MPAM systems, we use a default of 0 for minimum bandwidth controls as the maximum bandwidth controls only take effect if their value is higher than the minimum bandwidth value. I have specialised this on the ctrl->name which breaks your ctrl->type based classification but that's fixable by just adding a default field to membw.

This I am not sure about. In my understanding a typical "default" value means "no throttling" and, at least on Intel, this default hardware state has been summarized as "min" == "max" == "optimal".

Are you saying that on MPAM systems if "min" == "max" then max bandwidth controls do not take effect? Could you please elaborate what happens if "min" == "max"?

Quoted 9 lines
Quoted 6 lines

- No support for emulated controls (yet). The PoC is quite large already but I think it can be used as a base for emulated controls for which the software controller could be a potential first customer. In this PoC mounting with software controller will still display the original controller's properties.
- One open that needs to be addressed as part of support for emulated controls is how best to display emulation relationship via resctrl hierarchy.

What does emulated controls mean here? Is there some previous discussion you could point me at?

For emulated controls in context of MPAM I think the best reference is
https://lore.kernel.org/lkml/[email protected]/

Above is the email discussion that I attempted to visualize in the middle example in slide 6 ("resctrl controls vs. hardware controls") of
https://lpc.events/event/19/contributions/2093/attachments/1958/4172/resctrl%20Microconference%20LPC%202025%20Tokyo.pdf

When comparing the slide to Dave's text, please replace "MB_HW" from Dave's example with "MB_OPT" in the slide. I changed the name since I found the "HW" in an emulated control to be potentially confusing.

Quoted 11 lines
Quoted 6 lines

- No support for "read-modify-write" usage of schemata file. This is where we discussed (without agreement) on possibly introducing the "#" prefix to schemata file entries. This PoC does not support this prefix and the current assumption/expectation is that when user space changes a configuration only the new control values are written to schemata file. I thus do not have a plan to support this so please share opinions in this regard if you have some.

There is now less motivation from the MPAM side for this than when this was initially discussed. In pre-upstream versions of the MPAM patches a change in the MB resource control value would change both the mpam h/w mbw_min and mbw_max values but now (on non-broken h/w) we just change the mbw_max. (mbw_min kept at 0).

Ah, thanks for the correction. The email I linked above indeed refers to changing both min and max.

Quoted 2 lines

However, it would be useful not to be limited by percentages. In my quick

Indeed. Not being limited by percentages while still needing to have a backward compatible user interface is how we ended up with "emulated controls".

Quoted 10 lines

experimentation with your patches I used a percentage value for MB_MIN but it would be best to move away from this. For new controls I think we can mandate that user space has to discover the resolution from the info directly but how can we retrofit this. For MPAM, MB and MB_MAX, would control the same things. Could we just add MB_MAX with a h/w friendly scale and then reflect changes in MB_MAX in MB and vica versa with MB taking precedent if both are set? Old software can continue setting MB can move to using MB_MAX and take advantage of the improved control. (I don't think we should expose the MPAM hardware value directly as it has confusion over whether all 1s is 100% or not and we'd like to have something generic and friendly to the user.)

Sounds to me as though you are describing emulated controls. Exposing two controls in schemata file that essentially controls the same thing is what the emulated controls aim to solve and the resctrl hierarchies presented in slide #6 of that presentation (and discussed in the email thread) is how we contemplated how to represent the relationship among these controls to user space. So, considering your example resctrl may display something like:

info//

└── MB/
    └── resource_schemata/
        └── MB/
            └── MB_MAX/

Above hierarchy describes the relationship to user space that if MB is changed it will impact MB_MAX and vice-versa.

The one open I am aware of surrounding emulated controls is how to present some semblance of consistency to user space when considering all the possibilities the different architectures (and even within architectures) may have.

Quoted 7 lines
Quoted 3 lines

- Controls are independent for now. This means that, for example, if a resource supports a "MIN" and "MAX" control then this implementation would allow user to set the "maximum" control values to be less than the "minimum" control values.

I think this is ok as long as adding support for new controls in resctrl doesn't change the existing behaviour. In MPAM we dodged this by introducing MB as only affecting the h/w mbw_max and not mbw_min (as mentioned above).

I understand this to be a requirement for Intel where the spec contains "The Maximum Cap should be programmed to be greater than or equal to the Minimum and Optimal caps. Undesirable and undefined performance effects may result if cap programming guidelines are not followed."

I am currently thinking that resctrl should not try to be too smart here and if user space wants to make dramatic changes to min and max values then it should just ensure the ordering is appropriate. For example, attempting to set a new min to be larger than the old max would fail and user space should first increase the old max and then set a new min.

Quoted 80 lines
Quoted 69 lines

- PoC supports the "bitmap" control but does not (yet) expose properties of a bitmap control to the new info/<resource>/resource_schemata directory.

Accessing PoC
=============

Please consider the PoC as a rough draft. It has only been compile tested for Arm and known to be incomplete in Arm support. To help with experimenting I only fully adapted the Intel MBA resource to demo two dummy additional MBA controls. All architectures should immediately benefit from the new schema descriptions and new info/MB/resource_schemata hierarchy.

I considered the patches self too many for email. Instead, the PoC can be found at:

git://git.kernel.org/pub/scm/linux/kernel/git/reinette/linux.git branch resctrl/controls_rfc_v1

The work is based on v7.1-rc2 that also includes the following series (two of which has since been queued) included:

"selftests/resctrl: Fixes and improvements focused on Intel platforms"
https://lore.kernel.org/lkml/[email protected]/

"x86,fs/resctrl: Improve resctrl quality and consistency"
https://lore.kernel.org/lkml/[email protected]/

"x86,fs/resctrl: Pave the way for MPAM counter assignment"
https://lore.kernel.org/lkml/[email protected]/

Primary resctrl fs data structure changes
=========================================

Introduces a control represented by struct resctrl_ctrl that looks as below. To make the changes easier to follow I kept some of the original names to help communicate where familiar data structures land.

What to notice about a control is that it has some common properties required from all controls (scope, type, etc.) and then depending on the type of control (RESCTRL_CTRL_BITMAP or RESCTRL_CTRL_SCALAR) there are type specific properties.

/**
* struct resctrl_ctrl - A resource control
* @entry: List entry of rdt_resource::controls
* @scope: Scope of the resource that this control allocates
* @domains: RCU list of all control domains
* @type: The control type that determines the properties of the control,
* format string for displaying control values to user space, and
* parser of control values provided by user space.
* @name: Name of the control. Appended to final resource name
* (rdt_resource_final::name) to create final schema entry.
* Specifically, "rdt_resource_final::name"_"resctrl_ctrl::name".
* For example, with resource name "MB" and control name "MAX" the
* schema entry will be "MB_MAX".
* @cache: Cache allocation control properties.
* @membw: Bandwidth control properties.
*/

struct resctrl_ctrl {
	struct list_head	entry;
	enum resctrl_scope	scope;
	struct list_head	domains;
	enum resctrl_ctrl_type	type;
	enum resctrl_ctrl_name	name;
	union {
		struct resctrl_cache	cache;
		struct resctrl_membw	membw;
	};
};
Quoted 10 lines

Two members summarize how this new structure fits into the rest of resctrl:
a) resctrl_ctrl::entry
Since a resource can support multiple controls there is a new list in struct rdt_resource named "controls" that contains the list of all controls supported by the resource.
b) resctrl_ctrl::domains
Instead of the list of control domains belonging to a resource they now belong to the control self. By doing so resctrl can support resource controls at different scope for the same resource. This is intended to support some upcoming MPAM and RISC-V usages.

Quoted 7 lines

Please can you expand a bit on part b).

In an MPAM system we consider 3 resctrl resources, RDT_RESOURCE_L3, RDT_RESOURCE_L2 and RDT_RESOURCE_MBA which correspond to the L3 caches, L2 caches and memory bandwidth on egress from the L3 caches. The domain for each of these corresponds to the instance of the resource. That is, for RDT_RESOURCE_L2 there is a resource for each L2 instance, similarly for L3, and for

(I'm assuming above is typo and it is "there is a domain for each L2 instance"?)

Quoted 12 lines

RDT_RESOURCE_MBA there is a domain for each L3 cache. If we were to add suport for controls on a new cache level, say the L4, then I'd expect to add a new resource. For memory bandwidth, we'd like to be able to control b/w on the L2 egress (e.g. in a DSU). Wouldn't this too be a separate resource or would this be a new set of controls on the same resource?

New controls on the same resource
MB_MIN2
MB_MAX2
MB_PROP2
...

Quoted 4 lines

or
MB2_MIN
MB2_MAX
MB2_PROP

The way I currently see it is that controlling bandwidth at a different scope would be a new set of controls associated with the MB resource. There are more scenarios coming this way with AMD's "Global MBA" that is memory bandwidth allocation at NUMA node scope. If I understand correctly the "CPU-less Memory Node" that Nvidia shared at plumbers would need this also and control memory bandwidth allocation at the NUMA node scope. A related technology is Intel's region-aware MBA, which is still at L3 scope.

I fully agree that we need to figure out how to represent all of this to user space without turning the interface into something unintelligible. In the end this is required for user space to know what a domain ID represents.

Would it help to make the scope part of the control name? The ship has sailed for MB being associated with L3 scope but this could mean the "default" scope of MB resource is L3 (which user space can still confirm by looking at the control's "scope" file) and the others include scope in the name? Consider for example:
https://lore.kernel.org/lkml/[email protected]/

Quoted 3 lines

AFAIK, the DSU h/w just supports proportional bandwidth controls at the moment but we should consider what to do about the potential naming.

ack.

Quoted 8 lines

In the MPAM driver, we collect MSC into components (based on instances) and those into classes (components of the same type). Currently, a resource is mapped to a single class. (Two resources may map to the same class.)

I expect it is useful in the memory region and sub numa cases but I'd still expect the common case to be that the domains are the same within a control. Or am I missing something?

Domains of a control should all be at the same scope. Since the schemata file exposes the control with the different IDs representing the instances of the resource needing to be controlled it has to be clear to user space what the domain ID represents.

Quoted 34 lines
Quoted 26 lines

Example architectural data structure changes
============================================

An architecture can use the new control by following a similar pattern to resource and domain use by architectures. Consider the following for x86 where a new architecture specific struct resctrl_hw_ctrl includes struct resctrl_ctrl and any architecture private data needed to support the control:

/*
* struct resctrl_hw_ctrl - Arch private properties of a resource control
* @r_ctrl: Control properties exposed to resctrl file system
* @msr_base: Base MSR address where control values should be programmed
* @msr_update: Function pointer to update control values
*/

struct resctrl_hw_ctrl {
	struct resctrl_ctrl	r_ctrl;
	unsigned int		msr_base;
	void			(*msr_update)(struct msr_param *m);
};

Structure of patch series
=========================

Quoted 7 lines

As a PoC the series is not perfectly structured but to help navigate this work on a high level the changes can be categorized as follows:

Patch 1 to 11:
With a vision of what a "control" is, remove unused/unnecessary members, make clear what is a *resource* property vs a *control* property, do some renaming to help with the PoC.

Quoted 5 lines

A few of the changes are generic cleanup and could hopefully be dealt with before decisions on the larger PoC are made. I see:
fs/resctrl: Remove unused resctrl_membw::mb_map x86,fs/resctrl: Remove "arch_needs_linear"
Perhaps a few more.

ack.

Quoted 11 lines
Quoted 10 lines

Patch 12:
Introduce struct resctrl_ctrl and re-arrange existing struct rdt_resource members to form part of new rdt_resource::ctrl

Patch 13 to 44:
A lot of wrangling to introduce struct resctrl_ctrl to all code that needs to work with a control and/or domain without assuming that the control is the one and only control embedded in the resource it belongs to. Essentially, a lot of changes passing the control around in addition to the resource/domain.

Quoted 9 lines

You mention a few times in the commit message that you expect the cache resources to only have one control. On MPAM we have CMAX (and there looks to be a RISC-V equivalent) where the total number of bytes in the cache for a given closid is limited. The allocation must still respect the CPBM bitmap though. Looking at the code though I don't see much problem in adding this as an additional control. The assumption that these patches is making is not that there is only one control for cache resources but rather that cache portions are managed by the default cache resource control. Am I missing something or does that assessment make sense to you?

Your assessment is correct. There are still a few assumptions built into resctrl about there only being a single cache control and it being a bitmap control. Since I am not familiar with the other possible cache controls I instead focused on isolating the existing cache control. When I/we have better understanding about how additional cache controls behave this implementation can be adapted to support it.

Quoted 3 lines

I have been looking at adding CMAX control to resctrl and will have a go at basing what I have so far on top of this series.

Thank you!

...

Quoted 3 lines
Quoted 1 line
Quoted 1 line

Any feedback is appreciated.

Overall, this looks to be a big step in the right direction.

Glad to hear this.

Thank you very much.

Reinette

maintainer
#26 lore raw

Hi Reinette,

Quoted 55 lines · Reinette Chatre

Hi Ben,

Quoted 10 lines · Ben Horgan

Hi Reinette,

Quoted 7 lines · Reinette Chatre

Hi Everybody,

It has been a while since we discussed the resctrl changes required to support hardware that has controls with fine granularity or hardware that has multiple controls per resource. For reference, the most recent email discussion can be found at [1] with a summary of discussions in last year's plumbers slides [2].

Quoted 6 lines
Quoted 3 lines

I created a PoC that I believe supports what folks have agreed to so far. I hope this can help us to restart the discussion with the goal that resctrl gains support for upcoming hardware that require these features.

Thank you very much for doing this work. I believe this will be very useful for MPAM and other architectures.

Thank you very much for reviewing this.

Quoted 33 lines
Quoted 15 lines

Request regarding this PoC
==========================

Please consider this PoC as a "direction check" on the schema description and multiple control discussions held thus far.

Could folks working on enabling new hardware requiring this capability please consider if this is something you can build on and how it should be improved to support these upcoming capabilities?

Opens
=====

Quoted 9 lines

While the PoC aims to support what folks agreed on some opens remain:
- I attempted to make some MPAM supporting changes but these are all just compile tested. While MPAM should benefit from the new control properties I did not initialize them on MPAM and did not attempt refactor to separate out the architecture specific control properties (more on what this means later). I did attempt some MPAM refactoring that duplicates the MPAM domain to the control domain and monitoring domain lists in support of there being multiple controls each with its own list of control domains but it is definitely not good design.

I appreciate you including MPAM in this PoC. With this one line change I was able to boot an MPAM system and mount resctrl which appears to behave correctly. We can consider how best to do the code design later.

drivers/resctrl/mpam_resctrl.c +0 -0
@@ -1697,6 +1697,8 @@ int mpam_resctrl_setup(void)
 
Quoted 12 lines
Quoted 8 lines
drivers/resctrl/mpam_resctrl.c +1 -0
        /* Initialise the resctrl structures from the classes */
        for_each_mpam_resctrl_control(res, rid) {
+               INIT_LIST_HEAD(&res->resctrl_res.controls); // list_empty needs

to work
+
if (!res->class)
continue; // dummy resource

Thank you very much. I picked this up but ended up moving it earlier to be next to the mon_domains list initialization. Doing so made it easier for me to follow the initialization. That ok?

Yes, that makes sense.

Quoted 18 lines
Quoted 18 lines

I plumbed in support for the MB_MIN resource schema which also works under light testing. The only fs resctrl code change I needed was:

include/linux/resctrl.h +3 -0
@@ -483,6 +483,9 @@ static inline u32 resctrl_get_default_ctrlval(struct
 resctrl_ctrl *ctrl)
        case RESCTRL_CTRL_BITMAP:
                return BIT_MASK(ctrl->cache.cbm_len) - 1;
        case RESCTRL_CTRL_SCALAR:
+               if (ctrl->name == RESCTRL_CTRL_NAME_MIN)
+                       return ctrl->membw.min_bw;
+
                return ctrl->membw.max_bw;
        }
Quoted 9 lines
Quoted 5 lines

At least on MPAM systems, we use a default of 0 for minimum bandwidth controls as the maximum bandwidth controls only take effect if their value is higher than the minimum bandwidth value. I have specialised this on the ctrl->name which breaks your ctrl->type based classification but that's fixable by just adding a default field to membw.

This I am not sure about. In my understanding a typical "default" value means "no throttling" and, at least on Intel, this default hardware state has been summarized as "min" == "max" == "optimal".

Ok, this sounds odd to me but that is probably because I don't know what Intel systems do. On MPAM systems a MIN control is a boost rather than a throttling control. Although, you can always think of that as throttling the traffic with the other PARTIDs.

Quoted 3 lines

Are you saying that on MPAM systems if "min" == "max" then max bandwidth controls do not take effect? Could you please elaborate what happens if "min" == "max"?

Table 5-4 from section 5.2.8 of the IHI0099B.b shows the interaction between the min and maximum controls.

If used bandwidth is   The preference is  Description
Below the minimum      High               Only high requests compete with this

request.
Above the minimum:
Below the maximum Medium High requests are serviced first then

this request competes with other medium requests.

Above the maximum, Low Requests are not serviced if any high
when HARDLIM is 0 or medium requests are available.

Above the maximum, None Requests are not serviced
when HARDLIM is 1

So if we keep the minimum and the maximum controls values always the same then all traffic will be given "high" preference until the target bandwidth is reached. For some MPAM systems it is recommended to set the minimum value as 5% less than the maximum value to get a reliable target bandwidth. As 5% seems implementation specific and some systems don't have min controls it seemed better to just match the MB control with a maximum bandwidth control and let the user have freedom to choose the minimum bandwidth control when MB_MIN support is added.

If a default for the minimum of the maximum possible bandwidth is used (100%) then any change of the maximum won't have any effect as it's always less than minimum (if that's unchanged) and so all traffic is high preference. I now see from your reply below that you are planning on not allowing this kind of configuration.

If the minimum always tracks the maximum then we lose the distinction between medium and high preference traffic and so to reserve some high preference bandwidth for one control group we'd have to change the configuration in the other controls groups so that they're bandwidth preference is medium (minimum value at 0).

Quoted 14 lines
Quoted 10 lines
Quoted 7 lines

- No support for emulated controls (yet). The PoC is quite large already but I think it can be used as a base for emulated controls for which the software controller could be a potential first customer. In this PoC mounting with software controller will still display the original controller's properties.
- One open that needs to be addressed as part of support for emulated controls is how best to display emulation relationship via resctrl hierarchy.

What does emulated controls mean here? Is there some previous discussion you could point me at?

For emulated controls in context of MPAM I think the best reference is
https://lore.kernel.org/lkml/[email protected]/

Quoted 7 lines

Above is the email discussion that I attempted to visualize in the middle example in slide 6 ("resctrl controls vs. hardware controls") of
https://lpc.events/event/19/contributions/2093/attachments/1958/4172/resctrl%20Microconference%20LPC%202025%20Tokyo.pdf

When comparing the slide to Dave's text, please replace "MB_HW" from Dave's example with "MB_OPT" in the slide. I changed the name since I found the "HW" in an emulated control to be potentially confusing.

Ah I see, thanks for links and descriptions.

Quoted 47 lines
Quoted 9 lines
Quoted 8 lines

- No support for "read-modify-write" usage of schemata file. This is where we discussed (without agreement) on possibly introducing the "#" prefix to schemata file entries. This PoC does not support this prefix and the current assumption/expectation is that when user space changes a configuration only the new control values are written to schemata file. I thus do not have a plan to support this so please share opinions in this regard if you have some.

Quoted 4 lines

There is now less motivation from the MPAM side for this than when this was initially discussed. In pre-upstream versions of the MPAM patches a change in the MB resource control value would change both the mpam h/w mbw_min and mbw_max values but now (on non-broken h/w) we just change the mbw_max. (mbw_min kept at 0).

Ah, thanks for the correction. The email I linked above indeed refers to changing both min and max.

Quoted 2 lines

However, it would be useful not to be limited by percentages. In my quick

Indeed. Not being limited by percentages while still needing to have a backward compatible user interface is how we ended up with "emulated controls".

Quoted 10 lines

experimentation with your patches I used a percentage value for MB_MIN but it would be best to move away from this. For new controls I think we can mandate that user space has to discover the resolution from the info directly but how can we retrofit this. For MPAM, MB and MB_MAX, would control the same things. Could we just add MB_MAX with a h/w friendly scale and then reflect changes in MB_MAX in MB and vica versa with MB taking precedent if both are set? Old software can continue setting MB can move to using MB_MAX and take advantage of the improved control. (I don't think we should expose the MPAM hardware value directly as it has confusion over whether all 1s is 100% or not and we'd like to have something generic and friendly to the user.)

Sounds to me as though you are describing emulated controls. Exposing two controls in schemata file that essentially controls the same thing is what the emulated controls aim to solve and the resctrl hierarchies presented in slide #6 of that presentation (and discussed in the email thread) is how we contemplated how to represent the relationship among these controls to user space. So, considering your example resctrl may display something like:

info//

└── MB/
    └── resource_schemata/
        └── MB/
            └── MB_MAX/
Quoted 6 lines

Above hierarchy describes the relationship to user space that if MB is changed it will impact MB_MAX and vice-versa.

The one open I am aware of surrounding emulated controls is how to present some semblance of consistency to user space when considering all the possibilities the different architectures (and even within architectures) may have.

What other use cases do we have apart from MB and MB_MAX? I was wondering if this could be limited to a default control (L2, L3, MB..) with a single new style control (L2_*, L3_*, MB_ ...) under it.

Quoted 15 lines
Quoted 9 lines
Quoted 5 lines

- Controls are independent for now. This means that, for example, if a resource supports a "MIN" and "MAX" control then this implementation would allow user to set the "maximum" control values to be less than the "minimum" control values.

I think this is ok as long as adding support for new controls in resctrl doesn't change the existing behaviour. In MPAM we dodged this by introducing MB as only affecting the h/w mbw_max and not mbw_min (as mentioned above).

I understand this to be a requirement for Intel where the spec contains "The Maximum Cap should be programmed to be greater than or equal to the Minimum and Optimal caps. Undesirable and undefined performance effects may result if cap programming guidelines are not followed."

Quoted 5 lines

I am currently thinking that resctrl should not try to be too smart here and if user space wants to make dramatic changes to min and max values then it should just ensure the ordering is appropriate. For example, attempting to set a new min to be larger than the old max would fail and user space should first increase the old max and then set a new min.

Ok with me.

Quoted 83 lines
Quoted 83 lines
Quoted 70 lines

- PoC supports the "bitmap" control but does not (yet) expose properties of a bitmap control to the new info/<resource>/resource_schemata directory.

Accessing PoC
=============

Please consider the PoC as a rough draft. It has only been compile tested for Arm and known to be incomplete in Arm support. To help with experimenting I only fully adapted the Intel MBA resource to demo two dummy additional MBA controls. All architectures should immediately benefit from the new schema descriptions and new info/MB/resource_schemata hierarchy.

I considered the patches self too many for email. Instead, the PoC can be found at:

git://git.kernel.org/pub/scm/linux/kernel/git/reinette/linux.git branch resctrl/controls_rfc_v1

The work is based on v7.1-rc2 that also includes the following series (two of which has since been queued) included:

"selftests/resctrl: Fixes and improvements focused on Intel platforms"
https://lore.kernel.org/lkml/[email protected]/

"x86,fs/resctrl: Improve resctrl quality and consistency"
https://lore.kernel.org/lkml/[email protected]/

"x86,fs/resctrl: Pave the way for MPAM counter assignment"
https://lore.kernel.org/lkml/[email protected]/

Primary resctrl fs data structure changes
=========================================

Introduces a control represented by struct resctrl_ctrl that looks as below. To make the changes easier to follow I kept some of the original names to help communicate where familiar data structures land.

What to notice about a control is that it has some common properties required from all controls (scope, type, etc.) and then depending on the type of control (RESCTRL_CTRL_BITMAP or RESCTRL_CTRL_SCALAR) there are type specific properties.

/**
* struct resctrl_ctrl - A resource control
* @entry: List entry of rdt_resource::controls
* @scope: Scope of the resource that this control allocates
* @domains: RCU list of all control domains
* @type: The control type that determines the properties of the control,
* format string for displaying control values to user space, and
* parser of control values provided by user space.
* @name: Name of the control. Appended to final resource name
* (rdt_resource_final::name) to create final schema entry.
* Specifically, "rdt_resource_final::name"_"resctrl_ctrl::name".
* For example, with resource name "MB" and control name "MAX" the
* schema entry will be "MB_MAX".
* @cache: Cache allocation control properties.
* @membw: Bandwidth control properties.
*/

struct resctrl_ctrl {
	struct list_head	entry;
	enum resctrl_scope	scope;
	struct list_head	domains;
	enum resctrl_ctrl_type	type;
	enum resctrl_ctrl_name	name;
	union {
		struct resctrl_cache	cache;
		struct resctrl_membw	membw;
	};
};
Quoted 10 lines

Two members summarize how this new structure fits into the rest of resctrl:
a) resctrl_ctrl::entry
Since a resource can support multiple controls there is a new list in struct rdt_resource named "controls" that contains the list of all controls supported by the resource.
b) resctrl_ctrl::domains
Instead of the list of control domains belonging to a resource they now belong to the control self. By doing so resctrl can support resource controls at different scope for the same resource. This is intended to support some upcoming MPAM and RISC-V usages.

Please can you expand a bit on part b).

Quoted 7 lines
Quoted 5 lines

In an MPAM system we consider 3 resctrl resources, RDT_RESOURCE_L3, RDT_RESOURCE_L2 and RDT_RESOURCE_MBA which correspond to the L3 caches, L2 caches and memory bandwidth on egress from the L3 caches. The domain for each of these corresponds to the instance of the resource. That is, for RDT_RESOURCE_L2 there is a resource for each L2 instance, similarly for L3, and for

(I'm assuming above is typo and it is "there is a domain for each L2 instance"?)

yes, a mistake

Quoted 19 lines
Quoted 13 lines

RDT_RESOURCE_MBA there is a domain for each L3 cache. If we were to add suport for controls on a new cache level, say the L4, then I'd expect to add a new resource. For memory bandwidth, we'd like to be able to control b/w on the L2 egress (e.g. in a DSU). Wouldn't this too be a separate resource or would this be a new set of controls on the same resource?

New controls on the same resource
MB_MIN2
MB_MAX2
MB_PROP2
...

Quoted 4 lines

or
MB2_MIN
MB2_MAX
MB2_PROP

Quoted 6 lines

The way I currently see it is that controlling bandwidth at a different scope would be a new set of controls associated with the MB resource. There are more scenarios coming this way with AMD's "Global MBA" that is memory bandwidth allocation at NUMA node scope. If I understand correctly the "CPU-less Memory Node" that Nvidia shared at plumbers would need this also and control memory bandwidth allocation at the NUMA node scope.

Yes, in general for MSC at the memory controlers it would be good to scope these by NUMA node whether or not they are CPU-less or not.

Quoted 7 lines

A related technology is Intel's region-aware MBA, which is still at L3 scope.

I fully agree that we need to figure out how to represent all of this to user space without turning the interface into something unintelligible. In the end this is required for user space to know what a domain ID represents.

Quoted 5 lines

Would it help to make the scope part of the control name? The ship has sailed for MB being associated with L3 scope but this could mean the "default" scope of MB resource is L3 (which user space can still confirm by looking at the control's "scope" file) and the others include scope in the name? Consider for example:
https://lore.kernel.org/lkml/[email protected]/

This certainly helps with the naming.

The scope does have an effect on what causes a domain to be present or not. For existing scopes, such as L3 scope, that whether a domain is online or not is dependent on whether or not a set cpu is online and the cpu_read_lock is taken. However, for NUMA scope in MPAM (maybe not GMBA?) then whether or not the domain is online would need to depend on whether the memory is online or not and the memory hotplug lock will be needed to be taken. I am wondering if this sort of configuration means it's better to have the NUMA scoped memory bandwidth on a different resource or we just say ok and always take the memory hotplug lock , get_online_mems(), where we take the cpu_read_lock.

Quoted 13 lines
Quoted 5 lines

AFAIK, the DSU h/w just supports proportional bandwidth controls at the moment but we should consider what to do about the potential naming.

ack.

Quoted 5 lines

In the MPAM driver, we collect MSC into components (based on instances) and those into classes (components of the same type). Currently, a resource is mapped to a single class. (Two resources may map to the same class.)

Quoted 8 lines
Quoted 3 lines

I expect it is useful in the memory region and sub numa cases but I'd still expect the common case to be that the domains are the same within a control. Or am I missing something?

Domains of a control should all be at the same scope. Since the schemata file exposes the control with the different IDs representing the instances of the resource needing to be controlled it has to be clear to user space what the domain ID represents.

Agreed. (I meant to say the domains within a resource are likely to be the same for each control within the same resource.)

Thanks,

Ben

Quoted 90 lines
Quoted 35 lines
Quoted 27 lines

Example architectural data structure changes
============================================

An architecture can use the new control by following a similar pattern to resource and domain use by architectures. Consider the following for x86 where a new architecture specific struct resctrl_hw_ctrl includes struct resctrl_ctrl and any architecture private data needed to support the control:

/*
* struct resctrl_hw_ctrl - Arch private properties of a resource control
* @r_ctrl: Control properties exposed to resctrl file system
* @msr_base: Base MSR address where control values should be programmed
* @msr_update: Function pointer to update control values
*/

struct resctrl_hw_ctrl {
	struct resctrl_ctrl	r_ctrl;
	unsigned int		msr_base;
	void			(*msr_update)(struct msr_param *m);
};

Structure of patch series
=========================

Quoted 7 lines

As a PoC the series is not perfectly structured but to help navigate this work on a high level the changes can be categorized as follows:

Patch 1 to 11:
With a vision of what a "control" is, remove unused/unnecessary members, make clear what is a *resource* property vs a *control* property, do some renaming to help with the PoC.

Quoted 5 lines

A few of the changes are generic cleanup and could hopefully be dealt with before decisions on the larger PoC are made. I see:
fs/resctrl: Remove unused resctrl_membw::mb_map x86,fs/resctrl: Remove "arch_needs_linear"
Perhaps a few more.

ack.

Quoted 11 lines
Quoted 10 lines

Patch 12:
Introduce struct resctrl_ctrl and re-arrange existing struct rdt_resource members to form part of new rdt_resource::ctrl

Patch 13 to 44:
A lot of wrangling to introduce struct resctrl_ctrl to all code that needs to work with a control and/or domain without assuming that the control is the one and only control embedded in the resource it belongs to. Essentially, a lot of changes passing the control around in addition to the resource/domain.

Quoted 9 lines

You mention a few times in the commit message that you expect the cache resources to only have one control. On MPAM we have CMAX (and there looks to be a RISC-V equivalent) where the total number of bytes in the cache for a given closid is limited. The allocation must still respect the CPBM bitmap though. Looking at the code though I don't see much problem in adding this as an additional control. The assumption that these patches is making is not that there is only one control for cache resources but rather that cache portions are managed by the default cache resource control. Am I missing something or does that assessment make sense to you?

Your assessment is correct. There are still a few assumptions built into resctrl about there only being a single cache control and it being a bitmap control. Since I am not familiar with the other possible cache controls I instead focused on isolating the existing cache control. When I/we have better understanding about how additional cache controls behave this implementation can be adapted to support it.

Quoted 3 lines

I have been looking at adding CMAX control to resctrl and will have a go at basing what I have so far on top of this series.

Thank you!

...

Quoted 3 lines
Quoted 1 line
Quoted 1 line

Any feedback is appreciated.

Overall, this looks to be a big step in the right direction.

Glad to hear this.

Thank you very much.

Reinette

author maintainer
#27 lore raw

Hi Ben,

Quoted 3 lines · Ben Horgan
Quoted 2 lines · Reinette Chatre
Quoted 1 line · Ben Horgan

On 5/29/26 19:06, Reinette Chatre wrote:

...

Quoted 70 lines
Quoted 18 lines
Quoted 18 lines

I plumbed in support for the MB_MIN resource schema which also works under light testing. The only fs resctrl code change I needed was:

include/linux/resctrl.h +3 -0
@@ -483,6 +483,9 @@ static inline u32 resctrl_get_default_ctrlval(struct
 resctrl_ctrl *ctrl)
        case RESCTRL_CTRL_BITMAP:
                return BIT_MASK(ctrl->cache.cbm_len) - 1;
        case RESCTRL_CTRL_SCALAR:
+               if (ctrl->name == RESCTRL_CTRL_NAME_MIN)
+                       return ctrl->membw.min_bw;
+
                return ctrl->membw.max_bw;
        }
Quoted 9 lines
Quoted 5 lines

At least on MPAM systems, we use a default of 0 for minimum bandwidth controls as the maximum bandwidth controls only take effect if their value is higher than the minimum bandwidth value. I have specialised this on the ctrl->name which breaks your ctrl->type based classification but that's fixable by just adding a default field to membw.

This I am not sure about. In my understanding a typical "default" value means "no throttling" and, at least on Intel, this default hardware state has been summarized as "min" == "max" == "optimal".

Ok, this sounds odd to me but that is probably because I don't know what Intel systems do. On MPAM systems a MIN control is a boost rather than a throttling control. Although, you can always think of that as throttling the traffic with the other PARTIDs.

Quoted 3 lines

Are you saying that on MPAM systems if "min" == "max" then max bandwidth controls do not take effect? Could you please elaborate what happens if "min" == "max"?

Table 5-4 from section 5.2.8 of the IHI0099B.b shows the interaction between the min and maximum controls.

If used bandwidth is   The preference is  Description
Below the minimum      High               Only high requests compete with this

request.
Above the minimum:
Below the maximum Medium High requests are serviced first then

this request competes with other medium requests.

Above the maximum, Low Requests are not serviced if any high
when HARDLIM is 0 or medium requests are available.

Above the maximum, None Requests are not serviced
when HARDLIM is 1

So if we keep the minimum and the maximum controls values always the same then all traffic will be given "high" preference until the target bandwidth is reached. For some MPAM systems it is recommended to set the minimum value as 5% less than the maximum value to get a reliable target bandwidth. As 5% seems implementation specific and some systems don't have min controls it seemed better to just match the MB control with a maximum bandwidth control and let the user have freedom to choose the minimum bandwidth control when MB_MIN support is added.

If a default for the minimum of the maximum possible bandwidth is used (100%) then any change of the maximum won't have any effect as it's always less than minimum (if that's unchanged) and so all traffic is high preference. I now see from your reply below that you are planning on not allowing this kind of configuration.

Quoted 5 lines

If the minimum always tracks the maximum then we lose the distinction between medium and high preference traffic and so to reserve some high preference bandwidth for one control group we'd have to change the configuration in the other controls groups so that they're bandwidth preference is medium (minimum value at 0).

I do not think we are talking about the same thing here. I am *not* saying that minimum and maximum controls should always be the same.

The discussion is about a proposed change to resctrl_get_default_ctrlval(). resctrl uses this function in two places:
- When creating a new resource group:
The intention here is that when user space creates a new resource group it should be created with maximum allocations possible. For MBA this means "unthrottled". After creating the resource group user space can adjust allocations to match workload requirements.
- When unmounting the resctrl fs.
The intention here is that all controls are set to unthrottled to stop any possible impact to system when user space stops using resctrl.

resctrl_get_default_ctrlval() is thus intended to support an unthrottled baseline from where user space can make configuration changes as supported by hardware and required by workloads.

I see that the MPAM driver internally uses resctrl_get_default_ctrlval() in a couple of places and I am not considering this usage here. If internally MPAM has other usages for this function where it does not mean "unthrottled" then perhaps it would be better to create a new function that matches the usage?

Quoted 48 lines
Quoted 48 lines
Quoted 11 lines
Quoted 6 lines

- No support for "read-modify-write" usage of schemata file. This is where we discussed (without agreement) on possibly introducing the "#" prefix to schemata file entries. This PoC does not support this prefix and the current assumption/expectation is that when user space changes a configuration only the new control values are written to schemata file. I thus do not have a plan to support this so please share opinions in this regard if you have some.

There is now less motivation from the MPAM side for this than when this was initially discussed. In pre-upstream versions of the MPAM patches a change in the MB resource control value would change both the mpam h/w mbw_min and mbw_max values but now (on non-broken h/w) we just change the mbw_max. (mbw_min kept at 0).

Ah, thanks for the correction. The email I linked above indeed refers to changing both min and max.

Quoted 2 lines

However, it would be useful not to be limited by percentages. In my quick

Indeed. Not being limited by percentages while still needing to have a backward compatible user interface is how we ended up with "emulated controls".

Quoted 10 lines

experimentation with your patches I used a percentage value for MB_MIN but it would be best to move away from this. For new controls I think we can mandate that user space has to discover the resolution from the info directly but how can we retrofit this. For MPAM, MB and MB_MAX, would control the same things. Could we just add MB_MAX with a h/w friendly scale and then reflect changes in MB_MAX in MB and vica versa with MB taking precedent if both are set? Old software can continue setting MB can move to using MB_MAX and take advantage of the improved control. (I don't think we should expose the MPAM hardware value directly as it has confusion over whether all 1s is 100% or not and we'd like to have something generic and friendly to the user.)

Sounds to me as though you are describing emulated controls. Exposing two controls in schemata file that essentially controls the same thing is what the emulated controls aim to solve and the resctrl hierarchies presented in slide #6 of that presentation (and discussed in the email thread) is how we contemplated how to represent the relationship among these controls to user space. So, considering your example resctrl may display something like:

info//

└── MB/
    └── resource_schemata/
        └── MB/
            └── MB_MAX/

Above hierarchy describes the relationship to user space that if MB is changed it will impact MB_MAX and vice-versa.

Quoted 7 lines
Quoted 3 lines

The one open I am aware of surrounding emulated controls is how to present some semblance of consistency to user space when considering all the possibilities the different architectures (and even within architectures) may have.

What other use cases do we have apart from MB and MB_MAX? I was wondering if this could be limited to a default control (L2, L3, MB..) with a single new style control (L2_*, L3_*, MB_ ...) under it.

The motivation for these emulated controls is to not break a user space that does not understand the "info/<resource>/resource_schemata" interface. At this time user space expects every resource (not control) to have an entry in the schemata file. So yes, I also see this as limited to the default control.

Whether it implies that only a single (finer grained/hardware) control would be under it is not obvious to me since we already had one scenario where a legacy control is emulated by two hardware controls when considering the example on MPAM where the "MB" legacy control can be emulated with MPAM's "min" and "max" controls. An additional complication is that some of these architecture specs describe several controls but have their implementation as "optional" which presents a challenge when trying to create a sane and consistent hierarchy.

Quoted 13 lines
Quoted 13 lines
Quoted 7 lines
Quoted 3 lines

- Controls are independent for now. This means that, for example, if a resource supports a "MIN" and "MAX" control then this implementation would allow user to set the "maximum" control values to be less than the "minimum" control values.

I think this is ok as long as adding support for new controls in resctrl doesn't change the existing behaviour. In MPAM we dodged this by introducing MB as only affecting the h/w mbw_max and not mbw_min (as mentioned above).

I understand this to be a requirement for Intel where the spec contains "The Maximum Cap should be programmed to be greater than or equal to the Minimum and Optimal caps. Undesirable and undefined performance effects may result if cap programming guidelines are not followed."

Quoted 7 lines
Quoted 5 lines

I am currently thinking that resctrl should not try to be too smart here and if user space wants to make dramatic changes to min and max values then it should just ensure the ordering is appropriate. For example, attempting to set a new min to be larger than the old max would fail and user space should first increase the old max and then set a new min.

Ok with me.

Thank you for considering this.

...

Quoted 106 lines
Quoted 52 lines
Quoted 52 lines
Quoted 39 lines

Primary resctrl fs data structure changes
=========================================

Introduces a control represented by struct resctrl_ctrl that looks as below. To make the changes easier to follow I kept some of the original names to help communicate where familiar data structures land.

What to notice about a control is that it has some common properties required from all controls (scope, type, etc.) and then depending on the type of control (RESCTRL_CTRL_BITMAP or RESCTRL_CTRL_SCALAR) there are type specific properties.

/**
* struct resctrl_ctrl - A resource control
* @entry: List entry of rdt_resource::controls
* @scope: Scope of the resource that this control allocates
* @domains: RCU list of all control domains
* @type: The control type that determines the properties of the control,
* format string for displaying control values to user space, and
* parser of control values provided by user space.
* @name: Name of the control. Appended to final resource name
* (rdt_resource_final::name) to create final schema entry.
* Specifically, "rdt_resource_final::name"_"resctrl_ctrl::name".
* For example, with resource name "MB" and control name "MAX" the
* schema entry will be "MB_MAX".
* @cache: Cache allocation control properties.
* @membw: Bandwidth control properties.
*/

struct resctrl_ctrl {
	struct list_head	entry;
	enum resctrl_scope	scope;
	struct list_head	domains;
	enum resctrl_ctrl_type	type;
	enum resctrl_ctrl_name	name;
	union {
		struct resctrl_cache	cache;
		struct resctrl_membw	membw;
	};
};
Quoted 10 lines

Two members summarize how this new structure fits into the rest of resctrl:
a) resctrl_ctrl::entry
Since a resource can support multiple controls there is a new list in struct rdt_resource named "controls" that contains the list of all controls supported by the resource.
b) resctrl_ctrl::domains
Instead of the list of control domains belonging to a resource they now belong to the control self. By doing so resctrl can support resource controls at different scope for the same resource. This is intended to support some upcoming MPAM and RISC-V usages.

Please can you expand a bit on part b).

Quoted 7 lines
Quoted 5 lines

In an MPAM system we consider 3 resctrl resources, RDT_RESOURCE_L3, RDT_RESOURCE_L2 and RDT_RESOURCE_MBA which correspond to the L3 caches, L2 caches and memory bandwidth on egress from the L3 caches. The domain for each of these corresponds to the instance of the resource. That is, for RDT_RESOURCE_L2 there is a resource for each L2 instance, similarly for L3, and for

(I'm assuming above is typo and it is "there is a domain for each L2 instance"?)

yes, a mistake

Quoted 19 lines
Quoted 13 lines

RDT_RESOURCE_MBA there is a domain for each L3 cache. If we were to add suport for controls on a new cache level, say the L4, then I'd expect to add a new resource. For memory bandwidth, we'd like to be able to control b/w on the L2 egress (e.g. in a DSU). Wouldn't this too be a separate resource or would this be a new set of controls on the same resource?

New controls on the same resource
MB_MIN2
MB_MAX2
MB_PROP2
...

Quoted 4 lines

or
MB2_MIN
MB2_MAX
MB2_PROP

Quoted 6 lines

The way I currently see it is that controlling bandwidth at a different scope would be a new set of controls associated with the MB resource. There are more scenarios coming this way with AMD's "Global MBA" that is memory bandwidth allocation at NUMA node scope. If I understand correctly the "CPU-less Memory Node" that Nvidia shared at plumbers would need this also and control memory bandwidth allocation at the NUMA node scope.

Yes, in general for MSC at the memory controlers it would be good to scope these by NUMA node whether or not they are CPU-less or not.

Quoted 7 lines

A related technology is Intel's region-aware MBA, which is still at L3 scope.

I fully agree that we need to figure out how to represent all of this to user space without turning the interface into something unintelligible. In the end this is required for user space to know what a domain ID represents.

Quoted 5 lines

Would it help to make the scope part of the control name? The ship has sailed for MB being associated with L3 scope but this could mean the "default" scope of MB resource is L3 (which user space can still confirm by looking at the control's "scope" file) and the others include scope in the name? Consider for example:
https://lore.kernel.org/lkml/[email protected]/

This certainly helps with the naming.

Quoted 9 lines

The scope does have an effect on what causes a domain to be present or not. For existing scopes, such as L3 scope, that whether a domain is online or not is dependent on whether or not a set cpu is online and the cpu_read_lock is taken. However, for NUMA scope in MPAM (maybe not GMBA?) then whether or not the domain is online would need to depend on whether the memory is online or not and the memory hotplug lock will be needed to be taken. I am wondering if this sort of configuration means it's better to have the NUMA scoped memory bandwidth on a different resource or we just say ok and always take the memory hotplug lock , get_online_mems(), where we take the cpu_read_lock.

oh, thank you for bringing this up. I have not considered how the memory hotplug lock needs to be integrated. Taking cpus_read_lock() has permeated the entire subsystem. My initial thought is that having unique per-resource locking sounds complicated while always taking memory hotplug lock sounds much simpler. I do not see many users of get_online_mems() though.

Quoted 14 lines
Quoted 14 lines
Quoted 2 lines

AFAIK, the DSU h/w just supports proportional bandwidth controls at the moment but we should consider what to do about the potential naming.

ack.

Quoted 8 lines

In the MPAM driver, we collect MSC into components (based on instances) and those into classes (components of the same type). Currently, a resource is mapped to a single class. (Two resources may map to the same class.)

I expect it is useful in the memory region and sub numa cases but I'd still expect the common case to be that the domains are the same within a control. Or am I missing something?

Quoted 7 lines
Quoted 4 lines

Domains of a control should all be at the same scope. Since the schemata file exposes the control with the different IDs representing the instances of the resource needing to be controlled it has to be clear to user space what the domain ID represents.

Agreed. (I meant to say the domains within a resource are likely to be the same for each control within the same resource.)

This seems accurate for the resources that have implicit scope (the caches) but memory bandwidth as a resource is looking more like it needs to support allocation at different scopes.

Reinette

maintainer
#28 lore raw

Hi Reinette,

Quoted 85 lines · Reinette Chatre

Hi Ben,

Quoted 3 lines · Ben Horgan
Quoted 2 lines · Reinette Chatre
Quoted 1 line · Ben Horgan

On 5/29/26 19:06, Reinette Chatre wrote:

...

Quoted 70 lines
Quoted 18 lines
Quoted 18 lines

I plumbed in support for the MB_MIN resource schema which also works under light testing. The only fs resctrl code change I needed was:

include/linux/resctrl.h +3 -0
@@ -483,6 +483,9 @@ static inline u32 resctrl_get_default_ctrlval(struct
 resctrl_ctrl *ctrl)
        case RESCTRL_CTRL_BITMAP:
                return BIT_MASK(ctrl->cache.cbm_len) - 1;
        case RESCTRL_CTRL_SCALAR:
+               if (ctrl->name == RESCTRL_CTRL_NAME_MIN)
+                       return ctrl->membw.min_bw;
+
                return ctrl->membw.max_bw;
        }
Quoted 9 lines
Quoted 5 lines

At least on MPAM systems, we use a default of 0 for minimum bandwidth controls as the maximum bandwidth controls only take effect if their value is higher than the minimum bandwidth value. I have specialised this on the ctrl->name which breaks your ctrl->type based classification but that's fixable by just adding a default field to membw.

This I am not sure about. In my understanding a typical "default" value means "no throttling" and, at least on Intel, this default hardware state has been summarized as "min" == "max" == "optimal".

Ok, this sounds odd to me but that is probably because I don't know what Intel systems do. On MPAM systems a MIN control is a boost rather than a throttling control. Although, you can always think of that as throttling the traffic with the other PARTIDs.

Quoted 3 lines

Are you saying that on MPAM systems if "min" == "max" then max bandwidth controls do not take effect? Could you please elaborate what happens if "min" == "max"?

Table 5-4 from section 5.2.8 of the IHI0099B.b shows the interaction between the min and maximum controls.

If used bandwidth is   The preference is  Description
Below the minimum      High               Only high requests compete with this

request.
Above the minimum:
Below the maximum Medium High requests are serviced first then

this request competes with other medium requests.

Above the maximum, Low Requests are not serviced if any high
when HARDLIM is 0 or medium requests are available.

Above the maximum, None Requests are not serviced
when HARDLIM is 1

So if we keep the minimum and the maximum controls values always the same then all traffic will be given "high" preference until the target bandwidth is reached. For some MPAM systems it is recommended to set the minimum value as 5% less than the maximum value to get a reliable target bandwidth. As 5% seems implementation specific and some systems don't have min controls it seemed better to just match the MB control with a maximum bandwidth control and let the user have freedom to choose the minimum bandwidth control when MB_MIN support is added.

If a default for the minimum of the maximum possible bandwidth is used (100%) then any change of the maximum won't have any effect as it's always less than minimum (if that's unchanged) and so all traffic is high preference. I now see from your reply below that you are planning on not allowing this kind of configuration.

Quoted 5 lines

If the minimum always tracks the maximum then we lose the distinction between medium and high preference traffic and so to reserve some high preference bandwidth for one control group we'd have to change the configuration in the other controls groups so that they're bandwidth preference is medium (minimum value at 0).

Quoted 8 lines

I do not think we are talking about the same thing here. I am *not* saying that minimum and maximum controls should always be the same.

The discussion is about a proposed change to resctrl_get_default_ctrlval(). resctrl uses this function in two places:
- When creating a new resource group:
The intention here is that when user space creates a new resource group it should be created with maximum allocations possible. For MBA this means "unthrottled".

I would contend that for minimum controls that a policy of 'maximum allocation possible' isn't a useful default. I try and explain a bit more below.

Quoted 9 lines

After creating the resource group user space can adjust allocations to match workload requirements.
- When unmounting the resctrl fs.
The intention here is that all controls are set to unthrottled to stop any possible impact to system when user space stops using resctrl.

resctrl_get_default_ctrlval() is thus intended to support an unthrottled baseline from where user space can make configuration changes as supported by hardware and required by workloads.

The baseline that I see makes most sense for a minimum control is to have the default as 0. This just means that there is no "guaranteed"/high preference bandwidth reserved for the control group. I would say this still unthrottled but just not giving a boost. With this default the user can use MB (backed by max bandwidth) without having to know about MB_MIN (keeping it constant). If the default is 100% for min bandwidth then the user needs to know to set MB_MIN to be able to use MB. Having a default of 100% for max bandwidth, correspondingly means a user can change MB_MIN and see guaranteed bandwidth effects without having to know about MB/MB_MAX.

Does this make sense?

Quoted 5 lines

I see that the MPAM driver internally uses resctrl_get_default_ctrlval() in a couple of places and I am not considering this usage here. If internally MPAM has other usages for this function where it does not mean "unthrottled" then perhaps it would be better to create a new function that matches the usage?

I don't think the internal usage makes a difference here.

One process thing I was wondering about so that I know how to structure my patches. In the series you have a few patches which touch all architectures; these have the prefix mpam,x86,fs/resctrl. Is this how you would like cross architectures patches to look like or is it just for convenience in the rfc and a patch per-architecture is preferable?

Thanks,

Ben

author maintainer
#29 lore raw

Hi Ben,

Quoted 120 lines · Ben Horgan

Hi Reinette,

Quoted 85 lines · Reinette Chatre

Hi Ben,

Quoted 3 lines · Ben Horgan
Quoted 2 lines · Reinette Chatre
Quoted 1 line · Ben Horgan

On 5/29/26 19:06, Reinette Chatre wrote:

...

Quoted 70 lines
Quoted 18 lines
Quoted 18 lines

I plumbed in support for the MB_MIN resource schema which also works under light testing. The only fs resctrl code change I needed was:

include/linux/resctrl.h +3 -0
@@ -483,6 +483,9 @@ static inline u32 resctrl_get_default_ctrlval(struct
 resctrl_ctrl *ctrl)
        case RESCTRL_CTRL_BITMAP:
                return BIT_MASK(ctrl->cache.cbm_len) - 1;
        case RESCTRL_CTRL_SCALAR:
+               if (ctrl->name == RESCTRL_CTRL_NAME_MIN)
+                       return ctrl->membw.min_bw;
+
                return ctrl->membw.max_bw;
        }
Quoted 9 lines
Quoted 5 lines

At least on MPAM systems, we use a default of 0 for minimum bandwidth controls as the maximum bandwidth controls only take effect if their value is higher than the minimum bandwidth value. I have specialised this on the ctrl->name which breaks your ctrl->type based classification but that's fixable by just adding a default field to membw.

This I am not sure about. In my understanding a typical "default" value means "no throttling" and, at least on Intel, this default hardware state has been summarized as "min" == "max" == "optimal".

Ok, this sounds odd to me but that is probably because I don't know what Intel systems do. On MPAM systems a MIN control is a boost rather than a throttling control. Although, you can always think of that as throttling the traffic with the other PARTIDs.

Quoted 3 lines

Are you saying that on MPAM systems if "min" == "max" then max bandwidth controls do not take effect? Could you please elaborate what happens if "min" == "max"?

Table 5-4 from section 5.2.8 of the IHI0099B.b shows the interaction between the min and maximum controls.

If used bandwidth is   The preference is  Description
Below the minimum      High               Only high requests compete with this

request.
Above the minimum:
Below the maximum Medium High requests are serviced first then

this request competes with other medium requests.

Above the maximum, Low Requests are not serviced if any high
when HARDLIM is 0 or medium requests are available.

Above the maximum, None Requests are not serviced
when HARDLIM is 1

So if we keep the minimum and the maximum controls values always the same then all traffic will be given "high" preference until the target bandwidth is reached. For some MPAM systems it is recommended to set the minimum value as 5% less than the maximum value to get a reliable target bandwidth. As 5% seems implementation specific and some systems don't have min controls it seemed better to just match the MB control with a maximum bandwidth control and let the user have freedom to choose the minimum bandwidth control when MB_MIN support is added.

If a default for the minimum of the maximum possible bandwidth is used (100%) then any change of the maximum won't have any effect as it's always less than minimum (if that's unchanged) and so all traffic is high preference. I now see from your reply below that you are planning on not allowing this kind of configuration.

Quoted 5 lines

If the minimum always tracks the maximum then we lose the distinction between medium and high preference traffic and so to reserve some high preference bandwidth for one control group we'd have to change the configuration in the other controls groups so that they're bandwidth preference is medium (minimum value at 0).

Quoted 8 lines

I do not think we are talking about the same thing here. I am *not* saying that minimum and maximum controls should always be the same.

The discussion is about a proposed change to resctrl_get_default_ctrlval(). resctrl uses this function in two places:
- When creating a new resource group:
The intention here is that when user space creates a new resource group it should be created with maximum allocations possible. For MBA this means "unthrottled".

I would contend that for minimum controls that a policy of 'maximum allocation possible' isn't a useful default. I try and explain a bit more below.

Quoted 9 lines

After creating the resource group user space can adjust allocations to match workload requirements.
- When unmounting the resctrl fs.
The intention here is that all controls are set to unthrottled to stop any possible impact to system when user space stops using resctrl.

resctrl_get_default_ctrlval() is thus intended to support an unthrottled baseline from where user space can make configuration changes as supported by hardware and required by workloads.

The baseline that I see makes most sense for a minimum control is to have the default as 0. This just means that there is no "guaranteed"/high preference bandwidth reserved for the control group. I would say this still unthrottled but just not giving a boost. With this default the user can use MB (backed by max bandwidth) without having to know about MB_MIN (keeping it constant). If the default is 100% for min bandwidth then the user needs to know to set MB_MIN to be able to use MB. Having a default of 100% for max bandwidth, correspondingly means a user can change MB_MIN and see guaranteed bandwidth effects without having to know about MB/MB_MAX.

Quoted 1 line

Does this make sense?

I see. I've only considered the original scenario where MPAM's MB is emulated with both MIN and MAX controls based on examples in
https://lore.kernel.org/lkml/[email protected]/

There seems to be two issues here:
a) Since some systems require MIN to be 5% less than MAX MPAM driver may not know what the system MIN and MAX difference should be to get optimal bandwidth.
b) Some MPAM systems have MAX control but not the MIN control.

The proposal to only emulate MB with MAX is clear but it is not obvious why, on a system that support both MIN and MAX, MB cannot be emulated with both as in the original proposal. Is this motivated by (a) where MPAM driver just does not know what the "max" of a MIN control value should be? Just emulating MB with MAX control does not seem to eliminate this problem since between the fs and arch resctrl still needs to ensure that when user space writes a control value to the MIN control that it is valid for the underlying MPAM system.

It almost sounds as though there is an attempt to eliminate resctrl's usage of a "max" value for the MIN control since that is effectively unknown to MPAM but that does not look possible to me?

Quoted 6 lines
Quoted 4 lines

I see that the MPAM driver internally uses resctrl_get_default_ctrlval() in a couple of places and I am not considering this usage here. If internally MPAM has other usages for this function where it does not mean "unthrottled" then perhaps it would be better to create a new function that matches the usage?

I don't think the internal usage makes a difference here.

Thanks for checking.

Quoted 6 lines

One process thing I was wondering about so that I know how to structure my patches. In the series you have a few patches which touch all architectures; these have the prefix mpam,x86,fs/resctrl. Is this how you would like cross architectures patches to look like or is it just for convenience in the rfc and a patch per-architecture is preferable?

A per-architecture patch is preferable. I did try to keep changes separate as much as possible but some places all parts needed to be changed together to ensure that bisect continue to work. I expect I may have missed some places in the rush to get the PoC done and those need to be fixed.

Reinette

maintainer
#30 lore raw

Hi Reinette,

Quoted 123 lines · Reinette Chatre

Hi Ben,

Quoted 120 lines · Ben Horgan

Hi Reinette,

Quoted 85 lines · Reinette Chatre

Hi Ben,

Quoted 3 lines · Ben Horgan
Quoted 2 lines · Reinette Chatre
Quoted 1 line · Ben Horgan

On 5/29/26 19:06, Reinette Chatre wrote:

...

Quoted 70 lines
Quoted 18 lines
Quoted 18 lines

I plumbed in support for the MB_MIN resource schema which also works under light testing. The only fs resctrl code change I needed was:

include/linux/resctrl.h +3 -0
@@ -483,6 +483,9 @@ static inline u32 resctrl_get_default_ctrlval(struct
 resctrl_ctrl *ctrl)
        case RESCTRL_CTRL_BITMAP:
                return BIT_MASK(ctrl->cache.cbm_len) - 1;
        case RESCTRL_CTRL_SCALAR:
+               if (ctrl->name == RESCTRL_CTRL_NAME_MIN)
+                       return ctrl->membw.min_bw;
+
                return ctrl->membw.max_bw;
        }
Quoted 9 lines
Quoted 5 lines

At least on MPAM systems, we use a default of 0 for minimum bandwidth controls as the maximum bandwidth controls only take effect if their value is higher than the minimum bandwidth value. I have specialised this on the ctrl->name which breaks your ctrl->type based classification but that's fixable by just adding a default field to membw.

This I am not sure about. In my understanding a typical "default" value means "no throttling" and, at least on Intel, this default hardware state has been summarized as "min" == "max" == "optimal".

Ok, this sounds odd to me but that is probably because I don't know what Intel systems do. On MPAM systems a MIN control is a boost rather than a throttling control. Although, you can always think of that as throttling the traffic with the other PARTIDs.

Quoted 3 lines

Are you saying that on MPAM systems if "min" == "max" then max bandwidth controls do not take effect? Could you please elaborate what happens if "min" == "max"?

Table 5-4 from section 5.2.8 of the IHI0099B.b shows the interaction between the min and maximum controls.

If used bandwidth is   The preference is  Description
Below the minimum      High               Only high requests compete with this

request.
Above the minimum:
Below the maximum Medium High requests are serviced first then

this request competes with other medium requests.

Above the maximum, Low Requests are not serviced if any high
when HARDLIM is 0 or medium requests are available.

Above the maximum, None Requests are not serviced
when HARDLIM is 1

So if we keep the minimum and the maximum controls values always the same then all traffic will be given "high" preference until the target bandwidth is reached. For some MPAM systems it is recommended to set the minimum value as 5% less than the maximum value to get a reliable target bandwidth. As 5% seems implementation specific and some systems don't have min controls it seemed better to just match the MB control with a maximum bandwidth control and let the user have freedom to choose the minimum bandwidth control when MB_MIN support is added.

If a default for the minimum of the maximum possible bandwidth is used (100%) then any change of the maximum won't have any effect as it's always less than minimum (if that's unchanged) and so all traffic is high preference. I now see from your reply below that you are planning on not allowing this kind of configuration.

Quoted 5 lines

If the minimum always tracks the maximum then we lose the distinction between medium and high preference traffic and so to reserve some high preference bandwidth for one control group we'd have to change the configuration in the other controls groups so that they're bandwidth preference is medium (minimum value at 0).

Quoted 8 lines

I do not think we are talking about the same thing here. I am *not* saying that minimum and maximum controls should always be the same.

The discussion is about a proposed change to resctrl_get_default_ctrlval(). resctrl uses this function in two places:
- When creating a new resource group:
The intention here is that when user space creates a new resource group it should be created with maximum allocations possible. For MBA this means "unthrottled".

I would contend that for minimum controls that a policy of 'maximum allocation possible' isn't a useful default. I try and explain a bit more below.

Quoted 9 lines

After creating the resource group user space can adjust allocations to match workload requirements.
- When unmounting the resctrl fs.
The intention here is that all controls are set to unthrottled to stop any possible impact to system when user space stops using resctrl.

resctrl_get_default_ctrlval() is thus intended to support an unthrottled baseline from where user space can make configuration changes as supported by hardware and required by workloads.

The baseline that I see makes most sense for a minimum control is to have the default as 0. This just means that there is no "guaranteed"/high preference bandwidth reserved for the control group. I would say this still unthrottled but just not giving a boost. With this default the user can use MB (backed by max bandwidth) without having to know about MB_MIN (keeping it constant). If the default is 100% for min bandwidth then the user needs to know to set MB_MIN to be able to use MB. Having a default of 100% for max bandwidth, correspondingly means a user can change MB_MIN and see guaranteed bandwidth effects without having to know about MB/MB_MAX.

Quoted 5 lines
Quoted 1 line

Does this make sense?

I see. I've only considered the original scenario where MPAM's MB is emulated with both MIN and MAX controls based on examples in
https://lore.kernel.org/lkml/[email protected]/

Ok, but I'm not sure why that's relevant. I feel we are talking at cross purposely here and so I'll try and answer your questions but I'm likely missing the point.

Quoted 9 lines

There seems to be two issues here:
a) Since some systems require MIN to be 5% less than MAX MPAM driver may not know what the system MIN and MAX difference should be to get optimal bandwidth.
b) Some MPAM systems have MAX control but not the MIN control.

The proposal to only emulate MB with MAX is clear but it is not obvious why, on a system that support both MIN and MAX, MB cannot be emulated with both as in the original proposal.

I don't see the advantage of emulating MB with both MIN and MAX. Just going by the MPAM specification, a system keeping MIN at 0 and just setting MAX from MB, (MIN=0, MAX=MB) should behave the same as one always setting both, (MIN=MB, MAX=MB). In the MIN=0 case there is never any high preference traffic and in the MIN=MAX_MB case there is never any medium preference traffic. It seemed best to not rely on any platform specific heuristics to try and guess what's better and just wait til the time we could support MB_MIN in resctrl (and leave the decision up to the user). My expectation was that this would be the simplest course of action.

Is this motivated by (a) where MPAM driver just does not know

Quoted 1 line

what the "max" of a MIN control value should be?

What's the "max" of a MIN control?

The maximum value we can set it to?

We can work that. For MPAM it's writing all 1s to the register which for the minumum case represents ((2**mbw_min_wd)/2**mbw_min_wd)) * 100 %

Just emulating MB with MAX control

Quoted 7 lines

does not seem to eliminate this problem since between the fs and arch resctrl still needs to ensure that when user space writes a control value to the MIN control that it is valid for the underlying MPAM system.

It almost sounds as though there is an attempt to eliminate resctrl's usage of a "max" value for the MIN control since that is effectively unknown to MPAM but that does not look possible to me?

Sorry but I haven't understood what your saying. What does "resctrl's usage of a "max" value for MIN control" mean?

Quoted 17 lines
Quoted 7 lines
Quoted 5 lines

I see that the MPAM driver internally uses resctrl_get_default_ctrlval() in a couple of places and I am not considering this usage here. If internally MPAM has other usages for this function where it does not mean "unthrottled" then perhaps it would be better to create a new function that matches the usage?

I don't think the internal usage makes a difference here.

Thanks for checking.

Quoted 6 lines

One process thing I was wondering about so that I know how to structure my patches. In the series you have a few patches which touch all architectures; these have the prefix mpam,x86,fs/resctrl. Is this how you would like cross architectures patches to look like or is it just for convenience in the rfc and a patch per-architecture is preferable?

Quoted 4 lines

A per-architecture patch is preferable. I did try to keep changes separate as much as possible but some places all parts needed to be changed together to ensure that bisect continue to work. I expect I may have missed some places in the rush to get the PoC done and those need to be fixed.

I haven't checked if what you have could be better split but getting it out without worrying too much about such things seems the right call for the PoC.

Thanks,

Ben

Quoted 4 lines

Reinette

author maintainer
#31 lore raw

Hi Ben,

Quoted 140 lines · Ben Horgan
Quoted 119 lines · Reinette Chatre
Quoted 118 lines · Ben Horgan
Quoted 85 lines · Reinette Chatre

Hi Ben,

Quoted 3 lines · Ben Horgan
Quoted 2 lines · Reinette Chatre
Quoted 1 line · Ben Horgan

On 5/29/26 19:06, Reinette Chatre wrote:

...

Quoted 70 lines
Quoted 18 lines
Quoted 18 lines

I plumbed in support for the MB_MIN resource schema which also works under light testing. The only fs resctrl code change I needed was:

include/linux/resctrl.h +3 -0
@@ -483,6 +483,9 @@ static inline u32 resctrl_get_default_ctrlval(struct
 resctrl_ctrl *ctrl)
        case RESCTRL_CTRL_BITMAP:
                return BIT_MASK(ctrl->cache.cbm_len) - 1;
        case RESCTRL_CTRL_SCALAR:
+               if (ctrl->name == RESCTRL_CTRL_NAME_MIN)
+                       return ctrl->membw.min_bw;
+
                return ctrl->membw.max_bw;
        }
Quoted 9 lines
Quoted 5 lines

At least on MPAM systems, we use a default of 0 for minimum bandwidth controls as the maximum bandwidth controls only take effect if their value is higher than the minimum bandwidth value. I have specialised this on the ctrl->name which breaks your ctrl->type based classification but that's fixable by just adding a default field to membw.

This I am not sure about. In my understanding a typical "default" value means "no throttling" and, at least on Intel, this default hardware state has been summarized as "min" == "max" == "optimal".

Ok, this sounds odd to me but that is probably because I don't know what Intel systems do. On MPAM systems a MIN control is a boost rather than a throttling control. Although, you can always think of that as throttling the traffic with the other PARTIDs.

Quoted 3 lines

Are you saying that on MPAM systems if "min" == "max" then max bandwidth controls do not take effect? Could you please elaborate what happens if "min" == "max"?

Table 5-4 from section 5.2.8 of the IHI0099B.b shows the interaction between the min and maximum controls.

If used bandwidth is   The preference is  Description
Below the minimum      High               Only high requests compete with this

request.
Above the minimum:
Below the maximum Medium High requests are serviced first then

this request competes with other medium requests.

Above the maximum, Low Requests are not serviced if any high
when HARDLIM is 0 or medium requests are available.

Above the maximum, None Requests are not serviced
when HARDLIM is 1

So if we keep the minimum and the maximum controls values always the same then all traffic will be given "high" preference until the target bandwidth is reached. For some MPAM systems it is recommended to set the minimum value as 5% less than the maximum value to get a reliable target bandwidth. As 5% seems implementation specific and some systems don't have min controls it seemed better to just match the MB control with a maximum bandwidth control and let the user have freedom to choose the minimum bandwidth control when MB_MIN support is added.

If a default for the minimum of the maximum possible bandwidth is used (100%) then any change of the maximum won't have any effect as it's always less than minimum (if that's unchanged) and so all traffic is high preference. I now see from your reply below that you are planning on not allowing this kind of configuration.

Quoted 5 lines

If the minimum always tracks the maximum then we lose the distinction between medium and high preference traffic and so to reserve some high preference bandwidth for one control group we'd have to change the configuration in the other controls groups so that they're bandwidth preference is medium (minimum value at 0).

Quoted 8 lines

I do not think we are talking about the same thing here. I am *not* saying that minimum and maximum controls should always be the same.

The discussion is about a proposed change to resctrl_get_default_ctrlval(). resctrl uses this function in two places:
- When creating a new resource group:
The intention here is that when user space creates a new resource group it should be created with maximum allocations possible. For MBA this means "unthrottled".

I would contend that for minimum controls that a policy of 'maximum allocation possible' isn't a useful default. I try and explain a bit more below.

Quoted 9 lines

After creating the resource group user space can adjust allocations to match workload requirements.
- When unmounting the resctrl fs.
The intention here is that all controls are set to unthrottled to stop any possible impact to system when user space stops using resctrl.

resctrl_get_default_ctrlval() is thus intended to support an unthrottled baseline from where user space can make configuration changes as supported by hardware and required by workloads.

The baseline that I see makes most sense for a minimum control is to have the default as 0. This just means that there is no "guaranteed"/high preference bandwidth reserved for the control group. I would say this still unthrottled but just not giving a boost. With this default the user can use MB (backed by max bandwidth) without having to know about MB_MIN (keeping it constant). If the default is 100% for min bandwidth then the user needs to know to set MB_MIN to be able to use MB. Having a default of 100% for max bandwidth, correspondingly means a user can change MB_MIN and see guaranteed bandwidth effects without having to know about MB/MB_MAX.

Quoted 5 lines
Quoted 1 line

Does this make sense?

I see. I've only considered the original scenario where MPAM's MB is emulated with both MIN and MAX controls based on examples in
https://lore.kernel.org/lkml/[email protected]/

Ok, but I'm not sure why that's relevant. I feel we are talking at cross purposely here and so I'll try and answer your questions but I'm likely missing the point.

Quoted 9 lines

There seems to be two issues here:
a) Since some systems require MIN to be 5% less than MAX MPAM driver may not know what the system MIN and MAX difference should be to get optimal bandwidth.
b) Some MPAM systems have MAX control but not the MIN control.

The proposal to only emulate MB with MAX is clear but it is not obvious why, on a system that support both MIN and MAX, MB cannot be emulated with both as in the original proposal.

Quoted 9 lines

I don't see the advantage of emulating MB with both MIN and MAX. Just going by the MPAM specification, a system keeping MIN at 0 and just setting MAX from MB, (MIN=0, MAX=MB) should behave the same as one always setting both, (MIN=MB, MAX=MB). In the MIN=0 case there is never any high preference traffic and in the MIN=MAX_MB case there is never any medium preference traffic. It seemed best to not rely on any platform specific heuristics to try and guess what's better and just wait til the time we could support MB_MIN in resctrl (and leave the decision up to the user). My expectation was that this would be the simplest course of action.

This sounds fair. Two observations:
- The hierarchy exposed by resctrl may be different on systems that have the "same" controls.
For example, on an MPAM system (if I understand correctly) the user may see:
info/

  └── MB/
      └── resource_schemata/
          ├── MB/
          │   └── MB_MAX/
          └── MB_MIN/

Compared with a possible implementation on Intel that looks like:
info/

  └── MB/
      └── resource_schemata/
          ├── MB/
          │   └── MB_OPT/
          ├── MB_MAX/
          └── MB_MIN/

On Intel these controls are optional so could even look more like MPAM:
info/

  └── MB/
      └── resource_schemata/
          └── MB/
             └── MB_MAX/

Dave Martin had some musings about how to present controls to user space with some semblance of consistency but we were not able to finalize that.
At this time it seems that there may be agreement on needing hierarchies as above but I am concerned about the complicated interface that results from inconsistencies between systems. While complicated it does represent the relationships accurately and thus could also be seen as working as intended ... so perhaps we just need to take great care in documenting this hierarchy mechanism as the source for control information as opposed to the schemata file.

- I see that the default of zero for the MIN control can be appropriate. For the usage of resctrl_get_default_ctrlval() when creating a new control group this could be ok as a general setting since the user is expected to, after creating the resource group, adjust control values as required by the workloads it will contain. I need to make sure about this from all architectures and if 0 is not ok then we could use Drew's suggestion of architecture providing a specific reset value.
For the other usage of resctrl_get_default_ctrlval() that resets controls on unmount there is not an opportunity for user space to adjust and here more care needs to be taken to match architecture requirements. The architecture provided "reset" value does sound more appealing.

Quoted 7 lines

Is this motivated by (a) where MPAM driver just does not know

Quoted 1 line

what the "max" of a MIN control value should be?

What's the "max" of a MIN control?

The maximum value we can set it to?

The maximum value a user can set it to. I understood from an earlier comment that there are some MPAM systems where the MIN needs to be set to an implementation specific 5% less than MAX. I interpreted this to mean that it is difficult for the MPAM driver to always know what the maximum value is that a user can write to the MIN control.

Quoted 9 lines

We can work that. For MPAM it's writing all 1s to the register which for the minumum case represents ((2**mbw_min_wd)/2**mbw_min_wd)) * 100 %

Just emulating MB with MAX control

Quoted 4 lines

does not seem to eliminate this problem since between the fs and arch resctrl still needs to ensure that when user space writes a control value to the MIN control that it is valid for the underlying MPAM system.

Quoted 6 lines
Quoted 3 lines

It almost sounds as though there is an attempt to eliminate resctrl's usage of a "max" value for the MIN control since that is effectively unknown to MPAM but that does not look possible to me?

Sorry but I haven't understood what your saying. What does "resctrl's usage of a "max" value for MIN control" mean?

Basically it is resctrl fs's validation of user input. Specifically, in bw_validate() where the fs does this range check:

	if (bw < r->membw.min_bw || bw > r->membw.max_bw)

resctrl fs thus uses the "max" value of a control for user input checking and it seemed to me that it may be difficult for MPAM to lean that "max" from all systems but it sounds as though the plan is instead to use the max that the architecture supports?

Reinette

maintainer
#32 lore raw

Hi Reinette,

Quoted 158 lines · Reinette Chatre

Hi Ben,

Quoted 140 lines · Ben Horgan
Quoted 119 lines · Reinette Chatre
Quoted 118 lines · Ben Horgan
Quoted 85 lines · Reinette Chatre

Hi Ben,

Quoted 3 lines · Ben Horgan
Quoted 2 lines · Reinette Chatre
Quoted 1 line · Ben Horgan

On 5/29/26 19:06, Reinette Chatre wrote:

...

Quoted 70 lines
Quoted 18 lines
Quoted 18 lines

I plumbed in support for the MB_MIN resource schema which also works under light testing. The only fs resctrl code change I needed was:

include/linux/resctrl.h +3 -0
@@ -483,6 +483,9 @@ static inline u32 resctrl_get_default_ctrlval(struct
 resctrl_ctrl *ctrl)
        case RESCTRL_CTRL_BITMAP:
                return BIT_MASK(ctrl->cache.cbm_len) - 1;
        case RESCTRL_CTRL_SCALAR:
+               if (ctrl->name == RESCTRL_CTRL_NAME_MIN)
+                       return ctrl->membw.min_bw;
+
                return ctrl->membw.max_bw;
        }
Quoted 9 lines
Quoted 5 lines

At least on MPAM systems, we use a default of 0 for minimum bandwidth controls as the maximum bandwidth controls only take effect if their value is higher than the minimum bandwidth value. I have specialised this on the ctrl->name which breaks your ctrl->type based classification but that's fixable by just adding a default field to membw.

This I am not sure about. In my understanding a typical "default" value means "no throttling" and, at least on Intel, this default hardware state has been summarized as "min" == "max" == "optimal".

Ok, this sounds odd to me but that is probably because I don't know what Intel systems do. On MPAM systems a MIN control is a boost rather than a throttling control. Although, you can always think of that as throttling the traffic with the other PARTIDs.

Quoted 3 lines

Are you saying that on MPAM systems if "min" == "max" then max bandwidth controls do not take effect? Could you please elaborate what happens if "min" == "max"?

Table 5-4 from section 5.2.8 of the IHI0099B.b shows the interaction between the min and maximum controls.

If used bandwidth is   The preference is  Description
Below the minimum      High               Only high requests compete with this

request.
Above the minimum:
Below the maximum Medium High requests are serviced first then

this request competes with other medium requests.

Above the maximum, Low Requests are not serviced if any high
when HARDLIM is 0 or medium requests are available.

Above the maximum, None Requests are not serviced
when HARDLIM is 1

So if we keep the minimum and the maximum controls values always the same then all traffic will be given "high" preference until the target bandwidth is reached. For some MPAM systems it is recommended to set the minimum value as 5% less than the maximum value to get a reliable target bandwidth. As 5% seems implementation specific and some systems don't have min controls it seemed better to just match the MB control with a maximum bandwidth control and let the user have freedom to choose the minimum bandwidth control when MB_MIN support is added.

If a default for the minimum of the maximum possible bandwidth is used (100%) then any change of the maximum won't have any effect as it's always less than minimum (if that's unchanged) and so all traffic is high preference. I now see from your reply below that you are planning on not allowing this kind of configuration.

Quoted 5 lines

If the minimum always tracks the maximum then we lose the distinction between medium and high preference traffic and so to reserve some high preference bandwidth for one control group we'd have to change the configuration in the other controls groups so that they're bandwidth preference is medium (minimum value at 0).

Quoted 8 lines

I do not think we are talking about the same thing here. I am *not* saying that minimum and maximum controls should always be the same.

The discussion is about a proposed change to resctrl_get_default_ctrlval(). resctrl uses this function in two places:
- When creating a new resource group:
The intention here is that when user space creates a new resource group it should be created with maximum allocations possible. For MBA this means "unthrottled".

I would contend that for minimum controls that a policy of 'maximum allocation possible' isn't a useful default. I try and explain a bit more below.

Quoted 9 lines

After creating the resource group user space can adjust allocations to match workload requirements.
- When unmounting the resctrl fs.
The intention here is that all controls are set to unthrottled to stop any possible impact to system when user space stops using resctrl.

resctrl_get_default_ctrlval() is thus intended to support an unthrottled baseline from where user space can make configuration changes as supported by hardware and required by workloads.

The baseline that I see makes most sense for a minimum control is to have the default as 0. This just means that there is no "guaranteed"/high preference bandwidth reserved for the control group. I would say this still unthrottled but just not giving a boost. With this default the user can use MB (backed by max bandwidth) without having to know about MB_MIN (keeping it constant). If the default is 100% for min bandwidth then the user needs to know to set MB_MIN to be able to use MB. Having a default of 100% for max bandwidth, correspondingly means a user can change MB_MIN and see guaranteed bandwidth effects without having to know about MB/MB_MAX.

Quoted 5 lines
Quoted 1 line

Does this make sense?

I see. I've only considered the original scenario where MPAM's MB is emulated with both MIN and MAX controls based on examples in
https://lore.kernel.org/lkml/[email protected]/

Ok, but I'm not sure why that's relevant. I feel we are talking at cross purposely here and so I'll try and answer your questions but I'm likely missing the point.

Quoted 9 lines

There seems to be two issues here:
a) Since some systems require MIN to be 5% less than MAX MPAM driver may not know what the system MIN and MAX difference should be to get optimal bandwidth.
b) Some MPAM systems have MAX control but not the MIN control.

The proposal to only emulate MB with MAX is clear but it is not obvious why, on a system that support both MIN and MAX, MB cannot be emulated with both as in the original proposal.

Quoted 9 lines

I don't see the advantage of emulating MB with both MIN and MAX. Just going by the MPAM specification, a system keeping MIN at 0 and just setting MAX from MB, (MIN=0, MAX=MB) should behave the same as one always setting both, (MIN=MB, MAX=MB). In the MIN=0 case there is never any high preference traffic and in the MIN=MAX_MB case there is never any medium preference traffic. It seemed best to not rely on any platform specific heuristics to try and guess what's better and just wait til the time we could support MB_MIN in resctrl (and leave the decision up to the user). My expectation was that this would be the simplest course of action.

This sounds fair. Two observations:
- The hierarchy exposed by resctrl may be different on systems that have the "same" controls.
For example, on an MPAM system (if I understand correctly) the user may see:
info/

Quoted 5 lines
  └── MB/
      └── resource_schemata/
          ├── MB/
          │   └── MB_MAX/
          └── MB_MIN/

Yes, this matches my understanding.

Quoted 9 lines

Compared with a possible implementation on Intel that looks like:
info/

  └── MB/
      └── resource_schemata/
          ├── MB/
          │   └── MB_OPT/
          ├── MB_MAX/
          └── MB_MIN/

Not sure if my understanding is correct here...
In the kernel today is it rdt max that backs MB? (Ignoring the sw controller) If so wouldn't the meaning of MB change within the same platform on a kernel upgrade once the rdt optimal support is added?

Quoted 19 lines
Quoted 1 line

On Intel these controls are optional so could even look more like MPAM:

info/

  └── MB/
      └── resource_schemata/
          └── MB/
             └── MB_MAX/

Dave Martin had some musings about how to present controls to user space with some semblance of consistency but we were not able to finalize that.
At this time it seems that there may be agreement on needing hierarchies as above but I am concerned about the complicated interface that results from inconsistencies between systems. While complicated it does represent the relationships accurately and thus could also be seen as working as intended ... so perhaps we just need to take great care in documenting this hierarchy mechanism as the source for control information as opposed to the schemata file.

Quoted 11 lines

- I see that the default of zero for the MIN control can be appropriate. For the usage of resctrl_get_default_ctrlval() when creating a new control group this could be ok as a general setting since the user is expected to, after creating the resource group, adjust control values as required by the workloads it will contain. I need to make sure about this from all architectures and if 0 is not ok then we could use Drew's suggestion of architecture providing a specific reset value.
For the other usage of resctrl_get_default_ctrlval() that resets controls on unmount there is not an opportunity for user space to adjust and here more care needs to be taken to match architecture requirements. The architecture provided "reset" value does sound more appealing.

Yes, if 0 isn't generally applicable we'll need an architecture provided value.

Quoted 5 lines
Quoted 5 lines

Is this motivated by (a) where MPAM driver just does not know

Quoted 1 line

what the "max" of a MIN control value should be?

Quoted 8 lines
Quoted 3 lines

What's the "max" of a MIN control?

The maximum value we can set it to?

The maximum value a user can set it to. I understood from an earlier comment that there are some MPAM systems where the MIN needs to be set to an implementation specific 5% less than MAX. I interpreted this to mean that it is difficult for the MPAM driver to always know what the maximum value is that a user can write to the MIN control.

Sorry for the confusion.

Quoted 17 lines
Quoted 10 lines

We can work that. For MPAM it's writing all 1s to the register which for the minumum case represents ((2**mbw_min_wd)/2**mbw_min_wd)) * 100 %

Just emulating MB with MAX control

Quoted 4 lines

does not seem to eliminate this problem since between the fs and arch resctrl still needs to ensure that when user space writes a control value to the MIN control that it is valid for the underlying MPAM system.

Quoted 6 lines
Quoted 3 lines

It almost sounds as though there is an attempt to eliminate resctrl's usage of a "max" value for the MIN control since that is effectively unknown to MPAM but that does not look possible to me?

Sorry but I haven't understood what your saying. What does "resctrl's usage of a "max" value for MIN control" mean?

Quoted 7 lines

Basically it is resctrl fs's validation of user input. Specifically, in bw_validate() where the fs does this range check:

	if (bw < r->membw.min_bw || bw > r->membw.max_bw)

resctrl fs thus uses the "max" value of a control for user input checking and it seemed to me that it may be difficult for MPAM to lean that "max" from all systems but it sounds as though the plan is instead to use the max that the architecture supports?

Yes, can just size the max based on the number of configuration bits in the h/w.

Thanks,

Ben

Quoted 3 lines

Reinette

author maintainer
#33 lore raw

Hi Ben,

Quoted 38 lines · Ben Horgan
Quoted 19 lines · Reinette Chatre
Quoted 12 lines · Ben Horgan

On 6/8/26 17:16, Reinette Chatre wrote:

I don't see the advantage of emulating MB with both MIN and MAX. Just going by the MPAM specification, a system keeping MIN at 0 and just setting MAX from MB, (MIN=0, MAX=MB) should behave the same as one always setting both, (MIN=MB, MAX=MB). In the MIN=0 case there is never any high preference traffic and in the MIN=MAX_MB case there is never any medium preference traffic. It seemed best to not rely on any platform specific heuristics to try and guess what's better and just wait til the time we could support MB_MIN in resctrl (and leave the decision up to the user). My expectation was that this would be the simplest course of action.

This sounds fair. Two observations:
- The hierarchy exposed by resctrl may be different on systems that have the "same" controls.
For example, on an MPAM system (if I understand correctly) the user may see:
info/

Quoted 5 lines
  └── MB/
      └── resource_schemata/
          ├── MB/
          │   └── MB_MAX/
          └── MB_MIN/

Yes, this matches my understanding.

Quoted 9 lines

Compared with a possible implementation on Intel that looks like:
info/

  └── MB/
      └── resource_schemata/
          ├── MB/
          │   └── MB_OPT/
          ├── MB_MAX/
          └── MB_MIN/
Quoted 2 lines

Not sure if my understanding is correct here...
In the kernel today is it rdt max that backs MB? (Ignoring the sw controller)

resctrl does not have support for the RDT "MAX" controller yet. Since resctrl was created as part of enabling RDT the resctrl MB control maps exactly to RDT's original percentage based memory delay value that is an approximate. Newer hardware support three controls: optimal, minimum, and maximum. These controls have finer granularity than what the default percentage based control supports so emulation is needed.
So far I assumed that on these systems the default MB control would be emulated by the new "optimal" control but after these exchanges I can see there being an argument for it to be emulated by the new "maximum" control also. Apart from it implying a cap there is also the idea that the "maximum" control is more likely to be available on all platforms.

Quoted 2 lines

If so wouldn't the meaning of MB change within the same platform on a kernel upgrade once the rdt optimal support is added?

Good point. Thank you for bringing this up. While the existing RDT MB control is approximate the RDT spec does contain the statement "... should be viewed as a maximum bandwidth “cap” per-CLOS." which is quite clear. I'll follow up with RDT folks on this.

...

Quoted 19 lines
Quoted 19 lines
Quoted 8 lines

We can work that. For MPAM it's writing all 1s to the register which for the minumum case represents ((2**mbw_min_wd)/2**mbw_min_wd)) * 100 %

Just emulating MB with MAX control

Quoted 4 lines

does not seem to eliminate this problem since between the fs and arch resctrl still needs to ensure that when user space writes a control value to the MIN control that it is valid for the underlying MPAM system.

Quoted 6 lines
Quoted 3 lines

It almost sounds as though there is an attempt to eliminate resctrl's usage of a "max" value for the MIN control since that is effectively unknown to MPAM but that does not look possible to me?

Sorry but I haven't understood what your saying. What does "resctrl's usage of a "max" value for MIN control" mean?

Basically it is resctrl fs's validation of user input. Specifically, in bw_validate() where the fs does this range check:

	if (bw < r->membw.min_bw || bw > r->membw.max_bw)
Quoted 5 lines
Quoted 3 lines

resctrl fs thus uses the "max" value of a control for user input checking and it seemed to me that it may be difficult for MPAM to lean that "max" from all systems but it sounds as though the plan is instead to use the max that the architecture supports?

Yes, can just size the max based on the number of configuration bits in the h/w.

Ack. Thank you for the clarification.

Reinette

#34 lore raw

Hi Reinette,

Quoted 44 lines · Reinette Chatre

Hi Ben,

Quoted 38 lines · Ben Horgan
Quoted 19 lines · Reinette Chatre
Quoted 12 lines · Ben Horgan

On 6/8/26 17:16, Reinette Chatre wrote:

I don't see the advantage of emulating MB with both MIN and MAX. Just going by the MPAM specification, a system keeping MIN at 0 and just setting MAX from MB, (MIN=0, MAX=MB) should behave the same as one always setting both, (MIN=MB, MAX=MB). In the MIN=0 case there is never any high preference traffic and in the MIN=MAX_MB case there is never any medium preference traffic. It seemed best to not rely on any platform specific heuristics to try and guess what's better and just wait til the time we could support MB_MIN in resctrl (and leave the decision up to the user). My expectation was that this would be the simplest course of action.

This sounds fair. Two observations:
- The hierarchy exposed by resctrl may be different on systems that have the "same" controls.
For example, on an MPAM system (if I understand correctly) the user may see:
info/

Quoted 5 lines
   └── MB/
       └── resource_schemata/
           ├── MB/
           │   └── MB_MAX/
           └── MB_MIN/

Yes, this matches my understanding.

Quoted 9 lines

Compared with a possible implementation on Intel that looks like:
info/

   └── MB/
       └── resource_schemata/
           ├── MB/
           │   └── MB_OPT/
           ├── MB_MAX/
           └── MB_MIN/
Quoted 2 lines

Not sure if my understanding is correct here...
In the kernel today is it rdt max that backs MB? (Ignoring the sw controller)

Quoted 12 lines

resctrl does not have support for the RDT "MAX" controller yet. Since resctrl was created as part of enabling RDT the resctrl MB control maps exactly to RDT's original percentage based memory delay value that is an approximate. Newer hardware support three controls: optimal, minimum, and maximum. These controls have finer granularity than what the default percentage based control supports so emulation is needed.
So far I assumed that on these systems the default MB control would be emulated by the new "optimal" control but after these exchanges I can see there being an argument for it to be emulated by the new "maximum" control also. Apart from it implying a cap there is also the idea that the "maximum" control is more likely to be available on all platforms.

Regarding the region-aware RDT case, I wonder if we actually need to emulate the legacy MB control using MB_MAX. First, when we refer to the "legacy" for region-aware RDT, I suppose it corresponds to "MSR access" plus "percentage-based control".

case 1:
If the platform does not support region-aware RDT (no ERDT table is detected), the MB is naturally the "legacy" MB, and the info directory would look like:

info

└── MB
        └── resource_schemata
                └── MB

case 2:If the platform supports region-aware RDT (i.e., ERDT parsing succeeds), then the structure looks like below:

info

└── MB
        └── resource_schema
                └── MB                  <=== legacy
                └── MB_REGION0_OPT
                └── MB_REGION1_OPT
                └── MB_REGION0_MIN
                └── MB_REGION1_MIX
                └── MB_REGION0_MAX
                └── MB_REGION1_MAX

The OS supports runtime switching between the legacy MB and region-aware MB, via a MB scope control like:
/sys/fs/resctrl/info/MB/resource_schemata/mode [legacy] native
If the user wants to use the legacy interface, the system will fall back to the default [MSR-based, percentage]; if the user wants to use the new interface, the region-aware interfaces will be used. Since the two modes are mutually exclusive, there seems to be no need to use MB_REGION0_MAX to emulate legacy MB.

thanks,
Chenyu

#35 lore raw

Hi Reinette,

Quoted 78 lines · Yu C

Hi Reinette,

Quoted 45 lines · Reinette Chatre

Hi Ben,

Quoted 39 lines · Ben Horgan
Quoted 20 lines · Reinette Chatre
Quoted 12 lines · Ben Horgan

On 6/8/26 17:16, Reinette Chatre wrote:

I don't see the advantage of emulating MB with both MIN and MAX. Just going by the MPAM specification, a system keeping MIN at 0 and just setting MAX from MB, (MIN=0, MAX=MB) should behave the same as one always setting both, (MIN=MB, MAX=MB). In the MIN=0 case there is never any high preference traffic and in the MIN=MAX_MB case there is never any medium preference traffic. It seemed best to not rely on any platform specific heuristics to try and guess what's better and just wait til the time we could support MB_MIN in resctrl (and leave the decision up to the user). My expectation was that this would be the simplest course of action.

This sounds fair. Two observations:
- The hierarchy exposed by resctrl may be different on systems that have the "same"
   controls.
   For example, on an MPAM system (if I understand correctly) the user may see:
   info/

Quoted 5 lines
   └── MB/
       └── resource_schemata/
           ├── MB/
           │   └── MB_MAX/
           └── MB_MIN/

Yes, this matches my understanding.

Quoted 9 lines

   Compared with a possible implementation on Intel that looks like:
   info/

   └── MB/
       └── resource_schemata/
           ├── MB/
           │   └── MB_OPT/
           ├── MB_MAX/
           └── MB_MIN/
Quoted 2 lines

Not sure if my understanding is correct here...
In the kernel today is it rdt max that backs MB? (Ignoring the sw controller)

Quoted 12 lines

resctrl does not have support for the RDT "MAX" controller yet. Since resctrl was created as part of enabling RDT the resctrl MB control maps exactly to RDT's original percentage based memory delay value that is an approximate. Newer hardware support three controls: optimal, minimum, and maximum. These controls have finer granularity than what the default percentage based control supports so emulation is needed.
So far I assumed that on these systems the default MB control would be emulated by the new "optimal" control but after these exchanges I can see there being an argument for it to be emulated by the new "maximum" control also. Apart from it implying a cap there is also the idea that the "maximum" control is more likely to be available on all platforms.

Regarding the region-aware RDT case, I wonder if we actually need to emulate the legacy MB control using MB_MAX. First, when we refer to the "legacy" for region-aware RDT, I suppose it corresponds to "MSR access" plus "percentage-based control".

case 1:
If the platform does not support region-aware RDT (no ERDT table is detected), the MB is naturally the "legacy" MB, and the info directory would look like:

info

└── MB
         └── resource_schemata
                 └── MB

case 2:If the platform supports region-aware RDT (i.e., ERDT parsing succeeds), then the structure looks like below:

info

Quoted 10 lines
└── MB
         └── resource_schema
                 └── MB                  <=== legacy
                 └── MB_REGION0_OPT
                 └── MB_REGION1_OPT
                 └── MB_REGION0_MIN
                 └── MB_REGION1_MIX
                 └── MB_REGION0_MAX
                 └── MB_REGION1_MAX

This may be slightly off-topic from MAX emulation, but I have another thought regarding multi-controllers for rdt_resource:
As we know, with N regions, an MB resource will have a total of N × 3 controllers. Given that the current PoC iterates through every controller within the resource in resctrl_resource_ctrl_get(), could this increase lookup latency?

I studied the cgroup code and found that each controller for a cgroup resource uses a dedicated cftype. For example:
static struct cftype memory_files[] = {
{ .name = "min", .write = memory_min_write, .seq_show =
memory_min_show },
{ .name = "max", .write = memory_max_write, .seq_show =
memory_max_show },

    ...
};

The min/max memory controllers can be accessed in O(1) time using:
of_cft(of) -> kn->priv, and cft->write(of, buf, ...)

rftype is resctrl's equivalent of cftype, and schemata is currently implemented as a single rftype. Would it make sense to define a separate rftype for each resctrl controller(or maybe in the future consider that this is not in a critical path)

thanks,
Chenyu

author maintainer
#36 lore raw

Hi Chenyu,

Quoted 91 lines · Yu C

Hi Reinette,

Quoted 77 lines · Yu C

Hi Reinette,

Quoted 44 lines · Reinette Chatre

Hi Ben,

Quoted 38 lines · Ben Horgan
Quoted 19 lines · Reinette Chatre
Quoted 12 lines · Ben Horgan

On 6/8/26 17:16, Reinette Chatre wrote:

I don't see the advantage of emulating MB with both MIN and MAX. Just going by the MPAM specification, a system keeping MIN at 0 and just setting MAX from MB, (MIN=0, MAX=MB) should behave the same as one always setting both, (MIN=MB, MAX=MB). In the MIN=0 case there is never any high preference traffic and in the MIN=MAX_MB case there is never any medium preference traffic. It seemed best to not rely on any platform specific heuristics to try and guess what's better and just wait til the time we could support MB_MIN in resctrl (and leave the decision up to the user). My expectation was that this would be the simplest course of action.

This sounds fair. Two observations:
- The hierarchy exposed by resctrl may be different on systems that have the "same" controls.
   For example, on an MPAM system (if I understand correctly) the user may see:
   info/

Quoted 5 lines
   └── MB/
       └── resource_schemata/
           ├── MB/
           │   └── MB_MAX/
           └── MB_MIN/

Yes, this matches my understanding.

Quoted 9 lines

   Compared with a possible implementation on Intel that looks like:
   info/

   └── MB/
       └── resource_schemata/
           ├── MB/
           │   └── MB_OPT/
           ├── MB_MAX/
           └── MB_MIN/
Quoted 2 lines

Not sure if my understanding is correct here...
In the kernel today is it rdt max that backs MB? (Ignoring the sw controller)

Quoted 12 lines

resctrl does not have support for the RDT "MAX" controller yet. Since resctrl was created as part of enabling RDT the resctrl MB control maps exactly to RDT's original percentage based memory delay value that is an approximate. Newer hardware support three controls: optimal, minimum, and maximum. These controls have finer granularity than what the default percentage based control supports so emulation is needed.
So far I assumed that on these systems the default MB control would be emulated by the new "optimal" control but after these exchanges I can see there being an argument for it to be emulated by the new "maximum" control also. Apart from it implying a cap there is also the idea that the "maximum" control is more likely to be available on all platforms.

Regarding the region-aware RDT case, I wonder if we actually need to emulate the legacy MB control using MB_MAX. First, when we refer to the "legacy" for region-aware RDT, I suppose it corresponds to "MSR access" plus "percentage-based control".

case 1:
If the platform does not support region-aware RDT (no ERDT table is detected), the MB is naturally the "legacy" MB, and the info directory would look like:

info

└── MB
         └── resource_schemata
                 └── MB

case 2:If the platform supports region-aware RDT (i.e., ERDT parsing succeeds), then the structure looks like below:

info

Quoted 10 lines
└── MB
         └── resource_schema
                 └── MB                  <=== legacy
                 └── MB_REGION0_OPT
                 └── MB_REGION1_OPT
                 └── MB_REGION0_MIN
                 └── MB_REGION1_MIX
                 └── MB_REGION0_MAX
                 └── MB_REGION1_MAX
Quoted 6 lines

This may be slightly off-topic from MAX emulation, but I have another thought regarding multi-controllers for rdt_resource:
As we know, with N regions, an MB resource will have a total of N × 3 controllers. Given that the current PoC iterates through every controller within the resource in resctrl_resource_ctrl_get(), could this increase lookup latency?

resctrl_resource_ctrl_get() is only called when the user writes to the schemata file and since user can modify any of the enabled controls with a write to the schemata file it is necessary to iterate through all controls to find a match.

Quoted 8 lines

I studied the cgroup code and found that each controller for a cgroup resource uses a dedicated cftype. For example:
static struct cftype memory_files[] = {
    { .name = "min",  .write = memory_min_write,  .seq_show = memory_min_show }, { .name = "max",  .write = memory_max_write,  .seq_show = memory_max_show },
    ...
};

The PoC currently has "static struct rftype ctrl_files[] " that associates an rftype with every property of a control which enables all user space interactions with the individual controls to be direct.

Quoted 7 lines

The min/max memory controllers can be accessed in O(1) time using:
of_cft(of) -> kn->priv, and cft->write(of, buf, ...)

rftype is resctrl's equivalent of cftype, and schemata is currently implemented as a single rftype. Would it make sense to define a separate rftype for each resctrl controller(or maybe in the future consider that this is not in a critical path)

Your suggestion is not clear to me. The schemata file is associated with a control group, a resource group that has multiple allocations each backed by a different controller.

I do not think I fully understand your suggestion so would appreciate if you could provide more detail.

Reinette

#37 lore raw

Hi Reinette,

Quoted 2 lines · Reinette Chatre

Hi Chenyu,

[ ... ]

Quoted 35 lines
Quoted 28 lines
Quoted 17 lines

Regarding the region-aware RDT case, I wonder if we actually need to emulate the legacy MB control using MB_MAX. First, when we refer to the "legacy" for region-aware RDT, I suppose it corresponds to "MSR access" plus "percentage-based control".

case 1:
If the platform does not support region-aware RDT (no ERDT table is detected), the MB is naturally the "legacy" MB, and the info directory would look like:

info

└── MB
          └── resource_schemata
                  └── MB

case 2:If the platform supports region-aware RDT (i.e., ERDT parsing succeeds), then the structure looks like below:

info

Quoted 10 lines
└── MB
          └── resource_schema
                  └── MB                  <=== legacy
                  └── MB_REGION0_OPT
                  └── MB_REGION1_OPT
                  └── MB_REGION0_MIN
                  └── MB_REGION1_MIX
                  └── MB_REGION0_MAX
                  └── MB_REGION1_MAX
Quoted 6 lines

This may be slightly off-topic from MAX emulation, but I have another thought regarding multi-controllers for rdt_resource:
As we know, with N regions, an MB resource will have a total of N × 3 controllers. Given that the current PoC iterates through every controller within the resource in resctrl_resource_ctrl_get(), could this increase lookup latency?

Quoted 5 lines

resctrl_resource_ctrl_get() is only called when the user writes to the schemata file and since user can modify any of the enabled controls with a write to the schemata file it is necessary to iterate through all controls to find a match.

Yes, I was thinking if we can speed up this write by not iterating the control list.

Quoted 9 lines
Quoted 8 lines

I studied the cgroup code and found that each controller for a cgroup resource uses a dedicated cftype. For example:
static struct cftype memory_files[] = {
    { .name = "min",  .write = memory_min_write,  .seq_show = memory_min_show }, { .name = "max",  .write = memory_max_write,  .seq_show = memory_max_show },
    ...
};

Quoted 4 lines

The PoC currently has "static struct rftype ctrl_files[] " that associates an rftype with every property of a control which enables all user space interactions with the individual controls to be direct.

Yes we have dedicated rftype for the read-only info/properties.

Quoted 8 lines
Quoted 7 lines

The min/max memory controllers can be accessed in O(1) time using:
of_cft(of) -> kn->priv, and cft->write(of, buf, ...)

rftype is resctrl's equivalent of cftype, and schemata is currently implemented as a single rftype. Would it make sense to define a separate rftype for each resctrl controller(or maybe in the future consider that this is not in a critical path)

Quoted 6 lines

Your suggestion is not clear to me. The schemata file is associated with a control group, a resource group that has multiple allocations each backed by a different controller.

I do not think I fully understand your suggestion so would appreciate if you could provide more detail.

Sorry, I did not describe this precisely. My original thought is that when writing schemata, we need to locate the control by iterating through the control list:
rdtgroup_schemata_write
for_each_resource_ctrl(ctrl, r)

    if (!strncmp())
      return ctrl;

parse_line("0=80;1=60", ctrl)

I just wonder if we can skip iterating the control list, by giving each control a dedicated rftype(and rftype needs to introduce a pointer struct resctrl_ctrl *ctrl) rdtgroup_ctrl_schemata_write()
rft = of->kn->priv
parse_line("0=80;1=60", rft->ctrl)

thanks,
Chenyu

author maintainer
#38 lore raw

Hi Chenyu,

Quoted 10 lines · Yu C

On 6/11/2026 12:13 AM, Reinette Chatre wrote:

Sorry, I did not describe this precisely. My original thought is that when writing schemata, we need to locate the control by iterating through the control list:
rdtgroup_schemata_write
  for_each_resource_ctrl(ctrl, r)
    if (!strncmp())
      return ctrl;
  parse_line("0=80;1=60", ctrl)

Quoted 5 lines

I just wonder if we can skip iterating the control list, by giving each control a dedicated rftype(and rftype needs to introduce a pointer struct resctrl_ctrl *ctrl) rdtgroup_ctrl_schemata_write()
  rft = of->kn->priv

I interpret this to mean that this new "rftype" would be connected to the private data associated with the kernfs node associated with a resctrl file. Which resctrl file do you have in mind here? The schemata resctrl file has a kn but its private data points to the resource group to which the schemata file belongs. The only way to have private data of a resctrl file point to a control is if that file is clearly associated with just that one control. This is true for the files in the new resource_schemata directory but is not true for the files associated with a resource group.
When viewing the schemata file associated with a resource group the user expects to access all controls, not just one.

Quoted 2 lines

  parse_line("0=80;1=60", rft->ctrl)

Reinette

author maintainer
#39 lore raw
In reply to Chen, Yu C

Hi Chenyu,

Quoted 87 lines · Yu C

Hi Reinette,

Quoted 44 lines · Reinette Chatre

Hi Ben,

Quoted 38 lines · Ben Horgan
Quoted 19 lines · Reinette Chatre
Quoted 12 lines · Ben Horgan

On 6/8/26 17:16, Reinette Chatre wrote:

I don't see the advantage of emulating MB with both MIN and MAX. Just going by the MPAM specification, a system keeping MIN at 0 and just setting MAX from MB, (MIN=0, MAX=MB) should behave the same as one always setting both, (MIN=MB, MAX=MB). In the MIN=0 case there is never any high preference traffic and in the MIN=MAX_MB case there is never any medium preference traffic. It seemed best to not rely on any platform specific heuristics to try and guess what's better and just wait til the time we could support MB_MIN in resctrl (and leave the decision up to the user). My expectation was that this would be the simplest course of action.

This sounds fair. Two observations:
- The hierarchy exposed by resctrl may be different on systems that have the "same" controls.
   For example, on an MPAM system (if I understand correctly) the user may see:
   info/

Quoted 5 lines
   └── MB/
       └── resource_schemata/
           ├── MB/
           │   └── MB_MAX/
           └── MB_MIN/

Yes, this matches my understanding.

Quoted 9 lines

   Compared with a possible implementation on Intel that looks like:
   info/

   └── MB/
       └── resource_schemata/
           ├── MB/
           │   └── MB_OPT/
           ├── MB_MAX/
           └── MB_MIN/
Quoted 2 lines

Not sure if my understanding is correct here...
In the kernel today is it rdt max that backs MB? (Ignoring the sw controller)

Quoted 12 lines

resctrl does not have support for the RDT "MAX" controller yet. Since resctrl was created as part of enabling RDT the resctrl MB control maps exactly to RDT's original percentage based memory delay value that is an approximate. Newer hardware support three controls: optimal, minimum, and maximum. These controls have finer granularity than what the default percentage based control supports so emulation is needed.
So far I assumed that on these systems the default MB control would be emulated by the new "optimal" control but after these exchanges I can see there being an argument for it to be emulated by the new "maximum" control also. Apart from it implying a cap there is also the idea that the "maximum" control is more likely to be available on all platforms.

Regarding the region-aware RDT case, I wonder if we actually need to emulate the legacy MB control using MB_MAX. First, when we refer to the "legacy" for region-aware RDT, I suppose it corresponds to "MSR access" plus "percentage-based control".

case 1:
If the platform does not support region-aware RDT (no ERDT table is detected), the MB is naturally the "legacy" MB, and the info directory would look like:

info

└── MB
        └── resource_schemata
                └── MB

case 2:If the platform supports region-aware RDT (i.e., ERDT parsing succeeds), then the structure looks like below:

info

└── MB
        └── resource_schema
                └── MB                  <=== legacy
                └── MB_REGION0_OPT
                └── MB_REGION1_OPT
                └── MB_REGION0_MIN
                └── MB_REGION1_MIX
                └── MB_REGION0_MAX
                └── MB_REGION1_MAX
Quoted 9 lines

The OS supports runtime switching between the legacy MB and region-aware MB, via a MB scope control like:
/sys/fs/resctrl/info/MB/resource_schemata/mode [legacy] native
If the user wants to use the legacy interface, the system will fall back to the default [MSR-based, percentage]; if the user wants to use the new interface, the region-aware interfaces will be used. Since the two modes are mutually exclusive, there seems to be no need to use MB_REGION0_MAX to emulate legacy MB.

- This implies that all current and future ERDT capable systems will keep supporting MSR access with the legacy "percentage based control". Is this accurate? What will behavior be on an ERDT system that does not support MSR access with the percentage-based control?
- It is not obvious to me what you plan to have as default interface - I believe this should be "legacy" interface to ensure this will not break user space. resctrl should only switch to "native" after user demonstrates (by switching) familiarity with the new interface.

Reinette

#40 lore raw

Hi Reinette,

Quoted 2 lines · Reinette Chatre

Hi Chenyu,

[ ... ]

Quoted 37 lines
Quoted 27 lines

Regarding the region-aware RDT case, I wonder if we actually need to emulate the legacy MB control using MB_MAX. First, when we refer to the "legacy" for region-aware RDT, I suppose it corresponds to "MSR access" plus "percentage-based control".

case 1:
If the platform does not support region-aware RDT (no ERDT table is detected), the MB is naturally the "legacy" MB, and the info directory would look like:

info

└── MB
         └── resource_schemata
                 └── MB

case 2:If the platform supports region-aware RDT (i.e., ERDT parsing succeeds), then the structure looks like below:

info

└── MB
         └── resource_schema
                 └── MB                  <=== legacy
                 └── MB_REGION0_OPT
                 └── MB_REGION1_OPT
                 └── MB_REGION0_MIN
                 └── MB_REGION1_MIX
                 └── MB_REGION0_MAX
                 └── MB_REGION1_MAX
Quoted 9 lines

The OS supports runtime switching between the legacy MB and region-aware MB, via a MB scope control like:
/sys/fs/resctrl/info/MB/resource_schemata/mode [legacy] native
If the user wants to use the legacy interface, the system will fall back to the default [MSR-based, percentage]; if the user wants to use the new interface, the region-aware interfaces will be used. Since the two modes are mutually exclusive, there seems to be no need to use MB_REGION0_MAX to emulate legacy MB.

Quoted 3 lines

- This implies that all current and future ERDT capable systems will keep supporting MSR access with the legacy "percentage based control". Is this accurate?

I don't know the answer for now; it may depend on how the hardware maintains backward compatibility. Let me have a check.

Quoted 2 lines

What will behavior be on an ERDT system that does not support
MSR access with the percentage-based control?

Then in this case, we may need to emulate the legacy MBA if users do not wish to switch to the new interface.

Quoted 5 lines

- It is not obvious to me what you plan to have as default interface - I believe this should be "legacy" interface to ensure this will not break user space. resctrl should only switch to "native" after user demonstrates (by switching) familiarity with the new interface.

Yes, setting "legacy" as the default will accommodate users who rely on the old interfaces.

Quoted 4 lines

Reinette

#41 lore raw
In reply to Reinette Chatre

Hi Reinette,

Quoted 2 lines · Reinette Chatre

Hi Chenyu,

[ ... ]

Quoted 5 lines

- This implies that all current and future ERDT capable systems will keep supporting MSR access with the legacy "percentage based control". Is this accurate? What will behavior be on an ERDT system that does not support MSR access with the percentage-based control?

Would it make sense to encourage users to switch to MMIO-based access if the platform replaces MSR interfaces with MMIO-based ones? One issue with emulating MSR-based access is that there may not be a strict 1:1 mapping between the MSR range and the MMIO range. Additionally, I am unsure whether both are strictly linearly scalable, so it could be challenging to create a proper mapping for msr_value_emulated = map(mmio_value) IIUC.

thanks,
Chenyu

author maintainer
#42 lore raw

Hi Chenyu,

Quoted 9 lines · Yu C

On 6/10/2026 11:59 PM, Reinette Chatre wrote:
[ ... ]

Quoted 5 lines

-  This implies that all current and future ERDT capable systems will keep supporting MSR access with the legacy "percentage based control". Is this accurate? What will behavior be on an ERDT system that does not support MSR access with the percentage-based control?

Quoted 6 lines

Would it make sense to encourage users to switch to MMIO-based access if the platform replaces MSR interfaces with MMIO-based ones? One issue with emulating MSR-based access is that there may not be a strict 1:1 mapping between the MSR range and the MMIO range. Additionally, I am unsure whether both are strictly linearly scalable, so it could be challenging to create a proper mapping for msr_value_emulated = map(mmio_value) IIUC.

resctrl needs to continue to offer the MB percentage based control to not break any existing tools that may take a while to transition to the new control interface.

I believe all architectures will encourage their users to use the finer grained controls while I also expect users and their tools would gradually switch by themselves to benefit from the improved capabilities.

Reinette

#43 lore raw

Hi Reinette,

Quoted 19 lines · Reinette Chatre

Hi Chenyu,

Quoted 9 lines · Yu C

On 6/10/2026 11:59 PM, Reinette Chatre wrote:
[ ... ]

Quoted 5 lines

-  This implies that all current and future ERDT capable systems will keep
    supporting MSR access with the legacy "percentage based control". Is this accurate? What will behavior be on an ERDT system that does not support MSR access with the percentage-based control?

Quoted 6 lines

Would it make sense to encourage users to switch to MMIO-based access if the platform replaces MSR interfaces with MMIO-based ones? One issue with emulating MSR-based access is that there may not be a strict 1:1 mapping between the MSR range and the MMIO range. Additionally, I am unsure whether both are strictly linearly scalable, so it could be challenging to create a proper mapping for msr_value_emulated = map(mmio_value) IIUC.

Quoted 7 lines

resctrl needs to continue to offer the MB percentage based control to not break any existing tools that may take a while to transition to the new control interface.

I believe all architectures will encourage their users to use the finer grained controls while I also expect users and their tools would gradually switch by themselves to benefit from the improved capabilities.

The following are current thoughts on how Control B can emulate Control A. May I know if this direction is acceptable?

The main idea is that, Legacy MBA controller delegates to region-aware MBA controller via a pointer, with its hw_update function converting percentages to the target's value range.

struct resctrl_hw_ctrl {
	struct resctrl_ctrl  r_ctrl;
	unsigned int         msr_base;
	void                 (*hw_update)(struct hw_param *m);
	/* points to the backing controller */
	struct resctrl_ctrl  *r_ctrl_emul;  <-- newly added
};

On boot, region-aware controllers shall be created first, followed by the legacy MBA controller. When ERDT is enabled for legacy MBA, hw_update is assigned to mba_emul_update() instead of mba_wrmsr_intel(). Meanwhile, r_ctrl_emul points to the MAX controller of region0.

In mba_emul_update(), the legacy MSR value is scaled to the range of region0 MAX controller by referencing r_ctrl_emul->membw.max_bw and r_ctrl_emul->membw.min_bw. This maps the original input range [1, 100] to a new range [1, L], where L <= 255.

This emulation logic can also be extended to other controllers.

thanks,
Chenyu

maintainer
#44 lore raw

Hi Chenyu,

Quoted 50 lines · Yu C

Hi Reinette,

Quoted 32 lines · Reinette Chatre

Hi Chenyu,

Quoted 12 lines · Yu C

On 6/10/2026 11:59 PM, Reinette Chatre wrote:
[ ... ]

Quoted 8 lines

-  This implies that all current and future ERDT capable systems will keep
     supporting MSR access with the legacy "percentage based control". Is this
     accurate? What will behavior be on an ERDT system that does not support
     MSR access with the percentage-based control?

Quoted 11 lines

Would it make sense to encourage users to switch to MMIO-based access if the
platform replaces MSR interfaces with MMIO-based ones? One issue with emulating
MSR-based access is that there may not be a strict 1:1 mapping between the MSR
range and the MMIO range. Additionally, I am unsure whether both are strictly
linearly scalable, so it could be challenging to create a proper mapping for
msr_value_emulated = map(mmio_value) IIUC.

resctrl needs to continue to offer the MB percentage based control to not break any
existing tools that may take a while to transition to the new control interface.

Quoted 6 lines

I believe all architectures will encourage their users to use the finer grained
controls while I also expect users and their tools would gradually switch by themselves
to benefit from the improved capabilities.

The following are current thoughts on how Control B can emulate Control A. May I know if this direction is acceptable?

The main idea is that, Legacy MBA controller delegates to region-aware MBA controller via a pointer, with its hw_update function converting percentages
to the target's value range.

Quoted 7 lines

struct resctrl_hw_ctrl {
    struct resctrl_ctrl  r_ctrl;
    unsigned int         msr_base;
    void                 (*hw_update)(struct hw_param *m); /* points to the backing controller */
    struct resctrl_ctrl  *r_ctrl_emul;  <-- newly added };

What's the reason for this being in the architecture specific code? I would expect that the resctrl core code would need to know about any emulation even if the exact emulation is architecture specific?

Thanks,

Ben

Quoted 20 lines

On boot, region-aware controllers shall be created first, followed by the legacy MBA controller. When ERDT is enabled for legacy MBA, hw_update is assigned to mba_emul_update() instead of mba_wrmsr_intel(). Meanwhile, r_ctrl_emul points to the MAX controller of region0.

In mba_emul_update(), the legacy MSR value is scaled to the range of region0
MAX controller by referencing r_ctrl_emul->membw.max_bw and r_ctr_emul-

Quoted 1 line

membw.min_bw.

This maps the original input range [1, 100] to a new range [1, L], where L <= 255.

This emulation logic can also be extended to other controllers.

thanks,
Chenyu

#45 lore raw

Hi Ben,

Quoted 1 line · Ben Horgan

Hi Chenyu,

[ ... ]

Quoted 16 lines
Quoted 8 lines

The following are current thoughts on how Control B can emulate Control A. May I know if this direction is acceptable?

The main idea is that, Legacy MBA controller delegates to region-aware MBA controller via a pointer, with its hw_update function converting percentages
to the target's value range.

Quoted 7 lines

struct resctrl_hw_ctrl {
    struct resctrl_ctrl  r_ctrl;
    unsigned int         msr_base;
    void                 (*hw_update)(struct hw_param *m); /* points to the backing controller */
    struct resctrl_ctrl  *r_ctrl_emul;  <-- newly added
};

Quoted 4 lines

What's the reason for this being in the architecture specific code? I would expect that the resctrl core code would need to know about any emulation even if the exact emulation is architecture specific?

I was thinking that the low-level emulation was implemented by the hardware/architecture, so I put it within resctrl_hw_ctrl. But you are right - the generic resctrl might need to be aware of the emulation, because the "info" directory needs to display the emulation hierarchy. Let me have a try at moving it into the generic resctrl, like into struct resctrl_ctrl.

thanks,
Chenyu

author maintainer
#46 lore raw
In reply to Chen, Yu C

Hi Chenyu,

Quoted 41 lines · Yu C
Quoted 17 lines · Reinette Chatre
Quoted 9 lines · Yu C

On 6/10/2026 11:59 PM, Reinette Chatre wrote:
[ ... ]

Quoted 5 lines

-  This implies that all current and future ERDT capable systems will keep supporting MSR access with the legacy "percentage based control". Is this accurate? What will behavior be on an ERDT system that does not support MSR access with the percentage-based control?

Quoted 6 lines

Would it make sense to encourage users to switch to MMIO-based access if the platform replaces MSR interfaces with MMIO-based ones? One issue with emulating MSR-based access is that there may not be a strict 1:1 mapping between the MSR range and the MMIO range. Additionally, I am unsure whether both are strictly linearly scalable, so it could be challenging to create a proper mapping for msr_value_emulated = map(mmio_value) IIUC.

Quoted 7 lines

resctrl needs to continue to offer the MB percentage based control to not break any existing tools that may take a while to transition to the new control interface.

I believe all architectures will encourage their users to use the finer grained controls while I also expect users and their tools would gradually switch by themselves to benefit from the improved capabilities.

The following are current thoughts on how Control B can emulate Control A. May I know if this direction is acceptable?

The main idea is that, Legacy MBA controller delegates to region-aware MBA controller via a pointer, with its hw_update function converting percentages to the target's value range.

struct resctrl_hw_ctrl {
    struct resctrl_ctrl  r_ctrl;
    unsigned int         msr_base;
    void                 (*hw_update)(struct hw_param *m); /* points to the backing controller */
    struct resctrl_ctrl  *r_ctrl_emul;  <-- newly added };

Quoted 4 lines

On boot, region-aware controllers shall be created first, followed by the legacy MBA controller. When ERDT is enabled for legacy MBA, hw_update is assigned to mba_emul_update() instead of mba_wrmsr_intel(). Meanwhile, r_ctrl_emul points to the MAX controller of region0.

To maintain the generic "memory bandwidth allocation" I was expecting legacy MBA to be emulated by MAX controllers of all regions, not just one. Could you please elaborate how emulating with just one region is preferred?

Quoted 9 lines

In mba_emul_update(), the legacy MSR value is scaled to the range of region0 MAX controller by referencing r_ctrl_emul->membw.max_bw and r_ctrl_emul->membw.min_bw. This maps the original input range [1, 100] to a new range [1, L], where L <= 255.

This emulation logic can also be extended to other controllers.

thanks,
Chenyu

Reinette

#47 lore raw

Hi Reinette,

Quoted 44 lines · Reinette Chatre

Hi Chenyu,

Quoted 41 lines · Yu C
Quoted 17 lines · Reinette Chatre
Quoted 9 lines · Yu C

On 6/10/2026 11:59 PM, Reinette Chatre wrote:
[ ... ]

Quoted 5 lines

-  This implies that all current and future ERDT capable systems will keep
     supporting MSR access with the legacy "percentage based control". Is this accurate? What will behavior be on an ERDT system that does not support MSR access with the percentage-based control?

Quoted 6 lines

Would it make sense to encourage users to switch to MMIO-based access if the platform replaces MSR interfaces with MMIO-based ones? One issue with emulating MSR-based access is that there may not be a strict 1:1 mapping between the MSR range and the MMIO range. Additionally, I am unsure whether both are strictly linearly scalable, so it could be challenging to create a proper mapping for msr_value_emulated = map(mmio_value) IIUC.

Quoted 7 lines

resctrl needs to continue to offer the MB percentage based control to not break any existing tools that may take a while to transition to the new control interface.

I believe all architectures will encourage their users to use the finer grained controls while I also expect users and their tools would gradually switch by themselves to benefit from the improved capabilities.

The following are current thoughts on how Control B can emulate Control A. May I know if this direction is acceptable?

The main idea is that, Legacy MBA controller delegates to region-aware MBA controller via a pointer, with its hw_update function converting percentages to the target's value range.

struct resctrl_hw_ctrl {
    struct resctrl_ctrl  r_ctrl;
    unsigned int         msr_base;
    void                 (*hw_update)(struct hw_param *m); /* points to the backing controller */
    struct resctrl_ctrl  *r_ctrl_emul;  <-- newly added
};

Quoted 9 lines
Quoted 4 lines

On boot, region-aware controllers shall be created first, followed by the legacy MBA controller. When ERDT is enabled for legacy MBA, hw_update is assigned to mba_emul_update() instead of mba_wrmsr_intel(). Meanwhile, r_ctrl_emul points to the MAX controller of region0.

To maintain the generic "memory bandwidth allocation" I was expecting legacy MBA to be emulated by MAX controllers of all regions, not just one. Could you please elaborate how emulating with just one region is preferred?

You are right, legacy MBA introduces the request rate controller between the core (L2) and the L3, so it controls traffic to both DRAM, CXL,etc. Let me think about how to use a linked list for this emulation.

thanks,
Chenyu

#48 lore raw
In reply to Reinette Chatre
Quoted 7 lines · Reinette Chatre

- I see that the default of zero for the MIN control can be appropriate. For the usage of resctrl_get_default_ctrlval() when creating a new control group this could be ok as a general setting since the user is expected to, after creating the resource group, adjust control values as required by the workloads it will contain. I need to make sure about this from all architectures and if 0 is not ok then we could use Drew's suggestion of architecture providing a specific reset value.

Unfortunately, 0 will not work for the default value for Reserved bandwidth Blocks (Rbwb) in RISC-V CBQRI as it is required by the spec to be at least 1.

Thanks,
Drew

author maintainer
#49 lore raw

Hi Drew,

Quoted 9 lines · Drew Fustini
Quoted 7 lines · Reinette Chatre

- I see that the default of zero for the MIN control can be appropriate. For the usage of resctrl_get_default_ctrlval() when creating a new control group this could be ok as a general setting since the user is expected to, after creating the resource group, adjust control values as required by the workloads it will contain. I need to make sure about this from all architectures and if 0 is not ok then we could use Drew's suggestion of architecture providing a specific reset value.

Quoted 3 lines

Unfortunately, 0 will not work for the default value for Reserved bandwidth Blocks (Rbwb) in RISC-V CBQRI as it is required by the spec to be at least 1.

My mistake, I should not have written a hardcoded 0 but instead mirrored Ben's original suggestion of using ctrl->membw.min_bw.

Reinette

maintainer
#50 lore raw
In reply to Reinette Chatre

Reinette,

Tiny bug in "mpam,x86,fs/resctrl: Transition resource control to a list"

Quoted 5 lines

/*
* Return length needed to display longest control suffix.
* Add 1 for the "_" character when control name exists. */
size_t resctrl_resource_ctrl_max_len(struct rdt_resource *r)

Quoted 9 lines
{
	struct resctrl_ctrl *ctrl;
	size_t total = 0;
	size_t len;

	for_each_resource_ctrl(ctrl,r) {
		len = strlen(resctrl_ctrl_name_str(ctrl->name));
		if (len)
			total += 1 + len;

Should be:

			total = max(total, 1 + len);
Quoted 4 lines
	}

	return total;
}

Your sample code just converts "MB" over to using a list of controls. Would it also work for code & data priority? E.g. by having controls for "_CODE" and "_DATA" that are on the list when CDP is enabled, and just a no name control for the default?

-Tony

maintainer
#51 lore raw

Hi Tony,

Quoted 25 lines · Tony

Reinette,

Tiny bug in "mpam,x86,fs/resctrl: Transition resource control to a list"

Quoted 5 lines

/*
* Return length needed to display longest control suffix.
* Add 1 for the "_" character when control name exists. */
size_t resctrl_resource_ctrl_max_len(struct rdt_resource *r)

Quoted 9 lines
{
	struct resctrl_ctrl *ctrl;
	size_t total = 0;
	size_t len;

	for_each_resource_ctrl(ctrl,r) {
		len = strlen(resctrl_ctrl_name_str(ctrl->name));
		if (len)
			total += 1 + len;

Should be:

			total = max(total, 1 + len);
Quoted 2 lines
	}
Quoted 7 lines
Quoted 2 lines
	return total;
}

Your sample code just converts "MB" over to using a list of controls. Would it also work for code & data priority? E.g. by having controls for "_CODE" and "_DATA" that are on the list when CDP is enabled, and just a no name control for the default?

Is there a plan to expand to support CDP for things other than cache allocation? This would help for MPAM as we currently don't support the MB schema when CDP is enabled for L2 or L3 as we have no way of reliably supporting it for the caches but not the memory allocation.

Thanks,

Ben

Quoted 2 lines

-Tony

author maintainer
#52 lore raw
In reply to Luck, Tony

Hi Tony,

Quoted 14 lines · Tony

Reinette,

Tiny bug in "mpam,x86,fs/resctrl: Transition resource control to a list"

Quoted 10 lines

/*
* Return length needed to display longest control suffix.
* Add 1 for the "_" character when control name exists. */
size_t resctrl_resource_ctrl_max_len(struct rdt_resource *r)

{
	struct resctrl_ctrl *ctrl;
	size_t total = 0;
	size_t len;
Quoted 8 lines
Quoted 4 lines
	for_each_resource_ctrl(ctrl,r) {
		len = strlen(resctrl_ctrl_name_str(ctrl->name));
		if (len)
			total += 1 + len;

Should be:

			total = max(total, 1 + len);

Thank you very much. I squashed this locally.

Quoted 10 lines
Quoted 5 lines
	}

	return total;
}

Your sample code just converts "MB" over to using a list of controls. Would it also work for code & data priority? E.g. by having controls for "_CODE" and "_DATA" that are on the list when CDP is enabled, and just a no name control for the default?

This PoC also converts existing cache allocation to use a (one entry) list of controls. The current idea is that whether CDP is enabled or not there is just one list of controls associated with the resource but when CDP is enabled every domain associated with every control can accommodate both a "CODE" and "DATA" control value.

For cache controls this PoC indeed still has many assumptions about there being only one cache control. Even with the single control there are many CDP special cases so there may be some unexplored corners when adding multiple cache controls.

Reinette

#53 lore raw

Hi Reinette,

Quoted 22 lines · Reinette Chatre

Hi Tony,

Quoted 19 lines · Tony

Reinette,

Tiny bug in "mpam,x86,fs/resctrl: Transition resource control to a list"

Quoted 14 lines

/*
* Return length needed to display longest control suffix.
* Add 1 for the "_" character when control name exists. */
size_t resctrl_resource_ctrl_max_len(struct rdt_resource *r)

{
	struct resctrl_ctrl *ctrl;
	size_t total = 0;
	size_t len;

	for_each_resource_ctrl(ctrl,r) {
		len = strlen(resctrl_ctrl_name_str(ctrl->name));
		if (len)
			total += 1 + len;
Quoted 6 lines
Quoted 3 lines

Should be:

			total = max(total, 1 + len);

Thank you very much. I squashed this locally.

A minor alignment schemata issue is found when testing region-aware RDT:
[resctrl]# cat schemata

            MB:0=100
            MB_REGION0_OPT:0=255
            MB_REGION0_MIN:0=255
            MB_REGION0_MAX:0=255
            MB_REGION1_OPT:0=255
            MB_REGION1_MIN:0=255
            MB_REGION1_MAX:0=255
            L2:0=ffff;2=ffff;4=ffff
            L3:0=3ff

The schemata file right-justified only the resource name in a max_name_width field, then printed the control suffix afterwards without padding. Since max_name_width reserves space for name plus suffix, this left a run of spaces before every line. A small hacky fix:

fs/resctrl/ctrlmondata.c +0 -0
index 9b51464b66b7..7d517f3457bf 100644
@@ -521,6 +521,7 @@ ssize_t rdtgroup_schemata_write(struct kernfs_open_file *of,
 static void show_doms(struct seq_file *s, struct rdt_resource_final *f,
                      bool print_ctrl, int closid, struct resctrl_ctrl 

*ctrl)
{

+       char name[20];
        struct rdt_resource *r = f->res;
        struct rdt_ctrl_domain *dom;
        bool sep = false;
@@ -529,11 +530,13 @@ static void show_doms(struct seq_file *s, struct rdt_resource_final *f,
        /* Walking r->domains, ensure it can't race with cpuhp */
        lockdep_assert_cpus_held();

- if (print_ctrl)
- seq_printf(s, "%*s%s%s:", max_name_width, f->name,
- resctrl_ctrl_is_default(ctrl) ? "" : "_",
- resctrl_ctrl_is_default(ctrl) ?
- "" : resctrl_ctrl_name_str(ctrl->name));
+ if (print_ctrl) {
+ snprintf(name, sizeof(name), "%s%s%s", f->name,
+ resctrl_ctrl_is_default(ctrl) ? "" : "_",
+ resctrl_ctrl_is_default(ctrl) ?
+ "" : resctrl_ctrl_name_str(ctrl->name));
+ seq_printf(s, "%*s:", max_name_width, name);
+ }

        list_for_each_entry(dom, &ctrl->domains, hdr.list) {
                if (sep)
                        seq_puts(s, ";");
fs/resctrl/rdtgroup.c +6 -5
index c1aac0dcd332..916bf6a8eab4 100644
@@ -1654,6 +1654,7 @@ static int rdtgroup_size_show(struct kernfs_open_file *of,
        struct rdtgroup *rdtgrp;
        struct rdt_resource *r;
        unsigned int size;
+       char name[20];
        u32 ctrl_val;
        int ret = 0;
        u32 closid;
@@ -1693,11 +1694,11 @@ static int rdtgroup_size_show(struct kernfs_open_file *of,
                type = f->conf_type;
                for_each_resource_ctrl(ctrl, r) {
                        sep = false;
-                       seq_printf(s, "%*s", max_name_width, f->name);
-                       if (!resctrl_ctrl_is_default(ctrl))
-                               seq_printf(s, "_%s:", resctrl_ctrl_name_str(ctrl->name));
-                       else
-                               seq_putc(s, ':');
+                       snprintf(name, sizeof(name), "%s%s%s", f->name,
+                                resctrl_ctrl_is_default(ctrl) ? "" : "_",
+                                resctrl_ctrl_is_default(ctrl) ?
+                                 "" : resctrl_ctrl_name_str(ctrl->name));
+                       seq_printf(s, "%*s:", max_name_width, name);
                        list_for_each_entry(d, &ctrl->domains, hdr.list) {
                                if (sep)
                                        seq_putc(s, ';');

And it becomes:
[ resctrl]# cat schemata

            MB:0=100

MB_REGION0_OPT:0=255
MB_REGION0_MIN:0=255
MB_REGION0_MAX:0=255
MB_REGION1_OPT:0=255
MB_REGION1_MIN:0=255
MB_REGION1_MAX:0=255

            L2:0=ffff;2=ffff;4=ffff
            L3:0=3ff

thanks,
Chenyu

author maintainer
#54 lore raw

Hi Chenyu,

Quoted 113 lines · Yu C

Hi Reinette,

Quoted 22 lines · Reinette Chatre

Hi Tony,

Quoted 19 lines · Tony

Reinette,

Tiny bug in "mpam,x86,fs/resctrl: Transition resource control to a list"

Quoted 14 lines

/*
  * Return length needed to display longest control suffix.
  * Add 1 for the "_" character when control name exists. */
size_t resctrl_resource_ctrl_max_len(struct rdt_resource *r) {
    struct resctrl_ctrl *ctrl;
    size_t total = 0;
    size_t len;

    for_each_resource_ctrl(ctrl,r) {
        len = strlen(resctrl_ctrl_name_str(ctrl->name)); if (len)
            total += 1 + len;

Quoted 6 lines
Quoted 3 lines

Should be:

            total = max(total, 1 + len);

Thank you very much. I squashed this locally.

A minor alignment schemata issue is found when testing region-aware RDT:
[resctrl]# cat schemata
            MB:0=100
            MB_REGION0_OPT:0=255
            MB_REGION0_MIN:0=255
            MB_REGION0_MAX:0=255
            MB_REGION1_OPT:0=255
            MB_REGION1_MIN:0=255
            MB_REGION1_MAX:0=255
            L2:0=ffff;2=ffff;4=ffff
            L3:0=3ff

The schemata file right-justified only the resource name in a max_name_width field, then printed the control suffix afterwards without padding. Since max_name_width reserves space for name plus suffix, this left a run of spaces before every line. A small hacky fix:

fs/resctrl/ctrlmondata.c +8 -5
index 9b51464b66b7..7d517f3457bf 100644
@@ -521,6 +521,7 @@ ssize_t rdtgroup_schemata_write(struct kernfs_open_file *of,
  static void show_doms(struct seq_file *s, struct rdt_resource_final *f,
                       bool print_ctrl, int closid, struct resctrl_ctrl *ctrl)
  {
+       char name[20];
         struct rdt_resource *r = f->res;
         struct rdt_ctrl_domain *dom;
         bool sep = false;
@@ -529,11 +530,13 @@ static void show_doms(struct seq_file *s, struct rdt_resource_final *f,
         /* Walking r->domains, ensure it can't race with cpuhp */
         lockdep_assert_cpus_held();
 
-       if (print_ctrl)
-               seq_printf(s, "%*s%s%s:", max_name_width, f->name,
-                               resctrl_ctrl_is_default(ctrl) ? "" : "_",
-                               resctrl_ctrl_is_default(ctrl) ?
-                                "" : resctrl_ctrl_name_str(ctrl->name));
+       if (print_ctrl) {
+               snprintf(name, sizeof(name), "%s%s%s", f->name,
+                        resctrl_ctrl_is_default(ctrl) ? "" : "_",
+                        resctrl_ctrl_is_default(ctrl) ?
+                         "" : resctrl_ctrl_name_str(ctrl->name));
+               seq_printf(s, "%*s:", max_name_width, name);
+       }
         list_for_each_entry(dom, &ctrl->domains, hdr.list) {
                 if (sep)
                         seq_puts(s, ";");
fs/resctrl/rdtgroup.c +6 -5
index c1aac0dcd332..916bf6a8eab4 100644
@@ -1654,6 +1654,7 @@ static int rdtgroup_size_show(struct kernfs_open_file *of,
         struct rdtgroup *rdtgrp;
         struct rdt_resource *r;
         unsigned int size;
+       char name[20];
         u32 ctrl_val;
         int ret = 0;
         u32 closid;
@@ -1693,11 +1694,11 @@ static int rdtgroup_size_show(struct kernfs_open_file *of,
                 type = f->conf_type;
                 for_each_resource_ctrl(ctrl, r) {
                         sep = false;
-                       seq_printf(s, "%*s", max_name_width, f->name);
-                       if (!resctrl_ctrl_is_default(ctrl))
-                               seq_printf(s, "_%s:", resctrl_ctrl_name_str(ctrl->name));
-                       else
-                               seq_putc(s, ':');
+                       snprintf(name, sizeof(name), "%s%s%s", f->name,
+                                resctrl_ctrl_is_default(ctrl) ? "" : "_",
+                                resctrl_ctrl_is_default(ctrl) ?
+                                 "" : resctrl_ctrl_name_str(ctrl->name));
+                       seq_printf(s, "%*s:", max_name_width, name);
                         list_for_each_entry(d, &ctrl->domains, hdr.list) {
                                 if (sep)
                                         seq_putc(s, ';');
Quoted 12 lines

And it becomes:
[ resctrl]# cat schemata
            MB:0=100
MB_REGION0_OPT:0=255
MB_REGION0_MIN:0=255
MB_REGION0_MAX:0=255
MB_REGION1_OPT:0=255
MB_REGION1_MIN:0=255
MB_REGION1_MAX:0=255
            L2:0=ffff;2=ffff;4=ffff
            L3:0=3ff

Thank you for the report and the proposed fix. I'll look into making the fix generic.

Reinette

#55 lore raw
In reply to Reinette Chatre
Quoted 28 lines · Reinette Chatre

/**
* struct resctrl_ctrl - A resource control
* @entry: List entry of rdt_resource::controls
* @scope: Scope of the resource that this control allocates
* @domains: RCU list of all control domains
* @type: The control type that determines the properties of the control,
* format string for displaying control values to user space, and
* parser of control values provided by user space.
* @name: Name of the control. Appended to final resource name
* (rdt_resource_final::name) to create final schema entry.
* Specifically, "rdt_resource_final::name"_"resctrl_ctrl::name".
* For example, with resource name "MB" and control name "MAX" the
* schema entry will be "MB_MAX".
* @cache: Cache allocation control properties.
* @membw: Bandwidth control properties.
*/

struct resctrl_ctrl {
	struct list_head	entry;
	enum resctrl_scope	scope;
	struct list_head	domains;
	enum resctrl_ctrl_type	type;
	enum resctrl_ctrl_name	name;
	union {
		struct resctrl_cache	cache;
		struct resctrl_membw	membw;
	};
};
Quoted 10 lines

Two members summarize how this new structure fits into the rest of resctrl:
a) resctrl_ctrl::entry
Since a resource can support multiple controls there is a new list in struct rdt_resource named "controls" that contains the list of all controls supported by the resource.
b) resctrl_ctrl::domains
Instead of the list of control domains belonging to a resource they now belong to the control self. By doing so resctrl can support resource controls at different scope for the same resource. This is intended to support some upcoming MPAM and RISC-V usages.

The ability to change scope is much needed for RISC-V. There are compromises in my RFC [1] as a result of trying to map everything to either L2 or L3 scope.

I would also like to see a non-cpu cache scope for monitoring too, but would that be better discussed outside the context of this proof of concept?

Thanks,
Drew

[1] https://lore.kernel.org/all/[email protected]/

author maintainer
#56 lore raw

Hi Drew,

Quoted 40 lines · Drew Fustini
Quoted 28 lines · Reinette Chatre

/**
* struct resctrl_ctrl - A resource control
* @entry: List entry of rdt_resource::controls
* @scope: Scope of the resource that this control allocates
* @domains: RCU list of all control domains
* @type: The control type that determines the properties of the control,
* format string for displaying control values to user space, and
* parser of control values provided by user space.
* @name: Name of the control. Appended to final resource name
* (rdt_resource_final::name) to create final schema entry.
* Specifically, "rdt_resource_final::name"_"resctrl_ctrl::name".
* For example, with resource name "MB" and control name "MAX" the
* schema entry will be "MB_MAX".
* @cache: Cache allocation control properties.
* @membw: Bandwidth control properties.
*/

struct resctrl_ctrl {
	struct list_head	entry;
	enum resctrl_scope	scope;
	struct list_head	domains;
	enum resctrl_ctrl_type	type;
	enum resctrl_ctrl_name	name;
	union {
		struct resctrl_cache	cache;
		struct resctrl_membw	membw;
	};
};
Quoted 10 lines

Two members summarize how this new structure fits into the rest of resctrl:
a) resctrl_ctrl::entry
Since a resource can support multiple controls there is a new list in struct rdt_resource named "controls" that contains the list of all controls supported by the resource.
b) resctrl_ctrl::domains
Instead of the list of control domains belonging to a resource they now belong to the control self. By doing so resctrl can support resource controls at different scope for the same resource. This is intended to support some upcoming MPAM and RISC-V usages.

Quoted 7 lines

The ability to change scope is much needed for RISC-V. There are compromises in my RFC [1] as a result of trying to map everything to either L2 or L3 scope.

I would also like to see a non-cpu cache scope for monitoring too, but would that be better discussed outside the context of this proof of concept?

I also think it would be good for it to be clear that monitoring is based on
scope, not a resource. With the MB controls supporting different scope I do think that this would be a good next step. A previous musing from me on this topic can
be found (at the end of ) https://lore.kernel.org/lkml/[email protected]/

I have not yet considered how this can be built on top of this PoC though.

Reinette

#57 lore raw
Quoted 8 lines · Reinette Chatre
Quoted 7 lines

The ability to change scope is much needed for RISC-V. There are compromises in my RFC [1] as a result of trying to map everything to either L2 or L3 scope.

I would also like to see a non-cpu cache scope for monitoring too, but would that be better discussed outside the context of this proof of concept?

Quoted 6 lines

I also think it would be good for it to be clear that monitoring is based on
scope, not a resource. With the MB controls supporting different scope I do think that this would be a good next step. A previous musing from me on this topic can
be found (at the end of ) https://lore.kernel.org/lkml/[email protected]/

I have not yet considered how this can be built on top of this PoC though.

Thanks for explaining. I like how how you show an example of mon_data/mon_NODE_00/mbm_total_bytes in that thread. I believe that sort of scheme would work well for RISc-V as a bandwidth controller implementing the CBQRI spec can be located anywhere within the system.

Drew

#58 lore raw
In reply to Reinette Chatre

Hi Reinette,

Quoted 245 lines · Reinette Chatre

Hi Everybody,

It has been a while since we discussed the resctrl changes required to support hardware that has controls with fine granularity or hardware that has multiple controls per resource. For reference, the most recent email discussion can be found at [1] with a summary of discussions in last year's plumbers slides [2].

I created a PoC that I believe supports what folks have agreed to so far. I hope this can help us to restart the discussion with the goal that resctrl gains support for upcoming hardware that require these features.

Request regarding this PoC
==========================

Please consider this PoC as a "direction check" on the schema description and multiple control discussions held thus far.

Could folks working on enabling new hardware requiring this capability please consider if this is something you can build on and how it should be improved to support these upcoming capabilities?

Opens
=====

While the PoC aims to support what folks agreed on some opens remain:
- I attempted to make some MPAM supporting changes but these are all just compile tested. While MPAM should benefit from the new control properties I did not initialize them on MPAM and did not attempt refactor to separate out
the architecture specific control properties (more on what this means later). I did attempt some MPAM refactoring that duplicates the MPAM domain to the control domain and monitoring domain lists in support of there being multiple controls each with its own list of control domains but it is definitely not good design.
- No support for emulated controls (yet). The PoC is quite large already
but I think it can be used as a base for emulated controls for which the software controller could be a potential first customer. In this PoC mounting with software controller will still display the original controller's properties.
- One open that needs to be addressed as part of support for emulated controls is how best to display emulation relationship via resctrl hierarchy.
- No support for "read-modify-write" usage of schemata file. This is where we discussed (without agreement) on possibly introducing the "#" prefix to schemata file entries. This PoC does not support this prefix and the current assumption/expectation is that when user space changes a configuration only the new control values are written to schemata file. I thus do not have a plan to support this so please share opinions in this regard if you have some.
- Controls are independent for now. This means that, for example, if a resource supports a "MIN" and "MAX" control then this implementation would allow user to set the "maximum" control values to be less than the "minimum" control values.
- PoC supports the "bitmap" control but does not (yet) expose properties of a bitmap control to the new info/<resource>/resource_schemata directory.

Accessing PoC
=============

Please consider the PoC as a rough draft. It has only been compile tested for Arm and known to be incomplete in Arm support. To help with experimenting I only fully adapted the Intel MBA resource to demo two dummy additional MBA controls. All architectures should immediately benefit from the new schema descriptions and new info/MB/resource_schemata hierarchy.

I considered the patches self too many for email. Instead, the PoC can be found at:

git://git.kernel.org/pub/scm/linux/kernel/git/reinette/linux.git branch resctrl/controls_rfc_v1

The work is based on v7.1-rc2 that also includes the following series (two of which has since been queued) included:

"selftests/resctrl: Fixes and improvements focused on Intel platforms"
https://lore.kernel.org/lkml/[email protected]/

"x86,fs/resctrl: Improve resctrl quality and consistency"
https://lore.kernel.org/lkml/[email protected]/

"x86,fs/resctrl: Pave the way for MPAM counter assignment"
https://lore.kernel.org/lkml/[email protected]/

Primary resctrl fs data structure changes
=========================================

Introduces a control represented by struct resctrl_ctrl that looks as below. To make the changes easier to follow I kept some of the original names to help communicate where familiar data structures land.

What to notice about a control is that it has some common properties required from all controls (scope, type, etc.) and then depending on the type of control (RESCTRL_CTRL_BITMAP or RESCTRL_CTRL_SCALAR) there are type specific properties.

/**
* struct resctrl_ctrl - A resource control
* @entry: List entry of rdt_resource::controls
* @scope: Scope of the resource that this control allocates
* @domains: RCU list of all control domains
* @type: The control type that determines the properties of the control,
* format string for displaying control values to user space, and
* parser of control values provided by user space.
* @name: Name of the control. Appended to final resource name
* (rdt_resource_final::name) to create final schema entry.
* Specifically, "rdt_resource_final::name"_"resctrl_ctrl::name".
* For example, with resource name "MB" and control name "MAX" the
* schema entry will be "MB_MAX".
* @cache: Cache allocation control properties.
* @membw: Bandwidth control properties.
*/

struct resctrl_ctrl {
	struct list_head	entry;
	enum resctrl_scope	scope;
	struct list_head	domains;
	enum resctrl_ctrl_type	type;
	enum resctrl_ctrl_name	name;
	union {
		struct resctrl_cache	cache;
		struct resctrl_membw	membw;
	};
};

Two members summarize how this new structure fits into the rest of resctrl:
a) resctrl_ctrl::entry
Since a resource can support multiple controls there is a new list in struct rdt_resource named "controls" that contains the list of all controls supported by the resource.
b) resctrl_ctrl::domains
Instead of the list of control domains belonging to a resource they now belong to the control self. By doing so resctrl can support resource controls at different scope for the same resource. This is intended to support some upcoming MPAM and RISC-V usages.

Example architectural data structure changes
============================================

An architecture can use the new control by following a similar pattern to resource and domain use by architectures. Consider the following for x86 where a new architecture specific struct resctrl_hw_ctrl includes struct resctrl_ctrl and any architecture private data needed to support the control:

/*
* struct resctrl_hw_ctrl - Arch private properties of a resource control
* @r_ctrl: Control properties exposed to resctrl file system
* @msr_base: Base MSR address where control values should be programmed
* @msr_update: Function pointer to update control values
*/

struct resctrl_hw_ctrl {
	struct resctrl_ctrl	r_ctrl;
	unsigned int		msr_base;
	void			(*msr_update)(struct msr_param *m);
};

Structure of patch series
=========================

As a PoC the series is not perfectly structured but to help navigate this work on a high level the changes can be categorized as follows:

Patch 1 to 11:
With a vision of what a "control" is, remove unused/unnecessary members, make clear what is a *resource* property vs a *control* property, do some renaming to help with the PoC.

Patch 12:
Introduce struct resctrl_ctrl and re-arrange existing struct rdt_resource members to form part of new rdt_resource::ctrl

Patch 13 to 44:
A lot of wrangling to introduce struct resctrl_ctrl to all code that needs to work with a control and/or domain without assuming that the control is the one and only control embedded in the resource it belongs to. Essentially, a lot of changes passing the control around in addition to the resource/domain.

Patch 45:
Switch the single struct resctrl_ctrl member of struct rdt_resource to be a list of struct resctrl_ctrl.

Patch 47 to 49:
Introduce new info/<resource>/resource_schemata hierarchy to first only consist of properties already known to resctrl fs.

Patch 50 to 52:
Introduce the new control properties per [1], initialize them for x86, and expose them via info/<resource>/resource_schemata

Patch 53:
Let the new struct resctrl_hw_ctrl contain architecture's control properties.

Patch 54:
Teach resctrl fs about "MIN" and "MAX" controls.

Patch 55:
Sample of "MIN" and "MAX" memory bandwidth controls for x86.

Example interactions
====================

This series can be used on an x86 system where it will show two new dummy controls where it is possible to interact with the new controls.
For example:

# cat schemata
         MB_MAX:0=100;1=100
         MB_MIN:0=100;1=100
         MB:0=100;1=100
         L3:0=fff;1=fff
# echo 'MB_MIN:0=50' > schemata
# cat schemata
         MB_MAX:0=100;1=100
         MB_MIN:0=50;1=100
         MB:0=100;1=100
         L3:0=fff;1=fff

Writing to the dummy control will call a dummy callback that just prints to the kernel log:
"resctrl: Updata temporary MIN control on domain 0 with user value 50"

Example output of info/MB/:
/sys/fs/resctrl/info/MB/thread_throttle_mode:max /sys/fs/resctrl/info/MB/num_closids:15 /sys/fs/resctrl/info/MB/delay_linear:1 /sys/fs/resctrl/info/MB/min_bandwidth:10 /sys/fs/resctrl/info/MB/resource_schemata/MB/resolution:100 /sys/fs/resctrl/info/MB/resource_schemata/MB/tolerance:5 /sys/fs/resctrl/info/MB/resource_schemata/MB/type:scalar /sys/fs/resctrl/info/MB/resource_schemata/MB/min:10 /sys/fs/resctrl/info/MB/resource_schemata/MB/scale:1 /sys/fs/resctrl/info/MB/resource_schemata/MB/scope:L3 /sys/fs/resctrl/info/MB/resource_schemata/MB/unit:all /sys/fs/resctrl/info/MB/resource_schemata/MB/max:100 /sys/fs/resctrl/info/MB/resource_schemata/MB_MIN/resolution:100 /sys/fs/resctrl/info/MB/resource_schemata/MB_MIN/tolerance:5 /sys/fs/resctrl/info/MB/resource_schemata/MB_MIN/type:scalar /sys/fs/resctrl/info/MB/resource_schemata/MB_MIN/min:10 /sys/fs/resctrl/info/MB/resource_schemata/MB_MIN/scale:1 /sys/fs/resctrl/info/MB/resource_schemata/MB_MIN/scope:L3 /sys/fs/resctrl/info/MB/resource_schemata/MB_MIN/unit:all /sys/fs/resctrl/info/MB/resource_schemata/MB_MIN/max:100 /sys/fs/resctrl/info/MB/resource_schemata/MB_MAX/resolution:100 /sys/fs/resctrl/info/MB/resource_schemata/MB_MAX/tolerance:5 /sys/fs/resctrl/info/MB/resource_schemata/MB_MAX/type:scalar /sys/fs/resctrl/info/MB/resource_schemata/MB_MAX/min:10 /sys/fs/resctrl/info/MB/resource_schemata/MB_MAX/scale:1 /sys/fs/resctrl/info/MB/resource_schemata/MB_MAX/scope:L3 /sys/fs/resctrl/info/MB/resource_schemata/MB_MAX/unit:all /sys/fs/resctrl/info/MB/resource_schemata/MB_MAX/max:100 /sys/fs/resctrl/info/MB/bandwidth_gran:10

Quoted 7 lines

Any feedback is appreciated.

Reinette

[1] https://lore.kernel.org/lkml/aPtfMFfLV1l%[email protected]/
[2] https://lpc.events/event/19/contributions/2093/attachments/1958/4172/resctrl%20Microconference%20LPC%202025%20Tokyo.pdf

Re-based GMBA patches[3] on top of this series.

[3] https://lore.kernel.org/lkml/[email protected]/

#sudo mount -t resctrl resctrl /sys/fs/resctrl

#cat /sys/fs/resctrl/schemata
SMBA_GBL:0=4096;1=4096

     SMBA:0=8192;1=8192;2=8192;3=8192
   MB_GBL:0=4096;1=4096
       MB:0=8192;1=8192;2=8192;3=8192
       L3:0=ffff;1=ffff;2=ffff;3=ffff

Needed this change to get the schemata alignment correctly.

Note: Renamed GMB to MB_GBL and GSMBA to SMBA_GBL.

===========================================================

fs/resctrl/ctrlmondata.c +10 -5
index 979176b054e8..6123689e1250 100644
@@ -512,17 +512,22 @@ static void show_doms(struct seq_file *s, struct rdt_resource_final *f,
 {
        struct rdt_resource *r = f->res;
        struct rdt_ctrl_domain *dom;
+       char res_name[16];
        bool sep = false;
        u32 ctrl_val;
 
        /* Walking r->domains, ensure it can't race with cpuhp */
        lockdep_assert_cpus_held();
 
-       if (print_ctrl)
-               seq_printf(s, "%*s%s%s:", max_name_width, f->name,
-                               resctrl_ctrl_is_default(ctrl) ? "" : "_",
-                               resctrl_ctrl_is_default(ctrl) ?
-                                "" : resctrl_ctrl_name_str(ctrl->name));
+       if (print_ctrl) {
+               if (resctrl_ctrl_is_default(ctrl))
+                       snprintf(res_name, sizeof(res_name), "%s", f->name);
+               else
+                       snprintf(res_name, sizeof(res_name), "%s_%s",
+                                       f->name, resctrl_ctrl_name_str(ctrl->name));
+               seq_printf(s, "%*s:", max_name_width, res_name);
+       }
+
        list_for_each_entry(dom, &ctrl->domains, hdr.list) {
                if (sep)
                        seq_puts(s, ";");

====================================================================

$ tree /sys/fs/resctrl/info/MB/
/sys/fs/resctrl/info/MB/
├── bandwidth_gran
├── delay_linear
├── min_bandwidth
├── num_closids
└── resource_schemata
    ├── MB
    │   ├── max
    │   ├── min
    │   ├── resolution
    │   ├── scale
    │   ├── scope
    │   ├── tolerance
    │   ├── type
    │   └── unit
    └── MB_GBL
        ├── max
        ├── min
        ├── resolution
        ├── scale
        ├── scope
        ├── tolerance
        ├── type
        └── unit

4 directories, 20 files

$ tree /sys/fs/resctrl/info/SMBA/
/sys/fs/resctrl/info/SMBA/
├── bandwidth_gran
├── delay_linear
├── min_bandwidth
├── num_closids
└── resource_schemata
    ├── SMBA
    │   ├── max
    │   ├── min
    │   ├── resolution
    │   ├── scale
    │   ├── scope
    │   ├── tolerance
    │   ├── type
    │   └── unit
    └── SMBA_GBL
        ├── max
        ├── min
        ├── resolution
        ├── scale
        ├── scope
        ├── tolerance
        ├── type
        └── unit

4 directories, 20 files

Does the info directory hierarchy look right?

After going through the comments, it seems like it should cover most of the upcoming features. At least the Global MBA/SMBA stuff looks like it fits in well with the RFC.

What is your expectation for the RFC going forward?

Thanks
Babu

author maintainer
#59 lore raw

Hi Babu,

Quoted 2 lines · Babu

Re-based GMBA patches[3] on top of this series.

Thank you very much for trying it out.

Quoted 41 lines

[3] https://lore.kernel.org/lkml/[email protected]/

#sudo mount -t resctrl resctrl /sys/fs/resctrl

#cat /sys/fs/resctrl/schemata
SMBA_GBL:0=4096;1=4096
     SMBA:0=8192;1=8192;2=8192;3=8192
   MB_GBL:0=4096;1=4096
       MB:0=8192;1=8192;2=8192;3=8192
       L3:0=ffff;1=ffff;2=ffff;3=ffff

Needed this change to get the schemata alignment correctly.

Note: Renamed GMB to MB_GBL and GSMBA to SMBA_GBL.

===========================================================

fs/resctrl/ctrlmondata.c +3 -5
index 979176b054e8..6123689e1250 100644
@@ -512,17 +512,22 @@ static void show_doms(struct seq_file *s, struct rdt_resource_final *f,
  {
         struct rdt_resource *r = f->res;
         struct rdt_ctrl_domain *dom;
+       char res_name[16];
         bool sep = false;
         u32 ctrl_val;
 
         /* Walking r->domains, ensure it can't race with cpuhp */
         lockdep_assert_cpus_held();
 
-       if (print_ctrl)
-               seq_printf(s, "%*s%s%s:", max_name_width, f->name,
-                               resctrl_ctrl_is_default(ctrl) ? "" : "_",
-                               resctrl_ctrl_is_default(ctrl) ?
-                                "" : resctrl_ctrl_name_str(ctrl->name));
+       if (print_ctrl) {
+               if (resctrl_ctrl_is_default(ctrl))

fs/resctrl/ctrlmondata.c static void show_doms(struct seq_file *s, struct rdt_resource_final *f, · line 533

Quoted 10 lines
fs/resctrl/ctrlmondata.c +7 -0
+                       snprintf(res_name, sizeof(res_name), "%s", f->name);
+               else
+                       snprintf(res_name, sizeof(res_name), "%s_%s",
+                                       f->name, resctrl_ctrl_name_str(ctrl->name));
+               seq_printf(s, "%*s:", max_name_width, res_name);
+       }
+
         list_for_each_entry(dom, &ctrl->domains, hdr.list) {
                 if (sep)
                         seq_puts(s, ";");

This looks like a workaround for a bug. Are you perhaps running with Tony's fixup included?
https://lore.kernel.org/lkml/aiB2oJFPF-sqUVQx@agluck-desk3/

Quoted 57 lines

====================================================================

$ tree /sys/fs/resctrl/info/MB/
/sys/fs/resctrl/info/MB/
├── bandwidth_gran
├── delay_linear
├── min_bandwidth
├── num_closids
└── resource_schemata
    ├── MB
    │   ├── max
    │   ├── min
    │   ├── resolution
    │   ├── scale
    │   ├── scope
    │   ├── tolerance
    │   ├── type
    │   └── unit
    └── MB_GBL
        ├── max
        ├── min
        ├── resolution
        ├── scale
        ├── scope
        ├── tolerance
        ├── type
        └── unit

4 directories, 20 files

$ tree /sys/fs/resctrl/info/SMBA/
/sys/fs/resctrl/info/SMBA/
├── bandwidth_gran
├── delay_linear
├── min_bandwidth
├── num_closids
└── resource_schemata
    ├── SMBA
    │   ├── max
    │   ├── min
    │   ├── resolution
    │   ├── scale
    │   ├── scope
    │   ├── tolerance
    │   ├── type
    │   └── unit
    └── SMBA_GBL
        ├── max
        ├── min
        ├── resolution
        ├── scale
        ├── scope
        ├── tolerance
        ├── type
        └── unit
Quoted 3 lines

4 directories, 20 files

Does the info directory hierarchy look right?

I am not sure if the "GBL" extension is most informative. In this thread there were some motivations for the control names to include the scope. It could be useful to create some control naming customs to not have the schemata file become chaotic. Apart from that the hierarchy above matches what we have so far for the new schema descriptions.

Could you please also share the contents of the new files resulting from this implementation?

Quoted 3 lines

After going through the comments, it seems like it should cover most of the upcoming features. At least the Global MBA/SMBA stuff looks like it fits in well with the RFC.

That is good news.

Quoted 1 line

What is your expectation for the RFC going forward?

The "opens" of the RFC needs to be finalized as well as the items identified during this discussion. How to support emulated controls looks to be the biggest one. While the initial schema description may not need to support emulated controls, how to support it needs to be clear since the initial schema description needs to be accompanied with enough user documentation to support such future change.

It should be possible to start upstreaming at least the preparatory patches.

Reinette

reviewer
#60 lore raw

Hi Reinette,

Quoted 55 lines · Reinette Chatre

Hi Babu,

Quoted 2 lines · Babu

Re-based GMBA patches[3] on top of this series.

Thank you very much for trying it out.

Quoted 47 lines

[3] https://lore.kernel.org/lkml/[email protected]/

#sudo mount -t resctrl resctrl /sys/fs/resctrl

#cat /sys/fs/resctrl/schemata
SMBA_GBL:0=4096;1=4096
     SMBA:0=8192;1=8192;2=8192;3=8192
   MB_GBL:0=4096;1=4096
       MB:0=8192;1=8192;2=8192;3=8192
       L3:0=ffff;1=ffff;2=ffff;3=ffff

Needed this change to get the schemata alignment correctly.

Note: Renamed GMB to MB_GBL and GSMBA to SMBA_GBL.

===========================================================

fs/resctrl/ctrlmondata.c +10 -5
index 979176b054e8..6123689e1250 100644
@@ -512,17 +512,22 @@ static void show_doms(struct seq_file *s, struct rdt_resource_final *f,
  {
         struct rdt_resource *r = f->res;
         struct rdt_ctrl_domain *dom;
+       char res_name[16];
         bool sep = false;
         u32 ctrl_val;
 
         /* Walking r->domains, ensure it can't race with cpuhp */
         lockdep_assert_cpus_held();
 
-       if (print_ctrl)
-               seq_printf(s, "%*s%s%s:", max_name_width, f->name,
-                               resctrl_ctrl_is_default(ctrl) ? "" : "_",
-                               resctrl_ctrl_is_default(ctrl) ?
-                                "" : resctrl_ctrl_name_str(ctrl->name));
+       if (print_ctrl) {
+               if (resctrl_ctrl_is_default(ctrl))
+                       snprintf(res_name, sizeof(res_name), "%s", f->name);
+               else
+                       snprintf(res_name, sizeof(res_name), "%s_%s",
+                                       f->name, resctrl_ctrl_name_str(ctrl->name));
+               seq_printf(s, "%*s:", max_name_width, res_name);
+       }
+
Quoted 8 lines

fs/resctrl/ctrlmondata.c static void show_doms(struct seq_file *s, struct rdt_resource_final *f, · line 533

Quoted 4 lines
fs/resctrl/ctrlmondata.c +0 -0
         list_for_each_entry(dom, &ctrl->domains, hdr.list) {
                 if (sep)
                         seq_puts(s, ";");

This looks like a workaround for a bug. Are you perhaps running with Tony's fixup included?
https://lore.kernel.org/lkml/aiB2oJFPF-sqUVQx@agluck-desk3/

Ah.. I missed that.

Quoted 58 lines
Quoted 58 lines

====================================================================

$ tree /sys/fs/resctrl/info/MB/
/sys/fs/resctrl/info/MB/
├── bandwidth_gran
├── delay_linear
├── min_bandwidth
├── num_closids
└── resource_schemata
     ├── MB
     │   ├── max
     │   ├── min
     │   ├── resolution
     │   ├── scale
     │   ├── scope
     │   ├── tolerance
     │   ├── type
     │   └── unit
     └── MB_GBL
         ├── max
         ├── min
         ├── resolution
         ├── scale
         ├── scope
         ├── tolerance
         ├── type
         └── unit

4 directories, 20 files

$ tree /sys/fs/resctrl/info/SMBA/
/sys/fs/resctrl/info/SMBA/
├── bandwidth_gran
├── delay_linear
├── min_bandwidth
├── num_closids
└── resource_schemata
     ├── SMBA
     │   ├── max
     │   ├── min
     │   ├── resolution
     │   ├── scale
     │   ├── scope
     │   ├── tolerance
     │   ├── type
     │   └── unit
     └── SMBA_GBL
         ├── max
         ├── min
         ├── resolution
         ├── scale
         ├── scope
         ├── tolerance
         ├── type
         └── unit
Quoted 7 lines
Quoted 3 lines

4 directories, 20 files

Does the info directory hierarchy look right?

I am not sure if the "GBL" extension is most informative. In this thread there were some motivations for the control names to include the scope. It could be useful to create some control naming customs to not have the schemata file become chaotic.

Make sense.

Quoted 5 lines

Apart from that the hierarchy above matches what we have so far for the new schema descriptions.

Could you please also share the contents of the new files resulting from this implementation?

Used this script to print:

$tree -fi info/MB | while read f; do

    if [ -f "$f" ]; then
        printf "%s: %s\n" "$f" "$(cat "$f")"
    else
        echo "$f"
    fi

done

info/MB
info/MB/bandwidth_gran: 1
info/MB/delay_linear: 0
info/MB/min_bandwidth: 0
info/MB/num_closids: 16
info/MB/resource_schemata
info/MB/resource_schemata/MB info/MB/resource_schemata/MB/max: 8192 info/MB/resource_schemata/MB/min: 0 info/MB/resource_schemata/MB/resolution: 8 info/MB/resource_schemata/MB/scale: 1 info/MB/resource_schemata/MB/scope: L3 info/MB/resource_schemata/MB/tolerance: 0 info/MB/resource_schemata/MB/type: scalar info/MB/resource_schemata/MB/unit: GBps info/MB/resource_schemata/MB_GBL info/MB/resource_schemata/MB_GBL/max: 4096 info/MB/resource_schemata/MB_GBL/min: 0 info/MB/resource_schemata/MB_GBL/resolution: 1 info/MB/resource_schemata/MB_GBL/scale: 1 info/MB/resource_schemata/MB_GBL/scope: Unsupported control scope info/MB/resource_schemata/MB_GBL/tolerance: 0 info/MB/resource_schemata/MB_GBL/type: scalar info/MB/resource_schemata/MB_GBL/unit: GBps

4 directories, 20 files

info/SMBA
info/SMBA/bandwidth_gran: 1
info/SMBA/delay_linear: 0
info/SMBA/min_bandwidth: 0
info/SMBA/num_closids: 16
info/SMBA/resource_schemata info/SMBA/resource_schemata/SMBA info/SMBA/resource_schemata/SMBA/max: 8192 info/SMBA/resource_schemata/SMBA/min: 0 info/SMBA/resource_schemata/SMBA/resolution: 8 info/SMBA/resource_schemata/SMBA/scale: 1 info/SMBA/resource_schemata/SMBA/scope: L3 info/SMBA/resource_schemata/SMBA/tolerance: 0 info/SMBA/resource_schemata/SMBA/type: scalar info/SMBA/resource_schemata/SMBA/unit: GBps info/SMBA/resource_schemata/SMBA_GBL info/SMBA/resource_schemata/SMBA_GBL/max: 4096 info/SMBA/resource_schemata/SMBA_GBL/min: 0 info/SMBA/resource_schemata/SMBA_GBL/resolution: 1 info/SMBA/resource_schemata/SMBA_GBL/scale: 1 info/SMBA/resource_schemata/SMBA_GBL/scope: Unsupported control scope info/SMBA/resource_schemata/SMBA_GBL/tolerance: 0 info/SMBA/resource_schemata/SMBA_GBL/type: scalar info/SMBA/resource_schemata/SMBA_GBL/unit: GBps

4 directories, 20 files

Quoted 9 lines
Quoted 4 lines

After going through the comments, it seems like it should cover most of the upcoming features. At least the Global MBA/SMBA stuff looks like it fits in well with the RFC.

That is good news.

Quoted 1 line

What is your expectation for the RFC going forward?

Quoted 8 lines

The "opens" of the RFC needs to be finalized as well as the items identified during this discussion. How to support emulated controls looks to be the biggest one. While the initial schema description may not need to support emulated controls, how to support it needs to be clear since the initial schema description needs to be accompanied with enough user documentation to support such future change.

It should be possible to start upstreaming at least the preparatory patches.

Sounds good. Will keep an eye out for it.
Thanks
Babu

reviewer
#61 lore raw
In reply to Reinette Chatre

Hi, Reinette, Ben, Shaopen, et al,

On 5/29/26 11:06, Reinette Chatre wrote:

As Shaopen and Ben mentioned earlier, we are working on two MPAM features that may need to change schemata interface. The CPU-less feature was discussed on LPC (although the interfaces will be slightly different from the LPC). Hardlimit feature was not discussed yet.

It's good to discuss this further in this RFC thread before the new features RFCs will be sent out.

Overall, the new features can fit into this RFC well.

Quoted 193 lines

Hi Everybody,

It has been a while since we discussed the resctrl changes required to support hardware that has controls with fine granularity or hardware that has multiple controls per resource. For reference, the most recent email discussion can be found at [1] with a summary of discussions in last year's plumbers slides [2].

I created a PoC that I believe supports what folks have agreed to so far. I hope this can help us to restart the discussion with the goal that resctrl gains support for upcoming hardware that require these features.

Request regarding this PoC
==========================

Please consider this PoC as a "direction check" on the schema description and multiple control discussions held thus far.

Could folks working on enabling new hardware requiring this capability please consider if this is something you can build on and how it should be improved to support these upcoming capabilities?

Opens
=====

While the PoC aims to support what folks agreed on some opens remain:
- I attempted to make some MPAM supporting changes but these are all just compile tested. While MPAM should benefit from the new control properties I did not initialize them on MPAM and did not attempt refactor to separate out
the architecture specific control properties (more on what this means later). I did attempt some MPAM refactoring that duplicates the MPAM domain to the control domain and monitoring domain lists in support of there being multiple controls each with its own list of control domains but it is definitely not good design.
- No support for emulated controls (yet). The PoC is quite large already
but I think it can be used as a base for emulated controls for which the software controller could be a potential first customer. In this PoC mounting with software controller will still display the original controller's properties.
- One open that needs to be addressed as part of support for emulated controls is how best to display emulation relationship via resctrl hierarchy.
- No support for "read-modify-write" usage of schemata file. This is where we discussed (without agreement) on possibly introducing the "#" prefix to schemata file entries. This PoC does not support this prefix and the current assumption/expectation is that when user space changes a configuration only the new control values are written to schemata file. I thus do not have a plan to support this so please share opinions in this regard if you have some.
- Controls are independent for now. This means that, for example, if a resource supports a "MIN" and "MAX" control then this implementation would allow user to set the "maximum" control values to be less than the "minimum" control values.
- PoC supports the "bitmap" control but does not (yet) expose properties of a bitmap control to the new info/<resource>/resource_schemata directory.

Accessing PoC
=============

Please consider the PoC as a rough draft. It has only been compile tested for Arm and known to be incomplete in Arm support. To help with experimenting I only fully adapted the Intel MBA resource to demo two dummy additional MBA controls. All architectures should immediately benefit from the new schema descriptions and new info/MB/resource_schemata hierarchy.

I considered the patches self too many for email. Instead, the PoC can be found at:

git://git.kernel.org/pub/scm/linux/kernel/git/reinette/linux.git branch resctrl/controls_rfc_v1

The work is based on v7.1-rc2 that also includes the following series (two of which has since been queued) included:

"selftests/resctrl: Fixes and improvements focused on Intel platforms"
https://lore.kernel.org/lkml/[email protected]/

"x86,fs/resctrl: Improve resctrl quality and consistency"
https://lore.kernel.org/lkml/[email protected]/

"x86,fs/resctrl: Pave the way for MPAM counter assignment"
https://lore.kernel.org/lkml/[email protected]/

Primary resctrl fs data structure changes
=========================================

Introduces a control represented by struct resctrl_ctrl that looks as below. To make the changes easier to follow I kept some of the original names to help communicate where familiar data structures land.

What to notice about a control is that it has some common properties required from all controls (scope, type, etc.) and then depending on the type of control (RESCTRL_CTRL_BITMAP or RESCTRL_CTRL_SCALAR) there are type specific properties.

/**
* struct resctrl_ctrl - A resource control
* @entry: List entry of rdt_resource::controls
* @scope: Scope of the resource that this control allocates
* @domains: RCU list of all control domains
* @type: The control type that determines the properties of the control,
* format string for displaying control values to user space, and
* parser of control values provided by user space.
* @name: Name of the control. Appended to final resource name
* (rdt_resource_final::name) to create final schema entry.
* Specifically, "rdt_resource_final::name"_"resctrl_ctrl::name".
* For example, with resource name "MB" and control name "MAX" the
* schema entry will be "MB_MAX".
* @cache: Cache allocation control properties.
* @membw: Bandwidth control properties.
*/

struct resctrl_ctrl {
	struct list_head	entry;
	enum resctrl_scope	scope;
	struct list_head	domains;
	enum resctrl_ctrl_type	type;
	enum resctrl_ctrl_name	name;
	union {
		struct resctrl_cache	cache;
		struct resctrl_membw	membw;
	};
};

Two members summarize how this new structure fits into the rest of resctrl:
a) resctrl_ctrl::entry
Since a resource can support multiple controls there is a new list in struct rdt_resource named "controls" that contains the list of all controls supported by the resource.
b) resctrl_ctrl::domains
Instead of the list of control domains belonging to a resource they now belong to the control self. By doing so resctrl can support resource controls at different scope for the same resource. This is intended to support some upcoming MPAM and RISC-V usages.

Example architectural data structure changes
============================================

An architecture can use the new control by following a similar pattern to resource and domain use by architectures. Consider the following for x86 where a new architecture specific struct resctrl_hw_ctrl includes struct resctrl_ctrl and any architecture private data needed to support the control:

/*
* struct resctrl_hw_ctrl - Arch private properties of a resource control
* @r_ctrl: Control properties exposed to resctrl file system
* @msr_base: Base MSR address where control values should be programmed
* @msr_update: Function pointer to update control values
*/

struct resctrl_hw_ctrl {
	struct resctrl_ctrl	r_ctrl;
	unsigned int		msr_base;
	void			(*msr_update)(struct msr_param *m);
};

Structure of patch series
=========================

As a PoC the series is not perfectly structured but to help navigate this work on a high level the changes can be categorized as follows:

Patch 1 to 11:
With a vision of what a "control" is, remove unused/unnecessary members, make clear what is a *resource* property vs a *control* property, do some renaming to help with the PoC.

Patch 12:
Introduce struct resctrl_ctrl and re-arrange existing struct rdt_resource members to form part of new rdt_resource::ctrl

Patch 13 to 44:
A lot of wrangling to introduce struct resctrl_ctrl to all code that needs to work with a control and/or domain without assuming that the control is the one and only control embedded in the resource it belongs to. Essentially, a lot of changes passing the control around in addition to the resource/domain.

Patch 45:
Switch the single struct resctrl_ctrl member of struct rdt_resource to be a list of struct resctrl_ctrl.

Patch 47 to 49:
Introduce new info/<resource>/resource_schemata hierarchy to first only consist of properties already known to resctrl fs.

Patch 50 to 52:
Introduce the new control properties per [1], initialize them for x86, and expose them via info/<resource>/resource_schemata

Patch 53:
Let the new struct resctrl_hw_ctrl contain architecture's control properties.

Patch 54:
Teach resctrl fs about "MIN" and "MAX" controls.

Patch 55:
Sample of "MIN" and "MAX" memory bandwidth controls for x86.

Example interactions
====================

Quoted 8 lines

This series can be used on an x86 system where it will show two new dummy controls where it is possible to interact with the new controls.
For example:

# cat schemata
         MB_MAX:0=100;1=100
         MB_MIN:0=100;1=100
         MB:0=100;1=100

Some platforms may support CPU-less node which is represented by numa node id, examples:
1. CXL type 2 memory node which provides CXL memory without CPU and L3 on the node
2. GPU memory node that can be accessed by all CPUs but doesn't have a local CPU and L3 bound to. etc.

MPAM can allocate and monitor mem bandwidth on these memory node.
Since no CPU and L3 on the node, cache id cannot be used in "MB:" line. Instead, numa ids are used to identify MB allocation and monitoring.

For example, the MB allocation on CPU-less platforms could be:
MB:0=100;1=100;2=100;10=100;18=100;26=100

Where: domain id 0, 1, 2, etc are numa node id shown in /sys/devices/system/node directory or by numctl. 0: socket 0, node 0, CPUs, memory
1: socket 1, node 1, CPUs, memory
2: GPU 0, node 2, no CPU, memory only
10: GPU 1, node 10, no CPU, memory only
18: GPU 3, node 18, no CPU, memory only
26: GPU 4, node 26, no CPU, memory only

Arch specific driver (e.g. MPAM) detects CPU-less node. If there is any CPU-less node, use numa id in "MB:". Otherwise, fallback to legacy cache id.

There is another MPAM feature called MBW Max hardlimit which sets "MB:" allocation as hardlimit (i.e. MBW throttling percentage must be satisfied) per domain. Adding a new "MB_HLIM:" line in schemata. It's 1:1 mapped to "MB:" to control hardlimit of MB throttling percentage on each domain. By default hardlimit is off (0) and can be turned on to set MBW Max hardlimit on a domain.

For exmple:
MB_HLIM: 0=0;1=0;2=1;10=0;18=0;26=0
MB:0=100;1=100;2=80;10=100;18=100;26=100

On GPU memory numa node 2: cannot use more than 80% of total max mbw even if there is still idle mem bandwidth on this node).

MBW allocations on all other domains are soft limited, meaning MBW can be used more than specified if mem is idle.

Quoted 13 lines
         L3:0=fff;1=fff
# echo 'MB_MIN:0=50' > schemata
# cat schemata
         MB_MAX:0=100;1=100
         MB_MIN:0=50;1=100
         MB:0=100;1=100
         L3:0=fff;1=fff

Writing to the dummy control will call a dummy callback that just prints to the kernel log:
"resctrl: Updata temporary MIN control on domain 0 with user value 50"

Quoted 5 lines

Example output of info/MB/:
/sys/fs/resctrl/info/MB/thread_throttle_mode:max
/sys/fs/resctrl/info/MB/num_closids:15
/sys/fs/resctrl/info/MB/delay_linear:1
/sys/fs/resctrl/info/MB/min_bandwidth:10

Add two new MB info RO files:
1. /sys/fs/resctrl/info/MB/domain_id
It shows "numa" for using numa id in "MB:" or "cache" for using legacy cache id.

2. /sys/fs/resctrl/info/MB/max_lim
It shows number 0-3 for MPAM MBW max limit behaviors: 0 for supporting both softlimit and hardlimit, etc.

Quoted 1 line

/sys/fs/resctrl/info/MB/resource_schemata/MB/resolution:100

Is it more concise to s/resource_schemata/schemata/? "resource_" seems redundant in the context "info/MB".

Quoted 16 lines

/sys/fs/resctrl/info/MB/resource_schemata/MB/tolerance:5 /sys/fs/resctrl/info/MB/resource_schemata/MB/type:scalar /sys/fs/resctrl/info/MB/resource_schemata/MB/min:10 /sys/fs/resctrl/info/MB/resource_schemata/MB/scale:1 /sys/fs/resctrl/info/MB/resource_schemata/MB/scope:L3 /sys/fs/resctrl/info/MB/resource_schemata/MB/unit:all /sys/fs/resctrl/info/MB/resource_schemata/MB/max:100 /sys/fs/resctrl/info/MB/resource_schemata/MB_MIN/resolution:100 /sys/fs/resctrl/info/MB/resource_schemata/MB_MIN/tolerance:5 /sys/fs/resctrl/info/MB/resource_schemata/MB_MIN/type:scalar /sys/fs/resctrl/info/MB/resource_schemata/MB_MIN/min:10 /sys/fs/resctrl/info/MB/resource_schemata/MB_MIN/scale:1 /sys/fs/resctrl/info/MB/resource_schemata/MB_MIN/scope:L3 /sys/fs/resctrl/info/MB/resource_schemata/MB_MIN/unit:all /sys/fs/resctrl/info/MB/resource_schemata/MB_MIN/max:100 /sys/fs/resctrl/info/MB/resource_schemata/MB_MAX/resolution:100

Quoted 8 lines

/sys/fs/resctrl/info/MB/resource_schemata/MB_MAX/tolerance:5 /sys/fs/resctrl/info/MB/resource_schemata/MB_MAX/type:scalar /sys/fs/resctrl/info/MB/resource_schemata/MB_MAX/min:10 /sys/fs/resctrl/info/MB/resource_schemata/MB_MAX/scale:1 /sys/fs/resctrl/info/MB/resource_schemata/MB_MAX/scope:L3 /sys/fs/resctrl/info/MB/resource_schemata/MB_MAX/unit:all /sys/fs/resctrl/info/MB/resource_schemata/MB_MAX/max:100 /sys/fs/resctrl/info/MB/bandwidth_gran:10

For MBW monitoring, extend mon_data/ directory to monitor CPU-less memory node. For example,

On legacy platforms (i.e. L3 and memory are described in same MPAM ACPI MSC wich doesn't support CPU-less nodes):
mon_data/mbm_L3_01/llc_occupancy
mon_data/mbm_L3_01/mbm_total_bytes
mon_data/mbm_L3_02/llc_occupancy
mon_data/mbm_L3_02/mbm_total_bytes

On platforms with L3 and memory in separate MPAM ACPI MSCs but there is no CPU-less node:

mon_data/mbm_L3_01/llc_occupancy    <- cache id 1
mon_data/mbm_L3_02/llc_occupancy    <- cache id 2
mon_data/mbm_MB_00/mbm_total_bytes  <- numa node 0 (socket 0)
mon_data/mbm_MB_01/mbm_total_bytes  <- numa node 1 (socket 1)

On platforms with L3 and memory in separate MPAM ACPI MSCs and there are CPU-less nodes:

mon_data/mbm_L3_01/llc_occupancy    <- cache id 1
mon_data/mbm_L3_02/llc_occupancy    <- cache id 2
mon_data/mbm_MB_00/mbm_total_bytes  <- numa node 0 (socket 0)
mon_data/mbm_MB_01/mbm_total_bytes  <- numa node 1 (socket 1)
mon_data/mbm_MB_02/mbm_total_bytes  <- numa node 2 (GPU 0 mem)
mon_data/mbm_MB_10/mbm_total_bytes  <- numa node 10 (GPU 1 mem)
mon_data/mbm_MB_18/mbm_total_bytes  <- numa node 18 (GPU 2 mem)
mon_data/mbm_MB_26/mbm_total_bytes  <- numa node 26 (GPU 3 mem)

Thanks.

-Fenghua

author maintainer
#62 lore raw

Hi Fenghua,

Quoted 8 lines · Fenghua Yu

Hi, Reinette, Ben, Shaopen, et al,

On 5/29/26 11:06, Reinette Chatre wrote:

As Shaopen and Ben mentioned earlier, we are working on two MPAM features that may need to change schemata interface. The CPU-less feature was discussed on LPC (although the interfaces will be slightly different from the LPC).

I know. Here is where I tried to engage with you on needed interfaces after LPC:
https://lore.kernel.org/lkml/[email protected]/

Quoted 1 line

Hardlimit feature was not discussed yet.

It was considered. See discussion starting at
https://lore.kernel.org/lkml/[email protected]/

Quoted 1 line

It's good to discuss this further in this RFC thread before the new features RFCs will be sent out.

ack.

Quoted 224 lines

Overall, the new features can fit into this RFC well.

Quoted 193 lines

Hi Everybody,

It has been a while since we discussed the resctrl changes required to support hardware that has controls with fine granularity or hardware that has multiple controls per resource. For reference, the most recent email discussion can be found at [1] with a summary of discussions in last year's plumbers slides [2].

I created a PoC that I believe supports what folks have agreed to so far. I hope this can help us to restart the discussion with the goal that resctrl gains support for upcoming hardware that require these features.

Request regarding this PoC
==========================

Please consider this PoC as a "direction check" on the schema description and multiple control discussions held thus far.

Could folks working on enabling new hardware requiring this capability please consider if this is something you can build on and how it should be improved to support these upcoming capabilities?

Opens
=====

While the PoC aims to support what folks agreed on some opens remain:
- I attempted to make some MPAM supporting changes but these are all just compile tested. While MPAM should benefit from the new control properties I did not initialize them on MPAM and did not attempt refactor to separate out
   the architecture specific control properties (more on what this means later). I did attempt some MPAM refactoring that duplicates the MPAM domain to the control domain and monitoring domain lists in support of there being multiple controls each with its own list of control domains but it is definitely not good design.
- No support for emulated controls (yet). The PoC is quite large already
   but I think it can be used as a base for emulated controls for which the software controller could be a potential first customer. In this PoC mounting with software controller will still display the original controller's properties.
- One open that needs to be addressed as part of support for emulated controls is how best to display emulation relationship via resctrl hierarchy.
- No support for "read-modify-write" usage of schemata file. This is where we discussed (without agreement) on possibly introducing the "#" prefix to schemata file entries. This PoC does not support this prefix and the current assumption/expectation is that when user space changes a configuration only the new control values are written to schemata file. I thus do not have a plan to support this so please share opinions in this regard if you have some.
- Controls are independent for now. This means that, for example, if a resource supports a "MIN" and "MAX" control then this implementation would allow user to set the "maximum" control values to be less than the "minimum" control values.
- PoC supports the "bitmap" control but does not (yet) expose properties of a bitmap control to the new info/<resource>/resource_schemata directory.

Accessing PoC
=============

Please consider the PoC as a rough draft. It has only been compile tested for Arm and known to be incomplete in Arm support. To help with experimenting I only fully adapted the Intel MBA resource to demo two dummy additional MBA controls. All architectures should immediately benefit from the new schema descriptions and new info/MB/resource_schemata hierarchy.

I considered the patches self too many for email. Instead, the PoC can be found at:

git://git.kernel.org/pub/scm/linux/kernel/git/reinette/linux.git branch resctrl/controls_rfc_v1

The work is based on v7.1-rc2 that also includes the following series (two of which has since been queued) included:

"selftests/resctrl: Fixes and improvements focused on Intel platforms"
https://lore.kernel.org/lkml/[email protected]/

"x86,fs/resctrl: Improve resctrl quality and consistency"
https://lore.kernel.org/lkml/[email protected]/

"x86,fs/resctrl: Pave the way for MPAM counter assignment"
https://lore.kernel.org/lkml/[email protected]/

Primary resctrl fs data structure changes
=========================================

Introduces a control represented by struct resctrl_ctrl that looks as below. To make the changes easier to follow I kept some of the original names to help communicate where familiar data structures land.

What to notice about a control is that it has some common properties required from all controls (scope, type, etc.) and then depending on the type of control (RESCTRL_CTRL_BITMAP or RESCTRL_CTRL_SCALAR) there are type specific properties.

/**
  * struct resctrl_ctrl - A resource control
  * @entry:    List entry of rdt_resource::controls
  * @scope:    Scope of the resource that this control allocates
  * @domains:    RCU list of all control domains
  * @type:    The control type that determines the properties of the control, *        format string for displaying control values to user space, and *        parser of control values provided by user space.
  * @name:    Name of the control. Appended to final resource name *        (rdt_resource_final::name) to create final schema entry. *        Specifically, "rdt_resource_final::name"_"resctrl_ctrl::name". *        For example, with resource name "MB" and control name "MAX" the *        schema entry will be "MB_MAX".
  * @cache:    Cache allocation control properties.
  * @membw:    Bandwidth control properties.
  */
struct resctrl_ctrl {
    struct list_head    entry;
    enum resctrl_scope    scope;
    struct list_head    domains;
    enum resctrl_ctrl_type    type;
    enum resctrl_ctrl_name    name;
    union {
        struct resctrl_cache    cache;
        struct resctrl_membw    membw;
    };
};

Two members summarize how this new structure fits into the rest of resctrl:
a) resctrl_ctrl::entry
    Since a resource can support multiple controls there is a new list in struct rdt_resource named "controls" that contains the list of all controls supported by the resource.
b) resctrl_ctrl::domains
    Instead of the list of control domains belonging to a resource they now belong to the control self. By doing so resctrl can support resource controls at different scope for the same resource. This is intended to support some upcoming MPAM and RISC-V usages.

Example architectural data structure changes
============================================

An architecture can use the new control by following a similar pattern to resource and domain use by architectures. Consider the following for x86 where a new architecture specific struct resctrl_hw_ctrl includes struct resctrl_ctrl and any architecture private data needed to support the control:

/*
  * struct resctrl_hw_ctrl - Arch private properties of a resource control
  * @r_ctrl:    Control properties exposed to resctrl file system
  * @msr_base:    Base MSR address where control values should be programmed
  * @msr_update:    Function pointer to update control values
  */
struct resctrl_hw_ctrl {
    struct resctrl_ctrl    r_ctrl;
    unsigned int        msr_base;
    void            (*msr_update)(struct msr_param *m);
};

Structure of patch series
=========================

As a PoC the series is not perfectly structured but to help navigate this work on a high level the changes can be categorized as follows:

Patch 1 to 11:
With a vision of what a "control" is, remove unused/unnecessary members, make clear what is a *resource* property vs a *control* property, do some renaming to help with the PoC.

Patch 12:
Introduce struct resctrl_ctrl and re-arrange existing struct rdt_resource members to form part of new rdt_resource::ctrl

Patch 13 to 44:
A lot of wrangling to introduce struct resctrl_ctrl to all code that needs to work with a control and/or domain without assuming that the control is the one and only control embedded in the resource it belongs to. Essentially, a lot of changes passing the control around in addition to the resource/domain.

Patch 45:
Switch the single struct resctrl_ctrl member of struct rdt_resource to be a list of struct resctrl_ctrl.

Patch 47 to 49:
Introduce new info/<resource>/resource_schemata hierarchy to first only consist of properties already known to resctrl fs.

Patch 50 to 52:
Introduce the new control properties per [1], initialize them for x86, and expose them via info/<resource>/resource_schemata

Patch 53:
Let the new struct resctrl_hw_ctrl contain architecture's control properties.

Patch 54:
Teach resctrl fs about "MIN" and "MAX" controls.

Patch 55:
Sample of "MIN" and "MAX" memory bandwidth controls for x86.

Example interactions
====================

Quoted 8 lines

This series can be used on an x86 system where it will show two new dummy controls where it is possible to interact with the new controls.
For example:

# cat schemata
         MB_MAX:0=100;1=100
         MB_MIN:0=100;1=100
         MB:0=100;1=100

Some platforms may support CPU-less node which is represented by numa node id, examples:
1. CXL type 2 memory node which provides CXL memory without CPU and L3 on the node
2. GPU memory node that can be accessed by all CPUs but doesn't have a local CPU and L3 bound to. etc.

MPAM can allocate and monitor mem bandwidth on these memory node.
Since no CPU and L3 on the node, cache id cannot be used in "MB:" line. Instead, numa ids are used to identify MB allocation and monitoring.

For example, the MB allocation on CPU-less platforms could be:
MB:0=100;1=100;2=100;10=100;18=100;26=100

Where: domain id 0, 1, 2, etc are numa node id shown in /sys/devices/system/node directory or by numctl. 0: socket 0, node 0, CPUs, memory
1: socket 1, node 1, CPUs, memory
2: GPU 0, node 2, no CPU, memory only
10: GPU 1, node 10, no CPU, memory only
18: GPU 3, node 18, no CPU, memory only
26: GPU 4, node 26, no CPU, memory only

Quoted 1 line

Arch specific driver (e.g. MPAM) detects CPU-less node. If there is any CPU-less node, use numa id in "MB:". Otherwise, fallback to legacy cache id.

We always have to consider backward compatibility and to do so we cannot just retroactively change what domain ID represents when user space interacts with the "MB" control.

The legacy "MB" control is already defined and its domain ID represents an L3 cache ID. To support these new devices resctrl would need to expose a new control.

Quoted 8 lines

There is another MPAM feature called MBW Max hardlimit which sets "MB:" allocation as hardlimit (i.e. MBW throttling percentage must be satisfied) per domain. Adding a new "MB_HLIM:" line in schemata. It's 1:1 mapped to "MB:" to control hardlimit of MB throttling percentage on each domain. By default hardlimit is off (0) and can be turned on to set MBW Max hardlimit on a domain.

ack. This sounds like a new control associated with the MB resource. This is a boolean control as Dave highlighted in previous discussion so resctrl would need to know its properties.
See https://lore.kernel.org/lkml/[email protected]/

Quoted 8 lines

For exmple:
MB_HLIM: 0=0;1=0;2=1;10=0;18=0;26=0
MB:0=100;1=100;2=80;10=100;18=100;26=100

On GPU memory numa node 2: cannot use more than 80% of total max mbw even if there is still idle mem bandwidth on this node).

MBW allocations on all other domains are soft limited, meaning MBW can be used more than specified if mem is idle.

ack.

Quoted 19 lines
Quoted 13 lines

         L3:0=fff;1=fff

# echo 'MB_MIN:0=50' > schemata
# cat schemata
         MB_MAX:0=100;1=100
         MB_MIN:0=50;1=100
         MB:0=100;1=100
         L3:0=fff;1=fff

Writing to the dummy control will call a dummy callback that just prints to the kernel log:
"resctrl: Updata temporary MIN control on domain 0 with user value 50"

Quoted 5 lines

Example output of info/MB/:
/sys/fs/resctrl/info/MB/thread_throttle_mode:max
/sys/fs/resctrl/info/MB/num_closids:15
/sys/fs/resctrl/info/MB/delay_linear:1
/sys/fs/resctrl/info/MB/min_bandwidth:10

Quoted 3 lines

Add two new MB info RO files:
1. /sys/fs/resctrl/info/MB/domain_id
It shows "numa" for using numa id in "MB:" or "cache" for using legacy cache id.

This proposal introduces a *global* property to the MB *resource*? It does not seem as though this takes into account *anything* about how resctrl can support new hardware that has been discussed before, during, or after LPC. You have not participated in these discussions and now make an orthogonal proposal that does not take into account *any* of the requirements that we have been struggling with for months.

Why should this proposal be taken seriously? In your absence folks have been trying to accommodate how these upcoming products and be supported and the "scope" file associated with a control is intended to communicate to user space how the domain ID should be interpreted.

Why are you proposing something entirely different here without even acknowledging current approach and explaining why it does not work for you?

Quoted 3 lines

2. /sys/fs/resctrl/info/MB/max_lim
It shows number 0-3 for MPAM MBW max limit behaviors: 0 for supporting both softlimit and hardlimit, etc.

Again this adds another *global* property to the MB resource but then above you describe the new "MB_HLIM" schemata file entry that implies that it is a new control for the MB resource. Having it be a new control for the MB resource matches earlier discussions. To support this I thus expect it to be exposed as a new control with potentially a new type if any of the existing planned types do not suffice.

Quoted 3 lines
Quoted 1 line

/sys/fs/resctrl/info/MB/resource_schemata/MB/resolution:100

Is it more concise to s/resource_schemata/schemata/? "resource_" seems redundant in the context "info/MB".

We could do this, yes.

Quoted 26 lines
Quoted 17 lines

/sys/fs/resctrl/info/MB/resource_schemata/MB/tolerance:5 /sys/fs/resctrl/info/MB/resource_schemata/MB/type:scalar /sys/fs/resctrl/info/MB/resource_schemata/MB/min:10 /sys/fs/resctrl/info/MB/resource_schemata/MB/scale:1 /sys/fs/resctrl/info/MB/resource_schemata/MB/scope:L3 /sys/fs/resctrl/info/MB/resource_schemata/MB/unit:all /sys/fs/resctrl/info/MB/resource_schemata/MB/max:100 /sys/fs/resctrl/info/MB/resource_schemata/MB_MIN/resolution:100 /sys/fs/resctrl/info/MB/resource_schemata/MB_MIN/tolerance:5 /sys/fs/resctrl/info/MB/resource_schemata/MB_MIN/type:scalar /sys/fs/resctrl/info/MB/resource_schemata/MB_MIN/min:10 /sys/fs/resctrl/info/MB/resource_schemata/MB_MIN/scale:1 /sys/fs/resctrl/info/MB/resource_schemata/MB_MIN/scope:L3 /sys/fs/resctrl/info/MB/resource_schemata/MB_MIN/unit:all /sys/fs/resctrl/info/MB/resource_schemata/MB_MIN/max:100 /sys/fs/resctrl/info/MB/resource_schemata/MB_MAX/resolution:100

Quoted 8 lines

/sys/fs/resctrl/info/MB/resource_schemata/MB_MAX/tolerance:5 /sys/fs/resctrl/info/MB/resource_schemata/MB_MAX/type:scalar /sys/fs/resctrl/info/MB/resource_schemata/MB_MAX/min:10 /sys/fs/resctrl/info/MB/resource_schemata/MB_MAX/scale:1 /sys/fs/resctrl/info/MB/resource_schemata/MB_MAX/scope:L3 /sys/fs/resctrl/info/MB/resource_schemata/MB_MAX/unit:all /sys/fs/resctrl/info/MB/resource_schemata/MB_MAX/max:100 /sys/fs/resctrl/info/MB/bandwidth_gran:10

Quoted 1 line

For MBW monitoring, extend mon_data/ directory to monitor CPU-less memory node. For example,

Here is where I attempted to discuss with you how to support monitoring on these systems:
https://lore.kernel.org/lkml/[email protected]/

Here again you respond with something completely different without acknowledging the previous discussion or noting why that does not work for you.

Quoted 7 lines

On legacy platforms (i.e. L3 and memory are described in same MPAM ACPI MSC wich doesn't support CPU-less nodes):
mon_data/mbm_L3_01/llc_occupancy
mon_data/mbm_L3_01/mbm_total_bytes
mon_data/mbm_L3_02/llc_occupancy
mon_data/mbm_L3_02/mbm_total_bytes

Quoted 6 lines

On platforms with L3 and memory in separate MPAM ACPI MSCs but there is no CPU-less node:
mon_data/mbm_L3_01/llc_occupancy    <- cache id 1 mon_data/mbm_L3_02/llc_occupancy    <- cache id 2 mon_data/mbm_MB_00/mbm_total_bytes  <- numa node 0 (socket 0) mon_data/mbm_MB_01/mbm_total_bytes  <- numa node 1 (socket 1)

Here too I do not find it appropriate for resctrl to retroactively change its interface. "MB" is a resource and the above switches the resctrl interface to imply the monitoring data of a resource can be found in the mon directory that matches the resource name. This is not what resctrl does today. Doing something like above will result in resctrl having a confusing interface where "sometimes" memory bandwidth data can be found in the L3 directory and "sometimes" memory bandwidth data can be found in the MB directory.

As I described to you in December resctrl already exposes the monitoring data based on the *scope*. As you also point out above, today the memory bandwidth monitoring data at L3 scope can be found in the L3 directory. "L3" should thus not be interpreted as the resource L3 but the scope L3 since it contains MBM data today. When viewing it as such resctrl could internally be more explicit and separate monitoring scope from monitoring resource and present the monitoring data based on scope to remain intuitively backward compatible while obtaining support for these memory nodes.

Quoted 11 lines

On platforms with L3 and memory in separate MPAM ACPI MSCs and there are CPU-less nodes:
mon_data/mbm_L3_01/llc_occupancy    <- cache id 1 mon_data/mbm_L3_02/llc_occupancy    <- cache id 2 mon_data/mbm_MB_00/mbm_total_bytes  <- numa node 0 (socket 0) mon_data/mbm_MB_01/mbm_total_bytes  <- numa node 1 (socket 1) mon_data/mbm_MB_02/mbm_total_bytes  <- numa node 2 (GPU 0 mem) mon_data/mbm_MB_10/mbm_total_bytes  <- numa node 10 (GPU 1 mem) mon_data/mbm_MB_18/mbm_total_bytes  <- numa node 18 (GPU 2 mem) mon_data/mbm_MB_26/mbm_total_bytes  <- numa node 26 (GPU 3 mem)

To be backward compatible I find it more intuitive if instead this data is exposed as below:

mon_data/mbm_L3_01/llc_occupancy <- cache id 1
mon_data/mbm_L3_02/llc_occupancy <- cache id 2

mon_data/mbm_NODE_00/mbm_total_bytes  <- numa node 0 (socket 0)
mon_data/mbm_NODE_01/mbm_total_bytes  <- numa node 1 (socket 1)
mon_data/mbm_NODE_02/mbm_total_bytes  <- numa node 2 (GPU 0 mem)
mon_data/mbm_NODE_10/mbm_total_bytes  <- numa node 10 (GPU 1 mem)
mon_data/mbm_NODE_18/mbm_total_bytes  <- numa node 18 (GPU 2 mem)
mon_data/mbm_NODE_26/mbm_total_bytes  <- numa node 26 (GPU 3 mem)

When "mbm_total_bytes" move from mon_data/mbm_L3_x to mon_data/mbm_NODE_x it clearly indicates that it is memory bandwidth monitoring data moving from "L3" scope to "NODE" scope.

Reinette

reviewer
#63 lore raw

Hi, Reinette,

Quoted 7 lines · Reinette Chatre

Hi Fenghua,

Quoted 4 lines · Fenghua Yu

Hi, Reinette, Ben, Shaopen, et al,

On 5/29/26 11:06, Reinette Chatre wrote:

Quoted 7 lines
Quoted 4 lines

As Shaopen and Ben mentioned earlier, we are working on two MPAM features that may need to change schemata interface. The CPU-less feature was discussed on LPC (although the interfaces will be slightly different from the LPC).

I know. Here is where I tried to engage with you on needed interfaces after LPC:
https://lore.kernel.org/lkml/[email protected]/

MPAM ACPI defines MSC (Memory System Control) is defined in one of two ways (not both) on one platform:
1. L3 and memory together on each processor MSC
2. L3 in processor MSC and memory control/monitoring in different memory MSCs.

On type 1 platform, schemata is legacy:
MB:1=100;2=100 <-- cache id 1 and 2 as domain id

On type 2 platform, I will not reuse "MB:" name. Instead, define new resource name "MBN:" for numa node and schemata is:
MBN:0=100;1=100;2=100;10=100;18=100;26=100 <-- numa id 0, 1, 2, 10, 18,
26 as domain id
On type 2 platform, there won't be "MB:" line. Numa 0 and 1
are for mbm allocation on socket 0 and 1. 2,10, 18 and 26 are for GPU
memory nodes allocation.

BTW, Slow MBA (SMBA) is different from MBA Numa (MBN). SMBA still relies on L3 and the domain id in SMBA is still cache id. MBN depends on each memory controlor with numa id as domain id for both CPU and CPU-less memory nodes.

On type 1 platform, there is only MB:

info

└── MB
    └── resource_schemata
        ├── MB
        │   ├── max
        │   ├── min
        │   ├── resolution
        │   ├── scale
        │   ├── scope <== contains "L3"
        │   ├── tolerance
        │   ├── type
        │   └── unit

On type 2 platform, there is only MBN:
info

└── MBN
    └── resource_schemata
        ├── MBN
        │   ├── max
        │   ├── min
        │   ├── resolution
        │   ├── scale
        │   ├── scope <== contains "NUMA"
        │   ├── tolerance
        │   ├── type
        │   └── unit

This is different from the "scope" hierarchy discussed in the link. "MB" and "MBN" won't exist on the same platform.

I find it's hard (and not useful) to split "MB" for memory with CPU and "MBN" for CPU-less memory node. It's easier to have either "MB" for legacy memory with CPU or "MBN" for CPU-less memory.

Any thoughts? Does this update make sense?

Quoted 15 lines
Quoted 2 lines

Hardlimit feature was not discussed yet.

It was considered. See discussion starting at
https://lore.kernel.org/lkml/[email protected]/

Quoted 1 line

It's good to discuss this further in this RFC thread before the new features RFCs will be sent out.

ack.

Quoted 5 lines

Overall, the new features can fit into this RFC well.

Quoted 2 lines

Hi Everybody,

[SNIP]

Quoted 29 lines
Quoted 29 lines
Quoted 9 lines

This series can be used on an x86 system where it will show two new dummy controls where it is possible to interact with the new controls.
For example:

# cat schemata
          MB_MAX:0=100;1=100
          MB_MIN:0=100;1=100
          MB:0=100;1=100

Some platforms may support CPU-less node which is represented by numa node id, examples:
1. CXL type 2 memory node which provides CXL memory without CPU and L3 on the node
2. GPU memory node that can be accessed by all CPUs but doesn't have a local CPU and L3 bound to. etc.

MPAM can allocate and monitor mem bandwidth on these memory node.
Since no CPU and L3 on the node, cache id cannot be used in "MB:" line. Instead, numa ids are used to identify MB allocation and monitoring.

For example, the MB allocation on CPU-less platforms could be:
MB:0=100;1=100;2=100;10=100;18=100;26=100

Where: domain id 0, 1, 2, etc are numa node id shown in /sys/devices/system/node directory or by numctl. 0: socket 0, node 0, CPUs, memory
1: socket 1, node 1, CPUs, memory
2: GPU 0, node 2, no CPU, memory only
10: GPU 1, node 10, no CPU, memory only
18: GPU 3, node 18, no CPU, memory only
26: GPU 4, node 26, no CPU, memory only

Quoted 8 lines
Quoted 1 line

Arch specific driver (e.g. MPAM) detects CPU-less node. If there is any CPU-less node, use numa id in "MB:". Otherwise, fallback to legacy cache id.

We always have to consider backward compatibility and to do so we cannot just retroactively change what domain ID represents when user space interacts with the "MB" control.

The legacy "MB" control is already defined and its domain ID represents an L3 cache ID. To support these new devices resctrl would need to expose a new control.

Agree. Add new control "MBN" for Memory Bandwidth Allocation on numa node. See above.

Quoted 9 lines
Quoted 8 lines

There is another MPAM feature called MBW Max hardlimit which sets "MB:" allocation as hardlimit (i.e. MBW throttling percentage must be satisfied) per domain. Adding a new "MB_HLIM:" line in schemata. It's 1:1 mapped to "MB:" to control hardlimit of MB throttling percentage on each domain. By default hardlimit is off (0) and can be turned on to set MBW Max hardlimit on a domain.

Quoted 5 lines

ack. This sounds like a new control associated with the MB resource. This is a boolean control as Dave highlighted in previous discussion so resctrl would need to know its properties.
See https://lore.kernel.org/lkml/[email protected]/

Right. ("MB_HLIM" name may be adjusted accordingly when "MB_MAX" is available.)

Quoted 40 lines
Quoted 8 lines

For exmple:
MB_HLIM: 0=0;1=0;2=1;10=0;18=0;26=0
MB:0=100;1=100;2=80;10=100;18=100;26=100

On GPU memory numa node 2: cannot use more than 80% of total max mbw even if there is still idle mem bandwidth on this node).

MBW allocations on all other domains are soft limited, meaning MBW can be used more than specified if mem is idle.

ack.

Quoted 19 lines
Quoted 13 lines

         L3:0=fff;1=fff

# echo 'MB_MIN:0=50' > schemata
# cat schemata
          MB_MAX:0=100;1=100
          MB_MIN:0=50;1=100
          MB:0=100;1=100
          L3:0=fff;1=fff

Writing to the dummy control will call a dummy callback that just prints to the kernel log:
"resctrl: Updata temporary MIN control on domain 0 with user value 50"

Quoted 5 lines

Example output of info/MB/:
/sys/fs/resctrl/info/MB/thread_throttle_mode:max
/sys/fs/resctrl/info/MB/num_closids:15
/sys/fs/resctrl/info/MB/delay_linear:1
/sys/fs/resctrl/info/MB/min_bandwidth:10

Quoted 3 lines

Add two new MB info RO files:
1. /sys/fs/resctrl/info/MB/domain_id
It shows "numa" for using numa id in "MB:" or "cache" for using legacy cache id.

This proposal introduces a *global* property to the MB *resource*? It does not seem as though this takes into account *anything* about how resctrl can support new hardware that has been discussed before, during, or after LPC. You have not participated in these discussions and now make an orthogonal proposal that does not take into account *any* of the requirements that we have been struggling with for months.

Quoted 7 lines

Why should this proposal be taken seriously? In your absence folks have been trying to accommodate how these upcoming products and be supported and the "scope" file associated with a control is intended to communicate to user space how the domain ID should be interpreted.

Why are you proposing something entirely different here without even acknowledging current approach and explaining why it does not work for you?

So can I change this part to adding the following files in info dirctory?

1. For numa memory bw allocation (MBN):
/sys/fs/resctrl/info/MBN/resource_schemata/MBN/ /sys/fs/resctrl/info/MBN/resource_schemata/MBN/resolution:100 /sys/fs/resctrl/info/MBN/resource_schemata/MBN/tolerance:5 /sys/fs/resctrl/info/MBN/resource_schemata/MBN/type:scalar /sys/fs/resctrl/info/MBN/resource_schemata/MBN/min:10 /sys/fs/resctrl/info/MBN/resource_schemata/MBN/scale:1 /sys/fs/resctrl/info/MBN/resource_schemata/MBN/scope:NUMA /sys/fs/resctrl/info/MBN/resource_schemata/MBN/unit:all /sys/fs/resctrl/info/MBN/resource_schemata/MBN/max:100

Quoted 9 lines
Quoted 2 lines

2. /sys/fs/resctrl/info/MB/max_lim
It shows number 0-3 for MPAM MBW max limit behaviors: 0 for supporting both softlimit and hardlimit, etc.

Again this adds another *global* property to the MB resource but then above you describe the new "MB_HLIM" schemata file entry that implies that it is a new control for the MB resource. Having it be a new control for the MB resource matches earlier discussions. To support this I thus expect it to be exposed as a new control with potentially a new type if any of the existing planned types do not suffice.

How about adding these MB_HLIM dir and files in info?

/sys/fs/resctrl/info/MB_HLIM/resource_schemata/MB_HLIM/type: boolean /sys/fs/resctrl/info/MB_HLIM/resource_schemata/MB_HLIM/max_lim: 0

Quoted 95 lines
Quoted 3 lines
Quoted 1 line

/sys/fs/resctrl/info/MB/resource_schemata/MB/resolution:100

Is it more concise to s/resource_schemata/schemata/? "resource_" seems redundant in the context "info/MB".

We could do this, yes.

Quoted 26 lines
Quoted 17 lines

/sys/fs/resctrl/info/MB/resource_schemata/MB/tolerance:5 /sys/fs/resctrl/info/MB/resource_schemata/MB/type:scalar /sys/fs/resctrl/info/MB/resource_schemata/MB/min:10 /sys/fs/resctrl/info/MB/resource_schemata/MB/scale:1 /sys/fs/resctrl/info/MB/resource_schemata/MB/scope:L3 /sys/fs/resctrl/info/MB/resource_schemata/MB/unit:all /sys/fs/resctrl/info/MB/resource_schemata/MB/max:100 /sys/fs/resctrl/info/MB/resource_schemata/MB_MIN/resolution:100 /sys/fs/resctrl/info/MB/resource_schemata/MB_MIN/tolerance:5 /sys/fs/resctrl/info/MB/resource_schemata/MB_MIN/type:scalar /sys/fs/resctrl/info/MB/resource_schemata/MB_MIN/min:10 /sys/fs/resctrl/info/MB/resource_schemata/MB_MIN/scale:1 /sys/fs/resctrl/info/MB/resource_schemata/MB_MIN/scope:L3 /sys/fs/resctrl/info/MB/resource_schemata/MB_MIN/unit:all /sys/fs/resctrl/info/MB/resource_schemata/MB_MIN/max:100 /sys/fs/resctrl/info/MB/resource_schemata/MB_MAX/resolution:100

Quoted 8 lines

/sys/fs/resctrl/info/MB/resource_schemata/MB_MAX/tolerance:5 /sys/fs/resctrl/info/MB/resource_schemata/MB_MAX/type:scalar /sys/fs/resctrl/info/MB/resource_schemata/MB_MAX/min:10 /sys/fs/resctrl/info/MB/resource_schemata/MB_MAX/scale:1 /sys/fs/resctrl/info/MB/resource_schemata/MB_MAX/scope:L3 /sys/fs/resctrl/info/MB/resource_schemata/MB_MAX/unit:all /sys/fs/resctrl/info/MB/resource_schemata/MB_MAX/max:100 /sys/fs/resctrl/info/MB/bandwidth_gran:10

Quoted 1 line

For MBW monitoring, extend mon_data/ directory to monitor CPU-less memory node. For example,

Here is where I attempted to discuss with you how to support monitoring on these systems:
https://lore.kernel.org/lkml/[email protected]/

Here again you respond with something completely different without acknowledging the previous discussion or noting why that does not work for you.

Quoted 7 lines

On legacy platforms (i.e. L3 and memory are described in same MPAM ACPI MSC wich doesn't support CPU-less nodes):
mon_data/mbm_L3_01/llc_occupancy
mon_data/mbm_L3_01/mbm_total_bytes
mon_data/mbm_L3_02/llc_occupancy
mon_data/mbm_L3_02/mbm_total_bytes

Quoted 6 lines

On platforms with L3 and memory in separate MPAM ACPI MSCs but there is no CPU-less node:
mon_data/mbm_L3_01/llc_occupancy    <- cache id 1 mon_data/mbm_L3_02/llc_occupancy    <- cache id 2 mon_data/mbm_MB_00/mbm_total_bytes  <- numa node 0 (socket 0) mon_data/mbm_MB_01/mbm_total_bytes  <- numa node 1 (socket 1)

Here too I do not find it appropriate for resctrl to retroactively change its interface. "MB" is a resource and the above switches the resctrl interface to imply the monitoring data of a resource can be found in the mon directory that matches the resource name. This is not what resctrl does today. Doing something like above will result in resctrl having a confusing interface where "sometimes" memory bandwidth data can be found in the L3 directory and "sometimes" memory bandwidth data can be found in the MB directory.

As I described to you in December resctrl already exposes the monitoring data based on the *scope*. As you also point out above, today the memory bandwidth monitoring data at L3 scope can be found in the L3 directory. "L3" should thus not be interpreted as the resource L3 but the scope L3 since it contains MBM data today. When viewing it as such resctrl could internally be more explicit and separate monitoring scope from monitoring resource and present the monitoring data based on scope to remain intuitively backward compatible while obtaining support for these memory nodes.

Quoted 11 lines

On platforms with L3 and memory in separate MPAM ACPI MSCs and there are CPU-less nodes:
mon_data/mbm_L3_01/llc_occupancy    <- cache id 1 mon_data/mbm_L3_02/llc_occupancy    <- cache id 2 mon_data/mbm_MB_00/mbm_total_bytes  <- numa node 0 (socket 0) mon_data/mbm_MB_01/mbm_total_bytes  <- numa node 1 (socket 1) mon_data/mbm_MB_02/mbm_total_bytes  <- numa node 2 (GPU 0 mem) mon_data/mbm_MB_10/mbm_total_bytes  <- numa node 10 (GPU 1 mem) mon_data/mbm_MB_18/mbm_total_bytes  <- numa node 18 (GPU 2 mem) mon_data/mbm_MB_26/mbm_total_bytes  <- numa node 26 (GPU 3 mem)

To be backward compatible I find it more intuitive if instead this data is exposed as below:

mon_data/mbm_L3_01/llc_occupancy <- cache id 1
mon_data/mbm_L3_02/llc_occupancy <- cache id 2

mon_data/mbm_NODE_00/mbm_total_bytes  <- numa node 0 (socket 0)
mon_data/mbm_NODE_01/mbm_total_bytes  <- numa node 1 (socket 1)
mon_data/mbm_NODE_02/mbm_total_bytes  <- numa node 2 (GPU 0 mem)
mon_data/mbm_NODE_10/mbm_total_bytes  <- numa node 10 (GPU 1 mem)
mon_data/mbm_NODE_18/mbm_total_bytes  <- numa node 18 (GPU 2 mem)
mon_data/mbm_NODE_26/mbm_total_bytes  <- numa node 26 (GPU 3 mem)
Quoted 3 lines

When "mbm_total_bytes" move from mon_data/mbm_L3_x to mon_data/mbm_NODE_x it clearly indicates that it is memory bandwidth monitoring data moving from "L3" scope to "NODE" scope.

The scope in mon_data shown in https://lore.kernel.org/lkml/[email protected]/:

mon_data

├── mon_L3_00       <== monitoring data at scope L3
│   ├── llc_occupancy
│   ├── mbm_local_bytes
│   └── mbm_total_bytes
├── mon_L3_01       <== monitoring data at scope L3
│   ├── llc_occupancy
│   ├── mbm_local_bytes
│   └── mbm_total_bytes
├── mon_NODE_00     <== monitoring data at scope NODE
│   └── mbm_total_bytes
└── mon_NODE_01     <== monitoring data at scope NODE
    └── mbm_total_bytes

On some ARM platforms, for example, socket 0 (CPUs+L3) and socket 1 (CPUs+L3):
llc_occupancy is monitored through processor/L3. total_bytes are monitored through memory controlor.

Then the above "scope" is confused:
mon_L3_00 and mon_L3_01 only has llc_occupancy. The scope "L3" is fine here.
But mon_NODE_00 and 01 are confused because numa node 00 and 01 have both L3 and memory. The name "mon_NODE_00" seems monitor both llc_occupany and total_bytes but it only monitor total_bytes. And on CPU-less platforms, a CPU-less node does only have total_bytes which seems match "scope NODE".

With "scope NODE", can user tell if it has llc_occupancy or total_bytes or both?

If change "mon_MB_01" to "mon_MM_01", the scope is "MM" now which means monitoring memory (total_bytes) in numa node 1 with "scope MM" (not L3)?

Thanks.

-Fenghua

author maintainer
#64 lore raw

Hi Fenghua,

Quoted 25 lines · Fenghua Yu

Hi, Reinette,

Quoted 7 lines · Reinette Chatre

Hi Fenghua,

Quoted 4 lines · Fenghua Yu

Hi, Reinette, Ben, Shaopen, et al,

On 5/29/26 11:06, Reinette Chatre wrote:

Quoted 7 lines
Quoted 4 lines

As Shaopen and Ben mentioned earlier, we are working on two MPAM features that may need to change schemata interface. The CPU-less feature was discussed on LPC (although the interfaces will be slightly different from the LPC).

I know. Here is where I tried to engage with you on needed interfaces after LPC:
https://lore.kernel.org/lkml/[email protected]/

MPAM ACPI defines MSC (Memory System Control) is defined in one of two ways (not both) on one platform:
1. L3 and memory together on each processor MSC
2. L3 in processor MSC and memory control/monitoring in different memory MSCs.

On type 1 platform, schemata is legacy:
MB:1=100;2=100  <-- cache id 1 and 2 as domain id

Quoted 6 lines

On type 2 platform, I will not reuse "MB:" name. Instead, define new resource name "MBN:" for numa node and schemata is:
MBN:0=100;1=100;2=100;10=100;18=100;26=100 <-- numa id 0, 1, 2, 10, 18,
                           26 as domain id
On type 2 platform, there won't be "MB:" line. Numa 0 and 1
are for mbm allocation on socket 0 and 1. 2,10, 18 and 26 are for GPU
memory nodes allocation.

(to help make things explicit I will refer to what you call "MBN" as "MB_NODE" to make it explicit that it is memory bandwidth allocation at node scope)

I am trying to consider how this can be accomplished while also considering all the other new hardware features that resctrl need to support. Consider, for example, AMD's "Global MBA" (https://lore.kernel.org/lkml/[email protected]/) that throttles memory bandwidth at L3 scope but the user configures allocations at NODE scope. At this time the plan is to support this with a second control associated with the MB resource that can allocate memory bandwidth at node scope. See
https://lore.kernel.org/lkml/[email protected]/

If resctrl creates a new resource for node scoped memory bandwidth allocations to support these "type 2" systems then that will result in an inconsistent interface between architectures that we should avoid.

Have you been listening in on the discussions surrounding emulated controls? Considering that, would it be possible to support the "MB" control on a type "2" system but have it be backed by (emulated by) the underlying "MB_NODE" control?

resctrl could expose both controls on these "type 2" systems but make it clear that "MB" is emulated by "MB_NODE". For example:

info/

└── MB/
    └── resource_schemata/
        └── MB/
            └── MB_NODE/

User will see both controls in schemata file but when changes are made to "MB" control it will show in the "MB_NODE" control and vice-versa. User could also disable the "MB" control that will establish familiarity with the interface at which point resctrl can drop the "MB" control from the schemata file on these "type 2" systems.

Having the MB resource available with an MB control will keep resctrl backward compatible if there are any tools that expect that. If backward compatibility is not of concern then resctrl could initialize with the emulated control disabled by default. See discussion at
https://lore.kernel.org/lkml/[email protected]/
that describes a new resctrl capability in support of RISC-V and RDT.
With this resctrl could initialize with:

info/

└── MB/
    └── resource_schemata/
        ├── MB/
        │   ├── MB_NODE/
        │   │   └── status:enabled
        │   └── status:disabled
        └── mode:legacy [native]

With above a "type 2" system will boot with its schemata file just containing the "MB_NODE" control while info/MB describes the memory bandwidth resource.

If AMD is ok with naming their "Global MBA" allocations "MB_NODE" then a user working with AMD and these MPAM systems could find the controls in the same hierarchy and need not use external knowledge to determine how to interact with resctrl fs.

Quoted 4 lines

BTW, Slow MBA (SMBA) is different from MBA Numa (MBN). SMBA still relies on L3 and the domain id in SMBA is still cache id. MBN depends on each memory controlor with numa id as domain id for both CPU and CPU-less memory nodes.

ack. Similar to AMD's "Global MBA" there is also a new feature of "Global SMBA" that needs to be supported by resctrl.

Quoted 29 lines

On type 1 platform, there is only MB:

info

└── MB
    └── resource_schemata
        ├── MB
        │   ├── max
        │   ├── min
        │   ├── resolution
        │   ├── scale
        │   ├── scope <== contains "L3"
        │   ├── tolerance
        │   ├── type
        │   └── unit

On type 2 platform, there is only MBN:
info

└── MBN
    └── resource_schemata
        ├── MBN
        │   ├── max
        │   ├── min
        │   ├── resolution
        │   ├── scale
        │   ├── scope <== contains "NUMA"
        │   ├── tolerance
        │   ├── type
        │   └── unit
Quoted 5 lines

This is different from the "scope" hierarchy discussed in the link. "MB" and "MBN" won't exist on the same platform.

I find it's hard (and not useful) to split "MB" for memory with CPU and "MBN" for CPU-less memory node. It's easier to have either "MB" for legacy memory with CPU or "MBN" for CPU-less memory.

Please widen your considerations to include how resctrl can maintain backward compatibility and how enabling of these platforms can fit well with the "similar but not identical" hardware features from other architectures that also needs to be supported by resctrl.

Quoted 127 lines

Any thoughts? Does this update make sense?

Quoted 15 lines
Quoted 2 lines

Hardlimit feature was not discussed yet.

It was considered. See discussion starting at
https://lore.kernel.org/lkml/[email protected]/

Quoted 1 line

It's good to discuss this further in this RFC thread before the new features RFCs will be sent out.

ack.

Quoted 5 lines

Overall, the new features can fit into this RFC well.

Quoted 2 lines

Hi Everybody,

[SNIP]

Quoted 29 lines
Quoted 29 lines
Quoted 9 lines

This series can be used on an x86 system where it will show two new dummy controls where it is possible to interact with the new controls.
For example:

# cat schemata
          MB_MAX:0=100;1=100
          MB_MIN:0=100;1=100
          MB:0=100;1=100

Some platforms may support CPU-less node which is represented by numa node id, examples:
1. CXL type 2 memory node which provides CXL memory without CPU and L3 on the node
2. GPU memory node that can be accessed by all CPUs but doesn't have a local CPU and L3 bound to. etc.

MPAM can allocate and monitor mem bandwidth on these memory node.
Since no CPU and L3 on the node, cache id cannot be used in "MB:" line. Instead, numa ids are used to identify MB allocation and monitoring.

For example, the MB allocation on CPU-less platforms could be:
MB:0=100;1=100;2=100;10=100;18=100;26=100

Where: domain id 0, 1, 2, etc are numa node id shown in /sys/devices/system/node directory or by numctl. 0: socket 0, node 0, CPUs, memory
1: socket 1, node 1, CPUs, memory
2: GPU 0, node 2, no CPU, memory only
10: GPU 1, node 10, no CPU, memory only
18: GPU 3, node 18, no CPU, memory only
26: GPU 4, node 26, no CPU, memory only

Quoted 8 lines
Quoted 1 line

Arch specific driver (e.g. MPAM) detects CPU-less node. If there is any CPU-less node, use numa id in "MB:". Otherwise, fallback to legacy cache id.

We always have to consider backward compatibility and to do so we cannot just retroactively change what domain ID represents when user space interacts with the "MB" control.

The legacy "MB" control is already defined and its domain ID represents an L3 cache ID. To support these new devices resctrl would need to expose a new control.

Agree. Add new control "MBN" for Memory Bandwidth Allocation on numa node. See above.

Quoted 9 lines
Quoted 8 lines

There is another MPAM feature called MBW Max hardlimit which sets "MB:" allocation as hardlimit (i.e. MBW throttling percentage must be satisfied) per domain. Adding a new "MB_HLIM:" line in schemata. It's 1:1 mapped to "MB:" to control hardlimit of MB throttling percentage on each domain. By default hardlimit is off (0) and can be turned on to set MBW Max hardlimit on a domain.

Quoted 5 lines

ack. This sounds like a new control associated with the MB resource. This is a boolean control as Dave highlighted in previous discussion so resctrl would need to know its properties.
See https://lore.kernel.org/lkml/[email protected]/

Right. ("MB_HLIM" name may be adjusted accordingly when "MB_MAX" is available.)

Quoted 40 lines
Quoted 8 lines

For exmple:
MB_HLIM: 0=0;1=0;2=1;10=0;18=0;26=0
MB:0=100;1=100;2=80;10=100;18=100;26=100

On GPU memory numa node 2: cannot use more than 80% of total max mbw even if there is still idle mem bandwidth on this node).

MBW allocations on all other domains are soft limited, meaning MBW can be used more than specified if mem is idle.

ack.

Quoted 19 lines
Quoted 13 lines

          L3:0=fff;1=fff

# echo 'MB_MIN:0=50' > schemata
# cat schemata
          MB_MAX:0=100;1=100
          MB_MIN:0=50;1=100
          MB:0=100;1=100
          L3:0=fff;1=fff

Writing to the dummy control will call a dummy callback that just prints to the kernel log:
"resctrl: Updata temporary MIN control on domain 0 with user value 50"

Quoted 5 lines

Example output of info/MB/:
/sys/fs/resctrl/info/MB/thread_throttle_mode:max
/sys/fs/resctrl/info/MB/num_closids:15
/sys/fs/resctrl/info/MB/delay_linear:1
/sys/fs/resctrl/info/MB/min_bandwidth:10

Quoted 3 lines

Add two new MB info RO files:
1. /sys/fs/resctrl/info/MB/domain_id
It shows "numa" for using numa id in "MB:" or "cache" for using legacy cache id.

This proposal introduces a *global* property to the MB *resource*? It does not seem as though this takes into account *anything* about how resctrl can support new hardware that has been discussed before, during, or after LPC. You have not participated in these discussions and now make an orthogonal proposal that does not take into account *any* of the requirements that we have been struggling with for months.

Quoted 7 lines

Why should this proposal be taken seriously? In your absence folks have been trying to accommodate how these upcoming products and be supported and the "scope" file associated with a control is intended to communicate to user space how the domain ID should be interpreted.

Why are you proposing something entirely different here without even acknowledging current approach and explaining why it does not work for you?

So can I change this part to adding the following files in info dirctory?

Quoted 10 lines

1. For numa memory bw allocation (MBN):
/sys/fs/resctrl/info/MBN/resource_schemata/MBN/ /sys/fs/resctrl/info/MBN/resource_schemata/MBN/resolution:100 /sys/fs/resctrl/info/MBN/resource_schemata/MBN/tolerance:5 /sys/fs/resctrl/info/MBN/resource_schemata/MBN/type:scalar /sys/fs/resctrl/info/MBN/resource_schemata/MBN/min:10 /sys/fs/resctrl/info/MBN/resource_schemata/MBN/scale:1 /sys/fs/resctrl/info/MBN/resource_schemata/MBN/scope:NUMA /sys/fs/resctrl/info/MBN/resource_schemata/MBN/unit:all /sys/fs/resctrl/info/MBN/resource_schemata/MBN/max:100

This is not just about adding files to the info directory. The files, directories, their relationships, and content have meaning. All I see from these proposals is an attempt to slap some new files into resctrl without any consideration to present consistent interface to users and without consideration of other architectures that need to be supported by resctrl.

resctrl needs to provide a generic and consistent interface to user space irrespective of the underlying architecture. Architectures cannot just slap some new files for their convenience.

Quoted 11 lines
Quoted 10 lines
Quoted 3 lines

2. /sys/fs/resctrl/info/MB/max_lim
It shows number 0-3 for MPAM MBW max limit behaviors: 0 for supporting both softlimit and hardlimit, etc.

Again this adds another *global* property to the MB resource but then above you describe the new "MB_HLIM" schemata file entry that implies that it is a new control for the MB resource. Having it be a new control for the MB resource matches earlier discussions. To support this I thus expect it to be exposed as a new control with potentially a new type if any of the existing planned types do not suffice.

Quoted 4 lines

How about adding these MB_HLIM dir and files in info?

/sys/fs/resctrl/info/MB_HLIM/resource_schemata/MB_HLIM/type: boolean /sys/fs/resctrl/info/MB_HLIM/resource_schemata/MB_HLIM/max_lim: 0

This presents "MB_HLIM" as a *resource* to user space. It is not a resource but a *control* of a resource, no? I thus expect it to instead look something like below that makes it clear that MB_HARDMAX is a control of the MB resource.

info

└── MB
    └── resource_schemata
        ├── MB
        └── MB_HARDMAX
Quoted 120 lines
Quoted 95 lines
Quoted 3 lines
Quoted 1 line

/sys/fs/resctrl/info/MB/resource_schemata/MB/resolution:100

Is it more concise to s/resource_schemata/schemata/? "resource_" seems redundant in the context "info/MB".

We could do this, yes.

Quoted 26 lines
Quoted 17 lines

/sys/fs/resctrl/info/MB/resource_schemata/MB/tolerance:5 /sys/fs/resctrl/info/MB/resource_schemata/MB/type:scalar /sys/fs/resctrl/info/MB/resource_schemata/MB/min:10 /sys/fs/resctrl/info/MB/resource_schemata/MB/scale:1 /sys/fs/resctrl/info/MB/resource_schemata/MB/scope:L3 /sys/fs/resctrl/info/MB/resource_schemata/MB/unit:all /sys/fs/resctrl/info/MB/resource_schemata/MB/max:100 /sys/fs/resctrl/info/MB/resource_schemata/MB_MIN/resolution:100 /sys/fs/resctrl/info/MB/resource_schemata/MB_MIN/tolerance:5 /sys/fs/resctrl/info/MB/resource_schemata/MB_MIN/type:scalar /sys/fs/resctrl/info/MB/resource_schemata/MB_MIN/min:10 /sys/fs/resctrl/info/MB/resource_schemata/MB_MIN/scale:1 /sys/fs/resctrl/info/MB/resource_schemata/MB_MIN/scope:L3 /sys/fs/resctrl/info/MB/resource_schemata/MB_MIN/unit:all /sys/fs/resctrl/info/MB/resource_schemata/MB_MIN/max:100 /sys/fs/resctrl/info/MB/resource_schemata/MB_MAX/resolution:100

Quoted 8 lines

/sys/fs/resctrl/info/MB/resource_schemata/MB_MAX/tolerance:5 /sys/fs/resctrl/info/MB/resource_schemata/MB_MAX/type:scalar /sys/fs/resctrl/info/MB/resource_schemata/MB_MAX/min:10 /sys/fs/resctrl/info/MB/resource_schemata/MB_MAX/scale:1 /sys/fs/resctrl/info/MB/resource_schemata/MB_MAX/scope:L3 /sys/fs/resctrl/info/MB/resource_schemata/MB_MAX/unit:all /sys/fs/resctrl/info/MB/resource_schemata/MB_MAX/max:100 /sys/fs/resctrl/info/MB/bandwidth_gran:10

Quoted 1 line

For MBW monitoring, extend mon_data/ directory to monitor CPU-less memory node. For example,

Here is where I attempted to discuss with you how to support monitoring on these systems:
https://lore.kernel.org/lkml/[email protected]/

Here again you respond with something completely different without acknowledging the previous discussion or noting why that does not work for you.

Quoted 7 lines

On legacy platforms (i.e. L3 and memory are described in same MPAM ACPI MSC wich doesn't support CPU-less nodes):
mon_data/mbm_L3_01/llc_occupancy
mon_data/mbm_L3_01/mbm_total_bytes
mon_data/mbm_L3_02/llc_occupancy
mon_data/mbm_L3_02/mbm_total_bytes

Quoted 6 lines

On platforms with L3 and memory in separate MPAM ACPI MSCs but there is no CPU-less node:
mon_data/mbm_L3_01/llc_occupancy    <- cache id 1 mon_data/mbm_L3_02/llc_occupancy    <- cache id 2 mon_data/mbm_MB_00/mbm_total_bytes  <- numa node 0 (socket 0) mon_data/mbm_MB_01/mbm_total_bytes  <- numa node 1 (socket 1)

Here too I do not find it appropriate for resctrl to retroactively change its interface. "MB" is a resource and the above switches the resctrl interface to imply the monitoring data of a resource can be found in the mon directory that matches the resource name. This is not what resctrl does today. Doing something like above will result in resctrl having a confusing interface where "sometimes" memory bandwidth data can be found in the L3 directory and "sometimes" memory bandwidth data can be found in the MB directory.

As I described to you in December resctrl already exposes the monitoring data based on the *scope*. As you also point out above, today the memory bandwidth monitoring data at L3 scope can be found in the L3 directory. "L3" should thus not be interpreted as the resource L3 but the scope L3 since it contains MBM data today. When viewing it as such resctrl could internally be more explicit and separate monitoring scope from monitoring resource and present the monitoring data based on scope to remain intuitively backward compatible while obtaining support for these memory nodes.

Quoted 11 lines

On platforms with L3 and memory in separate MPAM ACPI MSCs and there are CPU-less nodes:
mon_data/mbm_L3_01/llc_occupancy    <- cache id 1 mon_data/mbm_L3_02/llc_occupancy    <- cache id 2 mon_data/mbm_MB_00/mbm_total_bytes  <- numa node 0 (socket 0) mon_data/mbm_MB_01/mbm_total_bytes  <- numa node 1 (socket 1) mon_data/mbm_MB_02/mbm_total_bytes  <- numa node 2 (GPU 0 mem) mon_data/mbm_MB_10/mbm_total_bytes  <- numa node 10 (GPU 1 mem) mon_data/mbm_MB_18/mbm_total_bytes  <- numa node 18 (GPU 2 mem) mon_data/mbm_MB_26/mbm_total_bytes  <- numa node 26 (GPU 3 mem)

To be backward compatible I find it more intuitive if instead this data is exposed as below:

mon_data/mbm_L3_01/llc_occupancy    <- cache id 1 mon_data/mbm_L3_02/llc_occupancy    <- cache id 2 mon_data/mbm_NODE_00/mbm_total_bytes  <- numa node 0 (socket 0) mon_data/mbm_NODE_01/mbm_total_bytes  <- numa node 1 (socket 1) mon_data/mbm_NODE_02/mbm_total_bytes  <- numa node 2 (GPU 0 mem) mon_data/mbm_NODE_10/mbm_total_bytes  <- numa node 10 (GPU 1 mem) mon_data/mbm_NODE_18/mbm_total_bytes  <- numa node 18 (GPU 2 mem) mon_data/mbm_NODE_26/mbm_total_bytes  <- numa node 26 (GPU 3 mem)

Quoted 3 lines

When "mbm_total_bytes" move from mon_data/mbm_L3_x to mon_data/mbm_NODE_x it clearly indicates that it is memory bandwidth monitoring data moving from "L3" scope to "NODE" scope.

The scope in mon_data shown in https://lore.kernel.org/lkml/[email protected]/:

mon_data

├── mon_L3_00       <== monitoring data at scope L3
│   ├── llc_occupancy
│   ├── mbm_local_bytes
│   └── mbm_total_bytes
├── mon_L3_01       <== monitoring data at scope L3
│   ├── llc_occupancy
│   ├── mbm_local_bytes
│   └── mbm_total_bytes
├── mon_NODE_00     <== monitoring data at scope NODE
│   └── mbm_total_bytes
└── mon_NODE_01     <== monitoring data at scope NODE
    └── mbm_total_bytes

On some ARM platforms, for example, socket 0 (CPUs+L3) and socket 1 (CPUs+L3):
llc_occupancy is monitored through processor/L3. total_bytes are monitored through memory controlor.

Quoted 4 lines

Then the above "scope" is confused:
mon_L3_00 and mon_L3_01 only has llc_occupancy. The scope "L3" is fine here. But mon_NODE_00 and 01 are confused because numa node 00 and 01 have both L3 and memory. The name "mon_NODE_00" seems monitor both

While the node may have L3 and memory the directory only represents what is monitored at the particular scope.

Quoted 4 lines

llc_occupany and total_bytes but it only monitor total_bytes. And on CPU-less platforms, a CPU-less node does only have total_bytes which seems match "scope NODE".
With "scope NODE", can user tell if it has llc_occupancy or total_bytes or both?

resctrl has a "mon_features" file associated with the monitoring scope that informs user space which events can be expected in a resource group's monitoring data directories.

Quoted 4 lines

If change "mon_MB_01" to "mon_MM_01", the scope is "MM" now which means monitoring memory (total_bytes) in numa node 1 with "scope MM" (not L3)?

This sounds redundant to me since the event names already have the resource embedded. "llc_occupancy" implies L3 occupancy
"mbm_local_bytes" ... the "mbm" implies this is memory bandwidth monitoring data.

User space can infer the scope at which the monitoring data is collected from the directory the event file is in.

Reinette

reviewer
#65 lore raw

Hi, Reinette,

Quoted 24 lines · Reinette Chatre

Hi Fenghua,

Quoted 21 lines · Fenghua Yu

Hi, Reinette,

Quoted 7 lines · Reinette Chatre

Hi Fenghua,

Quoted 4 lines · Fenghua Yu

Hi, Reinette, Ben, Shaopen, et al,

On 5/29/26 11:06, Reinette Chatre wrote:

Quoted 7 lines
Quoted 4 lines

As Shaopen and Ben mentioned earlier, we are working on two MPAM features that may need to change schemata interface. The CPU-less feature was discussed on LPC (although the interfaces will be slightly different from the LPC).

I know. Here is where I tried to engage with you on needed interfaces after LPC:
https://lore.kernel.org/lkml/[email protected]/

MPAM ACPI defines MSC (Memory System Control) is defined in one of two ways (not both) on one platform:
1. L3 and memory together on each processor MSC
2. L3 in processor MSC and memory control/monitoring in different memory MSCs.

Ben said there is type 3 platform:
3. L3 cache and memory bandwidth in processor MSCs and memory bandwidth in different memory MSCs.

Quoted 10 lines
Quoted 10 lines

On type 1 platform, schemata is legacy:
MB:1=100;2=100  <-- cache id 1 and 2 as domain id

On type 2 platform, I will not reuse "MB:" name. Instead, define new resource name "MBN:" for numa node and schemata is:
MBN:0=100;1=100;2=100;10=100;18=100;26=100 <-- numa id 0, 1, 2, 10, 18,
                           26 as domain id
On type 2 platform, there won't be "MB:" line. Numa 0 and 1
are for mbm allocation on socket 0 and 1. 2,10, 18 and 26 are for GPU
memory nodes allocation.

On type 3 platform, there could be "MB:" line for L3 cache and "MB_NODE:" for numa node. Example schemata is:

     MB:1=100;2=100                 <-- cache id 1 and 2 as domain id
MB_NODE:0=100;1=100;2=100;10=100;18=100;26=100 <-- numa id 0, 1, 2, 10,
                                                   18, 26 as domain id
Quoted 50 lines

(to help make things explicit I will refer to what you call "MBN" as "MB_NODE" to make it explicit that it is memory bandwidth allocation at node scope)

I am trying to consider how this can be accomplished while also considering all the other new hardware features that resctrl need to support. Consider, for example, AMD's "Global MBA" (https://lore.kernel.org/lkml/[email protected]/) that throttles memory bandwidth at L3 scope but the user configures allocations at NODE scope. At this time the plan is to support this with a second control associated with the MB resource that can allocate memory bandwidth at node scope. See
https://lore.kernel.org/lkml/[email protected]/

If resctrl creates a new resource for node scoped memory bandwidth allocations to support these "type 2" systems then that will result in an inconsistent interface between architectures that we should avoid.

Have you been listening in on the discussions surrounding emulated controls? Considering that, would it be possible to support the "MB" control on a type "2" system but have it be backed by (emulated by) the underlying "MB_NODE" control?

resctrl could expose both controls on these "type 2" systems but make it clear that "MB" is emulated by "MB_NODE". For example:

info/

└── MB/
     └── resource_schemata/
         └── MB/
             └── MB_NODE/

User will see both controls in schemata file but when changes are made to "MB" control it will show in the "MB_NODE" control and vice-versa. User could also disable the "MB" control that will establish familiarity with the interface at which point resctrl can drop the "MB" control from the schemata file on these "type 2" systems.

Having the MB resource available with an MB control will keep resctrl backward compatible if there are any tools that expect that. If backward compatibility is not of concern then resctrl could initialize with the emulated control disabled by default. See discussion at
https://lore.kernel.org/lkml/[email protected]/
that describes a new resctrl capability in support of RISC-V and RDT.
With this resctrl could initialize with:

info/

└── MB/
     └── resource_schemata/
         ├── MB/
         │   ├── MB_NODE/
         │   │   └── status:enabled
         │   └── status:disabled
         └── mode:legacy [native]
Quoted 3 lines

With above a "type 2" system will boot with its schemata file just containing the "MB_NODE" control while info/MB describes the memory bandwidth resource.

On type 3 machine, schemata has both MB in legacy mode with cache id as domain id and MB_NODE with numa id as domain id.

Is this directory OK?

info/

 └── MB/
      └── resource_schemata/
          ├── MB/
          │   ├── MB_NODE/
          │   │   └── status:disabled
          │   └── status:enabled
          ├── MB_NODE/
          └── mode:node

1. MB and MB_NODE are shown in parallel in inf/MB/resource_schemata/
2. mode is set as "node" meaning "MB" is for L3 and "MB_NODE" is for numa node
3. Emulation "MB_NODE" is disabled (or should the "MB_NODE" sub-dir be invisible?)

Quoted 3 lines

If AMD is ok with naming their "Global MBA" allocations "MB_NODE" then a user working with AMD and these MPAM systems could find the controls in the same hierarchy and need not use external knowledge to determine how to interact with resctrl fs.

MB_NODE is good for me.

Quoted 217 lines
Quoted 6 lines

BTW, Slow MBA (SMBA) is different from MBA Numa (MBN). SMBA still relies on L3 and the domain id in SMBA is still cache id. MBN depends on each memory controlor with numa id as domain id for both CPU and CPU-less memory nodes.

ack. Similar to AMD's "Global MBA" there is also a new feature of "Global SMBA" that needs to be supported by resctrl.

Quoted 29 lines

On type 1 platform, there is only MB:

info

└── MB
     └── resource_schemata
         ├── MB
         │   ├── max
         │   ├── min
         │   ├── resolution
         │   ├── scale
         │   ├── scope <== contains "L3"
         │   ├── tolerance
         │   ├── type
         │   └── unit

On type 2 platform, there is only MBN:
info

└── MBN
     └── resource_schemata
         ├── MBN
         │   ├── max
         │   ├── min
         │   ├── resolution
         │   ├── scale
         │   ├── scope <== contains "NUMA"
         │   ├── tolerance
         │   ├── type
         │   └── unit
Quoted 5 lines

This is different from the "scope" hierarchy discussed in the link. "MB" and "MBN" won't exist on the same platform.

I find it's hard (and not useful) to split "MB" for memory with CPU and "MBN" for CPU-less memory node. It's easier to have either "MB" for legacy memory with CPU or "MBN" for CPU-less memory.

Please widen your considerations to include how resctrl can maintain backward compatibility and how enabling of these platforms can fit well with the "similar but not identical" hardware features from other architectures that also needs to be supported by resctrl.

Quoted 127 lines

Any thoughts? Does this update make sense?

Quoted 15 lines
Quoted 2 lines

Hardlimit feature was not discussed yet.

It was considered. See discussion starting at
https://lore.kernel.org/lkml/[email protected]/

Quoted 1 line

It's good to discuss this further in this RFC thread before the new features RFCs will be sent out.

ack.

Quoted 5 lines

Overall, the new features can fit into this RFC well.

Quoted 2 lines

Hi Everybody,

[SNIP]

Quoted 29 lines
Quoted 29 lines
Quoted 9 lines

This series can be used on an x86 system where it will show two new dummy controls where it is possible to interact with the new controls.
For example:

# cat schemata
           MB_MAX:0=100;1=100
           MB_MIN:0=100;1=100
           MB:0=100;1=100

Some platforms may support CPU-less node which is represented by numa node id, examples:
1. CXL type 2 memory node which provides CXL memory without CPU and L3 on the node
2. GPU memory node that can be accessed by all CPUs but doesn't have a local CPU and L3 bound to. etc.

MPAM can allocate and monitor mem bandwidth on these memory node.
Since no CPU and L3 on the node, cache id cannot be used in "MB:" line. Instead, numa ids are used to identify MB allocation and monitoring.

For example, the MB allocation on CPU-less platforms could be:
MB:0=100;1=100;2=100;10=100;18=100;26=100

Where: domain id 0, 1, 2, etc are numa node id shown in /sys/devices/system/node directory or by numctl. 0: socket 0, node 0, CPUs, memory
1: socket 1, node 1, CPUs, memory
2: GPU 0, node 2, no CPU, memory only
10: GPU 1, node 10, no CPU, memory only
18: GPU 3, node 18, no CPU, memory only
26: GPU 4, node 26, no CPU, memory only

Quoted 8 lines
Quoted 1 line

Arch specific driver (e.g. MPAM) detects CPU-less node. If there is any CPU-less node, use numa id in "MB:". Otherwise, fallback to legacy cache id.

We always have to consider backward compatibility and to do so we cannot just retroactively change what domain ID represents when user space interacts with the "MB" control.

The legacy "MB" control is already defined and its domain ID represents an L3 cache ID. To support these new devices resctrl would need to expose a new control.

Agree. Add new control "MBN" for Memory Bandwidth Allocation on numa node. See above.

Quoted 9 lines
Quoted 8 lines

There is another MPAM feature called MBW Max hardlimit which sets "MB:" allocation as hardlimit (i.e. MBW throttling percentage must be satisfied) per domain. Adding a new "MB_HLIM:" line in schemata. It's 1:1 mapped to "MB:" to control hardlimit of MB throttling percentage on each domain. By default hardlimit is off (0) and can be turned on to set MBW Max hardlimit on a domain.

Quoted 5 lines

ack. This sounds like a new control associated with the MB resource. This is a boolean control as Dave highlighted in previous discussion so resctrl would need to know its properties.
See https://lore.kernel.org/lkml/[email protected]/

Right. ("MB_HLIM" name may be adjusted accordingly when "MB_MAX" is available.)

Quoted 40 lines
Quoted 8 lines

For exmple:
MB_HLIM: 0=0;1=0;2=1;10=0;18=0;26=0
MB:0=100;1=100;2=80;10=100;18=100;26=100

On GPU memory numa node 2: cannot use more than 80% of total max mbw even if there is still idle mem bandwidth on this node).

MBW allocations on all other domains are soft limited, meaning MBW can be used more than specified if mem is idle.

ack.

Quoted 19 lines
Quoted 13 lines

          L3:0=fff;1=fff

# echo 'MB_MIN:0=50' > schemata
# cat schemata
           MB_MAX:0=100;1=100
           MB_MIN:0=50;1=100
           MB:0=100;1=100
           L3:0=fff;1=fff

Writing to the dummy control will call a dummy callback that just prints to the kernel log:
"resctrl: Updata temporary MIN control on domain 0 with user value 50"

Quoted 5 lines

Example output of info/MB/:
/sys/fs/resctrl/info/MB/thread_throttle_mode:max
/sys/fs/resctrl/info/MB/num_closids:15
/sys/fs/resctrl/info/MB/delay_linear:1
/sys/fs/resctrl/info/MB/min_bandwidth:10

Quoted 3 lines

Add two new MB info RO files:
1. /sys/fs/resctrl/info/MB/domain_id
It shows "numa" for using numa id in "MB:" or "cache" for using legacy cache id.

This proposal introduces a *global* property to the MB *resource*? It does not seem as though this takes into account *anything* about how resctrl can support new hardware that has been discussed before, during, or after LPC. You have not participated in these discussions and now make an orthogonal proposal that does not take into account *any* of the requirements that we have been struggling with for months.

Quoted 7 lines

Why should this proposal be taken seriously? In your absence folks have been trying to accommodate how these upcoming products and be supported and the "scope" file associated with a control is intended to communicate to user space how the domain ID should be interpreted.

Why are you proposing something entirely different here without even acknowledging current approach and explaining why it does not work for you?

So can I change this part to adding the following files in info dirctory?

Quoted 10 lines

1. For numa memory bw allocation (MBN):
/sys/fs/resctrl/info/MBN/resource_schemata/MBN/ /sys/fs/resctrl/info/MBN/resource_schemata/MBN/resolution:100 /sys/fs/resctrl/info/MBN/resource_schemata/MBN/tolerance:5 /sys/fs/resctrl/info/MBN/resource_schemata/MBN/type:scalar /sys/fs/resctrl/info/MBN/resource_schemata/MBN/min:10 /sys/fs/resctrl/info/MBN/resource_schemata/MBN/scale:1 /sys/fs/resctrl/info/MBN/resource_schemata/MBN/scope:NUMA /sys/fs/resctrl/info/MBN/resource_schemata/MBN/unit:all /sys/fs/resctrl/info/MBN/resource_schemata/MBN/max:100

This is not just about adding files to the info directory. The files, directories, their relationships, and content have meaning. All I see from these proposals is an attempt to slap some new files into resctrl without any consideration to present consistent interface to users and without consideration of other architectures that need to be supported by resctrl.

resctrl needs to provide a generic and consistent interface to user space irrespective of the underlying architecture. Architectures cannot just slap some new files for their convenience.

Quoted 11 lines
Quoted 10 lines
Quoted 3 lines

2. /sys/fs/resctrl/info/MB/max_lim
It shows number 0-3 for MPAM MBW max limit behaviors: 0 for supporting both softlimit and hardlimit, etc.

Again this adds another *global* property to the MB resource but then above you describe the new "MB_HLIM" schemata file entry that implies that it is a new control for the MB resource. Having it be a new control for the MB resource matches earlier discussions. To support this I thus expect it to be exposed as a new control with potentially a new type if any of the existing planned types do not suffice.

Quoted 4 lines

How about adding these MB_HLIM dir and files in info?

/sys/fs/resctrl/info/MB_HLIM/resource_schemata/MB_HLIM/type: boolean /sys/fs/resctrl/info/MB_HLIM/resource_schemata/MB_HLIM/max_lim: 0

This presents "MB_HLIM" as a *resource* to user space. It is not a resource but a *control* of a resource, no? I thus expect it to instead look something like below that makes it clear that MB_HARDMAX is a control of the MB resource.

Quoted 5 lines

info

└── MB
     └── resource_schemata
         ├── MB
         └── MB_HARDMAX

Yes, this makes sense. I have changed to this hierarchy.

Quoted 130 lines
Quoted 123 lines
Quoted 98 lines
Quoted 6 lines
Quoted 4 lines

/sys/fs/resctrl/info/MB/resource_schemata/MB/resolution:100

Is it more concise to s/resource_schemata/schemata/? "resource_" seems redundant in the context "info/MB".

We could do this, yes.

Quoted 26 lines
Quoted 17 lines

/sys/fs/resctrl/info/MB/resource_schemata/MB/tolerance:5 /sys/fs/resctrl/info/MB/resource_schemata/MB/type:scalar /sys/fs/resctrl/info/MB/resource_schemata/MB/min:10 /sys/fs/resctrl/info/MB/resource_schemata/MB/scale:1 /sys/fs/resctrl/info/MB/resource_schemata/MB/scope:L3 /sys/fs/resctrl/info/MB/resource_schemata/MB/unit:all /sys/fs/resctrl/info/MB/resource_schemata/MB/max:100 /sys/fs/resctrl/info/MB/resource_schemata/MB_MIN/resolution:100 /sys/fs/resctrl/info/MB/resource_schemata/MB_MIN/tolerance:5 /sys/fs/resctrl/info/MB/resource_schemata/MB_MIN/type:scalar /sys/fs/resctrl/info/MB/resource_schemata/MB_MIN/min:10 /sys/fs/resctrl/info/MB/resource_schemata/MB_MIN/scale:1 /sys/fs/resctrl/info/MB/resource_schemata/MB_MIN/scope:L3 /sys/fs/resctrl/info/MB/resource_schemata/MB_MIN/unit:all /sys/fs/resctrl/info/MB/resource_schemata/MB_MIN/max:100 /sys/fs/resctrl/info/MB/resource_schemata/MB_MAX/resolution:100

Quoted 8 lines

/sys/fs/resctrl/info/MB/resource_schemata/MB_MAX/tolerance:5 /sys/fs/resctrl/info/MB/resource_schemata/MB_MAX/type:scalar /sys/fs/resctrl/info/MB/resource_schemata/MB_MAX/min:10 /sys/fs/resctrl/info/MB/resource_schemata/MB_MAX/scale:1 /sys/fs/resctrl/info/MB/resource_schemata/MB_MAX/scope:L3 /sys/fs/resctrl/info/MB/resource_schemata/MB_MAX/unit:all /sys/fs/resctrl/info/MB/resource_schemata/MB_MAX/max:100 /sys/fs/resctrl/info/MB/bandwidth_gran:10

Quoted 1 line

For MBW monitoring, extend mon_data/ directory to monitor CPU-less memory node. For example,

Here is where I attempted to discuss with you how to support monitoring on these systems:
https://lore.kernel.org/lkml/[email protected]/

Here again you respond with something completely different without acknowledging the previous discussion or noting why that does not work for you.

Quoted 7 lines

On legacy platforms (i.e. L3 and memory are described in same MPAM ACPI MSC wich doesn't support CPU-less nodes):
mon_data/mbm_L3_01/llc_occupancy
mon_data/mbm_L3_01/mbm_total_bytes
mon_data/mbm_L3_02/llc_occupancy
mon_data/mbm_L3_02/mbm_total_bytes

Quoted 6 lines

On platforms with L3 and memory in separate MPAM ACPI MSCs but there is no CPU-less node:
mon_data/mbm_L3_01/llc_occupancy    <- cache id 1 mon_data/mbm_L3_02/llc_occupancy    <- cache id 2 mon_data/mbm_MB_00/mbm_total_bytes  <- numa node 0 (socket 0) mon_data/mbm_MB_01/mbm_total_bytes  <- numa node 1 (socket 1)

Here too I do not find it appropriate for resctrl to retroactively change its interface. "MB" is a resource and the above switches the resctrl interface to imply the monitoring data of a resource can be found in the mon directory that matches the resource name. This is not what resctrl does today. Doing something like above will result in resctrl having a confusing interface where "sometimes" memory bandwidth data can be found in the L3 directory and "sometimes" memory bandwidth data can be found in the MB directory.

As I described to you in December resctrl already exposes the monitoring data based on the *scope*. As you also point out above, today the memory bandwidth monitoring data at L3 scope can be found in the L3 directory. "L3" should thus not be interpreted as the resource L3 but the scope L3 since it contains MBM data today. When viewing it as such resctrl could internally be more explicit and separate monitoring scope from monitoring resource and present the monitoring data based on scope to remain intuitively backward compatible while obtaining support for these memory nodes.

Quoted 11 lines

On platforms with L3 and memory in separate MPAM ACPI MSCs and there are CPU-less nodes:
mon_data/mbm_L3_01/llc_occupancy    <- cache id 1 mon_data/mbm_L3_02/llc_occupancy    <- cache id 2 mon_data/mbm_MB_00/mbm_total_bytes  <- numa node 0 (socket 0) mon_data/mbm_MB_01/mbm_total_bytes  <- numa node 1 (socket 1) mon_data/mbm_MB_02/mbm_total_bytes  <- numa node 2 (GPU 0 mem) mon_data/mbm_MB_10/mbm_total_bytes  <- numa node 10 (GPU 1 mem) mon_data/mbm_MB_18/mbm_total_bytes  <- numa node 18 (GPU 2 mem) mon_data/mbm_MB_26/mbm_total_bytes  <- numa node 26 (GPU 3 mem)

To be backward compatible I find it more intuitive if instead this data is exposed as below:

mon_data/mbm_L3_01/llc_occupancy    <- cache id 1 mon_data/mbm_L3_02/llc_occupancy    <- cache id 2 mon_data/mbm_NODE_00/mbm_total_bytes  <- numa node 0 (socket 0) mon_data/mbm_NODE_01/mbm_total_bytes  <- numa node 1 (socket 1) mon_data/mbm_NODE_02/mbm_total_bytes  <- numa node 2 (GPU 0 mem) mon_data/mbm_NODE_10/mbm_total_bytes  <- numa node 10 (GPU 1 mem) mon_data/mbm_NODE_18/mbm_total_bytes  <- numa node 18 (GPU 2 mem) mon_data/mbm_NODE_26/mbm_total_bytes  <- numa node 26 (GPU 3 mem)

Quoted 3 lines

When "mbm_total_bytes" move from mon_data/mbm_L3_x to mon_data/mbm_NODE_x it clearly indicates that it is memory bandwidth monitoring data moving from "L3" scope to "NODE" scope.

The scope in mon_data shown in https://lore.kernel.org/lkml/[email protected]/:

mon_data

├── mon_L3_00       <== monitoring data at scope L3
│   ├── llc_occupancy
│   ├── mbm_local_bytes
│   └── mbm_total_bytes
├── mon_L3_01       <== monitoring data at scope L3
│   ├── llc_occupancy
│   ├── mbm_local_bytes
│   └── mbm_total_bytes
├── mon_NODE_00     <== monitoring data at scope NODE
│   └── mbm_total_bytes
└── mon_NODE_01     <== monitoring data at scope NODE
     └── mbm_total_bytes

On some ARM platforms, for example, socket 0 (CPUs+L3) and socket 1 (CPUs+L3):
llc_occupancy is monitored through processor/L3. total_bytes are monitored through memory controlor.

Quoted 4 lines

Then the above "scope" is confused:
mon_L3_00 and mon_L3_01 only has llc_occupancy. The scope "L3" is fine here. But mon_NODE_00 and 01 are confused because numa node 00 and 01 have both L3 and memory. The name "mon_NODE_00" seems monitor both

While the node may have L3 and memory the directory only represents what is monitored at the particular scope.

Quoted 8 lines
Quoted 4 lines

llc_occupany and total_bytes but it only monitor total_bytes. And on CPU-less platforms, a CPU-less node does only have total_bytes which seems match "scope NODE".
With "scope NODE", can user tell if it has llc_occupancy or total_bytes or both?

resctrl has a "mon_features" file associated with the monitoring scope that informs user space which events can be expected in a resource group's monitoring data directories.

Right.

Quoted 5 lines
Quoted 5 lines

If change "mon_MB_01" to "mon_MM_01", the scope is "MM" now which means monitoring memory (total_bytes) in numa node 1 with "scope MM" (not L3)?

Quoted 7 lines

This sounds redundant to me since the event names already have the resource embedded. "llc_occupancy" implies L3 occupancy
"mbm_local_bytes" ... the "mbm" implies this is memory bandwidth monitoring data.

User space can infer the scope at which the monitoring data is collected from the directory the event file is in.

Right.

Thanks.

-Fenghua

author maintainer
#66 lore raw

Hi Fenghua,

Quoted 105 lines · Fenghua Yu
Quoted 10 lines · Reinette Chatre
Quoted 9 lines · Fenghua Yu
Quoted 8 lines · Reinette Chatre
Quoted 6 lines · Fenghua Yu

On 5/29/26 11:06, Reinette Chatre wrote:

As Shaopen and Ben mentioned earlier, we are working on two MPAM features that may need to change schemata interface. The CPU-less feature was discussed on LPC (although the interfaces will be slightly different from the LPC).

Quoted 6 lines
Quoted 6 lines
Quoted 2 lines

I know. Here is where I tried to engage with you on needed interfaces after LPC:
https://lore.kernel.org/lkml/[email protected]/

MPAM ACPI defines MSC (Memory System Control) is defined in one of two ways (not both) on one platform:
1. L3 and memory together on each processor MSC
2. L3 in processor MSC and memory control/monitoring in different memory MSCs.

Ben said there is type 3 platform:
3. L3 cache and memory bandwidth in processor MSCs and memory bandwidth in different memory MSCs.

Quoted 10 lines
Quoted 10 lines

On type 1 platform, schemata is legacy:
MB:1=100;2=100  <-- cache id 1 and 2 as domain id

On type 2 platform, I will not reuse "MB:" name. Instead, define new resource name "MBN:" for numa node and schemata is:
MBN:0=100;1=100;2=100;10=100;18=100;26=100 <-- numa id 0, 1, 2, 10, 18,
                            26 as domain id
On type 2 platform, there won't be "MB:" line. Numa 0 and 1
are for mbm allocation on socket 0 and 1. 2,10, 18 and 26 are for GPU
memory nodes allocation.

On type 3 platform, there could be "MB:" line for L3 cache and "MB_NODE:" for numa node. Example schemata is:

     MB:1=100;2=100                 <-- cache id 1 and 2 as domain id MB_NODE:0=100;1=100;2=100;10=100;18=100;26=100 <-- numa id 0, 1, 2, 10, 18, 26 as domain id

Quoted 50 lines

(to help make things explicit I will refer to what you call "MBN" as "MB_NODE" to make it explicit that it is memory bandwidth allocation at node scope)

I am trying to consider how this can be accomplished while also considering all the other new hardware features that resctrl need to support. Consider, for example, AMD's "Global MBA" (https://lore.kernel.org/lkml/[email protected]/) that throttles memory bandwidth at L3 scope but the user configures allocations at NODE scope. At this time the plan is to support this with a second control associated with the MB resource that can allocate memory bandwidth at node scope. See
https://lore.kernel.org/lkml/[email protected]/

If resctrl creates a new resource for node scoped memory bandwidth allocations to support these "type 2" systems then that will result in an inconsistent interface between architectures that we should avoid.

Have you been listening in on the discussions surrounding emulated controls? Considering that, would it be possible to support the "MB" control on a type "2" system but have it be backed by (emulated by) the underlying "MB_NODE" control?

resctrl could expose both controls on these "type 2" systems but make it clear that "MB" is emulated by "MB_NODE". For example:

info/

└── MB/
     └── resource_schemata/
         └── MB/
             └── MB_NODE/

User will see both controls in schemata file but when changes are made to "MB" control it will show in the "MB_NODE" control and vice-versa. User could also disable the "MB" control that will establish familiarity with the interface at which point resctrl can drop the "MB" control from the schemata file on these "type 2" systems.

Having the MB resource available with an MB control will keep resctrl backward compatible if there are any tools that expect that. If backward compatibility is not of concern then resctrl could initialize with the emulated control disabled by default. See discussion at
https://lore.kernel.org/lkml/[email protected]/
that describes a new resctrl capability in support of RISC-V and RDT.
With this resctrl could initialize with:

info/

└── MB/
     └── resource_schemata/
         ├── MB/
         │   ├── MB_NODE/
         │   │   └── status:enabled
         │   └── status:disabled
         └── mode:legacy [native]
Quoted 3 lines

With above a "type 2" system will boot with its schemata file just containing the "MB_NODE" control while info/MB describes the memory bandwidth resource.

On type 3 machine, schemata has both MB in legacy mode with cache id as domain id and MB_NODE with numa id as domain id.

Is this directory OK?

 info/

 └── MB/
      └── resource_schemata/
          ├── MB/
          │   ├── MB_NODE/
          │   │   └── status:disabled
          │   └── status:enabled
          ├── MB_NODE/
          └── mode:node
Quoted 1 line

1. MB and MB_NODE are shown in parallel in inf/MB/resource_schemata/

I do not think there is a need to expose an emulated MB_NODE control if the actual MB_NODE hardware control exists.

Quoted 1 line

2. mode is set as "node" meaning "MB" is for L3 and "MB_NODE" is for numa node

I assume you mean "scope" instead of "mode"? (more below)

Quoted 1 line

3. Emulation "MB_NODE" is disabled (or should the "MB_NODE" sub-dir be invisible?)

Right, I do not think emulation is needed here. No need to make it invisible since it should not exist.

From what I understand these "type 3" machines could be simplified to:

info/

└── MB/
    └── resource_schemata/
        ├── MB/
        │   └── scope:L3
        └── MB_NODE/
            └── scope:NODE

Beyond this I believe that MPAM currently emulates the MB control with its "MB_MAX" control and users may want to make bandwidth allocations at the fine granularity that it supports. Taking this into account the interface may end up looking like:

info/

└── MB/
    └── resource_schemata/
        ├── MB/
        │   ├── MB_MAX/
        │   │   └── scope:L3
        │   └── scope:L3
        └── MB_NODE/
            └── scope:NODE

A system like above will thus have three schemata file entries:
MB
MB_MAX
MB_NODE

Three schemata file entries would be unnecessary for users familiar with the finer granularity MB_MAX control so that is where the "mode" file can be used to disable the legacy MB control to just expose MB_MAX and MB_NODE on these systems.

Would that work for these systems?

...

Quoted 104 lines
Quoted 104 lines
Quoted 65 lines
Quoted 7 lines
Quoted 6 lines

There is another MPAM feature called MBW Max hardlimit which sets "MB:" allocation as hardlimit (i.e. MBW throttling percentage must be satisfied) per domain. Adding a new "MB_HLIM:" line in schemata. It's 1:1 mapped to "MB:" to control hardlimit of MB throttling percentage on each domain. By default hardlimit is off (0) and can be turned on to set MBW Max hardlimit on a domain.

Quoted 5 lines

ack. This sounds like a new control associated with the MB resource. This is a boolean control as Dave highlighted in previous discussion so resctrl would need to know its properties.
See https://lore.kernel.org/lkml/[email protected]/

Right. ("MB_HLIM" name may be adjusted accordingly when "MB_MAX" is available.)

Quoted 40 lines
Quoted 8 lines

For exmple:
MB_HLIM: 0=0;1=0;2=1;10=0;18=0;26=0
MB:0=100;1=100;2=80;10=100;18=100;26=100

On GPU memory numa node 2: cannot use more than 80% of total max mbw even if there is still idle mem bandwidth on this node).

MBW allocations on all other domains are soft limited, meaning MBW can be used more than specified if mem is idle.

ack.

Quoted 19 lines
Quoted 13 lines

           L3:0=fff;1=fff

# echo 'MB_MIN:0=50' > schemata
# cat schemata
           MB_MAX:0=100;1=100
           MB_MIN:0=50;1=100
           MB:0=100;1=100
           L3:0=fff;1=fff

Writing to the dummy control will call a dummy callback that just prints to the kernel log:
"resctrl: Updata temporary MIN control on domain 0 with user value 50"

Quoted 5 lines

Example output of info/MB/:
/sys/fs/resctrl/info/MB/thread_throttle_mode:max
/sys/fs/resctrl/info/MB/num_closids:15
/sys/fs/resctrl/info/MB/delay_linear:1
/sys/fs/resctrl/info/MB/min_bandwidth:10

Quoted 3 lines

Add two new MB info RO files:
1. /sys/fs/resctrl/info/MB/domain_id
It shows "numa" for using numa id in "MB:" or "cache" for using legacy cache id.

This proposal introduces a *global* property to the MB *resource*? It does not seem as though this takes into account *anything* about how resctrl can support new hardware that has been discussed before, during, or after LPC. You have not participated in these discussions and now make an orthogonal proposal that does not take into account *any* of the requirements that we have been struggling with for months.

Quoted 7 lines

Why should this proposal be taken seriously? In your absence folks have been trying to accommodate how these upcoming products and be supported and the "scope" file associated with a control is intended to communicate to user space how the domain ID should be interpreted.

Why are you proposing something entirely different here without even acknowledging current approach and explaining why it does not work for you?

So can I change this part to adding the following files in info dirctory?

Quoted 10 lines

1. For numa memory bw allocation (MBN):
/sys/fs/resctrl/info/MBN/resource_schemata/MBN/ /sys/fs/resctrl/info/MBN/resource_schemata/MBN/resolution:100 /sys/fs/resctrl/info/MBN/resource_schemata/MBN/tolerance:5 /sys/fs/resctrl/info/MBN/resource_schemata/MBN/type:scalar /sys/fs/resctrl/info/MBN/resource_schemata/MBN/min:10 /sys/fs/resctrl/info/MBN/resource_schemata/MBN/scale:1 /sys/fs/resctrl/info/MBN/resource_schemata/MBN/scope:NUMA /sys/fs/resctrl/info/MBN/resource_schemata/MBN/unit:all /sys/fs/resctrl/info/MBN/resource_schemata/MBN/max:100

This is not just about adding files to the info directory. The files, directories, their relationships, and content have meaning. All I see from these proposals is an attempt to slap some new files into resctrl without any consideration to present consistent interface to users and without consideration of other architectures that need to be supported by resctrl.

resctrl needs to provide a generic and consistent interface to user space irrespective of the underlying architecture. Architectures cannot just slap some new files for their convenience.

Quoted 11 lines
Quoted 10 lines
Quoted 3 lines

2. /sys/fs/resctrl/info/MB/max_lim
It shows number 0-3 for MPAM MBW max limit behaviors: 0 for supporting both softlimit and hardlimit, etc.

Again this adds another *global* property to the MB resource but then above you describe the new "MB_HLIM" schemata file entry that implies that it is a new control for the MB resource. Having it be a new control for the MB resource matches earlier discussions. To support this I thus expect it to be exposed as a new control with potentially a new type if any of the existing planned types do not suffice.

Quoted 4 lines

How about adding these MB_HLIM dir and files in info?

/sys/fs/resctrl/info/MB_HLIM/resource_schemata/MB_HLIM/type: boolean /sys/fs/resctrl/info/MB_HLIM/resource_schemata/MB_HLIM/max_lim: 0

This presents "MB_HLIM" as a *resource* to user space. It is not a resource but a *control* of a resource, no? I thus expect it to instead look something like below that makes it clear that MB_HARDMAX is a control of the MB resource.

Quoted 7 lines
Quoted 5 lines

info

└── MB
     └── resource_schemata
         ├── MB
         └── MB_HARDMAX

Yes, this makes sense. I have changed to this hierarchy.

Thank you very much for considering this approach.

Reinette

maintainer
#67 lore raw

Hi Reinette,

Quoted 145 lines · Reinette Chatre

Hi Fenghua,

Quoted 105 lines · Fenghua Yu
Quoted 10 lines · Reinette Chatre
Quoted 9 lines · Fenghua Yu
Quoted 8 lines · Reinette Chatre
Quoted 6 lines · Fenghua Yu

On 5/29/26 11:06, Reinette Chatre wrote:

As Shaopen and Ben mentioned earlier, we are working on two MPAM features that may need to change schemata interface. The CPU-less feature was discussed on LPC (although the interfaces will be slightly different from the LPC).

Quoted 6 lines
Quoted 6 lines
Quoted 2 lines

I know. Here is where I tried to engage with you on needed interfaces after LPC:
https://lore.kernel.org/lkml/[email protected]/

MPAM ACPI defines MSC (Memory System Control) is defined in one of two ways (not both) on one platform:
1. L3 and memory together on each processor MSC
2. L3 in processor MSC and memory control/monitoring in different memory MSCs.

Ben said there is type 3 platform:
3. L3 cache and memory bandwidth in processor MSCs and memory bandwidth in different memory MSCs.

Quoted 10 lines
Quoted 10 lines

On type 1 platform, schemata is legacy:
MB:1=100;2=100  <-- cache id 1 and 2 as domain id

On type 2 platform, I will not reuse "MB:" name. Instead, define new resource name "MBN:" for numa node and schemata is:
MBN:0=100;1=100;2=100;10=100;18=100;26=100 <-- numa id 0, 1, 2, 10, 18,
                            26 as domain id
On type 2 platform, there won't be "MB:" line. Numa 0 and 1
are for mbm allocation on socket 0 and 1. 2,10, 18 and 26 are for GPU
memory nodes allocation.

On type 3 platform, there could be "MB:" line for L3 cache and "MB_NODE:" for numa node. Example schemata is:

     MB:1=100;2=100                 <-- cache id 1 and 2 as domain id MB_NODE:0=100;1=100;2=100;10=100;18=100;26=100 <-- numa id 0, 1, 2, 10, 18, 26 as domain id

Quoted 50 lines

(to help make things explicit I will refer to what you call "MBN" as "MB_NODE" to make it explicit that it is memory bandwidth allocation at node scope)

I am trying to consider how this can be accomplished while also considering all the other new hardware features that resctrl need to support. Consider, for example, AMD's "Global MBA" (https://lore.kernel.org/lkml/[email protected]/) that throttles memory bandwidth at L3 scope but the user configures allocations at NODE scope. At this time the plan is to support this with a second control associated with the MB resource that can allocate memory bandwidth at node scope. See
https://lore.kernel.org/lkml/[email protected]/

If resctrl creates a new resource for node scoped memory bandwidth allocations to support these "type 2" systems then that will result in an inconsistent interface between architectures that we should avoid.

Have you been listening in on the discussions surrounding emulated controls? Considering that, would it be possible to support the "MB" control on a type "2" system but have it be backed by (emulated by) the underlying "MB_NODE" control?

resctrl could expose both controls on these "type 2" systems but make it clear that "MB" is emulated by "MB_NODE". For example:

info/

└── MB/
     └── resource_schemata/
         └── MB/
             └── MB_NODE/

User will see both controls in schemata file but when changes are made to "MB" control it will show in the "MB_NODE" control and vice-versa. User could also disable the "MB" control that will establish familiarity with the interface at which point resctrl can drop the "MB" control from the schemata file on these "type 2" systems.

Having the MB resource available with an MB control will keep resctrl backward compatible if there are any tools that expect that. If backward compatibility is not of concern then resctrl could initialize with the emulated control disabled by default. See discussion at
https://lore.kernel.org/lkml/[email protected]/
that describes a new resctrl capability in support of RISC-V and RDT.
With this resctrl could initialize with:

info/

└── MB/
     └── resource_schemata/
         ├── MB/
         │   ├── MB_NODE/
         │   │   └── status:enabled
         │   └── status:disabled
         └── mode:legacy [native]
Quoted 3 lines

With above a "type 2" system will boot with its schemata file just containing the "MB_NODE" control while info/MB describes the memory bandwidth resource.

On type 3 machine, schemata has both MB in legacy mode with cache id as domain id and MB_NODE with numa id as domain id.

Is this directory OK?

 info/

 └── MB/
      └── resource_schemata/
          ├── MB/
          │   ├── MB_NODE/
          │   │   └── status:disabled
          │   └── status:enabled
          ├── MB_NODE/
          └── mode:node
Quoted 1 line

1. MB and MB_NODE are shown in parallel in inf/MB/resource_schemata/

I do not think there is a need to expose an emulated MB_NODE control if the actual MB_NODE hardware control exists.

Quoted 1 line

2. mode is set as "node" meaning "MB" is for L3 and "MB_NODE" is for numa node

I assume you mean "scope" instead of "mode"? (more below)

Quoted 1 line

3. Emulation "MB_NODE" is disabled (or should the "MB_NODE" sub-dir be invisible?)

Right, I do not think emulation is needed here. No need to make it invisible since it should not exist.

From what I understand these "type 3" machines could be simplified to:

info/

└── MB/
    └── resource_schemata/
        ├── MB/
        │   └── scope:L3
        └── MB_NODE/
            └── scope:NODE

Beyond this I believe that MPAM currently emulates the MB control with its "MB_MAX" control and users may want to make bandwidth allocations at the fine granularity that it supports. Taking this into account the interface may end up looking like:

info/

└── MB/
    └── resource_schemata/
        ├── MB/
        │   ├── MB_MAX/
        │   │   └── scope:L3
        │   └── scope:L3
        └── MB_NODE/
            └── scope:NODE
Quoted 9 lines

A system like above will thus have three schemata file entries:
MB
MB_MAX
MB_NODE

Three schemata file entries would be unnecessary for users familiar with the finer granularity MB_MAX control so that is where the "mode" file can be used to disable the legacy MB control to just expose MB_MAX and MB_NODE on these systems.

Quoted 1 line

Would that work for these systems?

I'm not clear on the behaviour when the schemata file contains both MB and MB_MAX entries. I assume we should expect that the user might read back the whole file, modify what they expect to change and then write everything back. If legacy software does this and wants to change the MB schemata then the MB and MB_MAX values would be incompatible. Will we just ignore values that haven't changed and fail the write if two dependent schemata are changed at the same time?

Alternatively, could the "mode" file be used to switch between "MB" and "MB_MAX" and the two never need be shown at the same time. The user opts in to using the new interface, "MB_MAX" by setting "mode" and can just toggle back if they want to use "MB" directly again.

Thanks,

Ben

Quoted 120 lines

...

Quoted 104 lines
Quoted 104 lines
Quoted 65 lines
Quoted 7 lines
Quoted 6 lines

There is another MPAM feature called MBW Max hardlimit which sets "MB:" allocation as hardlimit (i.e. MBW throttling percentage must be satisfied) per domain. Adding a new "MB_HLIM:" line in schemata. It's 1:1 mapped to "MB:" to control hardlimit of MB throttling percentage on each domain. By default hardlimit is off (0) and can be turned on to set MBW Max hardlimit on a domain.

Quoted 5 lines

ack. This sounds like a new control associated with the MB resource. This is a boolean control as Dave highlighted in previous discussion so resctrl would need to know its properties.
See https://lore.kernel.org/lkml/[email protected]/

Right. ("MB_HLIM" name may be adjusted accordingly when "MB_MAX" is available.)

Quoted 40 lines
Quoted 8 lines

For exmple:
MB_HLIM: 0=0;1=0;2=1;10=0;18=0;26=0
MB:0=100;1=100;2=80;10=100;18=100;26=100

On GPU memory numa node 2: cannot use more than 80% of total max mbw even if there is still idle mem bandwidth on this node).

MBW allocations on all other domains are soft limited, meaning MBW can be used more than specified if mem is idle.

ack.

Quoted 19 lines
Quoted 13 lines

           L3:0=fff;1=fff

# echo 'MB_MIN:0=50' > schemata
# cat schemata
           MB_MAX:0=100;1=100
           MB_MIN:0=50;1=100
           MB:0=100;1=100
           L3:0=fff;1=fff

Writing to the dummy control will call a dummy callback that just prints to the kernel log:
"resctrl: Updata temporary MIN control on domain 0 with user value 50"

Quoted 5 lines

Example output of info/MB/:
/sys/fs/resctrl/info/MB/thread_throttle_mode:max
/sys/fs/resctrl/info/MB/num_closids:15
/sys/fs/resctrl/info/MB/delay_linear:1
/sys/fs/resctrl/info/MB/min_bandwidth:10

Quoted 3 lines

Add two new MB info RO files:
1. /sys/fs/resctrl/info/MB/domain_id
It shows "numa" for using numa id in "MB:" or "cache" for using legacy cache id.

This proposal introduces a *global* property to the MB *resource*? It does not seem as though this takes into account *anything* about how resctrl can support new hardware that has been discussed before, during, or after LPC. You have not participated in these discussions and now make an orthogonal proposal that does not take into account *any* of the requirements that we have been struggling with for months.

Quoted 7 lines

Why should this proposal be taken seriously? In your absence folks have been trying to accommodate how these upcoming products and be supported and the "scope" file associated with a control is intended to communicate to user space how the domain ID should be interpreted.

Why are you proposing something entirely different here without even acknowledging current approach and explaining why it does not work for you?

So can I change this part to adding the following files in info dirctory?

Quoted 10 lines

1. For numa memory bw allocation (MBN):
/sys/fs/resctrl/info/MBN/resource_schemata/MBN/ /sys/fs/resctrl/info/MBN/resource_schemata/MBN/resolution:100 /sys/fs/resctrl/info/MBN/resource_schemata/MBN/tolerance:5 /sys/fs/resctrl/info/MBN/resource_schemata/MBN/type:scalar /sys/fs/resctrl/info/MBN/resource_schemata/MBN/min:10 /sys/fs/resctrl/info/MBN/resource_schemata/MBN/scale:1 /sys/fs/resctrl/info/MBN/resource_schemata/MBN/scope:NUMA /sys/fs/resctrl/info/MBN/resource_schemata/MBN/unit:all /sys/fs/resctrl/info/MBN/resource_schemata/MBN/max:100

This is not just about adding files to the info directory. The files, directories, their relationships, and content have meaning. All I see from these proposals is an attempt to slap some new files into resctrl without any consideration to present consistent interface to users and without consideration of other architectures that need to be supported by resctrl.

resctrl needs to provide a generic and consistent interface to user space irrespective of the underlying architecture. Architectures cannot just slap some new files for their convenience.

Quoted 11 lines
Quoted 10 lines
Quoted 3 lines

2. /sys/fs/resctrl/info/MB/max_lim
It shows number 0-3 for MPAM MBW max limit behaviors: 0 for supporting both softlimit and hardlimit, etc.

Again this adds another *global* property to the MB resource but then above you describe the new "MB_HLIM" schemata file entry that implies that it is a new control for the MB resource. Having it be a new control for the MB resource matches earlier discussions. To support this I thus expect it to be exposed as a new control with potentially a new type if any of the existing planned types do not suffice.

Quoted 4 lines

How about adding these MB_HLIM dir and files in info?

/sys/fs/resctrl/info/MB_HLIM/resource_schemata/MB_HLIM/type: boolean /sys/fs/resctrl/info/MB_HLIM/resource_schemata/MB_HLIM/max_lim: 0

This presents "MB_HLIM" as a *resource* to user space. It is not a resource but a *control* of a resource, no? I thus expect it to instead look something like below that makes it clear that MB_HARDMAX is a control of the MB resource.

Quoted 7 lines
Quoted 5 lines

info

└── MB
     └── resource_schemata
         ├── MB
         └── MB_HARDMAX

Yes, this makes sense. I have changed to this hierarchy.

Thank you very much for considering this approach.

Reinette

author maintainer
#68 lore raw

Hi Ben,

Quoted 152 lines · Ben Horgan
Quoted 151 lines · Reinette Chatre
Quoted 105 lines · Fenghua Yu
Quoted 10 lines · Reinette Chatre
Quoted 9 lines · Fenghua Yu
Quoted 8 lines · Reinette Chatre
Quoted 6 lines · Fenghua Yu

On 5/29/26 11:06, Reinette Chatre wrote:

As Shaopen and Ben mentioned earlier, we are working on two MPAM features that may need to change schemata interface. The CPU-less feature was discussed on LPC (although the interfaces will be slightly different from the LPC).

Quoted 6 lines
Quoted 6 lines
Quoted 2 lines

I know. Here is where I tried to engage with you on needed interfaces after LPC:
https://lore.kernel.org/lkml/[email protected]/

MPAM ACPI defines MSC (Memory System Control) is defined in one of two ways (not both) on one platform:
1. L3 and memory together on each processor MSC
2. L3 in processor MSC and memory control/monitoring in different memory MSCs.

Ben said there is type 3 platform:
3. L3 cache and memory bandwidth in processor MSCs and memory bandwidth in different memory MSCs.

Quoted 10 lines
Quoted 10 lines

On type 1 platform, schemata is legacy:
MB:1=100;2=100  <-- cache id 1 and 2 as domain id

On type 2 platform, I will not reuse "MB:" name. Instead, define new resource name "MBN:" for numa node and schemata is:
MBN:0=100;1=100;2=100;10=100;18=100;26=100 <-- numa id 0, 1, 2, 10, 18,
                            26 as domain id
On type 2 platform, there won't be "MB:" line. Numa 0 and 1
are for mbm allocation on socket 0 and 1. 2,10, 18 and 26 are for GPU
memory nodes allocation.

On type 3 platform, there could be "MB:" line for L3 cache and "MB_NODE:" for numa node. Example schemata is:

     MB:1=100;2=100                 <-- cache id 1 and 2 as domain id MB_NODE:0=100;1=100;2=100;10=100;18=100;26=100 <-- numa id 0, 1, 2, 10, 18, 26 as domain id

Quoted 50 lines

(to help make things explicit I will refer to what you call "MBN" as "MB_NODE" to make it explicit that it is memory bandwidth allocation at node scope)

I am trying to consider how this can be accomplished while also considering all the other new hardware features that resctrl need to support. Consider, for example, AMD's "Global MBA" (https://lore.kernel.org/lkml/[email protected]/) that throttles memory bandwidth at L3 scope but the user configures allocations at NODE scope. At this time the plan is to support this with a second control associated with the MB resource that can allocate memory bandwidth at node scope. See
https://lore.kernel.org/lkml/[email protected]/

If resctrl creates a new resource for node scoped memory bandwidth allocations to support these "type 2" systems then that will result in an inconsistent interface between architectures that we should avoid.

Have you been listening in on the discussions surrounding emulated controls? Considering that, would it be possible to support the "MB" control on a type "2" system but have it be backed by (emulated by) the underlying "MB_NODE" control?

resctrl could expose both controls on these "type 2" systems but make it clear that "MB" is emulated by "MB_NODE". For example:

info/

└── MB/
     └── resource_schemata/
         └── MB/
             └── MB_NODE/

User will see both controls in schemata file but when changes are made to "MB" control it will show in the "MB_NODE" control and vice-versa. User could also disable the "MB" control that will establish familiarity with the interface at which point resctrl can drop the "MB" control from the schemata file on these "type 2" systems.

Having the MB resource available with an MB control will keep resctrl backward compatible if there are any tools that expect that. If backward compatibility is not of concern then resctrl could initialize with the emulated control disabled by default. See discussion at
https://lore.kernel.org/lkml/[email protected]/
that describes a new resctrl capability in support of RISC-V and RDT.
With this resctrl could initialize with:

info/

└── MB/
     └── resource_schemata/
         ├── MB/
         │   ├── MB_NODE/
         │   │   └── status:enabled
         │   └── status:disabled
         └── mode:legacy [native]
Quoted 3 lines

With above a "type 2" system will boot with its schemata file just containing the "MB_NODE" control while info/MB describes the memory bandwidth resource.

On type 3 machine, schemata has both MB in legacy mode with cache id as domain id and MB_NODE with numa id as domain id.

Is this directory OK?

 info/

 └── MB/
      └── resource_schemata/
          ├── MB/
          │   ├── MB_NODE/
          │   │   └── status:disabled
          │   └── status:enabled
          ├── MB_NODE/
          └── mode:node
Quoted 1 line

1. MB and MB_NODE are shown in parallel in inf/MB/resource_schemata/

I do not think there is a need to expose an emulated MB_NODE control if the actual MB_NODE hardware control exists.

Quoted 1 line

2. mode is set as "node" meaning "MB" is for L3 and "MB_NODE" is for numa node

I assume you mean "scope" instead of "mode"? (more below)

Quoted 1 line

3. Emulation "MB_NODE" is disabled (or should the "MB_NODE" sub-dir be invisible?)

Right, I do not think emulation is needed here. No need to make it invisible since it should not exist.

From what I understand these "type 3" machines could be simplified to:

info/

└── MB/
    └── resource_schemata/
        ├── MB/
        │   └── scope:L3
        └── MB_NODE/
            └── scope:NODE

Beyond this I believe that MPAM currently emulates the MB control with its "MB_MAX" control and users may want to make bandwidth allocations at the fine granularity that it supports. Taking this into account the interface may end up looking like:

info/

└── MB/
    └── resource_schemata/
        ├── MB/
        │   ├── MB_MAX/
        │   │   └── scope:L3
        │   └── scope:L3
        └── MB_NODE/
            └── scope:NODE

A system like above will thus have three schemata file entries:
MB
MB_MAX
MB_NODE

Three schemata file entries would be unnecessary for users familiar with the finer granularity MB_MAX control so that is where the "mode" file can be used to disable the legacy MB control to just expose MB_MAX and MB_NODE on these systems.

Quoted 7 lines
Quoted 1 line
Quoted 1 line

Would that work for these systems?

I'm not clear on the behaviour when the schemata file contains both MB and MB_MAX entries. I assume we should expect that the user might read back the whole file, modify what they expect to change and then write everything back. If legacy software does this and wants to change the MB schemata then the MB and MB_MAX values would be incompatible. Will we just ignore values that haven't changed and fail the write if two dependent schemata are changed at the same time?

An alternative would be to follow Dave Martin's suggestion to prefix emulated controls in schemata file with "#":
https://lore.kernel.org/lkml/[email protected]/
This was one element of original discussion that fizzled a bit since it was not obvious that user space would behave like this. Consider, for example, from Documentation/filesystems/resctrl.rst:

Reading/writing the schemata file
---------------------------------
... When writing you only need to specify those values which you wish to change. ..

Quoted 3 lines

Alternatively, could the "mode" file be used to switch between "MB" and "MB_MAX" and the two never need be shown at the same time. The user opts in to using the new interface, "MB_MAX" by setting "mode" and can just toggle back if they want to use "MB" directly again.

Interesting. So far the "mode" options have been "legacy" and "native" where "legacy" would show the legacy as well as emulated controls in the schemata file and "native" will only show the new controls. resctrl could make the default of "legacy" mean that *only* the legacy control is shown without insight into the controls it is being emulated with. The only insight to this would continue to be via the hierarchy in the info directory, which would show relationship but not the actual control values. Do you think there could be a need for users (excluding validation?) that may want to see the underlying control values used to emulate a legacy control?

I assume for backward compatibility that "legacy" would remain the default so a possible inconvenience here would be that users familiar with the new controls would forever need to switch the mode before being able to use them.

Having this clear distinction could simplify the implementation.

Reinette

maintainer
#69 lore raw

Hi Reinette,

Quoted 163 lines · Reinette Chatre

Hi Ben,

Quoted 152 lines · Ben Horgan
Quoted 151 lines · Reinette Chatre
Quoted 105 lines · Fenghua Yu
Quoted 10 lines · Reinette Chatre
Quoted 9 lines · Fenghua Yu
Quoted 8 lines · Reinette Chatre
Quoted 6 lines · Fenghua Yu

On 5/29/26 11:06, Reinette Chatre wrote:

As Shaopen and Ben mentioned earlier, we are working on two MPAM features that may need to change schemata interface. The CPU-less feature was discussed on LPC (although the interfaces will be slightly different from the LPC).

Quoted 6 lines
Quoted 6 lines
Quoted 2 lines

I know. Here is where I tried to engage with you on needed interfaces after LPC:
https://lore.kernel.org/lkml/[email protected]/

MPAM ACPI defines MSC (Memory System Control) is defined in one of two ways (not both) on one platform:
1. L3 and memory together on each processor MSC
2. L3 in processor MSC and memory control/monitoring in different memory MSCs.

Ben said there is type 3 platform:
3. L3 cache and memory bandwidth in processor MSCs and memory bandwidth in different memory MSCs.

Quoted 10 lines
Quoted 10 lines

On type 1 platform, schemata is legacy:
MB:1=100;2=100  <-- cache id 1 and 2 as domain id

On type 2 platform, I will not reuse "MB:" name. Instead, define new resource name "MBN:" for numa node and schemata is:
MBN:0=100;1=100;2=100;10=100;18=100;26=100 <-- numa id 0, 1, 2, 10, 18,
                            26 as domain id
On type 2 platform, there won't be "MB:" line. Numa 0 and 1
are for mbm allocation on socket 0 and 1. 2,10, 18 and 26 are for GPU
memory nodes allocation.

On type 3 platform, there could be "MB:" line for L3 cache and "MB_NODE:" for numa node. Example schemata is:

     MB:1=100;2=100                 <-- cache id 1 and 2 as domain id MB_NODE:0=100;1=100;2=100;10=100;18=100;26=100 <-- numa id 0, 1, 2, 10, 18, 26 as domain id

Quoted 50 lines

(to help make things explicit I will refer to what you call "MBN" as "MB_NODE" to make it explicit that it is memory bandwidth allocation at node scope)

I am trying to consider how this can be accomplished while also considering all the other new hardware features that resctrl need to support. Consider, for example, AMD's "Global MBA" (https://lore.kernel.org/lkml/[email protected]/) that throttles memory bandwidth at L3 scope but the user configures allocations at NODE scope. At this time the plan is to support this with a second control associated with the MB resource that can allocate memory bandwidth at node scope. See
https://lore.kernel.org/lkml/[email protected]/

If resctrl creates a new resource for node scoped memory bandwidth allocations to support these "type 2" systems then that will result in an inconsistent interface between architectures that we should avoid.

Have you been listening in on the discussions surrounding emulated controls? Considering that, would it be possible to support the "MB" control on a type "2" system but have it be backed by (emulated by) the underlying "MB_NODE" control?

resctrl could expose both controls on these "type 2" systems but make it clear that "MB" is emulated by "MB_NODE". For example:

info/

└── MB/
     └── resource_schemata/
         └── MB/
             └── MB_NODE/

User will see both controls in schemata file but when changes are made to "MB" control it will show in the "MB_NODE" control and vice-versa. User could also disable the "MB" control that will establish familiarity with the interface at which point resctrl can drop the "MB" control from the schemata file on these "type 2" systems.

Having the MB resource available with an MB control will keep resctrl backward compatible if there are any tools that expect that. If backward compatibility is not of concern then resctrl could initialize with the emulated control disabled by default. See discussion at
https://lore.kernel.org/lkml/[email protected]/
that describes a new resctrl capability in support of RISC-V and RDT.
With this resctrl could initialize with:

info/

└── MB/
     └── resource_schemata/
         ├── MB/
         │   ├── MB_NODE/
         │   │   └── status:enabled
         │   └── status:disabled
         └── mode:legacy [native]
Quoted 3 lines

With above a "type 2" system will boot with its schemata file just containing the "MB_NODE" control while info/MB describes the memory bandwidth resource.

On type 3 machine, schemata has both MB in legacy mode with cache id as domain id and MB_NODE with numa id as domain id.

Is this directory OK?

 info/

 └── MB/
      └── resource_schemata/
          ├── MB/
          │   ├── MB_NODE/
          │   │   └── status:disabled
          │   └── status:enabled
          ├── MB_NODE/
          └── mode:node
Quoted 1 line

1. MB and MB_NODE are shown in parallel in inf/MB/resource_schemata/

I do not think there is a need to expose an emulated MB_NODE control if the actual MB_NODE hardware control exists.

Quoted 1 line

2. mode is set as "node" meaning "MB" is for L3 and "MB_NODE" is for numa node

I assume you mean "scope" instead of "mode"? (more below)

Quoted 1 line

3. Emulation "MB_NODE" is disabled (or should the "MB_NODE" sub-dir be invisible?)

Right, I do not think emulation is needed here. No need to make it invisible since it should not exist.

From what I understand these "type 3" machines could be simplified to:

info/

└── MB/
    └── resource_schemata/
        ├── MB/
        │   └── scope:L3
        └── MB_NODE/
            └── scope:NODE

Beyond this I believe that MPAM currently emulates the MB control with its "MB_MAX" control and users may want to make bandwidth allocations at the fine granularity that it supports. Taking this into account the interface may end up looking like:

info/

└── MB/
    └── resource_schemata/
        ├── MB/
        │   ├── MB_MAX/
        │   │   └── scope:L3
        │   └── scope:L3
        └── MB_NODE/
            └── scope:NODE

A system like above will thus have three schemata file entries:
MB
MB_MAX
MB_NODE

Three schemata file entries would be unnecessary for users familiar with the finer granularity MB_MAX control so that is where the "mode" file can be used to disable the legacy MB control to just expose MB_MAX and MB_NODE on these systems.

Quoted 7 lines
Quoted 1 line
Quoted 1 line

Would that work for these systems?

I'm not clear on the behaviour when the schemata file contains both MB and MB_MAX entries. I assume we should expect that the user might read back the whole file, modify what they expect to change and then write everything back. If legacy software does this and wants to change the MB schemata then the MB and MB_MAX values would be incompatible. Will we just ignore values that haven't changed and fail the write if two dependent schemata are changed at the same time?

Quoted 3 lines

An alternative would be to follow Dave Martin's suggestion to prefix emulated controls in schemata file with "#":
https://lore.kernel.org/lkml/[email protected]/

I was aware of this but I had (mis)thought that the "mode" file was the alternative to this that was being taken forward.

Quoted 6 lines

This was one element of original discussion that fizzled a bit since it was not obvious that user space would behave like this. Consider, for example, from Documentation/filesystems/resctrl.rst:

Reading/writing the schemata file
---------------------------------
... When writing you only need to specify those values which you wish to change. ..

Yeah, I don't know but it does seem like something a user could reasonably expect to work.

Quoted 5 lines
Quoted 5 lines

Alternatively, could the "mode" file be used to switch between "MB" and "MB_MAX" and the two never need be shown at the same time. The user opts in to using the new interface, "MB_MAX" by setting "mode" and can just toggle back if they want to use "MB" directly again.

Quoted 7 lines

Interesting. So far the "mode" options have been "legacy" and "native" where "legacy" would show the legacy as well as emulated controls in the schemata file and "native" will only show the new controls. resctrl could make the default of "legacy" mean that *only* the legacy control is shown without insight into the controls it is being emulated with. The only insight to this would continue to be via the hierarchy in the info directory, which would show relationship but not the actual control values. Do you think there could be a need for users (excluding validation?) that may want to see the underlying control values used to emulate a legacy control?

They may want to see the underlying values to be able to migrate there legacy configuration to the native configuration but they can just write their legacy configuration and then toggle the mode to see the values in the native mode.

Quoted 4 lines

I assume for backward compatibility that "legacy" would remain the default so a possible inconvenience here would be that users familiar with the new controls would forever need to switch the mode before being able to use them.

This does also make the new controls slightly harder to discover.

Thanks,

Ben

Quoted 5 lines

Having this clear distinction could simplify the implementation.

Quoted 1 line

Reinette

maintainer
#70 lore raw
Quoted 8 lines
Quoted 6 lines

This was one element of original discussion that fizzled a bit since it was not obvious that user space would behave like this. Consider, for example, from Documentation/filesystems/resctrl.rst:

Reading/writing the schemata file
---------------------------------
... When writing you only need to specify those values which you wish to change. ..

Yeah, I don't know but it does seem like something a user could reasonably expect to work.

Personally I use one of two options to update schemata:

1) Just want to change something small:

	# echo "MB:1=50" > schemata
2) I want to make global change to some resource
	# vi schemata
		on L2 line: s/ffff/ff/g
	:wq

In case 2 every domain on every resource is written, but most of them to the existing value.

This may need careful consideration if the legacy "MB" lines are present together with the new controls for MIN, MAX etc.

Having a mode file to select whether legacy OR native controls are exposed would avoid a lot of complexity in the kernel. Yes, that would mean a user that wants to use the new native interface must add one "echo" to some startup script.

Or perhaps yet another "rdt=" command line option to pick whether mount defaults to legacy or native mode?

-Tony

maintainer
#71 lore raw

Hi Tony,

Quoted 11 lines · Tony
Quoted 8 lines
Quoted 6 lines

This was one element of original discussion that fizzled a bit since it was not obvious that user space would behave like this. Consider, for example, from Documentation/filesystems/resctrl.rst:

Reading/writing the schemata file
---------------------------------
... When writing you only need to specify those values which you wish to change. ..

Yeah, I don't know but it does seem like something a user could reasonably expect to work.

Personally I use one of two options to update schemata:

Quoted 8 lines

1) Just want to change something small:

	# echo "MB:1=50" > schemata
2) I want to make global change to some resource
	# vi schemata
		on L2 line: s/ffff/ff/g
	:wq

In case 2 every domain on every resource is written, but most of them to the existing value.

We've got a user :)

Quoted 10 lines

This may need careful consideration if the legacy "MB" lines are present together with the new controls for MIN, MAX etc.

Having a mode file to select whether legacy OR native controls are exposed would avoid a lot of complexity in the kernel. Yes, that would mean a user that wants to use the new native interface must add one "echo" to some startup script.

Or perhaps yet another "rdt=" command line option to pick whether mount defaults to legacy or native mode?

The "rdt=" options are x86 specific so I don't think we should use them for generic resctrl. A mount option might be end up being convenient.

Thanks,

Ben

Quoted 2 lines

-Tony

author maintainer
#72 lore raw

Hi Tony and Ben,

Quoted 33 lines · Ben Horgan

Hi Tony,

Quoted 11 lines · Tony
Quoted 8 lines
Quoted 6 lines

This was one element of original discussion that fizzled a bit since it was not obvious that user space would behave like this. Consider, for example, from Documentation/filesystems/resctrl.rst:

Reading/writing the schemata file
---------------------------------
... When writing you only need to specify those values which you wish to change. ..

Yeah, I don't know but it does seem like something a user could reasonably expect to work.

Personally I use one of two options to update schemata:

Quoted 8 lines

1) Just want to change something small:

	# echo "MB:1=50" > schemata
2) I want to make global change to some resource
	# vi schemata
		on L2 line: s/ffff/ff/g
	:wq

In case 2 every domain on every resource is written, but most of them to the existing value.

We've got a user :)

Quoted 8 lines

This may need careful consideration if the legacy "MB" lines are present together with the new controls for MIN, MAX etc.

Having a mode file to select whether legacy OR native controls are exposed would avoid a lot of complexity in the kernel. Yes, that would mean a user that wants to use the new native interface must add one "echo" to some startup script.

Quoted 5 lines
Quoted 2 lines

Or perhaps yet another "rdt=" command line option to pick whether mount defaults to legacy or native mode?

The "rdt=" options are x86 specific so I don't think we should use them for generic resctrl. A mount option might be end up being convenient.

What is the recommended way for users to discover which mount options are supported by a filesystem? I am not familiar with this area and a quick search showed that this could be supported if user space programmatically splits the mount into distinct steps to first discover if the filesystem's parser accepts the mount parameter before attempting to mount the filesystem. It sounds to me as though a mount parameter would place additional burden on user space?

I am also concerned about how resctrl could confidently build on top of such mount option. Would resctrl need to add new mount options to build on top of the ones introduced in the base or would resctrl need to create two different filesystems? Both sounds like a challenge to me.

Reinette

maintainer
#73 lore raw

Hi Reinette,

Quoted 40 lines · Reinette Chatre

Hi Tony and Ben,

Quoted 36 lines · Ben Horgan

Hi Tony,

Quoted 11 lines · Tony
Quoted 8 lines
Quoted 6 lines

This was one element of original discussion that fizzled a bit since it was not obvious that user space would behave like this. Consider, for example, from Documentation/filesystems/resctrl.rst:

Reading/writing the schemata file
---------------------------------
... When writing you only need to specify those values which you wish to change. ..

Yeah, I don't know but it does seem like something a user could reasonably expect to work.

Personally I use one of two options to update schemata:

Quoted 8 lines

1) Just want to change something small:

	# echo "MB:1=50" > schemata
2) I want to make global change to some resource
	# vi schemata
		on L2 line: s/ffff/ff/g
	:wq

In case 2 every domain on every resource is written, but most of them to the existing value.

We've got a user :)

Quoted 10 lines

This may need careful consideration if the legacy "MB" lines are present together with the new controls for MIN, MAX etc.

Having a mode file to select whether legacy OR native controls are exposed would avoid a lot of complexity in the kernel. Yes, that would mean a user that wants to use the new native interface must add one "echo" to some startup script.

Or perhaps yet another "rdt=" command line option to pick whether mount defaults to legacy or native mode?

Quoted 8 lines
Quoted 2 lines

The "rdt=" options are x86 specific so I don't think we should use them for generic resctrl. A mount option might be end up being convenient.

What is the recommended way for users to discover which mount options are supported by a filesystem? I am not familiar with this area and a quick search showed that this could be supported if user space programmatically splits the mount into distinct steps to first discover if the filesystem's parser accepts the mount parameter before attempting to mount the filesystem. It sounds to me as though a mount parameter would place additional burden on user space?

My thought here was that it could just be an easy way for the user to say they wanted to default to the native interface for all controls and that it would have no effect if there were no possible native controls. We would have to be careful to only introduce any command line or mount option once we have support for everything it can effect so its meaning doesn't change over time.

(I'm also not really familiar with mount option usage which is why I have just talked tangentially to your question.)

Thanks,

Ben

Quoted 7 lines

I am also concerned about how resctrl could confidently build on top of such mount option. Would resctrl need to add new mount options to build on top of the ones introduced in the base or would resctrl need to create two different filesystems? Both sounds like a challenge to me.

Reinette

author maintainer
#74 lore raw
In reply to Ben Horgan

Hi Ben, Chenyu, and Tony,

Quoted 3 lines · Ben Horgan
Quoted 2 lines · Reinette Chatre
Quoted 1 line · Ben Horgan

On 7/14/26 23:06, Reinette Chatre wrote:

...

Quoted 37 lines
Quoted 30 lines
Quoted 22 lines
Quoted 22 lines

Beyond this I believe that MPAM currently emulates the MB control with its "MB_MAX" control and users may want to make bandwidth allocations at the fine granularity that it supports. Taking this into account the interface may end up looking like:

info/

└── MB/
    └── resource_schemata/
        ├── MB/
        │   ├── MB_MAX/
        │   │   └── scope:L3
        │   └── scope:L3
        └── MB_NODE/
            └── scope:NODE

A system like above will thus have three schemata file entries:
MB
MB_MAX
MB_NODE

Three schemata file entries would be unnecessary for users familiar with the finer granularity MB_MAX control so that is where the "mode" file can be used to disable the legacy MB control to just expose MB_MAX and MB_NODE on these systems.

Quoted 7 lines
Quoted 1 line
Quoted 1 line

Would that work for these systems?

I'm not clear on the behaviour when the schemata file contains both MB and MB_MAX entries. I assume we should expect that the user might read back the whole file, modify what they expect to change and then write everything back. If legacy software does this and wants to change the MB schemata then the MB and MB_MAX values would be incompatible. Will we just ignore values that haven't changed and fail the write if two dependent schemata are changed at the same time?

Quoted 3 lines

An alternative would be to follow Dave Martin's suggestion to prefix emulated controls in schemata file with "#":
https://lore.kernel.org/lkml/[email protected]/

I was aware of this but I had (mis)thought that the "mode" file was the alternative to this that was being taken forward.

Quoted 8 lines
Quoted 6 lines

This was one element of original discussion that fizzled a bit since it was not obvious that user space would behave like this. Consider, for example, from Documentation/filesystems/resctrl.rst:

Reading/writing the schemata file
---------------------------------
... When writing you only need to specify those values which you wish to change. ..

Yeah, I don't know but it does seem like something a user could reasonably expect to work.

ok.

Quoted 16 lines
Quoted 10 lines
Quoted 3 lines

Alternatively, could the "mode" file be used to switch between "MB" and "MB_MAX" and the two never need be shown at the same time. The user opts in to using the new interface, "MB_MAX" by setting "mode" and can just toggle back if they want to use "MB" directly again.

Interesting. So far the "mode" options have been "legacy" and "native" where "legacy" would show the legacy as well as emulated controls in the schemata file and "native" will only show the new controls. resctrl could make the default of "legacy" mean that *only* the legacy control is shown without insight into the controls it is being emulated with. The only insight to this would continue to be via the hierarchy in the info directory, which would show relationship but not the actual control values. Do you think there could be a need for users (excluding validation?) that may want to see the underlying control values used to emulate a legacy control?

They may want to see the underlying values to be able to migrate there legacy configuration to the native configuration but they can just write their legacy configuration and then toggle the mode to see the values in the native mode.

Quoted 1 line
Quoted 6 lines
Quoted 3 lines

I assume for backward compatibility that "legacy" would remain the default so a possible inconvenience here would be that users familiar with the new controls would forever need to switch the mode before being able to use them.

This does also make the new controls slightly harder to discover.

What if resctrl combines the two suggestions? Specifically, for backward compatibility the mode will be "legacy" and if there are emulated controls then resctrl displays them in schemata file with "#" prefix but not(*) support any changes from user space to the underlying hardware controls. This will make new controls easier to discover and let user space see the underlying values, but not break a user space that may read schemata file, change a few values, and write entire file back.

How the "enabled" vs "disabled" state of a control works with this needs some confirmation since region-aware MBA support has this extra caveat of supporting MSR and ACPI interfaces which results in the relationship between "mode" and "status" of underlying controls not being consistent. This may be ok but please consider example below.

Thinking through this with examples as I understand MPAM and RDT region aware so far. Could this work for MPAM and region aware MBA?

(*) Should resctrl allow user space to change underlying control value of an emulated control in "legacy" mode?

MPAM
====
1) System default

info hierarchy
--------------
info/

└── MB/
    └── schemata/
        ├── MB/
        │   ├── MB_MAX/
        │   │   └── status:enabled
        │   └── status:enabled
        └── mode:[legacy] native

schemata file
-------------
MB:...
# MB_MAX:...

- In above, user space attempting to change MB_MAX via schemata file with or without "#" prefix will have no effect.
TBD: there should not be obstacles to resctrl supporting user space changes by removing the "#" prefix.
- MB_MAX control values would show the current active control values of MB_MAX control used to emulate the MB control

2) User space switches to fine grained controls

# echo "native" > /sys/fs/resctrl/info/MB/schemata/mode

info hierarchy
--------------
info/

└── MB/
    └── schemata/
        ├── MB/
        │   ├── MB_MAX/
        │   │   └── status:enabled
        │   └── status:disabled
        └── mode:legacy [native]

schemata file
-------------
MB_MAX:...

- Legacy control not displayed at all
- no # prefix

RDT region aware when using MSR interface for legacy controls
=============================================================
1) System default

info hierarchy
--------------
info/

└── MB/
    └── schemata/
        ├── MB/
        │   ├── MB_REGION0_MAX/
        │   │   └── status:disabled
        │   ├── MB_REGION1_MAX/
        │   │   └── status:disabled
        │   ├── MB_REGION2_MAX/
        │   │   └── status:disabled
        │   ├── MB_REGION3_MAX/
        │   │   └── status:disabled
        │   └── status:enabled
        └── mode:[legacy] native

schemata file
-------------
MB:...

2) User space switches to fine grained controls

# echo "native" > /sys/fs/resctrl/info/MB/schemata/mode

info hierarchy
--------------
info/

└── MB/
    └── schemata/
        ├── MB/
        │   ├── MB_REGION0_MAX/
        │   │   └── status:enabled
        │   ├── MB_REGION1_MAX/
        │   │   └── status:enabled
        │   ├── MB_REGION2_MAX/
        │   │   └── status:enabled
        │   ├── MB_REGION3_MAX/
        │   │   └── status:enabled
        │   └── status:disabled
        └── mode:legacy [native]

schemata file
-------------
MB_REGION0_MAX:...
MB_REGION1_MAX:...
MB_REGION2_MAX:...
MB_REGION3_MAX:...

RDT region aware when using ACPI interface for legacy controls
=============================================================
1) System default

info hierarchy
--------------
info/

└── MB/
    └── schemata/
        ├── MB/
        │   ├── MB_REGION0_MAX/
        │   │   └── status:enabled
        │   ├── MB_REGION1_MAX/
        │   │   └── status:enabled
        │   ├── MB_REGION2_MAX/
        │   │   └── status:enabled
        │   ├── MB_REGION3_MAX/
        │   │   └── status:enabled
        │   └── status:enabled
        └── mode:[legacy] native

schemata file
-------------
MB:...
# MB_REGION0_MAX:...
# MB_REGION1_MAX:...
# MB_REGION2_MAX:...
# MB_REGION3_MAX:...

2) User space switches to fine grained controls

# echo "native" > /sys/fs/resctrl/info/MB/schemata/mode

info hierarchy
--------------
info/

└── MB/
    └── schemata/
        ├── MB/
        │   ├── MB_REGION0_MAX/
        │   │   └── status:enabled
        │   ├── MB_REGION1_MAX/
        │   │   └── status:enabled
        │   ├── MB_REGION2_MAX/
        │   │   └── status:enabled
        │   ├── MB_REGION3_MAX/
        │   │   └── status:enabled
        │   └── status:disabled
        └── mode:legacy [native]

schemata file
-------------
MB_REGION0_MAX:...
MB_REGION1_MAX:...
MB_REGION2_MAX:...
MB_REGION3_MAX:...

Reinette

maintainer
#75 lore raw

Hi Reinette,

Quoted 77 lines · Reinette Chatre

Hi Ben, Chenyu, and Tony,

Quoted 3 lines · Ben Horgan
Quoted 2 lines · Reinette Chatre
Quoted 1 line · Ben Horgan

On 7/14/26 23:06, Reinette Chatre wrote:

...

Quoted 37 lines
Quoted 30 lines
Quoted 22 lines
Quoted 22 lines

Beyond this I believe that MPAM currently emulates the MB control with its "MB_MAX" control and users may want to make bandwidth allocations at the fine granularity that it supports. Taking this into account the interface may end up looking like:

info/

└── MB/
    └── resource_schemata/
        ├── MB/
        │   ├── MB_MAX/
        │   │   └── scope:L3
        │   └── scope:L3
        └── MB_NODE/
            └── scope:NODE

A system like above will thus have three schemata file entries:
MB
MB_MAX
MB_NODE

Three schemata file entries would be unnecessary for users familiar with the finer granularity MB_MAX control so that is where the "mode" file can be used to disable the legacy MB control to just expose MB_MAX and MB_NODE on these systems.

Quoted 7 lines
Quoted 1 line
Quoted 1 line

Would that work for these systems?

I'm not clear on the behaviour when the schemata file contains both MB and MB_MAX entries. I assume we should expect that the user might read back the whole file, modify what they expect to change and then write everything back. If legacy software does this and wants to change the MB schemata then the MB and MB_MAX values would be incompatible. Will we just ignore values that haven't changed and fail the write if two dependent schemata are changed at the same time?

Quoted 3 lines

An alternative would be to follow Dave Martin's suggestion to prefix emulated controls in schemata file with "#":
https://lore.kernel.org/lkml/[email protected]/

I was aware of this but I had (mis)thought that the "mode" file was the alternative to this that was being taken forward.

Quoted 8 lines
Quoted 6 lines

This was one element of original discussion that fizzled a bit since it was not obvious that user space would behave like this. Consider, for example, from Documentation/filesystems/resctrl.rst:

Reading/writing the schemata file
---------------------------------
... When writing you only need to specify those values which you wish to change. ..

Yeah, I don't know but it does seem like something a user could reasonably expect to work.

ok.

Quoted 20 lines
Quoted 10 lines
Quoted 3 lines

Alternatively, could the "mode" file be used to switch between "MB" and "MB_MAX" and the two never need be shown at the same time. The user opts in to using the new interface, "MB_MAX" by setting "mode" and can just toggle back if they want to use "MB" directly again.

Interesting. So far the "mode" options have been "legacy" and "native" where "legacy" would show the legacy as well as emulated controls in the schemata file and "native" will only show the new controls. resctrl could make the default of "legacy" mean that *only* the legacy control is shown without insight into the controls it is being emulated with. The only insight to this would continue to be via the hierarchy in the info directory, which would show relationship but not the actual control values. Do you think there could be a need for users (excluding validation?) that may want to see the underlying control values used to emulate a legacy control?

They may want to see the underlying values to be able to migrate there legacy configuration to the native configuration but they can just write their legacy configuration and then toggle the mode to see the values in the native mode.

Quoted 4 lines

I assume for backward compatibility that "legacy" would remain the default so a possible inconvenience here would be that users familiar with the new controls would forever need to switch the mode before being able to use them.

Quoted 7 lines
Quoted 2 lines

This does also make the new controls slightly harder to discover.

What if resctrl combines the two suggestions? Specifically, for backward compatibility the mode will be "legacy" and if there are emulated controls then resctrl displays them in schemata file with "#" prefix but not(*) support any changes from user space to the underlying hardware controls. This will make new controls easier to discover and let user space see the underlying values, but not break a user space that may read schemata file, change a few values, and write entire file back.

My initial impression is that this would work but is probably unnecessary.

Quoted 10 lines

How the "enabled" vs "disabled" state of a control works with this needs some confirmation since region-aware MBA support has this extra caveat of supporting MSR and ACPI interfaces which results in the relationship between "mode" and "status" of underlying controls not being consistent. This may be ok but please consider example below.

Thinking through this with examples as I understand MPAM and RDT region aware so far. Could this work for MPAM and region aware MBA?

(*) Should resctrl allow user space to change underlying control value of an emulated control in "legacy" mode?

I don't think this causes problems for MPAM but for RDT region aware controls couldn't you end up with a control state that isn't reachable just by configuring the legacy schema.

Quoted 16 lines

MPAM
====
1) System default

info hierarchy
--------------
info/

└── MB/
    └── schemata/
        ├── MB/
        │   ├── MB_MAX/
        │   │   └── status:enabled
        │   └── status:enabled
        └── mode:[legacy] native
Quoted 7 lines

schemata file
-------------
MB:...
# MB_MAX:...

- In above, user space attempting to change MB_MAX via schemata file with or without "#" prefix will have no effect.
TBD: there should not be obstacles to resctrl supporting user space changes by removing the "#" prefix.

Perhaps a different prefix for read only lines as opposed to commented lines.

Thanks,

Ben

Quoted 136 lines

- MB_MAX control values would show the current active control values of MB_MAX control used to emulate the MB control

2) User space switches to fine grained controls

# echo "native" > /sys/fs/resctrl/info/MB/schemata/mode

info hierarchy
--------------
info/

└── MB/
    └── schemata/
        ├── MB/
        │   ├── MB_MAX/
        │   │   └── status:enabled
        │   └── status:disabled
        └── mode:legacy [native]

schemata file
-------------
MB_MAX:...

- Legacy control not displayed at all
- no # prefix

RDT region aware when using MSR interface for legacy controls
=============================================================
1) System default

info hierarchy
--------------
info/

└── MB/
    └── schemata/
        ├── MB/
        │   ├── MB_REGION0_MAX/
        │   │   └── status:disabled
        │   ├── MB_REGION1_MAX/
        │   │   └── status:disabled
        │   ├── MB_REGION2_MAX/
        │   │   └── status:disabled
        │   ├── MB_REGION3_MAX/
        │   │   └── status:disabled
        │   └── status:enabled
        └── mode:[legacy] native

schemata file
-------------
MB:...

2) User space switches to fine grained controls

# echo "native" > /sys/fs/resctrl/info/MB/schemata/mode

info hierarchy
--------------
info/

└── MB/
    └── schemata/
        ├── MB/
        │   ├── MB_REGION0_MAX/
        │   │   └── status:enabled
        │   ├── MB_REGION1_MAX/
        │   │   └── status:enabled
        │   ├── MB_REGION2_MAX/
        │   │   └── status:enabled
        │   ├── MB_REGION3_MAX/
        │   │   └── status:enabled
        │   └── status:disabled
        └── mode:legacy [native]

schemata file
-------------
MB_REGION0_MAX:...
MB_REGION1_MAX:...
MB_REGION2_MAX:...
MB_REGION3_MAX:...

RDT region aware when using ACPI interface for legacy controls
=============================================================
1) System default

info hierarchy
--------------
info/

└── MB/
    └── schemata/
        ├── MB/
        │   ├── MB_REGION0_MAX/
        │   │   └── status:enabled
        │   ├── MB_REGION1_MAX/
        │   │   └── status:enabled
        │   ├── MB_REGION2_MAX/
        │   │   └── status:enabled
        │   ├── MB_REGION3_MAX/
        │   │   └── status:enabled
        │   └── status:enabled
        └── mode:[legacy] native

schemata file
-------------
MB:...
# MB_REGION0_MAX:...
# MB_REGION1_MAX:...
# MB_REGION2_MAX:...
# MB_REGION3_MAX:...

2) User space switches to fine grained controls

# echo "native" > /sys/fs/resctrl/info/MB/schemata/mode

info hierarchy
--------------
info/

└── MB/
    └── schemata/
        ├── MB/
        │   ├── MB_REGION0_MAX/
        │   │   └── status:enabled
        │   ├── MB_REGION1_MAX/
        │   │   └── status:enabled
        │   ├── MB_REGION2_MAX/
        │   │   └── status:enabled
        │   ├── MB_REGION3_MAX/
        │   │   └── status:enabled
        │   └── status:disabled
        └── mode:legacy [native]

schemata file
-------------
MB_REGION0_MAX:...
MB_REGION1_MAX:...
MB_REGION2_MAX:...
MB_REGION3_MAX:...

Reinette

author maintainer
#76 lore raw

Hi Ben,

Quoted 82 lines · Ben Horgan

Hi Reinette,

Quoted 79 lines · Reinette Chatre

Hi Ben, Chenyu, and Tony,

Quoted 3 lines · Ben Horgan
Quoted 2 lines · Reinette Chatre
Quoted 1 line · Ben Horgan

On 7/14/26 23:06, Reinette Chatre wrote:

...

Quoted 37 lines
Quoted 30 lines
Quoted 22 lines
Quoted 22 lines

Beyond this I believe that MPAM currently emulates the MB control with its "MB_MAX" control and users may want to make bandwidth allocations at the fine granularity that it supports. Taking this into account the interface may end up looking like:

info/

└── MB/
    └── resource_schemata/
        ├── MB/
        │   ├── MB_MAX/
        │   │   └── scope:L3
        │   └── scope:L3
        └── MB_NODE/
            └── scope:NODE

A system like above will thus have three schemata file entries:
MB
MB_MAX
MB_NODE

Three schemata file entries would be unnecessary for users familiar with the finer granularity MB_MAX control so that is where the "mode" file can be used to disable the legacy MB control to just expose MB_MAX and MB_NODE on these systems.

Quoted 7 lines
Quoted 1 line
Quoted 1 line

Would that work for these systems?

I'm not clear on the behaviour when the schemata file contains both MB and MB_MAX entries. I assume we should expect that the user might read back the whole file, modify what they expect to change and then write everything back. If legacy software does this and wants to change the MB schemata then the MB and MB_MAX values would be incompatible. Will we just ignore values that haven't changed and fail the write if two dependent schemata are changed at the same time?

Quoted 3 lines

An alternative would be to follow Dave Martin's suggestion to prefix emulated controls in schemata file with "#":
https://lore.kernel.org/lkml/[email protected]/

I was aware of this but I had (mis)thought that the "mode" file was the alternative to this that was being taken forward.

Quoted 8 lines
Quoted 6 lines

This was one element of original discussion that fizzled a bit since it was not obvious that user space would behave like this. Consider, for example, from Documentation/filesystems/resctrl.rst:

Reading/writing the schemata file
---------------------------------
... When writing you only need to specify those values which you wish to change. ..

Yeah, I don't know but it does seem like something a user could reasonably expect to work.

ok.

Quoted 22 lines
Quoted 10 lines
Quoted 3 lines

Alternatively, could the "mode" file be used to switch between "MB" and "MB_MAX" and the two never need be shown at the same time. The user opts in to using the new interface, "MB_MAX" by setting "mode" and can just toggle back if they want to use "MB" directly again.

Interesting. So far the "mode" options have been "legacy" and "native" where "legacy" would show the legacy as well as emulated controls in the schemata file and "native" will only show the new controls. resctrl could make the default of "legacy" mean that *only* the legacy control is shown without insight into the controls it is being emulated with. The only insight to this would continue to be via the hierarchy in the info directory, which would show relationship but not the actual control values. Do you think there could be a need for users (excluding validation?) that may want to see the underlying control values used to emulate a legacy control?

They may want to see the underlying values to be able to migrate there legacy configuration to the native configuration but they can just write their legacy configuration and then toggle the mode to see the values in the native mode.

Quoted 4 lines

I assume for backward compatibility that "legacy" would remain the default so a possible inconvenience here would be that users familiar with the new controls would forever need to switch the mode before being able to use them.

This does also make the new controls slightly harder to discover.

Quoted 7 lines
Quoted 5 lines

What if resctrl combines the two suggestions? Specifically, for backward compatibility the mode will be "legacy" and if there are emulated controls then resctrl displays them in schemata file with "#" prefix but not(*) support any changes from user space to the underlying hardware controls. This will make new controls easier to discover and let user space see the underlying values, but not break a user space that may read schemata file, change a few values, and write entire file back.

My initial impression is that this would work but is probably unnecessary.

ok. My goal was to present ideas to address the issues raised so far. Ideally this would result in discussion of pros/cons. If you find this unnecessary, could you please expand with some insight into which parts you find unnecessary or how you would prefer this solved? I am finding it difficult to interpret above response.

Quoted 5 lines
Quoted 5 lines

How the "enabled" vs "disabled" state of a control works with this needs some confirmation since region-aware MBA support has this extra caveat of supporting MSR and ACPI interfaces which results in the relationship between "mode" and "status" of underlying controls not being consistent. This may be ok but please consider example below.

Quoted 7 lines
Quoted 4 lines

Thinking through this with examples as I understand MPAM and RDT region aware so far. Could this work for MPAM and region aware MBA?

(*) Should resctrl allow user space to change underlying control value of an emulated control in "legacy" mode?

I don't think this causes problems for MPAM but for RDT region aware controls couldn't you end up with a control state that isn't reachable just by configuring the legacy schema.

Could you please highlight the scenario you refer to?

Quoted 22 lines
Quoted 22 lines

MPAM
====
1) System default

info hierarchy
--------------
info/

└── MB/
    └── schemata/
        ├── MB/
        │   ├── MB_MAX/
        │   │   └── status:enabled
        │   └── status:enabled
        └── mode:[legacy] native

schemata file
-------------
MB:...
# MB_MAX:...

Quoted 4 lines
Quoted 2 lines

- In above, user space attempting to change MB_MAX via schemata file with or without "#" prefix will have no effect.
TBD: there should not be obstacles to resctrl supporting user space changes by removing the "#" prefix.

Perhaps a different prefix for read only lines as opposed to commented lines.

Which prefix do you have in mind?

Reinette

maintainer
#77 lore raw

Hi Reinette,

Quoted 91 lines · Reinette Chatre

Hi Ben,

Quoted 88 lines · Ben Horgan

Hi Reinette,

Quoted 84 lines · Reinette Chatre

Hi Ben, Chenyu, and Tony,

Quoted 3 lines · Ben Horgan
Quoted 2 lines · Reinette Chatre
Quoted 1 line · Ben Horgan

On 7/14/26 23:06, Reinette Chatre wrote:

...

Quoted 37 lines
Quoted 30 lines
Quoted 22 lines
Quoted 22 lines

Beyond this I believe that MPAM currently emulates the MB control with its "MB_MAX" control and users may want to make bandwidth allocations at the fine granularity that it supports. Taking this into account the interface may end up looking like:

info/

└── MB/
    └── resource_schemata/
        ├── MB/
        │   ├── MB_MAX/
        │   │   └── scope:L3
        │   └── scope:L3
        └── MB_NODE/
            └── scope:NODE

A system like above will thus have three schemata file entries:
MB
MB_MAX
MB_NODE

Three schemata file entries would be unnecessary for users familiar with the finer granularity MB_MAX control so that is where the "mode" file can be used to disable the legacy MB control to just expose MB_MAX and MB_NODE on these systems.

Quoted 7 lines
Quoted 1 line
Quoted 1 line

Would that work for these systems?

I'm not clear on the behaviour when the schemata file contains both MB and MB_MAX entries. I assume we should expect that the user might read back the whole file, modify what they expect to change and then write everything back. If legacy software does this and wants to change the MB schemata then the MB and MB_MAX values would be incompatible. Will we just ignore values that haven't changed and fail the write if two dependent schemata are changed at the same time?

Quoted 3 lines

An alternative would be to follow Dave Martin's suggestion to prefix emulated controls in schemata file with "#":
https://lore.kernel.org/lkml/[email protected]/

I was aware of this but I had (mis)thought that the "mode" file was the alternative to this that was being taken forward.

Quoted 8 lines
Quoted 6 lines

This was one element of original discussion that fizzled a bit since it was not obvious that user space would behave like this. Consider, for example, from Documentation/filesystems/resctrl.rst:

Reading/writing the schemata file
---------------------------------
... When writing you only need to specify those values which you wish to change. ..

Yeah, I don't know but it does seem like something a user could reasonably expect to work.

ok.

Quoted 16 lines
Quoted 10 lines
Quoted 3 lines

Alternatively, could the "mode" file be used to switch between "MB" and "MB_MAX" and the two never need be shown at the same time. The user opts in to using the new interface, "MB_MAX" by setting "mode" and can just toggle back if they want to use "MB" directly again.

Interesting. So far the "mode" options have been "legacy" and "native" where "legacy" would show the legacy as well as emulated controls in the schemata file and "native" will only show the new controls. resctrl could make the default of "legacy" mean that *only* the legacy control is shown without insight into the controls it is being emulated with. The only insight to this would continue to be via the hierarchy in the info directory, which would show relationship but not the actual control values. Do you think there could be a need for users (excluding validation?) that may want to see the underlying control values used to emulate a legacy control?

They may want to see the underlying values to be able to migrate there legacy configuration to the native configuration but they can just write their legacy configuration and then toggle the mode to see the values in the native mode.

Quoted 1 line
Quoted 6 lines
Quoted 3 lines

I assume for backward compatibility that "legacy" would remain the default so a possible inconvenience here would be that users familiar with the new controls would forever need to switch the mode before being able to use them.

This does also make the new controls slightly harder to discover.

What if resctrl combines the two suggestions? Specifically, for backward compatibility the mode will be "legacy" and if there are emulated controls then resctrl displays them in schemata file with "#" prefix but not(*) support any changes from user space to the underlying hardware controls. This will make new controls easier to discover and let user space see the underlying values, but not break a user space that may read schemata file, change a few values, and write entire file back.

Quoted 5 lines
Quoted 1 line

My initial impression is that this would work but is probably unnecessary.

ok. My goal was to present ideas to address the issues raised so far. Ideally this would result in discussion of pros/cons. If you find this unnecessary, could you please expand with some insight into which parts you find unnecessary or how you would prefer this solved? I am finding it difficult to interpret above response.

Sorry, yes. I was a bit lazy in my reply yesterday. If the mode is easy and non-destructive to change then doesn't the user get the same information with just the burden of toggling the mode and then reading the schemata a second time. This does rely on the user being aware of the new behaviour though.

If the emulation isn't a, one to one, bijection then is changing mode expected to be destructive? More on this below.

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines

How the "enabled" vs "disabled" state of a control works with this needs some confirmation since region-aware MBA support has this extra caveat of supporting MSR and ACPI interfaces which results in the relationship between "mode" and "status" of underlying controls not being consistent. This may be ok but please consider example below.

Thinking through this with examples as I understand MPAM and RDT region aware so far. Could this work for MPAM and region aware MBA?

Quoted 6 lines
Quoted 4 lines
Quoted 1 line

(*) Should resctrl allow user space to change underlying control value of an emulated control in "legacy" mode?

I don't think this causes problems for MPAM but for RDT region aware controls couldn't you end up with a control state that isn't reachable just by configuring the legacy schema.

Could you please highlight the scenario you refer to?

My expectation here was that for any value of MB, X, the value for each of the regions would be the same, Y. (Not sure if this is actually the case.)

MB: X
# MB_REGION0_MAX:Y
# MB_REGION1_MAX:Y
# MB_REGION2_MAX:Y
# MB_REGION3_MAX:Y

So, if one of the regions was to be changed individually then it would be in a state not reachable by just changing the legacy control, MB. Potentially this complicated switching mode as well as uncommenting and writing a schemata.

I was thinking that in legacy mode that the region values would always be kept the same but perhaps the legacy (MB) value could just be the 'best estimate' and the native (MB_REGION0) values show the full story. 'best estimate' could be difficult to choose but allows switching mode to be non-destructive.

For MPAM, at least until Fenghua sent his series today on MBA control emulation [1], I was expecting that MPAM would only use emulation for exposing finer grade controls to the user and not allowing different underlying hardware to be used for the emulation. To me, it seems reasonable that a new control would involve a new schema but I do appreciate the benefit of being able to continue to use the old interface without stopping a new interface being used. The danger is that there are unexpected user visible side effects, e.g. domain lifetime, and that control behaviour is different from the user expectations.

[1] https://lore.kernel.org/linux-arm-kernel/[email protected]/

Quoted 22 lines
Quoted 22 lines
Quoted 22 lines

MPAM
====
1) System default

info hierarchy
--------------
info/

└── MB/
    └── schemata/
        ├── MB/
        │   ├── MB_MAX/
        │   │   └── status:enabled
        │   └── status:enabled
        └── mode:[legacy] native

schemata file
-------------
MB:...
# MB_MAX:...

Quoted 6 lines
Quoted 4 lines
Quoted 2 lines

- In above, user space attempting to change MB_MAX via schemata file with or without "#" prefix will have no effect.
TBD: there should not be obstacles to resctrl supporting user space changes by removing the "#" prefix.

Perhaps a different prefix for read only lines as opposed to commented lines.

Which prefix do you have in mind?

I was avoiding that.. but perhaps for read only lines perhaps they could be in brackets as it hints at it being an extra explanation.

(MB_MAX:...)

Thanks,

Ben

Quoted 4 lines

Reinette

author maintainer
#78 lore raw

Hi Ben,

Quoted 9 lines · Ben Horgan
Quoted 8 lines · Reinette Chatre
Quoted 7 lines · Ben Horgan
Quoted 6 lines · Reinette Chatre

Hi Ben, Chenyu, and Tony,

Quoted 3 lines · Ben Horgan
Quoted 2 lines · Reinette Chatre
Quoted 1 line · Ben Horgan

On 7/14/26 23:06, Reinette Chatre wrote:

...

Quoted 34 lines
Quoted 28 lines
Quoted 28 lines
Quoted 27 lines
Quoted 16 lines
Quoted 10 lines
Quoted 3 lines

Alternatively, could the "mode" file be used to switch between "MB" and "MB_MAX" and the two never need be shown at the same time. The user opts in to using the new interface, "MB_MAX" by setting "mode" and can just toggle back if they want to use "MB" directly again.

Interesting. So far the "mode" options have been "legacy" and "native" where "legacy" would show the legacy as well as emulated controls in the schemata file and "native" will only show the new controls. resctrl could make the default of "legacy" mean that *only* the legacy control is shown without insight into the controls it is being emulated with. The only insight to this would continue to be via the hierarchy in the info directory, which would show relationship but not the actual control values. Do you think there could be a need for users (excluding validation?) that may want to see the underlying control values used to emulate a legacy control?

They may want to see the underlying values to be able to migrate there legacy configuration to the native configuration but they can just write their legacy configuration and then toggle the mode to see the values in the native mode.

Quoted 1 line
Quoted 6 lines
Quoted 3 lines

I assume for backward compatibility that "legacy" would remain the default so a possible inconvenience here would be that users familiar with the new controls would forever need to switch the mode before being able to use them.

This does also make the new controls slightly harder to discover.

What if resctrl combines the two suggestions? Specifically, for backward compatibility the mode will be "legacy" and if there are emulated controls then resctrl displays them in schemata file with "#" prefix but not(*) support any changes from user space to the underlying hardware controls. This will make new controls easier to discover and let user space see the underlying values, but not break a user space that may read schemata file, change a few values, and write entire file back.

Quoted 5 lines
Quoted 1 line

My initial impression is that this would work but is probably unnecessary.

ok. My goal was to present ideas to address the issues raised so far. Ideally this would result in discussion of pros/cons. If you find this unnecessary, could you please expand with some insight into which parts you find unnecessary or how you would prefer this solved? I am finding it difficult to interpret above response.

Quoted 7 lines

Sorry, yes. I was a bit lazy in my reply yesterday. If the mode is easy and non-destructive to change then doesn't the user get the same information with just the burden of toggling the mode and then reading the schemata a second time. This does rely on the user being aware of the new behaviour though.

If the emulation isn't a, one to one, bijection then is changing mode expected to be destructive? More on this below.

There may be some corner cases (region aware may have some as you highlight below), but in general I do not expect changing mode to be destructive. The idea behind giving insight into underlying control values while legacy interface is in use is to address your earlier point that not doing so will make new controls slightly harder to discover. (more below)

Quoted 24 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

How the "enabled" vs "disabled" state of a control works with this needs some confirmation since region-aware MBA support has this extra caveat of supporting MSR and ACPI interfaces which results in the relationship between "mode" and "status" of underlying controls not being consistent. This may be ok but please consider example below.

Thinking through this with examples as I understand MPAM and RDT region aware so far. Could this work for MPAM and region aware MBA?

Quoted 6 lines
Quoted 4 lines
Quoted 1 line

(*) Should resctrl allow user space to change underlying control value of an emulated control in "legacy" mode?

I don't think this causes problems for MPAM but for RDT region aware controls couldn't you end up with a control state that isn't reachable just by configuring the legacy schema.

Could you please highlight the scenario you refer to?

My expectation here was that for any value of MB, X, the value for each of the regions would be the same, Y. (Not sure if this is actually the case.)

MB: X
# MB_REGION0_MAX:Y
# MB_REGION1_MAX:Y
# MB_REGION2_MAX:Y
# MB_REGION3_MAX:Y

Quoted 3 lines

So, if one of the regions was to be changed individually then it would be in a state not reachable by just changing the legacy control, MB. Potentially this complicated switching mode as well as uncommenting and writing a schemata.

Indeed. This scenario was highlighted in slide 8 of
https://lpc.events/event/19/contributions/2093/attachments/1958/4172/resctrl%20Microconference%20LPC%202025%20Tokyo.pdf
and discussed between Tony and Dave Martin in the thread starting at
https://lore.kernel.org/lkml/aPf0OKwDZ4XbmVRB@agluck-desk3/

Tony and Dave discussed a few scenarios and how resctrl could behave with different user interactions. On a high level I understood that the underlying controls will always, as you state below, "show the full story" and if user space interacts with them (instead of using the legacy control) then the control values associated with the legacy control cannot be relied on to represent accurate state to the point that it may even be better to not be visible.
You are right, this is a good motivation to not allow user to modify underlying controls when in "legacy" mode.

The remaining open is whether resctrl should display the (read-only) underlying control values in schemata file when in "legacy" mode. By default this does not work for region-aware since it will at least initially use different hardware interfaces for the different controls, essentially this means that there is no actual emulation and the legacy and region aware control are both legitimate hardware controls. Specifically, considering your example, there is no mapping from "X" to "Y".

The way region-aware is planning to address this is to use the "enabled" vs "disabled" status of a control to start with underlying controls disabled so that they are not displayed in schemata file when legacy mode is enabled.

If resctrl instead only displays legacy controls when in "legacy" mode then the "status" may no longer be needed for region-aware. RISC-V also considered using this "status" that I think may also be solved with this approach, I am not sure though.

Quoted 5 lines

I was thinking that in legacy mode that the region values would always be kept the same but perhaps the legacy (MB) value could just be the 'best estimate' and the native (MB_REGION0) values show the full story. 'best estimate' could be difficult to choose but allows switching mode to be non-destructive.

Quoted 7 lines

For MPAM, at least until Fenghua sent his series today on MBA control emulation [1], I was expecting that MPAM would only use emulation for exposing finer grade controls to the user and not allowing different underlying hardware to be used for the emulation. To me, it seems reasonable that a new control would involve a new schema but I do appreciate the benefit of being able to continue to use the old interface without stopping a new interface being used. The danger is that there are unexpected user visible side effects, e.g. domain lifetime, and that control behaviour is different from the user expectations.

I have not looked at Fenghua's series in detail but from earlier discussion I understood the high level problem to be that user space may have expectation that for every "resource" represented by a directory in info/ there is a matching entry in the schemata file. Whether this is an actual expectation from user space tools is not obvious to me though: there is already an exception since there is the "L3_MON" "resource" that does not have a schemata file entry.

To ensure backward compatibility the safest would be for resctrl to always provide a "MB" control via an entry in the schemata file when the "MB" resource is exposed via info/. If on the other hand resctrl is not expected to provide a "MB" entry in schemata file when there is a "MB" resource described in info/MB then resctrl should not be forced to always provide such emulation.

I would appreciate your thoughts here.

Quoted 27 lines

[1] https://lore.kernel.org/linux-arm-kernel/[email protected]/

Quoted 25 lines
Quoted 25 lines
Quoted 24 lines

MPAM
====
1) System default

info hierarchy
--------------
info/

└── MB/
    └── schemata/
        ├── MB/
        │   ├── MB_MAX/
        │   │   └── status:enabled
        │   └── status:enabled
        └── mode:[legacy] native

schemata file
-------------
MB:...
# MB_MAX:...

- In above, user space attempting to change MB_MAX via schemata file with or without "#" prefix will have no effect.
TBD: there should not be obstacles to resctrl supporting user space changes by removing the "#" prefix.

Quoted 8 lines
Quoted 3 lines
Quoted 1 line

Perhaps a different prefix for read only lines as opposed to commented lines.

Which prefix do you have in mind?

I was avoiding that.. but perhaps for read only lines perhaps they could be in brackets as it hints at it being an extra explanation.

(MB_MAX:...)

Sure (although it seems we are moving away from this approach).

Reinette

maintainer
#79 lore raw

Hi Reinette,

Quoted 136 lines · Reinette Chatre

Hi Ben,

Quoted 9 lines · Ben Horgan
Quoted 8 lines · Reinette Chatre
Quoted 7 lines · Ben Horgan
Quoted 6 lines · Reinette Chatre

Hi Ben, Chenyu, and Tony,

Quoted 3 lines · Ben Horgan
Quoted 2 lines · Reinette Chatre
Quoted 1 line · Ben Horgan

On 7/14/26 23:06, Reinette Chatre wrote:

...

Quoted 34 lines
Quoted 28 lines
Quoted 28 lines
Quoted 27 lines
Quoted 16 lines
Quoted 10 lines
Quoted 3 lines

Alternatively, could the "mode" file be used to switch between "MB" and "MB_MAX" and the two never need be shown at the same time. The user opts in to using the new interface, "MB_MAX" by setting "mode" and can just toggle back if they want to use "MB" directly again.

Interesting. So far the "mode" options have been "legacy" and "native" where "legacy" would show the legacy as well as emulated controls in the schemata file and "native" will only show the new controls. resctrl could make the default of "legacy" mean that *only* the legacy control is shown without insight into the controls it is being emulated with. The only insight to this would continue to be via the hierarchy in the info directory, which would show relationship but not the actual control values. Do you think there could be a need for users (excluding validation?) that may want to see the underlying control values used to emulate a legacy control?

They may want to see the underlying values to be able to migrate there legacy configuration to the native configuration but they can just write their legacy configuration and then toggle the mode to see the values in the native mode.

Quoted 1 line
Quoted 6 lines
Quoted 3 lines

I assume for backward compatibility that "legacy" would remain the default so a possible inconvenience here would be that users familiar with the new controls would forever need to switch the mode before being able to use them.

This does also make the new controls slightly harder to discover.

What if resctrl combines the two suggestions? Specifically, for backward compatibility the mode will be "legacy" and if there are emulated controls then resctrl displays them in schemata file with "#" prefix but not(*) support any changes from user space to the underlying hardware controls. This will make new controls easier to discover and let user space see the underlying values, but not break a user space that may read schemata file, change a few values, and write entire file back.

Quoted 5 lines
Quoted 1 line

My initial impression is that this would work but is probably unnecessary.

ok. My goal was to present ideas to address the issues raised so far. Ideally this would result in discussion of pros/cons. If you find this unnecessary, could you please expand with some insight into which parts you find unnecessary or how you would prefer this solved? I am finding it difficult to interpret above response.

Quoted 7 lines

Sorry, yes. I was a bit lazy in my reply yesterday. If the mode is easy and non-destructive to change then doesn't the user get the same information with just the burden of toggling the mode and then reading the schemata a second time. This does rely on the user being aware of the new behaviour though.

If the emulation isn't a, one to one, bijection then is changing mode expected to be destructive? More on this below.

There may be some corner cases (region aware may have some as you highlight below), but in general I do not expect changing mode to be destructive. The idea behind giving insight into underlying control values while legacy interface is in use is to address your earlier point that not doing so will make new controls slightly harder to discover. (more below)

Quoted 24 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

How the "enabled" vs "disabled" state of a control works with this needs some confirmation since region-aware MBA support has this extra caveat of supporting MSR and ACPI interfaces which results in the relationship between "mode" and "status" of underlying controls not being consistent. This may be ok but please consider example below.

Thinking through this with examples as I understand MPAM and RDT region aware so far. Could this work for MPAM and region aware MBA?

Quoted 6 lines
Quoted 4 lines
Quoted 1 line

(*) Should resctrl allow user space to change underlying control value of an emulated control in "legacy" mode?

I don't think this causes problems for MPAM but for RDT region aware controls couldn't you end up with a control state that isn't reachable just by configuring the legacy schema.

Could you please highlight the scenario you refer to?

My expectation here was that for any value of MB, X, the value for each of the regions would be the same, Y. (Not sure if this is actually the case.)

MB: X
# MB_REGION0_MAX:Y
# MB_REGION1_MAX:Y
# MB_REGION2_MAX:Y
# MB_REGION3_MAX:Y

Quoted 3 lines

So, if one of the regions was to be changed individually then it would be in a state not reachable by just changing the legacy control, MB. Potentially this complicated switching mode as well as uncommenting and writing a schemata.

Indeed. This scenario was highlighted in slide 8 of
https://lpc.events/event/19/contributions/2093/attachments/1958/4172/resctrl%20Microconference%20LPC%202025%20Tokyo.pdf
and discussed between Tony and Dave Martin in the thread starting at
https://lore.kernel.org/lkml/aPf0OKwDZ4XbmVRB@agluck-desk3/

Tony and Dave discussed a few scenarios and how resctrl could behave with different user interactions. On a high level I understood that the underlying controls will always, as you state below, "show the full story" and if user space interacts with them (instead of using the legacy control) then the control values associated with the legacy control cannot be relied on to represent accurate state to the point that it may even be better to not be visible.
You are right, this is a good motivation to not allow user to modify underlying controls when in "legacy" mode.

The remaining open is whether resctrl should display the (read-only) underlying control values in schemata file when in "legacy" mode. By default this does not work for region-aware since it will at least initially use different hardware interfaces for the different controls, essentially this means that there is no actual emulation and the legacy and region aware control are both legitimate hardware controls. Specifically, considering your example, there is no mapping from "X" to "Y".

The way region-aware is planning to address this is to use the "enabled" vs "disabled" status of a control to start with underlying controls disabled so that they are not displayed in schemata file when legacy mode is enabled.

If resctrl instead only displays legacy controls when in "legacy" mode then the "status" may no longer be needed for region-aware. RISC-V also considered using this "status" that I think may also be solved with this approach, I am not sure though.

Quoted 5 lines

I was thinking that in legacy mode that the region values would always be kept the same but perhaps the legacy (MB) value could just be the 'best estimate' and the native (MB_REGION0) values show the full story. 'best estimate' could be difficult to choose but allows switching mode to be non-destructive.

Quoted 7 lines

For MPAM, at least until Fenghua sent his series today on MBA control emulation [1], I was expecting that MPAM would only use emulation for exposing finer grade controls to the user and not allowing different underlying hardware to be used for the emulation. To me, it seems reasonable that a new control would involve a new schema but I do appreciate the benefit of being able to continue to use the old interface without stopping a new interface being used. The danger is that there are unexpected user visible side effects, e.g. domain lifetime, and that control behaviour is different from the user expectations.

I have not looked at Fenghua's series in detail but from earlier discussion I understood the high level problem to be that user space may have expectation that for every "resource" represented by a directory in info/ there is a matching entry in the schemata file. Whether this is an actual expectation from user space tools is not obvious to me though: there is already an exception since there is the "L3_MON" "resource" that does not have a schemata file entry.

Quoted 6 lines

To ensure backward compatibility the safest would be for resctrl to always provide a "MB" control via an entry in the schemata file when the "MB" resource is exposed via info/. If on the other hand resctrl is not expected to provide a "MB" entry in schemata file when there is a "MB" resource described in info/MB then resctrl should not be forced to always provide such emulation.

I would appreciate your thoughts here.

Ok, thanks for pointing this out, I hadn't previously properly understood the motivation for resource emulation.

As emulation adds complication and has the potential to break other user expectations I hope we can manage to keep it to only simple cases. For instance as I mentioned before, the lifetime of a resctrl domain lifetime is necessary different (when there are multiple NUMA nodes) for domains backed by MSC at the memory to MSC at the cache. At the memory they would be inaccessible when the NUMA node is powered off. Also, any partitioning is happening at a different point in the topology.

Can we get around this by by choosing a different organization of the info/ directory? We already have separate resource for L2 and L3. Perhaps, the same for MB, MB_NODE, (not sure about MB_REGION) and move the scope to be a property at info/<resource>/scope rather than info/<resource>/resource_schemata/ctrl/scope.

Can MB_REGION be considered an orthogonal new control or does using it require that the traditional intel MB (delay) not be configured?

On an MPAM system:

info

├── MB
│   ├── resource_schemata
│   │   ├── MB
│   │   └── MB_MAX
│   └── scope
└── MB_NODE
    ├── resource_schemata
    │   └── MB_NODE
    └── scope

On an x86 system:

info

├── MB
│   ├── resource_schemata
│   │   ├── MB
│   │   └── MB_MAX (Finer grained MB, more below)
│   └── scope
└── MB_REGION
    ├── resource_schemata
    │   └── MB_REGION
    └── scope

Do you think this helps?

This also brings another question. On MPAM systems the 'MB_MAX' is backed by the same MSC h/w as MB but it exposed a different interface to the user. If I understand correctly intel have an option to have finer grained control of MB (delay) as well and so it would make sense to use a common name rather than just going for the MPAM centric name of MB_MAX.

Thanks,

Ben

Quoted 40 lines
Quoted 28 lines

[1] https://lore.kernel.org/linux-arm-kernel/[email protected]/

Quoted 25 lines
Quoted 25 lines
Quoted 24 lines

MPAM
====
1) System default

info hierarchy
--------------
info/

└── MB/
    └── schemata/
        ├── MB/
        │   ├── MB_MAX/
        │   │   └── status:enabled
        │   └── status:enabled
        └── mode:[legacy] native

schemata file
-------------
MB:...
# MB_MAX:...

- In above, user space attempting to change MB_MAX via schemata file with or without "#" prefix will have no effect.
TBD: there should not be obstacles to resctrl supporting user space changes by removing the "#" prefix.

Quoted 8 lines
Quoted 3 lines
Quoted 1 line

Perhaps a different prefix for read only lines as opposed to commented lines.

Which prefix do you have in mind?

I was avoiding that.. but perhaps for read only lines perhaps they could be in brackets as it hints at it being an extra explanation.

(MB_MAX:...)

Sure (although it seems we are moving away from this approach).

Reinette

author maintainer
#80 lore raw

Hi Ben,

Quoted 155 lines · Ben Horgan

Hi Reinette,

Quoted 136 lines · Reinette Chatre

Hi Ben,

Quoted 9 lines · Ben Horgan
Quoted 8 lines · Reinette Chatre
Quoted 7 lines · Ben Horgan
Quoted 6 lines · Reinette Chatre

Hi Ben, Chenyu, and Tony,

Quoted 3 lines · Ben Horgan
Quoted 2 lines · Reinette Chatre
Quoted 1 line · Ben Horgan

On 7/14/26 23:06, Reinette Chatre wrote:

...

Quoted 34 lines
Quoted 28 lines
Quoted 28 lines
Quoted 27 lines
Quoted 16 lines
Quoted 10 lines
Quoted 3 lines

Alternatively, could the "mode" file be used to switch between "MB" and "MB_MAX" and the two never need be shown at the same time. The user opts in to using the new interface, "MB_MAX" by setting "mode" and can just toggle back if they want to use "MB" directly again.

Interesting. So far the "mode" options have been "legacy" and "native" where "legacy" would show the legacy as well as emulated controls in the schemata file and "native" will only show the new controls. resctrl could make the default of "legacy" mean that *only* the legacy control is shown without insight into the controls it is being emulated with. The only insight to this would continue to be via the hierarchy in the info directory, which would show relationship but not the actual control values. Do you think there could be a need for users (excluding validation?) that may want to see the underlying control values used to emulate a legacy control?

They may want to see the underlying values to be able to migrate there legacy configuration to the native configuration but they can just write their legacy configuration and then toggle the mode to see the values in the native mode.

Quoted 1 line
Quoted 6 lines
Quoted 3 lines

I assume for backward compatibility that "legacy" would remain the default so a possible inconvenience here would be that users familiar with the new controls would forever need to switch the mode before being able to use them.

This does also make the new controls slightly harder to discover.

What if resctrl combines the two suggestions? Specifically, for backward compatibility the mode will be "legacy" and if there are emulated controls then resctrl displays them in schemata file with "#" prefix but not(*) support any changes from user space to the underlying hardware controls. This will make new controls easier to discover and let user space see the underlying values, but not break a user space that may read schemata file, change a few values, and write entire file back.

Quoted 5 lines
Quoted 1 line

My initial impression is that this would work but is probably unnecessary.

ok. My goal was to present ideas to address the issues raised so far. Ideally this would result in discussion of pros/cons. If you find this unnecessary, could you please expand with some insight into which parts you find unnecessary or how you would prefer this solved? I am finding it difficult to interpret above response.

Quoted 7 lines

Sorry, yes. I was a bit lazy in my reply yesterday. If the mode is easy and non-destructive to change then doesn't the user get the same information with just the burden of toggling the mode and then reading the schemata a second time. This does rely on the user being aware of the new behaviour though.

If the emulation isn't a, one to one, bijection then is changing mode expected to be destructive? More on this below.

There may be some corner cases (region aware may have some as you highlight below), but in general I do not expect changing mode to be destructive. The idea behind giving insight into underlying control values while legacy interface is in use is to address your earlier point that not doing so will make new controls slightly harder to discover. (more below)

Quoted 24 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

How the "enabled" vs "disabled" state of a control works with this needs some confirmation since region-aware MBA support has this extra caveat of supporting MSR and ACPI interfaces which results in the relationship between "mode" and "status" of underlying controls not being consistent. This may be ok but please consider example below.

Thinking through this with examples as I understand MPAM and RDT region aware so far. Could this work for MPAM and region aware MBA?

Quoted 6 lines
Quoted 4 lines
Quoted 1 line

(*) Should resctrl allow user space to change underlying control value of an emulated control in "legacy" mode?

I don't think this causes problems for MPAM but for RDT region aware controls couldn't you end up with a control state that isn't reachable just by configuring the legacy schema.

Could you please highlight the scenario you refer to?

My expectation here was that for any value of MB, X, the value for each of the regions would be the same, Y. (Not sure if this is actually the case.)

MB: X
# MB_REGION0_MAX:Y
# MB_REGION1_MAX:Y
# MB_REGION2_MAX:Y
# MB_REGION3_MAX:Y

Quoted 3 lines

So, if one of the regions was to be changed individually then it would be in a state not reachable by just changing the legacy control, MB. Potentially this complicated switching mode as well as uncommenting and writing a schemata.

Indeed. This scenario was highlighted in slide 8 of
https://lpc.events/event/19/contributions/2093/attachments/1958/4172/resctrl%20Microconference%20LPC%202025%20Tokyo.pdf
and discussed between Tony and Dave Martin in the thread starting at
https://lore.kernel.org/lkml/aPf0OKwDZ4XbmVRB@agluck-desk3/

Tony and Dave discussed a few scenarios and how resctrl could behave with different user interactions. On a high level I understood that the underlying controls will always, as you state below, "show the full story" and if user space interacts with them (instead of using the legacy control) then the control values associated with the legacy control cannot be relied on to represent accurate state to the point that it may even be better to not be visible.
You are right, this is a good motivation to not allow user to modify underlying controls when in "legacy" mode.

The remaining open is whether resctrl should display the (read-only) underlying control values in schemata file when in "legacy" mode. By default this does not work for region-aware since it will at least initially use different hardware interfaces for the different controls, essentially this means that there is no actual emulation and the legacy and region aware control are both legitimate hardware controls. Specifically, considering your example, there is no mapping from "X" to "Y".

The way region-aware is planning to address this is to use the "enabled" vs "disabled" status of a control to start with underlying controls disabled so that they are not displayed in schemata file when legacy mode is enabled.

If resctrl instead only displays legacy controls when in "legacy" mode then the "status" may no longer be needed for region-aware. RISC-V also considered using this "status" that I think may also be solved with this approach, I am not sure though.

Quoted 5 lines

I was thinking that in legacy mode that the region values would always be kept the same but perhaps the legacy (MB) value could just be the 'best estimate' and the native (MB_REGION0) values show the full story. 'best estimate' could be difficult to choose but allows switching mode to be non-destructive.

Quoted 7 lines

For MPAM, at least until Fenghua sent his series today on MBA control emulation [1], I was expecting that MPAM would only use emulation for exposing finer grade controls to the user and not allowing different underlying hardware to be used for the emulation. To me, it seems reasonable that a new control would involve a new schema but I do appreciate the benefit of being able to continue to use the old interface without stopping a new interface being used. The danger is that there are unexpected user visible side effects, e.g. domain lifetime, and that control behaviour is different from the user expectations.

I have not looked at Fenghua's series in detail but from earlier discussion I understood the high level problem to be that user space may have expectation that for every "resource" represented by a directory in info/ there is a matching entry in the schemata file. Whether this is an actual expectation from user space tools is not obvious to me though: there is already an exception since there is the "L3_MON" "resource" that does not have a schemata file entry.

Quoted 6 lines

To ensure backward compatibility the safest would be for resctrl to always provide a "MB" control via an entry in the schemata file when the "MB" resource is exposed via info/. If on the other hand resctrl is not expected to provide a "MB" entry in schemata file when there is a "MB" resource described in info/MB then resctrl should not be forced to always provide such emulation.

I would appreciate your thoughts here.

Ok, thanks for pointing this out, I hadn't previously properly understood the motivation for resource emulation.

As emulation adds complication and has the potential to break other user expectations I hope we can manage to keep it to only simple cases. For instance as I mentioned before, the lifetime of a resctrl domain lifetime is necessary different (when there are multiple NUMA nodes) for domains backed by MSC at the memory to MSC at the cache. At the memory they would be inaccessible when the NUMA node is powered off. Also, any partitioning is happening at a different point in the topology.

Quoted 4 lines

Can we get around this by by choosing a different organization of the info/ directory? We already have separate resource for L2 and L3. Perhaps, the same for MB, MB_NODE, (not sure about MB_REGION) and move the scope to be a property at info/<resource>/scope rather than info/<resource>/resource_schemata/ctrl/scope.

Good point that L2 and L3, which is the same resource at different scope, are treated as different resources. With that view it seems reasonable to also consider MB and MB_NODE as different resources. The point here is that it is at a different scope so more specifically what a "domain ID" in the schemata file represents. With this view, MB_REGION is L3 scope that matches to the MB resource.

You will notice I get a bit lost in the discussion below so getting back here I would like to clarify if you propose that info/ contains a directory for each allocation scope of each resource with that directory containing descriptions of all the controls for that resource at the indicated scope or do you propose that info/ contains a directory for each control?

Quoted 3 lines

Can MB_REGION be considered an orthogonal new control or does using it require that the traditional intel MB (delay) not be configured?

The Intel systems that support both MSR ("traditional") and ACPI (region aware MBA) cannot use both concurrently. Not sure if this answers your question.

Just to clarify, there is no single "MB_REGION" control. When considering the "region aware MBA" feature I am currently aware of the following possible controls:
MB_REGION0_MIN
MB_REGION0_MAX
MB_REGION0_OPT
MB_REGION1_MIN
MB_REGION1_MAX
MB_REGION1_OPT
MB_REGION2_MIN
MB_REGION2_MAX
MB_REGION2_OPT
MB_REGION3_MIN
MB_REGION3_MAX
MB_REGION3_OPT

Quoted 12 lines

On an MPAM system:

info

├── MB
│   ├── resource_schemata
│   │   ├── MB
│   │   └── MB_MAX
│   └── scope
└── MB_NODE
    ├── resource_schemata
    │   └── MB_NODE
    └── scope

Above MB and MB_MAX are represented on the same level. My understanding is that MB_MAX is used by driver as the underlying hardware control for the percentage based MB exposed to user space. Thus, when user space changes the "MB" control via schemata file it is expected to also impact the underlying "MB_MAX" control. By representing them as above this relationship is lost. So far I understood that the "MB_MAX" control will be shown as a child of the "MB" control to show this relationship. Looks like you plan to change this but from what I understand this emulation is still relevant?

When thinking about MPAM, what would the underlying hardware control of "MB_NODE" be? It looks from above that it would either start out by itself having the properties of the underlying "MAX" control or is the plan to have it be a percentage based control backed by the underlying "MAX" hardware control?

I also understand MPAM to support more memory bandwidth controls ("MIN", "HARDMAX"/"HARDLIM", etc.). Do you envision them to exist within info/MB/resource_schemata/<control> as well as within info/MB_NODE/resource_schemata/<control>?

Quoted 15 lines

On an x86 system:

info

├── MB
│   ├── resource_schemata
│   │   ├── MB
│   │   └── MB_MAX (Finer grained MB, more below)
│   └── scope
└── MB_REGION
    ├── resource_schemata
    │   └── MB_REGION
    └── scope
Quoted 1 line

Do you think this helps?

"REGION" is not a new scope but instead region-aware MBA is controlled and manages bandwidth at L3 scope. Combine that with up to (currently) four regions each with three controls I find an interface like above potentially confusing to document in an intuitive way. Unless you are perhaps saying that we should introduce a new separate "MB_REGION" L3 scope resource (so let resctrl support multiple "MB" resources at the same scope?) and then *it* contains the twelve new controls within its resource_schemata directory?

Since the region-aware controls are orthogonal to the MSR based legacy control resctrl would still need a way for user space to switch from one to the other which implies a dependency between "MB" and "MB_REGION" that is not presented in above hierarchy.

I think I am missing quite a bit here as I try to navigate an interface so different from what we have discussed so far.
I would like to explore with more detail how this interface can handle the different scenarios we have discussed so far.

The other x86 feature to consider is AMD's upcoming "Global" MBA/SMBA that exposes memory bandwidth allocation in "groups of L3" that I understand could usually be mapped to NODE scope (but it remains controlled at L3 scope), except for one configuration where it is "SYSTEM"(?) scope.
Ref.: https://lore.kernel.org/lkml/8f77f498b1c77fa8fd8f5d5687f03ae598068544.1776980182.git.babu.moger@amd.com/

Quoted 5 lines

This also brings another question. On MPAM systems the 'MB_MAX' is backed by the same MSC h/w as MB but it exposed a different interface to the user. If I understand correctly intel have an option to have finer grained control of MB (delay) as well and so it would make sense to use a common name rather than just going for the MPAM centric name of MB_MAX.

Apologies but I was not able to parse above.

Reinette

maintainer
#81 lore raw

Hi Reinette,

Quoted 163 lines · Reinette Chatre

Hi Ben,

Quoted 155 lines · Ben Horgan

Hi Reinette,

Quoted 136 lines · Reinette Chatre

Hi Ben,

Quoted 9 lines · Ben Horgan
Quoted 8 lines · Reinette Chatre
Quoted 7 lines · Ben Horgan
Quoted 6 lines · Reinette Chatre

Hi Ben, Chenyu, and Tony,

Quoted 3 lines · Ben Horgan
Quoted 2 lines · Reinette Chatre
Quoted 1 line · Ben Horgan

On 7/14/26 23:06, Reinette Chatre wrote:

...

Quoted 34 lines
Quoted 28 lines
Quoted 28 lines
Quoted 27 lines
Quoted 16 lines
Quoted 10 lines
Quoted 3 lines

Alternatively, could the "mode" file be used to switch between "MB" and "MB_MAX" and the two never need be shown at the same time. The user opts in to using the new interface, "MB_MAX" by setting "mode" and can just toggle back if they want to use "MB" directly again.

Interesting. So far the "mode" options have been "legacy" and "native" where "legacy" would show the legacy as well as emulated controls in the schemata file and "native" will only show the new controls. resctrl could make the default of "legacy" mean that *only* the legacy control is shown without insight into the controls it is being emulated with. The only insight to this would continue to be via the hierarchy in the info directory, which would show relationship but not the actual control values. Do you think there could be a need for users (excluding validation?) that may want to see the underlying control values used to emulate a legacy control?

They may want to see the underlying values to be able to migrate there legacy configuration to the native configuration but they can just write their legacy configuration and then toggle the mode to see the values in the native mode.

Quoted 1 line
Quoted 6 lines
Quoted 3 lines

I assume for backward compatibility that "legacy" would remain the default so a possible inconvenience here would be that users familiar with the new controls would forever need to switch the mode before being able to use them.

This does also make the new controls slightly harder to discover.

What if resctrl combines the two suggestions? Specifically, for backward compatibility the mode will be "legacy" and if there are emulated controls then resctrl displays them in schemata file with "#" prefix but not(*) support any changes from user space to the underlying hardware controls. This will make new controls easier to discover and let user space see the underlying values, but not break a user space that may read schemata file, change a few values, and write entire file back.

Quoted 5 lines
Quoted 1 line

My initial impression is that this would work but is probably unnecessary.

ok. My goal was to present ideas to address the issues raised so far. Ideally this would result in discussion of pros/cons. If you find this unnecessary, could you please expand with some insight into which parts you find unnecessary or how you would prefer this solved? I am finding it difficult to interpret above response.

Quoted 7 lines

Sorry, yes. I was a bit lazy in my reply yesterday. If the mode is easy and non-destructive to change then doesn't the user get the same information with just the burden of toggling the mode and then reading the schemata a second time. This does rely on the user being aware of the new behaviour though.

If the emulation isn't a, one to one, bijection then is changing mode expected to be destructive? More on this below.

There may be some corner cases (region aware may have some as you highlight below), but in general I do not expect changing mode to be destructive. The idea behind giving insight into underlying control values while legacy interface is in use is to address your earlier point that not doing so will make new controls slightly harder to discover. (more below)

Quoted 24 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

How the "enabled" vs "disabled" state of a control works with this needs some confirmation since region-aware MBA support has this extra caveat of supporting MSR and ACPI interfaces which results in the relationship between "mode" and "status" of underlying controls not being consistent. This may be ok but please consider example below.

Thinking through this with examples as I understand MPAM and RDT region aware so far. Could this work for MPAM and region aware MBA?

Quoted 6 lines
Quoted 4 lines
Quoted 1 line

(*) Should resctrl allow user space to change underlying control value of an emulated control in "legacy" mode?

I don't think this causes problems for MPAM but for RDT region aware controls couldn't you end up with a control state that isn't reachable just by configuring the legacy schema.

Could you please highlight the scenario you refer to?

My expectation here was that for any value of MB, X, the value for each of the regions would be the same, Y. (Not sure if this is actually the case.)

MB: X
# MB_REGION0_MAX:Y
# MB_REGION1_MAX:Y
# MB_REGION2_MAX:Y
# MB_REGION3_MAX:Y

Quoted 3 lines

So, if one of the regions was to be changed individually then it would be in a state not reachable by just changing the legacy control, MB. Potentially this complicated switching mode as well as uncommenting and writing a schemata.

Indeed. This scenario was highlighted in slide 8 of
https://lpc.events/event/19/contributions/2093/attachments/1958/4172/resctrl%20Microconference%20LPC%202025%20Tokyo.pdf
and discussed between Tony and Dave Martin in the thread starting at
https://lore.kernel.org/lkml/aPf0OKwDZ4XbmVRB@agluck-desk3/

Tony and Dave discussed a few scenarios and how resctrl could behave with different user interactions. On a high level I understood that the underlying controls will always, as you state below, "show the full story" and if user space interacts with them (instead of using the legacy control) then the control values associated with the legacy control cannot be relied on to represent accurate state to the point that it may even be better to not be visible.
You are right, this is a good motivation to not allow user to modify underlying controls when in "legacy" mode.

The remaining open is whether resctrl should display the (read-only) underlying control values in schemata file when in "legacy" mode. By default this does not work for region-aware since it will at least initially use different hardware interfaces for the different controls, essentially this means that there is no actual emulation and the legacy and region aware control are both legitimate hardware controls. Specifically, considering your example, there is no mapping from "X" to "Y".

The way region-aware is planning to address this is to use the "enabled" vs "disabled" status of a control to start with underlying controls disabled so that they are not displayed in schemata file when legacy mode is enabled.

If resctrl instead only displays legacy controls when in "legacy" mode then the "status" may no longer be needed for region-aware. RISC-V also considered using this "status" that I think may also be solved with this approach, I am not sure though.

Quoted 5 lines

I was thinking that in legacy mode that the region values would always be kept the same but perhaps the legacy (MB) value could just be the 'best estimate' and the native (MB_REGION0) values show the full story. 'best estimate' could be difficult to choose but allows switching mode to be non-destructive.

Quoted 7 lines

For MPAM, at least until Fenghua sent his series today on MBA control emulation [1], I was expecting that MPAM would only use emulation for exposing finer grade controls to the user and not allowing different underlying hardware to be used for the emulation. To me, it seems reasonable that a new control would involve a new schema but I do appreciate the benefit of being able to continue to use the old interface without stopping a new interface being used. The danger is that there are unexpected user visible side effects, e.g. domain lifetime, and that control behaviour is different from the user expectations.

I have not looked at Fenghua's series in detail but from earlier discussion I understood the high level problem to be that user space may have expectation that for every "resource" represented by a directory in info/ there is a matching entry in the schemata file. Whether this is an actual expectation from user space tools is not obvious to me though: there is already an exception since there is the "L3_MON" "resource" that does not have a schemata file entry.

Quoted 6 lines

To ensure backward compatibility the safest would be for resctrl to always provide a "MB" control via an entry in the schemata file when the "MB" resource is exposed via info/. If on the other hand resctrl is not expected to provide a "MB" entry in schemata file when there is a "MB" resource described in info/MB then resctrl should not be forced to always provide such emulation.

I would appreciate your thoughts here.

Ok, thanks for pointing this out, I hadn't previously properly understood the motivation for resource emulation.

As emulation adds complication and has the potential to break other user expectations I hope we can manage to keep it to only simple cases. For instance as I mentioned before, the lifetime of a resctrl domain lifetime is necessary different (when there are multiple NUMA nodes) for domains backed by MSC at the memory to MSC at the cache. At the memory they would be inaccessible when the NUMA node is powered off. Also, any partitioning is happening at a different point in the topology.

Quoted 4 lines

Can we get around this by by choosing a different organization of the info/ directory? We already have separate resource for L2 and L3. Perhaps, the same for MB, MB_NODE, (not sure about MB_REGION) and move the scope to be a property at info/<resource>/scope rather than info/<resource>/resource_schemata/ctrl/scope.

Quoted 4 lines

Good point that L2 and L3, which is the same resource at different scope, are treated as different resources. With that view it seems reasonable to also consider MB and MB_NODE as different resources. The point here is that it is at a different scope so more specifically what a "domain ID" in the schemata file represents. With this view, MB_REGION is L3 scope that matches to the MB resource.

Ok. This makes sense to me. I had confused myself about what MB_REGION is.

Quoted 5 lines

You will notice I get a bit lost in the discussion below so getting back here I would like to clarify if you propose that info/ contains a directory for each allocation scope of each resource with that directory containing descriptions of all the controls for that resource at the indicated scope or do you propose that info/ contains a directory for each control?

The former, info/ contains a directory for each allocation scope of each resource.

info

├── L2
│   ├── resource_schemata
│   │   ├── L2
│   │   ├── L2_CMAX
│   │   └── L2_CMIN
│   └── scope : L2
├── L3
│   ├── resource_schemata
│   │   ├── L3
│   │   ├── L3_CMAX
│   │   └── L3_CMIN
│   └── scope : L3
├── MB
│   ├── resource_schemata
│   │   ├── MB
│   │   │   └── MB_MAX
│   │   ├── MB_MIN
│   │   ├── MB_PBM
│   │   └── MB_PROP
│   └── scope : L3
└── MB_NODE
    ├── resource_schemata
    │   ├── MB_NODE_MAX
    │   ├── MB_NODE_MIN
    │   ├── MB_NODE_PBM
    │   └── MB_NODE_PROP
    └── scope : NUMA NODE
Quoted 7 lines
Quoted 4 lines

Can MB_REGION be considered an orthogonal new control or does using it require that the traditional intel MB (delay) not be configured?

The Intel systems that support both MSR ("traditional") and ACPI (region aware MBA) cannot use both concurrently. Not sure if this answers your question.

Thanks, this answers my question.

Quoted 30 lines

Just to clarify, there is no single "MB_REGION" control. When considering the "region aware MBA" feature I am currently aware of the following possible controls:
MB_REGION0_MIN
MB_REGION0_MAX
MB_REGION0_OPT
MB_REGION1_MIN
MB_REGION1_MAX
MB_REGION1_OPT
MB_REGION2_MIN
MB_REGION2_MAX
MB_REGION2_OPT
MB_REGION3_MIN
MB_REGION3_MAX
MB_REGION3_OPT

Quoted 12 lines

On an MPAM system:

info

├── MB
│   ├── resource_schemata
│   │   ├── MB
│   │   └── MB_MAX
│   └── scope
└── MB_NODE
    ├── resource_schemata
    │   └── MB_NODE
    └── scope
Quoted 7 lines

Above MB and MB_MAX are represented on the same level. My understanding is that MB_MAX is used by driver as the underlying hardware control for the percentage based MB exposed to user space. Thus, when user space changes the "MB" control via schemata file it is expected to also impact the underlying "MB_MAX" control. By representing them as above this relationship is lost. So far I understood that the "MB_MAX" control will be shown as a child of the "MB" control to show this relationship. Looks like you plan to change this but from what I understand this emulation is still relevant?

Please just consider this a mistake. MB_MAX should be a child of MB.

info/

├── MB
│   ├── resource_schemata
│   │   └── MB
│   │       └── MB_MAX
│   └── scope
└── MB_NODE
    ├── resource_schemata
    │   └── MB_NODE_MAX (Changed from MB_NODE)
    └── scope
Quoted 5 lines

When thinking about MPAM, what would the underlying hardware control of "MB_NODE" be? It looks from above that it would either start out by itself having the properties of the underlying "MAX" control or is the plan to have it be a percentage based control backed by the underlying "MAX" hardware control?

The underlying hardware of MB_NODE would be essentially the same hardware as that backing MB_MAX, but at a different location in the SoC, at the memory controller rather than in the L3.

To correct myself slightly, I don't think we should have a control called MB_NODE, rather, it should be MB_NODE_MAX.

My understanding of previous discussions is that <resource_type>_<scope>_<control_type> it the pattern for control names and the pattern for resource names being <resource_type>_<scope>. (allowing for <scope> or <control_type> being missing to match existing naming.)

I don't think we should introduce more percent based controls and for new controls we can introduce a new format to describe them. Perhaps just the positive integer with a resolution supplied in info/ as discussed previously. Although, I have been pondering on whether we can do a bit better.

We could use hexadecimal point based format for controls which are a proportion of a resource and have a resolution which is a power of 2. The advantage of this is that the meaning of the value is independent of the granularity of the control (number of parts).

0 is represented as 0x0

1 as                0x1
1/2 as              0x0.8
7/256 as            0x0.07
1/2**28             0x0.00000001

etc

This maps well to the MPAM fixed-point fraction point format without having the weirdness of having values forced to 1 or 0 not being really 0. These MPAM h/w oddities can be hidden just by using the mbw_min mbw_max of a control. In MPAM this could be used in CMIN, CMAX, MB_MAX, MB_MIN and I would hope this would be useful for other architectures too. I am preparing some RFC patches on top of your PoC for consideration of this idea and to explore some of the proposals discussed relating to generic schemata and how they land in practice from the MPAM side.

Quoted 4 lines

I also understand MPAM to support more memory bandwidth controls ("MIN", "HARDMAX"/"HARDLIM", etc.). Do you envision them to exist within info/MB/resource_schemata/<control> as well as within info/MB_NODE/resource_schemata/<control>?

Yes, at least for MIN, see the info/ tree above. For HARDLIM, perhaps, but HARDLIM has the added complications that it is a property of the MBW_MAX control and that it may be configurable for each PARTID or a fixed property of the h/w. When HARDLIM is configurable the control name could be of the form <resource_type>_<scope>_<control>_<control_param> where <control_param> is HARDLIM and the full name for the HARDLIM configuration on the MB_NODE resource is MB_NODE_MAX_HARDLIM. There can also be an info/<resource>/resource_schemata/<control>/lim file which has values, soft, hard, configurable.

For CMAX, maximum cache capacity, there is an equivalent control SOFTLIM, which behaves as HARDLIM except the meaning of the bit is reversed. We can just use a consistent name in s/w though.

Quoted 24 lines
Quoted 16 lines

On an x86 system:

info

├── MB
│   ├── resource_schemata
│   │   ├── MB
│   │   └── MB_MAX (Finer grained MB, more below)
│   └── scope
└── MB_REGION
    ├── resource_schemata
    │   └── MB_REGION
    └── scope
Quoted 1 line

Do you think this helps?

"REGION" is not a new scope but instead region-aware MBA is controlled and manages bandwidth at L3 scope. Combine that with up to (currently) four regions each with three controls I find an interface like above potentially confusing to document in an intuitive way. Unless you are perhaps saying that we should introduce a new separate "MB_REGION" L3 scope resource (so let resctrl support multiple "MB" resources at the same scope?) and then *it* contains the twelve new controls within its resource_schemata directory?

Quoted 3 lines

Since the region-aware controls are orthogonal to the MSR based legacy control resctrl would still need a way for user space to switch from one to the other which implies a dependency between "MB" and "MB_REGION" that is not presented in above hierarchy.

OK. The /sys/fs/resctrl/info/MB/schemata/mode and the MB_REGION controls a child of MB you described previously seem s better fit than what I suggested.

Quoted 5 lines

I think I am missing quite a bit here as I try to navigate an interface so different from what we have discussed so far.
I would like to explore with more detail how this interface can handle the different scenarios we have discussed so far.

Certainly, I don't think we have got to the bottom of this yet.

Quoted 5 lines

The other x86 feature to consider is AMD's upcoming "Global" MBA/SMBA that exposes memory bandwidth allocation in "groups of L3" that I understand could usually be mapped to NODE scope (but it remains controlled at L3 scope), except for one configuration where it is "SYSTEM"(?) scope.
Ref.: https://lore.kernel.org/lkml/8f77f498b1c77fa8fd8f5d5687f03ae598068544.1776980182.git.babu.moger@amd.com/

Hmmm, I'm not sure that the scope can be considered to be NODE scope for GMBA. To me it seems to be accidental that it maps to the NUMA node but really the scope is just a grouping of L3 instances. For a control to NUMA scope I would expect the resctrl domains to go offline and online in sync with the NUMA nodes. For GMBA it looks like it would just going offline/online based on whether any of the CPUs and so L3 instances in the group are online. Am I correct here?

Assuming the domains are on L3 groups rather than NUMA also changes which end of the link the traffic is regulated and so how cross-NUMA traffic behaves differently. If the domain is an L3 group then a task running on a CPU affine to that L3 group won't be throttled unless that particular domain is throttled but with NUMA node domains it may be throttled if it has traffic going to that domain.

Quoted 8 lines
Quoted 6 lines

This also brings another question. On MPAM systems the 'MB_MAX' is backed by the same MSC h/w as MB but it exposed a different interface to the user. If I understand correctly intel have an option to have finer grained control of MB (delay) as well and so it would make sense to use a common name rather than just going for the MPAM centric name of MB_MAX.

Apologies but I was not able to parse above.

Ok, let me try to explain again (although it's probably not what we want to do). My intent here was to try and explore whether we can reuse naming and controls across architectures in the same way we already have for the L2/L3 cache portion bitmap and the existing MB control.

To quote from a previous mail of yours:
https://lore.kernel.org/lkml/[email protected]/
"""
For example, on an MPAM system (if I understand correctly) the user may see:
info/

  └── MB/
      └── resource_schemata/
          ├── MB/
          │   └── MB_MAX/
          └── MB_MIN/

Compared with a possible implementation on Intel that looks like:
info/

  └── MB/
      └── resource_schemata/
          ├── MB/
          │   └── MB_OPT/
          ├── MB_MAX/
          └── MB_MIN/

"""

In the two setups MPAM MB_MAX and intel MB_OPT play the same role, a finer grained control of the legacy MB control. I was thinking these could share a name (MB-PRECISE), but it probably doesn't make sense as MB_OPT and MB_MAX have different relationships to MB_MIN.

Thanks,

Ben

Quoted 3 lines

Reinette

author maintainer
#82 lore raw

Hi Ben,

Quoted 2 lines · Ben Horgan
Quoted 1 line · Reinette Chatre

On 7/20/26 6:30 AM, Ben Horgan wrote:

...>

Quoted 24 lines

The former, info/ contains a directory for each allocation scope of each resource.

info

├── L2
│   ├── resource_schemata
│   │   ├── L2
│   │   ├── L2_CMAX
│   │   └── L2_CMIN
│   └── scope : L2
├── L3
│   ├── resource_schemata
│   │   ├── L3
│   │   ├── L3_CMAX
│   │   └── L3_CMIN
│   └── scope : L3
├── MB
│   ├── resource_schemata
│   │   ├── MB
│   │   │   └── MB_MAX
│   │   ├── MB_MIN
│   │   ├── MB_PBM
│   │   └── MB_PROP
Quoted 9 lines
│   └── scope : L3
└── MB_NODE
    ├── resource_schemata
    │   ├── MB_NODE_MAX
    │   ├── MB_NODE_MIN
    │   ├── MB_NODE_PBM
    │   └── MB_NODE_PROP
    └── scope : NUMA NODE

At first glance this looks good to me. As you highlight below the nuances of how "Global S/MBA" can fit in here still needs to be worked out. One thing that resctrl may need to highlight when documenting this new capability is that while historically a resource had a matching schemata entry, this is no longer the case. As you show above some systems may have MB_NODE resource but no MB_NODE control while I expect that "Global MBA" (if it adopts this) may indeed have a MB_NODE resource with a MB_NODE control.

Quoted 12 lines

Please just consider this a mistake. MB_MAX should be a child of MB.

info/

├── MB
│   ├── resource_schemata
│   │   └── MB
│   │       └── MB_MAX
│   └── scope
└── MB_NODE
    ├── resource_schemata
    │   └── MB_NODE_MAX (Changed from MB_NODE)
    └── scope

ack.

Quoted 7 lines
Quoted 6 lines

When thinking about MPAM, what would the underlying hardware control of "MB_NODE" be? It looks from above that it would either start out by itself having the properties of the underlying "MAX" control or is the plan to have it be a percentage based control backed by the underlying "MAX" hardware control?

Quoted 5 lines

The underlying hardware of MB_NODE would be essentially the same hardware as that backing MB_MAX, but at a different location in the SoC, at the memory controller rather than in the L3.

To correct myself slightly, I don't think we should have a control called MB_NODE, rather, it should be MB_NODE_MAX.

ack.

Quoted 4 lines

My understanding of previous discussions is that <resource_type>_<scope>_<control_type> it the pattern for control names and the pattern for resource names being <resource_type>_<scope>. (allowing for <scope> or <control_type> being missing to match existing naming.)

This is where discussions are from my view also.

Quoted 3 lines

I don't think we should introduce more percent based controls and for new controls we can introduce a new format to describe them. Perhaps just the positive integer with a resolution supplied in info/

No, we should not introduce more percentage based controls per se. With the new schema format a percentage based control is just a variant of a proportional scalar control.

Quoted 11 lines

as discussed previously. Although, I have been pondering on whether we can do a bit better.

We could use hexadecimal point based format for controls which are a proportion of a resource and have a resolution which is a power of 2. The advantage of this is that the meaning of the value is independent of the granularity of the control (number of parts).

0 is represented as 0x0

1 as                0x1
1/2 as              0x0.8
7/256 as            0x0.07
1/2**28             0x0.00000001
Quoted 8 lines

etc

This maps well to the MPAM fixed-point fraction point format without having the weirdness of having values forced to 1 or 0 not being really 0. These MPAM h/w oddities can be hidden just by using the mbw_min mbw_max of a control. In MPAM this could be used in CMIN, CMAX, MB_MAX, MB_MIN and I would hope this would be useful for other architectures too. I am preparing some RFC patches on top of your PoC for consideration of this idea and to explore some of the proposals discussed relating to generic schemata and how they land in practice from the MPAM side.

I'm going to stand with Dave Martin [1] on this point with a preference to avoid floating point in resctrl input and output.
Could you please elaborate where values are forced to 1 or 0? The new schema format was intentionally created to *avoid* rounding errors (and parsing complexity).

[1] https://lore.kernel.org/lkml/[email protected]/

Quoted 10 lines
Quoted 3 lines

I also understand MPAM to support more memory bandwidth controls ("MIN", "HARDMAX"/"HARDLIM", etc.). Do you envision them to exist within info/MB/resource_schemata/<control> as well as within info/MB_NODE/resource_schemata/<control>?

Yes, at least for MIN, see the info/ tree above. For HARDLIM, perhaps, but HARDLIM has the added complications that it is a property of the MBW_MAX control and that it may be configurable for each PARTID or a fixed property of the h/w. When HARDLIM is configurable the control name could be of the form <resource_type>_<scope>_<control>_<control_param> where <control_param> is HARDLIM and the full name for the HARDLIM configuration on the MB_NODE resource is MB_NODE_MAX_HARDLIM. There can also be an info/<resource>/resource_schemata/<control>/lim file which has values, soft, hard, configurable.

ack. HARDLIM sounds like it would be a new control type. Perhaps a "boolean" type for which new control files need to be decided on? Sounds like you are headed in this direction and already have one control file in mind for this new type.

With this in mind the control could be built on top of what is being developed at the moment, possibly be presented to user space following Dave Martin's suggestion in
https://lore.kernel.org/lkml/[email protected]/:

| MB_HARDMAX: 0=0, 1=1, 2=1, 3=0 [...]

or

| MB_HARDMAX: 0=off, 1=on, 2=on, 3=off [...]

Quoted 28 lines

For CMAX, maximum cache capacity, there is an equivalent control SOFTLIM, which behaves as HARDLIM except the meaning of the bit is reversed. We can just use a consistent name in s/w though.

Quoted 24 lines
Quoted 16 lines

On an x86 system:

info

├── MB
│   ├── resource_schemata
│   │   ├── MB
│   │   └── MB_MAX (Finer grained MB, more below)
│   └── scope
└── MB_REGION
    ├── resource_schemata
    │   └── MB_REGION
    └── scope
Quoted 1 line

Do you think this helps?

"REGION" is not a new scope but instead region-aware MBA is controlled and manages bandwidth at L3 scope. Combine that with up to (currently) four regions each with three controls I find an interface like above potentially confusing to document in an intuitive way. Unless you are perhaps saying that we should introduce a new separate "MB_REGION" L3 scope resource (so let resctrl support multiple "MB" resources at the same scope?) and then *it* contains the twelve new controls within its resource_schemata directory?

Quoted 6 lines
Quoted 3 lines

Since the region-aware controls are orthogonal to the MSR based legacy control resctrl would still need a way for user space to switch from one to the other which implies a dependency between "MB" and "MB_REGION" that is not presented in above hierarchy.

OK. The /sys/fs/resctrl/info/MB/schemata/mode and the MB_REGION controls a child of MB you described previously seem s better fit than what I suggested.

ok, I'll keep following that.

Quoted 21 lines
Quoted 6 lines

I think I am missing quite a bit here as I try to navigate an interface so different from what we have discussed so far.
I would like to explore with more detail how this interface can handle the different scenarios we have discussed so far.

Certainly, I don't think we have got to the bottom of this yet.

Quoted 5 lines

The other x86 feature to consider is AMD's upcoming "Global" MBA/SMBA that exposes memory bandwidth allocation in "groups of L3" that I understand could usually be mapped to NODE scope (but it remains controlled at L3 scope), except for one configuration where it is "SYSTEM"(?) scope.
Ref.: https://lore.kernel.org/lkml/8f77f498b1c77fa8fd8f5d5687f03ae598068544.1776980182.git.babu.moger@amd.com/

Hmmm, I'm not sure that the scope can be considered to be NODE scope for GMBA. To me it seems to be accidental that it maps to the NUMA node but really the scope is just a grouping of L3 instances. For a control to NUMA scope I would expect the resctrl domains to go offline and online in sync with the NUMA nodes. For GMBA it looks like it would just going offline/online based on whether any of the CPUs and so L3 instances in the group are online. Am I correct here?

Quoted 5 lines

Assuming the domains are on L3 groups rather than NUMA also changes which end of the link the traffic is regulated and so how cross-NUMA traffic behaves differently. If the domain is an L3 group then a task running on a CPU affine to that L3 group won't be throttled unless that particular domain is throttled but with NUMA node domains it may be throttled if it has traffic going to that domain.

I'll defer to Babu for accurate answers about this hardware capability.

Quoted 32 lines
Quoted 8 lines
Quoted 6 lines

This also brings another question. On MPAM systems the 'MB_MAX' is backed by the same MSC h/w as MB but it exposed a different interface to the user. If I understand correctly intel have an option to have finer grained control of MB (delay) as well and so it would make sense to use a common name rather than just going for the MPAM centric name of MB_MAX.

Apologies but I was not able to parse above.

Ok, let me try to explain again (although it's probably not what we want to do). My intent here was to try and explore whether we can reuse naming and controls across architectures in the same way we already have for the L2/L3 cache portion bitmap and the existing MB control.

To quote from a previous mail of yours:
https://lore.kernel.org/lkml/[email protected]/
"""
For example, on an MPAM system (if I understand correctly) the user may see:
info/

  └── MB/
      └── resource_schemata/
          ├── MB/
          │   └── MB_MAX/
          └── MB_MIN/

Compared with a possible implementation on Intel that looks like:
info/

  └── MB/
      └── resource_schemata/
          ├── MB/
          │   └── MB_OPT/
          ├── MB_MAX/
          └── MB_MIN/
Quoted 6 lines

"""

In the two setups MPAM MB_MAX and intel MB_OPT play the same role, a finer grained control of the legacy MB control. I was thinking these could share a name (MB-PRECISE), but it probably doesn't make sense as MB_OPT and MB_MAX have different relationships to MB_MIN.

Apologies, I neglected to follow up on this after I clarified internally which RDT control should actually be used to emulate the percentage based control. My original thoughts that you highlight above is not correct the RDT also plans to use the "MAX" underlying hardware control for the percentage based MB control. With this RDT and MPAM should look more similar with RDT simplified (by dropping the region-aware terms) as:

info/

   └── MB/
       └── resource_schemata/
           ├── MB/
           │   └── MB_MAX/
           ├── MB_OPT/
           └── MB_MIN/

Reinette

reviewer
#83 lore raw

Hi Ben/Reinette,

Quoted 201 lines · Reinette Chatre

Hi Ben,

Quoted 2 lines · Ben Horgan
Quoted 1 line · Reinette Chatre

On 7/20/26 6:30 AM, Ben Horgan wrote:

...>

Quoted 24 lines

The former, info/ contains a directory for each allocation scope of each resource.

info

├── L2
│   ├── resource_schemata
│   │   ├── L2
│   │   ├── L2_CMAX
│   │   └── L2_CMIN
│   └── scope : L2
├── L3
│   ├── resource_schemata
│   │   ├── L3
│   │   ├── L3_CMAX
│   │   └── L3_CMIN
│   └── scope : L3
├── MB
│   ├── resource_schemata
│   │   ├── MB
│   │   │   └── MB_MAX
│   │   ├── MB_MIN
│   │   ├── MB_PBM
│   │   └── MB_PROP
Quoted 9 lines
│   └── scope : L3
└── MB_NODE
     ├── resource_schemata
     │   ├── MB_NODE_MAX
     │   ├── MB_NODE_MIN
     │   ├── MB_NODE_PBM
     │   └── MB_NODE_PROP
     └── scope : NUMA NODE

At first glance this looks good to me. As you highlight below the nuances of how "Global S/MBA" can fit in here still needs to be worked out. One thing that resctrl may need to highlight when documenting this new capability is that while historically a resource had a matching schemata entry, this is no longer the case. As you show above some systems may have MB_NODE resource but no MB_NODE control while I expect that "Global MBA" (if it adopts this) may indeed have a MB_NODE resource with a MB_NODE control.

Quoted 12 lines

Please just consider this a mistake. MB_MAX should be a child of MB.

info/

├── MB
│   ├── resource_schemata
│   │   └── MB
│   │       └── MB_MAX
│   └── scope
└── MB_NODE
     ├── resource_schemata
     │   └── MB_NODE_MAX (Changed from MB_NODE)
     └── scope

ack.

Quoted 7 lines
Quoted 6 lines

When thinking about MPAM, what would the underlying hardware control of "MB_NODE" be? It looks from above that it would either start out by itself having the properties of the underlying "MAX" control or is the plan to have it be a percentage based control backed by the underlying "MAX" hardware control?

Quoted 5 lines

The underlying hardware of MB_NODE would be essentially the same hardware as that backing MB_MAX, but at a different location in the SoC, at the memory controller rather than in the L3.

To correct myself slightly, I don't think we should have a control called MB_NODE, rather, it should be MB_NODE_MAX.

ack.

Quoted 4 lines

My understanding of previous discussions is that <resource_type>_<scope>_<control_type> it the pattern for control names and the pattern for resource names being <resource_type>_<scope>. (allowing for <scope> or <control_type> being missing to match existing naming.)

This is where discussions are from my view also.

Quoted 3 lines

I don't think we should introduce more percent based controls and for new controls we can introduce a new format to describe them. Perhaps just the positive integer with a resolution supplied in info/

No, we should not introduce more percentage based controls per se. With the new schema format a percentage based control is just a variant of a proportional scalar control.

Quoted 11 lines

as discussed previously. Although, I have been pondering on whether we can do a bit better.

We could use hexadecimal point based format for controls which are a proportion of a resource and have a resolution which is a power of 2. The advantage of this is that the meaning of the value is independent of the granularity of the control (number of parts).

0 is represented as 0x0

1 as                0x1
1/2 as              0x0.8
7/256 as            0x0.07
1/2**28             0x0.00000001
Quoted 8 lines

etc

This maps well to the MPAM fixed-point fraction point format without having the weirdness of having values forced to 1 or 0 not being really 0. These MPAM h/w oddities can be hidden just by using the mbw_min mbw_max of a control. In MPAM this could be used in CMIN, CMAX, MB_MAX, MB_MIN and I would hope this would be useful for other architectures too. I am preparing some RFC patches on top of your PoC for consideration of this idea and to explore some of the proposals discussed relating to generic schemata and how they land in practice from the MPAM side.

I'm going to stand with Dave Martin [1] on this point with a preference to avoid floating point in resctrl input and output.
Could you please elaborate where values are forced to 1 or 0? The new schema format was intentionally created to *avoid* rounding errors (and parsing complexity).

[1] https://lore.kernel.org/lkml/[email protected]/

Quoted 10 lines
Quoted 3 lines

I also understand MPAM to support more memory bandwidth controls ("MIN", "HARDMAX"/"HARDLIM", etc.). Do you envision them to exist within info/MB/resource_schemata/<control> as well as within info/MB_NODE/resource_schemata/<control>?

Yes, at least for MIN, see the info/ tree above. For HARDLIM, perhaps, but HARDLIM has the added complications that it is a property of the MBW_MAX control and that it may be configurable for each PARTID or a fixed property of the h/w. When HARDLIM is configurable the control name could be of the form <resource_type>_<scope>_<control>_<control_param> where <control_param> is HARDLIM and the full name for the HARDLIM configuration on the MB_NODE resource is MB_NODE_MAX_HARDLIM. There can also be an info/<resource>/resource_schemata/<control>/lim file which has values, soft, hard, configurable.

ack. HARDLIM sounds like it would be a new control type. Perhaps a "boolean" type for which new control files need to be decided on? Sounds like you are headed in this direction and already have one control file in mind for this new type.

With this in mind the control could be built on top of what is being developed at the moment, possibly be presented to user space following Dave Martin's suggestion in
https://lore.kernel.org/lkml/[email protected]/:

| MB_HARDMAX: 0=0, 1=1, 2=1, 3=0 [...]

or

| MB_HARDMAX: 0=off, 1=on, 2=on, 3=off [...]

Quoted 28 lines

For CMAX, maximum cache capacity, there is an equivalent control SOFTLIM, which behaves as HARDLIM except the meaning of the bit is reversed. We can just use a consistent name in s/w though.

Quoted 24 lines
Quoted 16 lines

On an x86 system:

info

├── MB
│   ├── resource_schemata
│   │   ├── MB
│   │   └── MB_MAX (Finer grained MB, more below)
│   └── scope
└── MB_REGION
     ├── resource_schemata
     │   └── MB_REGION
     └── scope
Quoted 1 line

Do you think this helps?

"REGION" is not a new scope but instead region-aware MBA is controlled and manages bandwidth at L3 scope. Combine that with up to (currently) four regions each with three controls I find an interface like above potentially confusing to document in an intuitive way. Unless you are perhaps saying that we should introduce a new separate "MB_REGION" L3 scope resource (so let resctrl support multiple "MB" resources at the same scope?) and then *it* contains the twelve new controls within its resource_schemata directory?

Quoted 6 lines
Quoted 3 lines

Since the region-aware controls are orthogonal to the MSR based legacy control resctrl would still need a way for user space to switch from one to the other which implies a dependency between "MB" and "MB_REGION" that is not presented in above hierarchy.

OK. The /sys/fs/resctrl/info/MB/schemata/mode and the MB_REGION controls a child of MB you described previously seem s better fit than what I suggested.

ok, I'll keep following that.

Quoted 21 lines
Quoted 6 lines

I think I am missing quite a bit here as I try to navigate an interface so different from what we have discussed so far.
I would like to explore with more detail how this interface can handle the different scenarios we have discussed so far.

Certainly, I don't think we have got to the bottom of this yet.

Quoted 5 lines

The other x86 feature to consider is AMD's upcoming "Global" MBA/SMBA that exposes memory bandwidth allocation in "groups of L3" that I understand could usually be mapped to NODE scope (but it remains controlled at L3 scope), except for one configuration where it is "SYSTEM"(?) scope.
Ref.: https://lore.kernel.org/lkml/8f77f498b1c77fa8fd8f5d5687f03ae598068544.1776980182.git.babu.moger@amd.com/

Hmmm, I'm not sure that the scope can be considered to be NODE scope for GMBA. To me it seems to be accidental that it maps to the NUMA node but really the scope is just a grouping of L3 instances. For a control to NUMA scope I would expect the resctrl domains to go offline and online in sync with the NUMA nodes. For GMBA it looks like it would just going offline/online based on whether any of the CPUs and so L3 instances in the group are online. Am I correct here?

Quoted 8 lines
Quoted 5 lines

Assuming the domains are on L3 groups rather than NUMA also changes which end of the link the traffic is regulated and so how cross-NUMA traffic behaves differently. If the domain is an L3 group then a task running on a CPU affine to that L3 group won't be throttled unless that particular domain is throttled but with NUMA node domains it may be throttled if it has traffic going to that domain.

I'll defer to Babu for accurate answers about this hardware capability.

To me, Global MBA should be considered a NODE-scoped resource. In some configurations it may appear as SYSTEM-scoped, but that is effectively equivalent to a single-node encompassing the entire system. In such cases, there is only one schemata entry controlling the whole system.

Yes, multiple L3 instances are grouped together to form a NODE. Internally, programming is still performed at the L3 level, but that implementation detail can be hidden from users and does not need to be exposed through the interface.

Thanks
Babu

Quoted 55 lines
Quoted 32 lines
Quoted 8 lines
Quoted 6 lines

This also brings another question. On MPAM systems the 'MB_MAX' is backed by the same MSC h/w as MB but it exposed a different interface to the user. If I understand correctly intel have an option to have finer grained control of MB (delay) as well and so it would make sense to use a common name rather than just going for the MPAM centric name of MB_MAX.

Apologies but I was not able to parse above.

Ok, let me try to explain again (although it's probably not what we want to do). My intent here was to try and explore whether we can reuse naming and controls across architectures in the same way we already have for the L2/L3 cache portion bitmap and the existing MB control.

To quote from a previous mail of yours:
https://lore.kernel.org/lkml/[email protected]/
"""
For example, on an MPAM system (if I understand correctly) the user may see:
info/

   └── MB/
       └── resource_schemata/
           ├── MB/
           │   └── MB_MAX/
           └── MB_MIN/

Compared with a possible implementation on Intel that looks like:
info/

   └── MB/
       └── resource_schemata/
           ├── MB/
           │   └── MB_OPT/
           ├── MB_MAX/
           └── MB_MIN/
Quoted 6 lines

"""

In the two setups MPAM MB_MAX and intel MB_OPT play the same role, a finer grained control of the legacy MB control. I was thinking these could share a name (MB-PRECISE), but it probably doesn't make sense as MB_OPT and MB_MAX have different relationships to MB_MIN.

Apologies, I neglected to follow up on this after I clarified internally which RDT control should actually be used to emulate the percentage based control. My original thoughts that you highlight above is not correct the RDT also plans to use the "MAX" underlying hardware control for the percentage based MB control. With this RDT and MPAM should look more similar with RDT simplified (by dropping the region-aware terms) as:

info/
└── MB/
└── resource_schemata/
├── MB/
│ └── MB_MAX/
├── MB_OPT/
└── MB_MIN/

Reinette

maintainer
#84 lore raw

Hi Babu,

Quoted 232 lines · Babu Moger

Hi Ben/Reinette,

Quoted 220 lines · Reinette Chatre

Hi Ben,

Quoted 2 lines · Ben Horgan
Quoted 1 line · Reinette Chatre

On 7/20/26 6:30 AM, Ben Horgan wrote:

...>

Quoted 24 lines

The former, info/ contains a directory for each allocation scope of each resource.

info

├── L2
│   ├── resource_schemata
│   │   ├── L2
│   │   ├── L2_CMAX
│   │   └── L2_CMIN
│   └── scope : L2
├── L3
│   ├── resource_schemata
│   │   ├── L3
│   │   ├── L3_CMAX
│   │   └── L3_CMIN
│   └── scope : L3
├── MB
│   ├── resource_schemata
│   │   ├── MB
│   │   │   └── MB_MAX
│   │   ├── MB_MIN
│   │   ├── MB_PBM
│   │   └── MB_PROP
Quoted 9 lines
│   └── scope : L3
└── MB_NODE
     ├── resource_schemata
     │   ├── MB_NODE_MAX
     │   ├── MB_NODE_MIN
     │   ├── MB_NODE_PBM
     │   └── MB_NODE_PROP
     └── scope : NUMA NODE

At first glance this looks good to me. As you highlight below the nuances of how "Global S/MBA" can fit
in here still needs to be worked out. One thing that resctrl may need to highlight when documenting this
new capability is that while historically a resource had a matching schemata entry, this is no longer the
case. As you show above some systems may have MB_NODE resource but no MB_NODE control while I expect that
"Global MBA" (if it adopts this) may indeed have a MB_NODE resource with a MB_NODE control.

Quoted 12 lines

Please just consider this a mistake. MB_MAX should be a child of MB.

info/

├── MB
│   ├── resource_schemata
│   │   └── MB
│   │       └── MB_MAX
│   └── scope
└── MB_NODE
     ├── resource_schemata
     │   └── MB_NODE_MAX (Changed from MB_NODE)
     └── scope

ack.

Quoted 7 lines
Quoted 6 lines

When thinking about MPAM, what would the underlying hardware control of "MB_NODE" be? It looks from above that it would either start out by itself having the properties of the underlying "MAX" control or is the plan to have it be a percentage based control backed by the underlying "MAX" hardware control?

Quoted 5 lines

The underlying hardware of MB_NODE would be essentially the same hardware as that backing MB_MAX, but at a different location in the SoC, at the memory controller rather than in the L3.

To correct myself slightly, I don't think we should have a control called MB_NODE, rather, it should be MB_NODE_MAX.

ack.

Quoted 4 lines

My understanding of previous discussions is that <resource_type>_<scope>_<control_type> it the pattern for control names and the pattern for resource names being <resource_type>_<scope>. (allowing for <scope> or <control_type> being missing to match existing naming.)

This is where discussions are from my view also.

Quoted 3 lines

I don't think we should introduce more percent based controls and for new controls we can introduce a new format to describe them. Perhaps just the positive integer with a resolution supplied in info/

No, we should not introduce more percentage based controls per se. With the new schema format a percentage based control is just a variant of a proportional scalar control.

Quoted 13 lines

as discussed previously. Although, I have been pondering on whether we can do a bit better.

We could use hexadecimal point based format for controls which are a proportion of a resource and have a resolution which is a power of 2. The advantage of this is that the meaning of the value is independent of the granularity of the control (number of parts).

0 is represented as 0x0
1 as                0x1
1/2 as              0x0.8
7/256 as            0x0.07
1/2**28             0x0.00000001
etc

Quoted 6 lines

This maps well to the MPAM fixed-point fraction point format without having the weirdness of having values forced to 1 or 0 not being really 0. These MPAM h/w oddities can be hidden just by using the mbw_min mbw_max of a control. In MPAM this could be used in CMIN, CMAX, MB_MAX, MB_MIN and I would hope this would be useful for other architectures too. I am preparing some RFC patches on top of your PoC for consideration of this idea and to explore some of the proposals discussed relating to generic schemata and how they land in practice from the MPAM side.

I'm going to stand with Dave Martin [1] on this point with a preference to avoid floating point in resctrl input and output.
Could you please elaborate where values are forced to 1 or 0? The new schema format was intentionally created to *avoid* rounding errors (and parsing complexity).

[1] https://lore.kernel.org/lkml/[email protected]/

Quoted 11 lines
Quoted 4 lines

I also understand MPAM to support more memory bandwidth controls ("MIN", "HARDMAX"/"HARDLIM", etc.).
Do you envision them to exist within info/MB/resource_schemata/<control> as well as within info/MB_NODE/resource_schemata/<control>?

Yes, at least for MIN, see the info/ tree above. For HARDLIM, perhaps, but HARDLIM has the added complications that it is a property of the MBW_MAX control and that it may be configurable for each PARTID or a fixed property of the h/w. When HARDLIM is configurable the control name could be of the form <resource_type>_<scope>_<control>_<control_param> where <control_param> is HARDLIM and the full name for the HARDLIM configuration on the MB_NODE resource is MB_NODE_MAX_HARDLIM. There can also be an info/<resource>/resource_schemata/<control>/lim file which has values, soft, hard, configurable.

ack. HARDLIM sounds like it would be a new control type. Perhaps a "boolean" type for which new control
files need to be decided on? Sounds like you are headed in this direction and already have one control file
in mind for this new type.

With this in mind the control could be built on top of what is being developed at the moment, possibly
be presented to user space following Dave Martin's suggestion in
https://lore.kernel.org/lkml/[email protected]/:

  | MB_HARDMAX: 0=0, 1=1, 2=1, 3=0 [...]

  or

  | MB_HARDMAX: 0=off, 1=on, 2=on, 3=off [...]

Quoted 32 lines

For CMAX, maximum cache capacity, there is an equivalent control SOFTLIM, which behaves as HARDLIM except the meaning of the bit is reversed. We can just use a consistent name in s/w though.

Quoted 28 lines
Quoted 16 lines

On an x86 system:

info

├── MB
│   ├── resource_schemata
│   │   ├── MB
│   │   └── MB_MAX (Finer grained MB, more below)
│   └── scope
└── MB_REGION
     ├── resource_schemata
     │   └── MB_REGION
     └── scope
Quoted 1 line

Do you think this helps?

"REGION" is not a new scope but instead region-aware MBA is controlled and manages bandwidth at L3 scope.
Combine that with up to (currently) four regions each with three controls I find an interface like above
potentially confusing to document in an intuitive way. Unless you are perhaps saying that we should introduce
a new separate  "MB_REGION" L3 scope resource (so let resctrl support multiple "MB" resources at the same
scope?) and then *it* contains the twelve new controls within its resource_schemata directory?

Quoted 8 lines
Quoted 5 lines

Since the region-aware controls are orthogonal to the MSR based legacy control resctrl would still need a
way for user space to switch from one to the other which implies a dependency between "MB" and "MB_REGION"
that is not presented in above hierarchy.

OK. The /sys/fs/resctrl/info/MB/schemata/mode and the MB_REGION controls a child of MB you described previously seem s better fit than what I suggested.

ok, I'll keep following that.

Quoted 26 lines
Quoted 8 lines

I think I am missing quite a bit here as I try to navigate an interface so different from what we have
discussed so far.
I would like to explore with more detail how this interface can handle the different scenarios we have
discussed so far.

Certainly, I don't think we have got to the bottom of this yet.

Quoted 8 lines

The other x86 feature to consider is AMD's upcoming "Global" MBA/SMBA that exposes memory bandwidth allocation
in "groups of L3" that I understand could usually be mapped to NODE scope (but it remains controlled at L3 scope),
except for one configuration where it is "SYSTEM"(?) scope.
Ref.: https://lore.kernel.org/
lkml/8f77f498b1c77fa8fd8f5d5687f03ae598068544.1776980182.git.babu.moger@amd.com/

Hmmm, I'm not sure that the scope can be considered to be NODE scope for GMBA. To me it seems to be accidental that it maps to the NUMA node but really the scope is just a grouping of L3 instances. For a control to NUMA scope I would expect the resctrl domains to go offline and online in sync with the NUMA nodes. For GMBA it looks like it would just going offline/online based on whether any of the CPUs and so L3 instances in the group are online. Am I correct here?

Quoted 8 lines
Quoted 5 lines

Assuming the domains are on L3 groups rather than NUMA also changes which end of the link the traffic is regulated and so how cross-NUMA traffic behaves differently. If the domain is an L3 group then a task running on a CPU affine to that L3 group won't be throttled unless that particular domain is throttled but with NUMA node domains it may be throttled if it has traffic going to that domain.

I'll defer to Babu for accurate answers about this hardware capability.

Quoted 7 lines

To me, Global MBA should be considered a NODE-scoped resource. In some configurations it may appear as SYSTEM-scoped, but that is effectively equivalent to a single-node encompassing the entire system. In such cases, there is only one schemata entry controlling the whole system.

Yes, multiple L3 instances are grouped together to form a NODE. Internally, programming is still performed at the L3 level, but that implementation detail can be hidden from users and does not need to be exposed through the interface.

We seem to have two things that can both, somewhat reasonably, be called NODE scope in the resctrl user interface but the behaviour required for an MPAM system and an AMD system appears different from the point of view of lifecycle of the resctrl domain.

For MPAM NUMA scope the MSC instance (MPAM hardware interface) is at the memory controller and so goes on and offline based on whether the NUMA node is offline or online. For AMD NUMA scope it looks to me that the lifecycle of the resctrl domains would be tied to the CPUs associated with the NUMA node. To me it does seem odd that a control with a domain associated with an offline NUMA node can continue to throttle (cross-NUMA) traffic.

Is there any GLBE Control Domain ID or similar that is exposed to the user, e.g. is sysfs, or is this just implicitly the NUMA id?

Thanks,

Ben

reviewer
#85 lore raw

Hi Ben,

Quoted 50 lines · Ben Horgan

Hi Babu,

Quoted 35 lines · Babu Moger

Hi Ben/Reinette,

Quoted 23 lines · Reinette Chatre

Hi Ben,

Quoted 2 lines · Ben Horgan
Quoted 1 line · Reinette Chatre

On 7/20/26 6:30 AM, Ben Horgan wrote:

...>

Quoted 17 lines

The former, info/ contains a directory for each allocation scope of each resource.

Quoted 8 lines

The other x86 feature to consider is AMD's upcoming "Global" MBA/SMBA that exposes memory bandwidth allocation
in "groups of L3" that I understand could usually be mapped to NODE scope (but it remains controlled at L3 scope),
except for one configuration where it is "SYSTEM"(?) scope.
Ref.: https://lore.kernel.org/
lkml/8f77f498b1c77fa8fd8f5d5687f03ae598068544.1776980182.git.babu.moger@amd.com/

Hmmm, I'm not sure that the scope can be considered to be NODE scope for GMBA. To me it seems to be accidental that it maps to the NUMA node but really the scope is just a grouping of L3 instances. For a control to NUMA scope I would expect the resctrl domains to go offline and online in sync with the NUMA nodes. For GMBA it looks like it would just going offline/online based on whether any of the CPUs and so L3 instances in the group are online. Am I correct here?

Quoted 8 lines
Quoted 5 lines

Assuming the domains are on L3 groups rather than NUMA also changes which end of the link the traffic is regulated and so how cross-NUMA traffic behaves differently. If the domain is an L3 group then a task running on a CPU affine to that L3 group won't be throttled unless that particular domain is throttled but with NUMA node domains it may be throttled if it has traffic going to that domain.

I'll defer to Babu for accurate answers about this hardware capability.

Quoted 7 lines

To me, Global MBA should be considered a NODE-scoped resource. In some configurations it may appear as SYSTEM-scoped, but that is effectively equivalent to a single-node encompassing the entire system. In such cases, there is only one schemata entry controlling the whole system.

Yes, multiple L3 instances are grouped together to form a NODE. Internally, programming is still performed at the L3 level, but that implementation detail can be hidden from users and does not need to be exposed through the interface.

We seem to have two things that can both, somewhat reasonably, be called NODE scope in the resctrl user interface but the behaviour required for an MPAM system and an AMD system appears different from the point of view of lifecycle of the resctrl domain.

Quoted 9 lines

For MPAM NUMA scope the MSC instance (MPAM hardware interface) is at the memory controller and so goes on and offline based on whether the NUMA node is offline or online. For AMD NUMA scope it looks to me that the lifecycle of the resctrl domains would be tied to the CPUs associated with the NUMA node. To me it does seem odd that a control with a domain associated with an offline NUMA node can continue to throttle (cross-NUMA) traffic.

Is there any GLBE Control Domain ID or similar that is exposed to the user, e.g. is sysfs, or is this just implicitly the NUMA id?

Yes, the GLBE Control Domain ID is exposed to the user. It is essentially equivalent to the NUMA ID.

Thanks
Babu

maintainer
#86 lore raw

Hi Babu,

Quoted 62 lines · Babu Moger

Hi Ben,

Quoted 59 lines · Ben Horgan

Hi Babu,

Quoted 38 lines · Babu Moger

Hi Ben/Reinette,

Quoted 25 lines · Reinette Chatre

Hi Ben,

Quoted 2 lines · Ben Horgan
Quoted 1 line · Reinette Chatre

On 7/20/26 6:30 AM, Ben Horgan wrote:

...>

Quoted 19 lines

The former, info/ contains a directory for each allocation scope of each resource.

Quoted 8 lines

The other x86 feature to consider is AMD's upcoming "Global" MBA/SMBA that exposes memory bandwidth allocation
in "groups of L3" that I understand could usually be mapped to NODE scope (but it remains controlled at L3 scope),
except for one configuration where it is "SYSTEM"(?) scope.
Ref.: https://lore.kernel.org/
lkml/8f77f498b1c77fa8fd8f5d5687f03ae598068544.1776980182.git.babu.moger@amd.com/

Hmmm, I'm not sure that the scope can be considered to be NODE scope for GMBA. To me it seems to be
accidental that it maps to the NUMA node but really the scope is just a grouping of L3 instances. For a control to NUMA scope I would expect the resctrl domains to go offline and online in sync with
the NUMA nodes. For GMBA it looks like it would just going offline/online based on whether any of the CPUs and so L3 instances in the group are online. Am I correct here?

Quoted 9 lines
Quoted 6 lines

Assuming the domains are on L3 groups rather than NUMA also changes which end of the link the traffic is regulated and so how cross-NUMA traffic behaves differently. If the domain is an L3 group
then a task running on a CPU affine to that L3 group won't be throttled unless that particular domain is throttled but with NUMA node domains it may be throttled if it has traffic going to that domain.

I'll defer to Babu for accurate answers about this hardware capability.

Quoted 7 lines

To me, Global MBA should be considered a NODE-scoped resource. In some configurations it may appear as SYSTEM-scoped, but that is effectively equivalent to a single-node encompassing the entire system. In such cases, there is only one schemata entry controlling the whole system.

Yes, multiple L3 instances are grouped together to form a NODE. Internally, programming is still performed at the L3 level, but that implementation detail can be hidden from users and does not need to be exposed through the interface.

We seem to have two things that can both, somewhat reasonably, be called NODE scope in the resctrl user interface but the behaviour required for an MPAM system and an AMD system appears different from the point of view of lifecycle of the resctrl domain.

For MPAM NUMA scope the MSC instance (MPAM hardware interface) is at the memory controller and so goes on and offline based on whether the NUMA node is offline or online. For AMD NUMA scope it looks to me that the lifecycle of the resctrl domains would be tied to the CPUs associated with the NUMA node. To me it does seem odd that a control with a domain associated with an offline NUMA node can continue to throttle (cross-NUMA) traffic.

Quoted 4 lines
Quoted 3 lines

Is there any GLBE Control Domain ID or similar that is exposed to the user, e.g. is sysfs, or is this just implicitly the NUMA id?

Yes, the GLBE Control Domain ID is exposed to the user. It is essentially equivalent to the NUMA ID.

What's the on/off lifecycle of these nodes? Does it follow the NUMA lifecycle as managed by the NUMA node notifiers documented in Documentation/core-api/memory-hotplug.rst or is it just linked to the cpu hotplug as is done currently for the resctrl cache based domains.

Thanks,

Ben

Quoted 3 lines

Thanks
Babu

reviewer
#87 lore raw

Hi Ben,

Quoted 65 lines · Ben Horgan

Hi Babu,

Quoted 62 lines · Babu Moger

Hi Ben,

Quoted 59 lines · Ben Horgan

Hi Babu,

Quoted 38 lines · Babu Moger

Hi Ben/Reinette,

Quoted 25 lines · Reinette Chatre

Hi Ben,

Quoted 2 lines · Ben Horgan
Quoted 1 line · Reinette Chatre

On 7/20/26 6:30 AM, Ben Horgan wrote:

...>

Quoted 19 lines

The former, info/ contains a directory for each allocation scope of each resource.

Quoted 8 lines

The other x86 feature to consider is AMD's upcoming "Global" MBA/SMBA that exposes memory bandwidth allocation
in "groups of L3" that I understand could usually be mapped to NODE scope (but it remains controlled at L3 scope),
except for one configuration where it is "SYSTEM"(?) scope.
Ref.: https://lore.kernel.org/
lkml/8f77f498b1c77fa8fd8f5d5687f03ae598068544.1776980182.git.babu.moger@amd.com/

Hmmm, I'm not sure that the scope can be considered to be NODE scope for GMBA. To me it seems to be
accidental that it maps to the NUMA node but really the scope is just a grouping of L3 instances. For a control to NUMA scope I would expect the resctrl domains to go offline and online in sync with
the NUMA nodes. For GMBA it looks like it would just going offline/online based on whether any of the CPUs and so L3 instances in the group are online. Am I correct here?

Quoted 9 lines
Quoted 6 lines

Assuming the domains are on L3 groups rather than NUMA also changes which end of the link the traffic is regulated and so how cross-NUMA traffic behaves differently. If the domain is an L3 group
then a task running on a CPU affine to that L3 group won't be throttled unless that particular domain is throttled but with NUMA node domains it may be throttled if it has traffic going to that domain.

I'll defer to Babu for accurate answers about this hardware capability.

Quoted 7 lines

To me, Global MBA should be considered a NODE-scoped resource. In some configurations it may appear as SYSTEM-scoped, but that is effectively equivalent to a single-node encompassing the entire system. In such cases, there is only one schemata entry controlling the whole system.

Yes, multiple L3 instances are grouped together to form a NODE. Internally, programming is still performed at the L3 level, but that implementation detail can be hidden from users and does not need to be exposed through the interface.

We seem to have two things that can both, somewhat reasonably, be called NODE scope in the resctrl user interface but the behaviour required for an MPAM system and an AMD system appears different from the point of view of lifecycle of the resctrl domain.

For MPAM NUMA scope the MSC instance (MPAM hardware interface) is at the memory controller and so goes on and offline based on whether the NUMA node is offline or online. For AMD NUMA scope it looks to me that the lifecycle of the resctrl domains would be tied to the CPUs associated with the NUMA node. To me it does seem odd that a control with a domain associated with an offline NUMA node can continue to throttle (cross-NUMA) traffic.

Quoted 9 lines
Quoted 4 lines
Quoted 3 lines

Is there any GLBE Control Domain ID or similar that is exposed to the user, e.g. is sysfs, or is this just implicitly the NUMA id?

Yes, the GLBE Control Domain ID is exposed to the user. It is essentially equivalent to the NUMA ID.

What's the on/off lifecycle of these nodes? Does it follow the NUMA lifecycle as managed by the NUMA node notifiers documented in Documentation/core-api/memory-hotplug.rst or is it just linked to the cpu hotplug as is done currently for the resctrl cache based domains.

It will follow the CPU hotplug lifecycle, similar to how it is currently handled for the resctrl cache-based domains.

Thanks,
Babu

maintainer
#88 lore raw

Hi Babu,

Quoted 77 lines · Babu Moger

Hi Ben,

Quoted 74 lines · Ben Horgan

Hi Babu,

Quoted 71 lines · Babu Moger

Hi Ben,

Quoted 68 lines · Ben Horgan

Hi Babu,

Quoted 46 lines · Babu Moger

Hi Ben/Reinette,

Quoted 35 lines · Reinette Chatre

Hi Ben,

Quoted 2 lines · Ben Horgan
Quoted 1 line · Reinette Chatre

On 7/20/26 6:30 AM, Ben Horgan wrote:

...>

Quoted 21 lines

The former, info/ contains a directory for each allocation scope of each resource.

Quoted 8 lines

The other x86 feature to consider is AMD's upcoming "Global" MBA/SMBA that exposes memory bandwidth allocation
in "groups of L3" that I understand could usually be mapped to NODE scope (but it remains controlled at L3 scope),
except for one configuration where it is "SYSTEM"(?) scope.
Ref.: https://lore.kernel.org/
lkml/8f77f498b1c77fa8fd8f5d5687f03ae598068544.1776980182.git.babu.moger@amd.com/

Hmmm, I'm not sure that the scope can be considered to be NODE scope for GMBA. To me it seems to be
accidental that it maps to the NUMA node but really the scope is just a grouping of L3 instances.
For a control to NUMA scope I would expect the resctrl domains to go offline and online in sync with
the NUMA nodes. For GMBA it looks like it would just going offline/online based on whether any of
the CPUs and so L3 instances in the group are online. Am I correct here?

Quoted 7 lines

Assuming the domains are on L3 groups rather than NUMA also changes which end of the link the traffic is regulated and so how cross-NUMA traffic behaves differently. If the domain is an L3 group
then a task running on a CPU affine to that L3 group won't be throttled unless that particular domain is throttled but with NUMA node domains it may be throttled if it has traffic going to that
domain.

Quoted 2 lines

I'll defer to Babu for accurate answers about this hardware capability.

To me, Global MBA should be considered a NODE-scoped resource. In some configurations it may appear
as SYSTEM-scoped, but that is effectively equivalent to a single-node encompassing the entire system. In such cases, there is only one schemata entry controlling the whole system.

Quoted 4 lines

Yes, multiple L3 instances are grouped together to form a NODE. Internally, programming is still performed at the L3 level, but that implementation detail can be hidden from users and does not need
to be exposed through the interface.

We seem to have two things that can both, somewhat reasonably, be called NODE scope in the resctrl user interface but the behaviour required for an MPAM system and an AMD system appears different from the point of view of lifecycle of the resctrl domain.

For MPAM NUMA scope the MSC instance (MPAM hardware interface) is at the memory controller and so goes on and offline based on whether the NUMA node is offline or online. For AMD NUMA scope it looks
to me that the lifecycle of the resctrl domains would be tied to the CPUs associated with the NUMA node. To me it does seem odd that a control with a domain associated with an offline NUMA node can continue to throttle (cross-NUMA) traffic.

Is there any GLBE Control Domain ID or similar that is exposed to the user, e.g. is sysfs, or is this just implicitly the NUMA id?

Quoted 8 lines
Quoted 6 lines
Quoted 1 line

Yes, the GLBE Control Domain ID is exposed to the user. It is essentially equivalent to the NUMA ID.

What's the on/off lifecycle of these nodes? Does it follow the NUMA lifecycle as managed by the NUMA node notifiers documented in Documentation/core-api/memory-hotplug.rst or is it just linked to the cpu hotplug as is done currently for the resctrl cache based domains.

It will follow the CPU hotplug lifecycle, similar to how it is currently handled for the resctrl cache-based domains.

Ok. MPAM MSC are associated with the memory controllers and so MPAM controls that have NUMA scope will follow the NUMA memory hotplug lifecycle. This points to them being different resources. Do you have any thoughts on how we would handle this difference?

Quoted 3 lines

Thanks,
Babu

reviewer
#89 lore raw

Hi Ben,

Quoted 86 lines · Ben Horgan

Hi Babu,

Quoted 83 lines · Babu Moger

Hi Ben,

Quoted 80 lines · Ben Horgan

Hi Babu,

Quoted 68 lines · Babu Moger

Hi Ben,

Quoted 65 lines · Ben Horgan

Hi Babu,

Quoted 46 lines · Babu Moger

Hi Ben/Reinette,

Quoted 35 lines · Reinette Chatre

Hi Ben,

Quoted 2 lines · Ben Horgan
Quoted 1 line · Reinette Chatre

On 7/20/26 6:30 AM, Ben Horgan wrote:

...>

Quoted 21 lines

The former, info/ contains a directory for each allocation scope of each resource.

Quoted 8 lines

The other x86 feature to consider is AMD's upcoming "Global" MBA/SMBA that exposes memory bandwidth allocation
in "groups of L3" that I understand could usually be mapped to NODE scope (but it remains controlled at L3 scope),
except for one configuration where it is "SYSTEM"(?) scope.
Ref.: https://lore.kernel.org/
lkml/8f77f498b1c77fa8fd8f5d5687f03ae598068544.1776980182.git.babu.moger@amd.com/

Hmmm, I'm not sure that the scope can be considered to be NODE scope for GMBA. To me it seems to be
accidental that it maps to the NUMA node but really the scope is just a grouping of L3 instances.
For a control to NUMA scope I would expect the resctrl domains to go offline and online in sync with
the NUMA nodes. For GMBA it looks like it would just going offline/online based on whether any of
the CPUs and so L3 instances in the group are online. Am I correct here?

Quoted 7 lines

Assuming the domains are on L3 groups rather than NUMA also changes which end of the link the traffic is regulated and so how cross-NUMA traffic behaves differently. If the domain is an L3 group
then a task running on a CPU affine to that L3 group won't be throttled unless that particular domain is throttled but with NUMA node domains it may be throttled if it has traffic going to that
domain.

Quoted 2 lines

I'll defer to Babu for accurate answers about this hardware capability.

To me, Global MBA should be considered a NODE-scoped resource. In some configurations it may appear
as SYSTEM-scoped, but that is effectively equivalent to a single-node encompassing the entire system. In such cases, there is only one schemata entry controlling the whole system.

Quoted 4 lines

Yes, multiple L3 instances are grouped together to form a NODE. Internally, programming is still performed at the L3 level, but that implementation detail can be hidden from users and does not need
to be exposed through the interface.

We seem to have two things that can both, somewhat reasonably, be called NODE scope in the resctrl user interface but the behaviour required for an MPAM system and an AMD system appears different from the point of view of lifecycle of the resctrl domain.

For MPAM NUMA scope the MSC instance (MPAM hardware interface) is at the memory controller and so goes on and offline based on whether the NUMA node is offline or online. For AMD NUMA scope it looks
to me that the lifecycle of the resctrl domains would be tied to the CPUs associated with the NUMA node. To me it does seem odd that a control with a domain associated with an offline NUMA node can continue to throttle (cross-NUMA) traffic.

Quoted 4 lines
Quoted 3 lines

Is there any GLBE Control Domain ID or similar that is exposed to the user, e.g. is sysfs, or is this just implicitly the NUMA id?

Yes, the GLBE Control Domain ID is exposed to the user. It is essentially equivalent to the NUMA ID.

What's the on/off lifecycle of these nodes? Does it follow the NUMA lifecycle as managed by the NUMA node notifiers documented in Documentation/core-api/memory-hotplug.rst or is it just linked to the cpu hotplug as is done currently for the resctrl cache based domains.

Quoted 7 lines
Quoted 2 lines

It will follow the CPU hotplug lifecycle, similar to how it is currently handled for the resctrl cache-based domains.

Ok. MPAM MSC are associated with the memory controllers and so MPAM controls that have NUMA scope will follow the NUMA memory hotplug lifecycle. This points to them being different resources. Do you have any thoughts on how we would handle this difference?

Currently, during CPU hotplug events, we call domain_add_cpu_ctrl(), which sets up the domain, CPU mask, and initial configuration values. The same approach will be used for Global MBA.

I'm not yet sure what needs to be done for MPAM memory hotplug. It appears there will be something different. One possibility is to introduce a flag to distinguish between different types of hotplug events(memory or CPU).

Thanks
Babu

author maintainer
#90 lore raw
In reply to Ben Horgan

Hi Ben and Babu,

Quoted 72 lines · Ben Horgan
Quoted 71 lines · Babu Moger
Quoted 70 lines · Ben Horgan
Quoted 60 lines · Babu Moger
Quoted 59 lines · Ben Horgan
Quoted 42 lines · Babu Moger
Quoted 33 lines · Reinette Chatre
Quoted 2 lines · Ben Horgan
Quoted 1 line · Reinette Chatre

On 7/20/26 6:30 AM, Ben Horgan wrote:

...>

Quoted 21 lines

The former, info/ contains a directory for each allocation scope of each resource.

Quoted 8 lines

The other x86 feature to consider is AMD's upcoming "Global" MBA/SMBA that exposes memory bandwidth allocation
in "groups of L3" that I understand could usually be mapped to NODE scope (but it remains controlled at L3 scope),
except for one configuration where it is "SYSTEM"(?) scope.
Ref.: https://lore.kernel.org/
lkml/8f77f498b1c77fa8fd8f5d5687f03ae598068544.1776980182.git.babu.moger@amd.com/

Hmmm, I'm not sure that the scope can be considered to be NODE scope for GMBA. To me it seems to be
accidental that it maps to the NUMA node but really the scope is just a grouping of L3 instances.
For a control to NUMA scope I would expect the resctrl domains to go offline and online in sync with
the NUMA nodes. For GMBA it looks like it would just going offline/online based on whether any of
the CPUs and so L3 instances in the group are online. Am I correct here?

Quoted 7 lines

Assuming the domains are on L3 groups rather than NUMA also changes which end of the link the traffic is regulated and so how cross-NUMA traffic behaves differently. If the domain is an L3 group
then a task running on a CPU affine to that L3 group won't be throttled unless that particular domain is throttled but with NUMA node domains it may be throttled if it has traffic going to that
domain.

Quoted 2 lines

I'll defer to Babu for accurate answers about this hardware capability.

To me, Global MBA should be considered a NODE-scoped resource. In some configurations it may appear
as SYSTEM-scoped, but that is effectively equivalent to a single-node encompassing the entire system. In such cases, there is only one schemata entry controlling the whole system.

Quoted 4 lines

Yes, multiple L3 instances are grouped together to form a NODE. Internally, programming is still performed at the L3 level, but that implementation detail can be hidden from users and does not need
to be exposed through the interface.

We seem to have two things that can both, somewhat reasonably, be called NODE scope in the resctrl user interface but the behaviour required for an MPAM system and an AMD system appears different from the point of view of lifecycle of the resctrl domain.

For MPAM NUMA scope the MSC instance (MPAM hardware interface) is at the memory controller and so goes on and offline based on whether the NUMA node is offline or online. For AMD NUMA scope it looks
to me that the lifecycle of the resctrl domains would be tied to the CPUs associated with the NUMA node. To me it does seem odd that a control with a domain associated with an offline NUMA node can continue to throttle (cross-NUMA) traffic.

Quoted 4 lines
Quoted 3 lines

Is there any GLBE Control Domain ID or similar that is exposed to the user, e.g. is sysfs, or is this just implicitly the NUMA id?

Yes, the GLBE Control Domain ID is exposed to the user. It is essentially equivalent to the NUMA ID.

What's the on/off lifecycle of these nodes? Does it follow the NUMA lifecycle as managed by the NUMA node notifiers documented in Documentation/core-api/memory-hotplug.rst or is it just linked to the cpu hotplug as is done currently for the resctrl cache based domains.

Quoted 6 lines
Quoted 2 lines

It will follow the CPU hotplug lifecycle, similar to how it is currently handled for the resctrl cache-based domains.

Ok. MPAM MSC are associated with the memory controllers and so MPAM controls that have NUMA scope will follow the NUMA memory hotplug lifecycle. This points to them being different resources. Do you have any thoughts on how we would handle this difference?

If I understand correctly, on an AMD GLBE system, even though it is "NUMA node scoped" it does not support memory bandwidth allocation for a NUMA node that is online but all its CPUs are offline. This is because GLBE is essentially L3 MBA that allows to set limits across multiple L3 domains. I think this is a good match for a control associated with the legacy MB resource that just has a different scope of allocation but how to do so without creating confusion with a "real" NUMA bandwidth allocation is not clear to me since it may end up looking like:

GLBE (make clear allocation is at L3 scope but domains are node scoped):
info/

└── MB/
    └── schemata/
        ├── MB/      /* scope of domain ID = L3   */
        └── MB_NODE/ /* scope of domain ID = node */

MPAM MSC (make clear allocation is at node scope:
info/

├── MB/
│   └── schemata/
│       └── MB/      /* scope of domain ID = L3   */
└── MB_NODE/
    └── schemata/
        └── MB_NODE/ /* scope of domain ID = node  */

Any suggestions?

I do not see how these can be considered different resources though. To me it looks like different controls operating at different scope for the same ("memory bandwidth") resource?

Reinette

maintainer
#91 lore raw

Hi Reinette,

Quoted 95 lines · Reinette Chatre

Hi Ben and Babu,

Quoted 72 lines · Ben Horgan
Quoted 71 lines · Babu Moger
Quoted 70 lines · Ben Horgan
Quoted 60 lines · Babu Moger
Quoted 59 lines · Ben Horgan
Quoted 42 lines · Babu Moger
Quoted 33 lines · Reinette Chatre
Quoted 2 lines · Ben Horgan
Quoted 1 line · Reinette Chatre

On 7/20/26 6:30 AM, Ben Horgan wrote:

...>

Quoted 21 lines

The former, info/ contains a directory for each allocation scope of each resource.

Quoted 8 lines

The other x86 feature to consider is AMD's upcoming "Global" MBA/SMBA that exposes memory bandwidth allocation
in "groups of L3" that I understand could usually be mapped to NODE scope (but it remains controlled at L3 scope),
except for one configuration where it is "SYSTEM"(?) scope.
Ref.: https://lore.kernel.org/
lkml/8f77f498b1c77fa8fd8f5d5687f03ae598068544.1776980182.git.babu.moger@amd.com/

Hmmm, I'm not sure that the scope can be considered to be NODE scope for GMBA. To me it seems to be
accidental that it maps to the NUMA node but really the scope is just a grouping of L3 instances.
For a control to NUMA scope I would expect the resctrl domains to go offline and online in sync with
the NUMA nodes. For GMBA it looks like it would just going offline/online based on whether any of
the CPUs and so L3 instances in the group are online. Am I correct here?

Quoted 7 lines

Assuming the domains are on L3 groups rather than NUMA also changes which end of the link the traffic is regulated and so how cross-NUMA traffic behaves differently. If the domain is an L3 group
then a task running on a CPU affine to that L3 group won't be throttled unless that particular domain is throttled but with NUMA node domains it may be throttled if it has traffic going to that
domain.

Quoted 2 lines

I'll defer to Babu for accurate answers about this hardware capability.

To me, Global MBA should be considered a NODE-scoped resource. In some configurations it may appear
as SYSTEM-scoped, but that is effectively equivalent to a single-node encompassing the entire system. In such cases, there is only one schemata entry controlling the whole system.

Quoted 4 lines

Yes, multiple L3 instances are grouped together to form a NODE. Internally, programming is still performed at the L3 level, but that implementation detail can be hidden from users and does not need
to be exposed through the interface.

We seem to have two things that can both, somewhat reasonably, be called NODE scope in the resctrl user interface but the behaviour required for an MPAM system and an AMD system appears different from the point of view of lifecycle of the resctrl domain.

For MPAM NUMA scope the MSC instance (MPAM hardware interface) is at the memory controller and so goes on and offline based on whether the NUMA node is offline or online. For AMD NUMA scope it looks
to me that the lifecycle of the resctrl domains would be tied to the CPUs associated with the NUMA node. To me it does seem odd that a control with a domain associated with an offline NUMA node can continue to throttle (cross-NUMA) traffic.

Quoted 4 lines
Quoted 3 lines

Is there any GLBE Control Domain ID or similar that is exposed to the user, e.g. is sysfs, or is this just implicitly the NUMA id?

Yes, the GLBE Control Domain ID is exposed to the user. It is essentially equivalent to the NUMA ID.

What's the on/off lifecycle of these nodes? Does it follow the NUMA lifecycle as managed by the NUMA node notifiers documented in Documentation/core-api/memory-hotplug.rst or is it just linked to the cpu hotplug as is done currently for the resctrl cache based domains.

Quoted 6 lines
Quoted 2 lines

It will follow the CPU hotplug lifecycle, similar to how it is currently handled for the resctrl cache-based domains.

Ok. MPAM MSC are associated with the memory controllers and so MPAM controls that have NUMA scope will follow the NUMA memory hotplug lifecycle. This points to them being different resources. Do you have any thoughts on how we would handle this difference?

If I understand correctly, on an AMD GLBE system, even though it is "NUMA node scoped" it does not support memory bandwidth allocation for a NUMA node that is online but all its CPUs are offline. This is because GLBE is essentially L3 MBA that allows to set limits across multiple L3 domains. I think this is a good match for a control associated with the legacy MB resource that just has a different scope of allocation but how to do so without creating confusion with a "real" NUMA bandwidth allocation is not clear to me since it may end up looking like:

GLBE (make clear allocation is at L3 scope but domains are node scoped):
info/

└── MB/
    └── schemata/
        ├── MB/      /* scope of domain ID = L3   */
        └── MB_NODE/ /* scope of domain ID = node */
Quoted 8 lines

MPAM MSC (make clear allocation is at node scope:
info/

├── MB/
│   └── schemata/
│       └── MB/      /* scope of domain ID = L3   */
└── MB_NODE/
    └── schemata/
        └── MB_NODE/ /* scope of domain ID = node  */

These info/ trees do express the difference between the two, GLBE and MPAM MSC at the memory. These both result in the same schemata file though.
MB:<cache_id>=...
MB_NODE:<node_id>...

This gives a potential conflict if GLBE is paired with memory allocation controls at the memory. I don't know if this is likely though.

Quoted 5 lines

Any suggestions?

I do not see how these can be considered different resources though. To me it looks like different controls operating at different scope for the same ("memory bandwidth") resource?

This is likely just a terminology issue. My thoughts are that, in a similar way to the L2 and L3 both being cache resources but a different level of the hierarchy, you can think of memory bandwidth at egress from L3 as a different resource from memory bandwidth at ingress to the memory controllers. Sure, they are interdependent but on an MPAM system you could have MSC at both locations. On a system with one L3 and one NUMA node and no other caches in between they are the same resource as they are just opposite ends of the same link but that's not the case when you can have cross NUMA traffic.

So in your diagram above info/MB and info/MB_NODE are different "resources" and info/MB/MB_NODE is node based scoping of the info/MB "resource". Do you have a good way of referring to these resource concepts so that we can be more precise in our language?

Thanks,

Ben

Quoted 2 lines

Reinette

author maintainer
#92 lore raw

Hi Everybody,

Quoted 125 lines · Ben Horgan
Quoted 93 lines · Reinette Chatre
Quoted 72 lines · Ben Horgan
Quoted 71 lines · Babu Moger
Quoted 70 lines · Ben Horgan
Quoted 60 lines · Babu Moger
Quoted 59 lines · Ben Horgan
Quoted 42 lines · Babu Moger
Quoted 33 lines · Reinette Chatre
Quoted 2 lines · Ben Horgan
Quoted 1 line · Reinette Chatre

On 7/20/26 6:30 AM, Ben Horgan wrote:

...>

Quoted 21 lines

The former, info/ contains a directory for each allocation scope of each resource.

Quoted 8 lines

The other x86 feature to consider is AMD's upcoming "Global" MBA/SMBA that exposes memory bandwidth allocation
in "groups of L3" that I understand could usually be mapped to NODE scope (but it remains controlled at L3 scope),
except for one configuration where it is "SYSTEM"(?) scope.
Ref.: https://lore.kernel.org/
lkml/8f77f498b1c77fa8fd8f5d5687f03ae598068544.1776980182.git.babu.moger@amd.com/

Hmmm, I'm not sure that the scope can be considered to be NODE scope for GMBA. To me it seems to be
accidental that it maps to the NUMA node but really the scope is just a grouping of L3 instances.
For a control to NUMA scope I would expect the resctrl domains to go offline and online in sync with
the NUMA nodes. For GMBA it looks like it would just going offline/online based on whether any of
the CPUs and so L3 instances in the group are online. Am I correct here?

Quoted 7 lines

Assuming the domains are on L3 groups rather than NUMA also changes which end of the link the traffic is regulated and so how cross-NUMA traffic behaves differently. If the domain is an L3 group
then a task running on a CPU affine to that L3 group won't be throttled unless that particular domain is throttled but with NUMA node domains it may be throttled if it has traffic going to that
domain.

Quoted 2 lines

I'll defer to Babu for accurate answers about this hardware capability.

To me, Global MBA should be considered a NODE-scoped resource. In some configurations it may appear
as SYSTEM-scoped, but that is effectively equivalent to a single-node encompassing the entire system. In such cases, there is only one schemata entry controlling the whole system.

Quoted 4 lines

Yes, multiple L3 instances are grouped together to form a NODE. Internally, programming is still performed at the L3 level, but that implementation detail can be hidden from users and does not need
to be exposed through the interface.

We seem to have two things that can both, somewhat reasonably, be called NODE scope in the resctrl user interface but the behaviour required for an MPAM system and an AMD system appears different from the point of view of lifecycle of the resctrl domain.

For MPAM NUMA scope the MSC instance (MPAM hardware interface) is at the memory controller and so goes on and offline based on whether the NUMA node is offline or online. For AMD NUMA scope it looks
to me that the lifecycle of the resctrl domains would be tied to the CPUs associated with the NUMA node. To me it does seem odd that a control with a domain associated with an offline NUMA node can continue to throttle (cross-NUMA) traffic.

Quoted 4 lines
Quoted 3 lines

Is there any GLBE Control Domain ID or similar that is exposed to the user, e.g. is sysfs, or is this just implicitly the NUMA id?

Yes, the GLBE Control Domain ID is exposed to the user. It is essentially equivalent to the NUMA ID.

What's the on/off lifecycle of these nodes? Does it follow the NUMA lifecycle as managed by the NUMA node notifiers documented in Documentation/core-api/memory-hotplug.rst or is it just linked to the cpu hotplug as is done currently for the resctrl cache based domains.

Quoted 6 lines
Quoted 2 lines

It will follow the CPU hotplug lifecycle, similar to how it is currently handled for the resctrl cache-based domains.

Ok. MPAM MSC are associated with the memory controllers and so MPAM controls that have NUMA scope will follow the NUMA memory hotplug lifecycle. This points to them being different resources. Do you have any thoughts on how we would handle this difference?

If I understand correctly, on an AMD GLBE system, even though it is "NUMA node scoped" it does not support memory bandwidth allocation for a NUMA node that is online but all its CPUs are offline. This is because GLBE is essentially L3 MBA that allows to set limits across multiple L3 domains. I think this is a good match for a control associated with the legacy MB resource that just has a different scope of allocation but how to do so without creating confusion with a "real" NUMA bandwidth allocation is not clear to me since it may end up looking like:

GLBE (make clear allocation is at L3 scope but domains are node scoped):
info/

└── MB/
    └── schemata/
        ├── MB/      /* scope of domain ID = L3   */
        └── MB_NODE/ /* scope of domain ID = node */
Quoted 8 lines

MPAM MSC (make clear allocation is at node scope:
info/

├── MB/
│   └── schemata/
│       └── MB/      /* scope of domain ID = L3   */
└── MB_NODE/
    └── schemata/
        └── MB_NODE/ /* scope of domain ID = node  */

These info/ trees do express the difference between the two, GLBE and MPAM MSC at the memory. These both result in the same schemata file though.
MB:<cache_id>=...
MB_NODE:<node_id>...

This gives a potential conflict if GLBE is paired with memory allocation controls at the memory. I don't know if this is likely though.

Quoted 5 lines

Any suggestions?

I do not see how these can be considered different resources though. To me it looks like different controls operating at different scope for the same ("memory bandwidth") resource?

This is likely just a terminology issue. My thoughts are that, in a similar way to the L2 and L3 both being cache resources but a different level of the hierarchy, you can think of memory bandwidth at egress from L3 as a different resource from memory bandwidth at ingress to the memory controllers. Sure, they are interdependent but on an MPAM system you could have MSC at both locations. On a system with one L3 and one NUMA node and no other caches in between they are the same resource as they are just opposite ends of the same link but that's not the case when you can have cross NUMA traffic.

Quoted 4 lines

So in your diagram above info/MB and info/MB_NODE are different "resources" and info/MB/MB_NODE is node based scoping of the info/MB "resource". Do you have a good way of referring to these resource concepts so that we can be more precise in our language?

This thread continues at https://lore.kernel.org/lkml/[email protected]/

Reinette

maintainer
#93 lore raw
In reply to Reinette Chatre

Hi Reinette,

Quoted 111 lines · Reinette Chatre

Hi Ben,

Quoted 2 lines · Ben Horgan
Quoted 1 line · Reinette Chatre

On 7/20/26 6:30 AM, Ben Horgan wrote:

...>

Quoted 24 lines

The former, info/ contains a directory for each allocation scope of each resource.

info

├── L2
│   ├── resource_schemata
│   │   ├── L2
│   │   ├── L2_CMAX
│   │   └── L2_CMIN
│   └── scope : L2
├── L3
│   ├── resource_schemata
│   │   ├── L3
│   │   ├── L3_CMAX
│   │   └── L3_CMIN
│   └── scope : L3
├── MB
│   ├── resource_schemata
│   │   ├── MB
│   │   │   └── MB_MAX
│   │   ├── MB_MIN
│   │   ├── MB_PBM
│   │   └── MB_PROP
Quoted 9 lines
│   └── scope : L3
└── MB_NODE
    ├── resource_schemata
    │   ├── MB_NODE_MAX
    │   ├── MB_NODE_MIN
    │   ├── MB_NODE_PBM
    │   └── MB_NODE_PROP
    └── scope : NUMA NODE

At first glance this looks good to me. As you highlight below the nuances of how "Global S/MBA" can fit in here still needs to be worked out. One thing that resctrl may need to highlight when documenting this new capability is that while historically a resource had a matching schemata entry, this is no longer the case. As you show above some systems may have MB_NODE resource but no MB_NODE control while I expect that "Global MBA" (if it adopts this) may indeed have a MB_NODE resource with a MB_NODE control.

Quoted 12 lines

Please just consider this a mistake. MB_MAX should be a child of MB.

info/

├── MB
│   ├── resource_schemata
│   │   └── MB
│   │       └── MB_MAX
│   └── scope
└── MB_NODE
    ├── resource_schemata
    │   └── MB_NODE_MAX (Changed from MB_NODE)
    └── scope

ack.

Quoted 7 lines
Quoted 6 lines

When thinking about MPAM, what would the underlying hardware control of "MB_NODE" be? It looks from above that it would either start out by itself having the properties of the underlying "MAX" control or is the plan to have it be a percentage based control backed by the underlying "MAX" hardware control?

Quoted 5 lines

The underlying hardware of MB_NODE would be essentially the same hardware as that backing MB_MAX, but at a different location in the SoC, at the memory controller rather than in the L3.

To correct myself slightly, I don't think we should have a control called MB_NODE, rather, it should be MB_NODE_MAX.

ack.

Quoted 4 lines

My understanding of previous discussions is that <resource_type>_<scope>_<control_type> it the pattern for control names and the pattern for resource names being <resource_type>_<scope>. (allowing for <scope> or <control_type> being missing to match existing naming.)

This is where discussions are from my view also.

Quoted 3 lines

I don't think we should introduce more percent based controls and for new controls we can introduce a new format to describe them. Perhaps just the positive integer with a resolution supplied in info/

No, we should not introduce more percentage based controls per se. With the new schema format a percentage based control is just a variant of a proportional scalar control.

Quoted 11 lines

as discussed previously. Although, I have been pondering on whether we can do a bit better.

We could use hexadecimal point based format for controls which are a proportion of a resource and have a resolution which is a power of 2. The advantage of this is that the meaning of the value is independent of the granularity of the control (number of parts).

0 is represented as 0x0

1 as                0x1
1/2 as              0x0.8
7/256 as            0x0.07
1/2**28             0x0.00000001
Quoted 8 lines

etc

This maps well to the MPAM fixed-point fraction point format without having the weirdness of having values forced to 1 or 0 not being really 0. These MPAM h/w oddities can be hidden just by using the mbw_min mbw_max of a control. In MPAM this could be used in CMIN, CMAX, MB_MAX, MB_MIN and I would hope this would be useful for other architectures too. I am preparing some RFC patches on top of your PoC for consideration of this idea and to explore some of the proposals discussed relating to generic schemata and how they land in practice from the MPAM side.

Quoted 2 lines

I'm going to stand with Dave Martin [1] on this point with a preference to avoid floating point in resctrl input and output.

Ok, but I'm not suggesting floating point. Dave's two objections to floating point were that it's not exact and that it's hard to parse. For the first it is exact as it is using base 16 and for the parsing it's essentially parsing a prefix, '0x0.', and then a hexadecimal value and shifting for resolution. 0 and 1 will need to be considered as special cases. What I did wonder if it was too weird a format for userspace. Anyhow, no point flogging a dead horse, let's just go with using a number in a proportional schema along with the resolution as Dave summarized in [1]

[1] https://lore.kernel.org/lkml/aPtfMFfLV1l%[email protected]/

Quoted 2 lines

Could you please elaborate where values are forced to 1 or 0? The new schema format was intentionally created to *avoid* rounding errors (and parsing complexity).

This a property of the MPAM fixed-point fraction point format in the hardware. It would need to be hidden whatever resctrl schema format is used, percentage, scalar, fraction ... etc.

For an MPAM MBW MAX control with a width of 4 bit the range of values in 0-0xF. In this set up the resolution is 16 but there are only 15 possible values. For MBW MAX a 0 in hardware the minimum value which becomes 1/16, ~6%.
See mpam_resctrl.c:mbw_max_to_percent() for how this is currently dealt with.

For an MPAM MBW MIN control we shift the other way and so 0 is 0 but (with 4 bits) 0xF is 15/16 ~94%. When the input resolution is a power of 2 these conversions between h/w value and schemata value can be slightly simplified but there is still some ugly offset of 1.

(The spec is a little less precise than this and uses weasel words like "Arm recommends..." but this seems to be the sensible interpretation. See ARM IHI0099B.c "MPAM system component specification", Section 9.3.)

Quoted 22 lines

[1] https://lore.kernel.org/lkml/[email protected]/

Quoted 10 lines
Quoted 3 lines

I also understand MPAM to support more memory bandwidth controls ("MIN", "HARDMAX"/"HARDLIM", etc.). Do you envision them to exist within info/MB/resource_schemata/<control> as well as within info/MB_NODE/resource_schemata/<control>?

Yes, at least for MIN, see the info/ tree above. For HARDLIM, perhaps, but HARDLIM has the added complications that it is a property of the MBW_MAX control and that it may be configurable for each PARTID or a fixed property of the h/w. When HARDLIM is configurable the control name could be of the form <resource_type>_<scope>_<control>_<control_param> where <control_param> is HARDLIM and the full name for the HARDLIM configuration on the MB_NODE resource is MB_NODE_MAX_HARDLIM. There can also be an info/<resource>/resource_schemata/<control>/lim file which has values, soft, hard, configurable.

ack. HARDLIM sounds like it would be a new control type. Perhaps a "boolean" type for which new control files need to be decided on? Sounds like you are headed in this direction and already have one control file in mind for this new type.

With this in mind the control could be built on top of what is being developed at the moment, possibly be presented to user space following Dave Martin's suggestion in
https://lore.kernel.org/lkml/[email protected]/:

Quoted 5 lines

| MB_HARDMAX: 0=0, 1=1, 2=1, 3=0 [...]

or

| MB_HARDMAX: 0=off, 1=on, 2=on, 3=off [...]

I'm happy with either of these format but have a slight preference for the second as on/off more clearly indicates this is a boolean with only two choices.

Quoted 111 lines
Quoted 29 lines

For CMAX, maximum cache capacity, there is an equivalent control SOFTLIM, which behaves as HARDLIM except the meaning of the bit is reversed. We can just use a consistent name in s/w though.

Quoted 24 lines
Quoted 16 lines

On an x86 system:

info

├── MB
│   ├── resource_schemata
│   │   ├── MB
│   │   └── MB_MAX (Finer grained MB, more below)
│   └── scope
└── MB_REGION
    ├── resource_schemata
    │   └── MB_REGION
    └── scope
Quoted 1 line

Do you think this helps?

"REGION" is not a new scope but instead region-aware MBA is controlled and manages bandwidth at L3 scope. Combine that with up to (currently) four regions each with three controls I find an interface like above potentially confusing to document in an intuitive way. Unless you are perhaps saying that we should introduce a new separate "MB_REGION" L3 scope resource (so let resctrl support multiple "MB" resources at the same scope?) and then *it* contains the twelve new controls within its resource_schemata directory?

Quoted 6 lines
Quoted 3 lines

Since the region-aware controls are orthogonal to the MSR based legacy control resctrl would still need a way for user space to switch from one to the other which implies a dependency between "MB" and "MB_REGION" that is not presented in above hierarchy.

OK. The /sys/fs/resctrl/info/MB/schemata/mode and the MB_REGION controls a child of MB you described previously seem s better fit than what I suggested.

ok, I'll keep following that.

Quoted 21 lines
Quoted 6 lines

I think I am missing quite a bit here as I try to navigate an interface so different from what we have discussed so far.
I would like to explore with more detail how this interface can handle the different scenarios we have discussed so far.

Certainly, I don't think we have got to the bottom of this yet.

Quoted 5 lines

The other x86 feature to consider is AMD's upcoming "Global" MBA/SMBA that exposes memory bandwidth allocation in "groups of L3" that I understand could usually be mapped to NODE scope (but it remains controlled at L3 scope), except for one configuration where it is "SYSTEM"(?) scope.
Ref.: https://lore.kernel.org/lkml/8f77f498b1c77fa8fd8f5d5687f03ae598068544.1776980182.git.babu.moger@amd.com/

Hmmm, I'm not sure that the scope can be considered to be NODE scope for GMBA. To me it seems to be accidental that it maps to the NUMA node but really the scope is just a grouping of L3 instances. For a control to NUMA scope I would expect the resctrl domains to go offline and online in sync with the NUMA nodes. For GMBA it looks like it would just going offline/online based on whether any of the CPUs and so L3 instances in the group are online. Am I correct here?

Quoted 5 lines

Assuming the domains are on L3 groups rather than NUMA also changes which end of the link the traffic is regulated and so how cross-NUMA traffic behaves differently. If the domain is an L3 group then a task running on a CPU affine to that L3 group won't be throttled unless that particular domain is throttled but with NUMA node domains it may be throttled if it has traffic going to that domain.

I'll defer to Babu for accurate answers about this hardware capability.

Quoted 32 lines
Quoted 8 lines
Quoted 6 lines

This also brings another question. On MPAM systems the 'MB_MAX' is backed by the same MSC h/w as MB but it exposed a different interface to the user. If I understand correctly intel have an option to have finer grained control of MB (delay) as well and so it would make sense to use a common name rather than just going for the MPAM centric name of MB_MAX.

Apologies but I was not able to parse above.

Ok, let me try to explain again (although it's probably not what we want to do). My intent here was to try and explore whether we can reuse naming and controls across architectures in the same way we already have for the L2/L3 cache portion bitmap and the existing MB control.

To quote from a previous mail of yours:
https://lore.kernel.org/lkml/[email protected]/
"""
For example, on an MPAM system (if I understand correctly) the user may see:
info/

  └── MB/
      └── resource_schemata/
          ├── MB/
          │   └── MB_MAX/
          └── MB_MIN/

Compared with a possible implementation on Intel that looks like:
info/

  └── MB/
      └── resource_schemata/
          ├── MB/
          │   └── MB_OPT/
          ├── MB_MAX/
          └── MB_MIN/
Quoted 6 lines

"""

In the two setups MPAM MB_MAX and intel MB_OPT play the same role, a finer grained control of the legacy MB control. I was thinking these could share a name (MB-PRECISE), but it probably doesn't make sense as MB_OPT and MB_MAX have different relationships to MB_MIN.

Apologies, I neglected to follow up on this after I clarified internally which RDT control should actually be used to emulate the percentage based control. My original thoughts that you highlight above is not correct the RDT also plans to use the "MAX" underlying hardware control for the percentage based MB control. With this RDT and MPAM should look more similar with RDT simplified (by dropping the region-aware terms) as:

Quoted 7 lines

info/

   └── MB/
       └── resource_schemata/
           ├── MB/
           │   └── MB_MAX/
           ├── MB_OPT/
           └── MB_MIN/

Ack

Thanks,

Ben

Quoted 3 lines

Reinette

author maintainer
#94 lore raw

Hi Ben,

Quoted 3 lines · Ben Horgan
Quoted 2 lines · Reinette Chatre
Quoted 1 line · Ben Horgan

On 7/20/26 23:54, Reinette Chatre wrote:

...

Quoted 29 lines
Quoted 26 lines
Quoted 2 lines

I don't think we should introduce more percent based controls and for new controls we can introduce a new format to describe them. Perhaps just the positive integer with a resolution supplied in info/

No, we should not introduce more percentage based controls per se. With the new schema format a percentage based control is just a variant of a proportional scalar control.

Quoted 11 lines

as discussed previously. Although, I have been pondering on whether we can do a bit better.

We could use hexadecimal point based format for controls which are a proportion of a resource and have a resolution which is a power of 2. The advantage of this is that the meaning of the value is independent of the granularity of the control (number of parts).

0 is represented as 0x0

1 as                0x1
1/2 as              0x0.8
7/256 as            0x0.07
1/2**28             0x0.00000001
Quoted 8 lines

etc

This maps well to the MPAM fixed-point fraction point format without having the weirdness of having values forced to 1 or 0 not being really 0. These MPAM h/w oddities can be hidden just by using the mbw_min mbw_max of a control. In MPAM this could be used in CMIN, CMAX, MB_MAX, MB_MIN and I would hope this would be useful for other architectures too. I am preparing some RFC patches on top of your PoC for consideration of this idea and to explore some of the proposals discussed relating to generic schemata and how they land in practice from the MPAM side.

Quoted 2 lines

I'm going to stand with Dave Martin [1] on this point with a preference to avoid floating point in resctrl input and output.

Quoted 8 lines

Ok, but I'm not suggesting floating point. Dave's two objections to floating point were that it's not exact and that it's hard to parse. For the first it is exact as it is using base 16 and for the parsing it's essentially parsing a prefix, '0x0.', and then a hexadecimal value and shifting for resolution. 0 and 1 will need to be considered as special cases. What I did wonder if it was too weird a format for userspace. Anyhow, no point flogging a dead horse, let's just go with using a number in a proportional schema along with the resolution as Dave summarized in [1]

[1] https://lore.kernel.org/lkml/aPtfMFfLV1l%[email protected]/

ok. I do like how the scalar control from that proposal accommodates a variety of proportional and absolute hardware controls without forcing/requiring hardware controls to be a particular shape.

Quoted 6 lines
Quoted 3 lines

Could you please elaborate where values are forced to 1 or 0? The new schema format was intentionally created to *avoid* rounding errors (and parsing complexity).

This a property of the MPAM fixed-point fraction point format in the hardware. It would need to be hidden whatever resctrl schema format is used, percentage, scalar, fraction ... etc.

Just a clarification, all three of the examples you mention (percentage, scalar, fraction) are currently expected to be supported by the single "scalar" control.

(see later for comment on "need to be hidden")

Quoted 5 lines

For an MPAM MBW MAX control with a width of 4 bit the range of values in 0-0xF. In this set up the resolution is 16 but there are only 15 possible values. For MBW MAX a 0 in hardware the minimum value which becomes 1/16, ~6%.
See mpam_resctrl.c:mbw_max_to_percent() for how this is currently dealt with.

Quoted 7 lines

For an MPAM MBW MIN control we shift the other way and so 0 is 0 but (with 4 bits) 0xF is 15/16 ~94%. When the input resolution is a power of 2 these conversions between h/w value and schemata value can be slightly simplified but there is still some ugly offset of 1.

(The spec is a little less precise than this and uses weasel words like "Arm recommends..." but this seems to be the sensible interpretation. See ARM IHI0099B.c "MPAM system component specification", Section 9.3.)

Thank you for the clarification. Sounds related to the weasel words in
https://lore.kernel.org/lkml/[email protected]/ that took us a while to settle on.

You mention above that this would need to be hidden but from what I understand this is accommodated by the "tolerance" property of the "scalar" control?

Per https://lore.kernel.org/lkml/aPtfMFfLV1l%[email protected]/:

Note on the "tolerance" parameter:

This is a new addition. On the MPAM side, the hardware has a choice about how to interpret the control value in some edge-case situations. We may not reasonably be able to probe for this, so it may be useful to warn software that there is an uncertainty margin.

Quoted 21 lines
Quoted 21 lines

[1] https://lore.kernel.org/lkml/[email protected]/

Quoted 10 lines
Quoted 3 lines

I also understand MPAM to support more memory bandwidth controls ("MIN", "HARDMAX"/"HARDLIM", etc.). Do you envision them to exist within info/MB/resource_schemata/<control> as well as within info/MB_NODE/resource_schemata/<control>?

Yes, at least for MIN, see the info/ tree above. For HARDLIM, perhaps, but HARDLIM has the added complications that it is a property of the MBW_MAX control and that it may be configurable for each PARTID or a fixed property of the h/w. When HARDLIM is configurable the control name could be of the form <resource_type>_<scope>_<control>_<control_param> where <control_param> is HARDLIM and the full name for the HARDLIM configuration on the MB_NODE resource is MB_NODE_MAX_HARDLIM. There can also be an info/<resource>/resource_schemata/<control>/lim file which has values, soft, hard, configurable.

ack. HARDLIM sounds like it would be a new control type. Perhaps a "boolean" type for which new control files need to be decided on? Sounds like you are headed in this direction and already have one control file in mind for this new type.

With this in mind the control could be built on top of what is being developed at the moment, possibly be presented to user space following Dave Martin's suggestion in
https://lore.kernel.org/lkml/[email protected]/:

Quoted 8 lines
Quoted 5 lines

| MB_HARDMAX: 0=0, 1=1, 2=1, 3=0 [...]

or

| MB_HARDMAX: 0=off, 1=on, 2=on, 3=off [...]

I'm happy with either of these format but have a slight preference for the second as on/off more clearly indicates this is a boolean with only two choices.

So far I am only aware of MPAM needing a boolean control which hints to me that MPAM can set direction here.

Reinette

maintainer
#95 lore raw

Hi Reinette,

Quoted 50 lines · Reinette Chatre

Hi Ben,

Quoted 3 lines · Ben Horgan
Quoted 2 lines · Reinette Chatre
Quoted 1 line · Ben Horgan

On 7/20/26 23:54, Reinette Chatre wrote:

...

Quoted 29 lines
Quoted 26 lines
Quoted 2 lines

I don't think we should introduce more percent based controls and for new controls we can introduce a new format to describe them. Perhaps just the positive integer with a resolution supplied in info/

No, we should not introduce more percentage based controls per se. With the new schema format a percentage based control is just a variant of a proportional scalar control.

Quoted 11 lines

as discussed previously. Although, I have been pondering on whether we can do a bit better.

We could use hexadecimal point based format for controls which are a proportion of a resource and have a resolution which is a power of 2. The advantage of this is that the meaning of the value is independent of the granularity of the control (number of parts).

0 is represented as 0x0

1 as                0x1
1/2 as              0x0.8
7/256 as            0x0.07
1/2**28             0x0.00000001
Quoted 8 lines

etc

This maps well to the MPAM fixed-point fraction point format without having the weirdness of having values forced to 1 or 0 not being really 0. These MPAM h/w oddities can be hidden just by using the mbw_min mbw_max of a control. In MPAM this could be used in CMIN, CMAX, MB_MAX, MB_MIN and I would hope this would be useful for other architectures too. I am preparing some RFC patches on top of your PoC for consideration of this idea and to explore some of the proposals discussed relating to generic schemata and how they land in practice from the MPAM side.

Quoted 2 lines

I'm going to stand with Dave Martin [1] on this point with a preference to avoid floating point in resctrl input and output.

Quoted 8 lines

Ok, but I'm not suggesting floating point. Dave's two objections to floating point were that it's not exact and that it's hard to parse. For the first it is exact as it is using base 16 and for the parsing it's essentially parsing a prefix, '0x0.', and then a hexadecimal value and shifting for resolution. 0 and 1 will need to be considered as special cases. What I did wonder if it was too weird a format for userspace. Anyhow, no point flogging a dead horse, let's just go with using a number in a proportional schema along with the resolution as Dave summarized in [1]

[1] https://lore.kernel.org/lkml/aPtfMFfLV1l%[email protected]/

ok. I do like how the scalar control from that proposal accommodates a variety of proportional and absolute hardware controls without forcing/requiring hardware controls to be a particular shape.

Quoted 1 line
Quoted 8 lines
Quoted 5 lines
Quoted 2 lines

Could you please elaborate where values are forced to 1 or 0? The new schema format was intentionally created to *avoid* rounding errors (and parsing complexity).

This a property of the MPAM fixed-point fraction point format in the hardware. It would need to be hidden whatever resctrl schema format is used, percentage, scalar, fraction ... etc.

Just a clarification, all three of the examples you mention (percentage, scalar, fraction) are currently expected to be supported by the single "scalar" control.

Ack

Quoted 22 lines

(see later for comment on "need to be hidden")

Quoted 5 lines

For an MPAM MBW MAX control with a width of 4 bit the range of values in 0-0xF. In this set up the resolution is 16 but there are only 15 possible values. For MBW MAX a 0 in hardware the minimum value which becomes 1/16, ~6%.
See mpam_resctrl.c:mbw_max_to_percent() for how this is currently dealt with.

Quoted 7 lines

For an MPAM MBW MIN control we shift the other way and so 0 is 0 but (with 4 bits) 0xF is 15/16 ~94%. When the input resolution is a power of 2 these conversions between h/w value and schemata value can be slightly simplified but there is still some ugly offset of 1.

(The spec is a little less precise than this and uses weasel words like "Arm recommends..." but this seems to be the sensible interpretation. See ARM IHI0099B.c "MPAM system component specification", Section 9.3.)

Thank you for the clarification. Sounds related to the weasel words in
https://lore.kernel.org/lkml/[email protected]/ that took us a while to settle on.

You mention above that this would need to be hidden but from what I understand this is accommodated by the "tolerance" property of the "scalar" control?

Quoted 8 lines

Per https://lore.kernel.org/lkml/aPtfMFfLV1l%[email protected]/:

Note on the "tolerance" parameter:

This is a new addition. On the MPAM side, the hardware has a choice about how to interpret the control value in some edge-case situations. We may not reasonably be able to probe for this, so it may be useful to warn software that there is an uncertainty margin.

Maybe, "tolerance" was the one parameter from that proposal that stuck out as odd to me. I haven't managed to understand the semantics. In what situations is this different from the granularity? Also, isn't the inaccuracy from this h/w interface confusion going to be trumped by inaccuracies and inconsistencies in the h/w implementation of the throttling? I'm ok with having the parameter as long as we know how to decide what values to have in it and that they provide additional value. However personally, I am unclear on what the correct values would be.

Thanks,

Ben

author maintainer
#96 lore raw

Hi Ben,

Quoted 86 lines · Ben Horgan
Quoted 48 lines · Reinette Chatre
Quoted 3 lines · Ben Horgan
Quoted 2 lines · Reinette Chatre
Quoted 1 line · Ben Horgan

On 7/20/26 23:54, Reinette Chatre wrote:

...

Quoted 29 lines
Quoted 26 lines
Quoted 2 lines

I don't think we should introduce more percent based controls and for new controls we can introduce a new format to describe them. Perhaps just the positive integer with a resolution supplied in info/

No, we should not introduce more percentage based controls per se. With the new schema format a percentage based control is just a variant of a proportional scalar control.

Quoted 11 lines

as discussed previously. Although, I have been pondering on whether we can do a bit better.

We could use hexadecimal point based format for controls which are a proportion of a resource and have a resolution which is a power of 2. The advantage of this is that the meaning of the value is independent of the granularity of the control (number of parts).

0 is represented as 0x0

1 as                0x1
1/2 as              0x0.8
7/256 as            0x0.07
1/2**28             0x0.00000001
Quoted 8 lines

etc

This maps well to the MPAM fixed-point fraction point format without having the weirdness of having values forced to 1 or 0 not being really 0. These MPAM h/w oddities can be hidden just by using the mbw_min mbw_max of a control. In MPAM this could be used in CMIN, CMAX, MB_MAX, MB_MIN and I would hope this would be useful for other architectures too. I am preparing some RFC patches on top of your PoC for consideration of this idea and to explore some of the proposals discussed relating to generic schemata and how they land in practice from the MPAM side.

Quoted 2 lines

I'm going to stand with Dave Martin [1] on this point with a preference to avoid floating point in resctrl input and output.

Quoted 8 lines

Ok, but I'm not suggesting floating point. Dave's two objections to floating point were that it's not exact and that it's hard to parse. For the first it is exact as it is using base 16 and for the parsing it's essentially parsing a prefix, '0x0.', and then a hexadecimal value and shifting for resolution. 0 and 1 will need to be considered as special cases. What I did wonder if it was too weird a format for userspace. Anyhow, no point flogging a dead horse, let's just go with using a number in a proportional schema along with the resolution as Dave summarized in [1]

[1] https://lore.kernel.org/lkml/aPtfMFfLV1l%[email protected]/

ok. I do like how the scalar control from that proposal accommodates a variety of proportional and absolute hardware controls without forcing/requiring hardware controls to be a particular shape.

Quoted 1 line
Quoted 8 lines
Quoted 5 lines
Quoted 2 lines

Could you please elaborate where values are forced to 1 or 0? The new schema format was intentionally created to *avoid* rounding errors (and parsing complexity).

This a property of the MPAM fixed-point fraction point format in the hardware. It would need to be hidden whatever resctrl schema format is used, percentage, scalar, fraction ... etc.

Just a clarification, all three of the examples you mention (percentage, scalar, fraction) are currently expected to be supported by the single "scalar" control.

Ack

Quoted 26 lines

(see later for comment on "need to be hidden")

Quoted 5 lines

For an MPAM MBW MAX control with a width of 4 bit the range of values in 0-0xF. In this set up the resolution is 16 but there are only 15 possible values. For MBW MAX a 0 in hardware the minimum value which becomes 1/16, ~6%.
See mpam_resctrl.c:mbw_max_to_percent() for how this is currently dealt with.

Quoted 7 lines

For an MPAM MBW MIN control we shift the other way and so 0 is 0 but (with 4 bits) 0xF is 15/16 ~94%. When the input resolution is a power of 2 these conversions between h/w value and schemata value can be slightly simplified but there is still some ugly offset of 1.

(The spec is a little less precise than this and uses weasel words like "Arm recommends..." but this seems to be the sensible interpretation. See ARM IHI0099B.c "MPAM system component specification", Section 9.3.)

Thank you for the clarification. Sounds related to the weasel words in
https://lore.kernel.org/lkml/[email protected]/ that took us a while to settle on.

You mention above that this would need to be hidden but from what I understand this is accommodated by the "tolerance" property of the "scalar" control?

Per https://lore.kernel.org/lkml/aPtfMFfLV1l%[email protected]/:

Note on the "tolerance" parameter:

Quoted 7 lines
Quoted 4 lines

This is a new addition. On the MPAM side, the hardware has a choice about how to interpret the control value in some edge-case situations. We may not reasonably be able to probe for this, so it may be useful to warn software that there is an uncertainty margin.

Maybe, "tolerance" was the one parameter from that proposal that stuck out as odd to me. I haven't managed to understand the semantics. In what situations is this different from the granularity?

I see "tolerance" as an evolution of what can be expressed with the granularity. Granularity as exposed via the "bandwidth_gran" file is rooted in the original RDT enabling that was percentage based and its documentation reflects that.

https://lore.kernel.org/lkml/[email protected]/ aimed to add weasel words to support usage by other architectures but could not really accommodate all architectural nuances. This is because "bandwidth_gran" is, at least claims to be in the documentation, percentage based and always rounded up.

The "tolerance" parameter enables architecture to provide difference between requested and actual control value with a finer grained precision than percentages. More in line with the actual properties of the control value found in the other files introduced as part of this work. Beyond that the "tolerance" parameter also makes it clear that the effective control value can be larger or smaller than requested.

Quoted 4 lines

Also, isn't the inaccuracy from this h/w interface confusion going to be trumped by inaccuracies and inconsistencies in the h/w implementation of the throttling? I'm ok with having the parameter as long as we know how to decide what values to have in it and that they provide additional value. However personally, I am unclear on what the correct values would be.

resctrl would not do any measurements in an attempt to provide very accurate "tolerance" values, instead the goal is to provide insight to which values are actually be programmed to the hardware.

I expect the scenario to be that user space writes a control value X to the schemata file and when the user reads back the schemata file it may actually contain a control value of X ± tolerance. Not seeing the original written control value may be unexpected to the user and not knowing how much the actual control value can vary may introduce uncertainty. Having the "tolerance" value exposed to user space gives insight to the fact that there may be variation while also set expectations on how big a difference the requested from actual control value can be.

Reinette

reviewer
#97 lore raw
In reply to Ben Horgan

Hi, Ben,

Quoted 323 lines · Ben Horgan

Hi Reinette,

Quoted 163 lines · Reinette Chatre

Hi Ben,

Quoted 155 lines · Ben Horgan

Hi Reinette,

Quoted 136 lines · Reinette Chatre

Hi Ben,

Quoted 9 lines · Ben Horgan
Quoted 8 lines · Reinette Chatre
Quoted 7 lines · Ben Horgan
Quoted 6 lines · Reinette Chatre

Hi Ben, Chenyu, and Tony,

Quoted 3 lines · Ben Horgan
Quoted 2 lines · Reinette Chatre
Quoted 1 line · Ben Horgan

On 7/14/26 23:06, Reinette Chatre wrote:

...

Quoted 34 lines
Quoted 28 lines
Quoted 28 lines
Quoted 27 lines
Quoted 16 lines
Quoted 10 lines
Quoted 3 lines

Alternatively, could the "mode" file be used to switch between "MB" and "MB_MAX" and the two never need be shown at the same time. The user opts in to using the new interface, "MB_MAX" by setting "mode" and can just toggle back if they want to use "MB" directly again.

Interesting. So far the "mode" options have been "legacy" and "native" where "legacy" would show the legacy as well as emulated controls in the schemata file and "native" will only show the new controls. resctrl could make the default of "legacy" mean that *only* the legacy control is shown without insight into the controls it is being emulated with. The only insight to this would continue to be via the hierarchy in the info directory, which would show relationship but not the actual control values. Do you think there could be a need for users (excluding validation?) that may want to see the underlying control values used to emulate a legacy control?

They may want to see the underlying values to be able to migrate there legacy configuration to the native configuration but they can just write their legacy configuration and then toggle the mode to see the values in the native mode.

Quoted 1 line
Quoted 6 lines
Quoted 3 lines

I assume for backward compatibility that "legacy" would remain the default so a possible inconvenience here would be that users familiar with the new controls would forever need to switch the mode before being able to use them.

This does also make the new controls slightly harder to discover.

What if resctrl combines the two suggestions? Specifically, for backward compatibility the mode will be "legacy" and if there are emulated controls then resctrl displays them in schemata file with "#" prefix but not(*) support any changes from user space to the underlying hardware controls. This will make new controls easier to discover and let user space see the underlying values, but not break a user space that may read schemata file, change a few values, and write entire file back.

Quoted 5 lines
Quoted 1 line

My initial impression is that this would work but is probably unnecessary.

ok. My goal was to present ideas to address the issues raised so far. Ideally this would result in discussion of pros/cons. If you find this unnecessary, could you please expand with some insight into which parts you find unnecessary or how you would prefer this solved? I am finding it difficult to interpret above response.

Quoted 7 lines

Sorry, yes. I was a bit lazy in my reply yesterday. If the mode is easy and non-destructive to change then doesn't the user get the same information with just the burden of toggling the mode and then reading the schemata a second time. This does rely on the user being aware of the new behaviour though.

If the emulation isn't a, one to one, bijection then is changing mode expected to be destructive? More on this below.

There may be some corner cases (region aware may have some as you highlight below), but in general I do not expect changing mode to be destructive. The idea behind giving insight into underlying control values while legacy interface is in use is to address your earlier point that not doing so will make new controls slightly harder to discover. (more below)

Quoted 24 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

How the "enabled" vs "disabled" state of a control works with this needs some confirmation since region-aware MBA support has this extra caveat of supporting MSR and ACPI interfaces which results in the relationship between "mode" and "status" of underlying controls not being consistent. This may be ok but please consider example below.

Thinking through this with examples as I understand MPAM and RDT region aware so far. Could this work for MPAM and region aware MBA?

Quoted 6 lines
Quoted 4 lines
Quoted 1 line

(*) Should resctrl allow user space to change underlying control value of an emulated control in "legacy" mode?

I don't think this causes problems for MPAM but for RDT region aware controls couldn't you end up with a control state that isn't reachable just by configuring the legacy schema.

Could you please highlight the scenario you refer to?

My expectation here was that for any value of MB, X, the value for each of the regions would be the same, Y. (Not sure if this is actually the case.)

MB: X
# MB_REGION0_MAX:Y
# MB_REGION1_MAX:Y
# MB_REGION2_MAX:Y
# MB_REGION3_MAX:Y

Quoted 3 lines

So, if one of the regions was to be changed individually then it would be in a state not reachable by just changing the legacy control, MB. Potentially this complicated switching mode as well as uncommenting and writing a schemata.

Indeed. This scenario was highlighted in slide 8 of
https://lpc.events/event/19/contributions/2093/attachments/1958/4172/resctrl%20Microconference%20LPC%202025%20Tokyo.pdf
and discussed between Tony and Dave Martin in the thread starting at
https://lore.kernel.org/lkml/aPf0OKwDZ4XbmVRB@agluck-desk3/

Tony and Dave discussed a few scenarios and how resctrl could behave with different user interactions. On a high level I understood that the underlying controls will always, as you state below, "show the full story" and if user space interacts with them (instead of using the legacy control) then the control values associated with the legacy control cannot be relied on to represent accurate state to the point that it may even be better to not be visible.
You are right, this is a good motivation to not allow user to modify underlying controls when in "legacy" mode.

The remaining open is whether resctrl should display the (read-only) underlying control values in schemata file when in "legacy" mode. By default this does not work for region-aware since it will at least initially use different hardware interfaces for the different controls, essentially this means that there is no actual emulation and the legacy and region aware control are both legitimate hardware controls. Specifically, considering your example, there is no mapping from "X" to "Y".

The way region-aware is planning to address this is to use the "enabled" vs "disabled" status of a control to start with underlying controls disabled so that they are not displayed in schemata file when legacy mode is enabled.

If resctrl instead only displays legacy controls when in "legacy" mode then the "status" may no longer be needed for region-aware. RISC-V also considered using this "status" that I think may also be solved with this approach, I am not sure though.

Quoted 5 lines

I was thinking that in legacy mode that the region values would always be kept the same but perhaps the legacy (MB) value could just be the 'best estimate' and the native (MB_REGION0) values show the full story. 'best estimate' could be difficult to choose but allows switching mode to be non-destructive.

Quoted 7 lines

For MPAM, at least until Fenghua sent his series today on MBA control emulation [1], I was expecting that MPAM would only use emulation for exposing finer grade controls to the user and not allowing different underlying hardware to be used for the emulation. To me, it seems reasonable that a new control would involve a new schema but I do appreciate the benefit of being able to continue to use the old interface without stopping a new interface being used. The danger is that there are unexpected user visible side effects, e.g. domain lifetime, and that control behaviour is different from the user expectations.

I have not looked at Fenghua's series in detail but from earlier discussion I understood the high level problem to be that user space may have expectation that for every "resource" represented by a directory in info/ there is a matching entry in the schemata file. Whether this is an actual expectation from user space tools is not obvious to me though: there is already an exception since there is the "L3_MON" "resource" that does not have a schemata file entry.

Quoted 6 lines

To ensure backward compatibility the safest would be for resctrl to always provide a "MB" control via an entry in the schemata file when the "MB" resource is exposed via info/. If on the other hand resctrl is not expected to provide a "MB" entry in schemata file when there is a "MB" resource described in info/MB then resctrl should not be forced to always provide such emulation.

I would appreciate your thoughts here.

Ok, thanks for pointing this out, I hadn't previously properly understood the motivation for resource emulation.

As emulation adds complication and has the potential to break other user expectations I hope we can manage to keep it to only simple cases. For instance as I mentioned before, the lifetime of a resctrl domain lifetime is necessary different (when there are multiple NUMA nodes) for domains backed by MSC at the memory to MSC at the cache. At the memory they would be inaccessible when the NUMA node is powered off. Also, any partitioning is happening at a different point in the topology.

Quoted 4 lines

Can we get around this by by choosing a different organization of the info/ directory? We already have separate resource for L2 and L3. Perhaps, the same for MB, MB_NODE, (not sure about MB_REGION) and move the scope to be a property at info/<resource>/scope rather than info/<resource>/resource_schemata/ctrl/scope.

Quoted 4 lines

Good point that L2 and L3, which is the same resource at different scope, are treated as different resources. With that view it seems reasonable to also consider MB and MB_NODE as different resources. The point here is that it is at a different scope so more specifically what a "domain ID" in the schemata file represents. With this view, MB_REGION is L3 scope that matches to the MB resource.

Ok. This makes sense to me. I had confused myself about what MB_REGION is.

Quoted 5 lines

You will notice I get a bit lost in the discussion below so getting back here I would like to clarify if you propose that info/ contains a directory for each allocation scope of each resource with that directory containing descriptions of all the controls for that resource at the indicated scope or do you propose that info/ contains a directory for each control?

The former, info/ contains a directory for each allocation scope of each resource.

info

├── L2
│   ├── resource_schemata
│   │   ├── L2
│   │   ├── L2_CMAX
│   │   └── L2_CMIN
│   └── scope : L2
├── L3
│   ├── resource_schemata
│   │   ├── L3
│   │   ├── L3_CMAX
│   │   └── L3_CMIN
│   └── scope : L3
├── MB
│   ├── resource_schemata
│   │   ├── MB
│   │   │   └── MB_MAX
│   │   ├── MB_MIN
│   │   ├── MB_PBM
│   │   └── MB_PROP
│   └── scope : L3
└── MB_NODE
     ├── resource_schemata
     │   ├── MB_NODE_MAX
     │   ├── MB_NODE_MIN
     │   ├── MB_NODE_PBM
     │   └── MB_NODE_PROP
     └── scope : NUMA NODE
Quoted 7 lines
Quoted 4 lines

Can MB_REGION be considered an orthogonal new control or does using it require that the traditional intel MB (delay) not be configured?

The Intel systems that support both MSR ("traditional") and ACPI (region aware MBA) cannot use both concurrently. Not sure if this answers your question.

Thanks, this answers my question.

Quoted 30 lines

Just to clarify, there is no single "MB_REGION" control. When considering the "region aware MBA" feature I am currently aware of the following possible controls:
MB_REGION0_MIN
MB_REGION0_MAX
MB_REGION0_OPT
MB_REGION1_MIN
MB_REGION1_MAX
MB_REGION1_OPT
MB_REGION2_MIN
MB_REGION2_MAX
MB_REGION2_OPT
MB_REGION3_MIN
MB_REGION3_MAX
MB_REGION3_OPT

Quoted 12 lines

On an MPAM system:

info

├── MB
│   ├── resource_schemata
│   │   ├── MB
│   │   └── MB_MAX
│   └── scope
└── MB_NODE
     ├── resource_schemata
     │   └── MB_NODE
     └── scope
Quoted 7 lines

Above MB and MB_MAX are represented on the same level. My understanding is that MB_MAX is used by driver as the underlying hardware control for the percentage based MB exposed to user space. Thus, when user space changes the "MB" control via schemata file it is expected to also impact the underlying "MB_MAX" control. By representing them as above this relationship is lost. So far I understood that the "MB_MAX" control will be shown as a child of the "MB" control to show this relationship. Looks like you plan to change this but from what I understand this emulation is still relevant?

Please just consider this a mistake. MB_MAX should be a child of MB.

info/

├── MB
│   ├── resource_schemata
│   │   └── MB
│   │       └── MB_MAX
│   └── scope
└── MB_NODE
     ├── resource_schemata
     │   └── MB_NODE_MAX (Changed from MB_NODE)
     └── scope
Quoted 5 lines

When thinking about MPAM, what would the underlying hardware control of "MB_NODE" be? It looks from above that it would either start out by itself having the properties of the underlying "MAX" control or is the plan to have it be a percentage based control backed by the underlying "MAX" hardware control?

The underlying hardware of MB_NODE would be essentially the same hardware as that backing MB_MAX, but at a different location in the SoC, at the memory controller rather than in the L3.

To correct myself slightly, I don't think we should have a control called MB_NODE, rather, it should be MB_NODE_MAX.

My understanding of previous discussions is that <resource_type>_<scope>_<control_type> it the pattern for control names and the pattern for resource names being <resource_type>_<scope>. (allowing for <scope> or <control_type> being missing to match existing naming.)

I don't think we should introduce more percent based controls and for new controls we can introduce a new format to describe them. Perhaps just the positive integer with a resolution supplied in info/ as discussed previously. Although, I have been pondering on whether we can do a bit better.

We could use hexadecimal point based format for controls which are a proportion of a resource and have a resolution which is a power of 2. The advantage of this is that the meaning of the value is independent of the granularity of the control (number of parts).

0 is represented as 0x0

1 as                0x1
1/2 as              0x0.8
7/256 as            0x0.07
1/2**28             0x0.00000001

etc

This maps well to the MPAM fixed-point fraction point format without having the weirdness of having values forced to 1 or 0 not being really 0. These MPAM h/w oddities can be hidden just by using the mbw_min mbw_max of a control. In MPAM this could be used in CMIN, CMAX, MB_MAX, MB_MIN and I would hope this would be useful for other architectures too. I am preparing some RFC patches on top of your PoC for consideration of this idea and to explore some of the proposals discussed relating to generic schemata and how they land in practice from the MPAM side.

Quoted 4 lines

I also understand MPAM to support more memory bandwidth controls ("MIN", "HARDMAX"/"HARDLIM", etc.). Do you envision them to exist within info/MB/resource_schemata/<control> as well as within info/MB_NODE/resource_schemata/<control>?

Yes, at least for MIN, see the info/ tree above. For HARDLIM, perhaps, but HARDLIM has the added complications that it is a property of the MBW_MAX control and that it may be configurable for each PARTID or a fixed property of the h/w. When HARDLIM is configurable the control name could be of the form <resource_type>_<scope>_<control>_<control_param> where <control_param> is HARDLIM and the full name for the HARDLIM configuration on the MB_NODE resource is MB_NODE_MAX_HARDLIM. There can also be an info/<resource>/resource_schemata/<control>/lim file which has values, soft, hard, configurable.

Quoted 3 lines

For CMAX, maximum cache capacity, there is an equivalent control SOFTLIM, which behaves as HARDLIM except the meaning of the bit is reversed. We can just use a consistent name in s/w though.

Is it possible to view MBW_MAX hard limit, CMAX soft limit, or future similar things as a "configuration" for a control, instead of a "control" itself?

A "configurations" is different from a "control" in that:
1. The configuration configures the control, e.g. toggle hard limit on MB control.
2. The configuration doesn't have the control's properties (e.g. gran).
3. The configuration is similar to "event_configs" in MB_MON, that can also configure MB_MON's event_filter.

MBW_MAX hard limit configures MB/MB_MAX control's hard limit. CMAX soft limit configures L3 control's soft limit.

Is it OK to implement configurations in a control?

Then in struct resctrl_ctrl, add "list_head configs" to track all configurations in this control.

$ cat schemata
        MB_NODE:0=100;1=100;2=100;10=100;18=100;26=100;34=100;35=100
# MB_MAXHLIM_NODE is a configuration of MB_NODE control
MB_MAXHLIM_NODE:0=1;1=0;2=0;10=0;18=0;26=0;34=0;35=0
             MB:0=100;1=100;2=100;10=100;18=100;26=100;34=100;35=100
# MB_MAXHLIM is a configuration of MB control
     MB_MAXHLIM:0=1;1=0;2=0;10=0;18=0;26=0;34=0;35=0
             L3:1=ffff;2=ffff

info/

│   └── resource_schemata
│       ├── MB (simulated by MB_NODE)
│       │   ├── configs          # all configs in MB control
│       │   │   └── MB_MAXHLIM   # MB_MAXHLIM config in MB control
│       │   │       └── type     # boolean
│       │   ├── max
│       │   ├── MB_NODE          # simulated MB
│       │   │   ├── configs             # all configs in MB_NODE control
│       │   │   │   └── MB_MAXHLIM_NODE # MB_MAXHLIM config in MB_NODE
│       │   │   │       └── type
│       │   │   ├── max
│       │   │   ├── min
│       │   │   ├── resolution
│       │   │   ├── scale
│       │   │   ├── scope
│       │   │   ├── status
│       │   │   ├── tolerance
│       │   │   ├── type
│       │   │   └── unit
│       │   ├── min
│       │   ├── resolution
│       │   ├── scale
│       │   ├── scope
│       │   ├── status
│       │   ├── tolerance
│       │   ├── type
│       │   └── unit
│       └── mode
Quoted 90 lines
Quoted 24 lines
Quoted 16 lines

On an x86 system:

info

├── MB
│   ├── resource_schemata
│   │   ├── MB
│   │   └── MB_MAX (Finer grained MB, more below)
│   └── scope
└── MB_REGION
     ├── resource_schemata
     │   └── MB_REGION
     └── scope
Quoted 1 line

Do you think this helps?

"REGION" is not a new scope but instead region-aware MBA is controlled and manages bandwidth at L3 scope. Combine that with up to (currently) four regions each with three controls I find an interface like above potentially confusing to document in an intuitive way. Unless you are perhaps saying that we should introduce a new separate "MB_REGION" L3 scope resource (so let resctrl support multiple "MB" resources at the same scope?) and then *it* contains the twelve new controls within its resource_schemata directory?

Quoted 3 lines

Since the region-aware controls are orthogonal to the MSR based legacy control resctrl would still need a way for user space to switch from one to the other which implies a dependency between "MB" and "MB_REGION" that is not presented in above hierarchy.

OK. The /sys/fs/resctrl/info/MB/schemata/mode and the MB_REGION controls a child of MB you described previously seem s better fit than what I suggested.

Quoted 5 lines

I think I am missing quite a bit here as I try to navigate an interface so different from what we have discussed so far.
I would like to explore with more detail how this interface can handle the different scenarios we have discussed so far.

Certainly, I don't think we have got to the bottom of this yet.

Quoted 5 lines

The other x86 feature to consider is AMD's upcoming "Global" MBA/SMBA that exposes memory bandwidth allocation in "groups of L3" that I understand could usually be mapped to NODE scope (but it remains controlled at L3 scope), except for one configuration where it is "SYSTEM"(?) scope.
Ref.: https://lore.kernel.org/lkml/8f77f498b1c77fa8fd8f5d5687f03ae598068544.1776980182.git.babu.moger@amd.com/

Hmmm, I'm not sure that the scope can be considered to be NODE scope for GMBA. To me it seems to be accidental that it maps to the NUMA node but really the scope is just a grouping of L3 instances. For a control to NUMA scope I would expect the resctrl domains to go offline and online in sync with the NUMA nodes. For GMBA it looks like it would just going offline/online based on whether any of the CPUs and so L3 instances in the group are online. Am I correct here?

Assuming the domains are on L3 groups rather than NUMA also changes which end of the link the traffic is regulated and so how cross-NUMA traffic behaves differently. If the domain is an L3 group then a task running on a CPU affine to that L3 group won't be throttled unless that particular domain is throttled but with NUMA node domains it may be throttled if it has traffic going to that domain.

Quoted 8 lines
Quoted 6 lines

This also brings another question. On MPAM systems the 'MB_MAX' is backed by the same MSC h/w as MB but it exposed a different interface to the user. If I understand correctly intel have an option to have finer grained control of MB (delay) as well and so it would make sense to use a common name rather than just going for the MPAM centric name of MB_MAX.

Apologies but I was not able to parse above.

Ok, let me try to explain again (although it's probably not what we want to do). My intent here was to try and explore whether we can reuse naming and controls across architectures in the same way we already have for the L2/L3 cache portion bitmap and the existing MB control.

To quote from a previous mail of yours:
https://lore.kernel.org/lkml/[email protected]/
"""
For example, on an MPAM system (if I understand correctly) the user may see:
info/

   └── MB/
       └── resource_schemata/
           ├── MB/
           │   └── MB_MAX/
           └── MB_MIN/

Compared with a possible implementation on Intel that looks like:
info/

   └── MB/
       └── resource_schemata/
           ├── MB/
           │   └── MB_OPT/
           ├── MB_MAX/
           └── MB_MIN/

"""

Quoted 13 lines

In the two setups MPAM MB_MAX and intel MB_OPT play the same role, a finer grained control of the legacy MB control. I was thinking these could share a name (MB-PRECISE), but it probably doesn't make sense as MB_OPT and MB_MAX have different relationships to MB_MIN.

Thanks,

Ben

Quoted 3 lines

Reinette

Thanks.

-Fenghua