Skip to contentSkip to conversation

x86/resctrl : Support AMD Assignable Bandwidth Monitoring Counters (ABMC)

PATCH Superseded Babu Moger 5 participants 121 messages May 15, 2025 – Jun 11, 2025 Page 2 of 2

Subsystems (3)

v13 of 18 Superseded Latest is v18 →

lore
Share & tools
lore
maintainer
#105 lore raw
In reply to Luck, Tony

Hi Tony,

Quoted 4 lines · Tony
Quoted 3 lines · Reinette Chatre

Hi Peter and Babu,

On 5/21/25 2:18 AM, Peter Newman wrote:

..

Quoted 36 lines
Quoted 17 lines
Quoted 4 lines

There's also the mongroup-RMID overcommit use case I described above[1]. On Intel we can safely assume that there are counters to back all RMIDs, so num_mbm_cntrs would be calculated directly from num_rmids.

This is about the:
There's now more interest in Google for allowing explicit control of where RMIDs are assigned on Intel platforms. Even though the number of RMIDs implemented by hardware tends to be roughly the number of containers they want to support, they often still need to create containers when all RMIDs have already been allocated, which is not currently allowed. Once the container has been created and starts running, it's no longer possible to move its threads into a monitoring group whenever RMIDs should become available again, so it's important for resctrl to maintain an accurate task list for a container even when RMIDs are not available.

Quoted 8 lines

I see a monitor group as a collection of tasks that need to be monitored together. The "task list" is the group of tasks that share a monitoring ID that is required to be a valid ID since when any of the tasks are scheduled that ID is written to the hardware. I intentionally tried to not use RMID since I believe this is required for all archs.
I thus do not understand how a task can start running when it does not have a valid monitoring ID. The idea of "deferred assignment" is not clear to me, there can never be "unmonitored tasks", no? I think I am missing something here.

In the AMD/RMID implemenentation this might be achieved with something extra in the task structure to denote whether a task is in a monitored group or not. E.g. We add "task->rmid_valid" as well as "task->rmid". Tasks in an unmonitored group retain their "task->rmid" (that's what identifies them as a member of a group) but have task->rmid_valid set to false. Context switch code would be updated to load "0" into the IA32_PQR_ASSOC.RMID field for tasks without a valid RMID. So they would still be monitored, but activity would be bundled with all tasks in the default resctrl group.

Quoted 2 lines

Presumably something analogous could be done for ARM/MPAM.

I do not interpret this as an unmonitored task but instead a task that belongs to the default resource group. Specifically, any data accumulated by such a task is attributed to the default resource group. Having tasks in a separate group but their monitoring data accumulating in/contributed to the default resource group (that has its own set of tasks) sounds wrong to me. Such an implementation makes any monitoring data of default resource group invalid, and by extension impossible to use default resource group to manage an allocation for a group of monitor groups if user space needs insight in monitoring data across all these monitor groups. User space will need to interact with resctrl differently and individually query monitor groups instead of CTRL_MON group once.

Reinette

maintainer
#106 lore raw
Quoted 39 lines
Quoted 36 lines
Quoted 17 lines
Quoted 4 lines

There's also the mongroup-RMID overcommit use case I described above[1]. On Intel we can safely assume that there are counters to back all RMIDs, so num_mbm_cntrs would be calculated directly from num_rmids.

This is about the:
There's now more interest in Google for allowing explicit control of where RMIDs are assigned on Intel platforms. Even though the number of RMIDs implemented by hardware tends to be roughly the number of containers they want to support, they often still need to create containers when all RMIDs have already been allocated, which is not currently allowed. Once the container has been created and starts running, it's no longer possible to move its threads into a monitoring group whenever RMIDs should become available again, so it's important for resctrl to maintain an accurate task list for a container even when RMIDs are not available.

Quoted 8 lines

I see a monitor group as a collection of tasks that need to be monitored together. The "task list" is the group of tasks that share a monitoring ID that is required to be a valid ID since when any of the tasks are scheduled that ID is written to the hardware. I intentionally tried to not use RMID since I believe this is required for all archs.
I thus do not understand how a task can start running when it does not have a valid monitoring ID. The idea of "deferred assignment" is not clear to me, there can never be "unmonitored tasks", no? I think I am missing something here.

In the AMD/RMID implemenentation this might be achieved with something extra in the task structure to denote whether a task is in a monitored group or not. E.g. We add "task->rmid_valid" as well as "task->rmid". Tasks in an unmonitored group retain their "task->rmid" (that's what identifies them as a member of a group) but have task->rmid_valid set to false. Context switch code would be updated to load "0" into the IA32_PQR_ASSOC.RMID field for tasks without a valid RMID. So they would still be monitored, but activity would be bundled with all tasks in the default resctrl group.

Quoted 2 lines

Presumably something analogous could be done for ARM/MPAM.

Quoted 11 lines

I do not interpret this as an unmonitored task but instead a task that belongs to the default resource group. Specifically, any data accumulated by such a task is attributed to the default resource group. Having tasks in a separate group but their monitoring data accumulating in/contributed to the default resource group (that has its own set of tasks) sounds wrong to me. Such an implementation makes any monitoring data of default resource group invalid, and by extension impossible to use default resource group to manage an allocation for a group of monitor groups if user space needs insight in monitoring data across all these monitor groups. User space will need to interact with resctrl differently and individually query monitor groups instead of CTRL_MON group once.

Maybe assign one of the limited supply of RMIDs for these "unmonitored" tasks. Populate a resctrl group named "unmonitored" that lists all the unmonitored tasks in a (read-only) "tasks" file. And supply all the counts for these tasks in normal looking "mon_data" directory.

-Tony

#107 lore raw

Hi Tony, Reinette,

Quoted 18 lines · Tony
Quoted 18 lines
Quoted 18 lines
Quoted 18 lines
Quoted 5 lines

There's also the mongroup-RMID overcommit use case I described above[1]. On Intel we can safely assume that there are counters to back all RMIDs, so num_mbm_cntrs would be calculated directly from num_rmids.

This is about the:
There's now more interest in Google for allowing explicit control of where RMIDs are assigned on Intel platforms. Even though the number of RMIDs implemented by hardware tends to be roughly the number of containers they want to support, they often still need to create containers when all RMIDs have already been allocated, which is not currently allowed. Once the container has been created and starts running, it's no longer possible to move its threads into a monitoring group whenever RMIDs should become available again, so it's important for resctrl to maintain an accurate task list for a container even when RMIDs are not available.

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

I see a monitor group as a collection of tasks that need to be monitored together. The "task list" is the group of tasks that share a monitoring ID that is required to be a valid ID since when any of the tasks are scheduled that ID is written to the hardware. I intentionally tried to not use RMID since I believe this is required for all archs.
I thus do not understand how a task can start running when it does not have a valid monitoring ID. The idea of "deferred assignment" is not clear to me, there can never be "unmonitored tasks", no? I think I am missing something here.

You are correct. I did forget to mention something...

Quoted 26 lines
Quoted 14 lines
Quoted 11 lines

In the AMD/RMID implemenentation this might be achieved with something extra in the task structure to denote whether a task is in a monitored group or not. E.g. We add "task->rmid_valid" as well as "task->rmid". Tasks in an unmonitored group retain their "task->rmid" (that's what identifies them as a member of a group) but have task->rmid_valid set to false. Context switch code would be updated to load "0" into the IA32_PQR_ASSOC.RMID field for tasks without a valid RMID. So they would still be monitored, but activity would be bundled with all tasks in the default resctrl group.

Quoted 2 lines

Presumably something analogous could be done for ARM/MPAM.

Quoted 11 lines

I do not interpret this as an unmonitored task but instead a task that belongs to the default resource group. Specifically, any data accumulated by such a task is attributed to the default resource group. Having tasks in a separate group but their monitoring data accumulating in/contributed to the default resource group (that has its own set of tasks) sounds wrong to me. Such an implementation makes any monitoring data of default resource group invalid, and by extension impossible to use default resource group to manage an allocation for a group of monitor groups if user space needs insight in monitoring data across all these monitor groups. User space will need to interact with resctrl differently and individually query monitor groups instead of CTRL_MON group once.

Quoted 4 lines

Maybe assign one of the limited supply of RMIDs for these "unmonitored" tasks. Populate a resctrl group named "unmonitored" that lists all the unmonitored tasks in a (read-only) "tasks" file. And supply all the counts for these tasks in normal looking "mon_data" directory.

I needed to switch to an rdtgroup struct pointer rather than hardware IDs in the task structure to indicate group membership[1], otherwise it's not possible to determine which tasks are in a group when it doesn't have a unique HW ID value.

Also this is required for shared assignment so that changing a group's IDs in a domain only requires updating running tasks rather than needing to search the entire task list, which would lead to the same problem we encountered in mongroup rename[2].

-Peter

[1] https://lore.kernel.org/lkml/[email protected]/
[2] https://lore.kernel.org/lkml/CALPaoCh0SbG1+VbbgcxjubE7Cc2Pb6QqhG3NH6X=WwsNfqNjtA@mail.gmail.com/

maintainer
#108 lore raw

Hi Peter,

Quoted 63 lines · Peter Newman

Hi Tony, Reinette,

Quoted 18 lines · Tony
Quoted 18 lines
Quoted 18 lines
Quoted 18 lines
Quoted 5 lines

There's also the mongroup-RMID overcommit use case I described above[1]. On Intel we can safely assume that there are counters to back all RMIDs, so num_mbm_cntrs would be calculated directly from num_rmids.

This is about the:
There's now more interest in Google for allowing explicit control of where RMIDs are assigned on Intel platforms. Even though the number of RMIDs implemented by hardware tends to be roughly the number of containers they want to support, they often still need to create containers when all RMIDs have already been allocated, which is not currently allowed. Once the container has been created and starts running, it's no longer possible to move its threads into a monitoring group whenever RMIDs should become available again, so it's important for resctrl to maintain an accurate task list for a container even when RMIDs are not available.

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

I see a monitor group as a collection of tasks that need to be monitored together. The "task list" is the group of tasks that share a monitoring ID that is required to be a valid ID since when any of the tasks are scheduled that ID is written to the hardware. I intentionally tried to not use RMID since I believe this is required for all archs.
I thus do not understand how a task can start running when it does not have a valid monitoring ID. The idea of "deferred assignment" is not clear to me, there can never be "unmonitored tasks", no? I think I am missing something here.

You are correct. I did forget to mention something...

Quoted 26 lines
Quoted 14 lines
Quoted 11 lines

In the AMD/RMID implemenentation this might be achieved with something extra in the task structure to denote whether a task is in a monitored group or not. E.g. We add "task->rmid_valid" as well as "task->rmid". Tasks in an unmonitored group retain their "task->rmid" (that's what identifies them as a member of a group) but have task->rmid_valid set to false. Context switch code would be updated to load "0" into the IA32_PQR_ASSOC.RMID field for tasks without a valid RMID. So they would still be monitored, but activity would be bundled with all tasks in the default resctrl group.

Quoted 2 lines

Presumably something analogous could be done for ARM/MPAM.

Quoted 11 lines

I do not interpret this as an unmonitored task but instead a task that belongs to the default resource group. Specifically, any data accumulated by such a task is attributed to the default resource group. Having tasks in a separate group but their monitoring data accumulating in/contributed to the default resource group (that has its own set of tasks) sounds wrong to me. Such an implementation makes any monitoring data of default resource group invalid, and by extension impossible to use default resource group to manage an allocation for a group of monitor groups if user space needs insight in monitoring data across all these monitor groups. User space will need to interact with resctrl differently and individually query monitor groups instead of CTRL_MON group once.

Quoted 4 lines

Maybe assign one of the limited supply of RMIDs for these "unmonitored" tasks. Populate a resctrl group named "unmonitored" that lists all the unmonitored tasks in a (read-only) "tasks" file. And supply all the counts for these tasks in normal looking "mon_data" directory.

Quoted 4 lines

I needed to switch to an rdtgroup struct pointer rather than hardware IDs in the task structure to indicate group membership[1], otherwise it's not possible to determine which tasks are in a group when it doesn't have a unique HW ID value.

Whether the task struct contains a pointer (albeit accompanied with its own complexities) does not address the issue that I am concerned about.

Looking at [1] I expect this new feature handles "unmonitored" groups by placing them in the default monitoring group, following Tony's first [3] suggestion.

When considering [1] by itself in the context of current resctrl all tasks should be members of resource groups that have valid HW monitoring IDs allocated. Using the default resource group in this way seems like addressing edge cases where pointer is not yet valid (unclear what these scenarios may be) instead of routing many tasks to the default group. I am not sure and I'll have to study that change closer to reason accurately.

From what I understand the new proposal that builds on [1] involves creating new monitor groups that are "unmonitored" for any length of time and when backed by the implementation in [1] this would mean these groups will actually still be monitored but the data attributed to the default resource group.

As I mentioned in response [4] to Tony this fundamentally changes the behavior users can expect from the default resource group. In addition, this breaks the first of the "Resource monitoring rules" from Documentation/filesystems/resctrl.rst:

1) If a task is a member of a MON group, or non-default CTRL_MON group then RDT events for the task will be reported in that group.

How does this fit with the ABMC work? I continue to think that I am missing parts of the discussion as it seems this new feature discussion mixed in with ABMC work.

Reinette

Quoted 10 lines

Also this is required for shared assignment so that changing a group's IDs in a domain only requires updating running tasks rather than needing to search the entire task list, which would lead to the same problem we encountered in mongroup rename[2].

-Peter

[1] https://lore.kernel.org/lkml/[email protected]/
[2] https://lore.kernel.org/lkml/CALPaoCh0SbG1+VbbgcxjubE7Cc2Pb6QqhG3NH6X=WwsNfqNjtA@mail.gmail.com/

[3] https://lore.kernel.org/lkml/aC5lL_qY00vd8qp4@agluck-desk3/
[4] https://lore.kernel.org/lkml/[email protected]/

maintainer
#109 lore raw
In reply to Peter Newman
Quoted 10 lines
Quoted 4 lines

Maybe assign one of the limited supply of RMIDs for these "unmonitored" tasks. Populate a resctrl group named "unmonitored" that lists all the unmonitored tasks in a (read-only) "tasks" file. And supply all the counts for these tasks in normal looking "mon_data" directory.

I needed to switch to an rdtgroup struct pointer rather than hardware IDs in the task structure to indicate group membership[1], otherwise it's not possible to determine which tasks are in a group when it doesn't have a unique HW ID value.

Quoted 4 lines

Also this is required for shared assignment so that changing a group's IDs in a domain only requires updating running tasks rather than needing to search the entire task list, which would lead to the same problem we encountered in mongroup rename[2].

Having a pointer to the rdtgroup in the task structure does make file system operations easier. But the cost appears to be more complexity (and memory references) in the context switch code.

Your patch[1] seems to do some extra work outside of the static_branch protected sections. So has a cost to context switch even if resctrl is not in use.

Chasing pointers "closid = rgrp->mon.parent->closid;" could be expensive when those miss in the cache.

Quoted 5 lines

-Peter

[1] https://lore.kernel.org/lkml/[email protected]/
[2] https://lore.kernel.org/lkml/CALPaoCh0SbG1+VbbgcxjubE7Cc2Pb6QqhG3NH6X=WwsNfqNjtA@mail.gmail.com/

-Tony

author reviewer
#110 lore raw
In reply to Reinette Chatre

Hi Reinette,

Quoted 127 lines · Reinette Chatre

Hi Peter and Babu,

Quoted 83 lines · Peter Newman

Hi Babu/Reinette,

Quoted 79 lines · Reinette Chatre

Hi Babu,

Quoted 75 lines · Babu

Hi Reinette,

Quoted 60 lines · Reinette Chatre

Hi Babu,

Quoted 47 lines · Babu

Hi Reinette,

Quoted 29 lines · Reinette Chatre

Hi Babu,

Quoted 2 lines · Babu
Quoted 1 line · Peter Newman

On Fri, May 16, 2025 at 12:52 AM Babu Moger <[email protected]> wrote:

...

Quoted 21 lines
Quoted 11 lines
Quoted 2 lines

/sys/fs/resctrl/info/L3_MON/num_mbm_cntrs: Reports the number of monitoring counters available for assignment.

Earlier I discussed with Reinette[1] what num_mbm_cntrs should represent in a "soft-ABMC" implementation where assignment is implemented by assigning an RMID, which would result in all events being assigned at once.

My main concern is how many "counters" you can assign by assigning RMIDs. I recall Reinette proposed reporting the number of groups which can be assigned separately from counters which can be assigned.

More context may be needed here. Currently, num_mbm_cntrs indicates the number of counters available per domain, which is 32.

At the moment, we can assign 2 counters to each group, meaning each RMID can be associated with 2 hardware counters. In theory, it's possible to assign all 32 hardware counters to a group—allowing one RMID to be linked with up to 32 counters. However, we currently lack the interface to support that level of assignment.

Quoted 8 lines
Quoted 2 lines

For now, the plan is to support basic assignment and expand functionality later once we have the necessary data structure and requirements.

Looks like some requirements did not make it into this implementation. Do you recall the discussion that resulted in you writing [2]? Looks like there is a question to Peter in there on how to determine how many "counters" are available in soft-ABMC. I interpreted [3] at that time to mean that this information would be available in a future AMD publication.

We already have a method to determine the number of counters in soft-ABMC mode, which Peter has addressed [4].

[4]
https://lore.kernel.org/lkml/[email protected]/

Quoted 6 lines

This appears to be more of a workaround, and I doubt it will be included in any official AMD documentation. Additionally, the long-term direction is moving towards ABMC.

I don’t believe this workaround needs to be part of the current series. It can be added later when soft-ABMC is implemented.

Agreed. What about the plans described in [2]? (Thanks to Peter for catching this!).

Quoted 9 lines

It is important to keep track of requirements while working on a feature to ensure that the implementation supports the planned use cases. Re-reading that thread it is not clear to me how soft-ABMC's per-group assignment would look. Could you please share how you see it progress from this implementation? This includes the single event vs. multiple event assignment. I would like to highlight that this is not a request for this to be supported in this implementation but there needs to be a plan for how this can be supported on top of interfaces established by this work.

Here’s my current understanding of soft-ABMC. Peter may have a more in-depth perspective on this.

Quoted 8 lines
Quoted 2 lines
Quoted 2 lines

Soft-ABMC:
a. num_mbm_cntrs: This is a software-defined limit based on the number of active RMIDs that can be supported. The value can be obtained using the code referenced in [4].

I would call it a hardware-defined limit that can be probed by software.

The main question is whether this file returns the exact number of RMIDs hardware can track or double that number (mbm_total_bytes + mbm_local_bytes) so that the value is always measured in events.

tl;dr: I continue [3] to find it most intuitive for num_mbm_cntrs to be the exact number of "active" RMIDs that the system can support *and* changing the name of the modes to help user interpret num_mbm_cntrs: "mbm_cntr_event_assign" for ABMC, "mbm_cntr_group_assign" for soft-ABMC.

details
-------

We are now back to the previous discussion about what user can expect from the interface. Let me try and re-cap that discussion so that we can all hopefully get back on the same page. Please add corrections/updates where needed.

soft-ABMC
---------
soft-ABMC manages "active" (term TBD) RMID assignment to monitor groups. When an "active" RMID is assigned to a monitor group then *all* MBM events (not LLC occupancy) in that monitor group are counted. "Active" RMID assignment can be done per domain.

Requirement: resctrl should accurately reflect which events are counted. That is, we do not want resctrl to pretend to allow user to assign an "active" RMID to only one event in a monitor group while all events are actually counted.

Caveat: To support rapid re-assignment of RMIDs to monitor groups, llc_occupancy event is disabled when soft-ABMC is enabled.

ABMC
----
ABMC manages (hardware) counter assignment to monitor group (RMID), event pairs. When a hardware counter is assigned to an RMID, event pair then only that RMID, event is counted. Hardware counter assignment can be done per domain.

Quoted 6 lines

shared assignment
-----------------
A shared assignment applies to both soft-ABMC and ABMC. A user can designate a "counter" (could be hardware counter or "active" RMID) as shared and that means the counter within that domain is shared between different monitor groups and actual assignment is scheduled by resctrl.

Good summary: Thanks.

Quoted 102 lines

user interface
--------------

Next, consider the interface while keeping above definitions and requirements in mind.

This series introduces (using implementation, not cover-letter):

/sys/fs/resctrl/info/L3_MON/num_mbm_cntrs
"num_mbm_cntrs":
The maximum number of monitoring counters (total of available and assigned counters) in each domain when the system supports mbm_cntr_assign mode.

/sys/fs/resctrl/mbm_L3_assignments
"mbm_L3_assignments":
This interface file is created when the mbm_cntr_assign mode is supported and shows the assignment status for each group.

Consider "mbm_L3_assignments" first. The interface is documented for ABMC support where it is possible to manage individual event assignment within monitor group.

For ABMC it is possible to assign just one event at a time and doing so consumes one counter in that domain:

a) Starting state on system with 32 counters per domain, two events in default resource group consumes two counters in that domain:

# cat /sys/fs/resctrl/info/L3_MON/num_mbm_cntrs
0=30;1=32
# cat /sys/fs/resctrl/mbm_L3_assignments
mbm_total_bytes:0=e;1=_
mbm_local_bytes:0=e;1=_

b) Assign counter to mbm_local_bytes in domain 1:

# echo "mbm_local_bytes:1=e" > /sys/fs/resctrl/mbm_L3_assignments
# cat /sys/fs/resctrl/mbm_L3_assignments
mbm_total_bytes:0=e;1=_
mbm_local_bytes:0=e;1=e
# cat /sys/fs/resctrl/info/L3_MON/num_mbm_cntrs
0=30;1=31

The question is how this should look on soft-ABMC system. Let's say hypothetically that on a soft-ABMC system it is possible to have 32 "active" RMIDs.

a) Starting state on system with 32 "active RMIDs" per domain, two events in default resource group consumes one RMID in that domain:

# cat /sys/fs/resctrl/mbm_L3_assignments
mbm_total_bytes:0=e;1=_
mbm_local_bytes:0=e;1=_

What should num_mbm_cntrs display?

Option A (counters are RMIDs):

# cat /sys/fs/resctrl/info/L3_MON/num_mbm_cntrs
0=31;1=32

Option B (pretend RMIDs are events):

# cat /sys/fs/resctrl/info/L3_MON/num_mbm_cntrs
0=62;1=64

b) Assign counter to mbm_local_bytes in domain 1:

# echo "mbm_local_bytes:1=e" > /sys/fs/resctrl/mbm_L3_assignments
# cat /sys/fs/resctrl/mbm_L3_assignments
mbm_total_bytes:0=e;1=e
mbm_local_bytes:0=e;1=e

Note that even though user requested only mbm_local_bytes to be assigned, it actually results in both mbm_total_bytes and mbm_local_bytes to be assigned. This ensures accurate state representation to user space but this also creates an inconsistent user interface between soft-ABMC and ABMC since user space intends to use the same interface but "sometimes" assigning one event results in assign of one event while "sometimes" it results in assign of multiple events.

wrt "num_mbm_cntrs"

Option A (counters are RMIDs):

# cat /sys/fs/resctrl/info/L3_MON/num_mbm_cntrs
0=31;1=31

Option B (pretend RMIDs are events):

# cat /sys/fs/resctrl/info/L3_MON/num_mbm_cntrs
0=62;1=62 

Neither option seems ideal to me since the interface cannot be consistent between ABMC and soft-ABMC.
As I mentioned in [2] it is not possible to hide ABMC and soft-ABMC behind the same interface. When user space wants to monitor a particular monitor group then it should be clear how that can be accomplished. Not knowing if an assignment/unassignment to/from an event would impact one or all events and whether it will consume one or multiple counters does not sound like a good interface to me.

As I understand current interface, user is required to know how ABMC and soft-ABMC is implemented to be able to configure the system. For example, if user has file like:

	# cat /sys/fs/resctrl/mbm_L3_assignments
	mbm_total_bytes:0=e;1=e
	mbm_local_bytes:0=e;1=e
user must know underlying implementation to be able to manage monitoring of
events and assigning counters otherwise it will be a surprise to lose monitoring
of all events when unassigning one event.
Quoted 4 lines

This is why I proposed in [3] that the name of the mode reflects how user can interact with the system. Instead of one "mbm_cntr_assign" mode there can be "mbm_cntr_event_assign" that is used for ABMC and "mbm_cntr_group_assign" that is used for soft-ABMC. The mode should make it clear what the system is capable of wrt counter assignments.

Yes, that makes sense. Perhaps we can also simplify it further:

# cat /sys/fs/resctrl/info/L3_MON/mbm_assign_mode:
[mbm_cntr_evt_assign] <- for ABMC
 mbm_cntr_grp_assign  <- for soft-ABMC
Quoted 7 lines

Considering this the interface should be clear:
num_mbm_cntrs: reflects the number of counters in each domain that can be assigned. In "mbm_cntr_event_assign" this will be the number of counters that can be assigned to each event within a monitoring group, in "mbm_cntr_group_assign" this will be the number of counters that can be assigned to entire monitoring groups impacting all MBM events.

Quoted 8 lines

mbm_L3_assignments: manages the counter assignment in each group. When user knows the mode is "mbm_cntr_event_assign"/"mbm_cntr_group_assign" then it should be clear to user space how the interface behaves wrt assignment, no surprises of multiple events impacted when assigning/unassigning single event.

For soft-ABMC I thus find it most intuitive for num_mbm_cntrs to be the exact number of "active" RMIDs that the system can support *and* changing the name of the modes to help user interpret num_mbm_cntrs.

Sure. The option A: fits well here then.

Option A (counters are RMIDs):

 # cat /sys/fs/resctrl/info/L3_MON/num_mbm_cntrs
 0=31;1=31
Quoted 54 lines
Quoted 6 lines

There's also the mongroup-RMID overcommit use case I described above[1]. On Intel we can safely assume that there are counters to back all RMIDs, so num_mbm_cntrs would be calculated directly from num_rmids.

This is about the:
There's now more interest in Google for allowing explicit control of where RMIDs are assigned on Intel platforms. Even though the number of RMIDs implemented by hardware tends to be roughly the number of containers they want to support, they often still need to create containers when all RMIDs have already been allocated, which is not currently allowed. Once the container has been created and starts running, it's no longer possible to move its threads into a monitoring group whenever RMIDs should become available again, so it's important for resctrl to maintain an accurate task list for a container even when RMIDs are not available.

I see a monitor group as a collection of tasks that need to be monitored together. The "task list" is the group of tasks that share a monitoring ID that is required to be a valid ID since when any of the tasks are scheduled that ID is written to the hardware. I intentionally tried to not use RMID since I believe this is required for all archs.
I thus do not understand how a task can start running when it does not have a valid monitoring ID. The idea of "deferred assignment" is not clear to me, there can never be "unmonitored tasks", no? I think I am missing something here.

Quoted 16 lines

I realized this use case is more difficult to implement on MPAM, because a PARTID is effectively a CLOSID+RMID, so deferring assigning a unique PARTID to a group also results in it being in a different allocation group. It will work if the unmonitored groups could find a way to share PARTIDs, but this has consequences on allocation - but hopefully no worse than sharing CLOSIDs on x86.

There's a lot of interest in monitoring ID overcommit in Google, so I think it's worth it for me to investigate the additional structural changes needed in resctrl (i.e., breaking the FS-level association between mongroups and HW monitoring IDs). Such a framework could be a better fit for soft-ABMC. For example, if overcommit is allowed, we would just report the number of simultaneous RMIDs we were able to probe as num_rmids. I would want the same shared assignment scheduler to be able to work with RMIDs and counters, though.

Quoted 4 lines

Thanks,
-Peter

[1] https://lore.kernel.org/lkml/CALPaoChSzzU5mzMZsdT6CeyEn0WD1qdT9fKCoNW_ty4tojtrkw@mail.gmail.com/

Reinette

[2] https://lore.kernel.org/lkml/[email protected]/
[3] https://lore.kernel.org/lkml/[email protected]/

Signature · 2 lines
Thanks
Babu Moger
maintainer
#111 lore raw

Hi Babu,

Quoted 1 line · Babu

On 5/21/25 18:03, Reinette Chatre wrote:

...

Quoted 10 lines
Quoted 4 lines

This is why I proposed in [3] that the name of the mode reflects how user can interact with the system. Instead of one "mbm_cntr_assign" mode there can be "mbm_cntr_event_assign" that is used for ABMC and "mbm_cntr_group_assign" that is used for soft-ABMC. The mode should make it clear what the system is capable of wrt counter assignments.

Yes, that makes sense. Perhaps we can also simplify it further:

# cat /sys/fs/resctrl/info/L3_MON/mbm_assign_mode:
[mbm_cntr_evt_assign] <- for ABMC
 mbm_cntr_grp_assign  <- for soft-ABMC

Looks good to me. Thank you.

Quoted 15 lines
Quoted 6 lines

Considering this the interface should be clear:
num_mbm_cntrs: reflects the number of counters in each domain that can be assigned. In "mbm_cntr_event_assign" this will be the number of counters that can be assigned to each event within a monitoring group, in "mbm_cntr_group_assign" this will be the number of counters that can be assigned to entire monitoring groups impacting all MBM events.

Quoted 8 lines

mbm_L3_assignments: manages the counter assignment in each group. When user knows the mode is "mbm_cntr_event_assign"/"mbm_cntr_group_assign" then it should be clear to user space how the interface behaves wrt assignment, no surprises of multiple events impacted when assigning/unassigning single event.

For soft-ABMC I thus find it most intuitive for num_mbm_cntrs to be the exact number of "active" RMIDs that the system can support *and* changing the name of the modes to help user interpret num_mbm_cntrs.

Quoted 5 lines

Sure. The option A: fits well here then.

Option A (counters are RMIDs):

 # cat /sys/fs/resctrl/info/L3_MON/num_mbm_cntrs
 0=31;1=31

Thank you for considering.

Please add the requirements from this discussion to your running list. Also please keep in mind how soft-ABMC intends to use the interfaces created by this work so that the documentation that accompanies the ABMC support in this series leaves enough "wiggle room" for soft-ABMC to be built on top.

Quoted 8 lines
Quoted 7 lines
Quoted 1 line

[1] https://lore.kernel.org/lkml/CALPaoChSzzU5mzMZsdT6CeyEn0WD1qdT9fKCoNW_ty4tojtrkw@mail.gmail.com/

Reinette

[2] https://lore.kernel.org/lkml/[email protected]/
[3] https://lore.kernel.org/lkml/[email protected]/

Reinette

author reviewer
#112 lore raw

Hi Reinette,

Quoted 41 lines · Reinette Chatre

Hi Babu,

Quoted 1 line · Babu

On 5/21/25 18:03, Reinette Chatre wrote:

...

Quoted 10 lines
Quoted 4 lines

This is why I proposed in [3] that the name of the mode reflects how user can interact with the system. Instead of one "mbm_cntr_assign" mode there can be "mbm_cntr_event_assign" that is used for ABMC and "mbm_cntr_group_assign" that is used for soft-ABMC. The mode should make it clear what the system is capable of wrt counter assignments.

Yes, that makes sense. Perhaps we can also simplify it further:

# cat /sys/fs/resctrl/info/L3_MON/mbm_assign_mode:
[mbm_cntr_evt_assign] <- for ABMC
 mbm_cntr_grp_assign  <- for soft-ABMC

Looks good to me. Thank you.

Quoted 15 lines
Quoted 6 lines

Considering this the interface should be clear:
num_mbm_cntrs: reflects the number of counters in each domain that can be assigned. In "mbm_cntr_event_assign" this will be the number of counters that can be assigned to each event within a monitoring group, in "mbm_cntr_group_assign" this will be the number of counters that can be assigned to entire monitoring groups impacting all MBM events.

Quoted 8 lines

mbm_L3_assignments: manages the counter assignment in each group. When user knows the mode is "mbm_cntr_event_assign"/"mbm_cntr_group_assign" then it should be clear to user space how the interface behaves wrt assignment, no surprises of multiple events impacted when assigning/unassigning single event.

For soft-ABMC I thus find it most intuitive for num_mbm_cntrs to be the exact number of "active" RMIDs that the system can support *and* changing the name of the modes to help user interpret num_mbm_cntrs.

Quoted 5 lines

Sure. The option A: fits well here then.

Option A (counters are RMIDs):

 # cat /sys/fs/resctrl/info/L3_MON/num_mbm_cntrs
 0=31;1=31
Quoted 5 lines

Thank you for considering.

Please add the requirements from this discussion to your running list. Also please keep in mind how soft-ABMC intends to use the interfaces created by this work so that the documentation that accompanies the ABMC support in this series leaves enough "wiggle room" for soft-ABMC to be built on top.

Sure. Thanks

Quoted 12 lines
Quoted 9 lines
Quoted 8 lines
Quoted 2 lines

[1] https://lore.kernel.org/lkml/CALPaoChSzzU5mzMZsdT6CeyEn0WD1qdT9fKCoNW_ty4tojtrkw@mail.gmail.com/

Reinette

[2] https://lore.kernel.org/lkml/[email protected]/
[3] https://lore.kernel.org/lkml/[email protected]/

Reinette

Signature · 2 lines
Thanks
Babu Moger
#113 lore raw
In reply to Reinette Chatre

Hi Reinette,

Quoted 12 lines · Reinette Chatre

Hi Babu,

Quoted 1 line · Babu

On 5/21/25 18:03, Reinette Chatre wrote:

...

Quoted 5 lines
Quoted 4 lines

This is why I proposed in [3] that the name of the mode reflects how user can interact with the system. Instead of one "mbm_cntr_assign" mode there can be "mbm_cntr_event_assign" that is used for ABMC and "mbm_cntr_group_assign" that is used for soft-ABMC. The mode should make it clear what the system is capable of wrt counter assignments.

Quoted 7 lines
Quoted 5 lines

Yes, that makes sense. Perhaps we can also simplify it further:

# cat /sys/fs/resctrl/info/L3_MON/mbm_assign_mode:
[mbm_cntr_evt_assign] <- for ABMC
  mbm_cntr_grp_assign  <- for soft-ABMC

Looks good to me. Thank you.

I am actually ready with v14 series. I have good feeling that we are getting closer to making these changes final.

So, Looking back again, it might make more sense to rename few user visible interfaces.

1. # cat /sys/fs/resctrl/info/L3_MON/mbm_assign_mode. [mbm_assign_event] <- for ABMC mbm_assign_group <- for soft-ABMC

This looks much more cleaner. It matches with "mbm_assign_mode"

Similarly, we can rename few functions and variable names to make little more readable.

2. mbm_cntr_assignable -> mbm_assignable

3. resctrl_arch_mbm_cntr_assign_enabled
-> >resctrl_arch_mbm_assign_enabled

4. mbm_cntr_assign_enabled -> mbm_assign_enabled

5. resctrl_arch_mbm_cntr_assign_set_one ->

resctrl_arch_mbm_assign_set_one.

6. There will few more functions. I will look into that if you agree with approach.

7. No need to change few of these below. These are related to actual counters. num_mbm_cntrs
available_mbm_cntrs

What do you think?

Thanks
Babu Moger

maintainer
#114 lore raw

Hi Babu,

Quoted 25 lines · Babu

Hi Reinette,

Quoted 12 lines · Reinette Chatre

Hi Babu,

Quoted 1 line · Babu

On 5/21/25 18:03, Reinette Chatre wrote:

...

Quoted 5 lines
Quoted 4 lines

This is why I proposed in [3] that the name of the mode reflects how user can interact with the system. Instead of one "mbm_cntr_assign" mode there can be "mbm_cntr_event_assign" that is used for ABMC and "mbm_cntr_group_assign" that is used for soft-ABMC. The mode should make it clear what the system is capable of wrt counter assignments.

Quoted 7 lines
Quoted 5 lines

Yes, that makes sense. Perhaps we can also simplify it further:

# cat /sys/fs/resctrl/info/L3_MON/mbm_assign_mode:
[mbm_cntr_evt_assign] <- for ABMC
  mbm_cntr_grp_assign  <- for soft-ABMC

Looks good to me. Thank you.

I am actually ready with v14 series. I have good feeling that we are getting closer to making these changes final.

Quoted 7 lines

So, Looking back again, it might make more sense to rename few user visible interfaces.

1. # cat /sys/fs/resctrl/info/L3_MON/mbm_assign_mode. [mbm_assign_event] <- for ABMC mbm_assign_group  <- for soft-ABMC

   This looks much more cleaner.  It matches with "mbm_assign_mode"

ah, I see, by dropping "cntr" it reduces confusion where ABMC assigns counters and soft-ABMC assigned RMID. This looks good.

Taking this further, the "assign" term in "mbm_assign_event" and "mbm_assign_group" may also be redundant considering that the filename, "mbm_assign_mode", already has "assign" in its name.

Quoted 5 lines

Similarly, we can rename few functions and variable names to make little more readable.

2. mbm_cntr_assignable -> mbm_assignable

I have no insight into how the soft-ABMC implementation will look and thus if it will build on this property. If soft-ABMC uses the property then making it more generic may help, but if it does not then it may make the code harder to read. Since this is all internal I'd vote for keeping it mbm_cntr_assignable since the current implementation directly associates it with hardware counters. I do not know if there will be a scenario where a system may support *both* event and group assignable counters. The idea did briefly come up[1]. If that may be possible then resctrl would need to distinguish them. Also, interesting to note that the example used in (1) above notes a system that supports both event and group assignment.

Quoted 3 lines

3. resctrl_arch_mbm_cntr_assign_enabled
 -> >resctrl_arch_mbm_assign_enabled

This is directly connected to choice for (2)

Quoted 1 line

4. mbm_cntr_assign_enabled -> mbm_assign_enabled

hmmm ... here mbm_cntr_assign_enabled is even more directly associated with hardware support for counter assignment. It is not clear what the benefit is to make it generic.

Quoted 4 lines

5. resctrl_arch_mbm_cntr_assign_set_one ->

   resctrl_arch_mbm_assign_set_one.

Same as (4)

Quoted 8 lines

6. There will few more functions. I will look into that if you agree with approach.

7. No need to change few of these below. These are related to actual counters. num_mbm_cntrs
   available_mbm_cntrs

What do you think?

It sounds to me as though you are aiming to make the ABMC implementation more generic in preparation for soft-ABMC support. If you have insight into the soft-ABMC implementation then please share the details for this to be taken into account. Until then I think it will be simpler for the implementation to be specific to the feature being enabled here. When soft-ABMC enabling arrives the needed changes can be made. Since this is about internals of resctrl (not the user interface) we are not as pressured to "get it right" while not having all information required to make these choices.

Reinette

[1] https://lore.kernel.org/lkml/CALPaoCj438UfH3QA_VnGo-pj2a_48sJufUWjBKT3MQatcMJ_Uw@mail.gmail.com/

#115 lore raw

Hi Reinette,

Quoted 34 lines · Reinette Chatre

Hi Babu,

Quoted 31 lines · Babu

Hi Reinette,

Quoted 12 lines · Reinette Chatre

Hi Babu,

Quoted 1 line · Babu

On 5/21/25 18:03, Reinette Chatre wrote:

...

Quoted 5 lines
Quoted 4 lines

This is why I proposed in [3] that the name of the mode reflects how user can interact with the system. Instead of one "mbm_cntr_assign" mode there can be "mbm_cntr_event_assign" that is used for ABMC and "mbm_cntr_group_assign" that is used for soft-ABMC. The mode should make it clear what the system is capable of wrt counter assignments.

Quoted 7 lines
Quoted 5 lines

Yes, that makes sense. Perhaps we can also simplify it further:

# cat /sys/fs/resctrl/info/L3_MON/mbm_assign_mode:
[mbm_cntr_evt_assign] <- for ABMC
   mbm_cntr_grp_assign  <- for soft-ABMC

Looks good to me. Thank you.

I am actually ready with v14 series. I have good feeling that we are getting closer to making these changes final.

So, Looking back again, it might make more sense to rename few user visible interfaces.

1. # cat /sys/fs/resctrl/info/L3_MON/mbm_assign_mode. [mbm_assign_event] <- for ABMC mbm_assign_group  <- for soft-ABMC

Quoted 7 lines
Quoted 1 line

   This looks much more cleaner.  It matches with "mbm_assign_mode"

ah, I see, by dropping "cntr" it reduces confusion where ABMC assigns counters and soft-ABMC assigned RMID. This looks good.

Taking this further, the "assign" term in "mbm_assign_event" and "mbm_assign_group" may also be redundant considering that the filename, "mbm_assign_mode", already has "assign" in its name.

ok. Sure. It will be

# cat /sys/fs/resctrl/info/L3_MON/mbm_assign_mode.
  [mbm_event] <- for ABMC
   mbm_group  <- for soft-ABMC
Quoted 7 lines
Quoted 6 lines

Similarly, we can rename few functions and variable names to make little more readable.

2. mbm_cntr_assignable -> mbm_assignable

Quoted 9 lines

I have no insight into how the soft-ABMC implementation will look and thus if it will build on this property. If soft-ABMC uses the property then making it more generic may help, but if it does not then it may make the code harder to read. Since this is all internal I'd vote for keeping it mbm_cntr_assignable since the current implementation directly associates it with hardware counters. I do not know if there will be a scenario where a system may support *both* event and group assignable counters. The idea did briefly come up[1]. If that may be possible then resctrl would need to distinguish them. Also, interesting to note that the example used in (1) above notes a system that supports both event and group assignment.

Ok. That is fine. Lets keep it as is then.

Quoted 6 lines
Quoted 4 lines

3. resctrl_arch_mbm_cntr_assign_enabled
 -> >resctrl_arch_mbm_assign_enabled

This is directly connected to choice for (2)

Ok.

Quoted 5 lines
Quoted 2 lines

4. mbm_cntr_assign_enabled -> mbm_assign_enabled

hmmm ... here mbm_cntr_assign_enabled is even more directly associated with hardware support for counter assignment. It is not clear what the benefit is to make it generic.

Ok.

Quoted 17 lines
Quoted 5 lines

5. resctrl_arch_mbm_cntr_assign_set_one ->

   resctrl_arch_mbm_assign_set_one.

Same as (4)

Quoted 8 lines

6. There will few more functions. I will look into that if you agree with approach.

7. No need to change few of these below. These are related to actual counters. num_mbm_cntrs
   available_mbm_cntrs

What do you think?

Quoted 8 lines

It sounds to me as though you are aiming to make the ABMC implementation more generic in preparation for soft-ABMC support. If you have insight into the soft-ABMC implementation then please share the details for this to be taken into account. Until then I think it will be simpler for the implementation to be specific to the feature being enabled here. When soft-ABMC enabling arrives the needed changes can be made. Since this is about internals of resctrl (not the user interface) we are not as pressured to "get it right" while not having all information required to make these choices.

Ok. Sure. That is fine. Lets keep the internals to implementation specific for now.

Thanks
Babu

Quoted 5 lines

Reinette

[1] https://lore.kernel.org/lkml/CALPaoCj438UfH3QA_VnGo-pj2a_48sJufUWjBKT3MQatcMJ_Uw@mail.gmail.com/

#116 lore raw
In reply to Reinette Chatre

Hi Reinette,

Quoted 97 lines · Reinette Chatre

Hi Babu,

Quoted 93 lines · Babu

Hi Reinette,

Quoted 60 lines · Reinette Chatre

Hi Babu,

Quoted 47 lines · Babu

Hi Reinette,

Quoted 29 lines · Reinette Chatre

Hi Babu,

Quoted 2 lines · Babu
Quoted 1 line · Peter Newman

On Fri, May 16, 2025 at 12:52 AM Babu Moger <[email protected]> wrote:

...

Quoted 21 lines
Quoted 11 lines
Quoted 2 lines

/sys/fs/resctrl/info/L3_MON/num_mbm_cntrs: Reports the number of monitoring counters available for assignment.

Earlier I discussed with Reinette[1] what num_mbm_cntrs should represent in a "soft-ABMC" implementation where assignment is implemented by assigning an RMID, which would result in all events being assigned at once.

My main concern is how many "counters" you can assign by assigning RMIDs. I recall Reinette proposed reporting the number of groups which can be assigned separately from counters which can be assigned.

More context may be needed here. Currently, num_mbm_cntrs indicates the number of counters available per domain, which is 32.

At the moment, we can assign 2 counters to each group, meaning each RMID can be associated with 2 hardware counters. In theory, it's possible to assign all 32 hardware counters to a group—allowing one RMID to be linked with up to 32 counters. However, we currently lack the interface to support that level of assignment.

Quoted 8 lines
Quoted 2 lines

For now, the plan is to support basic assignment and expand functionality later once we have the necessary data structure and requirements.

Looks like some requirements did not make it into this implementation. Do you recall the discussion that resulted in you writing [2]? Looks like there is a question to Peter in there on how to determine how many "counters" are available in soft-ABMC. I interpreted [3] at that time to mean that this information would be available in a future AMD publication.

We already have a method to determine the number of counters in soft-ABMC mode, which Peter has addressed [4].

[4]
https://lore.kernel.org/lkml/[email protected]/

Quoted 6 lines

This appears to be more of a workaround, and I doubt it will be included in any official AMD documentation. Additionally, the long-term direction is moving towards ABMC.

I don’t believe this workaround needs to be part of the current series. It can be added later when soft-ABMC is implemented.

Agreed. What about the plans described in [2]? (Thanks to Peter for catching this!).

Quoted 9 lines

It is important to keep track of requirements while working on a feature to ensure that the implementation supports the planned use cases. Re-reading that thread it is not clear to me how soft-ABMC's per-group assignment would look. Could you please share how you see it progress from this implementation? This includes the single event vs. multiple event assignment. I would like to highlight that this is not a request for this to be supported in this implementation but there needs to be a plan for how this can be supported on top of interfaces established by this work.

Here’s my current understanding of soft-ABMC. Peter may have a more in-depth perspective on this.

Soft-ABMC:
a. num_mbm_cntrs: This is a software-defined limit based on the number of active RMIDs that can be supported. The value can be obtained using the code referenced in [4].

b. Assignments: No hardware configuration is required. We simply need to ensure that no more than num_mbm_cntrs RMIDs are active at any given time.

c. Configuration: Controlled via /info/L3_MON/mbm_total_bytes_config and mbm_local_bytes_config.

d. Events: Only two events can be assigned(local and total).

ABMC:
a. num_mbm_cntrs: This is defined by the hardware.
b. Assignments: Requires special MSR writes to assign counters.
c. Configuration: Comes from /info/L3_MON/counter_configs/.
d. Events: More than two events can be assigned to a group (currently up to 2).

Commonalities:
a. Assignments can be either exclusive or shared in both these modes.

Quoted 8 lines
Quoted 1 line

Given these, I believe we can easily accommodate soft-ABMC in this interface.

This is not so obvious to me. It looks to me as though the user is forced to interpret the content of resctrl files differently based on soft-ABMC vs ABMC making the interface inconsistent and user thus needing to know details of implementations. This is what the previous discussion I linked to aimed to address. It sounds to me as though you believe that this is no longer an issue. Could you please show examples of what a user can expect from the interfaces and how a user will interact with the interfaces on both a non-ABMC and ABMC system?

At the interface level, I think mbm_L3_assignments on a non-ABMC system would only need to contain a single line:

0=s;1=s;...;31=s

But maybe for consistency we would synthesize a single, unmodifiable counter configuration to reflect that allocating an RMID in a domain results in assignment to all events and deallocating the RMID unassigns all events. We could call it "group" to say it's assigning at the group level, or perhaps just '*':

*:0=s;1=s;...;31=s

I'm not sure about allowing a '*' on ABMC hardware, because it could be interpreted as allocating a lot of counters when a large number of event configurations exist.

*:0=s;1=s;...;31=s

-Peter

Quoted 14 lines

Thank you

Reinette

Quoted 8 lines
Quoted 7 lines
Quoted 5 lines
Quoted 4 lines

[2] https://lore.kernel.org/lkml/[email protected]/
[3] https://lore.kernel.org/lkml/CALPaoCg3KpF94g2MEmfP_Ro2mQZYFA8sKVkmb+7isotKNgdY9A@mail.gmail.com/

maintainer
#117 lore raw

Hi Peter,

Quoted 3 lines · Peter Newman
Quoted 1 line · Reinette Chatre

On 5/20/25 4:25 PM, Moger, Babu wrote:

...

Quoted 30 lines
Quoted 21 lines
Quoted 21 lines

Here’s my current understanding of soft-ABMC. Peter may have a more in-depth perspective on this.

Soft-ABMC:
a. num_mbm_cntrs: This is a software-defined limit based on the number of active RMIDs that can be supported. The value can be obtained using the code referenced in [4].

b. Assignments: No hardware configuration is required. We simply need to ensure that no more than num_mbm_cntrs RMIDs are active at any given time.

c. Configuration: Controlled via /info/L3_MON/mbm_total_bytes_config and mbm_local_bytes_config.

d. Events: Only two events can be assigned(local and total).

ABMC:
a. num_mbm_cntrs: This is defined by the hardware.
b. Assignments: Requires special MSR writes to assign counters.
c. Configuration: Comes from /info/L3_MON/counter_configs/.
d. Events: More than two events can be assigned to a group (currently up to 2).

Commonalities:
a. Assignments can be either exclusive or shared in both these modes.

Quoted 8 lines
Quoted 1 line

Given these, I believe we can easily accommodate soft-ABMC in this interface.

This is not so obvious to me. It looks to me as though the user is forced to interpret the content of resctrl files differently based on soft-ABMC vs ABMC making the interface inconsistent and user thus needing to know details of implementations. This is what the previous discussion I linked to aimed to address. It sounds to me as though you believe that this is no longer an issue. Could you please show examples of what a user can expect from the interfaces and how a user will interact with the interfaces on both a non-ABMC and ABMC system?

Quoted 4 lines

At the interface level, I think mbm_L3_assignments on a non-ABMC system would only need to contain a single line:

0=s;1=s;...;31=s

It should be obvious to user space how to interpret the fields. When there is thus a single "mbm_cntr_assign" mode used for ABMC and soft-ABMC a single line like this would be difficult to parse since that would imply/require that user space knows whether it is running on ABMC or soft-ABMC system, which we should avoid.

If there are different modes, for example "mbm_cntr_event_assign" and "mbm_cntr_group_assign" then this could be used by user space to distinguish how to interact with mbm_L3_assignments making something like this possible.

Quoted 7 lines

But maybe for consistency we would synthesize a single, unmodifiable counter configuration to reflect that allocating an RMID in a domain results in assignment to all events and deallocating the RMID unassigns all events. We could call it "group" to say it's assigning at the group level, or perhaps just '*':

Quoted 8 lines

*:0=s;1=s;...;31=s

I'm not sure about allowing a '*' on ABMC hardware, because it could be interpreted as allocating a lot of counters when a large number of event configurations exist.

*:0=s;1=s;...;31=s

Either could work also. Whether it is "group" or "*" ABMC systems could respond with "not supported". Will think about this more but would like to hear your opinion about the flexibility that distinguishing between a "mbm_cntr_event_assign" and "mbm_cntr_group_assign" mode provides.

Reinette

Quoted 17 lines

-Peter

Quoted 14 lines

Thank you

Reinette

Quoted 8 lines
Quoted 7 lines
Quoted 5 lines
Quoted 4 lines

[2] https://lore.kernel.org/lkml/[email protected]/
[3] https://lore.kernel.org/lkml/CALPaoCg3KpF94g2MEmfP_Ro2mQZYFA8sKVkmb+7isotKNgdY9A@mail.gmail.com/

#118 lore raw

Hi Reinette,

Quoted 50 lines · Reinette Chatre

Hi Peter,

Quoted 3 lines · Peter Newman
Quoted 1 line · Reinette Chatre

On 5/20/25 4:25 PM, Moger, Babu wrote:

...

Quoted 30 lines
Quoted 21 lines
Quoted 21 lines

Here’s my current understanding of soft-ABMC. Peter may have a more in-depth perspective on this.

Soft-ABMC:
a. num_mbm_cntrs: This is a software-defined limit based on the number of active RMIDs that can be supported. The value can be obtained using the code referenced in [4].

b. Assignments: No hardware configuration is required. We simply need to ensure that no more than num_mbm_cntrs RMIDs are active at any given time.

c. Configuration: Controlled via /info/L3_MON/mbm_total_bytes_config and mbm_local_bytes_config.

d. Events: Only two events can be assigned(local and total).

ABMC:
a. num_mbm_cntrs: This is defined by the hardware.
b. Assignments: Requires special MSR writes to assign counters.
c. Configuration: Comes from /info/L3_MON/counter_configs/.
d. Events: More than two events can be assigned to a group (currently up to 2).

Commonalities:
a. Assignments can be either exclusive or shared in both these modes.

Quoted 8 lines
Quoted 1 line

Given these, I believe we can easily accommodate soft-ABMC in this interface.

This is not so obvious to me. It looks to me as though the user is forced to interpret the content of resctrl files differently based on soft-ABMC vs ABMC making the interface inconsistent and user thus needing to know details of implementations. This is what the previous discussion I linked to aimed to address. It sounds to me as though you believe that this is no longer an issue. Could you please show examples of what a user can expect from the interfaces and how a user will interact with the interfaces on both a non-ABMC and ABMC system?

Quoted 4 lines

At the interface level, I think mbm_L3_assignments on a non-ABMC system would only need to contain a single line:

0=s;1=s;...;31=s

It should be obvious to user space how to interpret the fields. When there is thus a single "mbm_cntr_assign" mode used for ABMC and soft-ABMC a single line like this would be difficult to parse since that would imply/require that user space knows whether it is running on ABMC or soft-ABMC system, which we should avoid.

Quoted 3 lines

If there are different modes, for example "mbm_cntr_event_assign" and "mbm_cntr_group_assign" then this could be used by user space to distinguish how to interact with mbm_L3_assignments making something like this possible.

I meant to say I was proposing the format of this file when in the group assignment mode. I didn't mean to imply that a separate mode wasn't needed.

Quoted 14 lines
Quoted 14 lines

But maybe for consistency we would synthesize a single, unmodifiable counter configuration to reflect that allocating an RMID in a domain results in assignment to all events and deallocating the RMID unassigns all events. We could call it "group" to say it's assigning at the group level, or perhaps just '*':

*:0=s;1=s;...;31=s

I'm not sure about allowing a '*' on ABMC hardware, because it could be interpreted as allocating a lot of counters when a large number of event configurations exist.

Quoted 7 lines
Quoted 2 lines

*:0=s;1=s;...;31=s

Either could work also. Whether it is "group" or "*" ABMC systems could respond with "not supported". Will think about this more but would like to hear your opinion about the flexibility that distinguishing between a "mbm_cntr_event_assign" and "mbm_cntr_group_assign" mode provides.

I agree it's clearer when they are separate modes. Between "*" and "group", I prefer "group" because it seems the least ambiguous.

I just want to make sure we'd never want both modes at the same time, such as an implementation with both a small number of monitoring IDs and a small number of MBM counters. I support one MPAM implementation that has a small number of PARTIDs and only one MBWU counter per domain. Fingers crossed that the number of PARTIDs it supports isn't small compared to the number of jobs we would run on it. Otherwise maybe it will work out to just pick the more limited of the two (monitor IDs or counters) and make allocation of one drive the other.

(In case you read this before my earlier reply[1], see the note about rdtgroup pointers in the task_struct, as this is a prerequisite for overcommitting HW monitor IDs.)

Thanks,
-Peter

[1] https://lore.kernel.org/lkml/CALPaoCjh_NXQLtNBqei=7a6Jsr17fEnPO+kqMaNq4xNu2UPDJA@mail.gmail.com/

Quoted 21 lines

Reinette

Quoted 17 lines

-Peter

Quoted 14 lines

Thank you

Reinette

Quoted 8 lines
Quoted 7 lines
Quoted 5 lines
Quoted 4 lines

[2] https://lore.kernel.org/lkml/[email protected]/
[3] https://lore.kernel.org/lkml/CALPaoCg3KpF94g2MEmfP_Ro2mQZYFA8sKVkmb+7isotKNgdY9A@mail.gmail.com/

maintainer
#119 lore raw

Hi Peter,

Quoted 3 lines · Peter Newman
Quoted 1 line · Reinette Chatre

On 5/21/25 7:27 AM, Peter Newman wrote:

...

Quoted 11 lines
Quoted 11 lines
Quoted 4 lines

At the interface level, I think mbm_L3_assignments on a non-ABMC system would only need to contain a single line:

0=s;1=s;...;31=s

It should be obvious to user space how to interpret the fields. When there is thus a single "mbm_cntr_assign" mode used for ABMC and soft-ABMC a single line like this would be difficult to parse since that would imply/require that user space knows whether it is running on ABMC or soft-ABMC system, which we should avoid.

Quoted 7 lines
Quoted 3 lines

If there are different modes, for example "mbm_cntr_event_assign" and "mbm_cntr_group_assign" then this could be used by user space to distinguish how to interact with mbm_L3_assignments making something like this possible.

I meant to say I was proposing the format of this file when in the group assignment mode. I didn't mean to imply that a separate mode wasn't needed.

Thanks for confirming.

Quoted 15 lines
Quoted 15 lines
Quoted 14 lines

But maybe for consistency we would synthesize a single, unmodifiable counter configuration to reflect that allocating an RMID in a domain results in assignment to all events and deallocating the RMID unassigns all events. We could call it "group" to say it's assigning at the group level, or perhaps just '*':

*:0=s;1=s;...;31=s

I'm not sure about allowing a '*' on ABMC hardware, because it could be interpreted as allocating a lot of counters when a large number of event configurations exist.

*:0=s;1=s;...;31=s

Quoted 7 lines
Quoted 4 lines

Either could work also. Whether it is "group" or "*" ABMC systems could respond with "not supported". Will think about this more but would like to hear your opinion about the flexibility that distinguishing between a "mbm_cntr_event_assign" and "mbm_cntr_group_assign" mode provides.

I agree it's clearer when they are separate modes. Between "*" and "group", I prefer "group" because it seems the least ambiguous.

Sounds good to me. resctrl will need extra guards to prevent user from creating an event named "group" but this matches what resctrl already needs to do for other parts (eg. user cannot create a monitor group named "mon_groups").

Quoted 3 lines

I just want to make sure we'd never want both modes at the same time, such as an implementation with both a small number of monitoring IDs

hmmm ... my assumption was that a system could only support one of these modes ("mbm_cntr_event_assign" or "mbm_cntr_group_assign") but it could be possible to have both possible on a system. But beyond that to have both active at the *same* time? That will take a lot of wrangling during runtime.

Quoted 6 lines

and a small number of MBM counters. I support one MPAM implementation that has a small number of PARTIDs and only one MBWU counter per domain. Fingers crossed that the number of PARTIDs it supports isn't small compared to the number of jobs we would run on it. Otherwise maybe it will work out to just pick the more limited of the two (monitor IDs or counters) and make allocation of one drive the other.

Could a scenario like this be addressed by "mbm_cntr_event_assign" mode gaining support for "shared assignment"?

Quoted 4 lines

(In case you read this before my earlier reply[1], see the note about rdtgroup pointers in the task_struct, as this is a prerequisite for overcommitting HW monitor IDs.)

Reinette

Quoted 1 line

[1] https://lore.kernel.org/lkml/CALPaoCjh_NXQLtNBqei=7a6Jsr17fEnPO+kqMaNq4xNu2UPDJA@mail.gmail.com/

...

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

[2] https://lore.kernel.org/lkml/[email protected]/
[3] https://lore.kernel.org/lkml/CALPaoCg3KpF94g2MEmfP_Ro2mQZYFA8sKVkmb+7isotKNgdY9A@mail.gmail.com/

maintainer
#120 lore raw
In reply to Babu Moger

Hi Babu,

Quoted 18 lines · Babu Moger

This series adds the support for Assignable Bandwidth Monitoring Counters (ABMC). It is also called QoS RMID Pinning feature

Series is written such that it is easier to support other assignable features supported from different vendors.

The feature details are documented in the APM listed below [1].
[1] AMD64 Architecture Programmer's Manual Volume 2: System Programming Publication # 24593 Revision 3.41 section 19.3.3.3 Assignable Bandwidth Monitoring (ABMC). The documentation is available at

The patches are based on top of commit 92a09c47464d0 (tag: v6.15-rc5, tip/irq/merge) Linux 6.15-rc5 plus
https://lore.kernel.org/lkml/[email protected]/

Quoted 7 lines

It is very clear these patches will go after James's resctrl FS/ARCH restructure. Hoping to avoid one review cycle due to the merge.

# Introduction

Users can create as many monitor groups as RMIDs supported by the hardware. However, bandwidth monitoring feature on AMD system only guarantees that

"bandwidth monitoring feature on AMD system" -> " the bandwidth monitoring feature on AMD systems"? or "the bandwidth monitoring feature on an AMD system". Not sure.

Quoted 7 lines

RMIDs currently assigned to a processor will be tracked by hardware. The counters of any other RMIDs which are no longer being tracked will be reset to zero. The MBM event counters return "Unavailable" for the RMIDs that are not tracked by hardware. So, there can be only limited number of groups that can give guaranteed monitoring numbers. With ever changing configurations there is no way to definitely know which of these groups are being tracked for certain point of time. Users do not have the option

"for certain point of time" -> "during a particular/certain(?) time"?

Quoted 1 line

to monitor a group or set of groups for certain period of time without

"for certain period of time" -> "for a certain period of time"?

This series contains many duplicate snippets. When you update one, please check that all the duplicates are updated also.

Quoted 8 lines

worrying about counter being reset in between.

The ABMC feature provides an option to the user to assign a hardware counter to an RMID, event pair and monitor the bandwidth as long as it is assigned. The assigned RMID will be tracked by the hardware until the user unassigns it manually. There is no need to worry about counters being reset during this period. Additionally, the user can specify a bitmask identifying the specific bandwidth types from the given source to track with the counter.

Instead of tacking it on as an "additionally" I see this capability now as essential to this new implementation. I tried to give this series a thorough review to help finalize this work but I kept being turned around by all the descriptions and finally it dawned that all the descriptions are at their code still based on the original "event ID" based implementation with either a small append or as little change as possible to adjust to the "extended event ID" based implementation.

The previous implementation still used (and copy&pasted many times) in these descriptions as "assign a hardware counter to an RMID, event pair" can only be accurate for this new implementation if an event is re-defined ... it is no longer the original constrained "event IDs" but instead an MBM event has become a generic name that identifies the configurable "bandwidth types" (but, see note about terminology later) to be monitored. This is never done.

I assume "the given source" is the assigned RMID? If so I think it will help to understand if this is specific: "bandwidth types from the assigned RMID ..."

I find this series to use several terms for the same concept,
for example, "bandwidth types", "memory transactions", "types of L3 transactions", "bandwidth sources", etc. This work will be easier to consume if it uses consistent and specific terminology.

Quoted 9 lines

Without ABMC enabled, monitoring will work in current 'default' mode without assignment option.

# History

Earlier implementation of ABMC had dependancy on BMEC (Bandwidth Monitoring Event Configuration). Peter had concerns with that implementation because it may be not be compatible with ARM's MPAM.

Quoted 7 lines

Here are the threads discussing the concerns and new interface to address the concerns.
https://lore.kernel.org/lkml/CALPaoCg97cLVVAcacnarp+880xjsedEWGJPXhYpy4P7=ky4MZw@mail.gmail.com/
https://lore.kernel.org/lkml/CALPaoCiii0vXOF06mfV=kVLBzhfNo0SFqt4kQGwGSGVUqvr2Dg@mail.gmail.com/

Here are the finalized requirements based on the discussion:

* Remove BMEC dependency on the ABMC feature.

Even stronger, BMEC and ABMC are now "incompatible" in that resctrl will not let them be used at the same time.

Quoted 8 lines

* Eliminate global assignment listing. The interface
/sys/fs/resctrl/info/L3_MON/mbm_assign_control is no longer required.

* Create the configuration directories at /sys/fs/resctrl/info/L3_MON/counter_configs/.
The configuration file names should be free-form, allowing users to create them as needed.

* Perform assignment listing at the group level by introducing mbm_L3_assignments

"the group level" -> "the monitoring group level"

Quoted 5 lines

in each monitoring group. The listing should provide the following details:

Event Configuration: Specifies the event configuration applied. This will be crucial when "mkdir" on event configuration is added in the future, leading to the creation of mon_data/mon_l3_*/<event configuration>.

hmmm ... sounds like it has become more natural to refer to it as "event configuration", which is a good match for what the purpose is. This thus sounds like good motivation to change "counter_configs" to "event_configs".

Quoted 4 lines

Domains: Identifies the domains where the configuration is applied, supporting multi-domain setups.

Assignment Type: Indicates whether the assignment is Exclusive (e or d), Shared (s), or Unassigned (_).

Could you please add definition of what "exclusive" and "shared" means?

Quoted 11 lines

* Provide option to enable or disable auto assignment when new group is created.

This series tries to address all the requirements listed above.

# Implementation details

Create a generic interface aimed to support user space assignment of scarce counters used for monitoring. First usage of interface is by ABMC with option to expand usage to "soft-ABMC" and MPAM counters in future.

Quoted 8 lines

Feature adds following interface files:

/sys/fs/resctrl/info/L3_MON/mbm_assign_mode: Reports the list of assignable monitoring features supported. The enclosed brackets indicate which feature is enabled.

/sys/fs/resctrl/info/L3_MON/num_mbm_cntrs: Reports the number of monitoring counters available for assignment.

Please aim to use consistent and clear terms to help understand this work. It is confusing that above uses "available" in description for num_mbm_cntrs and then below there is a new interface "available_mbm_cntrs" that uses the "available" term in name but not description.

Quoted 5 lines

/sys/fs/resctrl/info/L3_MON/available_mbm_cntrs: Reports the number of monitoring counters free in each domain.

/sys/fs/resctrl/info/L3_MON/counter_configs : Directory to hold the counter configuration.

Everywhere else seems to refer to this as "event configurations". Please just stick to one, "event configuration" seems most appropriate.

Quoted 3 lines

/sys/fs/resctrl/info/L3_MON/counter_configs/mbm_total_bytes/event_filter : Default configuration for MBM total events.

I think "default" should be dropped to make it clear that this is the actual configuration that is always used, not a static "default" that may be used in "some" circumstances.

Quoted 3 lines

/sys/fs/resctrl/info/L3_MON/counter_configs/mbm_local_bytes/event_filter : Default configuration for MBM local events.

Same wrt "default"

Quoted 2 lines

/sys/fs/resctrl/mbm_L3_assignments: Interface to list or modify assignment states on each group.

"Per monitor group interface to list or modify counters assigned to the group."? (Please improve.)

Quoted 4 lines

# Examples

a. Check if ABMC support is available

Please drop the "ABMC" from all the descriptions since this is intended to be a generic interface.

Quoted 6 lines

#mount -t resctrl resctrl /sys/fs/resctrl/

# cat /sys/fs/resctrl/info/L3_MON/mbm_assign_mode [mbm_cntr_assign]
default

I believe the naming has been finalized in
https://lore.kernel.org/lkml/[email protected]/.

Quoted 3 lines

ABMC feature is detected and it is enabled.

b. Check how many ABMC counters are available.

available -> supported? This will help distinguish it from the next interface file named "available_mbm_cntrs".

Quoted 3 lines
	# cat /sys/fs/resctrl/info/L3_MON/num_mbm_cntrs 
	32

Please update to reflect what implementation does.

Quoted 2 lines

c. Check how many ABMC counters are available in each domain.

"available" -> "available for assignment"

Quoted 4 lines
	# cat /sys/fs/resctrl/info/L3_MON/available_mbm_cntrs 
	0=30;1=30
Quoted 8 lines

d. Check default counter configuration.

	# cat /sys/fs/resctrl/info/L3_MON/counter_configs/mbm_total_bytes/event_filter 
	local_reads, remote_reads, local_non_temporal_writes, remote_non_temporal_writes,
        local_reads_slow_memory, remote_reads_slow_memory, dirty_victim_writes_all
	# cat /sys/fs/resctrl/info/L3_MON/counter_configs/mbm_local_bytes/event_filter 
	local_reads, local_non_temporal_writes, local_reads_slow_memory

Does not look like this matches implementation wrt spacing?

Quoted 3 lines

e. Series adds a new interface file "mbm_L3_assignments" in each monitoring group
to list and modify any group's monitoring states.

"any group's" -> "that group's"

Quoted 9 lines

The list is displayed in the following format:

<Event configuration>:<Domain id>=<Assignment type>

Event configuration: A valid event configuration listed in the /sys/fs/resctrl/info/L3_MON/counter_configs directory.

Domain ID: A valid domain ID number.

"A valid domain ID number" -> "A valid domain ID"

Quoted 5 lines

Assignment types:

_ : No event configuration assigned

Quoted 8 lines

e : Event configuration assigned in exclusive mode

To list the default group states:
# cat /sys/fs/resctrl/mbm_L3_assignments
mbm_total_bytes:0=e;1=e
mbm_local_bytes:0=e;1=e

To unassign the configuration of mbm_total_bytes on domain 0:

This unassigns a counter, as opposed to a configuration, no? How about "To unassign the counter associated with the mbm_total_bytes event"?

Quoted 2 lines
	#echo "mbm_total_bytes:0=_" > mbm_L3_assignments
	#cat mbm_L3_assignments

(May help to follow if the examples consistently uses full path.)

Quoted 4 lines
	mbm_total_bytes:0=_;1=e
	mbm_local_bytes:0=e;1=e

	To unassign the mbm_total_bytes configuration on all domains:

same wrt unassigning a counter

Quoted 2 lines
    	$echo "mbm_total_bytes:*=_" > mbm_L3_assignments
	$cat mbm_L3_assignments
# prompt is usually used for administrator and $ for user without
administrator privileges. Switching between # and $ in these examples 
is confusing.
Quoted 4 lines
	mbm_total_bytes:0=_;1=_
	mbm_local_bytes:0=e;1=e

	To assign the mbm_total_bytes configuration on all domains in exclusive mode:

same wrt unassigning a counter

Quoted 25 lines
    	$echo "mbm_total_bytes:*=e" > mbm_L3_assignments
	$cat mbm_L3_assignments
	mbm_total_bytes:0=e;1=e
	mbm_local_bytes:0=e;1=e

g. Read the events mbm_total_bytes and mbm_local_bytes of the default group.
There is no change in reading the events with ABMC. If the event is unassigned when reading, then the read will come back as "Unassigned".

	# cat /sys/fs/resctrl/mon_data/mon_L3_00/mbm_total_bytes
	779247936
	# cat /sys/fs/resctrl/mon_data/mon_L3_00/mbm_local_bytes 
	765207488

h. Check the default event configurations.

#cat /sys/fs/resctrl/info/L3_MON/counter_configs/mbm_total_bytes/event_filter local_reads, remote_reads, local_non_temporal_writes, remote_non_temporal_writes, local_reads_slow_memory, remote_reads_slow_memory, dirty_victim_writes_all

#cat /sys/fs/resctrl/info/L3_MON/counter_configs/mbm_local_bytes/event_filter local_reads, local_non_temporal_writes, local_reads_slow_memory

i. Change the event configuration for mbm_local_bytes.

Quoted 7 lines

#echo "local_reads, local_non_temporal_writes, local_reads_slow_memory, remote_reads" > /sys/fs/resctrl/info/L3_MON/counter_configs/mbm_local_bytes/event_filter

#cat /sys/fs/resctrl/info/L3_MON/counter_configs/mbm_local_bytes/event_filter local_reads, local_non_temporal_writes, local_reads_slow_memory, remote_reads

This will update the assignments where mbm_local_bytes are configured.

"This will update all (across all domains of all monitor groups) counter assignments associated with the mbm_local_bytes event." (Please improve).

Quoted 3 lines

j. Now read the total event again. The first read may come back with "Unavailable"
status. The subsequent read of mbm_total_bytes will display only the read events.

Was this intended to be example of reading *local* bytes after modification in previous step?

Quoted 7 lines
	
	#cat /sys/fs/resctrl/mon_data/mon_L3_00/mbm_total_bytes
	Unavailable
	#cat /sys/fs/resctrl/mon_data/mon_L3_00/mbm_total_bytes
	314101

k. Users will have the option to go back to 'default' mbm_assign_mode if required.

"Users will have the option" -> "Users have the option"

Quoted 2 lines

This can be done using the following command. Note that switching the mbm_assign_mode will reset all the MBM counters of all resctrl groups.

"all the MBM counters " -> "all the MBM counters (and thus all MBM events)"?

Quoted 10 lines
	# echo "default" > /sys/fs/resctrl/info/L3_MON/mbm_assign_mode
	# cat /sys/fs/resctrl/info/L3_MON/mbm_assign_mode
	mbm_cntr_assign
	[default]

l. Unmount the resctrl

#umount /sys/fs/resctrl/
---

Reinette