x86/resctrl : Support AMD Assignable Bandwidth Monitoring Counters (ABMC)
Subsystems (3)
Reinette Chatre
maintainerHi Tony,
Quoted 4 lines · Tony
Quoted 3 lines · Reinette Chatre
Hi Peter and Babu,
On 5/21/25 2:18 AM, Peter Newman wrote:
..
Quoted 36 lines
Quoted 17 lines
Quoted 4 lines
There's also the mongroup-RMID overcommit use case I described above[1]. On Intel we can safely assume that there are counters to back all RMIDs, so
num_mbm_cntrswould be calculated directly fromnum_rmids.This is about the:
There's now more interest in Google for allowing explicit control of where RMIDs are assigned on Intel platforms. Even though the number of RMIDs implemented by hardware tends to be roughly the number of containers they want to support, they often still need to create containers when all RMIDs have already been allocated, which is not currently allowed. Once the container has been created and starts running, it's no longer possible to move its threads into a monitoring group whenever RMIDs should become available again, so it's important for resctrl to maintain an accurate task list for a container even when RMIDs are not available.Quoted 8 lines
I see a monitor group as a collection of tasks that need to be monitored together. The "task list" is the group of tasks that share a monitoring ID that is required to be a valid ID since when any of the tasks are scheduled that ID is written to the hardware. I intentionally tried to not use RMID since I believe this is required for all archs.
I thus do not understand how a task can start running when it does not have a valid monitoring ID. The idea of "deferred assignment" is not clear to me, there can never be "unmonitored tasks", no? I think I am missing something here.In the AMD/RMID implemenentation this might be achieved with something extra in the task structure to denote whether a task is in a monitored group or not. E.g. We add "
task->rmid_valid" as well as "task->rmid". Tasks in an unmonitored group retain their "task->rmid" (that's what identifies them as a member of a group) but havetask->rmid_validset to false. Context switch code would be updated to load "0" into the IA32_PQR_ASSOC.RMID field for tasks without a valid RMID. So they would still be monitored, but activity would be bundled with all tasks in the default resctrl group.
Quoted 2 lines
Presumably something analogous could be done for ARM/MPAM.
I do not interpret this as an unmonitored task but instead a task that belongs to the default resource group. Specifically, any data accumulated by such a task is attributed to the default resource group. Having tasks in a separate group but their monitoring data accumulating in/contributed to the default resource group (that has its own set of tasks) sounds wrong to me. Such an implementation makes any monitoring data of default resource group invalid, and by extension impossible to use default resource group to manage an allocation for a group of monitor groups if user space needs insight in monitoring data across all these monitor groups. User space will need to interact with resctrl differently and individually query monitor groups instead of CTRL_MON group once.
Reinette
Luck, Tony
maintainerQuoted 39 lines
Quoted 36 lines
Quoted 17 lines
Quoted 4 lines
There's also the mongroup-RMID overcommit use case I described above[1]. On Intel we can safely assume that there are counters to back all RMIDs, so
num_mbm_cntrswould be calculated directly fromnum_rmids.This is about the:
There's now more interest in Google for allowing explicit control of where RMIDs are assigned on Intel platforms. Even though the number of RMIDs implemented by hardware tends to be roughly the number of containers they want to support, they often still need to create containers when all RMIDs have already been allocated, which is not currently allowed. Once the container has been created and starts running, it's no longer possible to move its threads into a monitoring group whenever RMIDs should become available again, so it's important for resctrl to maintain an accurate task list for a container even when RMIDs are not available.Quoted 8 lines
I see a monitor group as a collection of tasks that need to be monitored together. The "task list" is the group of tasks that share a monitoring ID that is required to be a valid ID since when any of the tasks are scheduled that ID is written to the hardware. I intentionally tried to not use RMID since I believe this is required for all archs.
I thus do not understand how a task can start running when it does not have a valid monitoring ID. The idea of "deferred assignment" is not clear to me, there can never be "unmonitored tasks", no? I think I am missing something here.In the AMD/RMID implemenentation this might be achieved with something extra in the task structure to denote whether a task is in a monitored group or not. E.g. We add "
task->rmid_valid" as well as "task->rmid". Tasks in an unmonitored group retain their "task->rmid" (that's what identifies them as a member of a group) but havetask->rmid_validset to false. Context switch code would be updated to load "0" into the IA32_PQR_ASSOC.RMID field for tasks without a valid RMID. So they would still be monitored, but activity would be bundled with all tasks in the default resctrl group.Quoted 2 lines
Presumably something analogous could be done for ARM/MPAM.
Quoted 11 lines
I do not interpret this as an unmonitored task but instead a task that belongs to the default resource group. Specifically, any data accumulated by such a task is attributed to the default resource group. Having tasks in a separate group but their monitoring data accumulating in/contributed to the default resource group (that has its own set of tasks) sounds wrong to me. Such an implementation makes any monitoring data of default resource group invalid, and by extension impossible to use default resource group to manage an allocation for a group of monitor groups if user space needs insight in monitoring data across all these monitor groups. User space will need to interact with resctrl differently and individually query monitor groups instead of
CTRL_MONgroup once.
Maybe assign one of the limited supply of RMIDs for these "unmonitored" tasks. Populate a resctrl group named "unmonitored" that lists all the unmonitored tasks in a (read-only) "tasks" file. And supply all the counts for these tasks in normal looking "mon_data" directory.
-Tony
Hi Tony, Reinette,
Quoted 18 lines · Tony
Quoted 18 lines
Quoted 18 lines
Quoted 18 lines
Quoted 5 lines
There's also the mongroup-RMID overcommit use case I described above[1]. On Intel we can safely assume that there are counters to back all RMIDs, so
num_mbm_cntrswould be calculated directly fromnum_rmids.This is about the:
There's now more interest in Google for allowing explicit control of where RMIDs are assigned on Intel platforms. Even though the number of RMIDs implemented by hardware tends to be roughly the number of containers they want to support, they often still need to create containers when all RMIDs have already been allocated, which is not currently allowed. Once the container has been created and starts running, it's no longer possible to move its threads into a monitoring group whenever RMIDs should become available again, so it's important for resctrl to maintain an accurate task list for a container even when RMIDs are not available.
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
I see a monitor group as a collection of tasks that need to be monitored together. The "task list" is the group of tasks that share a monitoring ID that is required to be a valid ID since when any of the tasks are scheduled that ID is written to the hardware. I intentionally tried to not use RMID since I believe this is required for all archs.
I thus do not understand how a task can start running when it does not have a valid monitoring ID. The idea of "deferred assignment" is not clear to me, there can never be "unmonitored tasks", no? I think I am missing something here.
You are correct. I did forget to mention something...
Quoted 26 lines
Quoted 14 lines
Quoted 11 lines
In the AMD/RMID implemenentation this might be achieved with something extra in the task structure to denote whether a task is in a monitored group or not. E.g. We add "
task->rmid_valid" as well as "task->rmid". Tasks in an unmonitored group retain their "task->rmid" (that's what identifies them as a member of a group) but havetask->rmid_validset to false. Context switch code would be updated to load "0" into the IA32_PQR_ASSOC.RMID field for tasks without a valid RMID. So they would still be monitored, but activity would be bundled with all tasks in the default resctrl group.Quoted 2 lines
Presumably something analogous could be done for ARM/MPAM.
Quoted 11 lines
I do not interpret this as an unmonitored task but instead a task that belongs to the default resource group. Specifically, any data accumulated by such a task is attributed to the default resource group. Having tasks in a separate group but their monitoring data accumulating in/contributed to the default resource group (that has its own set of tasks) sounds wrong to me. Such an implementation makes any monitoring data of default resource group invalid, and by extension impossible to use default resource group to manage an allocation for a group of monitor groups if user space needs insight in monitoring data across all these monitor groups. User space will need to interact with resctrl differently and individually query monitor groups instead of
CTRL_MONgroup once.
Quoted 4 lines
Maybe assign one of the limited supply of RMIDs for these "unmonitored" tasks. Populate a resctrl group named "unmonitored" that lists all the unmonitored tasks in a (read-only) "tasks" file. And supply all the counts for these tasks in normal looking "
mon_data" directory.
I needed to switch to an rdtgroup struct pointer rather than hardware IDs in the task structure to indicate group membership[1], otherwise it's not possible to determine which tasks are in a group when it doesn't have a unique HW ID value.
Also this is required for shared assignment so that changing a group's IDs in a domain only requires updating running tasks rather than needing to search the entire task list, which would lead to the same problem we encountered in mongroup rename[2].
-Peter
[1] https://lore.kernel.org/lkml/[email protected]/
[2] https://lore.kernel.org/lkml/CALPaoCh0SbG1+VbbgcxjubE7Cc2Pb6QqhG3NH6X=WwsNfqNjtA@mail.gmail.com/
Reinette Chatre
maintainerHi Peter,
Quoted 63 lines · Peter Newman
Hi Tony, Reinette,
Quoted 18 lines · Tony
Quoted 18 lines
Quoted 18 lines
Quoted 18 lines
Quoted 5 lines
There's also the mongroup-RMID overcommit use case I described above[1]. On Intel we can safely assume that there are counters to back all RMIDs, so
num_mbm_cntrswould be calculated directly fromnum_rmids.This is about the:
There's now more interest in Google for allowing explicit control of where RMIDs are assigned on Intel platforms. Even though the number of RMIDs implemented by hardware tends to be roughly the number of containers they want to support, they often still need to create containers when all RMIDs have already been allocated, which is not currently allowed. Once the container has been created and starts running, it's no longer possible to move its threads into a monitoring group whenever RMIDs should become available again, so it's important for resctrl to maintain an accurate task list for a container even when RMIDs are not available.Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
I see a monitor group as a collection of tasks that need to be monitored together. The "task list" is the group of tasks that share a monitoring ID that is required to be a valid ID since when any of the tasks are scheduled that ID is written to the hardware. I intentionally tried to not use RMID since I believe this is required for all archs.
I thus do not understand how a task can start running when it does not have a valid monitoring ID. The idea of "deferred assignment" is not clear to me, there can never be "unmonitored tasks", no? I think I am missing something here.You are correct. I did forget to mention something...
Quoted 26 lines
Quoted 14 lines
Quoted 11 lines
In the AMD/RMID implemenentation this might be achieved with something extra in the task structure to denote whether a task is in a monitored group or not. E.g. We add "
task->rmid_valid" as well as "task->rmid". Tasks in an unmonitored group retain their "task->rmid" (that's what identifies them as a member of a group) but havetask->rmid_validset to false. Context switch code would be updated to load "0" into the IA32_PQR_ASSOC.RMID field for tasks without a valid RMID. So they would still be monitored, but activity would be bundled with all tasks in the default resctrl group.Quoted 2 lines
Presumably something analogous could be done for ARM/MPAM.
Quoted 11 lines
I do not interpret this as an unmonitored task but instead a task that belongs to the default resource group. Specifically, any data accumulated by such a task is attributed to the default resource group. Having tasks in a separate group but their monitoring data accumulating in/contributed to the default resource group (that has its own set of tasks) sounds wrong to me. Such an implementation makes any monitoring data of default resource group invalid, and by extension impossible to use default resource group to manage an allocation for a group of monitor groups if user space needs insight in monitoring data across all these monitor groups. User space will need to interact with resctrl differently and individually query monitor groups instead of
CTRL_MONgroup once.Quoted 4 lines
Maybe assign one of the limited supply of RMIDs for these "unmonitored" tasks. Populate a resctrl group named "unmonitored" that lists all the unmonitored tasks in a (read-only) "tasks" file. And supply all the counts for these tasks in normal looking "
mon_data" directory.
Quoted 4 lines
I needed to switch to an rdtgroup struct pointer rather than hardware IDs in the task structure to indicate group membership[1], otherwise it's not possible to determine which tasks are in a group when it doesn't have a unique HW ID value.
Whether the task struct contains a pointer (albeit accompanied with its own complexities) does not address the issue that I am concerned about.
Looking at [1] I expect this new feature handles "unmonitored" groups by placing them in the default monitoring group, following Tony's first [3] suggestion.
When considering [1] by itself in the context of current resctrl all tasks should be members of resource groups that have valid HW monitoring IDs allocated. Using the default resource group in this way seems like addressing edge cases where pointer is not yet valid (unclear what these scenarios may be) instead of routing many tasks to the default group. I am not sure and I'll have to study that change closer to reason accurately.
From what I understand the new proposal that builds on [1] involves creating new monitor groups that are "unmonitored" for any length of time and when backed by the implementation in [1] this would mean these groups will actually still be monitored but the data attributed to the default resource group.
As I mentioned in response [4] to Tony this fundamentally changes the behavior users can expect from the default resource group. In addition, this breaks the first of the "Resource monitoring rules" from Documentation/filesystems/resctrl.rst:
1) If a task is a member of a MON group, or non-default CTRL_MON group then RDT events for the task will be reported in that group.
How does this fit with the ABMC work? I continue to think that I am missing parts of the discussion as it seems this new feature discussion mixed in with ABMC work.
Reinette
Quoted 10 lines
Also this is required for shared assignment so that changing a group's IDs in a domain only requires updating running tasks rather than needing to search the entire task list, which would lead to the same problem we encountered in mongroup rename[2].
-Peter
[1] https://lore.kernel.org/lkml/[email protected]/
[2] https://lore.kernel.org/lkml/CALPaoCh0SbG1+VbbgcxjubE7Cc2Pb6QqhG3NH6X=WwsNfqNjtA@mail.gmail.com/
[3] https://lore.kernel.org/lkml/aC5lL_qY00vd8qp4@agluck-desk3/
[4] https://lore.kernel.org/lkml/[email protected]/
Luck, Tony
maintainerQuoted 10 lines
Quoted 4 lines
Maybe assign one of the limited supply of RMIDs for these "unmonitored" tasks. Populate a resctrl group named "unmonitored" that lists all the unmonitored tasks in a (read-only) "tasks" file. And supply all the counts for these tasks in normal looking "
mon_data" directory.I needed to switch to an rdtgroup struct pointer rather than hardware IDs in the task structure to indicate group membership[1], otherwise it's not possible to determine which tasks are in a group when it doesn't have a unique HW ID value.
Quoted 4 lines
Also this is required for shared assignment so that changing a group's IDs in a domain only requires updating running tasks rather than needing to search the entire task list, which would lead to the same problem we encountered in mongroup rename[2].
Having a pointer to the rdtgroup in the task structure does make file system operations easier. But the cost appears to be more complexity (and memory references) in the context switch code.
Your patch[1] seems to do some extra work outside of the static_branch protected sections. So has a cost to context switch even if resctrl is not in use.
Chasing pointers "closid = rgrp->mon.parent->closid;" could be expensive when those miss in the cache.
Quoted 5 lines
-Peter
[1] https://lore.kernel.org/lkml/[email protected]/
[2] https://lore.kernel.org/lkml/CALPaoCh0SbG1+VbbgcxjubE7Cc2Pb6QqhG3NH6X=WwsNfqNjtA@mail.gmail.com/
-Tony
Moger, Babu
reviewerHi Reinette,
Quoted 127 lines · Reinette Chatre
Hi Peter and Babu,
Quoted 83 lines · Peter Newman
Hi Babu/Reinette,
Quoted 79 lines · Reinette Chatre
Hi Babu,
Quoted 75 lines · Babu
Hi Reinette,
Quoted 60 lines · Reinette Chatre
Hi Babu,
Quoted 47 lines · Babu
Hi Reinette,
Quoted 29 lines · Reinette Chatre
Hi Babu,
Quoted 2 lines · Babu
Quoted 1 line · Peter Newman
On Fri, May 16, 2025 at 12:52 AM Babu Moger <[email protected]> wrote:
...
Quoted 21 lines
Quoted 11 lines
Quoted 2 lines
/sys/fs/resctrl/info/L3_MON/num_mbm_cntrs: Reports the number of monitoring counters available for assignment.Earlier I discussed with Reinette[1] what
num_mbm_cntrsshould represent in a "soft-ABMC" implementation where assignment is implemented by assigning an RMID, which would result in all events being assigned at once.My main concern is how many "counters" you can assign by assigning RMIDs. I recall Reinette proposed reporting the number of groups which can be assigned separately from counters which can be assigned.
More context may be needed here. Currently,
num_mbm_cntrsindicates the number of counters available per domain, which is 32.At the moment, we can assign 2 counters to each group, meaning each RMID can be associated with 2 hardware counters. In theory, it's possible to assign all 32 hardware counters to a group—allowing one RMID to be linked with up to 32 counters. However, we currently lack the interface to support that level of assignment.
Quoted 8 lines
Quoted 2 lines
For now, the plan is to support basic assignment and expand functionality later once we have the necessary data structure and requirements.
Looks like some requirements did not make it into this implementation. Do you recall the discussion that resulted in you writing [2]? Looks like there is a question to Peter in there on how to determine how many "counters" are available in soft-ABMC. I interpreted [3] at that time to mean that this information would be available in a future AMD publication.
We already have a method to determine the number of counters in soft-ABMC mode, which Peter has addressed [4].
Quoted 6 lines
This appears to be more of a workaround, and I doubt it will be included in any official AMD documentation. Additionally, the long-term direction is moving towards ABMC.
I don’t believe this workaround needs to be part of the current series. It can be added later when soft-ABMC is implemented.
Agreed. What about the plans described in [2]? (Thanks to Peter for catching this!).
Quoted 9 lines
It is important to keep track of requirements while working on a feature to ensure that the implementation supports the planned use cases. Re-reading that thread it is not clear to me how soft-ABMC's per-group assignment would look. Could you please share how you see it progress from this implementation? This includes the single event vs. multiple event assignment. I would like to highlight that this is not a request for this to be supported in this implementation but there needs to be a plan for how this can be supported on top of interfaces established by this work.
Here’s my current understanding of soft-ABMC. Peter may have a more in-depth perspective on this.
Quoted 8 lines
Quoted 2 lines
Quoted 2 lines
Soft-ABMC:
a.num_mbm_cntrs: This is a software-defined limit based on the number of active RMIDs that can be supported. The value can be obtained using the code referenced in [4].I would call it a hardware-defined limit that can be probed by software.
The main question is whether this file returns the exact number of RMIDs hardware can track or double that number (
mbm_total_bytes+mbm_local_bytes) so that the value is always measured in events.tl;dr: I continue [3] to find it most intuitive for
num_mbm_cntrsto be the exact number of "active" RMIDs that the system can support *and* changing the name of the modes to help user interpretnum_mbm_cntrs: "mbm_cntr_event_assign" for ABMC, "mbm_cntr_group_assign" for soft-ABMC.details
-------We are now back to the previous discussion about what user can expect from the interface. Let me try and re-cap that discussion so that we can all hopefully get back on the same page. Please add corrections/updates where needed.
soft-ABMC
---------
soft-ABMC manages "active" (term TBD) RMID assignment to monitor groups. When an "active" RMID is assigned to a monitor group then *all* MBM events (not LLC occupancy) in that monitor group are counted. "Active" RMID assignment can be done per domain.Requirement: resctrl should accurately reflect which events are counted. That is, we do not want resctrl to pretend to allow user to assign an "active" RMID to only one event in a monitor group while all events are actually counted.
Caveat: To support rapid re-assignment of RMIDs to monitor groups,
llc_occupancyevent is disabled when soft-ABMC is enabled.ABMC
----
ABMC manages (hardware) counter assignment to monitor group (RMID), event pairs. When a hardware counter is assigned to an RMID, event pair then only that RMID, event is counted. Hardware counter assignment can be done per domain.
Quoted 6 lines
shared assignment
-----------------
A shared assignment applies to both soft-ABMC and ABMC. A user can designate a "counter" (could be hardware counter or "active" RMID) as shared and that means the counter within that domain is shared between different monitor groups and actual assignment is scheduled by resctrl.
Good summary: Thanks.
Quoted 102 lines
user interface
--------------Next, consider the interface while keeping above definitions and requirements in mind.
This series introduces (using implementation, not cover-letter):
/sys/fs/resctrl/info/L3_MON/num_mbm_cntrs
"num_mbm_cntrs":
The maximum number of monitoring counters (total of available and assigned counters) in each domain when the system supportsmbm_cntr_assignmode.
/sys/fs/resctrl/mbm_L3_assignments
"mbm_L3_assignments":
This interface file is created when thembm_cntr_assignmode is supported and shows the assignment status for each group.Consider "
mbm_L3_assignments" first. The interface is documented for ABMC support where it is possible to manage individual event assignment within monitor group.For ABMC it is possible to assign just one event at a time and doing so consumes one counter in that domain:
a) Starting state on system with 32 counters per domain, two events in default resource group consumes two counters in that domain:
# cat /sys/fs/resctrl/info/L3_MON/num_mbm_cntrs 0=30;1=32 # cat /sys/fs/resctrl/mbm_L3_assignments mbm_total_bytes:0=e;1=_ mbm_local_bytes:0=e;1=_b) Assign counter to
mbm_local_bytesin domain 1:# echo "mbm_local_bytes:1=e" > /sys/fs/resctrl/mbm_L3_assignments # cat /sys/fs/resctrl/mbm_L3_assignments mbm_total_bytes:0=e;1=_ mbm_local_bytes:0=e;1=e # cat /sys/fs/resctrl/info/L3_MON/num_mbm_cntrs 0=30;1=31The question is how this should look on soft-ABMC system. Let's say hypothetically that on a soft-ABMC system it is possible to have 32 "active" RMIDs.
a) Starting state on system with 32 "active RMIDs" per domain, two events in default resource group consumes one RMID in that domain:
# cat /sys/fs/resctrl/mbm_L3_assignments mbm_total_bytes:0=e;1=_ mbm_local_bytes:0=e;1=_What should
num_mbm_cntrsdisplay?Option A (counters are RMIDs):
# cat /sys/fs/resctrl/info/L3_MON/num_mbm_cntrs 0=31;1=32Option B (pretend RMIDs are events):
# cat /sys/fs/resctrl/info/L3_MON/num_mbm_cntrs 0=62;1=64b) Assign counter to
mbm_local_bytesin domain 1:# echo "mbm_local_bytes:1=e" > /sys/fs/resctrl/mbm_L3_assignments # cat /sys/fs/resctrl/mbm_L3_assignments mbm_total_bytes:0=e;1=e mbm_local_bytes:0=e;1=eNote that even though user requested only
mbm_local_bytesto be assigned, it actually results in bothmbm_total_bytesandmbm_local_bytesto be assigned. This ensures accurate state representation to user space but this also creates an inconsistent user interface between soft-ABMC and ABMC since user space intends to use the same interface but "sometimes" assigning one event results in assign of one event while "sometimes" it results in assign of multiple events.wrt "
num_mbm_cntrs"Option A (counters are RMIDs):
# cat /sys/fs/resctrl/info/L3_MON/num_mbm_cntrs 0=31;1=31Option B (pretend RMIDs are events):
# cat /sys/fs/resctrl/info/L3_MON/num_mbm_cntrs 0=62;1=62Neither option seems ideal to me since the interface cannot be consistent between ABMC and soft-ABMC.
As I mentioned in [2] it is not possible to hide ABMC and soft-ABMC behind the same interface. When user space wants to monitor a particular monitor group then it should be clear how that can be accomplished. Not knowing if an assignment/unassignment to/from an event would impact one or all events and whether it will consume one or multiple counters does not sound like a good interface to me.As I understand current interface, user is required to know how ABMC and soft-ABMC is implemented to be able to configure the system. For example, if user has file like:
# cat /sys/fs/resctrl/mbm_L3_assignments mbm_total_bytes:0=e;1=e mbm_local_bytes:0=e;1=e user must know underlying implementation to be able to manage monitoring of events and assigning counters otherwise it will be a surprise to lose monitoring of all events when unassigning one event.
Quoted 4 lines
This is why I proposed in [3] that the name of the mode reflects how user can interact with the system. Instead of one "
mbm_cntr_assign" mode there can be "mbm_cntr_event_assign" that is used for ABMC and "mbm_cntr_group_assign" that is used for soft-ABMC. The mode should make it clear what the system is capable of wrt counter assignments.
Yes, that makes sense. Perhaps we can also simplify it further:
# cat /sys/fs/resctrl/info/L3_MON/mbm_assign_mode: [mbm_cntr_evt_assign] <- for ABMC mbm_cntr_grp_assign <- for soft-ABMC
Quoted 7 lines
Considering this the interface should be clear:
num_mbm_cntrs: reflects the number of counters in each domain that can be assigned. In "mbm_cntr_event_assign" this will be the number of counters that can be assigned to each event within a monitoring group, in "mbm_cntr_group_assign" this will be the number of counters that can be assigned to entire monitoring groups impacting all MBM events.
Quoted 8 lines
mbm_L3_assignments: manages the counter assignment in each group. When user knows the mode is "mbm_cntr_event_assign"/"mbm_cntr_group_assign" then it should be clear to user space how the interface behaves wrt assignment, no surprises of multiple events impacted when assigning/unassigning single event.For soft-ABMC I thus find it most intuitive for
num_mbm_cntrsto be the exact number of "active" RMIDs that the system can support *and* changing the name of the modes to help user interpretnum_mbm_cntrs.
Sure. The option A: fits well here then.
Option A (counters are RMIDs):
# cat /sys/fs/resctrl/info/L3_MON/num_mbm_cntrs 0=31;1=31
Quoted 54 lines
Quoted 6 lines
There's also the mongroup-RMID overcommit use case I described above[1]. On Intel we can safely assume that there are counters to back all RMIDs, so
num_mbm_cntrswould be calculated directly fromnum_rmids.This is about the:
There's now more interest in Google for allowing explicit control of where RMIDs are assigned on Intel platforms. Even though the number of RMIDs implemented by hardware tends to be roughly the number of containers they want to support, they often still need to create containers when all RMIDs have already been allocated, which is not currently allowed. Once the container has been created and starts running, it's no longer possible to move its threads into a monitoring group whenever RMIDs should become available again, so it's important for resctrl to maintain an accurate task list for a container even when RMIDs are not available.I see a monitor group as a collection of tasks that need to be monitored together. The "task list" is the group of tasks that share a monitoring ID that is required to be a valid ID since when any of the tasks are scheduled that ID is written to the hardware. I intentionally tried to not use RMID since I believe this is required for all archs.
I thus do not understand how a task can start running when it does not have a valid monitoring ID. The idea of "deferred assignment" is not clear to me, there can never be "unmonitored tasks", no? I think I am missing something here.Quoted 16 lines
I realized this use case is more difficult to implement on MPAM, because a PARTID is effectively a CLOSID+RMID, so deferring assigning a unique PARTID to a group also results in it being in a different allocation group. It will work if the unmonitored groups could find a way to share PARTIDs, but this has consequences on allocation - but hopefully no worse than sharing CLOSIDs on x86.
There's a lot of interest in monitoring ID overcommit in Google, so I think it's worth it for me to investigate the additional structural changes needed in resctrl (i.e., breaking the FS-level association between mongroups and HW monitoring IDs). Such a framework could be a better fit for soft-ABMC. For example, if overcommit is allowed, we would just report the number of simultaneous RMIDs we were able to probe as
num_rmids. I would want the same shared assignment scheduler to be able to work with RMIDs and counters, though.Quoted 4 lines
Thanks,
-Peter[1] https://lore.kernel.org/lkml/CALPaoChSzzU5mzMZsdT6CeyEn0WD1qdT9fKCoNW_ty4tojtrkw@mail.gmail.com/
Reinette
[2] https://lore.kernel.org/lkml/[email protected]/
[3] https://lore.kernel.org/lkml/[email protected]/
Signature · 2 lines
Thanks Babu Moger
Reinette Chatre
maintainerHi Babu,
Quoted 1 line · Babu
On 5/21/25 18:03, Reinette Chatre wrote:
...
Quoted 10 lines
Quoted 4 lines
This is why I proposed in [3] that the name of the mode reflects how user can interact with the system. Instead of one "
mbm_cntr_assign" mode there can be "mbm_cntr_event_assign" that is used for ABMC and "mbm_cntr_group_assign" that is used for soft-ABMC. The mode should make it clear what the system is capable of wrt counter assignments.Yes, that makes sense. Perhaps we can also simplify it further:
# cat /sys/fs/resctrl/info/L3_MON/mbm_assign_mode: [mbm_cntr_evt_assign] <- for ABMC mbm_cntr_grp_assign <- for soft-ABMC
Looks good to me. Thank you.
Quoted 15 lines
Quoted 6 lines
Considering this the interface should be clear:
num_mbm_cntrs: reflects the number of counters in each domain that can be assigned. In "mbm_cntr_event_assign" this will be the number of counters that can be assigned to each event within a monitoring group, in "mbm_cntr_group_assign" this will be the number of counters that can be assigned to entire monitoring groups impacting all MBM events.Quoted 8 lines
mbm_L3_assignments: manages the counter assignment in each group. When user knows the mode is "mbm_cntr_event_assign"/"mbm_cntr_group_assign" then it should be clear to user space how the interface behaves wrt assignment, no surprises of multiple events impacted when assigning/unassigning single event.For soft-ABMC I thus find it most intuitive for
num_mbm_cntrsto be the exact number of "active" RMIDs that the system can support *and* changing the name of the modes to help user interpretnum_mbm_cntrs.
Quoted 5 lines
Sure. The option A: fits well here then.
Option A (counters are RMIDs):
# cat /sys/fs/resctrl/info/L3_MON/num_mbm_cntrs 0=31;1=31
Thank you for considering.
Please add the requirements from this discussion to your running list. Also please keep in mind how soft-ABMC intends to use the interfaces created by this work so that the documentation that accompanies the ABMC support in this series leaves enough "wiggle room" for soft-ABMC to be built on top.
Quoted 8 lines
Reinette
Moger, Babu
reviewerHi Reinette,
Quoted 41 lines · Reinette Chatre
Hi Babu,
Quoted 1 line · Babu
On 5/21/25 18:03, Reinette Chatre wrote:
...
Quoted 10 lines
Quoted 4 lines
This is why I proposed in [3] that the name of the mode reflects how user can interact with the system. Instead of one "
mbm_cntr_assign" mode there can be "mbm_cntr_event_assign" that is used for ABMC and "mbm_cntr_group_assign" that is used for soft-ABMC. The mode should make it clear what the system is capable of wrt counter assignments.Yes, that makes sense. Perhaps we can also simplify it further:
# cat /sys/fs/resctrl/info/L3_MON/mbm_assign_mode: [mbm_cntr_evt_assign] <- for ABMC mbm_cntr_grp_assign <- for soft-ABMCLooks good to me. Thank you.
Quoted 15 lines
Quoted 6 lines
Considering this the interface should be clear:
num_mbm_cntrs: reflects the number of counters in each domain that can be assigned. In "mbm_cntr_event_assign" this will be the number of counters that can be assigned to each event within a monitoring group, in "mbm_cntr_group_assign" this will be the number of counters that can be assigned to entire monitoring groups impacting all MBM events.Quoted 8 lines
mbm_L3_assignments: manages the counter assignment in each group. When user knows the mode is "mbm_cntr_event_assign"/"mbm_cntr_group_assign" then it should be clear to user space how the interface behaves wrt assignment, no surprises of multiple events impacted when assigning/unassigning single event.For soft-ABMC I thus find it most intuitive for
num_mbm_cntrsto be the exact number of "active" RMIDs that the system can support *and* changing the name of the modes to help user interpretnum_mbm_cntrs.Quoted 5 lines
Sure. The option A: fits well here then.
Option A (counters are RMIDs):
# cat /sys/fs/resctrl/info/L3_MON/num_mbm_cntrs 0=31;1=31
Quoted 5 lines
Thank you for considering.
Please add the requirements from this discussion to your running list. Also please keep in mind how soft-ABMC intends to use the interfaces created by this work so that the documentation that accompanies the ABMC support in this series leaves enough "wiggle room" for soft-ABMC to be built on top.
Sure. Thanks
Quoted 12 lines
Quoted 9 lines
Quoted 8 lines
Quoted 2 lines
[1] https://lore.kernel.org/lkml/CALPaoChSzzU5mzMZsdT6CeyEn0WD1qdT9fKCoNW_ty4tojtrkw@mail.gmail.com/
Reinette
[2] https://lore.kernel.org/lkml/[email protected]/
[3] https://lore.kernel.org/lkml/[email protected]/Reinette
Signature · 2 lines
Thanks Babu Moger
Hi Reinette,
Quoted 12 lines · Reinette Chatre
Hi Babu,
Quoted 1 line · Babu
On 5/21/25 18:03, Reinette Chatre wrote:
...
Quoted 5 lines
Quoted 4 lines
This is why I proposed in [3] that the name of the mode reflects how user can interact with the system. Instead of one "
mbm_cntr_assign" mode there can be "mbm_cntr_event_assign" that is used for ABMC and "mbm_cntr_group_assign" that is used for soft-ABMC. The mode should make it clear what the system is capable of wrt counter assignments.
Quoted 7 lines
Quoted 5 lines
Yes, that makes sense. Perhaps we can also simplify it further:
# cat /sys/fs/resctrl/info/L3_MON/mbm_assign_mode: [mbm_cntr_evt_assign] <- for ABMC mbm_cntr_grp_assign <- for soft-ABMCLooks good to me. Thank you.
I am actually ready with v14 series. I have good feeling that we are getting closer to making these changes final.
So, Looking back again, it might make more sense to rename few user visible interfaces.
1. # cat /sys/fs/resctrl/info/L3_MON/mbm_assign_mode. [mbm_assign_event] <- for ABMC mbm_assign_group <- for soft-ABMC
This looks much more cleaner. It matches with "mbm_assign_mode"
Similarly, we can rename few functions and variable names to make little more readable.
2. mbm_cntr_assignable -> mbm_assignable
3. resctrl_arch_mbm_cntr_assign_enabled
-> >resctrl_arch_mbm_assign_enabled
4. mbm_cntr_assign_enabled -> mbm_assign_enabled
5. resctrl_arch_mbm_cntr_assign_set_one ->
resctrl_arch_mbm_assign_set_one.
6. There will few more functions. I will look into that if you agree with approach.
7. No need to change few of these below. These are related to actual counters. num_mbm_cntrs
available_mbm_cntrs
What do you think?
Thanks
Babu Moger
Reinette Chatre
maintainerHi Babu,
Quoted 25 lines · Babu
Hi Reinette,
Quoted 12 lines · Reinette Chatre
Hi Babu,
Quoted 1 line · Babu
On 5/21/25 18:03, Reinette Chatre wrote:
...
Quoted 5 lines
Quoted 4 lines
This is why I proposed in [3] that the name of the mode reflects how user can interact with the system. Instead of one "
mbm_cntr_assign" mode there can be "mbm_cntr_event_assign" that is used for ABMC and "mbm_cntr_group_assign" that is used for soft-ABMC. The mode should make it clear what the system is capable of wrt counter assignments.Quoted 7 lines
Quoted 5 lines
Yes, that makes sense. Perhaps we can also simplify it further:
# cat /sys/fs/resctrl/info/L3_MON/mbm_assign_mode: [mbm_cntr_evt_assign] <- for ABMC mbm_cntr_grp_assign <- for soft-ABMCLooks good to me. Thank you.
I am actually ready with v14 series. I have good feeling that we are getting closer to making these changes final.
Quoted 7 lines
So, Looking back again, it might make more sense to rename few user visible interfaces.
1. # cat
/sys/fs/resctrl/info/L3_MON/mbm_assign_mode. [mbm_assign_event] <- for ABMC mbm_assign_group <- for soft-ABMCThis looks much more cleaner. It matches with "
mbm_assign_mode"
ah, I see, by dropping "cntr" it reduces confusion where ABMC assigns counters and soft-ABMC assigned RMID. This looks good.
Taking this further, the "assign" term in "mbm_assign_event" and "mbm_assign_group" may also be redundant considering that the filename, "mbm_assign_mode", already has "assign" in its name.
Quoted 5 lines
Similarly, we can rename few functions and variable names to make little more readable.
2.
mbm_cntr_assignable->mbm_assignable
I have no insight into how the soft-ABMC implementation will look and thus if it will build on this property. If soft-ABMC uses the property then making it more generic may help, but if it does not then it may make the code harder to read. Since this is all internal I'd vote for keeping it mbm_cntr_assignable since the current implementation directly associates it with hardware counters. I do not know if there will be a scenario where a system may support *both* event and group assignable counters. The idea did briefly come up[1]. If that may be possible then resctrl would need to distinguish them. Also, interesting to note that the example used in (1) above notes a system that supports both event and group assignment.
Quoted 3 lines
3.
resctrl_arch_mbm_cntr_assign_enabled
-> >resctrl_arch_mbm_assign_enabled
This is directly connected to choice for (2)
Quoted 1 line
4.
mbm_cntr_assign_enabled->mbm_assign_enabled
hmmm ... here mbm_cntr_assign_enabled is even more directly associated with hardware support for counter assignment. It is not clear what the benefit is to make it generic.
Quoted 4 lines
5.
resctrl_arch_mbm_cntr_assign_set_one->
resctrl_arch_mbm_assign_set_one.
Same as (4)
Quoted 8 lines
6. There will few more functions. I will look into that if you agree with approach.
7. No need to change few of these below. These are related to actual counters.
num_mbm_cntrs
available_mbm_cntrsWhat do you think?
It sounds to me as though you are aiming to make the ABMC implementation more generic in preparation for soft-ABMC support. If you have insight into the soft-ABMC implementation then please share the details for this to be taken into account. Until then I think it will be simpler for the implementation to be specific to the feature being enabled here. When soft-ABMC enabling arrives the needed changes can be made. Since this is about internals of resctrl (not the user interface) we are not as pressured to "get it right" while not having all information required to make these choices.
Reinette
[1] https://lore.kernel.org/lkml/CALPaoCj438UfH3QA_VnGo-pj2a_48sJufUWjBKT3MQatcMJ_Uw@mail.gmail.com/
Hi Reinette,
Quoted 34 lines · Reinette Chatre
Hi Babu,
Quoted 31 lines · Babu
Hi Reinette,
Quoted 12 lines · Reinette Chatre
Hi Babu,
Quoted 1 line · Babu
On 5/21/25 18:03, Reinette Chatre wrote:
...
Quoted 5 lines
Quoted 4 lines
This is why I proposed in [3] that the name of the mode reflects how user can interact with the system. Instead of one "
mbm_cntr_assign" mode there can be "mbm_cntr_event_assign" that is used for ABMC and "mbm_cntr_group_assign" that is used for soft-ABMC. The mode should make it clear what the system is capable of wrt counter assignments.Quoted 7 lines
Quoted 5 lines
Yes, that makes sense. Perhaps we can also simplify it further:
# cat /sys/fs/resctrl/info/L3_MON/mbm_assign_mode: [mbm_cntr_evt_assign] <- for ABMC mbm_cntr_grp_assign <- for soft-ABMCLooks good to me. Thank you.
I am actually ready with v14 series. I have good feeling that we are getting closer to making these changes final.
So, Looking back again, it might make more sense to rename few user visible interfaces.
1. # cat
/sys/fs/resctrl/info/L3_MON/mbm_assign_mode. [mbm_assign_event] <- for ABMC mbm_assign_group <- for soft-ABMC
Quoted 7 lines
Quoted 1 line
This looks much more cleaner. It matches with "
mbm_assign_mode"ah, I see, by dropping "cntr" it reduces confusion where ABMC assigns counters and soft-ABMC assigned RMID. This looks good.
Taking this further, the "assign" term in "
mbm_assign_event" and "mbm_assign_group" may also be redundant considering that the filename, "mbm_assign_mode", already has "assign" in its name.
ok. Sure. It will be
# cat /sys/fs/resctrl/info/L3_MON/mbm_assign_mode. [mbm_event] <- for ABMC mbm_group <- for soft-ABMC
Quoted 7 lines
Quoted 6 lines
Similarly, we can rename few functions and variable names to make little more readable.
2.
mbm_cntr_assignable->mbm_assignable
Quoted 9 lines
I have no insight into how the soft-ABMC implementation will look and thus if it will build on this property. If soft-ABMC uses the property then making it more generic may help, but if it does not then it may make the code harder to read. Since this is all internal I'd vote for keeping it
mbm_cntr_assignablesince the current implementation directly associates it with hardware counters. I do not know if there will be a scenario where a system may support *both* event and group assignable counters. The idea did briefly come up[1]. If that may be possible then resctrl would need to distinguish them. Also, interesting to note that the example used in (1) above notes a system that supports both event and group assignment.
Ok. That is fine. Lets keep it as is then.
Quoted 6 lines
Quoted 4 lines
3.
resctrl_arch_mbm_cntr_assign_enabled
-> >resctrl_arch_mbm_assign_enabledThis is directly connected to choice for (2)
Ok.
Quoted 5 lines
Quoted 2 lines
4.
mbm_cntr_assign_enabled->mbm_assign_enabledhmmm ... here
mbm_cntr_assign_enabledis even more directly associated with hardware support for counter assignment. It is not clear what the benefit is to make it generic.
Ok.
Quoted 17 lines
Quoted 5 lines
5.
resctrl_arch_mbm_cntr_assign_set_one->
resctrl_arch_mbm_assign_set_one.Same as (4)
Quoted 8 lines
6. There will few more functions. I will look into that if you agree with approach.
7. No need to change few of these below. These are related to actual counters.
num_mbm_cntrs
available_mbm_cntrsWhat do you think?
Quoted 8 lines
It sounds to me as though you are aiming to make the ABMC implementation more generic in preparation for soft-ABMC support. If you have insight into the soft-ABMC implementation then please share the details for this to be taken into account. Until then I think it will be simpler for the implementation to be specific to the feature being enabled here. When soft-ABMC enabling arrives the needed changes can be made. Since this is about internals of resctrl (not the user interface) we are not as pressured to "get it right" while not having all information required to make these choices.
Ok. Sure. That is fine. Lets keep the internals to implementation specific for now.
Thanks
Babu
Quoted 5 lines
Reinette
[1] https://lore.kernel.org/lkml/CALPaoCj438UfH3QA_VnGo-pj2a_48sJufUWjBKT3MQatcMJ_Uw@mail.gmail.com/
Hi Reinette,
Quoted 97 lines · Reinette Chatre
Hi Babu,
Quoted 93 lines · Babu
Hi Reinette,
Quoted 60 lines · Reinette Chatre
Hi Babu,
Quoted 47 lines · Babu
Hi Reinette,
Quoted 29 lines · Reinette Chatre
Hi Babu,
Quoted 2 lines · Babu
Quoted 1 line · Peter Newman
On Fri, May 16, 2025 at 12:52 AM Babu Moger <[email protected]> wrote:
...
Quoted 21 lines
Quoted 11 lines
Quoted 2 lines
/sys/fs/resctrl/info/L3_MON/num_mbm_cntrs: Reports the number of monitoring counters available for assignment.Earlier I discussed with Reinette[1] what
num_mbm_cntrsshould represent in a "soft-ABMC" implementation where assignment is implemented by assigning an RMID, which would result in all events being assigned at once.My main concern is how many "counters" you can assign by assigning RMIDs. I recall Reinette proposed reporting the number of groups which can be assigned separately from counters which can be assigned.
More context may be needed here. Currently,
num_mbm_cntrsindicates the number of counters available per domain, which is 32.At the moment, we can assign 2 counters to each group, meaning each RMID can be associated with 2 hardware counters. In theory, it's possible to assign all 32 hardware counters to a group—allowing one RMID to be linked with up to 32 counters. However, we currently lack the interface to support that level of assignment.
Quoted 8 lines
Quoted 2 lines
For now, the plan is to support basic assignment and expand functionality later once we have the necessary data structure and requirements.
Looks like some requirements did not make it into this implementation. Do you recall the discussion that resulted in you writing [2]? Looks like there is a question to Peter in there on how to determine how many "counters" are available in soft-ABMC. I interpreted [3] at that time to mean that this information would be available in a future AMD publication.
We already have a method to determine the number of counters in soft-ABMC mode, which Peter has addressed [4].
Quoted 6 lines
This appears to be more of a workaround, and I doubt it will be included in any official AMD documentation. Additionally, the long-term direction is moving towards ABMC.
I don’t believe this workaround needs to be part of the current series. It can be added later when soft-ABMC is implemented.
Agreed. What about the plans described in [2]? (Thanks to Peter for catching this!).
Quoted 9 lines
It is important to keep track of requirements while working on a feature to ensure that the implementation supports the planned use cases. Re-reading that thread it is not clear to me how soft-ABMC's per-group assignment would look. Could you please share how you see it progress from this implementation? This includes the single event vs. multiple event assignment. I would like to highlight that this is not a request for this to be supported in this implementation but there needs to be a plan for how this can be supported on top of interfaces established by this work.
Here’s my current understanding of soft-ABMC. Peter may have a more in-depth perspective on this.
Soft-ABMC:
a.num_mbm_cntrs: This is a software-defined limit based on the number of active RMIDs that can be supported. The value can be obtained using the code referenced in [4].b. Assignments: No hardware configuration is required. We simply need to ensure that no more than
num_mbm_cntrsRMIDs are active at any given time.c. Configuration: Controlled via /info/L3_MON/mbm_total_bytes_config and
mbm_local_bytes_config.d. Events: Only two events can be assigned(local and total).
ABMC:
a.num_mbm_cntrs: This is defined by the hardware.
b. Assignments: Requires special MSR writes to assign counters.
c. Configuration: Comes from /info/L3_MON/counter_configs/.
d. Events: More than two events can be assigned to a group (currently up to 2).Commonalities:
a. Assignments can be either exclusive or shared in both these modes.
Quoted 8 lines
Quoted 1 line
Given these, I believe we can easily accommodate soft-ABMC in this interface.
This is not so obvious to me. It looks to me as though the user is forced to interpret the content of resctrl files differently based on soft-ABMC vs ABMC making the interface inconsistent and user thus needing to know details of implementations. This is what the previous discussion I linked to aimed to address. It sounds to me as though you believe that this is no longer an issue. Could you please show examples of what a user can expect from the interfaces and how a user will interact with the interfaces on both a non-ABMC and ABMC system?
At the interface level, I think mbm_L3_assignments on a non-ABMC system would only need to contain a single line:
0=s;1=s;...;31=s
But maybe for consistency we would synthesize a single, unmodifiable counter configuration to reflect that allocating an RMID in a domain results in assignment to all events and deallocating the RMID unassigns all events. We could call it "group" to say it's assigning at the group level, or perhaps just '*':
*:0=s;1=s;...;31=s
I'm not sure about allowing a '*' on ABMC hardware, because it could be interpreted as allocating a lot of counters when a large number of event configurations exist.
*:0=s;1=s;...;31=s
-Peter
Quoted 14 lines
Thank you
Reinette
Quoted 8 lines
Quoted 7 lines
Quoted 5 lines
Reinette Chatre
maintainerHi Peter,
Quoted 3 lines · Peter Newman
Quoted 1 line · Reinette Chatre
On 5/20/25 4:25 PM, Moger, Babu wrote:
...
Quoted 30 lines
Quoted 21 lines
Quoted 21 lines
Here’s my current understanding of soft-ABMC. Peter may have a more in-depth perspective on this.
Soft-ABMC:
a.num_mbm_cntrs: This is a software-defined limit based on the number of active RMIDs that can be supported. The value can be obtained using the code referenced in [4].b. Assignments: No hardware configuration is required. We simply need to ensure that no more than
num_mbm_cntrsRMIDs are active at any given time.c. Configuration: Controlled via /info/L3_MON/mbm_total_bytes_config and
mbm_local_bytes_config.d. Events: Only two events can be assigned(local and total).
ABMC:
a.num_mbm_cntrs: This is defined by the hardware.
b. Assignments: Requires special MSR writes to assign counters.
c. Configuration: Comes from /info/L3_MON/counter_configs/.
d. Events: More than two events can be assigned to a group (currently up to 2).Commonalities:
a. Assignments can be either exclusive or shared in both these modes.Quoted 8 lines
Quoted 1 line
Given these, I believe we can easily accommodate soft-ABMC in this interface.
This is not so obvious to me. It looks to me as though the user is forced to interpret the content of resctrl files differently based on soft-ABMC vs ABMC making the interface inconsistent and user thus needing to know details of implementations. This is what the previous discussion I linked to aimed to address. It sounds to me as though you believe that this is no longer an issue. Could you please show examples of what a user can expect from the interfaces and how a user will interact with the interfaces on both a non-ABMC and ABMC system?
Quoted 4 lines
At the interface level, I think
mbm_L3_assignmentson a non-ABMC system would only need to contain a single line:0=s;1=s;...;31=s
It should be obvious to user space how to interpret the fields. When there is thus a single "mbm_cntr_assign" mode used for ABMC and soft-ABMC a single line like this would be difficult to parse since that would imply/require that user space knows whether it is running on ABMC or soft-ABMC system, which we should avoid.
If there are different modes, for example "mbm_cntr_event_assign" and "mbm_cntr_group_assign" then this could be used by user space to distinguish how to interact with mbm_L3_assignments making something like this possible.
Quoted 7 lines
But maybe for consistency we would synthesize a single, unmodifiable counter configuration to reflect that allocating an RMID in a domain results in assignment to all events and deallocating the RMID unassigns all events. We could call it "group" to say it's assigning at the group level, or perhaps just '*':
Quoted 8 lines
*:0=s;1=s;...;31=s
I'm not sure about allowing a '*' on ABMC hardware, because it could be interpreted as allocating a lot of counters when a large number of event configurations exist.
*:0=s;1=s;...;31=s
Either could work also. Whether it is "group" or "*" ABMC systems could respond with "not supported". Will think about this more but would like to hear your opinion about the flexibility that distinguishing between a "mbm_cntr_event_assign" and "mbm_cntr_group_assign" mode provides.
Reinette
Quoted 17 lines
-Peter
Quoted 14 lines
Thank you
Reinette
Quoted 8 lines
Quoted 7 lines
Quoted 5 lines
Hi Reinette,
Quoted 50 lines · Reinette Chatre
Hi Peter,
Quoted 3 lines · Peter Newman
Quoted 1 line · Reinette Chatre
On 5/20/25 4:25 PM, Moger, Babu wrote:
...
Quoted 30 lines
Quoted 21 lines
Quoted 21 lines
Here’s my current understanding of soft-ABMC. Peter may have a more in-depth perspective on this.
Soft-ABMC:
a.num_mbm_cntrs: This is a software-defined limit based on the number of active RMIDs that can be supported. The value can be obtained using the code referenced in [4].b. Assignments: No hardware configuration is required. We simply need to ensure that no more than
num_mbm_cntrsRMIDs are active at any given time.c. Configuration: Controlled via /info/L3_MON/mbm_total_bytes_config and
mbm_local_bytes_config.d. Events: Only two events can be assigned(local and total).
ABMC:
a.num_mbm_cntrs: This is defined by the hardware.
b. Assignments: Requires special MSR writes to assign counters.
c. Configuration: Comes from /info/L3_MON/counter_configs/.
d. Events: More than two events can be assigned to a group (currently up to 2).Commonalities:
a. Assignments can be either exclusive or shared in both these modes.Quoted 8 lines
Quoted 1 line
Given these, I believe we can easily accommodate soft-ABMC in this interface.
This is not so obvious to me. It looks to me as though the user is forced to interpret the content of resctrl files differently based on soft-ABMC vs ABMC making the interface inconsistent and user thus needing to know details of implementations. This is what the previous discussion I linked to aimed to address. It sounds to me as though you believe that this is no longer an issue. Could you please show examples of what a user can expect from the interfaces and how a user will interact with the interfaces on both a non-ABMC and ABMC system?
Quoted 4 lines
At the interface level, I think
mbm_L3_assignmentson a non-ABMC system would only need to contain a single line:0=s;1=s;...;31=s
It should be obvious to user space how to interpret the fields. When there is thus a single "
mbm_cntr_assign" mode used for ABMC and soft-ABMC a single line like this would be difficult to parse since that would imply/require that user space knows whether it is running on ABMC or soft-ABMC system, which we should avoid.
Quoted 3 lines
If there are different modes, for example "
mbm_cntr_event_assign" and "mbm_cntr_group_assign" then this could be used by user space to distinguish how to interact withmbm_L3_assignmentsmaking something like this possible.
I meant to say I was proposing the format of this file when in the group assignment mode. I didn't mean to imply that a separate mode wasn't needed.
Quoted 14 lines
Quoted 14 lines
But maybe for consistency we would synthesize a single, unmodifiable counter configuration to reflect that allocating an RMID in a domain results in assignment to all events and deallocating the RMID unassigns all events. We could call it "group" to say it's assigning at the group level, or perhaps just '*':
*:0=s;1=s;...;31=s
I'm not sure about allowing a '*' on ABMC hardware, because it could be interpreted as allocating a lot of counters when a large number of event configurations exist.
Quoted 7 lines
Quoted 2 lines
*:0=s;1=s;...;31=s
Either could work also. Whether it is "group" or "*" ABMC systems could respond with "not supported". Will think about this more but would like to hear your opinion about the flexibility that distinguishing between a "
mbm_cntr_event_assign" and "mbm_cntr_group_assign" mode provides.
I agree it's clearer when they are separate modes. Between "*" and "group", I prefer "group" because it seems the least ambiguous.
I just want to make sure we'd never want both modes at the same time, such as an implementation with both a small number of monitoring IDs and a small number of MBM counters. I support one MPAM implementation that has a small number of PARTIDs and only one MBWU counter per domain. Fingers crossed that the number of PARTIDs it supports isn't small compared to the number of jobs we would run on it. Otherwise maybe it will work out to just pick the more limited of the two (monitor IDs or counters) and make allocation of one drive the other.
(In case you read this before my earlier reply[1], see the note about rdtgroup pointers in the task_struct, as this is a prerequisite for overcommitting HW monitor IDs.)
Thanks,
-Peter
[1] https://lore.kernel.org/lkml/CALPaoCjh_NXQLtNBqei=7a6Jsr17fEnPO+kqMaNq4xNu2UPDJA@mail.gmail.com/
Quoted 21 lines
Reinette
Quoted 17 lines
-Peter
Quoted 14 lines
Thank you
Reinette
Quoted 8 lines
Quoted 7 lines
Quoted 5 lines
Reinette Chatre
maintainerHi Peter,
Quoted 3 lines · Peter Newman
Quoted 1 line · Reinette Chatre
On 5/21/25 7:27 AM, Peter Newman wrote:
...
Quoted 11 lines
Quoted 11 lines
Quoted 4 lines
At the interface level, I think
mbm_L3_assignmentson a non-ABMC system would only need to contain a single line:0=s;1=s;...;31=s
It should be obvious to user space how to interpret the fields. When there is thus a single "
mbm_cntr_assign" mode used for ABMC and soft-ABMC a single line like this would be difficult to parse since that would imply/require that user space knows whether it is running on ABMC or soft-ABMC system, which we should avoid.
Quoted 7 lines
Quoted 3 lines
If there are different modes, for example "
mbm_cntr_event_assign" and "mbm_cntr_group_assign" then this could be used by user space to distinguish how to interact withmbm_L3_assignmentsmaking something like this possible.I meant to say I was proposing the format of this file when in the group assignment mode. I didn't mean to imply that a separate mode wasn't needed.
Thanks for confirming.
Quoted 15 lines
Quoted 15 lines
Quoted 14 lines
But maybe for consistency we would synthesize a single, unmodifiable counter configuration to reflect that allocating an RMID in a domain results in assignment to all events and deallocating the RMID unassigns all events. We could call it "group" to say it's assigning at the group level, or perhaps just '*':
*:0=s;1=s;...;31=s
I'm not sure about allowing a '*' on ABMC hardware, because it could be interpreted as allocating a lot of counters when a large number of event configurations exist.
*:0=s;1=s;...;31=s
Quoted 7 lines
Quoted 4 lines
Either could work also. Whether it is "group" or "*" ABMC systems could respond with "not supported". Will think about this more but would like to hear your opinion about the flexibility that distinguishing between a "
mbm_cntr_event_assign" and "mbm_cntr_group_assign" mode provides.I agree it's clearer when they are separate modes. Between "*" and "group", I prefer "group" because it seems the least ambiguous.
Sounds good to me. resctrl will need extra guards to prevent user from creating an event named "group" but this matches what resctrl already needs to do for other parts (eg. user cannot create a monitor group named "mon_groups").
Quoted 3 lines
I just want to make sure we'd never want both modes at the same time, such as an implementation with both a small number of monitoring IDs
hmmm ... my assumption was that a system could only support one of these modes ("mbm_cntr_event_assign" or "mbm_cntr_group_assign") but it could be possible to have both possible on a system. But beyond that to have both active at the *same* time? That will take a lot of wrangling during runtime.
Quoted 6 lines
and a small number of MBM counters. I support one MPAM implementation that has a small number of PARTIDs and only one MBWU counter per domain. Fingers crossed that the number of PARTIDs it supports isn't small compared to the number of jobs we would run on it. Otherwise maybe it will work out to just pick the more limited of the two (monitor IDs or counters) and make allocation of one drive the other.
Could a scenario like this be addressed by "mbm_cntr_event_assign" mode gaining support for "shared assignment"?
Quoted 4 lines
(In case you read this before my earlier reply[1], see the note about rdtgroup pointers in the
task_struct, as this is a prerequisite for overcommitting HW monitor IDs.)
Reinette
Quoted 1 line
[1] https://lore.kernel.org/lkml/CALPaoCjh_NXQLtNBqei=7a6Jsr17fEnPO+kqMaNq4xNu2UPDJA@mail.gmail.com/
...
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Reinette Chatre
maintainerHi Babu,
Quoted 18 lines · Babu Moger
This series adds the support for Assignable Bandwidth Monitoring Counters (ABMC). It is also called QoS RMID Pinning feature
Series is written such that it is easier to support other assignable features supported from different vendors.
The feature details are documented in the APM listed below [1].
[1] AMD64 Architecture Programmer's Manual Volume 2: System Programming Publication # 24593 Revision 3.41 section 19.3.3.3 Assignable Bandwidth Monitoring (ABMC). The documentation is available atThe patches are based on top of commit
92a09c47464d0(tag: v6.15-rc5, tip/irq/merge) Linux 6.15-rc5 plus
https://lore.kernel.org/lkml/[email protected]/
Quoted 7 lines
It is very clear these patches will go after James's resctrl FS/ARCH restructure. Hoping to avoid one review cycle due to the merge.
# Introduction
Users can create as many monitor groups as RMIDs supported by the hardware. However, bandwidth monitoring feature on AMD system only guarantees that
"bandwidth monitoring feature on AMD system" -> " the bandwidth monitoring feature on AMD systems"? or "the bandwidth monitoring feature on an AMD system". Not sure.
Quoted 7 lines
RMIDs currently assigned to a processor will be tracked by hardware. The counters of any other RMIDs which are no longer being tracked will be reset to zero. The MBM event counters return "Unavailable" for the RMIDs that are not tracked by hardware. So, there can be only limited number of groups that can give guaranteed monitoring numbers. With ever changing configurations there is no way to definitely know which of these groups are being tracked for certain point of time. Users do not have the option
"for certain point of time" -> "during a particular/certain(?) time"?
Quoted 1 line
to monitor a group or set of groups for certain period of time without
"for certain period of time" -> "for a certain period of time"?
This series contains many duplicate snippets. When you update one, please check that all the duplicates are updated also.
Quoted 8 lines
worrying about counter being reset in between.
The ABMC feature provides an option to the user to assign a hardware counter to an RMID, event pair and monitor the bandwidth as long as it is assigned. The assigned RMID will be tracked by the hardware until the user unassigns it manually. There is no need to worry about counters being reset during this period. Additionally, the user can specify a bitmask identifying the specific bandwidth types from the given source to track with the counter.
Instead of tacking it on as an "additionally" I see this capability now as essential to this new implementation. I tried to give this series a thorough review to help finalize this work but I kept being turned around by all the descriptions and finally it dawned that all the descriptions are at their code still based on the original "event ID" based implementation with either a small append or as little change as possible to adjust to the "extended event ID" based implementation.
The previous implementation still used (and copy&pasted many times) in these descriptions as "assign a hardware counter to an RMID, event pair" can only be accurate for this new implementation if an event is re-defined ... it is no longer the original constrained "event IDs" but instead an MBM event has become a generic name that identifies the configurable "bandwidth types" (but, see note about terminology later) to be monitored. This is never done.
I assume "the given source" is the assigned RMID? If so I think it will help to understand if this is specific: "bandwidth types from the assigned RMID ..."
I find this series to use several terms for the same concept,
for example, "bandwidth types", "memory transactions", "types of L3 transactions", "bandwidth sources", etc. This work will be easier to consume if it uses consistent and specific terminology.
Quoted 9 lines
Without ABMC enabled, monitoring will work in current 'default' mode without assignment option.
# History
Earlier implementation of ABMC had dependancy on BMEC (Bandwidth Monitoring Event Configuration). Peter had concerns with that implementation because it may be not be compatible with ARM's MPAM.
Quoted 7 lines
Here are the threads discussing the concerns and new interface to address the concerns.
https://lore.kernel.org/lkml/CALPaoCg97cLVVAcacnarp+880xjsedEWGJPXhYpy4P7=ky4MZw@mail.gmail.com/
https://lore.kernel.org/lkml/CALPaoCiii0vXOF06mfV=kVLBzhfNo0SFqt4kQGwGSGVUqvr2Dg@mail.gmail.com/Here are the finalized requirements based on the discussion:
* Remove BMEC dependency on the ABMC feature.
Even stronger, BMEC and ABMC are now "incompatible" in that resctrl will not let them be used at the same time.
Quoted 8 lines
* Eliminate global assignment listing. The interface
/sys/fs/resctrl/info/L3_MON/mbm_assign_controlis no longer required.* Create the configuration directories at
/sys/fs/resctrl/info/L3_MON/counter_configs/.
The configuration file names should be free-form, allowing users to create them as needed.* Perform assignment listing at the group level by introducing
mbm_L3_assignments
"the group level" -> "the monitoring group level"
Quoted 5 lines
in each monitoring group. The listing should provide the following details:
Event Configuration: Specifies the event configuration applied. This will be crucial when "mkdir" on event configuration is added in the future, leading to the creation of mon_data/mon_l3_*/<event configuration>.
hmmm ... sounds like it has become more natural to refer to it as "event configuration", which is a good match for what the purpose is. This thus sounds like good motivation to change "counter_configs" to "event_configs".
Quoted 4 lines
Domains: Identifies the domains where the configuration is applied, supporting multi-domain setups.
Assignment Type: Indicates whether the assignment is Exclusive (e or d), Shared (s), or Unassigned (_).
Could you please add definition of what "exclusive" and "shared" means?
Quoted 11 lines
* Provide option to enable or disable auto assignment when new group is created.
This series tries to address all the requirements listed above.
# Implementation details
Create a generic interface aimed to support user space assignment of scarce counters used for monitoring. First usage of interface is by ABMC with option to expand usage to "soft-ABMC" and MPAM counters in future.
Quoted 8 lines
Feature adds following interface files:
/sys/fs/resctrl/info/L3_MON/mbm_assign_mode: Reports the list of assignable monitoring features supported. The enclosed brackets indicate which feature is enabled.
/sys/fs/resctrl/info/L3_MON/num_mbm_cntrs: Reports the number of monitoring counters available for assignment.
Please aim to use consistent and clear terms to help understand this work. It is confusing that above uses "available" in description for num_mbm_cntrs and then below there is a new interface "available_mbm_cntrs" that uses the "available" term in name but not description.
Quoted 5 lines
/sys/fs/resctrl/info/L3_MON/available_mbm_cntrs: Reports the number of monitoring counters free in each domain.
/sys/fs/resctrl/info/L3_MON/counter_configs: Directory to hold the counter configuration.
Everywhere else seems to refer to this as "event configurations". Please just stick to one, "event configuration" seems most appropriate.
Quoted 3 lines
/sys/fs/resctrl/info/L3_MON/counter_configs/mbm_total_bytes/event_filter: Default configuration for MBM total events.
I think "default" should be dropped to make it clear that this is the actual configuration that is always used, not a static "default" that may be used in "some" circumstances.
Quoted 3 lines
/sys/fs/resctrl/info/L3_MON/counter_configs/mbm_local_bytes/event_filter: Default configuration for MBM local events.
Same wrt "default"
Quoted 2 lines
/sys/fs/resctrl/mbm_L3_assignments: Interface to list or modify assignment states on each group.
"Per monitor group interface to list or modify counters assigned to the group."? (Please improve.)
Quoted 4 lines
# Examples
a. Check if ABMC support is available
Please drop the "ABMC" from all the descriptions since this is intended to be a generic interface.
Quoted 6 lines
#mount -t resctrl resctrl
/sys/fs/resctrl/# cat
/sys/fs/resctrl/info/L3_MON/mbm_assign_mode[mbm_cntr_assign]
default
I believe the naming has been finalized in
https://lore.kernel.org/lkml/[email protected]/.
Quoted 3 lines
ABMC feature is detected and it is enabled.
b. Check how many ABMC counters are available.
available -> supported? This will help distinguish it from the next interface file named "available_mbm_cntrs".
Quoted 3 lines
# cat /sys/fs/resctrl/info/L3_MON/num_mbm_cntrs 32
Please update to reflect what implementation does.
Quoted 2 lines
c. Check how many ABMC counters are available in each domain.
"available" -> "available for assignment"
Quoted 4 lines
# cat /sys/fs/resctrl/info/L3_MON/available_mbm_cntrs 0=30;1=30
Quoted 8 lines
d. Check default counter configuration.
# cat /sys/fs/resctrl/info/L3_MON/counter_configs/mbm_total_bytes/event_filter local_reads, remote_reads, local_non_temporal_writes, remote_non_temporal_writes, local_reads_slow_memory, remote_reads_slow_memory, dirty_victim_writes_all# cat /sys/fs/resctrl/info/L3_MON/counter_configs/mbm_local_bytes/event_filter local_reads, local_non_temporal_writes, local_reads_slow_memory
Does not look like this matches implementation wrt spacing?
Quoted 3 lines
e. Series adds a new interface file "
mbm_L3_assignments" in each monitoring group
to list and modify any group's monitoring states.
"any group's" -> "that group's"
Quoted 9 lines
The list is displayed in the following format:
<Event configuration>:<Domain id>=<Assignment type>
Event configuration: A valid event configuration listed in the
/sys/fs/resctrl/info/L3_MON/counter_configsdirectory.Domain ID: A valid domain ID number.
"A valid domain ID number" -> "A valid domain ID"
Quoted 5 lines
Assignment types:
_ : No event configuration assigned
Quoted 8 lines
e : Event configuration assigned in exclusive mode
To list the default group states:
# cat/sys/fs/resctrl/mbm_L3_assignments
mbm_total_bytes:0=e;1=e
mbm_local_bytes:0=e;1=eTo unassign the configuration of
mbm_total_byteson domain 0:
This unassigns a counter, as opposed to a configuration, no? How about "To unassign the counter associated with the mbm_total_bytes event"?
Quoted 2 lines
#echo "mbm_total_bytes:0=_" > mbm_L3_assignments #cat mbm_L3_assignments
(May help to follow if the examples consistently uses full path.)
Quoted 4 lines
mbm_total_bytes:0=_;1=e mbm_local_bytes:0=e;1=e To unassign the mbm_total_bytes configuration on all domains:
same wrt unassigning a counter
Quoted 2 lines
$echo "mbm_total_bytes:*=_" > mbm_L3_assignments $cat mbm_L3_assignments
# prompt is usually used for administrator and $ for user without administrator privileges. Switching between # and $ in these examples is confusing.
Quoted 4 lines
mbm_total_bytes:0=_;1=_ mbm_local_bytes:0=e;1=e To assign the mbm_total_bytes configuration on all domains in exclusive mode:
same wrt unassigning a counter
Quoted 25 lines
$echo "mbm_total_bytes:*=e" > mbm_L3_assignments $cat mbm_L3_assignments mbm_total_bytes:0=e;1=e mbm_local_bytes:0=e;1=eg. Read the events
mbm_total_bytesandmbm_local_bytesof the default group.
There is no change in reading the events with ABMC. If the event is unassigned when reading, then the read will come back as "Unassigned".# cat /sys/fs/resctrl/mon_data/mon_L3_00/mbm_total_bytes 779247936 # cat /sys/fs/resctrl/mon_data/mon_L3_00/mbm_local_bytes 765207488h. Check the default event configurations.
#cat
/sys/fs/resctrl/info/L3_MON/counter_configs/mbm_total_bytes/event_filterlocal_reads,remote_reads,local_non_temporal_writes,remote_non_temporal_writes,local_reads_slow_memory,remote_reads_slow_memory,dirty_victim_writes_all#cat
/sys/fs/resctrl/info/L3_MON/counter_configs/mbm_local_bytes/event_filterlocal_reads,local_non_temporal_writes,local_reads_slow_memoryi. Change the event configuration for
mbm_local_bytes.
Quoted 7 lines
#echo "
local_reads,local_non_temporal_writes,local_reads_slow_memory,remote_reads" >/sys/fs/resctrl/info/L3_MON/counter_configs/mbm_local_bytes/event_filter#cat
/sys/fs/resctrl/info/L3_MON/counter_configs/mbm_local_bytes/event_filterlocal_reads,local_non_temporal_writes,local_reads_slow_memory,remote_readsThis will update the assignments where
mbm_local_bytesare configured.
"This will update all (across all domains of all monitor groups) counter assignments associated with the mbm_local_bytes event." (Please improve).
Quoted 3 lines
j. Now read the total event again. The first read may come back with "Unavailable"
status. The subsequent read ofmbm_total_byteswill display only the read events.
Was this intended to be example of reading *local* bytes after modification in previous step?
Quoted 7 lines
#cat /sys/fs/resctrl/mon_data/mon_L3_00/mbm_total_bytes Unavailable #cat /sys/fs/resctrl/mon_data/mon_L3_00/mbm_total_bytes 314101k. Users will have the option to go back to 'default'
mbm_assign_modeif required.
"Users will have the option" -> "Users have the option"
Quoted 2 lines
This can be done using the following command. Note that switching the
mbm_assign_modewill reset all the MBM counters of all resctrl groups.
"all the MBM counters " -> "all the MBM counters (and thus all MBM events)"?
Quoted 10 lines
# echo "default" > /sys/fs/resctrl/info/L3_MON/mbm_assign_mode # cat /sys/fs/resctrl/info/L3_MON/mbm_assign_mode mbm_cntr_assign [default]l. Unmount the resctrl
#umount
/sys/fs/resctrl/
---
Reinette