Skip to contentSkip to conversation

x86/resctrl : Support AMD Assignable Bandwidth Monitoring Counters (ABMC)

PATCH Superseded Babu Moger 5 participants 121 messages May 15, 2025 – Jun 11, 2025 Page 1 of 2

Subsystems (3)

v13 of 18 Superseded Latest is v18 →

lore
Share & tools
lore

Babu Moger

author reviewer
#0 lore raw

v13 cover x86/resctrl : Support AMD Assignable Bandwidth Monitoring Counters (ABMC)

This series adds the support for Assignable Bandwidth Monitoring Counters (ABMC). It is also called QoS RMID Pinning feature

Series is written such that it is easier to support other assignable features supported from different vendors.

The feature details are documented in the APM listed below [1].
[1] AMD64 Architecture Programmer's Manual Volume 2: System Programming Publication # 24593 Revision 3.41 section 19.3.3.3 Assignable Bandwidth Monitoring (ABMC). The documentation is available at

The patches are based on top of commit 92a09c47464d0 (tag: v6.15-rc5, tip/irq/merge) Linux 6.15-rc5 plus
https://lore.kernel.org/lkml/[email protected]/

It is very clear these patches will go after James's resctrl FS/ARCH restructure. Hoping to avoid one review cycle due to the merge.

# Introduction

Users can create as many monitor groups as RMIDs supported by the hardware. However, bandwidth monitoring feature on AMD system only guarantees that RMIDs currently assigned to a processor will be tracked by hardware. The counters of any other RMIDs which are no longer being tracked will be reset to zero. The MBM event counters return "Unavailable" for the RMIDs that are not tracked by hardware. So, there can be only limited number of groups that can give guaranteed monitoring numbers. With ever changing configurations there is no way to definitely know which of these groups are being tracked for certain point of time. Users do not have the option to monitor a group or set of groups for certain period of time without worrying about counter being reset in between.

The ABMC feature provides an option to the user to assign a hardware counter to an RMID, event pair and monitor the bandwidth as long as it is assigned. The assigned RMID will be tracked by the hardware until the user unassigns it manually. There is no need to worry about counters being reset during this period. Additionally, the user can specify a bitmask identifying the specific bandwidth types from the given source to track with the counter.

Without ABMC enabled, monitoring will work in current 'default' mode without assignment option.

# History

Earlier implementation of ABMC had dependancy on BMEC (Bandwidth Monitoring Event Configuration). Peter had concerns with that implementation because it may be not be compatible with ARM's MPAM.

Here are the threads discussing the concerns and new interface to address the concerns.
https://lore.kernel.org/lkml/CALPaoCg97cLVVAcacnarp+880xjsedEWGJPXhYpy4P7=ky4MZw@mail.gmail.com/
https://lore.kernel.org/lkml/CALPaoCiii0vXOF06mfV=kVLBzhfNo0SFqt4kQGwGSGVUqvr2Dg@mail.gmail.com/

Here are the finalized requirements based on the discussion:

* Remove BMEC dependency on the ABMC feature.

* Eliminate global assignment listing. The interface
/sys/fs/resctrl/info/L3_MON/mbm_assign_control is no longer required.

* Create the configuration directories at /sys/fs/resctrl/info/L3_MON/counter_configs/.
The configuration file names should be free-form, allowing users to create them as needed.

* Perform assignment listing at the group level by introducing mbm_L3_assignments
in each monitoring group. The listing should provide the following details:

Event Configuration: Specifies the event configuration applied. This will be crucial when "mkdir" on event configuration is added in the future, leading to the creation of mon_data/mon_l3_*/<event configuration>.

Domains: Identifies the domains where the configuration is applied, supporting multi-domain setups.

Assignment Type: Indicates whether the assignment is Exclusive (e or d), Shared (s), or Unassigned (_).

* Provide option to enable or disable auto assignment when new group is created.

This series tries to address all the requirements listed above.

# Implementation details

Create a generic interface aimed to support user space assignment of scarce counters used for monitoring. First usage of interface is by ABMC with option to expand usage to "soft-ABMC" and MPAM counters in future.

Feature adds following interface files:

/sys/fs/resctrl/info/L3_MON/mbm_assign_mode: Reports the list of assignable monitoring features supported. The enclosed brackets indicate which feature is enabled.

/sys/fs/resctrl/info/L3_MON/num_mbm_cntrs: Reports the number of monitoring counters available for assignment.

/sys/fs/resctrl/info/L3_MON/available_mbm_cntrs: Reports the number of monitoring counters free in each domain.

/sys/fs/resctrl/info/L3_MON/counter_configs : Directory to hold the counter configuration.

/sys/fs/resctrl/info/L3_MON/counter_configs/mbm_total_bytes/event_filter : Default configuration for MBM total events.

/sys/fs/resctrl/info/L3_MON/counter_configs/mbm_local_bytes/event_filter : Default configuration for MBM local events.

/sys/fs/resctrl/mbm_L3_assignments: Interface to list or modify assignment states on each group.

# Examples

a. Check if ABMC support is available
#mount -t resctrl resctrl /sys/fs/resctrl/

# cat /sys/fs/resctrl/info/L3_MON/mbm_assign_mode [mbm_cntr_assign]
default

ABMC feature is detected and it is enabled.

b. Check how many ABMC counters are available.

	# cat /sys/fs/resctrl/info/L3_MON/num_mbm_cntrs 
	32

c. Check how many ABMC counters are available in each domain.

	# cat /sys/fs/resctrl/info/L3_MON/available_mbm_cntrs 
	0=30;1=30

d. Check default counter configuration.

	# cat /sys/fs/resctrl/info/L3_MON/counter_configs/mbm_total_bytes/event_filter 
	local_reads, remote_reads, local_non_temporal_writes, remote_non_temporal_writes,
        local_reads_slow_memory, remote_reads_slow_memory, dirty_victim_writes_all
	# cat /sys/fs/resctrl/info/L3_MON/counter_configs/mbm_local_bytes/event_filter 
	local_reads, local_non_temporal_writes, local_reads_slow_memory

e. Series adds a new interface file "mbm_L3_assignments" in each monitoring group
to list and modify any group's monitoring states.

The list is displayed in the following format:

<Event configuration>:<Domain id>=<Assignment type>

Event configuration: A valid event configuration listed in the /sys/fs/resctrl/info/L3_MON/counter_configs directory.

Domain ID: A valid domain ID number.

Assignment types:

_ : No event configuration assigned

e : Event configuration assigned in exclusive mode

To list the default group states:
# cat /sys/fs/resctrl/mbm_L3_assignments
mbm_total_bytes:0=e;1=e
mbm_local_bytes:0=e;1=e

To unassign the configuration of mbm_total_bytes on domain 0:
#echo "mbm_total_bytes:0=_" > mbm_L3_assignments #cat mbm_L3_assignments
mbm_total_bytes:0=_;1=e
mbm_local_bytes:0=e;1=e

To unassign the mbm_total_bytes configuration on all domains:
$echo "mbm_total_bytes:*=_" > mbm_L3_assignments $cat mbm_L3_assignments
mbm_total_bytes:0=_;1=_
mbm_local_bytes:0=e;1=e

To assign the mbm_total_bytes configuration on all domains in exclusive mode:
$echo "mbm_total_bytes:*=e" > mbm_L3_assignments
$cat mbm_L3_assignments
mbm_total_bytes:0=e;1=e
mbm_local_bytes:0=e;1=e

g. Read the events mbm_total_bytes and mbm_local_bytes of the default group.
There is no change in reading the events with ABMC. If the event is unassigned when reading, then the read will come back as "Unassigned".

	# cat /sys/fs/resctrl/mon_data/mon_L3_00/mbm_total_bytes
	779247936
	# cat /sys/fs/resctrl/mon_data/mon_L3_00/mbm_local_bytes 
	765207488

h. Check the default event configurations.

#cat /sys/fs/resctrl/info/L3_MON/counter_configs/mbm_total_bytes/event_filter local_reads, remote_reads, local_non_temporal_writes, remote_non_temporal_writes, local_reads_slow_memory, remote_reads_slow_memory, dirty_victim_writes_all

#cat /sys/fs/resctrl/info/L3_MON/counter_configs/mbm_local_bytes/event_filter local_reads, local_non_temporal_writes, local_reads_slow_memory

i. Change the event configuration for mbm_local_bytes.

#echo "local_reads, local_non_temporal_writes, local_reads_slow_memory, remote_reads" > /sys/fs/resctrl/info/L3_MON/counter_configs/mbm_local_bytes/event_filter

#cat /sys/fs/resctrl/info/L3_MON/counter_configs/mbm_local_bytes/event_filter local_reads, local_non_temporal_writes, local_reads_slow_memory, remote_reads

This will update the assignments where mbm_local_bytes are configured.

j. Now read the total event again. The first read may come back with "Unavailable"
status. The subsequent read of mbm_total_bytes will display only the read events.

	#cat /sys/fs/resctrl/mon_data/mon_L3_00/mbm_total_bytes
	Unavailable
	#cat /sys/fs/resctrl/mon_data/mon_L3_00/mbm_total_bytes
	314101

k. Users will have the option to go back to 'default' mbm_assign_mode if required.
This can be done using the following command. Note that switching the mbm_assign_mode will reset all the MBM counters of all resctrl groups.

	# echo "default" > /sys/fs/resctrl/info/L3_MON/mbm_assign_mode
	# cat /sys/fs/resctrl/info/L3_MON/mbm_assign_mode
	mbm_cntr_assign
	[default]

l. Unmount the resctrl

#umount /sys/fs/resctrl/
---
v13:
Removed BMEC related 2 patches which were in the previous series. It was related to optimization which can be doen later.

Patches are created on top of FS/ARCH restructure. So, major changes are due to FS/ARCH restructure. The files are split between arch/x86/kernel/cpu/resctrl/ and fs/resctrl/. So, functions are moved between these files accordingly.

Added fflag RFTYPE_RES_CACHE for mbm_assign_mode, num_mbm_cntrs, available_mbm_cntrs.

Removed the references to "mbm_assign_control".

Moved resctrl_arch_config_cntr() prototype to include/linux/resctrl.h. Changed resctrl_arch_config_cntr() to retun void from int to simplify few call sequences.

Added the event configuration details inside the evt_list in monitor domains. The avoids the need for new structure mbm_assign_config.

Passed evtid to functions resctrl_alloc_config_cntr() and resctrl_assign_cntr_event(). Event configuration value can be easily obtained from mon_evt list.

Added new patch to pass the entire struct rdtgroup to __mon_event_count(), mbm_update(), and related functions. We can easily get RMID,CLOSID etc from rdtgroup.

Added new function __cntr_id_read_phys() to handle ABMC event reading.

Added a new patch to hide BMEC related files when mbm_cntr_assign mode is enabled..

Added the call resctrl_init_evt_configuration() to setup the event configuration during init.

And few other commit message updates and user doc updates.

Removed Reviewed-by from few patches as patches have changed due to FS/ARCH restructure.

Let me know if I missed something.

v12:
This version is kind of RFC series with a new interface.

Removed Reviewed-by tag on few patches when the patch has changed.

Moved BMEC related patches (1 and 2) to beginning of the series. Removed the dependancy on BMEC to ABMC feature.

Removed the un-necessary initialization of mon_config_info structure. Changed wrmsrl instead of wrmsr to address the below comment.
https://lore.kernel.org/lkml/[email protected]/

Fixed the conflicts due to recent changes in rdt_resource data structure. Added new mbm_cfg_mask field to resctrl_mon.

Added the code to reset arch state inside _resctrl_abmc_enable().

Added the check CONFIG_RESCTRL_ASSIGN_FIXED to take care of arm platforms. This will be defined only in arm and not in x86.

Changed the code to display the max supported monitoring counters in each domain.

Fixed the struct mbm_cntr_cfg code documentation.
Moved the struct mbm_cntr_cfg definition to resctrl/internal.h as suggested by James.

Replaced seq_puts(s, ";") with seq_putc(s, ';');
Added missing rdt_last_cmd_clear() in resctrl_available_mbm_cntrs_show().

Added the check to reset the architecture-specific state only when assign is requested.

Added evt_cfg as the parameter to resctrl_arch_config_cntr() as the user will be passing the event configuration from /info/L3_MON/event_configs/.

Changed the check in resctrl_alloc_config_cntr() to reduce the indentation.
Fixed the handling error on first failure while assigning.
Added new parameter event configuration (evt_cfg) to get the event configuration from user space.

Added tte support for reading ABMC counters. This is bit involved change and affects lots of code.

New patch to support event configurations via new counter_configs method.

Removed mbm_cntr_reset() as it is not required while removing the group.

Added new patch to handle auto assign on group creation ("mbm_assign_on_mkdir")

Added couple of patches add interface for "mbm_L3_assignments" on each mon group.

Introduced mbm_cntr_free_all() and resctrl_reset_rmid_all() to clear counters and non-architectural states when monitor mode is changed.
https://lore.kernel.org/lkml/[email protected]/

Moved the resctrl_arch_mbm_cntr_assign_set_one to domain_add_cpu_mon().

Patches 17, 18, 19, 20, 21, 23, 24 are completely new to address the new interface requirement.

v11:
The commit 2937f9c361f7a ("x86/resctrl: Introduce resctrl_file_fflags_init() to initialize fflags") is already merged. Removed from the series.

Resolved minor conflicts due to code displacement in latest code.

Moved the monitoring related calls to monitor.c file when possible.
Moved some of the changes from include/linux/resctrl.h to arch/x86/kernel/cpu/resctrl/internal.h as requested by Reinette. This changes will be moved back when arch and non code is separated.

Renamed rdtgroup_mbm_assign_mode_show() to resctrl_mbm_assign_mode_show(). Renamed rdtgroup_num_mbm_cntrs_show() to resctrl_num_mbm_cntrs_show().

Moved the mon_config_info structure definition to internal.h.
Moved resctrl_arch_mon_event_config_get() and resctrl_arch_mon_event_config_set() to monitor.c file.

Moved resctrl_arch_assign_cntr() and resctrl_abmc_config_one_amd() to monitor.c. Added the code to reset the arch state in resctrl_arch_assign_cntr().
Also removed resctrl_arch_reset_rmid() inside IPI as the counters are reset from the callers.

Renamed rdtgroup_assign_cntr_event() to resctrl_assign_cntr_event(). Refactored the resctrl_assign_cntr_event().
Added functionality to exit on the first error during assignment. Simplified mbm_cntr_free().
Removed the function mbm_cntr_assigned(). Will be using mbm_cntr_get() to figure out if the counter is assigned or not.

Renamed rdtgroup_unassign_cntr_event() to resctrl_unassign_cntr_event(). Refactored the resctrl_unassign_cntr_event().

Moved mbm_cntr_reset() to monitor.c.
Added code reset non-architectural state in mbm_cntr_reset(). Added missing rdtgroup_unassign_cntrs() calls on failure path.

Domain can be NULL with SNC support so moved the unassign check in rdtgroup_mondata_show().

Renamed rdtgroup_mbm_assign_mode_write() to resctrl_mbm_assign_mode_write(). Added more details in resctrl.rst about mbm_cntr_assign mode. Re-arranged the text in resctrl.rst file in section mbm_cntr_assign.

Moved resctrl_arch_mbm_cntr_assign_set_one() to monitor.c

Added non-arch RMID reset in mbm_config_write_domain().
Removed resctrl_arch_reset_rmid() call in resctrl_abmc_config_one_amd(). Not required as reset of arch and non-arch rmid counters done from the callers. It simplies the IPI code.

Fixed printing the separator after each domain while listing the group assignments. Renamed rdtgroup_mbm_assign_control_show to resctrl_mbm_assign_control_show().

Fixed the static check warning with initializing dom_id in resctrl_process_flags()

Added change log in each patch for specific changes.

v10:
Major change is related to domain specific assignment.
Added struct mbm_cntr_cfg inside mon domains. This will handle
the domain specific assignments as discussed in below.
https://lore.kernel.org/lkml/CALPaoCj+zWq1vkHVbXYP0znJbe6Ke3PXPWjtri5AFgD9cQDCUg@mail.gmail.com/
I did not see the need to add cntr_id in mbm_state structure. Not used in the code. Following patches take care of these changes.
Patch 12, 13, 15, 16, 17, 18.

Added __init attribute to cache_alloc_hsw_probe(). Followed function prototype rules (preferred order is storage class before return type).

Moved the mon_config_info structure definition to resctrl.h

Added call resctrl_arch_reset_rmid() to reset the RMID in the domain inside IPI call resctrl_abmc_config_one_amd.

SMP and non-SMP call support is not required in resctrl_arch_config_cntr with new domain specific assign approach/data structure.

Assigned the counter before exposing the event files.
Moved the call rdtgroup_assign_cntrs() inside mkdir_rdt_prepare_rmid_alloc(). This is called both CNTR_MON and MON group creation.

Call mbm_cntr_reset() when unmounted to clear all the assignments.

Fixed the issue with finding the domain in multiple iterations in rdtgroup_process_flags().

Printed full error message with domain information when assign fails.

Taken care of other text comments in all the patches. Patch specific changes are in each patch.

If I missed something please point me and it is not intentional.

v9:
Patch 14 is a new addition.
Major change in patch 24.
Moved the fix patch to address __init attribute to begining of the series. Fixed all the call sequences. Added additional Fixed tags.

Added Reviewed-by where applicable.

Took care of couple of minor merge conflicts with latest code. Re-ordered the MSR in couple of instances.
Added available_mbm_cntrs (patch 14) to print the number of counter in a domain.

Used MBM_EVENT_ARRAY_INDEX macro to get the event index. Introduced rdtgroup_cntr_id_init() to initialize the cntr_id

Introduced new function resctrl_config_cntr to assign the counter, update the bitmap and reset the architectural state.
Taken care of error handling(freeing the counter) when assignment fails.

Changed rdtgroup_assign_cntrs() and rdtgroup_unassign_cntrs() to return void. Updated couple of rdtgroup_unassign_cntrs() calls properly.

Fixed problem changing the mode to mbm_cntr_assign mode when it is not supported. Added extra checks to detect if systems supports it.

https://lore.kernel.org/lkml/[email protected]/
As discussed in the above comment, introduced resctrl_mon_event_config_set to handle IPI. But sending another IPI inside IPI causes problem. Kernel reports SMP warning. So, introduced resctrl_arch_update_cntr() to send the command directly.

Fixed handling special case '//0=' and '//".
Removed extra strstr() call in rdtgroup_mbm_assign_control_write(). Added generic failure text when assignment operation fails. Corrected user documentation format texts.

v8:
Patches are getting into final stages.
Couple of changes Patch 8, Patch 19 and Patch 23.
Most of the other changes are related to rename and text message updates.

Details are in each patch. Here is the summary.

Added __init attribute to dom_data_init() in patch 8/25. Moved the mbm_cntrs_init() and mbm_cntrs_exit() functionality inside dom_data_init() and dom_data_exit() respectively.

Renamed resctrl_mbm_evt_config_init() to arch_mbm_evt_config_init() Renamed resctrl_arch_event_config_get() to resctrl_arch_mon_event_config_get().
resctrl_arch_event_config_set() to resctrl_arch_mon_event_config_set().

Rename resctrl_arch_assign_cntr to resctrl_arch_config_cntr. Renamed rdtgroup_assign_cntr() to rdtgroup_assign_cntr_event(). Added the code to return the error if rdtgroup_assign_cntr_event fails. Moved definition of MBM_EVENT_ARRAY_INDEX to resctrl/internal.h. Renamed rdtgroup_mbm_cntr_is_assigned to mbm_cntr_assigned_to_domain Added return error handling in resctrl_arch_config_cntr(). Renamed rdtgroup_assign_grp to rdtgroup_assign_cntrs.
Renamed rdtgroup_unassign_grp to rdtgroup_unassign_cntrs. Fixed the problem with unassigning the child MON groups of CTRL_MON group. Reset the internal counters after mbm_cntr_assign mode is changed. Renamed rdtgroup_mbm_cntr_reset() to mbm_cntr_reset()
Renamed resctrl_arch_mbm_cntr_assign_configure to
resctrl_arch_mbm_cntr_assign_set_one.

Used the same IPI as event update to modify the assignment. Could not do the way we discussed in the thread.
https://lore.kernel.org/lkml/[email protected]/
Needed to figure out event type to update the configuration.

Moved unassign first and assign during the assign modification. Assign none "_" takes priority. Cannot be mixed with other flags. Updated the documentation and .rst file format. htmldoc looks ok.

v7:
Major changes are related to FS and arch codes separation. Changed few interface names based on feedback.
Here are the summary and each patch contains changes specific the patch.

Removed WARN_ON for num_mbm_cntrs. Decided to dynamically allocate the bitmap. WARN_ON is not required anymore.

Renamed the function resctrl_arch_get_abmc_enabled() to resctrl_arch_mbm_cntr_assign_enabled().

Merged resctrl_arch_mbm_cntr_assign_disable, resctrl_arch_mbm_cntr_assign_disable and renamed to resctrl_arch_mbm_cntr_assign_set(). Passed the struct rdt_resource to these functions.

Removed resctrl_arch_reset_rmid_all() from arch code. This will be done from FS the caller.

Updated the descriptions/commit log in resctrl.rst to generic text. Removed ABMC references. Renamed mbm_mode to mbm_assign_mode.
Renamed mbm_control to mbm_assign_control.
Introduced mutex lock in rdtgroup_mbm_mode_show().

The 'legacy' mode is called 'default' mode.

Removed the static allocation and now allocating bitmap mbm_cntr_free_map dynamically.

Merged rdtgroup_assign_cntr(), rdtgroup_alloc_cntr() into one. Merged rdtgroup_unassign_cntr(), rdtgroup_free_cntr() into one.

Added struct rdt_resource to the interface functions resctrl_arch_assign_cntr () and resctrl_arch_unassign_cntr().
Rename rdtgroup_abmc_cfg() to resctrl_abmc_config_one_amd().

Added a new patch to fix counter assignment on event config changes.

Removed the references of ABMC from user interfaces.

Simplified the parsing (strsep(&token, "//") in rdtgroup_mbm_assign_control_write(). Added mutex lock in rdtgroup_mbm_assign_control_write() while processing.

Thomas Gleixner asked us to update https://gitlab.com/x86-cpuid.org/x86-cpuid-db.
It needs internal approval. We are working on it.

v6:
We still need to finalize few interface details on mbm_assign_mode and mbm_assign_control in case of ABMC and Soft-ABMC. We can continue the discussion with this series.

Added support for domain-id '*' to update all the domains at once. Fixed assign interface to allocate the counter if counter is not assigned.
Fixed unassign interface to free the counter if the counter is not assigned in any of the domains.

Renamed abmc_capable to mbm_cntr_assignable.

Renamed abmc_enabled to mbm_cntr_assign_enabled.
Used msr_set_bit and msr_clear_bit for msr updates.
Renamed resctrl_arch_abmc_enable() to resctrl_arch_mbm_cntr_assign_enable(). Renamed resctrl_arch_abmc_disable() to resctrl_arch_mbm_cntr_assign_disable().

Changed the display name from num_cntrs to num_mbm_cntrs.

Removed the variable mbm_cntrs_free_map_len. This is not required. Removed the call mbm_cntrs_init() in arch code. This needs to be done at higher level. Used DECLARE_BITMAP to initialize mbm_cntrs_free_map.
Removed unused config value definitions.

Introduced mbm_cntr_map to track counters at domain level. With this we dont need to send MSR read to read the counter configuration.

Separated all the counter id management to upper level in FS code.

Added checks to detect "Unassigned" before reading the RMID.

More details in each patch.

v5:
Rebase changes (because of SNC support)

Interface changes.
/sys/fs/resctrl/mbm_assign to /sys/fs/resctrl/mbm_assign_mode. /sys/fs/resctrl/mbm_assign_control to /sys/fs/resctrl/mbm_assign_control.

Added few arch specific routines.
resctrl_arch_get_abmc_enabled.
resctrl_arch_abmc_enable.
resctrl_arch_abmc_disable.

Few renames
num_cntrs_free_map -> mbm_cntrs_free_map num_cntrs_init -> mbm_cntrs_init arch_domain_mbm_evt_config -> resctrl_arch_mbm_evt_config

Introduced resctrl_arch_event_config_get and
resctrl_arch_event_config_set() to update event configuration.

Removed mon_state field mongroup. Added MON_CNTR_UNSET to initialize counters.

Renamed ctr_id to cntr_id for the hardware counter.

Report "Unassigned" in case the user attempts to read the events without assigning the counter.

ABMC is enabled during the boot up. Can be enabled or disabled later.

Fixed opcode and flags combination.

    '=_" is valid.
    "-_" amd "+_" is not valid.

Added all the comments as far as I know. If I missed something, it is not intentional.

v4:
Main change is domain specific event assignment.
Kept the ABMC feature as a default.
Dynamcic switching between ABMC and mbm_legacy is still allowed. We are still not clear about mount option.
Moved the monitoring related data in resctrl_mon structure from rdt_resource. Fixed the display of legacy and ABMC mode.
Used bimap APIs when possible.
Removed event configuration read from MSRs. We can use the internal saved data.(patch 12)
Added more comments about L3_QOS_ABMC_CFG MSR.
Added IPIs to read the assignment status for each domain (patch 18 and 19) More details in each patch.

v3:
This series adds the support for global assignment mode discussed in the thread. https://lore.kernel.org/lkml/[email protected]/
Removed the individual assignment mode and included the global assignment interface. Added following interface files.
a. /sys/fs/resctrl/info/L3_MON/mbm_assign
Used for displaying the current assignment mode and switch between ABMC and legacy mode.
b. /sys/fs/resctrl/info/L3_MON/mbm_assign_control
Used for lising the groups assignment mode and modify the assignment states.
c. Most of the changes are related to the new interface.
d. Addressed the comments from Reinette, James and Peter.
e. Hope I have addressed most of the major feedbacks discussed. If I missed
something then it is not intentional. Please feel free to comment.
f. Sending this as an RFC as per Reinette's comment. So, this is still open
for discussion.

v2:
a. Major change is the way ABMC is enabled. Earlier, user needed to remount
with -o abmc to enable ABMC feature. Removed that option now.
Now users can enable ABMC by "$echo 1 to /sys/fs/resctrl/info/L3_MON/mbm_assign_enable".

b. Added new word 21 to x86/cpufeatures.h.

c. Display unsupported if user attempts to read the events when ABMC is enabled
and event is not assigned.

d. Display monitor_state as "Unsupported" when ABMC is disabled.

e. Text updates and rebase to latest tip tree (as of Jan 18).

f. This series is still work in progress. I am yet to hear from ARM developers.

--------------------------------------------------------------------------------------

Previous revisions:
v12: https://lore.kernel.org/lkml/[email protected]/
v11: https://lore.kernel.org/lkml/[email protected]/
v10: https://lore.kernel.org/lkml/[email protected]/
v9: https://lore.kernel.org/lkml/[email protected]/
v8: https://lore.kernel.org/lkml/[email protected]/
v7: https://lore.kernel.org/lkml/[email protected]/
v6: https://lore.kernel.org/lkml/[email protected]/
v5: https://lore.kernel.org/lkml/[email protected]/
v4: https://lore.kernel.org/lkml/[email protected]/
v3: https://lore.kernel.org/lkml/[email protected]/
v2: https://lore.kernel.org/lkml/[email protected]/
v1: https://lore.kernel.org/lkml/[email protected]/

Babu Moger (27):
x86/cpufeatures: Add support for Assignable Bandwidth Monitoring
Counters (ABMC)
x86/resctrl: Add ABMC feature in the command line options x86/resctrl: Consolidate monitoring related data from rdt_resource x86/resctrl: Detect Assignable Bandwidth Monitoring feature details x86/resctrl: Add support to enable/disable AMD ABMC feature x86/resctrl: Introduce the interface to display monitor mode x86/resctrl: Introduce interface to display number of monitoring
counters
x86/resctrl: Introduce mbm_cntr_cfg to track assignable counters at
domain
x86/resctrl: Introduce interface to display number of free MBM
counters
x86/resctrl: Add data structures and definitions for ABMC assignment x86/resctrl: Implement resctrl_arch_config_cntr() to assign a counter
with ABMC
x86/resctrl: Introduce event configuration modes x86/resctrl: Add the functionality to assign MBM events x86/resctrl: Add the functionality to unassign MBM events x86/resctrl: Report 'Unassigned' for MBM events in mbm_cntr_assign
mode
x86/resctrl: Pass entire struct rdtgroup rather than passing
individual members
x86/resctrl: Add the support for reading ABMC counters x86/resctrl: Add definitions for MBM event configuration x86/resctrl: Add event configuration directory under info/L3_MON/ x86/resctrl: Provide interface to update the event configurations x86/resctrl: Introduce mbm_assign_on_mkdir to configure assignments x86/resctrl: Auto assign/unassign counters when mbm_cntr_assign is
enabled
x86/resctrl: Introduce mbm_L3_assignments to list assignments in a
group
x86/resctrl: Introduce the interface to modify assignments in a group x86/resctrl: Hide the BMEC related files when mbm_cnt_assign is
enabled
x86/resctrl: Introduce the interface to switch between monitor modes x86/resctrl: Configure mbm_cntr_assign mode if supported

 .../admin-guide/kernel-parameters.txt         |   2 +-
 Documentation/filesystems/resctrl.rst         | 188 +++++
 arch/x86/include/asm/cpufeatures.h            |   1 +
 arch/x86/include/asm/msr-index.h              |   2 +
 arch/x86/kernel/cpu/cpuid-deps.c              |   2 +
 arch/x86/kernel/cpu/resctrl/core.c            |  13 +-
 arch/x86/kernel/cpu/resctrl/internal.h        |  47 ++
 arch/x86/kernel/cpu/resctrl/monitor.c         | 176 +++-
 arch/x86/kernel/cpu/scattered.c               |   1 +
 fs/resctrl/ctrlmondata.c                      |  14 +
 fs/resctrl/internal.h                         |  37 +-
 fs/resctrl/monitor.c                          | 309 ++++++-
 fs/resctrl/rdtgroup.c                         | 768 +++++++++++++++++-
 include/linux/resctrl.h                       |  74 +-
 include/linux/resctrl_types.h                 |  11 +
 15 files changed, 1577 insertions(+), 68 deletions(-)
Signature · 1 line
2.34.1
author reviewer
#1 lore raw

PATCH v13 1/27 x86/cpufeatures: Add support for Assignable Bandwidth Monitoring Counters (ABMC)

Users can create as many monitor groups as RMIDs supported by the hardware. However, bandwidth monitoring feature on AMD system only guarantees that RMIDs currently assigned to a processor will be tracked by hardware. The counters of any other RMIDs which are no longer being tracked will be reset to zero. The MBM event counters return "Unavailable" for the RMIDs that are not tracked by hardware. So, there can be only limited number of groups that can give guaranteed monitoring numbers. With ever changing configurations there is no way to definitely know which of these groups are being tracked for certain point of time. Users do not have the option to monitor a group or set of groups for certain period of time without worrying about RMID being reset in between.

The ABMC feature provides an option to the user to assign a hardware counter to an RMID, event pair and monitor the bandwidth as long as it is assigned. The assigned RMID will be tracked by the hardware until the user unassigns it manually. There is no need to worry about counters being reset during this period. Additionally, the user can specify a bitmask identifying the specific bandwidth types from the given source to track with the counter.

Without ABMC enabled, monitoring will work in current mode without assignment option.

The Linux resctrl subsystem provides an interface that allows monitoring of up to two memory bandwidth events per group, selected from a combination of available total and local events. When ABMC is enabled, two events will be assigned to each group by default, in line with the current interface design. Users will also have the option to configure which types of memory transactions are counted by these events.

Due to the limited number of available counters (32), users may quickly exhaust the available counters. If the system runs out of assignable ABMC counters, the kernel will report an error. In such cases, users will nee dto unassign one or more active counters to free up countes for new assignments. The interface will provide options to assign or unassign events through the group-specific interface file.

The feature can be detected via CPUID_Fn80000020_EBX_x00 bit 5. Bits Description
5 ABMC (Assignable Bandwidth Monitoring Counters)

The feature details are documented in APM listed below [1].
[1] AMD64 Architecture Programmer's Manual Volume 2: System Programming Publication # 24593 Revision 3.41 section 19.3.3.3 Assignable Bandwidth Monitoring (ABMC).

---
Note: Checkpatch checks/warnings are ignored to maintain coding style.

v13: Updated the commit log with Linux interface details.

v12: Removed the dependancy on X86_FEATURE_BMEC.
Removed the Reviewed-by tag as patch has changed.

v11: No changes.

v10: No changes.

v9: Took care of couple of minor merge conflicts. No other changes.

v8: No changes.

v7: Removed "" from feature flags. Not required anymore.

    https://lore.kernel.org/lkml/20240817145058.GCZsC40neU4wkPXeVR@fat_crate.local/

v6: Added Reinette's Reviewed-by. Moved the Checkpatch note below ---.

v5: Minor rebase change and subject line update.

v4: Changes because of rebase. Feature word 21 has few more additions now.
Changed the text to "tracked by hardware" instead of active.

v3: Change because of rebase. Actual patch did not change.

v2: Added dependency on X86_FEATURE_BMEC.

---
 arch/x86/include/asm/cpufeatures.h | 1 +
 arch/x86/kernel/cpu/cpuid-deps.c   | 2 ++
 arch/x86/kernel/cpu/scattered.c    | 1 +
 3 files changed, 4 insertions(+)
arch/x86/include/asm/cpufeatures.h +1 -0
index 6c2c152d8a67..d5c14dc678df 100644
@@ -481,6 +481,7 @@
 #define X86_FEATURE_AMD_HETEROGENEOUS_CORES (21*32 + 6) /* Heterogeneous Core Topology */
 #define X86_FEATURE_AMD_WORKLOAD_CLASS	(21*32 + 7) /* Workload Classification */
 #define X86_FEATURE_PREFER_YMM		(21*32 + 8) /* Avoid ZMM registers due to downclocking */
+#define X86_FEATURE_ABMC		(21*32 + 9) /* Assignable Bandwidth Monitoring Counters */
 
 /*
  * BUG word(s)
arch/x86/kernel/cpu/cpuid-deps.c +2 -0
index a2fbea0be535..2f54831e04e5 100644
@@ -71,6 +71,8 @@ static const struct cpuid_dep cpuid_deps[] = {
 	{ X86_FEATURE_CQM_MBM_LOCAL,		X86_FEATURE_CQM_LLC   },
 	{ X86_FEATURE_BMEC,			X86_FEATURE_CQM_MBM_TOTAL   },
 	{ X86_FEATURE_BMEC,			X86_FEATURE_CQM_MBM_LOCAL   },
+	{ X86_FEATURE_ABMC,			X86_FEATURE_CQM_MBM_TOTAL   },
+	{ X86_FEATURE_ABMC,			X86_FEATURE_CQM_MBM_LOCAL   },
 	{ X86_FEATURE_AVX512_BF16,		X86_FEATURE_AVX512VL  },
 	{ X86_FEATURE_AVX512_FP16,		X86_FEATURE_AVX512BW  },
 	{ X86_FEATURE_ENQCMD,			X86_FEATURE_XSAVES    },
arch/x86/kernel/cpu/scattered.c +1 -0
index 16f3ca30626a..3b72b72270f1 100644
@@ -49,6 +49,7 @@ static const struct cpuid_bit cpuid_bits[] = {
 	{ X86_FEATURE_MBA,			CPUID_EBX,  6, 0x80000008, 0 },
 	{ X86_FEATURE_SMBA,			CPUID_EBX,  2, 0x80000020, 0 },
 	{ X86_FEATURE_BMEC,			CPUID_EBX,  3, 0x80000020, 0 },
+	{ X86_FEATURE_ABMC,			CPUID_EBX,  5, 0x80000020, 0 },
 	{ X86_FEATURE_AMD_WORKLOAD_CLASS,	CPUID_EAX, 22, 0x80000021, 0 },
 	{ X86_FEATURE_PERFMON_V2,		CPUID_EAX,  0, 0x80000022, 0 },
 	{ X86_FEATURE_AMD_LBR_V2,		CPUID_EAX,  1, 0x80000022, 0 },
Signature · 1 line
2.34.1
author reviewer
#8 lore raw

PATCH v13 2/27 x86/resctrl: Add ABMC feature in the command line options

In reply to Babu Moger

Add the command line option to enable or disable exposing the ABMC (Assignable Bandwidth Monitoring Counters) hardware feature to resctrl.

---
v13: Removed the Reviewed-by as the file resctrl.rst is moved to
Documentation/filesystems/resctrl.rst. In that sense patch has changed.

v12: No changes.

v11: No changes.

v10: No changes.

v9: No code changes. Added Reviewed-by.

v8: Commit message update.

v7: No changes

v6: No changes

v5: No changes

v4: No changes

v3: No changes

v2: No changes

---
 Documentation/admin-guide/kernel-parameters.txt | 2 +-
 Documentation/filesystems/resctrl.rst           | 1 +
 arch/x86/kernel/cpu/resctrl/core.c              | 2 ++
 3 files changed, 4 insertions(+), 1 deletion(-)
Documentation/admin-guide/kernel-parameters.txt +1 -1
index d9fd26b95b34..ed9761bb2e4a 100644
@@ -5988,7 +5988,7 @@
 	rdt=		[HW,X86,RDT]
 			Turn on/off individual RDT features. List is:
 			cmt, mbmtotal, mbmlocal, l3cat, l3cdp, l2cat, l2cdp,
-			mba, smba, bmec.
+			mba, smba, bmec, abmc.
 			E.g. to turn on cmt and turn off mba use:
 				rdt=cmt,!mba
 
Documentation/filesystems/resctrl.rst +1 -0
index c7949dd44f2f..c97fd77a107d 100644
@@ -26,6 +26,7 @@ MBM (Memory Bandwidth Monitoring)		"cqm_mbm_total", "cqm_mbm_local"
 MBA (Memory Bandwidth Allocation)		"mba"
 SMBA (Slow Memory Bandwidth Allocation)         ""
 BMEC (Bandwidth Monitoring Event Configuration) ""
+ABMC (Assignable Bandwidth Monitoring Counters) ""
 ===============================================	================================
 
 Historically, new features were made visible by default in /proc/cpuinfo. This
arch/x86/kernel/cpu/resctrl/core.c +2 -0
index 224bed28f341..15a1dfa92923 100644
@@ -704,6 +704,7 @@ enum {
 	RDT_FLAG_MBA,
 	RDT_FLAG_SMBA,
 	RDT_FLAG_BMEC,
+	RDT_FLAG_ABMC,
 };
 
 #define RDT_OPT(idx, n, f)	\
@@ -729,6 +730,7 @@ static struct rdt_options rdt_options[]  __ro_after_init = {
 	RDT_OPT(RDT_FLAG_MBA,	    "mba",	X86_FEATURE_MBA),
 	RDT_OPT(RDT_FLAG_SMBA,	    "smba",	X86_FEATURE_SMBA),
 	RDT_OPT(RDT_FLAG_BMEC,	    "bmec",	X86_FEATURE_BMEC),
+	RDT_OPT(RDT_FLAG_ABMC,	    "abmc",	X86_FEATURE_ABMC),
 };
 #define NUM_RDT_OPTIONS ARRAY_SIZE(rdt_options)
 
Signature · 1 line
2.34.1
author reviewer
#9 lore raw

PATCH v13 3/27 x86/resctrl: Consolidate monitoring related data from rdt_resource

In reply to Babu Moger

The cache allocation and memory bandwidth allocation feature properties are consolidated into struct resctrl_cache and struct resctrl_membw respectively.

In preparation for more monitoring properties that will clobber the existing resource struct more, re-organize the monitoring specific properties to also be in a separate structure.

---
v13: Changes due to FS/ARCH restructure.

v12: Fixed the conflicts due to recent changes in rdt_resource data structure.
Added new mbm_cfg_mask field to resctrl_mon.
Removed Reviewed-by tag as patch has changed.

v11: No changes.

v10: No changes.

v9: No changes.

v8: Added Reviewed-by from Reinette. No other changes.

v7: Added kernel doc for data structure. Minor text update.

v6: Update commit message and update kernel doc for rdt_resource.

v5: Commit message update.
Also changes related to data structure updates does to SNC support.

v4: New patch.

---
 arch/x86/kernel/cpu/resctrl/core.c    |  4 ++--
 arch/x86/kernel/cpu/resctrl/monitor.c | 12 ++++++------
 fs/resctrl/monitor.c                  |  8 ++++----
 fs/resctrl/rdtgroup.c                 | 12 ++++++------
 include/linux/resctrl.h               | 22 +++++++++++++++-------
 5 files changed, 33 insertions(+), 25 deletions(-)
arch/x86/kernel/cpu/resctrl/core.c +2 -2
index 15a1dfa92923..6859566398d6 100644
@@ -106,7 +106,7 @@ u32 resctrl_arch_system_num_rmid_idx(void)
 	struct rdt_resource *r = &rdt_resources_all[RDT_RESOURCE_L3].r_resctrl;
 
 	/* RMID are independent numbers for x86. num_rmid_idx == num_rmid */
-	return r->num_rmid;
+	return r->mon.num_rmid;
 }
 
 struct rdt_resource *resctrl_arch_get_resource(enum resctrl_res_level l)
@@ -534,7 +534,7 @@ static void domain_add_cpu_mon(int cpu, struct rdt_resource *r)
 
 	arch_mon_domain_online(r, d);
 
-	if (arch_domain_mbm_alloc(r->num_rmid, hw_dom)) {
+	if (arch_domain_mbm_alloc(r->mon.num_rmid, hw_dom)) {
 		mon_domain_free(hw_dom);
 		return;
 	}
arch/x86/kernel/cpu/resctrl/monitor.c +6 -6
index 3fc4d9f56f0d..aeb2a9283069 100644
@@ -134,7 +134,7 @@ static int logical_rmid_to_physical_rmid(int cpu, int lrmid)
 	if (snc_nodes_per_l3_cache == 1)
 		return lrmid;
 
-	return lrmid + (cpu_to_node(cpu) % snc_nodes_per_l3_cache) * r->num_rmid;
+	return lrmid + (cpu_to_node(cpu) % snc_nodes_per_l3_cache) * r->mon.num_rmid;
 }
 
 static int __rmid_read_phys(u32 prmid, enum resctrl_event_id eventid, u64 *val)
@@ -208,11 +208,11 @@ void resctrl_arch_reset_rmid_all(struct rdt_resource *r, struct rdt_mon_domain *
 
 	if (resctrl_arch_is_mbm_total_enabled())
 		memset(hw_dom->arch_mbm_total, 0,
-		       sizeof(*hw_dom->arch_mbm_total) * r->num_rmid);
+		       sizeof(*hw_dom->arch_mbm_total) * r->mon.num_rmid);
 
 	if (resctrl_arch_is_mbm_local_enabled())
 		memset(hw_dom->arch_mbm_local, 0,
-		       sizeof(*hw_dom->arch_mbm_local) * r->num_rmid);
+		       sizeof(*hw_dom->arch_mbm_local) * r->mon.num_rmid);
 }
 
 static u64 mbm_overflow_count(u64 prev_msr, u64 cur_msr, unsigned int width)
@@ -350,7 +350,7 @@ int __init rdt_get_mon_l3_config(struct rdt_resource *r)
 
 	resctrl_rmid_realloc_limit = boot_cpu_data.x86_cache_size * 1024;
 	hw_res->mon_scale = boot_cpu_data.x86_cache_occ_scale / snc_nodes_per_l3_cache;
-	r->num_rmid = (boot_cpu_data.x86_cache_max_rmid + 1) / snc_nodes_per_l3_cache;
+	r->mon.num_rmid = (boot_cpu_data.x86_cache_max_rmid + 1) / snc_nodes_per_l3_cache;
 	hw_res->mbm_width = MBM_CNTR_WIDTH_BASE;
 
 	if (mbm_offset > 0 && mbm_offset <= MBM_CNTR_WIDTH_OFFSET_MAX)
@@ -365,7 +365,7 @@ int __init rdt_get_mon_l3_config(struct rdt_resource *r)
 	 *
 	 * For a 35MB LLC and 56 RMIDs, this is ~1.8% of the LLC.
 	 */
-	threshold = resctrl_rmid_realloc_limit / r->num_rmid;
+	threshold = resctrl_rmid_realloc_limit / r->mon.num_rmid;
 
 	/*
 	 * Because num_rmid may not be a power of two, round the value
@@ -379,7 +379,7 @@ int __init rdt_get_mon_l3_config(struct rdt_resource *r)
 
 		/* Detect list of bandwidth sources that can be tracked */
 		cpuid_count(0x80000020, 3, &eax, &ebx, &ecx, &edx);
-		r->mbm_cfg_mask = ecx & MAX_EVT_CONFIG_BITS;
+		r->mon.mbm_cfg_mask = ecx & MAX_EVT_CONFIG_BITS;
 	}
 
 	r->mon_capable = true;
fs/resctrl/monitor.c +4 -4
index bde2801289d3..6ffa9d14a8b4 100644
@@ -866,14 +866,14 @@ static struct mon_evt mbm_local_event = {
  */
 static void l3_mon_evt_init(struct rdt_resource *r)
 {
-	INIT_LIST_HEAD(&r->evt_list);
+	INIT_LIST_HEAD(&r->mon.evt_list);
 
 	if (resctrl_arch_is_llc_occupancy_enabled())
-		list_add_tail(&llc_occupancy_event.list, &r->evt_list);
+		list_add_tail(&llc_occupancy_event.list, &r->mon.evt_list);
 	if (resctrl_arch_is_mbm_total_enabled())
-		list_add_tail(&mbm_total_event.list, &r->evt_list);
+		list_add_tail(&mbm_total_event.list, &r->mon.evt_list);
 	if (resctrl_arch_is_mbm_local_enabled())
-		list_add_tail(&mbm_local_event.list, &r->evt_list);
+		list_add_tail(&mbm_local_event.list, &r->mon.evt_list);
 }
 
 /**
fs/resctrl/rdtgroup.c +6 -6
index ec28228f6a8d..9412c7b64523 100644
@@ -1139,7 +1139,7 @@ static int rdt_num_rmids_show(struct kernfs_open_file *of,
 {
 	struct rdt_resource *r = rdt_kn_parent_priv(of->kn);
 
-	seq_printf(seq, "%d\n", r->num_rmid);
+	seq_printf(seq, "%d\n", r->mon.num_rmid);
 
 	return 0;
 }
@@ -1150,7 +1150,7 @@ static int rdt_mon_features_show(struct kernfs_open_file *of,
 	struct rdt_resource *r = rdt_kn_parent_priv(of->kn);
 	struct mon_evt *mevt;
 
-	list_for_each_entry(mevt, &r->evt_list, list) {
+	list_for_each_entry(mevt, &r->mon.evt_list, list) {
 		seq_printf(seq, "%s\n", mevt->name);
 		if (mevt->configurable)
 			seq_printf(seq, "%s_config\n", mevt->name);
@@ -1733,9 +1733,9 @@ static int mon_config_write(struct rdt_resource *r, char *tok, u32 evtid)
 	}
 
 	/* Value from user cannot be more than the supported set of events */
-	if ((val & r->mbm_cfg_mask) != val) {
+	if ((val & r->mon.mbm_cfg_mask) != val) {
 		rdt_last_cmd_printf("Invalid event configuration: max valid mask is 0x%02x\n",
-				    r->mbm_cfg_mask);
+				    r->mon.mbm_cfg_mask);
 		return -EINVAL;
 	}
 
@@ -3055,10 +3055,10 @@ static int mon_add_all_files(struct kernfs_node *kn, struct rdt_mon_domain *d,
 	struct mon_evt *mevt;
 	int ret, domid;
 
-	if (WARN_ON(list_empty(&r->evt_list)))
+	if (WARN_ON(list_empty(&r->mon.evt_list)))
 		return -EPERM;
 
-	list_for_each_entry(mevt, &r->evt_list, list) {
+	list_for_each_entry(mevt, &r->mon.evt_list, list) {
 		domid = do_sum ? d->ci->id : d->hdr.id;
 		priv = mon_get_kn_priv(r->rid, domid, mevt, do_sum);
 		if (WARN_ON_ONCE(!priv))
include/linux/resctrl.h +15 -7
index 9ba771f2ddea..2a8fa454d3e6 100644
@@ -255,40 +255,48 @@ enum resctrl_schema_fmt {
 	RESCTRL_SCHEMA_RANGE,
 };
 
+/**
+ * struct resctrl_mon - Monitoring related data of a resctrl resource
+ * @num_rmid:		Number of RMIDs available
+ * @mbm_cfg_mask:	Bandwidth sources that can be tracked when bandwidth
+ *			monitoring events can be configured.
+ * @evt_list:		List of monitoring events
+ */
+struct resctrl_mon {
+	int			num_rmid;
+	unsigned int		mbm_cfg_mask;
+	struct list_head	evt_list;
+};
+
 /**
  * struct rdt_resource - attributes of a resctrl resource
  * @rid:		The index of the resource
  * @alloc_capable:	Is allocation available on this machine
  * @mon_capable:	Is monitor feature available on this machine
- * @num_rmid:		Number of RMIDs available
  * @ctrl_scope:		Scope of this resource for control functions
  * @mon_scope:		Scope of this resource for monitor functions
  * @cache:		Cache allocation related data
  * @membw:		If the component has bandwidth controls, their properties.
+ * @mon:		Monitoring related data.
  * @ctrl_domains:	RCU list of all control domains for this resource
  * @mon_domains:	RCU list of all monitor domains for this resource
  * @name:		Name to use in "schemata" file.
  * @schema_fmt:		Which format string and parser is used for this schema.
- * @evt_list:		List of monitoring events
- * @mbm_cfg_mask:	Bandwidth sources that can be tracked when bandwidth
- *			monitoring events can be configured.
  * @cdp_capable:	Is the CDP feature available on this resource
  */
 struct rdt_resource {
 	int			rid;
 	bool			alloc_capable;
 	bool			mon_capable;
-	int			num_rmid;
 	enum resctrl_scope	ctrl_scope;
 	enum resctrl_scope	mon_scope;
 	struct resctrl_cache	cache;
 	struct resctrl_membw	membw;
+	struct resctrl_mon	mon;
 	struct list_head	ctrl_domains;
 	struct list_head	mon_domains;
 	char			*name;
 	enum resctrl_schema_fmt	schema_fmt;
-	struct list_head	evt_list;
-	unsigned int		mbm_cfg_mask;
 	bool			cdp_capable;
 };
 
Signature · 1 line
2.34.1
author reviewer
#12 lore raw

PATCH v13 4/27 x86/resctrl: Detect Assignable Bandwidth Monitoring feature details

In reply to Babu Moger

ABMC feature details are reported via CPUID Fn8000_0020_EBX_x5. Bits Description
15:0 MAX_ABMC Maximum Supported Assignable Bandwidth
Monitoring Counter ID + 1

The feature details are documented in APM listed below [1].
[1] AMD64 Architecture Programmer's Manual Volume 2: System Programming Publication # 24593 Revision 3.41 section 19.3.3.3 Assignable Bandwidth Monitoring (ABMC).

Detect the feature and number of assignable monitoring counters supported.

---
v13: No changes.

v12: Resolved conflicts because of latest merge.
Removed Reviewed-by as the patch has changed.

v11: No changes.

v10: No changes.

v9: Added Reviewed-by tag. No code changes

v8: Used GENMASK for the mask.

v7: Removed WARN_ON for num_mbm_cntrs. Decided to dynamically allocate the
bitmap. WARN_ON is not required anymore.
Removed redundant comments.

v6: Commit message update.
Renamed abmc_capable to mbm_cntr_assignable.

v5: Name change num_cntrs to num_mbm_cntrs.
Moved abmc_capable to resctrl_mon.

v4: Removed resctrl_arch_has_abmc(). Added all the code inline. We dont
need to separate this as arch code.

v3: Removed changes related to mon_features.
Moved rdt_cpu_has to core.c and added new function resctrl_arch_has_abmc. Also moved the fields mbm_assign_capable and mbm_assign_cntrs to rdt_resource. (James)

v2: Changed the field name to mbm_assign_capable from abmc_capable.

---
 arch/x86/kernel/cpu/resctrl/monitor.c | 9 +++++++--
 include/linux/resctrl.h               | 4 ++++
 2 files changed, 11 insertions(+), 2 deletions(-)
arch/x86/kernel/cpu/resctrl/monitor.c +7 -2
index aeb2a9283069..fd2761d9f3f7 100644
@@ -345,6 +345,7 @@ int __init rdt_get_mon_l3_config(struct rdt_resource *r)
 	unsigned int mbm_offset = boot_cpu_data.x86_cache_mbm_width_offset;
 	struct rdt_hw_resource *hw_res = resctrl_to_arch_res(r);
 	unsigned int threshold;
+	u32 eax, ebx, ecx, edx;
 
 	snc_nodes_per_l3_cache = snc_get_config();
 
@@ -375,13 +376,17 @@ int __init rdt_get_mon_l3_config(struct rdt_resource *r)
 	resctrl_rmid_realloc_threshold = resctrl_arch_round_mon_val(threshold);
 
 	if (rdt_cpu_has(X86_FEATURE_BMEC)) {
-		u32 eax, ebx, ecx, edx;
-
 		/* Detect list of bandwidth sources that can be tracked */
 		cpuid_count(0x80000020, 3, &eax, &ebx, &ecx, &edx);
 		r->mon.mbm_cfg_mask = ecx & MAX_EVT_CONFIG_BITS;
 	}
 
+	if (rdt_cpu_has(X86_FEATURE_ABMC)) {
+		r->mon.mbm_cntr_assignable = true;
+		cpuid_count(0x80000020, 5, &eax, &ebx, &ecx, &edx);
+		r->mon.num_mbm_cntrs = (ebx & GENMASK(15, 0)) + 1;
+	}
+
 	r->mon_capable = true;
 
 	return 0;
include/linux/resctrl.h +4 -0
index 2a8fa454d3e6..065fb6e38933 100644
@@ -260,11 +260,15 @@ enum resctrl_schema_fmt {
  * @num_rmid:		Number of RMIDs available
  * @mbm_cfg_mask:	Bandwidth sources that can be tracked when bandwidth
  *			monitoring events can be configured.
+ * @num_mbm_cntrs:	Number of assignable monitoring counters
+ * @mbm_cntr_assignable:Is system capable of supporting monitor assignment?
  * @evt_list:		List of monitoring events
  */
 struct resctrl_mon {
 	int			num_rmid;
 	unsigned int		mbm_cfg_mask;
+	int			num_mbm_cntrs;
+	bool			mbm_cntr_assignable;
 	struct list_head	evt_list;
 };
 
Signature · 1 line
2.34.1
author reviewer
#16 lore raw

PATCH v13 5/27 x86/resctrl: Add support to enable/disable AMD ABMC feature

In reply to Babu Moger

Add the functionality to enable/disable AMD ABMC feature.

AMD ABMC feature is enabled by setting enabled bit(0) in MSR L3_QOS_EXT_CFG. When the state of ABMC is changed, the MSR needs to be updated on all the logical processors in the QOS Domain.

Hardware counters will reset when ABMC state is changed.

The ABMC feature details are documented in APM listed below [1].
[1] AMD64 Architecture Programmer's Manual Volume 2: System Programming Publication # 24593 Revision 3.41 section 19.3.3.3 Assignable Bandwidth Monitoring (ABMC).

---
v13: Resolved minor conflicts with recent FS/ARCH restructure.

v12: Clarified the comment on _resctrl_abmc_enable().
Added the code to reset arch state in _resctrl_abmc_enable(). Resolved the conflicts with latest merge.

v11: Moved the monitoring related calls to monitor.c file.
Moved the changes from include/linux/resctrl.h to arch/x86/kernel/cpu/resctrl/internal.h. Removed the Reviewed-by tag as patch changed. Actual code did not change.

v10: No changes.

v9: Re-ordered the MSR and added Reviewed-by tag.

v8: Commit message update and moved around the comments about L3_QOS_EXT_CFG
to _resctrl_abmc_enable.

v7: Renamed the function
resctrl_arch_get_abmc_enabled() to resctrl_arch_mbm_cntr_assign_enabled().

Merged resctrl_arch_mbm_cntr_assign_disable, resctrl_arch_mbm_cntr_assign_disable and renamed to resctrl_arch_mbm_cntr_assign_set().

Moved the function definition to linux/resctrl.h.

Passed the struct rdt_resource to these functions.
Removed resctrl_arch_reset_rmid_all() from arch code. This will be done from the caller.

v6: Renamed abmc_enabled to mbm_cntr_assign_enabled.
Used msr_set_bit and msr_clear_bit for msr updates.
Renamed resctrl_arch_abmc_enable() to resctrl_arch_mbm_cntr_assign_enable(). Renamed resctrl_arch_abmc_disable() to resctrl_arch_mbm_cntr_assign_disable(). Made _resctrl_abmc_enable to return void.

v5: Renamed resctrl_abmc_enable to resctrl_arch_abmc_enable.
Renamed resctrl_abmc_disable to resctrl_arch_abmc_disable. Introduced resctrl_arch_get_abmc_enabled to get abmc state from non-arch code.
Renamed resctrl_abmc_set_all to _resctrl_abmc_enable(). Modified commit log to make it clear about AMD ABMC feature.

v3: No changes.

v2: Few text changes in commit message.

---
 arch/x86/include/asm/msr-index.h       |  1 +
 arch/x86/kernel/cpu/resctrl/internal.h |  5 +++
 arch/x86/kernel/cpu/resctrl/monitor.c  | 43 ++++++++++++++++++++++++++
 include/linux/resctrl.h                |  3 ++
 4 files changed, 52 insertions(+)
arch/x86/include/asm/msr-index.h +1 -0
index e6134ef2263d..3970e0b16e47 100644
@@ -1203,6 +1203,7 @@
 /* - AMD: */
 #define MSR_IA32_MBA_BW_BASE		0xc0000200
 #define MSR_IA32_SMBA_BW_BASE		0xc0000280
+#define MSR_IA32_L3_QOS_EXT_CFG		0xc00003ff
 #define MSR_IA32_EVT_CFG_BASE		0xc0000400
 
 /* AMD-V MSRs */
arch/x86/kernel/cpu/resctrl/internal.h +5 -0
index 5e3c41b36437..fcc9d23686a1 100644
@@ -37,6 +37,9 @@ struct arch_mbm_state {
 	u64	prev_msr;
 };
 
+/* Setting bit 0 in L3_QOS_EXT_CFG enables the ABMC feature. */
+#define ABMC_ENABLE_BIT			0
+
 /**
  * struct rdt_hw_ctrl_domain - Arch private attributes of a set of CPUs that share
  *			       a resource for a control function
@@ -102,6 +105,7 @@ struct msr_param {
  * @mon_scale:		cqm counter * mon_scale = occupancy in bytes
  * @mbm_width:		Monitor width, to detect and correct for overflow.
  * @cdp_enabled:	CDP state of this resource
+ * @mbm_cntr_assign_enabled:	ABMC feature is enabled
  *
  * Members of this structure are either private to the architecture
  * e.g. mbm_width, or accessed via helpers that provide abstraction. e.g.
@@ -115,6 +119,7 @@ struct rdt_hw_resource {
 	unsigned int		mon_scale;
 	unsigned int		mbm_width;
 	bool			cdp_enabled;
+	bool			mbm_cntr_assign_enabled;
 };
 
 static inline struct rdt_hw_resource *resctrl_to_arch_res(struct rdt_resource *r)
arch/x86/kernel/cpu/resctrl/monitor.c +43 -0

46 lines not on this page · raw message

include/linux/resctrl.h +3 -0
index 065fb6e38933..bdb264875ef6 100644
@@ -428,6 +428,9 @@ static inline u32 resctrl_get_config_index(u32 closid,
 bool resctrl_arch_get_cdp_enabled(enum resctrl_res_level l);
 int resctrl_arch_set_cdp_enabled(enum resctrl_res_level l, bool enable);
 
+bool resctrl_arch_mbm_cntr_assign_enabled(struct rdt_resource *r);
+int resctrl_arch_mbm_cntr_assign_set(struct rdt_resource *r, bool enable);
+
 /*
  * Update the ctrl_val and apply this config right now.
  * Must be called on one of the domain's CPUs.
Signature · 1 line
2.34.1
author reviewer
#19 lore raw

PATCH v13 6/27 x86/resctrl: Introduce the interface to display monitor mode

In reply to Babu Moger

Introduce the resctrl file "mbm_assign_mode" to list monitor modes supported.

The "mbm_cntr_assign" mode provides the option to assign a counter to an RMID, event pair and monitor the bandwidth as long as it is assigned.

On AMD systems "mbm_cntr_assign" mode is backed by the ABMC (Assignable Bandwidth Monitoring Counters) hardware feature and is enabled by default.

The "default" mode is the existing monitoring mode that works without the explicit counter assignment, instead relying on dynamic counter assignment by hardware that may result in hardware not dedicating a counter resulting in monitoring data reads returning "Unavailable".

Provide an interface to display the monitor mode on the system.

$ cat /sys/fs/resctrl/info/L3_MON/mbm_assign_mode
[mbm_cntr_assign]
default

Add IS_ENABLED(CONFIG_RESCTRL_ASSIGN_FIXED) check to support Arm64.

On x86, CONFIG_RESCTRL_ASSIGN_FIXED is not defined. On Arm64, it will be defined when the "mbm_cntr_assign" mode is supported.

Add IS_ENABLED(CONFIG_RESCTRL_ASSIGN_FIXED) check early to ensure the user interface remains compatible with upcoming Arm64 support. IS_ENABLED() safely evaluates to 0 when the configuration is not defined.

As a result, for MPAM, the display would be either:
[default]
or
[mbm_cntr_assign]

---
v13: Updated the commit log with motivation for adding CONFIG_RESCTRL_ASSIGN_FIXED.
Added fflag RFTYPE_RES_CACHE for mbm_assign_mode file.
Updated user doc. Removed the references to "mbm_assign_control". Resolved the conflicts with latest FS/ARCH code restructure.

v12: Minor text update in change log and user documentation.
Added the check CONFIG_RESCTRL_ASSIGN_FIXED to take care of arm platforms. This will be defined only in arm and not in x86.

v11: Renamed rdtgroup_mbm_assign_mode_show() to resctrl_mbm_assign_mode_show().
Removed few texts in resctrl.rst about AMD specific information. Updated few texts.

v10: Added few more text to user documentation clarify on the default mode.

v9: Updated user documentation based on comments.

v8: Commit message update.

v7: Updated the descriptions/commit log in resctrl.rst to generic text.
Thanks to James and Reinette.
Rename mbm_mode to mbm_assign_mode.
Introduced mutex lock in rdtgroup_mbm_mode_show().

v6: Added documentation for mbm_cntr_assign and legacy mode.
Moved mbm_mode fflags initialization to static initialization.

v5: Changed interface name to mbm_mode.
It will be always available even if ABMC feature is not supported. Added description in resctrl.rst about ABMC mode.
Fixed display abmc and legacy consistantly.

v4: Fixed the checks for legacy and abmc mode. Default it ABMC.

v3: New patch to display ABMC capability.

---
 Documentation/filesystems/resctrl.rst | 27 +++++++++++++++++++
 fs/resctrl/rdtgroup.c                 | 37 +++++++++++++++++++++++++++
 2 files changed, 64 insertions(+)
Documentation/filesystems/resctrl.rst +27 -0
index c97fd77a107d..8013418b6ca2 100644
@@ -257,6 +257,33 @@ with the following files:
 	    # cat /sys/fs/resctrl/info/L3_MON/mbm_local_bytes_config
 	    0=0x30;1=0x30;3=0x15;4=0x15
 
+"mbm_assign_mode":
+	Reports the list of monitoring modes supported. The enclosed brackets
+	indicate which mode is enabled.
+	::
+
+	  # cat /sys/fs/resctrl/info/L3_MON/mbm_assign_mode
+	  [mbm_cntr_assign]
+	  default
+
+	"mbm_cntr_assign":
+
+	In mbm_cntr_assign mode, a monitoring event can only accumulate data
+	while it is backed by a hardware counter. Use "mbm_L3_assignments" found
+	in each CTRL_MON and MON group to specify which of the events should have
+	a counter assigned. The number of counters available is described in the
+	"num_mbm_cntrs" file. Changing the mode may cause all counters on the
+	resource to reset.
+
+	"default":
+
+	In default mode, resctrl assumes there is a hardware counter for each
+	event within every CTRL_MON and MON group. On AMD platforms, it is
+	recommended to use the mbm_cntr_assign mode, if supported, to prevent
+	reset of MBM events between reads resulting from hardware re-allocating
+	counters. This can result in misleading values or display "Unavailable"
+	if no counter is assigned to the event.
+
 "max_threshold_occupancy":
 		Read/write file provides the largest value (in
 		bytes) at which a previously used LLC_occupancy
fs/resctrl/rdtgroup.c +37 -0
index 9412c7b64523..a9e1055df75f 100644
@@ -1801,6 +1801,36 @@ static ssize_t mbm_local_bytes_config_write(struct kernfs_open_file *of,
 	return ret ?: nbytes;
 }
 
+static int resctrl_mbm_assign_mode_show(struct kernfs_open_file *of,
+					struct seq_file *s, void *v)
+{
+	struct rdt_resource *r = rdt_kn_parent_priv(of->kn);
+	bool enabled;
+
+	mutex_lock(&rdtgroup_mutex);
+	enabled = resctrl_arch_mbm_cntr_assign_enabled(r);
+
+	if (r->mon.mbm_cntr_assignable) {
+		if (enabled)
+			seq_puts(s, "[mbm_cntr_assign]\n");
+		else
+			seq_puts(s, "[default]\n");
+
+		if (!IS_ENABLED(CONFIG_RESCTRL_ASSIGN_FIXED)) {
+			if (enabled)
+				seq_puts(s, "default\n");
+			else
+				seq_puts(s, "mbm_cntr_assign\n");
+		}
+	} else {
+		seq_puts(s, "[default]\n");
+	}
+
+	mutex_unlock(&rdtgroup_mutex);
+
+	return 0;
+}
+
 /* rdtgroup information files for one cache resource. */
 static struct rftype res_common_files[] = {
 	{
@@ -1913,6 +1943,13 @@ static struct rftype res_common_files[] = {
 		.seq_show	= mbm_local_bytes_config_show,
 		.write		= mbm_local_bytes_config_write,
 	},
+	{
+		.name		= "mbm_assign_mode",
+		.mode		= 0444,
+		.kf_ops		= &rdtgroup_kf_single_ops,
+		.seq_show	= resctrl_mbm_assign_mode_show,
+		.fflags		= RFTYPE_MON_INFO | RFTYPE_RES_CACHE,
+	},
 	{
 		.name		= "cpus",
 		.mode		= 0644,
Signature · 1 line
2.34.1
author reviewer
#22 lore raw

PATCH v13 7/27 x86/resctrl: Introduce interface to display number of monitoring counters

In reply to Babu Moger

The mbm_cntr_assign mode provides an option to the user to assign a counter to an RMID, event pair and monitor the bandwidth as long as the counter is assigned. Number of assignments depend on number of monitoring counters available.

Create 'num_mbm_cntrs' resctrl file that displays the number of monitoring counters supported in each domain. 'num_mbm_cntrs' is only visible to user space when the system supports mbm_cntr_assign mode.

---
v13: Updated the changelog.
Added fflags RFTYPE_RES_CACHE to the file num_mbm_cntrs.
Replaced seq_puts from seq_putc where applicable.
Resolved conflicts caused by the recent FS/ARCH code restructure. The files monitor.c/rdtgroup.c have been split between FS and ARCH directories.

v12: Changed the code to display the max supported monitoring counters in
each domain. Also updated the documentation.
Resolved the conflict with the latest code.

v11: Renamed rdtgroup_num_mbm_cntrs_show() to resctrl_num_mbm_cntrs_show().
Few monor text updates.

v10: No changes.

v9: Updated user document based on the comments.
Will add a new file available_mbm_cntrs later in the series.

v8: Commit message update and documentation update.

v7: Minor commit log text changes.

v6: No changes.

v5: Changed the display name from num_cntrs to num_mbm_cntrs.
Updated the commit message.
Moved the patch after mbm_mode is introduced.

v4: Changed the counter name to num_cntrs. And few text changes.

v3: Changed the field name to mbm_assign_cntrs.

v2: Changed the field name to mbm_assignable_counters from abmc_counter.

---
 Documentation/filesystems/resctrl.rst | 11 ++++++++++
 fs/resctrl/monitor.c                  |  4 ++++
 fs/resctrl/rdtgroup.c                 | 30 +++++++++++++++++++++++++++
 3 files changed, 45 insertions(+)
Documentation/filesystems/resctrl.rst +11 -0
index 8013418b6ca2..abbbbdf61062 100644
@@ -284,6 +284,17 @@ with the following files:
 	counters. This can result in misleading values or display "Unavailable"
 	if no counter is assigned to the event.
 
+"num_mbm_cntrs":
+	The maximum number of monitoring counters (total of available and assigned
+	counters) in each domain when the system supports mbm_cntr_assign mode.
+
+	For example, on a system with maximum of 32 memory bandwidth monitoring
+	counters in each of its L3 domains:
+	::
+
+	  # cat /sys/fs/resctrl/info/L3_MON/num_mbm_cntrs
+	  0=32;1=32
+
 "max_threshold_occupancy":
 		Read/write file provides the largest value (in
 		bytes) at which a previously used LLC_occupancy
fs/resctrl/monitor.c +4 -0
index 6ffa9d14a8b4..039516cf410d 100644
@@ -918,6 +918,10 @@ int resctrl_mon_resource_init(void)
 	else if (resctrl_arch_is_mbm_total_enabled())
 		mba_mbps_default_event = QOS_L3_MBM_TOTAL_EVENT_ID;
 
+	if (r->mon.mbm_cntr_assignable)
+		resctrl_file_fflags_init("num_mbm_cntrs",
+					 RFTYPE_MON_INFO | RFTYPE_RES_CACHE);
+
 	return 0;
 }
 
fs/resctrl/rdtgroup.c +30 -0
index a9e1055df75f..51f8f8d3ccbc 100644
@@ -1831,6 +1831,30 @@ static int resctrl_mbm_assign_mode_show(struct kernfs_open_file *of,
 	return 0;
 }
 
+static int resctrl_num_mbm_cntrs_show(struct kernfs_open_file *of,
+				      struct seq_file *s, void *v)
+{
+	struct rdt_resource *r = rdt_kn_parent_priv(of->kn);
+	struct rdt_mon_domain *dom;
+	bool sep = false;
+
+	cpus_read_lock();
+	mutex_lock(&rdtgroup_mutex);
+
+	list_for_each_entry(dom, &r->mon_domains, hdr.list) {
+		if (sep)
+			seq_putc(s, ';');
+
+		seq_printf(s, "%d=%d", dom->hdr.id, r->mon.num_mbm_cntrs);
+		sep = true;
+	}
+	seq_putc(s, '\n');
+
+	mutex_unlock(&rdtgroup_mutex);
+	cpus_read_unlock();
+	return 0;
+}
+
 /* rdtgroup information files for one cache resource. */
 static struct rftype res_common_files[] = {
 	{
@@ -1868,6 +1892,12 @@ static struct rftype res_common_files[] = {
 		.seq_show	= rdt_default_ctrl_show,
 		.fflags		= RFTYPE_CTRL_INFO | RFTYPE_RES_CACHE,
 	},
+	{
+		.name		= "num_mbm_cntrs",
+		.mode		= 0444,
+		.kf_ops		= &rdtgroup_kf_single_ops,
+		.seq_show	= resctrl_num_mbm_cntrs_show,
+	},
 	{
 		.name		= "min_cbm_bits",
 		.mode		= 0444,
Signature · 1 line
2.34.1
author reviewer
#23 lore raw

PATCH v13 8/27 x86/resctrl: Introduce mbm_cntr_cfg to track assignable counters at domain

In reply to Babu Moger

In mbm_cntr_assign mode hardware counters are assigned/unassigned to an MBM event of a monitor group. Hardware counters are assigned/unassigned at monitoring domain level.

Manage a monitoring domain's hardware counters using a per monitoring domain array of struct mbm_cntr_cfg that is indexed by the hardware counter ID. A hardware counter's configuration contains the MBM event ID and points to the monitoring group that it is assigned to, with a NULL pointer meaning that the hardware counter is available for assignment.

There is no direct way to determine which hardware counters are assigned to a particular monitoring group. Check every entry of every hardware counter configuration array in every monitoring domain to query which MBM events of a monitoring group is tracked by hardware. Such queries are acceptable because of a very small number of assignable counters (32 to 64).

---
v13: Resolved conflicts caused by the recent FS/ARCH code restructure.
The files monitor.c/rdtgroup.c have been split between FS and ARCH directories.

v12: Fixed the struct mbm_cntr_cfg code documentation.
Removed few strange charactors in changelog.
Added the counter range for better understanding. Moved the struct mbm_cntr_cfg definition to resctrl/internal.h as suggested by James.

v11: Refined the change log based on Reinette's feedback.
Fixed few style issues.

v10: Patch changed completely to handle the counters at domain level.
https://lore.kernel.org/lkml/CALPaoCj+zWq1vkHVbXYP0znJbe6Ke3PXPWjtri5AFgD9cQDCUg@mail.gmail.com/
Removed Reviewed-by tag.
Did not see the need to add cntr_id in mbm_state structure. Not used in the code.

v9: Added Reviewed-by tag. No other changes.

v8: Minor commit message changes.

v7: Added check mbm_cntr_assignable for allocating bitmap mbm_cntr_map

v6: New patch to add domain level assignment.

---
 fs/resctrl/rdtgroup.c   | 11 +++++++++++
 include/linux/resctrl.h | 16 ++++++++++++++++
 2 files changed, 27 insertions(+)
fs/resctrl/rdtgroup.c +11 -0
index 51f8f8d3ccbc..e2005fc9acd9 100644
@@ -4085,6 +4085,7 @@ static void rdtgroup_setup_default(void)
 
 static void domain_destroy_mon_state(struct rdt_mon_domain *d)
 {
+	kfree(d->cntr_cfg);
 	bitmap_free(d->rmid_busy_llc);
 	kfree(d->mbm_total);
 	kfree(d->mbm_local);
@@ -4171,6 +4172,16 @@ static int domain_setup_mon_state(struct rdt_resource *r, struct rdt_mon_domain
 			return -ENOMEM;
 		}
 	}
+	if (resctrl_is_mbm_enabled() && r->mon.mbm_cntr_assignable) {
+		tsize = sizeof(*d->cntr_cfg);
+		d->cntr_cfg = kcalloc(r->mon.num_mbm_cntrs, tsize, GFP_KERNEL);
+		if (!d->cntr_cfg) {
+			bitmap_free(d->rmid_busy_llc);
+			kfree(d->mbm_total);
+			kfree(d->mbm_local);
+			return -ENOMEM;
+		}
+	}
 
 	return 0;
 }
include/linux/resctrl.h +16 -0
index bdb264875ef6..d77981d1fcb9 100644
@@ -156,6 +156,20 @@ struct rdt_ctrl_domain {
 	u32				*mbps_val;
 };
 
+/**
+ * struct mbm_cntr_cfg - Assignable counter configuration
+ * @evtid:		MBM event to which the counter is assigned. Only valid
+ *			if @rdtgroup is not NULL.
+ * @evt_cfg:		Event configuration value.
+ * @rdtgrp:		resctrl group assigned to the counter. NULL if the
+ *			counter is free.
+ */
+struct mbm_cntr_cfg {
+	enum resctrl_event_id   evtid;
+	u32                     evt_cfg;
+	struct rdtgroup         *rdtgrp;
+};
+
 /**
  * struct rdt_mon_domain - group of CPUs sharing a resctrl monitor resource
  * @hdr:		common header for different domain types
@@ -167,6 +181,7 @@ struct rdt_ctrl_domain {
  * @cqm_limbo:		worker to periodically read CQM h/w counters
  * @mbm_work_cpu:	worker CPU for MBM h/w counters
  * @cqm_work_cpu:	worker CPU for CQM h/w counters
+ * @cntr_cfg:		assignable counters configuration
  */
 struct rdt_mon_domain {
 	struct rdt_domain_hdr		hdr;
@@ -178,6 +193,7 @@ struct rdt_mon_domain {
 	struct delayed_work		cqm_limbo;
 	int				mbm_work_cpu;
 	int				cqm_work_cpu;
+	struct mbm_cntr_cfg		*cntr_cfg;
 };
 
 /**
Signature · 1 line
2.34.1
author reviewer
#28 lore raw

PATCH v13 9/27 x86/resctrl: Introduce interface to display number of free MBM counters

In reply to Babu Moger

Provide the interface to display the number of monitoring counters available for assignment in each domain when mbm_cntr_assign mode is enabled.

---
v13: Resolved conflicts caused by the recent FS/ARCH code restructure.
The files monitor.c and rdtgroup.c file has now been split between the FS and ARCH directories.

v12: Minor change to change log.
Updated the documentation text with an example.
Replaced seq_puts(s, ";") with seq_putc(s, ';');
Added missing rdt_last_cmd_clear() in resctrl_available_mbm_cntrs_show().

v11: Rename rdtgroup_available_mbm_cntrs_show() to resctrl_available_mbm_cntrs_show().
Few minor text changes.

v10: Patch changed to handle the counters at domain level.
https://lore.kernel.org/lkml/CALPaoCj+zWq1vkHVbXYP0znJbe6Ke3PXPWjtri5AFgD9cQDCUg@mail.gmail.com/
So, display logic also changed now.

v9: New patch

---
 Documentation/filesystems/resctrl.rst | 11 ++++++
 fs/resctrl/monitor.c                  |  5 ++-
 fs/resctrl/rdtgroup.c                 | 48 +++++++++++++++++++++++++++
 3 files changed, 63 insertions(+), 1 deletion(-)
Documentation/filesystems/resctrl.rst +11 -0
index abbbbdf61062..2bfad43aac9c 100644
@@ -295,6 +295,17 @@ with the following files:
 	  # cat /sys/fs/resctrl/info/L3_MON/num_mbm_cntrs
 	  0=32;1=32
 
+"available_mbm_cntrs":
+	The number of monitoring counters available for assignment in each
+	domain when mbm_cntr_assign mode is enabled on the system.
+
+	For example, on a system with 30 available [hardware] monitoring counters
+	in each of its L3 domains:
+	::
+
+	  # cat /sys/fs/resctrl/info/L3_MON/available_mbm_cntrs
+	  0=30;1=30
+
 "max_threshold_occupancy":
 		Read/write file provides the largest value (in
 		bytes) at which a previously used LLC_occupancy
fs/resctrl/monitor.c +4 -1
index 039516cf410d..2548aee0151c 100644
@@ -918,9 +918,12 @@ int resctrl_mon_resource_init(void)
 	else if (resctrl_arch_is_mbm_total_enabled())
 		mba_mbps_default_event = QOS_L3_MBM_TOTAL_EVENT_ID;
 
-	if (r->mon.mbm_cntr_assignable)
+	if (r->mon.mbm_cntr_assignable) {
 		resctrl_file_fflags_init("num_mbm_cntrs",
 					 RFTYPE_MON_INFO | RFTYPE_RES_CACHE);
+		resctrl_file_fflags_init("available_mbm_cntrs",
+					 RFTYPE_MON_INFO | RFTYPE_RES_CACHE);
+	}
 
 	return 0;
 }
fs/resctrl/rdtgroup.c +48 -0

60 lines not on this page · raw message

Signature · 1 line
2.34.1
author reviewer
#29 lore raw

PATCH v13 10/27 x86/resctrl: Add data structures and definitions for ABMC assignment

In reply to Babu Moger

The ABMC feature provides an option to the user to assign a hardware counter to an RMID, event pair and monitor the bandwidth as long as the counter is assigned. The bandwidth events will be tracked by the hardware until the user changes the configuration. Each resctrl group can configure maximum two counters, one for total event and one for local event.

The ABMC feature implements an MSR L3_QOS_ABMC_CFG (C000_03FDh). ABMC counter assignment is done by setting the counter id, bandwidth source (RMID) and bandwidth configuration. Users will have the option to change the bandwidth configuration using resctrl interface which will be introduced later in the series.

Attempts to read or write the MSR when ABMC is not enabled will result in a #GP(0) exception.

Introduce the data structures and definitions for MSR L3_QOS_ABMC_CFG (0xC000_03FDh):
=========================================================================
Bits Mnemonic Description Access Reset
Type Value
=========================================================================
63 CfgEn Configuration Enable R/W 0

62 CtrEn Enable/disable counting R/W 0

61:53 – Reserved MBZ 0

52:48 CtrID Counter Identifier R/W 0

47 IsCOS BwSrc field is a CLOSID R/W 0
(not an RMID)

46:44 – Reserved MBZ 0

43:32 BwSrc Bandwidth Source R/W 0
(RMID or CLOSID)

31:0 BwType Bandwidth configuration R/W 0
to track for this counter
==========================================================================

The feature details are documented in the APM listed below [1].
[1] AMD64 Architecture Programmer's Manual Volume 2: System Programming Publication # 24593 Revision 3.41 section 19.3.3.3 Assignable Bandwidth Monitoring (ABMC).

---
v13: Removed the Reviewed-by tag as there is commit log change to remove
BMEC reference.

v12: No changes.

v11: No changes.

v10: No changes.

v9: Removed the references of L3_QOS_ABMC_DSC.
Text changes about configuration in kernel doc.

v8: Update the configuration notes in kernel_doc.
Few commit message update.

v7: Removed the reference of L3_QOS_ABMC_DSC as it is not used anymore.
Moved the configuration notes to kernel_doc.
Adjusted the tabs for l3_qos_abmc_cfg and checkpatch seems happy.

v6: Removed all the fs related changes.
Added note on CfgEn,CtrEn.
Removed the definitions which are not used.
Removed cntr_id initialization.

v5: Moved assignment flags here (path 10/19 of v4).
Added MON_CNTR_UNSET definition to initialize cntr_id's. More details in commit log.
Renamed few fields in l3_qos_abmc_cfg for readability.

v4: Added more descriptions.
Changed the name abmc_ctr_id to ctr_id.
Added L3_QOS_ABMC_DSC. Used for reading the configuration.

v3: No changes.

v2: No changes.

---
 arch/x86/include/asm/msr-index.h       |  1 +
 arch/x86/kernel/cpu/resctrl/internal.h | 35 ++++++++++++++++++++++++++
 2 files changed, 36 insertions(+)
arch/x86/include/asm/msr-index.h +1 -0
index 3970e0b16e47..b5b5ebead24f 100644
@@ -1203,6 +1203,7 @@
 /* - AMD: */
 #define MSR_IA32_MBA_BW_BASE		0xc0000200
 #define MSR_IA32_SMBA_BW_BASE		0xc0000280
+#define MSR_IA32_L3_QOS_ABMC_CFG	0xc00003fd
 #define MSR_IA32_L3_QOS_EXT_CFG		0xc00003ff
 #define MSR_IA32_EVT_CFG_BASE		0xc0000400
 
arch/x86/kernel/cpu/resctrl/internal.h +35 -0
index fcc9d23686a1..db6b0c28ee6b 100644
@@ -164,6 +164,41 @@ union cpuid_0x10_x_edx {
 	unsigned int full;
 };
 
+/*
+ * ABMC counters are configured by writing to L3_QOS_ABMC_CFG.
+ * @bw_type		: Bandwidth configuration (supported by BMEC)
+ *			  tracked by the @cntr_id.
+ * @bw_src		: Bandwidth source (RMID or CLOSID).
+ * @reserved1		: Reserved.
+ * @is_clos		: @bw_src field is a CLOSID (not an RMID).
+ * @cntr_id		: Counter identifier.
+ * @reserved		: Reserved.
+ * @cntr_en		: Counting enable bit.
+ * @cfg_en		: Configuration enable bit.
+ *
+ * Configuration and counting:
+ * Counter can be configured across multiple writes to MSR. Configuration
+ * is applied only when @cfg_en = 1. Counter @cntr_id is reset when the
+ * configuration is applied.
+ * @cfg_en = 1, @cntr_en = 0 : Apply @cntr_id configuration but do not
+ *                             count events.
+ * @cfg_en = 1, @cntr_en = 1 : Apply @cntr_id configuration and start
+ *                             counting events.
+ */
+union l3_qos_abmc_cfg {
+	struct {
+		unsigned long bw_type  :32,
+			      bw_src   :12,
+			      reserved1: 3,
+			      is_clos  : 1,
+			      cntr_id  : 5,
+			      reserved : 9,
+			      cntr_en  : 1,
+			      cfg_en   : 1;
+	} split;
+	unsigned long full;
+};
+
 void rdt_ctrl_update(void *arg);
 
 int rdt_get_mon_l3_config(struct rdt_resource *r);
Signature · 1 line
2.34.1
author reviewer
#32 lore raw

PATCH v13 11/27 x86/resctrl: Implement resctrl_arch_config_cntr() to assign a counter with ABMC

In reply to Babu Moger

The ABMC feature provides an option to the user to assign a hardware counter to an RMID, event pair and monitor the bandwidth as long as it is assigned. The assigned RMID will be tracked by the hardware until the user unassigns it manually.

Implement an architecture-specific handler to assign and unassign the counter. Configure counters by writing to the L3_QOS_ABMC_CFG MSR, specifying the counter ID, bandwidth source (RMID), and event configuration.

The feature details are documented in the APM listed below [1].
[1] AMD64 Architecture Programmer's Manual Volume 2: System Programming

    Publication # 24593 Revision 3.41 section 19.3.3.3 Assignable Bandwidth
    Monitoring (ABMC).

---
v13: Moved resctrl_arch_config_cntr() prototype to include/linux/resctrl.h.
Changed resctrl_arch_config_cntr() to retun void from int. Updated the kernal doc for the prototype.
Updated the code comment.

12: Added the check to reset the architecture-specific state only when
assign is requested.
Added evt_cfg as the parameter as the user will be passing the event configuration from /info/L3_MON/event_configs/.

v11: Moved resctrl_arch_assign_cntr() and resctrl_abmc_config_one_amd() to
monitor.c.
Added the code to reset the arch state in resctrl_arch_assign_cntr(). Also removed resctrl_arch_reset_rmid() inside IPI as the counters are reset from the callers.
Re-wrote commit message.

v10: Added call resctrl_arch_reset_rmid() to reset the RMID in the domain
inside IPI call.
SMP and non-SMP call support is not required in resctrl_arch_config_cntr with new domain specific assign approach/data structure. Commit message update.

v9: Removed the code to reset the architectural state. It will done
in another patch.

v8: Rename resctrl_arch_assign_cntr to resctrl_arch_config_cntr.

v7: Separated arch and fs functions. This patch only has arch implementation.
Added struct rdt_resource to the interface resctrl_arch_assign_cntr. Rename rdtgroup_abmc_cfg() to resctrl_abmc_config_one_amd().

v6: Removed mbm_cntr_alloc() from this patch to keep fs and arch code
separate.
Added code to update the counter assignment at domain level.

v5: Few name changes to match cntr_id.
Changed the function names to
rdtgroup_assign_cntr
resctr_arch_assign_cntr
More comments on commit log.
Added function summary.

v4: Commit message update.
User bitmap APIs where applicable. Changed the interfaces considering MPAM(arm). Added domain specific assignment.

v3: Removed the static from the prototype of rdtgroup_assign_abmc.
The function is not called directly from user anymore. These

---
 arch/x86/kernel/cpu/resctrl/monitor.c | 37 +++++++++++++++++++++++++++
 include/linux/resctrl.h               | 17 ++++++++++++
 2 files changed, 54 insertions(+)
arch/x86/kernel/cpu/resctrl/monitor.c +37 -0
index ff4b2abfa044..e31084f7babd 100644
@@ -448,3 +448,40 @@ inline bool resctrl_arch_mbm_cntr_assign_enabled(struct rdt_resource *r)
 {
 	return resctrl_to_arch_res(r)->mbm_cntr_assign_enabled;
 }
+
+static void resctrl_abmc_config_one_amd(void *info)
+{
+	union l3_qos_abmc_cfg *abmc_cfg = info;
+
+	wrmsrl(MSR_IA32_L3_QOS_ABMC_CFG, abmc_cfg->full);
+}
+
+/*
+ * Send an IPI to the domain to assign the counter to RMID, event pair.
+ */
+void resctrl_arch_config_cntr(struct rdt_resource *r, struct rdt_mon_domain *d,
+			      enum resctrl_event_id evtid, u32 rmid, u32 closid,
+			      u32 cntr_id, u32 evt_cfg, bool assign)
+{
+	struct rdt_hw_mon_domain *hw_dom = resctrl_to_arch_mon_dom(d);
+	union l3_qos_abmc_cfg abmc_cfg = { 0 };
+	struct arch_mbm_state *am;
+
+	abmc_cfg.split.cfg_en = 1;
+	abmc_cfg.split.cntr_en = assign ? 1 : 0;
+	abmc_cfg.split.cntr_id = cntr_id;
+	abmc_cfg.split.bw_src = rmid;
+	abmc_cfg.split.bw_type = evt_cfg;
+
+	smp_call_function_any(&d->hdr.cpu_mask, resctrl_abmc_config_one_amd, &abmc_cfg, 1);
+
+	/*
+	 * The hardware counter is reset (because cfg_en == 1) so there is no
+	 * need to record initial non-zero counts.
+	 */
+	if (assign) {
+		am = get_arch_mbm_state(hw_dom, rmid, evtid);
+		if (am)
+			memset(am, 0, sizeof(*am));
+	}
+}
include/linux/resctrl.h +17 -0
index d77981d1fcb9..59a4fe60ab46 100644
@@ -559,6 +559,23 @@ void resctrl_arch_reset_rmid_all(struct rdt_resource *r, struct rdt_mon_domain *
  */
 void resctrl_arch_reset_all_ctrls(struct rdt_resource *r);
 
+/**
+ * resctrl_arch_config_cntr() - Configure the counter id to RMID, event
+ *				pair on the domain.
+ * @r:			Resource structure.
+ * @d:			Domain that the counter id to be configured.
+ * @evtid:		Event type to configure.
+ * @rmid:		RMID to configure.
+ * @closid:		CLOSID to configure.
+ * @cntr_id:		Counter ID to configure.
+ * @evt_cfg:		MBM event configuration value representing reads,
+ *			writes etc.
+ * @assign:		Assign or unassign.
+ */
+void resctrl_arch_config_cntr(struct rdt_resource *r, struct rdt_mon_domain *d,
+			      enum resctrl_event_id evtid, u32 rmid, u32 closid,
+			      u32 cntr_id, u32 evt_cfg, bool assign);
+
 extern unsigned int resctrl_rmid_realloc_threshold;
 extern unsigned int resctrl_rmid_realloc_limit;
 
Signature · 1 line
2.34.1
author reviewer
#51 lore raw

PATCH v13 12/27 x86/resctrl: Introduce event configuration modes

In reply to Babu Moger

MBM events can be configured using either BMEC (Bandwidth Monitoring Event Configuration) or the mbm_cntr_assign mode.

Introduce a data structure to represent the various event configuration modes and their corresponding values.

---
v13: New patch to handle different event configuration types with
mbm_cntr_assign mode.

---
 fs/resctrl/internal.h         |  6 ++++--
 fs/resctrl/monitor.c          |  4 ++--
 fs/resctrl/rdtgroup.c         |  2 +-
 include/linux/resctrl_types.h | 11 +++++++++++
 4 files changed, 18 insertions(+), 5 deletions(-)
fs/resctrl/internal.h +4 -2
index 9a8cf6f11151..0fae374559ba 100644
@@ -55,13 +55,15 @@ static inline struct rdt_fs_context *rdt_fc2context(struct fs_context *fc)
  * struct mon_evt - Entry in the event list of a resource
  * @evtid:		event id
  * @name:		name of the event
- * @configurable:	true if the event is configurable
+ * @mbm_mode:		monitoring mode (BMEC or mbm_cntr_assign)
+ * @evt_cfg:		event configuration value decoding reads, writes.
  * @list:		entry in &rdt_resource->evt_list
  */
 struct mon_evt {
 	enum resctrl_event_id	evtid;
 	char			*name;
-	bool			configurable;
+	enum resctrl_mbm_mode	mbm_mode;
+	u32			evt_cfg;
 	struct list_head	list;
 };
 
fs/resctrl/monitor.c +2 -2
index 2548aee0151c..8e403587a02f 100644
@@ -903,12 +903,12 @@ int resctrl_mon_resource_init(void)
 	l3_mon_evt_init(r);
 
 	if (resctrl_arch_is_evt_configurable(QOS_L3_MBM_TOTAL_EVENT_ID)) {
-		mbm_total_event.configurable = true;
+		mbm_total_event.mbm_mode = MBM_MODE_BMEC;
 		resctrl_file_fflags_init("mbm_total_bytes_config",
 					 RFTYPE_MON_INFO | RFTYPE_RES_CACHE);
 	}
 	if (resctrl_arch_is_evt_configurable(QOS_L3_MBM_LOCAL_EVENT_ID)) {
-		mbm_local_event.configurable = true;
+		mbm_local_event.mbm_mode = MBM_MODE_BMEC;
 		resctrl_file_fflags_init("mbm_local_bytes_config",
 					 RFTYPE_MON_INFO | RFTYPE_RES_CACHE);
 	}
fs/resctrl/rdtgroup.c +1 -1
index 752750e3e443..f192b2736a77 100644
@@ -1152,7 +1152,7 @@ static int rdt_mon_features_show(struct kernfs_open_file *of,
 
 	list_for_each_entry(mevt, &r->mon.evt_list, list) {
 		seq_printf(seq, "%s\n", mevt->name);
-		if (mevt->configurable)
+		if (mevt->mbm_mode == MBM_MODE_BMEC)
 			seq_printf(seq, "%s_config\n", mevt->name);
 	}
 
include/linux/resctrl_types.h +11 -0
index a25fb9c4070d..26cd1fec72db 100644
@@ -47,4 +47,15 @@ enum resctrl_event_id {
 	QOS_NUM_EVENTS,
 };
 
+/*
+ * Event configuration mode.
+ * Events can be configured either in BMEC (Bandwidth Monitoring Event
+ * Configuration) mode or mbm_cntr_assign mode.
+ */
+enum resctrl_mbm_mode {
+	MBM_MODE_NONE,
+	MBM_MODE_BMEC,
+	MBM_MODE_ASSIGN,
+};
+
 #endif /* __LINUX_RESCTRL_TYPES_H */
Signature · 1 line
2.34.1
author reviewer
#54 lore raw

PATCH v13 13/27 x86/resctrl: Add the functionality to assign MBM events

In reply to Babu Moger

The mbm_cntr_assign mode offers "num_mbm_cntrs" number of counters that can be assigned to RMID, event pair and monitor the bandwidth as long as it is assigned.

Add the functionality to allocate and assign a counter to am RMID, event pair in the domain.

If all the counters are in use, kernel will log the error message "Unable to allocate counter in domain" in /sys/fs/resctrl/info/last_cmd_status when a new assignment is requested. Exit on the first failure when assigning counters across all the domains.

---
v13: Updated changelog.
Changed resctrl_arch_config_cntr() to return void instead of int. Just passing evtid is to resctrl_alloc_config_cntr() and resctrl_assign_cntr_event(). Event configuration value can be easily obtained from mon_evt list.
Introduced new function mbm_get_mon_event() to get event configuration value. Added prototype descriptions to mbm_cntr_get() and mbm_cntr_alloc(). Resolved conflicts caused by the recent FS/ARCH code restructure. The files monitor.c/rdtgroup.c have been split between FS and ARCH directories.

v12: Fixed typo in the subjest line.
Replaced several counters with "num_mbm_cntrs" counters.
Changed the check in resctrl_alloc_config_cntr() to reduce the indentation. Fixed the handling error on first failure.
Added domain id and event id on failure.
Fixed the return error override.
Added new parameter event configuration (evt_cfg) to get the event configuration from user space.

v11: Patch changed again quite a bit.
Moved the functions to monitor.c.
Renamed rdtgroup_assign_cntr_event() to resctrl_assign_cntr_event(). Refactored the resctrl_assign_cntr_event().
Added functionality to exit on the first error during assignment. Simplified mbm_cntr_free().
Removed the function mbm_cntr_assigned(). Will be using mbm_cntr_get() to figure out if the counter is assigned or not.
Updated commit message and code comments.

v10: Patch changed completely.
Counters are managed at the domain based on the discussion.
https://lore.kernel.org/lkml/CALPaoCj+zWq1vkHVbXYP0znJbe6Ke3PXPWjtri5AFgD9cQDCUg@mail.gmail.com/
Reset non-architectural MBM state.
Commit message update.

v9: Introduced new function resctrl_config_cntr to assign the counter, update
the bitmap and reset the architectural state.
Taken care of error handling(freeing the counter) when assignment fails. Moved mbm_cntr_assigned_to_domain here as it used in this patch. Minor text changes.

v8: Renamed rdtgroup_assign_cntr() to rdtgroup_assign_cntr_event().
Added the code to return the error if rdtgroup_assign_cntr_event fails. Moved definition of MBM_EVENT_ARRAY_INDEX to resctrl/internal.h. Updated typo in the comments.

v7: New patch. Moved all the FS code here.
Merged rdtgroup_assign_cntr and rdtgroup_alloc_cntr. Adde new #define MBM_EVENT_ARRAY_INDEX.

---
 fs/resctrl/internal.h |   3 +
 fs/resctrl/monitor.c  | 134 ++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 137 insertions(+)
fs/resctrl/internal.h +3 -0
index 0fae374559ba..ce4fcac91937 100644
@@ -377,6 +377,9 @@ bool closid_allocated(unsigned int closid);
 
 int resctrl_find_cleanest_closid(void);
 
+int resctrl_assign_cntr_event(struct rdt_resource *r, struct rdt_mon_domain *d,
+			      struct rdtgroup *rdtgrp, enum resctrl_event_id evtid);
+
 #ifdef CONFIG_RESCTRL_FS_PSEUDO_LOCK
 int rdtgroup_locksetup_enter(struct rdtgroup *rdtgrp);
 
fs/resctrl/monitor.c +134 -0

137 lines not on this page · raw message

Signature · 1 line
2.34.1
author reviewer
#57 lore raw

PATCH v13 14/27 x86/resctrl: Add the functionality to unassign MBM events

In reply to Babu Moger

The mbm_cntr_assign mode offers "num_mbm_cntrs" number of counters that can be assigned to an RMID, event pair and monitor the bandwidth as long as it is assigned. If all the counters are in use, the kernel will log the error message "Unable to allocate counter in domain" in /sys/fs/resctrl/info/last_cmd_status when a new assignment is requested.

To make space for a new assignment, users must unassign an already assigned counter and retry the assignment again.

Add the functionality to unassign and free the counters in the domain.

---
v13: Moved mbm_cntr_free() to this patch as it is used in here first.
Not required to pass evt_cfg to resctrl_unassign_cntr_event(). It is available via mbm_get_mon_event().
Resolved conflicts caused by the recent FS/ARCH code restructure. The monitor.c file has now been split between the FS and ARCH directories.

v12: Updated the commit text to make bit more clear.
Replaced several counters with "num_mbm_cntrs" counters.
Fixed typo in the subjest line.
Fixed the handling error on first failure.
Added domain id and event id on failure.
Added new parameter event configuration (evt_cfg) to provide the event from user space.

v11: Moved the functions to monitor.c.
Renamed rdtgroup_unassign_cntr_event() to resctrl_unassign_cntr_event(). Refactored the resctrl_unassign_cntr_event().
Updated commit message and code comments.

v10: Patch changed again.
Counters are managed at the domain based on the discussion.
https://lore.kernel.org/lkml/CALPaoCj+zWq1vkHVbXYP0znJbe6Ke3PXPWjtri5AFgD9cQDCUg@mail.gmail.com/
commit message update.

v9: Changes related to addition of new function resctrl_config_cntr().
The removed rdtgroup_mbm_cntr_is_assigned() as it was introduced already.
Text changes to take care comments.

v8: Renamed rdtgroup_mbm_cntr_is_assigned to mbm_cntr_assigned_to_domain
Added return error handling in resctrl_arch_config_cntr().

v7: Merged rdtgroup_unassign_cntr and rdtgroup_free_cntr functions.
Renamed rdtgroup_mbm_cntr_test() to rdtgroup_mbm_cntr_is_assigned(). Reworded the commit log little bit.

v6: Removed mbm_cntr_free from this patch.
Added counter test in all the domains and free if it is not assigned to any domains.

v5: Few name changes to match cntr_id.
Changed the function names to rdtgroup_unassign_cntr More comments on commit log.

v4: Added domain specific unassign feature.
Few name changes.

v3: Removed the static from the prototype of rdtgroup_unassign_abmc.
The function is not called directly from user anymore. These changes are related to global assignment interface.

v2: No changes.

---
 fs/resctrl/internal.h |  2 ++
 fs/resctrl/monitor.c  | 60 +++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 62 insertions(+)
fs/resctrl/internal.h +2 -0
index ce4fcac91937..64ddc107fcab 100644
@@ -379,6 +379,8 @@ int resctrl_find_cleanest_closid(void);
 
 int resctrl_assign_cntr_event(struct rdt_resource *r, struct rdt_mon_domain *d,
 			      struct rdtgroup *rdtgrp, enum resctrl_event_id evtid);
+int resctrl_unassign_cntr_event(struct rdt_resource *r, struct rdt_mon_domain *d,
+				struct rdtgroup *rdtgrp, enum resctrl_event_id evtid);
 
 #ifdef CONFIG_RESCTRL_FS_PSEUDO_LOCK
 int rdtgroup_locksetup_enter(struct rdtgroup *rdtgrp);
fs/resctrl/monitor.c +60 -0

69 lines not on this page · raw message

Signature · 1 line
2.34.1
author reviewer
#60 lore raw

PATCH v13 15/27 x86/resctrl: Report 'Unassigned' for MBM events in mbm_cntr_assign mode

In reply to Babu Moger

In mbm_cntr_assign mode, the hardware counter should be assigned to read the MBM events.

Report 'Unassigned' in case the user attempts to read the event without assigning a hardware counter.

Export resctrl_is_mbm_event() and mbm_cntr_get() to allow usage from other functions within fs/resctrl.

---
v13: Minor commitlog and user doc update.
Resolved conflicts caused by the recent FS/ARCH code restructure.
The monitor.c/rdtgroup.c files have been split between the FS and ARCH directories.

v12: Updated the documentation for more clarity.

v11: Domain can be NULL with SNC support so moved the unassign check in
rdtgroup_mondata_show().

v10: Moved the code to check the assign state inside mon_event_read().
Fixed few text comments.

v9: Used is_mbm_event() to check the event type.
Minor user documentation update.

v8: Used MBM_EVENT_ARRAY_INDEX to get the index for the MBM event.
Documentation update to make the text generic.

v7: Moved the documentation under "mon_data".
Updated the text little bit.

v6: Added more explaination in the resctrl.rst
Added checks to detect "Unassigned" before reading RMID.

v5: New patch.

---
 Documentation/filesystems/resctrl.rst |  8 ++++++++
 fs/resctrl/ctrlmondata.c              | 14 ++++++++++++++
 fs/resctrl/internal.h                 |  2 ++
 fs/resctrl/monitor.c                  |  4 ++--
 fs/resctrl/rdtgroup.c                 |  2 +-
 include/linux/resctrl.h               |  1 +
 6 files changed, 28 insertions(+), 3 deletions(-)
Documentation/filesystems/resctrl.rst +8 -0
index 2bfad43aac9c..5cf2d742f04c 100644
@@ -430,6 +430,14 @@ When monitoring is enabled all MON groups will also contain:
 	for the L3 cache they occupy). These are named "mon_sub_L3_YY"
 	where "YY" is the node number.
 
+	The mbm_cntr_assign mode offers "num_mbm_cntrs" number of counters
+	and allows users to assign a counter to mon_hw_id, event pair enabling
+	bandwidth monitoring for as long as the counter remains assigned.
+	The hardware will continue tracking the assigned mon_hw_id until
+	the user manually unassigns it, ensuring that counters are not reset
+	during this period. An MBM event returns 'Unassigned' when the event
+	does not have a hardware counter assigned.
+
 "mon_hw_id":
 	Available only with debug option. The identifier used by hardware
 	for the monitor group. On x86 this is the RMID.
fs/resctrl/ctrlmondata.c +14 -0
index 6ed2dfd4dbbd..f6b8ad24b0b5 100644
@@ -643,6 +643,18 @@ int rdtgroup_mondata_show(struct seq_file *m, void *arg)
 			goto out;
 		}
 		d = container_of(hdr, struct rdt_mon_domain, hdr);
+
+		/*
+		 * Report 'Unassigned' if mbm_cntr_assign mode is enabled and
+		 * counter is unassigned.
+		 */
+		if (resctrl_arch_mbm_cntr_assign_enabled(r) &&
+		    resctrl_is_mbm_event(evtid) &&
+		    (mbm_cntr_get(r, d, rdtgrp, evtid) < 0)) {
+			rr.err = -ENOENT;
+			goto checkresult;
+		}
+
 		mon_event_read(&rr, r, d, rdtgrp, &d->hdr.cpu_mask, evtid, false);
 	}
 
@@ -652,6 +664,8 @@ int rdtgroup_mondata_show(struct seq_file *m, void *arg)
 		seq_puts(m, "Error\n");
 	else if (rr.err == -EINVAL)
 		seq_puts(m, "Unavailable\n");
+	else if (rr.err == -ENOENT)
+		seq_puts(m, "Unassigned\n");
 	else
 		seq_printf(m, "%llu\n", rr.val);
 
fs/resctrl/internal.h +2 -0
index 64ddc107fcab..0dfd2efe68fc 100644
@@ -381,6 +381,8 @@ int resctrl_assign_cntr_event(struct rdt_resource *r, struct rdt_mon_domain *d,
 			      struct rdtgroup *rdtgrp, enum resctrl_event_id evtid);
 int resctrl_unassign_cntr_event(struct rdt_resource *r, struct rdt_mon_domain *d,
 				struct rdtgroup *rdtgrp, enum resctrl_event_id evtid);
+int mbm_cntr_get(struct rdt_resource *r, struct rdt_mon_domain *d,
+		 struct rdtgroup *rdtgrp, enum resctrl_event_id evtid);
 
 #ifdef CONFIG_RESCTRL_FS_PSEUDO_LOCK
 int rdtgroup_locksetup_enter(struct rdtgroup *rdtgrp);
fs/resctrl/monitor.c +2 -2
index fbc938bd3b23..c98a61bde179 100644
@@ -956,8 +956,8 @@ static void resctrl_config_cntr(struct rdt_resource *r, struct rdt_mon_domain *d
  * mbm_cntr_get() - Return the cntr_id for the matching evtid and rdtgrp in
  *		    cntr_cfg array.
  */
-static int mbm_cntr_get(struct rdt_resource *r, struct rdt_mon_domain *d,
-			struct rdtgroup *rdtgrp, enum resctrl_event_id evtid)
+int mbm_cntr_get(struct rdt_resource *r, struct rdt_mon_domain *d,
+		 struct rdtgroup *rdtgrp, enum resctrl_event_id evtid)
 {
 	int cntr_id;
 
fs/resctrl/rdtgroup.c +1 -1
index f192b2736a77..72317a5adee2 100644
@@ -127,7 +127,7 @@ static bool resctrl_is_mbm_enabled(void)
 		resctrl_arch_is_mbm_local_enabled());
 }
 
-static bool resctrl_is_mbm_event(int e)
+bool resctrl_is_mbm_event(int e)
 {
 	return (e >= QOS_L3_MBM_TOTAL_EVENT_ID &&
 		e <= QOS_L3_MBM_LOCAL_EVENT_ID);
include/linux/resctrl.h +1 -0
index 59a4fe60ab46..f78b6064230c 100644
@@ -441,6 +441,7 @@ static inline u32 resctrl_get_config_index(u32 closid,
 	}
 }
 
+bool resctrl_is_mbm_event(int e);
 bool resctrl_arch_get_cdp_enabled(enum resctrl_res_level l);
 int resctrl_arch_set_cdp_enabled(enum resctrl_res_level l, bool enable);
 
Signature · 1 line
2.34.1
author reviewer
#63 lore raw

PATCH v13 16/27 x86/resctrl: Pass entire struct rdtgroup rather than passing individual members

In reply to Babu Moger

The mbm_cntr_assign mode requires a cntr_id to read event data. The cntr_id is retrieved via mbm_cntr_get, which takes a struct rdtgroup as a parameter.

Passing the full rdtgroup also provides access to closid and rmid, both of which are necessary to read monitoring events.

Refactor the code to pass the entire struct rdtgroup instead of individual members in preparation for this requirement.

---
v13: New patch to pass the entire struct rdtgroup to __mon_event_count(),
mbm_update(), and related functions.

---
 fs/resctrl/monitor.c | 29 ++++++++++++++++-------------
 1 file changed, 16 insertions(+), 13 deletions(-)
fs/resctrl/monitor.c +16 -13
index c98a61bde179..a477be9cdb66 100644
@@ -357,9 +357,11 @@ static struct mbm_state *get_mbm_state(struct rdt_mon_domain *d, u32 closid,
 	}
 }
 
-static int __mon_event_count(u32 closid, u32 rmid, struct rmid_read *rr)
+static int __mon_event_count(struct rdtgroup *rdtgrp, struct rmid_read *rr)
 {
 	int cpu = smp_processor_id();
+	u32 closid = rdtgrp->closid;
+	u32 rmid = rdtgrp->mon.rmid;
 	struct rdt_mon_domain *d;
 	struct mbm_state *m;
 	int err, ret;
@@ -428,9 +430,11 @@ static int __mon_event_count(u32 closid, u32 rmid, struct rmid_read *rr)
  * __mon_event_count() is compared with the chunks value from the previous
  * invocation. This must be called once per second to maintain values in MBps.
  */
-static void mbm_bw_count(u32 closid, u32 rmid, struct rmid_read *rr)
+static void mbm_bw_count(struct rdtgroup *rdtgrp, struct rmid_read *rr)
 {
 	u64 cur_bw, bytes, cur_bytes;
+	u32 closid = rdtgrp->closid;
+	u32 rmid = rdtgrp->mon.rmid;
 	struct mbm_state *m;
 
 	m = get_mbm_state(rr->d, closid, rmid, rr->evtid);
@@ -459,7 +463,7 @@ void mon_event_count(void *info)
 
 	rdtgrp = rr->rgrp;
 
-	ret = __mon_event_count(rdtgrp->closid, rdtgrp->mon.rmid, rr);
+	ret = __mon_event_count(rdtgrp, rr);
 
 	/*
 	 * For Ctrl groups read data from child monitor groups and
@@ -470,8 +474,7 @@ void mon_event_count(void *info)
 
 	if (rdtgrp->type == RDTCTRL_GROUP) {
 		list_for_each_entry(entry, head, mon.crdtgrp_list) {
-			if (__mon_event_count(entry->closid, entry->mon.rmid,
-					      rr) == 0)
+			if (__mon_event_count(entry, rr) == 0)
 				ret = 0;
 		}
 	}
@@ -602,7 +605,7 @@ static void update_mba_bw(struct rdtgroup *rgrp, struct rdt_mon_domain *dom_mbm)
 }
 
 static void mbm_update_one_event(struct rdt_resource *r, struct rdt_mon_domain *d,
-				 u32 closid, u32 rmid, enum resctrl_event_id evtid)
+				 struct rdtgroup *rdtgrp, enum resctrl_event_id evtid)
 {
 	struct rmid_read rr = {0};
 
@@ -616,30 +619,30 @@ static void mbm_update_one_event(struct rdt_resource *r, struct rdt_mon_domain *
 		return;
 	}
 
-	__mon_event_count(closid, rmid, &rr);
+	__mon_event_count(rdtgrp, &rr);
 
 	/*
 	 * If the software controller is enabled, compute the
 	 * bandwidth for this event id.
 	 */
 	if (is_mba_sc(NULL))
-		mbm_bw_count(closid, rmid, &rr);
+		mbm_bw_count(rdtgrp, &rr);
 
 	resctrl_arch_mon_ctx_free(rr.r, rr.evtid, rr.arch_mon_ctx);
 }
 
 static void mbm_update(struct rdt_resource *r, struct rdt_mon_domain *d,
-		       u32 closid, u32 rmid)
+		       struct rdtgroup *rdtgrp)
 {
 	/*
 	 * This is protected from concurrent reads from user as both
 	 * the user and overflow handler hold the global mutex.
 	 */
 	if (resctrl_arch_is_mbm_total_enabled())
-		mbm_update_one_event(r, d, closid, rmid, QOS_L3_MBM_TOTAL_EVENT_ID);
+		mbm_update_one_event(r, d, rdtgrp, QOS_L3_MBM_TOTAL_EVENT_ID);
 
 	if (resctrl_arch_is_mbm_local_enabled())
-		mbm_update_one_event(r, d, closid, rmid, QOS_L3_MBM_LOCAL_EVENT_ID);
+		mbm_update_one_event(r, d, rdtgrp, QOS_L3_MBM_LOCAL_EVENT_ID);
 }
 
 /*
@@ -712,11 +715,11 @@ void mbm_handle_overflow(struct work_struct *work)
 	d = container_of(work, struct rdt_mon_domain, mbm_over.work);
 
 	list_for_each_entry(prgrp, &rdt_all_groups, rdtgroup_list) {
-		mbm_update(r, d, prgrp->closid, prgrp->mon.rmid);
+		mbm_update(r, d, prgrp);
 
 		head = &prgrp->mon.crdtgrp_list;
 		list_for_each_entry(crgrp, head, mon.crdtgrp_list)
-			mbm_update(r, d, crgrp->closid, crgrp->mon.rmid);
+			mbm_update(r, d, crgrp);
 
 		if (is_mba_sc(NULL))
 			update_mba_bw(prgrp, d);
Signature · 1 line
2.34.1
author reviewer
#66 lore raw

PATCH v13 17/27 x86/resctrl: Add the support for reading ABMC counters

In reply to Babu Moger

Software can read the assignable counters using the QM_EVTSEL and QM_CTR register pair.

QM_EVTSEL Register definition:
=======================================================
Bits Mnemonic Description
=======================================================
63:44 -- Reserved
43:32 RMID Resource Monitoring Identifier
31 ExtEvtID Extended Event Identifier
30:8 -- Reserved
7:0 EvtID Event Identifier
=======================================================

The contents of a specific counter can be read by setting the following fields in QM_EVTSEL.ExtendedEvtID = 1, QM_EVTSEL.EvtID = L3CacheABMC (=1) and setting [RMID] to the desired counter ID. Reading QM_CTR will then return the contents of the specified counter. The E bit will be set if the counter configuration was invalid, or if an invalid counter ID was set in the QM_EVTSEL[RMID] field.

Introduce __cntr_id_read_phys() to read the counter ID event data.

---
v13: Split the patch into 2. First one to handle the passing of rdtgroup structure to few
functions( __mon_event_count and mbm_update(). Second one to handle ABMC counter reading. Added new function __cntr_id_read_phys() to handle ABMC event reading. Updated kernel doc for resctrl_arch_reset_rmid() and resctrl_arch_rmid_read(). Resolved conflicts caused by the recent FS/ARCH code restructure.
The monitor.c file has now been split between the FS and ARCH directories.

v12: New patch to support extended event mode when ABMC is enabled.

---
 arch/x86/kernel/cpu/resctrl/internal.h |  6 +++
 arch/x86/kernel/cpu/resctrl/monitor.c  | 66 ++++++++++++++++++++++----
 fs/resctrl/monitor.c                   | 14 ++++--
 include/linux/resctrl.h                |  9 ++--
 4 files changed, 80 insertions(+), 15 deletions(-)
arch/x86/kernel/cpu/resctrl/internal.h +6 -0
index db6b0c28ee6b..3b0cdb5520c7 100644
@@ -40,6 +40,12 @@ struct arch_mbm_state {
 /* Setting bit 0 in L3_QOS_EXT_CFG enables the ABMC feature. */
 #define ABMC_ENABLE_BIT			0
 
+/*
+ * ABMC Qos Event Identifiers.
+ */
+#define ABMC_EXTENDED_EVT_ID		BIT(31)
+#define ABMC_EVT_ID			1
+
 /**
  * struct rdt_hw_ctrl_domain - Arch private attributes of a set of CPUs that share
  *			       a resource for a control function
arch/x86/kernel/cpu/resctrl/monitor.c +58 -8

96 lines not on this page · raw message

fs/resctrl/monitor.c +10 -4
index a477be9cdb66..72f3dfb5b903 100644
@@ -159,7 +159,11 @@ void __check_limbo(struct rdt_mon_domain *d, bool force_free)
 			break;
 
 		entry = __rmid_entry(idx);
-		if (resctrl_arch_rmid_read(r, d, entry->closid, entry->rmid,
+		/*
+		 * cntr_id is not relevant for QOS_L3_OCCUP_EVENT_ID.
+		 * Pass dummy value -1.
+		 */
+		if (resctrl_arch_rmid_read(r, d, entry->closid, entry->rmid, -1,
 					   QOS_L3_OCCUP_EVENT_ID, &val,
 					   arch_mon_ctx)) {
 			rmid_dirty = true;
@@ -359,6 +363,7 @@ static struct mbm_state *get_mbm_state(struct rdt_mon_domain *d, u32 closid,
 
 static int __mon_event_count(struct rdtgroup *rdtgrp, struct rmid_read *rr)
 {
+	int cntr_id = mbm_cntr_get(rr->r, rr->d, rdtgrp, rr->evtid);
 	int cpu = smp_processor_id();
 	u32 closid = rdtgrp->closid;
 	u32 rmid = rdtgrp->mon.rmid;
@@ -368,7 +373,7 @@ static int __mon_event_count(struct rdtgroup *rdtgrp, struct rmid_read *rr)
 	u64 tval = 0;
 
 	if (rr->first) {
-		resctrl_arch_reset_rmid(rr->r, rr->d, closid, rmid, rr->evtid);
+		resctrl_arch_reset_rmid(rr->r, rr->d, closid, rmid, cntr_id, rr->evtid);
 		m = get_mbm_state(rr->d, closid, rmid, rr->evtid);
 		if (m)
 			memset(m, 0, sizeof(struct mbm_state));
@@ -379,7 +384,7 @@ static int __mon_event_count(struct rdtgroup *rdtgrp, struct rmid_read *rr)
 		/* Reading a single domain, must be on a CPU in that domain. */
 		if (!cpumask_test_cpu(cpu, &rr->d->hdr.cpu_mask))
 			return -EINVAL;
-		rr->err = resctrl_arch_rmid_read(rr->r, rr->d, closid, rmid,
+		rr->err = resctrl_arch_rmid_read(rr->r, rr->d, closid, rmid, cntr_id,
 						 rr->evtid, &tval, rr->arch_mon_ctx);
 		if (rr->err)
 			return rr->err;
@@ -404,7 +409,8 @@ static int __mon_event_count(struct rdtgroup *rdtgrp, struct rmid_read *rr)
 	list_for_each_entry(d, &rr->r->mon_domains, hdr.list) {
 		if (d->ci->id != rr->ci->id)
 			continue;
-		err = resctrl_arch_rmid_read(rr->r, d, closid, rmid,
+		cntr_id = mbm_cntr_get(rr->r, d, rdtgrp, rr->evtid);
+		err = resctrl_arch_rmid_read(rr->r, d, closid, rmid, cntr_id,
 					     rr->evtid, &tval, rr->arch_mon_ctx);
 		if (!err) {
 			rr->val += tval;
include/linux/resctrl.h +6 -3
index f78b6064230c..cd24d1577e0a 100644
@@ -473,6 +473,7 @@ void resctrl_offline_cpu(unsigned int cpu);
  *			counter may match traffic of both @closid and @rmid, or @rmid
  *			only.
  * @rmid:		rmid of the counter to read.
+ * @cntr_id:		cntr_id to read MBM events with mbm_cntr_assign mode.
  * @eventid:		eventid to read, e.g. L3 occupancy.
  * @val:		result of the counter read in bytes.
  * @arch_mon_ctx:	An architecture specific value from
@@ -490,8 +491,9 @@ void resctrl_offline_cpu(unsigned int cpu);
  * 0 on success, or -EIO, -EINVAL etc on error.
  */
 int resctrl_arch_rmid_read(struct rdt_resource *r, struct rdt_mon_domain *d,
-			   u32 closid, u32 rmid, enum resctrl_event_id eventid,
-			   u64 *val, void *arch_mon_ctx);
+			   u32 closid, u32 rmid, int cntr_id,
+			   enum resctrl_event_id eventid, u64 *val,
+			   void *arch_mon_ctx);
 
 /**
  * resctrl_arch_rmid_read_context_check()  - warn about invalid contexts
@@ -532,12 +534,13 @@ struct rdt_domain_hdr *resctrl_find_domain(struct list_head *h, int id,
  * @closid:	closid that matches the rmid. Depending on the architecture, the
  *		counter may match traffic of both @closid and @rmid, or @rmid only.
  * @rmid:	The rmid whose counter values should be reset.
+ * @cntr_id:	The cntr_id to read MBM events with mbm_cntr_assign mode.
  * @eventid:	The eventid whose counter values should be reset.
  *
  * This can be called from any CPU.
  */
 void resctrl_arch_reset_rmid(struct rdt_resource *r, struct rdt_mon_domain *d,
-			     u32 closid, u32 rmid,
+			     u32 closid, u32 rmid, int cntr_id,
 			     enum resctrl_event_id eventid);
 
 /**
Signature · 1 line
2.34.1
author reviewer
#69 lore raw

PATCH v13 18/27 x86/resctrl: Add definitions for MBM event configuration

In reply to Babu Moger

The "mbm_cntr_assign" mode allows users to manually assign a hardware counter to a specific RMID and event pair. The events available for assignment are configurable.

By default, each resctrl group supports two MBM events: mbm_total_bytes and mbm_local_bytes. Each event corresponds to an MBM configuration that specifies the bandwidth sources tracked by the event.

Add definitions of supported bandwidth sources.

---
v13: Updated the changelog.
Removed the definitions from resctrl_types.h and moved to internal.h. Removed mbm_assign_config definition. Configurations will be part of mon_evt list.
Resolved conflicts caused by the recent FS/ARCH code restructure. The rdtgroup.c file has now been split between the FS and ARCH directories.

v12: New patch to support event configurations via new counter_configs
method.

---
 fs/resctrl/internal.h | 10 ++++++++++
 fs/resctrl/rdtgroup.c | 14 ++++++++++++++
 2 files changed, 24 insertions(+)
fs/resctrl/internal.h +10 -0
index 0dfd2efe68fc..019d00bf5adf 100644
@@ -203,6 +203,16 @@ struct rdtgroup {
 	struct pseudo_lock_region	*plr;
 };
 
+/**
+ * struct mbm_evt_value - Specific type of memory events.
+ * @evt_name:		Name of memory transaction type (read, write etc).
+ * @evt_val:		Value representing the memory transaction.
+ */
+struct mbm_evt_value {
+	char    evt_name[32];
+	u32     evt_val;
+};
+
 /* rdtgroup.flags */
 #define	RDT_DELETED		1
 
fs/resctrl/rdtgroup.c +14 -0
index 72317a5adee2..b109e91096b0 100644
@@ -75,6 +75,20 @@ static void rdtgroup_destroy_root(void);
 
 struct dentry *debugfs_resctrl;
 
+/* Number of memory transaction types that can be monitored */
+#define NUM_MBM_EVT_VALUES             7
+
+/* Decoded values for each type of memory events */
+struct mbm_evt_value mbm_evt_values[NUM_MBM_EVT_VALUES] = {
+	{"local_reads", READS_TO_LOCAL_MEM},
+	{"remote_reads", READS_TO_REMOTE_MEM},
+	{"local_non_temporal_writes", NON_TEMP_WRITE_TO_LOCAL_MEM},
+	{"remote_non_temporal_writes", NON_TEMP_WRITE_TO_REMOTE_MEM},
+	{"local_reads_slow_memory", READS_TO_LOCAL_S_MEM},
+	{"remote_reads_slow_memory", READS_TO_REMOTE_S_MEM},
+	{"dirty_victim_writes_all", DIRTY_VICTIMS_TO_ALL_MEM},
+};
+
 /*
  * Memory bandwidth monitoring event to use for the default CTRL_MON group
  * and each new CTRL_MON group created by the user.  Only relevant when
Signature · 1 line
2.34.1
author reviewer
#74 lore raw

PATCH v13 19/27 x86/resctrl: Add event configuration directory under info/L3_MON/

In reply to Babu Moger

Create the configuration directory and files for mbm_cntr_assign mode. These configurations will be used to assign MBM events in mbm_cntr_assign mode, with two default configurations created upon mounting.

Example:

$ cd /sys/fs/resctrl/
$ cat info/L3_MON/counter_configs/mbm_total_bytes/event_filter
  local_reads, remote_reads, local_non_temporal_writes,
  remote_non_temporal_writes, local_reads_slow_memory,
  remote_reads_slow_memory, dirty_victim_writes_all
$ cat info/L3_MON/counter_configs/mbm_local_bytes/event_filter
  local_reads, local_non_temporal_writes, local_reads_slow_memory

---
v13: Updated user doc (resctrl.rst).
Changed the name of the function resctrl_mkdir_info_configs to resctrl_mkdir_counter_configs().
Replaced seq_puts() with seq_putc() where applicable.
Removed RFTYPE_MON_CONFIG definition. Not required.
Changed the name of the flag RFTYPE_CONFIG to RFTYPE_ASSIGN_CONFIG. Reinette suggested RFTYPE_MBM_EVENT_CONFIG but RFTYPE_ASSIGN_CONFIG seemed shorter and pricise.
The configuration is created using evt_list.
Resolved conflicts caused by the recent FS/ARCH code restructure.
The monitor.c/rdtgroup.c files have been split between the FS and ARCH directories.

v12: New patch to hold the MBM event configurations for mbm_cntr_assign mode.

---
 Documentation/filesystems/resctrl.rst | 30 ++++++++++
 fs/resctrl/internal.h                 |  2 +
 fs/resctrl/monitor.c                  |  1 +
 fs/resctrl/rdtgroup.c                 | 80 +++++++++++++++++++++++++++
 4 files changed, 113 insertions(+)
Documentation/filesystems/resctrl.rst +30 -0
index 5cf2d742f04c..4eb9f007ba3d 100644
@@ -306,6 +306,36 @@ with the following files:
 	  # cat /sys/fs/resctrl/info/L3_MON/available_mbm_cntrs
 	  0=30;1=30
 
+"counter_configs":
+	When the "mbm_cntr_assign" mode is supported, a dedicated directory is created
+	under the "L3_MON" directory to store configuration files.
+
+	These files contain the list of configurable events. There are two default
+	configurations: mbm_local_bytes and mbm_total_bytes.
+
+	Following types of events are supported:
+
+	==== ========================= ============================================================
+	Bits Name   		         Description
+	==== ========================= ============================================================
+	6    dirty_victim_writes_all     Dirty Victims from the QOS domain to all types of memory
+	5    remote_reads_slow_memory    Reads to slow memory in the non-local NUMA domain
+	4    local_reads_slow_memory     Reads to slow memory in the local NUMA domain
+	3    remote_non_temporal_writes  Non-temporal writes to non-local NUMA domain
+	2    local_non_temporal_writes   Non-temporal writes to local NUMA domain
+	1    remote_reads                Reads to memory in the non-local NUMA domain
+	0    local_reads                 Reads to memory in the local NUMA domain
+	==== ========================= ==========================================================
+
+	For example::
+
+	  # cat /sys/fs/resctrl/info/L3_MON/counter_configs/mbm_total_bytes/event_filter
+	  local_reads, remote_reads, local_non_temporal_writes, remote_non_temporal_writes,
+	  local_reads_slow_memory, remote_reads_slow_memory, dirty_victim_writes_all
+
+	  # cat /sys/fs/resctrl/info/L3_MON/counter_configs/mbm_local_bytes/event_filter
+	  local_reads, local_non_temporal_writes, local_reads_slow_memory
+
 "max_threshold_occupancy":
 		Read/write file provides the largest value (in
 		bytes) at which a previously used LLC_occupancy
fs/resctrl/internal.h +2 -0
index 019d00bf5adf..446cc9cc61df 100644
@@ -238,6 +238,8 @@ struct mbm_evt_value {
 
 #define RFTYPE_DEBUG			BIT(10)
 
+#define RFTYPE_ASSIGN_CONFIG		BIT(11)
+
 #define RFTYPE_CTRL_INFO		(RFTYPE_INFO | RFTYPE_CTRL)
 
 #define RFTYPE_MON_INFO			(RFTYPE_INFO | RFTYPE_MON)
fs/resctrl/monitor.c +1 -0
index 72f3dfb5b903..1f72249a5c93 100644
@@ -932,6 +932,7 @@ int resctrl_mon_resource_init(void)
 					 RFTYPE_MON_INFO | RFTYPE_RES_CACHE);
 		resctrl_file_fflags_init("available_mbm_cntrs",
 					 RFTYPE_MON_INFO | RFTYPE_RES_CACHE);
+		resctrl_file_fflags_init("event_filter", RFTYPE_ASSIGN_CONFIG);
 	}
 
 	return 0;
fs/resctrl/rdtgroup.c +80 -0

104 lines not on this page · raw message

Signature · 1 line
2.34.1
author reviewer
#77 lore raw

PATCH v13 20/27 x86/resctrl: Provide interface to update the event configurations

In reply to Babu Moger

Users can modify the event configuration by writing to the event_filter interface file. The event configurations for mbm_cntr_assign mode are located in /sys/fs/resctrl/info/event_configs/.

Update the assignments of all groups when the event configuration is modified.

Example:

$ cd /sys/fs/resctrl/
$ cat info/L3_MON/counter_configs/mbm_local_bytes/event_filter
  local_reads,local_non_temporal_writes,local_reads_slow_memory
$ echo "local_reads,local_non_temporal_writes" >
  info/L3_MON/counter_configs/mbm_total_bytes/event_filter
$ cat info/L3_MON/counter_configs/mbm_total_bytes/event_filter
  local_reads,local_non_temporal_writes

---
v13: Updated changelog for imperative mode.
Added function description in the prototype.
Updated the user doc resctrl.rst to address few feedback.
Resolved conflicts caused by the recent FS/ARCH code restructure.
The rdtgroup.c/monitor.c file has now been split between the FS and ARCH directories.

v12: New patch to modify event configurations.

---
 Documentation/filesystems/resctrl.rst |  12 +++
 fs/resctrl/rdtgroup.c                 | 120 +++++++++++++++++++++++++-
 2 files changed, 131 insertions(+), 1 deletion(-)
Documentation/filesystems/resctrl.rst +12 -0
index 4eb9f007ba3d..9923276826db 100644
@@ -336,6 +336,18 @@ with the following files:
 	  # cat /sys/fs/resctrl/info/L3_MON/counter_configs/mbm_local_bytes/event_filter
 	  local_reads, local_non_temporal_writes, local_reads_slow_memory
 
+	Modify the event configuration by writing to the "event_filter" file within the
+	configuration directory. The read/write event_filter file contains the configuration
+	of the event that reflects which memory transactions are counted by it.
+
+	For example::
+
+	  # echo "local_reads, local_non_temporal_writes" >
+	    /sys/fs/resctrl/info/L3_MON/counter_configs/mbm_total_bytes/event_filter
+
+	  # cat /sys/fs/resctrl/info/L3_MON/counter_configs/mbm_total_bytes/event_filter
+	   local_reads, local_non_temporal_writes
+
 "max_threshold_occupancy":
 		Read/write file provides the largest value (in
 		bytes) at which a previously used LLC_occupancy
fs/resctrl/rdtgroup.c +119 -1

134 lines not on this page · raw message

Signature · 1 line
2.34.1
author reviewer
#80 lore raw

PATCH v13 21/27 x86/resctrl: Introduce mbm_assign_on_mkdir to configure assignments

In reply to Babu Moger

The mbm_cntr_assign mode provides an option to the user to assign a counter to an RMID, event pair and monitor the bandwidth as long as the counter is assigned.

Introduce a configuration option to automatically assign counter IDs when a resctrl group is created, provided the counters are available. By default, this option is enabled at boot.

---
v13: Added Suggested-by tag.
Resolved conflicts caused by the recent FS/ARCH code restructure.
The rdtgroup.c/monitor.c file has now been split between the FS and ARCH directories.

v12: New patch. Added after the discussion on the list.
https://lore.kernel.org/lkml/CALPaoCh8siZKjL_3yvOYGL4cF_n_38KpUFgHVGbQ86nD+Q2_SA@mail.gmail.com/

---
 Documentation/filesystems/resctrl.rst | 10 ++++++
 fs/resctrl/monitor.c                  |  2 ++
 fs/resctrl/rdtgroup.c                 | 44 +++++++++++++++++++++++++--
 include/linux/resctrl.h               |  2 ++
 4 files changed, 56 insertions(+), 2 deletions(-)
Documentation/filesystems/resctrl.rst +10 -0
index 9923276826db..356f1f918a86 100644
@@ -348,6 +348,16 @@ with the following files:
 	  # cat /sys/fs/resctrl/info/L3_MON/counter_configs/mbm_total_bytes/event_filter
 	   local_reads, local_non_temporal_writes
 
+"mbm_assign_on_mkdir":
+	Automatically assign the monitoring counters on resctrl group creation
+	if the counters are available. It is enabled by default on boot and users
+	can disable by writing to the interface.
+	::
+
+	  # echo 0 > /sys/fs/resctrl/info/L3_MON/mbm_assign_on_mkdir
+	  # cat /sys/fs/resctrl/info/L3_MON/mbm_assign_on_mkdir
+	  0
+
 "max_threshold_occupancy":
 		Read/write file provides the largest value (in
 		bytes) at which a previously used LLC_occupancy
fs/resctrl/monitor.c +2 -0
index 1f72249a5c93..5f6c4b662f3b 100644
@@ -933,6 +933,8 @@ int resctrl_mon_resource_init(void)
 		resctrl_file_fflags_init("available_mbm_cntrs",
 					 RFTYPE_MON_INFO | RFTYPE_RES_CACHE);
 		resctrl_file_fflags_init("event_filter", RFTYPE_ASSIGN_CONFIG);
+		resctrl_file_fflags_init("mbm_assign_on_mkdir", RFTYPE_MON_INFO |
+					 RFTYPE_RES_CACHE);
 	}
 
 	return 0;
fs/resctrl/rdtgroup.c +42 -2

62 lines not on this page · raw message

include/linux/resctrl.h +2 -0
index cd24d1577e0a..d6435abdde7b 100644
@@ -278,6 +278,7 @@ enum resctrl_schema_fmt {
  *			monitoring events can be configured.
  * @num_mbm_cntrs:	Number of assignable monitoring counters
  * @mbm_cntr_assignable:Is system capable of supporting monitor assignment?
+ * @mbm_assign_on_mkdir:Auto enable monitor assignment on mkdir?
  * @evt_list:		List of monitoring events
  */
 struct resctrl_mon {
@@ -285,6 +286,7 @@ struct resctrl_mon {
 	unsigned int		mbm_cfg_mask;
 	int			num_mbm_cntrs;
 	bool			mbm_cntr_assignable;
+	bool			mbm_assign_on_mkdir;
 	struct list_head	evt_list;
 };
 
Signature · 1 line
2.34.1
author reviewer
#85 lore raw

PATCH v13 22/27 x86/resctrl: Auto assign/unassign counters when mbm_cntr_assign is enabled

In reply to Babu Moger

Automatically assign or unassign counters when a resctrl group is created or deleted. By default, each group requires two counters: one for the MBM total event and one for the MBM local event.

The mbm_cntr_assign mode offers "num_mbm_cntrs" number of counters that can be assigned to an RMID, event pair and monitor the bandwidth as long as it is assigned. If these counters are exhausted, the kernel will log the error message "Unable to allocate counter in domain" in /sys/fs/resctrl/info/last_cmd_status when a new group is created.

However, the creation of a group should not fail due to assignment failures. Users have the flexibility to modify the assignments at a later time.

---
v13: Changes due to calling of resctrl_assign_cntr_event() and resctrl_unassign_cntr_event().
It only takes evtid. evt_cfg is not required anymore.
Resolved conflicts caused by the recent FS/ARCH code restructure.
The monitor.c/rdtgroup.c files have been split between the FS and ARCH directories.

v12: Removed mbm_cntr_reset() as it is not required while removing the group.
Update the commit text.
Added r->mon_capable check in rdtgroup_assign_cntrs() and rdtgroup_unassign_cntrs.

v11: Moved mbm_cntr_reset() to monitor.c.
Added code reset non-architectural state in mbm_cntr_reset(). Added missing rdtgroup_unassign_cntrs() calls on failure path.

v10: Assigned the counter before exposing the event files.
Moved the call rdtgroup_assign_cntrs() inside mkdir_rdt_prepare_rmid_alloc(). This is called both CNTR_MON and MON group creation.
Call mbm_cntr_reset() when unmounted to clear all the assignments. Taken care of few other feedback comments.

v9: Changed rdtgroup_assign_cntrs() and rdtgroup_unassign_cntrs() to return void.
Updated couple of rdtgroup_unassign_cntrs() calls properly. Updated function comments.

v8: Renamed rdtgroup_assign_grp to rdtgroup_assign_cntrs.
Renamed rdtgroup_unassign_grp to rdtgroup_unassign_cntrs. Fixed the problem with unassigning the child MON groups of CTRL_MON group.

v7: Reworded the commit message.
Removed the reference of ABMC with mbm_cntr_assign. Renamed the function rdtgroup_assign_cntrs to rdtgroup_assign_grp.

v6: Removed the redundant comments on all the calls of
rdtgroup_assign_cntrs. Updated the commit message.
Dropped printing error message on every call of rdtgroup_assign_cntrs.

v5: Removed the code to enable/disable ABMC during the mount.
That will be another patch.
Added arch callers to get the arch specific data. Renamed fuctions to match the other abmc function. Added code comments for assignment failures.

v4: Few name changes based on the upstream discussion.
Commit message update.

v3: This is a new patch. Patch addresses the upstream comment to enable
ABMC feature by default if the feature is available.

---
 arch/x86/kernel/cpu/resctrl/monitor.c |  1 +
 fs/resctrl/rdtgroup.c                 | 65 ++++++++++++++++++++++++++-
 2 files changed, 64 insertions(+), 2 deletions(-)
arch/x86/kernel/cpu/resctrl/monitor.c +1 -0
index 36a03dae6d8e..c3e15f4de0b4 100644
@@ -435,6 +435,7 @@ int __init rdt_get_mon_l3_config(struct rdt_resource *r)
 		r->mon.mbm_cntr_assignable = true;
 		cpuid_count(0x80000020, 5, &eax, &ebx, &ecx, &edx);
 		r->mon.num_mbm_cntrs = (ebx & GENMASK(15, 0)) + 1;
+		r->mon.mbm_assign_on_mkdir = true;
 	}
 
 	r->mon_capable = true;
fs/resctrl/rdtgroup.c +63 -2

130 lines not on this page · raw message

Signature · 1 line
2.34.1
author reviewer
#86 lore raw

PATCH v13 23/27 x86/resctrl: Introduce mbm_L3_assignments to list assignments in a group

In reply to Babu Moger

Introduce the interface to display the assignment states for each group when mbm_cntr_assign mode is enabled.

The list is displayed in the following format:
<Event configuration>:<Domain id>=<Assignment type>

Event configuration: A valid event configuration listed in the /sys/fs/resctrl/info/L3_MON/counter_configs directory.

Domain ID: A valid domain ID number.

The assignment type can be one of the following:

_ : No event configuration assigned

e : Event configuration assigned in exclusive mode

Example:
$cd /sys/fs/resctrl
$cat mbm_L3_assignments
mbm_total_bytes:0=e;1=e
mbm_local_bytes:0=e;1=e

---
v13: Changelog update.
Few changes in mbm_L3_assignments_show() after moving the event config to evt_list. Resolved conflicts caused by the recent FS/ARCH code restructure.
The rdtgroup.c/monitor.c files have been split between the FS and ARCH directories.

v12: New patch:
Assignment interface moved inside the group based the discussion
https://lore.kernel.org/lkml/CALPaoCiii0vXOF06mfV=kVLBzhfNo0SFqt4kQGwGSGVUqvr2Dg@mail.gmail.com/#t

---
 Documentation/filesystems/resctrl.rst | 28 +++++++++++++++
 fs/resctrl/monitor.c                  |  1 +
 fs/resctrl/rdtgroup.c                 | 52 +++++++++++++++++++++++++++
 3 files changed, 81 insertions(+)
Documentation/filesystems/resctrl.rst +28 -0
index 356f1f918a86..2350c1f21f4e 100644
@@ -504,6 +504,34 @@ When the "mba_MBps" mount option is used all CTRL_MON groups will also contain:
 	/sys/fs/resctrl/info/L3_MON/mon_features changes the input
 	event.
 
+"mbm_L3_assignments":
+	This interface file is created when the mbm_cntr_assign mode is supported
+	and shows the assignment status for each group.
+
+	The assignment list is displayed in the following format:
+
+	<Event configuration>:<Domain id>=<Assignment type>
+
+	Event configuration: A valid event configuration listed in the
+	/sys/fs/resctrl/info/L3_MON/counter_configs directory.
+
+	Domain ID: A valid domain ID number.
+
+	Assignment types:
+
+	_ : No event configuration assigned
+
+	e : Event configuration assigned in exclusive mode
+
+	Example:
+	To list the assignment states for the default group.
+	::
+
+	  # cd /sys/fs/resctrl
+	  # cat mbm_L3_assignments
+	    mbm_total_bytes:0=e;1=e
+	    mbm_local_bytes:0=e;1=e
+
 Resource allocation rules
 -------------------------
 
fs/resctrl/monitor.c +1 -0
index 5f6c4b662f3b..b982540ce4e3 100644
@@ -935,6 +935,7 @@ int resctrl_mon_resource_init(void)
 		resctrl_file_fflags_init("event_filter", RFTYPE_ASSIGN_CONFIG);
 		resctrl_file_fflags_init("mbm_assign_on_mkdir", RFTYPE_MON_INFO |
 					 RFTYPE_RES_CACHE);
+		resctrl_file_fflags_init("mbm_L3_assignments", RFTYPE_MON_BASE);
 	}
 
 	return 0;
fs/resctrl/rdtgroup.c +52 -0

64 lines not on this page · raw message

Signature · 1 line
2.34.1
author reviewer
#89 lore raw

PATCH v13 24/27 x86/resctrl: Introduce the interface to modify assignments in a group

In reply to Babu Moger

Introduce an interface to modify assignments within a group.

Modifications follow this format:
<Event configuration>:<Domain id>=<Assignment type>

The assignment type can be one of the following:

_ : No event configuration assigned

e : Event configuration assigned in exclusive mode

Domain id can be any valid domain ID number or '*' to update all the domains.

Example:
$cd /sys/fs/resctrl
$cat mbm_L3_assignments
mbm_total_bytes:0=e;1=e
mbm_local_bytes:0=e;1=e

To unassign the configuration of mbm_total_bytes on domain 0:

$echo "mbm_total_bytes:0=_" > mbm_L3_assignments
$cat mbm_L3_assignments
mbm_total_bytes:0=_;1=e
mbm_local_bytes:0=e;1=e

To unassign the mbm_total_bytes configuration on all domains:

$echo "mbm_total_bytes:*=_" > mbm_L3_assignments
$cat mbm_L3_assignments
mbm_total_bytes:0=_;1=_
mbm_local_bytes:0=e;1=e

---
v13: Few changes in mbm_L3_assignments_write() after moving the event config to evt_list.
Resolved conflicts caused by the recent FS/ARCH code restructure.

v12: New patch:
Assignment interface moved inside the group based the discussion
https://lore.kernel.org/lkml/CALPaoCiii0vXOF06mfV=kVLBzhfNo0SFqt4kQGwGSGVUqvr2Dg@mail.gmail.com/#t

---
 Documentation/filesystems/resctrl.rst |  29 ++++-
 fs/resctrl/internal.h                 |   9 ++
 fs/resctrl/rdtgroup.c                 | 165 +++++++++++++++++++++++++-
 3 files changed, 201 insertions(+), 2 deletions(-)
Documentation/filesystems/resctrl.rst +28 -1
index 2350c1f21f4e..d779554a2f91 100644
@@ -515,7 +515,7 @@ When the "mba_MBps" mount option is used all CTRL_MON groups will also contain:
 	Event configuration: A valid event configuration listed in the
 	/sys/fs/resctrl/info/L3_MON/counter_configs directory.
 
-	Domain ID: A valid domain ID number.
+	Domain ID: A valid domain ID number or '*' to update all the domains.
 
 	Assignment types:
 
@@ -532,6 +532,33 @@ When the "mba_MBps" mount option is used all CTRL_MON groups will also contain:
 	    mbm_total_bytes:0=e;1=e
 	    mbm_local_bytes:0=e;1=e
 
+	Modify the assignment states by writing to the interface file.
+
+	Example:
+	To unassign the configuration of mbm_total_bytes on domain 0:
+	::
+
+	 # echo "mbm_total_bytes:0=_" > mbm_L3_assignments
+	 # cat mbm_L3_assignments
+	 mbm_total_bytes:0=_;1=e
+	 mbm_local_bytes:0=e;1=e
+
+	To unassign the mbm_total_bytes configuration on all domains:
+	::
+
+	 # echo "mbm_total_bytes:*=_" > mbm_L3_assignments
+	 # cat mbm_L3_assignments
+	 mbm_total_bytes:0=_;1=_
+	 mbm_local_bytes:0=e;1=e
+
+	To assign the mbm_total_bytes configuration on all domains in exclusive mode:
+	::
+
+	 # echo "mbm_total_bytes:*=e" > mbm_L3_assignments
+	 # cat mbm_L3_assignments
+	 mbm_total_bytes:0=e;1=e
+	 mbm_local_bytes:0=e;1=e
+
 Resource allocation rules
 -------------------------
 
fs/resctrl/internal.h +9 -0
index 446cc9cc61df..a6069a5dfd49 100644
@@ -51,6 +51,15 @@ static inline struct rdt_fs_context *rdt_fc2context(struct fs_context *fc)
 	return container_of(kfc, struct rdt_fs_context, kfc);
 }
 
+/*
+ * Assignment types for mbm_cntr_assign mode
+ */
+enum {
+	ASSIGN_NONE		= 0,
+	ASSIGN_EXCLUSIVE,
+	ASSIGN_INVALID,
+};
+
 /**
  * struct mon_evt - Entry in the event list of a resource
  * @evtid:		event id
fs/resctrl/rdtgroup.c +164 -1

179 lines not on this page · raw message

Signature · 1 line
2.34.1
author reviewer
#92 lore raw

PATCH v13 25/27 x86/resctrl: Hide the BMEC related files when mbm_cnt_assign is enabled

In reply to Babu Moger

BMEC (Bandwidth Monitoring Event Configuration) and mbm_cntr_assign cannot be used simultaneously.

When mbm_cntr_assign is active, suppress visibility of BMEC-related files to prevent confusion.

The files /sys/fs/resctrl/info/L3_MON/mbm_total_bytes_config and /sys/fs/resctrl/info/L3_MON/mbm_local_bytes_config will not be visible when mbm_cntr_assign mode is enabled.

---
v13: New patch to hide BMEC related files.

---
 fs/resctrl/rdtgroup.c | 31 +++++++++++++++++++++++++++++++
 1 file changed, 31 insertions(+)
fs/resctrl/rdtgroup.c +31 -0
index ea1782723f81..d6bf2a50a105 100644
@@ -1815,6 +1815,33 @@ static ssize_t mbm_local_bytes_config_write(struct kernfs_open_file *of,
 	return ret ?: nbytes;
 }
 
+static void resctrl_bmec_files_show(struct rdt_resource *r, bool show)
+{
+	struct kernfs_node *kn_config, *l3_mon_kn;
+	char name[32];
+
+	sprintf(name, "%s_MON", r->name);
+	l3_mon_kn = kernfs_find_and_get(kn_info, name);
+	if (!l3_mon_kn)
+		return;
+
+	kn_config = kernfs_find_and_get(l3_mon_kn, "mbm_total_bytes_config");
+	if (kn_config) {
+		kernfs_get(kn_config);
+		kernfs_show(kn_config, show);
+		kernfs_put(kn_config);
+	}
+
+	kn_config = kernfs_find_and_get(l3_mon_kn, "mbm_local_bytes_config");
+	if (kn_config) {
+		kernfs_get(kn_config);
+		kernfs_show(kn_config, show);
+		kernfs_put(kn_config);
+	}
+
+	kernfs_put(l3_mon_kn);
+}
+
 static int resctrl_mbm_assign_mode_show(struct kernfs_open_file *of,
 					struct seq_file *s, void *v)
 {
@@ -2815,6 +2842,10 @@ static int rdtgroup_create_info_dir(struct kernfs_node *parent_kn)
 			ret = resctrl_mkdir_counter_configs(r, name);
 			if (ret)
 				goto out_destroy;
+
+			/* Hide BMEC related files if mbm_cntr_assign is enabled */
+			if (resctrl_arch_mbm_cntr_assign_enabled(r))
+				resctrl_bmec_files_show(r, false);
 		}
 	}
 
Signature · 1 line
2.34.1
author reviewer
#93 lore raw

PATCH v13 26/27 x86/resctrl: Introduce the interface to switch between monitor modes

In reply to Babu Moger

Resctrl subsystem can support two monitoring modes, "mbm_cntr_assign" or "default". In mbm_cntr_assign, monitoring event can only accumulate data while it is backed by a hardware counter. In "default" mode, resctrl assumes there is a hardware counter for each event within every CTRL_MON and MON group.

Introduce interface to switch between mbm_cntr_assign and default modes.

$ cat /sys/fs/resctrl/info/L3_MON/mbm_assign_mode
[mbm_cntr_assign]
default

To enable the "mbm_cntr_assign" monitoring mode:

$ echo "mbm_cntr_assign" > /sys/fs/resctrl/info/L3_MON/mbm_assign_mode

To enable the "default" monitoring mode:

$ echo "default" > /sys/fs/resctrl/info/L3_MON/mbm_assign_mode

MBM event counters are automatically reset as part of changing the mode. Clear both architectural and non-architectural event states to prevent overflow conditions during the next event read.

---
v13: Resolved the conflicts due to FS/ARCH restructure.
Introduced the new resctrl_init_evt_configuration() to initialize the event modes and configuration values.
Added the call to resctrl_bmec_files_show() hide/show BMEC related files.

v12: Fixed the documentation for a consistency.
Introduced mbm_cntr_free_all() and resctrl_reset_rmid_all() to clear counters and non-architectural states when monitor mode is changed.
https://lore.kernel.org/lkml/[email protected]/

v11: Changed the name of the function rdtgroup_mbm_assign_mode_write() to
resctrl_mbm_assign_mode_write().
Rewrote the commit message with context.
Added few more details in resctrl.rst about mbm_cntr_assign mode. Re-arranged the text in resctrl.rst file.

v10: The call mbm_cntr_reset() has been moved to earlier patch.
Minor documentation update.

v9: Fixed extra spaces in user documentation.
Fixed problem changing the mode to mbm_cntr_assign mode when it is not supported. Added extra checks to detect if systems supports it. Used the rdtgroup_cntr_id_init to initialize cntr_id.

v8: Reset the internal counters after mbm_cntr_assign mode is changed.
Renamed rdtgroup_mbm_cntr_reset() to mbm_cntr_reset() Updated the documentation to make text generic.

v7: Changed the interface name to mbm_assign_mode.
Removed the references of ABMC.
Added the changes to reset global and domain bitmaps. Added the changes to reset rmid.

v6: Changed the mode name to mbm_cntr_assign.
Moved all the FS related code here.
Added changes to reset mbm_cntr_map and resctrl group counters.

v5: Change log and mode description text correction.

v4: Minor commit text changes. Keep the default to ABMC when supported.
Fixed comments to reflect changed interface "mbm_mode".

v3: New patch to address the review comments from upstream.

---
 Documentation/filesystems/resctrl.rst | 25 ++++++++++-
 fs/resctrl/internal.h                 |  3 ++
 fs/resctrl/monitor.c                  | 53 +++++++++++++++++++---
 fs/resctrl/rdtgroup.c                 | 65 ++++++++++++++++++++++++++-
 4 files changed, 138 insertions(+), 8 deletions(-)
Documentation/filesystems/resctrl.rst +24 -1
index d779554a2f91..7c304821ce93 100644
@@ -259,7 +259,10 @@ with the following files:
 
 "mbm_assign_mode":
 	Reports the list of monitoring modes supported. The enclosed brackets
-	indicate which mode is enabled.
+	indicate which mode is enabled. The MBM events (mbm_total_bytes and/or
+	mbm_local_bytes) associated with counters may reset when "mbm_assign_mode"
+	is changed.
+
 	::
 
 	  # cat /sys/fs/resctrl/info/L3_MON/mbm_assign_mode
@@ -275,6 +278,16 @@ with the following files:
 	"num_mbm_cntrs" file. Changing the mode may cause all counters on the
 	resource to reset.
 
+	Moving to mbm_cntr_assign mode require users to assign the counters to
+	the events. Otherwise, the MBM event counters will return 'Unassigned'
+	when read.
+
+	The mode is beneficial for AMD platforms that support more CTRL_MON
+	and MON groups than available hardware counters. By default, this
+	feature is enabled on AMD platforms with the ABMC (Assignable Bandwidth
+	Monitoring Counters) capability, ensuring counters remain assigned even
+	when the corresponding RMID is not actively used by any processor.
+
 	"default":
 
 	In default mode, resctrl assumes there is a hardware counter for each
@@ -284,6 +297,16 @@ with the following files:
 	counters. This can result in misleading values or display "Unavailable"
 	if no counter is assigned to the event.
 
+	* To enable "mbm_cntr_assign" monitoring mode:
+	  ::
+
+	    # echo "mbm_cntr_assign" > /sys/fs/resctrl/info/L3_MON/mbm_assign_mode
+
+	* To enable "default" monitoring mode:
+	  ::
+
+	    # echo "default" > /sys/fs/resctrl/info/L3_MON/mbm_assign_mode
+
 "num_mbm_cntrs":
 	The maximum number of monitoring counters (total of available and assigned
 	counters) in each domain when the system supports mbm_cntr_assign mode.
fs/resctrl/internal.h +3 -0
index a6069a5dfd49..d5edb28a8df7 100644
@@ -404,6 +404,9 @@ int resctrl_unassign_cntr_event(struct rdt_resource *r, struct rdt_mon_domain *d
 				struct rdtgroup *rdtgrp, enum resctrl_event_id evtid);
 int mbm_cntr_get(struct rdt_resource *r, struct rdt_mon_domain *d,
 		 struct rdtgroup *rdtgrp, enum resctrl_event_id evtid);
+void resctrl_reset_rmid_all(struct rdt_resource *r, struct rdt_mon_domain *d);
+void mbm_cntr_free_all(struct rdt_resource *r, struct rdt_mon_domain *d);
+void resctrl_init_evt_configuration(struct rdt_resource *r, bool enable);
 
 #ifdef CONFIG_RESCTRL_FS_PSEUDO_LOCK
 int rdtgroup_locksetup_enter(struct rdtgroup *rdtgrp);
fs/resctrl/monitor.c +47 -6

81 lines not on this page · raw message

fs/resctrl/rdtgroup.c +64 -1

79 lines not on this page · raw message

Signature · 1 line
2.34.1
author reviewer
#94 lore raw

PATCH v13 27/27 x86/resctrl: Configure mbm_cntr_assign mode if supported

In reply to Babu Moger

Configure mbm_cntr_assign mode on AMD platforms. On AMD platforms, it is recommended to use the mbm_cntr_assign mode, if supported, to prevent the hardware from resetting counters between reads. This can result in misleading values or display "Unavailable" if no counter is assigned to the event.

The mbm_cntr_assign mode, referred to as ABMC (Assignable Bandwidth Monitoring Counters) on AMD, is enabled by default when supported by the system.

Update ABMC across all logical processors within the resctrl domain to ensure proper functionality.

---
v13 : Added the call resctrl_init_evt_configuration() to setup the event
configuration during init.
Resolved conflicts caused by the recent FS/ARCH code restructure.

v12: Moved the resctrl_arch_mbm_cntr_assign_set_one to domain_add_cpu_mon().
Updated the commit log.

v11: Commit text in imperative tone. Added few more details.
Moved resctrl_arch_mbm_cntr_assign_set_one() to monitor.c.

v10: Commit text in imperative tone.

v9: Minor code change due to merge. Actual code did not change.

v8: Renamed resctrl_arch_mbm_cntr_assign_configure to
resctrl_arch_mbm_cntr_assign_set_one.
Adde r->mon_capable check.
Commit message update.

v7: Introduced resctrl_arch_mbm_cntr_assign_configure() to configure.
Moved the default settings to rdt_get_mon_l3_config(). It should be done before the hotplug handler is called. It cannot be done at rdtgroup_init().

v6: Keeping the default enablement in arch init code for now.
This may need some discussion.
Renamed resctrl_arch_configure_abmc to resctrl_arch_mbm_cntr_assign_configure.

v5: New patch to enable ABMC by default.

---
 arch/x86/kernel/cpu/resctrl/core.c     | 7 +++++++
 arch/x86/kernel/cpu/resctrl/internal.h | 1 +
 arch/x86/kernel/cpu/resctrl/monitor.c  | 8 ++++++++
 3 files changed, 16 insertions(+)
arch/x86/kernel/cpu/resctrl/core.c +7 -0
index 6859566398d6..b59f5db96016 100644
@@ -514,6 +514,9 @@ static void domain_add_cpu_mon(int cpu, struct rdt_resource *r)
 		d = container_of(hdr, struct rdt_mon_domain, hdr);
 
 		cpumask_set_cpu(cpu, &d->hdr.cpu_mask);
+		/* Update the mbm_cntr_assign state for the CPU if supported */
+		if (r->mon.mbm_cntr_assignable)
+			resctrl_arch_mbm_cntr_assign_set_one(r);
 		return;
 	}
 
@@ -532,6 +535,10 @@ static void domain_add_cpu_mon(int cpu, struct rdt_resource *r)
 	}
 	cpumask_set_cpu(cpu, &d->hdr.cpu_mask);
 
+	/* Update the mbm_cntr_assign state for the CPU if supported */
+	if (r->mon.mbm_cntr_assignable)
+		resctrl_arch_mbm_cntr_assign_set_one(r);
+
 	arch_mon_domain_online(r, d);
 
 	if (arch_domain_mbm_alloc(r->mon.num_rmid, hw_dom)) {
arch/x86/kernel/cpu/resctrl/internal.h +1 -0
index 3b0cdb5520c7..85ebf60a9f1c 100644
@@ -214,5 +214,6 @@ bool rdt_cpu_has(int flag);
 void __init intel_rdt_mbm_apply_quirk(void);
 
 void rdt_domain_reconfigure_cdp(struct rdt_resource *r);
+void resctrl_arch_mbm_cntr_assign_set_one(struct rdt_resource *r);
 
 #endif /* _ASM_X86_RESCTRL_INTERNAL_H */
arch/x86/kernel/cpu/resctrl/monitor.c +8 -0
index c3e15f4de0b4..51a99b8e69d6 100644
@@ -436,6 +436,7 @@ int __init rdt_get_mon_l3_config(struct rdt_resource *r)
 		cpuid_count(0x80000020, 5, &eax, &ebx, &ecx, &edx);
 		r->mon.num_mbm_cntrs = (ebx & GENMASK(15, 0)) + 1;
 		r->mon.mbm_assign_on_mkdir = true;
+		hw_res->mbm_cntr_assign_enabled = true;
 	}
 
 	r->mon_capable = true;
@@ -536,3 +537,10 @@ void resctrl_arch_config_cntr(struct rdt_resource *r, struct rdt_mon_domain *d,
 			memset(am, 0, sizeof(*am));
 	}
 }
+
+void resctrl_arch_mbm_cntr_assign_set_one(struct rdt_resource *r)
+{
+	struct rdt_hw_resource *hw_res = resctrl_to_arch_res(r);
+
+	resctrl_abmc_set_one_amd(&hw_res->mbm_cntr_assign_enabled);
+}
Signature · 1 line
2.34.1
#95 lore raw
In reply to Babu Moger

Hi Babu,

Quoted 73 lines · Babu Moger

This series adds the support for Assignable Bandwidth Monitoring Counters (ABMC). It is also called QoS RMID Pinning feature

Series is written such that it is easier to support other assignable features supported from different vendors.

The feature details are documented in the APM listed below [1].
[1] AMD64 Architecture Programmer's Manual Volume 2: System Programming Publication # 24593 Revision 3.41 section 19.3.3.3 Assignable Bandwidth Monitoring (ABMC). The documentation is available at

The patches are based on top of commit 92a09c47464d0 (tag: v6.15-rc5, tip/irq/merge) Linux 6.15-rc5 plus
https://lore.kernel.org/lkml/[email protected]/

It is very clear these patches will go after James's resctrl FS/ARCH restructure. Hoping to avoid one review cycle due to the merge.

# Introduction

Users can create as many monitor groups as RMIDs supported by the hardware. However, bandwidth monitoring feature on AMD system only guarantees that RMIDs currently assigned to a processor will be tracked by hardware. The counters of any other RMIDs which are no longer being tracked will be reset to zero. The MBM event counters return "Unavailable" for the RMIDs that are not tracked by hardware. So, there can be only limited number of groups that can give guaranteed monitoring numbers. With ever changing configurations there is no way to definitely know which of these groups are being tracked for certain point of time. Users do not have the option to monitor a group or set of groups for certain period of time without worrying about counter being reset in between.

The ABMC feature provides an option to the user to assign a hardware counter to an RMID, event pair and monitor the bandwidth as long as it is assigned. The assigned RMID will be tracked by the hardware until the user unassigns it manually. There is no need to worry about counters being reset during this period. Additionally, the user can specify a bitmask identifying the specific bandwidth types from the given source to track with the counter.

Without ABMC enabled, monitoring will work in current 'default' mode without assignment option.

# History

Earlier implementation of ABMC had dependancy on BMEC (Bandwidth Monitoring Event Configuration). Peter had concerns with that implementation because it may be not be compatible with ARM's MPAM.

Here are the threads discussing the concerns and new interface to address the concerns.
https://lore.kernel.org/lkml/CALPaoCg97cLVVAcacnarp+880xjsedEWGJPXhYpy4P7=ky4MZw@mail.gmail.com/
https://lore.kernel.org/lkml/CALPaoCiii0vXOF06mfV=kVLBzhfNo0SFqt4kQGwGSGVUqvr2Dg@mail.gmail.com/

Here are the finalized requirements based on the discussion:

* Remove BMEC dependency on the ABMC feature.

* Eliminate global assignment listing. The interface
/sys/fs/resctrl/info/L3_MON/mbm_assign_control is no longer required.

* Create the configuration directories at /sys/fs/resctrl/info/L3_MON/counter_configs/.
The configuration file names should be free-form, allowing users to create them as needed.

* Perform assignment listing at the group level by introducing mbm_L3_assignments
in each monitoring group. The listing should provide the following details:

Event Configuration: Specifies the event configuration applied. This will be crucial when "mkdir" on event configuration is added in the future, leading to the creation of mon_data/mon_l3_*/<event configuration>.

Quoted 5 lines

Domains: Identifies the domains where the configuration is applied, supporting multi-domain setups.

Assignment Type: Indicates whether the assignment is Exclusive (e or d), Shared (s), or Unassigned (_).

* Provide option to enable or disable auto assignment when new group is created.

So far I was able to reenable MBM on AMD implementations (for some users) while deferring on the counter assignment interface discussion by just making shared assignment the default for newly-created groups. Until they want to upgrade assignments to exclusive or break down traffic with multiple counters to watch a particular group more closely, they won't need to change any assignments.

Just pointing out that this turned out to be a useful first step in deploying ABMC support.

Quoted 8 lines

This series tries to address all the requirements listed above.

# Implementation details

Create a generic interface aimed to support user space assignment of scarce counters used for monitoring. First usage of interface is by ABMC with option to expand usage to "soft-ABMC" and MPAM counters in future.

I'll try to identify any issues I've encountered with "soft-ABMC". Hopefully I'll be able to share a sample implementation based on these patches soon.

There's now more interest in Google for allowing explicit control of where RMIDs are assigned on Intel platforms. Even though the number of RMIDs implemented by hardware tends to be roughly the number of containers they want to support, they often still need to create containers when all RMIDs have already been allocated, which is not currently allowed. Once the container has been created and starts running, it's no longer possible to move its threads into a monitoring group whenever RMIDs should become available again, so it's important for resctrl to maintain an accurate task list for a container even when RMIDs are not available.

Quoted 9 lines

Feature adds following interface files:

/sys/fs/resctrl/info/L3_MON/mbm_assign_mode: Reports the list of assignable monitoring features supported. The enclosed brackets indicate which feature is enabled.

/sys/fs/resctrl/info/L3_MON/num_mbm_cntrs: Reports the number of monitoring counters available for assignment.

Earlier I discussed with Reinette[1] what num_mbm_cntrs should represent in a "soft-ABMC" implementation where assignment is implemented by assigning an RMID, which would result in all events being assigned at once.

My main concern is how many "counters" you can assign by assigning RMIDs. I recall Reinette proposed reporting the number of groups which can be assigned separately from counters which can be assigned.

Quoted 11 lines

/sys/fs/resctrl/info/L3_MON/available_mbm_cntrs: Reports the number of monitoring counters free in each domain.

/sys/fs/resctrl/info/L3_MON/counter_configs : Directory to hold the counter configuration.

/sys/fs/resctrl/info/L3_MON/counter_configs/mbm_total_bytes/event_filter : Default configuration for MBM total events.

/sys/fs/resctrl/info/L3_MON/counter_configs/mbm_local_bytes/event_filter : Default configuration for MBM local events.

IIUC, this needs to be implemented now so you can drop BMEC with this series?

Quoted 30 lines

/sys/fs/resctrl/mbm_L3_assignments: Interface to list or modify assignment states on each group.

# Examples

a. Check if ABMC support is available
#mount -t resctrl resctrl /sys/fs/resctrl/

# cat /sys/fs/resctrl/info/L3_MON/mbm_assign_mode [mbm_cntr_assign]
default

ABMC feature is detected and it is enabled.

b. Check how many ABMC counters are available.

        # cat /sys/fs/resctrl/info/L3_MON/num_mbm_cntrs
        32

c. Check how many ABMC counters are available in each domain.

        # cat /sys/fs/resctrl/info/L3_MON/available_mbm_cntrs
        0=30;1=30

d. Check default counter configuration.

        # cat /sys/fs/resctrl/info/L3_MON/counter_configs/mbm_total_bytes/event_filter
        local_reads, remote_reads, local_non_temporal_writes, remote_non_temporal_writes,
        local_reads_slow_memory, remote_reads_slow_memory, dirty_victim_writes_all
Quoted 5 lines
        # cat /sys/fs/resctrl/info/L3_MON/counter_configs/mbm_local_bytes/event_filter
        local_reads, local_non_temporal_writes, local_reads_slow_memory

e. Series adds a new interface file "mbm_L3_assignments" in each monitoring group
to list and modify any group's monitoring states.

To confirm, would we have "mbm_<resource_name>_assignments" for each resource where MBM-ish events could be assigned?

Quoted 4 lines

The list is displayed in the following format:

<Event configuration>:<Domain id>=<Assignment type>

For soft-ABMC assignment, is there just a single event configuration representing all the events tracked by the RMID?

Quoted 44 lines

Event configuration: A valid event configuration listed in the /sys/fs/resctrl/info/L3_MON/counter_configs directory.

Domain ID: A valid domain ID number.

Assignment types:

_ : No event configuration assigned

e : Event configuration assigned in exclusive mode

To list the default group states:
# cat /sys/fs/resctrl/mbm_L3_assignments
mbm_total_bytes:0=e;1=e
mbm_local_bytes:0=e;1=e

To unassign the configuration of mbm_total_bytes on domain 0:
#echo "mbm_total_bytes:0=_" > mbm_L3_assignments #cat mbm_L3_assignments
mbm_total_bytes:0=_;1=e
mbm_local_bytes:0=e;1=e

To unassign the mbm_total_bytes configuration on all domains:
$echo "mbm_total_bytes:*=_" > mbm_L3_assignments $cat mbm_L3_assignments
mbm_total_bytes:0=_;1=_
mbm_local_bytes:0=e;1=e

To assign the mbm_total_bytes configuration on all domains in exclusive mode:
$echo "mbm_total_bytes:*=e" > mbm_L3_assignments
$cat mbm_L3_assignments
mbm_total_bytes:0=e;1=e
mbm_local_bytes:0=e;1=e

g. Read the events mbm_total_bytes and mbm_local_bytes of the default group.
There is no change in reading the events with ABMC. If the event is unassigned when reading, then the read will come back as "Unassigned".

        # cat /sys/fs/resctrl/mon_data/mon_L3_00/mbm_total_bytes
        779247936
        # cat /sys/fs/resctrl/mon_data/mon_L3_00/mbm_local_bytes
        765207488
Quoted 8 lines

h. Check the default event configurations.

#cat /sys/fs/resctrl/info/L3_MON/counter_configs/mbm_total_bytes/event_filter local_reads, remote_reads, local_non_temporal_writes, remote_non_temporal_writes, local_reads_slow_memory, remote_reads_slow_memory, dirty_victim_writes_all

#cat /sys/fs/resctrl/info/L3_MON/counter_configs/mbm_local_bytes/event_filter local_reads, local_non_temporal_writes, local_reads_slow_memory

These look like the BMEC event names converted from camel case. Will event filter programming be portable?

Thanks,
-Peter

[1] https://lore.kernel.org/lkml/[email protected]/

author reviewer
#96 lore raw

Hi Peter,

Thanks for trying the series.

Quoted 89 lines · Peter Newman

Hi Babu,

Quoted 73 lines · Babu Moger

This series adds the support for Assignable Bandwidth Monitoring Counters (ABMC). It is also called QoS RMID Pinning feature

Series is written such that it is easier to support other assignable features supported from different vendors.

The feature details are documented in the APM listed below [1].
[1] AMD64 Architecture Programmer's Manual Volume 2: System Programming Publication # 24593 Revision 3.41 section 19.3.3.3 Assignable Bandwidth Monitoring (ABMC). The documentation is available at

The patches are based on top of commit 92a09c47464d0 (tag: v6.15-rc5, tip/irq/merge) Linux 6.15-rc5 plus
https://lore.kernel.org/lkml/[email protected]/

It is very clear these patches will go after James's resctrl FS/ARCH restructure. Hoping to avoid one review cycle due to the merge.

# Introduction

Users can create as many monitor groups as RMIDs supported by the hardware. However, bandwidth monitoring feature on AMD system only guarantees that RMIDs currently assigned to a processor will be tracked by hardware. The counters of any other RMIDs which are no longer being tracked will be reset to zero. The MBM event counters return "Unavailable" for the RMIDs that are not tracked by hardware. So, there can be only limited number of groups that can give guaranteed monitoring numbers. With ever changing configurations there is no way to definitely know which of these groups are being tracked for certain point of time. Users do not have the option to monitor a group or set of groups for certain period of time without worrying about counter being reset in between.

The ABMC feature provides an option to the user to assign a hardware counter to an RMID, event pair and monitor the bandwidth as long as it is assigned. The assigned RMID will be tracked by the hardware until the user unassigns it manually. There is no need to worry about counters being reset during this period. Additionally, the user can specify a bitmask identifying the specific bandwidth types from the given source to track with the counter.

Without ABMC enabled, monitoring will work in current 'default' mode without assignment option.

# History

Earlier implementation of ABMC had dependancy on BMEC (Bandwidth Monitoring Event Configuration). Peter had concerns with that implementation because it may be not be compatible with ARM's MPAM.

Here are the threads discussing the concerns and new interface to address the concerns.
https://lore.kernel.org/lkml/CALPaoCg97cLVVAcacnarp+880xjsedEWGJPXhYpy4P7=ky4MZw@mail.gmail.com/
https://lore.kernel.org/lkml/CALPaoCiii0vXOF06mfV=kVLBzhfNo0SFqt4kQGwGSGVUqvr2Dg@mail.gmail.com/

Here are the finalized requirements based on the discussion:

* Remove BMEC dependency on the ABMC feature.

* Eliminate global assignment listing. The interface
/sys/fs/resctrl/info/L3_MON/mbm_assign_control is no longer required.

* Create the configuration directories at /sys/fs/resctrl/info/L3_MON/counter_configs/.
The configuration file names should be free-form, allowing users to create them as needed.

* Perform assignment listing at the group level by introducing mbm_L3_assignments
in each monitoring group. The listing should provide the following details:

Event Configuration: Specifies the event configuration applied. This will be crucial when "mkdir" on event configuration is added in the future, leading to the creation of mon_data/mon_l3_*/<event configuration>.

Quoted 5 lines

Domains: Identifies the domains where the configuration is applied, supporting multi-domain setups.

Assignment Type: Indicates whether the assignment is Exclusive (e or d), Shared (s), or Unassigned (_).

* Provide option to enable or disable auto assignment when new group is created.

So far I was able to reenable MBM on AMD implementations (for some users) while deferring on the counter assignment interface discussion by just making shared assignment the default for newly-created groups. Until they want to upgrade assignments to exclusive or break down traffic with multiple counters to watch a particular group more closely, they won't need to change any assignments.

Quoted 2 lines

Just pointing out that this turned out to be a useful first step in deploying ABMC support.

Thank you.

Quoted 6 lines
Quoted 6 lines

This series tries to address all the requirements listed above.

# Implementation details

Quoted 7 lines
Quoted 3 lines

Create a generic interface aimed to support user space assignment of scarce counters used for monitoring. First usage of interface is by ABMC with option to expand usage to "soft-ABMC" and MPAM counters in future.

I'll try to identify any issues I've encountered with "soft-ABMC". Hopefully I'll be able to share a sample implementation based on these patches soon.

That would be wonderful.

Quoted 22 lines

There's now more interest in Google for allowing explicit control of where RMIDs are assigned on Intel platforms. Even though the number of RMIDs implemented by hardware tends to be roughly the number of containers they want to support, they often still need to create containers when all RMIDs have already been allocated, which is not currently allowed. Once the container has been created and starts running, it's no longer possible to move its threads into a monitoring group whenever RMIDs should become available again, so it's important for resctrl to maintain an accurate task list for a container even when RMIDs are not available.

Quoted 9 lines

Feature adds following interface files:

/sys/fs/resctrl/info/L3_MON/mbm_assign_mode: Reports the list of assignable monitoring features supported. The enclosed brackets indicate which feature is enabled.

/sys/fs/resctrl/info/L3_MON/num_mbm_cntrs: Reports the number of monitoring counters available for assignment.

Quoted 8 lines

Earlier I discussed with Reinette[1] what num_mbm_cntrs should represent in a "soft-ABMC" implementation where assignment is implemented by assigning an RMID, which would result in all events being assigned at once.

My main concern is how many "counters" you can assign by assigning RMIDs. I recall Reinette proposed reporting the number of groups which can be assigned separately from counters which can be assigned.

More context may be needed here. Currently, num_mbm_cntrs indicates the number of counters available per domain, which is 32.

At the moment, we can assign 2 counters to each group, meaning each RMID can be associated with 2 hardware counters. In theory, it's possible to assign all 32 hardware counters to a group—allowing one RMID to be linked with up to 32 counters. However, we currently lack the interface to support that level of assignment.

For now, the plan is to support basic assignment and expand functionality later once we have the necessary data structure and requirements.

Quoted 7 lines
Quoted 7 lines

/sys/fs/resctrl/info/L3_MON/available_mbm_cntrs: Reports the number of monitoring counters free in each domain.

/sys/fs/resctrl/info/L3_MON/counter_configs : Directory to hold the counter configuration.

Quoted 7 lines
Quoted 5 lines

/sys/fs/resctrl/info/L3_MON/counter_configs/mbm_total_bytes/event_filter : Default configuration for MBM total events.

/sys/fs/resctrl/info/L3_MON/counter_configs/mbm_local_bytes/event_filter : Default configuration for MBM local events.

IIUC, this needs to be implemented now so you can drop BMEC with this series?

This series hides the configuration files (mbm_local_bytes_config and mbm_total_bytes_config) required for BMEC when ABMC is enabled.

When the user switches back to "default" mode, BMEC becomes available again. I believe it's a good approach to keep it this way.

Quoted 31 lines
Quoted 31 lines

/sys/fs/resctrl/mbm_L3_assignments: Interface to list or modify assignment states on each group.

# Examples

a. Check if ABMC support is available
#mount -t resctrl resctrl /sys/fs/resctrl/

# cat /sys/fs/resctrl/info/L3_MON/mbm_assign_mode [mbm_cntr_assign]
default

ABMC feature is detected and it is enabled.

b. Check how many ABMC counters are available.

        # cat /sys/fs/resctrl/info/L3_MON/num_mbm_cntrs
        32

c. Check how many ABMC counters are available in each domain.

        # cat /sys/fs/resctrl/info/L3_MON/available_mbm_cntrs
        0=30;1=30

d. Check default counter configuration.

        # cat /sys/fs/resctrl/info/L3_MON/counter_configs/mbm_total_bytes/event_filter
        local_reads, remote_reads, local_non_temporal_writes, remote_non_temporal_writes,
        local_reads_slow_memory, remote_reads_slow_memory, dirty_victim_writes_all
Quoted 8 lines
Quoted 5 lines
        # cat /sys/fs/resctrl/info/L3_MON/counter_configs/mbm_local_bytes/event_filter
        local_reads, local_non_temporal_writes, local_reads_slow_memory

e. Series adds a new interface file "mbm_L3_assignments" in each monitoring group
to list and modify any group's monitoring states.

To confirm, would we have "mbm_<resource_name>_assignments" for each resource where MBM-ish events could be assigned?

This is a group-level property—it resides within each group and is not related to any specific resource.

Quoted 8 lines
Quoted 5 lines

The list is displayed in the following format:

<Event configuration>:<Domain id>=<Assignment type>

For soft-ABMC assignment, is there just a single event configuration representing all the events tracked by the RMID?

I’m not sure about the details of how soft-ABMC will be supported. It’s not available at the moment, but I believe it can be added once soft-ABMC support is in place.

Quoted 51 lines
Quoted 51 lines

Event configuration: A valid event configuration listed in the /sys/fs/resctrl/info/L3_MON/counter_configs directory.

Domain ID: A valid domain ID number.

Assignment types:

_ : No event configuration assigned

e : Event configuration assigned in exclusive mode

To list the default group states:
# cat /sys/fs/resctrl/mbm_L3_assignments
mbm_total_bytes:0=e;1=e
mbm_local_bytes:0=e;1=e

To unassign the configuration of mbm_total_bytes on domain 0:
#echo "mbm_total_bytes:0=_" > mbm_L3_assignments #cat mbm_L3_assignments
mbm_total_bytes:0=_;1=e
mbm_local_bytes:0=e;1=e

To unassign the mbm_total_bytes configuration on all domains:
$echo "mbm_total_bytes:*=_" > mbm_L3_assignments $cat mbm_L3_assignments
mbm_total_bytes:0=_;1=_
mbm_local_bytes:0=e;1=e

To assign the mbm_total_bytes configuration on all domains in exclusive mode:
$echo "mbm_total_bytes:*=e" > mbm_L3_assignments
$cat mbm_L3_assignments
mbm_total_bytes:0=e;1=e
mbm_local_bytes:0=e;1=e

g. Read the events mbm_total_bytes and mbm_local_bytes of the default group.
There is no change in reading the events with ABMC. If the event is unassigned when reading, then the read will come back as "Unassigned".

        # cat /sys/fs/resctrl/mon_data/mon_L3_00/mbm_total_bytes
        779247936
        # cat /sys/fs/resctrl/mon_data/mon_L3_00/mbm_local_bytes
        765207488

h. Check the default event configurations.

#cat /sys/fs/resctrl/info/L3_MON/counter_configs/mbm_total_bytes/event_filter local_reads, remote_reads, local_non_temporal_writes, remote_non_temporal_writes, local_reads_slow_memory, remote_reads_slow_memory, dirty_victim_writes_all

Quoted 5 lines
Quoted 2 lines

#cat /sys/fs/resctrl/info/L3_MON/counter_configs/mbm_local_bytes/event_filter local_reads, local_non_temporal_writes, local_reads_slow_memory

These look like the BMEC event names converted from camel case. Will event filter programming be portable?

Yes, that’s correct. The event types (reads, writes, etc.) supported by both BMEC and ABMC are the same, so I’ve used generalized names here.

As for portability, I can’t comment, since I’m not familiar with how event configuration is handled in MPAM or other architectures.

Quoted 6 lines

Thanks,
-Peter

[1] https://lore.kernel.org/lkml/[email protected]/

Signature · 2 lines
Thanks
Babu Moger
maintainer
#97 lore raw

Hi Babu,

Quoted 2 lines · Babu
Quoted 1 line · Peter Newman

On Fri, May 16, 2025 at 12:52 AM Babu Moger <[email protected]> wrote:

...

Quoted 15 lines
Quoted 11 lines
Quoted 2 lines

/sys/fs/resctrl/info/L3_MON/num_mbm_cntrs: Reports the number of monitoring counters available for assignment.

Earlier I discussed with Reinette[1] what num_mbm_cntrs should represent in a "soft-ABMC" implementation where assignment is implemented by assigning an RMID, which would result in all events being assigned at once.

My main concern is how many "counters" you can assign by assigning RMIDs. I recall Reinette proposed reporting the number of groups which can be assigned separately from counters which can be assigned.

More context may be needed here. Currently, num_mbm_cntrs indicates the number of counters available per domain, which is 32.

Quoted 8 lines

At the moment, we can assign 2 counters to each group, meaning each RMID can be associated with 2 hardware counters. In theory, it's possible to assign all 32 hardware counters to a group—allowing one RMID to be linked with up to 32 counters. However, we currently lack the interface to support that level of assignment.

For now, the plan is to support basic assignment and expand functionality later once we have the necessary data structure and requirements.

Looks like some requirements did not make it into this implementation. Do you recall the discussion that resulted in you writing [2]? Looks like there is a question to Peter in there on how to determine how many "counters" are available in soft-ABMC. I interpreted [3] at that time to mean that this information would be available in a future AMD publication.

Reinette

[2] https://lore.kernel.org/lkml/[email protected]/
[3] https://lore.kernel.org/lkml/CALPaoCg3KpF94g2MEmfP_Ro2mQZYFA8sKVkmb+7isotKNgdY9A@mail.gmail.com/

author reviewer
#98 lore raw

Hi Reinette,

Quoted 29 lines · Reinette Chatre

Hi Babu,

Quoted 2 lines · Babu
Quoted 1 line · Peter Newman

On Fri, May 16, 2025 at 12:52 AM Babu Moger <[email protected]> wrote:

...

Quoted 21 lines
Quoted 11 lines
Quoted 2 lines

/sys/fs/resctrl/info/L3_MON/num_mbm_cntrs: Reports the number of monitoring counters available for assignment.

Earlier I discussed with Reinette[1] what num_mbm_cntrs should represent in a "soft-ABMC" implementation where assignment is implemented by assigning an RMID, which would result in all events being assigned at once.

My main concern is how many "counters" you can assign by assigning RMIDs. I recall Reinette proposed reporting the number of groups which can be assigned separately from counters which can be assigned.

More context may be needed here. Currently, num_mbm_cntrs indicates the number of counters available per domain, which is 32.

At the moment, we can assign 2 counters to each group, meaning each RMID can be associated with 2 hardware counters. In theory, it's possible to assign all 32 hardware counters to a group—allowing one RMID to be linked with up to 32 counters. However, we currently lack the interface to support that level of assignment.

Quoted 8 lines
Quoted 2 lines

For now, the plan is to support basic assignment and expand functionality later once we have the necessary data structure and requirements.

Looks like some requirements did not make it into this implementation. Do you recall the discussion that resulted in you writing [2]? Looks like there is a question to Peter in there on how to determine how many "counters" are available in soft-ABMC. I interpreted [3] at that time to mean that this information would be available in a future AMD publication.

We already have a method to determine the number of counters in soft-ABMC mode, which Peter has addressed [4].

[4]
https://lore.kernel.org/lkml/[email protected]/

This appears to be more of a workaround, and I doubt it will be included in any official AMD documentation. Additionally, the long-term direction is moving towards ABMC.

I don’t believe this workaround needs to be part of the current series. It can be added later when soft-ABMC is implemented.

Quoted 5 lines

Reinette

[2] https://lore.kernel.org/lkml/[email protected]/
[3] https://lore.kernel.org/lkml/CALPaoCg3KpF94g2MEmfP_Ro2mQZYFA8sKVkmb+7isotKNgdY9A@mail.gmail.com/

Signature · 2 lines
Thanks
Babu Moger
maintainer
#99 lore raw

Hi Babu,

Quoted 47 lines · Babu

Hi Reinette,

Quoted 29 lines · Reinette Chatre

Hi Babu,

Quoted 2 lines · Babu
Quoted 1 line · Peter Newman

On Fri, May 16, 2025 at 12:52 AM Babu Moger <[email protected]> wrote:

...

Quoted 21 lines
Quoted 11 lines
Quoted 2 lines

/sys/fs/resctrl/info/L3_MON/num_mbm_cntrs: Reports the number of monitoring counters available for assignment.

Earlier I discussed with Reinette[1] what num_mbm_cntrs should represent in a "soft-ABMC" implementation where assignment is implemented by assigning an RMID, which would result in all events being assigned at once.

My main concern is how many "counters" you can assign by assigning RMIDs. I recall Reinette proposed reporting the number of groups which can be assigned separately from counters which can be assigned.

More context may be needed here. Currently, num_mbm_cntrs indicates the number of counters available per domain, which is 32.

At the moment, we can assign 2 counters to each group, meaning each RMID can be associated with 2 hardware counters. In theory, it's possible to assign all 32 hardware counters to a group—allowing one RMID to be linked with up to 32 counters. However, we currently lack the interface to support that level of assignment.

Quoted 8 lines
Quoted 2 lines

For now, the plan is to support basic assignment and expand functionality later once we have the necessary data structure and requirements.

Looks like some requirements did not make it into this implementation. Do you recall the discussion that resulted in you writing [2]? Looks like there is a question to Peter in there on how to determine how many "counters" are available in soft-ABMC. I interpreted [3] at that time to mean that this information would be available in a future AMD publication.

We already have a method to determine the number of counters in soft-ABMC mode, which Peter has addressed [4].

[4]
https://lore.kernel.org/lkml/[email protected]/

Quoted 6 lines

This appears to be more of a workaround, and I doubt it will be included in any official AMD documentation. Additionally, the long-term direction is moving towards ABMC.

I don’t believe this workaround needs to be part of the current series. It can be added later when soft-ABMC is implemented.

Agreed. What about the plans described in [2]? (Thanks to Peter for catching this!).

It is important to keep track of requirements while working on a feature to ensure that the implementation supports the planned use cases. Re-reading that thread it is not clear to me how soft-ABMC's per-group assignment would look. Could you please share how you see it progress from this implementation? This includes the single event vs. multiple event assignment. I would like to highlight that this is not a request for this to be supported in this implementation but there needs to be a plan for how this can be supported on top of interfaces established by this work.

Reinette

Quoted 7 lines
Quoted 6 lines

Reinette

[2] https://lore.kernel.org/lkml/[email protected]/
[3] https://lore.kernel.org/lkml/CALPaoCg3KpF94g2MEmfP_Ro2mQZYFA8sKVkmb+7isotKNgdY9A@mail.gmail.com/

#100 lore raw

Hi Reinette,

Quoted 60 lines · Reinette Chatre

Hi Babu,

Quoted 47 lines · Babu

Hi Reinette,

Quoted 29 lines · Reinette Chatre

Hi Babu,

Quoted 2 lines · Babu
Quoted 1 line · Peter Newman

On Fri, May 16, 2025 at 12:52 AM Babu Moger <[email protected]> wrote:

...

Quoted 21 lines
Quoted 11 lines
Quoted 2 lines

/sys/fs/resctrl/info/L3_MON/num_mbm_cntrs: Reports the number of monitoring counters available for assignment.

Earlier I discussed with Reinette[1] what num_mbm_cntrs should represent in a "soft-ABMC" implementation where assignment is implemented by assigning an RMID, which would result in all events being assigned at once.

My main concern is how many "counters" you can assign by assigning RMIDs. I recall Reinette proposed reporting the number of groups which can be assigned separately from counters which can be assigned.

More context may be needed here. Currently, num_mbm_cntrs indicates the number of counters available per domain, which is 32.

At the moment, we can assign 2 counters to each group, meaning each RMID can be associated with 2 hardware counters. In theory, it's possible to assign all 32 hardware counters to a group—allowing one RMID to be linked with up to 32 counters. However, we currently lack the interface to support that level of assignment.

Quoted 8 lines
Quoted 2 lines

For now, the plan is to support basic assignment and expand functionality later once we have the necessary data structure and requirements.

Looks like some requirements did not make it into this implementation. Do you recall the discussion that resulted in you writing [2]? Looks like there is a question to Peter in there on how to determine how many "counters" are available in soft-ABMC. I interpreted [3] at that time to mean that this information would be available in a future AMD publication.

We already have a method to determine the number of counters in soft-ABMC mode, which Peter has addressed [4].

[4]
https://lore.kernel.org/lkml/[email protected]/

Quoted 6 lines

This appears to be more of a workaround, and I doubt it will be included in any official AMD documentation. Additionally, the long-term direction is moving towards ABMC.

I don’t believe this workaround needs to be part of the current series. It can be added later when soft-ABMC is implemented.

Agreed. What about the plans described in [2]? (Thanks to Peter for catching this!).

Quoted 9 lines

It is important to keep track of requirements while working on a feature to ensure that the implementation supports the planned use cases. Re-reading that thread it is not clear to me how soft-ABMC's per-group assignment would look. Could you please share how you see it progress from this implementation? This includes the single event vs. multiple event assignment. I would like to highlight that this is not a request for this to be supported in this implementation but there needs to be a plan for how this can be supported on top of interfaces established by this work.

Here’s my current understanding of soft-ABMC. Peter may have a more in-depth perspective on this.

Soft-ABMC:
a. num_mbm_cntrs: This is a software-defined limit based on the number of active RMIDs that can be supported. The value can be obtained using the code referenced in [4].

b. Assignments: No hardware configuration is required. We simply need to ensure that no more than num_mbm_cntrs RMIDs are active at any given time.

c. Configuration: Controlled via /info/L3_MON/mbm_total_bytes_config and mbm_local_bytes_config.

d. Events: Only two events can be assigned(local and total).

ABMC:
a. num_mbm_cntrs: This is defined by the hardware.
b. Assignments: Requires special MSR writes to assign counters.
c. Configuration: Comes from /info/L3_MON/counter_configs/.
d. Events: More than two events can be assigned to a group (currently up to 2).

Commonalities:
a. Assignments can be either exclusive or shared in both these modes.

Given these, I believe we can easily accommodate soft-ABMC in this interface.

Quoted 6 lines
Quoted 4 lines
Quoted 3 lines

[2] https://lore.kernel.org/lkml/[email protected]/
[3] https://lore.kernel.org/lkml/CALPaoCg3KpF94g2MEmfP_Ro2mQZYFA8sKVkmb+7isotKNgdY9A@mail.gmail.com/

maintainer
#101 lore raw

Hi Babu,

Quoted 90 lines · Babu

Hi Reinette,

Quoted 60 lines · Reinette Chatre

Hi Babu,

Quoted 47 lines · Babu

Hi Reinette,

Quoted 29 lines · Reinette Chatre

Hi Babu,

Quoted 2 lines · Babu
Quoted 1 line · Peter Newman

On Fri, May 16, 2025 at 12:52 AM Babu Moger <[email protected]> wrote:

...

Quoted 21 lines
Quoted 11 lines
Quoted 2 lines

/sys/fs/resctrl/info/L3_MON/num_mbm_cntrs: Reports the number of monitoring counters available for assignment.

Earlier I discussed with Reinette[1] what num_mbm_cntrs should represent in a "soft-ABMC" implementation where assignment is implemented by assigning an RMID, which would result in all events being assigned at once.

My main concern is how many "counters" you can assign by assigning RMIDs. I recall Reinette proposed reporting the number of groups which can be assigned separately from counters which can be assigned.

More context may be needed here. Currently, num_mbm_cntrs indicates the number of counters available per domain, which is 32.

At the moment, we can assign 2 counters to each group, meaning each RMID can be associated with 2 hardware counters. In theory, it's possible to assign all 32 hardware counters to a group—allowing one RMID to be linked with up to 32 counters. However, we currently lack the interface to support that level of assignment.

Quoted 8 lines
Quoted 2 lines

For now, the plan is to support basic assignment and expand functionality later once we have the necessary data structure and requirements.

Looks like some requirements did not make it into this implementation. Do you recall the discussion that resulted in you writing [2]? Looks like there is a question to Peter in there on how to determine how many "counters" are available in soft-ABMC. I interpreted [3] at that time to mean that this information would be available in a future AMD publication.

We already have a method to determine the number of counters in soft-ABMC mode, which Peter has addressed [4].

[4]
https://lore.kernel.org/lkml/[email protected]/

Quoted 6 lines

This appears to be more of a workaround, and I doubt it will be included in any official AMD documentation. Additionally, the long-term direction is moving towards ABMC.

I don’t believe this workaround needs to be part of the current series. It can be added later when soft-ABMC is implemented.

Agreed. What about the plans described in [2]? (Thanks to Peter for catching this!).

Quoted 9 lines

It is important to keep track of requirements while working on a feature to ensure that the implementation supports the planned use cases. Re-reading that thread it is not clear to me how soft-ABMC's per-group assignment would look. Could you please share how you see it progress from this implementation? This includes the single event vs. multiple event assignment. I would like to highlight that this is not a request for this to be supported in this implementation but there needs to be a plan for how this can be supported on top of interfaces established by this work.

Here’s my current understanding of soft-ABMC. Peter may have a more in-depth perspective on this.

Soft-ABMC:
a. num_mbm_cntrs: This is a software-defined limit based on the number of active RMIDs that can be supported. The value can be obtained using the code referenced in [4].

b. Assignments: No hardware configuration is required. We simply need to ensure that no more than num_mbm_cntrs RMIDs are active at any given time.

c. Configuration: Controlled via /info/L3_MON/mbm_total_bytes_config and mbm_local_bytes_config.

d. Events: Only two events can be assigned(local and total).

ABMC:
a. num_mbm_cntrs: This is defined by the hardware.
b. Assignments: Requires special MSR writes to assign counters.
c. Configuration: Comes from /info/L3_MON/counter_configs/.
d. Events: More than two events can be assigned to a group (currently up to 2).

Quoted 4 lines

Commonalities:
a. Assignments can be either exclusive or shared in both these modes.

Given these, I believe we can easily accommodate soft-ABMC in this interface.

This is not so obvious to me. It looks to me as though the user is forced to interpret the content of resctrl files differently based on soft-ABMC vs ABMC making the interface inconsistent and user thus needing to know details of implementations. This is what the previous discussion I linked to aimed to address. It sounds to me as though you believe that this is no longer an issue. Could you please show examples of what a user can expect from the interfaces and how a user will interact with the interfaces on both a non-ABMC and ABMC system?

Thank you

Reinette

Quoted 8 lines
Quoted 7 lines
Quoted 5 lines
Quoted 4 lines

[2] https://lore.kernel.org/lkml/[email protected]/
[3] https://lore.kernel.org/lkml/CALPaoCg3KpF94g2MEmfP_Ro2mQZYFA8sKVkmb+7isotKNgdY9A@mail.gmail.com/

#102 lore raw

Hi Babu/Reinette,

Quoted 81 lines · Reinette Chatre

Hi Babu,

Quoted 77 lines · Babu

Hi Reinette,

Quoted 60 lines · Reinette Chatre

Hi Babu,

Quoted 47 lines · Babu

Hi Reinette,

Quoted 29 lines · Reinette Chatre

Hi Babu,

Quoted 2 lines · Babu
Quoted 1 line · Peter Newman

On Fri, May 16, 2025 at 12:52 AM Babu Moger <[email protected]> wrote:

...

Quoted 21 lines
Quoted 11 lines
Quoted 2 lines

/sys/fs/resctrl/info/L3_MON/num_mbm_cntrs: Reports the number of monitoring counters available for assignment.

Earlier I discussed with Reinette[1] what num_mbm_cntrs should represent in a "soft-ABMC" implementation where assignment is implemented by assigning an RMID, which would result in all events being assigned at once.

My main concern is how many "counters" you can assign by assigning RMIDs. I recall Reinette proposed reporting the number of groups which can be assigned separately from counters which can be assigned.

More context may be needed here. Currently, num_mbm_cntrs indicates the number of counters available per domain, which is 32.

At the moment, we can assign 2 counters to each group, meaning each RMID can be associated with 2 hardware counters. In theory, it's possible to assign all 32 hardware counters to a group—allowing one RMID to be linked with up to 32 counters. However, we currently lack the interface to support that level of assignment.

Quoted 8 lines
Quoted 2 lines

For now, the plan is to support basic assignment and expand functionality later once we have the necessary data structure and requirements.

Looks like some requirements did not make it into this implementation. Do you recall the discussion that resulted in you writing [2]? Looks like there is a question to Peter in there on how to determine how many "counters" are available in soft-ABMC. I interpreted [3] at that time to mean that this information would be available in a future AMD publication.

We already have a method to determine the number of counters in soft-ABMC mode, which Peter has addressed [4].

[4]
https://lore.kernel.org/lkml/[email protected]/

Quoted 6 lines

This appears to be more of a workaround, and I doubt it will be included in any official AMD documentation. Additionally, the long-term direction is moving towards ABMC.

I don’t believe this workaround needs to be part of the current series. It can be added later when soft-ABMC is implemented.

Agreed. What about the plans described in [2]? (Thanks to Peter for catching this!).

Quoted 9 lines

It is important to keep track of requirements while working on a feature to ensure that the implementation supports the planned use cases. Re-reading that thread it is not clear to me how soft-ABMC's per-group assignment would look. Could you please share how you see it progress from this implementation? This includes the single event vs. multiple event assignment. I would like to highlight that this is not a request for this to be supported in this implementation but there needs to be a plan for how this can be supported on top of interfaces established by this work.

Here’s my current understanding of soft-ABMC. Peter may have a more in-depth perspective on this.

Soft-ABMC:
a. num_mbm_cntrs: This is a software-defined limit based on the number of active RMIDs that can be supported. The value can be obtained using the code referenced in [4].

I would call it a hardware-defined limit that can be probed by software.

The main question is whether this file returns the exact number of RMIDs hardware can track or double that number (mbm_total_bytes + mbm_local_bytes) so that the value is always measured in events.

There's also the mongroup-RMID overcommit use case I described above[1]. On Intel we can safely assume that there are counters to back all RMIDs, so num_mbm_cntrs would be calculated directly from num_rmids.

I realized this use case is more difficult to implement on MPAM, because a PARTID is effectively a CLOSID+RMID, so deferring assigning a unique PARTID to a group also results in it being in a different allocation group. It will work if the unmonitored groups could find a way to share PARTIDs, but this has consequences on allocation - but hopefully no worse than sharing CLOSIDs on x86.

There's a lot of interest in monitoring ID overcommit in Google, so I think it's worth it for me to investigate the additional structural changes needed in resctrl (i.e., breaking the FS-level association between mongroups and HW monitoring IDs). Such a framework could be a better fit for soft-ABMC. For example, if overcommit is allowed, we would just report the number of simultaneous RMIDs we were able to probe as num_rmids. I would want the same shared assignment scheduler to be able to work with RMIDs and counters, though.

Thanks,
-Peter

[1] https://lore.kernel.org/lkml/CALPaoChSzzU5mzMZsdT6CeyEn0WD1qdT9fKCoNW_ty4tojtrkw@mail.gmail.com/

#116 lore raw
In reply to Reinette Chatre

Hi Reinette,

Quoted 97 lines · Reinette Chatre

Hi Babu,

Quoted 93 lines · Babu

Hi Reinette,

Quoted 60 lines · Reinette Chatre

Hi Babu,

Quoted 47 lines · Babu

Hi Reinette,

Quoted 29 lines · Reinette Chatre

Hi Babu,

Quoted 2 lines · Babu
Quoted 1 line · Peter Newman

On Fri, May 16, 2025 at 12:52 AM Babu Moger <[email protected]> wrote:

...

Quoted 21 lines
Quoted 11 lines
Quoted 2 lines

/sys/fs/resctrl/info/L3_MON/num_mbm_cntrs: Reports the number of monitoring counters available for assignment.

Earlier I discussed with Reinette[1] what num_mbm_cntrs should represent in a "soft-ABMC" implementation where assignment is implemented by assigning an RMID, which would result in all events being assigned at once.

My main concern is how many "counters" you can assign by assigning RMIDs. I recall Reinette proposed reporting the number of groups which can be assigned separately from counters which can be assigned.

More context may be needed here. Currently, num_mbm_cntrs indicates the number of counters available per domain, which is 32.

At the moment, we can assign 2 counters to each group, meaning each RMID can be associated with 2 hardware counters. In theory, it's possible to assign all 32 hardware counters to a group—allowing one RMID to be linked with up to 32 counters. However, we currently lack the interface to support that level of assignment.

Quoted 8 lines
Quoted 2 lines

For now, the plan is to support basic assignment and expand functionality later once we have the necessary data structure and requirements.

Looks like some requirements did not make it into this implementation. Do you recall the discussion that resulted in you writing [2]? Looks like there is a question to Peter in there on how to determine how many "counters" are available in soft-ABMC. I interpreted [3] at that time to mean that this information would be available in a future AMD publication.

We already have a method to determine the number of counters in soft-ABMC mode, which Peter has addressed [4].

[4]
https://lore.kernel.org/lkml/[email protected]/

Quoted 6 lines

This appears to be more of a workaround, and I doubt it will be included in any official AMD documentation. Additionally, the long-term direction is moving towards ABMC.

I don’t believe this workaround needs to be part of the current series. It can be added later when soft-ABMC is implemented.

Agreed. What about the plans described in [2]? (Thanks to Peter for catching this!).

Quoted 9 lines

It is important to keep track of requirements while working on a feature to ensure that the implementation supports the planned use cases. Re-reading that thread it is not clear to me how soft-ABMC's per-group assignment would look. Could you please share how you see it progress from this implementation? This includes the single event vs. multiple event assignment. I would like to highlight that this is not a request for this to be supported in this implementation but there needs to be a plan for how this can be supported on top of interfaces established by this work.

Here’s my current understanding of soft-ABMC. Peter may have a more in-depth perspective on this.

Soft-ABMC:
a. num_mbm_cntrs: This is a software-defined limit based on the number of active RMIDs that can be supported. The value can be obtained using the code referenced in [4].

b. Assignments: No hardware configuration is required. We simply need to ensure that no more than num_mbm_cntrs RMIDs are active at any given time.

c. Configuration: Controlled via /info/L3_MON/mbm_total_bytes_config and mbm_local_bytes_config.

d. Events: Only two events can be assigned(local and total).

ABMC:
a. num_mbm_cntrs: This is defined by the hardware.
b. Assignments: Requires special MSR writes to assign counters.
c. Configuration: Comes from /info/L3_MON/counter_configs/.
d. Events: More than two events can be assigned to a group (currently up to 2).

Commonalities:
a. Assignments can be either exclusive or shared in both these modes.

Quoted 8 lines
Quoted 1 line

Given these, I believe we can easily accommodate soft-ABMC in this interface.

This is not so obvious to me. It looks to me as though the user is forced to interpret the content of resctrl files differently based on soft-ABMC vs ABMC making the interface inconsistent and user thus needing to know details of implementations. This is what the previous discussion I linked to aimed to address. It sounds to me as though you believe that this is no longer an issue. Could you please show examples of what a user can expect from the interfaces and how a user will interact with the interfaces on both a non-ABMC and ABMC system?

At the interface level, I think mbm_L3_assignments on a non-ABMC system would only need to contain a single line:

0=s;1=s;...;31=s

But maybe for consistency we would synthesize a single, unmodifiable counter configuration to reflect that allocating an RMID in a domain results in assignment to all events and deallocating the RMID unassigns all events. We could call it "group" to say it's assigning at the group level, or perhaps just '*':

*:0=s;1=s;...;31=s

I'm not sure about allowing a '*' on ABMC hardware, because it could be interpreted as allocating a lot of counters when a large number of event configurations exist.

*:0=s;1=s;...;31=s

-Peter

Quoted 14 lines

Thank you

Reinette

Quoted 8 lines
Quoted 7 lines
Quoted 5 lines
Quoted 4 lines

[2] https://lore.kernel.org/lkml/[email protected]/
[3] https://lore.kernel.org/lkml/CALPaoCg3KpF94g2MEmfP_Ro2mQZYFA8sKVkmb+7isotKNgdY9A@mail.gmail.com/

maintainer
#103 lore raw
In reply to Peter Newman

Hi Peter and Babu,

Quoted 83 lines · Peter Newman

Hi Babu/Reinette,

Quoted 79 lines · Reinette Chatre

Hi Babu,

Quoted 75 lines · Babu

Hi Reinette,

Quoted 60 lines · Reinette Chatre

Hi Babu,

Quoted 47 lines · Babu

Hi Reinette,

Quoted 29 lines · Reinette Chatre

Hi Babu,

Quoted 2 lines · Babu
Quoted 1 line · Peter Newman

On Fri, May 16, 2025 at 12:52 AM Babu Moger <[email protected]> wrote:

...

Quoted 21 lines
Quoted 11 lines
Quoted 2 lines

/sys/fs/resctrl/info/L3_MON/num_mbm_cntrs: Reports the number of monitoring counters available for assignment.

Earlier I discussed with Reinette[1] what num_mbm_cntrs should represent in a "soft-ABMC" implementation where assignment is implemented by assigning an RMID, which would result in all events being assigned at once.

My main concern is how many "counters" you can assign by assigning RMIDs. I recall Reinette proposed reporting the number of groups which can be assigned separately from counters which can be assigned.

More context may be needed here. Currently, num_mbm_cntrs indicates the number of counters available per domain, which is 32.

At the moment, we can assign 2 counters to each group, meaning each RMID can be associated with 2 hardware counters. In theory, it's possible to assign all 32 hardware counters to a group—allowing one RMID to be linked with up to 32 counters. However, we currently lack the interface to support that level of assignment.

Quoted 8 lines
Quoted 2 lines

For now, the plan is to support basic assignment and expand functionality later once we have the necessary data structure and requirements.

Looks like some requirements did not make it into this implementation. Do you recall the discussion that resulted in you writing [2]? Looks like there is a question to Peter in there on how to determine how many "counters" are available in soft-ABMC. I interpreted [3] at that time to mean that this information would be available in a future AMD publication.

We already have a method to determine the number of counters in soft-ABMC mode, which Peter has addressed [4].

[4]
https://lore.kernel.org/lkml/[email protected]/

Quoted 6 lines

This appears to be more of a workaround, and I doubt it will be included in any official AMD documentation. Additionally, the long-term direction is moving towards ABMC.

I don’t believe this workaround needs to be part of the current series. It can be added later when soft-ABMC is implemented.

Agreed. What about the plans described in [2]? (Thanks to Peter for catching this!).

Quoted 9 lines

It is important to keep track of requirements while working on a feature to ensure that the implementation supports the planned use cases. Re-reading that thread it is not clear to me how soft-ABMC's per-group assignment would look. Could you please share how you see it progress from this implementation? This includes the single event vs. multiple event assignment. I would like to highlight that this is not a request for this to be supported in this implementation but there needs to be a plan for how this can be supported on top of interfaces established by this work.

Here’s my current understanding of soft-ABMC. Peter may have a more in-depth perspective on this.

Quoted 8 lines
Quoted 2 lines
Quoted 2 lines

Soft-ABMC:
a. num_mbm_cntrs: This is a software-defined limit based on the number of active RMIDs that can be supported. The value can be obtained using the code referenced in [4].

I would call it a hardware-defined limit that can be probed by software.

The main question is whether this file returns the exact number of RMIDs hardware can track or double that number (mbm_total_bytes + mbm_local_bytes) so that the value is always measured in events.

tl;dr: I continue [3] to find it most intuitive for num_mbm_cntrs to be the exact number of "active" RMIDs that the system can support *and* changing the name of the modes to help user interpret num_mbm_cntrs: "mbm_cntr_event_assign" for ABMC, "mbm_cntr_group_assign" for soft-ABMC.

details
-------

We are now back to the previous discussion about what user can expect from the interface. Let me try and re-cap that discussion so that we can all hopefully get back on the same page. Please add corrections/updates where needed.

soft-ABMC
---------
soft-ABMC manages "active" (term TBD) RMID assignment to monitor groups. When an "active" RMID is assigned to a monitor group then *all* MBM events (not LLC occupancy) in that monitor group are counted. "Active" RMID assignment can be done per domain.

Requirement: resctrl should accurately reflect which events are counted. That is, we do not want resctrl to pretend to allow user to assign an "active" RMID to only one event in a monitor group while all events are actually counted.

Caveat: To support rapid re-assignment of RMIDs to monitor groups, llc_occupancy event is disabled when soft-ABMC is enabled.

ABMC
----
ABMC manages (hardware) counter assignment to monitor group (RMID), event pairs. When a hardware counter is assigned to an RMID, event pair then only that RMID, event is counted. Hardware counter assignment can be done per domain.

shared assignment
-----------------
A shared assignment applies to both soft-ABMC and ABMC. A user can designate a "counter" (could be hardware counter or "active" RMID) as shared and that means the counter within that domain is shared between different monitor groups and actual assignment is scheduled by resctrl.

user interface
--------------

Next, consider the interface while keeping above definitions and requirements in mind.

This series introduces (using implementation, not cover-letter):

/sys/fs/resctrl/info/L3_MON/num_mbm_cntrs
"num_mbm_cntrs":
The maximum number of monitoring counters (total of available and assigned counters) in each domain when the system supports mbm_cntr_assign mode.

/sys/fs/resctrl/mbm_L3_assignments
"mbm_L3_assignments":
This interface file is created when the mbm_cntr_assign mode is supported and shows the assignment status for each group.

Consider "mbm_L3_assignments" first. The interface is documented for ABMC support where it is possible to manage individual event assignment within monitor group.

For ABMC it is possible to assign just one event at a time and doing so consumes one counter in that domain:

a) Starting state on system with 32 counters per domain, two events in default resource group consumes two counters in that domain:

# cat /sys/fs/resctrl/info/L3_MON/num_mbm_cntrs
0=30;1=32
# cat /sys/fs/resctrl/mbm_L3_assignments
mbm_total_bytes:0=e;1=_
mbm_local_bytes:0=e;1=_

b) Assign counter to mbm_local_bytes in domain 1:

# echo "mbm_local_bytes:1=e" > /sys/fs/resctrl/mbm_L3_assignments
# cat /sys/fs/resctrl/mbm_L3_assignments
mbm_total_bytes:0=e;1=_
mbm_local_bytes:0=e;1=e
# cat /sys/fs/resctrl/info/L3_MON/num_mbm_cntrs
0=30;1=31

The question is how this should look on soft-ABMC system. Let's say hypothetically that on a soft-ABMC system it is possible to have 32 "active" RMIDs.

a) Starting state on system with 32 "active RMIDs" per domain, two events in default resource group consumes one RMID in that domain:

# cat /sys/fs/resctrl/mbm_L3_assignments
mbm_total_bytes:0=e;1=_
mbm_local_bytes:0=e;1=_

What should num_mbm_cntrs display?

Option A (counters are RMIDs):

# cat /sys/fs/resctrl/info/L3_MON/num_mbm_cntrs
0=31;1=32

Option B (pretend RMIDs are events):

# cat /sys/fs/resctrl/info/L3_MON/num_mbm_cntrs
0=62;1=64

b) Assign counter to mbm_local_bytes in domain 1:

# echo "mbm_local_bytes:1=e" > /sys/fs/resctrl/mbm_L3_assignments
# cat /sys/fs/resctrl/mbm_L3_assignments
mbm_total_bytes:0=e;1=e
mbm_local_bytes:0=e;1=e

Note that even though user requested only mbm_local_bytes to be assigned, it actually results in both mbm_total_bytes and mbm_local_bytes to be assigned. This ensures accurate state representation to user space but this also creates an inconsistent user interface between soft-ABMC and ABMC since user space intends to use the same interface but "sometimes" assigning one event results in assign of one event while "sometimes" it results in assign of multiple events.

wrt "num_mbm_cntrs"

Option A (counters are RMIDs):

# cat /sys/fs/resctrl/info/L3_MON/num_mbm_cntrs
0=31;1=31

Option B (pretend RMIDs are events):

# cat /sys/fs/resctrl/info/L3_MON/num_mbm_cntrs
0=62;1=62 

Neither option seems ideal to me since the interface cannot be consistent between ABMC and soft-ABMC.
As I mentioned in [2] it is not possible to hide ABMC and soft-ABMC behind the same interface. When user space wants to monitor a particular monitor group then it should be clear how that can be accomplished. Not knowing if an assignment/unassignment to/from an event would impact one or all events and whether it will consume one or multiple counters does not sound like a good interface to me.

As I understand current interface, user is required to know how ABMC and soft-ABMC is implemented to be able to configure the system. For example, if user has file like:

	# cat /sys/fs/resctrl/mbm_L3_assignments
	mbm_total_bytes:0=e;1=e
	mbm_local_bytes:0=e;1=e
user must know underlying implementation to be able to manage monitoring of
events and assigning counters otherwise it will be a surprise to lose monitoring
of all events when unassigning one event.

This is why I proposed in [3] that the name of the mode reflects how user can interact with the system. Instead of one "mbm_cntr_assign" mode there can be "mbm_cntr_event_assign" that is used for ABMC and "mbm_cntr_group_assign" that is used for soft-ABMC. The mode should make it clear what the system is capable of wrt counter assignments.

Considering this the interface should be clear:
num_mbm_cntrs: reflects the number of counters in each domain that can be assigned. In "mbm_cntr_event_assign" this will be the number of counters that can be assigned to each event within a monitoring group, in "mbm_cntr_group_assign" this will be the number of counters that can be assigned to entire monitoring groups impacting all MBM events.

mbm_L3_assignments: manages the counter assignment in each group. When user knows the mode is "mbm_cntr_event_assign"/"mbm_cntr_group_assign" then it should be clear to user space how the interface behaves wrt assignment, no surprises of multiple events impacted when assigning/unassigning single event.

For soft-ABMC I thus find it most intuitive for num_mbm_cntrs to be the exact number of "active" RMIDs that the system can support *and* changing the name of the modes to help user interpret num_mbm_cntrs.

Quoted 5 lines

There's also the mongroup-RMID overcommit use case I described above[1]. On Intel we can safely assume that there are counters to back all RMIDs, so num_mbm_cntrs would be calculated directly from num_rmids.

This is about the:
There's now more interest in Google for allowing explicit control of where RMIDs are assigned on Intel platforms. Even though the number of RMIDs implemented by hardware tends to be roughly the number of containers they want to support, they often still need to create containers when all RMIDs have already been allocated, which is not currently allowed. Once the container has been created and starts running, it's no longer possible to move its threads into a monitoring group whenever RMIDs should become available again, so it's important for resctrl to maintain an accurate task list for a container even when RMIDs are not available.

I see a monitor group as a collection of tasks that need to be monitored together. The "task list" is the group of tasks that share a monitoring ID that is required to be a valid ID since when any of the tasks are scheduled that ID is written to the hardware. I intentionally tried to not use RMID since I believe this is required for all archs.
I thus do not understand how a task can start running when it does not have a valid monitoring ID. The idea of "deferred assignment" is not clear to me, there can never be "unmonitored tasks", no? I think I am missing something here.

Quoted 16 lines

I realized this use case is more difficult to implement on MPAM, because a PARTID is effectively a CLOSID+RMID, so deferring assigning a unique PARTID to a group also results in it being in a different allocation group. It will work if the unmonitored groups could find a way to share PARTIDs, but this has consequences on allocation - but hopefully no worse than sharing CLOSIDs on x86.

There's a lot of interest in monitoring ID overcommit in Google, so I think it's worth it for me to investigate the additional structural changes needed in resctrl (i.e., breaking the FS-level association between mongroups and HW monitoring IDs). Such a framework could be a better fit for soft-ABMC. For example, if overcommit is allowed, we would just report the number of simultaneous RMIDs we were able to probe as num_rmids. I would want the same shared assignment scheduler to be able to work with RMIDs and counters, though.

Quoted 4 lines

Thanks,
-Peter

[1] https://lore.kernel.org/lkml/CALPaoChSzzU5mzMZsdT6CeyEn0WD1qdT9fKCoNW_ty4tojtrkw@mail.gmail.com/

Reinette

[2] https://lore.kernel.org/lkml/[email protected]/
[3] https://lore.kernel.org/lkml/[email protected]/

maintainer
#117 lore raw
In reply to Peter Newman

Hi Peter,

Quoted 3 lines · Peter Newman
Quoted 1 line · Reinette Chatre

On 5/20/25 4:25 PM, Moger, Babu wrote:

...

Quoted 30 lines
Quoted 21 lines
Quoted 21 lines

Here’s my current understanding of soft-ABMC. Peter may have a more in-depth perspective on this.

Soft-ABMC:
a. num_mbm_cntrs: This is a software-defined limit based on the number of active RMIDs that can be supported. The value can be obtained using the code referenced in [4].

b. Assignments: No hardware configuration is required. We simply need to ensure that no more than num_mbm_cntrs RMIDs are active at any given time.

c. Configuration: Controlled via /info/L3_MON/mbm_total_bytes_config and mbm_local_bytes_config.

d. Events: Only two events can be assigned(local and total).

ABMC:
a. num_mbm_cntrs: This is defined by the hardware.
b. Assignments: Requires special MSR writes to assign counters.
c. Configuration: Comes from /info/L3_MON/counter_configs/.
d. Events: More than two events can be assigned to a group (currently up to 2).

Commonalities:
a. Assignments can be either exclusive or shared in both these modes.

Quoted 8 lines
Quoted 1 line

Given these, I believe we can easily accommodate soft-ABMC in this interface.

This is not so obvious to me. It looks to me as though the user is forced to interpret the content of resctrl files differently based on soft-ABMC vs ABMC making the interface inconsistent and user thus needing to know details of implementations. This is what the previous discussion I linked to aimed to address. It sounds to me as though you believe that this is no longer an issue. Could you please show examples of what a user can expect from the interfaces and how a user will interact with the interfaces on both a non-ABMC and ABMC system?

Quoted 4 lines

At the interface level, I think mbm_L3_assignments on a non-ABMC system would only need to contain a single line:

0=s;1=s;...;31=s

It should be obvious to user space how to interpret the fields. When there is thus a single "mbm_cntr_assign" mode used for ABMC and soft-ABMC a single line like this would be difficult to parse since that would imply/require that user space knows whether it is running on ABMC or soft-ABMC system, which we should avoid.

If there are different modes, for example "mbm_cntr_event_assign" and "mbm_cntr_group_assign" then this could be used by user space to distinguish how to interact with mbm_L3_assignments making something like this possible.

Quoted 7 lines

But maybe for consistency we would synthesize a single, unmodifiable counter configuration to reflect that allocating an RMID in a domain results in assignment to all events and deallocating the RMID unassigns all events. We could call it "group" to say it's assigning at the group level, or perhaps just '*':

Quoted 8 lines

*:0=s;1=s;...;31=s

I'm not sure about allowing a '*' on ABMC hardware, because it could be interpreted as allocating a lot of counters when a large number of event configurations exist.

*:0=s;1=s;...;31=s

Either could work also. Whether it is "group" or "*" ABMC systems could respond with "not supported". Will think about this more but would like to hear your opinion about the flexibility that distinguishing between a "mbm_cntr_event_assign" and "mbm_cntr_group_assign" mode provides.

Reinette

Quoted 17 lines

-Peter

Quoted 14 lines

Thank you

Reinette

Quoted 8 lines
Quoted 7 lines
Quoted 5 lines
Quoted 4 lines

[2] https://lore.kernel.org/lkml/[email protected]/
[3] https://lore.kernel.org/lkml/CALPaoCg3KpF94g2MEmfP_Ro2mQZYFA8sKVkmb+7isotKNgdY9A@mail.gmail.com/

maintainer
#104 lore raw
In reply to Reinette Chatre
Quoted 273 lines · Reinette Chatre

Hi Peter and Babu,

Quoted 83 lines · Peter Newman

Hi Babu/Reinette,

Quoted 79 lines · Reinette Chatre

Hi Babu,

Quoted 75 lines · Babu

Hi Reinette,

Quoted 60 lines · Reinette Chatre

Hi Babu,

Quoted 47 lines · Babu

Hi Reinette,

Quoted 29 lines · Reinette Chatre

Hi Babu,

Quoted 2 lines · Babu
Quoted 1 line · Peter Newman

On Fri, May 16, 2025 at 12:52 AM Babu Moger <[email protected]> wrote:

...

Quoted 21 lines
Quoted 11 lines
Quoted 2 lines

/sys/fs/resctrl/info/L3_MON/num_mbm_cntrs: Reports the number of monitoring counters available for assignment.

Earlier I discussed with Reinette[1] what num_mbm_cntrs should represent in a "soft-ABMC" implementation where assignment is implemented by assigning an RMID, which would result in all events being assigned at once.

My main concern is how many "counters" you can assign by assigning RMIDs. I recall Reinette proposed reporting the number of groups which can be assigned separately from counters which can be assigned.

More context may be needed here. Currently, num_mbm_cntrs indicates the number of counters available per domain, which is 32.

At the moment, we can assign 2 counters to each group, meaning each RMID can be associated with 2 hardware counters. In theory, it's possible to assign all 32 hardware counters to a group—allowing one RMID to be linked with up to 32 counters. However, we currently lack the interface to support that level of assignment.

Quoted 8 lines
Quoted 2 lines

For now, the plan is to support basic assignment and expand functionality later once we have the necessary data structure and requirements.

Looks like some requirements did not make it into this implementation. Do you recall the discussion that resulted in you writing [2]? Looks like there is a question to Peter in there on how to determine how many "counters" are available in soft-ABMC. I interpreted [3] at that time to mean that this information would be available in a future AMD publication.

We already have a method to determine the number of counters in soft-ABMC mode, which Peter has addressed [4].

[4]
https://lore.kernel.org/lkml/[email protected]/

Quoted 6 lines

This appears to be more of a workaround, and I doubt it will be included in any official AMD documentation. Additionally, the long-term direction is moving towards ABMC.

I don’t believe this workaround needs to be part of the current series. It can be added later when soft-ABMC is implemented.

Agreed. What about the plans described in [2]? (Thanks to Peter for catching this!).

Quoted 9 lines

It is important to keep track of requirements while working on a feature to ensure that the implementation supports the planned use cases. Re-reading that thread it is not clear to me how soft-ABMC's per-group assignment would look. Could you please share how you see it progress from this implementation? This includes the single event vs. multiple event assignment. I would like to highlight that this is not a request for this to be supported in this implementation but there needs to be a plan for how this can be supported on top of interfaces established by this work.

Here’s my current understanding of soft-ABMC. Peter may have a more in-depth perspective on this.

Quoted 8 lines
Quoted 2 lines
Quoted 2 lines

Soft-ABMC:
a. num_mbm_cntrs: This is a software-defined limit based on the number of active RMIDs that can be supported. The value can be obtained using the code referenced in [4].

I would call it a hardware-defined limit that can be probed by software.

The main question is whether this file returns the exact number of RMIDs hardware can track or double that number (mbm_total_bytes + mbm_local_bytes) so that the value is always measured in events.

tl;dr: I continue [3] to find it most intuitive for num_mbm_cntrs to be the exact number of "active" RMIDs that the system can support *and* changing the name of the modes to help user interpret num_mbm_cntrs: "mbm_cntr_event_assign" for ABMC, "mbm_cntr_group_assign" for soft-ABMC.

details
-------

We are now back to the previous discussion about what user can expect from the interface. Let me try and re-cap that discussion so that we can all hopefully get back on the same page. Please add corrections/updates where needed.

soft-ABMC
---------
soft-ABMC manages "active" (term TBD) RMID assignment to monitor groups. When an "active" RMID is assigned to a monitor group then *all* MBM events (not LLC occupancy) in that monitor group are counted. "Active" RMID assignment can be done per domain.

Requirement: resctrl should accurately reflect which events are counted. That is, we do not want resctrl to pretend to allow user to assign an "active" RMID to only one event in a monitor group while all events are actually counted.

Caveat: To support rapid re-assignment of RMIDs to monitor groups, llc_occupancy event is disabled when soft-ABMC is enabled.

ABMC
----
ABMC manages (hardware) counter assignment to monitor group (RMID), event pairs. When a hardware counter is assigned to an RMID, event pair then only that RMID, event is counted. Hardware counter assignment can be done per domain.

shared assignment
-----------------
A shared assignment applies to both soft-ABMC and ABMC. A user can designate a "counter" (could be hardware counter or "active" RMID) as shared and that means the counter within that domain is shared between different monitor groups and actual assignment is scheduled by resctrl.

user interface
--------------

Next, consider the interface while keeping above definitions and requirements in mind.

This series introduces (using implementation, not cover-letter):

/sys/fs/resctrl/info/L3_MON/num_mbm_cntrs
"num_mbm_cntrs":
The maximum number of monitoring counters (total of available and assigned counters) in each domain when the system supports mbm_cntr_assign mode.

/sys/fs/resctrl/mbm_L3_assignments
"mbm_L3_assignments":
This interface file is created when the mbm_cntr_assign mode is supported and shows the assignment status for each group.

Consider "mbm_L3_assignments" first. The interface is documented for ABMC support where it is possible to manage individual event assignment within monitor group.

For ABMC it is possible to assign just one event at a time and doing so consumes one counter in that domain:

a) Starting state on system with 32 counters per domain, two events in default resource group consumes two counters in that domain:

# cat /sys/fs/resctrl/info/L3_MON/num_mbm_cntrs
0=30;1=32
# cat /sys/fs/resctrl/mbm_L3_assignments
mbm_total_bytes:0=e;1=_
mbm_local_bytes:0=e;1=_

b) Assign counter to mbm_local_bytes in domain 1:

# echo "mbm_local_bytes:1=e" > /sys/fs/resctrl/mbm_L3_assignments
# cat /sys/fs/resctrl/mbm_L3_assignments
mbm_total_bytes:0=e;1=_
mbm_local_bytes:0=e;1=e
# cat /sys/fs/resctrl/info/L3_MON/num_mbm_cntrs
0=30;1=31

The question is how this should look on soft-ABMC system. Let's say hypothetically that on a soft-ABMC system it is possible to have 32 "active" RMIDs.

a) Starting state on system with 32 "active RMIDs" per domain, two events in default resource group consumes one RMID in that domain:

# cat /sys/fs/resctrl/mbm_L3_assignments
mbm_total_bytes:0=e;1=_
mbm_local_bytes:0=e;1=_

What should num_mbm_cntrs display?

Option A (counters are RMIDs):

# cat /sys/fs/resctrl/info/L3_MON/num_mbm_cntrs
0=31;1=32

Option B (pretend RMIDs are events):

# cat /sys/fs/resctrl/info/L3_MON/num_mbm_cntrs
0=62;1=64

b) Assign counter to mbm_local_bytes in domain 1:

# echo "mbm_local_bytes:1=e" > /sys/fs/resctrl/mbm_L3_assignments
# cat /sys/fs/resctrl/mbm_L3_assignments
mbm_total_bytes:0=e;1=e
mbm_local_bytes:0=e;1=e

Note that even though user requested only mbm_local_bytes to be assigned, it actually results in both mbm_total_bytes and mbm_local_bytes to be assigned. This ensures accurate state representation to user space but this also creates an inconsistent user interface between soft-ABMC and ABMC since user space intends to use the same interface but "sometimes" assigning one event results in assign of one event while "sometimes" it results in assign of multiple events.

wrt "num_mbm_cntrs"

Option A (counters are RMIDs):

# cat /sys/fs/resctrl/info/L3_MON/num_mbm_cntrs
0=31;1=31

Option B (pretend RMIDs are events):

# cat /sys/fs/resctrl/info/L3_MON/num_mbm_cntrs
0=62;1=62 

Neither option seems ideal to me since the interface cannot be consistent between ABMC and soft-ABMC.
As I mentioned in [2] it is not possible to hide ABMC and soft-ABMC behind the same interface. When user space wants to monitor a particular monitor group then it should be clear how that can be accomplished. Not knowing if an assignment/unassignment to/from an event would impact one or all events and whether it will consume one or multiple counters does not sound like a good interface to me.

As I understand current interface, user is required to know how ABMC and soft-ABMC is implemented to be able to configure the system. For example, if user has file like:

	# cat /sys/fs/resctrl/mbm_L3_assignments
	mbm_total_bytes:0=e;1=e
	mbm_local_bytes:0=e;1=e
user must know underlying implementation to be able to manage monitoring of
events and assigning counters otherwise it will be a surprise to lose monitoring
of all events when unassigning one event.

This is why I proposed in [3] that the name of the mode reflects how user can interact with the system. Instead of one "mbm_cntr_assign" mode there can be "mbm_cntr_event_assign" that is used for ABMC and "mbm_cntr_group_assign" that is used for soft-ABMC. The mode should make it clear what the system is capable of wrt counter assignments.

Considering this the interface should be clear:
num_mbm_cntrs: reflects the number of counters in each domain that can be assigned. In "mbm_cntr_event_assign" this will be the number of counters that can be assigned to each event within a monitoring group, in "mbm_cntr_group_assign" this will be the number of counters that can be assigned to entire monitoring groups impacting all MBM events.

mbm_L3_assignments: manages the counter assignment in each group. When user knows the mode is "mbm_cntr_event_assign"/"mbm_cntr_group_assign" then it should be clear to user space how the interface behaves wrt assignment, no surprises of multiple events impacted when assigning/unassigning single event.

For soft-ABMC I thus find it most intuitive for num_mbm_cntrs to be the exact number of "active" RMIDs that the system can support *and* changing the name of the modes to help user interpret num_mbm_cntrs.

Quoted 5 lines

There's also the mongroup-RMID overcommit use case I described above[1]. On Intel we can safely assume that there are counters to back all RMIDs, so num_mbm_cntrs would be calculated directly from num_rmids.

This is about the:
There's now more interest in Google for allowing explicit control of where RMIDs are assigned on Intel platforms. Even though the number of RMIDs implemented by hardware tends to be roughly the number of containers they want to support, they often still need to create containers when all RMIDs have already been allocated, which is not currently allowed. Once the container has been created and starts running, it's no longer possible to move its threads into a monitoring group whenever RMIDs should become available again, so it's important for resctrl to maintain an accurate task list for a container even when RMIDs are not available.

Quoted 8 lines

I see a monitor group as a collection of tasks that need to be monitored together. The "task list" is the group of tasks that share a monitoring ID that is required to be a valid ID since when any of the tasks are scheduled that ID is written to the hardware. I intentionally tried to not use RMID since I believe this is required for all archs.
I thus do not understand how a task can start running when it does not have a valid monitoring ID. The idea of "deferred assignment" is not clear to me, there can never be "unmonitored tasks", no? I think I am missing something here.

In the AMD/RMID implemenentation this might be achieved with something extra in the task structure to denote whether a task is in a monitored group or not. E.g. We add "task->rmid_valid" as well as "task->rmid". Tasks in an unmonitored group retain their "task->rmid" (that's what identifies them as a member of a group) but have task->rmid_valid set to false. Context switch code would be updated to load "0" into the IA32_PQR_ASSOC.RMID field for tasks without a valid RMID. So they would still be monitored, but activity would be bundled with all tasks in the default resctrl group.

Presumably something analogous could be done for ARM/MPAM.

Quoted 19 lines
Quoted 19 lines

I realized this use case is more difficult to implement on MPAM, because a PARTID is effectively a CLOSID+RMID, so deferring assigning a unique PARTID to a group also results in it being in a different allocation group. It will work if the unmonitored groups could find a way to share PARTIDs, but this has consequences on allocation - but hopefully no worse than sharing CLOSIDs on x86.

There's a lot of interest in monitoring ID overcommit in Google, so I think it's worth it for me to investigate the additional structural changes needed in resctrl (i.e., breaking the FS-level association between mongroups and HW monitoring IDs). Such a framework could be a better fit for soft-ABMC. For example, if overcommit is allowed, we would just report the number of simultaneous RMIDs we were able to probe as num_rmids. I would want the same shared assignment scheduler to be able to work with RMIDs and counters, though.

Thanks,
-Peter

Quoted 6 lines
Quoted 1 line

[1] https://lore.kernel.org/lkml/CALPaoChSzzU5mzMZsdT6CeyEn0WD1qdT9fKCoNW_ty4tojtrkw@mail.gmail.com/

Reinette

[2] https://lore.kernel.org/lkml/[email protected]/
[3] https://lore.kernel.org/lkml/[email protected]/

-Tony

maintainer
#105 lore raw

Hi Tony,

Quoted 4 lines · Tony
Quoted 3 lines · Reinette Chatre

Hi Peter and Babu,

On 5/21/25 2:18 AM, Peter Newman wrote:

..

Quoted 36 lines
Quoted 17 lines
Quoted 4 lines

There's also the mongroup-RMID overcommit use case I described above[1]. On Intel we can safely assume that there are counters to back all RMIDs, so num_mbm_cntrs would be calculated directly from num_rmids.

This is about the:
There's now more interest in Google for allowing explicit control of where RMIDs are assigned on Intel platforms. Even though the number of RMIDs implemented by hardware tends to be roughly the number of containers they want to support, they often still need to create containers when all RMIDs have already been allocated, which is not currently allowed. Once the container has been created and starts running, it's no longer possible to move its threads into a monitoring group whenever RMIDs should become available again, so it's important for resctrl to maintain an accurate task list for a container even when RMIDs are not available.

Quoted 8 lines

I see a monitor group as a collection of tasks that need to be monitored together. The "task list" is the group of tasks that share a monitoring ID that is required to be a valid ID since when any of the tasks are scheduled that ID is written to the hardware. I intentionally tried to not use RMID since I believe this is required for all archs.
I thus do not understand how a task can start running when it does not have a valid monitoring ID. The idea of "deferred assignment" is not clear to me, there can never be "unmonitored tasks", no? I think I am missing something here.

In the AMD/RMID implemenentation this might be achieved with something extra in the task structure to denote whether a task is in a monitored group or not. E.g. We add "task->rmid_valid" as well as "task->rmid". Tasks in an unmonitored group retain their "task->rmid" (that's what identifies them as a member of a group) but have task->rmid_valid set to false. Context switch code would be updated to load "0" into the IA32_PQR_ASSOC.RMID field for tasks without a valid RMID. So they would still be monitored, but activity would be bundled with all tasks in the default resctrl group.

Quoted 2 lines

Presumably something analogous could be done for ARM/MPAM.

I do not interpret this as an unmonitored task but instead a task that belongs to the default resource group. Specifically, any data accumulated by such a task is attributed to the default resource group. Having tasks in a separate group but their monitoring data accumulating in/contributed to the default resource group (that has its own set of tasks) sounds wrong to me. Such an implementation makes any monitoring data of default resource group invalid, and by extension impossible to use default resource group to manage an allocation for a group of monitor groups if user space needs insight in monitoring data across all these monitor groups. User space will need to interact with resctrl differently and individually query monitor groups instead of CTRL_MON group once.

Reinette

maintainer
#106 lore raw
Quoted 39 lines
Quoted 36 lines
Quoted 17 lines
Quoted 4 lines

There's also the mongroup-RMID overcommit use case I described above[1]. On Intel we can safely assume that there are counters to back all RMIDs, so num_mbm_cntrs would be calculated directly from num_rmids.

This is about the:
There's now more interest in Google for allowing explicit control of where RMIDs are assigned on Intel platforms. Even though the number of RMIDs implemented by hardware tends to be roughly the number of containers they want to support, they often still need to create containers when all RMIDs have already been allocated, which is not currently allowed. Once the container has been created and starts running, it's no longer possible to move its threads into a monitoring group whenever RMIDs should become available again, so it's important for resctrl to maintain an accurate task list for a container even when RMIDs are not available.

Quoted 8 lines

I see a monitor group as a collection of tasks that need to be monitored together. The "task list" is the group of tasks that share a monitoring ID that is required to be a valid ID since when any of the tasks are scheduled that ID is written to the hardware. I intentionally tried to not use RMID since I believe this is required for all archs.
I thus do not understand how a task can start running when it does not have a valid monitoring ID. The idea of "deferred assignment" is not clear to me, there can never be "unmonitored tasks", no? I think I am missing something here.

In the AMD/RMID implemenentation this might be achieved with something extra in the task structure to denote whether a task is in a monitored group or not. E.g. We add "task->rmid_valid" as well as "task->rmid". Tasks in an unmonitored group retain their "task->rmid" (that's what identifies them as a member of a group) but have task->rmid_valid set to false. Context switch code would be updated to load "0" into the IA32_PQR_ASSOC.RMID field for tasks without a valid RMID. So they would still be monitored, but activity would be bundled with all tasks in the default resctrl group.

Quoted 2 lines

Presumably something analogous could be done for ARM/MPAM.

Quoted 11 lines

I do not interpret this as an unmonitored task but instead a task that belongs to the default resource group. Specifically, any data accumulated by such a task is attributed to the default resource group. Having tasks in a separate group but their monitoring data accumulating in/contributed to the default resource group (that has its own set of tasks) sounds wrong to me. Such an implementation makes any monitoring data of default resource group invalid, and by extension impossible to use default resource group to manage an allocation for a group of monitor groups if user space needs insight in monitoring data across all these monitor groups. User space will need to interact with resctrl differently and individually query monitor groups instead of CTRL_MON group once.

Maybe assign one of the limited supply of RMIDs for these "unmonitored" tasks. Populate a resctrl group named "unmonitored" that lists all the unmonitored tasks in a (read-only) "tasks" file. And supply all the counts for these tasks in normal looking "mon_data" directory.

-Tony

#107 lore raw

Hi Tony, Reinette,

Quoted 18 lines · Tony
Quoted 18 lines
Quoted 18 lines
Quoted 18 lines
Quoted 5 lines

There's also the mongroup-RMID overcommit use case I described above[1]. On Intel we can safely assume that there are counters to back all RMIDs, so num_mbm_cntrs would be calculated directly from num_rmids.

This is about the:
There's now more interest in Google for allowing explicit control of where RMIDs are assigned on Intel platforms. Even though the number of RMIDs implemented by hardware tends to be roughly the number of containers they want to support, they often still need to create containers when all RMIDs have already been allocated, which is not currently allowed. Once the container has been created and starts running, it's no longer possible to move its threads into a monitoring group whenever RMIDs should become available again, so it's important for resctrl to maintain an accurate task list for a container even when RMIDs are not available.

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

I see a monitor group as a collection of tasks that need to be monitored together. The "task list" is the group of tasks that share a monitoring ID that is required to be a valid ID since when any of the tasks are scheduled that ID is written to the hardware. I intentionally tried to not use RMID since I believe this is required for all archs.
I thus do not understand how a task can start running when it does not have a valid monitoring ID. The idea of "deferred assignment" is not clear to me, there can never be "unmonitored tasks", no? I think I am missing something here.

You are correct. I did forget to mention something...

Quoted 26 lines
Quoted 14 lines
Quoted 11 lines

In the AMD/RMID implemenentation this might be achieved with something extra in the task structure to denote whether a task is in a monitored group or not. E.g. We add "task->rmid_valid" as well as "task->rmid". Tasks in an unmonitored group retain their "task->rmid" (that's what identifies them as a member of a group) but have task->rmid_valid set to false. Context switch code would be updated to load "0" into the IA32_PQR_ASSOC.RMID field for tasks without a valid RMID. So they would still be monitored, but activity would be bundled with all tasks in the default resctrl group.

Quoted 2 lines

Presumably something analogous could be done for ARM/MPAM.

Quoted 11 lines

I do not interpret this as an unmonitored task but instead a task that belongs to the default resource group. Specifically, any data accumulated by such a task is attributed to the default resource group. Having tasks in a separate group but their monitoring data accumulating in/contributed to the default resource group (that has its own set of tasks) sounds wrong to me. Such an implementation makes any monitoring data of default resource group invalid, and by extension impossible to use default resource group to manage an allocation for a group of monitor groups if user space needs insight in monitoring data across all these monitor groups. User space will need to interact with resctrl differently and individually query monitor groups instead of CTRL_MON group once.

Quoted 4 lines

Maybe assign one of the limited supply of RMIDs for these "unmonitored" tasks. Populate a resctrl group named "unmonitored" that lists all the unmonitored tasks in a (read-only) "tasks" file. And supply all the counts for these tasks in normal looking "mon_data" directory.

I needed to switch to an rdtgroup struct pointer rather than hardware IDs in the task structure to indicate group membership[1], otherwise it's not possible to determine which tasks are in a group when it doesn't have a unique HW ID value.

Also this is required for shared assignment so that changing a group's IDs in a domain only requires updating running tasks rather than needing to search the entire task list, which would lead to the same problem we encountered in mongroup rename[2].

-Peter

[1] https://lore.kernel.org/lkml/[email protected]/
[2] https://lore.kernel.org/lkml/CALPaoCh0SbG1+VbbgcxjubE7Cc2Pb6QqhG3NH6X=WwsNfqNjtA@mail.gmail.com/

#118 lore raw
In reply to Reinette Chatre

Hi Reinette,

Quoted 50 lines · Reinette Chatre

Hi Peter,

Quoted 3 lines · Peter Newman
Quoted 1 line · Reinette Chatre

On 5/20/25 4:25 PM, Moger, Babu wrote:

...

Quoted 30 lines
Quoted 21 lines
Quoted 21 lines

Here’s my current understanding of soft-ABMC. Peter may have a more in-depth perspective on this.

Soft-ABMC:
a. num_mbm_cntrs: This is a software-defined limit based on the number of active RMIDs that can be supported. The value can be obtained using the code referenced in [4].

b. Assignments: No hardware configuration is required. We simply need to ensure that no more than num_mbm_cntrs RMIDs are active at any given time.

c. Configuration: Controlled via /info/L3_MON/mbm_total_bytes_config and mbm_local_bytes_config.

d. Events: Only two events can be assigned(local and total).

ABMC:
a. num_mbm_cntrs: This is defined by the hardware.
b. Assignments: Requires special MSR writes to assign counters.
c. Configuration: Comes from /info/L3_MON/counter_configs/.
d. Events: More than two events can be assigned to a group (currently up to 2).

Commonalities:
a. Assignments can be either exclusive or shared in both these modes.

Quoted 8 lines
Quoted 1 line

Given these, I believe we can easily accommodate soft-ABMC in this interface.

This is not so obvious to me. It looks to me as though the user is forced to interpret the content of resctrl files differently based on soft-ABMC vs ABMC making the interface inconsistent and user thus needing to know details of implementations. This is what the previous discussion I linked to aimed to address. It sounds to me as though you believe that this is no longer an issue. Could you please show examples of what a user can expect from the interfaces and how a user will interact with the interfaces on both a non-ABMC and ABMC system?

Quoted 4 lines

At the interface level, I think mbm_L3_assignments on a non-ABMC system would only need to contain a single line:

0=s;1=s;...;31=s

It should be obvious to user space how to interpret the fields. When there is thus a single "mbm_cntr_assign" mode used for ABMC and soft-ABMC a single line like this would be difficult to parse since that would imply/require that user space knows whether it is running on ABMC or soft-ABMC system, which we should avoid.

Quoted 3 lines

If there are different modes, for example "mbm_cntr_event_assign" and "mbm_cntr_group_assign" then this could be used by user space to distinguish how to interact with mbm_L3_assignments making something like this possible.

I meant to say I was proposing the format of this file when in the group assignment mode. I didn't mean to imply that a separate mode wasn't needed.

Quoted 14 lines
Quoted 14 lines

But maybe for consistency we would synthesize a single, unmodifiable counter configuration to reflect that allocating an RMID in a domain results in assignment to all events and deallocating the RMID unassigns all events. We could call it "group" to say it's assigning at the group level, or perhaps just '*':

*:0=s;1=s;...;31=s

I'm not sure about allowing a '*' on ABMC hardware, because it could be interpreted as allocating a lot of counters when a large number of event configurations exist.

Quoted 7 lines
Quoted 2 lines

*:0=s;1=s;...;31=s

Either could work also. Whether it is "group" or "*" ABMC systems could respond with "not supported". Will think about this more but would like to hear your opinion about the flexibility that distinguishing between a "mbm_cntr_event_assign" and "mbm_cntr_group_assign" mode provides.

I agree it's clearer when they are separate modes. Between "*" and "group", I prefer "group" because it seems the least ambiguous.

I just want to make sure we'd never want both modes at the same time, such as an implementation with both a small number of monitoring IDs and a small number of MBM counters. I support one MPAM implementation that has a small number of PARTIDs and only one MBWU counter per domain. Fingers crossed that the number of PARTIDs it supports isn't small compared to the number of jobs we would run on it. Otherwise maybe it will work out to just pick the more limited of the two (monitor IDs or counters) and make allocation of one drive the other.

(In case you read this before my earlier reply[1], see the note about rdtgroup pointers in the task_struct, as this is a prerequisite for overcommitting HW monitor IDs.)

Thanks,
-Peter

[1] https://lore.kernel.org/lkml/CALPaoCjh_NXQLtNBqei=7a6Jsr17fEnPO+kqMaNq4xNu2UPDJA@mail.gmail.com/

Quoted 21 lines

Reinette

Quoted 17 lines

-Peter

Quoted 14 lines

Thank you

Reinette

Quoted 8 lines
Quoted 7 lines
Quoted 5 lines
Quoted 4 lines

[2] https://lore.kernel.org/lkml/[email protected]/
[3] https://lore.kernel.org/lkml/CALPaoCg3KpF94g2MEmfP_Ro2mQZYFA8sKVkmb+7isotKNgdY9A@mail.gmail.com/

author reviewer
#110 lore raw
In reply to Reinette Chatre

Hi Reinette,

Quoted 127 lines · Reinette Chatre

Hi Peter and Babu,

Quoted 83 lines · Peter Newman

Hi Babu/Reinette,

Quoted 79 lines · Reinette Chatre

Hi Babu,

Quoted 75 lines · Babu

Hi Reinette,

Quoted 60 lines · Reinette Chatre

Hi Babu,

Quoted 47 lines · Babu

Hi Reinette,

Quoted 29 lines · Reinette Chatre

Hi Babu,

Quoted 2 lines · Babu
Quoted 1 line · Peter Newman

On Fri, May 16, 2025 at 12:52 AM Babu Moger <[email protected]> wrote:

...

Quoted 21 lines
Quoted 11 lines
Quoted 2 lines

/sys/fs/resctrl/info/L3_MON/num_mbm_cntrs: Reports the number of monitoring counters available for assignment.

Earlier I discussed with Reinette[1] what num_mbm_cntrs should represent in a "soft-ABMC" implementation where assignment is implemented by assigning an RMID, which would result in all events being assigned at once.

My main concern is how many "counters" you can assign by assigning RMIDs. I recall Reinette proposed reporting the number of groups which can be assigned separately from counters which can be assigned.

More context may be needed here. Currently, num_mbm_cntrs indicates the number of counters available per domain, which is 32.

At the moment, we can assign 2 counters to each group, meaning each RMID can be associated with 2 hardware counters. In theory, it's possible to assign all 32 hardware counters to a group—allowing one RMID to be linked with up to 32 counters. However, we currently lack the interface to support that level of assignment.

Quoted 8 lines
Quoted 2 lines

For now, the plan is to support basic assignment and expand functionality later once we have the necessary data structure and requirements.

Looks like some requirements did not make it into this implementation. Do you recall the discussion that resulted in you writing [2]? Looks like there is a question to Peter in there on how to determine how many "counters" are available in soft-ABMC. I interpreted [3] at that time to mean that this information would be available in a future AMD publication.

We already have a method to determine the number of counters in soft-ABMC mode, which Peter has addressed [4].

[4]
https://lore.kernel.org/lkml/[email protected]/

Quoted 6 lines

This appears to be more of a workaround, and I doubt it will be included in any official AMD documentation. Additionally, the long-term direction is moving towards ABMC.

I don’t believe this workaround needs to be part of the current series. It can be added later when soft-ABMC is implemented.

Agreed. What about the plans described in [2]? (Thanks to Peter for catching this!).

Quoted 9 lines

It is important to keep track of requirements while working on a feature to ensure that the implementation supports the planned use cases. Re-reading that thread it is not clear to me how soft-ABMC's per-group assignment would look. Could you please share how you see it progress from this implementation? This includes the single event vs. multiple event assignment. I would like to highlight that this is not a request for this to be supported in this implementation but there needs to be a plan for how this can be supported on top of interfaces established by this work.

Here’s my current understanding of soft-ABMC. Peter may have a more in-depth perspective on this.

Quoted 8 lines
Quoted 2 lines
Quoted 2 lines

Soft-ABMC:
a. num_mbm_cntrs: This is a software-defined limit based on the number of active RMIDs that can be supported. The value can be obtained using the code referenced in [4].

I would call it a hardware-defined limit that can be probed by software.

The main question is whether this file returns the exact number of RMIDs hardware can track or double that number (mbm_total_bytes + mbm_local_bytes) so that the value is always measured in events.

tl;dr: I continue [3] to find it most intuitive for num_mbm_cntrs to be the exact number of "active" RMIDs that the system can support *and* changing the name of the modes to help user interpret num_mbm_cntrs: "mbm_cntr_event_assign" for ABMC, "mbm_cntr_group_assign" for soft-ABMC.

details
-------

We are now back to the previous discussion about what user can expect from the interface. Let me try and re-cap that discussion so that we can all hopefully get back on the same page. Please add corrections/updates where needed.

soft-ABMC
---------
soft-ABMC manages "active" (term TBD) RMID assignment to monitor groups. When an "active" RMID is assigned to a monitor group then *all* MBM events (not LLC occupancy) in that monitor group are counted. "Active" RMID assignment can be done per domain.

Requirement: resctrl should accurately reflect which events are counted. That is, we do not want resctrl to pretend to allow user to assign an "active" RMID to only one event in a monitor group while all events are actually counted.

Caveat: To support rapid re-assignment of RMIDs to monitor groups, llc_occupancy event is disabled when soft-ABMC is enabled.

ABMC
----
ABMC manages (hardware) counter assignment to monitor group (RMID), event pairs. When a hardware counter is assigned to an RMID, event pair then only that RMID, event is counted. Hardware counter assignment can be done per domain.

Quoted 6 lines

shared assignment
-----------------
A shared assignment applies to both soft-ABMC and ABMC. A user can designate a "counter" (could be hardware counter or "active" RMID) as shared and that means the counter within that domain is shared between different monitor groups and actual assignment is scheduled by resctrl.

Good summary: Thanks.

Quoted 102 lines

user interface
--------------

Next, consider the interface while keeping above definitions and requirements in mind.

This series introduces (using implementation, not cover-letter):

/sys/fs/resctrl/info/L3_MON/num_mbm_cntrs
"num_mbm_cntrs":
The maximum number of monitoring counters (total of available and assigned counters) in each domain when the system supports mbm_cntr_assign mode.

/sys/fs/resctrl/mbm_L3_assignments
"mbm_L3_assignments":
This interface file is created when the mbm_cntr_assign mode is supported and shows the assignment status for each group.

Consider "mbm_L3_assignments" first. The interface is documented for ABMC support where it is possible to manage individual event assignment within monitor group.

For ABMC it is possible to assign just one event at a time and doing so consumes one counter in that domain:

a) Starting state on system with 32 counters per domain, two events in default resource group consumes two counters in that domain:

# cat /sys/fs/resctrl/info/L3_MON/num_mbm_cntrs
0=30;1=32
# cat /sys/fs/resctrl/mbm_L3_assignments
mbm_total_bytes:0=e;1=_
mbm_local_bytes:0=e;1=_

b) Assign counter to mbm_local_bytes in domain 1:

# echo "mbm_local_bytes:1=e" > /sys/fs/resctrl/mbm_L3_assignments
# cat /sys/fs/resctrl/mbm_L3_assignments
mbm_total_bytes:0=e;1=_
mbm_local_bytes:0=e;1=e
# cat /sys/fs/resctrl/info/L3_MON/num_mbm_cntrs
0=30;1=31

The question is how this should look on soft-ABMC system. Let's say hypothetically that on a soft-ABMC system it is possible to have 32 "active" RMIDs.

a) Starting state on system with 32 "active RMIDs" per domain, two events in default resource group consumes one RMID in that domain:

# cat /sys/fs/resctrl/mbm_L3_assignments
mbm_total_bytes:0=e;1=_
mbm_local_bytes:0=e;1=_

What should num_mbm_cntrs display?

Option A (counters are RMIDs):

# cat /sys/fs/resctrl/info/L3_MON/num_mbm_cntrs
0=31;1=32

Option B (pretend RMIDs are events):

# cat /sys/fs/resctrl/info/L3_MON/num_mbm_cntrs
0=62;1=64

b) Assign counter to mbm_local_bytes in domain 1:

# echo "mbm_local_bytes:1=e" > /sys/fs/resctrl/mbm_L3_assignments
# cat /sys/fs/resctrl/mbm_L3_assignments
mbm_total_bytes:0=e;1=e
mbm_local_bytes:0=e;1=e

Note that even though user requested only mbm_local_bytes to be assigned, it actually results in both mbm_total_bytes and mbm_local_bytes to be assigned. This ensures accurate state representation to user space but this also creates an inconsistent user interface between soft-ABMC and ABMC since user space intends to use the same interface but "sometimes" assigning one event results in assign of one event while "sometimes" it results in assign of multiple events.

wrt "num_mbm_cntrs"

Option A (counters are RMIDs):

# cat /sys/fs/resctrl/info/L3_MON/num_mbm_cntrs
0=31;1=31

Option B (pretend RMIDs are events):

# cat /sys/fs/resctrl/info/L3_MON/num_mbm_cntrs
0=62;1=62 

Neither option seems ideal to me since the interface cannot be consistent between ABMC and soft-ABMC.
As I mentioned in [2] it is not possible to hide ABMC and soft-ABMC behind the same interface. When user space wants to monitor a particular monitor group then it should be clear how that can be accomplished. Not knowing if an assignment/unassignment to/from an event would impact one or all events and whether it will consume one or multiple counters does not sound like a good interface to me.

As I understand current interface, user is required to know how ABMC and soft-ABMC is implemented to be able to configure the system. For example, if user has file like:

	# cat /sys/fs/resctrl/mbm_L3_assignments
	mbm_total_bytes:0=e;1=e
	mbm_local_bytes:0=e;1=e
user must know underlying implementation to be able to manage monitoring of
events and assigning counters otherwise it will be a surprise to lose monitoring
of all events when unassigning one event.
Quoted 4 lines

This is why I proposed in [3] that the name of the mode reflects how user can interact with the system. Instead of one "mbm_cntr_assign" mode there can be "mbm_cntr_event_assign" that is used for ABMC and "mbm_cntr_group_assign" that is used for soft-ABMC. The mode should make it clear what the system is capable of wrt counter assignments.

Yes, that makes sense. Perhaps we can also simplify it further:

# cat /sys/fs/resctrl/info/L3_MON/mbm_assign_mode:
[mbm_cntr_evt_assign] <- for ABMC
 mbm_cntr_grp_assign  <- for soft-ABMC
Quoted 7 lines

Considering this the interface should be clear:
num_mbm_cntrs: reflects the number of counters in each domain that can be assigned. In "mbm_cntr_event_assign" this will be the number of counters that can be assigned to each event within a monitoring group, in "mbm_cntr_group_assign" this will be the number of counters that can be assigned to entire monitoring groups impacting all MBM events.

Quoted 8 lines

mbm_L3_assignments: manages the counter assignment in each group. When user knows the mode is "mbm_cntr_event_assign"/"mbm_cntr_group_assign" then it should be clear to user space how the interface behaves wrt assignment, no surprises of multiple events impacted when assigning/unassigning single event.

For soft-ABMC I thus find it most intuitive for num_mbm_cntrs to be the exact number of "active" RMIDs that the system can support *and* changing the name of the modes to help user interpret num_mbm_cntrs.

Sure. The option A: fits well here then.

Option A (counters are RMIDs):

 # cat /sys/fs/resctrl/info/L3_MON/num_mbm_cntrs
 0=31;1=31
Quoted 54 lines
Quoted 6 lines

There's also the mongroup-RMID overcommit use case I described above[1]. On Intel we can safely assume that there are counters to back all RMIDs, so num_mbm_cntrs would be calculated directly from num_rmids.

This is about the:
There's now more interest in Google for allowing explicit control of where RMIDs are assigned on Intel platforms. Even though the number of RMIDs implemented by hardware tends to be roughly the number of containers they want to support, they often still need to create containers when all RMIDs have already been allocated, which is not currently allowed. Once the container has been created and starts running, it's no longer possible to move its threads into a monitoring group whenever RMIDs should become available again, so it's important for resctrl to maintain an accurate task list for a container even when RMIDs are not available.

I see a monitor group as a collection of tasks that need to be monitored together. The "task list" is the group of tasks that share a monitoring ID that is required to be a valid ID since when any of the tasks are scheduled that ID is written to the hardware. I intentionally tried to not use RMID since I believe this is required for all archs.
I thus do not understand how a task can start running when it does not have a valid monitoring ID. The idea of "deferred assignment" is not clear to me, there can never be "unmonitored tasks", no? I think I am missing something here.

Quoted 16 lines

I realized this use case is more difficult to implement on MPAM, because a PARTID is effectively a CLOSID+RMID, so deferring assigning a unique PARTID to a group also results in it being in a different allocation group. It will work if the unmonitored groups could find a way to share PARTIDs, but this has consequences on allocation - but hopefully no worse than sharing CLOSIDs on x86.

There's a lot of interest in monitoring ID overcommit in Google, so I think it's worth it for me to investigate the additional structural changes needed in resctrl (i.e., breaking the FS-level association between mongroups and HW monitoring IDs). Such a framework could be a better fit for soft-ABMC. For example, if overcommit is allowed, we would just report the number of simultaneous RMIDs we were able to probe as num_rmids. I would want the same shared assignment scheduler to be able to work with RMIDs and counters, though.

Quoted 4 lines

Thanks,
-Peter

[1] https://lore.kernel.org/lkml/CALPaoChSzzU5mzMZsdT6CeyEn0WD1qdT9fKCoNW_ty4tojtrkw@mail.gmail.com/

Reinette

[2] https://lore.kernel.org/lkml/[email protected]/
[3] https://lore.kernel.org/lkml/[email protected]/

Signature · 2 lines
Thanks
Babu Moger
maintainer
#108 lore raw
In reply to Peter Newman

Hi Peter,

Quoted 63 lines · Peter Newman

Hi Tony, Reinette,

Quoted 18 lines · Tony
Quoted 18 lines
Quoted 18 lines
Quoted 18 lines
Quoted 5 lines

There's also the mongroup-RMID overcommit use case I described above[1]. On Intel we can safely assume that there are counters to back all RMIDs, so num_mbm_cntrs would be calculated directly from num_rmids.

This is about the:
There's now more interest in Google for allowing explicit control of where RMIDs are assigned on Intel platforms. Even though the number of RMIDs implemented by hardware tends to be roughly the number of containers they want to support, they often still need to create containers when all RMIDs have already been allocated, which is not currently allowed. Once the container has been created and starts running, it's no longer possible to move its threads into a monitoring group whenever RMIDs should become available again, so it's important for resctrl to maintain an accurate task list for a container even when RMIDs are not available.

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

I see a monitor group as a collection of tasks that need to be monitored together. The "task list" is the group of tasks that share a monitoring ID that is required to be a valid ID since when any of the tasks are scheduled that ID is written to the hardware. I intentionally tried to not use RMID since I believe this is required for all archs.
I thus do not understand how a task can start running when it does not have a valid monitoring ID. The idea of "deferred assignment" is not clear to me, there can never be "unmonitored tasks", no? I think I am missing something here.

You are correct. I did forget to mention something...

Quoted 26 lines
Quoted 14 lines
Quoted 11 lines

In the AMD/RMID implemenentation this might be achieved with something extra in the task structure to denote whether a task is in a monitored group or not. E.g. We add "task->rmid_valid" as well as "task->rmid". Tasks in an unmonitored group retain their "task->rmid" (that's what identifies them as a member of a group) but have task->rmid_valid set to false. Context switch code would be updated to load "0" into the IA32_PQR_ASSOC.RMID field for tasks without a valid RMID. So they would still be monitored, but activity would be bundled with all tasks in the default resctrl group.

Quoted 2 lines

Presumably something analogous could be done for ARM/MPAM.

Quoted 11 lines

I do not interpret this as an unmonitored task but instead a task that belongs to the default resource group. Specifically, any data accumulated by such a task is attributed to the default resource group. Having tasks in a separate group but their monitoring data accumulating in/contributed to the default resource group (that has its own set of tasks) sounds wrong to me. Such an implementation makes any monitoring data of default resource group invalid, and by extension impossible to use default resource group to manage an allocation for a group of monitor groups if user space needs insight in monitoring data across all these monitor groups. User space will need to interact with resctrl differently and individually query monitor groups instead of CTRL_MON group once.

Quoted 4 lines

Maybe assign one of the limited supply of RMIDs for these "unmonitored" tasks. Populate a resctrl group named "unmonitored" that lists all the unmonitored tasks in a (read-only) "tasks" file. And supply all the counts for these tasks in normal looking "mon_data" directory.

Quoted 4 lines

I needed to switch to an rdtgroup struct pointer rather than hardware IDs in the task structure to indicate group membership[1], otherwise it's not possible to determine which tasks are in a group when it doesn't have a unique HW ID value.

Whether the task struct contains a pointer (albeit accompanied with its own complexities) does not address the issue that I am concerned about.

Looking at [1] I expect this new feature handles "unmonitored" groups by placing them in the default monitoring group, following Tony's first [3] suggestion.

When considering [1] by itself in the context of current resctrl all tasks should be members of resource groups that have valid HW monitoring IDs allocated. Using the default resource group in this way seems like addressing edge cases where pointer is not yet valid (unclear what these scenarios may be) instead of routing many tasks to the default group. I am not sure and I'll have to study that change closer to reason accurately.

From what I understand the new proposal that builds on [1] involves creating new monitor groups that are "unmonitored" for any length of time and when backed by the implementation in [1] this would mean these groups will actually still be monitored but the data attributed to the default resource group.

As I mentioned in response [4] to Tony this fundamentally changes the behavior users can expect from the default resource group. In addition, this breaks the first of the "Resource monitoring rules" from Documentation/filesystems/resctrl.rst:

1) If a task is a member of a MON group, or non-default CTRL_MON group then RDT events for the task will be reported in that group.

How does this fit with the ABMC work? I continue to think that I am missing parts of the discussion as it seems this new feature discussion mixed in with ABMC work.

Reinette

Quoted 10 lines

Also this is required for shared assignment so that changing a group's IDs in a domain only requires updating running tasks rather than needing to search the entire task list, which would lead to the same problem we encountered in mongroup rename[2].

-Peter

[1] https://lore.kernel.org/lkml/[email protected]/
[2] https://lore.kernel.org/lkml/CALPaoCh0SbG1+VbbgcxjubE7Cc2Pb6QqhG3NH6X=WwsNfqNjtA@mail.gmail.com/

[3] https://lore.kernel.org/lkml/aC5lL_qY00vd8qp4@agluck-desk3/
[4] https://lore.kernel.org/lkml/[email protected]/

maintainer
#119 lore raw
In reply to Peter Newman

Hi Peter,

Quoted 3 lines · Peter Newman
Quoted 1 line · Reinette Chatre

On 5/21/25 7:27 AM, Peter Newman wrote:

...

Quoted 11 lines
Quoted 11 lines
Quoted 4 lines

At the interface level, I think mbm_L3_assignments on a non-ABMC system would only need to contain a single line:

0=s;1=s;...;31=s

It should be obvious to user space how to interpret the fields. When there is thus a single "mbm_cntr_assign" mode used for ABMC and soft-ABMC a single line like this would be difficult to parse since that would imply/require that user space knows whether it is running on ABMC or soft-ABMC system, which we should avoid.

Quoted 7 lines
Quoted 3 lines

If there are different modes, for example "mbm_cntr_event_assign" and "mbm_cntr_group_assign" then this could be used by user space to distinguish how to interact with mbm_L3_assignments making something like this possible.

I meant to say I was proposing the format of this file when in the group assignment mode. I didn't mean to imply that a separate mode wasn't needed.

Thanks for confirming.

Quoted 15 lines
Quoted 15 lines
Quoted 14 lines

But maybe for consistency we would synthesize a single, unmodifiable counter configuration to reflect that allocating an RMID in a domain results in assignment to all events and deallocating the RMID unassigns all events. We could call it "group" to say it's assigning at the group level, or perhaps just '*':

*:0=s;1=s;...;31=s

I'm not sure about allowing a '*' on ABMC hardware, because it could be interpreted as allocating a lot of counters when a large number of event configurations exist.

*:0=s;1=s;...;31=s

Quoted 7 lines
Quoted 4 lines

Either could work also. Whether it is "group" or "*" ABMC systems could respond with "not supported". Will think about this more but would like to hear your opinion about the flexibility that distinguishing between a "mbm_cntr_event_assign" and "mbm_cntr_group_assign" mode provides.

I agree it's clearer when they are separate modes. Between "*" and "group", I prefer "group" because it seems the least ambiguous.

Sounds good to me. resctrl will need extra guards to prevent user from creating an event named "group" but this matches what resctrl already needs to do for other parts (eg. user cannot create a monitor group named "mon_groups").

Quoted 3 lines

I just want to make sure we'd never want both modes at the same time, such as an implementation with both a small number of monitoring IDs

hmmm ... my assumption was that a system could only support one of these modes ("mbm_cntr_event_assign" or "mbm_cntr_group_assign") but it could be possible to have both possible on a system. But beyond that to have both active at the *same* time? That will take a lot of wrangling during runtime.

Quoted 6 lines

and a small number of MBM counters. I support one MPAM implementation that has a small number of PARTIDs and only one MBWU counter per domain. Fingers crossed that the number of PARTIDs it supports isn't small compared to the number of jobs we would run on it. Otherwise maybe it will work out to just pick the more limited of the two (monitor IDs or counters) and make allocation of one drive the other.

Could a scenario like this be addressed by "mbm_cntr_event_assign" mode gaining support for "shared assignment"?

Quoted 4 lines

(In case you read this before my earlier reply[1], see the note about rdtgroup pointers in the task_struct, as this is a prerequisite for overcommitting HW monitor IDs.)

Reinette

Quoted 1 line

[1] https://lore.kernel.org/lkml/CALPaoCjh_NXQLtNBqei=7a6Jsr17fEnPO+kqMaNq4xNu2UPDJA@mail.gmail.com/

...

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

[2] https://lore.kernel.org/lkml/[email protected]/
[3] https://lore.kernel.org/lkml/CALPaoCg3KpF94g2MEmfP_Ro2mQZYFA8sKVkmb+7isotKNgdY9A@mail.gmail.com/

maintainer
#111 lore raw
In reply to Moger, Babu

Hi Babu,

Quoted 1 line · Babu

On 5/21/25 18:03, Reinette Chatre wrote:

...

Quoted 10 lines
Quoted 4 lines

This is why I proposed in [3] that the name of the mode reflects how user can interact with the system. Instead of one "mbm_cntr_assign" mode there can be "mbm_cntr_event_assign" that is used for ABMC and "mbm_cntr_group_assign" that is used for soft-ABMC. The mode should make it clear what the system is capable of wrt counter assignments.

Yes, that makes sense. Perhaps we can also simplify it further:

# cat /sys/fs/resctrl/info/L3_MON/mbm_assign_mode:
[mbm_cntr_evt_assign] <- for ABMC
 mbm_cntr_grp_assign  <- for soft-ABMC

Looks good to me. Thank you.

Quoted 15 lines
Quoted 6 lines

Considering this the interface should be clear:
num_mbm_cntrs: reflects the number of counters in each domain that can be assigned. In "mbm_cntr_event_assign" this will be the number of counters that can be assigned to each event within a monitoring group, in "mbm_cntr_group_assign" this will be the number of counters that can be assigned to entire monitoring groups impacting all MBM events.

Quoted 8 lines

mbm_L3_assignments: manages the counter assignment in each group. When user knows the mode is "mbm_cntr_event_assign"/"mbm_cntr_group_assign" then it should be clear to user space how the interface behaves wrt assignment, no surprises of multiple events impacted when assigning/unassigning single event.

For soft-ABMC I thus find it most intuitive for num_mbm_cntrs to be the exact number of "active" RMIDs that the system can support *and* changing the name of the modes to help user interpret num_mbm_cntrs.

Quoted 5 lines

Sure. The option A: fits well here then.

Option A (counters are RMIDs):

 # cat /sys/fs/resctrl/info/L3_MON/num_mbm_cntrs
 0=31;1=31

Thank you for considering.

Please add the requirements from this discussion to your running list. Also please keep in mind how soft-ABMC intends to use the interfaces created by this work so that the documentation that accompanies the ABMC support in this series leaves enough "wiggle room" for soft-ABMC to be built on top.

Quoted 8 lines
Quoted 7 lines
Quoted 1 line

[1] https://lore.kernel.org/lkml/CALPaoChSzzU5mzMZsdT6CeyEn0WD1qdT9fKCoNW_ty4tojtrkw@mail.gmail.com/

Reinette

[2] https://lore.kernel.org/lkml/[email protected]/
[3] https://lore.kernel.org/lkml/[email protected]/

Reinette

maintainer
#109 lore raw
In reply to Peter Newman
Quoted 10 lines
Quoted 4 lines

Maybe assign one of the limited supply of RMIDs for these "unmonitored" tasks. Populate a resctrl group named "unmonitored" that lists all the unmonitored tasks in a (read-only) "tasks" file. And supply all the counts for these tasks in normal looking "mon_data" directory.

I needed to switch to an rdtgroup struct pointer rather than hardware IDs in the task structure to indicate group membership[1], otherwise it's not possible to determine which tasks are in a group when it doesn't have a unique HW ID value.

Quoted 4 lines

Also this is required for shared assignment so that changing a group's IDs in a domain only requires updating running tasks rather than needing to search the entire task list, which would lead to the same problem we encountered in mongroup rename[2].

Having a pointer to the rdtgroup in the task structure does make file system operations easier. But the cost appears to be more complexity (and memory references) in the context switch code.

Your patch[1] seems to do some extra work outside of the static_branch protected sections. So has a cost to context switch even if resctrl is not in use.

Chasing pointers "closid = rgrp->mon.parent->closid;" could be expensive when those miss in the cache.

Quoted 5 lines

-Peter

[1] https://lore.kernel.org/lkml/[email protected]/
[2] https://lore.kernel.org/lkml/CALPaoCh0SbG1+VbbgcxjubE7Cc2Pb6QqhG3NH6X=WwsNfqNjtA@mail.gmail.com/

-Tony

author reviewer
#112 lore raw
In reply to Reinette Chatre

Hi Reinette,

Quoted 41 lines · Reinette Chatre

Hi Babu,

Quoted 1 line · Babu

On 5/21/25 18:03, Reinette Chatre wrote:

...

Quoted 10 lines
Quoted 4 lines

This is why I proposed in [3] that the name of the mode reflects how user can interact with the system. Instead of one "mbm_cntr_assign" mode there can be "mbm_cntr_event_assign" that is used for ABMC and "mbm_cntr_group_assign" that is used for soft-ABMC. The mode should make it clear what the system is capable of wrt counter assignments.

Yes, that makes sense. Perhaps we can also simplify it further:

# cat /sys/fs/resctrl/info/L3_MON/mbm_assign_mode:
[mbm_cntr_evt_assign] <- for ABMC
 mbm_cntr_grp_assign  <- for soft-ABMC

Looks good to me. Thank you.

Quoted 15 lines
Quoted 6 lines

Considering this the interface should be clear:
num_mbm_cntrs: reflects the number of counters in each domain that can be assigned. In "mbm_cntr_event_assign" this will be the number of counters that can be assigned to each event within a monitoring group, in "mbm_cntr_group_assign" this will be the number of counters that can be assigned to entire monitoring groups impacting all MBM events.

Quoted 8 lines

mbm_L3_assignments: manages the counter assignment in each group. When user knows the mode is "mbm_cntr_event_assign"/"mbm_cntr_group_assign" then it should be clear to user space how the interface behaves wrt assignment, no surprises of multiple events impacted when assigning/unassigning single event.

For soft-ABMC I thus find it most intuitive for num_mbm_cntrs to be the exact number of "active" RMIDs that the system can support *and* changing the name of the modes to help user interpret num_mbm_cntrs.

Quoted 5 lines

Sure. The option A: fits well here then.

Option A (counters are RMIDs):

 # cat /sys/fs/resctrl/info/L3_MON/num_mbm_cntrs
 0=31;1=31
Quoted 5 lines

Thank you for considering.

Please add the requirements from this discussion to your running list. Also please keep in mind how soft-ABMC intends to use the interfaces created by this work so that the documentation that accompanies the ABMC support in this series leaves enough "wiggle room" for soft-ABMC to be built on top.

Sure. Thanks

Quoted 12 lines
Quoted 9 lines
Quoted 8 lines
Quoted 2 lines

[1] https://lore.kernel.org/lkml/CALPaoChSzzU5mzMZsdT6CeyEn0WD1qdT9fKCoNW_ty4tojtrkw@mail.gmail.com/

Reinette

[2] https://lore.kernel.org/lkml/[email protected]/
[3] https://lore.kernel.org/lkml/[email protected]/

Reinette

Signature · 2 lines
Thanks
Babu Moger
maintainer
#120 lore raw
In reply to Babu Moger

Hi Babu,

Quoted 18 lines · Babu Moger

This series adds the support for Assignable Bandwidth Monitoring Counters (ABMC). It is also called QoS RMID Pinning feature

Series is written such that it is easier to support other assignable features supported from different vendors.

The feature details are documented in the APM listed below [1].
[1] AMD64 Architecture Programmer's Manual Volume 2: System Programming Publication # 24593 Revision 3.41 section 19.3.3.3 Assignable Bandwidth Monitoring (ABMC). The documentation is available at

The patches are based on top of commit 92a09c47464d0 (tag: v6.15-rc5, tip/irq/merge) Linux 6.15-rc5 plus
https://lore.kernel.org/lkml/[email protected]/

Quoted 7 lines

It is very clear these patches will go after James's resctrl FS/ARCH restructure. Hoping to avoid one review cycle due to the merge.

# Introduction

Users can create as many monitor groups as RMIDs supported by the hardware. However, bandwidth monitoring feature on AMD system only guarantees that

"bandwidth monitoring feature on AMD system" -> " the bandwidth monitoring feature on AMD systems"? or "the bandwidth monitoring feature on an AMD system". Not sure.

Quoted 7 lines

RMIDs currently assigned to a processor will be tracked by hardware. The counters of any other RMIDs which are no longer being tracked will be reset to zero. The MBM event counters return "Unavailable" for the RMIDs that are not tracked by hardware. So, there can be only limited number of groups that can give guaranteed monitoring numbers. With ever changing configurations there is no way to definitely know which of these groups are being tracked for certain point of time. Users do not have the option

"for certain point of time" -> "during a particular/certain(?) time"?

Quoted 1 line

to monitor a group or set of groups for certain period of time without

"for certain period of time" -> "for a certain period of time"?

This series contains many duplicate snippets. When you update one, please check that all the duplicates are updated also.

Quoted 8 lines

worrying about counter being reset in between.

The ABMC feature provides an option to the user to assign a hardware counter to an RMID, event pair and monitor the bandwidth as long as it is assigned. The assigned RMID will be tracked by the hardware until the user unassigns it manually. There is no need to worry about counters being reset during this period. Additionally, the user can specify a bitmask identifying the specific bandwidth types from the given source to track with the counter.

Instead of tacking it on as an "additionally" I see this capability now as essential to this new implementation. I tried to give this series a thorough review to help finalize this work but I kept being turned around by all the descriptions and finally it dawned that all the descriptions are at their code still based on the original "event ID" based implementation with either a small append or as little change as possible to adjust to the "extended event ID" based implementation.

The previous implementation still used (and copy&pasted many times) in these descriptions as "assign a hardware counter to an RMID, event pair" can only be accurate for this new implementation if an event is re-defined ... it is no longer the original constrained "event IDs" but instead an MBM event has become a generic name that identifies the configurable "bandwidth types" (but, see note about terminology later) to be monitored. This is never done.

I assume "the given source" is the assigned RMID? If so I think it will help to understand if this is specific: "bandwidth types from the assigned RMID ..."

I find this series to use several terms for the same concept,
for example, "bandwidth types", "memory transactions", "types of L3 transactions", "bandwidth sources", etc. This work will be easier to consume if it uses consistent and specific terminology.

Quoted 9 lines

Without ABMC enabled, monitoring will work in current 'default' mode without assignment option.

# History

Earlier implementation of ABMC had dependancy on BMEC (Bandwidth Monitoring Event Configuration). Peter had concerns with that implementation because it may be not be compatible with ARM's MPAM.

Quoted 7 lines

Here are the threads discussing the concerns and new interface to address the concerns.
https://lore.kernel.org/lkml/CALPaoCg97cLVVAcacnarp+880xjsedEWGJPXhYpy4P7=ky4MZw@mail.gmail.com/
https://lore.kernel.org/lkml/CALPaoCiii0vXOF06mfV=kVLBzhfNo0SFqt4kQGwGSGVUqvr2Dg@mail.gmail.com/

Here are the finalized requirements based on the discussion:

* Remove BMEC dependency on the ABMC feature.

Even stronger, BMEC and ABMC are now "incompatible" in that resctrl will not let them be used at the same time.

Quoted 8 lines

* Eliminate global assignment listing. The interface
/sys/fs/resctrl/info/L3_MON/mbm_assign_control is no longer required.

* Create the configuration directories at /sys/fs/resctrl/info/L3_MON/counter_configs/.
The configuration file names should be free-form, allowing users to create them as needed.

* Perform assignment listing at the group level by introducing mbm_L3_assignments

"the group level" -> "the monitoring group level"

Quoted 5 lines

in each monitoring group. The listing should provide the following details:

Event Configuration: Specifies the event configuration applied. This will be crucial when "mkdir" on event configuration is added in the future, leading to the creation of mon_data/mon_l3_*/<event configuration>.

hmmm ... sounds like it has become more natural to refer to it as "event configuration", which is a good match for what the purpose is. This thus sounds like good motivation to change "counter_configs" to "event_configs".

Quoted 4 lines

Domains: Identifies the domains where the configuration is applied, supporting multi-domain setups.

Assignment Type: Indicates whether the assignment is Exclusive (e or d), Shared (s), or Unassigned (_).

Could you please add definition of what "exclusive" and "shared" means?

Quoted 11 lines

* Provide option to enable or disable auto assignment when new group is created.

This series tries to address all the requirements listed above.

# Implementation details

Create a generic interface aimed to support user space assignment of scarce counters used for monitoring. First usage of interface is by ABMC with option to expand usage to "soft-ABMC" and MPAM counters in future.

Quoted 8 lines

Feature adds following interface files:

/sys/fs/resctrl/info/L3_MON/mbm_assign_mode: Reports the list of assignable monitoring features supported. The enclosed brackets indicate which feature is enabled.

/sys/fs/resctrl/info/L3_MON/num_mbm_cntrs: Reports the number of monitoring counters available for assignment.

Please aim to use consistent and clear terms to help understand this work. It is confusing that above uses "available" in description for num_mbm_cntrs and then below there is a new interface "available_mbm_cntrs" that uses the "available" term in name but not description.

Quoted 5 lines

/sys/fs/resctrl/info/L3_MON/available_mbm_cntrs: Reports the number of monitoring counters free in each domain.

/sys/fs/resctrl/info/L3_MON/counter_configs : Directory to hold the counter configuration.

Everywhere else seems to refer to this as "event configurations". Please just stick to one, "event configuration" seems most appropriate.

Quoted 3 lines

/sys/fs/resctrl/info/L3_MON/counter_configs/mbm_total_bytes/event_filter : Default configuration for MBM total events.

I think "default" should be dropped to make it clear that this is the actual configuration that is always used, not a static "default" that may be used in "some" circumstances.

Quoted 3 lines

/sys/fs/resctrl/info/L3_MON/counter_configs/mbm_local_bytes/event_filter : Default configuration for MBM local events.

Same wrt "default"

Quoted 2 lines

/sys/fs/resctrl/mbm_L3_assignments: Interface to list or modify assignment states on each group.

"Per monitor group interface to list or modify counters assigned to the group."? (Please improve.)

Quoted 4 lines

# Examples

a. Check if ABMC support is available

Please drop the "ABMC" from all the descriptions since this is intended to be a generic interface.

Quoted 6 lines

#mount -t resctrl resctrl /sys/fs/resctrl/

# cat /sys/fs/resctrl/info/L3_MON/mbm_assign_mode [mbm_cntr_assign]
default

I believe the naming has been finalized in
https://lore.kernel.org/lkml/[email protected]/.

Quoted 3 lines

ABMC feature is detected and it is enabled.

b. Check how many ABMC counters are available.

available -> supported? This will help distinguish it from the next interface file named "available_mbm_cntrs".

Quoted 3 lines
	# cat /sys/fs/resctrl/info/L3_MON/num_mbm_cntrs 
	32

Please update to reflect what implementation does.

Quoted 2 lines

c. Check how many ABMC counters are available in each domain.

"available" -> "available for assignment"

Quoted 4 lines
	# cat /sys/fs/resctrl/info/L3_MON/available_mbm_cntrs 
	0=30;1=30
Quoted 8 lines

d. Check default counter configuration.

	# cat /sys/fs/resctrl/info/L3_MON/counter_configs/mbm_total_bytes/event_filter 
	local_reads, remote_reads, local_non_temporal_writes, remote_non_temporal_writes,
        local_reads_slow_memory, remote_reads_slow_memory, dirty_victim_writes_all
	# cat /sys/fs/resctrl/info/L3_MON/counter_configs/mbm_local_bytes/event_filter 
	local_reads, local_non_temporal_writes, local_reads_slow_memory

Does not look like this matches implementation wrt spacing?

Quoted 3 lines

e. Series adds a new interface file "mbm_L3_assignments" in each monitoring group
to list and modify any group's monitoring states.

"any group's" -> "that group's"

Quoted 9 lines

The list is displayed in the following format:

<Event configuration>:<Domain id>=<Assignment type>

Event configuration: A valid event configuration listed in the /sys/fs/resctrl/info/L3_MON/counter_configs directory.

Domain ID: A valid domain ID number.

"A valid domain ID number" -> "A valid domain ID"

Quoted 5 lines

Assignment types:

_ : No event configuration assigned

Quoted 8 lines

e : Event configuration assigned in exclusive mode

To list the default group states:
# cat /sys/fs/resctrl/mbm_L3_assignments
mbm_total_bytes:0=e;1=e
mbm_local_bytes:0=e;1=e

To unassign the configuration of mbm_total_bytes on domain 0:

This unassigns a counter, as opposed to a configuration, no? How about "To unassign the counter associated with the mbm_total_bytes event"?

Quoted 2 lines
	#echo "mbm_total_bytes:0=_" > mbm_L3_assignments
	#cat mbm_L3_assignments

(May help to follow if the examples consistently uses full path.)

Quoted 4 lines
	mbm_total_bytes:0=_;1=e
	mbm_local_bytes:0=e;1=e

	To unassign the mbm_total_bytes configuration on all domains:

same wrt unassigning a counter

Quoted 2 lines
    	$echo "mbm_total_bytes:*=_" > mbm_L3_assignments
	$cat mbm_L3_assignments
# prompt is usually used for administrator and $ for user without
administrator privileges. Switching between # and $ in these examples 
is confusing.
Quoted 4 lines
	mbm_total_bytes:0=_;1=_
	mbm_local_bytes:0=e;1=e

	To assign the mbm_total_bytes configuration on all domains in exclusive mode:

same wrt unassigning a counter

Quoted 25 lines
    	$echo "mbm_total_bytes:*=e" > mbm_L3_assignments
	$cat mbm_L3_assignments
	mbm_total_bytes:0=e;1=e
	mbm_local_bytes:0=e;1=e

g. Read the events mbm_total_bytes and mbm_local_bytes of the default group.
There is no change in reading the events with ABMC. If the event is unassigned when reading, then the read will come back as "Unassigned".

	# cat /sys/fs/resctrl/mon_data/mon_L3_00/mbm_total_bytes
	779247936
	# cat /sys/fs/resctrl/mon_data/mon_L3_00/mbm_local_bytes 
	765207488

h. Check the default event configurations.

#cat /sys/fs/resctrl/info/L3_MON/counter_configs/mbm_total_bytes/event_filter local_reads, remote_reads, local_non_temporal_writes, remote_non_temporal_writes, local_reads_slow_memory, remote_reads_slow_memory, dirty_victim_writes_all

#cat /sys/fs/resctrl/info/L3_MON/counter_configs/mbm_local_bytes/event_filter local_reads, local_non_temporal_writes, local_reads_slow_memory

i. Change the event configuration for mbm_local_bytes.

Quoted 7 lines

#echo "local_reads, local_non_temporal_writes, local_reads_slow_memory, remote_reads" > /sys/fs/resctrl/info/L3_MON/counter_configs/mbm_local_bytes/event_filter

#cat /sys/fs/resctrl/info/L3_MON/counter_configs/mbm_local_bytes/event_filter local_reads, local_non_temporal_writes, local_reads_slow_memory, remote_reads

This will update the assignments where mbm_local_bytes are configured.

"This will update all (across all domains of all monitor groups) counter assignments associated with the mbm_local_bytes event." (Please improve).

Quoted 3 lines

j. Now read the total event again. The first read may come back with "Unavailable"
status. The subsequent read of mbm_total_bytes will display only the read events.

Was this intended to be example of reading *local* bytes after modification in previous step?

Quoted 7 lines
	
	#cat /sys/fs/resctrl/mon_data/mon_L3_00/mbm_total_bytes
	Unavailable
	#cat /sys/fs/resctrl/mon_data/mon_L3_00/mbm_total_bytes
	314101

k. Users will have the option to go back to 'default' mbm_assign_mode if required.

"Users will have the option" -> "Users have the option"

Quoted 2 lines

This can be done using the following command. Note that switching the mbm_assign_mode will reset all the MBM counters of all resctrl groups.

"all the MBM counters " -> "all the MBM counters (and thus all MBM events)"?

Quoted 10 lines
	# echo "default" > /sys/fs/resctrl/info/L3_MON/mbm_assign_mode
	# cat /sys/fs/resctrl/info/L3_MON/mbm_assign_mode
	mbm_cntr_assign
	[default]

l. Unmount the resctrl

#umount /sys/fs/resctrl/
---

Reinette

maintainer
#2 lore raw

Re: [PATCH v13 01/27] x86/cpufeatures: Add support for Assignable Bandwidth Monitoring Counters (ABMC)

In reply to Babu Moger

Hi Babu,

Quoted 30 lines · Babu Moger

Users can create as many monitor groups as RMIDs supported by the hardware. However, bandwidth monitoring feature on AMD system only guarantees that RMIDs currently assigned to a processor will be tracked by hardware. The counters of any other RMIDs which are no longer being tracked will be reset to zero. The MBM event counters return "Unavailable" for the RMIDs that are not tracked by hardware. So, there can be only limited number of groups that can give guaranteed monitoring numbers. With ever changing configurations there is no way to definitely know which of these groups are being tracked for certain point of time. Users do not have the option to monitor a group or set of groups for certain period of time without worrying about RMID being reset in between.

The ABMC feature provides an option to the user to assign a hardware counter to an RMID, event pair and monitor the bandwidth as long as it is assigned. The assigned RMID will be tracked by the hardware until the user unassigns it manually. There is no need to worry about counters being reset during this period. Additionally, the user can specify a bitmask identifying the specific bandwidth types from the given source to track with the counter.

Without ABMC enabled, monitoring will work in current mode without assignment option.

The Linux resctrl subsystem provides an interface that allows monitoring of up to two memory bandwidth events per group, selected from a combination of available total and local events. When ABMC is enabled, two events will be assigned to each group by default, in line with the current interface design. Users will also have the option to configure which types of memory transactions are counted by these events.

Quoted 4 lines

Due to the limited number of available counters (32), users may quickly exhaust the available counters. If the system runs out of assignable ABMC counters, the kernel will report an error. In such cases, users will nee dto unassign one or more active counters to free up countes for new

"nee dto" -> "need to"
"countes" -> "counters"

Quoted 1 line

assignments. The interface will provide options to assign or unassign

"The interface will" -> "resctrl will"?

Quoted 3 lines

events through the group-specific interface file.

The feature can be detected via CPUID_Fn80000020_EBX_x00 bit 5.

"The feature can be detected" -> "The feature is detected"

Quoted 11 lines

Bits Description
5 ABMC (Assignable Bandwidth Monitoring Counters)

The feature details are documented in APM listed below [1].
[1] AMD64 Architecture Programmer's Manual Volume 2: System Programming Publication # 24593 Revision 3.41 section 19.3.3.3 Assignable Bandwidth Monitoring (ABMC).

---

...

Quoted 22 lines
 arch/x86/include/asm/cpufeatures.h | 1 +
 arch/x86/kernel/cpu/cpuid-deps.c   | 2 ++
 arch/x86/kernel/cpu/scattered.c    | 1 +
 3 files changed, 4 insertions(+)
arch/x86/include/asm/cpufeatures.h +1 -0
index 6c2c152d8a67..d5c14dc678df 100644
@@ -481,6 +481,7 @@
 #define X86_FEATURE_AMD_HETEROGENEOUS_CORES (21*32 + 6) /* Heterogeneous Core Topology */
 #define X86_FEATURE_AMD_WORKLOAD_CLASS	(21*32 + 7) /* Workload Classification */
 #define X86_FEATURE_PREFER_YMM		(21*32 + 8) /* Avoid ZMM registers due to downclocking */
+#define X86_FEATURE_ABMC		(21*32 + 9) /* Assignable Bandwidth Monitoring Counters */
 
 /*
  * BUG word(s)
arch/x86/kernel/cpu/cpuid-deps.c +0 -0
index a2fbea0be535..2f54831e04e5 100644

arch/x86/kernel/cpu/cpuid-deps.c static const struct cpuid_dep cpuid_deps[] = { · line 75

Quoted 5 lines
arch/x86/kernel/cpu/cpuid-deps.c +2 -0
@@ -71,6 +71,8 @@ static const struct cpuid_dep cpuid_deps[] = {
 	{ X86_FEATURE_CQM_MBM_LOCAL,		X86_FEATURE_CQM_LLC   },
 	{ X86_FEATURE_BMEC,			X86_FEATURE_CQM_MBM_TOTAL   },
 	{ X86_FEATURE_BMEC,			X86_FEATURE_CQM_MBM_LOCAL   },
+	{ X86_FEATURE_ABMC,			X86_FEATURE_CQM_MBM_TOTAL   },
+	{ X86_FEATURE_ABMC,			X86_FEATURE_CQM_MBM_LOCAL   },

Is this dependency still accurate now that the implementation switched to the "extended event ID" variant of ABMC that no longer uses the event IDs associated with X86_FEATURE_CQM_MBM_TOTAL and X86_FEATURE_CQM_MBM_LOCAL?

Quoted 8 lines
arch/x86/kernel/cpu/cpuid-deps.c +0 -0
 	{ X86_FEATURE_AVX512_BF16,		X86_FEATURE_AVX512VL  },
 	{ X86_FEATURE_AVX512_FP16,		X86_FEATURE_AVX512BW  },
 	{ X86_FEATURE_ENQCMD,			X86_FEATURE_XSAVES    },
arch/x86/kernel/cpu/scattered.c +0 -0
index 16f3ca30626a..3b72b72270f1 100644

arch/x86/kernel/cpu/scattered.c static const struct cpuid_bit cpuid_bits[] = { · line 55

Quoted 7 lines
arch/x86/kernel/cpu/scattered.c +1 -0
@@ -49,6 +49,7 @@ static const struct cpuid_bit cpuid_bits[] = {
 	{ X86_FEATURE_MBA,			CPUID_EBX,  6, 0x80000008, 0 },
 	{ X86_FEATURE_SMBA,			CPUID_EBX,  2, 0x80000020, 0 },
 	{ X86_FEATURE_BMEC,			CPUID_EBX,  3, 0x80000020, 0 },
+	{ X86_FEATURE_ABMC,			CPUID_EBX,  5, 0x80000020, 0 },
 	{ X86_FEATURE_AMD_WORKLOAD_CLASS,	CPUID_EAX, 22, 0x80000021, 0 },
 	{ X86_FEATURE_PERFMON_V2,		CPUID_EAX,  0, 0x80000022, 0 },
 	{ X86_FEATURE_AMD_LBR_V2,		CPUID_EAX,  1, 0x80000022, 0 },

Reinette

maintainer
#10 lore raw

Re: [PATCH v13 03/27] x86/resctrl: Consolidate monitoring related data from rdt_resource

In reply to Babu Moger

Hi Babu,

Quoted 5 lines · Babu Moger
include/linux/resctrl.h +0 -0
index 9ba771f2ddea..2a8fa454d3e6 100644

include/linux/resctrl.h enum resctrl_schema_fmt { · line 264

Quoted 10 lines
include/linux/resctrl.h +7 -0
@@ -255,40 +255,48 @@ enum resctrl_schema_fmt {
 	RESCTRL_SCHEMA_RANGE,
 };
 
+/**
+ * struct resctrl_mon - Monitoring related data of a resctrl resource
+ * @num_rmid:		Number of RMIDs available
+ * @mbm_cfg_mask:	Bandwidth sources that can be tracked when bandwidth
+ *			monitoring events can be configured.
+ * @evt_list:		List of monitoring events
+ */

Nit: this new comment portion can start with a clean slate by all sentences having good structure by starting with upper case and ending with period.

Reinette

maintainer
#13 lore raw

Re: [PATCH v13 04/27] x86/resctrl: Detect Assignable Bandwidth Monitoring feature details

In reply to Babu Moger

Hi Babu,

Quoted 15 lines · Babu Moger

ABMC feature details are reported via CPUID Fn8000_0020_EBX_x5. Bits Description
15:0 MAX_ABMC Maximum Supported Assignable Bandwidth
Monitoring Counter ID + 1

The feature details are documented in APM listed below [1].
[1] AMD64 Architecture Programmer's Manual Volume 2: System Programming Publication # 24593 Revision 3.41 section 19.3.3.3 Assignable Bandwidth Monitoring (ABMC).

Detect the feature and number of assignable monitoring counters supported.

---

...

Quoted 24 lines
---
 arch/x86/kernel/cpu/resctrl/monitor.c | 9 +++++++--
 include/linux/resctrl.h               | 4 ++++
 2 files changed, 11 insertions(+), 2 deletions(-)
arch/x86/kernel/cpu/resctrl/monitor.c +1 -2
index aeb2a9283069..fd2761d9f3f7 100644
@@ -345,6 +345,7 @@ int __init rdt_get_mon_l3_config(struct rdt_resource *r)
 	unsigned int mbm_offset = boot_cpu_data.x86_cache_mbm_width_offset;
 	struct rdt_hw_resource *hw_res = resctrl_to_arch_res(r);
 	unsigned int threshold;
+	u32 eax, ebx, ecx, edx;
 
 	snc_nodes_per_l3_cache = snc_get_config();
 
@@ -375,13 +376,17 @@ int __init rdt_get_mon_l3_config(struct rdt_resource *r)
 	resctrl_rmid_realloc_threshold = resctrl_arch_round_mon_val(threshold);
 
 	if (rdt_cpu_has(X86_FEATURE_BMEC)) {
-		u32 eax, ebx, ecx, edx;
-
 		/* Detect list of bandwidth sources that can be tracked */

arch/x86/kernel/cpu/resctrl/monitor.c int __init rdt_get_mon_l3_config(struct rdt_resource *r) · line 389

Quoted 10 lines
arch/x86/kernel/cpu/resctrl/monitor.c +6 -0
 		cpuid_count(0x80000020, 3, &eax, &ebx, &ecx, &edx);
 		r->mon.mbm_cfg_mask = ecx & MAX_EVT_CONFIG_BITS;
 	}
 
+	if (rdt_cpu_has(X86_FEATURE_ABMC)) {
+		r->mon.mbm_cntr_assignable = true;
+		cpuid_count(0x80000020, 5, &eax, &ebx, &ecx, &edx);
+		r->mon.num_mbm_cntrs = (ebx & GENMASK(15, 0)) + 1;
+	}
+

Shouldn't ABMC detection also include enumeration of which configurations are supported? From what I can tell, looking ahead patch #18 hardcodes definitions of all known "bandwidth types" (which term to use TBD) and then patch #20 allows *any* of these types to be configured irrespective of whether system supports it.
AMD spec mentions "The types of L3 transactions that ABMC can track are configurable and identified by CPUID Fn8000_0020_ECX_x3." It thus looks like the enumeration of r->mon.mbm_cfg_mask when BMEC is enabled is required for ABMC also and used by this implementation.

include/linux/resctrl.h enum resctrl_schema_fmt { · line 264

Quoted 13 lines
arch/x86/kernel/cpu/resctrl/monitor.c +0 -0
 	r->mon_capable = true;
 
 	return 0;
include/linux/resctrl.h +2 -0
index 2a8fa454d3e6..065fb6e38933 100644
@@ -260,11 +260,15 @@ enum resctrl_schema_fmt {
  * @num_rmid:		Number of RMIDs available
  * @mbm_cfg_mask:	Bandwidth sources that can be tracked when bandwidth
  *			monitoring events can be configured.
+ * @num_mbm_cntrs:	Number of assignable monitoring counters
+ * @mbm_cntr_assignable:Is system capable of supporting monitor assignment?

"monitor assignment" has not been used so far, was this intended to be "counter assignment"?

Reinette

maintainer
#20 lore raw

Re: [PATCH v13 06/27] x86/resctrl: Introduce the interface to display monitor mode

In reply to Babu Moger

Hi Babu,

On 5/15/25 3:51 PM, Babu Moger wrote:

No comments on changelog since I expect it to be reworked based on
https://lore.kernel.org/lkml/[email protected]/

Documentation/filesystems/resctrl.rst with the following files: · line 261

Quoted 8 lines
Documentation/filesystems/resctrl.rst +2 -0
@@ -257,6 +257,33 @@ with the following files:
 	    # cat /sys/fs/resctrl/info/L3_MON/mbm_local_bytes_config
 	    0=0x30;1=0x30;3=0x15;4=0x15
 
+"mbm_assign_mode":
+	Reports the list of monitoring modes supported. The enclosed brackets

Please try to avoid unnecessary words. For example,
"Reports the list of monitoring modes supported." -> "The supported monitoring modes."

Reinette

maintainer
#17 lore raw

Re: [PATCH v13 05/27] x86/resctrl: Add support to enable/disable AMD ABMC feature

In reply to Babu Moger

Hi Babu,

Quoted 16 lines · Babu Moger

Add the functionality to enable/disable AMD ABMC feature.

AMD ABMC feature is enabled by setting enabled bit(0) in MSR L3_QOS_EXT_CFG. When the state of ABMC is changed, the MSR needs to be updated on all the logical processors in the QOS Domain.

Hardware counters will reset when ABMC state is changed.

The ABMC feature details are documented in APM listed below [1].
[1] AMD64 Architecture Programmer's Manual Volume 2: System Programming Publication # 24593 Revision 3.41 section 19.3.3.3 Assignable Bandwidth Monitoring (ABMC).

---

...

Quoted 66 lines
---
 arch/x86/include/asm/msr-index.h       |  1 +
 arch/x86/kernel/cpu/resctrl/internal.h |  5 +++
 arch/x86/kernel/cpu/resctrl/monitor.c  | 43 ++++++++++++++++++++++++++
 include/linux/resctrl.h                |  3 ++
 4 files changed, 52 insertions(+)
arch/x86/include/asm/msr-index.h +1 -0
index e6134ef2263d..3970e0b16e47 100644
@@ -1203,6 +1203,7 @@
 /* - AMD: */
 #define MSR_IA32_MBA_BW_BASE		0xc0000200
 #define MSR_IA32_SMBA_BW_BASE		0xc0000280
+#define MSR_IA32_L3_QOS_EXT_CFG		0xc00003ff
 #define MSR_IA32_EVT_CFG_BASE		0xc0000400
 
 /* AMD-V MSRs */
arch/x86/kernel/cpu/resctrl/internal.h +5 -0
index 5e3c41b36437..fcc9d23686a1 100644
@@ -37,6 +37,9 @@ struct arch_mbm_state {
 	u64	prev_msr;
 };
 
+/* Setting bit 0 in L3_QOS_EXT_CFG enables the ABMC feature. */
+#define ABMC_ENABLE_BIT			0
+
 /**
  * struct rdt_hw_ctrl_domain - Arch private attributes of a set of CPUs that share
  *			       a resource for a control function
@@ -102,6 +105,7 @@ struct msr_param {
  * @mon_scale:		cqm counter * mon_scale = occupancy in bytes
  * @mbm_width:		Monitor width, to detect and correct for overflow.
  * @cdp_enabled:	CDP state of this resource
+ * @mbm_cntr_assign_enabled:	ABMC feature is enabled
  *
  * Members of this structure are either private to the architecture
  * e.g. mbm_width, or accessed via helpers that provide abstraction. e.g.
@@ -115,6 +119,7 @@ struct rdt_hw_resource {
 	unsigned int		mon_scale;
 	unsigned int		mbm_width;
 	bool			cdp_enabled;
+	bool			mbm_cntr_assign_enabled;
 };
 
 static inline struct rdt_hw_resource *resctrl_to_arch_res(struct rdt_resource *r)
arch/x86/kernel/cpu/resctrl/monitor.c +9 -0
index fd2761d9f3f7..ff4b2abfa044 100644
@@ -405,3 +405,46 @@ void __init intel_rdt_mbm_apply_quirk(void)
 	mbm_cf_rmidthreshold = mbm_cf_table[cf_index].rmidthreshold;
 	mbm_cf = mbm_cf_table[cf_index].cf;
 }
+
+static void resctrl_abmc_set_one_amd(void *arg)
+{
+	bool *enable = arg;
+
+	if (*enable)
+		msr_set_bit(MSR_IA32_L3_QOS_EXT_CFG, ABMC_ENABLE_BIT);
+	else
+		msr_clear_bit(MSR_IA32_L3_QOS_EXT_CFG, ABMC_ENABLE_BIT);

arch/x86/kernel/cpu/resctrl/monitor.c void __init intel_rdt_mbm_apply_quirk(void) · line 426

Quoted 10 lines
arch/x86/kernel/cpu/resctrl/monitor.c +10 -0
+}
+
+/*
+ * ABMC enable/disable requires update of L3_QOS_EXT_CFG MSR on all the CPUs
+ * associated with all monitor domains.
+ */
+static void _resctrl_abmc_enable(struct rdt_resource *r, bool enable)
+{
+	struct rdt_mon_domain *d;
+

It remains a challenge to consider these building blocks without insight into how/when they will be used. To help out, please add guardrails to help with review. For example, this could benefit from a:

	lockdep_assert_cpus_held();
Quoted 11 lines
arch/x86/kernel/cpu/resctrl/monitor.c +11 -0
+	list_for_each_entry(d, &r->mon_domains, hdr.list) {
+		on_each_cpu_mask(&d->hdr.cpu_mask,
+				 resctrl_abmc_set_one_amd, &enable, 1);
+		resctrl_arch_reset_rmid_all(r, d);
+	}
+}
+
+int resctrl_arch_mbm_cntr_assign_set(struct rdt_resource *r, bool enable)
+{
+	struct rdt_hw_resource *hw_res = resctrl_to_arch_res(r);
+

arch/x86/kernel/cpu/resctrl/monitor.c void __init intel_rdt_mbm_apply_quirk(void) · line 447

Quoted 10 lines
arch/x86/kernel/cpu/resctrl/monitor.c +10 -0
+	if (r->mon.mbm_cntr_assignable &&
+	    hw_res->mbm_cntr_assign_enabled != enable) {
+		_resctrl_abmc_enable(r, enable);
+		hw_res->mbm_cntr_assign_enabled = enable;
+	}
+
+	return 0;
+}
+
+inline bool resctrl_arch_mbm_cntr_assign_enabled(struct rdt_resource *r)

This "inline" in the .c file is unexpected. Why is this needed?

Quoted 8 lines
arch/x86/kernel/cpu/resctrl/monitor.c +3 -0
+{
+	return resctrl_to_arch_res(r)->mbm_cntr_assign_enabled;
+}
include/linux/resctrl.h +0 -0
index 065fb6e38933..bdb264875ef6 100644

include/linux/resctrl.h static inline u32 resctrl_get_config_index(u32 closid, · line 436

Quoted 9 lines
include/linux/resctrl.h +3 -0
@@ -428,6 +428,9 @@ static inline u32 resctrl_get_config_index(u32 closid,
 bool resctrl_arch_get_cdp_enabled(enum resctrl_res_level l);
 int resctrl_arch_set_cdp_enabled(enum resctrl_res_level l, bool enable);
 
+bool resctrl_arch_mbm_cntr_assign_enabled(struct rdt_resource *r);
+int resctrl_arch_mbm_cntr_assign_set(struct rdt_resource *r, bool enable);
+
 /*
  * Update the ctrl_val and apply this config right now.
  * Must be called on one of the domain's CPUs.

Reinette

maintainer
#24 lore raw

Re: [PATCH v13 08/27] x86/resctrl: Introduce mbm_cntr_cfg to track assignable counters at domain

In reply to Babu Moger

Hi Babu,

shortlog: "at domain" -> "per domain"?

Quoted 83 lines · Babu Moger

In mbm_cntr_assign mode hardware counters are assigned/unassigned to an MBM event of a monitor group. Hardware counters are assigned/unassigned at monitoring domain level.

Manage a monitoring domain's hardware counters using a per monitoring domain array of struct mbm_cntr_cfg that is indexed by the hardware counter ID. A hardware counter's configuration contains the MBM event ID and points to the monitoring group that it is assigned to, with a NULL pointer meaning that the hardware counter is available for assignment.

There is no direct way to determine which hardware counters are assigned to a particular monitoring group. Check every entry of every hardware counter configuration array in every monitoring domain to query which MBM events of a monitoring group is tracked by hardware. Such queries are acceptable because of a very small number of assignable counters (32 to 64).

---
v13: Resolved conflicts caused by the recent FS/ARCH code restructure.
The files monitor.c/rdtgroup.c have been split between FS and ARCH directories.

v12: Fixed the struct mbm_cntr_cfg code documentation.
Removed few strange charactors in changelog.
Added the counter range for better understanding. Moved the struct mbm_cntr_cfg definition to resctrl/internal.h as suggested by James.

v11: Refined the change log based on Reinette's feedback.
Fixed few style issues.

v10: Patch changed completely to handle the counters at domain level.
https://lore.kernel.org/lkml/CALPaoCj+zWq1vkHVbXYP0znJbe6Ke3PXPWjtri5AFgD9cQDCUg@mail.gmail.com/
Removed Reviewed-by tag.
Did not see the need to add cntr_id in mbm_state structure. Not used in the code.

v9: Added Reviewed-by tag. No other changes.

v8: Minor commit message changes.

v7: Added check mbm_cntr_assignable for allocating bitmap mbm_cntr_map

v6: New patch to add domain level assignment.

---
 fs/resctrl/rdtgroup.c   | 11 +++++++++++
 include/linux/resctrl.h | 16 ++++++++++++++++
 2 files changed, 27 insertions(+)
fs/resctrl/rdtgroup.c +11 -0
index 51f8f8d3ccbc..e2005fc9acd9 100644
@@ -4085,6 +4085,7 @@ static void rdtgroup_setup_default(void)
 
 static void domain_destroy_mon_state(struct rdt_mon_domain *d)
 {
+	kfree(d->cntr_cfg);
 	bitmap_free(d->rmid_busy_llc);
 	kfree(d->mbm_total);
 	kfree(d->mbm_local);
@@ -4171,6 +4172,16 @@ static int domain_setup_mon_state(struct rdt_resource *r, struct rdt_mon_domain
 			return -ENOMEM;
 		}
 	}
+	if (resctrl_is_mbm_enabled() && r->mon.mbm_cntr_assignable) {
+		tsize = sizeof(*d->cntr_cfg);
+		d->cntr_cfg = kcalloc(r->mon.num_mbm_cntrs, tsize, GFP_KERNEL);
+		if (!d->cntr_cfg) {
+			bitmap_free(d->rmid_busy_llc);
+			kfree(d->mbm_total);
+			kfree(d->mbm_local);
+			return -ENOMEM;
+		}
+	}
 
 	return 0;
 }
include/linux/resctrl.h +0 -0
index bdb264875ef6..d77981d1fcb9 100644

include/linux/resctrl.h struct rdt_ctrl_domain { · line 163

Quoted 8 lines
include/linux/resctrl.h +5 -0
@@ -156,6 +156,20 @@ struct rdt_ctrl_domain {
 	u32				*mbps_val;
 };
 
+/**
+ * struct mbm_cntr_cfg - Assignable counter configuration
+ * @evtid:		MBM event to which the counter is assigned. Only valid
+ *			if @rdtgroup is not NULL.
+ * @evt_cfg:		Event configuration value.

@evt_cfg is not introduced in changelog nor defined here. Please add a snippet here on what @evt_cfg's values represent. This is important since this is exposed as resctrl fs API to architectures so all architectures need to use same values when interacting with resctrl.

include/linux/resctrl.h struct rdt_ctrl_domain { · line 170

Quoted 7 lines
include/linux/resctrl.h +7 -0
+ * @rdtgrp:		resctrl group assigned to the counter. NULL if the
+ *			counter is free.
+ */
+struct mbm_cntr_cfg {
+	enum resctrl_event_id   evtid;
+	u32                     evt_cfg;
+	struct rdtgroup         *rdtgrp;

Please align struct member names using TABs.

include/linux/resctrl.h struct rdt_ctrl_domain { · line 184

Quoted 10 lines
include/linux/resctrl.h +3 -0
+};
+
 /**
  * struct rdt_mon_domain - group of CPUs sharing a resctrl monitor resource
  * @hdr:		common header for different domain types
@@ -167,6 +181,7 @@ struct rdt_ctrl_domain {
  * @cqm_limbo:		worker to periodically read CQM h/w counters
  * @mbm_work_cpu:	worker CPU for MBM h/w counters
  * @cqm_work_cpu:	worker CPU for CQM h/w counters
+ * @cntr_cfg:		assignable counters configuration

"array of assignable counters' configuration (indexed by counter ID)"

include/linux/resctrl.h struct rdt_mon_domain { · line 199

Quoted 11 lines
include/linux/resctrl.h +1 -0
  */
 struct rdt_mon_domain {
 	struct rdt_domain_hdr		hdr;
@@ -178,6 +193,7 @@ struct rdt_mon_domain {
 	struct delayed_work		cqm_limbo;
 	int				mbm_work_cpu;
 	int				cqm_work_cpu;
+	struct mbm_cntr_cfg		*cntr_cfg;
 };
 
 /**

Reinette

maintainer
#30 lore raw

Re: [PATCH v13 10/27] x86/resctrl: Add data structures and definitions for ABMC assignment

In reply to Babu Moger

Hi Babu,

Quoted 5 lines · Babu Moger

The ABMC feature provides an option to the user to assign a hardware counter to an RMID, event pair and monitor the bandwidth as long as the counter is assigned. The bandwidth events will be tracked by the hardware until the user changes the configuration. Each resctrl group can configure maximum two counters, one for total event and one for local event.

(please update, above describes previous design)

Quoted 6 lines

The ABMC feature implements an MSR L3_QOS_ABMC_CFG (C000_03FDh). ABMC counter assignment is done by setting the counter id, bandwidth source (RMID) and bandwidth configuration. Users will have the option to change the bandwidth configuration using resctrl interface which will be introduced later in the series.

"will be introduced later in the series" is similar to "in a subsequent patch" and should not be used in a changelog. Just describe what this patch does.

Quoted 96 lines

Attempts to read or write the MSR when ABMC is not enabled will result in a #GP(0) exception.

Introduce the data structures and definitions for MSR L3_QOS_ABMC_CFG (0xC000_03FDh):
=========================================================================
Bits Mnemonic Description Access Reset
Type Value
=========================================================================
63 CfgEn Configuration Enable R/W 0

62 CtrEn Enable/disable counting R/W 0

61:53 – Reserved MBZ 0

52:48 CtrID Counter Identifier R/W 0

47 IsCOS BwSrc field is a CLOSID R/W 0
(not an RMID)

46:44 – Reserved MBZ 0

43:32 BwSrc Bandwidth Source R/W 0
(RMID or CLOSID)

31:0 BwType Bandwidth configuration R/W 0
to track for this counter
==========================================================================

The feature details are documented in the APM listed below [1].
[1] AMD64 Architecture Programmer's Manual Volume 2: System Programming Publication # 24593 Revision 3.41 section 19.3.3.3 Assignable Bandwidth Monitoring (ABMC).

---
v13: Removed the Reviewed-by tag as there is commit log change to remove
BMEC reference.

v12: No changes.

v11: No changes.

v10: No changes.

v9: Removed the references of L3_QOS_ABMC_DSC.
Text changes about configuration in kernel doc.

v8: Update the configuration notes in kernel_doc.
Few commit message update.

v7: Removed the reference of L3_QOS_ABMC_DSC as it is not used anymore.
Moved the configuration notes to kernel_doc.
Adjusted the tabs for l3_qos_abmc_cfg and checkpatch seems happy.

v6: Removed all the fs related changes.
Added note on CfgEn,CtrEn.
Removed the definitions which are not used.
Removed cntr_id initialization.

v5: Moved assignment flags here (path 10/19 of v4).
Added MON_CNTR_UNSET definition to initialize cntr_id's. More details in commit log.
Renamed few fields in l3_qos_abmc_cfg for readability.

v4: Added more descriptions.
Changed the name abmc_ctr_id to ctr_id.
Added L3_QOS_ABMC_DSC. Used for reading the configuration.

v3: No changes.

v2: No changes.

---
 arch/x86/include/asm/msr-index.h       |  1 +
 arch/x86/kernel/cpu/resctrl/internal.h | 35 ++++++++++++++++++++++++++
 2 files changed, 36 insertions(+)
arch/x86/include/asm/msr-index.h +1 -0
index 3970e0b16e47..b5b5ebead24f 100644
@@ -1203,6 +1203,7 @@
 /* - AMD: */
 #define MSR_IA32_MBA_BW_BASE		0xc0000200
 #define MSR_IA32_SMBA_BW_BASE		0xc0000280
+#define MSR_IA32_L3_QOS_ABMC_CFG	0xc00003fd
 #define MSR_IA32_L3_QOS_EXT_CFG		0xc00003ff
 #define MSR_IA32_EVT_CFG_BASE		0xc0000400
 
arch/x86/kernel/cpu/resctrl/internal.h +0 -0
index fcc9d23686a1..db6b0c28ee6b 100644

arch/x86/kernel/cpu/resctrl/internal.h union cpuid_0x10_x_edx { · line 170

Quoted 7 lines
arch/x86/kernel/cpu/resctrl/internal.h +4 -0
@@ -164,6 +164,41 @@ union cpuid_0x10_x_edx {
 	unsigned int full;
 };
 
+/*
+ * ABMC counters are configured by writing to L3_QOS_ABMC_CFG.
+ * @bw_type		: Bandwidth configuration (supported by BMEC)
+ *			  tracked by the @cntr_id.

The "supported by BMEC" is unexpected with the new design that separated the two features.

Quoted 24 lines
arch/x86/kernel/cpu/resctrl/internal.h +24 -0
+ * @bw_src		: Bandwidth source (RMID or CLOSID).
+ * @reserved1		: Reserved.
+ * @is_clos		: @bw_src field is a CLOSID (not an RMID).
+ * @cntr_id		: Counter identifier.
+ * @reserved		: Reserved.
+ * @cntr_en		: Counting enable bit.
+ * @cfg_en		: Configuration enable bit.
+ *
+ * Configuration and counting:
+ * Counter can be configured across multiple writes to MSR. Configuration
+ * is applied only when @cfg_en = 1. Counter @cntr_id is reset when the
+ * configuration is applied.
+ * @cfg_en = 1, @cntr_en = 0 : Apply @cntr_id configuration but do not
+ *                             count events.
+ * @cfg_en = 1, @cntr_en = 1 : Apply @cntr_id configuration and start
+ *                             counting events.
+ */
+union l3_qos_abmc_cfg {
+	struct {
+		unsigned long bw_type  :32,
+			      bw_src   :12,
+			      reserved1: 3,
+			      is_clos  : 1,
+			      cntr_id  : 5,

arch/x86/kernel/cpu/resctrl/internal.h union cpuid_0x10_x_edx { · line 204

Quoted 10 lines
arch/x86/kernel/cpu/resctrl/internal.h +7 -0
+			      reserved : 9,
+			      cntr_en  : 1,
+			      cfg_en   : 1;
+	} split;
+	unsigned long full;
+};
+
 void rdt_ctrl_update(void *arg);
 
 int rdt_get_mon_l3_config(struct rdt_resource *r);

Reinette

maintainer
#33 lore raw

Re: [PATCH v13 11/27] x86/resctrl: Implement resctrl_arch_config_cntr() to assign a counter with ABMC

In reply to Babu Moger

Hi Babu,

Quoted 4 lines · Babu Moger

The ABMC feature provides an option to the user to assign a hardware counter to an RMID, event pair and monitor the bandwidth as long as it is assigned. The assigned RMID will be tracked by the hardware until the user unassigns it manually.

(please review this often repeated snippet to match new design)

Quoted 6 lines

Implement an architecture-specific handler to assign and unassign the counter. Configure counters by writing to the L3_QOS_ABMC_CFG MSR, specifying the counter ID, bandwidth source (RMID), and event configuration.

...

Quoted 22 lines
arch/x86/kernel/cpu/resctrl/monitor.c +14 -0
index ff4b2abfa044..e31084f7babd 100644
@@ -448,3 +448,40 @@ inline bool resctrl_arch_mbm_cntr_assign_enabled(struct rdt_resource *r)
 {
 	return resctrl_to_arch_res(r)->mbm_cntr_assign_enabled;
 }
+
+static void resctrl_abmc_config_one_amd(void *info)
+{
+	union l3_qos_abmc_cfg *abmc_cfg = info;
+
+	wrmsrl(MSR_IA32_L3_QOS_ABMC_CFG, abmc_cfg->full);
+}
+
+/*
+ * Send an IPI to the domain to assign the counter to RMID, event pair.
+ */
+void resctrl_arch_config_cntr(struct rdt_resource *r, struct rdt_mon_domain *d,
+			      enum resctrl_event_id evtid, u32 rmid, u32 closid,
+			      u32 cntr_id, u32 evt_cfg, bool assign)

arch/x86/kernel/cpu/resctrl/monitor.c inline bool resctrl_arch_mbm_cntr_assign_enabled(struct rdt_resource *r) · line 474

Quoted 10 lines
arch/x86/kernel/cpu/resctrl/monitor.c +10 -0
+{
+	struct rdt_hw_mon_domain *hw_dom = resctrl_to_arch_mon_dom(d);
+	union l3_qos_abmc_cfg abmc_cfg = { 0 };
+	struct arch_mbm_state *am;
+
+	abmc_cfg.split.cfg_en = 1;
+	abmc_cfg.split.cntr_en = assign ? 1 : 0;
+	abmc_cfg.split.cntr_id = cntr_id;
+	abmc_cfg.split.bw_src = rmid;
+	abmc_cfg.split.bw_type = evt_cfg;

Is evt_cfg really needed to be programmed when unassigning a counter? Looking ahead at patch #14 resctrl_free_config_cntr() needs to go through extra list walk to get this data but why would hardware need an accurate event configuration to *unassign* a counter?

It seems unnecessary to provide both the event ID *and* the configuration. resctrl_arch_config_cntr() could drop the "evt_cfg" parameter and instead there can be a new resctrl utility that architecture can use to query the event's configuration. Similar to resctrl_is_mon_event_enabled() introduced in
https://lore.kernel.org/lkml/[email protected]/ that exposes an event property.

It looks to me as though there are a couple of changes in the telemetry work
that would benefit this work. https://lore.kernel.org/lkml/[email protected]/
switches the monitor events to be maintained in an array indexed by event ID, eliminating the need for searching the evt_list that this work does in a couple of places. Also note the handy new for_each_mbm_event() helper (https://lore.kernel.org/lkml/[email protected]/).

Quoted 18 lines
arch/x86/kernel/cpu/resctrl/monitor.c +13 -0
+
+	smp_call_function_any(&d->hdr.cpu_mask, resctrl_abmc_config_one_amd, &abmc_cfg, 1);
+
+	/*
+	 * The hardware counter is reset (because cfg_en == 1) so there is no
+	 * need to record initial non-zero counts.
+	 */
+	if (assign) {
+		am = get_arch_mbm_state(hw_dom, rmid, evtid);
+		if (am)
+			memset(am, 0, sizeof(*am));
+	}
+}
include/linux/resctrl.h +0 -0
index d77981d1fcb9..59a4fe60ab46 100644

include/linux/resctrl.h void resctrl_arch_reset_rmid_all(struct rdt_resource *r, struct rdt_mon_domain * · line 564

Quoted 6 lines
include/linux/resctrl.h +3 -0
@@ -559,6 +559,23 @@ void resctrl_arch_reset_rmid_all(struct rdt_resource *r, struct rdt_mon_domain *
  */
 void resctrl_arch_reset_all_ctrls(struct rdt_resource *r);
 
+/**
+ * resctrl_arch_config_cntr() - Configure the counter id to RMID, event
+ *				pair on the domain.

The sentence seem strange, should "Configure the counter" perhaps be "Assign the counter"? Or if the naming requires "configure" ... "Configure the counter with its new RMID and event details."? Please feel free to improve.

include/linux/resctrl.h void resctrl_arch_reset_rmid_all(struct rdt_resource *r, struct rdt_mon_domain * · line 566

Quoted 2 lines
include/linux/resctrl.h +2 -0
+ * @r:			Resource structure.
+ * @d:			Domain that the counter id to be configured.

I am unable to parse description of @d.

include/linux/resctrl.h void resctrl_arch_reset_rmid_all(struct rdt_resource *r, struct rdt_mon_domain * · line 570

Quoted 4 lines
include/linux/resctrl.h +4 -0
+ * @evtid:		Event type to configure.
+ * @rmid:		RMID to configure.
+ * @closid:		CLOSID to configure.
+ * @cntr_id:		Counter ID to configure.

All four parameters descriptions end with "to configure" ... but it is actually only the counter that is configured while the rest is the data that the counter is configured with, no?

include/linux/resctrl.h void resctrl_arch_reset_rmid_all(struct rdt_resource *r, struct rdt_mon_domain * · line 572

Quoted 2 lines
include/linux/resctrl.h +2 -0
+ * @evt_cfg:		MBM event configuration value representing reads,
+ *			writes etc.

Needs definition about what the contents of @evt_cfg means. This is the API ...it cannot be vague like "reads, write, etc." but should be specific about which bit means what.

include/linux/resctrl.h void resctrl_arch_reset_rmid_all(struct rdt_resource *r, struct rdt_mon_domain * · line 573

Quoted 1 line
include/linux/resctrl.h +1 -0
+ * @assign:		Assign or unassign.

"True to assign the counter, false to unassign the counter."

Needs some context here about what architecture can expect on how this function will be called. For example, "Can be called from any CPU."

include/linux/resctrl.h void resctrl_arch_reset_rmid_all(struct rdt_resource *r, struct rdt_mon_domain * · line 581

Quoted 8 lines
include/linux/resctrl.h +5 -0
+ */
+void resctrl_arch_config_cntr(struct rdt_resource *r, struct rdt_mon_domain *d,
+			      enum resctrl_event_id evtid, u32 rmid, u32 closid,
+			      u32 cntr_id, u32 evt_cfg, bool assign);
+
 extern unsigned int resctrl_rmid_realloc_threshold;
 extern unsigned int resctrl_rmid_realloc_limit;
 

Reinette

maintainer
#52 lore raw

Re: [PATCH v13 12/27] x86/resctrl: Introduce event configuration modes

In reply to Babu Moger

Hi Babu,

Quoted 7 lines · Babu Moger

MBM events can be configured using either BMEC (Bandwidth Monitoring Event Configuration) or the mbm_cntr_assign mode.

Introduce a data structure to represent the various event configuration modes and their corresponding values.

I cannot recall suggesting this.

(/me digs)

Are you perhaps referring to https://lore.kernel.org/lkml/[email protected]/

This is not referring to new modes but the existing mbm_cntr_assign modes. resctrl knows which "mbm_cntr_assign" mode is active and it can use that to determine whether BMEC can be exposed to user space or not. There is already enough information in resctrl to know whether BMEC files should be exposed or not.

I think this work self makes clear that these modes are useless since patch #25 that determines whether to hide BMEC files doesn't even use it.

Quoted 20 lines

---
v13: New patch to handle different event configuration types with
mbm_cntr_assign mode.

---
 fs/resctrl/internal.h         |  6 ++++--
 fs/resctrl/monitor.c          |  4 ++--
 fs/resctrl/rdtgroup.c         |  2 +-
 include/linux/resctrl_types.h | 11 +++++++++++
 4 files changed, 18 insertions(+), 5 deletions(-)
fs/resctrl/internal.h +0 -1
index 9a8cf6f11151..0fae374559ba 100644
@@ -55,13 +55,15 @@ static inline struct rdt_fs_context *rdt_fc2context(struct fs_context *fc)
  * struct mon_evt - Entry in the event list of a resource
  * @evtid:		event id
  * @name:		name of the event
- * @configurable:	true if the event is configurable

fs/resctrl/internal.h static inline struct rdt_fs_context *rdt_fc2context(struct fs_context *fc) · line 66

Quoted 10 lines
fs/resctrl/internal.h +4 -1
+ * @mbm_mode:		monitoring mode (BMEC or mbm_cntr_assign)
+ * @evt_cfg:		event configuration value decoding reads, writes.
  * @list:		entry in &rdt_resource->evt_list
  */
 struct mon_evt {
 	enum resctrl_event_id	evtid;
 	char			*name;
-	bool			configurable;
+	enum resctrl_mbm_mode	mbm_mode;
+	u32			evt_cfg;

This very important yet totally unrelated member sneaked in without any mention.

Quoted 27 lines
fs/resctrl/internal.h +0 -0
 	struct list_head	list;
 };
 
fs/resctrl/monitor.c +2 -2
index 2548aee0151c..8e403587a02f 100644
@@ -903,12 +903,12 @@ int resctrl_mon_resource_init(void)
 	l3_mon_evt_init(r);
 
 	if (resctrl_arch_is_evt_configurable(QOS_L3_MBM_TOTAL_EVENT_ID)) {
-		mbm_total_event.configurable = true;
+		mbm_total_event.mbm_mode = MBM_MODE_BMEC;
 		resctrl_file_fflags_init("mbm_total_bytes_config",
 					 RFTYPE_MON_INFO | RFTYPE_RES_CACHE);
 	}
 	if (resctrl_arch_is_evt_configurable(QOS_L3_MBM_LOCAL_EVENT_ID)) {
-		mbm_local_event.configurable = true;
+		mbm_local_event.mbm_mode = MBM_MODE_BMEC;
 		resctrl_file_fflags_init("mbm_local_bytes_config",
 					 RFTYPE_MON_INFO | RFTYPE_RES_CACHE);
 	}
fs/resctrl/rdtgroup.c +0 -0
index 752750e3e443..f192b2736a77 100644

fs/resctrl/rdtgroup.c static int rdt_mon_features_show(struct kernfs_open_file *of, · line 1155

Quoted 5 lines
fs/resctrl/rdtgroup.c +1 -1
@@ -1152,7 +1152,7 @@ static int rdt_mon_features_show(struct kernfs_open_file *of,
 
 	list_for_each_entry(mevt, &r->mon.evt_list, list) {
 		seq_printf(seq, "%s\n", mevt->name);
-		if (mevt->configurable)
+		if (mevt->mbm_mode == MBM_MODE_BMEC)

This can instead be a call to a utility that returns whether BMEC should be visible based on resctrl_mon::mbm_cntr_assignable and rdt_hw_resource::mbm_cntr_assign_enabled (via resctrl_arch_mbm_cntr_assign_enabled() of course).

Quoted 13 lines
fs/resctrl/rdtgroup.c +0 -0
 			seq_printf(seq, "%s_config\n", mevt->name);
 	}
 
include/linux/resctrl_types.h +2 -0
index a25fb9c4070d..26cd1fec72db 100644
@@ -47,4 +47,15 @@ enum resctrl_event_id {
 	QOS_NUM_EVENTS,
 };
 
+/*
+ * Event configuration mode.

include/linux/resctrl_types.h enum resctrl_event_id { · line 61

Quoted 10 lines
include/linux/resctrl_types.h +9 -0
+ * Events can be configured either in BMEC (Bandwidth Monitoring Event
+ * Configuration) mode or mbm_cntr_assign mode.
+ */
+enum resctrl_mbm_mode {
+	MBM_MODE_NONE,
+	MBM_MODE_BMEC,
+	MBM_MODE_ASSIGN,
+};
+
 #endif /* __LINUX_RESCTRL_TYPES_H */

Reinette

maintainer
#34 lore raw

RE: [PATCH v13 11/27] x86/resctrl: Implement resctrl_arch_config_cntr() to assign a counter with ABMC

In reply to Reinette Chatre
Quoted 5 lines

It looks to me as though there are a couple of changes in the telemetry work
that would benefit this work. https://lore.kernel.org/lkml/[email protected]/
switches the monitor events to be maintained in an array indexed by event ID, eliminating the need for searching the evt_list that this work does in a couple of places. Also note the handy new for_each_mbm_event() helper (https://lore.kernel.org/lkml/[email protected]/).

Yesterday I ran through the exercise of rebasing my AET patches on top of these ABMC patches in order to check whether the ABMC patches painted resctrl into some corner that would be hard to get back out of.

Good news: they don't.

There was a bunch of manual patching to make the first four patches fit on top of the ABMC code, but I also noticed a few places where things were simpler after combining the two series.

Maybe a good path forward would be to take those first four patches from my AET series and then build ABMC on top of those.

-Tony

maintainer
#55 lore raw

Re: [PATCH v13 13/27] x86/resctrl: Add the functionality to assign MBM events

In reply to Babu Moger

Hi Babu,

Quoted 2 lines · Babu Moger

The mbm_cntr_assign mode offers "num_mbm_cntrs" number of counters that can be assigned to RMID, event pair and monitor the bandwidth as long

"RMID, event pairs"? (assuming at this point in new version it will be obvious what is meant by "event").

Quoted 3 lines

as it is assigned.

Add the functionality to allocate and assign a counter to am RMID, event

"am" -> "an"

Quoted 9 lines

pair in the domain.

If all the counters are in use, kernel will log the error message "Unable to allocate counter in domain" in /sys/fs/resctrl/info/last_cmd_status when a new assignment is requested. Exit on the first failure when assigning counters across all the domains.

---

...

Quoted 24 lines
---
 fs/resctrl/internal.h |   3 +
 fs/resctrl/monitor.c  | 134 ++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 137 insertions(+)
fs/resctrl/internal.h +3 -0
index 0fae374559ba..ce4fcac91937 100644
@@ -377,6 +377,9 @@ bool closid_allocated(unsigned int closid);
 
 int resctrl_find_cleanest_closid(void);
 
+int resctrl_assign_cntr_event(struct rdt_resource *r, struct rdt_mon_domain *d,
+			      struct rdtgroup *rdtgrp, enum resctrl_event_id evtid);
+
 #ifdef CONFIG_RESCTRL_FS_PSEUDO_LOCK
 int rdtgroup_locksetup_enter(struct rdtgroup *rdtgrp);
 
fs/resctrl/monitor.c +0 -0
index 8e403587a02f..d76fd0840946 100644

fs/resctrl/monitor.c void resctrl_mon_resource_exit(void) · line 940

Quoted 7 lines
fs/resctrl/monitor.c +4 -0
@@ -934,3 +934,137 @@ void resctrl_mon_resource_exit(void)
 
 	dom_data_exit(r);
 }
+
+/*
+ * Configure the counter for the event, RMID pair for the domain. Reset the
+ * non-architectural state to clear all the event counters.

clear *all* the event counters?

"Reset the non-architectural state to clear all the event counters." -> "Reset the associated non-architectural state."?

Also, please see https://lore.kernel.org/lkml/[email protected]/

Quoted 7 lines
fs/resctrl/monitor.c +7 -0
+ */
+static void resctrl_config_cntr(struct rdt_resource *r, struct rdt_mon_domain *d,
+				enum resctrl_event_id evtid, u32 rmid, u32 closid,
+				u32 cntr_id, u32 evt_cfg, bool assign)
+{
+	struct mbm_state *m;
+

fs/resctrl/monitor.c void resctrl_mon_resource_exit(void) · line 957

Quoted 10 lines
fs/resctrl/monitor.c +10 -0
+	resctrl_arch_config_cntr(r, d, evtid, rmid, closid, cntr_id, evt_cfg, assign);
+
+	m = get_mbm_state(d, closid, rmid, evtid);
+	if (m)
+		memset(m, 0, sizeof(struct mbm_state));
+}
+
+/*
+ * mbm_cntr_get() - Return the cntr_id for the matching evtid and rdtgrp in
+ *		    cntr_cfg array.

Please prefix parameter names with @ in description to make obvious what is refered to. Although "cntr_id" is a local variable so may be easier to parse if cntr_id is replaced with actual "counter ID" term while keeping rest as actual parameters. That makes cntr_cfg unneeded.

If intending to explain function context then failure return should also be documented. Even better would be to follow typical style of kernel-doc (even if not using /** start) and not mix and match so randomly.

fs/resctrl/monitor.c void resctrl_mon_resource_exit(void) · line 961

Quoted 4 lines
fs/resctrl/monitor.c +4 -0
+ */
+static int mbm_cntr_get(struct rdt_resource *r, struct rdt_mon_domain *d,
+			struct rdtgroup *rdtgrp, enum resctrl_event_id evtid)
+{

A subtle issue here is only evident from later patches, for example patch #17, that calls mbm_cntr_get() with a non MBM event ID from __mon_event_count().

If this usage is expected then these utilities needs extra checks to ensure they are only called with valid MBM event IDs.

fs/resctrl/monitor.c void resctrl_mon_resource_exit(void) · line 974

Quoted 13 lines
fs/resctrl/monitor.c +13 -0
+	int cntr_id;
+
+	for (cntr_id = 0; cntr_id < r->mon.num_mbm_cntrs; cntr_id++) {
+		if (d->cntr_cfg[cntr_id].rdtgrp == rdtgrp &&
+		    d->cntr_cfg[cntr_id].evtid == evtid)
+			return cntr_id;
+	}
+
+	return -ENOENT;
+}
+
+/*
+ * mbm_cntr_alloc() - Return the first free entry in cntr_cfg array.

"Return the first ...array." -> "Initilialize and return ID of a new counter, return -ENOSPC on failure." ? This is still an awkward use of kernel-doc ... better to be properly formatted.

Quoted 19 lines
fs/resctrl/monitor.c +19 -0
+ */
+static int mbm_cntr_alloc(struct rdt_resource *r, struct rdt_mon_domain *d,
+			  struct rdtgroup *rdtgrp, enum resctrl_event_id evtid)
+{
+	int cntr_id;
+
+	for (cntr_id = 0; cntr_id < r->mon.num_mbm_cntrs; cntr_id++) {
+		if (!d->cntr_cfg[cntr_id].rdtgrp) {
+			d->cntr_cfg[cntr_id].rdtgrp = rdtgrp;
+			d->cntr_cfg[cntr_id].evtid = evtid;
+			return cntr_id;
+		}
+	}
+
+	return -ENOSPC;
+}
+
+/*
+ * mbm_get_mon_event() - Return the mon_evt entry for the matching evtid.

fs/resctrl/monitor.c void resctrl_mon_resource_exit(void) · line 1003

Quoted 10 lines
fs/resctrl/monitor.c +10 -0
+ */
+static struct mon_evt *mbm_get_mon_event(struct rdt_resource *r,
+					 enum resctrl_event_id evtid)
+{
+	struct mon_evt *mevt;
+
+	list_for_each_entry(mevt, &r->mon.evt_list, list) {
+		if (mevt->evtid == evtid)
+			return mevt;
+	}

With changes from telemetry series this becomes an array lookup.

Quoted 19 lines
fs/resctrl/monitor.c +19 -0
+
+	return NULL;
+}
+
+/*
+ * Allocate a fresh counter and configure the event if not assigned already.
+ */
+static int resctrl_alloc_config_cntr(struct rdt_resource *r, struct rdt_mon_domain *d,
+				     struct rdtgroup *rdtgrp, enum resctrl_event_id evtid)
+{
+	struct mon_evt *mevt;
+	int cntr_id;
+
+	/* No need to allocate a new counter if it is already assigned */
+	cntr_id = mbm_cntr_get(r, d, rdtgrp, evtid);
+	if (cntr_id >= 0)
+		goto cntr_configure;
+
+	cntr_id = mbm_cntr_alloc(r, d, rdtgrp, evtid);

fs/resctrl/monitor.c void resctrl_mon_resource_exit(void) · line 1032

Quoted 10 lines
fs/resctrl/monitor.c +10 -0
+	if (cntr_id <  0) {
+		rdt_last_cmd_printf("Unable to allocate counter in domain %d\n",
+				    d->hdr.id);
+		return cntr_id;
+	}
+
+cntr_configure:
+	mevt = mbm_get_mon_event(r, evtid);
+	if (!mevt) {
+		rdt_last_cmd_printf("Invalid event id %d\n", evtid);

Difficult to see at this point but it seems that this is in kernel bug territory since user space provided text that is translated to event ID and here translated back to monitor event. This must succeed. Could this be simplified and back-and-forth avoided by passing the mon_evt instead of event ID?

fs/resctrl/monitor.c void resctrl_mon_resource_exit(void) · line 1040

Quoted 11 lines
fs/resctrl/monitor.c +8 -0
+		return -EINVAL;
+	}
 
 
 
+
+	/*
+	 * Skip reconfiguration if the event setup is current; otherwise,
+	 * update and apply the new configuration to the domain.
+	 */
+	if (mevt->evt_cfg != d->cntr_cfg[cntr_id].evt_cfg) {

Lost me. Previous patch silently created mon_event::evt_cfg without initializing it. Here it is compared and treated as the "source of truth" ... where does its value come from?

fs/resctrl/monitor.c void resctrl_mon_resource_exit(void) · line 1052

Quoted 12 lines
fs/resctrl/monitor.c +12 -0
+		d->cntr_cfg[cntr_id].evt_cfg = mevt->evt_cfg;
+		resctrl_config_cntr(r, d, evtid, rdtgrp->mon.rmid, rdtgrp->closid,
+				    cntr_id, mevt->evt_cfg, true);
+	}
+
+	return 0;
+}
+
+/*
+ * Assign a hardware counter to event @evtid of group @rdtgrp.
+ * Assign counters to all domains if @d is NULL; otherwise, assign the
+ * counter to the specified domain @d.

Can add here what is mentioned in changelog that this exits on first failure and so highlight that this can have partial assignment when exit on such failure.

Quoted 8 lines

+ */

fs/resctrl/monitor.c +7 -0
+int resctrl_assign_cntr_event(struct rdt_resource *r, struct rdt_mon_domain *d,
+			      struct rdtgroup *rdtgrp, enum resctrl_event_id evtid)
+{
+	int ret = 0;
+
+	if (!d) {
+		list_for_each_entry(d, &r->mon_domains, hdr.list) {

fs/resctrl/monitor.c void resctrl_mon_resource_exit(void) · line 1070

Quoted 10 lines
fs/resctrl/monitor.c +10 -0
+			ret = resctrl_alloc_config_cntr(r, d, rdtgrp, evtid);
+			if (ret)
+				return ret;
+		}
+	} else {
+		ret = resctrl_alloc_config_cntr(r, d, rdtgrp, evtid);
+	}
+
+	return ret;
+}

Reinette

maintainer
#58 lore raw

Re: [PATCH v13 14/27] x86/resctrl: Add the functionality to unassign MBM events

In reply to Babu Moger

Hi Babu,

Quoted 13 lines · Babu Moger

The mbm_cntr_assign mode offers "num_mbm_cntrs" number of counters that can be assigned to an RMID, event pair and monitor the bandwidth as long as it is assigned. If all the counters are in use, the kernel will log the error message "Unable to allocate counter in domain" in /sys/fs/resctrl/info/last_cmd_status when a new assignment is requested.

To make space for a new assignment, users must unassign an already assigned counter and retry the assignment again.

Add the functionality to unassign and free the counters in the domain.

---

...

Quoted 23 lines
---
 fs/resctrl/internal.h |  2 ++
 fs/resctrl/monitor.c  | 60 +++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 62 insertions(+)
fs/resctrl/internal.h +2 -0
index ce4fcac91937..64ddc107fcab 100644
@@ -379,6 +379,8 @@ int resctrl_find_cleanest_closid(void);
 
 int resctrl_assign_cntr_event(struct rdt_resource *r, struct rdt_mon_domain *d,
 			      struct rdtgroup *rdtgrp, enum resctrl_event_id evtid);
+int resctrl_unassign_cntr_event(struct rdt_resource *r, struct rdt_mon_domain *d,
+				struct rdtgroup *rdtgrp, enum resctrl_event_id evtid);
 
 #ifdef CONFIG_RESCTRL_FS_PSEUDO_LOCK
 int rdtgroup_locksetup_enter(struct rdtgroup *rdtgrp);
fs/resctrl/monitor.c +0 -0
index d76fd0840946..fbc938bd3b23 100644

fs/resctrl/monitor.c static int mbm_cntr_alloc(struct rdt_resource *r, struct rdt_mon_domain *d, · line 993

Quoted 5 lines
fs/resctrl/monitor.c +2 -0
@@ -989,6 +989,14 @@ static int mbm_cntr_alloc(struct rdt_resource *r, struct rdt_mon_domain *d,
 	return -ENOSPC;
 }
 
+/*
+ * mbm_cntr_free() -  Reset cntr_id to zero.

"Reset cntr_id to zero"? cntr_id is an index to an array. Please provide accurate and useful descriptions.

Quoted 22 lines
fs/resctrl/monitor.c +15 -0
+ */
+static void mbm_cntr_free(struct rdt_mon_domain *d, int cntr_id)
+{
+	memset(&d->cntr_cfg[cntr_id], 0, sizeof(struct mbm_cntr_cfg));
+}
+
 /*
  * mbm_get_mon_event() - Return the mon_evt entry for the matching evtid.
  */
@@ -1068,3 +1076,55 @@ int resctrl_assign_cntr_event(struct rdt_resource *r, struct rdt_mon_domain *d,
 
 	return ret;
 }
+
+/*
+ * Unassign and free the counter if assigned.
+ */
+static int resctrl_free_config_cntr(struct rdt_resource *r, struct rdt_mon_domain *d,
+				    struct rdtgroup *rdtgrp, enum resctrl_event_id evtid)
+{
+	struct mon_evt *mevt;
+	int cntr_id;

fs/resctrl/monitor.c int resctrl_assign_cntr_event(struct rdt_resource *r, struct rdt_mon_domain *d, · line 1097

Quoted 10 lines
fs/resctrl/monitor.c +10 -0
+
+	cntr_id = mbm_cntr_get(r, d, rdtgrp, evtid);
+
+	/* If there is no cntr_id assigned, nothing to do */
+	if (cntr_id < 0)
+		return 0;
+
+	mevt = mbm_get_mon_event(r, evtid);
+	if (!mevt) {
+		rdt_last_cmd_printf("Invalid event id %d\n", evtid);

Similar to previous comment this is in kernel bug territory and could be simplified by passing mon_evt instead. Although this is the unassign portion where evt_cfg seems unnecessary.

Quoted 23 lines
fs/resctrl/monitor.c +23 -0
+		return -EINVAL;
+	}
+
+	resctrl_config_cntr(r, d, evtid, rdtgrp->mon.rmid, rdtgrp->closid,
+			    cntr_id, mevt->evt_cfg, false);
+
+	mbm_cntr_free(d, cntr_id);
+
+	return 0;
+}
+
+/*
+ * Unassign a hardware counter associated with @evtid from the domain and
+ * the group. Unassign the counters from all the domains if @d is NULL else
+ * unassign from @d.
+ */
+int  resctrl_unassign_cntr_event(struct rdt_resource *r, struct rdt_mon_domain *d,
+				 struct rdtgroup *rdtgrp, enum resctrl_event_id evtid)
+{
+	int ret;
+
+	if (!d) {
+		list_for_each_entry(d, &r->mon_domains, hdr.list) {

fs/resctrl/monitor.c int resctrl_assign_cntr_event(struct rdt_resource *r, struct rdt_mon_domain *d, · line 1130

Quoted 10 lines
fs/resctrl/monitor.c +10 -0
+			ret = resctrl_free_config_cntr(r, d, rdtgrp, evtid);
+			if (ret)
+				return ret;
+		}
+	} else {
+		ret = resctrl_free_config_cntr(r, d, rdtgrp, evtid);
+	}
+
+	return ret;
+}

Reinette

maintainer
#61 lore raw

Re: [PATCH v13 15/27] x86/resctrl: Report 'Unassigned' for MBM events in mbm_cntr_assign mode

In reply to Babu Moger

Hi Babu,

Quoted 8 lines · Babu Moger

In mbm_cntr_assign mode, the hardware counter should be assigned to read the MBM events.

Report 'Unassigned' in case the user attempts to read the event without assigning a hardware counter.

Export resctrl_is_mbm_event() and mbm_cntr_get() to allow usage from other functions within fs/resctrl.

Please clarify that these two functions are exposed differently, resctrl_is_mbm_event() is added to include/linux/resctrl.h (also note similar change in
https://lore.kernel.org/lkml/[email protected]/)
so not just exposed to fs/resctrl but instead to resctrl fs as well as arch code
while mbm_cntr_get() remains internal to resctrl fs by being added to fs/resctrl/internal.h.

Quoted 19 lines

---

---
 Documentation/filesystems/resctrl.rst |  8 ++++++++
 fs/resctrl/ctrlmondata.c              | 14 ++++++++++++++
 fs/resctrl/internal.h                 |  2 ++
 fs/resctrl/monitor.c                  |  4 ++--
 fs/resctrl/rdtgroup.c                 |  2 +-
 include/linux/resctrl.h               |  1 +
 6 files changed, 28 insertions(+), 3 deletions(-)
Documentation/filesystems/resctrl.rst +0 -0
index 2bfad43aac9c..5cf2d742f04c 100644

Documentation/filesystems/resctrl.rst When monitoring is enabled all MON groups will also contain: · line 439

Quoted 10 lines
Documentation/filesystems/resctrl.rst +7 -0
@@ -430,6 +430,14 @@ When monitoring is enabled all MON groups will also contain:
 	for the L3 cache they occupy). These are named "mon_sub_L3_YY"
 	where "YY" is the node number.
 
+	The mbm_cntr_assign mode offers "num_mbm_cntrs" number of counters
+	and allows users to assign a counter to mon_hw_id, event pair enabling
+	bandwidth monitoring for as long as the counter remains assigned.
+	The hardware will continue tracking the assigned mon_hw_id until
+	the user manually unassigns it, ensuring that counters are not reset
+	during this period. An MBM event returns 'Unassigned' when the event
+	does not have a hardware counter assigned.

(please rework based on "event" vs "group" assignment ... not intending that "group" assignment be documented but the "event" assignment needs to be accurate for "group" assignment to be a simple extension)

Quoted 28 lines
Documentation/filesystems/resctrl.rst +1 -0
+
 "mon_hw_id":
 	Available only with debug option. The identifier used by hardware
 	for the monitor group. On x86 this is the RMID.
fs/resctrl/ctrlmondata.c +12 -0
index 6ed2dfd4dbbd..f6b8ad24b0b5 100644
@@ -643,6 +643,18 @@ int rdtgroup_mondata_show(struct seq_file *m, void *arg)
 			goto out;
 		}
 		d = container_of(hdr, struct rdt_mon_domain, hdr);
+
+		/*
+		 * Report 'Unassigned' if mbm_cntr_assign mode is enabled and
+		 * counter is unassigned.
+		 */
+		if (resctrl_arch_mbm_cntr_assign_enabled(r) &&
+		    resctrl_is_mbm_event(evtid) &&
+		    (mbm_cntr_get(r, d, rdtgrp, evtid) < 0)) {
+			rr.err = -ENOENT;
+			goto checkresult;
+		}
+
 		mon_event_read(&rr, r, d, rdtgrp, &d->hdr.cpu_mask, evtid, false);
 	}
 

fs/resctrl/ctrlmondata.c int rdtgroup_mondata_show(struct seq_file *m, void *arg) · line 671

Quoted 8 lines
fs/resctrl/ctrlmondata.c +2 -0
@@ -652,6 +664,8 @@ int rdtgroup_mondata_show(struct seq_file *m, void *arg)
 		seq_puts(m, "Error\n");
 	else if (rr.err == -EINVAL)
 		seq_puts(m, "Unavailable\n");
+	else if (rr.err == -ENOENT)
+		seq_puts(m, "Unassigned\n");
 	else
 		seq_printf(m, "%llu\n", rr.val);
 

It may be unexpected that this is treated as "-ENOENT" but the function returns success. This can be addressed with a comment when comparing the return codes to other hardware return codes.

Quoted 46 lines
fs/resctrl/internal.h +2 -0
index 64ddc107fcab..0dfd2efe68fc 100644
@@ -381,6 +381,8 @@ int resctrl_assign_cntr_event(struct rdt_resource *r, struct rdt_mon_domain *d,
 			      struct rdtgroup *rdtgrp, enum resctrl_event_id evtid);
 int resctrl_unassign_cntr_event(struct rdt_resource *r, struct rdt_mon_domain *d,
 				struct rdtgroup *rdtgrp, enum resctrl_event_id evtid);
+int mbm_cntr_get(struct rdt_resource *r, struct rdt_mon_domain *d,
+		 struct rdtgroup *rdtgrp, enum resctrl_event_id evtid);
 
 #ifdef CONFIG_RESCTRL_FS_PSEUDO_LOCK
 int rdtgroup_locksetup_enter(struct rdtgroup *rdtgrp);
fs/resctrl/monitor.c +2 -2
index fbc938bd3b23..c98a61bde179 100644
@@ -956,8 +956,8 @@ static void resctrl_config_cntr(struct rdt_resource *r, struct rdt_mon_domain *d
  * mbm_cntr_get() - Return the cntr_id for the matching evtid and rdtgrp in
  *		    cntr_cfg array.
  */
-static int mbm_cntr_get(struct rdt_resource *r, struct rdt_mon_domain *d,
-			struct rdtgroup *rdtgrp, enum resctrl_event_id evtid)
+int mbm_cntr_get(struct rdt_resource *r, struct rdt_mon_domain *d,
+		 struct rdtgroup *rdtgrp, enum resctrl_event_id evtid)
 {
 	int cntr_id;
 
fs/resctrl/rdtgroup.c +1 -1
index f192b2736a77..72317a5adee2 100644
@@ -127,7 +127,7 @@ static bool resctrl_is_mbm_enabled(void)
 		resctrl_arch_is_mbm_local_enabled());
 }
 
-static bool resctrl_is_mbm_event(int e)
+bool resctrl_is_mbm_event(int e)
 {
 	return (e >= QOS_L3_MBM_TOTAL_EVENT_ID &&
 		e <= QOS_L3_MBM_LOCAL_EVENT_ID);
include/linux/resctrl.h +0 -0
index 59a4fe60ab46..f78b6064230c 100644

include/linux/resctrl.h static inline u32 resctrl_get_config_index(u32 closid, · line 447

Quoted 7 lines
include/linux/resctrl.h +1 -0
@@ -441,6 +441,7 @@ static inline u32 resctrl_get_config_index(u32 closid,
 	}
 }
 
+bool resctrl_is_mbm_event(int e);
 bool resctrl_arch_get_cdp_enabled(enum resctrl_res_level l);
 int resctrl_arch_set_cdp_enabled(enum resctrl_res_level l, bool enable);
 

Reinette

maintainer
#64 lore raw

Re: [PATCH v13 16/27] x86/resctrl: Pass entire struct rdtgroup rather than passing individual members

In reply to Babu Moger

Hi Babu,

Quoted 1 line · Babu Moger

The mbm_cntr_assign mode requires a cntr_id to read event data. The

cntr_id -> "counter ID"

Quoted 1 line

cntr_id is retrieved via mbm_cntr_get, which takes a struct rdtgroup as

cntr_id -> "counter ID"

mbm_cntr_get -> mbm_cntr_get()

Quoted 3 lines

a parameter.

Passing the full rdtgroup also provides access to closid and rmid, both of

closid -> CLOSID
rmid -> RMID

Quoted 3 lines

which are necessary to read monitoring events.

Refactor the code to pass the entire struct rdtgroup instead of individual

"the entire" -> "a pointer to"

Quoted 5 lines

members in preparation for this requirement.

---

Patch looks good.

Reinette

maintainer
#67 lore raw

Re: [PATCH v13 17/27] x86/resctrl: Add the support for reading ABMC counters

In reply to Babu Moger

Hi Babu,

Quoted 2 lines · Babu Moger

Software can read the assignable counters using the QM_EVTSEL and QM_CTR register pair.

Please append with more context on how register pair is used to support the changes in this patch.

Quoted 12 lines

QM_EVTSEL Register definition:
=======================================================
Bits Mnemonic Description
=======================================================
63:44 -- Reserved
43:32 RMID Resource Monitoring Identifier
31 ExtEvtID Extended Event Identifier
30:8 -- Reserved
7:0 EvtID Event Identifier
=======================================================

Quoted 6 lines

The contents of a specific counter can be read by setting the following fields in QM_EVTSEL.ExtendedEvtID = 1, QM_EVTSEL.EvtID = L3CacheABMC (=1) and setting [RMID] to the desired counter ID. Reading QM_CTR will then return the contents of the specified counter. The E bit will be set if the counter configuration was invalid, or if an invalid counter ID was set in the QM_EVTSEL[RMID] field.

Please rewrite above in imperative tone.

Quoted 26 lines

Introduce __cntr_id_read_phys() to read the counter ID event data.

---
v13: Split the patch into 2. First one to handle the passing of rdtgroup structure to few
functions( __mon_event_count and mbm_update(). Second one to handle ABMC counter reading. Added new function __cntr_id_read_phys() to handle ABMC event reading. Updated kernel doc for resctrl_arch_reset_rmid() and resctrl_arch_rmid_read(). Resolved conflicts caused by the recent FS/ARCH code restructure.
The monitor.c file has now been split between the FS and ARCH directories.

v12: New patch to support extended event mode when ABMC is enabled.

---
 arch/x86/kernel/cpu/resctrl/internal.h |  6 +++
 arch/x86/kernel/cpu/resctrl/monitor.c  | 66 ++++++++++++++++++++++----
 fs/resctrl/monitor.c                   | 14 ++++--
 include/linux/resctrl.h                |  9 ++--
 4 files changed, 80 insertions(+), 15 deletions(-)
arch/x86/kernel/cpu/resctrl/internal.h +0 -0
index db6b0c28ee6b..3b0cdb5520c7 100644

arch/x86/kernel/cpu/resctrl/internal.h struct arch_mbm_state { · line 44

Quoted 5 lines
arch/x86/kernel/cpu/resctrl/internal.h +2 -0
@@ -40,6 +40,12 @@ struct arch_mbm_state {
 /* Setting bit 0 in L3_QOS_EXT_CFG enables the ABMC feature. */
 #define ABMC_ENABLE_BIT			0
 
+/*
+ * ABMC Qos Event Identifiers.

QoS?

arch/x86/kernel/cpu/resctrl/internal.h struct arch_mbm_state { · line 47

Quoted 3 lines
arch/x86/kernel/cpu/resctrl/internal.h +3 -0
+ */
+#define ABMC_EXTENDED_EVT_ID		BIT(31)
+#define ABMC_EVT_ID			1

Please use BIT(0) to be consistent.

Quoted 60 lines
arch/x86/kernel/cpu/resctrl/internal.h +1 -0
+
 /**
  * struct rdt_hw_ctrl_domain - Arch private attributes of a set of CPUs that share
  *			       a resource for a control function
arch/x86/kernel/cpu/resctrl/monitor.c +36 -1
index e31084f7babd..36a03dae6d8e 100644
@@ -161,6 +161,41 @@ static int __rmid_read_phys(u32 prmid, enum resctrl_event_id eventid, u64 *val)
 	return 0;
 }
 
+static int __cntr_id_read_phys(u32 cntr_id, u64 *val)
+{
+	u64 msr_val;
+
+	/*
+	 * QM_EVTSEL Register definition:
+	 * =======================================================
+	 * Bits    Mnemonic        Description
+	 * =======================================================
+	 * 63:44   --              Reserved
+	 * 43:32   RMID            Resource Monitoring Identifier
+	 * 31      ExtEvtID        Extended Event Identifier
+	 * 30:8    --              Reserved
+	 * 7:0     EvtID           Event Identifier
+	 * =======================================================
+	 * The contents of a specific counter can be read by setting the
+	 * following fields in QM_EVTSEL.ExtendedEvtID(=1) and
+	 * QM_EVTSEL.EvtID = L3CacheABMC (=1) and setting [RMID] to the
+	 * desired counter ID. Reading QM_CTR will then return the
+	 * contents of the specified counter. The E bit will be set if the
+	 * counter configuration was invalid, or if an invalid counter ID
+	 * was set in the QM_EVTSEL[RMID] field.
+	 */
+	wrmsr(MSR_IA32_QM_EVTSEL, ABMC_EXTENDED_EVT_ID | ABMC_EVT_ID, cntr_id);
+	rdmsrl(MSR_IA32_QM_CTR, msr_val);
+
+	if (msr_val & RMID_VAL_ERROR)
+		return -EIO;
+	if (msr_val & RMID_VAL_UNAVAIL)
+		return -EINVAL;
+
+	*val = msr_val;
+	return 0;
+}
+
 static struct arch_mbm_state *get_arch_mbm_state(struct rdt_hw_mon_domain *hw_dom,
 						 u32 rmid,
 						 enum resctrl_event_id eventid)
@@ -180,7 +215,7 @@ static struct arch_mbm_state *get_arch_mbm_state(struct rdt_hw_mon_domain *hw_do
 }
 
 void resctrl_arch_reset_rmid(struct rdt_resource *r, struct rdt_mon_domain *d,
-			     u32 unused, u32 rmid,
+			     u32 unused, u32 rmid, int cntr_id,
 			     enum resctrl_event_id eventid)
 {
 	struct rdt_hw_mon_domain *hw_dom = resctrl_to_arch_mon_dom(d);

arch/x86/kernel/cpu/resctrl/monitor.c void resctrl_arch_reset_rmid(struct rdt_resource *r, struct rdt_mon_domain *d, · line 232

Quoted 9 lines
arch/x86/kernel/cpu/resctrl/monitor.c +3 -3
@@ -192,9 +227,16 @@ void resctrl_arch_reset_rmid(struct rdt_resource *r, struct rdt_mon_domain *d,
 	if (am) {
 		memset(am, 0, sizeof(*am));
 
-		prmid = logical_rmid_to_physical_rmid(cpu, rmid);
-		/* Record any initial, non-zero count value. */
-		__rmid_read_phys(prmid, eventid, &am->prev_msr);
+		if (resctrl_arch_mbm_cntr_assign_enabled(r) &&
+		    resctrl_is_mbm_event(eventid)) {
+			if (cntr_id < 0)

This would be a bug, no? how about WARN_ON_ONCE()?

Quoted 22 lines
arch/x86/kernel/cpu/resctrl/monitor.c +9 -2
+				return;
+			__cntr_id_read_phys(cntr_id, &am->prev_msr);
+		} else {
+			prmid = logical_rmid_to_physical_rmid(cpu, rmid);
+			/* Record any initial, non-zero count value. */
+			__rmid_read_phys(prmid, eventid, &am->prev_msr);
+		}
 	}
 }
 
@@ -224,8 +266,8 @@ static u64 mbm_overflow_count(u64 prev_msr, u64 cur_msr, unsigned int width)
 }
 
 int resctrl_arch_rmid_read(struct rdt_resource *r, struct rdt_mon_domain *d,
-			   u32 unused, u32 rmid, enum resctrl_event_id eventid,
-			   u64 *val, void *ignored)
+			   u32 unused, u32 rmid, int cntr_id,
+			   enum resctrl_event_id eventid, u64 *val, void *ignored)
 {
 	struct rdt_hw_mon_domain *hw_dom = resctrl_to_arch_mon_dom(d);
 	struct rdt_hw_resource *hw_res = resctrl_to_arch_res(r);

arch/x86/kernel/cpu/resctrl/monitor.c int resctrl_arch_rmid_read(struct rdt_resource *r, struct rdt_mon_domain *d, · line 284

Quoted 8 lines
arch/x86/kernel/cpu/resctrl/monitor.c +3 -2
@@ -237,8 +279,16 @@ int resctrl_arch_rmid_read(struct rdt_resource *r, struct rdt_mon_domain *d,
 
 	resctrl_arch_rmid_read_context_check();
 
-	prmid = logical_rmid_to_physical_rmid(cpu, rmid);
-	ret = __rmid_read_phys(prmid, eventid, &msr_val);
+	if (resctrl_arch_mbm_cntr_assign_enabled(r) &&
+	    resctrl_is_mbm_event(eventid)) {
+		if (cntr_id < 0)

WARN_ON_ONCE()?

Quoted 28 lines
arch/x86/kernel/cpu/resctrl/monitor.c +7 -0
+			return cntr_id;
+		ret = __cntr_id_read_phys(cntr_id, &msr_val);
+	} else {
+		prmid = logical_rmid_to_physical_rmid(cpu, rmid);
+		ret = __rmid_read_phys(prmid, eventid, &msr_val);
+	}
+
 	if (ret)
 		return ret;
 
fs/resctrl/monitor.c +5 -1
index a477be9cdb66..72f3dfb5b903 100644
@@ -159,7 +159,11 @@ void __check_limbo(struct rdt_mon_domain *d, bool force_free)
 			break;
 
 		entry = __rmid_entry(idx);
-		if (resctrl_arch_rmid_read(r, d, entry->closid, entry->rmid,
+		/*
+		 * cntr_id is not relevant for QOS_L3_OCCUP_EVENT_ID.
+		 * Pass dummy value -1.
+		 */
+		if (resctrl_arch_rmid_read(r, d, entry->closid, entry->rmid, -1,
 					   QOS_L3_OCCUP_EVENT_ID, &val,
 					   arch_mon_ctx)) {
 			rmid_dirty = true;

fs/resctrl/monitor.c static struct mbm_state *get_mbm_state(struct rdt_mon_domain *d, u32 closid, · line 366

Quoted 4 lines
fs/resctrl/monitor.c +1 -0
@@ -359,6 +363,7 @@ static struct mbm_state *get_mbm_state(struct rdt_mon_domain *d, u32 closid,
 
 static int __mon_event_count(struct rdtgroup *rdtgrp, struct rmid_read *rr)
 {
+	int cntr_id = mbm_cntr_get(rr->r, rr->d, rdtgrp, rr->evtid);

So mbm_cntr_get() is called on *all* events (even non MBM) whether assignable counters are supported or not. I assume it relies on num_mbm_cntrs to be zero on non-ABMC systems but I think this needs to be explicit that mbm_cntr_get() returns -ENOENT in these cases. Any developer attempting to modify mbm_cntr_get() needs to be aware of this usage.

This is quite subtle that resctrl_arch_reset_rmid() and resctrl_arch_rmid_read() can be called with a negative counter ID. To help with code health this needs to be highlighted (more later).

Quoted 36 lines
fs/resctrl/monitor.c +4 -3
 	int cpu = smp_processor_id();
 	u32 closid = rdtgrp->closid;
 	u32 rmid = rdtgrp->mon.rmid;
@@ -368,7 +373,7 @@ static int __mon_event_count(struct rdtgroup *rdtgrp, struct rmid_read *rr)
 	u64 tval = 0;
 
 	if (rr->first) {
-		resctrl_arch_reset_rmid(rr->r, rr->d, closid, rmid, rr->evtid);
+		resctrl_arch_reset_rmid(rr->r, rr->d, closid, rmid, cntr_id, rr->evtid);
 		m = get_mbm_state(rr->d, closid, rmid, rr->evtid);
 		if (m)
 			memset(m, 0, sizeof(struct mbm_state));
@@ -379,7 +384,7 @@ static int __mon_event_count(struct rdtgroup *rdtgrp, struct rmid_read *rr)
 		/* Reading a single domain, must be on a CPU in that domain. */
 		if (!cpumask_test_cpu(cpu, &rr->d->hdr.cpu_mask))
 			return -EINVAL;
-		rr->err = resctrl_arch_rmid_read(rr->r, rr->d, closid, rmid,
+		rr->err = resctrl_arch_rmid_read(rr->r, rr->d, closid, rmid, cntr_id,
 						 rr->evtid, &tval, rr->arch_mon_ctx);
 		if (rr->err)
 			return rr->err;
@@ -404,7 +409,8 @@ static int __mon_event_count(struct rdtgroup *rdtgrp, struct rmid_read *rr)
 	list_for_each_entry(d, &rr->r->mon_domains, hdr.list) {
 		if (d->ci->id != rr->ci->id)
 			continue;
-		err = resctrl_arch_rmid_read(rr->r, d, closid, rmid,
+		cntr_id = mbm_cntr_get(rr->r, d, rdtgrp, rr->evtid);
+		err = resctrl_arch_rmid_read(rr->r, d, closid, rmid, cntr_id,
 					     rr->evtid, &tval, rr->arch_mon_ctx);
 		if (!err) {
 			rr->val += tval;
include/linux/resctrl.h +0 -0
index f78b6064230c..cd24d1577e0a 100644

include/linux/resctrl.h void resctrl_offline_cpu(unsigned int cpu); · line 476

Quoted 4 lines
include/linux/resctrl.h +1 -0
@@ -473,6 +473,7 @@ void resctrl_offline_cpu(unsigned int cpu);
  *			counter may match traffic of both @closid and @rmid, or @rmid
  *			only.
  * @rmid:		rmid of the counter to read.
+ * @cntr_id:		cntr_id to read MBM events with mbm_cntr_assign mode.

"Counter ID used to read MBM events in mbm_cntr_evt_assign mode. Only valid when
mbm_cntr_evt_assign mode is enabled and @eventid is an MBM event. Can be negative when invalid." (Please feel free to improve)

Quoted 16 lines
include/linux/resctrl.h +3 -2
  * @eventid:		eventid to read, e.g. L3 occupancy.
  * @val:		result of the counter read in bytes.
  * @arch_mon_ctx:	An architecture specific value from
@@ -490,8 +491,9 @@ void resctrl_offline_cpu(unsigned int cpu);
  * 0 on success, or -EIO, -EINVAL etc on error.
  */
 int resctrl_arch_rmid_read(struct rdt_resource *r, struct rdt_mon_domain *d,
-			   u32 closid, u32 rmid, enum resctrl_event_id eventid,
-			   u64 *val, void *arch_mon_ctx);
+			   u32 closid, u32 rmid, int cntr_id,
+			   enum resctrl_event_id eventid, u64 *val,
+			   void *arch_mon_ctx);
 
 /**
  * resctrl_arch_rmid_read_context_check()  - warn about invalid contexts

include/linux/resctrl.h struct rdt_domain_hdr *resctrl_find_domain(struct list_head *h, int id, · line 537

Quoted 4 lines
include/linux/resctrl.h +1 -0
@@ -532,12 +534,13 @@ struct rdt_domain_hdr *resctrl_find_domain(struct list_head *h, int id,
  * @closid:	closid that matches the rmid. Depending on the architecture, the
  *		counter may match traffic of both @closid and @rmid, or @rmid only.
  * @rmid:	The rmid whose counter values should be reset.
+ * @cntr_id:	The cntr_id to read MBM events with mbm_cntr_assign mode.

Same as above.

include/linux/resctrl.h struct rdt_domain_hdr *resctrl_find_domain(struct list_head *h, int id, · line 546

Quoted 10 lines
include/linux/resctrl.h +1 -1
  * @eventid:	The eventid whose counter values should be reset.
  *
  * This can be called from any CPU.
  */
 void resctrl_arch_reset_rmid(struct rdt_resource *r, struct rdt_mon_domain *d,
-			     u32 closid, u32 rmid,
+			     u32 closid, u32 rmid, int cntr_id,
 			     enum resctrl_event_id eventid);
 
 /**

Reinette

maintainer
#70 lore raw

Re: [PATCH v13 18/27] x86/resctrl: Add definitions for MBM event configuration

In reply to Babu Moger

Hi Babu,

Quoted 7 lines · Babu Moger

The "mbm_cntr_assign" mode allows users to manually assign a hardware counter to a specific RMID and event pair. The events available for assignment are configurable.

By default, each resctrl group supports two MBM events: mbm_total_bytes and mbm_local_bytes. Each event corresponds to an MBM configuration that specifies the bandwidth sources tracked by the event.

hmmm ... earlier I thought "bandwidth source" means RMID but here it seems to mean the memory transactions? The various terms are confusing.

Quoted 2 lines

Add definitions of supported bandwidth sources.

changelog uses "bandwidth sources" while the comments of patch uses "memory transactions" ... please be consistent with terms.

Quoted 22 lines

---
v13: Updated the changelog.
Removed the definitions from resctrl_types.h and moved to internal.h. Removed mbm_assign_config definition. Configurations will be part of mon_evt list.
Resolved conflicts caused by the recent FS/ARCH code restructure. The rdtgroup.c file has now been split between the FS and ARCH directories.

v12: New patch to support event configurations via new counter_configs
method.

---
 fs/resctrl/internal.h | 10 ++++++++++
 fs/resctrl/rdtgroup.c | 14 ++++++++++++++
 2 files changed, 24 insertions(+)
fs/resctrl/internal.h +0 -0
index 0dfd2efe68fc..019d00bf5adf 100644

fs/resctrl/internal.h struct rdtgroup { · line 207

Quoted 5 lines
fs/resctrl/internal.h +2 -0
@@ -203,6 +203,16 @@ struct rdtgroup {
 	struct pseudo_lock_region	*plr;
 };
 
+/**
+ * struct mbm_evt_value - Specific type of memory events.

I am trying to decipher the terminology. If these are events, then it becomes confusing since it becomes "these events are used to configure events". You mention "memory transaction" below, this sounds more accurate to me. Above could thus be:

struct mbm_evt_value - Memory transaction an MBM event can be configured with.

The name of the struct could also do with a rename to avoid the "event" term that conflicts with the actual MBM events. Maybe "mbm_cfg_value" ... I do not think this is a good name so please consider what would work better.

fs/resctrl/internal.h struct rdtgroup { · line 208

Quoted 1 line
fs/resctrl/internal.h +1 -0
+ * @evt_name:		Name of memory transaction type (read, write etc).

Unclear what "type" means ... maybe just "Name of memory transaction (read, write ...)"?

The "evt_" prefix looks unnecessary.

fs/resctrl/internal.h struct rdtgroup { · line 209

Quoted 1 line
fs/resctrl/internal.h +1 -0
+ * @evt_val:		Value representing the memory transaction.

This could just be "val" and the description could be specific:

"The bit used to represent the memory transaction within an event's configuration." Please feel free to improve.

fs/resctrl/internal.h struct rdtgroup { · line 213

Quoted 4 lines
fs/resctrl/internal.h +4 -0
+ */
+struct mbm_evt_value {
+	char    evt_name[32];
+	u32     evt_val;

Please space member names with TABs.

Quoted 10 lines
fs/resctrl/internal.h +2 -0
+};
+
 /* rdtgroup.flags */
 #define	RDT_DELETED		1
 
fs/resctrl/rdtgroup.c +0 -0
index 72317a5adee2..b109e91096b0 100644

fs/resctrl/rdtgroup.c static void rdtgroup_destroy_root(void); · line 78

Quoted 4 lines
fs/resctrl/rdtgroup.c +1 -0
@@ -75,6 +75,20 @@ static void rdtgroup_destroy_root(void);
 
 struct dentry *debugfs_resctrl;
 
+/* Number of memory transaction types that can be monitored */

"Number of memory transactions that an MBM event can be configured with."?

fs/resctrl/rdtgroup.c static void rdtgroup_destroy_root(void); · line 81

Quoted 3 lines
fs/resctrl/rdtgroup.c +3 -0
+#define NUM_MBM_EVT_VALUES             7
+
+/* Decoded values for each type of memory events */

Please be consistent with terminology. In the above lines it switches between "memory transaction types" and "memory events".

fs/resctrl/rdtgroup.c static void rdtgroup_destroy_root(void); · line 94

Quoted 13 lines
fs/resctrl/rdtgroup.c +10 -0
+struct mbm_evt_value mbm_evt_values[NUM_MBM_EVT_VALUES] = {
+	{"local_reads", READS_TO_LOCAL_MEM},
+	{"remote_reads", READS_TO_REMOTE_MEM},
+	{"local_non_temporal_writes", NON_TEMP_WRITE_TO_LOCAL_MEM},
+	{"remote_non_temporal_writes", NON_TEMP_WRITE_TO_REMOTE_MEM},
+	{"local_reads_slow_memory", READS_TO_LOCAL_S_MEM},
+	{"remote_reads_slow_memory", READS_TO_REMOTE_S_MEM},
+	{"dirty_victim_writes_all", DIRTY_VICTIMS_TO_ALL_MEM},
+};
+
 /*
  * Memory bandwidth monitoring event to use for the default CTRL_MON group
  * and each new CTRL_MON group created by the user.  Only relevant when

Reinette

maintainer
#75 lore raw

Re: [PATCH v13 19/27] x86/resctrl: Add event configuration directory under info/L3_MON/

In reply to Babu Moger

Hi Babu,

Quoted 3 lines · Babu Moger

Create the configuration directory and files for mbm_cntr_assign mode. These configurations will be used to assign MBM events in mbm_cntr_assign mode, with two default configurations created upon mounting.

This just jumps in with what the patch does. Requirements for proper changelog should be familiar by now. The changelog *always* starts with a context.

Sample:

"When assignable counters are supported the /sys/fs/resctrl/info/L3_MON/event_configs directory contains a sub-directory for each MBM event that can be assigned to a counter. The MBM event sub-directory contains a file named "event_filter" that is used to view and modify which memory transactions the MBM event is configured with.

Create the /sys/fs/resctrl/info/L3_MON/event_configs directory on resctrl mount and pre-populate it with directories for the two existing MBM events:
mbm_total_bytes and mbm_local_bytes. Create the "event_filter" file within each MBM event directory with the needed *show() that displays the memory transactions with which the MBM event is configured."

Quoted 38 lines

Example:

$ cd /sys/fs/resctrl/
$ cat info/L3_MON/counter_configs/mbm_total_bytes/event_filter
  local_reads, remote_reads, local_non_temporal_writes,
  remote_non_temporal_writes, local_reads_slow_memory,
  remote_reads_slow_memory, dirty_victim_writes_all
$ cat info/L3_MON/counter_configs/mbm_local_bytes/event_filter
  local_reads, local_non_temporal_writes, local_reads_slow_memory

---
v13: Updated user doc (resctrl.rst).
Changed the name of the function resctrl_mkdir_info_configs to resctrl_mkdir_counter_configs().
Replaced seq_puts() with seq_putc() where applicable.
Removed RFTYPE_MON_CONFIG definition. Not required.
Changed the name of the flag RFTYPE_CONFIG to RFTYPE_ASSIGN_CONFIG. Reinette suggested RFTYPE_MBM_EVENT_CONFIG but RFTYPE_ASSIGN_CONFIG seemed shorter and pricise.
The configuration is created using evt_list.
Resolved conflicts caused by the recent FS/ARCH code restructure.
The monitor.c/rdtgroup.c files have been split between the FS and ARCH directories.

v12: New patch to hold the MBM event configurations for mbm_cntr_assign mode.

---
 Documentation/filesystems/resctrl.rst | 30 ++++++++++
 fs/resctrl/internal.h                 |  2 +
 fs/resctrl/monitor.c                  |  1 +
 fs/resctrl/rdtgroup.c                 | 80 +++++++++++++++++++++++++++
 4 files changed, 113 insertions(+)
Documentation/filesystems/resctrl.rst +0 -0
index 5cf2d742f04c..4eb9f007ba3d 100644

Documentation/filesystems/resctrl.rst with the following files: · line 311

Quoted 6 lines
Documentation/filesystems/resctrl.rst +3 -0
@@ -306,6 +306,36 @@ with the following files:
 	  # cat /sys/fs/resctrl/info/L3_MON/available_mbm_cntrs
 	  0=30;1=30
 
+"counter_configs":
+	When the "mbm_cntr_assign" mode is supported, a dedicated directory is created
+	under the "L3_MON" directory to store configuration files.

? it does not contain files but directories for each event, no?

It will help if the text is specific. For example,
"event_configs":
Directory that exists when mbm_cntr_evt_assign is supported. Contains sub-directory for each MBM event that can be assigned to a counter. Each MBM event sub-directory ...

Documentation/filesystems/resctrl.rst with the following files: · line 313

Quoted 2 lines
Documentation/filesystems/resctrl.rst +2 -0
+
+	These files contain the list of configurable events. There are two default

So confusing ... terminology is all over the place. Which files are even talked about here? "configurable events" ... are these the memory transactions or MBM events?

Documentation/filesystems/resctrl.rst with the following files: · line 314

Quoted 1 line
Documentation/filesystems/resctrl.rst +1 -0
+	configurations: mbm_local_bytes and mbm_total_bytes.

"two default configurations"? These are not "configurations" but "events", no?

Documentation/filesystems/resctrl.rst with the following files: · line 316

Quoted 2 lines
Documentation/filesystems/resctrl.rst +2 -0
+
+	Following types of events are supported:

events -> memory transactions?

I am unable to parse the above.

Documentation/filesystems/resctrl.rst with the following files: · line 328

Quoted 12 lines
Documentation/filesystems/resctrl.rst +12 -0
+
+	==== ========================= ============================================================
+	Bits Name   		         Description
+	==== ========================= ============================================================
+	6    dirty_victim_writes_all     Dirty Victims from the QOS domain to all types of memory
+	5    remote_reads_slow_memory    Reads to slow memory in the non-local NUMA domain
+	4    local_reads_slow_memory     Reads to slow memory in the local NUMA domain
+	3    remote_non_temporal_writes  Non-temporal writes to non-local NUMA domain
+	2    local_non_temporal_writes   Non-temporal writes to local NUMA domain
+	1    remote_reads                Reads to memory in the non-local NUMA domain
+	0    local_reads                 Reads to memory in the local NUMA domain
+	==== ========================= ==========================================================

Why does user need to know the bit position used to represent the memory transaction?

Quoted 44 lines
Documentation/filesystems/resctrl.rst +10 -0
+
+	For example::
+
+	  # cat /sys/fs/resctrl/info/L3_MON/counter_configs/mbm_total_bytes/event_filter
+	  local_reads, remote_reads, local_non_temporal_writes, remote_non_temporal_writes,
+	  local_reads_slow_memory, remote_reads_slow_memory, dirty_victim_writes_all
+
+	  # cat /sys/fs/resctrl/info/L3_MON/counter_configs/mbm_local_bytes/event_filter
+	  local_reads, local_non_temporal_writes, local_reads_slow_memory
+
 "max_threshold_occupancy":
 		Read/write file provides the largest value (in
 		bytes) at which a previously used LLC_occupancy
fs/resctrl/internal.h +2 -0
index 019d00bf5adf..446cc9cc61df 100644
@@ -238,6 +238,8 @@ struct mbm_evt_value {
 
 #define RFTYPE_DEBUG			BIT(10)
 
+#define RFTYPE_ASSIGN_CONFIG		BIT(11)
+
 #define RFTYPE_CTRL_INFO		(RFTYPE_INFO | RFTYPE_CTRL)
 
 #define RFTYPE_MON_INFO			(RFTYPE_INFO | RFTYPE_MON)
fs/resctrl/monitor.c +1 -0
index 72f3dfb5b903..1f72249a5c93 100644
@@ -932,6 +932,7 @@ int resctrl_mon_resource_init(void)
 					 RFTYPE_MON_INFO | RFTYPE_RES_CACHE);
 		resctrl_file_fflags_init("available_mbm_cntrs",
 					 RFTYPE_MON_INFO | RFTYPE_RES_CACHE);
+		resctrl_file_fflags_init("event_filter", RFTYPE_ASSIGN_CONFIG);
 	}
 
 	return 0;
fs/resctrl/rdtgroup.c +0 -0
index b109e91096b0..cf84e3a382ac 100644
@@ -1911,6 +1911,25 @@ static int resctrl_available_mbm_cntrs_show(struct kernfs_open_file *of,
 	return ret;

fs/resctrl/rdtgroup.c static int resctrl_available_mbm_cntrs_show(struct kernfs_open_file *of, · line 1921

Quoted 10 lines
fs/resctrl/rdtgroup.c +8 -0
 }
 
+static int event_filter_show(struct kernfs_open_file *of, struct seq_file *seq, void *v)
+{
+	struct mon_evt *mevt = rdt_kn_parent_priv(of->kn);
+	bool sep = false;
+	int i;
+
+	for (i = 0; i < NUM_MBM_EVT_VALUES; i++) {
+		if (mevt->evt_cfg & mbm_evt_values[i].evt_val) {

Still no idea where mevt->evt_cfg comes from. Patch ordering issue?

Quoted 45 lines
fs/resctrl/rdtgroup.c +31 -0
+			if (sep)
+				seq_putc(seq, ',');
+			seq_printf(seq, "%s", mbm_evt_values[i].evt_name);
+			sep = true;
+		}
+	}
+	seq_putc(seq, '\n');
+
+	return 0;
+}
+
 /* rdtgroup information files for one cache resource. */
 static struct rftype res_common_files[] = {
 	{
@@ -2035,6 +2054,12 @@ static struct rftype res_common_files[] = {
 		.seq_show	= mbm_local_bytes_config_show,
 		.write		= mbm_local_bytes_config_write,
 	},
+	{
+		.name		= "event_filter",
+		.mode		= 0444,
+		.kf_ops		= &rdtgroup_kf_single_ops,
+		.seq_show	= event_filter_show,
+	},
 	{
 		.name		= "mbm_assign_mode",
 		.mode		= 0444,
@@ -2317,6 +2342,55 @@ static int rdtgroup_mkdir_info_resdir(void *priv, char *name,
 	return ret;
 }
 
+static int resctrl_mkdir_counter_configs(struct rdt_resource *r, char *name)
+{
+	struct kernfs_node *l3_mon_kn, *kn_subdir, *kn_subdir2;
+	struct mon_evt *mevt;
+	int ret;
+
+	l3_mon_kn = kernfs_find_and_get(kn_info, name);
+	if (!l3_mon_kn)
+		return -ENOENT;
+
+	kn_subdir = kernfs_create_dir(l3_mon_kn, "counter_configs", l3_mon_kn->mode, NULL);
+	if (IS_ERR(kn_subdir)) {
+		kernfs_put(l3_mon_kn);
+		return PTR_ERR(kn_subdir);

fs/resctrl/rdtgroup.c static int rdtgroup_mkdir_info_resdir(void *priv, char *name, · line 2368

Quoted 10 lines
fs/resctrl/rdtgroup.c +10 -0
+	}
+
+	ret = rdtgroup_kn_set_ugid(kn_subdir);
+	if (ret) {
+		kernfs_put(l3_mon_kn);
+		return ret;
+	}
+
+	list_for_each_entry(mevt, &r->mon.evt_list, list) {
+		if (mevt->mbm_mode == MBM_MODE_ASSIGN) {

I do not think this "mbm_mode" is needed, resctrl_mon::mbm_cntr_assignable is already used earlier, so would for_each_mbm_event() from the telemetry work be useful here?

fs/resctrl/rdtgroup.c static int rdtgroup_mkdir_info_resdir(void *priv, char *name, · line 2373

Quoted 5 lines
fs/resctrl/rdtgroup.c +5 -0
+			kn_subdir2 = kernfs_create_dir(kn_subdir, mevt->name,
+						       kn_subdir->mode, mevt);
+			if (IS_ERR(kn_subdir2)) {
+				ret = PTR_ERR(kn_subdir2);
+				goto config_out;

"grep goto fs/resctrl/rdtgroup.c" for naming conventions.

Quoted 6 lines
fs/resctrl/rdtgroup.c +6 -0
+			}
+
+			ret = rdtgroup_kn_set_ugid(kn_subdir2);
+			if (ret)
+				goto config_out;
+

fs/resctrl/rdtgroup.c static int rdtgroup_mkdir_info_resdir(void *priv, char *name, · line 2389

Quoted 10 lines
fs/resctrl/rdtgroup.c +10 -0
+			ret = rdtgroup_add_files(kn_subdir2, RFTYPE_ASSIGN_CONFIG);
+			if (!ret)
+				kernfs_activate(kn_subdir);
+		}
+	}
+
+config_out:
+	kernfs_put(l3_mon_kn);
+	if (ret)
+		kernfs_remove(kn_subdir);

This looks unnecessary since caller does kernfs_remove() on error return. Compare with how rdtgroup_mkdir_info_resdir() handles errors.

Quoted 10 lines
fs/resctrl/rdtgroup.c +4 -0
+
+	return ret;
+}
+
 static unsigned long fflags_from_resource(struct rdt_resource *r)
 {
 	switch (r->rid) {
@@ -2363,6 +2437,12 @@ static int rdtgroup_create_info_dir(struct kernfs_node *parent_kn)
 		ret = rdtgroup_mkdir_info_resdir(r, name, fflags);
 		if (ret)

fs/resctrl/rdtgroup.c static int rdtgroup_create_info_dir(struct kernfs_node *parent_kn) · line 2448

Quoted 10 lines
fs/resctrl/rdtgroup.c +6 -0
 			goto out_destroy;
+
+		if (r->mon.mbm_cntr_assignable) {
+			ret = resctrl_mkdir_counter_configs(r, name);
+			if (ret)
+				goto out_destroy;
+		}
 	}
 
 	ret = rdtgroup_kn_set_ugid(kn_info);

Reinette

maintainer
#78 lore raw

Re: [PATCH v13 20/27] x86/resctrl: Provide interface to update the event configurations

In reply to Babu Moger

Hi Babu,

Quoted 3 lines · Babu Moger

Users can modify the event configuration by writing to the event_filter interface file. The event configurations for mbm_cntr_assign mode are located in /sys/fs/resctrl/info/event_configs/.

heh ... looks like you also started thinking that "event_configs" is a better name (also missing L3_MON).

Quoted 34 lines

Update the assignments of all groups when the event configuration is modified.

Example:

$ cd /sys/fs/resctrl/
$ cat info/L3_MON/counter_configs/mbm_local_bytes/event_filter
  local_reads,local_non_temporal_writes,local_reads_slow_memory
$ echo "local_reads,local_non_temporal_writes" >
  info/L3_MON/counter_configs/mbm_total_bytes/event_filter
$ cat info/L3_MON/counter_configs/mbm_total_bytes/event_filter
  local_reads,local_non_temporal_writes

---
v13: Updated changelog for imperative mode.
Added function description in the prototype.
Updated the user doc resctrl.rst to address few feedback.
Resolved conflicts caused by the recent FS/ARCH code restructure.
The rdtgroup.c/monitor.c file has now been split between the FS and ARCH directories.

v12: New patch to modify event configurations.

---
 Documentation/filesystems/resctrl.rst |  12 +++
 fs/resctrl/rdtgroup.c                 | 120 +++++++++++++++++++++++++-
 2 files changed, 131 insertions(+), 1 deletion(-)
Documentation/filesystems/resctrl.rst +0 -0
index 4eb9f007ba3d..9923276826db 100644

...

fs/resctrl/rdtgroup.c static int event_filter_show(struct kernfs_open_file *of, struct seq_file *seq, · line 1935

Quoted 11 lines
fs/resctrl/rdtgroup.c +3 -0
index cf84e3a382ac..8c498b41be5d 100644
@@ -1930,6 +1930,123 @@ static int event_filter_show(struct kernfs_open_file *of, struct seq_file *seq,
 	return 0;
 }
 
+/**
+ * resctrl_group_assign - Update the counter assignments for the event in
+ *			  a group.

This name is very generic with an unexpected namespace. "rdtgroup_" prefix is often used for a function that operates on a rdtgroup. This can thus be "rdtgroup_assign_cntr()".

Quoted 11 lines
fs/resctrl/rdtgroup.c +11 -0
+ * @r:		Resource to which update needs to be done.
+ * @rdtgrp:	Resctrl group.
+ * @evtid:	Event ID.
+ * @evt_cfg:	Event configuration value.
+ */
+static int resctrl_group_assign(struct rdt_resource *r, struct rdtgroup *rdtgrp,
+				enum resctrl_event_id evtid, u32 evt_cfg)
+{
+	struct rdt_mon_domain *d;
+	int cntr_id;
+

fs/resctrl/rdtgroup.c static int event_filter_show(struct kernfs_open_file *of, struct seq_file *seq, · line 1956

Quoted 10 lines
fs/resctrl/rdtgroup.c +10 -0
+	list_for_each_entry(d, &r->mon_domains, hdr.list) {
+		cntr_id = mbm_cntr_get(r, d, rdtgrp, evtid);
+		if (cntr_id >= 0 && d->cntr_cfg[cntr_id].evt_cfg != evt_cfg) {
+			d->cntr_cfg[cntr_id].evt_cfg = evt_cfg;
+			resctrl_arch_config_cntr(r, d, evtid, rdtgrp->mon.rmid,
+						 rdtgrp->closid, cntr_id, evt_cfg, true);
+		}
+	}
+
+	return 0;

Can just return void?

fs/resctrl/rdtgroup.c static int event_filter_show(struct kernfs_open_file *of, struct seq_file *seq, · line 1961

Quoted 5 lines
fs/resctrl/rdtgroup.c +5 -0
+}
+
+/**
+ * resctrl_update_assign - Update the counter assignments for the event for all
+ *			   the groups.

Again very generic with "update" and "assign" that seem redundant? How about "resctrl_assign_cntr_allrdtgrp()"?

fs/resctrl/rdtgroup.c static int event_filter_show(struct kernfs_open_file *of, struct seq_file *seq, · line 1939

Quoted 3 lines
fs/resctrl/rdtgroup.c +3 -0
+ * @r:		Resource to which update needs to be done.
+ * @evtid:	Event ID.
+ * @evt_cfg:	Event configuration value.

Why are both event ID and evt_cfg needed? Could just passing mon_evt simplify this?

fs/resctrl/rdtgroup.c static int event_filter_show(struct kernfs_open_file *of, struct seq_file *seq, · line 1971

Quoted 7 lines
fs/resctrl/rdtgroup.c +7 -0
+ */
+static int resctrl_update_assign(struct rdt_resource *r, enum resctrl_event_id evtid,
+				 u32 evt_cfg)
+{
+	struct rdtgroup *prgrp, *crgrp;
+
+	/* Check if the cntr_id is associated to the event type updated */

Comment does not match code.

fs/resctrl/rdtgroup.c static int event_filter_show(struct kernfs_open_file *of, struct seq_file *seq, · line 1977

Quoted 6 lines
fs/resctrl/rdtgroup.c +6 -0
+	list_for_each_entry(prgrp, &rdt_all_groups, rdtgroup_list) {
+		resctrl_group_assign(r, prgrp, evtid, evt_cfg);
+
+		list_for_each_entry(crgrp, &prgrp->mon.crdtgrp_list, mon.crdtgrp_list) {
+			resctrl_group_assign(r, crgrp, evtid, evt_cfg);
+		}

Unnecessary braces?

fs/resctrl/rdtgroup.c static int event_filter_show(struct kernfs_open_file *of, struct seq_file *seq, · line 1956

Quoted 3 lines
fs/resctrl/rdtgroup.c +3 -0
+	}
+
+	return 0;

return void?

fs/resctrl/rdtgroup.c static int event_filter_show(struct kernfs_open_file *of, struct seq_file *seq, · line 1993

Quoted 13 lines
fs/resctrl/rdtgroup.c +13 -0
+}
+
+static int resctrl_process_configs(char *tok, u32 *val)
+{
+	char *evt_str;
+	bool found;
+	int i;
+
+next_config:
+	if (!tok || tok[0] == '\0')
+		return 0;
+
+	/* Start processing the strings for each event type */

Does comment intend to describe one iteration or all iterations? Also, "event type" -> "memory transaction"?

fs/resctrl/rdtgroup.c static int event_filter_show(struct kernfs_open_file *of, struct seq_file *seq, · line 1998

Quoted 5 lines
fs/resctrl/rdtgroup.c +5 -0
+	evt_str = strim(strsep(&tok, ","));
+	found = false;
+	for (i = 0; i < NUM_MBM_EVT_VALUES; i++) {
+		if (!strcmp(mbm_evt_values[i].evt_name, evt_str)) {
+			*val |=  mbm_evt_values[i].evt_val;

check spacing.

fs/resctrl/rdtgroup.c static int event_filter_show(struct kernfs_open_file *of, struct seq_file *seq, · line 2006

Quoted 8 lines
fs/resctrl/rdtgroup.c +8 -0
+			found = true;
+			break;
+		}
+	}
+
+	if (!found) {
+		rdt_last_cmd_printf("Invalid event type %s\n", evt_str);
+		return -EINVAL;

Looks like this will return partially initialized data. Please use a local variable in which to gather the new configuration and only assign that to provided pointer on success.

Quoted 18 lines
fs/resctrl/rdtgroup.c +18 -0
+	}
+
+	goto next_config;
+}
+
+static ssize_t event_filter_write(struct kernfs_open_file *of, char *buf,
+				  size_t nbytes, loff_t off)
+{
+	struct rdt_resource *r = resctrl_arch_get_resource(RDT_RESOURCE_L3);
+	struct mon_evt *mevt = rdt_kn_parent_priv(of->kn);
+	u32 evt_cfg = 0;
+	int ret = 0;
+
+	/* Valid input requires a trailing newline */
+	if (nbytes == 0 || buf[nbytes - 1] != '\n')
+		return -EINVAL;
+
+	buf[nbytes - 1] = '\0';

fs/resctrl/rdtgroup.c static int event_filter_show(struct kernfs_open_file *of, struct seq_file *seq, · line 2034

Quoted 10 lines
fs/resctrl/rdtgroup.c +10 -0
+
+	cpus_read_lock();
+	mutex_lock(&rdtgroup_mutex);
+
+	rdt_last_cmd_clear();
+
+	if (!resctrl_arch_mbm_cntr_assign_enabled(r)) {
+		rdt_last_cmd_puts("mbm_cntr_assign mode is not enabled\n");
+		ret = -EINVAL;
+		goto unlock_out;

"grep goto fs/resctrl/rdtgroup.c"

fs/resctrl/rdtgroup.c static int event_filter_show(struct kernfs_open_file *of, struct seq_file *seq, · line 2039

Quoted 5 lines
fs/resctrl/rdtgroup.c +5 -0
+	}
+
+	ret = resctrl_process_configs(buf, &evt_cfg);
+	if (!ret && mevt->evt_val != evt_cfg) {
+		mevt->evt_val = evt_cfg;

ah ... here it is. hmmm ... but it is mon_evt::evt_cfg, no? ah, fixed in next patch.

I still seem to be missing something because I expected mon_evt::evt_cfg of mbm_total_bytes and mbm_local_bytes to be initialized with a starting default. I missed where this is done in this series.

Quoted 15 lines
fs/resctrl/rdtgroup.c +10 -0
+		resctrl_update_assign(r, mevt->evtid, evt_cfg);
+	}
+
+unlock_out:
+	mutex_unlock(&rdtgroup_mutex);
+	cpus_read_unlock();
+
+	return ret ?: nbytes;
+}
+
 /* rdtgroup information files for one cache resource. */
 static struct rftype res_common_files[] = {
 	{
@@ -2056,9 +2173,10 @@ static struct rftype res_common_files[] = {
 	},

fs/resctrl/rdtgroup.c static struct rftype res_common_files[] = { · line 2182

Quoted 10 lines
fs/resctrl/rdtgroup.c +2 -1
 	{
 		.name		= "event_filter",
-		.mode		= 0444,
+		.mode		= 0644,
 		.kf_ops		= &rdtgroup_kf_single_ops,
 		.seq_show	= event_filter_show,
+		.write		= event_filter_write,
 	},
 	{
 		.name		= "mbm_assign_mode",

Reinette

maintainer
#87 lore raw

Re: [PATCH v13 23/27] x86/resctrl: Introduce mbm_L3_assignments to list assignments in a group

In reply to Babu Moger

Hi Babu,

Quoted 5 lines · Babu Moger

Introduce the interface to display the assignment states for each group when mbm_cntr_assign mode is enabled.

The list is displayed in the following format:
<Event configuration>:<Domain id>=<Assignment type>

Should this just be <Event>? The information is just the event name, not its configuration that will be in the "event_filter" file.

Quoted 39 lines

Event configuration: A valid event configuration listed in the /sys/fs/resctrl/info/L3_MON/counter_configs directory.

Domain ID: A valid domain ID number.

The assignment type can be one of the following:

_ : No event configuration assigned

e : Event configuration assigned in exclusive mode

Example:
$cd /sys/fs/resctrl
$cat mbm_L3_assignments
mbm_total_bytes:0=e;1=e
mbm_local_bytes:0=e;1=e

---
v13: Changelog update.
Few changes in mbm_L3_assignments_show() after moving the event config to evt_list. Resolved conflicts caused by the recent FS/ARCH code restructure.
The rdtgroup.c/monitor.c files have been split between the FS and ARCH directories.

v12: New patch:
Assignment interface moved inside the group based the discussion
https://lore.kernel.org/lkml/CALPaoCiii0vXOF06mfV=kVLBzhfNo0SFqt4kQGwGSGVUqvr2Dg@mail.gmail.com/#t

---
 Documentation/filesystems/resctrl.rst | 28 +++++++++++++++
 fs/resctrl/monitor.c                  |  1 +
 fs/resctrl/rdtgroup.c                 | 52 +++++++++++++++++++++++++++
 3 files changed, 81 insertions(+)
Documentation/filesystems/resctrl.rst +0 -0
index 356f1f918a86..2350c1f21f4e 100644

Documentation/filesystems/resctrl.rst When the "mba_MBps" mount option is used all CTRL_MON groups will also contain: · line 508

Quoted 5 lines
Documentation/filesystems/resctrl.rst +2 -0
@@ -504,6 +504,34 @@ When the "mba_MBps" mount option is used all CTRL_MON groups will also contain:
 	/sys/fs/resctrl/info/L3_MON/mon_features changes the input
 	event.
 
+"mbm_L3_assignments":
+	This interface file is created when the mbm_cntr_assign mode is supported

"This interface file is created when" -> "Exists when mbm_cntr_assign mode is supported"?

Documentation/filesystems/resctrl.rst When the "mba_MBps" mount option is used all CTRL_MON groups will also contain: · line 509

Quoted 1 line
Documentation/filesystems/resctrl.rst +1 -0
+	and shows the assignment status for each group.

This doc is in the portion documenting files in monitor groups. So rather:
"the assignment status for each group" -> "the counter assignment status for the MON group"?

Documentation/filesystems/resctrl.rst When the "mba_MBps" mount option is used all CTRL_MON groups will also contain: · line 513

Quoted 4 lines
Documentation/filesystems/resctrl.rst +4 -0
+
+	The assignment list is displayed in the following format:
+
+	<Event configuration>:<Domain id>=<Assignment type>

<Event configuration> -> <Event>

Documentation/filesystems/resctrl.rst When the "mba_MBps" mount option is used all CTRL_MON groups will also contain: · line 515

Quoted 2 lines
Documentation/filesystems/resctrl.rst +2 -0
+
+	Event configuration: A valid event configuration listed in the

"A valid event in the /sys/fs/resctrl/info/L3_MON/event_configs directory"

Documentation/filesystems/resctrl.rst When the "mba_MBps" mount option is used all CTRL_MON groups will also contain: · line 518

Quoted 3 lines
Documentation/filesystems/resctrl.rst +3 -0
+	/sys/fs/resctrl/info/L3_MON/counter_configs directory.
+
+	Domain ID: A valid domain ID number.

"A valid domain ID"

Documentation/filesystems/resctrl.rst When the "mba_MBps" mount option is used all CTRL_MON groups will also contain: · line 522

Quoted 4 lines
Documentation/filesystems/resctrl.rst +4 -0
+
+	Assignment types:
+
+	_ : No event configuration assigned

hmmm ... since the line has event as first field, would this not reflect the counter? That is "No counter assigned"

Documentation/filesystems/resctrl.rst When the "mba_MBps" mount option is used all CTRL_MON groups will also contain: · line 524

Quoted 2 lines
Documentation/filesystems/resctrl.rst +2 -0
+
+	e : Event configuration assigned in exclusive mode

"Counter assigned exclusively"? (with exclusive defined somewhere)

Documentation/filesystems/resctrl.rst When the "mba_MBps" mount option is used all CTRL_MON groups will also contain: · line 527

Quoted 3 lines
Documentation/filesystems/resctrl.rst +3 -0
+
+	Example:
+	To list the assignment states for the default group.

"the counter assignment states"?

Quoted 32 lines
Documentation/filesystems/resctrl.rst +7 -0
+	::
+
+	  # cd /sys/fs/resctrl
+	  # cat mbm_L3_assignments
+	    mbm_total_bytes:0=e;1=e
+	    mbm_local_bytes:0=e;1=e
+
 Resource allocation rules
 -------------------------
 
fs/resctrl/monitor.c +1 -0
index 5f6c4b662f3b..b982540ce4e3 100644
@@ -935,6 +935,7 @@ int resctrl_mon_resource_init(void)
 		resctrl_file_fflags_init("event_filter", RFTYPE_ASSIGN_CONFIG);
 		resctrl_file_fflags_init("mbm_assign_on_mkdir", RFTYPE_MON_INFO |
 					 RFTYPE_RES_CACHE);
+		resctrl_file_fflags_init("mbm_L3_assignments", RFTYPE_MON_BASE);
 	}
 
 	return 0;
fs/resctrl/rdtgroup.c +2 -0
index 931ea355f159..8d970b99bbbd 100644
@@ -2080,6 +2080,52 @@ static ssize_t resctrl_mbm_assign_on_mkdir_write(struct kernfs_open_file *of,
 	return ret ?: nbytes;
 }
 
+static int mbm_L3_assignments_show(struct kernfs_open_file *of, struct seq_file *s, void *v)
+{

fs/resctrl/rdtgroup.c static ssize_t resctrl_mbm_assign_on_mkdir_write(struct kernfs_open_file *of, · line 2094

Quoted 10 lines
fs/resctrl/rdtgroup.c +10 -0
+	struct rdt_resource *r = resctrl_arch_get_resource(RDT_RESOURCE_L3);
+	struct rdt_mon_domain *d;
+	struct rdtgroup *rdtgrp;
+	struct mon_evt *mevt;
+	int ret = 0;
+	bool sep;
+
+	rdtgrp = rdtgroup_kn_lock_live(of->kn);
+	if (!rdtgrp)
+		return -ENOENT;

Missing a rdtgroup_kn_unlock()?

fs/resctrl/rdtgroup.c static ssize_t resctrl_mbm_assign_on_mkdir_write(struct kernfs_open_file *of, · line 2100

Quoted 6 lines
fs/resctrl/rdtgroup.c +6 -0
+
+	rdt_last_cmd_clear();
+	if (!resctrl_arch_mbm_cntr_assign_enabled(r)) {
+		rdt_last_cmd_puts("mbm_cntr_assign mode not enabled\n");
+		ret = -ENOENT;
+		goto assign_out;

grep goto fs/resctrl/rdtgroup.c

fs/resctrl/rdtgroup.c static ssize_t resctrl_mbm_assign_on_mkdir_write(struct kernfs_open_file *of, · line 2103

Quoted 3 lines
fs/resctrl/rdtgroup.c +3 -0
+	}
+
+	list_for_each_entry(mevt, &r->mon.evt_list, list) {

can use for_each_mbm_event() and then below will not be needed?

Quoted 31 lines
fs/resctrl/rdtgroup.c +25 -0
+		if (mevt->mbm_mode != MBM_MODE_ASSIGN)
+			continue;
+
+		sep = false;
+		seq_printf(s, "%s:", mevt->name);
+		list_for_each_entry(d, &r->mon_domains, hdr.list) {
+			if (sep)
+				seq_putc(s, ';');
+
+			if (mbm_cntr_get(r, d, rdtgrp, mevt->evtid) >= 0)
+				seq_printf(s, "%d=e", d->hdr.id);
+			else
+				seq_printf(s, "%d=_", d->hdr.id);
+
+			sep = true;
+		}
+		seq_putc(s, '\n');
+	}
+
+assign_out:
+	rdtgroup_kn_unlock(of->kn);
+
+	return ret;
+}
+
 /* rdtgroup information files for one cache resource. */
 static struct rftype res_common_files[] = {
 	{
@@ -2218,6 +2264,12 @@ static struct rftype res_common_files[] = {
 		.seq_show	= event_filter_show,
 		.write		= event_filter_write,

fs/resctrl/rdtgroup.c static struct rftype res_common_files[] = { · line 2275

Quoted 10 lines
fs/resctrl/rdtgroup.c +6 -0
 	},
+	{
+		.name		= "mbm_L3_assignments",
+		.mode		= 0444,
+		.kf_ops		= &rdtgroup_kf_single_ops,
+		.seq_show	= mbm_L3_assignments_show,
+	},
 	{
 		.name		= "mbm_assign_mode",
 		.mode		= 0444,

Reinette

maintainer
#81 lore raw

Re: [PATCH v13 21/27] x86/resctrl: Introduce mbm_assign_on_mkdir to configure assignments

In reply to Babu Moger

Hi Babu,

Quoted 5 lines · Babu Moger

The mbm_cntr_assign mode provides an option to the user to assign a counter to an RMID, event pair and monitor the bandwidth as long as the counter is assigned.

Introduce a configuration option to automatically assign counter IDs

"assign counter IDs" -> "assign counter IDs to <what?>"

Quoted 24 lines

when a resctrl group is created, provided the counters are available. By default, this option is enabled at boot.

---
v13: Added Suggested-by tag.
Resolved conflicts caused by the recent FS/ARCH code restructure.
The rdtgroup.c/monitor.c file has now been split between the FS and ARCH directories.

v12: New patch. Added after the discussion on the list.
https://lore.kernel.org/lkml/CALPaoCh8siZKjL_3yvOYGL4cF_n_38KpUFgHVGbQ86nD+Q2_SA@mail.gmail.com/

---
 Documentation/filesystems/resctrl.rst | 10 ++++++
 fs/resctrl/monitor.c                  |  2 ++
 fs/resctrl/rdtgroup.c                 | 44 +++++++++++++++++++++++++--
 include/linux/resctrl.h               |  2 ++
 4 files changed, 56 insertions(+), 2 deletions(-)
Documentation/filesystems/resctrl.rst +0 -0
index 9923276826db..356f1f918a86 100644

Documentation/filesystems/resctrl.rst with the following files: · line 352

Quoted 5 lines
Documentation/filesystems/resctrl.rst +2 -0
@@ -348,6 +348,16 @@ with the following files:
 	  # cat /sys/fs/resctrl/info/L3_MON/counter_configs/mbm_total_bytes/event_filter
 	   local_reads, local_non_temporal_writes
 
+"mbm_assign_on_mkdir":
+	Automatically assign the monitoring counters on resctrl group creation

assign the monitoring counters to what?

Documentation/filesystems/resctrl.rst with the following files: · line 359

Quoted 7 lines
Documentation/filesystems/resctrl.rst +7 -0
+	if the counters are available. It is enabled by default on boot and users
+	can disable by writing to the interface.
+	::
+
+	  # echo 0 > /sys/fs/resctrl/info/L3_MON/mbm_assign_on_mkdir
+	  # cat /sys/fs/resctrl/info/L3_MON/mbm_assign_on_mkdir
+	  0

Please be explicit in docs what possible values are and what they mean.

Quoted 22 lines
Documentation/filesystems/resctrl.rst +1 -0
+
 "max_threshold_occupancy":
 		Read/write file provides the largest value (in
 		bytes) at which a previously used LLC_occupancy
fs/resctrl/monitor.c +2 -0
index 1f72249a5c93..5f6c4b662f3b 100644
@@ -933,6 +933,8 @@ int resctrl_mon_resource_init(void)
 		resctrl_file_fflags_init("available_mbm_cntrs",
 					 RFTYPE_MON_INFO | RFTYPE_RES_CACHE);
 		resctrl_file_fflags_init("event_filter", RFTYPE_ASSIGN_CONFIG);
+		resctrl_file_fflags_init("mbm_assign_on_mkdir", RFTYPE_MON_INFO |
+					 RFTYPE_RES_CACHE);
 	}
 
 	return 0;
fs/resctrl/rdtgroup.c +0 -0
index 8c498b41be5d..0093b323d858 100644

fs/resctrl/rdtgroup.c static ssize_t event_filter_write(struct kernfs_open_file *of, char *buf, · line 2042

Quoted 10 lines
fs/resctrl/rdtgroup.c +2 -2
@@ -2035,8 +2035,8 @@ static ssize_t event_filter_write(struct kernfs_open_file *of, char *buf,
 	}
 
 	ret = resctrl_process_configs(buf, &evt_cfg);
-	if (!ret && mevt->evt_val != evt_cfg) {
-		mevt->evt_val = evt_cfg;
+	if (!ret && mevt->evt_cfg != evt_cfg) {
+		mevt->evt_cfg = evt_cfg;
 		resctrl_update_assign(r, mevt->evtid, evt_cfg);
 	}
 

Needs to be squashed.

Quoted 16 lines
fs/resctrl/rdtgroup.c +12 -0
@@ -2047,6 +2047,39 @@ static ssize_t event_filter_write(struct kernfs_open_file *of, char *buf,
 	return ret ?: nbytes;
 }
 
+static int resctrl_mbm_assign_on_mkdir_show(struct kernfs_open_file *of,
+					    struct seq_file *s, void *v)
+{
+	struct rdt_resource *r = rdt_kn_parent_priv(of->kn);
+
+	seq_printf(s, "%u\n", r->mon.mbm_assign_on_mkdir);
+
+	return 0;
+}
+
+static ssize_t resctrl_mbm_assign_on_mkdir_write(struct kernfs_open_file *of,
+						 char *buf, size_t nbytes, loff_t off)

fs/resctrl/rdtgroup.c static ssize_t event_filter_write(struct kernfs_open_file *of, char *buf, · line 2071

Quoted 10 lines
fs/resctrl/rdtgroup.c +10 -0
+{
+	struct rdt_resource *r = rdt_kn_parent_priv(of->kn);
+	bool value;
+	int ret;
+
+	ret = kstrtobool(buf, &value);
+	if (ret)
+		return ret;
+
+	cpus_read_lock();

not traversing the domain list so hotplug lock not needed.

fs/resctrl/rdtgroup.c static ssize_t event_filter_write(struct kernfs_open_file *of, char *buf, · line 2072

Quoted 1 line
fs/resctrl/rdtgroup.c +1 -0
+	mutex_lock(&rdtgroup_mutex);

rdtgroup_mutex seems only needed because the message buffer is cleared below, and this is why it is not required in the show()?

Quoted 32 lines
fs/resctrl/rdtgroup.c +17 -0
+	rdt_last_cmd_clear();
+
+	r->mon.mbm_assign_on_mkdir = value;
+
+	mutex_unlock(&rdtgroup_mutex);
+	cpus_read_unlock();
+
+	return ret ?: nbytes;
+}
+
 /* rdtgroup information files for one cache resource. */
 static struct rftype res_common_files[] = {
 	{
@@ -2056,6 +2089,13 @@ static struct rftype res_common_files[] = {
 		.seq_show	= rdt_last_cmd_status_show,
 		.fflags		= RFTYPE_TOP_INFO,
 	},
+	{
+		.name		= "mbm_assign_on_mkdir",
+		.mode		= 0644,
+		.kf_ops		= &rdtgroup_kf_single_ops,
+		.seq_show	= resctrl_mbm_assign_on_mkdir_show,
+		.write		= resctrl_mbm_assign_on_mkdir_write,
+	},
 	{
 		.name		= "num_closids",
 		.mode		= 0444,
include/linux/resctrl.h +0 -0
index cd24d1577e0a..d6435abdde7b 100644

include/linux/resctrl.h enum resctrl_schema_fmt { · line 281

Quoted 4 lines
include/linux/resctrl.h +1 -0
@@ -278,6 +278,7 @@ enum resctrl_schema_fmt {
  *			monitoring events can be configured.
  * @num_mbm_cntrs:	Number of assignable monitoring counters
  * @mbm_cntr_assignable:Is system capable of supporting monitor assignment?
+ * @mbm_assign_on_mkdir:Auto enable monitor assignment on mkdir?

How is "monitor assignment" different from "counter assignment"?

include/linux/resctrl.h struct resctrl_mon { · line 292

Quoted 11 lines
include/linux/resctrl.h +1 -0
  * @evt_list:		List of monitoring events
  */
 struct resctrl_mon {
@@ -285,6 +286,7 @@ struct resctrl_mon {
 	unsigned int		mbm_cfg_mask;
 	int			num_mbm_cntrs;
 	bool			mbm_cntr_assignable;
+	bool			mbm_assign_on_mkdir;
 	struct list_head	evt_list;
 };
 

Reinette

maintainer
#35 lore raw

Re: [PATCH v13 11/27] x86/resctrl: Implement resctrl_arch_config_cntr() to assign a counter with ABMC

In reply to Luck, Tony
Quoted 10 lines · Tony
Quoted 5 lines

It looks to me as though there are a couple of changes in the telemetry work
that would benefit this work. https://lore.kernel.org/lkml/[email protected]/
switches the monitor events to be maintained in an array indexed by event ID, eliminating the need for searching the evt_list that this work does in a couple of places. Also note the handy new for_each_mbm_event() helper (https://lore.kernel.org/lkml/[email protected]/).

Yesterday I ran through the exercise of rebasing my AET patches on top of these ABMC patches in order to check whether the ABMC patches painted resctrl into some corner that would be hard to get back out of.

Quoted 8 lines

Good news: they don't.

There was a bunch of manual patching to make the first four patches fit on top of the ABMC code, but I also noticed a few places where things were simpler after combining the two series.

Maybe a good path forward would be to take those first four patches from my AET series and then build ABMC on top of those.

As an encouragement to try this direction, I took my four patches on top of tip x86/cache and then applied Babu's ABMC series.

Changes to Babu's code:
1) Adapt where needed for removal of evt_list. Use event array instead.
2) Use for_each_mbm_event() [Maybe didn't get all places?]
3) Bring the s/evt_val/evt_cfg/ fix into patch 20 from 21
4) Fix fir tree declaration for resctrl_process_assign()

I don't have an AMD system to check if the ABMC parts still work. But it does pass the resctrl self tests, so legacy isn't broken.

Patches in the "my_mbm_plus_babu_abmc" branch of my kernel.org
repo: git://git.kernel.org/pub/scm/linux/kernel/git/aegl/linux.git

-Tony

#90 lore raw

Re: [PATCH v13 24/27] x86/resctrl: Introduce the interface to modify assignments in a group

In reply to Babu Moger

Hi Babu,

Quoted 19 lines · Babu Moger
fs/resctrl/rdtgroup.c +11 -0
index 8d970b99bbbd..ea1782723f81 100644
@@ -2126,6 +2126,168 @@ static int mbm_L3_assignments_show(struct kernfs_open_file *of, struct seq_file
        return ret;
 }
 
+/*
+ * mbm_get_mon_event_by_name() - Return the mon_evt entry for the matching
+ * event name.
+ */
+static struct mon_evt *mbm_get_mon_event_by_name(struct rdt_resource *r,
+                                                char *name)
+{
+       struct mon_evt *mevt;
+
+       list_for_each_entry(mevt, &r->mon.evt_list, list) {
+               if (!strcmp(mevt->name, name))

fs/resctrl/rdtgroup.c static int mbm_L3_assignments_show(struct kernfs_open_file *of, struct seq_file · line 2149

Quoted 10 lines
fs/resctrl/rdtgroup.c +10 -0
+                       return mevt;
+       }
+
+       return NULL;
+}
+
+static unsigned int resctrl_get_assing_type(char *assign)
+{
+       unsigned int mon_state = ASSIGN_NONE;
+       int len = strlen(assign);
[  395.013183] BUG: kernel NULL pointer dereference, address: 0000000000000000
[  395.013426] #PF: supervisor read access in kernel mode
[  395.013600] #PF: error_code(0x0000) - not-present page
[  395.013779] PGD 39322c067 P4D 2a4f49067 PUD 2a4f4a067 PMD 0
[  395.013973] Oops: Oops: 0000 [#1] SMP DEBUG_PAGEALLOC NOPTI
[  395.014156] CPU: 37 UID: 0 PID: 24147 Comm: bash Not tainted
6.15.0-dbg-DEV #13 NONE
[  395.014403] Hardware name: Google Astoria-Turin/astoria, BIOS
0.20241223.2-0 01/17/2025
[  395.014652] RIP: 0010:strlen+0xb/0x20
[  395.014778] Code: 66 66 66 66 66 66 2e 0f 1f 84 00 00 00 00 00 90
90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 f3 0f 1e fa 48 c7 c0 ff
ff ff ff <80> 7c 07 01 00 48 8d 40 01 75 f5 c3 cc cc cc cc cc 0f 1f 40
00 90
[  395.015356] RSP: 0018:ffa000002f743d58 EFLAGS: 00010246
[  395.015522] RAX: ffffffffffffffff RBX: ff11000129a00600 RCX: 0000000000000000
[  395.015747] RDX: ff110001299f5253 RSI: ffffffff827b9651 RDI: 0000000000000000
[  395.015968] RBP: 0000000000000000 R08: 000000000000003d R09: 0000000000000000
[  395.016202] R10: ffffffff827b9652 R11: 0000000000000000 R12: ffffffff8305b7f8
[  395.016421] R13: ff110001299f5240 R14: 0000000000000014 R15: 0000000000000000
[  395.016644] FS:  00007f1281ff8b80(0000) GS:ff1100bdc8276000(0000)
knlGS:0000000000000000
[  395.016893] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  395.017071] CR2: 0000000000000000 CR3: 0000000420bc8002 CR4: 0000000000771ef0
[  395.017298] PKRU: 55555554
[  395.017388] Call Trace:
[  395.017471]  <TASK>
[  395.017545]  mbm_L3_assignments_write+0x2d4/0x4e0
[  395.017700]  kernfs_fop_write_iter+0x132/0x1c0
[  395.017851]  vfs_write+0x2bf/0x3c0
[  395.017963]  ksys_write+0x82/0x100
[  395.018074]  do_syscall_64+0xee/0x210
[  395.018198]  ? exc_page_fault+0x81/0xe0
[  395.018321]  entry_SYSCALL_64_after_hwframe+0x77/0x7f
[  395.018482] RIP: 0033:0x7f128177f8b3
[  395.018598] Code: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc
cc cc cc cc cc cc cc 48 8b 05 99 91 07 00 83 38 00 75 10 b8 01 00 00
00 0f 05 <48> 3d 01 f0 ff ff 73 4d c3 55 48 89 e5 41 57 41 56 53 50 48
89 d3
[  395.019167] RSP: 002b:00007ffff66e80f8 EFLAGS: 00000246 ORIG_RAX:
0000000000000001
[  395.019409] RAX: ffffffffffffffda RBX: 0000000000000014 RCX: 00007f128177f8b3
[  395.019636] RDX: 0000000000000014 RSI: 0000000001eedb60 RDI: 0000000000000001
[  395.019861] RBP: 00007ffff66e8120 R08: 0000000000000000 R09: 0000000000000000
[  395.020081] R10: 00007ffff66e81b0 R11: 0000000000000246 R12: 0000000001eedb60
[  395.020303] R13: 0000000000000001 R14: 00007f12817fa650 R15: 0000000000000014
[  395.020532]  </TASK>
Quoted 35 lines
fs/resctrl/rdtgroup.c +35 -0
+
+       if (!len || len > 1)
+               return ASSIGN_INVALID;
+
+       switch (*assign) {
+       case 'e':
+               mon_state = ASSIGN_EXCLUSIVE;
+               break;
+       case '_':
+               mon_state = ASSIGN_NONE;
+               break;
+       default:
+               mon_state = ASSIGN_INVALID;
+               break;
+       }
+
+       return mon_state;
+}
+
+static int resctrl_process_assign(struct rdt_resource *r, struct rdtgroup *rdtgrp,
+                                 char *config, char *tok)
+{
+       struct rdt_mon_domain *d;
+       char *dom_str, *id_str;
+       unsigned long dom_id = 0;
+       struct mon_evt *mevt;
+       int assign_type;
+       char domain[10];
+       bool found;
+       int ret;
+
+       mevt = mbm_get_mon_event_by_name(r, config);
+       if (!mevt) {
+               rdt_last_cmd_printf("Invalid assign configuration %s\n", config);
+               return  -ENOENT;

fs/resctrl/rdtgroup.c static int mbm_L3_assignments_show(struct kernfs_open_file *of, struct seq_file · line 2194

Quoted 10 lines
fs/resctrl/rdtgroup.c +10 -0
+       }
+
+next:
+       if (!tok || tok[0] == '\0')
+               return 0;
+
+       /* Start processing the strings for each domain */
+       dom_str = strim(strsep(&tok, ";"));
+
+       id_str = strsep(&dom_str, "=");

If there's no '=' then dom_str becomes NULL...

Quoted 16 lines
fs/resctrl/rdtgroup.c +16 -0
+
+       /* Check for domain id '*' which means all domains */
+       if (id_str && *id_str == '*') {
+               d = NULL;
+               goto check_state;
+       } else if (!id_str || kstrtoul(id_str, 10, &dom_id)) {
+               rdt_last_cmd_puts("Missing domain id\n");
+               return -EINVAL;
+       }
+
+       /* Verify if the dom_id is valid */
+       found = false;
+       list_for_each_entry(d, &r->mon_domains, hdr.list) {
+               if (d->hdr.id == dom_id) {
+                       found = true;
+                       break;

fs/resctrl/rdtgroup.c static int mbm_L3_assignments_show(struct kernfs_open_file *of, struct seq_file · line 2220

Quoted 10 lines
fs/resctrl/rdtgroup.c +10 -0
+               }
+       }
+
+       if (!found) {
+               rdt_last_cmd_printf("Invalid domain id %ld\n", dom_id);
+               return -EINVAL;
+       }
+
+check_state:
+       assign_type = resctrl_get_assing_type(dom_str);

then the resulting type of whatever this is supposed to mean is "panic"

Thanks,
-Peter

#36 lore raw

Re: [PATCH v13 11/27] x86/resctrl: Implement resctrl_arch_config_cntr() to assign a counter with ABMC

In reply to Luck, Tony

Hi Tony,

Quoted 18 lines · Tony
Quoted 16 lines · Tony
Quoted 5 lines

It looks to me as though there are a couple of changes in the telemetry work
that would benefit this work. https://lore.kernel.org/lkml/[email protected]/
switches the monitor events to be maintained in an array indexed by event ID, eliminating the need for searching the evt_list that this work does in a couple of places. Also note the handy new for_each_mbm_event() helper (https://lore.kernel.org/lkml/[email protected]/).

Yesterday I ran through the exercise of rebasing my AET patches on top of these ABMC patches in order to check whether the ABMC patches painted resctrl into some corner that would be hard to get back out of.

Good news: they don't.

There was a bunch of manual patching to make the first four patches fit on top of the ABMC code, but I also noticed a few places where things were simpler after combining the two series.

Quoted 5 lines
Quoted 2 lines

Maybe a good path forward would be to take those first four patches from my AET series and then build ABMC on top of those.

As an encouragement to try this direction, I took my four patches on top of tip x86/cache and then applied Babu's ABMC series.

I did the same thing last week, except in the other order, so I switched to your changes to test.

Quoted 7 lines

Changes to Babu's code:
1) Adapt where needed for removal of evt_list. Use event array instead.
2) Use for_each_mbm_event() [Maybe didn't get all places?]
3) Bring the s/evt_val/evt_cfg/ fix into patch 20 from 21
4) Fix fir tree declaration for resctrl_process_assign()

Quoted 5 lines

I don't have an AMD system to check if the ABMC parts still work. But it does pass the resctrl self tests, so legacy isn't broken.

Patches in the "my_mbm_plus_babu_abmc" branch of my kernel.org
repo: git://git.kernel.org/pub/scm/linux/kernel/git/aegl/linux.git

Thanks for applying my suggestion[1] about the array entry sizes, but you needed one more dereference:

(quoted diff) +0 -0

b/arch/x86/kernel/cpu/resctrl/core.c
index 1db6a61e27746..0c27e0a5a7b96 100644

arch/x86/kernel/cpu/resctrl/core.c +0 -0
@@ -399,7 +399,7 @@ static int domain_setup_ctrlval(struct

rdt_resource *r, struct rdt_ctrl_domain *
*/
static int arch_domain_mbm_alloc(u32 num_rmid, struct
rdt_hw_mon_domain *hw_dom)
{
- size_t tsize = sizeof(hw_dom->arch_mbm_states[0]);

+       size_t tsize = sizeof(*hw_dom->arch_mbm_states[0]);
        enum resctrl_event_id evt;
        int idx;
fs/resctrl/rdtgroup.c +0 -0
index 098ff002d2232..44ec33cb165f7 100644
@@ -4819,7 +4823,7 @@ void resctrl_offline_mon_domain(struct

rdt_resource *r, struct rdt_mon_domain *d
static int domain_setup_mon_state(struct rdt_resource *r, struct
rdt_mon_domain *d)

 {
        u32 idx_limit = resctrl_arch_system_num_rmid_idx();
-       size_t tsize = sizeof(d->mbm_states[0]);
+       size_t tsize = sizeof(*d->mbm_states[0]);
        enum resctrl_event_id evt;
        int idx;

You should be able to repro an array overrun without ABMC, and a page fault is likely if the system implements a lot of RMIDs. The AMD EPYC 9B45 I tested on implements 4096 RMIDs.

Thanks,
-Peter

[1] https://lore.kernel.org/lkml/CALPaoCj8yfzJ=5CkxTPQXc0-WRWpu0xKRX8v4FAWFGQKtXtMUw@mail.gmail.com/

author reviewer
#91 lore raw

Re: [PATCH v13 24/27] x86/resctrl: Introduce the interface to modify assignments in a group

In reply to Peter Newman

Hi Peter,

Quoted 73 lines · Peter Newman

Hi Babu,

Quoted 19 lines · Babu Moger
fs/resctrl/rdtgroup.c +11 -0
index 8d970b99bbbd..ea1782723f81 100644
@@ -2126,6 +2126,168 @@ static int mbm_L3_assignments_show(struct kernfs_open_file *of, struct seq_file
        return ret;
 }
 
+/*
+ * mbm_get_mon_event_by_name() - Return the mon_evt entry for the matching
+ * event name.
+ */
+static struct mon_evt *mbm_get_mon_event_by_name(struct rdt_resource *r,
+                                                char *name)
+{
+       struct mon_evt *mevt;
+
+       list_for_each_entry(mevt, &r->mon.evt_list, list) {
+               if (!strcmp(mevt->name, name))

fs/resctrl/rdtgroup.c static int mbm_L3_assignments_show(struct kernfs_open_file *of, struct seq_file · line 2149

Quoted 10 lines
fs/resctrl/rdtgroup.c +10 -0
+                       return mevt;
+       }
+
+       return NULL;
+}
+
+static unsigned int resctrl_get_assing_type(char *assign)
+{
+       unsigned int mon_state = ASSIGN_NONE;
+       int len = strlen(assign);
[  395.013183] BUG: kernel NULL pointer dereference, address: 0000000000000000
[  395.013426] #PF: supervisor read access in kernel mode
[  395.013600] #PF: error_code(0x0000) - not-present page
[  395.013779] PGD 39322c067 P4D 2a4f49067 PUD 2a4f4a067 PMD 0
[  395.013973] Oops: Oops: 0000 [#1] SMP DEBUG_PAGEALLOC NOPTI
[  395.014156] CPU: 37 UID: 0 PID: 24147 Comm: bash Not tainted
6.15.0-dbg-DEV #13 NONE
[  395.014403] Hardware name: Google Astoria-Turin/astoria, BIOS
0.20241223.2-0 01/17/2025
[  395.014652] RIP: 0010:strlen+0xb/0x20
[  395.014778] Code: 66 66 66 66 66 66 2e 0f 1f 84 00 00 00 00 00 90
90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 f3 0f 1e fa 48 c7 c0 ff
ff ff ff <80> 7c 07 01 00 48 8d 40 01 75 f5 c3 cc cc cc cc cc 0f 1f 40
00 90
[  395.015356] RSP: 0018:ffa000002f743d58 EFLAGS: 00010246
[  395.015522] RAX: ffffffffffffffff RBX: ff11000129a00600 RCX: 0000000000000000
[  395.015747] RDX: ff110001299f5253 RSI: ffffffff827b9651 RDI: 0000000000000000
[  395.015968] RBP: 0000000000000000 R08: 000000000000003d R09: 0000000000000000
[  395.016202] R10: ffffffff827b9652 R11: 0000000000000000 R12: ffffffff8305b7f8
[  395.016421] R13: ff110001299f5240 R14: 0000000000000014 R15: 0000000000000000
[  395.016644] FS:  00007f1281ff8b80(0000) GS:ff1100bdc8276000(0000)
knlGS:0000000000000000
[  395.016893] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  395.017071] CR2: 0000000000000000 CR3: 0000000420bc8002 CR4: 0000000000771ef0
[  395.017298] PKRU: 55555554
[  395.017388] Call Trace:
[  395.017471]  <TASK>
[  395.017545]  mbm_L3_assignments_write+0x2d4/0x4e0
[  395.017700]  kernfs_fop_write_iter+0x132/0x1c0
[  395.017851]  vfs_write+0x2bf/0x3c0
[  395.017963]  ksys_write+0x82/0x100
[  395.018074]  do_syscall_64+0xee/0x210
[  395.018198]  ? exc_page_fault+0x81/0xe0
[  395.018321]  entry_SYSCALL_64_after_hwframe+0x77/0x7f
[  395.018482] RIP: 0033:0x7f128177f8b3
[  395.018598] Code: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc

cc cc cc cc cc cc cc 48 8b 05 99 91 07 00 83 38 00 75 10 b8 01 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 4d c3 55 48 89 e5 41 57 41 56 53 50 48 89 d3

Quoted 9 lines
[  395.019167] RSP: 002b:00007ffff66e80f8 EFLAGS: 00000246 ORIG_RAX:
0000000000000001
[  395.019409] RAX: ffffffffffffffda RBX: 0000000000000014 RCX: 00007f128177f8b3
[  395.019636] RDX: 0000000000000014 RSI: 0000000001eedb60 RDI: 0000000000000001
[  395.019861] RBP: 00007ffff66e8120 R08: 0000000000000000 R09: 0000000000000000
[  395.020081] R10: 00007ffff66e81b0 R11: 0000000000000246 R12: 0000000001eedb60
[  395.020303] R13: 0000000000000001 R14: 00007f12817fa650 R15: 0000000000000014
[  395.020532]  </TASK>

Yes. Got it. Missing NULL check. Simplified the function now. Thanks

static unsigned int resctrl_get_assign_type(char *assign) {

        if (!assign || strlen(assign) != 1)
                return ASSIGN_INVALID;

        switch (*assign) {
        case 'e':
                return ASSIGN_EXCLUSIVE;
        case '_':
                return ASSIGN_NONE;
        default:
                return ASSIGN_INVALID;
        }
}
Quoted 39 lines
Quoted 39 lines
fs/resctrl/rdtgroup.c +39 -0
+
+       if (!len || len > 1)
+               return ASSIGN_INVALID;
+
+       switch (*assign) {
+       case 'e':
+               mon_state = ASSIGN_EXCLUSIVE;
+               break;
+       case '_':
+               mon_state = ASSIGN_NONE;
+               break;
+       default:
+               mon_state = ASSIGN_INVALID;
+               break;
+       }
+
+       return mon_state;
+}
+
+static int resctrl_process_assign(struct rdt_resource *r, struct rdtgroup *rdtgrp,
+                                 char *config, char *tok)
+{
+       struct rdt_mon_domain *d;
+       char *dom_str, *id_str;
+       unsigned long dom_id = 0;
+       struct mon_evt *mevt;
+       int assign_type;
+       char domain[10];
+       bool found;
+       int ret;
+
+       mevt = mbm_get_mon_event_by_name(r, config);
+       if (!mevt) {
+               rdt_last_cmd_printf("Invalid assign configuration %s\n", config);
+               return  -ENOENT;
+       }
+
+next:
+       if (!tok || tok[0] == '\0')
Quoted 8 lines

fs/resctrl/rdtgroup.c static int mbm_L3_assignments_show(struct kernfs_open_file *of, struct seq_file · line 2194

Quoted 6 lines
fs/resctrl/rdtgroup.c +6 -0
+               return 0;
+
+       /* Start processing the strings for each domain */
+       dom_str = strim(strsep(&tok, ";"));
+
+       id_str = strsep(&dom_str, "=");

If there's no '=' then dom_str becomes NULL...

Yea. That is correct.

Quoted 33 lines
Quoted 17 lines
fs/resctrl/rdtgroup.c +16 -0
+
+       /* Check for domain id '*' which means all domains */
+       if (id_str && *id_str == '*') {
+               d = NULL;
+               goto check_state;
+       } else if (!id_str || kstrtoul(id_str, 10, &dom_id)) {
+               rdt_last_cmd_puts("Missing domain id\n");
+               return -EINVAL;
+       }
+
+       /* Verify if the dom_id is valid */
+       found = false;
+       list_for_each_entry(d, &r->mon_domains, hdr.list) {
+               if (d->hdr.id == dom_id) {
+                       found = true;
+                       break;

fs/resctrl/rdtgroup.c static int mbm_L3_assignments_show(struct kernfs_open_file *of, struct seq_file · line 2220

Quoted 10 lines
fs/resctrl/rdtgroup.c +10 -0
+               }
+       }
+
+       if (!found) {
+               rdt_last_cmd_printf("Invalid domain id %ld\n", dom_id);
+               return -EINVAL;
+       }
+
+check_state:
+       assign_type = resctrl_get_assing_type(dom_str);

then the resulting type of whatever this is supposed to mean is "panic"

Thanks,
-Peter

Signature · 2 lines
Thanks
Babu Moger
author reviewer
#3 lore raw

Re: [PATCH v13 01/27] x86/cpufeatures: Add support for Assignable Bandwidth Monitoring Counters (ABMC)

In reply to Reinette Chatre

Hi Reinette,

Quoted 33 lines · Reinette Chatre

Hi Babu,

Quoted 30 lines · Babu Moger

Users can create as many monitor groups as RMIDs supported by the hardware. However, bandwidth monitoring feature on AMD system only guarantees that RMIDs currently assigned to a processor will be tracked by hardware. The counters of any other RMIDs which are no longer being tracked will be reset to zero. The MBM event counters return "Unavailable" for the RMIDs that are not tracked by hardware. So, there can be only limited number of groups that can give guaranteed monitoring numbers. With ever changing configurations there is no way to definitely know which of these groups are being tracked for certain point of time. Users do not have the option to monitor a group or set of groups for certain period of time without worrying about RMID being reset in between.

The ABMC feature provides an option to the user to assign a hardware counter to an RMID, event pair and monitor the bandwidth as long as it is assigned. The assigned RMID will be tracked by the hardware until the user unassigns it manually. There is no need to worry about counters being reset during this period. Additionally, the user can specify a bitmask identifying the specific bandwidth types from the given source to track with the counter.

Without ABMC enabled, monitoring will work in current mode without assignment option.

The Linux resctrl subsystem provides an interface that allows monitoring of up to two memory bandwidth events per group, selected from a combination of available total and local events. When ABMC is enabled, two events will be assigned to each group by default, in line with the current interface design. Users will also have the option to configure which types of memory transactions are counted by these events.

Quoted 7 lines
Quoted 4 lines

Due to the limited number of available counters (32), users may quickly exhaust the available counters. If the system runs out of assignable ABMC counters, the kernel will report an error. In such cases, users will nee dto unassign one or more active counters to free up countes for new

"nee dto" -> "need to"
"countes" -> "counters"

Sure.

Quoted 5 lines
Quoted 2 lines

assignments. The interface will provide options to assign or unassign

"The interface will" -> "resctrl will"?

Sure.

Quoted 6 lines
Quoted 3 lines

events through the group-specific interface file.

The feature can be detected via CPUID_Fn80000020_EBX_x00 bit 5.

"The feature can be detected" -> "The feature is detected"

Sure.

Quoted 28 lines
Quoted 11 lines

Bits Description
5 ABMC (Assignable Bandwidth Monitoring Counters)

The feature details are documented in APM listed below [1].
[1] AMD64 Architecture Programmer's Manual Volume 2: System Programming Publication # 24593 Revision 3.41 section 19.3.3.3 Assignable Bandwidth Monitoring (ABMC).

---

...

Quoted 15 lines
 arch/x86/include/asm/cpufeatures.h | 1 +
 arch/x86/kernel/cpu/cpuid-deps.c   | 2 ++
 arch/x86/kernel/cpu/scattered.c    | 1 +
 3 files changed, 4 insertions(+)
arch/x86/include/asm/cpufeatures.h +1 -0
index 6c2c152d8a67..d5c14dc678df 100644
@@ -481,6 +481,7 @@
 #define X86_FEATURE_AMD_HETEROGENEOUS_CORES (21*32 + 6) /* Heterogeneous Core Topology */
 #define X86_FEATURE_AMD_WORKLOAD_CLASS	(21*32 + 7) /* Workload Classification */
 #define X86_FEATURE_PREFER_YMM		(21*32 + 8) /* Avoid ZMM registers due to downclocking */
+#define X86_FEATURE_ABMC		(21*32 + 9) /* Assignable Bandwidth Monitoring Counters */
 
Quoted 16 lines

arch/x86/kernel/cpu/cpuid-deps.c static const struct cpuid_dep cpuid_deps[] = { · line 75

Quoted 12 lines
arch/x86/include/asm/cpufeatures.h +0 -0
 /*
  * BUG word(s)
arch/x86/kernel/cpu/cpuid-deps.c +2 -0
index a2fbea0be535..2f54831e04e5 100644
@@ -71,6 +71,8 @@ static const struct cpuid_dep cpuid_deps[] = {
 	{ X86_FEATURE_CQM_MBM_LOCAL,		X86_FEATURE_CQM_LLC   },
 	{ X86_FEATURE_BMEC,			X86_FEATURE_CQM_MBM_TOTAL   },
 	{ X86_FEATURE_BMEC,			X86_FEATURE_CQM_MBM_LOCAL   },
+	{ X86_FEATURE_ABMC,			X86_FEATURE_CQM_MBM_TOTAL   },
+	{ X86_FEATURE_ABMC,			X86_FEATURE_CQM_MBM_LOCAL   },

Is this dependency still accurate now that the implementation switched to the "extended event ID" variant of ABMC that no longer uses the event IDs associated with X86_FEATURE_CQM_MBM_TOTAL and X86_FEATURE_CQM_MBM_LOCAL?

That's a good question. Unfortunately, we may need to retain this dependency for now, as a significant portion of the code relies on functions like resctrl_is_mbm_event(), resctrl_is_mbm_enabled(), resctrl_arch_is_mbm_total_enabled(), and others.

Quoted 19 lines
Quoted 9 lines
arch/x86/kernel/cpu/cpuid-deps.c +0 -0
 	{ X86_FEATURE_AVX512_BF16,		X86_FEATURE_AVX512VL  },
 	{ X86_FEATURE_AVX512_FP16,		X86_FEATURE_AVX512BW  },
 	{ X86_FEATURE_ENQCMD,			X86_FEATURE_XSAVES    },
arch/x86/kernel/cpu/scattered.c +0 -0
index 16f3ca30626a..3b72b72270f1 100644

arch/x86/kernel/cpu/scattered.c static const struct cpuid_bit cpuid_bits[] = { · line 55

Quoted 7 lines
arch/x86/kernel/cpu/scattered.c +1 -0
@@ -49,6 +49,7 @@ static const struct cpuid_bit cpuid_bits[] = {
 	{ X86_FEATURE_MBA,			CPUID_EBX,  6, 0x80000008, 0 },
 	{ X86_FEATURE_SMBA,			CPUID_EBX,  2, 0x80000020, 0 },
 	{ X86_FEATURE_BMEC,			CPUID_EBX,  3, 0x80000020, 0 },
+	{ X86_FEATURE_ABMC,			CPUID_EBX,  5, 0x80000020, 0 },
 	{ X86_FEATURE_AMD_WORKLOAD_CLASS,	CPUID_EAX, 22, 0x80000021, 0 },
 	{ X86_FEATURE_PERFMON_V2,		CPUID_EAX,  0, 0x80000022, 0 },
 	{ X86_FEATURE_AMD_LBR_V2,		CPUID_EAX,  1, 0x80000022, 0 },

Reinette

Signature · 2 lines
Thanks
Babu Moger
maintainer
#4 lore raw

Re: [PATCH v13 01/27] x86/cpufeatures: Add support for Assignable Bandwidth Monitoring Counters (ABMC)

Hi Babu,

Quoted 18 lines · Babu
Quoted 16 lines · Reinette Chatre

arch/x86/kernel/cpu/cpuid-deps.c static const struct cpuid_dep cpuid_deps[] = { · line 75

Quoted 10 lines · Babu Moger
arch/x86/kernel/cpu/cpuid-deps.c +2 -0
index a2fbea0be535..2f54831e04e5 100644
@@ -71,6 +71,8 @@ static const struct cpuid_dep cpuid_deps[] = {
 	{ X86_FEATURE_CQM_MBM_LOCAL,		X86_FEATURE_CQM_LLC   },
 	{ X86_FEATURE_BMEC,			X86_FEATURE_CQM_MBM_TOTAL   },
 	{ X86_FEATURE_BMEC,			X86_FEATURE_CQM_MBM_LOCAL   },
+	{ X86_FEATURE_ABMC,			X86_FEATURE_CQM_MBM_TOTAL   },
+	{ X86_FEATURE_ABMC,			X86_FEATURE_CQM_MBM_LOCAL   },

Is this dependency still accurate now that the implementation switched to the "extended event ID" variant of ABMC that no longer uses the event IDs associated with X86_FEATURE_CQM_MBM_TOTAL and X86_FEATURE_CQM_MBM_LOCAL?

Quoted 5 lines

That's a good question. Unfortunately, we may need to retain this dependency for now, as a significant portion of the code relies on functions like resctrl_is_mbm_event(), resctrl_is_mbm_enabled(), resctrl_arch_is_mbm_total_enabled(), and others.

Avoiding needing to change code is not a valid reason.

I think that without this dependency the code will still rely on "functions like resctrl_is_mbm_event(), resctrl_is_mbm_enabled(), resctrl_arch_is_mbm_total_enabled(), and others." though.

The core shift is to stop thinking about QOS_L3_MBM_TOTAL_EVENT_ID to mean the same as X86_FEATURE_CQM_MBM_TOTAL, similarly to stop thinking about QOS_L3_MBM_LOCAL_EVENT_ID to mean the same as X86_FEATURE_CQM_MBM_LOCAL.

I expected that for backwards compatibility ABMC will start by enabling QOS_L3_MBM_TOTAL_EVENT_ID and QOS_L3_MBM_LOCAL_EVENT_ID as part of its initialization, configuring them with the current defaults for which memory transactions are expected to be monitored by each. With these events enabled the existing flows using, for example, resctrl_is_mbm_event(), will continue to work as expected, no?

This would require more familiarity with L3 monitoring enumeration on AMD since it will still be required to determine the number of RMIDs etc. but if ABMC does not actually depend on these CQM features then the current enumeration would need to be re-worked anyway.

Reinette

author reviewer
#5 lore raw

Re: [PATCH v13 01/27] x86/cpufeatures: Add support for Assignable Bandwidth Monitoring Counters (ABMC)

Hi Reinette,

Quoted 34 lines · Reinette Chatre

Hi Babu,

Quoted 18 lines · Babu
Quoted 16 lines · Reinette Chatre

arch/x86/kernel/cpu/cpuid-deps.c static const struct cpuid_dep cpuid_deps[] = { · line 75

Quoted 10 lines · Babu Moger
arch/x86/kernel/cpu/cpuid-deps.c +2 -0
index a2fbea0be535..2f54831e04e5 100644
@@ -71,6 +71,8 @@ static const struct cpuid_dep cpuid_deps[] = {
 	{ X86_FEATURE_CQM_MBM_LOCAL,		X86_FEATURE_CQM_LLC   },
 	{ X86_FEATURE_BMEC,			X86_FEATURE_CQM_MBM_TOTAL   },
 	{ X86_FEATURE_BMEC,			X86_FEATURE_CQM_MBM_LOCAL   },
+	{ X86_FEATURE_ABMC,			X86_FEATURE_CQM_MBM_TOTAL   },
+	{ X86_FEATURE_ABMC,			X86_FEATURE_CQM_MBM_LOCAL   },

Is this dependency still accurate now that the implementation switched to the "extended event ID" variant of ABMC that no longer uses the event IDs associated with X86_FEATURE_CQM_MBM_TOTAL and X86_FEATURE_CQM_MBM_LOCAL?

Quoted 5 lines

That's a good question. Unfortunately, we may need to retain this dependency for now, as a significant portion of the code relies on functions like resctrl_is_mbm_event(), resctrl_is_mbm_enabled(), resctrl_arch_is_mbm_total_enabled(), and others.

Avoiding needing to change code is not a valid reason.

I think that without this dependency the code will still rely on "functions like resctrl_is_mbm_event(), resctrl_is_mbm_enabled(), resctrl_arch_is_mbm_total_enabled(), and others." though.

Quoted 4 lines

The core shift is to stop thinking about QOS_L3_MBM_TOTAL_EVENT_ID to mean the same as X86_FEATURE_CQM_MBM_TOTAL, similarly to stop thinking about QOS_L3_MBM_LOCAL_EVENT_ID to mean the same as X86_FEATURE_CQM_MBM_LOCAL.

oh. ok.

Quoted 7 lines

I expected that for backwards compatibility ABMC will start by enabling QOS_L3_MBM_TOTAL_EVENT_ID and QOS_L3_MBM_LOCAL_EVENT_ID as part of its initialization, configuring them with the current defaults for which memory transactions are expected to be monitored by each. With these events enabled the existing flows using, for example, resctrl_is_mbm_event(), will continue to work as expected, no?

Yes. It will work as it uses event id.

Quoted 5 lines

This would require more familiarity with L3 monitoring enumeration on AMD since it will still be required to determine the number of RMIDs etc. but if ABMC does not actually depend on these CQM features then the current enumeration would need to be re-worked anyway.

Are you suggesting to remove the dependency and rework ABMC enumeration in get_rdt_mon_resources()?

Signature · 2 lines
Thanks
Babu Moger
author reviewer
#11 lore raw

Re: [PATCH v13 03/27] x86/resctrl: Consolidate monitoring related data from rdt_resource

In reply to Reinette Chatre

Hi Reinette,

Quoted 12 lines · Reinette Chatre

Hi Babu,

Quoted 8 lines · Babu Moger
include/linux/resctrl.h +0 -0
index 9ba771f2ddea..2a8fa454d3e6 100644
@@ -255,40 +255,48 @@ enum resctrl_schema_fmt {
 	RESCTRL_SCHEMA_RANGE,
 };
 
Quoted 10 lines

include/linux/resctrl.h enum resctrl_schema_fmt { · line 264

Quoted 7 lines
include/linux/resctrl.h +7 -0
+/**
+ * struct resctrl_mon - Monitoring related data of a resctrl resource
+ * @num_rmid:		Number of RMIDs available
+ * @mbm_cfg_mask:	Bandwidth sources that can be tracked when bandwidth
+ *			monitoring events can be configured.
+ * @evt_list:		List of monitoring events
+ */

Nit: this new comment portion can start with a clean slate by all sentences having good structure by starting with upper case and ending with period.

Sure. Will do.

Signature · 2 lines
Thanks
Babu Moger
author reviewer
#14 lore raw

Re: [PATCH v13 04/27] x86/resctrl: Detect Assignable Bandwidth Monitoring feature details

In reply to Reinette Chatre

Hi Reinette,

Quoted 49 lines · Reinette Chatre

Hi Babu,

Quoted 15 lines · Babu Moger

ABMC feature details are reported via CPUID Fn8000_0020_EBX_x5. Bits Description
15:0 MAX_ABMC Maximum Supported Assignable Bandwidth
Monitoring Counter ID + 1

The feature details are documented in APM listed below [1].
[1] AMD64 Architecture Programmer's Manual Volume 2: System Programming Publication # 24593 Revision 3.41 section 19.3.3.3 Assignable Bandwidth Monitoring (ABMC).

Detect the feature and number of assignable monitoring counters supported.

---

...

Quoted 28 lines
---
 arch/x86/kernel/cpu/resctrl/monitor.c | 9 +++++++--
 include/linux/resctrl.h               | 4 ++++
 2 files changed, 11 insertions(+), 2 deletions(-)
arch/x86/kernel/cpu/resctrl/monitor.c +1 -2
index aeb2a9283069..fd2761d9f3f7 100644
@@ -345,6 +345,7 @@ int __init rdt_get_mon_l3_config(struct rdt_resource *r)
 	unsigned int mbm_offset = boot_cpu_data.x86_cache_mbm_width_offset;
 	struct rdt_hw_resource *hw_res = resctrl_to_arch_res(r);
 	unsigned int threshold;
+	u32 eax, ebx, ecx, edx;
 
 	snc_nodes_per_l3_cache = snc_get_config();
 
@@ -375,13 +376,17 @@ int __init rdt_get_mon_l3_config(struct rdt_resource *r)
 	resctrl_rmid_realloc_threshold = resctrl_arch_round_mon_val(threshold);
 
 	if (rdt_cpu_has(X86_FEATURE_BMEC)) {
-		u32 eax, ebx, ecx, edx;
-
 		/* Detect list of bandwidth sources that can be tracked */
 		cpuid_count(0x80000020, 3, &eax, &ebx, &ecx, &edx);
 		r->mon.mbm_cfg_mask = ecx & MAX_EVT_CONFIG_BITS;
 	}
 
Quoted 16 lines

arch/x86/kernel/cpu/resctrl/monitor.c int __init rdt_get_mon_l3_config(struct rdt_resource *r) · line 389

Quoted 6 lines
arch/x86/kernel/cpu/resctrl/monitor.c +6 -0
+	if (rdt_cpu_has(X86_FEATURE_ABMC)) {
+		r->mon.mbm_cntr_assignable = true;
+		cpuid_count(0x80000020, 5, &eax, &ebx, &ecx, &edx);
+		r->mon.num_mbm_cntrs = (ebx & GENMASK(15, 0)) + 1;
+	}
+

Shouldn't ABMC detection also include enumeration of which configurations are supported? From what I can tell, looking ahead patch #18 hardcodes definitions of all known "bandwidth types" (which term to use TBD) and then patch #20 allows *any* of these types to be configured irrespective of whether system supports it.
AMD spec mentions "The types of L3 transactions that ABMC can track are configurable and identified by CPUID Fn8000_0020_ECX_x3." It thus looks like the enumeration of r->mon.mbm_cfg_mask when BMEC is enabled is required for ABMC also and used by this implementation.

That is true. Will add the following check.

if (rdt_cpu_has(X86_FEATURE_BMEC) || rdt_cpu_has(X86_FEATURE_ABMC)) {
/* Detect list of bandwidth sources that can be tracked */ cpuid_count(0x80000020, 3, &eax, &ebx, &ecx, &edx); r->mon.mbm_cfg_mask = ecx & MAX_EVT_CONFIG_BITS;
}

Quoted 20 lines
Quoted 9 lines
arch/x86/kernel/cpu/resctrl/monitor.c +0 -0
 	r->mon_capable = true;
 
 	return 0;
include/linux/resctrl.h +0 -0
index 2a8fa454d3e6..065fb6e38933 100644

include/linux/resctrl.h enum resctrl_schema_fmt { · line 264

Quoted 5 lines
include/linux/resctrl.h +2 -0
@@ -260,11 +260,15 @@ enum resctrl_schema_fmt {
  * @num_rmid:		Number of RMIDs available
  * @mbm_cfg_mask:	Bandwidth sources that can be tracked when bandwidth
  *			monitoring events can be configured.
+ * @num_mbm_cntrs:	Number of assignable monitoring counters
+ * @mbm_cntr_assignable:Is system capable of supporting monitor assignment?

"monitor assignment" has not been used so far, was this intended to be "counter assignment"?

Reinette

Signature · 2 lines
Thanks
Babu Moger
author reviewer
#15 lore raw

Re: [PATCH v13 04/27] x86/resctrl: Detect Assignable Bandwidth Monitoring feature details

In reply to Reinette Chatre

Hi Reinette,

Quoted 68 lines · Reinette Chatre

Hi Babu,

Quoted 15 lines · Babu Moger

ABMC feature details are reported via CPUID Fn8000_0020_EBX_x5. Bits Description
15:0 MAX_ABMC Maximum Supported Assignable Bandwidth
Monitoring Counter ID + 1

The feature details are documented in APM listed below [1].
[1] AMD64 Architecture Programmer's Manual Volume 2: System Programming Publication # 24593 Revision 3.41 section 19.3.3.3 Assignable Bandwidth Monitoring (ABMC).

Detect the feature and number of assignable monitoring counters supported.

---

...

Quoted 24 lines
---
 arch/x86/kernel/cpu/resctrl/monitor.c | 9 +++++++--
 include/linux/resctrl.h               | 4 ++++
 2 files changed, 11 insertions(+), 2 deletions(-)
arch/x86/kernel/cpu/resctrl/monitor.c +1 -2
index aeb2a9283069..fd2761d9f3f7 100644
@@ -345,6 +345,7 @@ int __init rdt_get_mon_l3_config(struct rdt_resource *r)
 	unsigned int mbm_offset = boot_cpu_data.x86_cache_mbm_width_offset;
 	struct rdt_hw_resource *hw_res = resctrl_to_arch_res(r);
 	unsigned int threshold;
+	u32 eax, ebx, ecx, edx;
 
 	snc_nodes_per_l3_cache = snc_get_config();
 
@@ -375,13 +376,17 @@ int __init rdt_get_mon_l3_config(struct rdt_resource *r)
 	resctrl_rmid_realloc_threshold = resctrl_arch_round_mon_val(threshold);
 
 	if (rdt_cpu_has(X86_FEATURE_BMEC)) {
-		u32 eax, ebx, ecx, edx;
-
 		/* Detect list of bandwidth sources that can be tracked */

arch/x86/kernel/cpu/resctrl/monitor.c int __init rdt_get_mon_l3_config(struct rdt_resource *r) · line 389

Quoted 10 lines
arch/x86/kernel/cpu/resctrl/monitor.c +6 -0
 		cpuid_count(0x80000020, 3, &eax, &ebx, &ecx, &edx);
 		r->mon.mbm_cfg_mask = ecx & MAX_EVT_CONFIG_BITS;
 	}
 
+	if (rdt_cpu_has(X86_FEATURE_ABMC)) {
+		r->mon.mbm_cntr_assignable = true;
+		cpuid_count(0x80000020, 5, &eax, &ebx, &ecx, &edx);
+		r->mon.num_mbm_cntrs = (ebx & GENMASK(15, 0)) + 1;
+	}
+

Shouldn't ABMC detection also include enumeration of which configurations are supported? From what I can tell, looking ahead patch #18 hardcodes definitions of all known "bandwidth types" (which term to use TBD) and then patch #20 allows *any* of these types to be configured irrespective of whether system supports it.
AMD spec mentions "The types of L3 transactions that ABMC can track are configurable and identified by CPUID Fn8000_0020_ECX_x3." It thus looks like the enumeration of r->mon.mbm_cfg_mask when BMEC is enabled is required for ABMC also and used by this implementation.

Quoted 2 lines
arch/x86/kernel/cpu/resctrl/monitor.c +0 -0
 	r->mon_capable = true;
 
Quoted 14 lines

include/linux/resctrl.h enum resctrl_schema_fmt { · line 264

Quoted 11 lines
arch/x86/kernel/cpu/resctrl/monitor.c +0 -0
 	return 0;
include/linux/resctrl.h +2 -0
index 2a8fa454d3e6..065fb6e38933 100644
@@ -260,11 +260,15 @@ enum resctrl_schema_fmt {
  * @num_rmid:		Number of RMIDs available
  * @mbm_cfg_mask:	Bandwidth sources that can be tracked when bandwidth
  *			monitoring events can be configured.
+ * @num_mbm_cntrs:	Number of assignable monitoring counters
+ * @mbm_cntr_assignable:Is system capable of supporting monitor assignment?

"monitor assignment" has not been used so far, was this intended to be "counter assignment"?

Missed to respond to this comment. Yes. Sure. Will correct it.

Signature · 2 lines
Thanks
Babu Moger
author reviewer
#18 lore raw

Re: [PATCH v13 05/27] x86/resctrl: Add support to enable/disable AMD ABMC feature

In reply to Reinette Chatre

Hi Reinette,

Quoted 103 lines · Reinette Chatre

Hi Babu,

Quoted 16 lines · Babu Moger

Add the functionality to enable/disable AMD ABMC feature.

AMD ABMC feature is enabled by setting enabled bit(0) in MSR L3_QOS_EXT_CFG. When the state of ABMC is changed, the MSR needs to be updated on all the logical processors in the QOS Domain.

Hardware counters will reset when ABMC state is changed.

The ABMC feature details are documented in APM listed below [1].
[1] AMD64 Architecture Programmer's Manual Volume 2: System Programming Publication # 24593 Revision 3.41 section 19.3.3.3 Assignable Bandwidth Monitoring (ABMC).

---

...

Quoted 66 lines
---
 arch/x86/include/asm/msr-index.h       |  1 +
 arch/x86/kernel/cpu/resctrl/internal.h |  5 +++
 arch/x86/kernel/cpu/resctrl/monitor.c  | 43 ++++++++++++++++++++++++++
 include/linux/resctrl.h                |  3 ++
 4 files changed, 52 insertions(+)
arch/x86/include/asm/msr-index.h +1 -0
index e6134ef2263d..3970e0b16e47 100644
@@ -1203,6 +1203,7 @@
 /* - AMD: */
 #define MSR_IA32_MBA_BW_BASE		0xc0000200
 #define MSR_IA32_SMBA_BW_BASE		0xc0000280
+#define MSR_IA32_L3_QOS_EXT_CFG		0xc00003ff
 #define MSR_IA32_EVT_CFG_BASE		0xc0000400
 
 /* AMD-V MSRs */
arch/x86/kernel/cpu/resctrl/internal.h +5 -0
index 5e3c41b36437..fcc9d23686a1 100644
@@ -37,6 +37,9 @@ struct arch_mbm_state {
 	u64	prev_msr;
 };
 
+/* Setting bit 0 in L3_QOS_EXT_CFG enables the ABMC feature. */
+#define ABMC_ENABLE_BIT			0
+
 /**
  * struct rdt_hw_ctrl_domain - Arch private attributes of a set of CPUs that share
  *			       a resource for a control function
@@ -102,6 +105,7 @@ struct msr_param {
  * @mon_scale:		cqm counter * mon_scale = occupancy in bytes
  * @mbm_width:		Monitor width, to detect and correct for overflow.
  * @cdp_enabled:	CDP state of this resource
+ * @mbm_cntr_assign_enabled:	ABMC feature is enabled
  *
  * Members of this structure are either private to the architecture
  * e.g. mbm_width, or accessed via helpers that provide abstraction. e.g.
@@ -115,6 +119,7 @@ struct rdt_hw_resource {
 	unsigned int		mon_scale;
 	unsigned int		mbm_width;
 	bool			cdp_enabled;
+	bool			mbm_cntr_assign_enabled;
 };
 
 static inline struct rdt_hw_resource *resctrl_to_arch_res(struct rdt_resource *r)
arch/x86/kernel/cpu/resctrl/monitor.c +9 -0
index fd2761d9f3f7..ff4b2abfa044 100644
@@ -405,3 +405,46 @@ void __init intel_rdt_mbm_apply_quirk(void)
 	mbm_cf_rmidthreshold = mbm_cf_table[cf_index].rmidthreshold;
 	mbm_cf = mbm_cf_table[cf_index].cf;
 }
+
+static void resctrl_abmc_set_one_amd(void *arg)
+{
+	bool *enable = arg;
+
+	if (*enable)
+		msr_set_bit(MSR_IA32_L3_QOS_EXT_CFG, ABMC_ENABLE_BIT);
+	else
+		msr_clear_bit(MSR_IA32_L3_QOS_EXT_CFG, ABMC_ENABLE_BIT);

arch/x86/kernel/cpu/resctrl/monitor.c void __init intel_rdt_mbm_apply_quirk(void) · line 426

Quoted 10 lines
arch/x86/kernel/cpu/resctrl/monitor.c +10 -0
+}
+
+/*
+ * ABMC enable/disable requires update of L3_QOS_EXT_CFG MSR on all the CPUs
+ * associated with all monitor domains.
+ */
+static void _resctrl_abmc_enable(struct rdt_resource *r, bool enable)
+{
+	struct rdt_mon_domain *d;
+

It remains a challenge to consider these building blocks without insight into how/when they will be used. To help out, please add guardrails to help with review. For example, this could benefit from a:

Quoted 1 line
	lockdep_assert_cpus_held();

Yes. Sure.

Quoted 16 lines
Quoted 16 lines
arch/x86/kernel/cpu/resctrl/monitor.c +15 -0
+	list_for_each_entry(d, &r->mon_domains, hdr.list) {
+		on_each_cpu_mask(&d->hdr.cpu_mask,
+				 resctrl_abmc_set_one_amd, &enable, 1);
+		resctrl_arch_reset_rmid_all(r, d);
+	}
+}
+
+int resctrl_arch_mbm_cntr_assign_set(struct rdt_resource *r, bool enable)
+{
+	struct rdt_hw_resource *hw_res = resctrl_to_arch_res(r);
+
+	if (r->mon.mbm_cntr_assignable &&
+	    hw_res->mbm_cntr_assign_enabled != enable) {
+		_resctrl_abmc_enable(r, enable);
+		hw_res->mbm_cntr_assign_enabled = enable;
Quoted 8 lines

arch/x86/kernel/cpu/resctrl/monitor.c void __init intel_rdt_mbm_apply_quirk(void) · line 447

Quoted 6 lines
arch/x86/kernel/cpu/resctrl/monitor.c +6 -0
+	}
+
+	return 0;
+}
+
+inline bool resctrl_arch_mbm_cntr_assign_enabled(struct rdt_resource *r)

This "inline" in the .c file is unexpected. Why is this needed?

Not required. No specific reason. Will remove it.

Quoted 22 lines
Quoted 9 lines
arch/x86/kernel/cpu/resctrl/monitor.c +3 -0
+{
+	return resctrl_to_arch_res(r)->mbm_cntr_assign_enabled;
+}
include/linux/resctrl.h +0 -0
index 065fb6e38933..bdb264875ef6 100644

include/linux/resctrl.h static inline u32 resctrl_get_config_index(u32 closid, · line 436

Quoted 9 lines
include/linux/resctrl.h +3 -0
@@ -428,6 +428,9 @@ static inline u32 resctrl_get_config_index(u32 closid,
 bool resctrl_arch_get_cdp_enabled(enum resctrl_res_level l);
 int resctrl_arch_set_cdp_enabled(enum resctrl_res_level l, bool enable);
 
+bool resctrl_arch_mbm_cntr_assign_enabled(struct rdt_resource *r);
+int resctrl_arch_mbm_cntr_assign_set(struct rdt_resource *r, bool enable);
+
 /*
  * Update the ctrl_val and apply this config right now.
  * Must be called on one of the domain's CPUs.

Reinette

Signature · 2 lines
Thanks
Babu Moger
author reviewer
#21 lore raw

Re: [PATCH v13 06/27] x86/resctrl: Introduce the interface to display monitor mode

In reply to Reinette Chatre

Hi Reinette,

Quoted 6 lines · Reinette Chatre

Hi Babu,

On 5/15/25 3:51 PM, Babu Moger wrote:

No comments on changelog since I expect it to be reworked based on
https://lore.kernel.org/lkml/[email protected]/

Yes. Sure.

Quoted 7 lines
Quoted 7 lines
Documentation/filesystems/resctrl.rst +0 -0
@@ -257,6 +257,33 @@ with the following files:
 	    # cat /sys/fs/resctrl/info/L3_MON/mbm_local_bytes_config
 	    0=0x30;1=0x30;3=0x15;4=0x15
 
Quoted 5 lines

Documentation/filesystems/resctrl.rst with the following files: · line 261

Quoted 2 lines
Documentation/filesystems/resctrl.rst +2 -0
+"mbm_assign_mode":
+	Reports the list of monitoring modes supported. The enclosed brackets

Please try to avoid unnecessary words. For example,
"Reports the list of monitoring modes supported." -> "The supported monitoring modes."

Sure.

Signature · 2 lines
Thanks
Babu Moger
maintainer
#37 lore raw

RE: [PATCH v13 11/27] x86/resctrl: Implement resctrl_arch_config_cntr() to assign a counter with ABMC

In reply to Peter Newman

Review

Quoted 9 lines

Thanks for applying my suggestion[1] about the array entry sizes, but you needed one more dereference:

(quoted diff) +2 -2
-       size_t tsize = sizeof(hw_dom->arch_mbm_states[0]);
+       size_t tsize = sizeof(*hw_dom->arch_mbm_states[0]);
 
-       size_t tsize = sizeof(d->mbm_states[0]);
+       size_t tsize = sizeof(*d->mbm_states[0]);

Indeed yes. Thanks.

-Tony

maintainer
#6 lore raw

Re: [PATCH v13 01/27] x86/cpufeatures: Add support for Assignable Bandwidth Monitoring Counters (ABMC)

In reply to Moger, Babu

Hi Babu,

Quoted 50 lines · Babu
Quoted 32 lines · Reinette Chatre
Quoted 18 lines · Babu
Quoted 16 lines · Reinette Chatre

arch/x86/kernel/cpu/cpuid-deps.c static const struct cpuid_dep cpuid_deps[] = { · line 75

Quoted 10 lines · Babu Moger
arch/x86/kernel/cpu/cpuid-deps.c +2 -0
index a2fbea0be535..2f54831e04e5 100644
@@ -71,6 +71,8 @@ static const struct cpuid_dep cpuid_deps[] = {
 	{ X86_FEATURE_CQM_MBM_LOCAL,		X86_FEATURE_CQM_LLC   },
 	{ X86_FEATURE_BMEC,			X86_FEATURE_CQM_MBM_TOTAL   },
 	{ X86_FEATURE_BMEC,			X86_FEATURE_CQM_MBM_LOCAL   },
+	{ X86_FEATURE_ABMC,			X86_FEATURE_CQM_MBM_TOTAL   },
+	{ X86_FEATURE_ABMC,			X86_FEATURE_CQM_MBM_LOCAL   },

Is this dependency still accurate now that the implementation switched to the "extended event ID" variant of ABMC that no longer uses the event IDs associated with X86_FEATURE_CQM_MBM_TOTAL and X86_FEATURE_CQM_MBM_LOCAL?

Quoted 5 lines

That's a good question. Unfortunately, we may need to retain this dependency for now, as a significant portion of the code relies on functions like resctrl_is_mbm_event(), resctrl_is_mbm_enabled(), resctrl_arch_is_mbm_total_enabled(), and others.

Avoiding needing to change code is not a valid reason.

I think that without this dependency the code will still rely on "functions like resctrl_is_mbm_event(), resctrl_is_mbm_enabled(), resctrl_arch_is_mbm_total_enabled(), and others." though.

Quoted 4 lines

The core shift is to stop thinking about QOS_L3_MBM_TOTAL_EVENT_ID to mean the same as X86_FEATURE_CQM_MBM_TOTAL, similarly to stop thinking about QOS_L3_MBM_LOCAL_EVENT_ID to mean the same as X86_FEATURE_CQM_MBM_LOCAL.

oh. ok.

Quoted 7 lines

I expected that for backwards compatibility ABMC will start by enabling QOS_L3_MBM_TOTAL_EVENT_ID and QOS_L3_MBM_LOCAL_EVENT_ID as part of its initialization, configuring them with the current defaults for which memory transactions are expected to be monitored by each. With these events enabled the existing flows using, for example, resctrl_is_mbm_event(), will continue to work as expected, no?

Yes. It will work as it uses event id.

Quoted 1 line
Quoted 8 lines
Quoted 4 lines

This would require more familiarity with L3 monitoring enumeration on AMD since it will still be required to determine the number of RMIDs etc. but if ABMC does not actually depend on these CQM features then the current enumeration would need to be re-worked anyway.

Are you suggesting to remove the dependency and rework ABMC enumeration in get_rdt_mon_resources()?

If you have an alternative proposal that would accurately reflect the ABMC and existing L3 MON features then we can surely consider it.

Reinette

#7 lore raw

Re: [PATCH v13 01/27] x86/cpufeatures: Add support for Assignable Bandwidth Monitoring Counters (ABMC)

Hi Reinette,

Quoted 58 lines · Reinette Chatre

Hi Babu,

Quoted 55 lines · Babu
Quoted 32 lines · Reinette Chatre
Quoted 18 lines · Babu
Quoted 16 lines · Reinette Chatre

arch/x86/kernel/cpu/cpuid-deps.c static const struct cpuid_dep cpuid_deps[] = { · line 75

Quoted 10 lines · Babu Moger
arch/x86/kernel/cpu/cpuid-deps.c +2 -0
index a2fbea0be535..2f54831e04e5 100644
@@ -71,6 +71,8 @@ static const struct cpuid_dep cpuid_deps[] = {
  	{ X86_FEATURE_CQM_MBM_LOCAL,		X86_FEATURE_CQM_LLC   },
  	{ X86_FEATURE_BMEC,			X86_FEATURE_CQM_MBM_TOTAL   },
  	{ X86_FEATURE_BMEC,			X86_FEATURE_CQM_MBM_LOCAL   },
+	{ X86_FEATURE_ABMC,			X86_FEATURE_CQM_MBM_TOTAL   },
+	{ X86_FEATURE_ABMC,			X86_FEATURE_CQM_MBM_LOCAL   },

Is this dependency still accurate now that the implementation switched to the "extended event ID" variant of ABMC that no longer uses the event IDs associated with X86_FEATURE_CQM_MBM_TOTAL and X86_FEATURE_CQM_MBM_LOCAL?

Quoted 5 lines

That's a good question. Unfortunately, we may need to retain this dependency for now, as a significant portion of the code relies on functions like resctrl_is_mbm_event(), resctrl_is_mbm_enabled(), resctrl_arch_is_mbm_total_enabled(), and others.

Avoiding needing to change code is not a valid reason.

I think that without this dependency the code will still rely on "functions like resctrl_is_mbm_event(), resctrl_is_mbm_enabled(), resctrl_arch_is_mbm_total_enabled(), and others." though.

Quoted 4 lines

The core shift is to stop thinking about QOS_L3_MBM_TOTAL_EVENT_ID to mean the same as X86_FEATURE_CQM_MBM_TOTAL, similarly to stop thinking about QOS_L3_MBM_LOCAL_EVENT_ID to mean the same as X86_FEATURE_CQM_MBM_LOCAL.

oh. ok.

Quoted 7 lines

I expected that for backwards compatibility ABMC will start by enabling QOS_L3_MBM_TOTAL_EVENT_ID and QOS_L3_MBM_LOCAL_EVENT_ID as part of its initialization, configuring them with the current defaults for which memory transactions are expected to be monitored by each. With these events enabled the existing flows using, for example, resctrl_is_mbm_event(), will continue to work as expected, no?

Yes. It will work as it uses event id.

Quoted 5 lines

This would require more familiarity with L3 monitoring enumeration on AMD since it will still be required to determine the number of RMIDs etc. but if ABMC does not actually depend on these CQM features then the current enumeration would need to be re-worked anyway.

Quoted 6 lines
Quoted 3 lines

Are you suggesting to remove the dependency and rework ABMC enumeration in get_rdt_mon_resources()?

If you have an alternative proposal that would accurately reflect the ABMC and existing L3 MON features then we can surely consider it.

I don't see any other option at this point. Will change it next revision.

Thanks
Babu

#25 lore raw

Re: [PATCH v13 08/27] x86/resctrl: Introduce mbm_cntr_cfg to track assignable counters at domain

In reply to Reinette Chatre

Hi Reinette,

Quoted 4 lines · Reinette Chatre

Hi Babu,

shortlog: "at domain" -> "per domain"?

Sure.

Quoted 87 lines · Babu Moger
Quoted 86 lines

In mbm_cntr_assign mode hardware counters are assigned/unassigned to an MBM event of a monitor group. Hardware counters are assigned/unassigned at monitoring domain level.

Manage a monitoring domain's hardware counters using a per monitoring domain array of struct mbm_cntr_cfg that is indexed by the hardware counter ID. A hardware counter's configuration contains the MBM event ID and points to the monitoring group that it is assigned to, with a NULL pointer meaning that the hardware counter is available for assignment.

There is no direct way to determine which hardware counters are assigned to a particular monitoring group. Check every entry of every hardware counter configuration array in every monitoring domain to query which MBM events of a monitoring group is tracked by hardware. Such queries are acceptable because of a very small number of assignable counters (32 to 64).

---
v13: Resolved conflicts caused by the recent FS/ARCH code restructure.
The files monitor.c/rdtgroup.c have been split between FS and ARCH directories.

v12: Fixed the struct mbm_cntr_cfg code documentation.
Removed few strange charactors in changelog.
Added the counter range for better understanding. Moved the struct mbm_cntr_cfg definition to resctrl/internal.h as suggested by James.

v11: Refined the change log based on Reinette's feedback.
Fixed few style issues.

v10: Patch changed completely to handle the counters at domain level.
https://lore.kernel.org/lkml/CALPaoCj+zWq1vkHVbXYP0znJbe6Ke3PXPWjtri5AFgD9cQDCUg@mail.gmail.com/
Removed Reviewed-by tag.
Did not see the need to add cntr_id in mbm_state structure. Not used in the code.

v9: Added Reviewed-by tag. No other changes.

v8: Minor commit message changes.

v7: Added check mbm_cntr_assignable for allocating bitmap mbm_cntr_map

v6: New patch to add domain level assignment.

---
  fs/resctrl/rdtgroup.c   | 11 +++++++++++
  include/linux/resctrl.h | 16 ++++++++++++++++
  2 files changed, 27 insertions(+)
fs/resctrl/rdtgroup.c +11 -0
index 51f8f8d3ccbc..e2005fc9acd9 100644
@@ -4085,6 +4085,7 @@ static void rdtgroup_setup_default(void)
 
  static void domain_destroy_mon_state(struct rdt_mon_domain *d)
  {
+	kfree(d->cntr_cfg);
  	bitmap_free(d->rmid_busy_llc);
  	kfree(d->mbm_total);
  	kfree(d->mbm_local);
@@ -4171,6 +4172,16 @@ static int domain_setup_mon_state(struct rdt_resource *r, struct rdt_mon_domain
  			return -ENOMEM;
  		}
  	}
+	if (resctrl_is_mbm_enabled() && r->mon.mbm_cntr_assignable) {
+		tsize = sizeof(*d->cntr_cfg);
+		d->cntr_cfg = kcalloc(r->mon.num_mbm_cntrs, tsize, GFP_KERNEL);
+		if (!d->cntr_cfg) {
+			bitmap_free(d->rmid_busy_llc);
+			kfree(d->mbm_total);
+			kfree(d->mbm_local);
+			return -ENOMEM;
+		}
+	}
 
  	return 0;
  }
include/linux/resctrl.h +0 -0
index bdb264875ef6..d77981d1fcb9 100644
@@ -156,6 +156,20 @@ struct rdt_ctrl_domain {
  	u32				*mbps_val;
  };
 
Quoted 10 lines

include/linux/resctrl.h struct rdt_ctrl_domain { · line 163

Quoted 5 lines
include/linux/resctrl.h +5 -0
+/**
+ * struct mbm_cntr_cfg - Assignable counter configuration
+ * @evtid:		MBM event to which the counter is assigned. Only valid
+ *			if @rdtgroup is not NULL.
+ * @evt_cfg:		Event configuration value.

@evt_cfg is not introduced in changelog nor defined here. Please add a snippet here on what @evt_cfg's values represent. This is important since this is exposed as resctrl fs API to architectures so all architectures need to use same values when interacting with resctrl.

Sure.

@evt_cfg: A value that represents memory transactions (e.g., reads, writes, etc.).

Quoted 10 lines

include/linux/resctrl.h struct rdt_ctrl_domain { · line 170

Quoted 8 lines
include/linux/resctrl.h +7 -0
+ * @rdtgrp:		resctrl group assigned to the counter. NULL if the
+ *			counter is free.
+ */
+struct mbm_cntr_cfg {
+	enum resctrl_event_id   evtid;
+	u32                     evt_cfg;
+	struct rdtgroup         *rdtgrp;

Please align struct member names using TABs.

Sure.

Quoted 13 lines

include/linux/resctrl.h struct rdt_ctrl_domain { · line 184

Quoted 11 lines
include/linux/resctrl.h +3 -0
+};
+
  /**
   * struct rdt_mon_domain - group of CPUs sharing a resctrl monitor resource
   * @hdr:		common header for different domain types
@@ -167,6 +181,7 @@ struct rdt_ctrl_domain {
   * @cqm_limbo:		worker to periodically read CQM h/w counters
   * @mbm_work_cpu:	worker CPU for MBM h/w counters
   * @cqm_work_cpu:	worker CPU for CQM h/w counters
+ * @cntr_cfg:		assignable counters configuration

"array of assignable counters' configuration (indexed by counter ID)"

Sure.

Quoted 11 lines
Quoted 11 lines
include/linux/resctrl.h +1 -0
   */
  struct rdt_mon_domain {
  	struct rdt_domain_hdr		hdr;
@@ -178,6 +193,7 @@ struct rdt_mon_domain {
  	struct delayed_work		cqm_limbo;
  	int				mbm_work_cpu;
  	int				cqm_work_cpu;
+	struct mbm_cntr_cfg		*cntr_cfg;
  };
 
Quoted 4 lines

include/linux/resctrl.h struct rdt_mon_domain { · line 199

Quoted 1 line
include/linux/resctrl.h +0 -0
  /**

Reinette

thanks
Babu

maintainer
#26 lore raw

Re: [PATCH v13 08/27] x86/resctrl: Introduce mbm_cntr_cfg to track assignable counters at domain

Hi Babu,

Quoted 21 lines · Babu
Quoted 11 lines · Reinette Chatre
Quoted 9 lines · Babu Moger
include/linux/resctrl.h +0 -0
index bdb264875ef6..d77981d1fcb9 100644
@@ -156,6 +156,20 @@ struct rdt_ctrl_domain {
       u32                *mbps_val;
   };

  +/**
+ * struct mbm_cntr_cfg - Assignable counter configuration

Quoted 8 lines
Quoted 3 lines

+ * @evtid:        MBM event to which the counter is assigned. Only valid
+ *            if @rdtgroup is not NULL.
+ * @evt_cfg:        Event configuration value.

@evt_cfg is not introduced in changelog nor defined here. Please add a snippet here on what @evt_cfg's values represent. This is important since this is exposed as resctrl fs API to architectures so all architectures need to use same values when interacting with resctrl.

Quoted 3 lines

Sure.

@evt_cfg: A value that represents memory transactions (e.g., reads, writes, etc.).

This still does not explain how an @evt_cfg value should be interpreted. For example, it could be something like below (please feel free to improve).

@evt_cfg: Event configuration created using the READS_TO_LOCAL_MEM, READS_TO_REMOTE_MEM, etc. bits
that represent the memory transactions being counted.

Reinette

#27 lore raw

Re: [PATCH v13 08/27] x86/resctrl: Introduce mbm_cntr_cfg to track assignable counters at domain

Hi Reinette,

Quoted 26 lines · Reinette Chatre

Hi Babu,

Quoted 23 lines · Babu
Quoted 11 lines · Reinette Chatre
Quoted 9 lines · Babu Moger
include/linux/resctrl.h +1 -0
index bdb264875ef6..d77981d1fcb9 100644
@@ -156,6 +156,20 @@ struct rdt_ctrl_domain {
       u32                *mbps_val;
   };
   +/**
+ * struct mbm_cntr_cfg - Assignable counter configuration
Quoted 8 lines

include/linux/resctrl.h struct rdt_ctrl_domain { · line 163

Quoted 3 lines
include/linux/resctrl.h +3 -0
+ * @evtid:        MBM event to which the counter is assigned. Only valid
+ *            if @rdtgroup is not NULL.
+ * @evt_cfg:        Event configuration value.

@evt_cfg is not introduced in changelog nor defined here. Please add a snippet here on what @evt_cfg's values represent. This is important since this is exposed as resctrl fs API to architectures so all architectures need to use same values when interacting with resctrl.

Sure.

Quoted 8 lines
Quoted 1 line

@evt_cfg: A value that represents memory transactions (e.g., reads, writes, etc.).

This still does not explain how an @evt_cfg value should be interpreted. For example, it could be something like below (please feel free to improve).

@evt_cfg: Event configuration created using the READS_TO_LOCAL_MEM, READS_TO_REMOTE_MEM, etc. bits
that represent the memory transactions being counted.

Looks good.
Thanks
Babu

#31 lore raw

Re: [PATCH v13 10/27] x86/resctrl: Add data structures and definitions for ABMC assignment

In reply to Reinette Chatre

Hi Reinette,

Quoted 11 lines · Reinette Chatre

Hi Babu,

Quoted 5 lines · Babu Moger

The ABMC feature provides an option to the user to assign a hardware counter to an RMID, event pair and monitor the bandwidth as long as the counter is assigned. The bandwidth events will be tracked by the hardware until the user changes the configuration. Each resctrl group can configure maximum two counters, one for total event and one for local event.

(please update, above describes previous design)

Ok. Will drop the last line.

Quoted 9 lines
Quoted 6 lines

The ABMC feature implements an MSR L3_QOS_ABMC_CFG (C000_03FDh). ABMC counter assignment is done by setting the counter id, bandwidth source (RMID) and bandwidth configuration. Users will have the option to change the bandwidth configuration using resctrl interface which will be introduced later in the series.

"will be introduced later in the series" is similar to "in a subsequent patch" and should not be used in a changelog. Just describe what this patch does.

ok. Last line is really not required.

Quoted 100 lines
Quoted 100 lines

Attempts to read or write the MSR when ABMC is not enabled will result in a #GP(0) exception.

Introduce the data structures and definitions for MSR L3_QOS_ABMC_CFG (0xC000_03FDh):
=========================================================================
Bits Mnemonic Description Access Reset
Type Value
=========================================================================
63 CfgEn Configuration Enable R/W 0

62 CtrEn Enable/disable counting R/W 0

61:53 – Reserved MBZ 0

52:48 CtrID Counter Identifier R/W 0

47 IsCOS BwSrc field is a CLOSID R/W 0
(not an RMID)

46:44 – Reserved MBZ 0

43:32 BwSrc Bandwidth Source R/W 0
(RMID or CLOSID)

31:0 BwType Bandwidth configuration R/W 0
to track for this counter
==========================================================================

The feature details are documented in the APM listed below [1].
[1] AMD64 Architecture Programmer's Manual Volume 2: System Programming Publication # 24593 Revision 3.41 section 19.3.3.3 Assignable Bandwidth Monitoring (ABMC).

---
v13: Removed the Reviewed-by tag as there is commit log change to remove
BMEC reference.

v12: No changes.

v11: No changes.

v10: No changes.

v9: Removed the references of L3_QOS_ABMC_DSC.
Text changes about configuration in kernel doc.

v8: Update the configuration notes in kernel_doc.
Few commit message update.

v7: Removed the reference of L3_QOS_ABMC_DSC as it is not used anymore.
Moved the configuration notes to kernel_doc.
Adjusted the tabs for l3_qos_abmc_cfg and checkpatch seems happy.

v6: Removed all the fs related changes.
Added note on CfgEn,CtrEn.
Removed the definitions which are not used.
Removed cntr_id initialization.

v5: Moved assignment flags here (path 10/19 of v4).
Added MON_CNTR_UNSET definition to initialize cntr_id's. More details in commit log.
Renamed few fields in l3_qos_abmc_cfg for readability.

v4: Added more descriptions.
Changed the name abmc_ctr_id to ctr_id.
Added L3_QOS_ABMC_DSC. Used for reading the configuration.

v3: No changes.

v2: No changes.

---
  arch/x86/include/asm/msr-index.h       |  1 +
  arch/x86/kernel/cpu/resctrl/internal.h | 35 ++++++++++++++++++++++++++
  2 files changed, 36 insertions(+)
arch/x86/include/asm/msr-index.h +1 -0
index 3970e0b16e47..b5b5ebead24f 100644
@@ -1203,6 +1203,7 @@
  /* - AMD: */
  #define MSR_IA32_MBA_BW_BASE		0xc0000200
  #define MSR_IA32_SMBA_BW_BASE		0xc0000280
+#define MSR_IA32_L3_QOS_ABMC_CFG	0xc00003fd
  #define MSR_IA32_L3_QOS_EXT_CFG		0xc00003ff
  #define MSR_IA32_EVT_CFG_BASE		0xc0000400
 
arch/x86/kernel/cpu/resctrl/internal.h +0 -0
index fcc9d23686a1..db6b0c28ee6b 100644
@@ -164,6 +164,41 @@ union cpuid_0x10_x_edx {
  	unsigned int full;
  };
 
Quoted 7 lines

arch/x86/kernel/cpu/resctrl/internal.h union cpuid_0x10_x_edx { · line 170

Quoted 4 lines
arch/x86/kernel/cpu/resctrl/internal.h +4 -0
+/*
+ * ABMC counters are configured by writing to L3_QOS_ABMC_CFG.
+ * @bw_type		: Bandwidth configuration (supported by BMEC)
+ *			  tracked by the @cntr_id.

The "supported by BMEC" is unexpected with the new design that separated the two features.

My bad. Will remove it.

Quoted 33 lines
Quoted 33 lines
arch/x86/kernel/cpu/resctrl/internal.h +31 -0
+ * @bw_src		: Bandwidth source (RMID or CLOSID).
+ * @reserved1		: Reserved.
+ * @is_clos		: @bw_src field is a CLOSID (not an RMID).
+ * @cntr_id		: Counter identifier.
+ * @reserved		: Reserved.
+ * @cntr_en		: Counting enable bit.
+ * @cfg_en		: Configuration enable bit.
+ *
+ * Configuration and counting:
+ * Counter can be configured across multiple writes to MSR. Configuration
+ * is applied only when @cfg_en = 1. Counter @cntr_id is reset when the
+ * configuration is applied.
+ * @cfg_en = 1, @cntr_en = 0 : Apply @cntr_id configuration but do not
+ *                             count events.
+ * @cfg_en = 1, @cntr_en = 1 : Apply @cntr_id configuration and start
+ *                             counting events.
+ */
+union l3_qos_abmc_cfg {
+	struct {
+		unsigned long bw_type  :32,
+			      bw_src   :12,
+			      reserved1: 3,
+			      is_clos  : 1,
+			      cntr_id  : 5,
+			      reserved : 9,
+			      cntr_en  : 1,
+			      cfg_en   : 1;
+	} split;
+	unsigned long full;
+};
+
  void rdt_ctrl_update(void *arg);
 
Quoted 4 lines

arch/x86/kernel/cpu/resctrl/internal.h union cpuid_0x10_x_edx { · line 204

Quoted 1 line
arch/x86/kernel/cpu/resctrl/internal.h +0 -0
  int rdt_get_mon_l3_config(struct rdt_resource *r);

Reinette

Thanks
Babu

#50 lore raw

Re: [PATCH v13 11/27] x86/resctrl: Implement resctrl_arch_config_cntr() to assign a counter with ABMC

In reply to Reinette Chatre

Hi Reinette,

Quoted 9 lines · Reinette Chatre

Hi Babu,

Quoted 4 lines · Babu Moger

The ABMC feature provides an option to the user to assign a hardware counter to an RMID, event pair and monitor the bandwidth as long as it is assigned. The assigned RMID will be tracked by the hardware until the user unassigns it manually.

(please review this often repeated snippet to match new design)

Sure.

Quoted 38 lines
Quoted 7 lines

Implement an architecture-specific handler to assign and unassign the counter. Configure counters by writing to the L3_QOS_ABMC_CFG MSR, specifying the counter ID, bandwidth source (RMID), and event configuration.

...

Quoted 28 lines
arch/x86/kernel/cpu/resctrl/monitor.c +20 -0
index ff4b2abfa044..e31084f7babd 100644
@@ -448,3 +448,40 @@ inline bool resctrl_arch_mbm_cntr_assign_enabled(struct rdt_resource *r)
  {
  	return resctrl_to_arch_res(r)->mbm_cntr_assign_enabled;
  }
+
+static void resctrl_abmc_config_one_amd(void *info)
+{
+	union l3_qos_abmc_cfg *abmc_cfg = info;
+
+	wrmsrl(MSR_IA32_L3_QOS_ABMC_CFG, abmc_cfg->full);
+}
+
+/*
+ * Send an IPI to the domain to assign the counter to RMID, event pair.
+ */
+void resctrl_arch_config_cntr(struct rdt_resource *r, struct rdt_mon_domain *d,
+			      enum resctrl_event_id evtid, u32 rmid, u32 closid,
+			      u32 cntr_id, u32 evt_cfg, bool assign)
+{
+	struct rdt_hw_mon_domain *hw_dom = resctrl_to_arch_mon_dom(d);
+	union l3_qos_abmc_cfg abmc_cfg = { 0 };
+	struct arch_mbm_state *am;
+
+	abmc_cfg.split.cfg_en = 1;
Quoted 8 lines

arch/x86/kernel/cpu/resctrl/monitor.c inline bool resctrl_arch_mbm_cntr_assign_enabled(struct rdt_resource *r) · line 474

Quoted 4 lines
arch/x86/kernel/cpu/resctrl/monitor.c +4 -0
+	abmc_cfg.split.cntr_en = assign ? 1 : 0;
+	abmc_cfg.split.cntr_id = cntr_id;
+	abmc_cfg.split.bw_src = rmid;
+	abmc_cfg.split.bw_type = evt_cfg;

Is evt_cfg really needed to be programmed when unassigning a counter? Looking ahead at patch #14 resctrl_free_config_cntr() needs to go through extra list walk to get this data but why would hardware need an accurate event configuration to *unassign* a counter?

evt_cfg is not required during unassign. I can remove it.

Quoted 7 lines

It seems unnecessary to provide both the event ID *and* the configuration. resctrl_arch_config_cntr() could drop the "evt_cfg" parameter and instead there can be a new resctrl utility that architecture can use to query the event's configuration. Similar to resctrl_is_mon_event_enabled() introduced in
https://lore.kernel.org/lkml/[email protected]/ that exposes an event property.

Sounds good.
I can add a new function resctrl_get_mon_event_config(evtid) and call it only during the "assign". It will be called inside resctrl_arch_config_cntr().

Quoted 6 lines

It looks to me as though there are a couple of changes in the telemetry work
that would benefit this work. https://lore.kernel.org/lkml/[email protected]/
switches the monitor events to be maintained in an array indexed by event ID, eliminating the need for searching the evt_list that this work does in a couple of places. Also note the handy new for_each_mbm_event() helper (https://lore.kernel.org/lkml/[email protected]/).

Sure. Looking at it now.

Quoted 23 lines
Quoted 23 lines
arch/x86/kernel/cpu/resctrl/monitor.c +13 -0
+
+	smp_call_function_any(&d->hdr.cpu_mask, resctrl_abmc_config_one_amd, &abmc_cfg, 1);
+
+	/*
+	 * The hardware counter is reset (because cfg_en == 1) so there is no
+	 * need to record initial non-zero counts.
+	 */
+	if (assign) {
+		am = get_arch_mbm_state(hw_dom, rmid, evtid);
+		if (am)
+			memset(am, 0, sizeof(*am));
+	}
+}
include/linux/resctrl.h +0 -0
index d77981d1fcb9..59a4fe60ab46 100644
@@ -559,6 +559,23 @@ void resctrl_arch_reset_rmid_all(struct rdt_resource *r, struct rdt_mon_domain *
   */
  void resctrl_arch_reset_all_ctrls(struct rdt_resource *r);
 
Quoted 8 lines

include/linux/resctrl.h void resctrl_arch_reset_rmid_all(struct rdt_resource *r, struct rdt_mon_domain * · line 564

Quoted 3 lines
include/linux/resctrl.h +3 -0
+/**
+ * resctrl_arch_config_cntr() - Configure the counter id to RMID, event
+ *				pair on the domain.

The sentence seem strange, should "Configure the counter" perhaps be "Assign the counter"? Or if the naming requires "configure" ... "Configure the counter with its new RMID and event details."? Please feel free to improve.

Last one looks good.

Quoted 5 lines

include/linux/resctrl.h void resctrl_arch_reset_rmid_all(struct rdt_resource *r, struct rdt_mon_domain * · line 566

Quoted 3 lines
include/linux/resctrl.h +2 -0
+ * @r:			Resource structure.
+ * @d:			Domain that the counter id to be configured.

I am unable to parse description of @d.

The domain in which the counter ID is to be configured.

Quoted 8 lines

include/linux/resctrl.h void resctrl_arch_reset_rmid_all(struct rdt_resource *r, struct rdt_mon_domain * · line 570

Quoted 5 lines
include/linux/resctrl.h +4 -0
+ * @evtid:		Event type to configure.
+ * @rmid:		RMID to configure.
+ * @closid:		CLOSID to configure.
+ * @cntr_id:		Counter ID to configure.

All four parameters descriptions end with "to configure" ... but it is actually only the counter that is configured while the rest is the data that the counter is configured with, no?

Will remove "to configure" from all the other fields except the cntr_id.

Quoted 7 lines

include/linux/resctrl.h void resctrl_arch_reset_rmid_all(struct rdt_resource *r, struct rdt_mon_domain * · line 572

Quoted 3 lines
include/linux/resctrl.h +2 -0
+ * @evt_cfg:		MBM event configuration value representing reads,
+ *			writes etc.

Needs definition about what the contents of @evt_cfg means. This is the API ...it cannot be vague like "reads, write, etc." but should be specific about which bit means what.

Copying your comment on other patch
https://lore.kernel.org/lkml/[email protected]/

@evt_cfg: Event configuration created using the READS_TO_LOCAL_MEM, READS_TO_REMOTE_MEM, etc. bits that represent the memory transactions being counted.

Quoted 5 lines

include/linux/resctrl.h void resctrl_arch_reset_rmid_all(struct rdt_resource *r, struct rdt_mon_domain * · line 573

Quoted 2 lines
include/linux/resctrl.h +1 -0
+ * @assign:		Assign or unassign.

"True to assign the counter, false to unassign the counter."

Sure.

Quoted 4 lines

Needs some context here about what architecture can expect on how this function will be called. For example, "Can be called from any CPU."

Sure.

Quoted 11 lines

include/linux/resctrl.h void resctrl_arch_reset_rmid_all(struct rdt_resource *r, struct rdt_mon_domain * · line 581

Quoted 8 lines
include/linux/resctrl.h +5 -0
+ */
+void resctrl_arch_config_cntr(struct rdt_resource *r, struct rdt_mon_domain *d,
+			      enum resctrl_event_id evtid, u32 rmid, u32 closid,
+			      u32 cntr_id, u32 evt_cfg, bool assign);
+
  extern unsigned int resctrl_rmid_realloc_threshold;
  extern unsigned int resctrl_rmid_realloc_limit;
 

Reinette

Thanks
Babu

#38 lore raw

Re: RE: [PATCH v13 11/27] x86/resctrl: Implement resctrl_arch_config_cntr() to assign a counter with ABMC

In reply to Luck, Tony

Hi Tony, Peter,

Quoted 4 lines · Tony
Quoted 3 lines

Thanks for applying my suggestion[1] about the array entry sizes, but you needed one more dereference:

Quoted 8 lines

Review

Quoted 5 lines
(quoted diff) +2 -2
-       size_t tsize = sizeof(hw_dom->arch_mbm_states[0]);
+       size_t tsize = sizeof(*hw_dom->arch_mbm_states[0]);
 
-       size_t tsize = sizeof(d->mbm_states[0]);
+       size_t tsize = sizeof(*d->mbm_states[0]);

Indeed yes. Thanks.

Tony, Thanks for porting patches.

I can actually pick your branch [1] and apply review comments on top for v14 series. Hope that is fine with everyone.
[1] https://git.kernel.org/pub/scm/linux/kernel/git/aegl/linux.git/log/?h=my_mbm_plus_babu_abmc

One question though: Where will the Peter's fix [2] go?
[2] https://lore.kernel.org/lkml/CALPaoCj7FBv_vfDp+4tgqo4p8T7Eov_Ys+CQRoAX6u43a4OTDQ@mail.gmail.com/

thanks
Babu

maintainer
#40 lore raw

RE: RE: [PATCH v13 11/27] x86/resctrl: Implement resctrl_arch_config_cntr() to assign a counter with ABMC

Quoted 1 line

I'll backport into my patches and then rebase the whole branch. Hopefully should not take long.

Done and force pushed to same branch at kernel .org.

Both Peter's changes went into my 4th patch, and your patches didn't touch those functions so git did all the real work.

-Tony

#41 lore raw

Re: RE: RE: [PATCH v13 11/27] x86/resctrl: Implement resctrl_arch_config_cntr() to assign a counter with ABMC

Hi Tony,

Quoted 7 lines · Tony
Quoted 1 line

I'll backport into my patches and then rebase the whole branch. Hopefully should not take long.

Done and force pushed to same branch at kernel .org.

Both Peter's changes went into my 4th patch, and your patches didn't touch those functions so git did all the real work.

I will base my v14 on top of this. Thanks
Babu

author reviewer
#53 lore raw

Re: [PATCH v13 12/27] x86/resctrl: Introduce event configuration modes

In reply to Reinette Chatre

Hi Reinette,

Quoted 9 lines · Reinette Chatre

Hi Babu,

Quoted 6 lines · Babu Moger

MBM events can be configured using either BMEC (Bandwidth Monitoring Event Configuration) or the mbm_cntr_assign mode.

Introduce a data structure to represent the various event configuration modes and their corresponding values.

Quoted 7 lines
Quoted 1 line

I cannot recall suggesting this.

(/me digs)

Are you perhaps referring to https://lore.kernel.org/lkml/[email protected]/

Yes.

Quoted 10 lines

This is not referring to new modes but the existing mbm_cntr_assign modes. resctrl knows which "mbm_cntr_assign" mode is active and it can use that to determine whether BMEC can be exposed to user space or not. There is already enough information in resctrl to know whether BMEC files should be exposed or not.

I think this work self makes clear that these modes are useless since patch #25 that determines whether to hide BMEC files doesn't even use it.

Sure. I will remove change related to mbm_mode.

Quoted 27 lines
Quoted 27 lines

---
v13: New patch to handle different event configuration types with
mbm_cntr_assign mode.

---
 fs/resctrl/internal.h         |  6 ++++--
 fs/resctrl/monitor.c          |  4 ++--
 fs/resctrl/rdtgroup.c         |  2 +-
 include/linux/resctrl_types.h | 11 +++++++++++
 4 files changed, 18 insertions(+), 5 deletions(-)
fs/resctrl/internal.h +2 -1
index 9a8cf6f11151..0fae374559ba 100644
@@ -55,13 +55,15 @@ static inline struct rdt_fs_context *rdt_fc2context(struct fs_context *fc)
  * struct mon_evt - Entry in the event list of a resource
  * @evtid:		event id
  * @name:		name of the event
- * @configurable:	true if the event is configurable
+ * @mbm_mode:		monitoring mode (BMEC or mbm_cntr_assign)
+ * @evt_cfg:		event configuration value decoding reads, writes.
  * @list:		entry in &rdt_resource->evt_list
  */
 struct mon_evt {
Quoted 8 lines

fs/resctrl/internal.h static inline struct rdt_fs_context *rdt_fc2context(struct fs_context *fc) · line 66

Quoted 5 lines
fs/resctrl/internal.h +2 -1
 	enum resctrl_event_id	evtid;
 	char			*name;
-	bool			configurable;
+	enum resctrl_mbm_mode	mbm_mode;
+	u32			evt_cfg;

This very important yet totally unrelated member sneaked in without any mention.

Yes. We only need evt_cfg. I will just add it as separate patch.

Quoted 29 lines
Quoted 29 lines
fs/resctrl/internal.h +0 -0
 	struct list_head	list;
 };
 
fs/resctrl/monitor.c +2 -2
index 2548aee0151c..8e403587a02f 100644
@@ -903,12 +903,12 @@ int resctrl_mon_resource_init(void)
 	l3_mon_evt_init(r);
 
 	if (resctrl_arch_is_evt_configurable(QOS_L3_MBM_TOTAL_EVENT_ID)) {
-		mbm_total_event.configurable = true;
+		mbm_total_event.mbm_mode = MBM_MODE_BMEC;
 		resctrl_file_fflags_init("mbm_total_bytes_config",
 					 RFTYPE_MON_INFO | RFTYPE_RES_CACHE);
 	}
 	if (resctrl_arch_is_evt_configurable(QOS_L3_MBM_LOCAL_EVENT_ID)) {
-		mbm_local_event.configurable = true;
+		mbm_local_event.mbm_mode = MBM_MODE_BMEC;
 		resctrl_file_fflags_init("mbm_local_bytes_config",
 					 RFTYPE_MON_INFO | RFTYPE_RES_CACHE);
 	}
fs/resctrl/rdtgroup.c +0 -0
index 752750e3e443..f192b2736a77 100644
@@ -1152,7 +1152,7 @@ static int rdt_mon_features_show(struct kernfs_open_file *of,
 
Quoted 8 lines

fs/resctrl/rdtgroup.c static int rdt_mon_features_show(struct kernfs_open_file *of, · line 1155

Quoted 4 lines
fs/resctrl/rdtgroup.c +1 -1
 	list_for_each_entry(mevt, &r->mon.evt_list, list) {
 		seq_printf(seq, "%s\n", mevt->name);
-		if (mevt->configurable)
+		if (mevt->mbm_mode == MBM_MODE_BMEC)

This can instead be a call to a utility that returns whether BMEC should be visible based on resctrl_mon::mbm_cntr_assignable and rdt_hw_resource::mbm_cntr_assign_enabled (via resctrl_arch_mbm_cntr_assign_enabled() of course).

Sure. Will do.

Quoted 27 lines
Quoted 14 lines
fs/resctrl/rdtgroup.c +0 -0
 			seq_printf(seq, "%s_config\n", mevt->name);
 	}
 
include/linux/resctrl_types.h +2 -0
index a25fb9c4070d..26cd1fec72db 100644
@@ -47,4 +47,15 @@ enum resctrl_event_id {
 	QOS_NUM_EVENTS,
 };
 
+/*
+ * Event configuration mode.

include/linux/resctrl_types.h enum resctrl_event_id { · line 61

Quoted 10 lines
include/linux/resctrl_types.h +9 -0
+ * Events can be configured either in BMEC (Bandwidth Monitoring Event
+ * Configuration) mode or mbm_cntr_assign mode.
+ */
+enum resctrl_mbm_mode {
+	MBM_MODE_NONE,
+	MBM_MODE_BMEC,
+	MBM_MODE_ASSIGN,
+};
+
 #endif /* __LINUX_RESCTRL_TYPES_H */

Reinette

Signature · 2 lines
Thanks
Babu Moger
author reviewer
#56 lore raw

Re: [PATCH v13 13/27] x86/resctrl: Add the functionality to assign MBM events

In reply to Reinette Chatre

Hi Reinette,

Quoted 8 lines · Reinette Chatre

Hi Babu,

Quoted 2 lines · Babu Moger

The mbm_cntr_assign mode offers "num_mbm_cntrs" number of counters that can be assigned to RMID, event pair and monitor the bandwidth as long

"RMID, event pairs"? (assuming at this point in new version it will be obvious what is meant by "event").

Sure.

Quoted 7 lines
Quoted 4 lines

as it is assigned.

Add the functionality to allocate and assign a counter to am RMID, event

"am" -> "an"

sure.

Quoted 46 lines
Quoted 9 lines

pair in the domain.

If all the counters are in use, kernel will log the error message "Unable to allocate counter in domain" in /sys/fs/resctrl/info/last_cmd_status when a new assignment is requested. Exit on the first failure when assigning counters across all the domains.

---

...

Quoted 24 lines
---
 fs/resctrl/internal.h |   3 +
 fs/resctrl/monitor.c  | 134 ++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 137 insertions(+)
fs/resctrl/internal.h +3 -0
index 0fae374559ba..ce4fcac91937 100644
@@ -377,6 +377,9 @@ bool closid_allocated(unsigned int closid);
 
 int resctrl_find_cleanest_closid(void);
 
+int resctrl_assign_cntr_event(struct rdt_resource *r, struct rdt_mon_domain *d,
+			      struct rdtgroup *rdtgrp, enum resctrl_event_id evtid);
+
 #ifdef CONFIG_RESCTRL_FS_PSEUDO_LOCK
 int rdtgroup_locksetup_enter(struct rdtgroup *rdtgrp);
 
fs/resctrl/monitor.c +0 -0
index 8e403587a02f..d76fd0840946 100644

fs/resctrl/monitor.c void resctrl_mon_resource_exit(void) · line 940

Quoted 7 lines
fs/resctrl/monitor.c +4 -0
@@ -934,3 +934,137 @@ void resctrl_mon_resource_exit(void)
 
 	dom_data_exit(r);
 }
+
+/*
+ * Configure the counter for the event, RMID pair for the domain. Reset the
+ * non-architectural state to clear all the event counters.

clear *all* the event counters?

Quoted 2 lines

"Reset the non-architectural state to clear all the event counters." -> "Reset the associated non-architectural state."?

ok.

Quoted 2 lines

Also, please see https://lore.kernel.org/lkml/[email protected]/

Yes. Sure.

Quoted 15 lines
Quoted 15 lines
fs/resctrl/monitor.c +14 -0
+ */
+static void resctrl_config_cntr(struct rdt_resource *r, struct rdt_mon_domain *d,
+				enum resctrl_event_id evtid, u32 rmid, u32 closid,
+				u32 cntr_id, u32 evt_cfg, bool assign)
+{
+	struct mbm_state *m;
+
+	resctrl_arch_config_cntr(r, d, evtid, rmid, closid, cntr_id, evt_cfg, assign);
+
+	m = get_mbm_state(d, closid, rmid, evtid);
+	if (m)
+		memset(m, 0, sizeof(struct mbm_state));
+}
+
Quoted 8 lines

fs/resctrl/monitor.c void resctrl_mon_resource_exit(void) · line 957

Quoted 3 lines
fs/resctrl/monitor.c +3 -0
+/*
+ * mbm_cntr_get() - Return the cntr_id for the matching evtid and rdtgrp in
+ *		    cntr_cfg array.

Please prefix parameter names with @ in description to make obvious what is refered to. Although "cntr_id" is a local variable so may be easier to parse if cntr_id is replaced with actual "counter ID" term while keeping rest as actual parameters. That makes cntr_cfg unneeded.

Sure.

Quoted 3 lines

If intending to explain function context then failure return should also be documented. Even better would be to follow typical style of kernel-doc (even if not using /** start) and not mix and match so randomly.

Sure.

Quoted 11 lines

fs/resctrl/monitor.c void resctrl_mon_resource_exit(void) · line 961

Quoted 5 lines
fs/resctrl/monitor.c +4 -0
+ */
+static int mbm_cntr_get(struct rdt_resource *r, struct rdt_mon_domain *d,
+			struct rdtgroup *rdtgrp, enum resctrl_event_id evtid)
+{

A subtle issue here is only evident from later patches, for example patch #17, that calls mbm_cntr_get() with a non MBM event ID from __mon_event_count().

If this usage is expected then these utilities needs extra checks to ensure they are only called with valid MBM event IDs.

Sure. Will add the check resctrl_is_mbm_event().

Quoted 17 lines
Quoted 4 lines
fs/resctrl/monitor.c +3 -0
+	int cntr_id;
+
+	for (cntr_id = 0; cntr_id < r->mon.num_mbm_cntrs; cntr_id++) {

fs/resctrl/monitor.c void resctrl_mon_resource_exit(void) · line 974

Quoted 10 lines
fs/resctrl/monitor.c +10 -0
+		if (d->cntr_cfg[cntr_id].rdtgrp == rdtgrp &&
+		    d->cntr_cfg[cntr_id].evtid == evtid)
+			return cntr_id;
+	}
+
+	return -ENOENT;
+}
+
+/*
+ * mbm_cntr_alloc() - Return the first free entry in cntr_cfg array.

"Return the first ...array." -> "Initilialize and return ID of a new counter, return -ENOSPC on failure." ? This is still an awkward use of kernel-doc ... better to be properly formatted.

Sure.

Quoted 24 lines
Quoted 24 lines
fs/resctrl/monitor.c +23 -0
+ */
+static int mbm_cntr_alloc(struct rdt_resource *r, struct rdt_mon_domain *d,
+			  struct rdtgroup *rdtgrp, enum resctrl_event_id evtid)
+{
+	int cntr_id;
+
+	for (cntr_id = 0; cntr_id < r->mon.num_mbm_cntrs; cntr_id++) {
+		if (!d->cntr_cfg[cntr_id].rdtgrp) {
+			d->cntr_cfg[cntr_id].rdtgrp = rdtgrp;
+			d->cntr_cfg[cntr_id].evtid = evtid;
+			return cntr_id;
+		}
+	}
+
+	return -ENOSPC;
+}
+
+/*
+ * mbm_get_mon_event() - Return the mon_evt entry for the matching evtid.
+ */
+static struct mon_evt *mbm_get_mon_event(struct rdt_resource *r,
+					 enum resctrl_event_id evtid)
+{
Quoted 8 lines

fs/resctrl/monitor.c void resctrl_mon_resource_exit(void) · line 1003

Quoted 6 lines
fs/resctrl/monitor.c +6 -0
+	struct mon_evt *mevt;
+
+	list_for_each_entry(mevt, &r->mon.evt_list, list) {
+		if (mevt->evtid == evtid)
+			return mevt;
+	}

With changes from telemetry series this becomes an array lookup.

Sure. Will look into this.

Quoted 27 lines
Quoted 27 lines
fs/resctrl/monitor.c +26 -0
+
+	return NULL;
+}
+
+/*
+ * Allocate a fresh counter and configure the event if not assigned already.
+ */
+static int resctrl_alloc_config_cntr(struct rdt_resource *r, struct rdt_mon_domain *d,
+				     struct rdtgroup *rdtgrp, enum resctrl_event_id evtid)
+{
+	struct mon_evt *mevt;
+	int cntr_id;
+
+	/* No need to allocate a new counter if it is already assigned */
+	cntr_id = mbm_cntr_get(r, d, rdtgrp, evtid);
+	if (cntr_id >= 0)
+		goto cntr_configure;
+
+	cntr_id = mbm_cntr_alloc(r, d, rdtgrp, evtid);
+	if (cntr_id <  0) {
+		rdt_last_cmd_printf("Unable to allocate counter in domain %d\n",
+				    d->hdr.id);
+		return cntr_id;
+	}
+
+cntr_configure:
Quoted 8 lines

fs/resctrl/monitor.c void resctrl_mon_resource_exit(void) · line 1032

Quoted 3 lines
fs/resctrl/monitor.c +3 -0
+	mevt = mbm_get_mon_event(r, evtid);
+	if (!mevt) {
+		rdt_last_cmd_printf("Invalid event id %d\n", evtid);

Difficult to see at this point but it seems that this is in kernel bug territory since user space provided text that is translated to event ID and here translated back to monitor event. This must succeed. Could this be simplified and back-and-forth avoided by passing the mon_evt instead of event ID?

We can do that.

Quoted 6 lines
Quoted 3 lines
fs/resctrl/monitor.c +2 -0
+		return -EINVAL;
+	}
Quoted 10 lines

fs/resctrl/monitor.c void resctrl_mon_resource_exit(void) · line 1040

Quoted 6 lines
fs/resctrl/monitor.c +6 -0
+
+	/*
+	 * Skip reconfiguration if the event setup is current; otherwise,
+	 * update and apply the new configuration to the domain.
+	 */
+	if (mevt->evt_cfg != d->cntr_cfg[cntr_id].evt_cfg) {

Lost me. Previous patch silently created mon_event::evt_cfg without initializing it. Here it is compared and treated as the "source of truth" ... where does its value come from?

Yes. That is correct. Will have to initialize evt_cfg when it is first introduced. Will do.

Quoted 16 lines

fs/resctrl/monitor.c void resctrl_mon_resource_exit(void) · line 1052

Quoted 13 lines
fs/resctrl/monitor.c +12 -0
+		d->cntr_cfg[cntr_id].evt_cfg = mevt->evt_cfg;
+		resctrl_config_cntr(r, d, evtid, rdtgrp->mon.rmid, rdtgrp->closid,
+				    cntr_id, mevt->evt_cfg, true);
+	}
+
+	return 0;
+}
+
+/*
+ * Assign a hardware counter to event @evtid of group @rdtgrp.
+ * Assign counters to all domains if @d is NULL; otherwise, assign the
+ * counter to the specified domain @d.

Can add here what is mentioned in changelog that this exits on first failure and so highlight that this can have partial assignment when exit on such failure.

Sure.

Quoted 22 lines
Quoted 9 lines
fs/resctrl/monitor.c +8 -0
+ */
+int resctrl_assign_cntr_event(struct rdt_resource *r, struct rdt_mon_domain *d,
+			      struct rdtgroup *rdtgrp, enum resctrl_event_id evtid)
+{
+	int ret = 0;
+
+	if (!d) {
+		list_for_each_entry(d, &r->mon_domains, hdr.list) {

fs/resctrl/monitor.c void resctrl_mon_resource_exit(void) · line 1070

Quoted 10 lines
fs/resctrl/monitor.c +10 -0
+			ret = resctrl_alloc_config_cntr(r, d, rdtgrp, evtid);
+			if (ret)
+				return ret;
+		}
+	} else {
+		ret = resctrl_alloc_config_cntr(r, d, rdtgrp, evtid);
+	}
+
+	return ret;
+}

Reinette

Signature · 2 lines
Thanks
Babu Moger
author reviewer
#59 lore raw

Re: [PATCH v13 14/27] x86/resctrl: Add the functionality to unassign MBM events

In reply to Reinette Chatre

Hi Reinette,

Quoted 44 lines · Reinette Chatre

Hi Babu,

Quoted 13 lines · Babu Moger

The mbm_cntr_assign mode offers "num_mbm_cntrs" number of counters that can be assigned to an RMID, event pair and monitor the bandwidth as long as it is assigned. If all the counters are in use, the kernel will log the error message "Unable to allocate counter in domain" in /sys/fs/resctrl/info/last_cmd_status when a new assignment is requested.

To make space for a new assignment, users must unassign an already assigned counter and retry the assignment again.

Add the functionality to unassign and free the counters in the domain.

---

...

Quoted 26 lines
---
 fs/resctrl/internal.h |  2 ++
 fs/resctrl/monitor.c  | 60 +++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 62 insertions(+)
fs/resctrl/internal.h +2 -0
index ce4fcac91937..64ddc107fcab 100644
@@ -379,6 +379,8 @@ int resctrl_find_cleanest_closid(void);
 
 int resctrl_assign_cntr_event(struct rdt_resource *r, struct rdt_mon_domain *d,
 			      struct rdtgroup *rdtgrp, enum resctrl_event_id evtid);
+int resctrl_unassign_cntr_event(struct rdt_resource *r, struct rdt_mon_domain *d,
+				struct rdtgroup *rdtgrp, enum resctrl_event_id evtid);
 
 #ifdef CONFIG_RESCTRL_FS_PSEUDO_LOCK
 int rdtgroup_locksetup_enter(struct rdtgroup *rdtgrp);
fs/resctrl/monitor.c +0 -0
index d76fd0840946..fbc938bd3b23 100644
@@ -989,6 +989,14 @@ static int mbm_cntr_alloc(struct rdt_resource *r, struct rdt_mon_domain *d,
 	return -ENOSPC;
 }
 
Quoted 5 lines

fs/resctrl/monitor.c static int mbm_cntr_alloc(struct rdt_resource *r, struct rdt_mon_domain *d, · line 993

Quoted 2 lines
fs/resctrl/monitor.c +2 -0
+/*
+ * mbm_cntr_free() -  Reset cntr_id to zero.

"Reset cntr_id to zero"? cntr_id is an index to an array. Please provide accurate and useful descriptions.

Yes. My bad. Will correct it.

Quoted 29 lines
Quoted 29 lines
fs/resctrl/monitor.c +21 -0
+ */
+static void mbm_cntr_free(struct rdt_mon_domain *d, int cntr_id)
+{
+	memset(&d->cntr_cfg[cntr_id], 0, sizeof(struct mbm_cntr_cfg));
+}
+
 /*
  * mbm_get_mon_event() - Return the mon_evt entry for the matching evtid.
  */
@@ -1068,3 +1076,55 @@ int resctrl_assign_cntr_event(struct rdt_resource *r, struct rdt_mon_domain *d,
 
 	return ret;
 }
+
+/*
+ * Unassign and free the counter if assigned.
+ */
+static int resctrl_free_config_cntr(struct rdt_resource *r, struct rdt_mon_domain *d,
+				    struct rdtgroup *rdtgrp, enum resctrl_event_id evtid)
+{
+	struct mon_evt *mevt;
+	int cntr_id;
+
+	cntr_id = mbm_cntr_get(r, d, rdtgrp, evtid);
+
+	/* If there is no cntr_id assigned, nothing to do */
+	if (cntr_id < 0)
+		return 0;
Quoted 8 lines

fs/resctrl/monitor.c int resctrl_assign_cntr_event(struct rdt_resource *r, struct rdt_mon_domain *d, · line 1097

Quoted 4 lines
fs/resctrl/monitor.c +4 -0
+
+	mevt = mbm_get_mon_event(r, evtid);
+	if (!mevt) {
+		rdt_last_cmd_printf("Invalid event id %d\n", evtid);

Similar to previous comment this is in kernel bug territory and could be simplified by passing mon_evt instead. Although this is the unassign portion where evt_cfg seems unnecessary.

Sure. The call mbm_get_mon_event() is not required anymore in this path.

Quoted 37 lines
Quoted 24 lines
fs/resctrl/monitor.c +23 -0
+		return -EINVAL;
+	}
+
+	resctrl_config_cntr(r, d, evtid, rdtgrp->mon.rmid, rdtgrp->closid,
+			    cntr_id, mevt->evt_cfg, false);
+
+	mbm_cntr_free(d, cntr_id);
+
+	return 0;
+}
+
+/*
+ * Unassign a hardware counter associated with @evtid from the domain and
+ * the group. Unassign the counters from all the domains if @d is NULL else
+ * unassign from @d.
+ */
+int  resctrl_unassign_cntr_event(struct rdt_resource *r, struct rdt_mon_domain *d,
+				 struct rdtgroup *rdtgrp, enum resctrl_event_id evtid)
+{
+	int ret;
+
+	if (!d) {
+		list_for_each_entry(d, &r->mon_domains, hdr.list) {

fs/resctrl/monitor.c int resctrl_assign_cntr_event(struct rdt_resource *r, struct rdt_mon_domain *d, · line 1130

Quoted 10 lines
fs/resctrl/monitor.c +10 -0
+			ret = resctrl_free_config_cntr(r, d, rdtgrp, evtid);
+			if (ret)
+				return ret;
+		}
+	} else {
+		ret = resctrl_free_config_cntr(r, d, rdtgrp, evtid);
+	}
+
+	return ret;
+}

Reinette

Signature · 2 lines
Thanks
Babu Moger
author reviewer
#62 lore raw

Re: [PATCH v13 15/27] x86/resctrl: Report 'Unassigned' for MBM events in mbm_cntr_assign mode

In reply to Reinette Chatre

Hi Reinette,

Quoted 12 lines · Reinette Chatre

Hi Babu,

Quoted 8 lines · Babu Moger

In mbm_cntr_assign mode, the hardware counter should be assigned to read the MBM events.

Report 'Unassigned' in case the user attempts to read the event without assigning a hardware counter.

Export resctrl_is_mbm_event() and mbm_cntr_get() to allow usage from other functions within fs/resctrl.

Quoted 7 lines

Please clarify that these two functions are exposed differently, resctrl_is_mbm_event() is added to include/linux/resctrl.h (also note similar change in
https://lore.kernel.org/lkml/[email protected]/)
so not just exposed to fs/resctrl but instead to resctrl fs as well as arch code
while mbm_cntr_get() remains internal to resctrl fs by being added to fs/resctrl/internal.h.

Sure. Will update the comment.
With Tony's changes(
https://lore.kernel.org/lkml/[email protected]/),
the resctrl_is_mbm_event() is not required in here. It is already there.

I will have to update the comment only on mbm_cntr_get(). Will do.

Quoted 22 lines
Quoted 22 lines

---

---
 Documentation/filesystems/resctrl.rst |  8 ++++++++
 fs/resctrl/ctrlmondata.c              | 14 ++++++++++++++
 fs/resctrl/internal.h                 |  2 ++
 fs/resctrl/monitor.c                  |  4 ++--
 fs/resctrl/rdtgroup.c                 |  2 +-
 include/linux/resctrl.h               |  1 +
 6 files changed, 28 insertions(+), 3 deletions(-)
Documentation/filesystems/resctrl.rst +0 -0
index 2bfad43aac9c..5cf2d742f04c 100644
@@ -430,6 +430,14 @@ When monitoring is enabled all MON groups will also contain:
 	for the L3 cache they occupy). These are named "mon_sub_L3_YY"
 	where "YY" is the node number.
 
Quoted 11 lines

Documentation/filesystems/resctrl.rst When monitoring is enabled all MON groups will also contain: · line 439

Quoted 7 lines
Documentation/filesystems/resctrl.rst +7 -0
+	The mbm_cntr_assign mode offers "num_mbm_cntrs" number of counters
+	and allows users to assign a counter to mon_hw_id, event pair enabling
+	bandwidth monitoring for as long as the counter remains assigned.
+	The hardware will continue tracking the assigned mon_hw_id until
+	the user manually unassigns it, ensuring that counters are not reset
+	during this period. An MBM event returns 'Unassigned' when the event
+	does not have a hardware counter assigned.

(please rework based on "event" vs "group" assignment ... not intending that "group" assignment be documented but the "event" assignment needs to be accurate for "group" assignment to be a simple extension)

Sure.

Quoted 28 lines
Quoted 28 lines
Documentation/filesystems/resctrl.rst +1 -0
+
 "mon_hw_id":
 	Available only with debug option. The identifier used by hardware
 	for the monitor group. On x86 this is the RMID.
fs/resctrl/ctrlmondata.c +12 -0
index 6ed2dfd4dbbd..f6b8ad24b0b5 100644
@@ -643,6 +643,18 @@ int rdtgroup_mondata_show(struct seq_file *m, void *arg)
 			goto out;
 		}
 		d = container_of(hdr, struct rdt_mon_domain, hdr);
+
+		/*
+		 * Report 'Unassigned' if mbm_cntr_assign mode is enabled and
+		 * counter is unassigned.
+		 */
+		if (resctrl_arch_mbm_cntr_assign_enabled(r) &&
+		    resctrl_is_mbm_event(evtid) &&
+		    (mbm_cntr_get(r, d, rdtgrp, evtid) < 0)) {
+			rr.err = -ENOENT;
+			goto checkresult;
+		}
+
 		mon_event_read(&rr, r, d, rdtgrp, &d->hdr.cpu_mask, evtid, false);
 	}
 
Quoted 13 lines

fs/resctrl/ctrlmondata.c int rdtgroup_mondata_show(struct seq_file *m, void *arg) · line 671

Quoted 9 lines
fs/resctrl/ctrlmondata.c +2 -0
@@ -652,6 +664,8 @@ int rdtgroup_mondata_show(struct seq_file *m, void *arg)
 		seq_puts(m, "Error\n");
 	else if (rr.err == -EINVAL)
 		seq_puts(m, "Unavailable\n");
+	else if (rr.err == -ENOENT)
+		seq_puts(m, "Unassigned\n");
 	else
 		seq_printf(m, "%llu\n", rr.val);
 

It may be unexpected that this is treated as "-ENOENT" but the function returns success. This can be addressed with a comment when comparing the return codes to other hardware return codes.

Will add the comment.

Quoted 58 lines
Quoted 47 lines
fs/resctrl/internal.h +2 -0
index 64ddc107fcab..0dfd2efe68fc 100644
@@ -381,6 +381,8 @@ int resctrl_assign_cntr_event(struct rdt_resource *r, struct rdt_mon_domain *d,
 			      struct rdtgroup *rdtgrp, enum resctrl_event_id evtid);
 int resctrl_unassign_cntr_event(struct rdt_resource *r, struct rdt_mon_domain *d,
 				struct rdtgroup *rdtgrp, enum resctrl_event_id evtid);
+int mbm_cntr_get(struct rdt_resource *r, struct rdt_mon_domain *d,
+		 struct rdtgroup *rdtgrp, enum resctrl_event_id evtid);
 
 #ifdef CONFIG_RESCTRL_FS_PSEUDO_LOCK
 int rdtgroup_locksetup_enter(struct rdtgroup *rdtgrp);
fs/resctrl/monitor.c +2 -2
index fbc938bd3b23..c98a61bde179 100644
@@ -956,8 +956,8 @@ static void resctrl_config_cntr(struct rdt_resource *r, struct rdt_mon_domain *d
  * mbm_cntr_get() - Return the cntr_id for the matching evtid and rdtgrp in
  *		    cntr_cfg array.
  */
-static int mbm_cntr_get(struct rdt_resource *r, struct rdt_mon_domain *d,
-			struct rdtgroup *rdtgrp, enum resctrl_event_id evtid)
+int mbm_cntr_get(struct rdt_resource *r, struct rdt_mon_domain *d,
+		 struct rdtgroup *rdtgrp, enum resctrl_event_id evtid)
 {
 	int cntr_id;
 
fs/resctrl/rdtgroup.c +1 -1
index f192b2736a77..72317a5adee2 100644
@@ -127,7 +127,7 @@ static bool resctrl_is_mbm_enabled(void)
 		resctrl_arch_is_mbm_local_enabled());
 }
 
-static bool resctrl_is_mbm_event(int e)
+bool resctrl_is_mbm_event(int e)
 {
 	return (e >= QOS_L3_MBM_TOTAL_EVENT_ID &&
 		e <= QOS_L3_MBM_LOCAL_EVENT_ID);
include/linux/resctrl.h +0 -0
index 59a4fe60ab46..f78b6064230c 100644

include/linux/resctrl.h static inline u32 resctrl_get_config_index(u32 closid, · line 447

Quoted 7 lines
include/linux/resctrl.h +1 -0
@@ -441,6 +441,7 @@ static inline u32 resctrl_get_config_index(u32 closid,
 	}
 }
 
+bool resctrl_is_mbm_event(int e);
 bool resctrl_arch_get_cdp_enabled(enum resctrl_res_level l);
 int resctrl_arch_set_cdp_enabled(enum resctrl_res_level l, bool enable);
 

Reinette

Signature · 2 lines
Thanks
Babu Moger
author reviewer
#65 lore raw

Re: [PATCH v13 16/27] x86/resctrl: Pass entire struct rdtgroup rather than passing individual members

In reply to Reinette Chatre

Hi Reinette,

Quoted 7 lines · Reinette Chatre

Hi Babu,

Quoted 1 line · Babu Moger

The mbm_cntr_assign mode requires a cntr_id to read event data. The

cntr_id -> "counter ID"

Sure.

Quoted 4 lines
Quoted 1 line

cntr_id is retrieved via mbm_cntr_get, which takes a struct rdtgroup as

cntr_id -> "counter ID"

Sure.

Quoted 2 lines

mbm_cntr_get -> mbm_cntr_get()

Sure.

Quoted 7 lines
Quoted 3 lines

a parameter.

Passing the full rdtgroup also provides access to closid and rmid, both of

closid -> CLOSID
rmid -> RMID

Sure.

Quoted 6 lines
Quoted 3 lines

which are necessary to read monitoring events.

Refactor the code to pass the entire struct rdtgroup instead of individual

"the entire" -> "a pointer to"

Sure.

Quoted 6 lines
Quoted 5 lines

members in preparation for this requirement.

---

Patch looks good.--

Thanks
Babu Moger

author reviewer
#68 lore raw

Re: [PATCH v13 17/27] x86/resctrl: Add the support for reading ABMC counters

In reply to Reinette Chatre

Hi Reinette,

Quoted 8 lines · Reinette Chatre

Hi Babu,

Quoted 2 lines · Babu Moger

Software can read the assignable counters using the QM_EVTSEL and QM_CTR register pair.

Please append with more context on how register pair is used to support the changes in this patch.

Sure.

Quoted 13 lines
Quoted 13 lines

QM_EVTSEL Register definition:
=======================================================
Bits Mnemonic Description
=======================================================
63:44 -- Reserved
43:32 RMID Resource Monitoring Identifier
31 ExtEvtID Extended Event Identifier
30:8 -- Reserved
7:0 EvtID Event Identifier
=======================================================

Quoted 8 lines
Quoted 6 lines

The contents of a specific counter can be read by setting the following fields in QM_EVTSEL.ExtendedEvtID = 1, QM_EVTSEL.EvtID = L3CacheABMC (=1) and setting [RMID] to the desired counter ID. Reading QM_CTR will then return the contents of the specified counter. The E bit will be set if the counter configuration was invalid, or if an invalid counter ID was set in the QM_EVTSEL[RMID] field.

Please rewrite above in imperative tone.

Sure.

Quoted 30 lines
Quoted 30 lines

Introduce __cntr_id_read_phys() to read the counter ID event data.

---
v13: Split the patch into 2. First one to handle the passing of rdtgroup structure to few
functions( __mon_event_count and mbm_update(). Second one to handle ABMC counter reading. Added new function __cntr_id_read_phys() to handle ABMC event reading. Updated kernel doc for resctrl_arch_reset_rmid() and resctrl_arch_rmid_read(). Resolved conflicts caused by the recent FS/ARCH code restructure.
The monitor.c file has now been split between the FS and ARCH directories.

v12: New patch to support extended event mode when ABMC is enabled.

---
 arch/x86/kernel/cpu/resctrl/internal.h |  6 +++
 arch/x86/kernel/cpu/resctrl/monitor.c  | 66 ++++++++++++++++++++++----
 fs/resctrl/monitor.c                   | 14 ++++--
 include/linux/resctrl.h                |  9 ++--
 4 files changed, 80 insertions(+), 15 deletions(-)
arch/x86/kernel/cpu/resctrl/internal.h +0 -0
index db6b0c28ee6b..3b0cdb5520c7 100644
@@ -40,6 +40,12 @@ struct arch_mbm_state {
 /* Setting bit 0 in L3_QOS_EXT_CFG enables the ABMC feature. */
 #define ABMC_ENABLE_BIT			0
 
Quoted 4 lines

arch/x86/kernel/cpu/resctrl/internal.h struct arch_mbm_state { · line 44

Quoted 2 lines
arch/x86/kernel/cpu/resctrl/internal.h +2 -0
+/*
+ * ABMC Qos Event Identifiers.

QoS?

Sure.

Quoted 7 lines

arch/x86/kernel/cpu/resctrl/internal.h struct arch_mbm_state { · line 47

Quoted 4 lines
arch/x86/kernel/cpu/resctrl/internal.h +3 -0
+ */
+#define ABMC_EXTENDED_EVT_ID		BIT(31)
+#define ABMC_EVT_ID			1

Please use BIT(0) to be consistent.

Sure.

Quoted 63 lines
Quoted 63 lines
arch/x86/kernel/cpu/resctrl/internal.h +1 -0
+
 /**
  * struct rdt_hw_ctrl_domain - Arch private attributes of a set of CPUs that share
  *			       a resource for a control function
arch/x86/kernel/cpu/resctrl/monitor.c +36 -1
index e31084f7babd..36a03dae6d8e 100644
@@ -161,6 +161,41 @@ static int __rmid_read_phys(u32 prmid, enum resctrl_event_id eventid, u64 *val)
 	return 0;
 }
 
+static int __cntr_id_read_phys(u32 cntr_id, u64 *val)
+{
+	u64 msr_val;
+
+	/*
+	 * QM_EVTSEL Register definition:
+	 * =======================================================
+	 * Bits    Mnemonic        Description
+	 * =======================================================
+	 * 63:44   --              Reserved
+	 * 43:32   RMID            Resource Monitoring Identifier
+	 * 31      ExtEvtID        Extended Event Identifier
+	 * 30:8    --              Reserved
+	 * 7:0     EvtID           Event Identifier
+	 * =======================================================
+	 * The contents of a specific counter can be read by setting the
+	 * following fields in QM_EVTSEL.ExtendedEvtID(=1) and
+	 * QM_EVTSEL.EvtID = L3CacheABMC (=1) and setting [RMID] to the
+	 * desired counter ID. Reading QM_CTR will then return the
+	 * contents of the specified counter. The E bit will be set if the
+	 * counter configuration was invalid, or if an invalid counter ID
+	 * was set in the QM_EVTSEL[RMID] field.
+	 */
+	wrmsr(MSR_IA32_QM_EVTSEL, ABMC_EXTENDED_EVT_ID | ABMC_EVT_ID, cntr_id);
+	rdmsrl(MSR_IA32_QM_CTR, msr_val);
+
+	if (msr_val & RMID_VAL_ERROR)
+		return -EIO;
+	if (msr_val & RMID_VAL_UNAVAIL)
+		return -EINVAL;
+
+	*val = msr_val;
+	return 0;
+}
+
 static struct arch_mbm_state *get_arch_mbm_state(struct rdt_hw_mon_domain *hw_dom,
 						 u32 rmid,
 						 enum resctrl_event_id eventid)
@@ -180,7 +215,7 @@ static struct arch_mbm_state *get_arch_mbm_state(struct rdt_hw_mon_domain *hw_do
 }
 
 void resctrl_arch_reset_rmid(struct rdt_resource *r, struct rdt_mon_domain *d,
-			     u32 unused, u32 rmid,
+			     u32 unused, u32 rmid, int cntr_id,
 			     enum resctrl_event_id eventid)
 {
 	struct rdt_hw_mon_domain *hw_dom = resctrl_to_arch_mon_dom(d);
@@ -192,9 +227,16 @@ void resctrl_arch_reset_rmid(struct rdt_resource *r, struct rdt_mon_domain *d,
 	if (am) {
 		memset(am, 0, sizeof(*am));
 
Quoted 8 lines

arch/x86/kernel/cpu/resctrl/monitor.c void resctrl_arch_reset_rmid(struct rdt_resource *r, struct rdt_mon_domain *d, · line 232

Quoted 6 lines
arch/x86/kernel/cpu/resctrl/monitor.c +3 -3
-		prmid = logical_rmid_to_physical_rmid(cpu, rmid);
-		/* Record any initial, non-zero count value. */
-		__rmid_read_phys(prmid, eventid, &am->prev_msr);
+		if (resctrl_arch_mbm_cntr_assign_enabled(r) &&
+		    resctrl_is_mbm_event(eventid)) {
+			if (cntr_id < 0)

This would be a bug, no? how about WARN_ON_ONCE()?

Yes. Will do that.

Quoted 26 lines
Quoted 26 lines
arch/x86/kernel/cpu/resctrl/monitor.c +9 -2
+				return;
+			__cntr_id_read_phys(cntr_id, &am->prev_msr);
+		} else {
+			prmid = logical_rmid_to_physical_rmid(cpu, rmid);
+			/* Record any initial, non-zero count value. */
+			__rmid_read_phys(prmid, eventid, &am->prev_msr);
+		}
 	}
 }
 
@@ -224,8 +266,8 @@ static u64 mbm_overflow_count(u64 prev_msr, u64 cur_msr, unsigned int width)
 }
 
 int resctrl_arch_rmid_read(struct rdt_resource *r, struct rdt_mon_domain *d,
-			   u32 unused, u32 rmid, enum resctrl_event_id eventid,
-			   u64 *val, void *ignored)
+			   u32 unused, u32 rmid, int cntr_id,
+			   enum resctrl_event_id eventid, u64 *val, void *ignored)
 {
 	struct rdt_hw_mon_domain *hw_dom = resctrl_to_arch_mon_dom(d);
 	struct rdt_hw_resource *hw_res = resctrl_to_arch_res(r);
@@ -237,8 +279,16 @@ int resctrl_arch_rmid_read(struct rdt_resource *r, struct rdt_mon_domain *d,
 
 	resctrl_arch_rmid_read_context_check();
 
Quoted 8 lines

arch/x86/kernel/cpu/resctrl/monitor.c int resctrl_arch_rmid_read(struct rdt_resource *r, struct rdt_mon_domain *d, · line 284

Quoted 5 lines
arch/x86/kernel/cpu/resctrl/monitor.c +3 -2
-	prmid = logical_rmid_to_physical_rmid(cpu, rmid);
-	ret = __rmid_read_phys(prmid, eventid, &msr_val);
+	if (resctrl_arch_mbm_cntr_assign_enabled(r) &&
+	    resctrl_is_mbm_event(eventid)) {
+		if (cntr_id < 0)

WARN_ON_ONCE()?

Yes.

Quoted 29 lines
Quoted 29 lines
arch/x86/kernel/cpu/resctrl/monitor.c +7 -0
+			return cntr_id;
+		ret = __cntr_id_read_phys(cntr_id, &msr_val);
+	} else {
+		prmid = logical_rmid_to_physical_rmid(cpu, rmid);
+		ret = __rmid_read_phys(prmid, eventid, &msr_val);
+	}
+
 	if (ret)
 		return ret;
 
fs/resctrl/monitor.c +5 -1
index a477be9cdb66..72f3dfb5b903 100644
@@ -159,7 +159,11 @@ void __check_limbo(struct rdt_mon_domain *d, bool force_free)
 			break;
 
 		entry = __rmid_entry(idx);
-		if (resctrl_arch_rmid_read(r, d, entry->closid, entry->rmid,
+		/*
+		 * cntr_id is not relevant for QOS_L3_OCCUP_EVENT_ID.
+		 * Pass dummy value -1.
+		 */
+		if (resctrl_arch_rmid_read(r, d, entry->closid, entry->rmid, -1,
 					   QOS_L3_OCCUP_EVENT_ID, &val,
 					   arch_mon_ctx)) {
 			rmid_dirty = true;
@@ -359,6 +363,7 @@ static struct mbm_state *get_mbm_state(struct rdt_mon_domain *d, u32 closid,
 
Quoted 9 lines

fs/resctrl/monitor.c static struct mbm_state *get_mbm_state(struct rdt_mon_domain *d, u32 closid, · line 366

Quoted 3 lines
fs/resctrl/monitor.c +1 -0
 static int __mon_event_count(struct rdtgroup *rdtgrp, struct rmid_read *rr)
 {
+	int cntr_id = mbm_cntr_get(rr->r, rr->d, rdtgrp, rr->evtid);

So mbm_cntr_get() is called on *all* events (even non MBM) whether assignable counters are supported or not. I assume it relies on num_mbm_cntrs to be zero on non-ABMC systems but I think this needs to be explicit that mbm_cntr_get() returns -ENOENT in these cases. Any developer attempting to modify mbm_cntr_get() needs to be aware of this usage.

Yes. Good point.

Quoted 3 lines

This is quite subtle that resctrl_arch_reset_rmid() and resctrl_arch_rmid_read() can be called with a negative counter ID. To help with code health this needs to be highlighted (more later).

Sure.

Quoted 37 lines
Quoted 37 lines
fs/resctrl/monitor.c +4 -3
 	int cpu = smp_processor_id();
 	u32 closid = rdtgrp->closid;
 	u32 rmid = rdtgrp->mon.rmid;
@@ -368,7 +373,7 @@ static int __mon_event_count(struct rdtgroup *rdtgrp, struct rmid_read *rr)
 	u64 tval = 0;
 
 	if (rr->first) {
-		resctrl_arch_reset_rmid(rr->r, rr->d, closid, rmid, rr->evtid);
+		resctrl_arch_reset_rmid(rr->r, rr->d, closid, rmid, cntr_id, rr->evtid);
 		m = get_mbm_state(rr->d, closid, rmid, rr->evtid);
 		if (m)
 			memset(m, 0, sizeof(struct mbm_state));
@@ -379,7 +384,7 @@ static int __mon_event_count(struct rdtgroup *rdtgrp, struct rmid_read *rr)
 		/* Reading a single domain, must be on a CPU in that domain. */
 		if (!cpumask_test_cpu(cpu, &rr->d->hdr.cpu_mask))
 			return -EINVAL;
-		rr->err = resctrl_arch_rmid_read(rr->r, rr->d, closid, rmid,
+		rr->err = resctrl_arch_rmid_read(rr->r, rr->d, closid, rmid, cntr_id,
 						 rr->evtid, &tval, rr->arch_mon_ctx);
 		if (rr->err)
 			return rr->err;
@@ -404,7 +409,8 @@ static int __mon_event_count(struct rdtgroup *rdtgrp, struct rmid_read *rr)
 	list_for_each_entry(d, &rr->r->mon_domains, hdr.list) {
 		if (d->ci->id != rr->ci->id)
 			continue;
-		err = resctrl_arch_rmid_read(rr->r, d, closid, rmid,
+		cntr_id = mbm_cntr_get(rr->r, d, rdtgrp, rr->evtid);
+		err = resctrl_arch_rmid_read(rr->r, d, closid, rmid, cntr_id,
 					     rr->evtid, &tval, rr->arch_mon_ctx);
 		if (!err) {
 			rr->val += tval;
include/linux/resctrl.h +0 -0
index f78b6064230c..cd24d1577e0a 100644
@@ -473,6 +473,7 @@ void resctrl_offline_cpu(unsigned int cpu);
Quoted 8 lines

include/linux/resctrl.h void resctrl_offline_cpu(unsigned int cpu); · line 476

Quoted 4 lines
include/linux/resctrl.h +1 -0
  *			counter may match traffic of both @closid and @rmid, or @rmid
  *			only.
  * @rmid:		rmid of the counter to read.
+ * @cntr_id:		cntr_id to read MBM events with mbm_cntr_assign mode.

"Counter ID used to read MBM events in mbm_cntr_evt_assign mode. Only valid when
mbm_cntr_evt_assign mode is enabled and @eventid is an MBM event. Can be negative when invalid." (Please feel free to improve)

Looks good.

Quoted 14 lines
Quoted 14 lines
include/linux/resctrl.h +3 -2
  * @eventid:		eventid to read, e.g. L3 occupancy.
  * @val:		result of the counter read in bytes.
  * @arch_mon_ctx:	An architecture specific value from
@@ -490,8 +491,9 @@ void resctrl_offline_cpu(unsigned int cpu);
  * 0 on success, or -EIO, -EINVAL etc on error.
  */
 int resctrl_arch_rmid_read(struct rdt_resource *r, struct rdt_mon_domain *d,
-			   u32 closid, u32 rmid, enum resctrl_event_id eventid,
-			   u64 *val, void *arch_mon_ctx);
+			   u32 closid, u32 rmid, int cntr_id,
+			   enum resctrl_event_id eventid, u64 *val,
+			   void *arch_mon_ctx);
 
Quoted 10 lines

include/linux/resctrl.h struct rdt_domain_hdr *resctrl_find_domain(struct list_head *h, int id, · line 537

Quoted 7 lines
include/linux/resctrl.h +1 -0
 /**
  * resctrl_arch_rmid_read_context_check()  - warn about invalid contexts
@@ -532,12 +534,13 @@ struct rdt_domain_hdr *resctrl_find_domain(struct list_head *h, int id,
  * @closid:	closid that matches the rmid. Depending on the architecture, the
  *		counter may match traffic of both @closid and @rmid, or @rmid only.
  * @rmid:	The rmid whose counter values should be reset.
+ * @cntr_id:	The cntr_id to read MBM events with mbm_cntr_assign mode.

Same as above.

Sure.

Quoted 13 lines

include/linux/resctrl.h struct rdt_domain_hdr *resctrl_find_domain(struct list_head *h, int id, · line 546

Quoted 10 lines
include/linux/resctrl.h +1 -1
  * @eventid:	The eventid whose counter values should be reset.
  *
  * This can be called from any CPU.
  */
 void resctrl_arch_reset_rmid(struct rdt_resource *r, struct rdt_mon_domain *d,
-			     u32 closid, u32 rmid,
+			     u32 closid, u32 rmid, int cntr_id,
 			     enum resctrl_event_id eventid);
 
 /**

Reinette

Signature · 2 lines
Thanks
Babu Moger
author reviewer
#71 lore raw

Re: [PATCH v13 18/27] x86/resctrl: Add definitions for MBM event configuration

In reply to Reinette Chatre

Hi Reinette,

Quoted 7 lines · Reinette Chatre

Hi Babu,

Quoted 4 lines · Babu Moger

The "mbm_cntr_assign" mode allows users to manually assign a hardware counter to a specific RMID and event pair. The events available for assignment are configurable.

Quoted 6 lines
Quoted 3 lines

By default, each resctrl group supports two MBM events: mbm_total_bytes and mbm_local_bytes. Each event corresponds to an MBM configuration that specifies the bandwidth sources tracked by the event.

hmmm ... earlier I thought "bandwidth source" means RMID but here it seems to mean the memory transactions? The various terms are confusing.

My bad. Yes. "bandwidth source" means RMID.

I should say "memory transactions"

Quoted 6 lines
Quoted 3 lines

Add definitions of supported bandwidth sources.

changelog uses "bandwidth sources" while the comments of patch uses "memory transactions" ... please be consistent with terms.

Sure.

Quoted 26 lines
Quoted 26 lines

---
v13: Updated the changelog.
Removed the definitions from resctrl_types.h and moved to internal.h. Removed mbm_assign_config definition. Configurations will be part of mon_evt list.
Resolved conflicts caused by the recent FS/ARCH code restructure. The rdtgroup.c file has now been split between the FS and ARCH directories.

v12: New patch to support event configurations via new counter_configs
method.

---
 fs/resctrl/internal.h | 10 ++++++++++
 fs/resctrl/rdtgroup.c | 14 ++++++++++++++
 2 files changed, 24 insertions(+)
fs/resctrl/internal.h +0 -0
index 0dfd2efe68fc..019d00bf5adf 100644
@@ -203,6 +203,16 @@ struct rdtgroup {
 	struct pseudo_lock_region	*plr;
 };
 
Quoted 8 lines

fs/resctrl/internal.h struct rdtgroup { · line 207

Quoted 2 lines
fs/resctrl/internal.h +2 -0
+/**
+ * struct mbm_evt_value - Specific type of memory events.

I am trying to decipher the terminology. If these are events, then it becomes confusing since it becomes "these events are used to configure events". You mention "memory transaction" below, this sounds more accurate to me. Above could thus be:

struct mbm_evt_value - Memory transaction an MBM event can be configured with.

Sure.

Quoted 4 lines

The name of the struct could also do with a rename to avoid the "event" term that conflicts with the actual MBM events. Maybe "mbm_cfg_value" ... I do not think this is a good name so please consider what would work better.

I can change it to "mbm_config_value".

Quoted 4 lines

fs/resctrl/internal.h struct rdtgroup { · line 208

Quoted 2 lines
fs/resctrl/internal.h +1 -0
+ * @evt_name:		Name of memory transaction type (read, write etc).

Unclear what "type" means ... maybe just "Name of memory transaction (read, write ...)"?

sure.

Quoted 2 lines

The "evt_" prefix looks unnecessary.

ok

Quoted 4 lines

fs/resctrl/internal.h struct rdtgroup { · line 209

Quoted 2 lines
fs/resctrl/internal.h +1 -0
+ * @evt_val:		Value representing the memory transaction.

This could just be "val" and the description could be specific:

ok.

Quoted 3 lines

"The bit used to represent the memory transaction within an event's configuration." Please feel free to improve.

Sounds good.

Quoted 7 lines

fs/resctrl/internal.h struct rdtgroup { · line 213

Quoted 5 lines
fs/resctrl/internal.h +4 -0
+ */
+struct mbm_evt_value {
+	char    evt_name[32];
+	u32     evt_val;

Please space member names with TABs.

Sure.

Quoted 12 lines
Quoted 12 lines
fs/resctrl/internal.h +2 -0
+};
+
 /* rdtgroup.flags */
 #define	RDT_DELETED		1
 
fs/resctrl/rdtgroup.c +0 -0
index 72317a5adee2..b109e91096b0 100644
@@ -75,6 +75,20 @@ static void rdtgroup_destroy_root(void);
 
Quoted 5 lines

fs/resctrl/rdtgroup.c static void rdtgroup_destroy_root(void); · line 78

Quoted 3 lines
fs/resctrl/rdtgroup.c +1 -0
 struct dentry *debugfs_resctrl;
 
+/* Number of memory transaction types that can be monitored */

"Number of memory transactions that an MBM event can be configured with."?

Sure.

Quoted 7 lines

fs/resctrl/rdtgroup.c static void rdtgroup_destroy_root(void); · line 81

Quoted 4 lines
fs/resctrl/rdtgroup.c +3 -0
+#define NUM_MBM_EVT_VALUES             7
+
+/* Decoded values for each type of memory events */

Please be consistent with terminology. In the above lines it switches between "memory transaction types" and "memory events".

"Decoded values for each type of memory transaction types"

Quoted 17 lines
Quoted 4 lines
fs/resctrl/rdtgroup.c +3 -0
+struct mbm_evt_value mbm_evt_values[NUM_MBM_EVT_VALUES] = {
+	{"local_reads", READS_TO_LOCAL_MEM},
+	{"remote_reads", READS_TO_REMOTE_MEM},

fs/resctrl/rdtgroup.c static void rdtgroup_destroy_root(void); · line 94

Quoted 10 lines
fs/resctrl/rdtgroup.c +7 -0
+	{"local_non_temporal_writes", NON_TEMP_WRITE_TO_LOCAL_MEM},
+	{"remote_non_temporal_writes", NON_TEMP_WRITE_TO_REMOTE_MEM},
+	{"local_reads_slow_memory", READS_TO_LOCAL_S_MEM},
+	{"remote_reads_slow_memory", READS_TO_REMOTE_S_MEM},
+	{"dirty_victim_writes_all", DIRTY_VICTIMS_TO_ALL_MEM},
+};
+
 /*
  * Memory bandwidth monitoring event to use for the default CTRL_MON group
  * and each new CTRL_MON group created by the user.  Only relevant when

Reinette

Signature · 2 lines
Thanks
Babu Moger
author reviewer
#76 lore raw

Re: [PATCH v13 19/27] x86/resctrl: Add event configuration directory under info/L3_MON/

In reply to Reinette Chatre

Hi Reinette,

Quoted 18 lines · Reinette Chatre

Hi Babu,

Quoted 3 lines · Babu Moger

Create the configuration directory and files for mbm_cntr_assign mode. These configurations will be used to assign MBM events in mbm_cntr_assign mode, with two default configurations created upon mounting.

This just jumps in with what the patch does. Requirements for proper changelog should be familiar by now. The changelog *always* starts with a context.

Sample:

"When assignable counters are supported the /sys/fs/resctrl/info/L3_MON/event_configs directory contains a sub-directory for each MBM event that can be assigned to a counter. The MBM event sub-directory contains a file named "event_filter" that is used to view and modify which memory transactions the MBM event is configured with.

Quoted 6 lines

Create the /sys/fs/resctrl/info/L3_MON/event_configs directory on resctrl mount and pre-populate it with directories for the two existing MBM events:
mbm_total_bytes and mbm_local_bytes. Create the "event_filter" file within each MBM event directory with the needed *show() that displays the memory transactions with which the MBM event is configured."

Looks good. Thanks. The directory name will be "event_configs". Will change it in the code as well.

Quoted 47 lines
Quoted 38 lines

Example:

$ cd /sys/fs/resctrl/
$ cat info/L3_MON/counter_configs/mbm_total_bytes/event_filter
  local_reads, remote_reads, local_non_temporal_writes,
  remote_non_temporal_writes, local_reads_slow_memory,
  remote_reads_slow_memory, dirty_victim_writes_all
$ cat info/L3_MON/counter_configs/mbm_local_bytes/event_filter
  local_reads, local_non_temporal_writes, local_reads_slow_memory

---
v13: Updated user doc (resctrl.rst).
Changed the name of the function resctrl_mkdir_info_configs to resctrl_mkdir_counter_configs().
Replaced seq_puts() with seq_putc() where applicable.
Removed RFTYPE_MON_CONFIG definition. Not required.
Changed the name of the flag RFTYPE_CONFIG to RFTYPE_ASSIGN_CONFIG. Reinette suggested RFTYPE_MBM_EVENT_CONFIG but RFTYPE_ASSIGN_CONFIG seemed shorter and pricise.
The configuration is created using evt_list.
Resolved conflicts caused by the recent FS/ARCH code restructure.
The monitor.c/rdtgroup.c files have been split between the FS and ARCH directories.

v12: New patch to hold the MBM event configurations for mbm_cntr_assign mode.

---
 Documentation/filesystems/resctrl.rst | 30 ++++++++++
 fs/resctrl/internal.h                 |  2 +
 fs/resctrl/monitor.c                  |  1 +
 fs/resctrl/rdtgroup.c                 | 80 +++++++++++++++++++++++++++
 4 files changed, 113 insertions(+)
Documentation/filesystems/resctrl.rst +0 -0
index 5cf2d742f04c..4eb9f007ba3d 100644

Documentation/filesystems/resctrl.rst with the following files: · line 311

Quoted 6 lines
Documentation/filesystems/resctrl.rst +3 -0
@@ -306,6 +306,36 @@ with the following files:
 	  # cat /sys/fs/resctrl/info/L3_MON/available_mbm_cntrs
 	  0=30;1=30
 
+"counter_configs":
+	When the "mbm_cntr_assign" mode is supported, a dedicated directory is created
+	under the "L3_MON" directory to store configuration files.

? it does not contain files but directories for each event, no?

Quoted 5 lines

It will help if the text is specific. For example,
"event_configs":
Directory that exists when mbm_cntr_evt_assign is supported. Contains sub-directory for each MBM event that can be assigned to a counter. Each MBM event sub-directory ...

Sounds good.

Quoted 6 lines

Documentation/filesystems/resctrl.rst with the following files: · line 313

Quoted 3 lines
Documentation/filesystems/resctrl.rst +2 -0
+
+	These files contain the list of configurable events. There are two default

So confusing ... terminology is all over the place. Which files are even talked about here? "configurable events" ... are these the memory transactions or MBM events?

Should be "memory trasactions"

Quoted 4 lines

Documentation/filesystems/resctrl.rst with the following files: · line 314

Quoted 2 lines
Documentation/filesystems/resctrl.rst +1 -0
+	configurations: mbm_local_bytes and mbm_total_bytes.

"two default configurations"? These are not "configurations" but "events", no?

Sure. Should be "two default events"

Quoted 5 lines

Documentation/filesystems/resctrl.rst with the following files: · line 316

Quoted 3 lines
Documentation/filesystems/resctrl.rst +2 -0
+
+	Following types of events are supported:

events -> memory transactions?

Sure.

Quoted 2 lines

I am unable to parse the above.

The following are the types of memory transactions that an MBM event can be configured with:

Quoted 16 lines
Quoted 4 lines
Documentation/filesystems/resctrl.rst +2 -0
+
+	==== ========================= ============================================================

Documentation/filesystems/resctrl.rst with the following files: · line 328

Quoted 10 lines
Documentation/filesystems/resctrl.rst +10 -0
+	Bits Name   		         Description
+	==== ========================= ============================================================
+	6    dirty_victim_writes_all     Dirty Victims from the QOS domain to all types of memory
+	5    remote_reads_slow_memory    Reads to slow memory in the non-local NUMA domain
+	4    local_reads_slow_memory     Reads to slow memory in the local NUMA domain
+	3    remote_non_temporal_writes  Non-temporal writes to non-local NUMA domain
+	2    local_non_temporal_writes   Non-temporal writes to local NUMA domain
+	1    remote_reads                Reads to memory in the non-local NUMA domain
+	0    local_reads                 Reads to memory in the local NUMA domain
+	==== ========================= ==========================================================

Why does user need to know the bit position used to represent the memory transaction?

Not required. Will remove it.

Quoted 47 lines
Quoted 47 lines
Documentation/filesystems/resctrl.rst +10 -0
+
+	For example::
+
+	  # cat /sys/fs/resctrl/info/L3_MON/counter_configs/mbm_total_bytes/event_filter
+	  local_reads, remote_reads, local_non_temporal_writes, remote_non_temporal_writes,
+	  local_reads_slow_memory, remote_reads_slow_memory, dirty_victim_writes_all
+
+	  # cat /sys/fs/resctrl/info/L3_MON/counter_configs/mbm_local_bytes/event_filter
+	  local_reads, local_non_temporal_writes, local_reads_slow_memory
+
 "max_threshold_occupancy":
 		Read/write file provides the largest value (in
 		bytes) at which a previously used LLC_occupancy
fs/resctrl/internal.h +2 -0
index 019d00bf5adf..446cc9cc61df 100644
@@ -238,6 +238,8 @@ struct mbm_evt_value {
 
 #define RFTYPE_DEBUG			BIT(10)
 
+#define RFTYPE_ASSIGN_CONFIG		BIT(11)
+
 #define RFTYPE_CTRL_INFO		(RFTYPE_INFO | RFTYPE_CTRL)
 
 #define RFTYPE_MON_INFO			(RFTYPE_INFO | RFTYPE_MON)
fs/resctrl/monitor.c +1 -0
index 72f3dfb5b903..1f72249a5c93 100644
@@ -932,6 +932,7 @@ int resctrl_mon_resource_init(void)
 					 RFTYPE_MON_INFO | RFTYPE_RES_CACHE);
 		resctrl_file_fflags_init("available_mbm_cntrs",
 					 RFTYPE_MON_INFO | RFTYPE_RES_CACHE);
+		resctrl_file_fflags_init("event_filter", RFTYPE_ASSIGN_CONFIG);
 	}
 
 	return 0;
fs/resctrl/rdtgroup.c +0 -0
index b109e91096b0..cf84e3a382ac 100644
@@ -1911,6 +1911,25 @@ static int resctrl_available_mbm_cntrs_show(struct kernfs_open_file *of,
 	return ret;
 }
 
Quoted 10 lines

fs/resctrl/rdtgroup.c static int resctrl_available_mbm_cntrs_show(struct kernfs_open_file *of, · line 1921

Quoted 8 lines
fs/resctrl/rdtgroup.c +8 -0
+static int event_filter_show(struct kernfs_open_file *of, struct seq_file *seq, void *v)
+{
+	struct mon_evt *mevt = rdt_kn_parent_priv(of->kn);
+	bool sep = false;
+	int i;
+
+	for (i = 0; i < NUM_MBM_EVT_VALUES; i++) {
+		if (mevt->evt_cfg & mbm_evt_values[i].evt_val) {

Still no idea where mevt->evt_cfg comes from. Patch ordering issue?

Yes.
Need to introduce evt_cfg member and also need to initialize the default values during the init. Will order it correctly to make little bit clear.

Quoted 51 lines
Quoted 51 lines
fs/resctrl/rdtgroup.c +36 -0
+			if (sep)
+				seq_putc(seq, ',');
+			seq_printf(seq, "%s", mbm_evt_values[i].evt_name);
+			sep = true;
+		}
+	}
+	seq_putc(seq, '\n');
+
+	return 0;
+}
+
 /* rdtgroup information files for one cache resource. */
 static struct rftype res_common_files[] = {
 	{
@@ -2035,6 +2054,12 @@ static struct rftype res_common_files[] = {
 		.seq_show	= mbm_local_bytes_config_show,
 		.write		= mbm_local_bytes_config_write,
 	},
+	{
+		.name		= "event_filter",
+		.mode		= 0444,
+		.kf_ops		= &rdtgroup_kf_single_ops,
+		.seq_show	= event_filter_show,
+	},
 	{
 		.name		= "mbm_assign_mode",
 		.mode		= 0444,
@@ -2317,6 +2342,55 @@ static int rdtgroup_mkdir_info_resdir(void *priv, char *name,
 	return ret;
 }
 
+static int resctrl_mkdir_counter_configs(struct rdt_resource *r, char *name)
+{
+	struct kernfs_node *l3_mon_kn, *kn_subdir, *kn_subdir2;
+	struct mon_evt *mevt;
+	int ret;
+
+	l3_mon_kn = kernfs_find_and_get(kn_info, name);
+	if (!l3_mon_kn)
+		return -ENOENT;
+
+	kn_subdir = kernfs_create_dir(l3_mon_kn, "counter_configs", l3_mon_kn->mode, NULL);
+	if (IS_ERR(kn_subdir)) {
+		kernfs_put(l3_mon_kn);
+		return PTR_ERR(kn_subdir);
+	}
+
+	ret = rdtgroup_kn_set_ugid(kn_subdir);
+	if (ret) {
+		kernfs_put(l3_mon_kn);
Quoted 8 lines

fs/resctrl/rdtgroup.c static int rdtgroup_mkdir_info_resdir(void *priv, char *name, · line 2368

Quoted 5 lines
fs/resctrl/rdtgroup.c +5 -0
+		return ret;
+	}
+
+	list_for_each_entry(mevt, &r->mon.evt_list, list) {
+		if (mevt->mbm_mode == MBM_MODE_ASSIGN) {

I do not think this "mbm_mode" is needed, resctrl_mon::mbm_cntr_assignable is already used earlier, so would for_each_mbm_event() from the telemetry work be useful here?

Yes. Will remove mbm_mode and use Tony's telemetry work.

Quoted 8 lines

fs/resctrl/rdtgroup.c static int rdtgroup_mkdir_info_resdir(void *priv, char *name, · line 2373

Quoted 6 lines
fs/resctrl/rdtgroup.c +5 -0
+			kn_subdir2 = kernfs_create_dir(kn_subdir, mevt->name,
+						       kn_subdir->mode, mevt);
+			if (IS_ERR(kn_subdir2)) {
+				ret = PTR_ERR(kn_subdir2);
+				goto config_out;

"grep goto fs/resctrl/rdtgroup.c" for naming conventions.

Yes. I see it. It should be "out_config".

Quoted 12 lines
Quoted 12 lines
fs/resctrl/rdtgroup.c +11 -0
+			}
+
+			ret = rdtgroup_kn_set_ugid(kn_subdir2);
+			if (ret)
+				goto config_out;
+
+			ret = rdtgroup_add_files(kn_subdir2, RFTYPE_ASSIGN_CONFIG);
+			if (!ret)
+				kernfs_activate(kn_subdir);
+		}
+	}
Quoted 8 lines

fs/resctrl/rdtgroup.c static int rdtgroup_mkdir_info_resdir(void *priv, char *name, · line 2389

Quoted 5 lines
fs/resctrl/rdtgroup.c +5 -0
+
+config_out:
+	kernfs_put(l3_mon_kn);
+	if (ret)
+		kernfs_remove(kn_subdir);

This looks unnecessary since caller does kernfs_remove() on error return. Compare with how rdtgroup_mkdir_info_resdir() handles errors.

Yes. Will remove it.

Quoted 24 lines
Quoted 11 lines
fs/resctrl/rdtgroup.c +4 -0
+
+	return ret;
+}
+
 static unsigned long fflags_from_resource(struct rdt_resource *r)
 {
 	switch (r->rid) {
@@ -2363,6 +2437,12 @@ static int rdtgroup_create_info_dir(struct kernfs_node *parent_kn)
 		ret = rdtgroup_mkdir_info_resdir(r, name, fflags);
 		if (ret)

fs/resctrl/rdtgroup.c static int rdtgroup_create_info_dir(struct kernfs_node *parent_kn) · line 2448

Quoted 10 lines
fs/resctrl/rdtgroup.c +6 -0
 			goto out_destroy;
+
+		if (r->mon.mbm_cntr_assignable) {
+			ret = resctrl_mkdir_counter_configs(r, name);
+			if (ret)
+				goto out_destroy;
+		}
 	}
 
 	ret = rdtgroup_kn_set_ugid(kn_info);

Reinette

Signature · 2 lines
Thanks
Babu Moger
maintainer
#72 lore raw

Re: [PATCH v13 18/27] x86/resctrl: Add definitions for MBM event configuration

In reply to Moger, Babu

Hi Babu,

Quoted 13 lines · Babu
Quoted 11 lines · Reinette Chatre
Quoted 2 lines · Babu Moger

+/**
+ * struct mbm_evt_value - Specific type of memory events.

I am trying to decipher the terminology. If these are events, then it becomes confusing since it becomes "these events are used to configure events". You mention "memory transaction" below, this sounds more accurate to me. Above could thus be:

struct mbm_evt_value - Memory transaction an MBM event can be configured with.

Quoted 8 lines

Sure.

Quoted 4 lines

The name of the struct could also do with a rename to avoid the "event" term that conflicts with the actual MBM events. Maybe "mbm_cfg_value" ... I do not think this is a good name so please consider what would work better.

I can change it to "mbm_config_value".

Looks good, thank you.

...

Quoted 8 lines
Quoted 6 lines

fs/resctrl/rdtgroup.c static void rdtgroup_destroy_root(void); · line 81

Quoted 3 lines
fs/resctrl/rdtgroup.c +3 -0
+#define NUM_MBM_EVT_VALUES             7
+
+/* Decoded values for each type of memory events */

Please be consistent with terminology. In the above lines it switches between "memory transaction types" and "memory events".

"Decoded values for each type of memory transaction types"

I do not think "type" is needed twice. Could also be:
"Decoded values of each memory transaction type."

Quoted 11 lines
Quoted 11 lines

fs/resctrl/rdtgroup.c static void rdtgroup_destroy_root(void); · line 90

Quoted 11 lines
fs/resctrl/rdtgroup.c +9 -0
+struct mbm_evt_value mbm_evt_values[NUM_MBM_EVT_VALUES] = {
+	{"local_reads", READS_TO_LOCAL_MEM},
+	{"remote_reads", READS_TO_REMOTE_MEM},
+	{"local_non_temporal_writes", NON_TEMP_WRITE_TO_LOCAL_MEM},
+	{"remote_non_temporal_writes", NON_TEMP_WRITE_TO_REMOTE_MEM},
+	{"local_reads_slow_memory", READS_TO_LOCAL_S_MEM},
+	{"remote_reads_slow_memory", READS_TO_REMOTE_S_MEM},
+	{"dirty_victim_writes_all", DIRTY_VICTIMS_TO_ALL_MEM},
+};

Reinette

#79 lore raw

Re: [PATCH v13 20/27] x86/resctrl: Provide interface to update the event configurations

In reply to Reinette Chatre

Hi Reinette,

Quoted 9 lines · Reinette Chatre

Hi Babu,

Quoted 3 lines · Babu Moger

Users can modify the event configuration by writing to the event_filter interface file. The event configurations for mbm_cntr_assign mode are located in /sys/fs/resctrl/info/event_configs/.

heh ... looks like you also started thinking that "event_configs" is a better name (also missing L3_MON).

Interesting... 'event_configs' seems to have made its way in somehow. Alright, I’ll go ahead and add L3_MON."

Quoted 46 lines
Quoted 35 lines

Update the assignments of all groups when the event configuration is modified.

Example:

$ cd /sys/fs/resctrl/
$ cat info/L3_MON/counter_configs/mbm_local_bytes/event_filter
   local_reads,local_non_temporal_writes,local_reads_slow_memory
$ echo "local_reads,local_non_temporal_writes" >
   info/L3_MON/counter_configs/mbm_total_bytes/event_filter
$ cat info/L3_MON/counter_configs/mbm_total_bytes/event_filter
   local_reads,local_non_temporal_writes

---
v13: Updated changelog for imperative mode.
Added function description in the prototype.
Updated the user doc resctrl.rst to address few feedback.
Resolved conflicts caused by the recent FS/ARCH code restructure.
The rdtgroup.c/monitor.c file has now been split between the FS and ARCH directories.

v12: New patch to modify event configurations.

---
  Documentation/filesystems/resctrl.rst |  12 +++
  fs/resctrl/rdtgroup.c                 | 120 +++++++++++++++++++++++++-
  2 files changed, 131 insertions(+), 1 deletion(-)
Documentation/filesystems/resctrl.rst +0 -0
index 4eb9f007ba3d..9923276826db 100644

...

Quoted 8 lines
fs/resctrl/rdtgroup.c +0 -0
index cf84e3a382ac..8c498b41be5d 100644
@@ -1930,6 +1930,123 @@ static int event_filter_show(struct kernfs_open_file *of, struct seq_file *seq,
  	return 0;
  }
 
Quoted 7 lines

fs/resctrl/rdtgroup.c static int event_filter_show(struct kernfs_open_file *of, struct seq_file *seq, · line 1935

Quoted 3 lines
fs/resctrl/rdtgroup.c +3 -0
+/**
+ * resctrl_group_assign - Update the counter assignments for the event in
+ *			  a group.

This name is very generic with an unexpected namespace. "rdtgroup_" prefix is often used for a function that operates on a rdtgroup. This can thus be "rdtgroup_assign_cntr()".

Sure.

Quoted 16 lines
Quoted 16 lines
fs/resctrl/rdtgroup.c +15 -0
+ * @r:		Resource to which update needs to be done.
+ * @rdtgrp:	Resctrl group.
+ * @evtid:	Event ID.
+ * @evt_cfg:	Event configuration value.
+ */
+static int resctrl_group_assign(struct rdt_resource *r, struct rdtgroup *rdtgrp,
+				enum resctrl_event_id evtid, u32 evt_cfg)
+{
+	struct rdt_mon_domain *d;
+	int cntr_id;
+
+	list_for_each_entry(d, &r->mon_domains, hdr.list) {
+		cntr_id = mbm_cntr_get(r, d, rdtgrp, evtid);
+		if (cntr_id >= 0 && d->cntr_cfg[cntr_id].evt_cfg != evt_cfg) {
+			d->cntr_cfg[cntr_id].evt_cfg = evt_cfg;
Quoted 8 lines

fs/resctrl/rdtgroup.c static int event_filter_show(struct kernfs_open_file *of, struct seq_file *seq, · line 1956

Quoted 6 lines
fs/resctrl/rdtgroup.c +6 -0
+			resctrl_arch_config_cntr(r, d, evtid, rdtgrp->mon.rmid,
+						 rdtgrp->closid, cntr_id, evt_cfg, true);
+		}
+	}
+
+	return 0;

Can just return void?

sure.

Quoted 9 lines

fs/resctrl/rdtgroup.c static int event_filter_show(struct kernfs_open_file *of, struct seq_file *seq, · line 1961

Quoted 6 lines
fs/resctrl/rdtgroup.c +5 -0
+}
+
+/**
+ * resctrl_update_assign - Update the counter assignments for the event for all
+ *			   the groups.

Again very generic with "update" and "assign" that seem redundant? How about "resctrl_assign_cntr_allrdtgrp()"?

Yes.

Quoted 7 lines

fs/resctrl/rdtgroup.c static int event_filter_show(struct kernfs_open_file *of, struct seq_file *seq, · line 1939

Quoted 4 lines
fs/resctrl/rdtgroup.c +3 -0
+ * @r:		Resource to which update needs to be done.
+ * @evtid:	Event ID.
+ * @evt_cfg:	Event configuration value.

Why are both event ID and evt_cfg needed? Could just passing mon_evt simplify this?

Quoted 9 lines

fs/resctrl/rdtgroup.c static int event_filter_show(struct kernfs_open_file *of, struct seq_file *seq, · line 1971

Quoted 7 lines
fs/resctrl/rdtgroup.c +7 -0
+ */
+static int resctrl_update_assign(struct rdt_resource *r, enum resctrl_event_id evtid,
+				 u32 evt_cfg)
+{
+	struct rdtgroup *prgrp, *crgrp;
+
+	/* Check if the cntr_id is associated to the event type updated */

Comment does not match code.

Will correct it.

Quoted 9 lines

fs/resctrl/rdtgroup.c static int event_filter_show(struct kernfs_open_file *of, struct seq_file *seq, · line 1977

Quoted 7 lines
fs/resctrl/rdtgroup.c +6 -0
+	list_for_each_entry(prgrp, &rdt_all_groups, rdtgroup_list) {
+		resctrl_group_assign(r, prgrp, evtid, evt_cfg);
+
+		list_for_each_entry(crgrp, &prgrp->mon.crdtgrp_list, mon.crdtgrp_list) {
+			resctrl_group_assign(r, crgrp, evtid, evt_cfg);
+		}

Unnecessary braces?

Yes.

Quoted 7 lines

fs/resctrl/rdtgroup.c static int event_filter_show(struct kernfs_open_file *of, struct seq_file *seq, · line 1956

Quoted 4 lines
fs/resctrl/rdtgroup.c +3 -0
+	}
+
+	return 0;

return void?

Sure.

Quoted 8 lines
Quoted 8 lines
fs/resctrl/rdtgroup.c +8 -0
+}
+
+static int resctrl_process_configs(char *tok, u32 *val)
+{
+	char *evt_str;
+	bool found;
+	int i;
+
Quoted 8 lines

fs/resctrl/rdtgroup.c static int event_filter_show(struct kernfs_open_file *of, struct seq_file *seq, · line 1993

Quoted 5 lines
fs/resctrl/rdtgroup.c +5 -0
+next_config:
+	if (!tok || tok[0] == '\0')
+		return 0;
+
+	/* Start processing the strings for each event type */

Does comment intend to describe one iteration or all iterations? Also, "event type" -> "memory transaction"?

Sure.

Quoted 8 lines

fs/resctrl/rdtgroup.c static int event_filter_show(struct kernfs_open_file *of, struct seq_file *seq, · line 1998

Quoted 6 lines
fs/resctrl/rdtgroup.c +5 -0
+	evt_str = strim(strsep(&tok, ","));
+	found = false;
+	for (i = 0; i < NUM_MBM_EVT_VALUES; i++) {
+		if (!strcmp(mbm_evt_values[i].evt_name, evt_str)) {
+			*val |=  mbm_evt_values[i].evt_val;

check spacing.

ok.

Quoted 13 lines

fs/resctrl/rdtgroup.c static int event_filter_show(struct kernfs_open_file *of, struct seq_file *seq, · line 2006

Quoted 9 lines
fs/resctrl/rdtgroup.c +8 -0
+			found = true;
+			break;
+		}
+	}
+
+	if (!found) {
+		rdt_last_cmd_printf("Invalid event type %s\n", evt_str);
+		return -EINVAL;

Looks like this will return partially initialized data. Please use a local variable in which to gather the new configuration and only assign that to provided pointer on success.

Yes. Makes sense.

Quoted 23 lines
Quoted 23 lines
fs/resctrl/rdtgroup.c +22 -0
+	}
+
+	goto next_config;
+}
+
+static ssize_t event_filter_write(struct kernfs_open_file *of, char *buf,
+				  size_t nbytes, loff_t off)
+{
+	struct rdt_resource *r = resctrl_arch_get_resource(RDT_RESOURCE_L3);
+	struct mon_evt *mevt = rdt_kn_parent_priv(of->kn);
+	u32 evt_cfg = 0;
+	int ret = 0;
+
+	/* Valid input requires a trailing newline */
+	if (nbytes == 0 || buf[nbytes - 1] != '\n')
+		return -EINVAL;
+
+	buf[nbytes - 1] = '\0';
+
+	cpus_read_lock();
+	mutex_lock(&rdtgroup_mutex);
+
Quoted 8 lines

fs/resctrl/rdtgroup.c static int event_filter_show(struct kernfs_open_file *of, struct seq_file *seq, · line 2034

Quoted 6 lines
fs/resctrl/rdtgroup.c +6 -0
+	rdt_last_cmd_clear();
+
+	if (!resctrl_arch_mbm_cntr_assign_enabled(r)) {
+		rdt_last_cmd_puts("mbm_cntr_assign mode is not enabled\n");
+		ret = -EINVAL;
+		goto unlock_out;

"grep goto fs/resctrl/rdtgroup.c"

Will change it to "out_unlock".

Quoted 9 lines

fs/resctrl/rdtgroup.c static int event_filter_show(struct kernfs_open_file *of, struct seq_file *seq, · line 2039

Quoted 6 lines
fs/resctrl/rdtgroup.c +5 -0
+	}
+
+	ret = resctrl_process_configs(buf, &evt_cfg);
+	if (!ret && mevt->evt_val != evt_cfg) {
+		mevt->evt_val = evt_cfg;

ah ... here it is. hmmm ... but it is mon_evt::evt_cfg, no? ah, fixed in next patch.

Yea. My bad.

Quoted 4 lines

I still seem to be missing something because I expected mon_evt::evt_cfg of mbm_total_bytes and mbm_local_bytes to be initialized with a starting default. I missed where this is done in this series.

Its not done in this series. Will do in next revision.

Quoted 29 lines
Quoted 16 lines
fs/resctrl/rdtgroup.c +10 -0
+		resctrl_update_assign(r, mevt->evtid, evt_cfg);
+	}
+
+unlock_out:
+	mutex_unlock(&rdtgroup_mutex);
+	cpus_read_unlock();
+
+	return ret ?: nbytes;
+}
+
  /* rdtgroup information files for one cache resource. */
  static struct rftype res_common_files[] = {
  	{
@@ -2056,9 +2173,10 @@ static struct rftype res_common_files[] = {
  	},

fs/resctrl/rdtgroup.c static struct rftype res_common_files[] = { · line 2182

Quoted 10 lines
fs/resctrl/rdtgroup.c +2 -1
  	{
  		.name		= "event_filter",
-		.mode		= 0444,
+		.mode		= 0644,
  		.kf_ops		= &rdtgroup_kf_single_ops,
  		.seq_show	= event_filter_show,
+		.write		= event_filter_write,
  	},
  	{
  		.name		= "mbm_assign_mode",

Reinette

#82 lore raw

Re: [PATCH v13 21/27] x86/resctrl: Introduce mbm_assign_on_mkdir to configure assignments

In reply to Reinette Chatre

Hi Reinette,

Quoted 10 lines · Reinette Chatre

Hi Babu,

Quoted 5 lines · Babu Moger

The mbm_cntr_assign mode provides an option to the user to assign a counter to an RMID, event pair and monitor the bandwidth as long as the counter is assigned.

Introduce a configuration option to automatically assign counter IDs

"assign counter IDs" -> "assign counter IDs to <what?>"

"Introduce a configuration option to automatically assign counter IDs to to an RMID, event pair when a resctrl group is created, provided the counter IDs are available."

Quoted 28 lines
Quoted 28 lines

when a resctrl group is created, provided the counters are available. By default, this option is enabled at boot.

---
v13: Added Suggested-by tag.
Resolved conflicts caused by the recent FS/ARCH code restructure.
The rdtgroup.c/monitor.c file has now been split between the FS and ARCH directories.

v12: New patch. Added after the discussion on the list.
https://lore.kernel.org/lkml/CALPaoCh8siZKjL_3yvOYGL4cF_n_38KpUFgHVGbQ86nD+Q2_SA@mail.gmail.com/

---
  Documentation/filesystems/resctrl.rst | 10 ++++++
  fs/resctrl/monitor.c                  |  2 ++
  fs/resctrl/rdtgroup.c                 | 44 +++++++++++++++++++++++++--
  include/linux/resctrl.h               |  2 ++
  4 files changed, 56 insertions(+), 2 deletions(-)
Documentation/filesystems/resctrl.rst +0 -0
index 9923276826db..356f1f918a86 100644
@@ -348,6 +348,16 @@ with the following files:
  	  # cat /sys/fs/resctrl/info/L3_MON/counter_configs/mbm_total_bytes/event_filter
  	   local_reads, local_non_temporal_writes
 
Quoted 4 lines

Documentation/filesystems/resctrl.rst with the following files: · line 352

Quoted 2 lines
Documentation/filesystems/resctrl.rst +2 -0
+"mbm_assign_on_mkdir":
+	Automatically assign the monitoring counters on resctrl group creation

assign the monitoring counters to what?

"Automatically assign counter IDs to an RMID, event pair on resctrl group creation if the counter IDs are available. It is enabled by default on boot and users can disable by writing to the interface."

Quoted 9 lines

Documentation/filesystems/resctrl.rst with the following files: · line 359

Quoted 7 lines
Documentation/filesystems/resctrl.rst +7 -0
+	if the counters are available. It is enabled by default on boot and users
+	can disable by writing to the interface.
+	::
+
+	  # echo 0 > /sys/fs/resctrl/info/L3_MON/mbm_assign_on_mkdir
+	  # cat /sys/fs/resctrl/info/L3_MON/mbm_assign_on_mkdir
+	  0

Please be explicit in docs what possible values are and what they mean.

Sure. I can print "enabled" or "disabled".

Quoted 25 lines
Quoted 25 lines
Documentation/filesystems/resctrl.rst +1 -0
+
  "max_threshold_occupancy":
  		Read/write file provides the largest value (in
  		bytes) at which a previously used LLC_occupancy
fs/resctrl/monitor.c +2 -0
index 1f72249a5c93..5f6c4b662f3b 100644
@@ -933,6 +933,8 @@ int resctrl_mon_resource_init(void)
  		resctrl_file_fflags_init("available_mbm_cntrs",
  					 RFTYPE_MON_INFO | RFTYPE_RES_CACHE);
  		resctrl_file_fflags_init("event_filter", RFTYPE_ASSIGN_CONFIG);
+		resctrl_file_fflags_init("mbm_assign_on_mkdir", RFTYPE_MON_INFO |
+					 RFTYPE_RES_CACHE);
  	}
 
  	return 0;
fs/resctrl/rdtgroup.c +0 -0
index 8c498b41be5d..0093b323d858 100644
@@ -2035,8 +2035,8 @@ static ssize_t event_filter_write(struct kernfs_open_file *of, char *buf,
  	}
 
Quoted 10 lines

fs/resctrl/rdtgroup.c static ssize_t event_filter_write(struct kernfs_open_file *of, char *buf, · line 2042

Quoted 8 lines
fs/resctrl/rdtgroup.c +2 -2
  	ret = resctrl_process_configs(buf, &evt_cfg);
-	if (!ret && mevt->evt_val != evt_cfg) {
-		mevt->evt_val = evt_cfg;
+	if (!ret && mevt->evt_cfg != evt_cfg) {
+		mevt->evt_cfg = evt_cfg;
  		resctrl_update_assign(r, mevt->evtid, evt_cfg);
  	}
 

Needs to be squashed.

Sure.

Quoted 21 lines
Quoted 21 lines
fs/resctrl/rdtgroup.c +16 -0
@@ -2047,6 +2047,39 @@ static ssize_t event_filter_write(struct kernfs_open_file *of, char *buf,
  	return ret ?: nbytes;
  }
 
+static int resctrl_mbm_assign_on_mkdir_show(struct kernfs_open_file *of,
+					    struct seq_file *s, void *v)
+{
+	struct rdt_resource *r = rdt_kn_parent_priv(of->kn);
+
+	seq_printf(s, "%u\n", r->mon.mbm_assign_on_mkdir);
+
+	return 0;
+}
+
+static ssize_t resctrl_mbm_assign_on_mkdir_write(struct kernfs_open_file *of,
+						 char *buf, size_t nbytes, loff_t off)
+{
+	struct rdt_resource *r = rdt_kn_parent_priv(of->kn);
+	bool value;
+	int ret;
Quoted 8 lines

fs/resctrl/rdtgroup.c static ssize_t event_filter_write(struct kernfs_open_file *of, char *buf, · line 2071

Quoted 6 lines
fs/resctrl/rdtgroup.c +6 -0
+
+	ret = kstrtobool(buf, &value);
+	if (ret)
+		return ret;
+
+	cpus_read_lock();

not traversing the domain list so hotplug lock not needed.

ok. Sure.

Quoted 5 lines

fs/resctrl/rdtgroup.c static ssize_t event_filter_write(struct kernfs_open_file *of, char *buf, · line 2072

Quoted 2 lines
fs/resctrl/rdtgroup.c +1 -0
+	mutex_lock(&rdtgroup_mutex);

rdtgroup_mutex seems only needed because the message buffer is cleared below, and this is why it is not required in the show()?

Hmm. I didnt think about that. Do you think it is required?

Quoted 31 lines
Quoted 31 lines
fs/resctrl/rdtgroup.c +17 -0
+	rdt_last_cmd_clear();
+
+	r->mon.mbm_assign_on_mkdir = value;
+
+	mutex_unlock(&rdtgroup_mutex);
+	cpus_read_unlock();
+
+	return ret ?: nbytes;
+}
+
  /* rdtgroup information files for one cache resource. */
  static struct rftype res_common_files[] = {
  	{
@@ -2056,6 +2089,13 @@ static struct rftype res_common_files[] = {
  		.seq_show	= rdt_last_cmd_status_show,
  		.fflags		= RFTYPE_TOP_INFO,
  	},
+	{
+		.name		= "mbm_assign_on_mkdir",
+		.mode		= 0644,
+		.kf_ops		= &rdtgroup_kf_single_ops,
+		.seq_show	= resctrl_mbm_assign_on_mkdir_show,
+		.write		= resctrl_mbm_assign_on_mkdir_write,
+	},
  	{
  		.name		= "num_closids",
  		.mode		= 0444,
include/linux/resctrl.h +0 -1
index cd24d1577e0a..d6435abdde7b 100644
--- a/include/linux/resctrl.h
Quoted 8 lines

include/linux/resctrl.h enum resctrl_schema_fmt { · line 281

Quoted 6 lines

+++ b/include/linux/resctrl.h

include/linux/resctrl.h +1 -0
@@ -278,6 +278,7 @@ enum resctrl_schema_fmt {
   *			monitoring events can be configured.
   * @num_mbm_cntrs:	Number of assignable monitoring counters
   * @mbm_cntr_assignable:Is system capable of supporting monitor assignment?
+ * @mbm_assign_on_mkdir:Auto enable monitor assignment on mkdir?

How is "monitor assignment" different from "counter assignment"?

I should be:

"Auto enable counter ID assignment on mkdir"

Quoted 15 lines

include/linux/resctrl.h struct resctrl_mon { · line 292

Quoted 12 lines
include/linux/resctrl.h +1 -0
   * @evt_list:		List of monitoring events
   */
  struct resctrl_mon {
@@ -285,6 +286,7 @@ struct resctrl_mon {
  	unsigned int		mbm_cfg_mask;
  	int			num_mbm_cntrs;
  	bool			mbm_cntr_assignable;
+	bool			mbm_assign_on_mkdir;
  	struct list_head	evt_list;
  };
 

Reinette

Thanks
Babu

#88 lore raw

Re: [PATCH v13 23/27] x86/resctrl: Introduce mbm_L3_assignments to list assignments in a group

In reply to Reinette Chatre

Hi Reinette,

Quoted 11 lines · Reinette Chatre

Hi Babu,

Quoted 5 lines · Babu Moger

Introduce the interface to display the assignment states for each group when mbm_cntr_assign mode is enabled.

The list is displayed in the following format:
<Event configuration>:<Domain id>=<Assignment type>

Should this just be <Event>? The information is just the event name, not its configuration that will be in the "event_filter" file.

Yes. Sure.

Quoted 5 lines
Quoted 5 lines

Event configuration: A valid event configuration listed in the /sys/fs/resctrl/info/L3_MON/counter_configs directory.

Quoted 8 lines
Quoted 8 lines

Domain ID: A valid domain ID number.

The assignment type can be one of the following:

_ : No event configuration assigned

e : Event configuration assigned in exclusive mode

This needs to change as well based on comments below(resctrl.rst). This is note for me.

Quoted 30 lines
Quoted 30 lines

Example:
$cd /sys/fs/resctrl
$cat mbm_L3_assignments
mbm_total_bytes:0=e;1=e
mbm_local_bytes:0=e;1=e

---
v13: Changelog update.
Few changes in mbm_L3_assignments_show() after moving the event config to evt_list. Resolved conflicts caused by the recent FS/ARCH code restructure.
The rdtgroup.c/monitor.c files have been split between the FS and ARCH directories.

v12: New patch:
Assignment interface moved inside the group based the discussion
https://lore.kernel.org/lkml/CALPaoCiii0vXOF06mfV=kVLBzhfNo0SFqt4kQGwGSGVUqvr2Dg@mail.gmail.com/#t

---
  Documentation/filesystems/resctrl.rst | 28 +++++++++++++++
  fs/resctrl/monitor.c                  |  1 +
  fs/resctrl/rdtgroup.c                 | 52 +++++++++++++++++++++++++++
  3 files changed, 81 insertions(+)
Documentation/filesystems/resctrl.rst +0 -0
index 356f1f918a86..2350c1f21f4e 100644
@@ -504,6 +504,34 @@ When the "mba_MBps" mount option is used all CTRL_MON groups will also contain:
  	/sys/fs/resctrl/info/L3_MON/mon_features changes the input
  	event.
 
Quoted 4 lines

Documentation/filesystems/resctrl.rst When the "mba_MBps" mount option is used all CTRL_MON groups will also contain: · line 508

Quoted 2 lines
Documentation/filesystems/resctrl.rst +2 -0
+"mbm_L3_assignments":
+	This interface file is created when the mbm_cntr_assign mode is supported

"This interface file is created when" -> "Exists when mbm_cntr_assign mode is supported"?

Sure,

Quoted 5 lines

Documentation/filesystems/resctrl.rst When the "mba_MBps" mount option is used all CTRL_MON groups will also contain: · line 509

Quoted 2 lines
Documentation/filesystems/resctrl.rst +1 -0
+	and shows the assignment status for each group.

This doc is in the portion documenting files in monitor groups. So rather:
"the assignment status for each group" -> "the counter assignment status for the MON group"?

ok.

Quoted 8 lines

Documentation/filesystems/resctrl.rst When the "mba_MBps" mount option is used all CTRL_MON groups will also contain: · line 513

Quoted 5 lines
Documentation/filesystems/resctrl.rst +4 -0
+
+	The assignment list is displayed in the following format:
+
+	<Event configuration>:<Domain id>=<Assignment type>

<Event configuration> -> <Event>

Ok.

Quoted 4 lines

Documentation/filesystems/resctrl.rst When the "mba_MBps" mount option is used all CTRL_MON groups will also contain: · line 515

Quoted 2 lines
Documentation/filesystems/resctrl.rst +2 -0
+
+	Event configuration: A valid event configuration listed in the

"A valid event in the /sys/fs/resctrl/info/L3_MON/event_configs directory"

Sure.

Quoted 7 lines

Documentation/filesystems/resctrl.rst When the "mba_MBps" mount option is used all CTRL_MON groups will also contain: · line 518

Quoted 4 lines
Documentation/filesystems/resctrl.rst +3 -0
+	/sys/fs/resctrl/info/L3_MON/counter_configs directory.
+
+	Domain ID: A valid domain ID number.

"A valid domain ID"

Quoted 7 lines

Documentation/filesystems/resctrl.rst When the "mba_MBps" mount option is used all CTRL_MON groups will also contain: · line 522

Quoted 4 lines
Documentation/filesystems/resctrl.rst +4 -0
+
+	Assignment types:
+
+	_ : No event configuration assigned

hmmm ... since the line has event as first field, would this not reflect the counter? That is "No counter assigned"

Sure.

Quoted 5 lines

Documentation/filesystems/resctrl.rst When the "mba_MBps" mount option is used all CTRL_MON groups will also contain: · line 524

Quoted 3 lines
Documentation/filesystems/resctrl.rst +2 -0
+
+	e : Event configuration assigned in exclusive mode

"Counter assigned exclusively"? (with exclusive defined somewhere)

Sure.

Quoted 6 lines

Documentation/filesystems/resctrl.rst When the "mba_MBps" mount option is used all CTRL_MON groups will also contain: · line 527

Quoted 4 lines
Documentation/filesystems/resctrl.rst +3 -0
+
+	Example:
+	To list the assignment states for the default group.

"the counter assignment states"?

ok

Quoted 31 lines
Quoted 31 lines
Documentation/filesystems/resctrl.rst +7 -0
+	::
+
+	  # cd /sys/fs/resctrl
+	  # cat mbm_L3_assignments
+	    mbm_total_bytes:0=e;1=e
+	    mbm_local_bytes:0=e;1=e
+
  Resource allocation rules
  -------------------------
 
fs/resctrl/monitor.c +1 -0
index 5f6c4b662f3b..b982540ce4e3 100644
@@ -935,6 +935,7 @@ int resctrl_mon_resource_init(void)
  		resctrl_file_fflags_init("event_filter", RFTYPE_ASSIGN_CONFIG);
  		resctrl_file_fflags_init("mbm_assign_on_mkdir", RFTYPE_MON_INFO |
  					 RFTYPE_RES_CACHE);
+		resctrl_file_fflags_init("mbm_L3_assignments", RFTYPE_MON_BASE);
  	}
 
  	return 0;
fs/resctrl/rdtgroup.c +0 -0
index 931ea355f159..8d970b99bbbd 100644
@@ -2080,6 +2080,52 @@ static ssize_t resctrl_mbm_assign_on_mkdir_write(struct kernfs_open_file *of,
  	return ret ?: nbytes;
  }
 
Quoted 14 lines

fs/resctrl/rdtgroup.c static ssize_t resctrl_mbm_assign_on_mkdir_write(struct kernfs_open_file *of, · line 2094

Quoted 12 lines
fs/resctrl/rdtgroup.c +12 -0
+static int mbm_L3_assignments_show(struct kernfs_open_file *of, struct seq_file *s, void *v)
+{
+	struct rdt_resource *r = resctrl_arch_get_resource(RDT_RESOURCE_L3);
+	struct rdt_mon_domain *d;
+	struct rdtgroup *rdtgrp;
+	struct mon_evt *mevt;
+	int ret = 0;
+	bool sep;
+
+	rdtgrp = rdtgroup_kn_lock_live(of->kn);
+	if (!rdtgrp)
+		return -ENOENT;

Missing a rdtgroup_kn_unlock()?

Yes. Will add it.

Quoted 9 lines

fs/resctrl/rdtgroup.c static ssize_t resctrl_mbm_assign_on_mkdir_write(struct kernfs_open_file *of, · line 2100

Quoted 7 lines
fs/resctrl/rdtgroup.c +6 -0
+
+	rdt_last_cmd_clear();
+	if (!resctrl_arch_mbm_cntr_assign_enabled(r)) {
+		rdt_last_cmd_puts("mbm_cntr_assign mode not enabled\n");
+		ret = -ENOENT;
+		goto assign_out;

grep goto fs/resctrl/rdtgroup.c

Will change it to "out_assign".

Quoted 6 lines

fs/resctrl/rdtgroup.c static ssize_t resctrl_mbm_assign_on_mkdir_write(struct kernfs_open_file *of, · line 2103

Quoted 4 lines
fs/resctrl/rdtgroup.c +3 -0
+	}
+
+	list_for_each_entry(mevt, &r->mon.evt_list, list) {

can use for_each_mbm_event() and then below will not be needed?

Yes.

Quoted 36 lines
Quoted 36 lines
fs/resctrl/rdtgroup.c +28 -0
+		if (mevt->mbm_mode != MBM_MODE_ASSIGN)
+			continue;
+
+		sep = false;
+		seq_printf(s, "%s:", mevt->name);
+		list_for_each_entry(d, &r->mon_domains, hdr.list) {
+			if (sep)
+				seq_putc(s, ';');
+
+			if (mbm_cntr_get(r, d, rdtgrp, mevt->evtid) >= 0)
+				seq_printf(s, "%d=e", d->hdr.id);
+			else
+				seq_printf(s, "%d=_", d->hdr.id);
+
+			sep = true;
+		}
+		seq_putc(s, '\n');
+	}
+
+assign_out:
+	rdtgroup_kn_unlock(of->kn);
+
+	return ret;
+}
+
  /* rdtgroup information files for one cache resource. */
  static struct rftype res_common_files[] = {
  	{
@@ -2218,6 +2264,12 @@ static struct rftype res_common_files[] = {
  		.seq_show	= event_filter_show,
  		.write		= event_filter_write,
  	},
+	{
+		.name		= "mbm_L3_assignments",
+		.mode		= 0444,
Quoted 9 lines

fs/resctrl/rdtgroup.c static struct rftype res_common_files[] = { · line 2275

Quoted 6 lines
fs/resctrl/rdtgroup.c +3 -0
+		.kf_ops		= &rdtgroup_kf_single_ops,
+		.seq_show	= mbm_L3_assignments_show,
+	},
  	{
  		.name		= "mbm_assign_mode",
  		.mode		= 0444,

Reinette

Thanks
Babu