x86/resctrl : Support AMD Assignable Bandwidth Monitoring Counters (ABMC)
Subsystems (3)
Babu Moger
reviewerv13 cover x86/resctrl : Support AMD Assignable Bandwidth Monitoring Counters (ABMC)
This series adds the support for Assignable Bandwidth Monitoring Counters (ABMC). It is also called QoS RMID Pinning feature
Series is written such that it is easier to support other assignable features supported from different vendors.
The feature details are documented in the APM listed below [1].
[1] AMD64 Architecture Programmer's Manual Volume 2: System Programming Publication # 24593 Revision 3.41 section 19.3.3.3 Assignable Bandwidth Monitoring (ABMC). The documentation is available at
The patches are based on top of commit 92a09c47464d0 (tag: v6.15-rc5, tip/irq/merge) Linux 6.15-rc5 plus
https://lore.kernel.org/lkml/[email protected]/
It is very clear these patches will go after James's resctrl FS/ARCH restructure. Hoping to avoid one review cycle due to the merge.
# Introduction
Users can create as many monitor groups as RMIDs supported by the hardware. However, bandwidth monitoring feature on AMD system only guarantees that RMIDs currently assigned to a processor will be tracked by hardware. The counters of any other RMIDs which are no longer being tracked will be reset to zero. The MBM event counters return "Unavailable" for the RMIDs that are not tracked by hardware. So, there can be only limited number of groups that can give guaranteed monitoring numbers. With ever changing configurations there is no way to definitely know which of these groups are being tracked for certain point of time. Users do not have the option to monitor a group or set of groups for certain period of time without worrying about counter being reset in between.
The ABMC feature provides an option to the user to assign a hardware counter to an RMID, event pair and monitor the bandwidth as long as it is assigned. The assigned RMID will be tracked by the hardware until the user unassigns it manually. There is no need to worry about counters being reset during this period. Additionally, the user can specify a bitmask identifying the specific bandwidth types from the given source to track with the counter.
Without ABMC enabled, monitoring will work in current 'default' mode without assignment option.
# History
Earlier implementation of ABMC had dependancy on BMEC (Bandwidth Monitoring Event Configuration). Peter had concerns with that implementation because it may be not be compatible with ARM's MPAM.
Here are the threads discussing the concerns and new interface to address the concerns.
https://lore.kernel.org/lkml/CALPaoCg97cLVVAcacnarp+880xjsedEWGJPXhYpy4P7=ky4MZw@mail.gmail.com/
https://lore.kernel.org/lkml/CALPaoCiii0vXOF06mfV=kVLBzhfNo0SFqt4kQGwGSGVUqvr2Dg@mail.gmail.com/
Here are the finalized requirements based on the discussion:
* Remove BMEC dependency on the ABMC feature.
* Eliminate global assignment listing. The interface
/sys/fs/resctrl/info/L3_MON/mbm_assign_control is no longer required.
* Create the configuration directories at /sys/fs/resctrl/info/L3_MON/counter_configs/.
The configuration file names should be free-form, allowing users to create them as needed.
* Perform assignment listing at the group level by introducing mbm_L3_assignments
in each monitoring group. The listing should provide the following details:
Event Configuration: Specifies the event configuration applied. This will be crucial when "mkdir" on event configuration is added in the future, leading to the creation of mon_data/mon_l3_*/<event configuration>.
Domains: Identifies the domains where the configuration is applied, supporting multi-domain setups.
Assignment Type: Indicates whether the assignment is Exclusive (e or d), Shared (s), or Unassigned (_).
* Provide option to enable or disable auto assignment when new group is created.
This series tries to address all the requirements listed above.
# Implementation details
Create a generic interface aimed to support user space assignment of scarce counters used for monitoring. First usage of interface is by ABMC with option to expand usage to "soft-ABMC" and MPAM counters in future.
Feature adds following interface files:
/sys/fs/resctrl/info/L3_MON/mbm_assign_mode: Reports the list of assignable monitoring features supported. The enclosed brackets indicate which feature is enabled.
/sys/fs/resctrl/info/L3_MON/num_mbm_cntrs: Reports the number of monitoring counters available for assignment.
/sys/fs/resctrl/info/L3_MON/available_mbm_cntrs: Reports the number of monitoring counters free in each domain.
/sys/fs/resctrl/info/L3_MON/counter_configs : Directory to hold the counter configuration.
/sys/fs/resctrl/info/L3_MON/counter_configs/mbm_total_bytes/event_filter : Default configuration for MBM total events.
/sys/fs/resctrl/info/L3_MON/counter_configs/mbm_local_bytes/event_filter : Default configuration for MBM local events.
/sys/fs/resctrl/mbm_L3_assignments: Interface to list or modify assignment states on each group.
# Examples
a. Check if ABMC support is available
#mount -t resctrl resctrl /sys/fs/resctrl/
# cat /sys/fs/resctrl/info/L3_MON/mbm_assign_mode [mbm_cntr_assign]
default
ABMC feature is detected and it is enabled.
b. Check how many ABMC counters are available.
# cat /sys/fs/resctrl/info/L3_MON/num_mbm_cntrs 32
c. Check how many ABMC counters are available in each domain.
# cat /sys/fs/resctrl/info/L3_MON/available_mbm_cntrs 0=30;1=30
d. Check default counter configuration.
# cat /sys/fs/resctrl/info/L3_MON/counter_configs/mbm_total_bytes/event_filter
local_reads, remote_reads, local_non_temporal_writes, remote_non_temporal_writes,
local_reads_slow_memory, remote_reads_slow_memory, dirty_victim_writes_all# cat /sys/fs/resctrl/info/L3_MON/counter_configs/mbm_local_bytes/event_filter local_reads, local_non_temporal_writes, local_reads_slow_memory
e. Series adds a new interface file "mbm_L3_assignments" in each monitoring group
to list and modify any group's monitoring states.
The list is displayed in the following format:
<Event configuration>:<Domain id>=<Assignment type>
Event configuration: A valid event configuration listed in the /sys/fs/resctrl/info/L3_MON/counter_configs directory.
Domain ID: A valid domain ID number.
Assignment types:
_ : No event configuration assigned
e : Event configuration assigned in exclusive mode
To list the default group states:
# cat /sys/fs/resctrl/mbm_L3_assignments
mbm_total_bytes:0=e;1=e
mbm_local_bytes:0=e;1=e
To unassign the configuration of mbm_total_bytes on domain 0:
#echo "mbm_total_bytes:0=_" > mbm_L3_assignments #cat mbm_L3_assignments
mbm_total_bytes:0=_;1=e
mbm_local_bytes:0=e;1=e
To unassign the mbm_total_bytes configuration on all domains:
$echo "mbm_total_bytes:*=_" > mbm_L3_assignments $cat mbm_L3_assignments
mbm_total_bytes:0=_;1=_
mbm_local_bytes:0=e;1=e
To assign the mbm_total_bytes configuration on all domains in exclusive mode:
$echo "mbm_total_bytes:*=e" > mbm_L3_assignments
$cat mbm_L3_assignments
mbm_total_bytes:0=e;1=e
mbm_local_bytes:0=e;1=e
g. Read the events mbm_total_bytes and mbm_local_bytes of the default group.
There is no change in reading the events with ABMC. If the event is unassigned when reading, then the read will come back as "Unassigned".
# cat /sys/fs/resctrl/mon_data/mon_L3_00/mbm_total_bytes 779247936 # cat /sys/fs/resctrl/mon_data/mon_L3_00/mbm_local_bytes 765207488
h. Check the default event configurations.
#cat /sys/fs/resctrl/info/L3_MON/counter_configs/mbm_total_bytes/event_filter local_reads, remote_reads, local_non_temporal_writes, remote_non_temporal_writes, local_reads_slow_memory, remote_reads_slow_memory, dirty_victim_writes_all
#cat /sys/fs/resctrl/info/L3_MON/counter_configs/mbm_local_bytes/event_filter local_reads, local_non_temporal_writes, local_reads_slow_memory
i. Change the event configuration for mbm_local_bytes.
#echo "local_reads, local_non_temporal_writes, local_reads_slow_memory, remote_reads" > /sys/fs/resctrl/info/L3_MON/counter_configs/mbm_local_bytes/event_filter
#cat /sys/fs/resctrl/info/L3_MON/counter_configs/mbm_local_bytes/event_filter local_reads, local_non_temporal_writes, local_reads_slow_memory, remote_reads
This will update the assignments where mbm_local_bytes are configured.
j. Now read the total event again. The first read may come back with "Unavailable"
status. The subsequent read of mbm_total_bytes will display only the read events.
#cat /sys/fs/resctrl/mon_data/mon_L3_00/mbm_total_bytes Unavailable #cat /sys/fs/resctrl/mon_data/mon_L3_00/mbm_total_bytes 314101
k. Users will have the option to go back to 'default' mbm_assign_mode if required.
This can be done using the following command. Note that switching the mbm_assign_mode will reset all the MBM counters of all resctrl groups.
# echo "default" > /sys/fs/resctrl/info/L3_MON/mbm_assign_mode # cat /sys/fs/resctrl/info/L3_MON/mbm_assign_mode mbm_cntr_assign [default]
l. Unmount the resctrl
#umount /sys/fs/resctrl/
---
v13:
Removed BMEC related 2 patches which were in the previous series. It was related to optimization which can be doen later.
Patches are created on top of FS/ARCH restructure. So, major changes are due to FS/ARCH restructure. The files are split between arch/x86/kernel/cpu/resctrl/ and fs/resctrl/. So, functions are moved between these files accordingly.
Added fflag RFTYPE_RES_CACHE for mbm_assign_mode, num_mbm_cntrs, available_mbm_cntrs.
Removed the references to "mbm_assign_control".
Moved resctrl_arch_config_cntr() prototype to include/linux/resctrl.h. Changed resctrl_arch_config_cntr() to retun void from int to simplify few call sequences.
Added the event configuration details inside the evt_list in monitor domains. The avoids the need for new structure mbm_assign_config.
Passed evtid to functions resctrl_alloc_config_cntr() and resctrl_assign_cntr_event(). Event configuration value can be easily obtained from mon_evt list.
Added new patch to pass the entire struct rdtgroup to __mon_event_count(), mbm_update(), and related functions. We can easily get RMID,CLOSID etc from rdtgroup.
Added new function __cntr_id_read_phys() to handle ABMC event reading.
Added a new patch to hide BMEC related files when mbm_cntr_assign mode is enabled..
Added the call resctrl_init_evt_configuration() to setup the event configuration during init.
And few other commit message updates and user doc updates.
Removed Reviewed-by from few patches as patches have changed due to FS/ARCH restructure.
Let me know if I missed something.
v12:
This version is kind of RFC series with a new interface.
Removed Reviewed-by tag on few patches when the patch has changed.
Moved BMEC related patches (1 and 2) to beginning of the series. Removed the dependancy on BMEC to ABMC feature.
Removed the un-necessary initialization of mon_config_info structure. Changed wrmsrl instead of wrmsr to address the below comment.
https://lore.kernel.org/lkml/[email protected]/
Fixed the conflicts due to recent changes in rdt_resource data structure. Added new mbm_cfg_mask field to resctrl_mon.
Added the code to reset arch state inside _resctrl_abmc_enable().
Added the check CONFIG_RESCTRL_ASSIGN_FIXED to take care of arm platforms. This will be defined only in arm and not in x86.
Changed the code to display the max supported monitoring counters in each domain.
Fixed the struct mbm_cntr_cfg code documentation.
Moved the struct mbm_cntr_cfg definition to resctrl/internal.h as suggested by James.
Replaced seq_puts(s, ";") with seq_putc(s, ';');
Added missing rdt_last_cmd_clear() in resctrl_available_mbm_cntrs_show().
Added the check to reset the architecture-specific state only when assign is requested.
Added evt_cfg as the parameter to resctrl_arch_config_cntr() as the user will be passing the event configuration from /info/L3_MON/event_configs/.
Changed the check in resctrl_alloc_config_cntr() to reduce the indentation.
Fixed the handling error on first failure while assigning.
Added new parameter event configuration (evt_cfg) to get the event configuration from user space.
Added tte support for reading ABMC counters. This is bit involved change and affects lots of code.
New patch to support event configurations via new counter_configs method.
Removed mbm_cntr_reset() as it is not required while removing the group.
Added new patch to handle auto assign on group creation ("mbm_assign_on_mkdir")
Added couple of patches add interface for "mbm_L3_assignments" on each mon group.
Introduced mbm_cntr_free_all() and resctrl_reset_rmid_all() to clear counters and non-architectural states when monitor mode is changed.
https://lore.kernel.org/lkml/[email protected]/
Moved the resctrl_arch_mbm_cntr_assign_set_one to domain_add_cpu_mon().
Patches 17, 18, 19, 20, 21, 23, 24 are completely new to address the new interface requirement.
v11:
The commit 2937f9c361f7a ("x86/resctrl: Introduce resctrl_file_fflags_init() to initialize fflags") is already merged. Removed from the series.
Resolved minor conflicts due to code displacement in latest code.
Moved the monitoring related calls to monitor.c file when possible.
Moved some of the changes from include/linux/resctrl.h to arch/x86/kernel/cpu/resctrl/internal.h as requested by Reinette. This changes will be moved back when arch and non code is separated.
Renamed rdtgroup_mbm_assign_mode_show() to resctrl_mbm_assign_mode_show(). Renamed rdtgroup_num_mbm_cntrs_show() to resctrl_num_mbm_cntrs_show().
Moved the mon_config_info structure definition to internal.h.
Moved resctrl_arch_mon_event_config_get() and resctrl_arch_mon_event_config_set() to monitor.c file.
Moved resctrl_arch_assign_cntr() and resctrl_abmc_config_one_amd() to monitor.c. Added the code to reset the arch state in resctrl_arch_assign_cntr().
Also removed resctrl_arch_reset_rmid() inside IPI as the counters are reset from the callers.
Renamed rdtgroup_assign_cntr_event() to resctrl_assign_cntr_event(). Refactored the resctrl_assign_cntr_event().
Added functionality to exit on the first error during assignment. Simplified mbm_cntr_free().
Removed the function mbm_cntr_assigned(). Will be using mbm_cntr_get() to figure out if the counter is assigned or not.
Renamed rdtgroup_unassign_cntr_event() to resctrl_unassign_cntr_event(). Refactored the resctrl_unassign_cntr_event().
Moved mbm_cntr_reset() to monitor.c.
Added code reset non-architectural state in mbm_cntr_reset(). Added missing rdtgroup_unassign_cntrs() calls on failure path.
Domain can be NULL with SNC support so moved the unassign check in rdtgroup_mondata_show().
Renamed rdtgroup_mbm_assign_mode_write() to resctrl_mbm_assign_mode_write(). Added more details in resctrl.rst about mbm_cntr_assign mode. Re-arranged the text in resctrl.rst file in section mbm_cntr_assign.
Moved resctrl_arch_mbm_cntr_assign_set_one() to monitor.c
Added non-arch RMID reset in mbm_config_write_domain().
Removed resctrl_arch_reset_rmid() call in resctrl_abmc_config_one_amd(). Not required as reset of arch and non-arch rmid counters done from the callers. It simplies the IPI code.
Fixed printing the separator after each domain while listing the group assignments. Renamed rdtgroup_mbm_assign_control_show to resctrl_mbm_assign_control_show().
Fixed the static check warning with initializing dom_id in resctrl_process_flags()
Added change log in each patch for specific changes.
v10:
Major change is related to domain specific assignment.
Added struct mbm_cntr_cfg inside mon domains. This will handle
the domain specific assignments as discussed in below.
https://lore.kernel.org/lkml/CALPaoCj+zWq1vkHVbXYP0znJbe6Ke3PXPWjtri5AFgD9cQDCUg@mail.gmail.com/
I did not see the need to add cntr_id in mbm_state structure. Not used in the code. Following patches take care of these changes.
Patch 12, 13, 15, 16, 17, 18.
Added __init attribute to cache_alloc_hsw_probe(). Followed function prototype rules (preferred order is storage class before return type).
Moved the mon_config_info structure definition to resctrl.h
Added call resctrl_arch_reset_rmid() to reset the RMID in the domain inside IPI call resctrl_abmc_config_one_amd.
SMP and non-SMP call support is not required in resctrl_arch_config_cntr with new domain specific assign approach/data structure.
Assigned the counter before exposing the event files.
Moved the call rdtgroup_assign_cntrs() inside mkdir_rdt_prepare_rmid_alloc(). This is called both CNTR_MON and MON group creation.
Call mbm_cntr_reset() when unmounted to clear all the assignments.
Fixed the issue with finding the domain in multiple iterations in rdtgroup_process_flags().
Printed full error message with domain information when assign fails.
Taken care of other text comments in all the patches. Patch specific changes are in each patch.
If I missed something please point me and it is not intentional.
v9:
Patch 14 is a new addition.
Major change in patch 24.
Moved the fix patch to address __init attribute to begining of the series. Fixed all the call sequences. Added additional Fixed tags.
Added Reviewed-by where applicable.
Took care of couple of minor merge conflicts with latest code. Re-ordered the MSR in couple of instances.
Added available_mbm_cntrs (patch 14) to print the number of counter in a domain.
Used MBM_EVENT_ARRAY_INDEX macro to get the event index. Introduced rdtgroup_cntr_id_init() to initialize the cntr_id
Introduced new function resctrl_config_cntr to assign the counter, update the bitmap and reset the architectural state.
Taken care of error handling(freeing the counter) when assignment fails.
Changed rdtgroup_assign_cntrs() and rdtgroup_unassign_cntrs() to return void. Updated couple of rdtgroup_unassign_cntrs() calls properly.
Fixed problem changing the mode to mbm_cntr_assign mode when it is not supported. Added extra checks to detect if systems supports it.
https://lore.kernel.org/lkml/[email protected]/
As discussed in the above comment, introduced resctrl_mon_event_config_set to handle IPI. But sending another IPI inside IPI causes problem. Kernel reports SMP warning. So, introduced resctrl_arch_update_cntr() to send the command directly.
Fixed handling special case '//0=' and '//".
Removed extra strstr() call in rdtgroup_mbm_assign_control_write(). Added generic failure text when assignment operation fails. Corrected user documentation format texts.
v8:
Patches are getting into final stages.
Couple of changes Patch 8, Patch 19 and Patch 23.
Most of the other changes are related to rename and text message updates.
Details are in each patch. Here is the summary.
Added __init attribute to dom_data_init() in patch 8/25. Moved the mbm_cntrs_init() and mbm_cntrs_exit() functionality inside dom_data_init() and dom_data_exit() respectively.
Renamed resctrl_mbm_evt_config_init() to arch_mbm_evt_config_init() Renamed resctrl_arch_event_config_get() to resctrl_arch_mon_event_config_get().
resctrl_arch_event_config_set() to resctrl_arch_mon_event_config_set().
Rename resctrl_arch_assign_cntr to resctrl_arch_config_cntr. Renamed rdtgroup_assign_cntr() to rdtgroup_assign_cntr_event(). Added the code to return the error if rdtgroup_assign_cntr_event fails. Moved definition of MBM_EVENT_ARRAY_INDEX to resctrl/internal.h. Renamed rdtgroup_mbm_cntr_is_assigned to mbm_cntr_assigned_to_domain Added return error handling in resctrl_arch_config_cntr(). Renamed rdtgroup_assign_grp to rdtgroup_assign_cntrs.
Renamed rdtgroup_unassign_grp to rdtgroup_unassign_cntrs. Fixed the problem with unassigning the child MON groups of CTRL_MON group. Reset the internal counters after mbm_cntr_assign mode is changed. Renamed rdtgroup_mbm_cntr_reset() to mbm_cntr_reset()
Renamed resctrl_arch_mbm_cntr_assign_configure to
resctrl_arch_mbm_cntr_assign_set_one.
Used the same IPI as event update to modify the assignment. Could not do the way we discussed in the thread.
https://lore.kernel.org/lkml/[email protected]/
Needed to figure out event type to update the configuration.
Moved unassign first and assign during the assign modification. Assign none "_" takes priority. Cannot be mixed with other flags. Updated the documentation and .rst file format. htmldoc looks ok.
v7:
Major changes are related to FS and arch codes separation. Changed few interface names based on feedback.
Here are the summary and each patch contains changes specific the patch.
Removed WARN_ON for num_mbm_cntrs. Decided to dynamically allocate the bitmap. WARN_ON is not required anymore.
Renamed the function resctrl_arch_get_abmc_enabled() to resctrl_arch_mbm_cntr_assign_enabled().
Merged resctrl_arch_mbm_cntr_assign_disable, resctrl_arch_mbm_cntr_assign_disable and renamed to resctrl_arch_mbm_cntr_assign_set(). Passed the struct rdt_resource to these functions.
Removed resctrl_arch_reset_rmid_all() from arch code. This will be done from FS the caller.
Updated the descriptions/commit log in resctrl.rst to generic text. Removed ABMC references. Renamed mbm_mode to mbm_assign_mode.
Renamed mbm_control to mbm_assign_control.
Introduced mutex lock in rdtgroup_mbm_mode_show().
The 'legacy' mode is called 'default' mode.
Removed the static allocation and now allocating bitmap mbm_cntr_free_map dynamically.
Merged rdtgroup_assign_cntr(), rdtgroup_alloc_cntr() into one. Merged rdtgroup_unassign_cntr(), rdtgroup_free_cntr() into one.
Added struct rdt_resource to the interface functions resctrl_arch_assign_cntr () and resctrl_arch_unassign_cntr().
Rename rdtgroup_abmc_cfg() to resctrl_abmc_config_one_amd().
Added a new patch to fix counter assignment on event config changes.
Removed the references of ABMC from user interfaces.
Simplified the parsing (strsep(&token, "//") in rdtgroup_mbm_assign_control_write(). Added mutex lock in rdtgroup_mbm_assign_control_write() while processing.
Thomas Gleixner asked us to update https://gitlab.com/x86-cpuid.org/x86-cpuid-db.
It needs internal approval. We are working on it.
v6:
We still need to finalize few interface details on mbm_assign_mode and mbm_assign_control in case of ABMC and Soft-ABMC. We can continue the discussion with this series.
Added support for domain-id '*' to update all the domains at once. Fixed assign interface to allocate the counter if counter is not assigned.
Fixed unassign interface to free the counter if the counter is not assigned in any of the domains.
Renamed abmc_capable to mbm_cntr_assignable.
Renamed abmc_enabled to mbm_cntr_assign_enabled.
Used msr_set_bit and msr_clear_bit for msr updates.
Renamed resctrl_arch_abmc_enable() to resctrl_arch_mbm_cntr_assign_enable(). Renamed resctrl_arch_abmc_disable() to resctrl_arch_mbm_cntr_assign_disable().
Changed the display name from num_cntrs to num_mbm_cntrs.
Removed the variable mbm_cntrs_free_map_len. This is not required. Removed the call mbm_cntrs_init() in arch code. This needs to be done at higher level. Used DECLARE_BITMAP to initialize mbm_cntrs_free_map.
Removed unused config value definitions.
Introduced mbm_cntr_map to track counters at domain level. With this we dont need to send MSR read to read the counter configuration.
Separated all the counter id management to upper level in FS code.
Added checks to detect "Unassigned" before reading the RMID.
More details in each patch.
v5:
Rebase changes (because of SNC support)
Interface changes.
/sys/fs/resctrl/mbm_assign to /sys/fs/resctrl/mbm_assign_mode. /sys/fs/resctrl/mbm_assign_control to /sys/fs/resctrl/mbm_assign_control.
Added few arch specific routines.
resctrl_arch_get_abmc_enabled.
resctrl_arch_abmc_enable.
resctrl_arch_abmc_disable.
Few renames
num_cntrs_free_map -> mbm_cntrs_free_map num_cntrs_init -> mbm_cntrs_init arch_domain_mbm_evt_config -> resctrl_arch_mbm_evt_config
Introduced resctrl_arch_event_config_get and
resctrl_arch_event_config_set() to update event configuration.
Removed mon_state field mongroup. Added MON_CNTR_UNSET to initialize counters.
Renamed ctr_id to cntr_id for the hardware counter.
Report "Unassigned" in case the user attempts to read the events without assigning the counter.
ABMC is enabled during the boot up. Can be enabled or disabled later.
Fixed opcode and flags combination.
'=_" is valid.
"-_" amd "+_" is not valid.Added all the comments as far as I know. If I missed something, it is not intentional.
v4:
Main change is domain specific event assignment.
Kept the ABMC feature as a default.
Dynamcic switching between ABMC and mbm_legacy is still allowed. We are still not clear about mount option.
Moved the monitoring related data in resctrl_mon structure from rdt_resource. Fixed the display of legacy and ABMC mode.
Used bimap APIs when possible.
Removed event configuration read from MSRs. We can use the internal saved data.(patch 12)
Added more comments about L3_QOS_ABMC_CFG MSR.
Added IPIs to read the assignment status for each domain (patch 18 and 19) More details in each patch.
v3:
This series adds the support for global assignment mode discussed in the thread. https://lore.kernel.org/lkml/[email protected]/
Removed the individual assignment mode and included the global assignment interface. Added following interface files.
a. /sys/fs/resctrl/info/L3_MON/mbm_assign
Used for displaying the current assignment mode and switch between ABMC and legacy mode.
b. /sys/fs/resctrl/info/L3_MON/mbm_assign_control
Used for lising the groups assignment mode and modify the assignment states.
c. Most of the changes are related to the new interface.
d. Addressed the comments from Reinette, James and Peter.
e. Hope I have addressed most of the major feedbacks discussed. If I missed
something then it is not intentional. Please feel free to comment.
f. Sending this as an RFC as per Reinette's comment. So, this is still open
for discussion.
v2:
a. Major change is the way ABMC is enabled. Earlier, user needed to remount
with -o abmc to enable ABMC feature. Removed that option now.
Now users can enable ABMC by "$echo 1 to /sys/fs/resctrl/info/L3_MON/mbm_assign_enable".
b. Added new word 21 to x86/cpufeatures.h.
c. Display unsupported if user attempts to read the events when ABMC is enabled
and event is not assigned.
d. Display monitor_state as "Unsupported" when ABMC is disabled.
e. Text updates and rebase to latest tip tree (as of Jan 18).
f. This series is still work in progress. I am yet to hear from ARM developers.
--------------------------------------------------------------------------------------
Previous revisions:
v12: https://lore.kernel.org/lkml/[email protected]/
v11: https://lore.kernel.org/lkml/[email protected]/
v10: https://lore.kernel.org/lkml/[email protected]/
v9: https://lore.kernel.org/lkml/[email protected]/
v8: https://lore.kernel.org/lkml/[email protected]/
v7: https://lore.kernel.org/lkml/[email protected]/
v6: https://lore.kernel.org/lkml/[email protected]/
v5: https://lore.kernel.org/lkml/[email protected]/
v4: https://lore.kernel.org/lkml/[email protected]/
v3: https://lore.kernel.org/lkml/[email protected]/
v2: https://lore.kernel.org/lkml/[email protected]/
v1: https://lore.kernel.org/lkml/[email protected]/
Babu Moger (27):
x86/cpufeatures: Add support for Assignable Bandwidth Monitoring
Counters (ABMC)
x86/resctrl: Add ABMC feature in the command line options x86/resctrl: Consolidate monitoring related data from rdt_resource x86/resctrl: Detect Assignable Bandwidth Monitoring feature details x86/resctrl: Add support to enable/disable AMD ABMC feature x86/resctrl: Introduce the interface to display monitor mode x86/resctrl: Introduce interface to display number of monitoring
counters
x86/resctrl: Introduce mbm_cntr_cfg to track assignable counters at
domain
x86/resctrl: Introduce interface to display number of free MBM
counters
x86/resctrl: Add data structures and definitions for ABMC assignment x86/resctrl: Implement resctrl_arch_config_cntr() to assign a counter
with ABMC
x86/resctrl: Introduce event configuration modes x86/resctrl: Add the functionality to assign MBM events x86/resctrl: Add the functionality to unassign MBM events x86/resctrl: Report 'Unassigned' for MBM events in mbm_cntr_assign
mode
x86/resctrl: Pass entire struct rdtgroup rather than passing
individual members
x86/resctrl: Add the support for reading ABMC counters x86/resctrl: Add definitions for MBM event configuration x86/resctrl: Add event configuration directory under info/L3_MON/ x86/resctrl: Provide interface to update the event configurations x86/resctrl: Introduce mbm_assign_on_mkdir to configure assignments x86/resctrl: Auto assign/unassign counters when mbm_cntr_assign is
enabled
x86/resctrl: Introduce mbm_L3_assignments to list assignments in a
group
x86/resctrl: Introduce the interface to modify assignments in a group x86/resctrl: Hide the BMEC related files when mbm_cnt_assign is
enabled
x86/resctrl: Introduce the interface to switch between monitor modes x86/resctrl: Configure mbm_cntr_assign mode if supported
.../admin-guide/kernel-parameters.txt | 2 +- Documentation/filesystems/resctrl.rst | 188 +++++ arch/x86/include/asm/cpufeatures.h | 1 + arch/x86/include/asm/msr-index.h | 2 + arch/x86/kernel/cpu/cpuid-deps.c | 2 + arch/x86/kernel/cpu/resctrl/core.c | 13 +- arch/x86/kernel/cpu/resctrl/internal.h | 47 ++ arch/x86/kernel/cpu/resctrl/monitor.c | 176 +++- arch/x86/kernel/cpu/scattered.c | 1 + fs/resctrl/ctrlmondata.c | 14 + fs/resctrl/internal.h | 37 +- fs/resctrl/monitor.c | 309 ++++++- fs/resctrl/rdtgroup.c | 768 +++++++++++++++++- include/linux/resctrl.h | 74 +- include/linux/resctrl_types.h | 11 + 15 files changed, 1577 insertions(+), 68 deletions(-)
Signature · 1 line
2.34.1
Babu Moger
reviewerPATCH v13 1/27 x86/cpufeatures: Add support for Assignable Bandwidth Monitoring Counters (ABMC)
Users can create as many monitor groups as RMIDs supported by the hardware. However, bandwidth monitoring feature on AMD system only guarantees that RMIDs currently assigned to a processor will be tracked by hardware. The counters of any other RMIDs which are no longer being tracked will be reset to zero. The MBM event counters return "Unavailable" for the RMIDs that are not tracked by hardware. So, there can be only limited number of groups that can give guaranteed monitoring numbers. With ever changing configurations there is no way to definitely know which of these groups are being tracked for certain point of time. Users do not have the option to monitor a group or set of groups for certain period of time without worrying about RMID being reset in between.
The ABMC feature provides an option to the user to assign a hardware counter to an RMID, event pair and monitor the bandwidth as long as it is assigned. The assigned RMID will be tracked by the hardware until the user unassigns it manually. There is no need to worry about counters being reset during this period. Additionally, the user can specify a bitmask identifying the specific bandwidth types from the given source to track with the counter.
Without ABMC enabled, monitoring will work in current mode without assignment option.
The Linux resctrl subsystem provides an interface that allows monitoring of up to two memory bandwidth events per group, selected from a combination of available total and local events. When ABMC is enabled, two events will be assigned to each group by default, in line with the current interface design. Users will also have the option to configure which types of memory transactions are counted by these events.
Due to the limited number of available counters (32), users may quickly exhaust the available counters. If the system runs out of assignable ABMC counters, the kernel will report an error. In such cases, users will nee dto unassign one or more active counters to free up countes for new assignments. The interface will provide options to assign or unassign events through the group-specific interface file.
The feature can be detected via CPUID_Fn80000020_EBX_x00 bit 5. Bits Description
5 ABMC (Assignable Bandwidth Monitoring Counters)
The feature details are documented in APM listed below [1].
[1] AMD64 Architecture Programmer's Manual Volume 2: System Programming Publication # 24593 Revision 3.41 section 19.3.3.3 Assignable Bandwidth Monitoring (ABMC).
---
Note: Checkpatch checks/warnings are ignored to maintain coding style.
v13: Updated the commit log with Linux interface details.
v12: Removed the dependancy on X86_FEATURE_BMEC.
Removed the Reviewed-by tag as patch has changed.
v11: No changes.
v10: No changes.
v9: Took care of couple of minor merge conflicts. No other changes.
v8: No changes.
v7: Removed "" from feature flags. Not required anymore.
https://lore.kernel.org/lkml/20240817145058.GCZsC40neU4wkPXeVR@fat_crate.local/
v6: Added Reinette's Reviewed-by. Moved the Checkpatch note below ---.
v5: Minor rebase change and subject line update.
v4: Changes because of rebase. Feature word 21 has few more additions now.
Changed the text to "tracked by hardware" instead of active.
v3: Change because of rebase. Actual patch did not change.
v2: Added dependency on X86_FEATURE_BMEC.
--- arch/x86/include/asm/cpufeatures.h | 1 + arch/x86/kernel/cpu/cpuid-deps.c | 2 ++ arch/x86/kernel/cpu/scattered.c | 1 + 3 files changed, 4 insertions(+)
arch/x86/include/asm/cpufeatures.h +1 -0
@@ -481,6 +481,7 @@#define X86_FEATURE_AMD_HETEROGENEOUS_CORES (21*32 + 6)/* Heterogeneous Core Topology */ #define X86_FEATURE_AMD_WORKLOAD_CLASS (21*32 + 7)/* Workload Classification */ #define X86_FEATURE_PREFER_YMM (21*32 + 8)/* Avoid ZMM registers due to downclocking */ +#define X86_FEATURE_ABMC (21*32 + 9)/* Assignable Bandwidth Monitoring Counters */ /* * BUG word(s)
arch/x86/kernel/cpu/cpuid-deps.c +2 -0
@@ -71,6 +71,8 @@ static const struct cpuid_dep cpuid_deps[] = { { X86_FEATURE_CQM_MBM_LOCAL, X86_FEATURE_CQM_LLC }, { X86_FEATURE_BMEC, X86_FEATURE_CQM_MBM_TOTAL }, { X86_FEATURE_BMEC, X86_FEATURE_CQM_MBM_LOCAL }, + { X86_FEATURE_ABMC, X86_FEATURE_CQM_MBM_TOTAL }, + { X86_FEATURE_ABMC, X86_FEATURE_CQM_MBM_LOCAL }, { X86_FEATURE_AVX512_BF16, X86_FEATURE_AVX512VL }, { X86_FEATURE_AVX512_FP16, X86_FEATURE_AVX512BW }, { X86_FEATURE_ENQCMD, X86_FEATURE_XSAVES },
arch/x86/kernel/cpu/scattered.c +1 -0
@@ -49,6 +49,7 @@ static const struct cpuid_bit cpuid_bits[] = { { X86_FEATURE_MBA, CPUID_EBX, 6, 0x80000008, 0 }, { X86_FEATURE_SMBA, CPUID_EBX, 2, 0x80000020, 0 }, { X86_FEATURE_BMEC, CPUID_EBX, 3, 0x80000020, 0 }, + { X86_FEATURE_ABMC, CPUID_EBX, 5, 0x80000020, 0 }, { X86_FEATURE_AMD_WORKLOAD_CLASS, CPUID_EAX, 22, 0x80000021, 0 }, { X86_FEATURE_PERFMON_V2, CPUID_EAX, 0, 0x80000022, 0 }, { X86_FEATURE_AMD_LBR_V2, CPUID_EAX, 1, 0x80000022, 0 },
Signature · 1 line
2.34.1
Reinette Chatre
maintainerRe: [PATCH v13 01/27] x86/cpufeatures: Add support for Assignable Bandwidth Monitoring Counters (ABMC)
Hi Babu,
Quoted 30 lines · Babu Moger
Users can create as many monitor groups as RMIDs supported by the hardware. However, bandwidth monitoring feature on AMD system only guarantees that RMIDs currently assigned to a processor will be tracked by hardware. The counters of any other RMIDs which are no longer being tracked will be reset to zero. The MBM event counters return "Unavailable" for the RMIDs that are not tracked by hardware. So, there can be only limited number of groups that can give guaranteed monitoring numbers. With ever changing configurations there is no way to definitely know which of these groups are being tracked for certain point of time. Users do not have the option to monitor a group or set of groups for certain period of time without worrying about RMID being reset in between.
The ABMC feature provides an option to the user to assign a hardware counter to an RMID, event pair and monitor the bandwidth as long as it is assigned. The assigned RMID will be tracked by the hardware until the user unassigns it manually. There is no need to worry about counters being reset during this period. Additionally, the user can specify a bitmask identifying the specific bandwidth types from the given source to track with the counter.
Without ABMC enabled, monitoring will work in current mode without assignment option.
The Linux resctrl subsystem provides an interface that allows monitoring of up to two memory bandwidth events per group, selected from a combination of available total and local events. When ABMC is enabled, two events will be assigned to each group by default, in line with the current interface design. Users will also have the option to configure which types of memory transactions are counted by these events.
Quoted 4 lines
Due to the limited number of available counters (32), users may quickly exhaust the available counters. If the system runs out of assignable ABMC counters, the kernel will report an error. In such cases, users will nee dto unassign one or more active counters to free up countes for new
"nee dto" -> "need to"
"countes" -> "counters"
Quoted 1 line
assignments. The interface will provide options to assign or unassign
"The interface will" -> "resctrl will"?
Quoted 3 lines
events through the group-specific interface file.
The feature can be detected via
CPUID_Fn80000020_EBX_x00bit 5.
"The feature can be detected" -> "The feature is detected"
Quoted 11 lines
Bits Description
5 ABMC (Assignable Bandwidth Monitoring Counters)The feature details are documented in APM listed below [1].
[1] AMD64 Architecture Programmer's Manual Volume 2: System Programming Publication # 24593 Revision 3.41 section 19.3.3.3 Assignable Bandwidth Monitoring (ABMC).---
...
Quoted 22 lines
arch/x86/include/asm/cpufeatures.h | 1 + arch/x86/kernel/cpu/cpuid-deps.c | 2 ++ arch/x86/kernel/cpu/scattered.c | 1 + 3 files changed, 4 insertions(+)arch/x86/include/asm/cpufeatures.h +1
-0@@ -481,6 +481,7 @@#define X86_FEATURE_AMD_HETEROGENEOUS_CORES (21*32 + 6)/* Heterogeneous Core Topology */ #define X86_FEATURE_AMD_WORKLOAD_CLASS (21*32 + 7)/* Workload Classification */ #define X86_FEATURE_PREFER_YMM (21*32 + 8)/* Avoid ZMM registers due to downclocking */ +#define X86_FEATURE_ABMC (21*32 + 9)/* Assignable Bandwidth Monitoring Counters */ /* * BUG word(s) arch/x86/kernel/cpu/cpuid-deps.c +0
-0
arch/x86/kernel/cpu/cpuid-deps.c static const struct cpuid_dep cpuid_deps[] = { · line 75
Quoted 5 lines
arch/x86/kernel/cpu/cpuid-deps.c +2
-0@@ -71,6 +71,8 @@ static const struct cpuid_dep cpuid_deps[] = { { X86_FEATURE_CQM_MBM_LOCAL, X86_FEATURE_CQM_LLC }, { X86_FEATURE_BMEC, X86_FEATURE_CQM_MBM_TOTAL }, { X86_FEATURE_BMEC, X86_FEATURE_CQM_MBM_LOCAL }, + { X86_FEATURE_ABMC, X86_FEATURE_CQM_MBM_TOTAL }, + { X86_FEATURE_ABMC, X86_FEATURE_CQM_MBM_LOCAL },
Is this dependency still accurate now that the implementation switched to the "extended event ID" variant of ABMC that no longer uses the event IDs associated with X86_FEATURE_CQM_MBM_TOTAL and X86_FEATURE_CQM_MBM_LOCAL?
Quoted 8 lines
arch/x86/kernel/cpu/cpuid-deps.c +0
-0{ X86_FEATURE_AVX512_BF16, X86_FEATURE_AVX512VL }, { X86_FEATURE_AVX512_FP16, X86_FEATURE_AVX512BW }, { X86_FEATURE_ENQCMD, X86_FEATURE_XSAVES },arch/x86/kernel/cpu/scattered.c +0
-0
arch/x86/kernel/cpu/scattered.c static const struct cpuid_bit cpuid_bits[] = { · line 55
Quoted 7 lines
arch/x86/kernel/cpu/scattered.c +1
-0@@ -49,6 +49,7 @@ static const struct cpuid_bit cpuid_bits[] = { { X86_FEATURE_MBA, CPUID_EBX, 6, 0x80000008, 0 }, { X86_FEATURE_SMBA, CPUID_EBX, 2, 0x80000020, 0 }, { X86_FEATURE_BMEC, CPUID_EBX, 3, 0x80000020, 0 }, + { X86_FEATURE_ABMC, CPUID_EBX, 5, 0x80000020, 0 }, { X86_FEATURE_AMD_WORKLOAD_CLASS, CPUID_EAX, 22, 0x80000021, 0 }, { X86_FEATURE_PERFMON_V2, CPUID_EAX, 0, 0x80000022, 0 }, { X86_FEATURE_AMD_LBR_V2, CPUID_EAX, 1, 0x80000022, 0 },
Reinette
Moger, Babu
reviewerRe: [PATCH v13 01/27] x86/cpufeatures: Add support for Assignable Bandwidth Monitoring Counters (ABMC)
Hi Reinette,
Quoted 33 lines · Reinette Chatre
Hi Babu,
Quoted 30 lines · Babu Moger
Users can create as many monitor groups as RMIDs supported by the hardware. However, bandwidth monitoring feature on AMD system only guarantees that RMIDs currently assigned to a processor will be tracked by hardware. The counters of any other RMIDs which are no longer being tracked will be reset to zero. The MBM event counters return "Unavailable" for the RMIDs that are not tracked by hardware. So, there can be only limited number of groups that can give guaranteed monitoring numbers. With ever changing configurations there is no way to definitely know which of these groups are being tracked for certain point of time. Users do not have the option to monitor a group or set of groups for certain period of time without worrying about RMID being reset in between.
The ABMC feature provides an option to the user to assign a hardware counter to an RMID, event pair and monitor the bandwidth as long as it is assigned. The assigned RMID will be tracked by the hardware until the user unassigns it manually. There is no need to worry about counters being reset during this period. Additionally, the user can specify a bitmask identifying the specific bandwidth types from the given source to track with the counter.
Without ABMC enabled, monitoring will work in current mode without assignment option.
The Linux resctrl subsystem provides an interface that allows monitoring of up to two memory bandwidth events per group, selected from a combination of available total and local events. When ABMC is enabled, two events will be assigned to each group by default, in line with the current interface design. Users will also have the option to configure which types of memory transactions are counted by these events.
Quoted 7 lines
Quoted 4 lines
Due to the limited number of available counters (32), users may quickly exhaust the available counters. If the system runs out of assignable ABMC counters, the kernel will report an error. In such cases, users will nee dto unassign one or more active counters to free up countes for new
"nee dto" -> "need to"
"countes" -> "counters"
Sure.
Quoted 5 lines
Quoted 2 lines
assignments. The interface will provide options to assign or unassign
"The interface will" -> "resctrl will"?
Sure.
Quoted 6 lines
Quoted 3 lines
events through the group-specific interface file.
The feature can be detected via
CPUID_Fn80000020_EBX_x00bit 5."The feature can be detected" -> "The feature is detected"
Sure.
Quoted 28 lines
Quoted 11 lines
Bits Description
5 ABMC (Assignable Bandwidth Monitoring Counters)The feature details are documented in APM listed below [1].
[1] AMD64 Architecture Programmer's Manual Volume 2: System Programming Publication # 24593 Revision 3.41 section 19.3.3.3 Assignable Bandwidth Monitoring (ABMC).---
...
Quoted 15 lines
arch/x86/include/asm/cpufeatures.h | 1 + arch/x86/kernel/cpu/cpuid-deps.c | 2 ++ arch/x86/kernel/cpu/scattered.c | 1 + 3 files changed, 4 insertions(+)arch/x86/include/asm/cpufeatures.h +1
-0@@ -481,6 +481,7 @@#define X86_FEATURE_AMD_HETEROGENEOUS_CORES (21*32 + 6)/* Heterogeneous Core Topology */ #define X86_FEATURE_AMD_WORKLOAD_CLASS (21*32 + 7)/* Workload Classification */ #define X86_FEATURE_PREFER_YMM (21*32 + 8)/* Avoid ZMM registers due to downclocking */ +#define X86_FEATURE_ABMC (21*32 + 9)/* Assignable Bandwidth Monitoring Counters */
Quoted 16 lines
arch/x86/kernel/cpu/cpuid-deps.c static const struct cpuid_dep cpuid_deps[] = { · line 75
Quoted 12 lines
arch/x86/include/asm/cpufeatures.h +0
-0/* * BUG word(s) arch/x86/kernel/cpu/cpuid-deps.c +2
-0@@ -71,6 +71,8 @@ static const struct cpuid_dep cpuid_deps[] = { { X86_FEATURE_CQM_MBM_LOCAL, X86_FEATURE_CQM_LLC }, { X86_FEATURE_BMEC, X86_FEATURE_CQM_MBM_TOTAL }, { X86_FEATURE_BMEC, X86_FEATURE_CQM_MBM_LOCAL }, + { X86_FEATURE_ABMC, X86_FEATURE_CQM_MBM_TOTAL }, + { X86_FEATURE_ABMC, X86_FEATURE_CQM_MBM_LOCAL },Is this dependency still accurate now that the implementation switched to the "extended event ID" variant of ABMC that no longer uses the event IDs associated with
X86_FEATURE_CQM_MBM_TOTALandX86_FEATURE_CQM_MBM_LOCAL?
That's a good question. Unfortunately, we may need to retain this dependency for now, as a significant portion of the code relies on functions like resctrl_is_mbm_event(), resctrl_is_mbm_enabled(), resctrl_arch_is_mbm_total_enabled(), and others.
Quoted 19 lines
Quoted 9 lines
arch/x86/kernel/cpu/cpuid-deps.c +0
-0{ X86_FEATURE_AVX512_BF16, X86_FEATURE_AVX512VL }, { X86_FEATURE_AVX512_FP16, X86_FEATURE_AVX512BW }, { X86_FEATURE_ENQCMD, X86_FEATURE_XSAVES },arch/x86/kernel/cpu/scattered.c +0
-0arch/x86/kernel/cpu/scattered.c static const struct cpuid_bit cpuid_bits[] = { · line 55
Quoted 7 lines
arch/x86/kernel/cpu/scattered.c +1
-0@@ -49,6 +49,7 @@ static const struct cpuid_bit cpuid_bits[] = { { X86_FEATURE_MBA, CPUID_EBX, 6, 0x80000008, 0 }, { X86_FEATURE_SMBA, CPUID_EBX, 2, 0x80000020, 0 }, { X86_FEATURE_BMEC, CPUID_EBX, 3, 0x80000020, 0 }, + { X86_FEATURE_ABMC, CPUID_EBX, 5, 0x80000020, 0 }, { X86_FEATURE_AMD_WORKLOAD_CLASS, CPUID_EAX, 22, 0x80000021, 0 }, { X86_FEATURE_PERFMON_V2, CPUID_EAX, 0, 0x80000022, 0 }, { X86_FEATURE_AMD_LBR_V2, CPUID_EAX, 1, 0x80000022, 0 },Reinette
Signature · 2 lines
Thanks Babu Moger
Reinette Chatre
maintainerRe: [PATCH v13 01/27] x86/cpufeatures: Add support for Assignable Bandwidth Monitoring Counters (ABMC)
Hi Babu,
Quoted 18 lines · Babu
Quoted 16 lines · Reinette Chatre
arch/x86/kernel/cpu/cpuid-deps.c static const struct cpuid_dep cpuid_deps[] = { · line 75
Quoted 10 lines · Babu Moger
arch/x86/kernel/cpu/cpuid-deps.c +2
-0@@ -71,6 +71,8 @@ static const struct cpuid_dep cpuid_deps[] = { { X86_FEATURE_CQM_MBM_LOCAL, X86_FEATURE_CQM_LLC }, { X86_FEATURE_BMEC, X86_FEATURE_CQM_MBM_TOTAL }, { X86_FEATURE_BMEC, X86_FEATURE_CQM_MBM_LOCAL }, + { X86_FEATURE_ABMC, X86_FEATURE_CQM_MBM_TOTAL }, + { X86_FEATURE_ABMC, X86_FEATURE_CQM_MBM_LOCAL },Is this dependency still accurate now that the implementation switched to the "extended event ID" variant of ABMC that no longer uses the event IDs associated with
X86_FEATURE_CQM_MBM_TOTALandX86_FEATURE_CQM_MBM_LOCAL?
Quoted 5 lines
That's a good question. Unfortunately, we may need to retain this dependency for now, as a significant portion of the code relies on functions like
resctrl_is_mbm_event(),resctrl_is_mbm_enabled(),resctrl_arch_is_mbm_total_enabled(), and others.
Avoiding needing to change code is not a valid reason.
I think that without this dependency the code will still rely on "functions like resctrl_is_mbm_event(), resctrl_is_mbm_enabled(), resctrl_arch_is_mbm_total_enabled(), and others." though.
The core shift is to stop thinking about QOS_L3_MBM_TOTAL_EVENT_ID to mean the same as X86_FEATURE_CQM_MBM_TOTAL, similarly to stop thinking about QOS_L3_MBM_LOCAL_EVENT_ID to mean the same as X86_FEATURE_CQM_MBM_LOCAL.
I expected that for backwards compatibility ABMC will start by enabling QOS_L3_MBM_TOTAL_EVENT_ID and QOS_L3_MBM_LOCAL_EVENT_ID as part of its initialization, configuring them with the current defaults for which memory transactions are expected to be monitored by each. With these events enabled the existing flows using, for example, resctrl_is_mbm_event(), will continue to work as expected, no?
This would require more familiarity with L3 monitoring enumeration on AMD since it will still be required to determine the number of RMIDs etc. but if ABMC does not actually depend on these CQM features then the current enumeration would need to be re-worked anyway.
Reinette
Moger, Babu
reviewerRe: [PATCH v13 01/27] x86/cpufeatures: Add support for Assignable Bandwidth Monitoring Counters (ABMC)
Hi Reinette,
Quoted 34 lines · Reinette Chatre
Hi Babu,
Quoted 18 lines · Babu
Quoted 16 lines · Reinette Chatre
arch/x86/kernel/cpu/cpuid-deps.c static const struct cpuid_dep cpuid_deps[] = { · line 75
Quoted 10 lines · Babu Moger
arch/x86/kernel/cpu/cpuid-deps.c +2
-0@@ -71,6 +71,8 @@ static const struct cpuid_dep cpuid_deps[] = { { X86_FEATURE_CQM_MBM_LOCAL, X86_FEATURE_CQM_LLC }, { X86_FEATURE_BMEC, X86_FEATURE_CQM_MBM_TOTAL }, { X86_FEATURE_BMEC, X86_FEATURE_CQM_MBM_LOCAL }, + { X86_FEATURE_ABMC, X86_FEATURE_CQM_MBM_TOTAL }, + { X86_FEATURE_ABMC, X86_FEATURE_CQM_MBM_LOCAL },Is this dependency still accurate now that the implementation switched to the "extended event ID" variant of ABMC that no longer uses the event IDs associated with
X86_FEATURE_CQM_MBM_TOTALandX86_FEATURE_CQM_MBM_LOCAL?Quoted 5 lines
That's a good question. Unfortunately, we may need to retain this dependency for now, as a significant portion of the code relies on functions like
resctrl_is_mbm_event(),resctrl_is_mbm_enabled(),resctrl_arch_is_mbm_total_enabled(), and others.Avoiding needing to change code is not a valid reason.
I think that without this dependency the code will still rely on "functions like
resctrl_is_mbm_event(),resctrl_is_mbm_enabled(),resctrl_arch_is_mbm_total_enabled(), and others." though.
Quoted 4 lines
The core shift is to stop thinking about
QOS_L3_MBM_TOTAL_EVENT_IDto mean the same asX86_FEATURE_CQM_MBM_TOTAL, similarly to stop thinking aboutQOS_L3_MBM_LOCAL_EVENT_IDto mean the same asX86_FEATURE_CQM_MBM_LOCAL.
oh. ok.
Quoted 7 lines
I expected that for backwards compatibility ABMC will start by enabling
QOS_L3_MBM_TOTAL_EVENT_IDandQOS_L3_MBM_LOCAL_EVENT_IDas part of its initialization, configuring them with the current defaults for which memory transactions are expected to be monitored by each. With these events enabled the existing flows using, for example,resctrl_is_mbm_event(), will continue to work as expected, no?
Yes. It will work as it uses event id.
Quoted 5 lines
This would require more familiarity with L3 monitoring enumeration on AMD since it will still be required to determine the number of RMIDs etc. but if ABMC does not actually depend on these CQM features then the current enumeration would need to be re-worked anyway.
Are you suggesting to remove the dependency and rework ABMC enumeration in get_rdt_mon_resources()?
Signature · 2 lines
Thanks Babu Moger
Reinette Chatre
maintainerRe: [PATCH v13 01/27] x86/cpufeatures: Add support for Assignable Bandwidth Monitoring Counters (ABMC)
Hi Babu,
Quoted 50 lines · Babu
Quoted 32 lines · Reinette Chatre
Quoted 18 lines · Babu
Quoted 16 lines · Reinette Chatre
arch/x86/kernel/cpu/cpuid-deps.c static const struct cpuid_dep cpuid_deps[] = { · line 75
Quoted 10 lines · Babu Moger
arch/x86/kernel/cpu/cpuid-deps.c +2
-0@@ -71,6 +71,8 @@ static const struct cpuid_dep cpuid_deps[] = { { X86_FEATURE_CQM_MBM_LOCAL, X86_FEATURE_CQM_LLC }, { X86_FEATURE_BMEC, X86_FEATURE_CQM_MBM_TOTAL }, { X86_FEATURE_BMEC, X86_FEATURE_CQM_MBM_LOCAL }, + { X86_FEATURE_ABMC, X86_FEATURE_CQM_MBM_TOTAL }, + { X86_FEATURE_ABMC, X86_FEATURE_CQM_MBM_LOCAL },Is this dependency still accurate now that the implementation switched to the "extended event ID" variant of ABMC that no longer uses the event IDs associated with
X86_FEATURE_CQM_MBM_TOTALandX86_FEATURE_CQM_MBM_LOCAL?Quoted 5 lines
That's a good question. Unfortunately, we may need to retain this dependency for now, as a significant portion of the code relies on functions like
resctrl_is_mbm_event(),resctrl_is_mbm_enabled(),resctrl_arch_is_mbm_total_enabled(), and others.Avoiding needing to change code is not a valid reason.
I think that without this dependency the code will still rely on "functions like
resctrl_is_mbm_event(),resctrl_is_mbm_enabled(),resctrl_arch_is_mbm_total_enabled(), and others." though.Quoted 4 lines
The core shift is to stop thinking about
QOS_L3_MBM_TOTAL_EVENT_IDto mean the same asX86_FEATURE_CQM_MBM_TOTAL, similarly to stop thinking aboutQOS_L3_MBM_LOCAL_EVENT_IDto mean the same asX86_FEATURE_CQM_MBM_LOCAL.oh. ok.
Quoted 7 lines
I expected that for backwards compatibility ABMC will start by enabling
QOS_L3_MBM_TOTAL_EVENT_IDandQOS_L3_MBM_LOCAL_EVENT_IDas part of its initialization, configuring them with the current defaults for which memory transactions are expected to be monitored by each. With these events enabled the existing flows using, for example,resctrl_is_mbm_event(), will continue to work as expected, no?Yes. It will work as it uses event id.
Quoted 1 line
Quoted 8 lines
Quoted 4 lines
This would require more familiarity with L3 monitoring enumeration on AMD since it will still be required to determine the number of RMIDs etc. but if ABMC does not actually depend on these CQM features then the current enumeration would need to be re-worked anyway.
Are you suggesting to remove the dependency and rework ABMC enumeration in
get_rdt_mon_resources()?
If you have an alternative proposal that would accurately reflect the ABMC and existing L3 MON features then we can surely consider it.
Reinette
Re: [PATCH v13 01/27] x86/cpufeatures: Add support for Assignable Bandwidth Monitoring Counters (ABMC)
Hi Reinette,
Quoted 58 lines · Reinette Chatre
Hi Babu,
Quoted 55 lines · Babu
Quoted 32 lines · Reinette Chatre
Quoted 18 lines · Babu
Quoted 16 lines · Reinette Chatre
arch/x86/kernel/cpu/cpuid-deps.c static const struct cpuid_dep cpuid_deps[] = { · line 75
Quoted 10 lines · Babu Moger
arch/x86/kernel/cpu/cpuid-deps.c +2
-0@@ -71,6 +71,8 @@ static const struct cpuid_dep cpuid_deps[] = { { X86_FEATURE_CQM_MBM_LOCAL, X86_FEATURE_CQM_LLC }, { X86_FEATURE_BMEC, X86_FEATURE_CQM_MBM_TOTAL }, { X86_FEATURE_BMEC, X86_FEATURE_CQM_MBM_LOCAL }, + { X86_FEATURE_ABMC, X86_FEATURE_CQM_MBM_TOTAL }, + { X86_FEATURE_ABMC, X86_FEATURE_CQM_MBM_LOCAL },Is this dependency still accurate now that the implementation switched to the "extended event ID" variant of ABMC that no longer uses the event IDs associated with
X86_FEATURE_CQM_MBM_TOTALandX86_FEATURE_CQM_MBM_LOCAL?Quoted 5 lines
That's a good question. Unfortunately, we may need to retain this dependency for now, as a significant portion of the code relies on functions like
resctrl_is_mbm_event(),resctrl_is_mbm_enabled(),resctrl_arch_is_mbm_total_enabled(), and others.Avoiding needing to change code is not a valid reason.
I think that without this dependency the code will still rely on "functions like
resctrl_is_mbm_event(),resctrl_is_mbm_enabled(),resctrl_arch_is_mbm_total_enabled(), and others." though.Quoted 4 lines
The core shift is to stop thinking about
QOS_L3_MBM_TOTAL_EVENT_IDto mean the same asX86_FEATURE_CQM_MBM_TOTAL, similarly to stop thinking aboutQOS_L3_MBM_LOCAL_EVENT_IDto mean the same asX86_FEATURE_CQM_MBM_LOCAL.oh. ok.
Quoted 7 lines
I expected that for backwards compatibility ABMC will start by enabling
QOS_L3_MBM_TOTAL_EVENT_IDandQOS_L3_MBM_LOCAL_EVENT_IDas part of its initialization, configuring them with the current defaults for which memory transactions are expected to be monitored by each. With these events enabled the existing flows using, for example,resctrl_is_mbm_event(), will continue to work as expected, no?Yes. It will work as it uses event id.
Quoted 5 lines
This would require more familiarity with L3 monitoring enumeration on AMD since it will still be required to determine the number of RMIDs etc. but if ABMC does not actually depend on these CQM features then the current enumeration would need to be re-worked anyway.
Quoted 6 lines
Quoted 3 lines
Are you suggesting to remove the dependency and rework ABMC enumeration in
get_rdt_mon_resources()?If you have an alternative proposal that would accurately reflect the ABMC and existing L3 MON features then we can surely consider it.
I don't see any other option at this point. Will change it next revision.
Thanks
Babu
Babu Moger
reviewerPATCH v13 2/27 x86/resctrl: Add ABMC feature in the command line options
In reply to Babu MogerAdd the command line option to enable or disable exposing the ABMC (Assignable Bandwidth Monitoring Counters) hardware feature to resctrl.
---
v13: Removed the Reviewed-by as the file resctrl.rst is moved to
Documentation/filesystems/resctrl.rst. In that sense patch has changed.
v12: No changes.
v11: No changes.
v10: No changes.
v9: No code changes. Added Reviewed-by.
v8: Commit message update.
v7: No changes
v6: No changes
v5: No changes
v4: No changes
v3: No changes
v2: No changes
--- Documentation/admin-guide/kernel-parameters.txt | 2 +- Documentation/filesystems/resctrl.rst | 1 + arch/x86/kernel/cpu/resctrl/core.c | 2 ++ 3 files changed, 4 insertions(+), 1 deletion(-)
Documentation/admin-guide/kernel-parameters.txt +1 -1
@@ -5988,7 +5988,7 @@ rdt= [HW,X86,RDT] Turn on/off individual RDT features. List is: cmt, mbmtotal, mbmlocal, l3cat, l3cdp, l2cat, l2cdp, - mba, smba, bmec. + mba, smba, bmec, abmc. E.g. to turn on cmt and turn off mba use: rdt=cmt,!mba
Documentation/filesystems/resctrl.rst +1 -0
@@ -26,6 +26,7 @@ MBM (Memory Bandwidth Monitoring) "cqm_mbm_total", "cqm_mbm_local" MBA (Memory Bandwidth Allocation) "mba" SMBA (Slow Memory Bandwidth Allocation) "" BMEC (Bandwidth Monitoring Event Configuration) "" +ABMC (Assignable Bandwidth Monitoring Counters) "" =============================================== ================================ Historically, new features were made visible by default in /proc/cpuinfo. This
arch/x86/kernel/cpu/resctrl/core.c +2 -0
@@ -704,6 +704,7 @@ enum {RDT_FLAG_MBA ,RDT_FLAG_SMBA ,RDT_FLAG_BMEC , + RDT_FLAG_ABMC, };#define RDT_OPT (idx, n, f) \
@@ -729,6 +730,7 @@ static struct rdt_options rdt_options[] __ro_after_init = {RDT_OPT (RDT_FLAG_MBA ,"mba" ,X86_FEATURE_MBA ),RDT_OPT (RDT_FLAG_SMBA ,"smba" ,X86_FEATURE_SMBA ),RDT_OPT (RDT_FLAG_BMEC ,"bmec" ,X86_FEATURE_BMEC ), + RDT_OPT(RDT_FLAG_ABMC, "abmc", X86_FEATURE_ABMC), };#define NUM_RDT_OPTIONS ARRAY_SIZE(rdt_options)
Signature · 1 line
2.34.1
Babu Moger
reviewerPATCH v13 3/27 x86/resctrl: Consolidate monitoring related data from rdt_resource
In reply to Babu MogerThe cache allocation and memory bandwidth allocation feature properties are consolidated into struct resctrl_cache and struct resctrl_membw respectively.
In preparation for more monitoring properties that will clobber the existing resource struct more, re-organize the monitoring specific properties to also be in a separate structure.
---
v13: Changes due to FS/ARCH restructure.
v12: Fixed the conflicts due to recent changes in rdt_resource data structure.
Added new mbm_cfg_mask field to resctrl_mon.
Removed Reviewed-by tag as patch has changed.
v11: No changes.
v10: No changes.
v9: No changes.
v8: Added Reviewed-by from Reinette. No other changes.
v7: Added kernel doc for data structure. Minor text update.
v6: Update commit message and update kernel doc for rdt_resource.
v5: Commit message update.
Also changes related to data structure updates does to SNC support.
v4: New patch.
--- arch/x86/kernel/cpu/resctrl/core.c | 4 ++-- arch/x86/kernel/cpu/resctrl/monitor.c | 12 ++++++------ fs/resctrl/monitor.c | 8 ++++---- fs/resctrl/rdtgroup.c | 12 ++++++------ include/linux/resctrl.h | 22 +++++++++++++++------- 5 files changed, 33 insertions(+), 25 deletions(-)
arch/x86/kernel/cpu/resctrl/core.c +2 -2
@@ -106,7 +106,7 @@ u32 resctrl_arch_system_num_rmid_idx(void)struct rdt_resource *r = &rdt_resources_all[RDT_RESOURCE_L3 ].r_resctrl;/* RMID are independent numbers for x86. num_rmid_idx == num_rmid */ -return r->num_rmid; +return r->mon.num_rmid; }struct rdt_resource *resctrl_arch_get_resource (enum resctrl_res_level l)
@@ -534,7 +534,7 @@ static void domain_add_cpu_mon(int cpu, struct rdt_resource *r)arch_mon_domain_online (r, d); -if (arch_domain_mbm_alloc (r->num_rmid, hw_dom)) { +if (arch_domain_mbm_alloc (r->mon.num_rmid, hw_dom)) {mon_domain_free (hw_dom);return ; }
arch/x86/kernel/cpu/resctrl/monitor.c +6 -6
@@ -134,7 +134,7 @@ static int logical_rmid_to_physical_rmid(int cpu, int lrmid)if (snc_nodes_per_l3_cache ==1 )return lrmid; -return lrmid + (cpu_to_node (cpu) % snc_nodes_per_l3_cache) * r->num_rmid; +return lrmid + (cpu_to_node (cpu) % snc_nodes_per_l3_cache) * r->mon.num_rmid; }static int __rmid_read_phys (u32 prmid,enum resctrl_event_id eventid,u64 *val)
@@ -208,11 +208,11 @@ void resctrl_arch_reset_rmid_all(struct rdt_resource *r, struct rdt_mon_domain *if (resctrl_arch_is_mbm_total_enabled ())memset (hw_dom->arch_mbm_total,0 , -sizeof (*hw_dom->arch_mbm_total) * r->num_rmid); +sizeof (*hw_dom->arch_mbm_total) * r->mon.num_rmid);if (resctrl_arch_is_mbm_local_enabled ())memset (hw_dom->arch_mbm_local,0 , -sizeof (*hw_dom->arch_mbm_local) * r->num_rmid); +sizeof (*hw_dom->arch_mbm_local) * r->mon.num_rmid); }static u64 mbm_overflow_count (u64 prev_msr,u64 cur_msr,unsigned int width)
@@ -350,7 +350,7 @@ int __init rdt_get_mon_l3_config(struct rdt_resource *r) resctrl_rmid_realloc_limit = boot_cpu_data.x86_cache_size *1024 ; hw_res->mon_scale = boot_cpu_data.x86_cache_occ_scale / snc_nodes_per_l3_cache; - r->num_rmid = (boot_cpu_data.x86_cache_max_rmid +1 ) / snc_nodes_per_l3_cache; + r->mon.num_rmid = (boot_cpu_data.x86_cache_max_rmid +1 ) / snc_nodes_per_l3_cache; hw_res->mbm_width =MBM_CNTR_WIDTH_BASE ;if (mbm_offset >0 && mbm_offset <=MBM_CNTR_WIDTH_OFFSET_MAX )
@@ -365,7 +365,7 @@ int __init rdt_get_mon_l3_config(struct rdt_resource *r)* * For a 35MB LLC and 56 RMIDs, this is ~1.8% of the LLC. */ - threshold = resctrl_rmid_realloc_limit / r->num_rmid; + threshold = resctrl_rmid_realloc_limit / r->mon.num_rmid;/* * Because num_rmid may not be a power of two, round the value
@@ -379,7 +379,7 @@ int __init rdt_get_mon_l3_config(struct rdt_resource *r)/* Detect list of bandwidth sources that can be tracked */ cpuid_count (0x80000020 ,3 , &eax, &ebx, &ecx, &edx); - r->mbm_cfg_mask = ecx &MAX_EVT_CONFIG_BITS ; + r->mon.mbm_cfg_mask = ecx &MAX_EVT_CONFIG_BITS ; } r->mon_capable = true;
fs/resctrl/monitor.c +4 -4
@@ -866,14 +866,14 @@ static struct mon_evt mbm_local_event = {*/ static void l3_mon_evt_init (struct rdt_resource *r) { -INIT_LIST_HEAD (&r->evt_list); +INIT_LIST_HEAD (&r->mon.evt_list);if (resctrl_arch_is_llc_occupancy_enabled ()) -list_add_tail (&llc_occupancy_event.list, &r->evt_list); +list_add_tail (&llc_occupancy_event.list, &r->mon.evt_list);if (resctrl_arch_is_mbm_total_enabled ()) -list_add_tail (&mbm_total_event.list, &r->evt_list); +list_add_tail (&mbm_total_event.list, &r->mon.evt_list);if (resctrl_arch_is_mbm_local_enabled ()) -list_add_tail (&mbm_local_event.list, &r->evt_list); +list_add_tail (&mbm_local_event.list, &r->mon.evt_list); }/**
fs/resctrl/rdtgroup.c +6 -6
@@ -1139,7 +1139,7 @@ static int rdt_num_rmids_show(struct kernfs_open_file *of, {struct rdt_resource *r =rdt_kn_parent_priv (of->kn); -seq_printf (seq,"%d\n" , r->num_rmid); +seq_printf (seq,"%d\n" , r->mon.num_rmid);return 0 ; }
@@ -1150,7 +1150,7 @@ static int rdt_mon_features_show(struct kernfs_open_file *of,struct rdt_resource *r =rdt_kn_parent_priv (of->kn);struct mon_evt *mevt; -list_for_each_entry (mevt, &r->evt_list, list) { +list_for_each_entry (mevt, &r->mon.evt_list, list) {seq_printf (seq,"%s\n" , mevt->name);if (mevt->configurable)seq_printf (seq,"%s_config\n" , mevt->name);
@@ -1733,9 +1733,9 @@ static int mon_config_write(struct rdt_resource *r, char *tok, u32 evtid) }/* Value from user cannot be more than the supported set of events */ -if ((val & r->mbm_cfg_mask) != val) { +if ((val & r->mon.mbm_cfg_mask) != val) {rdt_last_cmd_printf ("Invalid event configuration: max valid mask is 0x%02x\n" , - r->mbm_cfg_mask); + r->mon.mbm_cfg_mask);return -EINVAL ; }
@@ -3055,10 +3055,10 @@ static int mon_add_all_files(struct kernfs_node *kn, struct rdt_mon_domain *d,struct mon_evt *mevt;int ret, domid; -if (WARN_ON (list_empty (&r->evt_list))) +if (WARN_ON (list_empty (&r->mon.evt_list)))return -EPERM ; -list_for_each_entry (mevt, &r->evt_list, list) { +list_for_each_entry (mevt, &r->mon.evt_list, list) { domid = do_sum ? d->ci->id : d->hdr.id; priv =mon_get_kn_priv (r->rid, domid, mevt, do_sum);if (WARN_ON_ONCE (!priv))
include/linux/resctrl.h +15 -7
@@ -255,40 +255,48 @@ enum resctrl_schema_fmt { RESCTRL_SCHEMA_RANGE, }; +/** +* struct resctrl_mon - Monitoring related data of a resctrl resource +* @num_rmid: Number of RMIDs available +* @mbm_cfg_mask: Bandwidth sources that can be tracked when bandwidth +* monitoring events can be configured. +* @evt_list: List of monitoring events +*/ +struct resctrl_mon { +int num_rmid; +unsigned int mbm_cfg_mask; +struct list_head evt_list; +}; +/** * struct rdt_resource - attributes of a resctrl resource * @rid: The index of the resource * @alloc_capable: Is allocation available on this machine * @mon_capable: Is monitor feature available on this machine -* @num_rmid: Number of RMIDs available * @ctrl_scope: Scope of this resource for control functions * @mon_scope: Scope of this resource for monitor functions * @cache: Cache allocation related data * @membw: If the component has bandwidth controls, their properties. +* @mon: Monitoring related data. * @ctrl_domains: RCU list of all control domains for this resource * @mon_domains: RCU list of all monitor domains for this resource * @name: Name to use in "schemata" file. * @schema_fmt: Which format string and parser is used for this schema. -* @evt_list: List of monitoring events -* @mbm_cfg_mask: Bandwidth sources that can be tracked when bandwidth -* monitoring events can be configured. * @cdp_capable: Is the CDP feature available on this resource */ struct rdt_resource {int rid;bool alloc_capable;bool mon_capable; -int num_rmid;enum resctrl_scope ctrl_scope;enum resctrl_scope mon_scope;struct resctrl_cache cache;struct resctrl_membw membw; +struct resctrl_mon mon;struct list_head ctrl_domains;struct list_head mon_domains;char *name;enum resctrl_schema_fmt schema_fmt; -struct list_head evt_list; -unsigned int mbm_cfg_mask;bool cdp_capable; };
Signature · 1 line
2.34.1
Reinette Chatre
maintainerRe: [PATCH v13 03/27] x86/resctrl: Consolidate monitoring related data from rdt_resource
Hi Babu,
Quoted 5 lines · Babu Moger
include/linux/resctrl.h +0
-0
include/linux/resctrl.h enum resctrl_schema_fmt { · line 264
Quoted 10 lines
include/linux/resctrl.h +7
-0@@ -255,40 +255,48 @@ enum resctrl_schema_fmt { RESCTRL_SCHEMA_RANGE, }; +/** +* struct resctrl_mon - Monitoring related data of a resctrl resource +* @num_rmid: Number of RMIDs available +* @mbm_cfg_mask: Bandwidth sources that can be tracked when bandwidth +* monitoring events can be configured. +* @evt_list: List of monitoring events +*/
Nit: this new comment portion can start with a clean slate by all sentences having good structure by starting with upper case and ending with period.
Reinette
Moger, Babu
reviewerRe: [PATCH v13 03/27] x86/resctrl: Consolidate monitoring related data from rdt_resource
Hi Reinette,
Quoted 12 lines · Reinette Chatre
Hi Babu,
Quoted 8 lines · Babu Moger
include/linux/resctrl.h +0
-0@@ -255,40 +255,48 @@ enum resctrl_schema_fmt { RESCTRL_SCHEMA_RANGE, };
Quoted 10 lines
include/linux/resctrl.h enum resctrl_schema_fmt { · line 264
Quoted 7 lines
include/linux/resctrl.h +7
-0+/** +* struct resctrl_mon - Monitoring related data of a resctrl resource +* @num_rmid: Number of RMIDs available +* @mbm_cfg_mask: Bandwidth sources that can be tracked when bandwidth +* monitoring events can be configured. +* @evt_list: List of monitoring events +*/ Nit: this new comment portion can start with a clean slate by all sentences having good structure by starting with upper case and ending with period.
Sure. Will do.
Signature · 2 lines
Thanks Babu Moger
Babu Moger
reviewerPATCH v13 4/27 x86/resctrl: Detect Assignable Bandwidth Monitoring feature details
In reply to Babu MogerABMC feature details are reported via CPUID Fn8000_0020_EBX_x5. Bits Description
15:0 MAX_ABMC Maximum Supported Assignable Bandwidth
Monitoring Counter ID + 1
The feature details are documented in APM listed below [1].
[1] AMD64 Architecture Programmer's Manual Volume 2: System Programming Publication # 24593 Revision 3.41 section 19.3.3.3 Assignable Bandwidth Monitoring (ABMC).
Detect the feature and number of assignable monitoring counters supported.
---
v13: No changes.
v12: Resolved conflicts because of latest merge.
Removed Reviewed-by as the patch has changed.
v11: No changes.
v10: No changes.
v9: Added Reviewed-by tag. No code changes
v8: Used GENMASK for the mask.
v7: Removed WARN_ON for num_mbm_cntrs. Decided to dynamically allocate the
bitmap. WARN_ON is not required anymore.
Removed redundant comments.
v6: Commit message update.
Renamed abmc_capable to mbm_cntr_assignable.
v5: Name change num_cntrs to num_mbm_cntrs.
Moved abmc_capable to resctrl_mon.
v4: Removed resctrl_arch_has_abmc(). Added all the code inline. We dont
need to separate this as arch code.
v3: Removed changes related to mon_features.
Moved rdt_cpu_has to core.c and added new function resctrl_arch_has_abmc. Also moved the fields mbm_assign_capable and mbm_assign_cntrs to rdt_resource. (James)
v2: Changed the field name to mbm_assign_capable from abmc_capable.
--- arch/x86/kernel/cpu/resctrl/monitor.c | 9 +++++++-- include/linux/resctrl.h | 4 ++++ 2 files changed, 11 insertions(+), 2 deletions(-)
arch/x86/kernel/cpu/resctrl/monitor.c +7 -2
@@ -345,6 +345,7 @@ int __init rdt_get_mon_l3_config(struct rdt_resource *r)unsigned int mbm_offset = boot_cpu_data.x86_cache_mbm_width_offset;struct rdt_hw_resource *hw_res =resctrl_to_arch_res (r);unsigned int threshold; +u32 eax, ebx, ecx, edx; snc_nodes_per_l3_cache =snc_get_config ();
@@ -375,13 +376,17 @@ int __init rdt_get_mon_l3_config(struct rdt_resource *r) resctrl_rmid_realloc_threshold =resctrl_arch_round_mon_val (threshold);if (rdt_cpu_has (X86_FEATURE_BMEC )) { -u32 eax, ebx, ecx, edx; -/* Detect list of bandwidth sources that can be tracked */ cpuid_count (0x80000020 ,3 , &eax, &ebx, &ecx, &edx); r->mon.mbm_cfg_mask = ecx &MAX_EVT_CONFIG_BITS ; } +if (rdt_cpu_has (X86_FEATURE_ABMC )) { + r->mon.mbm_cntr_assignable = true; +cpuid_count (0x80000020 ,5 , &eax, &ebx, &ecx, &edx); + r->mon.num_mbm_cntrs = (ebx &GENMASK (15 ,0 )) +1 ; + } + r->mon_capable = true;return 0 ;
include/linux/resctrl.h +4 -0
@@ -260,11 +260,15 @@ enum resctrl_schema_fmt {* @num_rmid: Number of RMIDs available * @mbm_cfg_mask: Bandwidth sources that can be tracked when bandwidth * monitoring events can be configured. +* @num_mbm_cntrs: Number of assignable monitoring counters +* @mbm_cntr_assignable:Is system capable of supporting monitor assignment? * @evt_list: List of monitoring events */ struct resctrl_mon {int num_rmid;unsigned int mbm_cfg_mask; +int num_mbm_cntrs; +bool mbm_cntr_assignable;struct list_head evt_list; };
Signature · 1 line
2.34.1
Reinette Chatre
maintainerRe: [PATCH v13 04/27] x86/resctrl: Detect Assignable Bandwidth Monitoring feature details
Hi Babu,
Quoted 15 lines · Babu Moger
ABMC feature details are reported via CPUID
Fn8000_0020_EBX_x5. Bits Description
15:0MAX_ABMCMaximum Supported Assignable Bandwidth
Monitoring Counter ID + 1The feature details are documented in APM listed below [1].
[1] AMD64 Architecture Programmer's Manual Volume 2: System Programming Publication # 24593 Revision 3.41 section 19.3.3.3 Assignable Bandwidth Monitoring (ABMC).Detect the feature and number of assignable monitoring counters supported.
---
...
Quoted 24 lines
--- arch/x86/kernel/cpu/resctrl/monitor.c | 9 +++++++-- include/linux/resctrl.h | 4 ++++ 2 files changed, 11 insertions(+), 2 deletions(-)arch/x86/kernel/cpu/resctrl/monitor.c +1
-2@@ -345,6 +345,7 @@ int __init rdt_get_mon_l3_config(struct rdt_resource *r)unsigned int mbm_offset = boot_cpu_data.x86_cache_mbm_width_offset;struct rdt_hw_resource *hw_res =resctrl_to_arch_res (r);unsigned int threshold; +u32 eax, ebx, ecx, edx; snc_nodes_per_l3_cache =snc_get_config ();@@ -375,13 +376,17 @@ int __init rdt_get_mon_l3_config(struct rdt_resource *r) resctrl_rmid_realloc_threshold =resctrl_arch_round_mon_val (threshold);if (rdt_cpu_has (X86_FEATURE_BMEC )) { -u32 eax, ebx, ecx, edx; -/* Detect list of bandwidth sources that can be tracked */
arch/x86/kernel/cpu/resctrl/monitor.c int __init rdt_get_mon_l3_config(struct rdt_resource *r) · line 389
Quoted 10 lines
arch/x86/kernel/cpu/resctrl/monitor.c +6
-0cpuid_count (0x80000020 ,3 , &eax, &ebx, &ecx, &edx); r->mon.mbm_cfg_mask = ecx &MAX_EVT_CONFIG_BITS ; } +if (rdt_cpu_has (X86_FEATURE_ABMC )) { + r->mon.mbm_cntr_assignable = true; +cpuid_count (0x80000020 ,5 , &eax, &ebx, &ecx, &edx); + r->mon.num_mbm_cntrs = (ebx &GENMASK (15 ,0 )) +1 ; + } +
Shouldn't ABMC detection also include enumeration of which configurations are supported? From what I can tell, looking ahead patch #18 hardcodes definitions of all known "bandwidth types" (which term to use TBD) and then patch #20 allows *any* of these types to be configured irrespective of whether system supports it.
AMD spec mentions "The types of L3 transactions that ABMC can track are configurable and identified by CPUID Fn8000_0020_ECX_x3." It thus looks like the enumeration of r->mon.mbm_cfg_mask when BMEC is enabled is required for ABMC also and used by this implementation.
include/linux/resctrl.h enum resctrl_schema_fmt { · line 264
Quoted 13 lines
arch/x86/kernel/cpu/resctrl/monitor.c +0
-0r->mon_capable = true;return 0 ;include/linux/resctrl.h +2
-0@@ -260,11 +260,15 @@ enum resctrl_schema_fmt {* @num_rmid: Number of RMIDs available * @mbm_cfg_mask: Bandwidth sources that can be tracked when bandwidth * monitoring events can be configured. +* @num_mbm_cntrs: Number of assignable monitoring counters +* @mbm_cntr_assignable:Is system capable of supporting monitor assignment?
"monitor assignment" has not been used so far, was this intended to be "counter assignment"?
Reinette
Moger, Babu
reviewerRe: [PATCH v13 04/27] x86/resctrl: Detect Assignable Bandwidth Monitoring feature details
Hi Reinette,
Quoted 49 lines · Reinette Chatre
Hi Babu,
Quoted 15 lines · Babu Moger
ABMC feature details are reported via CPUID
Fn8000_0020_EBX_x5. Bits Description
15:0MAX_ABMCMaximum Supported Assignable Bandwidth
Monitoring Counter ID + 1The feature details are documented in APM listed below [1].
[1] AMD64 Architecture Programmer's Manual Volume 2: System Programming Publication # 24593 Revision 3.41 section 19.3.3.3 Assignable Bandwidth Monitoring (ABMC).Detect the feature and number of assignable monitoring counters supported.
---
...
Quoted 28 lines
--- arch/x86/kernel/cpu/resctrl/monitor.c | 9 +++++++-- include/linux/resctrl.h | 4 ++++ 2 files changed, 11 insertions(+), 2 deletions(-)arch/x86/kernel/cpu/resctrl/monitor.c +1
-2@@ -345,6 +345,7 @@ int __init rdt_get_mon_l3_config(struct rdt_resource *r)unsigned int mbm_offset = boot_cpu_data.x86_cache_mbm_width_offset;struct rdt_hw_resource *hw_res =resctrl_to_arch_res (r);unsigned int threshold; +u32 eax, ebx, ecx, edx; snc_nodes_per_l3_cache =snc_get_config ();@@ -375,13 +376,17 @@ int __init rdt_get_mon_l3_config(struct rdt_resource *r) resctrl_rmid_realloc_threshold =resctrl_arch_round_mon_val (threshold);if (rdt_cpu_has (X86_FEATURE_BMEC )) { -u32 eax, ebx, ecx, edx; -/* Detect list of bandwidth sources that can be tracked */ cpuid_count (0x80000020 ,3 , &eax, &ebx, &ecx, &edx); r->mon.mbm_cfg_mask = ecx &MAX_EVT_CONFIG_BITS ; }
Quoted 16 lines
arch/x86/kernel/cpu/resctrl/monitor.c int __init rdt_get_mon_l3_config(struct rdt_resource *r) · line 389
Quoted 6 lines
arch/x86/kernel/cpu/resctrl/monitor.c +6
-0+if (rdt_cpu_has (X86_FEATURE_ABMC )) { + r->mon.mbm_cntr_assignable = true; +cpuid_count (0x80000020 ,5 , &eax, &ebx, &ecx, &edx); + r->mon.num_mbm_cntrs = (ebx &GENMASK (15 ,0 )) +1 ; + } +Shouldn't ABMC detection also include enumeration of which configurations are supported? From what I can tell, looking ahead patch #18 hardcodes definitions of all known "bandwidth types" (which term to use TBD) and then patch #20 allows *any* of these types to be configured irrespective of whether system supports it.
AMD spec mentions "The types of L3 transactions that ABMC can track are configurable and identified by CPUIDFn8000_0020_ECX_x3." It thus looks like the enumeration ofr->mon.mbm_cfg_maskwhen BMEC is enabled is required for ABMC also and used by this implementation.
That is true. Will add the following check.
if (rdt_cpu_has(X86_FEATURE_BMEC) || rdt_cpu_has(X86_FEATURE_ABMC)) {
/* Detect list of bandwidth sources that can be tracked */ cpuid_count(0x80000020, 3, &eax, &ebx, &ecx, &edx); r->mon.mbm_cfg_mask = ecx & MAX_EVT_CONFIG_BITS;
}
Quoted 20 lines
Quoted 9 lines
arch/x86/kernel/cpu/resctrl/monitor.c +0
-0r->mon_capable = true;return 0 ;include/linux/resctrl.h +0
-0include/linux/resctrl.h enum resctrl_schema_fmt { · line 264
Quoted 5 lines
include/linux/resctrl.h +2
-0@@ -260,11 +260,15 @@ enum resctrl_schema_fmt {* @num_rmid: Number of RMIDs available * @mbm_cfg_mask: Bandwidth sources that can be tracked when bandwidth * monitoring events can be configured. +* @num_mbm_cntrs: Number of assignable monitoring counters +* @mbm_cntr_assignable:Is system capable of supporting monitor assignment? "monitor assignment" has not been used so far, was this intended to be "counter assignment"?
Reinette
Signature · 2 lines
Thanks Babu Moger
Moger, Babu
reviewerRe: [PATCH v13 04/27] x86/resctrl: Detect Assignable Bandwidth Monitoring feature details
In reply to Reinette ChatreHi Reinette,
Quoted 68 lines · Reinette Chatre
Hi Babu,
Quoted 15 lines · Babu Moger
ABMC feature details are reported via CPUID
Fn8000_0020_EBX_x5. Bits Description
15:0MAX_ABMCMaximum Supported Assignable Bandwidth
Monitoring Counter ID + 1The feature details are documented in APM listed below [1].
[1] AMD64 Architecture Programmer's Manual Volume 2: System Programming Publication # 24593 Revision 3.41 section 19.3.3.3 Assignable Bandwidth Monitoring (ABMC).Detect the feature and number of assignable monitoring counters supported.
---
...
Quoted 24 lines
--- arch/x86/kernel/cpu/resctrl/monitor.c | 9 +++++++-- include/linux/resctrl.h | 4 ++++ 2 files changed, 11 insertions(+), 2 deletions(-)arch/x86/kernel/cpu/resctrl/monitor.c +1
-2@@ -345,6 +345,7 @@ int __init rdt_get_mon_l3_config(struct rdt_resource *r)unsigned int mbm_offset = boot_cpu_data.x86_cache_mbm_width_offset;struct rdt_hw_resource *hw_res =resctrl_to_arch_res (r);unsigned int threshold; +u32 eax, ebx, ecx, edx; snc_nodes_per_l3_cache =snc_get_config ();@@ -375,13 +376,17 @@ int __init rdt_get_mon_l3_config(struct rdt_resource *r) resctrl_rmid_realloc_threshold =resctrl_arch_round_mon_val (threshold);if (rdt_cpu_has (X86_FEATURE_BMEC )) { -u32 eax, ebx, ecx, edx; -/* Detect list of bandwidth sources that can be tracked */ arch/x86/kernel/cpu/resctrl/monitor.c int __init rdt_get_mon_l3_config(struct rdt_resource *r) · line 389
Quoted 10 lines
arch/x86/kernel/cpu/resctrl/monitor.c +6
-0cpuid_count (0x80000020 ,3 , &eax, &ebx, &ecx, &edx); r->mon.mbm_cfg_mask = ecx &MAX_EVT_CONFIG_BITS ; } +if (rdt_cpu_has (X86_FEATURE_ABMC )) { + r->mon.mbm_cntr_assignable = true; +cpuid_count (0x80000020 ,5 , &eax, &ebx, &ecx, &edx); + r->mon.num_mbm_cntrs = (ebx &GENMASK (15 ,0 )) +1 ; + } +Shouldn't ABMC detection also include enumeration of which configurations are supported? From what I can tell, looking ahead patch #18 hardcodes definitions of all known "bandwidth types" (which term to use TBD) and then patch #20 allows *any* of these types to be configured irrespective of whether system supports it.
AMD spec mentions "The types of L3 transactions that ABMC can track are configurable and identified by CPUIDFn8000_0020_ECX_x3." It thus looks like the enumeration ofr->mon.mbm_cfg_maskwhen BMEC is enabled is required for ABMC also and used by this implementation.Quoted 2 lines
arch/x86/kernel/cpu/resctrl/monitor.c +0
-0r->mon_capable = true;
Quoted 14 lines
include/linux/resctrl.h enum resctrl_schema_fmt { · line 264
Quoted 11 lines
arch/x86/kernel/cpu/resctrl/monitor.c +0
-0return 0 ;include/linux/resctrl.h +2
-0@@ -260,11 +260,15 @@ enum resctrl_schema_fmt {* @num_rmid: Number of RMIDs available * @mbm_cfg_mask: Bandwidth sources that can be tracked when bandwidth * monitoring events can be configured. +* @num_mbm_cntrs: Number of assignable monitoring counters +* @mbm_cntr_assignable:Is system capable of supporting monitor assignment? "monitor assignment" has not been used so far, was this intended to be "counter assignment"?
Missed to respond to this comment. Yes. Sure. Will correct it.
Signature · 2 lines
Thanks Babu Moger
Babu Moger
reviewerPATCH v13 5/27 x86/resctrl: Add support to enable/disable AMD ABMC feature
In reply to Babu MogerAdd the functionality to enable/disable AMD ABMC feature.
AMD ABMC feature is enabled by setting enabled bit(0) in MSR L3_QOS_EXT_CFG. When the state of ABMC is changed, the MSR needs to be updated on all the logical processors in the QOS Domain.
Hardware counters will reset when ABMC state is changed.
The ABMC feature details are documented in APM listed below [1].
[1] AMD64 Architecture Programmer's Manual Volume 2: System Programming Publication # 24593 Revision 3.41 section 19.3.3.3 Assignable Bandwidth Monitoring (ABMC).
---
v13: Resolved minor conflicts with recent FS/ARCH restructure.
v12: Clarified the comment on _resctrl_abmc_enable().
Added the code to reset arch state in _resctrl_abmc_enable(). Resolved the conflicts with latest merge.
v11: Moved the monitoring related calls to monitor.c file.
Moved the changes from include/linux/resctrl.h to arch/x86/kernel/cpu/resctrl/internal.h. Removed the Reviewed-by tag as patch changed. Actual code did not change.
v10: No changes.
v9: Re-ordered the MSR and added Reviewed-by tag.
v8: Commit message update and moved around the comments about L3_QOS_EXT_CFG
to _resctrl_abmc_enable.
v7: Renamed the function
resctrl_arch_get_abmc_enabled() to resctrl_arch_mbm_cntr_assign_enabled().
Merged resctrl_arch_mbm_cntr_assign_disable, resctrl_arch_mbm_cntr_assign_disable and renamed to resctrl_arch_mbm_cntr_assign_set().
Moved the function definition to linux/resctrl.h.
Passed the struct rdt_resource to these functions.
Removed resctrl_arch_reset_rmid_all() from arch code. This will be done from the caller.
v6: Renamed abmc_enabled to mbm_cntr_assign_enabled.
Used msr_set_bit and msr_clear_bit for msr updates.
Renamed resctrl_arch_abmc_enable() to resctrl_arch_mbm_cntr_assign_enable(). Renamed resctrl_arch_abmc_disable() to resctrl_arch_mbm_cntr_assign_disable(). Made _resctrl_abmc_enable to return void.
v5: Renamed resctrl_abmc_enable to resctrl_arch_abmc_enable.
Renamed resctrl_abmc_disable to resctrl_arch_abmc_disable. Introduced resctrl_arch_get_abmc_enabled to get abmc state from non-arch code.
Renamed resctrl_abmc_set_all to _resctrl_abmc_enable(). Modified commit log to make it clear about AMD ABMC feature.
v3: No changes.
v2: Few text changes in commit message.
--- arch/x86/include/asm/msr-index.h | 1 + arch/x86/kernel/cpu/resctrl/internal.h | 5 +++ arch/x86/kernel/cpu/resctrl/monitor.c | 43 ++++++++++++++++++++++++++ include/linux/resctrl.h | 3 ++ 4 files changed, 52 insertions(+)
arch/x86/include/asm/msr-index.h +1 -0
@@ -1203,6 +1203,7 @@/* - AMD: */ #define MSR_IA32_MBA_BW_BASE 0xc0000200#define MSR_IA32_SMBA_BW_BASE 0xc0000280 +#define MSR_IA32_L3_QOS_EXT_CFG 0xc00003ff#define MSR_IA32_EVT_CFG_BASE 0xc0000400/* AMD-V MSRs */
arch/x86/kernel/cpu/resctrl/internal.h +5 -0
@@ -37,6 +37,9 @@ struct arch_mbm_state {u64 prev_msr; }; +/* Setting bit 0 in L3_QOS_EXT_CFG enables the ABMC feature. */ +#define ABMC_ENABLE_BIT 0 +/** * struct rdt_hw_ctrl_domain - Arch private attributes of a set of CPUs that share * a resource for a control function
@@ -102,6 +105,7 @@ struct msr_param {* @mon_scale: cqm counter * mon_scale = occupancy in bytes * @mbm_width: Monitor width, to detect and correct for overflow. * @cdp_enabled: CDP state of this resource +* @mbm_cntr_assign_enabled: ABMC feature is enabled * * Members of this structure are either private to the architecture * e.g. mbm_width, or accessed via helpers that provide abstraction. e.g.
@@ -115,6 +119,7 @@ struct rdt_hw_resource {unsigned int mon_scale;unsigned int mbm_width;bool cdp_enabled; +bool mbm_cntr_assign_enabled; };static inline struct rdt_hw_resource *resctrl_to_arch_res (struct rdt_resource *r)
arch/x86/kernel/cpu/resctrl/monitor.c +43 -0
46 lines not on this page · raw message
include/linux/resctrl.h +3 -0
@@ -428,6 +428,9 @@ static inline u32 resctrl_get_config_index(u32 closid,bool resctrl_arch_get_cdp_enabled (enum resctrl_res_level l);int resctrl_arch_set_cdp_enabled (enum resctrl_res_level l,bool enable); +bool resctrl_arch_mbm_cntr_assign_enabled (struct rdt_resource *r); +int resctrl_arch_mbm_cntr_assign_set (struct rdt_resource *r,bool enable); +/* * Update the ctrl_val and apply this config right now. * Must be called on one of the domain's CPUs.
Signature · 1 line
2.34.1
Reinette Chatre
maintainerRe: [PATCH v13 05/27] x86/resctrl: Add support to enable/disable AMD ABMC feature
Hi Babu,
Quoted 16 lines · Babu Moger
Add the functionality to enable/disable AMD ABMC feature.
AMD ABMC feature is enabled by setting enabled bit(0) in MSR
L3_QOS_EXT_CFG. When the state of ABMC is changed, the MSR needs to be updated on all the logical processors in the QOS Domain.Hardware counters will reset when ABMC state is changed.
The ABMC feature details are documented in APM listed below [1].
[1] AMD64 Architecture Programmer's Manual Volume 2: System Programming Publication # 24593 Revision 3.41 section 19.3.3.3 Assignable Bandwidth Monitoring (ABMC).---
...
Quoted 66 lines
--- arch/x86/include/asm/msr-index.h | 1 + arch/x86/kernel/cpu/resctrl/internal.h | 5 +++ arch/x86/kernel/cpu/resctrl/monitor.c | 43 ++++++++++++++++++++++++++ include/linux/resctrl.h | 3 ++ 4 files changed, 52 insertions(+)arch/x86/include/asm/msr-index.h +1
-0@@ -1203,6 +1203,7 @@/* - AMD: */ #define MSR_IA32_MBA_BW_BASE 0xc0000200#define MSR_IA32_SMBA_BW_BASE 0xc0000280 +#define MSR_IA32_L3_QOS_EXT_CFG 0xc00003ff#define MSR_IA32_EVT_CFG_BASE 0xc0000400/* AMD-V MSRs */ arch/x86/kernel/cpu/resctrl/internal.h +5
-0@@ -37,6 +37,9 @@ struct arch_mbm_state {u64 prev_msr; }; +/* Setting bit 0 in L3_QOS_EXT_CFG enables the ABMC feature. */ +#define ABMC_ENABLE_BIT 0 +/** * struct rdt_hw_ctrl_domain - Arch private attributes of a set of CPUs that share * a resource for a control function @@ -102,6 +105,7 @@ struct msr_param {* @mon_scale: cqm counter * mon_scale = occupancy in bytes * @mbm_width: Monitor width, to detect and correct for overflow. * @cdp_enabled: CDP state of this resource +* @mbm_cntr_assign_enabled: ABMC feature is enabled * * Members of this structure are either private to the architecture * e.g. mbm_width, or accessed via helpers that provide abstraction. e.g. @@ -115,6 +119,7 @@ struct rdt_hw_resource {unsigned int mon_scale;unsigned int mbm_width;bool cdp_enabled; +bool mbm_cntr_assign_enabled; };static inline struct rdt_hw_resource *resctrl_to_arch_res (struct rdt_resource *r)arch/x86/kernel/cpu/resctrl/monitor.c +9
-0@@ -405,3 +405,46 @@ void __init intel_rdt_mbm_apply_quirk(void) mbm_cf_rmidthreshold = mbm_cf_table[cf_index].rmidthreshold; mbm_cf = mbm_cf_table[cf_index].cf; } + +static void resctrl_abmc_set_one_amd (void *arg) +{ +bool *enable = arg; + +if (*enable) +msr_set_bit (MSR_IA32_L3_QOS_EXT_CFG ,ABMC_ENABLE_BIT ); +else +msr_clear_bit (MSR_IA32_L3_QOS_EXT_CFG ,ABMC_ENABLE_BIT );
arch/x86/kernel/cpu/resctrl/monitor.c void __init intel_rdt_mbm_apply_quirk(void) · line 426
Quoted 10 lines
arch/x86/kernel/cpu/resctrl/monitor.c +10
-0+} + +/* +* ABMC enable/disable requires update of L3_QOS_EXT_CFG MSR on all the CPUs +* associated with all monitor domains. +*/ +static void _resctrl_abmc_enable (struct rdt_resource *r,bool enable) +{ +struct rdt_mon_domain *d; +
It remains a challenge to consider these building blocks without insight into how/when they will be used. To help out, please add guardrails to help with review. For example, this could benefit from a:
lockdep_assert_cpus_held();
Quoted 11 lines
arch/x86/kernel/cpu/resctrl/monitor.c +11
-0+list_for_each_entry (d, &r->mon_domains, hdr.list) { +on_each_cpu_mask (&d->hdr.cpu_mask, + resctrl_abmc_set_one_amd, &enable,1 ); +resctrl_arch_reset_rmid_all (r, d); + } +} + +int resctrl_arch_mbm_cntr_assign_set (struct rdt_resource *r,bool enable) +{ +struct rdt_hw_resource *hw_res =resctrl_to_arch_res (r); +
arch/x86/kernel/cpu/resctrl/monitor.c void __init intel_rdt_mbm_apply_quirk(void) · line 447
Quoted 10 lines
arch/x86/kernel/cpu/resctrl/monitor.c +10
-0+if (r->mon.mbm_cntr_assignable && + hw_res->mbm_cntr_assign_enabled != enable) { +_resctrl_abmc_enable (r, enable); + hw_res->mbm_cntr_assign_enabled = enable; + } + +return 0 ; +} + +inline bool resctrl_arch_mbm_cntr_assign_enabled (struct rdt_resource *r)
This "inline" in the .c file is unexpected. Why is this needed?
Quoted 8 lines
arch/x86/kernel/cpu/resctrl/monitor.c +3
-0+{ +return resctrl_to_arch_res (r)->mbm_cntr_assign_enabled; +}include/linux/resctrl.h +0
-0
include/linux/resctrl.h static inline u32 resctrl_get_config_index(u32 closid, · line 436
Quoted 9 lines
include/linux/resctrl.h +3
-0@@ -428,6 +428,9 @@ static inline u32 resctrl_get_config_index(u32 closid,bool resctrl_arch_get_cdp_enabled (enum resctrl_res_level l);int resctrl_arch_set_cdp_enabled (enum resctrl_res_level l,bool enable); +bool resctrl_arch_mbm_cntr_assign_enabled (struct rdt_resource *r); +int resctrl_arch_mbm_cntr_assign_set (struct rdt_resource *r,bool enable); +/* * Update the ctrl_val and apply this config right now. * Must be called on one of the domain's CPUs.
Reinette
Moger, Babu
reviewerRe: [PATCH v13 05/27] x86/resctrl: Add support to enable/disable AMD ABMC feature
Hi Reinette,
Quoted 103 lines · Reinette Chatre
Hi Babu,
Quoted 16 lines · Babu Moger
Add the functionality to enable/disable AMD ABMC feature.
AMD ABMC feature is enabled by setting enabled bit(0) in MSR
L3_QOS_EXT_CFG. When the state of ABMC is changed, the MSR needs to be updated on all the logical processors in the QOS Domain.Hardware counters will reset when ABMC state is changed.
The ABMC feature details are documented in APM listed below [1].
[1] AMD64 Architecture Programmer's Manual Volume 2: System Programming Publication # 24593 Revision 3.41 section 19.3.3.3 Assignable Bandwidth Monitoring (ABMC).---
...
Quoted 66 lines
--- arch/x86/include/asm/msr-index.h | 1 + arch/x86/kernel/cpu/resctrl/internal.h | 5 +++ arch/x86/kernel/cpu/resctrl/monitor.c | 43 ++++++++++++++++++++++++++ include/linux/resctrl.h | 3 ++ 4 files changed, 52 insertions(+)arch/x86/include/asm/msr-index.h +1
-0@@ -1203,6 +1203,7 @@/* - AMD: */ #define MSR_IA32_MBA_BW_BASE 0xc0000200#define MSR_IA32_SMBA_BW_BASE 0xc0000280 +#define MSR_IA32_L3_QOS_EXT_CFG 0xc00003ff#define MSR_IA32_EVT_CFG_BASE 0xc0000400/* AMD-V MSRs */ arch/x86/kernel/cpu/resctrl/internal.h +5
-0@@ -37,6 +37,9 @@ struct arch_mbm_state {u64 prev_msr; }; +/* Setting bit 0 in L3_QOS_EXT_CFG enables the ABMC feature. */ +#define ABMC_ENABLE_BIT 0 +/** * struct rdt_hw_ctrl_domain - Arch private attributes of a set of CPUs that share * a resource for a control function @@ -102,6 +105,7 @@ struct msr_param {* @mon_scale: cqm counter * mon_scale = occupancy in bytes * @mbm_width: Monitor width, to detect and correct for overflow. * @cdp_enabled: CDP state of this resource +* @mbm_cntr_assign_enabled: ABMC feature is enabled * * Members of this structure are either private to the architecture * e.g. mbm_width, or accessed via helpers that provide abstraction. e.g. @@ -115,6 +119,7 @@ struct rdt_hw_resource {unsigned int mon_scale;unsigned int mbm_width;bool cdp_enabled; +bool mbm_cntr_assign_enabled; };static inline struct rdt_hw_resource *resctrl_to_arch_res (struct rdt_resource *r)arch/x86/kernel/cpu/resctrl/monitor.c +9
-0@@ -405,3 +405,46 @@ void __init intel_rdt_mbm_apply_quirk(void) mbm_cf_rmidthreshold = mbm_cf_table[cf_index].rmidthreshold; mbm_cf = mbm_cf_table[cf_index].cf; } + +static void resctrl_abmc_set_one_amd (void *arg) +{ +bool *enable = arg; + +if (*enable) +msr_set_bit (MSR_IA32_L3_QOS_EXT_CFG ,ABMC_ENABLE_BIT ); +else +msr_clear_bit (MSR_IA32_L3_QOS_EXT_CFG ,ABMC_ENABLE_BIT );arch/x86/kernel/cpu/resctrl/monitor.c void __init intel_rdt_mbm_apply_quirk(void) · line 426
Quoted 10 lines
arch/x86/kernel/cpu/resctrl/monitor.c +10
-0+} + +/* +* ABMC enable/disable requires update of L3_QOS_EXT_CFG MSR on all the CPUs +* associated with all monitor domains. +*/ +static void _resctrl_abmc_enable (struct rdt_resource *r,bool enable) +{ +struct rdt_mon_domain *d; +It remains a challenge to consider these building blocks without insight into how/when they will be used. To help out, please add guardrails to help with review. For example, this could benefit from a:
Quoted 1 line
lockdep_assert_cpus_held();
Yes. Sure.
Quoted 16 lines
Quoted 16 lines
arch/x86/kernel/cpu/resctrl/monitor.c +15
-0+list_for_each_entry (d, &r->mon_domains, hdr.list) { +on_each_cpu_mask (&d->hdr.cpu_mask, + resctrl_abmc_set_one_amd, &enable,1 ); +resctrl_arch_reset_rmid_all (r, d); + } +} + +int resctrl_arch_mbm_cntr_assign_set (struct rdt_resource *r,bool enable) +{ +struct rdt_hw_resource *hw_res =resctrl_to_arch_res (r); + +if (r->mon.mbm_cntr_assignable && + hw_res->mbm_cntr_assign_enabled != enable) { +_resctrl_abmc_enable (r, enable); + hw_res->mbm_cntr_assign_enabled = enable;
Quoted 8 lines
arch/x86/kernel/cpu/resctrl/monitor.c void __init intel_rdt_mbm_apply_quirk(void) · line 447
Quoted 6 lines
arch/x86/kernel/cpu/resctrl/monitor.c +6
-0+ } + +return 0 ; +} + +inline bool resctrl_arch_mbm_cntr_assign_enabled (struct rdt_resource *r)This "inline" in the .c file is unexpected. Why is this needed?
Not required. No specific reason. Will remove it.
Quoted 22 lines
Quoted 9 lines
arch/x86/kernel/cpu/resctrl/monitor.c +3
-0+{ +return resctrl_to_arch_res (r)->mbm_cntr_assign_enabled; +}include/linux/resctrl.h +0
-0include/linux/resctrl.h static inline u32 resctrl_get_config_index(u32 closid, · line 436
Quoted 9 lines
include/linux/resctrl.h +3
-0@@ -428,6 +428,9 @@ static inline u32 resctrl_get_config_index(u32 closid,bool resctrl_arch_get_cdp_enabled (enum resctrl_res_level l);int resctrl_arch_set_cdp_enabled (enum resctrl_res_level l,bool enable); +bool resctrl_arch_mbm_cntr_assign_enabled (struct rdt_resource *r); +int resctrl_arch_mbm_cntr_assign_set (struct rdt_resource *r,bool enable); +/* * Update the ctrl_val and apply this config right now. * Must be called on one of the domain's CPUs. Reinette
Signature · 2 lines
Thanks Babu Moger
Babu Moger
reviewerPATCH v13 6/27 x86/resctrl: Introduce the interface to display monitor mode
In reply to Babu MogerIntroduce the resctrl file "mbm_assign_mode" to list monitor modes supported.
The "mbm_cntr_assign" mode provides the option to assign a counter to an RMID, event pair and monitor the bandwidth as long as it is assigned.
On AMD systems "mbm_cntr_assign" mode is backed by the ABMC (Assignable Bandwidth Monitoring Counters) hardware feature and is enabled by default.
The "default" mode is the existing monitoring mode that works without the explicit counter assignment, instead relying on dynamic counter assignment by hardware that may result in hardware not dedicating a counter resulting in monitoring data reads returning "Unavailable".
Provide an interface to display the monitor mode on the system.
$ cat /sys/fs/resctrl/info/L3_MON/mbm_assign_mode [mbm_cntr_assign] default
Add IS_ENABLED(CONFIG_RESCTRL_ASSIGN_FIXED) check to support Arm64.
On x86, CONFIG_RESCTRL_ASSIGN_FIXED is not defined. On Arm64, it will be defined when the "mbm_cntr_assign" mode is supported.
Add IS_ENABLED(CONFIG_RESCTRL_ASSIGN_FIXED) check early to ensure the user interface remains compatible with upcoming Arm64 support. IS_ENABLED() safely evaluates to 0 when the configuration is not defined.
As a result, for MPAM, the display would be either:
[default]
or
[mbm_cntr_assign]
---
v13: Updated the commit log with motivation for adding CONFIG_RESCTRL_ASSIGN_FIXED.
Added fflag RFTYPE_RES_CACHE for mbm_assign_mode file.
Updated user doc. Removed the references to "mbm_assign_control". Resolved the conflicts with latest FS/ARCH code restructure.
v12: Minor text update in change log and user documentation.
Added the check CONFIG_RESCTRL_ASSIGN_FIXED to take care of arm platforms. This will be defined only in arm and not in x86.
v11: Renamed rdtgroup_mbm_assign_mode_show() to resctrl_mbm_assign_mode_show().
Removed few texts in resctrl.rst about AMD specific information. Updated few texts.
v10: Added few more text to user documentation clarify on the default mode.
v9: Updated user documentation based on comments.
v8: Commit message update.
v7: Updated the descriptions/commit log in resctrl.rst to generic text.
Thanks to James and Reinette.
Rename mbm_mode to mbm_assign_mode.
Introduced mutex lock in rdtgroup_mbm_mode_show().
v6: Added documentation for mbm_cntr_assign and legacy mode.
Moved mbm_mode fflags initialization to static initialization.
v5: Changed interface name to mbm_mode.
It will be always available even if ABMC feature is not supported. Added description in resctrl.rst about ABMC mode.
Fixed display abmc and legacy consistantly.
v4: Fixed the checks for legacy and abmc mode. Default it ABMC.
v3: New patch to display ABMC capability.
--- Documentation/filesystems/resctrl.rst | 27 +++++++++++++++++++ fs/resctrl/rdtgroup.c | 37 +++++++++++++++++++++++++++ 2 files changed, 64 insertions(+)
Documentation/filesystems/resctrl.rst +27 -0
@@ -257,6 +257,33 @@ with the following files: # cat /sys/fs/resctrl/info/L3_MON/mbm_local_bytes_config 0=0x30;1=0x30;3=0x15;4=0x15 +"mbm_assign_mode": + Reports the list of monitoring modes supported. The enclosed brackets + indicate which mode is enabled. + :: + + # cat /sys/fs/resctrl/info/L3_MON/mbm_assign_mode + [mbm_cntr_assign] + default + + "mbm_cntr_assign": + + In mbm_cntr_assign mode, a monitoring event can only accumulate data + while it is backed by a hardware counter. Use "mbm_L3_assignments" found + in each CTRL_MON and MON group to specify which of the events should have + a counter assigned. The number of counters available is described in the + "num_mbm_cntrs" file. Changing the mode may cause all counters on the + resource to reset. + + "default": + + In default mode, resctrl assumes there is a hardware counter for each + event within every CTRL_MON and MON group. On AMD platforms, it is + recommended to use the mbm_cntr_assign mode, if supported, to prevent + reset of MBM events between reads resulting from hardware re-allocating + counters. This can result in misleading values or display "Unavailable" + if no counter is assigned to the event. + "max_threshold_occupancy": Read/write file provides the largest value (in bytes) at which a previously used LLC_occupancy
fs/resctrl/rdtgroup.c +37 -0
@@ -1801,6 +1801,36 @@ static ssize_t mbm_local_bytes_config_write(struct kernfs_open_file *of,return ret ?: nbytes; } +static int resctrl_mbm_assign_mode_show (struct kernfs_open_file *of, +struct seq_file *s,void *v) +{ +struct rdt_resource *r =rdt_kn_parent_priv (of->kn); +bool enabled; + +mutex_lock (&rdtgroup_mutex); + enabled =resctrl_arch_mbm_cntr_assign_enabled (r); + +if (r->mon.mbm_cntr_assignable) { +if (enabled) +seq_puts (s,"[mbm_cntr_assign]\n" ); +else +seq_puts (s,"[default]\n" ); + +if (!IS_ENABLED (CONFIG_RESCTRL_ASSIGN_FIXED )) { +if (enabled) +seq_puts (s,"default\n" ); +else +seq_puts (s,"mbm_cntr_assign\n" ); + } + }else { +seq_puts (s,"[default]\n" ); + } + +mutex_unlock (&rdtgroup_mutex); + +return 0 ; +} +/* rdtgroup information files for one cache resource. */ static struct rftype res_common_files[] = { {
@@ -1913,6 +1943,13 @@ static struct rftype res_common_files[] = { .seq_show = mbm_local_bytes_config_show, .write = mbm_local_bytes_config_write, }, + { + .name = "mbm_assign_mode", + .mode = 0444, + .kf_ops = &rdtgroup_kf_single_ops, + .seq_show = resctrl_mbm_assign_mode_show, + .fflags = RFTYPE_MON_INFO | RFTYPE_RES_CACHE, + }, { .name = "cpus", .mode = 0644,
Signature · 1 line
2.34.1
Reinette Chatre
maintainerRe: [PATCH v13 06/27] x86/resctrl: Introduce the interface to display monitor mode
Hi Babu,
On 5/15/25 3:51 PM, Babu Moger wrote:
No comments on changelog since I expect it to be reworked based on
https://lore.kernel.org/lkml/[email protected]/
Documentation/filesystems/resctrl.rst with the following files: · line 261
Quoted 8 lines
Documentation/filesystems/resctrl.rst +2
-0@@ -257,6 +257,33 @@ with the following files: # cat /sys/fs/resctrl/info/L3_MON/mbm_local_bytes_config 0=0x30;1=0x30;3=0x15;4=0x15 +"mbm_assign_mode": + Reports the list of monitoring modes supported. The enclosed brackets
Please try to avoid unnecessary words. For example,
"Reports the list of monitoring modes supported." -> "The supported monitoring modes."
Reinette
Moger, Babu
reviewerRe: [PATCH v13 06/27] x86/resctrl: Introduce the interface to display monitor mode
Hi Reinette,
Quoted 6 lines · Reinette Chatre
Hi Babu,
On 5/15/25 3:51 PM, Babu Moger wrote:
No comments on changelog since I expect it to be reworked based on
https://lore.kernel.org/lkml/[email protected]/
Yes. Sure.
Quoted 7 lines
Quoted 7 lines
Documentation/filesystems/resctrl.rst +0
-0@@ -257,6 +257,33 @@ with the following files: # cat /sys/fs/resctrl/info/L3_MON/mbm_local_bytes_config 0=0x30;1=0x30;3=0x15;4=0x15
Quoted 5 lines
Documentation/filesystems/resctrl.rst with the following files: · line 261
Quoted 2 lines
Documentation/filesystems/resctrl.rst +2
-0+"mbm_assign_mode": + Reports the list of monitoring modes supported. The enclosed bracketsPlease try to avoid unnecessary words. For example,
"Reports the list of monitoring modes supported." -> "The supported monitoring modes."
Sure.
Signature · 2 lines
Thanks Babu Moger
Babu Moger
reviewerPATCH v13 7/27 x86/resctrl: Introduce interface to display number of monitoring counters
In reply to Babu MogerThe mbm_cntr_assign mode provides an option to the user to assign a counter to an RMID, event pair and monitor the bandwidth as long as the counter is assigned. Number of assignments depend on number of monitoring counters available.
Create 'num_mbm_cntrs' resctrl file that displays the number of monitoring counters supported in each domain. 'num_mbm_cntrs' is only visible to user space when the system supports mbm_cntr_assign mode.
---
v13: Updated the changelog.
Added fflags RFTYPE_RES_CACHE to the file num_mbm_cntrs.
Replaced seq_puts from seq_putc where applicable.
Resolved conflicts caused by the recent FS/ARCH code restructure. The files monitor.c/rdtgroup.c have been split between FS and ARCH directories.
v12: Changed the code to display the max supported monitoring counters in
each domain. Also updated the documentation.
Resolved the conflict with the latest code.
v11: Renamed rdtgroup_num_mbm_cntrs_show() to resctrl_num_mbm_cntrs_show().
Few monor text updates.
v10: No changes.
v9: Updated user document based on the comments.
Will add a new file available_mbm_cntrs later in the series.
v8: Commit message update and documentation update.
v7: Minor commit log text changes.
v6: No changes.
v5: Changed the display name from num_cntrs to num_mbm_cntrs.
Updated the commit message.
Moved the patch after mbm_mode is introduced.
v4: Changed the counter name to num_cntrs. And few text changes.
v3: Changed the field name to mbm_assign_cntrs.
v2: Changed the field name to mbm_assignable_counters from abmc_counter.
--- Documentation/filesystems/resctrl.rst | 11 ++++++++++ fs/resctrl/monitor.c | 4 ++++ fs/resctrl/rdtgroup.c | 30 +++++++++++++++++++++++++++ 3 files changed, 45 insertions(+)
Documentation/filesystems/resctrl.rst +11 -0
@@ -284,6 +284,17 @@ with the following files: counters. This can result in misleading values or display "Unavailable" if no counter is assigned to the event. +"num_mbm_cntrs": + The maximum number of monitoring counters (total of available and assigned + counters) in each domain when the system supports mbm_cntr_assign mode. + + For example, on a system with maximum of 32 memory bandwidth monitoring + counters in each of its L3 domains: + :: + + # cat /sys/fs/resctrl/info/L3_MON/num_mbm_cntrs + 0=32;1=32 + "max_threshold_occupancy": Read/write file provides the largest value (in bytes) at which a previously used LLC_occupancy
fs/resctrl/monitor.c +4 -0
@@ -918,6 +918,10 @@ int resctrl_mon_resource_init(void)else if (resctrl_arch_is_mbm_total_enabled ()) mba_mbps_default_event =QOS_L3_MBM_TOTAL_EVENT_ID ; +if (r->mon.mbm_cntr_assignable) +resctrl_file_fflags_init ("num_mbm_cntrs" , +RFTYPE_MON_INFO |RFTYPE_RES_CACHE ); +return 0 ; }
fs/resctrl/rdtgroup.c +30 -0
@@ -1831,6 +1831,30 @@ static int resctrl_mbm_assign_mode_show(struct kernfs_open_file *of,return 0 ; } +static int resctrl_num_mbm_cntrs_show (struct kernfs_open_file *of, +struct seq_file *s,void *v) +{ +struct rdt_resource *r =rdt_kn_parent_priv (of->kn); +struct rdt_mon_domain *dom; +bool sep = false; + +cpus_read_lock (); +mutex_lock (&rdtgroup_mutex); + +list_for_each_entry (dom, &r->mon_domains, hdr.list) { +if (sep) +seq_putc (s,';' ); + +seq_printf (s,"%d=%d" , dom->hdr.id, r->mon.num_mbm_cntrs); + sep = true; + } +seq_putc (s,'\n' ); + +mutex_unlock (&rdtgroup_mutex); +cpus_read_unlock (); +return 0 ; +} +/* rdtgroup information files for one cache resource. */ static struct rftype res_common_files[] = { {
@@ -1868,6 +1892,12 @@ static struct rftype res_common_files[] = { .seq_show = rdt_default_ctrl_show, .fflags = RFTYPE_CTRL_INFO | RFTYPE_RES_CACHE, }, + { + .name = "num_mbm_cntrs", + .mode = 0444, + .kf_ops = &rdtgroup_kf_single_ops, + .seq_show = resctrl_num_mbm_cntrs_show, + }, { .name = "min_cbm_bits", .mode = 0444,
Signature · 1 line
2.34.1
Babu Moger
reviewerPATCH v13 8/27 x86/resctrl: Introduce mbm_cntr_cfg to track assignable counters at domain
In reply to Babu MogerIn mbm_cntr_assign mode hardware counters are assigned/unassigned to an MBM event of a monitor group. Hardware counters are assigned/unassigned at monitoring domain level.
Manage a monitoring domain's hardware counters using a per monitoring domain array of struct mbm_cntr_cfg that is indexed by the hardware counter ID. A hardware counter's configuration contains the MBM event ID and points to the monitoring group that it is assigned to, with a NULL pointer meaning that the hardware counter is available for assignment.
There is no direct way to determine which hardware counters are assigned to a particular monitoring group. Check every entry of every hardware counter configuration array in every monitoring domain to query which MBM events of a monitoring group is tracked by hardware. Such queries are acceptable because of a very small number of assignable counters (32 to 64).
---
v13: Resolved conflicts caused by the recent FS/ARCH code restructure.
The files monitor.c/rdtgroup.c have been split between FS and ARCH directories.
v12: Fixed the struct mbm_cntr_cfg code documentation.
Removed few strange charactors in changelog.
Added the counter range for better understanding. Moved the struct mbm_cntr_cfg definition to resctrl/internal.h as suggested by James.
v11: Refined the change log based on Reinette's feedback.
Fixed few style issues.
v10: Patch changed completely to handle the counters at domain level.
https://lore.kernel.org/lkml/CALPaoCj+zWq1vkHVbXYP0znJbe6Ke3PXPWjtri5AFgD9cQDCUg@mail.gmail.com/
Removed Reviewed-by tag.
Did not see the need to add cntr_id in mbm_state structure. Not used in the code.
v9: Added Reviewed-by tag. No other changes.
v8: Minor commit message changes.
v7: Added check mbm_cntr_assignable for allocating bitmap mbm_cntr_map
v6: New patch to add domain level assignment.
--- fs/resctrl/rdtgroup.c | 11 +++++++++++ include/linux/resctrl.h | 16 ++++++++++++++++ 2 files changed, 27 insertions(+)
fs/resctrl/rdtgroup.c +11 -0
@@ -4085,6 +4085,7 @@ static void rdtgroup_setup_default(void)static void domain_destroy_mon_state (struct rdt_mon_domain *d) { +kfree (d->cntr_cfg);bitmap_free (d->rmid_busy_llc);kfree (d->mbm_total);kfree (d->mbm_local);
@@ -4171,6 +4172,16 @@ static int domain_setup_mon_state(struct rdt_resource *r, struct rdt_mon_domainreturn -ENOMEM ; } } +if (resctrl_is_mbm_enabled () && r->mon.mbm_cntr_assignable) { + tsize =sizeof (*d->cntr_cfg); + d->cntr_cfg =kcalloc (r->mon.num_mbm_cntrs, tsize,GFP_KERNEL ); +if (!d->cntr_cfg) { +bitmap_free (d->rmid_busy_llc); +kfree (d->mbm_total); +kfree (d->mbm_local); +return -ENOMEM ; + } + }return 0 ; }
include/linux/resctrl.h +16 -0
@@ -156,6 +156,20 @@ struct rdt_ctrl_domain {u32 *mbps_val; }; +/** +* struct mbm_cntr_cfg - Assignable counter configuration +* @evtid: MBM event to which the counter is assigned. Only valid +* if @rdtgroup is not NULL. +* @evt_cfg: Event configuration value. +* @rdtgrp: resctrl group assigned to the counter. NULL if the +* counter is free. +*/ +struct mbm_cntr_cfg { +enum resctrl_event_id evtid; +u32 evt_cfg; +struct rdtgroup *rdtgrp; +}; +/** * struct rdt_mon_domain - group of CPUs sharing a resctrl monitor resource * @hdr: common header for different domain types
@@ -167,6 +181,7 @@ struct rdt_ctrl_domain {* @cqm_limbo: worker to periodically read CQM h/w counters * @mbm_work_cpu: worker CPU for MBM h/w counters * @cqm_work_cpu: worker CPU for CQM h/w counters +* @cntr_cfg: assignable counters configuration */ struct rdt_mon_domain {struct rdt_domain_hdr hdr;
@@ -178,6 +193,7 @@ struct rdt_mon_domain {struct delayed_work cqm_limbo;int mbm_work_cpu;int cqm_work_cpu; +struct mbm_cntr_cfg *cntr_cfg; };/**
Signature · 1 line
2.34.1
Reinette Chatre
maintainerRe: [PATCH v13 08/27] x86/resctrl: Introduce mbm_cntr_cfg to track assignable counters at domain
Hi Babu,
shortlog: "at domain" -> "per domain"?
Quoted 83 lines · Babu Moger
In
mbm_cntr_assignmode hardware counters are assigned/unassigned to an MBM event of a monitor group. Hardware counters are assigned/unassigned at monitoring domain level.Manage a monitoring domain's hardware counters using a per monitoring domain array of struct
mbm_cntr_cfgthat is indexed by the hardware counter ID. A hardware counter's configuration contains the MBM event ID and points to the monitoring group that it is assigned to, with a NULL pointer meaning that the hardware counter is available for assignment.There is no direct way to determine which hardware counters are assigned to a particular monitoring group. Check every entry of every hardware counter configuration array in every monitoring domain to query which MBM events of a monitoring group is tracked by hardware. Such queries are acceptable because of a very small number of assignable counters (32 to 64).
Suggested-by Peter Newman <[email protected]>Signed-off-by Babu Moger <[email protected]>---
v13: Resolved conflicts caused by the recent FS/ARCH code restructure.
The filesmonitor.c/rdtgroup.chave been split between FS and ARCH directories.v12: Fixed the struct
mbm_cntr_cfgcode documentation.
Removed few strange charactors in changelog.
Added the counter range for better understanding. Moved the structmbm_cntr_cfgdefinition toresctrl/internal.has suggested by James.v11: Refined the change log based on Reinette's feedback.
Fixed few style issues.v10: Patch changed completely to handle the counters at domain level.
https://lore.kernel.org/lkml/CALPaoCj+zWq1vkHVbXYP0znJbe6Ke3PXPWjtri5AFgD9cQDCUg@mail.gmail.com/
Removed Reviewed-by tag.
Did not see the need to addcntr_idinmbm_statestructure. Not used in the code.v9: Added Reviewed-by tag. No other changes.
v8: Minor commit message changes.
v7: Added check
mbm_cntr_assignablefor allocating bitmapmbm_cntr_mapv6: New patch to add domain level assignment.
--- fs/resctrl/rdtgroup.c | 11 +++++++++++ include/linux/resctrl.h | 16 ++++++++++++++++ 2 files changed, 27 insertions(+)fs/resctrl/rdtgroup.c +11
-0@@ -4085,6 +4085,7 @@ static void rdtgroup_setup_default(void)static void domain_destroy_mon_state (struct rdt_mon_domain *d) { +kfree (d->cntr_cfg);bitmap_free (d->rmid_busy_llc);kfree (d->mbm_total);kfree (d->mbm_local);@@ -4171,6 +4172,16 @@ static int domain_setup_mon_state(struct rdt_resource *r, struct rdt_mon_domainreturn -ENOMEM ; } } +if (resctrl_is_mbm_enabled () && r->mon.mbm_cntr_assignable) { + tsize =sizeof (*d->cntr_cfg); + d->cntr_cfg =kcalloc (r->mon.num_mbm_cntrs, tsize,GFP_KERNEL ); +if (!d->cntr_cfg) { +bitmap_free (d->rmid_busy_llc); +kfree (d->mbm_total); +kfree (d->mbm_local); +return -ENOMEM ; + } + }return 0 ; }include/linux/resctrl.h +0
-0
include/linux/resctrl.h struct rdt_ctrl_domain { · line 163
Quoted 8 lines
include/linux/resctrl.h +5
-0@@ -156,6 +156,20 @@ struct rdt_ctrl_domain {u32 *mbps_val; }; +/** +* struct mbm_cntr_cfg - Assignable counter configuration +* @evtid: MBM event to which the counter is assigned. Only valid +* if @rdtgroup is not NULL. +* @evt_cfg: Event configuration value.
@evt_cfg is not introduced in changelog nor defined here. Please add a snippet here on what @evt_cfg's values represent. This is important since this is exposed as resctrl fs API to architectures so all architectures need to use same values when interacting with resctrl.
include/linux/resctrl.h struct rdt_ctrl_domain { · line 170
Quoted 7 lines
include/linux/resctrl.h +7
-0+* @rdtgrp: resctrl group assigned to the counter. NULL if the +* counter is free. +*/ +struct mbm_cntr_cfg { +enum resctrl_event_id evtid; +u32 evt_cfg; +struct rdtgroup *rdtgrp;
Please align struct member names using TABs.
include/linux/resctrl.h struct rdt_ctrl_domain { · line 184
Quoted 10 lines
include/linux/resctrl.h +3
-0+}; +/** * struct rdt_mon_domain - group of CPUs sharing a resctrl monitor resource * @hdr: common header for different domain types @@ -167,6 +181,7 @@ struct rdt_ctrl_domain {* @cqm_limbo: worker to periodically read CQM h/w counters * @mbm_work_cpu: worker CPU for MBM h/w counters * @cqm_work_cpu: worker CPU for CQM h/w counters +* @cntr_cfg: assignable counters configuration
"array of assignable counters' configuration (indexed by counter ID)"
include/linux/resctrl.h struct rdt_mon_domain { · line 199
Quoted 11 lines
include/linux/resctrl.h +1
-0*/ struct rdt_mon_domain {struct rdt_domain_hdr hdr;@@ -178,6 +193,7 @@ struct rdt_mon_domain {struct delayed_work cqm_limbo;int mbm_work_cpu;int cqm_work_cpu; +struct mbm_cntr_cfg *cntr_cfg; };/**
Reinette
Re: [PATCH v13 08/27] x86/resctrl: Introduce mbm_cntr_cfg to track assignable counters at domain
Hi Reinette,
Quoted 4 lines · Reinette Chatre
Hi Babu,
shortlog: "at domain" -> "per domain"?
Sure.
Quoted 87 lines · Babu Moger
Quoted 86 lines
In
mbm_cntr_assignmode hardware counters are assigned/unassigned to an MBM event of a monitor group. Hardware counters are assigned/unassigned at monitoring domain level.Manage a monitoring domain's hardware counters using a per monitoring domain array of struct
mbm_cntr_cfgthat is indexed by the hardware counter ID. A hardware counter's configuration contains the MBM event ID and points to the monitoring group that it is assigned to, with a NULL pointer meaning that the hardware counter is available for assignment.There is no direct way to determine which hardware counters are assigned to a particular monitoring group. Check every entry of every hardware counter configuration array in every monitoring domain to query which MBM events of a monitoring group is tracked by hardware. Such queries are acceptable because of a very small number of assignable counters (32 to 64).
Suggested-by Peter Newman <[email protected]>Signed-off-by Babu Moger <[email protected]>---
v13: Resolved conflicts caused by the recent FS/ARCH code restructure.
The filesmonitor.c/rdtgroup.chave been split between FS and ARCH directories.v12: Fixed the struct
mbm_cntr_cfgcode documentation.
Removed few strange charactors in changelog.
Added the counter range for better understanding. Moved the structmbm_cntr_cfgdefinition toresctrl/internal.has suggested by James.v11: Refined the change log based on Reinette's feedback.
Fixed few style issues.v10: Patch changed completely to handle the counters at domain level.
https://lore.kernel.org/lkml/CALPaoCj+zWq1vkHVbXYP0znJbe6Ke3PXPWjtri5AFgD9cQDCUg@mail.gmail.com/
Removed Reviewed-by tag.
Did not see the need to addcntr_idinmbm_statestructure. Not used in the code.v9: Added Reviewed-by tag. No other changes.
v8: Minor commit message changes.
v7: Added check
mbm_cntr_assignablefor allocating bitmapmbm_cntr_mapv6: New patch to add domain level assignment.
--- fs/resctrl/rdtgroup.c | 11 +++++++++++ include/linux/resctrl.h | 16 ++++++++++++++++ 2 files changed, 27 insertions(+)fs/resctrl/rdtgroup.c +11
-0@@ -4085,6 +4085,7 @@ static void rdtgroup_setup_default(void)static void domain_destroy_mon_state (struct rdt_mon_domain *d) { +kfree (d->cntr_cfg);bitmap_free (d->rmid_busy_llc);kfree (d->mbm_total);kfree (d->mbm_local);@@ -4171,6 +4172,16 @@ static int domain_setup_mon_state(struct rdt_resource *r, struct rdt_mon_domainreturn -ENOMEM ; } } +if (resctrl_is_mbm_enabled () && r->mon.mbm_cntr_assignable) { + tsize =sizeof (*d->cntr_cfg); + d->cntr_cfg =kcalloc (r->mon.num_mbm_cntrs, tsize,GFP_KERNEL ); +if (!d->cntr_cfg) { +bitmap_free (d->rmid_busy_llc); +kfree (d->mbm_total); +kfree (d->mbm_local); +return -ENOMEM ; + } + }return 0 ; }include/linux/resctrl.h +0
-0@@ -156,6 +156,20 @@ struct rdt_ctrl_domain {u32 *mbps_val; };
Quoted 10 lines
include/linux/resctrl.h struct rdt_ctrl_domain { · line 163
Quoted 5 lines
include/linux/resctrl.h +5
-0+/** +* struct mbm_cntr_cfg - Assignable counter configuration +* @evtid: MBM event to which the counter is assigned. Only valid +* if @rdtgroup is not NULL. +* @evt_cfg: Event configuration value. @evt_cfg is not introduced in changelog nor defined here. Please add a snippet here on what @evt_cfg's values represent. This is important since this is exposed as resctrl fs API to architectures so all architectures need to use same values when interacting with resctrl.
Sure.
@evt_cfg: A value that represents memory transactions (e.g., reads, writes, etc.).
Quoted 10 lines
include/linux/resctrl.h struct rdt_ctrl_domain { · line 170
Quoted 8 lines
include/linux/resctrl.h +7
-0+* @rdtgrp: resctrl group assigned to the counter. NULL if the +* counter is free. +*/ +struct mbm_cntr_cfg { +enum resctrl_event_id evtid; +u32 evt_cfg; +struct rdtgroup *rdtgrp;Please align struct member names using TABs.
Sure.
Quoted 13 lines
include/linux/resctrl.h struct rdt_ctrl_domain { · line 184
Quoted 11 lines
include/linux/resctrl.h +3
-0+}; +/** * struct rdt_mon_domain - group of CPUs sharing a resctrl monitor resource * @hdr: common header for different domain types @@ -167,6 +181,7 @@ struct rdt_ctrl_domain {* @cqm_limbo: worker to periodically read CQM h/w counters * @mbm_work_cpu: worker CPU for MBM h/w counters * @cqm_work_cpu: worker CPU for CQM h/w counters +* @cntr_cfg: assignable counters configuration "array of assignable counters' configuration (indexed by counter ID)"
Sure.
Quoted 11 lines
Quoted 11 lines
include/linux/resctrl.h +1
-0*/ struct rdt_mon_domain {struct rdt_domain_hdr hdr;@@ -178,6 +193,7 @@ struct rdt_mon_domain {struct delayed_work cqm_limbo;int mbm_work_cpu;int cqm_work_cpu; +struct mbm_cntr_cfg *cntr_cfg; };
Quoted 4 lines
include/linux/resctrl.h struct rdt_mon_domain { · line 199
Quoted 1 line
include/linux/resctrl.h +0
-0/** Reinette
thanks
Babu
Reinette Chatre
maintainerRe: [PATCH v13 08/27] x86/resctrl: Introduce mbm_cntr_cfg to track assignable counters at domain
Hi Babu,
Quoted 21 lines · Babu
Quoted 11 lines · Reinette Chatre
Quoted 9 lines · Babu Moger
include/linux/resctrl.h +0
-0@@ -156,6 +156,20 @@ struct rdt_ctrl_domain { u32 *mbps_val; };+/**
+ * structmbm_cntr_cfg- Assignable counter configurationQuoted 8 lines
Quoted 3 lines
+ * @evtid: MBM event to which the counter is assigned. Only valid
+ * if @rdtgroup is not NULL.
+ * @evt_cfg: Event configuration value.@evt_cfg is not introduced in changelog nor defined here. Please add a snippet here on what @evt_cfg's values represent. This is important since this is exposed as resctrl fs API to architectures so all architectures need to use same values when interacting with resctrl.
Quoted 3 lines
Sure.
@evt_cfg: A value that represents memory transactions (e.g., reads, writes, etc.).
This still does not explain how an @evt_cfg value should be interpreted. For example, it could be something like below (please feel free to improve).
@evt_cfg: Event configuration created using the READS_TO_LOCAL_MEM, READS_TO_REMOTE_MEM, etc. bits
that represent the memory transactions being counted.
Reinette
Re: [PATCH v13 08/27] x86/resctrl: Introduce mbm_cntr_cfg to track assignable counters at domain
Hi Reinette,
Quoted 26 lines · Reinette Chatre
Hi Babu,
Quoted 23 lines · Babu
Quoted 11 lines · Reinette Chatre
Quoted 9 lines · Babu Moger
include/linux/resctrl.h +1
-0@@ -156,6 +156,20 @@ struct rdt_ctrl_domain { u32 *mbps_val; }; +/** +* struct mbm_cntr_cfg - Assignable counter configuration Quoted 8 lines
include/linux/resctrl.h struct rdt_ctrl_domain { · line 163
Quoted 3 lines
include/linux/resctrl.h +3
-0+ * @evtid: MBM event to which the counter is assigned. Only valid + * if @rdtgroup is not NULL. + * @evt_cfg: Event configuration value.@evt_cfg is not introduced in changelog nor defined here. Please add a snippet here on what @evt_cfg's values represent. This is important since this is exposed as resctrl fs API to architectures so all architectures need to use same values when interacting with resctrl.
Sure.
Quoted 8 lines
Quoted 1 line
@evt_cfg: A value that represents memory transactions (e.g., reads, writes, etc.).
This still does not explain how an @evt_cfg value should be interpreted. For example, it could be something like below (please feel free to improve).
@evt_cfg: Event configuration created using the
READS_TO_LOCAL_MEM,READS_TO_REMOTE_MEM, etc. bits
that represent the memory transactions being counted.
Looks good.
Thanks
Babu
Babu Moger
reviewerPATCH v13 9/27 x86/resctrl: Introduce interface to display number of free MBM counters
In reply to Babu MogerProvide the interface to display the number of monitoring counters available for assignment in each domain when mbm_cntr_assign mode is enabled.
---
v13: Resolved conflicts caused by the recent FS/ARCH code restructure.
The files monitor.c and rdtgroup.c file has now been split between the FS and ARCH directories.
v12: Minor change to change log.
Updated the documentation text with an example.
Replaced seq_puts(s, ";") with seq_putc(s, ';');
Added missing rdt_last_cmd_clear() in resctrl_available_mbm_cntrs_show().
v11: Rename rdtgroup_available_mbm_cntrs_show() to resctrl_available_mbm_cntrs_show().
Few minor text changes.
v10: Patch changed to handle the counters at domain level.
https://lore.kernel.org/lkml/CALPaoCj+zWq1vkHVbXYP0znJbe6Ke3PXPWjtri5AFgD9cQDCUg@mail.gmail.com/
So, display logic also changed now.
v9: New patch
--- Documentation/filesystems/resctrl.rst | 11 ++++++ fs/resctrl/monitor.c | 5 ++- fs/resctrl/rdtgroup.c | 48 +++++++++++++++++++++++++++ 3 files changed, 63 insertions(+), 1 deletion(-)
Documentation/filesystems/resctrl.rst +11 -0
@@ -295,6 +295,17 @@ with the following files: # cat /sys/fs/resctrl/info/L3_MON/num_mbm_cntrs 0=32;1=32 +"available_mbm_cntrs": + The number of monitoring counters available for assignment in each + domain when mbm_cntr_assign mode is enabled on the system. + + For example, on a system with 30 available [hardware] monitoring counters + in each of its L3 domains: + :: + + # cat /sys/fs/resctrl/info/L3_MON/available_mbm_cntrs + 0=30;1=30 + "max_threshold_occupancy": Read/write file provides the largest value (in bytes) at which a previously used LLC_occupancy
fs/resctrl/monitor.c +4 -1
@@ -918,9 +918,12 @@ int resctrl_mon_resource_init(void)else if (resctrl_arch_is_mbm_total_enabled ()) mba_mbps_default_event =QOS_L3_MBM_TOTAL_EVENT_ID ; -if (r->mon.mbm_cntr_assignable) +if (r->mon.mbm_cntr_assignable) {resctrl_file_fflags_init ("num_mbm_cntrs" ,RFTYPE_MON_INFO |RFTYPE_RES_CACHE ); +resctrl_file_fflags_init ("available_mbm_cntrs" , +RFTYPE_MON_INFO |RFTYPE_RES_CACHE ); + }return 0 ; }
fs/resctrl/rdtgroup.c +48 -0
60 lines not on this page · raw message
Signature · 1 line
2.34.1
Babu Moger
reviewerPATCH v13 10/27 x86/resctrl: Add data structures and definitions for ABMC assignment
In reply to Babu MogerThe ABMC feature provides an option to the user to assign a hardware counter to an RMID, event pair and monitor the bandwidth as long as the counter is assigned. The bandwidth events will be tracked by the hardware until the user changes the configuration. Each resctrl group can configure maximum two counters, one for total event and one for local event.
The ABMC feature implements an MSR L3_QOS_ABMC_CFG (C000_03FDh). ABMC counter assignment is done by setting the counter id, bandwidth source (RMID) and bandwidth configuration. Users will have the option to change the bandwidth configuration using resctrl interface which will be introduced later in the series.
Attempts to read or write the MSR when ABMC is not enabled will result in a #GP(0) exception.
Introduce the data structures and definitions for MSR L3_QOS_ABMC_CFG (0xC000_03FDh):
=========================================================================
Bits Mnemonic Description Access Reset
Type Value
=========================================================================
63 CfgEn Configuration Enable R/W 0
62 CtrEn Enable/disable counting R/W 0
61:53 – Reserved MBZ 0
52:48 CtrID Counter Identifier R/W 0
47 IsCOS BwSrc field is a CLOSID R/W 0
(not an RMID)
46:44 – Reserved MBZ 0
43:32 BwSrc Bandwidth Source R/W 0
(RMID or CLOSID)
31:0 BwType Bandwidth configuration R/W 0
to track for this counter
==========================================================================
The feature details are documented in the APM listed below [1].
[1] AMD64 Architecture Programmer's Manual Volume 2: System Programming Publication # 24593 Revision 3.41 section 19.3.3.3 Assignable Bandwidth Monitoring (ABMC).
---
v13: Removed the Reviewed-by tag as there is commit log change to remove
BMEC reference.
v12: No changes.
v11: No changes.
v10: No changes.
v9: Removed the references of L3_QOS_ABMC_DSC.
Text changes about configuration in kernel doc.
v8: Update the configuration notes in kernel_doc.
Few commit message update.
v7: Removed the reference of L3_QOS_ABMC_DSC as it is not used anymore.
Moved the configuration notes to kernel_doc.
Adjusted the tabs for l3_qos_abmc_cfg and checkpatch seems happy.
v6: Removed all the fs related changes.
Added note on CfgEn,CtrEn.
Removed the definitions which are not used.
Removed cntr_id initialization.
v5: Moved assignment flags here (path 10/19 of v4).
Added MON_CNTR_UNSET definition to initialize cntr_id's. More details in commit log.
Renamed few fields in l3_qos_abmc_cfg for readability.
v4: Added more descriptions.
Changed the name abmc_ctr_id to ctr_id.
Added L3_QOS_ABMC_DSC. Used for reading the configuration.
v3: No changes.
v2: No changes.
--- arch/x86/include/asm/msr-index.h | 1 + arch/x86/kernel/cpu/resctrl/internal.h | 35 ++++++++++++++++++++++++++ 2 files changed, 36 insertions(+)
arch/x86/include/asm/msr-index.h +1 -0
@@ -1203,6 +1203,7 @@/* - AMD: */ #define MSR_IA32_MBA_BW_BASE 0xc0000200#define MSR_IA32_SMBA_BW_BASE 0xc0000280 +#define MSR_IA32_L3_QOS_ABMC_CFG 0xc00003fd#define MSR_IA32_L3_QOS_EXT_CFG 0xc00003ff#define MSR_IA32_EVT_CFG_BASE 0xc0000400
arch/x86/kernel/cpu/resctrl/internal.h +35 -0
@@ -164,6 +164,41 @@ union cpuid_0x10_x_edx {unsigned int full; }; +/* +* ABMC counters are configured by writing to L3_QOS_ABMC_CFG. +* @bw_type : Bandwidth configuration (supported by BMEC) +* tracked by the @cntr_id. +* @bw_src : Bandwidth source (RMID or CLOSID). +* @reserved1 : Reserved. +* @is_clos : @bw_src field is a CLOSID (not an RMID). +* @cntr_id : Counter identifier. +* @reserved : Reserved. +* @cntr_en : Counting enable bit. +* @cfg_en : Configuration enable bit. +* +* Configuration and counting: +* Counter can be configured across multiple writes to MSR. Configuration +* is applied only when @cfg_en = 1. Counter @cntr_id is reset when the +* configuration is applied. +* @cfg_en = 1, @cntr_en = 0 : Apply @cntr_id configuration but do not +* count events. +* @cfg_en = 1, @cntr_en = 1 : Apply @cntr_id configuration and start +* counting events. +*/ +union l3_qos_abmc_cfg { +struct { +unsigned long bw_type :32 , + bw_src :12 , + reserved1:3 , + is_clos :1 , + cntr_id :5 , + reserved :9 , + cntr_en :1 , + cfg_en :1 ; + } split; +unsigned long full; +}; +void rdt_ctrl_update (void *arg);int rdt_get_mon_l3_config (struct rdt_resource *r);
Signature · 1 line
2.34.1
Reinette Chatre
maintainerRe: [PATCH v13 10/27] x86/resctrl: Add data structures and definitions for ABMC assignment
Hi Babu,
Quoted 5 lines · Babu Moger
The ABMC feature provides an option to the user to assign a hardware counter to an RMID, event pair and monitor the bandwidth as long as the counter is assigned. The bandwidth events will be tracked by the hardware until the user changes the configuration. Each resctrl group can configure maximum two counters, one for total event and one for local event.
(please update, above describes previous design)
Quoted 6 lines
The ABMC feature implements an MSR
L3_QOS_ABMC_CFG(C000_03FDh). ABMC counter assignment is done by setting the counter id, bandwidth source (RMID) and bandwidth configuration. Users will have the option to change the bandwidth configuration using resctrl interface which will be introduced later in the series.
"will be introduced later in the series" is similar to "in a subsequent patch" and should not be used in a changelog. Just describe what this patch does.
Quoted 96 lines
Attempts to read or write the MSR when ABMC is not enabled will result in a #GP(0) exception.
Introduce the data structures and definitions for MSR
L3_QOS_ABMC_CFG(0xC000_03FDh):
=========================================================================
Bits Mnemonic Description Access Reset
Type Value
=========================================================================
63 CfgEn Configuration Enable R/W 062 CtrEn Enable/disable counting R/W 0
61:53 – Reserved MBZ 0
52:48 CtrID Counter Identifier R/W 0
47 IsCOS BwSrc field is a CLOSID R/W 0
(not an RMID)46:44 – Reserved MBZ 0
43:32 BwSrc Bandwidth Source R/W 0
(RMID or CLOSID)31:0 BwType Bandwidth configuration R/W 0
to track for this counter
==========================================================================The feature details are documented in the APM listed below [1].
[1] AMD64 Architecture Programmer's Manual Volume 2: System Programming Publication # 24593 Revision 3.41 section 19.3.3.3 Assignable Bandwidth Monitoring (ABMC).---
v13: Removed the Reviewed-by tag as there is commit log change to remove
BMEC reference.v12: No changes.
v11: No changes.
v10: No changes.
v9: Removed the references of
L3_QOS_ABMC_DSC.
Text changes about configuration in kernel doc.v8: Update the configuration notes in
kernel_doc.
Few commit message update.v7: Removed the reference of
L3_QOS_ABMC_DSCas it is not used anymore.
Moved the configuration notes tokernel_doc.
Adjusted the tabs forl3_qos_abmc_cfgand checkpatch seems happy.v6: Removed all the fs related changes.
Added note on CfgEn,CtrEn.
Removed the definitions which are not used.
Removedcntr_idinitialization.v5: Moved assignment flags here (path 10/19 of v4).
AddedMON_CNTR_UNSETdefinition to initialize cntr_id's. More details in commit log.
Renamed few fields inl3_qos_abmc_cfgfor readability.v4: Added more descriptions.
Changed the nameabmc_ctr_idtoctr_id.
AddedL3_QOS_ABMC_DSC. Used for reading the configuration.v3: No changes.
v2: No changes.
--- arch/x86/include/asm/msr-index.h | 1 + arch/x86/kernel/cpu/resctrl/internal.h | 35 ++++++++++++++++++++++++++ 2 files changed, 36 insertions(+)arch/x86/include/asm/msr-index.h +1
-0@@ -1203,6 +1203,7 @@/* - AMD: */ #define MSR_IA32_MBA_BW_BASE 0xc0000200#define MSR_IA32_SMBA_BW_BASE 0xc0000280 +#define MSR_IA32_L3_QOS_ABMC_CFG 0xc00003fd#define MSR_IA32_L3_QOS_EXT_CFG 0xc00003ff#define MSR_IA32_EVT_CFG_BASE 0xc0000400arch/x86/kernel/cpu/resctrl/internal.h +0
-0
arch/x86/kernel/cpu/resctrl/internal.h union cpuid_0x10_x_edx { · line 170
Quoted 7 lines
arch/x86/kernel/cpu/resctrl/internal.h +4
-0@@ -164,6 +164,41 @@ union cpuid_0x10_x_edx {unsigned int full; }; +/* +* ABMC counters are configured by writing to L3_QOS_ABMC_CFG. +* @bw_type : Bandwidth configuration (supported by BMEC) +* tracked by the @cntr_id.
The "supported by BMEC" is unexpected with the new design that separated the two features.
Quoted 24 lines
arch/x86/kernel/cpu/resctrl/internal.h +24
-0+* @bw_src : Bandwidth source (RMID or CLOSID). +* @reserved1 : Reserved. +* @is_clos : @bw_src field is a CLOSID (not an RMID). +* @cntr_id : Counter identifier. +* @reserved : Reserved. +* @cntr_en : Counting enable bit. +* @cfg_en : Configuration enable bit. +* +* Configuration and counting: +* Counter can be configured across multiple writes to MSR. Configuration +* is applied only when @cfg_en = 1. Counter @cntr_id is reset when the +* configuration is applied. +* @cfg_en = 1, @cntr_en = 0 : Apply @cntr_id configuration but do not +* count events. +* @cfg_en = 1, @cntr_en = 1 : Apply @cntr_id configuration and start +* counting events. +*/ +union l3_qos_abmc_cfg { +struct { +unsigned long bw_type :32 , + bw_src :12 , + reserved1:3 , + is_clos :1 , + cntr_id : 5,
arch/x86/kernel/cpu/resctrl/internal.h union cpuid_0x10_x_edx { · line 204
Quoted 10 lines
arch/x86/kernel/cpu/resctrl/internal.h +7
-0+ reserved : 9, + cntr_en : 1, + cfg_en : 1; + } split; +unsigned long full; +}; +void rdt_ctrl_update (void *arg);int rdt_get_mon_l3_config (struct rdt_resource *r);
Reinette
Re: [PATCH v13 10/27] x86/resctrl: Add data structures and definitions for ABMC assignment
Hi Reinette,
Quoted 11 lines · Reinette Chatre
Hi Babu,
Quoted 5 lines · Babu Moger
The ABMC feature provides an option to the user to assign a hardware counter to an RMID, event pair and monitor the bandwidth as long as the counter is assigned. The bandwidth events will be tracked by the hardware until the user changes the configuration. Each resctrl group can configure maximum two counters, one for total event and one for local event.
(please update, above describes previous design)
Ok. Will drop the last line.
Quoted 9 lines
Quoted 6 lines
The ABMC feature implements an MSR
L3_QOS_ABMC_CFG(C000_03FDh). ABMC counter assignment is done by setting the counter id, bandwidth source (RMID) and bandwidth configuration. Users will have the option to change the bandwidth configuration using resctrl interface which will be introduced later in the series."will be introduced later in the series" is similar to "in a subsequent patch" and should not be used in a changelog. Just describe what this patch does.
ok. Last line is really not required.
Quoted 100 lines
Quoted 100 lines
Attempts to read or write the MSR when ABMC is not enabled will result in a #GP(0) exception.
Introduce the data structures and definitions for MSR
L3_QOS_ABMC_CFG(0xC000_03FDh):
=========================================================================
Bits Mnemonic Description Access Reset
Type Value
=========================================================================
63 CfgEn Configuration Enable R/W 062 CtrEn Enable/disable counting R/W 0
61:53 – Reserved MBZ 0
52:48 CtrID Counter Identifier R/W 0
47 IsCOS BwSrc field is a CLOSID R/W 0
(not an RMID)46:44 – Reserved MBZ 0
43:32 BwSrc Bandwidth Source R/W 0
(RMID or CLOSID)31:0 BwType Bandwidth configuration R/W 0
to track for this counter
==========================================================================The feature details are documented in the APM listed below [1].
[1] AMD64 Architecture Programmer's Manual Volume 2: System Programming Publication # 24593 Revision 3.41 section 19.3.3.3 Assignable Bandwidth Monitoring (ABMC).---
v13: Removed the Reviewed-by tag as there is commit log change to remove
BMEC reference.v12: No changes.
v11: No changes.
v10: No changes.
v9: Removed the references of
L3_QOS_ABMC_DSC.
Text changes about configuration in kernel doc.v8: Update the configuration notes in
kernel_doc.
Few commit message update.v7: Removed the reference of
L3_QOS_ABMC_DSCas it is not used anymore.
Moved the configuration notes tokernel_doc.
Adjusted the tabs forl3_qos_abmc_cfgand checkpatch seems happy.v6: Removed all the fs related changes.
Added note on CfgEn,CtrEn.
Removed the definitions which are not used.
Removedcntr_idinitialization.v5: Moved assignment flags here (path 10/19 of v4).
AddedMON_CNTR_UNSETdefinition to initialize cntr_id's. More details in commit log.
Renamed few fields inl3_qos_abmc_cfgfor readability.v4: Added more descriptions.
Changed the nameabmc_ctr_idtoctr_id.
AddedL3_QOS_ABMC_DSC. Used for reading the configuration.v3: No changes.
v2: No changes.
--- arch/x86/include/asm/msr-index.h | 1 + arch/x86/kernel/cpu/resctrl/internal.h | 35 ++++++++++++++++++++++++++ 2 files changed, 36 insertions(+)arch/x86/include/asm/msr-index.h +1
-0@@ -1203,6 +1203,7 @@/* - AMD: */ #define MSR_IA32_MBA_BW_BASE 0xc0000200#define MSR_IA32_SMBA_BW_BASE 0xc0000280 +#define MSR_IA32_L3_QOS_ABMC_CFG 0xc00003fd#define MSR_IA32_L3_QOS_EXT_CFG 0xc00003ff#define MSR_IA32_EVT_CFG_BASE 0xc0000400arch/x86/kernel/cpu/resctrl/internal.h +0
-0@@ -164,6 +164,41 @@ union cpuid_0x10_x_edx {unsigned int full; };
Quoted 7 lines
arch/x86/kernel/cpu/resctrl/internal.h union cpuid_0x10_x_edx { · line 170
Quoted 4 lines
arch/x86/kernel/cpu/resctrl/internal.h +4
-0+/* +* ABMC counters are configured by writing to L3_QOS_ABMC_CFG. +* @bw_type : Bandwidth configuration (supported by BMEC) +* tracked by the @cntr_id. The "supported by BMEC" is unexpected with the new design that separated the two features.
My bad. Will remove it.
Quoted 33 lines
Quoted 33 lines
arch/x86/kernel/cpu/resctrl/internal.h +31
-0+* @bw_src : Bandwidth source (RMID or CLOSID). +* @reserved1 : Reserved. +* @is_clos : @bw_src field is a CLOSID (not an RMID). +* @cntr_id : Counter identifier. +* @reserved : Reserved. +* @cntr_en : Counting enable bit. +* @cfg_en : Configuration enable bit. +* +* Configuration and counting: +* Counter can be configured across multiple writes to MSR. Configuration +* is applied only when @cfg_en = 1. Counter @cntr_id is reset when the +* configuration is applied. +* @cfg_en = 1, @cntr_en = 0 : Apply @cntr_id configuration but do not +* count events. +* @cfg_en = 1, @cntr_en = 1 : Apply @cntr_id configuration and start +* counting events. +*/ +union l3_qos_abmc_cfg { +struct { +unsigned long bw_type :32 , + bw_src :12 , + reserved1:3 , + is_clos :1 , + cntr_id :5 , + reserved :9 , + cntr_en :1 , + cfg_en :1 ; + } split; +unsigned long full; +}; +void rdt_ctrl_update (void *arg);
Quoted 4 lines
arch/x86/kernel/cpu/resctrl/internal.h union cpuid_0x10_x_edx { · line 204
Quoted 1 line
arch/x86/kernel/cpu/resctrl/internal.h +0
-0int rdt_get_mon_l3_config (struct rdt_resource *r);Reinette
Thanks
Babu
Babu Moger
reviewerPATCH v13 11/27 x86/resctrl: Implement resctrl_arch_config_cntr() to assign a counter with ABMC
In reply to Babu MogerThe ABMC feature provides an option to the user to assign a hardware counter to an RMID, event pair and monitor the bandwidth as long as it is assigned. The assigned RMID will be tracked by the hardware until the user unassigns it manually.
Implement an architecture-specific handler to assign and unassign the counter. Configure counters by writing to the L3_QOS_ABMC_CFG MSR, specifying the counter ID, bandwidth source (RMID), and event configuration.
The feature details are documented in the APM listed below [1].
[1] AMD64 Architecture Programmer's Manual Volume 2: System Programming
Publication # 24593 Revision 3.41 section 19.3.3.3 Assignable Bandwidth
Monitoring (ABMC).---
v13: Moved resctrl_arch_config_cntr() prototype to include/linux/resctrl.h.
Changed resctrl_arch_config_cntr() to retun void from int. Updated the kernal doc for the prototype.
Updated the code comment.
12: Added the check to reset the architecture-specific state only when
assign is requested.
Added evt_cfg as the parameter as the user will be passing the event configuration from /info/L3_MON/event_configs/.
v11: Moved resctrl_arch_assign_cntr() and resctrl_abmc_config_one_amd() to
monitor.c.
Added the code to reset the arch state in resctrl_arch_assign_cntr(). Also removed resctrl_arch_reset_rmid() inside IPI as the counters are reset from the callers.
Re-wrote commit message.
v10: Added call resctrl_arch_reset_rmid() to reset the RMID in the domain
inside IPI call.
SMP and non-SMP call support is not required in resctrl_arch_config_cntr with new domain specific assign approach/data structure. Commit message update.
v9: Removed the code to reset the architectural state. It will done
in another patch.
v8: Rename resctrl_arch_assign_cntr to resctrl_arch_config_cntr.
v7: Separated arch and fs functions. This patch only has arch implementation.
Added struct rdt_resource to the interface resctrl_arch_assign_cntr. Rename rdtgroup_abmc_cfg() to resctrl_abmc_config_one_amd().
v6: Removed mbm_cntr_alloc() from this patch to keep fs and arch code
separate.
Added code to update the counter assignment at domain level.
v5: Few name changes to match cntr_id.
Changed the function names to
rdtgroup_assign_cntr
resctr_arch_assign_cntr
More comments on commit log.
Added function summary.
v4: Commit message update.
User bitmap APIs where applicable. Changed the interfaces considering MPAM(arm). Added domain specific assignment.
v3: Removed the static from the prototype of rdtgroup_assign_abmc.
The function is not called directly from user anymore. These
--- arch/x86/kernel/cpu/resctrl/monitor.c | 37 +++++++++++++++++++++++++++ include/linux/resctrl.h | 17 ++++++++++++ 2 files changed, 54 insertions(+)
arch/x86/kernel/cpu/resctrl/monitor.c +37 -0
@@ -448,3 +448,40 @@ inline bool resctrl_arch_mbm_cntr_assign_enabled(struct rdt_resource *r) {return resctrl_to_arch_res (r)->mbm_cntr_assign_enabled; } + +static void resctrl_abmc_config_one_amd (void *info) +{ +union l3_qos_abmc_cfg *abmc_cfg = info; + +wrmsrl (MSR_IA32_L3_QOS_ABMC_CFG , abmc_cfg->full); +} + +/* +* Send an IPI to the domain to assign the counter to RMID, event pair. +*/ +void resctrl_arch_config_cntr (struct rdt_resource *r,struct rdt_mon_domain *d, +enum resctrl_event_id evtid,u32 rmid,u32 closid, +u32 cntr_id,u32 evt_cfg,bool assign) +{ +struct rdt_hw_mon_domain *hw_dom =resctrl_to_arch_mon_dom (d); +union l3_qos_abmc_cfg abmc_cfg = {0 }; +struct arch_mbm_state *am; + + abmc_cfg.split.cfg_en =1 ; + abmc_cfg.split.cntr_en = assign ?1 :0 ; + abmc_cfg.split.cntr_id = cntr_id; + abmc_cfg.split.bw_src = rmid; + abmc_cfg.split.bw_type = evt_cfg; + +smp_call_function_any (&d->hdr.cpu_mask, resctrl_abmc_config_one_amd, &abmc_cfg,1 ); + +/* +* The hardware counter is reset (because cfg_en == 1) so there is no +* need to record initial non-zero counts. +*/ +if (assign) { + am =get_arch_mbm_state (hw_dom, rmid, evtid); +if (am) +memset (am,0 ,sizeof (*am)); + } +}
include/linux/resctrl.h +17 -0
@@ -559,6 +559,23 @@ void resctrl_arch_reset_rmid_all(struct rdt_resource *r, struct rdt_mon_domain **/ void resctrl_arch_reset_all_ctrls (struct rdt_resource *r); +/** +* resctrl_arch_config_cntr() - Configure the counter id to RMID, event +* pair on the domain. +* @r: Resource structure. +* @d: Domain that the counter id to be configured. +* @evtid: Event type to configure. +* @rmid: RMID to configure. +* @closid: CLOSID to configure. +* @cntr_id: Counter ID to configure. +* @evt_cfg: MBM event configuration value representing reads, +* writes etc. +* @assign: Assign or unassign. +*/ +void resctrl_arch_config_cntr (struct rdt_resource *r,struct rdt_mon_domain *d, +enum resctrl_event_id evtid,u32 rmid,u32 closid, +u32 cntr_id,u32 evt_cfg,bool assign); +extern unsigned int resctrl_rmid_realloc_threshold;extern unsigned int resctrl_rmid_realloc_limit;
Signature · 1 line
2.34.1
Reinette Chatre
maintainerRe: [PATCH v13 11/27] x86/resctrl: Implement resctrl_arch_config_cntr() to assign a counter with ABMC
Hi Babu,
Quoted 4 lines · Babu Moger
The ABMC feature provides an option to the user to assign a hardware counter to an RMID, event pair and monitor the bandwidth as long as it is assigned. The assigned RMID will be tracked by the hardware until the user unassigns it manually.
(please review this often repeated snippet to match new design)
Quoted 6 lines
Implement an architecture-specific handler to assign and unassign the counter. Configure counters by writing to the
L3_QOS_ABMC_CFGMSR, specifying the counter ID, bandwidth source (RMID), and event configuration.
...
Quoted 22 lines
arch/x86/kernel/cpu/resctrl/monitor.c +14
-0@@ -448,3 +448,40 @@ inline bool resctrl_arch_mbm_cntr_assign_enabled(struct rdt_resource *r) {return resctrl_to_arch_res (r)->mbm_cntr_assign_enabled; } + +static void resctrl_abmc_config_one_amd (void *info) +{ +union l3_qos_abmc_cfg *abmc_cfg = info; + +wrmsrl (MSR_IA32_L3_QOS_ABMC_CFG , abmc_cfg->full); +} + +/* +* Send an IPI to the domain to assign the counter to RMID, event pair. +*/ +void resctrl_arch_config_cntr (struct rdt_resource *r,struct rdt_mon_domain *d, +enum resctrl_event_id evtid,u32 rmid,u32 closid, +u32 cntr_id,u32 evt_cfg,bool assign)
arch/x86/kernel/cpu/resctrl/monitor.c inline bool resctrl_arch_mbm_cntr_assign_enabled(struct rdt_resource *r) · line 474
Quoted 10 lines
arch/x86/kernel/cpu/resctrl/monitor.c +10
-0+{ +struct rdt_hw_mon_domain *hw_dom =resctrl_to_arch_mon_dom (d); +union l3_qos_abmc_cfg abmc_cfg = {0 }; +struct arch_mbm_state *am; + + abmc_cfg.split.cfg_en =1 ; + abmc_cfg.split.cntr_en = assign ?1 :0 ; + abmc_cfg.split.cntr_id = cntr_id; + abmc_cfg.split.bw_src = rmid; + abmc_cfg.split.bw_type = evt_cfg;
Is evt_cfg really needed to be programmed when unassigning a counter? Looking ahead at patch #14 resctrl_free_config_cntr() needs to go through extra list walk to get this data but why would hardware need an accurate event configuration to *unassign* a counter?
It seems unnecessary to provide both the event ID *and* the configuration. resctrl_arch_config_cntr() could drop the "evt_cfg" parameter and instead there can be a new resctrl utility that architecture can use to query the event's configuration. Similar to resctrl_is_mon_event_enabled() introduced in
https://lore.kernel.org/lkml/[email protected]/ that exposes an event property.
It looks to me as though there are a couple of changes in the telemetry work
that would benefit this work. https://lore.kernel.org/lkml/[email protected]/
switches the monitor events to be maintained in an array indexed by event ID, eliminating the need for searching the evt_list that this work does in a couple of places. Also note the handy new for_each_mbm_event() helper (https://lore.kernel.org/lkml/[email protected]/).
Quoted 18 lines
arch/x86/kernel/cpu/resctrl/monitor.c +13
-0+ +smp_call_function_any (&d->hdr.cpu_mask, resctrl_abmc_config_one_amd, &abmc_cfg,1 ); + +/* +* The hardware counter is reset (because cfg_en == 1) so there is no +* need to record initial non-zero counts. +*/ +if (assign) { + am =get_arch_mbm_state (hw_dom, rmid, evtid); +if (am) +memset (am,0 ,sizeof (*am)); + } +}include/linux/resctrl.h +0
-0
include/linux/resctrl.h void resctrl_arch_reset_rmid_all(struct rdt_resource *r, struct rdt_mon_domain * · line 564
Quoted 6 lines
include/linux/resctrl.h +3
-0@@ -559,6 +559,23 @@ void resctrl_arch_reset_rmid_all(struct rdt_resource *r, struct rdt_mon_domain **/ void resctrl_arch_reset_all_ctrls (struct rdt_resource *r); +/** +* resctrl_arch_config_cntr() - Configure the counter id to RMID, event +* pair on the domain.
The sentence seem strange, should "Configure the counter" perhaps be "Assign the counter"? Or if the naming requires "configure" ... "Configure the counter with its new RMID and event details."? Please feel free to improve.
include/linux/resctrl.h void resctrl_arch_reset_rmid_all(struct rdt_resource *r, struct rdt_mon_domain * · line 566
Quoted 2 lines
include/linux/resctrl.h +2
-0+* @r: Resource structure. +* @d: Domain that the counter id to be configured.
I am unable to parse description of @d.
include/linux/resctrl.h void resctrl_arch_reset_rmid_all(struct rdt_resource *r, struct rdt_mon_domain * · line 570
Quoted 4 lines
include/linux/resctrl.h +4
-0+* @evtid: Event type to configure. +* @rmid: RMID to configure. +* @closid: CLOSID to configure. +* @cntr_id: Counter ID to configure.
All four parameters descriptions end with "to configure" ... but it is actually only the counter that is configured while the rest is the data that the counter is configured with, no?
include/linux/resctrl.h void resctrl_arch_reset_rmid_all(struct rdt_resource *r, struct rdt_mon_domain * · line 572
Quoted 2 lines
include/linux/resctrl.h +2
-0+* @evt_cfg: MBM event configuration value representing reads, +* writes etc.
Needs definition about what the contents of @evt_cfg means. This is the API ...it cannot be vague like "reads, write, etc." but should be specific about which bit means what.
include/linux/resctrl.h void resctrl_arch_reset_rmid_all(struct rdt_resource *r, struct rdt_mon_domain * · line 573
Quoted 1 line
include/linux/resctrl.h +1
-0+* @assign: Assign or unassign.
"True to assign the counter, false to unassign the counter."
Needs some context here about what architecture can expect on how this function will be called. For example, "Can be called from any CPU."
include/linux/resctrl.h void resctrl_arch_reset_rmid_all(struct rdt_resource *r, struct rdt_mon_domain * · line 581
Quoted 8 lines
include/linux/resctrl.h +5
-0+*/ +void resctrl_arch_config_cntr (struct rdt_resource *r,struct rdt_mon_domain *d, +enum resctrl_event_id evtid,u32 rmid,u32 closid, +u32 cntr_id,u32 evt_cfg,bool assign); +extern unsigned int resctrl_rmid_realloc_threshold;extern unsigned int resctrl_rmid_realloc_limit;
Reinette
Luck, Tony
maintainerRE: [PATCH v13 11/27] x86/resctrl: Implement resctrl_arch_config_cntr() to assign a counter with ABMC
Quoted 5 lines
It looks to me as though there are a couple of changes in the telemetry work
that would benefit this work. https://lore.kernel.org/lkml/[email protected]/
switches the monitor events to be maintained in an array indexed by event ID, eliminating the need for searching theevt_listthat this work does in a couple of places. Also note the handy newfor_each_mbm_event()helper (https://lore.kernel.org/lkml/[email protected]/).
Yesterday I ran through the exercise of rebasing my AET patches on top of these ABMC patches in order to check whether the ABMC patches painted resctrl into some corner that would be hard to get back out of.
Good news: they don't.
There was a bunch of manual patching to make the first four patches fit on top of the ABMC code, but I also noticed a few places where things were simpler after combining the two series.
Maybe a good path forward would be to take those first four patches from my AET series and then build ABMC on top of those.
-Tony
Luck, Tony
maintainerRe: [PATCH v13 11/27] x86/resctrl: Implement resctrl_arch_config_cntr() to assign a counter with ABMC
Quoted 10 lines · Tony
Quoted 5 lines
It looks to me as though there are a couple of changes in the telemetry work
that would benefit this work. https://lore.kernel.org/lkml/[email protected]/
switches the monitor events to be maintained in an array indexed by event ID, eliminating the need for searching theevt_listthat this work does in a couple of places. Also note the handy newfor_each_mbm_event()helper (https://lore.kernel.org/lkml/[email protected]/).Yesterday I ran through the exercise of rebasing my AET patches on top of these ABMC patches in order to check whether the ABMC patches painted resctrl into some corner that would be hard to get back out of.
Quoted 8 lines
Good news: they don't.
There was a bunch of manual patching to make the first four patches fit on top of the ABMC code, but I also noticed a few places where things were simpler after combining the two series.
Maybe a good path forward would be to take those first four patches from my AET series and then build ABMC on top of those.
As an encouragement to try this direction, I took my four patches on top of tip x86/cache and then applied Babu's ABMC series.
Changes to Babu's code:
1) Adapt where needed for removal of evt_list. Use event array instead.
2) Use for_each_mbm_event() [Maybe didn't get all places?]
3) Bring the s/evt_val/evt_cfg/ fix into patch 20 from 21
4) Fix fir tree declaration for resctrl_process_assign()
I don't have an AMD system to check if the ABMC parts still work. But it does pass the resctrl self tests, so legacy isn't broken.
Patches in the "my_mbm_plus_babu_abmc" branch of my kernel.org
repo: git://git.kernel.org/pub/scm/linux/kernel/git/aegl/linux.git
-Tony
Re: [PATCH v13 11/27] x86/resctrl: Implement resctrl_arch_config_cntr() to assign a counter with ABMC
Hi Tony,
Quoted 18 lines · Tony
Quoted 16 lines · Tony
Quoted 5 lines
It looks to me as though there are a couple of changes in the telemetry work
that would benefit this work. https://lore.kernel.org/lkml/[email protected]/
switches the monitor events to be maintained in an array indexed by event ID, eliminating the need for searching theevt_listthat this work does in a couple of places. Also note the handy newfor_each_mbm_event()helper (https://lore.kernel.org/lkml/[email protected]/).Yesterday I ran through the exercise of rebasing my AET patches on top of these ABMC patches in order to check whether the ABMC patches painted resctrl into some corner that would be hard to get back out of.
Good news: they don't.
There was a bunch of manual patching to make the first four patches fit on top of the ABMC code, but I also noticed a few places where things were simpler after combining the two series.
Quoted 5 lines
Quoted 2 lines
Maybe a good path forward would be to take those first four patches from my AET series and then build ABMC on top of those.
As an encouragement to try this direction, I took my four patches on top of tip x86/cache and then applied Babu's ABMC series.
I did the same thing last week, except in the other order, so I switched to your changes to test.
Quoted 7 lines
Changes to Babu's code:
1) Adapt where needed for removal ofevt_list. Use event array instead.
2) Usefor_each_mbm_event()[Maybe didn't get all places?]
3) Bring the s/evt_val/evt_cfg/ fix into patch 20 from 21
4) Fix fir tree declaration forresctrl_process_assign()
Quoted 5 lines
I don't have an AMD system to check if the ABMC parts still work. But it does pass the resctrl self tests, so legacy isn't broken.
Patches in the "
my_mbm_plus_babu_abmc" branch of my kernel.org
repo:git://git.kernel.org/pub/scm/linux/kernel/git/aegl/linux.git
Thanks for applying my suggestion[1] about the array entry sizes, but you needed one more dereference:
(quoted diff) +0 -0
b/arch/x86/kernel/cpu/resctrl/core.c
index 1db6a61e27746..0c27e0a5a7b96 100644
arch/x86/kernel/cpu/resctrl/core.c +0 -0
@@ -399,7 +399,7 @@ static int domain_setup_ctrlval(struct
rdt_resource *r, struct rdt_ctrl_domain *
*/
static int arch_domain_mbm_alloc(u32 num_rmid, structrdt_hw_mon_domain *hw_dom)
{
- size_t tsize = sizeof(hw_dom->arch_mbm_states[0]);
+ size_t tsize = sizeof(*hw_dom->arch_mbm_states[0]);
enum resctrl_event_id evt;
int idx;fs/resctrl/rdtgroup.c +0 -0
@@ -4819,7 +4823,7 @@ void resctrl_offline_mon_domain(struct
rdt_resource *r, struct rdt_mon_domain *d
static int domain_setup_mon_state(struct rdt_resource *r, structrdt_mon_domain *d)
{
u32 idx_limit = resctrl_arch_system_num_rmid_idx();
- size_t tsize = sizeof(d->mbm_states[0]);
+ size_t tsize = sizeof(*d->mbm_states[0]);
enum resctrl_event_id evt;
int idx;You should be able to repro an array overrun without ABMC, and a page fault is likely if the system implements a lot of RMIDs. The AMD EPYC 9B45 I tested on implements 4096 RMIDs.
Thanks,
-Peter
[1] https://lore.kernel.org/lkml/CALPaoCj8yfzJ=5CkxTPQXc0-WRWpu0xKRX8v4FAWFGQKtXtMUw@mail.gmail.com/
Luck, Tony
maintainerRE: [PATCH v13 11/27] x86/resctrl: Implement resctrl_arch_config_cntr() to assign a counter with ABMC
Review
Quoted 9 lines
Thanks for applying my suggestion[1] about the array entry sizes, but you needed one more dereference:
(quoted diff) +2
-2- size_t tsize = sizeof(hw_dom->arch_mbm_states[0]); + size_t tsize = sizeof(*hw_dom->arch_mbm_states[0]); - size_t tsize = sizeof(d->mbm_states[0]); + size_t tsize = sizeof(*d->mbm_states[0]);
Indeed yes. Thanks.
-Tony
Re: RE: [PATCH v13 11/27] x86/resctrl: Implement resctrl_arch_config_cntr() to assign a counter with ABMC
Hi Tony, Peter,
Quoted 4 lines · Tony
Quoted 3 lines
Thanks for applying my suggestion[1] about the array entry sizes, but you needed one more dereference:
Quoted 8 lines
Review
Quoted 5 lines
(quoted diff) +2
-2- size_t tsize = sizeof(hw_dom->arch_mbm_states[0]); + size_t tsize = sizeof(*hw_dom->arch_mbm_states[0]); - size_t tsize = sizeof(d->mbm_states[0]); + size_t tsize = sizeof(*d->mbm_states[0]);Indeed yes. Thanks.
Tony, Thanks for porting patches.
I can actually pick your branch [1] and apply review comments on top for v14 series. Hope that is fine with everyone.
[1] https://git.kernel.org/pub/scm/linux/kernel/git/aegl/linux.git/log/?h=my_mbm_plus_babu_abmc
One question though: Where will the Peter's fix [2] go?
[2] https://lore.kernel.org/lkml/CALPaoCj7FBv_vfDp+4tgqo4p8T7Eov_Ys+CQRoAX6u43a4OTDQ@mail.gmail.com/
thanks
Babu
Luck, Tony
maintainerRE: RE: [PATCH v13 11/27] x86/resctrl: Implement resctrl_arch_config_cntr() to assign a counter with ABMC
Quoted 3 lines
One question though: Where will the Peter's fix [2] go?
[2]
https://lore.kernel.org/lkml/CALPaoCj7FBv_vfDp+4tgqo4p8T7Eov_Ys+CQRoAX6u43a4OTDQ@mail.gmail.com/
Babu,
I'll backport into my patches and then rebase the whole branch. Hopefully should not take long.
-Tony
Luck, Tony
maintainerRE: RE: [PATCH v13 11/27] x86/resctrl: Implement resctrl_arch_config_cntr() to assign a counter with ABMC
Quoted 1 line
I'll backport into my patches and then rebase the whole branch. Hopefully should not take long.
Done and force pushed to same branch at kernel .org.
Both Peter's changes went into my 4th patch, and your patches didn't touch those functions so git did all the real work.
-Tony
Re: RE: RE: [PATCH v13 11/27] x86/resctrl: Implement resctrl_arch_config_cntr() to assign a counter with ABMC
Hi Tony,
Quoted 7 lines · Tony
Quoted 1 line
I'll backport into my patches and then rebase the whole branch. Hopefully should not take long.
Done and force pushed to same branch at kernel .org.
Both Peter's changes went into my 4th patch, and your patches didn't touch those functions so git did all the real work.
I will base my v14 on top of this. Thanks
Babu
Moger, Babu
reviewerRe: RE: [PATCH v13 11/27] x86/resctrl: Implement resctrl_arch_config_cntr() to assign a counter with ABMC
In reply to Moger, BabuHi Tony,
Quoted 23 lines · Babu
Hi Tony, Peter,
Quoted 4 lines · Tony
Quoted 3 lines
Thanks for applying my suggestion[1] about the array entry sizes, but you needed one more dereference:
Quoted 8 lines
Quoted 5 lines
-
size_ttsize =sizeof(hw_dom->arch_mbm_states[0]); +size_ttsize =sizeof(*hw_dom->arch_mbm_states[0]);-
size_ttsize =sizeof(d->mbm_states[0]);
+size_ttsize =sizeof(*d->mbm_states[0]);Indeed yes. Thanks.
Tony, Thanks for porting patches.
I can actually pick your branch [1] and apply review comments on top for v14 series. Hope that is fine with everyone.
[1]
https://git.kernel.org/pub/scm/linux/kernel/git/aegl/linux.git/log/?h=my_mbm_plus_babu_abmc
Quoted 8 lines
One question though: Where will the Peter's fix [2] go?
[2]
https://lore.kernel.org/lkml/CALPaoCj7FBv_vfDp+4tgqo4p8T7Eov_Ys+CQRoAX6u43a4OTDQ@mail.gmail.com/thanks
Babu
I'm currently working on v14 and plan to post the updated ABMC series tomorrow. I've used your multi-event support patches as the base:
x86, fs/resctrl: Consolidate monitor event descriptions
x86, fs/resctrl: Replace architecture event enabled checks
x86/resctrl: Remove 'rdt_mon_features' global variable
x86, fs/resctrl: Prepare for more monitor events
I noticed there are a few comments on your series here:
https://lore.kernel.org/lkml/[email protected]/
Let me know if you've updated the patches. If so, I’ll incorporate the latest version. Otherwise, I’ll proceed with the current base as-is.
Signature · 2 lines
Thanks Babu Moger
Tony Luck
maintainercover Latest version of initial patches in AET series
Babu,
Here's my latest version of those four patches (build on top of Qinyun's patch that I believe should go upstream via tip/urgent).
I believe I addressed all the comments[1] in these four patches
[1] https://lore.kernel.org/lkml/[email protected]/
-Tony
Qinyun Tan (1):
x86,fs/resctrl: Remove unappropriate references to cacheinfo in the
resctrl subsystem.
Tony Luck (4):
x86,fs/resctrl: Consolidate monitor event descriptions x86,fs/resctrl: Replace architecture event enabled checks x86/resctrl: Remove 'rdt_mon_features' global variable x86,fs/resctrl: Prepare for more monitor events
include/linux/resctrl.h | 28 +++++--- include/linux/resctrl_types.h | 15 ++-- arch/x86/include/asm/resctrl.h | 16 ----- arch/x86/kernel/cpu/resctrl/internal.h | 9 +-- fs/resctrl/internal.h | 17 +++-- arch/x86/kernel/cpu/resctrl/core.c | 65 +++++++++-------- arch/x86/kernel/cpu/resctrl/monitor.c | 41 +++++------ fs/resctrl/ctrlmondata.c | 17 +++-- fs/resctrl/monitor.c | 96 ++++++++++++++------------ fs/resctrl/rdtgroup.c | 79 +++++++++++---------- 10 files changed, 204 insertions(+), 179 deletions(-)
Signature · 1 line
2.49.0
Tony Luck
maintainerPATCH 1/5 x86,fs/resctrl: Remove unappropriate references to cacheinfo in the resctrl subsystem.
From: Qinyun Tan <[email protected]>
In the resctrl subsystem's Sub-NUMA Cluster (SNC) mode, the rdt_mon_domain structure representing a NUMA node relies on the cacheinfo interface (rdt_mon_domain::ci) to store L3 cache information (e.g., shared_cpu_map) for monitoring. The L3 cache information of a SNC NUMA node determines which domains are summed for the "top level" L3-scoped events.
rdt_mon_domain::ci is initialized using the first online CPU of a NUMA node. When this CPU goes offline, its shared_cpu_map is cleared to contain only the offline CPU itself. Subsequently, attempting to read counters via smp_call_on_cpu(offline_cpu) fails (and error ignored), returning zero values for "top-level events" without any error indication.
Replace the cacheinfo references in struct rdt_mon_domain and struct rmid_read with the cacheinfo ID (a unique identifier for the L3 cache).
rdt_domain_hdr::cpu_mask contains the online CPUs associated with that domain. When reading "top-level events", select a CPU from rdt_domain_hdr::cpu_mask and utilize its L3 shared_cpu_map to determine valid CPUs for reading RMID counter via the MSR interface.
Considering all CPUs associated with the L3 cache improves the chances of picking a housekeeping CPU on which the counter reading work can be queued, avoiding an unnecessary IPI.
--- include/linux/resctrl.h | 4 ++-- fs/resctrl/internal.h | 4 ++-- arch/x86/kernel/cpu/resctrl/core.c | 6 ++++-- fs/resctrl/ctrlmondata.c | 13 +++++++++---- fs/resctrl/monitor.c | 6 ++++-- fs/resctrl/rdtgroup.c | 6 +++--- 6 files changed, 24 insertions(+), 15 deletions(-)
include/linux/resctrl.h +2 -2
@@ -159,7 +159,7 @@ struct rdt_ctrl_domain {/** * struct rdt_mon_domain - group of CPUs sharing a resctrl monitor resource * @hdr: common header for different domain types -* @ci: cache info for this domain +* @ci_id: cache info id for this domain * @rmid_busy_llc: bitmap of which limbo RMIDs are above threshold * @mbm_total: saved state for MBM total bandwidth * @mbm_local: saved state for MBM local bandwidth
@@ -170,7 +170,7 @@ struct rdt_ctrl_domain {*/ struct rdt_mon_domain {struct rdt_domain_hdr hdr; -struct cacheinfo *ci; +unsigned int ci_id;unsigned long *rmid_busy_llc;struct mbm_state *mbm_total;struct mbm_state *mbm_local;
fs/resctrl/internal.h +2 -2
@@ -98,7 +98,7 @@ struct mon_data {* domains in @r sharing L3 @ci.id * @evtid: Which monitor event to read. * @first: Initialize MBM counter when true. -* @ci: Cacheinfo for L3. Only set when @d is NULL. Used when summing domains. +* @ci_id: Cacheinfo id for L3. Only set when @d is NULL. Used when summing domains. * @err: Error encountered when reading counter. * @val: Returned value of event counter. If @rgrp is a parent resource group, * @val includes the sum of event counts from its child resource groups.
@@ -112,7 +112,7 @@ struct rmid_read {struct rdt_mon_domain *d;enum resctrl_event_id evtid;bool first; -struct cacheinfo *ci; +unsigned int ci_id;int err;u64 val;void *arch_mon_ctx;
arch/x86/kernel/cpu/resctrl/core.c +4 -2
@@ -498,6 +498,7 @@ static void domain_add_cpu_mon(int cpu, struct rdt_resource *r)struct rdt_hw_mon_domain *hw_dom;struct rdt_domain_hdr *hdr;struct rdt_mon_domain *d; +struct cacheinfo *ci;int err;lockdep_assert_held (&domain_list_lock);
@@ -525,12 +526,13 @@ static void domain_add_cpu_mon(int cpu, struct rdt_resource *r) d = &hw_dom->d_resctrl; d->hdr.id = id; d->hdr.type =RESCTRL_MON_DOMAIN ; - d->ci =get_cpu_cacheinfo_level (cpu,RESCTRL_L3_CACHE ); -if (!d->ci) { + ci =get_cpu_cacheinfo_level (cpu,RESCTRL_L3_CACHE ); +if (!ci) {pr_warn_once ("Can't find L3 cache for CPU:%d resource %s\n" , cpu, r->name);mon_domain_free (hw_dom);return ; } + d->ci_id = ci->id;cpumask_set_cpu (cpu, &d->hdr.cpu_mask);arch_mon_domain_online (r, d);
fs/resctrl/ctrlmondata.c +9 -4
@@ -594,9 +594,10 @@ int rdtgroup_mondata_show(struct seq_file *m, void *arg)struct rmid_read rr = {0 };struct rdt_mon_domain *d;struct rdtgroup *rdtgrp; +int domid, cpu, ret =0 ;struct rdt_resource *r; +struct cacheinfo *ci;struct mon_data *md; -int domid, ret =0 ; rdtgrp =rdtgroup_kn_lock_live (of->kn);if (!rdtgrp) {
@@ -623,10 +624,14 @@ int rdtgroup_mondata_show(struct seq_file *m, void *arg)* one that matches this cache id. */ list_for_each_entry (d, &r->mon_domains, hdr.list) { -if (d->ci->id == domid) { - rr.ci = d->ci; +if (d->ci_id == domid) { + rr.ci_id = d->ci_id; + cpu =cpumask_any (&d->hdr.cpu_mask); + ci =get_cpu_cacheinfo_level (cpu,RESCTRL_L3_CACHE ); +if (!ci) +continue ;mon_event_read (&rr, r,NULL , rdtgrp, - &d->ci->shared_cpu_map, evtid, false); + &ci->shared_cpu_map, evtid, false); goto checkresult; } }
fs/resctrl/monitor.c +4 -2
@@ -361,6 +361,7 @@ static int __mon_event_count(u32 closid, u32 rmid, struct rmid_read *rr) {int cpu =smp_processor_id ();struct rdt_mon_domain *d; +struct cacheinfo *ci;struct mbm_state *m;int err, ret;u64 tval =0 ;
@@ -388,7 +389,8 @@ static int __mon_event_count(u32 closid, u32 rmid, struct rmid_read *rr) }/* Summing domains that share a cache, must be on a CPU for that cache. */ -if (!cpumask_test_cpu (cpu, &rr->ci->shared_cpu_map)) + ci =get_cpu_cacheinfo_level (cpu,RESCTRL_L3_CACHE ); +if (!ci || ci->id != rr->ci_id)return -EINVAL ;/*
@@ -400,7 +402,7 @@ static int __mon_event_count(u32 closid, u32 rmid, struct rmid_read *rr)*/ ret = -EINVAL ;list_for_each_entry (d, &rr->r->mon_domains, hdr.list) { -if (d->ci->id != rr->ci->id) +if (d->ci_id != rr->ci_id)continue ; err =resctrl_arch_rmid_read (rr->r, d, closid, rmid, rr->evtid, &tval, rr->arch_mon_ctx);
fs/resctrl/rdtgroup.c +3 -3
@@ -3036,7 +3036,7 @@ static void rmdir_mondata_subdir_allrdtgrp(struct rdt_resource *r,char name[32 ]; snc_mode = r->mon_scope ==RESCTRL_L3_NODE ; -sprintf (name,"mon_%s_%02d" , r->name, snc_mode ? d->ci->id : d->hdr.id); +sprintf (name,"mon_%s_%02d" , r->name, snc_mode ? d->ci_id : d->hdr.id);if (snc_mode)sprintf (subname,"mon_sub_%s_%02d" , r->name, d->hdr.id);
@@ -3061,7 +3061,7 @@ static int mon_add_all_files(struct kernfs_node *kn, struct rdt_mon_domain *d,return -EPERM ;list_for_each_entry (mevt, &r->evt_list, list) { - domid = do_sum ? d->ci->id : d->hdr.id; + domid = do_sum ? d->ci_id : d->hdr.id; priv =mon_get_kn_priv (r->rid, domid, mevt, do_sum);if (WARN_ON_ONCE (!priv))return -EINVAL ;
@@ -3089,7 +3089,7 @@ static int mkdir_mondata_subdir(struct kernfs_node *parent_kn,lockdep_assert_held (&rdtgroup_mutex); snc_mode = r->mon_scope ==RESCTRL_L3_NODE ; -sprintf (name,"mon_%s_%02d" , r->name, snc_mode ? d->ci->id : d->hdr.id); +sprintf (name,"mon_%s_%02d" , r->name, snc_mode ? d->ci_id : d->hdr.id); kn =kernfs_find_and_get (parent_kn, name);if (kn) {/*
Signature · 1 line
2.49.0
Tony Luck
maintainerPATCH 2/5 x86,fs/resctrl: Consolidate monitor event descriptions
In reply to Tony LuckThere are currently only three monitor events, all associated with the RDT_RESOURCE_L3 resource. Growing support for additional events will be easier with some restructuring to have a single point in file system code where all attributes of all events are defined.
Place all event descriptions into an array mon_event_all[]. Doing this has the beneficial side effect of removing the need for rdt_resource::evt_list.
Add resctrl_event_id::QOS_FIRST_EVENT for a lower bound on range checks for event ids and as the starting index to scan mon_event_all[].
Drop the code that builds evt_list and change the two places where the list is scanned to scan mon_event_all[] instead using a new helper macro for_each_mon_event().
Architecture code now informs file system code which events are available with resctrl_enable_mon_event().
--- include/linux/resctrl.h | 4 +- include/linux/resctrl_types.h | 12 ++++-- fs/resctrl/internal.h | 13 ++++-- arch/x86/kernel/cpu/resctrl/core.c | 12 ++++-- fs/resctrl/monitor.c | 63 +++++++++++++++--------------- fs/resctrl/rdtgroup.c | 11 +++--- 6 files changed, 66 insertions(+), 49 deletions(-)
include/linux/resctrl.h +2 -2
@@ -269,7 +269,6 @@ enum resctrl_schema_fmt {* @mon_domains: RCU list of all monitor domains for this resource * @name: Name to use in "schemata" file. * @schema_fmt: Which format string and parser is used for this schema. -* @evt_list: List of monitoring events * @mbm_cfg_mask: Bandwidth sources that can be tracked when bandwidth * monitoring events can be configured. * @cdp_capable: Is the CDP feature available on this resource
@@ -287,7 +286,6 @@ struct rdt_resource {struct list_head mon_domains;char *name;enum resctrl_schema_fmt schema_fmt; -struct list_head evt_list;unsigned int mbm_cfg_mask;bool cdp_capable; };
@@ -372,6 +370,8 @@ u32 resctrl_arch_get_num_closid(struct rdt_resource *r);u32 resctrl_arch_system_num_rmid_idx (void );int resctrl_arch_update_domains (struct rdt_resource *r,u32 closid); +void resctrl_enable_mon_event (enum resctrl_event_id eventid); +bool resctrl_arch_is_evt_configurable (enum resctrl_event_id evt);/**
include/linux/resctrl_types.h +8 -4
@@ -34,11 +34,15 @@/* Max event bits supported */ #define MAX_EVT_CONFIG_BITS GENMASK(6, 0) -/* -* Event IDs, the values match those used to program IA32_QM_EVTSEL before -* reading IA32_QM_CTR on RDT systems. -*/ +/* Event IDs */ enum resctrl_event_id { +/* Must match value of first event below */ +QOS_FIRST_EVENT =0x01 , + +/* +* These values match those used to program IA32_QM_EVTSEL before +* reading IA32_QM_CTR on RDT systems. +*/ QOS_L3_OCCUP_EVENT_ID =0x01 ,QOS_L3_MBM_TOTAL_EVENT_ID =0x02 ,QOS_L3_MBM_LOCAL_EVENT_ID =0x03 ,
fs/resctrl/internal.h +10 -3
@@ -52,19 +52,26 @@ static inline struct rdt_fs_context *rdt_fc2context(struct fs_context *fc) }/** -* struct mon_evt - Entry in the event list of a resource +* struct mon_evt - Description of a monitor event * @evtid: event id +* @rid: index of the resource for this event * @name: name of the event * @configurable: true if the event is configurable -* @list: entry in &rdt_resource->evt_list +* @enabled: true if the event is enabled */ struct mon_evt {enum resctrl_event_id evtid; +enum resctrl_res_level rid;char *name;bool configurable; -struct list_head list; +bool enabled; }; +extern struct mon_evt mon_event_all[QOS_NUM_EVENTS ]; + +#define for_each_mon_event (mevt) for (mevt = &mon_event_all[QOS_FIRST_EVENT]; \ + mevt < &mon_event_all[QOS_NUM_EVENTS]; mevt++) +/** * struct mon_data - Monitoring details for each event file. * @list: Member of the global @mon_data_kn_priv_list list.
arch/x86/kernel/cpu/resctrl/core.c +9 -3
@@ -864,12 +864,18 @@ static __init bool get_rdt_mon_resources(void) {struct rdt_resource *r = &rdt_resources_all[RDT_RESOURCE_L3 ].r_resctrl; -if (rdt_cpu_has (X86_FEATURE_CQM_OCCUP_LLC )) +if (rdt_cpu_has (X86_FEATURE_CQM_OCCUP_LLC )) { +resctrl_enable_mon_event (QOS_L3_OCCUP_EVENT_ID ); rdt_mon_features |= (1 <<QOS_L3_OCCUP_EVENT_ID ); -if (rdt_cpu_has (X86_FEATURE_CQM_MBM_TOTAL )) + } +if (rdt_cpu_has (X86_FEATURE_CQM_MBM_TOTAL )) { +resctrl_enable_mon_event (QOS_L3_MBM_TOTAL_EVENT_ID ); rdt_mon_features |= (1 <<QOS_L3_MBM_TOTAL_EVENT_ID ); -if (rdt_cpu_has (X86_FEATURE_CQM_MBM_LOCAL )) + } +if (rdt_cpu_has (X86_FEATURE_CQM_MBM_LOCAL )) { +resctrl_enable_mon_event (QOS_L3_MBM_LOCAL_EVENT_ID ); rdt_mon_features |= (1 <<QOS_L3_MBM_LOCAL_EVENT_ID ); + }if (!rdt_mon_features)return false;
fs/resctrl/monitor.c +31 -32
84 lines not on this page · raw message
fs/resctrl/rdtgroup.c +6 -5
@@ -1152,7 +1152,9 @@ static int rdt_mon_features_show(struct kernfs_open_file *of,struct rdt_resource *r =rdt_kn_parent_priv (of->kn);struct mon_evt *mevt; -list_for_each_entry (mevt, &r->evt_list, list) { +for_each_mon_event (mevt) { +if (mevt->rid != r->rid || !mevt->enabled) +continue ;seq_printf (seq,"%s\n" , mevt->name);if (mevt->configurable)seq_printf (seq,"%s_config\n" , mevt->name);
@@ -3057,10 +3059,9 @@ static int mon_add_all_files(struct kernfs_node *kn, struct rdt_mon_domain *d,struct mon_evt *mevt;int ret, domid; -if (WARN_ON (list_empty (&r->evt_list))) -return -EPERM ; - -list_for_each_entry (mevt, &r->evt_list, list) { +for_each_mon_event (mevt) { +if (mevt->rid != r->rid || !mevt->enabled) +continue ; domid = do_sum ? d->ci_id : d->hdr.id; priv =mon_get_kn_priv (r->rid, domid, mevt, do_sum);if (WARN_ON_ONCE (!priv))
Signature · 1 line
2.49.0
Tony Luck
maintainerPATCH 3/5 x86,fs/resctrl: Replace architecture event enabled checks
In reply to Tony LuckThe resctrl file system now has complete knowledge of the status of every event. So there is no need for per-event function calls to check.
Replace each of the resctrl_arch_is_{event}enabled() calls with resctrl_is_mon_event_enabled(QOS_{EVENT}).
No functional change.
--- include/linux/resctrl.h | 2 ++ arch/x86/include/asm/resctrl.h | 15 --------------- arch/x86/kernel/cpu/resctrl/core.c | 4 ++-- arch/x86/kernel/cpu/resctrl/monitor.c | 4 ++-- fs/resctrl/ctrlmondata.c | 4 ++-- fs/resctrl/monitor.c | 16 +++++++++++----- fs/resctrl/rdtgroup.c | 18 +++++++++--------- 7 files changed, 28 insertions(+), 35 deletions(-)
include/linux/resctrl.h +2 -0
@@ -372,6 +372,8 @@ int resctrl_arch_update_domains(struct rdt_resource *r, u32 closid);void resctrl_enable_mon_event (enum resctrl_event_id eventid); +bool resctrl_is_mon_event_enabled (enum resctrl_event_id eventid); +bool resctrl_arch_is_evt_configurable (enum resctrl_event_id evt);/**
arch/x86/include/asm/resctrl.h +0 -15
@@ -84,21 +84,6 @@ static inline void resctrl_arch_disable_mon(void)static_branch_dec_cpuslocked (&rdt_enable_key); } -static inline bool resctrl_arch_is_llc_occupancy_enabled (void ) -{ -return (rdt_mon_features & (1 <<QOS_L3_OCCUP_EVENT_ID )); -} - -static inline bool resctrl_arch_is_mbm_total_enabled (void ) -{ -return (rdt_mon_features & (1 <<QOS_L3_MBM_TOTAL_EVENT_ID )); -} - -static inline bool resctrl_arch_is_mbm_local_enabled (void ) -{ -return (rdt_mon_features & (1 <<QOS_L3_MBM_LOCAL_EVENT_ID )); -} -/* * __resctrl_sched_in() - Writes the task's CLOSid/RMID to IA32_PQR_MSR *
arch/x86/kernel/cpu/resctrl/core.c +2 -2
@@ -402,13 +402,13 @@ static int arch_domain_mbm_alloc(u32 num_rmid, struct rdt_hw_mon_domain *hw_dom) {size_t tsize; -if (resctrl_arch_is_mbm_total_enabled ()) { +if (resctrl_is_mon_event_enabled (QOS_L3_MBM_TOTAL_EVENT_ID )) { tsize =sizeof (*hw_dom->arch_mbm_total); hw_dom->arch_mbm_total =kcalloc (num_rmid, tsize,GFP_KERNEL );if (!hw_dom->arch_mbm_total)return -ENOMEM ; } -if (resctrl_arch_is_mbm_local_enabled ()) { +if (resctrl_is_mon_event_enabled (QOS_L3_MBM_LOCAL_EVENT_ID )) { tsize =sizeof (*hw_dom->arch_mbm_local); hw_dom->arch_mbm_local =kcalloc (num_rmid, tsize,GFP_KERNEL );if (!hw_dom->arch_mbm_local) {
arch/x86/kernel/cpu/resctrl/monitor.c +2 -2
@@ -207,11 +207,11 @@ void resctrl_arch_reset_rmid_all(struct rdt_resource *r, struct rdt_mon_domain * {struct rdt_hw_mon_domain *hw_dom =resctrl_to_arch_mon_dom (d); -if (resctrl_arch_is_mbm_total_enabled ()) +if (resctrl_is_mon_event_enabled (QOS_L3_MBM_TOTAL_EVENT_ID ))memset (hw_dom->arch_mbm_total,0 ,sizeof (*hw_dom->arch_mbm_total) * r->num_rmid); -if (resctrl_arch_is_mbm_local_enabled ()) +if (resctrl_is_mon_event_enabled (QOS_L3_MBM_LOCAL_EVENT_ID ))memset (hw_dom->arch_mbm_local,0 ,sizeof (*hw_dom->arch_mbm_local) * r->num_rmid); }
fs/resctrl/ctrlmondata.c +2 -2
@@ -473,12 +473,12 @@ ssize_t rdtgroup_mba_mbps_event_write(struct kernfs_open_file *of,rdt_last_cmd_clear ();if (!strcmp (buf,"mbm_local_bytes" )) { -if (resctrl_arch_is_mbm_local_enabled ()) +if (resctrl_is_mon_event_enabled (QOS_L3_MBM_LOCAL_EVENT_ID )) rdtgrp->mba_mbps_event =QOS_L3_MBM_LOCAL_EVENT_ID ;else ret = -EINVAL ; }else if (!strcmp (buf,"mbm_total_bytes" )) { -if (resctrl_arch_is_mbm_total_enabled ()) +if (resctrl_is_mon_event_enabled (QOS_L3_MBM_TOTAL_EVENT_ID )) rdtgrp->mba_mbps_event =QOS_L3_MBM_TOTAL_EVENT_ID ;else ret = -EINVAL ;
fs/resctrl/monitor.c +11 -5
@@ -336,7 +336,7 @@ void free_rmid(u32 closid, u32 rmid) entry =__rmid_entry (idx); -if (resctrl_arch_is_llc_occupancy_enabled ()) +if (resctrl_is_mon_event_enabled (QOS_L3_OCCUP_EVENT_ID ))add_rmid_to_limbo (entry);else list_add_tail (&entry->list, &rmid_free_lru);
@@ -637,10 +637,10 @@ static void mbm_update(struct rdt_resource *r, struct rdt_mon_domain *d,* This is protected from concurrent reads from user as both * the user and overflow handler hold the global mutex. */ -if (resctrl_arch_is_mbm_total_enabled ()) +if (resctrl_is_mon_event_enabled (QOS_L3_MBM_TOTAL_EVENT_ID ))mbm_update_one_event (r, d, closid, rmid,QOS_L3_MBM_TOTAL_EVENT_ID ); -if (resctrl_arch_is_mbm_local_enabled ()) +if (resctrl_is_mon_event_enabled (QOS_L3_MBM_LOCAL_EVENT_ID ))mbm_update_one_event (r, d, closid, rmid,QOS_L3_MBM_LOCAL_EVENT_ID ); }
@@ -879,6 +879,12 @@ void resctrl_enable_mon_event(enum resctrl_event_id eventid) mon_event_all[eventid].enabled = true; } +bool resctrl_is_mon_event_enabled (enum resctrl_event_id eventid) +{ +return eventid >=QOS_FIRST_EVENT && eventid <QOS_NUM_EVENTS && + mon_event_all[eventid].enabled; +} +/** * resctrl_mon_resource_init() - Initialise global monitoring structures. *
@@ -914,9 +920,9 @@ int resctrl_mon_resource_init(void) RFTYPE_MON_INFO | RFTYPE_RES_CACHE); } -if (resctrl_arch_is_mbm_local_enabled ()) +if (resctrl_is_mon_event_enabled (QOS_L3_MBM_LOCAL_EVENT_ID )) mba_mbps_default_event =QOS_L3_MBM_LOCAL_EVENT_ID ; -else if (resctrl_arch_is_mbm_total_enabled ()) +else if (resctrl_is_mon_event_enabled (QOS_L3_MBM_TOTAL_EVENT_ID )) mba_mbps_default_event =QOS_L3_MBM_TOTAL_EVENT_ID ;return 0 ;
fs/resctrl/rdtgroup.c +9 -9
@@ -123,8 +123,8 @@ void rdt_staged_configs_clear(void)static bool resctrl_is_mbm_enabled (void ) { -return (resctrl_arch_is_mbm_total_enabled () || -resctrl_arch_is_mbm_local_enabled ()); +return (resctrl_is_mon_event_enabled (QOS_L3_MBM_TOTAL_EVENT_ID ) || +resctrl_is_mon_event_enabled (QOS_L3_MBM_LOCAL_EVENT_ID )); }static bool resctrl_is_mbm_event (int e)
@@ -196,7 +196,7 @@ static int closid_alloc(void)lockdep_assert_held (&rdtgroup_mutex);if (IS_ENABLED (CONFIG_RESCTRL_RMID_DEPENDS_ON_CLOSID ) && -resctrl_arch_is_llc_occupancy_enabled ()) { +resctrl_is_mon_event_enabled (QOS_L3_OCCUP_EVENT_ID )) { cleanest_closid =resctrl_find_cleanest_closid ();if (cleanest_closid <0 )return cleanest_closid;
@@ -4051,7 +4051,7 @@ void resctrl_offline_mon_domain(struct rdt_resource *r, struct rdt_mon_domain *dif (resctrl_is_mbm_enabled ())cancel_delayed_work (&d->mbm_over); -if (resctrl_arch_is_llc_occupancy_enabled () &&has_busy_rmid (d)) { +if (resctrl_is_mon_event_enabled (QOS_L3_OCCUP_EVENT_ID ) &&has_busy_rmid (d)) {/* * When a package is going down, forcefully * decrement rmid->ebusy. There is no way to know
@@ -4087,12 +4087,12 @@ static int domain_setup_mon_state(struct rdt_resource *r, struct rdt_mon_domainu32 idx_limit =resctrl_arch_system_num_rmid_idx ();size_t tsize; -if (resctrl_arch_is_llc_occupancy_enabled ()) { +if (resctrl_is_mon_event_enabled (QOS_L3_OCCUP_EVENT_ID )) { d->rmid_busy_llc =bitmap_zalloc (idx_limit,GFP_KERNEL );if (!d->rmid_busy_llc)return -ENOMEM ; } -if (resctrl_arch_is_mbm_total_enabled ()) { +if (resctrl_is_mon_event_enabled (QOS_L3_MBM_TOTAL_EVENT_ID )) { tsize =sizeof (*d->mbm_total); d->mbm_total =kcalloc (idx_limit, tsize,GFP_KERNEL );if (!d->mbm_total) {
@@ -4100,7 +4100,7 @@ static int domain_setup_mon_state(struct rdt_resource *r, struct rdt_mon_domainreturn -ENOMEM ; } } -if (resctrl_arch_is_mbm_local_enabled ()) { +if (resctrl_is_mon_event_enabled (QOS_L3_MBM_LOCAL_EVENT_ID )) { tsize =sizeof (*d->mbm_local); d->mbm_local =kcalloc (idx_limit, tsize,GFP_KERNEL );if (!d->mbm_local) {
@@ -4145,7 +4145,7 @@ int resctrl_online_mon_domain(struct rdt_resource *r, struct rdt_mon_domain *d) RESCTRL_PICK_ANY_CPU); } -if (resctrl_arch_is_llc_occupancy_enabled ()) +if (resctrl_is_mon_event_enabled (QOS_L3_OCCUP_EVENT_ID ))INIT_DELAYED_WORK (&d->cqm_limbo, cqm_handle_limbo);/*
@@ -4220,7 +4220,7 @@ void resctrl_offline_cpu(unsigned int cpu)cancel_delayed_work (&d->mbm_over);mbm_setup_overflow_handler (d,0 , cpu); } -if (resctrl_arch_is_llc_occupancy_enabled () && +if (resctrl_is_mon_event_enabled (QOS_L3_OCCUP_EVENT_ID ) && cpu == d->cqm_work_cpu &&has_busy_rmid (d)) {cancel_delayed_work (&d->cqm_limbo);cqm_setup_limbo_handler (d,0 , cpu);
Signature · 1 line
2.49.0
Tony Luck
maintainerPATCH 4/5 x86/resctrl: Remove 'rdt_mon_features' global variable
In reply to Tony Luckrdt_mon_features is used as a bitmask of enabled monitor events. A monitor event's status is now maintained in mon_evt::enabled with all monitor events' mon_evt structures found in the filesystem's mon_event_all[] array.
Remove the remaining uses of rdt_mon_features.
--- arch/x86/include/asm/resctrl.h | 1 - arch/x86/kernel/cpu/resctrl/core.c | 9 +++++---- arch/x86/kernel/cpu/resctrl/monitor.c | 5 ----- 3 files changed, 5 insertions(+), 10 deletions(-)
arch/x86/include/asm/resctrl.h +0 -1
@@ -44,7 +44,6 @@ DECLARE_PER_CPU(struct resctrl_pqr_state, pqr_state);extern bool rdt_alloc_capable;extern bool rdt_mon_capable; -extern unsigned int rdt_mon_features;DECLARE_STATIC_KEY_FALSE (rdt_enable_key);DECLARE_STATIC_KEY_FALSE (rdt_alloc_enable_key);
arch/x86/kernel/cpu/resctrl/core.c +5 -4
@@ -863,21 +863,22 @@ static __init bool get_rdt_alloc_resources(void)static __init bool get_rdt_mon_resources (void ) {struct rdt_resource *r = &rdt_resources_all[RDT_RESOURCE_L3 ].r_resctrl; +bool ret = false;if (rdt_cpu_has (X86_FEATURE_CQM_OCCUP_LLC )) {resctrl_enable_mon_event (QOS_L3_OCCUP_EVENT_ID ); - rdt_mon_features |= (1 <<QOS_L3_OCCUP_EVENT_ID ); + ret = true; }if (rdt_cpu_has (X86_FEATURE_CQM_MBM_TOTAL )) {resctrl_enable_mon_event (QOS_L3_MBM_TOTAL_EVENT_ID ); - rdt_mon_features |= (1 <<QOS_L3_MBM_TOTAL_EVENT_ID ); + ret = true; }if (rdt_cpu_has (X86_FEATURE_CQM_MBM_LOCAL )) {resctrl_enable_mon_event (QOS_L3_MBM_LOCAL_EVENT_ID ); - rdt_mon_features |= (1 <<QOS_L3_MBM_LOCAL_EVENT_ID ); + ret = true; } -if (!rdt_mon_features) +if (!ret)return false;return !rdt_get_mon_l3_config (r);
arch/x86/kernel/cpu/resctrl/monitor.c +0 -5
@@ -31,11 +31,6 @@*/ bool rdt_mon_capable; -/* -* Global to indicate which monitoring events are enabled. -*/ -unsigned int rdt_mon_features; -#define CF (cf) ((unsigned long)(1048576 * (cf) + 0.5))static int snc_nodes_per_l3_cache =1 ;
Signature · 1 line
2.49.0
Tony Luck
maintainerPATCH 5/5 x86,fs/resctrl: Prepare for more monitor events
In reply to Tony LuckThere's a rule in computer programming that objects appear zero, once, or many times. So code accordingly.
There are two MBM events and resctrl is coded with a lot of
if (local)
do one thing
if (total)
do a different thing
Change the rdt_mon_domain and rdt_hw_mon_domain structures to hold arrays of pointers to per event data instead of explicit fields for total and local bandwidth.
Simplify by coding for many events using loops on which are enabled.
Move resctrl_is_mbm_event() to <linux/resctrl.h> so it can be used more widely. Also provide a for_each_mbm_event_id() helper macro.
Cleanup variable names in functions touched to consistently use "eventid" for those with type enum resctrl_event_id.
--- include/linux/resctrl.h | 18 +++++++--- include/linux/resctrl_types.h | 3 ++ arch/x86/kernel/cpu/resctrl/internal.h | 9 ++--- arch/x86/kernel/cpu/resctrl/core.c | 38 ++++++++++---------- arch/x86/kernel/cpu/resctrl/monitor.c | 36 +++++++++---------- fs/resctrl/monitor.c | 13 ++++--- fs/resctrl/rdtgroup.c | 48 ++++++++++++-------------- 7 files changed, 88 insertions(+), 77 deletions(-)
include/linux/resctrl.h +14 -4
@@ -161,8 +161,9 @@ struct rdt_ctrl_domain {* @hdr: common header for different domain types * @ci_id: cache info id for this domain * @rmid_busy_llc: bitmap of which limbo RMIDs are above threshold -* @mbm_total: saved state for MBM total bandwidth -* @mbm_local: saved state for MBM local bandwidth +* @mbm_states: Per-event pointer to the MBM event's saved state. +* An MBM event's state is an array of struct mbm_state +* indexed by RMID on x86 or combined CLOSID, RMID on Arm. * @mbm_over: worker to periodically read MBM h/w counters * @cqm_limbo: worker to periodically read CQM h/w counters * @mbm_work_cpu: worker CPU for MBM h/w counters
@@ -172,8 +173,7 @@ struct rdt_mon_domain {struct rdt_domain_hdr hdr;unsigned int ci_id;unsigned long *rmid_busy_llc; -struct mbm_state *mbm_total; -struct mbm_state *mbm_local; +struct mbm_state *mbm_states[QOS_NUM_L3_MBM_EVENTS ];struct delayed_work mbm_over;struct delayed_work cqm_limbo;int mbm_work_cpu;
@@ -376,6 +376,16 @@ bool resctrl_is_mon_event_enabled(enum resctrl_event_id eventid);bool resctrl_arch_is_evt_configurable (enum resctrl_event_id evt); +static inline bool resctrl_is_mbm_event (enum resctrl_event_id eventid) +{ +return (eventid >=QOS_L3_MBM_TOTAL_EVENT_ID && + eventid <=QOS_L3_MBM_LOCAL_EVENT_ID ); +} + +#define for_each_mbm_event_id (eventid) \ + for (eventid = QOS_L3_MBM_TOTAL_EVENT_ID; \ + eventid <= QOS_L3_MBM_LOCAL_EVENT_ID; eventid++) +/** * resctrl_arch_mon_event_config_write() - Write the config for an event. * @config_info: struct resctrl_mon_config_info describing the resource, domain
include/linux/resctrl_types.h +3 -0
@@ -51,4 +51,7 @@ enum resctrl_event_id { QOS_NUM_EVENTS, }; +#define QOS_NUM_L3_MBM_EVENTS (QOS_L3_MBM_LOCAL_EVENT_ID - QOS_L3_MBM_TOTAL_EVENT_ID + 1) +#define MBM_STATE_IDX (evt) ((evt) - QOS_L3_MBM_TOTAL_EVENT_ID) +#endif /* __LINUX_RESCTRL_TYPES_H */
arch/x86/kernel/cpu/resctrl/internal.h +5 -4
@@ -54,15 +54,16 @@ struct rdt_hw_ctrl_domain {* struct rdt_hw_mon_domain - Arch private attributes of a set of CPUs that share * a resource for a monitor function * @d_resctrl: Properties exposed to the resctrl file system -* @arch_mbm_total: arch private state for MBM total bandwidth -* @arch_mbm_local: arch private state for MBM local bandwidth +* @arch_mbm_states: arch private state for each MBM event +* @arch_mbm_states: Per-event pointer to the MBM event's saved state. +* An MBM event's state is an array of struct arch_mbm_state +* indexed by RMID on x86 or combined CLOSID, RMID on Arm. * * Members of this structure are accessed via helpers that provide abstraction. */ struct rdt_hw_mon_domain {struct rdt_mon_domain d_resctrl; -struct arch_mbm_state *arch_mbm_total; -struct arch_mbm_state *arch_mbm_local; +struct arch_mbm_state *arch_mbm_states[QOS_NUM_L3_MBM_EVENTS ]; };static inline struct rdt_hw_ctrl_domain *resctrl_to_arch_ctrl_dom (struct rdt_ctrl_domain *r)
arch/x86/kernel/cpu/resctrl/core.c +20 -18
@@ -365,8 +365,8 @@ static void ctrl_domain_free(struct rdt_hw_ctrl_domain *hw_dom)static void mon_domain_free (struct rdt_hw_mon_domain *hw_dom) { -kfree (hw_dom->arch_mbm_total); -kfree (hw_dom->arch_mbm_local); +for (int i =0 ; i <QOS_NUM_L3_MBM_EVENTS ; i++) +kfree (hw_dom->arch_mbm_states[i]);kfree (hw_dom); }
@@ -400,25 +400,27 @@ static int domain_setup_ctrlval(struct rdt_resource *r, struct rdt_ctrl_domain **/ static int arch_domain_mbm_alloc (u32 num_rmid,struct rdt_hw_mon_domain *hw_dom) { -size_t tsize; - -if (resctrl_is_mon_event_enabled (QOS_L3_MBM_TOTAL_EVENT_ID )) { - tsize =sizeof (*hw_dom->arch_mbm_total); - hw_dom->arch_mbm_total =kcalloc (num_rmid, tsize,GFP_KERNEL ); -if (!hw_dom->arch_mbm_total) -return -ENOMEM ; - } -if (resctrl_is_mon_event_enabled (QOS_L3_MBM_LOCAL_EVENT_ID )) { - tsize =sizeof (*hw_dom->arch_mbm_local); - hw_dom->arch_mbm_local =kcalloc (num_rmid, tsize,GFP_KERNEL ); -if (!hw_dom->arch_mbm_local) { -kfree (hw_dom->arch_mbm_total); - hw_dom->arch_mbm_total =NULL ; -return -ENOMEM ; - } +size_t tsize =sizeof (*hw_dom->arch_mbm_states[0 ]); +enum resctrl_event_id eventid; +int idx; + +for_each_mbm_event_id (eventid) { +if (!resctrl_is_mon_event_enabled (eventid)) +continue ; + idx =MBM_STATE_IDX (eventid); + hw_dom->arch_mbm_states[idx] =kcalloc (num_rmid, tsize,GFP_KERNEL ); +if (!hw_dom->arch_mbm_states[idx]) + goto cleanup; }return 0 ; +cleanup: +while (--idx >=0 ) { +kfree (hw_dom->arch_mbm_states[idx]); + hw_dom->arch_mbm_states[idx] =NULL ; + } + +return -ENOMEM ; }static int get_domain_id_from_scope (int cpu,enum resctrl_scope scope)
arch/x86/kernel/cpu/resctrl/monitor.c +17 -19
@@ -161,18 +161,14 @@ static struct arch_mbm_state *get_arch_mbm_state(struct rdt_hw_mon_domain *hw_dou32 rmid, enum resctrl_event_id eventid) { - switch (eventid) { - case QOS_L3_OCCUP_EVENT_ID: - return NULL; -case QOS_L3_MBM_TOTAL_EVENT_ID : -return &hw_dom->arch_mbm_total[rmid]; -case QOS_L3_MBM_LOCAL_EVENT_ID : -return &hw_dom->arch_mbm_local[rmid]; -default : -/* Never expect to get here */ -WARN_ON_ONCE (1 ); + struct arch_mbm_state *state; + +if (!resctrl_is_mbm_event (eventid))return NULL ; - } + + state = hw_dom->arch_mbm_states[MBM_STATE_IDX (eventid)]; + +return state ? &state[rmid] :NULL ; } void resctrl_arch_reset_rmid(struct rdt_resource *r, struct rdt_mon_domain *d,
@@ -201,14 +197,16 @@ void resctrl_arch_reset_rmid(struct rdt_resource *r, struct rdt_mon_domain *d,void resctrl_arch_reset_rmid_all (struct rdt_resource *r,struct rdt_mon_domain *d) {struct rdt_hw_mon_domain *hw_dom =resctrl_to_arch_mon_dom (d); - -if (resctrl_is_mon_event_enabled (QOS_L3_MBM_TOTAL_EVENT_ID )) -memset (hw_dom->arch_mbm_total,0 , -sizeof (*hw_dom->arch_mbm_total) * r->num_rmid); - -if (resctrl_is_mon_event_enabled (QOS_L3_MBM_LOCAL_EVENT_ID )) -memset (hw_dom->arch_mbm_local,0 , -sizeof (*hw_dom->arch_mbm_local) * r->num_rmid); +enum resctrl_event_id eventid; +int idx; + +for_each_mbm_event_id (eventid) { +if (!resctrl_is_mon_event_enabled (eventid)) +continue ; + idx =MBM_STATE_IDX (eventid); +memset (hw_dom->arch_mbm_states[idx],0 , +sizeof (struct arch_mbm_state ) * r->num_rmid); + } }static u64 mbm_overflow_count (u64 prev_msr,u64 cur_msr,unsigned int width)
fs/resctrl/monitor.c +6 -7
@@ -346,15 +346,14 @@ static struct mbm_state *get_mbm_state(struct rdt_mon_domain *d, u32 closid, u32 rmid, enum resctrl_event_id evtid) { u32 idx = resctrl_arch_rmid_idx_encode(closid, rmid); +struct mbm_state *state; -switch (evtid) { -case QOS_L3_MBM_TOTAL_EVENT_ID : -return &d->mbm_total[idx]; -case QOS_L3_MBM_LOCAL_EVENT_ID : -return &d->mbm_local[idx]; -default : +if (!resctrl_is_mbm_event (evtid))return NULL ; - } + + state = d->mbm_states[MBM_STATE_IDX (evtid)]; + +return state ? &state[idx] :NULL ; }static int __mon_event_count (u32 closid,u32 rmid,struct rmid_read *rr)
fs/resctrl/rdtgroup.c +23 -25
75 lines not on this page · raw message
Signature · 1 line
2.49.0
Moger, Babu
reviewerRe: [PATCH 0/5] Latest version of initial patches in AET series
In reply to Tony LuckHi Tony,
Quoted 4 lines · Tony Luck
Babu,
Here's my latest version of those four patches (build on top of Qinyun's patch that I believe should go upstream via tip/urgent).
Wonderful. Thank you. Will rebase my patches.
Quoted 29 lines
I believe I addressed all the comments[1] in these four patches
[1] https://lore.kernel.org/lkml/[email protected]/
-Tony
Qinyun Tan (1):
x86,fs/resctrl: Remove unappropriate references to cacheinfo in the
resctrl subsystem.Tony Luck (4):
x86,fs/resctrl: Consolidate monitor event descriptions x86,fs/resctrl: Replace architecture event enabled checks x86/resctrl: Remove 'rdt_mon_features' global variable x86,fs/resctrl: Prepare for more monitor eventsinclude/linux/resctrl.h | 28 +++++--- include/linux/resctrl_types.h | 15 ++-- arch/x86/include/asm/resctrl.h | 16 ----- arch/x86/kernel/cpu/resctrl/internal.h | 9 +-- fs/resctrl/internal.h | 17 +++-- arch/x86/kernel/cpu/resctrl/core.c | 65 +++++++++-------- arch/x86/kernel/cpu/resctrl/monitor.c | 41 +++++------ fs/resctrl/ctrlmondata.c | 17 +++-- fs/resctrl/monitor.c | 96 ++++++++++++++------------ fs/resctrl/rdtgroup.c | 79 +++++++++++---------- 10 files changed, 204 insertions(+), 179 deletions(-)
Signature · 2 lines
Thanks Babu Moger
Re: [PATCH v13 11/27] x86/resctrl: Implement resctrl_arch_config_cntr() to assign a counter with ABMC
In reply to Reinette ChatreHi Reinette,
Quoted 9 lines · Reinette Chatre
Hi Babu,
Quoted 4 lines · Babu Moger
The ABMC feature provides an option to the user to assign a hardware counter to an RMID, event pair and monitor the bandwidth as long as it is assigned. The assigned RMID will be tracked by the hardware until the user unassigns it manually.
(please review this often repeated snippet to match new design)
Sure.
Quoted 38 lines
Quoted 7 lines
Implement an architecture-specific handler to assign and unassign the counter. Configure counters by writing to the
L3_QOS_ABMC_CFGMSR, specifying the counter ID, bandwidth source (RMID), and event configuration....
Quoted 28 lines
arch/x86/kernel/cpu/resctrl/monitor.c +20
-0@@ -448,3 +448,40 @@ inline bool resctrl_arch_mbm_cntr_assign_enabled(struct rdt_resource *r) {return resctrl_to_arch_res (r)->mbm_cntr_assign_enabled; } + +static void resctrl_abmc_config_one_amd (void *info) +{ +union l3_qos_abmc_cfg *abmc_cfg = info; + +wrmsrl (MSR_IA32_L3_QOS_ABMC_CFG , abmc_cfg->full); +} + +/* +* Send an IPI to the domain to assign the counter to RMID, event pair. +*/ +void resctrl_arch_config_cntr (struct rdt_resource *r,struct rdt_mon_domain *d, +enum resctrl_event_id evtid,u32 rmid,u32 closid, +u32 cntr_id,u32 evt_cfg,bool assign) +{ +struct rdt_hw_mon_domain *hw_dom =resctrl_to_arch_mon_dom (d); +union l3_qos_abmc_cfg abmc_cfg = {0 }; +struct arch_mbm_state *am; + + abmc_cfg.split.cfg_en =1 ;
Quoted 8 lines
arch/x86/kernel/cpu/resctrl/monitor.c inline bool resctrl_arch_mbm_cntr_assign_enabled(struct rdt_resource *r) · line 474
Quoted 4 lines
arch/x86/kernel/cpu/resctrl/monitor.c +4
-0+ abmc_cfg.split.cntr_en = assign ?1 :0 ; + abmc_cfg.split.cntr_id = cntr_id; + abmc_cfg.split.bw_src = rmid; + abmc_cfg.split.bw_type = evt_cfg;Is
evt_cfgreally needed to be programmed when unassigning a counter? Looking ahead at patch #14resctrl_free_config_cntr()needs to go through extra list walk to get this data but why would hardware need an accurate event configuration to *unassign* a counter?
evt_cfg is not required during unassign. I can remove it.
Quoted 7 lines
It seems unnecessary to provide both the event ID *and* the configuration.
resctrl_arch_config_cntr()could drop the "evt_cfg" parameter and instead there can be a new resctrl utility that architecture can use to query the event's configuration. Similar toresctrl_is_mon_event_enabled()introduced in
https://lore.kernel.org/lkml/[email protected]/ that exposes an event property.
Sounds good.
I can add a new function resctrl_get_mon_event_config(evtid) and call it only during the "assign". It will be called inside resctrl_arch_config_cntr().
Quoted 6 lines
It looks to me as though there are a couple of changes in the telemetry work
that would benefit this work. https://lore.kernel.org/lkml/[email protected]/
switches the monitor events to be maintained in an array indexed by event ID, eliminating the need for searching theevt_listthat this work does in a couple of places. Also note the handy newfor_each_mbm_event()helper (https://lore.kernel.org/lkml/[email protected]/).
Sure. Looking at it now.
Quoted 23 lines
Quoted 23 lines
arch/x86/kernel/cpu/resctrl/monitor.c +13
-0+ +smp_call_function_any (&d->hdr.cpu_mask, resctrl_abmc_config_one_amd, &abmc_cfg,1 ); + +/* +* The hardware counter is reset (because cfg_en == 1) so there is no +* need to record initial non-zero counts. +*/ +if (assign) { + am =get_arch_mbm_state (hw_dom, rmid, evtid); +if (am) +memset (am,0 ,sizeof (*am)); + } +}include/linux/resctrl.h +0
-0@@ -559,6 +559,23 @@ void resctrl_arch_reset_rmid_all(struct rdt_resource *r, struct rdt_mon_domain **/ void resctrl_arch_reset_all_ctrls (struct rdt_resource *r);
Quoted 8 lines
include/linux/resctrl.h void resctrl_arch_reset_rmid_all(struct rdt_resource *r, struct rdt_mon_domain * · line 564
Quoted 3 lines
include/linux/resctrl.h +3
-0+/** +* resctrl_arch_config_cntr() - Configure the counter id to RMID, event +* pair on the domain. The sentence seem strange, should "Configure the counter" perhaps be "Assign the counter"? Or if the naming requires "configure" ... "Configure the counter with its new RMID and event details."? Please feel free to improve.
Last one looks good.
Quoted 5 lines
include/linux/resctrl.h void resctrl_arch_reset_rmid_all(struct rdt_resource *r, struct rdt_mon_domain * · line 566
Quoted 3 lines
include/linux/resctrl.h +2
-0+* @r: Resource structure. +* @d: Domain that the counter id to be configured. I am unable to parse description of @d.
The domain in which the counter ID is to be configured.
Quoted 8 lines
include/linux/resctrl.h void resctrl_arch_reset_rmid_all(struct rdt_resource *r, struct rdt_mon_domain * · line 570
Quoted 5 lines
include/linux/resctrl.h +4
-0+* @evtid: Event type to configure. +* @rmid: RMID to configure. +* @closid: CLOSID to configure. +* @cntr_id: Counter ID to configure. All four parameters descriptions end with "to configure" ... but it is actually only the counter that is configured while the rest is the data that the counter is configured with, no?
Will remove "to configure" from all the other fields except the cntr_id.
Quoted 7 lines
include/linux/resctrl.h void resctrl_arch_reset_rmid_all(struct rdt_resource *r, struct rdt_mon_domain * · line 572
Quoted 3 lines
include/linux/resctrl.h +2
-0+* @evt_cfg: MBM event configuration value representing reads, +* writes etc. Needs definition about what the contents of @evt_cfg means. This is the API ...it cannot be vague like "reads, write, etc." but should be specific about which bit means what.
Copying your comment on other patch
https://lore.kernel.org/lkml/[email protected]/
@evt_cfg: Event configuration created using the READS_TO_LOCAL_MEM, READS_TO_REMOTE_MEM, etc. bits that represent the memory transactions being counted.
Quoted 5 lines
include/linux/resctrl.h void resctrl_arch_reset_rmid_all(struct rdt_resource *r, struct rdt_mon_domain * · line 573
Quoted 2 lines
include/linux/resctrl.h +1
-0+* @assign: Assign or unassign. "True to assign the counter, false to unassign the counter."
Sure.
Quoted 4 lines
Needs some context here about what architecture can expect on how this function will be called. For example, "Can be called from any CPU."
Sure.
Quoted 11 lines
include/linux/resctrl.h void resctrl_arch_reset_rmid_all(struct rdt_resource *r, struct rdt_mon_domain * · line 581
Quoted 8 lines
include/linux/resctrl.h +5
-0+*/ +void resctrl_arch_config_cntr (struct rdt_resource *r,struct rdt_mon_domain *d, +enum resctrl_event_id evtid,u32 rmid,u32 closid, +u32 cntr_id,u32 evt_cfg,bool assign); +extern unsigned int resctrl_rmid_realloc_threshold;extern unsigned int resctrl_rmid_realloc_limit;Reinette
Thanks
Babu
Babu Moger
reviewerPATCH v13 12/27 x86/resctrl: Introduce event configuration modes
In reply to Babu MogerMBM events can be configured using either BMEC (Bandwidth Monitoring Event Configuration) or the mbm_cntr_assign mode.
Introduce a data structure to represent the various event configuration modes and their corresponding values.
---
v13: New patch to handle different event configuration types with
mbm_cntr_assign mode.
--- fs/resctrl/internal.h | 6 ++++-- fs/resctrl/monitor.c | 4 ++-- fs/resctrl/rdtgroup.c | 2 +- include/linux/resctrl_types.h | 11 +++++++++++ 4 files changed, 18 insertions(+), 5 deletions(-)
fs/resctrl/internal.h +4 -2
@@ -55,13 +55,15 @@ static inline struct rdt_fs_context *rdt_fc2context(struct fs_context *fc)* struct mon_evt - Entry in the event list of a resource * @evtid: event id * @name: name of the event -* @configurable: true if the event is configurable +* @mbm_mode: monitoring mode (BMEC or mbm_cntr_assign) +* @evt_cfg: event configuration value decoding reads, writes. * @list: entry in &rdt_resource->evt_list */ struct mon_evt {enum resctrl_event_id evtid;char *name; -bool configurable; +enum resctrl_mbm_mode mbm_mode; +u32 evt_cfg;struct list_head list; };
fs/resctrl/monitor.c +2 -2
@@ -903,12 +903,12 @@ int resctrl_mon_resource_init(void)l3_mon_evt_init (r);if (resctrl_arch_is_evt_configurable (QOS_L3_MBM_TOTAL_EVENT_ID )) { - mbm_total_event.configurable = true; + mbm_total_event.mbm_mode =MBM_MODE_BMEC ;resctrl_file_fflags_init ("mbm_total_bytes_config" ,RFTYPE_MON_INFO |RFTYPE_RES_CACHE ); }if (resctrl_arch_is_evt_configurable (QOS_L3_MBM_LOCAL_EVENT_ID )) { - mbm_local_event.configurable = true; + mbm_local_event.mbm_mode =MBM_MODE_BMEC ;resctrl_file_fflags_init ("mbm_local_bytes_config" ,RFTYPE_MON_INFO |RFTYPE_RES_CACHE ); }
fs/resctrl/rdtgroup.c +1 -1
@@ -1152,7 +1152,7 @@ static int rdt_mon_features_show(struct kernfs_open_file *of,list_for_each_entry (mevt, &r->mon.evt_list, list) {seq_printf (seq,"%s\n" , mevt->name); -if (mevt->configurable) +if (mevt->mbm_mode ==MBM_MODE_BMEC )seq_printf (seq,"%s_config\n" , mevt->name); }
include/linux/resctrl_types.h +11 -0
@@ -47,4 +47,15 @@ enum resctrl_event_id { QOS_NUM_EVENTS, }; +/* +* Event configuration mode. +* Events can be configured either in BMEC (Bandwidth Monitoring Event +* Configuration) mode or mbm_cntr_assign mode. +*/ +enum resctrl_mbm_mode { +MBM_MODE_NONE , +MBM_MODE_BMEC , +MBM_MODE_ASSIGN , +}; +#endif /* __LINUX_RESCTRL_TYPES_H */
Signature · 1 line
2.34.1
Reinette Chatre
maintainerRe: [PATCH v13 12/27] x86/resctrl: Introduce event configuration modes
Hi Babu,
Quoted 7 lines · Babu Moger
MBM events can be configured using either BMEC (Bandwidth Monitoring Event Configuration) or the
mbm_cntr_assignmode.Introduce a data structure to represent the various event configuration modes and their corresponding values.
Suggested-by Reinette Chatre <[email protected]>
I cannot recall suggesting this.
(/me digs)
Are you perhaps referring to https://lore.kernel.org/lkml/[email protected]/
This is not referring to new modes but the existing mbm_cntr_assign modes. resctrl knows which "mbm_cntr_assign" mode is active and it can use that to determine whether BMEC can be exposed to user space or not. There is already enough information in resctrl to know whether BMEC files should be exposed or not.
I think this work self makes clear that these modes are useless since patch #25 that determines whether to hide BMEC files doesn't even use it.
Quoted 20 lines
Signed-off-by Babu Moger <[email protected]>---
v13: New patch to handle different event configuration types with
mbm_cntr_assignmode.--- fs/resctrl/internal.h | 6 ++++-- fs/resctrl/monitor.c | 4 ++-- fs/resctrl/rdtgroup.c | 2 +- include/linux/resctrl_types.h | 11 +++++++++++ 4 files changed, 18 insertions(+), 5 deletions(-)fs/resctrl/internal.h +0
-1@@ -55,13 +55,15 @@ static inline struct rdt_fs_context *rdt_fc2context(struct fs_context *fc)* struct mon_evt - Entry in the event list of a resource * @evtid: event id * @name: name of the event -* @configurable: true if the event is configurable
fs/resctrl/internal.h static inline struct rdt_fs_context *rdt_fc2context(struct fs_context *fc) · line 66
Quoted 10 lines
fs/resctrl/internal.h +4
-1+* @mbm_mode: monitoring mode (BMEC or mbm_cntr_assign) +* @evt_cfg: event configuration value decoding reads, writes. * @list: entry in &rdt_resource->evt_list */ struct mon_evt {enum resctrl_event_id evtid;char *name; -bool configurable; +enum resctrl_mbm_mode mbm_mode; +u32 evt_cfg;
This very important yet totally unrelated member sneaked in without any mention.
Quoted 27 lines
fs/resctrl/internal.h +0
-0struct list_head list; };fs/resctrl/monitor.c +2
-2@@ -903,12 +903,12 @@ int resctrl_mon_resource_init(void)l3_mon_evt_init (r);if (resctrl_arch_is_evt_configurable (QOS_L3_MBM_TOTAL_EVENT_ID )) { - mbm_total_event.configurable = true; + mbm_total_event.mbm_mode =MBM_MODE_BMEC ;resctrl_file_fflags_init ("mbm_total_bytes_config" ,RFTYPE_MON_INFO |RFTYPE_RES_CACHE ); }if (resctrl_arch_is_evt_configurable (QOS_L3_MBM_LOCAL_EVENT_ID )) { - mbm_local_event.configurable = true; + mbm_local_event.mbm_mode =MBM_MODE_BMEC ;resctrl_file_fflags_init ("mbm_local_bytes_config" ,RFTYPE_MON_INFO |RFTYPE_RES_CACHE ); }fs/resctrl/rdtgroup.c +0
-0
fs/resctrl/rdtgroup.c static int rdt_mon_features_show(struct kernfs_open_file *of, · line 1155
Quoted 5 lines
fs/resctrl/rdtgroup.c +1
-1@@ -1152,7 +1152,7 @@ static int rdt_mon_features_show(struct kernfs_open_file *of,list_for_each_entry (mevt, &r->mon.evt_list, list) {seq_printf (seq,"%s\n" , mevt->name); -if (mevt->configurable) +if (mevt->mbm_mode ==MBM_MODE_BMEC )
This can instead be a call to a utility that returns whether BMEC should be visible based on resctrl_mon::mbm_cntr_assignable and rdt_hw_resource::mbm_cntr_assign_enabled (via resctrl_arch_mbm_cntr_assign_enabled() of course).
Quoted 13 lines
fs/resctrl/rdtgroup.c +0
-0seq_printf (seq,"%s_config\n" , mevt->name); }include/linux/resctrl_types.h +2
-0@@ -47,4 +47,15 @@ enum resctrl_event_id { QOS_NUM_EVENTS, }; +/* +* Event configuration mode.
include/linux/resctrl_types.h enum resctrl_event_id { · line 61
Quoted 10 lines
include/linux/resctrl_types.h +9
-0+* Events can be configured either in BMEC (Bandwidth Monitoring Event +* Configuration) mode or mbm_cntr_assign mode. +*/ +enum resctrl_mbm_mode { +MBM_MODE_NONE , +MBM_MODE_BMEC , +MBM_MODE_ASSIGN , +}; +#endif /* __LINUX_RESCTRL_TYPES_H */
Reinette
Moger, Babu
reviewerRe: [PATCH v13 12/27] x86/resctrl: Introduce event configuration modes
Hi Reinette,
Quoted 9 lines · Reinette Chatre
Hi Babu,
Quoted 6 lines · Babu Moger
MBM events can be configured using either BMEC (Bandwidth Monitoring Event Configuration) or the
mbm_cntr_assignmode.Introduce a data structure to represent the various event configuration modes and their corresponding values.
Quoted 7 lines
Quoted 1 line
Suggested-by Reinette Chatre <[email protected]>I cannot recall suggesting this.
(/me digs)
Are you perhaps referring to https://lore.kernel.org/lkml/[email protected]/
Yes.
Quoted 10 lines
This is not referring to new modes but the existing
mbm_cntr_assignmodes. resctrl knows which "mbm_cntr_assign" mode is active and it can use that to determine whether BMEC can be exposed to user space or not. There is already enough information in resctrl to know whether BMEC files should be exposed or not.I think this work self makes clear that these modes are useless since patch #25 that determines whether to hide BMEC files doesn't even use it.
Sure. I will remove change related to mbm_mode.
Quoted 27 lines
Quoted 27 lines
Signed-off-by Babu Moger <[email protected]>---
v13: New patch to handle different event configuration types with
mbm_cntr_assignmode.--- fs/resctrl/internal.h | 6 ++++-- fs/resctrl/monitor.c | 4 ++-- fs/resctrl/rdtgroup.c | 2 +- include/linux/resctrl_types.h | 11 +++++++++++ 4 files changed, 18 insertions(+), 5 deletions(-)fs/resctrl/internal.h +2
-1@@ -55,13 +55,15 @@ static inline struct rdt_fs_context *rdt_fc2context(struct fs_context *fc)* struct mon_evt - Entry in the event list of a resource * @evtid: event id * @name: name of the event -* @configurable: true if the event is configurable +* @mbm_mode: monitoring mode (BMEC or mbm_cntr_assign) +* @evt_cfg: event configuration value decoding reads, writes. * @list: entry in &rdt_resource->evt_list */ struct mon_evt {
Quoted 8 lines
fs/resctrl/internal.h static inline struct rdt_fs_context *rdt_fc2context(struct fs_context *fc) · line 66
Quoted 5 lines
fs/resctrl/internal.h +2
-1enum resctrl_event_id evtid;char *name; -bool configurable; +enum resctrl_mbm_mode mbm_mode; +u32 evt_cfg;This very important yet totally unrelated member sneaked in without any mention.
Yes. We only need evt_cfg. I will just add it as separate patch.
Quoted 29 lines
Quoted 29 lines
fs/resctrl/internal.h +0
-0struct list_head list; };fs/resctrl/monitor.c +2
-2@@ -903,12 +903,12 @@ int resctrl_mon_resource_init(void)l3_mon_evt_init (r);if (resctrl_arch_is_evt_configurable (QOS_L3_MBM_TOTAL_EVENT_ID )) { - mbm_total_event.configurable = true; + mbm_total_event.mbm_mode =MBM_MODE_BMEC ;resctrl_file_fflags_init ("mbm_total_bytes_config" ,RFTYPE_MON_INFO |RFTYPE_RES_CACHE ); }if (resctrl_arch_is_evt_configurable (QOS_L3_MBM_LOCAL_EVENT_ID )) { - mbm_local_event.configurable = true; + mbm_local_event.mbm_mode =MBM_MODE_BMEC ;resctrl_file_fflags_init ("mbm_local_bytes_config" ,RFTYPE_MON_INFO |RFTYPE_RES_CACHE ); }fs/resctrl/rdtgroup.c +0
-0@@ -1152,7 +1152,7 @@ static int rdt_mon_features_show(struct kernfs_open_file *of,
Quoted 8 lines
fs/resctrl/rdtgroup.c static int rdt_mon_features_show(struct kernfs_open_file *of, · line 1155
Quoted 4 lines
fs/resctrl/rdtgroup.c +1
-1list_for_each_entry (mevt, &r->mon.evt_list, list) {seq_printf (seq,"%s\n" , mevt->name); -if (mevt->configurable) +if (mevt->mbm_mode ==MBM_MODE_BMEC )This can instead be a call to a utility that returns whether BMEC should be visible based on
resctrl_mon::mbm_cntr_assignableandrdt_hw_resource::mbm_cntr_assign_enabled(viaresctrl_arch_mbm_cntr_assign_enabled()of course).
Sure. Will do.
Quoted 27 lines
Quoted 14 lines
fs/resctrl/rdtgroup.c +0
-0seq_printf (seq,"%s_config\n" , mevt->name); }include/linux/resctrl_types.h +2
-0@@ -47,4 +47,15 @@ enum resctrl_event_id { QOS_NUM_EVENTS, }; +/* +* Event configuration mode. include/linux/resctrl_types.h enum resctrl_event_id { · line 61
Quoted 10 lines
include/linux/resctrl_types.h +9
-0+* Events can be configured either in BMEC (Bandwidth Monitoring Event +* Configuration) mode or mbm_cntr_assign mode. +*/ +enum resctrl_mbm_mode { +MBM_MODE_NONE , +MBM_MODE_BMEC , +MBM_MODE_ASSIGN , +}; +#endif /* __LINUX_RESCTRL_TYPES_H */ Reinette
Signature · 2 lines
Thanks Babu Moger
Babu Moger
reviewerPATCH v13 13/27 x86/resctrl: Add the functionality to assign MBM events
In reply to Babu MogerThe mbm_cntr_assign mode offers "num_mbm_cntrs" number of counters that can be assigned to RMID, event pair and monitor the bandwidth as long as it is assigned.
Add the functionality to allocate and assign a counter to am RMID, event pair in the domain.
If all the counters are in use, kernel will log the error message "Unable to allocate counter in domain" in /sys/fs/resctrl/info/last_cmd_status when a new assignment is requested. Exit on the first failure when assigning counters across all the domains.
---
v13: Updated changelog.
Changed resctrl_arch_config_cntr() to return void instead of int. Just passing evtid is to resctrl_alloc_config_cntr() and resctrl_assign_cntr_event(). Event configuration value can be easily obtained from mon_evt list.
Introduced new function mbm_get_mon_event() to get event configuration value. Added prototype descriptions to mbm_cntr_get() and mbm_cntr_alloc(). Resolved conflicts caused by the recent FS/ARCH code restructure. The files monitor.c/rdtgroup.c have been split between FS and ARCH directories.
v12: Fixed typo in the subjest line.
Replaced several counters with "num_mbm_cntrs" counters.
Changed the check in resctrl_alloc_config_cntr() to reduce the indentation. Fixed the handling error on first failure.
Added domain id and event id on failure.
Fixed the return error override.
Added new parameter event configuration (evt_cfg) to get the event configuration from user space.
v11: Patch changed again quite a bit.
Moved the functions to monitor.c.
Renamed rdtgroup_assign_cntr_event() to resctrl_assign_cntr_event(). Refactored the resctrl_assign_cntr_event().
Added functionality to exit on the first error during assignment. Simplified mbm_cntr_free().
Removed the function mbm_cntr_assigned(). Will be using mbm_cntr_get() to figure out if the counter is assigned or not.
Updated commit message and code comments.
v10: Patch changed completely.
Counters are managed at the domain based on the discussion.
https://lore.kernel.org/lkml/CALPaoCj+zWq1vkHVbXYP0znJbe6Ke3PXPWjtri5AFgD9cQDCUg@mail.gmail.com/
Reset non-architectural MBM state.
Commit message update.
v9: Introduced new function resctrl_config_cntr to assign the counter, update
the bitmap and reset the architectural state.
Taken care of error handling(freeing the counter) when assignment fails. Moved mbm_cntr_assigned_to_domain here as it used in this patch. Minor text changes.
v8: Renamed rdtgroup_assign_cntr() to rdtgroup_assign_cntr_event().
Added the code to return the error if rdtgroup_assign_cntr_event fails. Moved definition of MBM_EVENT_ARRAY_INDEX to resctrl/internal.h. Updated typo in the comments.
v7: New patch. Moved all the FS code here.
Merged rdtgroup_assign_cntr and rdtgroup_alloc_cntr. Adde new #define MBM_EVENT_ARRAY_INDEX.
--- fs/resctrl/internal.h | 3 + fs/resctrl/monitor.c | 134 ++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 137 insertions(+)
fs/resctrl/internal.h +3 -0
@@ -377,6 +377,9 @@ bool closid_allocated(unsigned int closid);int resctrl_find_cleanest_closid (void ); +int resctrl_assign_cntr_event (struct rdt_resource *r,struct rdt_mon_domain *d, +struct rdtgroup *rdtgrp,enum resctrl_event_id evtid); +#ifdef CONFIG_RESCTRL_FS_PSEUDO_LOCK int rdtgroup_locksetup_enter (struct rdtgroup *rdtgrp);
fs/resctrl/monitor.c +134 -0
137 lines not on this page · raw message
Signature · 1 line
2.34.1
Reinette Chatre
maintainerRe: [PATCH v13 13/27] x86/resctrl: Add the functionality to assign MBM events
Hi Babu,
Quoted 2 lines · Babu Moger
The
mbm_cntr_assignmode offers "num_mbm_cntrs" number of counters that can be assigned to RMID, event pair and monitor the bandwidth as long
"RMID, event pairs"? (assuming at this point in new version it will be obvious what is meant by "event").
Quoted 3 lines
as it is assigned.
Add the functionality to allocate and assign a counter to am RMID, event
"am" -> "an"
Quoted 9 lines
pair in the domain.
If all the counters are in use, kernel will log the error message "Unable to allocate counter in domain" in
/sys/fs/resctrl/info/last_cmd_statuswhen a new assignment is requested. Exit on the first failure when assigning counters across all the domains.Signed-off-by Babu Moger <[email protected]>---
...
Quoted 24 lines
--- fs/resctrl/internal.h | 3 + fs/resctrl/monitor.c | 134 ++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 137 insertions(+)fs/resctrl/internal.h +3
-0@@ -377,6 +377,9 @@ bool closid_allocated(unsigned int closid);int resctrl_find_cleanest_closid (void ); +int resctrl_assign_cntr_event (struct rdt_resource *r,struct rdt_mon_domain *d, +struct rdtgroup *rdtgrp,enum resctrl_event_id evtid); +#ifdef CONFIG_RESCTRL_FS_PSEUDO_LOCK int rdtgroup_locksetup_enter (struct rdtgroup *rdtgrp);fs/resctrl/monitor.c +0
-0
fs/resctrl/monitor.c void resctrl_mon_resource_exit(void) · line 940
Quoted 7 lines
fs/resctrl/monitor.c +4
-0@@ -934,3 +934,137 @@ void resctrl_mon_resource_exit(void)dom_data_exit (r); } + +/* +* Configure the counter for the event, RMID pair for the domain. Reset the +* non-architectural state to clear all the event counters.
clear *all* the event counters?
"Reset the non-architectural state to clear all the event counters." -> "Reset the associated non-architectural state."?
Also, please see https://lore.kernel.org/lkml/[email protected]/
Quoted 7 lines
fs/resctrl/monitor.c +7
-0+*/ +static void resctrl_config_cntr (struct rdt_resource *r,struct rdt_mon_domain *d, +enum resctrl_event_id evtid,u32 rmid,u32 closid, +u32 cntr_id,u32 evt_cfg,bool assign) +{ +struct mbm_state *m; +
fs/resctrl/monitor.c void resctrl_mon_resource_exit(void) · line 957
Quoted 10 lines
fs/resctrl/monitor.c +10
-0+resctrl_arch_config_cntr (r, d, evtid, rmid, closid, cntr_id, evt_cfg, assign); + + m =get_mbm_state (d, closid, rmid, evtid); +if (m) +memset (m,0 ,sizeof (struct mbm_state )); +} + +/* +* mbm_cntr_get() - Return the cntr_id for the matching evtid and rdtgrp in +* cntr_cfg array.
Please prefix parameter names with @ in description to make obvious what is refered to. Although "cntr_id" is a local variable so may be easier to parse if cntr_id is replaced with actual "counter ID" term while keeping rest as actual parameters. That makes cntr_cfg unneeded.
If intending to explain function context then failure return should also be documented. Even better would be to follow typical style of kernel-doc (even if not using /** start) and not mix and match so randomly.
fs/resctrl/monitor.c void resctrl_mon_resource_exit(void) · line 961
Quoted 4 lines
fs/resctrl/monitor.c +4
-0+*/ +static int mbm_cntr_get (struct rdt_resource *r,struct rdt_mon_domain *d, +struct rdtgroup *rdtgrp,enum resctrl_event_id evtid) +{
A subtle issue here is only evident from later patches, for example patch #17, that calls mbm_cntr_get() with a non MBM event ID from __mon_event_count().
If this usage is expected then these utilities needs extra checks to ensure they are only called with valid MBM event IDs.
fs/resctrl/monitor.c void resctrl_mon_resource_exit(void) · line 974
Quoted 13 lines
fs/resctrl/monitor.c +13
-0+int cntr_id; + +for (cntr_id =0 ; cntr_id < r->mon.num_mbm_cntrs; cntr_id++) { +if (d->cntr_cfg[cntr_id].rdtgrp == rdtgrp && + d->cntr_cfg[cntr_id].evtid == evtid) +return cntr_id; + } + +return -ENOENT ; +} + +/* +* mbm_cntr_alloc() - Return the first free entry in cntr_cfg array.
"Return the first ...array." -> "Initilialize and return ID of a new counter, return -ENOSPC on failure." ? This is still an awkward use of kernel-doc ... better to be properly formatted.
Quoted 19 lines
fs/resctrl/monitor.c +19
-0+*/ +static int mbm_cntr_alloc (struct rdt_resource *r,struct rdt_mon_domain *d, +struct rdtgroup *rdtgrp,enum resctrl_event_id evtid) +{ +int cntr_id; + +for (cntr_id =0 ; cntr_id < r->mon.num_mbm_cntrs; cntr_id++) { +if (!d->cntr_cfg[cntr_id].rdtgrp) { + d->cntr_cfg[cntr_id].rdtgrp = rdtgrp; + d->cntr_cfg[cntr_id].evtid = evtid; +return cntr_id; + } + } + +return -ENOSPC ; +} + +/* +* mbm_get_mon_event() - Return the mon_evt entry for the matching evtid.
fs/resctrl/monitor.c void resctrl_mon_resource_exit(void) · line 1003
Quoted 10 lines
fs/resctrl/monitor.c +10
-0+*/ +static struct mon_evt *mbm_get_mon_event (struct rdt_resource *r, +enum resctrl_event_id evtid) +{ +struct mon_evt *mevt; + +list_for_each_entry (mevt, &r->mon.evt_list, list) { +if (mevt->evtid == evtid) +return mevt; + }
With changes from telemetry series this becomes an array lookup.
Quoted 19 lines
fs/resctrl/monitor.c +19
-0+ +return NULL ; +} + +/* +* Allocate a fresh counter and configure the event if not assigned already. +*/ +static int resctrl_alloc_config_cntr (struct rdt_resource *r,struct rdt_mon_domain *d, +struct rdtgroup *rdtgrp,enum resctrl_event_id evtid) +{ +struct mon_evt *mevt; +int cntr_id; + +/* No need to allocate a new counter if it is already assigned */ + cntr_id =mbm_cntr_get (r, d, rdtgrp, evtid); +if (cntr_id >=0 ) + goto cntr_configure; + + cntr_id =mbm_cntr_alloc (r, d, rdtgrp, evtid);
fs/resctrl/monitor.c void resctrl_mon_resource_exit(void) · line 1032
Quoted 10 lines
fs/resctrl/monitor.c +10
-0+if (cntr_id <0 ) { +rdt_last_cmd_printf ("Unable to allocate counter in domain %d\n" , + d->hdr.id); +return cntr_id; + } + +cntr_configure: + mevt =mbm_get_mon_event (r, evtid); +if (!mevt) { +rdt_last_cmd_printf ("Invalid event id %d\n" , evtid);
Difficult to see at this point but it seems that this is in kernel bug territory since user space provided text that is translated to event ID and here translated back to monitor event. This must succeed. Could this be simplified and back-and-forth avoided by passing the mon_evt instead of event ID?
fs/resctrl/monitor.c void resctrl_mon_resource_exit(void) · line 1040
Quoted 11 lines
fs/resctrl/monitor.c +8
-0+return -EINVAL ; + } + +/* +* Skip reconfiguration if the event setup is current; otherwise, +* update and apply the new configuration to the domain. +*/ +if (mevt->evt_cfg != d->cntr_cfg[cntr_id].evt_cfg) {
Lost me. Previous patch silently created mon_event::evt_cfg without initializing it. Here it is compared and treated as the "source of truth" ... where does its value come from?
fs/resctrl/monitor.c void resctrl_mon_resource_exit(void) · line 1052
Quoted 12 lines
fs/resctrl/monitor.c +12
-0+ d->cntr_cfg[cntr_id].evt_cfg = mevt->evt_cfg; +resctrl_config_cntr (r, d, evtid, rdtgrp->mon.rmid, rdtgrp->closid, + cntr_id, mevt->evt_cfg, true); + } + +return 0 ; +} + +/* +* Assign a hardware counter to event @evtid of group @rdtgrp. +* Assign counters to all domains if @d is NULL; otherwise, assign the +* counter to the specified domain @d.
Can add here what is mentioned in changelog that this exits on first failure and so highlight that this can have partial assignment when exit on such failure.
Quoted 8 lines
+ */
fs/resctrl/monitor.c +7
-0+int resctrl_assign_cntr_event (struct rdt_resource *r,struct rdt_mon_domain *d, +struct rdtgroup *rdtgrp,enum resctrl_event_id evtid) +{ +int ret =0 ; + +if (!d) { +list_for_each_entry (d, &r->mon_domains, hdr.list) {
fs/resctrl/monitor.c void resctrl_mon_resource_exit(void) · line 1070
Quoted 10 lines
fs/resctrl/monitor.c +10
-0+ ret =resctrl_alloc_config_cntr (r, d, rdtgrp, evtid); +if (ret) +return ret; + } + }else { + ret =resctrl_alloc_config_cntr (r, d, rdtgrp, evtid); + } + +return ret; +}
Reinette
Moger, Babu
reviewerRe: [PATCH v13 13/27] x86/resctrl: Add the functionality to assign MBM events
Hi Reinette,
Quoted 8 lines · Reinette Chatre
Hi Babu,
Quoted 2 lines · Babu Moger
The
mbm_cntr_assignmode offers "num_mbm_cntrs" number of counters that can be assigned to RMID, event pair and monitor the bandwidth as long"RMID, event pairs"? (assuming at this point in new version it will be obvious what is meant by "event").
Sure.
Quoted 7 lines
Quoted 4 lines
as it is assigned.
Add the functionality to allocate and assign a counter to am RMID, event
"am" -> "an"
sure.
Quoted 46 lines
Quoted 9 lines
pair in the domain.
If all the counters are in use, kernel will log the error message "Unable to allocate counter in domain" in
/sys/fs/resctrl/info/last_cmd_statuswhen a new assignment is requested. Exit on the first failure when assigning counters across all the domains.Signed-off-by Babu Moger <[email protected]>---
...
Quoted 24 lines
--- fs/resctrl/internal.h | 3 + fs/resctrl/monitor.c | 134 ++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 137 insertions(+)fs/resctrl/internal.h +3
-0@@ -377,6 +377,9 @@ bool closid_allocated(unsigned int closid);int resctrl_find_cleanest_closid (void ); +int resctrl_assign_cntr_event (struct rdt_resource *r,struct rdt_mon_domain *d, +struct rdtgroup *rdtgrp,enum resctrl_event_id evtid); +#ifdef CONFIG_RESCTRL_FS_PSEUDO_LOCK int rdtgroup_locksetup_enter (struct rdtgroup *rdtgrp);fs/resctrl/monitor.c +0
-0fs/resctrl/monitor.c void resctrl_mon_resource_exit(void) · line 940
Quoted 7 lines
fs/resctrl/monitor.c +4
-0@@ -934,3 +934,137 @@ void resctrl_mon_resource_exit(void)dom_data_exit (r); } + +/* +* Configure the counter for the event, RMID pair for the domain. Reset the +* non-architectural state to clear all the event counters. clear *all* the event counters?
Quoted 2 lines
"Reset the non-architectural state to clear all the event counters." -> "Reset the associated non-architectural state."?
ok.
Quoted 2 lines
Also, please see https://lore.kernel.org/lkml/[email protected]/
Yes. Sure.
Quoted 15 lines
Quoted 15 lines
fs/resctrl/monitor.c +14
-0+*/ +static void resctrl_config_cntr (struct rdt_resource *r,struct rdt_mon_domain *d, +enum resctrl_event_id evtid,u32 rmid,u32 closid, +u32 cntr_id,u32 evt_cfg,bool assign) +{ +struct mbm_state *m; + +resctrl_arch_config_cntr (r, d, evtid, rmid, closid, cntr_id, evt_cfg, assign); + + m =get_mbm_state (d, closid, rmid, evtid); +if (m) +memset (m,0 ,sizeof (struct mbm_state )); +} +
Quoted 8 lines
fs/resctrl/monitor.c void resctrl_mon_resource_exit(void) · line 957
Quoted 3 lines
fs/resctrl/monitor.c +3
-0+/* +* mbm_cntr_get() - Return the cntr_id for the matching evtid and rdtgrp in +* cntr_cfg array. Please prefix parameter names with @ in description to make obvious what is refered to. Although "
cntr_id" is a local variable so may be easier to parse ifcntr_idis replaced with actual "counter ID" term while keeping rest as actual parameters. That makescntr_cfgunneeded.
Sure.
Quoted 3 lines
If intending to explain function context then failure return should also be documented. Even better would be to follow typical style of kernel-doc (even if not using /** start) and not mix and match so randomly.
Sure.
Quoted 11 lines
fs/resctrl/monitor.c void resctrl_mon_resource_exit(void) · line 961
Quoted 5 lines
fs/resctrl/monitor.c +4
-0+*/ +static int mbm_cntr_get (struct rdt_resource *r,struct rdt_mon_domain *d, +struct rdtgroup *rdtgrp,enum resctrl_event_id evtid) +{A subtle issue here is only evident from later patches, for example patch #17, that calls
mbm_cntr_get()with a non MBM event ID from__mon_event_count().If this usage is expected then these utilities needs extra checks to ensure they are only called with valid MBM event IDs.
Sure. Will add the check resctrl_is_mbm_event().
Quoted 17 lines
Quoted 4 lines
fs/resctrl/monitor.c +3
-0+int cntr_id; + +for (cntr_id =0 ; cntr_id < r->mon.num_mbm_cntrs; cntr_id++) {fs/resctrl/monitor.c void resctrl_mon_resource_exit(void) · line 974
Quoted 10 lines
fs/resctrl/monitor.c +10
-0+if (d->cntr_cfg[cntr_id].rdtgrp == rdtgrp && + d->cntr_cfg[cntr_id].evtid == evtid) +return cntr_id; + } + +return -ENOENT ; +} + +/* +* mbm_cntr_alloc() - Return the first free entry in cntr_cfg array. "Return the first ...array." -> "Initilialize and return ID of a new counter, return
-ENOSPCon failure." ? This is still an awkward use of kernel-doc ... better to be properly formatted.
Sure.
Quoted 24 lines
Quoted 24 lines
fs/resctrl/monitor.c +23
-0+*/ +static int mbm_cntr_alloc (struct rdt_resource *r,struct rdt_mon_domain *d, +struct rdtgroup *rdtgrp,enum resctrl_event_id evtid) +{ +int cntr_id; + +for (cntr_id =0 ; cntr_id < r->mon.num_mbm_cntrs; cntr_id++) { +if (!d->cntr_cfg[cntr_id].rdtgrp) { + d->cntr_cfg[cntr_id].rdtgrp = rdtgrp; + d->cntr_cfg[cntr_id].evtid = evtid; +return cntr_id; + } + } + +return -ENOSPC ; +} + +/* +* mbm_get_mon_event() - Return the mon_evt entry for the matching evtid. +*/ +static struct mon_evt *mbm_get_mon_event (struct rdt_resource *r, +enum resctrl_event_id evtid) +{
Quoted 8 lines
fs/resctrl/monitor.c void resctrl_mon_resource_exit(void) · line 1003
Quoted 6 lines
fs/resctrl/monitor.c +6
-0+struct mon_evt *mevt; + +list_for_each_entry (mevt, &r->mon.evt_list, list) { +if (mevt->evtid == evtid) +return mevt; + }With changes from telemetry series this becomes an array lookup.
Sure. Will look into this.
Quoted 27 lines
Quoted 27 lines
fs/resctrl/monitor.c +26
-0+ +return NULL ; +} + +/* +* Allocate a fresh counter and configure the event if not assigned already. +*/ +static int resctrl_alloc_config_cntr (struct rdt_resource *r,struct rdt_mon_domain *d, +struct rdtgroup *rdtgrp,enum resctrl_event_id evtid) +{ +struct mon_evt *mevt; +int cntr_id; + +/* No need to allocate a new counter if it is already assigned */ + cntr_id =mbm_cntr_get (r, d, rdtgrp, evtid); +if (cntr_id >=0 ) + goto cntr_configure; + + cntr_id =mbm_cntr_alloc (r, d, rdtgrp, evtid); +if (cntr_id <0 ) { +rdt_last_cmd_printf ("Unable to allocate counter in domain %d\n" , + d->hdr.id); +return cntr_id; + } + +cntr_configure:
Quoted 8 lines
fs/resctrl/monitor.c void resctrl_mon_resource_exit(void) · line 1032
Quoted 3 lines
fs/resctrl/monitor.c +3
-0+ mevt =mbm_get_mon_event (r, evtid); +if (!mevt) { +rdt_last_cmd_printf ("Invalid event id %d\n" , evtid);Difficult to see at this point but it seems that this is in kernel bug territory since user space provided text that is translated to event ID and here translated back to monitor event. This must succeed. Could this be simplified and back-and-forth avoided by passing the
mon_evtinstead of event ID?
We can do that.
Quoted 6 lines
Quoted 3 lines
fs/resctrl/monitor.c +2
-0+return -EINVAL ; + }
Quoted 10 lines
fs/resctrl/monitor.c void resctrl_mon_resource_exit(void) · line 1040
Quoted 6 lines
fs/resctrl/monitor.c +6
-0+ +/* +* Skip reconfiguration if the event setup is current; otherwise, +* update and apply the new configuration to the domain. +*/ +if (mevt->evt_cfg != d->cntr_cfg[cntr_id].evt_cfg) {Lost me. Previous patch silently created
mon_event::evt_cfgwithout initializing it. Here it is compared and treated as the "source of truth" ... where does its value come from?
Yes. That is correct. Will have to initialize evt_cfg when it is first introduced. Will do.
Quoted 16 lines
fs/resctrl/monitor.c void resctrl_mon_resource_exit(void) · line 1052
Quoted 13 lines
fs/resctrl/monitor.c +12
-0+ d->cntr_cfg[cntr_id].evt_cfg = mevt->evt_cfg; +resctrl_config_cntr (r, d, evtid, rdtgrp->mon.rmid, rdtgrp->closid, + cntr_id, mevt->evt_cfg, true); + } + +return 0 ; +} + +/* +* Assign a hardware counter to event @evtid of group @rdtgrp. +* Assign counters to all domains if @d is NULL; otherwise, assign the +* counter to the specified domain @d. Can add here what is mentioned in changelog that this exits on first failure and so highlight that this can have partial assignment when exit on such failure.
Sure.
Quoted 22 lines
Quoted 9 lines
fs/resctrl/monitor.c +8
-0+*/ +int resctrl_assign_cntr_event (struct rdt_resource *r,struct rdt_mon_domain *d, +struct rdtgroup *rdtgrp,enum resctrl_event_id evtid) +{ +int ret =0 ; + +if (!d) { +list_for_each_entry (d, &r->mon_domains, hdr.list) {fs/resctrl/monitor.c void resctrl_mon_resource_exit(void) · line 1070
Quoted 10 lines
fs/resctrl/monitor.c +10
-0+ ret =resctrl_alloc_config_cntr (r, d, rdtgrp, evtid); +if (ret) +return ret; + } + }else { + ret =resctrl_alloc_config_cntr (r, d, rdtgrp, evtid); + } + +return ret; +}Reinette
Signature · 2 lines
Thanks Babu Moger
Babu Moger
reviewerPATCH v13 14/27 x86/resctrl: Add the functionality to unassign MBM events
In reply to Babu MogerThe mbm_cntr_assign mode offers "num_mbm_cntrs" number of counters that can be assigned to an RMID, event pair and monitor the bandwidth as long as it is assigned. If all the counters are in use, the kernel will log the error message "Unable to allocate counter in domain" in /sys/fs/resctrl/info/last_cmd_status when a new assignment is requested.
To make space for a new assignment, users must unassign an already assigned counter and retry the assignment again.
Add the functionality to unassign and free the counters in the domain.
---
v13: Moved mbm_cntr_free() to this patch as it is used in here first.
Not required to pass evt_cfg to resctrl_unassign_cntr_event(). It is available via mbm_get_mon_event().
Resolved conflicts caused by the recent FS/ARCH code restructure. The monitor.c file has now been split between the FS and ARCH directories.
v12: Updated the commit text to make bit more clear.
Replaced several counters with "num_mbm_cntrs" counters.
Fixed typo in the subjest line.
Fixed the handling error on first failure.
Added domain id and event id on failure.
Added new parameter event configuration (evt_cfg) to provide the event from user space.
v11: Moved the functions to monitor.c.
Renamed rdtgroup_unassign_cntr_event() to resctrl_unassign_cntr_event(). Refactored the resctrl_unassign_cntr_event().
Updated commit message and code comments.
v10: Patch changed again.
Counters are managed at the domain based on the discussion.
https://lore.kernel.org/lkml/CALPaoCj+zWq1vkHVbXYP0znJbe6Ke3PXPWjtri5AFgD9cQDCUg@mail.gmail.com/
commit message update.
v9: Changes related to addition of new function resctrl_config_cntr().
The removed rdtgroup_mbm_cntr_is_assigned() as it was introduced already.
Text changes to take care comments.
v8: Renamed rdtgroup_mbm_cntr_is_assigned to mbm_cntr_assigned_to_domain
Added return error handling in resctrl_arch_config_cntr().
v7: Merged rdtgroup_unassign_cntr and rdtgroup_free_cntr functions.
Renamed rdtgroup_mbm_cntr_test() to rdtgroup_mbm_cntr_is_assigned(). Reworded the commit log little bit.
v6: Removed mbm_cntr_free from this patch.
Added counter test in all the domains and free if it is not assigned to any domains.
v5: Few name changes to match cntr_id.
Changed the function names to rdtgroup_unassign_cntr More comments on commit log.
v4: Added domain specific unassign feature.
Few name changes.
v3: Removed the static from the prototype of rdtgroup_unassign_abmc.
The function is not called directly from user anymore. These changes are related to global assignment interface.
v2: No changes.
--- fs/resctrl/internal.h | 2 ++ fs/resctrl/monitor.c | 60 +++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 62 insertions(+)
fs/resctrl/internal.h +2 -0
@@ -379,6 +379,8 @@ int resctrl_find_cleanest_closid(void);int resctrl_assign_cntr_event (struct rdt_resource *r,struct rdt_mon_domain *d,struct rdtgroup *rdtgrp,enum resctrl_event_id evtid); +int resctrl_unassign_cntr_event (struct rdt_resource *r,struct rdt_mon_domain *d, +struct rdtgroup *rdtgrp,enum resctrl_event_id evtid);#ifdef CONFIG_RESCTRL_FS_PSEUDO_LOCK int rdtgroup_locksetup_enter (struct rdtgroup *rdtgrp);
fs/resctrl/monitor.c +60 -0
69 lines not on this page · raw message
Signature · 1 line
2.34.1
Reinette Chatre
maintainerRe: [PATCH v13 14/27] x86/resctrl: Add the functionality to unassign MBM events
Hi Babu,
Quoted 13 lines · Babu Moger
The
mbm_cntr_assignmode offers "num_mbm_cntrs" number of counters that can be assigned to an RMID, event pair and monitor the bandwidth as long as it is assigned. If all the counters are in use, the kernel will log the error message "Unable to allocate counter in domain" in/sys/fs/resctrl/info/last_cmd_statuswhen a new assignment is requested.To make space for a new assignment, users must unassign an already assigned counter and retry the assignment again.
Add the functionality to unassign and free the counters in the domain.
Signed-off-by Babu Moger <[email protected]>---
...
Quoted 23 lines
--- fs/resctrl/internal.h | 2 ++ fs/resctrl/monitor.c | 60 +++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 62 insertions(+)fs/resctrl/internal.h +2
-0@@ -379,6 +379,8 @@ int resctrl_find_cleanest_closid(void);int resctrl_assign_cntr_event (struct rdt_resource *r,struct rdt_mon_domain *d,struct rdtgroup *rdtgrp,enum resctrl_event_id evtid); +int resctrl_unassign_cntr_event (struct rdt_resource *r,struct rdt_mon_domain *d, +struct rdtgroup *rdtgrp,enum resctrl_event_id evtid);#ifdef CONFIG_RESCTRL_FS_PSEUDO_LOCK int rdtgroup_locksetup_enter (struct rdtgroup *rdtgrp);fs/resctrl/monitor.c +0
-0
fs/resctrl/monitor.c static int mbm_cntr_alloc(struct rdt_resource *r, struct rdt_mon_domain *d, · line 993
Quoted 5 lines
fs/resctrl/monitor.c +2
-0@@ -989,6 +989,14 @@ static int mbm_cntr_alloc(struct rdt_resource *r, struct rdt_mon_domain *d,return -ENOSPC ; } +/* +* mbm_cntr_free() - Reset cntr_id to zero.
"Reset cntr_id to zero"? cntr_id is an index to an array. Please provide accurate and useful descriptions.
Quoted 22 lines
fs/resctrl/monitor.c +15
-0+*/ +static void mbm_cntr_free (struct rdt_mon_domain *d,int cntr_id) +{ +memset (&d->cntr_cfg[cntr_id],0 ,sizeof (struct mbm_cntr_cfg )); +} +/* * mbm_get_mon_event() - Return the mon_evt entry for the matching evtid. */ @@ -1068,3 +1076,55 @@ int resctrl_assign_cntr_event(struct rdt_resource *r, struct rdt_mon_domain *d,return ret; } + +/* +* Unassign and free the counter if assigned. +*/ +static int resctrl_free_config_cntr (struct rdt_resource *r,struct rdt_mon_domain *d, +struct rdtgroup *rdtgrp,enum resctrl_event_id evtid) +{ +struct mon_evt *mevt; +int cntr_id;
fs/resctrl/monitor.c int resctrl_assign_cntr_event(struct rdt_resource *r, struct rdt_mon_domain *d, · line 1097
Quoted 10 lines
fs/resctrl/monitor.c +10
-0+ + cntr_id =mbm_cntr_get (r, d, rdtgrp, evtid); + +/* If there is no cntr_id assigned, nothing to do */ +if (cntr_id <0 ) +return 0 ; + + mevt =mbm_get_mon_event (r, evtid); +if (!mevt) { +rdt_last_cmd_printf ("Invalid event id %d\n" , evtid);
Similar to previous comment this is in kernel bug territory and could be simplified by passing mon_evt instead. Although this is the unassign portion where evt_cfg seems unnecessary.
Quoted 23 lines
fs/resctrl/monitor.c +23
-0+return -EINVAL ; + } + +resctrl_config_cntr (r, d, evtid, rdtgrp->mon.rmid, rdtgrp->closid, + cntr_id, mevt->evt_cfg, false); + +mbm_cntr_free (d, cntr_id); + +return 0 ; +} + +/* +* Unassign a hardware counter associated with @evtid from the domain and +* the group. Unassign the counters from all the domains if @d is NULL else +* unassign from @d. +*/ +int resctrl_unassign_cntr_event (struct rdt_resource *r,struct rdt_mon_domain *d, +struct rdtgroup *rdtgrp,enum resctrl_event_id evtid) +{ +int ret; + +if (!d) { +list_for_each_entry (d, &r->mon_domains, hdr.list) {
fs/resctrl/monitor.c int resctrl_assign_cntr_event(struct rdt_resource *r, struct rdt_mon_domain *d, · line 1130
Quoted 10 lines
fs/resctrl/monitor.c +10
-0+ ret =resctrl_free_config_cntr (r, d, rdtgrp, evtid); +if (ret) +return ret; + } + }else { + ret =resctrl_free_config_cntr (r, d, rdtgrp, evtid); + } + +return ret; +}
Reinette
Moger, Babu
reviewerRe: [PATCH v13 14/27] x86/resctrl: Add the functionality to unassign MBM events
Hi Reinette,
Quoted 44 lines · Reinette Chatre
Hi Babu,
Quoted 13 lines · Babu Moger
The
mbm_cntr_assignmode offers "num_mbm_cntrs" number of counters that can be assigned to an RMID, event pair and monitor the bandwidth as long as it is assigned. If all the counters are in use, the kernel will log the error message "Unable to allocate counter in domain" in/sys/fs/resctrl/info/last_cmd_statuswhen a new assignment is requested.To make space for a new assignment, users must unassign an already assigned counter and retry the assignment again.
Add the functionality to unassign and free the counters in the domain.
Signed-off-by Babu Moger <[email protected]>---
...
Quoted 26 lines
--- fs/resctrl/internal.h | 2 ++ fs/resctrl/monitor.c | 60 +++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 62 insertions(+)fs/resctrl/internal.h +2
-0@@ -379,6 +379,8 @@ int resctrl_find_cleanest_closid(void);int resctrl_assign_cntr_event (struct rdt_resource *r,struct rdt_mon_domain *d,struct rdtgroup *rdtgrp,enum resctrl_event_id evtid); +int resctrl_unassign_cntr_event (struct rdt_resource *r,struct rdt_mon_domain *d, +struct rdtgroup *rdtgrp,enum resctrl_event_id evtid);#ifdef CONFIG_RESCTRL_FS_PSEUDO_LOCK int rdtgroup_locksetup_enter (struct rdtgroup *rdtgrp);fs/resctrl/monitor.c +0
-0@@ -989,6 +989,14 @@ static int mbm_cntr_alloc(struct rdt_resource *r, struct rdt_mon_domain *d,return -ENOSPC ; }
Quoted 5 lines
fs/resctrl/monitor.c static int mbm_cntr_alloc(struct rdt_resource *r, struct rdt_mon_domain *d, · line 993
Quoted 2 lines
fs/resctrl/monitor.c +2
-0+/* +* mbm_cntr_free() - Reset cntr_id to zero. "Reset
cntr_idto zero"?cntr_idis an index to an array. Please provide accurate and useful descriptions.
Yes. My bad. Will correct it.
Quoted 29 lines
Quoted 29 lines
fs/resctrl/monitor.c +21
-0+*/ +static void mbm_cntr_free (struct rdt_mon_domain *d,int cntr_id) +{ +memset (&d->cntr_cfg[cntr_id],0 ,sizeof (struct mbm_cntr_cfg )); +} +/* * mbm_get_mon_event() - Return the mon_evt entry for the matching evtid. */ @@ -1068,3 +1076,55 @@ int resctrl_assign_cntr_event(struct rdt_resource *r, struct rdt_mon_domain *d,return ret; } + +/* +* Unassign and free the counter if assigned. +*/ +static int resctrl_free_config_cntr (struct rdt_resource *r,struct rdt_mon_domain *d, +struct rdtgroup *rdtgrp,enum resctrl_event_id evtid) +{ +struct mon_evt *mevt; +int cntr_id; + + cntr_id =mbm_cntr_get (r, d, rdtgrp, evtid); + +/* If there is no cntr_id assigned, nothing to do */ +if (cntr_id <0 ) +return 0 ;
Quoted 8 lines
fs/resctrl/monitor.c int resctrl_assign_cntr_event(struct rdt_resource *r, struct rdt_mon_domain *d, · line 1097
Quoted 4 lines
fs/resctrl/monitor.c +4
-0+ + mevt =mbm_get_mon_event (r, evtid); +if (!mevt) { +rdt_last_cmd_printf ("Invalid event id %d\n" , evtid);Similar to previous comment this is in kernel bug territory and could be simplified by passing
mon_evtinstead. Although this is the unassign portion whereevt_cfgseems unnecessary.
Sure. The call mbm_get_mon_event() is not required anymore in this path.
Quoted 37 lines
Quoted 24 lines
fs/resctrl/monitor.c +23
-0+return -EINVAL ; + } + +resctrl_config_cntr (r, d, evtid, rdtgrp->mon.rmid, rdtgrp->closid, + cntr_id, mevt->evt_cfg, false); + +mbm_cntr_free (d, cntr_id); + +return 0 ; +} + +/* +* Unassign a hardware counter associated with @evtid from the domain and +* the group. Unassign the counters from all the domains if @d is NULL else +* unassign from @d. +*/ +int resctrl_unassign_cntr_event (struct rdt_resource *r,struct rdt_mon_domain *d, +struct rdtgroup *rdtgrp,enum resctrl_event_id evtid) +{ +int ret; + +if (!d) { +list_for_each_entry (d, &r->mon_domains, hdr.list) {fs/resctrl/monitor.c int resctrl_assign_cntr_event(struct rdt_resource *r, struct rdt_mon_domain *d, · line 1130
Quoted 10 lines
fs/resctrl/monitor.c +10
-0+ ret =resctrl_free_config_cntr (r, d, rdtgrp, evtid); +if (ret) +return ret; + } + }else { + ret =resctrl_free_config_cntr (r, d, rdtgrp, evtid); + } + +return ret; +}Reinette
Signature · 2 lines
Thanks Babu Moger
Babu Moger
reviewerPATCH v13 15/27 x86/resctrl: Report 'Unassigned' for MBM events in mbm_cntr_assign mode
In reply to Babu MogerIn mbm_cntr_assign mode, the hardware counter should be assigned to read the MBM events.
Report 'Unassigned' in case the user attempts to read the event without assigning a hardware counter.
Export resctrl_is_mbm_event() and mbm_cntr_get() to allow usage from other functions within fs/resctrl.
---
v13: Minor commitlog and user doc update.
Resolved conflicts caused by the recent FS/ARCH code restructure.
The monitor.c/rdtgroup.c files have been split between the FS and ARCH directories.
v12: Updated the documentation for more clarity.
v11: Domain can be NULL with SNC support so moved the unassign check in
rdtgroup_mondata_show().
v10: Moved the code to check the assign state inside mon_event_read().
Fixed few text comments.
v9: Used is_mbm_event() to check the event type.
Minor user documentation update.
v8: Used MBM_EVENT_ARRAY_INDEX to get the index for the MBM event.
Documentation update to make the text generic.
v7: Moved the documentation under "mon_data".
Updated the text little bit.
v6: Added more explaination in the resctrl.rst
Added checks to detect "Unassigned" before reading RMID.
v5: New patch.
--- Documentation/filesystems/resctrl.rst | 8 ++++++++ fs/resctrl/ctrlmondata.c | 14 ++++++++++++++ fs/resctrl/internal.h | 2 ++ fs/resctrl/monitor.c | 4 ++-- fs/resctrl/rdtgroup.c | 2 +- include/linux/resctrl.h | 1 + 6 files changed, 28 insertions(+), 3 deletions(-)
Documentation/filesystems/resctrl.rst +8 -0
@@ -430,6 +430,14 @@ When monitoring is enabled all MON groups will also contain: for the L3 cache they occupy). These are named "mon_sub_L3_YY" where "YY" is the node number. + The mbm_cntr_assign mode offers "num_mbm_cntrs" number of counters + and allows users to assign a counter to mon_hw_id, event pair enabling + bandwidth monitoring for as long as the counter remains assigned. + The hardware will continue tracking the assigned mon_hw_id until + the user manually unassigns it, ensuring that counters are not reset + during this period. An MBM event returns 'Unassigned' when the event + does not have a hardware counter assigned. + "mon_hw_id": Available only with debug option. The identifier used by hardware for the monitor group. On x86 this is the RMID.
fs/resctrl/ctrlmondata.c +14 -0
@@ -643,6 +643,18 @@ int rdtgroup_mondata_show(struct seq_file *m, void *arg) goto out; } d = container_of(hdr, struct rdt_mon_domain, hdr); + +/* +* Report 'Unassigned' if mbm_cntr_assign mode is enabled and +* counter is unassigned. +*/ +if (resctrl_arch_mbm_cntr_assign_enabled (r) && +resctrl_is_mbm_event (evtid) && + (mbm_cntr_get (r, d, rdtgrp, evtid) <0 )) { + rr.err = -ENOENT ; + goto checkresult; + } +mon_event_read (&rr, r, d, rdtgrp, &d->hdr.cpu_mask, evtid, false); }
@@ -652,6 +664,8 @@ int rdtgroup_mondata_show(struct seq_file *m, void *arg)seq_puts (m,"Error\n" ); else if (rr.err == -EINVAL) seq_puts(m, "Unavailable\n"); + else if (rr.err == -ENOENT) + seq_puts(m, "Unassigned\n"); else seq_printf(m, "%llu\n", rr.val);
fs/resctrl/internal.h +2 -0
@@ -381,6 +381,8 @@ int resctrl_assign_cntr_event(struct rdt_resource *r, struct rdt_mon_domain *d, struct rdtgroup *rdtgrp, enum resctrl_event_id evtid);int resctrl_unassign_cntr_event (struct rdt_resource *r,struct rdt_mon_domain *d,struct rdtgroup *rdtgrp,enum resctrl_event_id evtid); +int mbm_cntr_get (struct rdt_resource *r,struct rdt_mon_domain *d, +struct rdtgroup *rdtgrp,enum resctrl_event_id evtid);#ifdef CONFIG_RESCTRL_FS_PSEUDO_LOCK int rdtgroup_locksetup_enter (struct rdtgroup *rdtgrp);
fs/resctrl/monitor.c +2 -2
@@ -956,8 +956,8 @@ static void resctrl_config_cntr(struct rdt_resource *r, struct rdt_mon_domain *d* mbm_cntr_get() - Return the cntr_id for the matching evtid and rdtgrp in * cntr_cfg array. */ -static int mbm_cntr_get (struct rdt_resource *r,struct rdt_mon_domain *d, -struct rdtgroup *rdtgrp,enum resctrl_event_id evtid) +int mbm_cntr_get (struct rdt_resource *r,struct rdt_mon_domain *d, +struct rdtgroup *rdtgrp,enum resctrl_event_id evtid) {int cntr_id;
fs/resctrl/rdtgroup.c +1 -1
@@ -127,7 +127,7 @@ static bool resctrl_is_mbm_enabled(void) resctrl_arch_is_mbm_local_enabled()); } -static bool resctrl_is_mbm_event (int e) +bool resctrl_is_mbm_event (int e) {return (e >=QOS_L3_MBM_TOTAL_EVENT_ID && e <=QOS_L3_MBM_LOCAL_EVENT_ID );
include/linux/resctrl.h +1 -0
@@ -441,6 +441,7 @@ static inline u32 resctrl_get_config_index(u32 closid, } } +bool resctrl_is_mbm_event (int e);bool resctrl_arch_get_cdp_enabled (enum resctrl_res_level l);int resctrl_arch_set_cdp_enabled (enum resctrl_res_level l,bool enable);
Signature · 1 line
2.34.1
Reinette Chatre
maintainerRe: [PATCH v13 15/27] x86/resctrl: Report 'Unassigned' for MBM events in mbm_cntr_assign mode
Hi Babu,
Quoted 8 lines · Babu Moger
In
mbm_cntr_assignmode, the hardware counter should be assigned to read the MBM events.Report 'Unassigned' in case the user attempts to read the event without assigning a hardware counter.
Export
resctrl_is_mbm_event()andmbm_cntr_get()to allow usage from other functions within fs/resctrl.
Please clarify that these two functions are exposed differently, resctrl_is_mbm_event() is added to include/linux/resctrl.h (also note similar change in
https://lore.kernel.org/lkml/[email protected]/)
so not just exposed to fs/resctrl but instead to resctrl fs as well as arch code
while mbm_cntr_get() remains internal to resctrl fs by being added to fs/resctrl/internal.h.
Quoted 19 lines
Signed-off-by Babu Moger <[email protected]>---
--- Documentation/filesystems/resctrl.rst | 8 ++++++++ fs/resctrl/ctrlmondata.c | 14 ++++++++++++++ fs/resctrl/internal.h | 2 ++ fs/resctrl/monitor.c | 4 ++-- fs/resctrl/rdtgroup.c | 2 +- include/linux/resctrl.h | 1 + 6 files changed, 28 insertions(+), 3 deletions(-)Documentation/filesystems/resctrl.rst +0
-0
Documentation/filesystems/resctrl.rst When monitoring is enabled all MON groups will also contain: · line 439
Quoted 10 lines
Documentation/filesystems/resctrl.rst +7
-0@@ -430,6 +430,14 @@ When monitoring is enabled all MON groups will also contain: for the L3 cache they occupy). These are named "mon_sub_L3_YY" where "YY" is the node number. + The mbm_cntr_assign mode offers "num_mbm_cntrs" number of counters + and allows users to assign a counter to mon_hw_id, event pair enabling + bandwidth monitoring for as long as the counter remains assigned. + The hardware will continue tracking the assigned mon_hw_id until + the user manually unassigns it, ensuring that counters are not reset + during this period. An MBM event returns 'Unassigned' when the event + does not have a hardware counter assigned.
(please rework based on "event" vs "group" assignment ... not intending that "group" assignment be documented but the "event" assignment needs to be accurate for "group" assignment to be a simple extension)
Quoted 28 lines
Documentation/filesystems/resctrl.rst +1
-0+ "mon_hw_id": Available only with debug option. The identifier used by hardware for the monitor group. On x86 this is the RMID.fs/resctrl/ctrlmondata.c +12
-0@@ -643,6 +643,18 @@ int rdtgroup_mondata_show(struct seq_file *m, void *arg) goto out; } d = container_of(hdr, struct rdt_mon_domain, hdr); + +/* +* Report 'Unassigned' if mbm_cntr_assign mode is enabled and +* counter is unassigned. +*/ +if (resctrl_arch_mbm_cntr_assign_enabled (r) && +resctrl_is_mbm_event (evtid) && + (mbm_cntr_get (r, d, rdtgrp, evtid) <0 )) { + rr.err = -ENOENT ; + goto checkresult; + } +mon_event_read (&rr, r, d, rdtgrp, &d->hdr.cpu_mask, evtid, false); }
fs/resctrl/ctrlmondata.c int rdtgroup_mondata_show(struct seq_file *m, void *arg) · line 671
Quoted 8 lines
fs/resctrl/ctrlmondata.c +2
-0@@ -652,6 +664,8 @@ int rdtgroup_mondata_show(struct seq_file *m, void *arg)seq_puts (m,"Error\n" ); else if (rr.err == -EINVAL) seq_puts(m, "Unavailable\n"); + else if (rr.err == -ENOENT) + seq_puts(m, "Unassigned\n"); else seq_printf(m, "%llu\n", rr.val);
It may be unexpected that this is treated as "-ENOENT" but the function returns success. This can be addressed with a comment when comparing the return codes to other hardware return codes.
Quoted 46 lines
fs/resctrl/internal.h +2
-0@@ -381,6 +381,8 @@ int resctrl_assign_cntr_event(struct rdt_resource *r, struct rdt_mon_domain *d, struct rdtgroup *rdtgrp, enum resctrl_event_id evtid);int resctrl_unassign_cntr_event (struct rdt_resource *r,struct rdt_mon_domain *d,struct rdtgroup *rdtgrp,enum resctrl_event_id evtid); +int mbm_cntr_get (struct rdt_resource *r,struct rdt_mon_domain *d, +struct rdtgroup *rdtgrp,enum resctrl_event_id evtid);#ifdef CONFIG_RESCTRL_FS_PSEUDO_LOCK int rdtgroup_locksetup_enter (struct rdtgroup *rdtgrp);fs/resctrl/monitor.c +2
-2@@ -956,8 +956,8 @@ static void resctrl_config_cntr(struct rdt_resource *r, struct rdt_mon_domain *d* mbm_cntr_get() - Return the cntr_id for the matching evtid and rdtgrp in * cntr_cfg array. */ -static int mbm_cntr_get (struct rdt_resource *r,struct rdt_mon_domain *d, -struct rdtgroup *rdtgrp,enum resctrl_event_id evtid) +int mbm_cntr_get (struct rdt_resource *r,struct rdt_mon_domain *d, +struct rdtgroup *rdtgrp,enum resctrl_event_id evtid) {int cntr_id;fs/resctrl/rdtgroup.c +1
-1@@ -127,7 +127,7 @@ static bool resctrl_is_mbm_enabled(void) resctrl_arch_is_mbm_local_enabled()); } -static bool resctrl_is_mbm_event (int e) +bool resctrl_is_mbm_event (int e) {return (e >=QOS_L3_MBM_TOTAL_EVENT_ID && e <=QOS_L3_MBM_LOCAL_EVENT_ID );include/linux/resctrl.h +0
-0
include/linux/resctrl.h static inline u32 resctrl_get_config_index(u32 closid, · line 447
Quoted 7 lines
include/linux/resctrl.h +1
-0@@ -441,6 +441,7 @@ static inline u32 resctrl_get_config_index(u32 closid, } } +bool resctrl_is_mbm_event (int e);bool resctrl_arch_get_cdp_enabled (enum resctrl_res_level l);int resctrl_arch_set_cdp_enabled (enum resctrl_res_level l,bool enable);
Reinette
Moger, Babu
reviewerRe: [PATCH v13 15/27] x86/resctrl: Report 'Unassigned' for MBM events in mbm_cntr_assign mode
Hi Reinette,
Quoted 12 lines · Reinette Chatre
Hi Babu,
Quoted 8 lines · Babu Moger
In
mbm_cntr_assignmode, the hardware counter should be assigned to read the MBM events.Report 'Unassigned' in case the user attempts to read the event without assigning a hardware counter.
Export
resctrl_is_mbm_event()andmbm_cntr_get()to allow usage from other functions within fs/resctrl.
Quoted 7 lines
Please clarify that these two functions are exposed differently,
resctrl_is_mbm_event()is added toinclude/linux/resctrl.h(also note similar change in
https://lore.kernel.org/lkml/[email protected]/)
so not just exposed to fs/resctrl but instead to resctrl fs as well as arch code
whilembm_cntr_get()remains internal to resctrl fs by being added tofs/resctrl/internal.h.
Sure. Will update the comment.
With Tony's changes(
https://lore.kernel.org/lkml/[email protected]/),
the resctrl_is_mbm_event() is not required in here. It is already there.
I will have to update the comment only on mbm_cntr_get(). Will do.
Quoted 22 lines
Quoted 22 lines
Signed-off-by Babu Moger <[email protected]>---
--- Documentation/filesystems/resctrl.rst | 8 ++++++++ fs/resctrl/ctrlmondata.c | 14 ++++++++++++++ fs/resctrl/internal.h | 2 ++ fs/resctrl/monitor.c | 4 ++-- fs/resctrl/rdtgroup.c | 2 +- include/linux/resctrl.h | 1 + 6 files changed, 28 insertions(+), 3 deletions(-)Documentation/filesystems/resctrl.rst +0
-0@@ -430,6 +430,14 @@ When monitoring is enabled all MON groups will also contain: for the L3 cache they occupy). These are named "mon_sub_L3_YY" where "YY" is the node number.
Quoted 11 lines
Documentation/filesystems/resctrl.rst When monitoring is enabled all MON groups will also contain: · line 439
Quoted 7 lines
Documentation/filesystems/resctrl.rst +7
-0+ The mbm_cntr_assign mode offers "num_mbm_cntrs" number of counters + and allows users to assign a counter to mon_hw_id, event pair enabling + bandwidth monitoring for as long as the counter remains assigned. + The hardware will continue tracking the assigned mon_hw_id until + the user manually unassigns it, ensuring that counters are not reset + during this period. An MBM event returns 'Unassigned' when the event + does not have a hardware counter assigned.(please rework based on "event" vs "group" assignment ... not intending that "group" assignment be documented but the "event" assignment needs to be accurate for "group" assignment to be a simple extension)
Sure.
Quoted 28 lines
Quoted 28 lines
Documentation/filesystems/resctrl.rst +1
-0+ "mon_hw_id": Available only with debug option. The identifier used by hardware for the monitor group. On x86 this is the RMID.fs/resctrl/ctrlmondata.c +12
-0@@ -643,6 +643,18 @@ int rdtgroup_mondata_show(struct seq_file *m, void *arg) goto out; } d = container_of(hdr, struct rdt_mon_domain, hdr); + +/* +* Report 'Unassigned' if mbm_cntr_assign mode is enabled and +* counter is unassigned. +*/ +if (resctrl_arch_mbm_cntr_assign_enabled (r) && +resctrl_is_mbm_event (evtid) && + (mbm_cntr_get (r, d, rdtgrp, evtid) <0 )) { + rr.err = -ENOENT ; + goto checkresult; + } +mon_event_read (&rr, r, d, rdtgrp, &d->hdr.cpu_mask, evtid, false); }
Quoted 13 lines
fs/resctrl/ctrlmondata.c int rdtgroup_mondata_show(struct seq_file *m, void *arg) · line 671
Quoted 9 lines
fs/resctrl/ctrlmondata.c +2
-0@@ -652,6 +664,8 @@ int rdtgroup_mondata_show(struct seq_file *m, void *arg)seq_puts (m,"Error\n" ); else if (rr.err == -EINVAL) seq_puts(m, "Unavailable\n"); + else if (rr.err == -ENOENT) + seq_puts(m, "Unassigned\n"); else seq_printf(m, "%llu\n", rr.val);It may be unexpected that this is treated as "
-ENOENT" but the function returns success. This can be addressed with a comment when comparing the return codes to other hardware return codes.
Will add the comment.
Quoted 58 lines
Quoted 47 lines
fs/resctrl/internal.h +2
-0@@ -381,6 +381,8 @@ int resctrl_assign_cntr_event(struct rdt_resource *r, struct rdt_mon_domain *d, struct rdtgroup *rdtgrp, enum resctrl_event_id evtid);int resctrl_unassign_cntr_event (struct rdt_resource *r,struct rdt_mon_domain *d,struct rdtgroup *rdtgrp,enum resctrl_event_id evtid); +int mbm_cntr_get (struct rdt_resource *r,struct rdt_mon_domain *d, +struct rdtgroup *rdtgrp,enum resctrl_event_id evtid);#ifdef CONFIG_RESCTRL_FS_PSEUDO_LOCK int rdtgroup_locksetup_enter (struct rdtgroup *rdtgrp);fs/resctrl/monitor.c +2
-2@@ -956,8 +956,8 @@ static void resctrl_config_cntr(struct rdt_resource *r, struct rdt_mon_domain *d* mbm_cntr_get() - Return the cntr_id for the matching evtid and rdtgrp in * cntr_cfg array. */ -static int mbm_cntr_get (struct rdt_resource *r,struct rdt_mon_domain *d, -struct rdtgroup *rdtgrp,enum resctrl_event_id evtid) +int mbm_cntr_get (struct rdt_resource *r,struct rdt_mon_domain *d, +struct rdtgroup *rdtgrp,enum resctrl_event_id evtid) {int cntr_id;fs/resctrl/rdtgroup.c +1
-1@@ -127,7 +127,7 @@ static bool resctrl_is_mbm_enabled(void) resctrl_arch_is_mbm_local_enabled()); } -static bool resctrl_is_mbm_event (int e) +bool resctrl_is_mbm_event (int e) {return (e >=QOS_L3_MBM_TOTAL_EVENT_ID && e <=QOS_L3_MBM_LOCAL_EVENT_ID );include/linux/resctrl.h +0
-0include/linux/resctrl.h static inline u32 resctrl_get_config_index(u32 closid, · line 447
Quoted 7 lines
include/linux/resctrl.h +1
-0@@ -441,6 +441,7 @@ static inline u32 resctrl_get_config_index(u32 closid, } } +bool resctrl_is_mbm_event (int e);bool resctrl_arch_get_cdp_enabled (enum resctrl_res_level l);int resctrl_arch_set_cdp_enabled (enum resctrl_res_level l,bool enable);Reinette
Signature · 2 lines
Thanks Babu Moger
Babu Moger
reviewerPATCH v13 16/27 x86/resctrl: Pass entire struct rdtgroup rather than passing individual members
In reply to Babu MogerThe mbm_cntr_assign mode requires a cntr_id to read event data. The cntr_id is retrieved via mbm_cntr_get, which takes a struct rdtgroup as a parameter.
Passing the full rdtgroup also provides access to closid and rmid, both of which are necessary to read monitoring events.
Refactor the code to pass the entire struct rdtgroup instead of individual members in preparation for this requirement.
---
v13: New patch to pass the entire struct rdtgroup to __mon_event_count(),
mbm_update(), and related functions.
--- fs/resctrl/monitor.c | 29 ++++++++++++++++------------- 1 file changed, 16 insertions(+), 13 deletions(-)
fs/resctrl/monitor.c +16 -13
@@ -357,9 +357,11 @@ static struct mbm_state *get_mbm_state(struct rdt_mon_domain *d, u32 closid, } } -static int __mon_event_count (u32 closid,u32 rmid,struct rmid_read *rr) +static int __mon_event_count (struct rdtgroup *rdtgrp,struct rmid_read *rr) {int cpu =smp_processor_id (); +u32 closid = rdtgrp->closid; +u32 rmid = rdtgrp->mon.rmid;struct rdt_mon_domain *d;struct mbm_state *m;int err, ret;
@@ -428,9 +430,11 @@ static int __mon_event_count(u32 closid, u32 rmid, struct rmid_read *rr)* __mon_event_count() is compared with the chunks value from the previous * invocation. This must be called once per second to maintain values in MBps. */ -static void mbm_bw_count (u32 closid,u32 rmid,struct rmid_read *rr) +static void mbm_bw_count (struct rdtgroup *rdtgrp,struct rmid_read *rr) {u64 cur_bw, bytes, cur_bytes; +u32 closid = rdtgrp->closid; +u32 rmid = rdtgrp->mon.rmid;struct mbm_state *m; m =get_mbm_state (rr->d, closid, rmid, rr->evtid);
@@ -459,7 +463,7 @@ void mon_event_count(void *info) rdtgrp = rr->rgrp; - ret =__mon_event_count (rdtgrp->closid, rdtgrp->mon.rmid, rr); + ret =__mon_event_count (rdtgrp, rr);/* * For Ctrl groups read data from child monitor groups and
@@ -470,8 +474,7 @@ void mon_event_count(void *info)if (rdtgrp->type ==RDTCTRL_GROUP ) {list_for_each_entry (entry, head, mon.crdtgrp_list) { -if (__mon_event_count (entry->closid, entry->mon.rmid, - rr) ==0 ) +if (__mon_event_count (entry, rr) ==0 ) ret =0 ; } }
@@ -602,7 +605,7 @@ static void update_mba_bw(struct rdtgroup *rgrp, struct rdt_mon_domain *dom_mbm) }static void mbm_update_one_event (struct rdt_resource *r,struct rdt_mon_domain *d, -u32 closid,u32 rmid,enum resctrl_event_id evtid) +struct rdtgroup *rdtgrp,enum resctrl_event_id evtid) {struct rmid_read rr = {0 };
@@ -616,30 +619,30 @@ static void mbm_update_one_event(struct rdt_resource *r, struct rdt_mon_domain *return ; } -__mon_event_count (closid, rmid, &rr); +__mon_event_count (rdtgrp, &rr);/* * If the software controller is enabled, compute the * bandwidth for this event id. */ if (is_mba_sc (NULL )) -mbm_bw_count (closid, rmid, &rr); +mbm_bw_count (rdtgrp, &rr);resctrl_arch_mon_ctx_free (rr.r, rr.evtid, rr.arch_mon_ctx); }static void mbm_update (struct rdt_resource *r,struct rdt_mon_domain *d, -u32 closid,u32 rmid) +struct rdtgroup *rdtgrp) {/* * This is protected from concurrent reads from user as both * the user and overflow handler hold the global mutex. */ if (resctrl_arch_is_mbm_total_enabled ()) -mbm_update_one_event (r, d, closid, rmid,QOS_L3_MBM_TOTAL_EVENT_ID ); +mbm_update_one_event (r, d, rdtgrp,QOS_L3_MBM_TOTAL_EVENT_ID );if (resctrl_arch_is_mbm_local_enabled ()) -mbm_update_one_event (r, d, closid, rmid,QOS_L3_MBM_LOCAL_EVENT_ID ); +mbm_update_one_event (r, d, rdtgrp,QOS_L3_MBM_LOCAL_EVENT_ID ); }/*
@@ -712,11 +715,11 @@ void mbm_handle_overflow(struct work_struct *work) d = container_of(work, struct rdt_mon_domain, mbm_over.work);list_for_each_entry (prgrp, &rdt_all_groups, rdtgroup_list) { -mbm_update (r, d, prgrp->closid, prgrp->mon.rmid); +mbm_update (r, d, prgrp); head = &prgrp->mon.crdtgrp_list;list_for_each_entry (crgrp, head, mon.crdtgrp_list) -mbm_update (r, d, crgrp->closid, crgrp->mon.rmid); +mbm_update (r, d, crgrp);if (is_mba_sc (NULL ))update_mba_bw (prgrp, d);
Signature · 1 line
2.34.1
Reinette Chatre
maintainerRe: [PATCH v13 16/27] x86/resctrl: Pass entire struct rdtgroup rather than passing individual members
Hi Babu,
Quoted 1 line · Babu Moger
The
mbm_cntr_assignmode requires acntr_idto read event data. The
cntr_id -> "counter ID"
Quoted 1 line
cntr_idis retrieved viambm_cntr_get, which takes a struct rdtgroup as
cntr_id -> "counter ID"
mbm_cntr_get -> mbm_cntr_get()
Quoted 3 lines
a parameter.
Passing the full rdtgroup also provides access to closid and rmid, both of
closid -> CLOSID
rmid -> RMID
Quoted 3 lines
which are necessary to read monitoring events.
Refactor the code to pass the entire struct rdtgroup instead of individual
"the entire" -> "a pointer to"
Quoted 5 lines
members in preparation for this requirement.
Suggested-by Reinette Chatre <[email protected]>Signed-off-by Babu Moger <[email protected]>---
Patch looks good.
Reinette
Moger, Babu
reviewerRe: [PATCH v13 16/27] x86/resctrl: Pass entire struct rdtgroup rather than passing individual members
Hi Reinette,
Quoted 7 lines · Reinette Chatre
Hi Babu,
Quoted 1 line · Babu Moger
The
mbm_cntr_assignmode requires acntr_idto read event data. The
cntr_id-> "counter ID"
Sure.
Quoted 4 lines
Quoted 1 line
cntr_idis retrieved viambm_cntr_get, which takes a struct rdtgroup as
cntr_id-> "counter ID"
Sure.
Quoted 2 lines
mbm_cntr_get->mbm_cntr_get()
Sure.
Quoted 7 lines
Quoted 3 lines
a parameter.
Passing the full rdtgroup also provides access to closid and rmid, both of
closid -> CLOSID
rmid -> RMID
Sure.
Quoted 6 lines
Quoted 3 lines
which are necessary to read monitoring events.
Refactor the code to pass the entire struct rdtgroup instead of individual
"the entire" -> "a pointer to"
Sure.
Quoted 6 lines
Quoted 5 lines
members in preparation for this requirement.
Suggested-by Reinette Chatre <[email protected]>Signed-off-by Babu Moger <[email protected]>---
Patch looks good.--
Thanks
Babu Moger
Babu Moger
reviewerPATCH v13 17/27 x86/resctrl: Add the support for reading ABMC counters
In reply to Babu MogerSoftware can read the assignable counters using the QM_EVTSEL and QM_CTR register pair.
QM_EVTSEL Register definition:
=======================================================
Bits Mnemonic Description
=======================================================
63:44 -- Reserved
43:32 RMID Resource Monitoring Identifier
31 ExtEvtID Extended Event Identifier
30:8 -- Reserved
7:0 EvtID Event Identifier
=======================================================
The contents of a specific counter can be read by setting the following fields in QM_EVTSEL.ExtendedEvtID = 1, QM_EVTSEL.EvtID = L3CacheABMC (=1) and setting [RMID] to the desired counter ID. Reading QM_CTR will then return the contents of the specified counter. The E bit will be set if the counter configuration was invalid, or if an invalid counter ID was set in the QM_EVTSEL[RMID] field.
Introduce __cntr_id_read_phys() to read the counter ID event data.
---
v13: Split the patch into 2. First one to handle the passing of rdtgroup structure to few
functions( __mon_event_count and mbm_update(). Second one to handle ABMC counter reading. Added new function __cntr_id_read_phys() to handle ABMC event reading. Updated kernel doc for resctrl_arch_reset_rmid() and resctrl_arch_rmid_read(). Resolved conflicts caused by the recent FS/ARCH code restructure.
The monitor.c file has now been split between the FS and ARCH directories.
v12: New patch to support extended event mode when ABMC is enabled.
--- arch/x86/kernel/cpu/resctrl/internal.h | 6 +++ arch/x86/kernel/cpu/resctrl/monitor.c | 66 ++++++++++++++++++++++---- fs/resctrl/monitor.c | 14 ++++-- include/linux/resctrl.h | 9 ++-- 4 files changed, 80 insertions(+), 15 deletions(-)
arch/x86/kernel/cpu/resctrl/internal.h +6 -0
@@ -40,6 +40,12 @@ struct arch_mbm_state {/* Setting bit 0 in L3_QOS_EXT_CFG enables the ABMC feature. */ #define ABMC_ENABLE_BIT 0 +/* +* ABMC Qos Event Identifiers. +*/ +#define ABMC_EXTENDED_EVT_ID BIT(31) +#define ABMC_EVT_ID 1 +/** * struct rdt_hw_ctrl_domain - Arch private attributes of a set of CPUs that share * a resource for a control function
arch/x86/kernel/cpu/resctrl/monitor.c +58 -8
96 lines not on this page · raw message
fs/resctrl/monitor.c +10 -4
@@ -159,7 +159,11 @@ void __check_limbo(struct rdt_mon_domain *d, bool force_free)break ; entry =__rmid_entry (idx); -if (resctrl_arch_rmid_read (r, d, entry->closid, entry->rmid, +/* +* cntr_id is not relevant for QOS_L3_OCCUP_EVENT_ID. +* Pass dummy value -1. +*/ +if (resctrl_arch_rmid_read (r, d, entry->closid, entry->rmid,-1 ,QOS_L3_OCCUP_EVENT_ID , &val, arch_mon_ctx)) { rmid_dirty = true;
@@ -359,6 +363,7 @@ static struct mbm_state *get_mbm_state(struct rdt_mon_domain *d, u32 closid,static int __mon_event_count (struct rdtgroup *rdtgrp,struct rmid_read *rr) { +int cntr_id =mbm_cntr_get (rr->r, rr->d, rdtgrp, rr->evtid);int cpu =smp_processor_id ();u32 closid = rdtgrp->closid;u32 rmid = rdtgrp->mon.rmid;
@@ -368,7 +373,7 @@ static int __mon_event_count(struct rdtgroup *rdtgrp, struct rmid_read *rr)u64 tval =0 ;if (rr->first) { -resctrl_arch_reset_rmid (rr->r, rr->d, closid, rmid, rr->evtid); +resctrl_arch_reset_rmid (rr->r, rr->d, closid, rmid, cntr_id, rr->evtid); m =get_mbm_state (rr->d, closid, rmid, rr->evtid);if (m)memset (m,0 ,sizeof (struct mbm_state ));
@@ -379,7 +384,7 @@ static int __mon_event_count(struct rdtgroup *rdtgrp, struct rmid_read *rr)/* Reading a single domain, must be on a CPU in that domain. */ if (!cpumask_test_cpu (cpu, &rr->d->hdr.cpu_mask))return -EINVAL ; - rr->err =resctrl_arch_rmid_read (rr->r, rr->d, closid, rmid, + rr->err =resctrl_arch_rmid_read (rr->r, rr->d, closid, rmid, cntr_id, rr->evtid, &tval, rr->arch_mon_ctx);if (rr->err)return rr->err;
@@ -404,7 +409,8 @@ static int __mon_event_count(struct rdtgroup *rdtgrp, struct rmid_read *rr)list_for_each_entry (d, &rr->r->mon_domains, hdr.list) {if (d->ci->id != rr->ci->id)continue ; - err =resctrl_arch_rmid_read (rr->r, d, closid, rmid, + cntr_id =mbm_cntr_get (rr->r, d, rdtgrp, rr->evtid); + err =resctrl_arch_rmid_read (rr->r, d, closid, rmid, cntr_id, rr->evtid, &tval, rr->arch_mon_ctx);if (!err) { rr->val += tval;
include/linux/resctrl.h +6 -3
@@ -473,6 +473,7 @@ void resctrl_offline_cpu(unsigned int cpu);* counter may match traffic of both @closid and @rmid, or @rmid * only. * @rmid: rmid of the counter to read. +* @cntr_id: cntr_id to read MBM events with mbm_cntr_assign mode. * @eventid: eventid to read, e.g. L3 occupancy. * @val: result of the counter read in bytes. * @arch_mon_ctx: An architecture specific value from
@@ -490,8 +491,9 @@ void resctrl_offline_cpu(unsigned int cpu);* 0 on success, or -EIO, -EINVAL etc on error. */ int resctrl_arch_rmid_read (struct rdt_resource *r,struct rdt_mon_domain *d, -u32 closid,u32 rmid,enum resctrl_event_id eventid, -u64 *val,void *arch_mon_ctx); +u32 closid,u32 rmid,int cntr_id, +enum resctrl_event_id eventid,u64 *val, +void *arch_mon_ctx);/** * resctrl_arch_rmid_read_context_check() - warn about invalid contexts
@@ -532,12 +534,13 @@ struct rdt_domain_hdr *resctrl_find_domain(struct list_head *h, int id,* @closid: closid that matches the rmid. Depending on the architecture, the * counter may match traffic of both @closid and @rmid, or @rmid only. * @rmid: The rmid whose counter values should be reset. +* @cntr_id: The cntr_id to read MBM events with mbm_cntr_assign mode. * @eventid: The eventid whose counter values should be reset. * * This can be called from any CPU. */ void resctrl_arch_reset_rmid (struct rdt_resource *r,struct rdt_mon_domain *d, -u32 closid,u32 rmid, +u32 closid,u32 rmid,int cntr_id,enum resctrl_event_id eventid);/**
Signature · 1 line
2.34.1
Reinette Chatre
maintainerRe: [PATCH v13 17/27] x86/resctrl: Add the support for reading ABMC counters
Hi Babu,
Quoted 2 lines · Babu Moger
Software can read the assignable counters using the
QM_EVTSELandQM_CTRregister pair.
Please append with more context on how register pair is used to support the changes in this patch.
Quoted 12 lines
QM_EVTSELRegister definition:
=======================================================
Bits Mnemonic Description
=======================================================
63:44 -- Reserved
43:32 RMID Resource Monitoring Identifier
31 ExtEvtID Extended Event Identifier
30:8 -- Reserved
7:0 EvtID Event Identifier
=======================================================
Quoted 6 lines
The contents of a specific counter can be read by setting the following fields in QM_EVTSEL.ExtendedEvtID = 1, QM_EVTSEL.EvtID = L3CacheABMC (=1) and setting [RMID] to the desired counter ID. Reading
QM_CTRwill then return the contents of the specified counter. The E bit will be set if the counter configuration was invalid, or if an invalid counter ID was set in the QM_EVTSEL[RMID] field.
Please rewrite above in imperative tone.
Quoted 26 lines
Introduce
__cntr_id_read_phys()to read the counter ID event data.Link https://www.amd.com/content/dam/amd/en/documents/processor-tech-docs/programmer-references/40332.pdfSigned-off-by Babu Moger <[email protected]>---
v13: Split the patch into 2. First one to handle the passing of rdtgroup structure to few
functions(__mon_event_countandmbm_update(). Second one to handle ABMC counter reading. Added new function__cntr_id_read_phys()to handle ABMC event reading. Updated kernel doc forresctrl_arch_reset_rmid()andresctrl_arch_rmid_read(). Resolved conflicts caused by the recent FS/ARCH code restructure.
Themonitor.cfile has now been split between the FS and ARCH directories.v12: New patch to support extended event mode when ABMC is enabled.
--- arch/x86/kernel/cpu/resctrl/internal.h | 6 +++ arch/x86/kernel/cpu/resctrl/monitor.c | 66 ++++++++++++++++++++++---- fs/resctrl/monitor.c | 14 ++++-- include/linux/resctrl.h | 9 ++-- 4 files changed, 80 insertions(+), 15 deletions(-)arch/x86/kernel/cpu/resctrl/internal.h +0
-0
arch/x86/kernel/cpu/resctrl/internal.h struct arch_mbm_state { · line 44
Quoted 5 lines
arch/x86/kernel/cpu/resctrl/internal.h +2
-0@@ -40,6 +40,12 @@ struct arch_mbm_state {/* Setting bit 0 in L3_QOS_EXT_CFG enables the ABMC feature. */ #define ABMC_ENABLE_BIT 0 +/* +* ABMC Qos Event Identifiers.
QoS?
arch/x86/kernel/cpu/resctrl/internal.h struct arch_mbm_state { · line 47
Quoted 3 lines
arch/x86/kernel/cpu/resctrl/internal.h +3
-0+*/ +#define ABMC_EXTENDED_EVT_ID BIT(31) +#define ABMC_EVT_ID 1
Please use BIT(0) to be consistent.
Quoted 60 lines
arch/x86/kernel/cpu/resctrl/internal.h +1
-0+/** * struct rdt_hw_ctrl_domain - Arch private attributes of a set of CPUs that share * a resource for a control function arch/x86/kernel/cpu/resctrl/monitor.c +36
-1@@ -161,6 +161,41 @@ static int __rmid_read_phys(u32 prmid, enum resctrl_event_id eventid, u64 *val)return 0 ; } +static int __cntr_id_read_phys (u32 cntr_id,u64 *val) +{ +u64 msr_val; + +/* +* QM_EVTSEL Register definition: +* ======================================================= +* Bits Mnemonic Description +* ======================================================= +* 63:44 -- Reserved +* 43:32 RMID Resource Monitoring Identifier +* 31 ExtEvtID Extended Event Identifier +* 30:8 -- Reserved +* 7:0 EvtID Event Identifier +* ======================================================= +* The contents of a specific counter can be read by setting the +* following fields in QM_EVTSEL.ExtendedEvtID(=1) and +* QM_EVTSEL.EvtID = L3CacheABMC (=1) and setting [RMID] to the +* desired counter ID. Reading QM_CTR will then return the +* contents of the specified counter. The E bit will be set if the +* counter configuration was invalid, or if an invalid counter ID +* was set in the QM_EVTSEL[RMID] field. +*/ +wrmsr (MSR_IA32_QM_EVTSEL ,ABMC_EXTENDED_EVT_ID |ABMC_EVT_ID , cntr_id); +rdmsrl (MSR_IA32_QM_CTR , msr_val); + +if (msr_val &RMID_VAL_ERROR ) +return -EIO ; +if (msr_val &RMID_VAL_UNAVAIL ) +return -EINVAL ; + + *val = msr_val; +return 0 ; +} +static struct arch_mbm_state *get_arch_mbm_state (struct rdt_hw_mon_domain *hw_dom,u32 rmid,enum resctrl_event_id eventid)@@ -180,7 +215,7 @@ static struct arch_mbm_state *get_arch_mbm_state(struct rdt_hw_mon_domain *hw_do }void resctrl_arch_reset_rmid (struct rdt_resource *r,struct rdt_mon_domain *d, -u32 unused,u32 rmid, +u32 unused,u32 rmid,int cntr_id,enum resctrl_event_id eventid) {struct rdt_hw_mon_domain *hw_dom =resctrl_to_arch_mon_dom (d);
arch/x86/kernel/cpu/resctrl/monitor.c void resctrl_arch_reset_rmid(struct rdt_resource *r, struct rdt_mon_domain *d, · line 232
Quoted 9 lines
arch/x86/kernel/cpu/resctrl/monitor.c +3
-3@@ -192,9 +227,16 @@ void resctrl_arch_reset_rmid(struct rdt_resource *r, struct rdt_mon_domain *d,if (am) {memset (am,0 ,sizeof (*am)); - prmid =logical_rmid_to_physical_rmid (cpu, rmid); -/* Record any initial, non-zero count value. */ -__rmid_read_phys (prmid, eventid, &am->prev_msr); +if (resctrl_arch_mbm_cntr_assign_enabled (r) && +resctrl_is_mbm_event (eventid)) { +if (cntr_id <0 )
This would be a bug, no? how about WARN_ON_ONCE()?
Quoted 22 lines
arch/x86/kernel/cpu/resctrl/monitor.c +9
-2+return ; +__cntr_id_read_phys (cntr_id, &am->prev_msr); + }else { + prmid =logical_rmid_to_physical_rmid (cpu, rmid); +/* Record any initial, non-zero count value. */ +__rmid_read_phys (prmid, eventid, &am->prev_msr); + } } }@@ -224,8 +266,8 @@ static u64 mbm_overflow_count(u64 prev_msr, u64 cur_msr, unsigned int width) }int resctrl_arch_rmid_read (struct rdt_resource *r,struct rdt_mon_domain *d, -u32 unused,u32 rmid,enum resctrl_event_id eventid, -u64 *val,void *ignored) +u32 unused,u32 rmid,int cntr_id, +enum resctrl_event_id eventid,u64 *val,void *ignored) {struct rdt_hw_mon_domain *hw_dom =resctrl_to_arch_mon_dom (d);struct rdt_hw_resource *hw_res =resctrl_to_arch_res (r);
arch/x86/kernel/cpu/resctrl/monitor.c int resctrl_arch_rmid_read(struct rdt_resource *r, struct rdt_mon_domain *d, · line 284
Quoted 8 lines
arch/x86/kernel/cpu/resctrl/monitor.c +3
-2@@ -237,8 +279,16 @@ int resctrl_arch_rmid_read(struct rdt_resource *r, struct rdt_mon_domain *d,resctrl_arch_rmid_read_context_check (); - prmid =logical_rmid_to_physical_rmid (cpu, rmid); - ret =__rmid_read_phys (prmid, eventid, &msr_val); +if (resctrl_arch_mbm_cntr_assign_enabled (r) && +resctrl_is_mbm_event (eventid)) { +if (cntr_id <0 )
WARN_ON_ONCE()?
Quoted 28 lines
arch/x86/kernel/cpu/resctrl/monitor.c +7
-0+return cntr_id; + ret =__cntr_id_read_phys (cntr_id, &msr_val); + }else { + prmid =logical_rmid_to_physical_rmid (cpu, rmid); + ret =__rmid_read_phys (prmid, eventid, &msr_val); + } +if (ret)return ret;fs/resctrl/monitor.c +5
-1@@ -159,7 +159,11 @@ void __check_limbo(struct rdt_mon_domain *d, bool force_free)break ; entry =__rmid_entry (idx); -if (resctrl_arch_rmid_read (r, d, entry->closid, entry->rmid, +/* +* cntr_id is not relevant for QOS_L3_OCCUP_EVENT_ID. +* Pass dummy value -1. +*/ +if (resctrl_arch_rmid_read (r, d, entry->closid, entry->rmid,-1 ,QOS_L3_OCCUP_EVENT_ID , &val, arch_mon_ctx)) { rmid_dirty = true;
fs/resctrl/monitor.c static struct mbm_state *get_mbm_state(struct rdt_mon_domain *d, u32 closid, · line 366
Quoted 4 lines
fs/resctrl/monitor.c +1
-0@@ -359,6 +363,7 @@ static struct mbm_state *get_mbm_state(struct rdt_mon_domain *d, u32 closid,static int __mon_event_count (struct rdtgroup *rdtgrp,struct rmid_read *rr) { +int cntr_id =mbm_cntr_get (rr->r, rr->d, rdtgrp, rr->evtid);
So mbm_cntr_get() is called on *all* events (even non MBM) whether assignable counters are supported or not. I assume it relies on num_mbm_cntrs to be zero on non-ABMC systems but I think this needs to be explicit that mbm_cntr_get() returns -ENOENT in these cases. Any developer attempting to modify mbm_cntr_get() needs to be aware of this usage.
This is quite subtle that resctrl_arch_reset_rmid() and resctrl_arch_rmid_read() can be called with a negative counter ID. To help with code health this needs to be highlighted (more later).
Quoted 36 lines
fs/resctrl/monitor.c +4
-3int cpu =smp_processor_id ();u32 closid = rdtgrp->closid;u32 rmid = rdtgrp->mon.rmid;@@ -368,7 +373,7 @@ static int __mon_event_count(struct rdtgroup *rdtgrp, struct rmid_read *rr)u64 tval =0 ;if (rr->first) { -resctrl_arch_reset_rmid (rr->r, rr->d, closid, rmid, rr->evtid); +resctrl_arch_reset_rmid (rr->r, rr->d, closid, rmid, cntr_id, rr->evtid); m =get_mbm_state (rr->d, closid, rmid, rr->evtid);if (m)memset (m,0 ,sizeof (struct mbm_state ));@@ -379,7 +384,7 @@ static int __mon_event_count(struct rdtgroup *rdtgrp, struct rmid_read *rr)/* Reading a single domain, must be on a CPU in that domain. */ if (!cpumask_test_cpu (cpu, &rr->d->hdr.cpu_mask))return -EINVAL ; - rr->err =resctrl_arch_rmid_read (rr->r, rr->d, closid, rmid, + rr->err =resctrl_arch_rmid_read (rr->r, rr->d, closid, rmid, cntr_id, rr->evtid, &tval, rr->arch_mon_ctx);if (rr->err)return rr->err;@@ -404,7 +409,8 @@ static int __mon_event_count(struct rdtgroup *rdtgrp, struct rmid_read *rr)list_for_each_entry (d, &rr->r->mon_domains, hdr.list) {if (d->ci->id != rr->ci->id)continue ; - err =resctrl_arch_rmid_read (rr->r, d, closid, rmid, + cntr_id =mbm_cntr_get (rr->r, d, rdtgrp, rr->evtid); + err =resctrl_arch_rmid_read (rr->r, d, closid, rmid, cntr_id, rr->evtid, &tval, rr->arch_mon_ctx);if (!err) { rr->val += tval;include/linux/resctrl.h +0
-0
include/linux/resctrl.h void resctrl_offline_cpu(unsigned int cpu); · line 476
Quoted 4 lines
include/linux/resctrl.h +1
-0@@ -473,6 +473,7 @@ void resctrl_offline_cpu(unsigned int cpu);* counter may match traffic of both @closid and @rmid, or @rmid * only. * @rmid: rmid of the counter to read. +* @cntr_id: cntr_id to read MBM events with mbm_cntr_assign mode.
"Counter ID used to read MBM events in mbm_cntr_evt_assign mode. Only valid when
mbm_cntr_evt_assign mode is enabled and @eventid is an MBM event. Can be negative when invalid." (Please feel free to improve)
Quoted 16 lines
include/linux/resctrl.h +3
-2* @eventid: eventid to read, e.g. L3 occupancy. * @val: result of the counter read in bytes. * @arch_mon_ctx: An architecture specific value from @@ -490,8 +491,9 @@ void resctrl_offline_cpu(unsigned int cpu);* 0 on success, or -EIO, -EINVAL etc on error. */ int resctrl_arch_rmid_read (struct rdt_resource *r,struct rdt_mon_domain *d, -u32 closid,u32 rmid,enum resctrl_event_id eventid, -u64 *val,void *arch_mon_ctx); +u32 closid,u32 rmid,int cntr_id, +enum resctrl_event_id eventid,u64 *val, +void *arch_mon_ctx);/** * resctrl_arch_rmid_read_context_check() - warn about invalid contexts
include/linux/resctrl.h struct rdt_domain_hdr *resctrl_find_domain(struct list_head *h, int id, · line 537
Quoted 4 lines
include/linux/resctrl.h +1
-0@@ -532,12 +534,13 @@ struct rdt_domain_hdr *resctrl_find_domain(struct list_head *h, int id,* @closid: closid that matches the rmid. Depending on the architecture, the * counter may match traffic of both @closid and @rmid, or @rmid only. * @rmid: The rmid whose counter values should be reset. +* @cntr_id: The cntr_id to read MBM events with mbm_cntr_assign mode.
Same as above.
include/linux/resctrl.h struct rdt_domain_hdr *resctrl_find_domain(struct list_head *h, int id, · line 546
Quoted 10 lines
include/linux/resctrl.h +1
-1* @eventid: The eventid whose counter values should be reset. * * This can be called from any CPU. */ void resctrl_arch_reset_rmid (struct rdt_resource *r,struct rdt_mon_domain *d, -u32 closid,u32 rmid, +u32 closid,u32 rmid,int cntr_id,enum resctrl_event_id eventid);/**
Reinette
Moger, Babu
reviewerRe: [PATCH v13 17/27] x86/resctrl: Add the support for reading ABMC counters
Hi Reinette,
Quoted 8 lines · Reinette Chatre
Hi Babu,
Quoted 2 lines · Babu Moger
Software can read the assignable counters using the
QM_EVTSELandQM_CTRregister pair.Please append with more context on how register pair is used to support the changes in this patch.
Sure.
Quoted 13 lines
Quoted 13 lines
QM_EVTSELRegister definition:
=======================================================
Bits Mnemonic Description
=======================================================
63:44 -- Reserved
43:32 RMID Resource Monitoring Identifier
31 ExtEvtID Extended Event Identifier
30:8 -- Reserved
7:0 EvtID Event Identifier
=======================================================
Quoted 8 lines
Quoted 6 lines
The contents of a specific counter can be read by setting the following fields in QM_EVTSEL.ExtendedEvtID = 1, QM_EVTSEL.EvtID = L3CacheABMC (=1) and setting [RMID] to the desired counter ID. Reading
QM_CTRwill then return the contents of the specified counter. The E bit will be set if the counter configuration was invalid, or if an invalid counter ID was set in the QM_EVTSEL[RMID] field.Please rewrite above in imperative tone.
Sure.
Quoted 30 lines
Quoted 30 lines
Introduce
__cntr_id_read_phys()to read the counter ID event data.Link https://www.amd.com/content/dam/amd/en/documents/processor-tech-docs/programmer-references/40332.pdfSigned-off-by Babu Moger <[email protected]>---
v13: Split the patch into 2. First one to handle the passing of rdtgroup structure to few
functions(__mon_event_countandmbm_update(). Second one to handle ABMC counter reading. Added new function__cntr_id_read_phys()to handle ABMC event reading. Updated kernel doc forresctrl_arch_reset_rmid()andresctrl_arch_rmid_read(). Resolved conflicts caused by the recent FS/ARCH code restructure.
Themonitor.cfile has now been split between the FS and ARCH directories.v12: New patch to support extended event mode when ABMC is enabled.
--- arch/x86/kernel/cpu/resctrl/internal.h | 6 +++ arch/x86/kernel/cpu/resctrl/monitor.c | 66 ++++++++++++++++++++++---- fs/resctrl/monitor.c | 14 ++++-- include/linux/resctrl.h | 9 ++-- 4 files changed, 80 insertions(+), 15 deletions(-)arch/x86/kernel/cpu/resctrl/internal.h +0
-0@@ -40,6 +40,12 @@ struct arch_mbm_state {/* Setting bit 0 in L3_QOS_EXT_CFG enables the ABMC feature. */ #define ABMC_ENABLE_BIT 0
Quoted 4 lines
arch/x86/kernel/cpu/resctrl/internal.h struct arch_mbm_state { · line 44
Quoted 2 lines
arch/x86/kernel/cpu/resctrl/internal.h +2
-0+/* +* ABMC Qos Event Identifiers. QoS?
Sure.
Quoted 7 lines
arch/x86/kernel/cpu/resctrl/internal.h struct arch_mbm_state { · line 47
Quoted 4 lines
arch/x86/kernel/cpu/resctrl/internal.h +3
-0+*/ +#define ABMC_EXTENDED_EVT_ID BIT(31) +#define ABMC_EVT_ID 1Please use BIT(0) to be consistent.
Sure.
Quoted 63 lines
Quoted 63 lines
arch/x86/kernel/cpu/resctrl/internal.h +1
-0+/** * struct rdt_hw_ctrl_domain - Arch private attributes of a set of CPUs that share * a resource for a control function arch/x86/kernel/cpu/resctrl/monitor.c +36
-1@@ -161,6 +161,41 @@ static int __rmid_read_phys(u32 prmid, enum resctrl_event_id eventid, u64 *val)return 0 ; } +static int __cntr_id_read_phys (u32 cntr_id,u64 *val) +{ +u64 msr_val; + +/* +* QM_EVTSEL Register definition: +* ======================================================= +* Bits Mnemonic Description +* ======================================================= +* 63:44 -- Reserved +* 43:32 RMID Resource Monitoring Identifier +* 31 ExtEvtID Extended Event Identifier +* 30:8 -- Reserved +* 7:0 EvtID Event Identifier +* ======================================================= +* The contents of a specific counter can be read by setting the +* following fields in QM_EVTSEL.ExtendedEvtID(=1) and +* QM_EVTSEL.EvtID = L3CacheABMC (=1) and setting [RMID] to the +* desired counter ID. Reading QM_CTR will then return the +* contents of the specified counter. The E bit will be set if the +* counter configuration was invalid, or if an invalid counter ID +* was set in the QM_EVTSEL[RMID] field. +*/ +wrmsr (MSR_IA32_QM_EVTSEL ,ABMC_EXTENDED_EVT_ID |ABMC_EVT_ID , cntr_id); +rdmsrl (MSR_IA32_QM_CTR , msr_val); + +if (msr_val &RMID_VAL_ERROR ) +return -EIO ; +if (msr_val &RMID_VAL_UNAVAIL ) +return -EINVAL ; + + *val = msr_val; +return 0 ; +} +static struct arch_mbm_state *get_arch_mbm_state (struct rdt_hw_mon_domain *hw_dom,u32 rmid,enum resctrl_event_id eventid)@@ -180,7 +215,7 @@ static struct arch_mbm_state *get_arch_mbm_state(struct rdt_hw_mon_domain *hw_do }void resctrl_arch_reset_rmid (struct rdt_resource *r,struct rdt_mon_domain *d, -u32 unused,u32 rmid, +u32 unused,u32 rmid,int cntr_id,enum resctrl_event_id eventid) {struct rdt_hw_mon_domain *hw_dom =resctrl_to_arch_mon_dom (d);@@ -192,9 +227,16 @@ void resctrl_arch_reset_rmid(struct rdt_resource *r, struct rdt_mon_domain *d,if (am) {memset (am,0 ,sizeof (*am));
Quoted 8 lines
arch/x86/kernel/cpu/resctrl/monitor.c void resctrl_arch_reset_rmid(struct rdt_resource *r, struct rdt_mon_domain *d, · line 232
Quoted 6 lines
arch/x86/kernel/cpu/resctrl/monitor.c +3
-3- prmid =logical_rmid_to_physical_rmid (cpu, rmid); -/* Record any initial, non-zero count value. */ -__rmid_read_phys (prmid, eventid, &am->prev_msr); +if (resctrl_arch_mbm_cntr_assign_enabled (r) && +resctrl_is_mbm_event (eventid)) { +if (cntr_id <0 )This would be a bug, no? how about
WARN_ON_ONCE()?
Yes. Will do that.
Quoted 26 lines
Quoted 26 lines
arch/x86/kernel/cpu/resctrl/monitor.c +9
-2+return ; +__cntr_id_read_phys (cntr_id, &am->prev_msr); + }else { + prmid =logical_rmid_to_physical_rmid (cpu, rmid); +/* Record any initial, non-zero count value. */ +__rmid_read_phys (prmid, eventid, &am->prev_msr); + } } }@@ -224,8 +266,8 @@ static u64 mbm_overflow_count(u64 prev_msr, u64 cur_msr, unsigned int width) }int resctrl_arch_rmid_read (struct rdt_resource *r,struct rdt_mon_domain *d, -u32 unused,u32 rmid,enum resctrl_event_id eventid, -u64 *val,void *ignored) +u32 unused,u32 rmid,int cntr_id, +enum resctrl_event_id eventid,u64 *val,void *ignored) {struct rdt_hw_mon_domain *hw_dom =resctrl_to_arch_mon_dom (d);struct rdt_hw_resource *hw_res =resctrl_to_arch_res (r);@@ -237,8 +279,16 @@ int resctrl_arch_rmid_read(struct rdt_resource *r, struct rdt_mon_domain *d,resctrl_arch_rmid_read_context_check ();
Quoted 8 lines
arch/x86/kernel/cpu/resctrl/monitor.c int resctrl_arch_rmid_read(struct rdt_resource *r, struct rdt_mon_domain *d, · line 284
Quoted 5 lines
arch/x86/kernel/cpu/resctrl/monitor.c +3
-2- prmid =logical_rmid_to_physical_rmid (cpu, rmid); - ret =__rmid_read_phys (prmid, eventid, &msr_val); +if (resctrl_arch_mbm_cntr_assign_enabled (r) && +resctrl_is_mbm_event (eventid)) { +if (cntr_id <0 )
WARN_ON_ONCE()?
Yes.
Quoted 29 lines
Quoted 29 lines
arch/x86/kernel/cpu/resctrl/monitor.c +7
-0+return cntr_id; + ret =__cntr_id_read_phys (cntr_id, &msr_val); + }else { + prmid =logical_rmid_to_physical_rmid (cpu, rmid); + ret =__rmid_read_phys (prmid, eventid, &msr_val); + } +if (ret)return ret;fs/resctrl/monitor.c +5
-1@@ -159,7 +159,11 @@ void __check_limbo(struct rdt_mon_domain *d, bool force_free)break ; entry =__rmid_entry (idx); -if (resctrl_arch_rmid_read (r, d, entry->closid, entry->rmid, +/* +* cntr_id is not relevant for QOS_L3_OCCUP_EVENT_ID. +* Pass dummy value -1. +*/ +if (resctrl_arch_rmid_read (r, d, entry->closid, entry->rmid,-1 ,QOS_L3_OCCUP_EVENT_ID , &val, arch_mon_ctx)) { rmid_dirty = true;@@ -359,6 +363,7 @@ static struct mbm_state *get_mbm_state(struct rdt_mon_domain *d, u32 closid,
Quoted 9 lines
fs/resctrl/monitor.c static struct mbm_state *get_mbm_state(struct rdt_mon_domain *d, u32 closid, · line 366
Quoted 3 lines
fs/resctrl/monitor.c +1
-0static int __mon_event_count (struct rdtgroup *rdtgrp,struct rmid_read *rr) { +int cntr_id =mbm_cntr_get (rr->r, rr->d, rdtgrp, rr->evtid);So
mbm_cntr_get()is called on *all* events (even non MBM) whether assignable counters are supported or not. I assume it relies onnum_mbm_cntrsto be zero on non-ABMC systems but I think this needs to be explicit thatmbm_cntr_get()returns-ENOENTin these cases. Any developer attempting to modifymbm_cntr_get()needs to be aware of this usage.
Yes. Good point.
Quoted 3 lines
This is quite subtle that
resctrl_arch_reset_rmid()andresctrl_arch_rmid_read()can be called with a negative counter ID. To help with code health this needs to be highlighted (more later).
Sure.
Quoted 37 lines
Quoted 37 lines
fs/resctrl/monitor.c +4
-3int cpu =smp_processor_id ();u32 closid = rdtgrp->closid;u32 rmid = rdtgrp->mon.rmid;@@ -368,7 +373,7 @@ static int __mon_event_count(struct rdtgroup *rdtgrp, struct rmid_read *rr)u64 tval =0 ;if (rr->first) { -resctrl_arch_reset_rmid (rr->r, rr->d, closid, rmid, rr->evtid); +resctrl_arch_reset_rmid (rr->r, rr->d, closid, rmid, cntr_id, rr->evtid); m =get_mbm_state (rr->d, closid, rmid, rr->evtid);if (m)memset (m,0 ,sizeof (struct mbm_state ));@@ -379,7 +384,7 @@ static int __mon_event_count(struct rdtgroup *rdtgrp, struct rmid_read *rr)/* Reading a single domain, must be on a CPU in that domain. */ if (!cpumask_test_cpu (cpu, &rr->d->hdr.cpu_mask))return -EINVAL ; - rr->err =resctrl_arch_rmid_read (rr->r, rr->d, closid, rmid, + rr->err =resctrl_arch_rmid_read (rr->r, rr->d, closid, rmid, cntr_id, rr->evtid, &tval, rr->arch_mon_ctx);if (rr->err)return rr->err;@@ -404,7 +409,8 @@ static int __mon_event_count(struct rdtgroup *rdtgrp, struct rmid_read *rr)list_for_each_entry (d, &rr->r->mon_domains, hdr.list) {if (d->ci->id != rr->ci->id)continue ; - err =resctrl_arch_rmid_read (rr->r, d, closid, rmid, + cntr_id =mbm_cntr_get (rr->r, d, rdtgrp, rr->evtid); + err =resctrl_arch_rmid_read (rr->r, d, closid, rmid, cntr_id, rr->evtid, &tval, rr->arch_mon_ctx);if (!err) { rr->val += tval;include/linux/resctrl.h +0
-0@@ -473,6 +473,7 @@ void resctrl_offline_cpu(unsigned int cpu);
Quoted 8 lines
include/linux/resctrl.h void resctrl_offline_cpu(unsigned int cpu); · line 476
Quoted 4 lines
include/linux/resctrl.h +1
-0* counter may match traffic of both @closid and @rmid, or @rmid * only. * @rmid: rmid of the counter to read. +* @cntr_id: cntr_id to read MBM events with mbm_cntr_assign mode. "Counter ID used to read MBM events in
mbm_cntr_evt_assignmode. Only valid when
mbm_cntr_evt_assignmode is enabled and @eventid is an MBM event. Can be negative when invalid." (Please feel free to improve)
Looks good.
Quoted 14 lines
Quoted 14 lines
include/linux/resctrl.h +3
-2* @eventid: eventid to read, e.g. L3 occupancy. * @val: result of the counter read in bytes. * @arch_mon_ctx: An architecture specific value from @@ -490,8 +491,9 @@ void resctrl_offline_cpu(unsigned int cpu);* 0 on success, or -EIO, -EINVAL etc on error. */ int resctrl_arch_rmid_read (struct rdt_resource *r,struct rdt_mon_domain *d, -u32 closid,u32 rmid,enum resctrl_event_id eventid, -u64 *val,void *arch_mon_ctx); +u32 closid,u32 rmid,int cntr_id, +enum resctrl_event_id eventid,u64 *val, +void *arch_mon_ctx);
Quoted 10 lines
include/linux/resctrl.h struct rdt_domain_hdr *resctrl_find_domain(struct list_head *h, int id, · line 537
Quoted 7 lines
include/linux/resctrl.h +1
-0/** * resctrl_arch_rmid_read_context_check() - warn about invalid contexts @@ -532,12 +534,13 @@ struct rdt_domain_hdr *resctrl_find_domain(struct list_head *h, int id,* @closid: closid that matches the rmid. Depending on the architecture, the * counter may match traffic of both @closid and @rmid, or @rmid only. * @rmid: The rmid whose counter values should be reset. +* @cntr_id: The cntr_id to read MBM events with mbm_cntr_assign mode. Same as above.
Sure.
Quoted 13 lines
include/linux/resctrl.h struct rdt_domain_hdr *resctrl_find_domain(struct list_head *h, int id, · line 546
Quoted 10 lines
include/linux/resctrl.h +1
-1* @eventid: The eventid whose counter values should be reset. * * This can be called from any CPU. */ void resctrl_arch_reset_rmid (struct rdt_resource *r,struct rdt_mon_domain *d, -u32 closid,u32 rmid, +u32 closid,u32 rmid,int cntr_id,enum resctrl_event_id eventid);/** Reinette
Signature · 2 lines
Thanks Babu Moger
Babu Moger
reviewerPATCH v13 18/27 x86/resctrl: Add definitions for MBM event configuration
In reply to Babu MogerThe "mbm_cntr_assign" mode allows users to manually assign a hardware counter to a specific RMID and event pair. The events available for assignment are configurable.
By default, each resctrl group supports two MBM events: mbm_total_bytes and mbm_local_bytes. Each event corresponds to an MBM configuration that specifies the bandwidth sources tracked by the event.
Add definitions of supported bandwidth sources.
---
v13: Updated the changelog.
Removed the definitions from resctrl_types.h and moved to internal.h. Removed mbm_assign_config definition. Configurations will be part of mon_evt list.
Resolved conflicts caused by the recent FS/ARCH code restructure. The rdtgroup.c file has now been split between the FS and ARCH directories.
v12: New patch to support event configurations via new counter_configs
method.
--- fs/resctrl/internal.h | 10 ++++++++++ fs/resctrl/rdtgroup.c | 14 ++++++++++++++ 2 files changed, 24 insertions(+)
fs/resctrl/internal.h +10 -0
@@ -203,6 +203,16 @@ struct rdtgroup {struct pseudo_lock_region *plr; }; +/** +* struct mbm_evt_value - Specific type of memory events. +* @evt_name: Name of memory transaction type (read, write etc). +* @evt_val: Value representing the memory transaction. +*/ +struct mbm_evt_value { +char evt_name[32 ]; +u32 evt_val; +}; +/* rdtgroup.flags */ #define RDT_DELETED 1
fs/resctrl/rdtgroup.c +14 -0
@@ -75,6 +75,20 @@ static void rdtgroup_destroy_root(void);struct dentry *debugfs_resctrl; +/* Number of memory transaction types that can be monitored */ +#define NUM_MBM_EVT_VALUES 7 + +/* Decoded values for each type of memory events */ +struct mbm_evt_value mbm_evt_values[NUM_MBM_EVT_VALUES ] = { + {"local_reads" ,READS_TO_LOCAL_MEM }, + {"remote_reads" ,READS_TO_REMOTE_MEM }, + {"local_non_temporal_writes" ,NON_TEMP_WRITE_TO_LOCAL_MEM }, + {"remote_non_temporal_writes" ,NON_TEMP_WRITE_TO_REMOTE_MEM }, + {"local_reads_slow_memory" ,READS_TO_LOCAL_S_MEM }, + {"remote_reads_slow_memory" ,READS_TO_REMOTE_S_MEM }, + {"dirty_victim_writes_all" ,DIRTY_VICTIMS_TO_ALL_MEM }, +}; +/* * Memory bandwidth monitoring event to use for the default CTRL_MON group * and each new CTRL_MON group created by the user. Only relevant when
Signature · 1 line
2.34.1
Reinette Chatre
maintainerRe: [PATCH v13 18/27] x86/resctrl: Add definitions for MBM event configuration
Hi Babu,
Quoted 7 lines · Babu Moger
The "
mbm_cntr_assign" mode allows users to manually assign a hardware counter to a specific RMID and event pair. The events available for assignment are configurable.By default, each resctrl group supports two MBM events:
mbm_total_bytesandmbm_local_bytes. Each event corresponds to an MBM configuration that specifies the bandwidth sources tracked by the event.
hmmm ... earlier I thought "bandwidth source" means RMID but here it seems to mean the memory transactions? The various terms are confusing.
Quoted 2 lines
Add definitions of supported bandwidth sources.
changelog uses "bandwidth sources" while the comments of patch uses "memory transactions" ... please be consistent with terms.
Quoted 22 lines
Signed-off-by Babu Moger <[email protected]>---
v13: Updated the changelog.
Removed the definitions fromresctrl_types.hand moved tointernal.h. Removedmbm_assign_configdefinition. Configurations will be part ofmon_evtlist.
Resolved conflicts caused by the recent FS/ARCH code restructure. Therdtgroup.cfile has now been split between the FS and ARCH directories.v12: New patch to support event configurations via new
counter_configs
method.--- fs/resctrl/internal.h | 10 ++++++++++ fs/resctrl/rdtgroup.c | 14 ++++++++++++++ 2 files changed, 24 insertions(+)fs/resctrl/internal.h +0
-0
fs/resctrl/internal.h struct rdtgroup { · line 207
Quoted 5 lines
fs/resctrl/internal.h +2
-0@@ -203,6 +203,16 @@ struct rdtgroup {struct pseudo_lock_region *plr; }; +/** +* struct mbm_evt_value - Specific type of memory events.
I am trying to decipher the terminology. If these are events, then it becomes confusing since it becomes "these events are used to configure events". You mention "memory transaction" below, this sounds more accurate to me. Above could thus be:
struct mbm_evt_value - Memory transaction an MBM event can be configured with.
The name of the struct could also do with a rename to avoid the "event" term that conflicts with the actual MBM events. Maybe "mbm_cfg_value" ... I do not think this is a good name so please consider what would work better.
fs/resctrl/internal.h struct rdtgroup { · line 208
Quoted 1 line
fs/resctrl/internal.h +1
-0+* @evt_name: Name of memory transaction type (read, write etc).
Unclear what "type" means ... maybe just "Name of memory transaction (read, write ...)"?
The "evt_" prefix looks unnecessary.
fs/resctrl/internal.h struct rdtgroup { · line 209
Quoted 1 line
fs/resctrl/internal.h +1
-0+* @evt_val: Value representing the memory transaction.
This could just be "val" and the description could be specific:
"The bit used to represent the memory transaction within an event's configuration." Please feel free to improve.
fs/resctrl/internal.h struct rdtgroup { · line 213
Quoted 4 lines
fs/resctrl/internal.h +4
-0+*/ +struct mbm_evt_value { +char evt_name[32 ]; +u32 evt_val;
Please space member names with TABs.
Quoted 10 lines
fs/resctrl/internal.h +2
-0+}; +/* rdtgroup.flags */ #define RDT_DELETED 1fs/resctrl/rdtgroup.c +0
-0
fs/resctrl/rdtgroup.c static void rdtgroup_destroy_root(void); · line 78
Quoted 4 lines
fs/resctrl/rdtgroup.c +1
-0@@ -75,6 +75,20 @@ static void rdtgroup_destroy_root(void);struct dentry *debugfs_resctrl; +/* Number of memory transaction types that can be monitored */
"Number of memory transactions that an MBM event can be configured with."?
fs/resctrl/rdtgroup.c static void rdtgroup_destroy_root(void); · line 81
Quoted 3 lines
fs/resctrl/rdtgroup.c +3
-0+#define NUM_MBM_EVT_VALUES 7 + +/* Decoded values for each type of memory events */
Please be consistent with terminology. In the above lines it switches between "memory transaction types" and "memory events".
fs/resctrl/rdtgroup.c static void rdtgroup_destroy_root(void); · line 94
Quoted 13 lines
fs/resctrl/rdtgroup.c +10
-0+struct mbm_evt_value mbm_evt_values[NUM_MBM_EVT_VALUES ] = { + {"local_reads" ,READS_TO_LOCAL_MEM }, + {"remote_reads" ,READS_TO_REMOTE_MEM }, + {"local_non_temporal_writes" ,NON_TEMP_WRITE_TO_LOCAL_MEM }, + {"remote_non_temporal_writes" ,NON_TEMP_WRITE_TO_REMOTE_MEM }, + {"local_reads_slow_memory" ,READS_TO_LOCAL_S_MEM }, + {"remote_reads_slow_memory" ,READS_TO_REMOTE_S_MEM }, + {"dirty_victim_writes_all" ,DIRTY_VICTIMS_TO_ALL_MEM }, +}; +/* * Memory bandwidth monitoring event to use for the default CTRL_MON group * and each new CTRL_MON group created by the user. Only relevant when
Reinette
Moger, Babu
reviewerRe: [PATCH v13 18/27] x86/resctrl: Add definitions for MBM event configuration
Hi Reinette,
Quoted 7 lines · Reinette Chatre
Hi Babu,
Quoted 4 lines · Babu Moger
The "
mbm_cntr_assign" mode allows users to manually assign a hardware counter to a specific RMID and event pair. The events available for assignment are configurable.
Quoted 6 lines
Quoted 3 lines
By default, each resctrl group supports two MBM events:
mbm_total_bytesandmbm_local_bytes. Each event corresponds to an MBM configuration that specifies the bandwidth sources tracked by the event.hmmm ... earlier I thought "bandwidth source" means RMID but here it seems to mean the memory transactions? The various terms are confusing.
My bad. Yes. "bandwidth source" means RMID.
I should say "memory transactions"
Quoted 6 lines
Quoted 3 lines
Add definitions of supported bandwidth sources.
changelog uses "bandwidth sources" while the comments of patch uses "memory transactions" ... please be consistent with terms.
Sure.
Quoted 26 lines
Quoted 26 lines
Signed-off-by Babu Moger <[email protected]>---
v13: Updated the changelog.
Removed the definitions fromresctrl_types.hand moved tointernal.h. Removedmbm_assign_configdefinition. Configurations will be part ofmon_evtlist.
Resolved conflicts caused by the recent FS/ARCH code restructure. Therdtgroup.cfile has now been split between the FS and ARCH directories.v12: New patch to support event configurations via new
counter_configs
method.--- fs/resctrl/internal.h | 10 ++++++++++ fs/resctrl/rdtgroup.c | 14 ++++++++++++++ 2 files changed, 24 insertions(+)fs/resctrl/internal.h +0
-0@@ -203,6 +203,16 @@ struct rdtgroup {struct pseudo_lock_region *plr; };
Quoted 8 lines
fs/resctrl/internal.h struct rdtgroup { · line 207
Quoted 2 lines
fs/resctrl/internal.h +2
-0+/** +* struct mbm_evt_value - Specific type of memory events. I am trying to decipher the terminology. If these are events, then it becomes confusing since it becomes "these events are used to configure events". You mention "memory transaction" below, this sounds more accurate to me. Above could thus be:
struct
mbm_evt_value- Memory transaction an MBM event can be configured with.
Sure.
Quoted 4 lines
The name of the struct could also do with a rename to avoid the "event" term that conflicts with the actual MBM events. Maybe "
mbm_cfg_value" ... I do not think this is a good name so please consider what would work better.
I can change it to "mbm_config_value".
Quoted 4 lines
fs/resctrl/internal.h struct rdtgroup { · line 208
Quoted 2 lines
fs/resctrl/internal.h +1
-0+* @evt_name: Name of memory transaction type (read, write etc). Unclear what "type" means ... maybe just "Name of memory transaction (read, write ...)"?
sure.
Quoted 2 lines
The "
evt_" prefix looks unnecessary.
ok
Quoted 4 lines
fs/resctrl/internal.h struct rdtgroup { · line 209
Quoted 2 lines
fs/resctrl/internal.h +1
-0+* @evt_val: Value representing the memory transaction. This could just be "val" and the description could be specific:
ok.
Quoted 3 lines
"The bit used to represent the memory transaction within an event's configuration." Please feel free to improve.
Sounds good.
Quoted 7 lines
fs/resctrl/internal.h struct rdtgroup { · line 213
Quoted 5 lines
fs/resctrl/internal.h +4
-0+*/ +struct mbm_evt_value { +char evt_name[32 ]; +u32 evt_val;Please space member names with TABs.
Sure.
Quoted 12 lines
Quoted 12 lines
fs/resctrl/internal.h +2
-0+}; +/* rdtgroup.flags */ #define RDT_DELETED 1fs/resctrl/rdtgroup.c +0
-0@@ -75,6 +75,20 @@ static void rdtgroup_destroy_root(void);
Quoted 5 lines
fs/resctrl/rdtgroup.c static void rdtgroup_destroy_root(void); · line 78
Quoted 3 lines
fs/resctrl/rdtgroup.c +1
-0struct dentry *debugfs_resctrl; +/* Number of memory transaction types that can be monitored */ "Number of memory transactions that an MBM event can be configured with."?
Sure.
Quoted 7 lines
fs/resctrl/rdtgroup.c static void rdtgroup_destroy_root(void); · line 81
Quoted 4 lines
fs/resctrl/rdtgroup.c +3
-0+#define NUM_MBM_EVT_VALUES 7 + +/* Decoded values for each type of memory events */ Please be consistent with terminology. In the above lines it switches between "memory transaction types" and "memory events".
"Decoded values for each type of memory transaction types"
Quoted 17 lines
Quoted 4 lines
fs/resctrl/rdtgroup.c +3
-0+struct mbm_evt_value mbm_evt_values[NUM_MBM_EVT_VALUES ] = { + {"local_reads" ,READS_TO_LOCAL_MEM }, + {"remote_reads" ,READS_TO_REMOTE_MEM },fs/resctrl/rdtgroup.c static void rdtgroup_destroy_root(void); · line 94
Quoted 10 lines
fs/resctrl/rdtgroup.c +7
-0+ {"local_non_temporal_writes", NON_TEMP_WRITE_TO_LOCAL_MEM}, + {"remote_non_temporal_writes", NON_TEMP_WRITE_TO_REMOTE_MEM}, + {"local_reads_slow_memory", READS_TO_LOCAL_S_MEM}, + {"remote_reads_slow_memory", READS_TO_REMOTE_S_MEM}, + {"dirty_victim_writes_all", DIRTY_VICTIMS_TO_ALL_MEM}, +}; +/* * Memory bandwidth monitoring event to use for the default CTRL_MON group * and each new CTRL_MON group created by the user. Only relevant when Reinette
Signature · 2 lines
Thanks Babu Moger
Reinette Chatre
maintainerRe: [PATCH v13 18/27] x86/resctrl: Add definitions for MBM event configuration
Hi Babu,
Quoted 13 lines · Babu
Quoted 11 lines · Reinette Chatre
Quoted 2 lines · Babu Moger
+/**
+ * structmbm_evt_value- Specific type of memory events.I am trying to decipher the terminology. If these are events, then it becomes confusing since it becomes "these events are used to configure events". You mention "memory transaction" below, this sounds more accurate to me. Above could thus be:
struct
mbm_evt_value- Memory transaction an MBM event can be configured with.
Quoted 8 lines
Sure.
Quoted 4 lines
The name of the struct could also do with a rename to avoid the "event" term that conflicts with the actual MBM events. Maybe "
mbm_cfg_value" ... I do not think this is a good name so please consider what would work better.I can change it to "
mbm_config_value".
Looks good, thank you.
...
Quoted 8 lines
Quoted 6 lines
fs/resctrl/rdtgroup.c static void rdtgroup_destroy_root(void); · line 81
Quoted 3 lines
fs/resctrl/rdtgroup.c +3
-0+#define NUM_MBM_EVT_VALUES 7 + +/* Decoded values for each type of memory events */ Please be consistent with terminology. In the above lines it switches between "memory transaction types" and "memory events".
"Decoded values for each type of memory transaction types"
I do not think "type" is needed twice. Could also be:
"Decoded values of each memory transaction type."
Quoted 11 lines
Quoted 11 lines
fs/resctrl/rdtgroup.c static void rdtgroup_destroy_root(void); · line 90
Quoted 11 lines
fs/resctrl/rdtgroup.c +9
-0+struct mbm_evt_value mbm_evt_values[NUM_MBM_EVT_VALUES ] = { + {"local_reads" ,READS_TO_LOCAL_MEM }, + {"remote_reads" ,READS_TO_REMOTE_MEM }, + {"local_non_temporal_writes" ,NON_TEMP_WRITE_TO_LOCAL_MEM }, + {"remote_non_temporal_writes" ,NON_TEMP_WRITE_TO_REMOTE_MEM }, + {"local_reads_slow_memory" ,READS_TO_LOCAL_S_MEM }, + {"remote_reads_slow_memory" ,READS_TO_REMOTE_S_MEM }, + {"dirty_victim_writes_all" ,DIRTY_VICTIMS_TO_ALL_MEM }, +};
Reinette
Re: [PATCH v13 18/27] x86/resctrl: Add definitions for MBM event configuration
Hi Reinette,
Quoted 36 lines · Reinette Chatre
Hi Babu,
Quoted 13 lines · Babu
Quoted 11 lines · Reinette Chatre
Quoted 2 lines · Babu Moger
+/**
+ * structmbm_evt_value- Specific type of memory events.I am trying to decipher the terminology. If these are events, then it becomes confusing since it becomes "these events are used to configure events". You mention "memory transaction" below, this sounds more accurate to me. Above could thus be:
struct
mbm_evt_value- Memory transaction an MBM event can be configured with.Quoted 8 lines
Sure.
Quoted 4 lines
The name of the struct could also do with a rename to avoid the "event" term that conflicts with the actual MBM events. Maybe "
mbm_cfg_value" ... I do not think this is a good name so please consider what would work better.I can change it to "
mbm_config_value".Looks good, thank you.
...
Quoted 7 lines
Quoted 6 lines
fs/resctrl/rdtgroup.c static void rdtgroup_destroy_root(void); · line 81
Quoted 3 lines
fs/resctrl/rdtgroup.c +3
-0+#define NUM_MBM_EVT_VALUES 7 + +/* Decoded values for each type of memory events */ Please be consistent with terminology. In the above lines it switches between "memory transaction types" and "memory events".
Quoted 4 lines
Quoted 1 line
"Decoded values for each type of memory transaction types"
I do not think "type" is needed twice. Could also be:
"Decoded values of each memory transaction type."
Sure.
Thanks
Babu
Babu Moger
reviewerPATCH v13 19/27 x86/resctrl: Add event configuration directory under info/L3_MON/
In reply to Babu MogerCreate the configuration directory and files for mbm_cntr_assign mode. These configurations will be used to assign MBM events in mbm_cntr_assign mode, with two default configurations created upon mounting.
Example:
$ cd /sys/fs/resctrl/ $ cat info/L3_MON/counter_configs/mbm_total_bytes/event_filter local_reads, remote_reads, local_non_temporal_writes, remote_non_temporal_writes, local_reads_slow_memory, remote_reads_slow_memory, dirty_victim_writes_all
$ cat info/L3_MON/counter_configs/mbm_local_bytes/event_filter local_reads, local_non_temporal_writes, local_reads_slow_memory
---
v13: Updated user doc (resctrl.rst).
Changed the name of the function resctrl_mkdir_info_configs to resctrl_mkdir_counter_configs().
Replaced seq_puts() with seq_putc() where applicable.
Removed RFTYPE_MON_CONFIG definition. Not required.
Changed the name of the flag RFTYPE_CONFIG to RFTYPE_ASSIGN_CONFIG. Reinette suggested RFTYPE_MBM_EVENT_CONFIG but RFTYPE_ASSIGN_CONFIG seemed shorter and pricise.
The configuration is created using evt_list.
Resolved conflicts caused by the recent FS/ARCH code restructure.
The monitor.c/rdtgroup.c files have been split between the FS and ARCH directories.
v12: New patch to hold the MBM event configurations for mbm_cntr_assign mode.
--- Documentation/filesystems/resctrl.rst | 30 ++++++++++ fs/resctrl/internal.h | 2 + fs/resctrl/monitor.c | 1 + fs/resctrl/rdtgroup.c | 80 +++++++++++++++++++++++++++ 4 files changed, 113 insertions(+)
Documentation/filesystems/resctrl.rst +30 -0
@@ -306,6 +306,36 @@ with the following files: # cat /sys/fs/resctrl/info/L3_MON/available_mbm_cntrs 0=30;1=30 +"counter_configs": + When the "mbm_cntr_assign" mode is supported, a dedicated directory is created + under the "L3_MON" directory to store configuration files. + + These files contain the list of configurable events. There are two default + configurations: mbm_local_bytes and mbm_total_bytes. + + Following types of events are supported: + + ==== ========================= ============================================================ + Bits Name Description + ==== ========================= ============================================================ + 6 dirty_victim_writes_all Dirty Victims from the QOS domain to all types of memory + 5 remote_reads_slow_memory Reads to slow memory in the non-local NUMA domain + 4 local_reads_slow_memory Reads to slow memory in the local NUMA domain + 3 remote_non_temporal_writes Non-temporal writes to non-local NUMA domain + 2 local_non_temporal_writes Non-temporal writes to local NUMA domain + 1 remote_reads Reads to memory in the non-local NUMA domain + 0 local_reads Reads to memory in the local NUMA domain + ==== ========================= ========================================================== + + For example:: + + # cat /sys/fs/resctrl/info/L3_MON/counter_configs/mbm_total_bytes/event_filter + local_reads, remote_reads, local_non_temporal_writes, remote_non_temporal_writes, + local_reads_slow_memory, remote_reads_slow_memory, dirty_victim_writes_all + + # cat /sys/fs/resctrl/info/L3_MON/counter_configs/mbm_local_bytes/event_filter + local_reads, local_non_temporal_writes, local_reads_slow_memory + "max_threshold_occupancy": Read/write file provides the largest value (in bytes) at which a previously used LLC_occupancy
fs/resctrl/internal.h +2 -0
@@ -238,6 +238,8 @@ struct mbm_evt_value {#define RFTYPE_DEBUG BIT(10) +#define RFTYPE_ASSIGN_CONFIG BIT(11) +#define RFTYPE_CTRL_INFO (RFTYPE_INFO | RFTYPE_CTRL)#define RFTYPE_MON_INFO (RFTYPE_INFO | RFTYPE_MON)
fs/resctrl/monitor.c +1 -0
@@ -932,6 +932,7 @@ int resctrl_mon_resource_init(void) RFTYPE_MON_INFO | RFTYPE_RES_CACHE);resctrl_file_fflags_init ("available_mbm_cntrs" ,RFTYPE_MON_INFO |RFTYPE_RES_CACHE ); +resctrl_file_fflags_init ("event_filter" ,RFTYPE_ASSIGN_CONFIG ); }return 0 ;
fs/resctrl/rdtgroup.c +80 -0
104 lines not on this page · raw message
Signature · 1 line
2.34.1
Reinette Chatre
maintainerRe: [PATCH v13 19/27] x86/resctrl: Add event configuration directory under info/L3_MON/
Hi Babu,
Quoted 3 lines · Babu Moger
Create the configuration directory and files for
mbm_cntr_assignmode. These configurations will be used to assign MBM events inmbm_cntr_assignmode, with two default configurations created upon mounting.
This just jumps in with what the patch does. Requirements for proper changelog should be familiar by now. The changelog *always* starts with a context.
Sample:
"When assignable counters are supported the /sys/fs/resctrl/info/L3_MON/event_configs directory contains a sub-directory for each MBM event that can be assigned to a counter. The MBM event sub-directory contains a file named "event_filter" that is used to view and modify which memory transactions the MBM event is configured with.
Create the /sys/fs/resctrl/info/L3_MON/event_configs directory on resctrl mount and pre-populate it with directories for the two existing MBM events:mbm_total_bytes and mbm_local_bytes. Create the "event_filter" file within each MBM event directory with the needed *show() that displays the memory transactions with which the MBM event is configured."
Quoted 38 lines
Example:
$ cd /sys/fs/resctrl/ $ cat info/L3_MON/counter_configs/mbm_total_bytes/event_filter local_reads, remote_reads, local_non_temporal_writes, remote_non_temporal_writes, local_reads_slow_memory, remote_reads_slow_memory, dirty_victim_writes_all$ cat info/L3_MON/counter_configs/mbm_local_bytes/event_filter local_reads, local_non_temporal_writes, local_reads_slow_memorySigned-off-by Babu Moger <[email protected]>---
v13: Updated user doc (resctrl.rst).
Changed the name of the functionresctrl_mkdir_info_configstoresctrl_mkdir_counter_configs().
Replacedseq_puts()withseq_putc()where applicable.
RemovedRFTYPE_MON_CONFIGdefinition. Not required.
Changed the name of the flagRFTYPE_CONFIGtoRFTYPE_ASSIGN_CONFIG. Reinette suggestedRFTYPE_MBM_EVENT_CONFIGbutRFTYPE_ASSIGN_CONFIGseemed shorter and pricise.
The configuration is created usingevt_list.
Resolved conflicts caused by the recent FS/ARCH code restructure.
Themonitor.c/rdtgroup.cfiles have been split between the FS and ARCH directories.v12: New patch to hold the MBM event configurations for
mbm_cntr_assignmode.--- Documentation/filesystems/resctrl.rst | 30 ++++++++++ fs/resctrl/internal.h | 2 + fs/resctrl/monitor.c | 1 + fs/resctrl/rdtgroup.c | 80 +++++++++++++++++++++++++++ 4 files changed, 113 insertions(+)Documentation/filesystems/resctrl.rst +0
-0
Documentation/filesystems/resctrl.rst with the following files: · line 311
Quoted 6 lines
Documentation/filesystems/resctrl.rst +3
-0@@ -306,6 +306,36 @@ with the following files: # cat /sys/fs/resctrl/info/L3_MON/available_mbm_cntrs 0=30;1=30 +"counter_configs": + When the "mbm_cntr_assign" mode is supported, a dedicated directory is created + under the "L3_MON" directory to store configuration files.
? it does not contain files but directories for each event, no?
It will help if the text is specific. For example,
"event_configs":
Directory that exists when mbm_cntr_evt_assign is supported. Contains sub-directory for each MBM event that can be assigned to a counter. Each MBM event sub-directory ...
Documentation/filesystems/resctrl.rst with the following files: · line 313
Quoted 2 lines
Documentation/filesystems/resctrl.rst +2
-0+ + These files contain the list of configurable events. There are two default
So confusing ... terminology is all over the place. Which files are even talked about here? "configurable events" ... are these the memory transactions or MBM events?
Documentation/filesystems/resctrl.rst with the following files: · line 314
Quoted 1 line
Documentation/filesystems/resctrl.rst +1
-0+ configurations: mbm_local_bytes and mbm_total_bytes.
"two default configurations"? These are not "configurations" but "events", no?
Documentation/filesystems/resctrl.rst with the following files: · line 316
Quoted 2 lines
Documentation/filesystems/resctrl.rst +2
-0+ + Following types of events are supported:
events -> memory transactions?
I am unable to parse the above.
Documentation/filesystems/resctrl.rst with the following files: · line 328
Quoted 12 lines
Documentation/filesystems/resctrl.rst +12
-0+ + ==== ========================= ============================================================ + Bits Name Description + ==== ========================= ============================================================ + 6 dirty_victim_writes_all Dirty Victims from the QOS domain to all types of memory + 5 remote_reads_slow_memory Reads to slow memory in the non-local NUMA domain + 4 local_reads_slow_memory Reads to slow memory in the local NUMA domain + 3 remote_non_temporal_writes Non-temporal writes to non-local NUMA domain + 2 local_non_temporal_writes Non-temporal writes to local NUMA domain + 1 remote_reads Reads to memory in the non-local NUMA domain + 0 local_reads Reads to memory in the local NUMA domain + ==== ========================= ==========================================================
Why does user need to know the bit position used to represent the memory transaction?
Quoted 44 lines
Documentation/filesystems/resctrl.rst +10
-0+ + For example:: + + # cat /sys/fs/resctrl/info/L3_MON/counter_configs/mbm_total_bytes/event_filter + local_reads, remote_reads, local_non_temporal_writes, remote_non_temporal_writes, + local_reads_slow_memory, remote_reads_slow_memory, dirty_victim_writes_all + + # cat /sys/fs/resctrl/info/L3_MON/counter_configs/mbm_local_bytes/event_filter + local_reads, local_non_temporal_writes, local_reads_slow_memory + "max_threshold_occupancy": Read/write file provides the largest value (in bytes) at which a previously used LLC_occupancyfs/resctrl/internal.h +2
-0@@ -238,6 +238,8 @@ struct mbm_evt_value {#define RFTYPE_DEBUG BIT(10) +#define RFTYPE_ASSIGN_CONFIG BIT(11) +#define RFTYPE_CTRL_INFO (RFTYPE_INFO | RFTYPE_CTRL)#define RFTYPE_MON_INFO (RFTYPE_INFO | RFTYPE_MON)fs/resctrl/monitor.c +1
-0@@ -932,6 +932,7 @@ int resctrl_mon_resource_init(void) RFTYPE_MON_INFO | RFTYPE_RES_CACHE);resctrl_file_fflags_init ("available_mbm_cntrs" ,RFTYPE_MON_INFO |RFTYPE_RES_CACHE ); +resctrl_file_fflags_init ("event_filter" ,RFTYPE_ASSIGN_CONFIG ); }return 0 ;fs/resctrl/rdtgroup.c +0
-0@@ -1911,6 +1911,25 @@ static int resctrl_available_mbm_cntrs_show(struct kernfs_open_file *of,return ret;
fs/resctrl/rdtgroup.c static int resctrl_available_mbm_cntrs_show(struct kernfs_open_file *of, · line 1921
Quoted 10 lines
fs/resctrl/rdtgroup.c +8
-0} +static int event_filter_show (struct kernfs_open_file *of,struct seq_file *seq,void *v) +{ +struct mon_evt *mevt =rdt_kn_parent_priv (of->kn); +bool sep = false; +int i; + +for (i =0 ; i <NUM_MBM_EVT_VALUES ; i++) { +if (mevt->evt_cfg & mbm_evt_values[i].evt_val) {
Still no idea where mevt->evt_cfg comes from. Patch ordering issue?
Quoted 45 lines
fs/resctrl/rdtgroup.c +31
-0+if (sep) +seq_putc (seq,',' ); +seq_printf (seq,"%s" , mbm_evt_values[i].evt_name); + sep = true; + } + } +seq_putc (seq,'\n' ); + +return 0 ; +} +/* rdtgroup information files for one cache resource. */ static struct rftype res_common_files[] = { {@@ -2035,6 +2054,12 @@ static struct rftype res_common_files[] = { .seq_show = mbm_local_bytes_config_show, .write = mbm_local_bytes_config_write, }, + { + .name = "event_filter", + .mode = 0444, + .kf_ops = &rdtgroup_kf_single_ops, + .seq_show = event_filter_show, + }, { .name = "mbm_assign_mode", .mode = 0444,@@ -2317,6 +2342,55 @@ static int rdtgroup_mkdir_info_resdir(void *priv, char *name,return ret; } +static int resctrl_mkdir_counter_configs (struct rdt_resource *r,char *name) +{ +struct kernfs_node *l3_mon_kn, *kn_subdir, *kn_subdir2; +struct mon_evt *mevt; +int ret; + + l3_mon_kn =kernfs_find_and_get (kn_info, name); +if (!l3_mon_kn) +return -ENOENT ; + + kn_subdir =kernfs_create_dir (l3_mon_kn,"counter_configs" , l3_mon_kn->mode,NULL ); +if (IS_ERR (kn_subdir)) { +kernfs_put (l3_mon_kn); +return PTR_ERR (kn_subdir);
fs/resctrl/rdtgroup.c static int rdtgroup_mkdir_info_resdir(void *priv, char *name, · line 2368
Quoted 10 lines
fs/resctrl/rdtgroup.c +10
-0+ } + + ret =rdtgroup_kn_set_ugid (kn_subdir); +if (ret) { +kernfs_put (l3_mon_kn); +return ret; + } + +list_for_each_entry (mevt, &r->mon.evt_list, list) { +if (mevt->mbm_mode ==MBM_MODE_ASSIGN ) {
I do not think this "mbm_mode" is needed, resctrl_mon::mbm_cntr_assignable is already used earlier, so would for_each_mbm_event() from the telemetry work be useful here?
fs/resctrl/rdtgroup.c static int rdtgroup_mkdir_info_resdir(void *priv, char *name, · line 2373
Quoted 5 lines
fs/resctrl/rdtgroup.c +5
-0+ kn_subdir2 =kernfs_create_dir (kn_subdir, mevt->name, + kn_subdir->mode, mevt); +if (IS_ERR (kn_subdir2)) { + ret =PTR_ERR (kn_subdir2); + goto config_out;
"grep goto fs/resctrl/rdtgroup.c" for naming conventions.
Quoted 6 lines
fs/resctrl/rdtgroup.c +6
-0+ } + + ret =rdtgroup_kn_set_ugid (kn_subdir2); +if (ret) + goto config_out; +
fs/resctrl/rdtgroup.c static int rdtgroup_mkdir_info_resdir(void *priv, char *name, · line 2389
Quoted 10 lines
fs/resctrl/rdtgroup.c +10
-0+ ret =rdtgroup_add_files (kn_subdir2,RFTYPE_ASSIGN_CONFIG ); +if (!ret) +kernfs_activate (kn_subdir); + } + } + +config_out: +kernfs_put (l3_mon_kn); +if (ret) +kernfs_remove (kn_subdir);
This looks unnecessary since caller does kernfs_remove() on error return. Compare with how rdtgroup_mkdir_info_resdir() handles errors.
Quoted 10 lines
fs/resctrl/rdtgroup.c +4
-0+ +return ret; +} +static unsigned long fflags_from_resource (struct rdt_resource *r) {switch (r->rid) {@@ -2363,6 +2437,12 @@ static int rdtgroup_create_info_dir(struct kernfs_node *parent_kn) ret =rdtgroup_mkdir_info_resdir (r, name, fflags);if (ret)
fs/resctrl/rdtgroup.c static int rdtgroup_create_info_dir(struct kernfs_node *parent_kn) · line 2448
Quoted 10 lines
fs/resctrl/rdtgroup.c +6
-0goto out_destroy; + +if (r->mon.mbm_cntr_assignable) { + ret =resctrl_mkdir_counter_configs (r, name); +if (ret) + goto out_destroy; + } } ret =rdtgroup_kn_set_ugid (kn_info);
Reinette
Moger, Babu
reviewerRe: [PATCH v13 19/27] x86/resctrl: Add event configuration directory under info/L3_MON/
Hi Reinette,
Quoted 18 lines · Reinette Chatre
Hi Babu,
Quoted 3 lines · Babu Moger
Create the configuration directory and files for
mbm_cntr_assignmode. These configurations will be used to assign MBM events inmbm_cntr_assignmode, with two default configurations created upon mounting.This just jumps in with what the patch does. Requirements for proper changelog should be familiar by now. The changelog *always* starts with a context.
Sample:
"When assignable counters are supported the
/sys/fs/resctrl/info/L3_MON/event_configsdirectory contains a sub-directory for each MBM event that can be assigned to a counter. The MBM event sub-directory contains a file named "event_filter" that is used to view and modify which memory transactions the MBM event is configured with.
Quoted 6 lines
Create the
/sys/fs/resctrl/info/L3_MON/event_configsdirectory on resctrl mount and pre-populate it with directories for the two existing MBM events:mbm_total_bytesandmbm_local_bytes. Create the "event_filter" file within each MBM event directory with the needed*show()that displays the memory transactions with which the MBM event is configured."
Looks good. Thanks. The directory name will be "event_configs". Will change it in the code as well.
Quoted 47 lines
Quoted 38 lines
Example:
$ cd /sys/fs/resctrl/ $ cat info/L3_MON/counter_configs/mbm_total_bytes/event_filter local_reads, remote_reads, local_non_temporal_writes, remote_non_temporal_writes, local_reads_slow_memory, remote_reads_slow_memory, dirty_victim_writes_all$ cat info/L3_MON/counter_configs/mbm_local_bytes/event_filter local_reads, local_non_temporal_writes, local_reads_slow_memorySigned-off-by Babu Moger <[email protected]>---
v13: Updated user doc (resctrl.rst).
Changed the name of the functionresctrl_mkdir_info_configstoresctrl_mkdir_counter_configs().
Replacedseq_puts()withseq_putc()where applicable.
RemovedRFTYPE_MON_CONFIGdefinition. Not required.
Changed the name of the flagRFTYPE_CONFIGtoRFTYPE_ASSIGN_CONFIG. Reinette suggestedRFTYPE_MBM_EVENT_CONFIGbutRFTYPE_ASSIGN_CONFIGseemed shorter and pricise.
The configuration is created usingevt_list.
Resolved conflicts caused by the recent FS/ARCH code restructure.
Themonitor.c/rdtgroup.cfiles have been split between the FS and ARCH directories.v12: New patch to hold the MBM event configurations for
mbm_cntr_assignmode.--- Documentation/filesystems/resctrl.rst | 30 ++++++++++ fs/resctrl/internal.h | 2 + fs/resctrl/monitor.c | 1 + fs/resctrl/rdtgroup.c | 80 +++++++++++++++++++++++++++ 4 files changed, 113 insertions(+)Documentation/filesystems/resctrl.rst +0
-0Documentation/filesystems/resctrl.rst with the following files: · line 311
Quoted 6 lines
Documentation/filesystems/resctrl.rst +3
-0@@ -306,6 +306,36 @@ with the following files: # cat /sys/fs/resctrl/info/L3_MON/available_mbm_cntrs 0=30;1=30 +"counter_configs": + When the "mbm_cntr_assign" mode is supported, a dedicated directory is created + under the "L3_MON" directory to store configuration files.? it does not contain files but directories for each event, no?
Quoted 5 lines
It will help if the text is specific. For example,
"event_configs":
Directory that exists whenmbm_cntr_evt_assignis supported. Contains sub-directory for each MBM event that can be assigned to a counter. Each MBM event sub-directory ...
Sounds good.
Quoted 6 lines
Documentation/filesystems/resctrl.rst with the following files: · line 313
Quoted 3 lines
Documentation/filesystems/resctrl.rst +2
-0+ + These files contain the list of configurable events. There are two defaultSo confusing ... terminology is all over the place. Which files are even talked about here? "configurable events" ... are these the memory transactions or MBM events?
Should be "memory trasactions"
Quoted 4 lines
Documentation/filesystems/resctrl.rst with the following files: · line 314
Quoted 2 lines
Documentation/filesystems/resctrl.rst +1
-0+ configurations: mbm_local_bytes and mbm_total_bytes."two default configurations"? These are not "configurations" but "events", no?
Sure. Should be "two default events"
Quoted 5 lines
Documentation/filesystems/resctrl.rst with the following files: · line 316
Quoted 3 lines
Documentation/filesystems/resctrl.rst +2
-0+ + Following types of events are supported:events -> memory transactions?
Sure.
Quoted 2 lines
I am unable to parse the above.
The following are the types of memory transactions that an MBM event can be configured with:
Quoted 16 lines
Quoted 4 lines
Documentation/filesystems/resctrl.rst +2
-0+ + ==== ========================= ============================================================Documentation/filesystems/resctrl.rst with the following files: · line 328
Quoted 10 lines
Documentation/filesystems/resctrl.rst +10
-0+ Bits Name Description + ==== ========================= ============================================================ + 6 dirty_victim_writes_all Dirty Victims from the QOS domain to all types of memory + 5 remote_reads_slow_memory Reads to slow memory in the non-local NUMA domain + 4 local_reads_slow_memory Reads to slow memory in the local NUMA domain + 3 remote_non_temporal_writes Non-temporal writes to non-local NUMA domain + 2 local_non_temporal_writes Non-temporal writes to local NUMA domain + 1 remote_reads Reads to memory in the non-local NUMA domain + 0 local_reads Reads to memory in the local NUMA domain + ==== ========================= ==========================================================Why does user need to know the bit position used to represent the memory transaction?
Not required. Will remove it.
Quoted 47 lines
Quoted 47 lines
Documentation/filesystems/resctrl.rst +10
-0+ + For example:: + + # cat /sys/fs/resctrl/info/L3_MON/counter_configs/mbm_total_bytes/event_filter + local_reads, remote_reads, local_non_temporal_writes, remote_non_temporal_writes, + local_reads_slow_memory, remote_reads_slow_memory, dirty_victim_writes_all + + # cat /sys/fs/resctrl/info/L3_MON/counter_configs/mbm_local_bytes/event_filter + local_reads, local_non_temporal_writes, local_reads_slow_memory + "max_threshold_occupancy": Read/write file provides the largest value (in bytes) at which a previously used LLC_occupancyfs/resctrl/internal.h +2
-0@@ -238,6 +238,8 @@ struct mbm_evt_value {#define RFTYPE_DEBUG BIT(10) +#define RFTYPE_ASSIGN_CONFIG BIT(11) +#define RFTYPE_CTRL_INFO (RFTYPE_INFO | RFTYPE_CTRL)#define RFTYPE_MON_INFO (RFTYPE_INFO | RFTYPE_MON)fs/resctrl/monitor.c +1
-0@@ -932,6 +932,7 @@ int resctrl_mon_resource_init(void) RFTYPE_MON_INFO | RFTYPE_RES_CACHE);resctrl_file_fflags_init ("available_mbm_cntrs" ,RFTYPE_MON_INFO |RFTYPE_RES_CACHE ); +resctrl_file_fflags_init ("event_filter" ,RFTYPE_ASSIGN_CONFIG ); }return 0 ;fs/resctrl/rdtgroup.c +0
-0@@ -1911,6 +1911,25 @@ static int resctrl_available_mbm_cntrs_show(struct kernfs_open_file *of,return ret; }
Quoted 10 lines
fs/resctrl/rdtgroup.c static int resctrl_available_mbm_cntrs_show(struct kernfs_open_file *of, · line 1921
Quoted 8 lines
fs/resctrl/rdtgroup.c +8
-0+static int event_filter_show (struct kernfs_open_file *of,struct seq_file *seq,void *v) +{ +struct mon_evt *mevt =rdt_kn_parent_priv (of->kn); +bool sep = false; +int i; + +for (i =0 ; i <NUM_MBM_EVT_VALUES ; i++) { +if (mevt->evt_cfg & mbm_evt_values[i].evt_val) {Still no idea where
mevt->evt_cfgcomes from. Patch ordering issue?
Yes.
Need to introduce evt_cfg member and also need to initialize the default values during the init. Will order it correctly to make little bit clear.
Quoted 51 lines
Quoted 51 lines
fs/resctrl/rdtgroup.c +36
-0+if (sep) +seq_putc (seq,',' ); +seq_printf (seq,"%s" , mbm_evt_values[i].evt_name); + sep = true; + } + } +seq_putc (seq,'\n' ); + +return 0 ; +} +/* rdtgroup information files for one cache resource. */ static struct rftype res_common_files[] = { {@@ -2035,6 +2054,12 @@ static struct rftype res_common_files[] = { .seq_show = mbm_local_bytes_config_show, .write = mbm_local_bytes_config_write, }, + { + .name = "event_filter", + .mode = 0444, + .kf_ops = &rdtgroup_kf_single_ops, + .seq_show = event_filter_show, + }, { .name = "mbm_assign_mode", .mode = 0444,@@ -2317,6 +2342,55 @@ static int rdtgroup_mkdir_info_resdir(void *priv, char *name,return ret; } +static int resctrl_mkdir_counter_configs (struct rdt_resource *r,char *name) +{ +struct kernfs_node *l3_mon_kn, *kn_subdir, *kn_subdir2; +struct mon_evt *mevt; +int ret; + + l3_mon_kn =kernfs_find_and_get (kn_info, name); +if (!l3_mon_kn) +return -ENOENT ; + + kn_subdir =kernfs_create_dir (l3_mon_kn,"counter_configs" , l3_mon_kn->mode,NULL ); +if (IS_ERR (kn_subdir)) { +kernfs_put (l3_mon_kn); +return PTR_ERR (kn_subdir); + } + + ret =rdtgroup_kn_set_ugid (kn_subdir); +if (ret) { +kernfs_put (l3_mon_kn);
Quoted 8 lines
fs/resctrl/rdtgroup.c static int rdtgroup_mkdir_info_resdir(void *priv, char *name, · line 2368
Quoted 5 lines
fs/resctrl/rdtgroup.c +5
-0+return ret; + } + +list_for_each_entry (mevt, &r->mon.evt_list, list) { +if (mevt->mbm_mode ==MBM_MODE_ASSIGN ) {I do not think this "
mbm_mode" is needed,resctrl_mon::mbm_cntr_assignableis already used earlier, so wouldfor_each_mbm_event()from the telemetry work be useful here?
Yes. Will remove mbm_mode and use Tony's telemetry work.
Quoted 8 lines
fs/resctrl/rdtgroup.c static int rdtgroup_mkdir_info_resdir(void *priv, char *name, · line 2373
Quoted 6 lines
fs/resctrl/rdtgroup.c +5
-0+ kn_subdir2 =kernfs_create_dir (kn_subdir, mevt->name, + kn_subdir->mode, mevt); +if (IS_ERR (kn_subdir2)) { + ret =PTR_ERR (kn_subdir2); + goto config_out;"grep goto
fs/resctrl/rdtgroup.c" for naming conventions.
Yes. I see it. It should be "out_config".
Quoted 12 lines
Quoted 12 lines
fs/resctrl/rdtgroup.c +11
-0+ } + + ret =rdtgroup_kn_set_ugid (kn_subdir2); +if (ret) + goto config_out; + + ret =rdtgroup_add_files (kn_subdir2,RFTYPE_ASSIGN_CONFIG ); +if (!ret) +kernfs_activate (kn_subdir); + } + }
Quoted 8 lines
fs/resctrl/rdtgroup.c static int rdtgroup_mkdir_info_resdir(void *priv, char *name, · line 2389
Quoted 5 lines
fs/resctrl/rdtgroup.c +5
-0+ +config_out: +kernfs_put (l3_mon_kn); +if (ret) +kernfs_remove (kn_subdir);This looks unnecessary since caller does
kernfs_remove()on error return. Compare with howrdtgroup_mkdir_info_resdir()handles errors.
Yes. Will remove it.
Quoted 24 lines
Quoted 11 lines
fs/resctrl/rdtgroup.c +4
-0+ +return ret; +} +static unsigned long fflags_from_resource (struct rdt_resource *r) {switch (r->rid) {@@ -2363,6 +2437,12 @@ static int rdtgroup_create_info_dir(struct kernfs_node *parent_kn) ret =rdtgroup_mkdir_info_resdir (r, name, fflags);if (ret)fs/resctrl/rdtgroup.c static int rdtgroup_create_info_dir(struct kernfs_node *parent_kn) · line 2448
Quoted 10 lines
fs/resctrl/rdtgroup.c +6
-0goto out_destroy; + +if (r->mon.mbm_cntr_assignable) { + ret =resctrl_mkdir_counter_configs (r, name); +if (ret) + goto out_destroy; + } } ret =rdtgroup_kn_set_ugid (kn_info);Reinette
Signature · 2 lines
Thanks Babu Moger
Babu Moger
reviewerPATCH v13 20/27 x86/resctrl: Provide interface to update the event configurations
In reply to Babu MogerUsers can modify the event configuration by writing to the event_filter interface file. The event configurations for mbm_cntr_assign mode are located in /sys/fs/resctrl/info/event_configs/.
Update the assignments of all groups when the event configuration is modified.
Example:
$ cd /sys/fs/resctrl/
$ cat info/L3_MON/counter_configs/mbm_local_bytes/event_filter local_reads,local_non_temporal_writes,local_reads_slow_memory
$ echo "local_reads,local_non_temporal_writes" > info/L3_MON/counter_configs/mbm_total_bytes/event_filter
$ cat info/L3_MON/counter_configs/mbm_total_bytes/event_filter local_reads,local_non_temporal_writes
---
v13: Updated changelog for imperative mode.
Added function description in the prototype.
Updated the user doc resctrl.rst to address few feedback.
Resolved conflicts caused by the recent FS/ARCH code restructure.
The rdtgroup.c/monitor.c file has now been split between the FS and ARCH directories.
v12: New patch to modify event configurations.
--- Documentation/filesystems/resctrl.rst | 12 +++ fs/resctrl/rdtgroup.c | 120 +++++++++++++++++++++++++- 2 files changed, 131 insertions(+), 1 deletion(-)
Documentation/filesystems/resctrl.rst +12 -0
@@ -336,6 +336,18 @@ with the following files: # cat /sys/fs/resctrl/info/L3_MON/counter_configs/mbm_local_bytes/event_filter local_reads, local_non_temporal_writes, local_reads_slow_memory + Modify the event configuration by writing to the "event_filter" file within the + configuration directory. The read/write event_filter file contains the configuration + of the event that reflects which memory transactions are counted by it. + + For example:: + + # echo "local_reads, local_non_temporal_writes" > + /sys/fs/resctrl/info/L3_MON/counter_configs/mbm_total_bytes/event_filter + + # cat /sys/fs/resctrl/info/L3_MON/counter_configs/mbm_total_bytes/event_filter + local_reads, local_non_temporal_writes + "max_threshold_occupancy": Read/write file provides the largest value (in bytes) at which a previously used LLC_occupancy
fs/resctrl/rdtgroup.c +119 -1
134 lines not on this page · raw message
Signature · 1 line
2.34.1
Reinette Chatre
maintainerRe: [PATCH v13 20/27] x86/resctrl: Provide interface to update the event configurations
Hi Babu,
Quoted 3 lines · Babu Moger
Users can modify the event configuration by writing to the
event_filterinterface file. The event configurations formbm_cntr_assignmode are located in/sys/fs/resctrl/info/event_configs/.
heh ... looks like you also started thinking that "event_configs" is a better name (also missing L3_MON).
Quoted 34 lines
Update the assignments of all groups when the event configuration is modified.
Example:
$ cd /sys/fs/resctrl/$ cat info/L3_MON/counter_configs/mbm_local_bytes/event_filter local_reads,local_non_temporal_writes,local_reads_slow_memory$ echo "local_reads,local_non_temporal_writes" > info/L3_MON/counter_configs/mbm_total_bytes/event_filter$ cat info/L3_MON/counter_configs/mbm_total_bytes/event_filter local_reads,local_non_temporal_writesSigned-off-by Babu Moger <[email protected]>---
v13: Updated changelog for imperative mode.
Added function description in the prototype.
Updated the user docresctrl.rstto address few feedback.
Resolved conflicts caused by the recent FS/ARCH code restructure.
Therdtgroup.c/monitor.cfile has now been split between the FS and ARCH directories.v12: New patch to modify event configurations.
--- Documentation/filesystems/resctrl.rst | 12 +++ fs/resctrl/rdtgroup.c | 120 +++++++++++++++++++++++++- 2 files changed, 131 insertions(+), 1 deletion(-)Documentation/filesystems/resctrl.rst +0
-0
...
fs/resctrl/rdtgroup.c static int event_filter_show(struct kernfs_open_file *of, struct seq_file *seq, · line 1935
Quoted 11 lines
fs/resctrl/rdtgroup.c +3
-0@@ -1930,6 +1930,123 @@ static int event_filter_show(struct kernfs_open_file *of, struct seq_file *seq,return 0 ; } +/** +* resctrl_group_assign - Update the counter assignments for the event in +* a group.
This name is very generic with an unexpected namespace. "rdtgroup_" prefix is often used for a function that operates on a rdtgroup. This can thus be "rdtgroup_assign_cntr()".
Quoted 11 lines
fs/resctrl/rdtgroup.c +11
-0+* @r: Resource to which update needs to be done. +* @rdtgrp: Resctrl group. +* @evtid: Event ID. +* @evt_cfg: Event configuration value. +*/ +static int resctrl_group_assign (struct rdt_resource *r,struct rdtgroup *rdtgrp, +enum resctrl_event_id evtid,u32 evt_cfg) +{ +struct rdt_mon_domain *d; +int cntr_id; +
fs/resctrl/rdtgroup.c static int event_filter_show(struct kernfs_open_file *of, struct seq_file *seq, · line 1956
Quoted 10 lines
fs/resctrl/rdtgroup.c +10
-0+list_for_each_entry (d, &r->mon_domains, hdr.list) { + cntr_id =mbm_cntr_get (r, d, rdtgrp, evtid); +if (cntr_id >=0 && d->cntr_cfg[cntr_id].evt_cfg != evt_cfg) { + d->cntr_cfg[cntr_id].evt_cfg = evt_cfg; +resctrl_arch_config_cntr (r, d, evtid, rdtgrp->mon.rmid, + rdtgrp->closid, cntr_id, evt_cfg, true); + } + } + +return 0 ;
Can just return void?
fs/resctrl/rdtgroup.c static int event_filter_show(struct kernfs_open_file *of, struct seq_file *seq, · line 1961
Quoted 5 lines
fs/resctrl/rdtgroup.c +5
-0+} + +/** +* resctrl_update_assign - Update the counter assignments for the event for all +* the groups.
Again very generic with "update" and "assign" that seem redundant? How about "resctrl_assign_cntr_allrdtgrp()"?
fs/resctrl/rdtgroup.c static int event_filter_show(struct kernfs_open_file *of, struct seq_file *seq, · line 1939
Quoted 3 lines
fs/resctrl/rdtgroup.c +3
-0+* @r: Resource to which update needs to be done. +* @evtid: Event ID. +* @evt_cfg: Event configuration value.
Why are both event ID and evt_cfg needed? Could just passing mon_evt simplify this?
fs/resctrl/rdtgroup.c static int event_filter_show(struct kernfs_open_file *of, struct seq_file *seq, · line 1971
Quoted 7 lines
fs/resctrl/rdtgroup.c +7
-0+*/ +static int resctrl_update_assign (struct rdt_resource *r,enum resctrl_event_id evtid, +u32 evt_cfg) +{ +struct rdtgroup *prgrp, *crgrp; + +/* Check if the cntr_id is associated to the event type updated */
Comment does not match code.
fs/resctrl/rdtgroup.c static int event_filter_show(struct kernfs_open_file *of, struct seq_file *seq, · line 1977
Quoted 6 lines
fs/resctrl/rdtgroup.c +6
-0+list_for_each_entry (prgrp, &rdt_all_groups, rdtgroup_list) { +resctrl_group_assign (r, prgrp, evtid, evt_cfg); + +list_for_each_entry (crgrp, &prgrp->mon.crdtgrp_list, mon.crdtgrp_list) { +resctrl_group_assign (r, crgrp, evtid, evt_cfg); + }
Unnecessary braces?
fs/resctrl/rdtgroup.c static int event_filter_show(struct kernfs_open_file *of, struct seq_file *seq, · line 1956
Quoted 3 lines
fs/resctrl/rdtgroup.c +3
-0+ } + +return 0 ;
return void?
fs/resctrl/rdtgroup.c static int event_filter_show(struct kernfs_open_file *of, struct seq_file *seq, · line 1993
Quoted 13 lines
fs/resctrl/rdtgroup.c +13
-0+} + +static int resctrl_process_configs (char *tok,u32 *val) +{ +char *evt_str; +bool found; +int i; + +next_config: +if (!tok || tok[0 ] =='\0' ) +return 0 ; + +/* Start processing the strings for each event type */
Does comment intend to describe one iteration or all iterations? Also, "event type" -> "memory transaction"?
fs/resctrl/rdtgroup.c static int event_filter_show(struct kernfs_open_file *of, struct seq_file *seq, · line 1998
Quoted 5 lines
fs/resctrl/rdtgroup.c +5
-0+ evt_str =strim (strsep (&tok,"," )); + found = false; +for (i =0 ; i <NUM_MBM_EVT_VALUES ; i++) { +if (!strcmp (mbm_evt_values[i].evt_name, evt_str)) { + *val |= mbm_evt_values[i].evt_val;
check spacing.
fs/resctrl/rdtgroup.c static int event_filter_show(struct kernfs_open_file *of, struct seq_file *seq, · line 2006
Quoted 8 lines
fs/resctrl/rdtgroup.c +8
-0+ found = true; +break ; + } + } + +if (!found) { +rdt_last_cmd_printf ("Invalid event type %s\n" , evt_str); +return -EINVAL ;
Looks like this will return partially initialized data. Please use a local variable in which to gather the new configuration and only assign that to provided pointer on success.
Quoted 18 lines
fs/resctrl/rdtgroup.c +18
-0+ } + + goto next_config; +} + +static ssize_t event_filter_write (struct kernfs_open_file *of,char *buf, +size_t nbytes,loff_t off) +{ +struct rdt_resource *r =resctrl_arch_get_resource (RDT_RESOURCE_L3 ); +struct mon_evt *mevt =rdt_kn_parent_priv (of->kn); +u32 evt_cfg =0 ; +int ret =0 ; + +/* Valid input requires a trailing newline */ +if (nbytes ==0 || buf[nbytes -1 ] !='\n' ) +return -EINVAL ; + + buf[nbytes -1 ] ='\0' ;
fs/resctrl/rdtgroup.c static int event_filter_show(struct kernfs_open_file *of, struct seq_file *seq, · line 2034
Quoted 10 lines
fs/resctrl/rdtgroup.c +10
-0+ +cpus_read_lock (); +mutex_lock (&rdtgroup_mutex); + +rdt_last_cmd_clear (); + +if (!resctrl_arch_mbm_cntr_assign_enabled (r)) { +rdt_last_cmd_puts ("mbm_cntr_assign mode is not enabled\n" ); + ret = -EINVAL ; + goto unlock_out;
"grep goto fs/resctrl/rdtgroup.c"
fs/resctrl/rdtgroup.c static int event_filter_show(struct kernfs_open_file *of, struct seq_file *seq, · line 2039
Quoted 5 lines
fs/resctrl/rdtgroup.c +5
-0+ } + + ret =resctrl_process_configs (buf, &evt_cfg); +if (!ret && mevt->evt_val != evt_cfg) { + mevt->evt_val = evt_cfg;
ah ... here it is. hmmm ... but it is mon_evt::evt_cfg, no? ah, fixed in next patch.
I still seem to be missing something because I expected mon_evt::evt_cfg of mbm_total_bytes and mbm_local_bytes to be initialized with a starting default. I missed where this is done in this series.
Quoted 15 lines
fs/resctrl/rdtgroup.c +10
-0+resctrl_update_assign (r, mevt->evtid, evt_cfg); + } + +unlock_out: +mutex_unlock (&rdtgroup_mutex); +cpus_read_unlock (); + +return ret ?: nbytes; +} +/* rdtgroup information files for one cache resource. */ static struct rftype res_common_files[] = { {@@ -2056,9 +2173,10 @@ static struct rftype res_common_files[] = { },
fs/resctrl/rdtgroup.c static struct rftype res_common_files[] = { · line 2182
Quoted 10 lines
fs/resctrl/rdtgroup.c +2
-1{ .name = "event_filter", - .mode = 0444, + .mode = 0644, .kf_ops = &rdtgroup_kf_single_ops, .seq_show = event_filter_show, + .write = event_filter_write, }, { .name = "mbm_assign_mode",
Reinette
Re: [PATCH v13 20/27] x86/resctrl: Provide interface to update the event configurations
Hi Reinette,
Quoted 9 lines · Reinette Chatre
Hi Babu,
Quoted 3 lines · Babu Moger
Users can modify the event configuration by writing to the
event_filterinterface file. The event configurations formbm_cntr_assignmode are located in/sys/fs/resctrl/info/event_configs/.heh ... looks like you also started thinking that "
event_configs" is a better name (also missingL3_MON).
Interesting... 'event_configs' seems to have made its way in somehow. Alright, I’ll go ahead and add L3_MON."
Quoted 46 lines
Quoted 35 lines
Update the assignments of all groups when the event configuration is modified.
Example:
$ cd /sys/fs/resctrl/$ cat info/L3_MON/counter_configs/mbm_local_bytes/event_filter local_reads,local_non_temporal_writes,local_reads_slow_memory$ echo "local_reads,local_non_temporal_writes" > info/L3_MON/counter_configs/mbm_total_bytes/event_filter$ cat info/L3_MON/counter_configs/mbm_total_bytes/event_filter local_reads,local_non_temporal_writesSigned-off-by Babu Moger <[email protected]>---
v13: Updated changelog for imperative mode.
Added function description in the prototype.
Updated the user docresctrl.rstto address few feedback.
Resolved conflicts caused by the recent FS/ARCH code restructure.
Therdtgroup.c/monitor.cfile has now been split between the FS and ARCH directories.v12: New patch to modify event configurations.
--- Documentation/filesystems/resctrl.rst | 12 +++ fs/resctrl/rdtgroup.c | 120 +++++++++++++++++++++++++- 2 files changed, 131 insertions(+), 1 deletion(-)Documentation/filesystems/resctrl.rst +0
-0...
Quoted 8 lines
fs/resctrl/rdtgroup.c +0
-0@@ -1930,6 +1930,123 @@ static int event_filter_show(struct kernfs_open_file *of, struct seq_file *seq,return 0 ; }
Quoted 7 lines
fs/resctrl/rdtgroup.c static int event_filter_show(struct kernfs_open_file *of, struct seq_file *seq, · line 1935
Quoted 3 lines
fs/resctrl/rdtgroup.c +3
-0+/** +* resctrl_group_assign - Update the counter assignments for the event in +* a group. This name is very generic with an unexpected namespace. "
rdtgroup_" prefix is often used for a function that operates on a rdtgroup. This can thus be "rdtgroup_assign_cntr()".
Sure.
Quoted 16 lines
Quoted 16 lines
fs/resctrl/rdtgroup.c +15
-0+* @r: Resource to which update needs to be done. +* @rdtgrp: Resctrl group. +* @evtid: Event ID. +* @evt_cfg: Event configuration value. +*/ +static int resctrl_group_assign (struct rdt_resource *r,struct rdtgroup *rdtgrp, +enum resctrl_event_id evtid,u32 evt_cfg) +{ +struct rdt_mon_domain *d; +int cntr_id; + +list_for_each_entry (d, &r->mon_domains, hdr.list) { + cntr_id =mbm_cntr_get (r, d, rdtgrp, evtid); +if (cntr_id >=0 && d->cntr_cfg[cntr_id].evt_cfg != evt_cfg) { + d->cntr_cfg[cntr_id].evt_cfg = evt_cfg;
Quoted 8 lines
fs/resctrl/rdtgroup.c static int event_filter_show(struct kernfs_open_file *of, struct seq_file *seq, · line 1956
Quoted 6 lines
fs/resctrl/rdtgroup.c +6
-0+resctrl_arch_config_cntr (r, d, evtid, rdtgrp->mon.rmid, + rdtgrp->closid, cntr_id, evt_cfg, true); + } + } + +return 0 ;Can just return void?
sure.
Quoted 9 lines
fs/resctrl/rdtgroup.c static int event_filter_show(struct kernfs_open_file *of, struct seq_file *seq, · line 1961
Quoted 6 lines
fs/resctrl/rdtgroup.c +5
-0+} + +/** +* resctrl_update_assign - Update the counter assignments for the event for all +* the groups. Again very generic with "update" and "assign" that seem redundant? How about "
resctrl_assign_cntr_allrdtgrp()"?
Yes.
Quoted 7 lines
fs/resctrl/rdtgroup.c static int event_filter_show(struct kernfs_open_file *of, struct seq_file *seq, · line 1939
Quoted 4 lines
fs/resctrl/rdtgroup.c +3
-0+* @r: Resource to which update needs to be done. +* @evtid: Event ID. +* @evt_cfg: Event configuration value. Why are both event ID and
evt_cfgneeded? Could just passingmon_evtsimplify this?
Quoted 9 lines
fs/resctrl/rdtgroup.c static int event_filter_show(struct kernfs_open_file *of, struct seq_file *seq, · line 1971
Quoted 7 lines
fs/resctrl/rdtgroup.c +7
-0+*/ +static int resctrl_update_assign (struct rdt_resource *r,enum resctrl_event_id evtid, +u32 evt_cfg) +{ +struct rdtgroup *prgrp, *crgrp; + +/* Check if the cntr_id is associated to the event type updated */ Comment does not match code.
Will correct it.
Quoted 9 lines
fs/resctrl/rdtgroup.c static int event_filter_show(struct kernfs_open_file *of, struct seq_file *seq, · line 1977
Quoted 7 lines
fs/resctrl/rdtgroup.c +6
-0+list_for_each_entry (prgrp, &rdt_all_groups, rdtgroup_list) { +resctrl_group_assign (r, prgrp, evtid, evt_cfg); + +list_for_each_entry (crgrp, &prgrp->mon.crdtgrp_list, mon.crdtgrp_list) { +resctrl_group_assign (r, crgrp, evtid, evt_cfg); + }Unnecessary braces?
Yes.
Quoted 7 lines
fs/resctrl/rdtgroup.c static int event_filter_show(struct kernfs_open_file *of, struct seq_file *seq, · line 1956
Quoted 4 lines
fs/resctrl/rdtgroup.c +3
-0+ } + +return 0 ;return void?
Sure.
Quoted 8 lines
Quoted 8 lines
fs/resctrl/rdtgroup.c +8
-0+} + +static int resctrl_process_configs (char *tok,u32 *val) +{ +char *evt_str; +bool found; +int i; +
Quoted 8 lines
fs/resctrl/rdtgroup.c static int event_filter_show(struct kernfs_open_file *of, struct seq_file *seq, · line 1993
Quoted 5 lines
fs/resctrl/rdtgroup.c +5
-0+next_config: +if (!tok || tok[0 ] =='\0' ) +return 0 ; + +/* Start processing the strings for each event type */ Does comment intend to describe one iteration or all iterations? Also, "event type" -> "memory transaction"?
Sure.
Quoted 8 lines
fs/resctrl/rdtgroup.c static int event_filter_show(struct kernfs_open_file *of, struct seq_file *seq, · line 1998
Quoted 6 lines
fs/resctrl/rdtgroup.c +5
-0+ evt_str =strim (strsep (&tok,"," )); + found = false; +for (i =0 ; i <NUM_MBM_EVT_VALUES ; i++) { +if (!strcmp (mbm_evt_values[i].evt_name, evt_str)) { + *val |= mbm_evt_values[i].evt_val;check spacing.
ok.
Quoted 13 lines
fs/resctrl/rdtgroup.c static int event_filter_show(struct kernfs_open_file *of, struct seq_file *seq, · line 2006
Quoted 9 lines
fs/resctrl/rdtgroup.c +8
-0+ found = true; +break ; + } + } + +if (!found) { +rdt_last_cmd_printf ("Invalid event type %s\n" , evt_str); +return -EINVAL ;Looks like this will return partially initialized data. Please use a local variable in which to gather the new configuration and only assign that to provided pointer on success.
Yes. Makes sense.
Quoted 23 lines
Quoted 23 lines
fs/resctrl/rdtgroup.c +22
-0+ } + + goto next_config; +} + +static ssize_t event_filter_write (struct kernfs_open_file *of,char *buf, +size_t nbytes,loff_t off) +{ +struct rdt_resource *r =resctrl_arch_get_resource (RDT_RESOURCE_L3 ); +struct mon_evt *mevt =rdt_kn_parent_priv (of->kn); +u32 evt_cfg =0 ; +int ret =0 ; + +/* Valid input requires a trailing newline */ +if (nbytes ==0 || buf[nbytes -1 ] !='\n' ) +return -EINVAL ; + + buf[nbytes -1 ] ='\0' ; + +cpus_read_lock (); +mutex_lock (&rdtgroup_mutex); +
Quoted 8 lines
fs/resctrl/rdtgroup.c static int event_filter_show(struct kernfs_open_file *of, struct seq_file *seq, · line 2034
Quoted 6 lines
fs/resctrl/rdtgroup.c +6
-0+rdt_last_cmd_clear (); + +if (!resctrl_arch_mbm_cntr_assign_enabled (r)) { +rdt_last_cmd_puts ("mbm_cntr_assign mode is not enabled\n" ); + ret = -EINVAL ; + goto unlock_out;"grep goto
fs/resctrl/rdtgroup.c"
Will change it to "out_unlock".
Quoted 9 lines
fs/resctrl/rdtgroup.c static int event_filter_show(struct kernfs_open_file *of, struct seq_file *seq, · line 2039
Quoted 6 lines
fs/resctrl/rdtgroup.c +5
-0+ } + + ret =resctrl_process_configs (buf, &evt_cfg); +if (!ret && mevt->evt_val != evt_cfg) { + mevt->evt_val = evt_cfg;ah ... here it is. hmmm ... but it is
mon_evt::evt_cfg, no? ah, fixed in next patch.
Yea. My bad.
Quoted 4 lines
I still seem to be missing something because I expected
mon_evt::evt_cfgofmbm_total_bytesandmbm_local_bytesto be initialized with a starting default. I missed where this is done in this series.
Its not done in this series. Will do in next revision.
Quoted 29 lines
Quoted 16 lines
fs/resctrl/rdtgroup.c +10
-0+resctrl_update_assign (r, mevt->evtid, evt_cfg); + } + +unlock_out: +mutex_unlock (&rdtgroup_mutex); +cpus_read_unlock (); + +return ret ?: nbytes; +} +/* rdtgroup information files for one cache resource. */ static struct rftype res_common_files[] = { {@@ -2056,9 +2173,10 @@ static struct rftype res_common_files[] = { },fs/resctrl/rdtgroup.c static struct rftype res_common_files[] = { · line 2182
Quoted 10 lines
fs/resctrl/rdtgroup.c +2
-1{ .name = "event_filter", - .mode = 0444, + .mode = 0644, .kf_ops = &rdtgroup_kf_single_ops, .seq_show = event_filter_show, + .write = event_filter_write, }, { .name = "mbm_assign_mode",Reinette
Babu Moger
reviewerPATCH v13 21/27 x86/resctrl: Introduce mbm_assign_on_mkdir to configure assignments
In reply to Babu MogerThe mbm_cntr_assign mode provides an option to the user to assign a counter to an RMID, event pair and monitor the bandwidth as long as the counter is assigned.
Introduce a configuration option to automatically assign counter IDs when a resctrl group is created, provided the counters are available. By default, this option is enabled at boot.
---
v13: Added Suggested-by tag.
Resolved conflicts caused by the recent FS/ARCH code restructure.
The rdtgroup.c/monitor.c file has now been split between the FS and ARCH directories.
v12: New patch. Added after the discussion on the list.
https://lore.kernel.org/lkml/CALPaoCh8siZKjL_3yvOYGL4cF_n_38KpUFgHVGbQ86nD+Q2_SA@mail.gmail.com/
--- Documentation/filesystems/resctrl.rst | 10 ++++++ fs/resctrl/monitor.c | 2 ++ fs/resctrl/rdtgroup.c | 44 +++++++++++++++++++++++++-- include/linux/resctrl.h | 2 ++ 4 files changed, 56 insertions(+), 2 deletions(-)
Documentation/filesystems/resctrl.rst +10 -0
@@ -348,6 +348,16 @@ with the following files: # cat /sys/fs/resctrl/info/L3_MON/counter_configs/mbm_total_bytes/event_filter local_reads, local_non_temporal_writes +"mbm_assign_on_mkdir": + Automatically assign the monitoring counters on resctrl group creation + if the counters are available. It is enabled by default on boot and users + can disable by writing to the interface. + :: + + # echo 0 > /sys/fs/resctrl/info/L3_MON/mbm_assign_on_mkdir + # cat /sys/fs/resctrl/info/L3_MON/mbm_assign_on_mkdir + 0 + "max_threshold_occupancy": Read/write file provides the largest value (in bytes) at which a previously used LLC_occupancy
fs/resctrl/monitor.c +2 -0
@@ -933,6 +933,8 @@ int resctrl_mon_resource_init(void)resctrl_file_fflags_init ("available_mbm_cntrs" ,RFTYPE_MON_INFO |RFTYPE_RES_CACHE );resctrl_file_fflags_init ("event_filter" ,RFTYPE_ASSIGN_CONFIG ); +resctrl_file_fflags_init ("mbm_assign_on_mkdir" ,RFTYPE_MON_INFO | +RFTYPE_RES_CACHE ); }return 0 ;
fs/resctrl/rdtgroup.c +42 -2
62 lines not on this page · raw message
include/linux/resctrl.h +2 -0
@@ -278,6 +278,7 @@ enum resctrl_schema_fmt {* monitoring events can be configured. * @num_mbm_cntrs: Number of assignable monitoring counters * @mbm_cntr_assignable:Is system capable of supporting monitor assignment? +* @mbm_assign_on_mkdir:Auto enable monitor assignment on mkdir? * @evt_list: List of monitoring events */ struct resctrl_mon {
@@ -285,6 +286,7 @@ struct resctrl_mon {unsigned int mbm_cfg_mask;int num_mbm_cntrs;bool mbm_cntr_assignable; +bool mbm_assign_on_mkdir;struct list_head evt_list; };
Signature · 1 line
2.34.1
Reinette Chatre
maintainerRe: [PATCH v13 21/27] x86/resctrl: Introduce mbm_assign_on_mkdir to configure assignments
Hi Babu,
Quoted 5 lines · Babu Moger
The
mbm_cntr_assignmode provides an option to the user to assign a counter to an RMID, event pair and monitor the bandwidth as long as the counter is assigned.Introduce a configuration option to automatically assign counter IDs
"assign counter IDs" -> "assign counter IDs to <what?>"
Quoted 24 lines
when a resctrl group is created, provided the counters are available. By default, this option is enabled at boot.
Suggested-by Peter Newman <[email protected]>Signed-off-by Babu Moger <[email protected]>---
v13: Added Suggested-by tag.
Resolved conflicts caused by the recent FS/ARCH code restructure.
Therdtgroup.c/monitor.cfile has now been split between the FS and ARCH directories.v12: New patch. Added after the discussion on the list.
https://lore.kernel.org/lkml/CALPaoCh8siZKjL_3yvOYGL4cF_n_38KpUFgHVGbQ86nD+Q2_SA@mail.gmail.com/--- Documentation/filesystems/resctrl.rst | 10 ++++++ fs/resctrl/monitor.c | 2 ++ fs/resctrl/rdtgroup.c | 44 +++++++++++++++++++++++++-- include/linux/resctrl.h | 2 ++ 4 files changed, 56 insertions(+), 2 deletions(-)Documentation/filesystems/resctrl.rst +0
-0
Documentation/filesystems/resctrl.rst with the following files: · line 352
Quoted 5 lines
Documentation/filesystems/resctrl.rst +2
-0@@ -348,6 +348,16 @@ with the following files: # cat /sys/fs/resctrl/info/L3_MON/counter_configs/mbm_total_bytes/event_filter local_reads, local_non_temporal_writes +"mbm_assign_on_mkdir": + Automatically assign the monitoring counters on resctrl group creation
assign the monitoring counters to what?
Documentation/filesystems/resctrl.rst with the following files: · line 359
Quoted 7 lines
Documentation/filesystems/resctrl.rst +7
-0+ if the counters are available. It is enabled by default on boot and users + can disable by writing to the interface. + :: + + # echo 0 > /sys/fs/resctrl/info/L3_MON/mbm_assign_on_mkdir + # cat /sys/fs/resctrl/info/L3_MON/mbm_assign_on_mkdir + 0
Please be explicit in docs what possible values are and what they mean.
Quoted 22 lines
Documentation/filesystems/resctrl.rst +1
-0+ "max_threshold_occupancy": Read/write file provides the largest value (in bytes) at which a previously used LLC_occupancyfs/resctrl/monitor.c +2
-0@@ -933,6 +933,8 @@ int resctrl_mon_resource_init(void)resctrl_file_fflags_init ("available_mbm_cntrs" ,RFTYPE_MON_INFO |RFTYPE_RES_CACHE );resctrl_file_fflags_init ("event_filter" ,RFTYPE_ASSIGN_CONFIG ); +resctrl_file_fflags_init ("mbm_assign_on_mkdir" ,RFTYPE_MON_INFO | +RFTYPE_RES_CACHE ); }return 0 ;fs/resctrl/rdtgroup.c +0
-0
fs/resctrl/rdtgroup.c static ssize_t event_filter_write(struct kernfs_open_file *of, char *buf, · line 2042
Quoted 10 lines
fs/resctrl/rdtgroup.c +2
-2@@ -2035,8 +2035,8 @@ static ssize_t event_filter_write(struct kernfs_open_file *of, char *buf, } ret =resctrl_process_configs (buf, &evt_cfg); -if (!ret && mevt->evt_val != evt_cfg) { - mevt->evt_val = evt_cfg; +if (!ret && mevt->evt_cfg != evt_cfg) { + mevt->evt_cfg = evt_cfg;resctrl_update_assign (r, mevt->evtid, evt_cfg); }
Needs to be squashed.
Quoted 16 lines
fs/resctrl/rdtgroup.c +12
-0@@ -2047,6 +2047,39 @@ static ssize_t event_filter_write(struct kernfs_open_file *of, char *buf,return ret ?: nbytes; } +static int resctrl_mbm_assign_on_mkdir_show (struct kernfs_open_file *of, +struct seq_file *s,void *v) +{ +struct rdt_resource *r =rdt_kn_parent_priv (of->kn); + +seq_printf (s,"%u\n" , r->mon.mbm_assign_on_mkdir); + +return 0 ; +} + +static ssize_t resctrl_mbm_assign_on_mkdir_write (struct kernfs_open_file *of, +char *buf,size_t nbytes,loff_t off)
fs/resctrl/rdtgroup.c static ssize_t event_filter_write(struct kernfs_open_file *of, char *buf, · line 2071
Quoted 10 lines
fs/resctrl/rdtgroup.c +10
-0+{ +struct rdt_resource *r =rdt_kn_parent_priv (of->kn); +bool value; +int ret; + + ret =kstrtobool (buf, &value); +if (ret) +return ret; + +cpus_read_lock ();
not traversing the domain list so hotplug lock not needed.
fs/resctrl/rdtgroup.c static ssize_t event_filter_write(struct kernfs_open_file *of, char *buf, · line 2072
Quoted 1 line
fs/resctrl/rdtgroup.c +1
-0+mutex_lock (&rdtgroup_mutex);
rdtgroup_mutex seems only needed because the message buffer is cleared below, and this is why it is not required in the show()?
Quoted 32 lines
fs/resctrl/rdtgroup.c +17
-0+rdt_last_cmd_clear (); + + r->mon.mbm_assign_on_mkdir = value; + +mutex_unlock (&rdtgroup_mutex); +cpus_read_unlock (); + +return ret ?: nbytes; +} +/* rdtgroup information files for one cache resource. */ static struct rftype res_common_files[] = { {@@ -2056,6 +2089,13 @@ static struct rftype res_common_files[] = { .seq_show = rdt_last_cmd_status_show, .fflags = RFTYPE_TOP_INFO, }, + { + .name = "mbm_assign_on_mkdir", + .mode = 0644, + .kf_ops = &rdtgroup_kf_single_ops, + .seq_show = resctrl_mbm_assign_on_mkdir_show, + .write = resctrl_mbm_assign_on_mkdir_write, + }, { .name = "num_closids", .mode = 0444,include/linux/resctrl.h +0
-0
include/linux/resctrl.h enum resctrl_schema_fmt { · line 281
Quoted 4 lines
include/linux/resctrl.h +1
-0@@ -278,6 +278,7 @@ enum resctrl_schema_fmt {* monitoring events can be configured. * @num_mbm_cntrs: Number of assignable monitoring counters * @mbm_cntr_assignable:Is system capable of supporting monitor assignment? +* @mbm_assign_on_mkdir:Auto enable monitor assignment on mkdir?
How is "monitor assignment" different from "counter assignment"?
include/linux/resctrl.h struct resctrl_mon { · line 292
Quoted 11 lines
include/linux/resctrl.h +1
-0* @evt_list: List of monitoring events */ struct resctrl_mon {@@ -285,6 +286,7 @@ struct resctrl_mon {unsigned int mbm_cfg_mask;int num_mbm_cntrs;bool mbm_cntr_assignable; +bool mbm_assign_on_mkdir;struct list_head evt_list; };
Reinette
Re: [PATCH v13 21/27] x86/resctrl: Introduce mbm_assign_on_mkdir to configure assignments
Hi Reinette,
Quoted 10 lines · Reinette Chatre
Hi Babu,
Quoted 5 lines · Babu Moger
The
mbm_cntr_assignmode provides an option to the user to assign a counter to an RMID, event pair and monitor the bandwidth as long as the counter is assigned.Introduce a configuration option to automatically assign counter IDs
"assign counter IDs" -> "assign counter IDs to <what?>"
"Introduce a configuration option to automatically assign counter IDs to to an RMID, event pair when a resctrl group is created, provided the counter IDs are available."
Quoted 28 lines
Quoted 28 lines
when a resctrl group is created, provided the counters are available. By default, this option is enabled at boot.
Suggested-by Peter Newman <[email protected]>Signed-off-by Babu Moger <[email protected]>---
v13: Added Suggested-by tag.
Resolved conflicts caused by the recent FS/ARCH code restructure.
Therdtgroup.c/monitor.cfile has now been split between the FS and ARCH directories.v12: New patch. Added after the discussion on the list.
https://lore.kernel.org/lkml/CALPaoCh8siZKjL_3yvOYGL4cF_n_38KpUFgHVGbQ86nD+Q2_SA@mail.gmail.com/--- Documentation/filesystems/resctrl.rst | 10 ++++++ fs/resctrl/monitor.c | 2 ++ fs/resctrl/rdtgroup.c | 44 +++++++++++++++++++++++++-- include/linux/resctrl.h | 2 ++ 4 files changed, 56 insertions(+), 2 deletions(-)Documentation/filesystems/resctrl.rst +0
-0@@ -348,6 +348,16 @@ with the following files: # cat /sys/fs/resctrl/info/L3_MON/counter_configs/mbm_total_bytes/event_filter local_reads, local_non_temporal_writes
Quoted 4 lines
Documentation/filesystems/resctrl.rst with the following files: · line 352
Quoted 2 lines
Documentation/filesystems/resctrl.rst +2
-0+"mbm_assign_on_mkdir": + Automatically assign the monitoring counters on resctrl group creationassign the monitoring counters to what?
"Automatically assign counter IDs to an RMID, event pair on resctrl group creation if the counter IDs are available. It is enabled by default on boot and users can disable by writing to the interface."
Quoted 9 lines
Documentation/filesystems/resctrl.rst with the following files: · line 359
Quoted 7 lines
Documentation/filesystems/resctrl.rst +7
-0+ if the counters are available. It is enabled by default on boot and users + can disable by writing to the interface. + :: + + # echo 0 > /sys/fs/resctrl/info/L3_MON/mbm_assign_on_mkdir + # cat /sys/fs/resctrl/info/L3_MON/mbm_assign_on_mkdir + 0Please be explicit in docs what possible values are and what they mean.
Sure. I can print "enabled" or "disabled".
Quoted 25 lines
Quoted 25 lines
Documentation/filesystems/resctrl.rst +1
-0+ "max_threshold_occupancy": Read/write file provides the largest value (in bytes) at which a previously used LLC_occupancyfs/resctrl/monitor.c +2
-0@@ -933,6 +933,8 @@ int resctrl_mon_resource_init(void)resctrl_file_fflags_init ("available_mbm_cntrs" ,RFTYPE_MON_INFO |RFTYPE_RES_CACHE );resctrl_file_fflags_init ("event_filter" ,RFTYPE_ASSIGN_CONFIG ); +resctrl_file_fflags_init ("mbm_assign_on_mkdir" ,RFTYPE_MON_INFO | +RFTYPE_RES_CACHE ); }return 0 ;fs/resctrl/rdtgroup.c +0
-0@@ -2035,8 +2035,8 @@ static ssize_t event_filter_write(struct kernfs_open_file *of, char *buf, }
Quoted 10 lines
fs/resctrl/rdtgroup.c static ssize_t event_filter_write(struct kernfs_open_file *of, char *buf, · line 2042
Quoted 8 lines
fs/resctrl/rdtgroup.c +2
-2ret =resctrl_process_configs (buf, &evt_cfg); -if (!ret && mevt->evt_val != evt_cfg) { - mevt->evt_val = evt_cfg; +if (!ret && mevt->evt_cfg != evt_cfg) { + mevt->evt_cfg = evt_cfg;resctrl_update_assign (r, mevt->evtid, evt_cfg); }Needs to be squashed.
Sure.
Quoted 21 lines
Quoted 21 lines
fs/resctrl/rdtgroup.c +16
-0@@ -2047,6 +2047,39 @@ static ssize_t event_filter_write(struct kernfs_open_file *of, char *buf,return ret ?: nbytes; } +static int resctrl_mbm_assign_on_mkdir_show (struct kernfs_open_file *of, +struct seq_file *s,void *v) +{ +struct rdt_resource *r =rdt_kn_parent_priv (of->kn); + +seq_printf (s,"%u\n" , r->mon.mbm_assign_on_mkdir); + +return 0 ; +} + +static ssize_t resctrl_mbm_assign_on_mkdir_write (struct kernfs_open_file *of, +char *buf,size_t nbytes,loff_t off) +{ +struct rdt_resource *r =rdt_kn_parent_priv (of->kn); +bool value; +int ret;
Quoted 8 lines
fs/resctrl/rdtgroup.c static ssize_t event_filter_write(struct kernfs_open_file *of, char *buf, · line 2071
Quoted 6 lines
fs/resctrl/rdtgroup.c +6
-0+ + ret =kstrtobool (buf, &value); +if (ret) +return ret; + +cpus_read_lock ();not traversing the domain list so hotplug lock not needed.
ok. Sure.
Quoted 5 lines
fs/resctrl/rdtgroup.c static ssize_t event_filter_write(struct kernfs_open_file *of, char *buf, · line 2072
Quoted 2 lines
fs/resctrl/rdtgroup.c +1
-0+mutex_lock (&rdtgroup_mutex);
rdtgroup_mutexseems only needed because the message buffer is cleared below, and this is why it is not required in theshow()?
Hmm. I didnt think about that. Do you think it is required?
Quoted 31 lines
Quoted 31 lines
fs/resctrl/rdtgroup.c +17
-0+rdt_last_cmd_clear (); + + r->mon.mbm_assign_on_mkdir = value; + +mutex_unlock (&rdtgroup_mutex); +cpus_read_unlock (); + +return ret ?: nbytes; +} +/* rdtgroup information files for one cache resource. */ static struct rftype res_common_files[] = { {@@ -2056,6 +2089,13 @@ static struct rftype res_common_files[] = { .seq_show = rdt_last_cmd_status_show, .fflags = RFTYPE_TOP_INFO, }, + { + .name = "mbm_assign_on_mkdir", + .mode = 0644, + .kf_ops = &rdtgroup_kf_single_ops, + .seq_show = resctrl_mbm_assign_on_mkdir_show, + .write = resctrl_mbm_assign_on_mkdir_write, + }, { .name = "num_closids", .mode = 0444,include/linux/resctrl.h +0
-1--- a/include/linux/resctrl.h
Quoted 8 lines
include/linux/resctrl.h enum resctrl_schema_fmt { · line 281
Quoted 6 lines
+++
b/include/linux/resctrl.hinclude/linux/resctrl.h +1
-0@@ -278,6 +278,7 @@ enum resctrl_schema_fmt {* monitoring events can be configured. * @num_mbm_cntrs: Number of assignable monitoring counters * @mbm_cntr_assignable:Is system capable of supporting monitor assignment? +* @mbm_assign_on_mkdir:Auto enable monitor assignment on mkdir? How is "monitor assignment" different from "counter assignment"?
I should be:
"Auto enable counter ID assignment on mkdir"
Quoted 15 lines
include/linux/resctrl.h struct resctrl_mon { · line 292
Quoted 12 lines
include/linux/resctrl.h +1
-0* @evt_list: List of monitoring events */ struct resctrl_mon {@@ -285,6 +286,7 @@ struct resctrl_mon {unsigned int mbm_cfg_mask;int num_mbm_cntrs;bool mbm_cntr_assignable; +bool mbm_assign_on_mkdir;struct list_head evt_list; };Reinette
Thanks
Babu
Reinette Chatre
maintainerRe: [PATCH v13 21/27] x86/resctrl: Introduce mbm_assign_on_mkdir to configure assignments
Hi Babu,
Quoted 10 lines · Babu
Hi Reinette,
Quoted 7 lines · Reinette Chatre
Hi Babu,
Quoted 4 lines · Babu Moger
The
mbm_cntr_assignmode provides an option to the user to assign a counter to an RMID, event pair and monitor the bandwidth as long as the counter is assigned.
Quoted 5 lines
Quoted 3 lines
Quoted 1 line
Introduce a configuration option to automatically assign counter IDs
"assign counter IDs" -> "assign counter IDs to <what?>"
"Introduce a configuration option to automatically assign counter IDs to to an RMID, event pair when a resctrl group is created, provided the counter IDs are available."
Stating that "counter IDs" (plural) are assigned to "an RMID, event pair" (singular) can be confusing.
How about something like (please feel free to improve):
"Introduce a user-configurable option that determines if a counter will automatically be assigned to an RMID, event pair when its associated monitor group is created via mkdir."
Quoted 33 lines
Quoted 21 lines
Quoted 21 lines
when a resctrl group is created, provided the counters are available. By default, this option is enabled at boot.
Suggested-by Peter Newman <[email protected]>Signed-off-by Babu Moger <[email protected]>---
v13: Added Suggested-by tag.
Resolved conflicts caused by the recent FS/ARCH code restructure.
Therdtgroup.c/monitor.cfile has now been split between the FS and ARCH directories.v12: New patch. Added after the discussion on the list.
https://lore.kernel.org/lkml/CALPaoCh8siZKjL_3yvOYGL4cF_n_38KpUFgHVGbQ86nD+Q2_SA@mail.gmail.com/
---
Documentation/filesystems/resctrl.rst| 10 ++++++ fs/resctrl/monitor.c | 2 ++ fs/resctrl/rdtgroup.c | 44 +++++++++++++++++++++++++-- include/linux/resctrl.h | 2 ++4 files changed, 56 insertions(+), 2 deletions(-)Quoted 11 lines
Quoted 9 lines
Documentation/filesystems/resctrl.rst +0
-0@@ -348,6 +348,16 @@ with the following files: # cat /sys/fs/resctrl/info/L3_MON/counter_configs/mbm_total_bytes/event_filter
local_reads,local_non_temporal_writes+"mbm_assign_on_mkdir":
+ Automatically assign the monitoring counters on resctrl group creationassign the monitoring counters to what?
Quoted 1 line
"Automatically assign counter IDs to an RMID, event pair on resctrl group creation if the counter IDs are available. It is enabled by default on boot and users can disable by writing to the interface."
Same here, please take care with the plural/singular usage.
Quoted 9 lines
Quoted 9 lines
Quoted 8 lines
+ if the counters are available. It is enabled by default on boot and users + can disable by writing to the interface.
+ ::
+
+ # echo 0 >/sys/fs/resctrl/info/L3_MON/mbm_assign_on_mkdir+ # cat/sys/fs/resctrl/info/L3_MON/mbm_assign_on_mkdir
+ 0
Quoted 3 lines
Quoted 1 line
Please be explicit in docs what possible values are and what they mean.
Sure. I can print "enabled" or "disabled".
I am not requesting a change in user interface self but instead clear documentation about what the input/output values mean. Even if the interface changes to "enabled"/"disabled" I assume the interface will still accept boolean values? Compare to the "sparse_masks" documentation on how the possible values are explicitly documented.
...
Quoted 18 lines
Quoted 6 lines
Quoted 6 lines
Documentation/filesystems/resctrl.rst +6
-0+static ssize_t resctrl_mbm_assign_on_mkdir_write(struct kernfs_open_file *of, + char *buf, size_t nbytes, loff_t off) +{ + struct rdt_resource *r = rdt_kn_parent_priv(of->kn); + bool value; + int ret;Quoted 8 lines
Documentation/filesystems/resctrl.rst
Quoted 6 lines
Documentation/filesystems/resctrl.rst +6
-0+ + ret = kstrtobool(buf, &value); + if (ret) + return ret; + + cpus_read_lock();not traversing the domain list so hotplug lock not needed.
ok. Sure.
Quoted 1 line
Quoted 6 lines
Quoted 4 lines
Quoted 1 line
+
mutex_lock(&rdtgroup_mutex);
rdtgroup_mutexseems only needed because the message buffer is cleared below, and this is why it is not required in theshow()?Hmm. I didnt think about that. Do you think it is required?
It is certainly required to be able to call rdt_last_cmd_clear() and since it then covers mbm_assign_on_mkdir I would prefer symmetry in consistently acquiring rdtgroup_mutex on both read and write while resctrl is mounted. Note that there is also other read usage on resctrl mount that is done with mutex held. Having the mutex acquired consistently will help to keep things simple.
...
Quoted 12 lines
Quoted 11 lines
include/linux/resctrl.h enum resctrl_schema_fmt { · line 281
Quoted 9 lines
include/linux/resctrl.h +1
-0@@ -278,6 +278,7 @@ enum resctrl_schema_fmt { * monitoring events can be configured. * @num_mbm_cntrs: Number of assignable monitoring counters * @mbm_cntr_assignable:Is system capable of supporting monitor assignment? + * @mbm_assign_on_mkdir:Auto enable monitor assignment on mkdir?How is "monitor assignment" different from "counter assignment"?
Quoted 3 lines
I should be:
"Auto enable counter ID assignment on mkdir"
hmmm ... I do not think this is about "Auto enable".
How about something like "Automatic counter assignment during monitor group create via mkdir?" or "True if counters should automatically be assigned to MBM events of monitor groups created via mkdir."
Reinette
Re: [PATCH v13 21/27] x86/resctrl: Introduce mbm_assign_on_mkdir to configure assignments
Hi Reinette,
Quoted 17 lines · Reinette Chatre
Hi Babu,
Quoted 14 lines · Babu
Hi Reinette,
Quoted 10 lines · Reinette Chatre
Hi Babu,
Quoted 5 lines · Babu Moger
The
mbm_cntr_assignmode provides an option to the user to assign a counter to an RMID, event pair and monitor the bandwidth as long as the counter is assigned.Introduce a configuration option to automatically assign counter IDs
"assign counter IDs" -> "assign counter IDs to <what?>"
Quoted 8 lines
Quoted 1 line
"Introduce a configuration option to automatically assign counter IDs to to an RMID, event pair when a resctrl group is created, provided the counter IDs are available."
Stating that "counter IDs" (plural) are assigned to "an RMID, event pair" (singular) can be confusing.
How about something like (please feel free to improve):
"Introduce a user-configurable option that determines if a counter will automatically be assigned to an RMID, event pair when its associated monitor group is created via mkdir."
Sure.
Quoted 35 lines
Quoted 35 lines
Quoted 34 lines
Quoted 28 lines
when a resctrl group is created, provided the counters are available. By default, this option is enabled at boot.
Suggested-by Peter Newman <[email protected]>Signed-off-by Babu Moger <[email protected]>---
v13: Added Suggested-by tag.
Resolved conflicts caused by the recent FS/ARCH code restructure.
Therdtgroup.c/monitor.cfile has now been split between the FS and ARCH directories.v12: New patch. Added after the discussion on the list.
https://lore.kernel.org/lkml/CALPaoCh8siZKjL_3yvOYGL4cF_n_38KpUFgHVGbQ86nD+Q2_SA@mail.gmail.com/--- Documentation/filesystems/resctrl.rst | 10 ++++++ fs/resctrl/monitor.c | 2 ++ fs/resctrl/rdtgroup.c | 44 +++++++++++++++++++++++++-- include/linux/resctrl.h | 2 ++ 4 files changed, 56 insertions(+), 2 deletions(-)Documentation/filesystems/resctrl.rst +0
-0Documentation/filesystems/resctrl.rst with the following files: · line 351
Quoted 4 lines
Documentation/filesystems/resctrl.rst +1
-0@@ -348,6 +348,16 @@ with the following files: # cat /sys/fs/resctrl/info/L3_MON/counter_configs/mbm_total_bytes/event_filter local_reads, local_non_temporal_writes +"mbm_assign_on_mkdir": + Automatically assign the monitoring counters on resctrl group creationassign the monitoring counters to what?
Quoted 3 lines
Quoted 1 line
"Automatically assign counter IDs to an RMID, event pair on resctrl group creation if the counter IDs are available. It is enabled by default on boot and users can disable by writing to the interface."
Same here, please take care with the plural/singular usage.
Sure.
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 9 lines
Documentation/filesystems/resctrl.rst +7
-0+ if the counters are available. It is enabled by default on boot and users + can disable by writing to the interface. + :: + + # echo 0 > /sys/fs/resctrl/info/L3_MON/mbm_assign_on_mkdir + # cat /sys/fs/resctrl/info/L3_MON/mbm_assign_on_mkdir + 0
Quoted 9 lines
Quoted 3 lines
Quoted 1 line
Please be explicit in docs what possible values are and what they mean.
Sure. I can print "enabled" or "disabled".
I am not requesting a change in user interface self but instead clear documentation about what the input/output values mean. Even if the interface changes to "enabled"/"disabled" I assume the interface will still accept boolean values? Compare to the "
sparse_masks" documentation on how the possible values are explicitly documented.
ok. Will look into that.
Quoted 25 lines
...
Quoted 23 lines
Quoted 6 lines
Quoted 6 lines
Documentation/filesystems/resctrl.rst +6
-0+static ssize_t resctrl_mbm_assign_on_mkdir_write(struct kernfs_open_file *of, + char *buf, size_t nbytes, loff_t off) +{ + struct rdt_resource *r = rdt_kn_parent_priv(of->kn); + bool value; + int ret;Quoted 8 lines
Documentation/filesystems/resctrl.rst
Quoted 6 lines
Documentation/filesystems/resctrl.rst +6
-0+ + ret = kstrtobool(buf, &value); + if (ret) + return ret; + + cpus_read_lock();not traversing the domain list so hotplug lock not needed.
ok. Sure.
Quoted 5 lines
Quoted 2 lines
+
mutex_lock(&rdtgroup_mutex);
rdtgroup_mutexseems only needed because the message buffer is cleared below, and this is why it is not required in theshow()?
Quoted 9 lines
Quoted 1 line
Hmm. I didnt think about that. Do you think it is required?
It is certainly required to be able to call
rdt_last_cmd_clear()and since it then coversmbm_assign_on_mkdirI would prefer symmetry in consistently acquiringrdtgroup_mutexon both read and write while resctrl is mounted. Note that there is also other read usage on resctrl mount that is done with mutex held. Having the mutex acquired consistently will help to keep things simple.
Sure. will do.
Quoted 14 lines
...
Quoted 12 lines
Quoted 11 lines
include/linux/resctrl.h enum resctrl_schema_fmt { · line 281
Quoted 9 lines
include/linux/resctrl.h +1
-0@@ -278,6 +278,7 @@ enum resctrl_schema_fmt { * monitoring events can be configured. * @num_mbm_cntrs: Number of assignable monitoring counters * @mbm_cntr_assignable:Is system capable of supporting monitor assignment? + * @mbm_assign_on_mkdir:Auto enable monitor assignment on mkdir?How is "monitor assignment" different from "counter assignment"?
Quoted 8 lines
Quoted 3 lines
I should be:
"Auto enable counter ID assignment on mkdir"
hmmm ... I do not think this is about "Auto enable".
How about something like "Automatic counter assignment during monitor group create via mkdir?" or "True if counters should automatically be assigned to MBM events of monitor groups created via mkdir."
Sure. Thanks
Babu
Babu Moger
reviewerPATCH v13 22/27 x86/resctrl: Auto assign/unassign counters when mbm_cntr_assign is enabled
In reply to Babu MogerAutomatically assign or unassign counters when a resctrl group is created or deleted. By default, each group requires two counters: one for the MBM total event and one for the MBM local event.
The mbm_cntr_assign mode offers "num_mbm_cntrs" number of counters that can be assigned to an RMID, event pair and monitor the bandwidth as long as it is assigned. If these counters are exhausted, the kernel will log the error message "Unable to allocate counter in domain" in /sys/fs/resctrl/info/last_cmd_status when a new group is created.
However, the creation of a group should not fail due to assignment failures. Users have the flexibility to modify the assignments at a later time.
---
v13: Changes due to calling of resctrl_assign_cntr_event() and resctrl_unassign_cntr_event().
It only takes evtid. evt_cfg is not required anymore.
Resolved conflicts caused by the recent FS/ARCH code restructure.
The monitor.c/rdtgroup.c files have been split between the FS and ARCH directories.
v12: Removed mbm_cntr_reset() as it is not required while removing the group.
Update the commit text.
Added r->mon_capable check in rdtgroup_assign_cntrs() and rdtgroup_unassign_cntrs.
v11: Moved mbm_cntr_reset() to monitor.c.
Added code reset non-architectural state in mbm_cntr_reset(). Added missing rdtgroup_unassign_cntrs() calls on failure path.
v10: Assigned the counter before exposing the event files.
Moved the call rdtgroup_assign_cntrs() inside mkdir_rdt_prepare_rmid_alloc(). This is called both CNTR_MON and MON group creation.
Call mbm_cntr_reset() when unmounted to clear all the assignments. Taken care of few other feedback comments.
v9: Changed rdtgroup_assign_cntrs() and rdtgroup_unassign_cntrs() to return void.
Updated couple of rdtgroup_unassign_cntrs() calls properly. Updated function comments.
v8: Renamed rdtgroup_assign_grp to rdtgroup_assign_cntrs.
Renamed rdtgroup_unassign_grp to rdtgroup_unassign_cntrs. Fixed the problem with unassigning the child MON groups of CTRL_MON group.
v7: Reworded the commit message.
Removed the reference of ABMC with mbm_cntr_assign. Renamed the function rdtgroup_assign_cntrs to rdtgroup_assign_grp.
v6: Removed the redundant comments on all the calls of
rdtgroup_assign_cntrs. Updated the commit message.
Dropped printing error message on every call of rdtgroup_assign_cntrs.
v5: Removed the code to enable/disable ABMC during the mount.
That will be another patch.
Added arch callers to get the arch specific data. Renamed fuctions to match the other abmc function. Added code comments for assignment failures.
v4: Few name changes based on the upstream discussion.
Commit message update.
v3: This is a new patch. Patch addresses the upstream comment to enable
ABMC feature by default if the feature is available.
--- arch/x86/kernel/cpu/resctrl/monitor.c | 1 + fs/resctrl/rdtgroup.c | 65 ++++++++++++++++++++++++++- 2 files changed, 64 insertions(+), 2 deletions(-)
arch/x86/kernel/cpu/resctrl/monitor.c +1 -0
@@ -435,6 +435,7 @@ int __init rdt_get_mon_l3_config(struct rdt_resource *r) r->mon.mbm_cntr_assignable = true;cpuid_count (0x80000020 ,5 , &eax, &ebx, &ecx, &edx); r->mon.num_mbm_cntrs = (ebx &GENMASK (15 ,0 )) +1 ; + r->mon.mbm_assign_on_mkdir = true; } r->mon_capable = true;
fs/resctrl/rdtgroup.c +63 -2
130 lines not on this page · raw message
Signature · 1 line
2.34.1
Babu Moger
reviewerPATCH v13 23/27 x86/resctrl: Introduce mbm_L3_assignments to list assignments in a group
In reply to Babu MogerIntroduce the interface to display the assignment states for each group when mbm_cntr_assign mode is enabled.
The list is displayed in the following format:
<Event configuration>:<Domain id>=<Assignment type>
Event configuration: A valid event configuration listed in the /sys/fs/resctrl/info/L3_MON/counter_configs directory.
Domain ID: A valid domain ID number.
The assignment type can be one of the following:
_ : No event configuration assigned
e : Event configuration assigned in exclusive mode
Example:
$cd /sys/fs/resctrl
$cat mbm_L3_assignments
mbm_total_bytes:0=e;1=e
mbm_local_bytes:0=e;1=e
---
v13: Changelog update.
Few changes in mbm_L3_assignments_show() after moving the event config to evt_list. Resolved conflicts caused by the recent FS/ARCH code restructure.
The rdtgroup.c/monitor.c files have been split between the FS and ARCH directories.
v12: New patch:
Assignment interface moved inside the group based the discussion
https://lore.kernel.org/lkml/CALPaoCiii0vXOF06mfV=kVLBzhfNo0SFqt4kQGwGSGVUqvr2Dg@mail.gmail.com/#t
--- Documentation/filesystems/resctrl.rst | 28 +++++++++++++++ fs/resctrl/monitor.c | 1 + fs/resctrl/rdtgroup.c | 52 +++++++++++++++++++++++++++ 3 files changed, 81 insertions(+)
Documentation/filesystems/resctrl.rst +28 -0
@@ -504,6 +504,34 @@ When the "mba_MBps" mount option is used all CTRL_MON groups will also contain: /sys/fs/resctrl/info/L3_MON/mon_features changes the input event. +"mbm_L3_assignments": + This interface file is created when the mbm_cntr_assign mode is supported + and shows the assignment status for each group. + + The assignment list is displayed in the following format: + + <Event configuration>:<Domain id>=<Assignment type> + + Event configuration: A valid event configuration listed in the + /sys/fs/resctrl/info/L3_MON/counter_configs directory. + + Domain ID: A valid domain ID number. + + Assignment types: + + _ : No event configuration assigned + + e : Event configuration assigned in exclusive mode + + Example: + To list the assignment states for the default group. + :: + + # cd /sys/fs/resctrl + # cat mbm_L3_assignments + mbm_total_bytes:0=e;1=e + mbm_local_bytes:0=e;1=e + Resource allocation rules -------------------------
fs/resctrl/monitor.c +1 -0
@@ -935,6 +935,7 @@ int resctrl_mon_resource_init(void)resctrl_file_fflags_init ("event_filter" ,RFTYPE_ASSIGN_CONFIG );resctrl_file_fflags_init ("mbm_assign_on_mkdir" ,RFTYPE_MON_INFO |RFTYPE_RES_CACHE ); +resctrl_file_fflags_init ("mbm_L3_assignments" ,RFTYPE_MON_BASE ); }return 0 ;
fs/resctrl/rdtgroup.c +52 -0
64 lines not on this page · raw message
Signature · 1 line
2.34.1
Reinette Chatre
maintainerRe: [PATCH v13 23/27] x86/resctrl: Introduce mbm_L3_assignments to list assignments in a group
Hi Babu,
Quoted 5 lines · Babu Moger
Introduce the interface to display the assignment states for each group when
mbm_cntr_assignmode is enabled.The list is displayed in the following format:
<Event configuration>:<Domain id>=<Assignment type>
Should this just be <Event>? The information is just the event name, not its configuration that will be in the "event_filter" file.
Quoted 39 lines
Event configuration: A valid event configuration listed in the
/sys/fs/resctrl/info/L3_MON/counter_configsdirectory.Domain ID: A valid domain ID number.
The assignment type can be one of the following:
_ : No event configuration assigned
e : Event configuration assigned in exclusive mode
Example:
$cd/sys/fs/resctrl
$catmbm_L3_assignments
mbm_total_bytes:0=e;1=e
mbm_local_bytes:0=e;1=eSigned-off-by Babu Moger <[email protected]>---
v13: Changelog update.
Few changes inmbm_L3_assignments_show()after moving the event config toevt_list. Resolved conflicts caused by the recent FS/ARCH code restructure.
Therdtgroup.c/monitor.cfiles have been split between the FS and ARCH directories.v12: New patch:
Assignment interface moved inside the group based the discussion
https://lore.kernel.org/lkml/CALPaoCiii0vXOF06mfV=kVLBzhfNo0SFqt4kQGwGSGVUqvr2Dg@mail.gmail.com/#t--- Documentation/filesystems/resctrl.rst | 28 +++++++++++++++ fs/resctrl/monitor.c | 1 + fs/resctrl/rdtgroup.c | 52 +++++++++++++++++++++++++++ 3 files changed, 81 insertions(+)Documentation/filesystems/resctrl.rst +0
-0
Documentation/filesystems/resctrl.rst When the "mba_MBps" mount option is used all CTRL_MON groups will also contain: · line 508
Quoted 5 lines
Documentation/filesystems/resctrl.rst +2
-0@@ -504,6 +504,34 @@ When the "mba_MBps" mount option is used all CTRL_MON groups will also contain: /sys/fs/resctrl/info/L3_MON/mon_features changes the input event. +"mbm_L3_assignments": + This interface file is created when the mbm_cntr_assign mode is supported
"This interface file is created when" -> "Exists when mbm_cntr_assign mode is supported"?
Documentation/filesystems/resctrl.rst When the "mba_MBps" mount option is used all CTRL_MON groups will also contain: · line 509
Quoted 1 line
Documentation/filesystems/resctrl.rst +1
-0+ and shows the assignment status for each group.
This doc is in the portion documenting files in monitor groups. So rather:
"the assignment status for each group" -> "the counter assignment status for the MON group"?
Documentation/filesystems/resctrl.rst When the "mba_MBps" mount option is used all CTRL_MON groups will also contain: · line 513
Quoted 4 lines
Documentation/filesystems/resctrl.rst +4
-0+ + The assignment list is displayed in the following format: + + <Event configuration>:<Domain id>=<Assignment type>
<Event configuration> -> <Event>
Documentation/filesystems/resctrl.rst When the "mba_MBps" mount option is used all CTRL_MON groups will also contain: · line 515
Quoted 2 lines
Documentation/filesystems/resctrl.rst +2
-0+ + Event configuration: A valid event configuration listed in the
"A valid event in the /sys/fs/resctrl/info/L3_MON/event_configs directory"
Documentation/filesystems/resctrl.rst When the "mba_MBps" mount option is used all CTRL_MON groups will also contain: · line 518
Quoted 3 lines
Documentation/filesystems/resctrl.rst +3
-0+ /sys/fs/resctrl/info/L3_MON/counter_configs directory. + + Domain ID: A valid domain ID number.
"A valid domain ID"
Documentation/filesystems/resctrl.rst When the "mba_MBps" mount option is used all CTRL_MON groups will also contain: · line 522
Quoted 4 lines
Documentation/filesystems/resctrl.rst +4
-0+ + Assignment types: + + _ : No event configuration assigned
hmmm ... since the line has event as first field, would this not reflect the counter? That is "No counter assigned"
Documentation/filesystems/resctrl.rst When the "mba_MBps" mount option is used all CTRL_MON groups will also contain: · line 524
Quoted 2 lines
Documentation/filesystems/resctrl.rst +2
-0+ + e : Event configuration assigned in exclusive mode
"Counter assigned exclusively"? (with exclusive defined somewhere)
Documentation/filesystems/resctrl.rst When the "mba_MBps" mount option is used all CTRL_MON groups will also contain: · line 527
Quoted 3 lines
Documentation/filesystems/resctrl.rst +3
-0+ + Example: + To list the assignment states for the default group.
"the counter assignment states"?
Quoted 32 lines
Documentation/filesystems/resctrl.rst +7
-0+ :: + + # cd /sys/fs/resctrl + # cat mbm_L3_assignments + mbm_total_bytes:0=e;1=e + mbm_local_bytes:0=e;1=e + Resource allocation rules -------------------------fs/resctrl/monitor.c +1
-0@@ -935,6 +935,7 @@ int resctrl_mon_resource_init(void)resctrl_file_fflags_init ("event_filter" ,RFTYPE_ASSIGN_CONFIG );resctrl_file_fflags_init ("mbm_assign_on_mkdir" ,RFTYPE_MON_INFO |RFTYPE_RES_CACHE ); +resctrl_file_fflags_init ("mbm_L3_assignments" ,RFTYPE_MON_BASE ); }return 0 ;fs/resctrl/rdtgroup.c +2
-0@@ -2080,6 +2080,52 @@ static ssize_t resctrl_mbm_assign_on_mkdir_write(struct kernfs_open_file *of,return ret ?: nbytes; } +static int mbm_L3_assignments_show (struct kernfs_open_file *of,struct seq_file *s,void *v) +{
fs/resctrl/rdtgroup.c static ssize_t resctrl_mbm_assign_on_mkdir_write(struct kernfs_open_file *of, · line 2094
Quoted 10 lines
fs/resctrl/rdtgroup.c +10
-0+struct rdt_resource *r =resctrl_arch_get_resource (RDT_RESOURCE_L3 ); +struct rdt_mon_domain *d; +struct rdtgroup *rdtgrp; +struct mon_evt *mevt; +int ret =0 ; +bool sep; + + rdtgrp =rdtgroup_kn_lock_live (of->kn); +if (!rdtgrp) +return -ENOENT ;
Missing a rdtgroup_kn_unlock()?
fs/resctrl/rdtgroup.c static ssize_t resctrl_mbm_assign_on_mkdir_write(struct kernfs_open_file *of, · line 2100
Quoted 6 lines
fs/resctrl/rdtgroup.c +6
-0+ +rdt_last_cmd_clear (); +if (!resctrl_arch_mbm_cntr_assign_enabled (r)) { +rdt_last_cmd_puts ("mbm_cntr_assign mode not enabled\n" ); + ret = -ENOENT ; + goto assign_out;
grep goto fs/resctrl/rdtgroup.c
fs/resctrl/rdtgroup.c static ssize_t resctrl_mbm_assign_on_mkdir_write(struct kernfs_open_file *of, · line 2103
Quoted 3 lines
fs/resctrl/rdtgroup.c +3
-0+ } + +list_for_each_entry (mevt, &r->mon.evt_list, list) {
can use for_each_mbm_event() and then below will not be needed?
Quoted 31 lines
fs/resctrl/rdtgroup.c +25
-0+if (mevt->mbm_mode !=MBM_MODE_ASSIGN ) +continue ; + + sep = false; +seq_printf (s,"%s:" , mevt->name); +list_for_each_entry (d, &r->mon_domains, hdr.list) { +if (sep) +seq_putc (s,';' ); + +if (mbm_cntr_get (r, d, rdtgrp, mevt->evtid) >=0 ) +seq_printf (s,"%d=e" , d->hdr.id); +else +seq_printf (s,"%d=_" , d->hdr.id); + + sep = true; + } +seq_putc (s,'\n' ); + } + +assign_out: +rdtgroup_kn_unlock (of->kn); + +return ret; +} +/* rdtgroup information files for one cache resource. */ static struct rftype res_common_files[] = { {@@ -2218,6 +2264,12 @@ static struct rftype res_common_files[] = { .seq_show = event_filter_show, .write = event_filter_write,
fs/resctrl/rdtgroup.c static struct rftype res_common_files[] = { · line 2275
Quoted 10 lines
fs/resctrl/rdtgroup.c +6
-0}, + { + .name = "mbm_L3_assignments", + .mode = 0444, + .kf_ops = &rdtgroup_kf_single_ops, + .seq_show = mbm_L3_assignments_show, + }, { .name = "mbm_assign_mode", .mode = 0444,
Reinette
Re: [PATCH v13 23/27] x86/resctrl: Introduce mbm_L3_assignments to list assignments in a group
Hi Reinette,
Quoted 11 lines · Reinette Chatre
Hi Babu,
Quoted 5 lines · Babu Moger
Introduce the interface to display the assignment states for each group when
mbm_cntr_assignmode is enabled.The list is displayed in the following format:
<Event configuration>:<Domain id>=<Assignment type>Should this just be <Event>? The information is just the event name, not its configuration that will be in the "
event_filter" file.
Yes. Sure.
Quoted 5 lines
Quoted 5 lines
Event configuration: A valid event configuration listed in the
/sys/fs/resctrl/info/L3_MON/counter_configsdirectory.
Quoted 8 lines
Quoted 8 lines
Domain ID: A valid domain ID number.
The assignment type can be one of the following:
_ : No event configuration assigned
e : Event configuration assigned in exclusive mode
This needs to change as well based on comments below(resctrl.rst). This is note for me.
Quoted 30 lines
Quoted 30 lines
Example:
$cd/sys/fs/resctrl
$catmbm_L3_assignments
mbm_total_bytes:0=e;1=e
mbm_local_bytes:0=e;1=eSigned-off-by Babu Moger <[email protected]>---
v13: Changelog update.
Few changes inmbm_L3_assignments_show()after moving the event config toevt_list. Resolved conflicts caused by the recent FS/ARCH code restructure.
Therdtgroup.c/monitor.cfiles have been split between the FS and ARCH directories.v12: New patch:
Assignment interface moved inside the group based the discussion
https://lore.kernel.org/lkml/CALPaoCiii0vXOF06mfV=kVLBzhfNo0SFqt4kQGwGSGVUqvr2Dg@mail.gmail.com/#t--- Documentation/filesystems/resctrl.rst | 28 +++++++++++++++ fs/resctrl/monitor.c | 1 + fs/resctrl/rdtgroup.c | 52 +++++++++++++++++++++++++++ 3 files changed, 81 insertions(+)Documentation/filesystems/resctrl.rst +0
-0@@ -504,6 +504,34 @@ When the "mba_MBps" mount option is used all CTRL_MON groups will also contain: /sys/fs/resctrl/info/L3_MON/mon_features changes the input event.
Quoted 4 lines
Documentation/filesystems/resctrl.rst When the "mba_MBps" mount option is used all CTRL_MON groups will also contain: · line 508
Quoted 2 lines
Documentation/filesystems/resctrl.rst +2
-0+"mbm_L3_assignments": + This interface file is created when the mbm_cntr_assign mode is supported"This interface file is created when" -> "Exists when
mbm_cntr_assignmode is supported"?
Sure,
Quoted 5 lines
Documentation/filesystems/resctrl.rst When the "mba_MBps" mount option is used all CTRL_MON groups will also contain: · line 509
Quoted 2 lines
Documentation/filesystems/resctrl.rst +1
-0+ and shows the assignment status for each group.This doc is in the portion documenting files in monitor groups. So rather:
"the assignment status for each group" -> "the counter assignment status for the MON group"?
ok.
Quoted 8 lines
Documentation/filesystems/resctrl.rst When the "mba_MBps" mount option is used all CTRL_MON groups will also contain: · line 513
Quoted 5 lines
Documentation/filesystems/resctrl.rst +4
-0+ + The assignment list is displayed in the following format: + + <Event configuration>:<Domain id>=<Assignment type><Event configuration> -> <Event>
Ok.
Quoted 4 lines
Documentation/filesystems/resctrl.rst When the "mba_MBps" mount option is used all CTRL_MON groups will also contain: · line 515
Quoted 2 lines
Documentation/filesystems/resctrl.rst +2
-0+ + Event configuration: A valid event configuration listed in the"A valid event in the
/sys/fs/resctrl/info/L3_MON/event_configsdirectory"
Sure.
Quoted 7 lines
Documentation/filesystems/resctrl.rst When the "mba_MBps" mount option is used all CTRL_MON groups will also contain: · line 518
Quoted 4 lines
Documentation/filesystems/resctrl.rst +3
-0+ /sys/fs/resctrl/info/L3_MON/counter_configs directory. + + Domain ID: A valid domain ID number."A valid domain ID"
Quoted 7 lines
Documentation/filesystems/resctrl.rst When the "mba_MBps" mount option is used all CTRL_MON groups will also contain: · line 522
Quoted 4 lines
Documentation/filesystems/resctrl.rst +4
-0+ + Assignment types: + + _ : No event configuration assignedhmmm ... since the line has event as first field, would this not reflect the counter? That is "No counter assigned"
Sure.
Quoted 5 lines
Documentation/filesystems/resctrl.rst When the "mba_MBps" mount option is used all CTRL_MON groups will also contain: · line 524
Quoted 3 lines
Documentation/filesystems/resctrl.rst +2
-0+ + e : Event configuration assigned in exclusive mode"Counter assigned exclusively"? (with exclusive defined somewhere)
Sure.
Quoted 6 lines
Documentation/filesystems/resctrl.rst When the "mba_MBps" mount option is used all CTRL_MON groups will also contain: · line 527
Quoted 4 lines
Documentation/filesystems/resctrl.rst +3
-0+ + Example: + To list the assignment states for the default group."the counter assignment states"?
ok
Quoted 31 lines
Quoted 31 lines
Documentation/filesystems/resctrl.rst +7
-0+ :: + + # cd /sys/fs/resctrl + # cat mbm_L3_assignments + mbm_total_bytes:0=e;1=e + mbm_local_bytes:0=e;1=e + Resource allocation rules -------------------------fs/resctrl/monitor.c +1
-0@@ -935,6 +935,7 @@ int resctrl_mon_resource_init(void)resctrl_file_fflags_init ("event_filter" ,RFTYPE_ASSIGN_CONFIG );resctrl_file_fflags_init ("mbm_assign_on_mkdir" ,RFTYPE_MON_INFO |RFTYPE_RES_CACHE ); +resctrl_file_fflags_init ("mbm_L3_assignments" ,RFTYPE_MON_BASE ); }return 0 ;fs/resctrl/rdtgroup.c +0
-0@@ -2080,6 +2080,52 @@ static ssize_t resctrl_mbm_assign_on_mkdir_write(struct kernfs_open_file *of,return ret ?: nbytes; }
Quoted 14 lines
fs/resctrl/rdtgroup.c static ssize_t resctrl_mbm_assign_on_mkdir_write(struct kernfs_open_file *of, · line 2094
Quoted 12 lines
fs/resctrl/rdtgroup.c +12
-0+static int mbm_L3_assignments_show (struct kernfs_open_file *of,struct seq_file *s,void *v) +{ +struct rdt_resource *r =resctrl_arch_get_resource (RDT_RESOURCE_L3 ); +struct rdt_mon_domain *d; +struct rdtgroup *rdtgrp; +struct mon_evt *mevt; +int ret =0 ; +bool sep; + + rdtgrp =rdtgroup_kn_lock_live (of->kn); +if (!rdtgrp) +return -ENOENT ;Missing a
rdtgroup_kn_unlock()?
Yes. Will add it.
Quoted 9 lines
fs/resctrl/rdtgroup.c static ssize_t resctrl_mbm_assign_on_mkdir_write(struct kernfs_open_file *of, · line 2100
Quoted 7 lines
fs/resctrl/rdtgroup.c +6
-0+ +rdt_last_cmd_clear (); +if (!resctrl_arch_mbm_cntr_assign_enabled (r)) { +rdt_last_cmd_puts ("mbm_cntr_assign mode not enabled\n" ); + ret = -ENOENT ; + goto assign_out;grep goto
fs/resctrl/rdtgroup.c
Will change it to "out_assign".
Quoted 6 lines
fs/resctrl/rdtgroup.c static ssize_t resctrl_mbm_assign_on_mkdir_write(struct kernfs_open_file *of, · line 2103
Quoted 4 lines
fs/resctrl/rdtgroup.c +3
-0+ } + +list_for_each_entry (mevt, &r->mon.evt_list, list) {can use
for_each_mbm_event()and then below will not be needed?
Yes.
Quoted 36 lines
Quoted 36 lines
fs/resctrl/rdtgroup.c +28
-0+if (mevt->mbm_mode !=MBM_MODE_ASSIGN ) +continue ; + + sep = false; +seq_printf (s,"%s:" , mevt->name); +list_for_each_entry (d, &r->mon_domains, hdr.list) { +if (sep) +seq_putc (s,';' ); + +if (mbm_cntr_get (r, d, rdtgrp, mevt->evtid) >=0 ) +seq_printf (s,"%d=e" , d->hdr.id); +else +seq_printf (s,"%d=_" , d->hdr.id); + + sep = true; + } +seq_putc (s,'\n' ); + } + +assign_out: +rdtgroup_kn_unlock (of->kn); + +return ret; +} +/* rdtgroup information files for one cache resource. */ static struct rftype res_common_files[] = { {@@ -2218,6 +2264,12 @@ static struct rftype res_common_files[] = { .seq_show = event_filter_show, .write = event_filter_write, }, + { + .name = "mbm_L3_assignments", + .mode = 0444,
Quoted 9 lines
fs/resctrl/rdtgroup.c static struct rftype res_common_files[] = { · line 2275
Quoted 6 lines
fs/resctrl/rdtgroup.c +3
-0+ .kf_ops = &rdtgroup_kf_single_ops, + .seq_show = mbm_L3_assignments_show, + }, { .name = "mbm_assign_mode", .mode = 0444,Reinette
Thanks
Babu
Babu Moger
reviewerPATCH v13 24/27 x86/resctrl: Introduce the interface to modify assignments in a group
In reply to Babu MogerIntroduce an interface to modify assignments within a group.
Modifications follow this format:
<Event configuration>:<Domain id>=<Assignment type>
The assignment type can be one of the following:
_ : No event configuration assigned
e : Event configuration assigned in exclusive mode
Domain id can be any valid domain ID number or '*' to update all the domains.
Example:
$cd /sys/fs/resctrl
$cat mbm_L3_assignments
mbm_total_bytes:0=e;1=e
mbm_local_bytes:0=e;1=e
To unassign the configuration of mbm_total_bytes on domain 0:
$echo "mbm_total_bytes:0=_" > mbm_L3_assignments
$cat mbm_L3_assignments
mbm_total_bytes:0=_;1=e
mbm_local_bytes:0=e;1=e
To unassign the mbm_total_bytes configuration on all domains:
$echo "mbm_total_bytes:*=_" > mbm_L3_assignments
$cat mbm_L3_assignments
mbm_total_bytes:0=_;1=_
mbm_local_bytes:0=e;1=e
---
v13: Few changes in mbm_L3_assignments_write() after moving the event config to evt_list.
Resolved conflicts caused by the recent FS/ARCH code restructure.
v12: New patch:
Assignment interface moved inside the group based the discussion
https://lore.kernel.org/lkml/CALPaoCiii0vXOF06mfV=kVLBzhfNo0SFqt4kQGwGSGVUqvr2Dg@mail.gmail.com/#t
--- Documentation/filesystems/resctrl.rst | 29 ++++- fs/resctrl/internal.h | 9 ++ fs/resctrl/rdtgroup.c | 165 +++++++++++++++++++++++++- 3 files changed, 201 insertions(+), 2 deletions(-)
Documentation/filesystems/resctrl.rst +28 -1
@@ -515,7 +515,7 @@ When the "mba_MBps" mount option is used all CTRL_MON groups will also contain: Event configuration: A valid event configuration listed in the /sys/fs/resctrl/info/L3_MON/counter_configs directory. - Domain ID: A valid domain ID number. + Domain ID: A valid domain ID number or '*' to update all the domains. Assignment types:
@@ -532,6 +532,33 @@ When the "mba_MBps" mount option is used all CTRL_MON groups will also contain: mbm_total_bytes:0=e;1=e mbm_local_bytes:0=e;1=e + Modify the assignment states by writing to the interface file. + + Example: + To unassign the configuration of mbm_total_bytes on domain 0: + :: + + # echo "mbm_total_bytes:0=_" > mbm_L3_assignments + # cat mbm_L3_assignments + mbm_total_bytes:0=_;1=e + mbm_local_bytes:0=e;1=e + + To unassign the mbm_total_bytes configuration on all domains: + :: + + # echo "mbm_total_bytes:*=_" > mbm_L3_assignments + # cat mbm_L3_assignments + mbm_total_bytes:0=_;1=_ + mbm_local_bytes:0=e;1=e + + To assign the mbm_total_bytes configuration on all domains in exclusive mode: + :: + + # echo "mbm_total_bytes:*=e" > mbm_L3_assignments + # cat mbm_L3_assignments + mbm_total_bytes:0=e;1=e + mbm_local_bytes:0=e;1=e + Resource allocation rules -------------------------
fs/resctrl/internal.h +9 -0
@@ -51,6 +51,15 @@ static inline struct rdt_fs_context *rdt_fc2context(struct fs_context *fc) return container_of(kfc, struct rdt_fs_context, kfc); } +/* +* Assignment types for mbm_cntr_assign mode +*/ +enum { +ASSIGN_NONE =0 , +ASSIGN_EXCLUSIVE , +ASSIGN_INVALID , +}; +/** * struct mon_evt - Entry in the event list of a resource * @evtid: event id
fs/resctrl/rdtgroup.c +164 -1
179 lines not on this page · raw message
Signature · 1 line
2.34.1
Re: [PATCH v13 24/27] x86/resctrl: Introduce the interface to modify assignments in a group
Hi Babu,
Quoted 19 lines · Babu Moger
fs/resctrl/rdtgroup.c +11
-0@@ -2126,6 +2126,168 @@ static int mbm_L3_assignments_show(struct kernfs_open_file *of, struct seq_filereturn ret; } +/* +* mbm_get_mon_event_by_name() - Return the mon_evt entry for the matching +* event name. +*/ +static struct mon_evt *mbm_get_mon_event_by_name (struct rdt_resource *r, +char *name) +{ +struct mon_evt *mevt; + +list_for_each_entry (mevt, &r->mon.evt_list, list) { +if (!strcmp (mevt->name, name))
fs/resctrl/rdtgroup.c static int mbm_L3_assignments_show(struct kernfs_open_file *of, struct seq_file · line 2149
Quoted 10 lines
fs/resctrl/rdtgroup.c +10
-0+return mevt; + } + +return NULL ; +} + +static unsigned int resctrl_get_assing_type (char *assign) +{ +unsigned int mon_state =ASSIGN_NONE ; +int len =strlen (assign);
[ 395.013183] BUG: kernel NULL pointer dereference, address: 0000000000000000 [ 395.013426] #PF: supervisor read access in kernel mode [ 395.013600] #PF: error_code(0x0000) - not-present page [ 395.013779] PGD 39322c067 P4D 2a4f49067 PUD 2a4f4a067 PMD 0 [ 395.013973] Oops: Oops: 0000 [#1] SMP DEBUG_PAGEALLOC NOPTI [ 395.014156] CPU: 37 UID: 0 PID: 24147 Comm: bash Not tainted 6.15.0-dbg-DEV #13 NONE [ 395.014403] Hardware name: Google Astoria-Turin/astoria, BIOS 0.20241223.2-0 01/17/2025 [ 395.014652] RIP: 0010:strlen+0xb/0x20 [ 395.014778] Code: 66 66 66 66 66 66 2e 0f 1f 84 00 00 00 00 00 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 f3 0f 1e fa 48 c7 c0 ff ff ff ff <80> 7c 07 01 00 48 8d 40 01 75 f5 c3 cc cc cc cc cc 0f 1f 40 00 90 [ 395.015356] RSP: 0018:ffa000002f743d58 EFLAGS: 00010246 [ 395.015522] RAX: ffffffffffffffff RBX: ff11000129a00600 RCX: 0000000000000000 [ 395.015747] RDX: ff110001299f5253 RSI: ffffffff827b9651 RDI: 0000000000000000 [ 395.015968] RBP: 0000000000000000 R08: 000000000000003d R09: 0000000000000000 [ 395.016202] R10: ffffffff827b9652 R11: 0000000000000000 R12: ffffffff8305b7f8 [ 395.016421] R13: ff110001299f5240 R14: 0000000000000014 R15: 0000000000000000 [ 395.016644] FS: 00007f1281ff8b80(0000) GS:ff1100bdc8276000(0000) knlGS:0000000000000000 [ 395.016893] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 395.017071] CR2: 0000000000000000 CR3: 0000000420bc8002 CR4: 0000000000771ef0 [ 395.017298] PKRU: 55555554 [ 395.017388] Call Trace: [ 395.017471] <TASK> [ 395.017545] mbm_L3_assignments_write+0x2d4/0x4e0 [ 395.017700] kernfs_fop_write_iter+0x132/0x1c0 [ 395.017851] vfs_write+0x2bf/0x3c0 [ 395.017963] ksys_write+0x82/0x100 [ 395.018074] do_syscall_64+0xee/0x210 [ 395.018198] ? exc_page_fault+0x81/0xe0 [ 395.018321] entry_SYSCALL_64_after_hwframe+0x77/0x7f [ 395.018482] RIP: 0033:0x7f128177f8b3 [ 395.018598] Code: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc 48 8b 05 99 91 07 00 83 38 00 75 10 b8 01 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 4d c3 55 48 89 e5 41 57 41 56 53 50 48 89 d3 [ 395.019167] RSP: 002b:00007ffff66e80f8 EFLAGS: 00000246 ORIG_RAX: 0000000000000001 [ 395.019409] RAX: ffffffffffffffda RBX: 0000000000000014 RCX: 00007f128177f8b3 [ 395.019636] RDX: 0000000000000014 RSI: 0000000001eedb60 RDI: 0000000000000001 [ 395.019861] RBP: 00007ffff66e8120 R08: 0000000000000000 R09: 0000000000000000 [ 395.020081] R10: 00007ffff66e81b0 R11: 0000000000000246 R12: 0000000001eedb60 [ 395.020303] R13: 0000000000000001 R14: 00007f12817fa650 R15: 0000000000000014 [ 395.020532] </TASK>
Quoted 35 lines
fs/resctrl/rdtgroup.c +35
-0+ +if (!len || len >1 ) +return ASSIGN_INVALID ; + +switch (*assign) { +case 'e' : + mon_state =ASSIGN_EXCLUSIVE ; +break ; +case '_' : + mon_state =ASSIGN_NONE ; +break ; +default : + mon_state =ASSIGN_INVALID ; +break ; + } + +return mon_state; +} + +static int resctrl_process_assign (struct rdt_resource *r,struct rdtgroup *rdtgrp, +char *config,char *tok) +{ +struct rdt_mon_domain *d; +char *dom_str, *id_str; +unsigned long dom_id =0 ; +struct mon_evt *mevt; +int assign_type; +char domain[10 ]; +bool found; +int ret; + + mevt =mbm_get_mon_event_by_name (r, config); +if (!mevt) { +rdt_last_cmd_printf ("Invalid assign configuration %s\n" , config); +return -ENOENT ;
fs/resctrl/rdtgroup.c static int mbm_L3_assignments_show(struct kernfs_open_file *of, struct seq_file · line 2194
Quoted 10 lines
fs/resctrl/rdtgroup.c +10
-0+ } + +next: +if (!tok || tok[0 ] =='\0' ) +return 0 ; + +/* Start processing the strings for each domain */ + dom_str =strim (strsep (&tok,";" )); + + id_str =strsep (&dom_str,"=" );
If there's no '=' then dom_str becomes NULL...
Quoted 16 lines
fs/resctrl/rdtgroup.c +16
-0+ +/* Check for domain id '*' which means all domains */ +if (id_str && *id_str =='*' ) { + d =NULL ; + goto check_state; + }else if (!id_str ||kstrtoul (id_str,10 , &dom_id)) { +rdt_last_cmd_puts ("Missing domain id\n" ); +return -EINVAL ; + } + +/* Verify if the dom_id is valid */ + found = false; +list_for_each_entry (d, &r->mon_domains, hdr.list) { +if (d->hdr.id == dom_id) { + found = true; +break ;
fs/resctrl/rdtgroup.c static int mbm_L3_assignments_show(struct kernfs_open_file *of, struct seq_file · line 2220
Quoted 10 lines
fs/resctrl/rdtgroup.c +10
-0+ } + } + +if (!found) { +rdt_last_cmd_printf ("Invalid domain id %ld\n" , dom_id); +return -EINVAL ; + } + +check_state: + assign_type =resctrl_get_assing_type (dom_str);
then the resulting type of whatever this is supposed to mean is "panic"
Thanks,
-Peter
Moger, Babu
reviewerRe: [PATCH v13 24/27] x86/resctrl: Introduce the interface to modify assignments in a group
Hi Peter,
Quoted 73 lines · Peter Newman
Hi Babu,
Quoted 19 lines · Babu Moger
fs/resctrl/rdtgroup.c +11
-0@@ -2126,6 +2126,168 @@ static int mbm_L3_assignments_show(struct kernfs_open_file *of, struct seq_filereturn ret; } +/* +* mbm_get_mon_event_by_name() - Return the mon_evt entry for the matching +* event name. +*/ +static struct mon_evt *mbm_get_mon_event_by_name (struct rdt_resource *r, +char *name) +{ +struct mon_evt *mevt; + +list_for_each_entry (mevt, &r->mon.evt_list, list) { +if (!strcmp (mevt->name, name))fs/resctrl/rdtgroup.c static int mbm_L3_assignments_show(struct kernfs_open_file *of, struct seq_file · line 2149
Quoted 10 lines
fs/resctrl/rdtgroup.c +10
-0+return mevt; + } + +return NULL ; +} + +static unsigned int resctrl_get_assing_type (char *assign) +{ +unsigned int mon_state =ASSIGN_NONE ; +int len =strlen (assign);[ 395.013183] BUG: kernel NULL pointer dereference, address: 0000000000000000 [ 395.013426] #PF: supervisor read access in kernel mode [ 395.013600] #PF: error_code(0x0000) - not-present page [ 395.013779] PGD 39322c067 P4D 2a4f49067 PUD 2a4f4a067 PMD 0 [ 395.013973] Oops: Oops: 0000 [#1] SMP DEBUG_PAGEALLOC NOPTI [ 395.014156] CPU: 37 UID: 0 PID: 24147 Comm: bash Not tainted 6.15.0-dbg-DEV #13 NONE [ 395.014403] Hardware name: Google Astoria-Turin/astoria, BIOS 0.20241223.2-0 01/17/2025 [ 395.014652] RIP: 0010:strlen+0xb/0x20 [ 395.014778] Code: 66 66 66 66 66 66 2e 0f 1f 84 00 00 00 00 00 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 f3 0f 1e fa 48 c7 c0 ff ff ff ff <80> 7c 07 01 00 48 8d 40 01 75 f5 c3 cc cc cc cc cc 0f 1f 40 00 90 [ 395.015356] RSP: 0018:ffa000002f743d58 EFLAGS: 00010246 [ 395.015522] RAX: ffffffffffffffff RBX: ff11000129a00600 RCX: 0000000000000000 [ 395.015747] RDX: ff110001299f5253 RSI: ffffffff827b9651 RDI: 0000000000000000 [ 395.015968] RBP: 0000000000000000 R08: 000000000000003d R09: 0000000000000000 [ 395.016202] R10: ffffffff827b9652 R11: 0000000000000000 R12: ffffffff8305b7f8 [ 395.016421] R13: ff110001299f5240 R14: 0000000000000014 R15: 0000000000000000 [ 395.016644] FS: 00007f1281ff8b80(0000) GS:ff1100bdc8276000(0000) knlGS:0000000000000000 [ 395.016893] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 395.017071] CR2: 0000000000000000 CR3: 0000000420bc8002 CR4: 0000000000771ef0 [ 395.017298] PKRU: 55555554 [ 395.017388] Call Trace: [ 395.017471] <TASK> [ 395.017545] mbm_L3_assignments_write+0x2d4/0x4e0 [ 395.017700] kernfs_fop_write_iter+0x132/0x1c0 [ 395.017851] vfs_write+0x2bf/0x3c0 [ 395.017963] ksys_write+0x82/0x100 [ 395.018074] do_syscall_64+0xee/0x210 [ 395.018198] ? exc_page_fault+0x81/0xe0 [ 395.018321] entry_SYSCALL_64_after_hwframe+0x77/0x7f [ 395.018482] RIP: 0033:0x7f128177f8b3 [ 395.018598] Code: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cccc cc cc cc cc cc cc 48 8b 05 99 91 07 00 83 38 00 75 10 b8 01 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 4d c3 55 48 89 e5 41 57 41 56 53 50 48 89 d3
Quoted 9 lines
[ 395.019167] RSP: 002b:00007ffff66e80f8 EFLAGS: 00000246 ORIG_RAX: 0000000000000001 [ 395.019409] RAX: ffffffffffffffda RBX: 0000000000000014 RCX: 00007f128177f8b3 [ 395.019636] RDX: 0000000000000014 RSI: 0000000001eedb60 RDI: 0000000000000001 [ 395.019861] RBP: 00007ffff66e8120 R08: 0000000000000000 R09: 0000000000000000 [ 395.020081] R10: 00007ffff66e81b0 R11: 0000000000000246 R12: 0000000001eedb60 [ 395.020303] R13: 0000000000000001 R14: 00007f12817fa650 R15: 0000000000000014 [ 395.020532] </TASK>
Yes. Got it. Missing NULL check. Simplified the function now. Thanks
static unsigned int resctrl_get_assign_type(char *assign) {
if (!assign || strlen(assign) != 1)
return ASSIGN_INVALID;
switch (*assign) {
case 'e':
return ASSIGN_EXCLUSIVE;
case '_':
return ASSIGN_NONE;
default:
return ASSIGN_INVALID;
}
}Quoted 39 lines
Quoted 39 lines
fs/resctrl/rdtgroup.c +39
-0+ +if (!len || len >1 ) +return ASSIGN_INVALID ; + +switch (*assign) { +case 'e' : + mon_state =ASSIGN_EXCLUSIVE ; +break ; +case '_' : + mon_state =ASSIGN_NONE ; +break ; +default : + mon_state =ASSIGN_INVALID ; +break ; + } + +return mon_state; +} + +static int resctrl_process_assign (struct rdt_resource *r,struct rdtgroup *rdtgrp, +char *config,char *tok) +{ +struct rdt_mon_domain *d; +char *dom_str, *id_str; +unsigned long dom_id =0 ; +struct mon_evt *mevt; +int assign_type; +char domain[10 ]; +bool found; +int ret; + + mevt =mbm_get_mon_event_by_name (r, config); +if (!mevt) { +rdt_last_cmd_printf ("Invalid assign configuration %s\n" , config); +return -ENOENT ; + } + +next: +if (!tok || tok[0 ] =='\0' )
Quoted 8 lines
fs/resctrl/rdtgroup.c static int mbm_L3_assignments_show(struct kernfs_open_file *of, struct seq_file · line 2194
Quoted 6 lines
fs/resctrl/rdtgroup.c +6
-0+return 0 ; + +/* Start processing the strings for each domain */ + dom_str =strim (strsep (&tok,";" )); + + id_str =strsep (&dom_str,"=" );If there's no '=' then
dom_strbecomes NULL...
Yea. That is correct.
Quoted 33 lines
Quoted 17 lines
fs/resctrl/rdtgroup.c +16
-0+ +/* Check for domain id '*' which means all domains */ +if (id_str && *id_str =='*' ) { + d =NULL ; + goto check_state; + }else if (!id_str ||kstrtoul (id_str,10 , &dom_id)) { +rdt_last_cmd_puts ("Missing domain id\n" ); +return -EINVAL ; + } + +/* Verify if the dom_id is valid */ + found = false; +list_for_each_entry (d, &r->mon_domains, hdr.list) { +if (d->hdr.id == dom_id) { + found = true; +break ;fs/resctrl/rdtgroup.c static int mbm_L3_assignments_show(struct kernfs_open_file *of, struct seq_file · line 2220
Quoted 10 lines
fs/resctrl/rdtgroup.c +10
-0+ } + } + +if (!found) { +rdt_last_cmd_printf ("Invalid domain id %ld\n" , dom_id); +return -EINVAL ; + } + +check_state: + assign_type =resctrl_get_assing_type (dom_str);then the resulting type of whatever this is supposed to mean is "panic"
Thanks,
-Peter
Signature · 2 lines
Thanks Babu Moger
Babu Moger
reviewerPATCH v13 25/27 x86/resctrl: Hide the BMEC related files when mbm_cnt_assign is enabled
In reply to Babu MogerBMEC (Bandwidth Monitoring Event Configuration) and mbm_cntr_assign cannot be used simultaneously.
When mbm_cntr_assign is active, suppress visibility of BMEC-related files to prevent confusion.
The files /sys/fs/resctrl/info/L3_MON/mbm_total_bytes_config and /sys/fs/resctrl/info/L3_MON/mbm_local_bytes_config will not be visible when mbm_cntr_assign mode is enabled.
---
v13: New patch to hide BMEC related files.
--- fs/resctrl/rdtgroup.c | 31 +++++++++++++++++++++++++++++++ 1 file changed, 31 insertions(+)
fs/resctrl/rdtgroup.c +31 -0
@@ -1815,6 +1815,33 @@ static ssize_t mbm_local_bytes_config_write(struct kernfs_open_file *of,return ret ?: nbytes; } +static void resctrl_bmec_files_show (struct rdt_resource *r,bool show) +{ +struct kernfs_node *kn_config, *l3_mon_kn; +char name[32 ]; + +sprintf (name,"%s_MON" , r->name); + l3_mon_kn =kernfs_find_and_get (kn_info, name); +if (!l3_mon_kn) +return ; + + kn_config =kernfs_find_and_get (l3_mon_kn,"mbm_total_bytes_config" ); +if (kn_config) { +kernfs_get (kn_config); +kernfs_show (kn_config, show); +kernfs_put (kn_config); + } + + kn_config =kernfs_find_and_get (l3_mon_kn,"mbm_local_bytes_config" ); +if (kn_config) { +kernfs_get (kn_config); +kernfs_show (kn_config, show); +kernfs_put (kn_config); + } + +kernfs_put (l3_mon_kn); +} +static int resctrl_mbm_assign_mode_show (struct kernfs_open_file *of,struct seq_file *s,void *v) {
@@ -2815,6 +2842,10 @@ static int rdtgroup_create_info_dir(struct kernfs_node *parent_kn) ret =resctrl_mkdir_counter_configs (r, name);if (ret) goto out_destroy; + +/* Hide BMEC related files if mbm_cntr_assign is enabled */ +if (resctrl_arch_mbm_cntr_assign_enabled (r)) +resctrl_bmec_files_show (r, false); } }
Signature · 1 line
2.34.1
Babu Moger
reviewerPATCH v13 26/27 x86/resctrl: Introduce the interface to switch between monitor modes
In reply to Babu MogerResctrl subsystem can support two monitoring modes, "mbm_cntr_assign" or "default". In mbm_cntr_assign, monitoring event can only accumulate data while it is backed by a hardware counter. In "default" mode, resctrl assumes there is a hardware counter for each event within every CTRL_MON and MON group.
Introduce interface to switch between mbm_cntr_assign and default modes.
$ cat /sys/fs/resctrl/info/L3_MON/mbm_assign_mode [mbm_cntr_assign] default
To enable the "mbm_cntr_assign" monitoring mode:
$ echo "mbm_cntr_assign" > /sys/fs/resctrl/info/L3_MON/mbm_assign_mode
To enable the "default" monitoring mode:
$ echo "default" > /sys/fs/resctrl/info/L3_MON/mbm_assign_mode
MBM event counters are automatically reset as part of changing the mode. Clear both architectural and non-architectural event states to prevent overflow conditions during the next event read.
---
v13: Resolved the conflicts due to FS/ARCH restructure.
Introduced the new resctrl_init_evt_configuration() to initialize the event modes and configuration values.
Added the call to resctrl_bmec_files_show() hide/show BMEC related files.
v12: Fixed the documentation for a consistency.
Introduced mbm_cntr_free_all() and resctrl_reset_rmid_all() to clear counters and non-architectural states when monitor mode is changed.
https://lore.kernel.org/lkml/[email protected]/
v11: Changed the name of the function rdtgroup_mbm_assign_mode_write() to
resctrl_mbm_assign_mode_write().
Rewrote the commit message with context.
Added few more details in resctrl.rst about mbm_cntr_assign mode. Re-arranged the text in resctrl.rst file.
v10: The call mbm_cntr_reset() has been moved to earlier patch.
Minor documentation update.
v9: Fixed extra spaces in user documentation.
Fixed problem changing the mode to mbm_cntr_assign mode when it is not supported. Added extra checks to detect if systems supports it. Used the rdtgroup_cntr_id_init to initialize cntr_id.
v8: Reset the internal counters after mbm_cntr_assign mode is changed.
Renamed rdtgroup_mbm_cntr_reset() to mbm_cntr_reset() Updated the documentation to make text generic.
v7: Changed the interface name to mbm_assign_mode.
Removed the references of ABMC.
Added the changes to reset global and domain bitmaps. Added the changes to reset rmid.
v6: Changed the mode name to mbm_cntr_assign.
Moved all the FS related code here.
Added changes to reset mbm_cntr_map and resctrl group counters.
v5: Change log and mode description text correction.
v4: Minor commit text changes. Keep the default to ABMC when supported.
Fixed comments to reflect changed interface "mbm_mode".
v3: New patch to address the review comments from upstream.
--- Documentation/filesystems/resctrl.rst | 25 ++++++++++- fs/resctrl/internal.h | 3 ++ fs/resctrl/monitor.c | 53 +++++++++++++++++++--- fs/resctrl/rdtgroup.c | 65 ++++++++++++++++++++++++++- 4 files changed, 138 insertions(+), 8 deletions(-)
Documentation/filesystems/resctrl.rst +24 -1
@@ -259,7 +259,10 @@ with the following files: "mbm_assign_mode": Reports the list of monitoring modes supported. The enclosed brackets - indicate which mode is enabled. + indicate which mode is enabled. The MBM events (mbm_total_bytes and/or + mbm_local_bytes) associated with counters may reset when "mbm_assign_mode" + is changed. + :: # cat /sys/fs/resctrl/info/L3_MON/mbm_assign_mode
@@ -275,6 +278,16 @@ with the following files: "num_mbm_cntrs" file. Changing the mode may cause all counters on the resource to reset. + Moving to mbm_cntr_assign mode require users to assign the counters to + the events. Otherwise, the MBM event counters will return 'Unassigned' + when read. + + The mode is beneficial for AMD platforms that support more CTRL_MON + and MON groups than available hardware counters. By default, this + feature is enabled on AMD platforms with the ABMC (Assignable Bandwidth + Monitoring Counters) capability, ensuring counters remain assigned even + when the corresponding RMID is not actively used by any processor. + "default": In default mode, resctrl assumes there is a hardware counter for each
@@ -284,6 +297,16 @@ with the following files: counters. This can result in misleading values or display "Unavailable" if no counter is assigned to the event. + * To enable "mbm_cntr_assign" monitoring mode: + :: + + # echo "mbm_cntr_assign" > /sys/fs/resctrl/info/L3_MON/mbm_assign_mode + + * To enable "default" monitoring mode: + :: + + # echo "default" > /sys/fs/resctrl/info/L3_MON/mbm_assign_mode + "num_mbm_cntrs": The maximum number of monitoring counters (total of available and assigned counters) in each domain when the system supports mbm_cntr_assign mode.
fs/resctrl/internal.h +3 -0
@@ -404,6 +404,9 @@ int resctrl_unassign_cntr_event(struct rdt_resource *r, struct rdt_mon_domain *d struct rdtgroup *rdtgrp, enum resctrl_event_id evtid);int mbm_cntr_get (struct rdt_resource *r,struct rdt_mon_domain *d,struct rdtgroup *rdtgrp,enum resctrl_event_id evtid); +void resctrl_reset_rmid_all (struct rdt_resource *r,struct rdt_mon_domain *d); +void mbm_cntr_free_all (struct rdt_resource *r,struct rdt_mon_domain *d); +void resctrl_init_evt_configuration (struct rdt_resource *r,bool enable);#ifdef CONFIG_RESCTRL_FS_PSEUDO_LOCK int rdtgroup_locksetup_enter (struct rdtgroup *rdtgrp);
fs/resctrl/monitor.c +47 -6
81 lines not on this page · raw message
fs/resctrl/rdtgroup.c +64 -1
79 lines not on this page · raw message
Signature · 1 line
2.34.1
Babu Moger
reviewerPATCH v13 27/27 x86/resctrl: Configure mbm_cntr_assign mode if supported
In reply to Babu MogerConfigure mbm_cntr_assign mode on AMD platforms. On AMD platforms, it is recommended to use the mbm_cntr_assign mode, if supported, to prevent the hardware from resetting counters between reads. This can result in misleading values or display "Unavailable" if no counter is assigned to the event.
The mbm_cntr_assign mode, referred to as ABMC (Assignable Bandwidth Monitoring Counters) on AMD, is enabled by default when supported by the system.
Update ABMC across all logical processors within the resctrl domain to ensure proper functionality.
---
v13 : Added the call resctrl_init_evt_configuration() to setup the event
configuration during init.
Resolved conflicts caused by the recent FS/ARCH code restructure.
v12: Moved the resctrl_arch_mbm_cntr_assign_set_one to domain_add_cpu_mon().
Updated the commit log.
v11: Commit text in imperative tone. Added few more details.
Moved resctrl_arch_mbm_cntr_assign_set_one() to monitor.c.
v10: Commit text in imperative tone.
v9: Minor code change due to merge. Actual code did not change.
v8: Renamed resctrl_arch_mbm_cntr_assign_configure to
resctrl_arch_mbm_cntr_assign_set_one.
Adde r->mon_capable check.
Commit message update.
v7: Introduced resctrl_arch_mbm_cntr_assign_configure() to configure.
Moved the default settings to rdt_get_mon_l3_config(). It should be done before the hotplug handler is called. It cannot be done at rdtgroup_init().
v6: Keeping the default enablement in arch init code for now.
This may need some discussion.
Renamed resctrl_arch_configure_abmc to resctrl_arch_mbm_cntr_assign_configure.
v5: New patch to enable ABMC by default.
--- arch/x86/kernel/cpu/resctrl/core.c | 7 +++++++ arch/x86/kernel/cpu/resctrl/internal.h | 1 + arch/x86/kernel/cpu/resctrl/monitor.c | 8 ++++++++ 3 files changed, 16 insertions(+)
arch/x86/kernel/cpu/resctrl/core.c +7 -0
@@ -514,6 +514,9 @@ static void domain_add_cpu_mon(int cpu, struct rdt_resource *r) d = container_of(hdr, struct rdt_mon_domain, hdr);cpumask_set_cpu (cpu, &d->hdr.cpu_mask); +/* Update the mbm_cntr_assign state for the CPU if supported */ +if (r->mon.mbm_cntr_assignable) +resctrl_arch_mbm_cntr_assign_set_one (r);return ; }
@@ -532,6 +535,10 @@ static void domain_add_cpu_mon(int cpu, struct rdt_resource *r) }cpumask_set_cpu (cpu, &d->hdr.cpu_mask); +/* Update the mbm_cntr_assign state for the CPU if supported */ +if (r->mon.mbm_cntr_assignable) +resctrl_arch_mbm_cntr_assign_set_one (r); +arch_mon_domain_online (r, d);if (arch_domain_mbm_alloc (r->mon.num_rmid, hw_dom)) {
arch/x86/kernel/cpu/resctrl/internal.h +1 -0
@@ -214,5 +214,6 @@ bool rdt_cpu_has(int flag);void __init intel_rdt_mbm_apply_quirk (void );void rdt_domain_reconfigure_cdp (struct rdt_resource *r); +void resctrl_arch_mbm_cntr_assign_set_one (struct rdt_resource *r);#endif /* _ASM_X86_RESCTRL_INTERNAL_H */
arch/x86/kernel/cpu/resctrl/monitor.c +8 -0
@@ -436,6 +436,7 @@ int __init rdt_get_mon_l3_config(struct rdt_resource *r)cpuid_count (0x80000020 ,5 , &eax, &ebx, &ecx, &edx); r->mon.num_mbm_cntrs = (ebx &GENMASK (15 ,0 )) +1 ; r->mon.mbm_assign_on_mkdir = true; + hw_res->mbm_cntr_assign_enabled = true; } r->mon_capable = true;
@@ -536,3 +537,10 @@ void resctrl_arch_config_cntr(struct rdt_resource *r, struct rdt_mon_domain *d,memset (am,0 ,sizeof (*am)); } } + +void resctrl_arch_mbm_cntr_assign_set_one (struct rdt_resource *r) +{ +struct rdt_hw_resource *hw_res =resctrl_to_arch_res (r); + +resctrl_abmc_set_one_amd (&hw_res->mbm_cntr_assign_enabled); +}
Signature · 1 line
2.34.1
Hi Babu,
Quoted 73 lines · Babu Moger
This series adds the support for Assignable Bandwidth Monitoring Counters (ABMC). It is also called QoS RMID Pinning feature
Series is written such that it is easier to support other assignable features supported from different vendors.
The feature details are documented in the APM listed below [1].
[1] AMD64 Architecture Programmer's Manual Volume 2: System Programming Publication # 24593 Revision 3.41 section 19.3.3.3 Assignable Bandwidth Monitoring (ABMC). The documentation is available atThe patches are based on top of commit
92a09c47464d0(tag: v6.15-rc5, tip/irq/merge) Linux 6.15-rc5 plus
https://lore.kernel.org/lkml/[email protected]/It is very clear these patches will go after James's resctrl FS/ARCH restructure. Hoping to avoid one review cycle due to the merge.
# Introduction
Users can create as many monitor groups as RMIDs supported by the hardware. However, bandwidth monitoring feature on AMD system only guarantees that RMIDs currently assigned to a processor will be tracked by hardware. The counters of any other RMIDs which are no longer being tracked will be reset to zero. The MBM event counters return "Unavailable" for the RMIDs that are not tracked by hardware. So, there can be only limited number of groups that can give guaranteed monitoring numbers. With ever changing configurations there is no way to definitely know which of these groups are being tracked for certain point of time. Users do not have the option to monitor a group or set of groups for certain period of time without worrying about counter being reset in between.
The ABMC feature provides an option to the user to assign a hardware counter to an RMID, event pair and monitor the bandwidth as long as it is assigned. The assigned RMID will be tracked by the hardware until the user unassigns it manually. There is no need to worry about counters being reset during this period. Additionally, the user can specify a bitmask identifying the specific bandwidth types from the given source to track with the counter.
Without ABMC enabled, monitoring will work in current 'default' mode without assignment option.
# History
Earlier implementation of ABMC had dependancy on BMEC (Bandwidth Monitoring Event Configuration). Peter had concerns with that implementation because it may be not be compatible with ARM's MPAM.
Here are the threads discussing the concerns and new interface to address the concerns.
https://lore.kernel.org/lkml/CALPaoCg97cLVVAcacnarp+880xjsedEWGJPXhYpy4P7=ky4MZw@mail.gmail.com/
https://lore.kernel.org/lkml/CALPaoCiii0vXOF06mfV=kVLBzhfNo0SFqt4kQGwGSGVUqvr2Dg@mail.gmail.com/Here are the finalized requirements based on the discussion:
* Remove BMEC dependency on the ABMC feature.
* Eliminate global assignment listing. The interface
/sys/fs/resctrl/info/L3_MON/mbm_assign_controlis no longer required.* Create the configuration directories at
/sys/fs/resctrl/info/L3_MON/counter_configs/.
The configuration file names should be free-form, allowing users to create them as needed.* Perform assignment listing at the group level by introducing
mbm_L3_assignments
in each monitoring group. The listing should provide the following details:Event Configuration: Specifies the event configuration applied. This will be crucial when "mkdir" on event configuration is added in the future, leading to the creation of mon_data/mon_l3_*/<event configuration>.
Quoted 5 lines
Domains: Identifies the domains where the configuration is applied, supporting multi-domain setups.
Assignment Type: Indicates whether the assignment is Exclusive (e or d), Shared (s), or Unassigned (_).
* Provide option to enable or disable auto assignment when new group is created.
So far I was able to reenable MBM on AMD implementations (for some users) while deferring on the counter assignment interface discussion by just making shared assignment the default for newly-created groups. Until they want to upgrade assignments to exclusive or break down traffic with multiple counters to watch a particular group more closely, they won't need to change any assignments.
Just pointing out that this turned out to be a useful first step in deploying ABMC support.
Quoted 8 lines
This series tries to address all the requirements listed above.
# Implementation details
Create a generic interface aimed to support user space assignment of scarce counters used for monitoring. First usage of interface is by ABMC with option to expand usage to "soft-ABMC" and MPAM counters in future.
I'll try to identify any issues I've encountered with "soft-ABMC". Hopefully I'll be able to share a sample implementation based on these patches soon.
There's now more interest in Google for allowing explicit control of where RMIDs are assigned on Intel platforms. Even though the number of RMIDs implemented by hardware tends to be roughly the number of containers they want to support, they often still need to create containers when all RMIDs have already been allocated, which is not currently allowed. Once the container has been created and starts running, it's no longer possible to move its threads into a monitoring group whenever RMIDs should become available again, so it's important for resctrl to maintain an accurate task list for a container even when RMIDs are not available.
Quoted 9 lines
Feature adds following interface files:
/sys/fs/resctrl/info/L3_MON/mbm_assign_mode: Reports the list of assignable monitoring features supported. The enclosed brackets indicate which feature is enabled.
/sys/fs/resctrl/info/L3_MON/num_mbm_cntrs: Reports the number of monitoring counters available for assignment.
Earlier I discussed with Reinette[1] what num_mbm_cntrs should represent in a "soft-ABMC" implementation where assignment is implemented by assigning an RMID, which would result in all events being assigned at once.
My main concern is how many "counters" you can assign by assigning RMIDs. I recall Reinette proposed reporting the number of groups which can be assigned separately from counters which can be assigned.
Quoted 11 lines
/sys/fs/resctrl/info/L3_MON/available_mbm_cntrs: Reports the number of monitoring counters free in each domain.
/sys/fs/resctrl/info/L3_MON/counter_configs: Directory to hold the counter configuration.
/sys/fs/resctrl/info/L3_MON/counter_configs/mbm_total_bytes/event_filter: Default configuration for MBM total events.
/sys/fs/resctrl/info/L3_MON/counter_configs/mbm_local_bytes/event_filter: Default configuration for MBM local events.
IIUC, this needs to be implemented now so you can drop BMEC with this series?
Quoted 30 lines
/sys/fs/resctrl/mbm_L3_assignments: Interface to list or modify assignment states on each group.# Examples
a. Check if ABMC support is available
#mount -t resctrl resctrl/sys/fs/resctrl/# cat
/sys/fs/resctrl/info/L3_MON/mbm_assign_mode[mbm_cntr_assign]
defaultABMC feature is detected and it is enabled.
b. Check how many ABMC counters are available.
# cat /sys/fs/resctrl/info/L3_MON/num_mbm_cntrs 32c. Check how many ABMC counters are available in each domain.
# cat /sys/fs/resctrl/info/L3_MON/available_mbm_cntrs 0=30;1=30d. Check default counter configuration.
# cat /sys/fs/resctrl/info/L3_MON/counter_configs/mbm_total_bytes/event_filter local_reads, remote_reads, local_non_temporal_writes, remote_non_temporal_writes, local_reads_slow_memory, remote_reads_slow_memory, dirty_victim_writes_all
Quoted 5 lines
# cat /sys/fs/resctrl/info/L3_MON/counter_configs/mbm_local_bytes/event_filter local_reads, local_non_temporal_writes, local_reads_slow_memorye. Series adds a new interface file "
mbm_L3_assignments" in each monitoring group
to list and modify any group's monitoring states.
To confirm, would we have "mbm_<resource_name>_assignments" for each resource where MBM-ish events could be assigned?
Quoted 4 lines
The list is displayed in the following format:
<Event configuration>:<Domain id>=<Assignment type>
For soft-ABMC assignment, is there just a single event configuration representing all the events tracked by the RMID?
Quoted 44 lines
Event configuration: A valid event configuration listed in the
/sys/fs/resctrl/info/L3_MON/counter_configsdirectory.Domain ID: A valid domain ID number.
Assignment types:
_ : No event configuration assigned
e : Event configuration assigned in exclusive mode
To list the default group states:
# cat/sys/fs/resctrl/mbm_L3_assignments
mbm_total_bytes:0=e;1=e
mbm_local_bytes:0=e;1=eTo unassign the configuration of
mbm_total_byteson domain 0:
#echo "mbm_total_bytes:0=_" >mbm_L3_assignments#catmbm_L3_assignments
mbm_total_bytes:0=_;1=e
mbm_local_bytes:0=e;1=eTo unassign the
mbm_total_bytesconfiguration on all domains:
$echo "mbm_total_bytes:*=_" >mbm_L3_assignments$catmbm_L3_assignments
mbm_total_bytes:0=_;1=_
mbm_local_bytes:0=e;1=eTo assign the
mbm_total_bytesconfiguration on all domains in exclusive mode:
$echo "mbm_total_bytes:*=e" >mbm_L3_assignments
$catmbm_L3_assignments
mbm_total_bytes:0=e;1=e
mbm_local_bytes:0=e;1=eg. Read the events
mbm_total_bytesandmbm_local_bytesof the default group.
There is no change in reading the events with ABMC. If the event is unassigned when reading, then the read will come back as "Unassigned".# cat /sys/fs/resctrl/mon_data/mon_L3_00/mbm_total_bytes 779247936 # cat /sys/fs/resctrl/mon_data/mon_L3_00/mbm_local_bytes 765207488
Quoted 8 lines
h. Check the default event configurations.
#cat
/sys/fs/resctrl/info/L3_MON/counter_configs/mbm_total_bytes/event_filterlocal_reads,remote_reads,local_non_temporal_writes,remote_non_temporal_writes,local_reads_slow_memory,remote_reads_slow_memory,dirty_victim_writes_all#cat
/sys/fs/resctrl/info/L3_MON/counter_configs/mbm_local_bytes/event_filterlocal_reads,local_non_temporal_writes,local_reads_slow_memory
These look like the BMEC event names converted from camel case. Will event filter programming be portable?
Thanks,
-Peter
Moger, Babu
reviewerHi Peter,
Thanks for trying the series.
Quoted 89 lines · Peter Newman
Hi Babu,
Quoted 73 lines · Babu Moger
This series adds the support for Assignable Bandwidth Monitoring Counters (ABMC). It is also called QoS RMID Pinning feature
Series is written such that it is easier to support other assignable features supported from different vendors.
The feature details are documented in the APM listed below [1].
[1] AMD64 Architecture Programmer's Manual Volume 2: System Programming Publication # 24593 Revision 3.41 section 19.3.3.3 Assignable Bandwidth Monitoring (ABMC). The documentation is available atThe patches are based on top of commit
92a09c47464d0(tag: v6.15-rc5, tip/irq/merge) Linux 6.15-rc5 plus
https://lore.kernel.org/lkml/[email protected]/It is very clear these patches will go after James's resctrl FS/ARCH restructure. Hoping to avoid one review cycle due to the merge.
# Introduction
Users can create as many monitor groups as RMIDs supported by the hardware. However, bandwidth monitoring feature on AMD system only guarantees that RMIDs currently assigned to a processor will be tracked by hardware. The counters of any other RMIDs which are no longer being tracked will be reset to zero. The MBM event counters return "Unavailable" for the RMIDs that are not tracked by hardware. So, there can be only limited number of groups that can give guaranteed monitoring numbers. With ever changing configurations there is no way to definitely know which of these groups are being tracked for certain point of time. Users do not have the option to monitor a group or set of groups for certain period of time without worrying about counter being reset in between.
The ABMC feature provides an option to the user to assign a hardware counter to an RMID, event pair and monitor the bandwidth as long as it is assigned. The assigned RMID will be tracked by the hardware until the user unassigns it manually. There is no need to worry about counters being reset during this period. Additionally, the user can specify a bitmask identifying the specific bandwidth types from the given source to track with the counter.
Without ABMC enabled, monitoring will work in current 'default' mode without assignment option.
# History
Earlier implementation of ABMC had dependancy on BMEC (Bandwidth Monitoring Event Configuration). Peter had concerns with that implementation because it may be not be compatible with ARM's MPAM.
Here are the threads discussing the concerns and new interface to address the concerns.
https://lore.kernel.org/lkml/CALPaoCg97cLVVAcacnarp+880xjsedEWGJPXhYpy4P7=ky4MZw@mail.gmail.com/
https://lore.kernel.org/lkml/CALPaoCiii0vXOF06mfV=kVLBzhfNo0SFqt4kQGwGSGVUqvr2Dg@mail.gmail.com/Here are the finalized requirements based on the discussion:
* Remove BMEC dependency on the ABMC feature.
* Eliminate global assignment listing. The interface
/sys/fs/resctrl/info/L3_MON/mbm_assign_controlis no longer required.* Create the configuration directories at
/sys/fs/resctrl/info/L3_MON/counter_configs/.
The configuration file names should be free-form, allowing users to create them as needed.* Perform assignment listing at the group level by introducing
mbm_L3_assignments
in each monitoring group. The listing should provide the following details:Event Configuration: Specifies the event configuration applied. This will be crucial when "mkdir" on event configuration is added in the future, leading to the creation of mon_data/mon_l3_*/<event configuration>.
Quoted 5 lines
Domains: Identifies the domains where the configuration is applied, supporting multi-domain setups.
Assignment Type: Indicates whether the assignment is Exclusive (e or d), Shared (s), or Unassigned (_).
* Provide option to enable or disable auto assignment when new group is created.
So far I was able to reenable MBM on AMD implementations (for some users) while deferring on the counter assignment interface discussion by just making shared assignment the default for newly-created groups. Until they want to upgrade assignments to exclusive or break down traffic with multiple counters to watch a particular group more closely, they won't need to change any assignments.
Quoted 2 lines
Just pointing out that this turned out to be a useful first step in deploying ABMC support.
Thank you.
Quoted 6 lines
Quoted 6 lines
This series tries to address all the requirements listed above.
# Implementation details
Quoted 7 lines
Quoted 3 lines
Create a generic interface aimed to support user space assignment of scarce counters used for monitoring. First usage of interface is by ABMC with option to expand usage to "soft-ABMC" and MPAM counters in future.
I'll try to identify any issues I've encountered with "soft-ABMC". Hopefully I'll be able to share a sample implementation based on these patches soon.
That would be wonderful.
Quoted 22 lines
There's now more interest in Google for allowing explicit control of where RMIDs are assigned on Intel platforms. Even though the number of RMIDs implemented by hardware tends to be roughly the number of containers they want to support, they often still need to create containers when all RMIDs have already been allocated, which is not currently allowed. Once the container has been created and starts running, it's no longer possible to move its threads into a monitoring group whenever RMIDs should become available again, so it's important for resctrl to maintain an accurate task list for a container even when RMIDs are not available.
Quoted 9 lines
Feature adds following interface files:
/sys/fs/resctrl/info/L3_MON/mbm_assign_mode: Reports the list of assignable monitoring features supported. The enclosed brackets indicate which feature is enabled.
/sys/fs/resctrl/info/L3_MON/num_mbm_cntrs: Reports the number of monitoring counters available for assignment.
Quoted 8 lines
Earlier I discussed with Reinette[1] what
num_mbm_cntrsshould represent in a "soft-ABMC" implementation where assignment is implemented by assigning an RMID, which would result in all events being assigned at once.My main concern is how many "counters" you can assign by assigning RMIDs. I recall Reinette proposed reporting the number of groups which can be assigned separately from counters which can be assigned.
More context may be needed here. Currently, num_mbm_cntrs indicates the number of counters available per domain, which is 32.
At the moment, we can assign 2 counters to each group, meaning each RMID can be associated with 2 hardware counters. In theory, it's possible to assign all 32 hardware counters to a group—allowing one RMID to be linked with up to 32 counters. However, we currently lack the interface to support that level of assignment.
For now, the plan is to support basic assignment and expand functionality later once we have the necessary data structure and requirements.
Quoted 7 lines
Quoted 7 lines
/sys/fs/resctrl/info/L3_MON/available_mbm_cntrs: Reports the number of monitoring counters free in each domain.
/sys/fs/resctrl/info/L3_MON/counter_configs: Directory to hold the counter configuration.
Quoted 7 lines
Quoted 5 lines
/sys/fs/resctrl/info/L3_MON/counter_configs/mbm_total_bytes/event_filter: Default configuration for MBM total events.
/sys/fs/resctrl/info/L3_MON/counter_configs/mbm_local_bytes/event_filter: Default configuration for MBM local events.IIUC, this needs to be implemented now so you can drop BMEC with this series?
This series hides the configuration files (mbm_local_bytes_config and mbm_total_bytes_config) required for BMEC when ABMC is enabled.
When the user switches back to "default" mode, BMEC becomes available again. I believe it's a good approach to keep it this way.
Quoted 31 lines
Quoted 31 lines
/sys/fs/resctrl/mbm_L3_assignments: Interface to list or modify assignment states on each group.# Examples
a. Check if ABMC support is available
#mount -t resctrl resctrl/sys/fs/resctrl/# cat
/sys/fs/resctrl/info/L3_MON/mbm_assign_mode[mbm_cntr_assign]
defaultABMC feature is detected and it is enabled.
b. Check how many ABMC counters are available.
# cat /sys/fs/resctrl/info/L3_MON/num_mbm_cntrs 32c. Check how many ABMC counters are available in each domain.
# cat /sys/fs/resctrl/info/L3_MON/available_mbm_cntrs 0=30;1=30d. Check default counter configuration.
# cat /sys/fs/resctrl/info/L3_MON/counter_configs/mbm_total_bytes/event_filter local_reads, remote_reads, local_non_temporal_writes, remote_non_temporal_writes, local_reads_slow_memory, remote_reads_slow_memory, dirty_victim_writes_all
Quoted 8 lines
Quoted 5 lines
# cat /sys/fs/resctrl/info/L3_MON/counter_configs/mbm_local_bytes/event_filter local_reads, local_non_temporal_writes, local_reads_slow_memorye. Series adds a new interface file "
mbm_L3_assignments" in each monitoring group
to list and modify any group's monitoring states.To confirm, would we have "mbm_<resource_name>_assignments" for each resource where MBM-ish events could be assigned?
This is a group-level property—it resides within each group and is not related to any specific resource.
Quoted 8 lines
Quoted 5 lines
The list is displayed in the following format:
<Event configuration>:<Domain id>=<Assignment type>
For soft-ABMC assignment, is there just a single event configuration representing all the events tracked by the RMID?
I’m not sure about the details of how soft-ABMC will be supported. It’s not available at the moment, but I believe it can be added once soft-ABMC support is in place.
Quoted 51 lines
Quoted 51 lines
Event configuration: A valid event configuration listed in the
/sys/fs/resctrl/info/L3_MON/counter_configsdirectory.Domain ID: A valid domain ID number.
Assignment types:
_ : No event configuration assigned
e : Event configuration assigned in exclusive mode
To list the default group states:
# cat/sys/fs/resctrl/mbm_L3_assignments
mbm_total_bytes:0=e;1=e
mbm_local_bytes:0=e;1=eTo unassign the configuration of
mbm_total_byteson domain 0:
#echo "mbm_total_bytes:0=_" >mbm_L3_assignments#catmbm_L3_assignments
mbm_total_bytes:0=_;1=e
mbm_local_bytes:0=e;1=eTo unassign the
mbm_total_bytesconfiguration on all domains:
$echo "mbm_total_bytes:*=_" >mbm_L3_assignments$catmbm_L3_assignments
mbm_total_bytes:0=_;1=_
mbm_local_bytes:0=e;1=eTo assign the
mbm_total_bytesconfiguration on all domains in exclusive mode:
$echo "mbm_total_bytes:*=e" >mbm_L3_assignments
$catmbm_L3_assignments
mbm_total_bytes:0=e;1=e
mbm_local_bytes:0=e;1=eg. Read the events
mbm_total_bytesandmbm_local_bytesof the default group.
There is no change in reading the events with ABMC. If the event is unassigned when reading, then the read will come back as "Unassigned".# cat /sys/fs/resctrl/mon_data/mon_L3_00/mbm_total_bytes 779247936 # cat /sys/fs/resctrl/mon_data/mon_L3_00/mbm_local_bytes 765207488h. Check the default event configurations.
#cat
/sys/fs/resctrl/info/L3_MON/counter_configs/mbm_total_bytes/event_filterlocal_reads,remote_reads,local_non_temporal_writes,remote_non_temporal_writes,local_reads_slow_memory,remote_reads_slow_memory,dirty_victim_writes_all
Quoted 5 lines
Quoted 2 lines
#cat
/sys/fs/resctrl/info/L3_MON/counter_configs/mbm_local_bytes/event_filterlocal_reads,local_non_temporal_writes,local_reads_slow_memoryThese look like the BMEC event names converted from camel case. Will event filter programming be portable?
Yes, that’s correct. The event types (reads, writes, etc.) supported by both BMEC and ABMC are the same, so I’ve used generalized names here.
As for portability, I can’t comment, since I’m not familiar with how event configuration is handled in MPAM or other architectures.
Quoted 6 lines
Thanks,
-Peter
Signature · 2 lines
Thanks Babu Moger
Reinette Chatre
maintainerHi Babu,
Quoted 2 lines · Babu
Quoted 1 line · Peter Newman
On Fri, May 16, 2025 at 12:52 AM Babu Moger <[email protected]> wrote:
...
Quoted 15 lines
Quoted 11 lines
Quoted 2 lines
/sys/fs/resctrl/info/L3_MON/num_mbm_cntrs: Reports the number of monitoring counters available for assignment.Earlier I discussed with Reinette[1] what
num_mbm_cntrsshould represent in a "soft-ABMC" implementation where assignment is implemented by assigning an RMID, which would result in all events being assigned at once.My main concern is how many "counters" you can assign by assigning RMIDs. I recall Reinette proposed reporting the number of groups which can be assigned separately from counters which can be assigned.
More context may be needed here. Currently,
num_mbm_cntrsindicates the number of counters available per domain, which is 32.
Quoted 8 lines
At the moment, we can assign 2 counters to each group, meaning each RMID can be associated with 2 hardware counters. In theory, it's possible to assign all 32 hardware counters to a group—allowing one RMID to be linked with up to 32 counters. However, we currently lack the interface to support that level of assignment.
For now, the plan is to support basic assignment and expand functionality later once we have the necessary data structure and requirements.
Looks like some requirements did not make it into this implementation. Do you recall the discussion that resulted in you writing [2]? Looks like there is a question to Peter in there on how to determine how many "counters" are available in soft-ABMC. I interpreted [3] at that time to mean that this information would be available in a future AMD publication.
Reinette
[2] https://lore.kernel.org/lkml/[email protected]/
[3] https://lore.kernel.org/lkml/CALPaoCg3KpF94g2MEmfP_Ro2mQZYFA8sKVkmb+7isotKNgdY9A@mail.gmail.com/
Moger, Babu
reviewerHi Reinette,
Quoted 29 lines · Reinette Chatre
Hi Babu,
Quoted 2 lines · Babu
Quoted 1 line · Peter Newman
On Fri, May 16, 2025 at 12:52 AM Babu Moger <[email protected]> wrote:
...
Quoted 21 lines
Quoted 11 lines
Quoted 2 lines
/sys/fs/resctrl/info/L3_MON/num_mbm_cntrs: Reports the number of monitoring counters available for assignment.Earlier I discussed with Reinette[1] what
num_mbm_cntrsshould represent in a "soft-ABMC" implementation where assignment is implemented by assigning an RMID, which would result in all events being assigned at once.My main concern is how many "counters" you can assign by assigning RMIDs. I recall Reinette proposed reporting the number of groups which can be assigned separately from counters which can be assigned.
More context may be needed here. Currently,
num_mbm_cntrsindicates the number of counters available per domain, which is 32.At the moment, we can assign 2 counters to each group, meaning each RMID can be associated with 2 hardware counters. In theory, it's possible to assign all 32 hardware counters to a group—allowing one RMID to be linked with up to 32 counters. However, we currently lack the interface to support that level of assignment.
Quoted 8 lines
Quoted 2 lines
For now, the plan is to support basic assignment and expand functionality later once we have the necessary data structure and requirements.
Looks like some requirements did not make it into this implementation. Do you recall the discussion that resulted in you writing [2]? Looks like there is a question to Peter in there on how to determine how many "counters" are available in soft-ABMC. I interpreted [3] at that time to mean that this information would be available in a future AMD publication.
We already have a method to determine the number of counters in soft-ABMC mode, which Peter has addressed [4].
[4]
https://lore.kernel.org/lkml/[email protected]/
This appears to be more of a workaround, and I doubt it will be included in any official AMD documentation. Additionally, the long-term direction is moving towards ABMC.
I don’t believe this workaround needs to be part of the current series. It can be added later when soft-ABMC is implemented.
Quoted 5 lines
Reinette
[2] https://lore.kernel.org/lkml/[email protected]/
[3] https://lore.kernel.org/lkml/CALPaoCg3KpF94g2MEmfP_Ro2mQZYFA8sKVkmb+7isotKNgdY9A@mail.gmail.com/
Signature · 2 lines
Thanks Babu Moger
Reinette Chatre
maintainerHi Babu,
Quoted 47 lines · Babu
Hi Reinette,
Quoted 29 lines · Reinette Chatre
Hi Babu,
Quoted 2 lines · Babu
Quoted 1 line · Peter Newman
On Fri, May 16, 2025 at 12:52 AM Babu Moger <[email protected]> wrote:
...
Quoted 21 lines
Quoted 11 lines
Quoted 2 lines
/sys/fs/resctrl/info/L3_MON/num_mbm_cntrs: Reports the number of monitoring counters available for assignment.Earlier I discussed with Reinette[1] what
num_mbm_cntrsshould represent in a "soft-ABMC" implementation where assignment is implemented by assigning an RMID, which would result in all events being assigned at once.My main concern is how many "counters" you can assign by assigning RMIDs. I recall Reinette proposed reporting the number of groups which can be assigned separately from counters which can be assigned.
More context may be needed here. Currently,
num_mbm_cntrsindicates the number of counters available per domain, which is 32.At the moment, we can assign 2 counters to each group, meaning each RMID can be associated with 2 hardware counters. In theory, it's possible to assign all 32 hardware counters to a group—allowing one RMID to be linked with up to 32 counters. However, we currently lack the interface to support that level of assignment.
Quoted 8 lines
Quoted 2 lines
For now, the plan is to support basic assignment and expand functionality later once we have the necessary data structure and requirements.
Looks like some requirements did not make it into this implementation. Do you recall the discussion that resulted in you writing [2]? Looks like there is a question to Peter in there on how to determine how many "counters" are available in soft-ABMC. I interpreted [3] at that time to mean that this information would be available in a future AMD publication.
We already have a method to determine the number of counters in soft-ABMC mode, which Peter has addressed [4].
Quoted 6 lines
This appears to be more of a workaround, and I doubt it will be included in any official AMD documentation. Additionally, the long-term direction is moving towards ABMC.
I don’t believe this workaround needs to be part of the current series. It can be added later when soft-ABMC is implemented.
Agreed. What about the plans described in [2]? (Thanks to Peter for catching this!).
It is important to keep track of requirements while working on a feature to ensure that the implementation supports the planned use cases. Re-reading that thread it is not clear to me how soft-ABMC's per-group assignment would look. Could you please share how you see it progress from this implementation? This includes the single event vs. multiple event assignment. I would like to highlight that this is not a request for this to be supported in this implementation but there needs to be a plan for how this can be supported on top of interfaces established by this work.
Reinette
Quoted 7 lines
Hi Reinette,
Quoted 60 lines · Reinette Chatre
Hi Babu,
Quoted 47 lines · Babu
Hi Reinette,
Quoted 29 lines · Reinette Chatre
Hi Babu,
Quoted 2 lines · Babu
Quoted 1 line · Peter Newman
On Fri, May 16, 2025 at 12:52 AM Babu Moger <[email protected]> wrote:
...
Quoted 21 lines
Quoted 11 lines
Quoted 2 lines
/sys/fs/resctrl/info/L3_MON/num_mbm_cntrs: Reports the number of monitoring counters available for assignment.Earlier I discussed with Reinette[1] what
num_mbm_cntrsshould represent in a "soft-ABMC" implementation where assignment is implemented by assigning an RMID, which would result in all events being assigned at once.My main concern is how many "counters" you can assign by assigning RMIDs. I recall Reinette proposed reporting the number of groups which can be assigned separately from counters which can be assigned.
More context may be needed here. Currently,
num_mbm_cntrsindicates the number of counters available per domain, which is 32.At the moment, we can assign 2 counters to each group, meaning each RMID can be associated with 2 hardware counters. In theory, it's possible to assign all 32 hardware counters to a group—allowing one RMID to be linked with up to 32 counters. However, we currently lack the interface to support that level of assignment.
Quoted 8 lines
Quoted 2 lines
For now, the plan is to support basic assignment and expand functionality later once we have the necessary data structure and requirements.
Looks like some requirements did not make it into this implementation. Do you recall the discussion that resulted in you writing [2]? Looks like there is a question to Peter in there on how to determine how many "counters" are available in soft-ABMC. I interpreted [3] at that time to mean that this information would be available in a future AMD publication.
We already have a method to determine the number of counters in soft-ABMC mode, which Peter has addressed [4].
Quoted 6 lines
This appears to be more of a workaround, and I doubt it will be included in any official AMD documentation. Additionally, the long-term direction is moving towards ABMC.
I don’t believe this workaround needs to be part of the current series. It can be added later when soft-ABMC is implemented.
Agreed. What about the plans described in [2]? (Thanks to Peter for catching this!).
Quoted 9 lines
It is important to keep track of requirements while working on a feature to ensure that the implementation supports the planned use cases. Re-reading that thread it is not clear to me how soft-ABMC's per-group assignment would look. Could you please share how you see it progress from this implementation? This includes the single event vs. multiple event assignment. I would like to highlight that this is not a request for this to be supported in this implementation but there needs to be a plan for how this can be supported on top of interfaces established by this work.
Here’s my current understanding of soft-ABMC. Peter may have a more in-depth perspective on this.
Soft-ABMC:
a. num_mbm_cntrs: This is a software-defined limit based on the number of active RMIDs that can be supported. The value can be obtained using the code referenced in [4].
b. Assignments: No hardware configuration is required. We simply need to ensure that no more than num_mbm_cntrs RMIDs are active at any given time.
c. Configuration: Controlled via /info/L3_MON/mbm_total_bytes_config and mbm_local_bytes_config.
d. Events: Only two events can be assigned(local and total).
ABMC:
a. num_mbm_cntrs: This is defined by the hardware.
b. Assignments: Requires special MSR writes to assign counters.
c. Configuration: Comes from /info/L3_MON/counter_configs/.
d. Events: More than two events can be assigned to a group (currently up to 2).
Commonalities:
a. Assignments can be either exclusive or shared in both these modes.
Given these, I believe we can easily accommodate soft-ABMC in this interface.
Quoted 6 lines
Quoted 4 lines
Reinette Chatre
maintainerHi Babu,
Quoted 90 lines · Babu
Hi Reinette,
Quoted 60 lines · Reinette Chatre
Hi Babu,
Quoted 47 lines · Babu
Hi Reinette,
Quoted 29 lines · Reinette Chatre
Hi Babu,
Quoted 2 lines · Babu
Quoted 1 line · Peter Newman
On Fri, May 16, 2025 at 12:52 AM Babu Moger <[email protected]> wrote:
...
Quoted 21 lines
Quoted 11 lines
Quoted 2 lines
/sys/fs/resctrl/info/L3_MON/num_mbm_cntrs: Reports the number of monitoring counters available for assignment.Earlier I discussed with Reinette[1] what
num_mbm_cntrsshould represent in a "soft-ABMC" implementation where assignment is implemented by assigning an RMID, which would result in all events being assigned at once.My main concern is how many "counters" you can assign by assigning RMIDs. I recall Reinette proposed reporting the number of groups which can be assigned separately from counters which can be assigned.
More context may be needed here. Currently,
num_mbm_cntrsindicates the number of counters available per domain, which is 32.At the moment, we can assign 2 counters to each group, meaning each RMID can be associated with 2 hardware counters. In theory, it's possible to assign all 32 hardware counters to a group—allowing one RMID to be linked with up to 32 counters. However, we currently lack the interface to support that level of assignment.
Quoted 8 lines
Quoted 2 lines
For now, the plan is to support basic assignment and expand functionality later once we have the necessary data structure and requirements.
Looks like some requirements did not make it into this implementation. Do you recall the discussion that resulted in you writing [2]? Looks like there is a question to Peter in there on how to determine how many "counters" are available in soft-ABMC. I interpreted [3] at that time to mean that this information would be available in a future AMD publication.
We already have a method to determine the number of counters in soft-ABMC mode, which Peter has addressed [4].
Quoted 6 lines
This appears to be more of a workaround, and I doubt it will be included in any official AMD documentation. Additionally, the long-term direction is moving towards ABMC.
I don’t believe this workaround needs to be part of the current series. It can be added later when soft-ABMC is implemented.
Agreed. What about the plans described in [2]? (Thanks to Peter for catching this!).
Quoted 9 lines
It is important to keep track of requirements while working on a feature to ensure that the implementation supports the planned use cases. Re-reading that thread it is not clear to me how soft-ABMC's per-group assignment would look. Could you please share how you see it progress from this implementation? This includes the single event vs. multiple event assignment. I would like to highlight that this is not a request for this to be supported in this implementation but there needs to be a plan for how this can be supported on top of interfaces established by this work.
Here’s my current understanding of soft-ABMC. Peter may have a more in-depth perspective on this.
Soft-ABMC:
a.num_mbm_cntrs: This is a software-defined limit based on the number of active RMIDs that can be supported. The value can be obtained using the code referenced in [4].b. Assignments: No hardware configuration is required. We simply need to ensure that no more than
num_mbm_cntrsRMIDs are active at any given time.c. Configuration: Controlled via /info/L3_MON/mbm_total_bytes_config and
mbm_local_bytes_config.d. Events: Only two events can be assigned(local and total).
ABMC:
a.num_mbm_cntrs: This is defined by the hardware.
b. Assignments: Requires special MSR writes to assign counters.
c. Configuration: Comes from /info/L3_MON/counter_configs/.
d. Events: More than two events can be assigned to a group (currently up to 2).
Quoted 4 lines
Commonalities:
a. Assignments can be either exclusive or shared in both these modes.Given these, I believe we can easily accommodate soft-ABMC in this interface.
This is not so obvious to me. It looks to me as though the user is forced to interpret the content of resctrl files differently based on soft-ABMC vs ABMC making the interface inconsistent and user thus needing to know details of implementations. This is what the previous discussion I linked to aimed to address. It sounds to me as though you believe that this is no longer an issue. Could you please show examples of what a user can expect from the interfaces and how a user will interact with the interfaces on both a non-ABMC and ABMC system?
Thank you
Reinette
Quoted 8 lines
Quoted 7 lines
Quoted 5 lines
Hi Babu/Reinette,
Quoted 81 lines · Reinette Chatre
Hi Babu,
Quoted 77 lines · Babu
Hi Reinette,
Quoted 60 lines · Reinette Chatre
Hi Babu,
Quoted 47 lines · Babu
Hi Reinette,
Quoted 29 lines · Reinette Chatre
Hi Babu,
Quoted 2 lines · Babu
Quoted 1 line · Peter Newman
On Fri, May 16, 2025 at 12:52 AM Babu Moger <[email protected]> wrote:
...
Quoted 21 lines
Quoted 11 lines
Quoted 2 lines
/sys/fs/resctrl/info/L3_MON/num_mbm_cntrs: Reports the number of monitoring counters available for assignment.Earlier I discussed with Reinette[1] what
num_mbm_cntrsshould represent in a "soft-ABMC" implementation where assignment is implemented by assigning an RMID, which would result in all events being assigned at once.My main concern is how many "counters" you can assign by assigning RMIDs. I recall Reinette proposed reporting the number of groups which can be assigned separately from counters which can be assigned.
More context may be needed here. Currently,
num_mbm_cntrsindicates the number of counters available per domain, which is 32.At the moment, we can assign 2 counters to each group, meaning each RMID can be associated with 2 hardware counters. In theory, it's possible to assign all 32 hardware counters to a group—allowing one RMID to be linked with up to 32 counters. However, we currently lack the interface to support that level of assignment.
Quoted 8 lines
Quoted 2 lines
For now, the plan is to support basic assignment and expand functionality later once we have the necessary data structure and requirements.
Looks like some requirements did not make it into this implementation. Do you recall the discussion that resulted in you writing [2]? Looks like there is a question to Peter in there on how to determine how many "counters" are available in soft-ABMC. I interpreted [3] at that time to mean that this information would be available in a future AMD publication.
We already have a method to determine the number of counters in soft-ABMC mode, which Peter has addressed [4].
Quoted 6 lines
This appears to be more of a workaround, and I doubt it will be included in any official AMD documentation. Additionally, the long-term direction is moving towards ABMC.
I don’t believe this workaround needs to be part of the current series. It can be added later when soft-ABMC is implemented.
Agreed. What about the plans described in [2]? (Thanks to Peter for catching this!).
Quoted 9 lines
It is important to keep track of requirements while working on a feature to ensure that the implementation supports the planned use cases. Re-reading that thread it is not clear to me how soft-ABMC's per-group assignment would look. Could you please share how you see it progress from this implementation? This includes the single event vs. multiple event assignment. I would like to highlight that this is not a request for this to be supported in this implementation but there needs to be a plan for how this can be supported on top of interfaces established by this work.
Here’s my current understanding of soft-ABMC. Peter may have a more in-depth perspective on this.
Soft-ABMC:
a.num_mbm_cntrs: This is a software-defined limit based on the number of active RMIDs that can be supported. The value can be obtained using the code referenced in [4].
I would call it a hardware-defined limit that can be probed by software.
The main question is whether this file returns the exact number of RMIDs hardware can track or double that number (mbm_total_bytes + mbm_local_bytes) so that the value is always measured in events.
There's also the mongroup-RMID overcommit use case I described above[1]. On Intel we can safely assume that there are counters to back all RMIDs, so num_mbm_cntrs would be calculated directly from num_rmids.
I realized this use case is more difficult to implement on MPAM, because a PARTID is effectively a CLOSID+RMID, so deferring assigning a unique PARTID to a group also results in it being in a different allocation group. It will work if the unmonitored groups could find a way to share PARTIDs, but this has consequences on allocation - but hopefully no worse than sharing CLOSIDs on x86.
There's a lot of interest in monitoring ID overcommit in Google, so I think it's worth it for me to investigate the additional structural changes needed in resctrl (i.e., breaking the FS-level association between mongroups and HW monitoring IDs). Such a framework could be a better fit for soft-ABMC. For example, if overcommit is allowed, we would just report the number of simultaneous RMIDs we were able to probe as num_rmids. I would want the same shared assignment scheduler to be able to work with RMIDs and counters, though.
Thanks,
-Peter
[1] https://lore.kernel.org/lkml/CALPaoChSzzU5mzMZsdT6CeyEn0WD1qdT9fKCoNW_ty4tojtrkw@mail.gmail.com/
Reinette Chatre
maintainerHi Peter and Babu,
Quoted 83 lines · Peter Newman
Hi Babu/Reinette,
Quoted 79 lines · Reinette Chatre
Hi Babu,
Quoted 75 lines · Babu
Hi Reinette,
Quoted 60 lines · Reinette Chatre
Hi Babu,
Quoted 47 lines · Babu
Hi Reinette,
Quoted 29 lines · Reinette Chatre
Hi Babu,
Quoted 2 lines · Babu
Quoted 1 line · Peter Newman
On Fri, May 16, 2025 at 12:52 AM Babu Moger <[email protected]> wrote:
...
Quoted 21 lines
Quoted 11 lines
Quoted 2 lines
/sys/fs/resctrl/info/L3_MON/num_mbm_cntrs: Reports the number of monitoring counters available for assignment.Earlier I discussed with Reinette[1] what
num_mbm_cntrsshould represent in a "soft-ABMC" implementation where assignment is implemented by assigning an RMID, which would result in all events being assigned at once.My main concern is how many "counters" you can assign by assigning RMIDs. I recall Reinette proposed reporting the number of groups which can be assigned separately from counters which can be assigned.
More context may be needed here. Currently,
num_mbm_cntrsindicates the number of counters available per domain, which is 32.At the moment, we can assign 2 counters to each group, meaning each RMID can be associated with 2 hardware counters. In theory, it's possible to assign all 32 hardware counters to a group—allowing one RMID to be linked with up to 32 counters. However, we currently lack the interface to support that level of assignment.
Quoted 8 lines
Quoted 2 lines
For now, the plan is to support basic assignment and expand functionality later once we have the necessary data structure and requirements.
Looks like some requirements did not make it into this implementation. Do you recall the discussion that resulted in you writing [2]? Looks like there is a question to Peter in there on how to determine how many "counters" are available in soft-ABMC. I interpreted [3] at that time to mean that this information would be available in a future AMD publication.
We already have a method to determine the number of counters in soft-ABMC mode, which Peter has addressed [4].
Quoted 6 lines
This appears to be more of a workaround, and I doubt it will be included in any official AMD documentation. Additionally, the long-term direction is moving towards ABMC.
I don’t believe this workaround needs to be part of the current series. It can be added later when soft-ABMC is implemented.
Agreed. What about the plans described in [2]? (Thanks to Peter for catching this!).
Quoted 9 lines
It is important to keep track of requirements while working on a feature to ensure that the implementation supports the planned use cases. Re-reading that thread it is not clear to me how soft-ABMC's per-group assignment would look. Could you please share how you see it progress from this implementation? This includes the single event vs. multiple event assignment. I would like to highlight that this is not a request for this to be supported in this implementation but there needs to be a plan for how this can be supported on top of interfaces established by this work.
Here’s my current understanding of soft-ABMC. Peter may have a more in-depth perspective on this.
Quoted 8 lines
Quoted 2 lines
Quoted 2 lines
Soft-ABMC:
a.num_mbm_cntrs: This is a software-defined limit based on the number of active RMIDs that can be supported. The value can be obtained using the code referenced in [4].I would call it a hardware-defined limit that can be probed by software.
The main question is whether this file returns the exact number of RMIDs hardware can track or double that number (
mbm_total_bytes+mbm_local_bytes) so that the value is always measured in events.
tl;dr: I continue [3] to find it most intuitive for num_mbm_cntrs to be the exact number of "active" RMIDs that the system can support *and* changing the name of the modes to help user interpret num_mbm_cntrs: "mbm_cntr_event_assign" for ABMC, "mbm_cntr_group_assign" for soft-ABMC.
details
-------
We are now back to the previous discussion about what user can expect from the interface. Let me try and re-cap that discussion so that we can all hopefully get back on the same page. Please add corrections/updates where needed.
soft-ABMC
---------
soft-ABMC manages "active" (term TBD) RMID assignment to monitor groups. When an "active" RMID is assigned to a monitor group then *all* MBM events (not LLC occupancy) in that monitor group are counted. "Active" RMID assignment can be done per domain.
Requirement: resctrl should accurately reflect which events are counted. That is, we do not want resctrl to pretend to allow user to assign an "active" RMID to only one event in a monitor group while all events are actually counted.
Caveat: To support rapid re-assignment of RMIDs to monitor groups, llc_occupancy event is disabled when soft-ABMC is enabled.
ABMC
----
ABMC manages (hardware) counter assignment to monitor group (RMID), event pairs. When a hardware counter is assigned to an RMID, event pair then only that RMID, event is counted. Hardware counter assignment can be done per domain.
shared assignment
-----------------
A shared assignment applies to both soft-ABMC and ABMC. A user can designate a "counter" (could be hardware counter or "active" RMID) as shared and that means the counter within that domain is shared between different monitor groups and actual assignment is scheduled by resctrl.
user interface
--------------
Next, consider the interface while keeping above definitions and requirements in mind.
This series introduces (using implementation, not cover-letter):
/sys/fs/resctrl/info/L3_MON/num_mbm_cntrs
"num_mbm_cntrs":
The maximum number of monitoring counters (total of available and assigned counters) in each domain when the system supports mbm_cntr_assign mode.
/sys/fs/resctrl/mbm_L3_assignments
"mbm_L3_assignments":
This interface file is created when the mbm_cntr_assign mode is supported and shows the assignment status for each group.
Consider "mbm_L3_assignments" first. The interface is documented for ABMC support where it is possible to manage individual event assignment within monitor group.
For ABMC it is possible to assign just one event at a time and doing so consumes one counter in that domain:
a) Starting state on system with 32 counters per domain, two events in default resource group consumes two counters in that domain:
# cat /sys/fs/resctrl/info/L3_MON/num_mbm_cntrs 0=30;1=32 # cat /sys/fs/resctrl/mbm_L3_assignments mbm_total_bytes:0=e;1=_ mbm_local_bytes:0=e;1=_
b) Assign counter to mbm_local_bytes in domain 1:
# echo "mbm_local_bytes:1=e" > /sys/fs/resctrl/mbm_L3_assignments # cat /sys/fs/resctrl/mbm_L3_assignments mbm_total_bytes:0=e;1=_ mbm_local_bytes:0=e;1=e # cat /sys/fs/resctrl/info/L3_MON/num_mbm_cntrs 0=30;1=31
The question is how this should look on soft-ABMC system. Let's say hypothetically that on a soft-ABMC system it is possible to have 32 "active" RMIDs.
a) Starting state on system with 32 "active RMIDs" per domain, two events in default resource group consumes one RMID in that domain:
# cat /sys/fs/resctrl/mbm_L3_assignments mbm_total_bytes:0=e;1=_ mbm_local_bytes:0=e;1=_
What should num_mbm_cntrs display?
Option A (counters are RMIDs):
# cat /sys/fs/resctrl/info/L3_MON/num_mbm_cntrs 0=31;1=32
Option B (pretend RMIDs are events):
# cat /sys/fs/resctrl/info/L3_MON/num_mbm_cntrs 0=62;1=64
b) Assign counter to mbm_local_bytes in domain 1:
# echo "mbm_local_bytes:1=e" > /sys/fs/resctrl/mbm_L3_assignments # cat /sys/fs/resctrl/mbm_L3_assignments mbm_total_bytes:0=e;1=e mbm_local_bytes:0=e;1=e
Note that even though user requested only mbm_local_bytes to be assigned, it actually results in both mbm_total_bytes and mbm_local_bytes to be assigned. This ensures accurate state representation to user space but this also creates an inconsistent user interface between soft-ABMC and ABMC since user space intends to use the same interface but "sometimes" assigning one event results in assign of one event while "sometimes" it results in assign of multiple events.
wrt "num_mbm_cntrs"
Option A (counters are RMIDs):
# cat /sys/fs/resctrl/info/L3_MON/num_mbm_cntrs 0=31;1=31
Option B (pretend RMIDs are events):
# cat /sys/fs/resctrl/info/L3_MON/num_mbm_cntrs 0=62;1=62
Neither option seems ideal to me since the interface cannot be consistent between ABMC and soft-ABMC.
As I mentioned in [2] it is not possible to hide ABMC and soft-ABMC behind the same interface. When user space wants to monitor a particular monitor group then it should be clear how that can be accomplished. Not knowing if an assignment/unassignment to/from an event would impact one or all events and whether it will consume one or multiple counters does not sound like a good interface to me.
As I understand current interface, user is required to know how ABMC and soft-ABMC is implemented to be able to configure the system. For example, if user has file like:
# cat /sys/fs/resctrl/mbm_L3_assignments mbm_total_bytes:0=e;1=e mbm_local_bytes:0=e;1=e user must know underlying implementation to be able to manage monitoring of events and assigning counters otherwise it will be a surprise to lose monitoring of all events when unassigning one event.
This is why I proposed in [3] that the name of the mode reflects how user can interact with the system. Instead of one "mbm_cntr_assign" mode there can be "mbm_cntr_event_assign" that is used for ABMC and "mbm_cntr_group_assign" that is used for soft-ABMC. The mode should make it clear what the system is capable of wrt counter assignments.
Considering this the interface should be clear:num_mbm_cntrs: reflects the number of counters in each domain that can be assigned. In "mbm_cntr_event_assign" this will be the number of counters that can be assigned to each event within a monitoring group, in "mbm_cntr_group_assign" this will be the number of counters that can be assigned to entire monitoring groups impacting all MBM events.
mbm_L3_assignments: manages the counter assignment in each group. When user knows the mode is "mbm_cntr_event_assign"/"mbm_cntr_group_assign" then it should be clear to user space how the interface behaves wrt assignment, no surprises of multiple events impacted when assigning/unassigning single event.
For soft-ABMC I thus find it most intuitive for num_mbm_cntrs to be the exact number of "active" RMIDs that the system can support *and* changing the name of the modes to help user interpret num_mbm_cntrs.
Quoted 5 lines
There's also the mongroup-RMID overcommit use case I described above[1]. On Intel we can safely assume that there are counters to back all RMIDs, so
num_mbm_cntrswould be calculated directly fromnum_rmids.
This is about the:
There's now more interest in Google for allowing explicit control of where RMIDs are assigned on Intel platforms. Even though the number of RMIDs implemented by hardware tends to be roughly the number of containers they want to support, they often still need to create containers when all RMIDs have already been allocated, which is not currently allowed. Once the container has been created and starts running, it's no longer possible to move its threads into a monitoring group whenever RMIDs should become available again, so it's important for resctrl to maintain an accurate task list for a container even when RMIDs are not available.
I see a monitor group as a collection of tasks that need to be monitored together. The "task list" is the group of tasks that share a monitoring ID that is required to be a valid ID since when any of the tasks are scheduled that ID is written to the hardware. I intentionally tried to not use RMID since I believe this is required for all archs.
I thus do not understand how a task can start running when it does not have a valid monitoring ID. The idea of "deferred assignment" is not clear to me, there can never be "unmonitored tasks", no? I think I am missing something here.
Quoted 16 lines
I realized this use case is more difficult to implement on MPAM, because a PARTID is effectively a CLOSID+RMID, so deferring assigning a unique PARTID to a group also results in it being in a different allocation group. It will work if the unmonitored groups could find a way to share PARTIDs, but this has consequences on allocation - but hopefully no worse than sharing CLOSIDs on x86.
There's a lot of interest in monitoring ID overcommit in Google, so I think it's worth it for me to investigate the additional structural changes needed in resctrl (i.e., breaking the FS-level association between mongroups and HW monitoring IDs). Such a framework could be a better fit for soft-ABMC. For example, if overcommit is allowed, we would just report the number of simultaneous RMIDs we were able to probe as
num_rmids. I would want the same shared assignment scheduler to be able to work with RMIDs and counters, though.
Quoted 4 lines
Thanks,
-Peter[1] https://lore.kernel.org/lkml/CALPaoChSzzU5mzMZsdT6CeyEn0WD1qdT9fKCoNW_ty4tojtrkw@mail.gmail.com/
Reinette
[2] https://lore.kernel.org/lkml/[email protected]/
[3] https://lore.kernel.org/lkml/[email protected]/
Luck, Tony
maintainerQuoted 273 lines · Reinette Chatre
Hi Peter and Babu,
Quoted 83 lines · Peter Newman
Hi Babu/Reinette,
Quoted 79 lines · Reinette Chatre
Hi Babu,
Quoted 75 lines · Babu
Hi Reinette,
Quoted 60 lines · Reinette Chatre
Hi Babu,
Quoted 47 lines · Babu
Hi Reinette,
Quoted 29 lines · Reinette Chatre
Hi Babu,
Quoted 2 lines · Babu
Quoted 1 line · Peter Newman
On Fri, May 16, 2025 at 12:52 AM Babu Moger <[email protected]> wrote:
...
Quoted 21 lines
Quoted 11 lines
Quoted 2 lines
/sys/fs/resctrl/info/L3_MON/num_mbm_cntrs: Reports the number of monitoring counters available for assignment.Earlier I discussed with Reinette[1] what
num_mbm_cntrsshould represent in a "soft-ABMC" implementation where assignment is implemented by assigning an RMID, which would result in all events being assigned at once.My main concern is how many "counters" you can assign by assigning RMIDs. I recall Reinette proposed reporting the number of groups which can be assigned separately from counters which can be assigned.
More context may be needed here. Currently,
num_mbm_cntrsindicates the number of counters available per domain, which is 32.At the moment, we can assign 2 counters to each group, meaning each RMID can be associated with 2 hardware counters. In theory, it's possible to assign all 32 hardware counters to a group—allowing one RMID to be linked with up to 32 counters. However, we currently lack the interface to support that level of assignment.
Quoted 8 lines
Quoted 2 lines
For now, the plan is to support basic assignment and expand functionality later once we have the necessary data structure and requirements.
Looks like some requirements did not make it into this implementation. Do you recall the discussion that resulted in you writing [2]? Looks like there is a question to Peter in there on how to determine how many "counters" are available in soft-ABMC. I interpreted [3] at that time to mean that this information would be available in a future AMD publication.
We already have a method to determine the number of counters in soft-ABMC mode, which Peter has addressed [4].
Quoted 6 lines
This appears to be more of a workaround, and I doubt it will be included in any official AMD documentation. Additionally, the long-term direction is moving towards ABMC.
I don’t believe this workaround needs to be part of the current series. It can be added later when soft-ABMC is implemented.
Agreed. What about the plans described in [2]? (Thanks to Peter for catching this!).
Quoted 9 lines
It is important to keep track of requirements while working on a feature to ensure that the implementation supports the planned use cases. Re-reading that thread it is not clear to me how soft-ABMC's per-group assignment would look. Could you please share how you see it progress from this implementation? This includes the single event vs. multiple event assignment. I would like to highlight that this is not a request for this to be supported in this implementation but there needs to be a plan for how this can be supported on top of interfaces established by this work.
Here’s my current understanding of soft-ABMC. Peter may have a more in-depth perspective on this.
Quoted 8 lines
Quoted 2 lines
Quoted 2 lines
Soft-ABMC:
a.num_mbm_cntrs: This is a software-defined limit based on the number of active RMIDs that can be supported. The value can be obtained using the code referenced in [4].I would call it a hardware-defined limit that can be probed by software.
The main question is whether this file returns the exact number of RMIDs hardware can track or double that number (
mbm_total_bytes+mbm_local_bytes) so that the value is always measured in events.tl;dr: I continue [3] to find it most intuitive for
num_mbm_cntrsto be the exact number of "active" RMIDs that the system can support *and* changing the name of the modes to help user interpretnum_mbm_cntrs: "mbm_cntr_event_assign" for ABMC, "mbm_cntr_group_assign" for soft-ABMC.details
-------We are now back to the previous discussion about what user can expect from the interface. Let me try and re-cap that discussion so that we can all hopefully get back on the same page. Please add corrections/updates where needed.
soft-ABMC
---------
soft-ABMC manages "active" (term TBD) RMID assignment to monitor groups. When an "active" RMID is assigned to a monitor group then *all* MBM events (not LLC occupancy) in that monitor group are counted. "Active" RMID assignment can be done per domain.Requirement: resctrl should accurately reflect which events are counted. That is, we do not want resctrl to pretend to allow user to assign an "active" RMID to only one event in a monitor group while all events are actually counted.
Caveat: To support rapid re-assignment of RMIDs to monitor groups,
llc_occupancyevent is disabled when soft-ABMC is enabled.ABMC
----
ABMC manages (hardware) counter assignment to monitor group (RMID), event pairs. When a hardware counter is assigned to an RMID, event pair then only that RMID, event is counted. Hardware counter assignment can be done per domain.shared assignment
-----------------
A shared assignment applies to both soft-ABMC and ABMC. A user can designate a "counter" (could be hardware counter or "active" RMID) as shared and that means the counter within that domain is shared between different monitor groups and actual assignment is scheduled by resctrl.user interface
--------------Next, consider the interface while keeping above definitions and requirements in mind.
This series introduces (using implementation, not cover-letter):
/sys/fs/resctrl/info/L3_MON/num_mbm_cntrs
"num_mbm_cntrs":
The maximum number of monitoring counters (total of available and assigned counters) in each domain when the system supportsmbm_cntr_assignmode.
/sys/fs/resctrl/mbm_L3_assignments
"mbm_L3_assignments":
This interface file is created when thembm_cntr_assignmode is supported and shows the assignment status for each group.Consider "
mbm_L3_assignments" first. The interface is documented for ABMC support where it is possible to manage individual event assignment within monitor group.For ABMC it is possible to assign just one event at a time and doing so consumes one counter in that domain:
a) Starting state on system with 32 counters per domain, two events in default resource group consumes two counters in that domain:
# cat /sys/fs/resctrl/info/L3_MON/num_mbm_cntrs 0=30;1=32 # cat /sys/fs/resctrl/mbm_L3_assignments mbm_total_bytes:0=e;1=_ mbm_local_bytes:0=e;1=_b) Assign counter to
mbm_local_bytesin domain 1:# echo "mbm_local_bytes:1=e" > /sys/fs/resctrl/mbm_L3_assignments # cat /sys/fs/resctrl/mbm_L3_assignments mbm_total_bytes:0=e;1=_ mbm_local_bytes:0=e;1=e # cat /sys/fs/resctrl/info/L3_MON/num_mbm_cntrs 0=30;1=31The question is how this should look on soft-ABMC system. Let's say hypothetically that on a soft-ABMC system it is possible to have 32 "active" RMIDs.
a) Starting state on system with 32 "active RMIDs" per domain, two events in default resource group consumes one RMID in that domain:
# cat /sys/fs/resctrl/mbm_L3_assignments mbm_total_bytes:0=e;1=_ mbm_local_bytes:0=e;1=_What should
num_mbm_cntrsdisplay?Option A (counters are RMIDs):
# cat /sys/fs/resctrl/info/L3_MON/num_mbm_cntrs 0=31;1=32Option B (pretend RMIDs are events):
# cat /sys/fs/resctrl/info/L3_MON/num_mbm_cntrs 0=62;1=64b) Assign counter to
mbm_local_bytesin domain 1:# echo "mbm_local_bytes:1=e" > /sys/fs/resctrl/mbm_L3_assignments # cat /sys/fs/resctrl/mbm_L3_assignments mbm_total_bytes:0=e;1=e mbm_local_bytes:0=e;1=eNote that even though user requested only
mbm_local_bytesto be assigned, it actually results in bothmbm_total_bytesandmbm_local_bytesto be assigned. This ensures accurate state representation to user space but this also creates an inconsistent user interface between soft-ABMC and ABMC since user space intends to use the same interface but "sometimes" assigning one event results in assign of one event while "sometimes" it results in assign of multiple events.wrt "
num_mbm_cntrs"Option A (counters are RMIDs):
# cat /sys/fs/resctrl/info/L3_MON/num_mbm_cntrs 0=31;1=31Option B (pretend RMIDs are events):
# cat /sys/fs/resctrl/info/L3_MON/num_mbm_cntrs 0=62;1=62Neither option seems ideal to me since the interface cannot be consistent between ABMC and soft-ABMC.
As I mentioned in [2] it is not possible to hide ABMC and soft-ABMC behind the same interface. When user space wants to monitor a particular monitor group then it should be clear how that can be accomplished. Not knowing if an assignment/unassignment to/from an event would impact one or all events and whether it will consume one or multiple counters does not sound like a good interface to me.As I understand current interface, user is required to know how ABMC and soft-ABMC is implemented to be able to configure the system. For example, if user has file like:
# cat /sys/fs/resctrl/mbm_L3_assignments mbm_total_bytes:0=e;1=e mbm_local_bytes:0=e;1=e user must know underlying implementation to be able to manage monitoring of events and assigning counters otherwise it will be a surprise to lose monitoring of all events when unassigning one event.This is why I proposed in [3] that the name of the mode reflects how user can interact with the system. Instead of one "
mbm_cntr_assign" mode there can be "mbm_cntr_event_assign" that is used for ABMC and "mbm_cntr_group_assign" that is used for soft-ABMC. The mode should make it clear what the system is capable of wrt counter assignments.Considering this the interface should be clear:
num_mbm_cntrs: reflects the number of counters in each domain that can be assigned. In "mbm_cntr_event_assign" this will be the number of counters that can be assigned to each event within a monitoring group, in "mbm_cntr_group_assign" this will be the number of counters that can be assigned to entire monitoring groups impacting all MBM events.
mbm_L3_assignments: manages the counter assignment in each group. When user knows the mode is "mbm_cntr_event_assign"/"mbm_cntr_group_assign" then it should be clear to user space how the interface behaves wrt assignment, no surprises of multiple events impacted when assigning/unassigning single event.For soft-ABMC I thus find it most intuitive for
num_mbm_cntrsto be the exact number of "active" RMIDs that the system can support *and* changing the name of the modes to help user interpretnum_mbm_cntrs.Quoted 5 lines
There's also the mongroup-RMID overcommit use case I described above[1]. On Intel we can safely assume that there are counters to back all RMIDs, so
num_mbm_cntrswould be calculated directly fromnum_rmids.This is about the:
There's now more interest in Google for allowing explicit control of where RMIDs are assigned on Intel platforms. Even though the number of RMIDs implemented by hardware tends to be roughly the number of containers they want to support, they often still need to create containers when all RMIDs have already been allocated, which is not currently allowed. Once the container has been created and starts running, it's no longer possible to move its threads into a monitoring group whenever RMIDs should become available again, so it's important for resctrl to maintain an accurate task list for a container even when RMIDs are not available.
Quoted 8 lines
I see a monitor group as a collection of tasks that need to be monitored together. The "task list" is the group of tasks that share a monitoring ID that is required to be a valid ID since when any of the tasks are scheduled that ID is written to the hardware. I intentionally tried to not use RMID since I believe this is required for all archs.
I thus do not understand how a task can start running when it does not have a valid monitoring ID. The idea of "deferred assignment" is not clear to me, there can never be "unmonitored tasks", no? I think I am missing something here.
In the AMD/RMID implemenentation this might be achieved with something extra in the task structure to denote whether a task is in a monitored group or not. E.g. We add "task->rmid_valid" as well as "task->rmid". Tasks in an unmonitored group retain their "task->rmid" (that's what identifies them as a member of a group) but have task->rmid_valid set to false. Context switch code would be updated to load "0" into the IA32_PQR_ASSOC.RMID field for tasks without a valid RMID. So they would still be monitored, but activity would be bundled with all tasks in the default resctrl group.
Presumably something analogous could be done for ARM/MPAM.
Quoted 19 lines
Quoted 19 lines
I realized this use case is more difficult to implement on MPAM, because a PARTID is effectively a CLOSID+RMID, so deferring assigning a unique PARTID to a group also results in it being in a different allocation group. It will work if the unmonitored groups could find a way to share PARTIDs, but this has consequences on allocation - but hopefully no worse than sharing CLOSIDs on x86.
There's a lot of interest in monitoring ID overcommit in Google, so I think it's worth it for me to investigate the additional structural changes needed in resctrl (i.e., breaking the FS-level association between mongroups and HW monitoring IDs). Such a framework could be a better fit for soft-ABMC. For example, if overcommit is allowed, we would just report the number of simultaneous RMIDs we were able to probe as
num_rmids. I would want the same shared assignment scheduler to be able to work with RMIDs and counters, though.Thanks,
-Peter
Quoted 6 lines
Quoted 1 line
[1] https://lore.kernel.org/lkml/CALPaoChSzzU5mzMZsdT6CeyEn0WD1qdT9fKCoNW_ty4tojtrkw@mail.gmail.com/
Reinette
[2] https://lore.kernel.org/lkml/[email protected]/
[3] https://lore.kernel.org/lkml/[email protected]/
-Tony