x86/resctrl : Support AMD Assignable Bandwidth Monitoring Counters (ABMC)
Subsystems (3)
Babu Moger
reviewerv8 cover x86/resctrl : Support AMD Assignable Bandwidth Monitoring Counters (ABMC)
This series adds the support for Assignable Bandwidth Monitoring Counters (ABMC). It is also called QoS RMID Pinning feature
Series is written such that it is easier to support other assignable features supported from different vendors.
The feature details are documented in the APM listed below [1].
[1] AMD64 Architecture Programmer's Manual Volume 2: System Programming Publication # 24593 Revision 3.41 section 19.3.3.3 Assignable Bandwidth Monitoring (ABMC). The documentation is available at
The patches are based on top of commit5b0c5f05fb2fe (tip/master) Merge branch into tip/master: 'x86/splitlock'
# Introduction
Users can create as many monitor groups as RMIDs supported by the hardware. However, bandwidth monitoring feature on AMD system only guarantees that RMIDs currently assigned to a processor will be tracked by hardware. The counters of any other RMIDs which are no longer being tracked will be reset to zero. The MBM event counters return "Unavailable" for the RMIDs that are not tracked by hardware. So, there can be only limited number of groups that can give guaranteed monitoring numbers. With ever changing configurations there is no way to definitely know which of these groups are being tracked for certain point of time. Users do not have the option to monitor a group or set of groups for certain period of time without worrying about RMID being reset in between.
The ABMC feature provides an option to the user to assign a hardware counter to an RMID, event pair and monitor the bandwidth as long as it is assigned. The assigned RMID will be tracked by the hardware until the user unassigns it manually. There is no need to worry about counters being reset during this period. Additionally, the user can specify a bitmask identifying the specific bandwidth types from the given source to track with the counter.
Without ABMC enabled, monitoring will work in current 'default' mode without assignment option.
# Linux Implementation
Create a generic interface aimed to support user space assignment of scarce counters used for monitoring. First usage of interface is by ABMC with option to expand usage to "soft-ABMC" and MPAM counters in future.
Feature adds following interface files:
/sys/fs/resctrl/info/L3_MON/mbm_assign_mode: Reports the list of assignable monitoring features supported. The enclosed brackets indicate which feature is enabled.
/sys/fs/resctrl/info/L3_MON/num_mbm_cntrs: Reports the number of monitoring counters available for assignment.
/sys/fs/resctrl/info/L3_MON/mbm_assign_control: Reports the resctrl group and monitor status of each group. Assignment state can be updated by writing to the interface.
# Examples
a. Check if ABMC support is available
#mount -t resctrl resctrl /sys/fs/resctrl/
#cat /sys/fs/resctrl/info/L3_MON/mbm_assign_mode [mbm_cntr_assign]
default
ABMC feature is detected and it is enabled.
b. Check how many ABMC counters are available.
#cat /sys/fs/resctrl/info/L3_MON/num_mbm_cntrs 32
c. Create few resctrl groups.
# mkdir /sys/fs/resctrl/mon_groups/child_default_mon_grp # mkdir /sys/fs/resctrl/non_default_ctrl_mon_grp # mkdir /sys/fs/resctrl/non_default_ctrl_mon_grp/mon_groups/child_non_default_mon_grp
d. This series adds a new interface file /sys/fs/resctrl/info/L3_MON/mbm_assign_control
to list and modify any group's monitoring states. File provides single place to list monitoring states of all the resctrl groups. It makes it easier for user space to learn about the used counters without needing to traverse all the groups thus reducing the number of file system calls.
The list follows the following format:
"<CTRL_MON group>/<MON group>/<domain_id>=<flags>"
Format for specific type of groups:
* Default CTRL_MON group:
"//<domain_id>=<flags>"
* Non-default CTRL_MON group:
"<CTRL_MON group>//<domain_id>=<flags>"
* Child MON group of default CTRL_MON group:
"/<MON group>/<domain_id>=<flags>"
* Child MON group of non-default CTRL_MON group:
"<CTRL_MON group>/<MON group>/<domain_id>=<flags>"
Flags can be one of the following:
t MBM total event is enabled.
l MBM local event is enabled.
tl Both total and local MBM events are enabled. _ None of the MBM events are enabled
Examples:
# cat /sys/fs/resctrl/info/L3_MON/mbm_assign_control non_default_ctrl_mon_grp//0=tl;1=tl; non_default_ctrl_mon_grp/child_non_default_mon_grp/0=tl;1=tl; //0=tl;1=tl;
/child_default_mon_grp/0=tl;1=tl;
There are four groups and all the groups have local and total event enabled on domain 0 and 1.
e. Update the group assignment states using the interface file /sys/fs/resctrl/info/L3_MON/mbm_assign_control.
The write format is similar to the above list format with addition of opcode for the assignment operation.
“<CTRL_MON group>/<MON group>/<domain_id><opcode><flags>”
* Default CTRL_MON group:
"//<domain_id><opcode><flags>"
* Non-default CTRL_MON group:
"<CTRL_MON group>//<domain_id><opcode><flags>"
* Child MON group of default CTRL_MON group:
"/<MON group>/<domain_id><opcode><flags>"
* Child MON group of non-default CTRL_MON group:
"<CTRL_MON group>/<MON group>/<domain_id><opcode><flags>"
Opcode can be one of the following:
= Update the assignment to match the flags.
+ Assign a new MBM event without impacting existing assignments.
- Unassign a MBM event from currently assigned events.
Flags can be one of the following:
t MBM total event.
l MBM local event.
tl Both total and local MBM events.
_ None of the MBM events. Only works with '=' opcode. This flag cannot be combined with other flags.
Initial group status:
# cat /sys/fs/resctrl/info/L3_MON/mbm_assign_control non_default_ctrl_mon_grp//0=tl;1=tl; non_default_ctrl_mon_grp/child_non_default_mon_grp/0=tl;1=tl; //0=tl;1=tl;
/child_default_mon_grp/0=tl;1=tl;
To update the default group to enable only total event on domain 0:
# echo "//0=t" > /sys/fs/resctrl/info/L3_MON/mbm_assign_control
Assignment status after the update:
# cat /sys/fs/resctrl/info/L3_MON/mbm_assign_control non_default_ctrl_mon_grp//0=tl;1=tl; non_default_ctrl_mon_grp/child_non_default_mon_grp/0=tl;1=tl; //0=t;1=tl;
/child_default_mon_grp/0=tl;1=tl;
To update the MON group child_default_mon_grp to remove total event on domain 1:
# echo "/child_default_mon_grp/1-t" > /sys/fs/resctrl/info/L3_MON/mbm_assign_control
Assignment status after the update:
$ cat /sys/fs/resctrl/info/L3_MON/mbm_assign_control non_default_ctrl_mon_grp//0=tl;1=tl; non_default_ctrl_mon_grp/child_non_default_mon_grp/0=tl;1=tl; //0=t;1=tl;
/child_default_mon_grp/0=tl;1=l;
To update the MON group non_default_ctrl_mon_grp/child_non_default_mon_grp to remove both local and total events on domain 1:
# echo "non_default_ctrl_mon_grp/child_non_default_mon_grp/1=_" >
/sys/fs/resctrl/info/L3_MON/mbm_assign_control
Assignment status after the update:
non_default_ctrl_mon_grp//0=tl;1=tl; non_default_ctrl_mon_grp/child_non_default_mon_grp/0=tl;1=_; //0=t;1=tl;
/child_default_mon_grp/0=tl;1=l;
To update the default group to add a local event domain 0. # echo "//0+l" > /sys/fs/resctrl/info/L3_MON/mbm_assign_control
Assignment status after the update:
# cat /sys/fs/resctrl/info/L3_MON/mbm_assign_control non_default_ctrl_mon_grp//0=tl;1=tl; non_default_ctrl_mon_grp/child_non_default_mon_grp/0=tl;1=_; //0=tl;1=tl;
/child_default_mon_grp/0=tl;1=l;
To update the non default CTRL_MON group non_default_ctrl_mon_grp to unassign all the MBM events on all the domains.
# echo "non_default_ctrl_mon_grp//*=_" > /sys/fs/resctrl/info/L3_MON/mbm_assign_control
Assignment status after the update:
# cat /sys/fs/resctrl/info/L3_MON/mbm_assign_control non_default_ctrl_mon_grp//0=_;1=_; non_default_ctrl_mon_grp/child_non_default_mon_grp/0=tl;1=_; //0=tl;1=tl;
/child_default_mon_grp/0=tl;1=l;
f. Read the event mbm_total_bytes and mbm_local_bytes of the default group.
There is no change in reading the events with ABMC. If the event is unassigned when reading, then the read will come back as "Unassigned".
# cat /sys/fs/resctrl/mon_data/mon_L3_00/mbm_total_bytes 779247936 # cat /sys/fs/resctrl/mon_data/mon_L3_00/mbm_local_bytes 765207488
g. Check the bandwidth configuration for the group. Note that bandwidth
configuration has a domain scope. Total event defaults to 0x7F (to count all the events) and local event defaults to 0x15 (to count all the local numa events). The event bitmap decoding is available at
https://www.kernel.org/doc/Documentation/x86/resctrl.rst
in section "mbm_total_bytes_config", "mbm_local_bytes_config":
#cat /sys/fs/resctrl/info/L3_MON/mbm_total_bytes_config 0=0x7f;1=0x7f #cat /sys/fs/resctrl/info/L3_MON/mbm_local_bytes_config 0=0x15;1=0x15
h. Change the bandwidth source for domain 0 for the total event to count only reads.
Note that this change effects total events on the domain 0.
#echo 0=0x33 > /sys/fs/resctrl/info/L3_MON/mbm_total_bytes_config #cat /sys/fs/resctrl/info/L3_MON/mbm_total_bytes_config 0=0x33;1=0x7F
i. Now read the total event again. The first read will come back with "Unavailable"
status. The subsequent read of mbm_total_bytes will display only the read events.
#cat /sys/fs/resctrl/mon_data/mon_L3_00/mbm_total_bytes Unavailable #cat /sys/fs/resctrl/mon_data/mon_L3_00/mbm_total_bytes 314101
j. Users will have the option to go back to 'default' mbm_assign_mode if required.
This can be done using the following command. Note that switching the mbm_assign_mode will reset all the MBM counters of all resctrl groups.
# echo "default" > /sys/fs/resctrl/info/L3_MON/mbm_assign_mode # cat /sys/fs/resctrl/info/L3_MON/mbm_assign_mode mbm_cntr_assign [default]
k. Unmount the resctrl
#umount /sys/fs/resctrl/
---
v8:
Patches are getting into final stages.
Couple of changes Patch 8, Patch 19 and Patch 23.
Most of the other changes are related to rename and text message updates.
Details are in each patch. Here is the summary.
Added __init attribute to dom_data_init() in patch 8/25. Moved the mbm_cntrs_init() and mbm_cntrs_exit() functionality inside dom_data_init() and dom_data_exit() respectively.
Renamed resctrl_mbm_evt_config_init() to arch_mbm_evt_config_init() Renamed resctrl_arch_event_config_get() to resctrl_arch_mon_event_config_get().
resctrl_arch_event_config_set() to resctrl_arch_mon_event_config_set().
Rename resctrl_arch_assign_cntr to resctrl_arch_config_cntr. Renamed rdtgroup_assign_cntr() to rdtgroup_assign_cntr_event(). Added the code to return the error if rdtgroup_assign_cntr_event fails. Moved definition of MBM_EVENT_ARRAY_INDEX to resctrl/internal.h. Renamed rdtgroup_mbm_cntr_is_assigned to mbm_cntr_assigned_to_domain Added return error handling in resctrl_arch_config_cntr(). Renamed rdtgroup_assign_grp to rdtgroup_assign_cntrs.
Renamed rdtgroup_unassign_grp to rdtgroup_unassign_cntrs. Fixed the problem with unassigning the child MON groups of CTRL_MON group. Reset the internal counters after mbm_cntr_assign mode is changed. Renamed rdtgroup_mbm_cntr_reset() to mbm_cntr_reset()
Renamed resctrl_arch_mbm_cntr_assign_configure to
resctrl_arch_mbm_cntr_assign_set_one.
Used the same IPI as event update to modify the assignment. Could not do the way we discussed in the thread.
https://lore.kernel.org/lkml/[email protected]/
Needed to figure out event type to update the configuration.
Moved unassign first and assign during the assign modification. Assign none "_" takes priority. Cannot be mixed with other flags. Updated the documentation and .rst file format. htmldoc looks ok.
v7:
Major changes are related to FS and arch codes separation. Changed few interface names based on feedback.
Here are the summary and each patch contains changes specific the patch.
Removed WARN_ON for num_mbm_cntrs. Decided to dynamically allocate the bitmap. WARN_ON is not required anymore.
Renamed the function resctrl_arch_get_abmc_enabled() to resctrl_arch_mbm_cntr_assign_enabled().
Merged resctrl_arch_mbm_cntr_assign_disable, resctrl_arch_mbm_cntr_assign_disable and renamed to resctrl_arch_mbm_cntr_assign_set(). Passed the struct rdt_resource to these functions.
Removed resctrl_arch_reset_rmid_all() from arch code. This will be done from FS the caller.
Updated the descriptions/commit log in resctrl.rst to generic text. Removed ABMC references. Renamed mbm_mode to mbm_assign_mode.
Renamed mbm_control to mbm_assign_control.
Introduced mutex lock in rdtgroup_mbm_mode_show().
The 'legacy' mode is called 'default' mode.
Removed the static allocation and now allocating bitmap mbm_cntr_free_map dynamically.
Merged rdtgroup_assign_cntr(), rdtgroup_alloc_cntr() into one. Merged rdtgroup_unassign_cntr(), rdtgroup_free_cntr() into one.
Added struct rdt_resource to the interface functions resctrl_arch_assign_cntr () and resctrl_arch_unassign_cntr().
Rename rdtgroup_abmc_cfg() to resctrl_abmc_config_one_amd().
Added a new patch to fix counter assignment on event config changes.
Removed the references of ABMC from user interfaces.
Simplified the parsing (strsep(&token, "//") in rdtgroup_mbm_assign_control_write(). Added mutex lock in rdtgroup_mbm_assign_control_write() while processing.
Thomas Gleixner asked us to update https://gitlab.com/x86-cpuid.org/x86-cpuid-db.
It needs internal approval. We are working on it.
v6:
We still need to finalize few interface details on mbm_assign_mode and mbm_assign_control in case of ABMC and Soft-ABMC. We can continue the discussion with this series.
Added support for domain-id '*' to update all the domains at once. Fixed assign interface to allocate the counter if counter is not assigned.
Fixed unassign interface to free the counter if the counter is not assigned in any of the domains.
Renamed abmc_capable to mbm_cntr_assignable.
Renamed abmc_enabled to mbm_cntr_assign_enabled.
Used msr_set_bit and msr_clear_bit for msr updates.
Renamed resctrl_arch_abmc_enable() to resctrl_arch_mbm_cntr_assign_enable(). Renamed resctrl_arch_abmc_disable() to resctrl_arch_mbm_cntr_assign_disable().
Changed the display name from num_cntrs to num_mbm_cntrs.
Removed the variable mbm_cntrs_free_map_len. This is not required. Removed the call mbm_cntrs_init() in arch code. This needs to be done at higher level. Used DECLARE_BITMAP to initialize mbm_cntrs_free_map.
Removed unused config value definitions.
Introduced mbm_cntr_map to track counters at domain level. With this we dont need to send MSR read to read the counter configuration.
Separated all the counter id management to upper level in FS code.
Added checks to detect "Unassigned" before reading the RMID.
More details in each patch.
v5:
Rebase changes (because of SNC support)
Interface changes.
/sys/fs/resctrl/mbm_assign to /sys/fs/resctrl/mbm_assign_mode. /sys/fs/resctrl/mbm_assign_control to /sys/fs/resctrl/mbm_assign_control.
Added few arch specific routines.
resctrl_arch_get_abmc_enabled.
resctrl_arch_abmc_enable.
resctrl_arch_abmc_disable.
Few renames
num_cntrs_free_map -> mbm_cntrs_free_map num_cntrs_init -> mbm_cntrs_init arch_domain_mbm_evt_config -> resctrl_arch_mbm_evt_config
Introduced resctrl_arch_event_config_get and
resctrl_arch_event_config_set() to update event configuration.
Removed mon_state field mongroup. Added MON_CNTR_UNSET to initialize counters.
Renamed ctr_id to cntr_id for the hardware counter.
Report "Unassigned" in case the user attempts to read the events without assigning the counter.
ABMC is enabled during the boot up. Can be enabled or disabled later.
Fixed opcode and flags combination.
'=_" is valid.
"-_" amd "+_" is not valid.Added all the comments as far as I know. If I missed something, it is not intentional.
v4:
Main change is domain specific event assignment.
Kept the ABMC feature as a default.
Dynamcic switching between ABMC and mbm_legacy is still allowed. We are still not clear about mount option.
Moved the monitoring related data in resctrl_mon structure from rdt_resource. Fixed the display of legacy and ABMC mode.
Used bimap APIs when possible.
Removed event configuration read from MSRs. We can use the internal saved data.(patch 12)
Added more comments about L3_QOS_ABMC_CFG MSR.
Added IPIs to read the assignment status for each domain (patch 18 and 19) More details in each patch.
v3:
This series adds the support for global assignment mode discussed in the thread. https://lore.kernel.org/lkml/[email protected]/
Removed the individual assignment mode and included the global assignment interface. Added following interface files.
a. /sys/fs/resctrl/info/L3_MON/mbm_assign
Used for displaying the current assignment mode and switch between ABMC and legacy mode.
b. /sys/fs/resctrl/info/L3_MON/mbm_assign_control
Used for lising the groups assignment mode and modify the assignment states.
c. Most of the changes are related to the new interface.
d. Addressed the comments from Reinette, James and Peter.
e. Hope I have addressed most of the major feedbacks discussed. If I missed
something then it is not intentional. Please feel free to comment.
f. Sending this as an RFC as per Reinette's comment. So, this is still open
for discussion.
v2:
a. Major change is the way ABMC is enabled. Earlier, user needed to remount
with -o abmc to enable ABMC feature. Removed that option now.
Now users can enable ABMC by "$echo 1 to /sys/fs/resctrl/info/L3_MON/mbm_assign_enable".
b. Added new word 21 to x86/cpufeatures.h.
c. Display unsupported if user attempts to read the events when ABMC is enabled
and event is not assigned.
d. Display monitor_state as "Unsupported" when ABMC is disabled.
e. Text updates and rebase to latest tip tree (as of Jan 18).
f. This series is still work in progress. I am yet to hear from ARM developers.
v7:
https://lore.kernel.org/lkml/[email protected]/
v6:
https://lore.kernel.org/lkml/[email protected]/
v5:
https://lore.kernel.org/lkml/[email protected]/
v4:
https://lore.kernel.org/lkml/[email protected]/
v3:
https://lore.kernel.org/lkml/[email protected]/
v2:
https://lore.kernel.org/lkml/[email protected]/
v1 :
https://lore.kernel.org/lkml/[email protected]/
Babu Moger (24):
x86/cpufeatures: Add support for Assignable Bandwidth Monitoring
Counters (ABMC)
x86/resctrl: Add ABMC feature in the command line options x86/resctrl: Consolidate monitoring related data from rdt_resource x86/resctrl: Detect Assignable Bandwidth Monitoring feature details x86/resctrl: Introduce resctrl_file_fflags_init() to initialize fflags x86/resctrl: Add support to enable/disable AMD ABMC feature x86/resctrl: Introduce the interface to display monitor mode x86/resctrl: Introduce interface to display number of monitoring
counters
x86/resctrl: Introduce bitmap mbm_cntr_free_map to track assignable
counters
x86/resctrl: Introduce mbm_total_cfg and mbm_local_cfg in struct
rdt_hw_mon_domain
x86/resctrl: Remove MSR reading of event configuration value x86/resctrl: Introduce mbm_cntr_map to track counters at domain x86/resctrl: Add data structures and definitions for ABMC assignment x86/resctrl: Introduce cntr_id in mongroup for assignments x86/resctrl: Implement resctrl_arch_assign_cntr to assign a counter
with ABMC
x86/resctrl: Add the interface to assign/update counter assignment x86/resctrl: Add the interface to unassign a MBM counter x86/resctrl: Auto Assign/unassign counters when mbm_cntr_assign is
enabled
x86/resctrl: Report "Unassigned" for MBM events in mbm_cntr_assign
mode
x86/resctrl: Introduce the interface to switch between monitor modes x86/resctrl: Configure mbm_cntr_assign mode if supported x86/resctrl: Update assignments on event configuration changes x86/resctrl: Introduce interface to list assignment states of all the
groups
x86/resctrl: Introduce interface to modify assignment states of the
groups
.../admin-guide/kernel-parameters.txt | 2 +- Documentation/arch/x86/resctrl.rst | 198 ++++ arch/x86/include/asm/cpufeatures.h | 1 + arch/x86/include/asm/msr-index.h | 2 + arch/x86/kernel/cpu/cpuid-deps.c | 3 + arch/x86/kernel/cpu/resctrl/core.c | 19 +- arch/x86/kernel/cpu/resctrl/ctrlmondata.c | 13 +- arch/x86/kernel/cpu/resctrl/internal.h | 77 +- arch/x86/kernel/cpu/resctrl/monitor.c | 90 +- arch/x86/kernel/cpu/resctrl/rdtgroup.c | 875 ++++++++++++++++-- arch/x86/kernel/cpu/scattered.c | 1 + include/linux/resctrl.h | 31 +- 12 files changed, 1227 insertions(+), 85 deletions(-)
Signature · 57 lines
2.34.1
Babu Moger (25):
x86/cpufeatures: Add support for Assignable Bandwidth Monitoring
Counters (ABMC)
x86/resctrl: Add ABMC feature in the command line options
x86/resctrl: Consolidate monitoring related data from rdt_resource
x86/resctrl: Detect Assignable Bandwidth Monitoring feature details
x86/resctrl: Introduce resctrl_file_fflags_init() to initialize fflags
x86/resctrl: Add support to enable/disable AMD ABMC feature
x86/resctrl: Introduce the interface to display monitor mode
x86/resctrl: Introduce interface to display number of monitoring
counters
x86/resctrl: Add __init attribute to dom_data_init()
x86/resctrl: Introduce bitmap mbm_cntr_free_map to track assignable
counters
x86/resctrl: Introduce mbm_total_cfg and mbm_local_cfg in struct
rdt_hw_mon_domain
x86/resctrl: Remove MSR reading of event configuration value
x86/resctrl: Introduce mbm_cntr_map to track assignable counters at
domain
x86/resctrl: Add data structures and definitions for ABMC assignment
x86/resctrl: Introduce cntr_id in mongroup for assignments
x86/resctrl: Implement resctrl_arch_config_cntr() to assign a counter
with ABMC
x86/resctrl: Add the interface to assign/update counter assignment
x86/resctrl: Add the interface to unassign a MBM counter
x86/resctrl: Auto assign/unassign counters when mbm_cntr_assign is
enabled
x86/resctrl: Report "Unassigned" for MBM events in mbm_cntr_assign
mode
x86/resctrl: Introduce the interface to switch between monitor modes
x86/resctrl: Configure mbm_cntr_assign mode if supported
x86/resctrl: Update assignments on event configuration changes
x86/resctrl: Introduce interface to list assignment states of all the
groups
x86/resctrl: Introduce interface to modify assignment states of the
groups
.../admin-guide/kernel-parameters.txt | 2 +-
Documentation/arch/x86/resctrl.rst | 221 +++++
arch/x86/include/asm/cpufeatures.h | 1 +
arch/x86/include/asm/msr-index.h | 2 +
arch/x86/kernel/cpu/cpuid-deps.c | 3 +
arch/x86/kernel/cpu/resctrl/core.c | 19 +-
arch/x86/kernel/cpu/resctrl/ctrlmondata.c | 13 +-
arch/x86/kernel/cpu/resctrl/internal.h | 87 +-
arch/x86/kernel/cpu/resctrl/monitor.c | 110 ++-
arch/x86/kernel/cpu/resctrl/rdtgroup.c | 899 ++++++++++++++++--
arch/x86/kernel/cpu/scattered.c | 1 +
include/linux/resctrl.h | 31 +-
12 files changed, 1293 insertions(+), 96 deletions(-)
--
2.34.1Babu Moger
reviewerPATCH v8 1/25 x86/cpufeatures: Add support for Assignable Bandwidth Monitoring Counters (ABMC)
Users can create as many monitor groups as RMIDs supported by the hardware. However, bandwidth monitoring feature on AMD system only guarantees that RMIDs currently assigned to a processor will be tracked by hardware. The counters of any other RMIDs which are no longer being tracked will be reset to zero. The MBM event counters return "Unavailable" for the RMIDs that are not tracked by hardware. So, there can be only limited number of groups that can give guaranteed monitoring numbers. With ever changing configurations there is no way to definitely know which of these groups are being tracked for certain point of time. Users do not have the option to monitor a group or set of groups for certain period of time without worrying about RMID being reset in between.
The ABMC feature provides an option to the user to assign a hardware counter to an RMID, event pair and monitor the bandwidth as long as it is assigned. The assigned RMID will be tracked by the hardware until the user unassigns it manually. There is no need to worry about counters being reset during this period. Additionally, the user can specify a bitmask identifying the specific bandwidth types from the given source to track with the counter.
Without ABMC enabled, monitoring will work in current mode without assignment option.
Linux resctrl subsystem provides the interface to count maximum of two memory bandwidth events per group, from a combination of available total and local events. Keeping the current interface, users can enable a maximum of 2 ABMC counters per group. User will also have the option to enable only one counter to the group. If the system runs out of assignable ABMC counters, kernel will display an error. Users need to disable an already enabled counter to make space for new assignments.
The feature can be detected via CPUID_Fn80000020_EBX_x00 bit 5. Bits Description
5 ABMC (Assignable Bandwidth Monitoring Counters)
The feature details are documented in APM listed below [1].
[1] AMD64 Architecture Programmer's Manual Volume 2: System Programming Publication # 24593 Revision 3.41 section 19.3.3.3 Assignable Bandwidth Monitoring (ABMC).
---
Note: Checkpatch checks/warnings are ignored to maintain coding style.
v8: No changes.
v7: Removed "" from feature flags. Not required anymore.
https://lore.kernel.org/lkml/20240817145058.GCZsC40neU4wkPXeVR@fat_crate.local/
v6: Added Reinette's Reviewed-by. Moved the Checkpatch note below ---.
v5: Minor rebase change and subject line update.
v4: Changes because of rebase. Feature word 21 has few more additions now.
Changed the text to "tracked by hardware" instead of active.
v3: Change because of rebase. Actual patch did not change.
v2: Added dependency on X86_FEATURE_BMEC.
--- arch/x86/include/asm/cpufeatures.h | 1 + arch/x86/kernel/cpu/cpuid-deps.c | 3 +++ arch/x86/kernel/cpu/scattered.c | 1 + 3 files changed, 5 insertions(+)
arch/x86/include/asm/cpufeatures.h +1 -0
@@ -473,6 +473,7 @@#define X86_FEATURE_CLEAR_BHB_HW (21*32+ 3)/* BHI_DIS_S HW control enabled */ #define X86_FEATURE_CLEAR_BHB_LOOP_ON_VMEXIT (21*32+ 4)/* Clear branch history at vmexit using SW loop */ #define X86_FEATURE_FAST_CPPC (21*32 + 5)/* AMD Fast CPPC */ +#define X86_FEATURE_ABMC (21*32 + 6)/* Assignable Bandwidth Monitoring Counters */ /* * BUG word(s)
arch/x86/kernel/cpu/cpuid-deps.c +3 -0
@@ -70,6 +70,9 @@ static const struct cpuid_dep cpuid_deps[] = { { X86_FEATURE_CQM_MBM_LOCAL, X86_FEATURE_CQM_LLC }, { X86_FEATURE_BMEC, X86_FEATURE_CQM_MBM_TOTAL }, { X86_FEATURE_BMEC, X86_FEATURE_CQM_MBM_LOCAL }, + { X86_FEATURE_ABMC, X86_FEATURE_CQM_MBM_TOTAL }, + { X86_FEATURE_ABMC, X86_FEATURE_CQM_MBM_LOCAL }, + { X86_FEATURE_ABMC, X86_FEATURE_BMEC }, { X86_FEATURE_AVX512_BF16, X86_FEATURE_AVX512VL }, { X86_FEATURE_AVX512_FP16, X86_FEATURE_AVX512BW }, { X86_FEATURE_ENQCMD, X86_FEATURE_XSAVES },
arch/x86/kernel/cpu/scattered.c +1 -0
@@ -49,6 +49,7 @@ static const struct cpuid_bit cpuid_bits[] = { { X86_FEATURE_MBA, CPUID_EBX, 6, 0x80000008, 0 }, { X86_FEATURE_SMBA, CPUID_EBX, 2, 0x80000020, 0 }, { X86_FEATURE_BMEC, CPUID_EBX, 3, 0x80000020, 0 }, + { X86_FEATURE_ABMC, CPUID_EBX, 5, 0x80000020, 0 }, { X86_FEATURE_PERFMON_V2, CPUID_EAX, 0, 0x80000022, 0 }, { X86_FEATURE_AMD_LBR_V2, CPUID_EAX, 1, 0x80000022, 0 }, { X86_FEATURE_AMD_LBR_PMC_FREEZE, CPUID_EAX, 2, 0x80000022, 0 },
Signature · 1 line
2.34.1
Babu Moger
reviewerPATCH v8 2/25 x86/resctrl: Add ABMC feature in the command line options
In reply to Babu MogerAdd the command line option to enable or disable exposing the ABMC (Assignable Bandwidth Monitoring Counters) hardware feature to resctrl.
---
v8: Commit message update.
v7: No changes
v6: No changes
v5: No changes
v4: No changes
v3: No changes
v2: No changes
--- Documentation/admin-guide/kernel-parameters.txt | 2 +- Documentation/arch/x86/resctrl.rst | 1 + arch/x86/kernel/cpu/resctrl/core.c | 2 ++ 3 files changed, 4 insertions(+), 1 deletion(-)
Documentation/admin-guide/kernel-parameters.txt +1 -1
@@ -5677,7 +5677,7 @@ rdt= [HW,X86,RDT] Turn on/off individual RDT features. List is: cmt, mbmtotal, mbmlocal, l3cat, l3cdp, l2cat, l2cdp, - mba, smba, bmec. + mba, smba, bmec, abmc. E.g. to turn on cmt and turn off mba use: rdt=cmt,!mba
Documentation/arch/x86/resctrl.rst +1 -0
@@ -26,6 +26,7 @@ MBM (Memory Bandwidth Monitoring) "cqm_mbm_total", "cqm_mbm_local" MBA (Memory Bandwidth Allocation) "mba" SMBA (Slow Memory Bandwidth Allocation) "" BMEC (Bandwidth Monitoring Event Configuration) "" +ABMC (Assignable Bandwidth Monitoring Counters) "" =============================================== ================================ Historically, new features were made visible by default in /proc/cpuinfo. This
arch/x86/kernel/cpu/resctrl/core.c +2 -0
@@ -809,6 +809,7 @@ enum {RDT_FLAG_MBA ,RDT_FLAG_SMBA ,RDT_FLAG_BMEC , + RDT_FLAG_ABMC, };#define RDT_OPT (idx, n, f) \
@@ -834,6 +835,7 @@ static struct rdt_options rdt_options[] __initdata = {RDT_OPT (RDT_FLAG_MBA ,"mba" ,X86_FEATURE_MBA ),RDT_OPT (RDT_FLAG_SMBA ,"smba" ,X86_FEATURE_SMBA ),RDT_OPT (RDT_FLAG_BMEC ,"bmec" ,X86_FEATURE_BMEC ), + RDT_OPT(RDT_FLAG_ABMC, "abmc", X86_FEATURE_ABMC), };#define NUM_RDT_OPTIONS ARRAY_SIZE(rdt_options)
Signature · 1 line
2.34.1
Re: [PATCH v8 02/25] x86/resctrl: Add ABMC feature in the command line options
Hi Babu,
Quoted 5 lines · Babu Moger
Add the command line option to enable or disable exposing the ABMC (Assignable Bandwidth Monitoring Counters) hardware feature to resctrl.
Signed-off-by Babu Moger <[email protected]>---
Reinette
Babu Moger
reviewerPATCH v8 3/25 x86/resctrl: Consolidate monitoring related data from rdt_resource
In reply to Babu MogerThe cache allocation and memory bandwidth allocation feature properties are consolidated into struct resctrl_cache and struct resctrl_membw respectively.
In preparation for more monitoring properties that will clobber the existing resource struct more, re-organize the monitoring specific properties to also be in a separate structure.
---
v8: Added Reviewed-by from Reinette. No other changes.
v7: Added kernel doc for data structure. Minor text update.
v6: Update commit message and update kernel doc for rdt_resource.
v5: Commit message update.
Also changes related to data structure updates does to SNC support.
v4: New patch.
--- arch/x86/kernel/cpu/resctrl/core.c | 4 ++-- arch/x86/kernel/cpu/resctrl/monitor.c | 18 +++++++++--------- arch/x86/kernel/cpu/resctrl/rdtgroup.c | 8 ++++---- include/linux/resctrl.h | 16 ++++++++++++---- 4 files changed, 27 insertions(+), 19 deletions(-)
arch/x86/kernel/cpu/resctrl/core.c +2 -2
@@ -124,7 +124,7 @@ u32 resctrl_arch_system_num_rmid_idx(void)struct rdt_resource *r = &rdt_resources_all[RDT_RESOURCE_L3 ].r_resctrl;/* RMID are independent numbers for x86. num_rmid_idx == num_rmid */ -return r->num_rmid; +return r->mon.num_rmid; }/*
@@ -625,7 +625,7 @@ static void domain_add_cpu_mon(int cpu, struct rdt_resource *r)arch_mon_domain_online (r, d); -if (arch_domain_mbm_alloc (r->num_rmid, hw_dom)) { +if (arch_domain_mbm_alloc (r->mon.num_rmid, hw_dom)) {mon_domain_free (hw_dom);return ; }
arch/x86/kernel/cpu/resctrl/monitor.c +9 -9
@@ -222,7 +222,7 @@ static int logical_rmid_to_physical_rmid(int cpu, int lrmid)if (snc_nodes_per_l3_cache ==1 )return lrmid; -return lrmid + (cpu_to_node (cpu) % snc_nodes_per_l3_cache) * r->num_rmid; +return lrmid + (cpu_to_node (cpu) % snc_nodes_per_l3_cache) * r->mon.num_rmid; }static int __rmid_read_phys (u32 prmid,enum resctrl_event_id eventid,u64 *val)
@@ -297,11 +297,11 @@ void resctrl_arch_reset_rmid_all(struct rdt_resource *r, struct rdt_mon_domain *if (is_mbm_total_enabled ())memset (hw_dom->arch_mbm_total,0 , -sizeof (*hw_dom->arch_mbm_total) * r->num_rmid); +sizeof (*hw_dom->arch_mbm_total) * r->mon.num_rmid);if (is_mbm_local_enabled ())memset (hw_dom->arch_mbm_local,0 , -sizeof (*hw_dom->arch_mbm_local) * r->num_rmid); +sizeof (*hw_dom->arch_mbm_local) * r->mon.num_rmid); }static u64 mbm_overflow_count (u64 prev_msr,u64 cur_msr,unsigned int width)
@@ -1083,14 +1083,14 @@ static struct mon_evt mbm_local_event = {*/ static void l3_mon_evt_init (struct rdt_resource *r) { -INIT_LIST_HEAD (&r->evt_list); +INIT_LIST_HEAD (&r->mon.evt_list);if (is_llc_occupancy_enabled ()) -list_add_tail (&llc_occupancy_event.list, &r->evt_list); +list_add_tail (&llc_occupancy_event.list, &r->mon.evt_list);if (is_mbm_total_enabled ()) -list_add_tail (&mbm_total_event.list, &r->evt_list); +list_add_tail (&mbm_total_event.list, &r->mon.evt_list);if (is_mbm_local_enabled ()) -list_add_tail (&mbm_local_event.list, &r->evt_list); +list_add_tail (&mbm_local_event.list, &r->mon.evt_list); }/*
@@ -1186,7 +1186,7 @@ int __init rdt_get_mon_l3_config(struct rdt_resource *r) resctrl_rmid_realloc_limit = boot_cpu_data.x86_cache_size *1024 ; hw_res->mon_scale = boot_cpu_data.x86_cache_occ_scale / snc_nodes_per_l3_cache; - r->num_rmid = (boot_cpu_data.x86_cache_max_rmid +1 ) / snc_nodes_per_l3_cache; + r->mon.num_rmid = (boot_cpu_data.x86_cache_max_rmid +1 ) / snc_nodes_per_l3_cache; hw_res->mbm_width =MBM_CNTR_WIDTH_BASE ;if (mbm_offset >0 && mbm_offset <=MBM_CNTR_WIDTH_OFFSET_MAX )
@@ -1201,7 +1201,7 @@ int __init rdt_get_mon_l3_config(struct rdt_resource *r)* * For a 35MB LLC and 56 RMIDs, this is ~1.8% of the LLC. */ - threshold = resctrl_rmid_realloc_limit / r->num_rmid; + threshold = resctrl_rmid_realloc_limit / r->mon.num_rmid;/* * Because num_rmid may not be a power of two, round the value
arch/x86/kernel/cpu/resctrl/rdtgroup.c +4 -4
@@ -1097,7 +1097,7 @@ static int rdt_num_rmids_show(struct kernfs_open_file *of, {struct rdt_resource *r = of->kn->parent->priv; -seq_printf (seq,"%d\n" , r->num_rmid); +seq_printf (seq,"%d\n" , r->mon.num_rmid);return 0 ; }
@@ -1108,7 +1108,7 @@ static int rdt_mon_features_show(struct kernfs_open_file *of,struct rdt_resource *r = of->kn->parent->priv;struct mon_evt *mevt; -list_for_each_entry (mevt, &r->evt_list, list) { +list_for_each_entry (mevt, &r->mon.evt_list, list) {seq_printf (seq,"%s\n" , mevt->name);if (mevt->configurable)seq_printf (seq,"%s_config\n" , mevt->name);
@@ -3057,13 +3057,13 @@ static int mon_add_all_files(struct kernfs_node *kn, struct rdt_mon_domain *d,struct mon_evt *mevt;int ret; -if (WARN_ON (list_empty (&r->evt_list))) +if (WARN_ON (list_empty (&r->mon.evt_list)))return -EPERM ; priv.u.rid = r->rid; priv.u.domid = do_sum ? d->ci->id : d->hdr.id; priv.u.sum = do_sum; -list_for_each_entry (mevt, &r->evt_list, list) { +list_for_each_entry (mevt, &r->mon.evt_list, list) { priv.u.evtid = mevt->evtid; ret =mon_addfile (kn, mevt->name, priv.priv);if (ret)
include/linux/resctrl.h +12 -4
@@ -182,16 +182,26 @@ enum resctrl_scope { RESCTRL_L3_NODE, }; +/** +* struct resctrl_mon - Monitoring related data of a resctrl resource +* @num_rmid: Number of RMIDs available +* @evt_list: List of monitoring events +*/ +struct resctrl_mon { +int num_rmid; +struct list_head evt_list; +}; +/** * struct rdt_resource - attributes of a resctrl resource * @rid: The index of the resource * @alloc_capable: Is allocation available on this machine * @mon_capable: Is monitor feature available on this machine -* @num_rmid: Number of RMIDs available * @ctrl_scope: Scope of this resource for control functions * @mon_scope: Scope of this resource for monitor functions * @cache: Cache allocation related data * @membw: If the component has bandwidth controls, their properties. +* @mon: Monitoring related data. * @ctrl_domains: RCU list of all control domains for this resource * @mon_domains: RCU list of all monitor domains for this resource * @name: Name to use in "schemata" file.
@@ -199,7 +209,6 @@ enum resctrl_scope {* @default_ctrl: Specifies default cache cbm or memory B/W percent. * @format_str: Per resource format string to show domain value * @parse_ctrlval: Per resource function pointer to parse control values -* @evt_list: List of monitoring events * @fflags: flags to choose base and info files * @cdp_capable: Is the CDP feature available on this resource */
@@ -207,11 +216,11 @@ struct rdt_resource {int rid;bool alloc_capable;bool mon_capable; -int num_rmid;enum resctrl_scope ctrl_scope;enum resctrl_scope mon_scope;struct resctrl_cache cache;struct resctrl_membw membw; +struct resctrl_mon mon;struct list_head ctrl_domains;struct list_head mon_domains;char *name;
@@ -221,7 +230,6 @@ struct rdt_resource {int (*parse_ctrlval)(struct rdt_parse_data *data,struct resctrl_schema *s,struct rdt_ctrl_domain *d); -struct list_head evt_list;unsigned long fflags;bool cdp_capable; };
Signature · 1 line
2.34.1
Babu Moger
reviewerPATCH v8 4/25 x86/resctrl: Detect Assignable Bandwidth Monitoring feature details
In reply to Babu MogerABMC feature details are reported via CPUID Fn8000_0020_EBX_x5. Bits Description
15:0 MAX_ABMC Maximum Supported Assignable Bandwidth
Monitoring Counter ID + 1
The feature details are documented in APM listed below [1].
[1] AMD64 Architecture Programmer's Manual Volume 2: System Programming Publication # 24593 Revision 3.41 section 19.3.3.3 Assignable Bandwidth Monitoring (ABMC).
Detect the feature and number of assignable monitoring counters supported.
---
v8: Used GENMASK for the mask.
v7: Removed WARN_ON for num_mbm_cntrs. Decided to dynamically allocate the
bitmap. WARN_ON is not required anymore.
Removed redundant comments.
v6: Commit message update.
Renamed abmc_capable to mbm_cntr_assignable.
v5: Name change num_cntrs to num_mbm_cntrs.
Moved abmc_capable to resctrl_mon.
v4: Removed resctrl_arch_has_abmc(). Added all the code inline. We dont
need to separate this as arch code.
v3: Removed changes related to mon_features.
Moved rdt_cpu_has to core.c and added new function resctrl_arch_has_abmc. Also moved the fields mbm_assign_capable and mbm_assign_cntrs to rdt_resource. (James)
v2: Changed the field name to mbm_assign_capable from abmc_capable.
--- arch/x86/kernel/cpu/resctrl/monitor.c | 6 ++++++ include/linux/resctrl.h | 4 ++++ 2 files changed, 10 insertions(+)
arch/x86/kernel/cpu/resctrl/monitor.c +6 -0
@@ -1229,6 +1229,12 @@ int __init rdt_get_mon_l3_config(struct rdt_resource *r) mbm_local_event.configurable = true;mbm_config_rftype_init ("mbm_local_bytes_config" ); } + +if (rdt_cpu_has (X86_FEATURE_ABMC )) { + r->mon.mbm_cntr_assignable = true; +cpuid_count (0x80000020 ,5 , &eax, &ebx, &ecx, &edx); + r->mon.num_mbm_cntrs = (ebx &GENMASK (15 ,0 )) +1 ; + } }l3_mon_evt_init (r);
include/linux/resctrl.h +4 -0
@@ -185,10 +185,14 @@ enum resctrl_scope {/** * struct resctrl_mon - Monitoring related data of a resctrl resource * @num_rmid: Number of RMIDs available +* @num_mbm_cntrs: Number of assignable monitoring counters +* @mbm_cntr_assignable:Is system capable of supporting monitor assignment? * @evt_list: List of monitoring events */ struct resctrl_mon {int num_rmid; +int num_mbm_cntrs; +bool mbm_cntr_assignable;struct list_head evt_list; };
Signature · 1 line
2.34.1
Re: [PATCH v8 04/25] x86/resctrl: Detect Assignable Bandwidth Monitoring feature details
Hi Babu,
Quoted 10 lines · Babu Moger
ABMC feature details are reported via CPUID
Fn8000_0020_EBX_x5. Bits Description
15:0MAX_ABMCMaximum Supported Assignable Bandwidth
Monitoring Counter ID + 1The feature details are documented in APM listed below [1].
[1] AMD64 Architecture Programmer's Manual Volume 2: System Programming Publication # 24593 Revision 3.41 section 19.3.3.3 Assignable Bandwidth Monitoring (ABMC).
Quoted 5 lines
Detect the feature and number of assignable monitoring counters supported.
---
Reinette
Babu Moger
reviewerPATCH v8 5/25 x86/resctrl: Introduce resctrl_file_fflags_init() to initialize fflags
In reply to Babu Mogerthread_throttle_mode_init() and mbm_config_rftype_init() both initialize fflags for resctrl files.
Adding new files will involve adding another function to initialize the fflags. This can be simplified by adding a new function resctrl_file_fflags_init() and passing the file name and flags to be initialized.
Consolidate fflags initialization into resctrl_file_fflags_init() and remove thread_throttle_mode_init() and mbm_config_rftype_init().
---
v8: No changes.
v7: No changes.
v6: Added Reviewed-by from Reinette.
v5: Commit message update.
v4: Commit message update.
v3: New patch to display ABMC capability.
--- arch/x86/kernel/cpu/resctrl/core.c | 4 +++- arch/x86/kernel/cpu/resctrl/internal.h | 4 ++-- arch/x86/kernel/cpu/resctrl/monitor.c | 6 ++++-- arch/x86/kernel/cpu/resctrl/rdtgroup.c | 16 +++------------- 4 files changed, 12 insertions(+), 18 deletions(-)
arch/x86/kernel/cpu/resctrl/core.c +3 -1
@@ -234,7 +234,9 @@ static bool __get_mem_config_intel(struct rdt_resource *r) r->membw.throttle_mode =THREAD_THROTTLE_PER_THREAD ; else r->membw.throttle_mode = THREAD_THROTTLE_MAX; -thread_throttle_mode_init (); + +resctrl_file_fflags_init ("thread_throttle_mode" , +RFTYPE_CTRL_INFO |RFTYPE_RES_MB ); r->alloc_capable = true;
arch/x86/kernel/cpu/resctrl/internal.h +2 -2
@@ -647,8 +647,8 @@ void cqm_handle_limbo(struct work_struct *work);bool has_busy_rmid (struct rdt_mon_domain *d);void __check_limbo (struct rdt_mon_domain *d,bool force_free);void rdt_domain_reconfigure_cdp (struct rdt_resource *r); -void __init thread_throttle_mode_init (void ); -void __init mbm_config_rftype_init (const char *config); +void __init resctrl_file_fflags_init (const char *config, +unsigned long fflags);void rdt_staged_configs_clear (void );bool closid_allocated (unsigned int closid);int resctrl_find_cleanest_closid (void );
arch/x86/kernel/cpu/resctrl/monitor.c +4 -2
@@ -1223,11 +1223,13 @@ int __init rdt_get_mon_l3_config(struct rdt_resource *r)if (rdt_cpu_has (X86_FEATURE_CQM_MBM_TOTAL )) { mbm_total_event.configurable = true; -mbm_config_rftype_init ("mbm_total_bytes_config" ); +resctrl_file_fflags_init ("mbm_total_bytes_config" , +RFTYPE_MON_INFO |RFTYPE_RES_CACHE ); }if (rdt_cpu_has (X86_FEATURE_CQM_MBM_LOCAL )) { mbm_local_event.configurable = true; -mbm_config_rftype_init ("mbm_local_bytes_config" ); +resctrl_file_fflags_init ("mbm_local_bytes_config" , +RFTYPE_MON_INFO |RFTYPE_RES_CACHE ); }if (rdt_cpu_has (X86_FEATURE_ABMC )) {
arch/x86/kernel/cpu/resctrl/rdtgroup.c +3 -13
@@ -2020,24 +2020,14 @@ static struct rftype *rdtgroup_get_rftype_by_name(const char *name)return NULL ; } -void __init thread_throttle_mode_init (void ) -{ -struct rftype *rft; - - rft =rdtgroup_get_rftype_by_name ("thread_throttle_mode" ); -if (!rft) -return ; - - rft->fflags =RFTYPE_CTRL_INFO |RFTYPE_RES_MB ; -} - -void __init mbm_config_rftype_init (const char *config) +void __init resctrl_file_fflags_init (const char *config, +unsigned long fflags) {struct rftype *rft; rft =rdtgroup_get_rftype_by_name (config);if (rft) - rft->fflags =RFTYPE_MON_INFO |RFTYPE_RES_CACHE ; + rft->fflags = fflags; }/**
Signature · 1 line
2.34.1
Babu Moger
reviewerPATCH v8 6/25 x86/resctrl: Add support to enable/disable AMD ABMC feature
In reply to Babu MogerAdd the functionality to enable/disable AMD ABMC feature.
AMD ABMC feature is enabled by setting enabled bit(0) in MSR L3_QOS_EXT_CFG. When the state of ABMC is changed, the MSR needs to be updated on all the logical processors in the QOS Domain.
Hardware counters will reset when ABMC state is changed.
The ABMC feature details are documented in APM listed below [1].
[1] AMD64 Architecture Programmer's Manual Volume 2: System Programming Publication # 24593 Revision 3.41 section 19.3.3.3 Assignable Bandwidth Monitoring (ABMC).
---
v8:
Commit message update and moved around the comments about L3_QOS_EXT_CFG to _resctrl_abmc_enable.
v7:
Renamed the function
resctrl_arch_get_abmc_enabled() to resctrl_arch_mbm_cntr_assign_enabled().
Merged resctrl_arch_mbm_cntr_assign_disable, resctrl_arch_mbm_cntr_assign_disable and renamed to resctrl_arch_mbm_cntr_assign_set().
Moved the function definition to linux/resctrl.h.
Passed the struct rdt_resource to these functions.
Removed resctrl_arch_reset_rmid_all() from arch code. This will be done from the caller.
v6: Renamed abmc_enabled to mbm_cntr_assign_enabled.
Used msr_set_bit and msr_clear_bit for msr updates.
Renamed resctrl_arch_abmc_enable() to resctrl_arch_mbm_cntr_assign_enable(). Renamed resctrl_arch_abmc_disable() to resctrl_arch_mbm_cntr_assign_disable(). Made _resctrl_abmc_enable to return void.
v5: Renamed resctrl_abmc_enable to resctrl_arch_abmc_enable.
Renamed resctrl_abmc_disable to resctrl_arch_abmc_disable. Introduced resctrl_arch_get_abmc_enabled to get abmc state from non-arch code.
Renamed resctrl_abmc_set_all to _resctrl_abmc_enable(). Modified commit log to make it clear about AMD ABMC feature.
v3: No changes.
v2: Few text changes in commit message.
--- arch/x86/include/asm/msr-index.h | 1 + arch/x86/kernel/cpu/resctrl/core.c | 5 ++++ arch/x86/kernel/cpu/resctrl/internal.h | 5 ++++ arch/x86/kernel/cpu/resctrl/rdtgroup.c | 36 ++++++++++++++++++++++++++ include/linux/resctrl.h | 3 +++ 5 files changed, 50 insertions(+)
arch/x86/include/asm/msr-index.h +1 -0
@@ -1195,6 +1195,7 @@#define MSR_IA32_MBA_BW_BASE 0xc0000200#define MSR_IA32_SMBA_BW_BASE 0xc0000280#define MSR_IA32_EVT_CFG_BASE 0xc0000400 +#define MSR_IA32_L3_QOS_EXT_CFG 0xc00003ff/* AMD-V MSRs */ #define MSR_VM_CR 0xc0010114
arch/x86/kernel/cpu/resctrl/core.c +5 -0
@@ -405,6 +405,11 @@ void rdt_ctrl_update(void *arg) hw_res->msr_update (m); } +bool resctrl_arch_mbm_cntr_assign_enabled (struct rdt_resource *r) +{ +return resctrl_to_arch_res (r)->mbm_cntr_assign_enabled; +} +/* * rdt_find_domain - Search for a domain id in a resource domain list. *
arch/x86/kernel/cpu/resctrl/internal.h +5 -0
@@ -56,6 +56,9 @@/* Max event bits supported */ #define MAX_EVT_CONFIG_BITS GENMASK(6, 0) +/* Setting bit 0 in L3_QOS_EXT_CFG enables the ABMC feature. */ +#define ABMC_ENABLE_BIT 0 +/** * cpumask_any_housekeeping() - Choose any CPU in @mask, preferring those that * aren't marked nohz_full
@@ -477,6 +480,7 @@ struct rdt_parse_data {* @mbm_cfg_mask: Bandwidth sources that can be tracked when Bandwidth * Monitoring Event Configuration (BMEC) is supported. * @cdp_enabled: CDP state of this resource +* @mbm_cntr_assign_enabled: ABMC feature is enabled * * Members of this structure are either private to the architecture * e.g. mbm_width, or accessed via helpers that provide abstraction. e.g.
@@ -491,6 +495,7 @@ struct rdt_hw_resource {unsigned int mbm_width;unsigned int mbm_cfg_mask;bool cdp_enabled; +bool mbm_cntr_assign_enabled; };static inline struct rdt_hw_resource *resctrl_to_arch_res (struct rdt_resource *r)
arch/x86/kernel/cpu/resctrl/rdtgroup.c +36 -0
@@ -2402,6 +2402,42 @@ int resctrl_arch_set_cdp_enabled(enum resctrl_res_level l, bool enable)return 0 ; } +static void resctrl_abmc_set_one_amd (void *arg) +{ +bool *enable = arg; + +if (*enable) +msr_set_bit (MSR_IA32_L3_QOS_EXT_CFG ,ABMC_ENABLE_BIT ); +else +msr_clear_bit (MSR_IA32_L3_QOS_EXT_CFG ,ABMC_ENABLE_BIT ); +} + +/* +* Update L3_QOS_EXT_CFG MSR on all the CPUs associated with the monitor +* domain. +*/ +static void _resctrl_abmc_enable (struct rdt_resource *r,bool enable) +{ +struct rdt_mon_domain *d; + +list_for_each_entry (d, &r->mon_domains, hdr.list) +on_each_cpu_mask (&d->hdr.cpu_mask, + resctrl_abmc_set_one_amd, &enable,1 ); +} + +int resctrl_arch_mbm_cntr_assign_set (struct rdt_resource *r,bool enable) +{ +struct rdt_hw_resource *hw_res =resctrl_to_arch_res (r); + +if (r->mon.mbm_cntr_assignable && + hw_res->mbm_cntr_assign_enabled != enable) { +_resctrl_abmc_enable (r, enable); + hw_res->mbm_cntr_assign_enabled = enable; + } + +return 0 ; +} +/* * We don't allow rdtgroup directories to be created anywhere * except the root directory. Thus when looking for the rdtgroup
include/linux/resctrl.h +3 -0
@@ -355,4 +355,7 @@ void resctrl_arch_reset_rmid_all(struct rdt_resource *r, struct rdt_mon_domain *extern unsigned int resctrl_rmid_realloc_threshold;extern unsigned int resctrl_rmid_realloc_limit; +int resctrl_arch_mbm_cntr_assign_set (struct rdt_resource *r,bool enable); +bool resctrl_arch_mbm_cntr_assign_enabled (struct rdt_resource *r); +#endif /* _RESCTRL_H */
Signature · 1 line
2.34.1
Tony Luck
maintainerRe: [PATCH v8 06/25] x86/resctrl: Add support to enable/disable AMD ABMC feature
arch/x86/include/asm/msr-index.h line 1198
Quoted 9 lines · Babu Moger
arch/x86/include/asm/msr-index.h +1
-0@@ -1195,6 +1195,7 @@#define MSR_IA32_MBA_BW_BASE 0xc0000200#define MSR_IA32_SMBA_BW_BASE 0xc0000280#define MSR_IA32_EVT_CFG_BASE 0xc0000400 +#define MSR_IA32_L3_QOS_EXT_CFG 0xc00003ff
Nitpick. Most of the MSRs in this file are in numerical order (within each functional grouping). So this belongs before MSR_IA32_EVT_CFG_BASE
Same in patch 14 which adds MSR_IA32_L3_QOS_ABMC_CFG
-Tony
Re: [PATCH v8 06/25] x86/resctrl: Add support to enable/disable AMD ABMC feature
Hi Tony,
Quoted 14 lines · Tony Luck
arch/x86/include/asm/msr-index.h line 1198
Quoted 9 lines · Babu Moger
arch/x86/include/asm/msr-index.h +1
-0@@ -1195,6 +1195,7 @@#define MSR_IA32_MBA_BW_BASE 0xc0000200#define MSR_IA32_SMBA_BW_BASE 0xc0000280#define MSR_IA32_EVT_CFG_BASE 0xc0000400 +#define MSR_IA32_L3_QOS_EXT_CFG 0xc00003ffNitpick. Most of the MSRs in this file are in numerical order (within each functional grouping). So this belongs before
MSR_IA32_EVT_CFG_BASE
Quoted 1 line
Same in patch 14 which adds
MSR_IA32_L3_QOS_ABMC_CFG
Yes. Will take care of this in next revision.
Signature · 1 line
- Babu Moger
Reinette Chatre
maintainerRe: [PATCH v8 06/25] x86/resctrl: Add support to enable/disable AMD ABMC feature
In reply to Babu MogerHi Babu,
Quoted 8 lines · Babu Moger
Add the functionality to enable/disable AMD ABMC feature.
AMD ABMC feature is enabled by setting enabled bit(0) in MSR
L3_QOS_EXT_CFG. When the state of ABMC is changed, the MSR needs to be updated on all the logical processors in the QOS Domain.Hardware counters will reset when ABMC state is changed.
Quoted 8 lines
The ABMC feature details are documented in APM listed below [1].
[1] AMD64 Architecture Programmer's Manual Volume 2: System Programming Publication # 24593 Revision 3.41 section 19.3.3.3 Assignable Bandwidth Monitoring (ABMC).---
With the MSRs ordered numerically per Tony's suggestion:
|Reviewed-by: Reinette Chatre <[email protected]>
Reinette
Babu Moger
reviewerPATCH v8 7/25 x86/resctrl: Introduce the interface to display monitor mode
In reply to Babu MogerIntroduce the interface file "mbm_assign_mode" to list monitor modes supported.
The "mbm_cntr_assign" mode provides the option to assign a counter to an RMID, event pair and monitor the bandwidth as long as it is assigned.
On AMD systems "mbm_cntr_assign" is backed by the ABMC (Assignable Bandwidth Monitoring Counters) hardware feature and is enabled by default.
The "default" mode is the existing monitoring mode that works without the explicit counter assignment, instead relying on dynamic counter assignment by hardware that may result in hardware not dedicating a counter resulting in monitoring data reads returning "Unavailable".
Provide an interface to display the monitor mode on the system. $cat /sys/fs/resctrl/info/L3_MON/mbm_assign_mode [mbm_cntr_assign]
default
Switching the mbm_assign_mode will reset all the MBM counters of all resctrl groups.
---
v8: Commit message update.
v7: Updated the descriptions/commit log in resctrl.rst to generic text.
Thanks to James and Reinette.
Rename mbm_mode to mbm_assign_mode.
Introduced mutex lock in rdtgroup_mbm_mode_show().
v6: Added documentation for mbm_cntr_assign and legacy mode.
Moved mbm_mode fflags initialization to static initialization.
v5: Changed interface name to mbm_mode.
It will be always available even if ABMC feature is not supported. Added description in resctrl.rst about ABMC mode.
Fixed display abmc and legacy consistantly.
v4: Fixed the checks for legacy and abmc mode. Default it ABMC.
v3: New patch to display ABMC capability.
--- Documentation/arch/x86/resctrl.rst | 34 ++++++++++++++++++++++++++ arch/x86/kernel/cpu/resctrl/rdtgroup.c | 31 +++++++++++++++++++++++ 2 files changed, 65 insertions(+)
Documentation/arch/x86/resctrl.rst +34 -0
@@ -257,6 +257,40 @@ with the following files: # cat /sys/fs/resctrl/info/L3_MON/mbm_local_bytes_config 0=0x30;1=0x30;3=0x15;4=0x15 +"mbm_assign_mode": + Reports the list of monitoring modes supported. The enclosed brackets + indicate which mode is enabled. + :: + + cat /sys/fs/resctrl/info/L3_MON/mbm_assign_mode + [mbm_cntr_assign] + default + + "mbm_cntr_assign": + + In mbm_cntr_assign mode user-space is able to specify which control + or monitor groups in resctrl should have a counter assigned using the + 'mbm_assign_control' file. The number of counters available is described + in the 'num_mbm_cntrs' file. Changing the mode may cause all counters on + a resource to reset. + + The mode is useful on platforms which support more control and monitor + groups than hardware counters, meaning 'unassigned' control or monitor + groups will report 'Unavailable' or count the traffic in an unpredictable + way. + + AMD Platforms with ABMC (Assignable Bandwidth Monitoring Counters) feature + enable this mode by default so that counters remain assigned even when the + corresponding RMID is not in use by any processor. + + "default": + + By default resctrl assumes each control and monitor group has a hardware + counter. Hardware that does not support 'mbm_cntr_assign' mode will still + allow more control or monitor groups than 'num_rmids' to be created. In + that case reading the mbm_total_bytes and mbm_local_bytes may report + 'Unavailable' if there is no counter associated with that group. + "max_threshold_occupancy": Read/write file provides the largest value (in bytes) at which a previously used LLC_occupancy
arch/x86/kernel/cpu/resctrl/rdtgroup.c +31 -0
@@ -845,6 +845,30 @@ static int rdtgroup_rmid_show(struct kernfs_open_file *of,return ret; } +static int rdtgroup_mbm_assign_mode_show (struct kernfs_open_file *of, +struct seq_file *s,void *v) +{ +struct rdt_resource *r = of->kn->parent->priv; + +mutex_lock (&rdtgroup_mutex); + +if (r->mon.mbm_cntr_assignable) { +if (resctrl_arch_mbm_cntr_assign_enabled (r)) { +seq_puts (s,"[mbm_cntr_assign]\n" ); +seq_puts (s,"default\n" ); + }else { +seq_puts (s,"mbm_cntr_assign\n" ); +seq_puts (s,"[default]\n" ); + } + }else { +seq_puts (s,"[default]\n" ); + } + +mutex_unlock (&rdtgroup_mutex); + +return 0 ; +} +#ifdef CONFIG_PROC_CPU_RESCTRL /*
@@ -1901,6 +1925,13 @@ static struct rftype res_common_files[] = { .seq_show = mbm_local_bytes_config_show, .write = mbm_local_bytes_config_write, }, + { + .name = "mbm_assign_mode", + .mode = 0444, + .kf_ops = &rdtgroup_kf_single_ops, + .seq_show = rdtgroup_mbm_assign_mode_show, + .fflags = RFTYPE_MON_INFO, + }, { .name = "cpus", .mode = 0644,
Signature · 1 line
2.34.1
Tony Luck
maintainerRe: [PATCH v8 07/25] x86/resctrl: Introduce the interface to display monitor mode
Quoted 21 lines · Babu Moger
+"mbm_assign_mode":
Documentation/arch/x86/resctrl.rst +20
-0+ Reports the list of monitoring modes supported. The enclosed brackets + indicate which mode is enabled. + :: + + cat /sys/fs/resctrl/info/L3_MON/mbm_assign_mode + [mbm_cntr_assign] + default + + "mbm_cntr_assign": + + In mbm_cntr_assign mode user-space is able to specify which control + or monitor groups in resctrl should have a counter assigned using the + 'mbm_assign_control' file. The number of counters available is described + in the 'num_mbm_cntrs' file. Changing the mode may cause all counters on + a resource to reset. + + The mode is useful on platforms which support more control and monitor + groups than hardware counters, meaning 'unassigned' control or monitor + groups will report 'Unavailable' or count the traffic in an unpredictable + way.
Documentation/arch/x86/resctrl.rst with the following files: · line 290
Quoted 10 lines
Documentation/arch/x86/resctrl.rst +10
-0+ + AMD Platforms with ABMC (Assignable Bandwidth Monitoring Counters) feature + enable this mode by default so that counters remain assigned even when the + corresponding RMID is not in use by any processor. + + "default": + + By default resctrl assumes each control and monitor group has a hardware + counter. Hardware that does not support 'mbm_cntr_assign' mode will still + allow more control or monitor groups than 'num_rmids' to be created. In
Should that be s/num_rmids/num_mbm_cntrs/ ?
-Tony
Moger, Babu
reviewerRe: [PATCH v8 07/25] x86/resctrl: Introduce the interface to display monitor mode
Hi Tony,
Thanks for reviewing the patches.
Quoted 26 lines · Tony Luck
Quoted 25 lines · Babu Moger
+"mbm_assign_mode":
Documentation/arch/x86/resctrl.rst +24
-0+ Reports the list of monitoring modes supported. The enclosed brackets + indicate which mode is enabled. + :: + + cat /sys/fs/resctrl/info/L3_MON/mbm_assign_mode + [mbm_cntr_assign] + default + + "mbm_cntr_assign": + + In mbm_cntr_assign mode user-space is able to specify which control + or monitor groups in resctrl should have a counter assigned using the + 'mbm_assign_control' file. The number of counters available is described + in the 'num_mbm_cntrs' file. Changing the mode may cause all counters on + a resource to reset. + + The mode is useful on platforms which support more control and monitor + groups than hardware counters, meaning 'unassigned' control or monitor + groups will report 'Unavailable' or count the traffic in an unpredictable + way. + + AMD Platforms with ABMC (Assignable Bandwidth Monitoring Counters) feature + enable this mode by default so that counters remain assigned even when the + corresponding RMID is not in use by any processor.
Quoted 8 lines
Documentation/arch/x86/resctrl.rst with the following files: · line 290
Quoted 6 lines
Documentation/arch/x86/resctrl.rst +6
-0+ + "default": + + By default resctrl assumes each control and monitor group has a hardware + counter. Hardware that does not support 'mbm_cntr_assign' mode will still + allow more control or monitor groups than 'num_rmids' to be created. InShould that be s/num_rmids/num_mbm_cntrs/ ?
It is actually num_rmids here as in default mode, num_rmid_cntrs are not available.
Signature · 2 lines
Thanks Babu Moger
Luck, Tony
maintainerRE: [PATCH v8 07/25] x86/resctrl: Introduce the interface to display monitor mode
Quoted 8 lines
Quoted 5 lines
Documentation/arch/x86/resctrl.rst with the following files: · line 290
Quoted 3 lines
Documentation/arch/x86/resctrl.rst +3
-0+ By default resctrl assumes each control and monitor group has a hardware + counter. Hardware that does not support 'mbm_cntr_assign' mode will still + allow more control or monitor groups than 'num_rmids' to be created. InShould that be s/num_rmids/num_mbm_cntrs/ ?
It is actually
num_rmidshere as in default mode,num_rmid_cntrsare not available.
Babu,
The code isn't working that way for me. I built & booted. Since I'm on an Intel machine without ABMC I'm in "default" mode. But I can't make more monitor groups that num_rmids.
-Tony
Moger, Babu
reviewerRe: [PATCH v8 07/25] x86/resctrl: Introduce the interface to display monitor mode
Hi Tony,
Quoted 6 lines · Tony
Quoted 6 lines
Quoted 5 lines
Documentation/arch/x86/resctrl.rst with the following files: · line 290
Quoted 3 lines
Documentation/arch/x86/resctrl.rst +3
-0+ By default resctrl assumes each control and monitor group has a hardware + counter. Hardware that does not support 'mbm_cntr_assign' mode will still + allow more control or monitor groups than 'num_rmids' to be created. InShould that be s/num_rmids/num_mbm_cntrs/ ?
Quoted 9 lines
Quoted 2 lines
It is actually
num_rmidshere as in default mode,num_rmid_cntrsare not available.Babu,
The code isn't working that way for me. I built & booted. Since I'm on an Intel machine without ABMC I'm in "default" mode. But I can't make more monitor groups that
num_rmids.
That is correct. We will have to change the text. How about?
"default":
By default resctrl assumes each control and monitor group has a hardware counter. Hardware that does not support 'mbm_cntr_assign' mode will still allow to create control or monitor groups up to num_rmids supported. In that case reading the mbm_total_bytes and mbm_local_bytes may report 'Unavailable' if there is no counter associated with that group.
Signature · 2 lines
Thanks Babu Moger
Luck, Tony
maintainerRE: [PATCH v8 07/25] x86/resctrl: Introduce the interface to display monitor mode
Quoted 8 lines
That is correct. We will have to change the text. How about?
"default":
By default resctrl assumes each control and monitor group has a hardware counter. Hardware that does not support 'mbm_cntr_assign' mode will still allow to create control or monitor groups up tonum_rmidssupported. In that case reading thembm_total_bytesandmbm_local_bytesmay report 'Unavailable' if there is no counter associated with that group.
Babu,
Looks good to me.
Thanks
-Tony
Reinette Chatre
maintainerRe: [PATCH v8 07/25] x86/resctrl: Introduce the interface to display monitor mode
In reply to Moger, BabuQuoted 17 lines · Babu
Quoted 6 lines · Tony
Quoted 6 lines
Quoted 5 lines
Documentation/arch/x86/resctrl.rst with the following files: · line 290
Quoted 3 lines
Documentation/arch/x86/resctrl.rst +3
-0+ By default resctrl assumes each control and monitor group has a hardware + counter. Hardware that does not support 'mbm_cntr_assign' mode will still + allow more control or monitor groups than 'num_rmids' to be created. InShould that be s/num_rmids/num_mbm_cntrs/ ?
Quoted 9 lines
Quoted 2 lines
It is actually
num_rmidshere as in default mode,num_rmid_cntrsare not available.Babu,
The code isn't working that way for me. I built & booted. Since I'm on an Intel machine without ABMC I'm in "default" mode. But I can't make more monitor groups that
num_rmids.
Quoted 5 lines
That is correct. We will have to change the text. How about?
"default":
By default resctrl assumes each control and monitor group has a hardware counter. Hardware that does not support 'mbm_cntr_assign' mode will still
I think this is independent from whether hardware supports 'mbm_cntr_assign' mode since a user could enable 'default' mode on hardware that supports 'mbm_cntr_assign'. This snippet is thus more about what is meant by 'default' mode than what is supported by hardware.
The docs already contain:
"num_rmids": ... This is the upper bound for how many "CTRL_MON" + "MON" groups can be created.
Neither of the 'mbm_assign_mode' options change this meaning of 'num_rmids' (i.e. no change in how many monitor groups can be created) so mentioning it in the 'default' portion but not in the 'mbm_cntr_assign' portion may create confusion.
Perhaps it can be simplified to:
In default mode resctrl assumes each CTRL_MON and MON group has a hardware counter. Reading mbm_total_bytes or mbm_local_bytes may report 'Unavailable' if there is no counter associated with that group.
Quoted 4 lines
allow to create control or monitor groups up to
num_rmidssupported. In that case reading thembm_total_bytesandmbm_local_bytesmay report 'Unavailable' if there is no counter associated with that group.
Reinette
Moger, Babu
reviewerRe: [PATCH v8 07/25] x86/resctrl: Introduce the interface to display monitor mode
Hi Reinette,
Quoted 43 lines · On 10/11/24 17:24, wrote:
Quoted 17 lines · Babu
Quoted 6 lines · Tony
Quoted 6 lines
Quoted 5 lines
Documentation/arch/x86/resctrl.rst with the following files: · line 290
Quoted 3 lines
Documentation/arch/x86/resctrl.rst +3
-0+ By default resctrl assumes each control and monitor group has a hardware + counter. Hardware that does not support 'mbm_cntr_assign' mode will still + allow more control or monitor groups than 'num_rmids' to be created. InShould that be s/num_rmids/num_mbm_cntrs/ ?
Quoted 9 lines
Quoted 2 lines
It is actually
num_rmidshere as in default mode,num_rmid_cntrsare not available.Babu,
The code isn't working that way for me. I built & booted. Since I'm on an Intel machine without ABMC I'm in "default" mode. But I can't make more monitor groups that
num_rmids.Quoted 5 lines
That is correct. We will have to change the text. How about?
"default":
By default resctrl assumes each control and monitor group has a hardware counter. Hardware that does not support 'mbm_cntr_assign' mode will stillI think this is independent from whether hardware supports '
mbm_cntr_assign' mode since a user could enable 'default' mode on hardware that supports 'mbm_cntr_assign'. This snippet is thus more about what is meant by 'default' mode than what is supported by hardware.The docs already contain:
"num_rmids": ... This is the upper bound for how many "CTRL_MON" + "MON" groups can be created.Neither of the '
mbm_assign_mode' options change this meaning of 'num_rmids' (i.e. no change in how many monitor groups can be created) so mentioning it in the 'default' portion but not in the 'mbm_cntr_assign' portion may create confusion.
Quoted 5 lines
Perhaps it can be simplified to:
In default mode resctrl assumes eachCTRL_MONand MON group has a hardware counter. Readingmbm_total_bytesormbm_local_bytesmay report 'Unavailable' if there is no counter associated with that group.
Sure.
Signature · 2 lines
Thanks Babu Moger
Reinette Chatre
maintainerRe: [PATCH v8 07/25] x86/resctrl: Introduce the interface to display monitor mode
In reply to Babu MogerHi Babu,
Quoted 14 lines · Babu Moger
Introduce the interface file "
mbm_assign_mode" to list monitor modes supported.The "
mbm_cntr_assign" mode provides the option to assign a counter to an RMID, event pair and monitor the bandwidth as long as it is assigned.On AMD systems "
mbm_cntr_assign" is backed by the ABMC (Assignable Bandwidth Monitoring Counters) hardware feature and is enabled by default.The "default" mode is the existing monitoring mode that works without the explicit counter assignment, instead relying on dynamic counter assignment by hardware that may result in hardware not dedicating a counter resulting in monitoring data reads returning "Unavailable".
Quoted 7 lines
Provide an interface to display the monitor mode on the system. $cat
/sys/fs/resctrl/info/L3_MON/mbm_assign_mode[mbm_cntr_assign]
defaultSwitching the
mbm_assign_modewill reset all the MBM counters of all resctrl groups.
Please note that this now contradicts the documentation. Perhaps this sentence can just be dropped since there is the documentation within the patch.
Quoted 2 lines
Signed-off-by Babu Moger <[email protected]>---
...
Quoted 11 lines
Documentation/arch/x86/resctrl.rst +3
-0@@ -257,6 +257,40 @@ with the following files: # cat /sys/fs/resctrl/info/L3_MON/mbm_local_bytes_config 0=0x30;1=0x30;3=0x15;4=0x15 +"mbm_assign_mode": + Reports the list of monitoring modes supported. The enclosed brackets + indicate which mode is enabled.
Documentation/arch/x86/resctrl.rst with the following files: · line 272
Quoted 10 lines
Documentation/arch/x86/resctrl.rst +10
-0+ :: + + cat /sys/fs/resctrl/info/L3_MON/mbm_assign_mode + [mbm_cntr_assign] + default + + "mbm_cntr_assign": + + In mbm_cntr_assign mode user-space is able to specify which control + or monitor groups in resctrl should have a counter assigned using the
Counters cannot be assigned to control groups. How about replacing all instances of "control and monitor groups" with "CTRL_MON and MON groups", similarly "control or monitor groups" with "CTRL_MON or MON groups".
Documentation/arch/x86/resctrl.rst with the following files: · line 273
Quoted 1 line
Documentation/arch/x86/resctrl.rst +1
-0+ 'mbm_assign_control' file. The number of counters available is described
Looking at the rest of the doc it seems that the custom is actually to place filenames in double quotes, like "mbm_assign_control".
Documentation/arch/x86/resctrl.rst with the following files: · line 280
Quoted 7 lines
Documentation/arch/x86/resctrl.rst +7
-0+ in the 'num_mbm_cntrs' file. Changing the mode may cause all counters on + a resource to reset. + + The mode is useful on platforms which support more control and monitor + groups than hardware counters, meaning 'unassigned' control or monitor + groups will report 'Unavailable' or count the traffic in an unpredictable + way.
Note two more instances of "control groups" above.
Please note that the above description implies that counter assignment is per-group. For
example, "specify which control or monitor groups in resctrl should have a counter
assigned" and "useful on platforms which support more control and monitor groups than hardware counters". This needs to be reworked to reflect that counters are assigned to events.
Documentation/arch/x86/resctrl.rst with the following files: · line 284
Quoted 4 lines
Documentation/arch/x86/resctrl.rst +4
-0+ + AMD Platforms with ABMC (Assignable Bandwidth Monitoring Counters) feature + enable this mode by default so that counters remain assigned even when the + corresponding RMID is not in use by any processor.
I assume this should remain RMID since this specifically talks about an x86 system?
Documentation/arch/x86/resctrl.rst with the following files: · line 293
Quoted 9 lines
Documentation/arch/x86/resctrl.rst +9
-0+ + "default": + + By default resctrl assumes each control and monitor group has a hardware + counter. Hardware that does not support 'mbm_cntr_assign' mode will still + allow more control or monitor groups than 'num_rmids' to be created. In + that case reading the mbm_total_bytes and mbm_local_bytes may report + 'Unavailable' if there is no counter associated with that group. +
I reconsidered my earlier suggestion and I believe it needs a correction since counter assignment is not per group:
In default mode resctrl assumes there is a hardware counter for each event within every CTRL_MON and MON group. Reading mbm_total_bytes or mbm_local_bytes may report 'Unavailable' if there is no counter associated with that event.
Please feel free to improve.
Documentation/arch/x86/resctrl.rst with the following files: · line 296
Quoted 3 lines
Documentation/arch/x86/resctrl.rst +0
-0"max_threshold_occupancy": Read/write file provides the largest value (in bytes) at which a previously used LLC_occupancy
The code change looks good to me.
Reinette
Moger, Babu
reviewerRe: [PATCH v8 07/25] x86/resctrl: Introduce the interface to display monitor mode
Hi Reinette
Quoted 22 lines · Reinette Chatre
Hi Babu,
Quoted 19 lines · Babu Moger
Introduce the interface file "
mbm_assign_mode" to list monitor modes supported.The "
mbm_cntr_assign" mode provides the option to assign a counter to an RMID, event pair and monitor the bandwidth as long as it is assigned.On AMD systems "
mbm_cntr_assign" is backed by the ABMC (Assignable Bandwidth Monitoring Counters) hardware feature and is enabled by default.The "default" mode is the existing monitoring mode that works without the explicit counter assignment, instead relying on dynamic counter assignment by hardware that may result in hardware not dedicating a counter resulting in monitoring data reads returning "Unavailable".
Provide an interface to display the monitor mode on the system. $cat
/sys/fs/resctrl/info/L3_MON/mbm_assign_mode[mbm_cntr_assign]
default
Quoted 5 lines
Quoted 2 lines
Switching the
mbm_assign_modewill reset all the MBM counters of all resctrl groups.Please note that this now contradicts the documentation. Perhaps this sentence can just be dropped since there is the documentation within the patch.
Sure. Will drop it.
Quoted 24 lines
Quoted 4 lines
Signed-off-by Babu Moger <[email protected]>---
...
Quoted 17 lines
Documentation/arch/x86/resctrl.rst +9
-0@@ -257,6 +257,40 @@ with the following files: # cat /sys/fs/resctrl/info/L3_MON/mbm_local_bytes_config 0=0x30;1=0x30;3=0x15;4=0x15 +"mbm_assign_mode": + Reports the list of monitoring modes supported. The enclosed brackets + indicate which mode is enabled. + :: + + cat /sys/fs/resctrl/info/L3_MON/mbm_assign_mode + [mbm_cntr_assign] + default +
Quoted 8 lines
Documentation/arch/x86/resctrl.rst with the following files: · line 272
Quoted 4 lines
Documentation/arch/x86/resctrl.rst +4
-0+ "mbm_cntr_assign": + + In mbm_cntr_assign mode user-space is able to specify which control + or monitor groups in resctrl should have a counter assigned using theCounters cannot be assigned to control groups. How about replacing all instances of "control and monitor groups" with "
CTRL_MONand MON groups", similarly "control or monitor groups" with "CTRL_MONor MON groups".
Ok.
Quoted 5 lines
Documentation/arch/x86/resctrl.rst with the following files: · line 273
Quoted 2 lines
Documentation/arch/x86/resctrl.rst +1
-0+ 'mbm_assign_control' file. The number of counters available is describedLooking at the rest of the doc it seems that the custom is actually to place filenames in double quotes, like "
mbm_assign_control".
Sure.
Quoted 11 lines
Documentation/arch/x86/resctrl.rst with the following files: · line 280
Quoted 8 lines
Documentation/arch/x86/resctrl.rst +7
-0+ in the 'num_mbm_cntrs' file. Changing the mode may cause all counters on + a resource to reset. + + The mode is useful on platforms which support more control and monitor + groups than hardware counters, meaning 'unassigned' control or monitor + groups will report 'Unavailable' or count the traffic in an unpredictable + way.Note two more instances of "control groups" above.
Quoted 5 lines
Please note that the above description implies that counter assignment is per-group. For
example, "specify which control or monitor groups in resctrl should have a counter
assigned" and "useful on platforms which support more control and monitor groups than hardware counters". This needs to be reworked to reflect that counters are assigned to events.
How about this?
The mode is useful on platforms which support more CTRL_MON and MON groups than the hardware counters, meaning 'unassigned' events on CTRL_MON or MON groups will report 'Unavailable' or count the traffic in an unpredictable way.
Quoted 7 lines
Documentation/arch/x86/resctrl.rst with the following files: · line 284
Quoted 5 lines
Documentation/arch/x86/resctrl.rst +4
-0+ + AMD Platforms with ABMC (Assignable Bandwidth Monitoring Counters) feature + enable this mode by default so that counters remain assigned even when the + corresponding RMID is not in use by any processor.I assume this should remain RMID since this specifically talks about an x86 system?
This was a suggestion from James. Let me know if you want me to change.
Quoted 14 lines
Documentation/arch/x86/resctrl.rst with the following files: · line 293
Quoted 10 lines
Documentation/arch/x86/resctrl.rst +9
-0+ + "default": + + By default resctrl assumes each control and monitor group has a hardware + counter. Hardware that does not support 'mbm_cntr_assign' mode will still + allow more control or monitor groups than 'num_rmids' to be created. In + that case reading the mbm_total_bytes and mbm_local_bytes may report + 'Unavailable' if there is no counter associated with that group. +I reconsidered my earlier suggestion and I believe it needs a correction since counter assignment is not per group:
Quoted 6 lines
In default mode resctrl assumes there is a hardware counter for each event within every
CTRL_MONand MON group. Readingmbm_total_bytesormbm_local_bytesmay report 'Unavailable' if there is no counter associated with that event.Please feel free to improve.
Looks good.
Quoted 6 lines
Documentation/arch/x86/resctrl.rst with the following files: · line 296
Quoted 4 lines
Documentation/arch/x86/resctrl.rst +0
-0"max_threshold_occupancy": Read/write file provides the largest value (in bytes) at which a previously used LLC_occupancyThe code change looks good to me.
Signature · 2 lines
Thanks Babu Moger
Reinette Chatre
maintainerRe: [PATCH v8 07/25] x86/resctrl: Introduce the interface to display monitor mode
Hi Babu,
Quoted 56 lines · Babu
Quoted 19 lines · Reinette Chatre
Quoted 17 lines · Babu Moger
Documentation/arch/x86/resctrl.rst +9
-0@@ -257,6 +257,40 @@ with the following files: # cat /sys/fs/resctrl/info/L3_MON/mbm_local_bytes_config 0=0x30;1=0x30;3=0x15;4=0x15 +"mbm_assign_mode": + Reports the list of monitoring modes supported. The enclosed brackets + indicate which mode is enabled. + :: + + cat /sys/fs/resctrl/info/L3_MON/mbm_assign_mode + [mbm_cntr_assign] + default +Quoted 8 lines
Documentation/arch/x86/resctrl.rst with the following files: · line 272
Quoted 4 lines
Documentation/arch/x86/resctrl.rst +4
-0+ "mbm_cntr_assign": + + In mbm_cntr_assign mode user-space is able to specify which control + or monitor groups in resctrl should have a counter assigned using theCounters cannot be assigned to control groups. How about replacing all instances of "control and monitor groups" with "
CTRL_MONand MON groups", similarly "control or monitor groups" with "CTRL_MONor MON groups".Ok.
Quoted 5 lines
Documentation/arch/x86/resctrl.rst with the following files: · line 273
Quoted 2 lines
Documentation/arch/x86/resctrl.rst +1
-0+ 'mbm_assign_control' file. The number of counters available is describedLooking at the rest of the doc it seems that the custom is actually to place filenames in double quotes, like "
mbm_assign_control".Sure.
Quoted 11 lines
Documentation/arch/x86/resctrl.rst with the following files: · line 280
Quoted 8 lines
Documentation/arch/x86/resctrl.rst +7
-0+ in the 'num_mbm_cntrs' file. Changing the mode may cause all counters on + a resource to reset. + + The mode is useful on platforms which support more control and monitor + groups than hardware counters, meaning 'unassigned' control or monitor + groups will report 'Unavailable' or count the traffic in an unpredictable + way.Note two more instances of "control groups" above.
Quoted 5 lines
Please note that the above description implies that counter assignment is per-group. For
example, "specify which control or monitor groups in resctrl should have a counter
assigned" and "useful on platforms which support more control and monitor groups than hardware counters". This needs to be reworked to reflect that counters are assigned to events.
Quoted 6 lines
How about this?
The mode is useful on platforms which support more
CTRL_MONand MON groups than the hardware counters, meaning 'unassigned' events onCTRL_MONor MON groups will report 'Unavailable' or count the traffic in an unpredictable way.
This rewrites the second paragraph of the section about "mbm_cntr_assign". It is not clear to me how this section will end up looking since the first paragraph still seems to refer to counters being assigned to groups ("specify which control or monitor groups in resctrl should have a counter assigned") while the later addition to this section by "x86/resctrl: Introduce the interface to switch between monitor modes" starts by specifying how counters are assigned to the MBM events ("The MBM events (mbm_total_bytes and/or mbm_local_bytes) associated counters").
Quoted 8 lines
Quoted 6 lines
Documentation/arch/x86/resctrl.rst with the following files: · line 284
Quoted 4 lines
Documentation/arch/x86/resctrl.rst +4
-0+ + AMD Platforms with ABMC (Assignable Bandwidth Monitoring Counters) feature + enable this mode by default so that counters remain assigned even when the + corresponding RMID is not in use by any processor.I assume this should remain RMID since this specifically talks about an x86 system?
This was a suggestion from James. Let me know if you want me to change.
I can proceed to assume this is a paragraph intended to be x86 specific. No need to change.
Reinette
Babu Moger
reviewerPATCH v8 8/25 x86/resctrl: Introduce interface to display number of monitoring counters
In reply to Babu MogerThe mbm_cntr_assign mode provides an option to the user to assign a counter to an RMID, event pair and monitor the bandwidth as long as the counter is assigned. Number of assignments depend on number of monitoring counters available.
Provide the interface to display the number of monitoring counters supported. The interface file 'num_mbm_cntrs' is available when an architecture supports mbm_cntr_assign mode.
---
v8: Commit message update and documentation update.
v7: Minor commit log text changes.
v6: No changes.
v5: Changed the display name from num_cntrs to num_mbm_cntrs.
Updated the commit message.
Moved the patch after mbm_mode is introduced.
v4: Changed the counter name to num_cntrs. And few text changes.
v3: Changed the field name to mbm_assign_cntrs.
v2: Changed the field name to mbm_assignable_counters from abmc_counte
--- Documentation/arch/x86/resctrl.rst | 4 ++++ arch/x86/kernel/cpu/resctrl/monitor.c | 1 + arch/x86/kernel/cpu/resctrl/rdtgroup.c | 16 ++++++++++++++++ 3 files changed, 21 insertions(+)
Documentation/arch/x86/resctrl.rst +4 -0
@@ -291,6 +291,10 @@ with the following files: that case reading the mbm_total_bytes and mbm_local_bytes may report 'Unavailable' if there is no counter associated with that group. +"num_mbm_cntrs": + The number of monitoring counters available for assignment when the + architecture supports mbm_cntr_assign mode. + "max_threshold_occupancy": Read/write file provides the largest value (in bytes) at which a previously used LLC_occupancy
arch/x86/kernel/cpu/resctrl/monitor.c +1 -0
@@ -1236,6 +1236,7 @@ int __init rdt_get_mon_l3_config(struct rdt_resource *r) r->mon.mbm_cntr_assignable = true;cpuid_count (0x80000020 ,5 , &eax, &ebx, &ecx, &edx); r->mon.num_mbm_cntrs = (ebx &GENMASK (15 ,0 )) +1 ; +resctrl_file_fflags_init ("num_mbm_cntrs" ,RFTYPE_MON_INFO ); } }
arch/x86/kernel/cpu/resctrl/rdtgroup.c +16 -0
@@ -869,6 +869,16 @@ static int rdtgroup_mbm_assign_mode_show(struct kernfs_open_file *of,return 0 ; } +static int rdtgroup_num_mbm_cntrs_show (struct kernfs_open_file *of, +struct seq_file *s,void *v) +{ +struct rdt_resource *r = of->kn->parent->priv; + +seq_printf (s,"%d\n" , r->mon.num_mbm_cntrs); + +return 0 ; +} +#ifdef CONFIG_PROC_CPU_RESCTRL /*
@@ -1940,6 +1950,12 @@ static struct rftype res_common_files[] = { .seq_show = rdtgroup_cpus_show, .fflags = RFTYPE_BASE, }, + { + .name = "num_mbm_cntrs", + .mode = 0444, + .kf_ops = &rdtgroup_kf_single_ops, + .seq_show = rdtgroup_num_mbm_cntrs_show, + }, { .name = "cpus_list", .mode = 0644,
Signature · 1 line
2.34.1
Tony Luck
maintainerRe: [PATCH v8 08/25] x86/resctrl: Introduce interface to display number of monitoring counters
Documentation/arch/x86/resctrl.rst with the following files: · line 296
Quoted 3 lines · Babu Moger
+"num_mbm_cntrs":
Documentation/arch/x86/resctrl.rst +2
-0+ The number of monitoring counters available for assignment when the + architecture supports mbm_cntr_assign mode.
It's not obvious (to me) how these counters work. When I create a group with both local and total monitoring enabled, does that use up two counters (even though I only used up one RMID)?
Are the counters multi-purpose. E.g. if I disable local counting on all groups, are the freed-up counters available for use to count total bandwidth on some additional groups?
From the examples it looks like if there are free counters available when user does mkdir, then they will be assigned to the new rdtgroup. If only one counter is free, does it get assigned to local or total?
Thanks
-Tony
Moger, Babu
reviewerRe: [PATCH v8 08/25] x86/resctrl: Introduce interface to display number of monitoring counters
Hi Tony,
Quoted 8 lines · Tony Luck
Documentation/arch/x86/resctrl.rst with the following files: · line 296
Quoted 3 lines · Babu Moger
+"num_mbm_cntrs":
Documentation/arch/x86/resctrl.rst +2
-0+ The number of monitoring counters available for assignment when the + architecture supports mbm_cntr_assign mode.It's not obvious (to me) how these counters work. When I create a group with both local and total monitoring enabled, does that use up two counters (even though I only used up one RMID)?
That is correct. One RMID can be associated with multiple h/w counters.
Quoted 4 lines
Are the counters multi-purpose. E.g. if I disable local counting on all groups, are the freed-up counters available for use to count total bandwidth on some additional groups?
Yes. That is correct.
With 32 counters you can enable both the events on up to 16 groups.
You can also enable only one event in up to 32 groups.
Quoted 5 lines
Quoted 2 lines
From the examples it looks like if there are free counters
available when user does mkdir, then they will be assigned to the new rdtgroup. If only one counter is free, does it get assigned to local or total?
Right now total event takes a priority.
All good points. How about this text:
"num_mbm_cntrs":
The number of monitoring counters available for assignment when the architecture supports mbm_cntr_assign mode.
Resctrl subsystem provides the interface to count maximum of two memory bandwidth events per group, from a combination of available total and local events. Keeping the current interface, users can enable a maximum of 2 counters per group. User will also have the option to enable only one counter to the group to maximize the number of groups monitored.
Signature · 2 lines
Thanks Babu Moger
Luck, Tony
maintainerRE: [PATCH v8 08/25] x86/resctrl: Introduce interface to display number of monitoring counters
Quoted 11 lines
All good points. How about this text:
"
num_mbm_cntrs":
The number of monitoring counters available for assignment when the architecture supportsmbm_cntr_assignmode.Resctrl subsystem provides the interface to count maximum of two memory bandwidth events per group, from a combination of available total and local events. Keeping the current interface, users can enable a maximum of 2 counters per group. User will also have the option to enable only one counter to the group to maximize the number of groups monitored.
Much better. Looks OK to me.
New questions:
1) Should resctrl provide a file to tell the user how many free counters are available? They can figure it out by counting all the 'l' and 't' in "mbm_assign_control" and subtracting that from "num_mbm_cntrs". But that seems complex.
2) Even more so because free counters might be different per socket if the user did some "0=tl;1=_" assignments as in one of your examples.
Maybe a UI like:
$ cat /sys/fs/resctrl/info/L3_MON/free_mbm_cntrs 0=5;1=9
-Tony
Moger, Babu
reviewerRe: [PATCH v8 08/25] x86/resctrl: Introduce interface to display number of monitoring counters
Hi Tony,
Quoted 6 lines · Tony
Quoted 6 lines
All good points. How about this text:
"
num_mbm_cntrs":
The number of monitoring counters available for assignment when the architecture supportsmbm_cntr_assignmode.
Quoted 7 lines
Quoted 5 lines
Resctrl subsystem provides the interface to count maximum of two memory bandwidth events per group, from a combination of available total and local events. Keeping the current interface, users can enable a maximum of 2 counters per group. User will also have the option to enable only one counter to the group to maximize the number of groups monitored.
Much better. Looks OK to me.
thanks
Quoted 7 lines
New questions:
1) Should resctrl provide a file to tell the user how many free counters are available? They can figure it out by counting all the 'l' and 't' in "
mbm_assign_control" and subtracting that from "num_mbm_cntrs". But that seems complex.
We have the information already in r->mon.mbm_cntr_free_map.
How about adding an extra text while printing num_mbm_cntrs?
$ cat /sys/fs/resctrl/info/L3_MON/num_mbm_cntrs Total 32, Available 16
There are all global counters, we don't differentiate between sockets just like number of CLOSIDs.
Quoted 12 lines
2) Even more so because free counters might be different per socket if the user did some "0=tl;1=_" assignments as in one of your examples.
Maybe a UI like:
$ cat /sys/fs/resctrl/info/L3_MON/free_mbm_cntrs 0=5;1=9-Tony
Signature · 2 lines
Thanks Babu Moger
Luck, Tony
maintainerRE: [PATCH v8 08/25] x86/resctrl: Introduce interface to display number of monitoring counters
Babu,
Quoted 6 lines
We have the information already in
r->mon.mbm_cntr_free_map.How about adding an extra text while printing
num_mbm_cntrs?$ cat /sys/fs/resctrl/info/L3_MON/num_mbm_cntrs Total 32, Available 16
Either that or:
Total 32
Available 16
which looks fractionally simpler to parse. But I don't have strong feelings.
Quoted 2 lines
There are all global counters, we don't differentiate between sockets just like number of CLOSIDs.
Interesting. So there is no real benefit from "0=tl;1=_" ... you are using up two counters, just not reporting them on socket 1.
Why have this complexity in mbm_assign_control syntax?
You could have just {grouppath}/{allocation}
where allocation is one of _, t, l, tl
-Tony
Moger, Babu
reviewerRe: [PATCH v8 08/25] x86/resctrl: Introduce interface to display number of monitoring counters
Hi Tony,
Quoted 4 lines · Tony
Babu,
Quoted 2 lines
We have the information already in
r->mon.mbm_cntr_free_map.
Quoted 9 lines
Quoted 4 lines
How about adding an extra text while printing
num_mbm_cntrs?$ cat /sys/fs/resctrl/info/L3_MON/num_mbm_cntrs Total 32, Available 16Either that or:
Total 32
Available 16
Sure. Fine with me.
Quoted 9 lines
which looks fractionally simpler to parse. But I don't have strong feelings.
Quoted 2 lines
There are all global counters, we don't differentiate between sockets just like number of CLOSIDs.
Interesting. So there is no real benefit from "0=tl;1=_" ... you are using up two counters, just not reporting them on socket 1.
Why have this complexity in
mbm_assign_controlsyntax?
Lets take an example:
$ cat /sys/fs/resctrl/info/L3_MON/num_mbm_cntrs Total 32 Available 30
# cat /sys/fs/resctrl/info/L3_MON/mbm_assign_control //0=tl;1=tl;
Here default group has taken two counters.
# cat /sys/fs/resctrl/info/L3_MON/mbm_assign_control //0=_;1=tl;
Here default group has two counters.
Domain 0 does not have counters applied. So, you wont be able to read the MBM values for domain 0.
Domain 1 has both the counters applied.
Domain level application is important.
This is similar to what we have with schemata. You can change the value in each individual domain.
#cat schemata
MB:0=2048;1=2048;2=2048;3=2048
L3:0=ffff;1=ffff;2=ffff;3=ffffQuoted 6 lines
You could have just {grouppath}/{allocation}
where allocation is one of _, t, l, tl
-Tony
Signature · 2 lines
Thanks Babu Moger
Luck, Tony
maintainerRE: [PATCH v8 08/25] x86/resctrl: Introduce interface to display number of monitoring counters
Quoted 12 lines
Quoted 1 line
Why have this complexity in
mbm_assign_controlsyntax?Lets take an example:
$ cat /sys/fs/resctrl/info/L3_MON/num_mbm_cntrs Total 32 Available 30# cat /sys/fs/resctrl/info/L3_MON/mbm_assign_control //0=tl;1=tl;Here default group has taken two counters.
Quoted 7 lines
# cat /sys/fs/resctrl/info/L3_MON/mbm_assign_control //0=_;1=tl;Here default group has two counters.
Domain 0 does not have counters applied. So, you wont be able to read the MBM values for domain 0.
Domain 1 has both the counters applied.
Is there some benefit from doing this? You are still using the same number of counters. You now can't read them from domain 0.
You said the counters are system-wide. Does that mean that in this case:
# cat /sys/fs/resctrl/info/L3_MON/mbm_assign_control //0=tl;1=tl;
there aren't separate counts from each of domain 0 and domain 1. I.e. if I read both I'd see the same value (sum of traffic on both domains):
$ grep . /sys/fs/resctrl/mon_data/*/*total* /sys/fs/resctrl /mon_data/mon_L3_00/mbm_total_bytes:260039467008 /sys/fs/resctrl /mon_data/mon_L3_01/mbm_total_bytes:260039467008
-Tony
Moger, Babu
reviewerRe: [PATCH v8 08/25] x86/resctrl: Introduce interface to display number of monitoring counters
Hi Tony,
Quoted 20 lines · Tony
Quoted 12 lines
Quoted 1 line
Why have this complexity in
mbm_assign_controlsyntax?Lets take an example:
$ cat /sys/fs/resctrl/info/L3_MON/num_mbm_cntrs Total 32 Available 30# cat /sys/fs/resctrl/info/L3_MON/mbm_assign_control //0=tl;1=tl;Here default group has taken two counters.
Quoted 7 lines
# cat /sys/fs/resctrl/info/L3_MON/mbm_assign_control //0=_;1=tl;Here default group has two counters.
Domain 0 does not have counters applied. So, you wont be able to read the MBM values for domain 0.
Domain 1 has both the counters applied.
Quoted 5 lines
Is there some benefit from doing this? You are still using the same number of counters. You now can't read them from domain 0.
You said the counters are system-wide. Does that mean that in this case:
Counter are system wide. We also keep track if the counters is applied to specific domain or not. We have two bitmaps to keep track of this.
There is a cost to applying counter to the domain(IPI needs to be sent to the domain).
Quoted 5 lines
# cat /sys/fs/resctrl/info/L3_MON/mbm_assign_control //0=tl;1=tl;there aren't separate counts from each of domain 0 and domain 1.
Yes. There is. Each domain has its own count. I am not sure about your config.
# cat /sys/fs/resctrl/info/L3_MON/mbm_assign_control //0=_;1=tl;
# grep . /sys/fs/resctrl/mon_data/*/*total* /sys/fs/resctrl/mon_data/mon_L3_00/mbm_total_bytes:Unassigned /sys/fs/resctrl/mon_data/mon_L3_01/mbm_total_bytes:22976
Quoted 7 lines
I.e. if I read both I'd see the same value (sum of traffic on both domains):
$ grep . /sys/fs/resctrl/mon_data/*/*total* /sys/fs/resctrl /mon_data/mon_L3_00/mbm_total_bytes:260039467008 /sys/fs/resctrl /mon_data/mon_L3_01/mbm_total_bytes:260039467008-Tony
Signature · 2 lines
Thanks Babu Moger
Tony Luck
maintainerRe: [PATCH v8 08/25] x86/resctrl: Introduce interface to display number of monitoring counters
Quoted 6 lines · Babu
Quoted 4 lines
# cat /sys/fs/resctrl/info/L3_MON/mbm_assign_control //0=tl;1=tl;there aren't separate counts from each of domain 0 and domain 1.
Yes. There is. Each domain has its own count. I am not sure about your config.
I've been reading the code and see better now.
There are a bunch (32) of counters per domain.
But you have a system-wide allocator. So when making a group you may allocate counters 2 and 3 for total and local respectively. Then configure the local instance of counter 2 on each domain (recording that in the per-domain bitmap) for total bandwidth. Ditto for counter 3 instances on each domain.
If the user updates the configuration to stop counting on domain 1. Then the per-domain bitmap is updated to show counters 2 and 3 are no longer in use on this domain. But those counters aren't freed (because domain 0 is still using them).
Is there some hardware limitation that would prevent re-using domain 1 counters 2 & 3 for some other group (RMID)?
Or is this just a s/w implementation detail because you have a system wide allocator for counters?
-Tony
Re: [PATCH v8 08/25] x86/resctrl: Introduce interface to display number of monitoring counters
Hi Tony,
Quoted 10 lines · Tony Luck
Quoted 6 lines · Babu
Quoted 4 lines
# cat /sys/fs/resctrl/info/L3_MON/mbm_assign_control //0=tl;1=tl;there aren't separate counts from each of domain 0 and domain 1.
Yes. There is. Each domain has its own count. I am not sure about your config.
I've been reading the code and see better now.
Quoted 8 lines
There are a bunch (32) of counters per domain.
But you have a system-wide allocator. So when making a group you may allocate counters 2 and 3 for total and local respectively. Then configure the local instance of counter 2 on each domain (recording that in the per-domain bitmap) for total bandwidth. Ditto for counter 3 instances on each domain.
Yes. That is correct.
Quoted 6 lines
If the user updates the configuration to stop counting on domain 1. Then the per-domain bitmap is updated to show counters 2 and 3 are no longer in use on this domain. But those counters aren't freed (because domain 0 is still using them).
Yes. Correct.
Quoted 7 lines
Is there some hardware limitation that would prevent re-using domain 1 counters 2 & 3 for some other group (RMID)?
Or is this just a s/w implementation detail because you have a system wide allocator for counters?
There is no hardware limitation. It is how resctrl is designed.
In case of Intel(with two sockets, 16 CLOSIDs), You can only create 16 groups. Each group will have two domains(domain 0 for socket 0 and domain 1 for socket 1).
# cat schemata
MB:0=100;1=100
L3:0=ffff;1=ffff;We may have to think of addressing this sometime in the future.
Signature · 1 line
- Babu Moger
Tony Luck
maintainerRe: [PATCH v8 08/25] x86/resctrl: Introduce interface to display number of monitoring counters
Quoted 12 lines · Babu
Quoted 6 lines
Is there some hardware limitation that would prevent re-using domain 1 counters 2 & 3 for some other group (RMID)?
Or is this just a s/w implementation detail because you have a system wide allocator for counters?
There is no hardware limitation. It is how resctrl is designed. In case of
Intel(with two sockets, 16 CLOSIDs), You can only create 16 groups. Each group will have two domains(domain 0 for socket 0 and domain 1 for socket 1).
Quoted 6 lines
# cat schemata MB:0=100;1=100 L3:0=ffff;1=ffff;We may have to think of addressing this sometime in the future.
In this example, the hardware would support using the instances of counters 2 & 3 on socket 1 for a different group (RMID). But your code doesn't alllow it because the instances of counters 2 & 3 are active on socket 0.
If you had a separate counter allocation pool for each domain you would not have this limitation. When counters 2 & 3 are freed on domain 1, they could be allocated to the domain 1 element of some other group.
Maybe that isn't an interesting use case, so not worth doing?
But if that is the goal, then there is no benefit in having /sys/fs/resctrl/info/L3_MON/mbm_assign_control allow different domains to choose different counter allocation policies.
E.g. in this example from Documentation:
/child_default_mon_grp/0=tl;1=l;
This group allocated two counters (because domain 0 is counting both total and local). Domain 1 is only counting local, but that means a counter on domain 1 is sitting idle. It can't be used because the matching counter is active on domain 0.
I.e. the user who chose this simply gave up being able to read total bandwidth on domain 1, but didn't get an extra counter in exchange for this sacrifice. That doesn't seem like a good deal.
I see two options for improvement:
1) Implement per-domain allocation of counters. Then a counter freed in a domain becomes available for use in that domain for other groups.
2) Go all-in on the global counter model and simplify the syntax of mbm_assign_control to allocate the same counters in all domains. That would simplify the parsing code.
-Tony
Reinette Chatre
maintainerRe: [PATCH v8 08/25] x86/resctrl: Introduce interface to display number of monitoring counters
Hi Tony,
Quoted 6 lines · Tony Luck
On Fri, Oct 11, 2024 at 03:49:48PM -0500, Moger, Babu wrote:
I.e. the user who chose this simply gave up being able to read total bandwidth on domain 1, but didn't get an extra counter in exchange for this sacrifice. That doesn't seem like a good deal.
As Babu mentioned earlier, this seems equivalent to the existing CLOSid management. For example, if a user assigns only CPUs from one domain to a resource group, it does not free up the CLOSID to create a new resource group dedicated to other domain(s).
Reinette
Moger, Babu
reviewerRe: [PATCH v8 08/25] x86/resctrl: Introduce interface to display number of monitoring counters
Quoted 10 lines · On 10/14/24 11:46, wrote:
Hi Tony,
Quoted 6 lines · Tony Luck
On Fri, Oct 11, 2024 at 03:49:48PM -0500, Moger, Babu wrote:
I.e. the user who chose this simply gave up being able to read total bandwidth on domain 1, but didn't get an extra counter in exchange for this sacrifice. That doesn't seem like a good deal.
Quoted 5 lines
As Babu mentioned earlier, this seems equivalent to the existing CLOSid management. For example, if a user assigns only CPUs from one domain to a resource group, it does not free up the CLOSID to create a new resource group dedicated to other domain(s).
Thanks for the confirmation here.
I was wondering if this works differently on Intel. I was trying to figure out on 2 socket intel system if we can create two separate resctrl groups sharing the same CLOSID (one group using CLOSID 1 on socket 0 and another group CLOSID 1 socket 1). No. We cannot do that.
Even though hardware supports separate allocation for each domain, resctrl design does not support that.
Signature · 2 lines
Thanks Babu Moger
Luck, Tony
maintainerRE: [PATCH v8 08/25] x86/resctrl: Introduce interface to display number of monitoring counters
Quoted 9 lines
Quoted 9 lines
Quoted 4 lines
I.e. the user who chose this simply gave up being able to read total bandwidth on domain 1, but didn't get an extra counter in exchange for this sacrifice. That doesn't seem like a good deal.
As Babu mentioned earlier, this seems equivalent to the existing CLOSid management. For example, if a user assigns only CPUs from one domain to a resource group, it does not free up the CLOSID to create a new resource group dedicated to other domain(s).
I hadn't considered the case where a user is assigning CPUs to resctrl groups instead of assigning tasks. With that context this makes sense to me now. Thanks.
Quoted 9 lines
Thanks for the confirmation here.
I was wondering if this works differently on Intel. I was trying to figure out on 2 socket intel system if we can create two separate resctrl groups sharing the same CLOSID (one group using CLOSID 1 on socket 0 and another group CLOSID 1 socket 1). No. We cannot do that.
Even though hardware supports separate allocation for each domain, resctrl design does not support that.
So CLOSIDs and counters are blanket assigned across all domains. I understand that now.
Back to my question of why complicate code and resctrl files by providing a mechanism to enable event counters differently per-domain.
"0=tl;1=_" requires allocation of the same counters as "0=tl;1=tl" or "0=t;1=l"
What advantage does it have over skipping the per-domain list and just providing a single value for all domains? You clearly expect this will be a common user request since you implemented the "*" means apply to all domains.
-Tony
Moger, Babu
reviewerRe: [PATCH v8 08/25] x86/resctrl: Introduce interface to display number of monitoring counters
Hi Tony,
Quoted 25 lines · Tony
Quoted 9 lines
Quoted 9 lines
Quoted 4 lines
I.e. the user who chose this simply gave up being able to read total bandwidth on domain 1, but didn't get an extra counter in exchange for this sacrifice. That doesn't seem like a good deal.
As Babu mentioned earlier, this seems equivalent to the existing CLOSid management. For example, if a user assigns only CPUs from one domain to a resource group, it does not free up the CLOSID to create a new resource group dedicated to other domain(s).
I hadn't considered the case where a user is assigning CPUs to resctrl groups instead of assigning tasks. With that context this makes sense to me now. Thanks.
Quoted 9 lines
Thanks for the confirmation here.
I was wondering if this works differently on Intel. I was trying to figure out on 2 socket intel system if we can create two separate resctrl groups sharing the same CLOSID (one group using CLOSID 1 on socket 0 and another group CLOSID 1 socket 1). No. We cannot do that.
Even though hardware supports separate allocation for each domain, resctrl design does not support that.
Quoted 8 lines
So CLOSIDs and counters are blanket assigned across all domains. I understand that now.
Back to my question of why complicate code and resctrl files by providing a mechanism to enable event counters differently per-domain.
"0=tl;1=_" requires allocation of the same counters as "0=tl;1=tl" or "0=t;1=l"
Yes. That is correct.
Quoted 6 lines
What advantage does it have over skipping the per-domain list and just providing a single value for all domains? You clearly expect this will be a common user request since you implemented the "*" means apply to all domains.
We started with a global assignment by applying assignment across all the domains initially.
But we wanted give a generic approach which allows both the options(domain specific assignment and global assignment with '*"). It is also matches with other managements (RMID/CLOSID management) we are doing in resctrl right now. Also, there is an extra IPI for each domain if user is only interested in on domain.
Some of the discussions are here.
https://lore.kernel.org/lkml/f7dac996d87b4144e4c786178a7fd3d218eaebe8.1711674410.git.babu.moger@amd.com/#r
Thanks
Babu Moger
Luck, Tony
maintainerRE: [PATCH v8 08/25] x86/resctrl: Introduce interface to display number of monitoring counters
Quoted 9 lines
Quoted 5 lines
What advantage does it have over skipping the per-domain list and just providing a single value for all domains? You clearly expect this will be a common user request since you implemented the "*" means apply to all domains.
We started with a global assignment by applying assignment across all the domains initially.
Quoted 8 lines
But we wanted give a generic approach which allows both the
options(domain specific assignment and global assignment with '*"). It is also matches with other managements (RMID/CLOSID management) we are doing in resctrl right now. Also, there is an extra IPI for each domain if user is only interested in on domain.Some of the discussions are here.
https://lore.kernel.org/lkml/f7dac996d87b4144e4c786178a7fd3d218eaebe8.1711674410.git.babu.moger@amd.com/#r
My summary of that:
Peter: Complex, don't need per-domain.
Reinette: Maybe some architecture might want per-domain.
Since you seem to want to keep the flexibility for a possible future where per-domain is needed. The "available_mbm_cntrs" file suggested in another thread would need to list available counters on each domain to avoid ABI problems should that future arrive.
$ cat num_mbm_counters 32
$ cat available_mbm_cntrs 0=12;1=9
Current implementation would show same number for all domains.
-Tony
Reinette Chatre
maintainerRe: [PATCH v8 08/25] x86/resctrl: Introduce interface to display number of monitoring counters
Hi Tony,
Quoted 15 lines · Tony
Quoted 15 lines
Quoted 5 lines
What advantage does it have over skipping the per-domain list and just providing a single value for all domains? You clearly expect this will be a common user request since you implemented the "*" means apply to all domains.
We started with a global assignment by applying assignment across all the domains initially.
But we wanted give a generic approach which allows both the
options(domain specific assignment and global assignment with '*"). It is also matches with other managements (RMID/CLOSID management) we are doing in resctrl right now. Also, there is an extra IPI for each domain if user is only interested in on domain.
Quoted 7 lines
Quoted 2 lines
Some of the discussions are here.
https://lore.kernel.org/lkml/f7dac996d87b4144e4c786178a7fd3d218eaebe8.1711674410.git.babu.moger@amd.com/#rMy summary of that:
Peter: Complex, don't need per-domain.
Reinette: Maybe some architecture might want per-domain.
To be specific ... we already have an architecture that supports per-domain:
AMD's ABMC. When I considered the lifetime of user interfaces (forever?) while knowing that ABMC does indeed support per-domain counter assignment it seems a good precaution for the user interface to support that, even if the first implementation does not.
There are two parts to this work: (a) the new user interface
and (b) support for ABMC. I believe that the user interface has to be flexible to support all ABMC features that users may want to take advantage of, even if the first implementation does not enable those features. In addition, the user interface should support future usages that we know if, "soft-ABMC" and MPAM.
I do not think that we should require all implementations to support everything made possible by user interface though. As I mentioned in that thread [1] I do think that the user _interface_ needs to be flexible by supporting domain level counter assignment, but that it may be possible that the _implementation_ only supports assignment to '*' domain values.
I thus do not think we should simplify the syntax of mbm_assign_control, but I also do not think we should require that all implementations support all that the syntax makes possible.
Quoted 10 lines
Since you seem to want to keep the flexibility for a possible future where per-domain is needed. The "
available_mbm_cntrs" file suggested in another thread would need to list available counters on each domain to avoid ABI problems should that future arrive.$ cat num_mbm_counters 32$ cat available_mbm_cntrs 0=12;1=9
Good point.
Quoted 3 lines
Current implementation would show same number for all domains.
Reinette
Moger, Babu
reviewerRe: [PATCH v8 08/25] x86/resctrl: Introduce interface to display number of monitoring counters
Hi Reinette/Tony,
Quoted 54 lines · On 10/14/24 15:05, wrote:
Hi Tony,
Quoted 15 lines · Tony
Quoted 15 lines
Quoted 5 lines
What advantage does it have over skipping the per-domain list and just providing a single value for all domains? You clearly expect this will be a common user request since you implemented the "*" means apply to all domains.
We started with a global assignment by applying assignment across all the domains initially.
But we wanted give a generic approach which allows both the
options(domain specific assignment and global assignment with '*"). It is also matches with other managements (RMID/CLOSID management) we are doing in resctrl right now. Also, there is an extra IPI for each domain if user is only interested in on domain.Quoted 7 lines
Quoted 2 lines
Some of the discussions are here.
https://lore.kernel.org/lkml/f7dac996d87b4144e4c786178a7fd3d218eaebe8.1711674410.git.babu.moger@amd.com/#rMy summary of that:
Peter: Complex, don't need per-domain.
Reinette: Maybe some architecture might want per-domain.To be specific ... we already have an architecture that supports per-domain:
AMD's ABMC. When I considered the lifetime of user interfaces (forever?) while knowing that ABMC does indeed support per-domain counter assignment it seems a good precaution for the user interface to support that, even if the first implementation does not.There are two parts to this work: (a) the new user interface
and (b) support for ABMC. I believe that the user interface has to be flexible to support all ABMC features that users may want to take advantage of, even if the first implementation does not enable those features. In addition, the user interface should support future usages that we know if, "soft-ABMC" and MPAM.I do not think that we should require all implementations to support everything made possible by user interface though. As I mentioned in that thread [1] I do think that the user _interface_ needs to be flexible by supporting domain level counter assignment, but that it may be possible that the _implementation_ only supports assignment to '*' domain values.
I thus do not think we should simplify the syntax of
mbm_assign_control, but I also do not think we should require that all implementations support all that the syntax makes possible.Quoted 5 lines
Since you seem to want to keep the flexibility for a possible future where per-domain is needed. The "
available_mbm_cntrs" file suggested in another thread would need to list available counters on each domain to avoid ABI problems should that future arrive.
Quoted 7 lines
Quoted 5 lines
$ cat num_mbm_counters 32$ cat available_mbm_cntrs 0=12;1=9Good point.
Ok. Will add it.
Thanks
Babu Moger
Moger, Babu
reviewerRe: [PATCH v8 08/25] x86/resctrl: Introduce interface to display number of monitoring counters
In reply to Reinette ChatreHi Reinette/Tony,
Quoted 54 lines · On 10/14/24 15:05, wrote:
Hi Tony,
Quoted 15 lines · Tony
Quoted 15 lines
Quoted 5 lines
What advantage does it have over skipping the per-domain list and just providing a single value for all domains? You clearly expect this will be a common user request since you implemented the "*" means apply to all domains.
We started with a global assignment by applying assignment across all the domains initially.
But we wanted give a generic approach which allows both the
options(domain specific assignment and global assignment with '*"). It is also matches with other managements (RMID/CLOSID management) we are doing in resctrl right now. Also, there is an extra IPI for each domain if user is only interested in on domain.Quoted 7 lines
Quoted 2 lines
Some of the discussions are here.
https://lore.kernel.org/lkml/f7dac996d87b4144e4c786178a7fd3d218eaebe8.1711674410.git.babu.moger@amd.com/#rMy summary of that:
Peter: Complex, don't need per-domain.
Reinette: Maybe some architecture might want per-domain.To be specific ... we already have an architecture that supports per-domain:
AMD's ABMC. When I considered the lifetime of user interfaces (forever?) while knowing that ABMC does indeed support per-domain counter assignment it seems a good precaution for the user interface to support that, even if the first implementation does not.There are two parts to this work: (a) the new user interface
and (b) support for ABMC. I believe that the user interface has to be flexible to support all ABMC features that users may want to take advantage of, even if the first implementation does not enable those features. In addition, the user interface should support future usages that we know if, "soft-ABMC" and MPAM.I do not think that we should require all implementations to support everything made possible by user interface though. As I mentioned in that thread [1] I do think that the user _interface_ needs to be flexible by supporting domain level counter assignment, but that it may be possible that the _implementation_ only supports assignment to '*' domain values.
I thus do not think we should simplify the syntax of
mbm_assign_control, but I also do not think we should require that all implementations support all that the syntax makes possible.Quoted 5 lines
Since you seem to want to keep the flexibility for a possible future where per-domain is needed. The "
available_mbm_cntrs" file suggested in another thread would need to list available counters on each domain to avoid ABI problems should that future arrive.
Quoted 8 lines
Quoted 5 lines
$ cat num_mbm_counters 32$ cat available_mbm_cntrs 0=12;1=9Good point.
Working on this now. Wanted to confirm if we really need domain specific information?
To me, it does not seem necessary for the user. User cannot make any decisions based on this information.
All user wants to know is if there are global counters available.
$ cat num_mbm_counters 32
$ cat available_mbm_cntrs 15
Signature · 2 lines
Thanks Babu Moger
Luck, Tony
maintainerRE: [PATCH v8 08/25] x86/resctrl: Introduce interface to display number of monitoring counters
Quoted 20 lines
Quoted 5 lines
Quoted 5 lines
Since you seem to want to keep the flexibility for a possible future where per-domain is needed. The "
available_mbm_cntrs" file suggested in another thread would need to list available counters on each domain to avoid ABI problems should that future arrive.Quoted 8 lines
Quoted 5 lines
$ cat num_mbm_counters 32$ cat available_mbm_cntrs 0=12;1=9Good point.
Working on this now. Wanted to confirm if we really need domain specific information?
To me, it does not seem necessary for the user. User cannot make any decisions based on this information.
Quoted 7 lines
All user wants to know is if there are global counters available.
$ cat num_mbm_counters 32$ cat available_mbm_cntrs 15
This approach paints resctrl into an ABI corner where it can't later update resctrl to track counters per-domain. Maybe you'll never want to do that, but some other architecture might want to have that flexibility.
-Tony
Re: RE: [PATCH v8 08/25] x86/resctrl: Introduce interface to display number of monitoring counters
Hi Tony,
Quoted 25 lines · Tony
Quoted 25 lines
Quoted 5 lines
Quoted 5 lines
Since you seem to want to keep the flexibility for a possible future where per-domain is needed. The "
available_mbm_cntrs" file suggested in another thread would need to list available counters on each domain to avoid ABI problems should that future arrive.Quoted 8 lines
Quoted 5 lines
$ cat num_mbm_counters 32$ cat available_mbm_cntrs 0=12;1=9Good point.
Working on this now. Wanted to confirm if we really need domain specific information?
To me, it does not seem necessary for the user. User cannot make any decisions based on this information.
All user wants to know is if there are global counters available.
$ cat num_mbm_counters 32
Quoted 6 lines
Quoted 2 lines
$ cat available_mbm_cntrs 15This approach paints resctrl into an ABI corner where it can't later update resctrl to track counters per-domain. Maybe you'll never want to do that, but some other architecture might want to have that flexibility.
Ok. Fine. Lets keep the per-domain counters.
Signature · 1 line
- Babu Moger
Reinette Chatre
maintainerRe: [PATCH v8 08/25] x86/resctrl: Introduce interface to display number of monitoring counters
In reply to Moger, BabuHi Tony and Babu,
Quoted 12 lines · Babu
Hi Tony,
Quoted 9 lines · Tony
Babu,
Quoted 6 lines
We have the information already in
r->mon.mbm_cntr_free_map.How about adding an extra text while printing
num_mbm_cntrs?$ cat /sys/fs/resctrl/info/L3_MON/num_mbm_cntrs Total 32, Available 16
Quoted 6 lines
Quoted 4 lines
Either that or:
Total 32
Available 16Sure. Fine with me.
I think separate files would be easier to parse and matches the existing resctrl interface in this regard. How about "available_mbm_cntrs"?
Reinette
Moger, Babu
reviewerRe: [PATCH v8 08/25] x86/resctrl: Introduce interface to display number of monitoring counters
Hi Reinette,
Quoted 20 lines · On 10/14/24 11:59, wrote:
Hi Tony and Babu,
Quoted 17 lines · Babu
Hi Tony,
Quoted 13 lines · Tony
Babu,
Quoted 6 lines
We have the information already in
r->mon.mbm_cntr_free_map.How about adding an extra text while printing
num_mbm_cntrs?$ cat /sys/fs/resctrl/info/L3_MON/num_mbm_cntrs Total 32, Available 16Either that or:
Total 32
Available 16
Quoted 4 lines
Quoted 1 line
Sure. Fine with me.
I think separate files would be easier to parse and matches the existing resctrl interface in this regard. How about "
available_mbm_cntrs"?
Sure.
Thanks
Babu Moger
Reinette Chatre
maintainerRe: [PATCH v8 08/25] x86/resctrl: Introduce interface to display number of monitoring counters
In reply to Moger, BabuHi Babu and Tony,
Quoted 8 lines · Babu
All good points. How about this text:
"
num_mbm_cntrs":
The number of monitoring counters available for assignment when the architecture supportsmbm_cntr_assignmode.Resctrl subsystem provides the interface to count maximum of two memory
subsystem -> filesystem
Quoted 1 line
bandwidth events per group, from a combination of available total and
Is this "from a combination of ..." snippet intended to hint at BMEC?
Quoted 1 line
local events. Keeping the current interface, users can enable a maximum of
What is meant by "Keeping the current interface"? Which interface? What will "current" mean when a user reads this documentation?
Quoted 1 line
2 counters per group. User will also have the option to enable only one
"User will also have" is talking about the future. When will this be the case?
Quoted 3 lines
counter to the group to maximize the number of groups monitored.
I think that we need to be careful when making this documentation so specific to the ABMC implementation. We already know that "soft-ABMC" is coming and Peter already shared [1] that with software assignment it will not be possible to assign counters to individual events.
The goal of this work is to create a generic interface and this is the documentation for it. If this documentation is created to be specific to the first implementation it will make it difficult to use this same interface to support other implementations.
Reinette
[1] https://lore.kernel.org/all/CALPaoCi_TBZnULHQpYns+H+30jODZvyQpUHJRDHNwjQzajrD=A@mail.gmail.com/
Moger, Babu
reviewerRe: [PATCH v8 08/25] x86/resctrl: Introduce interface to display number of monitoring counters
Hi Reinette,
Quoted 6 lines · Reinette Chatre
Hi Babu and Tony,
Quoted 3 lines · Babu
All good points. How about this text:
Quoted 7 lines
Quoted 5 lines
"
num_mbm_cntrs":
The number of monitoring counters available for assignment when the architecture supportsmbm_cntr_assignmode.Resctrl subsystem provides the interface to count maximum of two memory
subsystem -> filesystem
Sure.
Quoted 4 lines
Quoted 2 lines
bandwidth events per group, from a combination of available total and
Is this "from a combination of ..." snippet intended to hint at BMEC?
No. We support 2 MBM events right now. That is why I added combination of total and local. I can remove that text.
Quoted 5 lines
Quoted 2 lines
local events. Keeping the current interface, users can enable a maximum of
What is meant by "Keeping the current interface"? Which interface? What will "current" mean when a user reads this documentation?
I meant not to change any interface to support mbm_cntrl_assign feature.
Quoted 4 lines
Quoted 2 lines
2 counters per group. User will also have the option to enable only one
"User will also have" is talking about the future. When will this be the case?
Again.. will have change the text here.
Quoted 10 lines
Quoted 4 lines
counter to the group to maximize the number of groups monitored.
I think that we need to be careful when making this documentation so specific to the ABMC implementation. We already know that "soft-ABMC" is coming and Peter already shared [1] that with software assignment it will not be possible to assign counters to individual events.
Quoted 5 lines
The goal of this work is to create a generic interface and this is the documentation for it. If this documentation is created to be specific to the first implementation it will make it difficult to use this same interface to support other implementations.
Agree.
How about this?
"num_mbm_cntrs":
The number of monitoring counters available for assignment when the architecture supports mbm_cntr_assign mode.
The resctrl filesystem allows user track up to two memory bandwidth events per group, using a mix of total and local events. Users can enable up to 2 counters per group. There's also an option to enable just one counter per group, which allows monitoring more groups.
Signature · 2 lines
Thanks Babu Moger
Reinette Chatre
maintainerRe: [PATCH v8 08/25] x86/resctrl: Introduce interface to display number of monitoring counters
Hi Babu,
Quoted 58 lines · Babu
Quoted 11 lines · Reinette Chatre
Quoted 8 lines · Babu
All good points. How about this text:
"
num_mbm_cntrs":
The number of monitoring counters available for assignment when the architecture supportsmbm_cntr_assignmode.Resctrl subsystem provides the interface to count maximum of two memory
subsystem -> filesystem
Sure.
Quoted 4 lines
Quoted 2 lines
bandwidth events per group, from a combination of available total and
Is this "from a combination of ..." snippet intended to hint at BMEC?
No. We support 2 MBM events right now. That is why I added combination of total and local. I can remove that text.
Quoted 5 lines
Quoted 2 lines
local events. Keeping the current interface, users can enable a maximum of
What is meant by "Keeping the current interface"? Which interface? What will "current" mean when a user reads this documentation?
I meant not to change any interface to support
mbm_cntrl_assignfeature.Quoted 4 lines
Quoted 2 lines
2 counters per group. User will also have the option to enable only one
"User will also have" is talking about the future. When will this be the case?
Again.. will have change the text here.
Quoted 10 lines
Quoted 4 lines
counter to the group to maximize the number of groups monitored.
I think that we need to be careful when making this documentation so specific to the ABMC implementation. We already know that "soft-ABMC" is coming and Peter already shared [1] that with software assignment it will not be possible to assign counters to individual events.
Quoted 5 lines
The goal of this work is to create a generic interface and this is the documentation for it. If this documentation is created to be specific to the first implementation it will make it difficult to use this same interface to support other implementations.
Agree.
How about this?
Quoted 6 lines
"
num_mbm_cntrs":
The number of monitoring counters available for assignment when the architecture supportsmbm_cntr_assignmode.The resctrl filesystem allows user track up to two memory bandwidth events per group, using a mix of total and local events. Users can enable up to 2
"a mix of" remains unclear to me since there are only two options. I think we can be specific here.
Quoted 3 lines
counters per group. There's also an option to enable just one counter per group, which allows monitoring more groups.
How about below for the second paragraph:
The resctrl filesystem supports tracking up to two memory bandwidth events per monitoring group: mbm_total_bytes and/or mbm_local_bytes. Up to two counters can be assigned per monitoring group, one for each memory bandwidth event. More monitoring groups can be tracked by assigning one counter per monitoring group. However, doing so limits memory bandwidth tracking to a single memory bandwidth event per monitoring group.
Reinette
Moger, Babu
reviewerRe: [PATCH v8 08/25] x86/resctrl: Introduce interface to display number of monitoring counters
Hi Reinette,
Quoted 77 lines · Reinette Chatre
Hi Babu,
Quoted 58 lines · Babu
Quoted 11 lines · Reinette Chatre
Quoted 8 lines · Babu
All good points. How about this text:
"
num_mbm_cntrs":
The number of monitoring counters available for assignment when the architecture supportsmbm_cntr_assignmode.Resctrl subsystem provides the interface to count maximum of two memory
subsystem -> filesystem
Sure.
Quoted 4 lines
Quoted 2 lines
bandwidth events per group, from a combination of available total and
Is this "from a combination of ..." snippet intended to hint at BMEC?
No. We support 2 MBM events right now. That is why I added combination of total and local. I can remove that text.
Quoted 5 lines
Quoted 2 lines
local events. Keeping the current interface, users can enable a maximum of
What is meant by "Keeping the current interface"? Which interface? What will "current" mean when a user reads this documentation?
I meant not to change any interface to support
mbm_cntrl_assignfeature.Quoted 4 lines
Quoted 2 lines
2 counters per group. User will also have the option to enable only one
"User will also have" is talking about the future. When will this be the case?
Again.. will have change the text here.
Quoted 10 lines
Quoted 4 lines
counter to the group to maximize the number of groups monitored.
I think that we need to be careful when making this documentation so specific to the ABMC implementation. We already know that "soft-ABMC" is coming and Peter already shared [1] that with software assignment it will not be possible to assign counters to individual events.
Quoted 5 lines
The goal of this work is to create a generic interface and this is the documentation for it. If this documentation is created to be specific to the first implementation it will make it difficult to use this same interface to support other implementations.
Agree.
How about this?
Quoted 6 lines
"
num_mbm_cntrs":
The number of monitoring counters available for assignment when the architecture supportsmbm_cntr_assignmode.The resctrl filesystem allows user track up to two memory bandwidth events per group, using a mix of total and local events. Users can enable up to 2
"a mix of" remains unclear to me since there are only two options. I think we can be specific here.
Quoted 3 lines
counters per group. There's also an option to enable just one counter per group, which allows monitoring more groups.
How about below for the second paragraph:
Quoted 8 lines
The resctrl filesystem supports tracking up to two memory bandwidth events per monitoring group:
mbm_total_bytesand/ormbm_local_bytes. Up to two counters can be assigned per monitoring group, one for each memory bandwidth event. More monitoring groups can be tracked by assigning one counter per monitoring group. However, doing so limits memory bandwidth tracking to a single memory bandwidth event per monitoring group.
Sure. Looks good.
Signature · 2 lines
Thanks Babu Moger
Babu Moger
reviewerPATCH v8 9/25 x86/resctrl: Add __init attribute to dom_data_init()
In reply to Babu Mogerdom_data_init() is only called during the __init sequence. Add __init attribute like the rest of call sequence.
While at it, pass 'struct rdt_resource' to dom_data_init() and dom_data_exit() which will be used for mbm counter __init and__exit call sequence.
---
v8: New patch.
--- arch/x86/kernel/cpu/resctrl/core.c | 2 +- arch/x86/kernel/cpu/resctrl/internal.h | 2 +- arch/x86/kernel/cpu/resctrl/monitor.c | 8 ++++---- 3 files changed, 6 insertions(+), 6 deletions(-)
arch/x86/kernel/cpu/resctrl/core.c +1 -1
@@ -1140,7 +1140,7 @@ static void __exit resctrl_exit(void)rdtgroup_exit ();if (r->mon_capable) -rdt_put_mon_l3_config (); +rdt_put_mon_l3_config (r); }__exitcall (resctrl_exit);
arch/x86/kernel/cpu/resctrl/internal.h +1 -1
@@ -633,7 +633,7 @@ void closid_free(int closid);int alloc_rmid (u32 closid);void free_rmid (u32 closid,u32 rmid);int rdt_get_mon_l3_config (struct rdt_resource *r); -void __exit rdt_put_mon_l3_config(void); +void __exit rdt_put_mon_l3_config(struct rdt_resource *r);bool __init rdt_cpu_has (int flag);void mon_event_count (void *info);int rdtgroup_mondata_show (struct seq_file *m,void *arg);
arch/x86/kernel/cpu/resctrl/monitor.c +4 -4
@@ -983,7 +983,7 @@ void mbm_setup_overflow_handler(struct rdt_mon_domain *dom, unsigned long delay_schedule_delayed_work_on (cpu, &dom->mbm_over, delay); } -static int dom_data_init (struct rdt_resource *r) +static __init int dom_data_init (struct rdt_resource *r) {u32 idx_limit =resctrl_arch_system_num_rmid_idx ();u32 num_closid =resctrl_arch_get_num_closid (r);
@@ -1044,7 +1044,7 @@ static int dom_data_init(struct rdt_resource *r)return err; } -static void __exit dom_data_exit(void) +static void __exit dom_data_exit(struct rdt_resource *r) {mutex_lock (&rdtgroup_mutex);
@@ -1247,9 +1247,9 @@ int __init rdt_get_mon_l3_config(struct rdt_resource *r)return 0 ; } -void __exit rdt_put_mon_l3_config(void) +void __exit rdt_put_mon_l3_config(struct rdt_resource *r) { -dom_data_exit (); +dom_data_exit (r); }void __init intel_rdt_mbm_apply_quirk (void )
Signature · 1 line
2.34.1
Reinette Chatre
maintainerRe: [PATCH v8 09/25] x86/resctrl: Add __init attribute to dom_data_init()
Hi Babu,
Quoted 6 lines · Babu Moger
dom_data_init()is only called during the__initsequence. Add__initattribute like the rest of call sequence.While at it, pass 'struct
rdt_resource' todom_data_init()anddom_data_exit()which will be used for mbm counter__initand__exitcall sequence.
This patch needs to be split. Please move fixes to beginning of series and move the addition of the parameter to the patch where it is first used/needed.
Quoted 2 lines
Fixes bd334c86b5d7 ("x86/resctrl: Add __init attribute to rdt_get_mon_l3_config()")
For this change I think the following Fixes tag would be more accurate:
I think for a complete fix of the above commit it also needs to add __init storage class to l3_mon_evt_init().
The __init storage class is also missing from rdt_get_mon_l3_config() ... fixing that would indeed need the Fixes tag below:
Reinette
Moger, Babu
reviewerRe: [PATCH v8 09/25] x86/resctrl: Add __init attribute to dom_data_init()
Quoted 6 lines · Reinette Chatre
Hi Babu,
Quoted 3 lines · Babu Moger
dom_data_init()is only called during the__initsequence. Add__initattribute like the rest of call sequence.
Quoted 6 lines
Quoted 3 lines
While at it, pass 'struct
rdt_resource' todom_data_init()anddom_data_exit()which will be used for mbm counter__initand__exitcall sequence.This patch needs to be split. Please move fixes to beginning of series and move the addition of the parameter to the patch where it is first used/needed.
Sure. Will move the fixes to the beginning.
Quoted 9 lines
Quoted 3 lines
Fixes bd334c86b5d7 ("x86/resctrl: Add __init attribute to rdt_get_mon_l3_config()")For this change I think the following Fixes tag would be more accurate:
Fixes 6a445edce657 ("x86/intel_rdt/cqm: Add RDT monitoring initialization")I think for a complete fix of the above commit it also needs to add
__initstorage class tol3_mon_evt_init().
Yes. Sure.
Quoted 2 lines
The
__initstorage class is also missing fromrdt_get_mon_l3_config()...
1 internal.h _int rdt_get_mon_l3_config(struct rdt_resource *r); 2 monitor.c int __init rdt_get_mon_l3_config(struct rdt_resource *r)
rdt_get_mon_l3_config() has __init attribute already. But prototype in internal.h does not add the '__init'. Looks like that is ok.
Quoted 2 lines
fixing that would indeed need the Fixes tag below:
Fixes bd334c86b5d7 ("x86/resctrl: Add __init attribute to rdt_get_mon_l3_config()"
How about addressing both dom_data_init() and l3_mon_evt_init() in a single patch and adding 2 fixes flags?
rdt_get_mon_l3_config()")
Signature · 2 lines
Thanks Babu Moger
Reinette Chatre
maintainerRe: [PATCH v8 09/25] x86/resctrl: Add __init attribute to dom_data_init()
Hi Babu,
Quoted 31 lines · Babu
Quoted 6 lines · Reinette Chatre
Hi Babu,
Quoted 3 lines · Babu Moger
dom_data_init()is only called during the__initsequence. Add__initattribute like the rest of call sequence.Quoted 6 lines
Quoted 3 lines
While at it, pass 'struct
rdt_resource' todom_data_init()anddom_data_exit()which will be used for mbm counter__initand__exitcall sequence.This patch needs to be split. Please move fixes to beginning of series and move the addition of the parameter to the patch where it is first used/needed.
Sure. Will move the fixes to the beginning.
Quoted 9 lines
Quoted 3 lines
Fixes bd334c86b5d7 ("x86/resctrl: Add __init attribute to rdt_get_mon_l3_config()")For this change I think the following Fixes tag would be more accurate:
Fixes 6a445edce657 ("x86/intel_rdt/cqm: Add RDT monitoring initialization")I think for a complete fix of the above commit it also needs to add
__initstorage class tol3_mon_evt_init().Yes. Sure.
Quoted 1 line
Quoted 7 lines
Quoted 1 line
The
__initstorage class is also missing fromrdt_get_mon_l3_config()...1
internal.h_intrdt_get_mon_l3_config(struct rdt_resource *r); 2monitor.cint__initrdt_get_mon_l3_config(struct rdt_resource *r)
rdt_get_mon_l3_config()has__initattribute already. But prototype ininternal.hdoes not add the '__init'. Looks like that is ok.
I also think it may technically be ok since as far as I understand attributes the attributes will be merged. Even so, doing so does not match the current style where the storage class of declaration and definition are the same. See for example the partner function rdt_put_mon_l3_config().
Quoted 7 lines
Quoted 4 lines
fixing that would indeed need the Fixes tag below:
Fixes bd334c86b5d7 ("x86/resctrl: Add __init attribute to rdt_get_mon_l3_config()"How about addressing both
dom_data_init()andl3_mon_evt_init()in a single patch and adding 2 fixes flags?
... and add __init to declaration of rdt_get_mon_l3_config() ?
Quoted 4 lines
Fixes 6a445edce657 ("x86/intel_rdt/cqm: Add RDT monitoring initialization")Fixes bd334c86b5d7 ("x86/resctrl: Add __init attribute to
rdt_get_mon_l3_config()")
Reinette
Moger, Babu
reviewerRe: [PATCH v8 09/25] x86/resctrl: Add __init attribute to dom_data_init()
Hi Reinette,
Quoted 39 lines · Reinette Chatre
Hi Babu,
Quoted 36 lines · Babu
Quoted 6 lines · Reinette Chatre
Hi Babu,
Quoted 3 lines · Babu Moger
dom_data_init()is only called during the__initsequence. Add__initattribute like the rest of call sequence.Quoted 6 lines
Quoted 3 lines
While at it, pass 'struct
rdt_resource' todom_data_init()anddom_data_exit()which will be used for mbm counter__initand__exitcall sequence.This patch needs to be split. Please move fixes to beginning of series and move the addition of the parameter to the patch where it is first used/needed.
Sure. Will move the fixes to the beginning.
Quoted 9 lines
Quoted 3 lines
Fixes bd334c86b5d7 ("x86/resctrl: Add __init attribute to rdt_get_mon_l3_config()")For this change I think the following Fixes tag would be more accurate:
Fixes 6a445edce657 ("x86/intel_rdt/cqm: Add RDT monitoring initialization")I think for a complete fix of the above commit it also needs to add
__initstorage class tol3_mon_evt_init().Yes. Sure.
Quoted 2 lines
The
__initstorage class is also missing fromrdt_get_mon_l3_config()...1
internal.h_intrdt_get_mon_l3_config(struct rdt_resource *r); 2monitor.cint__initrdt_get_mon_l3_config(struct rdt_resource *r)
Quoted 7 lines
Quoted 2 lines
rdt_get_mon_l3_config()has__initattribute already. But prototype ininternal.hdoes not add the '__init'. Looks like that is ok.I also think it may technically be ok since as far as I understand attributes the attributes will be merged. Even so, doing so does not match the current style where the storage class of declaration and definition are the same. See for example the partner function
rdt_put_mon_l3_config().
Sure.
Quoted 10 lines
Quoted 8 lines
Quoted 5 lines
fixing that would indeed need the Fixes tag below:
Fixes bd334c86b5d7 ("x86/resctrl: Add __init attribute to rdt_get_mon_l3_config()"How about addressing both
dom_data_init()andl3_mon_evt_init()in a single patch and adding 2 fixes flags?... and add
__initto declaration ofrdt_get_mon_l3_config()?
Sure. Will do.
Quoted 8 lines
Quoted 5 lines
Fixes 6a445edce657 ("x86/intel_rdt/cqm: Add RDT monitoring initialization")Fixes bd334c86b5d7 ("x86/resctrl: Add __init attribute to
rdt_get_mon_l3_config()")Reinette
Signature · 2 lines
Thanks Babu Moger
Babu Moger
reviewerPATCH v8 10/25 x86/resctrl: Introduce bitmap mbm_cntr_free_map to track assignable counters
In reply to Babu MogerHardware provides a set of counters when mbm_assign_mode is supported. These counters are assigned to the MBM monitoring events of a MON group that needs to be tracked. The kernel must manage and track the available counters.
Introduce mbm_cntr_free_map bitmap to track available counters and set of routines to allocate and free the counters. Move dom_data_init() after mbm_cntr_assign detection.
---
v8: Moved the init and exit functionality inside dom_data_init()
and dom_data_exit() respectively.
v7: Removed the static allocation and now allocating bitmap mbm_cntr_free_map
dynamically.
Passed the struct rdt_resource mbm_cntr_alloc and mbm_cntr_free. Removed the reference of ABMC and changed it mbm_cntr_assign. Few other text changes.
v6: Removed the variable mbm_cntrs_free_map_len. This is not required.
Removed the call mbm_cntrs_init() in arch code. This needs to be done at higher level.
Used DECLARE_BITMAP to initialize mbm_cntrs_free_map. Moved all the counter interfaces mbm_cntr_alloc() and mbm_cntr_free() in here as part of separating arch and fs bits.
v5:
Updated the comments and commit log.
Few renames
num_cntrs_free_map -> mbm_cntrs_free_map num_cntrs_init -> mbm_cntrs_init
Added initialization in rdt_get_tree because the default ABMC enablement happens during the init.
v4: Changed the name to num_cntrs where applicable.
Used bitmap apis.
Added more comments for the globals.
v3: Changed the bitmap name to assign_cntrs_free_map. Removed abmc
from the name.
v2: Changed the bitmap name to assignable_counter_free_map from
abmc_counter_free_map.
--- arch/x86/kernel/cpu/resctrl/internal.h | 2 ++ arch/x86/kernel/cpu/resctrl/monitor.c | 43 +++++++++++++++++++++++--- arch/x86/kernel/cpu/resctrl/rdtgroup.c | 19 ++++++++++++ include/linux/resctrl.h | 2 ++ 4 files changed, 62 insertions(+), 4 deletions(-)
arch/x86/kernel/cpu/resctrl/internal.h +2 -0
@@ -654,6 +654,8 @@ void __check_limbo(struct rdt_mon_domain *d, bool force_free);void rdt_domain_reconfigure_cdp (struct rdt_resource *r);void __init resctrl_file_fflags_init (const char *config,unsigned long fflags); +int mbm_cntr_alloc (struct rdt_resource *r); +void mbm_cntr_free (struct rdt_resource *r,u32 cntr_id);void rdt_staged_configs_clear (void );bool closid_allocated (unsigned int closid);int resctrl_find_cleanest_closid (void );
arch/x86/kernel/cpu/resctrl/monitor.c +39 -4
73 lines not on this page · raw message
arch/x86/kernel/cpu/resctrl/rdtgroup.c +19 -0
@@ -185,6 +185,25 @@ bool closid_allocated(unsigned int closid)return !test_bit (closid, &closid_free_map); } +int mbm_cntr_alloc (struct rdt_resource *r) +{ +int cntr_id; + + cntr_id =find_first_bit (r->mon.mbm_cntr_free_map, + r->mon.num_mbm_cntrs); +if (cntr_id >= r->mon.num_mbm_cntrs) +return -ENOSPC ; + +__clear_bit (cntr_id, r->mon.mbm_cntr_free_map); + +return cntr_id; +} + +void mbm_cntr_free (struct rdt_resource *r,u32 cntr_id) +{ +__set_bit (cntr_id, r->mon.mbm_cntr_free_map); +} +/** * rdtgroup_mode_by_closid - Return mode of resource group with closid * @closid: closid if the resource group
include/linux/resctrl.h +2 -0
@@ -187,12 +187,14 @@ enum resctrl_scope {* @num_rmid: Number of RMIDs available * @num_mbm_cntrs: Number of assignable monitoring counters * @mbm_cntr_assignable:Is system capable of supporting monitor assignment? +* @mbm_cntr_free_map: bitmap of free MBM counters * @evt_list: List of monitoring events */ struct resctrl_mon {int num_rmid;int num_mbm_cntrs;bool mbm_cntr_assignable; +unsigned long *mbm_cntr_free_map;struct list_head evt_list; };
Signature · 1 line
2.34.1
Reinette Chatre
maintainerRe: [PATCH v8 10/25] x86/resctrl: Introduce bitmap mbm_cntr_free_map to track assignable counters
Hi Babu,
Quoted 8 lines · Babu Moger
Hardware provides a set of counters when
mbm_assign_modeis supported. These counters are assigned to the MBM monitoring events of a MON group that needs to be tracked. The kernel must manage and track the available counters.Introduce
mbm_cntr_free_mapbitmap to track available counters and set of routines to allocate and free the counters. Movedom_data_init()aftermbm_cntr_assigndetection.
Regarding "Move dom_data_init() after mbm_cntr_assign detection." - this is clear from the patch, please use changelog to explain *why*.
Quoted 30 lines
Signed-off-by Babu Moger <[email protected]>---
--- arch/x86/kernel/cpu/resctrl/internal.h | 2 ++ arch/x86/kernel/cpu/resctrl/monitor.c | 43 +++++++++++++++++++++++--- arch/x86/kernel/cpu/resctrl/rdtgroup.c | 19 ++++++++++++ include/linux/resctrl.h | 2 ++ 4 files changed, 62 insertions(+), 4 deletions(-)arch/x86/kernel/cpu/resctrl/internal.h +2
-0@@ -654,6 +654,8 @@ void __check_limbo(struct rdt_mon_domain *d, bool force_free);void rdt_domain_reconfigure_cdp (struct rdt_resource *r);void __init resctrl_file_fflags_init (const char *config,unsigned long fflags); +int mbm_cntr_alloc (struct rdt_resource *r); +void mbm_cntr_free (struct rdt_resource *r,u32 cntr_id);void rdt_staged_configs_clear (void );bool closid_allocated (unsigned int closid);int resctrl_find_cleanest_closid (void );arch/x86/kernel/cpu/resctrl/monitor.c +0
-0
arch/x86/kernel/cpu/resctrl/monitor.c void mbm_setup_overflow_handler(struct rdt_mon_domain *dom, unsigned long delay_ · line 990
Quoted 8 lines
arch/x86/kernel/cpu/resctrl/monitor.c +5
-0@@ -983,6 +983,27 @@ void mbm_setup_overflow_handler(struct rdt_mon_domain *dom, unsigned long delay_schedule_delayed_work_on (cpu, &dom->mbm_over, delay); } +/* +* Counter bitmap for tracking the available counters. +* 'mbm_cntr_assign' mode provides set of hardware counters for assigning +* RMID, event pair. Each RMID and event pair takes one hardware counter. +*/
"counters for assigning RMID, event pair" sounds strange and it seems like the same thing is mentioned twice.
How about:
Bitmap tracking the available hardware counters when operating in "mbm_cntr_assign" mode. A hardware counter can be assigned to a RMID, event pair.
Quoted 104 lines
arch/x86/kernel/cpu/resctrl/monitor.c +34
-4+static __init unsigned long *mbm_cntrs_init (struct rdt_resource *r) +{ + r->mon.mbm_cntr_free_map =bitmap_zalloc (r->mon.num_mbm_cntrs, +GFP_KERNEL ); +if (r->mon.mbm_cntr_free_map) +bitmap_fill (r->mon.mbm_cntr_free_map, r->mon.num_mbm_cntrs); + +return r->mon.mbm_cntr_free_map; +} + +static __exit void mbm_cntrs_exit(struct rdt_resource *r) +{ +bitmap_free (r->mon.mbm_cntr_free_map); + r->mon.mbm_cntr_free_map =NULL ; +} +static __init int dom_data_init (struct rdt_resource *r) {u32 idx_limit =resctrl_arch_system_num_rmid_idx ();@@ -1020,6 +1041,17 @@ static __init int dom_data_init(struct rdt_resource *r) goto out_unlock; } +if (r->mon.mbm_cntr_assignable && !mbm_cntrs_init (r)) { +if (IS_ENABLED (CONFIG_RESCTRL_RMID_DEPENDS_ON_CLOSID )) { +kfree (closid_num_dirty_rmid); + closid_num_dirty_rmid =NULL ; + } +kfree (rmid_ptrs); + rmid_ptrs =NULL ; + err = -ENOMEM ; + goto out_unlock; + } +for (i =0 ; i < idx_limit; i++) { entry = &rmid_ptrs[i];INIT_LIST_HEAD (&entry->list);@@ -1056,6 +1088,9 @@ static void __exit dom_data_exit(struct rdt_resource *r)kfree (rmid_ptrs); rmid_ptrs =NULL ; +if (r->mon.mbm_cntr_assignable) +mbm_cntrs_exit (r); +mutex_unlock (&rdtgroup_mutex); }@@ -1210,10 +1245,6 @@ int __init rdt_get_mon_l3_config(struct rdt_resource *r)*/ resctrl_rmid_realloc_threshold =resctrl_arch_round_mon_val (threshold); - ret =dom_data_init (r); -if (ret) -return ret; -if (rdt_cpu_has (X86_FEATURE_BMEC )) {u32 eax, ebx, ecx, edx;@@ -1240,6 +1271,10 @@ int __init rdt_get_mon_l3_config(struct rdt_resource *r) } } + ret =dom_data_init (r); +if (ret) +return ret; +l3_mon_evt_init (r); r->mon_capable = true;arch/x86/kernel/cpu/resctrl/rdtgroup.c +19
-0@@ -185,6 +185,25 @@ bool closid_allocated(unsigned int closid)return !test_bit (closid, &closid_free_map); } +int mbm_cntr_alloc (struct rdt_resource *r) +{ +int cntr_id; + + cntr_id =find_first_bit (r->mon.mbm_cntr_free_map, + r->mon.num_mbm_cntrs); +if (cntr_id >= r->mon.num_mbm_cntrs) +return -ENOSPC ; + +__clear_bit (cntr_id, r->mon.mbm_cntr_free_map); + +return cntr_id; +} + +void mbm_cntr_free (struct rdt_resource *r,u32 cntr_id) +{ +__set_bit (cntr_id, r->mon.mbm_cntr_free_map); +} +/** * rdtgroup_mode_by_closid - Return mode of resource group with closid * @closid: closid if the resource group include/linux/resctrl.h +0
-0
include/linux/resctrl.h enum resctrl_scope { · line 192
Quoted 6 lines
include/linux/resctrl.h +1
-0@@ -187,12 +187,14 @@ enum resctrl_scope {* @num_rmid: Number of RMIDs available * @num_mbm_cntrs: Number of assignable monitoring counters * @mbm_cntr_assignable:Is system capable of supporting monitor assignment? +* @mbm_cntr_free_map: bitmap of free MBM counters * @evt_list: List of monitoring events */
Please follow custom of existing doc and have description start with capital letter.
include/linux/resctrl.h enum resctrl_scope { · line 200
Quoted 8 lines
include/linux/resctrl.h +1
-0struct resctrl_mon {int num_rmid;int num_mbm_cntrs;bool mbm_cntr_assignable; +unsigned long *mbm_cntr_free_map;struct list_head evt_list; };
Reinette
Moger, Babu
reviewerRe: [PATCH v8 10/25] x86/resctrl: Introduce bitmap mbm_cntr_free_map to track assignable counters
Hi Reinette,
Quoted 8 lines · Reinette Chatre
Hi Babu,
Quoted 5 lines · Babu Moger
Hardware provides a set of counters when
mbm_assign_modeis supported. These counters are assigned to the MBM monitoring events of a MON group that needs to be tracked. The kernel must manage and track the available counters.
Quoted 6 lines
Quoted 3 lines
Introduce
mbm_cntr_free_mapbitmap to track available counters and set of routines to allocate and free the counters. Movedom_data_init()aftermbm_cntr_assigndetection.Regarding "Move
dom_data_init()aftermbm_cntr_assigndetection." - this is clear from the patch, please use changelog to explain *why*.
Will change it to.
dom_data_init() requires mbm_cntr_assign state to initialize mbm_cntr_free_map bitmap. Move dom_data_init() after mbm_cntr_assign detection.
Quoted 34 lines
Quoted 34 lines
Signed-off-by Babu Moger <[email protected]>---
--- arch/x86/kernel/cpu/resctrl/internal.h | 2 ++ arch/x86/kernel/cpu/resctrl/monitor.c | 43 +++++++++++++++++++++++--- arch/x86/kernel/cpu/resctrl/rdtgroup.c | 19 ++++++++++++ include/linux/resctrl.h | 2 ++ 4 files changed, 62 insertions(+), 4 deletions(-)arch/x86/kernel/cpu/resctrl/internal.h +2
-0@@ -654,6 +654,8 @@ void __check_limbo(struct rdt_mon_domain *d, bool force_free);void rdt_domain_reconfigure_cdp (struct rdt_resource *r);void __init resctrl_file_fflags_init (const char *config,unsigned long fflags); +int mbm_cntr_alloc (struct rdt_resource *r); +void mbm_cntr_free (struct rdt_resource *r,u32 cntr_id);void rdt_staged_configs_clear (void );bool closid_allocated (unsigned int closid);int resctrl_find_cleanest_closid (void );arch/x86/kernel/cpu/resctrl/monitor.c +0
-0@@ -983,6 +983,27 @@ void mbm_setup_overflow_handler(struct rdt_mon_domain *dom, unsigned long delay_schedule_delayed_work_on (cpu, &dom->mbm_over, delay); }
Quoted 12 lines
arch/x86/kernel/cpu/resctrl/monitor.c void mbm_setup_overflow_handler(struct rdt_mon_domain *dom, unsigned long delay_ · line 990
Quoted 5 lines
arch/x86/kernel/cpu/resctrl/monitor.c +5
-0+/* +* Counter bitmap for tracking the available counters. +* 'mbm_cntr_assign' mode provides set of hardware counters for assigning +* RMID, event pair. Each RMID and event pair takes one hardware counter. +*/ "counters for assigning RMID, event pair" sounds strange and it seems like the same thing is mentioned twice.
How about:
Bitmap tracking the available hardware counters when operating in "mbm_cntr_assign" mode. A hardware counter can be assigned to a RMID, event pair.
Sure.
Quoted 105 lines
Quoted 105 lines
arch/x86/kernel/cpu/resctrl/monitor.c +34
-4+static __init unsigned long *mbm_cntrs_init (struct rdt_resource *r) +{ + r->mon.mbm_cntr_free_map =bitmap_zalloc (r->mon.num_mbm_cntrs, +GFP_KERNEL ); +if (r->mon.mbm_cntr_free_map) +bitmap_fill (r->mon.mbm_cntr_free_map, r->mon.num_mbm_cntrs); + +return r->mon.mbm_cntr_free_map; +} + +static __exit void mbm_cntrs_exit(struct rdt_resource *r) +{ +bitmap_free (r->mon.mbm_cntr_free_map); + r->mon.mbm_cntr_free_map =NULL ; +} +static __init int dom_data_init (struct rdt_resource *r) {u32 idx_limit =resctrl_arch_system_num_rmid_idx ();@@ -1020,6 +1041,17 @@ static __init int dom_data_init(struct rdt_resource *r) goto out_unlock; } +if (r->mon.mbm_cntr_assignable && !mbm_cntrs_init (r)) { +if (IS_ENABLED (CONFIG_RESCTRL_RMID_DEPENDS_ON_CLOSID )) { +kfree (closid_num_dirty_rmid); + closid_num_dirty_rmid =NULL ; + } +kfree (rmid_ptrs); + rmid_ptrs =NULL ; + err = -ENOMEM ; + goto out_unlock; + } +for (i =0 ; i < idx_limit; i++) { entry = &rmid_ptrs[i];INIT_LIST_HEAD (&entry->list);@@ -1056,6 +1088,9 @@ static void __exit dom_data_exit(struct rdt_resource *r)kfree (rmid_ptrs); rmid_ptrs =NULL ; +if (r->mon.mbm_cntr_assignable) +mbm_cntrs_exit (r); +mutex_unlock (&rdtgroup_mutex); }@@ -1210,10 +1245,6 @@ int __init rdt_get_mon_l3_config(struct rdt_resource *r)*/ resctrl_rmid_realloc_threshold =resctrl_arch_round_mon_val (threshold); - ret =dom_data_init (r); -if (ret) -return ret; -if (rdt_cpu_has (X86_FEATURE_BMEC )) {u32 eax, ebx, ecx, edx;@@ -1240,6 +1271,10 @@ int __init rdt_get_mon_l3_config(struct rdt_resource *r) } } + ret =dom_data_init (r); +if (ret) +return ret; +l3_mon_evt_init (r); r->mon_capable = true;arch/x86/kernel/cpu/resctrl/rdtgroup.c +19
-0@@ -185,6 +185,25 @@ bool closid_allocated(unsigned int closid)return !test_bit (closid, &closid_free_map); } +int mbm_cntr_alloc (struct rdt_resource *r) +{ +int cntr_id; + + cntr_id =find_first_bit (r->mon.mbm_cntr_free_map, + r->mon.num_mbm_cntrs); +if (cntr_id >= r->mon.num_mbm_cntrs) +return -ENOSPC ; + +__clear_bit (cntr_id, r->mon.mbm_cntr_free_map); + +return cntr_id; +} + +void mbm_cntr_free (struct rdt_resource *r,u32 cntr_id) +{ +__set_bit (cntr_id, r->mon.mbm_cntr_free_map); +} +/** * rdtgroup_mode_by_closid - Return mode of resource group with closid * @closid: closid if the resource group include/linux/resctrl.h +0
-0@@ -187,12 +187,14 @@ enum resctrl_scope {
Quoted 8 lines
include/linux/resctrl.h enum resctrl_scope { · line 192
Quoted 6 lines
include/linux/resctrl.h +1
-0* @num_rmid: Number of RMIDs available * @num_mbm_cntrs: Number of assignable monitoring counters * @mbm_cntr_assignable:Is system capable of supporting monitor assignment? +* @mbm_cntr_free_map: bitmap of free MBM counters * @evt_list: List of monitoring events */ Please follow custom of existing doc and have description start with capital letter.
Sure.
Quoted 12 lines
include/linux/resctrl.h enum resctrl_scope { · line 200
Quoted 9 lines
include/linux/resctrl.h +1
-0struct resctrl_mon {int num_rmid;int num_mbm_cntrs;bool mbm_cntr_assignable; +unsigned long *mbm_cntr_free_map;struct list_head evt_list; };Reinette
Signature · 2 lines
Thanks Babu Moger
Babu Moger
reviewerPATCH v8 11/25 x86/resctrl: Introduce mbm_total_cfg and mbm_local_cfg in struct rdt_hw_mon_domain
In reply to Babu MogerIf the BMEC (Bandwidth Monitoring Event Configuration) feature is supported, the bandwidth events can be configured to track specific events. The event configuration is domain specific. ABMC (Assignable Bandwidth Monitoring Counters) feature needs event configuration information to assign a hardware counter to an RMID. Event configurations are not stored in resctrl but instead always read from or written to hardware directly when prompted by user space.
Read the event configuration from the hardware during the domain initialization. Save the configuration value in struct rdt_hw_mon_domain, so it can be used for counter assignment.
---
v8: Renamed resctrl_mbm_evt_config_init() to arch_mbm_evt_config_init()
Minor commit message update.
v7: Fixed initializing INVALID_CONFIG_VALUE to mbm_local_cfg in case of error.
v6: Renamed resctrl_arch_mbm_evt_config -> resctrl_mbm_evt_config_init
Initialized value to INVALID_CONFIG_VALUE if it is not configurable. Minor commit message update.
v5: Exported mon_event_config_index_get.
Renamed arch_domain_mbm_evt_config to resctrl_arch_mbm_evt_config.
v4: Read the configuration information from the hardware to initialize.
Added few commit messages.
Fixed the tab spaces.
v3: Minor changes related to rebase in mbm_config_write_domain.
v2: No changes.
--- arch/x86/kernel/cpu/resctrl/core.c | 2 ++ arch/x86/kernel/cpu/resctrl/internal.h | 9 +++++++++ arch/x86/kernel/cpu/resctrl/monitor.c | 26 ++++++++++++++++++++++++++ arch/x86/kernel/cpu/resctrl/rdtgroup.c | 4 +--- 4 files changed, 38 insertions(+), 3 deletions(-)
arch/x86/kernel/cpu/resctrl/core.c +2 -0
@@ -632,6 +632,8 @@ static void domain_add_cpu_mon(int cpu, struct rdt_resource *r)arch_mon_domain_online (r, d); +arch_mbm_evt_config_init (hw_dom); +if (arch_domain_mbm_alloc (r->mon.num_rmid, hw_dom)) {mon_domain_free (hw_dom);return ;
arch/x86/kernel/cpu/resctrl/internal.h +9 -0
@@ -56,6 +56,9 @@/* Max event bits supported */ #define MAX_EVT_CONFIG_BITS GENMASK(6, 0) +#define INVALID_CONFIG_VALUE U32_MAX +#define INVALID_CONFIG_INDEX UINT_MAX +/* Setting bit 0 in L3_QOS_EXT_CFG enables the ABMC feature. */ #define ABMC_ENABLE_BIT 0
@@ -401,6 +404,8 @@ struct rdt_hw_ctrl_domain {* @d_resctrl: Properties exposed to the resctrl file system * @arch_mbm_total: arch private state for MBM total bandwidth * @arch_mbm_local: arch private state for MBM local bandwidth +* @mbm_total_cfg: MBM total bandwidth configuration +* @mbm_local_cfg: MBM local bandwidth configuration * * Members of this structure are accessed via helpers that provide abstraction. */
@@ -408,6 +413,8 @@ struct rdt_hw_mon_domain {struct rdt_mon_domain d_resctrl;struct arch_mbm_state *arch_mbm_total;struct arch_mbm_state *arch_mbm_local; +u32 mbm_total_cfg; +u32 mbm_local_cfg; };static inline struct rdt_hw_ctrl_domain *resctrl_to_arch_ctrl_dom (struct rdt_ctrl_domain *r)
@@ -656,6 +663,8 @@ void __init resctrl_file_fflags_init(const char *config, unsigned long fflags);int mbm_cntr_alloc (struct rdt_resource *r);void mbm_cntr_free (struct rdt_resource *r,u32 cntr_id); +void arch_mbm_evt_config_init (struct rdt_hw_mon_domain *hw_dom); +unsigned int mon_event_config_index_get (u32 evtid);void rdt_staged_configs_clear (void );bool closid_allocated (unsigned int closid);int resctrl_find_cleanest_closid (void );
arch/x86/kernel/cpu/resctrl/monitor.c +26 -0
@@ -1282,6 +1282,32 @@ int __init rdt_get_mon_l3_config(struct rdt_resource *r)return 0 ; } +void arch_mbm_evt_config_init (struct rdt_hw_mon_domain *hw_dom) +{ +unsigned int index; +u64 msrval; + +/* +* Read the configuration registers QOS_EVT_CFG_n, where <n> is +* the BMEC event number (EvtID). +*/ +if (mbm_total_event.configurable) { + index =mon_event_config_index_get (QOS_L3_MBM_TOTAL_EVENT_ID ); +rdmsrl (MSR_IA32_EVT_CFG_BASE + index, msrval); + hw_dom->mbm_total_cfg = msrval &MAX_EVT_CONFIG_BITS ; + }else { + hw_dom->mbm_total_cfg =INVALID_CONFIG_VALUE ; + } + +if (mbm_local_event.configurable) { + index =mon_event_config_index_get (QOS_L3_MBM_LOCAL_EVENT_ID ); +rdmsrl (MSR_IA32_EVT_CFG_BASE + index, msrval); + hw_dom->mbm_local_cfg = msrval &MAX_EVT_CONFIG_BITS ; + }else { + hw_dom->mbm_local_cfg =INVALID_CONFIG_VALUE ; + } +} + void __exit rdt_put_mon_l3_config(struct rdt_resource *r) {dom_data_exit (r);
arch/x86/kernel/cpu/resctrl/rdtgroup.c +1 -3
@@ -1601,8 +1601,6 @@ struct mon_config_info {u32 mon_config; }; -#define INVALID_CONFIG_INDEX UINT_MAX -/** * mon_event_config_index_get - get the hardware index for the * configurable event
@@ -1612,7 +1610,7 @@ struct mon_config_info {* 1 for evtid == QOS_L3_MBM_LOCAL_EVENT_ID * INVALID_CONFIG_INDEX for invalid evtid */ -static inline unsigned int mon_event_config_index_get (u32 evtid) +unsigned int mon_event_config_index_get (u32 evtid) {switch (evtid) {case QOS_L3_MBM_TOTAL_EVENT_ID :
Signature · 1 line
2.34.1
Re: [PATCH v8 11/25] x86/resctrl: Introduce mbm_total_cfg and mbm_local_cfg in struct rdt_hw_mon_domain
Hi Babu,
Quoted 8 lines · Babu Moger
If the BMEC (Bandwidth Monitoring Event Configuration) feature is supported, the bandwidth events can be configured to track specific events. The event configuration is domain specific. ABMC (Assignable Bandwidth Monitoring Counters) feature needs event configuration information to assign a hardware counter to an RMID. Event configurations are not stored in resctrl but instead always read from or written to hardware directly when prompted by user space.
Quoted 6 lines
Read the event configuration from the hardware during the domain initialization. Save the configuration value in struct
rdt_hw_mon_domain, so it can be used for counter assignment.Signed-off-by Babu Moger <[email protected]>---
Reinette
Babu Moger
reviewerPATCH v8 12/25 x86/resctrl: Remove MSR reading of event configuration value
In reply to Babu MogerThe event configuration is domain specific and initialized during domain initialization. The values are stored in struct rdt_hw_mon_domain.
It is not required to read the configuration register every time user asks for it. Use the value stored in struct rdt_hw_mon_domain instead.
Introduce resctrl_arch_mon_event_config_get() and resctrl_arch_mon_event_config_set() to get/set architecture domain specific mbm_total_cfg/mbm_local_cfg values.
---
v8: Renamed
resctrl_arch_event_config_get() to resctrl_arch_mon_event_config_get(). resctrl_arch_event_config_set() to resctrl_arch_mon_event_config_set().
v7: Removed check if (val == INVALID_CONFIG_VALUE) as resctrl_arch_event_config_get
already prints warning.
Kept the Event config value definitions as is.
v6: Fixed inconstancy with types. Made all the types to u32 for config
value.
Removed few rdt_last_cmd_puts as it is not necessary. Removed unused config value definitions.
Few more updates to commit message.
v5: Introduced resctrl_arch_event_config_get and
resctrl_arch_event_config_get() based on our discussion.
https://lore.kernel.org/lkml/[email protected]/
v4: New patch.
--- arch/x86/kernel/cpu/resctrl/rdtgroup.c | 103 ++++++++++++++----------- include/linux/resctrl.h | 4 + 2 files changed, 62 insertions(+), 45 deletions(-)
arch/x86/kernel/cpu/resctrl/rdtgroup.c +58 -45
@@ -1597,10 +1597,57 @@ static int rdtgroup_size_show(struct kernfs_open_file *of, }struct mon_config_info { +struct rdt_mon_domain *d;u32 evtid;u32 mon_config; }; +u32 resctrl_arch_mon_event_config_get (struct rdt_mon_domain *d, +enum resctrl_event_id eventid) +{ +struct rdt_hw_mon_domain *hw_dom =resctrl_to_arch_mon_dom (d); + +switch (eventid) { +case QOS_L3_OCCUP_EVENT_ID : +break ; +case QOS_L3_MBM_TOTAL_EVENT_ID : +return hw_dom->mbm_total_cfg; +case QOS_L3_MBM_LOCAL_EVENT_ID : +return hw_dom->mbm_local_cfg; + } + +/* Never expect to get here */ +WARN_ON_ONCE (1 ); + +return INVALID_CONFIG_VALUE ; +} + +void resctrl_arch_mon_event_config_set (void *info) +{ +struct mon_config_info *mon_info = info; +struct rdt_hw_mon_domain *hw_dom; +unsigned int index; + + index =mon_event_config_index_get (mon_info->evtid); +if (index ==INVALID_CONFIG_INDEX ) +return ; + +wrmsr (MSR_IA32_EVT_CFG_BASE + index, mon_info->mon_config,0 ); + + hw_dom =resctrl_to_arch_mon_dom (mon_info->d); + +switch (mon_info->evtid) { +case QOS_L3_OCCUP_EVENT_ID : +break ; +case QOS_L3_MBM_TOTAL_EVENT_ID : + hw_dom->mbm_total_cfg = mon_info->mon_config; +break ; +case QOS_L3_MBM_LOCAL_EVENT_ID : + hw_dom->mbm_local_cfg = mon_info->mon_config; +break ; + } +} +/** * mon_event_config_index_get - get the hardware index for the * configurable event
@@ -1623,33 +1670,11 @@ unsigned int mon_event_config_index_get(u32 evtid) } } -static void mon_event_config_read (void *info) -{ -struct mon_config_info *mon_info = info; -unsigned int index; -u64 msrval; - - index =mon_event_config_index_get (mon_info->evtid); -if (index ==INVALID_CONFIG_INDEX ) { -pr_warn_once ("Invalid event id %d\n" , mon_info->evtid); -return ; - } -rdmsrl (MSR_IA32_EVT_CFG_BASE + index, msrval); - -/* Report only the valid event configuration bits */ - mon_info->mon_config = msrval &MAX_EVT_CONFIG_BITS ; -} - -static void mondata_config_read (struct rdt_mon_domain *d,struct mon_config_info *mon_info) -{ -smp_call_function_any (&d->hdr.cpu_mask, mon_event_config_read, mon_info,1 ); -} -static int mbm_config_show (struct seq_file *s,struct rdt_resource *r,u32 evtid) { -struct mon_config_info mon_info = {0 };struct rdt_mon_domain *dom;bool sep = false; +u32 val;cpus_read_lock ();mutex_lock (&rdtgroup_mutex);
@@ -1658,11 +1683,8 @@ static int mbm_config_show(struct seq_file *s, struct rdt_resource *r, u32 evtidif (sep)seq_puts (s,";" ); -memset (&mon_info,0 ,sizeof (struct mon_config_info )); - mon_info.evtid = evtid; -mondata_config_read (dom, &mon_info); - -seq_printf (s,"%d=0x%02x" , dom->hdr.id, mon_info.mon_config); + val =resctrl_arch_mon_event_config_get (dom, evtid); +seq_printf (s,"%d=0x%02x" , dom->hdr.id, val); sep = true; }seq_puts (s,"\n" );
@@ -1693,33 +1715,23 @@ static int mbm_local_bytes_config_show(struct kernfs_open_file *of,return 0 ; } -static void mon_event_config_write (void *info) -{ -struct mon_config_info *mon_info = info; -unsigned int index; - - index =mon_event_config_index_get (mon_info->evtid); -if (index ==INVALID_CONFIG_INDEX ) { -pr_warn_once ("Invalid event id %d\n" , mon_info->evtid); -return ; - } -wrmsr (MSR_IA32_EVT_CFG_BASE + index, mon_info->mon_config,0 ); -}static void mbm_config_write_domain (struct rdt_resource *r,struct rdt_mon_domain *d,u32 evtid,u32 val) {struct mon_config_info mon_info = {0 }; +u32 config_val;/* -* Read the current config value first. If both are the same then +* Check the current config value first. If both are the same then * no need to write it again. */ - mon_info.evtid = evtid; -mondata_config_read (d, &mon_info); -if (mon_info.mon_config == val) + config_val =resctrl_arch_mon_event_config_get (d, evtid); +if (config_val ==INVALID_CONFIG_VALUE || config_val == val)return ; + mon_info.d = d; + mon_info.evtid = evtid; mon_info.mon_config = val;/*
@@ -1728,7 +1740,8 @@ static void mbm_config_write_domain(struct rdt_resource *r,* are scoped at the domain level. Writing any of these MSRs * on one CPU is observed by all the CPUs in the domain. */ -smp_call_function_any (&d->hdr.cpu_mask, mon_event_config_write, +smp_call_function_any (&d->hdr.cpu_mask, + resctrl_arch_mon_event_config_set, &mon_info,1 );/*
include/linux/resctrl.h +4 -0
@@ -354,6 +354,10 @@ void resctrl_arch_reset_rmid(struct rdt_resource *r, struct rdt_mon_domain *d,*/ void resctrl_arch_reset_rmid_all (struct rdt_resource *r,struct rdt_mon_domain *d); +void resctrl_arch_mon_event_config_set (void *info); +u32 resctrl_arch_mon_event_config_get (struct rdt_mon_domain *d, +enum resctrl_event_id eventid); +extern unsigned int resctrl_rmid_realloc_threshold;extern unsigned int resctrl_rmid_realloc_limit;
Signature · 1 line
2.34.1
Reinette Chatre
maintainerRe: [PATCH v8 12/25] x86/resctrl: Remove MSR reading of event configuration value
Hi Babu,
Quoted 12 lines · Babu Moger
The event configuration is domain specific and initialized during domain initialization. The values are stored in struct
rdt_hw_mon_domain.It is not required to read the configuration register every time user asks for it. Use the value stored in struct
rdt_hw_mon_domaininstead.Introduce
resctrl_arch_mon_event_config_get()andresctrl_arch_mon_event_config_set()to get/set architecture domain specific mbm_total_cfg/mbm_local_cfg values.Signed-off-by Babu Moger <[email protected]>---
...
Quoted 7 lines
arch/x86/kernel/cpu/resctrl/rdtgroup.c +7
-0+void resctrl_arch_mon_event_config_set (void *info) +{ +struct mon_config_info *mon_info = info; +struct rdt_hw_mon_domain *hw_dom; +unsigned int index; + + index =mon_event_config_index_get (mon_info->evtid);
arch/x86/kernel/cpu/resctrl/rdtgroup.c static int rdtgroup_size_show(struct kernfs_open_file *of, · line 1641
Quoted 10 lines
arch/x86/kernel/cpu/resctrl/rdtgroup.c +10
-0+if (index ==INVALID_CONFIG_INDEX ) +return ; + +wrmsr (MSR_IA32_EVT_CFG_BASE + index, mon_info->mon_config,0 ); + + hw_dom =resctrl_to_arch_mon_dom (mon_info->d); + +switch (mon_info->evtid) { +case QOS_L3_OCCUP_EVENT_ID : +break ;
This check does no harm but I do not think it is necessary since earlier mon_event_config_index_get() would return INVALID_CONFIG_INDEX if the evtid is QOS_L3_OCCUP_EVENT_ID.
arch/x86/kernel/cpu/resctrl/rdtgroup.c static int rdtgroup_size_show(struct kernfs_open_file *of, · line 1646
Quoted 5 lines
arch/x86/kernel/cpu/resctrl/rdtgroup.c +5
-0+case QOS_L3_MBM_TOTAL_EVENT_ID : + hw_dom->mbm_total_cfg = mon_info->mon_config; +break ; +case QOS_L3_MBM_LOCAL_EVENT_ID : + hw_dom->mbm_local_cfg = mon_info->mon_config;
nit: unnecessary space
arch/x86/kernel/cpu/resctrl/rdtgroup.c static int rdtgroup_size_show(struct kernfs_open_file *of, · line 1624
Quoted 4 lines
arch/x86/kernel/cpu/resctrl/rdtgroup.c +4
-0+break ; + } +} +
Reinette
Moger, Babu
reviewerRe: [PATCH v8 12/25] x86/resctrl: Remove MSR reading of event configuration value
Hi Reinette,
Quoted 31 lines · Reinette Chatre
Hi Babu,
Quoted 12 lines · Babu Moger
The event configuration is domain specific and initialized during domain initialization. The values are stored in struct
rdt_hw_mon_domain.It is not required to read the configuration register every time user asks for it. Use the value stored in struct
rdt_hw_mon_domaininstead.Introduce
resctrl_arch_mon_event_config_get()andresctrl_arch_mon_event_config_set()to get/set architecture domain specific mbm_total_cfg/mbm_local_cfg values.Signed-off-by Babu Moger <[email protected]>---
...
Quoted 13 lines
arch/x86/kernel/cpu/resctrl/rdtgroup.c +13
-0+void resctrl_arch_mon_event_config_set (void *info) +{ +struct mon_config_info *mon_info = info; +struct rdt_hw_mon_domain *hw_dom; +unsigned int index; + + index =mon_event_config_index_get (mon_info->evtid); +if (index ==INVALID_CONFIG_INDEX ) +return ; + +wrmsr (MSR_IA32_EVT_CFG_BASE + index, mon_info->mon_config,0 ); + + hw_dom =resctrl_to_arch_mon_dom (mon_info->d);
Quoted 8 lines
arch/x86/kernel/cpu/resctrl/rdtgroup.c static int rdtgroup_size_show(struct kernfs_open_file *of, · line 1641
Quoted 4 lines
arch/x86/kernel/cpu/resctrl/rdtgroup.c +4
-0+ +switch (mon_info->evtid) { +case QOS_L3_OCCUP_EVENT_ID : +break ;This check does no harm but I do not think it is necessary since earlier
mon_event_config_index_get()would returnINVALID_CONFIG_INDEXif the evtid isQOS_L3_OCCUP_EVENT_ID.
Sure. Will remove it.
Quoted 8 lines
arch/x86/kernel/cpu/resctrl/rdtgroup.c static int rdtgroup_size_show(struct kernfs_open_file *of, · line 1646
Quoted 6 lines
arch/x86/kernel/cpu/resctrl/rdtgroup.c +5
-0+case QOS_L3_MBM_TOTAL_EVENT_ID : + hw_dom->mbm_total_cfg = mon_info->mon_config; +break ; +case QOS_L3_MBM_LOCAL_EVENT_ID : + hw_dom->mbm_local_cfg = mon_info->mon_config;nit: unnecessary space
Sure.
Quoted 9 lines
arch/x86/kernel/cpu/resctrl/rdtgroup.c static int rdtgroup_size_show(struct kernfs_open_file *of, · line 1624
Quoted 5 lines
arch/x86/kernel/cpu/resctrl/rdtgroup.c +4
-0+break ; + } +} +Reinette
Signature · 2 lines
Thanks Babu Moger
Babu Moger
reviewerPATCH v8 13/25 x86/resctrl: Introduce mbm_cntr_map to track assignable counters at domain
In reply to Babu MogerThe MBM counters are allocated globally and assigned to an RMID, event pair in a resctrl group. It is tracked by mbm_cntr_free_map. Counters are assigned to the domain based on the user input. It needs to be tracked at domain level also.
Add the mbm_cntr_map bitmap in struct rdt_mon_domain to keep track of assignment at domain level. The global counter at mbm_cntr_free_map can be released when assignment at all the domains are cleared.
---
v8: Minor commit message changes.
v7: Added check mbm_cntr_assignable for allocating bitmap mbm_cntr_map
v6: New patch to add domain level assignment.
--- arch/x86/kernel/cpu/resctrl/rdtgroup.c | 10 ++++++++++ include/linux/resctrl.h | 2 ++ 2 files changed, 12 insertions(+)
arch/x86/kernel/cpu/resctrl/rdtgroup.c +10 -0
@@ -4093,6 +4093,7 @@ static void __init rdtgroup_setup_default(void)static void domain_destroy_mon_state (struct rdt_mon_domain *d) { +bitmap_free (d->mbm_cntr_map);bitmap_free (d->rmid_busy_llc);kfree (d->mbm_total);kfree (d->mbm_local);
@@ -4166,6 +4167,15 @@ static int domain_setup_mon_state(struct rdt_resource *r, struct rdt_mon_domainreturn -ENOMEM ; } } +if (is_mbm_enabled () && r->mon.mbm_cntr_assignable) { + d->mbm_cntr_map =bitmap_zalloc (r->mon.num_mbm_cntrs,GFP_KERNEL ); +if (!d->mbm_cntr_map) { +bitmap_free (d->rmid_busy_llc); +kfree (d->mbm_total); +kfree (d->mbm_local); +return -ENOMEM ; + } + }return 0 ; }
include/linux/resctrl.h +2 -0
@@ -105,6 +105,7 @@ struct rdt_ctrl_domain {* @cqm_limbo: worker to periodically read CQM h/w counters * @mbm_work_cpu: worker CPU for MBM h/w counters * @cqm_work_cpu: worker CPU for CQM h/w counters +* @mbm_cntr_map: bitmap to track domain counter assignment */ struct rdt_mon_domain {struct rdt_domain_hdr hdr;
@@ -116,6 +117,7 @@ struct rdt_mon_domain {struct delayed_work cqm_limbo;int mbm_work_cpu;int cqm_work_cpu; +unsigned long *mbm_cntr_map; };/**
Signature · 1 line
2.34.1
Re: [PATCH v8 13/25] x86/resctrl: Introduce mbm_cntr_map to track assignable counters at domain
Hi Babu,
Quoted 11 lines · Babu Moger
The MBM counters are allocated globally and assigned to an RMID, event pair in a resctrl group. It is tracked by
mbm_cntr_free_map. Counters are assigned to the domain based on the user input. It needs to be tracked at domain level also.Add the
mbm_cntr_mapbitmap in structrdt_mon_domainto keep track of assignment at domain level. The global counter atmbm_cntr_free_mapcan be released when assignment at all the domains are cleared.Signed-off-by Babu Moger <[email protected]>---
Reinette
Babu Moger
reviewerPATCH v8 14/25 x86/resctrl: Add data structures and definitions for ABMC assignment
In reply to Babu MogerThe ABMC feature provides an option to the user to assign a hardware counter to an RMID, event pair and monitor the bandwidth as long as the counter is assigned. The bandwidth events will be tracked by the hardware until the user changes the configuration. Each resctrl group can configure maximum two counters, one for total event and one for local event.
The ABMC feature implements an MSR L3_QOS_ABMC_CFG (C000_03FDh). Configuration is done by setting the counter id, bandwidth source (RMID) and bandwidth configuration supported by BMEC (Bandwidth Monitoring Event Configuration).
Attempts to read or write the MSR when ABMC is not enabled will result in a #GP(0) exception.
Introduce the data structures and definitions for MSR L3_QOS_ABMC_CFG (0xC000_03FDh):
=========================================================================
Bits Mnemonic Description Access Reset
Type Value
=========================================================================
63 CfgEn Configuration Enable R/W 0
62 CtrEn Enable/disable counting R/W 0
61:53 – Reserved MBZ 0
52:48 CtrID Counter Identifier R/W 0
47 IsCOS BwSrc field is a CLOSID R/W 0
(not an RMID)
46:44 – Reserved MBZ 0
43:32 BwSrc Bandwidth Source R/W 0
(RMID or CLOSID)
31:0 BwType Bandwidth configuration R/W 0
to track for this counter
==========================================================================
The feature details are documented in the APM listed below [1].
[1] AMD64 Architecture Programmer's Manual Volume 2: System Programming Publication # 24593 Revision 3.41 section 19.3.3.3 Assignable Bandwidth Monitoring (ABMC).
---
v8: Update the configuration notes in kernel_doc.
Few commit message update.
v7: Removed the reference of L3_QOS_ABMC_DSC as it is not used anymore.
Moved the configuration notes to kernel_doc.
Adjusted the tabs for l3_qos_abmc_cfg and checkpatch seems happy.
v6: Removed all the fs related changes.
Added note on CfgEn,CtrEn.
Removed the definitions which are not used.
Removed cntr_id initialization.
v5: Moved assignment flags here (path 10/19 of v4).
Added MON_CNTR_UNSET definition to initialize cntr_id's. More details in commit log.
Renamed few fields in l3_qos_abmc_cfg for readability.
v4: Added more descriptions.
Changed the name abmc_ctr_id to ctr_id.
Added L3_QOS_ABMC_DSC. Used for reading the configuration.
v3: No changes.
v2: No changes.
--- arch/x86/include/asm/msr-index.h | 1 + arch/x86/kernel/cpu/resctrl/internal.h | 33 ++++++++++++++++++++++++++ 2 files changed, 34 insertions(+)
arch/x86/include/asm/msr-index.h +1 -0
@@ -1196,6 +1196,7 @@#define MSR_IA32_SMBA_BW_BASE 0xc0000280#define MSR_IA32_EVT_CFG_BASE 0xc0000400#define MSR_IA32_L3_QOS_EXT_CFG 0xc00003ff +#define MSR_IA32_L3_QOS_ABMC_CFG 0xc00003fd/* AMD-V MSRs */ #define MSR_VM_CR 0xc0010114
arch/x86/kernel/cpu/resctrl/internal.h +33 -0
@@ -602,6 +602,39 @@ union cpuid_0x10_x_edx {unsigned int full; }; +/* +* ABMC counters can be configured by writing to L3_QOS_ABMC_CFG. +* Reading L3_QOS_ABMC_DSC returns the configuration of the counter id +* specified in L3_QOS_ABMC_CFG.cntr_id. +* @bw_type : Bandwidth configuration(supported by BMEC) +* tracked by the @cntr_id. +* @bw_src : Bandwidth source (RMID or CLOSID). +* @reserved1 : Reserved. +* @is_clos : @bw_src field is a CLOSID (not an RMID). +* @cntr_id : Counter identifier. +* @reserved : Reserved. +* @cntr_en : Counting enable bit. +* @cfg_en : Configuration enable bit. +* +* Configuration and counting: +* cfg_en=0, : No configuration changes applied. +* cfg_en=1, cntr_en=0 : Configure cntr_id and but no counting the events. +* cfg_en=1, cntr_en=1 : Configure cntr_id and start counting the events. +*/ +union l3_qos_abmc_cfg { +struct { +unsigned long bw_type :32 , + bw_src :12 , + reserved1:3 , + is_clos :1 , + cntr_id :5 , + reserved :9 , + cntr_en :1 , + cfg_en :1 ; + } split; +unsigned long full; +}; +void rdt_last_cmd_clear (void );void rdt_last_cmd_puts (const char *s);__printf (1 ,2 )
Signature · 1 line
2.34.1
Reinette Chatre
maintainerRe: [PATCH v8 14/25] x86/resctrl: Add data structures and definitions for ABMC assignment
Hi Babu,
Quoted 48 lines · Babu Moger
The ABMC feature provides an option to the user to assign a hardware counter to an RMID, event pair and monitor the bandwidth as long as the counter is assigned. The bandwidth events will be tracked by the hardware until the user changes the configuration. Each resctrl group can configure maximum two counters, one for total event and one for local event.
The ABMC feature implements an MSR
L3_QOS_ABMC_CFG(C000_03FDh). Configuration is done by setting the counter id, bandwidth source (RMID) and bandwidth configuration supported by BMEC (Bandwidth Monitoring Event Configuration).Attempts to read or write the MSR when ABMC is not enabled will result in a #GP(0) exception.
Introduce the data structures and definitions for MSR
L3_QOS_ABMC_CFG(0xC000_03FDh):
=========================================================================
Bits Mnemonic Description Access Reset
Type Value
=========================================================================
63 CfgEn Configuration Enable R/W 062 CtrEn Enable/disable counting R/W 0
61:53 – Reserved MBZ 0
52:48 CtrID Counter Identifier R/W 0
47 IsCOS BwSrc field is a CLOSID R/W 0
(not an RMID)46:44 – Reserved MBZ 0
43:32 BwSrc Bandwidth Source R/W 0
(RMID or CLOSID)31:0 BwType Bandwidth configuration R/W 0
to track for this counter
==========================================================================The feature details are documented in the APM listed below [1].
[1] AMD64 Architecture Programmer's Manual Volume 2: System Programming Publication # 24593 Revision 3.41 section 19.3.3.3 Assignable Bandwidth Monitoring (ABMC).---
...
Quoted 10 lines
--- arch/x86/include/asm/msr-index.h | 1 + arch/x86/kernel/cpu/resctrl/internal.h | 33 ++++++++++++++++++++++++++ 2 files changed, 34 insertions(+)arch/x86/include/asm/msr-index.h +0
-0
arch/x86/include/asm/msr-index.h line 1200
Quoted 5 lines
arch/x86/include/asm/msr-index.h +1
-0@@ -1196,6 +1196,7 @@#define MSR_IA32_SMBA_BW_BASE 0xc0000280#define MSR_IA32_EVT_CFG_BASE 0xc0000400#define MSR_IA32_L3_QOS_EXT_CFG 0xc00003ff +#define MSR_IA32_L3_QOS_ABMC_CFG 0xc00003fd
As Tony mentioned, also please correct order of this MSR.
arch/x86/kernel/cpu/resctrl/internal.h union cpuid_0x10_x_edx { · line 606
Quoted 12 lines
arch/x86/include/asm/msr-index.h +0
-0/* AMD-V MSRs */ #define MSR_VM_CR 0xc0010114arch/x86/kernel/cpu/resctrl/internal.h +2
-0@@ -602,6 +602,39 @@ union cpuid_0x10_x_edx {unsigned int full; }; +/* +* ABMC counters can be configured by writing to L3_QOS_ABMC_CFG.
"ABMC counters are configured by writing to L3_QOS_ABMC_CFG."
arch/x86/kernel/cpu/resctrl/internal.h union cpuid_0x10_x_edx { · line 608
Quoted 2 lines
arch/x86/kernel/cpu/resctrl/internal.h +2
-0+* Reading L3_QOS_ABMC_DSC returns the configuration of the counter id +* specified in L3_QOS_ABMC_CFG.cntr_id.
First and only mention/use of L3_QOS_ABMC_DSC in this series. If this register is not used then references to it can be removed.
arch/x86/kernel/cpu/resctrl/internal.h union cpuid_0x10_x_edx { · line 609
Quoted 1 line
arch/x86/kernel/cpu/resctrl/internal.h +1
-0+* @bw_type : Bandwidth configuration(supported by BMEC)
"configuration(supported" -> "configuration (supported"
arch/x86/kernel/cpu/resctrl/internal.h union cpuid_0x10_x_edx { · line 620
Quoted 11 lines
arch/x86/kernel/cpu/resctrl/internal.h +11
-0+* tracked by the @cntr_id. +* @bw_src : Bandwidth source (RMID or CLOSID). +* @reserved1 : Reserved. +* @is_clos : @bw_src field is a CLOSID (not an RMID). +* @cntr_id : Counter identifier. +* @reserved : Reserved. +* @cntr_en : Counting enable bit. +* @cfg_en : Configuration enable bit. +* +* Configuration and counting: +* cfg_en=0, : No configuration changes applied.
Can this be expanded? (sidenote: It is taking a long time to get clarity on how to interact with hardware. These incremental cryptic fragments make it difficult to know how to interact with the hardware.)
For example, "No configuration changes applied. Counter can be configured across multiple writes to MSR while @cfg_en=0. Configuration applied when @cfg_en=1."
arch/x86/kernel/cpu/resctrl/internal.h union cpuid_0x10_x_edx { · line 621
Quoted 1 line
arch/x86/kernel/cpu/resctrl/internal.h +1
-0+* cfg_en=1, cntr_en=0 : Configure cntr_id and but no counting the events.
hmmm ... still the same (""but no counting the events") strange language I highlighted in V7 ...
I think it will make things easier to understand if similar language is used between the descriptions of the different fields.
"Apply @cntr_id configuration but do not count events."
arch/x86/kernel/cpu/resctrl/internal.h union cpuid_0x10_x_edx { · line 622
Quoted 1 line
arch/x86/kernel/cpu/resctrl/internal.h +1
-0+* cfg_en=1, cntr_en=1 : Configure cntr_id and start counting the events.
"Apply @cntr_id configuration and start counting events."
Can it be added here which of these settings (or combination of settings) result in counters being reset?
Quoted 8 lines
arch/x86/kernel/cpu/resctrl/internal.h +8
-0+*/ +union l3_qos_abmc_cfg { +struct { +unsigned long bw_type :32 , + bw_src :12 , + reserved1:3 , + is_clos :1 , + cntr_id : 5,
arch/x86/kernel/cpu/resctrl/internal.h union cpuid_0x10_x_edx { · line 640
Quoted 10 lines
arch/x86/kernel/cpu/resctrl/internal.h +7
-0+ reserved : 9, + cntr_en : 1, + cfg_en : 1; + } split; +unsigned long full; +}; +void rdt_last_cmd_clear (void );void rdt_last_cmd_puts (const char *s);__printf (1 ,2 )
Reinette
Moger, Babu
reviewerRe: [PATCH v8 14/25] x86/resctrl: Add data structures and definitions for ABMC assignment
Hi Reinette,
Quoted 59 lines · Reinette Chatre
Hi Babu,
Quoted 48 lines · Babu Moger
The ABMC feature provides an option to the user to assign a hardware counter to an RMID, event pair and monitor the bandwidth as long as the counter is assigned. The bandwidth events will be tracked by the hardware until the user changes the configuration. Each resctrl group can configure maximum two counters, one for total event and one for local event.
The ABMC feature implements an MSR
L3_QOS_ABMC_CFG(C000_03FDh). Configuration is done by setting the counter id, bandwidth source (RMID) and bandwidth configuration supported by BMEC (Bandwidth Monitoring Event Configuration).Attempts to read or write the MSR when ABMC is not enabled will result in a #GP(0) exception.
Introduce the data structures and definitions for MSR
L3_QOS_ABMC_CFG(0xC000_03FDh):
=========================================================================
Bits Mnemonic Description Access Reset
Type Value
=========================================================================
63 CfgEn Configuration Enable R/W 062 CtrEn Enable/disable counting R/W 0
61:53 – Reserved MBZ 0
52:48 CtrID Counter Identifier R/W 0
47 IsCOS BwSrc field is a CLOSID R/W 0
(not an RMID)46:44 – Reserved MBZ 0
43:32 BwSrc Bandwidth Source R/W 0
(RMID or CLOSID)31:0 BwType Bandwidth configuration R/W 0
to track for this counter
==========================================================================The feature details are documented in the APM listed below [1].
[1] AMD64 Architecture Programmer's Manual Volume 2: System Programming Publication # 24593 Revision 3.41 section 19.3.3.3 Assignable Bandwidth Monitoring (ABMC).---
...
Quoted 5 lines
--- arch/x86/include/asm/msr-index.h | 1 + arch/x86/kernel/cpu/resctrl/internal.h | 33 ++++++++++++++++++++++++++ 2 files changed, 34 insertions(+)
Quoted 12 lines
arch/x86/include/asm/msr-index.h line 1200
Quoted 10 lines
arch/x86/include/asm/msr-index.h +1
-0@@ -1196,6 +1196,7 @@#define MSR_IA32_SMBA_BW_BASE 0xc0000280#define MSR_IA32_EVT_CFG_BASE 0xc0000400#define MSR_IA32_L3_QOS_EXT_CFG 0xc00003ff +#define MSR_IA32_L3_QOS_ABMC_CFG 0xc00003fdAs Tony mentioned, also please correct order of this MSR.
Sure.
Quoted 11 lines
Quoted 11 lines
arch/x86/include/asm/msr-index.h +0
-0/* AMD-V MSRs */ #define MSR_VM_CR 0xc0010114arch/x86/kernel/cpu/resctrl/internal.h +0
-0@@ -602,6 +602,39 @@ union cpuid_0x10_x_edx {unsigned int full; };
Quoted 4 lines
arch/x86/kernel/cpu/resctrl/internal.h union cpuid_0x10_x_edx { · line 606
Quoted 2 lines
arch/x86/kernel/cpu/resctrl/internal.h +2
-0+/* +* ABMC counters can be configured by writing to L3_QOS_ABMC_CFG. "ABMC counters are configured by writing to
L3_QOS_ABMC_CFG."
Sure.
Quoted 6 lines
arch/x86/kernel/cpu/resctrl/internal.h union cpuid_0x10_x_edx { · line 608
Quoted 3 lines
arch/x86/kernel/cpu/resctrl/internal.h +2
-0+* Reading L3_QOS_ABMC_DSC returns the configuration of the counter id +* specified in L3_QOS_ABMC_CFG.cntr_id. First and only mention/use of
L3_QOS_ABMC_DSCin this series. If this register is not used then references to it can be removed.
Sure.
Quoted 4 lines
arch/x86/kernel/cpu/resctrl/internal.h union cpuid_0x10_x_edx { · line 609
Quoted 2 lines
arch/x86/kernel/cpu/resctrl/internal.h +1
-0+* @bw_type : Bandwidth configuration(supported by BMEC) "configuration(supported" -> "configuration (supported"
Sure.
Quoted 28 lines
arch/x86/kernel/cpu/resctrl/internal.h union cpuid_0x10_x_edx { · line 620
Quoted 12 lines
arch/x86/kernel/cpu/resctrl/internal.h +11
-0+* tracked by the @cntr_id. +* @bw_src : Bandwidth source (RMID or CLOSID). +* @reserved1 : Reserved. +* @is_clos : @bw_src field is a CLOSID (not an RMID). +* @cntr_id : Counter identifier. +* @reserved : Reserved. +* @cntr_en : Counting enable bit. +* @cfg_en : Configuration enable bit. +* +* Configuration and counting: +* cfg_en=0, : No configuration changes applied. Can this be expanded? (sidenote: It is taking a long time to get clarity on how to interact with hardware. These incremental cryptic fragments make it difficult to know how to interact with the hardware.)
For example, "No configuration changes applied. Counter can be configured across multiple writes to MSR while @cfg_en=0. Configuration applied when @cfg_en=1."
arch/x86/kernel/cpu/resctrl/internal.h union cpuid_0x10_x_edx { · line 621
Quoted 1 line
arch/x86/kernel/cpu/resctrl/internal.h +1
-0+* cfg_en=1, cntr_en=0 : Configure cntr_id and but no counting the events. hmmm ... still the same (""but no counting the events") strange language I highlighted in V7 ...
I think it will make things easier to understand if similar language is used between the descriptions of the different fields.
Quoted 8 lines
"Apply @cntr_id configuration but do not count events."
arch/x86/kernel/cpu/resctrl/internal.h union cpuid_0x10_x_edx { · line 622
Quoted 1 line
arch/x86/kernel/cpu/resctrl/internal.h +1
-0+* cfg_en=1, cntr_en=1 : Configure cntr_id and start counting the events. "Apply @cntr_id configuration and start counting events."
Can it be added here which of these settings (or combination of settings) result in counters being reset?
Any change in the configuration will reset the counters.
Little bit lost here. Let me summarize. How about this?
Configuration and counting:
Counter can be configured across multiple writes to MSR. Configuration is applied only when @cfg_en = 1. The event counters will reset when any of the configuration is changed.cfg_en = 1, cntr_en = 0 : Apply @cntr_id configuration but do not count events.cfg_en = 1, cntr_en = 1 : Apply @cntr_id configuration and start counting events.
Quoted 22 lines
Quoted 9 lines
arch/x86/kernel/cpu/resctrl/internal.h +8
-0+*/ +union l3_qos_abmc_cfg { +struct { +unsigned long bw_type :32 , + bw_src :12 , + reserved1:3 , + is_clos :1 , + cntr_id : 5,arch/x86/kernel/cpu/resctrl/internal.h union cpuid_0x10_x_edx { · line 640
Quoted 10 lines
arch/x86/kernel/cpu/resctrl/internal.h +7
-0+ reserved : 9, + cntr_en : 1, + cfg_en : 1; + } split; +unsigned long full; +}; +void rdt_last_cmd_clear (void );void rdt_last_cmd_puts (const char *s);__printf (1 ,2 )Reinette
Signature · 2 lines
Thanks Babu Moger
Reinette Chatre
maintainerRe: [PATCH v8 14/25] x86/resctrl: Add data structures and definitions for ABMC assignment
Hi Babu,
Quoted 39 lines · Babu
Quoted 29 lines · Reinette Chatre
Quoted 11 lines · Babu Moger
+ * tracked by the @cntr_id.
+ * @bw_src : Bandwidth source (RMID or CLOSID).
+ * @reserved1 : Reserved.
+ * @is_clos : @bw_src field is a CLOSID (not an RMID).
+ * @cntr_id : Counter identifier.
+ * @reserved : Reserved.
+ * @cntr_en : Counting enable bit.
+ * @cfg_en : Configuration enable bit.
+ *
+ * Configuration and counting:
+ * cfg_en=0, : No configuration changes applied.Can this be expanded? (sidenote: It is taking a long time to get clarity on how to interact with hardware. These incremental cryptic fragments make it difficult to know how to interact with the hardware.)
For example, "No configuration changes applied. Counter can be configured across multiple writes to MSR while @cfg_en=0. Configuration applied when @cfg_en=1."
Quoted 1 line
+ * cfg_en=1, cntr_en=0 : Configure
cntr_idand but no counting the events.hmmm ... still the same (""but no counting the events") strange language I highlighted in V7 ...
I think it will make things easier to understand if similar language is used between the descriptions of the different fields.
Quoted 8 lines
"Apply @cntr_id configuration but do not count events."
Quoted 1 line
+ * cfg_en=1, cntr_en=1 : Configure
cntr_idand start counting the events."Apply @cntr_id configuration and start counting events."
Can it be added here which of these settings (or combination of settings) result in counters being reset?
Quoted 8 lines
Any change in the configuration will reset the counters.
Little bit lost here. Let me summarize. How about this?
Configuration and counting:
Counter can be configured across multiple writes to MSR. Configuration is applied only when @cfg_en = 1. The event counters will reset when any of the configuration is changed.
Is is not clear to me what is meant with "when any of the configuration is changed". Are event counters reset with every write to the MSR, whether @cfg_en is set or not? If counters are only reset when @cfg_en is set then I think it should read "Counter @cntr_id is reset when the configuration is applied." Note this is also made specific to be related to *just* the counter being configured, not all event counters that "The event counters will reset ..." implies.
Quoted 5 lines
cfg_en= 1,cntr_en= 0 : Apply @cntr_id configuration but do not count events.cfg_en= 1,cntr_en= 1 : Apply @cntr_id configuration and start counting events.
Please use the @ prefix when referring to union members.
Thank you
Reinette
Re: [PATCH v8 14/25] x86/resctrl: Add data structures and definitions for ABMC assignment
Hi Reinette,
Quoted 51 lines · Reinette Chatre
Hi Babu,
Quoted 39 lines · Babu
Quoted 29 lines · Reinette Chatre
Quoted 11 lines · Babu Moger
+ * tracked by the @cntr_id.
+ * @bw_src : Bandwidth source (RMID or CLOSID).
+ * @reserved1 : Reserved.
+ * @is_clos : @bw_src field is a CLOSID (not an RMID).
+ * @cntr_id : Counter identifier.
+ * @reserved : Reserved.
+ * @cntr_en : Counting enable bit.
+ * @cfg_en : Configuration enable bit.
+ *
+ * Configuration and counting:
+ * cfg_en=0, : No configuration changes applied.Can this be expanded? (sidenote: It is taking a long time to get clarity on how to interact with hardware. These incremental cryptic fragments make it difficult to know how to interact with the hardware.)
For example, "No configuration changes applied. Counter can be configured across multiple writes to MSR while @cfg_en=0. Configuration applied when @cfg_en=1."
Quoted 1 line
+ * cfg_en=1, cntr_en=0 : Configure
cntr_idand but no counting the events.hmmm ... still the same (""but no counting the events") strange language I highlighted in V7 ...
I think it will make things easier to understand if similar language is used between the descriptions of the different fields.
Quoted 8 lines
"Apply @cntr_id configuration but do not count events."
Quoted 1 line
+ * cfg_en=1, cntr_en=1 : Configure
cntr_idand start counting the events."Apply @cntr_id configuration and start counting events."
Can it be added here which of these settings (or combination of settings) result in counters being reset?
Quoted 8 lines
Any change in the configuration will reset the counters.
Little bit lost here. Let me summarize. How about this?
Configuration and counting:
Counter can be configured across multiple writes to MSR. Configuration is applied only when @cfg_en = 1. The event counters will reset when any of the configuration is changed.
Quoted 4 lines
Is is not clear to me what is meant with "when any of the configuration is changed". Are event counters reset with every write to the MSR, whether @cfg_en is set or not? If counters are only reset when @cfg_en is set then I think it should read "Counter @cntr_id is reset when the configuration is applied." Note this is
Sure.
Quoted 10 lines
also made specific to be related to *just* the counter being configured, not all event counters that "The event counters will reset ..." implies.
Quoted 5 lines
cfg_en= 1,cntr_en= 0 : Apply @cntr_id configuration but do not count events.cfg_en= 1,cntr_en= 1 : Apply @cntr_id configuration and start counting events.Please use the @ prefix when referring to union members.
Ok. Sure.
Quoted 6 lines
Thank you
Reinette
Thanks
- Babu Moger
Babu Moger
reviewerPATCH v8 15/25 x86/resctrl: Introduce cntr_id in mongroup for assignments
In reply to Babu Mogermbm_cntr_assign feature provides an option to the user to assign a counter to an RMID, event pair and monitor the bandwidth as long as the counter is assigned. There can be two counters per monitor group, one for MBM total event and another for MBM local event.
Introduce cntr_id to manage the assignments.
---
v8: Minor commit message update.
v7: Minor comment update for cntr_id.
v6: New patch.
Separated FS and arch bits.
--- arch/x86/kernel/cpu/resctrl/internal.h | 7 +++++++ arch/x86/kernel/cpu/resctrl/rdtgroup.c | 6 ++++++ 2 files changed, 13 insertions(+)
arch/x86/kernel/cpu/resctrl/internal.h +7 -0
@@ -62,6 +62,11 @@/* Setting bit 0 in L3_QOS_EXT_CFG enables the ABMC feature. */ #define ABMC_ENABLE_BIT 0 +/* Maximum assignable counters per resctrl group */ +#define MAX_CNTRS 2 + +#define MON_CNTR_UNSET U32_MAX +/** * cpumask_any_housekeeping() - Choose any CPU in @mask, preferring those that * aren't marked nohz_full
@@ -231,12 +236,14 @@ enum rdtgrp_mode {* @parent: parent rdtgrp * @crdtgrp_list: child rdtgroup node list * @rmid: rmid for this rdtgroup +* @cntr_id: IDs of hardware counters assigned to monitor group */ struct mongroup {struct kernfs_node *mon_data_kn;struct rdtgroup *parent;struct list_head crdtgrp_list;u32 rmid; +u32 cntr_id[MAX_CNTRS ]; };/**
arch/x86/kernel/cpu/resctrl/rdtgroup.c +6 -0
@@ -3530,6 +3530,9 @@ static int mkdir_rdt_prepare_rmid_alloc(struct rdtgroup *rdtgrp) } rdtgrp->mon.rmid = ret; + rdtgrp->mon.cntr_id[0 ] =MON_CNTR_UNSET ; + rdtgrp->mon.cntr_id[1 ] =MON_CNTR_UNSET ; + ret =mkdir_mondata_all (rdtgrp->kn, rdtgrp, &rdtgrp->mon.mon_data_kn);if (ret) {rdt_last_cmd_puts ("kernfs subdir error\n" );
@@ -4084,6 +4087,9 @@ static void __init rdtgroup_setup_default(void) rdtgroup_default.closid =RESCTRL_RESERVED_CLOSID ; rdtgroup_default.mon.rmid =RESCTRL_RESERVED_RMID ; rdtgroup_default.type =RDTCTRL_GROUP ; + rdtgroup_default.mon.cntr_id[0 ] =MON_CNTR_UNSET ; + rdtgroup_default.mon.cntr_id[1 ] =MON_CNTR_UNSET ; +INIT_LIST_HEAD (&rdtgroup_default.mon.crdtgrp_list);list_add (&rdtgroup_default.rdtgroup_list, &rdt_all_groups);
Signature · 1 line
2.34.1
Reinette Chatre
maintainerRe: [PATCH v8 15/25] x86/resctrl: Introduce cntr_id in mongroup for assignments
Hi Babu,
Quoted 66 lines · Babu Moger
mbm_cntr_assignfeature provides an option to the user to assign a counter to an RMID, event pair and monitor the bandwidth as long as the counter is assigned. There can be two counters per monitor group, one for MBM total event and another for MBM local event.Introduce
cntr_idto manage the assignments.Signed-off-by Babu Moger <[email protected]>---
v8: Minor commit message update.v7: Minor comment update for
cntr_id.v6: New patch.
Separated FS and arch bits.--- arch/x86/kernel/cpu/resctrl/internal.h | 7 +++++++ arch/x86/kernel/cpu/resctrl/rdtgroup.c | 6 ++++++ 2 files changed, 13 insertions(+)arch/x86/kernel/cpu/resctrl/internal.h +7
-0@@ -62,6 +62,11 @@/* Setting bit 0 in L3_QOS_EXT_CFG enables the ABMC feature. */ #define ABMC_ENABLE_BIT 0 +/* Maximum assignable counters per resctrl group */ +#define MAX_CNTRS 2 + +#define MON_CNTR_UNSET U32_MAX +/** * cpumask_any_housekeeping() - Choose any CPU in @mask, preferring those that * aren't marked nohz_full @@ -231,12 +236,14 @@ enum rdtgrp_mode {* @parent: parent rdtgrp * @crdtgrp_list: child rdtgroup node list * @rmid: rmid for this rdtgroup +* @cntr_id: IDs of hardware counters assigned to monitor group */ struct mongroup {struct kernfs_node *mon_data_kn;struct rdtgroup *parent;struct list_head crdtgrp_list;u32 rmid; +u32 cntr_id[MAX_CNTRS ]; };/** arch/x86/kernel/cpu/resctrl/rdtgroup.c +3
-0@@ -3530,6 +3530,9 @@ static int mkdir_rdt_prepare_rmid_alloc(struct rdtgroup *rdtgrp) } rdtgrp->mon.rmid = ret; + rdtgrp->mon.cntr_id[0 ] =MON_CNTR_UNSET ; + rdtgrp->mon.cntr_id[1 ] =MON_CNTR_UNSET ; + ret =mkdir_mondata_all (rdtgrp->kn, rdtgrp, &rdtgrp->mon.mon_data_kn);if (ret) {rdt_last_cmd_puts ("kernfs subdir error\n" );
arch/x86/kernel/cpu/resctrl/rdtgroup.c static void __init rdtgroup_setup_default(void) · line 4092
Quoted 6 lines
arch/x86/kernel/cpu/resctrl/rdtgroup.c +3
-0@@ -4084,6 +4087,9 @@ static void __init rdtgroup_setup_default(void) rdtgroup_default.closid =RESCTRL_RESERVED_CLOSID ; rdtgroup_default.mon.rmid =RESCTRL_RESERVED_RMID ; rdtgroup_default.type =RDTCTRL_GROUP ; + rdtgroup_default.mon.cntr_id[0 ] =MON_CNTR_UNSET ; + rdtgroup_default.mon.cntr_id[1 ] =MON_CNTR_UNSET ; +
Could these magic constants be avoided by introducing MBM_EVENT_ARRAY_INDEX here and using it for the array index instead of "0" and "1"?
arch/x86/kernel/cpu/resctrl/rdtgroup.c static void __init rdtgroup_setup_default(void) · line 4095
Quoted 3 lines
arch/x86/kernel/cpu/resctrl/rdtgroup.c +0
-0INIT_LIST_HEAD (&rdtgroup_default.mon.crdtgrp_list);list_add (&rdtgroup_default.rdtgroup_list, &rdt_all_groups);
Reinette
Moger, Babu
reviewerRe: [PATCH v8 15/25] x86/resctrl: Introduce cntr_id in mongroup for assignments
Hi Reinette,
Quoted 62 lines · Reinette Chatre
Hi Babu,
Quoted 59 lines · Babu Moger
mbm_cntr_assignfeature provides an option to the user to assign a counter to an RMID, event pair and monitor the bandwidth as long as the counter is assigned. There can be two counters per monitor group, one for MBM total event and another for MBM local event.Introduce
cntr_idto manage the assignments.Signed-off-by Babu Moger <[email protected]>---
v8: Minor commit message update.v7: Minor comment update for
cntr_id.v6: New patch.
Separated FS and arch bits.--- arch/x86/kernel/cpu/resctrl/internal.h | 7 +++++++ arch/x86/kernel/cpu/resctrl/rdtgroup.c | 6 ++++++ 2 files changed, 13 insertions(+)arch/x86/kernel/cpu/resctrl/internal.h +7
-0@@ -62,6 +62,11 @@/* Setting bit 0 in L3_QOS_EXT_CFG enables the ABMC feature. */ #define ABMC_ENABLE_BIT 0 +/* Maximum assignable counters per resctrl group */ +#define MAX_CNTRS 2 + +#define MON_CNTR_UNSET U32_MAX +/** * cpumask_any_housekeeping() - Choose any CPU in @mask, preferring those that * aren't marked nohz_full @@ -231,12 +236,14 @@ enum rdtgrp_mode {* @parent: parent rdtgrp * @crdtgrp_list: child rdtgroup node list * @rmid: rmid for this rdtgroup +* @cntr_id: IDs of hardware counters assigned to monitor group */ struct mongroup {struct kernfs_node *mon_data_kn;struct rdtgroup *parent;struct list_head crdtgrp_list;u32 rmid; +u32 cntr_id[MAX_CNTRS ]; };/** arch/x86/kernel/cpu/resctrl/rdtgroup.c +0
-0@@ -3530,6 +3530,9 @@ static int mkdir_rdt_prepare_rmid_alloc(struct rdtgroup *rdtgrp) } rdtgrp->mon.rmid = ret;
Quoted 16 lines
arch/x86/kernel/cpu/resctrl/rdtgroup.c static void __init rdtgroup_setup_default(void) · line 4092
Quoted 13 lines
arch/x86/kernel/cpu/resctrl/rdtgroup.c +6
-0+ rdtgrp->mon.cntr_id[0 ] =MON_CNTR_UNSET ; + rdtgrp->mon.cntr_id[1 ] =MON_CNTR_UNSET ; + ret =mkdir_mondata_all (rdtgrp->kn, rdtgrp, &rdtgrp->mon.mon_data_kn);if (ret) {rdt_last_cmd_puts ("kernfs subdir error\n" );@@ -4084,6 +4087,9 @@ static void __init rdtgroup_setup_default(void) rdtgroup_default.closid =RESCTRL_RESERVED_CLOSID ; rdtgroup_default.mon.rmid =RESCTRL_RESERVED_RMID ; rdtgroup_default.type =RDTCTRL_GROUP ; + rdtgroup_default.mon.cntr_id[0 ] =MON_CNTR_UNSET ; + rdtgroup_default.mon.cntr_id[1 ] =MON_CNTR_UNSET ; +Could these magic constants be avoided by introducing
MBM_EVENT_ARRAY_INDEXhere and using it for the array index instead of "0" and "1"?
Sure. Will do.
Quoted 9 lines
arch/x86/kernel/cpu/resctrl/rdtgroup.c static void __init rdtgroup_setup_default(void) · line 4095
Quoted 4 lines
arch/x86/kernel/cpu/resctrl/rdtgroup.c +0
-0INIT_LIST_HEAD (&rdtgroup_default.mon.crdtgrp_list);list_add (&rdtgroup_default.rdtgroup_list, &rdt_all_groups);Reinette
Signature · 2 lines
Thanks Babu Moger
Babu Moger
reviewerPATCH v8 16/25 x86/resctrl: Implement resctrl_arch_config_cntr() to assign a counter with ABMC
In reply to Babu MogerThe ABMC feature provides an option to the user to assign a hardware counter to an RMID, event pair and monitor the bandwidth as long as it is assigned. The assigned RMID will be tracked by the hardware until the user unassigns it manually.
Counters are configured by writing to L3_QOS_ABMC_CFG MSR and specifying the counter id, bandwidth source, and bandwidth types.
Provide the interface to assign the counter ids to RMID.
The feature details are documented in the APM listed below [1].
[1] AMD64 Architecture Programmer's Manual Volume 2: System Programming
Publication # 24593 Revision 3.41 section 19.3.3.3 Assignable Bandwidth
Monitoring (ABMC).---
v8: Rename resctrl_arch_assign_cntr to resctrl_arch_config_cntr.
v7: Separated arch and fs functions. This patch only has arch implementation.
Added struct rdt_resource to the interface resctrl_arch_assign_cntr. Rename rdtgroup_abmc_cfg() to resctrl_abmc_config_one_amd().
v6: Removed mbm_cntr_alloc() from this patch to keep fs and arch code
separate.
Added code to update the counter assignment at domain level.
v5: Few name changes to match cntr_id.
Changed the function names to
rdtgroup_assign_cntr
resctr_arch_assign_cntr
More comments on commit log.
Added function summary.
v4: Commit message update.
User bitmap APIs where applicable. Changed the interfaces considering MPAM(arm). Added domain specific assignment.
v3: Removed the static from the prototype of rdtgroup_assign_abmc.
The function is not called directly from user anymore. These changes are related to global assignment interface.
v2: Minor text changes in commit message.
--- arch/x86/kernel/cpu/resctrl/internal.h | 3 ++ arch/x86/kernel/cpu/resctrl/rdtgroup.c | 45 ++++++++++++++++++++++++++ 2 files changed, 48 insertions(+)
arch/x86/kernel/cpu/resctrl/internal.h +3 -0
@@ -705,6 +705,9 @@ int mbm_cntr_alloc(struct rdt_resource *r);void mbm_cntr_free (struct rdt_resource *r,u32 cntr_id);void arch_mbm_evt_config_init (struct rdt_hw_mon_domain *hw_dom);unsigned int mon_event_config_index_get (u32 evtid); +int resctrl_arch_config_cntr (struct rdt_resource *r,struct rdt_mon_domain *d, +enum resctrl_event_id evtid,u32 rmid,u32 closid, +u32 cntr_id,bool assign);void rdt_staged_configs_clear (void );bool closid_allocated (unsigned int closid);int resctrl_find_cleanest_closid (void );
arch/x86/kernel/cpu/resctrl/rdtgroup.c +45 -0
51 lines not on this page · raw message
Signature · 1 line
2.34.1
Reinette Chatre
maintainerRe: [PATCH v8 16/25] x86/resctrl: Implement resctrl_arch_config_cntr() to assign a counter with ABMC
Hi Babu,
Quoted 18 lines · Babu Moger
The ABMC feature provides an option to the user to assign a hardware counter to an RMID, event pair and monitor the bandwidth as long as it is assigned. The assigned RMID will be tracked by the hardware until the user unassigns it manually.
Counters are configured by writing to
L3_QOS_ABMC_CFGMSR and specifying the counter id, bandwidth source, and bandwidth types.Provide the interface to assign the counter ids to RMID.
The feature details are documented in the APM listed below [1].
[1] AMD64 Architecture Programmer's Manual Volume 2: System ProgrammingPublication # 24593 Revision 3.41 section 19.3.3.3 Assignable Bandwidth Monitoring (ABMC).---
...
Quoted 59 lines
--- arch/x86/kernel/cpu/resctrl/internal.h | 3 ++ arch/x86/kernel/cpu/resctrl/rdtgroup.c | 45 ++++++++++++++++++++++++++ 2 files changed, 48 insertions(+)arch/x86/kernel/cpu/resctrl/internal.h +3
-0@@ -705,6 +705,9 @@ int mbm_cntr_alloc(struct rdt_resource *r);void mbm_cntr_free (struct rdt_resource *r,u32 cntr_id);void arch_mbm_evt_config_init (struct rdt_hw_mon_domain *hw_dom);unsigned int mon_event_config_index_get (u32 evtid); +int resctrl_arch_config_cntr (struct rdt_resource *r,struct rdt_mon_domain *d, +enum resctrl_event_id evtid,u32 rmid,u32 closid, +u32 cntr_id,bool assign);void rdt_staged_configs_clear (void );bool closid_allocated (unsigned int closid);int resctrl_find_cleanest_closid (void );arch/x86/kernel/cpu/resctrl/rdtgroup.c +32
-0@@ -1853,6 +1853,51 @@ static ssize_t mbm_local_bytes_config_write(struct kernfs_open_file *of,return ret ?: nbytes; } +static void resctrl_abmc_config_one_amd (void *info) +{ +u64 *msrval = info; + +wrmsrl (MSR_IA32_L3_QOS_ABMC_CFG , *msrval); +} + +/* +* Send an IPI to the domain to assign the counter to RMID, event pair. +*/ +int resctrl_arch_config_cntr (struct rdt_resource *r,struct rdt_mon_domain *d, +enum resctrl_event_id evtid,u32 rmid,u32 closid, +u32 cntr_id,bool assign) +{ +struct rdt_hw_mon_domain *hw_dom =resctrl_to_arch_mon_dom (d); +union l3_qos_abmc_cfg abmc_cfg = {0 }; +struct arch_mbm_state *arch_mbm; + + abmc_cfg.split.cfg_en =1 ; + abmc_cfg.split.cntr_en = assign ?1 :0 ; + abmc_cfg.split.cntr_id = cntr_id; + abmc_cfg.split.bw_src = rmid; + +/* Update the event configuration from the domain */ +if (evtid ==QOS_L3_MBM_TOTAL_EVENT_ID ) { + abmc_cfg.split.bw_type = hw_dom->mbm_total_cfg; + arch_mbm = &hw_dom->arch_mbm_total[rmid]; + }else { + abmc_cfg.split.bw_type = hw_dom->mbm_local_cfg; + arch_mbm = &hw_dom->arch_mbm_local[rmid]; + } +
arch/x86/kernel/cpu/resctrl/rdtgroup.c static ssize_t mbm_local_bytes_config_write(struct kernfs_open_file *of, · line 1897
Quoted 10 lines
arch/x86/kernel/cpu/resctrl/rdtgroup.c +10
-0+smp_call_function_any (&d->hdr.cpu_mask, resctrl_abmc_config_one_amd, + &abmc_cfg,1 ); + +/* +* Reset the architectural state so that reading of hardware +* counter is not considered as an overflow in next update. +*/ +if (arch_mbm) +memset (arch_mbm,0 ,sizeof (struct arch_mbm_state )); +
More on this later, but I do believe later code can be simplified if reset of architectural state is done by caller. This function should focus on just configuring the counter.
arch/x86/kernel/cpu/resctrl/rdtgroup.c static ssize_t mbm_local_bytes_config_write(struct kernfs_open_file *of, · line 1903
Quoted 6 lines
arch/x86/kernel/cpu/resctrl/rdtgroup.c +3
-0+return 0 ; +} +/* rdtgroup information files for one cache resource. */ static struct rftype res_common_files[] = { {
Reinette
Re: [PATCH v8 16/25] x86/resctrl: Implement resctrl_arch_config_cntr() to assign a counter with ABMC
Hi Reinette,
Quoted 89 lines · Reinette Chatre
Hi Babu,
Quoted 18 lines · Babu Moger
The ABMC feature provides an option to the user to assign a hardware counter to an RMID, event pair and monitor the bandwidth as long as it is assigned. The assigned RMID will be tracked by the hardware until the user unassigns it manually.
Counters are configured by writing to
L3_QOS_ABMC_CFGMSR and specifying the counter id, bandwidth source, and bandwidth types.Provide the interface to assign the counter ids to RMID.
The feature details are documented in the APM listed below [1].
[1] AMD64 Architecture Programmer's Manual Volume 2: System ProgrammingPublication # 24593 Revision 3.41 section 19.3.3.3 Assignable Bandwidth Monitoring (ABMC).---
...
Quoted 65 lines
--- arch/x86/kernel/cpu/resctrl/internal.h | 3 ++ arch/x86/kernel/cpu/resctrl/rdtgroup.c | 45 ++++++++++++++++++++++++++ 2 files changed, 48 insertions(+)arch/x86/kernel/cpu/resctrl/internal.h +3
-0@@ -705,6 +705,9 @@ int mbm_cntr_alloc(struct rdt_resource *r);void mbm_cntr_free (struct rdt_resource *r,u32 cntr_id);void arch_mbm_evt_config_init (struct rdt_hw_mon_domain *hw_dom);unsigned int mon_event_config_index_get (u32 evtid); +int resctrl_arch_config_cntr (struct rdt_resource *r,struct rdt_mon_domain *d, +enum resctrl_event_id evtid,u32 rmid,u32 closid, +u32 cntr_id,bool assign);void rdt_staged_configs_clear (void );bool closid_allocated (unsigned int closid);int resctrl_find_cleanest_closid (void );arch/x86/kernel/cpu/resctrl/rdtgroup.c +38
-0@@ -1853,6 +1853,51 @@ static ssize_t mbm_local_bytes_config_write(struct kernfs_open_file *of,return ret ?: nbytes; } +static void resctrl_abmc_config_one_amd (void *info) +{ +u64 *msrval = info; + +wrmsrl (MSR_IA32_L3_QOS_ABMC_CFG , *msrval); +} + +/* +* Send an IPI to the domain to assign the counter to RMID, event pair. +*/ +int resctrl_arch_config_cntr (struct rdt_resource *r,struct rdt_mon_domain *d, +enum resctrl_event_id evtid,u32 rmid,u32 closid, +u32 cntr_id,bool assign) +{ +struct rdt_hw_mon_domain *hw_dom =resctrl_to_arch_mon_dom (d); +union l3_qos_abmc_cfg abmc_cfg = {0 }; +struct arch_mbm_state *arch_mbm; + + abmc_cfg.split.cfg_en =1 ; + abmc_cfg.split.cntr_en = assign ?1 :0 ; + abmc_cfg.split.cntr_id = cntr_id; + abmc_cfg.split.bw_src = rmid; + +/* Update the event configuration from the domain */ +if (evtid ==QOS_L3_MBM_TOTAL_EVENT_ID ) { + abmc_cfg.split.bw_type = hw_dom->mbm_total_cfg; + arch_mbm = &hw_dom->arch_mbm_total[rmid]; + }else { + abmc_cfg.split.bw_type = hw_dom->mbm_local_cfg; + arch_mbm = &hw_dom->arch_mbm_local[rmid]; + } + +smp_call_function_any (&d->hdr.cpu_mask, resctrl_abmc_config_one_amd, + &abmc_cfg,1 ); + +/* +* Reset the architectural state so that reading of hardware +* counter is not considered as an overflow in next update.
Quoted 8 lines
arch/x86/kernel/cpu/resctrl/rdtgroup.c static ssize_t mbm_local_bytes_config_write(struct kernfs_open_file *of, · line 1897
Quoted 4 lines
arch/x86/kernel/cpu/resctrl/rdtgroup.c +4
-0+*/ +if (arch_mbm) +memset (arch_mbm,0 ,sizeof (struct arch_mbm_state )); +More on this later, but I do believe later code can be simplified if reset of architectural state is done by caller. This function should focus on just configuring the counter.
Yes. It can be done by the caller. Need to introduce arch handler to do this as it accesses the arch state.
Quoted 10 lines
arch/x86/kernel/cpu/resctrl/rdtgroup.c static ssize_t mbm_local_bytes_config_write(struct kernfs_open_file *of, · line 1903
Quoted 7 lines
arch/x86/kernel/cpu/resctrl/rdtgroup.c +3
-0+return 0 ; +} +/* rdtgroup information files for one cache resource. */ static struct rftype res_common_files[] = { {Reinette
Signature · 1 line
- Babu Moger
Babu Moger
reviewerPATCH v8 17/25 x86/resctrl: Add the interface to assign/update counter assignment
In reply to Babu MogerThe mbm_cntr_assign mode offers several hardware counters that can be assigned to an RMID-event pair and monitor the bandwidth as long as it is assigned.
Counters are managed at two levels. The global assignment is tracked using the mbm_cntr_free_map field in the struct resctrl_mon, while domain-specific assignments are tracked using the mbm_cntr_map field in the struct rdt_mon_domain. Allocation begins at the global level and is then applied individually to each domain.
Introduce an interface to allocate these counters and update the corresponding domains accordingly.
---
v8: Renamed rdtgroup_assign_cntr() to rdtgroup_assign_cntr_event().
Added the code to return the error if rdtgroup_assign_cntr_event fails. Moved definition of MBM_EVENT_ARRAY_INDEX to resctrl/internal.h. Updated typo in the comments.
v7: New patch. Moved all the FS code here.
Merged rdtgroup_assign_cntr and rdtgroup_alloc_cntr. Adde new #define MBM_EVENT_ARRAY_INDEX.
--- arch/x86/kernel/cpu/resctrl/internal.h | 9 +++++ arch/x86/kernel/cpu/resctrl/rdtgroup.c | 47 ++++++++++++++++++++++++++ 2 files changed, 56 insertions(+)
arch/x86/kernel/cpu/resctrl/internal.h +9 -0
@@ -67,6 +67,13 @@#define MON_CNTR_UNSET U32_MAX +/* +* Get the counter index for the assignable counter +* 0 for evtid == QOS_L3_MBM_TOTAL_EVENT_ID +* 1 for evtid == QOS_L3_MBM_LOCAL_EVENT_ID +*/ +#define MBM_EVENT_ARRAY_INDEX (_event) ((_event) - 2) +/** * cpumask_any_housekeeping() - Choose any CPU in @mask, preferring those that * aren't marked nohz_full
@@ -708,6 +715,8 @@ unsigned int mon_event_config_index_get(u32 evtid);int resctrl_arch_config_cntr (struct rdt_resource *r,struct rdt_mon_domain *d,enum resctrl_event_id evtid,u32 rmid,u32 closid,u32 cntr_id,bool assign); +int rdtgroup_assign_cntr_event (struct rdt_resource *r,struct rdtgroup *rdtgrp, +struct rdt_mon_domain *d,enum resctrl_event_id evtid);void rdt_staged_configs_clear (void );bool closid_allocated (unsigned int closid);int resctrl_find_cleanest_closid (void );
arch/x86/kernel/cpu/resctrl/rdtgroup.c +47 -0
53 lines not on this page · raw message
Signature · 1 line
2.34.1
Reinette Chatre
maintainerRe: [PATCH v8 17/25] x86/resctrl: Add the interface to assign/update counter assignment
Hi Babu,
Quoted 2 lines · Babu Moger
The
mbm_cntr_assignmode offers several hardware counters that can be assigned to an RMID-event pair and monitor the bandwidth as long as it
repeated nit (to be consistent): RMID, event
Quoted 31 lines
is assigned.
Counters are managed at two levels. The global assignment is tracked using the
mbm_cntr_free_mapfield in the structresctrl_mon, while domain-specific assignments are tracked using thembm_cntr_mapfield in the structrdt_mon_domain. Allocation begins at the global level and is then applied individually to each domain.Introduce an interface to allocate these counters and update the corresponding domains accordingly.
Signed-off-by Babu Moger <[email protected]>---
v8: Renamedrdtgroup_assign_cntr()tordtgroup_assign_cntr_event().
Added the code to return the error ifrdtgroup_assign_cntr_eventfails. Moved definition ofMBM_EVENT_ARRAY_INDEXtoresctrl/internal.h. Updated typo in the comments.v7: New patch. Moved all the FS code here.
Mergedrdtgroup_assign_cntrandrdtgroup_alloc_cntr. Adde new #defineMBM_EVENT_ARRAY_INDEX.--- arch/x86/kernel/cpu/resctrl/internal.h | 9 +++++ arch/x86/kernel/cpu/resctrl/rdtgroup.c | 47 ++++++++++++++++++++++++++ 2 files changed, 56 insertions(+)arch/x86/kernel/cpu/resctrl/internal.h +0
-0
arch/x86/kernel/cpu/resctrl/internal.h line 76
Quoted 10 lines
arch/x86/kernel/cpu/resctrl/internal.h +7
-0@@ -67,6 +67,13 @@#define MON_CNTR_UNSET U32_MAX +/* +* Get the counter index for the assignable counter +* 0 for evtid == QOS_L3_MBM_TOTAL_EVENT_ID +* 1 for evtid == QOS_L3_MBM_LOCAL_EVENT_ID +*/ +#define MBM_EVENT_ARRAY_INDEX (_event) ((_event) - 2) +
This can be moved to patch that introduces and initializes the array and used there.
Quoted 17 lines
arch/x86/kernel/cpu/resctrl/internal.h +2
-0/** * cpumask_any_housekeeping() - Choose any CPU in @mask, preferring those that * aren't marked nohz_full @@ -708,6 +715,8 @@ unsigned int mon_event_config_index_get(u32 evtid);int resctrl_arch_config_cntr (struct rdt_resource *r,struct rdt_mon_domain *d,enum resctrl_event_id evtid,u32 rmid,u32 closid,u32 cntr_id,bool assign); +int rdtgroup_assign_cntr_event (struct rdt_resource *r,struct rdtgroup *rdtgrp, +struct rdt_mon_domain *d,enum resctrl_event_id evtid);void rdt_staged_configs_clear (void );bool closid_allocated (unsigned int closid);int resctrl_find_cleanest_closid (void );arch/x86/kernel/cpu/resctrl/rdtgroup.c +0
-0
arch/x86/kernel/cpu/resctrl/rdtgroup.c int resctrl_arch_config_cntr(struct rdt_resource *r, struct rdt_mon_domain *d, · line 1902
Quoted 5 lines
arch/x86/kernel/cpu/resctrl/rdtgroup.c +2
-0@@ -1898,6 +1898,53 @@ int resctrl_arch_config_cntr(struct rdt_resource *r, struct rdt_mon_domain *d,return 0 ; } +/* +* Assign a hardware counter to the group.
hmmm ... counters are not assigned to groups. How about:
"Assign a hardware counter to event @evtid of group @rdtgrp"?
arch/x86/kernel/cpu/resctrl/rdtgroup.c int resctrl_arch_config_cntr(struct rdt_resource *r, struct rdt_mon_domain *d, · line 1904
Quoted 2 lines
arch/x86/kernel/cpu/resctrl/rdtgroup.c +2
-0+* Counter will be assigned to all the domains if rdt_mon_domain is NULL +* else the counter will be allocated to specific domain.
"will be allocated to" -> "will be assigned to"?
Quoted 19 lines
arch/x86/kernel/cpu/resctrl/rdtgroup.c +19
-0+*/ +int rdtgroup_assign_cntr_event (struct rdt_resource *r,struct rdtgroup *rdtgrp, +struct rdt_mon_domain *d,enum resctrl_event_id evtid) +{ +int index =MBM_EVENT_ARRAY_INDEX (evtid); +int cntr_id = rdtgrp->mon.cntr_id[index]; +int ret; + +/* +* Allocate a new counter id to the event if the counter is not +* assigned already. +*/ +if (cntr_id ==MON_CNTR_UNSET ) { + cntr_id =mbm_cntr_alloc (r); +if (cntr_id <0 ) { +rdt_last_cmd_puts ("Out of MBM assignable counters\n" ); +return -ENOSPC ; + } + rdtgrp->mon.cntr_id[index] = cntr_id;
arch/x86/kernel/cpu/resctrl/rdtgroup.c int resctrl_arch_config_cntr(struct rdt_resource *r, struct rdt_mon_domain *d, · line 1933
Quoted 10 lines
arch/x86/kernel/cpu/resctrl/rdtgroup.c +10
-0+ } + +if (!d) { +list_for_each_entry (d, &r->mon_domains, hdr.list) { + ret =resctrl_arch_config_cntr (r, d, evtid, rdtgrp->mon.rmid, + rdtgrp->closid, cntr_id, true); +if (ret) + goto out_done_assign; + +set_bit (cntr_id, d->mbm_cntr_map);
The code pattern above is repeated four times in this work, twice in rdtgroup_assign_cntr_event() and twice in rdtgroup_unassign_cntr_event(). This duplication should be avoided. It can be done in a function that also resets the architectural state.
arch/x86/kernel/cpu/resctrl/rdtgroup.c int resctrl_arch_config_cntr(struct rdt_resource *r, struct rdt_mon_domain *d, · line 1944
Quoted 11 lines
arch/x86/kernel/cpu/resctrl/rdtgroup.c +11
-0+ } + }else { + ret =resctrl_arch_config_cntr (r, d, evtid, rdtgrp->mon.rmid, + rdtgrp->closid, cntr_id, true); +if (ret) + goto out_done_assign; + +set_bit (cntr_id, d->mbm_cntr_map); + } + +out_done_assign:
Should a newly allocated counter not be freed if it could not be configured?
arch/x86/kernel/cpu/resctrl/rdtgroup.c int resctrl_arch_config_cntr(struct rdt_resource *r, struct rdt_mon_domain *d, · line 1950
Quoted 6 lines
arch/x86/kernel/cpu/resctrl/rdtgroup.c +3
-0+return ret; +} +/* rdtgroup information files for one cache resource. */ static struct rftype res_common_files[] = { {
Reinette
Re: [PATCH v8 17/25] x86/resctrl: Add the interface to assign/update counter assignment
Hi Reinette,
Quoted 7 lines · Reinette Chatre
Hi Babu,
Quoted 2 lines · Babu Moger
The
mbm_cntr_assignmode offers several hardware counters that can be assigned to an RMID-event pair and monitor the bandwidth as long as itrepeated nit (to be consistent): RMID, event
Sure.
Quoted 35 lines
Quoted 35 lines
is assigned.
Counters are managed at two levels. The global assignment is tracked using the
mbm_cntr_free_mapfield in the structresctrl_mon, while domain-specific assignments are tracked using thembm_cntr_mapfield in the structrdt_mon_domain. Allocation begins at the global level and is then applied individually to each domain.Introduce an interface to allocate these counters and update the corresponding domains accordingly.
Signed-off-by Babu Moger <[email protected]>---
v8: Renamedrdtgroup_assign_cntr()tordtgroup_assign_cntr_event().
Added the code to return the error ifrdtgroup_assign_cntr_eventfails. Moved definition ofMBM_EVENT_ARRAY_INDEXtoresctrl/internal.h. Updated typo in the comments.v7: New patch. Moved all the FS code here.
Mergedrdtgroup_assign_cntrandrdtgroup_alloc_cntr. Adde new #defineMBM_EVENT_ARRAY_INDEX.--- arch/x86/kernel/cpu/resctrl/internal.h | 9 +++++ arch/x86/kernel/cpu/resctrl/rdtgroup.c | 47 ++++++++++++++++++++++++++ 2 files changed, 56 insertions(+)arch/x86/kernel/cpu/resctrl/internal.h +0
-0@@ -67,6 +67,13 @@#define MON_CNTR_UNSET U32_MAX
Quoted 9 lines
arch/x86/kernel/cpu/resctrl/internal.h line 76
Quoted 7 lines
arch/x86/kernel/cpu/resctrl/internal.h +7
-0+/* +* Get the counter index for the assignable counter +* 0 for evtid == QOS_L3_MBM_TOTAL_EVENT_ID +* 1 for evtid == QOS_L3_MBM_LOCAL_EVENT_ID +*/ +#define MBM_EVENT_ARRAY_INDEX (_event) ((_event) - 2) +This can be moved to patch that introduces and initializes the array and used there.
Sure.
Quoted 21 lines
Quoted 21 lines
arch/x86/kernel/cpu/resctrl/internal.h +2
-0/** * cpumask_any_housekeeping() - Choose any CPU in @mask, preferring those that * aren't marked nohz_full @@ -708,6 +715,8 @@ unsigned int mon_event_config_index_get(u32 evtid);int resctrl_arch_config_cntr (struct rdt_resource *r,struct rdt_mon_domain *d,enum resctrl_event_id evtid,u32 rmid,u32 closid,u32 cntr_id,bool assign); +int rdtgroup_assign_cntr_event (struct rdt_resource *r,struct rdtgroup *rdtgrp, +struct rdt_mon_domain *d,enum resctrl_event_id evtid);void rdt_staged_configs_clear (void );bool closid_allocated (unsigned int closid);int resctrl_find_cleanest_closid (void );arch/x86/kernel/cpu/resctrl/rdtgroup.c +0
-0@@ -1898,6 +1898,53 @@ int resctrl_arch_config_cntr(struct rdt_resource *r, struct rdt_mon_domain *d,return 0 ; }
Quoted 5 lines
arch/x86/kernel/cpu/resctrl/rdtgroup.c int resctrl_arch_config_cntr(struct rdt_resource *r, struct rdt_mon_domain *d, · line 1902
Quoted 2 lines
arch/x86/kernel/cpu/resctrl/rdtgroup.c +2
-0+/* +* Assign a hardware counter to the group. hmmm ... counters are not assigned to groups. How about:
"Assign a hardware counter to event @evtid of group @rdtgrp"?
Sure.
Quoted 5 lines
arch/x86/kernel/cpu/resctrl/rdtgroup.c int resctrl_arch_config_cntr(struct rdt_resource *r, struct rdt_mon_domain *d, · line 1904
Quoted 3 lines
arch/x86/kernel/cpu/resctrl/rdtgroup.c +2
-0+* Counter will be assigned to all the domains if rdt_mon_domain is NULL +* else the counter will be allocated to specific domain. "will be allocated to" -> "will be assigned to"?
Sure.
Quoted 27 lines
Quoted 27 lines
arch/x86/kernel/cpu/resctrl/rdtgroup.c +26
-0+*/ +int rdtgroup_assign_cntr_event (struct rdt_resource *r,struct rdtgroup *rdtgrp, +struct rdt_mon_domain *d,enum resctrl_event_id evtid) +{ +int index =MBM_EVENT_ARRAY_INDEX (evtid); +int cntr_id = rdtgrp->mon.cntr_id[index]; +int ret; + +/* +* Allocate a new counter id to the event if the counter is not +* assigned already. +*/ +if (cntr_id ==MON_CNTR_UNSET ) { + cntr_id =mbm_cntr_alloc (r); +if (cntr_id <0 ) { +rdt_last_cmd_puts ("Out of MBM assignable counters\n" ); +return -ENOSPC ; + } + rdtgrp->mon.cntr_id[index] = cntr_id; + } + +if (!d) { +list_for_each_entry (d, &r->mon_domains, hdr.list) { + ret =resctrl_arch_config_cntr (r, d, evtid, rdtgrp->mon.rmid, + rdtgrp->closid, cntr_id, true); +if (ret)
Quoted 8 lines
arch/x86/kernel/cpu/resctrl/rdtgroup.c int resctrl_arch_config_cntr(struct rdt_resource *r, struct rdt_mon_domain *d, · line 1933
Quoted 3 lines
arch/x86/kernel/cpu/resctrl/rdtgroup.c +3
-0+ goto out_done_assign; + +set_bit (cntr_id, d->mbm_cntr_map);The code pattern above is repeated four times in this work, twice in
rdtgroup_assign_cntr_event()and twice inrdtgroup_unassign_cntr_event(). This duplication should be avoided. It can be done in a function that also resets the architectural state.
Are you suggesting to combine rdtgroup_assign_cntr_event() and rdtgroup_unassign_cntr_event()?
It can be done. We need a flag to tell if it is a assign or unassign.
Quoted 14 lines
arch/x86/kernel/cpu/resctrl/rdtgroup.c int resctrl_arch_config_cntr(struct rdt_resource *r, struct rdt_mon_domain *d, · line 1944
Quoted 12 lines
arch/x86/kernel/cpu/resctrl/rdtgroup.c +11
-0+ } + }else { + ret =resctrl_arch_config_cntr (r, d, evtid, rdtgrp->mon.rmid, + rdtgrp->closid, cntr_id, true); +if (ret) + goto out_done_assign; + +set_bit (cntr_id, d->mbm_cntr_map); + } + +out_done_assign:Should a newly allocated counter not be freed if it could not be configured?
Yes.
Quoted 10 lines
arch/x86/kernel/cpu/resctrl/rdtgroup.c int resctrl_arch_config_cntr(struct rdt_resource *r, struct rdt_mon_domain *d, · line 1950
Quoted 7 lines
arch/x86/kernel/cpu/resctrl/rdtgroup.c +3
-0+return ret; +} +/* rdtgroup information files for one cache resource. */ static struct rftype res_common_files[] = { {Reinette
Signature · 1 line
- Babu Moger
Reinette Chatre
maintainerRe: [PATCH v8 17/25] x86/resctrl: Add the interface to assign/update counter assignment
Hi Babu,
Quoted 38 lines · Babu
Quoted 28 lines · Reinette Chatre
Quoted 26 lines · Babu Moger
+ */
arch/x86/kernel/cpu/resctrl/rdtgroup.c +25
-0+int rdtgroup_assign_cntr_event (struct rdt_resource *r,struct rdtgroup *rdtgrp, + struct rdt_mon_domain *d, enum resctrl_event_id evtid) +{ + int index = MBM_EVENT_ARRAY_INDEX(evtid); + int cntr_id = rdtgrp->mon.cntr_id[index]; + int ret; + + /* + * Allocate a new counter id to the event if the counter is not + * assigned already. + */ + if (cntr_id == MON_CNTR_UNSET) { + cntr_id = mbm_cntr_alloc(r); + if (cntr_id < 0) { + rdt_last_cmd_puts("Out of MBM assignable counters\n"); + return -ENOSPC; + } + rdtgrp->mon.cntr_id[index] = cntr_id; + } + + if (!d) { + list_for_each_entry(d, &r->mon_domains, hdr.list) { + ret = resctrl_arch_config_cntr(r, d, evtid, rdtgrp->mon.rmid, + rdtgrp->closid, cntr_id, true); + if (ret)Quoted 8 lines
Quoted 3 lines
+ goto
out_done_assign;
+
+set_bit(cntr_id, d->mbm_cntr_map);The code pattern above is repeated four times in this work, twice in
rdtgroup_assign_cntr_event()and twice inrdtgroup_unassign_cntr_event(). This duplication should be avoided. It can be done in a function that also resets the architectural state.
Quoted 1 line
Are you suggesting to combine
rdtgroup_assign_cntr_event()andrdtgroup_unassign_cntr_event()?
No. My comment was about the following pattern that is repeated four times:
... ret = resctrl_arch_config_cntr(...) if (ret) ... set_bit()/clear_bit() ...
Quoted 1 line
It can be done. We need a flag to tell if it is a assign or unassign.
There is already a flag that is used by resctrl_arch_config_cntr(), the same parameters as resctrl_arch_config_cntr() can be used for a wrapper that just calls resctrl_arch_config_cntr() directly and uses that same flag to
select between set_bit() and clear_bit(). This wrapper can then also include the reset of architectural state.
Also, I do not think we need atomic bitops here so these can be __set_bit() and __clear_bit() that also matches how bits of mbm_cntr_free_map are managed earlier in series.
Reinette
Moger, Babu
reviewerRe: [PATCH v8 17/25] x86/resctrl: Add the interface to assign/update counter assignment
Hi Reinette,
Quoted 43 lines · Reinette Chatre
Hi Babu,
Quoted 38 lines · Babu
Quoted 28 lines · Reinette Chatre
Quoted 26 lines · Babu Moger
+ */
arch/x86/kernel/cpu/resctrl/rdtgroup.c +25
-0+int rdtgroup_assign_cntr_event (struct rdt_resource *r,struct rdtgroup *rdtgrp, + struct rdt_mon_domain *d, enum resctrl_event_id evtid) +{ + int index = MBM_EVENT_ARRAY_INDEX(evtid); + int cntr_id = rdtgrp->mon.cntr_id[index]; + int ret; + + /* + * Allocate a new counter id to the event if the counter is not + * assigned already. + */ + if (cntr_id == MON_CNTR_UNSET) { + cntr_id = mbm_cntr_alloc(r); + if (cntr_id < 0) { + rdt_last_cmd_puts("Out of MBM assignable counters\n"); + return -ENOSPC; + } + rdtgrp->mon.cntr_id[index] = cntr_id; + } + + if (!d) { + list_for_each_entry(d, &r->mon_domains, hdr.list) { + ret = resctrl_arch_config_cntr(r, d, evtid, rdtgrp->mon.rmid, + rdtgrp->closid, cntr_id, true); + if (ret)Quoted 8 lines
Quoted 3 lines
+ goto
out_done_assign;
+
+set_bit(cntr_id, d->mbm_cntr_map);The code pattern above is repeated four times in this work, twice in
rdtgroup_assign_cntr_event()and twice inrdtgroup_unassign_cntr_event(). This duplication should be avoided. It can be done in a function that also resets the architectural state.Quoted 1 line
Are you suggesting to combine
rdtgroup_assign_cntr_event()andrdtgroup_unassign_cntr_event()?
Quoted 8 lines
No. My comment was about the following pattern that is repeated four times:
... ret = resctrl_arch_config_cntr(...) if (ret) ... set_bit()/clear_bit() ...
ok.
Quoted 7 lines
Quoted 1 line
It can be done. We need a flag to tell if it is a assign or unassign.
There is already a flag that is used by
resctrl_arch_config_cntr(), the same parameters asresctrl_arch_config_cntr()can be used for a wrapper that just callsresctrl_arch_config_cntr()directly and uses that same flag to
select betweenset_bit()andclear_bit(). This wrapper can then also include the reset of architectural state.
ok. Got it, It will look like this.
+/*
+ * Wrapper to configure the counter in a domain.
+ */
+static int rdtgroup_config_cntr(struct rdt_resource *r,struct rdt_mon_domain *d,
+ enum resctrl_event_id evtid, u32 rmid, u32
closid,
+ u32 cntr_id, bool assign)
+{
+ int ret;
+
+ ret = resctrl_arch_config_cntr(r, d, evtid, rmid, closid, cntr_id,
assign);
+ if (ret)
+ return ret;
+
+ if (assign)
+ __set_bit(cntr_id, d->mbm_cntr_map);
+ else
+ __clear_bit(cntr_id, d->mbm_cntr_map);
+
+ /*
+ * Reset the architectural state so that reading of hardware + * counter is not considered as an overflow in next update. + */
+ resctrl_arch_reset_rmid(r, d, closid, rmid, evtid);
+
+ return ret;
+}
+
Quoted 5 lines
Also, I do not think we need atomic bitops here so these can be
__set_bit()and__clear_bit()that also matches how bits ofmbm_cntr_free_mapare managed earlier in series.
Signature · 2 lines
Thanks Babu Moger
Reinette Chatre
maintainerRe: [PATCH v8 17/25] x86/resctrl: Add the interface to assign/update counter assignment
Hi Babu,
Quoted 62 lines · Babu
Quoted 41 lines · Reinette Chatre
Quoted 38 lines · Babu
Quoted 28 lines · Reinette Chatre
Quoted 26 lines · Babu Moger
+ */
arch/x86/kernel/cpu/resctrl/rdtgroup.c +25
-0+int rdtgroup_assign_cntr_event (struct rdt_resource *r,struct rdtgroup *rdtgrp, + struct rdt_mon_domain *d, enum resctrl_event_id evtid) +{ + int index = MBM_EVENT_ARRAY_INDEX(evtid); + int cntr_id = rdtgrp->mon.cntr_id[index]; + int ret; + + /* + * Allocate a new counter id to the event if the counter is not + * assigned already. + */ + if (cntr_id == MON_CNTR_UNSET) { + cntr_id = mbm_cntr_alloc(r); + if (cntr_id < 0) { + rdt_last_cmd_puts("Out of MBM assignable counters\n"); + return -ENOSPC; + } + rdtgrp->mon.cntr_id[index] = cntr_id; + } + + if (!d) { + list_for_each_entry(d, &r->mon_domains, hdr.list) { + ret = resctrl_arch_config_cntr(r, d, evtid, rdtgrp->mon.rmid, + rdtgrp->closid, cntr_id, true); + if (ret)Quoted 8 lines
Quoted 3 lines
+ goto
out_done_assign;
+
+set_bit(cntr_id, d->mbm_cntr_map);The code pattern above is repeated four times in this work, twice in
rdtgroup_assign_cntr_event()and twice inrdtgroup_unassign_cntr_event(). This duplication should be avoided. It can be done in a function that also resets the architectural state.Quoted 1 line
Are you suggesting to combine
rdtgroup_assign_cntr_event()andrdtgroup_unassign_cntr_event()?Quoted 8 lines
No. My comment was about the following pattern that is repeated four times:
... ret = resctrl_arch_config_cntr(...) if (ret) ... set_bit()/clear_bit() ...ok.
Quoted 7 lines
Quoted 1 line
It can be done. We need a flag to tell if it is a assign or unassign.
There is already a flag that is used by
resctrl_arch_config_cntr(), the same parameters asresctrl_arch_config_cntr()can be used for a wrapper that just callsresctrl_arch_config_cntr()directly and uses that same flag to
select betweenset_bit()andclear_bit(). This wrapper can then also include the reset of architectural state.
Quoted 6 lines
ok. Got it, It will look like this.
+/*
+ * Wrapper to configure the counter in a domain.
+ */
Please replace comment with a description of what the function does.
Quoted 1 line
+static int rdtgroup_config_cntr(struct
rdt_resource*r,struct
While it keeps being a challenge to get naming right I do think this can start by replacing "rdtgroup" with "resctrl" (specifically, "rdtgroup_config_cntr() -> resctrl_config_cntr()") because, as seen with the parameters passed, this has nothing to do with rdtgroup.
Quoted 16 lines
rdt_mon_domain*d,
+ enumresctrl_event_idevtid, u32 rmid, u32
closid,arch/x86/kernel/cpu/resctrl/rdtgroup.c +5
-0+ u32 cntr_id, bool assign) +{ + int ret; + + ret = resctrl_arch_config_cntr(r, d, evtid, rmid, closid, cntr_id,assign);
(quoted diff) +7
-0+ if (ret) + return ret; + + if (assign) + __set_bit(cntr_id, d->mbm_cntr_map); + else + __clear_bit(cntr_id, d->mbm_cntr_map);
Review
Quoted 10 lines
(quoted diff) +10
-0+ + /* + * Reset the architectural state so that reading of hardware + * counter is not considered as an overflow in next update. + */ + resctrl_arch_reset_rmid(r, d, closid, rmid, evtid); + + return ret; +} +
Yes, this looks good. Thank you.
Reinette
Re: [PATCH v8 17/25] x86/resctrl: Add the interface to assign/update counter assignment
Hi Reinette,
Quoted 65 lines · Reinette Chatre
Hi Babu,
Quoted 62 lines · Babu
Quoted 41 lines · Reinette Chatre
Quoted 38 lines · Babu
Quoted 28 lines · Reinette Chatre
Quoted 26 lines · Babu Moger
+ */
arch/x86/kernel/cpu/resctrl/rdtgroup.c +25
-0+int rdtgroup_assign_cntr_event (struct rdt_resource *r,struct rdtgroup *rdtgrp, + struct rdt_mon_domain *d, enum resctrl_event_id evtid) +{ + int index = MBM_EVENT_ARRAY_INDEX(evtid); + int cntr_id = rdtgrp->mon.cntr_id[index]; + int ret; + + /* + * Allocate a new counter id to the event if the counter is not + * assigned already. + */ + if (cntr_id == MON_CNTR_UNSET) { + cntr_id = mbm_cntr_alloc(r); + if (cntr_id < 0) { + rdt_last_cmd_puts("Out of MBM assignable counters\n"); + return -ENOSPC; + } + rdtgrp->mon.cntr_id[index] = cntr_id; + } + + if (!d) { + list_for_each_entry(d, &r->mon_domains, hdr.list) { + ret = resctrl_arch_config_cntr(r, d, evtid, rdtgrp->mon.rmid, + rdtgrp->closid, cntr_id, true); + if (ret)Quoted 8 lines
Quoted 3 lines
+ goto
out_done_assign;
+
+set_bit(cntr_id, d->mbm_cntr_map);The code pattern above is repeated four times in this work, twice in
rdtgroup_assign_cntr_event()and twice inrdtgroup_unassign_cntr_event(). This duplication should be avoided. It can be done in a function that also resets the architectural state.Quoted 1 line
Are you suggesting to combine
rdtgroup_assign_cntr_event()andrdtgroup_unassign_cntr_event()?Quoted 8 lines
No. My comment was about the following pattern that is repeated four times:
... ret = resctrl_arch_config_cntr(...) if (ret) ... set_bit()/clear_bit() ...ok.
Quoted 7 lines
Quoted 1 line
It can be done. We need a flag to tell if it is a assign or unassign.
There is already a flag that is used by
resctrl_arch_config_cntr(), the same parameters asresctrl_arch_config_cntr()can be used for a wrapper that just callsresctrl_arch_config_cntr()directly and uses that same flag to
select betweenset_bit()andclear_bit(). This wrapper can then also include the reset of architectural state.
Quoted 8 lines
Quoted 6 lines
ok. Got it, It will look like this.
+/*
+ * Wrapper to configure the counter in a domain.
+ */Please replace comment with a description of what the function does.
sure.
Quoted 7 lines
Quoted 2 lines
+static int rdtgroup_config_cntr(struct
rdt_resource*r,structWhile it keeps being a challenge to get naming right I do think this can start by replacing "rdtgroup" with "resctrl" (specifically, "
rdtgroup_config_cntr()->resctrl_config_cntr()") because, as seen with the parameters passed, this has nothing to do with rdtgroup.
Sure.
Quoted 21 lines
Quoted 21 lines
rdt_mon_domain*d,
+ enumresctrl_event_idevtid, u32 rmid, u32
closid,arch/x86/kernel/cpu/resctrl/rdtgroup.c +5
-0+ u32 cntr_id, bool assign) +{ + int ret; + + ret = resctrl_arch_config_cntr(r, d, evtid, rmid, closid, cntr_id,assign);
(quoted diff) +11
-0+ if (ret) + return ret; + + if (assign) + __set_bit(cntr_id, d->mbm_cntr_map); + else + __clear_bit(cntr_id, d->mbm_cntr_map); + + /* + * Reset the architectural state so that reading of hardware + * counter is not considered as an overflow in next update.
Quoted 9 lines
arch/x86/kernel/cpu/resctrl/rdtgroup.c int resctrl_arch_config_cntr(struct rdt_resource *r, struct rdt_mon_domain *d, · line 1947
Quoted 6 lines
arch/x86/kernel/cpu/resctrl/rdtgroup.c +6
-0+*/ +resctrl_arch_reset_rmid (r, d, closid, rmid, evtid); + +return ret; +} +Yes, this looks good. Thank you.
Thanks-
- Babu Moger
Babu Moger
reviewerPATCH v8 18/25 x86/resctrl: Add the interface to unassign a MBM counter
In reply to Babu MogerThe mbm_cntr_assign mode provides a limited number of hardware counters that can be assigned to an RMID-event pair to monitor bandwidth while assigned. If all counters are in use, the kernel will show an error
message: "Out of MBM assignable counters" when a new assignment is requested. To make space for a new assignment, users must unassign an already assigned counter.
Introduce an interface that allows for the unassignment of counter IDs from both the group and the domain. Additionally, ensure that the global counter is released if it is no longer assigned to any domains.
---
v8: Renamed rdtgroup_mbm_cntr_is_assigned to mbm_cntr_assigned_to_domain
Added return error handling in resctrl_arch_config_cntr().
v7: Merged rdtgroup_unassign_cntr and rdtgroup_free_cntr functions.
Renamed rdtgroup_mbm_cntr_test() to rdtgroup_mbm_cntr_is_assigned(). Reworded the commit log little bit.
v6: Removed mbm_cntr_free from this patch.
Added counter test in all the domains and free if it is not assigned to any domains.
v5: Few name changes to match cntr_id.
Changed the function names to rdtgroup_unassign_cntr More comments on commit log.
v4: Added domain specific unassign feature.
Few name changes.
v3: Removed the static from the prototype of rdtgroup_unassign_abmc.
The function is not called directly from user anymore. These changes are related to global assignment interface.
v2: No changes.
--- arch/x86/kernel/cpu/resctrl/internal.h | 2 + arch/x86/kernel/cpu/resctrl/rdtgroup.c | 56 ++++++++++++++++++++++++++ 2 files changed, 58 insertions(+)
arch/x86/kernel/cpu/resctrl/internal.h +2 -0
@@ -717,6 +717,8 @@ int resctrl_arch_config_cntr(struct rdt_resource *r, struct rdt_mon_domain *d, u32 cntr_id, bool assign);int rdtgroup_assign_cntr_event (struct rdt_resource *r,struct rdtgroup *rdtgrp,struct rdt_mon_domain *d,enum resctrl_event_id evtid); +int rdtgroup_unassign_cntr_event (struct rdt_resource *r,struct rdtgroup *rdtgrp, +struct rdt_mon_domain *d,enum resctrl_event_id evtid);void rdt_staged_configs_clear (void );bool closid_allocated (unsigned int closid);int resctrl_find_cleanest_closid (void );
arch/x86/kernel/cpu/resctrl/rdtgroup.c +56 -0
62 lines not on this page · raw message
Signature · 1 line
2.34.1
Reinette Chatre
maintainerRe: [PATCH v8 18/25] x86/resctrl: Add the interface to unassign a MBM counter
Hi Babu,
Quoted 10 lines · Babu Moger
The
mbm_cntr_assignmode provides a limited number of hardware counters that can be assigned to an RMID-event pair to monitor bandwidth while assigned. If all counters are in use, the kernel will show an error
message: "Out of MBM assignable counters" when a new assignment is requested. To make space for a new assignment, users must unassign an already assigned counter.Introduce an interface that allows for the unassignment of counter IDs from both the group and the domain. Additionally, ensure that the global counter is released if it is no longer assigned to any domains.
Needs imperative tone ... "Release the global counter ..."
Quoted 3 lines
Signed-off-by Babu Moger <[email protected]>---
...
Quoted 29 lines
--- arch/x86/kernel/cpu/resctrl/internal.h | 2 + arch/x86/kernel/cpu/resctrl/rdtgroup.c | 56 ++++++++++++++++++++++++++ 2 files changed, 58 insertions(+)arch/x86/kernel/cpu/resctrl/internal.h +2
-0@@ -717,6 +717,8 @@ int resctrl_arch_config_cntr(struct rdt_resource *r, struct rdt_mon_domain *d, u32 cntr_id, bool assign);int rdtgroup_assign_cntr_event (struct rdt_resource *r,struct rdtgroup *rdtgrp,struct rdt_mon_domain *d,enum resctrl_event_id evtid); +int rdtgroup_unassign_cntr_event (struct rdt_resource *r,struct rdtgroup *rdtgrp, +struct rdt_mon_domain *d,enum resctrl_event_id evtid);void rdt_staged_configs_clear (void );bool closid_allocated (unsigned int closid);int resctrl_find_cleanest_closid (void );arch/x86/kernel/cpu/resctrl/rdtgroup.c +3
-0@@ -1945,6 +1945,62 @@ int rdtgroup_assign_cntr_event(struct rdt_resource *r, struct rdtgroup *rdtgrp,return ret; } +static bool mbm_cntr_assigned_to_domain (struct rdt_resource *r,u32 cntr_id) +{ +struct rdt_mon_domain *d;
arch/x86/kernel/cpu/resctrl/rdtgroup.c int rdtgroup_assign_cntr_event(struct rdt_resource *r, struct rdtgroup *rdtgrp, · line 1960
Quoted 10 lines
arch/x86/kernel/cpu/resctrl/rdtgroup.c +10
-0+ +list_for_each_entry (d, &r->mon_domains, hdr.list) +if (test_bit (cntr_id, d->mbm_cntr_map)) +return 1 ; + +return 0 ; +} + +/* +* Unassign a hardware counter from the domain and the group.
Not sure ... maybe "Unassign a hardware counter associated with @evtid from the domain and the group."?
arch/x86/kernel/cpu/resctrl/rdtgroup.c int rdtgroup_assign_cntr_event(struct rdt_resource *r, struct rdtgroup *rdtgrp, · line 1961
Quoted 1 line
arch/x86/kernel/cpu/resctrl/rdtgroup.c +1
-0+* Counter will be unassigned in all the domains if rdt_mon_domain is NULL
Please use imperative tone: "Unassign the counter from all the domains ...."
arch/x86/kernel/cpu/resctrl/rdtgroup.c int rdtgroup_assign_cntr_event(struct rdt_resource *r, struct rdtgroup *rdtgrp, · line 1962
Quoted 1 line
arch/x86/kernel/cpu/resctrl/rdtgroup.c +1
-0+* else the counter will be assigned to specific domain.
copy&paste error?
"assigned to specific domain" -> "unassign from specific domain"?
arch/x86/kernel/cpu/resctrl/rdtgroup.c int rdtgroup_assign_cntr_event(struct rdt_resource *r, struct rdtgroup *rdtgrp, · line 1963
Quoted 1 line
arch/x86/kernel/cpu/resctrl/rdtgroup.c +1
-0+* Global counter will be freed once it is unassigned from all the domains.
Needs imperative tone.
Quoted 18 lines
arch/x86/kernel/cpu/resctrl/rdtgroup.c +18
-0+*/ +int rdtgroup_unassign_cntr_event (struct rdt_resource *r,struct rdtgroup *rdtgrp, +struct rdt_mon_domain *d,enum resctrl_event_id evtid) +{ +int index =MBM_EVENT_ARRAY_INDEX (evtid); +int cntr_id = rdtgrp->mon.cntr_id[index]; +int ret; + +/* Return early if the counter is unassigned already */ +if (cntr_id ==MON_CNTR_UNSET ) +return 0 ; + +if (!d) { +list_for_each_entry (d, &r->mon_domains, hdr.list) { + ret =resctrl_arch_config_cntr (r, d, evtid, rdtgrp->mon.rmid, + rdtgrp->closid, cntr_id, false); +if (ret) + goto out_done_unassign;
arch/x86/kernel/cpu/resctrl/rdtgroup.c int rdtgroup_assign_cntr_event(struct rdt_resource *r, struct rdtgroup *rdtgrp, · line 1991
Quoted 10 lines
arch/x86/kernel/cpu/resctrl/rdtgroup.c +10
-0+ +clear_bit (cntr_id, d->mbm_cntr_map); + } + }else { + ret =resctrl_arch_config_cntr (r, d, evtid, rdtgrp->mon.rmid, + rdtgrp->closid, cntr_id, false); +if (ret) + goto out_done_unassign; + +clear_bit (cntr_id, d->mbm_cntr_map);
Please see comment to previous patch about the duplicate snippets. Snippets can be replaced with single function that also resets architectural state.
arch/x86/kernel/cpu/resctrl/rdtgroup.c int rdtgroup_assign_cntr_event(struct rdt_resource *r, struct rdtgroup *rdtgrp, · line 1994
Quoted 3 lines
arch/x86/kernel/cpu/resctrl/rdtgroup.c +3
-0+ } + +/* Update the counter bitmap */
What is the update?
arch/x86/kernel/cpu/resctrl/rdtgroup.c int rdtgroup_assign_cntr_event(struct rdt_resource *r, struct rdtgroup *rdtgrp, · line 2006
Quoted 12 lines
arch/x86/kernel/cpu/resctrl/rdtgroup.c +9
-0+if (!mbm_cntr_assigned_to_domain (r, cntr_id)) { +mbm_cntr_free (r, cntr_id); + rdtgrp->mon.cntr_id[index] =MON_CNTR_UNSET ; + } + +out_done_unassign: +return ret; +} +/* rdtgroup information files for one cache resource. */ static struct rftype res_common_files[] = { {
Reinette
Re: [PATCH v8 18/25] x86/resctrl: Add the interface to unassign a MBM counter
Hi Reinette,
Quoted 10 lines · Reinette Chatre
Hi Babu,
Quoted 7 lines · Babu Moger
The
mbm_cntr_assignmode provides a limited number of hardware counters that can be assigned to an RMID-event pair to monitor bandwidth while assigned. If all counters are in use, the kernel will show an error
message: "Out of MBM assignable counters" when a new assignment is requested. To make space for a new assignment, users must unassign an already assigned counter.
Quoted 5 lines
Quoted 3 lines
Introduce an interface that allows for the unassignment of counter IDs from both the group and the domain. Additionally, ensure that the global counter is released if it is no longer assigned to any domains.
Needs imperative tone ... "Release the global counter ..."
sure.
Quoted 32 lines
Quoted 4 lines
Signed-off-by Babu Moger <[email protected]>---
...
Quoted 26 lines
--- arch/x86/kernel/cpu/resctrl/internal.h | 2 + arch/x86/kernel/cpu/resctrl/rdtgroup.c | 56 ++++++++++++++++++++++++++ 2 files changed, 58 insertions(+)arch/x86/kernel/cpu/resctrl/internal.h +2
-0@@ -717,6 +717,8 @@ int resctrl_arch_config_cntr(struct rdt_resource *r, struct rdt_mon_domain *d, u32 cntr_id, bool assign);int rdtgroup_assign_cntr_event (struct rdt_resource *r,struct rdtgroup *rdtgrp,struct rdt_mon_domain *d,enum resctrl_event_id evtid); +int rdtgroup_unassign_cntr_event (struct rdt_resource *r,struct rdtgroup *rdtgrp, +struct rdt_mon_domain *d,enum resctrl_event_id evtid);void rdt_staged_configs_clear (void );bool closid_allocated (unsigned int closid);int resctrl_find_cleanest_closid (void );arch/x86/kernel/cpu/resctrl/rdtgroup.c +0
-0@@ -1945,6 +1945,62 @@ int rdtgroup_assign_cntr_event(struct rdt_resource *r, struct rdtgroup *rdtgrp,return ret; }
Quoted 16 lines
arch/x86/kernel/cpu/resctrl/rdtgroup.c int rdtgroup_assign_cntr_event(struct rdt_resource *r, struct rdtgroup *rdtgrp, · line 1960
Quoted 13 lines
arch/x86/kernel/cpu/resctrl/rdtgroup.c +13
-0+static bool mbm_cntr_assigned_to_domain (struct rdt_resource *r,u32 cntr_id) +{ +struct rdt_mon_domain *d; + +list_for_each_entry (d, &r->mon_domains, hdr.list) +if (test_bit (cntr_id, d->mbm_cntr_map)) +return 1 ; + +return 0 ; +} + +/* +* Unassign a hardware counter from the domain and the group. Not sure ... maybe "Unassign a hardware counter associated with @evtid from the domain and the group."?
ok.
Quoted 4 lines
arch/x86/kernel/cpu/resctrl/rdtgroup.c int rdtgroup_assign_cntr_event(struct rdt_resource *r, struct rdtgroup *rdtgrp, · line 1961
Quoted 2 lines
arch/x86/kernel/cpu/resctrl/rdtgroup.c +1
-0+* Counter will be unassigned in all the domains if rdt_mon_domain is NULL Please use imperative tone: "Unassign the counter from all the domains ...."
ok.
Quoted 5 lines
arch/x86/kernel/cpu/resctrl/rdtgroup.c int rdtgroup_assign_cntr_event(struct rdt_resource *r, struct rdtgroup *rdtgrp, · line 1962
Quoted 2 lines
arch/x86/kernel/cpu/resctrl/rdtgroup.c +1
-0+* else the counter will be assigned to specific domain. copy&paste error?
"assigned to specific domain" -> "unassign from specific domain"?
ok.
Quoted 28 lines
arch/x86/kernel/cpu/resctrl/rdtgroup.c int rdtgroup_assign_cntr_event(struct rdt_resource *r, struct rdtgroup *rdtgrp, · line 1963
Quoted 2 lines
arch/x86/kernel/cpu/resctrl/rdtgroup.c +1
-0+* Global counter will be freed once it is unassigned from all the domains. Needs imperative tone.
Quoted 23 lines
arch/x86/kernel/cpu/resctrl/rdtgroup.c +23
-0+*/ +int rdtgroup_unassign_cntr_event (struct rdt_resource *r,struct rdtgroup *rdtgrp, +struct rdt_mon_domain *d,enum resctrl_event_id evtid) +{ +int index =MBM_EVENT_ARRAY_INDEX (evtid); +int cntr_id = rdtgrp->mon.cntr_id[index]; +int ret; + +/* Return early if the counter is unassigned already */ +if (cntr_id ==MON_CNTR_UNSET ) +return 0 ; + +if (!d) { +list_for_each_entry (d, &r->mon_domains, hdr.list) { + ret =resctrl_arch_config_cntr (r, d, evtid, rdtgrp->mon.rmid, + rdtgrp->closid, cntr_id, false); +if (ret) + goto out_done_unassign; + +clear_bit (cntr_id, d->mbm_cntr_map); + } + }else { + ret = resctrl_arch_config_cntr(r, d, evtid, rdtgrp->mon.rmid,
Quoted 8 lines
arch/x86/kernel/cpu/resctrl/rdtgroup.c int rdtgroup_assign_cntr_event(struct rdt_resource *r, struct rdtgroup *rdtgrp, · line 1983
Quoted 5 lines
arch/x86/kernel/cpu/resctrl/rdtgroup.c +5
-0+ rdtgrp->closid, cntr_id, false); +if (ret) + goto out_done_unassign; + +clear_bit (cntr_id, d->mbm_cntr_map);Please see comment to previous patch about the duplicate snippets. Snippets can be replaced with single function that also resets architectural state.
Sure.
will combine rdtgroup_assign_cntr_event() andrdtgroup_unassign_cntr_event().
I need to rename the function. How about resctrl_configure_cntr_event()?
Quoted 6 lines
arch/x86/kernel/cpu/resctrl/rdtgroup.c int rdtgroup_assign_cntr_event(struct rdt_resource *r, struct rdtgroup *rdtgrp, · line 1994
Quoted 4 lines
arch/x86/kernel/cpu/resctrl/rdtgroup.c +3
-0+ } + +/* Update the counter bitmap */ What is the update?
Clear the counter bitmap.
Quoted 17 lines
arch/x86/kernel/cpu/resctrl/rdtgroup.c int rdtgroup_assign_cntr_event(struct rdt_resource *r, struct rdtgroup *rdtgrp, · line 2006
Quoted 13 lines
arch/x86/kernel/cpu/resctrl/rdtgroup.c +9
-0+if (!mbm_cntr_assigned_to_domain (r, cntr_id)) { +mbm_cntr_free (r, cntr_id); + rdtgrp->mon.cntr_id[index] =MON_CNTR_UNSET ; + } + +out_done_unassign: +return ret; +} +/* rdtgroup information files for one cache resource. */ static struct rftype res_common_files[] = { {Reinette
Signature · 1 line
- Babu Moger
Reinette Chatre
maintainerRe: [PATCH v8 18/25] x86/resctrl: Add the interface to unassign a MBM counter
Hi Babu,
Quoted 23 lines · Babu
Quoted 13 lines · Reinette Chatre
Quoted 11 lines · Babu Moger
+ if (!d) {
+list_for_each_entry(d, &r->mon_domains, hdr.list){
+ ret = resctrl_arch_config_cntr(r, d, evtid,rdtgrp->mon.rmid, +rdtgrp->closid,cntr_id, false); + if (ret)
+ gotoout_done_unassign;
+
+clear_bit(cntr_id, d->mbm_cntr_map);
+ }
+ } else {
+ ret = resctrl_arch_config_cntr(r, d, evtid,rdtgrp->mon.rmid,Quoted 8 lines
Quoted 5 lines
+
rdtgrp->closid,cntr_id, false); + if (ret)
+ gotoout_done_unassign;
+
+clear_bit(cntr_id, d->mbm_cntr_map);Please see comment to previous patch about the duplicate snippets. Snippets can be replaced with single function that also resets architectural state.
Quoted 4 lines
Sure.
will combine
rdtgroup_assign_cntr_event()andrdtgroup_unassign_cntr_event().
That is not what I suggested. I attempted to clarify in response to patch with original feedback:
https://lore.kernel.org/all/[email protected]/
Quoted 5 lines
I need to rename the function. How about
resctrl_configure_cntr_event()?Quoted 1 line
Quoted 7 lines
Quoted 5 lines
Quoted 3 lines
+ }
+
+ /* Update the counter bitmap */What is the update?
Clear the counter bitmap.
Could you please update the comment to be more specific? What is written can be seen from code.
Reinette
Babu Moger
reviewerPATCH v8 19/25 x86/resctrl: Auto assign/unassign counters when mbm_cntr_assign is enabled
In reply to Babu MogerAssign/unassign counters on resctrl group creation/deletion. Two counters are required per group, one for MBM total event and one for MBM local event.
There are a limited number of counters available for assignment. If these counters are exhausted, the kernel will display the error message: "Out of MBM assignable counters". However, it is not necessary to fail the creation of a group due to assignment failures. Users have the flexibility to modify the assignments at a later time.
---
v8: Renamed rdtgroup_assign_grp to rdtgroup_assign_cntrs.
Renamed rdtgroup_unassign_grp to rdtgroup_unassign_cntrs. Fixed the problem with unassigning the child MON groups of CTRL_MON group.
v7: Reworded the commit message.
Removed the reference of ABMC with mbm_cntr_assign. Renamed the function rdtgroup_assign_cntrs to rdtgroup_assign_grp.
v6: Removed the redundant comments on all the calls of
rdtgroup_assign_cntrs. Updated the commit message.
Dropped printing error message on every call of rdtgroup_assign_cntrs.
v5: Removed the code to enable/disable ABMC during the mount.
That will be another patch.
Added arch callers to get the arch specific data. Renamed fuctions to match the other abmc function. Added code comments for assignment failures.
v4: Few name changes based on the upstream discussion.
Commit message update.
v3: This is a new patch. Patch addresses the upstream comment to enable
ABMC feature by default if the feature is available.
--- arch/x86/kernel/cpu/resctrl/rdtgroup.c | 64 ++++++++++++++++++++++++++ 1 file changed, 64 insertions(+)
arch/x86/kernel/cpu/resctrl/rdtgroup.c +64 -0
@@ -2875,6 +2875,52 @@ static void schemata_list_destroy(void) } } +/* +* Called when a new group is created. If `mbm_cntr_assign` mode is enabled, +* counters are automatically assigned. Each group requires two counters: +* one for the total event and one for the local event. Due to the limited +* number of counters, assignments may fail in some cases. However, it is +* not necessary to fail the group creation. Users have the option to +* modify the assignments after the group has been created. +*/ +static int rdtgroup_assign_cntrs (struct rdtgroup *rdtgrp) +{ +struct rdt_resource *r = &rdt_resources_all[RDT_RESOURCE_L3 ].r_resctrl; +int ret =0 ; + +if (!resctrl_arch_mbm_cntr_assign_enabled (r)) +return 0 ; + +if (is_mbm_total_enabled ()) + ret =rdtgroup_assign_cntr_event (r, rdtgrp,NULL ,QOS_L3_MBM_TOTAL_EVENT_ID ); + +if (!ret &&is_mbm_local_enabled ()) + ret =rdtgroup_assign_cntr_event (r, rdtgrp,NULL ,QOS_L3_MBM_LOCAL_EVENT_ID ); + +return ret; +} + +/* +* Called when a group is deleted. Counters are unassigned if it was in +* assigned state. +*/ +static int rdtgroup_unassign_cntrs (struct rdtgroup *rdtgrp) +{ +struct rdt_resource *r = &rdt_resources_all[RDT_RESOURCE_L3 ].r_resctrl; +int ret =0 ; + +if (!resctrl_arch_mbm_cntr_assign_enabled (r)) +return 0 ; + +if (is_mbm_total_enabled ()) + ret =rdtgroup_unassign_cntr_event (r, rdtgrp,NULL ,QOS_L3_MBM_TOTAL_EVENT_ID ); + +if (!ret &&is_mbm_local_enabled ()) + ret =rdtgroup_unassign_cntr_event (r, rdtgrp,NULL ,QOS_L3_MBM_LOCAL_EVENT_ID ); + +return ret; +} +static int rdt_get_tree (struct fs_context *fc) {struct rdt_fs_context *ctx =rdt_fc2context (fc);
@@ -2934,6 +2980,8 @@ static int rdt_get_tree(struct fs_context *fc)if (ret <0 ) goto out_mongrp; rdtgroup_default.mon.mon_data_kn = kn_mondata; + +rdtgroup_assign_cntrs (&rdtgroup_default); } ret =rdt_pseudo_lock_init ();
@@ -2964,6 +3012,7 @@ static int rdt_get_tree(struct fs_context *fc) out_psl:rdt_pseudo_lock_release (); out_mondata: +rdtgroup_unassign_cntrs (&rdtgroup_default);if (resctrl_arch_mon_capable ())kernfs_remove (kn_mondata); out_mongrp:
@@ -3144,6 +3193,7 @@ static void free_all_child_rdtgrp(struct rdtgroup *rdtgrp) head = &rdtgrp->mon.crdtgrp_list;list_for_each_entry_safe (sentry, stmp, head, mon.crdtgrp_list) { +rdtgroup_unassign_cntrs (sentry);free_rmid (sentry->closid, sentry->mon.rmid);list_del (&sentry->mon.crdtgrp_list);
@@ -3184,6 +3234,8 @@ static void rmdir_all_sub(void)cpumask_or (&rdtgroup_default.cpu_mask, &rdtgroup_default.cpu_mask, &rdtgrp->cpu_mask); +rdtgroup_unassign_cntrs (rdtgrp); +free_rmid (rdtgrp->closid, rdtgrp->mon.rmid);kernfs_remove (rdtgrp->kn);
@@ -3223,6 +3275,8 @@ static void rdt_kill_sb(struct super_block *sb)resctrl_arch_disable_alloc ();if (resctrl_arch_mon_capable ())resctrl_arch_disable_mon (); + +rdtgroup_unassign_cntrs (&rdtgroup_default); resctrl_mounted = false;kernfs_kill_sb (sb);mutex_unlock (&rdtgroup_mutex);
@@ -3814,6 +3868,8 @@ static int rdtgroup_mkdir_mon(struct kernfs_node *parent_kn, goto out_unlock; } +rdtgroup_assign_cntrs (rdtgrp); +kernfs_activate (rdtgrp->kn);/*
@@ -3858,6 +3914,8 @@ static int rdtgroup_mkdir_ctrl_mon(struct kernfs_node *parent_kn,if (ret) goto out_closid_free; +rdtgroup_assign_cntrs (rdtgrp); +kernfs_activate (rdtgrp->kn); ret =rdtgroup_init_alloc (rdtgrp);
@@ -3883,6 +3941,7 @@ static int rdtgroup_mkdir_ctrl_mon(struct kernfs_node *parent_kn, out_del_list:list_del (&rdtgrp->rdtgroup_list); out_rmid_free: +rdtgroup_unassign_cntrs (rdtgrp);mkdir_rdt_prepare_rmid_free (rdtgrp); out_closid_free:closid_free (closid);
@@ -3953,6 +4012,9 @@ static int rdtgroup_rmdir_mon(struct rdtgroup *rdtgrp, cpumask_var_t tmpmask)update_closid_rmid (tmpmask,NULL ); rdtgrp->flags =RDT_DELETED ; + +rdtgroup_unassign_cntrs (rdtgrp); +free_rmid (rdtgrp->closid, rdtgrp->mon.rmid);/*
@@ -3999,6 +4061,8 @@ static int rdtgroup_rmdir_ctrl(struct rdtgroup *rdtgrp, cpumask_var_t tmpmask)cpumask_or (tmpmask, tmpmask, &rdtgrp->cpu_mask);update_closid_rmid (tmpmask,NULL ); +rdtgroup_unassign_cntrs (rdtgrp); +free_rmid (rdtgrp->closid, rdtgrp->mon.rmid);closid_free (rdtgrp->closid);
Signature · 1 line
2.34.1
Tony Luck
maintainerRe: [PATCH v8 19/25] x86/resctrl: Auto assign/unassign counters when mbm_cntr_assign is enabled
Quoted 11 lines · Babu Moger
arch/x86/kernel/cpu/resctrl/rdtgroup.c +11
-0+/* +* Called when a new group is created. If `mbm_cntr_assign` mode is enabled, +* counters are automatically assigned. Each group requires two counters: +* one for the total event and one for the local event. Due to the limited +* number of counters, assignments may fail in some cases. However, it is +* not necessary to fail the group creation. Users have the option to +* modify the assignments after the group has been created. +*/ +static int rdtgroup_assign_cntrs (struct rdtgroup *rdtgrp) +{ +struct rdt_resource *r = &rdt_resources_all[RDT_RESOURCE_L3 ].r_resctrl;
arch/x86/kernel/cpu/resctrl/rdtgroup.c static void schemata_list_destroy(void) · line 2898
Quoted 10 lines
arch/x86/kernel/cpu/resctrl/rdtgroup.c +10
-0+int ret =0 ; + +if (!resctrl_arch_mbm_cntr_assign_enabled (r)) +return 0 ; + +if (is_mbm_total_enabled ()) + ret =rdtgroup_assign_cntr_event (r, rdtgrp,NULL ,QOS_L3_MBM_TOTAL_EVENT_ID ); + +if (!ret &&is_mbm_local_enabled ()) + ret =rdtgroup_assign_cntr_event (r, rdtgrp,NULL ,QOS_L3_MBM_LOCAL_EVENT_ID );
This overwrites the value from allocating the counter for total event.
Review
Quoted 2 lines
(quoted diff) +2
-0+ + return ret;
But none of the callers check the return. Indeed it is ok (and expected) that counter allocation can fail.
Just make this a "void" function and delete the "ret" local variable.
Quoted 11 lines
arch/x86/kernel/cpu/resctrl/rdtgroup.c +11
-0+} + +/* +* Called when a group is deleted. Counters are unassigned if it was in +* assigned state. +*/ +static int rdtgroup_unassign_cntrs (struct rdtgroup *rdtgrp) +{ +struct rdt_resource *r = &rdt_resources_all[RDT_RESOURCE_L3 ].r_resctrl; +int ret =0 ; +
arch/x86/kernel/cpu/resctrl/rdtgroup.c static void schemata_list_destroy(void) · line 2921
Quoted 10 lines
arch/x86/kernel/cpu/resctrl/rdtgroup.c +10
-0+if (!resctrl_arch_mbm_cntr_assign_enabled (r)) +return 0 ; + +if (is_mbm_total_enabled ()) + ret =rdtgroup_unassign_cntr_event (r, rdtgrp,NULL ,QOS_L3_MBM_TOTAL_EVENT_ID ); + +if (!ret &&is_mbm_local_enabled ()) + ret =rdtgroup_unassign_cntr_event (r, rdtgrp,NULL ,QOS_L3_MBM_LOCAL_EVENT_ID ); + +return ret;
Ditto. No caller checks. Make this a void function. Dig down the call chain here. It looks like rdtgroup_unassign_cntr_event() can't fail, so it should be a void function too. Ditto resctrl_arch_config_cntr()
Quoted 1 line
+}
-Tony
Re: [PATCH v8 19/25] x86/resctrl: Auto assign/unassign counters when mbm_cntr_assign is enabled
Hi Tony,
Quoted 16 lines · Tony Luck
Quoted 15 lines · Babu Moger
arch/x86/kernel/cpu/resctrl/rdtgroup.c +15
-0+/* +* Called when a new group is created. If `mbm_cntr_assign` mode is enabled, +* counters are automatically assigned. Each group requires two counters: +* one for the total event and one for the local event. Due to the limited +* number of counters, assignments may fail in some cases. However, it is +* not necessary to fail the group creation. Users have the option to +* modify the assignments after the group has been created. +*/ +static int rdtgroup_assign_cntrs (struct rdtgroup *rdtgrp) +{ +struct rdt_resource *r = &rdt_resources_all[RDT_RESOURCE_L3 ].r_resctrl; +int ret =0 ; + +if (!resctrl_arch_mbm_cntr_assign_enabled (r)) +return 0 ;
Quoted 8 lines
arch/x86/kernel/cpu/resctrl/rdtgroup.c static void schemata_list_destroy(void) · line 2898
Quoted 6 lines
arch/x86/kernel/cpu/resctrl/rdtgroup.c +6
-0+ +if (is_mbm_total_enabled ()) + ret =rdtgroup_assign_cntr_event (r, rdtgrp,NULL ,QOS_L3_MBM_TOTAL_EVENT_ID ); + +if (!ret &&is_mbm_local_enabled ()) + ret =rdtgroup_assign_cntr_event (r, rdtgrp,NULL ,QOS_L3_MBM_LOCAL_EVENT_ID );This overwrites the value from allocating the counter for total event.
Total event and local events have two different indexes. Can you please elaborate?
Quoted 8 lines
Review
Quoted 3 lines
(quoted diff) +2
-0+ + return ret;But none of the callers check the return. Indeed it is ok (and expected) that counter allocation can fail.
Just make this a "void" function and delete the "ret" local variable.
Comment below.
Quoted 17 lines
Quoted 17 lines
arch/x86/kernel/cpu/resctrl/rdtgroup.c +17
-0+} + +/* +* Called when a group is deleted. Counters are unassigned if it was in +* assigned state. +*/ +static int rdtgroup_unassign_cntrs (struct rdtgroup *rdtgrp) +{ +struct rdt_resource *r = &rdt_resources_all[RDT_RESOURCE_L3 ].r_resctrl; +int ret =0 ; + +if (!resctrl_arch_mbm_cntr_assign_enabled (r)) +return 0 ; + +if (is_mbm_total_enabled ()) + ret =rdtgroup_unassign_cntr_event (r, rdtgrp,NULL ,QOS_L3_MBM_TOTAL_EVENT_ID ); +
Quoted 8 lines
arch/x86/kernel/cpu/resctrl/rdtgroup.c static void schemata_list_destroy(void) · line 2921
Quoted 4 lines
arch/x86/kernel/cpu/resctrl/rdtgroup.c +4
-0+if (!ret &&is_mbm_local_enabled ()) + ret =rdtgroup_unassign_cntr_event (r, rdtgrp,NULL ,QOS_L3_MBM_LOCAL_EVENT_ID ); + +return ret;Ditto. No caller checks. Make this a void function. Dig down the call chain here. It looks like
rdtgroup_unassign_cntr_event()can't fail, so it should be a void function too. Dittoresctrl_arch_config_cntr()
It was started a void function. In this case all the call sequence return 0. There is a possibility that other architectures can return failure(in arch calls resctrl_arch_config_cntr()). Keeping that in mind we added the check to handle the return values. Hope that helps. Thanks
Signature · 1 line
- Babu Moger
Luck, Tony
maintainerRE: [PATCH v8 19/25] x86/resctrl: Auto assign/unassign counters when mbm_cntr_assign is enabled
Quoted 10 lines
Quoted 10 lines
arch/x86/kernel/cpu/resctrl/rdtgroup.c static void schemata_list_destroy(void) · line 2895
Quoted 10 lines
arch/x86/kernel/cpu/resctrl/rdtgroup.c +10
-0+static int rdtgroup_assign_cntrs (struct rdtgroup *rdtgrp) +{ +struct rdt_resource *r = &rdt_resources_all[RDT_RESOURCE_L3 ].r_resctrl; +int ret =0 ; + +if (!resctrl_arch_mbm_cntr_assign_enabled (r)) +return 0 ; + +if (is_mbm_total_enabled ()) + ret =rdtgroup_assign_cntr_event (r, rdtgrp,NULL ,QOS_L3_MBM_TOTAL_EVENT_ID );
Consider that this call fails. "ret" indicates failure to allocate.
Quoted 3 lines
Quoted 3 lines
arch/x86/kernel/cpu/resctrl/rdtgroup.c static void schemata_list_destroy(void) · line 2898
Quoted 3 lines
arch/x86/kernel/cpu/resctrl/rdtgroup.c +3
-0+ +if (!ret &&is_mbm_local_enabled ()) + ret =rdtgroup_assign_cntr_event (r, rdtgrp,NULL ,QOS_L3_MBM_LOCAL_EVENT_ID );
Now this call succeeds. The failure of the previous call is forgotten as "ret" is overwritten with the success code.
Quoted 5 lines
Quoted 2 lines
This overwrites the value from allocating the counter for total event.
Total event and local events have two different indexes. Can you please elaborate?
See comments above. If you want a return code you need
int ret_local = 0, ret_total = 0; if (is_mbm_total_enabled()) ret_total = rdtgroup_assign_cntr_event(r, rdtgrp, NULL, QOS_L3_MBM_TOTAL_EVENT_ID); if (!ret && is_mbm_local_enabled()) ret_local = rdtgroup_assign_cntr_event(r, rdtgrp, NULL, QOS_L3_MBM_LOCAL_EVENT_ID); return some_function of ret_local and ret_total;
Not sure if you want to say success only if both of these calls succeeded. Or maybe if either worked?
But it all seems complicated since callers don't have to take any different action depending on whether allocation of a counter succeeds or fails.
-Tony
Moger, Babu
reviewerRe: RE: [PATCH v8 19/25] x86/resctrl: Auto assign/unassign counters when mbm_cntr_assign is enabled
Hi Tony,
Quoted 4 lines · Tony
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
arch/x86/kernel/cpu/resctrl/rdtgroup.c +4
-0+static int rdtgroup_assign_cntrs (struct rdtgroup *rdtgrp) +{ +struct rdt_resource *r = &rdt_resources_all[RDT_RESOURCE_L3 ].r_resctrl; +int ret =0 ;
Quoted 8 lines
Quoted 6 lines
Quoted 6 lines
arch/x86/kernel/cpu/resctrl/rdtgroup.c static void schemata_list_destroy(void) · line 2895
Quoted 6 lines
arch/x86/kernel/cpu/resctrl/rdtgroup.c +6
-0+ +if (!resctrl_arch_mbm_cntr_assign_enabled (r)) +return 0 ; + +if (is_mbm_total_enabled ()) + ret =rdtgroup_assign_cntr_event (r, rdtgrp,NULL ,QOS_L3_MBM_TOTAL_EVENT_ID );Consider that this call fails. "ret" indicates failure to allocate.
Look at this call
if (is_mbm_total_enabled())
ret = rdtgroup_assign_cntr_event(r, rdtgrp, NULL,QOS_L3_MBM_TOTAL_EVENT_ID);
If this call fails, it will return immediately.
Lets say ret = 1; (1 if for failure. 0 for success)
Quoted 7 lines
Quoted 4 lines
Quoted 4 lines
arch/x86/kernel/cpu/resctrl/rdtgroup.c static void schemata_list_destroy(void) · line 2898
Quoted 4 lines
arch/x86/kernel/cpu/resctrl/rdtgroup.c +3
-0+ +if (!ret &&is_mbm_local_enabled ()) + ret =rdtgroup_assign_cntr_event (r, rdtgrp,NULL ,QOS_L3_MBM_LOCAL_EVENT_ID );Now this call succeeds. The failure of the previous call is forgotten as "ret" is overwritten with the success code.
It will not make this call if the first call fails because of this check.
if (!ret && is_mbm_local_enabled())
ret = rdtgroup_assign_cntr_event(r, rdtgrp, NULL,QOS_L3_MBM_LOCAL_EVENT_ID);
return ret;
Here if (!1) evaluates to false.
Did I miss something?
Quoted 27 lines
Quoted 6 lines
Quoted 3 lines
This overwrites the value from allocating the counter for total event.
Total event and local events have two different indexes. Can you please elaborate?
See comments above. If you want a return code you need
int ret_local = 0, ret_total = 0; if (is_mbm_total_enabled()) ret_total = rdtgroup_assign_cntr_event(r, rdtgrp, NULL, QOS_L3_MBM_TOTAL_EVENT_ID); if (!ret && is_mbm_local_enabled()) ret_local = rdtgroup_assign_cntr_event(r, rdtgrp, NULL, QOS_L3_MBM_LOCAL_EVENT_ID); return some_function of ret_local and ret_total;Not sure if you want to say success only if both of these calls succeeded. Or maybe if either worked?
But it all seems complicated since callers don't have to take any different action depending on whether allocation of a counter succeeds or fails.
-Tony
Signature · 2 lines
Thanks Babu Moger
Luck, Tony
maintainerRE: RE: [PATCH v8 19/25] x86/resctrl: Auto assign/unassign counters when mbm_cntr_assign is enabled
Quoted 12 lines
Quoted 5 lines
Quoted 2 lines
Quoted 2 lines
arch/x86/kernel/cpu/resctrl/rdtgroup.c static void schemata_list_destroy(void) · line 2898
Quoted 2 lines
arch/x86/kernel/cpu/resctrl/rdtgroup.c +2
-0+if (!ret &&is_mbm_local_enabled ()) + ret =rdtgroup_assign_cntr_event (r, rdtgrp,NULL ,QOS_L3_MBM_LOCAL_EVENT_ID );Now this call succeeds. The failure of the previous call is forgotten as "ret" is overwritten with the success code.
It will not make this call if the first call fails because of this check.
if (!ret &&
is_mbm_local_enabled())
ret = rdtgroup_assign_cntr_event(r, rdtgrp, NULL,QOS_L3_MBM_LOCAL_EVENT_ID);
Quoted 5 lines
return ret;Here if (!1) evaluates to false.
Did I miss something?
You didn't.
I missed the check for ret in the local case.
It is still the case that callers don't care about the return value.
-Tony
Moger, Babu
reviewerRe: RE: RE: [PATCH v8 19/25] x86/resctrl: Auto assign/unassign counters when mbm_cntr_assign is enabled
Quoted 14 lines · Tony
Quoted 14 lines
Quoted 5 lines
Quoted 2 lines
Quoted 2 lines
arch/x86/kernel/cpu/resctrl/rdtgroup.c static void schemata_list_destroy(void) · line 2898
Quoted 2 lines
arch/x86/kernel/cpu/resctrl/rdtgroup.c +2
-0+if (!ret &&is_mbm_local_enabled ()) + ret =rdtgroup_assign_cntr_event (r, rdtgrp,NULL ,QOS_L3_MBM_LOCAL_EVENT_ID );Now this call succeeds. The failure of the previous call is forgotten as "ret" is overwritten with the success code.
It will not make this call if the first call fails because of this check.
if (!ret &&
is_mbm_local_enabled())
ret = rdtgroup_assign_cntr_event(r, rdtgrp, NULL,QOS_L3_MBM_LOCAL_EVENT_ID);return ret;
Quoted 7 lines
Quoted 3 lines
Here if (!1) evaluates to false.
Did I miss something?
You didn't.
I missed the check for ret in the local case.
That is fine,
Quoted 2 lines
It is still the case that callers don't care about the return value.
That is correct.
Signature · 2 lines
Thanks Babu Moger
Reinette Chatre
maintainerRe: [PATCH v8 19/25] x86/resctrl: Auto assign/unassign counters when mbm_cntr_assign is enabled
Hi Babu,
Quoted 7 lines · Babu
Quoted 2 lines · Tony
It is still the case that callers don't care about the return value.
That is correct.
Are you planning to change this? I think Tony has a good point that since assignment failures do not matter it unnecessarily complicates the code to have rdtgroup_assign_cntrs() return failure.
I also think the internals of rdtgroup_assign_cntrs() deserve a closer look. I assume that error handling within rdtgroup_assign_cntrs() was created with ABMC in mind. When only considering ABMC then the only reason why rdtgroup_assign_cntr_event() could fail is if the system ran out of counters and then indeed it makes no sense to attempt another call to rdtgroup_assign_cntr_event().
Now that the resctrl fs/arch split is clear the implementation does indeed expose another opportunity for failure ... if the arch callback, resctrl_arch_config_cntr() fails. It could thus be possible for the first rdtgroup_assign_cntr_event() to fail while the second succeeds. Earlier [1], Tony suggested to, within rdtgroup_assign_cntrs(), remove the local ret variable and have it return void. This sounds good to me. When doing so a function comment explaining the usage will be helpful.
I also think that rdtgroup_unassign_cntrs() deserves similar scrutiny. Even more so since I do not think that the second rdtgroup_unassign_cntr_event() should be prevented from running if the first rdtgroup_unassign_cntr_event() fails.
Reinette
Moger, Babu
reviewerRe: [PATCH v8 19/25] x86/resctrl: Auto assign/unassign counters when mbm_cntr_assign is enabled
Hi Reinette/Tony,
Quoted 28 lines · On 10/14/24 21:39, wrote:
Hi Babu,
Quoted 7 lines · Babu
Quoted 2 lines · Tony
It is still the case that callers don't care about the return value.
That is correct.
Are you planning to change this? I think Tony has a good point that since assignment failures do not matter it unnecessarily complicates the code to have
rdtgroup_assign_cntrs()return failure.I also think the internals of
rdtgroup_assign_cntrs()deserve a closer look. I assume that error handling withinrdtgroup_assign_cntrs()was created with ABMC in mind. When only considering ABMC then the only reason whyrdtgroup_assign_cntr_event()could fail is if the system ran out of counters and then indeed it makes no sense to attempt another call tordtgroup_assign_cntr_event().Now that the resctrl fs/arch split is clear the implementation does indeed expose another opportunity for failure ... if the arch callback,
resctrl_arch_config_cntr()fails. It could thus be possible for the firstrdtgroup_assign_cntr_event()to fail while the second succeeds. Earlier [1], Tony suggested to, withinrdtgroup_assign_cntrs(), remove the local ret variable and have it return void. This sounds good to me. When doing so a function comment explaining the usage will be helpful.
Quoted 3 lines
I also think that
rdtgroup_unassign_cntrs()deserves similar scrutiny. Even more so since I do not think that the secondrdtgroup_unassign_cntr_event()should be prevented from running if the firstrdtgroup_unassign_cntr_event()fails.
Sounds fine with me. Now it will look like this below.
static void rdtgroup_assign_cntrs(struct rdtgroup *rdtgrp)
{
struct rdt_resource *r = &rdt_resources_all[RDT_RESOURCE_L3].r_resctrl;
if (!resctrl_arch_mbm_cntr_assign_enabled(r))
return;
if (is_mbm_total_enabled())
rdtgroup_assign_cntr_event(r, rdtgrp, NULL, QOS_L3_MBM_TOTAL_EVENT_ID);
if (is_mbm_local_enabled())
rdtgroup_assign_cntr_event(r, rdtgrp, NULL, QOS_L3_MBM_LOCAL_EVENT_ID);
}
/*
* Called when a group is deleted. Counters are unassigned if it was in
* assigned state.
*/
static void rdtgroup_unassign_cntrs(struct rdtgroup *rdtgrp) {
struct rdt_resource *r = &rdt_resources_all[RDT_RESOURCE_L3].r_resctrl;
if (!resctrl_arch_mbm_cntr_assign_enabled(r))
return;
if (is_mbm_total_enabled())
rdtgroup_unassign_cntr_event(r, rdtgrp, NULL, QOS_L3_MBM_TOTAL_EVENT_ID);
if (is_mbm_local_enabled())
rdtgroup_unassign_cntr_event(r, rdtgrp, NULL, QOS_L3_MBM_LOCAL_EVENT_ID);
}
Quoted 5 lines
Reinette
Signature · 2 lines
Thanks Babu Moger
Luck, Tony
maintainerRE: [PATCH v8 19/25] x86/resctrl: Auto assign/unassign counters when mbm_cntr_assign is enabled
Quoted 29 lines
Sounds fine with me. Now it will look like this below.
static void
rdtgroup_assign_cntrs(struct rdtgroup *rdtgrp)
{
structrdt_resource*r = &rdt_resources_all[RDT_RESOURCE_L3].r_resctrl;if (!resctrl_arch_mbm_cntr_assign_enabled(r))
return;if (
is_mbm_total_enabled())
rdtgroup_assign_cntr_event(r, rdtgrp, NULL, QOS_L3_MBM_TOTAL_EVENT_ID);if (
is_mbm_local_enabled())
rdtgroup_assign_cntr_event(r, rdtgrp, NULL, QOS_L3_MBM_LOCAL_EVENT_ID);}
/*
* Called when a group is deleted. Counters are unassigned if it was in
* assigned state.
*/
static voidrdtgroup_unassign_cntrs(struct rdtgroup *rdtgrp){
structrdt_resource*r = &rdt_resources_all[RDT_RESOURCE_L3].r_resctrl;if (!resctrl_arch_mbm_cntr_assign_enabled(r))
return;
Quoted 7 lines
if (
is_mbm_total_enabled())
rdtgroup_unassign_cntr_event(r, rdtgrp, NULL, QOS_L3_MBM_TOTAL_EVENT_ID);if (
is_mbm_local_enabled())
rdtgroup_unassign_cntr_event(r, rdtgrp, NULL, QOS_L3_MBM_LOCAL_EVENT_ID);}
Much cleaner. Thanks.
-Tony
Reinette Chatre
maintainerRe: [PATCH v8 19/25] x86/resctrl: Auto assign/unassign counters when mbm_cntr_assign is enabled
In reply to Moger, BabuHi Babu,
Quoted 31 lines · Babu
Hi Reinette/Tony,
Quoted 28 lines · On 10/14/24 21:39, wrote:
Hi Babu,
Quoted 7 lines · Babu
Quoted 2 lines · Tony
It is still the case that callers don't care about the return value.
That is correct.
Are you planning to change this? I think Tony has a good point that since assignment failures do not matter it unnecessarily complicates the code to have
rdtgroup_assign_cntrs()return failure.I also think the internals of
rdtgroup_assign_cntrs()deserve a closer look. I assume that error handling withinrdtgroup_assign_cntrs()was created with ABMC in mind. When only considering ABMC then the only reason whyrdtgroup_assign_cntr_event()could fail is if the system ran out of counters and then indeed it makes no sense to attempt another call tordtgroup_assign_cntr_event().Now that the resctrl fs/arch split is clear the implementation does indeed expose another opportunity for failure ... if the arch callback,
resctrl_arch_config_cntr()fails. It could thus be possible for the firstrdtgroup_assign_cntr_event()to fail while the second succeeds. Earlier [1], Tony suggested to, withinrdtgroup_assign_cntrs(), remove the local ret variable and have it return void. This sounds good to me. When doing so a function comment explaining the usage will be helpful.
Quoted 6 lines
Quoted 3 lines
I also think that
rdtgroup_unassign_cntrs()deserves similar scrutiny. Even more so since I do not think that the secondrdtgroup_unassign_cntr_event()should be prevented from running if the firstrdtgroup_unassign_cntr_event()fails.Sounds fine with me. Now it will look like this below.
Thank you for considering.
Quoted 2 lines
I assume that you will keep rdtgroup_assign_cntrs() function comment? I think it may need some small changes to go with the function now returning void ... for example, saying "Each group *requires* two counters" and then not failing when two counters cannot be allocated seems suspect.
For example (please feel free to improve):
Called when a new group is created. If "mbm_cntr_assign" mode is enabled, counters are automatically assigned. Each group can accommodate two counters:
one for the total event and one for the local event. Assignments may fail due to the limited number of counters. However, it is not necessary to fail the group creation and thus no failure is returned. Users have the option to modify the counter assignments after the group has been created.
Quoted 26 lines
static void
rdtgroup_assign_cntrs(struct rdtgroup *rdtgrp)
{
structrdt_resource*r = &rdt_resources_all[RDT_RESOURCE_L3].r_resctrl;if (!resctrl_arch_mbm_cntr_assign_enabled(r))
return;if (
is_mbm_total_enabled())
rdtgroup_assign_cntr_event(r, rdtgrp, NULL, QOS_L3_MBM_TOTAL_EVENT_ID);if (
is_mbm_local_enabled())
rdtgroup_assign_cntr_event(r, rdtgrp, NULL, QOS_L3_MBM_LOCAL_EVENT_ID);}
/*
* Called when a group is deleted. Counters are unassigned if it was in
* assigned state.
*/
static voidrdtgroup_unassign_cntrs(struct rdtgroup *rdtgrp){
structrdt_resource*r = &rdt_resources_all[RDT_RESOURCE_L3].r_resctrl;if (!resctrl_arch_mbm_cntr_assign_enabled(r))
return;
Quoted 7 lines
if (
is_mbm_total_enabled())
rdtgroup_unassign_cntr_event(r, rdtgrp, NULL, QOS_L3_MBM_TOTAL_EVENT_ID);if (
is_mbm_local_enabled())
rdtgroup_unassign_cntr_event(r, rdtgrp, NULL, QOS_L3_MBM_LOCAL_EVENT_ID);}
Looks good to me, thank you.
Reinette
Moger, Babu
reviewerRe: [PATCH v8 19/25] x86/resctrl: Auto assign/unassign counters when mbm_cntr_assign is enabled
Hi Reinette,
Quoted 51 lines · On 10/15/24 12:18, wrote:
Hi Babu,
Quoted 31 lines · Babu
Hi Reinette/Tony,
Quoted 28 lines · On 10/14/24 21:39, wrote:
Hi Babu,
Quoted 7 lines · Babu
Quoted 2 lines · Tony
It is still the case that callers don't care about the return value.
That is correct.
Are you planning to change this? I think Tony has a good point that since assignment failures do not matter it unnecessarily complicates the code to have
rdtgroup_assign_cntrs()return failure.I also think the internals of
rdtgroup_assign_cntrs()deserve a closer look. I assume that error handling withinrdtgroup_assign_cntrs()was created with ABMC in mind. When only considering ABMC then the only reason whyrdtgroup_assign_cntr_event()could fail is if the system ran out of counters and then indeed it makes no sense to attempt another call tordtgroup_assign_cntr_event().Now that the resctrl fs/arch split is clear the implementation does indeed expose another opportunity for failure ... if the arch callback,
resctrl_arch_config_cntr()fails. It could thus be possible for the firstrdtgroup_assign_cntr_event()to fail while the second succeeds. Earlier [1], Tony suggested to, withinrdtgroup_assign_cntrs(), remove the local ret variable and have it return void. This sounds good to me. When doing so a function comment explaining the usage will be helpful.Quoted 6 lines
Quoted 3 lines
I also think that
rdtgroup_unassign_cntrs()deserves similar scrutiny. Even more so since I do not think that the secondrdtgroup_unassign_cntr_event()should be prevented from running if the firstrdtgroup_unassign_cntr_event()fails.Sounds fine with me. Now it will look like this below.
Thank you for considering.
Quoted 2 lines
I assume that you will keep
rdtgroup_assign_cntrs()function comment? I think it may need some small changes to go with the function now returning void ... for example, saying "Each group *requires* two counters" and then not failing when two counters cannot be allocated seems suspect.
Quoted 9 lines
For example (please feel free to improve):
Called when a new group is created. If "
mbm_cntr_assign" mode is enabled, counters are automatically assigned. Each group can accommodate two counters:
one for the total event and one for the local event. Assignments may fail due to the limited number of counters. However, it is not necessary to fail the group creation and thus no failure is returned. Users have the option to modify the counter assignments after the group has been created.
Looks good. Thanks
Signature · 2 lines
Thanks Babu Moger
Reinette Chatre
maintainerRe: [PATCH v8 19/25] x86/resctrl: Auto assign/unassign counters when mbm_cntr_assign is enabled
In reply to Babu MogerHi Babu,
Quoted 12 lines · Babu Moger
Assign/unassign counters on resctrl group creation/deletion. Two counters are required per group, one for MBM total event and one for MBM local event.
There are a limited number of counters available for assignment. If these counters are exhausted, the kernel will display the error message: "Out of MBM assignable counters". However, it is not necessary to fail the creation of a group due to assignment failures. Users have the flexibility to modify the assignments at a later time.
Signed-off-by Babu Moger <[email protected]>---
...
Quoted 8 lines
--- arch/x86/kernel/cpu/resctrl/rdtgroup.c | 64 ++++++++++++++++++++++++++ 1 file changed, 64 insertions(+)arch/x86/kernel/cpu/resctrl/rdtgroup.c +0
-0
...
Quoted 13 lines
static int
rdt_get_tree(struct fs_context *fc)
{arch/x86/kernel/cpu/resctrl/rdtgroup.c +2
-0struct rdt_fs_context *ctx =rdt_fc2context (fc);@@ -2934,6 +2980,8 @@ static int rdt_get_tree(struct fs_context *fc)if (ret <0 ) goto out_mongrp; rdtgroup_default.mon.mon_data_kn = kn_mondata; + +rdtgroup_assign_cntrs (&rdtgroup_default); } ret =rdt_pseudo_lock_init ();
arch/x86/kernel/cpu/resctrl/rdtgroup.c static int rdt_get_tree(struct fs_context *fc) · line 3017
Quoted 6 lines
arch/x86/kernel/cpu/resctrl/rdtgroup.c +1
-0@@ -2964,6 +3012,7 @@ static int rdt_get_tree(struct fs_context *fc) out_psl:rdt_pseudo_lock_release (); out_mondata: +rdtgroup_unassign_cntrs (&rdtgroup_default);if (resctrl_arch_mon_capable ())kernfs_remove (kn_mondata);
I think I mentioned this before ... this addition belongs within the "if (resctrl_arch_mon_capable())" to be symmetrical with where it was called from.
Quoted 19 lines
arch/x86/kernel/cpu/resctrl/rdtgroup.c +3
-0out_mongrp:@@ -3144,6 +3193,7 @@ static void free_all_child_rdtgrp(struct rdtgroup *rdtgrp) head = &rdtgrp->mon.crdtgrp_list;list_for_each_entry_safe (sentry, stmp, head, mon.crdtgrp_list) { +rdtgroup_unassign_cntrs (sentry);free_rmid (sentry->closid, sentry->mon.rmid);list_del (&sentry->mon.crdtgrp_list);@@ -3184,6 +3234,8 @@ static void rmdir_all_sub(void)cpumask_or (&rdtgroup_default.cpu_mask, &rdtgroup_default.cpu_mask, &rdtgrp->cpu_mask); +rdtgroup_unassign_cntrs (rdtgrp); +free_rmid (rdtgrp->closid, rdtgrp->mon.rmid);kernfs_remove (rdtgrp->kn);
arch/x86/kernel/cpu/resctrl/rdtgroup.c static void rdt_kill_sb(struct super_block *sb) · line 3279
Quoted 5 lines
arch/x86/kernel/cpu/resctrl/rdtgroup.c +2
-0@@ -3223,6 +3275,8 @@ static void rdt_kill_sb(struct super_block *sb)resctrl_arch_disable_alloc ();if (resctrl_arch_mon_capable ())resctrl_arch_disable_mon (); + +rdtgroup_unassign_cntrs (&rdtgroup_default);
Unassigning counters after monitoring is completely disabled seems late. I think this can be moved earlier to be right after the counters of all the other groups are unassigned.
Quoted 40 lines
arch/x86/kernel/cpu/resctrl/rdtgroup.c +8
-0resctrl_mounted = false;kernfs_kill_sb (sb);mutex_unlock (&rdtgroup_mutex);@@ -3814,6 +3868,8 @@ static int rdtgroup_mkdir_mon(struct kernfs_node *parent_kn, goto out_unlock; } +rdtgroup_assign_cntrs (rdtgrp); +kernfs_activate (rdtgrp->kn);/* @@ -3858,6 +3914,8 @@ static int rdtgroup_mkdir_ctrl_mon(struct kernfs_node *parent_kn,if (ret) goto out_closid_free; +rdtgroup_assign_cntrs (rdtgrp); +kernfs_activate (rdtgrp->kn); ret =rdtgroup_init_alloc (rdtgrp);@@ -3883,6 +3941,7 @@ static int rdtgroup_mkdir_ctrl_mon(struct kernfs_node *parent_kn, out_del_list:list_del (&rdtgrp->rdtgroup_list); out_rmid_free: +rdtgroup_unassign_cntrs (rdtgrp);mkdir_rdt_prepare_rmid_free (rdtgrp); out_closid_free:closid_free (closid);@@ -3953,6 +4012,9 @@ static int rdtgroup_rmdir_mon(struct rdtgroup *rdtgrp, cpumask_var_t tmpmask)update_closid_rmid (tmpmask,NULL ); rdtgrp->flags =RDT_DELETED ; + +rdtgroup_unassign_cntrs (rdtgrp); +free_rmid (rdtgrp->closid, rdtgrp->mon.rmid);/*
arch/x86/kernel/cpu/resctrl/rdtgroup.c static int rdtgroup_rmdir_ctrl(struct rdtgroup *rdtgrp, cpumask_var_t tmpmask) · line 4068
Quoted 8 lines
arch/x86/kernel/cpu/resctrl/rdtgroup.c +2
-0@@ -3999,6 +4061,8 @@ static int rdtgroup_rmdir_ctrl(struct rdtgroup *rdtgrp, cpumask_var_t tmpmask)cpumask_or (tmpmask, tmpmask, &rdtgrp->cpu_mask);update_closid_rmid (tmpmask,NULL ); +rdtgroup_unassign_cntrs (rdtgrp); +free_rmid (rdtgrp->closid, rdtgrp->mon.rmid);closid_free (rdtgrp->closid);
Reinette
Re: [PATCH v8 19/25] x86/resctrl: Auto assign/unassign counters when mbm_cntr_assign is enabled
Hi Reinette,
Quoted 39 lines · Reinette Chatre
Hi Babu,
Quoted 12 lines · Babu Moger
Assign/unassign counters on resctrl group creation/deletion. Two counters are required per group, one for MBM total event and one for MBM local event.
There are a limited number of counters available for assignment. If these counters are exhausted, the kernel will display the error message: "Out of MBM assignable counters". However, it is not necessary to fail the creation of a group due to assignment failures. Users have the flexibility to modify the assignments at a later time.
Signed-off-by Babu Moger <[email protected]>---
...
Quoted 8 lines
--- arch/x86/kernel/cpu/resctrl/rdtgroup.c | 64 ++++++++++++++++++++++++++ 1 file changed, 64 insertions(+)arch/x86/kernel/cpu/resctrl/rdtgroup.c +0
-0...
Quoted 11 lines
static int rdt_get_tree(struct fs_context *fc) { struct rdt_fs_context *ctx = rdt_fc2context(fc);arch/x86/kernel/cpu/resctrl/rdtgroup.c +2
-0@@ -2934,6 +2980,8 @@ static int rdt_get_tree(struct fs_context *fc)if (ret <0 ) goto out_mongrp; rdtgroup_default.mon.mon_data_kn = kn_mondata; + +rdtgroup_assign_cntrs (&rdtgroup_default); }
Quoted 11 lines
arch/x86/kernel/cpu/resctrl/rdtgroup.c static int rdt_get_tree(struct fs_context *fc) · line 3017
Quoted 8 lines
arch/x86/kernel/cpu/resctrl/rdtgroup.c +1
-0ret =rdt_pseudo_lock_init ();@@ -2964,6 +3012,7 @@ static int rdt_get_tree(struct fs_context *fc) out_psl:rdt_pseudo_lock_release (); out_mondata: +rdtgroup_unassign_cntrs (&rdtgroup_default);if (resctrl_arch_mon_capable ())kernfs_remove (kn_mondata);I think I mentioned this before ... this addition belongs within the "if (
resctrl_arch_mon_capable())" to be symmetrical with where it was called from.
Sure.
Quoted 18 lines
Quoted 18 lines
arch/x86/kernel/cpu/resctrl/rdtgroup.c +3
-0out_mongrp:@@ -3144,6 +3193,7 @@ static void free_all_child_rdtgrp(struct rdtgroup *rdtgrp) head = &rdtgrp->mon.crdtgrp_list;list_for_each_entry_safe (sentry, stmp, head, mon.crdtgrp_list) { +rdtgroup_unassign_cntrs (sentry);free_rmid (sentry->closid, sentry->mon.rmid);list_del (&sentry->mon.crdtgrp_list);@@ -3184,6 +3234,8 @@ static void rmdir_all_sub(void)cpumask_or (&rdtgroup_default.cpu_mask, &rdtgroup_default.cpu_mask, &rdtgrp->cpu_mask); +rdtgroup_unassign_cntrs (rdtgrp); +free_rmid (rdtgrp->closid, rdtgrp->mon.rmid);
Quoted 11 lines
arch/x86/kernel/cpu/resctrl/rdtgroup.c static void rdt_kill_sb(struct super_block *sb) · line 3279
Quoted 7 lines
arch/x86/kernel/cpu/resctrl/rdtgroup.c +2
-0kernfs_remove (rdtgrp->kn);@@ -3223,6 +3275,8 @@ static void rdt_kill_sb(struct super_block *sb)resctrl_arch_disable_alloc ();if (resctrl_arch_mon_capable ())resctrl_arch_disable_mon (); + +rdtgroup_unassign_cntrs (&rdtgroup_default);Unassigning counters after monitoring is completely disabled seems late. I think this can be moved earlier to be right after the counters of all the other groups are unassigned.
Sure. Right after rmdir_all_sub().
Quoted 44 lines
Quoted 44 lines
arch/x86/kernel/cpu/resctrl/rdtgroup.c +8
-0resctrl_mounted = false;kernfs_kill_sb (sb);mutex_unlock (&rdtgroup_mutex);@@ -3814,6 +3868,8 @@ static int rdtgroup_mkdir_mon(struct kernfs_node *parent_kn, goto out_unlock; } +rdtgroup_assign_cntrs (rdtgrp); +kernfs_activate (rdtgrp->kn);/* @@ -3858,6 +3914,8 @@ static int rdtgroup_mkdir_ctrl_mon(struct kernfs_node *parent_kn,if (ret) goto out_closid_free; +rdtgroup_assign_cntrs (rdtgrp); +kernfs_activate (rdtgrp->kn); ret =rdtgroup_init_alloc (rdtgrp);@@ -3883,6 +3941,7 @@ static int rdtgroup_mkdir_ctrl_mon(struct kernfs_node *parent_kn, out_del_list:list_del (&rdtgrp->rdtgroup_list); out_rmid_free: +rdtgroup_unassign_cntrs (rdtgrp);mkdir_rdt_prepare_rmid_free (rdtgrp); out_closid_free:closid_free (closid);@@ -3953,6 +4012,9 @@ static int rdtgroup_rmdir_mon(struct rdtgroup *rdtgrp, cpumask_var_t tmpmask)update_closid_rmid (tmpmask,NULL ); rdtgrp->flags =RDT_DELETED ; + +rdtgroup_unassign_cntrs (rdtgrp); +free_rmid (rdtgrp->closid, rdtgrp->mon.rmid);/* @@ -3999,6 +4061,8 @@ static int rdtgroup_rmdir_ctrl(struct rdtgroup *rdtgrp, cpumask_var_t tmpmask)cpumask_or (tmpmask, tmpmask, &rdtgrp->cpu_mask);update_closid_rmid (tmpmask,NULL );
Quoted 8 lines
arch/x86/kernel/cpu/resctrl/rdtgroup.c static int rdtgroup_rmdir_ctrl(struct rdtgroup *rdtgrp, cpumask_var_t tmpmask) · line 4068
Quoted 5 lines
arch/x86/kernel/cpu/resctrl/rdtgroup.c +2
-0+rdtgroup_unassign_cntrs (rdtgrp); +free_rmid (rdtgrp->closid, rdtgrp->mon.rmid);closid_free (rdtgrp->closid);Reinette
Thanks
- Babu Moger
Babu Moger
reviewerPATCH v8 20/25 x86/resctrl: Report "Unassigned" for MBM events in mbm_cntr_assign mode
In reply to Babu MogerIn mbm_cntr_assign mode, the hardware counter should be assigned to read the MBM events.
Report "Unassigned" in case the user attempts to read the events without assigning the counter.
---
v8: Used MBM_EVENT_ARRAY_INDEX to get the index for the MBM event.
Documentation update to make the text generic.
v7: Moved the documentation under "mon_data".
Updated the text little bit.
v6: Added more explaination in the resctrl.rst
Added checks to detect "Unassigned" before reading RMID.
v5: New patch.
--- Documentation/arch/x86/resctrl.rst | 10 ++++++++++ arch/x86/kernel/cpu/resctrl/ctrlmondata.c | 13 ++++++++++++- 2 files changed, 22 insertions(+), 1 deletion(-)
Documentation/arch/x86/resctrl.rst +10 -0
@@ -419,6 +419,16 @@ When monitoring is enabled all MON groups will also contain: for the L3 cache they occupy). These are named "mon_sub_L3_YY" where "YY" is the node number. + When supported the 'mbm_cntr_assign' mode allows users to assign a + counter to mon_hw_id, event pair enabling bandwidth monitoring for + as long as the counter remains assigned. The hardware will continue + tracking the assigned mon_hw_id until the user manually unassigns + it, ensuring that counters are not reset during this period. With + a limited number of counters, the system may run out of assignable + counters at some point. In that case, MBM event counters will return + "Unassigned" when the event is read. Users must manually assign a + counter to read the events. + "mon_hw_id": Available only with debug option. The identifier used by hardware for the monitor group. On x86 this is the RMID.
arch/x86/kernel/cpu/resctrl/ctrlmondata.c +12 -1
@@ -562,7 +562,7 @@ int rdtgroup_mondata_show(struct seq_file *m, void *arg)struct rdtgroup *rdtgrp;struct rdt_resource *r;union mon_data_bits md; -int ret =0 ; +int ret =0 , index; rdtgrp =rdtgroup_kn_lock_live (of->kn);if (!rdtgrp) {
@@ -576,6 +576,15 @@ int rdtgroup_mondata_show(struct seq_file *m, void *arg) evtid = md.u.evtid; r = &rdt_resources_all[resid].r_resctrl; +if (resctrl_arch_mbm_cntr_assign_enabled (r) && evtid !=QOS_L3_OCCUP_EVENT_ID ) { + index =MBM_EVENT_ARRAY_INDEX (evtid); +if (index !=INVALID_CONFIG_INDEX && + rdtgrp->mon.cntr_id[index] ==MON_CNTR_UNSET ) { + rr.err = -ENOENT ; + goto checkresult; + } + } +if (md.u.sum) {/* * This file requires summing across all domains that share
@@ -613,6 +622,8 @@ int rdtgroup_mondata_show(struct seq_file *m, void *arg)seq_puts (m,"Error\n" ); else if (rr.err == -EINVAL) seq_puts(m, "Unavailable\n"); + else if (rr.err == -ENOENT) + seq_puts(m, "Unassigned\n"); else seq_printf(m, "%llu\n", rr.val);
Signature · 1 line
2.34.1
Tony Luck
maintainerRe: [PATCH v8 20/25] x86/resctrl: Report "Unassigned" for MBM events in mbm_cntr_assign mode
arch/x86/kernel/cpu/resctrl/ctrlmondata.c int rdtgroup_mondata_show(struct seq_file *m, void *arg) · line 579
Quoted 5 lines · Babu Moger
arch/x86/kernel/cpu/resctrl/ctrlmondata.c +1
-0@@ -576,6 +576,15 @@ int rdtgroup_mondata_show(struct seq_file *m, void *arg) evtid = md.u.evtid; r = &rdt_resources_all[resid].r_resctrl; +if (resctrl_arch_mbm_cntr_assign_enabled (r) && evtid !=QOS_L3_OCCUP_EVENT_ID ) {
Better to write this as:
if (resctrl_arch_mbm_cntr_assign_enabled(r) && is_mbm_event(evtid)) {-Tony
Re: [PATCH v8 20/25] x86/resctrl: Report "Unassigned" for MBM events in mbm_cntr_assign mode
Hi Tony,
Quoted 11 lines · Tony Luck
arch/x86/kernel/cpu/resctrl/ctrlmondata.c int rdtgroup_mondata_show(struct seq_file *m, void *arg) · line 579
Quoted 5 lines · Babu Moger
arch/x86/kernel/cpu/resctrl/ctrlmondata.c +1
-0@@ -576,6 +576,15 @@ int rdtgroup_mondata_show(struct seq_file *m, void *arg) evtid = md.u.evtid; r = &rdt_resources_all[resid].r_resctrl; +if (resctrl_arch_mbm_cntr_assign_enabled (r) && evtid !=QOS_L3_OCCUP_EVENT_ID ) {Better to write this as:
if (resctrl_arch_mbm_cntr_assign_enabled(r) && is_mbm_event(evtid)) {
Sure. will do.
Thanks
- Babu Moger
Reinette Chatre
maintainerRe: [PATCH v8 20/25] x86/resctrl: Report "Unassigned" for MBM events in mbm_cntr_assign mode
In reply to Babu MogerHi Babu,
Quoted 28 lines · Babu Moger
In
mbm_cntr_assignmode, the hardware counter should be assigned to read the MBM events.Report "Unassigned" in case the user attempts to read the events without assigning the counter.
Signed-off-by Babu Moger <[email protected]>---
v8: UsedMBM_EVENT_ARRAY_INDEXto get the index for the MBM event.
Documentation update to make the text generic.v7: Moved the documentation under "
mon_data".
Updated the text little bit.v6: Added more explaination in the
resctrl.rst
Added checks to detect "Unassigned" before reading RMID.v5: New patch.
--- Documentation/arch/x86/resctrl.rst | 10 ++++++++++ arch/x86/kernel/cpu/resctrl/ctrlmondata.c | 13 ++++++++++++- 2 files changed, 22 insertions(+), 1 deletion(-)Documentation/arch/x86/resctrl.rst +0
-0
Documentation/arch/x86/resctrl.rst When monitoring is enabled all MON groups will also contain: · line 428
Quoted 10 lines
Documentation/arch/x86/resctrl.rst +7
-0@@ -419,6 +419,16 @@ When monitoring is enabled all MON groups will also contain: for the L3 cache they occupy). These are named "mon_sub_L3_YY" where "YY" is the node number. + When supported the 'mbm_cntr_assign' mode allows users to assign a + counter to mon_hw_id, event pair enabling bandwidth monitoring for + as long as the counter remains assigned. The hardware will continue + tracking the assigned mon_hw_id until the user manually unassigns + it, ensuring that counters are not reset during this period. With + a limited number of counters, the system may run out of assignable + counters at some point. In that case, MBM event counters will return
nit: "at some point" can be dropped for clarity.
Quoted 36 lines
Documentation/arch/x86/resctrl.rst +3
-0+ "Unassigned" when the event is read. Users must manually assign a + counter to read the events. + "mon_hw_id": Available only with debug option. The identifier used by hardware for the monitor group. On x86 this is the RMID.arch/x86/kernel/cpu/resctrl/ctrlmondata.c +10
-1@@ -562,7 +562,7 @@ int rdtgroup_mondata_show(struct seq_file *m, void *arg)struct rdtgroup *rdtgrp;struct rdt_resource *r;union mon_data_bits md; -int ret =0 ; +int ret =0 , index; rdtgrp =rdtgroup_kn_lock_live (of->kn);if (!rdtgrp) {@@ -576,6 +576,15 @@ int rdtgroup_mondata_show(struct seq_file *m, void *arg) evtid = md.u.evtid; r = &rdt_resources_all[resid].r_resctrl; +if (resctrl_arch_mbm_cntr_assign_enabled (r) && evtid !=QOS_L3_OCCUP_EVENT_ID ) { + index =MBM_EVENT_ARRAY_INDEX (evtid); +if (index !=INVALID_CONFIG_INDEX && + rdtgrp->mon.cntr_id[index] ==MON_CNTR_UNSET ) { + rr.err = -ENOENT ; + goto checkresult; + } + } +if (md.u.sum) {/* * This file requires summing across all domains that share
arch/x86/kernel/cpu/resctrl/ctrlmondata.c int rdtgroup_mondata_show(struct seq_file *m, void *arg) · line 629
Quoted 8 lines
arch/x86/kernel/cpu/resctrl/ctrlmondata.c +2
-0@@ -613,6 +622,8 @@ int rdtgroup_mondata_show(struct seq_file *m, void *arg)seq_puts (m,"Error\n" ); else if (rr.err == -EINVAL) seq_puts(m, "Unavailable\n"); + else if (rr.err == -ENOENT) + seq_puts(m, "Unassigned\n"); else seq_printf(m, "%llu\n", rr.val);
Reinette
Re: [PATCH v8 20/25] x86/resctrl: Report "Unassigned" for MBM events in mbm_cntr_assign mode
Hi Reinette,
Quoted 34 lines · Reinette Chatre
Hi Babu,
Quoted 31 lines · Babu Moger
In
mbm_cntr_assignmode, the hardware counter should be assigned to read the MBM events.Report "Unassigned" in case the user attempts to read the events without assigning the counter.
Signed-off-by Babu Moger <[email protected]>---
v8: UsedMBM_EVENT_ARRAY_INDEXto get the index for the MBM event.
Documentation update to make the text generic.v7: Moved the documentation under "
mon_data".
Updated the text little bit.v6: Added more explaination in the
resctrl.rst
Added checks to detect "Unassigned" before reading RMID.v5: New patch.
--- Documentation/arch/x86/resctrl.rst | 10 ++++++++++ arch/x86/kernel/cpu/resctrl/ctrlmondata.c | 13 ++++++++++++- 2 files changed, 22 insertions(+), 1 deletion(-)Documentation/arch/x86/resctrl.rst +0
-0@@ -419,6 +419,16 @@ When monitoring is enabled all MON groups will also contain: for the L3 cache they occupy). These are named "mon_sub_L3_YY" where "YY" is the node number.
Quoted 9 lines
Documentation/arch/x86/resctrl.rst When monitoring is enabled all MON groups will also contain: · line 428
Quoted 7 lines
Documentation/arch/x86/resctrl.rst +7
-0+ When supported the 'mbm_cntr_assign' mode allows users to assign a + counter to mon_hw_id, event pair enabling bandwidth monitoring for + as long as the counter remains assigned. The hardware will continue + tracking the assigned mon_hw_id until the user manually unassigns + it, ensuring that counters are not reset during this period. With + a limited number of counters, the system may run out of assignable + counters at some point. In that case, MBM event counters will returnnit: "at some point" can be dropped for clarity.
Sure.
Quoted 39 lines
Quoted 39 lines
Documentation/arch/x86/resctrl.rst +3
-0+ "Unassigned" when the event is read. Users must manually assign a + counter to read the events. + "mon_hw_id": Available only with debug option. The identifier used by hardware for the monitor group. On x86 this is the RMID.arch/x86/kernel/cpu/resctrl/ctrlmondata.c +10
-1@@ -562,7 +562,7 @@ int rdtgroup_mondata_show(struct seq_file *m, void *arg)struct rdtgroup *rdtgrp;struct rdt_resource *r;union mon_data_bits md; -int ret =0 ; +int ret =0 , index; rdtgrp =rdtgroup_kn_lock_live (of->kn);if (!rdtgrp) {@@ -576,6 +576,15 @@ int rdtgroup_mondata_show(struct seq_file *m, void *arg) evtid = md.u.evtid; r = &rdt_resources_all[resid].r_resctrl; +if (resctrl_arch_mbm_cntr_assign_enabled (r) && evtid !=QOS_L3_OCCUP_EVENT_ID ) { + index =MBM_EVENT_ARRAY_INDEX (evtid); +if (index !=INVALID_CONFIG_INDEX && + rdtgrp->mon.cntr_id[index] ==MON_CNTR_UNSET ) { + rr.err = -ENOENT ; + goto checkresult; + } + } +if (md.u.sum) {/* * This file requires summing across all domains that share @@ -613,6 +622,8 @@ int rdtgroup_mondata_show(struct seq_file *m, void *arg)seq_puts (m,"Error\n" ); else if (rr.err == -EINVAL)
Quoted 9 lines
arch/x86/kernel/cpu/resctrl/ctrlmondata.c int rdtgroup_mondata_show(struct seq_file *m, void *arg) · line 629
Quoted 6 lines
arch/x86/kernel/cpu/resctrl/ctrlmondata.c +2
-0seq_puts (m,"Unavailable\n" ); + else if (rr.err == -ENOENT) + seq_puts(m, "Unassigned\n"); else seq_printf(m, "%llu\n", rr.val);Reinette
Thanks
- Babu Moger
Babu Moger
reviewerPATCH v8 21/25 x86/resctrl: Introduce the interface to switch between monitor modes
In reply to Babu MogerIntroduce interface to switch between mbm_cntr_assign and default modes.
$ cat /sys/fs/resctrl/info/L3_MON/mbm_assign_mode [mbm_cntr_assign] default
To enable the "mbm_cntr_assign" mode:
$ echo "mbm_cntr_assign" > /sys/fs/resctrl/info/L3_MON/mbm_assign_mode
To enable the default monitoring mode:
$ echo "default" > /sys/fs/resctrl/info/L3_MON/mbm_assign_mode
MBM event counters will reset when mbm_assign_mode is changed.
---
v8: Reset the internal counters after mbm_cntr_assign mode is changed.
Renamed rdtgroup_mbm_cntr_reset() to mbm_cntr_reset() Updated the documentation to make text generic.
v7: Changed the interface name to mbm_assign_mode.
Removed the references of ABMC.
Added the changes to reset global and domain bitmaps. Added the changes to reset rmid.
v6: Changed the mode name to mbm_cntr_assign.
Moved all the FS related code here.
Added changes to reset mbm_cntr_map and resctrl group counters.
v5: Change log and mode description text correction.
v4: Minor commit text changes. Keep the default to ABMC when supported.
Fixed comments to reflect changed interface "mbm_mode".
v3: New patch to address the review comments from upstream.
--- Documentation/arch/x86/resctrl.rst | 15 ++++++ arch/x86/kernel/cpu/resctrl/rdtgroup.c | 75 +++++++++++++++++++++++++- 2 files changed, 89 insertions(+), 1 deletion(-)
Documentation/arch/x86/resctrl.rst +15 -0
@@ -291,6 +291,21 @@ with the following files: that case reading the mbm_total_bytes and mbm_local_bytes may report 'Unavailable' if there is no counter associated with that group. + * To enable "mbm_cntr_assign" mode: + :: + + # echo "mbm_cntr_assign" > /sys/fs/resctrl/info/L3_MON/mbm_assign_mode + + * To enable default monitoring mode: + :: + + # echo "default" > /sys/fs/resctrl/info/L3_MON/mbm_assign_mode + + The MBM events (mbm_total_bytes and/or mbm_local_bytes) associated counters + may reset when the mode is changed. Moving to mbm_cntr_assign mode will + require users to assign the counters to the events. Otherwise, the MBM + event counters will return "Unassigned" when read. + "num_mbm_cntrs": The number of monitoring counters available for assignment when the architecture supports mbm_cntr_assign mode.
arch/x86/kernel/cpu/resctrl/rdtgroup.c +74 -1
89 lines not on this page · raw message
Signature · 1 line
2.34.1
Reinette Chatre
maintainerRe: [PATCH v8 21/25] x86/resctrl: Introduce the interface to switch between monitor modes
Hi Babu,
Quoted 16 lines · Babu Moger
Introduce interface to switch between
mbm_cntr_assignand default modes.$ cat /sys/fs/resctrl/info/L3_MON/mbm_assign_mode [mbm_cntr_assign] defaultTo enable the "
mbm_cntr_assign" mode:$ echo "mbm_cntr_assign" > /sys/fs/resctrl/info/L3_MON/mbm_assign_modeTo enable the default monitoring mode:
$ echo "default" > /sys/fs/resctrl/info/L3_MON/mbm_assign_modeMBM event counters will reset when
mbm_assign_modeis changed.Signed-off-by Babu Moger <[email protected]>---
...
Quoted 10 lines
--- Documentation/arch/x86/resctrl.rst | 15 ++++++ arch/x86/kernel/cpu/resctrl/rdtgroup.c | 75 +++++++++++++++++++++++++- 2 files changed, 89 insertions(+), 1 deletion(-)Documentation/arch/x86/resctrl.rst +0
-0
Documentation/arch/x86/resctrl.rst with the following files: · line 297
Quoted 7 lines
Documentation/arch/x86/resctrl.rst +4
-0@@ -291,6 +291,21 @@ with the following files: that case reading the mbm_total_bytes and mbm_local_bytes may report 'Unavailable' if there is no counter associated with that group. + * To enable "mbm_cntr_assign" mode: + :: + + # echo "mbm_cntr_assign" > /sys/fs/resctrl/info/L3_MON/mbm_assign_mode
extra spaces
Documentation/arch/x86/resctrl.rst with the following files: · line 302
Quoted 5 lines
Documentation/arch/x86/resctrl.rst +5
-0+ + * To enable default monitoring mode: + :: + + # echo "default" > /sys/fs/resctrl/info/L3_MON/mbm_assign_mode
extra spaces
Documentation/arch/x86/resctrl.rst with the following files: · line 304
Quoted 2 lines
Documentation/arch/x86/resctrl.rst +2
-0+ + The MBM events (mbm_total_bytes and/or mbm_local_bytes) associated counters
I did ask you not to copy the text verbatim
https://lore.kernel.org/all/[email protected]/
Documentation/arch/x86/resctrl.rst with the following files: · line 306
Quoted 2 lines
Documentation/arch/x86/resctrl.rst +2
-0+ may reset when the mode is changed. Moving to mbm_cntr_assign mode will + require users to assign the counters to the events. Otherwise, the MBM
"will require" -> "require"
Quoted 31 lines
Documentation/arch/x86/resctrl.rst +2
-0+ event counters will return "Unassigned" when read. + "num_mbm_cntrs": The number of monitoring counters available for assignment when the architecture supports mbm_cntr_assign mode.arch/x86/kernel/cpu/resctrl/rdtgroup.c +18
-0@@ -888,6 +888,78 @@ static int rdtgroup_mbm_assign_mode_show(struct kernfs_open_file *of,return 0 ; } +static void mbm_cntr_reset (struct rdt_resource *r) +{ +struct rdtgroup *prgrp, *crgrp; +struct rdt_mon_domain *dom; + +/* +* Hardware counters will reset after switching the monitor mode. +* Reset the architectural state so that reading of hardware +* counter is not considered as an overflow in the next update. +* Also reset the domain counter bitmap. +*/ +list_for_each_entry (dom, &r->mon_domains, hdr.list) { +bitmap_zero (dom->mbm_cntr_map, r->mon.num_mbm_cntrs); +resctrl_arch_reset_rmid_all (r, dom); + } + +/* Reset global MBM counter map */ +bitmap_fill (r->mon.mbm_cntr_free_map, r->mon.num_mbm_cntrs);
arch/x86/kernel/cpu/resctrl/rdtgroup.c static int rdtgroup_mbm_assign_mode_show(struct kernfs_open_file *of, · line 918
Quoted 10 lines
arch/x86/kernel/cpu/resctrl/rdtgroup.c +10
-0+ +/* Reset the cntr_id's for all the monitor groups */ +list_for_each_entry (prgrp, &rdt_all_groups, rdtgroup_list) { + prgrp->mon.cntr_id[0 ] =MON_CNTR_UNSET ; + prgrp->mon.cntr_id[1 ] =MON_CNTR_UNSET ; +list_for_each_entry (crgrp, &prgrp->mon.crdtgrp_list, + mon.crdtgrp_list) { + crgrp->mon.cntr_id[0 ] =MON_CNTR_UNSET ; + crgrp->mon.cntr_id[1 ] =MON_CNTR_UNSET ; + }
Please use MBM_EVENT_ARRAY_INDEX
Quoted 20 lines
arch/x86/kernel/cpu/resctrl/rdtgroup.c +20
-0+ } +} + +static ssize_t rdtgroup_mbm_assign_mode_write (struct kernfs_open_file *of, +char *buf,size_t nbytes,loff_t off) +{ +struct rdt_resource *r = of->kn->parent->priv; +int ret =0 ; +bool enable; + +/* Valid input requires a trailing newline */ +if (nbytes ==0 || buf[nbytes -1 ] !='\n' ) +return -EINVAL ; + + buf[nbytes -1 ] ='\0' ; + +cpus_read_lock (); +mutex_lock (&rdtgroup_mutex); + +rdt_last_cmd_clear ();
arch/x86/kernel/cpu/resctrl/rdtgroup.c static int rdtgroup_mbm_assign_mode_show(struct kernfs_open_file *of, · line 948
Quoted 10 lines
arch/x86/kernel/cpu/resctrl/rdtgroup.c +10
-0+ +if (!strcmp (buf,"default" )) { + enable =0 ; + }else if (!strcmp (buf,"mbm_cntr_assign" )) { + enable =1 ; + }else { + ret = -EINVAL ; +rdt_last_cmd_puts ("Unsupported assign mode\n" ); + goto write_exit; + }
Please keep two things in mind:
* this file is always accessible, whether platform supports assignable counters or not.
* this is resctrl fs code.
So, considering above, how should user interpret the "Unsupported assign mode"? Shouldn't it also return this error if a user attempts to enable "mbm_cntr_assign" on a platform that does not support this mode?
arch/x86/kernel/cpu/resctrl/rdtgroup.c static int rdtgroup_mbm_assign_mode_show(struct kernfs_open_file *of, · line 950
Quoted 2 lines
arch/x86/kernel/cpu/resctrl/rdtgroup.c +2
-0+ +if (enable !=resctrl_arch_mbm_cntr_assign_enabled (r)) {
resctrl_arch_mbm_cntr_assign_enabled() returns true if mbm_cntr_assign mode is enabled, but when it returns false it could mean different things:
platform supports mbm_cntr_assign mode, but it is disabled, or platform does not support mbm_cntr_assign mode.
resctrl fs should not rely on all archs to duplicate the all the checking done in resctrl_arch_mbm_cntr_assign_set(). It should never ask arch to enable a mode that it knows the platform is not capable of.
Quoted 17 lines
arch/x86/kernel/cpu/resctrl/rdtgroup.c +12
-0+ ret =resctrl_arch_mbm_cntr_assign_set (r, enable); +if (!ret) +mbm_cntr_reset (r); + } + +write_exit: +mutex_unlock (&rdtgroup_mutex); +cpus_read_unlock (); + +return ret ?: nbytes; +} +static int rdtgroup_num_mbm_cntrs_show (struct kernfs_open_file *of,struct seq_file *s,void *v) {@@ -2115,9 +2187,10 @@ static struct rftype res_common_files[] = { },
arch/x86/kernel/cpu/resctrl/rdtgroup.c static struct rftype res_common_files[] = { · line 2196
Quoted 10 lines
arch/x86/kernel/cpu/resctrl/rdtgroup.c +2
-1{ .name = "mbm_assign_mode", - .mode = 0444, + .mode = 0644, .kf_ops = &rdtgroup_kf_single_ops, .seq_show = rdtgroup_mbm_assign_mode_show, + .write = rdtgroup_mbm_assign_mode_write, .fflags = RFTYPE_MON_INFO, }, {
Reinette
Re: [PATCH v8 21/25] x86/resctrl: Introduce the interface to switch between monitor modes
Hi Reinette,
Quoted 35 lines · Reinette Chatre
Hi Babu,
Quoted 16 lines · Babu Moger
Introduce interface to switch between
mbm_cntr_assignand default modes.$ cat /sys/fs/resctrl/info/L3_MON/mbm_assign_mode [mbm_cntr_assign] defaultTo enable the "
mbm_cntr_assign" mode:$ echo "mbm_cntr_assign" > /sys/fs/resctrl/info/L3_MON/mbm_assign_modeTo enable the default monitoring mode:
$ echo "default" > /sys/fs/resctrl/info/L3_MON/mbm_assign_modeMBM event counters will reset when
mbm_assign_modeis changed.Signed-off-by Babu Moger <[email protected]>---
...
Quoted 13 lines
--- Documentation/arch/x86/resctrl.rst | 15 ++++++ arch/x86/kernel/cpu/resctrl/rdtgroup.c | 75 +++++++++++++++++++++++++- 2 files changed, 89 insertions(+), 1 deletion(-)Documentation/arch/x86/resctrl.rst +0
-0@@ -291,6 +291,21 @@ with the following files: that case reading the mbm_total_bytes and mbm_local_bytes may report 'Unavailable' if there is no counter associated with that group.
Quoted 6 lines
Documentation/arch/x86/resctrl.rst with the following files: · line 297
Quoted 4 lines
Documentation/arch/x86/resctrl.rst +4
-0+ * To enable "mbm_cntr_assign" mode: + :: + + # echo "mbm_cntr_assign" > /sys/fs/resctrl/info/L3_MON/mbm_assign_modeextra spaces
Sure.
Quoted 8 lines
Documentation/arch/x86/resctrl.rst with the following files: · line 302
Quoted 6 lines
Documentation/arch/x86/resctrl.rst +5
-0+ + * To enable default monitoring mode: + :: + + # echo "default" > /sys/fs/resctrl/info/L3_MON/mbm_assign_modeextra spaces
Sure
Quoted 5 lines
Documentation/arch/x86/resctrl.rst with the following files: · line 304
Quoted 2 lines
Documentation/arch/x86/resctrl.rst +2
-0+ + The MBM events (mbm_total_bytes and/or mbm_local_bytes) associated countersI did ask you not to copy the text verbatim
https://lore.kernel.org/all/[email protected]/
Outch Yea.
The MBM events (mbm_total_bytes and/or mbm_local_bytes) associated with counters may reset when mbm_assign_mode is changed.
Quoted 41 lines
Documentation/arch/x86/resctrl.rst with the following files: · line 306
Quoted 3 lines
Documentation/arch/x86/resctrl.rst +2
-0+ may reset when the mode is changed. Moving to mbm_cntr_assign mode will + require users to assign the counters to the events. Otherwise, the MBM"will require" -> "require"
Quoted 35 lines
Documentation/arch/x86/resctrl.rst +2
-0+ event counters will return "Unassigned" when read. + "num_mbm_cntrs": The number of monitoring counters available for assignment when the architecture supports mbm_cntr_assign mode.arch/x86/kernel/cpu/resctrl/rdtgroup.c +22
-0@@ -888,6 +888,78 @@ static int rdtgroup_mbm_assign_mode_show(struct kernfs_open_file *of,return 0 ; } +static void mbm_cntr_reset (struct rdt_resource *r) +{ +struct rdtgroup *prgrp, *crgrp; +struct rdt_mon_domain *dom; + +/* +* Hardware counters will reset after switching the monitor mode. +* Reset the architectural state so that reading of hardware +* counter is not considered as an overflow in the next update. +* Also reset the domain counter bitmap. +*/ +list_for_each_entry (dom, &r->mon_domains, hdr.list) { +bitmap_zero (dom->mbm_cntr_map, r->mon.num_mbm_cntrs); +resctrl_arch_reset_rmid_all (r, dom); + } + +/* Reset global MBM counter map */ +bitmap_fill (r->mon.mbm_cntr_free_map, r->mon.num_mbm_cntrs); + +/* Reset the cntr_id's for all the monitor groups */ +list_for_each_entry (prgrp, &rdt_all_groups, rdtgroup_list) { + prgrp->mon.cntr_id[0 ] =MON_CNTR_UNSET ;
Quoted 8 lines
arch/x86/kernel/cpu/resctrl/rdtgroup.c static int rdtgroup_mbm_assign_mode_show(struct kernfs_open_file *of, · line 918
Quoted 6 lines
arch/x86/kernel/cpu/resctrl/rdtgroup.c +6
-0+ prgrp->mon.cntr_id[1 ] =MON_CNTR_UNSET ; +list_for_each_entry (crgrp, &prgrp->mon.crdtgrp_list, + mon.crdtgrp_list) { + crgrp->mon.cntr_id[0 ] =MON_CNTR_UNSET ; + crgrp->mon.cntr_id[1 ] =MON_CNTR_UNSET ; + }Please use
MBM_EVENT_ARRAY_INDEX
Sure.
Quoted 49 lines
Quoted 21 lines
arch/x86/kernel/cpu/resctrl/rdtgroup.c +20
-0+ } +} + +static ssize_t rdtgroup_mbm_assign_mode_write (struct kernfs_open_file *of, +char *buf,size_t nbytes,loff_t off) +{ +struct rdt_resource *r = of->kn->parent->priv; +int ret =0 ; +bool enable; + +/* Valid input requires a trailing newline */ +if (nbytes ==0 || buf[nbytes -1 ] !='\n' ) +return -EINVAL ; + + buf[nbytes -1 ] ='\0' ; + +cpus_read_lock (); +mutex_lock (&rdtgroup_mutex); + +rdt_last_cmd_clear ();arch/x86/kernel/cpu/resctrl/rdtgroup.c static int rdtgroup_mbm_assign_mode_show(struct kernfs_open_file *of, · line 948
Quoted 10 lines
arch/x86/kernel/cpu/resctrl/rdtgroup.c +10
-0+ +if (!strcmp (buf,"default" )) { + enable =0 ; + }else if (!strcmp (buf,"mbm_cntr_assign" )) { + enable =1 ; + }else { + ret = -EINVAL ; +rdt_last_cmd_puts ("Unsupported assign mode\n" ); + goto write_exit; + }Please keep two things in mind:
* this file is always accessible, whether platform supports assignable counters or not.
* this is resctrl fs code.So, considering above, how should user interpret the "Unsupported assign mode"? Shouldn't it also return this error if a user attempts to enable "
mbm_cntr_assign" on a platform that does not support this mode?arch/x86/kernel/cpu/resctrl/rdtgroup.c static int rdtgroup_mbm_assign_mode_show(struct kernfs_open_file *of, · line 950
Quoted 2 lines
arch/x86/kernel/cpu/resctrl/rdtgroup.c +2
-0+ +if (enable !=resctrl_arch_mbm_cntr_assign_enabled (r)) {
resctrl_arch_mbm_cntr_assign_enabled()returns true ifmbm_cntr_assignmode is enabled, but when it returns false it could mean different things:
platform supportsmbm_cntr_assignmode, but it is disabled, or platform does not supportmbm_cntr_assignmode.
Quoted 3 lines
resctrl fs should not rely on all archs to duplicate the all the checking done in
resctrl_arch_mbm_cntr_assign_set(). It should never ask arch to enable a mode that it knows the platform is not capable of.
Agree. This should take care of it. Good catch.
if (!strcmp(buf, "default")) {
enable = 0;
} else if (!strcmp(buf, "mbm_cntr_assign")) {
if (r->mon.mbm_cntr_assignable) {
enable = 1;
} else {
ret = -EINVAL;
rdt_last_cmd_puts("mbm_cntr_assign mode is not supported\n");
goto write_exit;
}
} else {
ret = -EINVAL;
rdt_last_cmd_puts("Unsupported assign mode\n");
goto write_exit;}
Quoted 32 lines
Quoted 18 lines
arch/x86/kernel/cpu/resctrl/rdtgroup.c +12
-0+ ret =resctrl_arch_mbm_cntr_assign_set (r, enable); +if (!ret) +mbm_cntr_reset (r); + } + +write_exit: +mutex_unlock (&rdtgroup_mutex); +cpus_read_unlock (); + +return ret ?: nbytes; +} +static int rdtgroup_num_mbm_cntrs_show (struct kernfs_open_file *of,struct seq_file *s,void *v) {@@ -2115,9 +2187,10 @@ static struct rftype res_common_files[] = { },arch/x86/kernel/cpu/resctrl/rdtgroup.c static struct rftype res_common_files[] = { · line 2196
Quoted 10 lines
arch/x86/kernel/cpu/resctrl/rdtgroup.c +2
-1{ .name = "mbm_assign_mode", - .mode = 0444, + .mode = 0644, .kf_ops = &rdtgroup_kf_single_ops, .seq_show = rdtgroup_mbm_assign_mode_show, + .write = rdtgroup_mbm_assign_mode_write, .fflags = RFTYPE_MON_INFO, }, {Reinette
Signature · 1 line
- Babu Moger
Babu Moger
reviewerPATCH v8 22/25 x86/resctrl: Configure mbm_cntr_assign mode if supported
In reply to Babu MogerConfigure mbm_cntr_assign on AMD. 'mbm_cntr_assign' mode in AMD is ABMC (Assignable Bandwidth Monitoring Counters). It is enabled by default when supported on the system.
When the ABMC is updated, it must be updated on all the logical processors in the resctrl domain.
---
v8: Renamed resctrl_arch_mbm_cntr_assign_configure to
resctrl_arch_mbm_cntr_assign_set_one.
Adde r->mon_capable check.
Commit message update.v7: Introduced resctrl_arch_mbm_cntr_assign_configure() to configure.
Moved the default settings to rdt_get_mon_l3_config(). It should be done before the hotplug handler is called. It cannot be done at rdtgroup_init().
v6: Keeping the default enablement in arch init code for now.
This may need some discussion.
Renamed resctrl_arch_configure_abmc to resctrl_arch_mbm_cntr_assign_configure.
v5: New patch to enable ABMC by default.
--- arch/x86/kernel/cpu/resctrl/internal.h | 1 + arch/x86/kernel/cpu/resctrl/monitor.c | 1 + arch/x86/kernel/cpu/resctrl/rdtgroup.c | 11 +++++++++++ 3 files changed, 13 insertions(+)
arch/x86/kernel/cpu/resctrl/internal.h +1 -0
@@ -719,6 +719,7 @@ int rdtgroup_assign_cntr_event(struct rdt_resource *r, struct rdtgroup *rdtgrp, struct rdt_mon_domain *d, enum resctrl_event_id evtid);int rdtgroup_unassign_cntr_event (struct rdt_resource *r,struct rdtgroup *rdtgrp,struct rdt_mon_domain *d,enum resctrl_event_id evtid); +void resctrl_arch_mbm_cntr_assign_set_one (struct rdt_resource *r);void rdt_staged_configs_clear (void );bool closid_allocated (unsigned int closid);int resctrl_find_cleanest_closid (void );
arch/x86/kernel/cpu/resctrl/monitor.c +1 -0
@@ -1268,6 +1268,7 @@ int __init rdt_get_mon_l3_config(struct rdt_resource *r)cpuid_count (0x80000020 ,5 , &eax, &ebx, &ecx, &edx); r->mon.num_mbm_cntrs = (ebx &GENMASK (15 ,0 )) +1 ;resctrl_file_fflags_init ("num_mbm_cntrs" ,RFTYPE_MON_INFO ); + hw_res->mbm_cntr_assign_enabled = true; } }
arch/x86/kernel/cpu/resctrl/rdtgroup.c +11 -0
@@ -2736,6 +2736,13 @@ int resctrl_arch_mbm_cntr_assign_set(struct rdt_resource *r, bool enable)return 0 ; } +void resctrl_arch_mbm_cntr_assign_set_one (struct rdt_resource *r) +{ +struct rdt_hw_resource *hw_res =resctrl_to_arch_res (r); + +resctrl_abmc_set_one_amd (&hw_res->mbm_cntr_assign_enabled); +} +/* * We don't allow rdtgroup directories to be created anywhere * except the root directory. Thus when looking for the rdtgroup
@@ -4523,9 +4530,13 @@ int resctrl_online_mon_domain(struct rdt_resource *r, struct rdt_mon_domain *d)void resctrl_online_cpu (unsigned int cpu) { +struct rdt_resource *r = &rdt_resources_all[RDT_RESOURCE_L3 ].r_resctrl; +mutex_lock (&rdtgroup_mutex);/* The CPU is set in default rdtgroup after online. */ cpumask_set_cpu (cpu, &rdtgroup_default.cpu_mask); +if (r->mon_capable && r->mon.mbm_cntr_assignable) +resctrl_arch_mbm_cntr_assign_set_one (r);mutex_unlock (&rdtgroup_mutex); }
Signature · 1 line
2.34.1
Babu Moger
reviewerPATCH v8 23/25 x86/resctrl: Update assignments on event configuration changes
In reply to Babu MogerUsers can modify the configuration of assignable events. Whenever the event configuration is updated, MBM assignments must be revised across all monitor groups within the impacted domains.
---
v8: Patch changed completely.
Updated the assignment on same IPI as the event is updated. Could not do the way we discussed in the thread.
https://lore.kernel.org/lkml/[email protected]/
Needed to figure out event type to update the configuration.
v7: New patch to update the assignments. Missed it earlier.
--- arch/x86/kernel/cpu/resctrl/rdtgroup.c | 49 ++++++++++++++++++++++++++ 1 file changed, 49 insertions(+)
arch/x86/kernel/cpu/resctrl/rdtgroup.c +49 -0
@@ -1669,6 +1669,7 @@ static int rdtgroup_size_show(struct kernfs_open_file *of, }struct mon_config_info { +struct rdt_resource *r;struct rdt_mon_domain *d;u32 evtid;u32 mon_config;
@@ -1694,11 +1695,46 @@ u32 resctrl_arch_mon_event_config_get(struct rdt_mon_domain *d,return INVALID_CONFIG_VALUE ; } +static void mbm_cntr_event_update (int cntr_id,unsigned int index,u32 val) +{ +union l3_qos_abmc_cfg abmc_cfg = {0 }; +struct rdtgroup *prgrp, *crgrp; +int update =0 ; + +/* Check if the cntr_id is associated to the event type updated */ +list_for_each_entry (prgrp, &rdt_all_groups, rdtgroup_list) { +if (prgrp->mon.cntr_id[index] == cntr_id) { + abmc_cfg.split.bw_src = prgrp->mon.rmid; + update =1 ; + goto out_update; + } +list_for_each_entry (crgrp, &prgrp->mon.crdtgrp_list, mon.crdtgrp_list) { +if (crgrp->mon.cntr_id[index] == cntr_id) { + abmc_cfg.split.bw_src = crgrp->mon.rmid; + update =1 ; + goto out_update; + } + } + } + +out_update: +if (update) { + abmc_cfg.split.cfg_en =1 ; + abmc_cfg.split.cntr_en =1 ; + abmc_cfg.split.cntr_id = cntr_id; + abmc_cfg.split.bw_type = val; +wrmsrl (MSR_IA32_L3_QOS_ABMC_CFG , abmc_cfg.full); + } +} +void resctrl_arch_mon_event_config_set (void *info) {struct mon_config_info *mon_info = info; +struct rdt_mon_domain *d = mon_info->d; +struct rdt_resource *r = mon_info->r;struct rdt_hw_mon_domain *hw_dom;unsigned int index; +int cntr_id; index =mon_event_config_index_get (mon_info->evtid);if (index ==INVALID_CONFIG_INDEX )
@@ -1718,6 +1754,18 @@ void resctrl_arch_mon_event_config_set(void *info) hw_dom->mbm_local_cfg = mon_info->mon_config;break ; } + +/* +* Update the assignment if the domain has the cntr_id's assigned +* to event type updated. +*/ +if (resctrl_arch_mbm_cntr_assign_enabled (r)) { +for (cntr_id =0 ; cntr_id < r->mon.num_mbm_cntrs; cntr_id++) { +if (test_bit (cntr_id, d->mbm_cntr_map)) +mbm_cntr_event_update (cntr_id, index, + mon_info->mon_config); + } + } }/**
@@ -1805,6 +1853,7 @@ static void mbm_config_write_domain(struct rdt_resource *r, mon_info.d = d; mon_info.evtid = evtid; mon_info.mon_config = val; + mon_info.r = r;/* * Update MSR_IA32_EVT_CFG_BASE MSR on one of the CPUs in the
Signature · 1 line
2.34.1
Reinette Chatre
maintainerRe: [PATCH v8 23/25] x86/resctrl: Update assignments on event configuration changes
Hi Babu,
Quoted 6 lines · Babu Moger
Users can modify the configuration of assignable events. Whenever the event configuration is updated, MBM assignments must be revised across all monitor groups within the impacted domains.
Signed-off-by Babu Moger <[email protected]>---
...
Quoted 30 lines
--- arch/x86/kernel/cpu/resctrl/rdtgroup.c | 49 ++++++++++++++++++++++++++ 1 file changed, 49 insertions(+)arch/x86/kernel/cpu/resctrl/rdtgroup.c +11
-0@@ -1669,6 +1669,7 @@ static int rdtgroup_size_show(struct kernfs_open_file *of, }struct mon_config_info { +struct rdt_resource *r;struct rdt_mon_domain *d;u32 evtid;u32 mon_config;@@ -1694,11 +1695,46 @@ u32 resctrl_arch_mon_event_config_get(struct rdt_mon_domain *d,return INVALID_CONFIG_VALUE ; } +static void mbm_cntr_event_update (int cntr_id,unsigned int index,u32 val) +{ +union l3_qos_abmc_cfg abmc_cfg = {0 }; +struct rdtgroup *prgrp, *crgrp; +int update =0 ; + +/* Check if the cntr_id is associated to the event type updated */ +list_for_each_entry (prgrp, &rdt_all_groups, rdtgroup_list) { +if (prgrp->mon.cntr_id[index] == cntr_id) { + abmc_cfg.split.bw_src = prgrp->mon.rmid;
arch/x86/kernel/cpu/resctrl/rdtgroup.c u32 resctrl_arch_mon_event_config_get(struct rdt_mon_domain *d, · line 1717
Quoted 10 lines
arch/x86/kernel/cpu/resctrl/rdtgroup.c +10
-0+ update =1 ; + goto out_update; + } +list_for_each_entry (crgrp, &prgrp->mon.crdtgrp_list, mon.crdtgrp_list) { +if (crgrp->mon.cntr_id[index] == cntr_id) { + abmc_cfg.split.bw_src = crgrp->mon.rmid; + update =1 ; + goto out_update; + } + }
This code looks like it is better suited for resctrl fs. Note that after the arch fs split struct rdtgroup is private to resctrl fs.
Quoted 43 lines
arch/x86/kernel/cpu/resctrl/rdtgroup.c +27
-0+ } + +out_update: +if (update) { + abmc_cfg.split.cfg_en =1 ; + abmc_cfg.split.cntr_en =1 ; + abmc_cfg.split.cntr_id = cntr_id; + abmc_cfg.split.bw_type = val; +wrmsrl (MSR_IA32_L3_QOS_ABMC_CFG , abmc_cfg.full); + } +} +void resctrl_arch_mon_event_config_set (void *info) {struct mon_config_info *mon_info = info; +struct rdt_mon_domain *d = mon_info->d; +struct rdt_resource *r = mon_info->r;struct rdt_hw_mon_domain *hw_dom;unsigned int index; +int cntr_id; index =mon_event_config_index_get (mon_info->evtid);if (index ==INVALID_CONFIG_INDEX )@@ -1718,6 +1754,18 @@ void resctrl_arch_mon_event_config_set(void *info) hw_dom->mbm_local_cfg = mon_info->mon_config;break ; } + +/* +* Update the assignment if the domain has the cntr_id's assigned +* to event type updated. +*/ +if (resctrl_arch_mbm_cntr_assign_enabled (r)) { +for (cntr_id =0 ; cntr_id < r->mon.num_mbm_cntrs; cntr_id++) { +if (test_bit (cntr_id, d->mbm_cntr_map)) +mbm_cntr_event_update (cntr_id, index, + mon_info->mon_config); + } + } }/**
arch/x86/kernel/cpu/resctrl/rdtgroup.c static void mbm_config_write_domain(struct rdt_resource *r, · line 1859
Quoted 7 lines
arch/x86/kernel/cpu/resctrl/rdtgroup.c +1
-0@@ -1805,6 +1853,7 @@ static void mbm_config_write_domain(struct rdt_resource *r, mon_info.d = d; mon_info.evtid = evtid; mon_info.mon_config = val; + mon_info.r = r;/* * Update MSR_IA32_EVT_CFG_BASE MSR on one of the CPUs in the
If I understand correctly, mbm_config_write_domain() paints itself into a corner by calling arch code via IPI. As seen above it needs resctrl help to get all the information and doing so from the arch helper is not appropriate.
How about calling a resctrl fs helper via IPI instead? For example:
resctrl_mon_event_config_set() {
resctrl_arch_mon_event_config_set();
if (resctrl_arch_mbm_cntr_assign_enabled(r)) {
for (cntr_id = 0; cntr_id < r->mon.num_mbm_cntrs; cntr_id++) {
if (test_bit(cntr_id, d->mbm_cntr_map)) {
/* determine rmid */
resctrl_arch_config_cntr()
}
}
}
}mbm_config_write_domain() {
... smp_call_function_any(&d->hdr.cpu_mask, resctrl_mon_event_config_set, ...) ...
}
By removing reset of arch state from resctrl_arch_config_cntr() this works well with the resctrl_arch_reset_rmid_all() that is done from mbm_config_write_domain(). Even though resctrl_arch_config_cntr() contains a smp_call_function_any() it should already be running on CPU in mask and thus should just run on local CPU.
Reinette
Re: [PATCH v8 23/25] x86/resctrl: Update assignments on event configuration changes
Hi Reinette,
Quoted 46 lines · Reinette Chatre
Hi Babu,
Quoted 6 lines · Babu Moger
Users can modify the configuration of assignable events. Whenever the event configuration is updated, MBM assignments must be revised across all monitor groups within the impacted domains.
Signed-off-by Babu Moger <[email protected]>---
...
Quoted 35 lines
--- arch/x86/kernel/cpu/resctrl/rdtgroup.c | 49 ++++++++++++++++++++++++++ 1 file changed, 49 insertions(+)arch/x86/kernel/cpu/resctrl/rdtgroup.c +16
-0@@ -1669,6 +1669,7 @@ static int rdtgroup_size_show(struct kernfs_open_file *of, }struct mon_config_info { +struct rdt_resource *r;struct rdt_mon_domain *d;u32 evtid;u32 mon_config;@@ -1694,11 +1695,46 @@ u32 resctrl_arch_mon_event_config_get(struct rdt_mon_domain *d,return INVALID_CONFIG_VALUE ; } +static void mbm_cntr_event_update (int cntr_id,unsigned int index,u32 val) +{ +union l3_qos_abmc_cfg abmc_cfg = {0 }; +struct rdtgroup *prgrp, *crgrp; +int update =0 ; + +/* Check if the cntr_id is associated to the event type updated */ +list_for_each_entry (prgrp, &rdt_all_groups, rdtgroup_list) { +if (prgrp->mon.cntr_id[index] == cntr_id) { + abmc_cfg.split.bw_src = prgrp->mon.rmid; + update =1 ; + goto out_update; + } +list_for_each_entry (crgrp, &prgrp->mon.crdtgrp_list, mon.crdtgrp_list) { +if (crgrp->mon.cntr_id[index] == cntr_id) {
Quoted 8 lines
arch/x86/kernel/cpu/resctrl/rdtgroup.c u32 resctrl_arch_mon_event_config_get(struct rdt_mon_domain *d, · line 1717
Quoted 5 lines
arch/x86/kernel/cpu/resctrl/rdtgroup.c +5
-0+ abmc_cfg.split.bw_src = crgrp->mon.rmid; + update =1 ; + goto out_update; + } + }This code looks like it is better suited for resctrl fs. Note that after the arch fs split struct rdtgroup is private to resctrl fs.
ok
Quoted 60 lines
Quoted 44 lines
arch/x86/kernel/cpu/resctrl/rdtgroup.c +27
-0+ } + +out_update: +if (update) { + abmc_cfg.split.cfg_en =1 ; + abmc_cfg.split.cntr_en =1 ; + abmc_cfg.split.cntr_id = cntr_id; + abmc_cfg.split.bw_type = val; +wrmsrl (MSR_IA32_L3_QOS_ABMC_CFG , abmc_cfg.full); + } +} +void resctrl_arch_mon_event_config_set (void *info) {struct mon_config_info *mon_info = info; +struct rdt_mon_domain *d = mon_info->d; +struct rdt_resource *r = mon_info->r;struct rdt_hw_mon_domain *hw_dom;unsigned int index; +int cntr_id; index =mon_event_config_index_get (mon_info->evtid);if (index ==INVALID_CONFIG_INDEX )@@ -1718,6 +1754,18 @@ void resctrl_arch_mon_event_config_set(void *info) hw_dom->mbm_local_cfg = mon_info->mon_config;break ; } + +/* +* Update the assignment if the domain has the cntr_id's assigned +* to event type updated. +*/ +if (resctrl_arch_mbm_cntr_assign_enabled (r)) { +for (cntr_id =0 ; cntr_id < r->mon.num_mbm_cntrs; cntr_id++) { +if (test_bit (cntr_id, d->mbm_cntr_map)) +mbm_cntr_event_update (cntr_id, index, + mon_info->mon_config); + } + } }/** arch/x86/kernel/cpu/resctrl/rdtgroup.c static void mbm_config_write_domain(struct rdt_resource *r, · line 1859
Quoted 7 lines
arch/x86/kernel/cpu/resctrl/rdtgroup.c +1
-0@@ -1805,6 +1853,7 @@ static void mbm_config_write_domain(struct rdt_resource *r, mon_info.d = d; mon_info.evtid = evtid; mon_info.mon_config = val; + mon_info.r = r;/* * Update MSR_IA32_EVT_CFG_BASE MSR on one of the CPUs in the If I understand correctly,
mbm_config_write_domain()paints itself into a corner by calling arch code via IPI. As seen above it needs resctrl help to get all the information and doing so from the arch helper is not appropriate.How about calling a resctrl fs helper via IPI instead? For example:
resctrl_mon_event_config_set(){
Quoted 7 lines
resctrl_arch_mon_event_config_set();
if (resctrl_arch_mbm_cntr_assign_enabled(r)) {
for (cntr_id = 0; cntr_id < r->mon.num_mbm_cntrs; cntr_id++) {
if (test_bit(cntr_id, d->mbm_cntr_map)) {
/* determine rmid */
resctrl_arch_config_cntr()The call resctrl_arch_config_cntr() requires both RMID and CLOSID. So, we will have to find the rdtgroup here (not just RMID, we need CLOSID also).
Yea. I think it can be done. Let me try.
Quoted 12 lines
} } } }
mbm_config_write_domain(){... smp_call_function_any(&d->hdr.cpu_mask, resctrl_mon_event_config_set, ...) ...
Quoted 6 lines
}
By removing reset of arch state from
resctrl_arch_config_cntr()this works well with theresctrl_arch_reset_rmid_all()that is done frommbm_config_write_domain(). Even thoughresctrl_arch_config_cntr()contains asmp_call_function_any()it should already be running on CPU in mask and thus should just run on local CPU.
Ok.
Signature · 1 line
- Babu Moger
Babu Moger
reviewerPATCH v8 24/25 x86/resctrl: Introduce interface to list assignment states of all the groups
In reply to Babu MogerProvide the interface to list the assignment states of all the resctrl groups in mbm_cntr_assign mode.
Example:
$mount -t resctrl resctrl /sys/fs/resctrl/ $cat /sys/fs/resctrl/info/L3_MON/mbm_assign_control //0=tl;1=tl;
List follows the following format:
"<CTRL_MON group>/<MON group>/<domain_id>=<flags>"
Format for specific type of groups:
- Default CTRL_MON group:
"//<domain_id>=<flags>"
- Non-default CTRL_MON group:
"<CTRL_MON group>//<domain_id>=<flags>"
- Child MON group of default CTRL_MON group:
"/<MON group>/<domain_id>=<flags>"
- Child MON group of non-default CTRL_MON group:
"<CTRL_MON group>/<MON group>/<domain_id>=<flags>"
Flags can be one of the following:
t MBM total event is enabled
l MBM local event is enabled
tl Both total and local MBM events are enabled
_ None of the MBM events are enabled
---
v8: Moved resctrl_mbm_event_assigned() in here as it is first used here.
Moved rdt_last_cmd_clear() before making any call. Updated the commit log.
Corrected the doc format.
v7: Renamed the interface name from 'mbm_control' to 'mbm_assign_control'
to match 'mbm_assign_mode'.
Removed Arch references from FS code.
Added rdt_last_cmd_clear() before the command processing. Added rdtgroup_mutex before all the calls.
Removed references of ABMC from FS code.
v6: The domain specific assignment can be determined looking at mbm_cntr_map.
Removed rdtgroup_abmc_dom_cfg() and rdtgroup_abmc_dom_state(). Removed the switch statement for the domain_state detection. Determined the flags incremently.
Removed special handling of default group while printing..
v5: Replaced "assignment flags" with "flags".
Changes related to mon structure.
Changes related renaming the interface from mbm_assign_control to mbm_control.
v4: Added functionality to query domain specific assigment in.
rdtgroup_abmc_dom_state().
v3: New patch.
Addresses the feedback to provide the global assignment interface.
https://lore.kernel.org/lkml/[email protected]/
--- Documentation/arch/x86/resctrl.rst | 44 +++++++++++++++ arch/x86/kernel/cpu/resctrl/monitor.c | 1 + arch/x86/kernel/cpu/resctrl/rdtgroup.c | 76 ++++++++++++++++++++++++++ 3 files changed, 121 insertions(+)
Documentation/arch/x86/resctrl.rst +44 -0
@@ -310,6 +310,50 @@ with the following files: The number of monitoring counters available for assignment when the architecture supports mbm_cntr_assign mode. +"mbm_assign_control": + Reports the resctrl group and monitor status of each group. + + List follows the following format: + "<CTRL_MON group>/<MON group>/<domain_id>=<flags>" + + Format for specific type of groups: + + * Default CTRL_MON group: + "//<domain_id>=<flags>" + + * Non-default CTRL_MON group: + "<CTRL_MON group>//<domain_id>=<flags>" + + * Child MON group of default CTRL_MON group: + "/<MON group>/<domain_id>=<flags>" + + * Child MON group of non-default CTRL_MON group: + "<CTRL_MON group>/<MON group>/<domain_id>=<flags>" + + Flags can be one of the following: + :: + + t MBM total event is assigned. + l MBM local event is assigned. + tl Both total and local MBM events are assigned. + _ None of the MBM events are assigned. + + Examples: + :: + + # mkdir /sys/fs/resctrl/mon_groups/child_default_mon_grp + # mkdir /sys/fs/resctrl/non_default_ctrl_mon_grp + # mkdir /sys/fs/resctrl/non_default_ctrl_mon_grp/mon_groups/child_non_default_mon_grp + + # cat /sys/fs/resctrl/info/L3_MON/mbm_assign_control + non_default_ctrl_mon_grp//0=tl;1=tl; + non_default_ctrl_mon_grp/child_non_default_mon_grp/0=tl;1=tl; + //0=tl;1=tl; + /child_default_mon_grp/0=tl;1=tl; + + There are four resctrl groups. All the groups have total and local MBM events + assigned on domain 0 and 1. + "max_threshold_occupancy": Read/write file provides the largest value (in bytes) at which a previously used LLC_occupancy
arch/x86/kernel/cpu/resctrl/monitor.c +1 -0
@@ -1269,6 +1269,7 @@ int __init rdt_get_mon_l3_config(struct rdt_resource *r) r->mon.num_mbm_cntrs = (ebx &GENMASK (15 ,0 )) +1 ;resctrl_file_fflags_init ("num_mbm_cntrs" ,RFTYPE_MON_INFO ); hw_res->mbm_cntr_assign_enabled = true; +resctrl_file_fflags_init ("mbm_assign_control" ,RFTYPE_MON_INFO ); } }
arch/x86/kernel/cpu/resctrl/rdtgroup.c +76 -0
88 lines not on this page · raw message
Signature · 1 line
2.34.1
Reinette Chatre
maintainerRe: [PATCH v8 24/25] x86/resctrl: Introduce interface to list assignment states of all the groups
Hi Babu,
Quoted 141 lines · Babu Moger
Provide the interface to list the assignment states of all the resctrl groups in
mbm_cntr_assignmode.Example:
$mount -t resctrl resctrl/sys/fs/resctrl/$cat/sys/fs/resctrl/info/L3_MON/mbm_assign_control//0=tl;1=tl;List follows the following format:
"<
CTRL_MONgroup>/<MON group>/<domain_id>=<flags>"Format for specific type of groups:
- Default
CTRL_MONgroup:
"//<domain_id>=<flags>"- Non-default
CTRL_MONgroup:
"<CTRL_MONgroup>//<domain_id>=<flags>"- Child MON group of default
CTRL_MONgroup:
"/<MON group>/<domain_id>=<flags>"- Child MON group of non-default
CTRL_MONgroup:
"<CTRL_MONgroup>/<MON group>/<domain_id>=<flags>"Flags can be one of the following:
t MBM total event is enabled
l MBM local event is enabled
tl Both total and local MBM events are enabled
_ None of the MBM events are enabledSigned-off-by Babu Moger <[email protected]>---
v8: Movedresctrl_mbm_event_assigned()in here as it is first used here.
Movedrdt_last_cmd_clear()before making any call. Updated the commit log.
Corrected the doc format.v7: Renamed the interface name from '
mbm_control' to 'mbm_assign_control'
to match 'mbm_assign_mode'.
Removed Arch references from FS code.
Addedrdt_last_cmd_clear()before the command processing. Addedrdtgroup_mutexbefore all the calls.
Removed references of ABMC from FS code.v6: The domain specific assignment can be determined looking at
mbm_cntr_map.
Removedrdtgroup_abmc_dom_cfg()andrdtgroup_abmc_dom_state(). Removed the switch statement for thedomain_statedetection. Determined the flags incremently.
Removed special handling of default group while printing..v5: Replaced "assignment flags" with "flags".
Changes related to mon structure.
Changes related renaming the interface frommbm_assign_controltombm_control.v4: Added functionality to query domain specific assigment in.
rdtgroup_abmc_dom_state().v3: New patch.
Addresses the feedback to provide the global assignment interface.
https://lore.kernel.org/lkml/[email protected]/--- Documentation/arch/x86/resctrl.rst | 44 +++++++++++++++ arch/x86/kernel/cpu/resctrl/monitor.c | 1 + arch/x86/kernel/cpu/resctrl/rdtgroup.c | 76 ++++++++++++++++++++++++++ 3 files changed, 121 insertions(+)Documentation/arch/x86/resctrl.rst +44
-0@@ -310,6 +310,50 @@ with the following files: The number of monitoring counters available for assignment when the architecture supports mbm_cntr_assign mode. +"mbm_assign_control": + Reports the resctrl group and monitor status of each group. + + List follows the following format: + "<CTRL_MON group>/<MON group>/<domain_id>=<flags>" + + Format for specific type of groups: + + * Default CTRL_MON group: + "//<domain_id>=<flags>" + + * Non-default CTRL_MON group: + "<CTRL_MON group>//<domain_id>=<flags>" + + * Child MON group of default CTRL_MON group: + "/<MON group>/<domain_id>=<flags>" + + * Child MON group of non-default CTRL_MON group: + "<CTRL_MON group>/<MON group>/<domain_id>=<flags>" + + Flags can be one of the following: + :: + + t MBM total event is assigned. + l MBM local event is assigned. + tl Both total and local MBM events are assigned. + _ None of the MBM events are assigned. + + Examples: + :: + + # mkdir /sys/fs/resctrl/mon_groups/child_default_mon_grp + # mkdir /sys/fs/resctrl/non_default_ctrl_mon_grp + # mkdir /sys/fs/resctrl/non_default_ctrl_mon_grp/mon_groups/child_non_default_mon_grp + + # cat /sys/fs/resctrl/info/L3_MON/mbm_assign_control + non_default_ctrl_mon_grp//0=tl;1=tl; + non_default_ctrl_mon_grp/child_non_default_mon_grp/0=tl;1=tl; + //0=tl;1=tl; + /child_default_mon_grp/0=tl;1=tl; + + There are four resctrl groups. All the groups have total and local MBM events + assigned on domain 0 and 1. + "max_threshold_occupancy": Read/write file provides the largest value (in bytes) at which a previously used LLC_occupancyarch/x86/kernel/cpu/resctrl/monitor.c +1
-0@@ -1269,6 +1269,7 @@ int __init rdt_get_mon_l3_config(struct rdt_resource *r) r->mon.num_mbm_cntrs = (ebx &GENMASK (15 ,0 )) +1 ;resctrl_file_fflags_init ("num_mbm_cntrs" ,RFTYPE_MON_INFO ); hw_res->mbm_cntr_assign_enabled = true; +resctrl_file_fflags_init ("mbm_assign_control" ,RFTYPE_MON_INFO ); } }arch/x86/kernel/cpu/resctrl/rdtgroup.c +0
-0
arch/x86/kernel/cpu/resctrl/rdtgroup.c static int rdtgroup_num_mbm_cntrs_show(struct kernfs_open_file *of, · line 974
Quoted 5 lines
arch/x86/kernel/cpu/resctrl/rdtgroup.c +2
-0@@ -970,6 +970,76 @@ static int rdtgroup_num_mbm_cntrs_show(struct kernfs_open_file *of,return 0 ; } +static bool resctrl_mbm_event_assigned (struct rdtgroup *rdtg, +struct rdt_mon_domain *d,u32 evtid)
u32 -> enum resctrl_event_id ?
Quoted 16 lines
arch/x86/kernel/cpu/resctrl/rdtgroup.c +16
-0+{ +int index =MBM_EVENT_ARRAY_INDEX (evtid); +int cntr_id = rdtg->mon.cntr_id[index]; + +return cntr_id !=MON_CNTR_UNSET &&test_bit (cntr_id, d->mbm_cntr_map); +} + +static char *rdtgroup_mon_state_to_str (struct rdtgroup *rdtgrp, +struct rdt_mon_domain *d,char *str) +{ +char *tmp = str; + +/* Query the total and local event flags for the domain */ +if (resctrl_mbm_event_assigned (rdtgrp, d,QOS_L3_MBM_TOTAL_EVENT_ID )) + *tmp++ ='t' ; +
arch/x86/kernel/cpu/resctrl/rdtgroup.c static int rdtgroup_num_mbm_cntrs_show(struct kernfs_open_file *of, · line 1000
Quoted 10 lines
arch/x86/kernel/cpu/resctrl/rdtgroup.c +10
-0+if (resctrl_mbm_event_assigned (rdtgrp, d,QOS_L3_MBM_LOCAL_EVENT_ID )) + *tmp++ ='l' ; + +if (tmp == str) + *tmp++ ='_' ; + + *tmp ='\0' ; +return str; +} +
Reinette
Moger, Babu
reviewerRe: [PATCH v8 24/25] x86/resctrl: Introduce interface to list assignment states of all the groups
Hi Reinette,
Quoted 147 lines · Reinette Chatre
Hi Babu,
Quoted 144 lines · Babu Moger
Provide the interface to list the assignment states of all the resctrl groups in
mbm_cntr_assignmode.Example:
$mount -t resctrl resctrl/sys/fs/resctrl/$cat/sys/fs/resctrl/info/L3_MON/mbm_assign_control//0=tl;1=tl;List follows the following format:
"<
CTRL_MONgroup>/<MON group>/<domain_id>=<flags>"Format for specific type of groups:
- Default
CTRL_MONgroup:
"//<domain_id>=<flags>"- Non-default
CTRL_MONgroup:
"<CTRL_MONgroup>//<domain_id>=<flags>"- Child MON group of default
CTRL_MONgroup:
"/<MON group>/<domain_id>=<flags>"- Child MON group of non-default
CTRL_MONgroup:
"<CTRL_MONgroup>/<MON group>/<domain_id>=<flags>"Flags can be one of the following:
t MBM total event is enabled
l MBM local event is enabled
tl Both total and local MBM events are enabled
_ None of the MBM events are enabledSigned-off-by Babu Moger <[email protected]>---
v8: Movedresctrl_mbm_event_assigned()in here as it is first used here.
Movedrdt_last_cmd_clear()before making any call. Updated the commit log.
Corrected the doc format.v7: Renamed the interface name from '
mbm_control' to 'mbm_assign_control'
to match 'mbm_assign_mode'.
Removed Arch references from FS code.
Addedrdt_last_cmd_clear()before the command processing. Addedrdtgroup_mutexbefore all the calls.
Removed references of ABMC from FS code.v6: The domain specific assignment can be determined looking at
mbm_cntr_map.
Removedrdtgroup_abmc_dom_cfg()andrdtgroup_abmc_dom_state(). Removed the switch statement for thedomain_statedetection. Determined the flags incremently.
Removed special handling of default group while printing..v5: Replaced "assignment flags" with "flags".
Changes related to mon structure.
Changes related renaming the interface frommbm_assign_controltombm_control.v4: Added functionality to query domain specific assigment in.
rdtgroup_abmc_dom_state().v3: New patch.
Addresses the feedback to provide the global assignment interface.
https://lore.kernel.org/lkml/[email protected]/--- Documentation/arch/x86/resctrl.rst | 44 +++++++++++++++ arch/x86/kernel/cpu/resctrl/monitor.c | 1 + arch/x86/kernel/cpu/resctrl/rdtgroup.c | 76 ++++++++++++++++++++++++++ 3 files changed, 121 insertions(+)Documentation/arch/x86/resctrl.rst +44
-0@@ -310,6 +310,50 @@ with the following files: The number of monitoring counters available for assignment when the architecture supports mbm_cntr_assign mode. +"mbm_assign_control": + Reports the resctrl group and monitor status of each group. + + List follows the following format: + "<CTRL_MON group>/<MON group>/<domain_id>=<flags>" + + Format for specific type of groups: + + * Default CTRL_MON group: + "//<domain_id>=<flags>" + + * Non-default CTRL_MON group: + "<CTRL_MON group>//<domain_id>=<flags>" + + * Child MON group of default CTRL_MON group: + "/<MON group>/<domain_id>=<flags>" + + * Child MON group of non-default CTRL_MON group: + "<CTRL_MON group>/<MON group>/<domain_id>=<flags>" + + Flags can be one of the following: + :: + + t MBM total event is assigned. + l MBM local event is assigned. + tl Both total and local MBM events are assigned. + _ None of the MBM events are assigned. + + Examples: + :: + + # mkdir /sys/fs/resctrl/mon_groups/child_default_mon_grp + # mkdir /sys/fs/resctrl/non_default_ctrl_mon_grp + # mkdir /sys/fs/resctrl/non_default_ctrl_mon_grp/mon_groups/child_non_default_mon_grp + + # cat /sys/fs/resctrl/info/L3_MON/mbm_assign_control + non_default_ctrl_mon_grp//0=tl;1=tl; + non_default_ctrl_mon_grp/child_non_default_mon_grp/0=tl;1=tl; + //0=tl;1=tl; + /child_default_mon_grp/0=tl;1=tl; + + There are four resctrl groups. All the groups have total and local MBM events + assigned on domain 0 and 1. + "max_threshold_occupancy": Read/write file provides the largest value (in bytes) at which a previously used LLC_occupancyarch/x86/kernel/cpu/resctrl/monitor.c +1
-0@@ -1269,6 +1269,7 @@ int __init rdt_get_mon_l3_config(struct rdt_resource *r) r->mon.num_mbm_cntrs = (ebx &GENMASK (15 ,0 )) +1 ;resctrl_file_fflags_init ("num_mbm_cntrs" ,RFTYPE_MON_INFO ); hw_res->mbm_cntr_assign_enabled = true; +resctrl_file_fflags_init ("mbm_assign_control" ,RFTYPE_MON_INFO ); } }arch/x86/kernel/cpu/resctrl/rdtgroup.c +0
-0@@ -970,6 +970,76 @@ static int rdtgroup_num_mbm_cntrs_show(struct kernfs_open_file *of,return 0 ; }
Quoted 4 lines
arch/x86/kernel/cpu/resctrl/rdtgroup.c static int rdtgroup_num_mbm_cntrs_show(struct kernfs_open_file *of, · line 974
Quoted 2 lines
arch/x86/kernel/cpu/resctrl/rdtgroup.c +2
-0+static bool resctrl_mbm_event_assigned (struct rdtgroup *rdtg, +struct rdt_mon_domain *d,u32 evtid)u32 -> enum
resctrl_event_id?
Sure.
Quoted 31 lines
Quoted 17 lines
arch/x86/kernel/cpu/resctrl/rdtgroup.c +16
-0+{ +int index =MBM_EVENT_ARRAY_INDEX (evtid); +int cntr_id = rdtg->mon.cntr_id[index]; + +return cntr_id !=MON_CNTR_UNSET &&test_bit (cntr_id, d->mbm_cntr_map); +} + +static char *rdtgroup_mon_state_to_str (struct rdtgroup *rdtgrp, +struct rdt_mon_domain *d,char *str) +{ +char *tmp = str; + +/* Query the total and local event flags for the domain */ +if (resctrl_mbm_event_assigned (rdtgrp, d,QOS_L3_MBM_TOTAL_EVENT_ID )) + *tmp++ ='t' ; +arch/x86/kernel/cpu/resctrl/rdtgroup.c static int rdtgroup_num_mbm_cntrs_show(struct kernfs_open_file *of, · line 1000
Quoted 10 lines
arch/x86/kernel/cpu/resctrl/rdtgroup.c +10
-0+if (resctrl_mbm_event_assigned (rdtgrp, d,QOS_L3_MBM_LOCAL_EVENT_ID )) + *tmp++ ='l' ; + +if (tmp == str) + *tmp++ ='_' ; + + *tmp ='\0' ; +return str; +} +Reinette
Signature · 2 lines
Thanks Babu Moger
Babu Moger
reviewerPATCH v8 25/25 x86/resctrl: Introduce interface to modify assignment states of the groups
In reply to Babu MogerIntroduce the interface to assign MBM events in mbm_cntr_assign mode.
Events can be enabled or disabled by writing to file /sys/fs/resctrl/info/L3_MON/mbm_assign_control
Format is similar to the list format with addition of opcode for the assignment operation.
"<CTRL_MON group>/<MON group>/<domain_id><opcode><flags>"
Format for specific type of groups:
* Default CTRL_MON group:
"//<domain_id><opcode><flags>"
* Non-default CTRL_MON group:
"<CTRL_MON group>//<domain_id><opcode><flags>"
* Child MON group of default CTRL_MON group:
"/<MON group>/<domain_id><opcode><flags>"
* Child MON group of non-default CTRL_MON group:
"<CTRL_MON group>/<MON group>/<domain_id><opcode><flags>"
Domain_id '*' will apply the flags on all the domains.
Opcode can be one of the following:
= Update the assignment to match the flags
+ Assign a new MBM event without impacting existing assignments.
- Unassign a MBM event from currently assigned events.
Assignment flags can be one of the following:
t MBM total event
l MBM local event
tl Both total and local MBM events
_ None of the MBM events. Valid only with '=' opcode. This flag cannot
be combined with other flags.
---
v8: Moved unassign as the first action during the assign modification.
Assign none "_" takes priority. Cannot be mixed with other flags. Updated the documentation and .rst file format. htmldoc looks ok.
v7: Simplified the parsing (strsep(&token, "//") in rdtgroup_mbm_assign_control_write().
Added mutex lock in rdtgroup_mbm_assign_control_write() while processing. Renamed rdtgroup_find_grp to rdtgroup_find_grp_by_name.
Fixed rdtgroup_str_to_mon_state to return error for invalid flags. Simplified the calls rdtgroup_assign_cntr by merging few functions earlier. Removed ABMC reference in FS code.
Reinette commented about handling the combination of flags like 'lt_' and '_lt'. Not sure if we need to change the behaviour here. Processed them sequencially right now. Users have the liberty to pass the flags. Restricting it might be a problem later.
v6: Added support assign all if domain id is '*'
Fixed the allocation of counter id if it not assigned already.
v5: Interface name changed from mbm_assign_control to mbm_control.
Fixed opcode and flags combination.
'=_" is valid.
"-_" amd "+_" is not valid.
Minor message update.
Renamed the function with prefix - rdtgroup_. Corrected few documentation mistakes.
Rebase related changes after SNC support.
v4: Added domain specific assignments. Fixed the opcode parsing.
v3: New patch.
Addresses the feedback to provide the global assignment interface.
https://lore.kernel.org/lkml/[email protected]/
--- Documentation/arch/x86/resctrl.rst | 115 +++++++++++- arch/x86/kernel/cpu/resctrl/internal.h | 10 ++ arch/x86/kernel/cpu/resctrl/rdtgroup.c | 233 ++++++++++++++++++++++++- 3 files changed, 356 insertions(+), 2 deletions(-)
Documentation/arch/x86/resctrl.rst +114 -1
@@ -336,7 +336,8 @@ with the following files: t MBM total event is assigned. l MBM local event is assigned. tl Both total and local MBM events are assigned. - _ None of the MBM events are assigned. + _ None of the MBM events are assigned. Only works with opcode '=' for write + and cannot be combined with other flags. Examples: ::
@@ -354,6 +355,118 @@ with the following files: There are four resctrl groups. All the groups have total and local MBM events assigned on domain 0 and 1. + Assignment state can be updated by writing to the interface. + + Format is similar to the list format with addition of opcode for the + assignment operation. + + "<CTRL_MON group>/<MON group>/<domain_id><opcode><flags>" + + Format for each type of groups: + + * Default CTRL_MON group: + "//<domain_id><opcode><flags>" + + * Non-default CTRL_MON group: + "<CTRL_MON group>//<domain_id><opcode><flags>" + + * Child MON group of default CTRL_MON group: + "/<MON group>/<domain_id><opcode><flags>" + + * Child MON group of non-default CTRL_MON group: + "<CTRL_MON group>/<MON group>/<domain_id><opcode><flags>" + + Domain_id '*' will apply the flags on all the domains. + + Opcode can be one of the following: + :: + + = Update the assignment to match the MBM event. + + Assign a new MBM event without impacting existing assignments. + - Unassign a MBM event from currently assigned events. + + Examples: + Initial group status: + :: + + # cat /sys/fs/resctrl/info/L3_MON/mbm_assign_control + non_default_ctrl_mon_grp//0=tl;1=tl; + non_default_ctrl_mon_grp/child_non_default_mon_grp/0=tl;1=tl; + //0=tl;1=tl; + /child_default_mon_grp/0=tl;1=tl; + + To update the default group to assign only total MBM event on domain 0: + :: + + # echo "//0=t" > /sys/fs/resctrl/info/L3_MON/mbm_assign_control + + Assignment status after the update: + :: + + # cat /sys/fs/resctrl/info/L3_MON/mbm_assign_control + non_default_ctrl_mon_grp//0=tl;1=tl; + non_default_ctrl_mon_grp/child_non_default_mon_grp/0=tl;1=tl; + //0=t;1=tl; + /child_default_mon_grp/0=tl;1=tl; + + To update the MON group child_default_mon_grp to remove total MBM event on domain 1: + :: + + # echo "/child_default_mon_grp/1-t" > /sys/fs/resctrl/info/L3_MON/mbm_assign_control + + Assignment status after the update: + :: + + $ cat /sys/fs/resctrl/info/L3_MON/mbm_assign_control + non_default_ctrl_mon_grp//0=tl;1=tl; + non_default_ctrl_mon_grp/child_non_default_mon_grp/0=tl;1=tl; + //0=t;1=tl; + /child_default_mon_grp/0=tl;1=l; + + To update the MON group non_default_ctrl_mon_grp/child_non_default_mon_grp to unassign + both local and total MBM events on domain 1: + :: + + # echo "non_default_ctrl_mon_grp/child_non_default_mon_grp/1=_" > + /sys/fs/resctrl/info/L3_MON/mbm_assign_control + + Assignment status after the update: + :: + + non_default_ctrl_mon_grp//0=tl;1=tl; + non_default_ctrl_mon_grp/child_non_default_mon_grp/0=tl;1=_; + //0=t;1=tl; + /child_default_mon_grp/0=tl;1=l; + + To update the default group to add a local MBM event domain 0. + :: + + # echo "//0+l" > /sys/fs/resctrl/info/L3_MON/mbm_assign_control + + Assignment status after the update: + :: + + # cat /sys/fs/resctrl/info/L3_MON/mbm_assign_control + non_default_ctrl_mon_grp//0=tl;1=tl; + non_default_ctrl_mon_grp/child_non_default_mon_grp/0=tl;1=_; + //0=tl;1=tl; + /child_default_mon_grp/0=tl;1=l; + + To update the non default CTRL_MON group non_default_ctrl_mon_grp to unassign all the + MBM events on all the domains. + :: + + # echo "non_default_ctrl_mon_grp//*=_" > /sys/fs/resctrl/info/L3_MON/mbm_assign_control + + Assignment status after the update: + :: + + #cat /sys/fs/resctrl/info/L3_MON/mbm_assign_control + non_default_ctrl_mon_grp//0=_;1=_; + non_default_ctrl_mon_grp/child_non_default_mon_grp/0=tl;1=_; + //0=tl;1=tl; + /child_default_mon_grp/0=tl;1=l; + "max_threshold_occupancy": Read/write file provides the largest value (in bytes) at which a previously used LLC_occupancy
arch/x86/kernel/cpu/resctrl/internal.h +10 -0
@@ -74,6 +74,16 @@*/ #define MBM_EVENT_ARRAY_INDEX (_event) ((_event) - 2) +/* +* Assignment flags for mbm_cntr_assign feature +*/ +enum { +ASSIGN_NONE =0 , +ASSIGN_TOTAL =BIT (QOS_L3_MBM_TOTAL_EVENT_ID ), +ASSIGN_LOCAL =BIT (QOS_L3_MBM_LOCAL_EVENT_ID ), +ASSIGN_INVALID , +}; +/** * cpumask_any_housekeeping() - Choose any CPU in @mask, preferring those that * aren't marked nohz_full
arch/x86/kernel/cpu/resctrl/rdtgroup.c +232 -1
247 lines not on this page · raw message
Signature · 1 line
2.34.1
Reinette Chatre
maintainerRe: [PATCH v8 25/25] x86/resctrl: Introduce interface to modify assignment states of the groups
Hi Babu,
Quoted 148 lines · Babu Moger
Introduce the interface to assign MBM events in
mbm_cntr_assignmode.Events can be enabled or disabled by writing to file
/sys/fs/resctrl/info/L3_MON/mbm_assign_controlFormat is similar to the list format with addition of opcode for the assignment operation.
"<CTRL_MONgroup>/<MON group>/<domain_id><opcode><flags>"Format for specific type of groups:
* Default
CTRL_MONgroup:"//<domain_id><opcode><flags>"* Non-default
CTRL_MONgroup:"<CTRL_MON group>//<domain_id><opcode><flags>"* Child MON group of default
CTRL_MONgroup:"/<MON group>/<domain_id><opcode><flags>"* Child MON group of non-default
CTRL_MONgroup:"<CTRL_MON group>/<MON group>/<domain_id><opcode><flags>"
Domain_id'*' will apply the flags on all the domains.Opcode can be one of the following:
= Update the assignment to match the flags
+ Assign a new MBM event without impacting existing assignments.
- Unassign a MBM event from currently assigned events.Assignment flags can be one of the following:
t MBM total event
l MBM local event
tl Both total and local MBM events
_ None of the MBM events. Valid only with '=' opcode. This flag cannot
be combined with other flags.Signed-off-by Babu Moger <[email protected]>---
v8: Moved unassign as the first action during the assign modification.
Assign none "_" takes priority. Cannot be mixed with other flags. Updated the documentation and .rst file format. htmldoc looks ok.v7: Simplified the parsing (
strsep(&token, "//")inrdtgroup_mbm_assign_control_write().
Added mutex lock inrdtgroup_mbm_assign_control_write()while processing. Renamedrdtgroup_find_grptordtgroup_find_grp_by_name.
Fixedrdtgroup_str_to_mon_stateto return error for invalid flags. Simplified the callsrdtgroup_assign_cntrby merging few functions earlier. Removed ABMC reference in FS code.
Reinette commented about handling the combination of flags like 'lt_' and '_lt'. Not sure if we need to change the behaviour here. Processed them sequencially right now. Users have the liberty to pass the flags. Restricting it might be a problem later.v6: Added support assign all if domain id is '*'
Fixed the allocation of counter id if it not assigned already.v5: Interface name changed from
mbm_assign_controltombm_control.
Fixed opcode and flags combination.
'=_" is valid.
"-_" amd "+_" is not valid.
Minor message update.
Renamed the function with prefix -rdtgroup_. Corrected few documentation mistakes.
Rebase related changes after SNC support.v4: Added domain specific assignments. Fixed the opcode parsing.
v3: New patch.
Addresses the feedback to provide the global assignment interface.
https://lore.kernel.org/lkml/[email protected]/--- Documentation/arch/x86/resctrl.rst | 115 +++++++++++- arch/x86/kernel/cpu/resctrl/internal.h | 10 ++ arch/x86/kernel/cpu/resctrl/rdtgroup.c | 233 ++++++++++++++++++++++++- 3 files changed, 356 insertions(+), 2 deletions(-)Documentation/arch/x86/resctrl.rst +55
-1@@ -336,7 +336,8 @@ with the following files: t MBM total event is assigned. l MBM local event is assigned. tl Both total and local MBM events are assigned. - _ None of the MBM events are assigned. + _ None of the MBM events are assigned. Only works with opcode '=' for write + and cannot be combined with other flags. Examples: ::@@ -354,6 +355,118 @@ with the following files: There are four resctrl groups. All the groups have total and local MBM events assigned on domain 0 and 1. + Assignment state can be updated by writing to the interface. + + Format is similar to the list format with addition of opcode for the + assignment operation. + + "<CTRL_MON group>/<MON group>/<domain_id><opcode><flags>" + + Format for each type of groups: + + * Default CTRL_MON group: + "//<domain_id><opcode><flags>" + + * Non-default CTRL_MON group: + "<CTRL_MON group>//<domain_id><opcode><flags>" + + * Child MON group of default CTRL_MON group: + "/<MON group>/<domain_id><opcode><flags>" + + * Child MON group of non-default CTRL_MON group: + "<CTRL_MON group>/<MON group>/<domain_id><opcode><flags>" + + Domain_id '*' will apply the flags on all the domains. + + Opcode can be one of the following: + :: + + = Update the assignment to match the MBM event. + + Assign a new MBM event without impacting existing assignments. + - Unassign a MBM event from currently assigned events. + + Examples: + Initial group status: + :: + + # cat /sys/fs/resctrl/info/L3_MON/mbm_assign_control + non_default_ctrl_mon_grp//0=tl;1=tl; + non_default_ctrl_mon_grp/child_non_default_mon_grp/0=tl;1=tl; + //0=tl;1=tl; + /child_default_mon_grp/0=tl;1=tl; + + To update the default group to assign only total MBM event on domain 0: + :: + + # echo "//0=t" > /sys/fs/resctrl/info/L3_MON/mbm_assign_control + + Assignment status after the update: + :: + + # cat /sys/fs/resctrl/info/L3_MON/mbm_assign_control + non_default_ctrl_mon_grp//0=tl;1=tl; + non_default_ctrl_mon_grp/child_non_default_mon_grp/0=tl;1=tl; + //0=t;1=tl; + /child_default_mon_grp/0=tl;1=tl;
Documentation/arch/x86/resctrl.rst with the following files: · line 420
Quoted 10 lines
Documentation/arch/x86/resctrl.rst +10
-0+ + To update the MON group child_default_mon_grp to remove total MBM event on domain 1: + :: + + # echo "/child_default_mon_grp/1-t" > /sys/fs/resctrl/info/L3_MON/mbm_assign_control + + Assignment status after the update: + :: + + $ cat /sys/fs/resctrl/info/L3_MON/mbm_assign_control
Please be consistent by always using "# cat", not sometimes "$ cat" as above.
Quoted 5 lines
Documentation/arch/x86/resctrl.rst +5
-0+ non_default_ctrl_mon_grp//0=tl;1=tl; + non_default_ctrl_mon_grp/child_non_default_mon_grp/0=tl;1=tl; + //0=t;1=tl; + /child_default_mon_grp/0=tl;1=l; +
Documentation/arch/x86/resctrl.rst with the following files: · line 435
Quoted 10 lines
Documentation/arch/x86/resctrl.rst +10
-0+ To update the MON group non_default_ctrl_mon_grp/child_non_default_mon_grp to unassign + both local and total MBM events on domain 1: + :: + + # echo "non_default_ctrl_mon_grp/child_non_default_mon_grp/1=_" > + /sys/fs/resctrl/info/L3_MON/mbm_assign_control + + Assignment status after the update: + :: +
Missing "# cat /sys/fs/resctrl/info/L3_MON/mbm_assign_control"
Quoted 19 lines
Documentation/arch/x86/resctrl.rst +19
-0+ non_default_ctrl_mon_grp//0=tl;1=tl; + non_default_ctrl_mon_grp/child_non_default_mon_grp/0=tl;1=_; + //0=t;1=tl; + /child_default_mon_grp/0=tl;1=l; + + To update the default group to add a local MBM event domain 0. + :: + + # echo "//0+l" > /sys/fs/resctrl/info/L3_MON/mbm_assign_control + + Assignment status after the update: + :: + + # cat /sys/fs/resctrl/info/L3_MON/mbm_assign_control + non_default_ctrl_mon_grp//0=tl;1=tl; + non_default_ctrl_mon_grp/child_non_default_mon_grp/0=tl;1=_; + //0=tl;1=tl; + /child_default_mon_grp/0=tl;1=l; +
Documentation/arch/x86/resctrl.rst with the following files: · line 464
Quoted 10 lines
Documentation/arch/x86/resctrl.rst +10
-0+ To update the non default CTRL_MON group non_default_ctrl_mon_grp to unassign all the + MBM events on all the domains. + :: + + # echo "non_default_ctrl_mon_grp//*=_" > /sys/fs/resctrl/info/L3_MON/mbm_assign_control + + Assignment status after the update: + :: + + #cat /sys/fs/resctrl/info/L3_MON/mbm_assign_control
Please be consistent with spacing "# cat" vs "#cat". This is very noticeable when viewing the formatted docs.
Quoted 194 lines
Documentation/arch/x86/resctrl.rst +5
-0+ non_default_ctrl_mon_grp//0=_;1=_; + non_default_ctrl_mon_grp/child_non_default_mon_grp/0=tl;1=_; + //0=tl;1=tl; + /child_default_mon_grp/0=tl;1=l; + "max_threshold_occupancy": Read/write file provides the largest value (in bytes) at which a previously used LLC_occupancyarch/x86/kernel/cpu/resctrl/internal.h +10
-0@@ -74,6 +74,16 @@*/ #define MBM_EVENT_ARRAY_INDEX (_event) ((_event) - 2) +/* +* Assignment flags for mbm_cntr_assign feature +*/ +enum { +ASSIGN_NONE =0 , +ASSIGN_TOTAL =BIT (QOS_L3_MBM_TOTAL_EVENT_ID ), +ASSIGN_LOCAL =BIT (QOS_L3_MBM_LOCAL_EVENT_ID ), +ASSIGN_INVALID , +}; +/** * cpumask_any_housekeeping() - Choose any CPU in @mask, preferring those that * aren't marked nohz_full arch/x86/kernel/cpu/resctrl/rdtgroup.c +157
-0@@ -1040,6 +1040,236 @@ static int rdtgroup_mbm_assign_control_show(struct kernfs_open_file *of,return 0 ; } +static int rdtgroup_str_to_mon_state (char *flag) +{ +int i, mon_state =ASSIGN_NONE ; + +for (i =0 ; i <strlen (flag); i++) { +switch (*(flag + i)) { +case 't' : + mon_state |=ASSIGN_TOTAL ; +break ; +case 'l' : + mon_state |=ASSIGN_LOCAL ; +break ; +case '_' : +return ASSIGN_NONE ; +default : +return ASSIGN_INVALID ; + } + } + +return mon_state; +} + +static struct rdtgroup *rdtgroup_find_grp_by_name (enum rdt_group_type rtype, +char *p_grp,char *c_grp) +{ +struct rdtgroup *rdtg, *crg; + +if (rtype ==RDTCTRL_GROUP && *p_grp =='\0' ) { +return &rdtgroup_default; + }else if (rtype ==RDTCTRL_GROUP ) { +list_for_each_entry (rdtg, &rdt_all_groups, rdtgroup_list) +if (!strcmp (p_grp, rdtg->kn->name)) +return rdtg; + }else if (rtype ==RDTMON_GROUP ) { +list_for_each_entry (rdtg, &rdt_all_groups, rdtgroup_list) { +if (!strcmp (p_grp, rdtg->kn->name)) { +list_for_each_entry (crg, &rdtg->mon.crdtgrp_list, + mon.crdtgrp_list) { +if (!strcmp (c_grp, crg->kn->name)) +return crg; + } + } + } + } + +return NULL ; +} + +static int rdtgroup_process_flags (struct rdt_resource *r, +enum rdt_group_type rtype, +char *p_grp,char *c_grp,char *tok) +{ +int op, mon_state, assign_state, unassign_state; +char *dom_str, *id_str, *op_str; +struct rdt_mon_domain *d; +struct rdtgroup *rdtgrp; +unsigned long dom_id; +int ret, found =0 ; + + rdtgrp =rdtgroup_find_grp_by_name (rtype, p_grp, c_grp); + +if (!rdtgrp) { +rdt_last_cmd_puts ("Not a valid resctrl group\n" ); +return -EINVAL ; + } + +next: +if (!tok || tok[0 ] =='\0' ) +return 0 ; + +/* Start processing the strings for each domain */ + dom_str =strim (strsep (&tok,";" )); + + op_str =strpbrk (dom_str,"=+-" ); + +if (op_str) { + op = *op_str; + }else { +rdt_last_cmd_puts ("Missing operation =, +, - character\n" ); +return -EINVAL ; + } + + id_str =strsep (&dom_str,"=+-" ); + +/* Check for domain id '*' which means all domains */ +if (id_str && *id_str =='*' ) { + d =NULL ; + goto check_state; + }else if (!id_str ||kstrtoul (id_str,10 , &dom_id)) { +rdt_last_cmd_puts ("Missing domain id\n" ); +return -EINVAL ; + } + +/* Verify if the dom_id is valid */ +list_for_each_entry (d, &r->mon_domains, hdr.list) { +if (d->hdr.id == dom_id) { + found =1 ; +break ; + } + } + +if (!found) { +rdt_last_cmd_printf ("Invalid domain id %ld\n" , dom_id); +return -EINVAL ; + } + +check_state: + mon_state =rdtgroup_str_to_mon_state (dom_str); + +if (mon_state ==ASSIGN_INVALID ) { +rdt_last_cmd_puts ("Invalid assign flag\n" ); + goto out_fail; + } + + assign_state =0 ; + unassign_state =0 ; + +switch (op) { +case '+' : +if (mon_state ==ASSIGN_NONE ) { +rdt_last_cmd_puts ("Invalid assign opcode\n" ); + goto out_fail; + } + assign_state = mon_state; +break ; +case '-' : +if (mon_state ==ASSIGN_NONE ) { +rdt_last_cmd_puts ("Invalid assign opcode\n" ); + goto out_fail; + } + unassign_state = mon_state; +break ; +case '=' : + assign_state = mon_state; + unassign_state = (ASSIGN_TOTAL |ASSIGN_LOCAL ) & ~assign_state; +break ; +default : +break ; + } + +if (unassign_state &ASSIGN_TOTAL ) { + ret =rdtgroup_unassign_cntr_event (r, rdtgrp, d,QOS_L3_MBM_TOTAL_EVENT_ID ); +if (ret) + goto out_fail; + } + +if (unassign_state &ASSIGN_LOCAL ) { + ret =rdtgroup_unassign_cntr_event (r, rdtgrp, d,QOS_L3_MBM_LOCAL_EVENT_ID ); +if (ret) + goto out_fail; + } + +if (assign_state &ASSIGN_TOTAL ) { + ret =rdtgroup_assign_cntr_event (r, rdtgrp, d,QOS_L3_MBM_TOTAL_EVENT_ID ); +if (ret) + goto out_fail; + }
arch/x86/kernel/cpu/resctrl/rdtgroup.c static int rdtgroup_mbm_assign_control_show(struct kernfs_open_file *of, · line 1209
Quoted 10 lines
arch/x86/kernel/cpu/resctrl/rdtgroup.c +10
-0+ +if (assign_state &ASSIGN_LOCAL ) { + ret =rdtgroup_assign_cntr_event (r, rdtgrp, d,QOS_L3_MBM_LOCAL_EVENT_ID ); +if (ret) + goto out_fail; + } + + goto next; + +out_fail:
Is it possible to print a message to the command status to give some details about which request failed? I am wondering about a scenario where a user changes multiple domains of multiple groups, since the operation does not undo changes, it will fail without information to user space about which setting triggered the failure and which settings succeeded. This is similar to what is done when user attempts to move several tasks ... the error will indicate which task triggered failure so that user space knows what completed successfully.
Quoted 13 lines
arch/x86/kernel/cpu/resctrl/rdtgroup.c +13
-0+ +return -EINVAL ; +} + +static ssize_t rdtgroup_mbm_assign_control_write (struct kernfs_open_file *of, +char *buf,size_t nbytes,loff_t off) +{ +struct rdt_resource *r = of->kn->parent->priv; +char *token, *cmon_grp, *mon_grp; +enum rdt_group_type rtype; +int ret; + +/* Valid input requires a trailing newline */
arch/x86/kernel/cpu/resctrl/rdtgroup.c static int rdtgroup_mbm_assign_control_show(struct kernfs_open_file *of, · line 1232
Quoted 10 lines
arch/x86/kernel/cpu/resctrl/rdtgroup.c +10
-0+if (nbytes ==0 || buf[nbytes -1 ] !='\n' ) +return -EINVAL ; + + buf[nbytes -1 ] ='\0' ; + +cpus_read_lock (); +mutex_lock (&rdtgroup_mutex); + +if (!resctrl_arch_mbm_cntr_assign_enabled (r)) { +rdt_last_cmd_puts ("mbm_cntr_assign mode is not enabled\n" );
Writing to last_cmd_status_buf here ...
arch/x86/kernel/cpu/resctrl/rdtgroup.c static int rdtgroup_mbm_assign_control_show(struct kernfs_open_file *of, · line 1238
Quoted 6 lines
arch/x86/kernel/cpu/resctrl/rdtgroup.c +6
-0+mutex_unlock (&rdtgroup_mutex); +cpus_read_unlock (); +return -EINVAL ; + } + +rdt_last_cmd_clear ();
... but initializing buffer here.
Sidenote: This was an issue before. If you receive comments about items in patches, please do check if those comments apply to other patches also.
Quoted 39 lines
arch/x86/kernel/cpu/resctrl/rdtgroup.c +34
-0+ +while ((token =strsep (&buf,"\n" )) !=NULL ) { +if (strstr (token,"/" )) { +/* +* The write command follows the following format: +* “<CTRL_MON group>/<MON group>/<domain_id><opcode><flags>” +* Extract the CTRL_MON group. +*/ + cmon_grp =strsep (&token,"/" ); + +/* +* Extract the MON_GROUP. +* strsep returns empty string for contiguous delimiters. +* Empty mon_grp here means it is a RDTCTRL_GROUP. +*/ + mon_grp =strsep (&token,"/" ); + +if (*mon_grp =='\0' ) + rtype =RDTCTRL_GROUP ; +else + rtype =RDTMON_GROUP ; + + ret =rdtgroup_process_flags (r, rtype, cmon_grp, mon_grp, token); +if (ret) +break ; + } + } + +mutex_unlock (&rdtgroup_mutex); +cpus_read_unlock (); + +return ret ?: nbytes; +} +#ifdef CONFIG_PROC_CPU_RESCTRL /* @@ -2328,9 +2558,10 @@ static struct rftype res_common_files[] = { },
arch/x86/kernel/cpu/resctrl/rdtgroup.c static struct rftype res_common_files[] = { · line 2567
Quoted 10 lines
arch/x86/kernel/cpu/resctrl/rdtgroup.c +2
-1{ .name = "mbm_assign_control", - .mode = 0444, + .mode = 0644, .kf_ops = &rdtgroup_kf_single_ops, .seq_show = rdtgroup_mbm_assign_control_show, + .write = rdtgroup_mbm_assign_control_write, }, { .name = "cpus_list",
On a high level this looks ok but this code needs to be more robust. This will parse data from user space that may include all kinds of input ... think malicious user or a buggy script. I am not able to test this code but I tried to work through what will happen under some wrong input and found some issues. For example, if user space provides input like '//\n' then rdtgroup_process_flags() will be called with token == NULL. This will result in rdtgroup_process_flags() returning "success", but fortunately do nothing, for this invalid input. A more severe example is with input like '//0=\n', from what I can tell this will result in rdtgroup_str_to_mon_state() called with dom_str==NULL that will treat this as ASSIGN_NONE and proceed as if user provided '//0=_'.
This was just some scenarios with basic input that could be typos, no real stress tests. I stopped here though since I believe it is already clear this needs to be more robust. Please do test this interface by exercising it with invalid input and corner cases.
Reinette
Moger, Babu
reviewerRe: [PATCH v8 25/25] x86/resctrl: Introduce interface to modify assignment states of the groups
Hi Reinette,
Quoted 155 lines · Reinette Chatre
Hi Babu,
Quoted 152 lines · Babu Moger
Introduce the interface to assign MBM events in
mbm_cntr_assignmode.Events can be enabled or disabled by writing to file
/sys/fs/resctrl/info/L3_MON/mbm_assign_controlFormat is similar to the list format with addition of opcode for the assignment operation.
"<CTRL_MONgroup>/<MON group>/<domain_id><opcode><flags>"Format for specific type of groups:
* Default
CTRL_MONgroup:"//<domain_id><opcode><flags>"* Non-default
CTRL_MONgroup:"<CTRL_MON group>//<domain_id><opcode><flags>"* Child MON group of default
CTRL_MONgroup:"/<MON group>/<domain_id><opcode><flags>"* Child MON group of non-default
CTRL_MONgroup:"<CTRL_MON group>/<MON group>/<domain_id><opcode><flags>"
Domain_id'*' will apply the flags on all the domains.Opcode can be one of the following:
= Update the assignment to match the flags
+ Assign a new MBM event without impacting existing assignments.
- Unassign a MBM event from currently assigned events.Assignment flags can be one of the following:
t MBM total event
l MBM local event
tl Both total and local MBM events
_ None of the MBM events. Valid only with '=' opcode. This flag cannot
be combined with other flags.Signed-off-by Babu Moger <[email protected]>---
v8: Moved unassign as the first action during the assign modification.
Assign none "_" takes priority. Cannot be mixed with other flags. Updated the documentation and .rst file format. htmldoc looks ok.v7: Simplified the parsing (
strsep(&token, "//")inrdtgroup_mbm_assign_control_write().
Added mutex lock inrdtgroup_mbm_assign_control_write()while processing. Renamedrdtgroup_find_grptordtgroup_find_grp_by_name.
Fixedrdtgroup_str_to_mon_stateto return error for invalid flags. Simplified the callsrdtgroup_assign_cntrby merging few functions earlier. Removed ABMC reference in FS code.
Reinette commented about handling the combination of flags like 'lt_' and '_lt'. Not sure if we need to change the behaviour here. Processed them sequencially right now. Users have the liberty to pass the flags. Restricting it might be a problem later.v6: Added support assign all if domain id is '*'
Fixed the allocation of counter id if it not assigned already.v5: Interface name changed from
mbm_assign_controltombm_control.
Fixed opcode and flags combination.
'=_" is valid.
"-_" amd "+_" is not valid.
Minor message update.
Renamed the function with prefix -rdtgroup_. Corrected few documentation mistakes.
Rebase related changes after SNC support.v4: Added domain specific assignments. Fixed the opcode parsing.
v3: New patch.
Addresses the feedback to provide the global assignment interface.
https://lore.kernel.org/lkml/[email protected]/--- Documentation/arch/x86/resctrl.rst | 115 +++++++++++- arch/x86/kernel/cpu/resctrl/internal.h | 10 ++ arch/x86/kernel/cpu/resctrl/rdtgroup.c | 233 ++++++++++++++++++++++++- 3 files changed, 356 insertions(+), 2 deletions(-)Documentation/arch/x86/resctrl.rst +59
-1@@ -336,7 +336,8 @@ with the following files: t MBM total event is assigned. l MBM local event is assigned. tl Both total and local MBM events are assigned. - _ None of the MBM events are assigned. + _ None of the MBM events are assigned. Only works with opcode '=' for write + and cannot be combined with other flags. Examples: ::@@ -354,6 +355,118 @@ with the following files: There are four resctrl groups. All the groups have total and local MBM events assigned on domain 0 and 1. + Assignment state can be updated by writing to the interface. + + Format is similar to the list format with addition of opcode for the + assignment operation. + + "<CTRL_MON group>/<MON group>/<domain_id><opcode><flags>" + + Format for each type of groups: + + * Default CTRL_MON group: + "//<domain_id><opcode><flags>" + + * Non-default CTRL_MON group: + "<CTRL_MON group>//<domain_id><opcode><flags>" + + * Child MON group of default CTRL_MON group: + "/<MON group>/<domain_id><opcode><flags>" + + * Child MON group of non-default CTRL_MON group: + "<CTRL_MON group>/<MON group>/<domain_id><opcode><flags>" + + Domain_id '*' will apply the flags on all the domains. + + Opcode can be one of the following: + :: + + = Update the assignment to match the MBM event. + + Assign a new MBM event without impacting existing assignments. + - Unassign a MBM event from currently assigned events. + + Examples: + Initial group status: + :: + + # cat /sys/fs/resctrl/info/L3_MON/mbm_assign_control + non_default_ctrl_mon_grp//0=tl;1=tl; + non_default_ctrl_mon_grp/child_non_default_mon_grp/0=tl;1=tl; + //0=tl;1=tl; + /child_default_mon_grp/0=tl;1=tl; + + To update the default group to assign only total MBM event on domain 0: + :: + + # echo "//0=t" > /sys/fs/resctrl/info/L3_MON/mbm_assign_control + + Assignment status after the update: + :: + + # cat /sys/fs/resctrl/info/L3_MON/mbm_assign_control + non_default_ctrl_mon_grp//0=tl;1=tl; + non_default_ctrl_mon_grp/child_non_default_mon_grp/0=tl;1=tl; + //0=t;1=tl; + /child_default_mon_grp/0=tl;1=tl; + + To update the MON group child_default_mon_grp to remove total MBM event on domain 1: + :: +
Quoted 8 lines
Documentation/arch/x86/resctrl.rst with the following files: · line 420
Quoted 6 lines
Documentation/arch/x86/resctrl.rst +6
-0+ # echo "/child_default_mon_grp/1-t" > /sys/fs/resctrl/info/L3_MON/mbm_assign_control + + Assignment status after the update: + :: + + $ cat /sys/fs/resctrl/info/L3_MON/mbm_assign_controlPlease be consistent by always using "# cat", not sometimes "$ cat" as above.
Sure.
Quoted 10 lines
Quoted 10 lines
Documentation/arch/x86/resctrl.rst +9
-0+ non_default_ctrl_mon_grp//0=tl;1=tl; + non_default_ctrl_mon_grp/child_non_default_mon_grp/0=tl;1=tl; + //0=t;1=tl; + /child_default_mon_grp/0=tl;1=l; + + To update the MON group non_default_ctrl_mon_grp/child_non_default_mon_grp to unassign + both local and total MBM events on domain 1: + :: +
Quoted 8 lines
Documentation/arch/x86/resctrl.rst with the following files: · line 435
Quoted 6 lines
Documentation/arch/x86/resctrl.rst +6
-0+ # echo "non_default_ctrl_mon_grp/child_non_default_mon_grp/1=_" > + /sys/fs/resctrl/info/L3_MON/mbm_assign_control + + Assignment status after the update: + :: +Missing "# cat
/sys/fs/resctrl/info/L3_MON/mbm_assign_control"
Sure.
Quoted 25 lines
Quoted 25 lines
Documentation/arch/x86/resctrl.rst +24
-0+ non_default_ctrl_mon_grp//0=tl;1=tl; + non_default_ctrl_mon_grp/child_non_default_mon_grp/0=tl;1=_; + //0=t;1=tl; + /child_default_mon_grp/0=tl;1=l; + + To update the default group to add a local MBM event domain 0. + :: + + # echo "//0+l" > /sys/fs/resctrl/info/L3_MON/mbm_assign_control + + Assignment status after the update: + :: + + # cat /sys/fs/resctrl/info/L3_MON/mbm_assign_control + non_default_ctrl_mon_grp//0=tl;1=tl; + non_default_ctrl_mon_grp/child_non_default_mon_grp/0=tl;1=_; + //0=tl;1=tl; + /child_default_mon_grp/0=tl;1=l; + + To update the non default CTRL_MON group non_default_ctrl_mon_grp to unassign all the + MBM events on all the domains. + :: + + # echo "non_default_ctrl_mon_grp//*=_" > /sys/fs/resctrl/info/L3_MON/mbm_assign_control
Quoted 8 lines
Documentation/arch/x86/resctrl.rst with the following files: · line 464
Quoted 5 lines
Documentation/arch/x86/resctrl.rst +5
-0+ + Assignment status after the update: + :: + + #cat /sys/fs/resctrl/info/L3_MON/mbm_assign_controlPlease be consistent with spacing "# cat" vs "#cat". This is very noticeable when viewing the formatted docs.
Sure.
Quoted 204 lines
Quoted 204 lines
Documentation/arch/x86/resctrl.rst +5
-0+ non_default_ctrl_mon_grp//0=_;1=_; + non_default_ctrl_mon_grp/child_non_default_mon_grp/0=tl;1=_; + //0=tl;1=tl; + /child_default_mon_grp/0=tl;1=l; + "max_threshold_occupancy": Read/write file provides the largest value (in bytes) at which a previously used LLC_occupancyarch/x86/kernel/cpu/resctrl/internal.h +10
-0@@ -74,6 +74,16 @@*/ #define MBM_EVENT_ARRAY_INDEX (_event) ((_event) - 2) +/* +* Assignment flags for mbm_cntr_assign feature +*/ +enum { +ASSIGN_NONE =0 , +ASSIGN_TOTAL =BIT (QOS_L3_MBM_TOTAL_EVENT_ID ), +ASSIGN_LOCAL =BIT (QOS_L3_MBM_LOCAL_EVENT_ID ), +ASSIGN_INVALID , +}; +/** * cpumask_any_housekeeping() - Choose any CPU in @mask, preferring those that * aren't marked nohz_full arch/x86/kernel/cpu/resctrl/rdtgroup.c +166
-0@@ -1040,6 +1040,236 @@ static int rdtgroup_mbm_assign_control_show(struct kernfs_open_file *of,return 0 ; } +static int rdtgroup_str_to_mon_state (char *flag) +{ +int i, mon_state =ASSIGN_NONE ; + +for (i =0 ; i <strlen (flag); i++) { +switch (*(flag + i)) { +case 't' : + mon_state |=ASSIGN_TOTAL ; +break ; +case 'l' : + mon_state |=ASSIGN_LOCAL ; +break ; +case '_' : +return ASSIGN_NONE ; +default : +return ASSIGN_INVALID ; + } + } + +return mon_state; +} + +static struct rdtgroup *rdtgroup_find_grp_by_name (enum rdt_group_type rtype, +char *p_grp,char *c_grp) +{ +struct rdtgroup *rdtg, *crg; + +if (rtype ==RDTCTRL_GROUP && *p_grp =='\0' ) { +return &rdtgroup_default; + }else if (rtype ==RDTCTRL_GROUP ) { +list_for_each_entry (rdtg, &rdt_all_groups, rdtgroup_list) +if (!strcmp (p_grp, rdtg->kn->name)) +return rdtg; + }else if (rtype ==RDTMON_GROUP ) { +list_for_each_entry (rdtg, &rdt_all_groups, rdtgroup_list) { +if (!strcmp (p_grp, rdtg->kn->name)) { +list_for_each_entry (crg, &rdtg->mon.crdtgrp_list, + mon.crdtgrp_list) { +if (!strcmp (c_grp, crg->kn->name)) +return crg; + } + } + } + } + +return NULL ; +} + +static int rdtgroup_process_flags (struct rdt_resource *r, +enum rdt_group_type rtype, +char *p_grp,char *c_grp,char *tok) +{ +int op, mon_state, assign_state, unassign_state; +char *dom_str, *id_str, *op_str; +struct rdt_mon_domain *d; +struct rdtgroup *rdtgrp; +unsigned long dom_id; +int ret, found =0 ; + + rdtgrp =rdtgroup_find_grp_by_name (rtype, p_grp, c_grp); + +if (!rdtgrp) { +rdt_last_cmd_puts ("Not a valid resctrl group\n" ); +return -EINVAL ; + } + +next: +if (!tok || tok[0 ] =='\0' ) +return 0 ; + +/* Start processing the strings for each domain */ + dom_str =strim (strsep (&tok,";" )); + + op_str =strpbrk (dom_str,"=+-" ); + +if (op_str) { + op = *op_str; + }else { +rdt_last_cmd_puts ("Missing operation =, +, - character\n" ); +return -EINVAL ; + } + + id_str =strsep (&dom_str,"=+-" ); + +/* Check for domain id '*' which means all domains */ +if (id_str && *id_str =='*' ) { + d =NULL ; + goto check_state; + }else if (!id_str ||kstrtoul (id_str,10 , &dom_id)) { +rdt_last_cmd_puts ("Missing domain id\n" ); +return -EINVAL ; + } + +/* Verify if the dom_id is valid */ +list_for_each_entry (d, &r->mon_domains, hdr.list) { +if (d->hdr.id == dom_id) { + found =1 ; +break ; + } + } + +if (!found) { +rdt_last_cmd_printf ("Invalid domain id %ld\n" , dom_id); +return -EINVAL ; + } + +check_state: + mon_state =rdtgroup_str_to_mon_state (dom_str); + +if (mon_state ==ASSIGN_INVALID ) { +rdt_last_cmd_puts ("Invalid assign flag\n" ); + goto out_fail; + } + + assign_state =0 ; + unassign_state =0 ; + +switch (op) { +case '+' : +if (mon_state ==ASSIGN_NONE ) { +rdt_last_cmd_puts ("Invalid assign opcode\n" ); + goto out_fail; + } + assign_state = mon_state; +break ; +case '-' : +if (mon_state ==ASSIGN_NONE ) { +rdt_last_cmd_puts ("Invalid assign opcode\n" ); + goto out_fail; + } + unassign_state = mon_state; +break ; +case '=' : + assign_state = mon_state; + unassign_state = (ASSIGN_TOTAL |ASSIGN_LOCAL ) & ~assign_state; +break ; +default : +break ; + } + +if (unassign_state &ASSIGN_TOTAL ) { + ret =rdtgroup_unassign_cntr_event (r, rdtgrp, d,QOS_L3_MBM_TOTAL_EVENT_ID ); +if (ret) + goto out_fail; + } + +if (unassign_state &ASSIGN_LOCAL ) { + ret =rdtgroup_unassign_cntr_event (r, rdtgrp, d,QOS_L3_MBM_LOCAL_EVENT_ID ); +if (ret) + goto out_fail; + } + +if (assign_state &ASSIGN_TOTAL ) { + ret =rdtgroup_assign_cntr_event (r, rdtgrp, d,QOS_L3_MBM_TOTAL_EVENT_ID ); +if (ret) + goto out_fail; + } + +if (assign_state &ASSIGN_LOCAL ) { + ret =rdtgroup_assign_cntr_event (r, rdtgrp, d,QOS_L3_MBM_LOCAL_EVENT_ID ); +if (ret) + goto out_fail; + } + + goto next; +
Quoted 8 lines
arch/x86/kernel/cpu/resctrl/rdtgroup.c static int rdtgroup_mbm_assign_control_show(struct kernfs_open_file *of, · line 1209
Quoted 1 line
arch/x86/kernel/cpu/resctrl/rdtgroup.c +1
-0+out_fail:Is it possible to print a message to the command status to give some details about which request failed? I am wondering about a scenario where a user changes multiple domains of multiple groups, since the operation does not undo changes, it will fail without information to user space about which setting triggered the failure and which settings succeeded. This is similar to what is done when user attempts to move several tasks ... the error will indicate which task triggered failure so that user space knows what completed successfully.
Will add something like this on failure.
rdt_last_cmd_printf("Total event assign failed on domain %d\n", dom_id);
Quoted 18 lines
Quoted 18 lines
arch/x86/kernel/cpu/resctrl/rdtgroup.c +17
-0+ +return -EINVAL ; +} + +static ssize_t rdtgroup_mbm_assign_control_write (struct kernfs_open_file *of, +char *buf,size_t nbytes,loff_t off) +{ +struct rdt_resource *r = of->kn->parent->priv; +char *token, *cmon_grp, *mon_grp; +enum rdt_group_type rtype; +int ret; + +/* Valid input requires a trailing newline */ +if (nbytes ==0 || buf[nbytes -1 ] !='\n' ) +return -EINVAL ; + + buf[nbytes -1 ] ='\0' ;
Quoted 8 lines
arch/x86/kernel/cpu/resctrl/rdtgroup.c static int rdtgroup_mbm_assign_control_show(struct kernfs_open_file *of, · line 1232
Quoted 6 lines
arch/x86/kernel/cpu/resctrl/rdtgroup.c +6
-0+ +cpus_read_lock (); +mutex_lock (&rdtgroup_mutex); + +if (!resctrl_arch_mbm_cntr_assign_enabled (r)) { +rdt_last_cmd_puts ("mbm_cntr_assign mode is not enabled\n" );Writing to
last_cmd_status_bufhere ...
Sure.
Quoted 11 lines
arch/x86/kernel/cpu/resctrl/rdtgroup.c static int rdtgroup_mbm_assign_control_show(struct kernfs_open_file *of, · line 1238
Quoted 7 lines
arch/x86/kernel/cpu/resctrl/rdtgroup.c +6
-0+mutex_unlock (&rdtgroup_mutex); +cpus_read_unlock (); +return -EINVAL ; + } + +rdt_last_cmd_clear ();... but initializing buffer here.
Sidenote: This was an issue before. If you receive comments about items in patches, please do check if those comments apply to other patches also.
Missed it.
Quoted 55 lines
Quoted 40 lines
arch/x86/kernel/cpu/resctrl/rdtgroup.c +34
-0+ +while ((token =strsep (&buf,"\n" )) !=NULL ) { +if (strstr (token,"/" )) { +/* +* The write command follows the following format: +* “<CTRL_MON group>/<MON group>/<domain_id><opcode><flags>” +* Extract the CTRL_MON group. +*/ + cmon_grp =strsep (&token,"/" ); + +/* +* Extract the MON_GROUP. +* strsep returns empty string for contiguous delimiters. +* Empty mon_grp here means it is a RDTCTRL_GROUP. +*/ + mon_grp =strsep (&token,"/" ); + +if (*mon_grp =='\0' ) + rtype =RDTCTRL_GROUP ; +else + rtype =RDTMON_GROUP ; + + ret =rdtgroup_process_flags (r, rtype, cmon_grp, mon_grp, token); +if (ret) +break ; + } + } + +mutex_unlock (&rdtgroup_mutex); +cpus_read_unlock (); + +return ret ?: nbytes; +} +#ifdef CONFIG_PROC_CPU_RESCTRL /* @@ -2328,9 +2558,10 @@ static struct rftype res_common_files[] = { },arch/x86/kernel/cpu/resctrl/rdtgroup.c static struct rftype res_common_files[] = { · line 2567
Quoted 10 lines
arch/x86/kernel/cpu/resctrl/rdtgroup.c +2
-1{ .name = "mbm_assign_control", - .mode = 0444, + .mode = 0644, .kf_ops = &rdtgroup_kf_single_ops, .seq_show = rdtgroup_mbm_assign_control_show, + .write = rdtgroup_mbm_assign_control_write, }, { .name = "cpus_list",On a high level this looks ok but this code needs to be more robust. This will parse data from user space that may include all kinds of input ... think malicious user or a buggy script. I am not able to test this code but I tried to work through what will happen under some wrong input and found some issues. For example, if user space provides
Quoted 8 lines
input like '//\n' then
rdtgroup_process_flags()will be called with token == NULL. This will result inrdtgroup_process_flags()returning "success", but fortunately do nothing, for this invalid input. A more severe example is with input like '//0=\n', from what I can tell this will result inrdtgroup_str_to_mon_state()called with dom_str==NULL that will treat this asASSIGN_NONEand proceed as if user provided '//0=_'.
This was just some scenarios with basic input that could be typos, no real stress tests. I stopped here though since I believe it is already clear this needs to be more robust. Please do test this interface by exercising it with invalid input and corner cases.
Agree.
But, tested the cases you mentioned above. It seems to handle as expected.
# cat /sys/fs/resctrl/info/L3_MON/mbm_assign_control //0=tl;1=tl;2=tl;3=tl;4=tl;5=tl;6=tl;7=tl;8=tl;9=tl;10=tl;11=tl;
#echo '//\n' > /sys/fs/resctrl/info/L3_MON/mbm_assign_control
bash: echo: write error: Invalid argument
# cat /sys/fs/resctrl/info/last_cmd_status Missing operation =, +, - character
#echo '//0=\n' > /sys/fs/resctrl/info/L3_MON/mbm_assign_control
bash: echo: write error: Invalid argument
#cat /sys/fs/resctrl/info/last_cmd_status
Invalid assign flag
#echo '/0=\n' > /sys/fs/resctrl/info/L3_MON/mbm_assign_control
bash: echo: write error: Invalid argument
# cat /sys/fs/resctrl/info/last_cmd_status Not a valid resctrl group
The assign state did not change.
#cat /sys/fs/resctrl/info/L3_MON/mbm_assign_control //0=tl;1=tl;2=tl;3=tl;4=tl;5=tl;6=tl;7=tl;8=tl;9=tl;10=tl;11=tl;
Sure. will test some more combinations to be sure.
Signature · 2 lines
Thanks Babu Moger
Reinette Chatre
maintainerRe: [PATCH v8 25/25] x86/resctrl: Introduce interface to modify assignment states of the groups
Hi Babu,
Quoted 131 lines · Babu
Quoted 121 lines · Reinette Chatre
Quoted 118 lines · Babu Moger
arch/x86/kernel/cpu/resctrl/rdtgroup.c +118
-0+static int rdtgroup_process_flags (struct rdt_resource *r, +enum rdt_group_type rtype, +char *p_grp,char *c_grp,char *tok) +{ +int op, mon_state, assign_state, unassign_state; +char *dom_str, *id_str, *op_str; +struct rdt_mon_domain *d; +struct rdtgroup *rdtgrp; +unsigned long dom_id; +int ret, found =0 ; + + rdtgrp =rdtgroup_find_grp_by_name (rtype, p_grp, c_grp); + +if (!rdtgrp) { +rdt_last_cmd_puts ("Not a valid resctrl group\n" ); +return -EINVAL ; + } + +next: +if (!tok || tok[0 ] =='\0' ) +return 0 ; + +/* Start processing the strings for each domain */ + dom_str =strim (strsep (&tok,";" )); + + op_str =strpbrk (dom_str,"=+-" ); + +if (op_str) { + op = *op_str; + }else { +rdt_last_cmd_puts ("Missing operation =, +, - character\n" ); +return -EINVAL ; + } + + id_str =strsep (&dom_str,"=+-" ); + +/* Check for domain id '*' which means all domains */ +if (id_str && *id_str =='*' ) { + d =NULL ; + goto check_state; + }else if (!id_str ||kstrtoul (id_str,10 , &dom_id)) { +rdt_last_cmd_puts ("Missing domain id\n" ); +return -EINVAL ; + } + +/* Verify if the dom_id is valid */ +list_for_each_entry (d, &r->mon_domains, hdr.list) { +if (d->hdr.id == dom_id) { + found =1 ; +break ; + } + } + +if (!found) { +rdt_last_cmd_printf ("Invalid domain id %ld\n" , dom_id); +return -EINVAL ; + } + +check_state: + mon_state =rdtgroup_str_to_mon_state (dom_str); + +if (mon_state ==ASSIGN_INVALID ) { +rdt_last_cmd_puts ("Invalid assign flag\n" ); + goto out_fail; + } + + assign_state =0 ; + unassign_state =0 ; + +switch (op) { +case '+' : +if (mon_state ==ASSIGN_NONE ) { +rdt_last_cmd_puts ("Invalid assign opcode\n" ); + goto out_fail; + } + assign_state = mon_state; +break ; +case '-' : +if (mon_state ==ASSIGN_NONE ) { +rdt_last_cmd_puts ("Invalid assign opcode\n" ); + goto out_fail; + } + unassign_state = mon_state; +break ; +case '=' : + assign_state = mon_state; + unassign_state = (ASSIGN_TOTAL |ASSIGN_LOCAL ) & ~assign_state; +break ; +default : +break ; + } + +if (unassign_state &ASSIGN_TOTAL ) { + ret =rdtgroup_unassign_cntr_event (r, rdtgrp, d,QOS_L3_MBM_TOTAL_EVENT_ID ); +if (ret) + goto out_fail; + } + +if (unassign_state &ASSIGN_LOCAL ) { + ret =rdtgroup_unassign_cntr_event (r, rdtgrp, d,QOS_L3_MBM_LOCAL_EVENT_ID ); +if (ret) + goto out_fail; + } + +if (assign_state &ASSIGN_TOTAL ) { + ret =rdtgroup_assign_cntr_event (r, rdtgrp, d,QOS_L3_MBM_TOTAL_EVENT_ID ); +if (ret) + goto out_fail; + } + +if (assign_state &ASSIGN_LOCAL ) { + ret =rdtgroup_assign_cntr_event (r, rdtgrp, d,QOS_L3_MBM_LOCAL_EVENT_ID ); +if (ret) + goto out_fail; + } + + goto next; +Quoted 8 lines
Quoted 1 line
+out_fail:
Is it possible to print a message to the command status to give some details about which request failed? I am wondering about a scenario where a user changes multiple domains of multiple groups, since the operation does not undo changes, it will fail without information to user space about which setting triggered the failure and which settings succeeded. This is similar to what is done when user attempts to move several tasks ... the error will indicate which task triggered failure so that user space knows what completed successfully.
Quoted 3 lines
Will add something like this on failure.
rdt_last_cmd_printf("Total event assign failed on domain %d\n", dom_id);
The user may provide changes for several groups in a single write. Could the CTRL_MON and MON group names also be printed? It is not clear to me if it will be easier to print the flags the user provides or verbose text that the flags translate to, that is "t" vs "Total event".
Quoted 40 lines
Quoted 17 lines
Quoted 17 lines
arch/x86/kernel/cpu/resctrl/rdtgroup.c +17
-0+ +return -EINVAL ; +} + +static ssize_t rdtgroup_mbm_assign_control_write (struct kernfs_open_file *of, +char *buf,size_t nbytes,loff_t off) +{ +struct rdt_resource *r = of->kn->parent->priv; +char *token, *cmon_grp, *mon_grp; +enum rdt_group_type rtype; +int ret; + +/* Valid input requires a trailing newline */ +if (nbytes ==0 || buf[nbytes -1 ] !='\n' ) +return -EINVAL ; + + buf[nbytes -1 ] ='\0' ;Quoted 8 lines
arch/x86/kernel/cpu/resctrl/rdtgroup.c static int rdtgroup_mbm_assign_control_show(struct kernfs_open_file *of, · line 1232
Quoted 6 lines
arch/x86/kernel/cpu/resctrl/rdtgroup.c +6
-0+ +cpus_read_lock (); +mutex_lock (&rdtgroup_mutex); + +if (!resctrl_arch_mbm_cntr_assign_enabled (r)) { +rdt_last_cmd_puts ("mbm_cntr_assign mode is not enabled\n" );Writing to
last_cmd_status_bufhere ...Sure.
Quoted 11 lines
arch/x86/kernel/cpu/resctrl/rdtgroup.c static int rdtgroup_mbm_assign_control_show(struct kernfs_open_file *of, · line 1238
Quoted 7 lines
arch/x86/kernel/cpu/resctrl/rdtgroup.c +6
-0+mutex_unlock (&rdtgroup_mutex); +cpus_read_unlock (); +return -EINVAL ; + } + +rdt_last_cmd_clear ();... but initializing buffer here.
Sidenote: This was an issue before. If you receive comments about items in patches, please do check if those comments apply to other patches also.
Quoted 6 lines
Missed it.
Quoted 4 lines
arch/x86/kernel/cpu/resctrl/rdtgroup.c static int rdtgroup_mbm_assign_control_show(struct kernfs_open_file *of, · line 1241
Quoted 4 lines
arch/x86/kernel/cpu/resctrl/rdtgroup.c +3
-0+ +while ((token =strsep (&buf,"\n" )) !=NULL ) { +if (strstr (token,"/" )) {
What is the purpose of this strstr() call?
Quoted 86 lines
Quoted 51 lines
Quoted 36 lines
arch/x86/kernel/cpu/resctrl/rdtgroup.c +31
-0+/* +* The write command follows the following format: +* “<CTRL_MON group>/<MON group>/<domain_id><opcode><flags>” +* Extract the CTRL_MON group. +*/ + cmon_grp =strsep (&token,"/" ); + +/* +* Extract the MON_GROUP. +* strsep returns empty string for contiguous delimiters. +* Empty mon_grp here means it is a RDTCTRL_GROUP. +*/ + mon_grp =strsep (&token,"/" ); + +if (*mon_grp =='\0' ) + rtype =RDTCTRL_GROUP ; +else + rtype =RDTMON_GROUP ; + + ret =rdtgroup_process_flags (r, rtype, cmon_grp, mon_grp, token); +if (ret) +break ; + } + } + +mutex_unlock (&rdtgroup_mutex); +cpus_read_unlock (); + +return ret ?: nbytes; +} +#ifdef CONFIG_PROC_CPU_RESCTRL /* @@ -2328,9 +2558,10 @@ static struct rftype res_common_files[] = { },arch/x86/kernel/cpu/resctrl/rdtgroup.c static struct rftype res_common_files[] = { · line 2567
Quoted 10 lines
arch/x86/kernel/cpu/resctrl/rdtgroup.c +2
-1{ .name = "mbm_assign_control", - .mode = 0444, + .mode = 0644, .kf_ops = &rdtgroup_kf_single_ops, .seq_show = rdtgroup_mbm_assign_control_show, + .write = rdtgroup_mbm_assign_control_write, }, { .name = "cpus_list",On a high level this looks ok but this code needs to be more robust. This will parse data from user space that may include all kinds of input ... think malicious user or a buggy script. I am not able to test this code but I tried to work through what will happen under some wrong input and found some issues. For example, if user space provides
Quoted 8 lines
input like '//\n' then
rdtgroup_process_flags()will be called with token == NULL. This will result inrdtgroup_process_flags()returning "success", but fortunately do nothing, for this invalid input. A more severe example is with input like '//0=\n', from what I can tell this will result inrdtgroup_str_to_mon_state()called with dom_str==NULL that will treat this asASSIGN_NONEand proceed as if user provided '//0=_'.
This was just some scenarios with basic input that could be typos, no real stress tests. I stopped here though since I believe it is already clear this needs to be more robust. Please do test this interface by exercising it with invalid input and corner cases.Agree.
But, tested the cases you mentioned above. It seems to handle as expected.
# cat /sys/fs/resctrl/info/L3_MON/mbm_assign_control //0=tl;1=tl;2=tl;3=tl;4=tl;5=tl;6=tl;7=tl;8=tl;9=tl;10=tl;11=tl;#echo '//\n' >
/sys/fs/resctrl/info/L3_MON/mbm_assign_control
bash: echo: write error: Invalid argument# cat /sys/fs/resctrl/info/last_cmd_status Missing operation =, +, - character#echo '//0=\n' >
/sys/fs/resctrl/info/L3_MON/mbm_assign_control
bash: echo: write error: Invalid argument#cat
/sys/fs/resctrl/info/last_cmd_status
Invalid assign flag#echo '/0=\n' >
/sys/fs/resctrl/info/L3_MON/mbm_assign_control
bash: echo: write error: Invalid argument# cat /sys/fs/resctrl/info/last_cmd_status Not a valid resctrl group
Quoted 5 lines
The assign state did not change.
#cat/sys/fs/resctrl/info/L3_MON/mbm_assign_control//0=tl;1=tl;2=tl;3=tl;4=tl;5=tl;6=tl;7=tl;8=tl;9=tl;10=tl;11=tl;Sure. will test some more combinations to be sure.
hmmm ... these are not quite the examples I shared since from what I can tell it adds a second \n that impacts the processing of string.
Could you please try:
# echo '//' > /sys/fs/resctrl/info/L3_MON/mbm_assign_control and # echo '//0=' > /sys/fs/resctrl/info/L3_MON/mbm_assign_control
Reinette
Re: [PATCH v8 25/25] x86/resctrl: Introduce interface to modify assignment states of the groups
Hi Reinette,
Quoted 134 lines · Reinette Chatre
Hi Babu,
Quoted 131 lines · Babu
Quoted 129 lines · Reinette Chatre
Quoted 109 lines · Babu Moger
(quoted diff) +109
-0+static int rdtgroup_process_flags(struct rdt_resource *r, + enum rdt_group_type rtype, + char *p_grp, char *c_grp, char *tok) +{ + int op, mon_state, assign_state, unassign_state; + char *dom_str, *id_str, *op_str; + struct rdt_mon_domain *d; + struct rdtgroup *rdtgrp; + unsigned long dom_id; + int ret, found = 0; + + rdtgrp = rdtgroup_find_grp_by_name(rtype, p_grp, c_grp); + + if (!rdtgrp) { + rdt_last_cmd_puts("Not a valid resctrl group\n"); + return -EINVAL; + } + +next: + if (!tok || tok[0] == '\0') + return 0; + + /* Start processing the strings for each domain */ + dom_str = strim(strsep(&tok, ";")); + + op_str = strpbrk(dom_str, "=+-"); + + if (op_str) { + op = *op_str; + } else { + rdt_last_cmd_puts("Missing operation =, +, - character\n"); + return -EINVAL; + } + + id_str = strsep(&dom_str, "=+-"); + + /* Check for domain id '*' which means all domains */ + if (id_str && *id_str == '*') { + d = NULL; + goto check_state; + } else if (!id_str || kstrtoul(id_str, 10, &dom_id)) { + rdt_last_cmd_puts("Missing domain id\n"); + return -EINVAL; + } + + /* Verify if the dom_id is valid */ + list_for_each_entry(d, &r->mon_domains, hdr.list) { + if (d->hdr.id == dom_id) { + found = 1; + break; + } + } + + if (!found) { + rdt_last_cmd_printf("Invalid domain id %ld\n", dom_id); + return -EINVAL; + } + +check_state: + mon_state = rdtgroup_str_to_mon_state(dom_str); + + if (mon_state == ASSIGN_INVALID) { + rdt_last_cmd_puts("Invalid assign flag\n"); + goto out_fail; + } + + assign_state = 0; + unassign_state = 0; + + switch (op) { + case '+': + if (mon_state == ASSIGN_NONE) { + rdt_last_cmd_puts("Invalid assign opcode\n"); + goto out_fail; + } + assign_state = mon_state; + break; + case '-': + if (mon_state == ASSIGN_NONE) { + rdt_last_cmd_puts("Invalid assign opcode\n"); + goto out_fail; + } + unassign_state = mon_state; + break; + case '=': + assign_state = mon_state; + unassign_state = (ASSIGN_TOTAL | ASSIGN_LOCAL) & ~assign_state; + break; + default: + break; + } + + if (unassign_state & ASSIGN_TOTAL) { + ret = rdtgroup_unassign_cntr_event(r, rdtgrp, d, QOS_L3_MBM_TOTAL_EVENT_ID); + if (ret) + goto out_fail; + } + + if (unassign_state & ASSIGN_LOCAL) { + ret = rdtgroup_unassign_cntr_event(r, rdtgrp, d, QOS_L3_MBM_LOCAL_EVENT_ID); + if (ret) + goto out_fail; + } + + if (assign_state & ASSIGN_TOTAL) { + ret = rdtgroup_assign_cntr_event(r, rdtgrp, d, QOS_L3_MBM_TOTAL_EVENT_ID); + if (ret) + goto out_fail; + }Review
Quoted 10 lines
(quoted diff) +10
-0+ + if (assign_state & ASSIGN_LOCAL) { + ret = rdtgroup_assign_cntr_event(r, rdtgrp, d, QOS_L3_MBM_LOCAL_EVENT_ID); + if (ret) + goto out_fail; + } + + goto next; + +out_fail:Is it possible to print a message to the command status to give some details about which request failed? I am wondering about a scenario where a user changes multiple domains of multiple groups, since the operation does not undo changes, it will fail without information to user space about which setting triggered the failure and which settings succeeded. This is similar to what is done when user attempts to move several tasks ... the error will indicate which task triggered failure so that user space knows what completed successfully.
Quoted 8 lines
Quoted 3 lines
Will add something like this on failure.
rdt_last_cmd_printf("Total event assign failed on domain %d\n", dom_id);The user may provide changes for several groups in a single write. Could the
CTRL_MONand MON group names also be printed? It is not clear to me if it will be easier to print the flags the user provides or verbose text that the flags translate to, that is "t" vs "Total event".
Yes. We can print generic messages with group names and flags
"Assignment operation +-= failed on resctrl group ABC with flags = lt"
Quoted 41 lines
Quoted 41 lines
Quoted 18 lines
Quoted 18 lines
(quoted diff) +17
-0+ + return -EINVAL; +} + +static ssize_t rdtgroup_mbm_assign_control_write(struct kernfs_open_file *of, + char *buf, size_t nbytes, loff_t off) +{ + struct rdt_resource *r = of->kn->parent->priv; + char *token, *cmon_grp, *mon_grp; + enum rdt_group_type rtype; + int ret; + + /* Valid input requires a trailing newline */ + if (nbytes == 0 || buf[nbytes - 1] != '\n') + return -EINVAL; + + buf[nbytes - 1] = '\0';Quoted 8 lines
Review
Quoted 6 lines
(quoted diff) +6
-0+ + cpus_read_lock(); + mutex_lock(&rdtgroup_mutex); + + if (!resctrl_arch_mbm_cntr_assign_enabled(r)) { + rdt_last_cmd_puts("mbm_cntr_assign mode is not enabled\n");Writing to
last_cmd_status_bufhere ...Sure.
Quoted 11 lines
Review
Quoted 7 lines
(quoted diff) +6
-0+ mutex_unlock(&rdtgroup_mutex); + cpus_read_unlock(); + return -EINVAL; + } + + rdt_last_cmd_clear();... but initializing buffer here.
Sidenote: This was an issue before. If you receive comments about items in patches, please do check if those comments apply to other patches also.
Quoted 8 lines
Quoted 6 lines
Missed it.
Quoted 4 lines
Review
Quoted 4 lines
(quoted diff) +3
-0+ + while ((token = strsep(&buf, "\n")) != NULL) { + if (strstr(token, "/")) {What is the purpose of this
strstr()call?
This is a carry over for v6. Not required. Will remove.
Quoted 93 lines
Quoted 87 lines
Quoted 52 lines
Quoted 37 lines
(quoted diff) +31
-0+ /* + * The write command follows the following format: + * “<CTRL_MON group>/<MON group>/<domain_id><opcode><flags>” + * Extract the CTRL_MON group. + */ + cmon_grp = strsep(&token, "/"); + + /* + * Extract the MON_GROUP. + * strsep returns empty string for contiguous delimiters. + * Empty mon_grp here means it is a RDTCTRL_GROUP. + */ + mon_grp = strsep(&token, "/"); + + if (*mon_grp == '\0') + rtype = RDTCTRL_GROUP; + else + rtype = RDTMON_GROUP; + + ret = rdtgroup_process_flags(r, rtype, cmon_grp, mon_grp, token); + if (ret) + break; + } + } + + mutex_unlock(&rdtgroup_mutex); + cpus_read_unlock(); + + return ret ?: nbytes; +} + #ifdef CONFIG_PROC_CPU_RESCTRL /*@@ -2328,9 +2558,10 @@ static struct rftype res_common_files[] = { },Review static struct rftype res_common_files[] = { · line 2567
Quoted 10 lines
(quoted diff) +2
-1{ .name = "mbm_assign_control", - .mode = 0444, + .mode = 0644, .kf_ops = &rdtgroup_kf_single_ops, .seq_show = rdtgroup_mbm_assign_control_show, + .write = rdtgroup_mbm_assign_control_write, }, { .name = "cpus_list",On a high level this looks ok but this code needs to be more robust. This will parse data from user space that may include all kinds of input ... think malicious user or a buggy script. I am not able to test this code but I tried to work through what will happen under some wrong input and found some issues. For example, if user space provides
Quoted 8 lines
input like '//\n' then
rdtgroup_process_flags()will be called with token == NULL. This will result inrdtgroup_process_flags()returning "success", but fortunately do nothing, for this invalid input. A more severe example is with input like '//0=\n', from what I can tell this will result inrdtgroup_str_to_mon_state()called with dom_str==NULL that will treat this asASSIGN_NONEand proceed as if user provided '//0=_'.
This was just some scenarios with basic input that could be typos, no real stress tests. I stopped here though since I believe it is already clear this needs to be more robust. Please do test this interface by exercising it with invalid input and corner cases.Agree.
But, tested the cases you mentioned above. It seems to handle as expected.
# cat /sys/fs/resctrl/info/L3_MON/mbm_assign_control //0=tl;1=tl;2=tl;3=tl;4=tl;5=tl;6=tl;7=tl;8=tl;9=tl;10=tl;11=tl;#echo '//\n' >
/sys/fs/resctrl/info/L3_MON/mbm_assign_control
bash: echo: write error: Invalid argument# cat /sys/fs/resctrl/info/last_cmd_status Missing operation =, +, - character#echo '//0=\n' >
/sys/fs/resctrl/info/L3_MON/mbm_assign_control
bash: echo: write error: Invalid argument#cat
/sys/fs/resctrl/info/last_cmd_status
Invalid assign flag#echo '/0=\n' >
/sys/fs/resctrl/info/L3_MON/mbm_assign_control
bash: echo: write error: Invalid argument# cat /sys/fs/resctrl/info/last_cmd_status Not a valid resctrl groupQuoted 5 lines
The assign state did not change.
#cat/sys/fs/resctrl/info/L3_MON/mbm_assign_control//0=tl;1=tl;2=tl;3=tl;4=tl;5=tl;6=tl;7=tl;8=tl;9=tl;10=tl;11=tl;Sure. will test some more combinations to be sure.
Quoted 8 lines
hmmm ... these are not quite the examples I shared since from what I can tell it adds a second \n that impacts the processing of string.
Could you please try:
# echo '//' > /sys/fs/resctrl/info/L3_MON/mbm_assign_control and # echo '//0=' > /sys/fs/resctrl/info/L3_MON/mbm_assign_control
Yes. You are right. Above cases does not work as expected.
This should fix. Will test more.
arch/x86/kernel/cpu/resctrl/rdtgroup.c +3 -0
@@ -1044,6 +1044,9 @@ static int rdtgroup_str_to_mon_state(char *flag) {int i, mon_state =ASSIGN_NONE ; +if (!strlen (flag)) +return ASSIGN_INVALID ; +for (i =0 ; i <strlen (flag); i++) {switch (*(flag + i)) {case 't' :
Thanks
- Babu Moger
Reinette Chatre
maintainerHi Babu,
Quoted 18 lines · Babu Moger
This series adds the support for Assignable Bandwidth Monitoring Counters (ABMC). It is also called QoS RMID Pinning feature
Series is written such that it is easier to support other assignable features supported from different vendors.
The feature details are documented in the APM listed below [1].
[1] AMD64 Architecture Programmer's Manual Volume 2: System Programming Publication # 24593 Revision 3.41 section 19.3.3.3 Assignable Bandwidth Monitoring (ABMC). The documentation is available atThe patches are based on top of commit
5b0c5f05fb2fe(tip/master) Merge branch into tip/master: 'x86/splitlock'# Introduction
Quoted 11 lines
Users can create as many monitor groups as RMIDs supported by the hardware. However, bandwidth monitoring feature on AMD system only guarantees that RMIDs currently assigned to a processor will be tracked by hardware. The counters of any other RMIDs which are no longer being tracked will be reset to zero. The MBM event counters return "Unavailable" for the RMIDs that are not tracked by hardware. So, there can be only limited number of groups that can give guaranteed monitoring numbers. With ever changing configurations there is no way to definitely know which of these groups are being tracked for certain point of time. Users do not have the option to monitor a group or set of groups for certain period of time without worrying about RMID being reset in between.
"worrying about RMID being reset in between" -> "worrying about counter being reset in between"?
Quoted 11 lines
The ABMC feature provides an option to the user to assign a hardware counter to an RMID, event pair and monitor the bandwidth as long as it is assigned. The assigned RMID will be tracked by the hardware until the user unassigns it manually. There is no need to worry about counters being reset during this period. Additionally, the user can specify a bitmask identifying the specific bandwidth types from the given source to track with the counter.
Without ABMC enabled, monitoring will work in current 'default' mode without assignment option.
Reinette
Moger, Babu
reviewerHi Reinette,
Thanks again for quick turn around for this series. Will start working on v9.
Quoted 33 lines · Reinette Chatre
Hi Babu,
Quoted 18 lines · Babu Moger
This series adds the support for Assignable Bandwidth Monitoring Counters (ABMC). It is also called QoS RMID Pinning feature
Series is written such that it is easier to support other assignable features supported from different vendors.
The feature details are documented in the APM listed below [1].
[1] AMD64 Architecture Programmer's Manual Volume 2: System Programming Publication # 24593 Revision 3.41 section 19.3.3.3 Assignable Bandwidth Monitoring (ABMC). The documentation is available atThe patches are based on top of commit
5b0c5f05fb2fe(tip/master) Merge branch into tip/master: 'x86/splitlock'# Introduction
Quoted 11 lines
Users can create as many monitor groups as RMIDs supported by the hardware. However, bandwidth monitoring feature on AMD system only guarantees that RMIDs currently assigned to a processor will be tracked by hardware. The counters of any other RMIDs which are no longer being tracked will be reset to zero. The MBM event counters return "Unavailable" for the RMIDs that are not tracked by hardware. So, there can be only limited number of groups that can give guaranteed monitoring numbers. With ever changing configurations there is no way to definitely know which of these groups are being tracked for certain point of time. Users do not have the option to monitor a group or set of groups for certain period of time without worrying about RMID being reset in between.
Quoted 2 lines
"worrying about RMID being reset in between" -> "worrying about counter being reset in between"?
Sure,
Quoted 15 lines
Quoted 9 lines
The ABMC feature provides an option to the user to assign a hardware counter to an RMID, event pair and monitor the bandwidth as long as it is assigned. The assigned RMID will be tracked by the hardware until the user unassigns it manually. There is no need to worry about counters being reset during this period. Additionally, the user can specify a bitmask identifying the specific bandwidth types from the given source to track with the counter.
Quoted 3 lines
Without ABMC enabled, monitoring will work in current 'default' mode without assignment option.
Reinette
Signature · 2 lines
Thanks Babu Moger