x86,fs/resctrl: Keep default MBM mode at boot and fix ABMC
Subsystems (4)
Babu Moger
reviewerv3 cover x86,fs/resctrl: Keep default MBM mode at boot and fix ABMC
Hi All,
This series restores default MBM assignment mode at boot, assigns counters when switching to mbm_event, and fixes ABMC programming for more than 32 counters.
Commit 0f1576e43adc ("x86/resctrl: Configure mbm_event mode if supported") enabled ABMC at boot. That breaks the pqos tool from intel-cmt-cat [1], which treats a non-numeric event read as zero bandwidth. pqos creates 16 or more groups and uses two counters per group. On a platform with 32 counters per domain, that consumes the pool, so further groups read "Unassigned" and pqos reports 0 MB/s.
Leaving the mode at "default" does not make pqos correct. pqos still treats "Unavailable" as zero. Once the default-mode counter pool is exceeded, hardware can re-allocate a counter between reads and pqos can report that as a wraparound. That shows up only after the pool is exceeded. Default mode uses one counter per monitoring group. On existing AMD platforms that pool is 64 counters and may be larger on newer hardware, so common deployments stay within it. mbm_event mode uses two counters per group, so the same 32 counters cover only 16 groups.
mbm_event remains available as an opt-in. Users that need stable measurements beyond the default-mode pool should enable it and rotate assignments. The limitation of default mode is documented.
Patch 1: widen cntr_id in L3_QOS_ABMC_CFG from 5 bits to 12 bits, and limit the enumerated counter count to 4096, which is what that field can encode. The APM [2] at [3] will be updated for the expanded cntr_id field.
Patch 2: when switching to mbm_event, assign counters to existing groups, including the default group, matching mkdir auto-assignment.
Patch 3: leave mbm_assign_mode as "default" at boot. This seems to be the patch that has generated the most discussion. Let me know, if additional justification in the documentation would be helpful, I'd be happy to add it.
Based on tip/master:
1aeb52f7869a Merge branch into tip/master: 'x86/tdx'
[1] https://github.com/intel/intel-cmt-cat/issues/311
[2] AMD64 Architecture Programmer's Manual Volume 2: System Programming, Publication #24593, Revision 3.41, Section 19.3.3.3 "Assignable Bandwidth Monitoring (ABMC)"
[3] https://bugzilla.kernel.org/show_bug.cgi?id=206537
v3:
- Patch 1: drop the 32-bit u64 change and the bw_src widening. Cap the enumerated counter count at 4096 instead of truncating the CPUID field.
- Patch 2: clean up comments. Add Reported-by, Closes, and Cc: stable.
- Patch 3: document one counter per group in default mode and two counters per group in mbm_event mode. Describe the default-mode pool and the pqos wraparound once that pool is exceeded.
v2:
- Added patch 2 to address Sashiko's comment regarding the documentation issue. In fact, it exposed a real issue. When switching to mbm_event mode, existing monitoring groups should be assigned counters whenever counters are available. This provides a smooth transition between modes and aligns the behavior with the existing auto-assignment mechanism.
https://sashiko.dev/#/patchset/8cb66e18e32e4087a9712c1e68ee6da614efe244.1784322818.git.babu.moger%40amd.com
- Combine the two v1 patches and add patch 2 (assign existing groups on mbm_event switch; Sashiko review).
- Keep boot-default separate from the encoding/truncation fixes.
- Document the default-mode counter-pool limitation.
v1: https://lore.kernel.org/lkml/980f39d3a0e0d9f73925e362f835aeef070a1bc5.1784322818.git.babu.moger@amd.com/
v2: https://lore.kernel.org/lkml/[email protected]/
Thanks
Babu Moger
Babu Moger (3):
x86/resctrl: Fix ABMC counter programming
fs/resctrl: Assign counters to existing groups when enabling mbm_event x86,fs/resctrl: Keep mbm_assign_mode in default mode at boot
Documentation/filesystems/resctrl.rst | 57 ++++++++++++++++++-------- arch/x86/kernel/cpu/resctrl/internal.h | 4 +- arch/x86/kernel/cpu/resctrl/monitor.c | 4 +- fs/resctrl/monitor.c | 35 +++++++++++++--- 4 files changed, 73 insertions(+), 27 deletions(-)
Signature · 1 line
2.43.0
Babu Moger
reviewerPATCH v3 1/3 x86/resctrl: Fix ABMC counter programming
AMD's Assignable Bandwidth Monitoring Counters (ABMC) are configured via MSR_IA32_L3_QOS_ABMC_CFG. The architecture [1] received an update that expands the counter ID field (l3_qos_abmc_cfg.split.cntr_id) from 5 to 12 bits.
Use the updated field width. The number of supported counters is enumerated separately. Mark this as a fix to the original ABMC support to avoid misconfigurations caused by truncating counter IDs on hardware that supports a large number of counters.
Also limit the number of supported counters to the maximum value that can be represented by the 12-bit cntr_id field if hardware reports more than 12 bits.
The AMD64 Architecture Programmer's Manual [1], available at [2], will be updated in a future revision to document the expanded cntr_id field.
[1] AMD64 Architecture Programmer's Manual Volume 2: System Programming, Publication #24593, Revision 3.41, Section 19.3.3.3 "Assignable Bandwidth Monitoring (ABMC)"
---
v3: Dropped the fix for truncation on 32-bit x86.
Removed the change bw_src field(RMID) width to 15 bits. Added new check to limit the number of counters to 12 bits.
v2: Moved the link tag to the last.
--- arch/x86/kernel/cpu/resctrl/internal.h | 4 ++-- arch/x86/kernel/cpu/resctrl/monitor.c | 3 ++- 2 files changed, 4 insertions(+), 3 deletions(-)
arch/x86/kernel/cpu/resctrl/internal.h +2 -2
@@ -214,8 +214,8 @@ union l3_qos_abmc_cfg { bw_src :12, reserved1: 3, is_clos : 1, - cntr_id : 5, - reserved : 9, + cntr_id :12, + reserved : 2, cntr_en : 1, cfg_en : 1; } split;
arch/x86/kernel/cpu/resctrl/monitor.c +2 -1
@@ -470,7 +470,8 @@ int __init rdt_get_l3_mon_config(struct rdt_resource *r) r->mon.mbm_cntr_assignable = true; r->mon.mbm_cntr_configurable = true;cpuid_count (0x80000020 ,5 , &eax, &ebx, &ecx, &edx); - r->mon.num_mbm_cntrs = (ebx &GENMASK (15 ,0 )) +1 ; +/* cntr_id is 12 bits and can only encode 4096 counters. */ + r->mon.num_mbm_cntrs =min ((ebx &GENMASK (15 ,0 )) +1 ,BIT (12 )); hw_res->mbm_cntr_assign_enabled = true; }
Signature · 1 line
2.43.0
Reinette Chatre
maintainerRe: [PATCH v3 1/3] x86/resctrl: Fix ABMC counter programming
Hi Babu,
Quoted 3 lines · Babu Moger
AMD's Assignable Bandwidth Monitoring Counters (ABMC) are configured via
MSR_IA32_L3_QOS_ABMC_CFG. The architecture [1] received an update that expands the counter ID field (l3_qos_abmc_cfg.split.cntr_id) from 5 to 12
This describes an architecture change as a code change (l3_qos_abmc_cfg)?
The subject also claims a "fix" but above establishes this just as an update because of architecture change without making clear what the problem needing fixing is. using the tip guidance when writing changelog will ensure that problem description is always clear.
Quoted 4 lines
bits.
Use the updated field width. The number of supported counters is enumerated
The second sentence of above paragraph seems out of place as it seems to add in a bit of additional context as part of the rest of the paragraph that describes the fix.
Quoted 1 line
separately. Mark this as a fix to the original ABMC support to avoid
Tag can speak for itself, just focus on describing the problem being fixed.
The problem and fix are fused together making it difficult to see what is being fixed here. Placing problem in its own paragraph helps to make this clear.
Quoted 6 lines
misconfigurations caused by truncating counter IDs on hardware that supports a large number of counters.
Also limit the number of supported counters to the maximum value that can be represented by the 12-bit
cntr_idfield if hardware reports more than 12 bits.
This is the actual fix though but is added as an afterthought?
Quoted 3 lines
The AMD64 Architecture Programmer's Manual [1], available at [2], will be updated in a future revision to document the expanded
cntr_idfield.
Consider an example changelog like below that aims to follow the x86 changelog guidelines:
AMD's Assignable Bandwidth Monitoring Counters (ABMC) are configured via MSR_IA32_L3_QOS_ABMC_CFG; MSR_IA32_L3_QOS_ABMC_CFG.cntr_id selects which counter the configuration applies to. The number of counters a platform supports (the number of possible values written to MSR_IA32_L3_QOS_ABMC_CFG.cntr_id) is enumerated separately via CPUID.
On a platform that enumerates more than 32 counters, writing a counter ID through the current 5-bit encoding truncates the ID and misprograms ABMC.
The AMD64 Architecture Programmer's Manual [1], available from [2], has been updated to widen MSR_IA32_L3_QOS_ABMC_CFG.cntr_id from 5 bits to 12 bits (the published revision 3.41 does not yet reflect this; a future revision will). The CPUID enumeration reports the maximum counter ID in a 16-bit field and can still report more counters than a 12-bit MSR_IA32_L3_QOS_ABMC_CFG.cntr_id can address.
Widen MSR_IA32_L3_QOS_ABMC_CFG.cntr_id to 12 bits to match the architecture. Cap the enumerated counter count at BIT(12) so every counter ID resctrl uses can be written to MSR_IA32_L3_QOS_ABMC_CFG.cntr_id without truncation.
Quoted 10 lines
[1] AMD64 Architecture Programmer's Manual Volume 2: System Programming, Publication #24593, Revision 3.41, Section 19.3.3.3 "Assignable Bandwidth Monitoring (ABMC)"
Fixes 84ecefb76674 ("x86/resctrl: Add data structures and definitions for ABMC assignment")Signed-off-by Babu Moger <[email protected]>---
Patch looks good to me.
Reinette
Babu Moger
reviewerRe: [PATCH v3 1/3] x86/resctrl: Fix ABMC counter programming
Hi Reinette,
Thanks for the quick review.
Quoted 7 lines · Reinette Chatre
Hi Babu,
Quoted 3 lines · Babu Moger
AMD's Assignable Bandwidth Monitoring Counters (ABMC) are configured via
MSR_IA32_L3_QOS_ABMC_CFG. The architecture [1] received an update that expands the counter ID field (l3_qos_abmc_cfg.split.cntr_id) from 5 to 12
Quoted 6 lines
This describes an architecture change as a code change (
l3_qos_abmc_cfg)?The subject also claims a "fix" but above establishes this just as an update because of architecture change without making clear what the problem needing fixing is. using the tip guidance when writing changelog will ensure that problem description is always clear.
Ack.>
Quoted 16 lines
Quoted 4 lines
bits.
Use the updated field width. The number of supported counters is enumerated
The second sentence of above paragraph seems out of place as it seems to add in a bit of additional context as part of the rest of the paragraph that describes the fix.
Quoted 1 line
separately. Mark this as a fix to the original ABMC support to avoid
Tag can speak for itself, just focus on describing the problem being fixed.
The problem and fix are fused together making it difficult to see what is being fixed here. Placing problem in its own paragraph helps to make this clear.
Quoted 9 lines
Quoted 6 lines
misconfigurations caused by truncating counter IDs on hardware that supports a large number of counters.
Also limit the number of supported counters to the maximum value that can be represented by the 12-bit
cntr_idfield if hardware reports more than 12 bits.This is the actual fix though but is added as an afterthought?
Yea.
Quoted 24 lines
Quoted 3 lines
The AMD64 Architecture Programmer's Manual [1], available at [2], will be updated in a future revision to document the expanded
cntr_idfield.Consider an example changelog like below that aims to follow the x86 changelog guidelines:
AMD's Assignable Bandwidth Monitoring Counters (ABMC) are configured via
MSR_IA32_L3_QOS_ABMC_CFG; MSR_IA32_L3_QOS_ABMC_CFG.cntr_id selects which counter the configuration applies to. The number of counters a platform supports (the number of possible values written to MSR_IA32_L3_QOS_ABMC_CFG.cntr_id) is enumerated separately via CPUID.On a platform that enumerates more than 32 counters, writing a counter ID through the current 5-bit encoding truncates the ID and misprograms ABMC.
The AMD64 Architecture Programmer's Manual [1], available from [2], has been updated to widen MSR_IA32_L3_QOS_ABMC_CFG.cntr_id from 5 bits to 12 bits (the published revision 3.41 does not yet reflect this; a future revision will). The CPUID enumeration reports the maximum counter ID in a 16-bit field and can still report more counters than a 12-bit MSR_IA32_L3_QOS_ABMC_CFG.cntr_id can address.
Quoted 5 lines
Widen MSR_IA32_L3_QOS_ABMC_CFG.cntr_id to 12 bits to match the architecture. Cap the enumerated counter count at BIT(12) so every counter ID resctrl uses can be written to MSR_IA32_L3_QOS_ABMC_CFG.cntr_id without truncation.
Looks good.
Quoted 11 lines
Quoted 10 lines
[1] AMD64 Architecture Programmer's Manual Volume 2: System Programming, Publication #24593, Revision 3.41, Section 19.3.3.3 "Assignable Bandwidth Monitoring (ABMC)"
Fixes 84ecefb76674 ("x86/resctrl: Add data structures and definitions for ABMC assignment")Signed-off-by Babu Moger <[email protected]>---
Patch looks good to me.
Thank you.
Babu
Babu Moger
reviewerPATCH v3 2/3 fs/resctrl: Assign counters to existing groups when enabling mbm_event
In reply to Babu MogerWhen the user enables counter assignment mode by writing "mbm_event" to /sys/fs/resctrl/info/L3_MON/mbm_assign_mode, resctrl resets all monitoring state and sets mbm_assign_on_mkdir for subsequent mkdir, but does not assign counters to groups that already exist, including the default group created at mount. The counters of those groups return "Unassigned" until the user assigns counters by hand.
Enable mbm_assign_on_mkdir and assign counters, while there are some available, to existing CTRL_MON and MON groups so the switch matches mkdir auto-assignment. An event left without a counter reads "Unassigned".
---
v3: Changelog update.
Code comment cleanup.
Added Cc to stable.
Added Reported-by and Closes tags.
v2: New patch.
This patch addresses the Sashiko comment about documentation issue where counters are not assigned automatically when mode is switched to mbm_event.
https://sashiko.dev/#/patchset/8cb66e18e32e4087a9712c1e68ee6da614efe244.1784322818.git.babu.moger%40amd.com
In fact, it exposed a real issue. When switching to mbm_event mode, existing monitoring groups should be assigned counters whenever counters are available. This provides a smooth transition between modes and aligns the behavior with the existing auto-assignment mechanism.
--- Documentation/filesystems/resctrl.rst | 7 ++++-- fs/resctrl/monitor.c | 35 +++++++++++++++++++++++---- 2 files changed, 35 insertions(+), 7 deletions(-)
Documentation/filesystems/resctrl.rst +5 -2
@@ -370,8 +370,11 @@ with the following files: of counters available is described in the "num_mbm_cntrs" file. Changing the mode may cause all counters on the resource to reset. - Moving to mbm_event counter assignment mode requires users to assign the counters - to the events. Otherwise, the MBM event counters will return 'Unassigned' when read. + Moving to mbm_event counter assignment mode enables "mbm_assign_on_mkdir" and + assigns counters to the events of all existing monitoring groups, including + the default group, while counters remain available. Consult + "mbm_L3_assignments" after switching to "mbm_event" mode for counter + assignment states of all monitoring groups. The mode is beneficial for AMD platforms that support more CTRL_MON and MON groups than available hardware counters. By default, this
fs/resctrl/monitor.c +30 -5
@@ -1300,8 +1300,7 @@ static int rdtgroup_assign_cntr_event(struct rdt_l3_mon_domain *d, struct rdtgro }/* -* rdtgroup_assign_cntrs() - Assign counters to MBM events. Called when -* a new group is created. +* rdtgroup_assign_cntrs() - Assign counters to MBM events. * * Each group can accommodate two counters per domain: one for the total * event and one for the local event. Assignments may fail due to the limited
@@ -1326,6 +1325,22 @@ void rdtgroup_assign_cntrs(struct rdtgroup *rdtgrp) &mon_event_all[QOS_L3_MBM_LOCAL_EVENT_ID]); } +/* +* resctrl_assign_cntrs_allrdtgrp() - Assign counters to the MBM events of +* every existing group. +*/ +static void resctrl_assign_cntrs_allrdtgrp (void ) +{ +struct rdtgroup *prgrp, *crgrp; + +list_for_each_entry (prgrp, &rdt_all_groups, rdtgroup_list) { +rdtgroup_assign_cntrs (prgrp); + +list_for_each_entry (crgrp, &prgrp->mon.crdtgrp_list, mon.crdtgrp_list) +rdtgroup_assign_cntrs (crgrp); + } +} +/* * rdtgroup_free_unassign_cntr() - Unassign and reset the counter ID configuration * for the event pointed to by @mevt within the domain @d and resctrl group @rdtgrp.
@@ -1599,9 +1614,6 @@ ssize_t resctrl_mbm_assign_mode_write(struct kernfs_open_file *of, char *buf, (READS_TO_LOCAL_MEM |READS_TO_LOCAL_S_MEM |NON_TEMP_WRITE_TO_LOCAL_MEM ); -/* Enable auto assignment when switching to "mbm_event" mode */ -if (enable) - r->mon.mbm_assign_on_mkdir = true;/* * Reset all the non-achitectural RMID state and assignable counters. */
@@ -1609,6 +1621,19 @@ ssize_t resctrl_mbm_assign_mode_write(struct kernfs_open_file *of, char *buf,mbm_cntr_free_all (r, d);resctrl_reset_rmid_all (r, d); } + +/* +* Counters were freed above, so both new groups (via mkdir) and +* the groups that already exist need assignments. Groups created +* while in "default" mode have no counter assigned, including the +* default group created when resctrl is mounted. Assign counters +* to them so that enabling the mode leaves the same assignments +* that mkdir would have made. +*/ +if (enable) { + r->mon.mbm_assign_on_mkdir = true; +resctrl_assign_cntrs_allrdtgrp (); + } } out_unlock:
Signature · 1 line
2.43.0
Reinette Chatre
maintainerRe: [PATCH v3 2/3] fs/resctrl: Assign counters to existing groups when enabling mbm_event
Hi Babu,
Quoted 6 lines · Babu Moger
When the user enables counter assignment mode by writing "
mbm_event" to/sys/fs/resctrl/info/L3_MON/mbm_assign_mode, resctrl resets all monitoring state and setsmbm_assign_on_mkdirfor subsequent mkdir, but does not assign counters to groups that already exist, including the default group created at mount. The counters of those groups return "Unassigned" until the user assigns counters by hand.
Above summarizes the context and then below follows with the change, what makes this deserve a "Fixes:" tag? What is the issue needing fixing?
Consider, for example:
When the user switches counter assignment mode by writing "mbm_event" to /sys/fs/resctrl/info/L3_MON/mbm_assign_mode, resctrl frees all assignable counters, resets per-domain RMID state, and enables counter auto-assignment exposed to user space as mbm_assign_on_mkdir.
Even though counter auto-assignment is enabled, groups that already exist at the time of the switch, including the default group created at mount, are not assigned a counter. All MBM events read "Unassigned" until the user assigns a counter by hand.
Walk every existing CTRL_MON and its MON children after the reset and assign counters to their MBM events. Enabling "mbm_event" now leaves the same per-group state that auto-assignment would have produced. There may be fewer available counters than MBM events across the existing groups; in that case stop assignment when no counters remain. Events in the remaining groups read "Unassigned", matching the behavior of creating a group when no counters are available.
Quoted 10 lines
Enable
mbm_assign_on_mkdirand assign counters, while there are some available, to existingCTRL_MONand MON groups so the switch matches mkdir auto-assignment. An event left without a counter reads "Unassigned".Fixes 8004ea01cf63 ("fs/resctrl: Introduce the interface to switch between monitor modes")Reported-by Sashiko <[email protected]>
tip tag order places "Cc:" after author's SoB.
Quoted 9 lines
Signed-off-by Babu Moger <[email protected]>---
fs/resctrl/monitor.c +0
-0
fs/resctrl/monitor.c static int rdtgroup_assign_cntr_event(struct rdt_l3_mon_domain *d, struct rdtgro · line 1303
Quoted 6 lines
fs/resctrl/monitor.c +1
-2@@ -1300,8 +1300,7 @@ static int rdtgroup_assign_cntr_event(struct rdt_l3_mon_domain *d, struct rdtgro }/* -* rdtgroup_assign_cntrs() - Assign counters to MBM events. Called when -* a new group is created. +* rdtgroup_assign_cntrs() - Assign counters to MBM events.
This patch adds another caller of rdtgroup_assign_cntrs() and above thus correctly updates the function comments to not just mention the original caller ...
fs/resctrl/monitor.c static int rdtgroup_assign_cntr_event(struct rdt_l3_mon_domain *d, struct rdtgro · line 1306
Quoted 3 lines
fs/resctrl/monitor.c +0
-0* * Each group can accommodate two counters per domain: one for the total * event and one for the local event. Assignments may fail due to the limited
... but the detailed function comments trimmed here still considers original caller as the only use case and uses original caller to justify why it does not need to return an error.
Reinette
Babu Moger
reviewerRe: [PATCH v3 2/3] fs/resctrl: Assign counters to existing groups when enabling mbm_event
Hi Reinette,
Quoted 27 lines · Reinette Chatre
Hi Babu,
Quoted 6 lines · Babu Moger
When the user enables counter assignment mode by writing "
mbm_event" to/sys/fs/resctrl/info/L3_MON/mbm_assign_mode, resctrl resets all monitoring state and setsmbm_assign_on_mkdirfor subsequent mkdir, but does not assign counters to groups that already exist, including the default group created at mount. The counters of those groups return "Unassigned" until the user assigns counters by hand.Above summarizes the context and then below follows with the change, what makes this deserve a "Fixes:" tag? What is the issue needing fixing?
Consider, for example:
When the user switches counter assignment mode by writing "
mbm_event" to/sys/fs/resctrl/info/L3_MON/mbm_assign_mode, resctrl frees all assignable counters, resets per-domain RMID state, and enables counter auto-assignment exposed to user space asmbm_assign_on_mkdir.Even though counter auto-assignment is enabled, groups that already exist at the time of the switch, including the default group created at mount, are not assigned a counter. All MBM events read "Unassigned" until the user assigns a counter by hand.
Quoted 9 lines
Walk every existing
CTRL_MONand its MON children after the reset and assign counters to their MBM events. Enabling "mbm_event" now leaves the same per-group state that auto-assignment would have produced. There may be fewer available counters than MBM events across the existing groups; in that case stop assignment when no counters remain. Events in the remaining groups read "Unassigned", matching the behavior of creating a group when no counters are available.
Looks good. Thanks
Quoted 6 lines
Quoted 6 lines
Enable
mbm_assign_on_mkdirand assign counters, while there are some available, to existingCTRL_MONand MON groups so the switch matches mkdir auto-assignment. An event left without a counter reads "Unassigned".
Quoted 6 lines
Quoted 4 lines
Fixes 8004ea01cf63 ("fs/resctrl: Introduce the interface to switch between monitor modes")Reported-by Sashiko <[email protected]>tip tag order places "Cc:" after author's SoB.
ack.
Quoted 20 lines
Quoted 10 lines
Signed-off-by Babu Moger <[email protected]>---
fs/resctrl/monitor.c +0
-0fs/resctrl/monitor.c static int rdtgroup_assign_cntr_event(struct rdt_l3_mon_domain *d, struct rdtgro · line 1303
Quoted 6 lines
fs/resctrl/monitor.c +1
-2@@ -1300,8 +1300,7 @@ static int rdtgroup_assign_cntr_event(struct rdt_l3_mon_domain *d, struct rdtgro }/* -* rdtgroup_assign_cntrs() - Assign counters to MBM events. Called when -* a new group is created. +* rdtgroup_assign_cntrs() - Assign counters to MBM events. This patch adds another caller of
rdtgroup_assign_cntrs()and above thus correctly updates the function comments to not just mention the original caller ...
Quoted 8 lines
fs/resctrl/monitor.c static int rdtgroup_assign_cntr_event(struct rdt_l3_mon_domain *d, struct rdtgro · line 1306
Quoted 3 lines
fs/resctrl/monitor.c +0
-0* * Each group can accommodate two counters per domain: one for the total * event and one for the local event. Assignments may fail due to the limited ... but the detailed function comments trimmed here still considers original caller as the only use case and uses original caller to justify why it does not need to return an error.
How about this?
/*
* rdtgroup_assign_cntrs() - Assign counters to MBM events.
*
* Each group can accommodate two counters per domain: one for the total
* event and one for the local event. Assignments may fail due to the
limited
* number of counters. However, it is not necessary to return failure.
* Users have the option to modify the counter assignments later if needed. */
Thanks
Ban
Reinette Chatre
maintainerRe: [PATCH v3 2/3] fs/resctrl: Assign counters to existing groups when enabling mbm_event
Hi Babu,
Quoted 8 lines · Babu Moger
How about this?
/*
*rdtgroup_assign_cntrs()- Assign counters to MBM events.
*
* Each group can accommodate two counters per domain: one for the total
* event and one for the local event. Assignments may fail due to the limited
* number of counters. However, it is not necessary to return failure.
This drops the original motivation for why an error is not returned, leaving the claim that "it is not necessary" to return a failure without an explanation why.
An example that adds a motivation:
/*
* rdtgroup_assign_cntrs() - Assign counters to enabled MBM events of a
* resource group
*
* Each group can accommodate two counters per domain: one for the total
* event and one for the local event. Assignments may fail due to the limited
* number of counters.
*
* Intended to be called in auto assignment scenarios ("mbm_assign_on_mkdir"
* enabled in "mbm_event" counter assignment mode) where counter assignment is
* not required to succeed. Thus no failure is returned. Users can modify the
* counter assignments afterwards if needed.
*/
Quoted 2 lines
* Users have the option to modify the counter assignments later if needed. */
Reinette
Re: [PATCH v3 2/3] fs/resctrl: Assign counters to existing groups when enabling mbm_event
Hi Reinette,
Quoted 18 lines · Reinette Chatre
Hi Babu,
Quoted 8 lines · Babu Moger
How about this?
/*
*rdtgroup_assign_cntrs()- Assign counters to MBM events.
*
* Each group can accommodate two counters per domain: one for the total
* event and one for the local event. Assignments may fail due to the limited
* number of counters. However, it is not necessary to return failure.This drops the original motivation for why an error is not returned, leaving the claim that "it is not necessary" to return a failure without an explanation why.
An example that adds a motivation:
Quoted 14 lines
/*
*rdtgroup_assign_cntrs()- Assign counters to enabled MBM events of a
* resource group
*
* Each group can accommodate two counters per domain: one for the total
* event and one for the local event. Assignments may fail due to the limited
* number of counters.
*
* Intended to be called in auto assignment scenarios ("mbm_assign_on_mkdir"
* enabled in "mbm_event" counter assignment mode) where counter assignment is
* not required to succeed. Thus no failure is returned. Users can modify the
* counter assignments afterwards if needed.
*/
Sounds good.
Thanks,
Babu
Babu Moger
reviewerPATCH v3 3/3 x86,fs/resctrl: Keep mbm_assign_mode in default mode at boot
In reply to Babu MogerThe "mbm_event" mode assigns hardware MBM counters to RMID/event pairs. It is intended for deployments that need to manage counter assignment on platforms where the number of monitoring groups exceeds the available hardware counters. Hardware counters are scarce, so mbm_event mode suits workflows that monitor a subset of groups at a time and rotate assignments as needed.
resctrl enables the "mbm_event" mode by default on hardware that supports it. That breaks the pqos tool [1], which assumes the historical default mode. pqos treats a non-numeric event read as zero bandwidth. It creates 16 or more monitoring groups and uses two counters per group. On a platform with 32 counters per domain, that consumes the pool, so further groups read "Unassigned" and pqos reports 0 MB/s for them:
$sudo pqos -m all:0-4
CORE IPC MISSES LLC[KB] MBL[MB/s] MBR[MB/s]
0 0.69 16k 192.0 0.0 0.0
1 0.40 3k 0.0 0.0 0.0
2 0.44 1k 64.0 0.0 0.0
3 0.39 1k 64.0 0.0 0.0
4 0.43 1k 0.0 0.0 0.0Leave mbm_assign_mode in "default" mode during initialization. Default mode uses one counter per monitoring group. On existing AMD platforms that pool is 64 counters and may be larger on newer hardware. The "mbm_event" mode uses two counters per group, so 32 counters cover only 16 groups.
Deployments within that pool receive accurate bandwidth measurements with default mode. The hardware supports 4096 monitoring groups. Beyond that pool, readings may be misleading or "Unavailable", with no user-visible indication.
Default mode is still limited once the number of monitoring groups exceeds that counter pool. After hardware re-allocates a counter, a read may return "Unavailable". pqos still treats "Unavailable" as zero. Successive reads can be a count, "Unavailable", then another count. pqos treats those jumps as wraparound and can report an inconsistent rate:
$sudo pqos -m all:0-4
CORE IPC MISSES LLC[KB] MBL[MB/s] MBR[MB/s] 0 0.76 60k 32.0 0.0 0.0 1 0.45 1k 32.0 0.0 0.0 2 1.57 107k 64.0 0.0 17592186044184.9 3 1.62 276k 4928.0 3.2 0.5
Users that need stable measurements beyond that pool should use "mbm_event" mode and rotate assignments as needed. Enable it with:
$echo mbm_event > /sys/fs/resctrl/info/L3_MON/mbm_assign_mode
--- Documentation/filesystems/resctrl.rst | 48 ++++++++++++++++++--------- arch/x86/kernel/cpu/resctrl/monitor.c | 1 - 2 files changed, 33 insertions(+), 16 deletions(-)
Documentation/filesystems/resctrl.rst +33 -15
@@ -354,8 +354,8 @@ with the following files: :: # cat /sys/fs/resctrl/info/L3_MON/mbm_assign_mode - [mbm_event] - default + [default] + mbm_event "mbm_event":
@@ -363,6 +363,9 @@ with the following files: pair and monitor the bandwidth usage as long as it is assigned. The hardware continues to track the assigned counter until it is explicitly unassigned by the user. Each event within a resctrl group can be assigned independently. + mbm_event mode uses one counter per event of one RMID. A monitoring group + can use two counters, one for mbm_total_bytes and one for mbm_local_bytes. + For example, 32 counters can monitor 16 groups. In this mode, a monitoring event can only accumulate data while it is backed by a hardware counter. Use "mbm_L3_assignments" found in each CTRL_MON and MON
@@ -376,20 +379,37 @@ with the following files: "mbm_L3_assignments" after switching to "mbm_event" mode for counter assignment states of all monitoring groups. - The mode is beneficial for AMD platforms that support more CTRL_MON - and MON groups than available hardware counters. By default, this - feature is enabled on AMD platforms with the ABMC (Assignable Bandwidth - Monitoring Counters) capability, ensuring counters remain assigned even - when the corresponding RMID is not actively used by any processor. + It is intended for deployments that need to manage counter assignment on + platforms where the number of monitoring groups exceeds the available + hardware counters. Hardware counters are scarce, so mbm_event mode suits + workflows that monitor a subset of groups at a time and rotate assignments + as needed. The mode is beneficial for AMD platforms that support more + CTRL_MON and MON groups than available hardware counters. The mbm_event + mode ensures counters remain assigned even when the corresponding RMID is + not actively monitored. "default": In default mode, resctrl assumes there is a hardware counter for each - event within every CTRL_MON and MON group. On AMD platforms, it is - recommended to use the mbm_event mode, if supported, to prevent reset of MBM - events between reads resulting from hardware re-allocating counters. This can - result in misleading values or display "Unavailable" if no counter is assigned - to the event. + event within every CTRL_MON and MON group. This mode is enabled by default. + + On AMD platforms that support more CTRL_MON and MON groups than hardware + counters, hardware dynamically shares a smaller pool of counters among + RMIDs. One counter from this pool is used per monitoring group and counts + every MBM event of that group's RMID. The size of the pool is not + enumerated to software (unlike "num_mbm_cntrs"), and "num_rmids" may be + much larger. For example, 64 counters in this pool monitor 64 groups. + Newer hardware may provide a larger pool. + + While the number of monitoring groups does not exceed that pool, a counter + stays attached to each RMID and readings remain accurate. Creating more + groups than the pool can cause hardware to re-allocate those counters + between successive reads of an event. Bandwidth values may then be + misleading, or a read may return "Unavailable" if no counter is allocated + to the RMID. There is no user-visible indication when this begins. + Users who need stable readings beyond that pool should switch to mbm_event + mode, if supported, and assign counters to the groups of interest + (rotating assignments as needed). * To enable "mbm_event" counter assignment mode: ::
@@ -1787,11 +1807,9 @@ View the llc occupancy snapshot:: Examples on working with mbm_assign_mode ======================================== -a. Check if MBM counter assignment mode is supported. +a. Check if MBM counter assignment mode is supported and enabled. :: - # mount -t resctrl resctrl /sys/fs/resctrl/ - # cat /sys/fs/resctrl/info/L3_MON/mbm_assign_mode [mbm_event] default
arch/x86/kernel/cpu/resctrl/monitor.c +0 -1
@@ -472,7 +472,6 @@ int __init rdt_get_l3_mon_config(struct rdt_resource *r)cpuid_count (0x80000020 ,5 , &eax, &ebx, &ecx, &edx);/* cntr_id is 12 bits and can only encode 4096 counters. */ r->mon.num_mbm_cntrs =min ((ebx &GENMASK (15 ,0 )) +1 ,BIT (12 )); - hw_res->mbm_cntr_assign_enabled = true; } r->mon_capable = true;
Signature · 1 line
2.43.0
Reinette Chatre
maintainerRe: [PATCH v3 3/3] x86,fs/resctrl: Keep mbm_assign_mode in default mode at boot
Hi Babu,
Quoted 4 lines · Babu Moger
The "
mbm_event" mode assigns hardware MBM counters to RMID/event pairs. It is intended for deployments that need to manage counter assignment on platforms where the number of monitoring groups exceeds the available hardware counters. Hardware counters are scarce, sombm_eventmode suits
"Hardware counters are scarce" -> "Hardware counters are scarce on these platforms" (to make it specific to the context and not a general statement)
Quoted 24 lines
workflows that monitor a subset of groups at a time and rotate assignments as needed.
resctrl enables the "
mbm_event" mode by default on hardware that supports it. That breaks the pqos tool [1], which assumes the historical default mode. pqos treats a non-numeric event read as zero bandwidth. It creates 16 or more monitoring groups and uses two counters per group. On a platform with 32 counters per domain, that consumes the pool, so further groups read "Unassigned" and pqos reports 0 MB/s for them:$sudo pqos -m all:0-4
CORE IPC MISSES LLC[KB] MBL[MB/s] MBR[MB/s]0 0.69 16k 192.0 0.0 0.0 1 0.40 3k 0.0 0.0 0.0 2 0.44 1k 64.0 0.0 0.0 3 0.39 1k 64.0 0.0 0.0 4 0.43 1k 0.0 0.0 0.0Leave
mbm_assign_modein "default" mode during initialization. Default mode uses one counter per monitoring group. On existing AMD platforms that pool is 64 counters and may be larger on newer hardware. The "mbm_event" mode uses two counters per group, so 32 counters cover only 16 groups.
Quoted 3 lines
Deployments within that pool receive accurate bandwidth measurements with default mode. The hardware supports 4096 monitoring groups. Beyond that pool, readings may be misleading or "Unavailable", with no
At this point I am lost because the word "pool" is used to mean three different things. It is used for the number of "ABMC counters" as well as the number of "non-ABMC counters" that exist on the same platform, above it seems to be used for number of RMIDs (number of monitoring groups) also. Could you please add some context at beginning of changelog to introduce the various counts applicable to the description, associate unique terms to these counts at that time, and then use those unique terms in the rest of the changelog?
Quoted 20 lines
user-visible indication.
Default mode is still limited once the number of monitoring groups exceeds that counter pool. After hardware re-allocates a counter, a read may return "Unavailable". pqos still treats "Unavailable" as zero. Successive reads can be a count, "Unavailable", then another count. pqos treats those jumps as wraparound and can report an inconsistent rate:
$sudo pqos -m all:0-4
CORE IPC MISSES LLC[KB] MBL[MB/s] MBR[MB/s] 0 0.76 60k 32.0 0.0 0.0 1 0.45 1k 32.0 0.0 0.0 2 1.57 107k 64.0 0.0 17592186044184.9 3 1.62 276k 4928.0 3.2 0.5Users that need stable measurements beyond that pool should use "
mbm_event" mode and rotate assignments as needed. Enable it with:
Quoted 7 lines
$echo
mbm_event>/sys/fs/resctrl/info/L3_MON/mbm_assign_modeFixes 0f1576e43adc ("x86/resctrl: Configure mbm_event mode if supported")Link https://github.com/intel/intel-cmt-cat # [1]Signed-off-by Babu Moger <[email protected]>
See "Ordering of commit tags" in Documentation/process/maintainer-tip.rst
Quoted 21 lines
--- Documentation/filesystems/resctrl.rst | 48 ++++++++++++++++++--------- arch/x86/kernel/cpu/resctrl/monitor.c | 1 - 2 files changed, 33 insertions(+), 16 deletions(-)Documentation/filesystems/resctrl.rst +2
-2@@ -354,8 +354,8 @@ with the following files: :: # cat /sys/fs/resctrl/info/L3_MON/mbm_assign_mode - [mbm_event] - default + [default] + mbm_event "mbm_event":
Documentation/filesystems/resctrl.rst with the following files: · line 368
Quoted 6 lines
Documentation/filesystems/resctrl.rst +3
-0@@ -363,6 +363,9 @@ with the following files: pair and monitor the bandwidth usage as long as it is assigned. The hardware continues to track the assigned counter until it is explicitly unassigned by the user. Each event within a resctrl group can be assigned independently. + mbm_event mode uses one counter per event of one RMID. A monitoring group + can use two counters, one for mbm_total_bytes and one for mbm_local_bytes. + For example, 32 counters can monitor 16 groups.
This addition looks redundant to me. "mbm_event mode uses one counter per event of one RMID." just repeats earlier "...assign a hardware counter to an RMID, event pair ...", no? The example being added ("A monitoring group can use two counters ...") also looks to be specific to AMD's usage and including platform specific example as part of generic introduction seems unnecessary.
Quoted 5 lines
Documentation/filesystems/resctrl.rst +0
-0In this mode, a monitoring event can only accumulate data while it is backed by a hardware counter. Use "mbm_L3_assignments" found in each CTRL_MON and MON@@ -376,20 +379,37 @@ with the following files: "mbm_L3_assignments" after switching to "mbm_event" mode for counter
Documentation/filesystems/resctrl.rst with the following files: · line 384
Quoted 10 lines
Documentation/filesystems/resctrl.rst +3
-5assignment states of all monitoring groups. - The mode is beneficial for AMD platforms that support more CTRL_MON - and MON groups than available hardware counters. By default, this - feature is enabled on AMD platforms with the ABMC (Assignable Bandwidth - Monitoring Counters) capability, ensuring counters remain assigned even - when the corresponding RMID is not actively used by any processor. + It is intended for deployments that need to manage counter assignment on + platforms where the number of monitoring groups exceeds the available + hardware counters. Hardware counters are scarce, so mbm_event mode suits
"Hardware counters are scarce" -> "Hardware counters are scarce on these platforms"?
Documentation/filesystems/resctrl.rst with the following files: · line 386
Quoted 2 lines
Documentation/filesystems/resctrl.rst +2
-0+ workflows that monitor a subset of groups at a time and rotate assignments + as needed. The mode is beneficial for AMD platforms that support more
Text from "The mode is beneficial for AMD ..." does not seem necessary here since it just duplicates the text added below?
Quoted 22 lines
Documentation/filesystems/resctrl.rst +13
-5+ CTRL_MON and MON groups than available hardware counters. The mbm_event + mode ensures counters remain assigned even when the corresponding RMID is + not actively monitored. "default": In default mode, resctrl assumes there is a hardware counter for each - event within every CTRL_MON and MON group. On AMD platforms, it is - recommended to use the mbm_event mode, if supported, to prevent reset of MBM - events between reads resulting from hardware re-allocating counters. This can - result in misleading values or display "Unavailable" if no counter is assigned - to the event. + event within every CTRL_MON and MON group. This mode is enabled by default. + + On AMD platforms that support more CTRL_MON and MON groups than hardware + counters, hardware dynamically shares a smaller pool of counters among + RMIDs. One counter from this pool is used per monitoring group and counts + every MBM event of that group's RMID. The size of the pool is not + enumerated to software (unlike "num_mbm_cntrs"), and "num_rmids" may be + much larger. For example, 64 counters in this pool monitor 64 groups. + Newer hardware may provide a larger pool. +
Documentation/filesystems/resctrl.rst with the following files: · line 413
Quoted 10 lines
Documentation/filesystems/resctrl.rst +9
-0+ While the number of monitoring groups does not exceed that pool, a counter + stays attached to each RMID and readings remain accurate. Creating more + groups than the pool can cause hardware to re-allocate those counters + between successive reads of an event. Bandwidth values may then be + misleading, or a read may return "Unavailable" if no counter is allocated + to the RMID. There is no user-visible indication when this begins. + Users who need stable readings beyond that pool should switch to mbm_event + mode, if supported, and assign counters to the groups of interest + (rotating assignments as needed).
Above usage of "pool" looks ok since it only uses the term for the "non-ABMC hardware counters" while using the accurate terms for the other counts.
Quoted 20 lines
Documentation/filesystems/resctrl.rst +1
-3* To enable "mbm_event" counter assignment mode: ::@@ -1787,11 +1807,9 @@ View the llc occupancy snapshot:: Examples on working with mbm_assign_mode ======================================== -a. Check if MBM counter assignment mode is supported. +a. Check if MBM counter assignment mode is supported and enabled. :: - # mount -t resctrl resctrl /sys/fs/resctrl/ - # cat /sys/fs/resctrl/info/L3_MON/mbm_assign_mode [mbm_event] defaultarch/x86/kernel/cpu/resctrl/monitor.c +0
-0
arch/x86/kernel/cpu/resctrl/monitor.c int __init rdt_get_l3_mon_config(struct rdt_resource *r) · line 477
Quoted 7 lines
arch/x86/kernel/cpu/resctrl/monitor.c +0
-1@@ -472,7 +472,6 @@ int __init rdt_get_l3_mon_config(struct rdt_resource *r)cpuid_count (0x80000020 ,5 , &eax, &ebx, &ecx, &edx);/* cntr_id is 12 bits and can only encode 4096 counters. */ r->mon.num_mbm_cntrs =min ((ebx &GENMASK (15 ,0 )) +1 ,BIT (12 )); - hw_res->mbm_cntr_assign_enabled = true; } r->mon_capable = true;
Rest looks good to me.
Reinette
Babu Moger
reviewerRe: [PATCH v3 3/3] x86,fs/resctrl: Keep mbm_assign_mode in default mode at boot
Hi Reinette,
Quoted 10 lines · Reinette Chatre
Hi Babu,
Quoted 4 lines · Babu Moger
The "
mbm_event" mode assigns hardware MBM counters to RMID/event pairs. It is intended for deployments that need to manage counter assignment on platforms where the number of monitoring groups exceeds the available hardware counters. Hardware counters are scarce, sombm_eventmode suits"Hardware counters are scarce" -> "Hardware counters are scarce on these platforms" (to make it specific to the context and not a general statement)
Sure.
Quoted 29 lines
Quoted 25 lines
workflows that monitor a subset of groups at a time and rotate assignments as needed.
resctrl enables the "
mbm_event" mode by default on hardware that supports it. That breaks the pqos tool [1], which assumes the historical default mode. pqos treats a non-numeric event read as zero bandwidth. It creates 16 or more monitoring groups and uses two counters per group. On a platform with 32 counters per domain, that consumes the pool, so further groups read "Unassigned" and pqos reports 0 MB/s for them:$sudo pqos -m all:0-4 CORE IPC MISSES LLC[KB] MBL[MB/s] MBR[MB/s] 0 0.69 16k 192.0 0.0 0.0 1 0.40 3k 0.0 0.0 0.0 2 0.44 1k 64.0 0.0 0.0 3 0.39 1k 64.0 0.0 0.0 4 0.43 1k 0.0 0.0 0.0Leave
mbm_assign_modein "default" mode during initialization. Default mode uses one counter per monitoring group. On existing AMD platforms that pool is 64 counters and may be larger on newer hardware. The "mbm_event" mode uses two counters per group, so 32 counters cover only 16 groups.Quoted 3 lines
Deployments within that pool receive accurate bandwidth measurements with default mode. The hardware supports 4096 monitoring groups. Beyond that pool, readings may be misleading or "Unavailable", with no
Quoted 8 lines
At this point I am lost because the word "pool" is used to mean three different things. It is used for the number of "ABMC counters" as well as the number of "non-ABMC counters" that exist on the same platform, above it seems to be used for number of RMIDs (number of monitoring groups) also. Could you please add some context at beginning of changelog to introduce the various counts applicable to the description, associate unique terms to these counts at that time, and then use those unique terms in the rest of the changelog?
Sure.
I can name as "ABMC counters pool" and "default active counters pool"
And update the text in terms of the words. RMID pool may not be required and will update the text around that.
Quoted 23 lines
Quoted 23 lines
user-visible indication.
Default mode is still limited once the number of monitoring groups exceeds that counter pool. After hardware re-allocates a counter, a read may return "Unavailable". pqos still treats "Unavailable" as zero. Successive reads can be a count, "Unavailable", then another count. pqos treats those jumps as wraparound and can report an inconsistent rate:
$sudo pqos -m all:0-4 CORE IPC MISSES LLC[KB] MBL[MB/s] MBR[MB/s] 0 0.76 60k 32.0 0.0 0.0 1 0.45 1k 32.0 0.0 0.0 2 1.57 107k 64.0 0.0 17592186044184.9 3 1.62 276k 4928.0 3.2 0.5Users that need stable measurements beyond that pool should use "
mbm_event" mode and rotate assignments as needed. Enable it with:$echo
mbm_event>/sys/fs/resctrl/info/L3_MON/mbm_assign_mode
Quoted 7 lines
Quoted 5 lines
Fixes 0f1576e43adc ("x86/resctrl: Configure mbm_event mode if supported")Link https://github.com/intel/intel-cmt-cat # [1]Signed-off-by Babu Moger <[email protected]>See "Ordering of commit tags" in
Documentation/process/maintainer-tip.rst
Ack.
Quoted 21 lines
Quoted 21 lines
--- Documentation/filesystems/resctrl.rst | 48 ++++++++++++++++++--------- arch/x86/kernel/cpu/resctrl/monitor.c | 1 - 2 files changed, 33 insertions(+), 16 deletions(-)Documentation/filesystems/resctrl.rst +2
-2@@ -354,8 +354,8 @@ with the following files: :: # cat /sys/fs/resctrl/info/L3_MON/mbm_assign_mode - [mbm_event] - default + [default] + mbm_event "mbm_event":
Quoted 13 lines
Documentation/filesystems/resctrl.rst with the following files: · line 368
Quoted 7 lines
Documentation/filesystems/resctrl.rst +3
-0@@ -363,6 +363,9 @@ with the following files: pair and monitor the bandwidth usage as long as it is assigned. The hardware continues to track the assigned counter until it is explicitly unassigned by the user. Each event within a resctrl group can be assigned independently. + mbm_event mode uses one counter per event of one RMID. A monitoring group + can use two counters, one for mbm_total_bytes and one for mbm_local_bytes. + For example, 32 counters can monitor 16 groups.This addition looks redundant to me. "
mbm_eventmode uses one counter per event of one RMID." just repeats earlier "...assign a hardware counter to an RMID, event pair ...", no? The example being added ("A monitoring group can use two counters ...") also looks to be specific to AMD's usage and including platform specific example as part of generic introduction seems unnecessary.
ok. Will remove this text.>
Quoted 7 lines
Quoted 7 lines
Documentation/filesystems/resctrl.rst +0
-0In this mode, a monitoring event can only accumulate data while it is backed by a hardware counter. Use "mbm_L3_assignments" found in each CTRL_MON and MON@@ -376,20 +379,37 @@ with the following files: "mbm_L3_assignments" after switching to "mbm_event" mode for counter assignment states of all monitoring groups.
Quoted 10 lines
Documentation/filesystems/resctrl.rst with the following files: · line 384
Quoted 8 lines
Documentation/filesystems/resctrl.rst +3
-5- The mode is beneficial for AMD platforms that support more CTRL_MON - and MON groups than available hardware counters. By default, this - feature is enabled on AMD platforms with the ABMC (Assignable Bandwidth - Monitoring Counters) capability, ensuring counters remain assigned even - when the corresponding RMID is not actively used by any processor. + It is intended for deployments that need to manage counter assignment on + platforms where the number of monitoring groups exceeds the available + hardware counters. Hardware counters are scarce, so mbm_event mode suits"Hardware counters are scarce" -> "Hardware counters are scarce on these platforms"?
Sure.
Quoted 6 lines
Documentation/filesystems/resctrl.rst with the following files: · line 386
Quoted 3 lines
Documentation/filesystems/resctrl.rst +2
-0+ workflows that monitor a subset of groups at a time and rotate assignments + as needed. The mode is beneficial for AMD platforms that support moreText from "The mode is beneficial for AMD ..." does not seem necessary here since it just duplicates the text added below?
Sure.
Quoted 28 lines
Quoted 28 lines
Documentation/filesystems/resctrl.rst +18
-5+ CTRL_MON and MON groups than available hardware counters. The mbm_event + mode ensures counters remain assigned even when the corresponding RMID is + not actively monitored. "default": In default mode, resctrl assumes there is a hardware counter for each - event within every CTRL_MON and MON group. On AMD platforms, it is - recommended to use the mbm_event mode, if supported, to prevent reset of MBM - events between reads resulting from hardware re-allocating counters. This can - result in misleading values or display "Unavailable" if no counter is assigned - to the event. + event within every CTRL_MON and MON group. This mode is enabled by default. + + On AMD platforms that support more CTRL_MON and MON groups than hardware + counters, hardware dynamically shares a smaller pool of counters among + RMIDs. One counter from this pool is used per monitoring group and counts + every MBM event of that group's RMID. The size of the pool is not + enumerated to software (unlike "num_mbm_cntrs"), and "num_rmids" may be + much larger. For example, 64 counters in this pool monitor 64 groups. + Newer hardware may provide a larger pool. + + While the number of monitoring groups does not exceed that pool, a counter + stays attached to each RMID and readings remain accurate. Creating more + groups than the pool can cause hardware to re-allocate those counters + between successive reads of an event. Bandwidth values may then be + misleading, or a read may return "Unavailable" if no counter is allocated
Quoted 8 lines
Documentation/filesystems/resctrl.rst with the following files: · line 413
Quoted 5 lines
Documentation/filesystems/resctrl.rst +4
-0+ to the RMID. There is no user-visible indication when this begins. + Users who need stable readings beyond that pool should switch to mbm_event + mode, if supported, and assign counters to the groups of interest + (rotating assignments as needed).Above usage of "pool" looks ok since it only uses the term for the "non-ABMC hardware counters" while using the accurate terms for the other counts.
Ack.
Quoted 27 lines
Quoted 27 lines
Documentation/filesystems/resctrl.rst +1
-3* To enable "mbm_event" counter assignment mode: ::@@ -1787,11 +1807,9 @@ View the llc occupancy snapshot:: Examples on working with mbm_assign_mode ======================================== -a. Check if MBM counter assignment mode is supported. +a. Check if MBM counter assignment mode is supported and enabled. :: - # mount -t resctrl resctrl /sys/fs/resctrl/ - # cat /sys/fs/resctrl/info/L3_MON/mbm_assign_mode [mbm_event] defaultarch/x86/kernel/cpu/resctrl/monitor.c +0
-1@@ -472,7 +472,6 @@ int __init rdt_get_l3_mon_config(struct rdt_resource *r)cpuid_count (0x80000020 ,5 , &eax, &ebx, &ecx, &edx);/* cntr_id is 12 bits and can only encode 4096 counters. */ r->mon.num_mbm_cntrs =min ((ebx &GENMASK (15 ,0 )) +1 ,BIT (12 )); - hw_res->mbm_cntr_assign_enabled = true; }
Quoted 4 lines
arch/x86/kernel/cpu/resctrl/monitor.c int __init rdt_get_l3_mon_config(struct rdt_resource *r) · line 477
Quoted 1 line
arch/x86/kernel/cpu/resctrl/monitor.c +0
-0r->mon_capable = true;Rest looks good to me.
Thanks
Babu
Reinette Chatre
maintainerRe: [PATCH v3 3/3] x86,fs/resctrl: Keep mbm_assign_mode in default mode at boot
Hi Babu,
Quoted 2 lines · Babu Moger
Quoted 1 line · Reinette Chatre
On 10/2/26 2:26 PM, Babu Moger wrote:
...
Quoted 37 lines
Quoted 28 lines
Quoted 24 lines
workflows that monitor a subset of groups at a time and rotate assignments as needed.
resctrl enables the "
mbm_event" mode by default on hardware that supports it. That breaks the pqos tool [1], which assumes the historical default mode. pqos treats a non-numeric event read as zero bandwidth. It creates 16 or more monitoring groups and uses two counters per group. On a platform with 32 counters per domain, that consumes the pool, so further groups read "Unassigned" and pqos reports 0 MB/s for them:$sudo pqos -m all:0-4
CORE IPC MISSES LLC[KB] MBL[MB/s] MBR[MB/s] 0 0.69 16k 192.0 0.0 0.0 1 0.40 3k 0.0 0.0 0.0 2 0.44 1k 64.0 0.0 0.0 3 0.39 1k 64.0 0.0 0.0 4 0.43 1k 0.0 0.0 0.0Leave
mbm_assign_modein "default" mode during initialization. Default mode uses one counter per monitoring group. On existing AMD platforms that pool is 64 counters and may be larger on newer hardware. The "mbm_event" mode uses two counters per group, so 32 counters cover only 16 groups.Quoted 3 lines
Deployments within that pool receive accurate bandwidth measurements with default mode. The hardware supports 4096 monitoring groups. Beyond that pool, readings may be misleading or "Unavailable", with no
Quoted 8 lines
At this point I am lost because the word "pool" is used to mean three different things. It is used for the number of "ABMC counters" as well as the number of "non-ABMC counters" that exist on the same platform, above it seems to be used for number of RMIDs (number of monitoring groups) also. Could you please add some context at beginning of changelog to introduce the various counts applicable to the description, associate unique terms to these counts at that time, and then use those unique terms in the rest of the changelog?
Quoted 2 lines
Sure.
I can name as "ABMC counters pool" and "default active counters pool"
hmmm ... "active counter" is new term that, when grepping this series, fs/resctrl/*, and arch/x86/kernel/cpu/resctrl/*, does not appear a single time.
If this new term is needed then please include a definition as part of the changelog context that makes it obvious what it refers to.
Reinette
Re: [PATCH v3 3/3] x86,fs/resctrl: Keep mbm_assign_mode in default mode at boot
Hi Reinette,
Quoted 43 lines · Reinette Chatre
Hi Babu,
Quoted 2 lines · Babu Moger
Quoted 1 line · Reinette Chatre
On 10/2/26 2:26 PM, Babu Moger wrote:
...
Quoted 37 lines
Quoted 36 lines
Quoted 24 lines
workflows that monitor a subset of groups at a time and rotate assignments as needed.
resctrl enables the "
mbm_event" mode by default on hardware that supports it. That breaks the pqos tool [1], which assumes the historical default mode. pqos treats a non-numeric event read as zero bandwidth. It creates 16 or more monitoring groups and uses two counters per group. On a platform with 32 counters per domain, that consumes the pool, so further groups read "Unassigned" and pqos reports 0 MB/s for them:$sudo pqos -m all:0-4
CORE IPC MISSES LLC[KB] MBL[MB/s] MBR[MB/s] 0 0.69 16k 192.0 0.0 0.0 1 0.40 3k 0.0 0.0 0.0 2 0.44 1k 64.0 0.0 0.0 3 0.39 1k 64.0 0.0 0.0 4 0.43 1k 0.0 0.0 0.0Leave
mbm_assign_modein "default" mode during initialization. Default mode uses one counter per monitoring group. On existing AMD platforms that pool is 64 counters and may be larger on newer hardware. The "mbm_event" mode uses two counters per group, so 32 counters cover only 16 groups.Quoted 3 lines
Deployments within that pool receive accurate bandwidth measurements with default mode. The hardware supports 4096 monitoring groups. Beyond that pool, readings may be misleading or "Unavailable", with no
At this point I am lost because the word "pool" is used to mean three different things. It is used for the number of "ABMC counters" as well as the number of "non-ABMC counters" that exist on the same platform, above it seems to be used for number of RMIDs (number of monitoring groups) also. Could you please add some context at beginning of changelog to introduce the various counts applicable to the description, associate unique terms to these counts at that time, and then use those unique terms in the rest of the changelog?
Quoted 9 lines
Quoted 2 lines
Sure.
I can name as "ABMC counters pool" and "default active counters pool"hmmm ... "active counter" is new term that, when grepping this series,
fs/resctrl/*, andarch/x86/kernel/cpu/resctrl/*, does not appear a single time.
If this new term is needed then please include a definition as part of the changelog context that makes it obvious what it refers to.
Sure. Will do.
Thanks
Babu