Allow AET to use PMT/TPMI as loadable modules
Subsystems (4)
Tony Luck
maintainerv4 cover Allow AET to use PMT/TPMI as loadable modules
Using symbol_request() eliminates the AET (Application Energy Telemetry) requirement that INTEL_TELEMETY_PMT and INTEL_TPMI be configured as built-in to the base kernel.
To make this work cleanly, refactor resctrl code so that architecture code is made aware of every mount/unmount operation so that state can be cleaned up on each unmount (instead of the notianal call from resctrl_arch_exit()).
AI-review-of-v3: https://sashiko.dev/#/patchset/20260327230208.18094-1-tony.luck%40intel.com
Changes since v3[1]
1) Switched from symbol_get() to symbol_request() so that required modules will be auto-loaded when needed instead of mounting the resctrl file system without AET events.
2) Add warnings to resctrl_{en,dis}able_mon_event() to ensure they are only used when the resctrl file system is not mounted.
3) Move the newly adding locking around mount/unmount from architecture to file system code. This makes it much simpler, avoiding obtaining a mutex in one function and releasing in another.
4) Reset rdt_resource::mon_capable to false as part of cleanup on unmount or for other mount failures. I have not updated rdt_mon_capable which technically needs to be an integer, incremented on mount, decremented on unmount to keep track of how many monitoring resources are enabled in the current mount cycle.
Tony Luck (7):
platform/x86/intel/pmt: Export PMT enumeration functions as GPL x86/resctrl: Drop setting of event_group::force_off when insufficient
RMIDs
fs/resctrl: Add interface to disable a monitor event fs,x86/resctrl: Add architecture hooks for every mount/unmount x86/resctrl: Resolve PMT and TPMI symbols at runtime x86/resctrl: Delete intel_aet_exit()
x86/resctrl: Downgrade dependency of AET on INTEL_PMT
include/linux/resctrl.h | 13 ++- arch/x86/kernel/cpu/resctrl/internal.h | 10 +- arch/x86/kernel/cpu/resctrl/core.c | 16 +++- arch/x86/kernel/cpu/resctrl/intel_aet.c | 102 ++++++++++++++++++--- drivers/platform/x86/intel/pmt/telemetry.c | 4 +- fs/resctrl/monitor.c | 16 ++++ fs/resctrl/rdtgroup.c | 43 ++++++--- arch/x86/Kconfig | 2 +- 8 files changed, 172 insertions(+), 34 deletions(-)
base-commit: 7aaa8047eafd0bd628065b15757d9b48c5f9c07d
Signature · 1 line
2.53.0
Tony Luck
maintainerPATCH v4 1/7 platform/x86/intel/pmt: Export PMT enumeration functions as GPL
The symbol_get() function requires that objects be GPL licensed.
--- drivers/platform/x86/intel/pmt/telemetry.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
drivers/platform/x86/intel/pmt/telemetry.c +2 -2
@@ -287,13 +287,13 @@ struct pmt_feature_group *intel_pmt_get_regions_by_feature(enum pmt_feature_id ireturn no_free_ptr (feature_group); } -EXPORT_SYMBOL (intel_pmt_get_regions_by_feature); +EXPORT_SYMBOL_GPL (intel_pmt_get_regions_by_feature);void intel_pmt_put_feature_group (struct pmt_feature_group *feature_group) {kref_put (&feature_group->kref, pmt_feature_group_release); } -EXPORT_SYMBOL (intel_pmt_put_feature_group); +EXPORT_SYMBOL_GPL (intel_pmt_put_feature_group);int pmt_telem_read (struct telem_endpoint *ep,u32 id,u64 *data,u32 count) {
Signature · 1 line
2.53.0
Reinette Chatre
maintainerRe: [PATCH v4 1/7] platform/x86/intel/pmt: Export PMT enumeration functions as GPL
Hi Tony,
Quoted 1 line · Tony Luck
The
symbol_get()function requires that objects be GPL licensed.
I do not know if this cryptic note is sufficient for the driver's maintainers. Looking at the dependencies among symbols it does look possible to add more context here that makes it clear that this is the right and obvious thing to do for this driver. Doing so makes inclusion of this change just a formality. If it is not the right and obvious thing to do then I similarly expect a better motivation than some planned usage since any planned usage may cause a debate about whether it should be supported.
Reinette
Re: [PATCH v4 1/7] platform/x86/intel/pmt: Export PMT enumeration functions as GPL
Quoted 13 lines · Reinette Chatre
Hi Tony,
Quoted 1 line · Tony Luck
The
symbol_get()function requires that objects be GPL licensed.I do not know if this cryptic note is sufficient for the driver's maintainers. Looking at the dependencies among symbols it does look possible to add more context here that makes it clear that this is the right and obvious thing to do for this driver. Doing so makes inclusion of this change just a formality. If it is not the right and obvious thing to do then I similarly expect a better motivation than some planned usage since any planned usage may cause a debate about whether it should be supported.
Quoted 1 line
Reinette
Tony,
Something along the lines of this would make it clearer:
"Prepare PMT enumeration interfaces for GPL-only users by converting intel_pmt_get_regions_by_feature() and intel_pmt_put_feature_group() to EXPORT_SYMBOL_GPL(). This enables future users of these interfaces to access them via symbol_get(), which requires GPL-exported symbols."
With that clarification, this looks good to me.
Re: [PATCH v4 1/7] platform/x86/intel/pmt: Export PMT enumeration functions as GPL
In reply to Tony LuckQuoted 1 line · Tony Luck
The
symbol_get()function requires that objects be GPL licensed.
Please don't add new uses of symbol_get, as we've been slowly trying to kill it off. If you have bidirectional dependencies just add an registration module in the middle to resolve it.
Luck, Tony
maintainerRE: [PATCH v4 1/7] platform/x86/intel/pmt: Export PMT enumeration functions as GPL
Quoted 5 lines
Quoted 1 line
The
symbol_get()function requires that objects be GPL licensed.Please don't add new uses of
symbol_get, as we've been slowly trying to kill it off. If you have bidirectional dependencies just add an registration module in the middle to resolve it.
In my case I have "resctrl" (which is built-in, and likely to stay that way) that needs functions from the "pmt_discovery" module to enumerate which event counters are present.
Should I add EXPORT_SYMBOL_GPL(some registration function) to resctrl
and have the pmt_discovery code call that to provide pointers to resctrl?
Thanks
-Tony
P.S.
When killing of some function, it would be nice if you added it to:
Documentation/process/deprecated.rst
to make this obvious to unwary developers.
Re: [PATCH v4 1/7] platform/x86/intel/pmt: Export PMT enumeration functions as GPL
Quoted 6 lines · Tony
Quoted 5 lines
Quoted 1 line
The
symbol_get()function requires that objects be GPL licensed.Please don't add new uses of
symbol_get, as we've been slowly trying to kill it off. If you have bidirectional dependencies just add an registration module in the middle to resolve it.
Quoted 7 lines
In my case I have "resctrl" (which is built-in, and likely to stay that way) that needs functions from the "
pmt_discovery" module to enumerate which event counters are present.Should I add
EXPORT_SYMBOL_GPL(some registration function)to resctrland have the
pmt_discoverycode call that to provide pointers to resctrl?
Yes, if your dependency only goes one-way against the normal flow, that's by far the easiest. Bonus point for bounding that registration on an object fitting the natural scope through OF/fwnode/acpi_dev or whatever fits here.
Tony Luck
maintainerPATCH v4 2/7 x86/resctrl: Drop setting of event_group::force_off when insufficient RMIDs
In reply to Tony LuckSetting the force_off flag does not matter when AET features are only enumerated once (on first mount).
In preparation for enumeration on every mount, drop this because it overrides user request with rdt= boot option to force enable a feature.
--- arch/x86/kernel/cpu/resctrl/intel_aet.c | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-)
arch/x86/kernel/cpu/resctrl/intel_aet.c +1 -3
@@ -214,10 +214,8 @@ static bool all_regions_have_sufficient_rmid(struct event_group *e, struct pmt_fif (!p->regions[i].addr)continue ; tr = &p->regions[i]; -if (tr->num_rmids < e->num_rmid) { - e->force_off = true; +if (tr->num_rmids < e->num_rmid)return false; - } }return true;
Signature · 1 line
2.53.0
Reinette Chatre
maintainerRe: [PATCH v4 2/7] x86/resctrl: Drop setting of event_group::force_off when insufficient RMIDs
Hi Tony,
Quoted 2 lines · Tony Luck
Setting the
force_offflag does not matter when AET features are only enumerated once (on first mount).
Missing motivation here. Consider for example:
Enumerating AET features multiple times is needed because ... ?
Quoted 3 lines
In preparation for enumeration on every mount, drop this because it overrides user request with rdt= boot option to force enable a feature.
Missing answer to:event_group::force_off is intended to reflect both the user and architecture's opinion. This removes the architecture setting entirely. This is safe because ...
Missing:
Change definition of event_group::force_off.
Reinette
Tony Luck
maintainerPATCH v4 3/7 fs/resctrl: Add interface to disable a monitor event
In reply to Tony LuckArchitecture code can ask file system code to enable events. But there is no way to clean up and disable events.
Add resctrl_disable_mon_event(). Adding/removing events is only possible when the file system is not mounted. Add a WARN_ON() to catch misuse.
--- include/linux/resctrl.h | 1 + fs/resctrl/monitor.c | 16 ++++++++++++++++ 2 files changed, 17 insertions(+)
include/linux/resctrl.h +1 -0
@@ -416,6 +416,7 @@ int resctrl_arch_update_domains(struct rdt_resource *r, u32 closid);bool resctrl_enable_mon_event (enum resctrl_event_id eventid,bool any_cpu,unsigned int binary_bits,void *arch_priv); +void resctrl_disable_mon_event (enum resctrl_event_id eventid);bool resctrl_is_mon_event_enabled (enum resctrl_event_id eventid);
fs/resctrl/monitor.c +16 -0
@@ -990,6 +990,8 @@ struct mon_evt mon_event_all[QOS_NUM_EVENTS] = {bool resctrl_enable_mon_event (enum resctrl_event_id eventid,bool any_cpu,unsigned int binary_bits,void *arch_priv) { +if (WARN_ON (resctrl_mounted)) +return false;if (WARN_ON_ONCE (eventid <QOS_FIRST_EVENT || eventid >=QOS_NUM_EVENTS || binary_bits >MAX_BINARY_BITS ))return false;
@@ -1010,6 +1012,20 @@ bool resctrl_enable_mon_event(enum resctrl_event_id eventid, bool any_cpu,return true; } +void resctrl_disable_mon_event (enum resctrl_event_id eventid) +{ +if (WARN_ON (resctrl_mounted)) +return ; +if (WARN_ON_ONCE (eventid <QOS_FIRST_EVENT || eventid >=QOS_NUM_EVENTS )) +return ; +if (!mon_event_all[eventid].enabled) { +pr_warn ("Repeat disable for event %d\n" , eventid); +return ; + } + + mon_event_all[eventid].enabled = false; +} +bool resctrl_is_mon_event_enabled (enum resctrl_event_id eventid) { return eventid >= QOS_FIRST_EVENT && eventid < QOS_NUM_EVENTS &&
Signature · 1 line
2.53.0
Reinette Chatre
maintainerRe: [PATCH v4 3/7] fs/resctrl: Add interface to disable a monitor event
Hi Tony,
Quoted 2 lines · Tony Luck
Architecture code can ask file system code to enable events. But there is no way to clean up and disable events.
Missing why it is required to disable events.
Quoted 2 lines
Add
resctrl_disable_mon_event().
(can be seen from patch)
Quoted 2 lines
Adding/removing events is only
possible when the file system is not mounted. Add aWARN_ON()to
While this is accurate that it should not be possible to enable/disable events while resctrl is mounted, this is *not* the *only* time when it should not be possible. Here it is unfortunately not straight forward since only some events require per-domain state which requires the event to be enabled before any domain comes online, potentially very early in initialization.
To me the addition of this warning adds false security.
Also, consider that resctrl_mounted is protected by rdtgroup_mutex and this addition gives architecture code free access without any protection.
To do this right resctrl may need to add more state to an event but how that may look is unclear since an architecture may require per-domain architectural state for an event while resctrl fs needs none. At this time these scenarios may just fall into the "architecture must do the right thing" category since it has best information on how state is managed for the events as they are enabled/disabled.
Reinette
Luck, Tony
maintainerRe: [PATCH v4 3/7] fs/resctrl: Add interface to disable a monitor event
Hi Reinette,
Quoted 7 lines · Reinette Chatre
Hi Tony,
Quoted 2 lines · Tony Luck
Architecture code can ask file system code to enable events. But there is no way to clean up and disable events.
Missing why it is required to disable events.
Will add rationale.
Quoted 4 lines
Quoted 2 lines
Add
resctrl_disable_mon_event().(can be seen from patch)
Will drop this part of comment.
Quoted 10 lines
Quoted 2 lines
Adding/removing events is only
possible when the file system is not mounted. Add aWARN_ON()toWhile this is accurate that it should not be possible to enable/disable events while resctrl is mounted, this is *not* the *only* time when it should not be possible. Here it is unfortunately not straight forward since only some events require per-domain state which requires the event to be enabled before any domain comes online, potentially very early in initialization.
Quoted 4 lines
To me the addition of this warning adds false security.
Also, consider that
resctrl_mountedis protected byrdtgroup_mutexand this addition gives architecture code free access without any protection.
This is a problem.
Quoted 3 lines
To do this right resctrl may need to add more state to an event but how that may look is unclear since an architecture may require per-domain architectural state for an event while resctrl fs needs none.
The extra state for an event could be a pair of pointers to file system functions to be used by resctrl_{en,disable}_mon_event() to allocate/clean up any state needed by file system code for each event.
But this might lead to a rabbit hole of adding complexity. May someday be useful if we ever make resctrl a loadable module.
Quoted 4 lines
At this
time these scenarios may just fall into the "architecture must do the right thing" category since it has best information on how state is managed for the events as they are enabled/disabled.
Are you suggesting to just drop the check for resctrl_mounted (as both a locking issue, and an incomplete solution)?
Quoted 1 line
Reinette
-Tony
Reinette Chatre
maintainerRe: [PATCH v4 3/7] fs/resctrl: Add interface to disable a monitor event
Hi Tony,
Quoted 1 line · Tony
On Fri, Apr 03, 2026 at 05:03:28PM -0700, Reinette Chatre wrote:
...
Quoted 8 lines
Quoted 5 lines
At this
time these scenarios may just fall into the "architecture must do the right thing" category since it has best information on how state is managed for the events as they are enabled/disabled.Are you suggesting to just drop the check for
resctrl_mounted(as both a locking issue, and an incomplete solution)?
I am indeed suggesting to "drop the check for resctrl_mounted" but instead of "just" doing that I think it worthwhile to add function comments to these two arch helpers in include/linux/resctrl.h that describes what needs to be considered when calling them. That is, describe "architecture must do the right thing" with some documentation about what needs to be considered. Such documentation may help us to start putting some boundaries on how these helpers can/should be used to help guide any future enhancements to make this more robust.
Reinette
Luck, Tony
maintainerRe: [PATCH v4 3/7] fs/resctrl: Add interface to disable a monitor event
Quoted 16 lines · Reinette Chatre
Hi Tony,
Quoted 1 line · Tony
On Fri, Apr 03, 2026 at 05:03:28PM -0700, Reinette Chatre wrote:
...
Quoted 8 lines
Quoted 5 lines
At this
time these scenarios may just fall into the "architecture must do the right thing" category since it has best information on how state is managed for the events as they are enabled/disabled.Are you suggesting to just drop the check for
resctrl_mounted(as both a locking issue, and an incomplete solution)?
Quoted 8 lines
I am indeed suggesting to "drop the check for
resctrl_mounted" but instead of "just" doing that I think it worthwhile to add function comments to these two arch helpers ininclude/linux/resctrl.hthat describes what needs to be considered when calling them. That is, describe "architecture must do the right thing" with some documentation about what needs to be considered. Such documentation may help us to start putting some boundaries on how these helpers can/should be used to help guide any future enhancements to make this more robust.
Something like this:
/*
* For events that require per-domain allocation, this routine must be called
* before CPU hot plug state begins allocating domain structures.
* For other events the requirement is that the file system must not be
* mounted when enabling events.
*/
bool resctrl_enable_mon_event(enum resctrl_event_id eventid, bool any_cpu,
unsigned int binary_bits, void *arch_priv)
...
/*
* This routine must not be called for events that require per-domain allcoation.
* For other events the requirement is that the file system must not be
* mounted when disabling events.
*/
void resctrl_disable_mon_event(enum resctrl_event_id eventid)
Quoted 2 lines
Reinette
Reinette Chatre
maintainerRe: [PATCH v4 3/7] fs/resctrl: Add interface to disable a monitor event
Hi Tony,
Quoted 26 lines · Tony
Quoted 16 lines · Reinette Chatre
Hi Tony,
Quoted 1 line · Tony
On Fri, Apr 03, 2026 at 05:03:28PM -0700, Reinette Chatre wrote:
...
Quoted 8 lines
Quoted 5 lines
At this
time these scenarios may just fall into the "architecture must do the right thing" category since it has best information on how state is managed for the events as they are enabled/disabled.Are you suggesting to just drop the check for
resctrl_mounted(as both a locking issue, and an incomplete solution)?Quoted 8 lines
I am indeed suggesting to "drop the check for
resctrl_mounted" but instead of "just" doing that I think it worthwhile to add function comments to these two arch helpers ininclude/linux/resctrl.hthat describes what needs to be considered when calling them. That is, describe "architecture must do the right thing" with some documentation about what needs to be considered. Such documentation may help us to start putting some boundaries on how these helpers can/should be used to help guide any future enhancements to make this more robust.
Quoted 4 lines
Something like this:
/*
* For events that require per-domain allocation, this routine must be called
Since all events are per-domain and all domains need allocation this is not clear. (nit: it is not necessary to say "this routine" when the context is clearly associated with the function).
I think referring to it as "per-domain state" is more accurate.
Quoted 1 line
* before CPU hot plug state begins allocating domain structures.
Architecture has some flexibility here if a resource is discovered after initialization, like what AET does.
Quoted 1 line
* For other events the requirement is that the file system must not be
"For other events" -> "For all events"?
Quoted 4 lines
* mounted when enabling events.
*/
bool resctrl_enable_mon_event(enumresctrl_event_ideventid, boolany_cpu,
unsigned intbinary_bits, void*arch_priv)
Since resctrl has AET to thank for all the additional parameters and this is another AET enhancement I think it would be appreciated if this be done properly. Consider, for example:
/**
* resctrl_enable_mon_event() - Enable monitoring event
* @eventid: ID of the event
* @any_cpu: True if event data can be read from any CPU.
* @binary_bits:Number of binary places of the fixed-point value expected to
* back a floating point event. Can only be set for floating point
* events.
* @arch_priv: Architecture private data associated with event. Passed back to
* architecture when reading the event via resctrl_arch_rmid_read().
*
* The file system must not be mounted when enabling an event.
*
* Events that require per-domain (architectural and/or filesystem) state must
* be enabled before the domain structures are allocated. For example before
* CPU hotplug callbacks that allocate domain structures are registered. If the
* architecture discovers a resource after initialization it should enable
* events needing per-domain state before any domain structure allocation which
* should be coordinated with the CPU hotplug callbacks.
*
* Return:
* true if event was successfully enabled, false otherwise.
*/
bool resctrl_enable_mon_event(enum resctrl_event_id eventid, bool any_cpu,
unsigned int binary_bits, void *arch_priv)
Quoted 10 lines
...
/*
* This routine must not be called for events that require per-domain allcoation.
* For other events the requirement is that the file system must not be
* mounted when disabling events.
*/
voidresctrl_disable_mon_event(enum resctrl_event_id eventid)
Similar here. Please note the "allcoation"
Reinette
Tony Luck
maintainerPATCH v4 4/7 fs,x86/resctrl: Add architecture hooks for every mount/unmount
In reply to Tony LuckAdd hooks for every mount/unmount of the resctrl file system so that architecture code can allocate on mount and free on unmount.
---
Note this patch disables enumeration of AET monitor events because the new mount/unmount hooks do not call intel_aet_get_events() (which is not ready for the change from "just on first mount" to "called on every mount"). That is resolved in the next patch.
include/linux/resctrl.h | 12 +++++-- arch/x86/kernel/cpu/resctrl/internal.h | 8 +++-- arch/x86/kernel/cpu/resctrl/core.c | 14 +++++++- arch/x86/kernel/cpu/resctrl/intel_aet.c | 13 ++++++++ fs/resctrl/rdtgroup.c | 43 ++++++++++++++++++------- 5 files changed, 74 insertions(+), 16 deletions(-)
include/linux/resctrl.h +10 -2
@@ -518,11 +518,19 @@ void resctrl_online_cpu(unsigned int cpu);void resctrl_offline_cpu (unsigned int cpu);/* -* Architecture hook called at beginning of first file system mount attempt. -* No locks are held. +* Architecture hooks for resctrl mount/unmount. +* Each is called with resctrl_mount_lock held. */ + +/* Called at beginning of each file system mount attempt. */ void resctrl_arch_pre_mount (void ); +/* Called to report success/failure of mount. */ +void resctrl_arch_mount_result (int ret); + +/* Called to report unmount. */ +void resctrl_arch_unmount (void ); +/** * resctrl_arch_rmid_read() - Read the eventid counter corresponding to rmid * for this resource and domain.
arch/x86/kernel/cpu/resctrl/internal.h +6 -2
@@ -234,14 +234,18 @@ void rdt_domain_reconfigure_cdp(struct rdt_resource *r);void resctrl_arch_mbm_cntr_assign_set_one (struct rdt_resource *r);#ifdef CONFIG_X86_CPU_RESCTRL_INTEL_AET -bool intel_aet_get_events (void ); +bool intel_aet_pre_mount (void ); +void intel_aet_mount_result (int ret); +void intel_aet_unmount (void ); void __exit intel_aet_exit(void);int intel_aet_read_event (int domid,u32 rmid,void *arch_priv,u64 *val);void intel_aet_mon_domain_setup (int cpu,int id,struct rdt_resource *r,struct list_head *add_pos);bool intel_handle_aet_option (bool force_off,char *tok);#else -static inline bool intel_aet_get_events (void ) {return false; } +static inline bool intel_aet_pre_mount (void ) {return false; } +static inline void intel_aet_mount_result (int ret) { } +static inline void intel_aet_unmount (void ) { } static inline void __exit intel_aet_exit(void) { }static inline int intel_aet_read_event (int domid,u32 rmid,void *arch_priv,u64 *val) {
arch/x86/kernel/cpu/resctrl/core.c +13 -1
@@ -769,8 +769,10 @@ void resctrl_arch_pre_mount(void)struct rdt_resource *r = &rdt_resources_all[RDT_RESOURCE_PERF_PKG ].r_resctrl;int cpu; -if (!intel_aet_get_events ()) +if (!intel_aet_pre_mount ()) { + r->mon_capable = false;return ; + }/* * Late discovery of telemetry events means the domains for the
@@ -786,6 +788,16 @@ void resctrl_arch_pre_mount(void)cpus_read_unlock (); } +void resctrl_arch_mount_result (int ret) +{ +intel_aet_mount_result (ret); +} + +void resctrl_arch_unmount (void ) +{ +intel_aet_unmount (); +} +enum {RDT_FLAG_CMT ,RDT_FLAG_MBM_TOTAL ,
arch/x86/kernel/cpu/resctrl/intel_aet.c +13 -0
@@ -335,6 +335,19 @@ void __exit intel_aet_exit(void) } } +bool intel_aet_pre_mount (void ) +{ +return false;// Temporary stub +} + +void intel_aet_mount_result (int ret) +{ +} + +void intel_aet_unmount (void ) +{ +} +#define DATA_VALID BIT_ULL(63)#define DATA_BITS GENMASK_ULL(62, 0)
fs/resctrl/rdtgroup.c +32 -11
87 lines not on this page · raw message
Signature · 1 line
2.53.0
Reinette Chatre
maintainerRe: [PATCH v4 4/7] fs,x86/resctrl: Add architecture hooks for every mount/unmount
Hi Tony,
Quoted 2 lines · Tony Luck
Add hooks for every mount/unmount of the resctrl file system so that architecture code can allocate on mount and free on unmount.
Please use the changelog to describe and motivate all the other things that this patch does.
Quoted 8 lines
Signed-off-by Tony Luck <[email protected]>---
Note this patch disables enumeration of AET monitor events because the new mount/unmount hooks do not call
intel_aet_get_events()(which is not ready for the change from "just on first mount" to "called on every mount"). That is resolved in the next patch.
This could be part of the proper changelog.
Could patches be re-ordered to support incremental changes?
Quoted 17 lines
include/linux/resctrl.h | 12 +++++-- arch/x86/kernel/cpu/resctrl/internal.h | 8 +++-- arch/x86/kernel/cpu/resctrl/core.c | 14 +++++++- arch/x86/kernel/cpu/resctrl/intel_aet.c | 13 ++++++++ fs/resctrl/rdtgroup.c | 43 ++++++++++++++++++------- 5 files changed, 74 insertions(+), 16 deletions(-)include/linux/resctrl.h +0
-1@@ -518,11 +518,19 @@ void resctrl_online_cpu(unsigned int cpu);void resctrl_offline_cpu (unsigned int cpu);/* -* Architecture hook called at beginning of first file system mount attempt.
include/linux/resctrl.h void resctrl_online_cpu(unsigned int cpu); · line 529
Quoted 10 lines
include/linux/resctrl.h +6
-1-* No locks are held. +* Architecture hooks for resctrl mount/unmount. +* Each is called with resctrl_mount_lock held. */ + +/* Called at beginning of each file system mount attempt. */ void resctrl_arch_pre_mount (void ); +/* Called to report success/failure of mount. */ +void resctrl_arch_mount_result (int ret);
Why is this needed? Why not just always call resctrl_arch_unmount()?
Quoted 37 lines
include/linux/resctrl.h +4
-0+ +/* Called to report unmount. */ +void resctrl_arch_unmount (void ); +/** * resctrl_arch_rmid_read() - Read the eventid counter corresponding to rmid * for this resource and domain. arch/x86/kernel/cpu/resctrl/internal.h +6
-2@@ -234,14 +234,18 @@ void rdt_domain_reconfigure_cdp(struct rdt_resource *r);void resctrl_arch_mbm_cntr_assign_set_one (struct rdt_resource *r);#ifdef CONFIG_X86_CPU_RESCTRL_INTEL_AET -bool intel_aet_get_events (void ); +bool intel_aet_pre_mount (void ); +void intel_aet_mount_result (int ret); +void intel_aet_unmount (void ); void __exit intel_aet_exit(void);int intel_aet_read_event (int domid,u32 rmid,void *arch_priv,u64 *val);void intel_aet_mon_domain_setup (int cpu,int id,struct rdt_resource *r,struct list_head *add_pos);bool intel_handle_aet_option (bool force_off,char *tok);#else -static inline bool intel_aet_get_events (void ) {return false; } +static inline bool intel_aet_pre_mount (void ) {return false; } +static inline void intel_aet_mount_result (int ret) { } +static inline void intel_aet_unmount (void ) { } static inline void __exit intel_aet_exit(void) { }static inline int intel_aet_read_event (int domid,u32 rmid,void *arch_priv,u64 *val) {arch/x86/kernel/cpu/resctrl/core.c +0
-0
arch/x86/kernel/cpu/resctrl/core.c void resctrl_arch_pre_mount(void) · line 773
Quoted 6 lines
arch/x86/kernel/cpu/resctrl/core.c +2
-1@@ -769,8 +769,10 @@ void resctrl_arch_pre_mount(void)struct rdt_resource *r = &rdt_resources_all[RDT_RESOURCE_PERF_PKG ].r_resctrl;int cpu; -if (!intel_aet_get_events ()) +if (!intel_aet_pre_mount ()) { + r->mon_capable = false;
Why is this needed?
Quoted 5 lines
arch/x86/kernel/cpu/resctrl/core.c +1
-0return ; + }/* * Late discovery of telemetry events means the domains for the
...
Quoted 34 lines
#include "
internal.h"arch/x86/kernel/cpu/resctrl/core.c +7
-9+/* Mutex protecting mount/unmount operations */ +static DEFINE_MUTEX (resctrl_mount_lock); +/* Mutex to protect rdtgroup access. */ DEFINE_MUTEX (rdtgroup_mutex);@@ -2788,17 +2790,8 @@ static int rdt_get_tree(struct fs_context *fc)struct rdt_resource *r;int ret; -DO_ONCE_SLEEPABLE (resctrl_arch_pre_mount); -cpus_read_lock ();mutex_lock (&rdtgroup_mutex); -/* -* resctrl file system can only be mounted once. -*/ -if (resctrl_mounted) { - ret = -EBUSY ; - goto out; - } ret =setup_rmid_lru_list ();if (ret)@@ -2900,6 +2893,30 @@ static int rdt_get_tree(struct fs_context *fc)return ret; } +static int rdt_get_tree_wrapper (struct fs_context *fc) +{ +int ret; +
arch/x86/kernel/cpu/resctrl/core.c static int rdt_get_tree(struct fs_context *fc) · line 2909
Quoted 10 lines
arch/x86/kernel/cpu/resctrl/core.c +10
-0+mutex_lock (&resctrl_mount_lock); + +/* +* resctrl file system can only be mounted once. +*/ +if (resctrl_mounted) { +mutex_unlock (&resctrl_mount_lock); +return -EBUSY ; + } +
This does not look right. Here too is resctrl_mounted accessed without rdtgroup_mutex held. This change implies that resctrl_mounted is now protected by resctrl_mount_lock but resctrl is not changed to respect this throughout resulting in unsafe access of resctrl_mounted.
Does this new resctrl_mount_lock need to be in resctrl fs? It really seems as though the needed synchronization belongs in the architecture. Could this instead be accomplished with a private mutex within the AET code?
arch/x86/kernel/cpu/resctrl/core.c static int rdt_get_tree(struct fs_context *fc) · line 2914
Quoted 5 lines
arch/x86/kernel/cpu/resctrl/core.c +5
-0+resctrl_arch_pre_mount (); + + ret =rdt_get_tree (fc); + +resctrl_arch_mount_result (ret);
Could this instead just call resctrl_arch_unmount() on failure?
Reinette
Luck, Tony
maintainerRe: [PATCH v4 4/7] fs,x86/resctrl: Add architecture hooks for every mount/unmount
Hi Reinette,
Quoted 8 lines · Reinette Chatre
Hi Tony,
Quoted 2 lines · Tony Luck
Add hooks for every mount/unmount of the resctrl file system so that architecture code can allocate on mount and free on unmount.
Please use the changelog to describe and motivate all the other things that this patch does.
OK. I will expand.
Quoted 4 lines
Quoted 4 lines
Signed-off-by Tony Luck <[email protected]>---
Quoted 8 lines
Quoted 4 lines
Note this patch disables enumeration of AET monitor events because the new mount/unmount hooks do not call
intel_aet_get_events()(which is not ready for the change from "just on first mount" to "called on every mount"). That is resolved in the next patch.This could be part of the proper changelog.
Could patches be re-ordered to support incremental changes?
I'll look again because several things have changed since I ordered the series this way. But some bits got overly complicated trying to make AET ready to be called multiple times. If I can't solve elegantly I'll move this into the proper changelog.
Quoted 25 lines
Quoted 25 lines
include/linux/resctrl.h | 12 +++++-- arch/x86/kernel/cpu/resctrl/internal.h | 8 +++-- arch/x86/kernel/cpu/resctrl/core.c | 14 +++++++- arch/x86/kernel/cpu/resctrl/intel_aet.c | 13 ++++++++ fs/resctrl/rdtgroup.c | 43 ++++++++++++++++++------- 5 files changed, 74 insertions(+), 16 deletions(-)include/linux/resctrl.h +4
-2@@ -518,11 +518,19 @@ void resctrl_online_cpu(unsigned int cpu);void resctrl_offline_cpu (unsigned int cpu);/* -* Architecture hook called at beginning of first file system mount attempt. -* No locks are held. +* Architecture hooks for resctrl mount/unmount. +* Each is called with resctrl_mount_lock held. */ + +/* Called at beginning of each file system mount attempt. */ void resctrl_arch_pre_mount (void );
Quoted 4 lines
include/linux/resctrl.h void resctrl_online_cpu(unsigned int cpu); · line 529
Quoted 2 lines
include/linux/resctrl.h +2
-0+/* Called to report success/failure of mount. */ +void resctrl_arch_mount_result (int ret);Why is this needed? Why not just always call
resctrl_arch_unmount()?
You are right. If the mount fails in the resctrl code, resctrl_arch_unmount() can clean things up.
Quoted 40 lines
Quoted 40 lines
include/linux/resctrl.h +4
-0+ +/* Called to report unmount. */ +void resctrl_arch_unmount (void ); +/** * resctrl_arch_rmid_read() - Read the eventid counter corresponding to rmid * for this resource and domain. arch/x86/kernel/cpu/resctrl/internal.h +6
-2@@ -234,14 +234,18 @@ void rdt_domain_reconfigure_cdp(struct rdt_resource *r);void resctrl_arch_mbm_cntr_assign_set_one (struct rdt_resource *r);#ifdef CONFIG_X86_CPU_RESCTRL_INTEL_AET -bool intel_aet_get_events (void ); +bool intel_aet_pre_mount (void ); +void intel_aet_mount_result (int ret); +void intel_aet_unmount (void ); void __exit intel_aet_exit(void);int intel_aet_read_event (int domid,u32 rmid,void *arch_priv,u64 *val);void intel_aet_mon_domain_setup (int cpu,int id,struct rdt_resource *r,struct list_head *add_pos);bool intel_handle_aet_option (bool force_off,char *tok);#else -static inline bool intel_aet_get_events (void ) {return false; } +static inline bool intel_aet_pre_mount (void ) {return false; } +static inline void intel_aet_mount_result (int ret) { } +static inline void intel_aet_unmount (void ) { } static inline void __exit intel_aet_exit(void) { }static inline int intel_aet_read_event (int domid,u32 rmid,void *arch_priv,u64 *val) {arch/x86/kernel/cpu/resctrl/core.c +0
-0@@ -769,8 +769,10 @@ void resctrl_arch_pre_mount(void)struct rdt_resource *r = &rdt_resources_all[RDT_RESOURCE_PERF_PKG ].r_resctrl;int cpu;
Quoted 5 lines
arch/x86/kernel/cpu/resctrl/core.c void resctrl_arch_pre_mount(void) · line 773
Quoted 3 lines
arch/x86/kernel/cpu/resctrl/core.c +2
-1-if (!intel_aet_get_events ()) +if (!intel_aet_pre_mount ()) { + r->mon_capable = false;Why is this needed?
I'll move to the next patch. It's needed for the case when AET works on some mount and then fails on a subsequent one. But that can't happen at this point in this patch series.
Quoted 59 lines
Quoted 5 lines
arch/x86/kernel/cpu/resctrl/core.c +1
-0return ; + }/* * Late discovery of telemetry events means the domains for the ...
Quoted 34 lines
#include "
internal.h"arch/x86/kernel/cpu/resctrl/core.c +7
-9+/* Mutex protecting mount/unmount operations */ +static DEFINE_MUTEX (resctrl_mount_lock); +/* Mutex to protect rdtgroup access. */ DEFINE_MUTEX (rdtgroup_mutex);@@ -2788,17 +2790,8 @@ static int rdt_get_tree(struct fs_context *fc)struct rdt_resource *r;int ret; -DO_ONCE_SLEEPABLE (resctrl_arch_pre_mount); -cpus_read_lock ();mutex_lock (&rdtgroup_mutex); -/* -* resctrl file system can only be mounted once. -*/ -if (resctrl_mounted) { - ret = -EBUSY ; - goto out; - } ret =setup_rmid_lru_list ();if (ret)@@ -2900,6 +2893,30 @@ static int rdt_get_tree(struct fs_context *fc)return ret; } +static int rdt_get_tree_wrapper (struct fs_context *fc) +{ +int ret; +arch/x86/kernel/cpu/resctrl/core.c static int rdt_get_tree(struct fs_context *fc) · line 2909
Quoted 10 lines
arch/x86/kernel/cpu/resctrl/core.c +10
-0+mutex_lock (&resctrl_mount_lock); + +/* +* resctrl file system can only be mounted once. +*/ +if (resctrl_mounted) { +mutex_unlock (&resctrl_mount_lock); +return -EBUSY ; + } +This does not look right. Here too is
resctrl_mountedaccessed withoutrdtgroup_mutexheld. This change implies thatresctrl_mountedis now protected byresctrl_mount_lockbut resctrl is not changed to respect this throughout resulting in unsafe access ofresctrl_mounted.
Quoted 3 lines
Does this new
resctrl_mount_lockneed to be in resctrl fs? It really seems as though the needed synchronization belongs in the architecture. Could this instead be accomplished with a private mutex within the AET code?
If you dig in lore for the v3 of this patch, you'll see I had the mutex in the AET code. But there were some complications.
1) Need to acquire in intel_aet_pre_mount() and release in intel_aet_mount_result() which is legal, but makes code more complex when call chains need to be compared to check that the mutex is being released correctly.
2) The "only mounted once" case meant extra state (AET_PRESENT, which you note in next patch may be redundant) because intel_aet_pre_mount() is called, but needs to do nothing.
Adding resctrl_mount_lock to the file system code made things simpler. The pre-mount code can't be called with rdtgroup_mutex held because it needs to build the domains. That needs cpus_read_lock() + mutex_lock(&domain_list_lock);
I need to add more comments on locking. resctrl_mounted is only modified when both resctrl_mount_lock AND rdtgroup_mutex are held. I believe that makes it safe to read the value of resctrl_mounted with just rdtgroup_mutex held.
Quoted 7 lines
arch/x86/kernel/cpu/resctrl/core.c static int rdt_get_tree(struct fs_context *fc) · line 2914
Quoted 5 lines
arch/x86/kernel/cpu/resctrl/core.c +5
-0+resctrl_arch_pre_mount (); + + ret =rdt_get_tree (fc); + +resctrl_arch_mount_result (ret);Could this instead just call
resctrl_arch_unmount()on failure?
Yes. Thanks for the suggestion.
Quoted 1 line
Reinette
-Tony
Reinette Chatre
maintainerRe: [PATCH v4 4/7] fs,x86/resctrl: Add architecture hooks for every mount/unmount
Hi Tony,
Quoted 19 lines · Tony
Quoted 6 lines · Reinette Chatre
Quoted 2 lines · Tony Luck
Add hooks for every mount/unmount of the resctrl file system so that architecture code can allocate on mount and free on unmount.
Please use the changelog to describe and motivate all the other things that this patch does.
OK. I will expand.
Quoted 9 lines
Quoted 8 lines
Signed-off-by Tony Luck <[email protected]>---
Note this patch disables enumeration of AET monitor events because the new mount/unmount hooks do not call
intel_aet_get_events()(which is not ready for the change from "just on first mount" to "called on every mount"). That is resolved in the next patch.
Quoted 8 lines
Quoted 3 lines
This could be part of the proper changelog.
Could patches be re-ordered to support incremental changes?
I'll look again because several things have changed since I ordered the series this way. But some bits got overly complicated trying to make AET ready to be called multiple times. If I can't solve elegantly I'll move this into the proper changelog.
Please mark patches as RFC when still working out details. When reviewing it helps to know whether something is being submitted for inclusion or not.
...
Quoted 28 lines
Quoted 24 lines
Quoted 8 lines
fs/resctrl/rdtgroup.c +4
-0@@ -2900,6 +2893,30 @@ static int rdt_get_tree(struct fs_context *fc)return ret; } +static int rdt_get_tree_wrapper (struct fs_context *fc) +{ +int ret; +fs/resctrl/rdtgroup.c static int rdt_get_tree(struct fs_context *fc) · line 2909
Quoted 10 lines
fs/resctrl/rdtgroup.c +10
-0+mutex_lock (&resctrl_mount_lock); + +/* +* resctrl file system can only be mounted once. +*/ +if (resctrl_mounted) { +mutex_unlock (&resctrl_mount_lock); +return -EBUSY ; + } +This does not look right. Here too is
resctrl_mountedaccessed withoutrdtgroup_mutexheld. This change implies thatresctrl_mountedis now protected byresctrl_mount_lockbut resctrl is not changed to respect this throughout resulting in unsafe access ofresctrl_mounted.Quoted 3 lines
Does this new
resctrl_mount_lockneed to be in resctrl fs? It really seems as though the needed synchronization belongs in the architecture. Could this instead be accomplished with a private mutex within the AET code?
Quoted 6 lines
If you dig in lore for the v3 of this patch, you'll see I had the mutex in the AET code. But there were some complications.
1) Need to acquire in
intel_aet_pre_mount()and release inintel_aet_mount_result()which is legal, but makes code more complex when call chains need to be compared to check that the mutex is being released correctly.
Why was it needed to hold mutex for so long? I cannot find explanation here or in changelog of v3. I did not remember correctly and considered the AET code to be doing the domain addition. Even so, I do think a mutex internal to the arch code can be used to manage the synchronization. Could you please elaborate why this cannot be done?
Quoted 3 lines
2) The "only mounted once" case meant extra state (
AET_PRESENT, which you note in next patch may be redundant) becauseintel_aet_pre_mount()is called, but needs to do nothing.
Right, I do not see need for extra state. In fact, since it is not clear to me that PMT enumeration will be complete when intel_pmt_get_regions_by_feature() is called it seemed worthwhile to only rely on event_group::pfg - if PMT enumeration was not complete during mount N it may be complete on mount N+1? This creates a poor user interface though since user would need an alternate way to know if AET is supported and then a "remount until it works" approach.
Quoted 2 lines
Adding
resctrl_mount_lockto the file system code made things simpler. The
Adding complications to resctrl fs to make things simpler for x86?
Quoted 2 lines
pre-mount code can't be called with
rdtgroup_mutexheld because it needs to build the domains. That needscpus_read_lock()+mutex_lock(&domain_list_lock);
ack. Can an arch-specific mutex be used instead?
Quoted 3 lines
I need to add more comments on locking.
resctrl_mountedis only modified when bothresctrl_mount_lockANDrdtgroup_mutexare held. I believe that makes it safe to read the value ofresctrl_mountedwith justrdtgroup_mutexheld.
...but not to read it with only resctrl_mount_lock held as in snippet above.
Reinette
Luck, Tony
maintainerRe: [PATCH v4 4/7] fs,x86/resctrl: Add architecture hooks for every mount/unmount
Quoted 26 lines · Reinette Chatre
Hi Tony,
Quoted 23 lines · Tony
Quoted 6 lines · Reinette Chatre
Quoted 2 lines · Tony Luck
Add hooks for every mount/unmount of the resctrl file system so that architecture code can allocate on mount and free on unmount.
Please use the changelog to describe and motivate all the other things that this patch does.
OK. I will expand.
Quoted 12 lines
Quoted 8 lines
Signed-off-by Tony Luck <[email protected]>---
Note this patch disables enumeration of AET monitor events because the new mount/unmount hooks do not call
intel_aet_get_events()(which is not ready for the change from "just on first mount" to "called on every mount"). That is resolved in the next patch.This could be part of the proper changelog.
Could patches be re-ordered to support incremental changes?
Quoted 7 lines
Quoted 4 lines
I'll look again because several things have changed since I ordered the series this way. But some bits got overly complicated trying to make AET ready to be called multiple times. If I can't solve elegantly I'll move this into the proper changelog.
Please mark patches as RFC when still working out details. When reviewing it helps to know whether something is being submitted for inclusion or not.
I thought I was close enough for minor tweaks. I was wrong. Especially so after learning from Christoph that symbol_get() and symbol_request() are quietly being deprecated. I'm going to use Christoph's suggestion of a registration function in the next version.
I have succesfully reordered these two patches to avoid breaking AET enumeration for the next series. The minor caveat this time is that architecture code is first updated to handle multiple mount/unmount cycles, but file system code still does the DO_ONCE_SLEEPABLE() to only call on first mount attempt. So the resctrl_arch_unmount() is defined but not called until the next patch.
It seems nicer to have a "just architecture" patch, followed by a "just file system" patch. But I could move the definition of resctrl_arch_unmount() into the file system patch that first uses it.
Quoted 33 lines
...
Quoted 31 lines
Quoted 24 lines
Quoted 8 lines
fs/resctrl/rdtgroup.c +4
-0@@ -2900,6 +2893,30 @@ static int rdt_get_tree(struct fs_context *fc)return ret; } +static int rdt_get_tree_wrapper (struct fs_context *fc) +{ +int ret; +fs/resctrl/rdtgroup.c static int rdt_get_tree(struct fs_context *fc) · line 2909
Quoted 10 lines
fs/resctrl/rdtgroup.c +10
-0+mutex_lock (&resctrl_mount_lock); + +/* +* resctrl file system can only be mounted once. +*/ +if (resctrl_mounted) { +mutex_unlock (&resctrl_mount_lock); +return -EBUSY ; + } +This does not look right. Here too is
resctrl_mountedaccessed withoutrdtgroup_mutexheld. This change implies thatresctrl_mountedis now protected byresctrl_mount_lockbut resctrl is not changed to respect this throughout resulting in unsafe access ofresctrl_mounted.Quoted 3 lines
Does this new
resctrl_mount_lockneed to be in resctrl fs? It really seems as though the needed synchronization belongs in the architecture. Could this instead be accomplished with a private mutex within the AET code?If you dig in lore for the v3 of this patch, you'll see I had the mutex in the AET code. But there were some complications.
Quoted 8 lines
Quoted 3 lines
1) Need to acquire in
intel_aet_pre_mount()and release inintel_aet_mount_result()which is legal, but makes code more complex when call chains need to be compared to check that the mutex is being released correctly.Why was it needed to hold mutex for so long? I cannot find explanation here or in changelog of v3. I did not remember correctly and considered the AET code to be doing the domain addition. Even so, I do think a mutex internal to the arch code can be used to manage the synchronization. Could you please elaborate why this cannot be done?
I tried to move the locks into architecture code. But main problem is still handling when a user tries to mount an already mounted resctrl file system and gets -EBUSY.
In that case file system calls resctrl_arch_pre_mount() with the file system mounted. You suggested that the AET code could detect and ignore a repeat enumeration by noting that the event_group "(*peg)->pfg" is non-NULL, set by the original enumeration. But that fails in this scenario:
# rmmod pmt_telemetry # mount -t resctrl resctrl /sys/fs/resctrl ... succeeds, but without AET present # modprobe pmt_telemetry # mount -t resctrl resctrl /sys/fs/resctrl ... enumeration success, but now calls resctrl_enable_mon_event() ... with the file system mounted
I think the bast solution for this is to change definition of resctrl_arch_pre_mount() from "called on every mount attempt" to "called only when resctrl is NOT mounted". This is because architecture code cannot tell whether the file system is mounted.
Quoted 11 lines
Quoted 4 lines
2) The "only mounted once" case meant extra state (
AET_PRESENT, which you note in next patch may be redundant) becauseintel_aet_pre_mount()is called, but needs to do nothing.Right, I do not see need for extra state. In fact, since it is not clear to me that PMT enumeration will be complete when
intel_pmt_get_regions_by_feature()is called it seemed worthwhile to only rely onevent_group::pfg- if PMT enumeration was not complete during mount N it may be complete on mount N+1? This creates a poor user interface though since user would need an alternate way to know if AET is supported and then a "remount until it works" approach.
The race remains, and is lost if resctrl is auto-mounted at boot from /etc/fstab.
The user can tell if AET is supported with:
$ grep ^ /sys/class/intel_pmt/*/guid
and checking if any of the RMID based event guids are present on the system.
Delta T for the race is small enough that delaying the mount to some other startup script should be sufficient. Users are likely to have such a script to create the CTRL_MON directories and configure schemata for their workload. So annoying, but easily solved.
Quoted 4 lines
Quoted 2 lines
Adding
resctrl_mount_lockto the file system code made things simpler. TheAdding complications to resctrl fs to make things simpler for x86?
I believe it is necessary, since architecture cannot tell if the file system is mounted.
Quoted 4 lines
Quoted 2 lines
pre-mount code can't be called with
rdtgroup_mutexheld because it needs to build the domains. That needscpus_read_lock()+mutex_lock(&domain_list_lock);ack. Can an arch-specific mutex be used instead?
See above.
Quoted 5 lines
Quoted 3 lines
I need to add more comments on locking.
resctrl_mountedis only modified when bothresctrl_mount_lockANDrdtgroup_mutexare held. I believe that makes it safe to read the value ofresctrl_mountedwith justrdtgroup_mutexheld....but not to read it with only
resctrl_mount_lockheld as in snippet above.
Holding either of resctrl_mount_lock or rdtgroup_mutex makes it safe to read the value of resctrl_mounted as it can only be modified when both mutexes are held.
Quoted 1 line
Reinette
-Tony
Reinette Chatre
maintainerRe: [PATCH v4 4/7] fs,x86/resctrl: Add architecture hooks for every mount/unmount
Hi Tony,
Quoted 47 lines · Tony
Quoted 37 lines · Reinette Chatre
Hi Tony,
Quoted 34 lines · Tony
Quoted 26 lines · Reinette Chatre
Quoted 8 lines · Tony Luck
fs/resctrl/rdtgroup.c +4
-0@@ -2900,6 +2893,30 @@ static int rdt_get_tree(struct fs_context *fc)return ret; } +static int rdt_get_tree_wrapper (struct fs_context *fc) +{ +int ret; +fs/resctrl/rdtgroup.c static int rdt_get_tree(struct fs_context *fc) · line 2909
Quoted 10 lines
fs/resctrl/rdtgroup.c +10
-0+mutex_lock (&resctrl_mount_lock); + +/* +* resctrl file system can only be mounted once. +*/ +if (resctrl_mounted) { +mutex_unlock (&resctrl_mount_lock); +return -EBUSY ; + } +This does not look right. Here too is
resctrl_mountedaccessed withoutrdtgroup_mutexheld. This change implies thatresctrl_mountedis now protected byresctrl_mount_lockbut resctrl is not changed to respect this throughout resulting in unsafe access ofresctrl_mounted.Quoted 3 lines
Does this new
resctrl_mount_lockneed to be in resctrl fs? It really seems as though the needed synchronization belongs in the architecture. Could this instead be accomplished with a private mutex within the AET code?If you dig in lore for the v3 of this patch, you'll see I had the mutex in the AET code. But there were some complications.
Quoted 8 lines
Quoted 3 lines
1) Need to acquire in
intel_aet_pre_mount()and release inintel_aet_mount_result()which is legal, but makes code more complex when call chains need to be compared to check that the mutex is being released correctly.Why was it needed to hold mutex for so long? I cannot find explanation here or in changelog of v3. I did not remember correctly and considered the AET code to be doing the domain addition. Even so, I do think a mutex internal to the arch code can be used to manage the synchronization. Could you please elaborate why this cannot be done?
Quoted 7 lines
I tried to move the locks into architecture code. But main problem is still handling when a user tries to mount an already mounted resctrl file system and gets
-EBUSY.In that case file system calls
resctrl_arch_pre_mount()with the file system mounted. You suggested that the AET code could detect and ignore a repeat enumeration by noting that theevent_group"(*peg)->pfg" is non-NULL, set by
It would be great if resctrl only needs to enumerate once but I do not see how that is possible since there is no clear indication from PMT driver whether (all) its data is ready or not.
Quoted 9 lines
the original enumeration. But that fails in this scenario:
# rmmod pmt_telemetry # mount -t resctrl resctrl /sys/fs/resctrl ... succeeds, but without AET present # modprobe pmt_telemetry # mount -t resctrl resctrl /sys/fs/resctrl ... enumeration success, but now calls resctrl_enable_mon_event() ... with the file system mounted
Thank you for catching this.
Quoted 4 lines
I think the bast solution for this is to change definition of
resctrl_arch_pre_mount()from "called on every mount attempt" to "called only when resctrl is NOT mounted". This is because architecture code cannot tell whether the file system is mounted.
Does this mean the addition of the extra locking in resctrl fs? I am not comfortable with that asymmetrical locking.
I think a problem here is that resctrl_enable_mon_event() gives the architecture code the capability of manipulating internal resctrl fs state directly. This is a consequence of the original design before the arch/fs split. I see the additional resctrl fs locking as an attempt to make it easier for the arch to manipulate fs state but I do not think we should go in that direction, instead I think it is better to improve the arch/fs boundaries here.
To that end, what if resctrl uses its familiar "capable" vs "enable" separation here? That is, the architecture informs resctrl fs whether it is *capable* of a feature and resctrl fs, based on interactions with user space, determines whether the feature is *enabled*?
For a simpler addition resctrl could obtain a new helper, for example, resctrl_capable_mon_event() that only marks an event as "capable". resctrl_enable_mon_event() could remain as the "early" call that marks events as capable
and enabled but may be deprecated. Both of these *have* to be called before any domains are created since per-domain state would now depend on whether the system is "capable" of an event or not. resctrl fs can assume that all "capable" events needs to be enabled and it can mark them so at the beginning of each mount, resctrl will only expose "enabled" events.
There are likely some simplifications on top of current implementation to not make this too invasive.
What do you think?
Quoted 15 lines
Quoted 10 lines
Quoted 3 lines
2) The "only mounted once" case meant extra state (
AET_PRESENT, which you note in next patch may be redundant) becauseintel_aet_pre_mount()is called, but needs to do nothing.Right, I do not see need for extra state. In fact, since it is not clear to me that PMT enumeration will be complete when
intel_pmt_get_regions_by_feature()is called it seemed worthwhile to only rely onevent_group::pfg- if PMT enumeration was not complete during mount N it may be complete on mount N+1? This creates a poor user interface though since user would need an alternate way to know if AET is supported and then a "remount until it works" approach.The race remains, and is lost if resctrl is auto-mounted at boot from
/etc/fstab.The user can tell if AET is supported with:
Quoted 8 lines
$ grep ^ /sys/class/intel_pmt/*/guidand checking if any of the RMID based event guids are present on the system.
Delta T for the race is small enough that delaying the mount to some other startup script should be sufficient. Users are likely to have such a script to create the
CTRL_MONdirectories and configure schemata for their workload. So annoying, but easily solved.
I do not have a clear understanding of how the new implementation with registration function will look to understand the races involved.
Quoted 18 lines
Quoted 3 lines
Quoted 1 line
Adding
resctrl_mount_lockto the file system code made things simpler. TheAdding complications to resctrl fs to make things simpler for x86?
I believe it is necessary, since architecture cannot tell if the file system is mounted.
Quoted 4 lines
Quoted 2 lines
pre-mount code can't be called with
rdtgroup_mutexheld because it needs to build the domains. That needscpus_read_lock()+mutex_lock(&domain_list_lock);ack. Can an arch-specific mutex be used instead?
See above.
Quoted 4 lines
Quoted 3 lines
I need to add more comments on locking.
resctrl_mountedis only modified when bothresctrl_mount_lockANDrdtgroup_mutexare held. I believe that makes it safe to read the value ofresctrl_mountedwith justrdtgroup_mutexheld.
Quoted 5 lines
Quoted 1 line
...but not to read it with only
resctrl_mount_lockheld as in snippet above.Holding either of
resctrl_mount_lockorrdtgroup_mutexmakes it safe to read the value ofresctrl_mountedas it can only be modified when both mutexes are held.
I am concerned about this approach due to it not being symmetrical and how resctrl fs now adds additional locking to accommodate drivers.
Reinette
Luck, Tony
maintainerRe: [PATCH v4 4/7] fs,x86/resctrl: Add architecture hooks for every mount/unmount
Hi Reinette,
Quoted 54 lines · Reinette Chatre
Hi Tony,
Quoted 51 lines · Tony
Quoted 37 lines · Reinette Chatre
Hi Tony,
Quoted 34 lines · Tony
Quoted 26 lines · Reinette Chatre
Quoted 8 lines · Tony Luck
fs/resctrl/rdtgroup.c +4
-0@@ -2900,6 +2893,30 @@ static int rdt_get_tree(struct fs_context *fc)return ret; } +static int rdt_get_tree_wrapper (struct fs_context *fc) +{ +int ret; +fs/resctrl/rdtgroup.c static int rdt_get_tree(struct fs_context *fc) · line 2909
Quoted 10 lines
fs/resctrl/rdtgroup.c +10
-0+mutex_lock (&resctrl_mount_lock); + +/* +* resctrl file system can only be mounted once. +*/ +if (resctrl_mounted) { +mutex_unlock (&resctrl_mount_lock); +return -EBUSY ; + } +This does not look right. Here too is
resctrl_mountedaccessed withoutrdtgroup_mutexheld. This change implies thatresctrl_mountedis now protected byresctrl_mount_lockbut resctrl is not changed to respect this throughout resulting in unsafe access ofresctrl_mounted.Quoted 3 lines
Does this new
resctrl_mount_lockneed to be in resctrl fs? It really seems as though the needed synchronization belongs in the architecture. Could this instead be accomplished with a private mutex within the AET code?If you dig in lore for the v3 of this patch, you'll see I had the mutex in the AET code. But there were some complications.
Quoted 8 lines
Quoted 3 lines
1) Need to acquire in
intel_aet_pre_mount()and release inintel_aet_mount_result()which is legal, but makes code more complex when call chains need to be compared to check that the mutex is being released correctly.Why was it needed to hold mutex for so long? I cannot find explanation here or in changelog of v3. I did not remember correctly and considered the AET code to be doing the domain addition. Even so, I do think a mutex internal to the arch code can be used to manage the synchronization. Could you please elaborate why this cannot be done?
I tried to move the locks into architecture code. But main problem is still handling when a user tries to mount an already mounted resctrl file system and gets
-EBUSY.
Quoted 7 lines
Quoted 3 lines
In that case file system calls
resctrl_arch_pre_mount()with the file system mounted. You suggested that the AET code could detect and ignore a repeat enumeration by noting that theevent_group"(*peg)->pfg" is non-NULL, set byIt would be great if resctrl only needs to enumerate once but I do not see how that is possible since there is no clear indication from PMT driver whether (all) its data is ready or not.
That was the root problem. But allowing INTEL_PMT_TELEMETRY to be a module adds the additional constraint that the module cannot be unloaded while resctrl is mounted. If that happens, the mappings to the MMIO space disappear. The existing intel_pmt_get_regions_by_feature() and intel_pmt_put_feature_group() do not do module_{get,put}() to prevent that. New series under development addresses this.
Quoted 13 lines
Quoted 9 lines
the original enumeration. But that fails in this scenario:
# rmmod pmt_telemetry # mount -t resctrl resctrl /sys/fs/resctrl ... succeeds, but without AET present # modprobe pmt_telemetry # mount -t resctrl resctrl /sys/fs/resctrl ... enumeration success, but now calls resctrl_enable_mon_event() ... with the file system mountedThank you for catching this.
Quoted 1 line
Quoted 6 lines
Quoted 3 lines
I think the bast solution for this is to change definition of
resctrl_arch_pre_mount()from "called on every mount attempt" to "called only when resctrl is NOT mounted". This is because architecture code cannot tell whether the file system is mounted.Does this mean the addition of the extra locking in resctrl fs? I am not comfortable with that asymmetrical locking.
Conceptually the change here is from "architecture must do all enumeration before file system code starts" to "architecture is allowed to make changes when the file system is not mounted".
There are currently several limits to what is safe to do because file system only does a partial cleanup on unmount. But general development direction has been to move initialization code into mount path. For AET things need minimal tweaks today. For some future feature more refactoring from "init" to "mount" might be needed.
Quoted 11 lines
I think a problem here is that
resctrl_enable_mon_event()gives the architecture code the capability of manipulating internal resctrl fs state directly. This is a consequence of the original design before thearch/fssplit. I see the additional resctrl fs locking as an attempt to make it easier for the arch to manipulate fs state but I do not think we should go in that direction, instead I think it is better to improve thearch/fsboundaries here.To that end, what if resctrl uses its familiar "capable" vs "enable" separation here? That is, the architecture informs resctrl fs whether it is *capable* of a feature and resctrl fs, based on interactions with user space, determines whether the feature is *enabled*?
For the AET features user space could look at /sys/class/intel_pmt/*/guid to see which events could be upgraded from "capable" to "enabled". But how can that be conveyed to "resctrl fs"? Adding more mount options would seem to be the only choice. Config files under "info/" are only available after mount. But by then domains have been created and the top level mon_data directory populated.
This also runs into problems if INTEL_PMT_TELEMETRY is unloaded between telling the file system that some set of events are "capable" and the file system asking to enable them.
Quoted 9 lines
For a simpler addition resctrl could obtain a new helper, for example,
resctrl_capable_mon_event()that only marks an event as "capable".resctrl_enable_mon_event()could remain as the "early" call that marks events as capable
and enabled but may be deprecated. Both of these *have* to be called before any domains are created since per-domain state would now depend on whether the system is "capable" of an event or not. resctrl fs can assume that all "capable" events needs to be enabled and it can mark them so at the beginning of each mount, resctrl will only expose "enabled" events.
Quoted 4 lines
There are likely some simplifications on top of current implementation to not make this too invasive.
What do you think?
I think it opens a new can of worms. Maybe the most challenging is for the file system to add a "hold" to the INTEL_PMT_TELEMETRY module when enabling an event.
Quoted 20 lines
Quoted 20 lines
Quoted 11 lines
Quoted 4 lines
2) The "only mounted once" case meant extra state (
AET_PRESENT, which you note in next patch may be redundant) becauseintel_aet_pre_mount()is called, but needs to do nothing.Right, I do not see need for extra state. In fact, since it is not clear to me that PMT enumeration will be complete when
intel_pmt_get_regions_by_feature()is called it seemed worthwhile to only rely onevent_group::pfg- if PMT enumeration was not complete during mount N it may be complete on mount N+1? This creates a poor user interface though since user would need an alternate way to know if AET is supported and then a "remount until it works" approach.The race remains, and is lost if resctrl is auto-mounted at boot from
/etc/fstab.The user can tell if AET is supported with:
$ grep ^ /sys/class/intel_pmt/*/guidand checking if any of the RMID based event guids are present on the system.
Quoted 7 lines
Quoted 4 lines
Delta T for the race is small enough that delaying the mount to some other startup script should be sufficient. Users are likely to have such a script to create the
CTRL_MONdirectories and configure schemata for their workload. So annoying, but easily solved.I do not have a clear understanding of how the new implementation with registration function will look to understand the races involved.
I think I'm ready with v5 of the series. I can post (as "RFC") so you can see, and comment, on the details.
Quoted 18 lines
Quoted 18 lines
Quoted 3 lines
Quoted 1 line
Adding
resctrl_mount_lockto the file system code made things simpler. TheAdding complications to resctrl fs to make things simpler for x86?
I believe it is necessary, since architecture cannot tell if the file system is mounted.
Quoted 4 lines
Quoted 2 lines
pre-mount code can't be called with
rdtgroup_mutexheld because it needs to build the domains. That needscpus_read_lock()+mutex_lock(&domain_list_lock);ack. Can an arch-specific mutex be used instead?
See above.
Quoted 4 lines
Quoted 3 lines
I need to add more comments on locking.
resctrl_mountedis only modified when bothresctrl_mount_lockANDrdtgroup_mutexare held. I believe that makes it safe to read the value ofresctrl_mountedwith justrdtgroup_mutexheld.
Quoted 8 lines
Quoted 5 lines
Quoted 1 line
...but not to read it with only
resctrl_mount_lockheld as in snippet above.Holding either of
resctrl_mount_lockorrdtgroup_mutexmakes it safe to read the value ofresctrl_mountedas it can only be modified when both mutexes are held.I am concerned about this approach due to it not being symmetrical and how resctrl fs now adds additional locking to accommodate drivers.
I'm not sure I see the asymmetry. File system code already calls architecture code with some set of locks held. This just adds a new lock (at the top of the locking hierarchy) that is held across calls to resctrl_arch_pre_mount() and resctrl_arch_unmount().
Quoted 3 lines
Reinette
-Tony
Reinette Chatre
maintainerRe: [PATCH v4 4/7] fs,x86/resctrl: Add architecture hooks for every mount/unmount
Hi Tony,
Quoted 59 lines · Tony
Quoted 50 lines · Reinette Chatre
Quoted 49 lines · Tony
Quoted 35 lines · Reinette Chatre
Quoted 34 lines · Tony
Quoted 26 lines · Reinette Chatre
Quoted 8 lines · Tony Luck
(quoted diff) +4
-0@@ -2900,6 +2893,30 @@ static int rdt_get_tree(struct fs_context *fc) return ret; } +static int rdt_get_tree_wrapper(struct fs_context *fc) +{ + int ret; +Review static int rdt_get_tree(struct fs_context *fc) · line 2909
Quoted 10 lines
(quoted diff) +10
-0+ mutex_lock(&resctrl_mount_lock); + + /* + * resctrl file system can only be mounted once. + */ + if (resctrl_mounted) { + mutex_unlock(&resctrl_mount_lock); + return -EBUSY; + } +This does not look right. Here too is
resctrl_mountedaccessed withoutrdtgroup_mutexheld. This change implies thatresctrl_mountedis now protected byresctrl_mount_lockbut resctrl is not changed to respect this throughout resulting in unsafe access ofresctrl_mounted.Quoted 3 lines
Does this new
resctrl_mount_lockneed to be in resctrl fs? It really seems as though the needed synchronization belongs in the architecture. Could this instead be accomplished with a private mutex within the AET code?If you dig in lore for the v3 of this patch, you'll see I had the mutex in the AET code. But there were some complications.
Quoted 8 lines
Quoted 3 lines
1) Need to acquire in
intel_aet_pre_mount()and release inintel_aet_mount_result()which is legal, but makes code more complex when call chains need to be compared to check that the mutex is being released correctly.Why was it needed to hold mutex for so long? I cannot find explanation here or in changelog of v3. I did not remember correctly and considered the AET code to be doing the domain addition. Even so, I do think a mutex internal to the arch code can be used to manage the synchronization. Could you please elaborate why this cannot be done?
I tried to move the locks into architecture code. But main problem is still handling when a user tries to mount an already mounted resctrl file system and gets
-EBUSY.Quoted 7 lines
Quoted 3 lines
In that case file system calls
resctrl_arch_pre_mount()with the file system mounted. You suggested that the AET code could detect and ignore a repeat enumeration by noting that theevent_group"(*peg)->pfg" is non-NULL, set byIt would be great if resctrl only needs to enumerate once but I do not see how that is possible since there is no clear indication from PMT driver whether (all) its data is ready or not.
Quoted 6 lines
That was the root problem. But allowing
INTEL_PMT_TELEMETRYto be a module adds the additional constraint that the module cannot be unloaded while resctrl is mounted. If that happens, the mappings to the MMIO space disappear. The existingintel_pmt_get_regions_by_feature()andintel_pmt_put_feature_group()do not do module_{get,put}() to prevent that. New series under development addresses this.
ack.
Quoted 18 lines
Quoted 18 lines
Quoted 10 lines
the original enumeration. But that fails in this scenario:
# rmmod pmt_telemetry # mount -t resctrl resctrl /sys/fs/resctrl ... succeeds, but without AET present # modprobe pmt_telemetry # mount -t resctrl resctrl /sys/fs/resctrl ... enumeration success, but now calls resctrl_enable_mon_event() ... with the file system mountedThank you for catching this.
Quoted 4 lines
I think the bast solution for this is to change definition of
resctrl_arch_pre_mount()from "called on every mount attempt" to "called only when resctrl is NOT mounted". This is because architecture code cannot tell whether the file system is mounted.
Quoted 6 lines
Quoted 2 lines
Does this mean the addition of the extra locking in resctrl fs? I am not comfortable with that asymmetrical locking.
Conceptually the change here is from "architecture must do all enumeration before file system code starts" to "architecture is allowed to make changes when the file system is not mounted".
This does not seem to be something that resctrl should enforce in general. Architecture could theoretically make changes any time it wants and stage them for resctrl fs to consume when it is able to do so.
Quoted 5 lines
There are currently several limits to what is safe to do because file system only does a partial cleanup on unmount. But general development direction has been to move initialization code into mount path. For AET things need minimal tweaks today. For some future feature more refactoring from "init" to "mount" might be needed.
Right ... and we need to consider how the architecture and filesystem boundaries are consistent so that this is managed well. Just adding a lock for convenience does not seem appropriate to me.
Quoted 8 lines
Quoted 8 lines
I think a problem here is that
resctrl_enable_mon_event()gives the architecture code the capability of manipulating internal resctrl fs state directly. This is a consequence of the original design before thearch/fssplit. I see the additional resctrl fs locking as an attempt to make it easier for the arch to manipulate fs state but I do not think we should go in that direction, instead I think it is better to improve thearch/fsboundaries here.
Quoted 8 lines
Quoted 4 lines
To that end, what if resctrl uses its familiar "capable" vs "enable" separation here? That is, the architecture informs resctrl fs whether it is *capable* of a feature and resctrl fs, based on interactions with user space, determines whether the feature is *enabled*?
For the AET features user space could look at
/sys/class/intel_pmt/*/guidto see which events could be upgraded from "capable" to "enabled". But how can that be conveyed to "resctrl fs"? Adding more mount options would seem to be
User space should not be involved here and I do not see any need to add new mount options. It is the architecture that marks events as capable if it supports the events.
Quoted 3 lines
the only choice. Config files under "info/" are only available after mount. But by then domains have been created and the top level
mon_datadirectory populated.
Why would this be needed?
Here is how I understand the relationship:
- arch enumerates a new feature at any time and calls resctrl_capable_mon_event() to mark any event discovered during enumeration as "capable".
- arch is responsible for domain creation and is trusted to do so *after* marking any events as "capable", any needed per-domain state is created for the "capable" events.
- when resctrl fs is mounted, before creating any files, resctrl fs automatically sets all "capable" events to "enabled". resctrl fs will create needed files only for "enabled" events.
- if an arch discovers new events after resctrl is mounted then it can still enumerate the events and mark them as "capable" - resctrl fs will pick that up on remount.
- arch can only disable an event as part of the unmount handler, this will clear "capable" as well as "enabled". This can be enforced with a check in the callback where only rdtgroup_mutex should be needed to access resctrl_mounted.
Quoted 4 lines
This also runs into problems if
INTEL_PMT_TELEMETRYis unloaded between telling the file system that some set of events are "capable" and the file system asking to enable them.
This is existing problem and sounds like you are solving this with the module_get() and module_put().
Quoted 10 lines
Quoted 10 lines
For a simpler addition resctrl could obtain a new helper, for example,
resctrl_capable_mon_event()that only marks an event as "capable".resctrl_enable_mon_event()could remain as the "early" call that marks events as capable
and enabled but may be deprecated. Both of these *have* to be called before any domains are created since per-domain state would now depend on whether the system is "capable" of an event or not. resctrl fs can assume that all "capable" events needs to be enabled and it can mark them so at the beginning of each mount, resctrl will only expose "enabled" events.
Quoted 8 lines
Quoted 4 lines
There are likely some simplifications on top of current implementation to not make this too invasive.
What do you think?
I think it opens a new can of worms. Maybe the most challenging is for the file system to add a "hold" to the
INTEL_PMT_TELEMETRYmodule when enabling an event.
This is a different problem that needs to be solved also and it sounds as though you have a solution for that.
Quoted 24 lines
Quoted 24 lines
Quoted 23 lines
Quoted 10 lines
Quoted 3 lines
2) The "only mounted once" case meant extra state (
AET_PRESENT, which you note in next patch may be redundant) becauseintel_aet_pre_mount()is called, but needs to do nothing.Right, I do not see need for extra state. In fact, since it is not clear to me that PMT enumeration will be complete when
intel_pmt_get_regions_by_feature()is called it seemed worthwhile to only rely onevent_group::pfg- if PMT enumeration was not complete during mount N it may be complete on mount N+1? This creates a poor user interface though since user would need an alternate way to know if AET is supported and then a "remount until it works" approach.The race remains, and is lost if resctrl is auto-mounted at boot from
/etc/fstab.The user can tell if AET is supported with:
$ grep ^ /sys/class/intel_pmt/*/guidand checking if any of the RMID based event guids are present on the system.
Delta T for the race is small enough that delaying the mount to some other startup script should be sufficient. Users are likely to have such a script to create the
CTRL_MONdirectories and configure schemata for their workload. So annoying, but easily solved.
Quoted 4 lines
Quoted 2 lines
I do not have a clear understanding of how the new implementation with registration function will look to understand the races involved.
I think I'm ready with v5 of the series. I can post (as "RFC") so you
You consider it ready without completing this discussion? I interpret this to mean that I am wasting my time trying to discuss.
Quoted 1 line
can see, and comment, on the details.
... and it is already posted ... before this discussion is complete ... unless you feel that the hour between your response here and the posting of the new version is sufficient grace for me to respond.
The first four versions of this work were all sent within one week. I tried to collaborate with you on v4 but now you sent v5 without completing the discussion. I cannot collaborate with such rapid attempts at throwing stuff to see what sticks.
Quoted 24 lines
Quoted 24 lines
Quoted 23 lines
Quoted 3 lines
Quoted 1 line
Adding
resctrl_mount_lockto the file system code made things simpler. TheAdding complications to resctrl fs to make things simpler for x86?
I believe it is necessary, since architecture cannot tell if the file system is mounted.
Quoted 4 lines
Quoted 2 lines
pre-mount code can't be called with
rdtgroup_mutexheld because it needs to build the domains. That needscpus_read_lock()+mutex_lock(&domain_list_lock);ack. Can an arch-specific mutex be used instead?
See above.
Quoted 5 lines
Quoted 3 lines
I need to add more comments on locking.
resctrl_mountedis only modified when bothresctrl_mount_lockANDrdtgroup_mutexare held. I believe that makes it safe to read the value ofresctrl_mountedwith justrdtgroup_mutexheld....but not to read it with only
resctrl_mount_lockheld as in snippet above.Holding either of
resctrl_mount_lockorrdtgroup_mutexmakes it safe to read the value ofresctrl_mountedas it can only be modified when both mutexes are held.
Quoted 7 lines
Quoted 2 lines
I am concerned about this approach due to it not being symmetrical and how resctrl fs now adds additional locking to accommodate drivers.
I'm not sure I see the asymmetry. File system code already calls architecture code with some set of locks held. This just adds a new lock (at the top of the locking hierarchy) that is held across calls to
resctrl_arch_pre_mount()andresctrl_arch_unmount().
The asymmetry is in the inconsistency which locks are used to interact with resctrl_mounted ... in this version at least.
Reinette
Luck, Tony
maintainerRE: [PATCH v4 4/7] fs,x86/resctrl: Add architecture hooks for every mount/unmount
Quoted 8 lines
Quoted 4 lines
Quoted 2 lines
I do not have a clear understanding of how the new implementation with registration function will look to understand the races involved.
I think I'm ready with v5 of the series. I can post (as "RFC") so you
You consider it ready without completing this discussion? I interpret this to mean that I am wasting my time trying to discuss.
Quoted 8 lines
Quoted 1 line
can see, and comment, on the details.
... and it is already posted ... before this discussion is complete ... unless you feel that the hour between your response here and the posting of the new version is sufficient grace for me to respond.
The first four versions of this work were all sent within one week. I tried to collaborate with you on v4 but now you sent v5 without completing the discussion. I cannot collaborate with such rapid attempts at throwing stuff to see what sticks.
My intent in posting (as RFC as suggested by you for code that isn't ready for upstream) was simply to give you visibility to the registration mechanism that I'd implemented in response to Christoph's comment that I should not use symbol_get() or symbol_request().
I'm looking at your suggestion for asynchronous update by architecture for use on next mount.
-Tony
Luck, Tony
maintainerRe: [PATCH v4 4/7] fs,x86/resctrl: Add architecture hooks for every mount/unmount
In reply to Reinette ChatreQuoted 100 lines · Reinette Chatre
Hi Tony,
Quoted 59 lines · Tony
Quoted 50 lines · Reinette Chatre
Quoted 49 lines · Tony
Quoted 35 lines · Reinette Chatre
Quoted 34 lines · Tony
Quoted 26 lines · Reinette Chatre
Quoted 8 lines · Tony Luck
(quoted diff) +4
-0@@ -2900,6 +2893,30 @@ static int rdt_get_tree(struct fs_context *fc) return ret; } +static int rdt_get_tree_wrapper(struct fs_context *fc) +{ + int ret; +Review static int rdt_get_tree(struct fs_context *fc) · line 2909
Quoted 10 lines
(quoted diff) +10
-0+ mutex_lock(&resctrl_mount_lock); + + /* + * resctrl file system can only be mounted once. + */ + if (resctrl_mounted) { + mutex_unlock(&resctrl_mount_lock); + return -EBUSY; + } +This does not look right. Here too is
resctrl_mountedaccessed withoutrdtgroup_mutexheld. This change implies thatresctrl_mountedis now protected byresctrl_mount_lockbut resctrl is not changed to respect this throughout resulting in unsafe access ofresctrl_mounted.Quoted 3 lines
Does this new
resctrl_mount_lockneed to be in resctrl fs? It really seems as though the needed synchronization belongs in the architecture. Could this instead be accomplished with a private mutex within the AET code?If you dig in lore for the v3 of this patch, you'll see I had the mutex in the AET code. But there were some complications.
Quoted 8 lines
Quoted 3 lines
1) Need to acquire in
intel_aet_pre_mount()and release inintel_aet_mount_result()which is legal, but makes code more complex when call chains need to be compared to check that the mutex is being released correctly.Why was it needed to hold mutex for so long? I cannot find explanation here or in changelog of v3. I did not remember correctly and considered the AET code to be doing the domain addition. Even so, I do think a mutex internal to the arch code can be used to manage the synchronization. Could you please elaborate why this cannot be done?
I tried to move the locks into architecture code. But main problem is still handling when a user tries to mount an already mounted resctrl file system and gets
-EBUSY.Quoted 7 lines
Quoted 3 lines
In that case file system calls
resctrl_arch_pre_mount()with the file system mounted. You suggested that the AET code could detect and ignore a repeat enumeration by noting that theevent_group"(*peg)->pfg" is non-NULL, set byIt would be great if resctrl only needs to enumerate once but I do not see how that is possible since there is no clear indication from PMT driver whether (all) its data is ready or not.
Quoted 6 lines
That was the root problem. But allowing
INTEL_PMT_TELEMETRYto be a module adds the additional constraint that the module cannot be unloaded while resctrl is mounted. If that happens, the mappings to the MMIO space disappear. The existingintel_pmt_get_regions_by_feature()andintel_pmt_put_feature_group()do not do module_{get,put}() to prevent that. New series under development addresses this.ack.
Quoted 18 lines
Quoted 18 lines
Quoted 10 lines
the original enumeration. But that fails in this scenario:
# rmmod pmt_telemetry # mount -t resctrl resctrl /sys/fs/resctrl ... succeeds, but without AET present # modprobe pmt_telemetry # mount -t resctrl resctrl /sys/fs/resctrl ... enumeration success, but now calls resctrl_enable_mon_event() ... with the file system mountedThank you for catching this.
Quoted 4 lines
I think the bast solution for this is to change definition of
resctrl_arch_pre_mount()from "called on every mount attempt" to "called only when resctrl is NOT mounted". This is because architecture code cannot tell whether the file system is mounted.Quoted 6 lines
Quoted 2 lines
Does this mean the addition of the extra locking in resctrl fs? I am not comfortable with that asymmetrical locking.
Conceptually the change here is from "architecture must do all enumeration before file system code starts" to "architecture is allowed to make changes when the file system is not mounted".
This does not seem to be something that resctrl should enforce in general. Architecture could theoretically make changes any time it wants and stage them for resctrl fs to consume when it is able to do so.
Quoted 8 lines
Quoted 5 lines
There are currently several limits to what is safe to do because file system only does a partial cleanup on unmount. But general development direction has been to move initialization code into mount path. For AET things need minimal tweaks today. For some future feature more refactoring from "init" to "mount" might be needed.
Right ... and we need to consider how the architecture and filesystem boundaries are consistent so that this is managed well. Just adding a lock for convenience does not seem appropriate to me.
I think it is more than convenience. Allowing file system and architecture to run asynchronously at all times means more race conditions need to be considered and addressed.
Quoted 24 lines
Quoted 8 lines
Quoted 8 lines
I think a problem here is that
resctrl_enable_mon_event()gives the architecture code the capability of manipulating internal resctrl fs state directly. This is a consequence of the original design before thearch/fssplit. I see the additional resctrl fs locking as an attempt to make it easier for the arch to manipulate fs state but I do not think we should go in that direction, instead I think it is better to improve thearch/fsboundaries here.Quoted 8 lines
Quoted 4 lines
To that end, what if resctrl uses its familiar "capable" vs "enable" separation here? That is, the architecture informs resctrl fs whether it is *capable* of a feature and resctrl fs, based on interactions with user space, determines whether the feature is *enabled*?
For the AET features user space could look at
/sys/class/intel_pmt/*/guidto see which events could be upgraded from "capable" to "enabled". But how can that be conveyed to "resctrl fs"? Adding more mount options would seem to beUser space should not be involved here and I do not see any need to add new mount options. It is the architecture that marks events as capable if it supports the events.
Quoted 3 lines
the only choice. Config files under "info/" are only available after mount. But by then domains have been created and the top level
mon_datadirectory populated.
Quoted 6 lines
Why would this be needed?
Here is how I understand the relationship:
- arch enumerates a new feature at any time and calls
resctrl_capable_mon_event()to mark any event discovered during enumeration as "capable".
This is possible.
Quoted 3 lines
- arch is responsible for domain creation and is trusted to do so *after* marking any events as "capable", any needed per-domain state is created for the "capable" events.
This creates a new problem. Domains for monitor resources are created for resources where rdt_resource::mon_capable is true. But this is visible by file system code. Maybe the same "capable" -> "enabled" trick can be used to separate architecture creation of domains from file system use?
Quoted 3 lines
- when resctrl fs is mounted, before creating any files, resctrl fs automatically sets all "capable" events to "enabled". resctrl fs will create needed files only for "enabled" events.
This has a new race. Instead of a binary problem of AET either being fully enabled or completely missing. User may see some subset of events if architecture was in the middle of looping over events marking them capable when file system is mounted.
Quoted 7 lines
- if an arch discovers new events after resctrl is mounted then it can still enumerate the events and mark them as "capable" - resctrl fs will pick that up on remount.
- arch can only disable an event as part of the unmount handler, this will clear "capable" as well as "enabled". This can be enforced with a check in the callback where only
rdtgroup_mutexshould be needed to accessresctrl_mounted.
Seems OK. But to make sure that events are accessible, architecture will now have to "hold" the pmt_telemetry module regardless of whether resctrl file system is mounted.
Since architecture doesn't have a reliable indication of when the file system is mounted (or if it ever will be mounted) the hold prevents pmt_telemtry from ever being unloaded.
Quoted 7 lines
Quoted 4 lines
This also runs into problems if
INTEL_PMT_TELEMETRYis unloaded between telling the file system that some set of events are "capable" and the file system asking to enable them.This is existing problem and sounds like you are solving this with the
module_get()andmodule_put().
My solution only works if architecture is called in initial part of file system mount, and trailing part of file system unmount. As detailed before these calls must be without rdtgroup_mutex held because of the need to hold cpus_read_lock() and domain_list_lock around domain creation and removal.
Quoted 46 lines
Quoted 10 lines
Quoted 10 lines
For a simpler addition resctrl could obtain a new helper, for example,
resctrl_capable_mon_event()that only marks an event as "capable".resctrl_enable_mon_event()could remain as the "early" call that marks events as capable
and enabled but may be deprecated. Both of these *have* to be called before any domains are created since per-domain state would now depend on whether the system is "capable" of an event or not. resctrl fs can assume that all "capable" events needs to be enabled and it can mark them so at the beginning of each mount, resctrl will only expose "enabled" events.Quoted 8 lines
Quoted 4 lines
There are likely some simplifications on top of current implementation to not make this too invasive.
What do you think?
I think it opens a new can of worms. Maybe the most challenging is for the file system to add a "hold" to the
INTEL_PMT_TELEMETRYmodule when enabling an event.This is a different problem that needs to be solved also and it sounds as though you have a solution for that.
Quoted 24 lines
Quoted 24 lines
Quoted 23 lines
Quoted 10 lines
Quoted 3 lines
2) The "only mounted once" case meant extra state (
AET_PRESENT, which you note in next patch may be redundant) becauseintel_aet_pre_mount()is called, but needs to do nothing.Right, I do not see need for extra state. In fact, since it is not clear to me that PMT enumeration will be complete when
intel_pmt_get_regions_by_feature()is called it seemed worthwhile to only rely onevent_group::pfg- if PMT enumeration was not complete during mount N it may be complete on mount N+1? This creates a poor user interface though since user would need an alternate way to know if AET is supported and then a "remount until it works" approach.The race remains, and is lost if resctrl is auto-mounted at boot from
/etc/fstab.The user can tell if AET is supported with:
$ grep ^ /sys/class/intel_pmt/*/guidand checking if any of the RMID based event guids are present on the system.
Delta T for the race is small enough that delaying the mount to some other startup script should be sufficient. Users are likely to have such a script to create the
CTRL_MONdirectories and configure schemata for their workload. So annoying, but easily solved.
Quoted 7 lines
Quoted 4 lines
Quoted 2 lines
I do not have a clear understanding of how the new implementation with registration function will look to understand the races involved.
I think I'm ready with v5 of the series. I can post (as "RFC") so you
You consider it ready without completing this discussion? I interpret this to mean that I am wasting my time trying to discuss.
My apologies again for this.
Quoted 36 lines
Quoted 1 line
can see, and comment, on the details.
... and it is already posted ... before this discussion is complete ... unless you feel that the hour between your response here and the posting of the new version is sufficient grace for me to respond.
The first four versions of this work were all sent within one week. I tried to collaborate with you on v4 but now you sent v5 without completing the discussion. I cannot collaborate with such rapid attempts at throwing stuff to see what sticks.Quoted 27 lines
Quoted 26 lines
Quoted 18 lines
Quoted 3 lines
Quoted 1 line
Adding
resctrl_mount_lockto the file system code made things simpler. TheAdding complications to resctrl fs to make things simpler for x86?
I believe it is necessary, since architecture cannot tell if the file system is mounted.
Quoted 4 lines
Quoted 2 lines
pre-mount code can't be called with
rdtgroup_mutexheld because it needs to build the domains. That needscpus_read_lock()+mutex_lock(&domain_list_lock);ack. Can an arch-specific mutex be used instead?
See above.
Quoted 4 lines
Quoted 3 lines
I need to add more comments on locking.
resctrl_mountedis only modified when bothresctrl_mount_lockANDrdtgroup_mutexare held. I believe that makes it safe to read the value ofresctrl_mountedwith justrdtgroup_mutexheld.Quoted 5 lines
Quoted 1 line
...but not to read it with only
resctrl_mount_lockheld as in snippet above.Holding either of
resctrl_mount_lockorrdtgroup_mutexmakes it safe to read the value ofresctrl_mountedas it can only be modified when both mutexes are held.I am concerned about this approach due to it not being symmetrical and how resctrl fs now adds additional locking to accommodate drivers.
Quoted 7 lines
Quoted 4 lines
I'm not sure I see the asymmetry. File system code already calls architecture code with some set of locks held. This just adds a new lock (at the top of the locking hierarchy) that is held across calls to
resctrl_arch_pre_mount()andresctrl_arch_unmount().The asymmetry is in the inconsistency which locks are used to interact with
resctrl_mounted... in this version at least.
"Double/split locking" is used at various places in the Linux kernel for data structures that need to be accessed read-only in separate contexts. Both locks must be held when updating the data structure. AI (Gemini) listed these high profile instances:
struct task_struct: protected by task->pi_lock and the rq->lock
VFS / dcache (Rename Operations): must hold i_rwsem for source and target of the rename
Netfilter / Connection Tracking: Modifying connection state requires a specific bucket lock and the status-change lock.
Quoted 2 lines
Reinette
-Tony
Reinette Chatre
maintainerRe: [PATCH v4 4/7] fs,x86/resctrl: Add architecture hooks for every mount/unmount
Hi Tony,
Quoted 7 lines · Tony
Quoted 4 lines · Reinette Chatre
Here is how I understand the relationship:
- arch enumerates a new feature at any time and calls
resctrl_capable_mon_event()to mark any event discovered during enumeration as "capable".
Quoted 8 lines
This is possible.
Quoted 3 lines
- arch is responsible for domain creation and is trusted to do so *after* marking any events as "capable", any needed per-domain state is created for the "capable" events.
This creates a new problem. Domains for monitor resources are created for resources where
rdt_resource::mon_capableis true. But this is
Right, and both (specifically, setting of rdt_resource::mon_capable and domain creation) are done by architecture.
Quoted 1 line
visible by file system code. Maybe the same "capable" -> "enabled"
Right, this is required to be visible to file system because the filesystem needs access to the domains as well as rdt_resource::mon_capable.
Quoted 2 lines
trick can be used to separate architecture creation of domains from file system use?
Architecture creation of domains is already separated from file system use of domains via cpus_read_lock(). This is already required for synchronization between architecture and resctrl fs and used by AET implementation that creates domains with cpus_read_lock() held. The domain list is an RCU list and can thus also be accessed from RCU read-side critical section but no such usage exists for x86 at this time. All accesses of domain list from file system is done with cpus_read_lock() held.
arch thus already controls when the domains it creates are visible to resctrl fs.
Quoted 8 lines
Quoted 3 lines
- when resctrl fs is mounted, before creating any files, resctrl fs automatically sets all "capable" events to "enabled". resctrl fs will create needed files only for "enabled" events.
This has a new race. Instead of a binary problem of AET either being fully enabled or completely missing. User may see some subset of events if architecture was in the middle of looping over events marking them capable when file system is mounted.
AET currently marks events as enabled without any locks held. It does so before setting rdt_resource::mon_capable though and will only set rdt_resource::mon_capable with cpus_read_lock() held, which rdt_get_tree requires to do anything.
One scenario to consider is if it takes a few attempts to enumerate the event groups which means that rdt_resource::mon_capable may already be set when a new event is enabled/marked as capable. This may be a problem, but would moving the call to mark event as capable to be with all the other changes to state used by resctrl fs address this? Specifically, by moving the call to mark event as capable to be with the snippet that sets rdt_resource::mon_capable and creates the domains (again).
With this the behavior of "fully enabled" or "completely missing" can be maintained? Although, there seems to be some other state of "only some event group(s)" enabled as opposed to "fully enabled", but I believe there will be no partial enabling of an event group.
Quoted 5 lines
Quoted 5 lines
- if an arch discovers new events after resctrl is mounted then it can still enumerate the events and mark them as "capable" - resctrl fs will pick that up on remount.
Quoted 7 lines
Quoted 3 lines
- arch can only disable an event as part of the unmount handler, this will clear "capable" as well as "enabled". This can be enforced with a check in the callback where only
rdtgroup_mutexshould be needed to accessresctrl_mounted.Seems OK. But to make sure that events are accessible, architecture will now have to "hold" the
pmt_telemetrymodule regardless of whether resctrl file system is mounted.
Could you please elaborate why this is required? if I understand correctly the "hold" on the pmt_telemetry module will be done by itself between the intel_pmt_get_regions_by_feature() and intel_pmt_put_feature_group() calls.
Quoted 4 lines
Since architecture doesn't have a reliable indication of when the file system is mounted (or if it ever will be mounted) the hold prevents
pmt_telemtryfrom ever being unloaded.
Why does architecture need an indication of when the file system is mounted?
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines
This also runs into problems if
INTEL_PMT_TELEMETRYis unloaded between telling the file system that some set of events are "capable" and the file system asking to enable them.
Quoted 8 lines
Quoted 2 lines
This is existing problem and sounds like you are solving this with the
module_get()andmodule_put().My solution only works if architecture is called in initial part of file system mount, and trailing part of file system unmount. As detailed before these calls must be without
rdtgroup_mutexheld because of the need to holdcpus_read_lock()anddomain_list_lockaround domain creation and removal.
Understood.
...
Quoted 5 lines
"Double/split locking" is used at various places in the Linux kernel for data structures that need to be accessed read-only in separate contexts. Both locks must be held when updating the data structure. AI (Gemini) listed these high profile instances:
Quoted 7 lines
struct
task_struct: protected bytask->pi_lockand therq->lockVFS / dcache (Rename Operations): must hold
i_rwsemfor source and target of the renameNetfilter / Connection Tracking: Modifying connection state requires a specific bucket lock and the status-change lock.
AI did not need to look outside resctrl where domain list update is protected by both cpus_read_lock() and domain_list_lock but reads just by cpus_read_lock() or RCU. I do not see why adding such additional complexity is needed here.
Reinette
Luck, Tony
maintainerRE: [PATCH v4 4/7] fs,x86/resctrl: Add architecture hooks for every mount/unmount
Quoted 7 lines
Quoted 3 lines
Seems OK. But to make sure that events are accessible, architecture will now have to "hold" the
pmt_telemetrymodule regardless of whether resctrl file system is mounted.Could you please elaborate why this is required? if I understand correctly the "hold" on the
pmt_telemetrymodule will be done by itself between theintel_pmt_get_regions_by_feature()andintel_pmt_put_feature_group()calls.
If architecture code runs completely asynchronously to resctrl mount/unmount when would it be safe to call intel_pmt_put_feature_group()?
-Tony
Reinette Chatre
maintainerRe: [PATCH v4 4/7] fs,x86/resctrl: Add architecture hooks for every mount/unmount
Hi Tony,
Quoted 9 lines · Tony
Quoted 7 lines
Quoted 3 lines
Seems OK. But to make sure that events are accessible, architecture will now have to "hold" the
pmt_telemetrymodule regardless of whether resctrl file system is mounted.Could you please elaborate why this is required? if I understand correctly the "hold" on the
pmt_telemetrymodule will be done by itself between theintel_pmt_get_regions_by_feature()andintel_pmt_put_feature_group()calls.If architecture code runs completely asynchronously to resctrl mount/unmount
Since there is shared state that is managed via existing locks I do not see architecture running completely asynchronously to resctrl mount/unmount.
Quoted 1 line
when would it be safe to call
intel_pmt_put_feature_group()?
I think I am missing something here since I have been assuming all along that intel_pmt_put_feature_group() is and will continue to be called from the new unmount handler.
Reinette
Luck, Tony
maintainerRE: [PATCH v4 4/7] fs,x86/resctrl: Add architecture hooks for every mount/unmount
Quoted 3 lines
I think I am missing something here since I have been assuming all along that
intel_pmt_put_feature_group()is and will continue to be called from the new unmount handler.
My problem is not with unmount. It is with resctrl_arch_pre_mount().
With no locks held, I can't tell whether this is a genuine attempt to mount, or a doomed-to-fail attempt to remount. There could be arbitrarily many parallel calls to resctrl_arch_pre_mount() leading to races that I find hard to handle.
In my earlier attempts to deal with this I acquired a lock in resctrl_arch_pre_mount() and held until the resctrl_arch_mount_result() call from filesystem.
This seemed complex, and so I moved the locking to the file system, and here we are today.
-Tony
Reinette Chatre
maintainerRe: [PATCH v4 4/7] fs,x86/resctrl: Add architecture hooks for every mount/unmount
Hi Tony,
Quoted 11 lines · Tony
Quoted 3 lines
I think I am missing something here since I have been assuming all along that
intel_pmt_put_feature_group()is and will continue to be called from the new unmount handler.My problem is not with unmount. It is with
resctrl_arch_pre_mount().With no locks held, I can't tell whether this is a genuine attempt to mount, or a doomed-to-fail attempt to remount. There could be arbitrarily many parallel calls to
resctrl_arch_pre_mount()leading to races that I find hard to handle.
Quoted 5 lines
In my earlier attempts to deal with this I acquired a lock in
resctrl_arch_pre_mount()and held until theresctrl_arch_mount_result()call from filesystem.This seemed complex, and so I moved the locking to the file system, and here we are today.
I am not able to decipher the issues you are facing since your response here deleted my responses to your questions on how I think this can be handled. If you find that my proposal to do this without an additional lock has flaws then please do point them out instead of ignoring them.
Reinette
Luck, Tony
maintainerRe: [PATCH v4 4/7] fs,x86/resctrl: Add architecture hooks for every mount/unmount
In reply to Reinette ChatreQuoted 7 lines · Reinette Chatre
Hi Tony,
Quoted 4 lines · Tony
Quoted 4 lines
Quoted 3 lines
Seems OK. But to make sure that events are accessible, architecture will now have to "hold" the
pmt_telemetrymodule regardless of whether resctrl file system is mounted.
Quoted 8 lines
Quoted 5 lines
Quoted 3 lines
Could you please elaborate why this is required? if I understand correctly the "hold" on the
pmt_telemetrymodule will be done by itself between theintel_pmt_get_regions_by_feature()andintel_pmt_put_feature_group()calls.If architecture code runs completely asynchronously to resctrl mount/unmount
Since there is shared state that is managed via existing locks I do not see architecture running completely asynchronously to resctrl mount/unmount.
I think I now have a solution without any new locks in the file system layer (just a call to resctrl_arch_pre_mount() on every mount and a call to resctrl_arch_unmount() if the mount fails, and on unmount).
Testing in progress. Cleanup to follow before posting.
Quoted 6 lines
Quoted 2 lines
when would it be safe to call
intel_pmt_put_feature_group()?I think I am missing something here since I have been assuming all along that
intel_pmt_put_feature_group()is and will continue to be called from the new unmount handler.
Correct.
Quoted 2 lines
Reinette
I'm looking at a more aggressive change to the Kconfig options.
Linus complained[1] that the biggest pain point for new people is getting the kernel configured. Extra CONFIG options was his prime focus.
So I'm looking at removing CONFIG_X86_CPU_RESCTRL_INTEL_AET completely and just making intel_aet.c included with CONFIG_X86_CPU_RESCTRL.
It does mean the resctrl will depend on X86_64, killing 32-bit support. That was discussed before and decision deferred. But there's other 64-bit MMIO stuff in the pipeline that means we have to make this change sooner or later.
Thoughts on unconditional inclusion and on removing 32-bit support?
-Tony
[1] https://lore.kernel.org/all/CAHk-=whigg3hvOy7c1j1MXFy6o6CHp0g4Tc3Y-MAk+XDssHU0A@mail.gmail.com
Reinette Chatre
maintainerRe: [PATCH v4 4/7] fs,x86/resctrl: Add architecture hooks for every mount/unmount
Hi Tony,
Quoted 14 lines · Tony
Quoted 13 lines · Reinette Chatre
Hi Tony,
Quoted 9 lines · Tony
Quoted 7 lines
Quoted 3 lines
Seems OK. But to make sure that events are accessible, architecture will now have to "hold" the
pmt_telemetrymodule regardless of whether resctrl file system is mounted.Could you please elaborate why this is required? if I understand correctly the "hold" on the
pmt_telemetrymodule will be done by itself between theintel_pmt_get_regions_by_feature()andintel_pmt_put_feature_group()calls.If architecture code runs completely asynchronously to resctrl mount/unmount
Quoted 8 lines
Quoted 2 lines
Since there is shared state that is managed via existing locks I do not see architecture running completely asynchronously to resctrl mount/unmount.
I think I now have a solution without any new locks in the file system layer (just a call to
resctrl_arch_pre_mount()on every mount and a call toresctrl_arch_unmount()if the mount fails, and on unmount).Testing in progress. Cleanup to follow before posting.
Thank you.
Quoted 7 lines
Quoted 6 lines
Quoted 2 lines
when would it be safe to call
intel_pmt_put_feature_group()?I think I am missing something here since I have been assuming all along that
intel_pmt_put_feature_group()is and will continue to be called from the new unmount handler.
Quoted 8 lines
Correct.
Quoted 2 lines
Reinette
I'm looking at a more aggressive change to the
Kconfigoptions.Linus complained[1] that the biggest pain point for new people is getting the kernel configured. Extra CONFIG options was his prime focus.
Thank you for pointing this out.
Quoted 3 lines
So I'm looking at removing
CONFIG_X86_CPU_RESCTRL_INTEL_AETcompletely and just makingintel_aet.cincluded withCONFIG_X86_CPU_RESCTRL.
Sounds reasonable but how the final dependencies would look like is not obvious to me. Consider for example this removal happening with today's Kconfig that may force a kernel that will run on AMD/Hygon to be built with INTEL_PMT_TELEMETRY/INTEL_TPMI. Something like this does not look right.
Quoted 5 lines
It does mean the resctrl will depend on
X86_64, killing 32-bit support. That was discussed before and decision deferred. But there's other 64-bit MMIO stuff in the pipeline that means we have to make this change sooner or later.Thoughts on unconditional inclusion and on removing 32-bit support?
I believe we already have a clear answer on 32-bit support for resctrl on *x86* from Thomas https://lore.kernel.org/lkml/87jzm3jckw.ffs@tglx/ and Babu https://lore.kernel.org/lkml/[email protected]/
Whether this can be extended to resctrl fs is not obvious to me but I think this 64-bit requirement is indeed limited to x86 here?
About the unconditional inclusion I would just like to ensure that the resulting dependencies are reasonable for all x86.
Reinette
Luck, Tony
maintainerRe: [PATCH v4 4/7] fs,x86/resctrl: Add architecture hooks for every mount/unmount
Quoted 59 lines · Reinette Chatre
Hi Tony,
Quoted 14 lines · Tony
Quoted 13 lines · Reinette Chatre
Hi Tony,
Quoted 9 lines · Tony
Quoted 7 lines
Quoted 3 lines
Seems OK. But to make sure that events are accessible, architecture will now have to "hold" the
pmt_telemetrymodule regardless of whether resctrl file system is mounted.Could you please elaborate why this is required? if I understand correctly the "hold" on the
pmt_telemetrymodule will be done by itself between theintel_pmt_get_regions_by_feature()andintel_pmt_put_feature_group()calls.If architecture code runs completely asynchronously to resctrl mount/unmount
Quoted 8 lines
Quoted 2 lines
Since there is shared state that is managed via existing locks I do not see architecture running completely asynchronously to resctrl mount/unmount.
I think I now have a solution without any new locks in the file system layer (just a call to
resctrl_arch_pre_mount()on every mount and a call toresctrl_arch_unmount()if the mount fails, and on unmount).Testing in progress. Cleanup to follow before posting.
Thank you.
Quoted 7 lines
Quoted 6 lines
Quoted 2 lines
when would it be safe to call
intel_pmt_put_feature_group()?I think I am missing something here since I have been assuming all along that
intel_pmt_put_feature_group()is and will continue to be called from the new unmount handler.Quoted 8 lines
Correct.
Quoted 2 lines
Reinette
I'm looking at a more aggressive change to the
Kconfigoptions.Linus complained[1] that the biggest pain point for new people is getting the kernel configured. Extra CONFIG options was his prime focus.
Thank you for pointing this out.
Quoted 3 lines
So I'm looking at removing
CONFIG_X86_CPU_RESCTRL_INTEL_AETcompletely and just makingintel_aet.cincluded withCONFIG_X86_CPU_RESCTRL.Sounds reasonable but how the final dependencies would look like is not obvious to me. Consider for example this removal happening with today's
Kconfigthat may force a kernel that will run on AMD/Hygon to be built with INTEL_PMT_TELEMETRY/INTEL_TPMI. Something like this does not look right.Quoted 4 lines
It does mean the resctrl will depend on
X86_64, killing 32-bit support. That was discussed before and decision deferred. But there's other 64-bit MMIO stuff in the pipeline that means we have to make this change sooner or later.
Quoted 5 lines
Quoted 1 line
Thoughts on unconditional inclusion and on removing 32-bit support?
I believe we already have a clear answer on 32-bit support for resctrl on *x86* from Thomas https://lore.kernel.org/lkml/87jzm3jckw.ffs@tglx/ and Babu https://lore.kernel.org/lkml/[email protected]/
Good.
Quoted 2 lines
Whether this can be extended to resctrl fs is not obvious to me but I think this 64-bit requirement is indeed limited to x86 here?
Just x86. The specific need is for readq() to do a 64-bit read of an event counter from MMIO space.
Quoted 2 lines
About the unconditional inclusion I would just like to ensure that the resulting dependencies are reasonable for all x86.
Only Intel needs INTEL_PMT_TELEMETRY. I tested out this Kconfig dependency:
depends on (INTEL_PMT_TELEMETRY != n) || !CPU_SUP_INTEL
and managed to configure X86_CPU_RESCTRL without INTEL_PMT_TELEMETRY for a configuration that set CPU_SUP_AMD but not CPU_SUP_INTEL in the "Processor type and features" -> "Supported processor vendors" menu.
Note that your need CONFIG_EXPERT=y to see this option.
Quoted 7 lines
Reinette
-Tony
Tony Luck
maintainerPATCH v4 5/7 x86/resctrl: Resolve PMT and TPMI symbols at runtime
In reply to Tony LuckThe resctrl subsystem is always built-in. Currently, it has a direct dependency on INTEL_PMT_TELEMETRY and INTEL_TPMI for Application Event Telemetry (AET) features. This forces these drivers to be built-in as well, even though they are logically separate.
Switch to using symbol_request() to resolve AET-related symbols at runtime. This allows PMT and TPMI to be built as loadable modules.
Update the mount/unmount logic to manage these references:
- Acquire symbol references and pin the modules during resctrl mount.
- Ensure AET structures are cleaned up and events disabled on unmount.
- Release symbol references and unpin modules on unmount or if mounting fails.
--- arch/x86/kernel/cpu/resctrl/intel_aet.c | 93 ++++++++++++++++++++++--- 1 file changed, 83 insertions(+), 10 deletions(-)
arch/x86/kernel/cpu/resctrl/intel_aet.c +83 -10
@@ -23,6 +23,7 @@#include <linux/intel_vsec.h> #include <linux/io.h> #include <linux/minmax.h> +#include <linux/module.h> #include <linux/printk.h> #include <linux/rculist.h> #include <linux/rcupdate.h>
@@ -289,6 +290,9 @@ static enum pmt_feature_id lookup_pfid(const char *pfname)return FEATURE_INVALID ; } +static struct pmt_feature_group *(*get_feature)(enum pmt_feature_id id); +static void (*put_feature)(struct pmt_feature_group *p); +/* * Request a copy of struct pmt_feature_group for each event group. If there is * one, the returned structure has an array of telemetry_region structures,
@@ -300,7 +304,7 @@ static enum pmt_feature_id lookup_pfid(const char *pfname)* struct pmt_feature_group to indicate that its events are successfully * enabled. */ -bool intel_aet_get_events (void ) +static bool aet_get_events (void ) {struct pmt_feature_group *p;enum pmt_feature_id pfid;
@@ -309,14 +313,14 @@ bool intel_aet_get_events(void)for_each_event_group (peg) { pfid =lookup_pfid ((*peg)->pfname); - p =intel_pmt_get_regions_by_feature (pfid); + p =get_feature (pfid);if (IS_ERR_OR_NULL (p))continue ;if (enable_events (*peg, p)) { (*peg)->pfg = p; ret = true; }else { -intel_pmt_put_feature_group (p); +put_feature (p); } }
@@ -325,27 +329,96 @@ bool intel_aet_get_events(void) void __exit intel_aet_exit(void) { -struct event_group **peg; +} -for_each_event_group (peg) { -if ((*peg)->pfg) { -intel_pmt_put_feature_group ((*peg)->pfg); - (*peg)->pfg =NULL ; - } +static bool get_pmt_references (void ) +{ + get_feature =symbol_request (intel_pmt_get_regions_by_feature); +if (!get_feature) +return false; + put_feature =symbol_request (intel_pmt_put_feature_group); +if (!put_feature) { +symbol_put (intel_pmt_get_regions_by_feature); + get_feature =NULL ; +return false; + } + +return true; +} + +static void put_pmt_references (void ) +{ +if (get_feature) { +symbol_put (intel_pmt_get_regions_by_feature); + get_feature =NULL ; + } +if (put_feature) { +symbol_put (intel_pmt_put_feature_group); + put_feature =NULL ; } } +static enum { +AET_UNINITIALIZED , +AET_PRESENT , +AET_NOT_PRESENT +} aet_state; +bool intel_aet_pre_mount (void ) { -return false;// Temporary stub +bool ret; + +if (aet_state ==AET_PRESENT ) +return true; + +if (aet_state ==AET_NOT_PRESENT || !get_pmt_references ()) +return false; + + ret =aet_get_events (); + +if (ret) { + aet_state =AET_PRESENT ; + }else { + aet_state =AET_NOT_PRESENT ; +put_pmt_references (); + } + +return ret; +} + +static void aet_cleanup (void ) +{ +struct event_group **peg; + +if (aet_state ==AET_PRESENT ) { +for_each_event_group (peg) { +if ((*peg)->pfg) { +struct event_group *e = *peg; + +for (int j =0 ; j < e->num_events; j++) +resctrl_disable_mon_event (e->evts[j].id); +put_feature ((*peg)->pfg); + (*peg)->pfg =NULL ; + } + } +put_pmt_references (); + aet_state =AET_UNINITIALIZED ; + } }void intel_aet_mount_result (int ret) { +struct rdt_resource *r = &rdt_resources_all[RDT_RESOURCE_PERF_PKG ].r_resctrl; + +if (ret) { + r->mon_capable = false; +aet_cleanup (); + } }void intel_aet_unmount (void ) { +aet_cleanup (); }#define DATA_VALID BIT_ULL(63)
Signature · 1 line
2.53.0
Reinette Chatre
maintainerRe: [PATCH v4 5/7] x86/resctrl: Resolve PMT and TPMI symbols at runtime
Hi Tony,
Quoted 5 lines · Tony Luck
The resctrl subsystem is always built-in. Currently, it has a direct dependency on
INTEL_PMT_TELEMETRYandINTEL_TPMIfor Application Event Telemetry (AET) features. This forces these drivers to be built-in as well, even though they are logically separate.
Quoted 8 lines
Switch to using
symbol_request()to resolve AET-related symbols at runtime. This allows PMT and TPMI to be built as loadable modules.Update the mount/unmount logic to manage these references:
- Acquire symbol references and pin the modules during resctrl mount.
- Ensure AET structures are cleaned up and events disabled on unmount.
- Release symbol references and unpin modules on unmount or if mounting fails.
If I remember correctly resctrl depends on the mount being done much later than driver initialization to ensure all PMT load and enumeration is complete by the time resctrl fs is mounted.
Now it looks like these modules can be loaded dynamically on resctrl mount with the functions learning about PMT feature groups called right after. Considering the change in timing, is it guaranteed that if intel_pmt_get_regions_by_feature() is called right after module probe that the telemetry driver would be done with its enumeration?
Quoted 111 lines
Signed-off-by Tony Luck <[email protected]>--- arch/x86/kernel/cpu/resctrl/intel_aet.c | 93 ++++++++++++++++++++++--- 1 file changed, 83 insertions(+), 10 deletions(-)arch/x86/kernel/cpu/resctrl/intel_aet.c +47
-10@@ -23,6 +23,7 @@#include <linux/intel_vsec.h> #include <linux/io.h> #include <linux/minmax.h> +#include <linux/module.h> #include <linux/printk.h> #include <linux/rculist.h> #include <linux/rcupdate.h> @@ -289,6 +290,9 @@ static enum pmt_feature_id lookup_pfid(const char *pfname)return FEATURE_INVALID ; } +static struct pmt_feature_group *(*get_feature)(enum pmt_feature_id id); +static void (*put_feature)(struct pmt_feature_group *p); +/* * Request a copy of struct pmt_feature_group for each event group. If there is * one, the returned structure has an array of telemetry_region structures, @@ -300,7 +304,7 @@ static enum pmt_feature_id lookup_pfid(const char *pfname)* struct pmt_feature_group to indicate that its events are successfully * enabled. */ -bool intel_aet_get_events (void ) +static bool aet_get_events (void ) {struct pmt_feature_group *p;enum pmt_feature_id pfid;@@ -309,14 +313,14 @@ bool intel_aet_get_events(void)for_each_event_group (peg) { pfid =lookup_pfid ((*peg)->pfname); - p =intel_pmt_get_regions_by_feature (pfid); + p =get_feature (pfid);if (IS_ERR_OR_NULL (p))continue ;if (enable_events (*peg, p)) { (*peg)->pfg = p; ret = true; }else { -intel_pmt_put_feature_group (p); +put_feature (p); } }@@ -325,27 +329,96 @@ bool intel_aet_get_events(void) void __exit intel_aet_exit(void) { -struct event_group **peg; +} -for_each_event_group (peg) { -if ((*peg)->pfg) { -intel_pmt_put_feature_group ((*peg)->pfg); - (*peg)->pfg =NULL ; - } +static bool get_pmt_references (void ) +{ + get_feature =symbol_request (intel_pmt_get_regions_by_feature); +if (!get_feature) +return false; + put_feature =symbol_request (intel_pmt_put_feature_group); +if (!put_feature) { +symbol_put (intel_pmt_get_regions_by_feature); + get_feature =NULL ; +return false; + } + +return true; +} + +static void put_pmt_references (void ) +{ +if (get_feature) { +symbol_put (intel_pmt_get_regions_by_feature); + get_feature =NULL ; + } +if (put_feature) { +symbol_put (intel_pmt_put_feature_group); + put_feature =NULL ; } } +static enum { +AET_UNINITIALIZED , +AET_PRESENT , +AET_NOT_PRESENT +} aet_state; +bool intel_aet_pre_mount (void ) { -return false;// Temporary stub +bool ret; + +if (aet_state ==AET_PRESENT ) +return true; + +if (aet_state ==AET_NOT_PRESENT || !get_pmt_references ()) +return false; + + ret =aet_get_events ();
arch/x86/kernel/cpu/resctrl/intel_aet.c bool intel_aet_get_events(void) · line 387
Quoted 10 lines
arch/x86/kernel/cpu/resctrl/intel_aet.c +10
-0+ +if (ret) { + aet_state =AET_PRESENT ; + }else { + aet_state =AET_NOT_PRESENT ; +put_pmt_references (); + } + +return ret; +}
I think I am missing something here. This seems to build a new state machine on top of existing state tracking.
What if, instead, just do:
aet_get_events()
{
for_each_event_group(peg) {
if ((*peg)->pfg) /* This means "AET_PRESENT" ? */
continue;
...
}
}Similarly, get_feature and put_feature maintains state about the symbols.
arch/x86/kernel/cpu/resctrl/intel_aet.c bool intel_aet_get_events(void) · line 393
Quoted 6 lines
arch/x86/kernel/cpu/resctrl/intel_aet.c +6
-0+ +static void aet_cleanup (void ) +{ +struct event_group **peg; + +if (aet_state ==AET_PRESENT ) {
I do not see why this check/state machine is needed, if enumeration was not done the loop should just do nothing because (*peg)->pfg will always be NULL.
arch/x86/kernel/cpu/resctrl/intel_aet.c bool intel_aet_get_events(void) · line 399
Quoted 6 lines
arch/x86/kernel/cpu/resctrl/intel_aet.c +6
-0+for_each_event_group (peg) { +if ((*peg)->pfg) { +struct event_group *e = *peg; + +for (int j =0 ; j < e->num_events; j++) +resctrl_disable_mon_event (e->evts[j].id);
Can this be expected to clean up/reset all changes from enable_events()? For example, how about event_group::num_rmid and rdt_resource::resctrl_mon::num_rmid?
arch/x86/kernel/cpu/resctrl/intel_aet.c bool intel_aet_get_events(void) · line 404
Quoted 5 lines
arch/x86/kernel/cpu/resctrl/intel_aet.c +5
-0+put_feature ((*peg)->pfg); + (*peg)->pfg =NULL ; + } + } +put_pmt_references ();
Similarly here, put_pmt_references() could just always be called and it will not do anything if there aren't references?
Quoted 8 lines
arch/x86/kernel/cpu/resctrl/intel_aet.c +4
-0+ aet_state =AET_UNINITIALIZED ; + } }void intel_aet_mount_result (int ret) { +struct rdt_resource *r = &rdt_resources_all[RDT_RESOURCE_PERF_PKG ].r_resctrl; +
arch/x86/kernel/cpu/resctrl/intel_aet.c bool intel_aet_get_events(void) · line 422
Quoted 10 lines
arch/x86/kernel/cpu/resctrl/intel_aet.c +5
-0+if (ret) { + r->mon_capable = false; +aet_cleanup (); + } }void intel_aet_unmount (void ) { +aet_cleanup (); }
I am expecting intel_aet_mount_result() and intel_aet_unmount() to look the same and that intel_aet_mount_result() thus becomes unnecessary.
Just setting r->mon_capable to false does not seem enough though ... how about all those monitoring domains that were created? This flow looks unexpected to me, what am I missing?
Reinette
Luck, Tony
maintainerRe: [PATCH v4 5/7] x86/resctrl: Resolve PMT and TPMI symbols at runtime
Hi Reinette,
Quoted 17 lines · Reinette Chatre
Hi Tony,
Quoted 5 lines · Tony Luck
The resctrl subsystem is always built-in. Currently, it has a direct dependency on
INTEL_PMT_TELEMETRYandINTEL_TPMIfor Application Event Telemetry (AET) features. This forces these drivers to be built-in as well, even though they are logically separate.Quoted 8 lines
Switch to using
symbol_request()to resolve AET-related symbols at runtime. This allows PMT and TPMI to be built as loadable modules.Update the mount/unmount logic to manage these references:
- Acquire symbol references and pin the modules during resctrl mount.
- Ensure AET structures are cleaned up and events disabled on unmount.
- Release symbol references and unpin modules on unmount or if mounting fails.
Quoted 8 lines
If I remember correctly resctrl depends on the mount being done much later than driver initialization to ensure all PMT load and enumeration is complete by the time resctrl fs is mounted.
Now it looks like these modules can be loaded dynamically on resctrl mount with the functions learning about PMT feature groups called right after. Considering the change in timing, is it guaranteed that ifintel_pmt_get_regions_by_feature()is called right after module probe that the telemetry driver would be done with its enumeration?
You do remmeber correctly. Your concerns are valid. I'd seen that the pmt_telemetry module was auto-loaded, but hadn't tracked how/when that happens.
The sequence is the intel_vsec driver is auto-loaded based on the OOBMSM PCIe device ID. That sets up an auxillary bus which pulls in the pmt_telemetry module. Adding a debug pr_info() I saw that the probe routine for the pmt discover module was called at kernel timestamp 42.720156 for socket 0 and 42.725646 for socket 1.
Adding a "resctrl" line to /etc/fstab attempts the mount at 39.667. Three seconds too early. No PMT events are found, and code in this V4 version of the patch series marks the system as AET_NOT_PRESENT and will never look again :-(
I can drop the AET_NOT_PRESENT state so that a retry will succeed. I don't see another fix other than to document this limitation.
Workarounds are:
1) Change the CONFIG to build pmt_telemetry into the kernel (where we are today, but haven't heard from Linux distros like Red Hat, SUSE etc. on whether this is acceptable.)
2) Delay mounting the resctrl file system.
Quoted 118 lines
Quoted 118 lines
Signed-off-by Tony Luck <[email protected]>--- arch/x86/kernel/cpu/resctrl/intel_aet.c | 93 ++++++++++++++++++++++--- 1 file changed, 83 insertions(+), 10 deletions(-)arch/x86/kernel/cpu/resctrl/intel_aet.c +53
-10@@ -23,6 +23,7 @@#include <linux/intel_vsec.h> #include <linux/io.h> #include <linux/minmax.h> +#include <linux/module.h> #include <linux/printk.h> #include <linux/rculist.h> #include <linux/rcupdate.h> @@ -289,6 +290,9 @@ static enum pmt_feature_id lookup_pfid(const char *pfname)return FEATURE_INVALID ; } +static struct pmt_feature_group *(*get_feature)(enum pmt_feature_id id); +static void (*put_feature)(struct pmt_feature_group *p); +/* * Request a copy of struct pmt_feature_group for each event group. If there is * one, the returned structure has an array of telemetry_region structures, @@ -300,7 +304,7 @@ static enum pmt_feature_id lookup_pfid(const char *pfname)* struct pmt_feature_group to indicate that its events are successfully * enabled. */ -bool intel_aet_get_events (void ) +static bool aet_get_events (void ) {struct pmt_feature_group *p;enum pmt_feature_id pfid;@@ -309,14 +313,14 @@ bool intel_aet_get_events(void)for_each_event_group (peg) { pfid =lookup_pfid ((*peg)->pfname); - p =intel_pmt_get_regions_by_feature (pfid); + p =get_feature (pfid);if (IS_ERR_OR_NULL (p))continue ;if (enable_events (*peg, p)) { (*peg)->pfg = p; ret = true; }else { -intel_pmt_put_feature_group (p); +put_feature (p); } }@@ -325,27 +329,96 @@ bool intel_aet_get_events(void) void __exit intel_aet_exit(void) { -struct event_group **peg; +} -for_each_event_group (peg) { -if ((*peg)->pfg) { -intel_pmt_put_feature_group ((*peg)->pfg); - (*peg)->pfg =NULL ; - } +static bool get_pmt_references (void ) +{ + get_feature =symbol_request (intel_pmt_get_regions_by_feature); +if (!get_feature) +return false; + put_feature =symbol_request (intel_pmt_put_feature_group); +if (!put_feature) { +symbol_put (intel_pmt_get_regions_by_feature); + get_feature =NULL ; +return false; + } + +return true; +} + +static void put_pmt_references (void ) +{ +if (get_feature) { +symbol_put (intel_pmt_get_regions_by_feature); + get_feature =NULL ; + } +if (put_feature) { +symbol_put (intel_pmt_put_feature_group); + put_feature =NULL ; } } +static enum { +AET_UNINITIALIZED , +AET_PRESENT , +AET_NOT_PRESENT +} aet_state; +bool intel_aet_pre_mount (void ) { -return false;// Temporary stub +bool ret; + +if (aet_state ==AET_PRESENT ) +return true; + +if (aet_state ==AET_NOT_PRESENT || !get_pmt_references ()) +return false; + + ret =aet_get_events (); + +if (ret) { + aet_state =AET_PRESENT ; + }else { + aet_state =AET_NOT_PRESENT ; +put_pmt_references ();
Quoted 8 lines
arch/x86/kernel/cpu/resctrl/intel_aet.c bool intel_aet_get_events(void) · line 387
Quoted 4 lines
arch/x86/kernel/cpu/resctrl/intel_aet.c +4
-0+ } + +return ret; +}I think I am missing something here. This seems to build a new state machine on top of existing state tracking.
What if, instead, just do:
The state machine was built to cope with the spurious call to intel_aet_pre_mount() in the case that user attempts to mount resctrl while it is already mounted. I removed that issue with the refactor in the file system layer, but didn't clean up here.
The only other part of this state machine was to avoid repeated calls to enumerate telemetry events on platforms that don't support them. But given the race to complete enumeration is lost on early mount, that seems to be a bad idea too and can be dropped.
Quoted 11 lines
aet_get_events()
{
for_each_event_group(peg) {
if ((*peg)->pfg) /* This means "AET_PRESENT" ? */
continue;
...
}
}Quoted 2 lines
Similarly,
get_featureandput_featuremaintains state about the symbols.
Yes.
Quoted 9 lines
arch/x86/kernel/cpu/resctrl/intel_aet.c bool intel_aet_get_events(void) · line 393
Quoted 6 lines
arch/x86/kernel/cpu/resctrl/intel_aet.c +6
-0+ +static void aet_cleanup (void ) +{ +struct event_group **peg; + +if (aet_state ==AET_PRESENT ) {I do not see why this check/state machine is needed, if enumeration was not done the loop should just do nothing because (*peg)->pfg will always be NULL.
Yes.
Quoted 9 lines
arch/x86/kernel/cpu/resctrl/intel_aet.c bool intel_aet_get_events(void) · line 399
Quoted 6 lines
arch/x86/kernel/cpu/resctrl/intel_aet.c +6
-0+for_each_event_group (peg) { +if ((*peg)->pfg) { +struct event_group *e = *peg; + +for (int j =0 ; j < e->num_events; j++) +resctrl_disable_mon_event (e->evts[j].id);Can this be expected to clean up/reset all changes from
enable_events()? For example, how aboutevent_group::num_rmidandrdt_resource::resctrl_mon::num_rmid?
I don't clean these up. But I don't see that they would change from one mount to the next.
Quoted 8 lines
arch/x86/kernel/cpu/resctrl/intel_aet.c bool intel_aet_get_events(void) · line 404
Quoted 5 lines
arch/x86/kernel/cpu/resctrl/intel_aet.c +5
-0+put_feature ((*peg)->pfg); + (*peg)->pfg =NULL ; + } + } +put_pmt_references ();Similarly here,
put_pmt_references()could just always be called and it will not do anything if there aren't references?
Yes.
Quoted 14 lines
Quoted 14 lines
arch/x86/kernel/cpu/resctrl/intel_aet.c +8
-0+ aet_state =AET_UNINITIALIZED ; + } }void intel_aet_mount_result (int ret) { +struct rdt_resource *r = &rdt_resources_all[RDT_RESOURCE_PERF_PKG ].r_resctrl; + +if (ret) { + r->mon_capable = false; +aet_cleanup (); + } }
Quoted 7 lines
arch/x86/kernel/cpu/resctrl/intel_aet.c bool intel_aet_get_events(void) · line 422
Quoted 4 lines
arch/x86/kernel/cpu/resctrl/intel_aet.c +1
-0void intel_aet_unmount (void ) { +aet_cleanup (); }I am expecting
intel_aet_mount_result()andintel_aet_unmount()to look the same and thatintel_aet_mount_result()thus becomes unnecessary.
Agreed. Per your comments on the previous patch in the series. File system can just call the architecture unmount function in the case that the mount failed. Action for architecture code is same for an aborted mount and for an unmount that follows a successful mount.
Quoted 2 lines
Just setting
r->mon_capableto false does not seem enough though ... how about all those monitoring domains that were created? This flow looks unexpected to me, what am I missing?
I'm missing the code to clean up the domains. Will add to next version.
Quoted 2 lines
Reinette
-Tony
Reinette Chatre
maintainerRe: [PATCH v4 5/7] x86/resctrl: Resolve PMT and TPMI symbols at runtime
Hi Tony,
On 4/7/26 11:13 AM, Luck, Tony wrote:
...
Quoted 5 lines
Adding a "resctrl" line to
/etc/fstabattempts the mount at 39.667. Three seconds too early. No PMT events are found, and code in this V4 version of the patch series marks the system asAET_NOT_PRESENTand will never look again :-(
Quoted 8 lines
I can drop the
AET_NOT_PRESENTstate so that a retry will succeed. I don't see another fix other than to document this limitation.Workarounds are:
1) Change the CONFIG to buildpmt_telemetryinto the kernel (where we are today, but haven't heard from Linux distros like Red Hat, SUSE etc. on whether this is acceptable.)
2) Delay mounting the resctrl file system.
As I mentioned in https://lore.kernel.org/lkml/[email protected]/
I find (2) to be a poor user interface since it (a) requires user space to somehow know that the system supports AET and then (b) either delay for some indeterminate time or repetitively (for some indeterminate count) remount resctrl to obtain needed features.
Considering all the complications, could you please provide the motivation for this series? I should have checked for this first. The cover letter does not contain this information.
Reinette
Luck, Tony
maintainerRe: [PATCH v4 5/7] x86/resctrl: Resolve PMT and TPMI symbols at runtime
Quoted 24 lines · Reinette Chatre
Hi Tony,
On 4/7/26 11:13 AM, Luck, Tony wrote:
...Quoted 5 lines
Adding a "resctrl" line to
/etc/fstabattempts the mount at 39.667. Three seconds too early. No PMT events are found, and code in this V4 version of the patch series marks the system asAET_NOT_PRESENTand will never look again :-(Quoted 8 lines
I can drop the
AET_NOT_PRESENTstate so that a retry will succeed. I don't see another fix other than to document this limitation.Workarounds are:
1) Change the CONFIG to buildpmt_telemetryinto the kernel (where we are today, but haven't heard from Linux distros like Red Hat, SUSE etc. on whether this is acceptable.)
2) Delay mounting the resctrl file system.As I mentioned in https://lore.kernel.org/lkml/[email protected]/
I find (2) to be a poor user interface since it (a) requires user space to somehow know that the system supports AET and then (b) either delay for some indeterminate time or repetitively (for some indeterminate count) remount resctrl to obtain needed features.
Quoted 3 lines
Considering all the complications, could you please provide the motivation for this series? I should have checked for this first. The cover letter does not contain this information.
Reinette,
I'm concerned that AET will not be available in Linux distros. The beta for Ubuntu 26.04 is available. I downloaded the iso and booted. It has the v7.0 kernel, so AET is in the source. But checking the config I see:
ubuntu@ubuntu:/boot$ grep VSEC config-7.0.0-10-generic CONFIG_INTEL_VSEC=m ubuntu@ubuntu:/boot$ grep INTEL_PMT config-7.0.0-10-generic CONFIG_INTEL_PMT_CLASS=m CONFIG_INTEL_PMT_TELEMETRY=m CONFIG_INTEL_PMT_CRASHLOG=m CONFIG_INTEL_PMT_DISCOVERY=m ubuntu@ubuntu:/boot$ grep AET config-7.0.0-10-generic ubuntu@ubuntu:/boot$
The default for most distributions is to build as much as possible as modules. So I expect to see the same when Fedora, Red Hat, SUSE.
Even our internal intel-next daily builds are configuring modules (I did get them to switch the relevant modules to built-in for one release cycle. But they feel they should match what is done in production systems, so they have switched back.
-Tony
Tony Luck
maintainerPATCH v4 6/7 x86/resctrl: Delete intel_aet_exit()
In reply to Tony LuckAll cleanup for AET is now handled during file system unmount.
Drop intel_aet_exit().
--- arch/x86/kernel/cpu/resctrl/internal.h | 2 -- arch/x86/kernel/cpu/resctrl/core.c | 2 -- arch/x86/kernel/cpu/resctrl/intel_aet.c | 4 ---- 3 files changed, 8 deletions(-)
arch/x86/kernel/cpu/resctrl/internal.h +0 -2
@@ -237,7 +237,6 @@ void resctrl_arch_mbm_cntr_assign_set_one(struct rdt_resource *r);bool intel_aet_pre_mount (void );void intel_aet_mount_result (int ret);void intel_aet_unmount (void ); -void __exit intel_aet_exit(void);int intel_aet_read_event (int domid,u32 rmid,void *arch_priv,u64 *val);void intel_aet_mon_domain_setup (int cpu,int id,struct rdt_resource *r,struct list_head *add_pos);
@@ -246,7 +245,6 @@ bool intel_handle_aet_option(bool force_off, char *tok);static inline bool intel_aet_pre_mount (void ) {return false; }static inline void intel_aet_mount_result (int ret) { }static inline void intel_aet_unmount (void ) { } -static inline void __exit intel_aet_exit(void) { }static inline int intel_aet_read_event (int domid,u32 rmid,void *arch_priv,u64 *val) {return -EINVAL ;
arch/x86/kernel/cpu/resctrl/core.c +0 -2
@@ -1172,8 +1172,6 @@ late_initcall(resctrl_arch_late_init); static void __exit resctrl_arch_exit(void) { -intel_aet_exit (); -cpuhp_remove_state (rdt_online);resctrl_exit ();
arch/x86/kernel/cpu/resctrl/intel_aet.c +0 -4
@@ -327,10 +327,6 @@ static bool aet_get_events(void)return ret; } -void __exit intel_aet_exit(void) -{ -} -static bool get_pmt_references (void ) { get_feature =symbol_request (intel_pmt_get_regions_by_feature);
Signature · 1 line
2.53.0
Tony Luck
maintainerPATCH v4 7/7 x86/resctrl: Downgrade dependency of AET on INTEL_PMT
In reply to Tony LuckThe INTEL_AET code now using symbol_get() to access the INTEL_PMT enumeration functions so no requirement that they be built-in to the base kernel.
Update the Kconfig dependencies.
--- arch/x86/Kconfig | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
arch/x86/Kconfig +1 -1
@@ -542,7 +542,7 @@ config X86_CPU_RESCTRL config X86_CPU_RESCTRL_INTEL_AET bool "Intel Application Energy Telemetry" - depends on X86_64 && X86_CPU_RESCTRL && CPU_SUP_INTEL && INTEL_PMT_TELEMETRY=y && INTEL_TPMI=y + depends on X86_64 && X86_CPU_RESCTRL && CPU_SUP_INTEL && INTEL_PMT_TELEMETRY != n && INTEL_TPMI != n help Enable per-RMID telemetry events in resctrl.
Signature · 1 line
2.53.0