Skip to contentSkip to conversation

x86,fs/resctrl: Improve resctrl quality and consistency

PATCH In review Reinette Chatre 3 participants 24 messages Jul 1 – Jul 22

Subsystems (2)
lore
Share & tools
lore

Reinette Chatre

author maintainer
#0 lore raw

v5 cover x86,fs/resctrl: Improve resctrl quality and consistency

Applies on top of v7.2-rc1 + v5 of "x86,fs/resctrl: Fix long-standing issues":
https://lore.kernel.org/lkml/[email protected]/

Changes since v4:
- v4: https://lore.kernel.org/lkml/[email protected]/
- Rebased on top of V5 "x86,fs/resctrl: Fix long-standing issues":
https://lore.kernel.org/lkml/[email protected]/
A significant change in that series is the addition of info_kn_lock()/info_kn_unlock() helpers to the info file handlers. With the addition of locking to the max_threshold_occ_write() in that series it seemed appropriate to re-order the last_cmd_status handling in max_threshold_occ_write() to be after the change to last_cmd_status custom to contain max_threshold_occ_write() changes to one patch.
- Add resctrl filename to new error messages to disambiguate the various "Invalid input" messages. (Tony)
- Add Tony's Reviewed-by tag to all patches.

Changes since v3:
- v3: https://lore.kernel.org/lkml/[email protected]/
- Rebased on v7.1-rc1.
- Drop "MAINTAINERS: Update resctrl entry" and "fs/resctrl: Add missing return value descriptions" that can be found in v7.1-rc1.
- Drop "fs/resctrl: Avoid "may be used uninitialized" warning" that just does unnecessary work to appease compiler warning that is disabled by default. (Boris)
- Use pr_warn() instead of WARN_ON() in "x86/resctrl: Protect against bad shift".

Changes since v2:
- v2: https://lore.kernel.org/lkml/[email protected]/
- Rebased on top of tip/master with pending resctrl selftest changes,
https://lore.kernel.org/lkml/[email protected]/,
applied on top to support testing with latest kernel that has pending resctrl changes merged.
- Drop "fs/resctrl: Use stricter checks on input to cpus/cpus_list file" that ended up being too strict and removed one use case. (Chris)
- max_threshold_occ_write() switch to guard(). (Chenyu)
- Add more detail to one return value description.
- Add tags.
- No opinions so far on needing to update last_cmd_status on success of all read commands. At this time last_cmd_status will continue to return previous failure message (if any) after a read command accessing static data succeeds.

Changes since v1:
- v1: https://lore.kernel.org/lkml/[email protected]/
- To simplify tracking, include patch to MAINTAINERS submitted separately:
https://lore.kernel.org/lkml/4274c478922c01f9ceebc805acf991f10a95519f.1771442788.git.reinette.chatre@intel.com/
- Follow recent upstream changes to add reference to tip tree handbook in MAINTAINERS entry.
- Use new pattern in resctrl to enable looping over all entries of an enum while making adding new enum entries more clear all while avoiding warnings when compiling with -Wswitch. (Ben)
- Add another last_cmd_status enhancement that appends "[truncated]" to output of info/last_cmd_status if the backing buffer overflowed.
- Please see patch changelogs for details of changes.
- Add tags.

Hi Everybody,

This is a collection of resctrl cleanups assembled together for convenience and simpler tracking. I'd be happy to split them up if it makes review and/or handling easier.

Summary of changes:

- Let resctrl pass stricter checks from various tools to provide a cleaner baseline with the goal to promote healthier contributions:
- ./tools/docs/kernel-doc -Wall -v <files>
- Build with W=12
- ./scripts/coccicheck
- Static checkers

- Use accurate and consistent type for all uses of resource ID.

- In the unlikely scenario that resctrl picked a wrong CPU to read an event from, pass the error through to user space instead of claiming to succeed and returning a (wrong) result.

- Since inception of last_cmd_status feature there have been mismatches between resctrl file operation failures and the contents of info/last_cmd_status. This pattern keeps propagating with each new resctrl feature. Establish a new baseline with a new pattern that ensures info/last_cmd_status contains an accurate failure description that matches the most recent resctrl file operation failure.

One potential open:

There remains an inconsistency between resctrl file operations that do/can _not_ fail and the contents of info/last_cmd_status. If a resctrl file operation fails and an informational error is printed to last_cmd_status then a subsequent reading of a resctrl file (specifically most of the files found in info/) may succeed while info/last_cmd_status may or may not return the error from previous failure.

Ensuring last_cmd_status is reset on every read carries the cost of taking rdtgroup_mutex on several more user space initiated paths and thus increase contention on rdtgroup_mutex. I opted to not make this change and instead focus this work on ensuring that last_cmd_status is accurate whenever there is a failure during any resctrl file operation. Please let me know if you have opinions in this regard.

Update v5: The addition of the info_kn_lock()/info_kn_unlock() helpers to info file handlers now means that rdtgroup_mutex is held every time an info file is opened. There is thus now less burden to reset last_cmd_status buffer on each info file access. Even so, since there has not been any opinions in this regard I opted to keep current behavior.

Any feedback is appreciated.

Regards,

Reinette

Reinette Chatre (10):
fs/resctrl: Use correct format specifier for printing error pointers x86/resctrl: Protect against bad shift
fs/resctrl: Change pattern used to track number of entries in enum fs/resctrl: Use accurate type for rdt_resource::rid fs/resctrl: Pass error reading event through to user space fs/resctrl: Use accurate and symmetric exit flows fs/resctrl: Change last_cmd_status custom during input parsing fs/resctrl: Add last_cmd_status support for writes to
max_threshold_occupancy
fs/resctrl: Communicate resource group deleted error via
last_cmd_status
fs/resctrl: Inform user space when status buffer overflowed

 arch/x86/kernel/cpu/resctrl/core.c |   6 +-
 fs/resctrl/ctrlmondata.c           |  71 ++++++++++-------
 fs/resctrl/monitor.c               |  78 ++++++++++--------
 fs/resctrl/rdtgroup.c              | 123 +++++++++++++++++------------
 include/linux/resctrl.h            |  11 +--
 5 files changed, 173 insertions(+), 116 deletions(-)
Signature · 1 line
2.54.0
author maintainer
#1 lore raw

PATCH v5 1/10 fs/resctrl: Use correct format specifier for printing error pointers

Use correct format specifier for error pointer as coccinelle suggests:

    .../fs/resctrl/monitor.c:148:8-15: WARNING: Consider using %pe to print PTR_ERR()
    .../fs/resctrl/monitor.c:760:9-16: WARNING: Consider using %pe to print PTR_ERR()

---
Changes since v4:
- Add Tony's Reviewed-by tag.

---
 fs/resctrl/monitor.c | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)
fs/resctrl/monitor.c +4 -4
index 7340b1d17f17..29b13a690cf3 100644
@@ -144,8 +144,8 @@ void __check_limbo(struct rdt_l3_mon_domain *d, bool force_free)
 	arch_priv = mon_event_all[QOS_L3_OCCUP_EVENT_ID].arch_priv;
 	arch_mon_ctx = resctrl_arch_mon_ctx_alloc(r, QOS_L3_OCCUP_EVENT_ID);
 	if (IS_ERR(arch_mon_ctx)) {
-		pr_warn_ratelimited("Failed to allocate monitor context: %ld",
-				    PTR_ERR(arch_mon_ctx));
+		pr_warn_ratelimited("Failed to allocate monitor context: %pe",
+				    arch_mon_ctx);
 		return;
 	}
 
@@ -771,8 +771,8 @@ static void mbm_update_one_event(struct rdt_resource *r, struct rdt_l3_mon_domai
 	} else {
 		rr.arch_mon_ctx = resctrl_arch_mon_ctx_alloc(rr.r, evtid);
 		if (IS_ERR(rr.arch_mon_ctx)) {
-			pr_warn_ratelimited("Failed to allocate monitor context: %ld",
-					    PTR_ERR(rr.arch_mon_ctx));
+			pr_warn_ratelimited("Failed to allocate monitor context: %pe",
+					    rr.arch_mon_ctx);
 			return;
 		}
 	}
Signature · 1 line
2.54.0
author maintainer
#2 lore raw

PATCH v5 2/10 x86/resctrl: Protect against bad shift

In reply to Reinette Chatre

The size of the bandwidth specifier field is enumerated from AMD hardware. resctrl uses this field width to determine the maximum bandwidth supported that is stored in resctrl_membw::max_bw to which user space allocation requests are compared for validity.

resctrl_membw::max_bw is of type u32 while the register containing the bandwidth specifier field, L3QOS_BW_CONTROL_n, is 64 bits. While not an issue with current hardware it is theoretically possible that enumeration of maximum bandwidth may trigger invalid behavior if a future system can use a bandwidth specifier field larger than 32 bits. Whether this could ever represent a reasonable bandwidth value is unknown but addressing the issue will appease static checkers.

Ensure resctrl can accommodate the hardware's bandwidth specifier field width with an additional check. Switch to BIT() instead of open-coding the bitshift to avoid signed integer overflow if the number of bits is a valid 31.

---
Changes since v3:
- Use pr_warn() instead of WARN_ON().
sashiko: https://sashiko.dev/#/patchset/cover.1775576382.git.reinette.chatre%40intel.com

Changes since v4:
- Add Tony's Reviewed-by tag.

---
 arch/x86/kernel/cpu/resctrl/core.c | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)
arch/x86/kernel/cpu/resctrl/core.c +5 -1
index bca782050198..9b9495174041 100644
@@ -247,7 +247,11 @@ static __init bool __rdt_get_mem_config_amd(struct rdt_resource *r)
 
 	cpuid_count(0x80000020, subleaf, &eax, &ebx, &ecx, &edx);
 	hw_res->num_closid = edx + 1;
-	r->membw.max_bw = 1 << eax;
+	if (BITS_PER_TYPE(r->membw.max_bw) <= eax) {
+		pr_warn("Unable to support hardware's maximum bandwidth\n");
+		return false;
+	}
+	r->membw.max_bw = BIT(eax);
 
 	/* AMD does not use delay */
 	r->membw.delay_linear = false;
Signature · 1 line
2.54.0
maintainer
#3 lore raw

Re: [PATCH v5 02/10] x86/resctrl: Protect against bad shift

Quoted 5 lines · Reinette Chatre
arch/x86/kernel/cpu/resctrl/core.c +0 -0
index bca782050198..9b9495174041 100644

arch/x86/kernel/cpu/resctrl/core.c static __init bool __rdt_get_mem_config_amd(struct rdt_resource *r) · line 254

Quoted 9 lines
arch/x86/kernel/cpu/resctrl/core.c +5 -1
@@ -247,7 +247,11 @@ static __init bool __rdt_get_mem_config_amd(struct rdt_resource *r)
 
 	cpuid_count(0x80000020, subleaf, &eax, &ebx, &ecx, &edx);
 	hw_res->num_closid = edx + 1;
-	r->membw.max_bw = 1 << eax;
+	if (BITS_PER_TYPE(r->membw.max_bw) <= eax) {
+		pr_warn("Unable to support hardware's maximum bandwidth\n");
+		return false;
+	}
+	r->membw.max_bw = BIT(eax);

Why?

Why go all the effort if the field can be 64 bits?

Why not make max_bw u64 and handle *all* possible values naturally?

Signature · 4 lines
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette
author maintainer
#4 lore raw

Re: [PATCH v5 02/10] x86/resctrl: Protect against bad shift

Hi Boris,

Quoted 18 lines · Borislav Petkov
Quoted 5 lines · Reinette Chatre
arch/x86/kernel/cpu/resctrl/core.c +0 -0
index bca782050198..9b9495174041 100644

arch/x86/kernel/cpu/resctrl/core.c static __init bool __rdt_get_mem_config_amd(struct rdt_resource *r) · line 254

Quoted 9 lines
arch/x86/kernel/cpu/resctrl/core.c +5 -1
@@ -247,7 +247,11 @@ static __init bool __rdt_get_mem_config_amd(struct rdt_resource *r)
 
 	cpuid_count(0x80000020, subleaf, &eax, &ebx, &ecx, &edx);
 	hw_res->num_closid = edx + 1;
-	r->membw.max_bw = 1 << eax;
+	if (BITS_PER_TYPE(r->membw.max_bw) <= eax) {
+		pr_warn("Unable to support hardware's maximum bandwidth\n");
+		return false;
+	}
+	r->membw.max_bw = BIT(eax);

Why?

Quoted 4 lines

Why go all the effort if the field can be 64 bits?

Why not make max_bw u64 and handle *all* possible values naturally?

resctrl does not support values larger than u32.

The value represented by max_bw provides the upper limit of control values that user space can configure via the schemata file, which in turn represents the control value programmed to hardware.

u32 is expected in several places:
- Parsing of user input in bw_validate() is done via kstrtou32() placing result in "u32 bw". It is this value that is compared against max_bw and would thus need to change as a consequence.
- Parsing of user input to schemata file via parse_bw() places parsed bandwidth in "u32 bw_val".
- Upon successful parsing the control value is staged by resctrl fs in resctrl_staged_config::new_ctrl that is u32.
- On receipt of programming request from resctrl fs, the x86 architectural specific code programs the control value to rdt_hw_ctrl_domain::ctrl_val[] that are u32.

resctrl, from fs to x86 architecture and now also MPAM, would thus require changes to support these AMD systems. If I get the math right this will be systems that support 1073741823.875GB/s and up?

If resctrl is required to support AMD hardware that can return up to 64bits here then that requires more changes than what I am able to take on at this time. From my side I prefer to stick to this resctrl safeguard for the scenario and leave support for this hardware to AMD.

Reinette

maintainer
#5 lore raw

Re: [PATCH v5 02/10] x86/resctrl: Protect against bad shift

Quoted 43 lines · Reinette Chatre

Hi Boris,

Quoted 18 lines · Borislav Petkov
Quoted 5 lines · Reinette Chatre
arch/x86/kernel/cpu/resctrl/core.c +0 -0
index bca782050198..9b9495174041 100644

arch/x86/kernel/cpu/resctrl/core.c static __init bool __rdt_get_mem_config_amd(struct rdt_resource *r) · line 254

Quoted 9 lines
arch/x86/kernel/cpu/resctrl/core.c +5 -1
@@ -247,7 +247,11 @@ static __init bool __rdt_get_mem_config_amd(struct rdt_resource *r)
 
 	cpuid_count(0x80000020, subleaf, &eax, &ebx, &ecx, &edx);
 	hw_res->num_closid = edx + 1;
-	r->membw.max_bw = 1 << eax;
+	if (BITS_PER_TYPE(r->membw.max_bw) <= eax) {
+		pr_warn("Unable to support hardware's maximum bandwidth\n");
+		return false;
+	}
+	r->membw.max_bw = BIT(eax);

Why?

Quoted 4 lines

Why go all the effort if the field can be 64 bits?

Why not make max_bw u64 and handle *all* possible values naturally?

resctrl does not support values larger than u32.

The value represented by max_bw provides the upper limit of control values that user space can configure via the schemata file, which in turn represents the control value programmed to hardware.

u32 is expected in several places:
- Parsing of user input in bw_validate() is done via kstrtou32() placing result in "u32 bw". It is this value that is compared against max_bw and would thus need to change as a consequence.
- Parsing of user input to schemata file via parse_bw() places parsed bandwidth in "u32 bw_val".
- Upon successful parsing the control value is staged by resctrl fs in resctrl_staged_config::new_ctrl that is u32.
- On receipt of programming request from resctrl fs, the x86 architectural specific code programs the control value to rdt_hw_ctrl_domain::ctrl_val[] that are u32.

Quoted 8 lines

resctrl, from fs to x86 architecture and now also MPAM, would thus require changes to support these AMD systems. If I get the math right this will be systems that support 1073741823.875GB/s and up?

If resctrl is required to support AMD hardware that can return up to 64bits here then that requires more changes than what I am able to take on at this time. From my side I prefer to stick to this resctrl safeguard for the scenario and leave support for this hardware to AMD.

So that bandwidth limit thing is read from those L3QOS_BW_CONTROL_n MSRs which have

BW_LEN-1:0 as the range which is the bandwidth limit, at BW_LEN there's the unlimited bandwidth limit

and that BW_LEN is read from CPUID.

And that is, kinda 12, right now.

And your worry is that some day that limit BW_LEN might go over 31?

Which your math shows how absurd this is.

So what are we actually fixing here?

Something some AI hallucinated that it might be possible at some point in the very distant, dark future?

Thx.

Signature · 4 lines
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette
author maintainer
#6 lore raw

Re: [PATCH v5 02/10] x86/resctrl: Protect against bad shift

Hi Boris,

Quoted 54 lines · Borislav Petkov
Quoted 41 lines · Reinette Chatre
Quoted 18 lines · Borislav Petkov
Quoted 5 lines · Reinette Chatre
arch/x86/kernel/cpu/resctrl/core.c +0 -0
index bca782050198..9b9495174041 100644

arch/x86/kernel/cpu/resctrl/core.c static __init bool __rdt_get_mem_config_amd(struct rdt_resource *r) · line 254

Quoted 9 lines
arch/x86/kernel/cpu/resctrl/core.c +5 -1
@@ -247,7 +247,11 @@ static __init bool __rdt_get_mem_config_amd(struct rdt_resource *r)
 
 	cpuid_count(0x80000020, subleaf, &eax, &ebx, &ecx, &edx);
 	hw_res->num_closid = edx + 1;
-	r->membw.max_bw = 1 << eax;
+	if (BITS_PER_TYPE(r->membw.max_bw) <= eax) {
+		pr_warn("Unable to support hardware's maximum bandwidth\n");
+		return false;
+	}
+	r->membw.max_bw = BIT(eax);

Why?

Quoted 4 lines

Why go all the effort if the field can be 64 bits?

Why not make max_bw u64 and handle *all* possible values naturally?

resctrl does not support values larger than u32.

The value represented by max_bw provides the upper limit of control values that user space can configure via the schemata file, which in turn represents the control value programmed to hardware.

u32 is expected in several places:
- Parsing of user input in bw_validate() is done via kstrtou32() placing result in "u32 bw". It is this value that is compared against max_bw and would thus need to change as a consequence.
- Parsing of user input to schemata file via parse_bw() places parsed bandwidth in "u32 bw_val".
- Upon successful parsing the control value is staged by resctrl fs in resctrl_staged_config::new_ctrl that is u32.
- On receipt of programming request from resctrl fs, the x86 architectural specific code programs the control value to rdt_hw_ctrl_domain::ctrl_val[] that are u32.

Quoted 8 lines

resctrl, from fs to x86 architecture and now also MPAM, would thus require changes to support these AMD systems. If I get the math right this will be systems that support 1073741823.875GB/s and up?

If resctrl is required to support AMD hardware that can return up to 64bits here then that requires more changes than what I am able to take on at this time. From my side I prefer to stick to this resctrl safeguard for the scenario and leave support for this hardware to AMD.

So that bandwidth limit thing is read from those L3QOS_BW_CONTROL_n MSRs which have

Quoted 8 lines

BW_LEN-1:0 as the range which is the bandwidth limit, at BW_LEN there's the unlimited bandwidth limit

and that BW_LEN is read from CPUID.

And that is, kinda 12, right now.

And your worry is that some day that limit BW_LEN might go over 31?

This is theoretical since the spec/hardware technically supports larger values than what resctrl supports. This could take the form of actual hardware or some model aimed at testing limits.

Quoted 7 lines

Which your math shows how absurd this is.

So what are we actually fixing here?

Something some AI hallucinated that it might be possible at some point in the very distant, dark future?

Not an AI hallucination but something encountered during my earlier attempts at getting resctrl to pass with fewer issues reported by the various deterministic tools, in this case a static checker, that I run as part of checking resctrl health.

I aim to highlight this caveat in the changelog with:
Whether this could ever represent a reasonable bandwidth value is unknown but addressing the issue will appease static checkers.

I see how such change could be considered absurd. I am ok to drop it from the series and just keep treating it's report locally as a false positive.

Reinette

maintainer
#7 lore raw

Re: [PATCH v5 02/10] x86/resctrl: Protect against bad shift

Quoted 2 lines · Reinette Chatre

I see how such change could be considered absurd. I am ok to drop it from the series and just keep treating it's report locally as a false positive.

Nah, but I'd like you, pls, to rewrite the commit message and explain more prominently

1. resctrl does not support values than u32 max_bw

2. what this is really fixing and where the potential issue can come from and what the situation is today.

I'd like to have more background in the commit message why we even arrived at this patch.

I wouldnt've even done it - we have bigger fish to fry - but if you think it is important for resctrl, sure.

Thx.

Signature · 4 lines
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette
author maintainer
#8 lore raw

Re: [PATCH v5 02/10] x86/resctrl: Protect against bad shift

Hi Boris,

Quoted 9 lines · Borislav Petkov
Quoted 2 lines · Reinette Chatre

I see how such change could be considered absurd. I am ok to drop it from the series and just keep treating it's report locally as a false positive.

Nah, but I'd like you, pls, to rewrite the commit message and explain more prominently

1. resctrl does not support values than u32 max_bw

Quoted 8 lines

2. what this is really fixing and where the potential issue can come from and what the situation is today.

I'd like to have more background in the commit message why we even arrived at this patch.

I wouldnt've even done it - we have bigger fish to fry - but if you think it is important for resctrl, sure.

I am trying to help with bigger things in resctrl and find it easier to do so when working on a baseline that is without implicit assumptions.

I plan to use changelog below that aims to address the items you highlighted and also changed the math to make the bandwidth boundary where resctrl stop working clear:

x86/resctrl: Protect against bad shift

The size of the bandwidth field is enumerated from AMD hardware. resctrl uses this field width to determine the maximum bandwidth supported that is stored in resctrl_membw::max_bw. User space allocation requests ("control values") are compared against this maximum for validity before being programmed to hardware.

resctrl filesystem and resctrl x86 architecture code only support u32 control
values: resctrl_membw::max_bw is a u32, the control value provided by user space is parsed into u32 local variables, and after validity checks, the control value is staged into the u32 resctrl_staged_config::new_ctrl for architecture consumption. The resctrl x86 architecture code in turn caches the new control value into the u32 array rdt_hw_ctrl_domain::ctrl_val[].

The AMD bandwidth field to which control values are written can be up to 64 bits wide. While not an issue with current hardware (bandwidths that require more than a u32, more than 536870911.875 GB/s, seem unreasonable today), it is theoretically possible that enumeration of maximum bandwidth field width will return values that are according to specification but cannot be supported by resctrl.

Static checkers complain about this size mismatch.

Fix the static checker complaint by explicitly encoding the fact that resctrl is unable to support all values that the hardware specification allows. Switch to BIT() instead of open-coding the bitshift to avoid signed integer overflow if the number of bits is a valid 31.

Reinette

maintainer
#9 lore raw

Re: [PATCH v5 02/10] x86/resctrl: Protect against bad shift

Quoted 2 lines · Reinette Chatre

I am trying to help with bigger things in resctrl and find it easier to do so when working on a baseline that is without implicit assumptions.

Ofc.

Quoted 2 lines

I plan to use changelog below that aims to address the items you highlighted and also changed the math to make the bandwidth boundary where resctrl stop working clear:

LGTM.

You can send me an updated patch now and I can integrate it into the pile.

Thx.

Signature · 4 lines
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette
author maintainer
#10 lore raw

Re: [PATCH v5 02/10] x86/resctrl: Protect against bad shift

Quoted 1 line · Borislav Petkov

You can send me an updated patch now and I can integrate it into the pile.

Thank you very much Boris. I'll do so shortly.

Reinette

author maintainer
#11 lore raw

PATCH v5 2/10 x86/resctrl: Protect against bad shift

In reply to Borislav Petkov

The size of the bandwidth field is enumerated from AMD hardware. resctrl uses this field width to determine the maximum bandwidth supported that is stored in resctrl_membw::max_bw. User space allocation requests ("control values") are compared against this maximum for validity before being programmed to hardware.

resctrl filesystem and resctrl x86 architecture code only support u32 control values: resctrl_membw::max_bw is a u32, the control value provided by user space is parsed into u32 local variables, and after validity checks, the control value is staged into the u32 resctrl_staged_config::new_ctrl for architecture consumption. The resctrl x86 architecture code in turn caches the new control value into the u32 array rdt_hw_ctrl_domain::ctrl_val[].

The AMD bandwidth field to which control values are written can be up to 64 bits wide. While not an issue with current hardware (bandwidths that require more than a u32, more than 536870911.875 GB/s, seem unreasonable today), it is theoretically possible that enumeration of maximum bandwidth field width will return values that are according to specification but cannot be supported by resctrl.

Static checkers complain about this size mismatch.

Fix the static checker complaint by explicitly encoding the fact that resctrl is unable to support all values that the hardware specification allows. Switch to BIT() instead of open-coding the bitshift to avoid signed integer overflow if the number of bits is a valid 31.

---
Changes since v3:
- Use pr_warn() instead of WARN_ON().
sashiko: https://sashiko.dev/#/patchset/cover.1775576382.git.reinette.chatre%40intel.com

Changes since v4:
- Add Tony's Reviewed-by tag.

Changes since v5:
- Rewrite changelog to highlight that resctrl cannot support bandwidth needing more than u32 and what this patch really fixes. (Boris)

---
 arch/x86/kernel/cpu/resctrl/core.c | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)
arch/x86/kernel/cpu/resctrl/core.c +5 -1
index bca782050198..9b9495174041 100644
@@ -247,7 +247,11 @@ static __init bool __rdt_get_mem_config_amd(struct rdt_resource *r)
 
 	cpuid_count(0x80000020, subleaf, &eax, &ebx, &ecx, &edx);
 	hw_res->num_closid = edx + 1;
-	r->membw.max_bw = 1 << eax;
+	if (BITS_PER_TYPE(r->membw.max_bw) <= eax) {
+		pr_warn("Unable to support hardware's maximum bandwidth\n");
+		return false;
+	}
+	r->membw.max_bw = BIT(eax);
 
 	/* AMD does not use delay */
 	r->membw.delay_linear = false;
Signature · 1 line
2.54.0
author maintainer
#12 lore raw

PATCH v5 3/10 fs/resctrl: Change pattern used to track number of entries in enum

In reply to Reinette Chatre

CDP_NUM_TYPES tracks the number of different configuration types that can be applied to a resource. CDP_NUM_TYPES is required to iterate over the different configurations but is not a member of enum resctrl_conf_type to avoid the warning generated with -Wswitch when CDP_NUM_TYPES lacks a case.

Add a new CDP_LAST enum entry used in CDP_NUM_TYPES definition to simplify adding a new enum entry. Do this to create a cleaner pattern for tracking the number of enum entries in resctrl in preparation for other enums needing to do so.

---
Changes since v2:
- Add Ben's RB tag.

Changes since v4:
- Add Tony's Reviewed-by tag.

---
 include/linux/resctrl.h | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
include/linux/resctrl.h +2 -1
index 73ff522448a0..c3ae0e8cb522 100644
@@ -69,9 +69,10 @@ enum resctrl_conf_type {
 	CDP_NONE,
 	CDP_CODE,
 	CDP_DATA,
+	CDP_LAST = CDP_DATA
 };
 
-#define CDP_NUM_TYPES	(CDP_DATA + 1)
+#define CDP_NUM_TYPES	(CDP_LAST + 1)
 
 /*
  * struct pseudo_lock_region - pseudo-lock region information
Signature · 1 line
2.54.0
author maintainer
#13 lore raw

PATCH v5 4/10 fs/resctrl: Use accurate type for rdt_resource::rid

In reply to Reinette Chatre

Every resctrl resource has a unique ID described by enum resctrl_res_level. enum resctrl_res_level is used in all resource ID initializations and all resource ID comparisons. All functions consuming the resource ID expects an enum resctrl_res_level. Of the four structures that contain a resource ID (struct mon_data, struct mon_evt, struct rdt_domain_hdr, and struct
rdt_resource) only struct rdt_resource does not use enum resctrl_res_level.

Switch the type of rdt_resource::rid to be enum resctrl_res_level to make it obvious what values are valid, match the type everywhere this member is used, and obtain benefits from tools that can flag any enum misuse.

Move define of RDT_NUM_RESOURCES outside the enum to enable tools to catch when a switch() on the resource ID does not handle all the resources and thus help flag which switch statements need an update when a new resource is added.

---
Changes since v1:
- Use RDT_RESOURCE_LAST = RDT_RESOURCE_PERF_PKG as last entry in enum.(Ben)

Changes since v2:
- Add Ben's RB tag.

Changes since v4:
- Add Tony's Reviewed-by tag.

---
 include/linux/resctrl.h | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)
include/linux/resctrl.h +4 -4
index c3ae0e8cb522..dd09c2ce9a0f 100644
@@ -54,11 +54,11 @@ enum resctrl_res_level {
 	RDT_RESOURCE_MBA,
 	RDT_RESOURCE_SMBA,
 	RDT_RESOURCE_PERF_PKG,
-
-	/* Must be the last */
-	RDT_NUM_RESOURCES,
+	RDT_RESOURCE_LAST = RDT_RESOURCE_PERF_PKG
 };
 
+#define RDT_NUM_RESOURCES (RDT_RESOURCE_LAST + 1)
+
 /**
  * enum resctrl_conf_type - The type of configuration.
  * @CDP_NONE:	No prioritisation, both code and data are controlled or monitored.
@@ -324,7 +324,7 @@ struct resctrl_mon {
  * @cdp_capable:	Is the CDP feature available on this resource
  */
 struct rdt_resource {
-	int			rid;
+	enum resctrl_res_level	rid;
 	bool			alloc_capable;
 	bool			mon_capable;
 	enum resctrl_scope	ctrl_scope;
Signature · 1 line
2.54.0
author maintainer
#14 lore raw

PATCH v5 5/10 fs/resctrl: Pass error reading event through to user space

In reply to Reinette Chatre

Reading of event data is managed through populating a struct rmid_read with properties of event needing to be read. This data is dispatched to an appropriate CPU and upon completion any error can be found in rmid_read::err, or on success the event data will be in rmid_read::val.

rmid_read::err is not updated in the unlikely scenario that the reading of the event was dispatched to a wrong CPU. If this ever occurs due to a bug in resctrl the user space read will return "success" but the data reported will be invalid.

Ensure accurate error reporting so that if there may be an issue with how resctrl picks a CPU it could be learned with an error to user space instead of silent failure.

---
Changes since v1:
- Add Ben's RB tag.

Changes since V4:
- Add Tony's Reviewed-by tag.

---
 fs/resctrl/monitor.c | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)
fs/resctrl/monitor.c +6 -2
index 29b13a690cf3..7f3fe055f7c9 100644
@@ -458,8 +458,10 @@ static int __l3_mon_event_count(struct rdtgroup *rdtgrp, struct rmid_read *rr)
 	}
 
 	/* Reading a single domain, must be on a CPU in that domain. */
-	if (!cpumask_test_cpu(cpu, &d->hdr.cpu_mask))
+	if (!cpumask_test_cpu(cpu, &d->hdr.cpu_mask)) {
+		rr->err = -EIO;
 		return -EINVAL;
+	}
 	if (rr->is_mbm_cntr)
 		rr->err = resctrl_arch_cntr_read(rr->r, d, closid, rmid, cntr_id,
 						 rr->evt->evtid, &tval);
@@ -496,8 +498,10 @@ static int __l3_mon_event_count_sum(struct rdtgroup *rdtgrp, struct rmid_read *r
 	}
 
 	/* Summing domains that share a cache, must be on a CPU for that cache. */
-	if (!cpumask_test_cpu(cpu, &rr->ci->shared_cpu_map))
+	if (!cpumask_test_cpu(cpu, &rr->ci->shared_cpu_map)) {
+		rr->err = -EIO;
 		return -EINVAL;
+	}
 
 	/*
 	 * Legacy files must report the sum of an event across all
Signature · 1 line
2.54.0
author maintainer
#15 lore raw

PATCH v5 6/10 fs/resctrl: Use accurate and symmetric exit flows

In reply to Reinette Chatre

During schemata file write handling there is one error exit path labeled "out" that handles all cleanup and unlocking needed on exit. The staged configs cleared during an error exit may not have been used at the time of exit making the clearing of the staged configs unnecessary.

Access the exit code using two labels and only clear the staged configuration if it was in use at the time of exit. Doing so makes the code flow obvious and simplifies upcoming changes that improve the handling of failures during early input parsing.

---
Changes since v4:
- Add Tony's Reviewed-by tag.

---
 fs/resctrl/ctrlmondata.c | 13 +++++++------
 1 file changed, 7 insertions(+), 6 deletions(-)
fs/resctrl/ctrlmondata.c +7 -6
index 2b29fb5a8702..e6ad8927cf50 100644
@@ -331,7 +331,7 @@ ssize_t rdtgroup_schemata_write(struct kernfs_open_file *of,
 	if (rdtgrp->mode == RDT_MODE_PSEUDO_LOCKED) {
 		ret = -EINVAL;
 		rdt_last_cmd_puts("Resource group is pseudo-locked\n");
-		goto out;
+		goto out_unlock;
 	}
 
 	rdt_staged_configs_clear();
@@ -341,16 +341,16 @@ ssize_t rdtgroup_schemata_write(struct kernfs_open_file *of,
 		if (!tok) {
 			rdt_last_cmd_puts("Missing ':'\n");
 			ret = -EINVAL;
-			goto out;
+			goto out_clear_staged;
 		}
 		if (tok[0] == '\0') {
 			rdt_last_cmd_printf("Missing '%s' value\n", resname);
 			ret = -EINVAL;
-			goto out;
+			goto out_clear_staged;
 		}
 		ret = rdtgroup_parse_resource(resname, tok, rdtgrp);
 		if (ret)
-			goto out;
+			goto out_clear_staged;
 	}
 
 	list_for_each_entry(s, &resctrl_schema_all, list) {
@@ -365,7 +365,7 @@ ssize_t rdtgroup_schemata_write(struct kernfs_open_file *of,
 
 		ret = resctrl_arch_update_domains(r, rdtgrp->closid);
 		if (ret)
-			goto out;
+			goto out_clear_staged;
 	}
 
 	if (rdtgrp->mode == RDT_MODE_PSEUDO_LOCKSETUP) {
@@ -378,8 +378,9 @@ ssize_t rdtgroup_schemata_write(struct kernfs_open_file *of,
 		ret = rdtgroup_pseudo_lock_create(rdtgrp);
 	}
 
-out:
+out_clear_staged:
 	rdt_staged_configs_clear();
+out_unlock:
 	rdtgroup_kn_unlock(of->kn);
 	return ret ?: nbytes;
 }
Signature · 1 line
2.54.0
author maintainer
#16 lore raw

PATCH v5 7/10 fs/resctrl: Change last_cmd_status custom during input parsing

In reply to Reinette Chatre

A pattern of usage of last_cmd_status was introduced during its enabling in commit

c377dcfbee80 ("x86/intel_rdt: Add diagnostics when writing the schemata file")

and since copied throughout resctrl to result in the following custom:

	..._write()
	{
		/* Early parsing of input, exit on failure. */

		/* Obtain rdtgroup_mutex */
		rdt_last_cmd_clear(); /* Clear last_cmd_status buffer */

		/*
		 * Act on user command, failures result in detail
		 * error message in last_cmd_status buffer via
		 * rdt_last_cmd_puts()/rdt_last_cmd_printf().
		 */

		/* Release rdtgroup_mutex */
	}

If resctrl exits with failure during early parsing of input there are two possible scenarios:
- The last_cmd_status buffer is empty and a user's read of info/last_cmd_status returns "ok".
- The last_cmd_status buffer contains details from an earlier ...write() failure and a user's read of info/last_cmd_status returns this outdated error description.

Writing to a resctrl file is considered a "resctrl command" and the resctrl documentation states the following about the last_cmd_status file:
"If the command failed, it will provide more information that can be
conveyed in the error returns from file operations."
Neither of the current scenarios is correct behavior.

Move early input parsing to be done with rdtgroup_mutex held after the last_cmd_status buffer is cleared. Let info/last_cmd_status be accurate when an error is encountered during parsing of user command.

---
Changes since v1:
- Add Ben's RB tag.

Changes since v4:
- Rebase on top of the "x86,fs/resctrl: Fix long-standing issues" series that added locking helpers to info file handlers.
- Add resctrl filename to new error messages to disambiguate the various "Invalid input" messages. (Tony)
- Add Tony's Reviewed-by tag.

---
 fs/resctrl/ctrlmondata.c | 55 ++++++++++++++++----------
 fs/resctrl/monitor.c     | 60 ++++++++++++++++------------
 fs/resctrl/rdtgroup.c    | 84 +++++++++++++++++++++++-----------------
 3 files changed, 120 insertions(+), 79 deletions(-)
fs/resctrl/ctrlmondata.c +35 -20
index e6ad8927cf50..8d5a8b1e659a 100644
@@ -312,11 +312,6 @@ ssize_t rdtgroup_schemata_write(struct kernfs_open_file *of,
 	char *tok, *resname;
 	int ret = 0;
 
-	/* Valid input requires a trailing newline */
-	if (nbytes == 0 || buf[nbytes - 1] != '\n')
-		return -EINVAL;
-	buf[nbytes - 1] = '\0';
-
 	rdtgrp = rdtgroup_kn_lock_live(of->kn);
 	if (!rdtgrp) {
 		rdtgroup_kn_unlock(of->kn);
@@ -324,6 +319,15 @@ ssize_t rdtgroup_schemata_write(struct kernfs_open_file *of,
 	}
 	rdt_last_cmd_clear();
 
+	/* Valid input requires a trailing newline */
+	if (nbytes == 0 || buf[nbytes - 1] != '\n') {
+		rdt_last_cmd_puts("schemata: Invalid input\n");
+		ret = -EINVAL;
+		goto out_unlock;
+	}
+
+	buf[nbytes - 1] = '\0';
+
 	/*
 	 * No changes to pseudo-locked region allowed. It has to be removed
 	 * and re-created instead.
@@ -466,11 +470,6 @@ ssize_t rdtgroup_mba_mbps_event_write(struct kernfs_open_file *of,
 	struct rdtgroup *rdtgrp;
 	int ret = 0;
 
-	/* Valid input requires a trailing newline */
-	if (nbytes == 0 || buf[nbytes - 1] != '\n')
-		return -EINVAL;
-	buf[nbytes - 1] = '\0';
-
 	rdtgrp = rdtgroup_kn_lock_live(of->kn);
 	if (!rdtgrp) {
 		rdtgroup_kn_unlock(of->kn);
@@ -478,6 +477,15 @@ ssize_t rdtgroup_mba_mbps_event_write(struct kernfs_open_file *of,
 	}
 	rdt_last_cmd_clear();
 
+	/* Valid input requires a trailing newline */
+	if (nbytes == 0 || buf[nbytes - 1] != '\n') {
+		rdt_last_cmd_puts("mba_MBps_event: Invalid input\n");
+		ret = -EINVAL;
+		goto out_unlock;
+	}
+
+	buf[nbytes - 1] = '\0';
+
 	if (!strcmp(buf, "mbm_local_bytes")) {
 		if (resctrl_is_mon_event_enabled(QOS_L3_MBM_LOCAL_EVENT_ID))
 			rdtgrp->mba_mbps_event = QOS_L3_MBM_LOCAL_EVENT_ID;
@@ -495,6 +503,7 @@ ssize_t rdtgroup_mba_mbps_event_write(struct kernfs_open_file *of,
 	if (ret)
 		rdt_last_cmd_printf("Unsupported event id '%s'\n", buf);
 
+out_unlock:
 	rdtgroup_kn_unlock(of->kn);
 
 	return ret ?: nbytes;
@@ -858,16 +867,18 @@ ssize_t resctrl_io_alloc_write(struct kernfs_open_file *of, char *buf,
 	bool enable;
 	int ret;
 
-	ret = kstrtobool(buf, &enable);
-	if (ret)
-		return ret;
-
 	if (!info_kn_lock(of->kn))
 		return -ENOENT;
 
 	r = s->res;
 	rdt_last_cmd_clear();
 
+	ret = kstrtobool(buf, &enable);
+	if (ret) {
+		rdt_last_cmd_puts("io_alloc: Invalid input\n");
+		goto out_unlock;
+	}
+
 	if (!r->cache.io_alloc_capable) {
 		rdt_last_cmd_printf("io_alloc is not supported on %s\n", s->name);
 		ret = -ENODEV;
@@ -1022,17 +1033,21 @@ ssize_t resctrl_io_alloc_cbm_write(struct kernfs_open_file *of, char *buf,
 	u32 io_alloc_closid;
 	int ret = 0;
 
-	/* Valid input requires a trailing newline */
-	if (nbytes == 0 || buf[nbytes - 1] != '\n')
-		return -EINVAL;
-
-	buf[nbytes - 1] = '\0';
-
 	if (!info_kn_lock(of->kn))
 		return -ENOENT;
 	rdt_last_cmd_clear();
 
 	r = s->res;
+
+	/* Valid input requires a trailing newline */
+	if (nbytes == 0 || buf[nbytes - 1] != '\n') {
+		rdt_last_cmd_puts("io_alloc_cbm: Invalid input\n");
+		ret = -EINVAL;
+		goto out_unlock;
+	}
+
+	buf[nbytes - 1] = '\0';
+
 	if (!r->cache.io_alloc_capable) {
 		rdt_last_cmd_printf("io_alloc is not supported on %s\n", s->name);
 		ret = -ENODEV;
fs/resctrl/monitor.c +36 -24

112 lines not on this page · raw message

fs/resctrl/rdtgroup.c +49 -35

167 lines not on this page · raw message

Signature · 1 line
2.54.0
author maintainer
#17 lore raw

PATCH v5 8/10 fs/resctrl: Add last_cmd_status support for writes to max_threshold_occupancy

In reply to Reinette Chatre

info/last_cmd_status is intended to contain more information if a write to any resctrl file fails. Writes to max_threshold_occupancy did not receive last_cmd_status support during initial last_cmd_status enabling. Add it now.

---
Changes since v1:
- Add Ben's RB tag.

Changes since v2:
- Switch to guard(). (Chenyu)
- Drop Ben's RB tag due to significant code change.

Changes since v4:
- Rebased on top of the "x86,fs/resctrl: Fix long-standing issues" series that added locking helpers to info file handlers.
- Add Tony's Reviewed-by tag.

---
 fs/resctrl/rdtgroup.c | 14 ++++++++++----
 1 file changed, 10 insertions(+), 4 deletions(-)
fs/resctrl/rdtgroup.c +10 -4
index a5fe76b0e9eb..0e428d337641 100644
@@ -1300,14 +1300,20 @@ static ssize_t max_threshold_occ_write(struct kernfs_open_file *of,
 	unsigned int bytes;
 	int ret;
 
-	ret = kstrtouint(buf, 0, &bytes);
-	if (ret)
-		return ret;
-
 	if (!info_kn_lock(of->kn))
 		return -ENOENT;
 
+	rdt_last_cmd_clear();
+
+	ret = kstrtouint(buf, 0, &bytes);
+	if (ret) {
+		rdt_last_cmd_puts("max_threshold_occupancy: Invalid input\n");
+		goto out_unlock;
+	}
+
 	if (bytes > resctrl_rmid_realloc_limit) {
+		rdt_last_cmd_printf("max_threshold_occupancy: Exceeds limit (before adjustment) of %u bytes\n",
+				    resctrl_rmid_realloc_limit);
 		ret = -EINVAL;
 		goto out_unlock;
 	}
Signature · 1 line
2.54.0
author maintainer
#18 lore raw

PATCH v5 9/10 fs/resctrl: Communicate resource group deleted error via last_cmd_status

In reply to Reinette Chatre

User space expects last_cmd_status to contain additional information if any resctrl command fails.

A resctrl command may be blocked waiting on the rdtgroup_mutex waiting for another command to finish and find that once the mutex is available that the resource group has since been deleted. In this scenario the command will fail while last_cmd_status contains either "ok" if the last_cmd_status buffer is empty or an outdated error from a previous command failure if last_cmd_status buffer has content.

Include clearing of last_cmd_status buffer as part of rdtgroup_kn_lock_live() that is used to obtain access and needed locking to a resource group before attempting a command on the group. With the last_cmd_status buffer ready, provide an appropriate message to user space if the resource group has been deleted.

No last_cmd_status treatment is needed for the remaining failure of rdtgroup_kn_lock_live() encountering a non-existent resource group since that could only occur during an attempt to obtain a resource group lock on a file in info/ which is an invalid usage.

---
Changes since v1:
- Add snippet about why it is safe to dereference the kn to get to resource group name when resource group is deleted.

Changes since v4:
- Add Tony's Reviewed-by tag.

---
 fs/resctrl/ctrlmondata.c |  3 ---
 fs/resctrl/monitor.c     |  2 --
 fs/resctrl/rdtgroup.c    | 20 +++++++++++---------
 3 files changed, 11 insertions(+), 14 deletions(-)
fs/resctrl/ctrlmondata.c +0 -3
index 8d5a8b1e659a..18ec9f564b5a 100644
@@ -317,7 +317,6 @@ ssize_t rdtgroup_schemata_write(struct kernfs_open_file *of,
 		rdtgroup_kn_unlock(of->kn);
 		return -ENOENT;
 	}
-	rdt_last_cmd_clear();
 
 	/* Valid input requires a trailing newline */
 	if (nbytes == 0 || buf[nbytes - 1] != '\n') {
@@ -434,7 +433,6 @@ int rdtgroup_schemata_show(struct kernfs_open_file *of,
 			}
 		} else if (rdtgrp->mode == RDT_MODE_PSEUDO_LOCKED) {
 			if (!rdtgrp->plr->d) {
-				rdt_last_cmd_clear();
 				rdt_last_cmd_puts("Cache domain offline\n");
 				ret = -ENODEV;
 			} else {
@@ -475,7 +473,6 @@ ssize_t rdtgroup_mba_mbps_event_write(struct kernfs_open_file *of,
 		rdtgroup_kn_unlock(of->kn);
 		return -ENOENT;
 	}
-	rdt_last_cmd_clear();
 
 	/* Valid input requires a trailing newline */
 	if (nbytes == 0 || buf[nbytes - 1] != '\n') {
fs/resctrl/monitor.c +0 -2
index b50da2b02594..73413cb128ea 100644
@@ -1696,7 +1696,6 @@ int mbm_L3_assignments_show(struct kernfs_open_file *of, struct seq_file *s, voi
 		goto out_unlock;
 	}
 
-	rdt_last_cmd_clear();
 	if (!resctrl_arch_mbm_cntr_assign_enabled(r)) {
 		rdt_last_cmd_puts("mbm_event counter assignment mode is not enabled\n");
 		ret = -EINVAL;
@@ -1836,7 +1835,6 @@ ssize_t mbm_L3_assignments_write(struct kernfs_open_file *of, char *buf,
 		rdtgroup_kn_unlock(of->kn);
 		return -ENOENT;
 	}
-	rdt_last_cmd_clear();
 
 	/* Valid input requires a trailing newline */
 	if (nbytes == 0 || buf[nbytes - 1] != '\n') {
fs/resctrl/rdtgroup.c +11 -9
index 0e428d337641..fa456c469386 100644
@@ -359,7 +359,6 @@ static int rdtgroup_cpus_show(struct kernfs_open_file *of,
 	if (rdtgrp) {
 		if (rdtgrp->mode == RDT_MODE_PSEUDO_LOCKED) {
 			if (!rdtgrp->plr->d) {
-				rdt_last_cmd_clear();
 				rdt_last_cmd_puts("Cache domain offline\n");
 				ret = -ENODEV;
 			} else {
@@ -522,8 +521,6 @@ static ssize_t rdtgroup_cpus_write(struct kernfs_open_file *of,
 		goto out_unlock;
 	}
 
-	rdt_last_cmd_clear();
-
 	if (!buf) {
 		rdt_last_cmd_printf("%s: Invalid input\n",
 				    is_cpu_list(of) ? "cpus_list" : "cpus");
@@ -788,7 +785,6 @@ static ssize_t rdtgroup_tasks_write(struct kernfs_open_file *of,
 		rdtgroup_kn_unlock(of->kn);
 		return -ENOENT;
 	}
-	rdt_last_cmd_clear();
 
 	if (rdtgrp->mode == RDT_MODE_PSEUDO_LOCKED ||
 	    rdtgrp->mode == RDT_MODE_PSEUDO_LOCKSETUP) {
@@ -1530,7 +1526,6 @@ static ssize_t rdtgroup_mode_write(struct kernfs_open_file *of,
 		return -ENOENT;
 	}
 
-	rdt_last_cmd_clear();
 	/* Valid input requires a trailing newline */
 	if (nbytes == 0 || buf[nbytes - 1] != '\n') {
 		rdt_last_cmd_puts("mode: Invalid input\n");
@@ -1667,7 +1662,6 @@ static int rdtgroup_size_show(struct kernfs_open_file *of,
 
 	if (rdtgrp->mode == RDT_MODE_PSEUDO_LOCKED) {
 		if (!rdtgrp->plr->d) {
-			rdt_last_cmd_clear();
 			rdt_last_cmd_puts("Cache domain offline\n");
 			ret = -ENODEV;
 		} else {
@@ -2728,10 +2722,20 @@ struct rdtgroup *rdtgroup_kn_lock_live(struct kernfs_node *kn)
 
 	cpus_read_lock();
 	mutex_lock(&rdtgroup_mutex);
+	rdt_last_cmd_clear();
 
 	/* Was this group deleted while we waited? */
-	if (rdtgrp->flags & RDT_DELETED)
+	if (rdtgrp->flags & RDT_DELETED) {
+		/*
+		 * It is safe to dereference kn to obtain the resource group's
+		 * name because one extra reference to kn is obtained
+		 * during resource group creation that will be released by
+		 * rdtgroup_remove() called by rdtgroup_kn_put().
+		 */
+		rdt_last_cmd_printf("Resource group %s deleted. No commands possible.\n",
+				    rdt_kn_name(rdtgrp->kn));
 		return NULL;
+	}
 
 	return rdtgrp;
 }
@@ -3945,8 +3949,6 @@ static int mkdir_rdt_prepare(struct kernfs_node *parent_kn,
 		goto out_unlock;
 	}
 
-	rdt_last_cmd_clear();
-
 	/*
 	 * Check that the parent directory for a monitor group is a "mon_groups"
 	 * directory.
Signature · 1 line
2.54.0
author maintainer
#19 lore raw

PATCH v5 10/10 fs/resctrl: Inform user space when status buffer overflowed

In reply to Reinette Chatre

resctrl fs commands are becoming more powerful with, for example, a user able to use syntax like '*' to make broad configuration changes. Such commands that span multiple domains may result in more than one message printed to the last_cmd_status buffer with more planned.

Display "[truncated]" to user space when displaying the last_cmd_status buffer to communicate if it overflowed. Upon encountering this user space is expected to combine details about the failure found in info/last_cmd_status with related resctrl files to learn the accurate system state after the command failure.

---
Changes since v2:
- Add Ben's RB tag.

Changes since V4:
- Rebased on "x86,fs/resctrl: Fix long-standing issues" series.
- Add Tony's Reviewed-by tag.

---
 fs/resctrl/rdtgroup.c | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)
fs/resctrl/rdtgroup.c +5 -2
index fa456c469386..efadadfe4ed5 100644
@@ -979,10 +979,13 @@ static int rdt_last_cmd_status_show(struct kernfs_open_file *of,
 	if (!info_kn_lock(of->kn))
 		return -ENOENT;
 	len = seq_buf_used(&last_cmd_status);
-	if (len)
+	if (len) {
 		seq_printf(seq, "%.*s", len, last_cmd_status_buf);
-	else
+		if (seq_buf_has_overflowed(&last_cmd_status))
+			seq_puts(seq, "[truncated]\n");
+	} else {
 		seq_puts(seq, "ok\n");
+	}
 	info_kn_unlock(of->kn);
 	return 0;
 }
Signature · 1 line
2.54.0
author maintainer
#20 lore raw
In reply to Reinette Chatre

Dear x86 maintainers,

Could you please consider this series for inclusion? It applies cleanly on top of x86/cache with HEAD at
2566b5cd6a27 ("fs/resctrl: Fix UAF from worker threads when domains are removed")

There is another resctrl series ("x86,fs/resctrl,arm_mpam: Factor MBA parse-time conversion to be per-arch") also ready for your consideration. There are no conflicts between it and this series, they can be considered in any order.

Thank you very much.

Reinette

[1] https://lore.kernel.org/lkml/[email protected]/

Quoted 133 lines · Reinette Chatre

Applies on top of v7.2-rc1 + v5 of "x86,fs/resctrl: Fix long-standing issues":
https://lore.kernel.org/lkml/[email protected]/

Changes since v4:
- v4: https://lore.kernel.org/lkml/[email protected]/
- Rebased on top of V5 "x86,fs/resctrl: Fix long-standing issues":
https://lore.kernel.org/lkml/[email protected]/
A significant change in that series is the addition of info_kn_lock()/info_kn_unlock() helpers to the info file handlers. With the addition of locking to the max_threshold_occ_write() in that series it seemed appropriate to re-order the last_cmd_status handling in max_threshold_occ_write() to be after the change to last_cmd_status custom to contain max_threshold_occ_write() changes to one patch.
- Add resctrl filename to new error messages to disambiguate the various "Invalid input" messages. (Tony)
- Add Tony's Reviewed-by tag to all patches.

Changes since v3:
- v3: https://lore.kernel.org/lkml/[email protected]/
- Rebased on v7.1-rc1.
- Drop "MAINTAINERS: Update resctrl entry" and "fs/resctrl: Add missing return value descriptions" that can be found in v7.1-rc1.
- Drop "fs/resctrl: Avoid "may be used uninitialized" warning" that just does unnecessary work to appease compiler warning that is disabled by default. (Boris)
- Use pr_warn() instead of WARN_ON() in "x86/resctrl: Protect against bad shift".

Changes since v2:
- v2: https://lore.kernel.org/lkml/[email protected]/
- Rebased on top of tip/master with pending resctrl selftest changes,
https://lore.kernel.org/lkml/[email protected]/,
applied on top to support testing with latest kernel that has pending resctrl changes merged.
- Drop "fs/resctrl: Use stricter checks on input to cpus/cpus_list file" that ended up being too strict and removed one use case. (Chris)
- max_threshold_occ_write() switch to guard(). (Chenyu)
- Add more detail to one return value description.
- Add tags.
- No opinions so far on needing to update last_cmd_status on success of all read commands. At this time last_cmd_status will continue to return previous failure message (if any) after a read command accessing static data succeeds.

Changes since v1:
- v1: https://lore.kernel.org/lkml/[email protected]/
- To simplify tracking, include patch to MAINTAINERS submitted separately:
https://lore.kernel.org/lkml/4274c478922c01f9ceebc805acf991f10a95519f.1771442788.git.reinette.chatre@intel.com/
- Follow recent upstream changes to add reference to tip tree handbook in MAINTAINERS entry.
- Use new pattern in resctrl to enable looping over all entries of an enum while making adding new enum entries more clear all while avoiding warnings when compiling with -Wswitch. (Ben)
- Add another last_cmd_status enhancement that appends "[truncated]" to output of info/last_cmd_status if the backing buffer overflowed.
- Please see patch changelogs for details of changes.
- Add tags.

Hi Everybody,

This is a collection of resctrl cleanups assembled together for convenience and simpler tracking. I'd be happy to split them up if it makes review and/or handling easier.

Summary of changes:

- Let resctrl pass stricter checks from various tools to provide a cleaner baseline with the goal to promote healthier contributions:
- ./tools/docs/kernel-doc -Wall -v <files>
- Build with W=12
- ./scripts/coccicheck
- Static checkers

- Use accurate and consistent type for all uses of resource ID.

- In the unlikely scenario that resctrl picked a wrong CPU to read an event from, pass the error through to user space instead of claiming to succeed and returning a (wrong) result.

- Since inception of last_cmd_status feature there have been mismatches between resctrl file operation failures and the contents of info/last_cmd_status. This pattern keeps propagating with each new resctrl feature. Establish a new baseline with a new pattern that ensures info/last_cmd_status contains an accurate failure description that matches the most recent resctrl file operation failure.

One potential open:

There remains an inconsistency between resctrl file operations that do/can _not_ fail and the contents of info/last_cmd_status. If a resctrl file operation fails and an informational error is printed to last_cmd_status then a subsequent reading of a resctrl file (specifically most of the files found in info/) may succeed while info/last_cmd_status may or may not return the error from previous failure.

Ensuring last_cmd_status is reset on every read carries the cost of taking rdtgroup_mutex on several more user space initiated paths and thus increase contention on rdtgroup_mutex. I opted to not make this change and instead focus this work on ensuring that last_cmd_status is accurate whenever there is a failure during any resctrl file operation. Please let me know if you have opinions in this regard.

Update v5: The addition of the info_kn_lock()/info_kn_unlock() helpers to info file handlers now means that rdtgroup_mutex is held every time an info file is opened. There is thus now less burden to reset last_cmd_status buffer on each info file access. Even so, since there has not been any opinions in this regard I opted to keep current behavior.

Any feedback is appreciated.

Regards,

Reinette

Reinette Chatre (10):
fs/resctrl: Use correct format specifier for printing error pointers x86/resctrl: Protect against bad shift
fs/resctrl: Change pattern used to track number of entries in enum fs/resctrl: Use accurate type for rdt_resource::rid fs/resctrl: Pass error reading event through to user space fs/resctrl: Use accurate and symmetric exit flows fs/resctrl: Change last_cmd_status custom during input parsing fs/resctrl: Add last_cmd_status support for writes to
max_threshold_occupancy
fs/resctrl: Communicate resource group deleted error via
last_cmd_status
fs/resctrl: Inform user space when status buffer overflowed

 arch/x86/kernel/cpu/resctrl/core.c |   6 +-
 fs/resctrl/ctrlmondata.c           |  71 ++++++++++-------
 fs/resctrl/monitor.c               |  78 ++++++++++--------
 fs/resctrl/rdtgroup.c              | 123 +++++++++++++++++------------
 include/linux/resctrl.h            |  11 +--
 5 files changed, 173 insertions(+), 116 deletions(-)
maintainer
#21 lore raw
Quoted 3 lines · Reinette Chatre

Dear x86 maintainers,

Could you please consider this series for inclusion?

/me looks.

Signature · 4 lines
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette
Tested-by Babu Moger on the v5 series
#22 lore raw
In reply to Reinette Chatre

Ran few tests on the series. Looks good.

Quoted 133 lines · Reinette Chatre

Applies on top of v7.2-rc1 + v5 of "x86,fs/resctrl: Fix long-standing issues":

   https://lore.kernel.org/lkml/[email protected]/

Changes since v4:
- v4: https://lore.kernel.org/lkml/[email protected]/
- Rebased on top of V5 "x86,fs/resctrl: Fix long-standing issues":
https://lore.kernel.org/lkml/[email protected]/
A significant change in that series is the addition of info_kn_lock()/info_kn_unlock() helpers to the info file handlers. With the addition of locking to the max_threshold_occ_write() in that series it seemed appropriate to re-order the last_cmd_status handling in max_threshold_occ_write() to be after the change to last_cmd_status custom to contain max_threshold_occ_write() changes to one patch.
- Add resctrl filename to new error messages to disambiguate the various "Invalid input" messages. (Tony)
- Add Tony's Reviewed-by tag to all patches.

Changes since v3:
- v3: https://lore.kernel.org/lkml/[email protected]/
- Rebased on v7.1-rc1.
- Drop "MAINTAINERS: Update resctrl entry" and "fs/resctrl: Add missing return value descriptions" that can be found in v7.1-rc1.
- Drop "fs/resctrl: Avoid "may be used uninitialized" warning" that just does unnecessary work to appease compiler warning that is disabled by default. (Boris)
- Use pr_warn() instead of WARN_ON() in "x86/resctrl: Protect against bad shift".

Changes since v2:
- v2: https://lore.kernel.org/lkml/[email protected]/
- Rebased on top of tip/master with pending resctrl selftest changes,
https://lore.kernel.org/lkml/[email protected]/,
applied on top to support testing with latest kernel that has pending resctrl changes merged.
- Drop "fs/resctrl: Use stricter checks on input to cpus/cpus_list file" that ended up being too strict and removed one use case. (Chris)
- max_threshold_occ_write() switch to guard(). (Chenyu)
- Add more detail to one return value description.
- Add tags.
- No opinions so far on needing to update last_cmd_status on success of all read commands. At this time last_cmd_status will continue to return previous failure message (if any) after a read command accessing static data succeeds.

Changes since v1:
- v1: https://lore.kernel.org/lkml/[email protected]/
- To simplify tracking, include patch to MAINTAINERS submitted separately:

   https://lore.kernel.org/lkml/4274c478922c01f9ceebc805acf991f10a95519f.1771442788.git.reinette.chatre@intel.com/

- Follow recent upstream changes to add reference to tip tree handbook in MAINTAINERS entry.
- Use new pattern in resctrl to enable looping over all entries of an enum while making adding new enum entries more clear all while avoiding warnings when compiling with -Wswitch. (Ben)
- Add another last_cmd_status enhancement that appends "[truncated]" to output of info/last_cmd_status if the backing buffer overflowed.
- Please see patch changelogs for details of changes.
- Add tags.

Hi Everybody,

This is a collection of resctrl cleanups assembled together for convenience and simpler tracking. I'd be happy to split them up if it makes review and/or handling easier.

Summary of changes:

- Let resctrl pass stricter checks from various tools to provide a cleaner baseline with the goal to promote healthier contributions:
- ./tools/docs/kernel-doc -Wall -v <files>
- Build with W=12
- ./scripts/coccicheck
- Static checkers

- Use accurate and consistent type for all uses of resource ID.

- In the unlikely scenario that resctrl picked a wrong CPU to read an event from, pass the error through to user space instead of claiming to succeed and returning a (wrong) result.

- Since inception of last_cmd_status feature there have been mismatches between resctrl file operation failures and the contents of info/last_cmd_status. This pattern keeps propagating with each new resctrl feature. Establish a new baseline with a new pattern that ensures info/last_cmd_status contains an accurate failure description that matches the most recent resctrl file operation failure.

One potential open:

There remains an inconsistency between resctrl file operations that do/can _not_ fail and the contents of info/last_cmd_status. If a resctrl file operation fails and an informational error is printed to last_cmd_status then a subsequent reading of a resctrl file (specifically most of the files found in info/) may succeed while info/last_cmd_status may or may not return the error from previous failure.

Ensuring last_cmd_status is reset on every read carries the cost of taking rdtgroup_mutex on several more user space initiated paths and thus increase contention on rdtgroup_mutex. I opted to not make this change and instead focus this work on ensuring that last_cmd_status is accurate whenever there is a failure during any resctrl file operation. Please let me know if you have opinions in this regard.

Update v5: The addition of the info_kn_lock()/info_kn_unlock() helpers to info file handlers now means that rdtgroup_mutex is held every time an info file is opened. There is thus now less burden to reset last_cmd_status buffer on each info file access. Even so, since there has not been any opinions in this regard I opted to keep current behavior.

Any feedback is appreciated.

Regards,

Reinette

Reinette Chatre (10):
fs/resctrl: Use correct format specifier for printing error pointers x86/resctrl: Protect against bad shift
fs/resctrl: Change pattern used to track number of entries in enum fs/resctrl: Use accurate type for rdt_resource::rid fs/resctrl: Pass error reading event through to user space fs/resctrl: Use accurate and symmetric exit flows fs/resctrl: Change last_cmd_status custom during input parsing fs/resctrl: Add last_cmd_status support for writes to
max_threshold_occupancy
fs/resctrl: Communicate resource group deleted error via
last_cmd_status
fs/resctrl: Inform user space when status buffer overflowed

  arch/x86/kernel/cpu/resctrl/core.c |   6 +-
  fs/resctrl/ctrlmondata.c           |  71 ++++++++++-------
  fs/resctrl/monitor.c               |  78 ++++++++++--------
  fs/resctrl/rdtgroup.c              | 123 +++++++++++++++++------------
  include/linux/resctrl.h            |  11 +--
  5 files changed, 173 insertions(+), 116 deletions(-)