Skip to contentSkip to conversation

x86/resctrl: Don't try to free nonexistent RMIDs

PATCH Superseded Dave Martin 2 participants 9 messages Jun 14, 2024 – Jun 19, 2024

Subsystems (2)

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore

Dave Martin

author reviewer
#0 lore raw

PATCH 1/1 x86/resctrl: Don't try to free nonexistent RMIDs

Commit 6791e0ea3071 ("x86/resctrl: Access per-rmid structures by index") adds logic to map individual monitoring groups into a global index space used for tracking allocated RMIDs.

That patch keept the logic to ignore requests to free the default RMID in free_rmid(), and this works fine on x86.

With arm64 MPAM, there is a latent bug here: on platforms with no monitors exposed through resctrl, each control group still gets a different monitoring group ID as seen by the hardware, since the CLOSID always forms part of the monitoring group ID. This means that when removing a control group, the code may try to free this group's default monitoring group RMID for real. If there are no monitors however, the RMID tracking table rmid_ptrs[] would be a waste of memory and is never allocated, leading to a splat when a free_rmid() tries to dereference the table.

One option would be to treat RMID 0 as special for every CLOSID, but this would be ugly since we still want to do bookkeeping for these monitoring group IDs when there are monitors present in the hardware.

Instead, add a gating check of resctrl_arch_mon_capable() in free_rmid(), and just do nothing if the hardware doesn't have monitors.

This fix mirrors the gating checks already present in mkdir_rdt_prepare_rmid_alloc() and elsewhere.

No functional change on x86.

Fixes 6791e0ea3071 ("x86/resctrl: Access per-rmid structures by index")

---

Based on v6.10-rc3.

Tested on x86 (But so far for the monitors-present case. Testing on Atom would be appreciated.)

Tested on arm64 for the no-monitors case.

---
 arch/x86/kernel/cpu/resctrl/monitor.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
arch/x86/kernel/cpu/resctrl/monitor.c +2 -1
index 2345e6836593..366f496ca3ce 100644
@@ -519,7 +519,8 @@ void free_rmid(u32 closid, u32 rmid)
 	 * allows architectures that ignore the closid parameter to avoid an
 	 * unnecessary check.
 	 */
-	if (idx == resctrl_arch_rmid_idx_encode(RESCTRL_RESERVED_CLOSID,
+	if (!resctrl_arch_mon_capable() ||
+	    idx == resctrl_arch_rmid_idx_encode(RESCTRL_RESERVED_CLOSID,
 						RESCTRL_RESERVED_RMID))
 		return;
 

base-commit: 83a7eefedc9b56fe7bfeff13b6c7356688ffa670

Signature · 1 line
2.34.1
maintainer
#1 lore raw

Hi Dave,

Quoted 5 lines · Dave Martin

Commit 6791e0ea3071 ("x86/resctrl: Access per-rmid structures by index") adds logic to map individual monitoring groups into a global index space used for tracking allocated RMIDs.

That patch keept the logic to ignore requests to free the default

keept -> kept

nitpick: I actually do not know if "that patch" gets same hate as "this patch" so to avoid any potential feedback about this I'd like to suggest that this is rewritten without this term. Perhaps something like: "Requests to free the default RMID in free_rmid() are ignored, and this works fine on x86."

Quoted 26 lines

RMID in free_rmid(), and this works fine on x86.

With arm64 MPAM, there is a latent bug here: on platforms with no monitors exposed through resctrl, each control group still gets a different monitoring group ID as seen by the hardware, since the CLOSID always forms part of the monitoring group ID. This means that when removing a control group, the code may try to free this group's default monitoring group RMID for real. If there are no monitors however, the RMID tracking table rmid_ptrs[] would be a waste of memory and is never allocated, leading to a splat when a free_rmid() tries to dereference the table.

One option would be to treat RMID 0 as special for every CLOSID, but this would be ugly since we still want to do bookkeeping for these monitoring group IDs when there are monitors present in the hardware.

Instead, add a gating check of resctrl_arch_mon_capable() in free_rmid(), and just do nothing if the hardware doesn't have monitors.

This fix mirrors the gating checks already present in mkdir_rdt_prepare_rmid_alloc() and elsewhere.

No functional change on x86.

Quoted 8 lines
Fixes 6791e0ea3071 ("x86/resctrl: Access per-rmid structures by index")

---

Based on v6.10-rc3.

Tested on x86 (But so far for the monitors-present case.

Tested by booting with "rdt=!cmt,!mbmtotal,!mbmlocal".

Quoted 13 lines

Testing on Atom would be appreciated.)

Tested on arm64 for the no-monitors case.

---
  arch/x86/kernel/cpu/resctrl/monitor.c | 3 ++-
  1 file changed, 2 insertions(+), 1 deletion(-)
arch/x86/kernel/cpu/resctrl/monitor.c +0 -0
index 2345e6836593..366f496ca3ce 100644

arch/x86/kernel/cpu/resctrl/monitor.c void free_rmid(u32 closid, u32 rmid) · line 526

Quoted 9 lines
arch/x86/kernel/cpu/resctrl/monitor.c +2 -1
@@ -519,7 +519,8 @@ void free_rmid(u32 closid, u32 rmid)
  	 * allows architectures that ignore the closid parameter to avoid an
  	 * unnecessary check.
  	 */
-	if (idx == resctrl_arch_rmid_idx_encode(RESCTRL_RESERVED_CLOSID,
+	if (!resctrl_arch_mon_capable() ||
+	    idx == resctrl_arch_rmid_idx_encode(RESCTRL_RESERVED_CLOSID,
  						RESCTRL_RESERVED_RMID))
  		return;
 

Thank you for catching this. This reveals how subtle resctrl is by calling free_rmid() in paths for convenience and then relying on an uninitialized variable for correct behavior.

This replaces this subtle assumption with an actual check.

With changelog comments addressed:
| Reviewed-by: Reinette Chatre <[email protected]>

Reinette

author reviewer
#2 lore raw

Hi Reinette,

Quoted 9 lines · Reinette Chatre

Hi Dave,

Quoted 5 lines · Dave Martin

Commit 6791e0ea3071 ("x86/resctrl: Access per-rmid structures by index") adds logic to map individual monitoring groups into a global index space used for tracking allocated RMIDs.

That patch keept the logic to ignore requests to free the default

Quoted 10 lines

keept -> kept

nitpick: I actually do not know if "that patch" gets same hate as "this patch" so to avoid any potential feedback about this I'd like to suggest that this is rewritten without this term. Perhaps something like: "Requests to free the default RMID in free_rmid() are ignored, and this works fine on x86."

Quoted 2 lines

RMID in free_rmid(), and this works fine on x86.

How about recasting the first paragraph into the past tense (since it relates a past commit), and rewording as "Requests to free the default RMID continued to be ignored in free_rmid(), and this works fine on x86."

(I agree that "this patch" would have been ambiguous. "That patch" was an attempt to be clearer, but felt a bit clumsy. Naming the commit again felt worse, though would have been clearer. I've noticed that people who do not have English as their first language tend to use "this" and "that" a little differently from native English speakers, so there is probably more scope for confusion here that I like to assume...)

Quoted 27 lines
Quoted 27 lines

With arm64 MPAM, there is a latent bug here: on platforms with no monitors exposed through resctrl, each control group still gets a different monitoring group ID as seen by the hardware, since the CLOSID always forms part of the monitoring group ID. This means that when removing a control group, the code may try to free this group's default monitoring group RMID for real. If there are no monitors however, the RMID tracking table rmid_ptrs[] would be a waste of memory and is never allocated, leading to a splat when a free_rmid() tries to dereference the table.

One option would be to treat RMID 0 as special for every CLOSID, but this would be ugly since we still want to do bookkeeping for these monitoring group IDs when there are monitors present in the hardware.

Instead, add a gating check of resctrl_arch_mon_capable() in free_rmid(), and just do nothing if the hardware doesn't have monitors.

This fix mirrors the gating checks already present in mkdir_rdt_prepare_rmid_alloc() and elsewhere.

No functional change on x86.

Fixes 6791e0ea3071 ("x86/resctrl: Access per-rmid structures by index")
Quoted 7 lines
Quoted 5 lines

---

Based on v6.10-rc3.

Tested on x86 (But so far for the monitors-present case.

Tested by booting with "rdt=!cmt,!mbmtotal,!mbmlocal".

Thanks (I take it that's your test, not a request to be more specific about mine?)

As it happens I tested with rdt=cmt,mbmtotal,mbmlocal,l3cat,l3cdp (though I made no effort to exercise these features other than running the selftests). I can note this in the commit if you prefer.

Quoted 27 lines
Quoted 13 lines

Testing on Atom would be appreciated.)

Tested on arm64 for the no-monitors case.

---
  arch/x86/kernel/cpu/resctrl/monitor.c | 3 ++-
  1 file changed, 2 insertions(+), 1 deletion(-)
arch/x86/kernel/cpu/resctrl/monitor.c +0 -0
index 2345e6836593..366f496ca3ce 100644

arch/x86/kernel/cpu/resctrl/monitor.c void free_rmid(u32 closid, u32 rmid) · line 526

Quoted 9 lines
arch/x86/kernel/cpu/resctrl/monitor.c +2 -1
@@ -519,7 +519,8 @@ void free_rmid(u32 closid, u32 rmid)
  	 * allows architectures that ignore the closid parameter to avoid an
  	 * unnecessary check.
  	 */
-	if (idx == resctrl_arch_rmid_idx_encode(RESCTRL_RESERVED_CLOSID,
+	if (!resctrl_arch_mon_capable() ||
+	    idx == resctrl_arch_rmid_idx_encode(RESCTRL_RESERVED_CLOSID,
  						RESCTRL_RESERVED_RMID))
  		return;
 

Thank you for catching this. This reveals how subtle resctrl is by calling free_rmid() in paths for convenience and then relying on an uninitialized variable for correct behavior.

Quoted 6 lines

This replaces this subtle assumption with an actual check.

With changelog comments addressed:
| Reviewed-by: Reinette Chatre <[email protected]>

Reinette

Thanks for that. I'll wait for your reply before respinning this.

Cheers
---Dave

maintainer Tested-by
#3 lore raw

Hi Dave,

Quoted 20 lines · Dave Martin

Hi Reinette,

Quoted 17 lines · Reinette Chatre

Hi Dave,

Quoted 5 lines · Dave Martin

Commit 6791e0ea3071 ("x86/resctrl: Access per-rmid structures by index") adds logic to map individual monitoring groups into a global index space used for tracking allocated RMIDs.

That patch keept the logic to ignore requests to free the default

keept -> kept

nitpick: I actually do not know if "that patch" gets same hate as "this patch" so to avoid any potential feedback about this I'd like to suggest that this is rewritten without this term. Perhaps something like: "Requests to free the default RMID in free_rmid() are ignored, and this works fine on x86."

Quoted 7 lines
Quoted 2 lines
Quoted 2 lines

RMID in free_rmid(), and this works fine on x86.

How about recasting the first paragraph into the past tense (since it relates a past commit), and rewording as "Requests to free the default RMID continued to be ignored in free_rmid(), and this works fine on x86."

Please keep it in the present tense. I do not see this as relating to a "past commit" but instead it is an existing commit responsible for current behavior. Documentation/process/maintainer-tip.rst contains some example changelogs created by x86 maintainers that captures their requirements. The beginning "context" portion is always in present tense.

Quoted 38 lines

(I agree that "this patch" would have been ambiguous. "That patch" was an attempt to be clearer, but felt a bit clumsy. Naming the commit again felt worse, though would have been clearer. I've noticed that people who do not have English as their first language tend to use "this" and "that" a little differently from native English speakers, so there is probably more scope for confusion here that I like to assume...)

Quoted 29 lines
Quoted 29 lines

With arm64 MPAM, there is a latent bug here: on platforms with no monitors exposed through resctrl, each control group still gets a different monitoring group ID as seen by the hardware, since the CLOSID always forms part of the monitoring group ID. This means that when removing a control group, the code may try to free this group's default monitoring group RMID for real. If there are no monitors however, the RMID tracking table rmid_ptrs[] would be a waste of memory and is never allocated, leading to a splat when a free_rmid() tries to dereference the table.

One option would be to treat RMID 0 as special for every CLOSID, but this would be ugly since we still want to do bookkeeping for these monitoring group IDs when there are monitors present in the hardware.

Instead, add a gating check of resctrl_arch_mon_capable() in free_rmid(), and just do nothing if the hardware doesn't have monitors.

This fix mirrors the gating checks already present in mkdir_rdt_prepare_rmid_alloc() and elsewhere.

No functional change on x86.

Fixes 6791e0ea3071 ("x86/resctrl: Access per-rmid structures by index")

---

Quoted 8 lines
Quoted 5 lines
Quoted 3 lines

Based on v6.10-rc3.

Tested on x86 (But so far for the monitors-present case.

Tested by booting with "rdt=!cmt,!mbmtotal,!mbmlocal".

Thanks (I take it that's your test, not a request to be more specific about mine?)

Yes, I did test it with those parameters. You are also welcome to add:

Quoted 4 lines

As it happens I tested with rdt=cmt,mbmtotal,mbmlocal,l3cat,l3cdp (though I made no effort to exercise these features other than running the selftests). I can note this in the commit if you prefer.

hmmm ... those parameters should not be necessary unless the system has those features forced off by default because of errata. Doing functional testing on these systems via such enabling is fine though.

Reinette

author reviewer
#4 lore raw

Hi,

Quoted 31 lines · Reinette Chatre

Hi Dave,

Quoted 20 lines · Dave Martin

Hi Reinette,

Quoted 17 lines · Reinette Chatre

Hi Dave,

Quoted 5 lines · Dave Martin

Commit 6791e0ea3071 ("x86/resctrl: Access per-rmid structures by index") adds logic to map individual monitoring groups into a global index space used for tracking allocated RMIDs.

That patch keept the logic to ignore requests to free the default

keept -> kept

nitpick: I actually do not know if "that patch" gets same hate as "this patch" so to avoid any potential feedback about this I'd like to suggest that this is rewritten without this term. Perhaps something like: "Requests to free the default RMID in free_rmid() are ignored, and this works fine on x86."

Quoted 7 lines
Quoted 2 lines
Quoted 2 lines

RMID in free_rmid(), and this works fine on x86.

How about recasting the first paragraph into the past tense (since it relates a past commit), and rewording as "Requests to free the default RMID continued to be ignored in free_rmid(), and this works fine on x86."

Quoted 6 lines

Please keep it in the present tense. I do not see this as relating to a "past commit" but instead it is an existing commit responsible for current behavior. Documentation/process/maintainer-tip.rst contains some example changelogs created by x86 maintainers that captures their requirements. The beginning "context" portion is always in present tense.

Fair enough. I'll reword this as per your suggestion.

[...]

Quoted 10 lines
Quoted 6 lines
Quoted 3 lines
Quoted 1 line

Tested on x86 (But so far for the monitors-present case.

Tested by booting with "rdt=!cmt,!mbmtotal,!mbmlocal".

Thanks (I take it that's your test, not a request to be more specific about mine?)

Yes, I did test it with those parameters. You are also welcome to add:

Ah, right. Thanks.

Quoted 10 lines
Quoted 3 lines

As it happens I tested with rdt=cmt,mbmtotal,mbmlocal,l3cat,l3cdp (though I made no effort to exercise these features other than running the selftests). I can note this in the commit if you prefer.

hmmm ... those parameters should not be necessary unless the system has those features forced off by default because of errata. Doing functional testing on these systems via such enabling is fine though.

Reinette

I got these kernel args from James. Apparently some of these features are defaulted to off by the firmware on the box I'm using, but we're not sure why. I haven't dug into it.

Cheers
---Dave

maintainer
#5 lore raw

Hi Dave,

Quoted 12 lines · Dave Martin
Quoted 11 lines · Reinette Chatre
Quoted 3 lines · Dave Martin

As it happens I tested with rdt=cmt,mbmtotal,mbmlocal,l3cat,l3cdp (though I made no effort to exercise these features other than running the selftests). I can note this in the commit if you prefer.

hmmm ... those parameters should not be necessary unless the system has those features forced off by default because of errata. Doing functional testing on these systems via such enabling is fine though.

Quoted 5 lines
Quoted 1 line

Reinette

I got these kernel args from James. Apparently some of these features are defaulted to off by the firmware on the box I'm using, but we're not sure why. I haven't dug into it.

resctrl does not provide capability to override features that the hardware does not advertise as supporting. resctrl does sometimes force hardware advertised features off because of errata (for example, see [1]) and the kernel "rdt=" parameter can be used to override that. This same option also enables administrators to ignore some features on their platforms.

Reinette

[1] https://lore.kernel.org/all/3aea0a3bae219062c812668bd9b7b8f1a25003ba.1503512900.git.tony.luck@intel.com/

author reviewer
#6 lore raw

Hi Reinette,

Quoted 27 lines · Reinette Chatre

Hi Dave,

Quoted 12 lines · Dave Martin
Quoted 11 lines · Reinette Chatre
Quoted 3 lines · Dave Martin

As it happens I tested with rdt=cmt,mbmtotal,mbmlocal,l3cat,l3cdp (though I made no effort to exercise these features other than running the selftests). I can note this in the commit if you prefer.

hmmm ... those parameters should not be necessary unless the system has those features forced off by default because of errata. Doing functional testing on these systems via such enabling is fine though.

Quoted 5 lines
Quoted 1 line

Reinette

I got these kernel args from James. Apparently some of these features are defaulted to off by the firmware on the box I'm using, but we're not sure why. I haven't dug into it.

resctrl does not provide capability to override features that the hardware does not advertise as supporting. resctrl does sometimes force hardware advertised features off because of errata (for example, see [1]) and the kernel "rdt=" parameter can be used to override that. This same option also enables administrators to ignore some features on their platforms.

Quoted 3 lines

Reinette

[1] https://lore.kernel.org/all/3aea0a3bae219062c812668bd9b7b8f1a25003ba.1503512900.git.tony.luck@intel.com/

Possibly the box in question falls under this case. I'll need to check with James and/or experiment.

Either way, I don't think this impacts the tested-ness of this patch, but please shout if you have concerns.

Cheers
---Dave

maintainer
#7 lore raw

Hi Dave,

Quoted 4 lines · Dave Martin

Either way, I don't think this impacts the tested-ness of this patch, but please shout if you have concerns.

I agree. The submission stated that it was tested on x86 "for the monitors-present case". When booting with "rdt=!cmt,!mbmtotal,!mbmlocal" (as I did in my test) resctrl is not able to discover monitoring support and it thus tested the "monitors _not_ present case".

Reinette

author reviewer
#8 lore raw

Hi,

Quoted 9 lines · Reinette Chatre

Hi Dave,

Quoted 4 lines · Dave Martin

Either way, I don't think this impacts the tested-ness of this patch, but please shout if you have concerns.

Quoted 7 lines

I agree. The submission stated that it was tested on x86 "for the monitors-present case". When booting with "rdt=!cmt,!mbmtotal,!mbmlocal" (as I did in my test) resctrl is not able to discover monitoring support and it thus tested the "monitors _not_ present case".

Reinette

Ah, right. I missed the significance of the options.

Thanks again!
---Dave

Quoted 2 lines