Skip to content

x86/resctrl: Merge the CDP resources

James Morse lkml stable 7 versions – Applied

Status from this message: The following commit has been merged into the x86/cache branch of tip:

Get series with b4
b4 am [email protected]

Versions newest first · linked by author and title

  1. v7 Latest24 patches 2 repliesApplied
  2. v624 patches 1 replySuperseded
  3. v524 patches 2 repliesSuperseded
Earlier versions · 4
VersionPostedPatchesRepliesStatus
v424 patches18 repliesSuperseded
v324 patches2 repliesSuperseded
v224 patches17 repliesSuperseded
v124 patches27 repliesSuperseded

Patches in v7 Applied

#Subject and filesSizeRepliesApplied
1/24x86/resctrl: Split struct rdt_resourcearch/x86/kernel/cpu/resctrl/core.c arch/x86/kernel/cpu/resctrl/ctrlmondata.c arch/x86/kernel/cpu/resctrl/internal.h +4 more+362 -2720 replies
2/24x86/resctrl: Split struct rdt_domainarch/x86/kernel/cpu/resctrl/core.c arch/x86/kernel/cpu/resctrl/ctrlmondata.c arch/x86/kernel/cpu/resctrl/internal.h +3 more+94 -600 replies
3/24x86/resctrl: Add a separate schema list for resctrlarch/x86/kernel/cpu/resctrl/internal.h arch/x86/kernel/cpu/resctrl/rdtgroup.c include/linux/resctrl.h+54 -10 replies
4/24x86/resctrl: Pass the schema in info dir's private pointerarch/x86/kernel/cpu/resctrl/rdtgroup.c+25 -130 replies
5/24x86/resctrl: Label the resources with their configuration typearch/x86/kernel/cpu/resctrl/core.c arch/x86/kernel/cpu/resctrl/internal.h arch/x86/kernel/cpu/resctrl/rdtgroup.c +1 more+24 -00 replies
6/24x86/resctrl: Walk the resctrl schema list instead of an arch listarch/x86/kernel/cpu/resctrl/ctrlmondata.c arch/x86/kernel/cpu/resctrl/rdtgroup.c+27 -140 replies
7/24x86/resctrl: Store the effective num_closid in the schemaarch/x86/kernel/cpu/resctrl/ctrlmondata.c arch/x86/kernel/cpu/resctrl/rdtgroup.c include/linux/resctrl.h+11 -150 replies
8/24x86/resctrl: Add resctrl_arch_get_num_closid()arch/x86/kernel/cpu/resctrl/core.c arch/x86/kernel/cpu/resctrl/internal.h arch/x86/kernel/cpu/resctrl/rdtgroup.c +1 more+18 -50 replies
9/24x86/resctrl: Pass the schema to resctrl filesystem functionsarch/x86/kernel/cpu/resctrl/ctrlmondata.c arch/x86/kernel/cpu/resctrl/internal.h arch/x86/kernel/cpu/resctrl/rdtgroup.c +1 more+29 -220 replies
10/24x86/resctrl: Swizzle rdt_resource and resctrl_schema in pseudo_lock_regionarch/x86/kernel/cpu/resctrl/ctrlmondata.c arch/x86/kernel/cpu/resctrl/internal.h arch/x86/kernel/cpu/resctrl/pseudo_lock.c +1 more+11 -110 replies
11/24x86/resctrl: Add a helper to read/set the CDP configurationarch/x86/kernel/cpu/resctrl/core.c arch/x86/kernel/cpu/resctrl/internal.h arch/x86/kernel/cpu/resctrl/pseudo_lock.c +2 more+62 -350 replies
12/24x86/resctrl: Move the schemata names into struct resctrl_schemaarch/x86/kernel/cpu/resctrl/core.c arch/x86/kernel/cpu/resctrl/ctrlmondata.c arch/x86/kernel/cpu/resctrl/rdtgroup.c +1 more+30 -160 replies
13/24x86/resctrl: Group staged configuration into a separate structarch/x86/kernel/cpu/resctrl/ctrlmondata.c arch/x86/kernel/cpu/resctrl/rdtgroup.c include/linux/resctrl.h+54 -270 replies
14/24x86/resctrl: Allow different CODE/DATA configurations to be stagedarch/x86/kernel/cpu/resctrl/ctrlmondata.c arch/x86/kernel/cpu/resctrl/rdtgroup.c include/linux/resctrl.h+20 -90 replies
15/24x86/resctrl: Rename update_domains() resctrl_arch_update_domains()arch/x86/kernel/cpu/resctrl/ctrlmondata.c arch/x86/kernel/cpu/resctrl/internal.h arch/x86/kernel/cpu/resctrl/rdtgroup.c +1 more+4 -40 replies
16/24x86/resctrl: Add a helper to read a closid's configurationarch/x86/kernel/cpu/resctrl/ctrlmondata.c arch/x86/kernel/cpu/resctrl/monitor.c arch/x86/kernel/cpu/resctrl/rdtgroup.c +1 more+37 -301 reply
17/24x86/resctrl: Pass configuration type to resctrl_arch_get_config()arch/x86/kernel/cpu/resctrl/ctrlmondata.c arch/x86/kernel/cpu/resctrl/monitor.c arch/x86/kernel/cpu/resctrl/rdtgroup.c +1 more+29 -160 replies
18/24x86/resctrl: Make ctrlval arrays the same sizearch/x86/kernel/cpu/resctrl/core.c arch/x86/kernel/cpu/resctrl/rdtgroup.c+18 -10 replies
19/24x86/resctrl: Apply offset correction when config is stagedarch/x86/kernel/cpu/resctrl/core.c arch/x86/kernel/cpu/resctrl/ctrlmondata.c arch/x86/kernel/cpu/resctrl/internal.h +1 more+27 -310 replies
20/24x86/resctrl: Calculate the index from the configuration typearch/x86/kernel/cpu/resctrl/core.c arch/x86/kernel/cpu/resctrl/ctrlmondata.c include/linux/resctrl.h+15 -280 replies
21/24x86/resctrl: Merge the ctrl_val arraysarch/x86/kernel/cpu/resctrl/core.c+61 -40 replies
22/24x86/resctrl: Remove rdt_cdp_peer_get()arch/x86/kernel/cpu/resctrl/rdtgroup.c+14 -850 replies
23/24x86/resctrl: Expand resctrl_arch_update_domains()'s msr_param rangearch/x86/kernel/cpu/resctrl/ctrlmondata.c+9 -30 replies
24/24x86/resctrl: Merge the CDP resourcesarch/x86/kernel/cpu/resctrl/core.c arch/x86/kernel/cpu/resctrl/internal.h arch/x86/kernel/cpu/resctrl/rdtgroup.c+85 -2390 replies

Reviews trailers per patch

All 24 patches have a Reviewed-by or Acked-by of their own; the series has one for all of them.

PatchJamie IlesReinette ChatreBabu Moger
SeriesA-b A-b
1/24R-b R-b T-b
2/24R-b R-b T-b
3/24R-b R-b T-b
4/24R-b R-b T-b
5/24R-b R-b T-b
6/24R-b R-b T-b
7/24R-b R-b T-b
8/24R-b R-b T-b
9/24R-b R-b T-b
10/24R-b R-b T-b
11/24R-b R-b T-b
12/24R-b R-b T-b
13/24R-b R-b T-b
14/24R-b R-b T-b
15/24R-b R-b T-b
16/24R-b R-b T-b
17/24R-b R-b T-b
18/24R-b R-b T-b
19/24R-b R-b T-b
20/24R-b R-b T-b
21/24R-b R-b T-b
22/24R-b R-b T-b
23/24R-b T-b
24/24R-b R-b T-b

Trailers given in replies are filled; those carried in the patches from earlier versions are outlined with dashes. Scroll the table sideways for all 3 reviewers.

Read the v7 thread

Cover letter [PATCH v7 00/24] x86/resctrl: Merge the CDP resources

Hi folks,

Changes since v6? Fixed some undefined behaviour warning in the first patch.
---

This series re-folds the resctrl code so the CDP resources (L3CODE et al) behaviour is all contained in the filesystem parts, with a minimum amount of arch specific code.

Arm have some CPU support for dividing caches into portions, and applying bandwidth limits at various points in the SoC. The collective term for these features is MPAM: Memory Partitioning and Monitoring.

MPAM is similar enough to Intel RDT, that it should use the defacto linux
interface: resctrl. This filesystem currently lives under arch/x86, and is tightly coupled to the architecture.
Ultimately, my plan is to split the existing resctrl code up to have an arch<->fs abstraction, then move all the bits out to fs/resctrl. From there MPAM can be wired up.

x86 might have two resources with cache controls, (L2 and L3) but has extra copies for CDP: L{2,3}{CODE,DATA}, which are marked as enabled if CDP is enabled for the corresponding cache.

MPAM has an equivalent feature to CDP, but its a property of the CPU, not the cache. Resctrl needs to have x86's odd/even behaviour, as that its the ABI, but this isn't how the MPAM hardware works. It is entirely possible that an in-kernel user of MPAM would not be using CDP, whereas resctrl is.

Pretending L3CODE and L3DATA are entirely separate resources is a neat trick, but doing this is specific to x86.
Doing this leaves the arch code in control of various parts of the filesystem ABI: the resources names, and the way the schemata are parsed. Allowing this stuff to vary between architectures is bad for user space.

This series collapses the CODE/DATA resources, moving all the user-visible resctrl ABI into what becomes the filesystem code. CDP becomes the type of configuration being applied to a cache. This is done by adding a struct resctrl_schema to the parts of resctrl that will move to fs. This holds the arch-code resource that is in use for this schema, along with other properties like the name, and whether the configuration being applied is CODE/DATA/BOTH.

This lets us fold the extra resources out of the arch code so that they don't need to be duplicated if the equivalent feature to CDP is missing, or implemented in a different way.

The first two patches split the resource and domain structs to have an arch specific 'hw' portion, and the rest that is visible to resctrl. Future series massage the resctrl code so there are no accesses to 'hw' structures in the parts of resctrl that will move to fs, providing helpers where necessary.

This series adds temporary scaffolding, which it removes a few patches later. This is to allow things like the ctrlval arrays and resources to be merged separately, which should make is easier to bisect. These things are marked temporary, and should all be gone by the end of the series.

This series is a little rough around the monitors, would a fake struct resctrl_schema for the monitors simplify things, or be a source of bugs?

A side effect of merging these resources, is their names are no longer printed in the kernel log at boot. e.g:

| resctrl: L3 allocation detected
| resctrl: MB allocation detected
| resctrl: L3 monitoring detected

would previously have had extra entries for 'L3CODE' and 'L3DATA'. User-space cannot rely on this to discover CDP support, as the kernel log may be inaccessible, may have been overwritten by newer messages, and because parsing the kernel log is a bad idea.

This series is based on v5.14-rc1, and can be retrieved from:
git://git.kernel.org/pub/scm/linux/kernel/git/morse/linux.git mpam/resctrl_merge_cdp/v7

v6: https://lore.kernel.org/lkml/[email protected]/
v5: https://lore.kernel.org/lkml/[email protected]/
v4: https://lore.kernel.org/lkml/[email protected]/
v3: https://lore.kernel.org/lkml/[email protected]/
v2: https://lore.kernel.org/lkml/[email protected]/
v1: https://lore.kernel.org/lkml/[email protected]/

Parts were previously posted as an RFC here:
https://lore.kernel.org/lkml/[email protected]/

Thanks,

James Morse (24):
x86/resctrl: Split struct rdt_resource
x86/resctrl: Split struct rdt_domain
x86/resctrl: Add a separate schema list for resctrl x86/resctrl: Pass the schema in info dir's private pointer x86/resctrl: Label the resources with their configuration type x86/resctrl: Walk the resctrl schema list instead of an arch list x86/resctrl: Store the effective num_closid in the schema x86/resctrl: Add resctrl_arch_get_num_closid() x86/resctrl: Pass the schema to resctrl filesystem functions x86/resctrl: Swizzle rdt_resource and resctrl_schema in
pseudo_lock_region
x86/resctrl: Add a helper to read/set the CDP configuration x86/resctrl: Move the schemata names into struct resctrl_schema x86/resctrl: Group staged configuration into a separate struct x86/resctrl: Allow different CODE/DATA configurations to be staged x86/resctrl: Rename update_domains() resctrl_arch_update_domains() x86/resctrl: Add a helper to read a closid's configuration x86/resctrl: Pass configuration type to resctrl_arch_get_config() x86/resctrl: Make ctrlval arrays the same size x86/resctrl: Apply offset correction when config is staged x86/resctrl: Calculate the index from the configuration type x86/resctrl: Merge the ctrl_val arrays
x86/resctrl: Remove rdt_cdp_peer_get()
x86/resctrl: Expand resctrl_arch_update_domains()'s msr_param range x86/resctrl: Merge the CDP resources

 arch/x86/kernel/cpu/resctrl/core.c        | 276 ++++++-------
 arch/x86/kernel/cpu/resctrl/ctrlmondata.c | 164 +++++---
 arch/x86/kernel/cpu/resctrl/internal.h    | 231 ++++-------
 arch/x86/kernel/cpu/resctrl/monitor.c     |  44 ++-
 arch/x86/kernel/cpu/resctrl/pseudo_lock.c |  12 +-
 arch/x86/kernel/cpu/resctrl/rdtgroup.c    | 459 ++++++++++++----------
 include/linux/resctrl.h                   | 185 +++++++++
 7 files changed, 775 insertions(+), 596 deletions(-)
Signature · 1 line
2.30.2