Skip to content

x86/resctrl: monitored closid+rmid together, separate arch/fs locking

James Morse lkml 9 versions – Applied

Status from this message: The following commit has been merged into the x86/cache branch of tip:

Get series with b4
b4 am [email protected]

Versions newest first · linked by author and title

  1. v9 Latest24 patches 30 repliesApplied
  2. v824 patches 25 repliesSuperseded
  3. v724 patches 78 repliesSuperseded
Earlier versions · 6
VersionPostedPatchesRepliesStatus
v624 patches55 repliesSuperseded
v524 patches52 repliesSuperseded
v424 patches37 repliesSuperseded
v319 patches50 repliesSuperseded
v218 patches54 repliesSuperseded
v118 patches23 repliesSuperseded

Patches in v9 Applied

#Subject and filesSizeRepliesApplied
1/24tick/nohz: Move tick_nohz_full_mask declaration outside the #ifdefinclude/linux/tick.h+8 -10 replies
2/24x86/resctrl: kfree() rmid_ptrs from resctrl_exit()arch/x86/kernel/cpu/resctrl/core.c arch/x86/kernel/cpu/resctrl/internal.h arch/x86/kernel/cpu/resctrl/monitor.c+22 -07 replies
3/24x86/resctrl: Create helper for RMID allocation and mondata dir creationarch/x86/kernel/cpu/resctrl/rdtgroup.c+27 -151 reply
4/24x86/resctrl: Move rmid allocation out of mkdir_rdt_prepare()arch/x86/kernel/cpu/resctrl/rdtgroup.c+26 -90 replies
5/24x86/resctrl: Track the closid with the rmidarch/x86/include/asm/resctrl.h arch/x86/kernel/cpu/resctrl/internal.h arch/x86/kernel/cpu/resctrl/monitor.c +3 more+77 -370 replies
6/24x86/resctrl: Access per-rmid structures by indexarch/x86/include/asm/resctrl.h arch/x86/kernel/cpu/resctrl/core.c arch/x86/kernel/cpu/resctrl/internal.h +2 more+100 -450 replies
7/24x86/resctrl: Allow RMID allocation to be scoped by CLOSIDarch/x86/kernel/cpu/resctrl/internal.h arch/x86/kernel/cpu/resctrl/monitor.c arch/x86/kernel/cpu/resctrl/pseudo_lock.c +1 more+41 -160 replies
8/24x86/resctrl: Track the number of dirty RMID a CLOSID hasarch/x86/kernel/cpu/resctrl/monitor.c+65 -100 replies
9/24x86/resctrl: Use __set_bit()/__clear_bit() instead of open codingarch/x86/kernel/cpu/resctrl/rdtgroup.c+12 -63 replies
10/24x86/resctrl: Allocate the cleanest CLOSID by searching closid_num_dirty_rmidarch/x86/kernel/cpu/resctrl/internal.h arch/x86/kernel/cpu/resctrl/monitor.c arch/x86/kernel/cpu/resctrl/rdtgroup.c+61 -50 replies
11/24x86/resctrl: Move CLOSID/RMID matching and setting to use helpersarch/x86/include/asm/resctrl.h arch/x86/kernel/cpu/resctrl/rdtgroup.c+56 -240 replies
12/24x86/resctrl: Add cpumask_any_housekeeping() for limbo/overflowarch/x86/kernel/cpu/resctrl/internal.h arch/x86/kernel/cpu/resctrl/monitor.c+37 -70 replies
13/24x86/resctrl: Queue mon_event_read() instead of sending an IPIarch/x86/kernel/cpu/resctrl/ctrlmondata.c arch/x86/kernel/cpu/resctrl/monitor.c+25 -30 replies
14/24x86/resctrl: Allow resctrl_arch_rmid_read() to sleeparch/x86/kernel/cpu/resctrl/monitor.c include/linux/resctrl.h+27 -210 replies
15/24x86/resctrl: Allow arch to allocate memory needed in resctrl_arch_rmid_read()arch/x86/include/asm/resctrl.h arch/x86/kernel/cpu/resctrl/ctrlmondata.c arch/x86/kernel/cpu/resctrl/internal.h +2 more+55 -40 replies
16/24x86/resctrl: Make resctrl_mounted checks explicitarch/x86/kernel/cpu/resctrl/internal.h arch/x86/kernel/cpu/resctrl/monitor.c arch/x86/kernel/cpu/resctrl/rdtgroup.c+28 -80 replies
17/24x86/resctrl: Move alloc/mon static keys into helpersarch/x86/include/asm/resctrl.h arch/x86/kernel/cpu/resctrl/internal.h arch/x86/kernel/cpu/resctrl/rdtgroup.c+24 -90 replies
18/24x86/resctrl: Make rdt_enable_key the arch's decision to switcharch/x86/include/asm/resctrl.h arch/x86/kernel/cpu/resctrl/rdtgroup.c+9 -60 replies
19/24x86/resctrl: Add helpers for system wide mon/alloc capablearch/x86/include/asm/resctrl.h arch/x86/kernel/cpu/resctrl/internal.h arch/x86/kernel/cpu/resctrl/monitor.c +2 more+37 -260 replies
20/24x86/resctrl: Add CPU online callback for resctrl workarch/x86/kernel/cpu/resctrl/core.c arch/x86/kernel/cpu/resctrl/rdtgroup.c include/linux/resctrl.h+13 -40 replies
21/24x86/resctrl: Allow overflow/limbo handlers to be scheduled on any-but cpuarch/x86/kernel/cpu/resctrl/core.c arch/x86/kernel/cpu/resctrl/ctrlmondata.c arch/x86/kernel/cpu/resctrl/internal.h +3 more+72 -211 reply
22/24x86/resctrl: Add CPU offline callback for resctrl workarch/x86/kernel/cpu/resctrl/core.c arch/x86/kernel/cpu/resctrl/rdtgroup.c include/linux/resctrl.h+30 -200 replies
23/24x86/resctrl: Move domain helper migration into resctrl_offline_cpu()arch/x86/kernel/cpu/resctrl/core.c arch/x86/kernel/cpu/resctrl/rdtgroup.c+18 -160 replies
24/24x86/resctrl: Separate arch and fs resctrl locksarch/x86/kernel/cpu/resctrl/core.c arch/x86/kernel/cpu/resctrl/ctrlmondata.c arch/x86/kernel/cpu/resctrl/monitor.c +3 more+112 -280 replies

Reviews trailers per patch

All 24 patches have a Reviewed-by or Acked-by of their own; the series has one for all of them.

PatchShaopeng TanPeter NewmanCarl WorthThomas GleixnerReinette ChatreBabu MogerIlpo JärvinenXin HaoDavid HildenbrandJames MorseMaciej Wieczor-Retman
SeriesR-b R-b T-b R-b R-b R-b R-b T-b T-b
1/24T-b R-b T-b T-b R-b A-b
2/24T-b R-b T-b R-b
3/24T-b R-b T-b T-b R-b T-b R-b R-b R-b
4/24T-b R-b T-b T-b R-b T-b R-b
5/24T-b R-b T-b T-b R-b T-b R-b
6/24T-b R-b T-b T-b R-b T-b R-b
7/24T-b R-b T-b T-b R-b T-b R-b
8/24T-b R-b T-b T-b R-b T-b R-b
9/24T-b R-b T-b T-b R-b T-b R-b R-b
10/24T-b R-b T-b T-b R-b T-b R-b
11/24T-b R-b T-b T-b R-b T-b R-b
12/24T-b R-b T-b T-b R-b T-b R-b
13/24T-b R-b T-b R-b T-b R-b T-b R-b
14/24T-b R-b T-b T-b R-b T-b R-b
15/24T-b R-b T-b T-b R-b T-b R-b
16/24T-b R-b T-b T-b R-b T-b R-b
17/24T-b R-b T-b T-b R-b T-b R-b
18/24T-b R-b T-b T-b R-b T-b R-b
19/24T-b R-b T-b T-b R-b T-b R-b
20/24T-b R-b T-b T-b R-b T-b R-b
21/24T-b R-b T-b T-b R-b T-b R-b
22/24T-b R-b T-b T-b R-b T-b R-b
23/24T-b R-b T-b T-b R-b T-b R-b
24/24T-b R-b T-b T-b R-b T-b R-b

Trailers given in replies are filled; those carried in the patches from earlier versions are outlined with dashes. Scroll the table sideways for all 11 reviewers.

Read the v9 thread

Cover letter [PATCH v9 00/24] x86/resctrl: monitored closid+rmid together, separate arch/fs locking

Hello!

It's been back and forth for whether this series should be rebased onto Tony's SNC series. This version isn't, its based on tip/x86/cache.
(I have the rebased-and-tested versions if anyone needs them)

This version just has minor cleanup from the previous one.
* An unusued parameter in unused code has gone,
* I've added a comment about the sizeing of the index array - it only matters on arm64.

Changes are also noted on each patch.

~

This series does two things, it changes resctrl to call resctrl_arch_rmid_read() in a way that works for MPAM, and it separates the locking so that the arch code and filesystem code don't have to share a mutex. I tried to split this as two series, but these touch similar call sites, so it would create more work.

(What's MPAM? See the cover letter of the first series. [1])

On x86 the RMID is an independent number. MPAMs equivalent is PMG, but this isn't an independent number - it extends the PARTID (same as CLOSID) space with bits that aren't used to select the configuration. The monitors can then be told to match specific PMG values, allowing monitor-groups to be created.

But, MPAM expects the monitors to always monitor by PARTID. The Cache-storage-utilisation counters can only work this way.
(In the MPAM spec not setting the MATCH_PARTID bit is made CONSTRAINED UNPREDICTABLE - which is Arm's term to mean portable software can't rely on this)

It gets worse, as some SoCs may have very few PMG bits. I've seen the datasheet for one that has a single bit of PMG space.

To be usable, MPAM's counters always need the PARTID and the PMG. For resctrl, this means always making the CLOSID available when the RMID is used.

To ensure RMID are always unique, this series combines the CLOSID and RMID into an index, and manages RMID based on that. For x86, the index and RMID would always be the same.

Currently the architecture specific code in the cpuhp callbacks takes the rdtgroup_mutex. This means the filesystem code would have to export this lock, resulting in an ill-defined interface between the two, and the possibility of cross-architecture lock-ordering head aches.

The second part of this series adds a domain_list_lock to protect writes to the domain list, and protects the domain list with RCU - or cpus_read_lock().

Use of RCU is to allow lockless readers of the domain list. To get MPAMs monitors working, its very likely they'll need to be plumbed up to perf. An uncore PMU driver would need to be a lockless reader of the domain list.

This series is based on tip/x86/caches's commit fc747eebef73, and can be retrieved from:
https://git.kernel.org/pub/scm/linux/kernel/git/morse/linux.git mpam/monitors_and_locking/v9

Bugs welcome,

Thanks,

James

[1] https://lore.kernel.org/lkml/[email protected]/
[v1] https://lore.kernel.org/all/[email protected]/
[v2] https://lore.kernel.org/lkml/[email protected]/
[v3] https://lore.kernel.org/r/[email protected]
[v4] https://lore.kernel.org/r/[email protected]
[v5] https://lore.kernel.org/lkml/[email protected]/
[v6] https://lore.kernel.org/all/[email protected]/
[v7] https://lore.kernel.org/r/[email protected]/
[v8] https://lore.kernel.org/lkml/[email protected]/

James Morse (24):
tick/nohz: Move tick_nohz_full_mask declaration outside the #ifdef x86/resctrl: kfree() rmid_ptrs from resctrl_exit() x86/resctrl: Create helper for RMID allocation and mondata dir
creation
x86/resctrl: Move rmid allocation out of mkdir_rdt_prepare() x86/resctrl: Track the closid with the rmid
x86/resctrl: Access per-rmid structures by index x86/resctrl: Allow RMID allocation to be scoped by CLOSID x86/resctrl: Track the number of dirty RMID a CLOSID has x86/resctrl: Use __set_bit()/__clear_bit() instead of open coding x86/resctrl: Allocate the cleanest CLOSID by searching
closid_num_dirty_rmid
x86/resctrl: Move CLOSID/RMID matching and setting to use helpers x86/resctrl: Add cpumask_any_housekeeping() for limbo/overflow x86/resctrl: Queue mon_event_read() instead of sending an IPI x86/resctrl: Allow resctrl_arch_rmid_read() to sleep x86/resctrl: Allow arch to allocate memory needed in
resctrl_arch_rmid_read()
x86/resctrl: Make resctrl_mounted checks explicit x86/resctrl: Move alloc/mon static keys into helpers x86/resctrl: Make rdt_enable_key the arch's decision to switch x86/resctrl: Add helpers for system wide mon/alloc capable x86/resctrl: Add CPU online callback for resctrl work x86/resctrl: Allow overflow/limbo handlers to be scheduled on any-but
cpu
x86/resctrl: Add CPU offline callback for resctrl work x86/resctrl: Move domain helper migration into resctrl_offline_cpu() x86/resctrl: Separate arch and fs resctrl locks

 arch/x86/include/asm/resctrl.h            |  90 +++++
 arch/x86/kernel/cpu/resctrl/core.c        | 102 ++---
 arch/x86/kernel/cpu/resctrl/ctrlmondata.c |  48 ++-
 arch/x86/kernel/cpu/resctrl/internal.h    |  67 +++-
 arch/x86/kernel/cpu/resctrl/monitor.c     | 463 +++++++++++++++++-----
 arch/x86/kernel/cpu/resctrl/pseudo_lock.c |  15 +-
 arch/x86/kernel/cpu/resctrl/rdtgroup.c    | 359 ++++++++++++-----
 include/linux/resctrl.h                   |  48 ++-
 include/linux/tick.h                      |   9 +-
 9 files changed, 921 insertions(+), 280 deletions(-)
Signature · 1 line
2.39.2