Skip to content

RFC Introduce region-aware RDT support

Chen Yu lkml 1 version In review

Get series with b4
b4 am [email protected]

Versions newest first

  1. v1 RFC Latest31 patches 5 repliesIn review

Patches in v1 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/31x86/topology: Export topo_lookup_cpuid() for resctrl usearch/x86/include/asm/apic.h arch/x86/kernel/cpu/topology.c+2 -10 replies
2/31x86/resctrl: Require 64-bit x86 for resctrl supportarch/x86/Kconfig+2 -20 replies
3/31x86/resctrl: Parse ACPI ERDT table and save CACD cpumask for RMDD domainsarch/x86/kernel/cpu/resctrl/Makefile arch/x86/kernel/cpu/resctrl/core.c arch/x86/kernel/cpu/resctrl/erdt.c +1 more+311 -10 replies
4/31x86/resctrl: Attach ACPI ERDT information to L3 mon domain on CPU onlinearch/x86/kernel/cpu/resctrl/core.c arch/x86/kernel/cpu/resctrl/erdt.c arch/x86/kernel/cpu/resctrl/internal.h+81 -10 replies
5/31x86/resctrl: Parse ACPI CMRC tablearch/x86/include/asm/resctrl.h arch/x86/kernel/cpu/resctrl/erdt.c arch/x86/kernel/cpu/resctrl/internal.h+64 -13 replies
6/31x86/resctrl: Refactor the monitor read functionarch/x86/kernel/cpu/resctrl/monitor.c+19 -80 replies
7/31fs/resctrl: Do not invoke smp_processor_id() in preemptible contextfs/resctrl/monitor.c+34 -140 replies
8/31x86/resctrl: Introduce erdt_cpu_has() and erdt_support()arch/x86/include/asm/resctrl.h arch/x86/kernel/cpu/resctrl/core.c arch/x86/kernel/cpu/resctrl/erdt.c +1 more+31 -00 replies
9/31x86/resctrl: Add MMIO-based LLC occupancy monitoring supportarch/x86/include/asm/resctrl.h arch/x86/kernel/cpu/resctrl/core.c arch/x86/kernel/cpu/resctrl/erdt.c +2 more+84 -40 replies
10/31Revert "x86/resctrl: NOT_FOR_INCLUSION: Example support for multiple controls"arch/x86/kernel/cpu/resctrl/core.c+21 -940 replies
11/31x86/resctrl: Rename struct resctrl_membw to struct resctrl_ctrl_scalararch/x86/kernel/cpu/resctrl/core.c drivers/resctrl/mpam_resctrl.c fs/resctrl/ctrlmondata.c +3 more+43 -430 replies
12/31x86/resctrl: Rename struct resctrl_cache to struct resctrl_ctrl_bitmaparch/x86/kernel/cpu/resctrl/core.c drivers/resctrl/mpam_resctrl.c fs/resctrl/ctrlmondata.c +3 more+43 -430 replies
13/31x86/resctrl: Add per-control and per-resource flagsarch/x86/kernel/cpu/resctrl/core.c drivers/resctrl/mpam_resctrl.c include/linux/resctrl.h+27 -00 replies
14/31x86/resctrl: Add emulation controller list to resctrl_ctrlarch/x86/kernel/cpu/resctrl/core.c include/linux/resctrl.h+8 -00 replies
15/31x86/resctrl: Parse ACPI MMRC tablearch/x86/kernel/cpu/resctrl/erdt.c arch/x86/kernel/cpu/resctrl/internal.h+56 -10 replies
16/31x86/resctrl: Replace "msr" in monitoring data identifiersarch/x86/kernel/cpu/resctrl/internal.h arch/x86/kernel/cpu/resctrl/monitor.c+19 -190 replies
17/31x86/resctrl: Introduce region aware MBM event definitionsfs/resctrl/monitor.c fs/resctrl/rdtgroup.c include/linux/resctrl.h +1 more+46 -40 replies
18/31x86/resctrl: Introduce memory region based MBM read callback on MMIO spacearch/x86/kernel/cpu/resctrl/erdt.c arch/x86/kernel/cpu/resctrl/internal.h arch/x86/kernel/cpu/resctrl/monitor.c +1 more+115 -40 replies
19/31x86/resctrl: Enable the region based events by adding them into the eventarch/x86/kernel/cpu/resctrl/core.c arch/x86/kernel/cpu/resctrl/erdt.c arch/x86/kernel/cpu/resctrl/internal.h +3 more+71 -90 replies
20/31x86/resctrl: Rename msr_update to hw_updatearch/x86/kernel/cpu/resctrl/core.c arch/x86/kernel/cpu/resctrl/ctrlmondata.c arch/x86/kernel/cpu/resctrl/internal.h +1 more+44 -440 replies
21/31x86/resctrl: Parse ACPI MARC tablearch/x86/kernel/cpu/resctrl/erdt.c arch/x86/kernel/cpu/resctrl/internal.h+79 -10 replies
22/31fs/resctrl: Add region-based control names and resctrl_ctrl_name_region()fs/resctrl/ctrlmondata.c include/linux/resctrl.h+40 -10 replies
23/31x86/resctrl: Add region aware MBA controllersarch/x86/kernel/cpu/resctrl/core.c arch/x86/kernel/cpu/resctrl/erdt.c arch/x86/kernel/cpu/resctrl/internal.h+75 -00 replies
24/31x86/resctrl: Attach ACPI ERDT information to ctrl domain on CPU onlinearch/x86/kernel/cpu/resctrl/core.c arch/x86/kernel/cpu/resctrl/erdt.c arch/x86/kernel/cpu/resctrl/internal.h+16 -00 replies
25/31x86/resctrl: Introduce region-based MBA write implementation on MMIO spacearch/x86/kernel/cpu/resctrl/erdt.c arch/x86/kernel/cpu/resctrl/internal.h+56 -02 replies
26/31x86/resctrl: Allow control writes from any CPU for MMIO controllersarch/x86/kernel/cpu/resctrl/ctrlmondata.c arch/x86/kernel/cpu/resctrl/erdt.c include/linux/resctrl.h+13 -10 replies
27/31x86/resctrl: Enable region-aware MBM/MBA via the RDT_CTRL registerarch/x86/kernel/cpu/resctrl/erdt.c+21 -00 replies
28/31x86/resctrl: Emulate the legacy MBA controller via the region MAX controlsarch/x86/kernel/cpu/resctrl/ctrlmondata.c arch/x86/kernel/cpu/resctrl/erdt.c include/linux/resctrl.h+88 -80 replies
29/31fs/resctrl: Expose emulation controllers in a resource_schemata subdirfs/resctrl/rdtgroup.c+27 -40 replies
30/31fs/resctrl: Fix excessive padding in schemata outputfs/resctrl/ctrlmondata.c fs/resctrl/rdtgroup.c+17 -130 replies
31/31x86,fs/resctrl: Update Documentation for region aware RDTDocumentation/filesystems/resctrl.rst+110 -00 replies

Reviews trailers per patch

1 of 31 patches has a Reviewed-by or Acked-by.

PatchHongyu NingThomas Gleixner
1/31T-b
2/31T-b
3/31T-b
4/31T-b
5/31T-b R-b
6/31T-b
7/31T-b
8/31T-b
9/31T-b
10/31
11/31
12/31
13/31
14/31
15/31
16/31
17/31
18/31
19/31
20/31
21/31
22/31
23/31
24/31
25/31
26/31
27/31
28/31
29/31
30/31
31/31

Trailers given in replies are filled; those carried in the patches from earlier versions are outlined with dashes.

Read the v1 thread

Cover letter [RFC PATCH 00/31] Introduce region-aware RDT support

Hi everyone,

This is a PoC patch set posted to support region-aware RDT on Intel platform. Its purpose is to demonstrate what region-aware RDT is and to seek for suggestion from the community on how Linux could support this feature properly. It also allows people to have an early try with this preliminary version if they would like to explore it in advance. This version is not intended for inclusion for upstream in its current shape.

This patch set is based on the generic schemata PoC patch set, which introduced multiple controllers per rdt_resource. The multi-controller support is the essential framework for region-aware RDT, as each memory region needs its own set of controls on the same resource. The generic schemata PoC is at:

git://git.kernel.org/pub/scm/linux/kernel/git/reinette/linux.git
branch resctrl/controls_rfc_v1

Since many details are still under discussion above, current patch set is evolving as well, and will be adjusted as the discussion and the decisions move forward.

What is region-aware RDT
========================

Intel hardware supports Region-Aware Memory Bandwidth Allocation (MBA) and Region-Aware Memory Bandwidth Monitoring (MBM). With Region-Aware MBA, independent bandwidth control (throttling) of L3 domain bandwidth to multiple regions is supported, enabling users to dynamically rebalance bandwidth control limits across different memory regions, each of which may have distinct bandwidth, latency, and capacity characteristics. Region-Aware MBM includes the capability to independently track multiple domains that are simultaneously accessing several memory regions. These memory regions correspond to different levels of memory tiers, such as directly attached memory (Tier 1), CXL-attached memory (Tier 2), and CXL accelerator devices with attached memory. Note, the region ID is per socket scope. Intel platform supports up to 4 regions for now.

Suppose there are 2 regions in each socket:

  +------------------------+------------------------+
  | +--------+ +--------+  | +--------+ +--------+  |
  | | L3_00  | | L3_01  |  | | L3_02  | | L3_03  |  |
  | +--------+ +--------+  | +--------+ +--------+  |
  | +-------------------+  | +-------------------+  |
  | |      IMC1         |  | |      IMC2         |  |
  | +-------------------+  | +-------------------+  |
  | +-------+   +-------+  | +-------+   +-------+  |
  | | DDR1  |   |  CXL1 |  | | DDR2  |   |  CXL2 |  |
  | +-------+   +-------+  | +-------+   +-------+  |
  |  socket0               |  socket1               |
  +------------------------+------------------------+

In above graph, from the perspective of a CPU associated with the L3_00 domain, Region 0 typically represents the local memory region (DDR1), while Region 2 typically represents the remote memory region (DDR2). Similarly, for CPUs attached to L3_00, Region 1 represents the local memory region (CXL1), and Region 3 represents the remote memory region (CXL2). Here the terms "local" and "remote" are defined at the socket level.

Take the region aware MBM for example. For the L3_00 domain, the memory bandwidth of Region 0 refers to the data transferred when the L3 miss occurs in L3_00 and the data is refilled from DDR1 - note, data refilled from L3_01 to L3_00 is not counted in. The bandwidth of Region 2, by contrast, refers to the data refilled from DDR2. Similarly, the same calculation logic applies to Region 1 (CXL1) and Region 3 (CXL2).

For Region-Aware MBA, when setting values for Region 0 on CPUs attached to L3_00, this configuration controls the traffic generated when data is transferred between cores and L3_00 targeting DDR1.

The resctrl interface
=====================

Region-aware MBM adds one file per region to each mon_data domain directory. For example, with 4 regions the following files are created:

mbm_region0_bytes, mbm_region1_bytes, mbm_region2_bytes and mbm_region3_bytes

For region-aware MBA, every region has its own set of controls, and each of them appears as a separate schemata line named "MB_REGION<n>_<type>", where <n> is the region number and <type> is one of OPT, MIN or MAX. Only the control types that the hardware reports in the MARC ACPI sub-table are present. The legacy "MB" control is emulated by the MAX control of every region, so writing "MB" throttles all regions at once, while writing an individual "MB_REGION<n>_MAX" line throttles only that region.

Below is an example on a platform with 2 memory regions, adjusting the tier1 local memory bandwidth (usually the DDR) of the second domain:

  # cat schemata
              MB:0=100;1=100;2=100;3=100
  MB_REGION0_OPT:0=511;1=511;2=511;3=511
  MB_REGION0_MIN:0=511;1=511;2=511;3=511
  MB_REGION0_MAX:0=511;1=511;2=511;3=511
  MB_REGION1_OPT:0=511;1=511;2=511;3=511
  MB_REGION1_MIN:0=511;1=511;2=511;3=511
  MB_REGION1_MAX:0=511;1=511;2=511;3=511
              L3:0=3ff;1=3ff;2=3ff;3=3ff
  # echo "MB_REGION0_MAX:1=200" > schemata
  # cat schemata
              MB:0=100;1=100;2=100;3=100
  MB_REGION0_OPT:0=511;1=511;2=511;3=511
  MB_REGION0_MIN:0=511;1=511;2=511;3=511
  MB_REGION0_MAX:0=511;1=200;2=511;3=511
  MB_REGION1_OPT:0=511;1=511;2=511;3=511
  MB_REGION1_MIN:0=511;1=511;2=511;3=511
  MB_REGION1_MAX:0=511;1=511;2=511;3=511
              L3:0=3ff;1=3ff;2=3ff;3=3ff

How this patch set is composed
==============================

Patches 1 to 9, from "x86/topology: Export topo_lookup_cpuid() for resctrl use" to "x86/resctrl: Add MMIO-based LLC occupancy monitoring support", add the MMIO-based CMT support, which has already been posted at:

https://lore.kernel.org/all/[email protected]/

They are included here because they have been slightly modified to apply on top of Reinette's generic schemata PoC.

Patches 10 to 14, from Revert "x86/resctrl: NOT_FOR_INCLUSION: Example support for multiple controls" to "x86/resctrl: Add emulation controller list to resctrl_ctrl", prepare for region-aware RDT.

Patches 15 to 19, from "x86/resctrl: Parse ACPI MMRC table" to "x86/resctrl: Enable the region based events by adding them into the event", add the region-aware MBM support.

Patches 20 to 30, from "x86/resctrl: Rename msr_update to hw_update" to the "fs/resctrl: Fix excessive padding in schemata output", add the region-aware MBA support.

Patch 31 is the documentation for region-aware RDT.

Any comments would be much appreciated.

thanks,
Chenyu

Anil S Keshavamurthy (1):
x86/resctrl: Parse ACPI ERDT table and save CACD cpumask for RMDD
domains

Chen Yu (29):
x86/topology: Export topo_lookup_cpuid() for resctrl use x86/resctrl: Require 64-bit x86 for resctrl support x86/resctrl: Attach ACPI ERDT information to L3 mon domain on CPU
online
x86/resctrl: Parse ACPI CMRC table
x86/resctrl: Refactor the monitor read function x86/resctrl: Introduce erdt_cpu_has() and erdt_support() x86/resctrl: Add MMIO-based LLC occupancy monitoring support Revert "x86/resctrl: NOT_FOR_INCLUSION: Example support for multiple
controls"
x86/resctrl: Rename struct resctrl_membw to struct resctrl_ctrl_scalar x86/resctrl: Rename struct resctrl_cache to struct resctrl_ctrl_bitmap x86/resctrl: Add per-control and per-resource flags x86/resctrl: Add emulation controller list to resctrl_ctrl x86/resctrl: Parse ACPI MMRC table
x86/resctrl: Replace "msr" in monitoring data identifiers x86/resctrl: Introduce region aware MBM event definitions x86/resctrl: Introduce memory region based MBM read callback on MMIO
space
x86/resctrl: Enable the region based events by adding them into the
event
x86/resctrl: Rename msr_update to hw_update
x86/resctrl: Parse ACPI MARC table
fs/resctrl: Add region-based control names and
resctrl_ctrl_name_region()
x86/resctrl: Add region aware MBA controllers
x86/resctrl: Attach ACPI ERDT information to ctrl domain on CPU online x86/resctrl: Introduce region-based MBA write implementation on MMIO
space
x86/resctrl: Allow control writes from any CPU for MMIO controllers x86/resctrl: Enable region-aware MBM/MBA via the RDT_CTRL register x86/resctrl: Emulate the legacy MBA controller via the region MAX
controls
fs/resctrl: Expose emulation controllers in a resource_schemata subdir fs/resctrl: Fix excessive padding in schemata output x86,fs/resctrl: Update Documentation for region aware RDT

Tony Luck (1):
fs/resctrl: Do not invoke smp_processor_id() in preemptible context

 Documentation/filesystems/resctrl.rst     | 110 +++
 arch/x86/Kconfig                          |   4 +-
 arch/x86/include/asm/apic.h               |   1 +
 arch/x86/include/asm/resctrl.h            |  11 +-
 arch/x86/kernel/cpu/resctrl/Makefile      |   1 +
 arch/x86/kernel/cpu/resctrl/core.c        | 258 +++----
 arch/x86/kernel/cpu/resctrl/ctrlmondata.c | 106 ++-
 arch/x86/kernel/cpu/resctrl/erdt.c        | 886 ++++++++++++++++++++++
 arch/x86/kernel/cpu/resctrl/internal.h    |  95 ++-
 arch/x86/kernel/cpu/resctrl/monitor.c     |  82 +-
 arch/x86/kernel/cpu/resctrl/rdtgroup.c    |  14 +-
 arch/x86/kernel/cpu/topology.c            |   2 +-
 drivers/resctrl/mpam_resctrl.c            |  18 +-
 fs/resctrl/ctrlmondata.c                  |  49 +-
 fs/resctrl/monitor.c                      |  73 +-
 fs/resctrl/pseudo_lock.c                  |   2 +-
 fs/resctrl/rdtgroup.c                     | 111 ++-
 include/linux/resctrl.h                   |  82 +-
 include/linux/resctrl_types.h             |  21 +-
 19 files changed, 1634 insertions(+), 292 deletions(-)
 create mode 100644 arch/x86/kernel/cpu/resctrl/erdt.c
Signature · 1 line
2.43.0