RFC Introduce region-aware RDT support
Get series with b4
b4 am [email protected]Versions newest first
- v1 RFC Latest31 patches 5 repliesIn review
Patches in v1 No applies recorded yet
| # | Subject and files | Size | Replies | Applied |
|---|---|---|---|---|
| 1/31 | x86/topology: Export topo_lookup_cpuid() for resctrl usearch/ arch/ | +2 | 0 replies | |
| 2/31 | x86/resctrl: Require 64-bit x86 for resctrl supportarch/ | +2 | 0 replies | |
| 3/31 | x86/resctrl: Parse ACPI ERDT table and save CACD cpumask for RMDD domainsarch/ arch/ arch/ +1 more | +311 | 0 replies | |
| 4/31 | x86/resctrl: Attach ACPI ERDT information to L3 mon domain on CPU onlinearch/ arch/ arch/ | +81 | 0 replies | |
| 5/31 | x86/resctrl: Parse ACPI CMRC tablearch/ arch/ arch/ | +64 | 3 replies | |
| 6/31 | x86/resctrl: Refactor the monitor read functionarch/ | +19 | 0 replies | |
| 7/31 | fs/resctrl: Do not invoke smp_processor_id() in preemptible contextfs/ | +34 | 0 replies | |
| 8/31 | x86/resctrl: Introduce erdt_cpu_has() and erdt_support()arch/ arch/ arch/ +1 more | +31 | 0 replies | |
| 9/31 | x86/resctrl: Add MMIO-based LLC occupancy monitoring supportarch/ arch/ arch/ +2 more | +84 | 0 replies | |
| 10/31 | Revert "x86/resctrl: NOT_FOR_INCLUSION: Example support for multiple controls"arch/ | +21 | 0 replies | |
| 11/31 | x86/resctrl: Rename struct resctrl_membw to struct resctrl_ctrl_scalararch/ drivers/ fs/ +3 more | +43 | 0 replies | |
| 12/31 | x86/resctrl: Rename struct resctrl_cache to struct resctrl_ctrl_bitmaparch/ drivers/ fs/ +3 more | +43 | 0 replies | |
| 13/31 | x86/resctrl: Add per-control and per-resource flagsarch/ drivers/ include/ | +27 | 0 replies | |
| 14/31 | x86/resctrl: Add emulation controller list to resctrl_ctrlarch/ include/ | +8 | 0 replies | |
| 15/31 | x86/resctrl: Parse ACPI MMRC tablearch/ arch/ | +56 | 0 replies | |
| 16/31 | x86/resctrl: Replace "msr" in monitoring data identifiersarch/ arch/ | +19 | 0 replies | |
| 17/31 | x86/resctrl: Introduce region aware MBM event definitionsfs/ fs/ include/ +1 more | +46 | 0 replies | |
| 18/31 | x86/resctrl: Introduce memory region based MBM read callback on MMIO spacearch/ arch/ arch/ +1 more | +115 | 0 replies | |
| 19/31 | x86/resctrl: Enable the region based events by adding them into the eventarch/ arch/ arch/ +3 more | +71 | 0 replies | |
| 20/31 | x86/resctrl: Rename msr_update to hw_updatearch/ arch/ arch/ +1 more | +44 | 0 replies | |
| 21/31 | x86/resctrl: Parse ACPI MARC tablearch/ arch/ | +79 | 0 replies | |
| 22/31 | fs/resctrl: Add region-based control names and resctrl_ctrl_name_region()fs/ include/ | +40 | 0 replies | |
| 23/31 | x86/resctrl: Add region aware MBA controllersarch/ arch/ arch/ | +75 | 0 replies | |
| 24/31 | x86/resctrl: Attach ACPI ERDT information to ctrl domain on CPU onlinearch/ arch/ arch/ | +16 | 0 replies | |
| 25/31 | x86/resctrl: Introduce region-based MBA write implementation on MMIO spacearch/ arch/ | +56 | 2 replies | |
| 26/31 | x86/resctrl: Allow control writes from any CPU for MMIO controllersarch/ arch/ include/ | +13 | 0 replies | |
| 27/31 | x86/resctrl: Enable region-aware MBM/MBA via the RDT_CTRL registerarch/ | +21 | 0 replies | |
| 28/31 | x86/resctrl: Emulate the legacy MBA controller via the region MAX controlsarch/ arch/ include/ | +88 | 0 replies | |
| 29/31 | fs/resctrl: Expose emulation controllers in a resource_schemata subdirfs/ | +27 | 0 replies | |
| 30/31 | fs/resctrl: Fix excessive padding in schemata outputfs/ fs/ | +17 | 0 replies | |
| 31/31 | x86,fs/resctrl: Update Documentation for region aware RDTDocumentation/ | +110 | 0 replies |
Reviews trailers per patch
1 of 31 patches has a Reviewed-by or Acked-by.
| Patch | Hongyu Ning | Thomas Gleixner |
|---|---|---|
| 1/31 | T-b | |
| 2/31 | T-b | |
| 3/31 | T-b | |
| 4/31 | T-b | |
| 5/31 | T-b | R-b |
| 6/31 | T-b | |
| 7/31 | T-b | |
| 8/31 | T-b | |
| 9/31 | T-b | |
| 10/31 | ||
| 11/31 | ||
| 12/31 | ||
| 13/31 | ||
| 14/31 | ||
| 15/31 | ||
| 16/31 | ||
| 17/31 | ||
| 18/31 | ||
| 19/31 | ||
| 20/31 | ||
| 21/31 | ||
| 22/31 | ||
| 23/31 | ||
| 24/31 | ||
| 25/31 | ||
| 26/31 | ||
| 27/31 | ||
| 28/31 | ||
| 29/31 | ||
| 30/31 | ||
| 31/31 |
Trailers given in replies are filled; those carried in the patches from earlier versions are outlined with dashes.
Cover letter [RFC PATCH 00/31] Introduce region-aware RDT support
Hi everyone,
This is a PoC patch set posted to support region-aware RDT on Intel platform. Its purpose is to demonstrate what region-aware RDT is and to seek for suggestion from the community on how Linux could support this feature properly. It also allows people to have an early try with this preliminary version if they would like to explore it in advance. This version is not intended for inclusion for upstream in its current shape.
This patch set is based on the generic schemata PoC patch set, which introduced multiple controllers per rdt_resource. The multi-controller support is the essential framework for region-aware RDT, as each memory region needs its own set of controls on the same resource. The generic schemata PoC is at:
git://git.kernel.org/pub/scm/linux/kernel/git/reinette/linux.git
branch resctrl/controls_rfc_v1
Since many details are still under discussion above, current patch set is evolving as well, and will be adjusted as the discussion and the decisions move forward.
What is region-aware RDT
========================
Intel hardware supports Region-Aware Memory Bandwidth Allocation (MBA) and Region-Aware Memory Bandwidth Monitoring (MBM). With Region-Aware MBA, independent bandwidth control (throttling) of L3 domain bandwidth to multiple regions is supported, enabling users to dynamically rebalance bandwidth control limits across different memory regions, each of which may have distinct bandwidth, latency, and capacity characteristics. Region-Aware MBM includes the capability to independently track multiple domains that are simultaneously accessing several memory regions. These memory regions correspond to different levels of memory tiers, such as directly attached memory (Tier 1), CXL-attached memory (Tier 2), and CXL accelerator devices with attached memory. Note, the region ID is per socket scope. Intel platform supports up to 4 regions for now.
Suppose there are 2 regions in each socket:
+------------------------+------------------------+ | +--------+ +--------+ | +--------+ +--------+ | | | L3_00 | | L3_01 | | | L3_02 | | L3_03 | | | +--------+ +--------+ | +--------+ +--------+ | | +-------------------+ | +-------------------+ | | | IMC1 | | | IMC2 | | | +-------------------+ | +-------------------+ | | +-------+ +-------+ | +-------+ +-------+ | | | DDR1 | | CXL1 | | | DDR2 | | CXL2 | | | +-------+ +-------+ | +-------+ +-------+ | | socket0 | socket1 | +------------------------+------------------------+
In above graph, from the perspective of a CPU associated with the L3_00 domain, Region 0 typically represents the local memory region (DDR1), while Region 2 typically represents the remote memory region (DDR2). Similarly, for CPUs attached to L3_00, Region 1 represents the local memory region (CXL1), and Region 3 represents the remote memory region (CXL2). Here the terms "local" and "remote" are defined at the socket level.
Take the region aware MBM for example. For the L3_00 domain, the memory bandwidth of Region 0 refers to the data transferred when the L3 miss occurs in L3_00 and the data is refilled from DDR1 - note, data refilled from L3_01 to L3_00 is not counted in. The bandwidth of Region 2, by contrast, refers to the data refilled from DDR2. Similarly, the same calculation logic applies to Region 1 (CXL1) and Region 3 (CXL2).
For Region-Aware MBA, when setting values for Region 0 on CPUs attached to L3_00, this configuration controls the traffic generated when data is transferred between cores and L3_00 targeting DDR1.
The resctrl interface
=====================
Region-aware MBM adds one file per region to each mon_data domain directory. For example, with 4 regions the following files are created:
mbm_region0_bytes, mbm_region1_bytes, mbm_region2_bytes and mbm_region3_bytes
For region-aware MBA, every region has its own set of controls, and each of them appears as a separate schemata line named "MB_REGION<n>_<type>", where <n> is the region number and <type> is one of OPT, MIN or MAX. Only the control types that the hardware reports in the MARC ACPI sub-table are present. The legacy "MB" control is emulated by the MAX control of every region, so writing "MB" throttles all regions at once, while writing an individual "MB_REGION<n>_MAX" line throttles only that region.
Below is an example on a platform with 2 memory regions, adjusting the tier1 local memory bandwidth (usually the DDR) of the second domain:
# cat schemata
MB:0=100;1=100;2=100;3=100
MB_REGION0_OPT:0=511;1=511;2=511;3=511
MB_REGION0_MIN:0=511;1=511;2=511;3=511
MB_REGION0_MAX:0=511;1=511;2=511;3=511
MB_REGION1_OPT:0=511;1=511;2=511;3=511
MB_REGION1_MIN:0=511;1=511;2=511;3=511
MB_REGION1_MAX:0=511;1=511;2=511;3=511
L3:0=3ff;1=3ff;2=3ff;3=3ff # echo "MB_REGION0_MAX:1=200" > schemata
# cat schemata
MB:0=100;1=100;2=100;3=100
MB_REGION0_OPT:0=511;1=511;2=511;3=511
MB_REGION0_MIN:0=511;1=511;2=511;3=511
MB_REGION0_MAX:0=511;1=200;2=511;3=511
MB_REGION1_OPT:0=511;1=511;2=511;3=511
MB_REGION1_MIN:0=511;1=511;2=511;3=511
MB_REGION1_MAX:0=511;1=511;2=511;3=511
L3:0=3ff;1=3ff;2=3ff;3=3ffHow this patch set is composed
==============================
Patches 1 to 9, from "x86/topology: Export topo_lookup_cpuid() for resctrl use" to "x86/resctrl: Add MMIO-based LLC occupancy monitoring support", add the MMIO-based CMT support, which has already been posted at:
https://lore.kernel.org/all/[email protected]/
They are included here because they have been slightly modified to apply on top of Reinette's generic schemata PoC.
Patches 10 to 14, from Revert "x86/resctrl: NOT_FOR_INCLUSION: Example support for multiple controls" to "x86/resctrl: Add emulation controller list to resctrl_ctrl", prepare for region-aware RDT.
Patches 15 to 19, from "x86/resctrl: Parse ACPI MMRC table" to "x86/resctrl: Enable the region based events by adding them into the event", add the region-aware MBM support.
Patches 20 to 30, from "x86/resctrl: Rename msr_update to hw_update" to the "fs/resctrl: Fix excessive padding in schemata output", add the region-aware MBA support.
Patch 31 is the documentation for region-aware RDT.
Any comments would be much appreciated.
thanks,
Chenyu
Anil S Keshavamurthy (1):
x86/resctrl: Parse ACPI ERDT table and save CACD cpumask for RMDD
domains
Chen Yu (29):
x86/topology: Export topo_lookup_cpuid() for resctrl use x86/resctrl: Require 64-bit x86 for resctrl support x86/resctrl: Attach ACPI ERDT information to L3 mon domain on CPU
online
x86/resctrl: Parse ACPI CMRC table
x86/resctrl: Refactor the monitor read function x86/resctrl: Introduce erdt_cpu_has() and erdt_support() x86/resctrl: Add MMIO-based LLC occupancy monitoring support Revert "x86/resctrl: NOT_FOR_INCLUSION: Example support for multiple
controls"
x86/resctrl: Rename struct resctrl_membw to struct resctrl_ctrl_scalar x86/resctrl: Rename struct resctrl_cache to struct resctrl_ctrl_bitmap x86/resctrl: Add per-control and per-resource flags x86/resctrl: Add emulation controller list to resctrl_ctrl x86/resctrl: Parse ACPI MMRC table
x86/resctrl: Replace "msr" in monitoring data identifiers x86/resctrl: Introduce region aware MBM event definitions x86/resctrl: Introduce memory region based MBM read callback on MMIO
space
x86/resctrl: Enable the region based events by adding them into the
event
x86/resctrl: Rename msr_update to hw_update
x86/resctrl: Parse ACPI MARC table
fs/resctrl: Add region-based control names and
resctrl_ctrl_name_region()
x86/resctrl: Add region aware MBA controllers
x86/resctrl: Attach ACPI ERDT information to ctrl domain on CPU online x86/resctrl: Introduce region-based MBA write implementation on MMIO
space
x86/resctrl: Allow control writes from any CPU for MMIO controllers x86/resctrl: Enable region-aware MBM/MBA via the RDT_CTRL register x86/resctrl: Emulate the legacy MBA controller via the region MAX
controls
fs/resctrl: Expose emulation controllers in a resource_schemata subdir fs/resctrl: Fix excessive padding in schemata output x86,fs/resctrl: Update Documentation for region aware RDT
Tony Luck (1):
fs/resctrl: Do not invoke smp_processor_id() in preemptible context
Documentation/filesystems/resctrl.rst | 110 +++ arch/x86/Kconfig | 4 +- arch/x86/include/asm/apic.h | 1 + arch/x86/include/asm/resctrl.h | 11 +- arch/x86/kernel/cpu/resctrl/Makefile | 1 + arch/x86/kernel/cpu/resctrl/core.c | 258 +++---- arch/x86/kernel/cpu/resctrl/ctrlmondata.c | 106 ++- arch/x86/kernel/cpu/resctrl/erdt.c | 886 ++++++++++++++++++++++ arch/x86/kernel/cpu/resctrl/internal.h | 95 ++- arch/x86/kernel/cpu/resctrl/monitor.c | 82 +- arch/x86/kernel/cpu/resctrl/rdtgroup.c | 14 +- arch/x86/kernel/cpu/topology.c | 2 +- drivers/resctrl/mpam_resctrl.c | 18 +- fs/resctrl/ctrlmondata.c | 49 +- fs/resctrl/monitor.c | 73 +- fs/resctrl/pseudo_lock.c | 2 +- fs/resctrl/rdtgroup.c | 111 ++- include/linux/resctrl.h | 82 +- include/linux/resctrl_types.h | 21 +- 19 files changed, 1634 insertions(+), 292 deletions(-) create mode 100644 arch/x86/kernel/cpu/resctrl/erdt.c
Signature · 1 line
2.43.0