RFC resctrl: MBA control emulation and ARM MPAM MB_NODE support

This series enables MBA bandwidth control emulation on ARM MPAM when the default MB control is disabled, using a node-scoped MB_NODE control as the native backend. It extends memory-bandwidth monitoring and control to memory-level MSCs or CPU-less NUMA nodes, and adds documentation and tests.

Background

On x86, MBA bandwidth control is typically associated with the L3 cache:
schemata lines use L3 cache ids and monitor domains appear as mon_L3_XX. On some ARM MPAM systems the MBA control is backed by a memory-level MSC rather than the L3 cache MSC. A memory-level MSC is represented as a NUMA node, and that node commonly has no CPUs of its own (a memory-only node that still participates in bandwidth control). For such a resource the control scope is NODE, schemata identifiers are NUMA node ids, monitor domains are named mon_NODE_XX, and a node-scoped MB_NODE control may be the only control with usable bandwidth hardware.

Because the memory-level MSC lives on a CPU-less NUMA node, the driver must handle nodes with no local CPUs. mpam_ris_get_affinity() derives affinity from the component's NUMA node id, so a memory-only node would otherwise end up with an empty CPU mask and never be registered. When the derived mask is empty, the driver falls back to cpu_possible_mask (masked by MSC accessibility) so the node still gets a resctrl domain. The borrowed affinity is kept out of class->affinity: those CPUs already contribute through CPU-ful nodes in the class, and subtracting the borrowed mask on teardown would remove CPUs that other live nodes still depend on. Domain setup iterates all MSC components per CPU (keyed by component), so each component — including these CPU-less memory nodes — gets its own control and monitor domain.

Examples of NODE and emulation interfaces on MPAM.

**MB disabled, MB_NODE enabled** (``MB`` emulated by ``MB_NODE``)::

info/MB/resource_schemata/
├── mode # [legacy] native
└── MB/
├── scope # NODE
├── status # disabled
└── MB_NODE/
├── scope # NODE
└── status # enabled
schemata: MB_NODE and MB keeps same values

        MB_NODE:0=100;1=100;2=100;10=100;18=100;26=100;34=100;35=100
             MB:0=100;1=100;2=100;10=100;18=100;26=100;34=100;35=100
             L3:1=ffff;2=ffff

mon_data/

mon_L3_01  mon_NODE_00  mon_NODE_02  mon_NODE_18  mon_NODE_34
mon_L3_02  mon_NODE_01  mon_NODE_10  mon_NODE_26  mon_NODE_35

mon_data/mon_L3_01/llc_occupancy
mon_data/mon_NODE_01/mbm_total_bytes

Switch to native by writing to "mode":

info/MB/resource_schemata/
├── mode # legacy [native]
├── MB/
│ └── scope # NODE
│ └── status # disabled
└── MB_NODE/ # MB_NODE is moved to here
├── scope # NODE
└── status # enabled

schemata: Only show MB_NODE. No "MB:" is emulated.

        MB_NODE:0=100;1=100;2=100;10=100;18=100;26=100;34=100;35=100
             L3:1=ffff;2=ffff

mon_data/

mon_L3_01  mon_NODE_00  mon_NODE_02  mon_NODE_18  mon_NODE_34
mon_L3_02  mon_NODE_01  mon_NODE_10  mon_NODE_26  mon_NODE_35

mon_data/mon_L3_01/llc_occupancy
mon_data/mon_NODE_01/mbm_total_bytes

**MB enabled, MB_NODE enabled**::

Same for legacy and native:

info/MB/resource_schemata/
├── mode # [legacy] native
├── MB/
│ ├── scope # L3
│ └── status # enabled
└── MB_NODE/
├── scope # NODE
└── status # enabled
schemata:

        MB_NODE:0=100;1=100;2=100;10=100;18=100;26=100;34=100;35=100
             MB:1=100;2=100
             L3:1=ffff;2=ffff

mon_data/

mon_L3_01  mon_NODE_00  mon_NODE_02  mon_NODE_18  mon_NODE_34
mon_L3_02  mon_NODE_01  mon_NODE_10  mon_NODE_26  mon_NODE_35

mon_data/mon_L3_01/llc_occupancy
mon_data/mon_L3_01/mbm_total_bytes
mon_data/mon_NODE_01/mbm_total_bytes

The patches can be compiled but not tested on x86.

The 23 patches are grouped as follows:

1. Fixes for previous issues (patches 1–2)

Correct resource_schemata control subdirectory ownership on non-root resctrl mounts, and fix a NULL address access in the ARM MPAM driver.

2. resctrl control emulation (patches 3–8)

Introduce the MBA emulation framework in the resctrl core: a writable legacy/native mode under info/MB/resource_schemata/, per-control status and nested schemata layout, schemata mirroring for controls without MBW hardware, and subdir rebuild when the mode changes. Patch 8 documents the user-visible behaviour in Documentation/filesystems/resctrl.rst.

This group and group 4 are trying to implement the concept in discussion [1].

The main purpose of this group is to make CPU-less NUMA Node patches work.

This implementation is only limited to MPAM CPU-less NUMA Node emulation. It needs to be enhanced to make another other emulation work, X86 memory region or RISC-V emulation etc.

It's totally fine for me to either replace this implementation with another implementation or continue to enhance this implementition for control emulations.

3. Memory bandwidth monitoring and control on memory-level MSCs (patches 9–17)

Generalise the monitor infrastructure beyond hard-wired L3 paths:
de-hardcode L3 monitor plumbing, expose MBA MBM counter assignment, name node-scoped monitor domains mon_NODE_<id>, add a node-scope MBM total event, and make MBM handling resource-aware.

On the ARM MPAM side, add memory-level MSC and ABMC support, refine L3 topology and class selection, refactor domain setup to iterate all MSC components per CPU (keyed by component), and handle CPU-less NUMA memory nodes that have no local CPUs but still participate in MPAM bandwidth control.

4. MB_NODE emulates MB (patch 18)

Wire MB_NODE as the native backend that emulates a disabled MB control in the ARM MPAM driver.

5. Documentation (patches 19–21)

Document memory-level MB control and NUMA nodes in the arm64 MPAM guide (patch 19), NODE-scoped MBA domains and mon_NODE_* monitoring in the resctrl documentation (patch 20), and an MB_NODE emulation example on ARM MPAM (patch 21).

6. Tests (patches 22–23)

Add a KUnit test for CPU-less NUMA node affinity handling and a kselftest that exercises MB emulation mode switching, resource_schemata hierarchy, and schemata mirroring on ARM MPAM.

To apply the patches, on top of [2],

Testing

# KUnit (CPU-less affinity) ./tools/testing/kunit/kunit.py run test_mpam_devices

  # kselftest (MB emulation; requires ARM MPAM with disabled MB control)
  cd tools/testing/selftests/resctrl
  ./resctrl_tests -a mb_emulation_test

The branch: https://github.com/fyu1/linux/ emul.cpu_less.rfc/ which is on top of [2].

[1] [RFC] mpam,x86,fs/resctrl: Generic schema description Proof of Concept
https://lore.kernel.org/lkml/[email protected]/
[2] Reinette's resctrl control branch: git://git.kernel.org/pub/scm/linux/kernel/git/reinette/linux.git branch resctrl/controls_rfc_v1

Fenghua Yu (23):
resctrl: Fix ownership of resource_schemata control subdirectories
arm_mpam: Fix NULL address access issue
resctrl: Expose MBA resource_schemata mode sysfs
resctrl: Expose per-control status in resource_schemata
resctrl: Add nested resource_schemata support for emulated controls
resctrl: Mirror schemata for controls without MBW hardware
resctrl: Rebuild resource_schemata subdirs on MBA mode change
Documentation: resctrl: document MBA control emulation
resctrl: De-hardcode L3 monitor infrastructure
resctrl: Expose MBA MBM counter assignment sysfs
resctrl: name node-scoped monitor domains mon_NODE_<id>
resctrl: Add node-scope MBM total event
resctrl: Make MBM paths resource-aware
arm_mpam: Support memory-level MSCs and ABMC per class
arm_mpam: Refine L3 topology and class selection
arm_mpam: Include all MSC components during domain setup
arm_mpam: Handle CPU-less numa nodes
arm_mpam: Emulate MB control with node-scoped MB_NODE control
Documentation: arm64: mpam: document memory-level MB control and NUMA nodes
Documentation: resctrl: document NODE-scoped MBA domains and mon_NODE monitoring
Documentation: resctrl: document MB_NODE emulation example on ARM MPAM
arm_mpam: Add KUnit test for CPU-less NUMA node affinity selftests/resctrl: Add MB emulation test for ARM MPAM

 Documentation/arch/arm64/mpam.rst             | 105 ++--
 Documentation/filesystems/resctrl.rst         | 184 ++++++-
 arch/x86/kernel/cpu/resctrl/core.c            |   2 +
 drivers/resctrl/mpam_devices.c                |  59 +-
 drivers/resctrl/mpam_internal.h               |   8 +
 drivers/resctrl/mpam_resctrl.c                | 488 ++++++++++++-----
 drivers/resctrl/test_mpam_devices.c           |  67 +++
 fs/resctrl/ctrlmondata.c                      |  99 +++-
 fs/resctrl/internal.h                         |   9 +-
 fs/resctrl/monitor.c                          | 225 +++++---
 fs/resctrl/rdtgroup.c                         | 507 +++++++++++++++---
 include/linux/resctrl.h                       |  62 ++-
 include/linux/resctrl_types.h                 |  15 +-
 tools/testing/selftests/resctrl/config        |   1 +
 .../selftests/resctrl/mb_emulation_test.c     | 460 ++++++++++++++++
 tools/testing/selftests/resctrl/resctrl.h     |   1 +
 .../testing/selftests/resctrl/resctrl_tests.c |   1 +
 17 files changed, 1942 insertions(+), 351 deletions(-)
 create mode 100644 tools/testing/selftests/resctrl/mb_emulation_test.c

2.43.0