x86,fs/resctrl: Add kernel-mode (e.g., PLZA) support to the resctrl subsystem

Hi All,

This series adds support for AMD's Privilege-Level Zero Association (PLZA), which allows the CPU to use a different CLOSID and/or RMID association while executing in kernel mode than it does in user mode.

The implementation is integrated through a generic kernel mode ("kmode") framework in fs/resctrl, enabling future architectures to support similar functionality without requiring changes to the core resctrl code. The intent is to provide a common infrastructure that architecture-specific implementations can hook into, minimizing future resctrl core modifications.

The features are documented in:

AMD64 Zen6 Platform Quality of Service (PQOS) Extensions, Publication # 69193 Revision 1.00, Issue Date March 2026

available at https://bugzilla.kernel.org/show_bug.cgi?id=206537

The patches are based on top of commit (tip/master):

c6279551a44c Merge branch into tip/master: 'x86/sev'

Any feedback is appreciated.

Background
==========

When memory bandwidth associated with a CLOSID is aggressively throttled, and a task with that CLOSID moves into kernel mode, the kernel operations are also aggressively throttled. This can stall forward progress and eventually degrade overall system performance.

Privilege-Level Zero Association (PLZA) allows the user to specify a CLOSID and/or RMID for execution at Privilege Level Zero. When PLZA is enabled on a CPU, kernel work at PL0 uses the CLOSID and/or RMID from MSR PQR_PLZA_ASSOC; otherwise, the CPU uses the CLOSID and RMID from PQR_ASSOC.

Design
======

A new sysfs file, info/kernel_mode, selects the kernel mode for resource allocation and monitoring of kernel work and, when the mode needs one, the resource group that backs it. Reads list the supported modes, with the active mode in square brackets. Writes use the same line without the brackets. Look at the thread below for design discussion.
https://lore.kernel.org/lkml/[email protected]/
https://lore.kernel.org/lkml/[email protected]/

Two kernel modes are exposed:

- inherit_user: kernel work uses the allocation and monitoring associations of the user task on whose behalf it runs. It takes no parameters.

- global_enable_per_cpu: kernel work uses the allocation and/or monitoring association of a backing CTRL_MON or MON group. ctrl= and mon= select assign or inherit for each. group= names the group and must be the last parameter, because a group name may contain ';'.

When mon=assign, a CTRL_MON group is shown as <CTRL_MON>// and a MON group as <CTRL_MON>/<MON>/. When mon=inherit, or monitoring is unsupported, a CTRL_MON group is shown as <CTRL_MON>/.

An omitted ctrl=, mon=, or group= selects the default: assign where the platform supports it, and the default CTRL_MON group. A partial write therefore restores the omitted fields and programs every online CPU. ctrl=inherit together with mon=inherit is rejected. A monitor group requires mon=assign. A failed write leaves the current association in place.

Per-rdtgroup files kmode_cpus and kmode_cpus_list scope the association to a subset of online CPUs. They are visible only on the group that backs the active association. A write enables the association on CPUs added to the mask and disables it on CPUs removed from the mask. An empty mask disables the association on every currently online CPU. A new global_enable_per_cpu association sets the mask to every online CPU. A CPU that comes online later joins the association, including after a narrowed or empty mask. A CPU that goes offline leaves it only when that CPU is in the mask.

A group that backs the association, or a control group whose monitor group does, cannot change mode, including entering pseudo-lock setup. Reparenting a backing monitor group is rejected. The backing group cannot be renamed until the association is cleared.

The arch hook, resctrl_arch_configure_kmode_global(), keeps the fs/resctrl layer arch-neutral. enable=false disables the mode on the given CPUs and still writes the same CLOSID, RMID, and per-ID enables, which is what the PLZA domain rule requires. It does not by itself switch the active mode back to inherit_user.

resctrl_set_kmode_support() lets architecture code register each extra mode during resctrl initialization. inherit_user is registered by default. On AMD, PLZA registers global_enable_per_cpu when the feature is available and the platform can allocate or monitor.

Only AMD PLZA is wired up here; Intel and ARM can add their own support later by implementing the hooks.

Examples
========

(See Documentation/filesystems/resctrl.rst, "kernel_mode", "kmode_cpus", and "Examples on working with kernel_mode", for the full UAPI.)

# Mount resctrl

  # mount -t resctrl resctrl /sys/fs/resctrl
  # cd /sys/fs/resctrl

# Read the supported modes. The active mode is bracketed for display # only; do not include brackets when writing.

  # cat info/kernel_mode
  [inherit_user]
  global_enable_per_cpu:ctrl=assign;mon=assign;group=//

# Create a CTRL_MON group and back kernel-mode allocation with it. # mon=inherit, so the group is shown as <CTRL_MON>/.

  # mkdir ctrl1
  # echo "global_enable_per_cpu:ctrl=assign;mon=inherit;group=ctrl1/" \
          > info/kernel_mode
  # cat info/kernel_mode
  inherit_user
  [global_enable_per_cpu:ctrl=assign;mon=inherit;group=ctrl1/]

  # kmode_cpus and kmode_cpus_list are visible only on the backing group.
  # ls ctrl1/kmode_cpus*
  ctrl1/kmode_cpus  ctrl1/kmode_cpus_list

# Restrict the association to a CPU subset. The association is enabled

  # on CPUs added to the mask and disabled on CPUs removed from it.
  # echo 0-3 > ctrl1/kmode_cpus_list
  # cat ctrl1/kmode_cpus
  f
  # cat ctrl1/kmode_cpus_list
  0-3

# Return to inherit_user. The inactive global_enable_per_cpu line

  # shows the defaults again.
  # echo "inherit_user" > info/kernel_mode
  # cat info/kernel_mode
  [inherit_user]
  global_enable_per_cpu:ctrl=assign;mon=assign;group=//

Tested on AMD with PLZA; builds on x86 without PLZA and does not expose global_enable_per_cpu unless the feature is available.

Layout
======

01-03 x86: PLZA CPU feature, command-line option, and MSR definitions. 04-07 fs/resctrl: kernel mode enum, arch hook, kmode state, and
resctrl_set_kmode_support().
08 x86/resctrl: register global_enable_per_cpu when PLZA is
available.
09 fs/resctrl: document last_cmd_status under a top-level info
section.

  10     fs/resctrl: info/kernel_mode read interface.
  11     fs/resctrl: hidden rdtgroup files.
  12     fs/resctrl: per-rdtgroup kmode_cpus/kmode_cpus_list.
  13     x86,fs/resctrl: CPU hotplug for the active association, and
         clear PLZA across kexec.
  14     fs/resctrl: deactivate the association when a group is removed.
  15     fs/resctrl: info/kernel_mode write interface.
  16     fs/resctrl: reject mode changes and reparenting while a group
         backs the association.
  17     fs/resctrl: show or hide kmode_cpus on the backing group.
  18     fs/resctrl: kmode_cpus/kmode_cpus_list writes.
  19     fs/resctrl: documentation and examples.

Changelog
=========
v7:
- Used "CLOSID and/or RMID association" throughout the series because feature operates on the MSR_IA32_PQR_PLZA_ASSOC association register. The association concept is already known in the resctrl subsystem. Please let me know if you see a better alternative.
- Rebase onto tip/master c6279551a44c ("Merge branch into tip/master:

    'x86/sev'").

- Document last_cmd_status in a new "Top level files" section of the info directory, before kernel_mode.
- info/kernel_mode includes the monitor component of group= only when mon=assign. A CTRL_MON group then reads as <CTRL_MON>//. When mon=inherit, or monitoring is unsupported, it reads as <CTRL_MON>/.
- An omitted ctrl=, mon=, or group= selects the defaults (assign where the platform supports it, and the default CTRL_MON group). A partial write restores the omitted fields and programs every online CPU.
- Reject ctrl=inherit together with mon=inherit. With both enables clear, CPL0 keeps the user task's IDs, which is inherit_user.
- group= is the last parameter, so a group name may contain ';'.
- Drop resctrl_kmode_switch(). Deactivation restores inherit_user and the default ctrl and mon values.
- Move kmode_cpus visibility to after activation and deactivation, and move the mode-change and reparent checks to after the kernel_mode write interface.
- Clear MSR_IA32_PQR_PLZA_ASSOC from clear_closid_rmid() so a kexec boot does not keep a previous kernel's PLZA state.

v6:
- Addressed all the comments as discussed in v5. Anything missing is not intentional. Please feel free to comment on any mistake.
- Rename assign_global_enable_per_cpu to global_enable_per_cpu and describe the interface as a kernel mode rather than a policy.
- Flatten kernel mode state into resctrl_kmode. The arch hook is resctrl_arch_configure_kmode_global().
- An omitted ctrl=, mon=, or group= keeps the value shown by reading info/kernel_mode. inherit_user takes no parameters. ctrl=inherit;mon=inherit remains global_enable_per_cpu.
- Selecting a new global_enable_per_cpu association sets that group's kmode_cpus to every online CPU. A CPU that comes online later joins the association even if user space had removed it from the mask.
- kmode_cpus files are created hidden and shown only while the group backs the active association.
- A monitor group requires mon=assign. Removing a control group detaches a child that backs the association before the parent's CLOSID is freed.
- Reject a mode change when the group or one of its monitor groups backs the association.
- Split kmode_cpus visibility and the mode-change rejection into their own patches.
- Use the parsing similar to ctrlmondata.c:parse_line(), ctrlmondata.c:resctrl_io_alloc_parse_line(), monitor.c:resctrl_parse_mbm_assignment().
- Re-wrote the changelog as a feature and not as a bug-fix.
- Remove implementation details in function headers.

v5:
- Collapse the two v4 global-assign modes into a single assign_global_enable_per_cpu policy with ctrl= and mon= options.
- Use "association" terminology consistently in code, errors, and documentation.
- Register assign_global_enable_per_cpu once during x86 resource discovery when PLZA is available.
- Reject mon=inherit when group= selects a monitor group.
- Split hidden-file support, deactivation-on-teardown, and PLZA mode registration into separate patches for easier review.
- Fix msr_pqr_plza_assoc truncation on 32-bit systems by using u64.
- Refresh documentation and add a readable end-to-end example walkthrough.
- Block pseudo-lock setup on kernel-mode-associated groups.
- Reject mode changes and rename while a group backs the active association.

v4:
- Reorder and split the series into 15 patches: separate read-only info/kernel_mode display from the write path; add hotplug support when a CPU comes online; add an end-to-end documentation/examples patch.
- Introduced resctrl_set_kmode_support() so architecture code can register supported kernel-mode policies during resctrl initialization.
- info/kernel_mode write: validate group type; run fail paths before tearing down the active association so errors retain the old state.
- kmode_cpus / kmode_cpus_list: writable with incremental enable/disable deltas; empty masks allowed; offline CPUs rejected.
- Hotplug: newly online CPUs are added to the associated group's kmode_cpu_mask and programmed when assign_global_enable_per_cpu is active.

v3:
- Generalise the layer beyond AMD: rename "PLZA mode" to "kernel mode" (kmode) in code, sysfs, and Documentation.
- Reset the association when the associated rdtgroup is removed, instead of leaving stale state.

v2:
- Similar to RFC with a new proposal; interface names were not final.
- Separated Global Bandwidth Enforcement (GLBE) from PLZA; this series only adds PLZA support.
- Used "kmode" instead of "PLZA" in the generic layer.

Previous versions:
v6: https://lore.kernel.org/lkml/[email protected]/
v5: https://lore.kernel.org/lkml/[email protected]/
v4: https://lore.kernel.org/lkml/[email protected]/
v3: https://lore.kernel.org/lkml/[email protected]/
v2: https://lore.kernel.org/lkml/[email protected]/
v1: https://lore.kernel.org/lkml/[email protected]/

Babu Moger (19):
x86/cpufeatures: Support Privilege Level Zero Association (PLZA) x86/resctrl: Add PLZA support to command-line options x86/resctrl: Add PLZA MSR definition and field layout fs/resctrl: Introduce kernel mode enum arm,x86,fs/resctrl: Introduce arch hook to program global per-CPU
kernel mode
fs/resctrl: Introduce data structures for kernel mode configuration fs/resctrl: Introduce resctrl_set_kmode_support() to register
supported modes
x86/resctrl: Register RESCTRL_GLOBAL_ENABLE_PER_CPU when PLZA is
available
fs/resctrl: Rearrange top-level files in info directory fs/resctrl: Add interface to display kernel mode status fs/resctrl: Add support for hidden resource group files fs/resctrl: Introduce kmode_cpus/kmode_cpus_list per rdtgroup x86,fs/resctrl: Add CPU hotplug support for kernel mode fs/resctrl: Deactivate kernel mode when the group is removed fs/resctrl: Extend info/kernel_mode to configure the kernel mode fs/resctrl: Reject mode changes for groups backing kernel mode fs/resctrl: Control visibility of global per-CPU kernel mode group
files
fs/resctrl: Allow userspace to write kmode_cpus/kmode_cpus_list fs/resctrl: Add kernel mode documentation and examples

 .../admin-guide/kernel-parameters.txt         |   2 +-
 Documentation/filesystems/resctrl.rst         | 255 ++++++-
 arch/x86/include/asm/cpufeatures.h            |   1 +
 arch/x86/include/asm/msr-index.h              |   1 +
 arch/x86/kernel/cpu/resctrl/core.c            |   9 +
 arch/x86/kernel/cpu/resctrl/ctrlmondata.c     |  38 +
 arch/x86/kernel/cpu/resctrl/internal.h        |  49 ++
 arch/x86/kernel/cpu/scattered.c               |   1 +
 drivers/resctrl/mpam_resctrl.c                |   7 +
 fs/resctrl/internal.h                         |  32 +
 fs/resctrl/rdtgroup.c                         | 693 ++++++++++++++++++
 include/linux/resctrl.h                       |  65 ++
 12 files changed, 1141 insertions(+), 12 deletions(-)

2.43.0