x86,fs/resctrl: Add kernel-mode (e.g., PLZA) support to the resctrl subsystem

When memory bandwidth associated with a CLOSID is aggressively throttled, and a task with that CLOSID moves into kernel mode, the kernel operations are also aggressively throttled. This can stall forward progress and eventually degrade overall system performance.

AMD hardware supports a feature Privilege Level Zero Association (PLZA), which allows the CPU's CLOSID and/or RMID association to be changed during the transition from user mode to kernel mode. This allows the kernel to run using a different CLOSID and/or RMID than user space, which can improve system performance and/or allow kernel work to be monitored separately.

Note that PLZA feature associates CLOSID and/or RMID to kernel work, and therefore influences resource allocation and/or monitoring for all resources, not only memory bandwidth.

The feature is detected via CPUID_Fn80000020_EBX_x00 [Bit 9]: Privilege Level Zero Association (PLZA).

The PLZA feature details are documented in [1] available from [2].

[1] AMD64 Zen6 Platform Quality of Service (PQOS) Extensions:

    Publication # 69193 Revision: 1.00, Issue Date: March 2026

Signed-off-by Babu Moger <[email protected]>

Acked-by Borislav Petkov (AMD) <[email protected]>

Link https://bugzilla.kernel.org/show_bug.cgi?id=206537 # [2]

---

Hi Babu,

Quoted 23 lines · Babu Moger

When memory bandwidth associated with a CLOSID is aggressively throttled, and a task with that CLOSID moves into kernel mode, the kernel operations are also aggressively throttled. This can stall forward progress and eventually degrade overall system performance.

AMD hardware supports a feature Privilege Level Zero Association (PLZA), which allows the CPU's CLOSID and/or RMID association to be changed during the transition from user mode to kernel mode. This allows the kernel to run using a different CLOSID and/or RMID than user space, which can improve system performance and/or allow kernel work to be monitored separately.

Note that PLZA feature associates CLOSID and/or RMID to kernel work, and therefore influences resource allocation and/or monitoring for all resources, not only memory bandwidth.

The feature is detected via CPUID_Fn80000020_EBX_x00 [Bit 9]: Privilege Level Zero Association (PLZA).

The PLZA feature details are documented in [1] available from [2].

[1] AMD64 Zen6 Platform Quality of Service (PQOS) Extensions:

     Publication # 69193 Revision: 1.00, Issue Date: March 2026

Quoted 4 lines

Signed-off-by Babu Moger <[email protected]>

Acked-by Borislav Petkov (AMD) <[email protected]>

Link https://bugzilla.kernel.org/show_bug.cgi?id=206537 # [2]

---

Reviewed-by Reinette Chatre <[email protected]>

Add a kernel command-line parameter to enable or disable the exposure of the PLZA (Privilege Level Zero Association) feature to resctrl.

This allows administrators to control PLZA visibility at boot time via the rdt= option (e.g. rdt=plza or rdt=!plza).

Add "plza" to the rdt= command-line parameter so PLZA exposure to resctrl can be enabled or disabled at boot. PLZA is exposed by default when supported; rdt=!plza suppresses it.

Signed-off-by Babu Moger <[email protected]>

---

Hi Babu,

Quoted 6 lines · Babu Moger

Add a kernel command-line parameter to enable or disable the exposure of the PLZA (Privilege Level Zero Association) feature to resctrl.

This allows administrators to control PLZA visibility at boot time via the rdt= option (e.g. rdt=plza or rdt=!plza).

Apologies for not being clear in previous review. This changelog has a lot of repetition. Above two paragraphs are redundant, paragraph below is sufficient by itself. One possible change would be to spell out PLZA in the paragraph below. Spelling PLZA out may not be necessary since it is spelled out in the patch self, so up to you.

Quoted 7 lines

Add "plza" to the rdt= command-line parameter so PLZA exposure to resctrl can be enabled or disabled at boot. PLZA is exposed by default when supported; rdt=!plza suppresses it.

Signed-off-by Babu Moger <[email protected]>

---

With first two paragraphs of changelog removed:
| Reviewed-by: Reinette Chatre <[email protected]>

Privilege Level Zero Association (PLZA) enables the kernel to switch to a different CLOSID and/or RMID than those used in user mode when entering kernel mode. CLOSID and RMID selection are independent and can be configured separately.

PLZA associations are programmed on a per-CPU basis through MSR_IA32_PQR_PLZA_ASSOC (0xc00003fc), which contains the kernel RMID and CLOSID values along with their corresponding enable bits.

Make the register and its field layout available so kernel mode allocation and monitoring associations can be configured independently.

Signed-off-by Babu Moger <[email protected]>

---

---
 arch/x86/include/asm/msr-index.h       |  1 +
 arch/x86/kernel/cpu/resctrl/internal.h | 49 ++++++++++++++++++++++++++
 2 files changed, 50 insertions(+)

arch/x86/include/asm/msr-index.h +1 -0

index 82ca6356dc62..1331291f7a36 100644
@@ -1304,6 +1304,7 @@
 /* - AMD: */
 #define MSR_IA32_MBA_BW_BASE		0xc0000200
 #define MSR_IA32_SMBA_BW_BASE		0xc0000280
+#define MSR_IA32_PQR_PLZA_ASSOC		0xc00003fc
 #define MSR_IA32_L3_QOS_ABMC_CFG	0xc00003fd
 #define MSR_IA32_L3_QOS_EXT_CFG		0xc00003ff
 #define MSR_IA32_EVT_CFG_BASE		0xc0000400

arch/x86/kernel/cpu/resctrl/internal.h +22 -0

index e3cfa0c10e92..44ecde20b1b4 100644
@@ -222,6 +222,55 @@ union l3_qos_abmc_cfg {
 	unsigned long full;
 };
 
+/*
+ * Privilege Level Zero Association (PLZA) is configured via
+ * MSR_IA32_PQR_PLZA_ASSOC.
+ *
+ * @rmid		: RMID programmed for PLZA.
+ * @reserved1		: Reserved.
+ * @rmid_en		: Enable RMID for PLZA.
+ * @closid		: CLOSID programmed for PLZA.
+ * @reserved2		: Reserved.
+ * @closid_en		: Enable CLOSID for PLZA.
+ * @reserved3		: Reserved.
+ * @plza_en		: Enable PLZA. When enabled, PLZA applies to the
+ *			  given CPU.
+ *
+ * The MSR_IA32_PQR_PLZA_ASSOC must be programmed on every CPU in a QoS
+ * domain. Each domain supports a single PLZA CLOSID and/or RMID
+ * association, and all fields except @plza_en must be programmed
+ * identically across the domain to ensure consistent behavior.
+ *
+ * CLOSID and RMID selections are controlled independently. When PLZA is
+ * enabled (@plza_en = 1), CPL0 uses CLOSID and RMID values from
+ * MSR_IA32_PQR_PLZA_ASSOC depending on the corresponding enable bits. When

arch/x86/kernel/cpu/resctrl/internal.h +10 -0

+ * when PLZA is disabled, with RMID and CLOSID used from MSR_IA32_PQR_ASSOC
+ * irrespective of privilege level.
+ */
+union msr_pqr_plza_assoc {
+	struct {
+		u64	rmid		:12,
+			reserved1	:19,
+			rmid_en		: 1,
+			closid		: 4,
+			reserved2	:11,

Hi Babu,

nit: subject claims multiple "definitions" while only adding one #define.

Quoted 4 lines

Privilege Level Zero Association (PLZA) enables the kernel to switch to a different CLOSID and/or RMID than those used in user mode when entering kernel mode. CLOSID and RMID selection are independent and can be configured separately.

That CLOSID and RMID are independent on x86 is clear to me. That they can be configured separately is not clear to me in this context. What does this "separate configuration" refer to? Looking at the later patches kernel mode allocation and monitoring associations are always configured together and written to same MSR at the same time?

Quoted 6 lines

PLZA associations are programmed on a per-CPU basis through MSR_IA32_PQR_PLZA_ASSOC (0xc00003fc), which contains the kernel RMID and CLOSID values along with their corresponding enable bits.

Quoted 2 lines

Make the register and its field layout available so kernel mode allocation and monitoring associations can be configured independently.

Could you please also elaborate what "configured independently" refers to? At this point it specifically refers to kernel mode and from what I understand kernel mode of allocation and monitoring are configured together.

Quoted 4 lines

Signed-off-by Babu Moger <[email protected]>

---

...

Quoted 51 lines

---
  arch/x86/include/asm/msr-index.h       |  1 +
  arch/x86/kernel/cpu/resctrl/internal.h | 49 ++++++++++++++++++++++++++
  2 files changed, 50 insertions(+)

arch/x86/include/asm/msr-index.h +1 -0

index 82ca6356dc62..1331291f7a36 100644
@@ -1304,6 +1304,7 @@
  /* - AMD: */
  #define MSR_IA32_MBA_BW_BASE		0xc0000200
  #define MSR_IA32_SMBA_BW_BASE		0xc0000280
+#define MSR_IA32_PQR_PLZA_ASSOC		0xc00003fc
  #define MSR_IA32_L3_QOS_ABMC_CFG	0xc00003fd
  #define MSR_IA32_L3_QOS_EXT_CFG		0xc00003ff
  #define MSR_IA32_EVT_CFG_BASE		0xc0000400

arch/x86/kernel/cpu/resctrl/internal.h +26 -0

index e3cfa0c10e92..44ecde20b1b4 100644
@@ -222,6 +222,55 @@ union l3_qos_abmc_cfg {
  	unsigned long full;
  };
 
+/*
+ * Privilege Level Zero Association (PLZA) is configured via
+ * MSR_IA32_PQR_PLZA_ASSOC.
+ *
+ * @rmid		: RMID programmed for PLZA.
+ * @reserved1		: Reserved.
+ * @rmid_en		: Enable RMID for PLZA.
+ * @closid		: CLOSID programmed for PLZA.
+ * @reserved2		: Reserved.
+ * @closid_en		: Enable CLOSID for PLZA.
+ * @reserved3		: Reserved.
+ * @plza_en		: Enable PLZA. When enabled, PLZA applies to the
+ *			  given CPU.
+ *
+ * The MSR_IA32_PQR_PLZA_ASSOC must be programmed on every CPU in a QoS
+ * domain. Each domain supports a single PLZA CLOSID and/or RMID
+ * association, and all fields except @plza_en must be programmed
+ * identically across the domain to ensure consistent behavior.
+ *
+ * CLOSID and RMID selections are controlled independently. When PLZA is
+ * enabled (@plza_en = 1), CPL0 uses CLOSID and RMID values from
+ * MSR_IA32_PQR_PLZA_ASSOC depending on the corresponding enable bits. When
+ * PLZA is disabled (@plza_en = 0), the CLOSID and RMID are always taken
+ * from MSR_IA32_PQR_ASSOC regardless of privilege level.
+ *
+ * - @rmid_en = 1: use the RMID from MSR_IA32_PQR_PLZA_ASSOC if @plza_en = 1.

arch/x86/kernel/cpu/resctrl/internal.h union l3_qos_abmc_cfg { · line 256

Quoted 6 lines

arch/x86/kernel/cpu/resctrl/internal.h +6 -0

+ * - @rmid_en = 0: use the RMID from MSR_IA32_PQR_ASSOC.
+ * - @closid_en = 1: use the CLOSID from MSR_IA32_PQR_PLZA_ASSOC if @plza_en = 1.
+ * - @closid_en = 0: use the CLOSID from MSR_IA32_PQR_ASSOC.
+ *
+ * This allows PLZA to override only RMID, only CLOSID, or both. If both
+ * rmid_en and closid_en are clear, the effective behavior is the same as

nit: rmid_en -> @rmid_en, closid_en -> @closid_en

Quoted 11 lines · Reinette Chatre

Quoted 4 lines · Babu Moger

Privilege Level Zero Association (PLZA) enables the kernel to switch to a different CLOSID and/or RMID than those used in user mode when entering kernel mode. CLOSID and RMID selection are independent and can be configured separately.

That CLOSID and RMID are independent on x86 is clear to me. That they can be configured separately is not clear to me in this context. What does this "separate configuration" refer to? Looking at the later patches kernel mode allocation and monitoring associations are always configured together and written to same MSR at the same time?

Good point. Its confusing. What I meant is that allocation (CLOSID) and monitoring (RMID) associations are independent PLZA features and can be enabled independently in hardware. For example, a platform may enable kernel mode CLOSID association without enabling kernel mode RMID association, or vice versa.

Yes. The kernel interface configures both together because they are carried in the same IA32_PQR_ASSOC MSR update.

How about?

Privilege Level Zero Association (PLZA) enables the kernel to use a different CLOSID and/or RMID in kernel mode than in user mode. Allocation (CLOSID) and monitoring (RMID) associations are independent PLZA features and may be enabled independently by hardware.

Hi Babu,

Quoted 22 lines · Babu Moger

Quoted 11 lines · Reinette Chatre

Quoted 4 lines · Babu Moger

Privilege Level Zero Association (PLZA) enables the kernel to switch to a different CLOSID and/or RMID than those used in user mode when entering kernel mode. CLOSID and RMID selection are independent and can be configured separately.

That CLOSID and RMID are independent on x86 is clear to me. That they can be configured separately is not clear to me in this context. What does this "separate configuration" refer to? Looking at the later patches kernel mode allocation and monitoring associations are always configured together and written to same MSR at the same time?

Good point. Its confusing. What I meant is that allocation (CLOSID) and monitoring (RMID) associations are independent PLZA features and can be enabled independently in hardware. For example, a platform may enable kernel mode CLOSID association without enabling kernel mode RMID association, or vice versa.

Yes. The kernel interface configures both together because they are carried in the same IA32_PQR_ASSOC MSR update.

Quoted 7 lines

How about?

Privilege Level Zero Association (PLZA) enables the kernel to use a different CLOSID and/or RMID in kernel mode than in user mode. Allocation (CLOSID) and monitoring (RMID) associations are independent PLZA features and may be enabled independently by hardware.

Which CLOSID (MSR_IA32_PQR_PLZA_ASSOC.closid) and/or RMID (MSR_IA32_PQR_PLZA_ASSOC.rmid) is used in kernel mode is different from whether the CLOSID (MSR_IA32_PQR_PLZA_ASSOC.closid_en) and/or RMID (MSR_IA32_PQR_PLZA_ASSOC.rmid_en) is used in kernel mode.

This series uses the high level "allocation and monitoring association" term interchangeably for both of these meanings and doing so causes confusion.

Please pick specific terms to refer to the CLOSID/RMID assigned to kernel mode vs whether the CLOSID/RMID assignment is enabled and use it throughout the series.

Quoted 35 lines · Reinette Chatre

Quoted 22 lines · Babu Moger

Quoted 11 lines · Reinette Chatre

Quoted 4 lines · Babu Moger

Privilege Level Zero Association (PLZA) enables the kernel to switch to a different CLOSID and/or RMID than those used in user mode when entering kernel mode. CLOSID and RMID selection are independent and can be configured separately.

That CLOSID and RMID are independent on x86 is clear to me. That they can be configured separately is not clear to me in this context. What does this "separate configuration" refer to? Looking at the later patches kernel mode allocation and monitoring associations are always configured together and written to same MSR at the same time?

Good point. Its confusing.  What I meant is that allocation (CLOSID) and monitoring (RMID) associations are independent PLZA features and can be enabled independently in hardware. For example, a platform may enable kernel mode CLOSID association without enabling kernel mode RMID association, or vice versa.

Yes. The kernel interface configures both together because they are carried in the same IA32_PQR_ASSOC MSR update.

Quoted 7 lines

How about?

Privilege Level Zero Association (PLZA) enables the kernel to use a different CLOSID and/or RMID in kernel mode than in user mode. Allocation (CLOSID) and monitoring (RMID) associations are independent PLZA features and may be enabled independently by hardware.

Which CLOSID (MSR_IA32_PQR_PLZA_ASSOC.closid) and/or RMID (MSR_IA32_PQR_PLZA_ASSOC.rmid) is used in kernel mode is different from whether the CLOSID (MSR_IA32_PQR_PLZA_ASSOC.closid_en) and/or RMID (MSR_IA32_PQR_PLZA_ASSOC.rmid_en) is used in kernel mode.

Quoted 6 lines

This series uses the high level "allocation and monitoring association" term interchangeably for both of these meanings and doing so causes confusion.

Please pick specific terms to refer to the CLOSID/RMID assigned to kernel mode vs whether the CLOSID/RMID assignment is enabled and use it throughout the series.

Another attempt to capture all the details:

"Privilege Level Zero Association (PLZA) enables the kernel to use a different CLOSID and/or RMID than those used in user mode when executing in kernel mode. CLOSID and RMID selection can be configured independently.

The kernel mode CLOSID and RMID are specified by MSR_IA32_PQR_PLZA_ASSOC.closid and MSR_IA32_PQR_PLZA_ASSOC.rmid, respectively. plza_en activates the PLZA operation, while closid_en and rmid_en independently control whether the corresponding kernel mode CLOSID or RMID is used.

Make the register and its field layout available so kernel mode allocation (CLOSID) and monitoring (RMID) selection can be configured independently."

Hi Babu,

Quoted 45 lines · Babu Moger

Quoted 35 lines · Reinette Chatre

Quoted 22 lines · Babu Moger

Quoted 11 lines · Reinette Chatre

Quoted 4 lines · Babu Moger

Privilege Level Zero Association (PLZA) enables the kernel to switch to a different CLOSID and/or RMID than those used in user mode when entering kernel mode. CLOSID and RMID selection are independent and can be configured separately.

That CLOSID and RMID are independent on x86 is clear to me. That they can be configured separately is not clear to me in this context. What does this "separate configuration" refer to? Looking at the later patches kernel mode allocation and monitoring associations are always configured together and written to same MSR at the same time?

Good point. Its confusing.  What I meant is that allocation (CLOSID) and monitoring (RMID) associations are independent PLZA features and can be enabled independently in hardware. For example, a platform may enable kernel mode CLOSID association without enabling kernel mode RMID association, or vice versa.

Yes. The kernel interface configures both together because they are carried in the same IA32_PQR_ASSOC MSR update.

Quoted 7 lines

How about?

Privilege Level Zero Association (PLZA) enables the kernel to use a different CLOSID and/or RMID in kernel mode than in user mode. Allocation (CLOSID) and monitoring (RMID) associations are independent PLZA features and may be enabled independently by hardware.

Which CLOSID (MSR_IA32_PQR_PLZA_ASSOC.closid) and/or RMID (MSR_IA32_PQR_PLZA_ASSOC.rmid) is used in kernel mode is different from whether the CLOSID (MSR_IA32_PQR_PLZA_ASSOC.closid_en) and/or RMID (MSR_IA32_PQR_PLZA_ASSOC.rmid_en) is used in kernel mode.

Quoted 6 lines

This series uses the high level "allocation and monitoring association" term interchangeably for both of these meanings and doing so causes confusion.

Please pick specific terms to refer to the CLOSID/RMID assigned to kernel mode vs whether the CLOSID/RMID assignment is enabled and use it throughout the series.

Another attempt to capture all the details:

Quoted 4 lines

"Privilege Level Zero Association (PLZA) enables the kernel to use a different CLOSID and/or RMID than those used in user mode when executing in kernel mode. CLOSID and RMID selection can be configured independently.

First sentence already contains "and/or" so I think the last sentence can just be dropped.

Quoted 6 lines

The kernel mode CLOSID and RMID are specified by MSR_IA32_PQR_PLZA_ASSOC.closid and MSR_IA32_PQR_PLZA_ASSOC.rmid, respectively. plza_en activates the PLZA operation, while closid_en and rmid_en independently control whether the corresponding kernel mode CLOSID or RMID is used.

Please aim to provide higher level description than what can be seen from the patch.

Quoted 4 lines

Make the register and its field layout available so kernel mode allocation (CLOSID) and monitoring (RMID) selection can be configured independently."

Apologies but I really do not understand why the "independently" distinction is needed since these values are always programmed together.

Here is an attempt, but it is written without knowing which terms you have picked to stick with:

x86/resctrl: Add PLZA MSR definition and field layout

Privilege Level Zero Association (PLZA) enables a CPU to use a different CLOSID and/or RMID when executing in kernel mode than those used in user mode.

PLZA is programmed per CPU through MSR_IA32_PQR_PLZA_ASSOC, which contains the kernel mode CLOSID and RMID, an enable bit for each, and a bit that enables PLZA on the CPU. All fields except the PLZA enable bit must be programmed identically on every CPU in a domain.

Make the register and its field layout available in preparation for resctrl support of kernel mode allocation and monitoring.

Kernel mode work can use different allocation and monitoring associations

than the originating user task. On x86, Privilege Level Zero Association (PLZA) enables the kernel to switch to a different CLOSID and/or RMID when entering kernel mode.

Architectures need a common way to name kernel modes before resctrl can report what is active or what the platform supports.

Introduce enum resctrl_kernel_mode:
- RESCTRL_INHERIT_USER: Kernel mode work inherits allocation and monitoring from the user task.
- RESCTRL_GLOBAL_ENABLE_PER_CPU: Kernel mode work uses allocation and/or monitoring associations that differ from the user task.

Signed-off-by Babu Moger <[email protected]>

---

---
 include/linux/resctrl.h | 29 +++++++++++++++++++++++++++++
 1 file changed, 29 insertions(+)

include/linux/resctrl.h +9 -0

index 10dfdca7f4bf..f6adff7f4e1f 100644
@@ -723,6 +723,35 @@ int resctrl_arch_io_alloc_enable(struct rdt_resource *r, bool enable);
  */
 bool resctrl_arch_get_io_alloc_enabled(struct rdt_resource *r);
 
+/**
+ * enum resctrl_kernel_mode - Kernel mode control and monitoring association.
+ *
+ * @RESCTRL_INHERIT_USER:
+ *     Kernel mode inherits both allocation and monitoring associations
+ *     from the user-space task. On x86, the kernel uses the same CLOSID and
+ *     RMID as the user-space task.
+ *
+ * @RESCTRL_GLOBAL_ENABLE_PER_CPU:

Hi Babu,

Quoted 1 line · Babu Moger

Kernel mode work can use different allocation and monitoring associations

Here I think "different allocation and monitoring associations" means different CLOSID/RMID, while other places "different allocation and monitoring associations" means different MSR_IA32_PQR_PLZA_ASSOC.closid_en and MSR_IA32_PQR_PLZA_ASSOC.rmid_en?

Quoted 8 lines

than the originating user task. On x86, Privilege Level Zero Association (PLZA) enables the kernel to switch to a different CLOSID and/or RMID when entering kernel mode.

Architectures need a common way to name kernel modes before resctrl can report what is active or what the platform supports.

Quoted 5 lines

Introduce enum resctrl_kernel_mode:
- RESCTRL_INHERIT_USER: Kernel mode work inherits allocation and monitoring from the user task.
- RESCTRL_GLOBAL_ENABLE_PER_CPU: Kernel mode work uses allocation and/or monitoring associations that differ from the user task.

As written this is not accurate because there is nothing that prevents the "kernel mode group" from being the same as a task's own group?

Could the description of RESCTRL_GLOBAL_ENABLE_PER_CPU be made more specific to include that (a) it is a single (global) association that applies to *all* user space tasks, and (b) it can be enabled/disabled on each CPU? With these two additional details reader will more easily understand why the name of this mode is "RESCTRL_GLOBAL_ENABLE_PER_CPU".

Quoted 4 lines

Signed-off-by Babu Moger <[email protected]>

---

...

Quoted 21 lines

---
  include/linux/resctrl.h | 29 +++++++++++++++++++++++++++++
  1 file changed, 29 insertions(+)

include/linux/resctrl.h +9 -0

index 10dfdca7f4bf..f6adff7f4e1f 100644
@@ -723,6 +723,35 @@ int resctrl_arch_io_alloc_enable(struct rdt_resource *r, bool enable);
   */
  bool resctrl_arch_get_io_alloc_enabled(struct rdt_resource *r);
 
+/**
+ * enum resctrl_kernel_mode - Kernel mode control and monitoring association.
+ *
+ * @RESCTRL_INHERIT_USER:
+ *     Kernel mode inherits both allocation and monitoring associations
+ *     from the user-space task. On x86, the kernel uses the same CLOSID and
+ *     RMID as the user-space task.
+ *
+ * @RESCTRL_GLOBAL_ENABLE_PER_CPU:

include/linux/resctrl.h int resctrl_arch_io_alloc_enable(struct rdt_resource *r, bool enable); · line 744

Quoted 10 lines

include/linux/resctrl.h +10 -0

+ *     All kernel mode work uses the same allocation and/or monitoring
+ *     association via an assigned CTRL_MON or MON group. The mode can
+ *     be enabled/disabled per-CPU. When the mode is disabled on a CPU
+ *     then the kernel work on that CPU inherits the allocation and
+ *     monitoring association from the user-space task, effectively the
+ *     kernel mode becomes RESCTRL_INHERIT_USER on that CPU.
+ *
+ *     All online CPUs are included by default. A subset may be selected
+ *     through the resctrl group interface, and a CTRL_MON or MON group
+ *     may be assigned to this mode. On x86, CLOSID and RMID are configured

The sentence starting with "A subset ..." repeats what is described in the first paragraph so it could just be dropped? This leaves the first sentence of second paragraph out of place, how about merging it into first paragraph like:

The mode can be enabled/disabled per-CPU, all online CPUs are enabled by default.

Quoted 7 lines · Reinette Chatre

Quoted 1 line · Babu Moger

Kernel mode work can use different allocation and monitoring associations

Here I think "different allocation and monitoring associations" means different CLOSID/RMID, while other places "different allocation and monitoring associations" means different MSR_IA32_PQR_PLZA_ASSOC.closid_en and MSR_IA32_PQR_PLZA_ASSOC.rmid_en?

That is correct. Hope text here is fine.

Hi Babu,

Quoted 9 lines · Babu Moger

Quoted 7 lines · Reinette Chatre

Quoted 1 line · Babu Moger

Kernel mode work can use different allocation and monitoring associations

Here I think "different allocation and monitoring associations" means different CLOSID/RMID, while other places "different allocation and monitoring associations" means different MSR_IA32_PQR_PLZA_ASSOC.closid_en and MSR_IA32_PQR_PLZA_ASSOC.rmid_en?

Quoted 2 lines

That is correct. Hope text here is fine.

I had to guess. This is the same topic as being discussed in patch #3. Could you please add clear definitions to eliminate the need for guessing?

Quoted 12 lines · Reinette Chatre

Quoted 11 lines · Babu Moger

Quoted 7 lines · Reinette Chatre

Quoted 1 line · Babu Moger

Kernel mode work can use different allocation and monitoring associations

Here I think "different allocation and monitoring associations" means different CLOSID/RMID, while other places "different allocation and monitoring associations" means different MSR_IA32_PQR_PLZA_ASSOC.closid_en and MSR_IA32_PQR_PLZA_ASSOC.rmid_en?

That is correct. Hope text here is fine.

Quoted 4 lines

I had to guess. This is the same topic as being discussed in patch #3. Could you please add clear definitions to eliminate the need for guessing?

Ack, rephrased the changelog to use CLOSID and RMID terminology to avoid the ambiguity around "allocation and monitoring associations". If you agree, we can use CLOSID and RMID consistently throughout the series.

"fs/resctrl: Introduce kernel mode enum

Kernel mode operation enables a CPU to use a different CLOSID and/or RMID when executing in kernel mode than those used in user mode.

Architectures need a common set of kernel mode definitions so resctrl

can report supported modes and indicate which mode is active.

Introduce enum resctrl_kernel_mode:

  - RESCTRL_INHERIT_USER: Kernel mode work inherits the CLOSID and/or RMID of the user task.

  - RESCTRL_GLOBAL_ENABLE_PER_CPU: Kernel mode work uses a single global CLOSID and/or RMID association shared by all userspace tasks. The mode can be enabled or disabled on a per-CPU basis.

Hi Babu,

Quoted 17 lines · Babu Moger

Quoted 16 lines · Reinette Chatre

Quoted 11 lines · Babu Moger

Quoted 7 lines · Reinette Chatre

Quoted 1 line · Babu Moger

Kernel mode work can use different allocation and monitoring associations

Here I think "different allocation and monitoring associations" means different CLOSID/RMID, while other places "different allocation and monitoring associations" means different MSR_IA32_PQR_PLZA_ASSOC.closid_en and MSR_IA32_PQR_PLZA_ASSOC.rmid_en?

That is correct. Hope text here is fine.

I had to guess. This is the same topic as being discussed in patch #3. Could you please add clear definitions to eliminate the need for guessing?

Quoted 4 lines

Ack, rephrased the changelog to use CLOSID and RMID terminology to avoid the ambiguity around "allocation and monitoring associations". If you agree, we can use CLOSID and RMID consistently throughout the series.

I agree, with emphasis on consistently.

Quoted 6 lines

"fs/resctrl: Introduce kernel mode enum

Kernel mode operation enables a CPU to use a different CLOSID and/or RMID when executing in kernel mode than those used in user mode.

Please take care to distinguish between the "kernel mode" that defines the policy here and "kernel work" that is what the policy governs. At this time this series even conflates the two in some places ("kernel mode work").

Quoted 2 lines

Architectures need a common set of kernel mode definitions so resctrl

It is resctrl filesystem that needs the common vocabulary.

Quoted 7 lines

can report supported modes and indicate which mode is active.

Introduce enum resctrl_kernel_mode:

  - RESCTRL_INHERIT_USER: Kernel mode work inherits the CLOSID and/or RMID of the user task.

This does not look right, it inherits CLOSID *and* RMID, unconditionally, no?

Quoted 5 lines

  - RESCTRL_GLOBAL_ENABLE_PER_CPU: Kernel mode work uses a single global CLOSID and/or RMID association shared by all userspace tasks. The mode can be enabled or disabled on a per-CPU basis.

"shared by all user space tasks" is not right? The CLOSID/RMID come from the backing CTRL_MON or MON group, so they are shared only with the tasks assigned to that group (possibly even none).

Below is a suggestion that aims to address the comments. Please do not just copy&paste without verifying it for accuracy and that it matches the terms you decided to stick with.

Kernel work need not use the CLOSID and RMID of the user task on whose behalf it runs. On x86, Privilege Level Zero Association (PLZA) enables a CPU to use a different CLOSID and/or RMID while executing at privilege level zero.

resctrl needs common definitions of how kernel work is managed so that architectures can advertise the kernel modes they support and user space can query them and select the one to use.

Introduce enum resctrl_kernel_mode:

- RESCTRL_INHERIT_USER: Kernel work uses the CLOSID and RMID of the user task.

- RESCTRL_GLOBAL_ENABLE_PER_CPU: All kernel work uses the CLOSID and/or RMID of a single resource group, which may also have user tasks assigned to it. The mode can be enabled or disabled per CPU; on a CPU where it is disabled, kernel work inherits the user task's CLOSID and RMID.

RESCTRL_GLOBAL_ENABLE_PER_CPU mode enables kernel work to use different allocation and monitoring associations than the originating user task.

The mode is enabled independently on each CPU, so the filesystem must program hardware when the mode is activated, when CPUs are added or removed, and when it is deactivated. Allocation and monitoring associations can be done independently, so the architecture callback takes separate control and monitoring options.

Add resctrl_arch_configure_kmode_global() with an x86 PLZA implementation and an empty MPAM stub for ARM systems that do not support this mode.

Signed-off-by Babu Moger <[email protected]>

---

---
 arch/x86/kernel/cpu/resctrl/ctrlmondata.c | 38 +++++++++++++++++++++++
 drivers/resctrl/mpam_resctrl.c            |  7 +++++
 include/linux/resctrl.h                   | 37 ++++++++++++++++++++++
 3 files changed, 82 insertions(+)

arch/x86/kernel/cpu/resctrl/ctrlmondata.c +5 -0

index 62044489b052..a343dc0c71c5 100644
@@ -137,3 +137,41 @@ int resctrl_arch_io_alloc_enable(struct rdt_resource *r, bool enable)
 
 	return 0;
 }
+
+static void resctrl_kmode_set_one_amd(void *arg)
+{
+	union msr_pqr_plza_assoc *plza = arg;
+

arch/x86/kernel/cpu/resctrl/ctrlmondata.c +23 -0

+ * @rmid:	RMID programmed into MSR_IA32_PQR_PLZA_ASSOC. Always
+ *		written; required even when @assign_mon is false.
+ * @assign_mon:	Whether PLZA should program the kernel-mode RMID.
+ * @enable:	Whether PLZA should program the kernel-mode association.
+ *
+ * Program PLZA on @cpu_mask for %RESCTRL_GLOBAL_ENABLE_PER_CPU. See
+ * union msr_pqr_plza_assoc for the hardware field meanings.
+ */
+void resctrl_arch_configure_kmode_global(const struct cpumask *cpu_mask,
+					 u32 closid, bool assign_ctrl,
+					 u32 rmid, bool assign_mon,
+					 bool enable)
+{
+	union msr_pqr_plza_assoc plza = { 0 };
+
+	plza.split.rmid = rmid;
+	plza.split.rmid_en = assign_mon;
+	plza.split.closid = closid;
+	plza.split.closid_en = assign_ctrl;
+	plza.split.plza_en = enable;
+
+	on_each_cpu_mask(cpu_mask, resctrl_kmode_set_one_amd, &plza, 1);
+}

drivers/resctrl/mpam_resctrl.c +7 -0

index f2c651e9f6ea..0c8594e98526 100644
@@ -139,6 +139,13 @@ bool resctrl_arch_get_io_alloc_enabled(struct rdt_resource *r)
 	return false;
 }
 
+void resctrl_arch_configure_kmode_global(const struct cpumask *cpu_mask,
+					 u32 closid, bool assign_ctrl,
+					 u32 rmid, bool assign_mon,
+					 bool enable)
+{
+}
+
 void resctrl_arch_pre_mount(void)
 {
 }

include/linux/resctrl.h +0 -0

index f6adff7f4e1f..dcb0beb847b6 100644

include/linux/resctrl.h +4 -0

+ *
+ * Contract between fs and arch: resctrl fs always provides the same
+ * closid, assign_ctrl, rmid, and assign_mon when activating
+ * %RESCTRL_GLOBAL_ENABLE_PER_CPU, during CPU add/remove while the mode is

Hi Babu,

There is no reference in upstream resctrl but during discussions around the "multiple controls" support we settled on a new subject prefix of "arm,x86,fs/resctrl" for patches that touch all these parts.

Quoted 8 lines

RESCTRL_GLOBAL_ENABLE_PER_CPU mode enables kernel work to use different allocation and monitoring associations than the originating user task.

The mode is enabled independently on each CPU, so the filesystem must program hardware when the mode is activated, when CPUs are added or removed, and when it is deactivated. Allocation and monitoring associations can be done independently, so the architecture callback takes separate control and monitoring options.

Going back and forth through this series a couple of times all these usages of "allocation and monitoring associations" in different contexts become a blur and at this point I do not know what it means at all.

Quoted 7 lines

Add resctrl_arch_configure_kmode_global() with an x86 PLZA implementation and an empty MPAM stub for ARM systems that do not support this mode.

Signed-off-by Babu Moger <[email protected]>

---

...

Quoted 25 lines

---
  arch/x86/kernel/cpu/resctrl/ctrlmondata.c | 38 +++++++++++++++++++++++
  drivers/resctrl/mpam_resctrl.c            |  7 +++++
  include/linux/resctrl.h                   | 37 ++++++++++++++++++++++
  3 files changed, 82 insertions(+)

arch/x86/kernel/cpu/resctrl/ctrlmondata.c +11 -0

index 62044489b052..a343dc0c71c5 100644
@@ -137,3 +137,41 @@ int resctrl_arch_io_alloc_enable(struct rdt_resource *r, bool enable)
 
  	return 0;
  }
+
+static void resctrl_kmode_set_one_amd(void *arg)
+{
+	union msr_pqr_plza_assoc *plza = arg;
+
+	wrmsrq(MSR_IA32_PQR_PLZA_ASSOC, plza->full);
+}
+
+/*
+ * resctrl_arch_configure_kmode_global - Program Privilege Level Zero
+ *					 Association (PLZA).

arch/x86/kernel/cpu/resctrl/ctrlmondata.c int resctrl_arch_io_alloc_enable(struct rdt_resource *r, bool enable) · line 154

Quoted 4 lines

arch/x86/kernel/cpu/resctrl/ctrlmondata.c +4 -0

+ * @cpu_mask:	CPUs whose PLZA MSR should be updated.
+ * @closid:	CLOSID programmed into MSR_IA32_PQR_PLZA_ASSOC. Always
+ *		written; required even when @assign_ctrl is false.
+ * @assign_ctrl: Whether PLZA should program the kernel-mode CLOSID.

At this point the series seems to switch terms from the "kernel mode" in previous patches to "kernel-mode" here. Could this all be consistently "kernel mode"?

On a similar note, the series switches between "global kernel mode" (like in this subject) and "global per-CPU kernel mode" (like in patch #14 subject).

Quoted 50 lines

arch/x86/kernel/cpu/resctrl/ctrlmondata.c +23 -0

+ * @rmid:	RMID programmed into MSR_IA32_PQR_PLZA_ASSOC. Always
+ *		written; required even when @assign_mon is false.
+ * @assign_mon:	Whether PLZA should program the kernel-mode RMID.
+ * @enable:	Whether PLZA should program the kernel-mode association.
+ *
+ * Program PLZA on @cpu_mask for %RESCTRL_GLOBAL_ENABLE_PER_CPU. See
+ * union msr_pqr_plza_assoc for the hardware field meanings.
+ */
+void resctrl_arch_configure_kmode_global(const struct cpumask *cpu_mask,
+					 u32 closid, bool assign_ctrl,
+					 u32 rmid, bool assign_mon,
+					 bool enable)
+{
+	union msr_pqr_plza_assoc plza = { 0 };
+
+	plza.split.rmid = rmid;
+	plza.split.rmid_en = assign_mon;
+	plza.split.closid = closid;
+	plza.split.closid_en = assign_ctrl;
+	plza.split.plza_en = enable;
+
+	on_each_cpu_mask(cpu_mask, resctrl_kmode_set_one_amd, &plza, 1);
+}

drivers/resctrl/mpam_resctrl.c +7 -0

index f2c651e9f6ea..0c8594e98526 100644
@@ -139,6 +139,13 @@ bool resctrl_arch_get_io_alloc_enabled(struct rdt_resource *r)
  	return false;
  }
 
+void resctrl_arch_configure_kmode_global(const struct cpumask *cpu_mask,
+					 u32 closid, bool assign_ctrl,
+					 u32 rmid, bool assign_mon,
+					 bool enable)
+{
+}
+
  void resctrl_arch_pre_mount(void)
  {
  }

include/linux/resctrl.h +0 -0

index f6adff7f4e1f..dcb0beb847b6 100644
@@ -752,6 +752,43 @@ enum resctrl_kernel_mode {
 
  #define RESCTRL_NUM_KERNEL_MODES (RESCTRL_KMODE_LAST + 1)
 

include/linux/resctrl.h enum resctrl_kernel_mode { · line 758

Quoted 4 lines

include/linux/resctrl.h +4 -0

+/**
+ * resctrl_arch_configure_kmode_global() - Program RESCTRL_GLOBAL_ENABLE_PER_CPU
+ *
+ * @cpu_mask:	CPUs whose kernel-mode association should be updated.

This could be made more specific about being online CPUs. For example,
@cpu_mask: Online CPUs on which to update the association.

include/linux/resctrl.h enum resctrl_kernel_mode { · line 762

Quoted 5 lines

include/linux/resctrl.h +4 -0

+ * @closid:	CLOSID that matches the RMID to program kernel mode. Depending
+ *		on the architecture, the counter may match traffic of both
+ *		@closid and @rmid, or @rmid only. Always provided; a valid
+ *		CLOSID is required even when @assign_ctrl is false.

Above description seems to be a concatenation of different texts. The "counter" text seems to be copied from resctrl_arch_rmid_read() and does not seem relevant here. The "CLOSID is required" is copied from the x86 arch helper where it is indeed required - but this does not apply to resctrl fs?

How about something like:
@closid: CLOSID of the resource group backing %RESCTRL_GLOBAL_ENABLE_PER_CPU mode.
Always provided, even when @assign_ctrl is false.

include/linux/resctrl.h enum resctrl_kernel_mode { · line 768

Quoted 7 lines

include/linux/resctrl.h +6 -0

+ * @assign_ctrl: true to assign @closid for kernel mode; false to inherit
+ *		association from the user-space task.
+ * @rmid:	RMID to program the kernel mode. Some architectures may use
+ *		CLOSID/RMID separately, others will consider them together.
+ *		Always provided; a valid RMID is required even when
+ *		@assign_mon is false.

(similar comment as for @closid)

include/linux/resctrl.h enum resctrl_kernel_mode { · line 772

Quoted 5 lines

include/linux/resctrl.h +4 -0

+ * @assign_mon:	true to assign @rmid for kernel mode; false to inherit
+ *		monitoring association from the user-space task.
+ * @enable:	true to enable kernel mode association on CPUs in @cpu_mask;
+ *		false to disable kernel mode.

"false to disable kernel mode" does not seem to capture all that is required from architecture. When considering resctrl_kmode_switch(), @enable being false is only thing resctrl fs does to put CPUs into RESCTRL_INHERIT_USER mode? So when @enable is false the architecture should disable %RESCTRL_GLOBAL_ENABLE_PER_CPU *and* enable %RESCTRL_INHERIT_USER?

include/linux/resctrl.h enum resctrl_kernel_mode { · line 780

Quoted 9 lines

include/linux/resctrl.h +8 -0

+ *
+ * This callback programs %RESCTRL_GLOBAL_ENABLE_PER_CPU. It is called in
+ * the following scenarios:
+ * - If the mode is active when user space changes the configuration, it is
+ *   first called to deactivate the mode on all CPUs where it is active.
+ * - When user space activates the mode, it is called with cpu_online_mask.
+ * - When user space adds a CPU to the active mode.
+ * - When user space removes a CPU from the active mode.

This attempt at listing all callsites is proving problematic since it is already out of date by the end of the series. I would thus suggest to drop listing the individual call sites and instead focus on writing a clear contract. Consider, for example, something like (but please verify, especially for accuracy and correct terms, do not just copy&paste):

An association starts when %RESCTRL_GLOBAL_ENABLE_PER_CPU is activated and ends when the mode is deactivated or user space changes the backing resource group, @assign_ctrl or @assign_mon. Every call for one association provides the same @closid, @assign_ctrl, @rmid and @assign_mon; only @cpu_mask and @enable differ between calls. Before an association ends, it is disabled on every CPU where it is enabled.

The filesystem tracks supported kernel modes, the active mode, and the control and monitoring association for the assigned group.

Introduce struct resctrl_kmode_cfg and initialize the default configuration from the platform's capabilities.

Signed-off-by Babu Moger <[email protected]>

---

---
 fs/resctrl/internal.h | 29 +++++++++++++++++++++++++++++
 fs/resctrl/rdtgroup.c | 25 +++++++++++++++++++++++++
 2 files changed, 54 insertions(+)

fs/resctrl/internal.h +0 -0

index e62a277dee85..98f04a2a762b 100644

fs/resctrl/internal.h +11 -0

+ * @rdtgrp:	Resource group backing %RESCTRL_GLOBAL_ENABLE_PER_CPU.
+ *		%NULL while that mode is not selected.
+ */
+struct resctrl_kmode_cfg {
+	DECLARE_BITMAP(kmode_sup, RESCTRL_NUM_KERNEL_MODES);
+	enum resctrl_kernel_mode	kmode_cur;
+	enum kmode_state		ctrl;
+	enum kmode_state		mon;
+	struct rdtgroup			*rdtgrp;
+};
+
 extern struct mutex rdtgroup_mutex;
 
 static inline const char *rdt_kn_name(const struct kernfs_node *kn)

fs/resctrl/rdtgroup.c +3 -0

index 68be9b903ac6..7c39792be1fa 100644
@@ -76,6 +76,9 @@ static void rdtgroup_destroy_root(void);
 
 struct dentry *debugfs_resctrl;
 
+/* Kernel mode for this system. */
+static struct resctrl_kmode_cfg resctrl_kmode;
+
 /*
  * Memory bandwidth monitoring event to use for the default CTRL_MON group
  * and each new CTRL_MON group created by the user.  Only relevant when

fs/resctrl/rdtgroup.c +11 -0

+	resctrl_kmode.kmode_cur = RESCTRL_INHERIT_USER;
+	__set_bit(RESCTRL_INHERIT_USER, resctrl_kmode.kmode_sup);
+
+	/*
+	 * Defaults used when RESCTRL_GLOBAL_ENABLE_PER_CPU mode is
+	 * activated without new values.
+	 */
+	resctrl_kmode.ctrl = KMODE_ASSIGN;
+	resctrl_kmode.mon = KMODE_ASSIGN;
+}
+
 void resctrl_file_fflags_init(const char *config, unsigned long fflags)
 {
 	struct rftype *rft;

Hi Babu,

nit: "for resctrl" in subject is redundant.

Quoted 2 lines

The filesystem tracks supported kernel modes, the active mode, and the control and monitoring association for the assigned group.

Please let the context describe the context at the time of the patch, not how things will work after entire series is applied.

Please use consistent terminology, this series switches between "assigned group" and "backing group" to refer to the same group.

Please use the changelog to guide the reader to understand the architecture and how things fit together. Many changelogs are very cryptic without explanation of what the patch does and why.

Quoted 3 lines

Introduce struct resctrl_kmode_cfg and initialize the default configuration from the platform's capabilities.

Please check all changelogs to ensure they all are able to provide an answer to "why". More specifically, please consider:

	https://lore.kernel.org/lkml/20250911165433.GBaML-yTUZHkywuJIe@fat_crate.local/

Quoted 3 lines

Signed-off-by Babu Moger <[email protected]>

---

...

Quoted 10 lines

---
  fs/resctrl/internal.h | 29 +++++++++++++++++++++++++++++
  fs/resctrl/rdtgroup.c | 25 +++++++++++++++++++++++++
  2 files changed, 54 insertions(+)

fs/resctrl/internal.h +0 -0

index e62a277dee85..98f04a2a762b 100644

fs/resctrl/internal.h struct mbm_state { · line 318

Quoted 5 lines

fs/resctrl/internal.h +2 -0

@@ -314,6 +314,35 @@ struct mbm_state {
  	u32	prev_bw;
  };
 
+/**
+ * enum kmode_state - Control or monitoring association

What does "control or monitoring association" mean? This is becoming an overloaded term used to mean different things and becoming confusing. For comparison, earlier patch introduced *another*
"control and monitoring association" enum:

enum resctrl_kernel_mode - Kernel mode control and monitoring association

fs/resctrl/internal.h struct mbm_state { · line 330

Quoted 13 lines

fs/resctrl/internal.h +12 -0

+ * @KMODE_INHERIT: Inherit from the user-space task.
+ * @KMODE_ASSIGN: Use the backing group's CLOSID or RMID.
+ */
+enum kmode_state {
+	KMODE_INHERIT,
+	KMODE_ASSIGN
+};
+
+/**
+ * struct resctrl_kmode_cfg - Kernel mode configuration
+ * @kmode_sup:	Bitmap of supported kernel modes.
+ * @kmode_cur:	Currently selected kernel mode.

Adding this comment during a second pass ... please consider later motivations about naming of these fields. This field is introduced as "kmode_cur" but when considering the language used in rest of series "active" seems more appropriate.

@kmode_sup could also just be @supported (the "kmode" is redundant).

fs/resctrl/internal.h struct mbm_state { · line 332

Quoted 3 lines

fs/resctrl/internal.h +2 -0

+ * @ctrl:	Control association for %RESCTRL_GLOBAL_ENABLE_PER_CPU.
+ *		Kept while %RESCTRL_INHERIT_USER is selected.

The "Kept while ..." does not seem to match implementation. Looks like resctrl_kmode_switch() always resets it to KMODE_ASSIGN?

At this point the reader is left to decipher the architecture via cryptic changelogs and inaccurate comments. It is very difficult to parse.

fs/resctrl/internal.h struct mbm_state { · line 334

Quoted 2 lines

fs/resctrl/internal.h +2 -0

+ * @mon:	Monitoring association for %RESCTRL_GLOBAL_ENABLE_PER_CPU.
+ *		Kept while %RESCTRL_INHERIT_USER is selected.

Similar comment about "Kept ..."

Quoted 33 lines

fs/resctrl/internal.h +11 -0

+ * @rdtgrp:	Resource group backing %RESCTRL_GLOBAL_ENABLE_PER_CPU.
+ *		%NULL while that mode is not selected.
+ */
+struct resctrl_kmode_cfg {
+	DECLARE_BITMAP(kmode_sup, RESCTRL_NUM_KERNEL_MODES);
+	enum resctrl_kernel_mode	kmode_cur;
+	enum kmode_state		ctrl;
+	enum kmode_state		mon;
+	struct rdtgroup			*rdtgrp;
+};
+
  extern struct mutex rdtgroup_mutex;
 
  static inline const char *rdt_kn_name(const struct kernfs_node *kn)

fs/resctrl/rdtgroup.c +3 -0

index 68be9b903ac6..7c39792be1fa 100644
@@ -76,6 +76,9 @@ static void rdtgroup_destroy_root(void);
 
  struct dentry *debugfs_resctrl;
 
+/* Kernel mode for this system. */
+static struct resctrl_kmode_cfg resctrl_kmode;
+
  /*
   * Memory bandwidth monitoring event to use for the default CTRL_MON group
   * and each new CTRL_MON group created by the user.  Only relevant when

@@ -2297,6 +2300,26 @@ static void io_alloc_init(void)
  	}
  }
 

fs/resctrl/rdtgroup.c static void io_alloc_init(void) · line 2305

Quoted 3 lines

fs/resctrl/rdtgroup.c +3 -0

+/*
+ * Set RESCTRL_INHERIT_USER as the default when the platform supports
+ * allocation or monitoring. Otherwise leave kernel mode unsupported.

This (and why) would be useful to describe in changelog.

fs/resctrl/rdtgroup.c static void io_alloc_init(void) · line 2311

Quoted 7 lines

fs/resctrl/rdtgroup.c +6 -0

+ */
+static void resctrl_kmode_init(void)
+{
+	if (!resctrl_arch_alloc_capable() && !resctrl_arch_mon_capable())
+		return;
+

This check seems unnecessary? resctrl_kmode_init() will only be called on a system that is capable of allocation or monitoring.
The planned AET changes will change this behavior see
https://lore.kernel.org/lkml/[email protected]/
but in that case it still seems as though initializing the kernel mode as below always needs to be done.

That begs the question why is resctrl_kmode_init() needed at all? Why not use a static initializer?

I find resctrl_kmode_init() to be potentially confusing since it does *not* actually initialize resctrl_kmode. This is only possible to see two patches later where get_rdt_resources() is changed so that x86 sets its supported modes and this runs *before* resctrl_kmode_init().

If anyone decides one day "this initialization code is missing a bitmap_zero()" then it will break PLZA.

Quoted 7 lines · Reinette Chatre

Quoted 4 lines · Babu Moger

+/**
+ * struct resctrl_kmode_cfg - Kernel mode configuration
+ * @kmode_sup:    Bitmap of supported kernel modes.
+ * @kmode_cur:    Currently selected kernel mode.

Quoted 5 lines

Adding this comment during a second pass ... please consider later motivations about naming of these fields. This field is introduced as "kmode_cur" but when considering the language used in rest of series "active" seems more appropriate.

@kmode_sup could also just be @supported (the "kmode" is redundant).

Sure. Will change kmode_cur to "current".

Hi Babu,

Quoted 8 lines · Babu

Quoted 7 lines · Reinette Chatre

Quoted 4 lines · Babu Moger

+/**
+ * struct resctrl_kmode_cfg - Kernel mode configuration
+ * @kmode_sup:    Bitmap of supported kernel modes.
+ * @kmode_cur:    Currently selected kernel mode.

Quoted 7 lines

Quoted 5 lines

Adding this comment during a second pass ... please consider later motivations about naming of these fields. This field is introduced as "kmode_cur" but when considering the language used in rest of series "active" seems more appropriate.

@kmode_sup could also just be @supported (the "kmode" is redundant).

Sure. Will change kmode_cur to "current".

Please do not use "current" as it is generally used to represent the current running task.

Why do you find "active" not appropriate? The text usually refers to it as the "active". Consider the resctrl.rst text that reads "When the mode is not active ...", changelogs refer to it as the "active" mode, user messages refer to it as the "active" mode, the functions are named rdtgroup_kmode_global_activate()/rdtgroup_kmode_global_deactivate().

The resctrl subsystem tracks kernel mode configuration. Architectures need

to register the supported kernel modes.

Add resctrl_set_kmode_support() so an architecture can register supported kernel modes.

Signed-off-by Babu Moger <[email protected]>

---

---
 fs/resctrl/rdtgroup.c   |  6 ++++++
 include/linux/resctrl.h | 15 +++++++++++++++
 2 files changed, 21 insertions(+)

fs/resctrl/rdtgroup.c +6 -0

index 7c39792be1fa..75b81d07a247 100644
@@ -2320,6 +2320,12 @@ static void resctrl_kmode_init(void)
 	resctrl_kmode.mon = KMODE_ASSIGN;
 }
 
+void resctrl_set_kmode_support(enum resctrl_kernel_mode kmode)
+{
+	if (kmode < RESCTRL_NUM_KERNEL_MODES)
+		__set_bit(kmode, resctrl_kmode.kmode_sup);
+}
+
 void resctrl_file_fflags_init(const char *config, unsigned long fflags)
 {
 	struct rftype *rft;

include/linux/resctrl.h +2 -0

index dcb0beb847b6..5553f6301bd4 100644
@@ -752,6 +752,21 @@ enum resctrl_kernel_mode {
 
 #define RESCTRL_NUM_KERNEL_MODES (RESCTRL_KMODE_LAST + 1)
 
+/**
+ * resctrl_set_kmode_support() - Register a supported kernel mode

Hi Babu,

Quoted 1 line · Babu Moger

The resctrl subsystem tracks kernel mode configuration. Architectures need

resctrl tracks nothing at this point. This is preparatory work to enable resctrl to do so.

Quoted 8 lines

to register the supported kernel modes.

Add resctrl_set_kmode_support() so an architecture can register supported kernel modes.

Signed-off-by Babu Moger <[email protected]>

---

...

Quoted 39 lines

---
  fs/resctrl/rdtgroup.c   |  6 ++++++
  include/linux/resctrl.h | 15 +++++++++++++++
  2 files changed, 21 insertions(+)

fs/resctrl/rdtgroup.c +6 -0

index 7c39792be1fa..75b81d07a247 100644
@@ -2320,6 +2320,12 @@ static void resctrl_kmode_init(void)
  	resctrl_kmode.mon = KMODE_ASSIGN;
  }
 
+void resctrl_set_kmode_support(enum resctrl_kernel_mode kmode)
+{
+	if (kmode < RESCTRL_NUM_KERNEL_MODES)
+		__set_bit(kmode, resctrl_kmode.kmode_sup);
+}
+
  void resctrl_file_fflags_init(const char *config, unsigned long fflags)
  {
  	struct rftype *rft;

include/linux/resctrl.h +9 -0

index dcb0beb847b6..5553f6301bd4 100644
@@ -752,6 +752,21 @@ enum resctrl_kernel_mode {
 
  #define RESCTRL_NUM_KERNEL_MODES (RESCTRL_KMODE_LAST + 1)
 
+/**
+ * resctrl_set_kmode_support() - Register a supported kernel mode
+ * @kmode:	Kernel mode supported by the architecture.
+ *
+ * RESCTRL_INHERIT_USER is registered by resctrl_kmode_init() and does
+ * not need to be registered again.
+ *
+ * Architectures use this interface during resctrl initialization to
+ * advertise supported kernel modes. Registration of a mode indicates

include/linux/resctrl.h enum resctrl_kernel_mode { · line 766

Quoted 3 lines

include/linux/resctrl.h +3 -0

+ * architectural support only. The final decision to expose and enable
+ * a mode is made by resctrl based on system capabilities and
+ * configuration.

The last sentence is not clear to me. There is no filtering of any modes before it is exposed to user space. It is the architecture that sets the modes based on the "system capabilities", this is not something the filesystem decides?

Hi Reinette,

Quoted 5 lines · Reinette Chatre

Hi Babu,

Quoted 1 line · Babu Moger

The resctrl subsystem tracks kernel mode configuration. Architectures need

Quoted 2 lines

resctrl tracks nothing at this point. This is preparatory work to enable resctrl to do so.

Yes. That is correct. Hope this looks ok.

Architectures may support different kernel modes that need to be exposed to resctrl.

Add resctrl_set_kmode_support() to allow an architecture to register its supported kernel modes. This is preparatory infrastructure for future kernel mode support in resctrl.

Hi Babu,

Quoted 11 lines · Babu

Hi Reinette,

Quoted 7 lines · Reinette Chatre

Hi Babu,

Quoted 1 line · Babu Moger

The resctrl subsystem tracks kernel mode configuration. Architectures need

resctrl tracks nothing at this point. This is preparatory work to enable resctrl to do so.

Quoted 4 lines

Yes. That is correct. Hope this looks ok.

Architectures may support different kernel modes that need to be exposed to resctrl.

nit: "need to be exposed to user space via resctrl filesystem."?

Quoted 5 lines

Add resctrl_set_kmode_support() to allow an architecture to register its supported kernel modes. This is preparatory infrastructure for future kernel mode support in resctrl.

Looks good, thank you.

PLZA (Privilege Level Zero Association) provides the global per-CPU kernel mode, RESCTRL_GLOBAL_ENABLE_PER_CPU, allowing kernel work to use allocation and monitoring associations that differ from user tasks.

Add x86 support for this kernel mode and make it available through the

resctrl kernel-mode infrastructure.

Signed-off-by Babu Moger <[email protected]>

---

Hi Babu,

Quoted 4 lines · Babu Moger

PLZA (Privilege Level Zero Association) provides the global per-CPU kernel mode, RESCTRL_GLOBAL_ENABLE_PER_CPU, allowing kernel work to use allocation and monitoring associations that differ from user tasks.

Quoted 1 line

Add x86 support for this kernel mode and make it available through the

nit: This patch does not add any support, it just registers the mode with resctrl fs (as the subject already indicates). Didn't patch #5 already claim to add support?

Resctrl kernel modes control how allocation and monitoring associations

are applied in kernel mode relative to user mode.

Add info/kernel_mode to expose the supported and active kernel modes via the resctrl filesystem. Supported modes are listed one per line, with the

active mode enclosed in square brackets.

Document the interface in Documentation/filesystems/resctrl.rst.

Signed-off-by Babu Moger <[email protected]>

---

---
 Documentation/filesystems/resctrl.rst |  48 +++++++++++++
 fs/resctrl/rdtgroup.c                 | 100 ++++++++++++++++++++++++++
 2 files changed, 148 insertions(+)

Documentation/filesystems/resctrl.rst +0 -0

index 73bf89d0ac83..73ccbb9594f5 100644

Documentation/filesystems/resctrl.rst +8 -0

+
+	Example of a system that supports allocation and monitoring, with
+	default mode inherit_user::
+
+	  # cat info/kernel_mode
+	  [inherit_user]
+	  global_enable_per_cpu:ctrl=assign;mon=assign;group=//
+
 Resource alloc and monitor groups
 =================================
 

fs/resctrl/rdtgroup.c +73 -0

index 75b81d07a247..e82709d64a00 100644
@@ -993,6 +993,99 @@ static int rdt_last_cmd_status_show(struct kernfs_open_file *of,
 	return 0;
 }
 
+/* Names used in info/kernel_mode; indexed by enum resctrl_kernel_mode */
+static const char * const resctrl_kmode_str[] = {
+	[RESCTRL_INHERIT_USER]		= "inherit_user",
+	[RESCTRL_GLOBAL_ENABLE_PER_CPU]	= "global_enable_per_cpu"
+};
+
+static_assert(ARRAY_SIZE(resctrl_kmode_str) == RESCTRL_NUM_KERNEL_MODES);
+
+static const char *resctrl_kmode_state_str(enum kmode_state state)
+{
+	return state == KMODE_ASSIGN ? "assign" : "inherit";
+}
+
+static void resctrl_kmode_group_path(struct rdtgroup *rdtgrp,
+				     const char **ctrl, const char **mon)
+{
+	*ctrl = "";
+	*mon = "";
+
+	if (rdtgrp->type == RDTMON_GROUP) {
+		*ctrl = rdt_kn_name(rdtgrp->mon.parent->kn);
+		*mon = rdt_kn_name(rdtgrp->kn);
+	} else {
+		*ctrl = rdt_kn_name(rdtgrp->kn);
+	}
+}
+
+/**
+ * resctrl_kernel_mode_show() - Display supported and active kernel modes
+ * @of:		kernfs open file
+ * @seq:	output seq_file
+ * @v:		unused
+ *
+ * Lists one line per mode and brackets the active mode. inherit_user
+ * has no parameters. global_enable_per_cpu prints ctrl= only when
+ * allocation is supported and mon= only when monitoring is supported,
+ * then group=.
+ *
+ * Return: 0 on success, or -ENOENT on error.
+ */
+static int resctrl_kernel_mode_show(struct kernfs_open_file *of,
+				    struct seq_file *seq, void *v)
+{
+	const char *ctrl_grp, *mon_grp;
+	enum resctrl_kernel_mode mode;
+	bool active;
+	char sep;
+
+	if (!info_kn_lock(of->kn))
+		return -ENOENT;
+
+	for (mode = 0; mode < RESCTRL_NUM_KERNEL_MODES; mode++) {
+		if (!test_bit(mode, resctrl_kmode.kmode_sup))
+			continue;
+
+		active = resctrl_kmode.kmode_cur == mode;
+
+		seq_printf(seq, active ? "[%s" : "%s", resctrl_kmode_str[mode]);
+
+		if (mode == RESCTRL_GLOBAL_ENABLE_PER_CPU) {
+			ctrl_grp = "";
+			mon_grp = "";
+			if (active) {
+				if (resctrl_kmode.rdtgrp)
+					resctrl_kmode_group_path(resctrl_kmode.rdtgrp,
+								 &ctrl_grp, &mon_grp);
+				else
+					pr_warn("%s active without a resource group\n",
+						resctrl_kmode_str[mode]);
+			}
+
+			sep = ':';
+			if (resctrl_arch_alloc_capable()) {

fs/resctrl/rdtgroup.c +10 -0

+		}
+
+		seq_puts(seq, active ? "]\n" : "\n");
+	}
+
+	info_kn_unlock(of->kn);
+
+	return 0;
+}
+
 void *rdt_kn_parent_priv(struct kernfs_node *kn)
 {
 	/*

@@ -1978,6 +2071,13 @@ static struct rftype res_common_files[] = {
 		.seq_show	= rdt_last_cmd_status_show,
 		.fflags		= RFTYPE_TOP_INFO,
 	},

Hi Babu,

Quoted 1 line · Babu Moger

Resctrl kernel modes control how allocation and monitoring associations

"allocation and monitoring associations" - unclear what this refers to at this point.

Quoted 4 lines

are applied in kernel mode relative to user mode.

Add info/kernel_mode to expose the supported and active kernel modes via the resctrl filesystem. Supported modes are listed one per line, with the

(needs imperative)

Quoted 3 lines

active mode enclosed in square brackets.

Document the interface in Documentation/filesystems/resctrl.rst.

Above sentence is not necessary, it is obvious from the patch.

Quoted 4 lines

Signed-off-by Babu Moger <[email protected]>

---

...

Quoted 13 lines

---
  Documentation/filesystems/resctrl.rst |  48 +++++++++++++
  fs/resctrl/rdtgroup.c                 | 100 ++++++++++++++++++++++++++
  2 files changed, 148 insertions(+)

Documentation/filesystems/resctrl.rst +0 -0

index 73bf89d0ac83..73ccbb9594f5 100644
@@ -527,6 +527,54 @@ conveyed in the error returns from file operations. E.g.
  	# cat info/last_cmd_status
  	mask f7 has non-consecutive 1-bits
 

Documentation/filesystems/resctrl.rst conveyed in the error returns from file operations. E.g. · line 531

Quoted 2 lines

Documentation/filesystems/resctrl.rst +2 -0

+"kernel_mode":
+	In the top level of the "info" directory, "kernel_mode" lists the

Note that just above this addition the documentation reads: "Finally, in the top level of the "info" directory there is a file named "last_cmd_status"." This addition thus creates a contradiction by adding another file after the "Finally". Could you please rework that portion of the documentation to have an introduction for the files in the top level info directory and make it clear how there are two files in this directory, similar to how the other directories are documented?

Documentation/filesystems/resctrl.rst conveyed in the error returns from file operations. E.g. · line 534

Quoted 4 lines

Documentation/filesystems/resctrl.rst +3 -0

+	kernel modes supported by the system, one per line, with the active
+	mode enclosed in square brackets. A kernel mode determines the
+	allocation and monitoring associations when a task switches from

Would user space know what "allocation and monitoring associations" means?

Quoted 11 lines

Documentation/filesystems/resctrl.rst +11 -0

+	user mode to kernel mode.
+
+	Possible kernel modes are:
+
+	"inherit_user":
+		Kernel mode work uses the allocation and monitoring associations
+		of the user-space task on whose behalf it runs.
+
+	"global_enable_per_cpu":
+		Kernel mode work uses the allocation and/or monitoring
+		association of a backing CTRL_MON or MON group. The mode can

Documentation/filesystems/resctrl.rst conveyed in the error returns from file operations. E.g. · line 546

Quoted 1 line

Documentation/filesystems/resctrl.rst +1 -0

+		be activated or deactivated per CPU. The line contains group=

Having a global "activate"/"deactivate" as well as a per CPU "activate"/"deactivate" is confusing. The series switches between language here but I find other places that refer to the global "activate"/"deactivate" and then per CPU "enable"/"disable" to be easier to follow. Doing so will match the text of the "kmode_cpus" file that reads "The association is enabled on CPUs added to the mask and disabled on CPUs removed".

Documentation/filesystems/resctrl.rst conveyed in the error returns from file operations. E.g. · line 556

Quoted 11 lines

Documentation/filesystems/resctrl.rst +10 -0

+		and whichever of ctrl= and mon= the platform supports::
+
+		  global_enable_per_cpu:ctrl=<assign|inherit>;mon=<assign|inherit>;group=<CTRL_MON>/<MON>/
+
+		"ctrl":
+			"assign" if kernel mode work uses the allocation
+			association of the resource group named by "group",
+			"inherit" if it uses the allocation association of
+			the user-space task. Present only on systems that
+			support allocation.

ok - this is already late in the series but this seems to make it clear that "allocation association" means the portions of resources allocated to this group. Thus the earlier " enum kmode_state - Control or monitoring association" is wrong?

Documentation/filesystems/resctrl.rst conveyed in the error returns from file operations. E.g. · line 565

Quoted 10 lines

Documentation/filesystems/resctrl.rst +9 -0

+		"mon":
+			"assign" if kernel mode work uses the monitoring
+			association of the resource group named by "group",
+			"inherit" if it uses the monitoring association of
+			the user-space task. Present only on systems that
+			support monitoring.
+		"group":
+			Path of the resource group, relative to the resctrl
+			mount point. A CTRL_MON group is shown as <CTRL_MON>//,

Shouldn't it just be <CTRL_MON>/ ?

Documentation/filesystems/resctrl.rst conveyed in the error returns from file operations. E.g. · line 569

Quoted 5 lines

Documentation/filesystems/resctrl.rst +4 -0

+			a MON group as <CTRL_MON>/<MON>/.
+
+		When the mode is not active, the parameters are the defaults
+		used if it is activated without new values.

"parameters are the defaults" seems to conflate two things: (a) the supported parameters, and (2) the default values of those supported parameters.

Quoted 99 lines

Documentation/filesystems/resctrl.rst +8 -0

+
+	Example of a system that supports allocation and monitoring, with
+	default mode inherit_user::
+
+	  # cat info/kernel_mode
+	  [inherit_user]
+	  global_enable_per_cpu:ctrl=assign;mon=assign;group=//
+
  Resource alloc and monitor groups
  =================================
 

fs/resctrl/rdtgroup.c +80 -0

index 75b81d07a247..e82709d64a00 100644
@@ -993,6 +993,99 @@ static int rdt_last_cmd_status_show(struct kernfs_open_file *of,
  	return 0;
  }
 
+/* Names used in info/kernel_mode; indexed by enum resctrl_kernel_mode */
+static const char * const resctrl_kmode_str[] = {
+	[RESCTRL_INHERIT_USER]		= "inherit_user",
+	[RESCTRL_GLOBAL_ENABLE_PER_CPU]	= "global_enable_per_cpu"
+};
+
+static_assert(ARRAY_SIZE(resctrl_kmode_str) == RESCTRL_NUM_KERNEL_MODES);
+
+static const char *resctrl_kmode_state_str(enum kmode_state state)
+{
+	return state == KMODE_ASSIGN ? "assign" : "inherit";
+}
+
+static void resctrl_kmode_group_path(struct rdtgroup *rdtgrp,
+				     const char **ctrl, const char **mon)
+{
+	*ctrl = "";
+	*mon = "";
+
+	if (rdtgrp->type == RDTMON_GROUP) {
+		*ctrl = rdt_kn_name(rdtgrp->mon.parent->kn);
+		*mon = rdt_kn_name(rdtgrp->kn);
+	} else {
+		*ctrl = rdt_kn_name(rdtgrp->kn);
+	}
+}
+
+/**
+ * resctrl_kernel_mode_show() - Display supported and active kernel modes
+ * @of:		kernfs open file
+ * @seq:	output seq_file
+ * @v:		unused
+ *
+ * Lists one line per mode and brackets the active mode. inherit_user
+ * has no parameters. global_enable_per_cpu prints ctrl= only when
+ * allocation is supported and mon= only when monitoring is supported,
+ * then group=.
+ *
+ * Return: 0 on success, or -ENOENT on error.
+ */
+static int resctrl_kernel_mode_show(struct kernfs_open_file *of,
+				    struct seq_file *seq, void *v)
+{
+	const char *ctrl_grp, *mon_grp;
+	enum resctrl_kernel_mode mode;
+	bool active;
+	char sep;
+
+	if (!info_kn_lock(of->kn))
+		return -ENOENT;
+
+	for (mode = 0; mode < RESCTRL_NUM_KERNEL_MODES; mode++) {
+		if (!test_bit(mode, resctrl_kmode.kmode_sup))
+			continue;
+
+		active = resctrl_kmode.kmode_cur == mode;
+
+		seq_printf(seq, active ? "[%s" : "%s", resctrl_kmode_str[mode]);
+
+		if (mode == RESCTRL_GLOBAL_ENABLE_PER_CPU) {
+			ctrl_grp = "";
+			mon_grp = "";
+			if (active) {
+				if (resctrl_kmode.rdtgrp)
+					resctrl_kmode_group_path(resctrl_kmode.rdtgrp,
+								 &ctrl_grp, &mon_grp);
+				else
+					pr_warn("%s active without a resource group\n",
+						resctrl_kmode_str[mode]);
+			}
+
+			sep = ':';
+			if (resctrl_arch_alloc_capable()) {
+				seq_printf(seq, "%cctrl=%s", sep,
+					   resctrl_kmode_state_str(resctrl_kmode.ctrl));
+				sep = ';';
+			}
+			if (resctrl_arch_mon_capable()) {
+				seq_printf(seq, "%cmon=%s", sep,
+					   resctrl_kmode_state_str(resctrl_kmode.mon));

fs/resctrl/rdtgroup.c static int rdt_last_cmd_status_show(struct kernfs_open_file *of, · line 1078

Quoted 3 lines

fs/resctrl/rdtgroup.c +3 -0

+				sep = ';';
+			}
+			seq_printf(seq, "%cgroup=%s/%s/", sep, ctrl_grp, mon_grp);

Looks like this will print a monitoring group on a system that does not support monitoring? I understand that the monitoring group will be the empty string in this scenario but in resctrl that has meaning of "default monitoring group" so printing this when monitoring is not supported is unexpected.

I also do not expect the monitoring group to be displayed if "mon=inherit" since that presents a contradiction with stating that "monitoring group is inherited" while printing a monitoring group.

Quoted 17 lines · Reinette Chatre

Documentation/filesystems/resctrl.rst conveyed in the error returns from file operations. E.g. · line 556

Quoted 10 lines · Babu Moger

+        and whichever of ctrl= and mon= the platform supports::

Documentation/filesystems/resctrl.rst +9 -0

+
+          global_enable_per_cpu:ctrl=<assign|inherit>;mon=<assign|inherit>;group=<CTRL_MON>/<MON>/
+
+        "ctrl":
+            "assign" if kernel mode work uses the allocation
+            association of the resource group named by "group",
+            "inherit" if it uses the allocation association of
+            the user-space task. Present only on systems that
+            support allocation.

ok - this is already late in the series but this seems to make it clear that "allocation association" means the portions of resources allocated to this group. Thus the earlier " enum kmode_state - Control or monitoring association" is wrong?

I think, it should be " enum kmode_state - Allocation or monitoring association"

Quoted 10 lines

Quoted 9 lines

+        "mon":
+            "assign" if kernel mode work uses the monitoring +            association of the resource group named by "group", +            "inherit" if it uses the monitoring association of +            the user-space task. Present only on systems that +            support monitoring.
+        "group":
+            Path of the resource group, relative to the resctrl +            mount point. A CTRL_MON group is shown as <CTRL_MON>//,

Quoted 1 line

Shouldn't it just be <CTRL_MON>/ ?

Or We can say "<CTRL_MON>/<MON>/" to be generic for both cases.

"The group is shown as  "<CTRL_MON>/<MON>/".

Quoted 14 lines

Quoted 5 lines

+
+            sep = ':';
+            if (resctrl_arch_alloc_capable()) {
+                seq_printf(seq, "%cctrl=%s", sep,
+                       resctrl_kmode_state_str(resctrl_kmode.ctrl));

Quoted 8 lines

+                sep = ';';
+            }
+            if (resctrl_arch_mon_capable()) {
+                seq_printf(seq, "%cmon=%s", sep,
+                       resctrl_kmode_state_str(resctrl_kmode.mon)); +                sep = ';';
+            }
+            seq_printf(seq, "%cgroup=%s/%s/", sep, ctrl_grp, mon_grp);

Quoted 5 lines

Looks like this will print a monitoring group on a system that does not support monitoring? I understand that the monitoring group will be the empty string in this scenario but in resctrl that has meaning of "default monitoring group" so printing this when monitoring is not supported is unexpected.

Ok. If monitoring not supported it will be "CTRL_MON/" (Example group=ctrl1/") and then default group is "/".

Is that what you are suggesting?

Quoted 5 lines

I also do not expect the monitoring group to be displayed if "mon=inherit" since that presents a contradiction with stating that "monitoring group is inherited" while printing a monitoring group.

Yes. We should not display "mon=inherit" when monitoring in not supported. Will check on this.

Hi Babu,

Quoted 19 lines · Babu

Quoted 17 lines · Reinette Chatre

Documentation/filesystems/resctrl.rst conveyed in the error returns from file operations. E.g. · line 556

Quoted 10 lines · Babu Moger

+        and whichever of ctrl= and mon= the platform supports::

Documentation/filesystems/resctrl.rst +9 -0

+
+          global_enable_per_cpu:ctrl=<assign|inherit>;mon=<assign|inherit>;group=<CTRL_MON>/<MON>/
+
+        "ctrl":
+            "assign" if kernel mode work uses the allocation
+            association of the resource group named by "group",
+            "inherit" if it uses the allocation association of
+            the user-space task. Present only on systems that
+            support allocation.

ok - this is already late in the series but this seems to make it clear that "allocation association" means the portions of resources allocated to this group. Thus the earlier " enum kmode_state - Control or monitoring association" is wrong?

Quoted 1 line

I think, it should be " enum kmode_state - Allocation or monitoring association"

Please don't. This is exactly the problem that I keep coming back to. If you change the name as you suggest then you solidify that you use the same term to mean two different things. Specifically, what above change implies is that the reader is left to decipher when:

"allocation and monitoring association" means
MSR_IA32_PQR_PLZA_ASSOC.closid and MSR_IA32_PQR_PLZA_ASSOC.rmid

vs when

"allocation and monitoring association" means
MSR_IA32_PQR_PLZA_ASSOC.closid_en and MSR_IA32_PQR_PLZA_ASSOC.rmid_en

Quoted 10 lines

Quoted 10 lines

Quoted 9 lines

+        "mon":
+            "assign" if kernel mode work uses the monitoring +            association of the resource group named by "group", +            "inherit" if it uses the monitoring association of +            the user-space task. Present only on systems that +            support monitoring.
+        "group":
+            Path of the resource group, relative to the resctrl +            mount point. A CTRL_MON group is shown as <CTRL_MON>//,

Quoted 5 lines

Quoted 1 line

Shouldn't it just be <CTRL_MON>/ ?

Or We can say "<CTRL_MON>/<MON>/" to be generic for both cases.

"The group is shown as  "<CTRL_MON>/<MON>/".

No, this depends on whether monitoring is enabled or not, or (if monitoring is supported) whether mon=inherit.

...

Quoted 20 lines

Quoted 14 lines

Quoted 5 lines

+
+            sep = ':';
+            if (resctrl_arch_alloc_capable()) {
+                seq_printf(seq, "%cctrl=%s", sep,
+                       resctrl_kmode_state_str(resctrl_kmode.ctrl));

Quoted 8 lines

+                sep = ';';
+            }
+            if (resctrl_arch_mon_capable()) {
+                seq_printf(seq, "%cmon=%s", sep,
+                       resctrl_kmode_state_str(resctrl_kmode.mon)); +                sep = ';';
+            }
+            seq_printf(seq, "%cgroup=%s/%s/", sep, ctrl_grp, mon_grp);

Quoted 5 lines

Looks like this will print a monitoring group on a system that does not support monitoring? I understand that the monitoring group will be the empty string in this scenario but in resctrl that has meaning of "default monitoring group" so printing this when monitoring is not supported is unexpected.

Quoted 3 lines

Ok. If monitoring not supported it will be "CTRL_MON/" (Example group=ctrl1/") and then default group is "/".

Is that what you are suggesting?

Yes. This matches the proc output.

Quoted 6 lines

Quoted 5 lines

I also do not expect the monitoring group to be displayed if "mon=inherit" since that presents a contradiction with stating that "monitoring group is inherited" while printing a monitoring group.

Quoted 2 lines

Yes. We should not display "mon=inherit" when monitoring in not supported. Will check on this.

This is needed yes, but what I was actually referring to was that if "mon=inherit" then the group parameter should not include a monitoring group. For example, "group=ctrl/". Including the name of a monitoring group when "mon=inherit" is confusing.

Some per-group resctrl files are only meaningful once a resource group has an active kernel mode association.

Creating them only at activation would complicate group setup and

teardown, while exposing them on every group would show empty files

with no clear purpose.

Add a hidden property to struct rftype to hide files at creation.

Suggested-by Reinette Chatre <[email protected]>

Signed-off-by Babu Moger <[email protected]>

---

Hi Babu,

Quoted 2 lines · Babu Moger

Some per-group resctrl files are only meaningful once a resource group has an active kernel mode association.

Could this elaborate more about how the files depend on the kernel mode and why they may not always be meaningful?

Quoted 2 lines

Creating them only at activation would complicate group setup and

Why would it complicate group setup and teardown? I think it would rather complicate mode activation and de-activation but since there is no detail about how this will complicate group setup it is difficult to compare. It is difficult to reason about a patch when the motivation is vague.

Quoted 1 line

teardown, while exposing them on every group would show empty files

"show empty files" - technically these files, as they are implemented, will not be empty but return errors

Quoted 3 lines

with no clear purpose.

Add a hidden property to struct rftype to hide files at creation.

Missing a "why".
This can append to describe how this adds the foundational support in preparation for features to restore visibility when the feature's files need to be exposed to user space.

Quoted 21 lines · Reinette Chatre

Quoted 2 lines · Babu Moger

Some per-group resctrl files are only meaningful once a resource group has an active kernel mode association.

Could this elaborate more about how the files depend on the kernel mode and why they may not always be meaningful?

Quoted 2 lines

Creating them only at activation would complicate group setup and

Why would it complicate group setup and teardown? I think it would rather complicate mode activation and de-activation but since there is no detail about how this will complicate group setup it is difficult to compare. It is difficult to reason about a patch when the motivation is vague.

Quoted 1 line

teardown, while exposing them on every group would show empty files

"show empty files" - technically these files, as they are implemented, will not be empty but return errors

Quoted 9 lines

Quoted 3 lines

with no clear purpose.

Add a hidden property to struct rftype to hide files at creation.

Missing a "why".
This can append to describe how this adds the foundational support in preparation for features to restore visibility when the feature's files need to be exposed to user space.

Will rephrase the whole changelog:

"Some per-group resctrl files are only applicable to resource groups with an active kernel mode association. Accessing these files before a kernel mode association is established would result in errors and does not provide useful information.

Creating and removing these files when a kernel mode is activated or deactivated would add complexity to the mode management path. Add a hidden property to struct rftype so files can be created with the resource group and exposed only when they become applicable.

This provides the infrastructure needed to control file visibility for future kernel-mode-specific resctrl interfaces."

Hi Babu,

Quoted 34 lines · Babu

Quoted 21 lines · Reinette Chatre

Quoted 2 lines · Babu Moger

Some per-group resctrl files are only meaningful once a resource group has an active kernel mode association.

Could this elaborate more about how the files depend on the kernel mode and why they may not always be meaningful?

Quoted 2 lines

Creating them only at activation would complicate group setup and

Why would it complicate group setup and teardown? I think it would rather complicate mode activation and de-activation but since there is no detail about how this will complicate group setup it is difficult to compare. It is difficult to reason about a patch when the motivation is vague.

Quoted 1 line

teardown, while exposing them on every group would show empty files

"show empty files" - technically these files, as they are implemented, will not be empty but return errors

Quoted 9 lines

Quoted 3 lines

with no clear purpose.

Add a hidden property to struct rftype to hide files at creation.

Missing a "why".
This can append to describe how this adds the foundational support in preparation for features to restore visibility when the feature's files need to be exposed to user space.

Will rephrase the whole changelog:

Quoted 4 lines

"Some per-group resctrl files are only applicable to resource groups with an active kernel mode association. Accessing these files before a kernel mode association is established would result in errors and does not provide useful information.

What does "and does not provide useful information" refer to? Does this mean that the errors are not useful?

Quoted 6 lines

Creating and removing these files when a kernel mode is activated or deactivated would add complexity to the mode management path. Add a hidden property to struct rftype so files can be created with the resource group and exposed only when they become applicable.

Please separate the problem and solution descriptions.

Quoted 35 lines · Reinette Chatre

Quoted 34 lines · Babu

Quoted 21 lines · Reinette Chatre

Quoted 2 lines · Babu Moger

Some per-group resctrl files are only meaningful once a resource group has an active kernel mode association.

Could this elaborate more about how the files depend on the kernel mode and why they may not always be meaningful?

Quoted 2 lines

Creating them only at activation would complicate group setup and

Why would it complicate group setup and teardown? I think it would rather complicate mode activation and de-activation but since there is no detail about how this will complicate group setup it is difficult to compare. It is difficult to reason about a patch when the motivation is vague.

Quoted 1 line

teardown, while exposing them on every group would show empty files

"show empty files" - technically these files, as they are implemented, will not be empty but return errors

Quoted 9 lines

Quoted 3 lines

with no clear purpose.

Add a hidden property to struct rftype to hide files at creation.

Missing a "why".
This can append to describe how this adds the foundational support in preparation for features to restore visibility when the feature's files need to be exposed to user space.

Will rephrase the whole changelog:

Quoted 7 lines

Quoted 4 lines

"Some per-group resctrl files are only applicable to resource groups with an active kernel mode association. Accessing these files before a kernel mode association is established would result in errors and does not provide useful information.

What does "and does not provide useful information" refer to? Does this mean that the errors are not useful?

Hmm..  One more attempt:

"Some per-group resctrl files are only applicable when a resource group has an active kernel mode association. Exposing these files before the mode is activated is not useful because they have no meaningful information to provide, while reporting errors on access offers little value. Dynamically creating and removing the files as associations are activated and deactivated would add complexity to the mode management path.

Add a hidden property to struct rftype so the files can be created with the resource group and exposed only when applicable.

This provides the infrastructure needed to control file visibility for future kernel-mode-specific resctrl interfaces."

Hi Babu,

Quoted 44 lines · Babu Moger

Quoted 35 lines · Reinette Chatre

Quoted 34 lines · Babu

Quoted 21 lines · Reinette Chatre

Quoted 2 lines · Babu Moger

Some per-group resctrl files are only meaningful once a resource group has an active kernel mode association.

Could this elaborate more about how the files depend on the kernel mode and why they may not always be meaningful?

Quoted 2 lines

Creating them only at activation would complicate group setup and

Why would it complicate group setup and teardown? I think it would rather complicate mode activation and de-activation but since there is no detail about how this will complicate group setup it is difficult to compare. It is difficult to reason about a patch when the motivation is vague.

Quoted 1 line

teardown, while exposing them on every group would show empty files

"show empty files" - technically these files, as they are implemented, will not be empty but return errors

Quoted 9 lines

Quoted 3 lines

with no clear purpose.

Add a hidden property to struct rftype to hide files at creation.

Missing a "why".
This can append to describe how this adds the foundational support in preparation for features to restore visibility when the feature's files need to be exposed to user space.

Will rephrase the whole changelog:

Quoted 7 lines

Quoted 4 lines

"Some per-group resctrl files are only applicable to resource groups with an active kernel mode association. Accessing these files before a kernel mode association is established would result in errors and does not provide useful information.

What does "and does not provide useful information" refer to? Does this mean that the errors are not useful?

Quoted 8 lines

Hmm..  One more attempt:

"Some per-group resctrl files are only applicable when a resource group has an active kernel mode association. Exposing these files before the mode is activated is not useful because they have no meaningful information to provide, while reporting errors on access offers little value. Dynamically creating and removing the files as associations are activated and deactivated would add complexity to the mode management path.

Add a hidden property to struct rftype so the files can be created with the resource group and exposed only when applicable.

This provides the infrastructure needed to control file visibility for future kernel-mode-specific resctrl interfaces."

I still find this motivation vague as it seems to aim to be generic when it requires to be specific since it is needed by the global per-CPU kernel mode support. Consider, for example, in comparison a changelog below that is specific to the problem that needs to be solved and why. Again this is an example that does not take into account the consistent terms that you aim to use in next version:

fs/resctrl: Add support for hidden resource group files

A resctrl file is visible to user space as soon as its resource group directory is activated. A file can only be hidden after it has been created, as is done for the info directory's BMEC event configuration files.

In the global_enable_per_cpu kernel mode, kernel work running on CPUs where the mode is active is allocated resources and/or monitored as part of a single backing resource group. The files needed to display and modify the CPUs on which this kernel mode is active are only meaningful in that group. Visible in every resource group, the files would return errors when accessed in any group other than the one backing the active kernel mode. Creating the files when the kernel mode is activated, and removing them when it is deactivated, would add failure paths to kernel mode activation that showing and hiding existing files does not have.

Allow a resource group file to be marked as hidden so that it is hidden when created, before its directory is activated, and is never visible to user space until resctrl restores its visibility. This provides the foundation in preparation for kernel mode support that shows the kmode_cpus and kmode_cpus_list files only in the resource group backing the active kernel mode.

The RESCTRL_GLOBAL_ENABLE_PER_CPU kernel mode allows tasks to use different allocation and monitoring associations when transitioning from user mode to

kernel mode. The mode is enabled/disabled on a per-CPU basis.

Add kmode_cpus and kmode_cpus_list to display the CPUs associated with a kernel mode. These files are exposed only when the global_enable_per_cpu

mode is active.

Signed-off-by Babu Moger <[email protected]>

---

fs/resctrl/internal.h +1 -0

  * @plr:			pseudo-locked region
  */
 struct rdtgroup {

@@ -229,6 +230,7 @@ struct rdtgroup {
 	struct mongroup			mon;
 	enum rdtgrp_mode		mode;
 	enum resctrl_event_id		mba_mbps_event;
+	struct cpumask			kmode_cpu_mask;
 	struct pseudo_lock_region	*plr;
 };
 

fs/resctrl/rdtgroup.c +0 -0

index 46b2cafd9411..0e638dc654c1 100644

fs/resctrl/rdtgroup.c +4 -0

+ * Supports both "kmode_cpus" (bitmap format) and "kmode_cpus_list"
+ * (range list format); the output format is selected accordingly.
+ *
+ * Returns -ENOENT on error.

fs/resctrl/rdtgroup.c +19 -0

+		ret = -ENOENT;
+	else
+		seq_printf(s, is_cpu_list(of) ? "%*pbl\n" : "%*pb\n",
+			   cpumask_pr_args(&rdtgrp->kmode_cpu_mask));
+	rdtgroup_kn_unlock(of->kn);
+
+	return ret;
+}
+
 /*
  * Update the PGR_ASSOC MSR on all cpus in @cpu_mask,
  *

@@ -2256,6 +2281,23 @@ static struct rftype res_common_files[] = {
 		.flags		= RFTYPE_FLAGS_CPUS_LIST,
 		.fflags		= RFTYPE_BASE,
 	},
+	{
+		.name		= "kmode_cpus",
+		.mode		= 0444,
+		.hidden		= true,
+		.kf_ops		= &rdtgroup_kf_single_ops,
+		.seq_show	= rdtgroup_kmode_cpus_show,
+		.fflags		= RFTYPE_BASE,
+	},
+	{
+		.name		= "kmode_cpus_list",

Hi Babu,

Quoted 2 lines · Babu Moger

The RESCTRL_GLOBAL_ENABLE_PER_CPU kernel mode allows tasks to use different allocation and monitoring associations when transitioning from user mode to

Could "different allocation and monitoring associations" just be "different CLOSID and RMID" (in places where this is what it refers to)? Being specific and consistent with terms would make this so much easier to understand.

Quoted 5 lines

kernel mode. The mode is enabled/disabled on a per-CPU basis.

Add kmode_cpus and kmode_cpus_list to display the CPUs associated with a kernel mode. These files are exposed only when the global_enable_per_cpu

"a kernel mode" -> this is specific to RESCTRL_GLOBAL_ENABLE_PER_CPU kernel mode node?

This switches from "RESCTRL_GLOBAL_ENABLE_PER_CPU kernel mode" to "global_enable_per_cpu mode" mid changelog.

Quoted 5 lines

mode is active.

Signed-off-by Babu Moger <[email protected]>

---

...> ---

Quoted 4 lines

  fs/resctrl/internal.h |  2 ++
  fs/resctrl/rdtgroup.c | 42 ++++++++++++++++++++++++++++++++++++++++++
  2 files changed, 44 insertions(+)

fs/resctrl/internal.h struct mongroup { · line 219

Quoted 9 lines

fs/resctrl/internal.h +1 -0

index 986b9c942fb2..6b8d14c4bd83 100644
@@ -216,6 +216,7 @@ struct mongroup {
   * @mon:			mongroup related data
   * @mode:			mode of resource group
   * @mba_mbps_event:		input monitoring event id when mba_sc is enabled
+ * @kmode_cpu_mask:		CPUs in this group's kernel-mode association

Could there be a caveat that the value is only valid when the resource group is backing the RESCTRL_GLOBAL_ENABLE_PER_CPU kernel mode?

Quoted 20 lines

fs/resctrl/internal.h +1 -0

   * @plr:			pseudo-locked region
   */
  struct rdtgroup {

@@ -229,6 +230,7 @@ struct rdtgroup {
  	struct mongroup			mon;
  	enum rdtgrp_mode		mode;
  	enum resctrl_event_id		mba_mbps_event;
+	struct cpumask			kmode_cpu_mask;
  	struct pseudo_lock_region	*plr;
  };
 

fs/resctrl/rdtgroup.c +0 -0

index 46b2cafd9411..0e638dc654c1 100644
@@ -385,6 +385,31 @@ static int rdtgroup_cpus_show(struct kernfs_open_file *of,
  	return ret;
  }
 

fs/resctrl/rdtgroup.c static int rdtgroup_cpus_show(struct kernfs_open_file *of, · line 389

Quoted 2 lines

fs/resctrl/rdtgroup.c +2 -0

+/*
+ * Display CPU masks for the kernel-mode associated resource group.

nit: "masks for" -> "mask of"?

Quoted 12 lines

fs/resctrl/rdtgroup.c +11 -0

+ * Supports both "kmode_cpus" (bitmap format) and "kmode_cpus_list"
+ * (range list format); the output format is selected accordingly.
+ *
+ * Returns -ENOENT on error.
+ */
+static int rdtgroup_kmode_cpus_show(struct kernfs_open_file *of,
+				    struct seq_file *s, void *v)
+{
+	struct rdtgroup *rdtgrp;
+	int ret = 0;
+

fs/resctrl/rdtgroup.c static int rdtgroup_cpus_show(struct kernfs_open_file *of, · line 403

Quoted 3 lines

fs/resctrl/rdtgroup.c +3 -0

+	rdtgrp = rdtgroup_kn_lock_live(of->kn);
+
+	if (!rdtgrp || rdtgrp != resctrl_kmode.rdtgrp)

Could the changelog please help reader understand the patches better? For example, the additional "rdtgrp != resctrl_kmode.rdtgrp" check is subtle but very important to maintain synchronization, but it is never mentioned why this is used.