fs,x86/resctrl: Add kernel-mode (e.g., PLZA) support to the resctrl subsystem

Add resctrl_kmode, resctrl_kmode_cfg, kernel mode bit defines, and resctrl_arch_get_kmode_cfg() for resctrl kernel mode (e.g. PLZA) support.

---
 include/linux/resctrl.h       | 10 ++++++++++
 include/linux/resctrl_types.h | 30 ++++++++++++++++++++++++++++++
 2 files changed, 40 insertions(+)

include/linux/resctrl.h +0 -0

index 006e57fd7ca5..2c36d1ac392f 100644
@@ -699,6 +699,16 @@ int resctrl_arch_io_alloc_enable(struct rdt_resource *r, bool enable);
  */
 bool resctrl_arch_get_io_alloc_enabled(struct rdt_resource *r);

include/linux/resctrl.h +1 -0

+
 extern unsigned int resctrl_rmid_realloc_threshold;
 extern unsigned int resctrl_rmid_realloc_limit;
 

include/linux/resctrl_types.h +0 -0

index a5f56faa18d2..6b78b08eab29 100644
@@ -65,7 +65,37 @@ enum resctrl_event_id {
 	QOS_NUM_EVENTS,
 };

include/linux/resctrl_types.h +13 -0

+
+/**
+ * struct resctrl_kmode_cfg - Resctrl kernel mode configuration
+ * @kmode:	Requested kernel mode.
+ * @kmode_cur:	Currently active kernel mode.
+ * @k_rdtgrp:	Resource control structure in use, or NULL otherwise.
+ */
+struct resctrl_kmode_cfg {
+	u32 kmode;
+	u32 kmode_cur;
+	struct rdtgroup *k_rdtgrp;
+};
+

Hi Babu,

Quoted 2 lines · Babu Moger

Add resctrl_kmode, resctrl_kmode_cfg, kernel mode bit defines, and resctrl_arch_get_kmode_cfg() for resctrl kernel mode (e.g. PLZA) support.

We should not have to start every series from scratch. Documentation/process/maintainer-tip.rst. Always.

Quoted 14 lines

---
  include/linux/resctrl.h       | 10 ++++++++++
  include/linux/resctrl_types.h | 30 ++++++++++++++++++++++++++++++
  2 files changed, 40 insertions(+)

include/linux/resctrl.h +0 -0

index 006e57fd7ca5..2c36d1ac392f 100644
@@ -699,6 +699,16 @@ int resctrl_arch_io_alloc_enable(struct rdt_resource *r, bool enable);
   */
  bool resctrl_arch_get_io_alloc_enabled(struct rdt_resource *r);
 

include/linux/resctrl.h int resctrl_arch_io_alloc_enable(struct rdt_resource *r, bool enable); · line 710

Quoted 9 lines

include/linux/resctrl.h +9 -0

+/**
+ * resctrl_arch_get_kmode_cfg() - Get resctrl kernel mode configuration
+ * @kcfg:	Filled with current kernel mode config (kmode, kmode_cur, k_rdtgrp).
+ *
+ * Used by the arch (e.g. x86) to report which kernel mode is active and,
+ * when a global assign mode is in use, which rdtgroup is assigned to
+ * kernel work.
+ */
+void resctrl_arch_get_kmode_cfg(struct resctrl_kmode_cfg *kcfg);

This interface does not look right. Would it not be resctrl fs that determines which resource group is assigned? This cannot be set by arch. Why does arch decide which mode is active? Is this not also resctrl fs? Should arch not just tell resctrl fs what it supports?

Quoted 13 lines

include/linux/resctrl.h +1 -0

+
  extern unsigned int resctrl_rmid_realloc_threshold;
  extern unsigned int resctrl_rmid_realloc_limit;
 

include/linux/resctrl_types.h +0 -0

index a5f56faa18d2..6b78b08eab29 100644
@@ -65,7 +65,37 @@ enum resctrl_event_id {
  	QOS_NUM_EVENTS,
  };
 

include/linux/resctrl_types.h enum resctrl_event_id { · line 76

Quoted 9 lines

include/linux/resctrl_types.h +9 -0

+/**
+ * struct resctrl_kmode - Resctrl kernel mode descriptor
+ * @name:	Human-readable name of the kernel mode.
+ * @val:	Bitmask value for the kernel mode (e.g. INHERIT_CTRL_AND_MON).
+ */
+struct resctrl_kmode {
+	char    name[32];
+	u32     val;
+};

There is no reason why this needs to be in a central header exposed to archs. Could this not be a static within the only function that uses it? Something like rdt_mode_str[]?

Quoted 17 lines

include/linux/resctrl_types.h +13 -0

+
+/**
+ * struct resctrl_kmode_cfg - Resctrl kernel mode configuration
+ * @kmode:	Requested kernel mode.
+ * @kmode_cur:	Currently active kernel mode.
+ * @k_rdtgrp:	Resource control structure in use, or NULL otherwise.
+ */
+struct resctrl_kmode_cfg {
+	u32 kmode;
+	u32 kmode_cur;
+	struct rdtgroup *k_rdtgrp;
+};
+
  #define QOS_NUM_L3_MBM_EVENTS	(QOS_L3_MBM_LOCAL_EVENT_ID - QOS_L3_MBM_TOTAL_EVENT_ID + 1)
  #define MBM_STATE_IDX(evt)	((evt) - QOS_L3_MBM_TOTAL_EVENT_ID)
 

include/linux/resctrl_types.h enum resctrl_event_id { · line 99

Quoted 7 lines

include/linux/resctrl_types.h +7 -0

+/* Resctrl kernel mode bits (e.g. for PLZA). */
+#define INHERIT_CTRL_AND_MON		BIT(0)	/* Kernel uses same CLOSID/RMID as user. */
+/* One CLOSID for all kernel work; RMID inherited from user. */
+#define GLOBAL_ASSIGN_CTRL_INHERIT_MON	BIT(1)
+/* One resource group (CLOSID+RMID) for all kernel work. */
+#define GLOBAL_ASSIGN_CTRL_ASSIGN_MON	BIT(2)
+#define RESCTRL_KERNEL_MODES_NUM	3

I think it will make the code much easier to understand if the different modes are described by an enum. For example,

	enum resctrl_kernel_modes {
		INHERIT_CTRL_AND_MON,
		GLOBAL_ASSIGN_CTRL_INHERIT_MON,
		GLOBAL_ASSIGN_CTRL_ASSIGN_MON,
		RESCTRL_KMODE_LAST = GLOBAL_ASSIGN_CTRL_ASSIGN_MON
	};
	#define RESCTRL_NUM_KERNEL_MODES (RESCTRL_KMODE_LAST + 1)

The supported kernel modes can still be managed as a bitmap with intuitive API using the enum that will make the code easier to read. For example, __set_bit(INHERIT_CTRL_AND_MON, ...) or BIT(INHERIT_CTRL_AND_MON). The naming is awkward at the moment though, we should improve here.

Implement the resctrl kernel mode (kmode) arch initialization.

- Add resctrl_arch_get_kmode_cfg() to fill the default kernel mode (INHERIT_CTRL_AND_MON). This can be extended later (e.g. for PLZA) to set additional modes.

- Add global resctrl_kcfg and resctrl_kmode_init() to initialize default values.

Signed-off-by Babu Moger <[email protected]>

---
v2: New patch to handle PLZA interfaces with /sys/fs/resctrl/info/ directory.

    https://lore.kernel.org/lkml/[email protected]/
---
 arch/x86/kernel/cpu/resctrl/core.c |  7 +++++++
 fs/resctrl/rdtgroup.c              | 10 ++++++++++
 2 files changed, 17 insertions(+)

arch/x86/kernel/cpu/resctrl/core.c +0 -0

index 7667cf7c4e94..4c3ab2d93909 100644

Hi Babu,

Quoted 5 lines · Babu Moger

Implement the resctrl kernel mode (kmode) arch initialization.

- Add resctrl_arch_get_kmode_cfg() to fill the default kernel mode (INHERIT_CTRL_AND_MON). This can be extended later (e.g. for PLZA) to set additional modes.

I do not think this is something that the architecture should set, at least at this time. Every mode has different requirements and this just lets the arch set it without any support for what configurations it implies. For example, if arch sets a different default mode than INHERIT_CTRL_AND_MON then PQR_PLZA_ASSOC needs to be programmed as the CPUs come online and this does not seem to accommodate this. This implementation appears to have significant assumptions on what architecture will end up setting since it is only considering PLZA.

Quoted 18 lines

- Add global resctrl_kcfg and resctrl_kmode_init() to initialize default values.

Signed-off-by Babu Moger <[email protected]>

---
v2: New patch to handle PLZA interfaces with /sys/fs/resctrl/info/ directory.

     https://lore.kernel.org/lkml/[email protected]/
---
  arch/x86/kernel/cpu/resctrl/core.c |  7 +++++++
  fs/resctrl/rdtgroup.c              | 10 ++++++++++
  2 files changed, 17 insertions(+)

arch/x86/kernel/cpu/resctrl/core.c +0 -0

index 7667cf7c4e94..4c3ab2d93909 100644

arch/x86/kernel/cpu/resctrl/core.c bool resctrl_arch_is_evt_configurable(enum resctrl_event_id evt) · line 900

Quoted 9 lines

arch/x86/kernel/cpu/resctrl/core.c +6 -0

@@ -892,6 +892,13 @@ bool resctrl_arch_is_evt_configurable(enum resctrl_event_id evt)
  	}
  }
 
+void resctrl_arch_get_kmode_cfg(struct resctrl_kmode_cfg *kcfg)
+{
+	kcfg->kmode = INHERIT_CTRL_AND_MON;
+	kcfg->kmode_cur = INHERIT_CTRL_AND_MON;
+	kcfg->k_rdtgrp = NULL;
+}

I already commented on the arch vs filesystem settings.

When using an arch helper this forces all architectures to support this helper. Is a helper required? Is it perhaps possible for arch to set a property instead? For example, how enumeration is handled?
I think the assumption here is that INHERIT_CTRL_AND_MON is the default and expected to be supported by all architectures. I do not see why arch should set this as default but instead this should be from resctrl fs. At the same time it is expected that the architecture supports this mode so there needs to be a failure if an architecture does not support this mode?

I'm going to stop here. I think the comments so far may result in major changes already making further detailed review of patches unnecessary.

Reinette

This series adds support for Privilege-Level Zero Association (PLZA) to the resctrl subsystem. PLZA is an AMD feature that allows specifying a CLOSID and/or RMID for execution in kernel mode (privilege level zero), so that kernel work is not subject to the same resource constrains as the current user-space task. This avoids kernel operations being aggressively throttled when a task's memory bandwidth is heavily limited.

The feature documentation is not yet publicly available, but it is expected to be released in the next few weeks. In the meantime, a brief description of the features is provided below.

Privilege Level Zero Association (PLZA)

Privilege Level Zero Association (PLZA) allows the hardware to automatically associate execution in Privilege Level Zero (CPL=0) with a specific COS (Class of Service) and/or RMID (Resource Monitoring Identifier). The QoS feature set already has a mechanism to associate execution on each logical processor with an RMID or COS. PLZA allows the system to override this per-thread association for a thread that is executing with CPL=0.
------------------------------------------------------------------------

The series introduces the feature in a way that supports the interface in a generic manner to accomodate MPAM or other vendor specific implimentation.

Below is the detailed requirements provided by Reinette:
https://lore.kernel.org/lkml/[email protected]/

Summary:
1. Kernel-mode/PLZA controls and status should be exposed under the resctrl info directory:/sys/fs/resctrl/info/, not as a separate or arch-specific path.

2. Add two info files

a. kernel_mode
Purpose: Control how resource allocation and monitoring apply in kernel mode (e.g. inherit from task vs global assign).

Read: List supported modes and show current one (e.g. with [brackets]).
Write: Set current mode by name (e.g. inherit_ctrl_and_mon, global_assign_ctrl_assign_mon).

b. kernel_mode_assignment

Purpose: When a “global assign” kernel mode is active, specify which resctrl group (CLOSID/RMID) is used for kernel work.

Read: Show the assigned group in a path-like form (e.g. //, ctrl1//, ctrl1/mon1/).
Write: Assign or clear the group used for kernel mode (and optionally clear with an empty write).

The patches are based on top of commit (v7.0.0-rc3) 839e91ce3f41b (tip/master) Merge branch into tip/master: 'x86/tdx'
------------------------------------------------------------------------

Examples: kernel_mode and kernel_mode_assignment

All paths below are under /sys/fs/resctrl/ (e.g. info/kernel_mode means /sys/fs/resctrl/info/kernel_mode). Resctrl must be mounted and the platform must support the relevant modes (e.g. AMD with PLZA).

1) kernel_mode — show and set the current kernel mode

Read supported modes and which one is active (current in brackets):

     $ cat info/kernel_mode
     [inherit_ctrl_and_mon]
     global_assign_ctrl_inherit_mon
     global_assign_ctrl_assign_mon

Set the active mode (e.g. use one CLOSID+RMID for all kernel work):

     $ echo "global_assign_ctrl_assign_mon" > info/kernel_mode
     $ cat info/kernel_mode
     inherit_ctrl_and_mon
     global_assign_ctrl_inherit_mon
     [global_assign_ctrl_assign_mon]

Mode meanings:
- inherit_ctrl_and_mon: kernel uses same CLOSID/RMID as the current task (default).
- global_assign_ctrl_inherit_mon: one CLOSID for all kernel work; RMID inherited from user.
- global_assign_ctrl_assign_mon: one resource group (CLOSID+RMID) for all kernel work.

2) kernel_mode_assignment — show and set which group is used for kernel work

Only relevant when kernel_mode is not "inherit_ctrl_and_mon". Read the

currently assigned group (path format is "CTRL_MON/MON/"):

     $ cat info/kernel_mode_assignment
     //

"//" means the default CTRL_MON group is assigned. Assign a specific group instead (e.g. a CTRL_MON group "ctrl1", or a MON group "mon1" under it):

     $ echo "ctrl1//" > info/kernel_mode_assignment
     $ cat info/kernel_mode_assignment
     ctrl1//

     $ echo "ctrl1/mon1/" > info/kernel_mode_assignment
     $ cat info/kernel_mode_assignment
     ctrl1/mon1/

Clear the assignment (no dedicated group for kernel work):

     $ echo >> info/kernel_mode_assignment
     $ cat info/kernel_mode_assignment
     Kmode is not configured

Errors (e.g. invalid group name or unsupported mode) are reported in info/last_cmd_status.

---

v2:
This is similar to RFC with new proposal. Names of the some interfaces are not final. Lets fix that later as we move forward.

Separated the two features: Global Bandwidth Enforcement (GLBE) and Privilege Level Zero Association (PLZA).

This series only adds support for PLZA.

Used the name of the feature as kmode instead of PLZA. That can be changed as well.

Tony suggested using global variables to store the kernel mode CLOSID and RMID. However, the kernel mode CLOSID and RMID are coming from rdtgroup structure with the new interface. Accessing them requires holding the associated lock, which would make the context switch path unnecessarily expensive. So, dropped the idea.
https://lore.kernel.org/lkml/aXuxVSbk1GR2ttzF@agluck-desk3/
Let me know if there are other ways to optimize this.

Hi Babu,

Quoted 22 lines · Babu Moger

This series adds support for Privilege-Level Zero Association (PLZA) to the resctrl subsystem. PLZA is an AMD feature that allows specifying a CLOSID and/or RMID for execution in kernel mode (privilege level zero), so that kernel work is not subject to the same resource constrains as the current user-space task. This avoids kernel operations being aggressively throttled when a task's memory bandwidth is heavily limited.

The feature documentation is not yet publicly available, but it is expected to be released in the next few weeks. In the meantime, a brief description of the features is provided below.

Privilege Level Zero Association (PLZA)

Privilege Level Zero Association (PLZA) allows the hardware to automatically associate execution in Privilege Level Zero (CPL=0) with a specific COS (Class of Service) and/or RMID (Resource Monitoring Identifier). The QoS feature set already has a mechanism to associate execution on each logical processor with an RMID or COS. PLZA allows the system to override this per-thread association for a thread that is executing with CPL=0.
------------------------------------------------------------------------

Quoted 5 lines

The series introduces the feature in a way that supports the interface in a generic manner to accomodate MPAM or other vendor specific implimentation.

Below is the detailed requirements provided by Reinette:
https://lore.kernel.org/lkml/[email protected]/

Our discussion considered how resctrl could support PLZA in a generic way while also preparing to support MPAM's variants and how PLZA may evolve to have similar capabilities when considering the capabilities of its registers.

This does not mean that your work needs to implement everything that was discussed. Instead, this work is expected to just support what PLZA is capable of today but do so in a way that the future enhancements could be added to.

This series is quite difficult to follow since it appears to implement a full featured generic interface while PLZA cannot take advantage of it.

Could you please simplify this work to focus on just enabling PLZA and only add interfaces needed to do so?

Quoted 54 lines

Summary:
1. Kernel-mode/PLZA controls and status should be exposed under the resctrl info directory:/sys/fs/resctrl/info/, not as a separate or arch-specific path.

2. Add two info files

a. kernel_mode
Purpose: Control how resource allocation and monitoring apply in kernel mode (e.g. inherit from task vs global assign).

Read: List supported modes and show current one (e.g. with [brackets]).
Write: Set current mode by name (e.g. inherit_ctrl_and_mon, global_assign_ctrl_assign_mon).

b. kernel_mode_assignment

Purpose: When a “global assign” kernel mode is active, specify which resctrl group (CLOSID/RMID) is used for kernel work.

Read: Show the assigned group in a path-like form (e.g. //, ctrl1//, ctrl1/mon1/).
Write: Assign or clear the group used for kernel mode (and optionally clear with an empty write).

The patches are based on top of commit (v7.0.0-rc3) 839e91ce3f41b (tip/master) Merge branch into tip/master: 'x86/tdx'
------------------------------------------------------------------------

Examples: kernel_mode and kernel_mode_assignment

All paths below are under /sys/fs/resctrl/ (e.g. info/kernel_mode means /sys/fs/resctrl/info/kernel_mode). Resctrl must be mounted and the platform must support the relevant modes (e.g. AMD with PLZA).

1) kernel_mode — show and set the current kernel mode

Read supported modes and which one is active (current in brackets):

$ cat info/kernel_mode
[inherit_ctrl_and_mon]
global_assign_ctrl_inherit_mon
global_assign_ctrl_assign_mon

Set the active mode (e.g. use one CLOSID+RMID for all kernel work):

$ echo "global_assign_ctrl_assign_mon" > info/kernel_mode $ cat info/kernel_mode
inherit_ctrl_and_mon global_assign_ctrl_inherit_mon [global_assign_ctrl_assign_mon]

Mode meanings:
- inherit_ctrl_and_mon: kernel uses same CLOSID/RMID as the current task (default).
- global_assign_ctrl_inherit_mon: one CLOSID for all kernel work; RMID inherited from user.
- global_assign_ctrl_assign_mon: one resource group (CLOSID+RMID) for all kernel work.

Quoted 3 lines

2) kernel_mode_assignment — show and set which group is used for kernel work

Only relevant when kernel_mode is not "inherit_ctrl_and_mon". Read the

To help with future usages please connect visibility of this file with the mode in info/kernel_mode. This helps us to support future modes with other resctrl files, possible within each resource group.
Specifically, kernel_mode_assignment is not visible to user space if mode is "inherit_ctrl_and_mon", while it is visible when mode is global_assign_ctrl_inherit_mon or global_assign_ctrl_assign_mon.

Quoted 2 lines

currently assigned group (path format is "CTRL_MON/MON/"):

The format depends on the mode, right? If the mode is "global_assign_ctrl_inherit_mon" then it should only contain a control group, alternatively, if the mode is "global_assign_ctrl_assign_mon" then it contains control and mon group. This gives resctrl future flexibility to change format for future modes.

We should also consider the scenario when it is a "monitoring only" system, which can happen independent from what hardware actually supports, for example, if user boots with "rdt=!l3cat,!l2cat,!mba,!smba". In this case I assume CLOS should just always be zero and thus only "default control group" is accepted?

Quoted 15 lines

      $ cat info/kernel_mode_assignment
      //

"//" means the default CTRL_MON group is assigned. Assign a specific group instead (e.g. a CTRL_MON group "ctrl1", or a MON group "mon1" under it):

$ echo "ctrl1//" > info/kernel_mode_assignment $ cat info/kernel_mode_assignment ctrl1//

$ echo "ctrl1/mon1/" > info/kernel_mode_assignment $ cat info/kernel_mode_assignment ctrl1/mon1/

Quoted 5 lines

Clear the assignment (no dedicated group for kernel work):

$ echo >> info/kernel_mode_assignment
$ cat info/kernel_mode_assignment
Kmode is not configured

This does not look right. Would this not create a conflict between info/kernel_mode and info/kernel_mode_assignment about what the current mode is? The way I see it info/kernel_mode_assignment must always contain a valid group.

Quoted 17 lines

Errors (e.g. invalid group name or unsupported mode) are reported in info/last_cmd_status.

---

v2:
This is similar to RFC with new proposal. Names of the some interfaces are not final. Lets fix that later as we move forward.

Separated the two features: Global Bandwidth Enforcement (GLBE) and Privilege Level Zero Association (PLZA).

This series only adds support for PLZA.

Used the name of the feature as kmode instead of PLZA. That can be changed as well.

Quoted 7 lines

Tony suggested using global variables to store the kernel mode CLOSID and RMID. However, the kernel mode CLOSID and RMID are coming from rdtgroup structure with the new interface. Accessing them requires holding the associated lock, which would make the context switch path unnecessarily expensive. So, dropped the idea.
https://lore.kernel.org/lkml/aXuxVSbk1GR2ttzF@agluck-desk3/
Let me know if there are other ways to optimize this.

I do not see why the context switch path needs to be touched at all with this implementation. Since PLZA only supports global assignment does it not mean that resctrl only needs to update PQR_PLZA_ASSOC when user writes to info/kernel_mode and info/kernel_mode_assignment?

Consider some of the scenarios:

resctrl mount with default state:

	# cat info/kernel_mode
	[inherit_ctrl_and_mon]
	global_assign_ctrl_inherit_mon
	global_assign_ctrl_assign_mon
	# ls info/kernel_mode_assignment
	ls: cannot access 'info/kernel_mode_assignment': No such file or directory

enable global_assign_ctrl_assign_mon mode:

	# echo "global_assign_ctrl_assign_mon" > info/kernel_mode

Expectation here is that when user space sets this mode as above then resctrl would in turn program MSR_IA32_PQR_PLZA_ASSOC on all CPUs to be:

	MSR_IA32_PQR_PLZA_ASSOC.rmid=0
	MSR_IA32_PQR_PLZA_ASSOC.rmid_en=1
	MSR_IA32_PQR_PLZA_ASSOC.closid=0
	MSR_IA32_PQR_PLZA_ASSOC.closid_en=1
	MSR_IA32_PQR_PLZA_ASSOC.plza_en=1

I do not see why it is necessary to maintain any per-CPU or per-task state or needing to touch the context switch code. Since PLZA only supports global could it not just set MSR_IA32_PQR_PLZA_ASSOC on all online CPUs and be done with it? Only caveat is that if a CPU is offline then this setting needs to be stashed so that MSR_IA32_PQR_PLZA_ASSOC can be set when new CPU comes online.

The way that rdtgroup_config_kmode() introduced in patch #11 assumes it is dealing with RDT_RESOURCE_L3 and traverses the resource domain list and resource group CPU mask seems unnecessary to me as well as error prone since the system may only have, for example, RDT_RESOURCE_MBA enabled or even just monitoring. Why not just set MSR_IA32_PQR_PLZA_ASSOC on all CPUs and be done?

To continue the scenarios ...

After user's setting above related files read:

	# cat info/kernel_mode
	inherit_ctrl_and_mon
	global_assign_ctrl_inherit_mon
	[global_assign_ctrl_assign_mon]
	# cat info/kernel_mode_assignment
	//

Modify group used by global_assign_ctrl_assign_mon mode:

	# echo 'ctrl1/mon1/' > info/kernel_mode_assignment

Expectation here is that when user space sets this then resctrl would program MSR_IA32_PQR_PLZA_ASSOC on all CPUs to be:

	MSR_IA32_PQR_PLZA_ASSOC.rmid=<rmid of mon1>
	MSR_IA32_PQR_PLZA_ASSOC.rmid_en=1
	MSR_IA32_PQR_PLZA_ASSOC.closid=<closid of ctrl1>
	MSR_IA32_PQR_PLZA_ASSOC.closid_en=1
	MSR_IA32_PQR_PLZA_ASSOC.plza_en=1

Enable global_assign_ctrl_inherit_mon mode:

	# echo "global_assign_ctrl_inherit_mon" > info/kernel_mode

Expectation here is that when user space sets this mode then resctrl would program MSR_IA32_PQR_PLZA_ASSOC on all CPUs to be:

	MSR_IA32_PQR_PLZA_ASSOC.rmid=0
	MSR_IA32_PQR_PLZA_ASSOC.rmid_en=0
	MSR_IA32_PQR_PLZA_ASSOC.closid=0
	MSR_IA32_PQR_PLZA_ASSOC.closid_en=1
	MSR_IA32_PQR_PLZA_ASSOC.plza_en=1

	# cat info/kernel_mode
	inherit_ctrl_and_mon
	[global_assign_ctrl_inherit_mon]
	global_assign_ctrl_assign_mon
	# cat info/kernel_mode_assignment <==== returns just a ctrl group
	/

Modify group used by global_assign_ctrl_inherit_mon mode:

	# echo ctrl1 > info/kernel_mode_assignment

Expectation here is that when user space sets this then resctrl would program MSR_IA32_PQR_PLZA_ASSOC on all CPUs to be:

	MSR_IA32_PQR_PLZA_ASSOC.rmid=0
	MSR_IA32_PQR_PLZA_ASSOC.rmid_en=0
	MSR_IA32_PQR_PLZA_ASSOC.closid=<closid of ctrl1>
	MSR_IA32_PQR_PLZA_ASSOC.closid_en=1
	MSR_IA32_PQR_PLZA_ASSOC.plza_en=1

	# cat info/kernel_mode_assignment <==== returns just a ctrl group
	ctrl/

Enable inherit_ctrl_and_mon mode:

	# echo "inherit_ctrl_and_mon" > info/kernel_mode

Expectation here is that when user space sets this mode then resctrl would program MSR_IA32_PQR_PLZA_ASSOC on all CPUs to be:

	MSR_IA32_PQR_PLZA_ASSOC.rmid=0
	MSR_IA32_PQR_PLZA_ASSOC.rmid_en=0
	MSR_IA32_PQR_PLZA_ASSOC.closid=0
	MSR_IA32_PQR_PLZA_ASSOC.closid_en=0
	MSR_IA32_PQR_PLZA_ASSOC.plza_en=0

At this point info/kernel_mode_assignment is not visible anymore:

	# ls info/kernel_mode_assignment
	ls: cannot access 'info/kernel_mode_assignment': No such file or directory

Quoted 1 line

From what I understand above exposes and enables full capability of PLZA. All the other

per-task and per-cpu handling in this series is not something that PLZA can benefit from. If this is not the case, what am I missing? Could this series be simplified to just support PLZA today? When next hardware with more capability needs to be supported resctrl could be enhanced to support it by using the more accurate information about what the hardware is capable of.

We also do not really know what use cases users prefer. This may even be sufficient.

Reinette

Hi Reinette,

Thanks for the review comments. Will address one by one.

Quoted 38 lines · Reinette Chatre

Hi Babu,

Quoted 22 lines · Babu Moger

This series adds support for Privilege-Level Zero Association (PLZA) to the resctrl subsystem. PLZA is an AMD feature that allows specifying a CLOSID and/or RMID for execution in kernel mode (privilege level zero), so that kernel work is not subject to the same resource constrains as the current user-space task. This avoids kernel operations being aggressively throttled when a task's memory bandwidth is heavily limited.

The feature documentation is not yet publicly available, but it is expected to be released in the next few weeks. In the meantime, a brief description of the features is provided below.

Privilege Level Zero Association (PLZA)

Privilege Level Zero Association (PLZA) allows the hardware to automatically associate execution in Privilege Level Zero (CPL=0) with a specific COS (Class of Service) and/or RMID (Resource Monitoring Identifier). The QoS feature set already has a mechanism to associate execution on each logical processor with an RMID or COS. PLZA allows the system to override this per-thread association for a thread that is executing with CPL=0.
------------------------------------------------------------------------

Quoted 5 lines

The series introduces the feature in a way that supports the interface in a generic manner to accomodate MPAM or other vendor specific implimentation.

Below is the detailed requirements provided by Reinette:
https://lore.kernel.org/lkml/[email protected]/

Our discussion considered how resctrl could support PLZA in a generic way while also preparing to support MPAM's variants and how PLZA may evolve to have similar capabilities when considering the capabilities of its registers.

This does not mean that your work needs to implement everything that was discussed. Instead, this work is expected to just support what PLZA is capable of today but do so in a way that the future enhancements could be added to.

Quoted 5 lines

This series is quite difficult to follow since it appears to implement a full featured generic interface while PLZA cannot take advantage of it.

Could you please simplify this work to focus on just enabling PLZA and only add interfaces needed to do so?

Sure. Will try. Lets continue the discussion.

Quoted 54 lines

Quoted 54 lines

Summary:
1. Kernel-mode/PLZA controls and status should be exposed under the resctrl info directory:/sys/fs/resctrl/info/, not as a separate or arch-specific path.

2. Add two info files

  a. kernel_mode
     Purpose: Control how resource allocation and monitoring apply in kernel mode (e.g. inherit from task vs global assign).

     Read: List supported modes and show current one (e.g. with [brackets]).
     Write: Set current mode by name (e.g. inherit_ctrl_and_mon, global_assign_ctrl_assign_mon).

b. kernel_mode_assignment

    Purpose: When a “global assign” kernel mode is active, specify which resctrl group (CLOSID/RMID) is used for kernel work.

    Read: Show the assigned group in a path-like form (e.g. //, ctrl1//, ctrl1/mon1/).
    Write: Assign or clear the group used for kernel mode (and optionally clear with an empty write).

The patches are based on top of commit (v7.0.0-rc3) 839e91ce3f41b (tip/master) Merge branch into tip/master: 'x86/tdx'
------------------------------------------------------------------------

Examples: kernel_mode and kernel_mode_assignment

All paths below are under /sys/fs/resctrl/ (e.g. info/kernel_mode means /sys/fs/resctrl/info/kernel_mode). Resctrl must be mounted and the platform must support the relevant modes (e.g. AMD with PLZA).

1) kernel_mode — show and set the current kernel mode

    Read supported modes and which one is active (current in brackets):

      $ cat info/kernel_mode
      [inherit_ctrl_and_mon]
      global_assign_ctrl_inherit_mon
      global_assign_ctrl_assign_mon

    Set the active mode (e.g. use one CLOSID+RMID for all kernel work):

      $ echo "global_assign_ctrl_assign_mon" > info/kernel_mode
      $ cat info/kernel_mode
      inherit_ctrl_and_mon
      global_assign_ctrl_inherit_mon
      [global_assign_ctrl_assign_mon]

    Mode meanings:
    - inherit_ctrl_and_mon: kernel uses same CLOSID/RMID as the current task (default).
    - global_assign_ctrl_inherit_mon: one CLOSID for all kernel work; RMID inherited from user.
    - global_assign_ctrl_assign_mon: one resource group (CLOSID+RMID) for all kernel work.

Quoted 8 lines

Quoted 3 lines

2) kernel_mode_assignment — show and set which group is used for kernel work

    Only relevant when kernel_mode is not "inherit_ctrl_and_mon". Read the

To help with future usages please connect visibility of this file with the mode in info/kernel_mode. This helps us to support future modes with other resctrl files, possible within each resource group.
Specifically, kernel_mode_assignment is not visible to user space if mode is "inherit_ctrl_and_mon", while it is visible when mode is global_assign_ctrl_inherit_mon or global_assign_ctrl_assign_mon.

Sure. Will do.

Quoted 1 line

Quoted 5 lines

Quoted 1 line

    currently assigned group (path format is "CTRL_MON/MON/"):

The format depends on the mode, right? If the mode is "global_assign_ctrl_inherit_mon" then it should only contain a control group, alternatively, if the mode is "global_assign_ctrl_assign_mon" then it contains control and mon group. This gives resctrl future flexibility to change format for future modes.

This can be done both ways.  Whole purpose of these groups is to get CLOSID and RMID to enable PLZA. User can echo CTRL_MON or MON group to kernel_mode_assignment in any of the modes.  We can decide what needs to be updated in MSR (PQR_PLZA_ASSOC) based on what kernel mode is selected.

Quoted 8 lines

Quoted 8 lines

      Tony suggested using global variables to store the kernel mode CLOSID and RMID. However, the kernel mode CLOSID and RMID are coming from rdtgroup structure with the new interface. Accessing them requires holding the associated lock, which would make the context switch path unnecessarily expensive. So, dropped the idea.
      https://lore.kernel.org/lkml/aXuxVSbk1GR2ttzF@agluck-desk3/
      Let me know if there are other ways to optimize this.

Quoted 4 lines

I do not see why the context switch path needs to be touched at all with this implementation. Since PLZA only supports global assignment does it not mean that resctrl only needs to update PQR_PLZA_ASSOC when user writes to info/kernel_mode and info/kernel_mode_assignment?

Each thread has an MSR to configure whether to associate privilege level zero execution with a separate COS and/or RMID, and the value of the COS and/or RMID.  PLZA may be enabled or disabled on a per-thread basis. However, the COS and RMID association and configuration must be the same for all threads in the QOS Domain.

So, PQR_PLZA_ASSOC is a per thread MSR just like PQR_ASSOC.

Privilege-Level Zero Association (PLZA) allows the user to specify a COS and/or RMID associated with execution in Privilege-Level Zero. When enabled on a HW thread, when that thread enters Privilige-Level Zero, transactions associated with that thread will be associated with the PLZA COS and/or RMID. Otherwise, the HW thread will be associated with the COS and RMID identified by  PQR_ASSOC.

More below.

Quoted 48 lines

Consider some of the scenarios:

resctrl mount with default state:

    # cat info/kernel_mode
    [inherit_ctrl_and_mon]
    global_assign_ctrl_inherit_mon
    global_assign_ctrl_assign_mon
    # ls info/kernel_mode_assignment
    ls: cannot access 'info/kernel_mode_assignment': No such file or directory

enable global_assign_ctrl_assign_mon mode:

    # echo "global_assign_ctrl_assign_mon" > info/kernel_mode

Expectation here is that when user space sets this mode as above then resctrl would in turn program MSR_IA32_PQR_PLZA_ASSOC on all CPUs to be:
    MSR_IA32_PQR_PLZA_ASSOC.rmid=0
    MSR_IA32_PQR_PLZA_ASSOC.rmid_en=1
    MSR_IA32_PQR_PLZA_ASSOC.closid=0 MSR_IA32_PQR_PLZA_ASSOC.closid_en=1 MSR_IA32_PQR_PLZA_ASSOC.plza_en=1

I do not see why it is necessary to maintain any per-CPU or per-task state or needing to touch the context switch code. Since PLZA only supports global could it not just set MSR_IA32_PQR_PLZA_ASSOC on all online CPUs and be done with it? Only caveat is that if a CPU is offline then this setting needs to be stashed so that MSR_IA32_PQR_PLZA_ASSOC can be set when new CPU comes online.

The way that rdtgroup_config_kmode() introduced in patch #11 assumes it is dealing with RDT_RESOURCE_L3 and traverses the resource domain list and resource group CPU mask seems unnecessary to me as well as error prone since the system may only have, for example, RDT_RESOURCE_MBA enabled or even just monitoring. Why not just set MSR_IA32_PQR_PLZA_ASSOC on all CPUs and be done?

To continue the scenarios ...

After user's setting above related files read:

    # cat info/kernel_mode
    inherit_ctrl_and_mon
    global_assign_ctrl_inherit_mon
    [global_assign_ctrl_assign_mon]
    # cat info/kernel_mode_assignment
    //

Modify group used by global_assign_ctrl_assign_mon mode:

    # echo 'ctrl1/mon1/' > info/kernel_mode_assignment

Quoted 7 lines

Expectation here is that when user space sets this then resctrl would program MSR_IA32_PQR_PLZA_ASSOC on all CPUs to be:
    MSR_IA32_PQR_PLZA_ASSOC.rmid=<rmid of mon1> MSR_IA32_PQR_PLZA_ASSOC.rmid_en=1 MSR_IA32_PQR_PLZA_ASSOC.closid=<closid of ctrl1> MSR_IA32_PQR_PLZA_ASSOC.closid_en=1 MSR_IA32_PQR_PLZA_ASSOC.plza_en=1

This works correctly when PLZA associations are defined by per CPU. For example, lets assume that *ctrl1* is assigned *CLOSID 1*.

In this scenario, every task in the system running on a any CPU will use the limits associated with *CLOSID 1* whenever it enters Privilege-Level Zero, because the CPU's *PQR_PLZA_ASSOC* register has PLZA enabled and CLOSID is 1.

Now consider task-based association:

We have two resctrl groups:

 * *ctrl1 -> CLOSID 1 -> task1.plza = 1   : *User wants PLZA be enabled for this task.
 * *ctrl2 -> CLOSID 2 -> task2.plza = 0   : *User wants PLZA disabled for this task.

Suppose *task1* is first scheduled on *CPU 0*. This behaves as expected: since CPU 0 's *PQR_PLZA_ASSOC* contains *CLOSID 1, plza_en =1*, task1 will use the limits from CLOSID 1 when it enters Privilege-Level Zero.

However, if *task2* later runs on *CPU 0*, we expect it to use *CLOSID 2* in both user mode and kernel mode, because user has PLZA disabled for this task. But CPU 0 still has *CLOSID 1, **plza_en =1* in its PQR_PLZA_ASSOC register.

As a result, task2 will incorrectly run with *CLOSID 1* when entering Privilege-Level Zero something we explicitly want to avoid.

At that point, PLZA must be disabled on CPU 0 to prevent the unintended association. Hope this explanation makes the issue clear.

Hi Babu,

Quoted 117 lines · Babu Moger

Hi Reinette,

Thanks for the review comments. Will address one by one.

Quoted 38 lines · Reinette Chatre

Hi Babu,

Quoted 22 lines · Babu Moger

This series adds support for Privilege-Level Zero Association (PLZA) to the resctrl subsystem. PLZA is an AMD feature that allows specifying a CLOSID and/or RMID for execution in kernel mode (privilege level zero), so that kernel work is not subject to the same resource constrains as the current user-space task. This avoids kernel operations being aggressively throttled when a task's memory bandwidth is heavily limited.

The feature documentation is not yet publicly available, but it is expected to be released in the next few weeks. In the meantime, a brief description of the features is provided below.

Privilege Level Zero Association (PLZA)

Privilege Level Zero Association (PLZA) allows the hardware to automatically associate execution in Privilege Level Zero (CPL=0) with a specific COS (Class of Service) and/or RMID (Resource Monitoring Identifier). The QoS feature set already has a mechanism to associate execution on each logical processor with an RMID or COS. PLZA allows the system to override this per-thread association for a thread that is executing with CPL=0.
------------------------------------------------------------------------

Quoted 5 lines

The series introduces the feature in a way that supports the interface in a generic manner to accomodate MPAM or other vendor specific implimentation.

Below is the detailed requirements provided by Reinette:
https://lore.kernel.org/lkml/[email protected]/

Our discussion considered how resctrl could support PLZA in a generic way while also preparing to support MPAM's variants and how PLZA may evolve to have similar capabilities when considering the capabilities of its registers.

This does not mean that your work needs to implement everything that was discussed. Instead, this work is expected to just support what PLZA is capable of today but do so in a way that the future enhancements could be added to.

Quoted 5 lines

This series is quite difficult to follow since it appears to implement a full featured generic interface while PLZA cannot take advantage of it.

Could you please simplify this work to focus on just enabling PLZA and only add interfaces needed to do so?

Sure. Will try. Lets continue the discussion.

Quoted 53 lines

Quoted 53 lines

Summary:
1. Kernel-mode/PLZA controls and status should be exposed under the resctrl info directory:/sys/fs/resctrl/info/, not as a separate or arch-specific path.

2. Add two info files

  a. kernel_mode
     Purpose: Control how resource allocation and monitoring apply in kernel mode (e.g. inherit from task vs global assign).

     Read: List supported modes and show current one (e.g. with [brackets]).
     Write: Set current mode by name (e.g. inherit_ctrl_and_mon, global_assign_ctrl_assign_mon).

b. kernel_mode_assignment

    Purpose: When a “global assign” kernel mode is active, specify which resctrl group (CLOSID/RMID) is used for kernel work.

    Read: Show the assigned group in a path-like form (e.g. //, ctrl1//, ctrl1/mon1/).
    Write: Assign or clear the group used for kernel mode (and optionally clear with an empty write).

The patches are based on top of commit (v7.0.0-rc3) 839e91ce3f41b (tip/master) Merge branch into tip/master: 'x86/tdx'
------------------------------------------------------------------------

Examples: kernel_mode and kernel_mode_assignment

All paths below are under /sys/fs/resctrl/ (e.g. info/kernel_mode means /sys/fs/resctrl/info/kernel_mode). Resctrl must be mounted and the platform must support the relevant modes (e.g. AMD with PLZA).

1) kernel_mode — show and set the current kernel mode

    Read supported modes and which one is active (current in brackets):

       $ cat info/kernel_mode
       [inherit_ctrl_and_mon]
       global_assign_ctrl_inherit_mon
       global_assign_ctrl_assign_mon

    Set the active mode (e.g. use one CLOSID+RMID for all kernel work):

       $ echo "global_assign_ctrl_assign_mon" > info/kernel_mode
       $ cat info/kernel_mode
       inherit_ctrl_and_mon
       global_assign_ctrl_inherit_mon
       [global_assign_ctrl_assign_mon]

    Mode meanings:
    - inherit_ctrl_and_mon: kernel uses same CLOSID/RMID as the current task (default).
    - global_assign_ctrl_inherit_mon: one CLOSID for all kernel work; RMID inherited from user.
    - global_assign_ctrl_assign_mon: one resource group (CLOSID+RMID) for all kernel work.

Quoted 8 lines

Quoted 3 lines

2) kernel_mode_assignment — show and set which group is used for kernel work

    Only relevant when kernel_mode is not "inherit_ctrl_and_mon". Read the

To help with future usages please connect visibility of this file with the mode in info/kernel_mode. This helps us to support future modes with other resctrl files, possible within each resource group.
Specifically, kernel_mode_assignment is not visible to user space if mode is "inherit_ctrl_and_mon", while it is visible when mode is global_assign_ctrl_inherit_mon or global_assign_ctrl_assign_mon.

Sure. Will do.

Quoted 5 lines

Quoted 1 line

    currently assigned group (path format is "CTRL_MON/MON/"):

The format depends on the mode, right? If the mode is "global_assign_ctrl_inherit_mon" then it should only contain a control group, alternatively, if the mode is "global_assign_ctrl_assign_mon" then it contains control and mon group. This gives resctrl future flexibility to change format for future modes.

Quoted 1 line

This can be done both ways.  Whole purpose of these groups is to get CLOSID and RMID to enable PLZA. User can echo CTRL_MON or MON group to kernel_mode_assignment in any of the modes.  We can decide what needs to be updated in MSR (PQR_PLZA_ASSOC) based on what kernel mode is selected.

The "both ways" are specific to one of the two active modes though. PLZA only needs the RMID when the mode is "global_assign_ctrl_assign_mon".

Displaying and parsing monitor group when the mode is "global_assign_ctrl_inherit_mon" creates an inconsistent interface since the mode only uses a control group. The interface to user space should match the mode otherwise it becomes confusing.

...

Quoted 7 lines

Quoted 7 lines

Quoted 7 lines

      Tony suggested using global variables to store the kernel mode CLOSID and RMID. However, the kernel mode CLOSID and RMID are coming from rdtgroup structure with the new interface. Accessing them requires holding the associated lock, which would make the context switch path unnecessarily expensive. So, dropped the idea.
      https://lore.kernel.org/lkml/aXuxVSbk1GR2ttzF@agluck-desk3/
      Let me know if there are other ways to optimize this.

Quoted 5 lines

Quoted 4 lines

I do not see why the context switch path needs to be touched at all with this implementation. Since PLZA only supports global assignment does it not mean that resctrl only needs to update PQR_PLZA_ASSOC when user writes to info/kernel_mode and info/kernel_mode_assignment?

Each thread has an MSR to configure whether to associate privilege level zero execution with a separate COS and/or RMID, and the value of the COS and/or RMID.  PLZA may be enabled or disabled on a per-thread basis. However, the COS and RMID association and configuration must be the same for all threads in the QOS Domain.

Based on previous comment in https://lore.kernel.org/lkml/[email protected]/
and this implementation all fields of PQR_PLZA_ASSOC except PQR_PLZA_ASSOC.plza_en must be the same for all CPUs on the system, not just per QoS domain. Could you please confirm?

Quoted 75 lines

So, PQR_PLZA_ASSOC is a per thread MSR just like PQR_ASSOC.

Privilege-Level Zero Association (PLZA) allows the user to specify a COS and/or RMID associated with execution in Privilege-Level Zero. When enabled on a HW thread, when that thread enters Privilige-Level Zero, transactions associated with that thread will be associated with the PLZA COS and/or RMID. Otherwise, the HW thread will be associated with the COS and RMID identified by  PQR_ASSOC.

More below.

Quoted 47 lines

Consider some of the scenarios:

resctrl mount with default state:

     # cat info/kernel_mode
     [inherit_ctrl_and_mon]
     global_assign_ctrl_inherit_mon
     global_assign_ctrl_assign_mon
     # ls info/kernel_mode_assignment
     ls: cannot access 'info/kernel_mode_assignment': No such file or directory

enable global_assign_ctrl_assign_mon mode:

     # echo "global_assign_ctrl_assign_mon" > info/kernel_mode

Expectation here is that when user space sets this mode as above then resctrl would in turn program MSR_IA32_PQR_PLZA_ASSOC on all CPUs to be:
    MSR_IA32_PQR_PLZA_ASSOC.rmid=0
    MSR_IA32_PQR_PLZA_ASSOC.rmid_en=1 MSR_IA32_PQR_PLZA_ASSOC.closid=0 MSR_IA32_PQR_PLZA_ASSOC.closid_en=1 MSR_IA32_PQR_PLZA_ASSOC.plza_en=1

I do not see why it is necessary to maintain any per-CPU or per-task state or needing to touch the context switch code. Since PLZA only supports global could it not just set MSR_IA32_PQR_PLZA_ASSOC on all online CPUs and be done with it? Only caveat is that if a CPU is offline then this setting needs to be stashed so that MSR_IA32_PQR_PLZA_ASSOC can be set when new CPU comes online.

The way that rdtgroup_config_kmode() introduced in patch #11 assumes it is dealing with RDT_RESOURCE_L3 and traverses the resource domain list and resource group CPU mask seems unnecessary to me as well as error prone since the system may only have, for example, RDT_RESOURCE_MBA enabled or even just monitoring. Why not just set MSR_IA32_PQR_PLZA_ASSOC on all CPUs and be done?

To continue the scenarios ...

After user's setting above related files read:

     # cat info/kernel_mode
     inherit_ctrl_and_mon
     global_assign_ctrl_inherit_mon
     [global_assign_ctrl_assign_mon]
     # cat info/kernel_mode_assignment
     //

Modify group used by global_assign_ctrl_assign_mon mode:

     # echo 'ctrl1/mon1/' > info/kernel_mode_assignment

Quoted 7 lines

Expectation here is that when user space sets this then resctrl would program MSR_IA32_PQR_PLZA_ASSOC on all CPUs to be:
    MSR_IA32_PQR_PLZA_ASSOC.rmid=<rmid of mon1> MSR_IA32_PQR_PLZA_ASSOC.rmid_en=1 MSR_IA32_PQR_PLZA_ASSOC.closid=<closid of ctrl1> MSR_IA32_PQR_PLZA_ASSOC.closid_en=1 MSR_IA32_PQR_PLZA_ASSOC.plza_en=1

This works correctly when PLZA associations are defined by per CPU. For example, lets assume that *ctrl1* is assigned *CLOSID 1*.

In this scenario, every task in the system running on a any CPU will use the limits associated with *CLOSID 1* whenever it enters Privilege-Level Zero, because the CPU's *PQR_PLZA_ASSOC* register has PLZA enabled and CLOSID is 1.

Now consider task-based association:

We have two resctrl groups:

 * *ctrl1 -> CLOSID 1 -> task1.plza = 1   : *User wants PLZA be enabled for this task.
 * *ctrl2 -> CLOSID 2 -> task2.plza = 0   : *User wants PLZA disabled for this task.

Quoted 8 lines

Suppose *task1* is first scheduled on *CPU 0*. This behaves as expected: since CPU 0 's *PQR_PLZA_ASSOC* contains *CLOSID 1, plza_en =1*, task1 will use the limits from CLOSID 1 when it enters Privilege-Level Zero.

However, if *task2* later runs on *CPU 0*, we expect it to use *CLOSID 2* in both user mode and kernel mode, because user has PLZA disabled for this task. But CPU 0 still has *CLOSID 1, **plza_en =1* in its PQR_PLZA_ASSOC register.

As a result, task2 will incorrectly run with *CLOSID 1* when entering Privilege-Level Zero something we explicitly want to avoid.

At that point, PLZA must be disabled on CPU 0 to prevent the unintended association. Hope this explanation makes the issue clear.

A couple of points:
- Looks like we still need to come to agreement what is meant by "global" when it comes to kernel mode.

In your description there is a "global" configuration, but the assignment is "per-task". To me this sounds like a new and distinct kernel_mode from the "global" modes considered so far. This seems to move to the "per_task" mode mentioned in but the implementation does not take into account any of the earlier discussions surrounding it:
https://lore.kernel.org/lkml/[email protected]/

We only learned about one use case in https://lore.kernel.org/lkml/CABPqkBSq=cgn-am4qorA_VN0vsbpbfDePSi7gubicpROB1=djw@mail.gmail.com/
As I understand this use case requires PLZA globally enabled for all tasks. Thus
I consider task assignment to be "global" when in the "global_*" kernel modes.
If this is indeed a common use case then supporting only global configuration
but then requiring user space to manually assign all tasks afterwards sounds
cumbersome for user space and also detrimental to system performance with all
the churn to modify all the task_structs involved. The accompanying documentation
does not mention all this additional user space interactions required by user
space to use this implementation.

I find this implementation difficult and inefficient to use in the one use case we know of. I would suggest that resctrl optimizes for the one known use case.

- This implementation ignores discussion on how existing resctrl files should not be repurposed.

This implementation allows user space to set a resource group in kernel_mode_assignment with the consequence that this resource group's "tasks" file changes behavior. I consider this a break of resctrl interface. We did briefly consider per-task configuration/assignment in previous discussion and the proposal was for it to use a new file (only when and if needed!).

- Now a user is required to write the task id of every task that participates in PLZA. Apart from the churn already mentioned this also breaks existing usage since it is no longer possible for new tasks to be added to this resource group. This creates an awkward interface where all tasks belonging to a resource group inherits the allocations/monitoring for their user space work and will get PLZA enabled whether user requested it or not while tasks from other resource groups need to be explicitly enabled. This creates an inconsistency when it comes to task assignment. The only way to "remove" PLZA from such a task would be to assign it to another resource group which may not have the user space allocations ... and once this is done the task cannot be moved back.
There is no requirement that CLOSID/RMID should be dedicated to kernel work but this implementation does so in an inconsistent way.

- Apart from the same issues as with repurposing of tasks file, why should same CPU allocation be used for kernel and user space?

Quoted 15 lines · Reinette Chatre

Quoted 14 lines · Babu Moger

Quoted 13 lines · Reinette Chatre

Quoted 7 lines · Babu Moger

       Tony suggested using global variables to store the kernel mode CLOSID and RMID. However, the kernel mode CLOSID and RMID are coming from rdtgroup structure with the new interface. Accessing them requires holding the associated lock, which would make the context switch path unnecessarily expensive. So, dropped the idea.
       https://lore.kernel.org/lkml/aXuxVSbk1GR2ttzF@agluck-desk3/
       Let me know if there are other ways to optimize this.

I do not see why the context switch path needs to be touched at all with this implementation. Since PLZA only supports global assignment does it not mean that resctrl only needs to update PQR_PLZA_ASSOC when user writes to info/kernel_mode and info/kernel_mode_assignment?

Quoted 4 lines

Quoted 1 line

Each thread has an MSR to configure whether to associate privilege level zero execution with a separate COS and/or RMID, and the value of the COS and/or RMID.  PLZA may be enabled or disabled on a per-thread basis. However, the COS and RMID association and configuration must be the same for all threads in the QOS Domain.

Based on previous comment in https://lore.kernel.org/lkml/[email protected]/
and this implementation all fields of PQR_PLZA_ASSOC except PQR_PLZA_ASSOC.plza_en must be the same for all CPUs on the system, not just per QoS domain. Could you please confirm?

Sorry for the confusion. It is "per QoS domain".

All the fields of PQR_PLZA_ASSOC except PQR_PLZA_ASSOC.plza_enmust be set to the same value for all HW threads in the QOS domain for consistent operation (Per-QosDomain).

Yes, I agree with your concerns. The goal here is to make the interface less disruptive while still addressing the different use cases.

     Background: Customers have identified an issue with the QoS Bandwidth Control feature: when a CLOS is aggressively throttled and execution transitions into kernel mode, kernel operations are also subject to the same aggressive throttling.

Quoted 1 line

Privilege-Level Zero Association (PLZA) allows a user to specify a

COS and/or RMID to be used during execution at Privilege Level Zero. When PLZA is enabled on a hardware thread, any execution that enters Privilege Level Zero will have its transactions associated with the PLZA COS and/or RMID. Otherwise, the thread continues to use the COS and RMID specified by |PQR_ASSOC|. In other words, the hardware provides a dedicated COS and/or RMID specifically for kernel-mode execution.

There are multiple ways this feature can be applied. For simplicity, the discussion below focuses only on CLOSID.

     1. Global PLZA enablement

PLZA can be configured as a global feature by setting |PQR_PLZA_ASSOC.closid = CLOSID| and |PQR_PLZA_ASSOC.plza_en = 1| on all threads in the system. A dedicated CLOSID is reserved for this purpose,

and all CPU threads use its allocations whenever they enter Privilege Level Zero. This CLOSID does not need to be associated with any resctrl group.

The user can explicitly enable or disable this feature.

There is no context switch overhead but there is no flexibility with this approach.

     2. Group based PLZA allocation :  PLZA is managed via dedicated restctrl group. A separate resctrl group can be created specifically for PLZA, with a dedicated CLOSID used exclusively for kernel mode execution. This approach can be further divided into two association models:

i) CPU based association
CPUs are assigned to the PLZA group, and PLZA is enabled only on those CPUs. This effectively creates a dedicated PLZA group. MSRs (| PQR_PLZA_ASSOC)| are programmed only when the user changes CPU assignments. This approach requires no changes to the context switch code and introduces no additional context switch overhead.

ii) Task based association
Tasks are explicitly assigned by the user to the PLZA group. Tasks need to be updated when user adds a new task. Also, this requires updates during task scheduling so that the MSRs (|PQR_PLZA_ASSOC)| are programmed on each context switch, which introduces additional context switch overhead.

I tried to fit these requirements into the interface files in /sys/ fs/resctrl/info/. I may have missed few things while trying to achieve it. As usual, I am open for the discussion and recommendations.

Hi Babu,

Quoted 21 lines · Babu Moger

Quoted 19 lines · Reinette Chatre

Quoted 10 lines · Babu Moger

Quoted 9 lines · Reinette Chatre

Quoted 7 lines · Babu Moger

       Tony suggested using global variables to store the kernel mode CLOSID and RMID. However, the kernel mode CLOSID and RMID are coming from rdtgroup structure with the new interface. Accessing them requires holding the associated lock, which would make the context switch path unnecessarily expensive. So, dropped the idea.
       https://lore.kernel.org/lkml/aXuxVSbk1GR2ttzF@agluck-desk3/
       Let me know if there are other ways to optimize this.

Quoted 5 lines

Quoted 4 lines

I do not see why the context switch path needs to be touched at all with this implementation. Since PLZA only supports global assignment does it not mean that resctrl only needs to update PQR_PLZA_ASSOC when user writes to info/kernel_mode and info/kernel_mode_assignment?

Each thread has an MSR to configure whether to associate privilege level zero execution with a separate COS and/or RMID, and the value of the COS and/or RMID.  PLZA may be enabled or disabled on a per-thread basis. However, the COS and RMID association and configuration must be the same for all threads in the QOS Domain.

Based on previous comment in https://lore.kernel.org/lkml/[email protected]/
and this implementation all fields of PQR_PLZA_ASSOC except PQR_PLZA_ASSOC.plza_en must be the same for all CPUs on the system, not just per QoS domain. Could you please confirm?

Quoted 3 lines

Sorry for the confusion. It is "per QoS domain".

All the fields of PQR_PLZA_ASSOC except PQR_PLZA_ASSOC.plza_enmust be set to the same value for all HW threads in the QOS domain for consistent operation (Per-QosDomain).

Thank you for clarifying. To build on this, what would be best way for resctrl to interpret this? As I see it all values in PQR_PLZA_ASSOC apply to *all* resources yet (theoretically?) every resource

can have domains that span different CPUs. There thus seem to be a built in assumption of what a "domain" means for PQR_PLZA_ASSOC so it sounds to me as though, instead of saying that "PQR_PLZA_ASSOC needs to be the same in QoS domain" it may be more accurate to, for example, say that "PQR_PLZA_ASSOC has L3 scope"?

This seems to be what this implementation does since it hardcodes PQR_PLZA_ASSOC scope to the L3 resource but that creates dependency to the L3 resource that would make PLZA unusable if, for example, the user boots with "rdt=!l3cat" while wanting to use PLZA to manage MBA allocations when in kernel?

...

Quoted 1 line

Yes, I agree with your concerns. The goal here is to make the interface less disruptive while still addressing the different use cases.

I consider changing resctrl behavior when values are written to existing resctrl files to be disruptive. This is something we explicitly discussed during v1 as something to be avoided so this implementation that overloads the tasks file again is unexpected.

Quoted 6 lines

     Background: Customers have identified an issue with the QoS Bandwidth Control feature: when a CLOS is aggressively throttled and execution transitions into kernel mode, kernel operations are also subject to the same aggressive throttling.

Quoted 8 lines

Quoted 1 line

Privilege-Level Zero Association (PLZA) allows a user to specify a

COS and/or RMID to be used during execution at Privilege Level Zero. When PLZA is enabled on a hardware thread, any execution that enters Privilege Level Zero will have its transactions associated with the PLZA COS and/or RMID. Otherwise, the thread continues to use the COS and RMID specified by |PQR_ASSOC|. In other words, the hardware provides a dedicated COS and/or RMID specifically for kernel-mode execution.

ack.

Quoted 6 lines

There are multiple ways this feature can be applied. For simplicity, the discussion below focuses only on CLOSID.

     1. Global PLZA enablement

Quoted 1 line

PLZA can be configured as a global feature by setting |PQR_PLZA_ASSOC.closid = CLOSID| and |PQR_PLZA_ASSOC.plza_en = 1| on all threads in the system. A dedicated CLOSID is reserved for this purpose,

Also discussed during v1 is that there is no need to dedicate a CLOSID for this purpose. There could be an "unthrottled" CLOSID to which all high priority user space tasks as well as all kernel work of all tasks are assigned.
If user space chooses to dedicate a CLOSID for kernel work then that should supported and interface can allow that, but there is no need for resctrl to enforce this.

Quoted 1 line

and all CPU threads use its allocations whenever they enter Privilege Level Zero. This CLOSID does not need to be associated with any resctrl group.

The CLOSID has to be associated with a resource group to be able to manage its resource allocations, no?

Quoted 2 lines

The user can explicitly enable or disable this feature.

ack.

Quoted 1 line

There is no context switch overhead but there is no flexibility with this approach.

Flexibility is subjective. As I understand this supports the only use case we learned about so far:
https://lore.kernel.org/lkml/CABPqkBSq=cgn-am4qorA_VN0vsbpbfDePSi7gubicpROB1=djw@mail.gmail.com/

Quoted 5 lines

     2. Group based PLZA allocation :  PLZA is managed via dedicated restctrl group. A separate resctrl group can be created specifically for PLZA, with a dedicated CLOSID used exclusively for kernel mode execution. This approach can be further divided into two association models:

So far this sounds like global allocation since both need a dedicated resource group. Whether this group is dedicated to kernel work or shared between kernel and user space work is up to the user. There is no motivation why CLOSID should ever be enforced to be exclusive for kernel mode execution.

Quoted 9 lines

i) CPU based association
CPUs are assigned to the PLZA group, and PLZA is enabled only on those CPUs. This effectively creates a dedicated PLZA group. MSRs (| PQR_PLZA_ASSOC)| are programmed only when the user changes CPU assignments. This approach requires no changes to the context switch code and introduces no additional context switch overhead.

Quoted 6 lines

ii) Task based association
Tasks are explicitly assigned by the user to the PLZA group. Tasks need to be updated when user adds a new task. Also, this requires updates during task scheduling so that the MSRs (|PQR_PLZA_ASSOC)| are programmed on each context switch, which introduces additional context switch overhead.

As discussed during v1 any changes needed to support per task assignment would need to be done with new files dedicated to this purpose. Do not overload the existing resctrl tasks/cpus/cpus_list files.

Quoted 5 lines

I tried to fit these requirements into the interface files in /sys/ fs/resctrl/info/. I may have missed few things while trying to achieve it. As usual, I am open for the discussion and recommendations.

Many of these items were already discussed as part of v1 so I think we may be talking past each other here. I tried to highlight the relevant points raised during v1 discussion that I thought there already was agreement on.

The one new aspect is that I assumed this implementation will only be for global configuration and assignment. It looks like you want to support both global configuration and per-task assignment. In the original I did not consider configuration and assignment to occur at different scope so we may need to come up with new modes to distinguish. Consider the addition of two modes as below:

	# cat info/kernel_mode
	[inherit_ctrl_and_mon]
	global_assign_ctrl_inherit_mon_set_all
	global_assign_ctrl_assign_mon_set_all
	global_assign_ctrl_inherit_mon_set_individual
	global_assign_ctrl_assign_mon_set_individual

Above introduces a "set_all" and "set_individual" suffix to the original two modes.

global_assign_ctrl_inherit_mon_set_all
global_assign_ctrl_assign_mon_set_all:

Above are the original two modes but makes it clear that when this mode is activated _all_ tasks run with the assignment.

global_assign_ctrl_inherit_mon_set_individual
global_assign_ctrl_assign_mon_set_individual:

Above are two new modes. In this mode user space also assigns a resource group globally but then needs to follow that up by activating every task separately to run with this assignment.
One way in which this can be accomplished could be to have "kernel_mode_tasks",
"kernel_mode_cpus", and "kernel_mode_cpus_list" files become visible (or be
created) in the resource group found in info/kernel_mode_assignment. User
space interacts with the new files to set which tasks and/or CPUs run with PLZA enabled.

Even so, as I understand global_assign_ctrl_inherit_mon_set_all and global_assign_ctrl_assign_mon_set_all addresses the only known use case. Do you know if there are use cases for global_assign_ctrl_inherit_mon_set_individual and global_assign_ctrl_assign_mon_set_individual? The latter two adds significant complexity to resctrl while I have not heard about any use case for it.

Hi Reinette,

Sorry for the late response. I was trying to get confirmation about the use case.

Quoted 9 lines · Babu Moger

Quoted 4 lines · Reinette Chatre

Quoted 3 lines · Babu Moger

Quoted 2 lines · Reinette Chatre

Quoted 1 line · Babu Moger

can have domains that span different CPUs. There thus seem to be a built in assumption of what a "domain" means for PQR_PLZA_ASSOC so it sounds to me as though, instead of saying that "PQR_PLZA_ASSOC needs to be the same in QoS domain" it may be more accurate to, for example, say that "PQR_PLZA_ASSOC has L3 scope"?

Yes.

Quoted 4 lines

This seems to be what this implementation does since it hardcodes PQR_PLZA_ASSOC scope to the L3 resource but that creates dependency to the L3 resource that would make PLZA unusable if, for example, the user boots with "rdt=!l3cat" while wanting to use PLZA to manage MBA allocations when in kernel?

Yes. that is correct. It should not be attached to one resource. We need to change it to global scope.

Quoted 5 lines

Quoted 5 lines

There are multiple ways this feature can be applied. For simplicity, the discussion below focuses only on CLOSID.

      1. Global PLZA enablement

Quoted 7 lines

Quoted 1 line

PLZA can be configured as a global feature by setting |PQR_PLZA_ASSOC.closid = CLOSID| and |PQR_PLZA_ASSOC.plza_en = 1| on all threads in the system. A dedicated CLOSID is reserved for this purpose,

Also discussed during v1 is that there is no need to dedicate a CLOSID for this purpose. There could be an "unthrottled" CLOSID to which all high priority user space tasks as well as all kernel work of all tasks are assigned.
If user space chooses to dedicate a CLOSID for kernel work then that should supported and interface can allow that, but there is no need for resctrl to enforce this.

Yes. I agree. The changes in context switch code is a concern.

You covered some of the cases I was thinking(xx_set_individual).

How about this idea?

I suggest splitting the PLZA into two distinct aspects:

1. How PLZA is applied within a resource group

2. How PLZA is monitored

Introduce a new file, "info/kmode_type", to describe how kmode applies in the system.

# cat info/kmode_type
[global] <- Kernel mode applies to the entire system (all CPUs/tasks)
  cpus   <- Kernel mode applies only to the CPUs in the group
  tasks  <- Kernel mode applies only to the tasks in the group

The "global" option is the default right now and it is current common use-case.

The "info/kmode_type -> cpus" option introduces new files "kmode_cpus" and "kmode_cpus_list" for users to apply kmode to specific set of CPUs. This lets users change the CPU set for PLZA.

The PLZA MSR is updated when user changes the association to the file. No context switch code changes are needed. This will be dedicated group. The current resctrl group files, "cpus, cpus_list

and tasks" will not be accessible in this mode. This option give

some flexibility for the user without the context switch overhead.

The "info/kmode_type -> tasks" option introduces a new file, "kmode_tasks", for users to apply kmode to specific set of tasks. This requires context switch changes. This will be dedicated group. The current resctrl group files, "cpus, cpus_list and tasks" will not be accessible in this mode. We currently have no use case for this, so it will not be supported now.

Add a file, "info/kmode_monitor", to describe how kmode is monitored.

# cat info/kmode_monitor
[inherit_ctrl_and_mon] <- Kernel uses the same CLOSID/RMID as user. Default option for the "global"
assign_ctrl_inherit_mon <- One CLOSID for all kernel work; RMID inherited from user.
assign_ctrl_assign_mon <- One resource group (CLOSID+RMID) for all kernel work. Default option for "cpu" type.

Rename “kernel_mode_assignment” to “kmode_group” to assign the specific group to kmode. This file usage is same as before.

#cat info/kmode_groups (Renamed "kernel_mode_assignment") //

Hi Babu,

Quoted 3 lines · Babu Moger

Hi Reinette,

Sorry for the late response. I was trying to get confirmation about the use case.

No problem. I appreciate that you did this so that we can make sure resctrl supports needed use cases.

Quoted 7 lines · Reinette Chatre

Quoted 5 lines · Babu Moger

Quoted 4 lines · Reinette Chatre

Quoted 3 lines · Babu Moger

Quoted 2 lines · Reinette Chatre

Quoted 1 line · Babu Moger

Quoted 5 lines

Quoted 3 lines

can have domains that span different CPUs. There thus seem to be a built in assumption of what a "domain" means for PQR_PLZA_ASSOC so it sounds to me as though, instead of saying that "PQR_PLZA_ASSOC needs to be the same in QoS domain" it may be more accurate to, for example, say that "PQR_PLZA_ASSOC has L3 scope"?

Yes.

Above is about L3 scope ...

Quoted 7 lines

Quoted 5 lines

This seems to be what this implementation does since it hardcodes PQR_PLZA_ASSOC scope to the L3 resource but that creates dependency to the L3 resource that would make PLZA unusable if, for example, the user boots with "rdt=!l3cat" while wanting to use PLZA to manage MBA allocations when in kernel?

Yes. that is correct. It should not be attached to one resource. We need to change it to global scope.

Can I interpret "global scope" as "all online CPUs"? Doing so will simplify

supporting this feature. It does not sound practical for a user wanting to assign different resource groups to kernel work done in different domains ... the guidance should instead be to just set the allocations of one resource group to what is needed in the different domains? There may be more flexibility when supporting per-domain RMIDs though but so far it sounds as though the focus is global. We can consider what needs to be done to support some type of "per-domain" assignment as exercise whether current interface could support it in the future.

...

Quoted 5 lines

Quoted 5 lines

Quoted 5 lines

There are multiple ways this feature can be applied. For simplicity, the discussion below focuses only on CLOSID.

      1. Global PLZA enablement

Quoted 7 lines

Quoted 7 lines

Quoted 1 line

PLZA can be configured as a global feature by setting |PQR_PLZA_ASSOC.closid = CLOSID| and |PQR_PLZA_ASSOC.plza_en = 1| on all threads in the system. A dedicated CLOSID is reserved for this purpose,

Also discussed during v1 is that there is no need to dedicate a CLOSID for this purpose. There could be an "unthrottled" CLOSID to which all high priority user space tasks as well as all kernel work of all tasks are assigned.
If user space chooses to dedicate a CLOSID for kernel work then that should supported and interface can allow that, but there is no need for resctrl to enforce this.

(above is comment about dedicated group - please see below)

Quoted 11 lines

Yes. I agree. The changes in context switch code is a concern.

You covered some of the cases I was thinking(xx_set_individual).

How about this idea?

I suggest splitting the PLZA into two distinct aspects:

1. How PLZA is applied within a resource group

2. How PLZA is monitored

I think I see where you are going here. While the "How PLZA is monitored" naming refers to "monitoring" I *think* what you are separating here is (a) how PLZA is configured (CLOSID and RMID settings) and (b) how that PLZA configuration is assigned to tasks/CPUs, not just within a resource group but across the system. Please see below.

Quoted 1 line

Introduce a new file, "info/kmode_type", to describe how kmode applies in the system.

ack. "in the system" as you have above, not "within a resource group" as mentioned before that.

Quoted 6 lines

# cat info/kmode_type
[global] <- Kernel mode applies to the entire system (all CPUs/tasks)
   cpus   <- Kernel mode applies only to the CPUs in the group
   tasks  <- Kernel mode applies only to the tasks in the group

Quoted 5 lines

The "global" option is the default right now and it is current common use-case.

The "info/kmode_type -> cpus" option introduces new files "kmode_cpus" and "kmode_cpus_list" for users to apply kmode to specific set of CPUs. This lets users change the CPU set for PLZA.

Where were you thinking about placing these files in the hierarchy?

Quoted 4 lines

The PLZA MSR is updated when user changes the association to the file. No context switch code changes are needed. This will be dedicated group. The current resctrl group files, "cpus, cpus_list

Why does this have to be a dedicated group? One of the conclusions from v1 discussion was that the "PLZA group" need *not* be a dedicated group. I repeated that in my earlier response that I left quoted above. You did not respond to these conclusions and statements in this regard while you keep coming back to this needing to be a dedicated group without providing a motivation to do so. Could you please elaborate why a dedicated group is required?

Quoted 3 lines

and tasks" will not be accessible in this mode. This option give

These files can continue to be accessible.

Quoted 2 lines

some flexibility for the user without the context switch overhead.

Dedicating a resource group to PLZA removes flexibility though, no?

Quoted 8 lines

The "info/kmode_type -> tasks" option introduces a new file, "kmode_tasks", for users to apply kmode to specific set of tasks. This requires context switch changes. This will be dedicated group. The current resctrl group files, "cpus, cpus_list and tasks" will not be accessible in this mode. We currently have no use case for this, so it will not be supported now.

Thank you for confirming. This is a relief.

Quoted 8 lines

Add a file, "info/kmode_monitor", to describe how kmode is monitored.

# cat info/kmode_monitor
[inherit_ctrl_and_mon] <- Kernel uses the same CLOSID/RMID as user. Default option for the "global"
assign_ctrl_inherit_mon <- One CLOSID for all kernel work; RMID inherited from user.
assign_ctrl_assign_mon <- One resource group (CLOSID+RMID) for all kernel work. Default option for "cpu" type.

My first thought is that the naming is confusing. resctrl has a very strong relationship between "RMID" and "monitoring" so naming a file "monitor" that deals with allocation/ctrl/CLOSID is potentially confusion.

Apart from that, while I think I understand where you are going by separating the mode into two files I am concerned about future complications needing to accommodate all different combinations of the (now) essentially two modes. My preference is thus to keep this simple by keeping the mode within one file.

Even so, when stepping back, it does not really look like we need to separate the "global" and "per CPU" modes. We could just have a single "per CPU" mode and the "global" is just its default of "all CPUs", no?

Consider, for example, the implementation just consisting of:

	# cat info/kernel_mode
	[inherit_ctrl_and_mon]
	global_assign_ctrl_inherit_mon_per_cpu
	global_assign_ctrl_assign_mon_per_cpu

Quoted 5 lines

Rename “kernel_mode_assignment” to “kmode_group” to assign the specific group to kmode. This file usage is same as before.

#cat info/kmode_groups (Renamed "kernel_mode_assignment") //

Please consider the intent of this file when thinking about names. The idea is that "info/kernel_mode" specifies the "mode" of how kernel work is handled and it determines the configuration files used in that mode as well as the syntax when interacting with those files. By renaming "kernel_mode_assignment" to "kmode_groups" it implicitly requires all future kernel mode enhancements to need some data related to "groups".

In summary, I think this can be simplified by introducing just two new files in info/ that enables the user to (a) select and (b) configure the "kernel mode". To start there can be just two modes, global_assign_ctrl_inherit_mon_per_cpu and global_assign_ctrl_assign_mon_per_cpu. global_assign_ctrl_inherit_mon_per_cpu mode requires a control group in kernel_mode_assignment while global_assign_ctrl_assign_mon_per_cpu requires a control and monitoring group.

The resource group in info/kernel_mode_assignment gets two additional files "kernel_mode_cpus" and "kernel_mode_cpus_list" that contains the CPUs enabled with the kernel mode configuration, by default it will be all online CPUs. The resource group can continue to be used to manage allocations of and monitor user space tasks. Specifically, the "cpus", "cpus_list", and "tasks" files remain.

A user wanting just "global" settings will get just that when writing the group to info/kernel_mode_assignment. A user wanting "per CPU" settings can follow the info/kernel_mode_assignment setting with changes to that resource group's kernel_mode_cpus/kernel_mode_cpus_list files. Any task running on a CPU that is *not* in kernel_mode_cpus/kernel_mode_cpus_list can be expected to inherit both CLOSID and RMID from user space for all kernel work.

Hi Reinette,

Quoted 22 lines · Reinette Chatre

Hi Babu,

Quoted 3 lines · Babu Moger

Hi Reinette,

Sorry for the late response. I was trying to get confirmation about the use case.

No problem. I appreciate that you did this so that we can make sure resctrl supports needed use cases.

Quoted 12 lines · Reinette Chatre

Quoted 9 lines · Babu Moger

Quoted 4 lines · Reinette Chatre

Quoted 3 lines · Babu Moger

Quoted 2 lines · Reinette Chatre

Quoted 1 line · Babu Moger

can have domains that span different CPUs. There thus seem to be a built in assumption of what a "domain" means for PQR_PLZA_ASSOC so it sounds to me as though, instead of saying that "PQR_PLZA_ASSOC needs to be the same in QoS domain" it may be more accurate to, for example, say that "PQR_PLZA_ASSOC has L3 scope"?

Quoted 3 lines

Quoted 1 line

Yes.

Above is about L3 scope ...

Yes. The scope for PQR_PLZA_ASSOC is L3.

Is that what you are asking here?

Quoted 10 lines

Quoted 6 lines

Quoted 4 lines

This seems to be what this implementation does since it hardcodes PQR_PLZA_ASSOC scope to the L3 resource but that creates dependency to the L3 resource that would make PLZA unusable if, for example, the user boots with "rdt=!l3cat" while wanting to use PLZA to manage MBA allocations when in kernel?

Yes. that is correct. It should not be attached to one resource. We need to change it to global scope.

Can I interpret "global scope" as "all online CPUs"? Doing so will simplify

Yes. That is correct.

Quoted 7 lines

supporting this feature. It does not sound practical for a user wanting to assign different resource groups to kernel work done in different domains ... the guidance should instead be to just set the allocations of one resource group to what is needed in the different domains? There may be more flexibility when supporting per-domain RMIDs though but so far it sounds as though the focus is global. We can consider what needs to be done to support some type of "per-domain" assignment as exercise whether current interface could support it in the future.

Yes. Makes sense.

Quoted 1 line

Quoted 10 lines

Quoted 3 lines

The PLZA MSR is updated when user changes the association to the file. No context switch code changes are needed. This will be dedicated group. The current resctrl group files, "cpus, cpus_list

Why does this have to be a dedicated group? One of the conclusions from v1 discussion was that the "PLZA group" need *not* be a dedicated group. I repeated that in my earlier response that I left quoted above. You did not respond to these conclusions and statements in this regard while you keep coming back to this needing to be a dedicated group without providing a motivation to do so. Could you please elaborate why a dedicated group is required?

If the same group applies identical limits to both user and kernel space, it essentially behaves like a current resctrl group. In that sense, it’s not really a PLZA group. PLZA’s key value is the ability to separate allocations between user space and kernel space. A

single CPU can belong to two groups: one group manages the user- space allocation for that CPU, while another manages the kernel-mode allocation.

This approach also simplifies file handling, which is another reason I prefer it.

That said, I’m open to not having a dedicated group if we can still support all the features that PLZA provides without it.

Quoted 11 lines

Quoted 6 lines

Add a file, "info/kmode_monitor", to describe how kmode is monitored.

# cat info/kmode_monitor
[inherit_ctrl_and_mon] <- Kernel uses the same CLOSID/RMID as user. Default option for the "global"
assign_ctrl_inherit_mon <- One CLOSID for all kernel work; RMID inherited from user.
assign_ctrl_assign_mon <- One resource group (CLOSID+RMID) for all kernel work. Default option for "cpu" type.

My first thought is that the naming is confusing. resctrl has a very strong relationship between "RMID" and "monitoring" so naming a file "monitor" that deals with allocation/ctrl/CLOSID is potentially confusion.

Quoted 8 lines

Apart from that, while I think I understand where you are going by separating the mode into two files I am concerned about future complications needing to accommodate all different combinations of the (now) essentially two modes. My preference is thus to keep this simple by keeping the mode within one file.

Even so, when stepping back, it does not really look like we need to separate the "global" and "per CPU" modes. We could just have a single "per CPU" mode and the "global" is just its default of "all CPUs", no?

Yes. That correct.

Quoted 30 lines

Consider, for example, the implementation just consisting of:

    # cat info/kernel_mode
    [inherit_ctrl_and_mon]
    global_assign_ctrl_inherit_mon_per_cpu
    global_assign_ctrl_assign_mon_per_cpu
 

Quoted 5 lines

Rename “kernel_mode_assignment” to “kmode_group” to assign the specific group to kmode. This file usage is same as before.

#cat info/kmode_groups (Renamed "kernel_mode_assignment") //

Please consider the intent of this file when thinking about names. The idea is that "info/kernel_mode" specifies the "mode" of how kernel work is handled and it determines the configuration files used in that mode as well as the syntax when interacting with those files. By renaming "kernel_mode_assignment" to "kmode_groups" it implicitly requires all future kernel mode enhancements to need some data related to "groups".

In summary, I think this can be simplified by introducing just two new files in info/ that enables the user to (a) select and (b) configure the "kernel mode". To start there can be just two modes, global_assign_ctrl_inherit_mon_per_cpu and global_assign_ctrl_assign_mon_per_cpu. global_assign_ctrl_inherit_mon_per_cpu mode requires a control group in kernel_mode_assignment while global_assign_ctrl_assign_mon_per_cpu requires a control and monitoring group.

The resource group in info/kernel_mode_assignment gets two additional files "kernel_mode_cpus" and "kernel_mode_cpus_list" that contains the CPUs enabled with the kernel mode configuration, by default it will be all online CPUs. The resource group can continue to be used to manage allocations of and monitor user space tasks. Specifically, the "cpus", "cpus_list", and "tasks" files remain.

Quoted 5 lines

A user wanting just "global" settings will get just that when writing the group to info/kernel_mode_assignment. A user wanting "per CPU" settings can follow the info/kernel_mode_assignment setting with changes to that resource group's kernel_mode_cpus/kernel_mode_cpus_list files. Any task running on a CPU that is *not* in kernel_mode_cpus/kernel_mode_cpus_list can be expected to inherit both CLOSID and RMID from user space for all kernel work.

After further consideration, I don’t think the info/kernel_mode file is necessary. There’s no need to enforce a specific mode for all the PLZA groups. Avoiding this constraint makes the design more flexible, particularly as we move toward supporting multiple PLZA groups in the future. MPAM already appears capable of handling more than one group—for example, one group could use inherit_ctrl_and_mon, while another could use global_assign_ctrl_inherit_mon_per_cpu.

The mode can simply be determined on a per-group basis. We can introduce two new files—kernel_mode_cpus and kernel_mode_cpus_list—within each resctrl group when kmode (or PLZA) is supported.

The info/kernel_mode_assignment file would indicate which resctrl group(or groups) is used for PLZA. The files—kernel_mode_cpus and kernel_mode_cpus_list would indicate how the plza is applied which each group.

Files and behavior:
- cpus / cpus_list:

CPUs listed here use the same allocation for both user and kernel space.

There is no change to the current semantics of these files.
If these files are empty, the group effectively becomes a PLZA-dedicated group.

- kernel_mode_cpus / kernel_mode_cpus_list:

These files determine whether a separate kernel allocation is applied. If empty, user and kernel share the same allocation.
If non-empty, the kernel uses a separate allocation.

The group can be CTL_MON or MON group. Based on type the group the CLOSID and RMID will be used to enable PLZA. If it is MON, then rmid_en = 1 when writing PLZA MSR.

Here’s the proposed flow:

# mount -t resctrl resctrl /sys/fs/resctrl/
# cd /sys/fs/resctrl/
# cat info/kernel_mode_assignment
//

By default, the root (default) group is PLZA-enabled when resctrl is mounted. All CPUs use CLOSID 0 for both user and kernel-mode allocation.

# cat cpus_list
1-64
# cat kmode_cpus_list
1-64

Next, create a new group for PLZA:

# mkdir plza_group

# echo "plza_group//" > info/kernel_mode_assignment

At this point, plza_group becomes the new PLZA-enabled group, and the PLZA-related MSRs are updated accordingly.

# cat plza_group/cpus_list
<empty>

# cat plza_group/kmode_cpus_list
1-64

The user can then update kmode_cpus_list to apply PLZA only to a specific subset of CPUs, if desired.

What do you think of this approach?

Hi Babu,

Quoted 29 lines · Babu Moger

Hi Reinette,

Quoted 25 lines · Reinette Chatre

Hi Babu,

Quoted 3 lines · Babu Moger

Hi Reinette,

Sorry for the late response. I was trying to get confirmation about the use case.

No problem. I appreciate that you did this so that we can make sure resctrl supports needed use cases.

Quoted 13 lines · Reinette Chatre

Quoted 9 lines · Babu Moger

Quoted 4 lines · Reinette Chatre

Quoted 3 lines · Babu Moger

Quoted 2 lines · Reinette Chatre

Quoted 1 line · Babu Moger

can have domains that span different CPUs. There thus seem to be a built in assumption of what a "domain" means for PQR_PLZA_ASSOC so it sounds to me as though, instead of saying that "PQR_PLZA_ASSOC needs to be the same in QoS domain" it may be more accurate to, for example, say that "PQR_PLZA_ASSOC has L3 scope"?

Yes.

Above is about L3 scope ...

Quoted 3 lines

Yes. The scope for PQR_PLZA_ASSOC is L3.

Is that what you are asking here?

I was trying to point out that there appears to be a mismatch between the actual scope and the planned implementation. As highlighted below during the discussion about "global" this is fine with me and I just wanted to confirm that this matches your intentions.

Quoted 26 lines

Quoted 11 lines

Quoted 9 lines

Quoted 7 lines

This seems to be what this implementation does since it hardcodes PQR_PLZA_ASSOC scope to the L3 resource but that creates dependency to the L3 resource that would make PLZA unusable if, for example, the user boots with "rdt=!l3cat" while wanting to use PLZA to manage MBA allocations when in kernel?

Yes. that is correct. It should not be attached to one resource. We need to change it to global scope.

Can I interpret "global scope" as "all online CPUs"? Doing so will simplify

Yes. That is correct.

Quoted 7 lines

supporting this feature. It does not sound practical for a user wanting to assign different resource groups to kernel work done in different domains ... the guidance should instead be to just set the allocations of one resource group to what is needed in the different domains? There may be more flexibility when supporting per-domain RMIDs though but so far it sounds as though the focus is global. We can consider what needs to be done to support some type of "per-domain" assignment as exercise whether current interface could support it in the future.

Yes. Makes sense.

Quoted 1 line

...

Quoted 11 lines

Quoted 10 lines

Quoted 3 lines

The PLZA MSR is updated when user changes the association to the file. No context switch code changes are needed. This will be dedicated group. The current resctrl group files, "cpus, cpus_list

Why does this have to be a dedicated group? One of the conclusions from v1 discussion was that the "PLZA group" need *not* be a dedicated group. I repeated that in my earlier response that I left quoted above. You did not respond to these conclusions and statements in this regard while you keep coming back to this needing to be a dedicated group without providing a motivation to do so. Could you please elaborate why a dedicated group is required?

Quoted 4 lines

If the same group applies identical limits to both user and kernel space, it essentially behaves like a current resctrl group. In that sense, it’s not really a PLZA group. PLZA’s key value is the ability to separate allocations between user space and kernel space. A

The plan has never been to force identical allocations for user and kernel space since that would go against this feature entirely. Even so, just as user and kernel space cannot be forced to have identical allocations they also cannot be forced to have different allocations. Specifically,
a task *can* use the same CLOSID for user and kernel space work just as easily as it can use *different* CLOSID for user and kernel space work. There should not be any CLOSID reserved just for kernel work. Or am I missing something?

Quoted 4 lines

single CPU can belong to two groups: one group manages the user- space allocation for that CPU, while another manages the kernel-mode allocation.

Exactly. This is why it is important to have two files for this CPU association within a resource group. The cpus/cpus_list file continues to be used as today while the new kernel_mode_cpus/kernel_mode_cpus_list is used for kernel work. With this a task can be associated with any resource group for its user space allocations but when it runs on one of the CPUs within kernel_mode_cpus then its kernel work will be done with allocations of the resource group the kernel_mode_cpus file belongs to, which may or may not be the same resource group that the user space task belongs to.

Quoted 3 lines

This approach also simplifies file handling, which is another reason I prefer it.

I *think* we have different interpretations of "dedicated group":
It sounds as though you interpret "dedicated group" as a way that enforces the same allocations to user space and kernel work.
I interpret "dedicated group" essentially as a CLOSID reserved for kernel work. Since I do not see that resctrl should dedicate a CLOSID/resource group for kernel work I have been pushing against such "dedicated group".

Quoted 2 lines

That said, I’m open to not having a dedicated group if we can still support all the features that PLZA provides without it.

I find that enabling user space to share CLOSID/RMID between user space and kernel space to indeed support what PLZA provides. I think I am missing something here since below proposal again attempts to isolate a resource group (CLOSID) for kernel work.

Quoted 59 lines

Quoted 12 lines

Quoted 7 lines

Add a file, "info/kmode_monitor", to describe how kmode is monitored.

# cat info/kmode_monitor
[inherit_ctrl_and_mon] <- Kernel uses the same CLOSID/RMID as user. Default option for the "global"
assign_ctrl_inherit_mon <- One CLOSID for all kernel work; RMID inherited from user.
assign_ctrl_assign_mon <- One resource group (CLOSID+RMID) for all kernel work. Default option for "cpu" type.

My first thought is that the naming is confusing. resctrl has a very strong relationship between "RMID" and "monitoring" so naming a file "monitor" that deals with allocation/ctrl/CLOSID is potentially confusion.

Quoted 8 lines

Apart from that, while I think I understand where you are going by separating the mode into two files I am concerned about future complications needing to accommodate all different combinations of the (now) essentially two modes. My preference is thus to keep this simple by keeping the mode within one file.

Even so, when stepping back, it does not really look like we need to separate the "global" and "per CPU" modes. We could just have a single "per CPU" mode and the "global" is just its default of "all CPUs", no?

Yes. That correct.

Quoted 30 lines

Consider, for example, the implementation just consisting of:

     # cat info/kernel_mode
     [inherit_ctrl_and_mon]
     global_assign_ctrl_inherit_mon_per_cpu
     global_assign_ctrl_assign_mon_per_cpu

Quoted 5 lines

Rename “kernel_mode_assignment” to “kmode_group” to assign the specific group to kmode. This file usage is same as before.

#cat info/kmode_groups (Renamed "kernel_mode_assignment") //

Please consider the intent of this file when thinking about names. The idea is that "info/kernel_mode" specifies the "mode" of how kernel work is handled and it determines the configuration files used in that mode as well as the syntax when interacting with those files. By renaming "kernel_mode_assignment" to "kmode_groups" it implicitly requires all future kernel mode enhancements to need some data related to "groups".

In summary, I think this can be simplified by introducing just two new files in info/ that enables the user to (a) select and (b) configure the "kernel mode". To start there can be just two modes, global_assign_ctrl_inherit_mon_per_cpu and global_assign_ctrl_assign_mon_per_cpu. global_assign_ctrl_inherit_mon_per_cpu mode requires a control group in kernel_mode_assignment while global_assign_ctrl_assign_mon_per_cpu requires a control and monitoring group.

The resource group in info/kernel_mode_assignment gets two additional files "kernel_mode_cpus" and "kernel_mode_cpus_list" that contains the CPUs enabled with the kernel mode configuration, by default it will be all online CPUs. The resource group can continue to be used to manage allocations of and monitor user space tasks. Specifically, the "cpus", "cpus_list", and "tasks" files remain.

Quoted 5 lines

A user wanting just "global" settings will get just that when writing the group to info/kernel_mode_assignment. A user wanting "per CPU" settings can follow the info/kernel_mode_assignment setting with changes to that resource group's kernel_mode_cpus/kernel_mode_cpus_list files. Any task running on a CPU that is *not* in kernel_mode_cpus/kernel_mode_cpus_list can be expected to inherit both CLOSID and RMID from user space for all kernel work.

Quoted 8 lines

After further consideration, I don’t think the info/kernel_mode file is necessary. There’s no need to enforce a specific mode for all the PLZA groups. Avoiding this constraint makes the design more flexible, particularly as we move toward supporting multiple PLZA groups in the future. MPAM already appears capable of handling more than one group—for example, one group could use inherit_ctrl_and_mon, while another could use global_assign_ctrl_inherit_mon_per_cpu.

You are looking ahead at future capabilities for which we do not know all requirements at this time. I think it is very good to consider how things may progress and your example of MPAM is of course on point. I believe the current design does consider this progression. Please see https://lore.kernel.org/lkml/[email protected]/
(search for "per_group_assign_ctrl_assign_mon"). In that exploration per-group assignment is actually accomplished with global files. I thus think we should not make such a big architectural decision that does not benefit the immediate feature using partial information. As it is, a "info/kernel_mode" gives the flexibility to expand to, if needed, configuration files within a resource group. That is why the intention is to associate the mode within info/kernel_mode with the presence/absence of info/kernel_mode_assignment (search for "Visibility depends on active mode in info/kernel_mode" in linked email) since in the future resctrl may need to enable a mode that needs configuration files within each resource group and when enabling such mode the per-resource group files will appear instead of the global info/kernel_mode_assignment.

Quoted 1 line

Quoted 1 line

The mode can simply be determined on a per-group basis. We can introduce two new files—kernel_mode_cpus and kernel_mode_cpus_list—within each resctrl group when kmode (or PLZA) is supported.

I think having these files in every resource group is confusing since user can only interact with these files in one resource group for current PLZA. Why not *just* have the files in the resource group that matches the group in info/kernel_mode_assignment?

Quoted 5 lines

The info/kernel_mode_assignment file would indicate which resctrl group(or groups) is used for PLZA. The files—kernel_mode_cpus and kernel_mode_cpus_list would indicate how the plza is applied which each group.

The "how PLZA is applied" should be learned from info/kernel_mode where user space learns whether RMID is inherited or not. While I find kernel_mode_cpus and kernel_mode_cpus_list to be just for configuration and just found in the resource group listed in info/kernel_mode_assignment.

Quoted 6 lines

Files and behavior:
- cpus / cpus_list:

CPUs listed here use the same allocation for both user and kernel space.

Both user and kernel space?

Monitoring would depend on info/kernel_mode_assignment ("inherit_mon") and kernel space allocation would depend on whether the CPU on which the task runs can be found in kernel_mode_cpus, no?

Quoted 4 lines

There is no change to the current semantics of these files.
If these files are empty, the group effectively becomes a PLZA-dedicated group.

I do not see it this way. If the cpu/cpus_list files are empty then it means that the tasks in the group will use their own CLOSID/RMID for user space allocation and monitoring. What allocations/monitoring is used by tasks when in kernel mode depends on whether the CPU the task is running on can be found in a kernel_mode_cpus/kernel_mode_cpuslist file. If the CPU the task is running on can be found in a kernel_mode_cpus/kernel_mode_cpuslist file then it will inherit whatever the PQR_PLZA setting of that CPU which is the allocation associated with the resource group to which that kernel_mode_cpus/kernel_mode_cpuslist belongs. If the CPU the task is running on cannot be found in kernel_mode_cpus/kernel_mode_cpuslist then its kernel work will inherit its user space allocations and monitoring.

Quoted 8 lines

- kernel_mode_cpus / kernel_mode_cpus_list:

These files determine whether a separate kernel allocation is applied. If empty, user and kernel share the same allocation.
If non-empty, the kernel uses a separate allocation.

The group can be CTL_MON or MON group. Based on type the group the CLOSID and RMID will be used to enable PLZA. If it is MON, then rmid_en = 1 when writing PLZA MSR.

This will be difficult to get right since CTRL_MON groups also have RMID assigned.

Quoted 14 lines

Here’s the proposed flow:

# mount -t resctrl resctrl /sys/fs/resctrl/
# cd /sys/fs/resctrl/
# cat info/kernel_mode_assignment
//

By default, the root (default) group is PLZA-enabled when resctrl is mounted. All CPUs use CLOSID 0 for both user and kernel-mode allocation.

# cat cpus_list
1-64
# cat kmode_cpus_list
1-64

Quoted 7 lines

Next, create a new group for PLZA:

# mkdir plza_group

# echo "plza_group//" > info/kernel_mode_assignment

At this point, plza_group becomes the new PLZA-enabled group, and the PLZA-related MSRs are updated accordingly.

It really looks like you are getting back to trying to dedicate a resource group to kernel work and that is not something that resctrl should enforce.

Quoted 10 lines

# cat plza_group/cpus_list
<empty>

# cat plza_group/kmode_cpus_list
1-64

The user can then update kmode_cpus_list to apply PLZA only to a specific subset of CPUs, if desired.

Quoted 1 line

What do you think of this approach?

It is difficult to predict how the "next" PLZA will actually end up looking like and I find resctrl creating a complicated interface to support this to be risky. Instead I would prefer to focus on efficiently supporting what PLZA can do today and make it extensible. Apart from that I find the implicit interface, "If it is MON, then rmid_en = 1" to be too architecture specific for a generic interface while also not able to accurately capture user's intent (i.e. user may indeed, for example, want "a CTRL_MON group to have rmid_en = 1"). Finally, I am just so confused about why the implementations keep needing to dedicate a resource group/CLOSID to kernel work.

Quoted 4 lines · Reinette Chatre

Quoted 1 line · Babu Moger

That said, I’m open to not having a dedicated group if we can still support all the features that PLZA provides without it.

Quoted 4 lines

I find that enabling user space to share CLOSID/RMID between user space and kernel space to indeed support what PLZA provides. I think I am missing something here since below proposal again attempts to isolate a resource group (CLOSID) for kernel work.

No. I dont want to isolate a group just for PLZA. All I am saying is, we should provide option to create a dedicated group if the user wants to do it.

Quoted 9 lines

Quoted 5 lines

The mode can simply be determined on a per-group basis. We can introduce two new files—kernel_mode_cpus and kernel_mode_cpus_list—within each resctrl group when kmode (or PLZA) is supported.

I think having these files in every resource group is confusing since user can only interact with these files in one resource group for current PLZA. Why not *just* have the files in the resource group that matches the group in info/kernel_mode_assignment?

The default group can also serve as the PLZA group.

#cat info/kernel_mode_assignment
//

At this point, the (kmode_cpus / kmode_cpus_list) files will exist in the default group:

Then user changes the PLZA group to "test".

#echo "test//" > info/kernel_mode_assignment

At this point, we expect the files "(kmode_cpus/kmode_cpus_list)" to be visible in "test//" group.

One open question is whether we should remove the visibility of these files from the default group. It’s unclear if we can safely do this dynamically.

An alternative approach would be to always keep the files present, but allow access to them only for groups that are listed in "info/kernel_mode_assignment".

Quoted 6 lines

Quoted 4 lines

Files and behavior:
- cpus / cpus_list:

CPUs listed here use the same allocation for both user and kernel space.

Both user and kernel space?

As it stands today, the CPU list is written to MSR_PQR_ASSOC, resulting in the same allocation for both user and kernel within a given CLOS.

Kernel-mode allocation changes only if specific CPUs are included in the kmode_cpus list.

Quoted 13 lines

Quoted 2 lines

There is no change to the current semantics of these files.
If these files are empty, the group effectively becomes a PLZA-dedicated group.

I do not see it this way. If the cpu/cpus_list files are empty then it means that the tasks in the group will use their own CLOSID/RMID for user space allocation and monitoring. What allocations/monitoring is used by tasks when in kernel mode depends on whether the CPU the task is running on can be found in a kernel_mode_cpus/kernel_mode_cpuslist file. If the CPU the task is running on can be found in a kernel_mode_cpus/kernel_mode_cpuslist file then it will inherit whatever the PQR_PLZA setting of that CPU which is the allocation associated with the resource group to which that kernel_mode_cpus/kernel_mode_cpuslist belongs. If the CPU the task is running on cannot be found in kernel_mode_cpus/kernel_mode_cpuslist then its kernel work will inherit its user space allocations and monitoring.

Yes. that is correct. I think our understanding is correct, but our implementation ideas are different it seems.

Let me make sure I understand what you mentioned earlier. Copied the text below from the thread for the context:

https://lore.kernel.org/lkml/[email protected]/
=====================================================================

Please consider the intent of this file when thinking about names. The idea is that "info/kernel_mode" specifies the "mode" of how kernel work is handled and it determines the configuration files used in that mode as well as the syntax when interacting with those files. By renaming "kernel_mode_assignment" to "kmode_groups" it implicitly requires all future kernel mode enhancements to need some data related to "groups".

In summary, I think this can be simplified by introducing just two new files in info/ that enables the user to (a) select and (b) configure the "kernel mode". To start there can be just two modes, global_assign_ctrl_inherit_mon_per_cpu and global_assign_ctrl_assign_mon_per_cpu. global_assign_ctrl_inherit_mon_per_cpu mode requires a control group in kernel_mode_assignment while global_assign_ctrl_assign_mon_per_cpu requires a control and monitoring group.

The resource group in info/kernel_mode_assignment gets two additional files "kernel_mode_cpus" and "kernel_mode_cpus_list" that contains the CPUs enabled with the kernel mode configuration, by default it will be all online CPUs. The resource group can continue to be used to manage allocations of and monitor user space tasks. Specifically, the "cpus", "cpus_list", and "tasks" files remain.

A user wanting just "global" settings will get just that when writing the group to info/kernel_mode_assignment. A user wanting "per CPU" settings can follow the info/kernel_mode_assignment setting with changes to that resource group's kernel_mode_cpus/kernel_mode_cpus_list files. Any task running on a CPU that is *not* in kernel_mode_cpus/kernel_mode_cpus_list can be expected to inherit both CLOSID and RMID from user space for all kernel work.

======================================================================

Let me try to get few clarification on things here.

# cat info/kernel_mode
  [inherit_ctrl_and_mon]
  global_assign_ctrl_inherit_mon_per_cpu
  global_assign_ctrl_assign_mon_per_cpu

My understanding of "inherit_ctrl_and_mon" is that the kernel inherits both the CLOS and the RMID from user space. Basically both user and kernel uses same CLOSID and RMID. This reflects the current behavior (without PLZA) correct? This would correspond to the

default group when resctrl is mounted.

The modes "global_assign_ctrl_inherit_mon_per_cpu" and "global_assign_ctrl_assign_mon_per_cpu" represent the actual PLZA modes.

Both of these modes introduce new files kernel_mode_cpus/ and kernel_mode_cpus_list in the resctrl group.

When the user echoes a group name into info/kernel_mode_assignment, PLZA is applied globally across all CPUs. This is default behavior.

If the user wants PLZA to apply only to a specific subset of CPUs, then the kernel_mode_cpus or kernel_mode_cpus_list files need to be updated accordingly.

global_assign_ctrl_inherit_mon_per_cpu : The group needs to be CTLR_MON group. This mode uses rmid_en=0 when writing PLZA MSR.

global_assign_ctrl_assign_mon_per_cpu: The group needs to be CTLR_MON/MON group. This mode uses rmid_en=1 when writing PLZA MSR.

Did I get it right?

Hi Babu,

Quoted 5 lines · Babu Moger

Quoted 4 lines · Reinette Chatre

Quoted 1 line · Babu Moger

That said, I’m open to not having a dedicated group if we can still support all the features that PLZA provides without it.

Quoted 8 lines

Quoted 4 lines

I find that enabling user space to share CLOSID/RMID between user space and kernel space to indeed support what PLZA provides. I think I am missing something here since below proposal again attempts to isolate a resource group (CLOSID) for kernel work.

No. I dont want to isolate a group just for PLZA. All I am saying is, we should provide option to create a dedicated group if the user wants to do it.

I agree. I do not see resctrl needing to do anything to accomplish this though. If the user wants a group dedicated to kernel mode/PLZA then all that is needed is for the user not to assign any tasks to this group, either via changes to the group's tasks file or via the group's cpus/cpus_list files.

Quoted 19 lines

Quoted 9 lines

Quoted 5 lines

The mode can simply be determined on a per-group basis. We can introduce two new files—kernel_mode_cpus and kernel_mode_cpus_list—within each resctrl group when kmode (or PLZA) is supported.

I think having these files in every resource group is confusing since user can only interact with these files in one resource group for current PLZA. Why not *just* have the files in the resource group that matches the group in info/kernel_mode_assignment?

The default group can also serve as the PLZA group.

#cat info/kernel_mode_assignment
//

At this point, the (kmode_cpus / kmode_cpus_list) files will exist in the default group:

Then user changes the PLZA group to "test".

Quoted 7 lines

#echo "test//" > info/kernel_mode_assignment

At this point, we expect the files "(kmode_cpus/kmode_cpus_list)" to be visible in "test//" group.

One open question is whether we should remove the visibility of these files from the default group. It’s unclear if we can safely do this dynamically.

An alternative approach would be to always keep the files present, but allow access to them only for groups that are listed in "info/kernel_mode_assignment".

The files appearing/disappearing is just how the user experiences the resctrl fs interface. Within resctrl the files could indeed always exist but resctrl can use the kernfs_show() API to show/hide them as needed. Similar to resctrl_bmec_files_show() that you created. Allowing/removing access becomes complicated because user space can always do a chmod to change permissions that resctrl would need to handle.

I do not know if there are sharp corners here when thinking about strange scenarios where user opens a file before resctrl changes visibility or permissions and then user space interacts with the file. This may be worthwhile to test to matter which mechanism is used.

Quoted 10 lines

Quoted 6 lines

Quoted 4 lines

Files and behavior:
- cpus / cpus_list:

CPUs listed here use the same allocation for both user and kernel space.

Both user and kernel space?

As it stands today, the CPU list is written to MSR_PQR_ASSOC, resulting in the same allocation for both user and kernel within a given CLOS.

Kernel-mode allocation changes only if specific CPUs are included in the kmode_cpus list.

ack.

Quoted 14 lines

Quoted 13 lines

Quoted 2 lines

There is no change to the current semantics of these files.
If these files are empty, the group effectively becomes a PLZA-dedicated group.

I do not see it this way. If the cpu/cpus_list files are empty then it means that the tasks in the group will use their own CLOSID/RMID for user space allocation and monitoring. What allocations/monitoring is used by tasks when in kernel mode depends on whether the CPU the task is running on can be found in a kernel_mode_cpus/kernel_mode_cpuslist file. If the CPU the task is running on can be found in a kernel_mode_cpus/kernel_mode_cpuslist file then it will inherit whatever the PQR_PLZA setting of that CPU which is the allocation associated with the resource group to which that kernel_mode_cpus/kernel_mode_cpuslist belongs. If the CPU the task is running on cannot be found in kernel_mode_cpus/kernel_mode_cpuslist then its kernel work will inherit its user space allocations and monitoring.

Quoted 1 line

Yes. that is correct. I think our understanding is correct, but our implementation ideas are different it seems.

While we have been sharing different ideas I have tried to be clear on *why* I made certain choices and attempted to provide specific feedback to your ideas. If you find your plan to be better then please respond to my feedback about it to help me understand why that may be the better solution. If you find your solution is better then could you please describe it with detail? At this time I do not have a clear understanding of what you propose.

...

Quoted 37 lines

Let me make sure I understand what you mentioned earlier. Copied the text below from the thread for the context:

https://lore.kernel.org/lkml/[email protected]/
=====================================================================

Please consider the intent of this file when thinking about names. The idea is that "info/kernel_mode" specifies the "mode" of how kernel work is handled and it determines the configuration files used in that mode as well as the syntax when interacting with those files. By renaming "kernel_mode_assignment" to "kmode_groups" it implicitly requires all future kernel mode enhancements to need some data related to "groups".

In summary, I think this can be simplified by introducing just two new files in info/ that enables the user to (a) select and (b) configure the "kernel mode". To start there can be just two modes, global_assign_ctrl_inherit_mon_per_cpu and global_assign_ctrl_assign_mon_per_cpu. global_assign_ctrl_inherit_mon_per_cpu mode requires a control group in kernel_mode_assignment while global_assign_ctrl_assign_mon_per_cpu requires a control and monitoring group.

The resource group in info/kernel_mode_assignment gets two additional files "kernel_mode_cpus" and "kernel_mode_cpus_list" that contains the CPUs enabled with the kernel mode configuration, by default it will be all online CPUs. The resource group can continue to be used to manage allocations of and monitor user space tasks. Specifically, the "cpus", "cpus_list", and "tasks" files remain.

A user wanting just "global" settings will get just that when writing the group to info/kernel_mode_assignment. A user wanting "per CPU" settings can follow the info/kernel_mode_assignment setting with changes to that resource group's kernel_mode_cpus/kernel_mode_cpus_list files. Any task running on a CPU that is *not* in kernel_mode_cpus/kernel_mode_cpus_list can be expected to inherit both CLOSID and RMID from user space for all kernel work.

======================================================================

Let me try to get few clarification on things here.

# cat info/kernel_mode
   [inherit_ctrl_and_mon]
   global_assign_ctrl_inherit_mon_per_cpu
   global_assign_ctrl_assign_mon_per_cpu

Quoted 4 lines

My understanding of "inherit_ctrl_and_mon" is that the kernel inherits both the CLOS and the RMID from user space. Basically both user and kernel uses same CLOSID and RMID. This reflects the current behavior (without PLZA) correct? This would correspond to the

Correct.

Quoted 3 lines

default group when resctrl is mounted.

Quoted 3 lines

The modes "global_assign_ctrl_inherit_mon_per_cpu" and "global_assign_ctrl_assign_mon_per_cpu" represent the actual PLZA modes.

Both of these modes introduce new files kernel_mode_cpus/ and kernel_mode_cpus_list in the resctrl group.

Right. To be specific when the user changes the mode to either "global_assign_ctrl_inherit_mon_per_cpu" or "global_assign_ctrl_assign_mon_per_cpu" the new files will be created in the default resource group with associated setting applied globally at that time.

Quoted 1 line · Reinette Chatre

On 4/8/26 1:45 PM, Babu Moger wrote:

Quoted 7 lines

Quoted 3 lines

The modes "global_assign_ctrl_inherit_mon_per_cpu" and "global_assign_ctrl_assign_mon_per_cpu" represent the actual PLZA modes.

Both of these modes introduce new files kernel_mode_cpus/ and kernel_mode_cpus_list in the resctrl group.

Right. To be specific when the user changes the mode to either "global_assign_ctrl_inherit_mon_per_cpu" or "global_assign_ctrl_assign_mon_per_cpu" the new files will be created in the default resource group with associated setting applied globally at that time.

If, at that point, "info/kernel_mode_assignment" points to // (the default group), is that correct?

And if "info/kernel_mode_assignment" points to a different group (for example, test//), then the kernel_mode_cpus/ and kernel_mode_cpus_list files will be created only under the test// group. Is that correct?

Hi Babu,

Quoted 2 lines · Babu

Quoted 1 line · Reinette Chatre

On 4/8/26 1:45 PM, Babu Moger wrote:

...

Quoted 9 lines

Quoted 7 lines

Quoted 3 lines

The modes "global_assign_ctrl_inherit_mon_per_cpu" and "global_assign_ctrl_assign_mon_per_cpu" represent the actual PLZA modes.

Both of these modes introduce new files kernel_mode_cpus/ and kernel_mode_cpus_list in the resctrl group.

Right. To be specific when the user changes the mode to either "global_assign_ctrl_inherit_mon_per_cpu" or "global_assign_ctrl_assign_mon_per_cpu" the new files will be created in the default resource group with associated setting applied globally at that time.

If, at that point, "info/kernel_mode_assignment" points to // (the default group), is that correct?

I see "info/kernel_mode_assignment" pointing to default group as the only option right after a mode switch away from "inherit_ctrl_and_mon".

To elaborate, the current idea is that the mode within info/kernel_mode determines which, if any, control files are presented to user space.
Assuming that the system boots up with:

	# cat info/kernel_mode
	[inherit_ctrl_and_mon]
	global_assign_ctrl_inherit_mon_per_cpu
	global_assign_ctrl_assign_mon_per_cpu

In above scenario "info/kernel_mode_assignment" does not exist (is not visible to user space).

When the user switches to either "global_assign_ctrl_inherit_mon_per_cpu" or 'global_assign_ctrl_assign_mon_per_cpu" then "info/kernel_mode_assignment" is created (or made visible to user space) and is expected to point to default group. User can change the group using "info/kernel_mode_assignment" at this point.

If the current scenario is below ...

	# cat info/kernel_mode
	[global_assign_ctrl_inherit_mon_per_cpu]
	inherit_ctrl_and_mon
	global_assign_ctrl_assign_mon_per_cpu

... then "info/kernel_mode_assignment" will exist but what it should contain if user switches mode at this point may be up for discussion.

option 1)
When user switches mode to "global_assign_ctrl_assign_mon_per_cpu" then the resource group in "info/kernel_mode_assignment" is reset to the default group and all CPUs PLZA state reset to match. The kernel_mode_cpus and kernel_mode_cpuslist files become visible in default resource group and they contain "all online CPUs".

option 2)
When user switches mode to "global_assign_ctrl_assign_mon_per_cpu" then the resource group in "info/kernel_mode_assignment" is kept and all CPUs PLZA state set to match it while also keeping the current values of that resource group's kernel_mode_cpus and kernel_mode_cpuslist files.

I am leaning towards "option 1" to keep it consistent with a switch from "inherit_ctrl_and_mon" and being deterministic about how a mode is started with

a clean slate. What are your thoughts? What would be use case where a user would want to switch between "global_assign_ctrl_inherit_mon_per_cpu" and "global_assign_ctrl_assign_mon_per_cpu" to just switch rmid_en on and off?

Quoted 6 lines

And if "info/kernel_mode_assignment" points to a different group (for example, test//), then the kernel_mode_cpus/ and kernel_mode_cpus_list files will be created only under the test// group. Is that correct?

I expect that if "info/kernel_mode_assignment" exists then the group listed within contains kernel_mode_cpus and kernel_mode_cpuslist. How the group ends up in "info/kernel_mode_assignment" could result from mode change or from write by user space.

Quoted 28 lines · Reinette Chatre

When the user switches to either "global_assign_ctrl_inherit_mon_per_cpu" or 'global_assign_ctrl_assign_mon_per_cpu" then "info/kernel_mode_assignment" is created (or made visible to user space) and is expected to point to default group. User can change the group using "info/kernel_mode_assignment" at this point.

If the current scenario is below ...

    # cat info/kernel_mode
    [global_assign_ctrl_inherit_mon_per_cpu]
    inherit_ctrl_and_mon
    global_assign_ctrl_assign_mon_per_cpu

... then "info/kernel_mode_assignment" will exist but what it should contain if user switches mode at this point may be up for discussion.

option 1)
When user switches mode to "global_assign_ctrl_assign_mon_per_cpu" then the resource group in "info/kernel_mode_assignment" is reset to the default group and all CPUs PLZA state reset to match. The kernel_mode_cpus and kernel_mode_cpuslist files become visible in default resource group and they contain "all online CPUs".

option 2)
When user switches mode to "global_assign_ctrl_assign_mon_per_cpu" then the resource group in "info/kernel_mode_assignment" is kept and all CPUs PLZA state set to match it while also keeping the current values of that resource group's kernel_mode_cpus and kernel_mode_cpuslist files.

Quoted 2 lines

I am leaning towards "option 1" to keep it consistent with a switch from "inherit_ctrl_and_mon" and being deterministic about how a mode is started with

Yes. The "option 1" seems appropriate.

Quoted 3 lines

a clean slate. What are your thoughts? What would be use case where a user would want to switch between "global_assign_ctrl_inherit_mon_per_cpu" and "global_assign_ctrl_assign_mon_per_cpu" to just switch rmid_en on and off?

This is a bit tricky.

Currently, our requirement is to have a CTRL_MON group for global_assign_ctrl_inherit_mon_per_cpu. In this scenario, we use the group’s CLOSID for PLZA configuration, and RMID is not used (rmid_en = 0) when setting up PLZA.

Our requirement is also to have a CTRL_MON/MON group for global_assign_ctrl_assign_mon_per_cpu. In this case as well, the group’s CLOSID and RMID (rmid_en = 1) both are used configure PLZA.

Actually, we should not allow these changes from global_assign_ctrl_inherit_mon_per_cpu to global_assign_ctrl_assign_mon_per_cpu or visa versa.

Hi Babu,

Quoted 42 lines · Babu

Quoted 28 lines · Reinette Chatre

When the user switches to either "global_assign_ctrl_inherit_mon_per_cpu" or 'global_assign_ctrl_assign_mon_per_cpu" then "info/kernel_mode_assignment" is created (or made visible to user space) and is expected to point to default group. User can change the group using "info/kernel_mode_assignment" at this point.

If the current scenario is below ...

     # cat info/kernel_mode
     [global_assign_ctrl_inherit_mon_per_cpu]
     inherit_ctrl_and_mon
     global_assign_ctrl_assign_mon_per_cpu

... then "info/kernel_mode_assignment" will exist but what it should contain if user switches mode at this point may be up for discussion.

option 1)
When user switches mode to "global_assign_ctrl_assign_mon_per_cpu" then the resource group in "info/kernel_mode_assignment" is reset to the default group and all CPUs PLZA state reset to match. The kernel_mode_cpus and kernel_mode_cpuslist files become visible in default resource group and they contain "all online CPUs".

option 2)
When user switches mode to "global_assign_ctrl_assign_mon_per_cpu" then the resource group in "info/kernel_mode_assignment" is kept and all CPUs PLZA state set to match it while also keeping the current values of that resource group's kernel_mode_cpus and kernel_mode_cpuslist files.

Quoted 2 lines

I am leaning towards "option 1" to keep it consistent with a switch from "inherit_ctrl_and_mon" and being deterministic about how a mode is started with

Yes. The "option 1" seems appropriate.

Quoted 3 lines

a clean slate. What are your thoughts? What would be use case where a user would want to switch between "global_assign_ctrl_inherit_mon_per_cpu" and "global_assign_ctrl_assign_mon_per_cpu" to just switch rmid_en on and off?

This is a bit tricky.

Quoted 8 lines

Currently, our requirement is to have a CTRL_MON group for global_assign_ctrl_inherit_mon_per_cpu. In this scenario, we use the group’s CLOSID for PLZA configuration, and RMID is not used (rmid_en = 0) when setting up PLZA.

Our requirement is also to have a CTRL_MON/MON group for global_assign_ctrl_assign_mon_per_cpu. In this case as well, the group’s CLOSID and RMID (rmid_en = 1) both are used configure PLZA.

ah, right. Good catch.

Quoted 1 line

Quoted 3 lines

Actually, we should not allow these changes from global_assign_ctrl_inherit_mon_per_cpu to global_assign_ctrl_assign_mon_per_cpu or visa versa.

resctrl could allow it but as part of the switch it resets the "kernel mode group" to be the default group every time? This would be the "option 1" above.

Reinette

Quoted 55 lines · Reinette Chatre

Quoted 42 lines · Babu

Quoted 28 lines · Reinette Chatre

When the user switches to either "global_assign_ctrl_inherit_mon_per_cpu" or 'global_assign_ctrl_assign_mon_per_cpu" then "info/kernel_mode_assignment" is created (or made visible to user space) and is expected to point to default group. User can change the group using "info/kernel_mode_assignment" at this point.

If the current scenario is below ...

     # cat info/kernel_mode
     [global_assign_ctrl_inherit_mon_per_cpu]
     inherit_ctrl_and_mon
     global_assign_ctrl_assign_mon_per_cpu

... then "info/kernel_mode_assignment" will exist but what it should contain if user switches mode at this point may be up for discussion.

option 1)
When user switches mode to "global_assign_ctrl_assign_mon_per_cpu" then the resource group in "info/kernel_mode_assignment" is reset to the default group and all CPUs PLZA state reset to match. The kernel_mode_cpus and kernel_mode_cpuslist files become visible in default resource group and they contain "all online CPUs".

option 2)
When user switches mode to "global_assign_ctrl_assign_mon_per_cpu" then the resource group in "info/kernel_mode_assignment" is kept and all CPUs PLZA state set to match it while also keeping the current values of that resource group's kernel_mode_cpus and kernel_mode_cpuslist files.

Quoted 2 lines

I am leaning towards "option 1" to keep it consistent with a switch from "inherit_ctrl_and_mon" and being deterministic about how a mode is started with

Yes. The "option 1" seems appropriate.

Quoted 3 lines

a clean slate. What are your thoughts? What would be use case where a user would want to switch between "global_assign_ctrl_inherit_mon_per_cpu" and "global_assign_ctrl_assign_mon_per_cpu" to just switch rmid_en on and off?

This is a bit tricky.

Quoted 8 lines

Currently, our requirement is to have a CTRL_MON group for global_assign_ctrl_inherit_mon_per_cpu. In this scenario, we use the group’s CLOSID for PLZA configuration, and RMID is not used (rmid_en = 0) when setting up PLZA.

Our requirement is also to have a CTRL_MON/MON group for global_assign_ctrl_assign_mon_per_cpu. In this case as well, the group’s CLOSID and RMID (rmid_en = 1)  both are used configure PLZA.

ah, right. Good catch.

Quoted 1 line

Quoted 6 lines

Quoted 3 lines

Actually, we should not allow these changes from global_assign_ctrl_inherit_mon_per_cpu  to global_assign_ctrl_assign_mon_per_cpu or visa versa.

resctrl could allow it but as part of the switch it resets the "kernel mode group" to be the default group every time? This would be the "option 1" above.

Other options.

Allow global_assign_ctrl_inherit_mon_per_cpu -> global_assign_ctrl_assign_mon_per_cpu. As part of the switch, reset the "kernel mode group" to the default group.

Allow global_assign_ctrl_assign_mon_per_cpu -> global_assign_ctrl_inherit_mon_per_cpu. In this case switch to CTRL_MON/MON -> CTRL_MON.

Hi Babu,

Quoted 67 lines · Babu

Quoted 55 lines · Reinette Chatre

Quoted 42 lines · Babu

Quoted 28 lines · Reinette Chatre

When the user switches to either "global_assign_ctrl_inherit_mon_per_cpu" or 'global_assign_ctrl_assign_mon_per_cpu" then "info/kernel_mode_assignment" is created (or made visible to user space) and is expected to point to default group. User can change the group using "info/kernel_mode_assignment" at this point.

If the current scenario is below ...

      # cat info/kernel_mode
      [global_assign_ctrl_inherit_mon_per_cpu]
      inherit_ctrl_and_mon
      global_assign_ctrl_assign_mon_per_cpu

... then "info/kernel_mode_assignment" will exist but what it should contain if user switches mode at this point may be up for discussion.

option 1)
When user switches mode to "global_assign_ctrl_assign_mon_per_cpu" then the resource group in "info/kernel_mode_assignment" is reset to the default group and all CPUs PLZA state reset to match. The kernel_mode_cpus and kernel_mode_cpuslist files become visible in default resource group and they contain "all online CPUs".

option 2)
When user switches mode to "global_assign_ctrl_assign_mon_per_cpu" then the resource group in "info/kernel_mode_assignment" is kept and all CPUs PLZA state set to match it while also keeping the current values of that resource group's kernel_mode_cpus and kernel_mode_cpuslist files.

Quoted 2 lines

I am leaning towards "option 1" to keep it consistent with a switch from "inherit_ctrl_and_mon" and being deterministic about how a mode is started with

Yes. The "option 1" seems appropriate.

Quoted 3 lines

a clean slate. What are your thoughts? What would be use case where a user would want to switch between "global_assign_ctrl_inherit_mon_per_cpu" and "global_assign_ctrl_assign_mon_per_cpu" to just switch rmid_en on and off?

This is a bit tricky.

Quoted 8 lines

Currently, our requirement is to have a CTRL_MON group for global_assign_ctrl_inherit_mon_per_cpu. In this scenario, we use the group’s CLOSID for PLZA configuration, and RMID is not used (rmid_en = 0) when setting up PLZA.

Our requirement is also to have a CTRL_MON/MON group for global_assign_ctrl_assign_mon_per_cpu. In this case as well, the group’s CLOSID and RMID (rmid_en = 1)  both are used configure PLZA.

ah, right. Good catch.

Quoted 1 line

Quoted 6 lines

Quoted 3 lines

Actually, we should not allow these changes from global_assign_ctrl_inherit_mon_per_cpu  to global_assign_ctrl_assign_mon_per_cpu or visa versa.

resctrl could allow it but as part of the switch it resets the "kernel mode group" to be the default group every time? This would be the "option 1" above.

Other options.

Allow global_assign_ctrl_inherit_mon_per_cpu -> global_assign_ctrl_assign_mon_per_cpu. As part of the switch, reset the "kernel mode group" to the default group.

Quoted 3 lines

Allow global_assign_ctrl_assign_mon_per_cpu -> global_assign_ctrl_inherit_mon_per_cpu. In this case switch to CTRL_MON/MON -> CTRL_MON.

ok. Could you please return the courtesy of providing feedback on the suggestion you are responding to and also include the motivation why your suggestion is the better option?

Hi Reinette,

Quoted 70 lines · Reinette Chatre

Hi Babu,

Quoted 67 lines · Babu

Quoted 55 lines · Reinette Chatre

Quoted 42 lines · Babu

Quoted 28 lines · Reinette Chatre

When the user switches to either "global_assign_ctrl_inherit_mon_per_cpu" or 'global_assign_ctrl_assign_mon_per_cpu" then "info/kernel_mode_assignment" is created (or made visible to user space) and is expected to point to default group. User can change the group using "info/kernel_mode_assignment" at this point.

If the current scenario is below ...

      # cat info/kernel_mode
      [global_assign_ctrl_inherit_mon_per_cpu]
      inherit_ctrl_and_mon
      global_assign_ctrl_assign_mon_per_cpu

... then "info/kernel_mode_assignment" will exist but what it should contain if user switches mode at this point may be up for discussion.

option 1)
When user switches mode to "global_assign_ctrl_assign_mon_per_cpu" then the resource group in "info/kernel_mode_assignment" is reset to the default group and all CPUs PLZA state reset to match. The kernel_mode_cpus and kernel_mode_cpuslist files become visible in default resource group and they contain "all online CPUs".

option 2)
When user switches mode to "global_assign_ctrl_assign_mon_per_cpu" then the resource group in "info/kernel_mode_assignment" is kept and all CPUs PLZA state set to match it while also keeping the current values of that resource group's kernel_mode_cpus and kernel_mode_cpuslist files.

Quoted 2 lines

I am leaning towards "option 1" to keep it consistent with a switch from "inherit_ctrl_and_mon" and being deterministic about how a mode is started with

Yes. The "option 1" seems appropriate.

Quoted 3 lines

a clean slate. What are your thoughts? What would be use case where a user would want to switch between "global_assign_ctrl_inherit_mon_per_cpu" and "global_assign_ctrl_assign_mon_per_cpu" to just switch rmid_en on and off?

This is a bit tricky.

Quoted 8 lines

Currently, our requirement is to have a CTRL_MON group for global_assign_ctrl_inherit_mon_per_cpu. In this scenario, we use the group’s CLOSID for PLZA configuration, and RMID is not used (rmid_en = 0) when setting up PLZA.

Our requirement is also to have a CTRL_MON/MON group for global_assign_ctrl_assign_mon_per_cpu. In this case as well, the group’s CLOSID and RMID (rmid_en = 1)  both are used configure PLZA.

ah, right. Good catch.

Quoted 1 line

Quoted 6 lines

Quoted 3 lines

Actually, we should not allow these changes from global_assign_ctrl_inherit_mon_per_cpu  to global_assign_ctrl_assign_mon_per_cpu or visa versa.

resctrl could allow it but as part of the switch it resets the "kernel mode group" to be the default group every time? This would be the "option 1" above.

Other options.

Allow global_assign_ctrl_inherit_mon_per_cpu -> global_assign_ctrl_assign_mon_per_cpu. As part of the switch, reset the "kernel mode group" to the default group.

Quoted 7 lines

Quoted 3 lines

Allow global_assign_ctrl_assign_mon_per_cpu -> global_assign_ctrl_inherit_mon_per_cpu. In this case switch to CTRL_MON/MON -> CTRL_MON.

ok. Could you please return the courtesy of providing feedback on the suggestion you are responding to and also include the motivation why your suggestion is the better option?

Yea. Sure.

We need to allow the switch between the modes. Otherwise only way to reset is to remount the resctrl filesystem. That is not a good option.

Allow global_assign_ctrl_inherit_mon_per_cpu -> global_assign_ctrl_assign_mon_per_cpu. As part of the switch, reset the "kernel mode group" to the default group.

This option is same as you suggested.

Allow global_assign_ctrl_assign_mon_per_cpu -> global_assign_ctrl_inherit_mon_per_cpu. In this case switch
to CTRL_MON/MON -> CTRL_MON. This option basically disables monitor (rmid_en=0). It is less disruptive. Move is between child group to parent group.

Hi Babu,

Quoted 89 lines · Babu

Hi Reinette,

Quoted 70 lines · Reinette Chatre

Hi Babu,

Quoted 67 lines · Babu

Quoted 55 lines · Reinette Chatre

Quoted 42 lines · Babu

Quoted 28 lines · Reinette Chatre

When the user switches to either "global_assign_ctrl_inherit_mon_per_cpu" or 'global_assign_ctrl_assign_mon_per_cpu" then "info/kernel_mode_assignment" is created (or made visible to user space) and is expected to point to default group. User can change the group using "info/kernel_mode_assignment" at this point.

If the current scenario is below ...

       # cat info/kernel_mode
       [global_assign_ctrl_inherit_mon_per_cpu]
       inherit_ctrl_and_mon
       global_assign_ctrl_assign_mon_per_cpu

... then "info/kernel_mode_assignment" will exist but what it should contain if user switches mode at this point may be up for discussion.

option 1)
When user switches mode to "global_assign_ctrl_assign_mon_per_cpu" then the resource group in "info/kernel_mode_assignment" is reset to the default group and all CPUs PLZA state reset to match. The kernel_mode_cpus and kernel_mode_cpuslist files become visible in default resource group and they contain "all online CPUs".

option 2)
When user switches mode to "global_assign_ctrl_assign_mon_per_cpu" then the resource group in "info/kernel_mode_assignment" is kept and all CPUs PLZA state set to match it while also keeping the current values of that resource group's kernel_mode_cpus and kernel_mode_cpuslist files.

Quoted 2 lines

I am leaning towards "option 1" to keep it consistent with a switch from "inherit_ctrl_and_mon" and being deterministic about how a mode is started with

Yes. The "option 1" seems appropriate.

Quoted 3 lines

a clean slate. What are your thoughts? What would be use case where a user would want to switch between "global_assign_ctrl_inherit_mon_per_cpu" and "global_assign_ctrl_assign_mon_per_cpu" to just switch rmid_en on and off?

This is a bit tricky.

Quoted 8 lines

Currently, our requirement is to have a CTRL_MON group for global_assign_ctrl_inherit_mon_per_cpu. In this scenario, we use the group’s CLOSID for PLZA configuration, and RMID is not used (rmid_en = 0) when setting up PLZA.

Our requirement is also to have a CTRL_MON/MON group for global_assign_ctrl_assign_mon_per_cpu. In this case as well, the group’s CLOSID and RMID (rmid_en = 1)  both are used configure PLZA.

ah, right. Good catch.

Quoted 1 line

Quoted 6 lines

Quoted 3 lines

Actually, we should not allow these changes from global_assign_ctrl_inherit_mon_per_cpu  to global_assign_ctrl_assign_mon_per_cpu or visa versa.

resctrl could allow it but as part of the switch it resets the "kernel mode group" to be the default group every time? This would be the "option 1" above.

Other options.

Allow global_assign_ctrl_inherit_mon_per_cpu -> global_assign_ctrl_assign_mon_per_cpu. As part of the switch, reset the "kernel mode group" to the default group.

Quoted 7 lines

Quoted 3 lines

Allow global_assign_ctrl_assign_mon_per_cpu -> global_assign_ctrl_inherit_mon_per_cpu. In this case switch to CTRL_MON/MON -> CTRL_MON.

ok. Could you please return the courtesy of providing feedback on the suggestion you are responding to and also include the motivation why your suggestion is the better option?

Yea. Sure.

We need to allow the switch between the modes. Otherwise only way to reset is to remount the resctrl filesystem. That is not a good option.

Allow global_assign_ctrl_inherit_mon_per_cpu -> global_assign_ctrl_assign_mon_per_cpu. As part of the switch, reset the "kernel mode group" to the default group.

This option is same as you suggested.

Quoted 2 lines

Allow global_assign_ctrl_assign_mon_per_cpu -> global_assign_ctrl_inherit_mon_per_cpu. In this case switch
to CTRL_MON/MON -> CTRL_MON. This option basically disables monitor (rmid_en=0). It is less disruptive. Move is between child group to parent group.

ok. I am concerned that this creates an inconsistent interface. Specifically, sometimes when switching the mode the kernel group will reset and sometimes it won't. This inconsistency may be more apparent when writing the user documentation as part of this work. If you are able to clearly explain how this resctrl fs interface behaves (this cannot be about PLZA internals as above) then this could work.

Hi Babu,

Quoted 89 lines · Babu

Hi Reinette,

Quoted 70 lines · Reinette Chatre

Hi Babu,

Quoted 67 lines · Babu

Quoted 55 lines · Reinette Chatre

Quoted 42 lines · Babu

Quoted 28 lines · Reinette Chatre

When the user switches to either "global_assign_ctrl_inherit_mon_per_cpu" or 'global_assign_ctrl_assign_mon_per_cpu" then "info/kernel_mode_assignment" is created (or made visible to user space) and is expected to point to default group. User can change the group using "info/kernel_mode_assignment" at this point.

If the current scenario is below ...

       # cat info/kernel_mode
       [global_assign_ctrl_inherit_mon_per_cpu]
       inherit_ctrl_and_mon
       global_assign_ctrl_assign_mon_per_cpu

... then "info/kernel_mode_assignment" will exist but what it should contain if user switches mode at this point may be up for discussion.

option 1)
When user switches mode to "global_assign_ctrl_assign_mon_per_cpu" then the resource group in "info/kernel_mode_assignment" is reset to the default group and all CPUs PLZA state reset to match. The kernel_mode_cpus and kernel_mode_cpuslist files become visible in default resource group and they contain "all online CPUs".

option 2)
When user switches mode to "global_assign_ctrl_assign_mon_per_cpu" then the resource group in "info/kernel_mode_assignment" is kept and all CPUs PLZA state set to match it while also keeping the current values of that resource group's kernel_mode_cpus and kernel_mode_cpuslist files.

Quoted 2 lines

I am leaning towards "option 1" to keep it consistent with a switch from "inherit_ctrl_and_mon" and being deterministic about how a mode is started with

Yes. The "option 1" seems appropriate.

Quoted 3 lines

a clean slate. What are your thoughts? What would be use case where a user would want to switch between "global_assign_ctrl_inherit_mon_per_cpu" and "global_assign_ctrl_assign_mon_per_cpu" to just switch rmid_en on and off?

This is a bit tricky.

Quoted 8 lines

Currently, our requirement is to have a CTRL_MON group for global_assign_ctrl_inherit_mon_per_cpu. In this scenario, we use the group’s CLOSID for PLZA configuration, and RMID is not used (rmid_en = 0) when setting up PLZA.

Our requirement is also to have a CTRL_MON/MON group for global_assign_ctrl_assign_mon_per_cpu. In this case as well, the group’s CLOSID and RMID (rmid_en = 1)  both are used configure PLZA.

ah, right. Good catch.

Quoted 1 line

Quoted 6 lines

Quoted 3 lines

Actually, we should not allow these changes from global_assign_ctrl_inherit_mon_per_cpu  to global_assign_ctrl_assign_mon_per_cpu or visa versa.

resctrl could allow it but as part of the switch it resets the "kernel mode group" to be the default group every time? This would be the "option 1" above.

Other options.

Allow global_assign_ctrl_inherit_mon_per_cpu -> global_assign_ctrl_assign_mon_per_cpu. As part of the switch, reset the "kernel mode group" to the default group.

Quoted 7 lines

Quoted 3 lines

Allow global_assign_ctrl_assign_mon_per_cpu -> global_assign_ctrl_inherit_mon_per_cpu. In this case switch to CTRL_MON/MON -> CTRL_MON.

ok. Could you please return the courtesy of providing feedback on the suggestion you are responding to and also include the motivation why your suggestion is the better option?

Yea. Sure.

We need to allow the switch between the modes. Otherwise only way to reset is to remount the resctrl filesystem. That is not a good option.

Allow global_assign_ctrl_inherit_mon_per_cpu -> global_assign_ctrl_assign_mon_per_cpu. As part of the switch, reset the "kernel mode group" to the default group.

This option is same as you suggested.

Quoted 2 lines

Allow global_assign_ctrl_assign_mon_per_cpu -> global_assign_ctrl_inherit_mon_per_cpu. In this case switch
to CTRL_MON/MON -> CTRL_MON. This option basically disables monitor (rmid_en=0). It is less disruptive. Move is between child group to parent group.

ok. I am concerned that this creates an inconsistent interface. Specifically, sometimes when switching the mode the kernel group will reset and sometimes it won't. This inconsistency may be more apparent when writing the user documentation as part of this work. If you are able to clearly explain how this resctrl fs interface behaves (this cannot be about PLZA internals as above) then this could work.

Quoted 88 lines · Reinette Chatre

Quoted 87 lines · Babu

Quoted 70 lines · Reinette Chatre

Hi Babu,

Quoted 67 lines · Babu

Quoted 55 lines · Reinette Chatre

Quoted 42 lines · Babu

Quoted 28 lines · Reinette Chatre

When the user switches to either "global_assign_ctrl_inherit_mon_per_cpu" or 'global_assign_ctrl_assign_mon_per_cpu" then "info/kernel_mode_assignment" is created (or made visible to user space) and is expected to point to default group. User can change the group using "info/kernel_mode_assignment" at this point.

If the current scenario is below ...

       # cat info/kernel_mode
       [global_assign_ctrl_inherit_mon_per_cpu]
       inherit_ctrl_and_mon
       global_assign_ctrl_assign_mon_per_cpu

... then "info/kernel_mode_assignment" will exist but what it should contain if user switches mode at this point may be up for discussion.

option 1)
When user switches mode to "global_assign_ctrl_assign_mon_per_cpu" then the resource group in "info/kernel_mode_assignment" is reset to the default group and all CPUs PLZA state reset to match. The kernel_mode_cpus and kernel_mode_cpuslist files become visible in default resource group and they contain "all online CPUs".

option 2)
When user switches mode to "global_assign_ctrl_assign_mon_per_cpu" then the resource group in "info/kernel_mode_assignment" is kept and all CPUs PLZA state set to match it while also keeping the current values of that resource group's kernel_mode_cpus and kernel_mode_cpuslist files.

Quoted 2 lines

I am leaning towards "option 1" to keep it consistent with a switch from "inherit_ctrl_and_mon" and being deterministic about how a mode is started with

Yes. The "option 1" seems appropriate.

Quoted 3 lines

a clean slate. What are your thoughts? What would be use case where a user would want to switch between "global_assign_ctrl_inherit_mon_per_cpu" and "global_assign_ctrl_assign_mon_per_cpu" to just switch rmid_en on and off?

This is a bit tricky.

Quoted 8 lines

Currently, our requirement is to have a CTRL_MON group for global_assign_ctrl_inherit_mon_per_cpu. In this scenario, we use the group’s CLOSID for PLZA configuration, and RMID is not used (rmid_en = 0) when setting up PLZA.

Our requirement is also to have a CTRL_MON/MON group for global_assign_ctrl_assign_mon_per_cpu. In this case as well, the group’s CLOSID and RMID (rmid_en = 1)  both are used configure PLZA.

ah, right. Good catch.

Quoted 1 line

Quoted 6 lines

Quoted 3 lines

Actually, we should not allow these changes from global_assign_ctrl_inherit_mon_per_cpu  to global_assign_ctrl_assign_mon_per_cpu or visa versa.

resctrl could allow it but as part of the switch it resets the "kernel mode group" to be the default group every time? This would be the "option 1" above.

Other options.

Allow global_assign_ctrl_inherit_mon_per_cpu -> global_assign_ctrl_assign_mon_per_cpu. As part of the switch, reset the "kernel mode group" to the default group.

Quoted 7 lines

Quoted 3 lines

Allow global_assign_ctrl_assign_mon_per_cpu -> global_assign_ctrl_inherit_mon_per_cpu. In this case switch to CTRL_MON/MON -> CTRL_MON.

ok. Could you please return the courtesy of providing feedback on the suggestion you are responding to and also include the motivation why your suggestion is the better option?

Yea. Sure.

We need to allow the switch between the modes. Otherwise only way to reset is to remount the resctrl filesystem. That is not a good option.

Allow global_assign_ctrl_inherit_mon_per_cpu -> global_assign_ctrl_assign_mon_per_cpu. As part of the switch, reset the "kernel mode group" to the default group.

This option is same as you suggested.

Quoted 8 lines

Quoted 2 lines

Allow global_assign_ctrl_assign_mon_per_cpu -> global_assign_ctrl_inherit_mon_per_cpu. In this case switch
to CTRL_MON/MON -> CTRL_MON. This option basically disables monitor (rmid_en=0). It is less disruptive. Move is between child group to parent group.

ok. I am concerned that this creates an inconsistent interface. Specifically, sometimes when switching the mode the kernel group will reset and sometimes it won't. This inconsistency may be more apparent when writing the user documentation as part of this work. If you are able to clearly explain how this resctrl fs interface behaves (this cannot be about PLZA internals as above) then this could work.

Started working on these changes. May be it is better to discuss this before to avoid one more revision.

The current mode change behavior is very restrictive.

For example:

# cat info/kernel_mode
      inherit_ctrl_and_mon
      [global_assign_ctrl_assign_mon_per_cpu]
       global_assign_ctrl_inherit_mon_per_cpu

# cat info/kernel_mode_assignment
     ctrl1/mon1/

In this state, we cannot change kernel_mode to inherit_ctrl_and_mon. The expectation, however, is that inherit_ctrl_and_mon should always map to the RDTCTRL_GROUP.

A similar issue exists when switching between global_assign_ctrl_inherit_mon_per_cpu and global_assign_ctrl_assign_mon_per_cpu (in either direction).

The same problem also occurs when modifying the kernel_mode_assignment group. If the current group is an RDTMON_GROUP, we can't assign another RDTCTRL_GROUP without changing both mode and group together.

Hi Babu,

Quoted 113 lines · Babu Moger

Quoted 88 lines · Reinette Chatre

Quoted 87 lines · Babu

Quoted 70 lines · Reinette Chatre

Hi Babu,

Quoted 67 lines · Babu

Quoted 55 lines · Reinette Chatre

Quoted 42 lines · Babu

Quoted 28 lines · Reinette Chatre

When the user switches to either "global_assign_ctrl_inherit_mon_per_cpu" or 'global_assign_ctrl_assign_mon_per_cpu" then "info/kernel_mode_assignment" is created (or made visible to user space) and is expected to point to default group. User can change the group using "info/kernel_mode_assignment" at this point.

If the current scenario is below ...

        # cat info/kernel_mode
        [global_assign_ctrl_inherit_mon_per_cpu]
        inherit_ctrl_and_mon
        global_assign_ctrl_assign_mon_per_cpu

... then "info/kernel_mode_assignment" will exist but what it should contain if user switches mode at this point may be up for discussion.

option 1)
When user switches mode to "global_assign_ctrl_assign_mon_per_cpu" then the resource group in "info/kernel_mode_assignment" is reset to the default group and all CPUs PLZA state reset to match. The kernel_mode_cpus and kernel_mode_cpuslist files become visible in default resource group and they contain "all online CPUs".

option 2)
When user switches mode to "global_assign_ctrl_assign_mon_per_cpu" then the resource group in "info/kernel_mode_assignment" is kept and all CPUs PLZA state set to match it while also keeping the current values of that resource group's kernel_mode_cpus and kernel_mode_cpuslist files.

Quoted 2 lines

I am leaning towards "option 1" to keep it consistent with a switch from "inherit_ctrl_and_mon" and being deterministic about how a mode is started with

Yes. The "option 1" seems appropriate.

Quoted 3 lines

a clean slate. What are your thoughts? What would be use case where a user would want to switch between "global_assign_ctrl_inherit_mon_per_cpu" and "global_assign_ctrl_assign_mon_per_cpu" to just switch rmid_en on and off?

This is a bit tricky.

Quoted 8 lines

Currently, our requirement is to have a CTRL_MON group for global_assign_ctrl_inherit_mon_per_cpu. In this scenario, we use the group’s CLOSID for PLZA configuration, and RMID is not used (rmid_en = 0) when setting up PLZA.

Our requirement is also to have a CTRL_MON/MON group for global_assign_ctrl_assign_mon_per_cpu. In this case as well, the group’s CLOSID and RMID (rmid_en = 1)  both are used configure PLZA.

ah, right. Good catch.

Quoted 1 line

Quoted 6 lines

Quoted 3 lines

Actually, we should not allow these changes from global_assign_ctrl_inherit_mon_per_cpu  to global_assign_ctrl_assign_mon_per_cpu or visa versa.

resctrl could allow it but as part of the switch it resets the "kernel mode group" to be the default group every time? This would be the "option 1" above.

Other options.

Allow global_assign_ctrl_inherit_mon_per_cpu -> global_assign_ctrl_assign_mon_per_cpu. As part of the switch, reset the "kernel mode group" to the default group.

Quoted 7 lines

Quoted 3 lines

Allow global_assign_ctrl_assign_mon_per_cpu -> global_assign_ctrl_inherit_mon_per_cpu. In this case switch to CTRL_MON/MON -> CTRL_MON.

ok. Could you please return the courtesy of providing feedback on the suggestion you are responding to and also include the motivation why your suggestion is the better option?

Yea. Sure.

We need to allow the switch between the modes. Otherwise only way to reset is to remount the resctrl filesystem. That is not a good option.

Allow global_assign_ctrl_inherit_mon_per_cpu -> global_assign_ctrl_assign_mon_per_cpu. As part of the switch, reset the "kernel mode group" to the default group.

This option is same as you suggested.

Quoted 8 lines

Quoted 2 lines

Allow global_assign_ctrl_assign_mon_per_cpu -> global_assign_ctrl_inherit_mon_per_cpu. In this case switch
to CTRL_MON/MON -> CTRL_MON. This option basically disables monitor (rmid_en=0). It is less disruptive. Move is between child group to parent group.

ok. I am concerned that this creates an inconsistent interface. Specifically, sometimes when switching the mode the kernel group will reset and sometimes it won't. This inconsistency may be more apparent when writing the user documentation as part of this work. If you are able to clearly explain how this resctrl fs interface behaves (this cannot be about PLZA internals as above) then this could work.

Started working on these changes. May be it is better to discuss this before to avoid one more revision.

The current mode change behavior is very restrictive.

For example:

# cat info/kernel_mode
       inherit_ctrl_and_mon
       [global_assign_ctrl_assign_mon_per_cpu]
        global_assign_ctrl_inherit_mon_per_cpu

# cat info/kernel_mode_assignment
      ctrl1/mon1/

Quoted 1 line

In this state, we cannot change kernel_mode to inherit_ctrl_and_mon. The expectation, however, is that inherit_ctrl_and_mon should always map to the RDTCTRL_GROUP.

Could you please provide details behind the "we cannot change kernel_mode to inherit_ctrl_and_mon" statement? Why is this not possible?

I do not see "inherit_ctrl_and_mon" to map to *any* group though. Expectation is that when user changes mode to "inherit_ctrl_and_mon" then info/kernel_mode_assignment would become invisible to user space.

Quoted 6 lines

A similar issue exists when switching between global_assign_ctrl_inherit_mon_per_cpu and global_assign_ctrl_assign_mon_per_cpu (in either direction).

What similar issue? Could you please provide some detail to help me understand what the issue is? Isn't this what we just discussed in thread you are replying to? That is, you were looking at developing that interface that I viewed as "inconsistent"?

Quoted 3 lines

The same problem also occurs when modifying the kernel_mode_assignment group. If the current group is an RDTMON_GROUP, we can't assign another RDTCTRL_GROUP without changing both mode and group together.

Same problem? Still unclear what the problem is. So far three problems are mentioned but I am not able to decipher what the problems are. Could you please elaborate?
When modifying the kernel_mode_assignment group I expect that the interface will only accept a MON group when in "assign_mon" mode and a CTRL group when in "inherit_mon" mode.
I do not understand what you mean with *another* RDTCTRL_GROUP. Only one group can be assigned at any time, no?

Reinette

Quoted 15 lines · Reinette Chatre

Quoted 13 lines · Babu Moger

The current mode change behavior is very restrictive.

For example:

# cat info/kernel_mode
       inherit_ctrl_and_mon
       [global_assign_ctrl_assign_mon_per_cpu]
        global_assign_ctrl_inherit_mon_per_cpu

# cat info/kernel_mode_assignment
      ctrl1/mon1/

Quoted 8 lines

Quoted 1 line

In this state, we cannot change kernel_mode to inherit_ctrl_and_mon. The expectation, however, is that inherit_ctrl_and_mon should always map to the RDTCTRL_GROUP.

Could you please provide details behind the "we cannot change kernel_mode to inherit_ctrl_and_mon" statement? Why is this not possible?

I do not see "inherit_ctrl_and_mon" to map to *any* group though. Expectation is that when user changes mode to "inherit_ctrl_and_mon" then info/kernel_mode_assignment would become invisible to user space.

Ok. That is fine.

Sorry for not making it clear. Let’s consider the following scenario.

The system boots with these default settings:

# cat info/kernel_mode
[inherit_ctrl_and_mon]
global_assign_ctrl_assign_mon_per_cpu
global_assign_ctrl_inherit_mon_per_cpu

At this point, the interface info/kernel_mode_assignment is not visible.

Next, lets create a new control group:

# mkdir ctrl1

We want to designate this group as the new kernel-mode group.

First operation: Change the mode:

# echo "global_assign_ctrl_inherit_mon_per_cpu" > info/kernel_mode

At this stage, only the kernel mode is being changed. However, there is no way to know which control group the user intends to assign to kernel mode. All we know here is the selected mode.

After this operation, the info/kernel_mode_assignment interface should become visible. But the question is: what should it contain or point to at this moment?

# cat info/kernel_mode_assignment
??

Next operation: Assign the group

# echo "ctrl1//" > info/kernel_mode_assignment

Now the intended control group (ctrl1) is explicitly specified for kernel mode. In summary, changing the kernel mode requires two distinct inputs:

- Selecting the kernel mode.
- Specifying the control group to be used for that mode.

Hope this makes sense.

Hi Babu,

Quoted 46 lines · Babu

Quoted 15 lines · Reinette Chatre

Quoted 13 lines · Babu Moger

The current mode change behavior is very restrictive.

For example:

# cat info/kernel_mode
        inherit_ctrl_and_mon
        [global_assign_ctrl_assign_mon_per_cpu]
         global_assign_ctrl_inherit_mon_per_cpu

# cat info/kernel_mode_assignment
       ctrl1/mon1/

Quoted 8 lines

Quoted 1 line

In this state, we cannot change kernel_mode to inherit_ctrl_and_mon. The expectation, however, is that inherit_ctrl_and_mon should always map to the RDTCTRL_GROUP.

Could you please provide details behind the "we cannot change kernel_mode to inherit_ctrl_and_mon" statement? Why is this not possible?

I do not see "inherit_ctrl_and_mon" to map to *any* group though. Expectation is that when user changes mode to "inherit_ctrl_and_mon" then info/kernel_mode_assignment would become invisible to user space.

Ok. That is fine.

Sorry for not making it clear. Let’s consider the following scenario.

The system boots with these default settings:

# cat info/kernel_mode
[inherit_ctrl_and_mon]
global_assign_ctrl_assign_mon_per_cpu
global_assign_ctrl_inherit_mon_per_cpu

At this point, the interface info/kernel_mode_assignment is not visible.

Next, lets create a new control group:

# mkdir ctrl1

We want to designate this group as the new kernel-mode group.

Quoted 7 lines

First operation: Change the mode:

# echo "global_assign_ctrl_inherit_mon_per_cpu" > info/kernel_mode

At this stage, only the kernel mode is being changed. However, there is no way to know which control group the user intends to assign to kernel mode. All we know here is the selected mode.

After this operation, the info/kernel_mode_assignment interface should become visible. But the question is: what should it contain or point to at this moment?

This was considered as part of original proposal per
https://lore.kernel.org/lkml/[email protected]/
(search for "default value") where the idea was that the group should be initialized to the default group.

Quoted 3 lines

# cat info/kernel_mode_assignment
??

After

# echo "global_assign_ctrl_inherit_mon_per_cpu" > info/kernel_mode
# cat info/kernel_mode_assignment
/

After

# echo "global_assign_ctrl_assign_mon_per_cpu" > info/kernel_mode
# cat info/kernel_mode_assignment
//

(although this is where previous discussion comes in on how interface can become inconsistent depending on what the previous kernel mode was)

Quoted 6 lines

Next operation: Assign the group

# echo "ctrl1//" > info/kernel_mode_assignment

Quoted 9 lines

Now the intended control group (ctrl1) is explicitly specified for kernel mode. In summary, changing the kernel mode requires two distinct inputs:

- Selecting the kernel mode.
- Specifying the control group to be used for that mode.

Hope this makes sense.

Understood. Could you please elaborate what the problem is with making it so? Are you trying to eliminate one per-CPU register write? Is this something that ends up being very expensive? I assumed that a register designed to support modification during context switch should be fast. Or is it the IPI you are concerned about? Please help me to understand what the actual problem is that you are trying to solve.
I think it is reasonable to start with defaults when changing the mode which I do not expect users to change often.

We already discussed moving back to the default group on every mode switch. Doing so here would once again cause extra MSR writes on each mode transition, which is undesirable.