Skip to contentSkip to conversation

Resctrl - rewrite (WIP)

RFC PATCH In review Tony Luck 3 participants 16 messages Jun 20, 2023 – Aug 1, 2023

Subsystems (4)
lore
Share & tools
lore

Tony Luck

author maintainer
#0 lore raw

cover Resctrl - rewrite (WIP)

Back in April I posted some RFC patches that added a "driver registration" interface to the core resctrl code so that additional resource control and monitor features could be added without further complicating the core code. Link to that discussion:

https://lore.kernel.org/all/[email protected]/

Reinette gave the feedback that it would be better to base the module registration on the resctrl resource structure. Reinette also pointed me to work from James Morse, and some additional discussion happened here:

https://lore.kernel.org/all/ZG%2FMZVrWYrCHm%2Ffr@agluck-desk3/

James provided details on where ARM's MPAM has similarities and differences from the Intel Resource Director Technology and AMD's similar implementation. Drew Fustini was also pulled into that conversation to comment on RISC-V CBQRI.

From those discussions I believed we need a do-over on the core /sys/fs/resctrl implementation to make it friendlier for architecural variations. Here's what I have so far.

=========================================================================

| N.B. This is a general direction check. There are many obvious        |
| rough edges (e.g. some careful thought needs to happen on locking     |
| for the files in /sys/fs/resctrl that are "owned" by modules that     |
| can be unloaded). I'm mostly looking for feedback from AMD, ARM and   |
| RISCV on whether this is a foundation to build on, whether some small |
| tweaks could make it better, or if this is still going to be really   |
| hard for architectures that have radical divergence from the Intel    |
| model.                                                                |

=========================================================================

First patch is my attempt at architecture neutral code. All mention of "RDT", "CLOSID" and "RMID" have been expunged. When creating a new group this code calls arch_alloc_resctrl_ids() to allocate an opaque "resctrl_ids" value.

Q: I made this a "u64" because that neatly allows storage of both an x86 CLOSID and RMID (in a handy representation that matches the bit layout of the Intel IA32_PQR_ASSOC model specific register). If other architectures need something more complex it could be a "typedef resctrl_id_t" ... there are a couple of places where we would need a comparison function.

I broke the code into several source files that handle different sub-functions of core code to make it easier to navigate. Much of the code here should look familiar as I did a lot of s/rdtgroup/resctrl_group/ on functions from the original resctrl code.

By itself the core code is useless. Cannot even be built as the controlling Kconfig option "CONFIG_RESCTRL2_FS" must be invoked by a "select" request from architecture specific code that provides the necessary "arch_*()" functions to make everything work.

Module registration is handled in fs/resctrl2/resources.c and can be done before or after mounting /sys/fs/resctrl. Current code won't let you make any new resource groups until a module implementing a control function is loaded to supply the information on how many groups the architecture supports.

Second patch is all the Intel X86 code (with some of the AMD bits included, but by no means all of them).

I've implemented modules for most of the legacy Intel control and monitor functions. Many of these share common code (by means of a symlinked source file ... I couldn't figure out how to make Kconfig build both rdt_l3_cat.ko and rdt_l3_cdp.ko from the same source file with a different set of $(CFLAGS)).

Users can pick which features they want by loading modules that implement the bits they want. E.g. CDP is enabled by loading that rdt_l3_cdp.ko module instead of rdt_l3_cat.ko (there's some code to prevent both being loaded together).

I started on the hooks for the "mba_MBps" feedback from MBM driver, but in this code drop I just have a simple module that reports the bandwidth for each group instead of the byte count. I just need to create a module that has both MBA control and MBM monitoring resources with a periodic comparison of actual bandwidth with desired, that then tweaks the MBA controls up/down as needed.

I haven't ventured to read all the pseudo-locking code, but it looks as though providing the driver with a way to tell core code that a group is exclusive instead of shared (which tells core code not to allow assignment of tasks or CPUs to the group) may be all the surgery needed to core code. The x86 module will be more complex that the toys I've produced so far, but should be able to leverage much from the existing resctrl implementation.

Tony Luck (2):
resctrl2: Add all the generic code
resctrl2: Arch x86 modules for most of the legacy control/monitor
functions

 include/linux/resctrl.h                    | 107 +++++
 include/linux/sched.h                      |   3 +
 arch/x86/include/asm/resctrl.h             |  38 ++
 fs/resctrl2/arch/x86/rdt.h                 |  22 +
 fs/resctrl2/internal.h                     | 110 +++++
 arch/x86/kernel/cpu/amd.c                  |   3 +
 arch/x86/kernel/cpu/intel.c                |   3 +
 arch/x86/kernel/process_32.c               |   1 +
 arch/x86/kernel/process_64.c               |   3 +
 fs/resctrl2/arch/x86/alloc.c               | 119 +++++
 fs/resctrl2/arch/x86/rdt_l2_cat.c          |   1 +
 fs/resctrl2/arch/x86/rdt_l2_cdp.c          |   1 +
 fs/resctrl2/arch/x86/rdt_l3_cat.c          | 349 +++++++++++++++
 fs/resctrl2/arch/x86/rdt_l3_cdp.c          |   1 +
 fs/resctrl2/arch/x86/rdt_l3_mba.c          | 251 +++++++++++
 fs/resctrl2/arch/x86/rdt_llc_occupancy.c   | 100 +++++
 fs/resctrl2/arch/x86/rdt_mbm_adjust.c      |  91 ++++
 fs/resctrl2/arch/x86/rdt_mbm_local_bytes.c |   1 +
 fs/resctrl2/arch/x86/rdt_mbm_local_rate.c  |   1 +
 fs/resctrl2/arch/x86/rdt_mbm_total_bytes.c |   1 +
 fs/resctrl2/arch/x86/rdt_mbm_total_rate.c  |   1 +
 fs/resctrl2/arch/x86/rdt_monitor.c         | 491 +++++++++++++++++++++
 fs/resctrl2/cpu.c                          | 315 +++++++++++++
 fs/resctrl2/directory.c                    | 295 +++++++++++++
 fs/resctrl2/domain.c                       |  99 +++++
 fs/resctrl2/info.c                         |  99 +++++
 fs/resctrl2/kernfs.c                       |  58 +++
 fs/resctrl2/locking.c                      |  52 +++
 fs/resctrl2/resources.c                    |  85 ++++
 fs/resctrl2/root.c                         | 173 ++++++++
 fs/resctrl2/schemata.c                     | 110 +++++
 fs/resctrl2/tasks.c                        | 193 ++++++++
 arch/x86/Kconfig                           |  81 +++-
 fs/Kconfig                                 |   1 +
 fs/Makefile                                |   1 +
 fs/resctrl2/Kconfig                        |   5 +
 fs/resctrl2/Makefile                       |  14 +
 fs/resctrl2/arch/x86/Makefile              |  29 ++
 38 files changed, 3306 insertions(+), 2 deletions(-)
 create mode 100644 fs/resctrl2/arch/x86/rdt.h
 create mode 100644 fs/resctrl2/internal.h
 create mode 100644 fs/resctrl2/arch/x86/alloc.c
 create mode 120000 fs/resctrl2/arch/x86/rdt_l2_cat.c
 create mode 120000 fs/resctrl2/arch/x86/rdt_l2_cdp.c
 create mode 100644 fs/resctrl2/arch/x86/rdt_l3_cat.c
 create mode 120000 fs/resctrl2/arch/x86/rdt_l3_cdp.c
 create mode 100644 fs/resctrl2/arch/x86/rdt_l3_mba.c
 create mode 100644 fs/resctrl2/arch/x86/rdt_llc_occupancy.c
 create mode 100644 fs/resctrl2/arch/x86/rdt_mbm_adjust.c
 create mode 120000 fs/resctrl2/arch/x86/rdt_mbm_local_bytes.c
 create mode 120000 fs/resctrl2/arch/x86/rdt_mbm_local_rate.c
 create mode 120000 fs/resctrl2/arch/x86/rdt_mbm_total_bytes.c
 create mode 120000 fs/resctrl2/arch/x86/rdt_mbm_total_rate.c
 create mode 100644 fs/resctrl2/arch/x86/rdt_monitor.c
 create mode 100644 fs/resctrl2/cpu.c
 create mode 100644 fs/resctrl2/directory.c
 create mode 100644 fs/resctrl2/domain.c
 create mode 100644 fs/resctrl2/info.c
 create mode 100644 fs/resctrl2/kernfs.c
 create mode 100644 fs/resctrl2/locking.c
 create mode 100644 fs/resctrl2/resources.c
 create mode 100644 fs/resctrl2/root.c
 create mode 100644 fs/resctrl2/schemata.c
 create mode 100644 fs/resctrl2/tasks.c
 create mode 100644 fs/resctrl2/Kconfig
 create mode 100644 fs/resctrl2/Makefile
 create mode 100644 fs/resctrl2/arch/x86/Makefile

base-commit: 45a3e24f65e90a047bef86f927ebdc4c710edaa1

Signature · 1 line
2.40.1
author maintainer
#1 lore raw

RFC PATCH 1/2 resctrl2: Add all the generic code

Generic resctrly code is responsible for:

0) Mounting and unmounting /sys/fs/resctrl
1) Creating and removing groups
2) Associating tasks with groups
3) Associating CPUs with groups
4) Context switch, calling architecture code to begin control/monitor for the nex task based on which groups it is in, and which CPU it is running on.
5) Registering and unregistering architecture modules for all control and monitor functions.

N.B. Much of this code is just the old resctrl code with a bunch of renaming and refactoring.

---
 include/linux/resctrl.h | 107 ++++++++++++++
 include/linux/sched.h   |   3 +
 fs/resctrl2/internal.h  | 110 ++++++++++++++
 fs/resctrl2/cpu.c       | 315 ++++++++++++++++++++++++++++++++++++++++
 fs/resctrl2/directory.c | 295 +++++++++++++++++++++++++++++++++++++
 fs/resctrl2/domain.c    |  99 +++++++++++++
 fs/resctrl2/info.c      |  99 +++++++++++++
 fs/resctrl2/kernfs.c    |  58 ++++++++
 fs/resctrl2/locking.c   |  52 +++++++
 fs/resctrl2/resources.c |  85 +++++++++++
 fs/resctrl2/root.c      | 173 ++++++++++++++++++++++
 fs/resctrl2/schemata.c  | 110 ++++++++++++++
 fs/resctrl2/tasks.c     | 193 ++++++++++++++++++++++++
 fs/Kconfig              |   1 +
 fs/Makefile             |   1 +
 fs/resctrl2/Kconfig     |   5 +
 fs/resctrl2/Makefile    |  14 ++
 17 files changed, 1720 insertions(+)
 create mode 100644 fs/resctrl2/internal.h
 create mode 100644 fs/resctrl2/cpu.c
 create mode 100644 fs/resctrl2/directory.c
 create mode 100644 fs/resctrl2/domain.c
 create mode 100644 fs/resctrl2/info.c
 create mode 100644 fs/resctrl2/kernfs.c
 create mode 100644 fs/resctrl2/locking.c
 create mode 100644 fs/resctrl2/resources.c
 create mode 100644 fs/resctrl2/root.c
 create mode 100644 fs/resctrl2/schemata.c
 create mode 100644 fs/resctrl2/tasks.c
 create mode 100644 fs/resctrl2/Kconfig
 create mode 100644 fs/resctrl2/Makefile
include/linux/resctrl.h +107 -0
index 8334eeacfec5..889d9c65d8d9 100644
@@ -2,6 +2,8 @@
 #ifndef _RESCTRL_H
 #define _RESCTRL_H
 
+#ifdef CONFIG_X86_CPU_RESCTRL
+
 #include <linux/kernel.h>
 #include <linux/list.h>
 #include <linux/pid.h>
@@ -264,4 +266,109 @@ void resctrl_arch_reset_rmid_all(struct rdt_resource *r, struct rdt_domain *d);
 extern unsigned int resctrl_rmid_realloc_threshold;
 extern unsigned int resctrl_rmid_realloc_limit;
 
+#endif /* CONFIG_X86_CPU_RESCTRL */
+
+#ifdef CONFIG_RESCTRL2_FS
+
+#include <linux/sched.h>
+#include <linux/jump_label.h>
+
+bool arch_check_resctrl_support(void);
+void arch_resctrl_apply_ids(u64 resctrl_ids);
+extern u64 arch_resctrl_default_ids;
+
+DECLARE_STATIC_KEY_FALSE(resctrl_enable_key);
+
+struct resctrl_per_cpu_state {
+	u64	cached_resctrl_ids;
+	u64	default_resctrl_ids;
+};
+
+DECLARE_PER_CPU(struct resctrl_per_cpu_state, resctrl_per_cpu_state);
+
+static inline void resctrl_sched_in(struct task_struct *tsk)
+{
+	struct resctrl_per_cpu_state *state;
+	u64 new_resctrl_ids;
+
+	if (!static_branch_likely(&resctrl_enable_key))
+		return;
+
+	state = this_cpu_ptr(&resctrl_per_cpu_state);
+	new_resctrl_ids = state->default_resctrl_ids;
+
+	if (tsk->resctrl_ids != arch_resctrl_default_ids)
+		new_resctrl_ids = tsk->resctrl_ids;
+
+	if (new_resctrl_ids != state->cached_resctrl_ids) {
+		state->cached_resctrl_ids = new_resctrl_ids;
+		arch_resctrl_apply_ids(new_resctrl_ids);
+	}
+}
+
+/* Unclear if this is still useful */
+static inline void resctrl_cpu_detect(struct cpuinfo_x86 *c) {}
+
+enum resctrl_type {
+	RESCTRL_CONTROL,
+	RESCTRL_MONITOR,
+};
+
+enum resctrl_scope {
+	RESCTRL_CORE,
+	RESCTRL_L2CACHE,
+	RESCTRL_L3CACHE,
+	RESCTRL_SOCKET,
+};
+
+enum resctrl_domain_update {
+	RESCTRL_DOMAIN_ADD,
+	RESCTRL_DOMAIN_ADD_CPU,
+	RESCTRL_DOMAIN_DELETE_CPU,
+	RESCTRL_DOMAIN_DELETE,
+};
+
+struct resctrl_domain {
+	struct list_head	list;
+	struct cpumask		cpu_mask;
+	int			id;
+};
+
+struct resctrl_fileinfo {
+	char			*name;
+	struct kernfs_ops	*ops;
+	void			*priv;
+};
+
+struct resctrl_resource {
+	char			*name;
+	int			archtag;
+	struct list_head	list;
+	int			type;
+	enum resctrl_scope	scope;
+	size_t			domain_size;
+	struct list_head	domains;
+	void			(*domain_update)(struct resctrl_resource *r, int what, int cpu, struct resctrl_domain *d);
+
+	char			*infodir;
+	struct resctrl_fileinfo	*infofiles;
+
+	// bits for control resources
+	int			num_alloc_ids;
+	void			(*show)(struct resctrl_resource *r, struct seq_file *m, u64 resctrl_ids);
+	void			(*resetstaging)(struct resctrl_resource *r, u64 resctrl_ids);
+	int			(*parse)(struct resctrl_resource *r, char *tok, u64 resctrl_ids);
+	void			(*applychanges)(struct resctrl_resource *r, u64 resctrl_ids);
+
+	// bits for monitor resources
+	char			*mon_domain_dir;
+	char			*mon_domain_file;
+	struct kernfs_ops	*mod_domain_ops;
+	int			mon_event;
+};
+
+int resctrl_register_ctrl_resource(struct resctrl_resource *r);
+void resctrl_unregister_ctrl_resource(struct resctrl_resource *r);
+
+#endif /* CONFIG_RESCTRL2_FS */
 #endif /* _RESCTRL_H */
include/linux/sched.h +3 -0
index eed5d65b8d1f..ab5e861bcb80 100644
@@ -1228,6 +1228,9 @@ struct task_struct {
 	u32				closid;
 	u32				rmid;
 #endif
+#ifdef CONFIG_RESCTRL2_FS
+	u64				resctrl_ids;
+#endif
 #ifdef CONFIG_FUTEX
 	struct robust_list_head __user	*robust_list;
 #ifdef CONFIG_COMPAT
fs/resctrl2/internal.h new file +110 -0

110 lines not on this page · raw message

fs/resctrl2/cpu.c new file +315 -0

315 lines not on this page · raw message

fs/resctrl2/directory.c new file +295 -0

295 lines not on this page · raw message

fs/resctrl2/domain.c new file +99 -0

99 lines not on this page · raw message

fs/resctrl2/info.c new file +99 -0

99 lines not on this page · raw message

fs/resctrl2/kernfs.c new file +58 -0

58 lines not on this page · raw message

fs/resctrl2/locking.c new file +52 -0

52 lines not on this page · raw message

fs/resctrl2/resources.c new file +85 -0

85 lines not on this page · raw message

fs/resctrl2/root.c new file +173 -0

173 lines not on this page · raw message

fs/resctrl2/schemata.c new file +110 -0

110 lines not on this page · raw message

fs/resctrl2/tasks.c new file +193 -0

193 lines not on this page · raw message

fs/Kconfig +1 -0
index 18d034ec7953..f4ebf8f5204b 100644
@@ -45,6 +45,7 @@ source "fs/btrfs/Kconfig"
 source "fs/nilfs2/Kconfig"
 source "fs/f2fs/Kconfig"
 source "fs/zonefs/Kconfig"
+source "fs/resctrl2/Kconfig"
 
 endif # BLOCK
 
fs/Makefile +1 -0
index 5bfdbf0d7037..0aab615ceb4d 100644
@@ -135,3 +135,4 @@ obj-$(CONFIG_EFIVAR_FS)		+= efivarfs/
 obj-$(CONFIG_EROFS_FS)		+= erofs/
 obj-$(CONFIG_VBOXSF_FS)		+= vboxsf/
 obj-$(CONFIG_ZONEFS_FS)		+= zonefs/
+obj-$(CONFIG_RESCTRL2_FS)	+= resctrl2/
fs/resctrl2/Kconfig new file +5 -0
new file mode 100644
index 000000000000..6ee9162b96c1
@@ -0,0 +1,5 @@
+#
+# Architectures that support resource control will select this
+#
+config RESCTRL2_FS
+	bool
fs/resctrl2/Makefile new file +14 -0
new file mode 100644
index 000000000000..debf91ebcb6a
@@ -0,0 +1,14 @@
+# SPDX-License-Identifier: GPL-2.0
+
+obj-$(CONFIG_RESCTRL2_FS) += cpu.o
+obj-$(CONFIG_RESCTRL2_FS) += directory.o
+obj-$(CONFIG_RESCTRL2_FS) += domain.o
+obj-$(CONFIG_RESCTRL2_FS) += info.o
+obj-$(CONFIG_RESCTRL2_FS) += kernfs.o
+obj-$(CONFIG_RESCTRL2_FS) += locking.o
+obj-$(CONFIG_RESCTRL2_FS) += resources.o
+obj-$(CONFIG_RESCTRL2_FS) += root.o
+obj-$(CONFIG_RESCTRL2_FS) += schemata.o
+obj-$(CONFIG_RESCTRL2_FS) += tasks.o
+
+obj-$(CONFIG_X86_CPU_RESCTRL2) += arch/x86/
Signature · 1 line
2.40.1
author maintainer
#2 lore raw

RFC PATCH 2/2 resctrl2: Arch x86 modules for most of the legacy control/monitor functions

In reply to Tony Luck

Missing:
1) "mba_MBps" feedback from MBM
2) Pseudo-locking
3) AMD SMBA/BMEC
4) Fixes and additions after v6.4-rc6
5) Other stuff I haven't noticed

N.B. This is proof-of-concept code ... the locking (or lack thereof) for most of the files controlled by modules needs to be addressed.

---
 arch/x86/include/asm/resctrl.h             |  38 ++
 fs/resctrl2/arch/x86/rdt.h                 |  22 +
 arch/x86/kernel/cpu/amd.c                  |   3 +
 arch/x86/kernel/cpu/intel.c                |   3 +
 arch/x86/kernel/process_32.c               |   1 +
 arch/x86/kernel/process_64.c               |   3 +
 fs/resctrl2/arch/x86/alloc.c               | 119 +++++
 fs/resctrl2/arch/x86/rdt_l2_cat.c          |   1 +
 fs/resctrl2/arch/x86/rdt_l2_cdp.c          |   1 +
 fs/resctrl2/arch/x86/rdt_l3_cat.c          | 349 +++++++++++++++
 fs/resctrl2/arch/x86/rdt_l3_cdp.c          |   1 +
 fs/resctrl2/arch/x86/rdt_l3_mba.c          | 251 +++++++++++
 fs/resctrl2/arch/x86/rdt_llc_occupancy.c   | 100 +++++
 fs/resctrl2/arch/x86/rdt_mbm_adjust.c      |  91 ++++
 fs/resctrl2/arch/x86/rdt_mbm_local_bytes.c |   1 +
 fs/resctrl2/arch/x86/rdt_mbm_local_rate.c  |   1 +
 fs/resctrl2/arch/x86/rdt_mbm_total_bytes.c |   1 +
 fs/resctrl2/arch/x86/rdt_mbm_total_rate.c  |   1 +
 fs/resctrl2/arch/x86/rdt_monitor.c         | 491 +++++++++++++++++++++
 arch/x86/Kconfig                           |  81 +++-
 fs/resctrl2/arch/x86/Makefile              |  29 ++
 21 files changed, 1586 insertions(+), 2 deletions(-)
 create mode 100644 fs/resctrl2/arch/x86/rdt.h
 create mode 100644 fs/resctrl2/arch/x86/alloc.c
 create mode 120000 fs/resctrl2/arch/x86/rdt_l2_cat.c
 create mode 120000 fs/resctrl2/arch/x86/rdt_l2_cdp.c
 create mode 100644 fs/resctrl2/arch/x86/rdt_l3_cat.c
 create mode 120000 fs/resctrl2/arch/x86/rdt_l3_cdp.c
 create mode 100644 fs/resctrl2/arch/x86/rdt_l3_mba.c
 create mode 100644 fs/resctrl2/arch/x86/rdt_llc_occupancy.c
 create mode 100644 fs/resctrl2/arch/x86/rdt_mbm_adjust.c
 create mode 120000 fs/resctrl2/arch/x86/rdt_mbm_local_bytes.c
 create mode 120000 fs/resctrl2/arch/x86/rdt_mbm_local_rate.c
 create mode 120000 fs/resctrl2/arch/x86/rdt_mbm_total_bytes.c
 create mode 120000 fs/resctrl2/arch/x86/rdt_mbm_total_rate.c
 create mode 100644 fs/resctrl2/arch/x86/rdt_monitor.c
 create mode 100644 fs/resctrl2/arch/x86/Makefile
arch/x86/include/asm/resctrl.h +38 -0
index 255a78d9d906..efbad3175680 100644
@@ -96,6 +96,44 @@ static inline void resctrl_sched_in(struct task_struct *tsk)
 
 void resctrl_cpu_detect(struct cpuinfo_x86 *c);
 
+#elif defined(CONFIG_X86_CPU_RESCTRL2)
+
+bool arch_alloc_resctrl_ids(struct resctrl_group *rg);
+void arch_free_resctrl_ids(struct resctrl_group *rg);
+bool arch_init_alloc_ids(struct resctrl_resource *r);
+int rmid_alloc(int prmid);
+void rmid_free(int rmid);
+void arch_add_monitor(int mon_event);
+void arch_del_monitor(int mon_event);
+u64 rdt_rmid_read(int domain_id, int rmid, int event);
+void rdt_mbm_apply_quirk(int num_rmids);
+u64 get_corrected_mbm_count(u32 rmid, unsigned long val);
+
+static inline bool is_closid_match(struct task_struct *t, struct resctrl_group *rg)
+{
+	return (t->resctrl_ids >> 32) == (rg->resctrl_ids >> 32);
+}
+
+static inline bool arch_is_resctrl_id_match(struct task_struct *t, struct resctrl_group *rg)
+{
+	if (rg->type == DIR_MON)
+		return t->resctrl_ids == rg->resctrl_ids;
+	return is_closid_match(t, rg);
+}
+
+static inline bool arch_set_task_ids(struct task_struct *t, struct resctrl_group *rg)
+{
+	if (rg->type == DIR_MON) {
+		if (!is_closid_match(t, rg)) {
+			//rdt_last_cmd_puts("Can't move task to different control group\n");
+			return false;
+		}
+	}
+
+	WRITE_ONCE(t->resctrl_ids, rg->resctrl_ids);
+
+	return true;
+}
 #else
 
 static inline void resctrl_sched_in(struct task_struct *tsk) {}
fs/resctrl2/arch/x86/rdt.h new file +22 -0
new file mode 100644
index 000000000000..98c8147bac8e
@@ -0,0 +1,22 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/* Copyright(c) 2023 Intel Corporation. */
+
+/* H/W supported RDT monitor events */
+#define EV_LLC		1
+#define EV_TOT		2
+#define EV_LOC		3
+#define EV_MAX		4
+
+/* S/W events */
+#define EV_TOTRATE	4
+#define EV_LOCRATE	5
+
+#define RESCTRL_FILE_DEF(X, fmt)			\
+static int X##_show(struct seq_file *sf, void *v)	\
+{							\
+	seq_printf(sf, fmt, X);				\
+	return 0;					\
+}							\
+static struct kernfs_ops X##_ops = {			\
+	.seq_show	= X##_show			\
+};
arch/x86/kernel/cpu/amd.c +3 -0
index 571abf808ea3..fbd95edcd75b 100644
@@ -9,6 +9,7 @@
 #include <linux/sched/clock.h>
 #include <linux/random.h>
 #include <linux/topology.h>
+#include <linux/resctrl.h>
 #include <asm/processor.h>
 #include <asm/apic.h>
 #include <asm/cacheinfo.h>
@@ -19,7 +20,9 @@
 #include <asm/pci-direct.h>
 #include <asm/delay.h>
 #include <asm/debugreg.h>
+#ifndef CONFIG_RESCTRL2_FS
 #include <asm/resctrl.h>
+#endif
 
 #ifdef CONFIG_X86_64
 # include <asm/mmconfig.h>
arch/x86/kernel/cpu/intel.c +3 -0
index 1c4639588ff9..66829e06b01f 100644
@@ -14,6 +14,7 @@
 #include <linux/workqueue.h>
 #include <linux/delay.h>
 #include <linux/cpuhotplug.h>
+#include <linux/resctrl.h>
 
 #include <asm/cpufeature.h>
 #include <asm/msr.h>
@@ -26,7 +27,9 @@
 #include <asm/cpu_device_id.h>
 #include <asm/cmdline.h>
 #include <asm/traps.h>
+#ifndef CONFIG_RESCTRL2_FS
 #include <asm/resctrl.h>
+#endif
 #include <asm/numa.h>
 #include <asm/thermal.h>
 
arch/x86/kernel/process_32.c +1 -0
index 708c87b88cc1..8cf70973a331 100644
@@ -38,6 +38,7 @@
 #include <linux/io.h>
 #include <linux/kdebug.h>
 #include <linux/syscalls.h>
+#include <linux/resctrl.h>
 
 #include <asm/ldt.h>
 #include <asm/processor.h>
arch/x86/kernel/process_64.c +3 -0
index 3d181c16a2f6..7df3cf1dbcaa 100644
@@ -40,6 +40,7 @@
 #include <linux/ftrace.h>
 #include <linux/syscalls.h>
 #include <linux/iommu.h>
+#include <linux/resctrl.h>
 
 #include <asm/processor.h>
 #include <asm/pkru.h>
@@ -53,7 +54,9 @@
 #include <asm/switch_to.h>
 #include <asm/xen/hypervisor.h>
 #include <asm/vdso.h>
+#ifndef CONFIG_RESCTRL2_FS
 #include <asm/resctrl.h>
+#endif
 #include <asm/unistd.h>
 #include <asm/fsgsbase.h>
 #ifdef CONFIG_IA32_EMULATION
fs/resctrl2/arch/x86/alloc.c new file +119 -0

119 lines not on this page · raw message

fs/resctrl2/arch/x86/rdt_l2_cat.c new file +1 -0
new file mode 120000
index 000000000000..d25679af550e
@@ -0,0 +1 @@
+rdt_l3_cat.c
\ No newline at end of file
fs/resctrl2/arch/x86/rdt_l2_cdp.c new file +1 -0
new file mode 120000
index 000000000000..d25679af550e
@@ -0,0 +1 @@
+rdt_l3_cat.c
\ No newline at end of file
fs/resctrl2/arch/x86/rdt_l3_cat.c new file +349 -0

349 lines not on this page · raw message

fs/resctrl2/arch/x86/rdt_l3_cdp.c new file +1 -0
new file mode 120000
index 000000000000..d25679af550e
@@ -0,0 +1 @@
+rdt_l3_cat.c
\ No newline at end of file
fs/resctrl2/arch/x86/rdt_l3_mba.c new file +251 -0

251 lines not on this page · raw message

fs/resctrl2/arch/x86/rdt_llc_occupancy.c new file +100 -0

100 lines not on this page · raw message

fs/resctrl2/arch/x86/rdt_mbm_adjust.c new file +91 -0

91 lines not on this page · raw message

fs/resctrl2/arch/x86/rdt_mbm_local_bytes.c new file +1 -0
new file mode 120000
index 000000000000..e15bfdd4143e
@@ -0,0 +1 @@
+rdt_llc_occupancy.c
\ No newline at end of file
fs/resctrl2/arch/x86/rdt_mbm_local_rate.c new file +1 -0
new file mode 120000
index 000000000000..e15bfdd4143e
@@ -0,0 +1 @@
+rdt_llc_occupancy.c
\ No newline at end of file
fs/resctrl2/arch/x86/rdt_mbm_total_bytes.c new file +1 -0
new file mode 120000
index 000000000000..e15bfdd4143e
@@ -0,0 +1 @@
+rdt_llc_occupancy.c
\ No newline at end of file
fs/resctrl2/arch/x86/rdt_mbm_total_rate.c new file +1 -0
new file mode 120000
index 000000000000..e15bfdd4143e
@@ -0,0 +1 @@
+rdt_llc_occupancy.c
\ No newline at end of file
fs/resctrl2/arch/x86/rdt_monitor.c new file +491 -0

491 lines not on this page · raw message

arch/x86/Kconfig +79 -2

95 lines not on this page · raw message

fs/resctrl2/arch/x86/Makefile new file +29 -0
new file mode 100644
index 000000000000..393a9412f151
@@ -0,0 +1,29 @@
+# SPDX-License-Identifier: GPL-2.0
+
+obj-$(CONFIG_X86_CPU_RESCTRL2) += alloc.o
+obj-$(CONFIG_X86_CPU_RESCTRL2) += rdt_monitor.o
+obj-$(CONFIG_X86_CPU_RESCTRL2) += rdt_mbm_adjust.o
+
+CFLAGS_rdt_l3_cat.o += -DCACHE_LEVEL=3
+CFLAGS_rdt_l2_cat.o += -DCACHE_LEVEL=2
+CFLAGS_rdt_l3_cdp.o += -DCACHE_LEVEL=3 -DCDP
+CFLAGS_rdt_l2_cdp.o += -DCACHE_LEVEL=2 -DCDP
+
+obj-$(CONFIG_X86_RDT_L3_CAT)		+= rdt_l3_cat.o
+obj-$(CONFIG_X86_RDT_L2_CAT)		+= rdt_l2_cat.o
+obj-$(CONFIG_X86_RDT_L3_CDP)		+= rdt_l3_cdp.o
+obj-$(CONFIG_X86_RDT_L2_CDP)		+= rdt_l2_cdp.o
+
+CFLAGS_rdt_llc_occupancy.o		+= -DEVENT=1
+CFLAGS_rdt_mbm_total_bytes.o		+= -DEVENT=2
+CFLAGS_rdt_mbm_local_bytes.o		+= -DEVENT=3
+CFLAGS_rdt_mbm_total_rate.o		+= -DEVENT=4
+CFLAGS_rdt_mbm_local_rate.o		+= -DEVENT=5
+
+obj-$(CONFIG_X86_RDT_LLC_OCCUPANCY)	+= rdt_llc_occupancy.o
+obj-$(CONFIG_X86_RDT_MBM_TOTAL_BYTES)	+= rdt_mbm_total_bytes.o
+obj-$(CONFIG_X86_RDT_MBM_LOCAL_BYTES)	+= rdt_mbm_local_bytes.o
+obj-$(CONFIG_X86_RDT_MBM_TOTAL_RATE)	+= rdt_mbm_total_rate.o
+obj-$(CONFIG_X86_RDT_MBM_LOCAL_RATE)	+= rdt_mbm_local_rate.o
+
+obj-$(CONFIG_X86_RDT_L3_MBA)		+= rdt_l3_mba.o
Signature · 1 line
2.40.1
#3 lore raw

Re: [RFC PATCH 2/2] resctrl2: Arch x86 modules for most of the legacy control/monitor functions

Hi Tony,

Quoted 6 lines · Tony Luck
fs/resctrl2/arch/x86/rdt_monitor.c new file +6 -0
+struct rmid {
+       struct list_head        list;
+       struct list_head        child_list;
+       bool                    is_parent;
+static void __rdt_rmid_read(void *info)
+{

fs/resctrl2/arch/x86/rdt_monitor.c line 107

Quoted 10 lines
fs/resctrl2/arch/x86/rdt_monitor.c new file +10 -0
+       struct rrmid_info *rr = info;
+       unsigned long flags;
+       struct rmid *cr, *r;
+       struct mydomain *m;
+       u64 chunks;
+
+       m = get_mydomain(rr->domain);
+
+       if (rr->event <= EV_LOC) {
+               spin_lock_irqsave(&m->msr_lock, flags);

Will there ultimately be any locking at the filesystem layer? I recall from feedback on my change adding a spinlock here[1] before that the filesystem-layer locking took care of this.

Quoted 39 lines
fs/resctrl2/arch/x86/rdt_monitor.c new file +39 -0
+               wrmsrl(MSR_IA32_QM_EVTSEL, (rr->rmid << 32) | rr->event);
+               rdmsrl(MSR_IA32_QM_CTR, chunks);
+       } else {
+               chunks = 0;
+       }
+
+       rr->chunks = adjust(m, rr->rmid, rr->event, chunks);
+
+       r = &rmid_array[rr->rmid];
+       if (r->is_parent && !list_empty(&r->child_list)) {
+               list_for_each_entry(cr, &r->child_list, child_list) {
+                       u64 crmid = cr - rmid_array;
+
+                       if (rr->event <= EV_LOC) {
+                               wrmsrl(MSR_IA32_QM_EVTSEL, (crmid << 32) | rr->event);
+                               rdmsrl(MSR_IA32_QM_CTR, chunks);
+                       } else {
+                               chunks = 0;
+                       }
+
+                       rr->chunks += adjust(m, crmid, rr->event, chunks);
+               }
+       }
+
+       if (rr->event <= EV_LOC)
+               spin_unlock_irqrestore(&m->msr_lock, flags);
+}
+
+u64 rdt_rmid_read(int domain_id, int rmid, int event)
+{
+       struct resctrl_domain *d;
+       struct rrmid_info rr;
+       struct mydomain *m;
+
+       list_for_each_entry(d, &monitor.domains, list)
+               if (d->id == domain_id)
+                       goto found;
+       return ~0ull;
+found:

fs/resctrl2/arch/x86/rdt_monitor.c line 320

Quoted 10 lines
fs/resctrl2/arch/x86/rdt_monitor.c new file +10 -0
+       m = get_mydomain(d);
+
+       rr.domain = d;
+       rr.rmid = rmid;
+       rr.event = event;
+
+       if (event <= EV_LOC)
+               smp_call_function_any(&d->cpu_mask, __rdt_rmid_read, &rr, 1);
+       else
+               __rdt_rmid_read(&rr);

I like that the driver is responsible for deciding where IPIs need to be sent, but it looks like the consequence is that RDT-level code wants to add in the child monitors' event counts once executing within the correct domain. The one-per-domain IPI assumption from the current resctrl code being wrong is probably harder to overcome than needing to figure out what additional RMIDs to read, but I'd really need to know the synchronization requirements for __rdt_rmid_read() to inspect the monitoring group hierarchy.

Would you continue to promise that the FS structure won't change during a monitor read? To us, the biggest priority for parallelization is reading all the domain-group combinations in the system, because we have a lot of them and want the tightest possible snapshot of bandwidth usage, broken down by group.

Thanks!
-Peter

[1] https://lore.kernel.org/all/[email protected]/

author maintainer
#4 lore raw

RE: [RFC PATCH 2/2] resctrl2: Arch x86 modules for most of the legacy control/monitor functions

Quoted 6 lines

fs/resctrl2/arch/x86/rdt_monitor.c line 271

Quoted 2 lines
fs/resctrl2/arch/x86/rdt_monitor.c new file +2 -0
+       if (rr->event <= EV_LOC) {
+               spin_lock_irqsave(&m->msr_lock, flags);

Will there ultimately be any locking at the filesystem layer? I recall from feedback on my change adding a spinlock here[1] before that the filesystem-layer locking took care of this.

Yes. Working on that now. The spinlock will go away when everything is protected. by resctrl_mutex.

Quoted 5 lines

fs/resctrl2/arch/x86/rdt_monitor.c line 320

Quoted 4 lines
fs/resctrl2/arch/x86/rdt_monitor.c new file +4 -0
+       if (event <= EV_LOC)
+               smp_call_function_any(&d->cpu_mask, __rdt_rmid_read, &rr, 1);
+       else
+               __rdt_rmid_read(&rr);
Quoted 8 lines

I like that the driver is responsible for deciding where IPIs need to be sent, but it looks like the consequence is that RDT-level code wants to add in the child monitors' event counts once executing within the correct domain. The one-per-domain IPI assumption from the current resctrl code being wrong is probably harder to overcome than needing to figure out what additional RMIDs to read, but I'd really need to know the synchronization requirements for __rdt_rmid_read() to inspect the monitoring group hierarchy.

There are likely some gaps in current code. But I think they should be fixable.

Quoted 5 lines

Would you continue to promise that the FS structure won't change during a monitor read? To us, the biggest priority for parallelization is reading all the domain-group combinations in the system, because we have a lot of them and want the tightest possible snapshot of bandwidth usage, broken down by group.

The mbm_poll() code that makes sure that counters don't wrap is doing all the expensive wrmsr(QM_EVTSEL);rdmsr(QM_COUNT) once per second to give you the data you want. But existing resctrl filesystem doesn't let you do a bulk read. I have some ideas on how to provide something better. One question: do you really need that snapshot to be system-wide? Or can you live with separate L3-scoped snapshots that aren't tightly synchronized with each other?

-Tony

#5 lore raw

Re: [RFC PATCH 2/2] resctrl2: Arch x86 modules for most of the legacy control/monitor functions

Hi Tony,

Quoted 3 lines · Tony

The mbm_poll() code that makes sure that counters don't wrap is doing all the expensive wrmsr(QM_EVTSEL);rdmsr(QM_COUNT) once per second to give you the data you want.

I was doing that in the soft RMID series I posted earlier because it simplified things, but then I had some realizations about how much error +/- 1 second on the sampling point could result in[1]. We usually measure the bandwidth rate with a 5-second window, so a reading that's up to one second old would mean a 20% error in the bandwidth calculation.

Quoted 5 lines

But existing resctrl
filesystem doesn't let you do a bulk read. I have some ideas on how to provide something better. One question: do you really need that snapshot to be system-wide? Or can you live with separate L3-scoped snapshots that aren't tightly synchronized with each other?

The most demanding use case is responding to a bandwidth saturation emergency, where the admin wants a snapshot of all jobs' bandwidth rates to identify which one(s) to kill. Jobs aren't confined to L3 domains, so I expect it's more important to get a tight snapshot for an RMID-at-a-time. Going domain-at-a-time could create more races where a high-bandwidth job migrates at the right time and avoids detection.

One workaround we've been using to cut down on IPIs is providing a usermode threshold knob for limiting how old a reported event count can be, but this is more useful for soft RMIDs where reading a single CPU's counters makes all RMIDs stored counts current, and IPIs resulting from userspace reads of different groups in the same domain are all redundant.

We could probably work with second-old event counts if we knew the exact time they were recorded so we'd know the exact time delta for the bandwidth count delta, but that seems like it would be awkward in the current interface.

-Peter

[1] https://lore.kernel.org/lkml/CALPaoCi_N8JHbP4zFD=ijBh5-=4Q0Ec-LrspYNGGnj4G6A6U0g@mail.gmail.com/

author maintainer
#6 lore raw

Re: [RFC PATCH 2/2] resctrl2: Arch x86 modules for most of the legacy control/monitor functions

Quoted 7 lines · Peter Newman

Hi Tony,

Quoted 3 lines · Tony

The mbm_poll() code that makes sure that counters don't wrap is doing all the expensive wrmsr(QM_EVTSEL);rdmsr(QM_COUNT) once per second to give you the data you want.

Quoted 6 lines

I was doing that in the soft RMID series I posted earlier because it simplified things, but then I had some realizations about how much error +/- 1 second on the sampling point could result in[1]. We usually measure the bandwidth rate with a 5-second window, so a reading that's up to one second old would mean a 20% error in the bandwidth calculation.

I just pushed the latest version of the resctrl2 patches to the resctrl2_v65rc1 branch of git://git.kernel.org/pub/scm/linux/kernel/git/aegl/linux.git

As well as locking, bug fixes, and general updates it includes an experimental feature to provide summary MBM information for each node. E.g. (both "total" and "local" rates are provided). Note that you have to load modules rdt_mbm_local_bytes and rdt_mbm_total_bytes so that the MBM overflow threads are running. I should fix the code to print "n/a" instead of "0" if they are not.

$ cat /sys/fs/resctrl/mon_data/mon_L3_00/mbm_summary
    3638     3638 /g2
    3087     3087 /g2/m2
    3267     3267 /g2/m1
    3443     3443 /g1
    3629     3629 /g1/m2
    3588     3587 /g1/m1
    3999     3993 /
    3370     3369 /m2
    3432     3432 /m1

The rates are produced once per second by the MBM overflow code. They compute MBytes/sec as "chunks since last poll" divided by (now - then). I'm using jiffies for the times which may be good enough. "now - then" is one second (maybe more if the kernel thread doing the MBM polling is delayed from running).

I should fix the summarization code to work the same as the regular MBM files (i.e. make the parent control directory report the sum of all its children).

The code also attempts (but fails) to make these mbm_summary files poll(2)-able. With the wakeup dependent on aggregate measure bandwidth compared against a configurable threshold:

$ cat /sys/fs/resctrl/info/L3_MON/mbm_poll_threshold
10000000

There's something wrong though. Poll(2) always says there is data to be read. I only see one other piece of kernel code implementing poll on kernfs (in the cgroup code). Perhaps my problem is inability to write an appliction that uses poll(2) correctly.

Let me know if this all seems like a useful direction. Maybe the polling part is overkill and it is sufficient to just have a cheap way to get all the bandwidths even if the values seen might be up to one second old.

-Tony

author maintainer
#7 lore raw
In reply to Tony Luck
Quoted 1 line
38 files changed, 3306 insertions(+), 2 deletions(-)

Lest this be too scary, I'll note the original resctrl code looks like:

$ find arch/x86/kernel/cpu/resctrl  -type f | xargs wc -l
    43 arch/x86/kernel/cpu/resctrl/pseudo_lock_event.h
     4 arch/x86/kernel/cpu/resctrl/Makefile
   996 arch/x86/kernel/cpu/resctrl/core.c
   581 arch/x86/kernel/cpu/resctrl/ctrlmondata.c
   560 arch/x86/kernel/cpu/resctrl/internal.h
   845 arch/x86/kernel/cpu/resctrl/monitor.c
  1600 arch/x86/kernel/cpu/resctrl/pseudo_lock.c
  3733 arch/x86/kernel/cpu/resctrl/rdtgroup.c
  8362 total

I'm haven't included pseudo_lock ... but I have most of the rest, so I'm confident this will end up as a net reduction in lines of code.

-Tony

#8 lore raw
In reply to Tony Luck
Quoted 39 lines · Tony Luck

Back in April I posted some RFC patches that added a "driver registration" interface to the core resctrl code so that additional resource control and monitor features could be added without further complicating the core code. Link to that discussion:

https://lore.kernel.org/all/[email protected]/

Reinette gave the feedback that it would be better to base the module registration on the resctrl resource structure. Reinette also pointed me to work from James Morse, and some additional discussion happened here:

https://lore.kernel.org/all/ZG%2FMZVrWYrCHm%2Ffr@agluck-desk3/

James provided details on where ARM's MPAM has similarities and differences from the Intel Resource Director Technology and AMD's similar implementation. Drew Fustini was also pulled into that conversation to comment on RISC-V CBQRI.

Quoted 1 line

From those discussions I believed we need a do-over on the core

/sys/fs/resctrl implementation to make it friendlier for architecural variations. Here's what I have so far.

=========================================================================

| N.B. This is a general direction check. There are many obvious        |
| rough edges (e.g. some careful thought needs to happen on locking     |
| for the files in /sys/fs/resctrl that are "owned" by modules that     |
| can be unloaded). I'm mostly looking for feedback from AMD, ARM and   |
| RISCV on whether this is a foundation to build on, whether some small |
| tweaks could make it better, or if this is still going to be really   |
| hard for architectures that have radical divergence from the Intel    |
| model.                                                                |

=========================================================================

First patch is my attempt at architecture neutral code. All mention of "RDT", "CLOSID" and "RMID" have been expunged. When creating a new group this code calls arch_alloc_resctrl_ids() to allocate an opaque "resctrl_ids" value.

Quoted 6 lines

Q: I made this a "u64" because that neatly allows storage of both an x86 CLOSID and RMID (in a handy representation that matches the bit layout of the Intel IA32_PQR_ASSOC model specific register). If other architectures need something more complex it could be a "typedef resctrl_id_t" ... there are a couple of places where we would need a comparison function.

This works okay for RISC-V. The Ssqosid extension defines a 32-bit register sqoscfg (see chapter 2 of CBQRI spec [0]). This contains a 12-bit MCID field (similar to an RMID) and 12-bit RCID field (similar to an CLOSID).

Quoted 11 lines

I broke the code into several source files that handle different sub-functions of core code to make it easier to navigate. Much of the code here should look familiar as I did a lot of s/rdtgroup/resctrl_group/ on functions from the original resctrl code.

By itself the core code is useless. Cannot even be built as the controlling Kconfig option "CONFIG_RESCTRL2_FS" must be invoked by a "select" request from architecture specific code that provides the necessary "arch_*()" functions to make everything work.

I would like to try to rebase the RISC-V CBQRI resctrl RFC [1] on top of this patch series instead of the mpam snapshot branch [2].

I had a patch in my RFC that added config option RISCV_ISA_SSQOSID which selects ARCH_HAS_CPU_RESCTRL and RESCTRL_FS [3]. It seems I would need to change that to select CONFIG_RESCTRL2_FS ?

A patch [4] in that RFC adds the "arch_*()" functions in arch/riscv/kernel/qos/qos_resctrl.c

thanks,
drew

[0] https://github.com/riscv-non-isa/riscv-cbqri/blob/main/riscv-cbqri.pdf
[1] https://lore.kernel.org/linux-riscv/[email protected]/
[2] https://gitlab.arm.com/linux-arm/linux-jm/-/tree/mpam/snaphot/20230406
[3] https://lore.kernel.org/linux-riscv/[email protected]/
[4] https://lore.kernel.org/linux-riscv/[email protected]/

author maintainer
#9 lore raw
Quoted 7 lines
Quoted 4 lines

By itself the core code is useless. Cannot even be built as the controlling Kconfig option "CONFIG_RESCTRL2_FS" must be invoked by a "select" request from architecture specific code that provides the necessary "arch_*()" functions to make everything work.

I would like to try to rebase the RISC-V CBQRI resctrl RFC [1] on top of this patch series instead of the mpam snapshot branch [2].

Thanks. That should help shake out any places where I've left in Intel-isms, or my abstraction is insufficient to handle your architecture.

I've made some significant changes since I posted those patches. I pushed the latest version to:

git://git.kernel.org/pub/scm/linux/kernel/git/aegl/linux.git resctrl2_v64

Quoted 3 lines

I had a patch in my RFC that added config option RISCV_ISA_SSQOSID which selects ARCH_HAS_CPU_RESCTRL and RESCTRL_FS [3]. It seems I would need to change that to select CONFIG_RESCTRL2_FS ?

Yes. Just have your architecture CONFIG option select RESCTRL2_FS

Quoted 2 lines

A patch [4] in that RFC adds the "arch_*()" functions in arch/riscv/kernel/qos/qos_resctrl.c

Yes. This is an area that may need some tweaking to get the prototypes for the arch_*() functions right.

I put all the x86 architecture code under fs/resctrl2/arch/x86/ .... mostly so I can do quick test builds of both the common code and architecture code with "make fs/resctrl2/" ... maybe in the end-game they should be under arch/x86 rather than adding arch specific subdirs under generic top-level directories (though I see a smattering of "x86" directories in several places.

-Tony

author maintainer
#10 lore raw
Quoted 4 lines · Tony

I've made some significant changes since I posted those patches. I pushed the latest version to:

git://git.kernel.org/pub/scm/linux/kernel/git/aegl/linux.git resctrl2_v64

I just pushed one big commit with all the bits I've updated so far this week. Fixes some serious issues as well as general cleanup.

HEAD is now:

afb7cdd4d640 resctrl2: Many cleanups, fixes, and new functionality

If you've started writing your own architecture specific modules there are some small interface changes. Most should be found by the compiler barfing, but the new ".reset()" resource function called during unmount of /sys/fs/resctrl might be less obvious.

-Tony

#11 lore raw
Quoted 11 lines · Tony Luck
Quoted 4 lines · Tony

I've made some significant changes since I posted those patches. I pushed the latest version to:

git://git.kernel.org/pub/scm/linux/kernel/git/aegl/linux.git resctrl2_v64

I just pushed one big commit with all the bits I've updated so far this week. Fixes some serious issues as well as general cleanup.

HEAD is now:

Quoted 8 lines

afb7cdd4d640 resctrl2: Many cleanups, fixes, and new functionality

If you've started writing your own architecture specific modules there are some small interface changes. Most should be found by the compiler barfing, but the new ".reset()" resource function called during unmount of /sys/fs/resctrl might be less obvious.

-Tony

I have access to a Xeon Silver 4310 machine which reports to have cat_l3, cqm_mbm_local, cqm_mbm_total and mba.

I would like to test resctrl2 on it so I can better understand how it works. I think that will help me understand how to adapt the RISC-V CBQRI resctrl proof-of-concept to use resctrl2.

Would you be able to provide an example of how you loaded the necessary resctrl2 kernel modules?

Also, is resctrl2_v65rc1 the latest to branch to test?

Thank you,
Drew

author maintainer
#12 lore raw
Quoted 9 lines · Drew Fustini

I have access to a Xeon Silver 4310 machine which reports to have cat_l3, cqm_mbm_local, cqm_mbm_total and mba.

I would like to test resctrl2 on it so I can better understand how it works. I think that will help me understand how to adapt the RISC-V CBQRI resctrl proof-of-concept to use resctrl2.

Would you be able to provide an example of how you loaded the necessary resctrl2 kernel modules?

Drew,

Sure. You simply mount the filesystem, and then load modules for whichever features you'd like to use. This will enable everything you list above:

# mount -t resctrl resctrl /sys/fs/resctrl
# modprobe rdt_l3_cat
# modprobe rdt_llc_occupancy
# modprobe rdt_mbm_local_bytes
# modprobe rdt_mbm_total_bytes
# modprobe rdt_l3_mba

There are some experimental extras. E.g.

# modprobe rdt_mbm_total_rate
# modprobe rdt_mbm_local_rate

Will each add an extra file to the mon_data directories to report the data rate in MB/s. The value reported is calculated by the once-per-second counter roll-over code in the kernel. So it might be up to one second out of date, but it is very cheap to read since it doesn't involve MSR access (or cross processor interrupts if you are reading from a CPU in a different scope).

You can unload modules without unmounting the filesystem and load different ones to get different data/control. E.g. to switch from L3CAT to L3CDP (which you don't list as supported, so this may not work for you:

# rmmod rdt_l3_cat
# modprobe rdt_l3_cdp

Or to switch from the default MBA that uses percentages to specify throttling to the MBM->MBA feedback code that uses MB/s in the schemata file:

# rmmod rdt_l3_mba
# modprobe rdt_l3_mba_MBps
Quoted 2 lines

Also, is resctrl2_v65rc1 the latest to branch to test?

Yes. That's the latest. There haven't been any updates for a few days because I'm working on a module to support pseudo-locking. I'm half-way there (can do most of the bits to set a group into pseudo-locked mode ... about to work on the cleanup when the group is removed, the filesystem unmounted, or the module unloaded).

-Tony

author maintainer
#13 lore raw
Quoted 9 lines · Tony Luck
Quoted 2 lines · Drew Fustini

Also, is resctrl2_v65rc1 the latest to branch to test?

Yes. That's the latest. There haven't been any updates for a few days because I'm working on a module to support pseudo-locking. I'm half-way there (can do most of the bits to set a group into pseudo-locked mode ... about to work on the cleanup when the group is removed, the filesystem unmounted, or the module unloaded).

Updated version available at:

git://git.kernel.org/pub/scm/linux/kernel/git/aegl/linux.git

Branch: resctrl2_v65rc4

Some minor fixes to core code, also changes to support pseudo-locking (core code for the "mode" file plus some new functions in the resource structure to call into modules to support this).

-Tony

#14 lore raw
In reply to Tony Luck

Hi Tony,

Quoted 33 lines · Tony Luck

Back in April I posted some RFC patches that added a "driver registration" interface to the core resctrl code so that additional resource control and monitor features could be added without further complicating the core code. Link to that discussion:

https://lore.kernel.org/all/[email protected]/

Reinette gave the feedback that it would be better to base the module registration on the resctrl resource structure. Reinette also pointed me to work from James Morse, and some additional discussion happened here:

https://lore.kernel.org/all/ZG%2FMZVrWYrCHm%2Ffr@agluck-desk3/

James provided details on where ARM's MPAM has similarities and differences from the Intel Resource Director Technology and AMD's similar implementation. Drew Fustini was also pulled into that conversation to comment on RISC-V CBQRI.

From those discussions I believed we need a do-over on the core /sys/fs/resctrl implementation to make it friendlier for architecural variations. Here's what I have so far.

=========================================================================

| N.B. This is a general direction check. There are many obvious        |
| rough edges (e.g. some careful thought needs to happen on locking     |
| for the files in /sys/fs/resctrl that are "owned" by modules that     |
| can be unloaded). I'm mostly looking for feedback from AMD, ARM and   |
| RISCV on whether this is a foundation to build on, whether some small |
| tweaks could make it better, or if this is still going to be really   |
| hard for architectures that have radical divergence from the Intel    |
| model.                                                                |
Quoted 1 line

=========================================================================

Thanks for working on this! I played with these changes locally on some of our machines and they seemed reasonably functional so far and was happy to see dynamically adding and removing resources working.

I will need to try working with the code to give it a serious evaluation, though. Would you consider it ready for me to try re-implementing soft RMIDs on it?

I'm also very interested in James's opinion and what this means for the ongoing MPAM upstreaming.

Thanks!
-Peter

author maintainer
#15 lore raw
Quoted 3 lines

Thanks for working on this! I played with these changes locally on some of our machines and they seemed reasonably functional so far and was happy to see dynamically adding and removing resources working.

Thanks for taking it for a spin on some additional systems.

Quoted 3 lines

I will need to try working with the code to give it a serious evaluation, though. Would you consider it ready for me to try re-implementing soft RMIDs on it?

Current code is:
1) Lacking locking for access to files created on behalf of dynamic loaded modules. So, I expect the system to crash if you unload a module while simultaneously accessing one of those files.
2) Lacking error checking and cleanup code paths to undo partial operations when things fail to allocate.
3) The unmount (sb_kill()) code has fallen far behind development of other features. So expect memory leaks if you unmount and remount.

It should be OK to play around with this version, but things will go wrong when the system is under stress. Do not use in production!!

All of the RMID allocation and understanding of hierarchy between control and monitor groups is localized in fs/resctrl2/arch/x86/rdt_monitor.c I think I'm mostly done with the functionality I need in that file, so work you do there for soft RMIDs is unlikely to need refactoring for other changes that I'm making.

Quoted 2 lines

I'm also very interested in James's opinion and what this means for the ongoing MPAM upstreaming.

Me too. I'm hopeful that my code can be a better base than the legacy resctrl code. But it needs an MPAM expert to really assess that.

-Tony