Resctrl - rewrite (WIP)
Subsystems (4)
Tony Luck
maintainercover Resctrl - rewrite (WIP)
Back in April I posted some RFC patches that added a "driver registration" interface to the core resctrl code so that additional resource control and monitor features could be added without further complicating the core code. Link to that discussion:
https://lore.kernel.org/all/[email protected]/
Reinette gave the feedback that it would be better to base the module registration on the resctrl resource structure. Reinette also pointed me to work from James Morse, and some additional discussion happened here:
https://lore.kernel.org/all/ZG%2FMZVrWYrCHm%2Ffr@agluck-desk3/
James provided details on where ARM's MPAM has similarities and differences from the Intel Resource Director Technology and AMD's similar implementation. Drew Fustini was also pulled into that conversation to comment on RISC-V CBQRI.
From those discussions I believed we need a do-over on the core /sys/fs/resctrl implementation to make it friendlier for architecural variations. Here's what I have so far.
=========================================================================
| N.B. This is a general direction check. There are many obvious | | rough edges (e.g. some careful thought needs to happen on locking | | for the files in /sys/fs/resctrl that are "owned" by modules that | | can be unloaded). I'm mostly looking for feedback from AMD, ARM and | | RISCV on whether this is a foundation to build on, whether some small | | tweaks could make it better, or if this is still going to be really | | hard for architectures that have radical divergence from the Intel | | model. |
=========================================================================
First patch is my attempt at architecture neutral code. All mention of "RDT", "CLOSID" and "RMID" have been expunged. When creating a new group this code calls arch_alloc_resctrl_ids() to allocate an opaque "resctrl_ids" value.
Q: I made this a "u64" because that neatly allows storage of both an x86 CLOSID and RMID (in a handy representation that matches the bit layout of the Intel IA32_PQR_ASSOC model specific register). If other architectures need something more complex it could be a "typedef resctrl_id_t" ... there are a couple of places where we would need a comparison function.
I broke the code into several source files that handle different sub-functions of core code to make it easier to navigate. Much of the code here should look familiar as I did a lot of s/rdtgroup/resctrl_group/ on functions from the original resctrl code.
By itself the core code is useless. Cannot even be built as the controlling Kconfig option "CONFIG_RESCTRL2_FS" must be invoked by a "select" request from architecture specific code that provides the necessary "arch_*()" functions to make everything work.
Module registration is handled in fs/resctrl2/resources.c and can be done before or after mounting /sys/fs/resctrl. Current code won't let you make any new resource groups until a module implementing a control function is loaded to supply the information on how many groups the architecture supports.
Second patch is all the Intel X86 code (with some of the AMD bits included, but by no means all of them).
I've implemented modules for most of the legacy Intel control and monitor functions. Many of these share common code (by means of a symlinked source file ... I couldn't figure out how to make Kconfig build both rdt_l3_cat.ko and rdt_l3_cdp.ko from the same source file with a different set of $(CFLAGS)).
Users can pick which features they want by loading modules that implement the bits they want. E.g. CDP is enabled by loading that rdt_l3_cdp.ko module instead of rdt_l3_cat.ko (there's some code to prevent both being loaded together).
I started on the hooks for the "mba_MBps" feedback from MBM driver, but in this code drop I just have a simple module that reports the bandwidth for each group instead of the byte count. I just need to create a module that has both MBA control and MBM monitoring resources with a periodic comparison of actual bandwidth with desired, that then tweaks the MBA controls up/down as needed.
I haven't ventured to read all the pseudo-locking code, but it looks as though providing the driver with a way to tell core code that a group is exclusive instead of shared (which tells core code not to allow assignment of tasks or CPUs to the group) may be all the surgery needed to core code. The x86 module will be more complex that the toys I've produced so far, but should be able to leverage much from the existing resctrl implementation.
Tony Luck (2):
resctrl2: Add all the generic code
resctrl2: Arch x86 modules for most of the legacy control/monitor
functions
include/linux/resctrl.h | 107 +++++ include/linux/sched.h | 3 + arch/x86/include/asm/resctrl.h | 38 ++ fs/resctrl2/arch/x86/rdt.h | 22 + fs/resctrl2/internal.h | 110 +++++ arch/x86/kernel/cpu/amd.c | 3 + arch/x86/kernel/cpu/intel.c | 3 + arch/x86/kernel/process_32.c | 1 + arch/x86/kernel/process_64.c | 3 + fs/resctrl2/arch/x86/alloc.c | 119 +++++ fs/resctrl2/arch/x86/rdt_l2_cat.c | 1 + fs/resctrl2/arch/x86/rdt_l2_cdp.c | 1 + fs/resctrl2/arch/x86/rdt_l3_cat.c | 349 +++++++++++++++ fs/resctrl2/arch/x86/rdt_l3_cdp.c | 1 + fs/resctrl2/arch/x86/rdt_l3_mba.c | 251 +++++++++++ fs/resctrl2/arch/x86/rdt_llc_occupancy.c | 100 +++++ fs/resctrl2/arch/x86/rdt_mbm_adjust.c | 91 ++++ fs/resctrl2/arch/x86/rdt_mbm_local_bytes.c | 1 + fs/resctrl2/arch/x86/rdt_mbm_local_rate.c | 1 + fs/resctrl2/arch/x86/rdt_mbm_total_bytes.c | 1 + fs/resctrl2/arch/x86/rdt_mbm_total_rate.c | 1 + fs/resctrl2/arch/x86/rdt_monitor.c | 491 +++++++++++++++++++++ fs/resctrl2/cpu.c | 315 +++++++++++++ fs/resctrl2/directory.c | 295 +++++++++++++ fs/resctrl2/domain.c | 99 +++++ fs/resctrl2/info.c | 99 +++++ fs/resctrl2/kernfs.c | 58 +++ fs/resctrl2/locking.c | 52 +++ fs/resctrl2/resources.c | 85 ++++ fs/resctrl2/root.c | 173 ++++++++ fs/resctrl2/schemata.c | 110 +++++ fs/resctrl2/tasks.c | 193 ++++++++ arch/x86/Kconfig | 81 +++- fs/Kconfig | 1 + fs/Makefile | 1 + fs/resctrl2/Kconfig | 5 + fs/resctrl2/Makefile | 14 + fs/resctrl2/arch/x86/Makefile | 29 ++ 38 files changed, 3306 insertions(+), 2 deletions(-) create mode 100644 fs/resctrl2/arch/x86/rdt.h create mode 100644 fs/resctrl2/internal.h create mode 100644 fs/resctrl2/arch/x86/alloc.c create mode 120000 fs/resctrl2/arch/x86/rdt_l2_cat.c create mode 120000 fs/resctrl2/arch/x86/rdt_l2_cdp.c create mode 100644 fs/resctrl2/arch/x86/rdt_l3_cat.c create mode 120000 fs/resctrl2/arch/x86/rdt_l3_cdp.c create mode 100644 fs/resctrl2/arch/x86/rdt_l3_mba.c create mode 100644 fs/resctrl2/arch/x86/rdt_llc_occupancy.c create mode 100644 fs/resctrl2/arch/x86/rdt_mbm_adjust.c create mode 120000 fs/resctrl2/arch/x86/rdt_mbm_local_bytes.c create mode 120000 fs/resctrl2/arch/x86/rdt_mbm_local_rate.c create mode 120000 fs/resctrl2/arch/x86/rdt_mbm_total_bytes.c create mode 120000 fs/resctrl2/arch/x86/rdt_mbm_total_rate.c create mode 100644 fs/resctrl2/arch/x86/rdt_monitor.c create mode 100644 fs/resctrl2/cpu.c create mode 100644 fs/resctrl2/directory.c create mode 100644 fs/resctrl2/domain.c create mode 100644 fs/resctrl2/info.c create mode 100644 fs/resctrl2/kernfs.c create mode 100644 fs/resctrl2/locking.c create mode 100644 fs/resctrl2/resources.c create mode 100644 fs/resctrl2/root.c create mode 100644 fs/resctrl2/schemata.c create mode 100644 fs/resctrl2/tasks.c create mode 100644 fs/resctrl2/Kconfig create mode 100644 fs/resctrl2/Makefile create mode 100644 fs/resctrl2/arch/x86/Makefile
base-commit: 45a3e24f65e90a047bef86f927ebdc4c710edaa1
Signature · 1 line
2.40.1
Tony Luck
maintainerRFC PATCH 1/2 resctrl2: Add all the generic code
Generic resctrly code is responsible for:
0) Mounting and unmounting /sys/fs/resctrl
1) Creating and removing groups
2) Associating tasks with groups
3) Associating CPUs with groups
4) Context switch, calling architecture code to begin control/monitor for the nex task based on which groups it is in, and which CPU it is running on.
5) Registering and unregistering architecture modules for all control and monitor functions.
N.B. Much of this code is just the old resctrl code with a bunch of renaming and refactoring.
--- include/linux/resctrl.h | 107 ++++++++++++++ include/linux/sched.h | 3 + fs/resctrl2/internal.h | 110 ++++++++++++++ fs/resctrl2/cpu.c | 315 ++++++++++++++++++++++++++++++++++++++++ fs/resctrl2/directory.c | 295 +++++++++++++++++++++++++++++++++++++ fs/resctrl2/domain.c | 99 +++++++++++++ fs/resctrl2/info.c | 99 +++++++++++++ fs/resctrl2/kernfs.c | 58 ++++++++ fs/resctrl2/locking.c | 52 +++++++ fs/resctrl2/resources.c | 85 +++++++++++ fs/resctrl2/root.c | 173 ++++++++++++++++++++++ fs/resctrl2/schemata.c | 110 ++++++++++++++ fs/resctrl2/tasks.c | 193 ++++++++++++++++++++++++ fs/Kconfig | 1 + fs/Makefile | 1 + fs/resctrl2/Kconfig | 5 + fs/resctrl2/Makefile | 14 ++ 17 files changed, 1720 insertions(+) create mode 100644 fs/resctrl2/internal.h create mode 100644 fs/resctrl2/cpu.c create mode 100644 fs/resctrl2/directory.c create mode 100644 fs/resctrl2/domain.c create mode 100644 fs/resctrl2/info.c create mode 100644 fs/resctrl2/kernfs.c create mode 100644 fs/resctrl2/locking.c create mode 100644 fs/resctrl2/resources.c create mode 100644 fs/resctrl2/root.c create mode 100644 fs/resctrl2/schemata.c create mode 100644 fs/resctrl2/tasks.c create mode 100644 fs/resctrl2/Kconfig create mode 100644 fs/resctrl2/Makefile
include/linux/resctrl.h +107 -0
@@ -2,6 +2,8 @@#ifndef _RESCTRL_H#define _RESCTRL_H +#ifdef CONFIG_X86_CPU_RESCTRL +#include <linux/kernel.h> #include <linux/list.h> #include <linux/pid.h>
@@ -264,4 +266,109 @@ void resctrl_arch_reset_rmid_all(struct rdt_resource *r, struct rdt_domain *d);extern unsigned int resctrl_rmid_realloc_threshold;extern unsigned int resctrl_rmid_realloc_limit; +#endif /* CONFIG_X86_CPU_RESCTRL */ + +#ifdef CONFIG_RESCTRL2_FS + +#include <linux/sched.h> +#include <linux/jump_label.h> + +bool arch_check_resctrl_support (void ); +void arch_resctrl_apply_ids (u64 resctrl_ids); +extern u64 arch_resctrl_default_ids; + +DECLARE_STATIC_KEY_FALSE (resctrl_enable_key); + +struct resctrl_per_cpu_state { +u64 cached_resctrl_ids; +u64 default_resctrl_ids; +}; + +DECLARE_PER_CPU(struct resctrl_per_cpu_state, resctrl_per_cpu_state); + +static inline void resctrl_sched_in (struct task_struct *tsk) +{ +struct resctrl_per_cpu_state *state; +u64 new_resctrl_ids; + +if (!static_branch_likely (&resctrl_enable_key)) +return ; + + state =this_cpu_ptr (&resctrl_per_cpu_state); + new_resctrl_ids = state->default_resctrl_ids; + +if (tsk->resctrl_ids != arch_resctrl_default_ids) + new_resctrl_ids = tsk->resctrl_ids; + +if (new_resctrl_ids != state->cached_resctrl_ids) { + state->cached_resctrl_ids = new_resctrl_ids; +arch_resctrl_apply_ids (new_resctrl_ids); + } +} + +/* Unclear if this is still useful */ +static inline void resctrl_cpu_detect (struct cpuinfo_x86 *c) {} + +enum resctrl_type { +RESCTRL_CONTROL , +RESCTRL_MONITOR , +}; + +enum resctrl_scope { +RESCTRL_CORE , +RESCTRL_L2CACHE , +RESCTRL_L3CACHE , +RESCTRL_SOCKET , +}; + +enum resctrl_domain_update { +RESCTRL_DOMAIN_ADD , +RESCTRL_DOMAIN_ADD_CPU , +RESCTRL_DOMAIN_DELETE_CPU , +RESCTRL_DOMAIN_DELETE , +}; + +struct resctrl_domain { +struct list_head list; +struct cpumask cpu_mask; +int id; +}; + +struct resctrl_fileinfo { +char *name; +struct kernfs_ops *ops; +void *priv; +}; + +struct resctrl_resource { +char *name; +int archtag; +struct list_head list; +int type; +enum resctrl_scope scope; +size_t domain_size; +struct list_head domains; +void (*domain_update)(struct resctrl_resource *r,int what,int cpu,struct resctrl_domain *d); + +char *infodir; +struct resctrl_fileinfo *infofiles; + +// bits for control resources +int num_alloc_ids; +void (*show)(struct resctrl_resource *r,struct seq_file *m,u64 resctrl_ids); +void (*resetstaging)(struct resctrl_resource *r,u64 resctrl_ids); +int (*parse)(struct resctrl_resource *r,char *tok,u64 resctrl_ids); +void (*applychanges)(struct resctrl_resource *r,u64 resctrl_ids); + +// bits for monitor resources +char *mon_domain_dir; +char *mon_domain_file; +struct kernfs_ops *mod_domain_ops; +int mon_event; +}; + +int resctrl_register_ctrl_resource (struct resctrl_resource *r); +void resctrl_unregister_ctrl_resource (struct resctrl_resource *r); + +#endif /* CONFIG_RESCTRL2_FS */ #endif /* _RESCTRL_H */
include/linux/sched.h +3 -0
@@ -1228,6 +1228,9 @@ struct task_struct {u32 closid;u32 rmid;#endif +#ifdef CONFIG_RESCTRL2_FS +u64 resctrl_ids; +#endif #ifdef CONFIG_FUTEX struct robust_list_head __user *robust_list;#ifdef CONFIG_COMPAT
fs/resctrl2/internal.h new file +110 -0
110 lines not on this page · raw message
fs/resctrl2/cpu.c new file +315 -0
315 lines not on this page · raw message
fs/resctrl2/directory.c new file +295 -0
295 lines not on this page · raw message
fs/resctrl2/domain.c new file +99 -0
99 lines not on this page · raw message
fs/resctrl2/info.c new file +99 -0
99 lines not on this page · raw message
fs/resctrl2/kernfs.c new file +58 -0
58 lines not on this page · raw message
fs/resctrl2/locking.c new file +52 -0
52 lines not on this page · raw message
fs/resctrl2/resources.c new file +85 -0
85 lines not on this page · raw message
fs/resctrl2/root.c new file +173 -0
173 lines not on this page · raw message
fs/resctrl2/schemata.c new file +110 -0
110 lines not on this page · raw message
fs/resctrl2/tasks.c new file +193 -0
193 lines not on this page · raw message
fs/Kconfig +1 -0
@@ -45,6 +45,7 @@ source "fs/btrfs/Kconfig" source "fs/nilfs2/Kconfig" source "fs/f2fs/Kconfig" source "fs/zonefs/Kconfig" +source "fs/resctrl2/Kconfig" endif # BLOCK
fs/Makefile +1 -0
@@ -135,3 +135,4 @@ obj-$(CONFIG_EFIVAR_FS) += efivarfs/ obj-$(CONFIG_EROFS_FS) += erofs/ obj-$(CONFIG_VBOXSF_FS) += vboxsf/ obj-$(CONFIG_ZONEFS_FS) += zonefs/ +obj-$(CONFIG_RESCTRL2_FS) += resctrl2/
fs/resctrl2/Kconfig new file +5 -0
@@ -0,0 +1,5 @@ +# +# Architectures that support resource control will select this +# +config RESCTRL2_FS + bool
fs/resctrl2/Makefile new file +14 -0
@@ -0,0 +1,14 @@ +# SPDX-License-Identifier: GPL-2.0 + +obj-$(CONFIG_RESCTRL2_FS) += cpu.o +obj-$(CONFIG_RESCTRL2_FS) += directory.o +obj-$(CONFIG_RESCTRL2_FS) += domain.o +obj-$(CONFIG_RESCTRL2_FS) += info.o +obj-$(CONFIG_RESCTRL2_FS) += kernfs.o +obj-$(CONFIG_RESCTRL2_FS) += locking.o +obj-$(CONFIG_RESCTRL2_FS) += resources.o +obj-$(CONFIG_RESCTRL2_FS) += root.o +obj-$(CONFIG_RESCTRL2_FS) += schemata.o +obj-$(CONFIG_RESCTRL2_FS) += tasks.o + +obj-$(CONFIG_X86_CPU_RESCTRL2) += arch/x86/
Signature · 1 line
2.40.1
Tony Luck
maintainerRFC PATCH 2/2 resctrl2: Arch x86 modules for most of the legacy control/monitor functions
In reply to Tony LuckMissing:
1) "mba_MBps" feedback from MBM
2) Pseudo-locking
3) AMD SMBA/BMEC
4) Fixes and additions after v6.4-rc6
5) Other stuff I haven't noticed
N.B. This is proof-of-concept code ... the locking (or lack thereof) for most of the files controlled by modules needs to be addressed.
--- arch/x86/include/asm/resctrl.h | 38 ++ fs/resctrl2/arch/x86/rdt.h | 22 + arch/x86/kernel/cpu/amd.c | 3 + arch/x86/kernel/cpu/intel.c | 3 + arch/x86/kernel/process_32.c | 1 + arch/x86/kernel/process_64.c | 3 + fs/resctrl2/arch/x86/alloc.c | 119 +++++ fs/resctrl2/arch/x86/rdt_l2_cat.c | 1 + fs/resctrl2/arch/x86/rdt_l2_cdp.c | 1 + fs/resctrl2/arch/x86/rdt_l3_cat.c | 349 +++++++++++++++ fs/resctrl2/arch/x86/rdt_l3_cdp.c | 1 + fs/resctrl2/arch/x86/rdt_l3_mba.c | 251 +++++++++++ fs/resctrl2/arch/x86/rdt_llc_occupancy.c | 100 +++++ fs/resctrl2/arch/x86/rdt_mbm_adjust.c | 91 ++++ fs/resctrl2/arch/x86/rdt_mbm_local_bytes.c | 1 + fs/resctrl2/arch/x86/rdt_mbm_local_rate.c | 1 + fs/resctrl2/arch/x86/rdt_mbm_total_bytes.c | 1 + fs/resctrl2/arch/x86/rdt_mbm_total_rate.c | 1 + fs/resctrl2/arch/x86/rdt_monitor.c | 491 +++++++++++++++++++++ arch/x86/Kconfig | 81 +++- fs/resctrl2/arch/x86/Makefile | 29 ++ 21 files changed, 1586 insertions(+), 2 deletions(-) create mode 100644 fs/resctrl2/arch/x86/rdt.h create mode 100644 fs/resctrl2/arch/x86/alloc.c create mode 120000 fs/resctrl2/arch/x86/rdt_l2_cat.c create mode 120000 fs/resctrl2/arch/x86/rdt_l2_cdp.c create mode 100644 fs/resctrl2/arch/x86/rdt_l3_cat.c create mode 120000 fs/resctrl2/arch/x86/rdt_l3_cdp.c create mode 100644 fs/resctrl2/arch/x86/rdt_l3_mba.c create mode 100644 fs/resctrl2/arch/x86/rdt_llc_occupancy.c create mode 100644 fs/resctrl2/arch/x86/rdt_mbm_adjust.c create mode 120000 fs/resctrl2/arch/x86/rdt_mbm_local_bytes.c create mode 120000 fs/resctrl2/arch/x86/rdt_mbm_local_rate.c create mode 120000 fs/resctrl2/arch/x86/rdt_mbm_total_bytes.c create mode 120000 fs/resctrl2/arch/x86/rdt_mbm_total_rate.c create mode 100644 fs/resctrl2/arch/x86/rdt_monitor.c create mode 100644 fs/resctrl2/arch/x86/Makefile
arch/x86/include/asm/resctrl.h +38 -0
@@ -96,6 +96,44 @@ static inline void resctrl_sched_in(struct task_struct *tsk)void resctrl_cpu_detect (struct cpuinfo_x86 *c); +#elif defined(CONFIG_X86_CPU_RESCTRL2 ) + +bool arch_alloc_resctrl_ids (struct resctrl_group *rg); +void arch_free_resctrl_ids (struct resctrl_group *rg); +bool arch_init_alloc_ids (struct resctrl_resource *r); +int rmid_alloc (int prmid); +void rmid_free (int rmid); +void arch_add_monitor (int mon_event); +void arch_del_monitor (int mon_event); +u64 rdt_rmid_read (int domain_id,int rmid,int event); +void rdt_mbm_apply_quirk (int num_rmids); +u64 get_corrected_mbm_count (u32 rmid,unsigned long val); + +static inline bool is_closid_match (struct task_struct *t,struct resctrl_group *rg) +{ +return (t->resctrl_ids >>32 ) == (rg->resctrl_ids >>32 ); +} + +static inline bool arch_is_resctrl_id_match (struct task_struct *t,struct resctrl_group *rg) +{ +if (rg->type ==DIR_MON ) +return t->resctrl_ids == rg->resctrl_ids; +return is_closid_match (t, rg); +} + +static inline bool arch_set_task_ids (struct task_struct *t,struct resctrl_group *rg) +{ +if (rg->type ==DIR_MON ) { +if (!is_closid_match (t, rg)) { +//rdt_last_cmd_puts("Can't move task to different control group\n"); +return false; + } + } + +WRITE_ONCE (t->resctrl_ids, rg->resctrl_ids); + +return true; +}#else static inline void resctrl_sched_in (struct task_struct *tsk) {}
fs/resctrl2/arch/x86/rdt.h new file +22 -0
@@ -0,0 +1,22 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* Copyright(c) 2023 Intel Corporation. */ + +/* H/W supported RDT monitor events */ +#define EV_LLC 1 +#define EV_TOT 2 +#define EV_LOC 3 +#define EV_MAX 4 + +/* S/W events */ +#define EV_TOTRATE 4 +#define EV_LOCRATE 5 + +#define RESCTRL_FILE_DEF (X , fmt) \ +static int X##_show(struct seq_file *sf, void *v) \ +{ \ + seq_printf(sf, fmt, X); \ + return 0; \ +} \ +static struct kernfs_ops X##_ops = { \ + .seq_show = X##_show \ +};
arch/x86/kernel/cpu/amd.c +3 -0
@@ -9,6 +9,7 @@#include <linux/sched/clock.h> #include <linux/random.h> #include <linux/topology.h> +#include <linux/resctrl.h> #include <asm/processor.h> #include <asm/apic.h> #include <asm/cacheinfo.h>
@@ -19,7 +20,9 @@#include <asm/pci-direct.h> #include <asm/delay.h> #include <asm/debugreg.h> +#ifndef CONFIG_RESCTRL2_FS #include <asm/resctrl.h> +#endif #ifdef CONFIG_X86_64 # include <asm/mmconfig.h>
arch/x86/kernel/cpu/intel.c +3 -0
@@ -14,6 +14,7 @@#include <linux/workqueue.h> #include <linux/delay.h> #include <linux/cpuhotplug.h> +#include <linux/resctrl.h> #include <asm/cpufeature.h> #include <asm/msr.h>
@@ -26,7 +27,9 @@#include <asm/cpu_device_id.h> #include <asm/cmdline.h> #include <asm/traps.h> +#ifndef CONFIG_RESCTRL2_FS #include <asm/resctrl.h> +#endif #include <asm/numa.h> #include <asm/thermal.h>
arch/x86/kernel/process_32.c +1 -0
@@ -38,6 +38,7 @@#include <linux/io.h> #include <linux/kdebug.h> #include <linux/syscalls.h> +#include <linux/resctrl.h> #include <asm/ldt.h> #include <asm/processor.h>
arch/x86/kernel/process_64.c +3 -0
@@ -40,6 +40,7 @@#include <linux/ftrace.h> #include <linux/syscalls.h> #include <linux/iommu.h> +#include <linux/resctrl.h> #include <asm/processor.h> #include <asm/pkru.h>
@@ -53,7 +54,9 @@#include <asm/switch_to.h> #include <asm/xen/hypervisor.h> #include <asm/vdso.h> +#ifndef CONFIG_RESCTRL2_FS #include <asm/resctrl.h> +#endif #include <asm/unistd.h> #include <asm/fsgsbase.h> #ifdef CONFIG_IA32_EMULATION
fs/resctrl2/arch/x86/alloc.c new file +119 -0
119 lines not on this page · raw message
fs/resctrl2/arch/x86/rdt_l2_cat.c new file +1 -0
@@ -0,0 +1 @@ +rdt_l3_cat.c
fs/resctrl2/arch/x86/rdt_l2_cdp.c new file +1 -0
@@ -0,0 +1 @@ +rdt_l3_cat.c
fs/resctrl2/arch/x86/rdt_l3_cat.c new file +349 -0
349 lines not on this page · raw message
fs/resctrl2/arch/x86/rdt_l3_cdp.c new file +1 -0
@@ -0,0 +1 @@ +rdt_l3_cat.c
fs/resctrl2/arch/x86/rdt_l3_mba.c new file +251 -0
251 lines not on this page · raw message
fs/resctrl2/arch/x86/rdt_llc_occupancy.c new file +100 -0
100 lines not on this page · raw message
fs/resctrl2/arch/x86/rdt_mbm_adjust.c new file +91 -0
91 lines not on this page · raw message
fs/resctrl2/arch/x86/rdt_mbm_local_bytes.c new file +1 -0
@@ -0,0 +1 @@ +rdt_llc_occupancy.c
fs/resctrl2/arch/x86/rdt_mbm_local_rate.c new file +1 -0
@@ -0,0 +1 @@ +rdt_llc_occupancy.c
fs/resctrl2/arch/x86/rdt_mbm_total_bytes.c new file +1 -0
@@ -0,0 +1 @@ +rdt_llc_occupancy.c
fs/resctrl2/arch/x86/rdt_mbm_total_rate.c new file +1 -0
@@ -0,0 +1 @@ +rdt_llc_occupancy.c
fs/resctrl2/arch/x86/rdt_monitor.c new file +491 -0
491 lines not on this page · raw message
arch/x86/Kconfig +79 -2
95 lines not on this page · raw message
fs/resctrl2/arch/x86/Makefile new file +29 -0
@@ -0,0 +1,29 @@ +# SPDX-License-Identifier: GPL-2.0 + +obj-$(CONFIG_X86_CPU_RESCTRL2) += alloc.o +obj-$(CONFIG_X86_CPU_RESCTRL2) += rdt_monitor.o +obj-$(CONFIG_X86_CPU_RESCTRL2) += rdt_mbm_adjust.o + +CFLAGS_rdt_l3_cat.o += -DCACHE_LEVEL=3 +CFLAGS_rdt_l2_cat.o += -DCACHE_LEVEL=2 +CFLAGS_rdt_l3_cdp.o += -DCACHE_LEVEL=3 -DCDP +CFLAGS_rdt_l2_cdp.o += -DCACHE_LEVEL=2 -DCDP + +obj-$(CONFIG_X86_RDT_L3_CAT) += rdt_l3_cat.o +obj-$(CONFIG_X86_RDT_L2_CAT) += rdt_l2_cat.o +obj-$(CONFIG_X86_RDT_L3_CDP) += rdt_l3_cdp.o +obj-$(CONFIG_X86_RDT_L2_CDP) += rdt_l2_cdp.o + +CFLAGS_rdt_llc_occupancy.o += -DEVENT=1 +CFLAGS_rdt_mbm_total_bytes.o += -DEVENT=2 +CFLAGS_rdt_mbm_local_bytes.o += -DEVENT=3 +CFLAGS_rdt_mbm_total_rate.o += -DEVENT=4 +CFLAGS_rdt_mbm_local_rate.o += -DEVENT=5 + +obj-$(CONFIG_X86_RDT_LLC_OCCUPANCY) += rdt_llc_occupancy.o +obj-$(CONFIG_X86_RDT_MBM_TOTAL_BYTES) += rdt_mbm_total_bytes.o +obj-$(CONFIG_X86_RDT_MBM_LOCAL_BYTES) += rdt_mbm_local_bytes.o +obj-$(CONFIG_X86_RDT_MBM_TOTAL_RATE) += rdt_mbm_total_rate.o +obj-$(CONFIG_X86_RDT_MBM_LOCAL_RATE) += rdt_mbm_local_rate.o + +obj-$(CONFIG_X86_RDT_L3_MBA) += rdt_l3_mba.o
Signature · 1 line
2.40.1
Luck, Tony
maintainerQuoted 1 line
38 files changed, 3306 insertions(+), 2 deletions(-)
Lest this be too scary, I'll note the original resctrl code looks like:
$ find arch/x86/kernel/cpu/resctrl -type f | xargs wc -l
43 arch/x86/kernel/cpu/resctrl/pseudo_lock_event.h
4 arch/x86/kernel/cpu/resctrl/Makefile
996 arch/x86/kernel/cpu/resctrl/core.c
581 arch/x86/kernel/cpu/resctrl/ctrlmondata.c
560 arch/x86/kernel/cpu/resctrl/internal.h
845 arch/x86/kernel/cpu/resctrl/monitor.c
1600 arch/x86/kernel/cpu/resctrl/pseudo_lock.c
3733 arch/x86/kernel/cpu/resctrl/rdtgroup.c
8362 totalI'm haven't included pseudo_lock ... but I have most of the rest, so I'm confident this will end up as a net reduction in lines of code.
-Tony
Quoted 39 lines · Tony Luck
Back in April I posted some RFC patches that added a "driver registration" interface to the core resctrl code so that additional resource control and monitor features could be added without further complicating the core code. Link to that discussion:
https://lore.kernel.org/all/[email protected]/
Reinette gave the feedback that it would be better to base the module registration on the resctrl resource structure. Reinette also pointed me to work from James Morse, and some additional discussion happened here:
https://lore.kernel.org/all/ZG%2FMZVrWYrCHm%2Ffr@agluck-desk3/
James provided details on where ARM's MPAM has similarities and differences from the Intel Resource Director Technology and AMD's similar implementation. Drew Fustini was also pulled into that conversation to comment on RISC-V CBQRI.
Quoted 1 line
From those discussions I believed we need a do-over on the core
/sys/fs/resctrlimplementation to make it friendlier for architecural variations. Here's what I have so far.=========================================================================
| N.B. This is a general direction check. There are many obvious | | rough edges (e.g. some careful thought needs to happen on locking | | for the files in /sys/fs/resctrl that are "owned" by modules that | | can be unloaded). I'm mostly looking for feedback from AMD, ARM and | | RISCV on whether this is a foundation to build on, whether some small | | tweaks could make it better, or if this is still going to be really | | hard for architectures that have radical divergence from the Intel | | model. |=========================================================================
First patch is my attempt at architecture neutral code. All mention of "RDT", "CLOSID" and "RMID" have been expunged. When creating a new group this code calls
arch_alloc_resctrl_ids()to allocate an opaque "resctrl_ids" value.
Quoted 6 lines
Q: I made this a "u64" because that neatly allows storage of both an x86 CLOSID and RMID (in a handy representation that matches the bit layout of the Intel
IA32_PQR_ASSOCmodel specific register). If other architectures need something more complex it could be a "typedefresctrl_id_t" ... there are a couple of places where we would need a comparison function.
This works okay for RISC-V. The Ssqosid extension defines a 32-bit register sqoscfg (see chapter 2 of CBQRI spec [0]). This contains a 12-bit MCID field (similar to an RMID) and 12-bit RCID field (similar to an CLOSID).
Quoted 11 lines
I broke the code into several source files that handle different sub-functions of core code to make it easier to navigate. Much of the code here should look familiar as I did a lot of s/rdtgroup/resctrl_group/ on functions from the original resctrl code.
By itself the core code is useless. Cannot even be built as the controlling
Kconfigoption "CONFIG_RESCTRL2_FS" must be invoked by a "select" request from architecture specific code that provides the necessary "arch_*()" functions to make everything work.
I would like to try to rebase the RISC-V CBQRI resctrl RFC [1] on top of this patch series instead of the mpam snapshot branch [2].
I had a patch in my RFC that added config option RISCV_ISA_SSQOSID which selects ARCH_HAS_CPU_RESCTRL and RESCTRL_FS [3]. It seems I would need to change that to select CONFIG_RESCTRL2_FS ?
A patch [4] in that RFC adds the "arch_*()" functions in arch/riscv/kernel/qos/qos_resctrl.c
thanks,
drew
[0] https://github.com/riscv-non-isa/riscv-cbqri/blob/main/riscv-cbqri.pdf
[1] https://lore.kernel.org/linux-riscv/[email protected]/
[2] https://gitlab.arm.com/linux-arm/linux-jm/-/tree/mpam/snaphot/20230406
[3] https://lore.kernel.org/linux-riscv/[email protected]/
[4] https://lore.kernel.org/linux-riscv/[email protected]/
Luck, Tony
maintainerQuoted 7 lines
Quoted 4 lines
By itself the core code is useless. Cannot even be built as the controlling
Kconfigoption "CONFIG_RESCTRL2_FS" must be invoked by a "select" request from architecture specific code that provides the necessary "arch_*()" functions to make everything work.I would like to try to rebase the RISC-V CBQRI resctrl RFC [1] on top of this patch series instead of the mpam snapshot branch [2].
Thanks. That should help shake out any places where I've left in Intel-isms, or my abstraction is insufficient to handle your architecture.
I've made some significant changes since I posted those patches. I pushed the latest version to:
git://git.kernel.org/pub/scm/linux/kernel/git/aegl/linux.git resctrl2_v64
Quoted 3 lines
I had a patch in my RFC that added config option
RISCV_ISA_SSQOSIDwhich selectsARCH_HAS_CPU_RESCTRLandRESCTRL_FS[3]. It seems I would need to change that to selectCONFIG_RESCTRL2_FS?
Yes. Just have your architecture CONFIG option select RESCTRL2_FS
Quoted 2 lines
A patch [4] in that RFC adds the "arch_*()" functions in
arch/riscv/kernel/qos/qos_resctrl.c
Yes. This is an area that may need some tweaking to get the prototypes for the arch_*() functions right.
I put all the x86 architecture code under fs/resctrl2/arch/x86/ .... mostly so I can do quick test builds of both the common code and architecture code with "make fs/resctrl2/" ... maybe in the end-game they should be under arch/x86 rather than adding arch specific subdirs under generic top-level directories (though I see a smattering of "x86" directories in several places.
-Tony
Hi Tony,
Quoted 33 lines · Tony Luck
Back in April I posted some RFC patches that added a "driver registration" interface to the core resctrl code so that additional resource control and monitor features could be added without further complicating the core code. Link to that discussion:
https://lore.kernel.org/all/[email protected]/
Reinette gave the feedback that it would be better to base the module registration on the resctrl resource structure. Reinette also pointed me to work from James Morse, and some additional discussion happened here:
https://lore.kernel.org/all/ZG%2FMZVrWYrCHm%2Ffr@agluck-desk3/
James provided details on where ARM's MPAM has similarities and differences from the Intel Resource Director Technology and AMD's similar implementation. Drew Fustini was also pulled into that conversation to comment on RISC-V CBQRI.
From those discussions I believed we need a do-over on the core
/sys/fs/resctrlimplementation to make it friendlier for architecural variations. Here's what I have so far.=========================================================================
| N.B. This is a general direction check. There are many obvious | | rough edges (e.g. some careful thought needs to happen on locking | | for the files in /sys/fs/resctrl that are "owned" by modules that | | can be unloaded). I'm mostly looking for feedback from AMD, ARM and | | RISCV on whether this is a foundation to build on, whether some small | | tweaks could make it better, or if this is still going to be really | | hard for architectures that have radical divergence from the Intel | | model. |
Quoted 1 line
=========================================================================
Thanks for working on this! I played with these changes locally on some of our machines and they seemed reasonably functional so far and was happy to see dynamically adding and removing resources working.
I will need to try working with the code to give it a serious evaluation, though. Would you consider it ready for me to try re-implementing soft RMIDs on it?
I'm also very interested in James's opinion and what this means for the ongoing MPAM upstreaming.
Thanks!
-Peter
Luck, Tony
maintainerQuoted 3 lines
Thanks for working on this! I played with these changes locally on some of our machines and they seemed reasonably functional so far and was happy to see dynamically adding and removing resources working.
Thanks for taking it for a spin on some additional systems.
Quoted 3 lines
I will need to try working with the code to give it a serious evaluation, though. Would you consider it ready for me to try re-implementing soft RMIDs on it?
Current code is:
1) Lacking locking for access to files created on behalf of dynamic loaded modules. So, I expect the system to crash if you unload a module while simultaneously accessing one of those files.
2) Lacking error checking and cleanup code paths to undo partial operations when things fail to allocate.
3) The unmount (sb_kill()) code has fallen far behind development of other features. So expect memory leaks if you unmount and remount.
It should be OK to play around with this version, but things will go wrong when the system is under stress. Do not use in production!!
All of the RMID allocation and understanding of hierarchy between control and monitor groups is localized in fs/resctrl2/arch/x86/rdt_monitor.c I think I'm mostly done with the functionality I need in that file, so work you do there for soft RMIDs is unlikely to need refactoring for other changes that I'm making.
Quoted 2 lines
I'm also very interested in James's opinion and what this means for the ongoing MPAM upstreaming.
Me too. I'm hopeful that my code can be a better base than the legacy resctrl code. But it needs an MPAM expert to really assess that.
-Tony
Tony Luck
maintainerQuoted 4 lines · Tony
I've made some significant changes since I posted those patches. I pushed the latest version to:
git://git.kernel.org/pub/scm/linux/kernel/git/aegl/linux.gitresctrl2_v64
I just pushed one big commit with all the bits I've updated so far this week. Fixes some serious issues as well as general cleanup.
HEAD is now:
afb7cdd4d640 resctrl2: Many cleanups, fixes, and new functionality
If you've started writing your own architecture specific modules there are some small interface changes. Most should be found by the compiler barfing, but the new ".reset()" resource function called during unmount of /sys/fs/resctrl might be less obvious.
-Tony
Re: [RFC PATCH 2/2] resctrl2: Arch x86 modules for most of the legacy control/monitor functions
In reply to Tony LuckHi Tony,
Quoted 6 lines · Tony Luck
fs/resctrl2/arch/x86/rdt_monitor.c new file +6
-0+struct rmid { + struct list_head list; + struct list_head child_list; + bool is_parent; +static void __rdt_rmid_read(void *info) +{
fs/resctrl2/arch/x86/rdt_monitor.c line 107
Quoted 10 lines
fs/resctrl2/arch/x86/rdt_monitor.c new file +10
-0+struct rrmid_info *rr = info; +unsigned long flags; +struct rmid *cr, *r; +struct mydomain *m; +u64 chunks; + + m =get_mydomain (rr->domain); + +if (rr->event <=EV_LOC ) { +spin_lock_irqsave (&m->msr_lock, flags);
Will there ultimately be any locking at the filesystem layer? I recall from feedback on my change adding a spinlock here[1] before that the filesystem-layer locking took care of this.
Quoted 39 lines
fs/resctrl2/arch/x86/rdt_monitor.c new file +39
-0+wrmsrl (MSR_IA32_QM_EVTSEL , (rr->rmid <<32 ) | rr->event); +rdmsrl (MSR_IA32_QM_CTR , chunks); + }else { + chunks =0 ; + } + + rr->chunks =adjust (m, rr->rmid, rr->event, chunks); + + r = &rmid_array[rr->rmid]; +if (r->is_parent && !list_empty (&r->child_list)) { +list_for_each_entry (cr, &r->child_list, child_list) { +u64 crmid = cr - rmid_array; + +if (rr->event <=EV_LOC ) { +wrmsrl (MSR_IA32_QM_EVTSEL , (crmid <<32 ) | rr->event); +rdmsrl (MSR_IA32_QM_CTR , chunks); + }else { + chunks =0 ; + } + + rr->chunks +=adjust (m, crmid, rr->event, chunks); + } + } + +if (rr->event <=EV_LOC ) +spin_unlock_irqrestore (&m->msr_lock, flags); +} + +u64 rdt_rmid_read (int domain_id,int rmid,int event) +{ +struct resctrl_domain *d; +struct rrmid_info rr; +struct mydomain *m; + +list_for_each_entry (d, &monitor.domains, list) +if (d->id == domain_id) + goto found; +return ~0ull ; +found:
fs/resctrl2/arch/x86/rdt_monitor.c line 320
Quoted 10 lines
fs/resctrl2/arch/x86/rdt_monitor.c new file +10
-0+ m =get_mydomain (d); + + rr.domain = d; + rr.rmid = rmid; + rr.event = event; + +if (event <=EV_LOC ) +smp_call_function_any (&d->cpu_mask, __rdt_rmid_read, &rr,1 ); +else +__rdt_rmid_read (&rr);
I like that the driver is responsible for deciding where IPIs need to be sent, but it looks like the consequence is that RDT-level code wants to add in the child monitors' event counts once executing within the correct domain. The one-per-domain IPI assumption from the current resctrl code being wrong is probably harder to overcome than needing to figure out what additional RMIDs to read, but I'd really need to know the synchronization requirements for __rdt_rmid_read() to inspect the monitoring group hierarchy.
Would you continue to promise that the FS structure won't change during a monitor read? To us, the biggest priority for parallelization is reading all the domain-group combinations in the system, because we have a lot of them and want the tightest possible snapshot of bandwidth usage, broken down by group.
Thanks!
-Peter
Luck, Tony
maintainerRE: [RFC PATCH 2/2] resctrl2: Arch x86 modules for most of the legacy control/monitor functions
Quoted 6 lines
fs/resctrl2/arch/x86/rdt_monitor.c line 271
Quoted 2 lines
fs/resctrl2/arch/x86/rdt_monitor.c new file +2
-0+if (rr->event <=EV_LOC ) { +spin_lock_irqsave (&m->msr_lock, flags);Will there ultimately be any locking at the filesystem layer? I recall from feedback on my change adding a spinlock here[1] before that the filesystem-layer locking took care of this.
Yes. Working on that now. The spinlock will go away when everything is protected. by resctrl_mutex.
Quoted 5 lines
fs/resctrl2/arch/x86/rdt_monitor.c line 320
Quoted 4 lines
fs/resctrl2/arch/x86/rdt_monitor.c new file +4
-0+if (event <=EV_LOC ) +smp_call_function_any (&d->cpu_mask, __rdt_rmid_read, &rr,1 ); +else +__rdt_rmid_read (&rr);
Quoted 8 lines
I like that the driver is responsible for deciding where IPIs need to be sent, but it looks like the consequence is that RDT-level code wants to add in the child monitors' event counts once executing within the correct domain. The one-per-domain IPI assumption from the current resctrl code being wrong is probably harder to overcome than needing to figure out what additional RMIDs to read, but I'd really need to know the synchronization requirements for
__rdt_rmid_read()to inspect the monitoring group hierarchy.
There are likely some gaps in current code. But I think they should be fixable.
Quoted 5 lines
Would you continue to promise that the FS structure won't change during a monitor read? To us, the biggest priority for parallelization is reading all the domain-group combinations in the system, because we have a lot of them and want the tightest possible snapshot of bandwidth usage, broken down by group.
The mbm_poll() code that makes sure that counters don't wrap is doing all the expensive wrmsr(QM_EVTSEL);rdmsr(QM_COUNT) once per second to give you the data you want. But existing resctrl filesystem doesn't let you do a bulk read. I have some ideas on how to provide something better. One question: do you really need that snapshot to be system-wide? Or can you live with separate L3-scoped snapshots that aren't tightly synchronized with each other?
-Tony
Re: [RFC PATCH 2/2] resctrl2: Arch x86 modules for most of the legacy control/monitor functions
Hi Tony,
Quoted 3 lines · Tony
The
mbm_poll()code that makes sure that counters don't wrap is doing all the expensive wrmsr(QM_EVTSEL);rdmsr(QM_COUNT) once per second to give you the data you want.
I was doing that in the soft RMID series I posted earlier because it simplified things, but then I had some realizations about how much error +/- 1 second on the sampling point could result in[1]. We usually measure the bandwidth rate with a 5-second window, so a reading that's up to one second old would mean a 20% error in the bandwidth calculation.
Quoted 5 lines
But existing resctrl
filesystem doesn't let you do a bulk read. I have some ideas on how to provide something better. One question: do you really need that snapshot to be system-wide? Or can you live with separate L3-scoped snapshots that aren't tightly synchronized with each other?
The most demanding use case is responding to a bandwidth saturation emergency, where the admin wants a snapshot of all jobs' bandwidth rates to identify which one(s) to kill. Jobs aren't confined to L3 domains, so I expect it's more important to get a tight snapshot for an RMID-at-a-time. Going domain-at-a-time could create more races where a high-bandwidth job migrates at the right time and avoids detection.
One workaround we've been using to cut down on IPIs is providing a usermode threshold knob for limiting how old a reported event count can be, but this is more useful for soft RMIDs where reading a single CPU's counters makes all RMIDs stored counts current, and IPIs resulting from userspace reads of different groups in the same domain are all redundant.
We could probably work with second-old event counts if we knew the exact time they were recorded so we'd know the exact time delta for the bandwidth count delta, but that seems like it would be awkward in the current interface.
-Peter
[1] https://lore.kernel.org/lkml/CALPaoCi_N8JHbP4zFD=ijBh5-=4Q0Ec-LrspYNGGnj4G6A6U0g@mail.gmail.com/
Tony Luck
maintainerRe: [RFC PATCH 2/2] resctrl2: Arch x86 modules for most of the legacy control/monitor functions
Quoted 7 lines · Peter Newman
Hi Tony,
Quoted 3 lines · Tony
The
mbm_poll()code that makes sure that counters don't wrap is doing all the expensive wrmsr(QM_EVTSEL);rdmsr(QM_COUNT) once per second to give you the data you want.
Quoted 6 lines
I was doing that in the soft RMID series I posted earlier because it simplified things, but then I had some realizations about how much error +/- 1 second on the sampling point could result in[1]. We usually measure the bandwidth rate with a 5-second window, so a reading that's up to one second old would mean a 20% error in the bandwidth calculation.
I just pushed the latest version of the resctrl2 patches to the resctrl2_v65rc1 branch of git://git.kernel.org/pub/scm/linux/kernel/git/aegl/linux.git
As well as locking, bug fixes, and general updates it includes an experimental feature to provide summary MBM information for each node. E.g. (both "total" and "local" rates are provided). Note that you have to load modules rdt_mbm_local_bytes and rdt_mbm_total_bytes so that the MBM overflow threads are running. I should fix the code to print "n/a" instead of "0" if they are not.
$ cat /sys/fs/resctrl/mon_data/mon_L3_00/mbm_summary
3638 3638 /g2
3087 3087 /g2/m2
3267 3267 /g2/m1
3443 3443 /g1
3629 3629 /g1/m2
3588 3587 /g1/m1
3999 3993 /
3370 3369 /m2
3432 3432 /m1The rates are produced once per second by the MBM overflow code. They compute MBytes/sec as "chunks since last poll" divided by (now - then). I'm using jiffies for the times which may be good enough. "now - then" is one second (maybe more if the kernel thread doing the MBM polling is delayed from running).
I should fix the summarization code to work the same as the regular MBM files (i.e. make the parent control directory report the sum of all its children).
The code also attempts (but fails) to make these mbm_summary files poll(2)-able. With the wakeup dependent on aggregate measure bandwidth compared against a configurable threshold:
$ cat /sys/fs/resctrl/info/L3_MON/mbm_poll_threshold 10000000
There's something wrong though. Poll(2) always says there is data to be read. I only see one other piece of kernel code implementing poll on kernfs (in the cgroup code). Perhaps my problem is inability to write an appliction that uses poll(2) correctly.
Let me know if this all seems like a useful direction. Maybe the polling part is overkill and it is sufficient to just have a cheap way to get all the bandwidths even if the values seen might be up to one second old.
-Tony
Quoted 11 lines · Tony Luck
Quoted 4 lines · Tony
I've made some significant changes since I posted those patches. I pushed the latest version to:
git://git.kernel.org/pub/scm/linux/kernel/git/aegl/linux.gitresctrl2_v64I just pushed one big commit with all the bits I've updated so far this week. Fixes some serious issues as well as general cleanup.
HEAD is now:
Quoted 8 lines
afb7cdd4d640resctrl2: Many cleanups, fixes, and new functionalityIf you've started writing your own architecture specific modules there are some small interface changes. Most should be found by the compiler barfing, but the new ".reset()" resource function called during unmount of
/sys/fs/resctrlmight be less obvious.-Tony
I have access to a Xeon Silver 4310 machine which reports to have cat_l3, cqm_mbm_local, cqm_mbm_total and mba.
I would like to test resctrl2 on it so I can better understand how it works. I think that will help me understand how to adapt the RISC-V CBQRI resctrl proof-of-concept to use resctrl2.
Would you be able to provide an example of how you loaded the necessary resctrl2 kernel modules?
Also, is resctrl2_v65rc1 the latest to branch to test?
Thank you,
Drew
Tony Luck
maintainerQuoted 9 lines · Drew Fustini
I have access to a Xeon Silver 4310 machine which reports to have
cat_l3,cqm_mbm_local,cqm_mbm_totaland mba.I would like to test resctrl2 on it so I can better understand how it works. I think that will help me understand how to adapt the RISC-V CBQRI resctrl proof-of-concept to use resctrl2.
Would you be able to provide an example of how you loaded the necessary resctrl2 kernel modules?
Drew,
Sure. You simply mount the filesystem, and then load modules for whichever features you'd like to use. This will enable everything you list above:
# mount -t resctrl resctrl /sys/fs/resctrl # modprobe rdt_l3_cat # modprobe rdt_llc_occupancy # modprobe rdt_mbm_local_bytes # modprobe rdt_mbm_total_bytes # modprobe rdt_l3_mba
There are some experimental extras. E.g.
# modprobe rdt_mbm_total_rate # modprobe rdt_mbm_local_rate
Will each add an extra file to the mon_data directories to report the data rate in MB/s. The value reported is calculated by the once-per-second counter roll-over code in the kernel. So it might be up to one second out of date, but it is very cheap to read since it doesn't involve MSR access (or cross processor interrupts if you are reading from a CPU in a different scope).
You can unload modules without unmounting the filesystem and load different ones to get different data/control. E.g. to switch from L3CAT to L3CDP (which you don't list as supported, so this may not work for you:
# rmmod rdt_l3_cat # modprobe rdt_l3_cdp
Or to switch from the default MBA that uses percentages to specify throttling to the MBM->MBA feedback code that uses MB/s in the schemata file:
# rmmod rdt_l3_mba # modprobe rdt_l3_mba_MBps
Quoted 2 lines
Also, is
resctrl2_v65rc1the latest to branch to test?
Yes. That's the latest. There haven't been any updates for a few days because I'm working on a module to support pseudo-locking. I'm half-way there (can do most of the bits to set a group into pseudo-locked mode ... about to work on the cleanup when the group is removed, the filesystem unmounted, or the module unloaded).
-Tony
Tony Luck
maintainerQuoted 9 lines · Tony Luck
Quoted 2 lines · Drew Fustini
Also, is
resctrl2_v65rc1the latest to branch to test?Yes. That's the latest. There haven't been any updates for a few days because I'm working on a module to support pseudo-locking. I'm half-way there (can do most of the bits to set a group into pseudo-locked mode ... about to work on the cleanup when the group is removed, the filesystem unmounted, or the module unloaded).
Updated version available at:
git://git.kernel.org/pub/scm/linux/kernel/git/aegl/linux.git
Branch: resctrl2_v65rc4
Some minor fixes to core code, also changes to support pseudo-locking (core code for the "mode" file plus some new functions in the resource structure to call into modules to support this).
-Tony