x86/resctrl: Enable user to view thread or core throttling mode
Subsystems (3)
cover x86/resctrl: Enable user to view thread or core throttling mode
This series enumerates Memory Bandwidth Allocation (MBA) per-thread feature and introduces a new resctrl file "thread_throttle_mode". The file shows either "per-thread" on newer systems that implement the MBA per-thread feature or "max" on other systems that allocate maximum throttling per-core.
Details about the feature can be found in the commit description and in Chapter 9 of the most recent Intel ISE available from
https://software.intel.com/sites/default/files/managed/c5/15/architecture-instruction-set-extensions-programming-reference.pdf
This series replaces the original MBA thread throttling patch series found at
https://lore.kernel.org/lkml/[email protected]/
This is a simplified version of the previous series that skips the min/max mode and just provides the per-core vs. per-thread indicator.
Fenghua Yu (2):
x86/resctrl: Enumerate per-thread MBA
x86/resctrl: Enable user to view thread or core throttling mode
Documentation/x86/resctrl_ui.rst | 18 ++++++++- arch/x86/include/asm/cpufeatures.h | 1 + arch/x86/kernel/cpu/cpuid-deps.c | 1 + arch/x86/kernel/cpu/resctrl/core.c | 11 ++++++ arch/x86/kernel/cpu/resctrl/internal.h | 30 ++++++++++++--- arch/x86/kernel/cpu/resctrl/rdtgroup.c | 53 +++++++++++++++++++++++++- arch/x86/kernel/cpu/scattered.c | 1 + 7 files changed, 106 insertions(+), 9 deletions(-)
Signature · 1 line
2.19.1
PATCH 1/2 x86/resctrl: Enumerate per-thread MBA
Some systems support per-thread Memory Bandwidth Allocation (MBA) which applies a throttling delay value to each hardware thread instead of to a core. Per-thread MBA is enumerated by CPUID.
No feature flag is shown in /proc/cpuinfo. User applications need to check a resctrl throttling mode info file to know if the feature is supported.
--- arch/x86/include/asm/cpufeatures.h | 1 + arch/x86/kernel/cpu/cpuid-deps.c | 1 + arch/x86/kernel/cpu/scattered.c | 1 + 3 files changed, 3 insertions(+)
arch/x86/include/asm/cpufeatures.h +1 -0
@@ -286,6 +286,7 @@#define X86_FEATURE_FENCE_SWAPGS_USER (11*32+ 4)/* "" LFENCE in user entry SWAPGS path */ #define X86_FEATURE_FENCE_SWAPGS_KERNEL (11*32+ 5)/* "" LFENCE in kernel entry SWAPGS path */ #define X86_FEATURE_SPLIT_LOCK_DETECT (11*32+ 6)/* #AC for split lock */ +#define X86_FEATURE_PER_THREAD_MBA (11*32+ 7)/* "" Per-thread Memory Bandwidth Allocation */ /* Intel-defined CPU features, CPUID level 0x00000007:1 (EAX), word 12 */ #define X86_FEATURE_AVX512_BF16 (12*32+ 5)/* AVX512 BFLOAT16 instructions */
arch/x86/kernel/cpu/cpuid-deps.c +1 -0
@@ -69,6 +69,7 @@ static const struct cpuid_dep cpuid_deps[] = { { X86_FEATURE_CQM_MBM_TOTAL, X86_FEATURE_CQM_LLC }, { X86_FEATURE_CQM_MBM_LOCAL, X86_FEATURE_CQM_LLC }, { X86_FEATURE_AVX512_BF16, X86_FEATURE_AVX512VL }, + { X86_FEATURE_PER_THREAD_MBA, X86_FEATURE_MBA }, {} };
arch/x86/kernel/cpu/scattered.c +1 -0
@@ -35,6 +35,7 @@ static const struct cpuid_bit cpuid_bits[] = { { X86_FEATURE_CDP_L3, CPUID_ECX, 2, 0x00000010, 1 }, { X86_FEATURE_CDP_L2, CPUID_ECX, 2, 0x00000010, 2 }, { X86_FEATURE_MBA, CPUID_EBX, 3, 0x00000010, 0 }, + { X86_FEATURE_PER_THREAD_MBA, CPUID_ECX, 0, 0x00000010, 3 }, { X86_FEATURE_HW_PSTATE, CPUID_EDX, 7, 0x80000007, 0 }, { X86_FEATURE_CPB, CPUID_EDX, 9, 0x80000007, 0 }, { X86_FEATURE_PROC_FEEDBACK, CPUID_EDX, 11, 0x80000007, 0 },
Signature · 1 line
2.19.1
PATCH 2/2 x86/resctrl: Enable user to view thread or core throttling mode
In reply to Fenghua YuEarly Intel hardware implementations of Memory Bandwidth Allocation (MBA) could only control bandwidth at the processor core level. This meant that when two processes with different bandwidth allocations ran simultaneously on the same core the hardware had to resolve this difference. It did so by applying the higher throttling value (lower bandwidth) to both processes.
Newer implementations can apply different throttling values to each thread on a core.
Introduce a new resctrl file, "thread_throttle_mode", on Intel systems that shows to the user how throttling values are allocated, per-core or per-thread.
On systems that support per-core throttling, the file will display "max". On newer systems that support per-thread throttling, the file will display "per-thread".
AMD confirmed in
https://lore.kernel.org/lkml/[email protected]
that AMD bandwidth allocation is already at thread level but that the AMD implementation does not use a memory delay throttle mode. So to avoid confusion the thread throttling mode would be UNDEFINED on AMD systems and the "thread_throttle_mode" file will not be visible.
--- Documentation/x86/resctrl_ui.rst | 18 ++++++++- arch/x86/kernel/cpu/resctrl/core.c | 11 ++++++ arch/x86/kernel/cpu/resctrl/internal.h | 30 ++++++++++++--- arch/x86/kernel/cpu/resctrl/rdtgroup.c | 53 +++++++++++++++++++++++++- 4 files changed, 103 insertions(+), 9 deletions(-)
Documentation/x86/resctrl_ui.rst +16 -2
@@ -138,6 +138,18 @@ with respect to allocation: non-linear. This field is purely informational only. +"thread_throttle_mode": + Indicator on Intel systems of how tasks running on threads + of a physical core are throttled in cases where they + request different memory bandwidth percentages: + + "max": + the smallest percentage is applied + to all threads + "per-thread": + bandwidth percentages are directly applied to + the threads running on the core + If RDT monitoring is available there will be an "L3_MON" directory with the following files:
@@ -364,8 +376,10 @@ to the next control step available on the hardware. The bandwidth throttling is a core specific mechanism on some of Intel SKUs. Using a high bandwidth and a low bandwidth setting on two threads -sharing a core will result in both threads being throttled to use the -low bandwidth. The fact that Memory bandwidth allocation(MBA) is a core +sharing a core may result in both threads being throttled to use the +low bandwidth (see "thread_throttle_mode"). + +The fact that Memory bandwidth allocation(MBA) may be a core specific mechanism where as memory bandwidth monitoring(MBM) is done at the package level may lead to confusion when users try to apply control via the MBA and then monitor the bandwidth to see if the controls are
arch/x86/kernel/cpu/resctrl/core.c +11 -0
@@ -270,6 +270,12 @@ static bool __get_mem_config_intel(struct rdt_resource *r) } r->data_width =3 ; +if (boot_cpu_has (X86_FEATURE_PER_THREAD_MBA )) + r->membw.arch_throttle_mode =THREAD_THROTTLE_PER_THREAD ; +else + r->membw.arch_throttle_mode =THREAD_THROTTLE_MAX ; +thread_throttle_mode_init (); + r->alloc_capable = true; r->alloc_enabled = true;
@@ -289,6 +295,11 @@ static bool __rdt_get_mem_config_amd(struct rdt_resource *r)/* AMD does not use delay */ r->membw.delay_linear = false; +/* +* AMD does not use memory delay throttle model to control +* the allocation like Intel does. +*/ + r->membw.arch_throttle_mode =THREAD_THROTTLE_UNDEFINED ; r->membw.min_bw =0 ; r->membw.bw_gran =1 ;/* Max value is 2048, Data width should be 4 in decimal */
arch/x86/kernel/cpu/resctrl/internal.h +24 -6
@@ -369,6 +369,20 @@ struct rdt_cache {unsigned int shareable_bits; }; +/** +* enum membw_throttle_mode - System's memory bandwidth throttling mode +* @THREAD_THROTTLE_UNDEFINED: Not relevant to the system +* @THREAD_THROTTLE_MAX: Memory bandwidth is throttled at the core +* always using smallest bandwidth percentage +* assigned to threads, aka "max throttling" +* @THREAD_THROTTLE_PER_THREAD: Memory bandwidth is throttled at the thread +*/ +enum membw_throttle_mode { +THREAD_THROTTLE_UNDEFINED =0 , +THREAD_THROTTLE_MAX , +THREAD_THROTTLE_PER_THREAD , +}; +/** * struct rdt_membw - Memory bandwidth allocation related data * @max_delay: Max throttle delay. Delay is the hardware
@@ -376,16 +390,19 @@ struct rdt_cache {* @min_bw: Minimum memory bandwidth percentage user can request * @bw_gran: Granularity at which the memory bandwidth is allocated * @delay_linear: True if memory B/W delay is in linear scale +* @arch_throttle_mode: Bandwidth throttling mode when threads request +* different memory bandwidths * @mba_sc: True if MBA software controller(mba_sc) is enabled * @mb_map: Mapping of memory B/W percentage to memory B/W delay */ struct rdt_membw { -u32 max_delay; -u32 min_bw; -u32 bw_gran; -u32 delay_linear; -bool mba_sc; -u32 *mb_map; +u32 max_delay; +u32 min_bw; +u32 bw_gran; +u32 delay_linear; +enum membw_throttle_mode arch_throttle_mode; +bool mba_sc; +u32 *mb_map; };static inline bool is_llc_occupancy_enabled (void )
@@ -612,5 +629,6 @@ void __check_limbo(struct rdt_domain *d, bool force_free);bool cbm_validate_intel (char *buf,u32 *data,struct rdt_resource *r);bool cbm_validate_amd (char *buf,u32 *data,struct rdt_resource *r);void rdt_domain_reconfigure_cdp (struct rdt_resource *r); +void __init thread_throttle_mode_init (void );#endif /* _ASM_X86_RESCTRL_INTERNAL_H */
arch/x86/kernel/cpu/resctrl/rdtgroup.c +52 -1
77 lines not on this page · raw message
Signature · 1 line
2.19.1
Borislav Petkov
maintainerarch/x86/kernel/cpu/resctrl/internal.h struct rdt_cache { · line 403
Quoted 12 lines · Fenghua Yu
struct
rdt_membw{arch/x86/kernel/cpu/resctrl/internal.h +5
-6-u32 max_delay; -u32 min_bw; -u32 bw_gran; -u32 delay_linear; -bool mba_sc; -u32 *mb_map; +u32 max_delay; +u32 min_bw; +u32 bw_gran; +u32 delay_linear; +enum membw_throttle_mode arch_throttle_mode;
I guess "arch" is implicit so you can
s/arch_throttle_mode/throttle_mode/g
...
Quoted 1 line
+static struct rftype *rdtgroup_rftype_by_name(const char *name)
That function name needs a verb.
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteQuoted 7 lines · Borislav Petkov
arch/x86/kernel/cpu/resctrl/internal.h struct rdt_cache { · line 403
Quoted 2 lines · Fenghua Yu
struct
rdt_membw{arch/x86/kernel/cpu/resctrl/internal.h +1
-0+enum membw_throttle_mode arch_throttle_mode;I guess "arch" is implicit so you can
s/arch_throttle_mode/throttle_mode/g
Ok. Will fix this.
Quoted 6 lines
...
Quoted 1 line
+static struct rftype *rdtgroup_rftype_by_name(const char *name)
That function name needs a verb.
I will change it to rdtgroup_rftype_get_by_name().
Thank you very much for your review!
-Fenghua
Hi, X86 maintainers,
Quoted 10 lines · Fenghua Yu
This series enumerates Memory Bandwidth Allocation (MBA) per-thread feature and introduces a new resctrl file "
thread_throttle_mode". The file shows either "per-thread" on newer systems that implement the MBA per-thread feature or "max" on other systems that allocate maximum throttling per-core.Details about the feature can be found in the commit description and in Chapter 9 of the most recent Intel ISE available from
https://software.intel.com/sites/default/files/managed/c5/15/architecture-instruction-set-extensions-programming-reference.pdf
Quoted 7 lines
This series replaces the original MBA thread throttling patch series found at
https://lore.kernel.org/lkml/[email protected]/This is a simplified version of the previous series that skips the min/max mode and just provides the per-core vs. per-thread indicator.
Any comment on this series? Could you please help push to upstream?
Thanks.
-Fenghua