x86/fpu: track AVX-512 usage of tasks
Subsystems (2)
PATCH v4 1/2 x86/fpu: track AVX-512 usage of tasks
User space tools which do automated task placement need information about AVX-512 usage of tasks, because AVX-512 usage could cause core turbo frequency drop and impact the running task on the sibling CPU.
The XSAVE hardware structure has bits that indicate when valid state is present in registers unique to AVX-512 use. Use these bits to indicate when AVX-512 has been in use and add per-task AVX-512 state tracking to context switch.
The tracking turns on the usage flag at the next context switch of the task, but requires 3 consecutive context switches with no usage to clear it. This decay is required because well-written AVX-512 applications are expected to clear this state when not actively using AVX-512 registers.
Although this mechanism is imprecise and can theoretically have both false-positives and false-negatives, it has been measured to be precise enough to be useful under real-world workloads like tensorflow and linpack.
If higher precision is required, suggest user space tools to use the PMU-based mechanisms in combination.
--- arch/x86/include/asm/fpu/internal.h | 22 ++++++++++++++++++++++ arch/x86/include/asm/fpu/types.h | 8 ++++++++ 2 files changed, 30 insertions(+)
arch/x86/include/asm/fpu/internal.h +22 -0
@@ -275,6 +275,27 @@ static inline void copy_fxregs_to_kernel(struct fpu *fpu) : "D" (st), "m" (*st), "a" (lmask), "d" (hmask) \ : "memory") +#define AVX512_STATE_DECAY_COUNT 3 +/* +* This function is called during context switch to update AVX512 state +*/ +static inline void update_avx512_state (struct fpu *fpu) +{ +/* +* AVX512 state is tracked here because its use is known to slow +* the max clock speed of the core. +* +* However, AVX512-using tasks are expected to clear this state when +* not actively using these registers. Thus, this tracking mechanism +* can miss. To ensure that false-negatives do not immediately show +* up, decay the usage count over time. +*/ +if (fpu->state.xsave.header.xfeatures &XFEATURE_MASK_AVX512 ) + fpu->avx512_usage =AVX512_STATE_DECAY_COUNT ; +else if (fpu->avx512_usage) + fpu->avx512_usage--; +} +/* * This function is called only during boot time when x86 caps are not set * up and alternative can not be used yet.
@@ -411,6 +432,7 @@ static inline int copy_fpregs_to_fpstate(struct fpu *fpu) {if (likely (use_xsave ())) {copy_xregs_to_kernel (&fpu->state.xsave); +update_avx512_state (fpu);return 1 ; }
arch/x86/include/asm/fpu/types.h +8 -0
@@ -302,6 +302,14 @@ struct fpu {*/ unsigned char initialized; +/* +* @avx512_usage: +* +* Records the usage of AVX512 registers. A value of non-zero is used +* to indicate whether these AVX512 registers recently had valid state. +*/ +unsigned char avx512_usage; +/* * @state: *
Signature · 1 line
2.17.1
PATCH v4 2/2 proc: add AVX-512 usage to /proc/pid/status
AVX-512 components usage could cause core turbo frequency drop. So it's useful to expose AVX-512 components usage as a heuristic hint for the user space job scheduler to cluster the AVX-512 using tasks together.
Example:
$ cat /proc/pid/status | grep AVX512_hint AVX512_hint: 1
The hint number '0' indicates the task recently didn't use AVX-512 components thus unlikely has frequency drop issue. And the number '1' indicates the task recently used AVX-512 components thus could cause core frequency drop. User space tools may want to further check by:
$ perf stat --pid <pid> -e core_power.lvl2_turbo_license -- sleep 1
Performance counter stats for process id '3558':
3,251,565,961 core_power.lvl2_turbo_license
1.004031387 seconds time elapsedNon-zero counter value confirms that the task causes frequency drop.
--- arch/x86/kernel/fpu/xstate.c | 19 +++++++++++++++++++ fs/proc/array.c | 5 +++++ 2 files changed, 24 insertions(+)
arch/x86/kernel/fpu/xstate.c +19 -0
@@ -7,6 +7,7 @@#include <linux/cpu.h> #include <linux/mman.h> #include <linux/pkeys.h> +#include <linux/seq_file.h> #include <asm/fpu/api.h> #include <asm/fpu/internal.h>
@@ -1245,3 +1246,21 @@ int copy_user_to_xstate(struct xregs_state *xsave, const void __user *ubuf)return 0 ; } + +/* +* Report CPU specific thread state +*/ +void arch_task_state (struct seq_file *m,struct task_struct *task) +{ +/* +* Check the processor and build option if AVX512 is supported. +*/ +if (!cpu_feature_enabled (X86_FEATURE_AVX512F )) +return ; +/* +* Report AVX-512 components usage: +*/ +seq_put_decimal_ull (m,"AVX512_hint:\t" , + task->thread.fpu.avx512_usage ?1 :0 ); +seq_putc (m,'\n' ); +}
fs/proc/array.c +5 -0
@@ -392,6 +392,10 @@ static inline void task_core_dumping(struct seq_file *m, struct mm_struct *mm)seq_putc (m,'\n' ); } +void __weak arch_task_state (struct seq_file *m,struct task_struct *task) +{ +} +int proc_pid_status (struct seq_file *m,struct pid_namespace *ns,struct pid *pid,struct task_struct *task) {
@@ -414,6 +418,7 @@ int proc_pid_status(struct seq_file *m, struct pid_namespace *ns,task_cpus_allowed (m, task);cpuset_task_status_allowed (m, task);task_context_switch_counts (m, task); +arch_task_state (m, task);return 0 ; }
Signature · 1 line
2.17.1
Re: [PATCH v4 2/2] proc: add AVX-512 usage to /proc/pid/status
Quoted 57 lines · Aubrey Li
AVX-512 components usage could cause core turbo frequency drop. So it's useful to expose AVX-512 components usage as a heuristic hint for the user space job scheduler to cluster the AVX-512 using tasks together.
Example:
$ cat /proc/pid/status | grep AVX512_hint AVX512_hint: 1The hint number '0' indicates the task recently didn't use AVX-512 components thus unlikely has frequency drop issue. And the number '1' indicates the task recently used AVX-512 components thus could cause core frequency drop. User space tools may want to further check by:
$ perf stat --pid <pid> -e core_power.lvl2_turbo_license -- sleep 1Performance counter stats for process id '3558':
3,251,565,961 core_power.lvl2_turbo_license 1.004031387 seconds time elapsedNon-zero counter value confirms that the task causes frequency drop.
Signed-off-by Aubrey Li <[email protected]>Cc Peter Zijlstra <[email protected]>Cc Andi Kleen <[email protected]>Cc Tim Chen <[email protected]>Cc Dave Hansen <[email protected]>Cc Arjan van de Ven <[email protected]>--- arch/x86/kernel/fpu/xstate.c | 19 +++++++++++++++++++ fs/proc/array.c | 5 +++++ 2 files changed, 24 insertions(+)arch/x86/kernel/fpu/xstate.c +7
-0@@ -7,6 +7,7 @@#include <linux/cpu.h> #include <linux/mman.h> #include <linux/pkeys.h> +#include <linux/seq_file.h> #include <asm/fpu/api.h> #include <asm/fpu/internal.h> @@ -1245,3 +1246,21 @@ int copy_user_to_xstate(struct xregs_state *xsave, const void __user *ubuf)return 0 ; } + +/* +* Report CPU specific thread state +*/ +void arch_task_state (struct seq_file *m,struct task_struct *task) +{
arch/x86/kernel/fpu/xstate.c int copy_user_to_xstate(struct xregs_state *xsave, const void __user *ubuf) · line 1264
Quoted 10 lines
arch/x86/kernel/fpu/xstate.c +10
-0+/* +* Check the processor and build option if AVX512 is supported. +*/ +if (!cpu_feature_enabled (X86_FEATURE_AVX512F )) +return ; +/* +* Report AVX-512 components usage: +*/ +seq_put_decimal_ull (m,"AVX512_hint:\t" , + task->thread.fpu.avx512_usage ?1 :0 );
I believe you need some kind of documentation of this new interface in kernel's Documentation. So an admin doesn't need to look into the kernel code to figure out what this /proc/pid/status is supposed to do.
Tim
Quoted 5 lines · Aubrey Li
The tracking turns on the usage flag at the next context switch of the task, but requires 3 consecutive context switches with no usage to clear it. This decay is required because well-written AVX-512 applications are expected to clear this state when not actively using AVX-512 registers.
One concern about this: Given a HZ=1000 system, this means that the flag needs to get scanned every ~3ms. That's a pretty good amount of scanning on a system with hundreds or thousands of tasks running around.
How many tasks does this scale to until you're eating up an entire CPU or two just scanning /proc?
Quoted 7 lines · Dave Hansen
Quoted 5 lines · Aubrey Li
The tracking turns on the usage flag at the next context switch of the task, but requires 3 consecutive context switches with no usage to clear it. This decay is required because well-written AVX-512 applications are expected to clear this state when not actively using AVX-512 registers.
Quoted 7 lines
One concern about this: Given a HZ=1000 system, this means that the flag needs to get scanned every ~3ms. That's a pretty good amount of scanning on a system with hundreds or thousands of tasks running around.
How many tasks does this scale to until you're eating up an entire CPU or two just scanning /proc?
The reason why I believe that an exponential decay average, or some other kind of weighted decay of the AVX512 usage is better.
Tim
Quoted 7 lines · Dave Hansen
Quoted 5 lines · Aubrey Li
The tracking turns on the usage flag at the next context switch of the task, but requires 3 consecutive context switches with no usage to clear it. This decay is required because well-written AVX-512 applications are expected to clear this state when not actively using AVX-512 registers.
Quoted 6 lines
One concern about this: Given a HZ=1000 system, this means that the flag needs to get scanned every ~3ms. That's a pretty good amount of scanning on a system with hundreds or thousands of tasks running around.
How many tasks does this scale to until you're eating up an entire CPU or two just scanning /proc?
Yes that's why we may need to propagate it to cgroups in the kernel, because user daemons don't really want to track every TID.
But per pid is a start so that people can start experimenting with this.
Then with some experience fancier interfaces for it can be implemented.
-Andi
Quoted 7 lines · Dave Hansen
Quoted 5 lines · Aubrey Li
The tracking turns on the usage flag at the next context switch of the task, but requires 3 consecutive context switches with no usage to clear it. This decay is required because well-written AVX-512 applications are expected to clear this state when not actively using AVX-512 registers.
Quoted 7 lines
One concern about this: Given a HZ=1000 system, this means that the flag needs to get scanned every ~3ms. That's a pretty good amount of scanning on a system with hundreds or thousands of tasks running around.
How many tasks does this scale to until you're eating up an entire CPU or two just scanning /proc?
Do we have a real requirement to do this in practical environment? AFAIK, 1s or even 5s is good enough in some customers environment.
Thanks,
-Aubrey
Quoted 12 lines · Aubrey
Quoted 11 lines · Dave Hansen
Quoted 5 lines · Aubrey Li
The tracking turns on the usage flag at the next context switch of the task, but requires 3 consecutive context switches with no usage to clear it. This decay is required because well-written AVX-512 applications are expected to clear this state when not actively using AVX-512 registers.
One concern about this: Given a HZ=1000 system, this means that the flag needs to get scanned every ~3ms. That's a pretty good amount of scanning on a system with hundreds or thousands of tasks running around.
Quoted 6 lines
Quoted 3 lines
How many tasks does this scale to until you're eating up an entire CPU or two just scanning /proc?
Do we have a real requirement to do this in practical environment? AFAIK, 1s or even 5s is good enough in some customers environment.
maybe instead of a 1/0 bit, it's useful to store the timestamp of the last time we found the task to use avx? (need to find a good time unit)
Quoted 17 lines · Arjan van de Ven
Quoted 16 lines · Aubrey
Quoted 14 lines · Dave Hansen
Quoted 5 lines · Aubrey Li
The tracking turns on the usage flag at the next context switch of the task, but requires 3 consecutive context switches with no usage to clear it. This decay is required because well-written AVX-512 applications are expected to clear this state when not actively using AVX-512 registers.
One concern about this: Given a HZ=1000 system, this means that the flag needs to get scanned every ~3ms. That's a pretty good amount of scanning on a system with hundreds or thousands of tasks running around.
How many tasks does this scale to until you're eating up an entire CPU or two just scanning /proc?
Quoted 7 lines
Quoted 2 lines
Do we have a real requirement to do this in practical environment? AFAIK, 1s or even 5s is good enough in some customers environment.
maybe instead of a 1/0 bit, it's useful to store the timestamp of the last time we found the task to use avx? (need to find a good time unit)
Are you suggesting kernel does not do any translation, just provide a fact to the user space tool and let user space tool to decide how to use this info?
So how does user space tool use this timestamp in your mind?
Thanks,
-Aubrey
Quoted 8 lines · Aubrey
Quoted 4 lines
maybe instead of a 1/0 bit, it's useful to store the timestamp of the last time we found the task to use avx? (need to find a good time unit)
Are you suggesting kernel does not do any translation, just provide a fact to the user space tool and let user space tool to decide how to use this info?
So how does user space tool use this timestamp in your mind?
Couple of things...
If we have a timestamp, we don't need a decay. That means less tuning and also less work at runtime in the common case.
Let's say we report milliseconds. The app itself looks at the number and could now say the following:
1. task A used AVX512 1ms ago, I might need to segregate it
2. task B used AVX512 10000ms ago, time to stop segregating it
3. task C used AVX512 1ms ago, and was using it 100ms ago (during the last scan), it's a regular AVX512 user.
That way, you don't have to *catch* tasks in the window between use and the end of the decay period.
to update AVX512 state
Quoted 22 lines
+ */
arch/x86/include/asm/fpu/internal.h +17
-0+static inline void update_avx512_state (struct fpu *fpu) +{ +/* +* AVX512 state is tracked here because its use is known to slow +* the max clock speed of the core. +* +* However, AVX512-using tasks are expected to clear this state when +* not actively using these registers. Thus, this tracking mechanism +* can miss. To ensure that false-negatives do not immediately show +* up, decay the usage count over time. +*/ +if (fpu->state.xsave.header.xfeatures &XFEATURE_MASK_AVX512 ) + fpu->avx512_usage =AVX512_STATE_DECAY_COUNT ; +else if (fpu->avx512_usage) + fpu->avx512_usage--; +} +/* * This function is called only during boot time when x86 caps are not set * up and alternative can not be used yet.
arch/x86/include/asm/fpu/internal.h static inline int copy_fpregs_to_fpstate(struct fpu *fpu) · line 437
Quoted 6 lines
arch/x86/include/asm/fpu/internal.h +1
-0@@ -411,6 +432,7 @@ static inline int copy_fpregs_to_fpstate(struct fpu *fpu) {if (likely (use_xsave ())) {copy_xregs_to_kernel (&fpu->state.xsave); +update_avx512_state (fpu);return 1 ; }
Is there a reason we shouldn't do:
if (!cpu_feature_enabled(X86_FEATURE_AVX512F)) update_avx512_state(fpu);
?
Quoted 33 lines · Dave Hansen
to update AVX512 state
Quoted 22 lines
+ */
arch/x86/include/asm/fpu/internal.h +17
-0+static inline void update_avx512_state (struct fpu *fpu) +{ +/* +* AVX512 state is tracked here because its use is known to slow +* the max clock speed of the core. +* +* However, AVX512-using tasks are expected to clear this state when +* not actively using these registers. Thus, this tracking mechanism +* can miss. To ensure that false-negatives do not immediately show +* up, decay the usage count over time. +*/ +if (fpu->state.xsave.header.xfeatures &XFEATURE_MASK_AVX512 ) + fpu->avx512_usage =AVX512_STATE_DECAY_COUNT ; +else if (fpu->avx512_usage) + fpu->avx512_usage--; +} +/* * This function is called only during boot time when x86 caps are not set * up and alternative can not be used yet. arch/x86/include/asm/fpu/internal.h static inline int copy_fpregs_to_fpstate(struct fpu *fpu) · line 437
Quoted 6 lines
arch/x86/include/asm/fpu/internal.h +1
-0@@ -411,6 +432,7 @@ static inline int copy_fpregs_to_fpstate(struct fpu *fpu) {if (likely (use_xsave ())) {copy_xregs_to_kernel (&fpu->state.xsave); +update_avx512_state (fpu);return 1 ; }Is there a reason we shouldn't do:
Quoted 5 lines
if (!cpu_feature_enabled(X86_FEATURE_AVX512F)) update_avx512_state(fpu);?
Why _!_ ?
Quoted 8 lines · Aubrey
Quoted 7 lines
Is there a reason we shouldn't do:
if (!cpu_feature_enabled(X86_FEATURE_AVX512F)) update_avx512_state(fpu);?
Why _!_ ?
Sorry, got it backwards. I think I was considering having you do a
if (!cpu_feature_enabled(X86_FEATURE_AVX512F)) return;
inside of update_avx512_state(), but I got the state mixed up in my head.
You don't need the '!'.
From: Aubrey Li
Quoted 10 lines
Sent: 11 December 2018 00:25
User space tools which do automated task placement need information about AVX-512 usage of tasks, because AVX-512 usage could cause core turbo frequency drop and impact the running task on the sibling CPU.
The XSAVE hardware structure has bits that indicate when valid state is present in registers unique to AVX-512 use. Use these bits to indicate when AVX-512 has been in use and add per-task AVX-512 state tracking to context switch.
Isn't a thread likely to clear the AVX registers at the end of a function that uses them.
In particular this removes the massive overhead on certain cpus of switching between two AVX modes.
So it is actually unlikely that XSAVE will need to save them at all?
As I've also said before the registers are caller saved and since systems calls are normal function calls the application code would have to save them across a system call.
This allows the kernel to zero the registers on system call entry again meaning that XSAVE won't normally have to save them.
David
-
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK Registration No: 1397386 (Wales)
Quoted 5 lines
Isn't a thread likely to clear the AVX registers at the end of a function that uses them.
In particular this removes the massive overhead on certain cpus of switching between two AVX modes.
So it is actually unlikely that XSAVE will need to save them at all?
Only if context switches only happened on function boundaries, which is obviously not the case.
Yes the detection mechanism is not 100% accurate, but if AVX is used significantly it should eventually detect it. Think of it as a statistical sampling heuristic.
Quoted 6 lines
As I've also said before the registers are caller saved and since systems calls are normal function calls the application code would have to save them across a system call.
This allows the kernel to zero the registers on system call entry again meaning that XSAVE won't normally have to save them.
While I agree this would be nice, the Linux system call ABI wasn't defined like this, so it cannot be done at this point.
-Andi