Skip to contentSkip to conversation

/proc/pid/status: Add support for architecture specific output

PATCH Superseded Aubrey Li 5 participants 14 messages Feb 24, 2019 – Apr 8, 2019

Subsystems (4)

v13 of 16 Superseded Latest is v16 →

lore
Share & tools
lore

Aubrey Li

author
#0 lore raw

PATCH v13 1/3 /proc/pid/status: Add support for architecture specific output

The architecture specific information of the running processes could be useful to the userland. Add support to examine process architecture specific information externally.

Cc Peter Zijlstra <[email protected]>
Cc Andi Kleen <[email protected]>
Cc Tim Chen <[email protected]>
Cc Dave Hansen <[email protected]>
Cc Arjan van de Ven <[email protected]>
---
 fs/proc/array.c         | 5 +++++
 include/linux/proc_fs.h | 2 ++
 2 files changed, 7 insertions(+)
fs/proc/array.c +5 -0
index 9d428d5a0ac8..ea7a981f289c 100644
@@ -401,6 +401,10 @@ static inline void task_thp_status(struct seq_file *m, struct mm_struct *mm)
 	seq_printf(m, "THP_enabled:\t%d\n", thp_enabled);
 }
 
+void __weak arch_proc_pid_status(struct seq_file *m, struct task_struct *task)
+{
+}
+
 int proc_pid_status(struct seq_file *m, struct pid_namespace *ns,
 			struct pid *pid, struct task_struct *task)
 {
@@ -424,6 +428,7 @@ int proc_pid_status(struct seq_file *m, struct pid_namespace *ns,
 	task_cpus_allowed(m, task);
 	cpuset_task_status_allowed(m, task);
 	task_context_switch_counts(m, task);
+	arch_proc_pid_status(m, task);
 	return 0;
 }
 
include/linux/proc_fs.h +2 -0
index d0e1f1522a78..1de9ba1b064f 100644
@@ -73,6 +73,8 @@ struct proc_dir_entry *proc_create_net_single_write(const char *name, umode_t mo
 						    int (*show)(struct seq_file *, void *),
 						    proc_write_t write,
 						    void *data);
+/* Add support for architecture specific output in /proc/pid/status */
+extern void arch_proc_pid_status(struct seq_file *m, struct task_struct *task);
 
 #else /* CONFIG_PROC_FS */
 
Signature · 1 line
2.17.1
author
#1 lore raw

PATCH v13 2/3 x86,/proc/pid/status: Add AVX-512 usage elapsed time

AVX-512 components use could cause core turbo frequency drop. So it's useful to expose AVX-512 usage elapsed time as a heuristic hint for the user space job scheduler to cluster the AVX-512 using tasks together.

Tensorflow example:

$ while [ 1 ]; do cat /proc/pid/status | grep AVX; sleep 1; done
AVX512_elapsed_ms:	4
AVX512_elapsed_ms:	8
AVX512_elapsed_ms:	4

This means that 4 milliseconds have elapsed since the AVX512 usage of tensorflow task was detected when the task was scheduled out.

Or:

$ cat /proc/pid/status | grep AVX512_elapsed_ms
AVX512_elapsed_ms:      -1

The number '-1' indicates that no AVX512 usage recorded before thus the task unlikely has frequency drop issue.

User space tools may want to further check by:

$ perf stat --pid <pid> -e core_power.lvl2_turbo_license -- sleep 1

Performance counter stats for process id '3558':

     3,251,565,961      core_power.lvl2_turbo_license

       1.004031387 seconds time elapsed

Non-zero counter value confirms that the task causes frequency drop.

Cc Peter Zijlstra <[email protected]>
Cc Andi Kleen <[email protected]>
Cc Tim Chen <[email protected]>
Cc Dave Hansen <[email protected]>
Cc Arjan van de Ven <[email protected]>
---
 arch/x86/kernel/fpu/xstate.c | 42 ++++++++++++++++++++++++++++++++++++
 1 file changed, 42 insertions(+)
arch/x86/kernel/fpu/xstate.c +42 -0
index 9cc108456d0b..e480a535eeb2 100644
@@ -7,6 +7,8 @@
 #include <linux/cpu.h>
 #include <linux/mman.h>
 #include <linux/pkeys.h>
+#include <linux/seq_file.h>
+#include <linux/proc_fs.h>
 
 #include <asm/fpu/api.h>
 #include <asm/fpu/internal.h>
@@ -1243,3 +1245,43 @@ int copy_user_to_xstate(struct xregs_state *xsave, const void __user *ubuf)
 
 	return 0;
 }
+
+/*
+ * Report the amount of time elapsed in millisecond since last AVX512
+ * use in the task.
+ */
+static void avx512_status(struct seq_file *m, struct task_struct *task)
+{
+	unsigned long timestamp = task->thread.fpu.avx512_timestamp;
+	long delta;
+
+	if (!timestamp) {
+		/*
+		 * Report -1 if no AVX512 usage
+		 */
+		delta = -1;
+	} else {
+		delta = (long)(jiffies - timestamp);
+		/*
+		 * Cap to LONG_MAX if time difference > LONG_MAX
+		 */
+		if (delta < 0)
+			delta = LONG_MAX;
+		delta = jiffies_to_msecs(delta);
+	}
+
+	seq_put_decimal_ll(m, "AVX512_elapsed_ms:\t", delta);
+	seq_putc(m, '\n');
+}
+
+/*
+ * Report architecture specific information
+ */
+void arch_proc_pid_status(struct seq_file *m, struct task_struct *task)
+{
+	/*
+	 * Report AVX512 state if the processor and build option supported.
+	 */
+	if (cpu_feature_enabled(X86_FEATURE_AVX512F))
+		avx512_status(m, task);
+}
Signature · 1 line
2.17.1
#2 lore raw

Re: [PATCH v13 2/3] x86,/proc/pid/status: Add AVX-512 usage elapsed time

Quoted 11 lines · Aubrey Li

AVX-512 components use could cause core turbo frequency drop. So it's useful to expose AVX-512 usage elapsed time as a heuristic hint for the user space job scheduler to cluster the AVX-512 using tasks together.

Tensorflow example:

$ while [ 1 ]; do cat /proc/pid/status | grep AVX; sleep 1; done
AVX512_elapsed_ms:      4
AVX512_elapsed_ms:      8
AVX512_elapsed_ms:      4
Quoted 6 lines

This means that 4 milliseconds have elapsed since the AVX512 usage of tensorflow task was detected when the task was scheduled out.

Or:

$ cat /proc/pid/status | grep AVX512_elapsed_ms
AVX512_elapsed_ms:      -1

(Very nitpicky, feel free to ignore: If you change the /proc/pid to /proc/tid in the commit message, it becomes clearer that this status is really per-task/thread, not per-process/threadgroup.)

[...]

arch/x86/kernel/fpu/xstate.c int copy_user_to_xstate(struct xregs_state *xsave, const void __user *ubuf) · line 1255

Quoted 8 lines
arch/x86/kernel/fpu/xstate.c +8 -0
+
+/*
+ * Report the amount of time elapsed in millisecond since last AVX512
+ * use in the task.
+ */
+static void avx512_status(struct seq_file *m, struct task_struct *task)
+{
+       unsigned long timestamp = task->thread.fpu.avx512_timestamp;

This is theoretically a data race, right? Should this have a READ_ONCE() on it?

Is there something that zeroes out the avx512_timestamp on fork()/clone(), or will every child inherit the avx512 timestamp? As far as I can tell, the timestamp is inherited; I think it would be nicer to zero it out at that point. Either way, It might be worth documenting this decision.

author
#3 lore raw

Re: [PATCH v13 2/3] x86,/proc/pid/status: Add AVX-512 usage elapsed time

Quoted 15 lines · Jann Horn
Quoted 14 lines · Aubrey Li

AVX-512 components use could cause core turbo frequency drop. So it's useful to expose AVX-512 usage elapsed time as a heuristic hint for the user space job scheduler to cluster the AVX-512 using tasks together.

Tensorflow example:

$ while [ 1 ]; do cat /proc/pid/status | grep AVX; sleep 1; done
AVX512_elapsed_ms:      4
AVX512_elapsed_ms:      8
AVX512_elapsed_ms:      4

This means that 4 milliseconds have elapsed since the AVX512 usage of tensorflow task was detected when the task was scheduled out.

Quoted 7 lines
Quoted 3 lines

Or:

$ cat /proc/pid/status | grep AVX512_elapsed_ms
AVX512_elapsed_ms:      -1

(Very nitpicky, feel free to ignore: If you change the /proc/pid to /proc/tid in the commit message, it becomes clearer that this status is really per-task/thread, not per-process/threadgroup.)

Thanks, I'll refine.

Quoted 12 lines

[...]

arch/x86/kernel/fpu/xstate.c int copy_user_to_xstate(struct xregs_state *xsave, const void __user *ubuf) · line 1255

Quoted 8 lines
arch/x86/kernel/fpu/xstate.c +8 -0
+
+/*
+ * Report the amount of time elapsed in millisecond since last AVX512
+ * use in the task.
+ */
+static void avx512_status(struct seq_file *m, struct task_struct *task)
+{
+       unsigned long timestamp = task->thread.fpu.avx512_timestamp;

This is theoretically a data race, right? Should this have a READ_ONCE() on it?

Thanks, I'll refine.

Quoted 7 lines

Is there something that zeroes out the avx512_timestamp on fork()/clone(), or will every child inherit the avx512 timestamp? As far as I can tell, the timestamp is inherited; I think it would be nicer to zero it out at that point. Either way, It might be worth documenting this decision.

This timestamp is not inherited, see below:

_do_fork()
->copy_process()
-->dup_task_struct()
--->arch_dup_task_struct()
---->fpu__copy()

Thanks,
-Aubrey

author
#4 lore raw

PATCH v13 3/3 Documentation/filesystems/proc.txt: add AVX512_elapsed_ms

In reply to Aubrey Li

Added AVX512_elapsed_ms in /proc/<pid>/status. Report it in Documentation/filesystems/proc.txt

Cc Peter Zijlstra <[email protected]>
Cc Andi Kleen <[email protected]>
Cc Tim Chen <[email protected]>
Cc Dave Hansen <[email protected]>
Cc Arjan van de Ven <[email protected]>
---
 Documentation/filesystems/proc.txt | 29 ++++++++++++++++++++++++++++-
 1 file changed, 28 insertions(+), 1 deletion(-)
Documentation/filesystems/proc.txt +28 -1
index 66cad5c86171..c4a9e48681ad 100644
@@ -207,6 +207,7 @@ read the file /proc/PID/status:
   Speculation_Store_Bypass:       thread vulnerable
   voluntary_ctxt_switches:        0
   nonvoluntary_ctxt_switches:     1
+  AVX512_elapsed_ms:	8
 
 This shows you nearly the same information you would get if you viewed it with
 the ps  command.  In  fact,  ps  uses  the  proc  file  system  to  obtain its
@@ -224,7 +225,7 @@ asynchronous manner and the value may not be very precise. To see a precise
 snapshot of a moment, you can see /proc/<pid>/smaps file and scan page table.
 It's slow but very precise.
 
-Table 1-2: Contents of the status files (as of 4.19)
+Table 1-2: Contents of the status files (as of 5.1)
 ..............................................................................
  Field                       Content
  Name                        filename of the executable
@@ -289,6 +290,32 @@ Table 1-2: Contents of the status files (as of 4.19)
  Mems_allowed_list           Same as previous, but in "list format"
  voluntary_ctxt_switches     number of voluntary context switches
  nonvoluntary_ctxt_switches  number of non voluntary context switches
+ AVX512_elapsed_ms           time elapsed since last AVX512 usage recorded
+
+ AVX512_elapsed_ms:
+ ------------------
+  If AVX512 is supported on the machine, this entry shows the milliseconds
+  elapsed since the last time AVX512 usage was recorded. The recording
+  happens on a best effort basis when a task is scheduled out. This means
+  that the value depends on two factors:
+
+    1) The time which the task spent on the CPU without being scheduled
+       out. With CPU isolation and a single runnable task this can take
+       several seconds.
+
+    2) The time since the task was scheduled out last. Depending on the
+       reason for being scheduled out (time slice exhausted, syscall ...)
+       this can be arbitrary long time.
+
+  As a consequence the value cannot be considered precise and authoritative
+  information. The application which uses this information has to be aware
+  of the overall scenario on the system in order to determine whether a
+  task is a real AVX512 user or not.
+
+  A special value of '-1' indicates that no AVX512 usage was recorded, thus
+  the task is unlikely an AVX512 user, but depends on the workload and the
+  scheduling scenario, it also could be a false negative mentioned above.
+
 ..............................................................................
 
 Table 1-3: Contents of the statm files (as of 2.6.8-rc3)
Signature · 1 line
2.17.1
#5 lore raw
In reply to Aubrey Li
Quoted 10 lines · Aubrey Li

The architecture specific information of the running processes could be useful to the userland. Add support to examine process architecture specific information externally.

Cc Peter Zijlstra <[email protected]>
Cc Andi Kleen <[email protected]>
Cc Tim Chen <[email protected]>
Cc Dave Hansen <[email protected]>
Cc Arjan van de Ven <[email protected]>

This really lacks

Cc Linux API <[email protected]>
Cc Alexey Dobriyan <[email protected]>
Cc Andrew Morton <[email protected]>

Cc'ed now.

Quoted 45 lines
---
 fs/proc/array.c         | 5 +++++
 include/linux/proc_fs.h | 2 ++
 2 files changed, 7 insertions(+)
fs/proc/array.c +5 -0
index 9d428d5a0ac8..ea7a981f289c 100644
@@ -401,6 +401,10 @@ static inline void task_thp_status(struct seq_file *m, struct mm_struct *mm)
 	seq_printf(m, "THP_enabled:\t%d\n", thp_enabled);
 }
 
+void __weak arch_proc_pid_status(struct seq_file *m, struct task_struct *task)
+{
+}
+
 int proc_pid_status(struct seq_file *m, struct pid_namespace *ns,
 			struct pid *pid, struct task_struct *task)
 {
@@ -424,6 +428,7 @@ int proc_pid_status(struct seq_file *m, struct pid_namespace *ns,
 	task_cpus_allowed(m, task);
 	cpuset_task_status_allowed(m, task);
 	task_context_switch_counts(m, task);
+	arch_proc_pid_status(m, task);
 	return 0;
 }
 
include/linux/proc_fs.h +2 -0
index d0e1f1522a78..1de9ba1b064f 100644
@@ -73,6 +73,8 @@ struct proc_dir_entry *proc_create_net_single_write(const char *name, umode_t mo
 						    int (*show)(struct seq_file *, void *),
 						    proc_write_t write,
 						    void *data);
+/* Add support for architecture specific output in /proc/pid/status */
+extern void arch_proc_pid_status(struct seq_file *m, struct task_struct *task);
 
 #else /* CONFIG_PROC_FS */
 
Signature · 1 line
2.17.1
#6 lore raw
Quoted 2 lines · Thomas Gleixner

include/linux/proc_fs.h struct proc_dir_entry *proc_create_net_single_write(const char *name, umode_t mo · line 77

Quoted 2 lines
include/linux/proc_fs.h +2 -0
+/* Add support for architecture specific output in /proc/pid/status */
+extern void arch_proc_pid_status(struct seq_file *m, struct task_struct *task);
     ^^^^^^

Unnecessary extern.

author
#7 lore raw
Quoted 7 lines · Alexey Dobriyan
Quoted 2 lines · Thomas Gleixner

include/linux/proc_fs.h struct proc_dir_entry *proc_create_net_single_write(const char *name, umode_t mo · line 77

Quoted 2 lines
include/linux/proc_fs.h +2 -0
+/* Add support for architecture specific output in /proc/pid/status */
+extern void arch_proc_pid_status(struct seq_file *m, struct task_struct *task);
     ^^^^^^

Unnecessary extern.

The linkage is default extern, but with this functions and variables can be treated the same way.

Is it mandatory not to use it explicitly? ./script/checkpatch.pl did not report this.

Thanks,
-Aubrey

#8 lore raw
Quoted 6 lines · Aubrey
Quoted 5 lines · Alexey Dobriyan
Quoted 2 lines · Thomas Gleixner

include/linux/proc_fs.h struct proc_dir_entry *proc_create_net_single_write(const char *name, umode_t mo · line 77

Quoted 2 lines
include/linux/proc_fs.h +2 -0
+/* Add support for architecture specific output in /proc/pid/status */
+extern void arch_proc_pid_status(struct seq_file *m, struct task_struct *task);
     ^^^^^^
Quoted 7 lines
Quoted 2 lines

Unnecessary extern.

The linkage is default extern, but with this functions and variables can be treated the same way.

Is it mandatory not to use it explicitly? ./script/checkpatch.pl did not report this.

"extern" is only necessary for variables. For prototypes, it is more typing and more characters on the screen.

checkpatch.pl doesn't report it because opening floodgates is not the plan.

Yours,
Extern Police.

author
#9 lore raw
Quoted 15 lines · Alexey Dobriyan
Quoted 6 lines · Aubrey
Quoted 5 lines · Alexey Dobriyan
Quoted 2 lines · Thomas Gleixner

include/linux/proc_fs.h struct proc_dir_entry *proc_create_net_single_write(const char *name, umode_t mo · line 77

Quoted 2 lines
include/linux/proc_fs.h +2 -0
+/* Add support for architecture specific output in /proc/pid/status */
+extern void arch_proc_pid_status(struct seq_file *m, struct task_struct *task);
     ^^^^^^
Quoted 7 lines
Quoted 2 lines

Unnecessary extern.

The linkage is default extern, but with this functions and variables can be treated the same way.

Is it mandatory not to use it explicitly? ./script/checkpatch.pl did not report this.

Quoted 8 lines

"extern" is only necessary for variables. For prototypes, it is more typing and more characters on the screen.

checkpatch.pl doesn't report it because opening floodgates is not the plan.

Yours,
Extern Police.

No problem, Mr. Police, I can remove it in the next version.

Thanks,
-Aubrey

#10 lore raw
In reply to Thomas Gleixner
Quoted 14 lines · Thomas Gleixner
Quoted 10 lines · Aubrey Li

The architecture specific information of the running processes could be useful to the userland. Add support to examine process architecture specific information externally.

Cc Peter Zijlstra <[email protected]>
Cc Andi Kleen <[email protected]>
Cc Tim Chen <[email protected]>
Cc Dave Hansen <[email protected]>
Cc Arjan van de Ven <[email protected]>
Quoted 8 lines

This really lacks

Cc Linux API <[email protected]>
Cc Alexey Dobriyan <[email protected]>
Cc Andrew Morton <[email protected]>

Cc'ed now.

I certainly understand why you want to expose this info, but would it make more sense to instead add an arch_status file in /proc with architecture-specific info? Or maybe an x86_status field for x86 status, etc.

author
#11 lore raw
Quoted 21 lines · Andy Lutomirski
Quoted 20 lines · Thomas Gleixner
Quoted 10 lines · Aubrey Li

The architecture specific information of the running processes could be useful to the userland. Add support to examine process architecture specific information externally.

Cc Peter Zijlstra <[email protected]>
Cc Andi Kleen <[email protected]>
Cc Tim Chen <[email protected]>
Cc Dave Hansen <[email protected]>
Cc Arjan van de Ven <[email protected]>

This really lacks

Cc Linux API <[email protected]>
Cc Alexey Dobriyan <[email protected]>
Cc Andrew Morton <[email protected]>
Quoted 8 lines
Quoted 2 lines

Cc'ed now.

I certainly understand why you want to expose this info, but would it make more sense to instead add an arch_status file in /proc with architecture-specific info? Or maybe an x86_status field for x86 status, etc.

I tried this, but no other architecture showed interest in arch_status under /proc.

Thanks,
-Aubrey

#12 lore raw
Quoted 26 lines · Aubrey
Quoted 24 lines · Andy Lutomirski
Quoted 22 lines · Thomas Gleixner
Quoted 10 lines · Aubrey Li

The architecture specific information of the running processes could be useful to the userland. Add support to examine process architecture specific information externally.

Cc Peter Zijlstra <[email protected]>
Cc Andi Kleen <[email protected]>
Cc Tim Chen <[email protected]>
Cc Dave Hansen <[email protected]>
Cc Arjan van de Ven <[email protected]>

This really lacks

Cc Linux API <[email protected]>
Cc Alexey Dobriyan <[email protected]>
Cc Andrew Morton <[email protected]>

Cc'ed now.

Quoted 9 lines
Quoted 5 lines

I certainly understand why you want to expose this info, but would it make more sense to instead add an arch_status file in /proc with architecture-specific info? Or maybe an x86_status field for x86 status, etc.

I tried this, but no other architecture showed interest in arch_status under /proc.

Why is that a problem? It could exist on x86 and not exist on other arches until needed.

author
#13 lore raw
Quoted 33 lines · Andy Lutomirski
Quoted 32 lines · Aubrey
Quoted 29 lines · Andy Lutomirski
Quoted 14 lines · Thomas Gleixner
Quoted 10 lines · Aubrey Li

The architecture specific information of the running processes could be useful to the userland. Add support to examine process architecture specific information externally.

Cc Peter Zijlstra <[email protected]>
Cc Andi Kleen <[email protected]>
Cc Tim Chen <[email protected]>
Cc Dave Hansen <[email protected]>
Cc Arjan van de Ven <[email protected]>
Quoted 8 lines

This really lacks

Cc Linux API <[email protected]>
Cc Alexey Dobriyan <[email protected]>
Cc Andrew Morton <[email protected]>

Cc'ed now.

I certainly understand why you want to expose this info, but would it make more sense to instead add an arch_status file in /proc with architecture-specific info? Or maybe an x86_status field for x86 status, etc.

Quoted 7 lines
Quoted 3 lines

I tried this, but no other architecture showed interest in arch_status under /proc.

Why is that a problem? It could exist on x86 and not exist on other arches until needed.

I placed it in tid_base_stuff, under live_patch entry, so it exists for all arches, is there a better way to do this?

Thanks,
-Aubrey