Skip to contentSkip to conversation

/proc/pid/status: Add support for architecture specific output

PATCH Superseded Aubrey Li 4 participants 8 messages Apr 16, 2019 – Apr 19, 2019

Subsystems (4)

v15 of 16 Superseded Latest is v16 →

lore
Share & tools
lore

Aubrey Li

author
#0 lore raw

PATCH v15 1/3 /proc/pid/status: Add support for architecture specific output

The architecture specific information of the running processes could be useful to the userland. Add support to examine process architecture specific information externally.

Cc Peter Zijlstra <[email protected]>
Cc Andi Kleen <[email protected]>
Cc Tim Chen <[email protected]>
Cc Dave Hansen <[email protected]>
Cc Arjan van de Ven <[email protected]>
Cc Linux API <[email protected]>
Cc Alexey Dobriyan <[email protected]>
Cc Andrew Morton <[email protected]>
---
 fs/proc/array.c | 6 ++++++
 1 file changed, 6 insertions(+)
fs/proc/array.c +6 -0
index 2edbb657f859..87bc7e882d35 100644
@@ -96,6 +96,11 @@
 #include <asm/processor.h>
 #include "internal.h"
 
+/* Add support for architecture specific output in /proc/pid/status */
+#ifndef	arch_proc_pid_status
+#define	arch_proc_pid_status(m, task)
+#endif
+
 void proc_task_name(struct seq_file *m, struct task_struct *p, bool escape)
 {
 	char *buf;
@@ -424,6 +429,7 @@ int proc_pid_status(struct seq_file *m, struct pid_namespace *ns,
 	task_cpus_allowed(m, task);
 	cpuset_task_status_allowed(m, task);
 	task_context_switch_counts(m, task);
+	arch_proc_pid_status(m, task);
 	return 0;
 }
 
Signature · 1 line
2.21.0
author
#1 lore raw

PATCH v15 2/3 x86,/proc/pid/status: Add AVX-512 usage elapsed time

AVX-512 components use could cause core turbo frequency drop. So it's useful to expose AVX-512 usage elapsed time as a heuristic hint for the user space job scheduler to cluster the AVX-512 using tasks together.

Tensorflow example:

$ while [ 1 ]; do cat /proc/tid/status | grep AVX; sleep 1; done
AVX512_elapsed_ms:      4
AVX512_elapsed_ms:      8
AVX512_elapsed_ms:      4

This means that 4 milliseconds have elapsed since the AVX512 usage of tensorflow task was detected when the task was scheduled out.

Or:

$ cat /proc/tid/status | grep AVX512_elapsed_ms
AVX512_elapsed_ms:      -1

The number '-1' indicates that no AVX512 usage recorded before thus the task unlikely has frequency drop issue.

User space tools may want to further check by:

$ perf stat --pid <pid> -e core_power.lvl2_turbo_license -- sleep 1

Performance counter stats for process id '3558':

     3,251,565,961      core_power.lvl2_turbo_license

       1.004031387 seconds time elapsed

Non-zero counter value confirms that the task causes frequency drop.

Cc Peter Zijlstra <[email protected]>
Cc Andi Kleen <[email protected]>
Cc Tim Chen <[email protected]>
Cc Dave Hansen <[email protected]>
Cc Arjan van de Ven <[email protected]>
Cc Linux API <[email protected]>
Cc Alexey Dobriyan <[email protected]>
Cc Andrew Morton <[email protected]>
---
 arch/x86/include/asm/processor.h |  4 +++
 arch/x86/kernel/fpu/xstate.c     | 42 ++++++++++++++++++++++++++++++++
 2 files changed, 46 insertions(+)
arch/x86/include/asm/processor.h +4 -0
index 2bb3a648fc12..5a7271ab78d8 100644
@@ -991,4 +991,8 @@ enum l1tf_mitigations {
 
 extern enum l1tf_mitigations l1tf_mitigation;
 
+/* Add support for architecture specific output in /proc/pid/status */
+void arch_proc_pid_status(struct seq_file *m, struct task_struct *task);
+#define arch_proc_pid_status arch_proc_pid_status
+
 #endif /* _ASM_X86_PROCESSOR_H */
arch/x86/kernel/fpu/xstate.c +42 -0

51 lines not on this page · raw message

Signature · 1 line
2.21.0
author
#2 lore raw

PATCH v15 3/3 Documentation/filesystems/proc.txt: add AVX512_elapsed_ms

In reply to Aubrey Li

Added AVX512_elapsed_ms in /proc/<pid>/status. Report it in Documentation/filesystems/proc.txt

Cc Peter Zijlstra <[email protected]>
Cc Andi Kleen <[email protected]>
Cc Tim Chen <[email protected]>
Cc Dave Hansen <[email protected]>
Cc Arjan van de Ven <[email protected]>
Cc Linux API <[email protected]>
Cc Alexey Dobriyan <[email protected]>
Cc Andrew Morton <[email protected]>
---
 Documentation/filesystems/proc.txt | 29 ++++++++++++++++++++++++++++-
 1 file changed, 28 insertions(+), 1 deletion(-)
Documentation/filesystems/proc.txt +28 -1
index 66cad5c86171..c4a9e48681ad 100644
@@ -207,6 +207,7 @@ read the file /proc/PID/status:
   Speculation_Store_Bypass:       thread vulnerable
   voluntary_ctxt_switches:        0
   nonvoluntary_ctxt_switches:     1
+  AVX512_elapsed_ms:	8
 
 This shows you nearly the same information you would get if you viewed it with
 the ps  command.  In  fact,  ps  uses  the  proc  file  system  to  obtain its
@@ -224,7 +225,7 @@ asynchronous manner and the value may not be very precise. To see a precise
 snapshot of a moment, you can see /proc/<pid>/smaps file and scan page table.
 It's slow but very precise.
 
-Table 1-2: Contents of the status files (as of 4.19)
+Table 1-2: Contents of the status files (as of 5.1)
 ..............................................................................
  Field                       Content
  Name                        filename of the executable
@@ -289,6 +290,32 @@ Table 1-2: Contents of the status files (as of 4.19)
  Mems_allowed_list           Same as previous, but in "list format"
  voluntary_ctxt_switches     number of voluntary context switches
  nonvoluntary_ctxt_switches  number of non voluntary context switches
+ AVX512_elapsed_ms           time elapsed since last AVX512 usage recorded
+
+ AVX512_elapsed_ms:
+ ------------------
+  If AVX512 is supported on the machine, this entry shows the milliseconds
+  elapsed since the last time AVX512 usage was recorded. The recording
+  happens on a best effort basis when a task is scheduled out. This means
+  that the value depends on two factors:
+
+    1) The time which the task spent on the CPU without being scheduled
+       out. With CPU isolation and a single runnable task this can take
+       several seconds.
+
+    2) The time since the task was scheduled out last. Depending on the
+       reason for being scheduled out (time slice exhausted, syscall ...)
+       this can be arbitrary long time.
+
+  As a consequence the value cannot be considered precise and authoritative
+  information. The application which uses this information has to be aware
+  of the overall scenario on the system in order to determine whether a
+  task is a real AVX512 user or not.
+
+  A special value of '-1' indicates that no AVX512 usage was recorded, thus
+  the task is unlikely an AVX512 user, but depends on the workload and the
+  scheduling scenario, it also could be a false negative mentioned above.
+
 ..............................................................................
 
 Table 1-3: Contents of the statm files (as of 2.6.8-rc3)
Signature · 1 line
2.21.0
#3 lore raw
In reply to Aubrey Li
Quoted 3 lines · On Tue, 16 Apr 2019 14:32:48 +0800 Aubrey Li <[email protected]> wrote:

The architecture specific information of the running processes could be useful to the userland. Add support to examine process architecture specific information externally.

The implementation looks just fine to me. Have you had any feedback on the overall desirability of adding this feature?

fs/proc/array.c line 102

Quoted 10 lines
fs/proc/array.c +4 -0
@@ -96,6 +96,11 @@
 #include <asm/processor.h>
 #include "internal.h"
 
+/* Add support for architecture specific output in /proc/pid/status */
+#ifndef	arch_proc_pid_status
+#define	arch_proc_pid_status(m, task)
+#endif

To this I suggest adding

/* arch_proc_pid_status() must be defined in asm/processor.h */

Because we've regularly had different architectures defining such things in different headers, resulting in a mess.

author
#4 lore raw
Quoted 9 lines · Andrew Morton
Quoted 3 lines · On Tue, 16 Apr 2019 14:32:48 +0800 Aubrey Li <[email protected]> wrote:

The architecture specific information of the running processes could be useful to the userland. Add support to examine process architecture specific information externally.

The implementation looks just fine to me. Have you had any feedback on the overall desirability of adding this feature?

This is orientated by the customer's complain. When the customer colocated their latency critical tasks with AVX512 tasks, the latency critical tasks were affected due to core frequency slowing down by AVX512 use. So we propose this interface for the user space tools to identify AVX512 using task and apply user space dispatching policy. We may have subsequent effort based on this proposal.

Quoted 13 lines

fs/proc/array.c line 102

Quoted 10 lines
fs/proc/array.c +4 -0
@@ -96,6 +96,11 @@
 #include <asm/processor.h>
 #include "internal.h"
 
+/* Add support for architecture specific output in /proc/pid/status */
+#ifndef	arch_proc_pid_status
+#define	arch_proc_pid_status(m, task)
+#endif

To this I suggest adding

Quoted 4 lines

/* arch_proc_pid_status() must be defined in asm/processor.h */

Because we've regularly had different architectures defining such things in different headers, resulting in a mess.

Thanks, I'll add it in the next version.
-Aubrey

#5 lore raw
In reply to Andrew Morton
Quoted 9 lines · Andrew Morton
Quoted 3 lines · On Tue, 16 Apr 2019 14:32:48 +0800 Aubrey Li <[email protected]> wrote:

The architecture specific information of the running processes could be useful to the userland. Add support to examine process architecture specific information externally.

The implementation looks just fine to me. Have you had any feedback on the overall desirability of adding this feature?

I think I've been the most outspoken, and my not-all-that-strong opinion is that I don't really like it. /proc/PID/status is already a bit of a mess, and I don't think we really want it to be a mess that is different on different architectures. Hence my suggestion of /proc/PID/x86_status instead. Or we could do /proc/PID/arch_status, I suppose, and make sure that everything in it is namespaced. (We could easily end up with a situation where status fields from more than one architecture are present. Think i386 + x86_64. Thata's also the case where qemu userspace emulation is used to run binaries meant for one architecture on a different architecture's kernel.)

--Andy

#6 lore raw
Quoted 8 lines · Andy Lutomirski
Quoted 7 lines · Andrew Morton
Quoted 3 lines · On Tue, 16 Apr 2019 14:32:48 +0800 Aubrey Li <[email protected]> wrote:

The architecture specific information of the running processes could be useful to the userland. Add support to examine process architecture specific information externally.

Quoted 8 lines
Quoted 2 lines

The implementation looks just fine to me. Have you had any feedback on the overall desirability of adding this feature?

I think I've been the most outspoken, and my not-all-that-strong opinion is that I don't really like it. /proc/PID/status is already a bit of a mess, and I don't think we really want it to be a mess that is different on different architectures. Hence my suggestion of /proc/PID/x86_status instead. Or we could do /proc/PID/arch_status, I

arch_status looks like the right thing to do.

Thanks,

tglx

author
#7 lore raw
Quoted 13 lines · Thomas Gleixner
Quoted 11 lines · Andy Lutomirski
Quoted 9 lines · Andrew Morton
Quoted 3 lines · On Tue, 16 Apr 2019 14:32:48 +0800 Aubrey Li <[email protected]> wrote:

The architecture specific information of the running processes could be useful to the userland. Add support to examine process architecture specific information externally.

The implementation looks just fine to me. Have you had any feedback on the overall desirability of adding this feature?

Quoted 7 lines
Quoted 5 lines

I think I've been the most outspoken, and my not-all-that-strong opinion is that I don't really like it. /proc/PID/status is already a bit of a mess, and I don't think we really want it to be a mess that is different on different architectures. Hence my suggestion of /proc/PID/x86_status instead. Or we could do /proc/PID/arch_status, I

arch_status looks like the right thing to do.

Thanks Andy and Thomas, let me change the patch to use arch_status instead.