Skip to contentSkip to conversation

Don't show PF_IO_WORKER in /proc/<pid>/task/

PATCH In review Jens Axboe 6 participants 31 messages Mar 25, 2021 – Apr 1, 2021

Subsystems (7)
lore
Share & tools
lore

Jens Axboe

author maintainer
#0 lore raw

cover Don't show PF_IO_WORKER in /proc/<pid>/task/

Hi,

Stefan reports that attaching to a task with io_uring will leave gdb very confused and just repeatedly attempting to attach to the IO threads, even though it receives an -EPERM every time. This patchset proposes to skip PF_IO_WORKER threads as same_thread_group(), except for accounting purposes which we still desire.

We also skip listing the IO threads in /proc/<pid>/task/ so that gdb doesn't think it should stop and attach to them. This makes us consistent with earlier kernels, where these async threads were not related to the ring owning task, and hence gdb (and others) ignored them anyway.

Seems to me that this is the right approach, but open to comments on if others agree with this. Oleg, I did see your messages as well on SIGSTOP, and as was discussed with Eric as well, this is something we most certainly can revisit. I do think that the visibility of these threads is a separate issue. Even with SIGSTOP implemented (which I did try as well), we're never going to allow ptrace attach and hence gdb would still be broken. Hence I'd rather treat them as separate issues to attack.

Signature · 1 line
Jens Axboe
author maintainer
#1 lore raw

PATCH 1/2 kernel: don't include PF_IO_WORKERs as part of same_thread_group()

Don't pretend that the IO threads are in the same thread group, the only case where that seems to be desired is for accounting purposes. Add a special accounting function for that and make the scheduler side use it.

For signals and ptrace, we don't allow them to be treated as threads anyway.

---
 include/linux/sched/signal.h | 9 ++++++++-
 kernel/sched/cputime.c       | 2 +-
 2 files changed, 9 insertions(+), 2 deletions(-)
include/linux/sched/signal.h +8 -1
index 3f6a0fcaa10c..4f621e386abf 100644
@@ -668,11 +668,18 @@ static inline bool thread_group_leader(struct task_struct *p)
 }
 
 static inline
-bool same_thread_group(struct task_struct *p1, struct task_struct *p2)
+bool same_thread_group_account(struct task_struct *p1, struct task_struct *p2)
 {
 	return p1->signal == p2->signal;
 }
 
+static inline
+bool same_thread_group(struct task_struct *p1, struct task_struct *p2)
+{
+	return same_thread_group_account(p1, p2) &&
+			!((p1->flags | p2->flags) & PF_IO_WORKER);
+}
+
 static inline struct task_struct *next_thread(const struct task_struct *p)
 {
 	return list_entry_rcu(p->thread_group.next,
kernel/sched/cputime.c +1 -1
index 5f611658eeab..625110cacc2a 100644
@@ -307,7 +307,7 @@ void thread_group_cputime(struct task_struct *tsk, struct task_cputime *times)
 	 * those pending times and rely only on values updated on tick or
 	 * other scheduler action.
 	 */
-	if (same_thread_group(current, tsk))
+	if (same_thread_group_account(current, tsk))
 		(void) task_sched_runtime(current);
 
 	rcu_read_lock();
Signature · 1 line
2.31.0
author maintainer
#2 lore raw

PATCH 2/2 proc: don't show PF_IO_WORKER threads as threads in /proc/<pid>/task/

In reply to Jens Axboe

We don't allow SIGSTOP and ptrace attach to these threads, and that confuses applications like gdb that assume they can attach to any thread listed in /proc/<pid>/task/. gdb then enters an infinite loop of retrying attach, even though it fails with the same error (-EPERM) every time.

Skip over PF_IO_WORKER threads in the proc task setup. We can't just terminate the when we find a PF_IO_WORKER thread, as there's no real ordering here. It's perfectly feasible to have the first thread be an IO worker, and then a real thread after that. Hence just implement the skip.

---
 fs/proc/base.c | 22 ++++++++++++++--------
 1 file changed, 14 insertions(+), 8 deletions(-)
fs/proc/base.c +14 -8
index 3851bfcdba56..abff2fe10bfa 100644
@@ -3723,7 +3723,7 @@ static struct task_struct *first_tid(struct pid *pid, int tid, loff_t f_pos,
 	 */
 	pos = task = task->group_leader;
 	do {
-		if (!nr--)
+		if (same_thread_group(task, pos) && !nr--)
 			goto found;
 	} while_each_thread(task, pos);
 fail:
@@ -3744,16 +3744,22 @@ static struct task_struct *first_tid(struct pid *pid, int tid, loff_t f_pos,
  */
 static struct task_struct *next_tid(struct task_struct *start)
 {
-	struct task_struct *pos = NULL;
+	struct task_struct *tmp, *pos = NULL;
+
 	rcu_read_lock();
-	if (pid_alive(start)) {
-		pos = next_thread(start);
-		if (thread_group_leader(pos))
-			pos = NULL;
-		else
-			get_task_struct(pos);
+	if (!pid_alive(start))
+		goto no_thread;
+	list_for_each_entry_rcu(tmp, &start->thread_group, thread_group) {
+		if (!thread_group_leader(tmp) && same_thread_group(start, tmp)) {
+			get_task_struct(tmp);
+			pos = tmp;
+			break;
+		}
 	}
+no_thread:
 	rcu_read_unlock();
+	if (!pos)
+		return NULL;
 	put_task_struct(start);
 	return pos;
 }
Signature · 1 line
2.31.0
#4 lore raw
In reply to Jens Axboe
Quoted 13 lines · Jens Axboe

Hi,

Stefan reports that attaching to a task with io_uring will leave gdb very confused and just repeatedly attempting to attach to the IO threads, even though it receives an -EPERM every time. This patchset proposes to skip PF_IO_WORKER threads as same_thread_group(), except for accounting purposes which we still desire.

We also skip listing the IO threads in /proc/<pid>/task/ so that gdb doesn't think it should stop and attach to them. This makes us consistent with earlier kernels, where these async threads were not related to the ring owning task, and hence gdb (and others) ignored them anyway.

Quoted 7 lines

Seems to me that this is the right approach, but open to comments on if others agree with this. Oleg, I did see your messages as well on SIGSTOP, and as was discussed with Eric as well, this is something we most certainly can revisit. I do think that the visibility of these threads is a separate issue. Even with SIGSTOP implemented (which I did try as well), we're never going to allow ptrace attach and hence gdb would still be broken. Hence I'd rather treat them as separate issues to attack.

A quick skim shows that these threads are not showing up anywhere in proc which appears to be a problem, as it hides them from top.

Sysadmins need the ability to dig into a system and find out where all their cpu usage or io's have gone when there is a problem. I general I think this argues that these threads should show up as threads of the process so I am not even certain this is the right fix to deal with gdb.

Eric

#5 lore raw
Quoted 8 lines · Eric W. Biederman

A quick skim shows that these threads are not showing up anywhere in proc which appears to be a problem, as it hides them from top.

Sysadmins need the ability to dig into a system and find out where all their cpu usage or io's have gone when there is a problem. I general I think this argues that these threads should show up as threads of the process so I am not even certain this is the right fix to deal with gdb.

Yeah, I do think that hiding them is the wrong model, because it also hides them from "ps" etc, which is very wrong.

I don't know what the gdb logic is, but maybe there's some other option that makes gdb not react to them?

Linus

author maintainer
#29 lore raw
In reply to Eric W. Biederman
Quoted 23 lines · Eric W. Biederman
Quoted 13 lines · Jens Axboe

Hi,

Stefan reports that attaching to a task with io_uring will leave gdb very confused and just repeatedly attempting to attach to the IO threads, even though it receives an -EPERM every time. This patchset proposes to skip PF_IO_WORKER threads as same_thread_group(), except for accounting purposes which we still desire.

We also skip listing the IO threads in /proc/<pid>/task/ so that gdb doesn't think it should stop and attach to them. This makes us consistent with earlier kernels, where these async threads were not related to the ring owning task, and hence gdb (and others) ignored them anyway.

Quoted 7 lines

Seems to me that this is the right approach, but open to comments on if others agree with this. Oleg, I did see your messages as well on SIGSTOP, and as was discussed with Eric as well, this is something we most certainly can revisit. I do think that the visibility of these threads is a separate issue. Even with SIGSTOP implemented (which I did try as well), we're never going to allow ptrace attach and hence gdb would still be broken. Hence I'd rather treat them as separate issues to attack.

Quoted 7 lines

A quick skim shows that these threads are not showing up anywhere in proc which appears to be a problem, as it hides them from top.

Sysadmins need the ability to dig into a system and find out where all their cpu usage or io's have gone when there is a problem. I general I think this argues that these threads should show up as threads of the process so I am not even certain this is the right fix to deal with gdb.

That's a good point, overall hiding was not really what I desired, just getting them out of gdb's hands. And arguably it _is_ a gdb bug, but...

Signature · 1 line
Jens Axboe
author maintainer
#6 lore raw
In reply to Linus Torvalds
Quoted 6 lines · Linus Torvalds
Quoted 4 lines · Eric W. Biederman

A quick skim shows that these threads are not showing up anywhere in proc which appears to be a problem, as it hides them from top.

Quoted 7 lines
Quoted 4 lines

Sysadmins need the ability to dig into a system and find out where all their cpu usage or io's have gone when there is a problem. I general I think this argues that these threads should show up as threads of the process so I am not even certain this is the right fix to deal with gdb.

Yeah, I do think that hiding them is the wrong model, because it also hides them from "ps" etc, which is very wrong.

Totally agree.

Quoted 2 lines

I don't know what the gdb logic is, but maybe there's some other option that makes gdb not react to them?

Guess it's time to dig out the gdb source... I'll take a look.

Signature · 1 line
Jens Axboe
#7 lore raw
In reply to Linus Torvalds
Quoted 3 lines · Linus Torvalds

I don't know what the gdb logic is, but maybe there's some other option that makes gdb not react to them?

.. maybe we could have a different name for them under the task/ subdirectory, for example (not just the pid)? Although that probably messes up 'ps' too..

Linus

author maintainer
#8 lore raw
Quoted 9 lines · Linus Torvalds
Quoted 3 lines · Linus Torvalds

I don't know what the gdb logic is, but maybe there's some other option that makes gdb not react to them?

.. maybe we could have a different name for them under the task/ subdirectory, for example (not just the pid)? Although that probably messes up 'ps' too..

Heh, I can try, but my guess is that it would mess up _something_, if not ps/top.

Signature · 1 line
Jens Axboe
#14 lore raw
In reply to Linus Torvalds
Quoted 10 lines · Linus Torvalds
Quoted 3 lines · Linus Torvalds

I don't know what the gdb logic is, but maybe there's some other option that makes gdb not react to them?

.. maybe we could have a different name for them under the task/ subdirectory, for example (not just the pid)? Although that probably messes up 'ps' too..

Actually, maybe the right model is to simply make all the io threads take signals, and get rid of all the special cases.

Sure, the signals will never be delivered to user space, but if we

- just made the thread loop do "get_signal()" when there are pending signals

- allowed ptrace_attach on them

they'd look pretty much like regular threads that just never do the user-space part of signal handling.

The whole "signals are very special for IO threads" thing has caused so many problems, that maybe the solution is simply to _not_ make them special?

Linus

#9 lore raw
In reply to Jens Axboe
Quoted 7 lines · Jens Axboe
Quoted 6 lines · Linus Torvalds
Quoted 3 lines · Linus Torvalds

I don't know what the gdb logic is, but maybe there's some other option that makes gdb not react to them?

Quoted 6 lines
Quoted 3 lines

.. maybe we could have a different name for them under the task/ subdirectory, for example (not just the pid)? Although that probably messes up 'ps' too..

Heh, I can try, but my guess is that it would mess up _something_, if not ps/top.

Hmm.

So looking quickly the flip side of the coin is gdb (and other debuggers) needs a way to know these threads are special, so it can know not to attach.

I suspect getting -EPERM (or possibly a different error code) when attempting attach is the right was to know that a thread is not available to be debugged.

Eric

maintainer
#30 lore raw
In reply to Jens Axboe

I didn't even try to read this series yet, will try tomorrow.

But sorry, I can't resist...

Quoted 9 lines · Jens Axboe
Quoted 7 lines · Eric W. Biederman
Quoted 5 lines · Jens Axboe

Hi,

Stefan reports that attaching to a task with io_uring will leave gdb very confused and just repeatedly attempting to attach to the IO threads, even though it receives an -EPERM every time.

Heh. As expected :/

Quoted 1 line

And arguably it _is_ a gdb bug, but...

Why do you think so?

Oleg.

maintainer
#10 lore raw
In reply to Eric W. Biederman
Quoted 4 lines · Eric W. Biederman

So looking quickly the flip side of the coin is gdb (and other debuggers) needs a way to know these threads are special, so it can know not to attach.

may be,

Quoted 3 lines

I suspect getting -EPERM (or possibly a different error code) when attempting attach is the right was to know that a thread is not available to be debugged.

may be.

But I don't think we can blame gdb. The kernel changed the rules, and this broke gdb. IOW, I don't agree this is gdb bug.

Oleg.

author maintainer
#15 lore raw
In reply to Linus Torvalds
Quoted 20 lines · Linus Torvalds
Quoted 10 lines · Linus Torvalds
Quoted 3 lines · Linus Torvalds

I don't know what the gdb logic is, but maybe there's some other option that makes gdb not react to them?

.. maybe we could have a different name for them under the task/ subdirectory, for example (not just the pid)? Although that probably messes up 'ps' too..

Actually, maybe the right model is to simply make all the io threads take signals, and get rid of all the special cases.

Sure, the signals will never be delivered to user space, but if we

- just made the thread loop do "get_signal()" when there are pending signals

Quoted 8 lines

- allowed ptrace_attach on them

they'd look pretty much like regular threads that just never do the user-space part of signal handling.

The whole "signals are very special for IO threads" thing has caused so many problems, that maybe the solution is simply to _not_ make them special?

Just to wrap up the previous one, yes it broke all sorts of things to make the 'tid' directory different. They just end up being hidden anyway through that, for both ps and top.

Yes, I do think that maybe it's better to just embrace maybe just embrace the signals, and have everything just work by default. It's better than continually trying to make the threads special. I'll see if there are some demons lurking down that path.

Signature · 1 line
Jens Axboe
author maintainer
#13 lore raw
In reply to Eric W. Biederman
Quoted 18 lines · Eric W. Biederman
Quoted 7 lines · Jens Axboe
Quoted 6 lines · Linus Torvalds
Quoted 3 lines · Linus Torvalds

I don't know what the gdb logic is, but maybe there's some other option that makes gdb not react to them?

Quoted 6 lines
Quoted 3 lines

.. maybe we could have a different name for them under the task/ subdirectory, for example (not just the pid)? Although that probably messes up 'ps' too..

Heh, I can try, but my guess is that it would mess up _something_, if not ps/top.

Hmm.

Quoted 7 lines

So looking quickly the flip side of the coin is gdb (and other debuggers) needs a way to know these threads are special, so it can know not to attach.

I suspect getting -EPERM (or possibly a different error code) when attempting attach is the right was to know that a thread is not available to be debugged.

But that's what's being returned right now, and gdb seemingly doesn't really handle that. And even if it was just a gdb issue that could be fixed it (it definitely is), I'd still greatly prefer not having to do that. It just takes too long for packages to get updated in distros, and it'd be years until it got fixed widely. Secondly, I'm even more worried about cases that we haven't seen yet. I doubt that gdb is the only thing that'd fall over, not expecting threads in there that it cannot attach to.

Signature · 1 line
Jens Axboe
author maintainer
#11 lore raw
In reply to Oleg Nesterov
Quoted 8 lines · Oleg Nesterov
Quoted 4 lines · Eric W. Biederman

So looking quickly the flip side of the coin is gdb (and other debuggers) needs a way to know these threads are special, so it can know not to attach.

may be,

Quoted 8 lines
Quoted 3 lines

I suspect getting -EPERM (or possibly a different error code) when attempting attach is the right was to know that a thread is not available to be debugged.

may be.

But I don't think we can blame gdb. The kernel changed the rules, and this broke gdb. IOW, I don't agree this is gdb bug.

Right, that's what I was getting at too - and it's likely not just gdb. We have to ensure that we don't break this use case, which seems to imply that we:

1) Just make it work, or
2) Make them hidden in such a way that gdb doesn't see them, but regular tooling does

#2 seems fraught with peril, and maybe not even possible.

Signature · 1 line
Jens Axboe
maintainer
#23 lore raw
In reply to Linus Torvalds
Quoted 4 lines · Linus Torvalds

The whole "signals are very special for IO threads" thing has caused so many problems, that maybe the solution is simply to _not_ make them special?

Or may be IO threads should not abuse CLONE_THREAD?

Why does create_io_thread() abuse CLONE_THREAD ?

One reason (I think) is that this implies SIGKILL when the process exits/execs, anything else?

Oleg.

#27 lore raw
In reply to Linus Torvalds
Quoted 20 lines · Linus Torvalds
Quoted 10 lines · Linus Torvalds
Quoted 3 lines · Linus Torvalds

I don't know what the gdb logic is, but maybe there's some other option that makes gdb not react to them?

.. maybe we could have a different name for them under the task/ subdirectory, for example (not just the pid)? Although that probably messes up 'ps' too..

Actually, maybe the right model is to simply make all the io threads take signals, and get rid of all the special cases.

Sure, the signals will never be delivered to user space, but if we

- just made the thread loop do "get_signal()" when there are pending signals

Quoted 8 lines

- allowed ptrace_attach on them

they'd look pretty much like regular threads that just never do the user-space part of signal handling.

The whole "signals are very special for IO threads" thing has caused so many problems, that maybe the solution is simply to _not_ make them special?

The special case in check_kill_permission is certainly unnecessary. Having the signal blocked is enough to prevent signal_pending() from being true.

The most straight forward thing I can see is to allow ptrace_attach and to modify ptrace_check_attach to always return -ESRCH for io workers unless ignore_state is set causing none of the other ptrace operations to work.

That is what a long running in-kernel thread would do today so user-space aka gdb may actually cope with it.

We might be able to support if io workers start supporting SIGSTOP but I am not at all certain.

Eric

#12 lore raw
In reply to Oleg Nesterov
Quoted 8 lines · Oleg Nesterov
Quoted 4 lines · Eric W. Biederman

So looking quickly the flip side of the coin is gdb (and other debuggers) needs a way to know these threads are special, so it can know not to attach.

may be,

Quoted 8 lines
Quoted 3 lines

I suspect getting -EPERM (or possibly a different error code) when attempting attach is the right was to know that a thread is not available to be debugged.

may be.

But I don't think we can blame gdb. The kernel changed the rules, and this broke gdb. IOW, I don't agree this is gdb bug.

My point would be it is not strictly a regression either. It is gdb not handling new functionality.

If we can be backwards compatible and make ptrace_attach work that is preferable. If we can't saying the handful of ptrace using applications need an upgrade to support processes that use io_uring may be acceptable.

I don't see any easy to implement path that is guaranteed to work.

Eric

#24 lore raw
In reply to Oleg Nesterov
Quoted 6 lines · Oleg Nesterov
Quoted 4 lines · Linus Torvalds

The whole "signals are very special for IO threads" thing has caused so many problems, that maybe the solution is simply to _not_ make them special?

Quoted 6 lines

Or may be IO threads should not abuse CLONE_THREAD?

Why does create_io_thread() abuse CLONE_THREAD ?

One reason (I think) is that this implies SIGKILL when the process exits/execs, anything else?

A lot.

The io workers perform work on behave of the ordinary userspace threads. Some of that work is opening files. For things like rlimits to work properly you need to share the signal_struct. But odds are if you find anything in signal_struct (not counting signals) there will be an io_uring code path that can exercise it as io_uring can traverse the filesystem, open files and read/write files. So io_uring can exercise all of proc.

Using create_io_thread with CLONE_THREAD is the least problematic way (including all of the signal and ptrace problems we are looking at right now) to implement the io worker threads.

They _really_ are threads of the process that just never execute any code in userspace.

Eric

#25 lore raw

Am 25.03.21 um 21:55 schrieb Eric W. Biederman:

Quoted 25 lines
Quoted 6 lines · Oleg Nesterov
Quoted 4 lines · Linus Torvalds

The whole "signals are very special for IO threads" thing has caused so many problems, that maybe the solution is simply to _not_ make them special?

Quoted 6 lines

Or may be IO threads should not abuse CLONE_THREAD?

Why does create_io_thread() abuse CLONE_THREAD ?

One reason (I think) is that this implies SIGKILL when the process exits/execs, anything else?

A lot.

The io workers perform work on behave of the ordinary userspace threads. Some of that work is opening files. For things like rlimits to work properly you need to share the signal_struct. But odds are if you find anything in signal_struct (not counting signals) there will be an io_uring code path that can exercise it as io_uring can traverse the filesystem, open files and read/write files. So io_uring can exercise all of proc.

Quoted 6 lines

Using create_io_thread with CLONE_THREAD is the least problematic way (including all of the signal and ptrace problems we are looking at right now) to implement the io worker threads.

They _really_ are threads of the process that just never execute any code in userspace.

So they should look like a userspace thread sitting in something like epoll_pwait() with all signals blocked, which will never return to userspace again?

I think that would be useful, but I also think that userspace should see:
- /proc/$tidofiothread/cmdline as empty (in order to let ps and top use [iou-wrk-$tidofuserspacethread])
- /proc/$tidofiothread/exe as symlink to that not exists
- all of /proc/$tidofiothread/ shows root.root as owner and group
and things which still allow write access to /proc/$tidofiothread/comm similar things with rw permissions should still disallow modifications:

For the other kernel threads e.g. "[cryptd]" I see the following:

LANG=C ls -l /proc/653 | grep rw
ls: cannot read symbolic link '/proc/653/exe': No such file or directory

-rw-r--r--  1 root root 0 Mar 25 22:09 autogroup
-rw-r--r--  1 root root 0 Mar 25 22:09 comm
-rw-r--r--  1 root root 0 Mar 25 22:09 coredump_filter
lrwxrwxrwx  1 root root 0 Mar 25 22:09 cwd -> /
lrwxrwxrwx  1 root root 0 Mar 25 22:09 exe
-rw-r--r--  1 root root 0 Mar 25 22:09 gid_map
-rw-r--r--  1 root root 0 Mar 25 22:09 loginuid
-rw-------  1 root root 0 Mar 25 22:09 mem
-rw-r--r--  1 root root 0 Mar 25 22:09 oom_adj
-rw-r--r--  1 root root 0 Mar 25 22:09 oom_score_adj
-rw-r--r--  1 root root 0 Mar 25 22:09 projid_map
lrwxrwxrwx  1 root root 0 Mar 25 22:09 root -> /
-rw-r--r--  1 root root 0 Mar 25 22:09 sched
-rw-r--r--  1 root root 0 Mar 25 22:09 setgroups
-rw-r--r--  1 root root 0 Mar 25 22:09 timens_offsets
-rw-rw-rw-  1 root root 0 Mar 25 22:09 timerslack_ns
-rw-r--r--  1 root root 0 Mar 25 22:09 uid_map

And this:

LANG=C echo "bla" > /proc/653/comm
-bash: echo: write error: Invalid argument

LANG=C echo "bla" > /proc/653/gid_map
-bash: echo: write error: Operation not permitted

Can't we do the same for iothreads regarding /proc? Just make things read only there and empty "cmdline"/"exe"?

Maybe I'm too naive, but that what I'd assume as a userspace developer/admin.

Does at least parts of it make any sense?

metze

author maintainer
#16 lore raw
In reply to Jens Axboe
Quoted 30 lines · Jens Axboe
Quoted 20 lines · Linus Torvalds
Quoted 10 lines · Linus Torvalds
Quoted 3 lines · Linus Torvalds

I don't know what the gdb logic is, but maybe there's some other option that makes gdb not react to them?

.. maybe we could have a different name for them under the task/ subdirectory, for example (not just the pid)? Although that probably messes up 'ps' too..

Actually, maybe the right model is to simply make all the io threads take signals, and get rid of all the special cases.

Sure, the signals will never be delivered to user space, but if we

- just made the thread loop do "get_signal()" when there are pending signals

Quoted 8 lines

- allowed ptrace_attach on them

they'd look pretty much like regular threads that just never do the user-space part of signal handling.

The whole "signals are very special for IO threads" thing has caused so many problems, that maybe the solution is simply to _not_ make them special?

Quoted 8 lines

Just to wrap up the previous one, yes it broke all sorts of things to make the 'tid' directory different. They just end up being hidden anyway through that, for both ps and top.

Yes, I do think that maybe it's better to just embrace maybe just embrace the signals, and have everything just work by default. It's better than continually trying to make the threads special. I'll see if there are some demons lurking down that path.

In the spirit of "let's just try it", I ran with the below patch. With that, I can gdb attach just fine to a test case that creates an io_uring and a regular thread with pthread_create(). The regular thread uses the ring, so you end up with two iou-mgr threads. Attach:

[root@archlinux ~]# gdb -p 360
[snip gdb noise]
Attaching to process 360
[New LWP 361]
[New LWP 362]
[New LWP 363]

warning: Selected architecture i386:x86-64 is not compatible with reported target architecture i386

warning: Architecture rejected target-supplied description
Error while reading shared library symbols for /usr/lib/libpthread.so.0:
Cannot find user-level thread for LWP 363: generic error 0x00007f7aa526e125 in clock_nanosleep@GLIBC_2.2.5 () from /usr/lib/libc.so.6 (gdb) info threads

  Id   Target Id             Frame 
* 1    LWP 360 "io_uring"    0x00007f7aa526e125 in clock_nanosleep@GLIBC_2.2.5 ()
   from /usr/lib/libc.so.6
  2    LWP 361 "iou-mgr-360" 0x0000000000000000 in ?? ()
  3    LWP 362 "io_uring"    0x00007f7aa52a0a9d in syscall () from /usr/lib/libc.so.6
  4    LWP 363 "iou-mgr-362" 0x0000000000000000 in ?? ()

(gdb) thread 2
[Switching to thread 2 (LWP 361)]
#0 0x0000000000000000 in ?? ()
(gdb) bt
#0 0x0000000000000000 in ?? ()
Backtrace stopped: Cannot access memory at address 0x0
(gdb) cont
Continuing.
^C
Thread 1 "io_uring" received signal SIGINT, Interrupt. [Switching to LWP 360]
0x00007f7aa526e125 in clock_nanosleep@GLIBC_2.2.5 () from /usr/lib/libc.so.6 (gdb) q
A debugging session is active.

Inferior 1 [process 360] will be detached.

Quit anyway? (y or n) y
Detaching from program: /root/git/fio/t/io_uring, process 360 [Inferior 1 (process 360) detached]

The iou-mgr-x threads are stopped just fine, gdb obviously can't get any real info out of them. But it works... Regular test cases work fine too, just a sanity check. Didn't expect them not to.

Only thing that I dislike a bit, but I guess that's just a Linuxism, is that if can now kill an io_uring owning task by sending a signal to one of its IO thread workers.

fs/io-wq.c +17 -4
index b7c1fa932cb3..2dbdc552f3ba 100644
@@ -505,8 +505,14 @@ static int io_wqe_worker(void *data)
 		ret = schedule_timeout(WORKER_IDLE_TIMEOUT);
 		if (try_to_freeze() || ret)
 			continue;
-		if (fatal_signal_pending(current))
-			break;
+		if (signal_pending(current)) {
+			struct ksignal ksig;
+
+			if (fatal_signal_pending(current))
+				break;
+			get_signal(&ksig);
+			continue;
+		}
 		/* timed out, exit unless we're the fixed worker */
 		if (test_bit(IO_WQ_BIT_EXIT, &wq->state) ||
 		    !(worker->flags & IO_WORKER_F_FIXED))
@@ -715,8 +721,15 @@ static int io_wq_manager(void *data)
 		io_wq_check_workers(wq);
 		schedule_timeout(HZ);
 		try_to_freeze();
-		if (fatal_signal_pending(current))
-			set_bit(IO_WQ_BIT_EXIT, &wq->state);
+		if (signal_pending(current)) {
+			struct ksignal ksig;
+
+			if (fatal_signal_pending(current))
+				set_bit(IO_WQ_BIT_EXIT, &wq->state);
+			else
+				get_signal(&ksig);
+			continue;
+		}
 	} while (!test_bit(IO_WQ_BIT_EXIT, &wq->state));
 
 	io_wq_check_workers(wq);
fs/io_uring.c +8 -2
index 54ea561db4a5..3a9d021db328 100644
@@ -6765,8 +6765,14 @@ static int io_sq_thread(void *data)
 			timeout = jiffies + sqd->sq_thread_idle;
 			continue;
 		}
-		if (fatal_signal_pending(current))
-			break;
+		if (signal_pending(current)) {
+			struct ksignal ksig;
+
+			if (fatal_signal_pending(current))
+				break;
+			get_signal(&ksig);
+			continue;
+		}
 		sqt_spin = false;
 		cap_entries = !list_is_singular(&sqd->ctx_list);
 		list_for_each_entry(ctx, &sqd->ctx_list, sqd_list) {
kernel/fork.c +1 -0
index d3171e8e88e5..3b45d0f04044 100644
@@ -2436,6 +2436,7 @@ struct task_struct *create_io_thread(int (*fn)(void *), void *arg, int node)
 	if (!IS_ERR(tsk)) {
 		sigfillset(&tsk->blocked);
 		sigdelsetmask(&tsk->blocked, sigmask(SIGKILL));
+		sigdelsetmask(&tsk->blocked, sigmask(SIGSTOP));
 	}
 	return tsk;
 }
kernel/ptrace.c +1 -1
index 821cf1723814..61db50f7ca86 100644
@@ -375,7 +375,7 @@ static int ptrace_attach(struct task_struct *task, long request,
 	audit_ptrace(task);
 
 	retval = -EPERM;
-	if (unlikely(task->flags & (PF_KTHREAD | PF_IO_WORKER)))
+	if (unlikely(task->flags & PF_KTHREAD))
 		goto out;
 	if (same_thread_group(task, current))
 		goto out;
kernel/signal.c +2 -6
index f2a1b898da29..a5700557eb50 100644
@@ -91,7 +91,7 @@ static bool sig_task_ignored(struct task_struct *t, int sig, bool force)
 		return true;
 
 	/* Only allow kernel generated signals to this kthread */
-	if (unlikely((t->flags & (PF_KTHREAD | PF_IO_WORKER)) &&
+	if (unlikely((t->flags & PF_KTHREAD) &&
 		     (handler == SIG_KTHREAD_KERNEL) && !force))
 		return true;
 
@@ -288,8 +288,7 @@ bool task_set_jobctl_pending(struct task_struct *task, unsigned long mask)
 			JOBCTL_STOP_SIGMASK | JOBCTL_TRAPPING));
 	BUG_ON((mask & JOBCTL_TRAPPING) && !(mask & JOBCTL_PENDING_MASK));
 
-	if (unlikely(fatal_signal_pending(task) ||
-		     (task->flags & (PF_EXITING | PF_IO_WORKER))))
+	if (unlikely(fatal_signal_pending(task) || task->flags & PF_EXITING))
 		return false;
 
 	if (mask & JOBCTL_STOP_SIGMASK)
@@ -834,9 +833,6 @@ static int check_kill_permission(int sig, struct kernel_siginfo *info,
 
 	if (!valid_signal(sig))
 		return -EINVAL;
-	/* PF_IO_WORKER threads don't take any signals */
-	if (t->flags & PF_IO_WORKER)
-		return -ESRCH;
 
 	if (!si_fromuser(info))
 		return 0;
Signature · 1 line
Jens Axboe
#26 lore raw
In reply to Stefan Metzmacher

Am 25.03.21 um 22:20 schrieb Stefan Metzmacher:

Quoted 31 lines

Am 25.03.21 um 21:55 schrieb Eric W. Biederman:

Quoted 29 lines
Quoted 6 lines · Oleg Nesterov
Quoted 4 lines · Linus Torvalds

The whole "signals are very special for IO threads" thing has caused so many problems, that maybe the solution is simply to _not_ make them special?

Quoted 6 lines

Or may be IO threads should not abuse CLONE_THREAD?

Why does create_io_thread() abuse CLONE_THREAD ?

One reason (I think) is that this implies SIGKILL when the process exits/execs, anything else?

A lot.

The io workers perform work on behave of the ordinary userspace threads. Some of that work is opening files. For things like rlimits to work properly you need to share the signal_struct. But odds are if you find anything in signal_struct (not counting signals) there will be an io_uring code path that can exercise it as io_uring can traverse the filesystem, open files and read/write files. So io_uring can exercise all of proc.

Using create_io_thread with CLONE_THREAD is the least problematic way (including all of the signal and ptrace problems we are looking at right now) to implement the io worker threads.

Quoted 5 lines
Quoted 2 lines

They _really_ are threads of the process that just never execute any code in userspace.

So they should look like a userspace thread sitting in something like epoll_pwait() with all signals blocked, which will never return to userspace again?

Would gdb work with that?
The question is what backtrace gdb would show for that thread.

Is it possible to block SIGSTOP/SIGCONT?

I also think that all signals to an iothread should not be delivered to other threads and it may only react on a direct SIGSTOP/SIGCONT. I guess even SIGKILL should be ignored as the shutdown should happen via the exit path of the iothread parent only.

Quoted 37 lines

I think that would be useful, but I also think that userspace should see:
- /proc/$tidofiothread/cmdline as empty (in order to let ps and top use [iou-wrk-$tidofuserspacethread])
- /proc/$tidofiothread/exe as symlink to that not exists
- all of /proc/$tidofiothread/ shows root.root as owner and group
and things which still allow write access to /proc/$tidofiothread/comm similar things with rw permissions should still disallow modifications:

For the other kernel threads e.g. "[cryptd]" I see the following:

LANG=C ls -l /proc/653 | grep rw
ls: cannot read symbolic link '/proc/653/exe': No such file or directory

(quoted diff) +0 -3
-rw-r--r--  1 root root 0 Mar 25 22:09 autogroup
-rw-r--r--  1 root root 0 Mar 25 22:09 comm
-rw-r--r--  1 root root 0 Mar 25 22:09 coredump_filter

lrwxrwxrwx 1 root root 0 Mar 25 22:09 cwd -> /
lrwxrwxrwx 1 root root 0 Mar 25 22:09 exe

(quoted diff) +0 -6
-rw-r--r--  1 root root 0 Mar 25 22:09 gid_map
-rw-r--r--  1 root root 0 Mar 25 22:09 loginuid
-rw-------  1 root root 0 Mar 25 22:09 mem
-rw-r--r--  1 root root 0 Mar 25 22:09 oom_adj
-rw-r--r--  1 root root 0 Mar 25 22:09 oom_score_adj
-rw-r--r--  1 root root 0 Mar 25 22:09 projid_map

lrwxrwxrwx 1 root root 0 Mar 25 22:09 root -> /

(quoted diff) +0 -5
-rw-r--r--  1 root root 0 Mar 25 22:09 sched
-rw-r--r--  1 root root 0 Mar 25 22:09 setgroups
-rw-r--r--  1 root root 0 Mar 25 22:09 timens_offsets
-rw-rw-rw-  1 root root 0 Mar 25 22:09 timerslack_ns
-rw-r--r--  1 root root 0 Mar 25 22:09 uid_map

And this:

LANG=C echo "bla" > /proc/653/comm
-bash: echo: write error: Invalid argument

LANG=C echo "bla" > /proc/653/gid_map
-bash: echo: write error: Operation not permitted

Quoted 6 lines

Can't we do the same for iothreads regarding /proc? Just make things read only there and empty "cmdline"/"exe"?

Maybe I'm too naive, but that what I'd assume as a userspace developer/admin.

Does at least parts of it make any sense?

I think the strange glibc setuid() behavior should also be tests here, I guess we don't want that to reset the credentials of an iothread!

Another idea would be to have the iothreads as a child process with it's threads, but again I'm only looking as an admin to what I'd except to see under /proc via ps and top.

metze

author maintainer
#28 lore raw
In reply to Eric W. Biederman
Quoted 41 lines · Eric W. Biederman
Quoted 20 lines · Linus Torvalds
Quoted 10 lines · Linus Torvalds
Quoted 3 lines · Linus Torvalds

I don't know what the gdb logic is, but maybe there's some other option that makes gdb not react to them?

.. maybe we could have a different name for them under the task/ subdirectory, for example (not just the pid)? Although that probably messes up 'ps' too..

Actually, maybe the right model is to simply make all the io threads take signals, and get rid of all the special cases.

Sure, the signals will never be delivered to user space, but if we

- just made the thread loop do "get_signal()" when there are pending signals

Quoted 8 lines

- allowed ptrace_attach on them

they'd look pretty much like regular threads that just never do the user-space part of signal handling.

The whole "signals are very special for IO threads" thing has caused so many problems, that maybe the solution is simply to _not_ make them special?

The special case in check_kill_permission is certainly unnecessary. Having the signal blocked is enough to prevent signal_pending() from being true.

The most straight forward thing I can see is to allow ptrace_attach and to modify ptrace_check_attach to always return -ESRCH for io workers unless ignore_state is set causing none of the other ptrace operations to work.

Quoted 6 lines

That is what a long running in-kernel thread would do today so user-space aka gdb may actually cope with it.

We might be able to support if io workers start supporting SIGSTOP but I am not at all certain.

See patch just send out as a POC, mostly, not fully sanitized yet. But I did try to return -ESRCH from ptrace_check_attach() if it's an IO thread and ignore_state isn't set:

if (!ignore_state && child->flags & PF_IO_WORKER)

	return -ESRCH;

and that causes gdb to abort at that thread. For the same test case as in the previous email, you get:

Attaching to process 358
[New LWP 359]
[New LWP 360]
[New LWP 361]
Couldn't get CS register: No such process.
(gdb) 0x00007ffa58537125 in ?? ()

(gdb) bt
#0 0x00007ffa58537125 in ?? ()
#1 0x0000000000000000 in ?? ()
(gdb) info threads

  Id   Target Id             Frame 
* 1    LWP 358 "io_uring"    0x00007ffa58537125 in ?? ()
  2    LWP 359 "iou-mgr-358" Couldn't get registers: No such process.

(gdb) q
A debugging session is active.

Inferior 1 [process 358] will be detached.

Quit anyway? (y or n) y
Couldn't write debug register: No such process.

where 360 here is a regular pthread created thread, and 361 is another iou-mgr-x task. While gdb behaves better in this case, it does still prevent you from inspecting thread 3 which would be totally valid.

Signature · 1 line
Jens Axboe
#17 lore raw
In reply to Jens Axboe

Am 25.03.21 um 22:44 schrieb Jens Axboe:

Quoted 82 lines · Jens Axboe
Quoted 30 lines · Linus Torvalds
Quoted 20 lines
Quoted 10 lines · Linus Torvalds
Quoted 3 lines · Linus Torvalds

I don't know what the gdb logic is, but maybe there's some other option that makes gdb not react to them?

.. maybe we could have a different name for them under the task/ subdirectory, for example (not just the pid)? Although that probably messes up 'ps' too..

Actually, maybe the right model is to simply make all the io threads take signals, and get rid of all the special cases.

Sure, the signals will never be delivered to user space, but if we

- just made the thread loop do "get_signal()" when there are pending signals

Quoted 8 lines

- allowed ptrace_attach on them

they'd look pretty much like regular threads that just never do the user-space part of signal handling.

The whole "signals are very special for IO threads" thing has caused so many problems, that maybe the solution is simply to _not_ make them special?

Quoted 8 lines

Just to wrap up the previous one, yes it broke all sorts of things to make the 'tid' directory different. They just end up being hidden anyway through that, for both ps and top.

Yes, I do think that maybe it's better to just embrace maybe just embrace the signals, and have everything just work by default. It's better than continually trying to make the threads special. I'll see if there are some demons lurking down that path.

In the spirit of "let's just try it", I ran with the below patch. With that, I can gdb attach just fine to a test case that creates an io_uring and a regular thread with pthread_create(). The regular thread uses the ring, so you end up with two iou-mgr threads. Attach:

[root@archlinux ~]# gdb -p 360
[snip gdb noise]
Attaching to process 360
[New LWP 361]
[New LWP 362]
[New LWP 363]

warning: Selected architecture i386:x86-64 is not compatible with reported target architecture i386

warning: Architecture rejected target-supplied description
Error while reading shared library symbols for /usr/lib/libpthread.so.0:
Cannot find user-level thread for LWP 363: generic error 0x00007f7aa526e125 in clock_nanosleep@GLIBC_2.2.5 () from /usr/lib/libc.so.6 (gdb) info threads

  Id   Target Id             Frame 
* 1    LWP 360 "io_uring"    0x00007f7aa526e125 in clock_nanosleep@GLIBC_2.2.5 ()
   from /usr/lib/libc.so.6
  2    LWP 361 "iou-mgr-360" 0x0000000000000000 in ?? ()
  3    LWP 362 "io_uring"    0x00007f7aa52a0a9d in syscall () from /usr/lib/libc.so.6
  4    LWP 363 "iou-mgr-362" 0x0000000000000000 in ?? ()

(gdb) thread 2
[Switching to thread 2 (LWP 361)]
#0 0x0000000000000000 in ?? ()
(gdb) bt
#0 0x0000000000000000 in ?? ()
Backtrace stopped: Cannot access memory at address 0x0
(gdb) cont
Continuing.
^C
Thread 1 "io_uring" received signal SIGINT, Interrupt. [Switching to LWP 360]
0x00007f7aa526e125 in clock_nanosleep@GLIBC_2.2.5 () from /usr/lib/libc.so.6 (gdb) q
A debugging session is active.

Inferior 1 [process 360] will be detached.

Quoted 7 lines

Quit anyway? (y or n) y
Detaching from program: /root/git/fio/t/io_uring, process 360 [Inferior 1 (process 360) detached]

The iou-mgr-x threads are stopped just fine, gdb obviously can't get any real info out of them. But it works... Regular test cases work fine too, just a sanity check. Didn't expect them not to.

I guess that's basically what I tried to describe when I said they should look like a userspace process that is blocked in a syscall forever.

Quoted 3 lines

Only thing that I dislike a bit, but I guess that's just a Linuxism, is that if can now kill an io_uring owning task by sending a signal to one of its IO thread workers.

Can't we just only allow SIGSTOP, which will be only delivered to the iothread itself? And also SIGKILL should not be allowed from userspace.

And /proc/$iothread/ should be read only and owned by root with "cmdline" and "exe" being empty.

Thanks!
metze

#21 lore raw
In reply to Jens Axboe
Quoted 6 lines · Jens Axboe

In the spirit of "let's just try it", I ran with the below patch. With that, I can gdb attach just fine to a test case that creates an io_uring and a regular thread with pthread_create(). The regular thread uses the ring, so you end up with two iou-mgr threads. Attach:

Quoted 6 lines

[root@archlinux ~]# gdb -p 360
[snip gdb noise]
Attaching to process 360
[New LWP 361]
[New LWP 362]
[New LWP 363]

[..]

Looks fairly sane to me.

I think this ends up being the right approach - just the final part (famous last words) of "io_uring threads act like normal threads".

Doing it for VM and FS got rid of all the special cases there, and now doing it for signal handling gets rid of all these ptrace etc issues.

And the fact that a noticeable part of the patch was removing the PF_IO_WORKER tests again looks like a very good sign to me.

In fact, I think you could now remove all the freezer hacks too - because get_signal() will now do the proper try_to_freeze(), so all those freezer things are stale as well.

Yeah, it's still going to be different in that there's no real user space return, and so it will never look _entirely_ like a normal thread, but on the whole I really like how this does seem to get rid of another batch of special cases.

Linus

author maintainer
#22 lore raw
Quoted 17 lines · Linus Torvalds
Quoted 6 lines · Jens Axboe

In the spirit of "let's just try it", I ran with the below patch. With that, I can gdb attach just fine to a test case that creates an io_uring and a regular thread with pthread_create(). The regular thread uses the ring, so you end up with two iou-mgr threads. Attach:

Quoted 6 lines

[root@archlinux ~]# gdb -p 360
[snip gdb noise]
Attaching to process 360
[New LWP 361]
[New LWP 362]
[New LWP 363]

[..]

Looks fairly sane to me.

Quoted 8 lines

I think this ends up being the right approach - just the final part (famous last words) of "io_uring threads act like normal threads".

Doing it for VM and FS got rid of all the special cases there, and now doing it for signal handling gets rid of all these ptrace etc issues.

And the fact that a noticeable part of the patch was removing the PF_IO_WORKER tests again looks like a very good sign to me.

I agree, and in fact there are more PF_IO_WORKER checks that can go too. The patch is just the bare minimum.

Quoted 3 lines

In fact, I think you could now remove all the freezer hacks too - because get_signal() will now do the proper try_to_freeze(), so all those freezer things are stale as well.

Yep

Quoted 4 lines

Yeah, it's still going to be different in that there's no real user space return, and so it will never look _entirely_ like a normal thread, but on the whole I really like how this does seem to get rid of another batch of special cases.

That's what makes me feel better too. I think was so hung up on the "never take signals" that it just didn't occur to me to go this route instead.

I'll send out a clean series.

Signature · 1 line
Jens Axboe
author maintainer
#18 lore raw
In reply to Stefan Metzmacher
Quoted 88 lines · Stefan Metzmacher

Am 25.03.21 um 22:44 schrieb Jens Axboe:

Quoted 86 lines · Jens Axboe
Quoted 30 lines · Linus Torvalds
Quoted 20 lines
Quoted 10 lines · Linus Torvalds
Quoted 3 lines · Linus Torvalds

I don't know what the gdb logic is, but maybe there's some other option that makes gdb not react to them?

.. maybe we could have a different name for them under the task/ subdirectory, for example (not just the pid)? Although that probably messes up 'ps' too..

Actually, maybe the right model is to simply make all the io threads take signals, and get rid of all the special cases.

Sure, the signals will never be delivered to user space, but if we

- just made the thread loop do "get_signal()" when there are pending signals

Quoted 8 lines

- allowed ptrace_attach on them

they'd look pretty much like regular threads that just never do the user-space part of signal handling.

The whole "signals are very special for IO threads" thing has caused so many problems, that maybe the solution is simply to _not_ make them special?

Quoted 8 lines

Just to wrap up the previous one, yes it broke all sorts of things to make the 'tid' directory different. They just end up being hidden anyway through that, for both ps and top.

Yes, I do think that maybe it's better to just embrace maybe just embrace the signals, and have everything just work by default. It's better than continually trying to make the threads special. I'll see if there are some demons lurking down that path.

In the spirit of "let's just try it", I ran with the below patch. With that, I can gdb attach just fine to a test case that creates an io_uring and a regular thread with pthread_create(). The regular thread uses the ring, so you end up with two iou-mgr threads. Attach:

[root@archlinux ~]# gdb -p 360
[snip gdb noise]
Attaching to process 360
[New LWP 361]
[New LWP 362]
[New LWP 363]

warning: Selected architecture i386:x86-64 is not compatible with reported target architecture i386

warning: Architecture rejected target-supplied description
Error while reading shared library symbols for /usr/lib/libpthread.so.0:
Cannot find user-level thread for LWP 363: generic error 0x00007f7aa526e125 in clock_nanosleep@GLIBC_2.2.5 () from /usr/lib/libc.so.6 (gdb) info threads

  Id   Target Id             Frame 
* 1    LWP 360 "io_uring"    0x00007f7aa526e125 in clock_nanosleep@GLIBC_2.2.5 ()
   from /usr/lib/libc.so.6
  2    LWP 361 "iou-mgr-360" 0x0000000000000000 in ?? ()
  3    LWP 362 "io_uring"    0x00007f7aa52a0a9d in syscall () from /usr/lib/libc.so.6
  4    LWP 363 "iou-mgr-362" 0x0000000000000000 in ?? ()

(gdb) thread 2
[Switching to thread 2 (LWP 361)]
#0 0x0000000000000000 in ?? ()
(gdb) bt
#0 0x0000000000000000 in ?? ()
Backtrace stopped: Cannot access memory at address 0x0
(gdb) cont
Continuing.
^C
Thread 1 "io_uring" received signal SIGINT, Interrupt. [Switching to LWP 360]
0x00007f7aa526e125 in clock_nanosleep@GLIBC_2.2.5 () from /usr/lib/libc.so.6 (gdb) q
A debugging session is active.

Inferior 1 [process 360] will be detached.

Quit anyway? (y or n) y
Detaching from program: /root/git/fio/t/io_uring, process 360 [Inferior 1 (process 360) detached]

Quoted 7 lines
Quoted 3 lines

The iou-mgr-x threads are stopped just fine, gdb obviously can't get any real info out of them. But it works... Regular test cases work fine too, just a sanity check. Didn't expect them not to.

I guess that's basically what I tried to describe when I said they should look like a userspace process that is blocked in a syscall forever.

Right, that's almost what they look like, in practice that is what they look like.

Quoted 6 lines
Quoted 3 lines

Only thing that I dislike a bit, but I guess that's just a Linuxism, is that if can now kill an io_uring owning task by sending a signal to one of its IO thread workers.

Can't we just only allow SIGSTOP, which will be only delivered to the iothread itself? And also SIGKILL should not be allowed from userspace.

I don't think we can sanely block them, and we to cleanup and teardown normally regardless of who gets the signal (owner or one of the threads). So I'm not _too_ hung up on the "io thread gets signal goes to owner" as that is what happens with normal threads too, though I would prefer if that wasn't the case. But overall I feel better just embracing the thread model, rather than having something that kinda sorta looks like a thread, but differs in odd ways.

Quoted 2 lines

And /proc/$iothread/ should be read only and owned by root with "cmdline" and "exe" being empty.

I know you brought this one up as part of your series, not sure I get why you want it owned by root and read-only? cmdline and exe, yeah those could be hidden, but is there really any point?

Maybe I'm missing something here, if so, do clue me in!

Signature · 1 line
Jens Axboe
#19 lore raw

Hi Jens,

Quoted 8 lines
Quoted 2 lines

And /proc/$iothread/ should be read only and owned by root with "cmdline" and "exe" being empty.

I know you brought this one up as part of your series, not sure I get why you want it owned by root and read-only? cmdline and exe, yeah those could be hidden, but is there really any point?

Maybe I'm missing something here, if so, do clue me in!

I looked through /proc and I think it's mostly similar to the unshare() case, if userspace wants to do stupid things like changing "comm" of iothreads, it gets what was asked for.

But the "cmdline" hiding would be very useful.

While most tools use "comm", by default.

ps -eLf or 'iotop' use "cmdline".

Some processes use setproctitle to change "cmdline" in order to identify the process better, without the 15 chars comm restriction, that's why I very often press 'c' in 'top' to see the cmdline, in that case it would be very helpful to see '[iou-wrk-1234]' instead of the seeing the cmdline.

So I'd very much prefer if this could be applied:
https://lore.kernel.org/io-uring/d4487f959c778d0b1d4c5738b75bcff17d21df5b.1616197787.git.metze@samba.org/T/#u

If you want I can add a comment and a more verbose commit message...

metze

#3 lore raw

[proc] 43b2a76b1a: will-it-scale.per_process_ops -11.3% regression

In reply to Jens Axboe
FYI, we noticed a -11.3% regression of will-it-scale.per_process_ops due to commit:

Greeting,

FYI, we noticed a -11.3% regression of will-it-scale.per_process_ops due to commit:

commit: 43b2a76b1a5abcc9833b463bef137d35cbb85cdd ("[PATCH 2/2] proc: don't show PF_IO_WORKER threads as threads in /proc/<pid>/task/")
url: https://github.com/0day-ci/linux/commits/Jens-Axboe/Don-t-show-PF_IO_WORKER-in-proc-pid-task/20210326-004554
base: https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git a74e6a014c9d4d4161061f770c9b4f98372ac778

in testcase: will-it-scale
on test machine: 192 threads Intel(R) Xeon(R) Platinum 9242 CPU @ 2.30GHz with 192G memory with following parameters:

	nr_task: 16
	mode: process
	test: eventfd1
	cpufreq_governor: performance
	ucode: 0x5003006

test-description: Will It Scale takes a testcase and runs it from 1 through to n parallel copies to see if the testcase will scale. It builds both a process and threads based test in order to see any differences between the two.
test-url: https://github.com/antonblanchard/will-it-scale

In addition to that, the commit also has significant impact on the following tests:

+------------------+---------------------------------------------------------------------------+
| testcase: change | will-it-scale: will-it-scale.per_process_ops -11.1% regression            |
| test machine     | 104 threads Skylake with 192G memory                                      |
| test parameters  | cpufreq_governor=performance                                              |
|                  | mode=process                                                              |
|                  | nr_task=16                                                                |
|                  | test=malloc1                                                              |
|                  | ucode=0x2006a0a                                                           |
+------------------+---------------------------------------------------------------------------+
| testcase: change | will-it-scale: will-it-scale.per_process_ops -6.3% regression             |
| test machine     | 104 threads Skylake with 192G memory                                      |
| test parameters  | cpufreq_governor=performance                                              |
|                  | mode=process                                                              |
|                  | nr_task=100%                                                              |
|                  | test=eventfd1                                                             |
|                  | ucode=0x2006a0a                                                           |
+------------------+---------------------------------------------------------------------------+
| testcase: change | will-it-scale: will-it-scale.per_thread_ops -2.2% regression              |
| test machine     | 48 threads Intel(R) Xeon(R) CPU E5-2697 v2 @ 2.70GHz with 112G memory     |
| test parameters  | cpufreq_governor=performance                                              |
|                  | mode=thread                                                               |
|                  | nr_task=100%                                                              |
|                  | test=poll1                                                                |
|                  | ucode=0x42e                                                               |
+------------------+---------------------------------------------------------------------------+
| testcase: change | will-it-scale: will-it-scale.per_process_ops -2.6% regression             |
| test machine     | 48 threads Intel(R) Xeon(R) CPU E5-2697 v2 @ 2.70GHz with 112G memory     |
| test parameters  | cpufreq_governor=performance                                              |
|                  | mode=process                                                              |
|                  | nr_task=50%                                                               |
|                  | test=pread1                                                               |
|                  | ucode=0x42e                                                               |
+------------------+---------------------------------------------------------------------------+
| testcase: change | will-it-scale: will-it-scale.per_process_ops -15.3% regression            |
| test machine     | 104 threads Skylake with 192G memory                                      |
| test parameters  | cpufreq_governor=performance                                              |
|                  | mode=process                                                              |
|                  | nr_task=50%                                                               |
|                  | test=brk1                                                                 |
|                  | ucode=0x2006a0a                                                           |
+------------------+---------------------------------------------------------------------------+
| testcase: change | will-it-scale: will-it-scale.per_thread_ops -7.1% regression              |
| test machine     | 144 threads Intel(R) Xeon(R) CPU E7-8890 v3 @ 2.50GHz with 512G memory    |
| test parameters  | cpufreq_governor=performance                                              |
|                  | mode=thread                                                               |
|                  | nr_task=16                                                                |
|                  | test=pipe1                                                                |
|                  | ucode=0x16                                                                |
+------------------+---------------------------------------------------------------------------+
| testcase: change | will-it-scale: will-it-scale.per_process_ops -21.4% regression            |
| test machine     | 192 threads Intel(R) Xeon(R) Platinum 9242 CPU @ 2.30GHz with 192G memory |
| test parameters  | cpufreq_governor=performance                                              |
|                  | mode=process                                                              |
|                  | nr_task=100%                                                              |
|                  | test=eventfd1                                                             |
|                  | ucode=0x5003006                                                           |
+------------------+---------------------------------------------------------------------------+
| testcase: change | unixbench: unixbench.score -4.4% regression                               |
| test machine     | 96 threads Intel(R) Xeon(R) CPU @ 2.30GHz with 128G memory                |
| test parameters  | cpufreq_governor=performance                                              |
|                  | nr_task=1                                                                 |
|                  | runtime=300s                                                              |
|                  | test=pipe                                                                 |
|                  | ucode=0x4003006                                                           |
+------------------+---------------------------------------------------------------------------+
| testcase: change | will-it-scale: will-it-scale.per_process_ops -20.3% regression            |
| test machine     | 104 threads Skylake with 192G memory                                      |
| test parameters  | cpufreq_governor=performance                                              |
|                  | mode=process                                                              |
|                  | nr_task=16                                                                |
|                  | test=open1                                                                |
|                  | ucode=0x2006a0a                                                           |
+------------------+---------------------------------------------------------------------------+
| testcase: change | will-it-scale: will-it-scale.per_thread_ops -17.7% regression             |
| test machine     | 144 threads Intel(R) Xeon(R) Gold 5318H CPU @ 2.50GHz with 128G memory    |
| test parameters  | cpufreq_governor=performance                                              |
|                  | mode=thread                                                               |
|                  | nr_task=50%                                                               |
|                  | test=readseek1                                                            |
|                  | ucode=0x700001e                                                           |
+------------------+---------------------------------------------------------------------------+
| testcase: change | will-it-scale: will-it-scale.per_thread_ops -11.2% regression             |
| test machine     | 192 threads Intel(R) Xeon(R) Platinum 9242 CPU @ 2.30GHz with 192G memory |
| test parameters  | cpufreq_governor=performance                                              |
|                  | mode=thread                                                               |
|                  | nr_task=16                                                                |
|                  | test=eventfd1                                                             |
|                  | ucode=0x5003006                                                           |
+------------------+---------------------------------------------------------------------------+
| testcase: change | will-it-scale: will-it-scale.per_thread_ops -1.1% regression              |
| test machine     | 48 threads Intel(R) Xeon(R) CPU E5-2697 v2 @ 2.70GHz with 112G memory     |
| test parameters  | cpufreq_governor=performance                                              |
|                  | mode=thread                                                               |
|                  | nr_task=50%                                                               |
|                  | test=pthread_mutex2                                                       |
|                  | ucode=0x42e                                                               |
+------------------+---------------------------------------------------------------------------+
| testcase: change | will-it-scale: will-it-scale.per_thread_ops -6.7% regression              |
| test machine     | 104 threads Skylake with 192G memory                                      |
| test parameters  | cpufreq_governor=performance                                              |
|                  | mode=thread                                                               |
|                  | nr_task=100%                                                              |
|                  | test=eventfd1                                                             |
|                  | ucode=0x2006a0a                                                           |
+------------------+---------------------------------------------------------------------------+
| testcase: change | will-it-scale: will-it-scale.per_process_ops -11.2% regression            |
| test machine     | 104 threads Skylake with 192G memory                                      |
| test parameters  | cpufreq_governor=performance                                              |
|                  | mode=process                                                              |
|                  | nr_task=50%                                                               |
|                  | test=lock1                                                                |
|                  | ucode=0x2006a0a                                                           |
+------------------+---------------------------------------------------------------------------+
| testcase: change | will-it-scale: will-it-scale.per_thread_ops 3.3% improvement              |
| test machine     | 144 threads Intel(R) Xeon(R) CPU E7-8890 v3 @ 2.50GHz with 512G memory    |
| test parameters  | cpufreq_governor=performance                                              |
|                  | mode=thread                                                               |
|                  | nr_task=50%                                                               |
|                  | test=sched_yield                                                          |
|                  | ucode=0x16                                                                |
+------------------+---------------------------------------------------------------------------+
| testcase: change | fxmark: fxmark.hdd_ext4_MWRL_9_bufferedio.works/sec -11.4% regression     |
| test machine     | 288 threads Intel(R) Xeon Phi(TM) CPU 7295 @ 1.50GHz with 80G memory      |
| test parameters  | cpufreq_governor=performance                                              |
|                  | directio=bufferedio                                                       |
|                  | disk=1HDD                                                                 |
|                  | fstype=ext4                                                               |
|                  | media=hdd                                                                 |
|                  | test=MWRL                                                                 |
|                  | ucode=0x11                                                                |
+------------------+---------------------------------------------------------------------------+
| testcase: change | fxmark: fxmark.hdd_btrfs_MWRL_9_bufferedio.works/sec -31.7% regression    |
| test machine     | 288 threads Intel(R) Xeon Phi(TM) CPU 7295 @ 1.50GHz with 80G memory      |
| test parameters  | cpufreq_governor=performance                                              |
|                  | directio=bufferedio                                                       |
|                  | disk=1HDD                                                                 |
|                  | fstype=btrfs                                                              |
|                  | media=hdd                                                                 |
|                  | test=MWRL                                                                 |
|                  | ucode=0x11                                                                |
+------------------+---------------------------------------------------------------------------+
| testcase: change | stress-ng: stress-ng.netdev.ops_per_sec -16.5% regression                 |
| test machine     | 96 threads Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz with 192G memory      |
| test parameters  | class=network                                                             |
|                  | cpufreq_governor=performance                                              |
|                  | disk=1HDD                                                                 |
|                  | nr_threads=100%                                                           |
|                  | test=netdev                                                               |
|                  | testtime=60s                                                              |
|                  | ucode=0x5003006                                                           |
+------------------+---------------------------------------------------------------------------+
| testcase: change | fxmark: fxmark.hdd_xfs_DRBH_2_directio.works/sec -14.3% regression        |
| test machine     | 288 threads Intel(R) Xeon Phi(TM) CPU 7295 @ 1.50GHz with 80G memory      |
| test parameters  | cpufreq_governor=performance                                              |
|                  | directio=directio                                                         |
|                  | disk=1HDD                                                                 |
|                  | fstype=xfs                                                                |
|                  | media=hdd                                                                 |
|                  | test=DRBH                                                                 |
|                  | ucode=0x11                                                                |
+------------------+---------------------------------------------------------------------------+
| testcase: change | stress-ng: stress-ng.fanotify.ops_per_sec 192.8% improvement              |
| test machine     | 48 threads Intel(R) Xeon(R) CPU E5-2697 v2 @ 2.70GHz with 112G memory     |
| test parameters  | class=filesystem                                                          |
|                  | cpufreq_governor=performance                                              |
|                  | disk=1HDD                                                                 |
|                  | fs=btrfs                                                                  |
|                  | nr_threads=10%                                                            |
|                  | test=fanotify                                                             |
|                  | testtime=60s                                                              |
|                  | ucode=0x42e                                                               |
+------------------+---------------------------------------------------------------------------+
| testcase: change | fxmark: fxmark.hdd_xfs_MWCM_9_bufferedio.works/sec -9.0% regression       |
| test machine     | 288 threads Intel(R) Xeon Phi(TM) CPU 7295 @ 1.50GHz with 80G memory      |
| test parameters  | cpufreq_governor=performance                                              |
|                  | directio=bufferedio                                                       |
|                  | disk=1HDD                                                                 |
|                  | fstype=xfs                                                                |
|                  | media=hdd                                                                 |
|                  | test=MWCM                                                                 |
|                  | ucode=0x11                                                                |
+------------------+---------------------------------------------------------------------------+

If you fix the issue, kindly add following tag

Details are as below:
-------------------------------------------------------------------------------------------------->

To reproduce:

git clone https://github.com/intel/lkp-tests.git
cd lkp-tests
bin/lkp install job.yaml # job file is attached in this email
bin/lkp split-job --compatible job.yaml
bin/lkp run compatible-job.yaml

=========================================================================================
compiler/cpufreq_governor/kconfig/mode/nr_task/rootfs/tbox_group/test/testcase/ucode:
gcc-9/performance/x86_64-rhel-8.3/process/16/debian-10.4-x86_64-20200603.cgz/lkp-csl-2ap2/eventfd1/will-it-scale/0x5003006

commit:
dde38d5b80 ("kernel: don't include PF_IO_WORKERs as part of same_thread_group()") 43b2a76b1a ("proc: don't show PF_IO_WORKER threads as threads in /proc/<pid>/task/")

dde38d5b805358c7 43b2a76b1a5abcc9833b463bef1
---------------- ---------------------------
%stddev %change %stddev
\ | \

  47173522           -11.3%   41822683        will-it-scale.16.processes
   2948344           -11.3%    2613917        will-it-scale.per_process_ops
  47173522           -11.3%   41822683        will-it-scale.workload
    334392 ± 62%    +222.6%    1078700 ± 43%  numa-numastat.node3.local_node
    359226 ± 51%    +217.8%    1141654 ± 38%  numa-numastat.node3.numa_hit
      1.63 ± 20%      -0.6        1.01 ±  8%  mpstat.cpu.all.irq%
      6.70            +0.8        7.53        mpstat.cpu.all.sys%
      1.41            -0.1        1.27        mpstat.cpu.all.usr%
   1387381          +194.5%    4086263 ±  5%  vmstat.memory.cache
     15.17 ±  2%     +14.3%      17.33 ±  2%  vmstat.procs.r
      2193 ±  3%      +9.6%       2402        vmstat.system.cs
   1429848 ± 16%     +26.2%    1804776 ± 12%  sched_debug.cfs_rq:/.spread0.max
    566.65           +74.3%     987.56        sched_debug.cfs_rq:/.util_est_enqueued.max
    331.80 ±  3%     +11.2%     368.82        sched_debug.cpu.curr->pid.avg
      0.09 ±  4%     +11.1%       0.10        sched_debug.cpu.nr_running.avg
     54610 ±  3%   +4930.8%    2747384 ±  7%  meminfo.Active
     54610 ±  3%   +4930.8%    2747384 ±  7%  meminfo.Active(anon)
    173070           +13.0%     195601        meminfo.AnonHugePages
    265559           +15.2%     305891        meminfo.AnonPages
   1256498          +214.2%    3948319 ±  5%  meminfo.Cached
    509316          +545.4%    3287375 ±  6%  meminfo.Committed_AS
    285355           +13.8%     324656        meminfo.Inactive
    285355           +13.8%     324656        meminfo.Inactive(anon)
   3296220          +104.7%    6746177 ±  3%  meminfo.Memused
      4791 ±  3%     +16.9%       5602 ±  2%  meminfo.PageTables
     75032 ±  2%   +3587.6%    2766852 ±  7%  meminfo.Shmem
   4299183 ±  3%     +57.3%    6764473 ±  3%  meminfo.max_used_kB
    179.17 ±158%   +4225.3%       7749 ±160%  numa-vmstat.node0.nr_active_anon
    179.17 ±158%   +4225.3%       7749 ±160%  numa-vmstat.node0.nr_zone_active_anon
    196.17 ±129%  +95954.0%     188425 ± 98%  numa-vmstat.node1.nr_active_anon
     73277 ±  2%    +258.3%     262578 ± 70%  numa-vmstat.node1.nr_file_pages
      2075 ± 82%   +9064.7%     190228 ± 97%  numa-vmstat.node1.nr_shmem
    196.17 ±129%  +95954.0%     188425 ± 98%  numa-vmstat.node1.nr_zone_active_anon
    254.67 ±134%  +67712.8%     172696 ± 70%  numa-vmstat.node2.nr_active_anon
     76891 ±  4%    +221.5%     247214 ± 49%  numa-vmstat.node2.nr_file_pages
    455.33 ±108%  +38042.8%     173677 ± 70%  numa-vmstat.node2.nr_shmem
    254.67 ±134%  +67712.8%     172696 ± 70%  numa-vmstat.node2.nr_zone_active_anon
     12988 ±  5%   +2347.1%     317842 ± 58%  numa-vmstat.node3.nr_active_anon
     88291 ±  3%    +344.8%     392756 ± 47%  numa-vmstat.node3.nr_file_pages
     14154 ±  8%   +2154.8%     319144 ± 58%  numa-vmstat.node3.nr_shmem
     12988 ±  5%   +2347.1%     317842 ± 58%  numa-vmstat.node3.nr_zone_active_anon
    707483 ± 24%     +53.0%    1082663 ± 30%  numa-vmstat.node3.numa_hit
    717.67 ±158%   +4214.9%      30966 ±160%  numa-meminfo.node0.Active
    717.67 ±158%   +4214.9%      30966 ±160%  numa-meminfo.node0.Active(anon)
    898016 ±  7%     +18.0%    1059629 ±  8%  numa-meminfo.node0.MemUsed
    785.33 ±129%  +95815.7%     753258 ± 98%  numa-meminfo.node1.Active
    785.33 ±129%  +95815.7%     753258 ± 98%  numa-meminfo.node1.Active(anon)
    293110 ±  2%    +258.2%    1049866 ± 70%  numa-meminfo.node1.FilePages
    737028 ±  9%    +128.1%    1681518 ± 47%  numa-meminfo.node1.MemUsed
      8303 ± 82%   +9058.8%     760466 ± 97%  numa-meminfo.node1.Shmem
      1019 ±134%  +67579.3%     690216 ± 70%  numa-meminfo.node2.Active
      1019 ±134%  +67579.3%     690216 ± 70%  numa-meminfo.node2.Active(anon)
     94946 ± 62%    +232.2%     315437 ± 45%  numa-meminfo.node2.AnonPages.max
    307567 ±  4%    +221.3%     988285 ± 49%  numa-meminfo.node2.FilePages
    808971 ±  8%    +111.6%    1712135 ± 29%  numa-meminfo.node2.MemUsed
      1822 ±108%  +37980.2%     694137 ± 70%  numa-meminfo.node2.Shmem
     51989 ±  5%   +2342.9%    1270067 ± 58%  numa-meminfo.node3.Active
     51989 ±  5%   +2342.9%    1270067 ± 58%  numa-meminfo.node3.Active(anon)
    353216 ±  3%    +344.4%    1569725 ± 47%  numa-meminfo.node3.FilePages
    853866 ± 11%    +168.2%    2290171 ± 35%  numa-meminfo.node3.MemUsed
     56665 ±  8%   +2150.6%    1275276 ± 58%  numa-meminfo.node3.Shmem
     13627 ±  3%   +4930.0%     685442 ±  7%  proc-vmstat.nr_active_anon
     66425           +15.2%      76490        proc-vmstat.nr_anon_pages
   4835055            -1.8%    4749110        proc-vmstat.nr_dirty_background_threshold
   9681934            -1.8%    9509833        proc-vmstat.nr_dirty_threshold
    314100          +213.8%     985675 ±  5%  proc-vmstat.nr_file_pages
  48604271            -1.8%   47743555        proc-vmstat.nr_free_pages
     71374           +13.7%      81180        proc-vmstat.nr_inactive_anon
      9741            +1.6%       9892        proc-vmstat.nr_mapped
      1203 ±  2%     +16.7%       1403 ±  2%  proc-vmstat.nr_page_table_pages
     18733 ±  2%   +3584.9%     690308 ±  7%  proc-vmstat.nr_shmem
     72469            +7.5%      77891 ±  2%  proc-vmstat.nr_slab_unreclaimable
     13627 ±  3%   +4930.0%     685442 ±  7%  proc-vmstat.nr_zone_active_anon
     71374           +13.7%      81180        proc-vmstat.nr_zone_inactive_anon
   1243541          +141.3%    3000844 ±  3%  proc-vmstat.numa_hit
    983775          +178.6%    2741036 ±  4%  proc-vmstat.numa_local
     43874 ± 21%     +65.0%      72400 ± 30%  proc-vmstat.numa_pte_updates
     18545 ±  3%    +398.9%      92528 ±  6%  proc-vmstat.pgactivate
   1320163          +224.0%    4277306 ±  5%  proc-vmstat.pgalloc_normal
   1176045            -2.5%    1146897        proc-vmstat.pgfault
   1325260 ±  5%    +101.6%    2671978 ±  6%  proc-vmstat.pgfree
      2999 ±  5%     -10.0%       2698 ±  7%  slabinfo.PING.active_objs
      2999 ±  5%     -10.0%       2698 ±  7%  slabinfo.PING.num_objs
     73711 ±  6%     +27.9%      94307 ±  4%  slabinfo.filp.active_objs
      1160 ±  6%     +28.1%       1486 ±  4%  slabinfo.filp.active_slabs
     74267 ±  6%     +28.1%      95154 ±  4%  slabinfo.filp.num_objs
      1160 ±  6%     +28.1%       1486 ±  4%  slabinfo.filp.num_slabs
     28063 ±  2%     +17.3%      32913 ±  5%  slabinfo.kmalloc-512.active_objs
     28448 ±  2%     +16.8%      33214 ±  4%  slabinfo.kmalloc-512.num_objs
     21843 ±  2%     -27.7%      15791 ±  3%  slabinfo.proc_inode_cache.active_objs
    458.33 ±  3%     -23.3%     351.67 ±  4%  slabinfo.proc_inode_cache.active_slabs
     22024 ±  3%     -23.2%      16912 ±  4%  slabinfo.proc_inode_cache.num_objs
    458.33 ±  3%     -23.3%     351.67 ±  4%  slabinfo.proc_inode_cache.num_slabs
     27129           +38.3%      37516 ±  2%  slabinfo.radix_tree_node.active_objs
    484.00           +38.3%     669.50 ±  2%  slabinfo.radix_tree_node.active_slabs
     27129           +38.3%      37516 ±  2%  slabinfo.radix_tree_node.num_objs
    484.00           +38.3%     669.50 ±  2%  slabinfo.radix_tree_node.num_slabs
      5470 ±  5%      +9.7%       6000 ±  5%  slabinfo.signal_cache.active_objs
      5506 ±  6%      +9.0%       6000 ±  5%  slabinfo.signal_cache.num_objs
      2158           +20.5%       2601 ±  7%  slabinfo.task_struct.active_objs
      2172           +20.4%       2616 ±  7%  slabinfo.task_struct.active_slabs
      2172           +20.4%       2616 ±  7%  slabinfo.task_struct.num_objs
      2172           +20.4%       2616 ±  7%  slabinfo.task_struct.num_slabs
     16909 ± 10%     -19.8%      13563 ±  6%  softirqs.CPU10.RCU
     14967 ±  7%     -15.2%      12694 ±  2%  softirqs.CPU104.RCU
     17878 ±  6%     -17.9%      14671 ±  8%  softirqs.CPU12.RCU
     16947 ± 14%     -18.3%      13844 ±  5%  softirqs.CPU13.RCU
     17862 ± 14%     -18.7%      14521 ±  3%  softirqs.CPU14.RCU
     13535 ± 18%    +114.7%      29055 ± 34%  softirqs.CPU144.RCU
     12200 ± 17%     -16.2%      10227 ±  8%  softirqs.CPU163.RCU
     12305 ± 13%    +140.2%      29562 ± 18%  softirqs.CPU168.RCU
     12133 ± 12%     -18.8%       9856 ±  8%  softirqs.CPU171.RCU
     12087 ± 12%     -19.3%       9753 ±  8%  softirqs.CPU174.RCU
     14277 ± 15%     -24.5%      10781 ± 17%  softirqs.CPU185.RCU
     13292 ± 16%     -19.2%      10746 ± 16%  softirqs.CPU186.RCU
     13156 ± 16%     -20.7%      10435 ± 16%  softirqs.CPU187.RCU
     17562 ± 11%     -22.5%      13605 ±  8%  softirqs.CPU2.RCU
     16919 ± 11%     -19.1%      13690 ±  8%  softirqs.CPU3.RCU
     17426 ± 10%     -19.4%      14052 ±  8%  softirqs.CPU4.RCU
     13629 ± 19%    +171.3%      36969 ± 32%  softirqs.CPU48.RCU
     40218 ±  4%     -12.1%      35333 ±  8%  softirqs.CPU48.SCHED
     17180 ± 10%     -19.8%      13783 ±  6%  softirqs.CPU5.RCU
     17300 ± 12%     -18.4%      14110 ±  8%  softirqs.CPU6.RCU
     12435 ± 16%     -17.2%      10295 ±  9%  softirqs.CPU67.RCU
     16439 ± 13%     -20.2%      13110 ±  8%  softirqs.CPU7.RCU
     12481 ± 14%    +229.3%      41098 ± 16%  softirqs.CPU72.RCU
     39776 ±  3%     -19.1%      32174 ±  8%  softirqs.CPU72.SCHED
     16186 ± 10%     -20.8%      12820 ±  7%  softirqs.CPU9.RCU
     18637 ±  4%     +96.3%      36588 ± 17%  softirqs.CPU95.SCHED
      1.17 ±  5%      -0.1        1.03 ±  2%  perf-stat.i.branch-miss-rate%
  1.92e+08 ±  5%     -12.3%  1.685e+08 ±  2%  perf-stat.i.branch-misses
     13.96 ± 72%     +16.8       30.78 ± 22%  perf-stat.i.cache-miss-rate%
   4525225 ± 10%    +331.5%   19527153 ± 19%  perf-stat.i.cache-misses
      2145 ±  3%      +9.8%       2356        perf-stat.i.context-switches
      0.69           +15.5%       0.80        perf-stat.i.cpi
  5.64e+10           +14.7%  6.472e+10        perf-stat.i.cpu-cycles
     12806 ± 12%     -57.3%       5465 ± 15%  perf-stat.i.cycles-between-cache-misses
 2.373e+10            -3.9%   2.28e+10        perf-stat.i.dTLB-loads
 1.588e+10            -4.5%  1.516e+10        perf-stat.i.dTLB-stores
     98.13            -0.8       97.32        perf-stat.i.iTLB-load-miss-rate%
 1.763e+08            -9.3%    1.6e+08        perf-stat.i.iTLB-load-misses
    463.51            +9.5%     507.53        perf-stat.i.instructions-per-iTLB-miss
      1.44           -13.4%       1.25        perf-stat.i.ipc
      1.39 ±  5%     -84.1%       0.22 ± 84%  perf-stat.i.major-faults
      0.29           +14.7%       0.34        perf-stat.i.metric.GHz
    292.33            -3.1%     283.30        perf-stat.i.metric.M/sec
      3778            -2.6%       3681        perf-stat.i.minor-faults
     91.63            +4.1       95.73        perf-stat.i.node-load-miss-rate%
    437707 ± 55%   +1002.0%    4823604 ± 27%  perf-stat.i.node-load-misses
     39983 ± 39%    +313.0%     165142 ± 20%  perf-stat.i.node-loads
     94366 ± 21%   +1359.2%    1376997 ± 17%  perf-stat.i.node-store-misses
      3780            -2.6%       3681        perf-stat.i.page-faults
      1.17 ±  5%      -0.1        1.03 ±  2%  perf-stat.overall.branch-miss-rate%
      0.69           +15.6%       0.80        perf-stat.overall.cpi
     12633 ± 12%     -72.6%       3457 ± 21%  perf-stat.overall.cycles-between-cache-misses
     98.18            -0.8       97.37        perf-stat.overall.iTLB-load-miss-rate%
    462.23            +9.4%     505.87        perf-stat.overall.instructions-per-iTLB-miss
      1.45           -13.5%       1.25        perf-stat.overall.ipc
     90.92            +5.4       96.31        perf-stat.overall.node-load-miss-rate%
     83.08 ± 13%     +13.9       96.99        perf-stat.overall.node-store-miss-rate%
    520246           +12.0%     582888        perf-stat.overall.path-length
 1.914e+08 ±  5%     -12.3%  1.679e+08 ±  2%  perf-stat.ps.branch-misses
   4510225 ± 10%    +331.6%   19466927 ± 19%  perf-stat.ps.cache-misses
      2138 ±  3%      +9.8%       2348        perf-stat.ps.context-switches
 5.621e+10           +14.7%   6.45e+10        perf-stat.ps.cpu-cycles
    197.85            -1.4%     195.09        perf-stat.ps.cpu-migrations
 2.365e+10            -3.9%  2.272e+10        perf-stat.ps.dTLB-loads
 1.583e+10            -4.5%  1.511e+10        perf-stat.ps.dTLB-stores
 1.757e+08            -9.3%  1.594e+08        perf-stat.ps.iTLB-load-misses
      1.38 ±  5%     -84.1%       0.22 ± 84%  perf-stat.ps.major-faults
      3765            -2.6%       3668        perf-stat.ps.minor-faults
    436378 ± 55%   +1002.0%    4808882 ± 27%  perf-stat.ps.node-load-misses
     39914 ± 39%    +312.5%     164641 ± 20%  perf-stat.ps.node-loads
     94069 ± 21%   +1359.2%    1372706 ± 17%  perf-stat.ps.node-store-misses
      3766            -2.6%       3668        perf-stat.ps.page-faults
     37.68 ± 15%     -37.7        0.00        perf-profile.calltrace.cycles-pp.secondary_startup_64_no_verify
     36.34 ± 14%     -36.3        0.00        perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64_no_verify
     36.34 ± 14%     -36.3        0.00        perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
     36.34 ± 14%     -36.3        0.00        perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
     35.40 ±  9%     -35.4        0.00        perf-profile.calltrace.cycles-pp.read
     35.16 ± 15%     -35.2        0.00        perf-profile.calltrace.cycles-pp.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
     34.65 ± 14%     -34.7        0.00        perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary
     33.00 ± 17%     -33.0        0.00        perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry
     30.91 ±  9%     -30.9        0.00        perf-profile.calltrace.cycles-pp.write
     23.94 ±  9%     -23.9        0.00        perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.read
     21.78 ±  9%     -21.8        0.00        perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.read
     20.85 ±  9%     -20.9        0.00        perf-profile.calltrace.cycles-pp.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe.read
     19.55 ±  9%     -19.6        0.00        perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.write
     18.44 ±  9%     -18.4        0.00        perf-profile.calltrace.cycles-pp.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe.read
     17.39 ±  9%     -17.4        0.00        perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.write
     16.48 ±  9%     -16.5        0.00        perf-profile.calltrace.cycles-pp.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.write
     14.08 ±  9%     -14.1        0.00        perf-profile.calltrace.cycles-pp.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.write
     10.62 ±  9%     -10.6        0.00        perf-profile.calltrace.cycles-pp.new_sync_read.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe
      8.92 ±  9%      -8.9        0.00        perf-profile.calltrace.cycles-pp.eventfd_read.new_sync_read.vfs_read.ksys_read.do_syscall_64
      7.90 ±  9%      -7.9        0.00        perf-profile.calltrace.cycles-pp.eventfd_write.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
      6.58 ±  9%      -6.6        0.00        perf-profile.calltrace.cycles-pp.__entry_text_start.write
      6.55 ±  8%      -6.6        0.00        perf-profile.calltrace.cycles-pp.__entry_text_start.read
      5.20 ±  9%      -5.2        0.00        perf-profile.calltrace.cycles-pp._copy_to_iter.eventfd_read.new_sync_read.vfs_read.ksys_read
     43.75 ±  9%     -43.8        0.00        perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
     39.43 ±  9%     -39.4        0.00        perf-profile.children.cycles-pp.do_syscall_64
     37.68 ± 15%     -37.7        0.00        perf-profile.children.cycles-pp.secondary_startup_64_no_verify
     37.68 ± 15%     -37.7        0.00        perf-profile.children.cycles-pp.cpu_startup_entry
     37.68 ± 15%     -37.7        0.00        perf-profile.children.cycles-pp.do_idle
     36.51 ± 15%     -36.5        0.00        perf-profile.children.cycles-pp.cpuidle_enter
     36.51 ± 15%     -36.5        0.00        perf-profile.children.cycles-pp.cpuidle_enter_state
     36.34 ± 14%     -36.3        0.00        perf-profile.children.cycles-pp.start_secondary
     35.41 ±  9%     -35.4        0.00        perf-profile.children.cycles-pp.read
     33.03 ± 17%     -33.0        0.00        perf-profile.children.cycles-pp.intel_idle
     30.90 ±  9%     -30.9        0.00        perf-profile.children.cycles-pp.write
     20.96 ±  9%     -21.0        0.00        perf-profile.children.cycles-pp.ksys_read
     18.59 ±  9%     -18.6        0.00        perf-profile.children.cycles-pp.vfs_read
     16.59 ±  9%     -16.6        0.00        perf-profile.children.cycles-pp.ksys_write
     14.22 ±  9%     -14.2        0.00        perf-profile.children.cycles-pp.vfs_write
     10.75 ±  9%     -10.7        0.00        perf-profile.children.cycles-pp.new_sync_read
      9.12 ±  9%      -9.1        0.00        perf-profile.children.cycles-pp.eventfd_read
      8.41 ±  8%      -8.4        0.00        perf-profile.children.cycles-pp.__entry_text_start
      8.20 ±  9%      -8.2        0.00        perf-profile.children.cycles-pp.security_file_permission
      8.03 ±  9%      -8.0        0.00        perf-profile.children.cycles-pp.eventfd_write
      6.46 ±  9%      -6.5        0.00        perf-profile.children.cycles-pp.syscall_return_via_sysret
      5.46 ±  8%      -5.5        0.00        perf-profile.children.cycles-pp.common_file_perm
      5.24 ±  9%      -5.2        0.00        perf-profile.children.cycles-pp._copy_to_iter
     33.03 ± 17%     -33.0        0.00        perf-profile.self.cycles-pp.intel_idle
      6.46 ±  9%      -6.5        0.00        perf-profile.self.cycles-pp.syscall_return_via_sysret
      0.01 ± 47%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown]
      0.02 ± 90%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.devkmsg_read.vfs_read.ksys_read.do_syscall_64
      0.01 ± 35%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.do_nanosleep.hrtimer_nanosleep.__x64_sys_nanosleep.do_syscall_64
      0.01 ± 92%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.do_syslog.part.0.kmsg_read.vfs_read
      0.56 ±221%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
      0.01 ± 45%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64
      0.01 ± 34%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.futex_wait_queue_me.futex_wait.do_futex.__x64_sys_futex
      0.00 ± 57%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.pipe_read.new_sync_read.vfs_read.ksys_read
      0.03 ±115%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.rcu_gp_kthread.kthread.ret_from_fork
      0.01 ± 91%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait.__x64_sys_epoll_wait
      0.01 ± 64%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_select
      0.01 ± 38%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_sys_poll
      0.02 ± 56%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.schedule_timeout.kcompactd.kthread.ret_from_fork
      0.01 ± 88%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.schedule_timeout.rcu_gp_kthread.kthread.ret_from_fork
      0.01 ± 50%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.schedule_timeout.wait_for_completion.__flush_work.lru_add_drain_all
      0.01 ± 28%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.smpboot_thread_fn.kthread.ret_from_fork
      0.00 ± 19%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.wait_for_partner.fifo_open.do_dentry_open.do_open.isra
      0.24 ±206%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.worker_thread.kthread.ret_from_fork
      0.02 ± 50%    -100.0%       0.00        perf-sched.sch_delay.max.ms.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown]
      0.04 ± 97%    -100.0%       0.00        perf-sched.sch_delay.max.ms.devkmsg_read.vfs_read.ksys_read.do_syscall_64
      0.03 ± 52%    -100.0%       0.00        perf-sched.sch_delay.max.ms.do_nanosleep.hrtimer_nanosleep.__x64_sys_nanosleep.do_syscall_64
      0.02 ±117%    -100.0%       0.00        perf-sched.sch_delay.max.ms.do_syslog.part.0.kmsg_read.vfs_read
    166.75 ±223%    -100.0%       0.00        perf-sched.sch_delay.max.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
      0.07 ± 79%    -100.0%       0.00        perf-sched.sch_delay.max.ms.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64
      0.01 ± 31%    -100.0%       0.00        perf-sched.sch_delay.max.ms.exit_to_user_mode_prepare.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe.[unknown]
      0.01 ± 35%    -100.0%       0.00        perf-sched.sch_delay.max.ms.futex_wait_queue_me.futex_wait.do_futex.__x64_sys_futex
      0.32 ±176%    -100.0%       0.00        perf-sched.sch_delay.max.ms.pipe_read.new_sync_read.vfs_read.ksys_read
      0.06 ± 13%    -100.0%       0.00        perf-sched.sch_delay.max.ms.preempt_schedule_common.__cond_resched.stop_one_cpu.affine_move_task.__set_cpus_allowed_ptr
      0.05 ±110%    -100.0%       0.00        perf-sched.sch_delay.max.ms.rcu_gp_kthread.kthread.ret_from_fork
      0.04 ±120%    -100.0%       0.00        perf-sched.sch_delay.max.ms.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait.__x64_sys_epoll_wait
      0.01 ± 95%    -100.0%       0.00        perf-sched.sch_delay.max.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_select
      0.02 ± 45%    -100.0%       0.00        perf-sched.sch_delay.max.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_sys_poll
      0.19 ±192%    -100.0%       0.00        perf-sched.sch_delay.max.ms.schedule_timeout.kcompactd.kthread.ret_from_fork
      0.88 ±209%    -100.0%       0.00        perf-sched.sch_delay.max.ms.schedule_timeout.rcu_gp_kthread.kthread.ret_from_fork
      0.01 ± 44%    -100.0%       0.00        perf-sched.sch_delay.max.ms.schedule_timeout.wait_for_completion.__flush_work.lru_add_drain_all
      1.46 ±211%    -100.0%       0.00        perf-sched.sch_delay.max.ms.smpboot_thread_fn.kthread.ret_from_fork
      0.01 ± 43%    -100.0%       0.00        perf-sched.sch_delay.max.ms.wait_for_partner.fifo_open.do_dentry_open.do_open.isra
    141.01 ±218%    -100.0%       0.00        perf-sched.sch_delay.max.ms.worker_thread.kthread.ret_from_fork
      0.03 ±115%    -100.0%       0.00        perf-sched.total_sch_delay.average.ms
    308.12 ±140%    -100.0%       0.00        perf-sched.total_sch_delay.max.ms
    218.18 ±  4%    -100.0%       0.00        perf-sched.total_wait_and_delay.average.ms
     11274 ±  3%    -100.0%       0.00        perf-sched.total_wait_and_delay.count.ms
      8874 ±  3%    -100.0%       0.00        perf-sched.total_wait_and_delay.max.ms
    218.15 ±  4%    -100.0%       0.00        perf-sched.total_wait_time.average.ms
      8874 ±  3%    -100.0%       0.00        perf-sched.total_wait_time.max.ms
    899.50          -100.0%       0.00        perf-sched.wait_and_delay.avg.ms.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown]
      1631 ± 33%    -100.0%       0.00        perf-sched.wait_and_delay.avg.ms.devkmsg_read.vfs_read.ksys_read.do_syscall_64
    797.01 ±  4%    -100.0%       0.00        perf-sched.wait_and_delay.avg.ms.do_nanosleep.hrtimer_nanosleep.__x64_sys_nanosleep.do_syscall_64
      1631 ± 33%    -100.0%       0.00        perf-sched.wait_and_delay.avg.ms.do_syslog.part.0.kmsg_read.vfs_read
    218.64 ±  3%    -100.0%       0.00        perf-sched.wait_and_delay.avg.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
      0.74 ±  2%    -100.0%       0.00        perf-sched.wait_and_delay.avg.ms.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64
      0.03 ±  6%    -100.0%       0.00        perf-sched.wait_and_delay.avg.ms.exit_to_user_mode_prepare.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe.[unknown]
      1013 ± 22%    -100.0%       0.00        perf-sched.wait_and_delay.avg.ms.futex_wait_queue_me.futex_wait.do_futex.__x64_sys_futex
     54.40 ±  3%    -100.0%       0.00        perf-sched.wait_and_delay.avg.ms.pipe_read.new_sync_read.vfs_read.ksys_read
      4.57 ± 11%    -100.0%       0.00        perf-sched.wait_and_delay.avg.ms.preempt_schedule_common.__cond_resched.stop_one_cpu.affine_move_task.__set_cpus_allowed_ptr
      2.76 ±  8%    -100.0%       0.00        perf-sched.wait_and_delay.avg.ms.rcu_gp_kthread.kthread.ret_from_fork
    518.43 ± 19%    -100.0%       0.00        perf-sched.wait_and_delay.avg.ms.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait.__x64_sys_epoll_wait
    599.96 ± 13%    -100.0%       0.00        perf-sched.wait_and_delay.avg.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_sys_poll
    478.26          -100.0%       0.00        perf-sched.wait_and_delay.avg.ms.schedule_timeout.kcompactd.kthread.ret_from_fork
      6.30 ± 23%    -100.0%       0.00        perf-sched.wait_and_delay.avg.ms.schedule_timeout.rcu_gp_kthread.kthread.ret_from_fork
    691.51          -100.0%       0.00        perf-sched.wait_and_delay.avg.ms.smpboot_thread_fn.kthread.ret_from_fork
      0.00 ± 19%    -100.0%       0.00        perf-sched.wait_and_delay.avg.ms.wait_for_partner.fifo_open.do_dentry_open.do_open.isra
    440.34 ±  6%    -100.0%       0.00        perf-sched.wait_and_delay.avg.ms.worker_thread.kthread.ret_from_fork
     10.00          -100.0%       0.00        perf-sched.wait_and_delay.count.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown]
      5.50 ± 27%    -100.0%       0.00        perf-sched.wait_and_delay.count.devkmsg_read.vfs_read.ksys_read.do_syscall_64
     22.67 ±  4%    -100.0%       0.00        perf-sched.wait_and_delay.count.do_nanosleep.hrtimer_nanosleep.__x64_sys_nanosleep.do_syscall_64
      5.50 ± 27%    -100.0%       0.00        perf-sched.wait_and_delay.count.do_syslog.part.0.kmsg_read.vfs_read
    302.50          -100.0%       0.00        perf-sched.wait_and_delay.count.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
    294.67          -100.0%       0.00        perf-sched.wait_and_delay.count.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64
    215.83 ± 12%    -100.0%       0.00        perf-sched.wait_and_delay.count.exit_to_user_mode_prepare.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe.[unknown]
      3.50 ± 42%    -100.0%       0.00        perf-sched.wait_and_delay.count.futex_wait_queue_me.futex_wait.do_futex.__x64_sys_futex
      2947 ±  2%    -100.0%       0.00        perf-sched.wait_and_delay.count.pipe_read.new_sync_read.vfs_read.ksys_read
      2208 ±  4%    -100.0%       0.00        perf-sched.wait_and_delay.count.preempt_schedule_common.__cond_resched.stop_one_cpu.affine_move_task.__set_cpus_allowed_ptr
    100.50 ± 24%    -100.0%       0.00        perf-sched.wait_and_delay.count.rcu_gp_kthread.kthread.ret_from_fork
     22.83 ± 35%    -100.0%       0.00        perf-sched.wait_and_delay.count.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait.__x64_sys_epoll_wait
     55.17 ±  9%    -100.0%       0.00        perf-sched.wait_and_delay.count.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_sys_poll
     78.33 ±  2%    -100.0%       0.00        perf-sched.wait_and_delay.count.schedule_timeout.kcompactd.kthread.ret_from_fork
      1642 ± 27%    -100.0%       0.00        perf-sched.wait_and_delay.count.schedule_timeout.rcu_gp_kthread.kthread.ret_from_fork
      2593 ±  3%    -100.0%       0.00        perf-sched.wait_and_delay.count.smpboot_thread_fn.kthread.ret_from_fork
     71.33          -100.0%       0.00        perf-sched.wait_and_delay.count.wait_for_partner.fifo_open.do_dentry_open.do_open.isra
    651.50 ±  4%    -100.0%       0.00        perf-sched.wait_and_delay.count.worker_thread.kthread.ret_from_fork
    999.62          -100.0%       0.00        perf-sched.wait_and_delay.max.ms.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown]
      4739 ± 36%    -100.0%       0.00        perf-sched.wait_and_delay.max.ms.devkmsg_read.vfs_read.ksys_read.do_syscall_64
      1000          -100.0%       0.00        perf-sched.wait_and_delay.max.ms.do_nanosleep.hrtimer_nanosleep.__x64_sys_nanosleep.do_syscall_64
      4739 ± 36%    -100.0%       0.00        perf-sched.wait_and_delay.max.ms.do_syslog.part.0.kmsg_read.vfs_read
      3519 ± 49%    -100.0%       0.00        perf-sched.wait_and_delay.max.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
     16.30          -100.0%       0.00        perf-sched.wait_and_delay.max.ms.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64
      0.12 ± 30%    -100.0%       0.00        perf-sched.wait_and_delay.max.ms.exit_to_user_mode_prepare.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe.[unknown]
      2009 ± 22%    -100.0%       0.00        perf-sched.wait_and_delay.max.ms.futex_wait_queue_me.futex_wait.do_futex.__x64_sys_futex
      4741 ± 36%    -100.0%       0.00        perf-sched.wait_and_delay.max.ms.pipe_read.new_sync_read.vfs_read.ksys_read
      1768 ± 43%    -100.0%       0.00        perf-sched.wait_and_delay.max.ms.preempt_schedule_common.__cond_resched.stop_one_cpu.affine_move_task.__set_cpus_allowed_ptr
      4.86 ±  6%    -100.0%       0.00        perf-sched.wait_and_delay.max.ms.rcu_gp_kthread.kthread.ret_from_fork
      4385 ± 40%    -100.0%       0.00        perf-sched.wait_and_delay.max.ms.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait.__x64_sys_epoll_wait
      6937 ± 12%    -100.0%       0.00        perf-sched.wait_and_delay.max.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_sys_poll
    504.87          -100.0%       0.00        perf-sched.wait_and_delay.max.ms.schedule_timeout.kcompactd.kthread.ret_from_fork
    210.35 ± 49%    -100.0%       0.00        perf-sched.wait_and_delay.max.ms.schedule_timeout.rcu_gp_kthread.kthread.ret_from_fork
      1226 ± 35%    -100.0%       0.00        perf-sched.wait_and_delay.max.ms.smpboot_thread_fn.kthread.ret_from_fork
      0.01 ± 43%    -100.0%       0.00        perf-sched.wait_and_delay.max.ms.wait_for_partner.fifo_open.do_dentry_open.do_open.isra
      8874 ±  3%    -100.0%       0.00        perf-sched.wait_and_delay.max.ms.worker_thread.kthread.ret_from_fork
    899.48          -100.0%       0.00        perf-sched.wait_time.avg.ms.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown]
      1631 ± 33%    -100.0%       0.00        perf-sched.wait_time.avg.ms.devkmsg_read.vfs_read.ksys_read.do_syscall_64
    797.00 ±  4%    -100.0%       0.00        perf-sched.wait_time.avg.ms.do_nanosleep.hrtimer_nanosleep.__x64_sys_nanosleep.do_syscall_64
      1631 ± 33%    -100.0%       0.00        perf-sched.wait_time.avg.ms.do_syslog.part.0.kmsg_read.vfs_read
    218.08 ±  3%    -100.0%       0.00        perf-sched.wait_time.avg.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
      0.74          -100.0%       0.00        perf-sched.wait_time.avg.ms.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64
      0.02 ± 18%    -100.0%       0.00        perf-sched.wait_time.avg.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt.[unknown]
      0.03 ±  6%    -100.0%       0.00        perf-sched.wait_time.avg.ms.exit_to_user_mode_prepare.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe.[unknown]
      1013 ± 22%    -100.0%       0.00        perf-sched.wait_time.avg.ms.futex_wait_queue_me.futex_wait.do_futex.__x64_sys_futex
     54.39 ±  3%    -100.0%       0.00        perf-sched.wait_time.avg.ms.pipe_read.new_sync_read.vfs_read.ksys_read
      4.57 ± 11%    -100.0%       0.00        perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.stop_one_cpu.affine_move_task.__set_cpus_allowed_ptr
      2.74 ±  9%    -100.0%       0.00        perf-sched.wait_time.avg.ms.rcu_gp_kthread.kthread.ret_from_fork
    518.41 ± 19%    -100.0%       0.00        perf-sched.wait_time.avg.ms.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait.__x64_sys_epoll_wait
    599.95 ± 13%    -100.0%       0.00        perf-sched.wait_time.avg.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_sys_poll
    478.24          -100.0%       0.00        perf-sched.wait_time.avg.ms.schedule_timeout.kcompactd.kthread.ret_from_fork
      0.01 ± 50%    -100.0%       0.00        perf-sched.wait_time.avg.ms.schedule_timeout.khugepaged.kthread.ret_from_fork
      6.29 ± 23%    -100.0%       0.00        perf-sched.wait_time.avg.ms.schedule_timeout.rcu_gp_kthread.kthread.ret_from_fork
    691.51          -100.0%       0.00        perf-sched.wait_time.avg.ms.smpboot_thread_fn.kthread.ret_from_fork
    440.10 ±  6%    -100.0%       0.00        perf-sched.wait_time.avg.ms.worker_thread.kthread.ret_from_fork
    999.61          -100.0%       0.00        perf-sched.wait_time.max.ms.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown]
      4739 ± 36%    -100.0%       0.00        perf-sched.wait_time.max.ms.devkmsg_read.vfs_read.ksys_read.do_syscall_64
      1000          -100.0%       0.00        perf-sched.wait_time.max.ms.do_nanosleep.hrtimer_nanosleep.__x64_sys_nanosleep.do_syscall_64
      4739 ± 36%    -100.0%       0.00        perf-sched.wait_time.max.ms.do_syslog.part.0.kmsg_read.vfs_read
      3519 ± 49%    -100.0%       0.00        perf-sched.wait_time.max.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
     16.30          -100.0%       0.00        perf-sched.wait_time.max.ms.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64
      0.07 ± 58%    -100.0%       0.00        perf-sched.wait_time.max.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt.[unknown]
      0.12 ± 30%    -100.0%       0.00        perf-sched.wait_time.max.ms.exit_to_user_mode_prepare.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe.[unknown]
      2009 ± 22%    -100.0%       0.00        perf-sched.wait_time.max.ms.futex_wait_queue_me.futex_wait.do_futex.__x64_sys_futex
      4741 ± 36%    -100.0%       0.00        perf-sched.wait_time.max.ms.pipe_read.new_sync_read.vfs_read.ksys_read
      1768 ± 43%    -100.0%       0.00        perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.stop_one_cpu.affine_move_task.__set_cpus_allowed_ptr
      4.83 ±  7%    -100.0%       0.00        perf-sched.wait_time.max.ms.rcu_gp_kthread.kthread.ret_from_fork
      4384 ± 40%    -100.0%       0.00        perf-sched.wait_time.max.ms.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait.__x64_sys_epoll_wait
      6937 ± 12%    -100.0%       0.00        perf-sched.wait_time.max.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_sys_poll
    504.85          -100.0%       0.00        perf-sched.wait_time.max.ms.schedule_timeout.kcompactd.kthread.ret_from_fork
      0.01 ± 50%    -100.0%       0.00        perf-sched.wait_time.max.ms.schedule_timeout.khugepaged.kthread.ret_from_fork
    210.32 ± 49%    -100.0%       0.00        perf-sched.wait_time.max.ms.schedule_timeout.rcu_gp_kthread.kthread.ret_from_fork
      1226 ± 35%    -100.0%       0.00        perf-sched.wait_time.max.ms.smpboot_thread_fn.kthread.ret_from_fork
      8874 ±  3%    -100.0%       0.00        perf-sched.wait_time.max.ms.worker_thread.kthread.ret_from_fork
      4703 ± 53%    -100.0%       0.00        interrupts.CPU0.NMI:Non-maskable_interrupts
      4703 ± 53%    -100.0%       0.00        interrupts.CPU0.PMI:Performance_monitoring_interrupts
      5649 ± 32%    -100.0%       0.00        interrupts.CPU1.NMI:Non-maskable_interrupts
      5649 ± 32%    -100.0%       0.00        interrupts.CPU1.PMI:Performance_monitoring_interrupts
      6583 ± 37%    -100.0%       0.00        interrupts.CPU10.NMI:Non-maskable_interrupts
      6583 ± 37%    -100.0%       0.00        interrupts.CPU10.PMI:Performance_monitoring_interrupts
      5033 ± 40%    -100.0%       0.00        interrupts.CPU100.NMI:Non-maskable_interrupts
      5033 ± 40%    -100.0%       0.00        interrupts.CPU100.PMI:Performance_monitoring_interrupts
      7687 ± 23%    -100.0%       1.00 ±223%  interrupts.CPU101.NMI:Non-maskable_interrupts
      7687 ± 23%    -100.0%       1.00 ±223%  interrupts.CPU101.PMI:Performance_monitoring_interrupts
      5400 ± 39%    -100.0%       0.00        interrupts.CPU102.NMI:Non-maskable_interrupts
      5400 ± 39%    -100.0%       0.00        interrupts.CPU102.PMI:Performance_monitoring_interrupts
      4291 ± 23%    -100.0%       0.00        interrupts.CPU103.NMI:Non-maskable_interrupts
      4291 ± 23%    -100.0%       0.00        interrupts.CPU103.PMI:Performance_monitoring_interrupts
      4648 ± 36%    -100.0%       0.00        interrupts.CPU104.NMI:Non-maskable_interrupts
      4648 ± 36%    -100.0%       0.00        interrupts.CPU104.PMI:Performance_monitoring_interrupts
      3899 ± 31%    -100.0%       0.00        interrupts.CPU105.NMI:Non-maskable_interrupts
      3899 ± 31%    -100.0%       0.00        interrupts.CPU105.PMI:Performance_monitoring_interrupts
      5477 ± 47%    -100.0%       0.00        interrupts.CPU106.NMI:Non-maskable_interrupts
      5477 ± 47%    -100.0%       0.00        interrupts.CPU106.PMI:Performance_monitoring_interrupts
      6150 ± 40%    -100.0%       0.00        interrupts.CPU107.NMI:Non-maskable_interrupts
      6150 ± 40%    -100.0%       0.00        interrupts.CPU107.PMI:Performance_monitoring_interrupts
      4603 ± 53%    -100.0%       0.00        interrupts.CPU108.NMI:Non-maskable_interrupts
      4603 ± 53%    -100.0%       0.00        interrupts.CPU108.PMI:Performance_monitoring_interrupts
      6039 ± 40%    -100.0%       0.00        interrupts.CPU109.NMI:Non-maskable_interrupts
      6039 ± 40%    -100.0%       0.00        interrupts.CPU109.PMI:Performance_monitoring_interrupts
      5284 ± 46%    -100.0%       0.00        interrupts.CPU11.NMI:Non-maskable_interrupts
      5284 ± 46%    -100.0%       0.00        interrupts.CPU11.PMI:Performance_monitoring_interrupts
      6376 ± 35%    -100.0%       0.00        interrupts.CPU110.NMI:Non-maskable_interrupts
      6376 ± 35%    -100.0%       0.00        interrupts.CPU110.PMI:Performance_monitoring_interrupts
      4144 ± 38%    -100.0%       0.00        interrupts.CPU111.NMI:Non-maskable_interrupts
      4144 ± 38%    -100.0%       0.00        interrupts.CPU111.PMI:Performance_monitoring_interrupts
    248.67 ± 79%    -100.0%       0.00        interrupts.CPU112.NMI:Non-maskable_interrupts
    248.67 ± 79%    -100.0%       0.00        interrupts.CPU112.PMI:Performance_monitoring_interrupts
    109.50 ±  9%    -100.0%       0.00        interrupts.CPU113.NMI:Non-maskable_interrupts
    109.50 ±  9%    -100.0%       0.00        interrupts.CPU113.PMI:Performance_monitoring_interrupts
    228.67 ±120%    -100.0%       0.00        interrupts.CPU114.NMI:Non-maskable_interrupts
    228.67 ±120%    -100.0%       0.00        interrupts.CPU114.PMI:Performance_monitoring_interrupts
    109.50 ± 33%    -100.0%       0.00        interrupts.CPU115.NMI:Non-maskable_interrupts
    109.50 ± 33%    -100.0%       0.00        interrupts.CPU115.PMI:Performance_monitoring_interrupts
    106.50 ± 27%    -100.0%       0.00        interrupts.CPU116.NMI:Non-maskable_interrupts
    106.50 ± 27%    -100.0%       0.00        interrupts.CPU116.PMI:Performance_monitoring_interrupts
    107.00 ± 10%    -100.0%       0.00        interrupts.CPU117.NMI:Non-maskable_interrupts
    107.00 ± 10%    -100.0%       0.00        interrupts.CPU117.PMI:Performance_monitoring_interrupts
     89.50 ± 29%    -100.0%       0.00        interrupts.CPU118.NMI:Non-maskable_interrupts
     89.50 ± 29%    -100.0%       0.00        interrupts.CPU118.PMI:Performance_monitoring_interrupts
     97.33 ± 18%    -100.0%       0.00        interrupts.CPU119.NMI:Non-maskable_interrupts
     97.33 ± 18%    -100.0%       0.00        interrupts.CPU119.PMI:Performance_monitoring_interrupts
      7484 ± 36%    -100.0%       0.00        interrupts.CPU12.NMI:Non-maskable_interrupts
      7484 ± 36%    -100.0%       0.00        interrupts.CPU12.PMI:Performance_monitoring_interrupts
    114.33 ± 32%    -100.0%       0.00        interrupts.CPU120.NMI:Non-maskable_interrupts
    114.33 ± 32%    -100.0%       0.00        interrupts.CPU120.PMI:Performance_monitoring_interrupts
    105.50 ± 40%    -100.0%       0.00        interrupts.CPU121.NMI:Non-maskable_interrupts
    105.50 ± 40%    -100.0%       0.00        interrupts.CPU121.PMI:Performance_monitoring_interrupts
    108.00 ± 45%    -100.0%       0.00        interrupts.CPU122.NMI:Non-maskable_interrupts
    108.00 ± 45%    -100.0%       0.00        interrupts.CPU122.PMI:Performance_monitoring_interrupts
    139.17 ± 78%    -100.0%       0.00        interrupts.CPU123.NMI:Non-maskable_interrupts
    139.17 ± 78%    -100.0%       0.00        interrupts.CPU123.PMI:Performance_monitoring_interrupts
    113.83 ± 44%    -100.0%       0.00        interrupts.CPU124.NMI:Non-maskable_interrupts
    113.83 ± 44%    -100.0%       0.00        interrupts.CPU124.PMI:Performance_monitoring_interrupts
    124.83 ± 47%    -100.0%       0.00        interrupts.CPU125.NMI:Non-maskable_interrupts
    124.83 ± 47%    -100.0%       0.00        interrupts.CPU125.PMI:Performance_monitoring_interrupts
    139.83 ± 60%    -100.0%       0.00        interrupts.CPU126.NMI:Non-maskable_interrupts
    139.83 ± 60%    -100.0%       0.00        interrupts.CPU126.PMI:Performance_monitoring_interrupts
    110.67 ± 47%     -99.1%       1.00 ±223%  interrupts.CPU127.NMI:Non-maskable_interrupts
    110.67 ± 47%     -99.1%       1.00 ±223%  interrupts.CPU127.PMI:Performance_monitoring_interrupts
    105.00 ± 42%    -100.0%       0.00        interrupts.CPU128.NMI:Non-maskable_interrupts
    105.00 ± 42%    -100.0%       0.00        interrupts.CPU128.PMI:Performance_monitoring_interrupts
    102.00 ± 40%    -100.0%       0.00        interrupts.CPU129.NMI:Non-maskable_interrupts
    102.00 ± 40%    -100.0%       0.00        interrupts.CPU129.PMI:Performance_monitoring_interrupts
      6284 ± 40%    -100.0%       0.00        interrupts.CPU13.NMI:Non-maskable_interrupts
      6284 ± 40%    -100.0%       0.00        interrupts.CPU13.PMI:Performance_monitoring_interrupts
    112.83 ± 32%    -100.0%       0.00        interrupts.CPU130.NMI:Non-maskable_interrupts
    112.83 ± 32%    -100.0%       0.00        interrupts.CPU130.PMI:Performance_monitoring_interrupts
    118.50 ± 37%    -100.0%       0.00        interrupts.CPU131.NMI:Non-maskable_interrupts
    118.50 ± 37%    -100.0%       0.00        interrupts.CPU131.PMI:Performance_monitoring_interrupts
    115.17 ± 33%    -100.0%       0.00        interrupts.CPU132.NMI:Non-maskable_interrupts
    115.17 ± 33%    -100.0%       0.00        interrupts.CPU132.PMI:Performance_monitoring_interrupts
    112.50 ± 31%    -100.0%       0.00        interrupts.CPU133.NMI:Non-maskable_interrupts
    112.50 ± 31%    -100.0%       0.00        interrupts.CPU133.PMI:Performance_monitoring_interrupts
    105.33 ± 38%    -100.0%       0.00        interrupts.CPU134.NMI:Non-maskable_interrupts
    105.33 ± 38%    -100.0%       0.00        interrupts.CPU134.PMI:Performance_monitoring_interrupts
    102.33 ± 40%    -100.0%       0.00        interrupts.CPU135.NMI:Non-maskable_interrupts
    102.33 ± 40%    -100.0%       0.00        interrupts.CPU135.PMI:Performance_monitoring_interrupts
    102.83 ± 38%    -100.0%       0.00        interrupts.CPU136.NMI:Non-maskable_interrupts
    102.83 ± 38%    -100.0%       0.00        interrupts.CPU136.PMI:Performance_monitoring_interrupts
    102.33 ± 40%    -100.0%       0.00        interrupts.CPU137.NMI:Non-maskable_interrupts
    102.33 ± 40%    -100.0%       0.00        interrupts.CPU137.PMI:Performance_monitoring_interrupts
    104.00 ± 40%    -100.0%       0.00        interrupts.CPU138.NMI:Non-maskable_interrupts
    104.00 ± 40%    -100.0%       0.00        interrupts.CPU138.PMI:Performance_monitoring_interrupts
    110.00 ± 30%    -100.0%       0.00        interrupts.CPU139.NMI:Non-maskable_interrupts
    110.00 ± 30%    -100.0%       0.00        interrupts.CPU139.PMI:Performance_monitoring_interrupts
      5200 ± 40%    -100.0%       0.00        interrupts.CPU14.NMI:Non-maskable_interrupts
      5200 ± 40%    -100.0%       0.00        interrupts.CPU14.PMI:Performance_monitoring_interrupts
    102.50 ± 40%    -100.0%       0.00        interrupts.CPU140.NMI:Non-maskable_interrupts
    102.50 ± 40%    -100.0%       0.00        interrupts.CPU140.PMI:Performance_monitoring_interrupts
    103.50 ± 40%     -99.0%       1.00 ±223%  interrupts.CPU141.NMI:Non-maskable_interrupts
    103.50 ± 40%     -99.0%       1.00 ±223%  interrupts.CPU141.PMI:Performance_monitoring_interrupts
    103.67 ± 41%    -100.0%       0.00        interrupts.CPU142.NMI:Non-maskable_interrupts
    103.67 ± 41%    -100.0%       0.00        interrupts.CPU142.PMI:Performance_monitoring_interrupts
    104.33 ± 41%    -100.0%       0.00        interrupts.CPU143.NMI:Non-maskable_interrupts
    104.33 ± 41%    -100.0%       0.00        interrupts.CPU143.PMI:Performance_monitoring_interrupts
    168.33 ± 42%    -100.0%       0.00        interrupts.CPU144.NMI:Non-maskable_interrupts
    168.33 ± 42%    -100.0%       0.00        interrupts.CPU144.PMI:Performance_monitoring_interrupts
    103.33 ± 39%    -100.0%       0.00        interrupts.CPU145.NMI:Non-maskable_interrupts
    103.33 ± 39%    -100.0%       0.00        interrupts.CPU145.PMI:Performance_monitoring_interrupts
    106.83 ± 38%    -100.0%       0.00        interrupts.CPU146.NMI:Non-maskable_interrupts
    106.83 ± 38%    -100.0%       0.00        interrupts.CPU146.PMI:Performance_monitoring_interrupts
    114.50 ± 31%    -100.0%       0.00        interrupts.CPU147.NMI:Non-maskable_interrupts
    114.50 ± 31%    -100.0%       0.00        interrupts.CPU147.PMI:Performance_monitoring_interrupts
    116.17 ± 30%    -100.0%       0.00        interrupts.CPU148.NMI:Non-maskable_interrupts
    116.17 ± 30%    -100.0%       0.00        interrupts.CPU148.PMI:Performance_monitoring_interrupts
    124.33 ± 34%    -100.0%       0.00        interrupts.CPU149.NMI:Non-maskable_interrupts
    124.33 ± 34%    -100.0%       0.00        interrupts.CPU149.PMI:Performance_monitoring_interrupts
      7696 ± 20%    -100.0%       0.00        interrupts.CPU15.NMI:Non-maskable_interrupts
      7696 ± 20%    -100.0%       0.00        interrupts.CPU15.PMI:Performance_monitoring_interrupts
    124.83 ± 31%    -100.0%       0.00        interrupts.CPU150.NMI:Non-maskable_interrupts
    124.83 ± 31%    -100.0%       0.00        interrupts.CPU150.PMI:Performance_monitoring_interrupts
    124.00 ± 36%    -100.0%       0.00        interrupts.CPU151.NMI:Non-maskable_interrupts
    124.00 ± 36%    -100.0%       0.00        interrupts.CPU151.PMI:Performance_monitoring_interrupts
    114.33 ± 31%    -100.0%       0.00        interrupts.CPU152.NMI:Non-maskable_interrupts
    114.33 ± 31%    -100.0%       0.00        interrupts.CPU152.PMI:Performance_monitoring_interrupts
    118.17 ± 30%    -100.0%       0.00        interrupts.CPU153.NMI:Non-maskable_interrupts
    118.17 ± 30%    -100.0%       0.00        interrupts.CPU153.PMI:Performance_monitoring_interrupts
    116.33 ± 33%    -100.0%       0.00        interrupts.CPU154.NMI:Non-maskable_interrupts
    116.33 ± 33%    -100.0%       0.00        interrupts.CPU154.PMI:Performance_monitoring_interrupts
    118.17 ± 32%    -100.0%       0.00        interrupts.CPU155.NMI:Non-maskable_interrupts
    118.17 ± 32%    -100.0%       0.00        interrupts.CPU155.PMI:Performance_monitoring_interrupts
    121.83 ± 36%    -100.0%       0.00        interrupts.CPU156.NMI:Non-maskable_interrupts
    121.83 ± 36%    -100.0%       0.00        interrupts.CPU156.PMI:Performance_monitoring_interrupts
    124.33 ± 18%    -100.0%       0.00        interrupts.CPU157.NMI:Non-maskable_interrupts
    124.33 ± 18%    -100.0%       0.00        interrupts.CPU157.PMI:Performance_monitoring_interrupts
    131.67 ± 21%    -100.0%       0.00        interrupts.CPU158.NMI:Non-maskable_interrupts
    131.67 ± 21%    -100.0%       0.00        interrupts.CPU158.PMI:Performance_monitoring_interrupts
    123.33 ± 18%    -100.0%       0.00        interrupts.CPU159.NMI:Non-maskable_interrupts
    123.33 ± 18%    -100.0%       0.00        interrupts.CPU159.PMI:Performance_monitoring_interrupts
    164.33 ± 58%    -100.0%       0.00        interrupts.CPU16.NMI:Non-maskable_interrupts
    164.33 ± 58%    -100.0%       0.00        interrupts.CPU16.PMI:Performance_monitoring_interrupts
    125.33 ± 18%    -100.0%       0.00        interrupts.CPU160.NMI:Non-maskable_interrupts
    125.33 ± 18%    -100.0%       0.00        interrupts.CPU160.PMI:Performance_monitoring_interrupts
    126.50 ± 21%    -100.0%       0.00        interrupts.CPU161.NMI:Non-maskable_interrupts
    126.50 ± 21%    -100.0%       0.00        interrupts.CPU161.PMI:Performance_monitoring_interrupts
    124.00 ± 19%     -99.2%       1.00 ±223%  interrupts.CPU162.NMI:Non-maskable_interrupts
    124.00 ± 19%     -99.2%       1.00 ±223%  interrupts.CPU162.PMI:Performance_monitoring_interrupts
    127.67 ± 21%    -100.0%       0.00        interrupts.CPU163.NMI:Non-maskable_interrupts
    127.67 ± 21%    -100.0%       0.00        interrupts.CPU163.PMI:Performance_monitoring_interrupts
    124.83 ± 22%    -100.0%       0.00        interrupts.CPU164.NMI:Non-maskable_interrupts
    124.83 ± 22%    -100.0%       0.00        interrupts.CPU164.PMI:Performance_monitoring_interrupts
    112.83 ± 31%    -100.0%       0.00        interrupts.CPU165.NMI:Non-maskable_interrupts
    112.83 ± 31%    -100.0%       0.00        interrupts.CPU165.PMI:Performance_monitoring_interrupts
    114.50 ± 31%    -100.0%       0.00        interrupts.CPU166.NMI:Non-maskable_interrupts
    114.50 ± 31%    -100.0%       0.00        interrupts.CPU166.PMI:Performance_monitoring_interrupts
    135.83 ± 35%    -100.0%       0.00        interrupts.CPU167.NMI:Non-maskable_interrupts
    135.83 ± 35%    -100.0%       0.00        interrupts.CPU167.PMI:Performance_monitoring_interrupts
    119.83 ± 44%    -100.0%       0.00        interrupts.CPU168.NMI:Non-maskable_interrupts
    119.83 ± 44%    -100.0%       0.00        interrupts.CPU168.PMI:Performance_monitoring_interrupts
    130.67 ± 16%    -100.0%       0.00        interrupts.CPU169.NMI:Non-maskable_interrupts
    130.67 ± 16%    -100.0%       0.00        interrupts.CPU169.PMI:Performance_monitoring_interrupts
    106.50 ±  9%    -100.0%       0.00        interrupts.CPU17.NMI:Non-maskable_interrupts
    106.50 ±  9%    -100.0%       0.00        interrupts.CPU17.PMI:Performance_monitoring_interrupts
    126.83 ± 17%    -100.0%       0.00        interrupts.CPU170.NMI:Non-maskable_interrupts
    126.83 ± 17%    -100.0%       0.00        interrupts.CPU170.PMI:Performance_monitoring_interrupts
    112.50 ± 21%    -100.0%       0.00        interrupts.CPU171.NMI:Non-maskable_interrupts
    112.50 ± 21%    -100.0%       0.00        interrupts.CPU171.PMI:Performance_monitoring_interrupts
    123.33 ± 16%    -100.0%       0.00        interrupts.CPU172.NMI:Non-maskable_interrupts
    123.33 ± 16%    -100.0%       0.00        interrupts.CPU172.PMI:Performance_monitoring_interrupts
    126.17 ± 15%    -100.0%       0.00        interrupts.CPU173.NMI:Non-maskable_interrupts
    126.17 ± 15%    -100.0%       0.00        interrupts.CPU173.PMI:Performance_monitoring_interrupts
    124.00 ± 16%    -100.0%       0.00        interrupts.CPU174.NMI:Non-maskable_interrupts
    124.00 ± 16%    -100.0%       0.00        interrupts.CPU174.PMI:Performance_monitoring_interrupts
    161.17 ± 60%    -100.0%       0.00        interrupts.CPU175.NMI:Non-maskable_interrupts
    161.17 ± 60%    -100.0%       0.00        interrupts.CPU175.PMI:Performance_monitoring_interrupts
    125.17 ± 18%    -100.0%       0.00        interrupts.CPU176.NMI:Non-maskable_interrupts
    125.17 ± 18%    -100.0%       0.00        interrupts.CPU176.PMI:Performance_monitoring_interrupts
    179.67 ± 70%    -100.0%       0.00        interrupts.CPU177.NMI:Non-maskable_interrupts
    179.67 ± 70%    -100.0%       0.00        interrupts.CPU177.PMI:Performance_monitoring_interrupts
    124.50 ± 19%    -100.0%       0.00        interrupts.CPU178.NMI:Non-maskable_interrupts
    124.50 ± 19%    -100.0%       0.00        interrupts.CPU178.PMI:Performance_monitoring_interrupts
    127.67 ± 18%    -100.0%       0.00        interrupts.CPU179.NMI:Non-maskable_interrupts
    127.67 ± 18%    -100.0%       0.00        interrupts.CPU179.PMI:Performance_monitoring_interrupts
    195.67 ±100%    -100.0%       0.00        interrupts.CPU18.NMI:Non-maskable_interrupts
    195.67 ±100%    -100.0%       0.00        interrupts.CPU18.PMI:Performance_monitoring_interrupts
    125.00 ± 18%    -100.0%       0.00        interrupts.CPU180.NMI:Non-maskable_interrupts
    125.00 ± 18%    -100.0%       0.00        interrupts.CPU180.PMI:Performance_monitoring_interrupts
    124.33 ± 17%    -100.0%       0.00        interrupts.CPU181.NMI:Non-maskable_interrupts
    124.33 ± 17%    -100.0%       0.00        interrupts.CPU181.PMI:Performance_monitoring_interrupts
    139.33 ± 27%    -100.0%       0.00        interrupts.CPU182.NMI:Non-maskable_interrupts
    139.33 ± 27%    -100.0%       0.00        interrupts.CPU182.PMI:Performance_monitoring_interrupts
    132.67 ± 25%    -100.0%       0.00        interrupts.CPU183.NMI:Non-maskable_interrupts
    132.67 ± 25%    -100.0%       0.00        interrupts.CPU183.PMI:Performance_monitoring_interrupts
    123.67 ± 17%    -100.0%       0.00        interrupts.CPU184.NMI:Non-maskable_interrupts
    123.67 ± 17%    -100.0%       0.00        interrupts.CPU184.PMI:Performance_monitoring_interrupts
    127.50 ± 19%    -100.0%       0.00        interrupts.CPU185.NMI:Non-maskable_interrupts
    127.50 ± 19%    -100.0%       0.00        interrupts.CPU185.PMI:Performance_monitoring_interrupts
    128.67 ± 17%    -100.0%       0.00        interrupts.CPU186.NMI:Non-maskable_interrupts
    128.67 ± 17%    -100.0%       0.00        interrupts.CPU186.PMI:Performance_monitoring_interrupts
    126.33 ± 17%    -100.0%       0.00        interrupts.CPU187.NMI:Non-maskable_interrupts
    126.33 ± 17%    -100.0%       0.00        interrupts.CPU187.PMI:Performance_monitoring_interrupts
    124.50 ± 17%    -100.0%       0.00        interrupts.CPU188.NMI:Non-maskable_interrupts
    124.50 ± 17%    -100.0%       0.00        interrupts.CPU188.PMI:Performance_monitoring_interrupts
    128.17 ± 23%    -100.0%       0.00        interrupts.CPU189.NMI:Non-maskable_interrupts
    128.17 ± 23%    -100.0%       0.00        interrupts.CPU189.PMI:Performance_monitoring_interrupts
     32977 ±216%     -97.3%     887.50        interrupts.CPU19.CAL:Function_call_interrupts
    111.33 ± 36%    -100.0%       0.00        interrupts.CPU19.NMI:Non-maskable_interrupts
    111.33 ± 36%    -100.0%       0.00        interrupts.CPU19.PMI:Performance_monitoring_interrupts
    124.00 ± 17%    -100.0%       0.00        interrupts.CPU190.NMI:Non-maskable_interrupts
    124.00 ± 17%    -100.0%       0.00        interrupts.CPU190.PMI:Performance_monitoring_interrupts
    542.50 ± 21%    -100.0%       0.00        interrupts.CPU191.NMI:Non-maskable_interrupts
    542.50 ± 21%    -100.0%       0.00        interrupts.CPU191.PMI:Performance_monitoring_interrupts
      7354 ± 36%    -100.0%       0.00        interrupts.CPU2.NMI:Non-maskable_interrupts
      7354 ± 36%    -100.0%       0.00        interrupts.CPU2.PMI:Performance_monitoring_interrupts
     97.67 ± 39%    -100.0%       0.00        interrupts.CPU20.NMI:Non-maskable_interrupts
     97.67 ± 39%    -100.0%       0.00        interrupts.CPU20.PMI:Performance_monitoring_interrupts
     87.67 ± 32%    -100.0%       0.00        interrupts.CPU21.NMI:Non-maskable_interrupts
     87.67 ± 32%    -100.0%       0.00        interrupts.CPU21.PMI:Performance_monitoring_interrupts
     82.50 ± 38%    -100.0%       0.00        interrupts.CPU22.NMI:Non-maskable_interrupts
     82.50 ± 38%    -100.0%       0.00        interrupts.CPU22.PMI:Performance_monitoring_interrupts
     89.00 ± 31%    -100.0%       0.00        interrupts.CPU23.NMI:Non-maskable_interrupts
     89.00 ± 31%    -100.0%       0.00        interrupts.CPU23.PMI:Performance_monitoring_interrupts
     98.17 ± 49%    -100.0%       0.00        interrupts.CPU24.NMI:Non-maskable_interrupts
     98.17 ± 49%    -100.0%       0.00        interrupts.CPU24.PMI:Performance_monitoring_interrupts
     87.50 ± 57%    -100.0%       0.00        interrupts.CPU25.NMI:Non-maskable_interrupts
     87.50 ± 57%    -100.0%       0.00        interrupts.CPU25.PMI:Performance_monitoring_interrupts
     99.67 ± 53%    -100.0%       0.00        interrupts.CPU26.NMI:Non-maskable_interrupts
     99.67 ± 53%    -100.0%       0.00        interrupts.CPU26.PMI:Performance_monitoring_interrupts
    115.83 ± 53%    -100.0%       0.00        interrupts.CPU27.NMI:Non-maskable_interrupts
    115.83 ± 53%    -100.0%       0.00        interrupts.CPU27.PMI:Performance_monitoring_interrupts
    130.83 ± 23%    -100.0%       0.00        interrupts.CPU28.NMI:Non-maskable_interrupts
    130.83 ± 23%    -100.0%       0.00        interrupts.CPU28.PMI:Performance_monitoring_interrupts
    124.50 ± 47%    -100.0%       0.00        interrupts.CPU29.NMI:Non-maskable_interrupts
    124.50 ± 47%    -100.0%       0.00        interrupts.CPU29.PMI:Performance_monitoring_interrupts
      5843 ± 41%    -100.0%       0.00        interrupts.CPU3.NMI:Non-maskable_interrupts
      5843 ± 41%    -100.0%       0.00        interrupts.CPU3.PMI:Performance_monitoring_interrupts
    184.50 ±108%    -100.0%       0.00        interrupts.CPU30.NMI:Non-maskable_interrupts
    184.50 ±108%    -100.0%       0.00        interrupts.CPU30.PMI:Performance_monitoring_interrupts
     94.33 ± 49%    -100.0%       0.00        interrupts.CPU31.NMI:Non-maskable_interrupts
     94.33 ± 49%    -100.0%       0.00        interrupts.CPU31.PMI:Performance_monitoring_interrupts
     93.83 ± 50%    -100.0%       0.00        interrupts.CPU32.NMI:Non-maskable_interrupts
     93.83 ± 50%    -100.0%       0.00        interrupts.CPU32.PMI:Performance_monitoring_interrupts
     88.17 ± 60%    -100.0%       0.00        interrupts.CPU33.NMI:Non-maskable_interrupts
     88.17 ± 60%    -100.0%       0.00        interrupts.CPU33.PMI:Performance_monitoring_interrupts
     85.67 ± 57%    -100.0%       0.00        interrupts.CPU34.NMI:Non-maskable_interrupts
     85.67 ± 57%    -100.0%       0.00        interrupts.CPU34.PMI:Performance_monitoring_interrupts
     98.50 ± 52%    -100.0%       0.00        interrupts.CPU35.NMI:Non-maskable_interrupts
     98.50 ± 52%    -100.0%       0.00        interrupts.CPU35.PMI:Performance_monitoring_interrupts
     96.50 ± 51%    -100.0%       0.00        interrupts.CPU36.NMI:Non-maskable_interrupts
     96.50 ± 51%    -100.0%       0.00        interrupts.CPU36.PMI:Performance_monitoring_interrupts
     94.67 ± 49%    -100.0%       0.00        interrupts.CPU37.NMI:Non-maskable_interrupts
     94.67 ± 49%    -100.0%       0.00        interrupts.CPU37.PMI:Performance_monitoring_interrupts
    105.67 ± 40%    -100.0%       0.00        interrupts.CPU38.NMI:Non-maskable_interrupts
    105.67 ± 40%    -100.0%       0.00        interrupts.CPU38.PMI:Performance_monitoring_interrupts
    116.67 ± 36%    -100.0%       0.00        interrupts.CPU39.NMI:Non-maskable_interrupts
    116.67 ± 36%    -100.0%       0.00        interrupts.CPU39.PMI:Performance_monitoring_interrupts
      5561 ± 38%    -100.0%       0.00        interrupts.CPU4.NMI:Non-maskable_interrupts
      5561 ± 38%    -100.0%       0.00        interrupts.CPU4.PMI:Performance_monitoring_interrupts
    121.00 ± 16%    -100.0%       0.00        interrupts.CPU40.NMI:Non-maskable_interrupts
    121.00 ± 16%    -100.0%       0.00        interrupts.CPU40.PMI:Performance_monitoring_interrupts
    120.83 ± 19%    -100.0%       0.00        interrupts.CPU41.NMI:Non-maskable_interrupts
    120.83 ± 19%    -100.0%       0.00        interrupts.CPU41.PMI:Performance_monitoring_interrupts
    122.17 ± 20%    -100.0%       0.00        interrupts.CPU42.NMI:Non-maskable_interrupts
    122.17 ± 20%    -100.0%       0.00        interrupts.CPU42.PMI:Performance_monitoring_interrupts
    119.67 ± 19%    -100.0%       0.00        interrupts.CPU43.NMI:Non-maskable_interrupts
    119.67 ± 19%    -100.0%       0.00        interrupts.CPU43.PMI:Performance_monitoring_interrupts
    121.00 ± 19%    -100.0%       0.00        interrupts.CPU44.NMI:Non-maskable_interrupts
    121.00 ± 19%    -100.0%       0.00        interrupts.CPU44.PMI:Performance_monitoring_interrupts
    121.17 ± 19%    -100.0%       0.00        interrupts.CPU45.NMI:Non-maskable_interrupts
    121.17 ± 19%    -100.0%       0.00        interrupts.CPU45.PMI:Performance_monitoring_interrupts
    127.50 ± 25%    -100.0%       0.00        interrupts.CPU46.NMI:Non-maskable_interrupts
    127.50 ± 25%    -100.0%       0.00        interrupts.CPU46.PMI:Performance_monitoring_interrupts
    122.50 ± 20%    -100.0%       0.00        interrupts.CPU47.NMI:Non-maskable_interrupts
    122.50 ± 20%    -100.0%       0.00        interrupts.CPU47.PMI:Performance_monitoring_interrupts
    175.50 ± 30%    -100.0%       0.00        interrupts.CPU48.NMI:Non-maskable_interrupts
    175.50 ± 30%    -100.0%       0.00        interrupts.CPU48.PMI:Performance_monitoring_interrupts
    120.50 ± 17%    -100.0%       0.00        interrupts.CPU49.NMI:Non-maskable_interrupts
    120.50 ± 17%    -100.0%       0.00        interrupts.CPU49.PMI:Performance_monitoring_interrupts
      4208 ± 49%    -100.0%       0.00        interrupts.CPU5.NMI:Non-maskable_interrupts
      4208 ± 49%    -100.0%       0.00        interrupts.CPU5.PMI:Performance_monitoring_interrupts
    123.67 ± 16%    -100.0%       0.00        interrupts.CPU50.NMI:Non-maskable_interrupts
    123.67 ± 16%    -100.0%       0.00        interrupts.CPU50.PMI:Performance_monitoring_interrupts
    121.67 ± 17%    -100.0%       0.00        interrupts.CPU51.NMI:Non-maskable_interrupts
    121.67 ± 17%    -100.0%       0.00        interrupts.CPU51.PMI:Performance_monitoring_interrupts
    124.67 ± 16%    -100.0%       0.00        interrupts.CPU52.NMI:Non-maskable_interrupts
    124.67 ± 16%    -100.0%       0.00        interrupts.CPU52.PMI:Performance_monitoring_interrupts
    126.00 ± 19%    -100.0%       0.00        interrupts.CPU53.NMI:Non-maskable_interrupts
    126.00 ± 19%    -100.0%       0.00        interrupts.CPU53.PMI:Performance_monitoring_interrupts
    127.50 ± 18%    -100.0%       0.00        interrupts.CPU54.NMI:Non-maskable_interrupts
    127.50 ± 18%    -100.0%       0.00        interrupts.CPU54.PMI:Performance_monitoring_interrupts
    131.17 ± 20%    -100.0%       0.00        interrupts.CPU55.NMI:Non-maskable_interrupts
    131.17 ± 20%    -100.0%       0.00        interrupts.CPU55.PMI:Performance_monitoring_interrupts
    124.00 ± 17%    -100.0%       0.00        interrupts.CPU56.NMI:Non-maskable_interrupts
    124.00 ± 17%    -100.0%       0.00        interrupts.CPU56.PMI:Performance_monitoring_interrupts
    126.00 ± 17%    -100.0%       0.00        interrupts.CPU57.NMI:Non-maskable_interrupts
    126.00 ± 17%    -100.0%       0.00        interrupts.CPU57.PMI:Performance_monitoring_interrupts
    126.17 ± 24%    -100.0%       0.00        interrupts.CPU58.NMI:Non-maskable_interrupts
    126.17 ± 24%    -100.0%       0.00        interrupts.CPU58.PMI:Performance_monitoring_interrupts
    132.67 ± 27%    -100.0%       0.00        interrupts.CPU59.NMI:Non-maskable_interrupts
    132.67 ± 27%    -100.0%       0.00        interrupts.CPU59.PMI:Performance_monitoring_interrupts
      5412 ± 43%    -100.0%       0.00        interrupts.CPU6.NMI:Non-maskable_interrupts
      5412 ± 43%    -100.0%       0.00        interrupts.CPU6.PMI:Performance_monitoring_interrupts
    123.83 ± 20%    -100.0%       0.00        interrupts.CPU60.NMI:Non-maskable_interrupts
    123.83 ± 20%    -100.0%       0.00        interrupts.CPU60.PMI:Performance_monitoring_interrupts
    124.33 ± 17%    -100.0%       0.00        interrupts.CPU61.NMI:Non-maskable_interrupts
    124.33 ± 17%    -100.0%       0.00        interrupts.CPU61.PMI:Performance_monitoring_interrupts
    134.17 ± 25%    -100.0%       0.00        interrupts.CPU62.NMI:Non-maskable_interrupts
    134.17 ± 25%    -100.0%       0.00        interrupts.CPU62.PMI:Performance_monitoring_interrupts
    122.17 ± 19%    -100.0%       0.00        interrupts.CPU63.NMI:Non-maskable_interrupts
    122.17 ± 19%    -100.0%       0.00        interrupts.CPU63.PMI:Performance_monitoring_interrupts
    124.00 ± 19%    -100.0%       0.00        interrupts.CPU64.NMI:Non-maskable_interrupts
    124.00 ± 19%    -100.0%       0.00        interrupts.CPU64.PMI:Performance_monitoring_interrupts
    126.00 ± 23%    -100.0%       0.00        interrupts.CPU65.NMI:Non-maskable_interrupts
    126.00 ± 23%    -100.0%       0.00        interrupts.CPU65.PMI:Performance_monitoring_interrupts
    124.00 ± 20%    -100.0%       0.00        interrupts.CPU66.NMI:Non-maskable_interrupts
    124.00 ± 20%    -100.0%       0.00        interrupts.CPU66.PMI:Performance_monitoring_interrupts
    127.83 ± 21%    -100.0%       0.00        interrupts.CPU67.NMI:Non-maskable_interrupts
    127.83 ± 21%    -100.0%       0.00        interrupts.CPU67.PMI:Performance_monitoring_interrupts
    123.33 ± 20%    -100.0%       0.00        interrupts.CPU68.NMI:Non-maskable_interrupts
    123.33 ± 20%    -100.0%       0.00        interrupts.CPU68.PMI:Performance_monitoring_interrupts
    122.33 ± 19%    -100.0%       0.00        interrupts.CPU69.NMI:Non-maskable_interrupts
    122.33 ± 19%    -100.0%       0.00        interrupts.CPU69.PMI:Performance_monitoring_interrupts
      7639 ± 12%    -100.0%       0.00        interrupts.CPU7.NMI:Non-maskable_interrupts
      7639 ± 12%    -100.0%       0.00        interrupts.CPU7.PMI:Performance_monitoring_interrupts
    124.67 ± 18%    -100.0%       0.00        interrupts.CPU70.NMI:Non-maskable_interrupts
    124.67 ± 18%    -100.0%       0.00        interrupts.CPU70.PMI:Performance_monitoring_interrupts
    141.83 ± 20%    -100.0%       0.00        interrupts.CPU71.NMI:Non-maskable_interrupts
    141.83 ± 20%    -100.0%       0.00        interrupts.CPU71.PMI:Performance_monitoring_interrupts
    131.50 ± 18%    -100.0%       0.00        interrupts.CPU72.NMI:Non-maskable_interrupts
    131.50 ± 18%    -100.0%       0.00        interrupts.CPU72.PMI:Performance_monitoring_interrupts
    134.17 ± 19%    -100.0%       0.00        interrupts.CPU73.NMI:Non-maskable_interrupts
    134.17 ± 19%    -100.0%       0.00        interrupts.CPU73.PMI:Performance_monitoring_interrupts
    126.83 ± 19%    -100.0%       0.00        interrupts.CPU74.NMI:Non-maskable_interrupts
    126.83 ± 19%    -100.0%       0.00        interrupts.CPU74.PMI:Performance_monitoring_interrupts
    126.00 ± 19%    -100.0%       0.00        interrupts.CPU75.NMI:Non-maskable_interrupts
    126.00 ± 19%    -100.0%       0.00        interrupts.CPU75.PMI:Performance_monitoring_interrupts
    123.83 ± 17%    -100.0%       0.00        interrupts.CPU76.NMI:Non-maskable_interrupts
    123.83 ± 17%    -100.0%       0.00        interrupts.CPU76.PMI:Performance_monitoring_interrupts
    125.17 ± 16%    -100.0%       0.00        interrupts.CPU77.NMI:Non-maskable_interrupts
    125.17 ± 16%    -100.0%       0.00        interrupts.CPU77.PMI:Performance_monitoring_interrupts
    125.50 ± 18%    -100.0%       0.00        interrupts.CPU78.NMI:Non-maskable_interrupts
    125.50 ± 18%    -100.0%       0.00        interrupts.CPU78.PMI:Performance_monitoring_interrupts
    162.33 ± 62%    -100.0%       0.00        interrupts.CPU79.NMI:Non-maskable_interrupts
    162.33 ± 62%    -100.0%       0.00        interrupts.CPU79.PMI:Performance_monitoring_interrupts
      6455 ± 37%    -100.0%       0.00        interrupts.CPU8.NMI:Non-maskable_interrupts
      6455 ± 37%    -100.0%       0.00        interrupts.CPU8.PMI:Performance_monitoring_interrupts
    103.67 ± 38%    -100.0%       0.00        interrupts.CPU80.NMI:Non-maskable_interrupts
    103.67 ± 38%    -100.0%       0.00        interrupts.CPU80.PMI:Performance_monitoring_interrupts
    143.83 ± 40%    -100.0%       0.00        interrupts.CPU81.NMI:Non-maskable_interrupts
    143.83 ± 40%    -100.0%       0.00        interrupts.CPU81.PMI:Performance_monitoring_interrupts
    122.17 ± 17%    -100.0%       0.00        interrupts.CPU82.NMI:Non-maskable_interrupts
    122.17 ± 17%    -100.0%       0.00        interrupts.CPU82.PMI:Performance_monitoring_interrupts
    125.50 ± 17%    -100.0%       0.00        interrupts.CPU83.NMI:Non-maskable_interrupts
    125.50 ± 17%    -100.0%       0.00        interrupts.CPU83.PMI:Performance_monitoring_interrupts
    125.33 ± 17%    -100.0%       0.00        interrupts.CPU84.NMI:Non-maskable_interrupts
    125.33 ± 17%    -100.0%       0.00        interrupts.CPU84.PMI:Performance_monitoring_interrupts
    124.17 ± 17%    -100.0%       0.00        interrupts.CPU85.NMI:Non-maskable_interrupts
    124.17 ± 17%    -100.0%       0.00        interrupts.CPU85.PMI:Performance_monitoring_interrupts
    127.17 ± 17%    -100.0%       0.00        interrupts.CPU86.NMI:Non-maskable_interrupts
    127.17 ± 17%    -100.0%       0.00        interrupts.CPU86.PMI:Performance_monitoring_interrupts
    124.17 ± 17%    -100.0%       0.00        interrupts.CPU87.NMI:Non-maskable_interrupts
    124.17 ± 17%    -100.0%       0.00        interrupts.CPU87.PMI:Performance_monitoring_interrupts
    123.00 ± 17%    -100.0%       0.00        interrupts.CPU88.NMI:Non-maskable_interrupts
    123.00 ± 17%    -100.0%       0.00        interrupts.CPU88.PMI:Performance_monitoring_interrupts
    125.33 ± 17%    -100.0%       0.00        interrupts.CPU89.NMI:Non-maskable_interrupts
    125.33 ± 17%    -100.0%       0.00        interrupts.CPU89.PMI:Performance_monitoring_interrupts
      7938 ± 13%    -100.0%       0.00        interrupts.CPU9.NMI:Non-maskable_interrupts
      7938 ± 13%    -100.0%       0.00        interrupts.CPU9.PMI:Performance_monitoring_interrupts
    124.50 ± 34%    -100.0%       0.00        interrupts.CPU90.NMI:Non-maskable_interrupts
    124.50 ± 34%    -100.0%       0.00        interrupts.CPU90.PMI:Performance_monitoring_interrupts
    125.67 ± 17%    -100.0%       0.00        interrupts.CPU91.NMI:Non-maskable_interrupts
    125.67 ± 17%    -100.0%       0.00        interrupts.CPU91.PMI:Performance_monitoring_interrupts
    122.83 ± 17%    -100.0%       0.00        interrupts.CPU92.NMI:Non-maskable_interrupts
    122.83 ± 17%    -100.0%       0.00        interrupts.CPU92.PMI:Performance_monitoring_interrupts
    121.83 ± 17%    -100.0%       0.00        interrupts.CPU93.NMI:Non-maskable_interrupts
    121.83 ± 17%    -100.0%       0.00        interrupts.CPU93.PMI:Performance_monitoring_interrupts
    112.00 ± 29%    -100.0%       0.00        interrupts.CPU94.NMI:Non-maskable_interrupts
    112.00 ± 29%    -100.0%       0.00        interrupts.CPU94.PMI:Performance_monitoring_interrupts
    293.33 ± 29%    -100.0%       0.00        interrupts.CPU95.NMI:Non-maskable_interrupts
    293.33 ± 29%    -100.0%       0.00        interrupts.CPU95.PMI:Performance_monitoring_interrupts
      7124 ± 23%    -100.0%       0.00        interrupts.CPU96.NMI:Non-maskable_interrupts
      7124 ± 23%    -100.0%       0.00        interrupts.CPU96.PMI:Performance_monitoring_interrupts
      5931 ± 44%    -100.0%       0.00        interrupts.CPU97.NMI:Non-maskable_interrupts
      5931 ± 44%    -100.0%       0.00        interrupts.CPU97.PMI:Performance_monitoring_interrupts
      4942 ± 44%    -100.0%       0.00        interrupts.CPU98.NMI:Non-maskable_interrupts
      4942 ± 44%    -100.0%       0.00        interrupts.CPU98.PMI:Performance_monitoring_interrupts
      5446 ± 42%    -100.0%       0.00        interrupts.CPU99.NMI:Non-maskable_interrupts
      5446 ± 42%    -100.0%       0.00        interrupts.CPU99.PMI:Performance_monitoring_interrupts
    206627 ±  9%    -100.0%       4.00 ± 70%  interrupts.NMI:Non-maskable_interrupts
    206627 ±  9%    -100.0%       4.00 ± 70%  interrupts.PMI:Performance_monitoring_interrupts


                                                                                
                              will-it-scale.16.processes                        

4.8e+07 +-----------------------------------------------------------------+

          |+.++++.++                   :                     +   .+++.++++.+|   
  4.7e+07 |-+       :                  :                      +++           |   
  4.6e+07 |-+       +.+++.++++.+++.++++                                     |   
          |                                                                 |   
  4.5e+07 |-+                                                               |   
          |                                                                 |   
  4.4e+07 |-+                                                               |   
          |                                                                 |   
  4.3e+07 |-+                                                               |   
  4.2e+07 |-+   O       O O    OOO    O                                     |   
          |  OOO  OOO       O       O                                       |   
  4.1e+07 |O+         OO   O O     O O                                      |   
          |                                                                 |   
    4e+07 +-----------------------------------------------------------------+   

will-it-scale.per_process_ops

3e+06 +----------------------------------------------------------------+
| .+ ++.++ : : + +. |

  2.95e+06 |++ +     :                 :                      ++.+ +  ++++.+|   
   2.9e+06 |-+       ++.+++.++++.++++.++                                    |   
           |                                                                |   
  2.85e+06 |-+                                                              |   
   2.8e+06 |-+                                                              |   
           |                                                                |   
  2.75e+06 |-+                                                              |   
   2.7e+06 |-+                                                              |   
           |                                                                |   
  2.65e+06 |-+   O       O       O                                          |   
   2.6e+06 |-+  O   O     O  O O  O O  O                                    |   
           |O OO   O OO     O O    O  O                                     |   

2.55e+06 +----------------------------------------------------------------+

will-it-scale.workload

4.8e+07 +-----------------------------------------------------------------+

          |+.++++.++                   :                     +   .+++.++++.+|   
  4.7e+07 |-+       :                  :                      +++           |   
  4.6e+07 |-+       +.+++.++++.+++.++++                                     |   
          |                                                                 |   
  4.5e+07 |-+                                                               |   
          |                                                                 |   
  4.4e+07 |-+                                                               |   
          |                                                                 |   
  4.3e+07 |-+                                                               |   
  4.2e+07 |-+   O       O O    OOO    O                                     |   
          |  OOO  OOO       O       O                                       |   
  4.1e+07 |O+         OO   O O     O O                                      |   
          |                                                                 |   
    4e+07 +-----------------------------------------------------------------+   

perf-sched.total_wait_time.max.ms

10000 +-------------------------------------------------------------------+

        |                       :           :        :                      |   
   9500 |-+ +               +   ::  +    +  :      + :           +          |   
        |   :     .+       + :  ::  ::.+ :  :   + :: ::         + :         |   
        |   :    + :    +  : +  :+  :+  : : :   :::: ::         : :    +    |   
   9000 |-+: :   :  :  +: :  :  : : :   + :::  + +  : : +       :  :+ ::  .+|   
        |: : :  :   + +  ::   ::  ::      :: : :    : : :+.+ +  :  + :: :+  |   
   8500 |:.+ :  +    +   :    ::   :       : ::     : : :   +:  :    +  ::  |   
        |+   +.+         +    ::   +       + ::     + : :     ::        +   |   
   8000 |-+                   ::             ::        :      ::            |   
        |                      :             :         :      ::            |   
        |                      +             +         +       :            |   
   7500 |-+                                                    +            |   
        |                                                                   |   
   7000 +-------------------------------------------------------------------+   

perf-sched.total_wait_time.average.ms

235 +---------------------------------------------------------------------+

      |                                     +.        +                     |   
  230 |-+                        +          : +       :+                    |   
  225 |-+    +       +           :: +       : :       : :            +.     |   
      |+     :+      :+      +   :: :+     :   :     :  :         +.+  +    |   
  220 |-+    ::  +  +  + +.  :: :  :  +  .+:   +   +.:   :+       :     +  +|   
      |  ++ : :  : :    +  +: : +  +  : +  +    : +  +   + :     :      :  :|   
  215 |-+ : :  : :::        +  +       ::       : :        :    ++      :  :|   
      |   : :  : :+                    +         +          :  +        : : |   
  210 |-+  ::  : :                                          + +          :: |   
  205 |-+  :   ::                                            +           :: |   
      |    :    :                                                        :: |   
  200 |-+  +    :                                                        :  |   
      |         +                                                        +  |   
  195 +---------------------------------------------------------------------+   

perf-sched.total_sch_delay.max.ms

1100 +--------------------------------------------------------------------+

  1000 |-+                                                                 +|   
       |                                    +        +                    +:|   
   900 |-+                                  :        :                   + :|   
   800 |-+                                  :        :                   : :|   
   700 |-+                                  :        :                   : :|   
   600 |-+                                 ::        :                   : :|   
       |                                   ::        :                   : :|   
   500 |-+                                 : :       ::                  :  |   
   400 |-+                                 : :      : :                 :   |   
   300 |-+                                 : :      : :                 :   |   
   200 |-+                                 : :      : :                 :   |   
       |                                  :  :      : :                 :   |   
   100 |-+                                :  :      : :                 :   |   
     0 +--------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                       perf-sched.total_sch_delay.average.ms                    

0.1 +--------------------------------------------------------------------+

  0.09 |-+                                  +        +                     :|   
       |                                    :        :                    ::|   
  0.08 |-+                                  :        :                    ::|   
  0.07 |-+                                 ::        :                   + :|   
       |                                   ::        :                   : :|   
  0.06 |-+                                 ::        :                   : :|   
  0.05 |-+                                 ::       ::                   :  |   
  0.04 |-+                                 : :      : :                  :  |   
       |                                   : :      : :                 :   |   
  0.03 |-+                                :  :      : :                 :   |   
  0.02 |-+                                :  :      : :                 :   |   
       |                                  +  :     .+ :             +   :   |   
  0.01 |+.+ +.+++.++. ++.++ .+++.++ .++ .+   :+.+ +   :.+ +.+++.++.+ :.++   |   
     0 +--------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                      perf-sched.total_wait_and_delay.count.ms                  

12600 +-------------------------------------------------------------------+

  12400 |-+      +                                                          |   
        |        :                                                          |   
  12200 |-+  +   :                                                       +  |   
  12000 |-+  :   ::          +                                           :  |   
  11800 |-+  :   ::      +   :                                           :: |   
  11600 |-+  ::  ::      ::  ::                              +          : : |   
        |    ::  ::      ::  :: ++                           ::         + : |   
  11400 |-+ : : : :      : +: : : +  +                      : :         : : |   
  11200 |-.+: : : : +    : ::  :   + :: ++.    + +.+    +   +  + +  +  :   :|   
  11000 |++:: : :  ::  +:   :  +   : :: :  +  + ::  +   :+. :   + : :+.+   +|   
  10800 |-+ :  ::  : ::::   +       :  +    :+  +    :.+   +      ::        |   
        |   +  ::  + :: :           +       +        +             +        |   
  10600 |-+    ++    +  +                                                   |   

10400 +-------------------------------------------------------------------+

perf-sched.total_wait_and_delay.max.ms

10000 +-------------------------------------------------------------------+

        |                       :           :        :                      |   
   9500 |-+ +               +   ::  +    +  :      + :           +          |   
        |   :     .+       + :  ::  ::.+ :  :   + :: ::         + :         |   
        |   :    + :    +  : +  :+  :+  : : :   :::: ::         : :    +    |   
   9000 |-+: :   :  :  +: :  :  : : :   + :::  + +  : : +       :  :+ ::  .+|   
        |: : :  :   + +  ::   ::  ::      :: : :    : : :+.+ +  :  + :: :+  |   
   8500 |:.+ :  +    +   :    ::   :       : ::     : : :   +:  :    +  ::  |   
        |+   +.+         +    ::   +       + ::     + : :     ::        +   |   
   8000 |-+                   ::             ::        :      ::            |   
        |                      :             :         :      ::            |   
        |                      +             +         +       :            |   
   7500 |-+                                                    +            |   
        |                                                                   |   
   7000 +-------------------------------------------------------------------+   

perf-sched.total_wait_and_delay.average.ms

235 +---------------------------------------------------------------------+

      |                                     +.        +                     |   
  230 |-+            +           +          : +       :+                    |   
  225 |-+    +       ::          :: +       : :       : :            +.     |   
      |+     :+     : +      +   :: :+     :   :     :  :         +.+  +    |   
  220 |-+    ::  +  +  + +.  :: :  :  +  .++   +   +.+   :+       :     +  +|   
      |  ++ : :  : :    +  +: : +  +  : +       : +      + :     :      :  :|   
  215 |-+ : :  : :::        +  +       ::       : :        :    ++      :  :|   
      |   : :  : :+                    +         +          :  +        : : |   
  210 |-+  ::  : :                                          + +          :: |   
  205 |-+  :   ::                                            +           :: |   
      |    :    :                                                        :: |   
  200 |-+  +    :                                                        :  |   
      |         +                                                        +  |   
  195 +---------------------------------------------------------------------+   

perf-sched.wait_time.avg.ms.pipe_read.new_sync_read.vfs_read.ksys_read

58 +----------------------------------------------------------------------+

     |                       +                                              |   
  57 |-+                     :                                      +       |   
  56 |-++                    :                                      :  .+  +|   
     |  :   +                :           +         +               :: + :  :|   
  55 |-::   :  +            : :          ::        :               :: : : ::|   
     | : :  ::::            : :          ::        ::              : ::  :: |   
  54 |-: :  ::::            : :          : :       ::              : ::  :: |   
     | : : : +  :  +        : +     +   :  +      : +.            :  :   :  |   
  53 |:+ + :    :  :+.+  .+ : :  + + :  :   ++.   :   ++.+++.+ +.++  :   +  |   
  52 |++  +:    +.+    ++ : :  :: +  ++.+      ++.+           ::     +      |   
     |     +               :   ::                             +             |   
  51 |-+                   :   +                                            |   
     |                     +                                                |   
  50 +----------------------------------------------------------------------+   

perf-sched.sch_delay.max.ms.pipe_read.new_sync_read.vfs_read.ksys_read

1.6 +---------------------------------------------------------------------+

      |               +                                                :    |   
  1.4 |-+             :                                                :    |   
  1.2 |-+             :                                                :    |   
      |               :                                               ::    |   
    1 |-+             ::                                              ::    |   
      |               ::                                              ::    |   
  0.8 |-+             ::                                              : :   |   
      |              : :                                              : :   |   
  0.6 |-+            : :                                              : :   |   
  0.4 |-+            : :                                              : :   |   
      |              :  :                                            :  :   |   
  0.2 |-+            :  :                                            :  :   |   
      | .+  .+ .+    :  :    +.          .+        +.              .+:  + .+|   
    0 +---------------------------------------------------------------------+   

perf-sched.sch_delay.avg.ms.pipe_read.new_sync_read.vfs_read.ksys_read

0.005 +------------------------------------------------------------------+

         |                     :                                            |   
  0.0045 |-+                   :                                            |   
   0.004 |-+    + +            :           +        +               +  +    |   
         |      : :            ::          :        :               :  :    |   
  0.0035 |-+    : :            ::          :        :               :  :    |   
         |     :: ::           ::          :        ::              : ::    |   
   0.003 |-++  :: ::           ::          :        ::              : : :  +|   
         |  :  :: ::          : :          :        ::              : : :  :|   
  0.0025 |-::  : : :          : :         : :      : :             : :: : ::|   
   0.002 |-::  : : :     +    : :         : :      : :             : :: + ::|   
         | : : : : :     :    :  :        : :      : :             : :: : : |   
  0.0015 |-: ::  :  :   : :   :  :        : :      :  :            : :   :: |   
         |:  ::  :  :   : :   :  :        : :      :  :            : :   :  |   
   0.001 +------------------------------------------------------------------+   

perf-sched.wait_and_delay.count.pipe_read.new_sync_read.vfs_read.ksys_read

3150 +--------------------------------------------------------------------+

       |+                    :                                              |   
  3100 |:+               +  ::   +                                          |   
  3050 |:+  +           + : ::  ::     +.+   ++. ++   +.+     +.            |   
       | : +:     +    +  +: :  : :+.++  :   :  + :   :  :   +  + .+ +   +  |   
  3000 |-: : :    :+.++    + : +  ::     :  +      : +   ++.+    + : :   :  |   
       | : : :   :           : :  +      :  :      : :             : ::  :: |   
  2950 |-: : : + :            ::          ::       : :             : :: : : |   
       | ::  : ::             ::          ::       ::               ::: : ::|   
  2900 |-+:   : +             ::          :         :               :  :+  :|   
  2850 |-+:   +               :           +         +               :  +   +|   
       |  +                   :                                     :       |   
  2800 |-+                    :                                     +       |   
       |                      +                                             |   
  2750 +--------------------------------------------------------------------+   

perf-sched.wait_and_delay.avg.ms.pipe_read.new_sync_read.vfs_read.ksys_read

58 +----------------------------------------------------------------------+

     |                       +                                              |   
  57 |-+                     :                                      +       |   
  56 |-++                    :                                      :  .+  +|   
     |  :   +                :           +         +               :: + :  :|   
  55 |-::   :  +            : :          ::        :               :: : : ::|   
     | : :  ::::            : :          ::        ::              : ::  :: |   
  54 |-: :  ::::            : :          : :       ::              : ::  :: |   
     | : : : +  :  +        : +     +   :  +      : +.            :  :   :  |   
  53 |:+ + :    :  :+.+  .+ : :  + + :  :   ++.   :   ++.+++.+ +.++  :   +  |   
  52 |++  +:    +.+    ++ : :  :: +  ++.+      ++.+           ::     +      |   
     |     +               :   ::                             +             |   
  51 |-+                   :   +                                            |   
     |                     +                                                |   
  50 +----------------------------------------------------------------------+   

perf-sched.wait_time.max.ms.smpboot_thread_fn.kthread.ret_from_fork

5500 +--------------------------------------------------------------------+

       |                                                      +             |   
  5000 |-+                                                    :             |   
  4500 |-+                                                    :             |   
       |                                                      :             |   
  4000 |-+                                                    ::            |   
  3500 |-+                                                    ::            |   
       |                                                      ::            |   
  3000 |-+                                                   : :            |   
  2500 |-+                                                   : :            |   
       |                                                    +: :         +  |   
  2000 |-+                                                  ::  :        :: |   
  1500 |-+                                                 : :  :        :: |   
       |                                                   : :  :       :  :|   
  1000 +--------------------------------------------------------------------+   

perf-sched.wait_time.avg.ms.smpboot_thread_fn.kthread.ret_from_fork

720 +---------------------------------------------------------------------+

      |                          +          +.        +                     |   
  710 |-+   .+          +   +   +:  +       : +       :+                    |   
      |+   + :         : :  :   : : ::      : :       : +                +. |   
  700 |:+  :  :  ++.   : +.::  +  : :: +   :   :.+   :   +    +. ++      : +|   
      | : +   +  :  +++    +:  :   :  +:   +   + :   +    +.+ : + :  +   :  |   
  690 |-: :   :  :           ::    +   :   :     :   :      : :    : :  :   |   
      | : :    : :           ::        :   :     :  :       : :    : :: +   |   
  680 |-::     ::            :          : :       : :       : :    :: : :   |   
      |  :     ::            +          : :       ::         :      : : :   |   
  670 |-++      :                       : +       :+         :      +  :    |   
      |         +                       :+        ::         :         +    |   
  660 |-+                               +         +          +              |   
      |                                                                     |   
  650 +---------------------------------------------------------------------+   

perf-sched.sch_delay.avg.ms.smpboot_thread_fn.kthread.ret_from_fork

0.018 +-------------------------------------------------------------------+

        |                      +           +        +                       |   
  0.016 |-+                    :           :        :                       |   
        |                      :           :        :                       |   
  0.014 |-+                   ::           :        :               +       |   
        |                     : :         ::        :               :       |   
  0.012 |-+                   : :         ::        :               :       |   
        |  +     +            : :         : :       ::             ::       |   
   0.01 |-+:     :           :  :         : :      : :             ::  +    |   
        | ::     :: +      + +  ++.+      : :      : :      ++   +.+ : ::   |   
  0.008 |-: :   : : :     :: :     :      : :      : :      : :  :   :: +   |   
        | : :  +:  : :    : ::      :    :  :      : :      : : :    :: :  +|   
  0.006 |:+ + + +  + +.+ +  :       ++. +:  ++.  +.: +.+ +.:   ++    :   :+ |   
        |+   +          +   +          + +     ++  +    +  +         +   +  |   
  0.004 +-------------------------------------------------------------------+   

perf-sched.wait_and_delay.count.smpboot_thread_fn.kthread.ret_from_fork

2700 +--------------------------------------------------------------------+

  2680 |-+:     +     +  ++  ++  ++  +   ++  ++   +.+ +.+    +      ++.++   |   
       | ::     :     :  ::  ::  ::  :   ::  ::   : : : :    :      :   :   |   
  2660 |-::     :     :  ::  :: : :  :   ::  ::   : : : :    :      :   :   |   
  2640 |-::     ::    :  ::  : :: : ::   : : : :  : : : :    :      :   :   |   
  2620 |-::     ::    : : : :  :: : ::  :  : : :  : : : :    :      :   :   |   
  2600 |:+:     ::    : : : :  :  : ::  :  : : :  : : : :    :      :   :   |   
       |+  :+   ::    :::  ::  +   :: : :  : : :  :  ::  :   ::     :    :  |   
  2580 |:+ ::  : :   : ::  ::      :: : :  ::  : :   :   :  : :    :     :  |   
  2560 |:+ ::: : :   : ::  ::      :: : :  ::  : :   :   :  : :    :     :  |   
  2540 |:+ ::: :  :  : ::  ::      :  : :   :   ::   :   :  : :    :     :  |   
  2520 |-+ : : :  :  : :   :       :  ::    :   ::   :   :  : :    :     :  |   
       |   :  ::  :  : :   :       +  ::    :   ::   :   :  : :    :     :  |   
  2500 |-+ +  ++  ++.+ +   +          ++    +   ++   +   ++.+ +.++.+     +.+|   

2480 +--------------------------------------------------------------------+

perf-sched.wait_and_delay.max.ms.smpboot_thread_fn.kthread.ret_from_fork

5500 +--------------------------------------------------------------------+

       |                                                      +             |   
  5000 |-+                                                    :             |   
  4500 |-+                                                    :             |   
       |                                                      :             |   
  4000 |-+                                                    ::            |   
  3500 |-+                                                    ::            |   
       |                                                      ::            |   
  3000 |-+                                                   : :            |   
  2500 |-+                                                   : :            |   
       |                                                    +: :         +  |   
  2000 |-+                                                  ::  :        :: |   
  1500 |-+                                                 : :  :        :: |   
       |                                                   : :  :       :  :|   
  1000 +--------------------------------------------------------------------+   

perf-sched.wait_and_delay.avg.ms.smpboot_thread_fn.kthread.ret_from_fork

720 +---------------------------------------------------------------------+

      |                          +          +.        +                     |   
  710 |-+   .+          +   +   +:  +       : +       :+                    |   
      |+   + :         : :  :   : : ::      : :       : +                +. |   
  700 |:+  :  :  ++.   : +.::  +  : :: +   :   :.+   :   +    +. ++      : +|   
      | : +   +  :  +++    +:  :   :  +:   +   + :   +    +.+ : + :  +   :  |   
  690 |-: :   :  :           ::    +   :   :     :   :      : :    : :  :   |   
      | : :    : :           ::        :   :     :  :       : :    : :: +   |   
  680 |-::     ::            :          : :       : :       : :    :: : :   |   
      |  :     ::            +          : :       ::         :      : : :   |   
  670 |-++      :                       : +       :+         :      +  :    |   
      |         +                       :+        ::         :         +    |   
  660 |-+                               +         +          +              |   
      |                                                                     |   
  650 +---------------------------------------------------------------------+   

10 +----------------------------------------------------------------------+

     |                                                                      |   
   9 |-+                                    +         +                     |   
     |       +     ++.                      :         :                     |   
   8 |++     :  +  :  +           +         ::       : :          +  +      |   
     | : +  +:  :: :  :      +   + + +  +  : +.+  +  : +          :: :+     |   
   7 |-: :  : : :::   :  .+  :  :   +: : +.+   : : ++   + +       :::  :    |   
     |  + : : : : :    :+ : : : :     ::        ::       +:    + :  +  :    |   
   6 |-+  : : : : +    ::  :: :+      +         +          :.++ +:      :  +|   
     |    ::  ::       +   :  +                            +     +      +  :|   
   5 |-+  ::   :           +                                            : : |   
     |     :   :                                                         :: |   
   4 |-+   +   +                                                         +  |   
     |                                                                      |   
   3 +----------------------------------------------------------------------+   

14 +----------------------------------------------------------------------+

     |                                                                      |   
  12 |-+        +                       +         +                         |   
     |          :                       :         :                         |   
  10 |-+        ::                      ::        ::                        |   
     |          ::                      :+        :+                        |   
   8 |-+        ::                     : :       : :                        |   
     |          ::                     : :       : :                        |   
   6 |-+        : :                    :  :      : :                        |   
     |         :  :    +               :  :      : :           +         +  |   
   4 |-+       :  +    :            +  :  :      :  :         +:         :  |   
     |         :  :    :            :  :  :      :  :         : :        :: |   
   2 |-+       :   :  : :          : ::    :    :   :         : :       : : |   
     |         :   :  : :          : ::    :    :   :        :   :      :  :|   
   0 +----------------------------------------------------------------------+   

2600 +--------------------------------------------------------------------+

       |        +                                                        :  |   
  2400 |-+  +   :                                                        :  |   
  2200 |-+  :   :                                                        :: |   
       |    :   ::                                                       :: |   
  2000 |-+  ::  ::                                                      : : |   
       |    ::  ::           +                                          : : |   
  1800 |-+ : :  ::       +   : +                            +           :  :|   
       |   : : : : +     :  :: :+                           :+   +      +  :|   
  1600 |-+ :  :: : :    : : : :: +     +         +         :  +.+ :     :  +|   
  1400 |-++:  ::  :::   : +:  :   :  + :+        ::      + :      : +   :   |   
       | + +  +:  : :  :   +  +   + + +  ++.+ +.+ +.++  + +        ::: :    |   
  1200 |++     :  +  :++           +        : :      : :           + +.+    |   
       |       +     +                       :        ::                    |   
  1000 +--------------------------------------------------------------------+   

10 +----------------------------------------------------------------------+

     |                                                                      |   
   9 |-+                                    +         +                     |   
     |       +     ++.                      :         :                     |   
   8 |++     :  +  :  +           +         ::       : :          +  ++     |   
     | : +  +:  :: :  :      +   + + +  +  : +.+  +  : +          :: : :    |   
   7 |-: :  : : :::   :  .+  :  :   +: : +.+   : : ++   + +       :::  :    |   
     |  + : : : : :    :+ : : : :     ::        ::       +:    + :  +   :   |   
   6 |-+  : : : : +    ::  :: :+      +         +          :.++ +:      +  +|   
     |    ::  ::       +   :  +                            +     +      :  :|   
   5 |-+  ::   :           +                                             :: |   
     |     :   :                                                         :: |   
   4 |-+   +   +                                                         +  |   
     |                                                                      |   
   3 +----------------------------------------------------------------------+   

perf-sched.wait_time.max.ms.worker_thread.kthread.ret_from_fork

10000 +-------------------------------------------------------------------+

        |                       :           :        :                      |   
   9500 |-+ +               +   :   +    +  :      + :           +          |   
        |   :     .+       + :  ::  ::.+ :  :   + :: ::         + :         |   
        |   :    + :    +  : +  ::  :+  : : :   :::: ::         : :    +    |   
   9000 |-+: :   :  :  +: :  :  ::  :   + :::  + +  : : +       :  :+ ::  .+|   
        |: : :  :   + +  ::   :: + :      :: : :    : : :+.+ +  :  + :: :+  |   
   8500 |:.+ :  +    +   :    ::  +:       : ::     : : :   +:  :    +  ::  |   
        |+   +.+         +    ::   +       + ::     + : :     ::        +   |   
   8000 |-+                   ::             ::        :      ::            |   
        |                      :             :         :      ::            |   
        |                      +             +         +       :            |   
   7500 |-+                                                    +            |   
        |                                                                   |   
   7000 +-------------------------------------------------------------------+   

perf-sched.wait_time.avg.ms.worker_thread.kthread.ret_from_fork

500 +---------------------------------------------------------------------+

      |                                                                 +   |   
  480 |-+                                                               :   |   
      |         +                                                       :   |   
  460 |-+       :                                                   +  ::   |   
      |  +      :                                                   :  + :  |   
  440 |-+:     : :        .+++.+        +         +                : ::  :  |   
      | ::     : :      ++     :      + ::     +  :                : ::  : +|   
  420 |:: :+.  : :   + +       :     +: ::    + : ::   +  +      + : +   :+ |   
      |:  :: + : +   :+         :+  +  :  ++ :  :: +.+ :+ ::     : :     +  |   
  400 |++ :   :   :  :          ::+ :  :    ::   :    :  + : +  + :         |   
      |   +   +   +.:           :  +   +    +    +    +     +:  : +         |   
  380 |-+           +           +                             ::            |   
      |                                                       ::            |   
  360 +---------------------------------------------------------------------+   

perf-sched.sch_delay.max.ms.worker_thread.kthread.ret_from_fork

1000 +--------------------------------------------------------------------+

   900 |-+                                  +        +                      |   
       |                                    :        :                   +  |   
   800 |-+                                  :        :                   :  |   
   700 |-+                                  :        :                   :  |   
       |                                   ::        :                   :: |   
   600 |-+                                 ::        :                   :: |   
   500 |-+                                 ::        :                   :: |   
   400 |-+                                 : :      : :                  :: |   
       |                                   : :      : :                 : : |   
   300 |-+                                 : :      : :                 : : |   
   200 |-+                                :  :      : :                 :  :|   
       |                                  :  :      : :                 :  :|   
   100 |-+                                :  :      : :                 :  :|   
     0 +--------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
          perf-sched.sch_delay.avg.ms.worker_thread.kthread.ret_from_fork       

1.6 +---------------------------------------------------------------------+

      |                                    :         :                      |   
  1.4 |-+                                  :         :                   +  |   
  1.2 |-+                                  :         :                   :  |   
      |                                    :        ::                   :  |   
    1 |-+                                  :        ::                   :: |   
      |                                    :        ::                   :: |   
  0.8 |-+                                  ::       : :                  :: |   
      |                                   : :       : :                  :: |   
  0.6 |-+                                 : :       : :                 : : |   
  0.4 |-+                                 : :       : :                 : : |   
      |                                   : :      :  :                 :  :|   
  0.2 |-+                                 : :      :  :                 :  :|   
      |                                   : :      :  :                 :  :|   
    0 +---------------------------------------------------------------------+   

perf-sched.wait_and_delay.count.worker_thread.kthread.ret_from_fork

700 +---------------------------------------------------------------------+

      |                                                                 :   |   
  680 |-+                                                              +:   |   
      |                     +                                          ::   |   
  660 |-+                   :                                         :  : +|   
      |         +           :           +         +                 + :  :::|   
  640 |-+    + : +         : :       .+ :+    ++. ::   +         + +: :  :: |   
      |+    + :: :   ++.   : : +    +  +  +  +   + +.  :+    +   :+  :   +  |   
  620 |-:  +  +   :  :  ++.+ :+ ++.+       ++        ++  ++.+ :.+    +      |   
      | :  :      +.:        +                                +             |   
  600 |-++ :        +                                                       |   
      |  ::                                                                 |   
  580 |-+ :                                                                 |   
      |   +                                                                 |   
  560 +---------------------------------------------------------------------+   

perf-sched.wait_and_delay.max.ms.worker_thread.kthread.ret_from_fork

10000 +-------------------------------------------------------------------+

        |                       :           :        :                      |   
   9500 |-+ +               +   :   +    +  :      + :           +          |   
        |   :     .+       + :  ::  ::.+ :  :   + :: ::         + :         |   
        |   :    + :    +  : +  ::  :+  : : :   :::: ::         : :    +    |   
   9000 |-+: :   :  :  +: :  :  ::  :   + :::  + +  : : +       :  :+ ::  .+|   
        |: : :  :   + +  ::   :: + :      :: : :    : : :+.+ +  :  + :: :+  |   
   8500 |:.+ :  +    +   :    ::  +:       : ::     : : :   +:  :    +  ::  |   
        |+   +.+         +    ::   +       + ::     + : :     ::        +   |   
   8000 |-+                   ::             ::        :      ::            |   
        |                      :             :         :      ::            |   
        |                      +             +         +       :            |   
   7500 |-+                                                    +            |   
        |                                                                   |   
   7000 +-------------------------------------------------------------------+   

perf-sched.wait_and_delay.avg.ms.worker_thread.kthread.ret_from_fork

500 +---------------------------------------------------------------------+

      |                                                                 +   |   
  480 |-+                                                               :   |   
      |         +                                                       :   |   
  460 |-+       :                                                   +  ::   |   
      |  +      :                                                   :  + :  |   
  440 |-+:     : :      + .+++.+        +         +                : ::  :  |   
      | ::     : :     : +     :      + ::     +  :                : ::  : +|   
  420 |:: :+.  : :   + :       :     +: ::    + : ::   +  +      + : +   :+ |   
      |:  :: + : +   :+         :+  +  :  ++ :  :: +.+ :+ ::     : :     +  |   
  400 |++ :   :   :  :          ::+ :  :    ::   :    :  + : +  + :         |   
      |   +   +   +.:           :  +   +    +    +    +     +:  : +         |   
  380 |-+           +           +                             ::            |   
      |                                                       ::            |   
  360 +---------------------------------------------------------------------+   

235 +---------------------------------------------------------------------+

      |                                                           :         |   
  230 |-+                        +     +     .+  +     +          :      +  |   
      |                         +:     ::  ++ :  ::  ++:    +     ::     :  |   
  225 |-+            +         + :     :+  :  :  :+  :  :   :     :: +   :: |   
      |       +      :        :   :    ::  :  : : :  :  :  ::   + :: :   :: |   
  220 |-+     :      ::       :   :    : : :   :: : :   :  : :  : :: ::  :: |   
      |       ::    : +.+  + +    :   :  : :   :: : :    :+  :  :: : :: :  :|   
  215 |-+ +   ::    :   :  : :    :   :  ::    ::  ::    +   + ::: : :: :  +|   
      |   :  : :  +.+   : :: :    :   :  ::    :   ::        : : :  : : :   |   
  210 |:+ :: : :  :      :: :     :   +   :    +   :         : : :  :  :+   |   
      |+.: : :  ::       :  :      : +    +        +          :: :  :  +    |   
  205 |-++ +.+  ++       +  +      :+                         :  +  +       |   
      |                            +                          +             |   
  200 +---------------------------------------------------------------------+   

1200 +--------------------------------------------------------------------+

       |                                                                    |   
  1000 |-+                                                                 +|   
       |                                                                   :|   
       |                                                                   :|   
   800 |-+                                                                 :|   
       |                                                                  ::|   
   600 |-+                                                                ::|   
       |                                                                  : |   
   400 |-+                                                                : |   
       |                                                                  : |   
       |                                                                  : |   
   200 |-+                                                               :  |   
       |                                                                 :  |   
     0 +--------------------------------------------------------------------+   

3.5 +---------------------------------------------------------------------+

      |                                                                    +|   
    3 |-+                                                                  :|   
      |                                                                    :|   
  2.5 |-+                                                                  :|   
      |                                                                   ::|   
    2 |-+                                                                 ::|   
      |                                                                   ::|   
  1.5 |-+                                                                 : |   
      |                                                                   : |   
    1 |-+                                                                 : |   
      |                                                                  :  |   
  0.5 |-+                                                                :  |   
      |                                                                  :  |   
    0 +---------------------------------------------------------------------+   

307 +---------------------------------------------------------------------+

      |:  ::        :   :      :       :    : :  :    ::    :   :       :   |   
  306 |:+ ::        :  ::      :       :    : :  :    ::    :   :       :   |   
      |:  ::        :  ::      :       :    : :  :    ::    :   :       :   |   
      |:: : :      :: : :     ::       :    : : ::    : :  ::  ::      ::   |   
  305 |:: : :      :: + :     ::       :    : : ::    : :  ::  ::      ::   |   
      |:: : :      :: : :     ::       :    : : ::    : :  ::  ::      ::   |   
  304 |:: : :    + : ::  :    : :+     ::   :  :: :   : :  : : : :     + :  |   
      | ::  :    : : ::  :    : ::    : :  :   :: :  :  :  : : : :     : :  |   
  303 |-::  :    : : ::  :    : :::   : :  :   :: :  :  :  : : : :+   :  :  |   
      | ::  :    : : :   :    : :::   : :  :   :: :  :  :  : : : ::   :  :  |   
      |  :   :  : :  :   :   :  : :   : :  :   :  :  :   ::  ::  :::  :  :  |   
  302 |-+:   :  : :  :   :   :  : :   : :  :   :  :  :   ::  ::  : :  :  :  |   
      |  :   :  : :  :   :   :  :  :  : :  :   :  :  :   ::  ::  :  ::   :  |   
  301 +---------------------------------------------------------------------+   

235 +---------------------------------------------------------------------+

      |                                                           :         |   
  230 |-+                        +     +     .+  +     +          :      +  |   
      |                         +:     ::  ++ :  ::  ++:    +     ::     :: |   
  225 |-+            +         + :     :+  :  :  :+  :  :   :     :: +   :: |   
      |       +      :        :   :    ::  :  : : :  :  :  ::   + :: :   : :|   
  220 |-+     :      ::       :   :    : : :   :: : :   :  : :  : :: ::  : +|   
      |       ::    : +.+  + +    :   :  : :   :: : :    :+  :  :: : :: :   |   
  215 |-+ +   ::    :   :  : :    :   :  ::    ::  ::    +   + ::: : :: :   |   
      |   :  : :  +.+   : :: :    :   :  ::    :   ::        : : :  : : :   |   
  210 |:+ :: : :  :      :: :     :   +   :    +   :         : : :  :  :+   |   
      |+.: : :  ::       :  :      : +    +        +          :: :  :  +    |   
  205 |-++ +.+  ++       +  +      :+                         :  +  +       |   
      |                            +                          +             |   
  200 +---------------------------------------------------------------------+   

perf-sched.wait_time.max.ms.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64

16.9 +--------------------------------------------------------------------+

  16.8 |-+                :+.+                                              |   
       |                  :  :       +                                      |   
  16.7 |:+                :  :       :                                   +  |   
  16.6 |++                :  :      ::                                   :  |   
       |:                 :  :   +  ::       +        +       +          :: |   
  16.5 |-:      +.       :   :  :: +  :      ::       :+  +  + :         :: |   
  16.4 |-:      : +      :    : :: :  :      :+       : + :: : :         :::|   
  16.3 |-:  +   : :      :    :+ : :  :      : :      : : :::   :       :  :|   
       | :  ::  :  :  +  :    ::  ::  :+    :  : +   :   :  +   +       :  +|   
  16.2 |-+:+ : :   +. :  :    ::  :   + + +.:   + :.+:   +      :       :   |   
  16.1 |-++   +:     + :.+    :   :      +  +     +  +           :  +   +   |   
       |       +       +      +   +                              +. :+  :   |   
    16 |-+                                                         +  +:    |   

15.9 +--------------------------------------------------------------------+

perf-sched.wait_time.avg.ms.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64

0.81 +--------------------------------------------------------------------+

   0.8 |-+                    +                                             |   
       |      +               :                                             |   
  0.79 |-+    :               :           +         +                       |   
  0.78 |-+    :               :           :         :               +       |   
  0.77 |-+   ::               ::          ::        :               :       |   
  0.76 |-++  :: +.           : :          ::       ::               :       |   
       |  :  : :: +          : :          ::       : :              :: +   +|   
  0.75 |-: : : :: :          : +         : :       : :             : :::  ::|   
  0.74 |-: : : ::  :         : :   +     :  :+     : :+.        ++.: :: : ::|   
  0.73 |++ +:  :   +   +.++  +  :  :+  + :  ::+. +:  :: +      +   + +  ++  |   
  0.72 |:+  :  :    + +    :+   : :  ++ +:  +   + :  +  : +. ++             |   
       |    +  +     +     +     :+      +        +      :: +               |   
  0.71 |-+                       +                       :                  |   

0.7 +--------------------------------------------------------------------+

perf-sched.sch_delay.max.ms.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64

0.16 +--------------------------------------------------------------------+

       |      +                           +         +                       |   
  0.14 |-+    : +                         :         :                       |   
  0.12 |-+    : :             +           :         :                  +   +|   
       |  +   : :             :           ::        :                  ::  :|   
   0.1 |-+:  :: ::            :           ::       ::               +  :+  :|   
       |  :  :: ::            :           ::       ::               : : : ::|   
  0.08 |-::  : :::            :           ::       : :              : : : ::|   
       | ::  : : :            ::         : :       : :              : : : : |   
  0.06 |-: : : : :           : :         : :       : :              :::  :: |   
  0.04 |-: : : : :           : :         :  :      : :             : ::  :  |   
       |:  ::  :  :          : :         :  :     :  :             : :   :  |   
  0.02 |:+ ::  +  :          : :         :  :     :  :             : :   +  |   
       |+  ++     ++.+++.+++.+ +.+++.+++.+  +++.+++  ++.+++.+++.++.+ +      |   
     0 +--------------------------------------------------------------------+   

perf-sched.sch_delay.avg.ms.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64

0.025 +-------------------------------------------------------------------+

        |                      :                                            |   
        |                      :           +        +                       |   
   0.02 |-+                    :           :        :                       |   
        |      +              ::           :        :                       |   
        |      :              ::          ::        :                       |   
  0.015 |-+    :              : :         ::        :               +       |   
        |     :: +            : :         : :       ::              :       |   
   0.01 |-++  : ::            : :         : :      : :              :  ++  +|   
        |  :  : :::           : :         : :      : :              :: ::  :|   
        | : : : : :          :  :        :  :      : :             : ::  :: |   
  0.005 |-: ::  :  :         :  :        :  :      : :             : ::  :: |   
        |+  ++  +  +++.+++.+++  ++.+++.+++  ++.+++.+ +.+++.+++.+++.+ +   +  |   
        |                                                                   |   
      0 +-------------------------------------------------------------------+   

perf-sched.wait_and_delay.count.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_6

310 +---------------------------------------------------------------------+

      |                                                                     |   
  305 |++ +         + +.+              +   ++.+  +   +++        +           |   
      |:  ::     + + ::  :       +.    ::  :   :+ :  :  +      + :          |   
  300 |-: :+  +  :+  +   +.++  ++  ++.+ +  :   +  +  :   ++.+++  ++  +   +  |   
      | : ::  :  :          :  :        :  :      :  :            :  ::  :: |   
  295 |-: : : :: :          :  :        :  :      : :              : ::  :: |   
      | ::  : :::           : :          : :      : :              : : ::  :|   
  290 |-+:  ::  :           : :          : :      : :              : : ++  +|   
      |  :  ::  +            ::          ::        ::              ::       |   
  285 |-++   :               ::          ::        ::               :       |   
      |      +               ::          ::        ::               :       |   
  280 |-+                    :            :        :                +       |   
      |                      :            :        :                        |   
  275 +---------------------------------------------------------------------+   

16.9 +--------------------------------------------------------------------+

  16.8 |-+                :+.+                                              |   
       |                  :  :       +                                      |   
  16.7 |:+                :  :       :                                   +  |   
  16.6 |++                :  :      ::                        +          :  |   
       |:                 :  :   +  ::       +        +       :          :: |   
  16.5 |-:      +.       :   :  +: +  :      ::       :+  +  + :         :::|   
  16.4 |-:      : +      :    :+ : :  :      :+       : + :: : :         : :|   
  16.3 |-:  +   : :      :    :: : :  :      : :      : : :::   :       :  +|   
       | :  ::  :  :  +  :    ::  ::  :+    :  : +   :   :  +   +       :   |   
  16.2 |-+:+ : :   +. :  :    :   :   + + +.:   + :.+:   +      :       :   |   
  16.1 |-++   +:     + :.+    :   :      +  +     +  +           :  +   +   |   
       |       +       +      +   +                              +. :+. :   |   
    16 |-+                                                         +   +    |   

15.9 +--------------------------------------------------------------------+

0.84 +--------------------------------------------------------------------+

       |                      +                                             |   
  0.82 |-+                    :                                             |   
       |      +               :           +         +                       |   
   0.8 |-+    :               :           :         :                       |   
       |      :               ::          ::        :               +       |   
  0.78 |-+   :: +            : :          ::       ::               :       |   
       |  +  :: :+           : :          ::       : :              :  +   +|   
  0.76 |-+:  : :: +          : :         : :       : :              :: :   :|   
       | : : : ::  :         : +         :  :      : :             : :: : ::|   
  0.74 |-: ::  :   +    .+   +  :  +.  + :  :++. +:  :+.+       ++.: :: + : |   
       |+  +:  :    + ++  + +   :  : ++ +:  +   + :  +  : +.   +   + +   +  |   
  0.72 |-+  +  +     +     +     :+      +        +      :: +++             |   
       |                         +                       +                  |   
   0.7 +--------------------------------------------------------------------+   

6 +-----------------------------------------------------------------------+

    |                                                                       |   
  5 |-+                         +                                           |   
    |                           :                                           |   
    |                           :                                           |   
  4 |-+                         ::                                          |   
    |                           ::                                          |   
  3 |-+                         ::                                          |   
    |:                          ::                                          |   
  2 |:+                        : :                                          |   
    |:                         : :                                          |   
    |:                         :  :                                         |   
  1 |:+                        :  :                                         |   
    |:                         :  :                                         |   
  0 +-----------------------------------------------------------------------+   

270 +---------------------------------------------------------------------+

  260 |-+       +                                                        :  |   
      |         ::                                                       :  |   
  250 |-+       :+         + +                                   +      : : |   
  240 |-+       ::      + +: :                                   :      : : |   
      |        : :      :+ : ::                                  :      + : |   
  230 |-+      : :      :   : : +                                :      : : |   
  220 |++      : :     :    : : :   +  +         +           +  : :     : : |   
  210 |:+      :  :    :    + : :  +:  :+.+      :++         :: : :    :  : |   
      |  +     :  :    :       : ::  : :  : +   :  :  +      :+.: :  +.:   :|   
  200 |-+: +.  :  :    :       : ::  : :  : :+  :   : ::    :   + :.+  +   :|   
  190 |-+ :: +:   :.++:        + +   ::    :: + :   : :+.+ .+     +        +|   
      |   :   :   +   +               :    :   :     :    +                 |   
  180 |-+ +   +                       +    +   +     +                      |   

170 +---------------------------------------------------------------------+

6 +-----------------------------------------------------------------------+

    |                                                                       |   
  5 |-+                         +                                           |   
    |                           :                                           |   
    |                           :                                           |   
  4 |-+                         ::                                          |   
    |                           ::                                          |   
  3 |-+                         ::                                          |   
    |:                          ::                                          |   
  2 |:+                        : :                                          |   
    |:                         : :                                          |   
    |:                         :  :                                         |   
  1 |:+                        :  :                                         |   
    |:                         :  :                                         |   
  0 +-----------------------------------------------------------------------+   

perf-sched.wait_time.max.ms.rcu_gp_kthread.kthread.ret_from_fork

5.2 +---------------------------------------------------------------------+

      |                                                                     |   
    5 |+.+++. +.++  +++.++.+++.+++.++.+   +++.++   +.+++.++   +.+++.++.+ +.+|   
      |      +   :  :                 :   :    :   :      :   :        : :  |   
  4.8 |-+        :  :                 :   :    :   :      :   :        : :  |   
      |          : :                  :  :      :  :       :  :        : :  |   
  4.6 |-+        : :                  :  :      :  :       :  :        : :  |   
      |           ::                   : :      :  :       :  :         :   |   
  4.4 |-+         ::                   : :      : :        : :          :   |   
      |           ::                   : :      : :        : :          :   |   
  4.2 |-+         :                    ::        ::         ::          :   |   
      |           :                    ::        ::         ::          +   |   
    4 |-+         +                    ++        ++         ++              |   
      |                                                                     |   
  3.8 +---------------------------------------------------------------------+   

perf-sched.wait_time.avg.ms.rcu_gp_kthread.kthread.ret_from_fork

3.2 +---------------------------------------------------------------------+

      |   +                        +                            +           |   
  3.1 |-+ :         +              ::                           :    +      |   
    3 |-+ :         : +            :+                    +     ::    :      |   
      |  ::         ::::        + : :       +         +  :+    : :   ::     |   
  2.9 |+.+ :  +    : + : +.+    : :  :    + :      +  : : :   :  :   ::     |   
  2.8 |:+  :  :+   :    ::  :  :: :  :   + : :     :+:: :  :  +  +  : :    :|   
      |:   :  : +  :    +   +  +: :  :  +  + :    +  + ::  :  :   +.: :    +|   
  2.7 |-+  +.:  :  :        : :  ::   : :     :   :    :    : :     +  :  + |   
  2.6 |-+    +   ::          ::  :    ++      ++.+     +    +:         + +  |   
      |          ::          +   :                           :         : :  |   
  2.5 |-+        ++              +                           +          ::  |   
  2.4 |-+                                                               :   |   
      |                                                                 +   |   
  2.3 +---------------------------------------------------------------------+   

perf-sched.wait_and_delay.count.rcu_gp_kthread.kthread.ret_from_fork

220 +---------------------------------------------------------------------+

      |    +                                                                |   
  200 |-+  :                                                                |   
  180 |-+  :                                                                |   
      |    ::   +           +                                               |   
  160 |-+  ::   :           :                                 +             |   
      |    ::  ::           :   +                             :             |   
  140 |-+ : :  : :       + : :  :                             ::        +   |   
      |   : :  : :       :+: :  :   +.+        +.        +   : :        :  +|   
  120 |+.+: :  : :       : + : : :  :  +  +    : + +    + +.++  :   +   :  :|   
  100 |-+::  ::  ++.    :    +.+ :  :   :+ :  +   ::+  +        +  + : : :: |   
      |   :  :+     +++.:        : :    +  + :    +  + :        : +  +.: :: |   
   80 |-+ +  +          +        +.+        ::        :          ::    + :  |   
      |                                     +         +          :       +  |   
   60 +---------------------------------------------------------------------+   

perf-sched.wait_and_delay.max.ms.rcu_gp_kthread.kthread.ret_from_fork

5.2 +---------------------------------------------------------------------+

      |                                   +        +.                       |   
    5 |+.+++.++.++  +++.++.+++.+++.++.+   :++.++   : +++.++   +.+++.++.+ +.+|   
      |          :  :                 :   :    :   :      :   :        : :  |   
      |          :  :                 :   :    :   :      :   :        : :  |   
  4.8 |-+        :  :                 :  :      :  :       :  :        : :  |   
      |          : :                  :  :      :  :       :  :        : :  |   
  4.6 |-+        : :                  :  :      :  :       :  :         ::  |   
      |           ::                   : :      : :        :  :         :   |   
  4.4 |-+         ::                   : :      : :        : :          :   |   
      |           ::                   : :      : :        : :          :   |   
      |           ::                   ::        ::         ::          :   |   
  4.2 |-+         :                    ::        ::         ::          +   |   
      |           :                    ::        ::         ::              |   
    4 +---------------------------------------------------------------------+   

perf-sched.wait_and_delay.avg.ms.rcu_gp_kthread.kthread.ret_from_fork

3.2 +---------------------------------------------------------------------+

      |                            +                                        |   
  3.1 |-+ +         +              ::                           +    +      |   
    3 |-+ :         : +            :+                    +      :    :      |   
      |  ::         :::         + : :     + +      +  +  ::    ::    ::     |   
  2.9 |-.+:        : + : +.+    : :  :    : :      :: : : +    : :  : :     |   
      |+   :  +    :   : : :   :: :  :   : :::     :: : : :   +  :  : :    :|   
  2.8 |:+  :  :+   :    :   :  +: :  :   : : :    :  : ::  :  :  + .+ :    :|   
      |:   : :  +  :    +   +  : ::  :  +  + :    +  + ::  :  :   +    :   +|   
  2.7 |-+  +.+  :  :        : :  ::   : :     :   :    :    : :        :  : |   
  2.6 |-+        ::          ::  :    ++      ++.+     +    +:         +  : |   
      |          ::          +   :                           :         : +  |   
  2.5 |-+        ++              +                           +          ::  |   
      |                                                                 +   |   
  2.4 +---------------------------------------------------------------------+   

perf-sched.wait_time.max.ms.schedule_timeout.kcompactd.kthread.ret_from_fork

505.2 +-------------------------------------------------------------------+

        |                                                                   |   
    505 |+.+++.+++.+ +.+++.+++.+++.+++.+++. ++.+++.+ +.+++.+   +++.+++.+ +.+|   
        |           +                      +        +      :   :       : :  |   
        |                                                  :   :       : :  |   
  504.8 |-+                                                :   :       : :  |   
        |                                                  :  :        : :  |   
  504.6 |-+                                                :  :        : :  |   
        |                                                   : :         :   |   
  504.4 |-+                                                 : :         :   |   
        |                                                   : :         :   |   
        |                                                   : :         :   |   
  504.2 |-+                                                 ::          :   |   
        |                                                   ::          +   |   
    504 +-------------------------------------------------------------------+   

perf-sched.wait_time.avg.ms.schedule_timeout.kcompactd.kthread.ret_from_fork

479 +-------------------------------------------------------------------+
| .+ |

  478.8 |+.++  +++.+ +.+++.+++  ++.+++  ++.+++.+ +.+++.+++ : +.++  +   + +.+|   
        |   :  :   : :        : :    :  :      : :         : :  :  :   : : :|   
  478.6 |-+ : :    : :        ::     :  :      : :         : :  :  :   : : :|   
  478.4 |-+  ::    : :         +      : :      : :         : :  : ::  :: : :|   
        |    ::     ::                : :      : :         : :  : ::  :: : :|   
  478.2 |-+  :      :                 : :      : :          ::  : ::  : :  :|   
        |    +      :                 : :       ::          :    :: : : :   |   
    478 |-+         :                 ::        :           :    :: ::  :   |   
        |           +                 ::        :           :    :: :+  :   |   
  477.8 |-+                            :        :           :    :  ::  +   |   
  477.6 |-+                            :        :           :    :  :       |   
        |                              :        :           +    +  +       |   

477.4 +-------------------------------------------------------------------+

perf-sched.sch_delay.max.ms.schedule_timeout.kcompactd.kthread.ret_from_fork

1.2 +---------------------------------------------------------------------+

      |                                                                     |   
    1 |-+                                                              +    |   
      |                                                                :    |   
      |                                                                :    |   
  0.8 |-+                                                              :    |   
      |                                                               ::    |   
  0.6 |-+                                                             ::    |   
      |                                                               : :   |   
  0.4 |-+                                                             : :   |   
      |                                                               : :   |   
      |                                                               : :   |   
  0.2 |-+                                                            :  :   |   
      |                                                            .+:  : .+|   
    0 +---------------------------------------------------------------------+   

perf-sched.sch_delay.avg.ms.schedule_timeout.kcompactd.kthread.ret_from_fork

0.04 +-------------------------------------------------------------------+

        |                                                                   |   
  0.035 |-+                                                            +    |   
        |                                                              :    |   
   0.03 |-+                                                            :    |   
        |                                                             ::    |   
  0.025 |-+                                                           : :  +|   
        |                                                             : :  :|   
   0.02 |-+                                                           : : ::|   
        |                                                             : : : |   
  0.015 |-++   + +             +           +        +               +:  + : |   
        | + : + ::+           + :         + :       ::              ::   :  |   
   0.01 |++ ++  +  +  .+++.+ +  ++.+++.+++  ++.+++.+ +.+++.+++.+++.+ +   +  |   
        |           ++      +                                               |   
  0.005 +-------------------------------------------------------------------+   

perf-sched.wait_and_delay.count.schedule_timeout.kcompactd.kthread.ret_from_fork

    80 +--------------------------------------------------------------------+   
       |   :  :   :  :       : :     : :      :  :        :  :  :  :   : : :|   
  79.5 |-+ : :    :  :        :      : :      :  :        :  :  :  :   : : :|   
    79 |-+ : :    : :         +      : :      :  :        :  :  :  :  :: : :|   
       |    ::    : :                : :       : :         : :  : ::  :: : :|   
  78.5 |-+  ::     ::                : :       : :         : :  : ::  : :: :|   
       |    :      ::                : :       : :         : :  : ::  : :  :|   
    78 |-+  +      ::                 ::       : :         : :   :: : : :   |   
       |           :                  :        ::          ::    :: ::  :   |   
  77.5 |-+         :                  :        ::          ::    :: ::  :   |   
    77 |-+         +                  :        ::          ::    :: :+  +   |   
       |                              :         :           :    :  ::      |   
  76.5 |-+                            :         :           :    :  ::      |   
       |                              :         :           :    :  :       |   
    76 +--------------------------------------------------------------------+   

perf-sched.wait_and_delay.max.ms.schedule_timeout.kcompactd.kthread.ret_from_for

505.2 +-------------------------------------------------------------------+

        |                                                             .+    |   
    505 |+.+++.+++.+ +.+++.+++.+++.+++.+++. ++.+++.+ +.+++.+   +++.+++ : +.+|   
        |           +                      +        +      :   :       : :  |   
        |                                                  :   :       : :  |   
  504.8 |-+                                                :   :       : :  |   
        |                                                  :  :        : :  |   
  504.6 |-+                                                :  :         ::  |   
        |                                                   : :         :   |   
  504.4 |-+                                                 : :         :   |   
        |                                                   : :         :   |   
        |                                                   : :         :   |   
  504.2 |-+                                                 ::          :   |   
        |                                                   ::          +   |   
    504 +-------------------------------------------------------------------+   

perf-sched.wait_and_delay.avg.ms.schedule_timeout.kcompactd.kthread.ret_from_for

479 +-------------------------------------------------------------------+
| +. .+ |

  478.8 |+.++  +++.+ +.++  +++  ++.+++  ++.+++.+ +.+++.+++ : +.++  +   + +.+|   
        |   :  :   : :        : :    :  :      : :         : :  :  :   : : :|   
  478.6 |-+ : :    : :        ::     :  :      : :         : :  :  :   : : :|   
  478.4 |-+  ::    : :         +      : :      : :         : :  : ::  :: : :|   
        |    ::     ::                : :      : :         : :  : ::  : :: :|   
  478.2 |-+  :      :                 : :      : :          ::  : ::  : :  :|   
        |    +      :                 ::        :           :    :: : : :   |   
    478 |-+         :                 ::        :           :    :: ::  :   |   
        |           +                 ::        :           :    :: :+  +   |   
  477.8 |-+                            :        :           :    :  ::      |   
  477.6 |-+                            :        :           :    :  :       |   
        |                              +        +           +    +  +       |   

477.4 +-------------------------------------------------------------------+

perf-sched.sch_delay.avg.ms.wait_for_partner.fifo_open.do_dentry_open.do_open.is

0.004 +------------------------------------------------------------------+

         |        :                                                 :       |   
         |        :                                                 :       |   
  0.0035 |-+      :                                                 :       |   
         |        ::                                                ::      |   
         |        ::                                                ::      |   
         |        ::                                                ::      |   
   0.003 |-+      ::           +           +        +               :+.+   +|   
         |       : :           :           :        :              :   :   :|   
         |       : :           ::          :        ::             :   :  ::|   
  0.0025 |-+     : :           ::          :        ::             :   :  ::|   
         |       :  :         : :         : :      : :             :    : : |   
         |       :  :         : :         : :      : :             :    : : |   
         |       :  :         :  :        : :      :  :            :    ::  |   
   0.002 +------------------------------------------------------------------+   

72 +--------------------------------------------------------------------+
|: :: : : : : :: : : : : :: |
|: :: : : : : :: : : : : :: |

  71.8 |:+ ::          : :     :       :     ::  :    : :       :      ::   |   
       |:: : :         : :     ::      ::    : : :    : :      ::     : :   |   
       |:: : :         : :     ::      ::    : : :    : :      ::     : :   |   
  71.6 |:: : :         : :     ::      ::    : : :    : :      ::     : :   |   
       |:: : :         :  :    ::      ::    : : ::   :  :     : :    :  :  |   
  71.4 |-::  :        :   :   : :     : :   :  :: :  :   :     : :    :  :  |   
       | ::  :        :   :   : :     : :   :  :: :  :   :     : :    :  :  |   
       | ::  :        :   :   : :     : :   :  :: :  :   :     : :    :  :  |   
  71.2 |-+:   :       :   :   :  :    :  :  :   : :  :   :    :  :   :   :  |   
       |  :   :       :   :   :  :    :  :  :   : :  :   :    :  :   :   :  |   
       |  :   :       :   :   :  :    :  :  :   : :  :   :    :  :   :   :  |   
    71 +--------------------------------------------------------------------+   

0.004 +------------------------------------------------------------------+

         |        :                                                 :       |   
         |        :                                                 :       |   
  0.0035 |-+      :                                                 :       |   
         |        ::                                                ::      |   
         |        ::                                                ::      |   
         |        ::                                                ::      |   
   0.003 |-+      ::           +           +        +               :+.+   +|   
         |       : :           :           :        :              :   :   :|   
         |       : :           ::          :        ::             :   :  ::|   
  0.0025 |-+     : :           ::          :        ::             :   :  ::|   
         |       :  :         : :         : :      : :             :    : : |   
         |       :  :         : :         : :      : :             :    : : |   
         |       :  :         :  :        : :      :  :            :    ::  |   
   0.002 +------------------------------------------------------------------+   

7500 +--------------------------------------------------------------------+

       |+  ++.++   +.+++.++ .+ +.+++.+++.+  +++.+++  ++.++  +++.++.+++.++  +|   
  7000 |:+ :    +.+        +  +          :  :     :  :   :  :           :  :|   
       |:  :                             :  :     :  :   :  :           :  :|   
       | : :                             : :       : :   : :            : : |   
  6500 |-: :                             : :       : :   : :            : : |   
       | : :                             : :       : :   : :            : : |   
  6000 |-::                               ::       ::     ::             :: |   
       | ::                               ::       ::     ::             :: |   
  5500 |-::                               ::       ::     ::             :: |   
       |  :                               :         :     :              :  |   
       |  :                               :         :     :              :  |   
  5000 |-++                               +         +     +              +  |   
       |                                                                    |   
  4500 +--------------------------------------------------------------------+   

750 +---------------------------------------------------------------------+

      |      +                                                              |   
  700 |++    ::   +              +      +         +         +    +  +       |   
      |:    : +   :+             :+     :         :         :   +: ::  +    |   
  650 |::   :  :  : +            : +    :         :         :   : :: :+ +   |   
      | :  :   : :   :+          : :    :: +      :  +     : : :  +  +  :   |   
  600 |-:  +    ++   +:         :  :   : : ::    ::  :: .+ : : :        :  +|   
      |  : :           :    ++. :   :  : : :+.+  ::  :++ : : +:         :  :|   
  550 |-+++            : +.+   ++   :.+: : :   +.: ::     ::  +         : : |   
      |                 ::          +  + : :     + ::     :              :: |   
  500 |-+               +                ::        ::     +              :: |   
      |                                   :        ::                    :: |   
  450 |-+                                 :        :                     :  |   
      |                                   +        +                     +  |   
  400 +---------------------------------------------------------------------+   

0.045 +-------------------------------------------------------------------+

        |                                                               +   |   
   0.04 |-+                                                             :   |   
  0.035 |-+                                                             :   |   
        |                                                           +  ::   |   
   0.03 |-+                                                         :  +:   |   
        |                                                           :  : :  |   
  0.025 |++                    +   +       +        +              :: :  :  |   
        |:                    ::  ::      ::        :        +    .+ ::  :  |   
   0.02 |-:    +++.      +    : : : :     : :  +   : :  +    ::  +   ::  : +|   
  0.015 |-:   +    +   ++ : ++  ++  ++.+ +  : + +  + :  :  +: : +    :   :+ |   
        | : ++      + +   : :           ::  ++   :+  +.+ :+ +  +     +   +  |   
   0.01 |-: :        +     :            +        +       +                  |   
        |  :               +                                                |   
  0.005 +-------------------------------------------------------------------+   

70 +----------------------------------------------------------------------+

     |                 +                                                    |   
  65 |-+ +             :      +       +         +                        +  |   
     |   ::            ::     :+      :         :                        :  |   
     |   ::            ::    : :     + :       + :           ++          :: |   
  60 |-+ : :   +       :+    + :     : :       : :           ::          :: |   
     |   : +   ::     :  +  :   :   +  :    + :  :   .+  +  : :         :  :|   
  55 |:+:  :  : +.    :   + :   :  :   :  .+ ::  :  +  :+ : :  :  +     :  +|   
     |: :  :  :   +   :    +    :  :    :+   +    :+   +  ++   :  :: +  :   |   
  50 |+.:   :+     ++.+         : +     +         +            +.+ : ::.+   |   
     |  +   ::                  : :                                 + +     |   
     |      +                    ::                                         |   
  45 |-+                         :                                          |   
     |                           +                                          |   
  40 +----------------------------------------------------------------------+   

7500 +--------------------------------------------------------------------+

       |+  ++.++   +.+++.++ .+ +.+++.+++.+  +++.+++  ++.++  +++.++.+++.++  +|   
  7000 |:+ :    +.+        +  +          :  :     :  :   :  :           :  :|   
       |:  :                             :  :     :  :   :  :           :  :|   
       | : :                             : :       : :   : :            : : |   
  6500 |-: :                             : :       : :   : :            : : |   
       | : :                             : :       : :   : :            : : |   
  6000 |-::                               ::       ::     ::             :: |   
       | ::                               ::       ::     ::             :: |   
  5500 |-::                               ::       ::     ::             :: |   
       |  :                               :         :     :              :  |   
       |  :                               :         :     :              :  |   
  5000 |-++                               +         +     +              +  |   
       |                                                                    |   
  4500 +--------------------------------------------------------------------+   

750 +---------------------------------------------------------------------+

      |      +                                                              |   
  700 |++    ::   +              +      +         +         +    +  +       |   
      |:    : +   :+             :+     :         :         :   +: ::  +    |   
  650 |::   :  :  : +            : +    :         :         :   : :: :+ +   |   
      | :  :   : :   :+          : :    :: +      :  +     : : :  +  +  :   |   
  600 |-:  +    ++   +:         :  :   : : ::    ::  :: .+ : : :        :  +|   
      |  : :           :    ++. :   :  : : :+.+  ::  :++ : : +:         :  :|   
  550 |-+++            : +.+   ++   :.+: : :   +.: ::     ::  +         : : |   
      |                 ::          +  + : :     + ::     :              :: |   
  500 |-+               +                ::        ::     +              :: |   
      |                                   :        ::                    :: |   
  450 |-+                                 :        :                     :  |   
      |                                   +        +                     +  |   
  400 +---------------------------------------------------------------------+   

1200 +--------------------------------------------------------------------+

       |                         +                                          |   
  1100 |-+                       :                                          |   
  1000 |-+                      ::           +        +       +             |   
       |                     +  ::           :        ::      ::            |   
   900 |-+                   :  ::    +      :: +     ::      ::            |   
   800 |-+             +    ::  : :   :      ::+:     : :     : :           |   
       |              + +   :: :  :+.+ :.+  : +  :+  :  +    :  +           |   
   700 |-+            :  + :  :+  ::   + :  :    +:  :  : +  :  :  +        |   
   600 |-+           +    ++  ::  ::     :  +      : +   ::+ :   :+:        |   
       |+  +        +         :   :       ::       : :   +  ++   +  :    +  |   
   500 |-+ :+.++   +          +   +       ::        :               ++. + +:|   
   400 |-++     +. :                      +         +                  +   +|   
       |          +                                                         |   
   300 +--------------------------------------------------------------------+   

45 +----------------------------------------------------------------------+

     |  +                                                                   |   
  40 |-::      +             +           +         +                +       |   
     | ::     +:             :   +       :         :                ::      |   
  35 |++ : + + :  +          :   :       :         :     +          :+   +  |   
     |   :+: : :  :         ::   :       ::        :     :   +   + : :   :  |   
  30 |-+ + : : :  ::      + ::  ::       ::        :     :   :   : : :   :: |   
     |      ::  :: +      : : : ::      : :       : :   ::  ::   : :  :  :: |   
  25 |-+    :   :: :     :: : : ::      : :       : :   : : ::  :: :  : : : |   
     |      :   :: :     : :: : : :     : :       : :   : : : : : ::  : :  :|   
  20 |-+    +   :: :     : :  ::  :     :  :      : :   : : : : : :   +.:  +|   
     |          :   :    : :  ::  :     +  :      + :  :  ::  : : :     +  :|   
  15 |-+        +   :   :  +  ++  :    +   ++    +  +.+:  ++  +:  +         |   
     |              :  ++         :.+ +      +. +      +       +            |   
  10 +----------------------------------------------------------------------+   

1200 +--------------------------------------------------------------------+

       |                         +                                          |   
  1100 |-+                       :                                          |   
  1000 |-+                      ::           +        +       +             |   
       |                     +  ::           :        ::      ::            |   
   900 |-+                   :  ::    +      :: +     ::      ::            |   
   800 |-+             +    ::  : :   :      ::+:     : :     : :           |   
       |              + +   :: :  :+.+ :.+  : +  :+  :  +    :  +           |   
   700 |-+            :  + :  :+  ::   + :  :    +:  :  : +  :  :  +        |   
   600 |-+           +    ++  ::  ::     :  +      : +   ::+ :   :+:        |   
       |+  +        +         :   :       ::       : :   +  ++   +  :    +  |   
   500 |-+ :+.++   +          +   +       ::        :               ++. + +:|   
   400 |-++     +. :                      +         +                  +   +|   
       |          +                                                         |   
   300 +--------------------------------------------------------------------+   

0.14 +--------------------------------------------------------------------+

       |             ++    +.+ +  +          +        +     +   +  +   : +  |   
  0.12 |-+           ::    : : :  :          :        :     :   :  :   : :  |   
       |             ::    :  ::  :          :        :     :   :  :   : :  |   
       |            : :    :  : : ::         :        ::   ::  :: ::  : ::: |   
   0.1 |-+          : :    :  + : ::         :        ::   ::  : :::  : : : |   
       |            : :    :    : ::         :        ::   ::  : :::  : : : |   
  0.08 |-+          :  :   :    :: +        : :      : :   : : : :: : : : : |   
       |            :  : +:     :: :        : :      : :   : : : :  : : + : |   
  0.06 |-+      +.+ :  : ::     ::  :       : :      : :  :  : : +  : :   : |   
       |        : ::   ::::      :  :       : :      :  : :  ::     ::     :|   
       |    +  +   :   :: :      :   :+   +.: : +  .+:  : +  ::     ::     :|   
  0.04 |+.++ + :   +   +  :      +   + +.+  + :+ ++  +  :+   ++     ++     +|   
       |      +           +                   +         +                   |   
  0.02 +--------------------------------------------------------------------+   

0.032 +-------------------------------------------------------------------+

        |                                    :         :        +           |   
   0.03 |-+                          +       :         :        :      +    |   
        |                    +.++    :       ::       ::        :      :+   |   
  0.028 |-+            +     :  :  + ::      ::       ::       ::      ::   |   
        |             ::   + :  :  :: :      ::       : :      +:     :  :  |   
  0.026 |-+ +         : :  ::   : : : :     : :       : :   + :  :    :  :  |   
        |+  :+.+    ++  + : +    :: + :     :  :      : :   :::  : +  :  +  |   
  0.024 |:+:    +   :    ::      ::    :    :  :      : :  + +   :::  :   + |   
        |: +     +.:     +       ::    +    :  ++    :  + +      :::  :    +|   
  0.022 |-+        +             :     :  .+:   :   +:   +       +  ::     :|   
        |                        +      :+  +    :.+ +              ::     :|   
   0.02 |-+                             +        +                  ++      |   
        |                                                                   |   
  0.018 +-------------------------------------------------------------------+   

1000.15 +-----------------------------------------------------------------+

  1000.14 |-+                   :                                      +   :|   
          |     +  +            :                                      :   :|   
  1000.13 |-+   :  :            :           +        +                 :  ::|   
  1000.12 |-+   :  :            :           :        :                 :  ::|   
          |     :: :            :           :        :                 :  ::|   
  1000.11 |-+   :: :            :          ::       ::                 :  ::|   
   1000.1 |-+   :: :           : :         ::       ::                ::  : |   
  1000.09 |-+  : :: :          : :         : :      : :               : : : |   
          |    : :: :          : :         : :      : :               : ::  |   
  1000.08 |-+  :  : :          : :         : :      : :        +      : ::  |   
  1000.07 |-+  :  : :    .+  + : :       + : :    + : :        :+   +.+ :+  |   
          |    :  + :   +  :+ +: :.+  +.+::  :+.++::  :+.+  +.+  +  :   ::  |   
  1000.06 |:+ ++    +.+ :  +   + +  :+    :  +     :  +   ++      ++    :   |   

1000.05 +-----------------------------------------------------------------+

860 +---------------------------------------------------------------------+

      | :  :    : : :      : :            :        :     :   :   :  ::      |   
  840 |-:  ::  :: : :      : ::           :        ::    :   :   :  : :     |   
      |  :: :  :: : :     :: ::          ::        ::   ::   :   : :  :     |   
  820 |-+++ : + : :  :    :: ::          : :       ::   : :  ::  :::  :  +  |   
      |     : : : :  :    :: ::          : :      : :   : :  ::  :::  :  :: |   
  800 |-+    :: : :  :    : :  :         : :      :  :  : : : : : ::   : :: |   
      |      :   ::  :   :  :  :        :  :      :  : :  : : : : :    : : :|   
  780 |-+    +   :   ++.++  :  +++.++.+++  +  ++.++  + +  + : + : +    +:  +|   
      |          :          :              : :       : :   ::  ::      ::   |   
  760 |-+        :          :               ::        :    ::  ::       :   |   
      |          :          +               +         +     +   +       +   |   
  740 |-+        :                                                          |   
      |          :                                                          |   
  720 +---------------------------------------------------------------------+   

25 +--------------------------------------------------------------------+
| : |

  24.5 |-+        :                                                         |   
    24 |-+        :          +               +        +     +   +       +   |   
       |         ::          :               :        ::    :   :       :   |   
  23.5 |-+       ::          :               ::       ::   ::  ::       :   |   
       |         ::         ::              : :      :  :  ::  ::      ::   |   
    23 |-+    +  : :  ++.++ :: +.+++.+++.+  + +.+++  +  + + : + :  +   + : +|   
       |      :  : :  :   : :: :         :  :     :  :  : : : : :  :   : :: |   
  22.5 |-+   : : : :  :   : : ::         : :       : :  : :  ::  :::  :  :: |   
    22 |-+++ : + : :  :   : : ::         : :       : :  : :  ::  :::  :  +  |   
       |  :: : ::  : :     :: :           ::       ::    :   :   :: : :     |   
  21.5 |-:  ::  :  : :     :  :           ::       ::    :   :   :: : :     |   
       | :  :   :  : :     :  :           :         :    :   :   :  ::      |   
    21 +--------------------------------------------------------------------+   

1000.2 +-----------------------------------------------------------------+

          |                                                            +   :|   
  1000.18 |-+                   +                                      :   :|   
          |                     :                                      :  ::|   
  1000.16 |-+   +  +            :           +        +                 :  ::|   
  1000.14 |-+   :  :            :           :        :                 :  ::|   
          |     :: :            :           :        :                ::  : |   
  1000.12 |-+   :: :           : :         ::       ::                : : : |   
          |    : :: :          : :         ::       ::                : ::  |   
   1000.1 |-+  :  : :        + : :       + : :    + : :        ++    .+ ::  |   
          |    :  + :    .+  :+: :       : : :+.  : : :+.      : +  +   :+  |   
  1000.08 |-+  +    +.+ +  ++  + +.+ ++.+ :  :: ++ :  :: + ++.+   ++    ::  |   
  1000.06 |:.++        ::           ::    :  :     :  :   +             +   |   
          |+           +            +     +  +     +  +                     |   
  1000.04 +-----------------------------------------------------------------+   

860 +---------------------------------------------------------------------+

      | :  :    : : :      : :            :        :     :   :   :  ::      |   
  840 |-:  ::  :: : :      : ::           :        ::    :   :   :  : :     |   
      |  :: :  :: : :     :: ::          ::        ::   ::   :   : :  :     |   
  820 |-+++ : + : :  :    :: ::          : :       ::   : :  ::  :::  :  +  |   
      |     : : : :  :    :: ::          : :      : :   : :  ::  :::  :  :: |   
  800 |-+    :: : :  :    : :  :         : :      :  :  : : : : : ::   : :: |   
      |      :   ::  :   :  :  :        :  :      :  : :  : : : : :    : : :|   
  780 |-+    +   :   ++.++  :  +++.++.+++  +  ++.++  + +  + : + : +    +:  +|   
      |          :          :              : :       : :   ::  ::      ::   |   
  760 |-+        :          :               ::        :    ::  ::       :   |   
      |          :          +               +         +     +   +       +   |   
  740 |-+        :                                                          |   
      |          :                                                          |   
  720 +---------------------------------------------------------------------+   

999.75 +------------------------------------------------------------------+

         |        +                                                         |   
   999.7 |-+      :                                                         |   
         |        ::         +                                              |   
         |   +  +: :  +   +  ::   +      +        +                 +   +   |   
  999.65 |-+ :  :: :  :  +: : ++. ::    : : +    : :  +            + :.+:   |   
         | .+ :: + :  :::  ::    + +    : + ::   : +  :  .+       +  +  :  +|   
   999.6 |++  ::    :: ::  +       :   +  : :: ++  : : :+  :  ++ +       :::|   
         |    +     :+ +            :  :  : : ::   : : ::  + +  ::       :::|   
  999.55 |-+        +               :  :   :  +     :: +    +   +        +  |   
         |                          : :    :        :                       |   
         |                           ::    +        +                       |   
   999.5 |-+                         ++                                     |   
         |                                                                  |   
  999.45 +------------------------------------------------------------------+   

900 +---------------------------------------------------------------------+

      |:                        :  :                                        |   
  880 |:+                       :  :                                        |   
      |:                        :  :                                        |   
      |:                        : :                                         |   
  860 |:+                       : :                                         |   
      |:                         ::                                         |   
  840 |-+                        ::                                         |   
      |                          ::                                         |   
  820 |-+                        ::                                         |   
      |                          :                                          |   
      |                          :                                          |   
  800 |-+                        +                                          |   
      |                                                                     |   
  780 +---------------------------------------------------------------------+   

10.1 +-------------------------------------------------------------------+

  10.08 |-+                                                                 |   
        |                                                                   |   
  10.06 |-+                                                                 |   
  10.04 |-+                                                                 |   
  10.02 |-+                                                                 |   
     10 |+.+++.+++.+++.+++.+++.+++.+++.+++.+++.+++.+++.+++.+++.+++.+++.+++.+|   
        |                                                                   |   
   9.98 |-+                                                                 |   
   9.96 |-+                                                                 |   
   9.94 |-+                                                                 |   
   9.92 |-+                                                                 |   
        |                                                                   |   
    9.9 |-+                                                                 |   

9.88 +-------------------------------------------------------------------+

999.75 +------------------------------------------------------------------+

         |        :                                                         |   
   999.7 |-+      :          +                                              |   
         |      +: :         :                                      +       |   
         |   +  :: :  +   + : :+. +      +  +     +   +             :  ++   |   
  999.65 |-+ : : + :  :  + :: +  + :    : + ::   : +  :           .+ :+ :  +|   
         | .+ ::   :  ::+  +       +    : : ::   : : : : .+   +  +   +   :::|   
   999.6 |++  +     :: +           :   +  : : :++  : : :+  + : + :       :::|   
         |          :+              :  :   :  +     :: +    ::  +        +  |   
  999.55 |-+        +               :  :   :        :       +               |   
         |                          : :    +        +                       |   
         |                           ::                                     |   
   999.5 |-+                         ++                                     |   
         |                                                                  |   
  999.45 +------------------------------------------------------------------+   

900 +---------------------------------------------------------------------+

      |:                        :  :                                        |   
  880 |:+                       :  :                                        |   
      |:                        :  :                                        |   
      |:                        : :                                         |   
  860 |:+                       : :                                         |   
      |:                         ::                                         |   
  840 |-+                        ::                                         |   
      |                          ::                                         |   
  820 |-+                        ::                                         |   
      |                          :                                          |   
      |                          :                                          |   
  800 |-+                        +                                          |   
      |                                                                     |   
  780 +---------------------------------------------------------------------+   

perf-sched.sch_delay.avg.ms.devkmsg_read.vfs_read.ksys_read.do_syscall_64

0.06 +--------------------------------------------------------------------+

       |        :                         :         :                  :    |   
  0.05 |-+      :                         :         :                  :    |   
       |      + ::                        ::       ::                  :    |   
       |      : ::                        ::       ::                 ::    |   
  0.04 |-+    : ::                        ::       ::                 ::   +|   
       |     :: ::                       : :       : :                ::   :|   
  0.03 |-+   ::: :            +          : :       : :              + : : ::|   
       |  +  : : :            :          : :       : :              : : : ::|   
  0.02 |-+:  : :  :           :          :  :     :  :              : : : : |   
       | :: :  :  :          : :         :  :     :  :             : :  : : |   
       | : ::  :  :     .+ +.+ :       +.+  :    ++  :     .+      : :  ::  |   
  0.01 |++ :+  +  ++. ++  +    +. + .+ :    + +. :   + .+++  ++.++.+ +  :+  |   
       |   +         +           + +  +      +  +     +                 +   |   
     0 +--------------------------------------------------------------------+   

perf-sched.wait_and_delay.count.devkmsg_read.vfs_read.ksys_read.do_syscall_64

8 +-----------------------------------------------------------------------+

    |  :   :  :                                                     :       |   
    |  :   :  :                                                     :       |   
  7 |-::   : ::             +            +         +               :: +.+  +|   
    | ::   : ::             :            :         :               :: : :  :|   
    | ::   : ::             :            :         :               :: : :  :|   
  6 |-: : : :::             :           ::        ::               : :: : ::|   
    |:  : : :: :            ::          : :       : :              : ::  :: |   
  5 |++ : : :: :            ::          : :       : :              : :   :: |   
    |   : : :  :           : :          : :       : :             :  :   :: |   
    |   : : :  :           : :         :  :      :  :             :  :   :  |   
  4 |-+ +.+ +  :           : +     ++.++  +.+++.++  ++.++.+++.++.++  +   +  |   
    |          :           :  :    :                                        |   
    |          :           :  :   :                                         |   
  3 +-----------------------------------------------------------------------+   

perf-sched.wait_and_delay.count.do_syslog.part.0.kmsg_read.vfs_read

8 +-----------------------------------------------------------------------+

    |  :   :  :                                                     :       |   
    |  :   :  :                                                     :       |   
  7 |-::   : ::             +            +         +               :: +.+  +|   
    | ::   : ::             :            :         :               :: : :  :|   
    | ::   : ::             :            :         :               :: : :  :|   
  6 |-: : : :::             :           ::        ::               : :: : ::|   
    |:  : : :: :            ::          : :       : :              : ::  :: |   
  5 |++ : : :: :            ::          : :       : :              : :   :: |   
    |   : : :  :           : :          : :       : :             :  :   :: |   
    |   : : :  :           : :         :  :      :  :             :  :   :  |   
  4 |-+ +.+ +  :           : +     ++.++  +.+++.++  ++.++.+++.++.++  +   +  |   
    |          :           :  :    :                                        |   
    |          :           :  :   :                                         |   
  3 +-----------------------------------------------------------------------+   

0.09 +--------------------------------------------------------------------+

       |        +                         +         +                       |   
  0.08 |-++     :                         :         :                       |   
  0.07 |-+:     :                         :         :                       |   
       |  :     ::                        ::       ::                       |   
  0.06 |-::     ::                        ::       ::                       |   
  0.05 |-::     ::                        ::       ::                       |   
       | ::    : :                       : :       : :                 +    |   
  0.04 |-: :   : :                       : :       : :                 :    |   
  0.03 |-: :   : :                       : :       : :              + ::    |   
       |:  :   :  :                    + :  :    +:  :              : : :   |   
  0.02 |:+ :   :  :           +        :+:  :    ::  :             : :: :   |   
  0.01 |++ : .++  ++. +    +. ::.++   :  +  +++.: +  ++.+++.+  .+ .+ :  :+. |   
       |   ++        + +.++  + +   +.++         +            ++  +   +  +  +|   
     0 +--------------------------------------------------------------------+   

perf-sched.wait_time.max.ms.schedule_timeout.khugepaged.kthread.ret_from_fork

0.09 +--------------------------------------------------------------------+

       |                      +                                             |   
  0.08 |-+                    :                                             |   
  0.07 |-+                    :                                             |   
       |                      :                                             |   
  0.06 |-+                    :                                             |   
  0.05 |-+                   ::                                             |   
       |                     : :                                            |   
  0.04 |-+                   : :                                            |   
  0.03 |-+                   : :                                            |   
       |+                    + :   +                      +                 |   
  0.02 |:+               +  :  :   ::     +         +     ::                |   
  0.01 |-++ +.+ +.+   + + + :  +. : : ++. :+   .++ + :   : :     +.+ +.  +.+|   
       |   +   +   +.+ +   +     ++  +   +  +++   +  ++.++  +++.+   +  ++   |   
     0 +--------------------------------------------------------------------+   

perf-sched.wait_time.avg.ms.schedule_timeout.khugepaged.kthread.ret_from_fork

0.09 +--------------------------------------------------------------------+

       |                      +                                             |   
  0.08 |-+                    :                                             |   
  0.07 |-+                    :                                             |   
       |                      :                                             |   
  0.06 |-+                    :                                             |   
  0.05 |-+                   ::                                             |   
       |                     : :                                            |   
  0.04 |-+                   : :                                            |   
  0.03 |-+                   : :                                            |   
       |+                    + :   +                      +                 |   
  0.02 |:+               +  :  :   ::     +         +     ::                |   
  0.01 |-++ +.+ +.+   + + + :  +. : : ++. :+   .++ + :   : :     +.+ +.  +.+|   
       |   +   +   +.+ +   +     ++  +   +  +++   +  ++.++  +++.+   +  ++   |   
     0 +--------------------------------------------------------------------+   

[*] bisect-good sample
[O] bisect-bad sample

***************************************************************************************************
lkp-skl-fpga01: 104 threads Skylake with 192G memory
=========================================================================================
compiler/cpufreq_governor/kconfig/mode/nr_task/rootfs/tbox_group/test/testcase/ucode:
gcc-9/performance/x86_64-rhel-8.3/process/16/debian-10.4-x86_64-20200603.cgz/lkp-skl-fpga01/malloc1/will-it-scale/0x2006a0a

commit:
dde38d5b80 ("kernel: don't include PF_IO_WORKERs as part of same_thread_group()") 43b2a76b1a ("proc: don't show PF_IO_WORKER threads as threads in /proc/<pid>/task/")

dde38d5b805358c7 43b2a76b1a5abcc9833b463bef1
---------------- ---------------------------
%stddev %change %stddev
\ | \
1105101 -11.1% 982154 ± 2% will-it-scale.16.processes

     83.20            -1.4%      82.01        will-it-scale.16.processes_idle
     69068           -11.1%      61384 ±  2%  will-it-scale.per_process_ops
   1105101           -11.1%     982154 ±  2%  will-it-scale.workload
      1.36 ± 15%      -0.7        0.67 ±  4%  mpstat.cpu.all.irq%
 6.653e+08           -11.0%  5.921e+08 ±  2%  numa-numastat.node0.local_node
 6.653e+08           -11.0%  5.921e+08 ±  2%  numa-numastat.node0.numa_hit
    626019 ±  3%    +223.0%    2022084 ± 22%  numa-numastat.node1.local_node
    677357 ±  4%    +202.5%    2049009 ± 21%  numa-numastat.node1.numa_hit
     83.00            -1.2%      82.00        vmstat.cpu.id
   1358743          +147.9%    3368229 ±  2%  vmstat.memory.cache
     15.17 ±  2%     +12.1%      17.00        vmstat.procs.r
      1602           +10.6%       1772        vmstat.system.cs
    599024 ±  2%     +70.6%    1022074 ± 38%  numa-meminfo.node0.FilePages
   1251096 ±  5%     +48.3%    1855764 ± 20%  numa-meminfo.node0.MemUsed
     55683 ±  4%   +2832.5%    1632907 ± 28%  numa-meminfo.node1.Active
     55560 ±  4%   +2830.2%    1628037 ± 29%  numa-meminfo.node1.Active(anon)
    661513 ±  2%    +238.7%    2240271 ± 20%  numa-meminfo.node1.FilePages
   1318940 ±  5%    +137.5%    3132286 ± 13%  numa-meminfo.node1.MemUsed
     64592 ±  4%   +2434.8%    1637306 ± 28%  numa-meminfo.node1.Shmem
     86167 ± 72%     -40.6%      51159 ±  2%  sched_debug.cfs_rq:/.load.max
     27024 ± 20%     -18.4%      22050        sched_debug.cfs_rq:/.load.stddev
    490216 ±  4%     +18.4%     580187 ±  5%  sched_debug.cfs_rq:/.min_vruntime.avg
    225.01 ± 12%     -23.1%     172.92 ± 10%  sched_debug.cfs_rq:/.util_est_enqueued.avg
    348.14 ±  5%     -23.6%     266.02 ±  7%  sched_debug.cfs_rq:/.util_est_enqueued.stddev
      1334 ± 25%     -48.8%     682.65 ±  3%  sched_debug.cpu.clock_task.stddev
      0.16 ±  2%     +11.0%       0.18        sched_debug.cpu.nr_running.avg
     57321 ±  3%   +3485.8%    2055409 ±  3%  meminfo.Active
     56950 ±  3%   +3497.8%    2048955 ±  4%  meminfo.Active(anon)
   1260522          +158.6%    3260277 ±  2%  meminfo.Cached
   2367306           +88.5%    4462262 ±  2%  meminfo.Committed_AS
   2569992           +94.0%    4985892        meminfo.Memused
      4496           +11.4%       5009        meminfo.PageTables
     76116 ±  2%   +2615.4%    2066827 ±  4%  meminfo.Shmem
   3023094           +64.9%    4985892        meminfo.max_used_kB
    149755 ±  2%     +70.6%     255457 ± 38%  numa-vmstat.node0.nr_file_pages
 3.314e+08           -10.4%   2.97e+08 ±  2%  numa-vmstat.node0.numa_hit
 3.313e+08           -10.4%   2.97e+08 ±  2%  numa-vmstat.node0.numa_local
     13864 ±  4%   +2833.3%     406677 ± 29%  numa-vmstat.node1.nr_active_anon
    165371 ±  2%    +238.5%     559735 ± 20%  numa-vmstat.node1.nr_file_pages
     16141 ±  4%   +2433.9%     408994 ± 28%  numa-vmstat.node1.nr_shmem
     13864 ±  4%   +2833.3%     406677 ± 29%  numa-vmstat.node1.nr_zone_active_anon
   1129707 ±  2%     +59.3%    1799567 ±  9%  numa-vmstat.node1.numa_hit
    947670 ± 11%     +68.0%    1591804 ± 11%  numa-vmstat.node1.numa_local
     31447 ±  2%     +41.8%      44578 ±  3%  slabinfo.filp.active_objs
    993.50 ±  2%     +40.7%       1398 ±  3%  slabinfo.filp.active_slabs
     31810 ±  2%     +40.7%      44750 ±  3%  slabinfo.filp.num_objs
    993.50 ±  2%     +40.7%       1398 ±  3%  slabinfo.filp.num_slabs
      8576           +52.7%      13093 ± 13%  slabinfo.kmalloc-256.active_objs
    268.17           +53.3%     411.17 ± 13%  slabinfo.kmalloc-256.active_slabs
      8605           +53.1%      13173 ± 13%  slabinfo.kmalloc-256.num_objs
    268.17           +53.3%     411.17 ± 13%  slabinfo.kmalloc-256.num_slabs
     13727           -35.8%       8815        slabinfo.proc_inode_cache.active_objs
     13729           -35.8%       8815        slabinfo.proc_inode_cache.num_objs
     22048           +36.6%      30115        slabinfo.radix_tree_node.active_objs
    393.00           +36.8%     537.67        slabinfo.radix_tree_node.active_slabs
     22048           +36.6%      30118        slabinfo.radix_tree_node.num_objs
    393.00           +36.8%     537.67        slabinfo.radix_tree_node.num_slabs
     14238 ±  3%   +3503.0%     513000 ±  4%  proc-vmstat.nr_active_anon
     61574            +8.1%      66554        proc-vmstat.nr_anon_pages
   4824998            -1.2%    4764837        proc-vmstat.nr_dirty_background_threshold
   9661794            -1.2%    9541325        proc-vmstat.nr_dirty_threshold
    315136          +158.9%     815833 ±  2%  proc-vmstat.nr_file_pages
  48538649            -1.2%   47933894        proc-vmstat.nr_free_pages
     66233            +7.0%      70862        proc-vmstat.nr_inactive_anon
      1122           +11.3%       1249        proc-vmstat.nr_page_table_pages
     19033 ±  2%   +2618.8%     517468 ±  4%  proc-vmstat.nr_shmem
     47871            +3.3%      49439        proc-vmstat.nr_slab_unreclaimable
     14238 ±  3%   +3503.0%     513000 ±  4%  proc-vmstat.nr_zone_active_anon
     66233            +7.0%      70862        proc-vmstat.nr_zone_inactive_anon
      2897 ± 91%    +114.1%       6202 ± 39%  proc-vmstat.numa_hint_faults
 6.656e+08           -10.8%   5.94e+08        proc-vmstat.numa_hit
 6.655e+08           -10.8%  5.939e+08        proc-vmstat.numa_local
     20412 ±  4%    +236.7%      68727 ±  3%  proc-vmstat.pgactivate
 6.661e+08           -10.6%  5.952e+08        proc-vmstat.pgalloc_normal
 3.335e+08           -11.0%  2.967e+08 ±  2%  proc-vmstat.pgfault
 6.661e+08           -10.8%   5.94e+08        proc-vmstat.pgfree
      8040 ±  9%     +16.9%       9395 ±  9%  softirqs.CPU1.RCU
      7061 ±  7%     +30.9%       9240 ± 12%  softirqs.CPU16.RCU
      7511 ± 10%     +43.0%      10743 ± 19%  softirqs.CPU17.RCU
      7696 ± 10%     +17.0%       9002 ± 11%  softirqs.CPU2.RCU
      7261 ± 13%     +32.3%       9605 ± 15%  softirqs.CPU20.RCU
      6764 ± 13%     +28.7%       8706 ±  8%  softirqs.CPU21.RCU
      5969 ±  8%     +95.5%      11670 ± 30%  softirqs.CPU27.RCU
      5875 ±  4%     +52.7%       8971 ± 31%  softirqs.CPU29.RCU
      6642 ±  7%     +29.9%       8626 ± 13%  softirqs.CPU45.RCU
      6641 ±  7%     +35.4%       8991 ± 22%  softirqs.CPU47.RCU
      6629 ±  7%     +30.5%       8648 ± 15%  softirqs.CPU48.RCU
      6314 ±  8%     +53.3%       9679 ± 31%  softirqs.CPU49.RCU
     17807 ±  7%    +120.2%      39216 ±  4%  softirqs.CPU51.SCHED
      6839 ±  7%     +38.3%       9462 ±  5%  softirqs.CPU58.RCU
      6665 ±  7%     +33.8%       8921 ±  7%  softirqs.CPU59.RCU
      7542 ±  9%     +17.3%       8847 ±  8%  softirqs.CPU66.RCU
      4833 ± 17%     +78.2%       8612 ± 28%  softirqs.CPU77.RCU
      7677 ± 13%     +24.4%       9552 ± 12%  softirqs.CPU78.RCU
     41305           -12.3%      36206 ±  8%  softirqs.CPU78.SCHED
      7588 ± 10%     +21.5%       9222 ± 19%  softirqs.CPU84.RCU
    780375 ±  6%     +19.5%     932332 ±  4%  softirqs.RCU
     43219 ±  7%     +23.9%      53545 ±  2%  softirqs.TIMER
     52.18 ± 10%     -52.2        0.00        perf-profile.calltrace.cycles-pp.__munmap
     51.08 ± 10%     -51.1        0.00        perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__munmap
     50.08 ± 10%     -50.1        0.00        perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__munmap
     50.05 ± 10%     -50.1        0.00        perf-profile.calltrace.cycles-pp.__x64_sys_munmap.do_syscall_64.entry_SYSCALL_64_after_hwframe.__munmap
     49.94 ± 10%     -49.9        0.00        perf-profile.calltrace.cycles-pp.__vm_munmap.__x64_sys_munmap.do_syscall_64.entry_SYSCALL_64_after_hwframe.__munmap
     49.77 ± 10%     -49.8        0.00        perf-profile.calltrace.cycles-pp.__do_munmap.__vm_munmap.__x64_sys_munmap.do_syscall_64.entry_SYSCALL_64_after_hwframe
     48.99 ± 10%     -49.0        0.00        perf-profile.calltrace.cycles-pp.unmap_region.__do_munmap.__vm_munmap.__x64_sys_munmap.do_syscall_64
     37.69 ± 10%     -37.7        0.00        perf-profile.calltrace.cycles-pp.tlb_finish_mmu.unmap_region.__do_munmap.__vm_munmap.__x64_sys_munmap
     37.63 ± 10%     -37.6        0.00        perf-profile.calltrace.cycles-pp.tlb_flush_mmu.tlb_finish_mmu.unmap_region.__do_munmap.__vm_munmap
     34.89 ± 19%     -34.9        0.00        perf-profile.calltrace.cycles-pp.secondary_startup_64_no_verify
     33.92 ± 17%     -33.9        0.00        perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
     33.92 ± 17%     -33.9        0.00        perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
     33.92 ± 17%     -33.9        0.00        perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64_no_verify
     33.57 ± 17%     -33.6        0.00        perf-profile.calltrace.cycles-pp.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
     33.46 ± 17%     -33.5        0.00        perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary
     33.02 ± 21%     -33.0        0.00        perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry
     31.26 ±  9%     -31.3        0.00        perf-profile.calltrace.cycles-pp.flush_tlb_mm_range.tlb_flush_mmu.tlb_finish_mmu.unmap_region.__do_munmap
     30.90 ±  9%     -30.9        0.00        perf-profile.calltrace.cycles-pp.flush_tlb_func_common.flush_tlb_mm_range.tlb_flush_mmu.tlb_finish_mmu.unmap_region
     30.60 ±  9%     -30.6        0.00        perf-profile.calltrace.cycles-pp.native_flush_tlb_one_user.flush_tlb_func_common.flush_tlb_mm_range.tlb_flush_mmu.tlb_finish_mmu
      5.94 ± 11%      -5.9        0.00        perf-profile.calltrace.cycles-pp.release_pages.tlb_flush_mmu.tlb_finish_mmu.unmap_region.__do_munmap
      5.90 ± 10%      -5.9        0.00        perf-profile.calltrace.cycles-pp.asm_exc_page_fault
      5.77 ± 10%      -5.8        0.00        perf-profile.calltrace.cycles-pp.unmap_vmas.unmap_region.__do_munmap.__vm_munmap.__x64_sys_munmap
      5.60 ± 10%      -5.6        0.00        perf-profile.calltrace.cycles-pp.unmap_page_range.unmap_vmas.unmap_region.__do_munmap.__vm_munmap
      5.03 ± 10%      -5.0        0.00        perf-profile.calltrace.cycles-pp.exc_page_fault.asm_exc_page_fault
      4.99 ± 10%      -5.0        0.00        perf-profile.calltrace.cycles-pp.do_user_addr_fault.exc_page_fault.asm_exc_page_fault
     54.84 ± 10%     -54.8        0.00        perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
     52.90 ± 10%     -52.9        0.00        perf-profile.children.cycles-pp.do_syscall_64
     52.27 ± 10%     -52.3        0.00        perf-profile.children.cycles-pp.__munmap
     50.05 ± 10%     -50.1        0.00        perf-profile.children.cycles-pp.__x64_sys_munmap
     49.95 ± 10%     -49.9        0.00        perf-profile.children.cycles-pp.__vm_munmap
     49.78 ± 10%     -49.8        0.00        perf-profile.children.cycles-pp.__do_munmap
     49.00 ± 10%     -49.0        0.00        perf-profile.children.cycles-pp.unmap_region
     37.69 ± 10%     -37.7        0.00        perf-profile.children.cycles-pp.tlb_finish_mmu
     37.64 ± 10%     -37.6        0.00        perf-profile.children.cycles-pp.tlb_flush_mmu
     34.89 ± 19%     -34.9        0.00        perf-profile.children.cycles-pp.secondary_startup_64_no_verify
     34.89 ± 19%     -34.9        0.00        perf-profile.children.cycles-pp.cpu_startup_entry
     34.89 ± 19%     -34.9        0.00        perf-profile.children.cycles-pp.do_idle
     34.53 ± 19%     -34.5        0.00        perf-profile.children.cycles-pp.cpuidle_enter
     34.53 ± 19%     -34.5        0.00        perf-profile.children.cycles-pp.cpuidle_enter_state
     33.92 ± 17%     -33.9        0.00        perf-profile.children.cycles-pp.start_secondary
     33.11 ± 21%     -33.1        0.00        perf-profile.children.cycles-pp.intel_idle
     31.28 ±  9%     -31.3        0.00        perf-profile.children.cycles-pp.flush_tlb_mm_range
     30.95 ±  9%     -30.9        0.00        perf-profile.children.cycles-pp.flush_tlb_func_common
     30.65 ±  9%     -30.7        0.00        perf-profile.children.cycles-pp.native_flush_tlb_one_user
      6.63 ± 12%      -6.6        0.00        perf-profile.children.cycles-pp.release_pages
      5.96 ± 10%      -6.0        0.00        perf-profile.children.cycles-pp.asm_exc_page_fault
      5.78 ± 10%      -5.8        0.00        perf-profile.children.cycles-pp.unmap_vmas
      5.71 ± 10%      -5.7        0.00        perf-profile.children.cycles-pp.unmap_page_range
      5.05 ± 10%      -5.1        0.00        perf-profile.children.cycles-pp.exc_page_fault
      5.01 ± 10%      -5.0        0.00        perf-profile.children.cycles-pp.do_user_addr_fault
      4.68 ± 11%      -4.7        0.00        perf-profile.children.cycles-pp.__mmap
     33.11 ± 21%     -33.1        0.00        perf-profile.self.cycles-pp.intel_idle
     30.60 ±  9%     -30.6        0.00        perf-profile.self.cycles-pp.native_flush_tlb_one_user
 5.794e+09           +10.5%  6.403e+09        perf-stat.i.branch-instructions
      0.67 ± 37%     +11.0       11.68 ±  6%  perf-stat.i.cache-miss-rate%
   1466858 ± 45%   +1979.5%   30503394 ±  6%  perf-stat.i.cache-misses
 2.278e+08 ±  8%     +14.8%  2.616e+08 ±  3%  perf-stat.i.cache-references
      1573           +10.8%       1742        perf-stat.i.context-switches
 4.718e+10           +11.7%   5.27e+10        perf-stat.i.cpu-cycles
    108.07            -1.8%     106.11        perf-stat.i.cpu-migrations
     48430 ± 64%     -96.3%       1798 ±  9%  perf-stat.i.cycles-between-cache-misses
      0.08 ± 11%      -0.0        0.06        perf-stat.i.dTLB-load-miss-rate%
 6.336e+09           +10.0%  6.967e+09        perf-stat.i.dTLB-loads
      0.15            -0.0        0.11        perf-stat.i.dTLB-store-miss-rate%
   4511491           -12.9%    3930479        perf-stat.i.dTLB-store-misses
  2.97e+09           +16.4%  3.457e+09        perf-stat.i.dTLB-stores
      3.67 ±  5%      +0.6        4.28 ±  8%  perf-stat.i.iTLB-load-miss-rate%
  97452404            -9.9%   87781036        perf-stat.i.iTLB-loads
  2.63e+10           +11.9%  2.944e+10        perf-stat.i.instructions
      0.85 ± 13%     -86.7%       0.11 ± 60%  perf-stat.i.major-faults
      0.45           +11.7%       0.51        perf-stat.i.metric.GHz
      0.94 ± 20%     -69.9%       0.28 ±  5%  perf-stat.i.metric.K/sec
    148.39           +11.4%     165.31        perf-stat.i.metric.M/sec
   1104170           -11.0%     982353 ±  2%  perf-stat.i.minor-faults
    124679 ± 19%   +4411.6%    5625000 ±  4%  perf-stat.i.node-load-misses
     18816 ± 18%   +3435.6%     665298 ±  6%  perf-stat.i.node-loads
     91.87            +7.0       98.92        perf-stat.i.node-store-miss-rate%
     31781 ±  7%  +10338.3%    3317480 ±  8%  perf-stat.i.node-store-misses
      5872 ±  9%    +393.6%      28983 ±  5%  perf-stat.i.node-stores
   1104171           -11.0%     982353 ±  2%  perf-stat.i.page-faults
      0.63 ± 41%     +11.0       11.67 ±  6%  perf-stat.overall.cache-miss-rate%
     43058 ± 57%     -96.0%       1734 ±  6%  perf-stat.overall.cycles-between-cache-misses
      0.08 ± 11%      -0.0        0.06        perf-stat.overall.dTLB-load-miss-rate%
      0.15            -0.0        0.11        perf-stat.overall.dTLB-store-miss-rate%
      3.66 ±  5%      +0.6        4.26 ±  8%  perf-stat.overall.iTLB-load-miss-rate%
     86.88            +2.5       89.40        perf-stat.overall.node-load-miss-rate%
     84.40           +14.7       99.13        perf-stat.overall.node-store-miss-rate%
   7160811           +26.0%    9025782        perf-stat.overall.path-length
 5.774e+09           +10.5%  6.382e+09        perf-stat.ps.branch-instructions
   1461552 ± 45%   +1979.8%   30397813 ±  6%  perf-stat.ps.cache-misses
 2.271e+08 ±  8%     +14.8%  2.608e+08 ±  3%  perf-stat.ps.cache-references
      1568           +10.8%       1737        perf-stat.ps.context-switches
 4.702e+10           +11.7%  5.252e+10        perf-stat.ps.cpu-cycles
    107.68            -1.8%     105.75        perf-stat.ps.cpu-migrations
 6.315e+09           +10.0%  6.943e+09        perf-stat.ps.dTLB-loads
   4496294           -12.9%    3917284 ±  2%  perf-stat.ps.dTLB-store-misses
  2.96e+09           +16.4%  3.446e+09        perf-stat.ps.dTLB-stores
  97123947            -9.9%   87486204 ±  2%  perf-stat.ps.iTLB-loads
 2.621e+10           +11.9%  2.934e+10        perf-stat.ps.instructions
      0.84 ± 13%     -86.7%       0.11 ± 60%  perf-stat.ps.major-faults
   1100449           -11.0%     979055 ±  2%  perf-stat.ps.minor-faults
    124211 ± 19%   +4413.3%    5605997 ±  4%  perf-stat.ps.node-load-misses
     18709 ± 18%   +3444.3%     663126 ±  6%  perf-stat.ps.node-loads
     31654 ±  7%  +10343.2%    3305726 ±  8%  perf-stat.ps.node-store-misses
      5835 ±  9%    +394.9%      28881 ±  5%  perf-stat.ps.node-stores
   1100450           -11.0%     979055 ±  2%  perf-stat.ps.page-faults
 7.914e+12           +12.0%  8.863e+12        perf-stat.total.instructions
      0.01 ± 58%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown]
      0.01 ± 48%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.devkmsg_read.vfs_read.ksys_read.do_syscall_64
      0.01 ± 41%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.do_nanosleep.hrtimer_nanosleep.__x64_sys_nanosleep.do_syscall_64
      0.01 ± 29%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.do_syslog.part.0.kmsg_read.vfs_read
      0.01 ± 31%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
      0.01 ± 74%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64
      0.01 ± 64%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.futex_wait_queue_me.futex_wait.do_futex.__x64_sys_futex
      0.00 ± 38%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.pipe_read.new_sync_read.vfs_read.ksys_read
      0.02 ±173%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.rcu_gp_kthread.kthread.ret_from_fork
      0.01 ± 32%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait.__x64_sys_epoll_wait
      0.01 ± 23%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_select
      0.00 ± 51%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_sys_poll
      0.01 ± 24%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.schedule_timeout.kcompactd.kthread.ret_from_fork
      0.01 ±111%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.schedule_timeout.rcu_gp_kthread.kthread.ret_from_fork
      0.01 ± 89%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.schedule_timeout.wait_for_completion.__flush_work.lru_add_drain_all
      0.01 ± 41%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.smpboot_thread_fn.kthread.ret_from_fork
      0.00 ± 11%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.wait_for_partner.fifo_open.do_dentry_open.do_open.isra
      0.02 ± 20%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.worker_thread.kthread.ret_from_fork
      0.01 ± 79%    -100.0%       0.00        perf-sched.sch_delay.max.ms.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown]
      0.02 ±127%    -100.0%       0.00        perf-sched.sch_delay.max.ms.devkmsg_read.vfs_read.ksys_read.do_syscall_64
      0.02 ± 55%    -100.0%       0.00        perf-sched.sch_delay.max.ms.do_nanosleep.hrtimer_nanosleep.__x64_sys_nanosleep.do_syscall_64
      0.02 ± 89%    -100.0%       0.00        perf-sched.sch_delay.max.ms.do_syslog.part.0.kmsg_read.vfs_read
      0.06 ± 62%    -100.0%       0.00        perf-sched.sch_delay.max.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
      0.06 ± 85%    -100.0%       0.00        perf-sched.sch_delay.max.ms.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64
      0.01 ± 21%    -100.0%       0.00        perf-sched.sch_delay.max.ms.exit_to_user_mode_prepare.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe.[unknown]
      0.03 ±121%    -100.0%       0.00        perf-sched.sch_delay.max.ms.futex_wait_queue_me.futex_wait.do_futex.__x64_sys_futex
      0.07 ± 65%    -100.0%       0.00        perf-sched.sch_delay.max.ms.pipe_read.new_sync_read.vfs_read.ksys_read
      0.04 ± 43%    -100.0%       0.00        perf-sched.sch_delay.max.ms.preempt_schedule_common.__cond_resched.stop_one_cpu.affine_move_task.__set_cpus_allowed_ptr
      0.02 ±168%    -100.0%       0.00        perf-sched.sch_delay.max.ms.rcu_gp_kthread.kthread.ret_from_fork
      0.04 ± 80%    -100.0%       0.00        perf-sched.sch_delay.max.ms.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait.__x64_sys_epoll_wait
      0.01 ± 16%    -100.0%       0.00        perf-sched.sch_delay.max.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_select
      0.01 ± 51%    -100.0%       0.00        perf-sched.sch_delay.max.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_sys_poll
      0.02 ± 24%    -100.0%       0.00        perf-sched.sch_delay.max.ms.schedule_timeout.kcompactd.kthread.ret_from_fork
      6.20 ±108%    -100.0%       0.00        perf-sched.sch_delay.max.ms.schedule_timeout.rcu_gp_kthread.kthread.ret_from_fork
      0.01 ± 85%    -100.0%       0.00        perf-sched.sch_delay.max.ms.schedule_timeout.wait_for_completion.__flush_work.lru_add_drain_all
      1.42 ±217%    -100.0%       0.00        perf-sched.sch_delay.max.ms.smpboot_thread_fn.kthread.ret_from_fork
      0.01 ± 51%    -100.0%       0.00        perf-sched.sch_delay.max.ms.wait_for_partner.fifo_open.do_dentry_open.do_open.isra
      3.20          -100.0%       0.00        perf-sched.sch_delay.max.ms.worker_thread.kthread.ret_from_fork
      0.01 ± 56%    -100.0%       0.00        perf-sched.total_sch_delay.average.ms
      8.10 ± 68%    -100.0%       0.00        perf-sched.total_sch_delay.max.ms
    205.13 ±  2%    -100.0%       0.00        perf-sched.total_wait_and_delay.average.ms
      8507 ±  2%    -100.0%       0.00        perf-sched.total_wait_and_delay.count.ms
      8963 ±  5%    -100.0%       0.00        perf-sched.total_wait_and_delay.max.ms
    205.12 ±  2%    -100.0%       0.00        perf-sched.total_wait_time.average.ms
      8963 ±  5%    -100.0%       0.00        perf-sched.total_wait_time.max.ms
    899.67          -100.0%       0.00        perf-sched.wait_and_delay.avg.ms.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown]
      1963 ± 41%    -100.0%       0.00        perf-sched.wait_and_delay.avg.ms.devkmsg_read.vfs_read.ksys_read.do_syscall_64
    833.08 ±  4%    -100.0%       0.00        perf-sched.wait_and_delay.avg.ms.do_nanosleep.hrtimer_nanosleep.__x64_sys_nanosleep.do_syscall_64
      1963 ± 41%    -100.0%       0.00        perf-sched.wait_and_delay.avg.ms.do_syslog.part.0.kmsg_read.vfs_read
    272.32 ±  4%    -100.0%       0.00        perf-sched.wait_and_delay.avg.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
      0.91 ±  4%    -100.0%       0.00        perf-sched.wait_and_delay.avg.ms.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64
     69.10 ±  4%    -100.0%       0.00        perf-sched.wait_and_delay.avg.ms.pipe_read.new_sync_read.vfs_read.ksys_read
      0.02 ±  5%    -100.0%       0.00        perf-sched.wait_and_delay.avg.ms.preempt_schedule_common.__cond_resched.remove_vma.__do_munmap.__vm_munmap
      8.95 ±  8%    -100.0%       0.00        perf-sched.wait_and_delay.avg.ms.preempt_schedule_common.__cond_resched.stop_one_cpu.affine_move_task.__set_cpus_allowed_ptr
      2.66 ±  9%    -100.0%       0.00        perf-sched.wait_and_delay.avg.ms.rcu_gp_kthread.kthread.ret_from_fork
    580.43 ± 25%    -100.0%       0.00        perf-sched.wait_and_delay.avg.ms.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait.__x64_sys_epoll_wait
    519.81 ± 12%    -100.0%       0.00        perf-sched.wait_and_delay.avg.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_sys_poll
    478.81          -100.0%       0.00        perf-sched.wait_and_delay.avg.ms.schedule_timeout.kcompactd.kthread.ret_from_fork
      6.18 ± 11%    -100.0%       0.00        perf-sched.wait_and_delay.avg.ms.schedule_timeout.rcu_gp_kthread.kthread.ret_from_fork
    697.60          -100.0%       0.00        perf-sched.wait_and_delay.avg.ms.smpboot_thread_fn.kthread.ret_from_fork
      0.00 ± 11%    -100.0%       0.00        perf-sched.wait_and_delay.avg.ms.wait_for_partner.fifo_open.do_dentry_open.do_open.isra
    543.97 ±  4%    -100.0%       0.00        perf-sched.wait_and_delay.avg.ms.worker_thread.kthread.ret_from_fork
     10.00          -100.0%       0.00        perf-sched.wait_and_delay.count.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown]
      5.17 ± 36%    -100.0%       0.00        perf-sched.wait_and_delay.count.devkmsg_read.vfs_read.ksys_read.do_syscall_64
     21.67 ±  4%    -100.0%       0.00        perf-sched.wait_and_delay.count.do_nanosleep.hrtimer_nanosleep.__x64_sys_nanosleep.do_syscall_64
      5.17 ± 36%    -100.0%       0.00        perf-sched.wait_and_delay.count.do_syslog.part.0.kmsg_read.vfs_read
    247.00          -100.0%       0.00        perf-sched.wait_and_delay.count.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
    238.33 ±  4%    -100.0%       0.00        perf-sched.wait_and_delay.count.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64
      2342 ±  4%    -100.0%       0.00        perf-sched.wait_and_delay.count.pipe_read.new_sync_read.vfs_read.ksys_read
    265.00 ±  5%    -100.0%       0.00        perf-sched.wait_and_delay.count.preempt_schedule_common.__cond_resched.remove_vma.__do_munmap.__vm_munmap
      1200 ±  4%    -100.0%       0.00        perf-sched.wait_and_delay.count.preempt_schedule_common.__cond_resched.stop_one_cpu.affine_move_task.__set_cpus_allowed_ptr
     63.67 ± 13%    -100.0%       0.00        perf-sched.wait_and_delay.count.rcu_gp_kthread.kthread.ret_from_fork
     29.00 ± 33%    -100.0%       0.00        perf-sched.wait_and_delay.count.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait.__x64_sys_epoll_wait
     64.83 ±  5%    -100.0%       0.00        perf-sched.wait_and_delay.count.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_sys_poll
     40.00          -100.0%       0.00        perf-sched.wait_and_delay.count.schedule_timeout.kcompactd.kthread.ret_from_fork
      1611 ± 11%    -100.0%       0.00        perf-sched.wait_and_delay.count.schedule_timeout.rcu_gp_kthread.kthread.ret_from_fork
      1408 ±  3%    -100.0%       0.00        perf-sched.wait_and_delay.count.smpboot_thread_fn.kthread.ret_from_fork
     72.00          -100.0%       0.00        perf-sched.wait_and_delay.count.wait_for_partner.fifo_open.do_dentry_open.do_open.isra
    726.33 ±  3%    -100.0%       0.00        perf-sched.wait_and_delay.count.worker_thread.kthread.ret_from_fork
    999.73          -100.0%       0.00        perf-sched.wait_and_delay.max.ms.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown]
      5749 ± 44%    -100.0%       0.00        perf-sched.wait_and_delay.max.ms.devkmsg_read.vfs_read.ksys_read.do_syscall_64
      1000          -100.0%       0.00        perf-sched.wait_and_delay.max.ms.do_nanosleep.hrtimer_nanosleep.__x64_sys_nanosleep.do_syscall_64
      5749 ± 44%    -100.0%       0.00        perf-sched.wait_and_delay.max.ms.do_syslog.part.0.kmsg_read.vfs_read
      4683 ± 54%    -100.0%       0.00        perf-sched.wait_and_delay.max.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
     16.38 ±  2%    -100.0%       0.00        perf-sched.wait_and_delay.max.ms.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64
      5751 ± 44%    -100.0%       0.00        perf-sched.wait_and_delay.max.ms.pipe_read.new_sync_read.vfs_read.ksys_read
      0.11 ± 49%    -100.0%       0.00        perf-sched.wait_and_delay.max.ms.preempt_schedule_common.__cond_resched.remove_vma.__do_munmap.__vm_munmap
      1807 ± 44%    -100.0%       0.00        perf-sched.wait_and_delay.max.ms.preempt_schedule_common.__cond_resched.stop_one_cpu.affine_move_task.__set_cpus_allowed_ptr
      4.85 ±  7%    -100.0%       0.00        perf-sched.wait_and_delay.max.ms.rcu_gp_kthread.kthread.ret_from_fork
      5338 ± 48%    -100.0%       0.00        perf-sched.wait_and_delay.max.ms.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait.__x64_sys_epoll_wait
      6078 ± 15%    -100.0%       0.00        perf-sched.wait_and_delay.max.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_sys_poll
    505.00          -100.0%       0.00        perf-sched.wait_and_delay.max.ms.schedule_timeout.kcompactd.kthread.ret_from_fork
    265.84 ± 15%    -100.0%       0.00        perf-sched.wait_and_delay.max.ms.schedule_timeout.rcu_gp_kthread.kthread.ret_from_fork
      1193 ± 31%    -100.0%       0.00        perf-sched.wait_and_delay.max.ms.smpboot_thread_fn.kthread.ret_from_fork
      0.01 ± 51%    -100.0%       0.00        perf-sched.wait_and_delay.max.ms.wait_for_partner.fifo_open.do_dentry_open.do_open.isra
      8783 ±  7%    -100.0%       0.00        perf-sched.wait_and_delay.max.ms.worker_thread.kthread.ret_from_fork
    899.66          -100.0%       0.00        perf-sched.wait_time.avg.ms.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown]
      1963 ± 41%    -100.0%       0.00        perf-sched.wait_time.avg.ms.devkmsg_read.vfs_read.ksys_read.do_syscall_64
    833.07 ±  4%    -100.0%       0.00        perf-sched.wait_time.avg.ms.do_nanosleep.hrtimer_nanosleep.__x64_sys_nanosleep.do_syscall_64
      1963 ± 41%    -100.0%       0.00        perf-sched.wait_time.avg.ms.do_syslog.part.0.kmsg_read.vfs_read
    272.31 ±  4%    -100.0%       0.00        perf-sched.wait_time.avg.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
      0.90 ±  3%    -100.0%       0.00        perf-sched.wait_time.avg.ms.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64
      0.02 ± 32%    -100.0%       0.00        perf-sched.wait_time.avg.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_exc_page_fault.[unknown]
      0.02 ± 19%    -100.0%       0.00        perf-sched.wait_time.avg.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt.[unknown]
      0.08 ±163%    -100.0%       0.00        perf-sched.wait_time.avg.ms.exit_to_user_mode_prepare.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe.[unknown]
     69.09 ±  4%    -100.0%       0.00        perf-sched.wait_time.avg.ms.pipe_read.new_sync_read.vfs_read.ksys_read
      0.02 ± 47%    -100.0%       0.00        perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.__alloc_pages_nodemask.pte_alloc_one.__pte_alloc
      0.02 ± 17%    -100.0%       0.00        perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.__anon_vma_prepare.do_anonymous_page.__handle_mm_fault
     72.45 ±223%    -100.0%       0.00        perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.do_user_addr_fault.exc_page_fault.asm_exc_page_fault
      0.02 ± 72%    -100.0%       0.00        perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.down_write_killable.__vm_munmap.__x64_sys_munmap
      0.02 ± 60%    -100.0%       0.00        perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.down_write_killable.vm_mmap_pgoff.ksys_mmap_pgoff
      0.02 ± 55%    -100.0%       0.00        perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.kmem_cache_alloc.__anon_vma_prepare.do_anonymous_page
      0.01 ± 58%    -100.0%       0.00        perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.kmem_cache_alloc.vm_area_alloc.mmap_region
      0.02 ±  5%    -100.0%       0.00        perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.remove_vma.__do_munmap.__vm_munmap
      8.95 ±  8%    -100.0%       0.00        perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.stop_one_cpu.affine_move_task.__set_cpus_allowed_ptr
      0.02 ± 19%    -100.0%       0.00        perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.unmap_page_range.unmap_vmas.unmap_region
      0.02 ± 48%    -100.0%       0.00        perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.unmap_vmas.unmap_region.__do_munmap
      0.01 ± 63%    -100.0%       0.00        perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.zap_pte_range.unmap_page_range.unmap_vmas
      2.64 ±  9%    -100.0%       0.00        perf-sched.wait_time.avg.ms.rcu_gp_kthread.kthread.ret_from_fork
    580.43 ± 25%    -100.0%       0.00        perf-sched.wait_time.avg.ms.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait.__x64_sys_epoll_wait
    519.81 ± 12%    -100.0%       0.00        perf-sched.wait_time.avg.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_sys_poll
    478.80          -100.0%       0.00        perf-sched.wait_time.avg.ms.schedule_timeout.kcompactd.kthread.ret_from_fork
      0.03 ±100%    -100.0%       0.00        perf-sched.wait_time.avg.ms.schedule_timeout.khugepaged.kthread.ret_from_fork
      6.17 ± 11%    -100.0%       0.00        perf-sched.wait_time.avg.ms.schedule_timeout.rcu_gp_kthread.kthread.ret_from_fork
    697.59          -100.0%       0.00        perf-sched.wait_time.avg.ms.smpboot_thread_fn.kthread.ret_from_fork
    543.95 ±  4%    -100.0%       0.00        perf-sched.wait_time.avg.ms.worker_thread.kthread.ret_from_fork
    999.72          -100.0%       0.00        perf-sched.wait_time.max.ms.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown]
      5749 ± 44%    -100.0%       0.00        perf-sched.wait_time.max.ms.devkmsg_read.vfs_read.ksys_read.do_syscall_64
      1000          -100.0%       0.00        perf-sched.wait_time.max.ms.do_nanosleep.hrtimer_nanosleep.__x64_sys_nanosleep.do_syscall_64
      5749 ± 44%    -100.0%       0.00        perf-sched.wait_time.max.ms.do_syslog.part.0.kmsg_read.vfs_read
      4683 ± 54%    -100.0%       0.00        perf-sched.wait_time.max.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
     16.38 ±  2%    -100.0%       0.00        perf-sched.wait_time.max.ms.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64
      0.04 ± 11%    -100.0%       0.00        perf-sched.wait_time.max.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_exc_page_fault.[unknown]
      0.07 ± 57%    -100.0%       0.00        perf-sched.wait_time.max.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt.[unknown]
      1.22 ±208%    -100.0%       0.00        perf-sched.wait_time.max.ms.exit_to_user_mode_prepare.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe.[unknown]
      5751 ± 44%    -100.0%       0.00        perf-sched.wait_time.max.ms.pipe_read.new_sync_read.vfs_read.ksys_read
      0.04 ± 41%    -100.0%       0.00        perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.__alloc_pages_nodemask.pte_alloc_one.__pte_alloc
      0.07 ± 65%    -100.0%       0.00        perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.__anon_vma_prepare.do_anonymous_page.__handle_mm_fault
      1086 ±223%    -100.0%       0.00        perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.do_user_addr_fault.exc_page_fault.asm_exc_page_fault
      0.05 ±112%    -100.0%       0.00        perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.down_write_killable.__vm_munmap.__x64_sys_munmap
      0.06 ± 83%    -100.0%       0.00        perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.down_write_killable.vm_mmap_pgoff.ksys_mmap_pgoff
      0.03 ± 37%    -100.0%       0.00        perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.kmem_cache_alloc.__anon_vma_prepare.do_anonymous_page
      0.03 ± 59%    -100.0%       0.00        perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.kmem_cache_alloc.vm_area_alloc.mmap_region
      0.11 ± 49%    -100.0%       0.00        perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.remove_vma.__do_munmap.__vm_munmap
      1807 ± 44%    -100.0%       0.00        perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.stop_one_cpu.affine_move_task.__set_cpus_allowed_ptr
      0.08 ± 59%    -100.0%       0.00        perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.unmap_page_range.unmap_vmas.unmap_region
      0.09 ± 70%    -100.0%       0.00        perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.unmap_vmas.unmap_region.__do_munmap
      0.02 ± 75%    -100.0%       0.00        perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.zap_pte_range.unmap_page_range.unmap_vmas
      4.83 ±  7%    -100.0%       0.00        perf-sched.wait_time.max.ms.rcu_gp_kthread.kthread.ret_from_fork
      5338 ± 48%    -100.0%       0.00        perf-sched.wait_time.max.ms.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait.__x64_sys_epoll_wait
      6078 ± 15%    -100.0%       0.00        perf-sched.wait_time.max.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_sys_poll
    505.00          -100.0%       0.00        perf-sched.wait_time.max.ms.schedule_timeout.kcompactd.kthread.ret_from_fork
      0.03 ±100%    -100.0%       0.00        perf-sched.wait_time.max.ms.schedule_timeout.khugepaged.kthread.ret_from_fork
    265.82 ± 15%    -100.0%       0.00        perf-sched.wait_time.max.ms.schedule_timeout.rcu_gp_kthread.kthread.ret_from_fork
      1193 ± 31%    -100.0%       0.00        perf-sched.wait_time.max.ms.smpboot_thread_fn.kthread.ret_from_fork
      8783 ±  7%    -100.0%       0.00        perf-sched.wait_time.max.ms.worker_thread.kthread.ret_from_fork
    676.83 ±104%    +927.6%       6954 ±136%  interrupts.39:PCI-MSI.67633154-edge.eth0-TxRx-1
      5953 ± 35%    -100.0%       0.00        interrupts.CPU0.NMI:Non-maskable_interrupts
      5953 ± 35%    -100.0%       0.00        interrupts.CPU0.PMI:Performance_monitoring_interrupts
      6263 ± 21%    -100.0%       0.00        interrupts.CPU1.NMI:Non-maskable_interrupts
      6263 ± 21%    -100.0%       0.00        interrupts.CPU1.PMI:Performance_monitoring_interrupts
      6259 ± 36%    -100.0%       0.00        interrupts.CPU10.NMI:Non-maskable_interrupts
      6259 ± 36%    -100.0%       0.00        interrupts.CPU10.PMI:Performance_monitoring_interrupts
    101.00 ± 33%    -100.0%       0.00        interrupts.CPU100.NMI:Non-maskable_interrupts
    101.00 ± 33%    -100.0%       0.00        interrupts.CPU100.PMI:Performance_monitoring_interrupts
    124.33 ± 52%    -100.0%       0.00        interrupts.CPU101.NMI:Non-maskable_interrupts
    124.33 ± 52%    -100.0%       0.00        interrupts.CPU101.PMI:Performance_monitoring_interrupts
    122.67 ± 64%    -100.0%       0.00        interrupts.CPU102.NMI:Non-maskable_interrupts
    122.67 ± 64%    -100.0%       0.00        interrupts.CPU102.PMI:Performance_monitoring_interrupts
    418.33 ± 32%    -100.0%       0.00        interrupts.CPU103.NMI:Non-maskable_interrupts
    418.33 ± 32%    -100.0%       0.00        interrupts.CPU103.PMI:Performance_monitoring_interrupts
      5750 ± 38%    -100.0%       0.00        interrupts.CPU11.NMI:Non-maskable_interrupts
      5750 ± 38%    -100.0%       0.00        interrupts.CPU11.PMI:Performance_monitoring_interrupts
      5182 ± 53%    -100.0%       0.00        interrupts.CPU12.NMI:Non-maskable_interrupts
      5182 ± 53%    -100.0%       0.00        interrupts.CPU12.PMI:Performance_monitoring_interrupts
      4527 ± 38%    -100.0%       0.00        interrupts.CPU13.NMI:Non-maskable_interrupts
      4527 ± 38%    -100.0%       0.00        interrupts.CPU13.PMI:Performance_monitoring_interrupts
      5809 ± 45%    -100.0%       0.00        interrupts.CPU14.NMI:Non-maskable_interrupts
      5809 ± 45%    -100.0%       0.00        interrupts.CPU14.PMI:Performance_monitoring_interrupts
      4868 ± 41%    -100.0%       0.00        interrupts.CPU15.NMI:Non-maskable_interrupts
      4868 ± 41%    -100.0%       0.00        interrupts.CPU15.PMI:Performance_monitoring_interrupts
      2.00 ±119%   +4000.0%      82.00 ±131%  interrupts.CPU19.RES:Rescheduling_interrupts
      4642 ± 50%    -100.0%       0.00        interrupts.CPU2.NMI:Non-maskable_interrupts
      4642 ± 50%    -100.0%       0.00        interrupts.CPU2.PMI:Performance_monitoring_interrupts
      1.00 ±115%   +7700.0%      78.00 ±138%  interrupts.CPU20.RES:Rescheduling_interrupts
     75.83 ± 29%    -100.0%       0.00        interrupts.CPU22.NMI:Non-maskable_interrupts
     75.83 ± 29%    -100.0%       0.00        interrupts.CPU22.PMI:Performance_monitoring_interrupts
     77.83 ± 25%    -100.0%       0.00        interrupts.CPU24.NMI:Non-maskable_interrupts
     77.83 ± 25%    -100.0%       0.00        interrupts.CPU24.PMI:Performance_monitoring_interrupts
    121.00 ± 41%    -100.0%       0.00        interrupts.CPU26.NMI:Non-maskable_interrupts
    121.00 ± 41%    -100.0%       0.00        interrupts.CPU26.PMI:Performance_monitoring_interrupts
    193.50 ± 94%    -100.0%       0.00        interrupts.CPU27.NMI:Non-maskable_interrupts
    193.50 ± 94%    -100.0%       0.00        interrupts.CPU27.PMI:Performance_monitoring_interrupts
     99.00 ± 35%    -100.0%       0.00        interrupts.CPU28.NMI:Non-maskable_interrupts
     99.00 ± 35%    -100.0%       0.00        interrupts.CPU28.PMI:Performance_monitoring_interrupts
    191.33 ±108%    -100.0%       0.00        interrupts.CPU29.NMI:Non-maskable_interrupts
    191.33 ±108%    -100.0%       0.00        interrupts.CPU29.PMI:Performance_monitoring_interrupts
      4235 ± 46%    -100.0%       0.00        interrupts.CPU3.NMI:Non-maskable_interrupts
      4235 ± 46%    -100.0%       0.00        interrupts.CPU3.PMI:Performance_monitoring_interrupts
    195.33 ±116%    -100.0%       0.00        interrupts.CPU30.NMI:Non-maskable_interrupts
    195.33 ±116%    -100.0%       0.00        interrupts.CPU30.PMI:Performance_monitoring_interrupts
    676.83 ±104%    +927.6%       6954 ±136%  interrupts.CPU31.39:PCI-MSI.67633154-edge.eth0-TxRx-1
     95.50 ± 34%    -100.0%       0.00        interrupts.CPU31.NMI:Non-maskable_interrupts
     95.50 ± 34%    -100.0%       0.00        interrupts.CPU31.PMI:Performance_monitoring_interrupts
    215.50 ±111%    -100.0%       0.00        interrupts.CPU32.NMI:Non-maskable_interrupts
    215.50 ±111%    -100.0%       0.00        interrupts.CPU32.PMI:Performance_monitoring_interrupts
    548.33 ±  5%     +62.7%     892.17 ± 30%  interrupts.CPU33.CAL:Function_call_interrupts
    106.50 ± 29%    -100.0%       0.00        interrupts.CPU33.NMI:Non-maskable_interrupts
    106.50 ± 29%    -100.0%       0.00        interrupts.CPU33.PMI:Performance_monitoring_interrupts
     98.50 ± 36%    -100.0%       0.00        interrupts.CPU34.NMI:Non-maskable_interrupts
     98.50 ± 36%    -100.0%       0.00        interrupts.CPU34.PMI:Performance_monitoring_interrupts
     97.50 ± 41%    -100.0%       0.00        interrupts.CPU35.NMI:Non-maskable_interrupts
     97.50 ± 41%    -100.0%       0.00        interrupts.CPU35.PMI:Performance_monitoring_interrupts
    179.00 ±102%    -100.0%       0.00        interrupts.CPU36.NMI:Non-maskable_interrupts
    179.00 ±102%    -100.0%       0.00        interrupts.CPU36.PMI:Performance_monitoring_interrupts
    108.50 ± 23%    -100.0%       0.00        interrupts.CPU37.NMI:Non-maskable_interrupts
    108.50 ± 23%    -100.0%       0.00        interrupts.CPU37.PMI:Performance_monitoring_interrupts
    103.00 ± 28%    -100.0%       0.00        interrupts.CPU38.NMI:Non-maskable_interrupts
    103.00 ± 28%    -100.0%       0.00        interrupts.CPU38.PMI:Performance_monitoring_interrupts
     93.50 ± 33%    -100.0%       0.00        interrupts.CPU39.NMI:Non-maskable_interrupts
     93.50 ± 33%    -100.0%       0.00        interrupts.CPU39.PMI:Performance_monitoring_interrupts
      3954 ± 46%    -100.0%       0.00        interrupts.CPU4.NMI:Non-maskable_interrupts
      3954 ± 46%    -100.0%       0.00        interrupts.CPU4.PMI:Performance_monitoring_interrupts
     93.67 ± 31%    -100.0%       0.00        interrupts.CPU40.NMI:Non-maskable_interrupts
     93.67 ± 31%    -100.0%       0.00        interrupts.CPU40.PMI:Performance_monitoring_interrupts
    102.33 ± 43%    -100.0%       0.00        interrupts.CPU41.NMI:Non-maskable_interrupts
    102.33 ± 43%    -100.0%       0.00        interrupts.CPU41.PMI:Performance_monitoring_interrupts
     93.00 ± 29%    -100.0%       0.00        interrupts.CPU42.NMI:Non-maskable_interrupts
     93.00 ± 29%    -100.0%       0.00        interrupts.CPU42.PMI:Performance_monitoring_interrupts
     99.83 ± 33%    -100.0%       0.00        interrupts.CPU43.NMI:Non-maskable_interrupts
     99.83 ± 33%    -100.0%       0.00        interrupts.CPU43.PMI:Performance_monitoring_interrupts
    124.83 ± 38%    -100.0%       0.00        interrupts.CPU44.NMI:Non-maskable_interrupts
    124.83 ± 38%    -100.0%       0.00        interrupts.CPU44.PMI:Performance_monitoring_interrupts
    101.83 ± 37%    -100.0%       0.00        interrupts.CPU45.NMI:Non-maskable_interrupts
    101.83 ± 37%    -100.0%       0.00        interrupts.CPU45.PMI:Performance_monitoring_interrupts
    102.00 ± 41%    -100.0%       0.00        interrupts.CPU46.NMI:Non-maskable_interrupts
    102.00 ± 41%    -100.0%       0.00        interrupts.CPU46.PMI:Performance_monitoring_interrupts
     96.50 ± 34%    -100.0%       0.00        interrupts.CPU47.NMI:Non-maskable_interrupts
     96.50 ± 34%    -100.0%       0.00        interrupts.CPU47.PMI:Performance_monitoring_interrupts
    105.33 ± 28%    -100.0%       0.00        interrupts.CPU48.NMI:Non-maskable_interrupts
    105.33 ± 28%    -100.0%       0.00        interrupts.CPU48.PMI:Performance_monitoring_interrupts
    139.67 ± 56%    -100.0%       0.00        interrupts.CPU49.NMI:Non-maskable_interrupts
    139.67 ± 56%    -100.0%       0.00        interrupts.CPU49.PMI:Performance_monitoring_interrupts
      4293 ± 48%    -100.0%       0.00        interrupts.CPU5.NMI:Non-maskable_interrupts
      4293 ± 48%    -100.0%       0.00        interrupts.CPU5.PMI:Performance_monitoring_interrupts
    173.17 ±113%    -100.0%       0.00        interrupts.CPU50.NMI:Non-maskable_interrupts
    173.17 ±113%    -100.0%       0.00        interrupts.CPU50.PMI:Performance_monitoring_interrupts
    270.33 ± 28%    -100.0%       0.00        interrupts.CPU51.NMI:Non-maskable_interrupts
    270.33 ± 28%    -100.0%       0.00        interrupts.CPU51.PMI:Performance_monitoring_interrupts
      5508 ± 39%    -100.0%       0.00        interrupts.CPU52.NMI:Non-maskable_interrupts
      5508 ± 39%    -100.0%       0.00        interrupts.CPU52.PMI:Performance_monitoring_interrupts
      4786 ± 40%    -100.0%       0.00        interrupts.CPU53.NMI:Non-maskable_interrupts
      4786 ± 40%    -100.0%       0.00        interrupts.CPU53.PMI:Performance_monitoring_interrupts
      6828 ± 17%    -100.0%       0.00        interrupts.CPU54.NMI:Non-maskable_interrupts
      6828 ± 17%    -100.0%       0.00        interrupts.CPU54.PMI:Performance_monitoring_interrupts
      6512 ± 24%    -100.0%       0.00        interrupts.CPU55.NMI:Non-maskable_interrupts
      6512 ± 24%    -100.0%       0.00        interrupts.CPU55.PMI:Performance_monitoring_interrupts
      6860 ± 26%    -100.0%       1.00 ±223%  interrupts.CPU56.NMI:Non-maskable_interrupts
      6860 ± 26%    -100.0%       1.00 ±223%  interrupts.CPU56.PMI:Performance_monitoring_interrupts
      6435 ± 27%    -100.0%       0.00        interrupts.CPU57.NMI:Non-maskable_interrupts
      6435 ± 27%    -100.0%       0.00        interrupts.CPU57.PMI:Performance_monitoring_interrupts
      5567 ± 36%    -100.0%       0.00        interrupts.CPU58.NMI:Non-maskable_interrupts
      5567 ± 36%    -100.0%       0.00        interrupts.CPU58.PMI:Performance_monitoring_interrupts
      5703 ± 39%    -100.0%       0.00        interrupts.CPU59.NMI:Non-maskable_interrupts
      5703 ± 39%    -100.0%       0.00        interrupts.CPU59.PMI:Performance_monitoring_interrupts
      5098 ± 43%    -100.0%       0.00        interrupts.CPU6.NMI:Non-maskable_interrupts
      5098 ± 43%    -100.0%       0.00        interrupts.CPU6.PMI:Performance_monitoring_interrupts
      6022 ± 27%    -100.0%       0.00        interrupts.CPU60.NMI:Non-maskable_interrupts
      6022 ± 27%    -100.0%       0.00        interrupts.CPU60.PMI:Performance_monitoring_interrupts
      5896 ± 37%    -100.0%       0.00        interrupts.CPU61.NMI:Non-maskable_interrupts
      5896 ± 37%    -100.0%       0.00        interrupts.CPU61.PMI:Performance_monitoring_interrupts
      5451 ± 36%    -100.0%       0.00        interrupts.CPU62.NMI:Non-maskable_interrupts
      5451 ± 36%    -100.0%       0.00        interrupts.CPU62.PMI:Performance_monitoring_interrupts
      5960 ± 27%    -100.0%       0.00        interrupts.CPU63.NMI:Non-maskable_interrupts
      5960 ± 27%    -100.0%       0.00        interrupts.CPU63.PMI:Performance_monitoring_interrupts
      6291 ± 28%    -100.0%       0.00        interrupts.CPU64.NMI:Non-maskable_interrupts
      6291 ± 28%    -100.0%       0.00        interrupts.CPU64.PMI:Performance_monitoring_interrupts
      5867 ± 38%    -100.0%       0.00        interrupts.CPU65.NMI:Non-maskable_interrupts
      5867 ± 38%    -100.0%       0.00        interrupts.CPU65.PMI:Performance_monitoring_interrupts
      5665 ± 33%    -100.0%       0.00        interrupts.CPU66.NMI:Non-maskable_interrupts
      5665 ± 33%    -100.0%       0.00        interrupts.CPU66.PMI:Performance_monitoring_interrupts
      5844 ± 27%    -100.0%       1.00 ±223%  interrupts.CPU67.NMI:Non-maskable_interrupts
      5844 ± 27%    -100.0%       1.00 ±223%  interrupts.CPU67.PMI:Performance_monitoring_interrupts
      5506 ± 36%    -100.0%       0.00        interrupts.CPU7.NMI:Non-maskable_interrupts
      5506 ± 36%    -100.0%       0.00        interrupts.CPU7.PMI:Performance_monitoring_interrupts
     76.50 ± 28%    -100.0%       0.00        interrupts.CPU70.NMI:Non-maskable_interrupts
     76.50 ± 28%    -100.0%       0.00        interrupts.CPU70.PMI:Performance_monitoring_interrupts
     77.50 ± 21%    -100.0%       0.00        interrupts.CPU71.NMI:Non-maskable_interrupts
     77.50 ± 21%    -100.0%       0.00        interrupts.CPU71.PMI:Performance_monitoring_interrupts
    101.17 ± 52%    -100.0%       0.00        interrupts.CPU74.NMI:Non-maskable_interrupts
    101.17 ± 52%    -100.0%       0.00        interrupts.CPU74.PMI:Performance_monitoring_interrupts
     79.83 ± 26%    -100.0%       0.00        interrupts.CPU75.NMI:Non-maskable_interrupts
     79.83 ± 26%    -100.0%       0.00        interrupts.CPU75.PMI:Performance_monitoring_interrupts
     83.33 ± 12%    -100.0%       0.00        interrupts.CPU76.NMI:Non-maskable_interrupts
     83.33 ± 12%    -100.0%       0.00        interrupts.CPU76.PMI:Performance_monitoring_interrupts
    126.83 ± 30%    -100.0%       0.00        interrupts.CPU78.NMI:Non-maskable_interrupts
    126.83 ± 30%    -100.0%       0.00        interrupts.CPU78.PMI:Performance_monitoring_interrupts
    164.33 ± 77%    -100.0%       0.00        interrupts.CPU79.NMI:Non-maskable_interrupts
    164.33 ± 77%    -100.0%       0.00        interrupts.CPU79.PMI:Performance_monitoring_interrupts
      5231 ± 44%    -100.0%       0.00        interrupts.CPU8.NMI:Non-maskable_interrupts
      5231 ± 44%    -100.0%       0.00        interrupts.CPU8.PMI:Performance_monitoring_interrupts
     98.50 ± 47%    -100.0%       0.00        interrupts.CPU80.NMI:Non-maskable_interrupts
     98.50 ± 47%    -100.0%       0.00        interrupts.CPU80.PMI:Performance_monitoring_interrupts
    184.83 ± 91%    -100.0%       0.00        interrupts.CPU81.NMI:Non-maskable_interrupts
    184.83 ± 91%    -100.0%       0.00        interrupts.CPU81.PMI:Performance_monitoring_interrupts
    144.50 ± 78%    -100.0%       0.00        interrupts.CPU82.NMI:Non-maskable_interrupts
    144.50 ± 78%    -100.0%       0.00        interrupts.CPU82.PMI:Performance_monitoring_interrupts
    102.83 ± 40%    -100.0%       0.00        interrupts.CPU83.NMI:Non-maskable_interrupts
    102.83 ± 40%    -100.0%       0.00        interrupts.CPU83.PMI:Performance_monitoring_interrupts
    138.50 ± 75%    -100.0%       0.00        interrupts.CPU84.NMI:Non-maskable_interrupts
    138.50 ± 75%    -100.0%       0.00        interrupts.CPU84.PMI:Performance_monitoring_interrupts
     99.17 ± 49%    -100.0%       0.00        interrupts.CPU85.NMI:Non-maskable_interrupts
     99.17 ± 49%    -100.0%       0.00        interrupts.CPU85.PMI:Performance_monitoring_interrupts
     92.33 ± 46%    -100.0%       0.00        interrupts.CPU86.NMI:Non-maskable_interrupts
     92.33 ± 46%    -100.0%       0.00        interrupts.CPU86.PMI:Performance_monitoring_interrupts
     98.67 ± 53%    -100.0%       0.00        interrupts.CPU87.NMI:Non-maskable_interrupts
     98.67 ± 53%    -100.0%       0.00        interrupts.CPU87.PMI:Performance_monitoring_interrupts
    143.50 ± 74%    -100.0%       0.00        interrupts.CPU88.NMI:Non-maskable_interrupts
    143.50 ± 74%    -100.0%       0.00        interrupts.CPU88.PMI:Performance_monitoring_interrupts
     97.50 ± 34%    -100.0%       0.00        interrupts.CPU89.NMI:Non-maskable_interrupts
     97.50 ± 34%    -100.0%       0.00        interrupts.CPU89.PMI:Performance_monitoring_interrupts
      5814 ± 35%    -100.0%       0.00        interrupts.CPU9.NMI:Non-maskable_interrupts
      5814 ± 35%    -100.0%       0.00        interrupts.CPU9.PMI:Performance_monitoring_interrupts
     94.67 ± 37%    -100.0%       0.00        interrupts.CPU90.NMI:Non-maskable_interrupts
     94.67 ± 37%    -100.0%       0.00        interrupts.CPU90.PMI:Performance_monitoring_interrupts
     98.33 ± 38%    -100.0%       0.00        interrupts.CPU91.NMI:Non-maskable_interrupts
     98.33 ± 38%    -100.0%       0.00        interrupts.CPU91.PMI:Performance_monitoring_interrupts
     98.33 ± 35%    -100.0%       0.00        interrupts.CPU92.NMI:Non-maskable_interrupts
     98.33 ± 35%    -100.0%       0.00        interrupts.CPU92.PMI:Performance_monitoring_interrupts
     98.67 ± 37%    -100.0%       0.00        interrupts.CPU93.NMI:Non-maskable_interrupts
     98.67 ± 37%    -100.0%       0.00        interrupts.CPU93.PMI:Performance_monitoring_interrupts
    100.33 ± 25%    -100.0%       0.00        interrupts.CPU94.NMI:Non-maskable_interrupts
    100.33 ± 25%    -100.0%       0.00        interrupts.CPU94.PMI:Performance_monitoring_interrupts
    122.17 ± 20%    -100.0%       0.00        interrupts.CPU95.NMI:Non-maskable_interrupts
    122.17 ± 20%    -100.0%       0.00        interrupts.CPU95.PMI:Performance_monitoring_interrupts
    128.00 ± 44%    -100.0%       0.00        interrupts.CPU96.NMI:Non-maskable_interrupts
    128.00 ± 44%    -100.0%       0.00        interrupts.CPU96.PMI:Performance_monitoring_interrupts
    120.33 ± 40%     -99.2%       1.00 ±223%  interrupts.CPU97.NMI:Non-maskable_interrupts
    120.33 ± 40%     -99.2%       1.00 ±223%  interrupts.CPU97.PMI:Performance_monitoring_interrupts
    100.50 ± 38%    -100.0%       0.00        interrupts.CPU98.NMI:Non-maskable_interrupts
    100.50 ± 38%    -100.0%       0.00        interrupts.CPU98.PMI:Performance_monitoring_interrupts
     97.33 ± 36%    -100.0%       0.00        interrupts.CPU99.NMI:Non-maskable_interrupts
     97.33 ± 36%    -100.0%       0.00        interrupts.CPU99.PMI:Performance_monitoring_interrupts
    186821 ± 14%    -100.0%       4.00 ± 70%  interrupts.NMI:Non-maskable_interrupts
    186821 ± 14%    -100.0%       4.00 ± 70%  interrupts.PMI:Performance_monitoring_interrupts

***************************************************************************************************
lkp-skl-fpga01: 104 threads Skylake with 192G memory
=========================================================================================
compiler/cpufreq_governor/kconfig/mode/nr_task/rootfs/tbox_group/test/testcase/ucode:
gcc-9/performance/x86_64-rhel-8.3/process/100%/debian-10.4-x86_64-20200603.cgz/lkp-skl-fpga01/eventfd1/will-it-scale/0x2006a0a

commit:
dde38d5b80 ("kernel: don't include PF_IO_WORKERs as part of same_thread_group()") 43b2a76b1a ("proc: don't show PF_IO_WORKER threads as threads in /proc/<pid>/task/")

dde38d5b805358c7 43b2a76b1a5abcc9833b463bef1
---------------- ---------------------------
%stddev %change %stddev
\ | \

  53817815            -6.3%   50434581        will-it-scale.104.processes
    517478            -6.3%     484947        will-it-scale.per_process_ops
  53817815            -6.3%   50434581        will-it-scale.workload
      3.61 ±  4%      +2.8        6.38        mpstat.cpu.all.idle%
      0.02 ± 15%      +0.0        0.03        mpstat.cpu.all.soft%
   1335406 ±  6%     +43.6%    1917517 ± 15%  numa-meminfo.node0.MemUsed
     26350 ±  8%     -32.8%      17695 ± 18%  numa-meminfo.node1.Mapped
    351261 ±  8%     +91.6%     672866 ± 13%  numa-numastat.node0.local_node
    396995           +84.4%     731886 ± 12%  numa-numastat.node0.numa_hit
    965410 ±  9%     +23.1%    1188249 ±  7%  numa-vmstat.node0.numa_hit
      6639 ±  8%     -32.7%       4466 ± 18%  numa-vmstat.node1.nr_mapped
  14273927 ± 17%    +200.1%   42839522 ±114%  cpuidle.C6.time
     46336 ±  7%     +63.0%      75510 ± 68%  cpuidle.C6.usage
     12753 ±  6%    +100.7%      25591 ± 73%  cpuidle.POLL.time
     55.00            -3.6%      53.00        vmstat.cpu.sy
     43.00            +4.7%      45.00        vmstat.cpu.us
   1537032           +38.4%    2127542        vmstat.memory.cache
      1502 ±  5%     +40.3%       2107        vmstat.system.cs
    212844            -2.4%     207815        vmstat.system.in
      8605 ±  2%     +13.3%       9747 ±  3%  slabinfo.kmalloc-256.active_objs
      8654 ±  2%     +13.2%       9795 ±  3%  slabinfo.kmalloc-256.num_objs
     14367           -37.7%       8957        slabinfo.proc_inode_cache.active_objs
    299.50           -38.0%     185.83        slabinfo.proc_inode_cache.active_slabs
     14395           -37.8%       8957        slabinfo.proc_inode_cache.num_objs
    299.50           -38.0%     185.83        slabinfo.proc_inode_cache.num_slabs
    224400          +269.5%     829186 ±  2%  meminfo.Active
    223868          +270.3%     828946 ±  2%  meminfo.Active(anon)
   1437431           +41.2%    2029667        meminfo.Cached
   2408715           +27.3%    3065135        meminfo.Committed_AS
     51176           -26.0%      37863        meminfo.Mapped
   2783939           +35.1%    3760068        meminfo.Memused
    251360          +237.0%     846957 ±  2%  meminfo.Shmem
   3254687           +15.5%    3760328        meminfo.max_used_kB
    217046 ±  5%    +122.4%     482701 ± 45%  sched_debug.cfs_rq:/.min_vruntime.stddev
      0.27 ±122%    +481.4%       1.57 ± 44%  sched_debug.cfs_rq:/.removed.runnable_avg.avg
      2.64 ±122%    +303.8%      10.65 ± 19%  sched_debug.cfs_rq:/.removed.runnable_avg.stddev
      0.27 ±122%    +481.4%       1.57 ± 44%  sched_debug.cfs_rq:/.removed.util_avg.avg
      2.64 ±122%    +303.8%      10.65 ± 19%  sched_debug.cfs_rq:/.removed.util_avg.stddev
      1499 ±  4%     +30.4%       1955 ±  4%  sched_debug.cfs_rq:/.runnable_avg.max
     95.35 ±  7%     +87.9%     179.14 ±  7%  sched_debug.cfs_rq:/.runnable_avg.stddev
    216964 ±  5%    +122.5%     482737 ± 45%  sched_debug.cfs_rq:/.spread0.stddev
      1093 ±  3%     +19.2%       1304 ±  7%  sched_debug.cfs_rq:/.util_avg.max
    796.11 ±  5%     -25.3%     594.42 ±  6%  sched_debug.cfs_rq:/.util_avg.min
     56.73 ± 10%     +62.3%      92.05 ±  8%  sched_debug.cfs_rq:/.util_avg.stddev
    668.86 ±  4%     +90.2%       1272 ±  2%  sched_debug.cfs_rq:/.util_est_enqueued.max
     78.80 ±  3%     +74.1%     137.19 ±  3%  sched_debug.cfs_rq:/.util_est_enqueued.stddev
     11.98 ±  5%     +26.0%      15.09 ±  6%  sched_debug.cpu.clock.stddev
      0.16 ± 11%     +27.3%       0.20 ±  6%  sched_debug.cpu.nr_running.stddev
      3824 ±  3%     +22.4%       4681        sched_debug.cpu.nr_switches.avg
      1470 ±  4%     +34.5%       1978 ± 12%  sched_debug.cpu.nr_switches.min
     55875          +271.0%     207304 ±  2%  proc-vmstat.nr_active_anon
     70608            +6.7%      75350        proc-vmstat.nr_anon_pages
    359234           +41.3%     507487        proc-vmstat.nr_file_pages
     77322            +3.1%      79694        proc-vmstat.nr_inactive_anon
     12886           -25.6%       9591        proc-vmstat.nr_mapped
      2518            +4.6%       2634        proc-vmstat.nr_page_table_pages
     62716          +237.7%     211808 ±  2%  proc-vmstat.nr_shmem
     24871            -4.3%      23800        proc-vmstat.nr_slab_reclaimable
     55875          +271.0%     207304 ±  2%  proc-vmstat.nr_zone_active_anon
     77322            +3.1%      79694        proc-vmstat.nr_zone_inactive_anon
     14803 ± 29%     -62.5%       5557 ± 62%  proc-vmstat.numa_hint_faults
     13291 ± 28%     -78.1%       2913 ± 64%  proc-vmstat.numa_hint_faults_local
    943105           +45.2%    1369352        proc-vmstat.numa_hit
    849367           +50.2%    1275624 ±  2%  proc-vmstat.numa_local
    148542 ± 12%     -64.5%      52803 ± 31%  proc-vmstat.numa_pte_updates
     86501           -67.2%      28371 ±  2%  proc-vmstat.pgactivate
    993722           +55.6%    1546560 ±  2%  proc-vmstat.pgalloc_normal
    977248           -12.4%     856153        proc-vmstat.pgfault
    883680            +7.2%     947019 ±  4%  proc-vmstat.pgfree
     55.30           -55.3        0.00        perf-profile.calltrace.cycles-pp.read
     45.47           -45.5        0.00        perf-profile.calltrace.cycles-pp.write
     32.09           -32.1        0.00        perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.read
     25.67           -25.7        0.00        perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.write
     18.87           -18.9        0.00        perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.read
     17.08           -17.1        0.00        perf-profile.calltrace.cycles-pp.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe.read
     15.43           -15.4        0.00        perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_safe_stack.read
     13.05           -13.0        0.00        perf-profile.calltrace.cycles-pp.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe.write
     12.95           -12.9        0.00        perf-profile.calltrace.cycles-pp.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe.read
     12.64           -12.6        0.00        perf-profile.calltrace.cycles-pp.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe.read
     12.10           -12.1        0.00        perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.write
     10.85           -10.8        0.00        perf-profile.calltrace.cycles-pp.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.write
      9.56            -9.6        0.00        perf-profile.calltrace.cycles-pp.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.write
      8.90            -8.9        0.00        perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_safe_stack.write
      8.07            -8.1        0.00        perf-profile.calltrace.cycles-pp.new_sync_read.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe
      7.57            -7.6        0.00        perf-profile.calltrace.cycles-pp.syscall_return_via_sysret.write
      7.48            -7.5        0.00        perf-profile.calltrace.cycles-pp.syscall_return_via_sysret.read
      6.75            -6.8        0.00        perf-profile.calltrace.cycles-pp.eventfd_read.new_sync_read.vfs_read.ksys_read.do_syscall_64
      6.32            -6.3        0.00        perf-profile.calltrace.cycles-pp.__entry_text_start.read
      6.30            -6.3        0.00        perf-profile.calltrace.cycles-pp.__entry_text_start.write
      5.92            -5.9        0.00        perf-profile.calltrace.cycles-pp.eventfd_write.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
     57.92           -57.9        0.00        perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
     55.52           -55.5        0.00        perf-profile.children.cycles-pp.read
     45.70           -45.7        0.00        perf-profile.children.cycles-pp.write
     31.08           -31.1        0.00        perf-profile.children.cycles-pp.do_syscall_64
     25.84           -25.8        0.00        perf-profile.children.cycles-pp.syscall_exit_to_user_mode
     22.59           -22.6        0.00        perf-profile.children.cycles-pp.__entry_text_start
     17.14           -17.1        0.00        perf-profile.children.cycles-pp.ksys_read
     16.66           -16.7        0.00        perf-profile.children.cycles-pp.syscall_return_via_sysret
     13.08           -13.1        0.00        perf-profile.children.cycles-pp.vfs_read
     12.68           -12.7        0.00        perf-profile.children.cycles-pp.entry_SYSCALL_64_safe_stack
     10.91           -10.9        0.00        perf-profile.children.cycles-pp.ksys_write
      9.69            -9.7        0.00        perf-profile.children.cycles-pp.vfs_write
      8.13            -8.1        0.00        perf-profile.children.cycles-pp.new_sync_read
      6.96            -7.0        0.00        perf-profile.children.cycles-pp.eventfd_read
      6.05            -6.1        0.00        perf-profile.children.cycles-pp.eventfd_write
      5.20            -5.2        0.00        perf-profile.children.cycles-pp.security_file_permission
     24.51           -24.5        0.00        perf-profile.self.cycles-pp.syscall_exit_to_user_mode
     20.88           -20.9        0.00        perf-profile.self.cycles-pp.__entry_text_start
     16.64           -16.6        0.00        perf-profile.self.cycles-pp.syscall_return_via_sysret
      0.06 ±  4%    +338.4%       0.26 ±  3%  perf-stat.i.MPKI
 1.897e+10            -3.8%  1.825e+10        perf-stat.i.branch-instructions
      1.73            -0.0        1.69        perf-stat.i.branch-miss-rate%
 3.262e+08            -5.6%  3.078e+08        perf-stat.i.branch-misses
    555892 ±  4%    +628.5%    4049656 ± 44%  perf-stat.i.cache-misses
   4726936 ±  5%    +389.3%   23128136 ±  4%  perf-stat.i.cache-references
      1469 ±  5%     +41.4%       2078        perf-stat.i.context-switches
      2.98            +4.6%       3.11        perf-stat.i.cpi
    649902 ±  5%     -84.5%     100578 ± 65%  perf-stat.i.cycles-between-cache-misses
      0.39            -0.0        0.38        perf-stat.i.dTLB-load-miss-rate%
 1.074e+08            -6.1%  1.008e+08        perf-stat.i.dTLB-load-misses
 2.749e+10            -4.5%  2.625e+10        perf-stat.i.dTLB-loads
      0.00            +0.0        0.00        perf-stat.i.dTLB-store-miss-rate%
    102626            +3.6%     106273        perf-stat.i.dTLB-store-misses
  1.86e+10            -4.6%  1.774e+10        perf-stat.i.dTLB-stores
 1.084e+08            -5.9%   1.02e+08        perf-stat.i.iTLB-load-misses
 9.457e+10            -3.8%  9.094e+10        perf-stat.i.instructions
      1036            -7.8%     955.05        perf-stat.i.instructions-per-iTLB-miss
      0.34            -4.3%       0.32        perf-stat.i.ipc
      0.91 ±  5%     -68.9%       0.28 ± 22%  perf-stat.i.major-faults
      1.99 ±  4%     -65.9%       0.68 ± 36%  perf-stat.i.metric.K/sec
    625.66            -4.3%     598.87        perf-stat.i.metric.M/sec
      3112           -13.0%       2709        perf-stat.i.minor-faults
    103775 ±  3%    +559.2%     684048 ± 43%  perf-stat.i.node-load-misses
     21080 ±  5%   +1052.0%     242839 ± 61%  perf-stat.i.node-store-misses
      9471 ±  5%     +92.5%      18236 ±  6%  perf-stat.i.node-stores
      3113           -13.0%       2709        perf-stat.i.page-faults
      0.05 ±  5%    +403.5%       0.25 ±  4%  perf-stat.overall.MPKI
      1.72            -0.0        1.69        perf-stat.overall.branch-miss-rate%
      2.98            +4.6%       3.12        perf-stat.overall.cpi
    500646 ±  3%     -80.4%      98361 ± 64%  perf-stat.overall.cycles-between-cache-misses
      0.39            -0.0        0.38        perf-stat.overall.dTLB-load-miss-rate%
      0.00            +0.0        0.00        perf-stat.overall.dTLB-store-miss-rate%
    872.51            +2.2%     891.54        perf-stat.overall.instructions-per-iTLB-miss
      0.34            -4.4%       0.32        perf-stat.overall.ipc
     69.85           +17.4       87.30 ±  3%  perf-stat.overall.node-load-miss-rate%
    529805            +2.4%     542598        perf-stat.overall.path-length
 1.891e+10            -3.8%  1.819e+10        perf-stat.ps.branch-instructions
 3.251e+08            -5.6%  3.068e+08        perf-stat.ps.branch-misses
    562046 ±  3%    +618.1%    4036217 ± 44%  perf-stat.ps.cache-misses
   4761203 ±  5%    +384.2%   23051468 ±  4%  perf-stat.ps.cache-references
      1464 ±  5%     +41.4%       2071        perf-stat.ps.context-switches
  1.07e+08            -6.1%  1.005e+08        perf-stat.ps.dTLB-load-misses
  2.74e+10            -4.5%  2.617e+10        perf-stat.ps.dTLB-loads
    102534            +3.3%     105923        perf-stat.ps.dTLB-store-misses
 1.854e+10            -4.6%  1.768e+10        perf-stat.ps.dTLB-stores
  1.08e+08            -5.9%  1.017e+08        perf-stat.ps.iTLB-load-misses
 9.425e+10            -3.8%  9.064e+10        perf-stat.ps.instructions
      0.91 ±  5%     -68.9%       0.28 ± 22%  perf-stat.ps.major-faults
      3112           -13.2%       2701        perf-stat.ps.minor-faults
    103489 ±  3%    +558.8%     681779 ± 43%  perf-stat.ps.node-load-misses
     21019 ±  5%   +1051.4%     242030 ± 61%  perf-stat.ps.node-store-misses
      9655 ±  5%     +88.4%      18191 ±  6%  perf-stat.ps.node-stores
      3113           -13.2%       2701        perf-stat.ps.page-faults
 2.851e+13            -4.0%  2.737e+13        perf-stat.total.instructions
     29272 ± 20%    +217.2%      92856 ± 68%  softirqs.CPU0.RCU
     28556 ± 22%     +75.4%      50096 ± 14%  softirqs.CPU1.RCU
     25887 ± 26%     +81.2%      46903 ±  8%  softirqs.CPU10.RCU
     25505 ± 21%    +109.4%      53408 ± 28%  softirqs.CPU100.RCU
     25573 ± 21%    +162.1%      67035 ± 77%  softirqs.CPU101.RCU
     25640 ± 21%     +79.3%      45965 ± 10%  softirqs.CPU102.RCU
     26246 ± 22%     +70.8%      44817 ±  9%  softirqs.CPU103.RCU
     26572 ± 23%     +95.4%      51920 ± 16%  softirqs.CPU11.RCU
     26735 ± 22%     +79.1%      47872 ± 13%  softirqs.CPU12.RCU
     26905 ± 23%     +81.3%      48783 ± 15%  softirqs.CPU13.RCU
     26212 ± 25%     +81.4%      47540 ± 17%  softirqs.CPU14.RCU
     30184 ± 21%     +79.6%      54204 ± 14%  softirqs.CPU15.RCU
     30139 ± 21%     +70.9%      51501 ±  5%  softirqs.CPU16.RCU
     30448 ± 21%     +67.6%      51036 ±  6%  softirqs.CPU17.RCU
     30542 ± 22%     +77.7%      54274 ± 13%  softirqs.CPU18.RCU
     30572 ± 21%     +75.3%      53595 ±  8%  softirqs.CPU19.RCU
     27505 ± 23%     +59.9%      43983        softirqs.CPU2.RCU
     30494 ± 21%     +69.0%      51532 ± 10%  softirqs.CPU20.RCU
     30404 ± 22%     +74.1%      52921 ± 10%  softirqs.CPU21.RCU
     30124 ± 21%     +58.1%      47615 ±  9%  softirqs.CPU22.RCU
     30208 ± 22%     +70.2%      51406 ±  7%  softirqs.CPU23.RCU
     30500 ± 22%     +80.9%      55170 ± 11%  softirqs.CPU24.RCU
     30498 ± 22%     +70.0%      51841 ±  9%  softirqs.CPU25.RCU
     30289 ± 20%     +67.2%      50632 ± 16%  softirqs.CPU26.RCU
     28512 ± 22%     +57.4%      44882 ±  6%  softirqs.CPU27.RCU
     27424 ± 24%     +71.2%      46950 ± 19%  softirqs.CPU28.RCU
     27144 ± 24%     +85.1%      50233 ± 17%  softirqs.CPU29.RCU
     27428 ± 23%    +167.5%      73379 ± 71%  softirqs.CPU3.RCU
     28605 ± 23%     +80.0%      51501 ± 19%  softirqs.CPU30.RCU
     28492 ± 22%     +68.5%      48006 ±  6%  softirqs.CPU31.RCU
     25999 ±  6%     +96.0%      50956 ± 12%  softirqs.CPU32.RCU
     29537 ± 22%     +98.2%      58554 ± 42%  softirqs.CPU33.RCU
     27675 ± 26%     +67.1%      46239        softirqs.CPU34.RCU
     28291 ± 23%     +63.6%      46274        softirqs.CPU35.RCU
     28351 ± 22%     +93.7%      54930 ± 19%  softirqs.CPU36.RCU
     28464 ± 22%     +81.1%      51539 ± 14%  softirqs.CPU37.RCU
     27484 ± 24%     +70.2%      46785 ± 15%  softirqs.CPU38.RCU
     28024 ± 22%     +62.7%      45599        softirqs.CPU39.RCU
     26917 ± 22%     +61.0%      43345 ±  5%  softirqs.CPU4.RCU
     28337 ± 22%     +84.7%      52325 ± 12%  softirqs.CPU40.RCU
     28016 ± 22%     +62.4%      45501        softirqs.CPU41.RCU
     28151 ± 22%     +98.1%      55757 ± 15%  softirqs.CPU42.RCU
     28436 ± 22%     +76.5%      50190 ± 12%  softirqs.CPU43.RCU
     28569 ± 21%    +172.5%      77851 ± 74%  softirqs.CPU44.RCU
     27278 ± 23%     +71.2%      46688 ± 10%  softirqs.CPU45.RCU
     27129 ± 26%     +78.2%      48341 ± 19%  softirqs.CPU46.RCU
     26888 ± 22%     +71.4%      46081 ± 14%  softirqs.CPU47.RCU
     27117 ± 22%    +100.5%      54364 ± 27%  softirqs.CPU48.RCU
     27341 ± 22%    +147.5%      67674 ± 77%  softirqs.CPU49.RCU
     26690 ± 23%     +67.4%      44668 ± 17%  softirqs.CPU5.RCU
     27347 ± 23%     +69.9%      46451 ±  8%  softirqs.CPU50.RCU
     27667 ± 23%     +65.8%      45881 ± 10%  softirqs.CPU51.RCU
     31835 ± 18%    +179.5%      88967 ± 64%  softirqs.CPU52.RCU
     33461 ± 18%     +63.1%      54589 ±  8%  softirqs.CPU53.RCU
     30080 ± 24%     +59.8%      48060 ±  2%  softirqs.CPU54.RCU
     30214 ± 20%    +158.6%      78143 ± 64%  softirqs.CPU55.RCU
     29822 ± 20%     +57.3%      46914 ± 13%  softirqs.CPU56.RCU
     29623 ± 20%     +72.3%      51053 ± 15%  softirqs.CPU57.RCU
     29602 ± 20%     +66.1%      49167 ±  2%  softirqs.CPU58.RCU
     29167 ± 21%     +68.3%      49077 ± 12%  softirqs.CPU59.RCU
     26548 ± 22%     +65.7%      43979 ±  3%  softirqs.CPU6.RCU
     26874 ± 22%     +70.5%      45825 ±  5%  softirqs.CPU60.RCU
     26636 ± 22%     +78.1%      47440 ± 14%  softirqs.CPU61.RCU
     26156 ± 25%     +85.3%      48456 ± 10%  softirqs.CPU62.RCU
     26814 ± 23%     +92.8%      51700 ± 14%  softirqs.CPU63.RCU
     27189 ± 22%     +84.2%      50080 ± 16%  softirqs.CPU64.RCU
     26912 ± 23%     +88.2%      50645 ± 14%  softirqs.CPU65.RCU
     26746 ± 23%     +81.1%      48444 ± 12%  softirqs.CPU66.RCU
     27265 ± 21%     +80.9%      49332 ± 11%  softirqs.CPU67.RCU
     27233 ± 22%     +75.9%      47915 ±  8%  softirqs.CPU68.RCU
     26988 ± 23%     +73.6%      46859 ±  5%  softirqs.CPU69.RCU
     27187 ± 22%     +59.3%      43299 ± 15%  softirqs.CPU7.RCU
     27143 ± 22%     +81.7%      49309 ± 10%  softirqs.CPU70.RCU
     27192 ± 22%     +91.1%      51955 ± 15%  softirqs.CPU71.RCU
     26927 ± 23%     +76.0%      47381 ± 11%  softirqs.CPU72.RCU
     26744 ± 23%     +77.8%      47540 ±  7%  softirqs.CPU73.RCU
     26600 ± 23%     +72.3%      45837 ±  4%  softirqs.CPU74.RCU
     25739 ± 20%     +82.1%      46883 ± 11%  softirqs.CPU75.RCU
     25979 ± 19%     +79.8%      46706 ±  9%  softirqs.CPU76.RCU
     26063 ± 19%     +81.5%      47295 ± 14%  softirqs.CPU77.RCU
     32477 ± 18%     +59.1%      51671 ± 25%  softirqs.CPU78.RCU
     32395 ± 18%     +50.0%      48577 ±  6%  softirqs.CPU79.RCU
     27059 ± 22%     +65.2%      44713 ±  4%  softirqs.CPU8.RCU
     29857 ± 20%     +68.1%      50192 ± 19%  softirqs.CPU80.RCU
     28839 ± 21%     +84.8%      53291 ± 20%  softirqs.CPU81.RCU
     28496 ± 20%     +76.8%      50394 ± 18%  softirqs.CPU82.RCU
     28388 ± 20%     +66.6%      47286 ±  4%  softirqs.CPU83.RCU
     27419 ± 15%     +83.1%      50211 ± 10%  softirqs.CPU84.RCU
     27740 ± 21%    +105.0%      56870 ± 41%  softirqs.CPU85.RCU
     27446 ± 23%     +69.0%      46371        softirqs.CPU86.RCU
     28118 ± 22%     +63.5%      45970        softirqs.CPU87.RCU
     27945 ± 21%    +100.2%      55955 ± 23%  softirqs.CPU88.RCU
     28060 ± 21%     +86.2%      52260 ± 14%  softirqs.CPU89.RCU
     26821 ± 22%     +71.5%      45988 ± 14%  softirqs.CPU9.RCU
     25391 ± 22%     +75.7%      44618 ± 14%  softirqs.CPU90.RCU
     25372 ± 22%     +66.8%      42331        softirqs.CPU91.RCU
     25418 ± 21%     +85.1%      47051 ± 10%  softirqs.CPU92.RCU
     25647 ± 21%     +65.2%      42374        softirqs.CPU93.RCU
     25635 ± 21%     +90.0%      48706 ± 10%  softirqs.CPU94.RCU
     25658 ± 21%     +85.6%      47613 ± 16%  softirqs.CPU95.RCU
     25595 ± 21%    +193.0%      74990 ± 79%  softirqs.CPU96.RCU
     25644 ± 21%     +80.8%      46355 ± 11%  softirqs.CPU97.RCU
     25404 ± 21%     +87.6%      47661 ± 21%  softirqs.CPU98.RCU
     25554 ± 21%     +72.6%      44106 ± 10%  softirqs.CPU99.RCU
   2897508 ± 21%     +83.5%    5316315        softirqs.RCU
     40871 ± 12%     +43.0%      58456 ±  2%  softirqs.TIMER
      0.02 ±  2%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown]
      0.92 ±217%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.devkmsg_read.vfs_read.ksys_read.do_syscall_64
      0.02 ±  5%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.do_nanosleep.hrtimer_nanosleep.__x64_sys_nanosleep.do_syscall_64
      4.58 ± 60%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.do_syslog.part.0.kmsg_read.vfs_read
      0.60 ± 18%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
      0.03 ± 66%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64
      0.03 ± 26%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.futex_wait_queue_me.futex_wait.do_futex.__x64_sys_futex
      0.01 ±  9%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.pipe_read.new_sync_read.vfs_read.ksys_read
      9.01          -100.0%       0.00        perf-sched.sch_delay.avg.ms.preempt_schedule_common.__cond_resched.stop_one_cpu.sched_exec.bprm_execve
      0.02 ±  2%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.rcu_gp_kthread.kthread.ret_from_fork
      0.28 ±134%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait.__x64_sys_epoll_wait
      0.02 ±  9%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_select
      0.00          -100.0%       0.00        perf-sched.sch_delay.avg.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_sys_poll
      0.02 ±  4%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.schedule_timeout.kcompactd.kthread.ret_from_fork
      0.02          -100.0%       0.00        perf-sched.sch_delay.avg.ms.schedule_timeout.rcu_gp_kthread.kthread.ret_from_fork
      0.01 ±  4%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.smpboot_thread_fn.kthread.ret_from_fork
      0.97 ± 15%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.wait_for_partner.fifo_open.do_dentry_open.do_open.isra
      0.04 ± 24%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.worker_thread.kthread.ret_from_fork
      0.02 ±  7%    -100.0%       0.00        perf-sched.sch_delay.max.ms.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown]
      2.07 ±220%    -100.0%       0.00        perf-sched.sch_delay.max.ms.devkmsg_read.vfs_read.ksys_read.do_syscall_64
      0.02 ± 24%    -100.0%       0.00        perf-sched.sch_delay.max.ms.do_nanosleep.hrtimer_nanosleep.__x64_sys_nanosleep.do_syscall_64
     14.48 ± 56%    -100.0%       0.00        perf-sched.sch_delay.max.ms.do_syslog.part.0.kmsg_read.vfs_read
     14.35 ± 26%    -100.0%       0.00        perf-sched.sch_delay.max.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
      2.30 ±219%    -100.0%       0.00        perf-sched.sch_delay.max.ms.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64
      5.48 ±102%    -100.0%       0.00        perf-sched.sch_delay.max.ms.exit_to_user_mode_prepare.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe.[unknown]
      0.04 ± 46%    -100.0%       0.00        perf-sched.sch_delay.max.ms.futex_wait_queue_me.futex_wait.do_futex.__x64_sys_futex
      0.22 ± 36%    -100.0%       0.00        perf-sched.sch_delay.max.ms.pipe_read.new_sync_read.vfs_read.ksys_read
      0.04 ± 45%    -100.0%       0.00        perf-sched.sch_delay.max.ms.preempt_schedule_common.__cond_resched.stop_one_cpu.affine_move_task.__set_cpus_allowed_ptr
     22.37 ± 19%    -100.0%       0.00        perf-sched.sch_delay.max.ms.preempt_schedule_common.__cond_resched.stop_one_cpu.sched_exec.bprm_execve
      0.03 ± 17%    -100.0%       0.00        perf-sched.sch_delay.max.ms.rcu_gp_kthread.kthread.ret_from_fork
      4.26 ±131%    -100.0%       0.00        perf-sched.sch_delay.max.ms.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait.__x64_sys_epoll_wait
      0.02 ± 10%    -100.0%       0.00        perf-sched.sch_delay.max.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_select
      0.03 ± 28%    -100.0%       0.00        perf-sched.sch_delay.max.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_sys_poll
      0.02 ± 22%    -100.0%       0.00        perf-sched.sch_delay.max.ms.schedule_timeout.kcompactd.kthread.ret_from_fork
      0.05 ± 66%    -100.0%       0.00        perf-sched.sch_delay.max.ms.schedule_timeout.rcu_gp_kthread.kthread.ret_from_fork
      0.04 ± 61%    -100.0%       0.00        perf-sched.sch_delay.max.ms.smpboot_thread_fn.kthread.ret_from_fork
     14.51 ± 23%    -100.0%       0.00        perf-sched.sch_delay.max.ms.wait_for_partner.fifo_open.do_dentry_open.do_open.isra
      9.11 ± 43%    -100.0%       0.00        perf-sched.sch_delay.max.ms.worker_thread.kthread.ret_from_fork
      0.18 ±  6%    -100.0%       0.00        perf-sched.total_sch_delay.average.ms
     22.51 ± 19%    -100.0%       0.00        perf-sched.total_sch_delay.max.ms
    145.21 ±  9%    -100.0%       0.00        perf-sched.total_wait_and_delay.average.ms
     12308 ±  6%    -100.0%       0.00        perf-sched.total_wait_and_delay.count.ms
      8429 ±  9%    -100.0%       0.00        perf-sched.total_wait_and_delay.max.ms
    145.02 ±  9%    -100.0%       0.00        perf-sched.total_wait_time.average.ms
      8429 ±  9%    -100.0%       0.00        perf-sched.total_wait_time.max.ms
    882.81 ±  4%    -100.0%       0.00        perf-sched.wait_and_delay.avg.ms.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown]
    917.42 ± 28%    -100.0%       0.00        perf-sched.wait_and_delay.avg.ms.devkmsg_read.vfs_read.ksys_read.do_syscall_64
    803.52 ±  4%    -100.0%       0.00        perf-sched.wait_and_delay.avg.ms.do_nanosleep.hrtimer_nanosleep.__x64_sys_nanosleep.do_syscall_64
      1002 ± 31%    -100.0%       0.00        perf-sched.wait_and_delay.avg.ms.do_syslog.part.0.kmsg_read.vfs_read
    265.93 ±  3%    -100.0%       0.00        perf-sched.wait_and_delay.avg.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
    109.91 ±  7%    -100.0%       0.00        perf-sched.wait_and_delay.avg.ms.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64
      0.24 ± 19%    -100.0%       0.00        perf-sched.wait_and_delay.avg.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt.[unknown]
      0.16 ± 16%    -100.0%       0.00        perf-sched.wait_and_delay.avg.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_call_function_single.[unknown]
      0.12 ± 35%    -100.0%       0.00        perf-sched.wait_and_delay.avg.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_reschedule_ipi.[unknown]
      0.27 ± 44%    -100.0%       0.00        perf-sched.wait_and_delay.avg.ms.exit_to_user_mode_prepare.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe.[unknown]
    208.74 ±  4%    -100.0%       0.00        perf-sched.wait_and_delay.avg.ms.pipe_read.new_sync_read.vfs_read.ksys_read
      8.98          -100.0%       0.00        perf-sched.wait_and_delay.avg.ms.preempt_schedule_common.__cond_resched.stop_one_cpu.affine_move_task.__set_cpus_allowed_ptr
      9.02          -100.0%       0.00        perf-sched.wait_and_delay.avg.ms.preempt_schedule_common.__cond_resched.stop_one_cpu.sched_exec.bprm_execve
      2.64 ±  4%    -100.0%       0.00        perf-sched.wait_and_delay.avg.ms.rcu_gp_kthread.kthread.ret_from_fork
    526.58 ± 47%    -100.0%       0.00        perf-sched.wait_and_delay.avg.ms.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait.__x64_sys_epoll_wait
    157.83 ±  5%    -100.0%       0.00        perf-sched.wait_and_delay.avg.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_sys_poll
    478.58          -100.0%       0.00        perf-sched.wait_and_delay.avg.ms.schedule_timeout.kcompactd.kthread.ret_from_fork
      6.98 ± 20%    -100.0%       0.00        perf-sched.wait_and_delay.avg.ms.schedule_timeout.rcu_gp_kthread.kthread.ret_from_fork
    586.89 ±  2%    -100.0%       0.00        perf-sched.wait_and_delay.avg.ms.smpboot_thread_fn.kthread.ret_from_fork
    584.74 ± 11%    -100.0%       0.00        perf-sched.wait_and_delay.avg.ms.worker_thread.kthread.ret_from_fork
     10.00          -100.0%       0.00        perf-sched.wait_and_delay.count.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown]
      4.67 ± 10%    -100.0%       0.00        perf-sched.wait_and_delay.count.devkmsg_read.vfs_read.ksys_read.do_syscall_64
     22.50 ±  4%    -100.0%       0.00        perf-sched.wait_and_delay.count.do_nanosleep.hrtimer_nanosleep.__x64_sys_nanosleep.do_syscall_64
      4.33 ± 10%    -100.0%       0.00        perf-sched.wait_and_delay.count.do_syslog.part.0.kmsg_read.vfs_read
    236.83          -100.0%       0.00        perf-sched.wait_and_delay.count.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
    268.00          -100.0%       0.00        perf-sched.wait_and_delay.count.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64
      1662 ± 16%    -100.0%       0.00        perf-sched.wait_and_delay.count.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt.[unknown]
    238.17 ±  4%    -100.0%       0.00        perf-sched.wait_and_delay.count.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_call_function_single.[unknown]
    181.33 ± 13%    -100.0%       0.00        perf-sched.wait_and_delay.count.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_reschedule_ipi.[unknown]
      3252 ±  4%    -100.0%       0.00        perf-sched.wait_and_delay.count.exit_to_user_mode_prepare.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe.[unknown]
    624.83 ±  3%    -100.0%       0.00        perf-sched.wait_and_delay.count.pipe_read.new_sync_read.vfs_read.ksys_read
      1248          -100.0%       0.00        perf-sched.wait_and_delay.count.preempt_schedule_common.__cond_resched.stop_one_cpu.affine_move_task.__set_cpus_allowed_ptr
    212.33          -100.0%       0.00        perf-sched.wait_and_delay.count.preempt_schedule_common.__cond_resched.stop_one_cpu.sched_exec.bprm_execve
    111.00 ± 47%    -100.0%       0.00        perf-sched.wait_and_delay.count.rcu_gp_kthread.kthread.ret_from_fork
     19.50 ± 51%    -100.0%       0.00        perf-sched.wait_and_delay.count.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait.__x64_sys_epoll_wait
    158.50 ±  6%    -100.0%       0.00        perf-sched.wait_and_delay.count.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_sys_poll
     39.67          -100.0%       0.00        perf-sched.wait_and_delay.count.schedule_timeout.kcompactd.kthread.ret_from_fork
      1469 ± 26%    -100.0%       0.00        perf-sched.wait_and_delay.count.schedule_timeout.rcu_gp_kthread.kthread.ret_from_fork
      1834 ±  2%    -100.0%       0.00        perf-sched.wait_and_delay.count.smpboot_thread_fn.kthread.ret_from_fork
    601.00 ± 11%    -100.0%       0.00        perf-sched.wait_and_delay.count.worker_thread.kthread.ret_from_fork
    999.44          -100.0%       0.00        perf-sched.wait_and_delay.max.ms.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown]
      2750 ± 28%    -100.0%       0.00        perf-sched.wait_and_delay.max.ms.devkmsg_read.vfs_read.ksys_read.do_syscall_64
      1000          -100.0%       0.00        perf-sched.wait_and_delay.max.ms.do_nanosleep.hrtimer_nanosleep.__x64_sys_nanosleep.do_syscall_64
      2763 ± 28%    -100.0%       0.00        perf-sched.wait_and_delay.max.ms.do_syslog.part.0.kmsg_read.vfs_read
      1841 ±102%    -100.0%       0.00        perf-sched.wait_and_delay.max.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
      1021          -100.0%       0.00        perf-sched.wait_and_delay.max.ms.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64
     23.20 ± 62%    -100.0%       0.00        perf-sched.wait_and_delay.max.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt.[unknown]
      8.54 ± 46%    -100.0%       0.00        perf-sched.wait_and_delay.max.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_call_function_single.[unknown]
      6.14 ± 99%    -100.0%       0.00        perf-sched.wait_and_delay.max.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_reschedule_ipi.[unknown]
    206.18 ±172%    -100.0%       0.00        perf-sched.wait_and_delay.max.ms.exit_to_user_mode_prepare.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe.[unknown]
      2752 ± 28%    -100.0%       0.00        perf-sched.wait_and_delay.max.ms.pipe_read.new_sync_read.vfs_read.ksys_read
      2177          -100.0%       0.00        perf-sched.wait_and_delay.max.ms.preempt_schedule_common.__cond_resched.stop_one_cpu.affine_move_task.__set_cpus_allowed_ptr
     22.37 ± 19%    -100.0%       0.00        perf-sched.wait_and_delay.max.ms.preempt_schedule_common.__cond_resched.stop_one_cpu.sched_exec.bprm_execve
      5.02          -100.0%       0.00        perf-sched.wait_and_delay.max.ms.rcu_gp_kthread.kthread.ret_from_fork
      4004 ± 60%    -100.0%       0.00        perf-sched.wait_and_delay.max.ms.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait.__x64_sys_epoll_wait
      5816          -100.0%       0.00        perf-sched.wait_and_delay.max.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_sys_poll
    505.01          -100.0%       0.00        perf-sched.wait_and_delay.max.ms.schedule_timeout.kcompactd.kthread.ret_from_fork
    362.67 ± 42%    -100.0%       0.00        perf-sched.wait_and_delay.max.ms.schedule_timeout.rcu_gp_kthread.kthread.ret_from_fork
      7190 ± 18%    -100.0%       0.00        perf-sched.wait_and_delay.max.ms.smpboot_thread_fn.kthread.ret_from_fork
      7974 ± 11%    -100.0%       0.00        perf-sched.wait_and_delay.max.ms.worker_thread.kthread.ret_from_fork
    882.79 ±  4%    -100.0%       0.00        perf-sched.wait_time.avg.ms.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown]
    916.50 ± 28%    -100.0%       0.00        perf-sched.wait_time.avg.ms.devkmsg_read.vfs_read.ksys_read.do_syscall_64
    803.50 ±  4%    -100.0%       0.00        perf-sched.wait_time.avg.ms.do_nanosleep.hrtimer_nanosleep.__x64_sys_nanosleep.do_syscall_64
    997.70 ± 31%    -100.0%       0.00        perf-sched.wait_time.avg.ms.do_syslog.part.0.kmsg_read.vfs_read
    265.33 ±  3%    -100.0%       0.00        perf-sched.wait_time.avg.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
    109.88 ±  7%    -100.0%       0.00        perf-sched.wait_time.avg.ms.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64
      0.24 ± 19%    -100.0%       0.00        perf-sched.wait_time.avg.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt.[unknown]
      0.16 ± 16%    -100.0%       0.00        perf-sched.wait_time.avg.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_call_function_single.[unknown]
      0.10 ± 86%    -100.0%       0.00        perf-sched.wait_time.avg.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_irq_work.[unknown]
      0.12 ± 35%    -100.0%       0.00        perf-sched.wait_time.avg.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_reschedule_ipi.[unknown]
      0.27 ± 45%    -100.0%       0.00        perf-sched.wait_time.avg.ms.exit_to_user_mode_prepare.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe.[unknown]
    208.72 ±  4%    -100.0%       0.00        perf-sched.wait_time.avg.ms.pipe_read.new_sync_read.vfs_read.ksys_read
      9.91 ± 20%    -100.0%       0.00        perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.dput.__fput.task_work_run
    439.75 ±115%    -100.0%       0.00        perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.generic_perform_write.__generic_file_write_iter.generic_file_write_iter
      8.98          -100.0%       0.00        perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.stop_one_cpu.affine_move_task.__set_cpus_allowed_ptr
      2.62 ±  4%    -100.0%       0.00        perf-sched.wait_time.avg.ms.rcu_gp_kthread.kthread.ret_from_fork
    526.30 ± 47%    -100.0%       0.00        perf-sched.wait_time.avg.ms.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait.__x64_sys_epoll_wait
    157.83 ±  5%    -100.0%       0.00        perf-sched.wait_time.avg.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_sys_poll
    478.56          -100.0%       0.00        perf-sched.wait_time.avg.ms.schedule_timeout.kcompactd.kthread.ret_from_fork
      3.13 ± 44%    -100.0%       0.00        perf-sched.wait_time.avg.ms.schedule_timeout.khugepaged.kthread.ret_from_fork
      6.96 ± 20%    -100.0%       0.00        perf-sched.wait_time.avg.ms.schedule_timeout.rcu_gp_kthread.kthread.ret_from_fork
    586.88 ±  2%    -100.0%       0.00        perf-sched.wait_time.avg.ms.smpboot_thread_fn.kthread.ret_from_fork
      2.18 ± 19%    -100.0%       0.00        perf-sched.wait_time.avg.ms.wait_for_partner.fifo_open.do_dentry_open.do_open.isra
    584.70 ± 11%    -100.0%       0.00        perf-sched.wait_time.avg.ms.worker_thread.kthread.ret_from_fork
    999.42          -100.0%       0.00        perf-sched.wait_time.max.ms.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown]
      2748 ± 28%    -100.0%       0.00        perf-sched.wait_time.max.ms.devkmsg_read.vfs_read.ksys_read.do_syscall_64
      1000          -100.0%       0.00        perf-sched.wait_time.max.ms.do_nanosleep.hrtimer_nanosleep.__x64_sys_nanosleep.do_syscall_64
      2754 ± 28%    -100.0%       0.00        perf-sched.wait_time.max.ms.do_syslog.part.0.kmsg_read.vfs_read
      1841 ±102%    -100.0%       0.00        perf-sched.wait_time.max.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
      1021          -100.0%       0.00        perf-sched.wait_time.max.ms.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64
     23.20 ± 62%    -100.0%       0.00        perf-sched.wait_time.max.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt.[unknown]
      8.54 ± 46%    -100.0%       0.00        perf-sched.wait_time.max.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_call_function_single.[unknown]
      1.24 ±105%    -100.0%       0.00        perf-sched.wait_time.max.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_irq_work.[unknown]
      6.14 ± 99%    -100.0%       0.00        perf-sched.wait_time.max.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_reschedule_ipi.[unknown]
    206.17 ±172%    -100.0%       0.00        perf-sched.wait_time.max.ms.exit_to_user_mode_prepare.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe.[unknown]
      2752 ± 28%    -100.0%       0.00        perf-sched.wait_time.max.ms.pipe_read.new_sync_read.vfs_read.ksys_read
     11.86 ± 16%    -100.0%       0.00        perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.dput.__fput.task_work_run
      3351 ± 99%    -100.0%       0.00        perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.generic_perform_write.__generic_file_write_iter.generic_file_write_iter
      2177          -100.0%       0.00        perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.stop_one_cpu.affine_move_task.__set_cpus_allowed_ptr
      5.00          -100.0%       0.00        perf-sched.wait_time.max.ms.rcu_gp_kthread.kthread.ret_from_fork
      4004 ± 60%    -100.0%       0.00        perf-sched.wait_time.max.ms.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait.__x64_sys_epoll_wait
      5816          -100.0%       0.00        perf-sched.wait_time.max.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_sys_poll
    504.99          -100.0%       0.00        perf-sched.wait_time.max.ms.schedule_timeout.kcompactd.kthread.ret_from_fork
      3.13 ± 44%    -100.0%       0.00        perf-sched.wait_time.max.ms.schedule_timeout.khugepaged.kthread.ret_from_fork
    362.65 ± 42%    -100.0%       0.00        perf-sched.wait_time.max.ms.schedule_timeout.rcu_gp_kthread.kthread.ret_from_fork
      7190 ± 18%    -100.0%       0.00        perf-sched.wait_time.max.ms.smpboot_thread_fn.kthread.ret_from_fork
     12.80          -100.0%       0.00        perf-sched.wait_time.max.ms.wait_for_partner.fifo_open.do_dentry_open.do_open.isra
      7974 ± 11%    -100.0%       0.00        perf-sched.wait_time.max.ms.worker_thread.kthread.ret_from_fork
    104127           -31.8%      71024 ±  2%  interrupts.CAL:Function_call_interrupts
      7546          -100.0%       0.00        interrupts.CPU0.NMI:Non-maskable_interrupts
      7546          -100.0%       0.00        interrupts.CPU0.PMI:Performance_monitoring_interrupts
      1652 ± 23%     -54.3%     755.83 ± 37%  interrupts.CPU1.CAL:Function_call_interrupts
      5650 ± 33%    -100.0%       0.00        interrupts.CPU1.NMI:Non-maskable_interrupts
      5650 ± 33%    -100.0%       0.00        interrupts.CPU1.PMI:Performance_monitoring_interrupts
    677.33 ± 29%     -38.5%     416.83 ± 16%  interrupts.CPU1.RES:Rescheduling_interrupts
      6911 ± 20%    -100.0%       0.00        interrupts.CPU10.NMI:Non-maskable_interrupts
      6911 ± 20%    -100.0%       0.00        interrupts.CPU10.PMI:Performance_monitoring_interrupts
    329.33           +99.6%     657.33 ± 38%  interrupts.CPU10.RES:Rescheduling_interrupts
      7180 ± 20%    -100.0%       0.00        interrupts.CPU100.NMI:Non-maskable_interrupts
      7180 ± 20%    -100.0%       0.00        interrupts.CPU100.PMI:Performance_monitoring_interrupts
    331.33 ±  7%     +56.6%     519.00 ± 40%  interrupts.CPU100.RES:Rescheduling_interrupts
      7181 ± 20%    -100.0%       0.00        interrupts.CPU101.NMI:Non-maskable_interrupts
      7181 ± 20%    -100.0%       0.00        interrupts.CPU101.PMI:Performance_monitoring_interrupts
    324.67 ±  4%     +40.1%     454.83 ± 13%  interrupts.CPU101.RES:Rescheduling_interrupts
      7181 ± 20%    -100.0%       0.00        interrupts.CPU102.NMI:Non-maskable_interrupts
      7181 ± 20%    -100.0%       0.00        interrupts.CPU102.PMI:Performance_monitoring_interrupts
      7177 ± 20%    -100.0%       0.00        interrupts.CPU103.NMI:Non-maskable_interrupts
      7177 ± 20%    -100.0%       0.00        interrupts.CPU103.PMI:Performance_monitoring_interrupts
      6911 ± 20%    -100.0%       0.00        interrupts.CPU11.NMI:Non-maskable_interrupts
      6911 ± 20%    -100.0%       0.00        interrupts.CPU11.PMI:Performance_monitoring_interrupts
      6909 ± 20%    -100.0%       0.00        interrupts.CPU12.NMI:Non-maskable_interrupts
      6909 ± 20%    -100.0%       0.00        interrupts.CPU12.PMI:Performance_monitoring_interrupts
    339.33 ±  6%     +38.2%     468.83 ± 35%  interrupts.CPU12.RES:Rescheduling_interrupts
      6914 ± 20%    -100.0%       0.00        interrupts.CPU13.NMI:Non-maskable_interrupts
      6914 ± 20%    -100.0%       0.00        interrupts.CPU13.PMI:Performance_monitoring_interrupts
    340.83 ±  7%     +22.1%     416.17 ± 11%  interrupts.CPU13.RES:Rescheduling_interrupts
      7549          -100.0%       0.00        interrupts.CPU14.NMI:Non-maskable_interrupts
      7549          -100.0%       0.00        interrupts.CPU14.PMI:Performance_monitoring_interrupts
      7542          -100.0%       0.00        interrupts.CPU15.NMI:Non-maskable_interrupts
      7542          -100.0%       0.00        interrupts.CPU15.PMI:Performance_monitoring_interrupts
      7555          -100.0%       0.00        interrupts.CPU16.NMI:Non-maskable_interrupts
      7555          -100.0%       0.00        interrupts.CPU16.PMI:Performance_monitoring_interrupts
    324.83           +29.8%     421.50 ± 13%  interrupts.CPU16.RES:Rescheduling_interrupts
      6289 ± 28%    -100.0%       0.00        interrupts.CPU17.NMI:Non-maskable_interrupts
      6289 ± 28%    -100.0%       0.00        interrupts.CPU17.PMI:Performance_monitoring_interrupts
    321.67           +18.7%     381.83 ±  4%  interrupts.CPU17.RES:Rescheduling_interrupts
      6919 ± 20%    -100.0%       0.00        interrupts.CPU18.NMI:Non-maskable_interrupts
      6919 ± 20%    -100.0%       0.00        interrupts.CPU18.PMI:Performance_monitoring_interrupts
    324.17           +73.3%     561.67 ± 33%  interrupts.CPU18.RES:Rescheduling_interrupts
      6914 ± 20%    -100.0%       0.00        interrupts.CPU19.NMI:Non-maskable_interrupts
      6914 ± 20%    -100.0%       0.00        interrupts.CPU19.PMI:Performance_monitoring_interrupts
    321.17           +28.2%     411.83 ± 12%  interrupts.CPU19.RES:Rescheduling_interrupts
      1214 ± 42%     -55.4%     541.50 ± 16%  interrupts.CPU2.CAL:Function_call_interrupts
      5651 ± 33%    -100.0%       0.00        interrupts.CPU2.NMI:Non-maskable_interrupts
      5651 ± 33%    -100.0%       0.00        interrupts.CPU2.PMI:Performance_monitoring_interrupts
    501.17 ± 18%     -24.5%     378.50 ± 13%  interrupts.CPU2.RES:Rescheduling_interrupts
      6914 ± 20%    -100.0%       0.00        interrupts.CPU20.NMI:Non-maskable_interrupts
      6914 ± 20%    -100.0%       0.00        interrupts.CPU20.PMI:Performance_monitoring_interrupts
    319.17           +19.1%     380.17 ±  5%  interrupts.CPU20.RES:Rescheduling_interrupts
      7544          -100.0%       0.00        interrupts.CPU21.NMI:Non-maskable_interrupts
      7544          -100.0%       0.00        interrupts.CPU21.PMI:Performance_monitoring_interrupts
    332.00 ±  7%     +23.4%     409.67 ± 15%  interrupts.CPU21.RES:Rescheduling_interrupts
      7544          -100.0%       0.00        interrupts.CPU22.NMI:Non-maskable_interrupts
      7544          -100.0%       0.00        interrupts.CPU22.PMI:Performance_monitoring_interrupts
    318.00           +52.5%     485.00 ± 41%  interrupts.CPU22.RES:Rescheduling_interrupts
      7544          -100.0%       0.00        interrupts.CPU23.NMI:Non-maskable_interrupts
      7544          -100.0%       0.00        interrupts.CPU23.PMI:Performance_monitoring_interrupts
    328.17 ±  6%     +17.1%     384.33 ±  7%  interrupts.CPU23.RES:Rescheduling_interrupts
      6914 ± 20%    -100.0%       0.00        interrupts.CPU24.NMI:Non-maskable_interrupts
      6914 ± 20%    -100.0%       0.00        interrupts.CPU24.PMI:Performance_monitoring_interrupts
    322.33           +17.9%     380.17 ±  7%  interrupts.CPU24.RES:Rescheduling_interrupts
      5655 ± 33%    -100.0%       0.00        interrupts.CPU25.NMI:Non-maskable_interrupts
      5655 ± 33%    -100.0%       0.00        interrupts.CPU25.PMI:Performance_monitoring_interrupts
      1325 ± 23%     -46.9%     704.17 ± 39%  interrupts.CPU26.CAL:Function_call_interrupts
      6526 ± 28%    -100.0%       0.00        interrupts.CPU26.NMI:Non-maskable_interrupts
      6526 ± 28%    -100.0%       0.00        interrupts.CPU26.PMI:Performance_monitoring_interrupts
      1712 ± 39%     -66.3%     577.83 ±  3%  interrupts.CPU27.CAL:Function_call_interrupts
      6527 ± 28%    -100.0%       1.00 ±223%  interrupts.CPU27.NMI:Non-maskable_interrupts
      6527 ± 28%    -100.0%       1.00 ±223%  interrupts.CPU27.PMI:Performance_monitoring_interrupts
    732.83 ± 41%     -54.4%     334.50 ±  5%  interrupts.CPU27.RES:Rescheduling_interrupts
      1110 ± 26%     -47.9%     578.67 ±  3%  interrupts.CPU28.CAL:Function_call_interrupts
      6526 ± 28%    -100.0%       0.00        interrupts.CPU28.NMI:Non-maskable_interrupts
      6526 ± 28%    -100.0%       0.00        interrupts.CPU28.PMI:Performance_monitoring_interrupts
    461.67 ± 17%     -27.7%     334.00 ±  5%  interrupts.CPU28.RES:Rescheduling_interrupts
    675.50 ±  5%     -16.7%     563.00 ±  4%  interrupts.CPU29.CAL:Function_call_interrupts
      7180 ± 20%    -100.0%       0.00        interrupts.CPU29.NMI:Non-maskable_interrupts
      7180 ± 20%    -100.0%       0.00        interrupts.CPU29.PMI:Performance_monitoring_interrupts
      6281 ± 28%    -100.0%       0.00        interrupts.CPU3.NMI:Non-maskable_interrupts
      6281 ± 28%    -100.0%       0.00        interrupts.CPU3.PMI:Performance_monitoring_interrupts
      7181 ± 20%    -100.0%       1.00 ±223%  interrupts.CPU30.NMI:Non-maskable_interrupts
      7181 ± 20%    -100.0%       1.00 ±223%  interrupts.CPU30.PMI:Performance_monitoring_interrupts
      7181 ± 20%    -100.0%       0.00        interrupts.CPU31.NMI:Non-maskable_interrupts
      7181 ± 20%    -100.0%       0.00        interrupts.CPU31.PMI:Performance_monitoring_interrupts
      7180 ± 20%    -100.0%       0.00        interrupts.CPU32.NMI:Non-maskable_interrupts
      7180 ± 20%    -100.0%       0.00        interrupts.CPU32.PMI:Performance_monitoring_interrupts
      6528 ± 28%    -100.0%       0.00        interrupts.CPU33.NMI:Non-maskable_interrupts
      6528 ± 28%    -100.0%       0.00        interrupts.CPU33.PMI:Performance_monitoring_interrupts
      7182 ± 20%    -100.0%       0.00        interrupts.CPU34.NMI:Non-maskable_interrupts
      7182 ± 20%    -100.0%       0.00        interrupts.CPU34.PMI:Performance_monitoring_interrupts
      5875 ± 33%    -100.0%       0.00        interrupts.CPU35.NMI:Non-maskable_interrupts
      5875 ± 33%    -100.0%       0.00        interrupts.CPU35.PMI:Performance_monitoring_interrupts
      5874 ± 33%    -100.0%       0.00        interrupts.CPU36.NMI:Non-maskable_interrupts
      5874 ± 33%    -100.0%       0.00        interrupts.CPU36.PMI:Performance_monitoring_interrupts
      5873 ± 33%    -100.0%       0.00        interrupts.CPU37.NMI:Non-maskable_interrupts
      5873 ± 33%    -100.0%       0.00        interrupts.CPU37.PMI:Performance_monitoring_interrupts
      5873 ± 33%    -100.0%       0.00        interrupts.CPU38.NMI:Non-maskable_interrupts
      5873 ± 33%    -100.0%       0.00        interrupts.CPU38.PMI:Performance_monitoring_interrupts
      5222 ± 35%    -100.0%       0.00        interrupts.CPU39.NMI:Non-maskable_interrupts
      5222 ± 35%    -100.0%       0.00        interrupts.CPU39.PMI:Performance_monitoring_interrupts
      7538          -100.0%       0.00        interrupts.CPU4.NMI:Non-maskable_interrupts
      7538          -100.0%       0.00        interrupts.CPU4.PMI:Performance_monitoring_interrupts
      6528 ± 28%    -100.0%       0.00        interrupts.CPU40.NMI:Non-maskable_interrupts
      6528 ± 28%    -100.0%       0.00        interrupts.CPU40.PMI:Performance_monitoring_interrupts
    315.00           +17.4%     369.83 ± 12%  interrupts.CPU40.RES:Rescheduling_interrupts
      6528 ± 28%    -100.0%       0.00        interrupts.CPU41.NMI:Non-maskable_interrupts
      6528 ± 28%    -100.0%       0.00        interrupts.CPU41.PMI:Performance_monitoring_interrupts
      6528 ± 28%    -100.0%       0.00        interrupts.CPU42.NMI:Non-maskable_interrupts
      6528 ± 28%    -100.0%       0.00        interrupts.CPU42.PMI:Performance_monitoring_interrupts
      7181 ± 20%    -100.0%       0.00        interrupts.CPU43.NMI:Non-maskable_interrupts
      7181 ± 20%    -100.0%       0.00        interrupts.CPU43.PMI:Performance_monitoring_interrupts
    314.33           +25.9%     395.67 ± 28%  interrupts.CPU43.RES:Rescheduling_interrupts
      6529 ± 28%    -100.0%       0.00        interrupts.CPU44.NMI:Non-maskable_interrupts
      6529 ± 28%    -100.0%       0.00        interrupts.CPU44.PMI:Performance_monitoring_interrupts
    313.17           +10.8%     347.00 ±  7%  interrupts.CPU44.RES:Rescheduling_interrupts
      6527 ± 28%    -100.0%       0.00        interrupts.CPU45.NMI:Non-maskable_interrupts
      6527 ± 28%    -100.0%       0.00        interrupts.CPU45.PMI:Performance_monitoring_interrupts
      6527 ± 28%    -100.0%       0.00        interrupts.CPU46.NMI:Non-maskable_interrupts
      6527 ± 28%    -100.0%       0.00        interrupts.CPU46.PMI:Performance_monitoring_interrupts
      6528 ± 28%    -100.0%       0.00        interrupts.CPU47.NMI:Non-maskable_interrupts
      6528 ± 28%    -100.0%       0.00        interrupts.CPU47.PMI:Performance_monitoring_interrupts
      6528 ± 28%    -100.0%       0.00        interrupts.CPU48.NMI:Non-maskable_interrupts
      6528 ± 28%    -100.0%       0.00        interrupts.CPU48.PMI:Performance_monitoring_interrupts
    317.00 ±  3%      +9.1%     346.00 ±  5%  interrupts.CPU48.RES:Rescheduling_interrupts
      5874 ± 33%    -100.0%       0.00        interrupts.CPU49.NMI:Non-maskable_interrupts
      5874 ± 33%    -100.0%       0.00        interrupts.CPU49.PMI:Performance_monitoring_interrupts
    318.67 ±  3%     +23.7%     394.33 ± 25%  interrupts.CPU49.RES:Rescheduling_interrupts
      6910 ± 20%    -100.0%       0.00        interrupts.CPU5.NMI:Non-maskable_interrupts
      6910 ± 20%    -100.0%       0.00        interrupts.CPU5.PMI:Performance_monitoring_interrupts
      5874 ± 33%    -100.0%       0.00        interrupts.CPU50.NMI:Non-maskable_interrupts
      5874 ± 33%    -100.0%       0.00        interrupts.CPU50.PMI:Performance_monitoring_interrupts
      7179 ± 20%    -100.0%       0.00        interrupts.CPU51.NMI:Non-maskable_interrupts
      7179 ± 20%    -100.0%       0.00        interrupts.CPU51.PMI:Performance_monitoring_interrupts
      1361 ± 16%     -49.9%     682.00 ± 15%  interrupts.CPU52.CAL:Function_call_interrupts
      6916 ± 20%    -100.0%       0.00        interrupts.CPU52.NMI:Non-maskable_interrupts
      6916 ± 20%    -100.0%       0.00        interrupts.CPU52.PMI:Performance_monitoring_interrupts
      5463 ± 43%     -85.3%     801.83 ± 59%  interrupts.CPU53.CAL:Function_call_interrupts
      7545          -100.0%       0.00        interrupts.CPU53.NMI:Non-maskable_interrupts
      7545          -100.0%       0.00        interrupts.CPU53.PMI:Performance_monitoring_interrupts
      6538 ± 41%     -89.4%     694.00 ± 28%  interrupts.CPU54.CAL:Function_call_interrupts
      7541          -100.0%       0.00        interrupts.CPU54.NMI:Non-maskable_interrupts
      7541          -100.0%       0.00        interrupts.CPU54.PMI:Performance_monitoring_interrupts
      2701 ± 33%     -70.1%     808.67 ± 33%  interrupts.CPU55.CAL:Function_call_interrupts
      6913 ± 20%    -100.0%       0.00        interrupts.CPU55.NMI:Non-maskable_interrupts
      6913 ± 20%    -100.0%       0.00        interrupts.CPU55.PMI:Performance_monitoring_interrupts
      6921 ± 20%    -100.0%       0.00        interrupts.CPU56.NMI:Non-maskable_interrupts
      6921 ± 20%    -100.0%       0.00        interrupts.CPU56.PMI:Performance_monitoring_interrupts
      6912 ± 20%    -100.0%       0.00        interrupts.CPU57.NMI:Non-maskable_interrupts
      6912 ± 20%    -100.0%       0.00        interrupts.CPU57.PMI:Performance_monitoring_interrupts
      6913 ± 20%    -100.0%       0.00        interrupts.CPU58.NMI:Non-maskable_interrupts
      6913 ± 20%    -100.0%       0.00        interrupts.CPU58.PMI:Performance_monitoring_interrupts
      6283 ± 28%    -100.0%       0.00        interrupts.CPU59.NMI:Non-maskable_interrupts
      6283 ± 28%    -100.0%       0.00        interrupts.CPU59.PMI:Performance_monitoring_interrupts
    398.83 ± 11%    +385.6%       1936 ± 87%  interrupts.CPU59.RES:Rescheduling_interrupts
      6913 ± 20%    -100.0%       0.00        interrupts.CPU6.NMI:Non-maskable_interrupts
      6913 ± 20%    -100.0%       0.00        interrupts.CPU6.PMI:Performance_monitoring_interrupts
      6910 ± 20%    -100.0%       0.00        interrupts.CPU60.NMI:Non-maskable_interrupts
      6910 ± 20%    -100.0%       0.00        interrupts.CPU60.PMI:Performance_monitoring_interrupts
      6910 ± 20%    -100.0%       0.00        interrupts.CPU61.NMI:Non-maskable_interrupts
      6910 ± 20%    -100.0%       0.00        interrupts.CPU61.PMI:Performance_monitoring_interrupts
    581.67 ± 42%    +177.0%       1611 ± 46%  interrupts.CPU61.RES:Rescheduling_interrupts
      6281 ± 28%    -100.0%       0.00        interrupts.CPU62.NMI:Non-maskable_interrupts
      6281 ± 28%    -100.0%       0.00        interrupts.CPU62.PMI:Performance_monitoring_interrupts
    547.83 ± 14%     +41.8%     777.00 ± 20%  interrupts.CPU63.CAL:Function_call_interrupts
      7536          -100.0%       0.00        interrupts.CPU63.NMI:Non-maskable_interrupts
      7536          -100.0%       0.00        interrupts.CPU63.PMI:Performance_monitoring_interrupts
    347.83 ± 10%    +470.8%       1985 ± 46%  interrupts.CPU63.RES:Rescheduling_interrupts
      7537          -100.0%       0.00        interrupts.CPU64.NMI:Non-maskable_interrupts
      7537          -100.0%       0.00        interrupts.CPU64.PMI:Performance_monitoring_interrupts
      7537          -100.0%       0.00        interrupts.CPU65.NMI:Non-maskable_interrupts
      7537          -100.0%       0.00        interrupts.CPU65.PMI:Performance_monitoring_interrupts
      7538          -100.0%       0.00        interrupts.CPU66.NMI:Non-maskable_interrupts
      7538          -100.0%       0.00        interrupts.CPU66.PMI:Performance_monitoring_interrupts
      7543          -100.0%       0.00        interrupts.CPU67.NMI:Non-maskable_interrupts
      7543          -100.0%       0.00        interrupts.CPU67.PMI:Performance_monitoring_interrupts
    385.17 ± 28%    +410.0%       1964 ± 55%  interrupts.CPU67.RES:Rescheduling_interrupts
      7536          -100.0%       0.00        interrupts.CPU68.NMI:Non-maskable_interrupts
      7536          -100.0%       0.00        interrupts.CPU68.PMI:Performance_monitoring_interrupts
    347.00 ± 13%    +370.2%       1631 ± 79%  interrupts.CPU68.RES:Rescheduling_interrupts
      7537          -100.0%       0.00        interrupts.CPU69.NMI:Non-maskable_interrupts
      7537          -100.0%       0.00        interrupts.CPU69.PMI:Performance_monitoring_interrupts
    376.67 ± 30%    +524.3%       2351 ±108%  interrupts.CPU69.RES:Rescheduling_interrupts
      6911 ± 20%    -100.0%       0.00        interrupts.CPU7.NMI:Non-maskable_interrupts
      6911 ± 20%    -100.0%       0.00        interrupts.CPU7.PMI:Performance_monitoring_interrupts
      6276 ± 28%    -100.0%       0.00        interrupts.CPU70.NMI:Non-maskable_interrupts
      6276 ± 28%    -100.0%       0.00        interrupts.CPU70.PMI:Performance_monitoring_interrupts
    439.17 ± 52%    +374.2%       2082 ± 43%  interrupts.CPU70.RES:Rescheduling_interrupts
    571.83           +81.9%       1040 ± 59%  interrupts.CPU71.CAL:Function_call_interrupts
      6276 ± 28%    -100.0%       0.00        interrupts.CPU71.NMI:Non-maskable_interrupts
      6276 ± 28%    -100.0%       0.00        interrupts.CPU71.PMI:Performance_monitoring_interrupts
    325.50 ±  2%    +525.2%       2035 ± 43%  interrupts.CPU71.RES:Rescheduling_interrupts
      6276 ± 28%    -100.0%       0.00        interrupts.CPU72.NMI:Non-maskable_interrupts
      6276 ± 28%    -100.0%       0.00        interrupts.CPU72.PMI:Performance_monitoring_interrupts
      6276 ± 28%    -100.0%       0.00        interrupts.CPU73.NMI:Non-maskable_interrupts
      6276 ± 28%    -100.0%       0.00        interrupts.CPU73.PMI:Performance_monitoring_interrupts
    567.50           +52.8%     867.17 ± 34%  interrupts.CPU74.CAL:Function_call_interrupts
      6908 ± 20%    -100.0%       0.00        interrupts.CPU74.NMI:Non-maskable_interrupts
      6908 ± 20%    -100.0%       0.00        interrupts.CPU74.PMI:Performance_monitoring_interrupts
    321.00          +685.6%       2521 ± 68%  interrupts.CPU74.RES:Rescheduling_interrupts
      6907 ± 20%    -100.0%       0.00        interrupts.CPU75.NMI:Non-maskable_interrupts
      6907 ± 20%    -100.0%       0.00        interrupts.CPU75.PMI:Performance_monitoring_interrupts
      6906 ± 20%    -100.0%       1.00 ±223%  interrupts.CPU76.NMI:Non-maskable_interrupts
      6906 ± 20%    -100.0%       1.00 ±223%  interrupts.CPU76.PMI:Performance_monitoring_interrupts
    323.00 ±  2%    +724.5%       2663 ±110%  interrupts.CPU76.RES:Rescheduling_interrupts
      7534          -100.0%       0.00        interrupts.CPU77.NMI:Non-maskable_interrupts
      7534          -100.0%       0.00        interrupts.CPU77.PMI:Performance_monitoring_interrupts
    348.50 ± 18%    +833.9%       3254 ± 63%  interrupts.CPU77.RES:Rescheduling_interrupts
      1637 ± 25%     -57.0%     703.50 ± 29%  interrupts.CPU78.CAL:Function_call_interrupts
      7833          -100.0%       0.00        interrupts.CPU78.NMI:Non-maskable_interrupts
      7833          -100.0%       0.00        interrupts.CPU78.PMI:Performance_monitoring_interrupts
      6155 ± 29%     -86.1%     858.50 ± 49%  interrupts.CPU79.CAL:Function_call_interrupts
      7834          -100.0%       0.00        interrupts.CPU79.NMI:Non-maskable_interrupts
      7834          -100.0%       0.00        interrupts.CPU79.PMI:Performance_monitoring_interrupts
      6915 ± 20%    -100.0%       0.00        interrupts.CPU8.NMI:Non-maskable_interrupts
      6915 ± 20%    -100.0%       0.00        interrupts.CPU8.PMI:Performance_monitoring_interrupts
      6996 ± 48%     -88.5%     805.50 ± 43%  interrupts.CPU80.CAL:Function_call_interrupts
      7835          -100.0%       0.00        interrupts.CPU80.NMI:Non-maskable_interrupts
      7835          -100.0%       0.00        interrupts.CPU80.PMI:Performance_monitoring_interrupts
      2300 ± 58%     -71.8%     649.67 ± 58%  interrupts.CPU80.RES:Rescheduling_interrupts
      1938 ± 43%     -70.1%     579.17 ±  5%  interrupts.CPU81.CAL:Function_call_interrupts
      7834          -100.0%       0.00        interrupts.CPU81.NMI:Non-maskable_interrupts
      7834          -100.0%       0.00        interrupts.CPU81.PMI:Performance_monitoring_interrupts
      1473 ± 58%     -58.7%     608.17 ± 12%  interrupts.CPU82.CAL:Function_call_interrupts
      7834          -100.0%       0.00        interrupts.CPU82.NMI:Non-maskable_interrupts
      7834          -100.0%       0.00        interrupts.CPU82.PMI:Performance_monitoring_interrupts
    801.67 ± 24%     -27.1%     584.67 ±  7%  interrupts.CPU83.CAL:Function_call_interrupts
      7834          -100.0%       0.00        interrupts.CPU83.NMI:Non-maskable_interrupts
      7834          -100.0%       0.00        interrupts.CPU83.PMI:Performance_monitoring_interrupts
      7833          -100.0%       0.00        interrupts.CPU84.NMI:Non-maskable_interrupts
      7833          -100.0%       0.00        interrupts.CPU84.PMI:Performance_monitoring_interrupts
      7182 ± 20%    -100.0%       0.00        interrupts.CPU85.NMI:Non-maskable_interrupts
      7182 ± 20%    -100.0%       0.00        interrupts.CPU85.PMI:Performance_monitoring_interrupts
      7181 ± 20%    -100.0%       0.00        interrupts.CPU86.NMI:Non-maskable_interrupts
      7181 ± 20%    -100.0%       0.00        interrupts.CPU86.PMI:Performance_monitoring_interrupts
      6528 ± 28%    -100.0%       0.00        interrupts.CPU87.NMI:Non-maskable_interrupts
      6528 ± 28%    -100.0%       0.00        interrupts.CPU87.PMI:Performance_monitoring_interrupts
      7180 ± 20%    -100.0%       0.00        interrupts.CPU88.NMI:Non-maskable_interrupts
      7180 ± 20%    -100.0%       0.00        interrupts.CPU88.PMI:Performance_monitoring_interrupts
      7834          -100.0%       0.00        interrupts.CPU89.NMI:Non-maskable_interrupts
      7834          -100.0%       0.00        interrupts.CPU89.PMI:Performance_monitoring_interrupts
      7539          -100.0%       0.00        interrupts.CPU9.NMI:Non-maskable_interrupts
      7539          -100.0%       0.00        interrupts.CPU9.PMI:Performance_monitoring_interrupts
      7835          -100.0%       1.00 ±223%  interrupts.CPU90.NMI:Non-maskable_interrupts
      7835          -100.0%       1.00 ±223%  interrupts.CPU90.PMI:Performance_monitoring_interrupts
      7835          -100.0%       0.00        interrupts.CPU91.NMI:Non-maskable_interrupts
      7835          -100.0%       0.00        interrupts.CPU91.PMI:Performance_monitoring_interrupts
      7836          -100.0%       0.00        interrupts.CPU92.NMI:Non-maskable_interrupts
      7836          -100.0%       0.00        interrupts.CPU92.PMI:Performance_monitoring_interrupts
    333.17 ±  5%     +89.7%     632.00 ± 38%  interrupts.CPU92.RES:Rescheduling_interrupts
      7834          -100.0%       0.00        interrupts.CPU93.NMI:Non-maskable_interrupts
      7834          -100.0%       0.00        interrupts.CPU93.PMI:Performance_monitoring_interrupts
    334.67 ±  9%     +59.2%     532.83 ± 26%  interrupts.CPU93.RES:Rescheduling_interrupts
      7834          -100.0%       0.00        interrupts.CPU94.NMI:Non-maskable_interrupts
      7834          -100.0%       0.00        interrupts.CPU94.PMI:Performance_monitoring_interrupts
      7834          -100.0%       0.00        interrupts.CPU95.NMI:Non-maskable_interrupts
      7834          -100.0%       0.00        interrupts.CPU95.PMI:Performance_monitoring_interrupts
      7834          -100.0%       0.00        interrupts.CPU96.NMI:Non-maskable_interrupts
      7834          -100.0%       0.00        interrupts.CPU96.PMI:Performance_monitoring_interrupts
    326.33 ±  2%    +205.5%     996.83 ±103%  interrupts.CPU96.RES:Rescheduling_interrupts
      7833          -100.0%       0.00        interrupts.CPU97.NMI:Non-maskable_interrupts
      7833          -100.0%       0.00        interrupts.CPU97.PMI:Performance_monitoring_interrupts
      7834          -100.0%       0.00        interrupts.CPU98.NMI:Non-maskable_interrupts
      7834          -100.0%       0.00        interrupts.CPU98.PMI:Performance_monitoring_interrupts
      7833          -100.0%       0.00        interrupts.CPU99.NMI:Non-maskable_interrupts
      7833          -100.0%       0.00        interrupts.CPU99.PMI:Performance_monitoring_interrupts
    192.67 ±  4%    -100.0%       0.00        interrupts.IWI:IRQ_work_interrupts
    729455 ±  4%    -100.0%       4.00 ± 70%  interrupts.NMI:Non-maskable_interrupts
    729455 ±  4%    -100.0%       4.00 ± 70%  interrupts.PMI:Performance_monitoring_interrupts
     51460 ±  7%     +86.2%      95806 ±  2%  interrupts.RES:Rescheduling_interrupts
    314.50 ±  3%     -28.1%     226.17 ± 30%  interrupts.TLB:TLB_shootdowns

***************************************************************************************************
lkp-ivb-2ep1: 48 threads Intel(R) Xeon(R) CPU E5-2697 v2 @ 2.70GHz with 112G memory
=========================================================================================
compiler/cpufreq_governor/kconfig/mode/nr_task/rootfs/tbox_group/test/testcase/ucode:
gcc-9/performance/x86_64-rhel-8.3/thread/100%/debian-10.4-x86_64-20200603.cgz/lkp-ivb-2ep1/poll1/will-it-scale/0x42e

commit:
dde38d5b80 ("kernel: don't include PF_IO_WORKERs as part of same_thread_group()") 43b2a76b1a ("proc: don't show PF_IO_WORKER threads as threads in /proc/<pid>/task/")

dde38d5b805358c7 43b2a76b1a5abcc9833b463bef1
---------------- ---------------------------
%stddev %change %stddev
\ | \

  59738282            -2.2%   58422246        will-it-scale.48.threads
   1244547            -2.2%    1217129        will-it-scale.per_thread_ops
  59738282            -2.2%   58422246        will-it-scale.workload
      2.54            +3.5        6.04 ±  2%  mpstat.cpu.all.idle%
      0.02 ± 14%      +0.0        0.04        mpstat.cpu.all.soft%
   1379034           +56.6%    2159868        vmstat.memory.cache
      1272 ±  2%     +50.3%       1912        vmstat.system.cs
    328771 ±  6%     +75.5%     577027 ± 20%  numa-numastat.node0.local_node
    364262 ±  5%     +65.8%     603798 ± 21%  numa-numastat.node0.numa_hit
    356834 ±  5%     +68.2%     600293 ± 20%  numa-numastat.node1.local_node
    364821 ±  5%     +69.1%     616999 ± 21%  numa-numastat.node1.numa_hit
    101943          +765.8%     882625 ±  4%  meminfo.Active
    101943          +765.8%     882625 ±  4%  meminfo.Active(anon)
   1297433           +60.0%    2075764        meminfo.Cached
    815318          +100.9%    1638128 ±  2%  meminfo.Committed_AS
   2185803           +43.6%    3138095        meminfo.Memused
    116458          +668.3%     894794 ±  4%  meminfo.Shmem
   2409774           +30.2%    3138133        meminfo.max_used_kB
     13355 ± 41%    +822.5%     123204 ± 15%  numa-vmstat.node0.nr_active_anon
    161959 ±  3%     +67.0%     270513 ±  7%  numa-vmstat.node0.nr_file_pages
     14956 ± 39%    +736.5%     125114 ± 15%  numa-vmstat.node0.nr_shmem
     13355 ± 41%    +822.5%     123204 ± 15%  numa-vmstat.node0.nr_zone_active_anon
     12128 ± 46%    +704.6%      97588 ± 11%  numa-vmstat.node1.nr_active_anon
    162378 ±  3%     +53.1%     248564 ±  5%  numa-vmstat.node1.nr_file_pages
     14137 ± 42%    +598.3%      98721 ± 11%  numa-vmstat.node1.nr_shmem
     12128 ± 46%    +704.6%      97588 ± 11%  numa-vmstat.node1.nr_zone_active_anon
      5488 ±  3%     +33.7%       7337 ±  3%  slabinfo.kmalloc-256.active_objs
      5529 ±  3%     +33.7%       7392 ±  3%  slabinfo.kmalloc-256.num_objs
      9068 ±  2%      -9.6%       8198        slabinfo.proc_inode_cache.active_objs
      9068 ±  2%      -9.6%       8198        slabinfo.proc_inode_cache.num_objs
     17898           +15.2%      20618        slabinfo.radix_tree_node.active_objs
    638.83           +15.2%     736.00        slabinfo.radix_tree_node.active_slabs
     17898           +15.2%      20618        slabinfo.radix_tree_node.num_objs
    638.83           +15.2%     736.00        slabinfo.radix_tree_node.num_slabs
     53322 ± 41%    +823.0%     492188 ± 15%  numa-meminfo.node0.Active
     53322 ± 41%    +823.0%     492188 ± 15%  numa-meminfo.node0.Active(anon)
    647688 ±  3%     +67.0%    1081422 ±  7%  numa-meminfo.node0.FilePages
   1128100 ±  3%     +42.7%    1609970 ±  5%  numa-meminfo.node0.MemUsed
     59674 ± 39%    +737.6%     499822 ± 15%  numa-meminfo.node0.Shmem
     48465 ± 46%    +704.4%     389842 ± 11%  numa-meminfo.node1.Active
     48465 ± 46%    +704.4%     389842 ± 11%  numa-meminfo.node1.Active(anon)
    649545 ±  3%     +53.0%     993748 ±  5%  numa-meminfo.node1.FilePages
   1057395 ±  3%     +44.5%    1527564 ±  4%  numa-meminfo.node1.MemUsed
     56584 ± 42%    +597.0%     394377 ± 11%  numa-meminfo.node1.Shmem
      9204 ± 93%    +100.9%      18488 ± 64%  sched_debug.cfs_rq:/.load.stddev
      1403 ±  4%     +39.6%       1958 ±  2%  sched_debug.cfs_rq:/.runnable_avg.max
    104.09 ±  5%    +127.0%     236.30 ±  3%  sched_debug.cfs_rq:/.runnable_avg.stddev
      1112 ±  2%     +19.1%       1325 ±  2%  sched_debug.cfs_rq:/.util_avg.max
    835.58           -28.3%     598.81 ±  5%  sched_debug.cfs_rq:/.util_avg.min
     61.94 ±  5%     +87.5%     116.16 ±  5%  sched_debug.cfs_rq:/.util_avg.stddev
      1022           +60.9%       1645 ± 12%  sched_debug.cfs_rq:/.util_est_enqueued.max
    105.35 ± 65%    +112.0%     223.36 ± 12%  sched_debug.cfs_rq:/.util_est_enqueued.stddev
    707908 ±  9%     +24.2%     879248 ±  5%  sched_debug.cpu.avg_idle.avg
   1063850 ± 13%     +37.8%    1465737 ±  6%  sched_debug.cpu.avg_idle.max
    126226 ± 11%     +26.1%     159118 ± 15%  sched_debug.cpu.avg_idle.stddev
      0.00 ± 20%     -18.6%       0.00 ±  2%  sched_debug.cpu.next_balance.stddev
      0.23 ±  5%     +28.7%       0.29 ±  4%  sched_debug.cpu.nr_running.stddev
      6079 ±  2%     +32.7%       8069        sched_debug.cpu.nr_switches.avg
     20560 ±  5%     -25.0%      15421 ±  7%  sched_debug.cpu.nr_switches.max
      2039 ± 17%     +95.6%       3988 ± 13%  sched_debug.cpu.nr_switches.min
      4387 ±  3%     -42.5%       2520 ± 19%  sched_debug.cpu.nr_switches.stddev
     25464          +766.1%     220557 ±  4%  proc-vmstat.nr_active_anon
     63374            +2.9%      65240        proc-vmstat.nr_anon_pages
    324325           +60.0%     518843        proc-vmstat.nr_file_pages
     66898            +1.8%      68101        proc-vmstat.nr_inactive_anon
      9338            -5.6%       8819        proc-vmstat.nr_mapped
      1084            +9.0%       1182        proc-vmstat.nr_page_table_pages
     29081          +668.9%     223601 ±  4%  proc-vmstat.nr_shmem
     25464          +766.1%     220557 ±  4%  proc-vmstat.nr_zone_active_anon
     66898            +1.8%      68101        proc-vmstat.nr_zone_inactive_anon
     10303 ± 28%     -56.9%       4446 ± 52%  proc-vmstat.numa_hint_faults_local
    760864           +64.3%    1249911 ±  2%  proc-vmstat.numa_hit
    717323           +68.2%    1206367 ±  2%  proc-vmstat.numa_local
    145574 ± 13%     -42.7%      83394 ± 28%  proc-vmstat.numa_pte_updates
     37598           -20.5%      29891 ±  4%  proc-vmstat.pgactivate
    816352           +71.9%    1403351 ±  2%  proc-vmstat.pgalloc_normal
    893418            -6.9%     831535        proc-vmstat.pgfault
    762113           +14.3%     871330 ±  2%  proc-vmstat.pgfree
     50605            -6.2%      47486 ±  3%  proc-vmstat.pgreuse
     98.56           -98.6        0.00        perf-profile.calltrace.cycles-pp.__poll
     51.59           -51.6        0.00        perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__poll
     32.22           -32.2        0.00        perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_safe_stack.__poll
     26.60           -26.6        0.00        perf-profile.calltrace.cycles-pp.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe.__poll
     23.39           -23.4        0.00        perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__poll
     20.60           -20.6        0.00        perf-profile.calltrace.cycles-pp.__x64_sys_poll.do_syscall_64.entry_SYSCALL_64_after_hwframe.__poll
     17.01           -17.0        0.00        perf-profile.calltrace.cycles-pp.do_sys_poll.__x64_sys_poll.do_syscall_64.entry_SYSCALL_64_after_hwframe.__poll
     14.54           -14.5        0.00        perf-profile.calltrace.cycles-pp.syscall_return_via_sysret.__poll
     13.78           -13.8        0.00        perf-profile.calltrace.cycles-pp.__entry_text_start.__poll
     98.80           -98.8        0.00        perf-profile.children.cycles-pp.__poll
     51.95           -52.0        0.00        perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
     27.71           -27.7        0.00        perf-profile.children.cycles-pp.__entry_text_start
     26.72           -26.7        0.00        perf-profile.children.cycles-pp.syscall_exit_to_user_mode
     23.59           -23.6        0.00        perf-profile.children.cycles-pp.do_syscall_64
     20.71           -20.7        0.00        perf-profile.children.cycles-pp.__x64_sys_poll
     17.47           -17.5        0.00        perf-profile.children.cycles-pp.do_sys_poll
     16.54           -16.5        0.00        perf-profile.children.cycles-pp.entry_SYSCALL_64_safe_stack
     16.32           -16.3        0.00        perf-profile.children.cycles-pp.syscall_return_via_sysret
     25.93           -25.9        0.00        perf-profile.self.cycles-pp.__entry_text_start
     25.01           -25.0        0.00        perf-profile.self.cycles-pp.syscall_exit_to_user_mode
     16.28           -16.3        0.00        perf-profile.self.cycles-pp.syscall_return_via_sysret
      6.99            -7.0        0.00        perf-profile.self.cycles-pp.do_sys_poll
     40603 ± 14%    +119.9%      89296 ± 21%  softirqs.CPU0.RCU
     39894 ± 14%     +72.5%      68812 ± 16%  softirqs.CPU1.RCU
     35093 ± 15%     +80.3%      63288 ± 14%  softirqs.CPU10.RCU
     35614 ± 15%     +90.9%      67979 ± 11%  softirqs.CPU11.RCU
     35000 ± 12%     +79.5%      62812 ±  7%  softirqs.CPU12.RCU
     31579 ± 11%     +98.5%      62671 ± 12%  softirqs.CPU13.RCU
     32174 ± 10%    +114.7%      69079 ± 13%  softirqs.CPU14.RCU
     31104 ± 11%    +109.4%      65136 ±  9%  softirqs.CPU15.RCU
     36318 ± 14%     +92.3%      69837 ±  7%  softirqs.CPU16.RCU
     32951 ± 18%    +105.5%      67709 ± 13%  softirqs.CPU17.RCU
     36326 ± 14%     +90.5%      69212 ±  6%  softirqs.CPU18.RCU
     34802 ± 13%     +89.3%      65899 ± 15%  softirqs.CPU19.RCU
     38578 ± 20%     +90.1%      73339 ± 19%  softirqs.CPU2.RCU
     35302 ± 14%     +87.9%      66341 ±  7%  softirqs.CPU20.RCU
     35113 ± 16%     +93.3%      67861        softirqs.CPU21.RCU
     35292 ± 15%     +90.3%      67170 ±  3%  softirqs.CPU22.RCU
     35612 ± 16%     +82.5%      64992 ±  5%  softirqs.CPU23.RCU
     31301 ± 16%    +149.5%      78086 ± 21%  softirqs.CPU24.RCU
     34412 ± 14%     +87.9%      64657 ± 20%  softirqs.CPU25.RCU
     32752 ± 14%    +101.7%      66067 ± 15%  softirqs.CPU26.RCU
     34903 ± 16%     +79.1%      62519 ± 15%  softirqs.CPU27.RCU
     34158 ± 16%     +76.9%      60417 ±  9%  softirqs.CPU28.RCU
     33748 ± 17%     +89.0%      63791 ± 20%  softirqs.CPU29.RCU
     38876 ± 15%     +73.5%      67453 ±  9%  softirqs.CPU3.RCU
     33417 ± 16%    +113.8%      71458 ± 17%  softirqs.CPU30.RCU
     33281 ± 16%     +83.8%      61177 ± 15%  softirqs.CPU31.RCU
     29887 ± 16%     +80.3%      53893 ± 12%  softirqs.CPU32.RCU
     30644 ± 16%     +94.5%      59592 ±  6%  softirqs.CPU33.RCU
     31044 ± 20%     +69.1%      52510 ± 18%  softirqs.CPU34.RCU
     29665 ± 14%     +86.4%      55307 ± 12%  softirqs.CPU35.RCU
     35628 ± 16%     +83.5%      65376 ± 12%  softirqs.CPU36.RCU
     32107 ± 18%    +103.8%      65444 ± 10%  softirqs.CPU37.RCU
     34464 ± 14%    +100.6%      69129 ±  7%  softirqs.CPU38.RCU
     34575 ± 15%     +92.1%      66419 ± 13%  softirqs.CPU39.RCU
     37352 ± 16%     +75.6%      65588 ±  9%  softirqs.CPU4.RCU
     38060 ± 15%     +87.5%      71346 ±  4%  softirqs.CPU40.RCU
     32929 ± 23%    +101.2%      66264 ± 17%  softirqs.CPU41.RCU
     37962 ± 13%     +90.0%      72112 ±  5%  softirqs.CPU42.RCU
     36473 ± 14%     +88.1%      68587 ±  9%  softirqs.CPU43.RCU
     36698 ± 14%     +81.1%      66468 ±  3%  softirqs.CPU44.RCU
     36159 ± 16%     +85.3%      67014 ±  3%  softirqs.CPU45.RCU
     36154 ± 16%     +86.9%      67582 ±  5%  softirqs.CPU46.RCU
     36546 ± 16%     +76.5%      64495 ±  4%  softirqs.CPU47.RCU
     36519 ± 15%     +91.3%      69871 ± 22%  softirqs.CPU5.RCU
     36807 ± 17%     +92.5%      70871 ± 13%  softirqs.CPU6.RCU
     35242 ± 16%     +83.1%      64543 ± 13%  softirqs.CPU7.RCU
     36761 ± 16%     +72.9%      63562 ±  7%  softirqs.CPU8.RCU
     37277 ± 11%     +92.7%      71838 ±  9%  softirqs.CPU9.RCU
   1677186 ± 14%     +90.5%    3194899        softirqs.RCU
     33514 ±  3%     +52.1%      50986        softirqs.TIMER
      0.09 ± 10%    +204.1%       0.26 ±  6%  perf-stat.i.MPKI
  8.85e+09            +3.4%  9.151e+09        perf-stat.i.branch-instructions
  63173486            +4.0%   65686220        perf-stat.i.branch-misses
      7.05            +2.4        9.48 ±  9%  perf-stat.i.cache-miss-rate%
    246018 ±  2%    +332.2%    1063300 ±  7%  perf-stat.i.cache-misses
   2755412 ±  3%    +299.4%   11004931 ±  6%  perf-stat.i.cache-references
      1244 ±  2%     +51.6%       1886        perf-stat.i.context-switches
      3.31            -3.3%       3.20        perf-stat.i.cpi
    111.57 ±  6%     +37.7%     153.66 ±  2%  perf-stat.i.cpu-migrations
    955562 ±  3%     -85.0%     143699 ±  8%  perf-stat.i.cycles-between-cache-misses
      1.17            -0.0        1.13        perf-stat.i.dTLB-load-miss-rate%
 1.108e+10            +2.5%  1.136e+10        perf-stat.i.dTLB-loads
      1.14            -0.0        1.11        perf-stat.i.dTLB-store-miss-rate%
 1.323e+08            -1.9%  1.298e+08        perf-stat.i.dTLB-store-misses
 4.299e+10            +3.4%  4.445e+10        perf-stat.i.instructions
      0.30            +3.5%       0.31        perf-stat.i.ipc
      0.78 ± 25%     -64.3%       0.28 ± 47%  perf-stat.i.major-faults
      1.20 ±  7%     -83.2%       0.20 ± 52%  perf-stat.i.metric.K/sec
    655.44            +2.2%     669.80        perf-stat.i.metric.M/sec
      2858            -7.1%       2654        perf-stat.i.minor-faults
     46.94            +1.3       48.26        perf-stat.i.node-load-miss-rate%
     93609 ±  2%    +365.9%     436129 ± 11%  perf-stat.i.node-load-misses
    116813 ±  2%    +301.2%     468642 ± 11%  perf-stat.i.node-loads
     33.38 ±  6%      +9.2       42.59        perf-stat.i.node-store-miss-rate%
     50561 ±  5%    +725.2%     417237 ±  7%  perf-stat.i.node-store-misses
    116603 ± 11%    +383.2%     563385 ±  6%  perf-stat.i.node-stores
      2859            -7.2%       2654        perf-stat.i.page-faults
      0.06 ±  3%    +285.5%       0.25 ±  6%  perf-stat.overall.MPKI
      3.31            -3.3%       3.21        perf-stat.overall.cpi
    576612 ±  2%     -76.6%     134705 ±  7%  perf-stat.overall.cycles-between-cache-misses
      1.17            -0.0        1.14        perf-stat.overall.dTLB-load-miss-rate%
      1.15            -0.0        1.12        perf-stat.overall.dTLB-store-miss-rate%
      0.30            +3.4%       0.31        perf-stat.overall.ipc
     44.41            +3.8       48.20        perf-stat.overall.node-load-miss-rate%
     30.38 ±  8%     +12.1       42.53        perf-stat.overall.node-store-miss-rate%
    216892            +5.7%     229166        perf-stat.overall.path-length
 8.821e+09            +3.4%   9.12e+09        perf-stat.ps.branch-instructions
  62993650            +3.9%   65475101        perf-stat.ps.branch-misses
    246469 ±  2%    +330.2%    1060339 ±  7%  perf-stat.ps.cache-misses
   2753111 ±  3%    +298.5%   10970294 ±  6%  perf-stat.ps.cache-references
      1240 ±  2%     +51.5%       1880        perf-stat.ps.context-switches
    111.34 ±  6%     +37.6%     153.15 ±  2%  perf-stat.ps.cpu-migrations
 1.105e+10            +2.5%  1.132e+10        perf-stat.ps.dTLB-loads
 1.319e+08            -1.9%  1.293e+08        perf-stat.ps.dTLB-store-misses
 4.285e+10            +3.4%   4.43e+10        perf-stat.ps.instructions
      0.78 ± 25%     -64.3%       0.28 ± 47%  perf-stat.ps.major-faults
      2851            -7.2%       2646        perf-stat.ps.minor-faults
     93481 ±  2%    +365.0%     434677 ± 11%  perf-stat.ps.node-load-misses
    117037 ±  2%    +299.1%     467127 ± 11%  perf-stat.ps.node-loads
     50493 ±  5%    +723.7%     415907 ±  7%  perf-stat.ps.node-store-misses
    116604 ± 11%    +381.7%     561730 ±  6%  perf-stat.ps.node-stores
      2852            -7.2%       2646        perf-stat.ps.page-faults
 1.296e+13            +3.3%  1.339e+13        perf-stat.total.instructions
    128311           -24.4%      96996 ±  2%  interrupts.CAL:Function_call_interrupts
      7980 ±  7%    -100.0%       0.00        interrupts.CPU0.NMI:Non-maskable_interrupts
      7980 ±  7%    -100.0%       0.00        interrupts.CPU0.PMI:Performance_monitoring_interrupts
      5151 ± 26%     -59.6%       2080 ± 11%  interrupts.CPU1.CAL:Function_call_interrupts
      5489 ± 27%    -100.0%       0.00        interrupts.CPU1.NMI:Non-maskable_interrupts
      5489 ± 27%    -100.0%       0.00        interrupts.CPU1.PMI:Performance_monitoring_interrupts
      5906 ± 31%    -100.0%       0.00        interrupts.CPU10.NMI:Non-maskable_interrupts
      5906 ± 31%    -100.0%       0.00        interrupts.CPU10.PMI:Performance_monitoring_interrupts
      6677 ± 27%    -100.0%       0.00        interrupts.CPU11.NMI:Non-maskable_interrupts
      6677 ± 27%    -100.0%       0.00        interrupts.CPU11.PMI:Performance_monitoring_interrupts
    372.50 ± 11%    +139.5%     892.17 ± 47%  interrupts.CPU11.RES:Rescheduling_interrupts
      3246 ±  6%     -35.5%       2093 ± 12%  interrupts.CPU12.CAL:Function_call_interrupts
      5948 ± 25%    -100.0%       0.00        interrupts.CPU12.NMI:Non-maskable_interrupts
      5948 ± 25%    -100.0%       0.00        interrupts.CPU12.PMI:Performance_monitoring_interrupts
      5726 ± 26%     -64.9%       2011 ±  5%  interrupts.CPU13.CAL:Function_call_interrupts
      7394 ± 20%    -100.0%       0.00        interrupts.CPU13.NMI:Non-maskable_interrupts
      7394 ± 20%    -100.0%       0.00        interrupts.CPU13.PMI:Performance_monitoring_interrupts
      7596 ± 15%     -75.0%       1901 ±  3%  interrupts.CPU14.CAL:Function_call_interrupts
      5949 ± 31%    -100.0%       0.00        interrupts.CPU14.NMI:Non-maskable_interrupts
      5949 ± 31%    -100.0%       0.00        interrupts.CPU14.PMI:Performance_monitoring_interrupts
      1372 ± 41%     -57.9%     578.33 ± 22%  interrupts.CPU14.RES:Rescheduling_interrupts
      3586 ± 32%     -46.9%       1903 ±  6%  interrupts.CPU15.CAL:Function_call_interrupts
      6700 ± 27%    -100.0%       0.00        interrupts.CPU15.NMI:Non-maskable_interrupts
      6700 ± 27%    -100.0%       0.00        interrupts.CPU15.PMI:Performance_monitoring_interrupts
      5647 ± 27%    -100.0%       0.00        interrupts.CPU16.NMI:Non-maskable_interrupts
      5647 ± 27%    -100.0%       0.00        interrupts.CPU16.PMI:Performance_monitoring_interrupts
      6007 ± 31%    -100.0%       0.00        interrupts.CPU17.NMI:Non-maskable_interrupts
      6007 ± 31%    -100.0%       0.00        interrupts.CPU17.PMI:Performance_monitoring_interrupts
      6206 ± 33%    -100.0%       0.00        interrupts.CPU18.NMI:Non-maskable_interrupts
      6206 ± 33%    -100.0%       0.00        interrupts.CPU18.PMI:Performance_monitoring_interrupts
    413.67 ± 17%     +96.3%     812.17 ± 24%  interrupts.CPU18.RES:Rescheduling_interrupts
      5518 ± 35%    -100.0%       0.00        interrupts.CPU19.NMI:Non-maskable_interrupts
      5518 ± 35%    -100.0%       0.00        interrupts.CPU19.PMI:Performance_monitoring_interrupts
    399.33 ± 24%     +92.7%     769.33 ± 37%  interrupts.CPU19.RES:Rescheduling_interrupts
      6808 ± 13%     -70.0%       2044 ± 11%  interrupts.CPU2.CAL:Function_call_interrupts
      5454 ± 28%    -100.0%       0.00        interrupts.CPU2.NMI:Non-maskable_interrupts
      5454 ± 28%    -100.0%       0.00        interrupts.CPU2.PMI:Performance_monitoring_interrupts
      1100 ±  4%      -9.1%       1000 ±  3%  interrupts.CPU2.TLB:TLB_shootdowns
      7581 ± 20%    -100.0%       0.00        interrupts.CPU20.NMI:Non-maskable_interrupts
      7581 ± 20%    -100.0%       0.00        interrupts.CPU20.PMI:Performance_monitoring_interrupts
      7109 ± 21%    -100.0%       0.00        interrupts.CPU21.NMI:Non-maskable_interrupts
      7109 ± 21%    -100.0%       0.00        interrupts.CPU21.PMI:Performance_monitoring_interrupts
      7036 ± 21%    -100.0%       0.00        interrupts.CPU22.NMI:Non-maskable_interrupts
      7036 ± 21%    -100.0%       0.00        interrupts.CPU22.PMI:Performance_monitoring_interrupts
    414.33 ± 27%    +135.4%     975.50 ± 38%  interrupts.CPU22.RES:Rescheduling_interrupts
      7434 ± 20%    -100.0%       0.00        interrupts.CPU23.NMI:Non-maskable_interrupts
      7434 ± 20%    -100.0%       0.00        interrupts.CPU23.PMI:Performance_monitoring_interrupts
    333.00 ±  4%    +132.5%     774.33 ± 59%  interrupts.CPU23.RES:Rescheduling_interrupts
      1966 ±  4%      -9.9%       1770 ±  5%  interrupts.CPU24.CAL:Function_call_interrupts
      6889 ± 28%    -100.0%       0.00        interrupts.CPU24.NMI:Non-maskable_interrupts
      6889 ± 28%    -100.0%       0.00        interrupts.CPU24.PMI:Performance_monitoring_interrupts
      1082 ±  4%     -15.5%     914.83 ±  6%  interrupts.CPU24.TLB:TLB_shootdowns
      2259 ±  9%     -13.3%       1957 ±  6%  interrupts.CPU25.CAL:Function_call_interrupts
      7215 ± 22%    -100.0%       0.00        interrupts.CPU25.NMI:Non-maskable_interrupts
      7215 ± 22%    -100.0%       0.00        interrupts.CPU25.PMI:Performance_monitoring_interrupts
      2705 ± 14%     -27.5%       1960 ±  9%  interrupts.CPU26.CAL:Function_call_interrupts
      7554 ± 18%    -100.0%       1.17 ±223%  interrupts.CPU26.NMI:Non-maskable_interrupts
      7554 ± 18%    -100.0%       1.17 ±223%  interrupts.CPU26.PMI:Performance_monitoring_interrupts
      6887 ± 28%    -100.0%       0.00        interrupts.CPU27.NMI:Non-maskable_interrupts
      6887 ± 28%    -100.0%       0.00        interrupts.CPU27.PMI:Performance_monitoring_interrupts
      7579 ± 20%    -100.0%       0.00        interrupts.CPU28.NMI:Non-maskable_interrupts
      7579 ± 20%    -100.0%       0.00        interrupts.CPU28.PMI:Performance_monitoring_interrupts
      6761 ± 27%    -100.0%       0.00        interrupts.CPU29.NMI:Non-maskable_interrupts
      6761 ± 27%    -100.0%       0.00        interrupts.CPU29.PMI:Performance_monitoring_interrupts
      5819 ± 17%     -66.0%       1978 ±  8%  interrupts.CPU3.CAL:Function_call_interrupts
      6654 ± 27%    -100.0%       0.00        interrupts.CPU3.NMI:Non-maskable_interrupts
      6654 ± 27%    -100.0%       0.00        interrupts.CPU3.PMI:Performance_monitoring_interrupts
      1105 ±  3%     -10.9%     985.50 ±  6%  interrupts.CPU3.TLB:TLB_shootdowns
      7517 ± 19%    -100.0%       0.00        interrupts.CPU30.NMI:Non-maskable_interrupts
      7517 ± 19%    -100.0%       0.00        interrupts.CPU30.PMI:Performance_monitoring_interrupts
      5579 ± 34%    -100.0%       0.00        interrupts.CPU31.NMI:Non-maskable_interrupts
      5579 ± 34%    -100.0%       0.00        interrupts.CPU31.PMI:Performance_monitoring_interrupts
    378.50 ± 20%    +448.7%       2077 ± 62%  interrupts.CPU31.RES:Rescheduling_interrupts
      5652 ± 33%    -100.0%       0.00        interrupts.CPU32.NMI:Non-maskable_interrupts
      5652 ± 33%    -100.0%       0.00        interrupts.CPU32.PMI:Performance_monitoring_interrupts
    450.50 ± 54%    +251.9%       1585 ± 86%  interrupts.CPU32.RES:Rescheduling_interrupts
      1092 ±  4%      -8.6%     998.33 ±  4%  interrupts.CPU32.TLB:TLB_shootdowns
      6900 ± 28%    -100.0%       0.00        interrupts.CPU33.NMI:Non-maskable_interrupts
      6900 ± 28%    -100.0%       0.00        interrupts.CPU33.PMI:Performance_monitoring_interrupts
      1097           -10.9%     978.50 ±  7%  interrupts.CPU33.TLB:TLB_shootdowns
      6518 ± 28%    -100.0%       0.00        interrupts.CPU34.NMI:Non-maskable_interrupts
      6518 ± 28%    -100.0%       0.00        interrupts.CPU34.PMI:Performance_monitoring_interrupts
    366.33 ± 14%    +303.8%       1479 ± 35%  interrupts.CPU34.RES:Rescheduling_interrupts
      6948 ± 27%    -100.0%       0.00        interrupts.CPU35.NMI:Non-maskable_interrupts
      6948 ± 27%    -100.0%       0.00        interrupts.CPU35.PMI:Performance_monitoring_interrupts
    496.17 ± 65%    +218.7%       1581 ± 68%  interrupts.CPU35.RES:Rescheduling_interrupts
      7540 ± 20%    -100.0%       0.00        interrupts.CPU36.NMI:Non-maskable_interrupts
      7540 ± 20%    -100.0%       0.00        interrupts.CPU36.PMI:Performance_monitoring_interrupts
    401.33 ± 14%    +243.2%       1377 ± 71%  interrupts.CPU36.RES:Rescheduling_interrupts
      1107 ±  2%     -11.6%     979.17 ±  5%  interrupts.CPU36.TLB:TLB_shootdowns
      3479 ± 30%     -44.0%       1947 ± 12%  interrupts.CPU37.CAL:Function_call_interrupts
      6836 ± 27%    -100.0%       0.00        interrupts.CPU37.NMI:Non-maskable_interrupts
      6836 ± 27%    -100.0%       0.00        interrupts.CPU37.PMI:Performance_monitoring_interrupts
      1093 ±  4%      -8.4%       1001 ±  4%  interrupts.CPU37.TLB:TLB_shootdowns
      6880 ± 27%    -100.0%       0.00        interrupts.CPU38.NMI:Non-maskable_interrupts
      6880 ± 27%    -100.0%       0.00        interrupts.CPU38.PMI:Performance_monitoring_interrupts
    407.17 ± 14%    +130.5%     938.67 ± 53%  interrupts.CPU38.RES:Rescheduling_interrupts
      6901 ± 28%    -100.0%       0.00        interrupts.CPU39.NMI:Non-maskable_interrupts
      6901 ± 28%    -100.0%       0.00        interrupts.CPU39.PMI:Performance_monitoring_interrupts
      3726 ± 21%     -48.3%       1928 ±  3%  interrupts.CPU4.CAL:Function_call_interrupts
      6048 ± 32%    -100.0%       0.00        interrupts.CPU4.NMI:Non-maskable_interrupts
      6048 ± 32%    -100.0%       0.00        interrupts.CPU4.PMI:Performance_monitoring_interrupts
      6901 ± 28%    -100.0%       0.00        interrupts.CPU40.NMI:Non-maskable_interrupts
      6901 ± 28%    -100.0%       0.00        interrupts.CPU40.PMI:Performance_monitoring_interrupts
      6895 ± 27%    -100.0%       0.00        interrupts.CPU41.NMI:Non-maskable_interrupts
      6895 ± 27%    -100.0%       0.00        interrupts.CPU41.PMI:Performance_monitoring_interrupts
      7580 ± 20%    -100.0%       0.00        interrupts.CPU42.NMI:Non-maskable_interrupts
      7580 ± 20%    -100.0%       0.00        interrupts.CPU42.PMI:Performance_monitoring_interrupts
    349.83 ±  8%    +188.9%       1010 ± 40%  interrupts.CPU42.RES:Rescheduling_interrupts
      7660 ± 17%    -100.0%       0.00        interrupts.CPU43.NMI:Non-maskable_interrupts
      7660 ± 17%    -100.0%       0.00        interrupts.CPU43.PMI:Performance_monitoring_interrupts
      6208 ± 33%    -100.0%       0.00        interrupts.CPU44.NMI:Non-maskable_interrupts
      6208 ± 33%    -100.0%       0.00        interrupts.CPU44.PMI:Performance_monitoring_interrupts
    395.50 ± 15%    +223.7%       1280 ± 57%  interrupts.CPU44.RES:Rescheduling_interrupts
      1104 ±  3%      -8.9%       1005 ±  4%  interrupts.CPU44.TLB:TLB_shootdowns
      5515 ± 35%    -100.0%       0.00        interrupts.CPU45.NMI:Non-maskable_interrupts
      5515 ± 35%    -100.0%       0.00        interrupts.CPU45.PMI:Performance_monitoring_interrupts
    374.17 ± 10%    +227.8%       1226 ± 72%  interrupts.CPU45.RES:Rescheduling_interrupts
      7557 ± 20%    -100.0%       0.00        interrupts.CPU46.NMI:Non-maskable_interrupts
      7557 ± 20%    -100.0%       0.00        interrupts.CPU46.PMI:Performance_monitoring_interrupts
    346.33 ±  7%    +251.3%       1216 ± 71%  interrupts.CPU46.RES:Rescheduling_interrupts
      6123 ± 32%    -100.0%       0.00        interrupts.CPU47.NMI:Non-maskable_interrupts
      6123 ± 32%    -100.0%       0.00        interrupts.CPU47.PMI:Performance_monitoring_interrupts
    360.67 ±  5%    +183.0%       1020 ± 79%  interrupts.CPU47.RES:Rescheduling_interrupts
      4732 ± 28%    -100.0%       0.00        interrupts.CPU5.NMI:Non-maskable_interrupts
      4732 ± 28%    -100.0%       0.00        interrupts.CPU5.PMI:Performance_monitoring_interrupts
      5158 ± 30%    -100.0%       0.00        interrupts.CPU6.NMI:Non-maskable_interrupts
      5158 ± 30%    -100.0%       0.00        interrupts.CPU6.PMI:Performance_monitoring_interrupts
      4821 ± 31%    -100.0%       0.00        interrupts.CPU7.NMI:Non-maskable_interrupts
      4821 ± 31%    -100.0%       0.00        interrupts.CPU7.PMI:Performance_monitoring_interrupts
      7091 ± 20%    -100.0%       0.00        interrupts.CPU8.NMI:Non-maskable_interrupts
      7091 ± 20%    -100.0%       0.00        interrupts.CPU8.PMI:Performance_monitoring_interrupts
      5671 ± 28%    -100.0%       0.00        interrupts.CPU9.NMI:Non-maskable_interrupts
      5671 ± 28%    -100.0%       0.00        interrupts.CPU9.PMI:Performance_monitoring_interrupts
    383.50 ±  9%    +133.6%     895.67 ± 51%  interrupts.CPU9.RES:Rescheduling_interrupts
    314820 ±  7%    -100.0%       1.17 ±223%  interrupts.NMI:Non-maskable_interrupts
    314820 ±  7%    -100.0%       1.17 ±223%  interrupts.PMI:Performance_monitoring_interrupts
     27370 ±  6%    +104.9%      56082 ±  2%  interrupts.RES:Rescheduling_interrupts
      0.02 ±  4%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown]
      2.03 ±101%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.devkmsg_read.vfs_read.ksys_read.do_syscall_64
      0.02 ±  2%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.do_nanosleep.hrtimer_nanosleep.__x64_sys_nanosleep.do_syscall_64
      3.23 ±120%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.do_syslog.part.0.kmsg_read.vfs_read
      2.68 ± 16%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
      0.02 ±  2%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64
      0.19 ± 14%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.exit_to_user_mode_prepare.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe.[unknown]
      0.03 ± 22%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.futex_wait_queue_me.futex_wait.do_futex.__x64_sys_futex
      0.02 ± 10%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.pipe_read.new_sync_read.vfs_read.ksys_read
     10.55 ± 11%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.preempt_schedule_common.__cond_resched.stop_one_cpu.sched_exec.bprm_execve
      0.08 ±155%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.rcu_gp_kthread.kthread.ret_from_fork
      0.05 ± 28%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait.__x64_sys_epoll_wait
      0.02 ± 11%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_select
      0.00 ± 14%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_sys_poll
      0.02 ±  2%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.schedule_timeout.kcompactd.kthread.ret_from_fork
      0.02 ±  4%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.schedule_timeout.rcu_gp_kthread.kthread.ret_from_fork
      0.03 ±126%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.smpboot_thread_fn.kthread.ret_from_fork
      8.47 ±  7%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.wait_for_partner.fifo_open.do_dentry_open.do_open.isra
      0.06 ± 43%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.worker_thread.kthread.ret_from_fork
      0.02 ±  9%    -100.0%       0.00        perf-sched.sch_delay.max.ms.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown]
     10.11 ±101%    -100.0%       0.00        perf-sched.sch_delay.max.ms.devkmsg_read.vfs_read.ksys_read.do_syscall_64
      0.03 ± 18%    -100.0%       0.00        perf-sched.sch_delay.max.ms.do_nanosleep.hrtimer_nanosleep.__x64_sys_nanosleep.do_syscall_64
     11.64 ± 94%    -100.0%       0.00        perf-sched.sch_delay.max.ms.do_syslog.part.0.kmsg_read.vfs_read
     19.20 ±  8%    -100.0%       0.00        perf-sched.sch_delay.max.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
      0.05 ± 49%    -100.0%       0.00        perf-sched.sch_delay.max.ms.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64
     18.89 ± 11%    -100.0%       0.00        perf-sched.sch_delay.max.ms.exit_to_user_mode_prepare.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe.[unknown]
      0.04 ± 28%    -100.0%       0.00        perf-sched.sch_delay.max.ms.futex_wait_queue_me.futex_wait.do_futex.__x64_sys_futex
      0.33 ± 28%    -100.0%       0.00        perf-sched.sch_delay.max.ms.pipe_read.new_sync_read.vfs_read.ksys_read
      0.03 ±  5%    -100.0%       0.00        perf-sched.sch_delay.max.ms.preempt_schedule_common.__cond_resched.stop_one_cpu.affine_move_task.__set_cpus_allowed_ptr
     35.38 ±105%    -100.0%       0.00        perf-sched.sch_delay.max.ms.preempt_schedule_common.__cond_resched.stop_one_cpu.sched_exec.bprm_execve
      5.07 ±199%    -100.0%       0.00        perf-sched.sch_delay.max.ms.rcu_gp_kthread.kthread.ret_from_fork
      0.42 ±  4%    -100.0%       0.00        perf-sched.sch_delay.max.ms.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait.__x64_sys_epoll_wait
      0.02 ± 14%    -100.0%       0.00        perf-sched.sch_delay.max.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_select
      0.02 ± 25%    -100.0%       0.00        perf-sched.sch_delay.max.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_sys_poll
      0.02 ± 11%    -100.0%       0.00        perf-sched.sch_delay.max.ms.schedule_timeout.kcompactd.kthread.ret_from_fork
      0.21 ± 84%    -100.0%       0.00        perf-sched.sch_delay.max.ms.schedule_timeout.rcu_gp_kthread.kthread.ret_from_fork
     12.35 ±217%    -100.0%       0.00        perf-sched.sch_delay.max.ms.smpboot_thread_fn.kthread.ret_from_fork
     20.33 ±  4%    -100.0%       0.00        perf-sched.sch_delay.max.ms.wait_for_partner.fifo_open.do_dentry_open.do_open.isra
     15.52 ± 54%    -100.0%       0.00        perf-sched.sch_delay.max.ms.worker_thread.kthread.ret_from_fork
      0.21 ±  8%    -100.0%       0.00        perf-sched.total_sch_delay.average.ms
     48.71 ± 73%    -100.0%       0.00        perf-sched.total_sch_delay.max.ms
     98.46 ±  9%    -100.0%       0.00        perf-sched.total_wait_and_delay.average.ms
     10953 ± 10%    -100.0%       0.00        perf-sched.total_wait_and_delay.count.ms
      7483 ±  5%    -100.0%       0.00        perf-sched.total_wait_and_delay.max.ms
     98.25 ±  9%    -100.0%       0.00        perf-sched.total_wait_time.average.ms
      7483 ±  5%    -100.0%       0.00        perf-sched.total_wait_time.max.ms
    882.95 ±  4%    -100.0%       0.00        perf-sched.wait_and_delay.avg.ms.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown]
    487.24 ± 60%    -100.0%       0.00        perf-sched.wait_and_delay.avg.ms.devkmsg_read.vfs_read.ksys_read.do_syscall_64
    821.03 ±  4%    -100.0%       0.00        perf-sched.wait_and_delay.avg.ms.do_nanosleep.hrtimer_nanosleep.__x64_sys_nanosleep.do_syscall_64
    506.66 ± 77%    -100.0%       0.00        perf-sched.wait_and_delay.avg.ms.do_syslog.part.0.kmsg_read.vfs_read
    268.27 ±  2%    -100.0%       0.00        perf-sched.wait_and_delay.avg.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
     99.97 ± 13%    -100.0%       0.00        perf-sched.wait_and_delay.avg.ms.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64
      0.37 ± 71%    -100.0%       0.00        perf-sched.wait_and_delay.avg.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt.[unknown]
      0.22 ± 21%    -100.0%       0.00        perf-sched.wait_and_delay.avg.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_call_function_single.[unknown]
      0.18 ± 57%    -100.0%       0.00        perf-sched.wait_and_delay.avg.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_reschedule_ipi.[unknown]
      0.41 ± 11%    -100.0%       0.00        perf-sched.wait_and_delay.avg.ms.exit_to_user_mode_prepare.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe.[unknown]
     83.78 ± 18%    -100.0%       0.00        perf-sched.wait_and_delay.avg.ms.pipe_read.new_sync_read.vfs_read.ksys_read
     18.86          -100.0%       0.00        perf-sched.wait_and_delay.avg.ms.preempt_schedule_common.__cond_resched.stop_one_cpu.affine_move_task.__set_cpus_allowed_ptr
    284.95 ± 40%    -100.0%       0.00        perf-sched.wait_and_delay.avg.ms.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait.__x64_sys_epoll_wait
    239.54 ±  9%    -100.0%       0.00        perf-sched.wait_and_delay.avg.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_sys_poll
    478.81          -100.0%       0.00        perf-sched.wait_and_delay.avg.ms.schedule_timeout.kcompactd.kthread.ret_from_fork
      6.28 ± 15%    -100.0%       0.00        perf-sched.wait_and_delay.avg.ms.schedule_timeout.rcu_gp_kthread.kthread.ret_from_fork
    653.01          -100.0%       0.00        perf-sched.wait_and_delay.avg.ms.smpboot_thread_fn.kthread.ret_from_fork
      8.87 ±  5%    -100.0%       0.00        perf-sched.wait_and_delay.avg.ms.wait_for_partner.fifo_open.do_dentry_open.do_open.isra
    565.29 ±  5%    -100.0%       0.00        perf-sched.wait_and_delay.avg.ms.worker_thread.kthread.ret_from_fork
     10.00          -100.0%       0.00        perf-sched.wait_and_delay.count.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown]
      5.50 ±  9%    -100.0%       0.00        perf-sched.wait_and_delay.count.devkmsg_read.vfs_read.ksys_read.do_syscall_64
     22.00 ±  4%    -100.0%       0.00        perf-sched.wait_and_delay.count.do_nanosleep.hrtimer_nanosleep.__x64_sys_nanosleep.do_syscall_64
      5.67 ±  8%    -100.0%       0.00        perf-sched.wait_and_delay.count.do_syslog.part.0.kmsg_read.vfs_read
    228.00          -100.0%       0.00        perf-sched.wait_and_delay.count.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
    264.00 ±  2%    -100.0%       0.00        perf-sched.wait_and_delay.count.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64
      1517 ± 13%    -100.0%       0.00        perf-sched.wait_and_delay.count.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt.[unknown]
    178.17 ± 10%    -100.0%       0.00        perf-sched.wait_and_delay.count.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_call_function_single.[unknown]
    104.50 ± 12%    -100.0%       0.00        perf-sched.wait_and_delay.count.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_reschedule_ipi.[unknown]
      3237 ± 14%    -100.0%       0.00        perf-sched.wait_and_delay.count.exit_to_user_mode_prepare.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe.[unknown]
      1581 ± 22%    -100.0%       0.00        perf-sched.wait_and_delay.count.pipe_read.new_sync_read.vfs_read.ksys_read
    576.00          -100.0%       0.00        perf-sched.wait_and_delay.count.preempt_schedule_common.__cond_resched.stop_one_cpu.affine_move_task.__set_cpus_allowed_ptr
     23.33 ± 46%    -100.0%       0.00        perf-sched.wait_and_delay.count.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait.__x64_sys_epoll_wait
    105.50 ±  7%    -100.0%       0.00        perf-sched.wait_and_delay.count.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_sys_poll
     40.00          -100.0%       0.00        perf-sched.wait_and_delay.count.schedule_timeout.kcompactd.kthread.ret_from_fork
      1598 ± 21%    -100.0%       0.00        perf-sched.wait_and_delay.count.schedule_timeout.rcu_gp_kthread.kthread.ret_from_fork
    749.00          -100.0%       0.00        perf-sched.wait_and_delay.count.smpboot_thread_fn.kthread.ret_from_fork
     66.17          -100.0%       0.00        perf-sched.wait_and_delay.count.wait_for_partner.fifo_open.do_dentry_open.do_open.isra
    436.17 ±  2%    -100.0%       0.00        perf-sched.wait_and_delay.count.worker_thread.kthread.ret_from_fork
    999.61          -100.0%       0.00        perf-sched.wait_and_delay.max.ms.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown]
      2337 ± 50%    -100.0%       0.00        perf-sched.wait_and_delay.max.ms.devkmsg_read.vfs_read.ksys_read.do_syscall_64
      1000          -100.0%       0.00        perf-sched.wait_and_delay.max.ms.do_nanosleep.hrtimer_nanosleep.__x64_sys_nanosleep.do_syscall_64
      2368 ± 49%    -100.0%       0.00        perf-sched.wait_and_delay.max.ms.do_syslog.part.0.kmsg_read.vfs_read
      1989 ± 84%    -100.0%       0.00        perf-sched.wait_and_delay.max.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
      1022          -100.0%       0.00        perf-sched.wait_and_delay.max.ms.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64
    193.25 ±186%    -100.0%       0.00        perf-sched.wait_and_delay.max.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt.[unknown]
      9.91 ± 74%    -100.0%       0.00        perf-sched.wait_and_delay.max.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_call_function_single.[unknown]
      9.67 ±113%    -100.0%       0.00        perf-sched.wait_and_delay.max.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_reschedule_ipi.[unknown]
     28.27 ± 19%    -100.0%       0.00        perf-sched.wait_and_delay.max.ms.exit_to_user_mode_prepare.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe.[unknown]
      2338 ± 50%    -100.0%       0.00        perf-sched.wait_and_delay.max.ms.pipe_read.new_sync_read.vfs_read.ksys_read
      1849          -100.0%       0.00        perf-sched.wait_and_delay.max.ms.preempt_schedule_common.__cond_resched.stop_one_cpu.affine_move_task.__set_cpus_allowed_ptr
      1999 ± 57%    -100.0%       0.00        perf-sched.wait_and_delay.max.ms.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait.__x64_sys_epoll_wait
      5216 ± 36%    -100.0%       0.00        perf-sched.wait_and_delay.max.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_sys_poll
    505.01          -100.0%       0.00        perf-sched.wait_and_delay.max.ms.schedule_timeout.kcompactd.kthread.ret_from_fork
    254.85 ± 29%    -100.0%       0.00        perf-sched.wait_and_delay.max.ms.schedule_timeout.rcu_gp_kthread.kthread.ret_from_fork
      5585 ± 20%    -100.0%       0.00        perf-sched.wait_and_delay.max.ms.smpboot_thread_fn.kthread.ret_from_fork
     20.33 ±  4%    -100.0%       0.00        perf-sched.wait_and_delay.max.ms.wait_for_partner.fifo_open.do_dentry_open.do_open.isra
      7413 ±  6%    -100.0%       0.00        perf-sched.wait_and_delay.max.ms.worker_thread.kthread.ret_from_fork
    882.94 ±  4%    -100.0%       0.00        perf-sched.wait_time.avg.ms.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown]
    485.21 ± 60%    -100.0%       0.00        perf-sched.wait_time.avg.ms.devkmsg_read.vfs_read.ksys_read.do_syscall_64
    821.01 ±  4%    -100.0%       0.00        perf-sched.wait_time.avg.ms.do_nanosleep.hrtimer_nanosleep.__x64_sys_nanosleep.do_syscall_64
    503.43 ± 77%    -100.0%       0.00        perf-sched.wait_time.avg.ms.do_syslog.part.0.kmsg_read.vfs_read
    265.59 ±  2%    -100.0%       0.00        perf-sched.wait_time.avg.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
     99.95 ± 13%    -100.0%       0.00        perf-sched.wait_time.avg.ms.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64
      0.36 ± 71%    -100.0%       0.00        perf-sched.wait_time.avg.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt.[unknown]
      0.21 ± 19%    -100.0%       0.00        perf-sched.wait_time.avg.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_call_function_single.[unknown]
      1.02 ±216%    -100.0%       0.00        perf-sched.wait_time.avg.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_irq_work.[unknown]
      0.17 ± 62%    -100.0%       0.00        perf-sched.wait_time.avg.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_reschedule_ipi.[unknown]
      0.22 ±  9%    -100.0%       0.00        perf-sched.wait_time.avg.ms.exit_to_user_mode_prepare.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe.[unknown]
    300.42 ±204%    -100.0%       0.00        perf-sched.wait_time.avg.ms.futex_wait_queue_me.futex_wait.do_futex.__x64_sys_futex
     83.76 ± 18%    -100.0%       0.00        perf-sched.wait_time.avg.ms.pipe_read.new_sync_read.vfs_read.ksys_read
     10.87 ± 11%    -100.0%       0.00        perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.dput.__fput.task_work_run
    741.75 ±104%    -100.0%       0.00        perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.generic_perform_write.__generic_file_write_iter.generic_file_write_iter
      3.83 ± 19%    -100.0%       0.00        perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.khugepaged.kthread.ret_from_fork
     18.86          -100.0%       0.00        perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.stop_one_cpu.affine_move_task.__set_cpus_allowed_ptr
      2.77 ± 13%    -100.0%       0.00        perf-sched.wait_time.avg.ms.rcu_gp_kthread.kthread.ret_from_fork
    284.90 ± 40%    -100.0%       0.00        perf-sched.wait_time.avg.ms.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait.__x64_sys_epoll_wait
    239.53 ±  9%    -100.0%       0.00        perf-sched.wait_time.avg.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_sys_poll
    478.79          -100.0%       0.00        perf-sched.wait_time.avg.ms.schedule_timeout.kcompactd.kthread.ret_from_fork
      6.71 ± 12%    -100.0%       0.00        perf-sched.wait_time.avg.ms.schedule_timeout.khugepaged.kthread.ret_from_fork
      6.26 ± 15%    -100.0%       0.00        perf-sched.wait_time.avg.ms.schedule_timeout.rcu_gp_kthread.kthread.ret_from_fork
    652.98          -100.0%       0.00        perf-sched.wait_time.avg.ms.smpboot_thread_fn.kthread.ret_from_fork
      0.40 ± 53%    -100.0%       0.00        perf-sched.wait_time.avg.ms.wait_for_partner.fifo_open.do_dentry_open.do_open.isra
    565.24 ±  5%    -100.0%       0.00        perf-sched.wait_time.avg.ms.worker_thread.kthread.ret_from_fork
    999.59          -100.0%       0.00        perf-sched.wait_time.max.ms.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown]
      2337 ± 50%    -100.0%       0.00        perf-sched.wait_time.max.ms.devkmsg_read.vfs_read.ksys_read.do_syscall_64
      1000          -100.0%       0.00        perf-sched.wait_time.max.ms.do_nanosleep.hrtimer_nanosleep.__x64_sys_nanosleep.do_syscall_64
      2357 ± 49%    -100.0%       0.00        perf-sched.wait_time.max.ms.do_syslog.part.0.kmsg_read.vfs_read
      1986 ± 84%    -100.0%       0.00        perf-sched.wait_time.max.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
      1022          -100.0%       0.00        perf-sched.wait_time.max.ms.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64
    193.25 ±186%    -100.0%       0.00        perf-sched.wait_time.max.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt.[unknown]
      7.96 ± 94%    -100.0%       0.00        perf-sched.wait_time.max.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_call_function_single.[unknown]
     10.10 ±219%    -100.0%       0.00        perf-sched.wait_time.max.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_irq_work.[unknown]
      9.53 ±115%    -100.0%       0.00        perf-sched.wait_time.max.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_reschedule_ipi.[unknown]
     27.50 ± 25%    -100.0%       0.00        perf-sched.wait_time.max.ms.exit_to_user_mode_prepare.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe.[unknown]
    869.52 ±211%    -100.0%       0.00        perf-sched.wait_time.max.ms.futex_wait_queue_me.futex_wait.do_futex.__x64_sys_futex
      2338 ± 50%    -100.0%       0.00        perf-sched.wait_time.max.ms.pipe_read.new_sync_read.vfs_read.ksys_read
     12.88 ±  2%    -100.0%       0.00        perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.dput.__fput.task_work_run
      3266 ± 99%    -100.0%       0.00        perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.generic_perform_write.__generic_file_write_iter.generic_file_write_iter
      6.44 ± 21%    -100.0%       0.00        perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.khugepaged.kthread.ret_from_fork
      1849          -100.0%       0.00        perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.stop_one_cpu.affine_move_task.__set_cpus_allowed_ptr
     10.00 ±116%    -100.0%       0.00        perf-sched.wait_time.max.ms.rcu_gp_kthread.kthread.ret_from_fork
      1999 ± 57%    -100.0%       0.00        perf-sched.wait_time.max.ms.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait.__x64_sys_epoll_wait
      5216 ± 36%    -100.0%       0.00        perf-sched.wait_time.max.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_sys_poll
    504.99          -100.0%       0.00        perf-sched.wait_time.max.ms.schedule_timeout.kcompactd.kthread.ret_from_fork
      6.71 ± 12%    -100.0%       0.00        perf-sched.wait_time.max.ms.schedule_timeout.khugepaged.kthread.ret_from_fork
    254.82 ± 29%    -100.0%       0.00        perf-sched.wait_time.max.ms.schedule_timeout.rcu_gp_kthread.kthread.ret_from_fork
      5585 ± 20%    -100.0%       0.00        perf-sched.wait_time.max.ms.smpboot_thread_fn.kthread.ret_from_fork
      7.41 ± 59%    -100.0%       0.00        perf-sched.wait_time.max.ms.wait_for_partner.fifo_open.do_dentry_open.do_open.isra
      7413 ±  6%    -100.0%       0.00        perf-sched.wait_time.max.ms.worker_thread.kthread.ret_from_fork

***************************************************************************************************
lkp-ivb-2ep1: 48 threads Intel(R) Xeon(R) CPU E5-2697 v2 @ 2.70GHz with 112G memory
=========================================================================================
compiler/cpufreq_governor/kconfig/mode/nr_task/rootfs/tbox_group/test/testcase/ucode:
gcc-9/performance/x86_64-rhel-8.3/process/50%/debian-10.4-x86_64-20200603.cgz/lkp-ivb-2ep1/pread1/will-it-scale/0x42e

commit:
dde38d5b80 ("kernel: don't include PF_IO_WORKERs as part of same_thread_group()") 43b2a76b1a ("proc: don't show PF_IO_WORKER threads as threads in /proc/<pid>/task/")

dde38d5b805358c7 43b2a76b1a5abcc9833b463bef1
---------------- ---------------------------
%stddev %change %stddev
\ | \

  32174247            -2.6%   31345253        will-it-scale.24.processes
     49.34            -8.3%      45.25        will-it-scale.24.processes_idle
   1340593            -2.6%    1306051        will-it-scale.per_process_ops
  32174247            -2.6%   31345253        will-it-scale.workload
      0.03            +0.0        0.04 ±  3%  mpstat.cpu.all.soft%
    327098 ±  5%    +118.6%     715075 ±  5%  numa-numastat.node0.local_node
    357493 ±  2%    +106.8%     739292 ±  4%  numa-numastat.node0.numa_hit
    372736 ±  4%     +93.8%     722461 ±  3%  numa-numastat.node1.local_node
    385821           +92.2%     741716        numa-numastat.node1.numa_hit
     49.00            -8.2%      45.00        vmstat.cpu.id
   1337175           +92.9%    2579543        vmstat.memory.cache
      1225 ±  2%     +47.6%       1808 ±  2%  vmstat.system.cs
     97244            -9.5%      87974 ± 20%  vmstat.system.in
     61834         +2002.0%    1299755        meminfo.Active
     61834         +2002.0%    1299755        meminfo.Active(anon)
   1255800           +98.5%    2492662        meminfo.Cached
    441192          +290.3%    1722149        meminfo.Committed_AS
   2148004           +67.0%    3586556        meminfo.Memused
     74825         +1653.0%    1311691        meminfo.Shmem
   2339292           +53.3%    3586556        meminfo.max_used_kB
      0.53            +8.7%       0.58 ±  2%  sched_debug.cfs_rq:/.nr_running.avg
    225.94 ±  2%     +15.3%     260.59 ±  3%  sched_debug.cfs_rq:/.util_est_enqueued.avg
    584.12 ±  4%     +70.5%     995.71 ±  2%  sched_debug.cfs_rq:/.util_est_enqueued.max
    231.12 ±  2%     +13.1%     261.49        sched_debug.cfs_rq:/.util_est_enqueued.stddev
    611.20 ±  3%     +12.9%     689.89 ±  7%  sched_debug.cpu.clock_task.stddev
      5469 ±  7%     +26.7%       6932 ±  6%  sched_debug.cpu.nr_switches.avg
     15262 ± 15%     +72.5%      26320 ± 12%  sched_debug.cpu.nr_switches.max
      3393 ± 13%     +56.6%       5313 ±  7%  sched_debug.cpu.nr_switches.stddev
     14380 ±  3%     +17.7%      16924 ±  4%  slabinfo.filp.active_objs
     14621 ±  2%     +17.5%      17182 ±  4%  slabinfo.filp.num_objs
      5668 ±  2%     +11.9%       6341 ±  2%  slabinfo.kmalloc-256.active_objs
      5756 ±  2%     +13.0%       6503 ±  2%  slabinfo.kmalloc-256.num_objs
      9145            -9.5%       8277 ±  2%  slabinfo.proc_inode_cache.active_objs
      9145            -9.5%       8277 ±  2%  slabinfo.proc_inode_cache.num_objs
     17859           +26.4%      22573        slabinfo.radix_tree_node.active_objs
    637.33           +26.4%     805.83        slabinfo.radix_tree_node.active_slabs
     17859           +26.4%      22573        slabinfo.radix_tree_node.num_objs
    637.33           +26.4%     805.83        slabinfo.radix_tree_node.num_slabs
      4183 ± 76%  +16432.3%     691573 ± 18%  numa-meminfo.node0.Active
      4183 ± 76%  +16432.3%     691573 ± 18%  numa-meminfo.node0.Active(anon)
    594660 ±  2%    +117.1%    1290807 ±  9%  numa-meminfo.node0.FilePages
   1089205 ±  7%     +74.9%    1905557 ±  4%  numa-meminfo.node0.MemUsed
      2271 ± 15%     +19.0%       2702 ±  6%  numa-meminfo.node0.PageTables
     10495 ± 55%   +6560.5%     699019 ± 18%  numa-meminfo.node0.Shmem
     57656 ±  5%    +958.4%     610240 ± 18%  numa-meminfo.node1.Active
     57656 ±  5%    +958.4%     610240 ± 18%  numa-meminfo.node1.Active(anon)
    661155 ±  2%     +82.1%    1203912 ±  9%  numa-meminfo.node1.FilePages
   1058311 ±  7%     +59.0%    1683051 ±  4%  numa-meminfo.node1.MemUsed
     64345 ±  8%    +855.4%     614730 ± 18%  numa-meminfo.node1.Shmem
      1045 ± 76%  +16398.7%     172521 ± 18%  numa-vmstat.node0.nr_active_anon
    148663 ±  2%    +116.8%     322329 ±  9%  numa-vmstat.node0.nr_file_pages
    567.50 ± 16%     +19.2%     676.33 ±  6%  numa-vmstat.node0.nr_page_table_pages
      2621 ± 55%   +6551.1%     174381 ± 18%  numa-vmstat.node0.nr_shmem
      1045 ± 76%  +16398.7%     172521 ± 18%  numa-vmstat.node0.nr_zone_active_anon
    663612 ± 11%     +34.6%     893276 ±  4%  numa-vmstat.node0.numa_hit
    630366 ± 10%     +37.7%     867909 ±  5%  numa-vmstat.node0.numa_local
     14426 ±  5%    +955.1%     152209 ± 18%  numa-vmstat.node1.nr_active_anon
    165264 ±  2%     +81.9%     300627 ±  9%  numa-vmstat.node1.nr_file_pages
     16062 ±  8%    +854.6%     153332 ± 18%  numa-vmstat.node1.nr_shmem
     14426 ±  5%    +955.1%     152208 ± 18%  numa-vmstat.node1.nr_zone_active_anon
    721032 ± 10%     +19.7%     862855 ±  6%  numa-vmstat.node1.numa_hit
     15452         +2002.0%     324798        proc-vmstat.nr_active_anon
     59520            +4.1%      61985        proc-vmstat.nr_anon_pages
   2812461            -1.3%    2776554        proc-vmstat.nr_dirty_background_threshold
   5631801            -1.3%    5559898        proc-vmstat.nr_dirty_threshold
    313924           +98.5%     623025        proc-vmstat.nr_file_pages
  28307278            -1.3%   27947675        proc-vmstat.nr_free_pages
     62640            +3.5%      64853        proc-vmstat.nr_inactive_anon
      1155            +8.2%       1250        proc-vmstat.nr_page_table_pages
     18681         +1654.6%     327783        proc-vmstat.nr_shmem
     20318            +1.9%      20696        proc-vmstat.nr_slab_reclaimable
     25125            +2.8%      25818        proc-vmstat.nr_slab_unreclaimable
     15452         +2002.0%     324798        proc-vmstat.nr_zone_active_anon
     62640            +3.5%      64853        proc-vmstat.nr_zone_inactive_anon
    771109           +95.3%    1505616        proc-vmstat.numa_hit
    727606          +100.9%    1462123        proc-vmstat.numa_local
     62333 ± 25%     -44.5%      34565 ± 48%  proc-vmstat.numa_pte_updates
     23012           +88.6%      43401        proc-vmstat.pgactivate
    819919          +104.4%    1676106 ±  2%  proc-vmstat.pgalloc_normal
    901629            -3.4%     870576        proc-vmstat.pgfault
    783570           +19.4%     935666 ±  3%  proc-vmstat.pgfree
     72.90 ±  2%     -72.9        0.00        perf-profile.calltrace.cycles-pp.__libc_pread
     53.42 ±  2%     -53.4        0.00        perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__libc_pread
     35.64 ±  2%     -35.6        0.00        perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_pread
     33.46 ±  2%     -33.5        0.00        perf-profile.calltrace.cycles-pp.ksys_pread64.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_pread
     31.79 ±  2%     -31.8        0.00        perf-profile.calltrace.cycles-pp.vfs_read.ksys_pread64.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_pread
     26.04 ±  5%     -26.0        0.00        perf-profile.calltrace.cycles-pp.secondary_startup_64_no_verify
     25.98 ±  5%     -26.0        0.00        perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry
     25.65 ±  2%     -25.7        0.00        perf-profile.calltrace.cycles-pp.new_sync_read.vfs_read.ksys_pread64.do_syscall_64.entry_SYSCALL_64_after_hwframe
     25.08 ±  5%     -25.1        0.00        perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
     25.08 ±  5%     -25.1        0.00        perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
     25.08 ±  5%     -25.1        0.00        perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64_no_verify
     25.06 ±  5%     -25.1        0.00        perf-profile.calltrace.cycles-pp.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
     25.06 ±  5%     -25.1        0.00        perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary
     24.49 ±  2%     -24.5        0.00        perf-profile.calltrace.cycles-pp.shmem_file_read_iter.new_sync_read.vfs_read.ksys_pread64.do_syscall_64
     16.81           -16.8        0.00        perf-profile.calltrace.cycles-pp.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe.__libc_pread
     11.46           -11.5        0.00        perf-profile.calltrace.cycles-pp.copy_page_to_iter.shmem_file_read_iter.new_sync_read.vfs_read.ksys_pread64
      9.80            -9.8        0.00        perf-profile.calltrace.cycles-pp.copyout.copy_page_to_iter.shmem_file_read_iter.new_sync_read.vfs_read
      9.66            -9.7        0.00        perf-profile.calltrace.cycles-pp.copy_user_enhanced_fast_string.copyout.copy_page_to_iter.shmem_file_read_iter.new_sync_read
      9.16 ±  2%      -9.2        0.00        perf-profile.calltrace.cycles-pp.__entry_text_start.__libc_pread
      8.47            -8.5        0.00        perf-profile.calltrace.cycles-pp.syscall_return_via_sysret.__libc_pread
      6.27 ±  2%      -6.3        0.00        perf-profile.calltrace.cycles-pp.shmem_getpage_gfp.shmem_file_read_iter.new_sync_read.vfs_read.ksys_pread64
     73.26 ±  2%     -73.3        0.00        perf-profile.children.cycles-pp.__libc_pread
     53.67 ±  2%     -53.7        0.00        perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
     35.82 ±  2%     -35.8        0.00        perf-profile.children.cycles-pp.do_syscall_64
     33.53 ±  2%     -33.5        0.00        perf-profile.children.cycles-pp.ksys_pread64
     31.94 ±  2%     -31.9        0.00        perf-profile.children.cycles-pp.vfs_read
     26.04 ±  5%     -26.0        0.00        perf-profile.children.cycles-pp.secondary_startup_64_no_verify
     26.04 ±  5%     -26.0        0.00        perf-profile.children.cycles-pp.cpu_startup_entry
     26.04 ±  5%     -26.0        0.00        perf-profile.children.cycles-pp.do_idle
     26.02 ±  5%     -26.0        0.00        perf-profile.children.cycles-pp.cpuidle_enter
     26.02 ±  5%     -26.0        0.00        perf-profile.children.cycles-pp.cpuidle_enter_state
     25.99 ±  5%     -26.0        0.00        perf-profile.children.cycles-pp.intel_idle
     25.69 ±  2%     -25.7        0.00        perf-profile.children.cycles-pp.new_sync_read
     25.08 ±  5%     -25.1        0.00        perf-profile.children.cycles-pp.start_secondary
     24.59 ±  2%     -24.6        0.00        perf-profile.children.cycles-pp.shmem_file_read_iter
     16.90           -16.9        0.00        perf-profile.children.cycles-pp.syscall_exit_to_user_mode
     11.56           -11.6        0.00        perf-profile.children.cycles-pp.copy_page_to_iter
      9.87            -9.9        0.00        perf-profile.children.cycles-pp.copyout
      9.71            -9.7        0.00        perf-profile.children.cycles-pp.copy_user_enhanced_fast_string
      9.52 ±  2%      -9.5        0.00        perf-profile.children.cycles-pp.syscall_return_via_sysret
      8.98            -9.0        0.00        perf-profile.children.cycles-pp.__entry_text_start
      6.29 ±  2%      -6.3        0.00        perf-profile.children.cycles-pp.shmem_getpage_gfp
     25.99 ±  5%     -26.0        0.00        perf-profile.self.cycles-pp.intel_idle
     16.30           -16.3        0.00        perf-profile.self.cycles-pp.syscall_exit_to_user_mode
      9.60            -9.6        0.00        perf-profile.self.cycles-pp.copy_user_enhanced_fast_string
      9.48 ±  2%      -9.5        0.00        perf-profile.self.cycles-pp.syscall_return_via_sysret
      7.91 ±  2%      -7.9        0.00        perf-profile.self.cycles-pp.__entry_text_start
     19897 ± 14%    +948.1%     208538 ±107%  softirqs.CPU0.RCU
     22322 ± 25%    +435.8%     119597 ±145%  softirqs.CPU1.RCU
     25408 ± 22%     +86.4%      47368 ± 14%  softirqs.CPU10.RCU
     21039 ± 19%     +90.9%      40158 ± 18%  softirqs.CPU11.RCU
     15984 ± 17%    +706.3%     128875 ±137%  softirqs.CPU12.RCU
     20208 ±  8%    +705.7%     162828 ±117%  softirqs.CPU13.RCU
     20333 ± 22%     +64.4%      33427 ± 13%  softirqs.CPU14.RCU
     20027 ± 19%     +63.8%      32798 ± 24%  softirqs.CPU15.RCU
     23328 ± 19%     +79.4%      41842 ± 24%  softirqs.CPU16.RCU
     19957 ± 22%    +106.7%      41245 ± 24%  softirqs.CPU17.RCU
     18919 ± 31%    +108.0%      39358 ± 26%  softirqs.CPU18.RCU
     21789 ± 18%     +67.6%      36521 ± 16%  softirqs.CPU19.RCU
     24863 ± 29%     +82.5%      45387 ± 24%  softirqs.CPU2.RCU
     20013 ± 18%     +98.5%      39732 ± 14%  softirqs.CPU20.RCU
     20732 ± 18%    +112.8%      44129 ± 16%  softirqs.CPU21.RCU
     19935 ± 17%    +501.2%     119845 ±149%  softirqs.CPU22.RCU
     19639 ± 13%    +138.7%      46875 ± 16%  softirqs.CPU23.RCU
     22718 ± 16%     +78.6%      40573 ± 14%  softirqs.CPU24.RCU
     16177 ± 28%    +131.3%      37422 ± 15%  softirqs.CPU25.RCU
     17400 ± 18%     +97.5%      34370 ± 19%  softirqs.CPU26.RCU
     16930 ± 13%    +588.6%     116579 ±153%  softirqs.CPU27.RCU
     16811 ± 21%    +117.9%      36632 ± 18%  softirqs.CPU28.RCU
     18904 ± 21%    +537.8%     120568 ±149%  softirqs.CPU29.RCU
     23340 ± 24%     +88.6%      44009 ±  9%  softirqs.CPU3.RCU
     16236 ± 19%    +100.2%      32511 ± 26%  softirqs.CPU30.RCU
     17583 ± 22%     +87.9%      33031 ± 18%  softirqs.CPU31.RCU
     15133 ± 17%     +97.0%      29817 ± 10%  softirqs.CPU32.RCU
     14155 ±  6%    +142.6%      34335 ± 19%  softirqs.CPU33.RCU
     14907 ±  8%    +105.2%      30596 ± 17%  softirqs.CPU34.RCU
     17290 ± 26%    +106.8%      35749 ± 18%  softirqs.CPU35.RCU
     17657 ± 26%     +89.5%      33455 ± 13%  softirqs.CPU36.RCU
     15973 ± 12%    +123.0%      35617 ± 18%  softirqs.CPU37.RCU
     16076 ±  7%    +625.1%     116569 ±156%  softirqs.CPU38.RCU
     17260 ± 13%    +123.5%      38585 ± 12%  softirqs.CPU39.RCU
     22100 ± 24%    +116.2%      47774 ± 33%  softirqs.CPU4.RCU
     17436 ± 15%    +128.0%      39762 ± 17%  softirqs.CPU40.RCU
     18958 ± 30%    +538.8%     121115 ±145%  softirqs.CPU41.RCU
     19060 ± 14%    +110.9%      40191 ± 17%  softirqs.CPU42.RCU
     17989 ± 14%    +123.7%      40249 ± 21%  softirqs.CPU43.RCU
     18994 ± 16%    +106.2%      39168 ± 22%  softirqs.CPU44.RCU
     18178 ± 11%    +104.4%      37160 ± 23%  softirqs.CPU45.RCU
     19523 ± 12%     +90.3%      37155 ± 11%  softirqs.CPU46.RCU
     19627 ± 20%     +61.9%      31774 ± 15%  softirqs.CPU47.RCU
     20905 ± 17%    +104.9%      42825 ± 19%  softirqs.CPU5.RCU
     21597 ± 24%     +90.4%      41126 ± 27%  softirqs.CPU6.RCU
     21684 ± 19%    +365.1%     100860 ±117%  softirqs.CPU7.RCU
     22081 ± 14%    +112.4%      46911 ± 22%  softirqs.CPU8.RCU
    933853 ±  8%    +198.4%    2786674 ±  6%  softirqs.RCU
     25875 ±  5%     +66.9%      43189 ±  4%  softirqs.TIMER
      0.24 ± 15%    +108.0%       0.51 ±  6%  perf-stat.i.MPKI
 1.079e+10            +4.6%  1.128e+10        perf-stat.i.branch-instructions
      0.94            -0.0        0.91        perf-stat.i.branch-miss-rate%
 1.003e+08            +2.0%  1.023e+08        perf-stat.i.branch-misses
      1.90 ± 13%     +10.3       12.23 ±  9%  perf-stat.i.cache-miss-rate%
    271081 ±  3%   +1125.1%    3321051 ±  8%  perf-stat.i.cache-misses
  12121406 ± 15%    +124.5%   27215732 ±  7%  perf-stat.i.cache-references
      1200 ±  2%     +48.6%       1783 ±  2%  perf-stat.i.context-switches
      1.38            +3.2%       1.42        perf-stat.i.cpi
 7.223e+10            +8.0%  7.801e+10        perf-stat.i.cpu-cycles
     71.77            -2.8%      69.73        perf-stat.i.cpu-migrations
    391619 ±  2%     -92.1%      30968 ± 50%  perf-stat.i.cycles-between-cache-misses
      0.34            -0.0        0.33        perf-stat.i.dTLB-store-miss-rate%
  64281882            -1.2%   63510875        perf-stat.i.dTLB-store-misses
  33566536            -4.4%   32101494        perf-stat.i.iTLB-load-misses
 5.247e+10            +4.6%  5.489e+10        perf-stat.i.instructions
      1577            +9.5%       1726        perf-stat.i.instructions-per-iTLB-miss
      0.73            -3.1%       0.70        perf-stat.i.ipc
      0.40 ±  4%     -78.5%       0.09 ± 34%  perf-stat.i.major-faults
      1.50            +8.0%       1.63        perf-stat.i.metric.GHz
      0.72 ±  9%     -78.5%       0.15        perf-stat.i.metric.K/sec
      1123            +1.4%       1138        perf-stat.i.metric.M/sec
      2891            -3.6%       2786        perf-stat.i.minor-faults
     47.36            +1.6       48.98        perf-stat.i.node-load-miss-rate%
    107976 ±  3%   +1660.0%    1900404 ± 11%  perf-stat.i.node-load-misses
    130609         +1396.4%    1954489 ± 11%  perf-stat.i.node-loads
     34.59 ±  6%     +11.5       46.05 ±  4%  perf-stat.i.node-store-miss-rate%
     59311 ±  7%   +2382.6%    1472493 ± 27%  perf-stat.i.node-store-misses
    119688         +1282.0%    1654044 ± 25%  perf-stat.i.node-stores
      2891            -3.6%       2786        perf-stat.i.page-faults
      0.23 ± 15%    +114.6%       0.50 ±  7%  perf-stat.overall.MPKI
      0.93            -0.0        0.91        perf-stat.overall.branch-miss-rate%
      2.29 ± 15%     +10.0       12.25 ±  9%  perf-stat.overall.cache-miss-rate%
      1.38            +3.2%       1.42        perf-stat.overall.cpi
    266141 ±  3%     -91.1%      23675 ±  8%  perf-stat.overall.cycles-between-cache-misses
      0.34            -0.0        0.33        perf-stat.overall.dTLB-store-miss-rate%
      1563            +9.4%       1710        perf-stat.overall.instructions-per-iTLB-miss
      0.73            -3.1%       0.70        perf-stat.overall.ipc
     45.24            +4.1       49.29        perf-stat.overall.node-load-miss-rate%
     33.09 ±  4%     +13.9       46.98        perf-stat.overall.node-store-miss-rate%
    490908            +7.4%     527414        perf-stat.overall.path-length
 1.075e+10            +4.6%  1.124e+10        perf-stat.ps.branch-instructions
  99957025            +2.0%   1.02e+08        perf-stat.ps.branch-misses
    270839 ±  3%   +1122.1%    3309953 ±  8%  perf-stat.ps.cache-misses
  12084255 ± 15%    +124.5%   27126455 ±  7%  perf-stat.ps.cache-references
      1196 ±  2%     +48.5%       1777 ±  2%  perf-stat.ps.context-switches
 7.199e+10            +8.0%  7.774e+10        perf-stat.ps.cpu-cycles
     71.54            -2.8%      69.51        perf-stat.ps.cpu-migrations
  64064667            -1.2%   63294229        perf-stat.ps.dTLB-store-misses
  33453056            -4.4%   31991765        perf-stat.ps.iTLB-load-misses
 5.229e+10            +4.6%  5.471e+10        perf-stat.ps.instructions
      0.40 ±  4%     -78.4%       0.09 ± 34%  perf-stat.ps.major-faults
      2882            -3.6%       2777        perf-stat.ps.minor-faults
    107680 ±  3%   +1658.7%    1893794 ± 11%  perf-stat.ps.node-load-misses
    130298         +1394.8%    1947735 ± 11%  perf-stat.ps.node-loads
     59232 ±  7%   +2377.4%    1467437 ± 27%  perf-stat.ps.node-store-misses
    119549         +1278.9%    1648476 ± 25%  perf-stat.ps.node-stores
      2882            -3.7%       2777        perf-stat.ps.page-faults
 1.579e+13            +4.7%  1.653e+13        perf-stat.total.instructions
     57963            -6.5%      54206        interrupts.CAL:Function_call_interrupts
      2708 ± 34%    -100.0%       0.00        interrupts.CPU0.NMI:Non-maskable_interrupts
      2708 ± 34%    -100.0%       0.00        interrupts.CPU0.PMI:Performance_monitoring_interrupts
      3773 ± 61%    -100.0%       0.00        interrupts.CPU1.NMI:Non-maskable_interrupts
      3773 ± 61%    -100.0%       0.00        interrupts.CPU1.PMI:Performance_monitoring_interrupts
      5117 ± 48%    -100.0%       0.00        interrupts.CPU10.NMI:Non-maskable_interrupts
      5117 ± 48%    -100.0%       0.00        interrupts.CPU10.PMI:Performance_monitoring_interrupts
      3932 ± 31%    -100.0%       0.00        interrupts.CPU11.NMI:Non-maskable_interrupts
      3932 ± 31%    -100.0%       0.00        interrupts.CPU11.PMI:Performance_monitoring_interrupts
      4655 ± 60%    -100.0%       0.00        interrupts.CPU12.NMI:Non-maskable_interrupts
      4655 ± 60%    -100.0%       0.00        interrupts.CPU12.PMI:Performance_monitoring_interrupts
      4903 ± 51%    -100.0%       0.00        interrupts.CPU13.NMI:Non-maskable_interrupts
      4903 ± 51%    -100.0%       0.00        interrupts.CPU13.PMI:Performance_monitoring_interrupts
      4380 ± 64%    -100.0%       0.00        interrupts.CPU14.NMI:Non-maskable_interrupts
      4380 ± 64%    -100.0%       0.00        interrupts.CPU14.PMI:Performance_monitoring_interrupts
      3342 ± 54%    -100.0%       0.00        interrupts.CPU15.NMI:Non-maskable_interrupts
      3342 ± 54%    -100.0%       0.00        interrupts.CPU15.PMI:Performance_monitoring_interrupts
      4933 ± 50%    -100.0%       0.00        interrupts.CPU16.NMI:Non-maskable_interrupts
      4933 ± 50%    -100.0%       0.00        interrupts.CPU16.PMI:Performance_monitoring_interrupts
      4272 ± 67%    -100.0%       0.00        interrupts.CPU17.NMI:Non-maskable_interrupts
      4272 ± 67%    -100.0%       0.00        interrupts.CPU17.PMI:Performance_monitoring_interrupts
      4640 ± 37%    -100.0%       0.00        interrupts.CPU18.NMI:Non-maskable_interrupts
      4640 ± 37%    -100.0%       0.00        interrupts.CPU18.PMI:Performance_monitoring_interrupts
      3997 ± 28%    -100.0%       0.00        interrupts.CPU19.NMI:Non-maskable_interrupts
      3997 ± 28%    -100.0%       0.00        interrupts.CPU19.PMI:Performance_monitoring_interrupts
      4229 ± 47%    -100.0%       0.00        interrupts.CPU2.NMI:Non-maskable_interrupts
      4229 ± 47%    -100.0%       0.00        interrupts.CPU2.PMI:Performance_monitoring_interrupts
      3888 ± 55%    -100.0%       0.00        interrupts.CPU20.NMI:Non-maskable_interrupts
      3888 ± 55%    -100.0%       0.00        interrupts.CPU20.PMI:Performance_monitoring_interrupts
      5098 ± 48%    -100.0%       0.00        interrupts.CPU21.NMI:Non-maskable_interrupts
      5098 ± 48%    -100.0%       0.00        interrupts.CPU21.PMI:Performance_monitoring_interrupts
      4788 ± 51%    -100.0%       0.00        interrupts.CPU22.NMI:Non-maskable_interrupts
      4788 ± 51%    -100.0%       0.00        interrupts.CPU22.PMI:Performance_monitoring_interrupts
      4535 ± 42%    -100.0%       0.00        interrupts.CPU23.NMI:Non-maskable_interrupts
      4535 ± 42%    -100.0%       0.00        interrupts.CPU23.PMI:Performance_monitoring_interrupts
      6687 ± 34%    -100.0%       0.00        interrupts.CPU24.NMI:Non-maskable_interrupts
      6687 ± 34%    -100.0%       0.00        interrupts.CPU24.PMI:Performance_monitoring_interrupts
      3988 ± 52%    -100.0%       0.00        interrupts.CPU25.NMI:Non-maskable_interrupts
      3988 ± 52%    -100.0%       0.00        interrupts.CPU25.PMI:Performance_monitoring_interrupts
      2905 ± 37%    -100.0%       0.00        interrupts.CPU26.NMI:Non-maskable_interrupts
      2905 ± 37%    -100.0%       0.00        interrupts.CPU26.PMI:Performance_monitoring_interrupts
      4086 ± 66%    -100.0%       0.00        interrupts.CPU27.NMI:Non-maskable_interrupts
      4086 ± 66%    -100.0%       0.00        interrupts.CPU27.PMI:Performance_monitoring_interrupts
      5137 ± 53%    -100.0%       0.00        interrupts.CPU28.NMI:Non-maskable_interrupts
      5137 ± 53%    -100.0%       0.00        interrupts.CPU28.PMI:Performance_monitoring_interrupts
      4698 ± 43%    -100.0%       0.00        interrupts.CPU29.NMI:Non-maskable_interrupts
      4698 ± 43%    -100.0%       0.00        interrupts.CPU29.PMI:Performance_monitoring_interrupts
      6014 ± 39%    -100.0%       0.00        interrupts.CPU3.NMI:Non-maskable_interrupts
      6014 ± 39%    -100.0%       0.00        interrupts.CPU3.PMI:Performance_monitoring_interrupts
      3091 ± 30%    -100.0%       0.00        interrupts.CPU30.NMI:Non-maskable_interrupts
      3091 ± 30%    -100.0%       0.00        interrupts.CPU30.PMI:Performance_monitoring_interrupts
      4042 ± 52%    -100.0%       0.00        interrupts.CPU31.NMI:Non-maskable_interrupts
      4042 ± 52%    -100.0%       0.00        interrupts.CPU31.PMI:Performance_monitoring_interrupts
      2411 ± 43%    -100.0%       0.00        interrupts.CPU32.NMI:Non-maskable_interrupts
      2411 ± 43%    -100.0%       0.00        interrupts.CPU32.PMI:Performance_monitoring_interrupts
      1840 ± 26%     -37.6%       1148 ± 23%  interrupts.CPU33.CAL:Function_call_interrupts
      2570 ± 96%    -100.0%       0.00        interrupts.CPU33.NMI:Non-maskable_interrupts
      2570 ± 96%    -100.0%       0.00        interrupts.CPU33.PMI:Performance_monitoring_interrupts
    100.00 ± 16%     +77.2%     177.17 ± 28%  interrupts.CPU33.RES:Rescheduling_interrupts
      2373 ± 42%    -100.0%       0.00        interrupts.CPU34.NMI:Non-maskable_interrupts
      2373 ± 42%    -100.0%       0.00        interrupts.CPU34.PMI:Performance_monitoring_interrupts
      3532 ± 73%    -100.0%       0.00        interrupts.CPU35.NMI:Non-maskable_interrupts
      3532 ± 73%    -100.0%       0.00        interrupts.CPU35.PMI:Performance_monitoring_interrupts
      3293 ± 74%    -100.0%       0.00        interrupts.CPU36.NMI:Non-maskable_interrupts
      3293 ± 74%    -100.0%       0.00        interrupts.CPU36.PMI:Performance_monitoring_interrupts
      2805 ± 42%    -100.0%       0.00        interrupts.CPU37.NMI:Non-maskable_interrupts
      2805 ± 42%    -100.0%       0.00        interrupts.CPU37.PMI:Performance_monitoring_interrupts
      4824 ± 42%    -100.0%       0.00        interrupts.CPU38.NMI:Non-maskable_interrupts
      4824 ± 42%    -100.0%       0.00        interrupts.CPU38.PMI:Performance_monitoring_interrupts
    120.00 ± 52%     +89.4%     227.33 ± 23%  interrupts.CPU38.RES:Rescheduling_interrupts
      4038 ± 52%    -100.0%       0.00        interrupts.CPU39.NMI:Non-maskable_interrupts
      4038 ± 52%    -100.0%       0.00        interrupts.CPU39.PMI:Performance_monitoring_interrupts
      3067 ± 39%    -100.0%       0.00        interrupts.CPU4.NMI:Non-maskable_interrupts
      3067 ± 39%    -100.0%       0.00        interrupts.CPU4.PMI:Performance_monitoring_interrupts
      4436 ± 60%    -100.0%       1.00 ±223%  interrupts.CPU40.NMI:Non-maskable_interrupts
      4436 ± 60%    -100.0%       1.00 ±223%  interrupts.CPU40.PMI:Performance_monitoring_interrupts
      4425 ± 40%    -100.0%       0.00        interrupts.CPU41.NMI:Non-maskable_interrupts
      4425 ± 40%    -100.0%       0.00        interrupts.CPU41.PMI:Performance_monitoring_interrupts
      3077 ± 79%    -100.0%       0.00        interrupts.CPU42.NMI:Non-maskable_interrupts
      3077 ± 79%    -100.0%       0.00        interrupts.CPU42.PMI:Performance_monitoring_interrupts
      2626 ± 32%    -100.0%       0.00        interrupts.CPU43.NMI:Non-maskable_interrupts
      2626 ± 32%    -100.0%       0.00        interrupts.CPU43.PMI:Performance_monitoring_interrupts
      5302 ± 43%    -100.0%       0.00        interrupts.CPU44.NMI:Non-maskable_interrupts
      5302 ± 43%    -100.0%       0.00        interrupts.CPU44.PMI:Performance_monitoring_interrupts
      2654 ± 35%    -100.0%       0.00        interrupts.CPU45.NMI:Non-maskable_interrupts
      2654 ± 35%    -100.0%       0.00        interrupts.CPU45.PMI:Performance_monitoring_interrupts
      3091 ± 41%    -100.0%       0.00        interrupts.CPU46.NMI:Non-maskable_interrupts
      3091 ± 41%    -100.0%       0.00        interrupts.CPU46.PMI:Performance_monitoring_interrupts
      3730 ± 44%    -100.0%       0.00        interrupts.CPU47.NMI:Non-maskable_interrupts
      3730 ± 44%    -100.0%       0.00        interrupts.CPU47.PMI:Performance_monitoring_interrupts
      4877 ± 53%    -100.0%       0.00        interrupts.CPU5.NMI:Non-maskable_interrupts
      4877 ± 53%    -100.0%       0.00        interrupts.CPU5.PMI:Performance_monitoring_interrupts
      3268 ± 29%    -100.0%       0.00        interrupts.CPU6.NMI:Non-maskable_interrupts
      3268 ± 29%    -100.0%       0.00        interrupts.CPU6.PMI:Performance_monitoring_interrupts
      5376 ± 50%    -100.0%       0.00        interrupts.CPU7.NMI:Non-maskable_interrupts
      5376 ± 50%    -100.0%       0.00        interrupts.CPU7.PMI:Performance_monitoring_interrupts
      3774 ± 23%    -100.0%       0.00        interrupts.CPU8.NMI:Non-maskable_interrupts
      3774 ± 23%    -100.0%       0.00        interrupts.CPU8.PMI:Performance_monitoring_interrupts
      6783 ± 32%    -100.0%       0.00        interrupts.CPU9.NMI:Non-maskable_interrupts
      6783 ± 32%    -100.0%       0.00        interrupts.CPU9.PMI:Performance_monitoring_interrupts
    289.33 ±  5%     -21.8%     226.17 ± 15%  interrupts.CPU9.RES:Rescheduling_interrupts
    196194 ±  3%    -100.0%       1.00 ±223%  interrupts.NMI:Non-maskable_interrupts
    196194 ±  3%    -100.0%       1.00 ±223%  interrupts.PMI:Performance_monitoring_interrupts
      0.01 ±  5%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown]
      0.01 ±  7%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.devkmsg_read.vfs_read.ksys_read.do_syscall_64
      0.01 ±  5%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.do_nanosleep.hrtimer_nanosleep.__x64_sys_nanosleep.do_syscall_64
      0.01 ±  8%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.do_syslog.part.0.kmsg_read.vfs_read
      0.01          -100.0%       0.00        perf-sched.sch_delay.avg.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
      0.01          -100.0%       0.00        perf-sched.sch_delay.avg.ms.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64
      0.01 ± 12%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.futex_wait_queue_me.futex_wait.do_futex.__x64_sys_futex
      0.00          -100.0%       0.00        perf-sched.sch_delay.avg.ms.pipe_read.new_sync_read.vfs_read.ksys_read
      0.01 ± 40%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.preempt_schedule_common.__cond_resched.stop_one_cpu.sched_exec.bprm_execve
      0.01 ±  7%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.rcu_gp_kthread.kthread.ret_from_fork
      0.01 ±  5%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait.__x64_sys_epoll_wait
      0.01 ± 26%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_select
      0.00 ± 14%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_sys_poll
      0.01 ±  6%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.schedule_timeout.kcompactd.kthread.ret_from_fork
      0.01 ±  7%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.schedule_timeout.rcu_gp_kthread.kthread.ret_from_fork
      0.02 ± 40%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.smpboot_thread_fn.kthread.ret_from_fork
      0.01 ±  8%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.wait_for_partner.fifo_open.do_dentry_open.do_open.isra
      0.04 ±  4%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.worker_thread.kthread.ret_from_fork
      0.01 ± 21%    -100.0%       0.00        perf-sched.sch_delay.max.ms.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown]
      0.02 ±  9%    -100.0%       0.00        perf-sched.sch_delay.max.ms.devkmsg_read.vfs_read.ksys_read.do_syscall_64
      0.02 ± 45%    -100.0%       0.00        perf-sched.sch_delay.max.ms.do_nanosleep.hrtimer_nanosleep.__x64_sys_nanosleep.do_syscall_64
      0.02 ± 17%    -100.0%       0.00        perf-sched.sch_delay.max.ms.do_syslog.part.0.kmsg_read.vfs_read
      0.03 ± 25%    -100.0%       0.00        perf-sched.sch_delay.max.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
      0.04 ± 44%    -100.0%       0.00        perf-sched.sch_delay.max.ms.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64
      0.01 ± 20%    -100.0%       0.00        perf-sched.sch_delay.max.ms.exit_to_user_mode_prepare.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe.[unknown]
      0.02 ± 21%    -100.0%       0.00        perf-sched.sch_delay.max.ms.futex_wait_queue_me.futex_wait.do_futex.__x64_sys_futex
      0.04 ± 14%    -100.0%       0.00        perf-sched.sch_delay.max.ms.pipe_read.new_sync_read.vfs_read.ksys_read
      0.02 ± 11%    -100.0%       0.00        perf-sched.sch_delay.max.ms.preempt_schedule_common.__cond_resched.stop_one_cpu.affine_move_task.__set_cpus_allowed_ptr
      0.02 ± 48%    -100.0%       0.00        perf-sched.sch_delay.max.ms.preempt_schedule_common.__cond_resched.stop_one_cpu.sched_exec.bprm_execve
      0.01 ± 30%    -100.0%       0.00        perf-sched.sch_delay.max.ms.rcu_gp_kthread.kthread.ret_from_fork
      0.02 ± 21%    -100.0%       0.00        perf-sched.sch_delay.max.ms.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait.__x64_sys_epoll_wait
      0.01 ± 23%    -100.0%       0.00        perf-sched.sch_delay.max.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_select
      0.01 ± 10%    -100.0%       0.00        perf-sched.sch_delay.max.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_sys_poll
      0.02 ± 41%    -100.0%       0.00        perf-sched.sch_delay.max.ms.schedule_timeout.kcompactd.kthread.ret_from_fork
      0.03 ± 31%    -100.0%       0.00        perf-sched.sch_delay.max.ms.schedule_timeout.rcu_gp_kthread.kthread.ret_from_fork
      6.02 ± 51%    -100.0%       0.00        perf-sched.sch_delay.max.ms.smpboot_thread_fn.kthread.ret_from_fork
      0.02 ±  7%    -100.0%       0.00        perf-sched.sch_delay.max.ms.wait_for_partner.fifo_open.do_dentry_open.do_open.isra
     12.45          -100.0%       0.00        perf-sched.sch_delay.max.ms.worker_thread.kthread.ret_from_fork
      0.01 ±  9%    -100.0%       0.00        perf-sched.total_sch_delay.average.ms
     12.45          -100.0%       0.00        perf-sched.total_sch_delay.max.ms
    162.59 ±  7%    -100.0%       0.00        perf-sched.total_wait_and_delay.average.ms
      6367 ±  9%    -100.0%       0.00        perf-sched.total_wait_and_delay.count.ms
      7735 ±  4%    -100.0%       0.00        perf-sched.total_wait_and_delay.max.ms
    162.58 ±  7%    -100.0%       0.00        perf-sched.total_wait_time.average.ms
      7735 ±  4%    -100.0%       0.00        perf-sched.total_wait_time.max.ms
    899.64          -100.0%       0.00        perf-sched.wait_and_delay.avg.ms.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown]
    469.21 ± 73%    -100.0%       0.00        perf-sched.wait_and_delay.avg.ms.devkmsg_read.vfs_read.ksys_read.do_syscall_64
    808.95 ±  4%    -100.0%       0.00        perf-sched.wait_and_delay.avg.ms.do_nanosleep.hrtimer_nanosleep.__x64_sys_nanosleep.do_syscall_64
    469.22 ± 73%    -100.0%       0.00        perf-sched.wait_and_delay.avg.ms.do_syslog.part.0.kmsg_read.vfs_read
    272.98 ±  5%    -100.0%       0.00        perf-sched.wait_and_delay.avg.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
      1.43          -100.0%       0.00        perf-sched.wait_and_delay.avg.ms.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64
      0.09 ± 61%    -100.0%       0.00        perf-sched.wait_and_delay.avg.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt.[unknown]
      0.04 ± 16%    -100.0%       0.00        perf-sched.wait_and_delay.avg.ms.exit_to_user_mode_prepare.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe.[unknown]
     71.33 ±  2%    -100.0%       0.00        perf-sched.wait_and_delay.avg.ms.pipe_read.new_sync_read.vfs_read.ksys_read
      0.04 ± 22%    -100.0%       0.00        perf-sched.wait_and_delay.avg.ms.preempt_schedule_common.__cond_resched.pagecache_get_page.shmem_getpage_gfp.shmem_file_read_iter
     17.39          -100.0%       0.00        perf-sched.wait_and_delay.avg.ms.preempt_schedule_common.__cond_resched.stop_one_cpu.affine_move_task.__set_cpus_allowed_ptr
    587.99 ±  9%    -100.0%       0.00        perf-sched.wait_and_delay.avg.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_sys_poll
    478.78          -100.0%       0.00        perf-sched.wait_and_delay.avg.ms.schedule_timeout.kcompactd.kthread.ret_from_fork
      9.25 ± 31%    -100.0%       0.00        perf-sched.wait_and_delay.avg.ms.schedule_timeout.rcu_gp_kthread.kthread.ret_from_fork
    652.76          -100.0%       0.00        perf-sched.wait_and_delay.avg.ms.smpboot_thread_fn.kthread.ret_from_fork
      0.01 ±  8%    -100.0%       0.00        perf-sched.wait_and_delay.avg.ms.wait_for_partner.fifo_open.do_dentry_open.do_open.isra
    528.39 ±  2%    -100.0%       0.00        perf-sched.wait_and_delay.avg.ms.worker_thread.kthread.ret_from_fork
     10.00          -100.0%       0.00        perf-sched.wait_and_delay.count.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown]
      7.33 ±  6%    -100.0%       0.00        perf-sched.wait_and_delay.count.devkmsg_read.vfs_read.ksys_read.do_syscall_64
     22.33 ±  4%    -100.0%       0.00        perf-sched.wait_and_delay.count.do_nanosleep.hrtimer_nanosleep.__x64_sys_nanosleep.do_syscall_64
      7.33 ±  6%    -100.0%       0.00        perf-sched.wait_and_delay.count.do_syslog.part.0.kmsg_read.vfs_read
    246.67          -100.0%       0.00        perf-sched.wait_and_delay.count.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
    245.67          -100.0%       0.00        perf-sched.wait_and_delay.count.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64
     47.83 ± 35%    -100.0%       0.00        perf-sched.wait_and_delay.count.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt.[unknown]
    185.00 ± 10%    -100.0%       0.00        perf-sched.wait_and_delay.count.exit_to_user_mode_prepare.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe.[unknown]
      2200          -100.0%       0.00        perf-sched.wait_and_delay.count.pipe_read.new_sync_read.vfs_read.ksys_read
    113.33 ± 11%    -100.0%       0.00        perf-sched.wait_and_delay.count.preempt_schedule_common.__cond_resched.pagecache_get_page.shmem_getpage_gfp.shmem_file_read_iter
    576.00          -100.0%       0.00        perf-sched.wait_and_delay.count.preempt_schedule_common.__cond_resched.stop_one_cpu.affine_move_task.__set_cpus_allowed_ptr
     50.67 ± 11%    -100.0%       0.00        perf-sched.wait_and_delay.count.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_sys_poll
     40.00          -100.0%       0.00        perf-sched.wait_and_delay.count.schedule_timeout.kcompactd.kthread.ret_from_fork
      1261 ± 52%    -100.0%       0.00        perf-sched.wait_and_delay.count.schedule_timeout.rcu_gp_kthread.kthread.ret_from_fork
    680.33          -100.0%       0.00        perf-sched.wait_and_delay.count.smpboot_thread_fn.kthread.ret_from_fork
     72.00          -100.0%       0.00        perf-sched.wait_and_delay.count.wait_for_partner.fifo_open.do_dentry_open.do_open.isra
    456.50          -100.0%       0.00        perf-sched.wait_and_delay.count.worker_thread.kthread.ret_from_fork
    999.63          -100.0%       0.00        perf-sched.wait_and_delay.max.ms.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown]
      3039 ± 72%    -100.0%       0.00        perf-sched.wait_and_delay.max.ms.devkmsg_read.vfs_read.ksys_read.do_syscall_64
      1000          -100.0%       0.00        perf-sched.wait_and_delay.max.ms.do_nanosleep.hrtimer_nanosleep.__x64_sys_nanosleep.do_syscall_64
      3039 ± 72%    -100.0%       0.00        perf-sched.wait_and_delay.max.ms.do_syslog.part.0.kmsg_read.vfs_read
      3228 ± 60%    -100.0%       0.00        perf-sched.wait_and_delay.max.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
     31.13          -100.0%       0.00        perf-sched.wait_and_delay.max.ms.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64
      1.00 ±178%    -100.0%       0.00        perf-sched.wait_and_delay.max.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt.[unknown]
      0.58 ±115%    -100.0%       0.00        perf-sched.wait_and_delay.max.ms.exit_to_user_mode_prepare.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe.[unknown]
      3041 ± 72%    -100.0%       0.00        perf-sched.wait_and_delay.max.ms.pipe_read.new_sync_read.vfs_read.ksys_read
      0.17 ± 52%    -100.0%       0.00        perf-sched.wait_and_delay.max.ms.preempt_schedule_common.__cond_resched.pagecache_get_page.shmem_getpage_gfp.shmem_file_read_iter
      1029 ±  2%    -100.0%       0.00        perf-sched.wait_and_delay.max.ms.preempt_schedule_common.__cond_resched.stop_one_cpu.affine_move_task.__set_cpus_allowed_ptr
      6493 ± 16%    -100.0%       0.00        perf-sched.wait_and_delay.max.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_sys_poll
    504.51          -100.0%       0.00        perf-sched.wait_and_delay.max.ms.schedule_timeout.kcompactd.kthread.ret_from_fork
    289.34 ± 25%    -100.0%       0.00        perf-sched.wait_and_delay.max.ms.schedule_timeout.rcu_gp_kthread.kthread.ret_from_fork
      2708 ± 61%    -100.0%       0.00        perf-sched.wait_and_delay.max.ms.smpboot_thread_fn.kthread.ret_from_fork
      0.02 ±  7%    -100.0%       0.00        perf-sched.wait_and_delay.max.ms.wait_for_partner.fifo_open.do_dentry_open.do_open.isra
      6444 ± 17%    -100.0%       0.00        perf-sched.wait_and_delay.max.ms.worker_thread.kthread.ret_from_fork
    899.63          -100.0%       0.00        perf-sched.wait_time.avg.ms.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown]
    469.20 ± 73%    -100.0%       0.00        perf-sched.wait_time.avg.ms.devkmsg_read.vfs_read.ksys_read.do_syscall_64
    808.94 ±  4%    -100.0%       0.00        perf-sched.wait_time.avg.ms.do_nanosleep.hrtimer_nanosleep.__x64_sys_nanosleep.do_syscall_64
    469.20 ± 73%    -100.0%       0.00        perf-sched.wait_time.avg.ms.do_syslog.part.0.kmsg_read.vfs_read
    272.97 ±  5%    -100.0%       0.00        perf-sched.wait_time.avg.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
      1.41          -100.0%       0.00        perf-sched.wait_time.avg.ms.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64
      0.09 ± 61%    -100.0%       0.00        perf-sched.wait_time.avg.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt.[unknown]
      0.06 ± 90%    -100.0%       0.00        perf-sched.wait_time.avg.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_call_function_single.[unknown]
      0.04 ± 21%    -100.0%       0.00        perf-sched.wait_time.avg.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_reschedule_ipi.[unknown]
      0.04 ± 16%    -100.0%       0.00        perf-sched.wait_time.avg.ms.exit_to_user_mode_prepare.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe.[unknown]
    537.84 ±134%    -100.0%       0.00        perf-sched.wait_time.avg.ms.futex_wait_queue_me.futex_wait.do_futex.__x64_sys_futex
     71.33 ±  2%    -100.0%       0.00        perf-sched.wait_time.avg.ms.pipe_read.new_sync_read.vfs_read.ksys_read
      0.03          -100.0%       0.00        perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.copy_page_to_iter.shmem_file_read_iter.new_sync_read
      0.04 ± 22%    -100.0%       0.00        perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.pagecache_get_page.shmem_getpage_gfp.shmem_file_read_iter
     17.39          -100.0%       0.00        perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.stop_one_cpu.affine_move_task.__set_cpus_allowed_ptr
      2.56 ± 13%    -100.0%       0.00        perf-sched.wait_time.avg.ms.rcu_gp_kthread.kthread.ret_from_fork
    365.76 ± 36%    -100.0%       0.00        perf-sched.wait_time.avg.ms.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait.__x64_sys_epoll_wait
    587.98 ±  9%    -100.0%       0.00        perf-sched.wait_time.avg.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_sys_poll
    478.78          -100.0%       0.00        perf-sched.wait_time.avg.ms.schedule_timeout.kcompactd.kthread.ret_from_fork
      0.02 ± 39%    -100.0%       0.00        perf-sched.wait_time.avg.ms.schedule_timeout.khugepaged.kthread.ret_from_fork
      9.25 ± 31%    -100.0%       0.00        perf-sched.wait_time.avg.ms.schedule_timeout.rcu_gp_kthread.kthread.ret_from_fork
    652.75          -100.0%       0.00        perf-sched.wait_time.avg.ms.smpboot_thread_fn.kthread.ret_from_fork
    528.35 ±  2%    -100.0%       0.00        perf-sched.wait_time.avg.ms.worker_thread.kthread.ret_from_fork
    999.62          -100.0%       0.00        perf-sched.wait_time.max.ms.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown]
      3039 ± 72%    -100.0%       0.00        perf-sched.wait_time.max.ms.devkmsg_read.vfs_read.ksys_read.do_syscall_64
      1000          -100.0%       0.00        perf-sched.wait_time.max.ms.do_nanosleep.hrtimer_nanosleep.__x64_sys_nanosleep.do_syscall_64
      3039 ± 72%    -100.0%       0.00        perf-sched.wait_time.max.ms.do_syslog.part.0.kmsg_read.vfs_read
      3228 ± 60%    -100.0%       0.00        perf-sched.wait_time.max.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
     31.12          -100.0%       0.00        perf-sched.wait_time.max.ms.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64
      1.00 ±178%    -100.0%       0.00        perf-sched.wait_time.max.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt.[unknown]
      0.06 ± 90%    -100.0%       0.00        perf-sched.wait_time.max.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_call_function_single.[unknown]
      0.09 ± 44%    -100.0%       0.00        perf-sched.wait_time.max.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_reschedule_ipi.[unknown]
      0.58 ±115%    -100.0%       0.00        perf-sched.wait_time.max.ms.exit_to_user_mode_prepare.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe.[unknown]
      2081 ±138%    -100.0%       0.00        perf-sched.wait_time.max.ms.futex_wait_queue_me.futex_wait.do_futex.__x64_sys_futex
      3041 ± 72%    -100.0%       0.00        perf-sched.wait_time.max.ms.pipe_read.new_sync_read.vfs_read.ksys_read
      0.10 ± 41%    -100.0%       0.00        perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.copy_page_to_iter.shmem_file_read_iter.new_sync_read
      0.17 ± 52%    -100.0%       0.00        perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.pagecache_get_page.shmem_getpage_gfp.shmem_file_read_iter
      1029 ±  2%    -100.0%       0.00        perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.stop_one_cpu.affine_move_task.__set_cpus_allowed_ptr
      4.49 ± 11%    -100.0%       0.00        perf-sched.wait_time.max.ms.rcu_gp_kthread.kthread.ret_from_fork
      3794 ± 70%    -100.0%       0.00        perf-sched.wait_time.max.ms.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait.__x64_sys_epoll_wait
      6493 ± 16%    -100.0%       0.00        perf-sched.wait_time.max.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_sys_poll
    504.50          -100.0%       0.00        perf-sched.wait_time.max.ms.schedule_timeout.kcompactd.kthread.ret_from_fork
      0.02 ± 39%    -100.0%       0.00        perf-sched.wait_time.max.ms.schedule_timeout.khugepaged.kthread.ret_from_fork
    289.33 ± 25%    -100.0%       0.00        perf-sched.wait_time.max.ms.schedule_timeout.rcu_gp_kthread.kthread.ret_from_fork
      2708 ± 61%    -100.0%       0.00        perf-sched.wait_time.max.ms.smpboot_thread_fn.kthread.ret_from_fork
      6444 ± 17%    -100.0%       0.00        perf-sched.wait_time.max.ms.worker_thread.kthread.ret_from_fork

***************************************************************************************************
lkp-skl-fpga01: 104 threads Skylake with 192G memory
=========================================================================================
compiler/cpufreq_governor/kconfig/mode/nr_task/rootfs/tbox_group/test/testcase/ucode:
gcc-9/performance/x86_64-rhel-8.3/process/50%/debian-10.4-x86_64-20200603.cgz/lkp-skl-fpga01/brk1/will-it-scale/0x2006a0a

commit:
dde38d5b80 ("kernel: don't include PF_IO_WORKERs as part of same_thread_group()") 43b2a76b1a ("proc: don't show PF_IO_WORKER threads as threads in /proc/<pid>/task/")

dde38d5b805358c7 43b2a76b1a5abcc9833b463bef1
---------------- ---------------------------
%stddev %change %stddev
\ | \

  66531819           -15.3%   56377631        will-it-scale.52.processes
     49.47            -3.8%      47.59        will-it-scale.52.processes_idle
   1279457           -15.3%    1084184        will-it-scale.per_process_ops
      9487            +0.9%       9569        will-it-scale.time.maximum_resident_set_size
  66531819           -15.3%   56377631        will-it-scale.workload
     52376 ±  7%     +32.8%      69564 ± 27%  cpuidle.POLL.time
      0.04 ±  5%      +0.0        0.05 ±  6%  mpstat.cpu.all.soft%
     27.94            +3.2       31.18        mpstat.cpu.all.sys%
     20.15            -3.0       17.13        mpstat.cpu.all.usr%
    534.28 ±  5%     -41.2%     314.38 ±  6%  sched_debug.cfs_rq:/.util_est_enqueued.avg
    423.00 ±  2%     -29.6%     297.65 ±  3%  sched_debug.cfs_rq:/.util_est_enqueued.stddev
      3100 ± 17%     +42.9%       4430 ± 16%  sched_debug.cpu.nr_switches.stddev
    357400 ±  8%    +116.1%     772492 ± 11%  numa-numastat.node0.local_node
    408607          +101.1%     821517 ±  8%  numa-numastat.node0.numa_hit
    431312 ±  6%     +73.2%     747209 ±  7%  numa-numastat.node1.local_node
    473757           +67.1%     791872 ±  4%  numa-numastat.node1.numa_hit
    494.33 ±  9%  +35972.6%     178318 ± 34%  numa-vmstat.node0.nr_active_anon
    151965 ±  2%    +118.3%     331721 ± 18%  numa-vmstat.node0.nr_file_pages
      3549 ± 25%   +5006.2%     181234 ± 33%  numa-vmstat.node0.nr_shmem
    494.33 ±  9%  +35972.6%     178318 ± 34%  numa-vmstat.node0.nr_zone_active_anon
    994186 ±  6%     +25.6%    1248383 ± 10%  numa-vmstat.node0.numa_hit
     50.00            -4.0%      48.00        vmstat.cpu.id
     20.00           -11.7%      17.67 ±  2%  vmstat.cpu.us
   1433837           +79.8%    2577361        vmstat.memory.cache
      1572 ±  5%     +31.2%       2063 ±  4%  vmstat.system.cs
    211278            -1.8%     207503        vmstat.system.in
      2088 ±  9%  +34105.3%     714491 ± 34%  numa-meminfo.node0.Active
      1982 ±  9%  +35929.8%     714170 ± 34%  numa-meminfo.node0.Active(anon)
    607865 ±  2%    +118.4%    1327776 ± 18%  numa-meminfo.node0.FilePages
   1359425 ±  3%     +65.5%    2249289 ± 12%  numa-meminfo.node0.MemUsed
     14202 ± 25%   +5010.6%     725830 ± 33%  numa-meminfo.node0.Shmem
   1323966 ±  3%     +50.8%    1996033 ± 12%  numa-meminfo.node1.MemUsed
    130438 ±  3%    +876.0%    1273092 ±  3%  meminfo.Active
    130230 ±  3%    +877.2%    1272567 ±  3%  meminfo.Active(anon)
   1334231           +85.6%    2476409        meminfo.Cached
    971466          +123.4%    2170088 ±  2%  meminfo.Committed_AS
   2683686           +58.2%    4245240        meminfo.Memused
    151555 ±  2%    +751.5%    1290535 ±  3%  meminfo.Shmem
   3145259           +35.0%    4245265        meminfo.max_used_kB
      8674 ±  3%      +9.2%       9469 ±  2%  slabinfo.kmalloc-256.active_objs
      8722 ±  3%      +9.7%       9564 ±  2%  slabinfo.kmalloc-256.num_objs
    959.83 ±  8%     +15.3%       1107 ±  5%  slabinfo.mnt_cache.active_objs
    959.83 ±  8%     +15.3%       1107 ±  5%  slabinfo.mnt_cache.num_objs
     14118           -36.3%       8989        slabinfo.proc_inode_cache.active_objs
    293.50           -36.4%     186.67        slabinfo.proc_inode_cache.active_slabs
     14118           -36.3%       8989        slabinfo.proc_inode_cache.num_objs
    293.50           -36.4%     186.67        slabinfo.proc_inode_cache.num_slabs
     22359           +19.9%      26804        slabinfo.radix_tree_node.active_objs
     22359           +19.9%      26804        slabinfo.radix_tree_node.num_objs
     32514 ±  3%    +877.6%     317847 ±  3%  proc-vmstat.nr_active_anon
     65254            +8.2%      70573        proc-vmstat.nr_anon_pages
    333544           +85.5%     618811        proc-vmstat.nr_file_pages
     70493            +6.3%      74935        proc-vmstat.nr_inactive_anon
     10402            -7.8%       9585        proc-vmstat.nr_mapped
      1551            +7.4%       1665        proc-vmstat.nr_page_table_pages
     37874 ±  2%    +751.1%     322340 ±  3%  proc-vmstat.nr_shmem
     24876            -2.6%      24238        proc-vmstat.nr_slab_reclaimable
     48365            +1.5%      49089        proc-vmstat.nr_slab_unreclaimable
     32514 ±  3%    +877.6%     317847 ±  3%  proc-vmstat.nr_zone_active_anon
     70493            +6.3%      74935        proc-vmstat.nr_zone_inactive_anon
      7610 ± 40%     -41.4%       4459 ± 53%  proc-vmstat.numa_hint_faults
      4133 ±  8%     -48.5%       2129 ±  9%  proc-vmstat.numa_hint_faults_local
    912421           +80.1%    1643260 ±  2%  proc-vmstat.numa_hit
    818742           +89.3%    1549535 ±  2%  proc-vmstat.numa_local
     85725 ± 18%     -59.3%      34864 ± 30%  proc-vmstat.numa_pte_updates
     49420 ±  2%     -13.4%      42800 ±  3%  proc-vmstat.pgactivate
    969571           +91.4%    1856181 ±  4%  proc-vmstat.pgalloc_normal
    941570            -7.5%     871253        proc-vmstat.pgfault
    898227           +14.9%    1031957 ±  6%  proc-vmstat.pgfree
     19059 ± 33%    +287.5%      73861 ±128%  softirqs.CPU100.RCU
     25846 ± 36%     -65.1%       9030 ± 79%  softirqs.CPU16.SCHED
     25015 ± 30%     +51.9%      38003 ± 17%  softirqs.CPU19.RCU
     22651 ± 15%     +36.8%      30997 ± 15%  softirqs.CPU2.RCU
     23715 ± 22%     +69.0%      40085 ±  9%  softirqs.CPU24.RCU
     24337 ± 28%     +74.8%      42545 ± 47%  softirqs.CPU31.RCU
     20850 ± 19%     +66.3%      34668 ± 20%  softirqs.CPU32.RCU
     22721 ± 21%     +57.4%      35756 ± 17%  softirqs.CPU40.RCU
     17821 ± 27%     +87.9%      33480 ± 12%  softirqs.CPU46.RCU
     21298 ± 22%     +49.7%      31874 ± 15%  softirqs.CPU50.RCU
     18539 ± 21%     +58.8%      29435 ± 21%  softirqs.CPU55.RCU
     19198 ± 20%     +48.3%      28475 ± 19%  softirqs.CPU57.RCU
     17888 ± 25%     +61.7%      28930 ± 25%  softirqs.CPU60.RCU
     19030 ± 25%     +66.1%      31604 ± 24%  softirqs.CPU61.RCU
     18768 ± 23%     +60.9%      30204 ± 24%  softirqs.CPU63.RCU
     20254 ± 30%    +431.3%     107621 ±162%  softirqs.CPU64.RCU
     19895 ± 20%     +47.9%      29429 ± 20%  softirqs.CPU65.RCU
     18800 ± 23%     +65.0%      31020 ± 21%  softirqs.CPU66.RCU
     19145 ± 22%     +63.5%      31294 ± 25%  softirqs.CPU74.RCU
     18727 ± 25%    +367.0%      87458 ±151%  softirqs.CPU77.RCU
     22560 ± 22%     +51.6%      34208 ± 14%  softirqs.CPU81.RCU
     17142 ± 25%     +65.2%      28315 ± 23%  softirqs.CPU85.RCU
     18137 ± 23%     +58.5%      28747 ± 28%  softirqs.CPU89.RCU
     17197 ± 25%     +83.1%      31484 ± 34%  softirqs.CPU91.RCU
     18710 ± 22%     +61.9%      30283 ± 19%  softirqs.CPU94.RCU
     18735 ± 20%     +33.9%      25080 ± 21%  softirqs.CPU95.RCU
     17769 ± 51%     -60.3%       7047 ± 44%  softirqs.NET_RX
   2222678 ± 29%     +80.1%    4003205 ± 12%  softirqs.RCU
     35500 ± 12%     +44.8%      51399 ±  3%  softirqs.TIMER
     67.45 ±  9%     -67.4        0.00        perf-profile.calltrace.cycles-pp.brk
     48.31 ±  9%     -48.3        0.00        perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.brk
     32.16 ± 20%     -32.2        0.00        perf-profile.calltrace.cycles-pp.secondary_startup_64_no_verify
     32.09 ± 20%     -32.1        0.00        perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry
     31.51 ± 20%     -31.5        0.00        perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
     31.51 ± 20%     -31.5        0.00        perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
     31.51 ± 20%     -31.5        0.00        perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64_no_verify
     31.51 ± 20%     -31.5        0.00        perf-profile.calltrace.cycles-pp.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
     31.51 ± 20%     -31.5        0.00        perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary
     31.32 ±  9%     -31.3        0.00        perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.brk
     30.75 ±  9%     -30.8        0.00        perf-profile.calltrace.cycles-pp.__x64_sys_brk.do_syscall_64.entry_SYSCALL_64_after_hwframe.brk
     16.21 ±  9%     -16.2        0.00        perf-profile.calltrace.cycles-pp.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe.brk
     15.04 ±  9%     -15.0        0.00        perf-profile.calltrace.cycles-pp.do_brk_flags.__x64_sys_brk.do_syscall_64.entry_SYSCALL_64_after_hwframe.brk
     11.50 ±  9%     -11.5        0.00        perf-profile.calltrace.cycles-pp.__do_munmap.__x64_sys_brk.do_syscall_64.entry_SYSCALL_64_after_hwframe.brk
      8.87 ±  9%      -8.9        0.00        perf-profile.calltrace.cycles-pp.__entry_text_start.brk
      7.98 ±  9%      -8.0        0.00        perf-profile.calltrace.cycles-pp.syscall_return_via_sysret.brk
      5.55 ±  9%      -5.6        0.00        perf-profile.calltrace.cycles-pp.perf_event_mmap.do_brk_flags.__x64_sys_brk.do_syscall_64.entry_SYSCALL_64_after_hwframe
      5.21 ±  9%      -5.2        0.00        perf-profile.calltrace.cycles-pp.unmap_region.__do_munmap.__x64_sys_brk.do_syscall_64.entry_SYSCALL_64_after_hwframe
     67.49 ±  9%     -67.5        0.00        perf-profile.children.cycles-pp.brk
     48.47 ±  9%     -48.5        0.00        perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
     32.16 ± 20%     -32.2        0.00        perf-profile.children.cycles-pp.secondary_startup_64_no_verify
     32.16 ± 20%     -32.2        0.00        perf-profile.children.cycles-pp.cpu_startup_entry
     32.16 ± 20%     -32.2        0.00        perf-profile.children.cycles-pp.do_idle
     32.16 ± 20%     -32.2        0.00        perf-profile.children.cycles-pp.cpuidle_enter
     32.16 ± 20%     -32.2        0.00        perf-profile.children.cycles-pp.cpuidle_enter_state
     32.16 ± 20%     -32.2        0.00        perf-profile.children.cycles-pp.intel_idle
     31.51 ± 20%     -31.5        0.00        perf-profile.children.cycles-pp.start_secondary
     31.43 ±  9%     -31.4        0.00        perf-profile.children.cycles-pp.do_syscall_64
     30.84 ±  9%     -30.8        0.00        perf-profile.children.cycles-pp.__x64_sys_brk
     16.29 ±  9%     -16.3        0.00        perf-profile.children.cycles-pp.syscall_exit_to_user_mode
     15.10 ±  9%     -15.1        0.00        perf-profile.children.cycles-pp.do_brk_flags
     11.55 ±  9%     -11.6        0.00        perf-profile.children.cycles-pp.__do_munmap
      9.08 ± 10%      -9.1        0.00        perf-profile.children.cycles-pp.syscall_return_via_sysret
      8.47 ±  9%      -8.5        0.00        perf-profile.children.cycles-pp.__entry_text_start
      5.63 ±  9%      -5.6        0.00        perf-profile.children.cycles-pp.perf_event_mmap
      5.29 ±  9%      -5.3        0.00        perf-profile.children.cycles-pp.unmap_region
     32.16 ± 20%     -32.2        0.00        perf-profile.self.cycles-pp.intel_idle
     15.96 ±  9%     -16.0        0.00        perf-profile.self.cycles-pp.syscall_exit_to_user_mode
      9.05 ±  9%      -9.0        0.00        perf-profile.self.cycles-pp.syscall_return_via_sysret
      7.34 ±  9%      -7.3        0.00        perf-profile.self.cycles-pp.__entry_text_start
      0.06 ± 23%    +849.3%       0.52 ± 27%  perf-stat.i.MPKI
 2.817e+10            -7.5%  2.605e+10        perf-stat.i.branch-instructions
      0.70            +0.1        0.79        perf-stat.i.branch-miss-rate%
 1.965e+08            +4.2%  2.047e+08        perf-stat.i.branch-misses
   1348212 ± 93%    +927.5%   13853011 ± 20%  perf-stat.i.cache-misses
   6337510 ± 28%    +909.9%   64003716 ± 27%  perf-stat.i.cache-references
      1541 ±  5%     +31.9%       2033 ±  4%  perf-stat.i.context-switches
      1.08           +12.9%       1.22        perf-stat.i.cpi
 1.448e+11            +4.4%  1.512e+11        perf-stat.i.cpu-cycles
    132.05            -4.9%     125.55 ±  4%  perf-stat.i.cpu-migrations
    224930 ± 19%     -94.6%      12223 ± 28%  perf-stat.i.cycles-between-cache-misses
      0.17            -0.0        0.16        perf-stat.i.dTLB-load-miss-rate%
  66339784           -14.7%   56566077        perf-stat.i.dTLB-load-misses
 3.941e+10            -8.0%  3.624e+10        perf-stat.i.dTLB-loads
     52761           -11.0%      46938        perf-stat.i.dTLB-store-misses
 2.163e+10            -7.8%  1.994e+10        perf-stat.i.dTLB-stores
  68108407           -15.1%   57828873 ±  2%  perf-stat.i.iTLB-load-misses
  1.34e+11            -7.6%  1.239e+11        perf-stat.i.instructions
      1997            +8.9%       2174 ±  2%  perf-stat.i.instructions-per-iTLB-miss
      0.93           -11.4%       0.82        perf-stat.i.ipc
      0.72           -94.5%       0.04 ± 33%  perf-stat.i.major-faults
      1.39            +4.4%       1.45        perf-stat.i.metric.GHz
    857.73            -7.7%     791.38        perf-stat.i.metric.M/sec
      3021            -7.7%       2788        perf-stat.i.minor-faults
     89.68            -3.9       85.77        perf-stat.i.node-load-miss-rate%
    315756 ±104%    +546.6%    2041712 ± 20%  perf-stat.i.node-load-misses
     55314 ±104%    +530.4%     348680 ± 25%  perf-stat.i.node-loads
     34490 ± 45%   +2319.4%     834464 ± 31%  perf-stat.i.node-store-misses
      8436 ±  5%    +145.9%      20748 ±  6%  perf-stat.i.node-stores
      3021            -7.7%       2788        perf-stat.i.page-faults
      0.05 ± 28%    +987.0%       0.52 ± 27%  perf-stat.overall.MPKI
      0.70            +0.1        0.79        perf-stat.overall.branch-miss-rate%
      1.08           +13.0%       1.22        perf-stat.overall.cpi
    166572 ± 41%     -93.1%      11460 ± 23%  perf-stat.overall.cycles-between-cache-misses
      0.17            -0.0        0.16        perf-stat.overall.dTLB-load-miss-rate%
      0.00            -0.0        0.00        perf-stat.overall.dTLB-store-miss-rate%
      1968            +8.9%       2143        perf-stat.overall.instructions-per-iTLB-miss
      0.93           -11.5%       0.82        perf-stat.overall.ipc
     78.34 ±  6%     +18.6       96.97 ±  2%  perf-stat.overall.node-store-miss-rate%
    606198            +9.2%     661776        perf-stat.overall.path-length
 2.807e+10            -7.5%  2.597e+10        perf-stat.ps.branch-instructions
 1.959e+08            +4.1%   2.04e+08        perf-stat.ps.branch-misses
   1346058 ± 93%    +925.6%   13805032 ± 20%  perf-stat.ps.cache-misses
   6345091 ± 28%    +905.4%   63791617 ± 27%  perf-stat.ps.cache-references
      1536 ±  5%     +31.9%       2026 ±  4%  perf-stat.ps.context-switches
 1.443e+11            +4.4%  1.507e+11        perf-stat.ps.cpu-cycles
    132.04            -5.2%     125.17 ±  4%  perf-stat.ps.cpu-migrations
  66096851           -14.7%   56375461        perf-stat.ps.dTLB-load-misses
 3.927e+10            -8.0%  3.612e+10        perf-stat.ps.dTLB-loads
     52611           -11.1%      46790        perf-stat.ps.dTLB-store-misses
 2.155e+10            -7.8%  1.987e+10        perf-stat.ps.dTLB-stores
  67859804           -15.1%   57635424 ±  2%  perf-stat.ps.iTLB-load-misses
 1.336e+11            -7.5%  1.235e+11        perf-stat.ps.instructions
      0.72           -94.4%       0.04 ± 33%  perf-stat.ps.major-faults
      3011            -7.7%       2779        perf-stat.ps.minor-faults
    314425 ±104%    +547.1%    2034667 ± 20%  perf-stat.ps.node-load-misses
     55599 ±103%    +525.0%     347495 ± 25%  perf-stat.ps.node-loads
     34364 ± 45%   +2319.6%     831507 ± 31%  perf-stat.ps.node-store-misses
      8502 ±  5%    +143.5%      20703 ±  6%  perf-stat.ps.node-stores
      3012            -7.7%       2779        perf-stat.ps.page-faults
 4.033e+13            -7.5%  3.731e+13        perf-stat.total.instructions
      0.01 ±  9%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown]
      0.01 ± 11%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.devkmsg_read.vfs_read.ksys_read.do_syscall_64
      0.01 ±  6%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.do_nanosleep.hrtimer_nanosleep.__x64_sys_nanosleep.do_syscall_64
      0.01 ± 13%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.do_syslog.part.0.kmsg_read.vfs_read
      0.01 ±  5%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
      0.01 ±  3%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64
      0.01 ± 20%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.futex_wait_queue_me.futex_wait.do_futex.__x64_sys_futex
      0.00          -100.0%       0.00        perf-sched.sch_delay.avg.ms.pipe_read.new_sync_read.vfs_read.ksys_read
      0.01 ±  7%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.rcu_gp_kthread.kthread.ret_from_fork
      0.01 ± 15%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait.__x64_sys_epoll_wait
      0.01 ± 21%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_select
      0.00 ± 28%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_sys_poll
      0.01          -100.0%       0.00        perf-sched.sch_delay.avg.ms.schedule_timeout.kcompactd.kthread.ret_from_fork
      0.01          -100.0%       0.00        perf-sched.sch_delay.avg.ms.schedule_timeout.rcu_gp_kthread.kthread.ret_from_fork
      0.01 ± 11%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.schedule_timeout.wait_for_completion.__flush_work.lru_add_drain_all
      0.01          -100.0%       0.00        perf-sched.sch_delay.avg.ms.smpboot_thread_fn.kthread.ret_from_fork
      0.01 ±  6%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.wait_for_partner.fifo_open.do_dentry_open.do_open.isra
      0.02 ± 16%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.worker_thread.kthread.ret_from_fork
      0.01 ± 53%    -100.0%       0.00        perf-sched.sch_delay.max.ms.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown]
      0.01 ± 23%    -100.0%       0.00        perf-sched.sch_delay.max.ms.devkmsg_read.vfs_read.ksys_read.do_syscall_64
      0.01 ± 14%    -100.0%       0.00        perf-sched.sch_delay.max.ms.do_nanosleep.hrtimer_nanosleep.__x64_sys_nanosleep.do_syscall_64
      0.02 ± 27%    -100.0%       0.00        perf-sched.sch_delay.max.ms.do_syslog.part.0.kmsg_read.vfs_read
      0.03 ± 64%    -100.0%       0.00        perf-sched.sch_delay.max.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
      0.02 ± 12%    -100.0%       0.00        perf-sched.sch_delay.max.ms.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64
      0.01 ± 19%    -100.0%       0.00        perf-sched.sch_delay.max.ms.exit_to_user_mode_prepare.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe.[unknown]
      0.01 ± 18%    -100.0%       0.00        perf-sched.sch_delay.max.ms.futex_wait_queue_me.futex_wait.do_futex.__x64_sys_futex
      0.03 ± 10%    -100.0%       0.00        perf-sched.sch_delay.max.ms.pipe_read.new_sync_read.vfs_read.ksys_read
      0.02 ± 15%    -100.0%       0.00        perf-sched.sch_delay.max.ms.preempt_schedule_common.__cond_resched.stop_one_cpu.affine_move_task.__set_cpus_allowed_ptr
      0.01 ± 19%    -100.0%       0.00        perf-sched.sch_delay.max.ms.rcu_gp_kthread.kthread.ret_from_fork
      0.03 ± 78%    -100.0%       0.00        perf-sched.sch_delay.max.ms.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait.__x64_sys_epoll_wait
      0.01 ± 29%    -100.0%       0.00        perf-sched.sch_delay.max.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_select
      0.01 ± 51%    -100.0%       0.00        perf-sched.sch_delay.max.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_sys_poll
      0.01 ± 42%    -100.0%       0.00        perf-sched.sch_delay.max.ms.schedule_timeout.kcompactd.kthread.ret_from_fork
      0.02 ± 26%    -100.0%       0.00        perf-sched.sch_delay.max.ms.schedule_timeout.rcu_gp_kthread.kthread.ret_from_fork
      0.01 ± 11%    -100.0%       0.00        perf-sched.sch_delay.max.ms.schedule_timeout.wait_for_completion.__flush_work.lru_add_drain_all
      0.02 ± 22%    -100.0%       0.00        perf-sched.sch_delay.max.ms.smpboot_thread_fn.kthread.ret_from_fork
      0.02 ± 66%    -100.0%       0.00        perf-sched.sch_delay.max.ms.wait_for_partner.fifo_open.do_dentry_open.do_open.isra
      5.62 ±  2%    -100.0%       0.00        perf-sched.sch_delay.max.ms.worker_thread.kthread.ret_from_fork
      0.01          -100.0%       0.00        perf-sched.total_sch_delay.average.ms
      5.62 ±  2%    -100.0%       0.00        perf-sched.total_sch_delay.max.ms
    195.80 ±  6%    -100.0%       0.00        perf-sched.total_wait_and_delay.average.ms
      8759 ±  6%    -100.0%       0.00        perf-sched.total_wait_and_delay.count.ms
      8233 ±  6%    -100.0%       0.00        perf-sched.total_wait_and_delay.max.ms
    195.80 ±  6%    -100.0%       0.00        perf-sched.total_wait_time.average.ms
      8233 ±  6%    -100.0%       0.00        perf-sched.total_wait_time.max.ms
    899.53          -100.0%       0.00        perf-sched.wait_and_delay.avg.ms.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown]
      1969          -100.0%       0.00        perf-sched.wait_and_delay.avg.ms.devkmsg_read.vfs_read.ksys_read.do_syscall_64
    809.06 ±  4%    -100.0%       0.00        perf-sched.wait_and_delay.avg.ms.do_nanosleep.hrtimer_nanosleep.__x64_sys_nanosleep.do_syscall_64
      1969          -100.0%       0.00        perf-sched.wait_and_delay.avg.ms.do_syslog.part.0.kmsg_read.vfs_read
    269.47 ±  4%    -100.0%       0.00        perf-sched.wait_and_delay.avg.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
      1.43          -100.0%       0.00        perf-sched.wait_and_delay.avg.ms.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64
      0.04 ± 18%    -100.0%       0.00        perf-sched.wait_and_delay.avg.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt.[unknown]
      0.03 ±  5%    -100.0%       0.00        perf-sched.wait_and_delay.avg.ms.exit_to_user_mode_prepare.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe.[unknown]
     64.05          -100.0%       0.00        perf-sched.wait_and_delay.avg.ms.pipe_read.new_sync_read.vfs_read.ksys_read
      0.04 ± 30%    -100.0%       0.00        perf-sched.wait_and_delay.avg.ms.preempt_schedule_common.__cond_resched.down_write_killable.__x64_sys_brk.do_syscall_64
      0.04 ± 16%    -100.0%       0.00        perf-sched.wait_and_delay.avg.ms.preempt_schedule_common.__cond_resched.kmem_cache_alloc.vm_area_alloc.do_brk_flags
      7.87 ±  6%    -100.0%       0.00        perf-sched.wait_and_delay.avg.ms.preempt_schedule_common.__cond_resched.stop_one_cpu.affine_move_task.__set_cpus_allowed_ptr
      0.03 ±  6%    -100.0%       0.00        perf-sched.wait_and_delay.avg.ms.preempt_schedule_common.__cond_resched.unmap_vmas.unmap_region.__do_munmap
      2.56 ±  8%    -100.0%       0.00        perf-sched.wait_and_delay.avg.ms.rcu_gp_kthread.kthread.ret_from_fork
    595.03 ± 12%    -100.0%       0.00        perf-sched.wait_and_delay.avg.ms.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait.__x64_sys_epoll_wait
    326.53 ± 27%    -100.0%       0.00        perf-sched.wait_and_delay.avg.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_sys_poll
    478.79          -100.0%       0.00        perf-sched.wait_and_delay.avg.ms.schedule_timeout.kcompactd.kthread.ret_from_fork
      8.32 ± 25%    -100.0%       0.00        perf-sched.wait_and_delay.avg.ms.schedule_timeout.rcu_gp_kthread.kthread.ret_from_fork
    676.61 ±  2%    -100.0%       0.00        perf-sched.wait_and_delay.avg.ms.smpboot_thread_fn.kthread.ret_from_fork
      0.01 ±  6%    -100.0%       0.00        perf-sched.wait_and_delay.avg.ms.wait_for_partner.fifo_open.do_dentry_open.do_open.isra
    623.29 ±  4%    -100.0%       0.00        perf-sched.wait_and_delay.avg.ms.worker_thread.kthread.ret_from_fork
     10.00          -100.0%       0.00        perf-sched.wait_and_delay.count.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown]
      4.00          -100.0%       0.00        perf-sched.wait_and_delay.count.devkmsg_read.vfs_read.ksys_read.do_syscall_64
     22.33 ±  4%    -100.0%       0.00        perf-sched.wait_and_delay.count.do_nanosleep.hrtimer_nanosleep.__x64_sys_nanosleep.do_syscall_64
      4.00          -100.0%       0.00        perf-sched.wait_and_delay.count.do_syslog.part.0.kmsg_read.vfs_read
    247.00          -100.0%       0.00        perf-sched.wait_and_delay.count.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
    246.00          -100.0%       0.00        perf-sched.wait_and_delay.count.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64
    118.83 ± 38%    -100.0%       0.00        perf-sched.wait_and_delay.count.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt.[unknown]
    342.83 ± 14%    -100.0%       0.00        perf-sched.wait_and_delay.count.exit_to_user_mode_prepare.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe.[unknown]
      2518          -100.0%       0.00        perf-sched.wait_and_delay.count.pipe_read.new_sync_read.vfs_read.ksys_read
    115.33 ±  6%    -100.0%       0.00        perf-sched.wait_and_delay.count.preempt_schedule_common.__cond_resched.down_write_killable.__x64_sys_brk.do_syscall_64
     61.83 ±  7%    -100.0%       0.00        perf-sched.wait_and_delay.count.preempt_schedule_common.__cond_resched.kmem_cache_alloc.vm_area_alloc.do_brk_flags
      1248          -100.0%       0.00        perf-sched.wait_and_delay.count.preempt_schedule_common.__cond_resched.stop_one_cpu.affine_move_task.__set_cpus_allowed_ptr
     54.17 ± 12%    -100.0%       0.00        perf-sched.wait_and_delay.count.preempt_schedule_common.__cond_resched.unmap_vmas.unmap_region.__do_munmap
    143.67 ± 55%    -100.0%       0.00        perf-sched.wait_and_delay.count.rcu_gp_kthread.kthread.ret_from_fork
     23.67 ± 45%    -100.0%       0.00        perf-sched.wait_and_delay.count.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait.__x64_sys_epoll_wait
     83.83 ± 25%    -100.0%       0.00        perf-sched.wait_and_delay.count.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_sys_poll
     40.00          -100.0%       0.00        perf-sched.wait_and_delay.count.schedule_timeout.kcompactd.kthread.ret_from_fork
      1272 ± 37%    -100.0%       0.00        perf-sched.wait_and_delay.count.schedule_timeout.rcu_gp_kthread.kthread.ret_from_fork
      1464          -100.0%       0.00        perf-sched.wait_and_delay.count.smpboot_thread_fn.kthread.ret_from_fork
     72.00          -100.0%       0.00        perf-sched.wait_and_delay.count.wait_for_partner.fifo_open.do_dentry_open.do_open.isra
    575.00 ±  8%    -100.0%       0.00        perf-sched.wait_and_delay.count.worker_thread.kthread.ret_from_fork
    999.50          -100.0%       0.00        perf-sched.wait_and_delay.max.ms.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown]
      7815          -100.0%       0.00        perf-sched.wait_and_delay.max.ms.devkmsg_read.vfs_read.ksys_read.do_syscall_64
      1000          -100.0%       0.00        perf-sched.wait_and_delay.max.ms.do_nanosleep.hrtimer_nanosleep.__x64_sys_nanosleep.do_syscall_64
      7815          -100.0%       0.00        perf-sched.wait_and_delay.max.ms.do_syslog.part.0.kmsg_read.vfs_read
      3095 ± 56%    -100.0%       0.00        perf-sched.wait_and_delay.max.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
     33.25 ±  2%    -100.0%       0.00        perf-sched.wait_and_delay.max.ms.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64
      0.88 ±146%    -100.0%       0.00        perf-sched.wait_and_delay.max.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt.[unknown]
      0.19 ±106%    -100.0%       0.00        perf-sched.wait_and_delay.max.ms.exit_to_user_mode_prepare.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe.[unknown]
      7819          -100.0%       0.00        perf-sched.wait_and_delay.max.ms.pipe_read.new_sync_read.vfs_read.ksys_read
      0.78 ±149%    -100.0%       0.00        perf-sched.wait_and_delay.max.ms.preempt_schedule_common.__cond_resched.down_write_killable.__x64_sys_brk.do_syscall_64
      0.25 ±144%    -100.0%       0.00        perf-sched.wait_and_delay.max.ms.preempt_schedule_common.__cond_resched.kmem_cache_alloc.vm_area_alloc.do_brk_flags
      1172 ± 32%    -100.0%       0.00        perf-sched.wait_and_delay.max.ms.preempt_schedule_common.__cond_resched.stop_one_cpu.affine_move_task.__set_cpus_allowed_ptr
      0.06 ± 27%    -100.0%       0.00        perf-sched.wait_and_delay.max.ms.preempt_schedule_common.__cond_resched.unmap_vmas.unmap_region.__do_munmap
      5.00          -100.0%       0.00        perf-sched.wait_and_delay.max.ms.rcu_gp_kthread.kthread.ret_from_fork
      7407          -100.0%       0.00        perf-sched.wait_and_delay.max.ms.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait.__x64_sys_epoll_wait
      5250 ± 36%    -100.0%       0.00        perf-sched.wait_and_delay.max.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_sys_poll
    504.84          -100.0%       0.00        perf-sched.wait_and_delay.max.ms.schedule_timeout.kcompactd.kthread.ret_from_fork
    243.33 ± 50%    -100.0%       0.00        perf-sched.wait_and_delay.max.ms.schedule_timeout.rcu_gp_kthread.kthread.ret_from_fork
      2729 ± 47%    -100.0%       0.00        perf-sched.wait_and_delay.max.ms.smpboot_thread_fn.kthread.ret_from_fork
      0.03 ± 63%    -100.0%       0.00        perf-sched.wait_and_delay.max.ms.wait_for_partner.fifo_open.do_dentry_open.do_open.isra
      7699 ± 14%    -100.0%       0.00        perf-sched.wait_and_delay.max.ms.worker_thread.kthread.ret_from_fork
    899.53          -100.0%       0.00        perf-sched.wait_time.avg.ms.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown]
      1969          -100.0%       0.00        perf-sched.wait_time.avg.ms.devkmsg_read.vfs_read.ksys_read.do_syscall_64
    809.06 ±  4%    -100.0%       0.00        perf-sched.wait_time.avg.ms.do_nanosleep.hrtimer_nanosleep.__x64_sys_nanosleep.do_syscall_64
      1969          -100.0%       0.00        perf-sched.wait_time.avg.ms.do_syslog.part.0.kmsg_read.vfs_read
    269.46 ±  4%    -100.0%       0.00        perf-sched.wait_time.avg.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
      1.42          -100.0%       0.00        perf-sched.wait_time.avg.ms.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64
      0.04 ± 18%    -100.0%       0.00        perf-sched.wait_time.avg.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt.[unknown]
      0.03 ± 29%    -100.0%       0.00        perf-sched.wait_time.avg.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_reschedule_ipi.[unknown]
      0.03 ±  4%    -100.0%       0.00        perf-sched.wait_time.avg.ms.exit_to_user_mode_prepare.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe.[unknown]
     28.97 ± 27%    -100.0%       0.00        perf-sched.wait_time.avg.ms.futex_wait_queue_me.futex_wait.do_futex.__x64_sys_futex
     64.05          -100.0%       0.00        perf-sched.wait_time.avg.ms.pipe_read.new_sync_read.vfs_read.ksys_read
      0.04 ± 30%    -100.0%       0.00        perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.down_write_killable.__x64_sys_brk.do_syscall_64
      0.04 ± 16%    -100.0%       0.00        perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.kmem_cache_alloc.vm_area_alloc.do_brk_flags
      0.03 ±  8%    -100.0%       0.00        perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.remove_vma.__do_munmap.__x64_sys_brk
      7.87 ±  6%    -100.0%       0.00        perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.stop_one_cpu.affine_move_task.__set_cpus_allowed_ptr
      0.04 ±  6%    -100.0%       0.00        perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.unmap_page_range.unmap_vmas.unmap_region
      0.03 ±  6%    -100.0%       0.00        perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.unmap_vmas.unmap_region.__do_munmap
      2.56 ±  8%    -100.0%       0.00        perf-sched.wait_time.avg.ms.rcu_gp_kthread.kthread.ret_from_fork
    595.02 ± 12%    -100.0%       0.00        perf-sched.wait_time.avg.ms.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait.__x64_sys_epoll_wait
    326.53 ± 27%    -100.0%       0.00        perf-sched.wait_time.avg.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_sys_poll
    478.79          -100.0%       0.00        perf-sched.wait_time.avg.ms.schedule_timeout.kcompactd.kthread.ret_from_fork
      0.01 ± 63%    -100.0%       0.00        perf-sched.wait_time.avg.ms.schedule_timeout.khugepaged.kthread.ret_from_fork
      8.31 ± 25%    -100.0%       0.00        perf-sched.wait_time.avg.ms.schedule_timeout.rcu_gp_kthread.kthread.ret_from_fork
    676.60 ±  2%    -100.0%       0.00        perf-sched.wait_time.avg.ms.smpboot_thread_fn.kthread.ret_from_fork
    623.27 ±  4%    -100.0%       0.00        perf-sched.wait_time.avg.ms.worker_thread.kthread.ret_from_fork
    999.50          -100.0%       0.00        perf-sched.wait_time.max.ms.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown]
      7815          -100.0%       0.00        perf-sched.wait_time.max.ms.devkmsg_read.vfs_read.ksys_read.do_syscall_64
      1000          -100.0%       0.00        perf-sched.wait_time.max.ms.do_nanosleep.hrtimer_nanosleep.__x64_sys_nanosleep.do_syscall_64
      7815          -100.0%       0.00        perf-sched.wait_time.max.ms.do_syslog.part.0.kmsg_read.vfs_read
      3095 ± 56%    -100.0%       0.00        perf-sched.wait_time.max.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
     33.24 ±  2%    -100.0%       0.00        perf-sched.wait_time.max.ms.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64
      0.88 ±146%    -100.0%       0.00        perf-sched.wait_time.max.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt.[unknown]
      0.10 ±106%    -100.0%       0.00        perf-sched.wait_time.max.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_reschedule_ipi.[unknown]
      0.19 ±106%    -100.0%       0.00        perf-sched.wait_time.max.ms.exit_to_user_mode_prepare.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe.[unknown]
     57.94 ± 27%    -100.0%       0.00        perf-sched.wait_time.max.ms.futex_wait_queue_me.futex_wait.do_futex.__x64_sys_futex
      7819          -100.0%       0.00        perf-sched.wait_time.max.ms.pipe_read.new_sync_read.vfs_read.ksys_read
      0.78 ±149%    -100.0%       0.00        perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.down_write_killable.__x64_sys_brk.do_syscall_64
      0.25 ±144%    -100.0%       0.00        perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.kmem_cache_alloc.vm_area_alloc.do_brk_flags
      0.05          -100.0%       0.00        perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.remove_vma.__do_munmap.__x64_sys_brk
      1172 ± 32%    -100.0%       0.00        perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.stop_one_cpu.affine_move_task.__set_cpus_allowed_ptr
      0.09 ± 37%    -100.0%       0.00        perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.unmap_page_range.unmap_vmas.unmap_region
      0.06 ± 27%    -100.0%       0.00        perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.unmap_vmas.unmap_region.__do_munmap
      4.99          -100.0%       0.00        perf-sched.wait_time.max.ms.rcu_gp_kthread.kthread.ret_from_fork
      7407          -100.0%       0.00        perf-sched.wait_time.max.ms.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait.__x64_sys_epoll_wait
      5250 ± 36%    -100.0%       0.00        perf-sched.wait_time.max.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_sys_poll
    504.83          -100.0%       0.00        perf-sched.wait_time.max.ms.schedule_timeout.kcompactd.kthread.ret_from_fork
      0.01 ± 63%    -100.0%       0.00        perf-sched.wait_time.max.ms.schedule_timeout.khugepaged.kthread.ret_from_fork
    243.32 ± 50%    -100.0%       0.00        perf-sched.wait_time.max.ms.schedule_timeout.rcu_gp_kthread.kthread.ret_from_fork
      2729 ± 47%    -100.0%       0.00        perf-sched.wait_time.max.ms.smpboot_thread_fn.kthread.ret_from_fork
      7699 ± 14%    -100.0%       0.00        perf-sched.wait_time.max.ms.worker_thread.kthread.ret_from_fork
     76758 ±  2%      -8.6%      70119 ±  2%  interrupts.CAL:Function_call_interrupts
      4021 ± 30%    -100.0%       0.00        interrupts.CPU0.NMI:Non-maskable_interrupts
      4021 ± 30%    -100.0%       0.00        interrupts.CPU0.PMI:Performance_monitoring_interrupts
      5976 ± 37%    -100.0%       0.00        interrupts.CPU1.NMI:Non-maskable_interrupts
      5976 ± 37%    -100.0%       0.00        interrupts.CPU1.PMI:Performance_monitoring_interrupts
      5375 ± 45%    -100.0%       0.00        interrupts.CPU10.NMI:Non-maskable_interrupts
      5375 ± 45%    -100.0%       0.00        interrupts.CPU10.PMI:Performance_monitoring_interrupts
      4174 ± 47%    -100.0%       0.00        interrupts.CPU100.NMI:Non-maskable_interrupts
      4174 ± 47%    -100.0%       0.00        interrupts.CPU100.PMI:Performance_monitoring_interrupts
      5327 ± 42%    -100.0%       0.00        interrupts.CPU101.NMI:Non-maskable_interrupts
      5327 ± 42%    -100.0%       0.00        interrupts.CPU101.PMI:Performance_monitoring_interrupts
      4581 ± 38%    -100.0%       0.00        interrupts.CPU102.NMI:Non-maskable_interrupts
      4581 ± 38%    -100.0%       0.00        interrupts.CPU102.PMI:Performance_monitoring_interrupts
      5337 ± 26%    -100.0%       0.00        interrupts.CPU103.NMI:Non-maskable_interrupts
      5337 ± 26%    -100.0%       0.00        interrupts.CPU103.PMI:Performance_monitoring_interrupts
      3615 ± 45%    -100.0%       0.00        interrupts.CPU11.NMI:Non-maskable_interrupts
      3615 ± 45%    -100.0%       0.00        interrupts.CPU11.PMI:Performance_monitoring_interrupts
      4389 ± 57%    -100.0%       0.00        interrupts.CPU12.NMI:Non-maskable_interrupts
      4389 ± 57%    -100.0%       0.00        interrupts.CPU12.PMI:Performance_monitoring_interrupts
      5230 ± 44%    -100.0%       0.00        interrupts.CPU13.NMI:Non-maskable_interrupts
      5230 ± 44%    -100.0%       0.00        interrupts.CPU13.PMI:Performance_monitoring_interrupts
      4857 ± 37%    -100.0%       0.00        interrupts.CPU14.NMI:Non-maskable_interrupts
      4857 ± 37%    -100.0%       0.00        interrupts.CPU14.PMI:Performance_monitoring_interrupts
      4571 ± 51%    -100.0%       0.00        interrupts.CPU15.NMI:Non-maskable_interrupts
      4571 ± 51%    -100.0%       0.00        interrupts.CPU15.PMI:Performance_monitoring_interrupts
    769.50 ± 20%     -28.1%     553.00 ±  8%  interrupts.CPU16.CAL:Function_call_interrupts
      3950 ± 54%    -100.0%       0.00        interrupts.CPU16.NMI:Non-maskable_interrupts
      3950 ± 54%    -100.0%       0.00        interrupts.CPU16.PMI:Performance_monitoring_interrupts
      4485 ± 30%    -100.0%       0.00        interrupts.CPU17.NMI:Non-maskable_interrupts
      4485 ± 30%    -100.0%       0.00        interrupts.CPU17.PMI:Performance_monitoring_interrupts
      5139 ± 35%    -100.0%       0.00        interrupts.CPU18.NMI:Non-maskable_interrupts
      5139 ± 35%    -100.0%       0.00        interrupts.CPU18.PMI:Performance_monitoring_interrupts
      5525 ± 45%    -100.0%       0.00        interrupts.CPU19.NMI:Non-maskable_interrupts
      5525 ± 45%    -100.0%       0.00        interrupts.CPU19.PMI:Performance_monitoring_interrupts
      6327 ± 30%    -100.0%       0.00        interrupts.CPU2.NMI:Non-maskable_interrupts
      6327 ± 30%    -100.0%       0.00        interrupts.CPU2.PMI:Performance_monitoring_interrupts
    732.67 ± 24%     -21.5%     575.17 ± 12%  interrupts.CPU20.CAL:Function_call_interrupts
      3597 ± 54%    -100.0%       0.00        interrupts.CPU20.NMI:Non-maskable_interrupts
      3597 ± 54%    -100.0%       0.00        interrupts.CPU20.PMI:Performance_monitoring_interrupts
      4856 ± 42%    -100.0%       0.00        interrupts.CPU21.NMI:Non-maskable_interrupts
      4856 ± 42%    -100.0%       0.00        interrupts.CPU21.PMI:Performance_monitoring_interrupts
      6101 ± 38%    -100.0%       0.00        interrupts.CPU22.NMI:Non-maskable_interrupts
      6101 ± 38%    -100.0%       0.00        interrupts.CPU22.PMI:Performance_monitoring_interrupts
      4139 ± 57%    -100.0%       0.00        interrupts.CPU23.NMI:Non-maskable_interrupts
      4139 ± 57%    -100.0%       0.00        interrupts.CPU23.PMI:Performance_monitoring_interrupts
    647.00 ± 14%     -15.8%     544.67 ±  3%  interrupts.CPU24.CAL:Function_call_interrupts
      5617 ± 37%    -100.0%       0.00        interrupts.CPU24.NMI:Non-maskable_interrupts
      5617 ± 37%    -100.0%       0.00        interrupts.CPU24.PMI:Performance_monitoring_interrupts
      6282 ± 27%    -100.0%       0.00        interrupts.CPU25.NMI:Non-maskable_interrupts
      6282 ± 27%    -100.0%       0.00        interrupts.CPU25.PMI:Performance_monitoring_interrupts
      5144 ± 54%    -100.0%       0.00        interrupts.CPU26.NMI:Non-maskable_interrupts
      5144 ± 54%    -100.0%       0.00        interrupts.CPU26.PMI:Performance_monitoring_interrupts
      6256 ± 27%    -100.0%       0.00        interrupts.CPU27.NMI:Non-maskable_interrupts
      6256 ± 27%    -100.0%       0.00        interrupts.CPU27.PMI:Performance_monitoring_interrupts
      5805 ± 38%    -100.0%       0.00        interrupts.CPU28.NMI:Non-maskable_interrupts
      5805 ± 38%    -100.0%       0.00        interrupts.CPU28.PMI:Performance_monitoring_interrupts
      3944 ± 55%    -100.0%       0.00        interrupts.CPU29.NMI:Non-maskable_interrupts
      3944 ± 55%    -100.0%       0.00        interrupts.CPU29.PMI:Performance_monitoring_interrupts
      5923 ± 33%    -100.0%       0.00        interrupts.CPU3.NMI:Non-maskable_interrupts
      5923 ± 33%    -100.0%       0.00        interrupts.CPU3.PMI:Performance_monitoring_interrupts
      4233 ± 47%    -100.0%       0.00        interrupts.CPU30.NMI:Non-maskable_interrupts
      4233 ± 47%    -100.0%       0.00        interrupts.CPU30.PMI:Performance_monitoring_interrupts
      7332 ±  9%    -100.0%       0.00        interrupts.CPU31.NMI:Non-maskable_interrupts
      7332 ±  9%    -100.0%       0.00        interrupts.CPU31.PMI:Performance_monitoring_interrupts
      4973 ± 44%    -100.0%       0.00        interrupts.CPU32.NMI:Non-maskable_interrupts
      4973 ± 44%    -100.0%       0.00        interrupts.CPU32.PMI:Performance_monitoring_interrupts
      7439 ± 10%    -100.0%       0.00        interrupts.CPU33.NMI:Non-maskable_interrupts
      7439 ± 10%    -100.0%       0.00        interrupts.CPU33.PMI:Performance_monitoring_interrupts
      7900          -100.0%       0.00        interrupts.CPU34.NMI:Non-maskable_interrupts
      7900          -100.0%       0.00        interrupts.CPU34.PMI:Performance_monitoring_interrupts
      6148 ± 31%    -100.0%       0.00        interrupts.CPU35.NMI:Non-maskable_interrupts
      6148 ± 31%    -100.0%       0.00        interrupts.CPU35.PMI:Performance_monitoring_interrupts
      6697 ± 28%    -100.0%       0.00        interrupts.CPU36.NMI:Non-maskable_interrupts
      6697 ± 28%    -100.0%       0.00        interrupts.CPU36.PMI:Performance_monitoring_interrupts
      6696 ± 28%    -100.0%       0.00        interrupts.CPU37.NMI:Non-maskable_interrupts
      6696 ± 28%    -100.0%       0.00        interrupts.CPU37.PMI:Performance_monitoring_interrupts
      7901          -100.0%       0.00        interrupts.CPU38.NMI:Non-maskable_interrupts
      7901          -100.0%       0.00        interrupts.CPU38.PMI:Performance_monitoring_interrupts
      7061 ± 26%    -100.0%       0.00        interrupts.CPU39.NMI:Non-maskable_interrupts
      7061 ± 26%    -100.0%       0.00        interrupts.CPU39.PMI:Performance_monitoring_interrupts
      5894 ± 24%    -100.0%       0.00        interrupts.CPU4.NMI:Non-maskable_interrupts
      5894 ± 24%    -100.0%       0.00        interrupts.CPU4.PMI:Performance_monitoring_interrupts
      6487 ± 30%    -100.0%       0.00        interrupts.CPU40.NMI:Non-maskable_interrupts
      6487 ± 30%    -100.0%       0.00        interrupts.CPU40.PMI:Performance_monitoring_interrupts
      6538 ± 29%    -100.0%       0.00        interrupts.CPU41.NMI:Non-maskable_interrupts
      6538 ± 29%    -100.0%       0.00        interrupts.CPU41.PMI:Performance_monitoring_interrupts
      6221 ± 38%    -100.0%       0.00        interrupts.CPU42.NMI:Non-maskable_interrupts
      6221 ± 38%    -100.0%       0.00        interrupts.CPU42.PMI:Performance_monitoring_interrupts
      6657 ± 28%    -100.0%       0.00        interrupts.CPU43.NMI:Non-maskable_interrupts
      6657 ± 28%    -100.0%       0.00        interrupts.CPU43.PMI:Performance_monitoring_interrupts
      6069 ± 32%    -100.0%       0.00        interrupts.CPU44.NMI:Non-maskable_interrupts
      6069 ± 32%    -100.0%       0.00        interrupts.CPU44.PMI:Performance_monitoring_interrupts
      7011 ± 26%    -100.0%       0.00        interrupts.CPU45.NMI:Non-maskable_interrupts
      7011 ± 26%    -100.0%       0.00        interrupts.CPU45.PMI:Performance_monitoring_interrupts
      5375 ± 35%    -100.0%       0.00        interrupts.CPU46.NMI:Non-maskable_interrupts
      5375 ± 35%    -100.0%       0.00        interrupts.CPU46.PMI:Performance_monitoring_interrupts
      5413 ± 35%    -100.0%       0.00        interrupts.CPU47.NMI:Non-maskable_interrupts
      5413 ± 35%    -100.0%       0.00        interrupts.CPU47.PMI:Performance_monitoring_interrupts
      7061 ± 26%    -100.0%       0.00        interrupts.CPU48.NMI:Non-maskable_interrupts
      7061 ± 26%    -100.0%       0.00        interrupts.CPU48.PMI:Performance_monitoring_interrupts
      5134 ± 33%    -100.0%       0.00        interrupts.CPU49.NMI:Non-maskable_interrupts
      5134 ± 33%    -100.0%       0.00        interrupts.CPU49.PMI:Performance_monitoring_interrupts
      5876 ± 36%    -100.0%       0.00        interrupts.CPU5.NMI:Non-maskable_interrupts
      5876 ± 36%    -100.0%       0.00        interrupts.CPU5.PMI:Performance_monitoring_interrupts
      6389 ± 35%    -100.0%       0.00        interrupts.CPU50.NMI:Non-maskable_interrupts
      6389 ± 35%    -100.0%       0.00        interrupts.CPU50.PMI:Performance_monitoring_interrupts
      5871 ± 26%    -100.0%       0.00        interrupts.CPU51.NMI:Non-maskable_interrupts
      5871 ± 26%    -100.0%       0.00        interrupts.CPU51.PMI:Performance_monitoring_interrupts
      6226 ± 27%    -100.0%       0.00        interrupts.CPU52.NMI:Non-maskable_interrupts
      6226 ± 27%    -100.0%       0.00        interrupts.CPU52.PMI:Performance_monitoring_interrupts
      4904 ± 33%    -100.0%       0.00        interrupts.CPU53.NMI:Non-maskable_interrupts
      4904 ± 33%    -100.0%       0.00        interrupts.CPU53.PMI:Performance_monitoring_interrupts
      3705 ± 42%    -100.0%       0.00        interrupts.CPU54.NMI:Non-maskable_interrupts
      3705 ± 42%    -100.0%       0.00        interrupts.CPU54.PMI:Performance_monitoring_interrupts
      4316 ± 42%    -100.0%       0.00        interrupts.CPU55.NMI:Non-maskable_interrupts
      4316 ± 42%    -100.0%       0.00        interrupts.CPU55.PMI:Performance_monitoring_interrupts
      4496 ± 50%    -100.0%       0.00        interrupts.CPU56.NMI:Non-maskable_interrupts
      4496 ± 50%    -100.0%       0.00        interrupts.CPU56.PMI:Performance_monitoring_interrupts
      5172 ± 50%    -100.0%       0.00        interrupts.CPU57.NMI:Non-maskable_interrupts
      5172 ± 50%    -100.0%       0.00        interrupts.CPU57.PMI:Performance_monitoring_interrupts
      4153 ± 45%    -100.0%       1.00 ±223%  interrupts.CPU58.NMI:Non-maskable_interrupts
      4153 ± 45%    -100.0%       1.00 ±223%  interrupts.CPU58.PMI:Performance_monitoring_interrupts
      4949 ± 44%    -100.0%       0.00        interrupts.CPU59.NMI:Non-maskable_interrupts
      4949 ± 44%    -100.0%       0.00        interrupts.CPU59.PMI:Performance_monitoring_interrupts
      6086 ± 28%    -100.0%       0.00        interrupts.CPU6.NMI:Non-maskable_interrupts
      6086 ± 28%    -100.0%       0.00        interrupts.CPU6.PMI:Performance_monitoring_interrupts
      4156 ± 45%    -100.0%       0.00        interrupts.CPU60.NMI:Non-maskable_interrupts
      4156 ± 45%    -100.0%       0.00        interrupts.CPU60.PMI:Performance_monitoring_interrupts
      4486 ± 50%    -100.0%       0.00        interrupts.CPU61.NMI:Non-maskable_interrupts
      4486 ± 50%    -100.0%       0.00        interrupts.CPU61.PMI:Performance_monitoring_interrupts
      1459 ±121%     -62.1%     553.67 ± 11%  interrupts.CPU62.CAL:Function_call_interrupts
      4623 ± 40%    -100.0%       0.00        interrupts.CPU62.NMI:Non-maskable_interrupts
      4623 ± 40%    -100.0%       0.00        interrupts.CPU62.PMI:Performance_monitoring_interrupts
      5745 ± 32%    -100.0%       0.00        interrupts.CPU63.NMI:Non-maskable_interrupts
      5745 ± 32%    -100.0%       0.00        interrupts.CPU63.PMI:Performance_monitoring_interrupts
      5320 ± 35%    -100.0%       0.00        interrupts.CPU64.NMI:Non-maskable_interrupts
      5320 ± 35%    -100.0%       0.00        interrupts.CPU64.PMI:Performance_monitoring_interrupts
      4127 ± 36%    -100.0%       0.00        interrupts.CPU65.NMI:Non-maskable_interrupts
      4127 ± 36%    -100.0%       0.00        interrupts.CPU65.PMI:Performance_monitoring_interrupts
      4024 ± 51%    -100.0%       0.00        interrupts.CPU66.NMI:Non-maskable_interrupts
      4024 ± 51%    -100.0%       0.00        interrupts.CPU66.PMI:Performance_monitoring_interrupts
      4786 ± 36%    -100.0%       0.00        interrupts.CPU67.NMI:Non-maskable_interrupts
      4786 ± 36%    -100.0%       0.00        interrupts.CPU67.PMI:Performance_monitoring_interrupts
      6864 ± 26%    -100.0%       0.00        interrupts.CPU68.NMI:Non-maskable_interrupts
      6864 ± 26%    -100.0%       0.00        interrupts.CPU68.PMI:Performance_monitoring_interrupts
    183.00 ± 39%     -65.5%      63.17 ± 80%  interrupts.CPU68.RES:Rescheduling_interrupts
      5280 ± 41%    -100.0%       0.00        interrupts.CPU69.NMI:Non-maskable_interrupts
      5280 ± 41%    -100.0%       0.00        interrupts.CPU69.PMI:Performance_monitoring_interrupts
      5936 ± 33%    -100.0%       0.00        interrupts.CPU7.NMI:Non-maskable_interrupts
      5936 ± 33%    -100.0%       0.00        interrupts.CPU7.PMI:Performance_monitoring_interrupts
      3813 ± 35%    -100.0%       1.00 ±223%  interrupts.CPU70.NMI:Non-maskable_interrupts
      3813 ± 35%    -100.0%       1.00 ±223%  interrupts.CPU70.PMI:Performance_monitoring_interrupts
      5284 ± 44%    -100.0%       0.00        interrupts.CPU71.NMI:Non-maskable_interrupts
      5284 ± 44%    -100.0%       0.00        interrupts.CPU71.PMI:Performance_monitoring_interrupts
      6574 ± 27%    -100.0%       0.00        interrupts.CPU72.NMI:Non-maskable_interrupts
      6574 ± 27%    -100.0%       0.00        interrupts.CPU72.PMI:Performance_monitoring_interrupts
      4859 ± 35%    -100.0%       0.00        interrupts.CPU73.NMI:Non-maskable_interrupts
      4859 ± 35%    -100.0%       0.00        interrupts.CPU73.PMI:Performance_monitoring_interrupts
      4462 ± 36%    -100.0%       0.00        interrupts.CPU74.NMI:Non-maskable_interrupts
      4462 ± 36%    -100.0%       0.00        interrupts.CPU74.PMI:Performance_monitoring_interrupts
      6674 ± 26%    -100.0%       0.00        interrupts.CPU75.NMI:Non-maskable_interrupts
      6674 ± 26%    -100.0%       0.00        interrupts.CPU75.PMI:Performance_monitoring_interrupts
      4385 ± 49%    -100.0%       0.00        interrupts.CPU76.NMI:Non-maskable_interrupts
      4385 ± 49%    -100.0%       0.00        interrupts.CPU76.PMI:Performance_monitoring_interrupts
    730.50 ± 17%     -21.4%     574.50 ±  6%  interrupts.CPU77.CAL:Function_call_interrupts
      5248 ± 45%    -100.0%       0.00        interrupts.CPU77.NMI:Non-maskable_interrupts
      5248 ± 45%    -100.0%       0.00        interrupts.CPU77.PMI:Performance_monitoring_interrupts
      6331 ± 28%    -100.0%       0.00        interrupts.CPU78.NMI:Non-maskable_interrupts
      6331 ± 28%    -100.0%       0.00        interrupts.CPU78.PMI:Performance_monitoring_interrupts
      4801 ± 38%    -100.0%       0.00        interrupts.CPU79.NMI:Non-maskable_interrupts
      4801 ± 38%    -100.0%       0.00        interrupts.CPU79.PMI:Performance_monitoring_interrupts
      5608 ± 37%    -100.0%       0.00        interrupts.CPU8.NMI:Non-maskable_interrupts
      5608 ± 37%    -100.0%       0.00        interrupts.CPU8.PMI:Performance_monitoring_interrupts
      5432 ± 45%    -100.0%       1.00 ±223%  interrupts.CPU80.NMI:Non-maskable_interrupts
      5432 ± 45%    -100.0%       1.00 ±223%  interrupts.CPU80.PMI:Performance_monitoring_interrupts
      6875 ± 22%    -100.0%       1.00 ±223%  interrupts.CPU81.NMI:Non-maskable_interrupts
      6875 ± 22%    -100.0%       1.00 ±223%  interrupts.CPU81.PMI:Performance_monitoring_interrupts
      6375 ± 34%    -100.0%       0.00        interrupts.CPU82.NMI:Non-maskable_interrupts
      6375 ± 34%    -100.0%       0.00        interrupts.CPU82.PMI:Performance_monitoring_interrupts
      3909 ± 43%    -100.0%       0.00        interrupts.CPU83.NMI:Non-maskable_interrupts
      3909 ± 43%    -100.0%       0.00        interrupts.CPU83.PMI:Performance_monitoring_interrupts
      6750 ± 26%    -100.0%       0.00        interrupts.CPU84.NMI:Non-maskable_interrupts
      6750 ± 26%    -100.0%       0.00        interrupts.CPU84.PMI:Performance_monitoring_interrupts
      4277 ± 47%    -100.0%       0.00        interrupts.CPU85.NMI:Non-maskable_interrupts
      4277 ± 47%    -100.0%       0.00        interrupts.CPU85.PMI:Performance_monitoring_interrupts
      3813 ± 35%    -100.0%       0.00        interrupts.CPU86.NMI:Non-maskable_interrupts
      3813 ± 35%    -100.0%       0.00        interrupts.CPU86.PMI:Performance_monitoring_interrupts
      4906 ± 43%    -100.0%       1.00 ±223%  interrupts.CPU87.NMI:Non-maskable_interrupts
      4906 ± 43%    -100.0%       1.00 ±223%  interrupts.CPU87.PMI:Performance_monitoring_interrupts
      5015 ± 45%    -100.0%       0.00        interrupts.CPU88.NMI:Non-maskable_interrupts
      5015 ± 45%    -100.0%       0.00        interrupts.CPU88.PMI:Performance_monitoring_interrupts
      4539 ± 52%    -100.0%       0.00        interrupts.CPU89.NMI:Non-maskable_interrupts
      4539 ± 52%    -100.0%       0.00        interrupts.CPU89.PMI:Performance_monitoring_interrupts
      4803 ± 43%    -100.0%       0.00        interrupts.CPU9.NMI:Non-maskable_interrupts
      4803 ± 43%    -100.0%       0.00        interrupts.CPU9.PMI:Performance_monitoring_interrupts
      3339 ± 31%    -100.0%       0.00        interrupts.CPU90.NMI:Non-maskable_interrupts
      3339 ± 31%    -100.0%       0.00        interrupts.CPU90.PMI:Performance_monitoring_interrupts
      3704 ± 50%    -100.0%       0.00        interrupts.CPU91.NMI:Non-maskable_interrupts
      3704 ± 50%    -100.0%       0.00        interrupts.CPU91.PMI:Performance_monitoring_interrupts
      4157 ± 44%    -100.0%       0.00        interrupts.CPU92.NMI:Non-maskable_interrupts
      4157 ± 44%    -100.0%       0.00        interrupts.CPU92.PMI:Performance_monitoring_interrupts
      3429 ± 37%    -100.0%       0.00        interrupts.CPU93.NMI:Non-maskable_interrupts
      3429 ± 37%    -100.0%       0.00        interrupts.CPU93.PMI:Performance_monitoring_interrupts
      4539 ± 52%    -100.0%       0.00        interrupts.CPU94.NMI:Non-maskable_interrupts
      4539 ± 52%    -100.0%       0.00        interrupts.CPU94.PMI:Performance_monitoring_interrupts
      4583 ± 41%    -100.0%       0.00        interrupts.CPU95.NMI:Non-maskable_interrupts
      4583 ± 41%    -100.0%       0.00        interrupts.CPU95.PMI:Performance_monitoring_interrupts
      5170 ± 44%    -100.0%       0.00        interrupts.CPU96.NMI:Non-maskable_interrupts
      5170 ± 44%    -100.0%       0.00        interrupts.CPU96.PMI:Performance_monitoring_interrupts
      4205 ± 46%    -100.0%       1.00 ±223%  interrupts.CPU97.NMI:Non-maskable_interrupts
      4205 ± 46%    -100.0%       1.00 ±223%  interrupts.CPU97.PMI:Performance_monitoring_interrupts
      5019 ± 39%    -100.0%       0.00        interrupts.CPU98.NMI:Non-maskable_interrupts
      5019 ± 39%    -100.0%       0.00        interrupts.CPU98.PMI:Performance_monitoring_interrupts
      4982 ± 38%    -100.0%       0.00        interrupts.CPU99.NMI:Non-maskable_interrupts
      4982 ± 38%    -100.0%       0.00        interrupts.CPU99.PMI:Performance_monitoring_interrupts
    107.67 ± 16%    -100.0%       0.00        interrupts.IWI:IRQ_work_interrupts
    550173 ±  9%    -100.0%       6.00        interrupts.NMI:Non-maskable_interrupts
    550173 ±  9%    -100.0%       6.00        interrupts.PMI:Performance_monitoring_interrupts

***************************************************************************************************
lkp-hsw-4ex1: 144 threads Intel(R) Xeon(R) CPU E7-8890 v3 @ 2.50GHz with 512G memory
=========================================================================================
compiler/cpufreq_governor/kconfig/mode/nr_task/rootfs/tbox_group/test/testcase/ucode:
gcc-9/performance/x86_64-rhel-8.3/thread/16/debian-10.4-x86_64-20200603.cgz/lkp-hsw-4ex1/pipe1/will-it-scale/0x16

commit:
dde38d5b80 ("kernel: don't include PF_IO_WORKERs as part of same_thread_group()") 43b2a76b1a ("proc: don't show PF_IO_WORKER threads as threads in /proc/<pid>/task/")

dde38d5b805358c7 43b2a76b1a5abcc9833b463bef1
---------------- ---------------------------
%stddev %change %stddev
\ | \

  10347715            -7.1%    9613244        will-it-scale.16.threads
    646731            -7.1%     600827        will-it-scale.per_thread_ops
  10347715            -7.1%    9613244        will-it-scale.workload
   2161039 ±140%     -89.9%     217227 ± 11%  cpuidle.POLL.time
    778537 ± 94%     -90.1%      76755 ± 15%  cpuidle.POLL.usage
      2.35 ±  5%      -0.6        1.75 ±  3%  mpstat.cpu.all.irq%
      0.09 ±  3%      -0.0        0.08 ±  2%  mpstat.cpu.all.soft%
      5.16            +0.5        5.67        mpstat.cpu.all.sys%
    126150 ± 67%    +455.9%     701316 ± 13%  numa-numastat.node2.local_node
    190120 ± 44%    +302.2%     764691 ± 11%  numa-numastat.node2.numa_hit
    343721 ± 40%    +124.8%     772821 ± 19%  numa-numastat.node3.local_node
    411537 ± 35%    +104.5%     841447 ± 16%  numa-numastat.node3.numa_hit
     15406 ±  9%     -19.7%      12379 ± 13%  sched_debug.cfs_rq:/.load.avg
      1538 ±  9%     -28.6%       1098 ± 10%  sched_debug.cpu.clock_task.stddev
      3291           +20.7%       3971 ±  5%  sched_debug.cpu.nr_switches.avg
    -16.72           -28.2%     -12.00        sched_debug.cpu.nr_uninterruptible.min
     86.17            -1.4%      85.00        vmstat.cpu.id
   1367031          +179.8%    3824534        vmstat.memory.cache
     16.00           +12.5%      18.00        vmstat.procs.r
      1773           +41.5%       2508        vmstat.system.cs
     49509 ±  3%   +4948.7%    2499569 ±  2%  meminfo.Active
     49509 ±  3%   +4948.7%    2499569 ±  2%  meminfo.Active(anon)
   1253240          +195.4%    3702564        meminfo.Cached
    532084          +471.8%    3042485        meminfo.Committed_AS
   3151775           +94.6%    6133656        meminfo.Memused
      4357           +12.6%       4906        meminfo.PageTables
     71968 ±  2%   +3403.3%    2521291 ±  2%  meminfo.Shmem
   3803846           +61.3%    6134154        meminfo.max_used_kB
     54196 ±  4%     +25.7%      68114 ±  4%  slabinfo.filp.active_objs
    855.67 ±  4%     +26.0%       1078 ±  4%  slabinfo.filp.active_slabs
     54795 ±  4%     +26.0%      69029 ±  4%  slabinfo.filp.num_objs
    855.67 ±  4%     +26.0%       1078 ±  4%  slabinfo.filp.num_slabs
     19864           -29.1%      14091        slabinfo.proc_inode_cache.active_objs
    413.17           -28.8%     294.17        slabinfo.proc_inode_cache.active_slabs
     19865           -28.8%      14140        slabinfo.proc_inode_cache.num_objs
    413.17           -28.8%     294.17        slabinfo.proc_inode_cache.num_slabs
     24290           +36.2%      33073        slabinfo.radix_tree_node.active_objs
    433.50           +36.2%     590.33        slabinfo.radix_tree_node.active_slabs
     24290           +36.2%      33073        slabinfo.radix_tree_node.num_objs
    433.50           +36.2%     590.33        slabinfo.radix_tree_node.num_slabs
    243.00 ± 67%  +12967.1%      31753 ± 95%  numa-vmstat.node0.nr_active_anon
      1329 ±107%   +2462.5%      34073 ± 87%  numa-vmstat.node0.nr_shmem
    243.00 ± 67%  +12967.1%      31753 ± 95%  numa-vmstat.node0.nr_zone_active_anon
    390.50 ± 70%  +51012.8%     199595 ± 33%  numa-vmstat.node1.nr_active_anon
     74888 ±  3%    +267.5%     275184 ± 24%  numa-vmstat.node1.nr_file_pages
      2116 ± 81%   +9380.3%     200665 ± 33%  numa-vmstat.node1.nr_shmem
    390.50 ± 70%  +51012.8%     199595 ± 33%  numa-vmstat.node1.nr_zone_active_anon
    219.50 ± 74%  +87889.5%     193137 ± 42%  numa-vmstat.node2.nr_active_anon
     75475 ±  5%    +250.7%     264662 ± 31%  numa-vmstat.node2.nr_file_pages
      1579 ± 57%  +12147.1%     193483 ± 42%  numa-vmstat.node2.nr_shmem
    219.50 ± 74%  +87889.5%     193137 ± 42%  numa-vmstat.node2.nr_zone_active_anon
     11535 ±  4%   +1630.4%     199614 ± 60%  numa-vmstat.node3.nr_active_anon
     85408 ±  3%    +226.2%     278565 ± 42%  numa-vmstat.node3.nr_file_pages
     12935 ± 14%   +1456.2%     201307 ± 59%  numa-vmstat.node3.nr_shmem
     11535 ±  4%   +1630.4%     199614 ± 60%  numa-vmstat.node3.nr_zone_active_anon
    575672 ± 16%     +57.8%     908500 ±  8%  numa-vmstat.node3.numa_hit
    439401 ± 23%     +70.1%     747636 ± 11%  numa-vmstat.node3.numa_local
     12360 ±  3%   +4953.6%     624669 ±  2%  proc-vmstat.nr_active_anon
     64392            +8.8%      70043        proc-vmstat.nr_anon_pages
    313288          +195.4%     925417        proc-vmstat.nr_file_pages
     69914            +7.7%      75279        proc-vmstat.nr_inactive_anon
      1091           +12.4%       1226        proc-vmstat.nr_page_table_pages
     17970 ±  2%   +3406.4%     630099 ±  2%  proc-vmstat.nr_shmem
     53352            +4.7%      55880        proc-vmstat.nr_slab_unreclaimable
     12360 ±  3%   +4953.6%     624669 ±  2%  proc-vmstat.nr_zone_active_anon
     69914            +7.7%      75279        proc-vmstat.nr_zone_inactive_anon
   1201307          +109.1%    2512032        proc-vmstat.numa_hit
    968119          +135.4%    2278868        proc-vmstat.numa_local
     13054 ± 49%     -95.5%     588.67 ± 34%  proc-vmstat.numa_pages_migrated
     17335 ±  4%    +382.6%      83662 ±  2%  proc-vmstat.pgactivate
   1269551          +125.3%    2860508 ±  2%  proc-vmstat.pgalloc_normal
   1172329            -2.8%    1138971        proc-vmstat.pgfault
   1183638           +10.8%    1311710 ±  5%  proc-vmstat.pgfree
     13054 ± 49%     -95.5%     588.67 ± 34%  proc-vmstat.pgmigrate_success
     38556 ± 13%    +163.9%     101756 ± 30%  syscalls.sys_close.max
    568.17           -22.5%     440.50 ±  6%  syscalls.sys_close.med
  17718871 ± 11%  -4.3e+06    13405659 ± 21%  syscalls.sys_close.noise.2%
  17644152 ± 11%  -4.3e+06    13344394 ± 22%  syscalls.sys_close.noise.5%
    103226 ± 13%    +267.9%     379798 ± 12%  syscalls.sys_openat.max
      3284           +72.8%       5675 ±  6%  syscalls.sys_openat.med
  99825845 ±  3%  -4.1e+07    58578210 ± 13%  syscalls.sys_openat.noise.100%
 1.404e+08 ±  3%  -3.3e+07   1.078e+08 ± 13%  syscalls.sys_openat.noise.2%
 1.332e+08 ±  3%  -4.5e+07    87987720 ± 18%  syscalls.sys_openat.noise.25%
   1.4e+08 ±  3%  -3.4e+07   1.065e+08 ± 14%  syscalls.sys_openat.noise.5%
 1.183e+08 ±  3%  -4.4e+07    74351118 ± 24%  syscalls.sys_openat.noise.50%
 1.066e+08 ±  3%  -3.9e+07    67567568 ± 24%  syscalls.sys_openat.noise.75%
 1.074e+11 ±  6%  -3.8e+10   6.928e+10 ± 19%  syscalls.sys_read.noise.100%
 1.074e+11 ±  6%  -3.8e+10   6.928e+10 ± 19%  syscalls.sys_read.noise.2%
 1.074e+11 ±  6%  -3.8e+10   6.928e+10 ± 19%  syscalls.sys_read.noise.25%
 1.074e+11 ±  6%  -3.8e+10   6.928e+10 ± 19%  syscalls.sys_read.noise.5%
 1.074e+11 ±  6%  -3.8e+10   6.928e+10 ± 19%  syscalls.sys_read.noise.50%
 1.074e+11 ±  6%  -3.8e+10   6.928e+10 ± 19%  syscalls.sys_read.noise.75%
    973.33 ± 67%  +12957.0%     127087 ± 95%  numa-meminfo.node0.Active
    973.33 ± 67%  +12957.0%     127087 ± 95%  numa-meminfo.node0.Active(anon)
    973600 ±  8%     +18.3%    1152023 ±  8%  numa-meminfo.node0.MemUsed
      5320 ±107%   +2463.2%     136369 ± 87%  numa-meminfo.node0.Shmem
      1563 ± 70%  +51094.1%     800163 ± 33%  numa-meminfo.node1.Active
      1563 ± 70%  +51094.1%     800163 ± 33%  numa-meminfo.node1.Active(anon)
    299552 ±  3%    +268.1%    1102521 ± 24%  numa-meminfo.node1.FilePages
    693218 ±  6%    +133.1%    1615690 ± 16%  numa-meminfo.node1.MemUsed
      8467 ± 81%   +9400.8%     804445 ± 33%  numa-meminfo.node1.Shmem
    879.67 ± 73%  +87936.5%     774427 ± 42%  numa-meminfo.node2.Active
    879.67 ± 73%  +87936.5%     774427 ± 42%  numa-meminfo.node2.Active(anon)
    301902 ±  5%    +251.3%    1060528 ± 31%  numa-meminfo.node2.FilePages
    728495 ± 15%    +119.9%    1601674 ± 20%  numa-meminfo.node2.MemUsed
      6321 ± 57%  +12173.6%     775813 ± 42%  numa-meminfo.node2.Shmem
     46166 ±  4%   +1633.7%     800379 ± 60%  numa-meminfo.node3.Active
     46166 ±  4%   +1633.7%     800379 ± 60%  numa-meminfo.node3.Active(anon)
    341668 ±  3%    +226.7%    1116179 ± 42%  numa-meminfo.node3.FilePages
    756726 ±  7%    +133.5%    1766769 ± 28%  numa-meminfo.node3.MemUsed
     51779 ± 14%   +1458.8%     807147 ± 59%  numa-meminfo.node3.Shmem
     36.03 ± 18%     -36.0        0.00        perf-profile.calltrace.cycles-pp.secondary_startup_64_no_verify
     34.77 ± 16%     -34.8        0.00        perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
     34.77 ± 16%     -34.8        0.00        perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64_no_verify
     34.76 ± 16%     -34.8        0.00        perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
     33.91 ± 16%     -33.9        0.00        perf-profile.calltrace.cycles-pp.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
     33.74 ± 17%     -33.7        0.00        perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary
     32.55 ± 21%     -32.5        0.00        perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry
     31.89 ± 10%     -31.9        0.00        perf-profile.calltrace.cycles-pp.__libc_read
     31.50 ± 10%     -31.5        0.00        perf-profile.calltrace.cycles-pp.__libc_write
     24.59 ± 10%     -24.6        0.00        perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__libc_read
     24.14 ± 10%     -24.1        0.00        perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__libc_write
     12.22 ± 10%     -12.2        0.00        perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_read
     12.04 ± 10%     -12.0        0.00        perf-profile.calltrace.cycles-pp.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe.__libc_write
     11.97 ± 10%     -12.0        0.00        perf-profile.calltrace.cycles-pp.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe.__libc_read
     11.73 ± 10%     -11.7        0.00        perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_write
      8.45 ± 10%      -8.4        0.00        perf-profile.calltrace.cycles-pp.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_read
      7.88 ±  9%      -7.9        0.00        perf-profile.calltrace.cycles-pp.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_write
      7.20 ± 10%      -7.2        0.00        perf-profile.calltrace.cycles-pp.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_read
      6.64 ±  9%      -6.6        0.00        perf-profile.calltrace.cycles-pp.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_write
      5.12 ± 10%      -5.1        0.00        perf-profile.calltrace.cycles-pp.new_sync_read.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe
      4.97 ± 10%      -5.0        0.00        perf-profile.calltrace.cycles-pp.new_sync_write.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
     48.91 ± 10%     -48.9        0.00        perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
     36.03 ± 18%     -36.0        0.00        perf-profile.children.cycles-pp.secondary_startup_64_no_verify
     36.03 ± 18%     -36.0        0.00        perf-profile.children.cycles-pp.cpu_startup_entry
     36.03 ± 18%     -36.0        0.00        perf-profile.children.cycles-pp.do_idle
     35.17 ± 19%     -35.2        0.00        perf-profile.children.cycles-pp.cpuidle_enter
     35.16 ± 19%     -35.2        0.00        perf-profile.children.cycles-pp.cpuidle_enter_state
     34.77 ± 16%     -34.8        0.00        perf-profile.children.cycles-pp.start_secondary
     32.55 ± 21%     -32.5        0.00        perf-profile.children.cycles-pp.intel_idle
     32.38 ± 10%     -32.4        0.00        perf-profile.children.cycles-pp.__libc_read
     31.78 ± 10%     -31.8        0.00        perf-profile.children.cycles-pp.__libc_write
     24.08 ± 10%     -24.1        0.00        perf-profile.children.cycles-pp.syscall_exit_to_user_mode
     24.06 ± 10%     -24.1        0.00        perf-profile.children.cycles-pp.do_syscall_64
      8.49 ± 10%      -8.5        0.00        perf-profile.children.cycles-pp.ksys_read
      7.92 ±  9%      -7.9        0.00        perf-profile.children.cycles-pp.ksys_write
      7.27 ± 10%      -7.3        0.00        perf-profile.children.cycles-pp.vfs_read
      6.70 ±  9%      -6.7        0.00        perf-profile.children.cycles-pp.vfs_write
      6.68 ± 10%      -6.7        0.00        perf-profile.children.cycles-pp.syscall_return_via_sysret
      6.38 ± 10%      -6.4        0.00        perf-profile.children.cycles-pp.syscall_trace_enter
      6.34 ± 10%      -6.3        0.00        perf-profile.children.cycles-pp.__entry_text_start
      6.15 ± 10%      -6.1        0.00        perf-profile.children.cycles-pp.trace_buffer_lock_reserve
      5.96 ± 10%      -6.0        0.00        perf-profile.children.cycles-pp.ftrace_syscall_enter
      5.66 ± 10%      -5.7        0.00        perf-profile.children.cycles-pp.syscall_exit_to_user_mode_prepare
      5.42 ± 11%      -5.4        0.00        perf-profile.children.cycles-pp.ring_buffer_lock_reserve
      5.29 ± 10%      -5.3        0.00        perf-profile.children.cycles-pp.ftrace_syscall_exit
      5.16 ± 11%      -5.2        0.00        perf-profile.children.cycles-pp.new_sync_read
      5.03 ±  9%      -5.0        0.00        perf-profile.children.cycles-pp.new_sync_write
      4.69 ± 10%      -4.7        0.00        perf-profile.children.cycles-pp.pipe_read
     32.55 ± 21%     -32.5        0.00        perf-profile.self.cycles-pp.intel_idle
     18.11 ± 10%     -18.1        0.00        perf-profile.self.cycles-pp.syscall_exit_to_user_mode
      6.65 ± 10%      -6.7        0.00        perf-profile.self.cycles-pp.syscall_return_via_sysret
      5.45 ± 10%      -5.5        0.00        perf-profile.self.cycles-pp.__entry_text_start
      1.86            +9.5%       2.04        perf-stat.i.MPKI
 7.787e+09           +13.4%  8.828e+09        perf-stat.i.branch-instructions
      1.27            -0.1        1.14        perf-stat.i.branch-miss-rate%
      0.34 ±  9%      +5.1        5.46 ± 13%  perf-stat.i.cache-miss-rate%
    245601 ±  9%   +1822.1%    4720745 ± 13%  perf-stat.i.cache-misses
  70035252           +24.3%   87072170        perf-stat.i.cache-references
      1724           +42.7%       2461        perf-stat.i.context-switches
 5.293e+10           +12.2%  5.936e+10        perf-stat.i.cpu-cycles
    251199 ± 10%     -88.3%      29314 ± 26%  perf-stat.i.cycles-between-cache-misses
      0.21            -0.0        0.19        perf-stat.i.dTLB-load-miss-rate%
 1.139e+10            +9.2%  1.244e+10        perf-stat.i.dTLB-loads
      0.26            -0.0        0.23        perf-stat.i.dTLB-store-miss-rate%
 8.301e+09            +7.0%  8.884e+09        perf-stat.i.dTLB-stores
     86.29            -5.0       81.30 ±  2%  perf-stat.i.iTLB-load-miss-rate%
   3599988 ± 13%     +44.8%    5213290 ± 14%  perf-stat.i.iTLB-loads
 3.804e+10           +13.4%  4.312e+10        perf-stat.i.instructions
      1686 ±  2%     +13.5%       1913 ±  4%  perf-stat.i.instructions-per-iTLB-miss
      1.02 ±  2%     -94.3%       0.06 ± 44%  perf-stat.i.major-faults
      0.37           +12.2%       0.41        perf-stat.i.metric.GHz
      1.09 ± 10%     -24.1%       0.83 ± 14%  perf-stat.i.metric.K/sec
    191.47            +9.8%     210.19        perf-stat.i.metric.M/sec
      3770            -3.1%       3653        perf-stat.i.minor-faults
     94.62            +3.2       97.82        perf-stat.i.node-load-miss-rate%
    155456 ± 10%   +1936.0%    3165118 ± 16%  perf-stat.i.node-load-misses
     13817 ±  8%    +243.8%      47505 ± 43%  perf-stat.i.node-loads
     46.19 ± 12%     +39.2       85.35 ±  2%  perf-stat.i.node-store-miss-rate%
     32493 ± 13%   +4121.7%    1371781 ± 11%  perf-stat.i.node-store-misses
     40563 ± 14%    +220.5%     130023 ±  8%  perf-stat.i.node-stores
      3771            -3.1%       3653        perf-stat.i.page-faults
      1.84            +9.7%       2.02        perf-stat.overall.MPKI
      1.26            -0.1        1.13        perf-stat.overall.branch-miss-rate%
      0.35 ±  9%      +5.1        5.42 ± 13%  perf-stat.overall.cache-miss-rate%
    216967 ±  8%     -94.1%      12875 ± 16%  perf-stat.overall.cycles-between-cache-misses
      0.21            -0.0        0.19        perf-stat.overall.dTLB-load-miss-rate%
      0.26            -0.0        0.23        perf-stat.overall.dTLB-store-miss-rate%
     86.34            -5.0       81.34 ±  2%  perf-stat.overall.iTLB-load-miss-rate%
      1677 ±  2%     +13.5%       1904 ±  4%  perf-stat.overall.instructions-per-iTLB-miss
     91.77            +6.7       98.44        perf-stat.overall.node-load-miss-rate%
     44.57 ± 11%     +46.7       91.25        perf-stat.overall.node-store-miss-rate%
   1106216           +22.1%    1350408        perf-stat.overall.path-length
  7.76e+09           +13.4%  8.798e+09        perf-stat.ps.branch-instructions
    245116 ±  9%   +1819.1%    4704109 ± 13%  perf-stat.ps.cache-misses
  69795445           +24.3%   86773526        perf-stat.ps.cache-references
      1719           +42.7%       2452        perf-stat.ps.context-switches
 5.275e+10           +12.2%  5.916e+10        perf-stat.ps.cpu-cycles
 1.135e+10            +9.2%   1.24e+10        perf-stat.ps.dTLB-loads
 8.273e+09            +7.0%  8.853e+09        perf-stat.ps.dTLB-stores
   3587022 ± 13%     +44.8%    5195656 ± 14%  perf-stat.ps.iTLB-loads
 3.792e+10           +13.3%  4.298e+10        perf-stat.ps.instructions
      1.01 ±  2%     -94.3%       0.06 ± 44%  perf-stat.ps.major-faults
      3757            -3.1%       3640        perf-stat.ps.minor-faults
    155084 ± 10%   +1933.7%    3153975 ± 16%  perf-stat.ps.node-load-misses
     13831 ±  8%    +242.2%      47330 ± 43%  perf-stat.ps.node-loads
     32417 ± 13%   +4116.6%    1366944 ± 11%  perf-stat.ps.node-store-misses
     40467 ± 14%    +220.2%     129580 ±  8%  perf-stat.ps.node-stores
      3758            -3.1%       3640        perf-stat.ps.page-faults
 1.145e+13           +13.4%  1.298e+13        perf-stat.total.instructions
      0.02 ± 23%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.__traceiter_sched_switch.__traceiter_sched_switch.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.04 ± 18%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.__traceiter_sched_switch.__traceiter_sched_switch.devkmsg_read.vfs_read.ksys_read
      0.01 ± 21%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.__traceiter_sched_switch.__traceiter_sched_switch.do_nanosleep.hrtimer_nanosleep.__x64_sys_nanosleep
      0.03 ± 32%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.__traceiter_sched_switch.__traceiter_sched_switch.do_syslog.part.0
      0.01 ±  7%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.__traceiter_sched_switch.__traceiter_sched_switch.do_task_dead.do_exit.do_group_exit
      0.01 ± 19%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.__traceiter_sched_switch.__traceiter_sched_switch.do_wait.kernel_wait4.__do_sys_wait4
      0.03 ± 37%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.__traceiter_sched_switch.__traceiter_sched_switch.futex_wait_queue_me.futex_wait.do_futex
      0.00 ± 14%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.__traceiter_sched_switch.__traceiter_sched_switch.pipe_read.new_sync_read.vfs_read
      0.05 ± 34%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.__traceiter_sched_switch.__traceiter_sched_switch.rcu_gp_kthread.kthread.ret_from_fork
      0.02 ± 24%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.__traceiter_sched_switch.__traceiter_sched_switch.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait
      0.01 ± 20%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.__traceiter_sched_switch.__traceiter_sched_switch.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop
      0.02 ± 31%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.__traceiter_sched_switch.__traceiter_sched_switch.schedule_timeout.io_schedule_timeout.wait_for_completion_io
      0.02 ± 61%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.__traceiter_sched_switch.__traceiter_sched_switch.schedule_timeout.kcompactd.kthread
      0.04 ± 43%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.__traceiter_sched_switch.__traceiter_sched_switch.schedule_timeout.rcu_gp_kthread.kthread
      0.02 ± 40%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.__traceiter_sched_switch.__traceiter_sched_switch.schedule_timeout.wait_for_completion.__flush_work
      0.01 ±  4%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.__traceiter_sched_switch.__traceiter_sched_switch.smpboot_thread_fn.kthread.ret_from_fork
      0.00 ± 14%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.__traceiter_sched_switch.__traceiter_sched_switch.wait_for_partner.fifo_open.do_dentry_open
      1.91 ±218%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.__traceiter_sched_switch.__traceiter_sched_switch.worker_thread.kthread.ret_from_fork
      0.03 ± 42%    -100.0%       0.00        perf-sched.sch_delay.max.ms.__traceiter_sched_switch.__traceiter_sched_switch.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.06 ± 19%    -100.0%       0.00        perf-sched.sch_delay.max.ms.__traceiter_sched_switch.__traceiter_sched_switch.devkmsg_read.vfs_read.ksys_read
      0.03 ± 49%    -100.0%       0.00        perf-sched.sch_delay.max.ms.__traceiter_sched_switch.__traceiter_sched_switch.do_nanosleep.hrtimer_nanosleep.__x64_sys_nanosleep
      0.05 ± 23%    -100.0%       0.00        perf-sched.sch_delay.max.ms.__traceiter_sched_switch.__traceiter_sched_switch.do_syslog.part.0
      0.05 ±  3%    -100.0%       0.00        perf-sched.sch_delay.max.ms.__traceiter_sched_switch.__traceiter_sched_switch.do_task_dead.do_exit.do_group_exit
      0.06 ±  9%    -100.0%       0.00        perf-sched.sch_delay.max.ms.__traceiter_sched_switch.__traceiter_sched_switch.do_wait.kernel_wait4.__do_sys_wait4
      0.03 ± 47%    -100.0%       0.00        perf-sched.sch_delay.max.ms.__traceiter_sched_switch.__traceiter_sched_switch.exit_to_user_mode_prepare.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe
      0.05 ± 33%    -100.0%       0.00        perf-sched.sch_delay.max.ms.__traceiter_sched_switch.__traceiter_sched_switch.futex_wait_queue_me.futex_wait.do_futex
      0.06 ±  8%    -100.0%       0.00        perf-sched.sch_delay.max.ms.__traceiter_sched_switch.__traceiter_sched_switch.pipe_read.new_sync_read.vfs_read
      0.04 ± 12%    -100.0%       0.00        perf-sched.sch_delay.max.ms.__traceiter_sched_switch.__traceiter_sched_switch.preempt_schedule_common.__cond_resched.stop_one_cpu
      0.07 ± 30%    -100.0%       0.00        perf-sched.sch_delay.max.ms.__traceiter_sched_switch.__traceiter_sched_switch.rcu_gp_kthread.kthread.ret_from_fork
      0.05 ± 20%    -100.0%       0.00        perf-sched.sch_delay.max.ms.__traceiter_sched_switch.__traceiter_sched_switch.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait
      0.04 ± 29%    -100.0%       0.00        perf-sched.sch_delay.max.ms.__traceiter_sched_switch.__traceiter_sched_switch.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop
      0.02 ± 31%    -100.0%       0.00        perf-sched.sch_delay.max.ms.__traceiter_sched_switch.__traceiter_sched_switch.schedule_timeout.io_schedule_timeout.wait_for_completion_io
      0.55 ±208%    -100.0%       0.00        perf-sched.sch_delay.max.ms.__traceiter_sched_switch.__traceiter_sched_switch.schedule_timeout.kcompactd.kthread
      6.73 ±140%    -100.0%       0.00        perf-sched.sch_delay.max.ms.__traceiter_sched_switch.__traceiter_sched_switch.schedule_timeout.rcu_gp_kthread.kthread
      0.02 ± 51%    -100.0%       0.00        perf-sched.sch_delay.max.ms.__traceiter_sched_switch.__traceiter_sched_switch.schedule_timeout.wait_for_completion.__flush_work
      6.77 ±  9%    -100.0%       0.00        perf-sched.sch_delay.max.ms.__traceiter_sched_switch.__traceiter_sched_switch.smpboot_thread_fn.kthread.ret_from_fork
      0.03 ± 28%    -100.0%       0.00        perf-sched.sch_delay.max.ms.__traceiter_sched_switch.__traceiter_sched_switch.wait_for_partner.fifo_open.do_dentry_open
    298.90 ±213%    -100.0%       0.00        perf-sched.sch_delay.max.ms.__traceiter_sched_switch.__traceiter_sched_switch.worker_thread.kthread.ret_from_fork
      0.14 ±206%    -100.0%       0.00        perf-sched.total_sch_delay.average.ms
    300.24 ±212%    -100.0%       0.00        perf-sched.total_sch_delay.max.ms
    215.33          -100.0%       0.00        perf-sched.total_wait_and_delay.average.ms
      9201          -100.0%       0.00        perf-sched.total_wait_and_delay.count.ms
      8311 ±  8%    -100.0%       0.00        perf-sched.total_wait_and_delay.max.ms
    215.19          -100.0%       0.00        perf-sched.total_wait_time.average.ms
      8311 ±  8%    -100.0%       0.00        perf-sched.total_wait_time.max.ms
    899.66          -100.0%       0.00        perf-sched.wait_and_delay.avg.ms.__traceiter_sched_switch.__traceiter_sched_switch.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe
    259.95 ± 13%    -100.0%       0.00        perf-sched.wait_and_delay.avg.ms.__traceiter_sched_switch.__traceiter_sched_switch.devkmsg_read.vfs_read.ksys_read
    781.14 ±  5%    -100.0%       0.00        perf-sched.wait_and_delay.avg.ms.__traceiter_sched_switch.__traceiter_sched_switch.do_nanosleep.hrtimer_nanosleep.__x64_sys_nanosleep
    259.95 ± 13%    -100.0%       0.00        perf-sched.wait_and_delay.avg.ms.__traceiter_sched_switch.__traceiter_sched_switch.do_syslog.part.0
    211.48 ±  5%    -100.0%       0.00        perf-sched.wait_and_delay.avg.ms.__traceiter_sched_switch.__traceiter_sched_switch.do_task_dead.do_exit.do_group_exit
      0.88          -100.0%       0.00        perf-sched.wait_and_delay.avg.ms.__traceiter_sched_switch.__traceiter_sched_switch.do_wait.kernel_wait4.__do_sys_wait4
      0.05 ± 26%    -100.0%       0.00        perf-sched.wait_and_delay.avg.ms.__traceiter_sched_switch.__traceiter_sched_switch.exit_to_user_mode_prepare.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe
     53.04          -100.0%       0.00        perf-sched.wait_and_delay.avg.ms.__traceiter_sched_switch.__traceiter_sched_switch.pipe_read.new_sync_read.vfs_read
      0.05 ± 16%    -100.0%       0.00        perf-sched.wait_and_delay.avg.ms.__traceiter_sched_switch.__traceiter_sched_switch.preempt_schedule_common.__cond_resched.mutex_lock
      5.67 ±  3%    -100.0%       0.00        perf-sched.wait_and_delay.avg.ms.__traceiter_sched_switch.__traceiter_sched_switch.preempt_schedule_common.__cond_resched.stop_one_cpu
      2.82 ±  4%    -100.0%       0.00        perf-sched.wait_and_delay.avg.ms.__traceiter_sched_switch.__traceiter_sched_switch.rcu_gp_kthread.kthread.ret_from_fork
    580.11 ±  6%    -100.0%       0.00        perf-sched.wait_and_delay.avg.ms.__traceiter_sched_switch.__traceiter_sched_switch.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop
    677.37 ± 30%    -100.0%       0.00        perf-sched.wait_and_delay.avg.ms.__traceiter_sched_switch.__traceiter_sched_switch.schedule_timeout.io_schedule_timeout.wait_for_completion_io
    478.71          -100.0%       0.00        perf-sched.wait_and_delay.avg.ms.__traceiter_sched_switch.__traceiter_sched_switch.schedule_timeout.kcompactd.kthread
     12.51 ±  3%    -100.0%       0.00        perf-sched.wait_and_delay.avg.ms.__traceiter_sched_switch.__traceiter_sched_switch.schedule_timeout.rcu_gp_kthread.kthread
    687.31 ±  2%    -100.0%       0.00        perf-sched.wait_and_delay.avg.ms.__traceiter_sched_switch.__traceiter_sched_switch.smpboot_thread_fn.kthread.ret_from_fork
      0.00 ± 14%    -100.0%       0.00        perf-sched.wait_and_delay.avg.ms.__traceiter_sched_switch.__traceiter_sched_switch.wait_for_partner.fifo_open.do_dentry_open
    438.87 ± 11%    -100.0%       0.00        perf-sched.wait_and_delay.avg.ms.__traceiter_sched_switch.__traceiter_sched_switch.worker_thread.kthread.ret_from_fork
     10.00          -100.0%       0.00        perf-sched.wait_and_delay.count.__traceiter_sched_switch.__traceiter_sched_switch.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe
      6.33 ± 11%    -100.0%       0.00        perf-sched.wait_and_delay.count.__traceiter_sched_switch.__traceiter_sched_switch.devkmsg_read.vfs_read.ksys_read
     23.17 ±  5%    -100.0%       0.00        perf-sched.wait_and_delay.count.__traceiter_sched_switch.__traceiter_sched_switch.do_nanosleep.hrtimer_nanosleep.__x64_sys_nanosleep
      6.33 ± 11%    -100.0%       0.00        perf-sched.wait_and_delay.count.__traceiter_sched_switch.__traceiter_sched_switch.do_syslog.part.0
    313.50          -100.0%       0.00        perf-sched.wait_and_delay.count.__traceiter_sched_switch.__traceiter_sched_switch.do_task_dead.do_exit.do_group_exit
    310.83          -100.0%       0.00        perf-sched.wait_and_delay.count.__traceiter_sched_switch.__traceiter_sched_switch.do_wait.kernel_wait4.__do_sys_wait4
     69.00 ± 43%    -100.0%       0.00        perf-sched.wait_and_delay.count.__traceiter_sched_switch.__traceiter_sched_switch.exit_to_user_mode_prepare.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe
      2916          -100.0%       0.00        perf-sched.wait_and_delay.count.__traceiter_sched_switch.__traceiter_sched_switch.pipe_read.new_sync_read.vfs_read
     82.83 ± 13%    -100.0%       0.00        perf-sched.wait_and_delay.count.__traceiter_sched_switch.__traceiter_sched_switch.preempt_schedule_common.__cond_resched.mutex_lock
      1655 ±  4%    -100.0%       0.00        perf-sched.wait_and_delay.count.__traceiter_sched_switch.__traceiter_sched_switch.preempt_schedule_common.__cond_resched.stop_one_cpu
     83.50 ±  7%    -100.0%       0.00        perf-sched.wait_and_delay.count.__traceiter_sched_switch.__traceiter_sched_switch.rcu_gp_kthread.kthread.ret_from_fork
     59.50 ±  6%    -100.0%       0.00        perf-sched.wait_and_delay.count.__traceiter_sched_switch.__traceiter_sched_switch.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop
      4.83 ±  7%    -100.0%       0.00        perf-sched.wait_and_delay.count.__traceiter_sched_switch.__traceiter_sched_switch.schedule_timeout.io_schedule_timeout.wait_for_completion_io
     79.67          -100.0%       0.00        perf-sched.wait_and_delay.count.__traceiter_sched_switch.__traceiter_sched_switch.schedule_timeout.kcompactd.kthread
    782.17 ±  3%    -100.0%       0.00        perf-sched.wait_and_delay.count.__traceiter_sched_switch.__traceiter_sched_switch.schedule_timeout.rcu_gp_kthread.kthread
      1947 ±  3%    -100.0%       0.00        perf-sched.wait_and_delay.count.__traceiter_sched_switch.__traceiter_sched_switch.smpboot_thread_fn.kthread.ret_from_fork
     73.00          -100.0%       0.00        perf-sched.wait_and_delay.count.__traceiter_sched_switch.__traceiter_sched_switch.wait_for_partner.fifo_open.do_dentry_open
    658.50 ±  2%    -100.0%       0.00        perf-sched.wait_and_delay.count.__traceiter_sched_switch.__traceiter_sched_switch.worker_thread.kthread.ret_from_fork
    999.71          -100.0%       0.00        perf-sched.wait_and_delay.max.ms.__traceiter_sched_switch.__traceiter_sched_switch.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe
      1537 ±  5%    -100.0%       0.00        perf-sched.wait_and_delay.max.ms.__traceiter_sched_switch.__traceiter_sched_switch.devkmsg_read.vfs_read.ksys_read
      1000          -100.0%       0.00        perf-sched.wait_and_delay.max.ms.__traceiter_sched_switch.__traceiter_sched_switch.do_nanosleep.hrtimer_nanosleep.__x64_sys_nanosleep
      1537 ±  5%    -100.0%       0.00        perf-sched.wait_and_delay.max.ms.__traceiter_sched_switch.__traceiter_sched_switch.do_syslog.part.0
      1966 ± 59%    -100.0%       0.00        perf-sched.wait_and_delay.max.ms.__traceiter_sched_switch.__traceiter_sched_switch.do_task_dead.do_exit.do_group_exit
     20.30          -100.0%       0.00        perf-sched.wait_and_delay.max.ms.__traceiter_sched_switch.__traceiter_sched_switch.do_wait.kernel_wait4.__do_sys_wait4
      0.48 ±164%    -100.0%       0.00        perf-sched.wait_and_delay.max.ms.__traceiter_sched_switch.__traceiter_sched_switch.exit_to_user_mode_prepare.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe
      1540 ±  5%    -100.0%       0.00        perf-sched.wait_and_delay.max.ms.__traceiter_sched_switch.__traceiter_sched_switch.pipe_read.new_sync_read.vfs_read
      0.12 ± 46%    -100.0%       0.00        perf-sched.wait_and_delay.max.ms.__traceiter_sched_switch.__traceiter_sched_switch.preempt_schedule_common.__cond_resched.mutex_lock
      1071 ± 14%    -100.0%       0.00        perf-sched.wait_and_delay.max.ms.__traceiter_sched_switch.__traceiter_sched_switch.preempt_schedule_common.__cond_resched.stop_one_cpu
      4.54 ± 10%    -100.0%       0.00        perf-sched.wait_and_delay.max.ms.__traceiter_sched_switch.__traceiter_sched_switch.rcu_gp_kthread.kthread.ret_from_fork
      7489          -100.0%       0.00        perf-sched.wait_and_delay.max.ms.__traceiter_sched_switch.__traceiter_sched_switch.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop
      1151 ± 23%    -100.0%       0.00        perf-sched.wait_and_delay.max.ms.__traceiter_sched_switch.__traceiter_sched_switch.schedule_timeout.io_schedule_timeout.wait_for_completion_io
    505.63          -100.0%       0.00        perf-sched.wait_and_delay.max.ms.__traceiter_sched_switch.__traceiter_sched_switch.schedule_timeout.kcompactd.kthread
    376.36 ± 10%    -100.0%       0.00        perf-sched.wait_and_delay.max.ms.__traceiter_sched_switch.__traceiter_sched_switch.schedule_timeout.rcu_gp_kthread.kthread
      1691 ± 88%    -100.0%       0.00        perf-sched.wait_and_delay.max.ms.__traceiter_sched_switch.__traceiter_sched_switch.smpboot_thread_fn.kthread.ret_from_fork
      0.03 ± 28%    -100.0%       0.00        perf-sched.wait_and_delay.max.ms.__traceiter_sched_switch.__traceiter_sched_switch.wait_for_partner.fifo_open.do_dentry_open
      7711 ± 21%    -100.0%       0.00        perf-sched.wait_and_delay.max.ms.__traceiter_sched_switch.__traceiter_sched_switch.worker_thread.kthread.ret_from_fork
    899.64          -100.0%       0.00        perf-sched.wait_time.avg.ms.__traceiter_sched_switch.__traceiter_sched_switch.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe
    259.91 ± 13%    -100.0%       0.00        perf-sched.wait_time.avg.ms.__traceiter_sched_switch.__traceiter_sched_switch.devkmsg_read.vfs_read.ksys_read
    781.13 ±  5%    -100.0%       0.00        perf-sched.wait_time.avg.ms.__traceiter_sched_switch.__traceiter_sched_switch.do_nanosleep.hrtimer_nanosleep.__x64_sys_nanosleep
    259.91 ± 13%    -100.0%       0.00        perf-sched.wait_time.avg.ms.__traceiter_sched_switch.__traceiter_sched_switch.do_syslog.part.0
    211.47 ±  5%    -100.0%       0.00        perf-sched.wait_time.avg.ms.__traceiter_sched_switch.__traceiter_sched_switch.do_task_dead.do_exit.do_group_exit
      0.86          -100.0%       0.00        perf-sched.wait_time.avg.ms.__traceiter_sched_switch.__traceiter_sched_switch.do_wait.kernel_wait4.__do_sys_wait4
      0.06 ± 49%    -100.0%       0.00        perf-sched.wait_time.avg.ms.__traceiter_sched_switch.__traceiter_sched_switch.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt
      0.04 ± 11%    -100.0%       0.00        perf-sched.wait_time.avg.ms.__traceiter_sched_switch.__traceiter_sched_switch.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_irq_work
      0.06 ± 32%    -100.0%       0.00        perf-sched.wait_time.avg.ms.__traceiter_sched_switch.__traceiter_sched_switch.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_reschedule_ipi
      0.05 ± 26%    -100.0%       0.00        perf-sched.wait_time.avg.ms.__traceiter_sched_switch.__traceiter_sched_switch.exit_to_user_mode_prepare.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe
    208.50 ±196%    -100.0%       0.00        perf-sched.wait_time.avg.ms.__traceiter_sched_switch.__traceiter_sched_switch.futex_wait_queue_me.futex_wait.do_futex
     53.03          -100.0%       0.00        perf-sched.wait_time.avg.ms.__traceiter_sched_switch.__traceiter_sched_switch.pipe_read.new_sync_read.vfs_read
      0.05 ± 23%    -100.0%       0.00        perf-sched.wait_time.avg.ms.__traceiter_sched_switch.__traceiter_sched_switch.preempt_schedule_common.__cond_resched.copy_page_from_iter
      0.06 ± 47%    -100.0%       0.00        perf-sched.wait_time.avg.ms.__traceiter_sched_switch.__traceiter_sched_switch.preempt_schedule_common.__cond_resched.copy_page_to_iter
      0.05 ± 16%    -100.0%       0.00        perf-sched.wait_time.avg.ms.__traceiter_sched_switch.__traceiter_sched_switch.preempt_schedule_common.__cond_resched.mutex_lock
      5.67 ±  3%    -100.0%       0.00        perf-sched.wait_time.avg.ms.__traceiter_sched_switch.__traceiter_sched_switch.preempt_schedule_common.__cond_resched.stop_one_cpu
      2.77 ±  4%    -100.0%       0.00        perf-sched.wait_time.avg.ms.__traceiter_sched_switch.__traceiter_sched_switch.rcu_gp_kthread.kthread.ret_from_fork
    247.85 ± 48%    -100.0%       0.00        perf-sched.wait_time.avg.ms.__traceiter_sched_switch.__traceiter_sched_switch.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait
    580.11 ±  6%    -100.0%       0.00        perf-sched.wait_time.avg.ms.__traceiter_sched_switch.__traceiter_sched_switch.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop
    677.35 ± 30%    -100.0%       0.00        perf-sched.wait_time.avg.ms.__traceiter_sched_switch.__traceiter_sched_switch.schedule_timeout.io_schedule_timeout.wait_for_completion_io
    478.69          -100.0%       0.00        perf-sched.wait_time.avg.ms.__traceiter_sched_switch.__traceiter_sched_switch.schedule_timeout.kcompactd.kthread
      0.01 ±110%    -100.0%       0.00        perf-sched.wait_time.avg.ms.__traceiter_sched_switch.__traceiter_sched_switch.schedule_timeout.khugepaged.kthread
     12.47 ±  3%    -100.0%       0.00        perf-sched.wait_time.avg.ms.__traceiter_sched_switch.__traceiter_sched_switch.schedule_timeout.rcu_gp_kthread.kthread
    687.30 ±  2%    -100.0%       0.00        perf-sched.wait_time.avg.ms.__traceiter_sched_switch.__traceiter_sched_switch.smpboot_thread_fn.kthread.ret_from_fork
    436.96 ± 11%    -100.0%       0.00        perf-sched.wait_time.avg.ms.__traceiter_sched_switch.__traceiter_sched_switch.worker_thread.kthread.ret_from_fork
    999.69          -100.0%       0.00        perf-sched.wait_time.max.ms.__traceiter_sched_switch.__traceiter_sched_switch.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe
      1537 ±  5%    -100.0%       0.00        perf-sched.wait_time.max.ms.__traceiter_sched_switch.__traceiter_sched_switch.devkmsg_read.vfs_read.ksys_read
      1000          -100.0%       0.00        perf-sched.wait_time.max.ms.__traceiter_sched_switch.__traceiter_sched_switch.do_nanosleep.hrtimer_nanosleep.__x64_sys_nanosleep
      1537 ±  5%    -100.0%       0.00        perf-sched.wait_time.max.ms.__traceiter_sched_switch.__traceiter_sched_switch.do_syslog.part.0
      1966 ± 59%    -100.0%       0.00        perf-sched.wait_time.max.ms.__traceiter_sched_switch.__traceiter_sched_switch.do_task_dead.do_exit.do_group_exit
     20.30          -100.0%       0.00        perf-sched.wait_time.max.ms.__traceiter_sched_switch.__traceiter_sched_switch.do_wait.kernel_wait4.__do_sys_wait4
      0.46 ±162%    -100.0%       0.00        perf-sched.wait_time.max.ms.__traceiter_sched_switch.__traceiter_sched_switch.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt
      0.05 ± 13%    -100.0%       0.00        perf-sched.wait_time.max.ms.__traceiter_sched_switch.__traceiter_sched_switch.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_irq_work
      0.13 ±115%    -100.0%       0.00        perf-sched.wait_time.max.ms.__traceiter_sched_switch.__traceiter_sched_switch.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_reschedule_ipi
      0.48 ±164%    -100.0%       0.00        perf-sched.wait_time.max.ms.__traceiter_sched_switch.__traceiter_sched_switch.exit_to_user_mode_prepare.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe
      1152 ±213%    -100.0%       0.00        perf-sched.wait_time.max.ms.__traceiter_sched_switch.__traceiter_sched_switch.futex_wait_queue_me.futex_wait.do_futex
      1540 ±  5%    -100.0%       0.00        perf-sched.wait_time.max.ms.__traceiter_sched_switch.__traceiter_sched_switch.pipe_read.new_sync_read.vfs_read
      0.08 ± 71%    -100.0%       0.00        perf-sched.wait_time.max.ms.__traceiter_sched_switch.__traceiter_sched_switch.preempt_schedule_common.__cond_resched.copy_page_from_iter
      0.09 ± 73%    -100.0%       0.00        perf-sched.wait_time.max.ms.__traceiter_sched_switch.__traceiter_sched_switch.preempt_schedule_common.__cond_resched.copy_page_to_iter
      0.12 ± 46%    -100.0%       0.00        perf-sched.wait_time.max.ms.__traceiter_sched_switch.__traceiter_sched_switch.preempt_schedule_common.__cond_resched.mutex_lock
      1070 ± 14%    -100.0%       0.00        perf-sched.wait_time.max.ms.__traceiter_sched_switch.__traceiter_sched_switch.preempt_schedule_common.__cond_resched.stop_one_cpu
      4.50 ± 10%    -100.0%       0.00        perf-sched.wait_time.max.ms.__traceiter_sched_switch.__traceiter_sched_switch.rcu_gp_kthread.kthread.ret_from_fork
      2024 ± 94%    -100.0%       0.00        perf-sched.wait_time.max.ms.__traceiter_sched_switch.__traceiter_sched_switch.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait
      7489          -100.0%       0.00        perf-sched.wait_time.max.ms.__traceiter_sched_switch.__traceiter_sched_switch.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop
      1151 ± 23%    -100.0%       0.00        perf-sched.wait_time.max.ms.__traceiter_sched_switch.__traceiter_sched_switch.schedule_timeout.io_schedule_timeout.wait_for_completion_io
    505.62          -100.0%       0.00        perf-sched.wait_time.max.ms.__traceiter_sched_switch.__traceiter_sched_switch.schedule_timeout.kcompactd.kthread
      0.01 ±110%    -100.0%       0.00        perf-sched.wait_time.max.ms.__traceiter_sched_switch.__traceiter_sched_switch.schedule_timeout.khugepaged.kthread
    376.31 ± 10%    -100.0%       0.00        perf-sched.wait_time.max.ms.__traceiter_sched_switch.__traceiter_sched_switch.schedule_timeout.rcu_gp_kthread.kthread
      1691 ± 88%    -100.0%       0.00        perf-sched.wait_time.max.ms.__traceiter_sched_switch.__traceiter_sched_switch.smpboot_thread_fn.kthread.ret_from_fork
      7711 ± 21%    -100.0%       0.00        perf-sched.wait_time.max.ms.__traceiter_sched_switch.__traceiter_sched_switch.worker_thread.kthread.ret_from_fork
     19423 ± 16%    +181.4%      54663 ±  9%  softirqs.CPU0.RCU
     17368 ± 14%    +203.7%      52743 ±  7%  softirqs.CPU1.RCU
     17971 ± 17%    +220.1%      57519 ±  4%  softirqs.CPU10.RCU
     14509 ± 18%    +188.4%      41847 ± 18%  softirqs.CPU100.RCU
     16450 ± 21%    +192.8%      48170 ±  9%  softirqs.CPU101.RCU
     16134 ± 22%    +191.2%      46984 ± 12%  softirqs.CPU102.RCU
     15942 ± 23%    +210.5%      49505 ± 15%  softirqs.CPU103.RCU
     14345 ± 18%    +222.9%      46315 ± 13%  softirqs.CPU104.RCU
     15933 ± 22%    +194.7%      46949 ±  6%  softirqs.CPU105.RCU
     16305 ± 22%    +205.6%      49821 ± 12%  softirqs.CPU106.RCU
     15880 ± 24%    +195.4%      46907 ±  6%  softirqs.CPU107.RCU
     10021 ± 14%    +802.9%      90484 ± 21%  softirqs.CPU108.RCU
      9881 ± 11%    +346.2%      44092 ± 30%  softirqs.CPU109.RCU
     18504 ± 11%    +229.5%      60977 ±  4%  softirqs.CPU11.RCU
      8863 ±  8%    +273.6%      33113 ± 23%  softirqs.CPU110.RCU
      9235 ±  7%    +231.7%      30633 ± 18%  softirqs.CPU111.RCU
     12750 ± 17%    +283.6%      48906 ± 25%  softirqs.CPU112.RCU
     12897 ± 17%    +240.0%      43852 ± 11%  softirqs.CPU113.RCU
      8779 ±  5%    +231.7%      29116 ± 13%  softirqs.CPU114.RCU
     12658 ± 26%    +248.9%      44164 ± 22%  softirqs.CPU115.RCU
     11441 ± 11%    +247.2%      39726 ± 10%  softirqs.CPU116.RCU
     13127 ± 15%    +233.2%      43736 ± 12%  softirqs.CPU117.RCU
     15022 ± 18%    +222.7%      48483 ±  6%  softirqs.CPU118.RCU
     16019 ± 21%    +222.8%      51715 ±  9%  softirqs.CPU119.RCU
     18445 ± 13%    +229.4%      60762 ±  5%  softirqs.CPU12.RCU
     16411 ± 22%    +272.3%      61102 ± 22%  softirqs.CPU120.RCU
     15637 ± 21%    +249.0%      54573 ± 15%  softirqs.CPU121.RCU
     13994 ± 20%    +245.8%      48393 ± 13%  softirqs.CPU122.RCU
     16481 ± 17%    +240.0%      56032 ±  8%  softirqs.CPU123.RCU
     16313 ± 22%    +244.0%      56119 ± 11%  softirqs.CPU124.RCU
     15193 ± 19%    +238.4%      51414 ± 15%  softirqs.CPU125.RCU
      9449 ± 22%    +684.6%      74140 ± 29%  softirqs.CPU126.RCU
      8026 ±  7%    +341.6%      35447 ± 32%  softirqs.CPU127.RCU
      9925 ± 12%    +316.8%      41374 ± 26%  softirqs.CPU128.RCU
     10407 ± 10%    +284.2%      39989 ± 17%  softirqs.CPU129.RCU
     18780 ±  8%    +211.9%      58578 ±  8%  softirqs.CPU13.RCU
     12368 ± 14%    +234.7%      41402 ±  8%  softirqs.CPU130.RCU
     13023 ± 13%    +212.5%      40692 ± 12%  softirqs.CPU131.RCU
     12325 ± 16%    +210.6%      38280 ± 13%  softirqs.CPU132.RCU
     12901 ± 17%    +328.4%      55266 ± 25%  softirqs.CPU133.RCU
     11583 ± 18%    +274.1%      43338 ± 14%  softirqs.CPU134.RCU
     12702 ± 14%    +269.0%      46874 ± 13%  softirqs.CPU135.RCU
     14088 ± 18%    +234.1%      47075 ±  9%  softirqs.CPU136.RCU
     15252 ± 19%    +254.4%      54060 ± 17%  softirqs.CPU137.RCU
     14751 ± 21%    +247.2%      51225 ± 13%  softirqs.CPU138.RCU
     14324 ± 20%    +241.1%      48856 ±  9%  softirqs.CPU139.RCU
     16718 ± 24%    +256.5%      59604 ±  4%  softirqs.CPU14.RCU
     13748 ± 16%    +249.7%      48071 ±  9%  softirqs.CPU140.RCU
     13603 ± 22%    +276.9%      51270 ±  7%  softirqs.CPU141.RCU
     15426 ± 19%    +231.4%      51124 ±  8%  softirqs.CPU142.RCU
     14208 ± 17%    +246.0%      49156 ±  9%  softirqs.CPU143.RCU
     18660 ± 10%    +208.6%      57578 ±  7%  softirqs.CPU15.RCU
      7546 ± 10%    +338.4%      33085 ± 56%  softirqs.CPU16.RCU
      8489 ±  7%    +304.3%      34320 ± 55%  softirqs.CPU17.RCU
     15143 ± 21%   +1246.1%     203848 ± 12%  softirqs.CPU18.RCU
     40325 ±  3%     -21.5%      31673 ±  7%  softirqs.CPU18.SCHED
     12313 ± 12%    +435.6%      65947 ± 21%  softirqs.CPU19.RCU
     17395 ±  9%    +207.3%      53452 ±  9%  softirqs.CPU2.RCU
     10912 ± 12%    +233.8%      36423 ±  9%  softirqs.CPU20.RCU
     11902 ±  9%    +246.3%      41217 ± 15%  softirqs.CPU21.RCU
     14465 ± 18%    +275.2%      54269 ± 46%  softirqs.CPU22.RCU
     14245 ± 18%    +261.1%      51444 ± 24%  softirqs.CPU23.RCU
     13959 ± 21%    +162.6%      36661 ± 14%  softirqs.CPU24.RCU
     14336 ± 18%    +177.9%      39840 ± 12%  softirqs.CPU25.RCU
     12796 ± 15%    +273.2%      47753 ± 27%  softirqs.CPU26.RCU
     14604 ± 20%    +248.6%      50910 ± 34%  softirqs.CPU27.RCU
     15773 ± 19%    +181.0%      44330 ± 11%  softirqs.CPU28.RCU
     16353 ± 20%    +225.3%      53200 ± 15%  softirqs.CPU29.RCU
     16735 ± 17%    +243.5%      57482 ±  4%  softirqs.CPU3.RCU
     16013 ± 24%    +190.4%      46509 ±  6%  softirqs.CPU30.RCU
     16103 ± 21%    +242.3%      55121 ± 36%  softirqs.CPU31.RCU
      9610 ± 11%    +349.1%      43162 ± 50%  softirqs.CPU32.RCU
     11333 ± 12%    +233.9%      37847 ± 15%  softirqs.CPU33.RCU
     11324 ± 14%    +247.2%      39320 ±  9%  softirqs.CPU34.RCU
     11168 ± 15%    +210.2%      34641 ± 20%  softirqs.CPU35.RCU
     14767 ± 20%   +1293.9%     205842 ± 19%  softirqs.CPU36.RCU
     12741 ± 15%    +449.9%      70065 ± 13%  softirqs.CPU37.RCU
     11457 ± 12%    +379.8%      54980 ± 31%  softirqs.CPU38.RCU
     12440 ± 15%    +307.9%      50748 ± 16%  softirqs.CPU39.RCU
     17911 ±  9%    +184.9%      51035 ± 19%  softirqs.CPU4.RCU
     13898 ± 19%    +301.5%      55802 ± 21%  softirqs.CPU40.RCU
     14268 ± 15%    +244.9%      49204 ± 20%  softirqs.CPU41.RCU
      9776 ±  4%    +260.8%      35272 ± 24%  softirqs.CPU42.RCU
     12653 ± 13%    +275.9%      47564 ± 20%  softirqs.CPU43.RCU
     12635 ± 15%    +267.8%      46472 ± 11%  softirqs.CPU44.RCU
     14269 ± 17%    +234.5%      47725 ± 15%  softirqs.CPU45.RCU
     14850 ± 17%    +229.5%      48938 ± 14%  softirqs.CPU46.RCU
     14489 ± 20%    +235.0%      48536 ± 14%  softirqs.CPU47.RCU
     12510 ± 16%    +294.0%      49293 ± 20%  softirqs.CPU48.RCU
     11967 ±  9%    +237.0%      40330 ± 14%  softirqs.CPU49.RCU
     18541 ± 14%    +195.2%      54740 ±  6%  softirqs.CPU5.RCU
     10043 ±  8%    +296.6%      39831 ± 32%  softirqs.CPU50.RCU
     12549 ± 17%    +243.7%      43135 ±  8%  softirqs.CPU51.RCU
     13016 ± 16%    +231.6%      43156 ± 15%  softirqs.CPU52.RCU
     11202 ± 13%    +244.3%      38567 ± 19%  softirqs.CPU53.RCU
     11751 ± 24%   +1443.5%     181385 ± 11%  softirqs.CPU54.RCU
     40026 ±  2%     -17.4%      33063 ±  6%  softirqs.CPU54.SCHED
     10619 ± 13%    +557.8%      69857 ± 30%  softirqs.CPU55.RCU
     11473 ± 20%    +328.5%      49168 ± 29%  softirqs.CPU56.RCU
     10539 ± 11%    +337.0%      46052 ± 19%  softirqs.CPU57.RCU
     12270 ± 13%    +256.0%      43680 ± 16%  softirqs.CPU58.RCU
     12315 ± 14%    +246.1%      42625 ± 18%  softirqs.CPU59.RCU
     17153 ± 13%    +216.0%      54205 ±  7%  softirqs.CPU6.RCU
     11927 ± 13%    +289.7%      46480 ± 24%  softirqs.CPU60.RCU
     12446 ± 17%    +315.3%      51691 ± 48%  softirqs.CPU61.RCU
     11852 ± 17%    +267.2%      43518 ± 23%  softirqs.CPU62.RCU
     12387 ± 13%    +251.1%      43496 ± 11%  softirqs.CPU63.RCU
     10996 ± 14%    +232.5%      36563 ± 18%  softirqs.CPU64.RCU
     12143 ± 23%    +262.1%      43967 ± 16%  softirqs.CPU65.RCU
     12599 ± 19%    +274.8%      47226 ± 22%  softirqs.CPU66.RCU
     11427 ± 16%    +285.8%      44090 ± 24%  softirqs.CPU67.RCU
     10829 ± 11%    +257.9%      38760 ± 25%  softirqs.CPU68.RCU
     12273 ± 14%    +262.0%      44435 ±  9%  softirqs.CPU69.RCU
     17499 ±  8%    +227.2%      57257 ±  6%  softirqs.CPU7.RCU
     13200 ± 17%    +232.9%      43942 ±  7%  softirqs.CPU70.RCU
     10909 ± 11%    +240.2%      37115 ± 10%  softirqs.CPU71.RCU
     16827 ± 21%    +144.9%      41204        softirqs.CPU71.SCHED
     13148 ±  5%    +222.0%      42332 ±  4%  softirqs.CPU72.RCU
     12524 ± 17%    +233.6%      41781 ±  3%  softirqs.CPU73.RCU
     11992 ±  9%    +228.0%      39332 ±  4%  softirqs.CPU74.RCU
     12262 ± 10%    +224.1%      39743 ±  6%  softirqs.CPU75.RCU
     12221 ±  8%    +215.6%      38564 ±  5%  softirqs.CPU76.RCU
     12596 ±  8%    +213.9%      39538 ±  7%  softirqs.CPU77.RCU
     12097 ±  7%    +228.9%      39794 ±  4%  softirqs.CPU78.RCU
     11903 ±  7%    +218.4%      37895 ±  6%  softirqs.CPU79.RCU
     17361 ± 12%    +221.1%      55751 ±  7%  softirqs.CPU8.RCU
     12445 ±  8%    +221.8%      40052 ±  7%  softirqs.CPU80.RCU
     12004 ±  7%    +225.0%      39015 ±  4%  softirqs.CPU81.RCU
     12000 ±  6%    +229.3%      39517 ±  8%  softirqs.CPU82.RCU
     12504 ±  9%    +219.9%      40003 ±  6%  softirqs.CPU83.RCU
     12440 ±  8%    +216.1%      39328 ±  4%  softirqs.CPU84.RCU
     12262 ±  8%    +230.4%      40513 ±  7%  softirqs.CPU85.RCU
     12229 ±  9%    +225.8%      39845 ±  8%  softirqs.CPU86.RCU
     11983 ±  7%    +234.5%      40084 ±  5%  softirqs.CPU87.RCU
      7843 ±  8%    +252.7%      27661 ± 27%  softirqs.CPU88.RCU
      8306 ± 12%    +243.3%      28516 ± 23%  softirqs.CPU89.RCU
     17622 ± 11%    +230.0%      58148 ±  4%  softirqs.CPU9.RCU
     12073 ± 23%    +619.4%      86857 ± 23%  softirqs.CPU90.RCU
     10332 ± 19%    +290.2%      40318 ± 27%  softirqs.CPU91.RCU
      8983 ±  8%    +221.7%      28896 ± 17%  softirqs.CPU92.RCU
      9696 ± 10%    +239.8%      32944 ± 15%  softirqs.CPU93.RCU
     10886 ± 12%    +245.6%      37620 ± 21%  softirqs.CPU94.RCU
     10441 ± 15%    +214.6%      32849 ± 16%  softirqs.CPU95.RCU
     12834 ± 20%    +158.0%      33113 ±  9%  softirqs.CPU96.RCU
     13146 ± 17%    +159.9%      34160 ±  8%  softirqs.CPU97.RCU
     10393 ±  8%    +267.4%      38183 ± 19%  softirqs.CPU98.RCU
     12492 ± 17%    +277.9%      47207 ± 26%  softirqs.CPU99.RCU
   1898417 ± 11%    +276.1%    7139397        softirqs.RCU
     24540 ±  2%     +96.5%      48221        softirqs.TIMER
      4440 ± 54%    -100.0%       0.00        interrupts.CPU0.NMI:Non-maskable_interrupts
      4440 ± 54%    -100.0%       0.00        interrupts.CPU0.PMI:Performance_monitoring_interrupts
      2777 ±126%     -62.7%       1037 ±  6%  interrupts.CPU1.CAL:Function_call_interrupts
      6995 ± 26%    -100.0%       0.00        interrupts.CPU1.NMI:Non-maskable_interrupts
      6995 ± 26%    -100.0%       0.00        interrupts.CPU1.PMI:Performance_monitoring_interrupts
      5665 ± 38%    -100.0%       0.00        interrupts.CPU10.NMI:Non-maskable_interrupts
      5665 ± 38%    -100.0%       0.00        interrupts.CPU10.PMI:Performance_monitoring_interrupts
    149.50 ± 35%    -100.0%       0.00        interrupts.CPU100.NMI:Non-maskable_interrupts
    149.50 ± 35%    -100.0%       0.00        interrupts.CPU100.PMI:Performance_monitoring_interrupts
    118.17 ± 35%    -100.0%       0.00        interrupts.CPU101.NMI:Non-maskable_interrupts
    118.17 ± 35%    -100.0%       0.00        interrupts.CPU101.PMI:Performance_monitoring_interrupts
    162.17 ± 31%    -100.0%       0.00        interrupts.CPU102.NMI:Non-maskable_interrupts
    162.17 ± 31%    -100.0%       0.00        interrupts.CPU102.PMI:Performance_monitoring_interrupts
    141.33 ± 32%    -100.0%       0.00        interrupts.CPU103.NMI:Non-maskable_interrupts
    141.33 ± 32%    -100.0%       0.00        interrupts.CPU103.PMI:Performance_monitoring_interrupts
    152.50 ± 37%    -100.0%       0.00        interrupts.CPU104.NMI:Non-maskable_interrupts
    152.50 ± 37%    -100.0%       0.00        interrupts.CPU104.PMI:Performance_monitoring_interrupts
    151.67 ± 37%    -100.0%       0.00        interrupts.CPU105.NMI:Non-maskable_interrupts
    151.67 ± 37%    -100.0%       0.00        interrupts.CPU105.PMI:Performance_monitoring_interrupts
    152.67 ± 37%    -100.0%       0.00        interrupts.CPU106.NMI:Non-maskable_interrupts
    152.67 ± 37%    -100.0%       0.00        interrupts.CPU106.PMI:Performance_monitoring_interrupts
    203.50 ± 34%    -100.0%       0.00        interrupts.CPU107.NMI:Non-maskable_interrupts
    203.50 ± 34%    -100.0%       0.00        interrupts.CPU107.PMI:Performance_monitoring_interrupts
    223.83 ± 40%    -100.0%       0.00        interrupts.CPU108.NMI:Non-maskable_interrupts
    223.83 ± 40%    -100.0%       0.00        interrupts.CPU108.PMI:Performance_monitoring_interrupts
    174.50 ± 29%     -99.4%       1.00 ±223%  interrupts.CPU109.NMI:Non-maskable_interrupts
    174.50 ± 29%     -99.4%       1.00 ±223%  interrupts.CPU109.PMI:Performance_monitoring_interrupts
      4690 ± 55%    -100.0%       0.00        interrupts.CPU11.NMI:Non-maskable_interrupts
      4690 ± 55%    -100.0%       0.00        interrupts.CPU11.PMI:Performance_monitoring_interrupts
    147.67 ± 28%    -100.0%       0.00        interrupts.CPU110.NMI:Non-maskable_interrupts
    147.67 ± 28%    -100.0%       0.00        interrupts.CPU110.PMI:Performance_monitoring_interrupts
    149.67 ± 38%    -100.0%       0.00        interrupts.CPU111.NMI:Non-maskable_interrupts
    149.67 ± 38%    -100.0%       0.00        interrupts.CPU111.PMI:Performance_monitoring_interrupts
    133.17 ± 36%    -100.0%       0.00        interrupts.CPU112.NMI:Non-maskable_interrupts
    133.17 ± 36%    -100.0%       0.00        interrupts.CPU112.PMI:Performance_monitoring_interrupts
    160.83 ± 26%    -100.0%       0.00        interrupts.CPU113.NMI:Non-maskable_interrupts
    160.83 ± 26%    -100.0%       0.00        interrupts.CPU113.PMI:Performance_monitoring_interrupts
    129.33 ± 38%    -100.0%       0.00        interrupts.CPU114.NMI:Non-maskable_interrupts
    129.33 ± 38%    -100.0%       0.00        interrupts.CPU114.PMI:Performance_monitoring_interrupts
    147.00 ± 37%    -100.0%       0.00        interrupts.CPU115.NMI:Non-maskable_interrupts
    147.00 ± 37%    -100.0%       0.00        interrupts.CPU115.PMI:Performance_monitoring_interrupts
    112.00 ± 36%    -100.0%       0.00        interrupts.CPU116.NMI:Non-maskable_interrupts
    112.00 ± 36%    -100.0%       0.00        interrupts.CPU116.PMI:Performance_monitoring_interrupts
    133.00 ± 39%    -100.0%       0.00        interrupts.CPU117.NMI:Non-maskable_interrupts
    133.00 ± 39%    -100.0%       0.00        interrupts.CPU117.PMI:Performance_monitoring_interrupts
     98.00 ±  8%    -100.0%       0.00        interrupts.CPU118.NMI:Non-maskable_interrupts
     98.00 ±  8%    -100.0%       0.00        interrupts.CPU118.PMI:Performance_monitoring_interrupts
    131.17 ± 30%    -100.0%       0.00        interrupts.CPU119.NMI:Non-maskable_interrupts
    131.17 ± 30%    -100.0%       0.00        interrupts.CPU119.PMI:Performance_monitoring_interrupts
      5913 ± 32%    -100.0%       0.00        interrupts.CPU12.NMI:Non-maskable_interrupts
      5913 ± 32%    -100.0%       0.00        interrupts.CPU12.PMI:Performance_monitoring_interrupts
    116.83 ± 35%    -100.0%       0.00        interrupts.CPU120.NMI:Non-maskable_interrupts
    116.83 ± 35%    -100.0%       0.00        interrupts.CPU120.PMI:Performance_monitoring_interrupts
    113.00 ± 21%    -100.0%       0.00        interrupts.CPU121.NMI:Non-maskable_interrupts
    113.00 ± 21%    -100.0%       0.00        interrupts.CPU121.PMI:Performance_monitoring_interrupts
     98.83 ±  6%    -100.0%       0.00        interrupts.CPU122.NMI:Non-maskable_interrupts
     98.83 ±  6%    -100.0%       0.00        interrupts.CPU122.PMI:Performance_monitoring_interrupts
    142.00 ± 61%    -100.0%       0.00        interrupts.CPU123.NMI:Non-maskable_interrupts
    142.00 ± 61%    -100.0%       0.00        interrupts.CPU123.PMI:Performance_monitoring_interrupts
    119.17 ± 39%    -100.0%       0.00        interrupts.CPU124.NMI:Non-maskable_interrupts
    119.17 ± 39%    -100.0%       0.00        interrupts.CPU124.PMI:Performance_monitoring_interrupts
    133.67 ± 38%    -100.0%       0.00        interrupts.CPU125.NMI:Non-maskable_interrupts
    133.67 ± 38%    -100.0%       0.00        interrupts.CPU125.PMI:Performance_monitoring_interrupts
    173.17 ± 48%    -100.0%       0.00        interrupts.CPU126.NMI:Non-maskable_interrupts
    173.17 ± 48%    -100.0%       0.00        interrupts.CPU126.PMI:Performance_monitoring_interrupts
    137.00 ± 32%    -100.0%       0.00        interrupts.CPU127.NMI:Non-maskable_interrupts
    137.00 ± 32%    -100.0%       0.00        interrupts.CPU127.PMI:Performance_monitoring_interrupts
    117.83 ± 32%    -100.0%       0.00        interrupts.CPU128.NMI:Non-maskable_interrupts
    117.83 ± 32%    -100.0%       0.00        interrupts.CPU128.PMI:Performance_monitoring_interrupts
    122.33 ± 34%    -100.0%       0.00        interrupts.CPU129.NMI:Non-maskable_interrupts
    122.33 ± 34%    -100.0%       0.00        interrupts.CPU129.PMI:Performance_monitoring_interrupts
      5466 ± 50%    -100.0%       0.00        interrupts.CPU13.NMI:Non-maskable_interrupts
      5466 ± 50%    -100.0%       0.00        interrupts.CPU13.PMI:Performance_monitoring_interrupts
    101.50 ±  8%    -100.0%       0.00        interrupts.CPU130.NMI:Non-maskable_interrupts
    101.50 ±  8%    -100.0%       0.00        interrupts.CPU130.PMI:Performance_monitoring_interrupts
    116.17 ± 29%    -100.0%       0.00        interrupts.CPU131.NMI:Non-maskable_interrupts
    116.17 ± 29%    -100.0%       0.00        interrupts.CPU131.PMI:Performance_monitoring_interrupts
    215.17 ± 98%    -100.0%       0.00        interrupts.CPU132.NMI:Non-maskable_interrupts
    215.17 ± 98%    -100.0%       0.00        interrupts.CPU132.PMI:Performance_monitoring_interrupts
    114.67 ± 32%    -100.0%       0.00        interrupts.CPU133.NMI:Non-maskable_interrupts
    114.67 ± 32%    -100.0%       0.00        interrupts.CPU133.PMI:Performance_monitoring_interrupts
    164.33 ± 27%    -100.0%       0.00        interrupts.CPU134.NMI:Non-maskable_interrupts
    164.33 ± 27%    -100.0%       0.00        interrupts.CPU134.PMI:Performance_monitoring_interrupts
    118.67 ± 34%    -100.0%       0.00        interrupts.CPU135.NMI:Non-maskable_interrupts
    118.67 ± 34%    -100.0%       0.00        interrupts.CPU135.PMI:Performance_monitoring_interrupts
    131.83 ± 30%    -100.0%       0.00        interrupts.CPU136.NMI:Non-maskable_interrupts
    131.83 ± 30%    -100.0%       0.00        interrupts.CPU136.PMI:Performance_monitoring_interrupts
    118.67 ± 33%    -100.0%       0.00        interrupts.CPU137.NMI:Non-maskable_interrupts
    118.67 ± 33%    -100.0%       0.00        interrupts.CPU137.PMI:Performance_monitoring_interrupts
    122.50 ± 31%    -100.0%       0.00        interrupts.CPU138.NMI:Non-maskable_interrupts
    122.50 ± 31%    -100.0%       0.00        interrupts.CPU138.PMI:Performance_monitoring_interrupts
    100.67 ±  7%    -100.0%       0.00        interrupts.CPU139.NMI:Non-maskable_interrupts
    100.67 ±  7%    -100.0%       0.00        interrupts.CPU139.PMI:Performance_monitoring_interrupts
      5232 ± 48%    -100.0%       0.00        interrupts.CPU14.NMI:Non-maskable_interrupts
      5232 ± 48%    -100.0%       0.00        interrupts.CPU14.PMI:Performance_monitoring_interrupts
    118.33 ± 36%    -100.0%       0.00        interrupts.CPU140.NMI:Non-maskable_interrupts
    118.33 ± 36%    -100.0%       0.00        interrupts.CPU140.PMI:Performance_monitoring_interrupts
    118.00 ± 31%    -100.0%       0.00        interrupts.CPU141.NMI:Non-maskable_interrupts
    118.00 ± 31%    -100.0%       0.00        interrupts.CPU141.PMI:Performance_monitoring_interrupts
    144.83 ± 45%    -100.0%       0.00        interrupts.CPU142.NMI:Non-maskable_interrupts
    144.83 ± 45%    -100.0%       0.00        interrupts.CPU142.PMI:Performance_monitoring_interrupts
    361.83 ± 30%    -100.0%       0.00        interrupts.CPU143.NMI:Non-maskable_interrupts
    361.83 ± 30%    -100.0%       0.00        interrupts.CPU143.PMI:Performance_monitoring_interrupts
      6247 ± 35%    -100.0%       0.00        interrupts.CPU15.NMI:Non-maskable_interrupts
      6247 ± 35%    -100.0%       0.00        interrupts.CPU15.PMI:Performance_monitoring_interrupts
    425.50 ± 14%     -23.4%     325.83 ± 14%  interrupts.CPU15.TLB:TLB_shootdowns
    172.17 ± 41%    -100.0%       0.00        interrupts.CPU16.NMI:Non-maskable_interrupts
    172.17 ± 41%    -100.0%       0.00        interrupts.CPU16.PMI:Performance_monitoring_interrupts
    164.50 ± 30%    -100.0%       0.00        interrupts.CPU17.NMI:Non-maskable_interrupts
    164.50 ± 30%    -100.0%       0.00        interrupts.CPU17.PMI:Performance_monitoring_interrupts
    181.00 ± 20%    -100.0%       0.00        interrupts.CPU18.NMI:Non-maskable_interrupts
    181.00 ± 20%    -100.0%       0.00        interrupts.CPU18.PMI:Performance_monitoring_interrupts
    159.17 ± 33%    -100.0%       0.00        interrupts.CPU19.NMI:Non-maskable_interrupts
    159.17 ± 33%    -100.0%       0.00        interrupts.CPU19.PMI:Performance_monitoring_interrupts
      6587 ± 33%    -100.0%       0.00        interrupts.CPU2.NMI:Non-maskable_interrupts
      6587 ± 33%    -100.0%       0.00        interrupts.CPU2.PMI:Performance_monitoring_interrupts
    147.50 ± 28%    -100.0%       0.00        interrupts.CPU20.NMI:Non-maskable_interrupts
    147.50 ± 28%    -100.0%       0.00        interrupts.CPU20.PMI:Performance_monitoring_interrupts
    160.67 ± 23%    -100.0%       0.00        interrupts.CPU21.NMI:Non-maskable_interrupts
    160.67 ± 23%    -100.0%       0.00        interrupts.CPU21.PMI:Performance_monitoring_interrupts
    183.33 ± 17%    -100.0%       0.00        interrupts.CPU22.NMI:Non-maskable_interrupts
    183.33 ± 17%    -100.0%       0.00        interrupts.CPU22.PMI:Performance_monitoring_interrupts
    156.67 ± 38%    -100.0%       0.00        interrupts.CPU23.NMI:Non-maskable_interrupts
    156.67 ± 38%    -100.0%       0.00        interrupts.CPU23.PMI:Performance_monitoring_interrupts
    164.67 ± 26%    -100.0%       0.00        interrupts.CPU24.NMI:Non-maskable_interrupts
    164.67 ± 26%    -100.0%       0.00        interrupts.CPU24.PMI:Performance_monitoring_interrupts
    176.50 ± 26%    -100.0%       0.00        interrupts.CPU25.NMI:Non-maskable_interrupts
    176.50 ± 26%    -100.0%       0.00        interrupts.CPU25.PMI:Performance_monitoring_interrupts
    153.50 ± 33%    -100.0%       0.00        interrupts.CPU26.NMI:Non-maskable_interrupts
    153.50 ± 33%    -100.0%       0.00        interrupts.CPU26.PMI:Performance_monitoring_interrupts
    144.50 ± 26%    -100.0%       0.00        interrupts.CPU27.NMI:Non-maskable_interrupts
    144.50 ± 26%    -100.0%       0.00        interrupts.CPU27.PMI:Performance_monitoring_interrupts
    135.00 ± 37%    -100.0%       0.00        interrupts.CPU28.NMI:Non-maskable_interrupts
    135.00 ± 37%    -100.0%       0.00        interrupts.CPU28.PMI:Performance_monitoring_interrupts
    131.17 ± 29%    -100.0%       0.00        interrupts.CPU29.NMI:Non-maskable_interrupts
    131.17 ± 29%    -100.0%       0.00        interrupts.CPU29.PMI:Performance_monitoring_interrupts
      6274 ± 42%    -100.0%       0.00        interrupts.CPU3.NMI:Non-maskable_interrupts
      6274 ± 42%    -100.0%       0.00        interrupts.CPU3.PMI:Performance_monitoring_interrupts
    147.33 ± 29%    -100.0%       0.00        interrupts.CPU30.NMI:Non-maskable_interrupts
    147.33 ± 29%    -100.0%       0.00        interrupts.CPU30.PMI:Performance_monitoring_interrupts
    118.00 ± 21%    -100.0%       0.00        interrupts.CPU31.NMI:Non-maskable_interrupts
    118.00 ± 21%    -100.0%       0.00        interrupts.CPU31.PMI:Performance_monitoring_interrupts
    137.83 ± 34%    -100.0%       0.00        interrupts.CPU32.NMI:Non-maskable_interrupts
    137.83 ± 34%    -100.0%       0.00        interrupts.CPU32.PMI:Performance_monitoring_interrupts
    131.67 ± 31%    -100.0%       0.00        interrupts.CPU33.NMI:Non-maskable_interrupts
    131.67 ± 31%    -100.0%       0.00        interrupts.CPU33.PMI:Performance_monitoring_interrupts
    152.33 ± 30%    -100.0%       0.00        interrupts.CPU34.NMI:Non-maskable_interrupts
    152.33 ± 30%    -100.0%       0.00        interrupts.CPU34.PMI:Performance_monitoring_interrupts
    177.33 ± 54%    -100.0%       0.00        interrupts.CPU35.NMI:Non-maskable_interrupts
    177.33 ± 54%    -100.0%       0.00        interrupts.CPU35.PMI:Performance_monitoring_interrupts
    173.33 ± 45%    -100.0%       0.00        interrupts.CPU36.NMI:Non-maskable_interrupts
    173.33 ± 45%    -100.0%       0.00        interrupts.CPU36.PMI:Performance_monitoring_interrupts
    143.50 ± 38%    -100.0%       0.00        interrupts.CPU37.NMI:Non-maskable_interrupts
    143.50 ± 38%    -100.0%       0.00        interrupts.CPU37.PMI:Performance_monitoring_interrupts
    124.33 ± 27%    -100.0%       0.00        interrupts.CPU38.NMI:Non-maskable_interrupts
    124.33 ± 27%    -100.0%       0.00        interrupts.CPU38.PMI:Performance_monitoring_interrupts
    145.67 ± 29%    -100.0%       0.00        interrupts.CPU39.NMI:Non-maskable_interrupts
    145.67 ± 29%    -100.0%       0.00        interrupts.CPU39.PMI:Performance_monitoring_interrupts
      5122 ± 38%    -100.0%       0.00        interrupts.CPU4.NMI:Non-maskable_interrupts
      5122 ± 38%    -100.0%       0.00        interrupts.CPU4.PMI:Performance_monitoring_interrupts
    142.67 ± 29%     -99.3%       1.00 ±223%  interrupts.CPU40.NMI:Non-maskable_interrupts
    142.67 ± 29%     -99.3%       1.00 ±223%  interrupts.CPU40.PMI:Performance_monitoring_interrupts
    151.50 ± 26%    -100.0%       0.00        interrupts.CPU41.NMI:Non-maskable_interrupts
    151.50 ± 26%    -100.0%       0.00        interrupts.CPU41.PMI:Performance_monitoring_interrupts
    150.67 ± 29%    -100.0%       0.00        interrupts.CPU42.NMI:Non-maskable_interrupts
    150.67 ± 29%    -100.0%       0.00        interrupts.CPU42.PMI:Performance_monitoring_interrupts
    143.50 ± 30%    -100.0%       0.00        interrupts.CPU43.NMI:Non-maskable_interrupts
    143.50 ± 30%    -100.0%       0.00        interrupts.CPU43.PMI:Performance_monitoring_interrupts
    160.67 ± 26%    -100.0%       0.00        interrupts.CPU44.NMI:Non-maskable_interrupts
    160.67 ± 26%    -100.0%       0.00        interrupts.CPU44.PMI:Performance_monitoring_interrupts
    122.67 ± 30%    -100.0%       0.00        interrupts.CPU45.NMI:Non-maskable_interrupts
    122.67 ± 30%    -100.0%       0.00        interrupts.CPU45.PMI:Performance_monitoring_interrupts
    177.17 ± 20%    -100.0%       0.00        interrupts.CPU46.NMI:Non-maskable_interrupts
    177.17 ± 20%    -100.0%       0.00        interrupts.CPU46.PMI:Performance_monitoring_interrupts
    186.83 ± 24%    -100.0%       0.00        interrupts.CPU47.NMI:Non-maskable_interrupts
    186.83 ± 24%    -100.0%       0.00        interrupts.CPU47.PMI:Performance_monitoring_interrupts
    141.00 ± 23%    -100.0%       0.00        interrupts.CPU48.NMI:Non-maskable_interrupts
    141.00 ± 23%    -100.0%       0.00        interrupts.CPU48.PMI:Performance_monitoring_interrupts
    170.50 ± 30%    -100.0%       0.00        interrupts.CPU49.NMI:Non-maskable_interrupts
    170.50 ± 30%    -100.0%       0.00        interrupts.CPU49.PMI:Performance_monitoring_interrupts
      5397 ± 51%    -100.0%       0.00        interrupts.CPU5.NMI:Non-maskable_interrupts
      5397 ± 51%    -100.0%       0.00        interrupts.CPU5.PMI:Performance_monitoring_interrupts
    169.17 ± 23%    -100.0%       0.00        interrupts.CPU50.NMI:Non-maskable_interrupts
    169.17 ± 23%    -100.0%       0.00        interrupts.CPU50.PMI:Performance_monitoring_interrupts
    181.83 ± 19%    -100.0%       0.00        interrupts.CPU51.NMI:Non-maskable_interrupts
    181.83 ± 19%    -100.0%       0.00        interrupts.CPU51.PMI:Performance_monitoring_interrupts
    136.67 ± 22%    -100.0%       0.00        interrupts.CPU52.NMI:Non-maskable_interrupts
    136.67 ± 22%    -100.0%       0.00        interrupts.CPU52.PMI:Performance_monitoring_interrupts
    183.17 ± 19%    -100.0%       0.00        interrupts.CPU53.NMI:Non-maskable_interrupts
    183.17 ± 19%    -100.0%       0.00        interrupts.CPU53.PMI:Performance_monitoring_interrupts
    197.50 ± 47%    -100.0%       0.00        interrupts.CPU54.NMI:Non-maskable_interrupts
    197.50 ± 47%    -100.0%       0.00        interrupts.CPU54.PMI:Performance_monitoring_interrupts
    173.00 ± 32%    -100.0%       0.00        interrupts.CPU55.NMI:Non-maskable_interrupts
    173.00 ± 32%    -100.0%       0.00        interrupts.CPU55.PMI:Performance_monitoring_interrupts
    162.17 ± 22%    -100.0%       0.00        interrupts.CPU56.NMI:Non-maskable_interrupts
    162.17 ± 22%    -100.0%       0.00        interrupts.CPU56.PMI:Performance_monitoring_interrupts
    155.00 ± 27%    -100.0%       0.00        interrupts.CPU57.NMI:Non-maskable_interrupts
    155.00 ± 27%    -100.0%       0.00        interrupts.CPU57.PMI:Performance_monitoring_interrupts
    171.50 ± 19%    -100.0%       0.00        interrupts.CPU58.NMI:Non-maskable_interrupts
    171.50 ± 19%    -100.0%       0.00        interrupts.CPU58.PMI:Performance_monitoring_interrupts
    182.33 ± 16%    -100.0%       0.00        interrupts.CPU59.NMI:Non-maskable_interrupts
    182.33 ± 16%    -100.0%       0.00        interrupts.CPU59.PMI:Performance_monitoring_interrupts
      5701 ± 45%    -100.0%       1.00 ±223%  interrupts.CPU6.NMI:Non-maskable_interrupts
      5701 ± 45%    -100.0%       1.00 ±223%  interrupts.CPU6.PMI:Performance_monitoring_interrupts
    207.50 ± 44%    -100.0%       0.00        interrupts.CPU60.NMI:Non-maskable_interrupts
    207.50 ± 44%    -100.0%       0.00        interrupts.CPU60.PMI:Performance_monitoring_interrupts
    175.00 ± 25%    -100.0%       0.00        interrupts.CPU61.NMI:Non-maskable_interrupts
    175.00 ± 25%    -100.0%       0.00        interrupts.CPU61.PMI:Performance_monitoring_interrupts
    160.17 ± 25%    -100.0%       0.00        interrupts.CPU62.NMI:Non-maskable_interrupts
    160.17 ± 25%    -100.0%       0.00        interrupts.CPU62.PMI:Performance_monitoring_interrupts
    175.00 ± 18%    -100.0%       0.00        interrupts.CPU63.NMI:Non-maskable_interrupts
    175.00 ± 18%    -100.0%       0.00        interrupts.CPU63.PMI:Performance_monitoring_interrupts
    133.33 ± 34%    -100.0%       0.00        interrupts.CPU64.NMI:Non-maskable_interrupts
    133.33 ± 34%    -100.0%       0.00        interrupts.CPU64.PMI:Performance_monitoring_interrupts
    160.00 ± 23%    -100.0%       0.00        interrupts.CPU65.NMI:Non-maskable_interrupts
    160.00 ± 23%    -100.0%       0.00        interrupts.CPU65.PMI:Performance_monitoring_interrupts
    171.67 ± 24%    -100.0%       0.00        interrupts.CPU66.NMI:Non-maskable_interrupts
    171.67 ± 24%    -100.0%       0.00        interrupts.CPU66.PMI:Performance_monitoring_interrupts
    197.00 ± 19%    -100.0%       0.00        interrupts.CPU67.NMI:Non-maskable_interrupts
    197.00 ± 19%    -100.0%       0.00        interrupts.CPU67.PMI:Performance_monitoring_interrupts
    181.67 ± 19%    -100.0%       0.00        interrupts.CPU68.NMI:Non-maskable_interrupts
    181.67 ± 19%    -100.0%       0.00        interrupts.CPU68.PMI:Performance_monitoring_interrupts
    189.67 ± 23%    -100.0%       0.00        interrupts.CPU69.NMI:Non-maskable_interrupts
    189.67 ± 23%    -100.0%       0.00        interrupts.CPU69.PMI:Performance_monitoring_interrupts
      5634 ± 32%    -100.0%       0.00        interrupts.CPU7.NMI:Non-maskable_interrupts
      5634 ± 32%    -100.0%       0.00        interrupts.CPU7.PMI:Performance_monitoring_interrupts
    168.67 ± 26%    -100.0%       0.00        interrupts.CPU70.NMI:Non-maskable_interrupts
    168.67 ± 26%    -100.0%       0.00        interrupts.CPU70.PMI:Performance_monitoring_interrupts
    351.83 ± 29%    -100.0%       0.00        interrupts.CPU71.NMI:Non-maskable_interrupts
    351.83 ± 29%    -100.0%       0.00        interrupts.CPU71.PMI:Performance_monitoring_interrupts
      5467 ± 50%    -100.0%       0.00        interrupts.CPU72.NMI:Non-maskable_interrupts
      5467 ± 50%    -100.0%       0.00        interrupts.CPU72.PMI:Performance_monitoring_interrupts
      3198 ± 44%    -100.0%       0.00        interrupts.CPU73.NMI:Non-maskable_interrupts
      3198 ± 44%    -100.0%       0.00        interrupts.CPU73.PMI:Performance_monitoring_interrupts
      3548 ± 30%    -100.0%       0.00        interrupts.CPU74.NMI:Non-maskable_interrupts
      3548 ± 30%    -100.0%       0.00        interrupts.CPU74.PMI:Performance_monitoring_interrupts
      2786 ± 38%    -100.0%       0.00        interrupts.CPU75.NMI:Non-maskable_interrupts
      2786 ± 38%    -100.0%       0.00        interrupts.CPU75.PMI:Performance_monitoring_interrupts
      3942 ± 61%    -100.0%       0.00        interrupts.CPU76.NMI:Non-maskable_interrupts
      3942 ± 61%    -100.0%       0.00        interrupts.CPU76.PMI:Performance_monitoring_interrupts
      5265 ± 45%    -100.0%       1.00 ±223%  interrupts.CPU77.NMI:Non-maskable_interrupts
      5265 ± 45%    -100.0%       1.00 ±223%  interrupts.CPU77.PMI:Performance_monitoring_interrupts
      3939 ± 58%    -100.0%       0.00        interrupts.CPU78.NMI:Non-maskable_interrupts
      3939 ± 58%    -100.0%       0.00        interrupts.CPU78.PMI:Performance_monitoring_interrupts
      4860 ± 54%    -100.0%       0.00        interrupts.CPU79.NMI:Non-maskable_interrupts
      4860 ± 54%    -100.0%       0.00        interrupts.CPU79.PMI:Performance_monitoring_interrupts
      4825 ± 35%    -100.0%       0.00        interrupts.CPU8.NMI:Non-maskable_interrupts
      4825 ± 35%    -100.0%       0.00        interrupts.CPU8.PMI:Performance_monitoring_interrupts
      4414 ± 51%    -100.0%       0.00        interrupts.CPU80.NMI:Non-maskable_interrupts
      4414 ± 51%    -100.0%       0.00        interrupts.CPU80.PMI:Performance_monitoring_interrupts

44 lines not shown · raw message

17,039 lines not shown · raw message

Part of this message is not shown here. View the raw message.

#20 lore raw
In reply to Stefan Metzmacher

Hi Jens,

Quoted 22 lines
Quoted 5 lines

I know you brought this one up as part of your series, not sure I get why you want it owned by root and read-only? cmdline and exe, yeah those could be hidden, but is there really any point?

Maybe I'm missing something here, if so, do clue me in!

I looked through /proc and I think it's mostly similar to the unshare() case, if userspace wants to do stupid things like changing "comm" of iothreads, it gets what was asked for.

But the "cmdline" hiding would be very useful.

While most tools use "comm", by default.

ps -eLf or 'iotop' use "cmdline".

Some processes use setproctitle to change "cmdline" in order to identify the process better, without the 15 chars comm restriction, that's why I very often press 'c' in 'top' to see the cmdline, in that case it would be very helpful to see '[iou-wrk-1234]' instead of the seeing the cmdline.

Quoted 4 lines

So I'd very much prefer if this could be applied:
https://lore.kernel.org/io-uring/d4487f959c778d0b1d4c5738b75bcff17d21df5b.1616197787.git.metze@samba.org/T/#u

If you want I can add a comment and a more verbose commit message...

I noticed that 'iotop' actually appends ' [iou-wrk-1234]' to the cmdline value, so that leaves us with 'ps -eLf' and 'top' (with 'c').

pstree -a -t -p is also fine:

      │   └─io_uring-cp,1315 /root/kernel/linux-image-5.12.0-rc2+-dbg_5.12.0-rc2+-5_amd64.deb file
      │       ├─{iou-mgr-1315},1316
      │       ├─{iou-wrk-1315},1317
      │       ├─{iou-wrk-1315},1318
      │       ├─{iou-wrk-1315},1319
      │       ├─{iou-wrk-1315},1320

In the spirit of "avoid special PF_IO_WORKER checks" I guess it's ok to leave of as is...

metze