Don't show PF_IO_WORKER in /proc/<pid>/task/
Subsystems (7)
Jens Axboe
maintainercover Don't show PF_IO_WORKER in /proc/<pid>/task/
Hi,
Stefan reports that attaching to a task with io_uring will leave gdb very confused and just repeatedly attempting to attach to the IO threads, even though it receives an -EPERM every time. This patchset proposes to skip PF_IO_WORKER threads as same_thread_group(), except for accounting purposes which we still desire.
We also skip listing the IO threads in /proc/<pid>/task/ so that gdb doesn't think it should stop and attach to them. This makes us consistent with earlier kernels, where these async threads were not related to the ring owning task, and hence gdb (and others) ignored them anyway.
Seems to me that this is the right approach, but open to comments on if others agree with this. Oleg, I did see your messages as well on SIGSTOP, and as was discussed with Eric as well, this is something we most certainly can revisit. I do think that the visibility of these threads is a separate issue. Even with SIGSTOP implemented (which I did try as well), we're never going to allow ptrace attach and hence gdb would still be broken. Hence I'd rather treat them as separate issues to attack.
Signature · 1 line
Jens Axboe
Jens Axboe
maintainerPATCH 1/2 kernel: don't include PF_IO_WORKERs as part of same_thread_group()
Don't pretend that the IO threads are in the same thread group, the only case where that seems to be desired is for accounting purposes. Add a special accounting function for that and make the scheduler side use it.
For signals and ptrace, we don't allow them to be treated as threads anyway.
--- include/linux/sched/signal.h | 9 ++++++++- kernel/sched/cputime.c | 2 +- 2 files changed, 9 insertions(+), 2 deletions(-)
include/linux/sched/signal.h +8 -1
@@ -668,11 +668,18 @@ static inline bool thread_group_leader(struct task_struct *p) }static inline -bool same_thread_group (struct task_struct *p1,struct task_struct *p2) +bool same_thread_group_account (struct task_struct *p1,struct task_struct *p2) {return p1->signal == p2->signal; } +static inline +bool same_thread_group (struct task_struct *p1,struct task_struct *p2) +{ +return same_thread_group_account (p1, p2) && + !((p1->flags | p2->flags) &PF_IO_WORKER ); +} +static inline struct task_struct *next_thread (const struct task_struct *p) { return list_entry_rcu(p->thread_group.next,
kernel/sched/cputime.c +1 -1
@@ -307,7 +307,7 @@ void thread_group_cputime(struct task_struct *tsk, struct task_cputime *times)* those pending times and rely only on values updated on tick or * other scheduler action. */ -if (same_thread_group (current, tsk)) +if (same_thread_group_account (current, tsk)) (void )task_sched_runtime (current);rcu_read_lock ();
Signature · 1 line
2.31.0
Jens Axboe
maintainerPATCH 2/2 proc: don't show PF_IO_WORKER threads as threads in /proc/<pid>/task/
In reply to Jens AxboeWe don't allow SIGSTOP and ptrace attach to these threads, and that confuses applications like gdb that assume they can attach to any thread listed in /proc/<pid>/task/. gdb then enters an infinite loop of retrying attach, even though it fails with the same error (-EPERM) every time.
Skip over PF_IO_WORKER threads in the proc task setup. We can't just terminate the when we find a PF_IO_WORKER thread, as there's no real ordering here. It's perfectly feasible to have the first thread be an IO worker, and then a real thread after that. Hence just implement the skip.
--- fs/proc/base.c | 22 ++++++++++++++-------- 1 file changed, 14 insertions(+), 8 deletions(-)
fs/proc/base.c +14 -8
@@ -3723,7 +3723,7 @@ static struct task_struct *first_tid(struct pid *pid, int tid, loff_t f_pos,*/ pos = task = task->group_leader; do { - if (!nr--) + if (same_thread_group(task, pos) && !nr--) goto found; } while_each_thread(task, pos); fail:
@@ -3744,16 +3744,22 @@ static struct task_struct *first_tid(struct pid *pid, int tid, loff_t f_pos,*/ static struct task_struct *next_tid (struct task_struct *start) { -struct task_struct *pos =NULL ; +struct task_struct *tmp, *pos =NULL ; +rcu_read_lock (); -if (pid_alive (start)) { - pos =next_thread (start); -if (thread_group_leader (pos)) - pos =NULL ; -else -get_task_struct (pos); +if (!pid_alive (start)) + goto no_thread; +list_for_each_entry_rcu (tmp, &start->thread_group, thread_group) { +if (!thread_group_leader (tmp) &&same_thread_group (start, tmp)) { +get_task_struct (tmp); + pos = tmp; +break ; + } } +no_thread:rcu_read_unlock (); +if (!pos) +return NULL ;put_task_struct (start);return pos; }
Signature · 1 line
2.31.0
kernel test robot
build bot[proc] 43b2a76b1a: will-it-scale.per_process_ops -11.3% regression
FYI, we noticed a -11.3% regression of will-it-scale.per_process_ops due to commit:
Greeting,
FYI, we noticed a -11.3% regression of will-it-scale.per_process_ops due to commit:
commit: 43b2a76b1a5abcc9833b463bef137d35cbb85cdd ("[PATCH 2/2] proc: don't show PF_IO_WORKER threads as threads in /proc/<pid>/task/")
url: https://github.com/0day-ci/linux/commits/Jens-Axboe/Don-t-show-PF_IO_WORKER-in-proc-pid-task/20210326-004554
base: https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git a74e6a014c9d4d4161061f770c9b4f98372ac778
in testcase: will-it-scale
on test machine: 192 threads Intel(R) Xeon(R) Platinum 9242 CPU @ 2.30GHz with 192G memory with following parameters:
nr_task: 16 mode: process test: eventfd1 cpufreq_governor: performance ucode: 0x5003006
test-description: Will It Scale takes a testcase and runs it from 1 through to n parallel copies to see if the testcase will scale. It builds both a process and threads based test in order to see any differences between the two.
test-url: https://github.com/antonblanchard/will-it-scale
In addition to that, the commit also has significant impact on the following tests:
+------------------+---------------------------------------------------------------------------+ | testcase: change | will-it-scale: will-it-scale.per_process_ops -11.1% regression | | test machine | 104 threads Skylake with 192G memory | | test parameters | cpufreq_governor=performance | | | mode=process | | | nr_task=16 | | | test=malloc1 | | | ucode=0x2006a0a | +------------------+---------------------------------------------------------------------------+ | testcase: change | will-it-scale: will-it-scale.per_process_ops -6.3% regression | | test machine | 104 threads Skylake with 192G memory | | test parameters | cpufreq_governor=performance | | | mode=process | | | nr_task=100% | | | test=eventfd1 | | | ucode=0x2006a0a | +------------------+---------------------------------------------------------------------------+ | testcase: change | will-it-scale: will-it-scale.per_thread_ops -2.2% regression | | test machine | 48 threads Intel(R) Xeon(R) CPU E5-2697 v2 @ 2.70GHz with 112G memory | | test parameters | cpufreq_governor=performance | | | mode=thread | | | nr_task=100% | | | test=poll1 | | | ucode=0x42e | +------------------+---------------------------------------------------------------------------+ | testcase: change | will-it-scale: will-it-scale.per_process_ops -2.6% regression | | test machine | 48 threads Intel(R) Xeon(R) CPU E5-2697 v2 @ 2.70GHz with 112G memory | | test parameters | cpufreq_governor=performance | | | mode=process | | | nr_task=50% | | | test=pread1 | | | ucode=0x42e | +------------------+---------------------------------------------------------------------------+ | testcase: change | will-it-scale: will-it-scale.per_process_ops -15.3% regression | | test machine | 104 threads Skylake with 192G memory | | test parameters | cpufreq_governor=performance | | | mode=process | | | nr_task=50% | | | test=brk1 | | | ucode=0x2006a0a | +------------------+---------------------------------------------------------------------------+ | testcase: change | will-it-scale: will-it-scale.per_thread_ops -7.1% regression | | test machine | 144 threads Intel(R) Xeon(R) CPU E7-8890 v3 @ 2.50GHz with 512G memory | | test parameters | cpufreq_governor=performance | | | mode=thread | | | nr_task=16 | | | test=pipe1 | | | ucode=0x16 | +------------------+---------------------------------------------------------------------------+ | testcase: change | will-it-scale: will-it-scale.per_process_ops -21.4% regression | | test machine | 192 threads Intel(R) Xeon(R) Platinum 9242 CPU @ 2.30GHz with 192G memory | | test parameters | cpufreq_governor=performance | | | mode=process | | | nr_task=100% | | | test=eventfd1 | | | ucode=0x5003006 | +------------------+---------------------------------------------------------------------------+ | testcase: change | unixbench: unixbench.score -4.4% regression | | test machine | 96 threads Intel(R) Xeon(R) CPU @ 2.30GHz with 128G memory | | test parameters | cpufreq_governor=performance | | | nr_task=1 | | | runtime=300s | | | test=pipe | | | ucode=0x4003006 | +------------------+---------------------------------------------------------------------------+ | testcase: change | will-it-scale: will-it-scale.per_process_ops -20.3% regression | | test machine | 104 threads Skylake with 192G memory | | test parameters | cpufreq_governor=performance | | | mode=process | | | nr_task=16 | | | test=open1 | | | ucode=0x2006a0a | +------------------+---------------------------------------------------------------------------+ | testcase: change | will-it-scale: will-it-scale.per_thread_ops -17.7% regression | | test machine | 144 threads Intel(R) Xeon(R) Gold 5318H CPU @ 2.50GHz with 128G memory | | test parameters | cpufreq_governor=performance | | | mode=thread | | | nr_task=50% | | | test=readseek1 | | | ucode=0x700001e | +------------------+---------------------------------------------------------------------------+ | testcase: change | will-it-scale: will-it-scale.per_thread_ops -11.2% regression | | test machine | 192 threads Intel(R) Xeon(R) Platinum 9242 CPU @ 2.30GHz with 192G memory | | test parameters | cpufreq_governor=performance | | | mode=thread | | | nr_task=16 | | | test=eventfd1 | | | ucode=0x5003006 | +------------------+---------------------------------------------------------------------------+ | testcase: change | will-it-scale: will-it-scale.per_thread_ops -1.1% regression | | test machine | 48 threads Intel(R) Xeon(R) CPU E5-2697 v2 @ 2.70GHz with 112G memory | | test parameters | cpufreq_governor=performance | | | mode=thread | | | nr_task=50% | | | test=pthread_mutex2 | | | ucode=0x42e | +------------------+---------------------------------------------------------------------------+ | testcase: change | will-it-scale: will-it-scale.per_thread_ops -6.7% regression | | test machine | 104 threads Skylake with 192G memory | | test parameters | cpufreq_governor=performance | | | mode=thread | | | nr_task=100% | | | test=eventfd1 | | | ucode=0x2006a0a | +------------------+---------------------------------------------------------------------------+ | testcase: change | will-it-scale: will-it-scale.per_process_ops -11.2% regression | | test machine | 104 threads Skylake with 192G memory | | test parameters | cpufreq_governor=performance | | | mode=process | | | nr_task=50% | | | test=lock1 | | | ucode=0x2006a0a | +------------------+---------------------------------------------------------------------------+ | testcase: change | will-it-scale: will-it-scale.per_thread_ops 3.3% improvement | | test machine | 144 threads Intel(R) Xeon(R) CPU E7-8890 v3 @ 2.50GHz with 512G memory | | test parameters | cpufreq_governor=performance | | | mode=thread | | | nr_task=50% | | | test=sched_yield | | | ucode=0x16 | +------------------+---------------------------------------------------------------------------+ | testcase: change | fxmark: fxmark.hdd_ext4_MWRL_9_bufferedio.works/sec -11.4% regression | | test machine | 288 threads Intel(R) Xeon Phi(TM) CPU 7295 @ 1.50GHz with 80G memory | | test parameters | cpufreq_governor=performance | | | directio=bufferedio | | | disk=1HDD | | | fstype=ext4 | | | media=hdd | | | test=MWRL | | | ucode=0x11 | +------------------+---------------------------------------------------------------------------+ | testcase: change | fxmark: fxmark.hdd_btrfs_MWRL_9_bufferedio.works/sec -31.7% regression | | test machine | 288 threads Intel(R) Xeon Phi(TM) CPU 7295 @ 1.50GHz with 80G memory | | test parameters | cpufreq_governor=performance | | | directio=bufferedio | | | disk=1HDD | | | fstype=btrfs | | | media=hdd | | | test=MWRL | | | ucode=0x11 | +------------------+---------------------------------------------------------------------------+ | testcase: change | stress-ng: stress-ng.netdev.ops_per_sec -16.5% regression | | test machine | 96 threads Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz with 192G memory | | test parameters | class=network | | | cpufreq_governor=performance | | | disk=1HDD | | | nr_threads=100% | | | test=netdev | | | testtime=60s | | | ucode=0x5003006 | +------------------+---------------------------------------------------------------------------+ | testcase: change | fxmark: fxmark.hdd_xfs_DRBH_2_directio.works/sec -14.3% regression | | test machine | 288 threads Intel(R) Xeon Phi(TM) CPU 7295 @ 1.50GHz with 80G memory | | test parameters | cpufreq_governor=performance | | | directio=directio | | | disk=1HDD | | | fstype=xfs | | | media=hdd | | | test=DRBH | | | ucode=0x11 | +------------------+---------------------------------------------------------------------------+ | testcase: change | stress-ng: stress-ng.fanotify.ops_per_sec 192.8% improvement | | test machine | 48 threads Intel(R) Xeon(R) CPU E5-2697 v2 @ 2.70GHz with 112G memory | | test parameters | class=filesystem | | | cpufreq_governor=performance | | | disk=1HDD | | | fs=btrfs | | | nr_threads=10% | | | test=fanotify | | | testtime=60s | | | ucode=0x42e | +------------------+---------------------------------------------------------------------------+ | testcase: change | fxmark: fxmark.hdd_xfs_MWCM_9_bufferedio.works/sec -9.0% regression | | test machine | 288 threads Intel(R) Xeon Phi(TM) CPU 7295 @ 1.50GHz with 80G memory | | test parameters | cpufreq_governor=performance | | | directio=bufferedio | | | disk=1HDD | | | fstype=xfs | | | media=hdd | | | test=MWCM | | | ucode=0x11 | +------------------+---------------------------------------------------------------------------+
If you fix the issue, kindly add following tag
Details are as below:
-------------------------------------------------------------------------------------------------->
To reproduce:
git clone https://github.com/intel/lkp-tests.git
cd lkp-tests
bin/lkp install job.yaml # job file is attached in this email
bin/lkp split-job --compatible job.yaml
bin/lkp run compatible-job.yaml
=========================================================================================
compiler/cpufreq_governor/kconfig/mode/nr_task/rootfs/tbox_group/test/testcase/ucode:
gcc-9/performance/x86_64-rhel-8.3/process/16/debian-10.4-x86_64-20200603.cgz/lkp-csl-2ap2/eventfd1/will-it-scale/0x5003006
commit:
dde38d5b80 ("kernel: don't include PF_IO_WORKERs as part of same_thread_group()") 43b2a76b1a ("proc: don't show PF_IO_WORKER threads as threads in /proc/<pid>/task/")
dde38d5b805358c7 43b2a76b1a5abcc9833b463bef1
---------------- ---------------------------
%stddev %change %stddev
\ | \
47173522 -11.3% 41822683 will-it-scale.16.processes
2948344 -11.3% 2613917 will-it-scale.per_process_ops
47173522 -11.3% 41822683 will-it-scale.workload
334392 ± 62% +222.6% 1078700 ± 43% numa-numastat.node3.local_node
359226 ± 51% +217.8% 1141654 ± 38% numa-numastat.node3.numa_hit
1.63 ± 20% -0.6 1.01 ± 8% mpstat.cpu.all.irq%
6.70 +0.8 7.53 mpstat.cpu.all.sys%
1.41 -0.1 1.27 mpstat.cpu.all.usr%
1387381 +194.5% 4086263 ± 5% vmstat.memory.cache
15.17 ± 2% +14.3% 17.33 ± 2% vmstat.procs.r
2193 ± 3% +9.6% 2402 vmstat.system.cs
1429848 ± 16% +26.2% 1804776 ± 12% sched_debug.cfs_rq:/.spread0.max
566.65 +74.3% 987.56 sched_debug.cfs_rq:/.util_est_enqueued.max
331.80 ± 3% +11.2% 368.82 sched_debug.cpu.curr->pid.avg
0.09 ± 4% +11.1% 0.10 sched_debug.cpu.nr_running.avg
54610 ± 3% +4930.8% 2747384 ± 7% meminfo.Active
54610 ± 3% +4930.8% 2747384 ± 7% meminfo.Active(anon)
173070 +13.0% 195601 meminfo.AnonHugePages
265559 +15.2% 305891 meminfo.AnonPages
1256498 +214.2% 3948319 ± 5% meminfo.Cached
509316 +545.4% 3287375 ± 6% meminfo.Committed_AS
285355 +13.8% 324656 meminfo.Inactive
285355 +13.8% 324656 meminfo.Inactive(anon)
3296220 +104.7% 6746177 ± 3% meminfo.Memused
4791 ± 3% +16.9% 5602 ± 2% meminfo.PageTables
75032 ± 2% +3587.6% 2766852 ± 7% meminfo.Shmem
4299183 ± 3% +57.3% 6764473 ± 3% meminfo.max_used_kB
179.17 ±158% +4225.3% 7749 ±160% numa-vmstat.node0.nr_active_anon
179.17 ±158% +4225.3% 7749 ±160% numa-vmstat.node0.nr_zone_active_anon
196.17 ±129% +95954.0% 188425 ± 98% numa-vmstat.node1.nr_active_anon
73277 ± 2% +258.3% 262578 ± 70% numa-vmstat.node1.nr_file_pages
2075 ± 82% +9064.7% 190228 ± 97% numa-vmstat.node1.nr_shmem
196.17 ±129% +95954.0% 188425 ± 98% numa-vmstat.node1.nr_zone_active_anon
254.67 ±134% +67712.8% 172696 ± 70% numa-vmstat.node2.nr_active_anon
76891 ± 4% +221.5% 247214 ± 49% numa-vmstat.node2.nr_file_pages
455.33 ±108% +38042.8% 173677 ± 70% numa-vmstat.node2.nr_shmem
254.67 ±134% +67712.8% 172696 ± 70% numa-vmstat.node2.nr_zone_active_anon
12988 ± 5% +2347.1% 317842 ± 58% numa-vmstat.node3.nr_active_anon
88291 ± 3% +344.8% 392756 ± 47% numa-vmstat.node3.nr_file_pages
14154 ± 8% +2154.8% 319144 ± 58% numa-vmstat.node3.nr_shmem
12988 ± 5% +2347.1% 317842 ± 58% numa-vmstat.node3.nr_zone_active_anon
707483 ± 24% +53.0% 1082663 ± 30% numa-vmstat.node3.numa_hit
717.67 ±158% +4214.9% 30966 ±160% numa-meminfo.node0.Active
717.67 ±158% +4214.9% 30966 ±160% numa-meminfo.node0.Active(anon)
898016 ± 7% +18.0% 1059629 ± 8% numa-meminfo.node0.MemUsed
785.33 ±129% +95815.7% 753258 ± 98% numa-meminfo.node1.Active
785.33 ±129% +95815.7% 753258 ± 98% numa-meminfo.node1.Active(anon)
293110 ± 2% +258.2% 1049866 ± 70% numa-meminfo.node1.FilePages
737028 ± 9% +128.1% 1681518 ± 47% numa-meminfo.node1.MemUsed
8303 ± 82% +9058.8% 760466 ± 97% numa-meminfo.node1.Shmem
1019 ±134% +67579.3% 690216 ± 70% numa-meminfo.node2.Active
1019 ±134% +67579.3% 690216 ± 70% numa-meminfo.node2.Active(anon)
94946 ± 62% +232.2% 315437 ± 45% numa-meminfo.node2.AnonPages.max
307567 ± 4% +221.3% 988285 ± 49% numa-meminfo.node2.FilePages
808971 ± 8% +111.6% 1712135 ± 29% numa-meminfo.node2.MemUsed
1822 ±108% +37980.2% 694137 ± 70% numa-meminfo.node2.Shmem
51989 ± 5% +2342.9% 1270067 ± 58% numa-meminfo.node3.Active
51989 ± 5% +2342.9% 1270067 ± 58% numa-meminfo.node3.Active(anon)
353216 ± 3% +344.4% 1569725 ± 47% numa-meminfo.node3.FilePages
853866 ± 11% +168.2% 2290171 ± 35% numa-meminfo.node3.MemUsed
56665 ± 8% +2150.6% 1275276 ± 58% numa-meminfo.node3.Shmem
13627 ± 3% +4930.0% 685442 ± 7% proc-vmstat.nr_active_anon
66425 +15.2% 76490 proc-vmstat.nr_anon_pages
4835055 -1.8% 4749110 proc-vmstat.nr_dirty_background_threshold
9681934 -1.8% 9509833 proc-vmstat.nr_dirty_threshold
314100 +213.8% 985675 ± 5% proc-vmstat.nr_file_pages
48604271 -1.8% 47743555 proc-vmstat.nr_free_pages
71374 +13.7% 81180 proc-vmstat.nr_inactive_anon
9741 +1.6% 9892 proc-vmstat.nr_mapped
1203 ± 2% +16.7% 1403 ± 2% proc-vmstat.nr_page_table_pages
18733 ± 2% +3584.9% 690308 ± 7% proc-vmstat.nr_shmem
72469 +7.5% 77891 ± 2% proc-vmstat.nr_slab_unreclaimable
13627 ± 3% +4930.0% 685442 ± 7% proc-vmstat.nr_zone_active_anon
71374 +13.7% 81180 proc-vmstat.nr_zone_inactive_anon
1243541 +141.3% 3000844 ± 3% proc-vmstat.numa_hit
983775 +178.6% 2741036 ± 4% proc-vmstat.numa_local
43874 ± 21% +65.0% 72400 ± 30% proc-vmstat.numa_pte_updates
18545 ± 3% +398.9% 92528 ± 6% proc-vmstat.pgactivate
1320163 +224.0% 4277306 ± 5% proc-vmstat.pgalloc_normal
1176045 -2.5% 1146897 proc-vmstat.pgfault
1325260 ± 5% +101.6% 2671978 ± 6% proc-vmstat.pgfree
2999 ± 5% -10.0% 2698 ± 7% slabinfo.PING.active_objs
2999 ± 5% -10.0% 2698 ± 7% slabinfo.PING.num_objs
73711 ± 6% +27.9% 94307 ± 4% slabinfo.filp.active_objs
1160 ± 6% +28.1% 1486 ± 4% slabinfo.filp.active_slabs
74267 ± 6% +28.1% 95154 ± 4% slabinfo.filp.num_objs
1160 ± 6% +28.1% 1486 ± 4% slabinfo.filp.num_slabs
28063 ± 2% +17.3% 32913 ± 5% slabinfo.kmalloc-512.active_objs
28448 ± 2% +16.8% 33214 ± 4% slabinfo.kmalloc-512.num_objs
21843 ± 2% -27.7% 15791 ± 3% slabinfo.proc_inode_cache.active_objs
458.33 ± 3% -23.3% 351.67 ± 4% slabinfo.proc_inode_cache.active_slabs
22024 ± 3% -23.2% 16912 ± 4% slabinfo.proc_inode_cache.num_objs
458.33 ± 3% -23.3% 351.67 ± 4% slabinfo.proc_inode_cache.num_slabs
27129 +38.3% 37516 ± 2% slabinfo.radix_tree_node.active_objs
484.00 +38.3% 669.50 ± 2% slabinfo.radix_tree_node.active_slabs
27129 +38.3% 37516 ± 2% slabinfo.radix_tree_node.num_objs
484.00 +38.3% 669.50 ± 2% slabinfo.radix_tree_node.num_slabs
5470 ± 5% +9.7% 6000 ± 5% slabinfo.signal_cache.active_objs
5506 ± 6% +9.0% 6000 ± 5% slabinfo.signal_cache.num_objs
2158 +20.5% 2601 ± 7% slabinfo.task_struct.active_objs
2172 +20.4% 2616 ± 7% slabinfo.task_struct.active_slabs
2172 +20.4% 2616 ± 7% slabinfo.task_struct.num_objs
2172 +20.4% 2616 ± 7% slabinfo.task_struct.num_slabs
16909 ± 10% -19.8% 13563 ± 6% softirqs.CPU10.RCU
14967 ± 7% -15.2% 12694 ± 2% softirqs.CPU104.RCU
17878 ± 6% -17.9% 14671 ± 8% softirqs.CPU12.RCU
16947 ± 14% -18.3% 13844 ± 5% softirqs.CPU13.RCU
17862 ± 14% -18.7% 14521 ± 3% softirqs.CPU14.RCU
13535 ± 18% +114.7% 29055 ± 34% softirqs.CPU144.RCU
12200 ± 17% -16.2% 10227 ± 8% softirqs.CPU163.RCU
12305 ± 13% +140.2% 29562 ± 18% softirqs.CPU168.RCU
12133 ± 12% -18.8% 9856 ± 8% softirqs.CPU171.RCU
12087 ± 12% -19.3% 9753 ± 8% softirqs.CPU174.RCU
14277 ± 15% -24.5% 10781 ± 17% softirqs.CPU185.RCU
13292 ± 16% -19.2% 10746 ± 16% softirqs.CPU186.RCU
13156 ± 16% -20.7% 10435 ± 16% softirqs.CPU187.RCU
17562 ± 11% -22.5% 13605 ± 8% softirqs.CPU2.RCU
16919 ± 11% -19.1% 13690 ± 8% softirqs.CPU3.RCU
17426 ± 10% -19.4% 14052 ± 8% softirqs.CPU4.RCU
13629 ± 19% +171.3% 36969 ± 32% softirqs.CPU48.RCU
40218 ± 4% -12.1% 35333 ± 8% softirqs.CPU48.SCHED
17180 ± 10% -19.8% 13783 ± 6% softirqs.CPU5.RCU
17300 ± 12% -18.4% 14110 ± 8% softirqs.CPU6.RCU
12435 ± 16% -17.2% 10295 ± 9% softirqs.CPU67.RCU
16439 ± 13% -20.2% 13110 ± 8% softirqs.CPU7.RCU
12481 ± 14% +229.3% 41098 ± 16% softirqs.CPU72.RCU
39776 ± 3% -19.1% 32174 ± 8% softirqs.CPU72.SCHED
16186 ± 10% -20.8% 12820 ± 7% softirqs.CPU9.RCU
18637 ± 4% +96.3% 36588 ± 17% softirqs.CPU95.SCHED
1.17 ± 5% -0.1 1.03 ± 2% perf-stat.i.branch-miss-rate%
1.92e+08 ± 5% -12.3% 1.685e+08 ± 2% perf-stat.i.branch-misses
13.96 ± 72% +16.8 30.78 ± 22% perf-stat.i.cache-miss-rate%
4525225 ± 10% +331.5% 19527153 ± 19% perf-stat.i.cache-misses
2145 ± 3% +9.8% 2356 perf-stat.i.context-switches
0.69 +15.5% 0.80 perf-stat.i.cpi
5.64e+10 +14.7% 6.472e+10 perf-stat.i.cpu-cycles
12806 ± 12% -57.3% 5465 ± 15% perf-stat.i.cycles-between-cache-misses
2.373e+10 -3.9% 2.28e+10 perf-stat.i.dTLB-loads
1.588e+10 -4.5% 1.516e+10 perf-stat.i.dTLB-stores
98.13 -0.8 97.32 perf-stat.i.iTLB-load-miss-rate%
1.763e+08 -9.3% 1.6e+08 perf-stat.i.iTLB-load-misses
463.51 +9.5% 507.53 perf-stat.i.instructions-per-iTLB-miss
1.44 -13.4% 1.25 perf-stat.i.ipc
1.39 ± 5% -84.1% 0.22 ± 84% perf-stat.i.major-faults
0.29 +14.7% 0.34 perf-stat.i.metric.GHz
292.33 -3.1% 283.30 perf-stat.i.metric.M/sec
3778 -2.6% 3681 perf-stat.i.minor-faults
91.63 +4.1 95.73 perf-stat.i.node-load-miss-rate%
437707 ± 55% +1002.0% 4823604 ± 27% perf-stat.i.node-load-misses
39983 ± 39% +313.0% 165142 ± 20% perf-stat.i.node-loads
94366 ± 21% +1359.2% 1376997 ± 17% perf-stat.i.node-store-misses
3780 -2.6% 3681 perf-stat.i.page-faults
1.17 ± 5% -0.1 1.03 ± 2% perf-stat.overall.branch-miss-rate%
0.69 +15.6% 0.80 perf-stat.overall.cpi
12633 ± 12% -72.6% 3457 ± 21% perf-stat.overall.cycles-between-cache-misses
98.18 -0.8 97.37 perf-stat.overall.iTLB-load-miss-rate%
462.23 +9.4% 505.87 perf-stat.overall.instructions-per-iTLB-miss
1.45 -13.5% 1.25 perf-stat.overall.ipc
90.92 +5.4 96.31 perf-stat.overall.node-load-miss-rate%
83.08 ± 13% +13.9 96.99 perf-stat.overall.node-store-miss-rate%
520246 +12.0% 582888 perf-stat.overall.path-length
1.914e+08 ± 5% -12.3% 1.679e+08 ± 2% perf-stat.ps.branch-misses
4510225 ± 10% +331.6% 19466927 ± 19% perf-stat.ps.cache-misses
2138 ± 3% +9.8% 2348 perf-stat.ps.context-switches
5.621e+10 +14.7% 6.45e+10 perf-stat.ps.cpu-cycles
197.85 -1.4% 195.09 perf-stat.ps.cpu-migrations
2.365e+10 -3.9% 2.272e+10 perf-stat.ps.dTLB-loads
1.583e+10 -4.5% 1.511e+10 perf-stat.ps.dTLB-stores
1.757e+08 -9.3% 1.594e+08 perf-stat.ps.iTLB-load-misses
1.38 ± 5% -84.1% 0.22 ± 84% perf-stat.ps.major-faults
3765 -2.6% 3668 perf-stat.ps.minor-faults
436378 ± 55% +1002.0% 4808882 ± 27% perf-stat.ps.node-load-misses
39914 ± 39% +312.5% 164641 ± 20% perf-stat.ps.node-loads
94069 ± 21% +1359.2% 1372706 ± 17% perf-stat.ps.node-store-misses
3766 -2.6% 3668 perf-stat.ps.page-faults
37.68 ± 15% -37.7 0.00 perf-profile.calltrace.cycles-pp.secondary_startup_64_no_verify
36.34 ± 14% -36.3 0.00 perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64_no_verify
36.34 ± 14% -36.3 0.00 perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
36.34 ± 14% -36.3 0.00 perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
35.40 ± 9% -35.4 0.00 perf-profile.calltrace.cycles-pp.read
35.16 ± 15% -35.2 0.00 perf-profile.calltrace.cycles-pp.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
34.65 ± 14% -34.7 0.00 perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary
33.00 ± 17% -33.0 0.00 perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry
30.91 ± 9% -30.9 0.00 perf-profile.calltrace.cycles-pp.write
23.94 ± 9% -23.9 0.00 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.read
21.78 ± 9% -21.8 0.00 perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.read
20.85 ± 9% -20.9 0.00 perf-profile.calltrace.cycles-pp.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe.read
19.55 ± 9% -19.6 0.00 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.write
18.44 ± 9% -18.4 0.00 perf-profile.calltrace.cycles-pp.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe.read
17.39 ± 9% -17.4 0.00 perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.write
16.48 ± 9% -16.5 0.00 perf-profile.calltrace.cycles-pp.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.write
14.08 ± 9% -14.1 0.00 perf-profile.calltrace.cycles-pp.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.write
10.62 ± 9% -10.6 0.00 perf-profile.calltrace.cycles-pp.new_sync_read.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe
8.92 ± 9% -8.9 0.00 perf-profile.calltrace.cycles-pp.eventfd_read.new_sync_read.vfs_read.ksys_read.do_syscall_64
7.90 ± 9% -7.9 0.00 perf-profile.calltrace.cycles-pp.eventfd_write.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
6.58 ± 9% -6.6 0.00 perf-profile.calltrace.cycles-pp.__entry_text_start.write
6.55 ± 8% -6.6 0.00 perf-profile.calltrace.cycles-pp.__entry_text_start.read
5.20 ± 9% -5.2 0.00 perf-profile.calltrace.cycles-pp._copy_to_iter.eventfd_read.new_sync_read.vfs_read.ksys_read
43.75 ± 9% -43.8 0.00 perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
39.43 ± 9% -39.4 0.00 perf-profile.children.cycles-pp.do_syscall_64
37.68 ± 15% -37.7 0.00 perf-profile.children.cycles-pp.secondary_startup_64_no_verify
37.68 ± 15% -37.7 0.00 perf-profile.children.cycles-pp.cpu_startup_entry
37.68 ± 15% -37.7 0.00 perf-profile.children.cycles-pp.do_idle
36.51 ± 15% -36.5 0.00 perf-profile.children.cycles-pp.cpuidle_enter
36.51 ± 15% -36.5 0.00 perf-profile.children.cycles-pp.cpuidle_enter_state
36.34 ± 14% -36.3 0.00 perf-profile.children.cycles-pp.start_secondary
35.41 ± 9% -35.4 0.00 perf-profile.children.cycles-pp.read
33.03 ± 17% -33.0 0.00 perf-profile.children.cycles-pp.intel_idle
30.90 ± 9% -30.9 0.00 perf-profile.children.cycles-pp.write
20.96 ± 9% -21.0 0.00 perf-profile.children.cycles-pp.ksys_read
18.59 ± 9% -18.6 0.00 perf-profile.children.cycles-pp.vfs_read
16.59 ± 9% -16.6 0.00 perf-profile.children.cycles-pp.ksys_write
14.22 ± 9% -14.2 0.00 perf-profile.children.cycles-pp.vfs_write
10.75 ± 9% -10.7 0.00 perf-profile.children.cycles-pp.new_sync_read
9.12 ± 9% -9.1 0.00 perf-profile.children.cycles-pp.eventfd_read
8.41 ± 8% -8.4 0.00 perf-profile.children.cycles-pp.__entry_text_start
8.20 ± 9% -8.2 0.00 perf-profile.children.cycles-pp.security_file_permission
8.03 ± 9% -8.0 0.00 perf-profile.children.cycles-pp.eventfd_write
6.46 ± 9% -6.5 0.00 perf-profile.children.cycles-pp.syscall_return_via_sysret
5.46 ± 8% -5.5 0.00 perf-profile.children.cycles-pp.common_file_perm
5.24 ± 9% -5.2 0.00 perf-profile.children.cycles-pp._copy_to_iter
33.03 ± 17% -33.0 0.00 perf-profile.self.cycles-pp.intel_idle
6.46 ± 9% -6.5 0.00 perf-profile.self.cycles-pp.syscall_return_via_sysret
0.01 ± 47% -100.0% 0.00 perf-sched.sch_delay.avg.ms.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown]
0.02 ± 90% -100.0% 0.00 perf-sched.sch_delay.avg.ms.devkmsg_read.vfs_read.ksys_read.do_syscall_64
0.01 ± 35% -100.0% 0.00 perf-sched.sch_delay.avg.ms.do_nanosleep.hrtimer_nanosleep.__x64_sys_nanosleep.do_syscall_64
0.01 ± 92% -100.0% 0.00 perf-sched.sch_delay.avg.ms.do_syslog.part.0.kmsg_read.vfs_read
0.56 ±221% -100.0% 0.00 perf-sched.sch_delay.avg.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
0.01 ± 45% -100.0% 0.00 perf-sched.sch_delay.avg.ms.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64
0.01 ± 34% -100.0% 0.00 perf-sched.sch_delay.avg.ms.futex_wait_queue_me.futex_wait.do_futex.__x64_sys_futex
0.00 ± 57% -100.0% 0.00 perf-sched.sch_delay.avg.ms.pipe_read.new_sync_read.vfs_read.ksys_read
0.03 ±115% -100.0% 0.00 perf-sched.sch_delay.avg.ms.rcu_gp_kthread.kthread.ret_from_fork
0.01 ± 91% -100.0% 0.00 perf-sched.sch_delay.avg.ms.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait.__x64_sys_epoll_wait
0.01 ± 64% -100.0% 0.00 perf-sched.sch_delay.avg.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_select
0.01 ± 38% -100.0% 0.00 perf-sched.sch_delay.avg.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_sys_poll
0.02 ± 56% -100.0% 0.00 perf-sched.sch_delay.avg.ms.schedule_timeout.kcompactd.kthread.ret_from_fork
0.01 ± 88% -100.0% 0.00 perf-sched.sch_delay.avg.ms.schedule_timeout.rcu_gp_kthread.kthread.ret_from_fork
0.01 ± 50% -100.0% 0.00 perf-sched.sch_delay.avg.ms.schedule_timeout.wait_for_completion.__flush_work.lru_add_drain_all
0.01 ± 28% -100.0% 0.00 perf-sched.sch_delay.avg.ms.smpboot_thread_fn.kthread.ret_from_fork
0.00 ± 19% -100.0% 0.00 perf-sched.sch_delay.avg.ms.wait_for_partner.fifo_open.do_dentry_open.do_open.isra
0.24 ±206% -100.0% 0.00 perf-sched.sch_delay.avg.ms.worker_thread.kthread.ret_from_fork
0.02 ± 50% -100.0% 0.00 perf-sched.sch_delay.max.ms.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown]
0.04 ± 97% -100.0% 0.00 perf-sched.sch_delay.max.ms.devkmsg_read.vfs_read.ksys_read.do_syscall_64
0.03 ± 52% -100.0% 0.00 perf-sched.sch_delay.max.ms.do_nanosleep.hrtimer_nanosleep.__x64_sys_nanosleep.do_syscall_64
0.02 ±117% -100.0% 0.00 perf-sched.sch_delay.max.ms.do_syslog.part.0.kmsg_read.vfs_read
166.75 ±223% -100.0% 0.00 perf-sched.sch_delay.max.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
0.07 ± 79% -100.0% 0.00 perf-sched.sch_delay.max.ms.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64
0.01 ± 31% -100.0% 0.00 perf-sched.sch_delay.max.ms.exit_to_user_mode_prepare.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe.[unknown]
0.01 ± 35% -100.0% 0.00 perf-sched.sch_delay.max.ms.futex_wait_queue_me.futex_wait.do_futex.__x64_sys_futex
0.32 ±176% -100.0% 0.00 perf-sched.sch_delay.max.ms.pipe_read.new_sync_read.vfs_read.ksys_read
0.06 ± 13% -100.0% 0.00 perf-sched.sch_delay.max.ms.preempt_schedule_common.__cond_resched.stop_one_cpu.affine_move_task.__set_cpus_allowed_ptr
0.05 ±110% -100.0% 0.00 perf-sched.sch_delay.max.ms.rcu_gp_kthread.kthread.ret_from_fork
0.04 ±120% -100.0% 0.00 perf-sched.sch_delay.max.ms.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait.__x64_sys_epoll_wait
0.01 ± 95% -100.0% 0.00 perf-sched.sch_delay.max.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_select
0.02 ± 45% -100.0% 0.00 perf-sched.sch_delay.max.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_sys_poll
0.19 ±192% -100.0% 0.00 perf-sched.sch_delay.max.ms.schedule_timeout.kcompactd.kthread.ret_from_fork
0.88 ±209% -100.0% 0.00 perf-sched.sch_delay.max.ms.schedule_timeout.rcu_gp_kthread.kthread.ret_from_fork
0.01 ± 44% -100.0% 0.00 perf-sched.sch_delay.max.ms.schedule_timeout.wait_for_completion.__flush_work.lru_add_drain_all
1.46 ±211% -100.0% 0.00 perf-sched.sch_delay.max.ms.smpboot_thread_fn.kthread.ret_from_fork
0.01 ± 43% -100.0% 0.00 perf-sched.sch_delay.max.ms.wait_for_partner.fifo_open.do_dentry_open.do_open.isra
141.01 ±218% -100.0% 0.00 perf-sched.sch_delay.max.ms.worker_thread.kthread.ret_from_fork
0.03 ±115% -100.0% 0.00 perf-sched.total_sch_delay.average.ms
308.12 ±140% -100.0% 0.00 perf-sched.total_sch_delay.max.ms
218.18 ± 4% -100.0% 0.00 perf-sched.total_wait_and_delay.average.ms
11274 ± 3% -100.0% 0.00 perf-sched.total_wait_and_delay.count.ms
8874 ± 3% -100.0% 0.00 perf-sched.total_wait_and_delay.max.ms
218.15 ± 4% -100.0% 0.00 perf-sched.total_wait_time.average.ms
8874 ± 3% -100.0% 0.00 perf-sched.total_wait_time.max.ms
899.50 -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown]
1631 ± 33% -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.devkmsg_read.vfs_read.ksys_read.do_syscall_64
797.01 ± 4% -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.do_nanosleep.hrtimer_nanosleep.__x64_sys_nanosleep.do_syscall_64
1631 ± 33% -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.do_syslog.part.0.kmsg_read.vfs_read
218.64 ± 3% -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
0.74 ± 2% -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64
0.03 ± 6% -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.exit_to_user_mode_prepare.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe.[unknown]
1013 ± 22% -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.futex_wait_queue_me.futex_wait.do_futex.__x64_sys_futex
54.40 ± 3% -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.pipe_read.new_sync_read.vfs_read.ksys_read
4.57 ± 11% -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.preempt_schedule_common.__cond_resched.stop_one_cpu.affine_move_task.__set_cpus_allowed_ptr
2.76 ± 8% -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.rcu_gp_kthread.kthread.ret_from_fork
518.43 ± 19% -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait.__x64_sys_epoll_wait
599.96 ± 13% -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_sys_poll
478.26 -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.schedule_timeout.kcompactd.kthread.ret_from_fork
6.30 ± 23% -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.schedule_timeout.rcu_gp_kthread.kthread.ret_from_fork
691.51 -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.smpboot_thread_fn.kthread.ret_from_fork
0.00 ± 19% -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.wait_for_partner.fifo_open.do_dentry_open.do_open.isra
440.34 ± 6% -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.worker_thread.kthread.ret_from_fork
10.00 -100.0% 0.00 perf-sched.wait_and_delay.count.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown]
5.50 ± 27% -100.0% 0.00 perf-sched.wait_and_delay.count.devkmsg_read.vfs_read.ksys_read.do_syscall_64
22.67 ± 4% -100.0% 0.00 perf-sched.wait_and_delay.count.do_nanosleep.hrtimer_nanosleep.__x64_sys_nanosleep.do_syscall_64
5.50 ± 27% -100.0% 0.00 perf-sched.wait_and_delay.count.do_syslog.part.0.kmsg_read.vfs_read
302.50 -100.0% 0.00 perf-sched.wait_and_delay.count.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
294.67 -100.0% 0.00 perf-sched.wait_and_delay.count.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64
215.83 ± 12% -100.0% 0.00 perf-sched.wait_and_delay.count.exit_to_user_mode_prepare.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe.[unknown]
3.50 ± 42% -100.0% 0.00 perf-sched.wait_and_delay.count.futex_wait_queue_me.futex_wait.do_futex.__x64_sys_futex
2947 ± 2% -100.0% 0.00 perf-sched.wait_and_delay.count.pipe_read.new_sync_read.vfs_read.ksys_read
2208 ± 4% -100.0% 0.00 perf-sched.wait_and_delay.count.preempt_schedule_common.__cond_resched.stop_one_cpu.affine_move_task.__set_cpus_allowed_ptr
100.50 ± 24% -100.0% 0.00 perf-sched.wait_and_delay.count.rcu_gp_kthread.kthread.ret_from_fork
22.83 ± 35% -100.0% 0.00 perf-sched.wait_and_delay.count.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait.__x64_sys_epoll_wait
55.17 ± 9% -100.0% 0.00 perf-sched.wait_and_delay.count.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_sys_poll
78.33 ± 2% -100.0% 0.00 perf-sched.wait_and_delay.count.schedule_timeout.kcompactd.kthread.ret_from_fork
1642 ± 27% -100.0% 0.00 perf-sched.wait_and_delay.count.schedule_timeout.rcu_gp_kthread.kthread.ret_from_fork
2593 ± 3% -100.0% 0.00 perf-sched.wait_and_delay.count.smpboot_thread_fn.kthread.ret_from_fork
71.33 -100.0% 0.00 perf-sched.wait_and_delay.count.wait_for_partner.fifo_open.do_dentry_open.do_open.isra
651.50 ± 4% -100.0% 0.00 perf-sched.wait_and_delay.count.worker_thread.kthread.ret_from_fork
999.62 -100.0% 0.00 perf-sched.wait_and_delay.max.ms.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown]
4739 ± 36% -100.0% 0.00 perf-sched.wait_and_delay.max.ms.devkmsg_read.vfs_read.ksys_read.do_syscall_64
1000 -100.0% 0.00 perf-sched.wait_and_delay.max.ms.do_nanosleep.hrtimer_nanosleep.__x64_sys_nanosleep.do_syscall_64
4739 ± 36% -100.0% 0.00 perf-sched.wait_and_delay.max.ms.do_syslog.part.0.kmsg_read.vfs_read
3519 ± 49% -100.0% 0.00 perf-sched.wait_and_delay.max.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
16.30 -100.0% 0.00 perf-sched.wait_and_delay.max.ms.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64
0.12 ± 30% -100.0% 0.00 perf-sched.wait_and_delay.max.ms.exit_to_user_mode_prepare.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe.[unknown]
2009 ± 22% -100.0% 0.00 perf-sched.wait_and_delay.max.ms.futex_wait_queue_me.futex_wait.do_futex.__x64_sys_futex
4741 ± 36% -100.0% 0.00 perf-sched.wait_and_delay.max.ms.pipe_read.new_sync_read.vfs_read.ksys_read
1768 ± 43% -100.0% 0.00 perf-sched.wait_and_delay.max.ms.preempt_schedule_common.__cond_resched.stop_one_cpu.affine_move_task.__set_cpus_allowed_ptr
4.86 ± 6% -100.0% 0.00 perf-sched.wait_and_delay.max.ms.rcu_gp_kthread.kthread.ret_from_fork
4385 ± 40% -100.0% 0.00 perf-sched.wait_and_delay.max.ms.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait.__x64_sys_epoll_wait
6937 ± 12% -100.0% 0.00 perf-sched.wait_and_delay.max.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_sys_poll
504.87 -100.0% 0.00 perf-sched.wait_and_delay.max.ms.schedule_timeout.kcompactd.kthread.ret_from_fork
210.35 ± 49% -100.0% 0.00 perf-sched.wait_and_delay.max.ms.schedule_timeout.rcu_gp_kthread.kthread.ret_from_fork
1226 ± 35% -100.0% 0.00 perf-sched.wait_and_delay.max.ms.smpboot_thread_fn.kthread.ret_from_fork
0.01 ± 43% -100.0% 0.00 perf-sched.wait_and_delay.max.ms.wait_for_partner.fifo_open.do_dentry_open.do_open.isra
8874 ± 3% -100.0% 0.00 perf-sched.wait_and_delay.max.ms.worker_thread.kthread.ret_from_fork
899.48 -100.0% 0.00 perf-sched.wait_time.avg.ms.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown]
1631 ± 33% -100.0% 0.00 perf-sched.wait_time.avg.ms.devkmsg_read.vfs_read.ksys_read.do_syscall_64
797.00 ± 4% -100.0% 0.00 perf-sched.wait_time.avg.ms.do_nanosleep.hrtimer_nanosleep.__x64_sys_nanosleep.do_syscall_64
1631 ± 33% -100.0% 0.00 perf-sched.wait_time.avg.ms.do_syslog.part.0.kmsg_read.vfs_read
218.08 ± 3% -100.0% 0.00 perf-sched.wait_time.avg.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
0.74 -100.0% 0.00 perf-sched.wait_time.avg.ms.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64
0.02 ± 18% -100.0% 0.00 perf-sched.wait_time.avg.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt.[unknown]
0.03 ± 6% -100.0% 0.00 perf-sched.wait_time.avg.ms.exit_to_user_mode_prepare.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe.[unknown]
1013 ± 22% -100.0% 0.00 perf-sched.wait_time.avg.ms.futex_wait_queue_me.futex_wait.do_futex.__x64_sys_futex
54.39 ± 3% -100.0% 0.00 perf-sched.wait_time.avg.ms.pipe_read.new_sync_read.vfs_read.ksys_read
4.57 ± 11% -100.0% 0.00 perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.stop_one_cpu.affine_move_task.__set_cpus_allowed_ptr
2.74 ± 9% -100.0% 0.00 perf-sched.wait_time.avg.ms.rcu_gp_kthread.kthread.ret_from_fork
518.41 ± 19% -100.0% 0.00 perf-sched.wait_time.avg.ms.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait.__x64_sys_epoll_wait
599.95 ± 13% -100.0% 0.00 perf-sched.wait_time.avg.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_sys_poll
478.24 -100.0% 0.00 perf-sched.wait_time.avg.ms.schedule_timeout.kcompactd.kthread.ret_from_fork
0.01 ± 50% -100.0% 0.00 perf-sched.wait_time.avg.ms.schedule_timeout.khugepaged.kthread.ret_from_fork
6.29 ± 23% -100.0% 0.00 perf-sched.wait_time.avg.ms.schedule_timeout.rcu_gp_kthread.kthread.ret_from_fork
691.51 -100.0% 0.00 perf-sched.wait_time.avg.ms.smpboot_thread_fn.kthread.ret_from_fork
440.10 ± 6% -100.0% 0.00 perf-sched.wait_time.avg.ms.worker_thread.kthread.ret_from_fork
999.61 -100.0% 0.00 perf-sched.wait_time.max.ms.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown]
4739 ± 36% -100.0% 0.00 perf-sched.wait_time.max.ms.devkmsg_read.vfs_read.ksys_read.do_syscall_64
1000 -100.0% 0.00 perf-sched.wait_time.max.ms.do_nanosleep.hrtimer_nanosleep.__x64_sys_nanosleep.do_syscall_64
4739 ± 36% -100.0% 0.00 perf-sched.wait_time.max.ms.do_syslog.part.0.kmsg_read.vfs_read
3519 ± 49% -100.0% 0.00 perf-sched.wait_time.max.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
16.30 -100.0% 0.00 perf-sched.wait_time.max.ms.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64
0.07 ± 58% -100.0% 0.00 perf-sched.wait_time.max.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt.[unknown]
0.12 ± 30% -100.0% 0.00 perf-sched.wait_time.max.ms.exit_to_user_mode_prepare.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe.[unknown]
2009 ± 22% -100.0% 0.00 perf-sched.wait_time.max.ms.futex_wait_queue_me.futex_wait.do_futex.__x64_sys_futex
4741 ± 36% -100.0% 0.00 perf-sched.wait_time.max.ms.pipe_read.new_sync_read.vfs_read.ksys_read
1768 ± 43% -100.0% 0.00 perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.stop_one_cpu.affine_move_task.__set_cpus_allowed_ptr
4.83 ± 7% -100.0% 0.00 perf-sched.wait_time.max.ms.rcu_gp_kthread.kthread.ret_from_fork
4384 ± 40% -100.0% 0.00 perf-sched.wait_time.max.ms.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait.__x64_sys_epoll_wait
6937 ± 12% -100.0% 0.00 perf-sched.wait_time.max.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_sys_poll
504.85 -100.0% 0.00 perf-sched.wait_time.max.ms.schedule_timeout.kcompactd.kthread.ret_from_fork
0.01 ± 50% -100.0% 0.00 perf-sched.wait_time.max.ms.schedule_timeout.khugepaged.kthread.ret_from_fork
210.32 ± 49% -100.0% 0.00 perf-sched.wait_time.max.ms.schedule_timeout.rcu_gp_kthread.kthread.ret_from_fork
1226 ± 35% -100.0% 0.00 perf-sched.wait_time.max.ms.smpboot_thread_fn.kthread.ret_from_fork
8874 ± 3% -100.0% 0.00 perf-sched.wait_time.max.ms.worker_thread.kthread.ret_from_fork
4703 ± 53% -100.0% 0.00 interrupts.CPU0.NMI:Non-maskable_interrupts
4703 ± 53% -100.0% 0.00 interrupts.CPU0.PMI:Performance_monitoring_interrupts
5649 ± 32% -100.0% 0.00 interrupts.CPU1.NMI:Non-maskable_interrupts
5649 ± 32% -100.0% 0.00 interrupts.CPU1.PMI:Performance_monitoring_interrupts
6583 ± 37% -100.0% 0.00 interrupts.CPU10.NMI:Non-maskable_interrupts
6583 ± 37% -100.0% 0.00 interrupts.CPU10.PMI:Performance_monitoring_interrupts
5033 ± 40% -100.0% 0.00 interrupts.CPU100.NMI:Non-maskable_interrupts
5033 ± 40% -100.0% 0.00 interrupts.CPU100.PMI:Performance_monitoring_interrupts
7687 ± 23% -100.0% 1.00 ±223% interrupts.CPU101.NMI:Non-maskable_interrupts
7687 ± 23% -100.0% 1.00 ±223% interrupts.CPU101.PMI:Performance_monitoring_interrupts
5400 ± 39% -100.0% 0.00 interrupts.CPU102.NMI:Non-maskable_interrupts
5400 ± 39% -100.0% 0.00 interrupts.CPU102.PMI:Performance_monitoring_interrupts
4291 ± 23% -100.0% 0.00 interrupts.CPU103.NMI:Non-maskable_interrupts
4291 ± 23% -100.0% 0.00 interrupts.CPU103.PMI:Performance_monitoring_interrupts
4648 ± 36% -100.0% 0.00 interrupts.CPU104.NMI:Non-maskable_interrupts
4648 ± 36% -100.0% 0.00 interrupts.CPU104.PMI:Performance_monitoring_interrupts
3899 ± 31% -100.0% 0.00 interrupts.CPU105.NMI:Non-maskable_interrupts
3899 ± 31% -100.0% 0.00 interrupts.CPU105.PMI:Performance_monitoring_interrupts
5477 ± 47% -100.0% 0.00 interrupts.CPU106.NMI:Non-maskable_interrupts
5477 ± 47% -100.0% 0.00 interrupts.CPU106.PMI:Performance_monitoring_interrupts
6150 ± 40% -100.0% 0.00 interrupts.CPU107.NMI:Non-maskable_interrupts
6150 ± 40% -100.0% 0.00 interrupts.CPU107.PMI:Performance_monitoring_interrupts
4603 ± 53% -100.0% 0.00 interrupts.CPU108.NMI:Non-maskable_interrupts
4603 ± 53% -100.0% 0.00 interrupts.CPU108.PMI:Performance_monitoring_interrupts
6039 ± 40% -100.0% 0.00 interrupts.CPU109.NMI:Non-maskable_interrupts
6039 ± 40% -100.0% 0.00 interrupts.CPU109.PMI:Performance_monitoring_interrupts
5284 ± 46% -100.0% 0.00 interrupts.CPU11.NMI:Non-maskable_interrupts
5284 ± 46% -100.0% 0.00 interrupts.CPU11.PMI:Performance_monitoring_interrupts
6376 ± 35% -100.0% 0.00 interrupts.CPU110.NMI:Non-maskable_interrupts
6376 ± 35% -100.0% 0.00 interrupts.CPU110.PMI:Performance_monitoring_interrupts
4144 ± 38% -100.0% 0.00 interrupts.CPU111.NMI:Non-maskable_interrupts
4144 ± 38% -100.0% 0.00 interrupts.CPU111.PMI:Performance_monitoring_interrupts
248.67 ± 79% -100.0% 0.00 interrupts.CPU112.NMI:Non-maskable_interrupts
248.67 ± 79% -100.0% 0.00 interrupts.CPU112.PMI:Performance_monitoring_interrupts
109.50 ± 9% -100.0% 0.00 interrupts.CPU113.NMI:Non-maskable_interrupts
109.50 ± 9% -100.0% 0.00 interrupts.CPU113.PMI:Performance_monitoring_interrupts
228.67 ±120% -100.0% 0.00 interrupts.CPU114.NMI:Non-maskable_interrupts
228.67 ±120% -100.0% 0.00 interrupts.CPU114.PMI:Performance_monitoring_interrupts
109.50 ± 33% -100.0% 0.00 interrupts.CPU115.NMI:Non-maskable_interrupts
109.50 ± 33% -100.0% 0.00 interrupts.CPU115.PMI:Performance_monitoring_interrupts
106.50 ± 27% -100.0% 0.00 interrupts.CPU116.NMI:Non-maskable_interrupts
106.50 ± 27% -100.0% 0.00 interrupts.CPU116.PMI:Performance_monitoring_interrupts
107.00 ± 10% -100.0% 0.00 interrupts.CPU117.NMI:Non-maskable_interrupts
107.00 ± 10% -100.0% 0.00 interrupts.CPU117.PMI:Performance_monitoring_interrupts
89.50 ± 29% -100.0% 0.00 interrupts.CPU118.NMI:Non-maskable_interrupts
89.50 ± 29% -100.0% 0.00 interrupts.CPU118.PMI:Performance_monitoring_interrupts
97.33 ± 18% -100.0% 0.00 interrupts.CPU119.NMI:Non-maskable_interrupts
97.33 ± 18% -100.0% 0.00 interrupts.CPU119.PMI:Performance_monitoring_interrupts
7484 ± 36% -100.0% 0.00 interrupts.CPU12.NMI:Non-maskable_interrupts
7484 ± 36% -100.0% 0.00 interrupts.CPU12.PMI:Performance_monitoring_interrupts
114.33 ± 32% -100.0% 0.00 interrupts.CPU120.NMI:Non-maskable_interrupts
114.33 ± 32% -100.0% 0.00 interrupts.CPU120.PMI:Performance_monitoring_interrupts
105.50 ± 40% -100.0% 0.00 interrupts.CPU121.NMI:Non-maskable_interrupts
105.50 ± 40% -100.0% 0.00 interrupts.CPU121.PMI:Performance_monitoring_interrupts
108.00 ± 45% -100.0% 0.00 interrupts.CPU122.NMI:Non-maskable_interrupts
108.00 ± 45% -100.0% 0.00 interrupts.CPU122.PMI:Performance_monitoring_interrupts
139.17 ± 78% -100.0% 0.00 interrupts.CPU123.NMI:Non-maskable_interrupts
139.17 ± 78% -100.0% 0.00 interrupts.CPU123.PMI:Performance_monitoring_interrupts
113.83 ± 44% -100.0% 0.00 interrupts.CPU124.NMI:Non-maskable_interrupts
113.83 ± 44% -100.0% 0.00 interrupts.CPU124.PMI:Performance_monitoring_interrupts
124.83 ± 47% -100.0% 0.00 interrupts.CPU125.NMI:Non-maskable_interrupts
124.83 ± 47% -100.0% 0.00 interrupts.CPU125.PMI:Performance_monitoring_interrupts
139.83 ± 60% -100.0% 0.00 interrupts.CPU126.NMI:Non-maskable_interrupts
139.83 ± 60% -100.0% 0.00 interrupts.CPU126.PMI:Performance_monitoring_interrupts
110.67 ± 47% -99.1% 1.00 ±223% interrupts.CPU127.NMI:Non-maskable_interrupts
110.67 ± 47% -99.1% 1.00 ±223% interrupts.CPU127.PMI:Performance_monitoring_interrupts
105.00 ± 42% -100.0% 0.00 interrupts.CPU128.NMI:Non-maskable_interrupts
105.00 ± 42% -100.0% 0.00 interrupts.CPU128.PMI:Performance_monitoring_interrupts
102.00 ± 40% -100.0% 0.00 interrupts.CPU129.NMI:Non-maskable_interrupts
102.00 ± 40% -100.0% 0.00 interrupts.CPU129.PMI:Performance_monitoring_interrupts
6284 ± 40% -100.0% 0.00 interrupts.CPU13.NMI:Non-maskable_interrupts
6284 ± 40% -100.0% 0.00 interrupts.CPU13.PMI:Performance_monitoring_interrupts
112.83 ± 32% -100.0% 0.00 interrupts.CPU130.NMI:Non-maskable_interrupts
112.83 ± 32% -100.0% 0.00 interrupts.CPU130.PMI:Performance_monitoring_interrupts
118.50 ± 37% -100.0% 0.00 interrupts.CPU131.NMI:Non-maskable_interrupts
118.50 ± 37% -100.0% 0.00 interrupts.CPU131.PMI:Performance_monitoring_interrupts
115.17 ± 33% -100.0% 0.00 interrupts.CPU132.NMI:Non-maskable_interrupts
115.17 ± 33% -100.0% 0.00 interrupts.CPU132.PMI:Performance_monitoring_interrupts
112.50 ± 31% -100.0% 0.00 interrupts.CPU133.NMI:Non-maskable_interrupts
112.50 ± 31% -100.0% 0.00 interrupts.CPU133.PMI:Performance_monitoring_interrupts
105.33 ± 38% -100.0% 0.00 interrupts.CPU134.NMI:Non-maskable_interrupts
105.33 ± 38% -100.0% 0.00 interrupts.CPU134.PMI:Performance_monitoring_interrupts
102.33 ± 40% -100.0% 0.00 interrupts.CPU135.NMI:Non-maskable_interrupts
102.33 ± 40% -100.0% 0.00 interrupts.CPU135.PMI:Performance_monitoring_interrupts
102.83 ± 38% -100.0% 0.00 interrupts.CPU136.NMI:Non-maskable_interrupts
102.83 ± 38% -100.0% 0.00 interrupts.CPU136.PMI:Performance_monitoring_interrupts
102.33 ± 40% -100.0% 0.00 interrupts.CPU137.NMI:Non-maskable_interrupts
102.33 ± 40% -100.0% 0.00 interrupts.CPU137.PMI:Performance_monitoring_interrupts
104.00 ± 40% -100.0% 0.00 interrupts.CPU138.NMI:Non-maskable_interrupts
104.00 ± 40% -100.0% 0.00 interrupts.CPU138.PMI:Performance_monitoring_interrupts
110.00 ± 30% -100.0% 0.00 interrupts.CPU139.NMI:Non-maskable_interrupts
110.00 ± 30% -100.0% 0.00 interrupts.CPU139.PMI:Performance_monitoring_interrupts
5200 ± 40% -100.0% 0.00 interrupts.CPU14.NMI:Non-maskable_interrupts
5200 ± 40% -100.0% 0.00 interrupts.CPU14.PMI:Performance_monitoring_interrupts
102.50 ± 40% -100.0% 0.00 interrupts.CPU140.NMI:Non-maskable_interrupts
102.50 ± 40% -100.0% 0.00 interrupts.CPU140.PMI:Performance_monitoring_interrupts
103.50 ± 40% -99.0% 1.00 ±223% interrupts.CPU141.NMI:Non-maskable_interrupts
103.50 ± 40% -99.0% 1.00 ±223% interrupts.CPU141.PMI:Performance_monitoring_interrupts
103.67 ± 41% -100.0% 0.00 interrupts.CPU142.NMI:Non-maskable_interrupts
103.67 ± 41% -100.0% 0.00 interrupts.CPU142.PMI:Performance_monitoring_interrupts
104.33 ± 41% -100.0% 0.00 interrupts.CPU143.NMI:Non-maskable_interrupts
104.33 ± 41% -100.0% 0.00 interrupts.CPU143.PMI:Performance_monitoring_interrupts
168.33 ± 42% -100.0% 0.00 interrupts.CPU144.NMI:Non-maskable_interrupts
168.33 ± 42% -100.0% 0.00 interrupts.CPU144.PMI:Performance_monitoring_interrupts
103.33 ± 39% -100.0% 0.00 interrupts.CPU145.NMI:Non-maskable_interrupts
103.33 ± 39% -100.0% 0.00 interrupts.CPU145.PMI:Performance_monitoring_interrupts
106.83 ± 38% -100.0% 0.00 interrupts.CPU146.NMI:Non-maskable_interrupts
106.83 ± 38% -100.0% 0.00 interrupts.CPU146.PMI:Performance_monitoring_interrupts
114.50 ± 31% -100.0% 0.00 interrupts.CPU147.NMI:Non-maskable_interrupts
114.50 ± 31% -100.0% 0.00 interrupts.CPU147.PMI:Performance_monitoring_interrupts
116.17 ± 30% -100.0% 0.00 interrupts.CPU148.NMI:Non-maskable_interrupts
116.17 ± 30% -100.0% 0.00 interrupts.CPU148.PMI:Performance_monitoring_interrupts
124.33 ± 34% -100.0% 0.00 interrupts.CPU149.NMI:Non-maskable_interrupts
124.33 ± 34% -100.0% 0.00 interrupts.CPU149.PMI:Performance_monitoring_interrupts
7696 ± 20% -100.0% 0.00 interrupts.CPU15.NMI:Non-maskable_interrupts
7696 ± 20% -100.0% 0.00 interrupts.CPU15.PMI:Performance_monitoring_interrupts
124.83 ± 31% -100.0% 0.00 interrupts.CPU150.NMI:Non-maskable_interrupts
124.83 ± 31% -100.0% 0.00 interrupts.CPU150.PMI:Performance_monitoring_interrupts
124.00 ± 36% -100.0% 0.00 interrupts.CPU151.NMI:Non-maskable_interrupts
124.00 ± 36% -100.0% 0.00 interrupts.CPU151.PMI:Performance_monitoring_interrupts
114.33 ± 31% -100.0% 0.00 interrupts.CPU152.NMI:Non-maskable_interrupts
114.33 ± 31% -100.0% 0.00 interrupts.CPU152.PMI:Performance_monitoring_interrupts
118.17 ± 30% -100.0% 0.00 interrupts.CPU153.NMI:Non-maskable_interrupts
118.17 ± 30% -100.0% 0.00 interrupts.CPU153.PMI:Performance_monitoring_interrupts
116.33 ± 33% -100.0% 0.00 interrupts.CPU154.NMI:Non-maskable_interrupts
116.33 ± 33% -100.0% 0.00 interrupts.CPU154.PMI:Performance_monitoring_interrupts
118.17 ± 32% -100.0% 0.00 interrupts.CPU155.NMI:Non-maskable_interrupts
118.17 ± 32% -100.0% 0.00 interrupts.CPU155.PMI:Performance_monitoring_interrupts
121.83 ± 36% -100.0% 0.00 interrupts.CPU156.NMI:Non-maskable_interrupts
121.83 ± 36% -100.0% 0.00 interrupts.CPU156.PMI:Performance_monitoring_interrupts
124.33 ± 18% -100.0% 0.00 interrupts.CPU157.NMI:Non-maskable_interrupts
124.33 ± 18% -100.0% 0.00 interrupts.CPU157.PMI:Performance_monitoring_interrupts
131.67 ± 21% -100.0% 0.00 interrupts.CPU158.NMI:Non-maskable_interrupts
131.67 ± 21% -100.0% 0.00 interrupts.CPU158.PMI:Performance_monitoring_interrupts
123.33 ± 18% -100.0% 0.00 interrupts.CPU159.NMI:Non-maskable_interrupts
123.33 ± 18% -100.0% 0.00 interrupts.CPU159.PMI:Performance_monitoring_interrupts
164.33 ± 58% -100.0% 0.00 interrupts.CPU16.NMI:Non-maskable_interrupts
164.33 ± 58% -100.0% 0.00 interrupts.CPU16.PMI:Performance_monitoring_interrupts
125.33 ± 18% -100.0% 0.00 interrupts.CPU160.NMI:Non-maskable_interrupts
125.33 ± 18% -100.0% 0.00 interrupts.CPU160.PMI:Performance_monitoring_interrupts
126.50 ± 21% -100.0% 0.00 interrupts.CPU161.NMI:Non-maskable_interrupts
126.50 ± 21% -100.0% 0.00 interrupts.CPU161.PMI:Performance_monitoring_interrupts
124.00 ± 19% -99.2% 1.00 ±223% interrupts.CPU162.NMI:Non-maskable_interrupts
124.00 ± 19% -99.2% 1.00 ±223% interrupts.CPU162.PMI:Performance_monitoring_interrupts
127.67 ± 21% -100.0% 0.00 interrupts.CPU163.NMI:Non-maskable_interrupts
127.67 ± 21% -100.0% 0.00 interrupts.CPU163.PMI:Performance_monitoring_interrupts
124.83 ± 22% -100.0% 0.00 interrupts.CPU164.NMI:Non-maskable_interrupts
124.83 ± 22% -100.0% 0.00 interrupts.CPU164.PMI:Performance_monitoring_interrupts
112.83 ± 31% -100.0% 0.00 interrupts.CPU165.NMI:Non-maskable_interrupts
112.83 ± 31% -100.0% 0.00 interrupts.CPU165.PMI:Performance_monitoring_interrupts
114.50 ± 31% -100.0% 0.00 interrupts.CPU166.NMI:Non-maskable_interrupts
114.50 ± 31% -100.0% 0.00 interrupts.CPU166.PMI:Performance_monitoring_interrupts
135.83 ± 35% -100.0% 0.00 interrupts.CPU167.NMI:Non-maskable_interrupts
135.83 ± 35% -100.0% 0.00 interrupts.CPU167.PMI:Performance_monitoring_interrupts
119.83 ± 44% -100.0% 0.00 interrupts.CPU168.NMI:Non-maskable_interrupts
119.83 ± 44% -100.0% 0.00 interrupts.CPU168.PMI:Performance_monitoring_interrupts
130.67 ± 16% -100.0% 0.00 interrupts.CPU169.NMI:Non-maskable_interrupts
130.67 ± 16% -100.0% 0.00 interrupts.CPU169.PMI:Performance_monitoring_interrupts
106.50 ± 9% -100.0% 0.00 interrupts.CPU17.NMI:Non-maskable_interrupts
106.50 ± 9% -100.0% 0.00 interrupts.CPU17.PMI:Performance_monitoring_interrupts
126.83 ± 17% -100.0% 0.00 interrupts.CPU170.NMI:Non-maskable_interrupts
126.83 ± 17% -100.0% 0.00 interrupts.CPU170.PMI:Performance_monitoring_interrupts
112.50 ± 21% -100.0% 0.00 interrupts.CPU171.NMI:Non-maskable_interrupts
112.50 ± 21% -100.0% 0.00 interrupts.CPU171.PMI:Performance_monitoring_interrupts
123.33 ± 16% -100.0% 0.00 interrupts.CPU172.NMI:Non-maskable_interrupts
123.33 ± 16% -100.0% 0.00 interrupts.CPU172.PMI:Performance_monitoring_interrupts
126.17 ± 15% -100.0% 0.00 interrupts.CPU173.NMI:Non-maskable_interrupts
126.17 ± 15% -100.0% 0.00 interrupts.CPU173.PMI:Performance_monitoring_interrupts
124.00 ± 16% -100.0% 0.00 interrupts.CPU174.NMI:Non-maskable_interrupts
124.00 ± 16% -100.0% 0.00 interrupts.CPU174.PMI:Performance_monitoring_interrupts
161.17 ± 60% -100.0% 0.00 interrupts.CPU175.NMI:Non-maskable_interrupts
161.17 ± 60% -100.0% 0.00 interrupts.CPU175.PMI:Performance_monitoring_interrupts
125.17 ± 18% -100.0% 0.00 interrupts.CPU176.NMI:Non-maskable_interrupts
125.17 ± 18% -100.0% 0.00 interrupts.CPU176.PMI:Performance_monitoring_interrupts
179.67 ± 70% -100.0% 0.00 interrupts.CPU177.NMI:Non-maskable_interrupts
179.67 ± 70% -100.0% 0.00 interrupts.CPU177.PMI:Performance_monitoring_interrupts
124.50 ± 19% -100.0% 0.00 interrupts.CPU178.NMI:Non-maskable_interrupts
124.50 ± 19% -100.0% 0.00 interrupts.CPU178.PMI:Performance_monitoring_interrupts
127.67 ± 18% -100.0% 0.00 interrupts.CPU179.NMI:Non-maskable_interrupts
127.67 ± 18% -100.0% 0.00 interrupts.CPU179.PMI:Performance_monitoring_interrupts
195.67 ±100% -100.0% 0.00 interrupts.CPU18.NMI:Non-maskable_interrupts
195.67 ±100% -100.0% 0.00 interrupts.CPU18.PMI:Performance_monitoring_interrupts
125.00 ± 18% -100.0% 0.00 interrupts.CPU180.NMI:Non-maskable_interrupts
125.00 ± 18% -100.0% 0.00 interrupts.CPU180.PMI:Performance_monitoring_interrupts
124.33 ± 17% -100.0% 0.00 interrupts.CPU181.NMI:Non-maskable_interrupts
124.33 ± 17% -100.0% 0.00 interrupts.CPU181.PMI:Performance_monitoring_interrupts
139.33 ± 27% -100.0% 0.00 interrupts.CPU182.NMI:Non-maskable_interrupts
139.33 ± 27% -100.0% 0.00 interrupts.CPU182.PMI:Performance_monitoring_interrupts
132.67 ± 25% -100.0% 0.00 interrupts.CPU183.NMI:Non-maskable_interrupts
132.67 ± 25% -100.0% 0.00 interrupts.CPU183.PMI:Performance_monitoring_interrupts
123.67 ± 17% -100.0% 0.00 interrupts.CPU184.NMI:Non-maskable_interrupts
123.67 ± 17% -100.0% 0.00 interrupts.CPU184.PMI:Performance_monitoring_interrupts
127.50 ± 19% -100.0% 0.00 interrupts.CPU185.NMI:Non-maskable_interrupts
127.50 ± 19% -100.0% 0.00 interrupts.CPU185.PMI:Performance_monitoring_interrupts
128.67 ± 17% -100.0% 0.00 interrupts.CPU186.NMI:Non-maskable_interrupts
128.67 ± 17% -100.0% 0.00 interrupts.CPU186.PMI:Performance_monitoring_interrupts
126.33 ± 17% -100.0% 0.00 interrupts.CPU187.NMI:Non-maskable_interrupts
126.33 ± 17% -100.0% 0.00 interrupts.CPU187.PMI:Performance_monitoring_interrupts
124.50 ± 17% -100.0% 0.00 interrupts.CPU188.NMI:Non-maskable_interrupts
124.50 ± 17% -100.0% 0.00 interrupts.CPU188.PMI:Performance_monitoring_interrupts
128.17 ± 23% -100.0% 0.00 interrupts.CPU189.NMI:Non-maskable_interrupts
128.17 ± 23% -100.0% 0.00 interrupts.CPU189.PMI:Performance_monitoring_interrupts
32977 ±216% -97.3% 887.50 interrupts.CPU19.CAL:Function_call_interrupts
111.33 ± 36% -100.0% 0.00 interrupts.CPU19.NMI:Non-maskable_interrupts
111.33 ± 36% -100.0% 0.00 interrupts.CPU19.PMI:Performance_monitoring_interrupts
124.00 ± 17% -100.0% 0.00 interrupts.CPU190.NMI:Non-maskable_interrupts
124.00 ± 17% -100.0% 0.00 interrupts.CPU190.PMI:Performance_monitoring_interrupts
542.50 ± 21% -100.0% 0.00 interrupts.CPU191.NMI:Non-maskable_interrupts
542.50 ± 21% -100.0% 0.00 interrupts.CPU191.PMI:Performance_monitoring_interrupts
7354 ± 36% -100.0% 0.00 interrupts.CPU2.NMI:Non-maskable_interrupts
7354 ± 36% -100.0% 0.00 interrupts.CPU2.PMI:Performance_monitoring_interrupts
97.67 ± 39% -100.0% 0.00 interrupts.CPU20.NMI:Non-maskable_interrupts
97.67 ± 39% -100.0% 0.00 interrupts.CPU20.PMI:Performance_monitoring_interrupts
87.67 ± 32% -100.0% 0.00 interrupts.CPU21.NMI:Non-maskable_interrupts
87.67 ± 32% -100.0% 0.00 interrupts.CPU21.PMI:Performance_monitoring_interrupts
82.50 ± 38% -100.0% 0.00 interrupts.CPU22.NMI:Non-maskable_interrupts
82.50 ± 38% -100.0% 0.00 interrupts.CPU22.PMI:Performance_monitoring_interrupts
89.00 ± 31% -100.0% 0.00 interrupts.CPU23.NMI:Non-maskable_interrupts
89.00 ± 31% -100.0% 0.00 interrupts.CPU23.PMI:Performance_monitoring_interrupts
98.17 ± 49% -100.0% 0.00 interrupts.CPU24.NMI:Non-maskable_interrupts
98.17 ± 49% -100.0% 0.00 interrupts.CPU24.PMI:Performance_monitoring_interrupts
87.50 ± 57% -100.0% 0.00 interrupts.CPU25.NMI:Non-maskable_interrupts
87.50 ± 57% -100.0% 0.00 interrupts.CPU25.PMI:Performance_monitoring_interrupts
99.67 ± 53% -100.0% 0.00 interrupts.CPU26.NMI:Non-maskable_interrupts
99.67 ± 53% -100.0% 0.00 interrupts.CPU26.PMI:Performance_monitoring_interrupts
115.83 ± 53% -100.0% 0.00 interrupts.CPU27.NMI:Non-maskable_interrupts
115.83 ± 53% -100.0% 0.00 interrupts.CPU27.PMI:Performance_monitoring_interrupts
130.83 ± 23% -100.0% 0.00 interrupts.CPU28.NMI:Non-maskable_interrupts
130.83 ± 23% -100.0% 0.00 interrupts.CPU28.PMI:Performance_monitoring_interrupts
124.50 ± 47% -100.0% 0.00 interrupts.CPU29.NMI:Non-maskable_interrupts
124.50 ± 47% -100.0% 0.00 interrupts.CPU29.PMI:Performance_monitoring_interrupts
5843 ± 41% -100.0% 0.00 interrupts.CPU3.NMI:Non-maskable_interrupts
5843 ± 41% -100.0% 0.00 interrupts.CPU3.PMI:Performance_monitoring_interrupts
184.50 ±108% -100.0% 0.00 interrupts.CPU30.NMI:Non-maskable_interrupts
184.50 ±108% -100.0% 0.00 interrupts.CPU30.PMI:Performance_monitoring_interrupts
94.33 ± 49% -100.0% 0.00 interrupts.CPU31.NMI:Non-maskable_interrupts
94.33 ± 49% -100.0% 0.00 interrupts.CPU31.PMI:Performance_monitoring_interrupts
93.83 ± 50% -100.0% 0.00 interrupts.CPU32.NMI:Non-maskable_interrupts
93.83 ± 50% -100.0% 0.00 interrupts.CPU32.PMI:Performance_monitoring_interrupts
88.17 ± 60% -100.0% 0.00 interrupts.CPU33.NMI:Non-maskable_interrupts
88.17 ± 60% -100.0% 0.00 interrupts.CPU33.PMI:Performance_monitoring_interrupts
85.67 ± 57% -100.0% 0.00 interrupts.CPU34.NMI:Non-maskable_interrupts
85.67 ± 57% -100.0% 0.00 interrupts.CPU34.PMI:Performance_monitoring_interrupts
98.50 ± 52% -100.0% 0.00 interrupts.CPU35.NMI:Non-maskable_interrupts
98.50 ± 52% -100.0% 0.00 interrupts.CPU35.PMI:Performance_monitoring_interrupts
96.50 ± 51% -100.0% 0.00 interrupts.CPU36.NMI:Non-maskable_interrupts
96.50 ± 51% -100.0% 0.00 interrupts.CPU36.PMI:Performance_monitoring_interrupts
94.67 ± 49% -100.0% 0.00 interrupts.CPU37.NMI:Non-maskable_interrupts
94.67 ± 49% -100.0% 0.00 interrupts.CPU37.PMI:Performance_monitoring_interrupts
105.67 ± 40% -100.0% 0.00 interrupts.CPU38.NMI:Non-maskable_interrupts
105.67 ± 40% -100.0% 0.00 interrupts.CPU38.PMI:Performance_monitoring_interrupts
116.67 ± 36% -100.0% 0.00 interrupts.CPU39.NMI:Non-maskable_interrupts
116.67 ± 36% -100.0% 0.00 interrupts.CPU39.PMI:Performance_monitoring_interrupts
5561 ± 38% -100.0% 0.00 interrupts.CPU4.NMI:Non-maskable_interrupts
5561 ± 38% -100.0% 0.00 interrupts.CPU4.PMI:Performance_monitoring_interrupts
121.00 ± 16% -100.0% 0.00 interrupts.CPU40.NMI:Non-maskable_interrupts
121.00 ± 16% -100.0% 0.00 interrupts.CPU40.PMI:Performance_monitoring_interrupts
120.83 ± 19% -100.0% 0.00 interrupts.CPU41.NMI:Non-maskable_interrupts
120.83 ± 19% -100.0% 0.00 interrupts.CPU41.PMI:Performance_monitoring_interrupts
122.17 ± 20% -100.0% 0.00 interrupts.CPU42.NMI:Non-maskable_interrupts
122.17 ± 20% -100.0% 0.00 interrupts.CPU42.PMI:Performance_monitoring_interrupts
119.67 ± 19% -100.0% 0.00 interrupts.CPU43.NMI:Non-maskable_interrupts
119.67 ± 19% -100.0% 0.00 interrupts.CPU43.PMI:Performance_monitoring_interrupts
121.00 ± 19% -100.0% 0.00 interrupts.CPU44.NMI:Non-maskable_interrupts
121.00 ± 19% -100.0% 0.00 interrupts.CPU44.PMI:Performance_monitoring_interrupts
121.17 ± 19% -100.0% 0.00 interrupts.CPU45.NMI:Non-maskable_interrupts
121.17 ± 19% -100.0% 0.00 interrupts.CPU45.PMI:Performance_monitoring_interrupts
127.50 ± 25% -100.0% 0.00 interrupts.CPU46.NMI:Non-maskable_interrupts
127.50 ± 25% -100.0% 0.00 interrupts.CPU46.PMI:Performance_monitoring_interrupts
122.50 ± 20% -100.0% 0.00 interrupts.CPU47.NMI:Non-maskable_interrupts
122.50 ± 20% -100.0% 0.00 interrupts.CPU47.PMI:Performance_monitoring_interrupts
175.50 ± 30% -100.0% 0.00 interrupts.CPU48.NMI:Non-maskable_interrupts
175.50 ± 30% -100.0% 0.00 interrupts.CPU48.PMI:Performance_monitoring_interrupts
120.50 ± 17% -100.0% 0.00 interrupts.CPU49.NMI:Non-maskable_interrupts
120.50 ± 17% -100.0% 0.00 interrupts.CPU49.PMI:Performance_monitoring_interrupts
4208 ± 49% -100.0% 0.00 interrupts.CPU5.NMI:Non-maskable_interrupts
4208 ± 49% -100.0% 0.00 interrupts.CPU5.PMI:Performance_monitoring_interrupts
123.67 ± 16% -100.0% 0.00 interrupts.CPU50.NMI:Non-maskable_interrupts
123.67 ± 16% -100.0% 0.00 interrupts.CPU50.PMI:Performance_monitoring_interrupts
121.67 ± 17% -100.0% 0.00 interrupts.CPU51.NMI:Non-maskable_interrupts
121.67 ± 17% -100.0% 0.00 interrupts.CPU51.PMI:Performance_monitoring_interrupts
124.67 ± 16% -100.0% 0.00 interrupts.CPU52.NMI:Non-maskable_interrupts
124.67 ± 16% -100.0% 0.00 interrupts.CPU52.PMI:Performance_monitoring_interrupts
126.00 ± 19% -100.0% 0.00 interrupts.CPU53.NMI:Non-maskable_interrupts
126.00 ± 19% -100.0% 0.00 interrupts.CPU53.PMI:Performance_monitoring_interrupts
127.50 ± 18% -100.0% 0.00 interrupts.CPU54.NMI:Non-maskable_interrupts
127.50 ± 18% -100.0% 0.00 interrupts.CPU54.PMI:Performance_monitoring_interrupts
131.17 ± 20% -100.0% 0.00 interrupts.CPU55.NMI:Non-maskable_interrupts
131.17 ± 20% -100.0% 0.00 interrupts.CPU55.PMI:Performance_monitoring_interrupts
124.00 ± 17% -100.0% 0.00 interrupts.CPU56.NMI:Non-maskable_interrupts
124.00 ± 17% -100.0% 0.00 interrupts.CPU56.PMI:Performance_monitoring_interrupts
126.00 ± 17% -100.0% 0.00 interrupts.CPU57.NMI:Non-maskable_interrupts
126.00 ± 17% -100.0% 0.00 interrupts.CPU57.PMI:Performance_monitoring_interrupts
126.17 ± 24% -100.0% 0.00 interrupts.CPU58.NMI:Non-maskable_interrupts
126.17 ± 24% -100.0% 0.00 interrupts.CPU58.PMI:Performance_monitoring_interrupts
132.67 ± 27% -100.0% 0.00 interrupts.CPU59.NMI:Non-maskable_interrupts
132.67 ± 27% -100.0% 0.00 interrupts.CPU59.PMI:Performance_monitoring_interrupts
5412 ± 43% -100.0% 0.00 interrupts.CPU6.NMI:Non-maskable_interrupts
5412 ± 43% -100.0% 0.00 interrupts.CPU6.PMI:Performance_monitoring_interrupts
123.83 ± 20% -100.0% 0.00 interrupts.CPU60.NMI:Non-maskable_interrupts
123.83 ± 20% -100.0% 0.00 interrupts.CPU60.PMI:Performance_monitoring_interrupts
124.33 ± 17% -100.0% 0.00 interrupts.CPU61.NMI:Non-maskable_interrupts
124.33 ± 17% -100.0% 0.00 interrupts.CPU61.PMI:Performance_monitoring_interrupts
134.17 ± 25% -100.0% 0.00 interrupts.CPU62.NMI:Non-maskable_interrupts
134.17 ± 25% -100.0% 0.00 interrupts.CPU62.PMI:Performance_monitoring_interrupts
122.17 ± 19% -100.0% 0.00 interrupts.CPU63.NMI:Non-maskable_interrupts
122.17 ± 19% -100.0% 0.00 interrupts.CPU63.PMI:Performance_monitoring_interrupts
124.00 ± 19% -100.0% 0.00 interrupts.CPU64.NMI:Non-maskable_interrupts
124.00 ± 19% -100.0% 0.00 interrupts.CPU64.PMI:Performance_monitoring_interrupts
126.00 ± 23% -100.0% 0.00 interrupts.CPU65.NMI:Non-maskable_interrupts
126.00 ± 23% -100.0% 0.00 interrupts.CPU65.PMI:Performance_monitoring_interrupts
124.00 ± 20% -100.0% 0.00 interrupts.CPU66.NMI:Non-maskable_interrupts
124.00 ± 20% -100.0% 0.00 interrupts.CPU66.PMI:Performance_monitoring_interrupts
127.83 ± 21% -100.0% 0.00 interrupts.CPU67.NMI:Non-maskable_interrupts
127.83 ± 21% -100.0% 0.00 interrupts.CPU67.PMI:Performance_monitoring_interrupts
123.33 ± 20% -100.0% 0.00 interrupts.CPU68.NMI:Non-maskable_interrupts
123.33 ± 20% -100.0% 0.00 interrupts.CPU68.PMI:Performance_monitoring_interrupts
122.33 ± 19% -100.0% 0.00 interrupts.CPU69.NMI:Non-maskable_interrupts
122.33 ± 19% -100.0% 0.00 interrupts.CPU69.PMI:Performance_monitoring_interrupts
7639 ± 12% -100.0% 0.00 interrupts.CPU7.NMI:Non-maskable_interrupts
7639 ± 12% -100.0% 0.00 interrupts.CPU7.PMI:Performance_monitoring_interrupts
124.67 ± 18% -100.0% 0.00 interrupts.CPU70.NMI:Non-maskable_interrupts
124.67 ± 18% -100.0% 0.00 interrupts.CPU70.PMI:Performance_monitoring_interrupts
141.83 ± 20% -100.0% 0.00 interrupts.CPU71.NMI:Non-maskable_interrupts
141.83 ± 20% -100.0% 0.00 interrupts.CPU71.PMI:Performance_monitoring_interrupts
131.50 ± 18% -100.0% 0.00 interrupts.CPU72.NMI:Non-maskable_interrupts
131.50 ± 18% -100.0% 0.00 interrupts.CPU72.PMI:Performance_monitoring_interrupts
134.17 ± 19% -100.0% 0.00 interrupts.CPU73.NMI:Non-maskable_interrupts
134.17 ± 19% -100.0% 0.00 interrupts.CPU73.PMI:Performance_monitoring_interrupts
126.83 ± 19% -100.0% 0.00 interrupts.CPU74.NMI:Non-maskable_interrupts
126.83 ± 19% -100.0% 0.00 interrupts.CPU74.PMI:Performance_monitoring_interrupts
126.00 ± 19% -100.0% 0.00 interrupts.CPU75.NMI:Non-maskable_interrupts
126.00 ± 19% -100.0% 0.00 interrupts.CPU75.PMI:Performance_monitoring_interrupts
123.83 ± 17% -100.0% 0.00 interrupts.CPU76.NMI:Non-maskable_interrupts
123.83 ± 17% -100.0% 0.00 interrupts.CPU76.PMI:Performance_monitoring_interrupts
125.17 ± 16% -100.0% 0.00 interrupts.CPU77.NMI:Non-maskable_interrupts
125.17 ± 16% -100.0% 0.00 interrupts.CPU77.PMI:Performance_monitoring_interrupts
125.50 ± 18% -100.0% 0.00 interrupts.CPU78.NMI:Non-maskable_interrupts
125.50 ± 18% -100.0% 0.00 interrupts.CPU78.PMI:Performance_monitoring_interrupts
162.33 ± 62% -100.0% 0.00 interrupts.CPU79.NMI:Non-maskable_interrupts
162.33 ± 62% -100.0% 0.00 interrupts.CPU79.PMI:Performance_monitoring_interrupts
6455 ± 37% -100.0% 0.00 interrupts.CPU8.NMI:Non-maskable_interrupts
6455 ± 37% -100.0% 0.00 interrupts.CPU8.PMI:Performance_monitoring_interrupts
103.67 ± 38% -100.0% 0.00 interrupts.CPU80.NMI:Non-maskable_interrupts
103.67 ± 38% -100.0% 0.00 interrupts.CPU80.PMI:Performance_monitoring_interrupts
143.83 ± 40% -100.0% 0.00 interrupts.CPU81.NMI:Non-maskable_interrupts
143.83 ± 40% -100.0% 0.00 interrupts.CPU81.PMI:Performance_monitoring_interrupts
122.17 ± 17% -100.0% 0.00 interrupts.CPU82.NMI:Non-maskable_interrupts
122.17 ± 17% -100.0% 0.00 interrupts.CPU82.PMI:Performance_monitoring_interrupts
125.50 ± 17% -100.0% 0.00 interrupts.CPU83.NMI:Non-maskable_interrupts
125.50 ± 17% -100.0% 0.00 interrupts.CPU83.PMI:Performance_monitoring_interrupts
125.33 ± 17% -100.0% 0.00 interrupts.CPU84.NMI:Non-maskable_interrupts
125.33 ± 17% -100.0% 0.00 interrupts.CPU84.PMI:Performance_monitoring_interrupts
124.17 ± 17% -100.0% 0.00 interrupts.CPU85.NMI:Non-maskable_interrupts
124.17 ± 17% -100.0% 0.00 interrupts.CPU85.PMI:Performance_monitoring_interrupts
127.17 ± 17% -100.0% 0.00 interrupts.CPU86.NMI:Non-maskable_interrupts
127.17 ± 17% -100.0% 0.00 interrupts.CPU86.PMI:Performance_monitoring_interrupts
124.17 ± 17% -100.0% 0.00 interrupts.CPU87.NMI:Non-maskable_interrupts
124.17 ± 17% -100.0% 0.00 interrupts.CPU87.PMI:Performance_monitoring_interrupts
123.00 ± 17% -100.0% 0.00 interrupts.CPU88.NMI:Non-maskable_interrupts
123.00 ± 17% -100.0% 0.00 interrupts.CPU88.PMI:Performance_monitoring_interrupts
125.33 ± 17% -100.0% 0.00 interrupts.CPU89.NMI:Non-maskable_interrupts
125.33 ± 17% -100.0% 0.00 interrupts.CPU89.PMI:Performance_monitoring_interrupts
7938 ± 13% -100.0% 0.00 interrupts.CPU9.NMI:Non-maskable_interrupts
7938 ± 13% -100.0% 0.00 interrupts.CPU9.PMI:Performance_monitoring_interrupts
124.50 ± 34% -100.0% 0.00 interrupts.CPU90.NMI:Non-maskable_interrupts
124.50 ± 34% -100.0% 0.00 interrupts.CPU90.PMI:Performance_monitoring_interrupts
125.67 ± 17% -100.0% 0.00 interrupts.CPU91.NMI:Non-maskable_interrupts
125.67 ± 17% -100.0% 0.00 interrupts.CPU91.PMI:Performance_monitoring_interrupts
122.83 ± 17% -100.0% 0.00 interrupts.CPU92.NMI:Non-maskable_interrupts
122.83 ± 17% -100.0% 0.00 interrupts.CPU92.PMI:Performance_monitoring_interrupts
121.83 ± 17% -100.0% 0.00 interrupts.CPU93.NMI:Non-maskable_interrupts
121.83 ± 17% -100.0% 0.00 interrupts.CPU93.PMI:Performance_monitoring_interrupts
112.00 ± 29% -100.0% 0.00 interrupts.CPU94.NMI:Non-maskable_interrupts
112.00 ± 29% -100.0% 0.00 interrupts.CPU94.PMI:Performance_monitoring_interrupts
293.33 ± 29% -100.0% 0.00 interrupts.CPU95.NMI:Non-maskable_interrupts
293.33 ± 29% -100.0% 0.00 interrupts.CPU95.PMI:Performance_monitoring_interrupts
7124 ± 23% -100.0% 0.00 interrupts.CPU96.NMI:Non-maskable_interrupts
7124 ± 23% -100.0% 0.00 interrupts.CPU96.PMI:Performance_monitoring_interrupts
5931 ± 44% -100.0% 0.00 interrupts.CPU97.NMI:Non-maskable_interrupts
5931 ± 44% -100.0% 0.00 interrupts.CPU97.PMI:Performance_monitoring_interrupts
4942 ± 44% -100.0% 0.00 interrupts.CPU98.NMI:Non-maskable_interrupts
4942 ± 44% -100.0% 0.00 interrupts.CPU98.PMI:Performance_monitoring_interrupts
5446 ± 42% -100.0% 0.00 interrupts.CPU99.NMI:Non-maskable_interrupts
5446 ± 42% -100.0% 0.00 interrupts.CPU99.PMI:Performance_monitoring_interrupts
206627 ± 9% -100.0% 4.00 ± 70% interrupts.NMI:Non-maskable_interrupts
206627 ± 9% -100.0% 4.00 ± 70% interrupts.PMI:Performance_monitoring_interrupts
will-it-scale.16.processes 4.8e+07 +-----------------------------------------------------------------+
|+.++++.++ : + .+++.++++.+|
4.7e+07 |-+ : : +++ |
4.6e+07 |-+ +.+++.++++.+++.++++ |
| |
4.5e+07 |-+ |
| |
4.4e+07 |-+ |
| |
4.3e+07 |-+ |
4.2e+07 |-+ O O O OOO O |
| OOO OOO O O |
4.1e+07 |O+ OO O O O O |
| |
4e+07 +-----------------------------------------------------------------+ will-it-scale.per_process_ops
3e+06 +----------------------------------------------------------------+
| .+ ++.++ : : + +. |
2.95e+06 |++ + : : ++.+ + ++++.+|
2.9e+06 |-+ ++.+++.++++.++++.++ |
| |
2.85e+06 |-+ |
2.8e+06 |-+ |
| |
2.75e+06 |-+ |
2.7e+06 |-+ |
| |
2.65e+06 |-+ O O O |
2.6e+06 |-+ O O O O O O O O |
|O OO O OO O O O O | 2.55e+06 +----------------------------------------------------------------+
will-it-scale.workload
4.8e+07 +-----------------------------------------------------------------+
|+.++++.++ : + .+++.++++.+|
4.7e+07 |-+ : : +++ |
4.6e+07 |-+ +.+++.++++.+++.++++ |
| |
4.5e+07 |-+ |
| |
4.4e+07 |-+ |
| |
4.3e+07 |-+ |
4.2e+07 |-+ O O O OOO O |
| OOO OOO O O |
4.1e+07 |O+ OO O O O O |
| |
4e+07 +-----------------------------------------------------------------+ perf-sched.total_wait_time.max.ms
10000 +-------------------------------------------------------------------+
| : : : |
9500 |-+ + + :: + + : + : + |
| : .+ + : :: ::.+ : : + :: :: + : |
| : + : + : + :+ :+ : : : :::: :: : : + |
9000 |-+: : : : +: : : : : : + ::: + + : : + : :+ :: .+|
|: : : : + + :: :: :: :: : : : : :+.+ + : + :: :+ |
8500 |:.+ : + + : :: : : :: : : : +: : + :: |
|+ +.+ + :: + + :: + : : :: + |
8000 |-+ :: :: : :: |
| : : : :: |
| + + + : |
7500 |-+ + |
| |
7000 +-------------------------------------------------------------------+ perf-sched.total_wait_time.average.ms
235 +---------------------------------------------------------------------+
| +. + |
230 |-+ + : + :+ |
225 |-+ + + :: + : : : : +. |
|+ :+ :+ + :: :+ : : : : +.+ + |
220 |-+ :: + + + +. :: : : + .+: + +.: :+ : + +|
| ++ : : : : + +: : + + : + + : + + + : : : :|
215 |-+ : : : ::: + + :: : : : ++ : :|
| : : : :+ + + : + : : |
210 |-+ :: : : + + :: |
205 |-+ : :: + :: |
| : : :: |
200 |-+ + : : |
| + + |
195 +---------------------------------------------------------------------+ perf-sched.total_sch_delay.max.ms
1100 +--------------------------------------------------------------------+
1000 |-+ +|
| + + +:|
900 |-+ : : + :|
800 |-+ : : : :|
700 |-+ : : : :|
600 |-+ :: : : :|
| :: : : :|
500 |-+ : : :: : |
400 |-+ : : : : : |
300 |-+ : : : : : |
200 |-+ : : : : : |
| : : : : : |
100 |-+ : : : : : |
0 +--------------------------------------------------------------------+
perf-sched.total_sch_delay.average.ms 0.1 +--------------------------------------------------------------------+
0.09 |-+ + + :|
| : : ::|
0.08 |-+ : : ::|
0.07 |-+ :: : + :|
| :: : : :|
0.06 |-+ :: : : :|
0.05 |-+ :: :: : |
0.04 |-+ : : : : : |
| : : : : : |
0.03 |-+ : : : : : |
0.02 |-+ : : : : : |
| + : .+ : + : |
0.01 |+.+ +.+++.++. ++.++ .+++.++ .++ .+ :+.+ + :.+ +.+++.++.+ :.++ |
0 +--------------------------------------------------------------------+
perf-sched.total_wait_and_delay.count.ms 12600 +-------------------------------------------------------------------+
12400 |-+ + |
| : |
12200 |-+ + : + |
12000 |-+ : :: + : |
11800 |-+ : :: + : :: |
11600 |-+ :: :: :: :: + : : |
| :: :: :: :: ++ :: + : |
11400 |-+ : : : : : +: : : + + : : : : |
11200 |-.+: : : : + : :: : + :: ++. + +.+ + + + + + : :|
11000 |++:: : : :: +: : + : :: : + + :: + :+. : + : :+.+ +|
10800 |-+ : :: : :::: + : + :+ + :.+ + :: |
| + :: + :: : + + + + |
10600 |-+ ++ + + | 10400 +-------------------------------------------------------------------+
perf-sched.total_wait_and_delay.max.ms
10000 +-------------------------------------------------------------------+
| : : : |
9500 |-+ + + :: + + : + : + |
| : .+ + : :: ::.+ : : + :: :: + : |
| : + : + : + :+ :+ : : : :::: :: : : + |
9000 |-+: : : : +: : : : : : + ::: + + : : + : :+ :: .+|
|: : : : + + :: :: :: :: : : : : :+.+ + : + :: :+ |
8500 |:.+ : + + : :: : : :: : : : +: : + :: |
|+ +.+ + :: + + :: + : : :: + |
8000 |-+ :: :: : :: |
| : : : :: |
| + + + : |
7500 |-+ + |
| |
7000 +-------------------------------------------------------------------+ perf-sched.total_wait_and_delay.average.ms
235 +---------------------------------------------------------------------+
| +. + |
230 |-+ + + : + :+ |
225 |-+ + :: :: + : : : : +. |
|+ :+ : + + :: :+ : : : : +.+ + |
220 |-+ :: + + + +. :: : : + .++ + +.+ :+ : + +|
| ++ : : : : + +: : + + : + : + + : : : :|
215 |-+ : : : ::: + + :: : : : ++ : :|
| : : : :+ + + : + : : |
210 |-+ :: : : + + :: |
205 |-+ : :: + :: |
| : : :: |
200 |-+ + : : |
| + + |
195 +---------------------------------------------------------------------+ perf-sched.wait_time.avg.ms.pipe_read.new_sync_read.vfs_read.ksys_read
58 +----------------------------------------------------------------------+
| + |
57 |-+ : + |
56 |-++ : : .+ +|
| : + : + + :: + : :|
55 |-:: : + : : :: : :: : : ::|
| : : :::: : : :: :: : :: :: |
54 |-: : :::: : : : : :: : :: :: |
| : : : + : + : + + : + : +. : : : |
53 |:+ + : : :+.+ .+ : : + + : : ++. : ++.+++.+ +.++ : + |
52 |++ +: +.+ ++ : : :: + ++.+ ++.+ :: + |
| + : :: + |
51 |-+ : + |
| + |
50 +----------------------------------------------------------------------+ perf-sched.sch_delay.max.ms.pipe_read.new_sync_read.vfs_read.ksys_read
1.6 +---------------------------------------------------------------------+
| + : |
1.4 |-+ : : |
1.2 |-+ : : |
| : :: |
1 |-+ :: :: |
| :: :: |
0.8 |-+ :: : : |
| : : : : |
0.6 |-+ : : : : |
0.4 |-+ : : : : |
| : : : : |
0.2 |-+ : : : : |
| .+ .+ .+ : : +. .+ +. .+: + .+|
0 +---------------------------------------------------------------------+ perf-sched.sch_delay.avg.ms.pipe_read.new_sync_read.vfs_read.ksys_read
0.005 +------------------------------------------------------------------+
| : |
0.0045 |-+ : |
0.004 |-+ + + : + + + + |
| : : :: : : : : |
0.0035 |-+ : : :: : : : : |
| :: :: :: : :: : :: |
0.003 |-++ :: :: :: : :: : : : +|
| : :: :: : : : :: : : : :|
0.0025 |-:: : : : : : : : : : : :: : ::|
0.002 |-:: : : : + : : : : : : : :: + ::|
| : : : : : : : : : : : : : :: : : |
0.0015 |-: :: : : : : : : : : : : : : :: |
|: :: : : : : : : : : : : : : : |
0.001 +------------------------------------------------------------------+ perf-sched.wait_and_delay.count.pipe_read.new_sync_read.vfs_read.ksys_read
3150 +--------------------------------------------------------------------+
|+ : |
3100 |:+ + :: + |
3050 |:+ + + : :: :: +.+ ++. ++ +.+ +. |
| : +: + + +: : : :+.++ : : + : : : + + .+ + + |
3000 |-: : : :+.++ + : + :: : + : + ++.+ + : : : |
| : : : : : : + : : : : : :: :: |
2950 |-: : : + : :: :: : : : :: : : |
| :: : :: :: :: :: ::: : ::|
2900 |-+: : + :: : : : :+ :|
2850 |-+: + : + + : + +|
| + : : |
2800 |-+ : + |
| + |
2750 +--------------------------------------------------------------------+ perf-sched.wait_and_delay.avg.ms.pipe_read.new_sync_read.vfs_read.ksys_read
58 +----------------------------------------------------------------------+
| + |
57 |-+ : + |
56 |-++ : : .+ +|
| : + : + + :: + : :|
55 |-:: : + : : :: : :: : : ::|
| : : :::: : : :: :: : :: :: |
54 |-: : :::: : : : : :: : :: :: |
| : : : + : + : + + : + : +. : : : |
53 |:+ + : : :+.+ .+ : : + + : : ++. : ++.+++.+ +.++ : + |
52 |++ +: +.+ ++ : : :: + ++.+ ++.+ :: + |
| + : :: + |
51 |-+ : + |
| + |
50 +----------------------------------------------------------------------+ perf-sched.wait_time.max.ms.smpboot_thread_fn.kthread.ret_from_fork
5500 +--------------------------------------------------------------------+
| + |
5000 |-+ : |
4500 |-+ : |
| : |
4000 |-+ :: |
3500 |-+ :: |
| :: |
3000 |-+ : : |
2500 |-+ : : |
| +: : + |
2000 |-+ :: : :: |
1500 |-+ : : : :: |
| : : : : :|
1000 +--------------------------------------------------------------------+ perf-sched.wait_time.avg.ms.smpboot_thread_fn.kthread.ret_from_fork
720 +---------------------------------------------------------------------+
| + +. + |
710 |-+ .+ + + +: + : + :+ |
|+ + : : : : : : :: : : : + +. |
700 |:+ : : ++. : +.:: + : :: + : :.+ : + +. ++ : +|
| : + + : +++ +: : : +: + + : + +.+ : + : + : |
690 |-: : : : :: + : : : : : : : : : |
| : : : : :: : : : : : : : :: + |
680 |-:: :: : : : : : : : :: : : |
| : :: + : : :: : : : : |
670 |-++ : : + :+ : + : |
| + :+ :: : + |
660 |-+ + + + |
| |
650 +---------------------------------------------------------------------+ perf-sched.sch_delay.avg.ms.smpboot_thread_fn.kthread.ret_from_fork
0.018 +-------------------------------------------------------------------+
| + + + |
0.016 |-+ : : : |
| : : : |
0.014 |-+ :: : : + |
| : : :: : : |
0.012 |-+ : : :: : : |
| + + : : : : :: :: |
0.01 |-+: : : : : : : : :: + |
| :: :: + + + ++.+ : : : : ++ +.+ : :: |
0.008 |-: : : : : :: : : : : : : : : : :: + |
| : : +: : : : :: : : : : : : : : :: : +|
0.006 |:+ + + + + +.+ + : ++. +: ++. +.: +.+ +.: ++ : :+ |
|+ + + + + + ++ + + + + + |
0.004 +-------------------------------------------------------------------+ perf-sched.wait_and_delay.count.smpboot_thread_fn.kthread.ret_from_fork
2700 +--------------------------------------------------------------------+
2680 |-+: + + ++ ++ ++ + ++ ++ +.+ +.+ + ++.++ |
| :: : : :: :: :: : :: :: : : : : : : : |
2660 |-:: : : :: :: : : : :: :: : : : : : : : |
2640 |-:: :: : :: : :: : :: : : : : : : : : : : : |
2620 |-:: :: : : : : :: : :: : : : : : : : : : : : |
2600 |:+: :: : : : : : : :: : : : : : : : : : : : |
|+ :+ :: ::: :: + :: : : : : : : :: : :: : : |
2580 |:+ :: : : : :: :: :: : : :: : : : : : : : : |
2560 |:+ ::: : : : :: :: :: : : :: : : : : : : : : |
2540 |:+ ::: : : : :: :: : : : : :: : : : : : : |
2520 |-+ : : : : : : : : :: : :: : : : : : : |
| : :: : : : : + :: : :: : : : : : : |
2500 |-+ + ++ ++.+ + + ++ + ++ + ++.+ +.++.+ +.+| 2480 +--------------------------------------------------------------------+
perf-sched.wait_and_delay.max.ms.smpboot_thread_fn.kthread.ret_from_fork
5500 +--------------------------------------------------------------------+
| + |
5000 |-+ : |
4500 |-+ : |
| : |
4000 |-+ :: |
3500 |-+ :: |
| :: |
3000 |-+ : : |
2500 |-+ : : |
| +: : + |
2000 |-+ :: : :: |
1500 |-+ : : : :: |
| : : : : :|
1000 +--------------------------------------------------------------------+ perf-sched.wait_and_delay.avg.ms.smpboot_thread_fn.kthread.ret_from_fork
720 +---------------------------------------------------------------------+
| + +. + |
710 |-+ .+ + + +: + : + :+ |
|+ + : : : : : : :: : : : + +. |
700 |:+ : : ++. : +.:: + : :: + : :.+ : + +. ++ : +|
| : + + : +++ +: : : +: + + : + +.+ : + : + : |
690 |-: : : : :: + : : : : : : : : : |
| : : : : :: : : : : : : : :: + |
680 |-:: :: : : : : : : : :: : : |
| : :: + : : :: : : : : |
670 |-++ : : + :+ : + : |
| + :+ :: : + |
660 |-+ + + + |
| |
650 +---------------------------------------------------------------------+ 10 +----------------------------------------------------------------------+
| |
9 |-+ + + |
| + ++. : : |
8 |++ : + : + + :: : : + + |
| : + +: :: : : + + + + + : +.+ + : + :: :+ |
7 |-: : : : ::: : .+ : : +: : +.+ : : ++ + + ::: : |
| + : : : : : :+ : : : : :: :: +: + : + : |
6 |-+ : : : : + :: :: :+ + + :.++ +: : +|
| :: :: + : + + + + :|
5 |-+ :: : + : : |
| : : :: |
4 |-+ + + + |
| |
3 +----------------------------------------------------------------------+ 14 +----------------------------------------------------------------------+
| |
12 |-+ + + + |
| : : : |
10 |-+ :: :: :: |
| :: :+ :+ |
8 |-+ :: : : : : |
| :: : : : : |
6 |-+ : : : : : : |
| : : + : : : : + + |
4 |-+ : + : + : : : : +: : |
| : : : : : : : : : : :: |
2 |-+ : : : : : :: : : : : : : : |
| : : : : : :: : : : : : : :|
0 +----------------------------------------------------------------------+ 2600 +--------------------------------------------------------------------+
| + : |
2400 |-+ + : : |
2200 |-+ : : :: |
| : :: :: |
2000 |-+ :: :: : : |
| :: :: + : : |
1800 |-+ : : :: + : + + : :|
| : : : : + : :: :+ :+ + + :|
1600 |-+ : :: : : : : : :: + + + : +.+ : : +|
1400 |-++: :: ::: : +: : : + :+ :: + : : + : |
| + + +: : : : + + + + + ++.+ +.+ +.++ + + ::: : |
1200 |++ : + :++ + : : : : + +.+ |
| + + : :: |
1000 +--------------------------------------------------------------------+ 10 +----------------------------------------------------------------------+
| |
9 |-+ + + |
| + ++. : : |
8 |++ : + : + + :: : : + ++ |
| : + +: :: : : + + + + + : +.+ + : + :: : : |
7 |-: : : : ::: : .+ : : +: : +.+ : : ++ + + ::: : |
| + : : : : : :+ : : : : :: :: +: + : + : |
6 |-+ : : : : + :: :: :+ + + :.++ +: + +|
| :: :: + : + + + : :|
5 |-+ :: : + :: |
| : : :: |
4 |-+ + + + |
| |
3 +----------------------------------------------------------------------+ perf-sched.wait_time.max.ms.worker_thread.kthread.ret_from_fork
10000 +-------------------------------------------------------------------+
| : : : |
9500 |-+ + + : + + : + : + |
| : .+ + : :: ::.+ : : + :: :: + : |
| : + : + : + :: :+ : : : :::: :: : : + |
9000 |-+: : : : +: : : :: : + ::: + + : : + : :+ :: .+|
|: : : : + + :: :: + : :: : : : : :+.+ + : + :: :+ |
8500 |:.+ : + + : :: +: : :: : : : +: : + :: |
|+ +.+ + :: + + :: + : : :: + |
8000 |-+ :: :: : :: |
| : : : :: |
| + + + : |
7500 |-+ + |
| |
7000 +-------------------------------------------------------------------+ perf-sched.wait_time.avg.ms.worker_thread.kthread.ret_from_fork
500 +---------------------------------------------------------------------+
| + |
480 |-+ : |
| + : |
460 |-+ : + :: |
| + : : + : |
440 |-+: : : .+++.+ + + : :: : |
| :: : : ++ : + :: + : : :: : +|
420 |:: :+. : : + + : +: :: + : :: + + + : + :+ |
|: :: + : + :+ :+ + : ++ : :: +.+ :+ :: : : + |
400 |++ : : : : ::+ : : :: : : + : + + : |
| + + +.: : + + + + + +: : + |
380 |-+ + + :: |
| :: |
360 +---------------------------------------------------------------------+ perf-sched.sch_delay.max.ms.worker_thread.kthread.ret_from_fork
1000 +--------------------------------------------------------------------+
900 |-+ + + |
| : : + |
800 |-+ : : : |
700 |-+ : : : |
| :: : :: |
600 |-+ :: : :: |
500 |-+ :: : :: |
400 |-+ : : : : :: |
| : : : : : : |
300 |-+ : : : : : : |
200 |-+ : : : : : :|
| : : : : : :|
100 |-+ : : : : : :|
0 +--------------------------------------------------------------------+
perf-sched.sch_delay.avg.ms.worker_thread.kthread.ret_from_fork 1.6 +---------------------------------------------------------------------+
| : : |
1.4 |-+ : : + |
1.2 |-+ : : : |
| : :: : |
1 |-+ : :: :: |
| : :: :: |
0.8 |-+ :: : : :: |
| : : : : :: |
0.6 |-+ : : : : : : |
0.4 |-+ : : : : : : |
| : : : : : :|
0.2 |-+ : : : : : :|
| : : : : : :|
0 +---------------------------------------------------------------------+ perf-sched.wait_and_delay.count.worker_thread.kthread.ret_from_fork
700 +---------------------------------------------------------------------+
| : |
680 |-+ +: |
| + :: |
660 |-+ : : : +|
| + : + + + : :::|
640 |-+ + : + : : .+ :+ ++. :: + + +: : :: |
|+ + :: : ++. : : + + + + + + +. :+ + :+ : + |
620 |-: + + : : ++.+ :+ ++.+ ++ ++ ++.+ :.+ + |
| : : +.: + + |
600 |-++ : + |
| :: |
580 |-+ : |
| + |
560 +---------------------------------------------------------------------+ perf-sched.wait_and_delay.max.ms.worker_thread.kthread.ret_from_fork
10000 +-------------------------------------------------------------------+
| : : : |
9500 |-+ + + : + + : + : + |
| : .+ + : :: ::.+ : : + :: :: + : |
| : + : + : + :: :+ : : : :::: :: : : + |
9000 |-+: : : : +: : : :: : + ::: + + : : + : :+ :: .+|
|: : : : + + :: :: + : :: : : : : :+.+ + : + :: :+ |
8500 |:.+ : + + : :: +: : :: : : : +: : + :: |
|+ +.+ + :: + + :: + : : :: + |
8000 |-+ :: :: : :: |
| : : : :: |
| + + + : |
7500 |-+ + |
| |
7000 +-------------------------------------------------------------------+ perf-sched.wait_and_delay.avg.ms.worker_thread.kthread.ret_from_fork
500 +---------------------------------------------------------------------+
| + |
480 |-+ : |
| + : |
460 |-+ : + :: |
| + : : + : |
440 |-+: : : + .+++.+ + + : :: : |
| :: : : : + : + :: + : : :: : +|
420 |:: :+. : : + : : +: :: + : :: + + + : + :+ |
|: :: + : + :+ :+ + : ++ : :: +.+ :+ :: : : + |
400 |++ : : : : ::+ : : :: : : + : + + : |
| + + +.: : + + + + + +: : + |
380 |-+ + + :: |
| :: |
360 +---------------------------------------------------------------------+ 235 +---------------------------------------------------------------------+
| : |
230 |-+ + + .+ + + : + |
| +: :: ++ : :: ++: + :: : |
225 |-+ + + : :+ : : :+ : : : :: + :: |
| + : : : :: : : : : : : :: + :: : :: |
220 |-+ : :: : : : : : :: : : : : : : :: :: :: |
| :: : +.+ + + : : : : :: : : :+ : :: : :: : :|
215 |-+ + :: : : : : : : :: :: :: + + ::: : :: : +|
| : : : +.+ : :: : : : :: : :: : : : : : : |
210 |:+ :: : : : :: : : + : + : : : : : :+ |
|+.: : : :: : : : + + + :: : : + |
205 |-++ +.+ ++ + + :+ : + + |
| + + |
200 +---------------------------------------------------------------------+ 1200 +--------------------------------------------------------------------+
| |
1000 |-+ +|
| :|
| :|
800 |-+ :|
| ::|
600 |-+ ::|
| : |
400 |-+ : |
| : |
| : |
200 |-+ : |
| : |
0 +--------------------------------------------------------------------+ 3.5 +---------------------------------------------------------------------+
| +|
3 |-+ :|
| :|
2.5 |-+ :|
| ::|
2 |-+ ::|
| ::|
1.5 |-+ : |
| : |
1 |-+ : |
| : |
0.5 |-+ : |
| : |
0 +---------------------------------------------------------------------+ 307 +---------------------------------------------------------------------+
|: :: : : : : : : : :: : : : |
306 |:+ :: : :: : : : : : :: : : : |
|: :: : :: : : : : : :: : : : |
|:: : : :: : : :: : : : :: : : :: :: :: |
305 |:: : : :: + : :: : : : :: : : :: :: :: |
|:: : : :: : : :: : : : :: : : :: :: :: |
304 |:: : : + : :: : : :+ :: : :: : : : : : : : + : |
| :: : : : :: : : :: : : : :: : : : : : : : : : |
303 |-:: : : : :: : : ::: : : : :: : : : : : : :+ : : |
| :: : : : : : : ::: : : : :: : : : : : : :: : : |
| : : : : : : : : : : : : : : : :: :: ::: : : |
302 |-+: : : : : : : : : : : : : : : :: :: : : : : |
| : : : : : : : : : : : : : : : :: :: : :: : |
301 +---------------------------------------------------------------------+ 235 +---------------------------------------------------------------------+
| : |
230 |-+ + + .+ + + : + |
| +: :: ++ : :: ++: + :: :: |
225 |-+ + + : :+ : : :+ : : : :: + :: |
| + : : : :: : : : : : : :: + :: : : :|
220 |-+ : :: : : : : : :: : : : : : : :: :: : +|
| :: : +.+ + + : : : : :: : : :+ : :: : :: : |
215 |-+ + :: : : : : : : :: :: :: + + ::: : :: : |
| : : : +.+ : :: : : : :: : :: : : : : : : |
210 |:+ :: : : : :: : : + : + : : : : : :+ |
|+.: : : :: : : : + + + :: : : + |
205 |-++ +.+ ++ + + :+ : + + |
| + + |
200 +---------------------------------------------------------------------+ perf-sched.wait_time.max.ms.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64
16.9 +--------------------------------------------------------------------+
16.8 |-+ :+.+ |
| : : + |
16.7 |:+ : : : + |
16.6 |++ : : :: : |
|: : : + :: + + + :: |
16.5 |-: +. : : :: + : :: :+ + + : :: |
16.4 |-: : + : : :: : : :+ : + :: : : :::|
16.3 |-: + : : : :+ : : : : : : : ::: : : :|
| : :: : : + : :: :: :+ : : + : : + + : +|
16.2 |-+:+ : : +. : : :: : + + +.: + :.+: + : : |
16.1 |-++ +: + :.+ : : + + + + : + + |
| + + + + +. :+ : |
16 |-+ + +: | 15.9 +--------------------------------------------------------------------+
perf-sched.wait_time.avg.ms.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64
0.81 +--------------------------------------------------------------------+
0.8 |-+ + |
| + : |
0.79 |-+ : : + + |
0.78 |-+ : : : : + |
0.77 |-+ :: :: :: : : |
0.76 |-++ :: +. : : :: :: : |
| : : :: + : : :: : : :: + +|
0.75 |-: : : :: : : + : : : : : ::: ::|
0.74 |-: : : :: : : : + : :+ : :+. ++.: :: : ::|
0.73 |++ +: : + +.++ + : :+ + : ::+. +: :: + + + + ++ |
0.72 |:+ : : + + :+ : : ++ +: + + : + : +. ++ |
| + + + + :+ + + :: + |
0.71 |-+ + : | 0.7 +--------------------------------------------------------------------+
perf-sched.sch_delay.max.ms.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64
0.16 +--------------------------------------------------------------------+
| + + + |
0.14 |-+ : + : : |
0.12 |-+ : : + : : + +|
| + : : : :: : :: :|
0.1 |-+: :: :: : :: :: + :+ :|
| : :: :: : :: :: : : : ::|
0.08 |-:: : ::: : :: : : : : : ::|
| :: : : : :: : : : : : : : : |
0.06 |-: : : : : : : : : : : ::: :: |
0.04 |-: : : : : : : : : : : : :: : |
|: :: : : : : : : : : : : : |
0.02 |:+ :: + : : : : : : : : : + |
|+ ++ ++.+++.+++.+ +.+++.+++.+ +++.+++ ++.+++.+++.++.+ + |
0 +--------------------------------------------------------------------+ perf-sched.sch_delay.avg.ms.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64
0.025 +-------------------------------------------------------------------+
| : |
| : + + |
0.02 |-+ : : : |
| + :: : : |
| : :: :: : |
0.015 |-+ : : : :: : + |
| :: + : : : : :: : |
0.01 |-++ : :: : : : : : : : ++ +|
| : : ::: : : : : : : :: :: :|
| : : : : : : : : : : : : :: :: |
0.005 |-: :: : : : : : : : : : :: :: |
|+ ++ + +++.+++.+++ ++.+++.+++ ++.+++.+ +.+++.+++.+++.+ + + |
| |
0 +-------------------------------------------------------------------+ perf-sched.wait_and_delay.count.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_6
310 +---------------------------------------------------------------------+
| |
305 |++ + + +.+ + ++.+ + +++ + |
|: :: + + :: : +. :: : :+ : : + + : |
300 |-: :+ + :+ + +.++ ++ ++.+ + : + + : ++.+++ ++ + + |
| : :: : : : : : : : : : :: :: |
295 |-: : : :: : : : : : : : : :: :: |
| :: : ::: : : : : : : : : :: :|
290 |-+: :: : : : : : : : : : ++ +|
| : :: + :: :: :: :: |
285 |-++ : :: :: :: : |
| + :: :: :: : |
280 |-+ : : : + |
| : : : |
275 +---------------------------------------------------------------------+ 16.9 +--------------------------------------------------------------------+
16.8 |-+ :+.+ |
| : : + |
16.7 |:+ : : : + |
16.6 |++ : : :: + : |
|: : : + :: + + : :: |
16.5 |-: +. : : +: + : :: :+ + + : :::|
16.4 |-: : + : :+ : : : :+ : + :: : : : :|
16.3 |-: + : : : :: : : : : : : : ::: : : +|
| : :: : : + : :: :: :+ : : + : : + + : |
16.2 |-+:+ : : +. : : : : + + +.: + :.+: + : : |
16.1 |-++ +: + :.+ : : + + + + : + + |
| + + + + +. :+. : |
16 |-+ + + | 15.9 +--------------------------------------------------------------------+
0.84 +--------------------------------------------------------------------+
| + |
0.82 |-+ : |
| + : + + |
0.8 |-+ : : : : |
| : :: :: : + |
0.78 |-+ :: + : : :: :: : |
| + :: :+ : : :: : : : + +|
0.76 |-+: : :: + : : : : : : :: : :|
| : : : :: : : + : : : : : :: : ::|
0.74 |-: :: : + .+ + : +. + : :++. +: :+.+ ++.: :: + : |
|+ +: : + ++ + + : : ++ +: + + : + : +. + + + + |
0.72 |-+ + + + + :+ + + :: +++ |
| + + |
0.7 +--------------------------------------------------------------------+ 6 +-----------------------------------------------------------------------+
| |
5 |-+ + |
| : |
| : |
4 |-+ :: |
| :: |
3 |-+ :: |
|: :: |
2 |:+ : : |
|: : : |
|: : : |
1 |:+ : : |
|: : : |
0 +-----------------------------------------------------------------------+ 270 +---------------------------------------------------------------------+
260 |-+ + : |
| :: : |
250 |-+ :+ + + + : : |
240 |-+ :: + +: : : : : |
| : : :+ : :: : + : |
230 |-+ : : : : : + : : : |
220 |++ : : : : : : + + + + : : : : |
210 |:+ : : : + : : +: :+.+ :++ :: : : : : |
| + : : : : :: : : : + : : + :+.: : +.: :|
200 |-+: +. : : : : :: : : : :+ : : :: : + :.+ + :|
190 |-+ :: +: :.++: + + :: :: + : : :+.+ .+ + +|
| : : + + : : : : + |
180 |-+ + + + + + + | 170 +---------------------------------------------------------------------+
6 +-----------------------------------------------------------------------+
| |
5 |-+ + |
| : |
| : |
4 |-+ :: |
| :: |
3 |-+ :: |
|: :: |
2 |:+ : : |
|: : : |
|: : : |
1 |:+ : : |
|: : : |
0 +-----------------------------------------------------------------------+ perf-sched.wait_time.max.ms.rcu_gp_kthread.kthread.ret_from_fork
5.2 +---------------------------------------------------------------------+
| |
5 |+.+++. +.++ +++.++.+++.+++.++.+ +++.++ +.+++.++ +.+++.++.+ +.+|
| + : : : : : : : : : : |
4.8 |-+ : : : : : : : : : : |
| : : : : : : : : : : |
4.6 |-+ : : : : : : : : : : |
| :: : : : : : : : |
4.4 |-+ :: : : : : : : : |
| :: : : : : : : : |
4.2 |-+ : :: :: :: : |
| : :: :: :: + |
4 |-+ + ++ ++ ++ |
| |
3.8 +---------------------------------------------------------------------+ perf-sched.wait_time.avg.ms.rcu_gp_kthread.kthread.ret_from_fork
3.2 +---------------------------------------------------------------------+
| + + + |
3.1 |-+ : + :: : + |
3 |-+ : : + :+ + :: : |
| :: :::: + : : + + :+ : : :: |
2.9 |+.+ : + : + : +.+ : : : + : + : : : : : :: |
2.8 |:+ : :+ : :: : :: : : + : : :+:: : : + + : : :|
|: : : + : + + +: : : + + : + + :: : : +.: : +|
2.7 |-+ +.: : : : : :: : : : : : : : + : + |
2.6 |-+ + :: :: : ++ ++.+ + +: + + |
| :: + : : : : |
2.5 |-+ ++ + + :: |
2.4 |-+ : |
| + |
2.3 +---------------------------------------------------------------------+ perf-sched.wait_and_delay.count.rcu_gp_kthread.kthread.ret_from_fork
220 +---------------------------------------------------------------------+
| + |
200 |-+ : |
180 |-+ : |
| :: + + |
160 |-+ :: : : + |
| :: :: : + : |
140 |-+ : : : : + : : : :: + |
| : : : : :+: : : +.+ +. + : : : +|
120 |+.+: : : : : + : : : : + + : + + + +.++ : + : :|
100 |-+:: :: ++. : +.+ : : :+ : + ::+ + + + : : :: |
| : :+ +++.: : : + + : + + : : + +.: :: |
80 |-+ + + + +.+ :: : :: + : |
| + + : + |
60 +---------------------------------------------------------------------+ perf-sched.wait_and_delay.max.ms.rcu_gp_kthread.kthread.ret_from_fork
5.2 +---------------------------------------------------------------------+
| + +. |
5 |+.+++.++.++ +++.++.+++.+++.++.+ :++.++ : +++.++ +.+++.++.+ +.+|
| : : : : : : : : : : |
| : : : : : : : : : : |
4.8 |-+ : : : : : : : : : : |
| : : : : : : : : : : |
4.6 |-+ : : : : : : : : :: |
| :: : : : : : : : |
4.4 |-+ :: : : : : : : : |
| :: : : : : : : : |
| :: :: :: :: : |
4.2 |-+ : :: :: :: + |
| : :: :: :: |
4 +---------------------------------------------------------------------+ perf-sched.wait_and_delay.avg.ms.rcu_gp_kthread.kthread.ret_from_fork
3.2 +---------------------------------------------------------------------+
| + |
3.1 |-+ + + :: + + |
3 |-+ : : + :+ + : : |
| :: ::: + : : + + + + :: :: :: |
2.9 |-.+: : + : +.+ : : : : : :: : : + : : : : |
|+ : + : : : : :: : : : ::: :: : : : + : : : :|
2.8 |:+ : :+ : : : +: : : : : : : : :: : : + .+ : :|
|: : : + : + + : :: : + + : + + :: : : + : +|
2.7 |-+ +.+ : : : : :: : : : : : : : : : |
2.6 |-+ :: :: : ++ ++.+ + +: + : |
| :: + : : : + |
2.5 |-+ ++ + + :: |
| + |
2.4 +---------------------------------------------------------------------+ perf-sched.wait_time.max.ms.schedule_timeout.kcompactd.kthread.ret_from_fork
505.2 +-------------------------------------------------------------------+
| |
505 |+.+++.+++.+ +.+++.+++.+++.+++.+++. ++.+++.+ +.+++.+ +++.+++.+ +.+|
| + + + : : : : |
| : : : : |
504.8 |-+ : : : : |
| : : : : |
504.6 |-+ : : : : |
| : : : |
504.4 |-+ : : : |
| : : : |
| : : : |
504.2 |-+ :: : |
| :: + |
504 +-------------------------------------------------------------------+ perf-sched.wait_time.avg.ms.schedule_timeout.kcompactd.kthread.ret_from_fork
479 +-------------------------------------------------------------------+
| .+ |
478.8 |+.++ +++.+ +.+++.+++ ++.+++ ++.+++.+ +.+++.+++ : +.++ + + +.+|
| : : : : : : : : : : : : : : : : :|
478.6 |-+ : : : : :: : : : : : : : : : : :|
478.4 |-+ :: : : + : : : : : : : :: :: : :|
| :: :: : : : : : : : :: :: : :|
478.2 |-+ : : : : : : :: : :: : : :|
| + : : : :: : :: : : : |
478 |-+ : :: : : :: :: : |
| + :: : : :: :+ : |
477.8 |-+ : : : : :: + |
477.6 |-+ : : : : : |
| : : + + + | 477.4 +-------------------------------------------------------------------+
perf-sched.sch_delay.max.ms.schedule_timeout.kcompactd.kthread.ret_from_fork
1.2 +---------------------------------------------------------------------+
| |
1 |-+ + |
| : |
| : |
0.8 |-+ : |
| :: |
0.6 |-+ :: |
| : : |
0.4 |-+ : : |
| : : |
| : : |
0.2 |-+ : : |
| .+: : .+|
0 +---------------------------------------------------------------------+ perf-sched.sch_delay.avg.ms.schedule_timeout.kcompactd.kthread.ret_from_fork
0.04 +-------------------------------------------------------------------+
| |
0.035 |-+ + |
| : |
0.03 |-+ : |
| :: |
0.025 |-+ : : +|
| : : :|
0.02 |-+ : : ::|
| : : : |
0.015 |-++ + + + + + +: + : |
| + : + ::+ + : + : :: :: : |
0.01 |++ ++ + + .+++.+ + ++.+++.+++ ++.+++.+ +.+++.+++.+++.+ + + |
| ++ + |
0.005 +-------------------------------------------------------------------+ perf-sched.wait_and_delay.count.schedule_timeout.kcompactd.kthread.ret_from_fork
80 +--------------------------------------------------------------------+
| : : : : : : : : : : : : : : : : :|
79.5 |-+ : : : : : : : : : : : : : : : :|
79 |-+ : : : : + : : : : : : : : :: : :|
| :: : : : : : : : : : :: :: : :|
78.5 |-+ :: :: : : : : : : : :: : :: :|
| : :: : : : : : : : :: : : :|
78 |-+ + :: :: : : : : :: : : : |
| : : :: :: :: :: : |
77.5 |-+ : : :: :: :: :: : |
77 |-+ + : :: :: :: :+ + |
| : : : : :: |
76.5 |-+ : : : : :: |
| : : : : : |
76 +--------------------------------------------------------------------+ perf-sched.wait_and_delay.max.ms.schedule_timeout.kcompactd.kthread.ret_from_for
505.2 +-------------------------------------------------------------------+
| .+ |
505 |+.+++.+++.+ +.+++.+++.+++.+++.+++. ++.+++.+ +.+++.+ +++.+++ : +.+|
| + + + : : : : |
| : : : : |
504.8 |-+ : : : : |
| : : : : |
504.6 |-+ : : :: |
| : : : |
504.4 |-+ : : : |
| : : : |
| : : : |
504.2 |-+ :: : |
| :: + |
504 +-------------------------------------------------------------------+ perf-sched.wait_and_delay.avg.ms.schedule_timeout.kcompactd.kthread.ret_from_for
479 +-------------------------------------------------------------------+
| +. .+ |
478.8 |+.++ +++.+ +.++ +++ ++.+++ ++.+++.+ +.+++.+++ : +.++ + + +.+|
| : : : : : : : : : : : : : : : : :|
478.6 |-+ : : : : :: : : : : : : : : : : :|
478.4 |-+ :: : : + : : : : : : : :: :: : :|
| :: :: : : : : : : : :: : :: :|
478.2 |-+ : : : : : : :: : :: : : :|
| + : :: : : :: : : : |
478 |-+ : :: : : :: :: : |
| + :: : : :: :+ + |
477.8 |-+ : : : : :: |
477.6 |-+ : : : : : |
| + + + + + | 477.4 +-------------------------------------------------------------------+
perf-sched.sch_delay.avg.ms.wait_for_partner.fifo_open.do_dentry_open.do_open.is
0.004 +------------------------------------------------------------------+
| : : |
| : : |
0.0035 |-+ : : |
| :: :: |
| :: :: |
| :: :: |
0.003 |-+ :: + + + :+.+ +|
| : : : : : : : :|
| : : :: : :: : : ::|
0.0025 |-+ : : :: : :: : : ::|
| : : : : : : : : : : : |
| : : : : : : : : : : : |
| : : : : : : : : : :: |
0.002 +------------------------------------------------------------------+ 72 +--------------------------------------------------------------------+
|: :: : : : : :: : : : : :: |
|: :: : : : : :: : : : : :: |
71.8 |:+ :: : : : : :: : : : : :: |
|:: : : : : :: :: : : : : : :: : : |
|:: : : : : :: :: : : : : : :: : : |
71.6 |:: : : : : :: :: : : : : : :: : : |
|:: : : : : :: :: : : :: : : : : : : |
71.4 |-:: : : : : : : : : :: : : : : : : : |
| :: : : : : : : : : :: : : : : : : : |
| :: : : : : : : : : :: : : : : : : : |
71.2 |-+: : : : : : : : : : : : : : : : : |
| : : : : : : : : : : : : : : : : : |
| : : : : : : : : : : : : : : : : : |
71 +--------------------------------------------------------------------+ 0.004 +------------------------------------------------------------------+
| : : |
| : : |
0.0035 |-+ : : |
| :: :: |
| :: :: |
| :: :: |
0.003 |-+ :: + + + :+.+ +|
| : : : : : : : :|
| : : :: : :: : : ::|
0.0025 |-+ : : :: : :: : : ::|
| : : : : : : : : : : : |
| : : : : : : : : : : : |
| : : : : : : : : : :: |
0.002 +------------------------------------------------------------------+ 7500 +--------------------------------------------------------------------+
|+ ++.++ +.+++.++ .+ +.+++.+++.+ +++.+++ ++.++ +++.++.+++.++ +|
7000 |:+ : +.+ + + : : : : : : : :|
|: : : : : : : : : :|
| : : : : : : : : : : |
6500 |-: : : : : : : : : : |
| : : : : : : : : : : |
6000 |-:: :: :: :: :: |
| :: :: :: :: :: |
5500 |-:: :: :: :: :: |
| : : : : : |
| : : : : : |
5000 |-++ + + + + |
| |
4500 +--------------------------------------------------------------------+ 750 +---------------------------------------------------------------------+
| + |
700 |++ :: + + + + + + + |
|: : + :+ :+ : : : +: :: + |
650 |:: : : : + : + : : : : :: :+ + |
| : : : : :+ : : :: + : + : : : + + : |
600 |-: + ++ +: : : : : :: :: :: .+ : : : : +|
| : : : ++. : : : : :+.+ :: :++ : : +: : :|
550 |-+++ : +.+ ++ :.+: : : +.: :: :: + : : |
| :: + + : : + :: : :: |
500 |-+ + :: :: + :: |
| : :: :: |
450 |-+ : : : |
| + + + |
400 +---------------------------------------------------------------------+ 0.045 +-------------------------------------------------------------------+
| + |
0.04 |-+ : |
0.035 |-+ : |
| + :: |
0.03 |-+ : +: |
| : : : |
0.025 |++ + + + + :: : : |
|: :: :: :: : + .+ :: : |
0.02 |-: +++. + : : : : : : + : : + :: + :: : +|
0.015 |-: + + ++ : ++ ++ ++.+ + : + + + : : +: : + : :+ |
| : ++ + + : : :: ++ :+ +.+ :+ + + + + |
0.01 |-: : + : + + + |
| : + |
0.005 +-------------------------------------------------------------------+ 70 +----------------------------------------------------------------------+
| + |
65 |-+ + : + + + + |
| :: :: :+ : : : |
| :: :: : : + : + : ++ :: |
60 |-+ : : + :+ + : : : : : :: :: |
| : + :: : + : : + : + : : .+ + : : : :|
55 |:+: : : +. : + : : : : .+ :: : + :+ : : : + : +|
|: : : : + : + : : :+ + :+ + ++ : :: + : |
50 |+.: :+ ++.+ : + + + +.+ : ::.+ |
| + :: : : + + |
| + :: |
45 |-+ : |
| + |
40 +----------------------------------------------------------------------+ 7500 +--------------------------------------------------------------------+
|+ ++.++ +.+++.++ .+ +.+++.+++.+ +++.+++ ++.++ +++.++.+++.++ +|
7000 |:+ : +.+ + + : : : : : : : :|
|: : : : : : : : : :|
| : : : : : : : : : : |
6500 |-: : : : : : : : : : |
| : : : : : : : : : : |
6000 |-:: :: :: :: :: |
| :: :: :: :: :: |
5500 |-:: :: :: :: :: |
| : : : : : |
| : : : : : |
5000 |-++ + + + + |
| |
4500 +--------------------------------------------------------------------+ 750 +---------------------------------------------------------------------+
| + |
700 |++ :: + + + + + + + |
|: : + :+ :+ : : : +: :: + |
650 |:: : : : + : + : : : : :: :+ + |
| : : : : :+ : : :: + : + : : : + + : |
600 |-: + ++ +: : : : : :: :: :: .+ : : : : +|
| : : : ++. : : : : :+.+ :: :++ : : +: : :|
550 |-+++ : +.+ ++ :.+: : : +.: :: :: + : : |
| :: + + : : + :: : :: |
500 |-+ + :: :: + :: |
| : :: :: |
450 |-+ : : : |
| + + + |
400 +---------------------------------------------------------------------+ 1200 +--------------------------------------------------------------------+
| + |
1100 |-+ : |
1000 |-+ :: + + + |
| + :: : :: :: |
900 |-+ : :: + :: + :: :: |
800 |-+ + :: : : : ::+: : : : : |
| + + :: : :+.+ :.+ : + :+ : + : + |
700 |-+ : + : :+ :: + : : +: : : + : : + |
600 |-+ + ++ :: :: : + : + ::+ : :+: |
|+ + + : : :: : : + ++ + : + |
500 |-+ :+.++ + + + :: : ++. + +:|
400 |-++ +. : + + + +|
| + |
300 +--------------------------------------------------------------------+ 45 +----------------------------------------------------------------------+
| + |
40 |-:: + + + + + |
| :: +: : + : : :: |
35 |++ : + + : + : : : : + :+ + |
| :+: : : : :: : :: : : + + : : : |
30 |-+ + : : : :: + :: :: :: : : : : : : :: |
| :: :: + : : : :: : : : : :: :: : : : :: |
25 |-+ : :: : :: : : :: : : : : : : :: :: : : : : |
| : :: : : :: : : : : : : : : : : : : :: : : :|
20 |-+ + :: : : : :: : : : : : : : : : : : +.: +|
| : : : : :: : + : + : : :: : : : + :|
15 |-+ + : : + ++ : + ++ + +.+: ++ +: + |
| : ++ :.+ + +. + + + |
10 +----------------------------------------------------------------------+ 1200 +--------------------------------------------------------------------+
| + |
1100 |-+ : |
1000 |-+ :: + + + |
| + :: : :: :: |
900 |-+ : :: + :: + :: :: |
800 |-+ + :: : : : ::+: : : : : |
| + + :: : :+.+ :.+ : + :+ : + : + |
700 |-+ : + : :+ :: + : : +: : : + : : + |
600 |-+ + ++ :: :: : + : + ::+ : :+: |
|+ + + : : :: : : + ++ + : + |
500 |-+ :+.++ + + + :: : ++. + +:|
400 |-++ +. : + + + +|
| + |
300 +--------------------------------------------------------------------+ 0.14 +--------------------------------------------------------------------+
| ++ +.+ + + + + + + + : + |
0.12 |-+ :: : : : : : : : : : : : |
| :: : :: : : : : : : : : |
| : : : : : :: : :: :: :: :: : ::: |
0.1 |-+ : : : + : :: : :: :: : ::: : : : |
| : : : : :: : :: :: : ::: : : : |
0.08 |-+ : : : :: + : : : : : : : :: : : : : |
| : : +: :: : : : : : : : : : : : + : |
0.06 |-+ +.+ : : :: :: : : : : : : : : + : : : |
| : :: :::: : : : : : : : :: :: :|
| + + : :: : : :+ +.: : + .+: : + :: :: :|
0.04 |+.++ + : + + : + + +.+ + :+ ++ + :+ ++ ++ +|
| + + + + |
0.02 +--------------------------------------------------------------------+ 0.032 +-------------------------------------------------------------------+
| : : + |
0.03 |-+ + : : : + |
| +.++ : :: :: : :+ |
0.028 |-+ + : : + :: :: :: :: :: |
| :: + : : :: : :: : : +: : : |
0.026 |-+ + : : :: : : : : : : : : + : : : : |
|+ :+.+ ++ + : + :: + : : : : : ::: : + : + |
0.024 |:+: + : :: :: : : : : : + + ::: : + |
|: + +.: + :: + : ++ : + + ::: : +|
0.022 |-+ + : : .+: : +: + + :: :|
| + :+ + :.+ + :: :|
0.02 |-+ + + ++ |
| |
0.018 +-------------------------------------------------------------------+ 1000.15 +-----------------------------------------------------------------+
1000.14 |-+ : + :|
| + + : : :|
1000.13 |-+ : : : + + : ::|
1000.12 |-+ : : : : : : ::|
| :: : : : : : ::|
1000.11 |-+ :: : : :: :: : ::|
1000.1 |-+ :: : : : :: :: :: : |
1000.09 |-+ : :: : : : : : : : : : : |
| : :: : : : : : : : : :: |
1000.08 |-+ : : : : : : : : : + : :: |
1000.07 |-+ : : : .+ + : : + : : + : : :+ +.+ :+ |
| : + : + :+ +: :.+ +.+:: :+.++:: :+.+ +.+ + : :: |
1000.06 |:+ ++ +.+ : + + + :+ : + : + ++ ++ : | 1000.05 +-----------------------------------------------------------------+
860 +---------------------------------------------------------------------+
| : : : : : : : : : : : : :: |
840 |-: :: :: : : : :: : :: : : : : : |
| :: : :: : : :: :: :: :: :: : : : : |
820 |-+++ : + : : : :: :: : : :: : : :: ::: : + |
| : : : : : :: :: : : : : : : :: ::: : :: |
800 |-+ :: : : : : : : : : : : : : : : : :: : :: |
| : :: : : : : : : : : : : : : : : : : :|
780 |-+ + : ++.++ : +++.++.+++ + ++.++ + + + : + : + +: +|
| : : : : : : :: :: :: |
760 |-+ : : :: : :: :: : |
| : + + + + + + |
740 |-+ : |
| : |
720 +---------------------------------------------------------------------+ 25 +--------------------------------------------------------------------+
| : |
24.5 |-+ : |
24 |-+ : + + + + + + |
| :: : : :: : : : |
23.5 |-+ :: : :: :: :: :: : |
| :: :: : : : : :: :: :: |
23 |-+ + : : ++.++ :: +.+++.+++.+ + +.+++ + + + : + : + + : +|
| : : : : : :: : : : : : : : : : : : : :: |
22.5 |-+ : : : : : : : :: : : : : : : :: ::: : :: |
22 |-+++ : + : : : : : :: : : : : : : :: ::: : + |
| :: : :: : : :: : :: :: : : :: : : |
21.5 |-: :: : : : : : :: :: : : :: : : |
| : : : : : : : : : : : : :: |
21 +--------------------------------------------------------------------+ 1000.2 +-----------------------------------------------------------------+
| + :|
1000.18 |-+ + : :|
| : : ::|
1000.16 |-+ + + : + + : ::|
1000.14 |-+ : : : : : : ::|
| :: : : : : :: : |
1000.12 |-+ :: : : : :: :: : : : |
| : :: : : : :: :: : :: |
1000.1 |-+ : : : + : : + : : + : : ++ .+ :: |
| : + : .+ :+: : : : :+. : : :+. : + + :+ |
1000.08 |-+ + +.+ + ++ + +.+ ++.+ : :: ++ : :: + ++.+ ++ :: |
1000.06 |:.++ :: :: : : : : + + |
|+ + + + + + + |
1000.04 +-----------------------------------------------------------------+ 860 +---------------------------------------------------------------------+
| : : : : : : : : : : : : :: |
840 |-: :: :: : : : :: : :: : : : : : |
| :: : :: : : :: :: :: :: :: : : : : |
820 |-+++ : + : : : :: :: : : :: : : :: ::: : + |
| : : : : : :: :: : : : : : : :: ::: : :: |
800 |-+ :: : : : : : : : : : : : : : : : :: : :: |
| : :: : : : : : : : : : : : : : : : : :|
780 |-+ + : ++.++ : +++.++.+++ + ++.++ + + + : + : + +: +|
| : : : : : : :: :: :: |
760 |-+ : : :: : :: :: : |
| : + + + + + + |
740 |-+ : |
| : |
720 +---------------------------------------------------------------------+ 999.75 +------------------------------------------------------------------+
| + |
999.7 |-+ : |
| :: + |
| + +: : + + :: + + + + + |
999.65 |-+ : :: : : +: : ++. :: : : + : : + + :.+: |
| .+ :: + : ::: :: + + : + :: : + : .+ + + : +|
999.6 |++ :: :: :: + : + : :: ++ : : :+ : ++ + :::|
| + :+ + : : : : :: : : :: + + :: :::|
999.55 |-+ + : : : + :: + + + + |
| : : : : |
| :: + + |
999.5 |-+ ++ |
| |
999.45 +------------------------------------------------------------------+ 900 +---------------------------------------------------------------------+
|: : : |
880 |:+ : : |
|: : : |
|: : : |
860 |:+ : : |
|: :: |
840 |-+ :: |
| :: |
820 |-+ :: |
| : |
| : |
800 |-+ + |
| |
780 +---------------------------------------------------------------------+ 10.1 +-------------------------------------------------------------------+
10.08 |-+ |
| |
10.06 |-+ |
10.04 |-+ |
10.02 |-+ |
10 |+.+++.+++.+++.+++.+++.+++.+++.+++.+++.+++.+++.+++.+++.+++.+++.+++.+|
| |
9.98 |-+ |
9.96 |-+ |
9.94 |-+ |
9.92 |-+ |
| |
9.9 |-+ | 9.88 +-------------------------------------------------------------------+
999.75 +------------------------------------------------------------------+
| : |
999.7 |-+ : + |
| +: : : + |
| + :: : + + : :+. + + + + + : ++ |
999.65 |-+ : : + : : + :: + + : : + :: : + : .+ :+ : +|
| .+ :: : ::+ + + : : :: : : : : .+ + + + :::|
999.6 |++ + :: + : + : : :++ : : :+ + : + : :::|
| :+ : : : + :: + :: + + |
999.55 |-+ + : : : : + |
| : : + + |
| :: |
999.5 |-+ ++ |
| |
999.45 +------------------------------------------------------------------+ 900 +---------------------------------------------------------------------+
|: : : |
880 |:+ : : |
|: : : |
|: : : |
860 |:+ : : |
|: :: |
840 |-+ :: |
| :: |
820 |-+ :: |
| : |
| : |
800 |-+ + |
| |
780 +---------------------------------------------------------------------+ perf-sched.sch_delay.avg.ms.devkmsg_read.vfs_read.ksys_read.do_syscall_64
0.06 +--------------------------------------------------------------------+
| : : : : |
0.05 |-+ : : : : |
| + :: :: :: : |
| : :: :: :: :: |
0.04 |-+ : :: :: :: :: +|
| :: :: : : : : :: :|
0.03 |-+ ::: : + : : : : + : : ::|
| + : : : : : : : : : : : ::|
0.02 |-+: : : : : : : : : : : : : |
| :: : : : : : : : : : : : : : |
| : :: : : .+ +.+ : +.+ : ++ : .+ : : :: |
0.01 |++ :+ + ++. ++ + +. + .+ : + +. : + .+++ ++.++.+ + :+ |
| + + + + + + + + + |
0 +--------------------------------------------------------------------+ perf-sched.wait_and_delay.count.devkmsg_read.vfs_read.ksys_read.do_syscall_64
8 +-----------------------------------------------------------------------+
| : : : : |
| : : : : |
7 |-:: : :: + + + :: +.+ +|
| :: : :: : : : :: : : :|
| :: : :: : : : :: : : :|
6 |-: : : ::: : :: :: : :: : ::|
|: : : :: : :: : : : : : :: :: |
5 |++ : : :: : :: : : : : : : :: |
| : : : : : : : : : : : : :: |
| : : : : : : : : : : : : : |
4 |-+ +.+ + : : + ++.++ +.+++.++ ++.++.+++.++.++ + + |
| : : : : |
| : : : : |
3 +-----------------------------------------------------------------------+ perf-sched.wait_and_delay.count.do_syslog.part.0.kmsg_read.vfs_read
8 +-----------------------------------------------------------------------+
| : : : : |
| : : : : |
7 |-:: : :: + + + :: +.+ +|
| :: : :: : : : :: : : :|
| :: : :: : : : :: : : :|
6 |-: : : ::: : :: :: : :: : ::|
|: : : :: : :: : : : : : :: :: |
5 |++ : : :: : :: : : : : : : :: |
| : : : : : : : : : : : : :: |
| : : : : : : : : : : : : : |
4 |-+ +.+ + : : + ++.++ +.+++.++ ++.++.+++.++.++ + + |
| : : : : |
| : : : : |
3 +-----------------------------------------------------------------------+ 0.09 +--------------------------------------------------------------------+
| + + + |
0.08 |-++ : : : |
0.07 |-+: : : : |
| : :: :: :: |
0.06 |-:: :: :: :: |
0.05 |-:: :: :: :: |
| :: : : : : : : + |
0.04 |-: : : : : : : : : |
0.03 |-: : : : : : : : + :: |
|: : : : + : : +: : : : : |
0.02 |:+ : : : + :+: : :: : : :: : |
0.01 |++ : .++ ++. + +. ::.++ : + +++.: + ++.+++.+ .+ .+ : :+. |
| ++ + +.++ + + +.++ + ++ + + + +|
0 +--------------------------------------------------------------------+ perf-sched.wait_time.max.ms.schedule_timeout.khugepaged.kthread.ret_from_fork
0.09 +--------------------------------------------------------------------+
| + |
0.08 |-+ : |
0.07 |-+ : |
| : |
0.06 |-+ : |
0.05 |-+ :: |
| : : |
0.04 |-+ : : |
0.03 |-+ : : |
|+ + : + + |
0.02 |:+ + : : :: + + :: |
0.01 |-++ +.+ +.+ + + + : +. : : ++. :+ .++ + : : : +.+ +. +.+|
| + + +.+ + + ++ + + +++ + ++.++ +++.+ + ++ |
0 +--------------------------------------------------------------------+ perf-sched.wait_time.avg.ms.schedule_timeout.khugepaged.kthread.ret_from_fork
0.09 +--------------------------------------------------------------------+
| + |
0.08 |-+ : |
0.07 |-+ : |
| : |
0.06 |-+ : |
0.05 |-+ :: |
| : : |
0.04 |-+ : : |
0.03 |-+ : : |
|+ + : + + |
0.02 |:+ + : : :: + + :: |
0.01 |-++ +.+ +.+ + + + : +. : : ++. :+ .++ + : : : +.+ +. +.+|
| + + +.+ + + ++ + + +++ + ++.++ +++.+ + ++ |
0 +--------------------------------------------------------------------+ [*] bisect-good sample
[O] bisect-bad sample
***************************************************************************************************
lkp-skl-fpga01: 104 threads Skylake with 192G memory
=========================================================================================
compiler/cpufreq_governor/kconfig/mode/nr_task/rootfs/tbox_group/test/testcase/ucode:
gcc-9/performance/x86_64-rhel-8.3/process/16/debian-10.4-x86_64-20200603.cgz/lkp-skl-fpga01/malloc1/will-it-scale/0x2006a0a
commit:
dde38d5b80 ("kernel: don't include PF_IO_WORKERs as part of same_thread_group()") 43b2a76b1a ("proc: don't show PF_IO_WORKER threads as threads in /proc/<pid>/task/")
dde38d5b805358c7 43b2a76b1a5abcc9833b463bef1
---------------- ---------------------------
%stddev %change %stddev
\ | \
1105101 -11.1% 982154 ± 2% will-it-scale.16.processes
83.20 -1.4% 82.01 will-it-scale.16.processes_idle
69068 -11.1% 61384 ± 2% will-it-scale.per_process_ops
1105101 -11.1% 982154 ± 2% will-it-scale.workload
1.36 ± 15% -0.7 0.67 ± 4% mpstat.cpu.all.irq%
6.653e+08 -11.0% 5.921e+08 ± 2% numa-numastat.node0.local_node
6.653e+08 -11.0% 5.921e+08 ± 2% numa-numastat.node0.numa_hit
626019 ± 3% +223.0% 2022084 ± 22% numa-numastat.node1.local_node
677357 ± 4% +202.5% 2049009 ± 21% numa-numastat.node1.numa_hit
83.00 -1.2% 82.00 vmstat.cpu.id
1358743 +147.9% 3368229 ± 2% vmstat.memory.cache
15.17 ± 2% +12.1% 17.00 vmstat.procs.r
1602 +10.6% 1772 vmstat.system.cs
599024 ± 2% +70.6% 1022074 ± 38% numa-meminfo.node0.FilePages
1251096 ± 5% +48.3% 1855764 ± 20% numa-meminfo.node0.MemUsed
55683 ± 4% +2832.5% 1632907 ± 28% numa-meminfo.node1.Active
55560 ± 4% +2830.2% 1628037 ± 29% numa-meminfo.node1.Active(anon)
661513 ± 2% +238.7% 2240271 ± 20% numa-meminfo.node1.FilePages
1318940 ± 5% +137.5% 3132286 ± 13% numa-meminfo.node1.MemUsed
64592 ± 4% +2434.8% 1637306 ± 28% numa-meminfo.node1.Shmem
86167 ± 72% -40.6% 51159 ± 2% sched_debug.cfs_rq:/.load.max
27024 ± 20% -18.4% 22050 sched_debug.cfs_rq:/.load.stddev
490216 ± 4% +18.4% 580187 ± 5% sched_debug.cfs_rq:/.min_vruntime.avg
225.01 ± 12% -23.1% 172.92 ± 10% sched_debug.cfs_rq:/.util_est_enqueued.avg
348.14 ± 5% -23.6% 266.02 ± 7% sched_debug.cfs_rq:/.util_est_enqueued.stddev
1334 ± 25% -48.8% 682.65 ± 3% sched_debug.cpu.clock_task.stddev
0.16 ± 2% +11.0% 0.18 sched_debug.cpu.nr_running.avg
57321 ± 3% +3485.8% 2055409 ± 3% meminfo.Active
56950 ± 3% +3497.8% 2048955 ± 4% meminfo.Active(anon)
1260522 +158.6% 3260277 ± 2% meminfo.Cached
2367306 +88.5% 4462262 ± 2% meminfo.Committed_AS
2569992 +94.0% 4985892 meminfo.Memused
4496 +11.4% 5009 meminfo.PageTables
76116 ± 2% +2615.4% 2066827 ± 4% meminfo.Shmem
3023094 +64.9% 4985892 meminfo.max_used_kB
149755 ± 2% +70.6% 255457 ± 38% numa-vmstat.node0.nr_file_pages
3.314e+08 -10.4% 2.97e+08 ± 2% numa-vmstat.node0.numa_hit
3.313e+08 -10.4% 2.97e+08 ± 2% numa-vmstat.node0.numa_local
13864 ± 4% +2833.3% 406677 ± 29% numa-vmstat.node1.nr_active_anon
165371 ± 2% +238.5% 559735 ± 20% numa-vmstat.node1.nr_file_pages
16141 ± 4% +2433.9% 408994 ± 28% numa-vmstat.node1.nr_shmem
13864 ± 4% +2833.3% 406677 ± 29% numa-vmstat.node1.nr_zone_active_anon
1129707 ± 2% +59.3% 1799567 ± 9% numa-vmstat.node1.numa_hit
947670 ± 11% +68.0% 1591804 ± 11% numa-vmstat.node1.numa_local
31447 ± 2% +41.8% 44578 ± 3% slabinfo.filp.active_objs
993.50 ± 2% +40.7% 1398 ± 3% slabinfo.filp.active_slabs
31810 ± 2% +40.7% 44750 ± 3% slabinfo.filp.num_objs
993.50 ± 2% +40.7% 1398 ± 3% slabinfo.filp.num_slabs
8576 +52.7% 13093 ± 13% slabinfo.kmalloc-256.active_objs
268.17 +53.3% 411.17 ± 13% slabinfo.kmalloc-256.active_slabs
8605 +53.1% 13173 ± 13% slabinfo.kmalloc-256.num_objs
268.17 +53.3% 411.17 ± 13% slabinfo.kmalloc-256.num_slabs
13727 -35.8% 8815 slabinfo.proc_inode_cache.active_objs
13729 -35.8% 8815 slabinfo.proc_inode_cache.num_objs
22048 +36.6% 30115 slabinfo.radix_tree_node.active_objs
393.00 +36.8% 537.67 slabinfo.radix_tree_node.active_slabs
22048 +36.6% 30118 slabinfo.radix_tree_node.num_objs
393.00 +36.8% 537.67 slabinfo.radix_tree_node.num_slabs
14238 ± 3% +3503.0% 513000 ± 4% proc-vmstat.nr_active_anon
61574 +8.1% 66554 proc-vmstat.nr_anon_pages
4824998 -1.2% 4764837 proc-vmstat.nr_dirty_background_threshold
9661794 -1.2% 9541325 proc-vmstat.nr_dirty_threshold
315136 +158.9% 815833 ± 2% proc-vmstat.nr_file_pages
48538649 -1.2% 47933894 proc-vmstat.nr_free_pages
66233 +7.0% 70862 proc-vmstat.nr_inactive_anon
1122 +11.3% 1249 proc-vmstat.nr_page_table_pages
19033 ± 2% +2618.8% 517468 ± 4% proc-vmstat.nr_shmem
47871 +3.3% 49439 proc-vmstat.nr_slab_unreclaimable
14238 ± 3% +3503.0% 513000 ± 4% proc-vmstat.nr_zone_active_anon
66233 +7.0% 70862 proc-vmstat.nr_zone_inactive_anon
2897 ± 91% +114.1% 6202 ± 39% proc-vmstat.numa_hint_faults
6.656e+08 -10.8% 5.94e+08 proc-vmstat.numa_hit
6.655e+08 -10.8% 5.939e+08 proc-vmstat.numa_local
20412 ± 4% +236.7% 68727 ± 3% proc-vmstat.pgactivate
6.661e+08 -10.6% 5.952e+08 proc-vmstat.pgalloc_normal
3.335e+08 -11.0% 2.967e+08 ± 2% proc-vmstat.pgfault
6.661e+08 -10.8% 5.94e+08 proc-vmstat.pgfree
8040 ± 9% +16.9% 9395 ± 9% softirqs.CPU1.RCU
7061 ± 7% +30.9% 9240 ± 12% softirqs.CPU16.RCU
7511 ± 10% +43.0% 10743 ± 19% softirqs.CPU17.RCU
7696 ± 10% +17.0% 9002 ± 11% softirqs.CPU2.RCU
7261 ± 13% +32.3% 9605 ± 15% softirqs.CPU20.RCU
6764 ± 13% +28.7% 8706 ± 8% softirqs.CPU21.RCU
5969 ± 8% +95.5% 11670 ± 30% softirqs.CPU27.RCU
5875 ± 4% +52.7% 8971 ± 31% softirqs.CPU29.RCU
6642 ± 7% +29.9% 8626 ± 13% softirqs.CPU45.RCU
6641 ± 7% +35.4% 8991 ± 22% softirqs.CPU47.RCU
6629 ± 7% +30.5% 8648 ± 15% softirqs.CPU48.RCU
6314 ± 8% +53.3% 9679 ± 31% softirqs.CPU49.RCU
17807 ± 7% +120.2% 39216 ± 4% softirqs.CPU51.SCHED
6839 ± 7% +38.3% 9462 ± 5% softirqs.CPU58.RCU
6665 ± 7% +33.8% 8921 ± 7% softirqs.CPU59.RCU
7542 ± 9% +17.3% 8847 ± 8% softirqs.CPU66.RCU
4833 ± 17% +78.2% 8612 ± 28% softirqs.CPU77.RCU
7677 ± 13% +24.4% 9552 ± 12% softirqs.CPU78.RCU
41305 -12.3% 36206 ± 8% softirqs.CPU78.SCHED
7588 ± 10% +21.5% 9222 ± 19% softirqs.CPU84.RCU
780375 ± 6% +19.5% 932332 ± 4% softirqs.RCU
43219 ± 7% +23.9% 53545 ± 2% softirqs.TIMER
52.18 ± 10% -52.2 0.00 perf-profile.calltrace.cycles-pp.__munmap
51.08 ± 10% -51.1 0.00 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__munmap
50.08 ± 10% -50.1 0.00 perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__munmap
50.05 ± 10% -50.1 0.00 perf-profile.calltrace.cycles-pp.__x64_sys_munmap.do_syscall_64.entry_SYSCALL_64_after_hwframe.__munmap
49.94 ± 10% -49.9 0.00 perf-profile.calltrace.cycles-pp.__vm_munmap.__x64_sys_munmap.do_syscall_64.entry_SYSCALL_64_after_hwframe.__munmap
49.77 ± 10% -49.8 0.00 perf-profile.calltrace.cycles-pp.__do_munmap.__vm_munmap.__x64_sys_munmap.do_syscall_64.entry_SYSCALL_64_after_hwframe
48.99 ± 10% -49.0 0.00 perf-profile.calltrace.cycles-pp.unmap_region.__do_munmap.__vm_munmap.__x64_sys_munmap.do_syscall_64
37.69 ± 10% -37.7 0.00 perf-profile.calltrace.cycles-pp.tlb_finish_mmu.unmap_region.__do_munmap.__vm_munmap.__x64_sys_munmap
37.63 ± 10% -37.6 0.00 perf-profile.calltrace.cycles-pp.tlb_flush_mmu.tlb_finish_mmu.unmap_region.__do_munmap.__vm_munmap
34.89 ± 19% -34.9 0.00 perf-profile.calltrace.cycles-pp.secondary_startup_64_no_verify
33.92 ± 17% -33.9 0.00 perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
33.92 ± 17% -33.9 0.00 perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
33.92 ± 17% -33.9 0.00 perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64_no_verify
33.57 ± 17% -33.6 0.00 perf-profile.calltrace.cycles-pp.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
33.46 ± 17% -33.5 0.00 perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary
33.02 ± 21% -33.0 0.00 perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry
31.26 ± 9% -31.3 0.00 perf-profile.calltrace.cycles-pp.flush_tlb_mm_range.tlb_flush_mmu.tlb_finish_mmu.unmap_region.__do_munmap
30.90 ± 9% -30.9 0.00 perf-profile.calltrace.cycles-pp.flush_tlb_func_common.flush_tlb_mm_range.tlb_flush_mmu.tlb_finish_mmu.unmap_region
30.60 ± 9% -30.6 0.00 perf-profile.calltrace.cycles-pp.native_flush_tlb_one_user.flush_tlb_func_common.flush_tlb_mm_range.tlb_flush_mmu.tlb_finish_mmu
5.94 ± 11% -5.9 0.00 perf-profile.calltrace.cycles-pp.release_pages.tlb_flush_mmu.tlb_finish_mmu.unmap_region.__do_munmap
5.90 ± 10% -5.9 0.00 perf-profile.calltrace.cycles-pp.asm_exc_page_fault
5.77 ± 10% -5.8 0.00 perf-profile.calltrace.cycles-pp.unmap_vmas.unmap_region.__do_munmap.__vm_munmap.__x64_sys_munmap
5.60 ± 10% -5.6 0.00 perf-profile.calltrace.cycles-pp.unmap_page_range.unmap_vmas.unmap_region.__do_munmap.__vm_munmap
5.03 ± 10% -5.0 0.00 perf-profile.calltrace.cycles-pp.exc_page_fault.asm_exc_page_fault
4.99 ± 10% -5.0 0.00 perf-profile.calltrace.cycles-pp.do_user_addr_fault.exc_page_fault.asm_exc_page_fault
54.84 ± 10% -54.8 0.00 perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
52.90 ± 10% -52.9 0.00 perf-profile.children.cycles-pp.do_syscall_64
52.27 ± 10% -52.3 0.00 perf-profile.children.cycles-pp.__munmap
50.05 ± 10% -50.1 0.00 perf-profile.children.cycles-pp.__x64_sys_munmap
49.95 ± 10% -49.9 0.00 perf-profile.children.cycles-pp.__vm_munmap
49.78 ± 10% -49.8 0.00 perf-profile.children.cycles-pp.__do_munmap
49.00 ± 10% -49.0 0.00 perf-profile.children.cycles-pp.unmap_region
37.69 ± 10% -37.7 0.00 perf-profile.children.cycles-pp.tlb_finish_mmu
37.64 ± 10% -37.6 0.00 perf-profile.children.cycles-pp.tlb_flush_mmu
34.89 ± 19% -34.9 0.00 perf-profile.children.cycles-pp.secondary_startup_64_no_verify
34.89 ± 19% -34.9 0.00 perf-profile.children.cycles-pp.cpu_startup_entry
34.89 ± 19% -34.9 0.00 perf-profile.children.cycles-pp.do_idle
34.53 ± 19% -34.5 0.00 perf-profile.children.cycles-pp.cpuidle_enter
34.53 ± 19% -34.5 0.00 perf-profile.children.cycles-pp.cpuidle_enter_state
33.92 ± 17% -33.9 0.00 perf-profile.children.cycles-pp.start_secondary
33.11 ± 21% -33.1 0.00 perf-profile.children.cycles-pp.intel_idle
31.28 ± 9% -31.3 0.00 perf-profile.children.cycles-pp.flush_tlb_mm_range
30.95 ± 9% -30.9 0.00 perf-profile.children.cycles-pp.flush_tlb_func_common
30.65 ± 9% -30.7 0.00 perf-profile.children.cycles-pp.native_flush_tlb_one_user
6.63 ± 12% -6.6 0.00 perf-profile.children.cycles-pp.release_pages
5.96 ± 10% -6.0 0.00 perf-profile.children.cycles-pp.asm_exc_page_fault
5.78 ± 10% -5.8 0.00 perf-profile.children.cycles-pp.unmap_vmas
5.71 ± 10% -5.7 0.00 perf-profile.children.cycles-pp.unmap_page_range
5.05 ± 10% -5.1 0.00 perf-profile.children.cycles-pp.exc_page_fault
5.01 ± 10% -5.0 0.00 perf-profile.children.cycles-pp.do_user_addr_fault
4.68 ± 11% -4.7 0.00 perf-profile.children.cycles-pp.__mmap
33.11 ± 21% -33.1 0.00 perf-profile.self.cycles-pp.intel_idle
30.60 ± 9% -30.6 0.00 perf-profile.self.cycles-pp.native_flush_tlb_one_user
5.794e+09 +10.5% 6.403e+09 perf-stat.i.branch-instructions
0.67 ± 37% +11.0 11.68 ± 6% perf-stat.i.cache-miss-rate%
1466858 ± 45% +1979.5% 30503394 ± 6% perf-stat.i.cache-misses
2.278e+08 ± 8% +14.8% 2.616e+08 ± 3% perf-stat.i.cache-references
1573 +10.8% 1742 perf-stat.i.context-switches
4.718e+10 +11.7% 5.27e+10 perf-stat.i.cpu-cycles
108.07 -1.8% 106.11 perf-stat.i.cpu-migrations
48430 ± 64% -96.3% 1798 ± 9% perf-stat.i.cycles-between-cache-misses
0.08 ± 11% -0.0 0.06 perf-stat.i.dTLB-load-miss-rate%
6.336e+09 +10.0% 6.967e+09 perf-stat.i.dTLB-loads
0.15 -0.0 0.11 perf-stat.i.dTLB-store-miss-rate%
4511491 -12.9% 3930479 perf-stat.i.dTLB-store-misses
2.97e+09 +16.4% 3.457e+09 perf-stat.i.dTLB-stores
3.67 ± 5% +0.6 4.28 ± 8% perf-stat.i.iTLB-load-miss-rate%
97452404 -9.9% 87781036 perf-stat.i.iTLB-loads
2.63e+10 +11.9% 2.944e+10 perf-stat.i.instructions
0.85 ± 13% -86.7% 0.11 ± 60% perf-stat.i.major-faults
0.45 +11.7% 0.51 perf-stat.i.metric.GHz
0.94 ± 20% -69.9% 0.28 ± 5% perf-stat.i.metric.K/sec
148.39 +11.4% 165.31 perf-stat.i.metric.M/sec
1104170 -11.0% 982353 ± 2% perf-stat.i.minor-faults
124679 ± 19% +4411.6% 5625000 ± 4% perf-stat.i.node-load-misses
18816 ± 18% +3435.6% 665298 ± 6% perf-stat.i.node-loads
91.87 +7.0 98.92 perf-stat.i.node-store-miss-rate%
31781 ± 7% +10338.3% 3317480 ± 8% perf-stat.i.node-store-misses
5872 ± 9% +393.6% 28983 ± 5% perf-stat.i.node-stores
1104171 -11.0% 982353 ± 2% perf-stat.i.page-faults
0.63 ± 41% +11.0 11.67 ± 6% perf-stat.overall.cache-miss-rate%
43058 ± 57% -96.0% 1734 ± 6% perf-stat.overall.cycles-between-cache-misses
0.08 ± 11% -0.0 0.06 perf-stat.overall.dTLB-load-miss-rate%
0.15 -0.0 0.11 perf-stat.overall.dTLB-store-miss-rate%
3.66 ± 5% +0.6 4.26 ± 8% perf-stat.overall.iTLB-load-miss-rate%
86.88 +2.5 89.40 perf-stat.overall.node-load-miss-rate%
84.40 +14.7 99.13 perf-stat.overall.node-store-miss-rate%
7160811 +26.0% 9025782 perf-stat.overall.path-length
5.774e+09 +10.5% 6.382e+09 perf-stat.ps.branch-instructions
1461552 ± 45% +1979.8% 30397813 ± 6% perf-stat.ps.cache-misses
2.271e+08 ± 8% +14.8% 2.608e+08 ± 3% perf-stat.ps.cache-references
1568 +10.8% 1737 perf-stat.ps.context-switches
4.702e+10 +11.7% 5.252e+10 perf-stat.ps.cpu-cycles
107.68 -1.8% 105.75 perf-stat.ps.cpu-migrations
6.315e+09 +10.0% 6.943e+09 perf-stat.ps.dTLB-loads
4496294 -12.9% 3917284 ± 2% perf-stat.ps.dTLB-store-misses
2.96e+09 +16.4% 3.446e+09 perf-stat.ps.dTLB-stores
97123947 -9.9% 87486204 ± 2% perf-stat.ps.iTLB-loads
2.621e+10 +11.9% 2.934e+10 perf-stat.ps.instructions
0.84 ± 13% -86.7% 0.11 ± 60% perf-stat.ps.major-faults
1100449 -11.0% 979055 ± 2% perf-stat.ps.minor-faults
124211 ± 19% +4413.3% 5605997 ± 4% perf-stat.ps.node-load-misses
18709 ± 18% +3444.3% 663126 ± 6% perf-stat.ps.node-loads
31654 ± 7% +10343.2% 3305726 ± 8% perf-stat.ps.node-store-misses
5835 ± 9% +394.9% 28881 ± 5% perf-stat.ps.node-stores
1100450 -11.0% 979055 ± 2% perf-stat.ps.page-faults
7.914e+12 +12.0% 8.863e+12 perf-stat.total.instructions
0.01 ± 58% -100.0% 0.00 perf-sched.sch_delay.avg.ms.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown]
0.01 ± 48% -100.0% 0.00 perf-sched.sch_delay.avg.ms.devkmsg_read.vfs_read.ksys_read.do_syscall_64
0.01 ± 41% -100.0% 0.00 perf-sched.sch_delay.avg.ms.do_nanosleep.hrtimer_nanosleep.__x64_sys_nanosleep.do_syscall_64
0.01 ± 29% -100.0% 0.00 perf-sched.sch_delay.avg.ms.do_syslog.part.0.kmsg_read.vfs_read
0.01 ± 31% -100.0% 0.00 perf-sched.sch_delay.avg.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
0.01 ± 74% -100.0% 0.00 perf-sched.sch_delay.avg.ms.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64
0.01 ± 64% -100.0% 0.00 perf-sched.sch_delay.avg.ms.futex_wait_queue_me.futex_wait.do_futex.__x64_sys_futex
0.00 ± 38% -100.0% 0.00 perf-sched.sch_delay.avg.ms.pipe_read.new_sync_read.vfs_read.ksys_read
0.02 ±173% -100.0% 0.00 perf-sched.sch_delay.avg.ms.rcu_gp_kthread.kthread.ret_from_fork
0.01 ± 32% -100.0% 0.00 perf-sched.sch_delay.avg.ms.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait.__x64_sys_epoll_wait
0.01 ± 23% -100.0% 0.00 perf-sched.sch_delay.avg.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_select
0.00 ± 51% -100.0% 0.00 perf-sched.sch_delay.avg.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_sys_poll
0.01 ± 24% -100.0% 0.00 perf-sched.sch_delay.avg.ms.schedule_timeout.kcompactd.kthread.ret_from_fork
0.01 ±111% -100.0% 0.00 perf-sched.sch_delay.avg.ms.schedule_timeout.rcu_gp_kthread.kthread.ret_from_fork
0.01 ± 89% -100.0% 0.00 perf-sched.sch_delay.avg.ms.schedule_timeout.wait_for_completion.__flush_work.lru_add_drain_all
0.01 ± 41% -100.0% 0.00 perf-sched.sch_delay.avg.ms.smpboot_thread_fn.kthread.ret_from_fork
0.00 ± 11% -100.0% 0.00 perf-sched.sch_delay.avg.ms.wait_for_partner.fifo_open.do_dentry_open.do_open.isra
0.02 ± 20% -100.0% 0.00 perf-sched.sch_delay.avg.ms.worker_thread.kthread.ret_from_fork
0.01 ± 79% -100.0% 0.00 perf-sched.sch_delay.max.ms.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown]
0.02 ±127% -100.0% 0.00 perf-sched.sch_delay.max.ms.devkmsg_read.vfs_read.ksys_read.do_syscall_64
0.02 ± 55% -100.0% 0.00 perf-sched.sch_delay.max.ms.do_nanosleep.hrtimer_nanosleep.__x64_sys_nanosleep.do_syscall_64
0.02 ± 89% -100.0% 0.00 perf-sched.sch_delay.max.ms.do_syslog.part.0.kmsg_read.vfs_read
0.06 ± 62% -100.0% 0.00 perf-sched.sch_delay.max.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
0.06 ± 85% -100.0% 0.00 perf-sched.sch_delay.max.ms.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64
0.01 ± 21% -100.0% 0.00 perf-sched.sch_delay.max.ms.exit_to_user_mode_prepare.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe.[unknown]
0.03 ±121% -100.0% 0.00 perf-sched.sch_delay.max.ms.futex_wait_queue_me.futex_wait.do_futex.__x64_sys_futex
0.07 ± 65% -100.0% 0.00 perf-sched.sch_delay.max.ms.pipe_read.new_sync_read.vfs_read.ksys_read
0.04 ± 43% -100.0% 0.00 perf-sched.sch_delay.max.ms.preempt_schedule_common.__cond_resched.stop_one_cpu.affine_move_task.__set_cpus_allowed_ptr
0.02 ±168% -100.0% 0.00 perf-sched.sch_delay.max.ms.rcu_gp_kthread.kthread.ret_from_fork
0.04 ± 80% -100.0% 0.00 perf-sched.sch_delay.max.ms.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait.__x64_sys_epoll_wait
0.01 ± 16% -100.0% 0.00 perf-sched.sch_delay.max.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_select
0.01 ± 51% -100.0% 0.00 perf-sched.sch_delay.max.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_sys_poll
0.02 ± 24% -100.0% 0.00 perf-sched.sch_delay.max.ms.schedule_timeout.kcompactd.kthread.ret_from_fork
6.20 ±108% -100.0% 0.00 perf-sched.sch_delay.max.ms.schedule_timeout.rcu_gp_kthread.kthread.ret_from_fork
0.01 ± 85% -100.0% 0.00 perf-sched.sch_delay.max.ms.schedule_timeout.wait_for_completion.__flush_work.lru_add_drain_all
1.42 ±217% -100.0% 0.00 perf-sched.sch_delay.max.ms.smpboot_thread_fn.kthread.ret_from_fork
0.01 ± 51% -100.0% 0.00 perf-sched.sch_delay.max.ms.wait_for_partner.fifo_open.do_dentry_open.do_open.isra
3.20 -100.0% 0.00 perf-sched.sch_delay.max.ms.worker_thread.kthread.ret_from_fork
0.01 ± 56% -100.0% 0.00 perf-sched.total_sch_delay.average.ms
8.10 ± 68% -100.0% 0.00 perf-sched.total_sch_delay.max.ms
205.13 ± 2% -100.0% 0.00 perf-sched.total_wait_and_delay.average.ms
8507 ± 2% -100.0% 0.00 perf-sched.total_wait_and_delay.count.ms
8963 ± 5% -100.0% 0.00 perf-sched.total_wait_and_delay.max.ms
205.12 ± 2% -100.0% 0.00 perf-sched.total_wait_time.average.ms
8963 ± 5% -100.0% 0.00 perf-sched.total_wait_time.max.ms
899.67 -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown]
1963 ± 41% -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.devkmsg_read.vfs_read.ksys_read.do_syscall_64
833.08 ± 4% -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.do_nanosleep.hrtimer_nanosleep.__x64_sys_nanosleep.do_syscall_64
1963 ± 41% -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.do_syslog.part.0.kmsg_read.vfs_read
272.32 ± 4% -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
0.91 ± 4% -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64
69.10 ± 4% -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.pipe_read.new_sync_read.vfs_read.ksys_read
0.02 ± 5% -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.preempt_schedule_common.__cond_resched.remove_vma.__do_munmap.__vm_munmap
8.95 ± 8% -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.preempt_schedule_common.__cond_resched.stop_one_cpu.affine_move_task.__set_cpus_allowed_ptr
2.66 ± 9% -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.rcu_gp_kthread.kthread.ret_from_fork
580.43 ± 25% -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait.__x64_sys_epoll_wait
519.81 ± 12% -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_sys_poll
478.81 -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.schedule_timeout.kcompactd.kthread.ret_from_fork
6.18 ± 11% -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.schedule_timeout.rcu_gp_kthread.kthread.ret_from_fork
697.60 -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.smpboot_thread_fn.kthread.ret_from_fork
0.00 ± 11% -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.wait_for_partner.fifo_open.do_dentry_open.do_open.isra
543.97 ± 4% -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.worker_thread.kthread.ret_from_fork
10.00 -100.0% 0.00 perf-sched.wait_and_delay.count.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown]
5.17 ± 36% -100.0% 0.00 perf-sched.wait_and_delay.count.devkmsg_read.vfs_read.ksys_read.do_syscall_64
21.67 ± 4% -100.0% 0.00 perf-sched.wait_and_delay.count.do_nanosleep.hrtimer_nanosleep.__x64_sys_nanosleep.do_syscall_64
5.17 ± 36% -100.0% 0.00 perf-sched.wait_and_delay.count.do_syslog.part.0.kmsg_read.vfs_read
247.00 -100.0% 0.00 perf-sched.wait_and_delay.count.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
238.33 ± 4% -100.0% 0.00 perf-sched.wait_and_delay.count.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64
2342 ± 4% -100.0% 0.00 perf-sched.wait_and_delay.count.pipe_read.new_sync_read.vfs_read.ksys_read
265.00 ± 5% -100.0% 0.00 perf-sched.wait_and_delay.count.preempt_schedule_common.__cond_resched.remove_vma.__do_munmap.__vm_munmap
1200 ± 4% -100.0% 0.00 perf-sched.wait_and_delay.count.preempt_schedule_common.__cond_resched.stop_one_cpu.affine_move_task.__set_cpus_allowed_ptr
63.67 ± 13% -100.0% 0.00 perf-sched.wait_and_delay.count.rcu_gp_kthread.kthread.ret_from_fork
29.00 ± 33% -100.0% 0.00 perf-sched.wait_and_delay.count.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait.__x64_sys_epoll_wait
64.83 ± 5% -100.0% 0.00 perf-sched.wait_and_delay.count.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_sys_poll
40.00 -100.0% 0.00 perf-sched.wait_and_delay.count.schedule_timeout.kcompactd.kthread.ret_from_fork
1611 ± 11% -100.0% 0.00 perf-sched.wait_and_delay.count.schedule_timeout.rcu_gp_kthread.kthread.ret_from_fork
1408 ± 3% -100.0% 0.00 perf-sched.wait_and_delay.count.smpboot_thread_fn.kthread.ret_from_fork
72.00 -100.0% 0.00 perf-sched.wait_and_delay.count.wait_for_partner.fifo_open.do_dentry_open.do_open.isra
726.33 ± 3% -100.0% 0.00 perf-sched.wait_and_delay.count.worker_thread.kthread.ret_from_fork
999.73 -100.0% 0.00 perf-sched.wait_and_delay.max.ms.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown]
5749 ± 44% -100.0% 0.00 perf-sched.wait_and_delay.max.ms.devkmsg_read.vfs_read.ksys_read.do_syscall_64
1000 -100.0% 0.00 perf-sched.wait_and_delay.max.ms.do_nanosleep.hrtimer_nanosleep.__x64_sys_nanosleep.do_syscall_64
5749 ± 44% -100.0% 0.00 perf-sched.wait_and_delay.max.ms.do_syslog.part.0.kmsg_read.vfs_read
4683 ± 54% -100.0% 0.00 perf-sched.wait_and_delay.max.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
16.38 ± 2% -100.0% 0.00 perf-sched.wait_and_delay.max.ms.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64
5751 ± 44% -100.0% 0.00 perf-sched.wait_and_delay.max.ms.pipe_read.new_sync_read.vfs_read.ksys_read
0.11 ± 49% -100.0% 0.00 perf-sched.wait_and_delay.max.ms.preempt_schedule_common.__cond_resched.remove_vma.__do_munmap.__vm_munmap
1807 ± 44% -100.0% 0.00 perf-sched.wait_and_delay.max.ms.preempt_schedule_common.__cond_resched.stop_one_cpu.affine_move_task.__set_cpus_allowed_ptr
4.85 ± 7% -100.0% 0.00 perf-sched.wait_and_delay.max.ms.rcu_gp_kthread.kthread.ret_from_fork
5338 ± 48% -100.0% 0.00 perf-sched.wait_and_delay.max.ms.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait.__x64_sys_epoll_wait
6078 ± 15% -100.0% 0.00 perf-sched.wait_and_delay.max.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_sys_poll
505.00 -100.0% 0.00 perf-sched.wait_and_delay.max.ms.schedule_timeout.kcompactd.kthread.ret_from_fork
265.84 ± 15% -100.0% 0.00 perf-sched.wait_and_delay.max.ms.schedule_timeout.rcu_gp_kthread.kthread.ret_from_fork
1193 ± 31% -100.0% 0.00 perf-sched.wait_and_delay.max.ms.smpboot_thread_fn.kthread.ret_from_fork
0.01 ± 51% -100.0% 0.00 perf-sched.wait_and_delay.max.ms.wait_for_partner.fifo_open.do_dentry_open.do_open.isra
8783 ± 7% -100.0% 0.00 perf-sched.wait_and_delay.max.ms.worker_thread.kthread.ret_from_fork
899.66 -100.0% 0.00 perf-sched.wait_time.avg.ms.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown]
1963 ± 41% -100.0% 0.00 perf-sched.wait_time.avg.ms.devkmsg_read.vfs_read.ksys_read.do_syscall_64
833.07 ± 4% -100.0% 0.00 perf-sched.wait_time.avg.ms.do_nanosleep.hrtimer_nanosleep.__x64_sys_nanosleep.do_syscall_64
1963 ± 41% -100.0% 0.00 perf-sched.wait_time.avg.ms.do_syslog.part.0.kmsg_read.vfs_read
272.31 ± 4% -100.0% 0.00 perf-sched.wait_time.avg.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
0.90 ± 3% -100.0% 0.00 perf-sched.wait_time.avg.ms.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64
0.02 ± 32% -100.0% 0.00 perf-sched.wait_time.avg.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_exc_page_fault.[unknown]
0.02 ± 19% -100.0% 0.00 perf-sched.wait_time.avg.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt.[unknown]
0.08 ±163% -100.0% 0.00 perf-sched.wait_time.avg.ms.exit_to_user_mode_prepare.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe.[unknown]
69.09 ± 4% -100.0% 0.00 perf-sched.wait_time.avg.ms.pipe_read.new_sync_read.vfs_read.ksys_read
0.02 ± 47% -100.0% 0.00 perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.__alloc_pages_nodemask.pte_alloc_one.__pte_alloc
0.02 ± 17% -100.0% 0.00 perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.__anon_vma_prepare.do_anonymous_page.__handle_mm_fault
72.45 ±223% -100.0% 0.00 perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.do_user_addr_fault.exc_page_fault.asm_exc_page_fault
0.02 ± 72% -100.0% 0.00 perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.down_write_killable.__vm_munmap.__x64_sys_munmap
0.02 ± 60% -100.0% 0.00 perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.down_write_killable.vm_mmap_pgoff.ksys_mmap_pgoff
0.02 ± 55% -100.0% 0.00 perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.kmem_cache_alloc.__anon_vma_prepare.do_anonymous_page
0.01 ± 58% -100.0% 0.00 perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.kmem_cache_alloc.vm_area_alloc.mmap_region
0.02 ± 5% -100.0% 0.00 perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.remove_vma.__do_munmap.__vm_munmap
8.95 ± 8% -100.0% 0.00 perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.stop_one_cpu.affine_move_task.__set_cpus_allowed_ptr
0.02 ± 19% -100.0% 0.00 perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.unmap_page_range.unmap_vmas.unmap_region
0.02 ± 48% -100.0% 0.00 perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.unmap_vmas.unmap_region.__do_munmap
0.01 ± 63% -100.0% 0.00 perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.zap_pte_range.unmap_page_range.unmap_vmas
2.64 ± 9% -100.0% 0.00 perf-sched.wait_time.avg.ms.rcu_gp_kthread.kthread.ret_from_fork
580.43 ± 25% -100.0% 0.00 perf-sched.wait_time.avg.ms.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait.__x64_sys_epoll_wait
519.81 ± 12% -100.0% 0.00 perf-sched.wait_time.avg.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_sys_poll
478.80 -100.0% 0.00 perf-sched.wait_time.avg.ms.schedule_timeout.kcompactd.kthread.ret_from_fork
0.03 ±100% -100.0% 0.00 perf-sched.wait_time.avg.ms.schedule_timeout.khugepaged.kthread.ret_from_fork
6.17 ± 11% -100.0% 0.00 perf-sched.wait_time.avg.ms.schedule_timeout.rcu_gp_kthread.kthread.ret_from_fork
697.59 -100.0% 0.00 perf-sched.wait_time.avg.ms.smpboot_thread_fn.kthread.ret_from_fork
543.95 ± 4% -100.0% 0.00 perf-sched.wait_time.avg.ms.worker_thread.kthread.ret_from_fork
999.72 -100.0% 0.00 perf-sched.wait_time.max.ms.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown]
5749 ± 44% -100.0% 0.00 perf-sched.wait_time.max.ms.devkmsg_read.vfs_read.ksys_read.do_syscall_64
1000 -100.0% 0.00 perf-sched.wait_time.max.ms.do_nanosleep.hrtimer_nanosleep.__x64_sys_nanosleep.do_syscall_64
5749 ± 44% -100.0% 0.00 perf-sched.wait_time.max.ms.do_syslog.part.0.kmsg_read.vfs_read
4683 ± 54% -100.0% 0.00 perf-sched.wait_time.max.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
16.38 ± 2% -100.0% 0.00 perf-sched.wait_time.max.ms.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64
0.04 ± 11% -100.0% 0.00 perf-sched.wait_time.max.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_exc_page_fault.[unknown]
0.07 ± 57% -100.0% 0.00 perf-sched.wait_time.max.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt.[unknown]
1.22 ±208% -100.0% 0.00 perf-sched.wait_time.max.ms.exit_to_user_mode_prepare.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe.[unknown]
5751 ± 44% -100.0% 0.00 perf-sched.wait_time.max.ms.pipe_read.new_sync_read.vfs_read.ksys_read
0.04 ± 41% -100.0% 0.00 perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.__alloc_pages_nodemask.pte_alloc_one.__pte_alloc
0.07 ± 65% -100.0% 0.00 perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.__anon_vma_prepare.do_anonymous_page.__handle_mm_fault
1086 ±223% -100.0% 0.00 perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.do_user_addr_fault.exc_page_fault.asm_exc_page_fault
0.05 ±112% -100.0% 0.00 perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.down_write_killable.__vm_munmap.__x64_sys_munmap
0.06 ± 83% -100.0% 0.00 perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.down_write_killable.vm_mmap_pgoff.ksys_mmap_pgoff
0.03 ± 37% -100.0% 0.00 perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.kmem_cache_alloc.__anon_vma_prepare.do_anonymous_page
0.03 ± 59% -100.0% 0.00 perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.kmem_cache_alloc.vm_area_alloc.mmap_region
0.11 ± 49% -100.0% 0.00 perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.remove_vma.__do_munmap.__vm_munmap
1807 ± 44% -100.0% 0.00 perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.stop_one_cpu.affine_move_task.__set_cpus_allowed_ptr
0.08 ± 59% -100.0% 0.00 perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.unmap_page_range.unmap_vmas.unmap_region
0.09 ± 70% -100.0% 0.00 perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.unmap_vmas.unmap_region.__do_munmap
0.02 ± 75% -100.0% 0.00 perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.zap_pte_range.unmap_page_range.unmap_vmas
4.83 ± 7% -100.0% 0.00 perf-sched.wait_time.max.ms.rcu_gp_kthread.kthread.ret_from_fork
5338 ± 48% -100.0% 0.00 perf-sched.wait_time.max.ms.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait.__x64_sys_epoll_wait
6078 ± 15% -100.0% 0.00 perf-sched.wait_time.max.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_sys_poll
505.00 -100.0% 0.00 perf-sched.wait_time.max.ms.schedule_timeout.kcompactd.kthread.ret_from_fork
0.03 ±100% -100.0% 0.00 perf-sched.wait_time.max.ms.schedule_timeout.khugepaged.kthread.ret_from_fork
265.82 ± 15% -100.0% 0.00 perf-sched.wait_time.max.ms.schedule_timeout.rcu_gp_kthread.kthread.ret_from_fork
1193 ± 31% -100.0% 0.00 perf-sched.wait_time.max.ms.smpboot_thread_fn.kthread.ret_from_fork
8783 ± 7% -100.0% 0.00 perf-sched.wait_time.max.ms.worker_thread.kthread.ret_from_fork
676.83 ±104% +927.6% 6954 ±136% interrupts.39:PCI-MSI.67633154-edge.eth0-TxRx-1
5953 ± 35% -100.0% 0.00 interrupts.CPU0.NMI:Non-maskable_interrupts
5953 ± 35% -100.0% 0.00 interrupts.CPU0.PMI:Performance_monitoring_interrupts
6263 ± 21% -100.0% 0.00 interrupts.CPU1.NMI:Non-maskable_interrupts
6263 ± 21% -100.0% 0.00 interrupts.CPU1.PMI:Performance_monitoring_interrupts
6259 ± 36% -100.0% 0.00 interrupts.CPU10.NMI:Non-maskable_interrupts
6259 ± 36% -100.0% 0.00 interrupts.CPU10.PMI:Performance_monitoring_interrupts
101.00 ± 33% -100.0% 0.00 interrupts.CPU100.NMI:Non-maskable_interrupts
101.00 ± 33% -100.0% 0.00 interrupts.CPU100.PMI:Performance_monitoring_interrupts
124.33 ± 52% -100.0% 0.00 interrupts.CPU101.NMI:Non-maskable_interrupts
124.33 ± 52% -100.0% 0.00 interrupts.CPU101.PMI:Performance_monitoring_interrupts
122.67 ± 64% -100.0% 0.00 interrupts.CPU102.NMI:Non-maskable_interrupts
122.67 ± 64% -100.0% 0.00 interrupts.CPU102.PMI:Performance_monitoring_interrupts
418.33 ± 32% -100.0% 0.00 interrupts.CPU103.NMI:Non-maskable_interrupts
418.33 ± 32% -100.0% 0.00 interrupts.CPU103.PMI:Performance_monitoring_interrupts
5750 ± 38% -100.0% 0.00 interrupts.CPU11.NMI:Non-maskable_interrupts
5750 ± 38% -100.0% 0.00 interrupts.CPU11.PMI:Performance_monitoring_interrupts
5182 ± 53% -100.0% 0.00 interrupts.CPU12.NMI:Non-maskable_interrupts
5182 ± 53% -100.0% 0.00 interrupts.CPU12.PMI:Performance_monitoring_interrupts
4527 ± 38% -100.0% 0.00 interrupts.CPU13.NMI:Non-maskable_interrupts
4527 ± 38% -100.0% 0.00 interrupts.CPU13.PMI:Performance_monitoring_interrupts
5809 ± 45% -100.0% 0.00 interrupts.CPU14.NMI:Non-maskable_interrupts
5809 ± 45% -100.0% 0.00 interrupts.CPU14.PMI:Performance_monitoring_interrupts
4868 ± 41% -100.0% 0.00 interrupts.CPU15.NMI:Non-maskable_interrupts
4868 ± 41% -100.0% 0.00 interrupts.CPU15.PMI:Performance_monitoring_interrupts
2.00 ±119% +4000.0% 82.00 ±131% interrupts.CPU19.RES:Rescheduling_interrupts
4642 ± 50% -100.0% 0.00 interrupts.CPU2.NMI:Non-maskable_interrupts
4642 ± 50% -100.0% 0.00 interrupts.CPU2.PMI:Performance_monitoring_interrupts
1.00 ±115% +7700.0% 78.00 ±138% interrupts.CPU20.RES:Rescheduling_interrupts
75.83 ± 29% -100.0% 0.00 interrupts.CPU22.NMI:Non-maskable_interrupts
75.83 ± 29% -100.0% 0.00 interrupts.CPU22.PMI:Performance_monitoring_interrupts
77.83 ± 25% -100.0% 0.00 interrupts.CPU24.NMI:Non-maskable_interrupts
77.83 ± 25% -100.0% 0.00 interrupts.CPU24.PMI:Performance_monitoring_interrupts
121.00 ± 41% -100.0% 0.00 interrupts.CPU26.NMI:Non-maskable_interrupts
121.00 ± 41% -100.0% 0.00 interrupts.CPU26.PMI:Performance_monitoring_interrupts
193.50 ± 94% -100.0% 0.00 interrupts.CPU27.NMI:Non-maskable_interrupts
193.50 ± 94% -100.0% 0.00 interrupts.CPU27.PMI:Performance_monitoring_interrupts
99.00 ± 35% -100.0% 0.00 interrupts.CPU28.NMI:Non-maskable_interrupts
99.00 ± 35% -100.0% 0.00 interrupts.CPU28.PMI:Performance_monitoring_interrupts
191.33 ±108% -100.0% 0.00 interrupts.CPU29.NMI:Non-maskable_interrupts
191.33 ±108% -100.0% 0.00 interrupts.CPU29.PMI:Performance_monitoring_interrupts
4235 ± 46% -100.0% 0.00 interrupts.CPU3.NMI:Non-maskable_interrupts
4235 ± 46% -100.0% 0.00 interrupts.CPU3.PMI:Performance_monitoring_interrupts
195.33 ±116% -100.0% 0.00 interrupts.CPU30.NMI:Non-maskable_interrupts
195.33 ±116% -100.0% 0.00 interrupts.CPU30.PMI:Performance_monitoring_interrupts
676.83 ±104% +927.6% 6954 ±136% interrupts.CPU31.39:PCI-MSI.67633154-edge.eth0-TxRx-1
95.50 ± 34% -100.0% 0.00 interrupts.CPU31.NMI:Non-maskable_interrupts
95.50 ± 34% -100.0% 0.00 interrupts.CPU31.PMI:Performance_monitoring_interrupts
215.50 ±111% -100.0% 0.00 interrupts.CPU32.NMI:Non-maskable_interrupts
215.50 ±111% -100.0% 0.00 interrupts.CPU32.PMI:Performance_monitoring_interrupts
548.33 ± 5% +62.7% 892.17 ± 30% interrupts.CPU33.CAL:Function_call_interrupts
106.50 ± 29% -100.0% 0.00 interrupts.CPU33.NMI:Non-maskable_interrupts
106.50 ± 29% -100.0% 0.00 interrupts.CPU33.PMI:Performance_monitoring_interrupts
98.50 ± 36% -100.0% 0.00 interrupts.CPU34.NMI:Non-maskable_interrupts
98.50 ± 36% -100.0% 0.00 interrupts.CPU34.PMI:Performance_monitoring_interrupts
97.50 ± 41% -100.0% 0.00 interrupts.CPU35.NMI:Non-maskable_interrupts
97.50 ± 41% -100.0% 0.00 interrupts.CPU35.PMI:Performance_monitoring_interrupts
179.00 ±102% -100.0% 0.00 interrupts.CPU36.NMI:Non-maskable_interrupts
179.00 ±102% -100.0% 0.00 interrupts.CPU36.PMI:Performance_monitoring_interrupts
108.50 ± 23% -100.0% 0.00 interrupts.CPU37.NMI:Non-maskable_interrupts
108.50 ± 23% -100.0% 0.00 interrupts.CPU37.PMI:Performance_monitoring_interrupts
103.00 ± 28% -100.0% 0.00 interrupts.CPU38.NMI:Non-maskable_interrupts
103.00 ± 28% -100.0% 0.00 interrupts.CPU38.PMI:Performance_monitoring_interrupts
93.50 ± 33% -100.0% 0.00 interrupts.CPU39.NMI:Non-maskable_interrupts
93.50 ± 33% -100.0% 0.00 interrupts.CPU39.PMI:Performance_monitoring_interrupts
3954 ± 46% -100.0% 0.00 interrupts.CPU4.NMI:Non-maskable_interrupts
3954 ± 46% -100.0% 0.00 interrupts.CPU4.PMI:Performance_monitoring_interrupts
93.67 ± 31% -100.0% 0.00 interrupts.CPU40.NMI:Non-maskable_interrupts
93.67 ± 31% -100.0% 0.00 interrupts.CPU40.PMI:Performance_monitoring_interrupts
102.33 ± 43% -100.0% 0.00 interrupts.CPU41.NMI:Non-maskable_interrupts
102.33 ± 43% -100.0% 0.00 interrupts.CPU41.PMI:Performance_monitoring_interrupts
93.00 ± 29% -100.0% 0.00 interrupts.CPU42.NMI:Non-maskable_interrupts
93.00 ± 29% -100.0% 0.00 interrupts.CPU42.PMI:Performance_monitoring_interrupts
99.83 ± 33% -100.0% 0.00 interrupts.CPU43.NMI:Non-maskable_interrupts
99.83 ± 33% -100.0% 0.00 interrupts.CPU43.PMI:Performance_monitoring_interrupts
124.83 ± 38% -100.0% 0.00 interrupts.CPU44.NMI:Non-maskable_interrupts
124.83 ± 38% -100.0% 0.00 interrupts.CPU44.PMI:Performance_monitoring_interrupts
101.83 ± 37% -100.0% 0.00 interrupts.CPU45.NMI:Non-maskable_interrupts
101.83 ± 37% -100.0% 0.00 interrupts.CPU45.PMI:Performance_monitoring_interrupts
102.00 ± 41% -100.0% 0.00 interrupts.CPU46.NMI:Non-maskable_interrupts
102.00 ± 41% -100.0% 0.00 interrupts.CPU46.PMI:Performance_monitoring_interrupts
96.50 ± 34% -100.0% 0.00 interrupts.CPU47.NMI:Non-maskable_interrupts
96.50 ± 34% -100.0% 0.00 interrupts.CPU47.PMI:Performance_monitoring_interrupts
105.33 ± 28% -100.0% 0.00 interrupts.CPU48.NMI:Non-maskable_interrupts
105.33 ± 28% -100.0% 0.00 interrupts.CPU48.PMI:Performance_monitoring_interrupts
139.67 ± 56% -100.0% 0.00 interrupts.CPU49.NMI:Non-maskable_interrupts
139.67 ± 56% -100.0% 0.00 interrupts.CPU49.PMI:Performance_monitoring_interrupts
4293 ± 48% -100.0% 0.00 interrupts.CPU5.NMI:Non-maskable_interrupts
4293 ± 48% -100.0% 0.00 interrupts.CPU5.PMI:Performance_monitoring_interrupts
173.17 ±113% -100.0% 0.00 interrupts.CPU50.NMI:Non-maskable_interrupts
173.17 ±113% -100.0% 0.00 interrupts.CPU50.PMI:Performance_monitoring_interrupts
270.33 ± 28% -100.0% 0.00 interrupts.CPU51.NMI:Non-maskable_interrupts
270.33 ± 28% -100.0% 0.00 interrupts.CPU51.PMI:Performance_monitoring_interrupts
5508 ± 39% -100.0% 0.00 interrupts.CPU52.NMI:Non-maskable_interrupts
5508 ± 39% -100.0% 0.00 interrupts.CPU52.PMI:Performance_monitoring_interrupts
4786 ± 40% -100.0% 0.00 interrupts.CPU53.NMI:Non-maskable_interrupts
4786 ± 40% -100.0% 0.00 interrupts.CPU53.PMI:Performance_monitoring_interrupts
6828 ± 17% -100.0% 0.00 interrupts.CPU54.NMI:Non-maskable_interrupts
6828 ± 17% -100.0% 0.00 interrupts.CPU54.PMI:Performance_monitoring_interrupts
6512 ± 24% -100.0% 0.00 interrupts.CPU55.NMI:Non-maskable_interrupts
6512 ± 24% -100.0% 0.00 interrupts.CPU55.PMI:Performance_monitoring_interrupts
6860 ± 26% -100.0% 1.00 ±223% interrupts.CPU56.NMI:Non-maskable_interrupts
6860 ± 26% -100.0% 1.00 ±223% interrupts.CPU56.PMI:Performance_monitoring_interrupts
6435 ± 27% -100.0% 0.00 interrupts.CPU57.NMI:Non-maskable_interrupts
6435 ± 27% -100.0% 0.00 interrupts.CPU57.PMI:Performance_monitoring_interrupts
5567 ± 36% -100.0% 0.00 interrupts.CPU58.NMI:Non-maskable_interrupts
5567 ± 36% -100.0% 0.00 interrupts.CPU58.PMI:Performance_monitoring_interrupts
5703 ± 39% -100.0% 0.00 interrupts.CPU59.NMI:Non-maskable_interrupts
5703 ± 39% -100.0% 0.00 interrupts.CPU59.PMI:Performance_monitoring_interrupts
5098 ± 43% -100.0% 0.00 interrupts.CPU6.NMI:Non-maskable_interrupts
5098 ± 43% -100.0% 0.00 interrupts.CPU6.PMI:Performance_monitoring_interrupts
6022 ± 27% -100.0% 0.00 interrupts.CPU60.NMI:Non-maskable_interrupts
6022 ± 27% -100.0% 0.00 interrupts.CPU60.PMI:Performance_monitoring_interrupts
5896 ± 37% -100.0% 0.00 interrupts.CPU61.NMI:Non-maskable_interrupts
5896 ± 37% -100.0% 0.00 interrupts.CPU61.PMI:Performance_monitoring_interrupts
5451 ± 36% -100.0% 0.00 interrupts.CPU62.NMI:Non-maskable_interrupts
5451 ± 36% -100.0% 0.00 interrupts.CPU62.PMI:Performance_monitoring_interrupts
5960 ± 27% -100.0% 0.00 interrupts.CPU63.NMI:Non-maskable_interrupts
5960 ± 27% -100.0% 0.00 interrupts.CPU63.PMI:Performance_monitoring_interrupts
6291 ± 28% -100.0% 0.00 interrupts.CPU64.NMI:Non-maskable_interrupts
6291 ± 28% -100.0% 0.00 interrupts.CPU64.PMI:Performance_monitoring_interrupts
5867 ± 38% -100.0% 0.00 interrupts.CPU65.NMI:Non-maskable_interrupts
5867 ± 38% -100.0% 0.00 interrupts.CPU65.PMI:Performance_monitoring_interrupts
5665 ± 33% -100.0% 0.00 interrupts.CPU66.NMI:Non-maskable_interrupts
5665 ± 33% -100.0% 0.00 interrupts.CPU66.PMI:Performance_monitoring_interrupts
5844 ± 27% -100.0% 1.00 ±223% interrupts.CPU67.NMI:Non-maskable_interrupts
5844 ± 27% -100.0% 1.00 ±223% interrupts.CPU67.PMI:Performance_monitoring_interrupts
5506 ± 36% -100.0% 0.00 interrupts.CPU7.NMI:Non-maskable_interrupts
5506 ± 36% -100.0% 0.00 interrupts.CPU7.PMI:Performance_monitoring_interrupts
76.50 ± 28% -100.0% 0.00 interrupts.CPU70.NMI:Non-maskable_interrupts
76.50 ± 28% -100.0% 0.00 interrupts.CPU70.PMI:Performance_monitoring_interrupts
77.50 ± 21% -100.0% 0.00 interrupts.CPU71.NMI:Non-maskable_interrupts
77.50 ± 21% -100.0% 0.00 interrupts.CPU71.PMI:Performance_monitoring_interrupts
101.17 ± 52% -100.0% 0.00 interrupts.CPU74.NMI:Non-maskable_interrupts
101.17 ± 52% -100.0% 0.00 interrupts.CPU74.PMI:Performance_monitoring_interrupts
79.83 ± 26% -100.0% 0.00 interrupts.CPU75.NMI:Non-maskable_interrupts
79.83 ± 26% -100.0% 0.00 interrupts.CPU75.PMI:Performance_monitoring_interrupts
83.33 ± 12% -100.0% 0.00 interrupts.CPU76.NMI:Non-maskable_interrupts
83.33 ± 12% -100.0% 0.00 interrupts.CPU76.PMI:Performance_monitoring_interrupts
126.83 ± 30% -100.0% 0.00 interrupts.CPU78.NMI:Non-maskable_interrupts
126.83 ± 30% -100.0% 0.00 interrupts.CPU78.PMI:Performance_monitoring_interrupts
164.33 ± 77% -100.0% 0.00 interrupts.CPU79.NMI:Non-maskable_interrupts
164.33 ± 77% -100.0% 0.00 interrupts.CPU79.PMI:Performance_monitoring_interrupts
5231 ± 44% -100.0% 0.00 interrupts.CPU8.NMI:Non-maskable_interrupts
5231 ± 44% -100.0% 0.00 interrupts.CPU8.PMI:Performance_monitoring_interrupts
98.50 ± 47% -100.0% 0.00 interrupts.CPU80.NMI:Non-maskable_interrupts
98.50 ± 47% -100.0% 0.00 interrupts.CPU80.PMI:Performance_monitoring_interrupts
184.83 ± 91% -100.0% 0.00 interrupts.CPU81.NMI:Non-maskable_interrupts
184.83 ± 91% -100.0% 0.00 interrupts.CPU81.PMI:Performance_monitoring_interrupts
144.50 ± 78% -100.0% 0.00 interrupts.CPU82.NMI:Non-maskable_interrupts
144.50 ± 78% -100.0% 0.00 interrupts.CPU82.PMI:Performance_monitoring_interrupts
102.83 ± 40% -100.0% 0.00 interrupts.CPU83.NMI:Non-maskable_interrupts
102.83 ± 40% -100.0% 0.00 interrupts.CPU83.PMI:Performance_monitoring_interrupts
138.50 ± 75% -100.0% 0.00 interrupts.CPU84.NMI:Non-maskable_interrupts
138.50 ± 75% -100.0% 0.00 interrupts.CPU84.PMI:Performance_monitoring_interrupts
99.17 ± 49% -100.0% 0.00 interrupts.CPU85.NMI:Non-maskable_interrupts
99.17 ± 49% -100.0% 0.00 interrupts.CPU85.PMI:Performance_monitoring_interrupts
92.33 ± 46% -100.0% 0.00 interrupts.CPU86.NMI:Non-maskable_interrupts
92.33 ± 46% -100.0% 0.00 interrupts.CPU86.PMI:Performance_monitoring_interrupts
98.67 ± 53% -100.0% 0.00 interrupts.CPU87.NMI:Non-maskable_interrupts
98.67 ± 53% -100.0% 0.00 interrupts.CPU87.PMI:Performance_monitoring_interrupts
143.50 ± 74% -100.0% 0.00 interrupts.CPU88.NMI:Non-maskable_interrupts
143.50 ± 74% -100.0% 0.00 interrupts.CPU88.PMI:Performance_monitoring_interrupts
97.50 ± 34% -100.0% 0.00 interrupts.CPU89.NMI:Non-maskable_interrupts
97.50 ± 34% -100.0% 0.00 interrupts.CPU89.PMI:Performance_monitoring_interrupts
5814 ± 35% -100.0% 0.00 interrupts.CPU9.NMI:Non-maskable_interrupts
5814 ± 35% -100.0% 0.00 interrupts.CPU9.PMI:Performance_monitoring_interrupts
94.67 ± 37% -100.0% 0.00 interrupts.CPU90.NMI:Non-maskable_interrupts
94.67 ± 37% -100.0% 0.00 interrupts.CPU90.PMI:Performance_monitoring_interrupts
98.33 ± 38% -100.0% 0.00 interrupts.CPU91.NMI:Non-maskable_interrupts
98.33 ± 38% -100.0% 0.00 interrupts.CPU91.PMI:Performance_monitoring_interrupts
98.33 ± 35% -100.0% 0.00 interrupts.CPU92.NMI:Non-maskable_interrupts
98.33 ± 35% -100.0% 0.00 interrupts.CPU92.PMI:Performance_monitoring_interrupts
98.67 ± 37% -100.0% 0.00 interrupts.CPU93.NMI:Non-maskable_interrupts
98.67 ± 37% -100.0% 0.00 interrupts.CPU93.PMI:Performance_monitoring_interrupts
100.33 ± 25% -100.0% 0.00 interrupts.CPU94.NMI:Non-maskable_interrupts
100.33 ± 25% -100.0% 0.00 interrupts.CPU94.PMI:Performance_monitoring_interrupts
122.17 ± 20% -100.0% 0.00 interrupts.CPU95.NMI:Non-maskable_interrupts
122.17 ± 20% -100.0% 0.00 interrupts.CPU95.PMI:Performance_monitoring_interrupts
128.00 ± 44% -100.0% 0.00 interrupts.CPU96.NMI:Non-maskable_interrupts
128.00 ± 44% -100.0% 0.00 interrupts.CPU96.PMI:Performance_monitoring_interrupts
120.33 ± 40% -99.2% 1.00 ±223% interrupts.CPU97.NMI:Non-maskable_interrupts
120.33 ± 40% -99.2% 1.00 ±223% interrupts.CPU97.PMI:Performance_monitoring_interrupts
100.50 ± 38% -100.0% 0.00 interrupts.CPU98.NMI:Non-maskable_interrupts
100.50 ± 38% -100.0% 0.00 interrupts.CPU98.PMI:Performance_monitoring_interrupts
97.33 ± 36% -100.0% 0.00 interrupts.CPU99.NMI:Non-maskable_interrupts
97.33 ± 36% -100.0% 0.00 interrupts.CPU99.PMI:Performance_monitoring_interrupts
186821 ± 14% -100.0% 4.00 ± 70% interrupts.NMI:Non-maskable_interrupts
186821 ± 14% -100.0% 4.00 ± 70% interrupts.PMI:Performance_monitoring_interrupts***************************************************************************************************
lkp-skl-fpga01: 104 threads Skylake with 192G memory
=========================================================================================
compiler/cpufreq_governor/kconfig/mode/nr_task/rootfs/tbox_group/test/testcase/ucode:
gcc-9/performance/x86_64-rhel-8.3/process/100%/debian-10.4-x86_64-20200603.cgz/lkp-skl-fpga01/eventfd1/will-it-scale/0x2006a0a
commit:
dde38d5b80 ("kernel: don't include PF_IO_WORKERs as part of same_thread_group()") 43b2a76b1a ("proc: don't show PF_IO_WORKER threads as threads in /proc/<pid>/task/")
dde38d5b805358c7 43b2a76b1a5abcc9833b463bef1
---------------- ---------------------------
%stddev %change %stddev
\ | \
53817815 -6.3% 50434581 will-it-scale.104.processes
517478 -6.3% 484947 will-it-scale.per_process_ops
53817815 -6.3% 50434581 will-it-scale.workload
3.61 ± 4% +2.8 6.38 mpstat.cpu.all.idle%
0.02 ± 15% +0.0 0.03 mpstat.cpu.all.soft%
1335406 ± 6% +43.6% 1917517 ± 15% numa-meminfo.node0.MemUsed
26350 ± 8% -32.8% 17695 ± 18% numa-meminfo.node1.Mapped
351261 ± 8% +91.6% 672866 ± 13% numa-numastat.node0.local_node
396995 +84.4% 731886 ± 12% numa-numastat.node0.numa_hit
965410 ± 9% +23.1% 1188249 ± 7% numa-vmstat.node0.numa_hit
6639 ± 8% -32.7% 4466 ± 18% numa-vmstat.node1.nr_mapped
14273927 ± 17% +200.1% 42839522 ±114% cpuidle.C6.time
46336 ± 7% +63.0% 75510 ± 68% cpuidle.C6.usage
12753 ± 6% +100.7% 25591 ± 73% cpuidle.POLL.time
55.00 -3.6% 53.00 vmstat.cpu.sy
43.00 +4.7% 45.00 vmstat.cpu.us
1537032 +38.4% 2127542 vmstat.memory.cache
1502 ± 5% +40.3% 2107 vmstat.system.cs
212844 -2.4% 207815 vmstat.system.in
8605 ± 2% +13.3% 9747 ± 3% slabinfo.kmalloc-256.active_objs
8654 ± 2% +13.2% 9795 ± 3% slabinfo.kmalloc-256.num_objs
14367 -37.7% 8957 slabinfo.proc_inode_cache.active_objs
299.50 -38.0% 185.83 slabinfo.proc_inode_cache.active_slabs
14395 -37.8% 8957 slabinfo.proc_inode_cache.num_objs
299.50 -38.0% 185.83 slabinfo.proc_inode_cache.num_slabs
224400 +269.5% 829186 ± 2% meminfo.Active
223868 +270.3% 828946 ± 2% meminfo.Active(anon)
1437431 +41.2% 2029667 meminfo.Cached
2408715 +27.3% 3065135 meminfo.Committed_AS
51176 -26.0% 37863 meminfo.Mapped
2783939 +35.1% 3760068 meminfo.Memused
251360 +237.0% 846957 ± 2% meminfo.Shmem
3254687 +15.5% 3760328 meminfo.max_used_kB
217046 ± 5% +122.4% 482701 ± 45% sched_debug.cfs_rq:/.min_vruntime.stddev
0.27 ±122% +481.4% 1.57 ± 44% sched_debug.cfs_rq:/.removed.runnable_avg.avg
2.64 ±122% +303.8% 10.65 ± 19% sched_debug.cfs_rq:/.removed.runnable_avg.stddev
0.27 ±122% +481.4% 1.57 ± 44% sched_debug.cfs_rq:/.removed.util_avg.avg
2.64 ±122% +303.8% 10.65 ± 19% sched_debug.cfs_rq:/.removed.util_avg.stddev
1499 ± 4% +30.4% 1955 ± 4% sched_debug.cfs_rq:/.runnable_avg.max
95.35 ± 7% +87.9% 179.14 ± 7% sched_debug.cfs_rq:/.runnable_avg.stddev
216964 ± 5% +122.5% 482737 ± 45% sched_debug.cfs_rq:/.spread0.stddev
1093 ± 3% +19.2% 1304 ± 7% sched_debug.cfs_rq:/.util_avg.max
796.11 ± 5% -25.3% 594.42 ± 6% sched_debug.cfs_rq:/.util_avg.min
56.73 ± 10% +62.3% 92.05 ± 8% sched_debug.cfs_rq:/.util_avg.stddev
668.86 ± 4% +90.2% 1272 ± 2% sched_debug.cfs_rq:/.util_est_enqueued.max
78.80 ± 3% +74.1% 137.19 ± 3% sched_debug.cfs_rq:/.util_est_enqueued.stddev
11.98 ± 5% +26.0% 15.09 ± 6% sched_debug.cpu.clock.stddev
0.16 ± 11% +27.3% 0.20 ± 6% sched_debug.cpu.nr_running.stddev
3824 ± 3% +22.4% 4681 sched_debug.cpu.nr_switches.avg
1470 ± 4% +34.5% 1978 ± 12% sched_debug.cpu.nr_switches.min
55875 +271.0% 207304 ± 2% proc-vmstat.nr_active_anon
70608 +6.7% 75350 proc-vmstat.nr_anon_pages
359234 +41.3% 507487 proc-vmstat.nr_file_pages
77322 +3.1% 79694 proc-vmstat.nr_inactive_anon
12886 -25.6% 9591 proc-vmstat.nr_mapped
2518 +4.6% 2634 proc-vmstat.nr_page_table_pages
62716 +237.7% 211808 ± 2% proc-vmstat.nr_shmem
24871 -4.3% 23800 proc-vmstat.nr_slab_reclaimable
55875 +271.0% 207304 ± 2% proc-vmstat.nr_zone_active_anon
77322 +3.1% 79694 proc-vmstat.nr_zone_inactive_anon
14803 ± 29% -62.5% 5557 ± 62% proc-vmstat.numa_hint_faults
13291 ± 28% -78.1% 2913 ± 64% proc-vmstat.numa_hint_faults_local
943105 +45.2% 1369352 proc-vmstat.numa_hit
849367 +50.2% 1275624 ± 2% proc-vmstat.numa_local
148542 ± 12% -64.5% 52803 ± 31% proc-vmstat.numa_pte_updates
86501 -67.2% 28371 ± 2% proc-vmstat.pgactivate
993722 +55.6% 1546560 ± 2% proc-vmstat.pgalloc_normal
977248 -12.4% 856153 proc-vmstat.pgfault
883680 +7.2% 947019 ± 4% proc-vmstat.pgfree
55.30 -55.3 0.00 perf-profile.calltrace.cycles-pp.read
45.47 -45.5 0.00 perf-profile.calltrace.cycles-pp.write
32.09 -32.1 0.00 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.read
25.67 -25.7 0.00 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.write
18.87 -18.9 0.00 perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.read
17.08 -17.1 0.00 perf-profile.calltrace.cycles-pp.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe.read
15.43 -15.4 0.00 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_safe_stack.read
13.05 -13.0 0.00 perf-profile.calltrace.cycles-pp.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe.write
12.95 -12.9 0.00 perf-profile.calltrace.cycles-pp.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe.read
12.64 -12.6 0.00 perf-profile.calltrace.cycles-pp.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe.read
12.10 -12.1 0.00 perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.write
10.85 -10.8 0.00 perf-profile.calltrace.cycles-pp.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.write
9.56 -9.6 0.00 perf-profile.calltrace.cycles-pp.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.write
8.90 -8.9 0.00 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_safe_stack.write
8.07 -8.1 0.00 perf-profile.calltrace.cycles-pp.new_sync_read.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe
7.57 -7.6 0.00 perf-profile.calltrace.cycles-pp.syscall_return_via_sysret.write
7.48 -7.5 0.00 perf-profile.calltrace.cycles-pp.syscall_return_via_sysret.read
6.75 -6.8 0.00 perf-profile.calltrace.cycles-pp.eventfd_read.new_sync_read.vfs_read.ksys_read.do_syscall_64
6.32 -6.3 0.00 perf-profile.calltrace.cycles-pp.__entry_text_start.read
6.30 -6.3 0.00 perf-profile.calltrace.cycles-pp.__entry_text_start.write
5.92 -5.9 0.00 perf-profile.calltrace.cycles-pp.eventfd_write.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
57.92 -57.9 0.00 perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
55.52 -55.5 0.00 perf-profile.children.cycles-pp.read
45.70 -45.7 0.00 perf-profile.children.cycles-pp.write
31.08 -31.1 0.00 perf-profile.children.cycles-pp.do_syscall_64
25.84 -25.8 0.00 perf-profile.children.cycles-pp.syscall_exit_to_user_mode
22.59 -22.6 0.00 perf-profile.children.cycles-pp.__entry_text_start
17.14 -17.1 0.00 perf-profile.children.cycles-pp.ksys_read
16.66 -16.7 0.00 perf-profile.children.cycles-pp.syscall_return_via_sysret
13.08 -13.1 0.00 perf-profile.children.cycles-pp.vfs_read
12.68 -12.7 0.00 perf-profile.children.cycles-pp.entry_SYSCALL_64_safe_stack
10.91 -10.9 0.00 perf-profile.children.cycles-pp.ksys_write
9.69 -9.7 0.00 perf-profile.children.cycles-pp.vfs_write
8.13 -8.1 0.00 perf-profile.children.cycles-pp.new_sync_read
6.96 -7.0 0.00 perf-profile.children.cycles-pp.eventfd_read
6.05 -6.1 0.00 perf-profile.children.cycles-pp.eventfd_write
5.20 -5.2 0.00 perf-profile.children.cycles-pp.security_file_permission
24.51 -24.5 0.00 perf-profile.self.cycles-pp.syscall_exit_to_user_mode
20.88 -20.9 0.00 perf-profile.self.cycles-pp.__entry_text_start
16.64 -16.6 0.00 perf-profile.self.cycles-pp.syscall_return_via_sysret
0.06 ± 4% +338.4% 0.26 ± 3% perf-stat.i.MPKI
1.897e+10 -3.8% 1.825e+10 perf-stat.i.branch-instructions
1.73 -0.0 1.69 perf-stat.i.branch-miss-rate%
3.262e+08 -5.6% 3.078e+08 perf-stat.i.branch-misses
555892 ± 4% +628.5% 4049656 ± 44% perf-stat.i.cache-misses
4726936 ± 5% +389.3% 23128136 ± 4% perf-stat.i.cache-references
1469 ± 5% +41.4% 2078 perf-stat.i.context-switches
2.98 +4.6% 3.11 perf-stat.i.cpi
649902 ± 5% -84.5% 100578 ± 65% perf-stat.i.cycles-between-cache-misses
0.39 -0.0 0.38 perf-stat.i.dTLB-load-miss-rate%
1.074e+08 -6.1% 1.008e+08 perf-stat.i.dTLB-load-misses
2.749e+10 -4.5% 2.625e+10 perf-stat.i.dTLB-loads
0.00 +0.0 0.00 perf-stat.i.dTLB-store-miss-rate%
102626 +3.6% 106273 perf-stat.i.dTLB-store-misses
1.86e+10 -4.6% 1.774e+10 perf-stat.i.dTLB-stores
1.084e+08 -5.9% 1.02e+08 perf-stat.i.iTLB-load-misses
9.457e+10 -3.8% 9.094e+10 perf-stat.i.instructions
1036 -7.8% 955.05 perf-stat.i.instructions-per-iTLB-miss
0.34 -4.3% 0.32 perf-stat.i.ipc
0.91 ± 5% -68.9% 0.28 ± 22% perf-stat.i.major-faults
1.99 ± 4% -65.9% 0.68 ± 36% perf-stat.i.metric.K/sec
625.66 -4.3% 598.87 perf-stat.i.metric.M/sec
3112 -13.0% 2709 perf-stat.i.minor-faults
103775 ± 3% +559.2% 684048 ± 43% perf-stat.i.node-load-misses
21080 ± 5% +1052.0% 242839 ± 61% perf-stat.i.node-store-misses
9471 ± 5% +92.5% 18236 ± 6% perf-stat.i.node-stores
3113 -13.0% 2709 perf-stat.i.page-faults
0.05 ± 5% +403.5% 0.25 ± 4% perf-stat.overall.MPKI
1.72 -0.0 1.69 perf-stat.overall.branch-miss-rate%
2.98 +4.6% 3.12 perf-stat.overall.cpi
500646 ± 3% -80.4% 98361 ± 64% perf-stat.overall.cycles-between-cache-misses
0.39 -0.0 0.38 perf-stat.overall.dTLB-load-miss-rate%
0.00 +0.0 0.00 perf-stat.overall.dTLB-store-miss-rate%
872.51 +2.2% 891.54 perf-stat.overall.instructions-per-iTLB-miss
0.34 -4.4% 0.32 perf-stat.overall.ipc
69.85 +17.4 87.30 ± 3% perf-stat.overall.node-load-miss-rate%
529805 +2.4% 542598 perf-stat.overall.path-length
1.891e+10 -3.8% 1.819e+10 perf-stat.ps.branch-instructions
3.251e+08 -5.6% 3.068e+08 perf-stat.ps.branch-misses
562046 ± 3% +618.1% 4036217 ± 44% perf-stat.ps.cache-misses
4761203 ± 5% +384.2% 23051468 ± 4% perf-stat.ps.cache-references
1464 ± 5% +41.4% 2071 perf-stat.ps.context-switches
1.07e+08 -6.1% 1.005e+08 perf-stat.ps.dTLB-load-misses
2.74e+10 -4.5% 2.617e+10 perf-stat.ps.dTLB-loads
102534 +3.3% 105923 perf-stat.ps.dTLB-store-misses
1.854e+10 -4.6% 1.768e+10 perf-stat.ps.dTLB-stores
1.08e+08 -5.9% 1.017e+08 perf-stat.ps.iTLB-load-misses
9.425e+10 -3.8% 9.064e+10 perf-stat.ps.instructions
0.91 ± 5% -68.9% 0.28 ± 22% perf-stat.ps.major-faults
3112 -13.2% 2701 perf-stat.ps.minor-faults
103489 ± 3% +558.8% 681779 ± 43% perf-stat.ps.node-load-misses
21019 ± 5% +1051.4% 242030 ± 61% perf-stat.ps.node-store-misses
9655 ± 5% +88.4% 18191 ± 6% perf-stat.ps.node-stores
3113 -13.2% 2701 perf-stat.ps.page-faults
2.851e+13 -4.0% 2.737e+13 perf-stat.total.instructions
29272 ± 20% +217.2% 92856 ± 68% softirqs.CPU0.RCU
28556 ± 22% +75.4% 50096 ± 14% softirqs.CPU1.RCU
25887 ± 26% +81.2% 46903 ± 8% softirqs.CPU10.RCU
25505 ± 21% +109.4% 53408 ± 28% softirqs.CPU100.RCU
25573 ± 21% +162.1% 67035 ± 77% softirqs.CPU101.RCU
25640 ± 21% +79.3% 45965 ± 10% softirqs.CPU102.RCU
26246 ± 22% +70.8% 44817 ± 9% softirqs.CPU103.RCU
26572 ± 23% +95.4% 51920 ± 16% softirqs.CPU11.RCU
26735 ± 22% +79.1% 47872 ± 13% softirqs.CPU12.RCU
26905 ± 23% +81.3% 48783 ± 15% softirqs.CPU13.RCU
26212 ± 25% +81.4% 47540 ± 17% softirqs.CPU14.RCU
30184 ± 21% +79.6% 54204 ± 14% softirqs.CPU15.RCU
30139 ± 21% +70.9% 51501 ± 5% softirqs.CPU16.RCU
30448 ± 21% +67.6% 51036 ± 6% softirqs.CPU17.RCU
30542 ± 22% +77.7% 54274 ± 13% softirqs.CPU18.RCU
30572 ± 21% +75.3% 53595 ± 8% softirqs.CPU19.RCU
27505 ± 23% +59.9% 43983 softirqs.CPU2.RCU
30494 ± 21% +69.0% 51532 ± 10% softirqs.CPU20.RCU
30404 ± 22% +74.1% 52921 ± 10% softirqs.CPU21.RCU
30124 ± 21% +58.1% 47615 ± 9% softirqs.CPU22.RCU
30208 ± 22% +70.2% 51406 ± 7% softirqs.CPU23.RCU
30500 ± 22% +80.9% 55170 ± 11% softirqs.CPU24.RCU
30498 ± 22% +70.0% 51841 ± 9% softirqs.CPU25.RCU
30289 ± 20% +67.2% 50632 ± 16% softirqs.CPU26.RCU
28512 ± 22% +57.4% 44882 ± 6% softirqs.CPU27.RCU
27424 ± 24% +71.2% 46950 ± 19% softirqs.CPU28.RCU
27144 ± 24% +85.1% 50233 ± 17% softirqs.CPU29.RCU
27428 ± 23% +167.5% 73379 ± 71% softirqs.CPU3.RCU
28605 ± 23% +80.0% 51501 ± 19% softirqs.CPU30.RCU
28492 ± 22% +68.5% 48006 ± 6% softirqs.CPU31.RCU
25999 ± 6% +96.0% 50956 ± 12% softirqs.CPU32.RCU
29537 ± 22% +98.2% 58554 ± 42% softirqs.CPU33.RCU
27675 ± 26% +67.1% 46239 softirqs.CPU34.RCU
28291 ± 23% +63.6% 46274 softirqs.CPU35.RCU
28351 ± 22% +93.7% 54930 ± 19% softirqs.CPU36.RCU
28464 ± 22% +81.1% 51539 ± 14% softirqs.CPU37.RCU
27484 ± 24% +70.2% 46785 ± 15% softirqs.CPU38.RCU
28024 ± 22% +62.7% 45599 softirqs.CPU39.RCU
26917 ± 22% +61.0% 43345 ± 5% softirqs.CPU4.RCU
28337 ± 22% +84.7% 52325 ± 12% softirqs.CPU40.RCU
28016 ± 22% +62.4% 45501 softirqs.CPU41.RCU
28151 ± 22% +98.1% 55757 ± 15% softirqs.CPU42.RCU
28436 ± 22% +76.5% 50190 ± 12% softirqs.CPU43.RCU
28569 ± 21% +172.5% 77851 ± 74% softirqs.CPU44.RCU
27278 ± 23% +71.2% 46688 ± 10% softirqs.CPU45.RCU
27129 ± 26% +78.2% 48341 ± 19% softirqs.CPU46.RCU
26888 ± 22% +71.4% 46081 ± 14% softirqs.CPU47.RCU
27117 ± 22% +100.5% 54364 ± 27% softirqs.CPU48.RCU
27341 ± 22% +147.5% 67674 ± 77% softirqs.CPU49.RCU
26690 ± 23% +67.4% 44668 ± 17% softirqs.CPU5.RCU
27347 ± 23% +69.9% 46451 ± 8% softirqs.CPU50.RCU
27667 ± 23% +65.8% 45881 ± 10% softirqs.CPU51.RCU
31835 ± 18% +179.5% 88967 ± 64% softirqs.CPU52.RCU
33461 ± 18% +63.1% 54589 ± 8% softirqs.CPU53.RCU
30080 ± 24% +59.8% 48060 ± 2% softirqs.CPU54.RCU
30214 ± 20% +158.6% 78143 ± 64% softirqs.CPU55.RCU
29822 ± 20% +57.3% 46914 ± 13% softirqs.CPU56.RCU
29623 ± 20% +72.3% 51053 ± 15% softirqs.CPU57.RCU
29602 ± 20% +66.1% 49167 ± 2% softirqs.CPU58.RCU
29167 ± 21% +68.3% 49077 ± 12% softirqs.CPU59.RCU
26548 ± 22% +65.7% 43979 ± 3% softirqs.CPU6.RCU
26874 ± 22% +70.5% 45825 ± 5% softirqs.CPU60.RCU
26636 ± 22% +78.1% 47440 ± 14% softirqs.CPU61.RCU
26156 ± 25% +85.3% 48456 ± 10% softirqs.CPU62.RCU
26814 ± 23% +92.8% 51700 ± 14% softirqs.CPU63.RCU
27189 ± 22% +84.2% 50080 ± 16% softirqs.CPU64.RCU
26912 ± 23% +88.2% 50645 ± 14% softirqs.CPU65.RCU
26746 ± 23% +81.1% 48444 ± 12% softirqs.CPU66.RCU
27265 ± 21% +80.9% 49332 ± 11% softirqs.CPU67.RCU
27233 ± 22% +75.9% 47915 ± 8% softirqs.CPU68.RCU
26988 ± 23% +73.6% 46859 ± 5% softirqs.CPU69.RCU
27187 ± 22% +59.3% 43299 ± 15% softirqs.CPU7.RCU
27143 ± 22% +81.7% 49309 ± 10% softirqs.CPU70.RCU
27192 ± 22% +91.1% 51955 ± 15% softirqs.CPU71.RCU
26927 ± 23% +76.0% 47381 ± 11% softirqs.CPU72.RCU
26744 ± 23% +77.8% 47540 ± 7% softirqs.CPU73.RCU
26600 ± 23% +72.3% 45837 ± 4% softirqs.CPU74.RCU
25739 ± 20% +82.1% 46883 ± 11% softirqs.CPU75.RCU
25979 ± 19% +79.8% 46706 ± 9% softirqs.CPU76.RCU
26063 ± 19% +81.5% 47295 ± 14% softirqs.CPU77.RCU
32477 ± 18% +59.1% 51671 ± 25% softirqs.CPU78.RCU
32395 ± 18% +50.0% 48577 ± 6% softirqs.CPU79.RCU
27059 ± 22% +65.2% 44713 ± 4% softirqs.CPU8.RCU
29857 ± 20% +68.1% 50192 ± 19% softirqs.CPU80.RCU
28839 ± 21% +84.8% 53291 ± 20% softirqs.CPU81.RCU
28496 ± 20% +76.8% 50394 ± 18% softirqs.CPU82.RCU
28388 ± 20% +66.6% 47286 ± 4% softirqs.CPU83.RCU
27419 ± 15% +83.1% 50211 ± 10% softirqs.CPU84.RCU
27740 ± 21% +105.0% 56870 ± 41% softirqs.CPU85.RCU
27446 ± 23% +69.0% 46371 softirqs.CPU86.RCU
28118 ± 22% +63.5% 45970 softirqs.CPU87.RCU
27945 ± 21% +100.2% 55955 ± 23% softirqs.CPU88.RCU
28060 ± 21% +86.2% 52260 ± 14% softirqs.CPU89.RCU
26821 ± 22% +71.5% 45988 ± 14% softirqs.CPU9.RCU
25391 ± 22% +75.7% 44618 ± 14% softirqs.CPU90.RCU
25372 ± 22% +66.8% 42331 softirqs.CPU91.RCU
25418 ± 21% +85.1% 47051 ± 10% softirqs.CPU92.RCU
25647 ± 21% +65.2% 42374 softirqs.CPU93.RCU
25635 ± 21% +90.0% 48706 ± 10% softirqs.CPU94.RCU
25658 ± 21% +85.6% 47613 ± 16% softirqs.CPU95.RCU
25595 ± 21% +193.0% 74990 ± 79% softirqs.CPU96.RCU
25644 ± 21% +80.8% 46355 ± 11% softirqs.CPU97.RCU
25404 ± 21% +87.6% 47661 ± 21% softirqs.CPU98.RCU
25554 ± 21% +72.6% 44106 ± 10% softirqs.CPU99.RCU
2897508 ± 21% +83.5% 5316315 softirqs.RCU
40871 ± 12% +43.0% 58456 ± 2% softirqs.TIMER
0.02 ± 2% -100.0% 0.00 perf-sched.sch_delay.avg.ms.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown]
0.92 ±217% -100.0% 0.00 perf-sched.sch_delay.avg.ms.devkmsg_read.vfs_read.ksys_read.do_syscall_64
0.02 ± 5% -100.0% 0.00 perf-sched.sch_delay.avg.ms.do_nanosleep.hrtimer_nanosleep.__x64_sys_nanosleep.do_syscall_64
4.58 ± 60% -100.0% 0.00 perf-sched.sch_delay.avg.ms.do_syslog.part.0.kmsg_read.vfs_read
0.60 ± 18% -100.0% 0.00 perf-sched.sch_delay.avg.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
0.03 ± 66% -100.0% 0.00 perf-sched.sch_delay.avg.ms.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64
0.03 ± 26% -100.0% 0.00 perf-sched.sch_delay.avg.ms.futex_wait_queue_me.futex_wait.do_futex.__x64_sys_futex
0.01 ± 9% -100.0% 0.00 perf-sched.sch_delay.avg.ms.pipe_read.new_sync_read.vfs_read.ksys_read
9.01 -100.0% 0.00 perf-sched.sch_delay.avg.ms.preempt_schedule_common.__cond_resched.stop_one_cpu.sched_exec.bprm_execve
0.02 ± 2% -100.0% 0.00 perf-sched.sch_delay.avg.ms.rcu_gp_kthread.kthread.ret_from_fork
0.28 ±134% -100.0% 0.00 perf-sched.sch_delay.avg.ms.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait.__x64_sys_epoll_wait
0.02 ± 9% -100.0% 0.00 perf-sched.sch_delay.avg.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_select
0.00 -100.0% 0.00 perf-sched.sch_delay.avg.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_sys_poll
0.02 ± 4% -100.0% 0.00 perf-sched.sch_delay.avg.ms.schedule_timeout.kcompactd.kthread.ret_from_fork
0.02 -100.0% 0.00 perf-sched.sch_delay.avg.ms.schedule_timeout.rcu_gp_kthread.kthread.ret_from_fork
0.01 ± 4% -100.0% 0.00 perf-sched.sch_delay.avg.ms.smpboot_thread_fn.kthread.ret_from_fork
0.97 ± 15% -100.0% 0.00 perf-sched.sch_delay.avg.ms.wait_for_partner.fifo_open.do_dentry_open.do_open.isra
0.04 ± 24% -100.0% 0.00 perf-sched.sch_delay.avg.ms.worker_thread.kthread.ret_from_fork
0.02 ± 7% -100.0% 0.00 perf-sched.sch_delay.max.ms.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown]
2.07 ±220% -100.0% 0.00 perf-sched.sch_delay.max.ms.devkmsg_read.vfs_read.ksys_read.do_syscall_64
0.02 ± 24% -100.0% 0.00 perf-sched.sch_delay.max.ms.do_nanosleep.hrtimer_nanosleep.__x64_sys_nanosleep.do_syscall_64
14.48 ± 56% -100.0% 0.00 perf-sched.sch_delay.max.ms.do_syslog.part.0.kmsg_read.vfs_read
14.35 ± 26% -100.0% 0.00 perf-sched.sch_delay.max.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
2.30 ±219% -100.0% 0.00 perf-sched.sch_delay.max.ms.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64
5.48 ±102% -100.0% 0.00 perf-sched.sch_delay.max.ms.exit_to_user_mode_prepare.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe.[unknown]
0.04 ± 46% -100.0% 0.00 perf-sched.sch_delay.max.ms.futex_wait_queue_me.futex_wait.do_futex.__x64_sys_futex
0.22 ± 36% -100.0% 0.00 perf-sched.sch_delay.max.ms.pipe_read.new_sync_read.vfs_read.ksys_read
0.04 ± 45% -100.0% 0.00 perf-sched.sch_delay.max.ms.preempt_schedule_common.__cond_resched.stop_one_cpu.affine_move_task.__set_cpus_allowed_ptr
22.37 ± 19% -100.0% 0.00 perf-sched.sch_delay.max.ms.preempt_schedule_common.__cond_resched.stop_one_cpu.sched_exec.bprm_execve
0.03 ± 17% -100.0% 0.00 perf-sched.sch_delay.max.ms.rcu_gp_kthread.kthread.ret_from_fork
4.26 ±131% -100.0% 0.00 perf-sched.sch_delay.max.ms.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait.__x64_sys_epoll_wait
0.02 ± 10% -100.0% 0.00 perf-sched.sch_delay.max.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_select
0.03 ± 28% -100.0% 0.00 perf-sched.sch_delay.max.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_sys_poll
0.02 ± 22% -100.0% 0.00 perf-sched.sch_delay.max.ms.schedule_timeout.kcompactd.kthread.ret_from_fork
0.05 ± 66% -100.0% 0.00 perf-sched.sch_delay.max.ms.schedule_timeout.rcu_gp_kthread.kthread.ret_from_fork
0.04 ± 61% -100.0% 0.00 perf-sched.sch_delay.max.ms.smpboot_thread_fn.kthread.ret_from_fork
14.51 ± 23% -100.0% 0.00 perf-sched.sch_delay.max.ms.wait_for_partner.fifo_open.do_dentry_open.do_open.isra
9.11 ± 43% -100.0% 0.00 perf-sched.sch_delay.max.ms.worker_thread.kthread.ret_from_fork
0.18 ± 6% -100.0% 0.00 perf-sched.total_sch_delay.average.ms
22.51 ± 19% -100.0% 0.00 perf-sched.total_sch_delay.max.ms
145.21 ± 9% -100.0% 0.00 perf-sched.total_wait_and_delay.average.ms
12308 ± 6% -100.0% 0.00 perf-sched.total_wait_and_delay.count.ms
8429 ± 9% -100.0% 0.00 perf-sched.total_wait_and_delay.max.ms
145.02 ± 9% -100.0% 0.00 perf-sched.total_wait_time.average.ms
8429 ± 9% -100.0% 0.00 perf-sched.total_wait_time.max.ms
882.81 ± 4% -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown]
917.42 ± 28% -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.devkmsg_read.vfs_read.ksys_read.do_syscall_64
803.52 ± 4% -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.do_nanosleep.hrtimer_nanosleep.__x64_sys_nanosleep.do_syscall_64
1002 ± 31% -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.do_syslog.part.0.kmsg_read.vfs_read
265.93 ± 3% -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
109.91 ± 7% -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64
0.24 ± 19% -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt.[unknown]
0.16 ± 16% -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_call_function_single.[unknown]
0.12 ± 35% -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_reschedule_ipi.[unknown]
0.27 ± 44% -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.exit_to_user_mode_prepare.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe.[unknown]
208.74 ± 4% -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.pipe_read.new_sync_read.vfs_read.ksys_read
8.98 -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.preempt_schedule_common.__cond_resched.stop_one_cpu.affine_move_task.__set_cpus_allowed_ptr
9.02 -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.preempt_schedule_common.__cond_resched.stop_one_cpu.sched_exec.bprm_execve
2.64 ± 4% -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.rcu_gp_kthread.kthread.ret_from_fork
526.58 ± 47% -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait.__x64_sys_epoll_wait
157.83 ± 5% -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_sys_poll
478.58 -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.schedule_timeout.kcompactd.kthread.ret_from_fork
6.98 ± 20% -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.schedule_timeout.rcu_gp_kthread.kthread.ret_from_fork
586.89 ± 2% -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.smpboot_thread_fn.kthread.ret_from_fork
584.74 ± 11% -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.worker_thread.kthread.ret_from_fork
10.00 -100.0% 0.00 perf-sched.wait_and_delay.count.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown]
4.67 ± 10% -100.0% 0.00 perf-sched.wait_and_delay.count.devkmsg_read.vfs_read.ksys_read.do_syscall_64
22.50 ± 4% -100.0% 0.00 perf-sched.wait_and_delay.count.do_nanosleep.hrtimer_nanosleep.__x64_sys_nanosleep.do_syscall_64
4.33 ± 10% -100.0% 0.00 perf-sched.wait_and_delay.count.do_syslog.part.0.kmsg_read.vfs_read
236.83 -100.0% 0.00 perf-sched.wait_and_delay.count.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
268.00 -100.0% 0.00 perf-sched.wait_and_delay.count.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64
1662 ± 16% -100.0% 0.00 perf-sched.wait_and_delay.count.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt.[unknown]
238.17 ± 4% -100.0% 0.00 perf-sched.wait_and_delay.count.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_call_function_single.[unknown]
181.33 ± 13% -100.0% 0.00 perf-sched.wait_and_delay.count.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_reschedule_ipi.[unknown]
3252 ± 4% -100.0% 0.00 perf-sched.wait_and_delay.count.exit_to_user_mode_prepare.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe.[unknown]
624.83 ± 3% -100.0% 0.00 perf-sched.wait_and_delay.count.pipe_read.new_sync_read.vfs_read.ksys_read
1248 -100.0% 0.00 perf-sched.wait_and_delay.count.preempt_schedule_common.__cond_resched.stop_one_cpu.affine_move_task.__set_cpus_allowed_ptr
212.33 -100.0% 0.00 perf-sched.wait_and_delay.count.preempt_schedule_common.__cond_resched.stop_one_cpu.sched_exec.bprm_execve
111.00 ± 47% -100.0% 0.00 perf-sched.wait_and_delay.count.rcu_gp_kthread.kthread.ret_from_fork
19.50 ± 51% -100.0% 0.00 perf-sched.wait_and_delay.count.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait.__x64_sys_epoll_wait
158.50 ± 6% -100.0% 0.00 perf-sched.wait_and_delay.count.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_sys_poll
39.67 -100.0% 0.00 perf-sched.wait_and_delay.count.schedule_timeout.kcompactd.kthread.ret_from_fork
1469 ± 26% -100.0% 0.00 perf-sched.wait_and_delay.count.schedule_timeout.rcu_gp_kthread.kthread.ret_from_fork
1834 ± 2% -100.0% 0.00 perf-sched.wait_and_delay.count.smpboot_thread_fn.kthread.ret_from_fork
601.00 ± 11% -100.0% 0.00 perf-sched.wait_and_delay.count.worker_thread.kthread.ret_from_fork
999.44 -100.0% 0.00 perf-sched.wait_and_delay.max.ms.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown]
2750 ± 28% -100.0% 0.00 perf-sched.wait_and_delay.max.ms.devkmsg_read.vfs_read.ksys_read.do_syscall_64
1000 -100.0% 0.00 perf-sched.wait_and_delay.max.ms.do_nanosleep.hrtimer_nanosleep.__x64_sys_nanosleep.do_syscall_64
2763 ± 28% -100.0% 0.00 perf-sched.wait_and_delay.max.ms.do_syslog.part.0.kmsg_read.vfs_read
1841 ±102% -100.0% 0.00 perf-sched.wait_and_delay.max.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
1021 -100.0% 0.00 perf-sched.wait_and_delay.max.ms.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64
23.20 ± 62% -100.0% 0.00 perf-sched.wait_and_delay.max.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt.[unknown]
8.54 ± 46% -100.0% 0.00 perf-sched.wait_and_delay.max.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_call_function_single.[unknown]
6.14 ± 99% -100.0% 0.00 perf-sched.wait_and_delay.max.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_reschedule_ipi.[unknown]
206.18 ±172% -100.0% 0.00 perf-sched.wait_and_delay.max.ms.exit_to_user_mode_prepare.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe.[unknown]
2752 ± 28% -100.0% 0.00 perf-sched.wait_and_delay.max.ms.pipe_read.new_sync_read.vfs_read.ksys_read
2177 -100.0% 0.00 perf-sched.wait_and_delay.max.ms.preempt_schedule_common.__cond_resched.stop_one_cpu.affine_move_task.__set_cpus_allowed_ptr
22.37 ± 19% -100.0% 0.00 perf-sched.wait_and_delay.max.ms.preempt_schedule_common.__cond_resched.stop_one_cpu.sched_exec.bprm_execve
5.02 -100.0% 0.00 perf-sched.wait_and_delay.max.ms.rcu_gp_kthread.kthread.ret_from_fork
4004 ± 60% -100.0% 0.00 perf-sched.wait_and_delay.max.ms.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait.__x64_sys_epoll_wait
5816 -100.0% 0.00 perf-sched.wait_and_delay.max.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_sys_poll
505.01 -100.0% 0.00 perf-sched.wait_and_delay.max.ms.schedule_timeout.kcompactd.kthread.ret_from_fork
362.67 ± 42% -100.0% 0.00 perf-sched.wait_and_delay.max.ms.schedule_timeout.rcu_gp_kthread.kthread.ret_from_fork
7190 ± 18% -100.0% 0.00 perf-sched.wait_and_delay.max.ms.smpboot_thread_fn.kthread.ret_from_fork
7974 ± 11% -100.0% 0.00 perf-sched.wait_and_delay.max.ms.worker_thread.kthread.ret_from_fork
882.79 ± 4% -100.0% 0.00 perf-sched.wait_time.avg.ms.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown]
916.50 ± 28% -100.0% 0.00 perf-sched.wait_time.avg.ms.devkmsg_read.vfs_read.ksys_read.do_syscall_64
803.50 ± 4% -100.0% 0.00 perf-sched.wait_time.avg.ms.do_nanosleep.hrtimer_nanosleep.__x64_sys_nanosleep.do_syscall_64
997.70 ± 31% -100.0% 0.00 perf-sched.wait_time.avg.ms.do_syslog.part.0.kmsg_read.vfs_read
265.33 ± 3% -100.0% 0.00 perf-sched.wait_time.avg.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
109.88 ± 7% -100.0% 0.00 perf-sched.wait_time.avg.ms.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64
0.24 ± 19% -100.0% 0.00 perf-sched.wait_time.avg.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt.[unknown]
0.16 ± 16% -100.0% 0.00 perf-sched.wait_time.avg.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_call_function_single.[unknown]
0.10 ± 86% -100.0% 0.00 perf-sched.wait_time.avg.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_irq_work.[unknown]
0.12 ± 35% -100.0% 0.00 perf-sched.wait_time.avg.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_reschedule_ipi.[unknown]
0.27 ± 45% -100.0% 0.00 perf-sched.wait_time.avg.ms.exit_to_user_mode_prepare.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe.[unknown]
208.72 ± 4% -100.0% 0.00 perf-sched.wait_time.avg.ms.pipe_read.new_sync_read.vfs_read.ksys_read
9.91 ± 20% -100.0% 0.00 perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.dput.__fput.task_work_run
439.75 ±115% -100.0% 0.00 perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.generic_perform_write.__generic_file_write_iter.generic_file_write_iter
8.98 -100.0% 0.00 perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.stop_one_cpu.affine_move_task.__set_cpus_allowed_ptr
2.62 ± 4% -100.0% 0.00 perf-sched.wait_time.avg.ms.rcu_gp_kthread.kthread.ret_from_fork
526.30 ± 47% -100.0% 0.00 perf-sched.wait_time.avg.ms.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait.__x64_sys_epoll_wait
157.83 ± 5% -100.0% 0.00 perf-sched.wait_time.avg.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_sys_poll
478.56 -100.0% 0.00 perf-sched.wait_time.avg.ms.schedule_timeout.kcompactd.kthread.ret_from_fork
3.13 ± 44% -100.0% 0.00 perf-sched.wait_time.avg.ms.schedule_timeout.khugepaged.kthread.ret_from_fork
6.96 ± 20% -100.0% 0.00 perf-sched.wait_time.avg.ms.schedule_timeout.rcu_gp_kthread.kthread.ret_from_fork
586.88 ± 2% -100.0% 0.00 perf-sched.wait_time.avg.ms.smpboot_thread_fn.kthread.ret_from_fork
2.18 ± 19% -100.0% 0.00 perf-sched.wait_time.avg.ms.wait_for_partner.fifo_open.do_dentry_open.do_open.isra
584.70 ± 11% -100.0% 0.00 perf-sched.wait_time.avg.ms.worker_thread.kthread.ret_from_fork
999.42 -100.0% 0.00 perf-sched.wait_time.max.ms.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown]
2748 ± 28% -100.0% 0.00 perf-sched.wait_time.max.ms.devkmsg_read.vfs_read.ksys_read.do_syscall_64
1000 -100.0% 0.00 perf-sched.wait_time.max.ms.do_nanosleep.hrtimer_nanosleep.__x64_sys_nanosleep.do_syscall_64
2754 ± 28% -100.0% 0.00 perf-sched.wait_time.max.ms.do_syslog.part.0.kmsg_read.vfs_read
1841 ±102% -100.0% 0.00 perf-sched.wait_time.max.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
1021 -100.0% 0.00 perf-sched.wait_time.max.ms.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64
23.20 ± 62% -100.0% 0.00 perf-sched.wait_time.max.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt.[unknown]
8.54 ± 46% -100.0% 0.00 perf-sched.wait_time.max.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_call_function_single.[unknown]
1.24 ±105% -100.0% 0.00 perf-sched.wait_time.max.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_irq_work.[unknown]
6.14 ± 99% -100.0% 0.00 perf-sched.wait_time.max.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_reschedule_ipi.[unknown]
206.17 ±172% -100.0% 0.00 perf-sched.wait_time.max.ms.exit_to_user_mode_prepare.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe.[unknown]
2752 ± 28% -100.0% 0.00 perf-sched.wait_time.max.ms.pipe_read.new_sync_read.vfs_read.ksys_read
11.86 ± 16% -100.0% 0.00 perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.dput.__fput.task_work_run
3351 ± 99% -100.0% 0.00 perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.generic_perform_write.__generic_file_write_iter.generic_file_write_iter
2177 -100.0% 0.00 perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.stop_one_cpu.affine_move_task.__set_cpus_allowed_ptr
5.00 -100.0% 0.00 perf-sched.wait_time.max.ms.rcu_gp_kthread.kthread.ret_from_fork
4004 ± 60% -100.0% 0.00 perf-sched.wait_time.max.ms.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait.__x64_sys_epoll_wait
5816 -100.0% 0.00 perf-sched.wait_time.max.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_sys_poll
504.99 -100.0% 0.00 perf-sched.wait_time.max.ms.schedule_timeout.kcompactd.kthread.ret_from_fork
3.13 ± 44% -100.0% 0.00 perf-sched.wait_time.max.ms.schedule_timeout.khugepaged.kthread.ret_from_fork
362.65 ± 42% -100.0% 0.00 perf-sched.wait_time.max.ms.schedule_timeout.rcu_gp_kthread.kthread.ret_from_fork
7190 ± 18% -100.0% 0.00 perf-sched.wait_time.max.ms.smpboot_thread_fn.kthread.ret_from_fork
12.80 -100.0% 0.00 perf-sched.wait_time.max.ms.wait_for_partner.fifo_open.do_dentry_open.do_open.isra
7974 ± 11% -100.0% 0.00 perf-sched.wait_time.max.ms.worker_thread.kthread.ret_from_fork
104127 -31.8% 71024 ± 2% interrupts.CAL:Function_call_interrupts
7546 -100.0% 0.00 interrupts.CPU0.NMI:Non-maskable_interrupts
7546 -100.0% 0.00 interrupts.CPU0.PMI:Performance_monitoring_interrupts
1652 ± 23% -54.3% 755.83 ± 37% interrupts.CPU1.CAL:Function_call_interrupts
5650 ± 33% -100.0% 0.00 interrupts.CPU1.NMI:Non-maskable_interrupts
5650 ± 33% -100.0% 0.00 interrupts.CPU1.PMI:Performance_monitoring_interrupts
677.33 ± 29% -38.5% 416.83 ± 16% interrupts.CPU1.RES:Rescheduling_interrupts
6911 ± 20% -100.0% 0.00 interrupts.CPU10.NMI:Non-maskable_interrupts
6911 ± 20% -100.0% 0.00 interrupts.CPU10.PMI:Performance_monitoring_interrupts
329.33 +99.6% 657.33 ± 38% interrupts.CPU10.RES:Rescheduling_interrupts
7180 ± 20% -100.0% 0.00 interrupts.CPU100.NMI:Non-maskable_interrupts
7180 ± 20% -100.0% 0.00 interrupts.CPU100.PMI:Performance_monitoring_interrupts
331.33 ± 7% +56.6% 519.00 ± 40% interrupts.CPU100.RES:Rescheduling_interrupts
7181 ± 20% -100.0% 0.00 interrupts.CPU101.NMI:Non-maskable_interrupts
7181 ± 20% -100.0% 0.00 interrupts.CPU101.PMI:Performance_monitoring_interrupts
324.67 ± 4% +40.1% 454.83 ± 13% interrupts.CPU101.RES:Rescheduling_interrupts
7181 ± 20% -100.0% 0.00 interrupts.CPU102.NMI:Non-maskable_interrupts
7181 ± 20% -100.0% 0.00 interrupts.CPU102.PMI:Performance_monitoring_interrupts
7177 ± 20% -100.0% 0.00 interrupts.CPU103.NMI:Non-maskable_interrupts
7177 ± 20% -100.0% 0.00 interrupts.CPU103.PMI:Performance_monitoring_interrupts
6911 ± 20% -100.0% 0.00 interrupts.CPU11.NMI:Non-maskable_interrupts
6911 ± 20% -100.0% 0.00 interrupts.CPU11.PMI:Performance_monitoring_interrupts
6909 ± 20% -100.0% 0.00 interrupts.CPU12.NMI:Non-maskable_interrupts
6909 ± 20% -100.0% 0.00 interrupts.CPU12.PMI:Performance_monitoring_interrupts
339.33 ± 6% +38.2% 468.83 ± 35% interrupts.CPU12.RES:Rescheduling_interrupts
6914 ± 20% -100.0% 0.00 interrupts.CPU13.NMI:Non-maskable_interrupts
6914 ± 20% -100.0% 0.00 interrupts.CPU13.PMI:Performance_monitoring_interrupts
340.83 ± 7% +22.1% 416.17 ± 11% interrupts.CPU13.RES:Rescheduling_interrupts
7549 -100.0% 0.00 interrupts.CPU14.NMI:Non-maskable_interrupts
7549 -100.0% 0.00 interrupts.CPU14.PMI:Performance_monitoring_interrupts
7542 -100.0% 0.00 interrupts.CPU15.NMI:Non-maskable_interrupts
7542 -100.0% 0.00 interrupts.CPU15.PMI:Performance_monitoring_interrupts
7555 -100.0% 0.00 interrupts.CPU16.NMI:Non-maskable_interrupts
7555 -100.0% 0.00 interrupts.CPU16.PMI:Performance_monitoring_interrupts
324.83 +29.8% 421.50 ± 13% interrupts.CPU16.RES:Rescheduling_interrupts
6289 ± 28% -100.0% 0.00 interrupts.CPU17.NMI:Non-maskable_interrupts
6289 ± 28% -100.0% 0.00 interrupts.CPU17.PMI:Performance_monitoring_interrupts
321.67 +18.7% 381.83 ± 4% interrupts.CPU17.RES:Rescheduling_interrupts
6919 ± 20% -100.0% 0.00 interrupts.CPU18.NMI:Non-maskable_interrupts
6919 ± 20% -100.0% 0.00 interrupts.CPU18.PMI:Performance_monitoring_interrupts
324.17 +73.3% 561.67 ± 33% interrupts.CPU18.RES:Rescheduling_interrupts
6914 ± 20% -100.0% 0.00 interrupts.CPU19.NMI:Non-maskable_interrupts
6914 ± 20% -100.0% 0.00 interrupts.CPU19.PMI:Performance_monitoring_interrupts
321.17 +28.2% 411.83 ± 12% interrupts.CPU19.RES:Rescheduling_interrupts
1214 ± 42% -55.4% 541.50 ± 16% interrupts.CPU2.CAL:Function_call_interrupts
5651 ± 33% -100.0% 0.00 interrupts.CPU2.NMI:Non-maskable_interrupts
5651 ± 33% -100.0% 0.00 interrupts.CPU2.PMI:Performance_monitoring_interrupts
501.17 ± 18% -24.5% 378.50 ± 13% interrupts.CPU2.RES:Rescheduling_interrupts
6914 ± 20% -100.0% 0.00 interrupts.CPU20.NMI:Non-maskable_interrupts
6914 ± 20% -100.0% 0.00 interrupts.CPU20.PMI:Performance_monitoring_interrupts
319.17 +19.1% 380.17 ± 5% interrupts.CPU20.RES:Rescheduling_interrupts
7544 -100.0% 0.00 interrupts.CPU21.NMI:Non-maskable_interrupts
7544 -100.0% 0.00 interrupts.CPU21.PMI:Performance_monitoring_interrupts
332.00 ± 7% +23.4% 409.67 ± 15% interrupts.CPU21.RES:Rescheduling_interrupts
7544 -100.0% 0.00 interrupts.CPU22.NMI:Non-maskable_interrupts
7544 -100.0% 0.00 interrupts.CPU22.PMI:Performance_monitoring_interrupts
318.00 +52.5% 485.00 ± 41% interrupts.CPU22.RES:Rescheduling_interrupts
7544 -100.0% 0.00 interrupts.CPU23.NMI:Non-maskable_interrupts
7544 -100.0% 0.00 interrupts.CPU23.PMI:Performance_monitoring_interrupts
328.17 ± 6% +17.1% 384.33 ± 7% interrupts.CPU23.RES:Rescheduling_interrupts
6914 ± 20% -100.0% 0.00 interrupts.CPU24.NMI:Non-maskable_interrupts
6914 ± 20% -100.0% 0.00 interrupts.CPU24.PMI:Performance_monitoring_interrupts
322.33 +17.9% 380.17 ± 7% interrupts.CPU24.RES:Rescheduling_interrupts
5655 ± 33% -100.0% 0.00 interrupts.CPU25.NMI:Non-maskable_interrupts
5655 ± 33% -100.0% 0.00 interrupts.CPU25.PMI:Performance_monitoring_interrupts
1325 ± 23% -46.9% 704.17 ± 39% interrupts.CPU26.CAL:Function_call_interrupts
6526 ± 28% -100.0% 0.00 interrupts.CPU26.NMI:Non-maskable_interrupts
6526 ± 28% -100.0% 0.00 interrupts.CPU26.PMI:Performance_monitoring_interrupts
1712 ± 39% -66.3% 577.83 ± 3% interrupts.CPU27.CAL:Function_call_interrupts
6527 ± 28% -100.0% 1.00 ±223% interrupts.CPU27.NMI:Non-maskable_interrupts
6527 ± 28% -100.0% 1.00 ±223% interrupts.CPU27.PMI:Performance_monitoring_interrupts
732.83 ± 41% -54.4% 334.50 ± 5% interrupts.CPU27.RES:Rescheduling_interrupts
1110 ± 26% -47.9% 578.67 ± 3% interrupts.CPU28.CAL:Function_call_interrupts
6526 ± 28% -100.0% 0.00 interrupts.CPU28.NMI:Non-maskable_interrupts
6526 ± 28% -100.0% 0.00 interrupts.CPU28.PMI:Performance_monitoring_interrupts
461.67 ± 17% -27.7% 334.00 ± 5% interrupts.CPU28.RES:Rescheduling_interrupts
675.50 ± 5% -16.7% 563.00 ± 4% interrupts.CPU29.CAL:Function_call_interrupts
7180 ± 20% -100.0% 0.00 interrupts.CPU29.NMI:Non-maskable_interrupts
7180 ± 20% -100.0% 0.00 interrupts.CPU29.PMI:Performance_monitoring_interrupts
6281 ± 28% -100.0% 0.00 interrupts.CPU3.NMI:Non-maskable_interrupts
6281 ± 28% -100.0% 0.00 interrupts.CPU3.PMI:Performance_monitoring_interrupts
7181 ± 20% -100.0% 1.00 ±223% interrupts.CPU30.NMI:Non-maskable_interrupts
7181 ± 20% -100.0% 1.00 ±223% interrupts.CPU30.PMI:Performance_monitoring_interrupts
7181 ± 20% -100.0% 0.00 interrupts.CPU31.NMI:Non-maskable_interrupts
7181 ± 20% -100.0% 0.00 interrupts.CPU31.PMI:Performance_monitoring_interrupts
7180 ± 20% -100.0% 0.00 interrupts.CPU32.NMI:Non-maskable_interrupts
7180 ± 20% -100.0% 0.00 interrupts.CPU32.PMI:Performance_monitoring_interrupts
6528 ± 28% -100.0% 0.00 interrupts.CPU33.NMI:Non-maskable_interrupts
6528 ± 28% -100.0% 0.00 interrupts.CPU33.PMI:Performance_monitoring_interrupts
7182 ± 20% -100.0% 0.00 interrupts.CPU34.NMI:Non-maskable_interrupts
7182 ± 20% -100.0% 0.00 interrupts.CPU34.PMI:Performance_monitoring_interrupts
5875 ± 33% -100.0% 0.00 interrupts.CPU35.NMI:Non-maskable_interrupts
5875 ± 33% -100.0% 0.00 interrupts.CPU35.PMI:Performance_monitoring_interrupts
5874 ± 33% -100.0% 0.00 interrupts.CPU36.NMI:Non-maskable_interrupts
5874 ± 33% -100.0% 0.00 interrupts.CPU36.PMI:Performance_monitoring_interrupts
5873 ± 33% -100.0% 0.00 interrupts.CPU37.NMI:Non-maskable_interrupts
5873 ± 33% -100.0% 0.00 interrupts.CPU37.PMI:Performance_monitoring_interrupts
5873 ± 33% -100.0% 0.00 interrupts.CPU38.NMI:Non-maskable_interrupts
5873 ± 33% -100.0% 0.00 interrupts.CPU38.PMI:Performance_monitoring_interrupts
5222 ± 35% -100.0% 0.00 interrupts.CPU39.NMI:Non-maskable_interrupts
5222 ± 35% -100.0% 0.00 interrupts.CPU39.PMI:Performance_monitoring_interrupts
7538 -100.0% 0.00 interrupts.CPU4.NMI:Non-maskable_interrupts
7538 -100.0% 0.00 interrupts.CPU4.PMI:Performance_monitoring_interrupts
6528 ± 28% -100.0% 0.00 interrupts.CPU40.NMI:Non-maskable_interrupts
6528 ± 28% -100.0% 0.00 interrupts.CPU40.PMI:Performance_monitoring_interrupts
315.00 +17.4% 369.83 ± 12% interrupts.CPU40.RES:Rescheduling_interrupts
6528 ± 28% -100.0% 0.00 interrupts.CPU41.NMI:Non-maskable_interrupts
6528 ± 28% -100.0% 0.00 interrupts.CPU41.PMI:Performance_monitoring_interrupts
6528 ± 28% -100.0% 0.00 interrupts.CPU42.NMI:Non-maskable_interrupts
6528 ± 28% -100.0% 0.00 interrupts.CPU42.PMI:Performance_monitoring_interrupts
7181 ± 20% -100.0% 0.00 interrupts.CPU43.NMI:Non-maskable_interrupts
7181 ± 20% -100.0% 0.00 interrupts.CPU43.PMI:Performance_monitoring_interrupts
314.33 +25.9% 395.67 ± 28% interrupts.CPU43.RES:Rescheduling_interrupts
6529 ± 28% -100.0% 0.00 interrupts.CPU44.NMI:Non-maskable_interrupts
6529 ± 28% -100.0% 0.00 interrupts.CPU44.PMI:Performance_monitoring_interrupts
313.17 +10.8% 347.00 ± 7% interrupts.CPU44.RES:Rescheduling_interrupts
6527 ± 28% -100.0% 0.00 interrupts.CPU45.NMI:Non-maskable_interrupts
6527 ± 28% -100.0% 0.00 interrupts.CPU45.PMI:Performance_monitoring_interrupts
6527 ± 28% -100.0% 0.00 interrupts.CPU46.NMI:Non-maskable_interrupts
6527 ± 28% -100.0% 0.00 interrupts.CPU46.PMI:Performance_monitoring_interrupts
6528 ± 28% -100.0% 0.00 interrupts.CPU47.NMI:Non-maskable_interrupts
6528 ± 28% -100.0% 0.00 interrupts.CPU47.PMI:Performance_monitoring_interrupts
6528 ± 28% -100.0% 0.00 interrupts.CPU48.NMI:Non-maskable_interrupts
6528 ± 28% -100.0% 0.00 interrupts.CPU48.PMI:Performance_monitoring_interrupts
317.00 ± 3% +9.1% 346.00 ± 5% interrupts.CPU48.RES:Rescheduling_interrupts
5874 ± 33% -100.0% 0.00 interrupts.CPU49.NMI:Non-maskable_interrupts
5874 ± 33% -100.0% 0.00 interrupts.CPU49.PMI:Performance_monitoring_interrupts
318.67 ± 3% +23.7% 394.33 ± 25% interrupts.CPU49.RES:Rescheduling_interrupts
6910 ± 20% -100.0% 0.00 interrupts.CPU5.NMI:Non-maskable_interrupts
6910 ± 20% -100.0% 0.00 interrupts.CPU5.PMI:Performance_monitoring_interrupts
5874 ± 33% -100.0% 0.00 interrupts.CPU50.NMI:Non-maskable_interrupts
5874 ± 33% -100.0% 0.00 interrupts.CPU50.PMI:Performance_monitoring_interrupts
7179 ± 20% -100.0% 0.00 interrupts.CPU51.NMI:Non-maskable_interrupts
7179 ± 20% -100.0% 0.00 interrupts.CPU51.PMI:Performance_monitoring_interrupts
1361 ± 16% -49.9% 682.00 ± 15% interrupts.CPU52.CAL:Function_call_interrupts
6916 ± 20% -100.0% 0.00 interrupts.CPU52.NMI:Non-maskable_interrupts
6916 ± 20% -100.0% 0.00 interrupts.CPU52.PMI:Performance_monitoring_interrupts
5463 ± 43% -85.3% 801.83 ± 59% interrupts.CPU53.CAL:Function_call_interrupts
7545 -100.0% 0.00 interrupts.CPU53.NMI:Non-maskable_interrupts
7545 -100.0% 0.00 interrupts.CPU53.PMI:Performance_monitoring_interrupts
6538 ± 41% -89.4% 694.00 ± 28% interrupts.CPU54.CAL:Function_call_interrupts
7541 -100.0% 0.00 interrupts.CPU54.NMI:Non-maskable_interrupts
7541 -100.0% 0.00 interrupts.CPU54.PMI:Performance_monitoring_interrupts
2701 ± 33% -70.1% 808.67 ± 33% interrupts.CPU55.CAL:Function_call_interrupts
6913 ± 20% -100.0% 0.00 interrupts.CPU55.NMI:Non-maskable_interrupts
6913 ± 20% -100.0% 0.00 interrupts.CPU55.PMI:Performance_monitoring_interrupts
6921 ± 20% -100.0% 0.00 interrupts.CPU56.NMI:Non-maskable_interrupts
6921 ± 20% -100.0% 0.00 interrupts.CPU56.PMI:Performance_monitoring_interrupts
6912 ± 20% -100.0% 0.00 interrupts.CPU57.NMI:Non-maskable_interrupts
6912 ± 20% -100.0% 0.00 interrupts.CPU57.PMI:Performance_monitoring_interrupts
6913 ± 20% -100.0% 0.00 interrupts.CPU58.NMI:Non-maskable_interrupts
6913 ± 20% -100.0% 0.00 interrupts.CPU58.PMI:Performance_monitoring_interrupts
6283 ± 28% -100.0% 0.00 interrupts.CPU59.NMI:Non-maskable_interrupts
6283 ± 28% -100.0% 0.00 interrupts.CPU59.PMI:Performance_monitoring_interrupts
398.83 ± 11% +385.6% 1936 ± 87% interrupts.CPU59.RES:Rescheduling_interrupts
6913 ± 20% -100.0% 0.00 interrupts.CPU6.NMI:Non-maskable_interrupts
6913 ± 20% -100.0% 0.00 interrupts.CPU6.PMI:Performance_monitoring_interrupts
6910 ± 20% -100.0% 0.00 interrupts.CPU60.NMI:Non-maskable_interrupts
6910 ± 20% -100.0% 0.00 interrupts.CPU60.PMI:Performance_monitoring_interrupts
6910 ± 20% -100.0% 0.00 interrupts.CPU61.NMI:Non-maskable_interrupts
6910 ± 20% -100.0% 0.00 interrupts.CPU61.PMI:Performance_monitoring_interrupts
581.67 ± 42% +177.0% 1611 ± 46% interrupts.CPU61.RES:Rescheduling_interrupts
6281 ± 28% -100.0% 0.00 interrupts.CPU62.NMI:Non-maskable_interrupts
6281 ± 28% -100.0% 0.00 interrupts.CPU62.PMI:Performance_monitoring_interrupts
547.83 ± 14% +41.8% 777.00 ± 20% interrupts.CPU63.CAL:Function_call_interrupts
7536 -100.0% 0.00 interrupts.CPU63.NMI:Non-maskable_interrupts
7536 -100.0% 0.00 interrupts.CPU63.PMI:Performance_monitoring_interrupts
347.83 ± 10% +470.8% 1985 ± 46% interrupts.CPU63.RES:Rescheduling_interrupts
7537 -100.0% 0.00 interrupts.CPU64.NMI:Non-maskable_interrupts
7537 -100.0% 0.00 interrupts.CPU64.PMI:Performance_monitoring_interrupts
7537 -100.0% 0.00 interrupts.CPU65.NMI:Non-maskable_interrupts
7537 -100.0% 0.00 interrupts.CPU65.PMI:Performance_monitoring_interrupts
7538 -100.0% 0.00 interrupts.CPU66.NMI:Non-maskable_interrupts
7538 -100.0% 0.00 interrupts.CPU66.PMI:Performance_monitoring_interrupts
7543 -100.0% 0.00 interrupts.CPU67.NMI:Non-maskable_interrupts
7543 -100.0% 0.00 interrupts.CPU67.PMI:Performance_monitoring_interrupts
385.17 ± 28% +410.0% 1964 ± 55% interrupts.CPU67.RES:Rescheduling_interrupts
7536 -100.0% 0.00 interrupts.CPU68.NMI:Non-maskable_interrupts
7536 -100.0% 0.00 interrupts.CPU68.PMI:Performance_monitoring_interrupts
347.00 ± 13% +370.2% 1631 ± 79% interrupts.CPU68.RES:Rescheduling_interrupts
7537 -100.0% 0.00 interrupts.CPU69.NMI:Non-maskable_interrupts
7537 -100.0% 0.00 interrupts.CPU69.PMI:Performance_monitoring_interrupts
376.67 ± 30% +524.3% 2351 ±108% interrupts.CPU69.RES:Rescheduling_interrupts
6911 ± 20% -100.0% 0.00 interrupts.CPU7.NMI:Non-maskable_interrupts
6911 ± 20% -100.0% 0.00 interrupts.CPU7.PMI:Performance_monitoring_interrupts
6276 ± 28% -100.0% 0.00 interrupts.CPU70.NMI:Non-maskable_interrupts
6276 ± 28% -100.0% 0.00 interrupts.CPU70.PMI:Performance_monitoring_interrupts
439.17 ± 52% +374.2% 2082 ± 43% interrupts.CPU70.RES:Rescheduling_interrupts
571.83 +81.9% 1040 ± 59% interrupts.CPU71.CAL:Function_call_interrupts
6276 ± 28% -100.0% 0.00 interrupts.CPU71.NMI:Non-maskable_interrupts
6276 ± 28% -100.0% 0.00 interrupts.CPU71.PMI:Performance_monitoring_interrupts
325.50 ± 2% +525.2% 2035 ± 43% interrupts.CPU71.RES:Rescheduling_interrupts
6276 ± 28% -100.0% 0.00 interrupts.CPU72.NMI:Non-maskable_interrupts
6276 ± 28% -100.0% 0.00 interrupts.CPU72.PMI:Performance_monitoring_interrupts
6276 ± 28% -100.0% 0.00 interrupts.CPU73.NMI:Non-maskable_interrupts
6276 ± 28% -100.0% 0.00 interrupts.CPU73.PMI:Performance_monitoring_interrupts
567.50 +52.8% 867.17 ± 34% interrupts.CPU74.CAL:Function_call_interrupts
6908 ± 20% -100.0% 0.00 interrupts.CPU74.NMI:Non-maskable_interrupts
6908 ± 20% -100.0% 0.00 interrupts.CPU74.PMI:Performance_monitoring_interrupts
321.00 +685.6% 2521 ± 68% interrupts.CPU74.RES:Rescheduling_interrupts
6907 ± 20% -100.0% 0.00 interrupts.CPU75.NMI:Non-maskable_interrupts
6907 ± 20% -100.0% 0.00 interrupts.CPU75.PMI:Performance_monitoring_interrupts
6906 ± 20% -100.0% 1.00 ±223% interrupts.CPU76.NMI:Non-maskable_interrupts
6906 ± 20% -100.0% 1.00 ±223% interrupts.CPU76.PMI:Performance_monitoring_interrupts
323.00 ± 2% +724.5% 2663 ±110% interrupts.CPU76.RES:Rescheduling_interrupts
7534 -100.0% 0.00 interrupts.CPU77.NMI:Non-maskable_interrupts
7534 -100.0% 0.00 interrupts.CPU77.PMI:Performance_monitoring_interrupts
348.50 ± 18% +833.9% 3254 ± 63% interrupts.CPU77.RES:Rescheduling_interrupts
1637 ± 25% -57.0% 703.50 ± 29% interrupts.CPU78.CAL:Function_call_interrupts
7833 -100.0% 0.00 interrupts.CPU78.NMI:Non-maskable_interrupts
7833 -100.0% 0.00 interrupts.CPU78.PMI:Performance_monitoring_interrupts
6155 ± 29% -86.1% 858.50 ± 49% interrupts.CPU79.CAL:Function_call_interrupts
7834 -100.0% 0.00 interrupts.CPU79.NMI:Non-maskable_interrupts
7834 -100.0% 0.00 interrupts.CPU79.PMI:Performance_monitoring_interrupts
6915 ± 20% -100.0% 0.00 interrupts.CPU8.NMI:Non-maskable_interrupts
6915 ± 20% -100.0% 0.00 interrupts.CPU8.PMI:Performance_monitoring_interrupts
6996 ± 48% -88.5% 805.50 ± 43% interrupts.CPU80.CAL:Function_call_interrupts
7835 -100.0% 0.00 interrupts.CPU80.NMI:Non-maskable_interrupts
7835 -100.0% 0.00 interrupts.CPU80.PMI:Performance_monitoring_interrupts
2300 ± 58% -71.8% 649.67 ± 58% interrupts.CPU80.RES:Rescheduling_interrupts
1938 ± 43% -70.1% 579.17 ± 5% interrupts.CPU81.CAL:Function_call_interrupts
7834 -100.0% 0.00 interrupts.CPU81.NMI:Non-maskable_interrupts
7834 -100.0% 0.00 interrupts.CPU81.PMI:Performance_monitoring_interrupts
1473 ± 58% -58.7% 608.17 ± 12% interrupts.CPU82.CAL:Function_call_interrupts
7834 -100.0% 0.00 interrupts.CPU82.NMI:Non-maskable_interrupts
7834 -100.0% 0.00 interrupts.CPU82.PMI:Performance_monitoring_interrupts
801.67 ± 24% -27.1% 584.67 ± 7% interrupts.CPU83.CAL:Function_call_interrupts
7834 -100.0% 0.00 interrupts.CPU83.NMI:Non-maskable_interrupts
7834 -100.0% 0.00 interrupts.CPU83.PMI:Performance_monitoring_interrupts
7833 -100.0% 0.00 interrupts.CPU84.NMI:Non-maskable_interrupts
7833 -100.0% 0.00 interrupts.CPU84.PMI:Performance_monitoring_interrupts
7182 ± 20% -100.0% 0.00 interrupts.CPU85.NMI:Non-maskable_interrupts
7182 ± 20% -100.0% 0.00 interrupts.CPU85.PMI:Performance_monitoring_interrupts
7181 ± 20% -100.0% 0.00 interrupts.CPU86.NMI:Non-maskable_interrupts
7181 ± 20% -100.0% 0.00 interrupts.CPU86.PMI:Performance_monitoring_interrupts
6528 ± 28% -100.0% 0.00 interrupts.CPU87.NMI:Non-maskable_interrupts
6528 ± 28% -100.0% 0.00 interrupts.CPU87.PMI:Performance_monitoring_interrupts
7180 ± 20% -100.0% 0.00 interrupts.CPU88.NMI:Non-maskable_interrupts
7180 ± 20% -100.0% 0.00 interrupts.CPU88.PMI:Performance_monitoring_interrupts
7834 -100.0% 0.00 interrupts.CPU89.NMI:Non-maskable_interrupts
7834 -100.0% 0.00 interrupts.CPU89.PMI:Performance_monitoring_interrupts
7539 -100.0% 0.00 interrupts.CPU9.NMI:Non-maskable_interrupts
7539 -100.0% 0.00 interrupts.CPU9.PMI:Performance_monitoring_interrupts
7835 -100.0% 1.00 ±223% interrupts.CPU90.NMI:Non-maskable_interrupts
7835 -100.0% 1.00 ±223% interrupts.CPU90.PMI:Performance_monitoring_interrupts
7835 -100.0% 0.00 interrupts.CPU91.NMI:Non-maskable_interrupts
7835 -100.0% 0.00 interrupts.CPU91.PMI:Performance_monitoring_interrupts
7836 -100.0% 0.00 interrupts.CPU92.NMI:Non-maskable_interrupts
7836 -100.0% 0.00 interrupts.CPU92.PMI:Performance_monitoring_interrupts
333.17 ± 5% +89.7% 632.00 ± 38% interrupts.CPU92.RES:Rescheduling_interrupts
7834 -100.0% 0.00 interrupts.CPU93.NMI:Non-maskable_interrupts
7834 -100.0% 0.00 interrupts.CPU93.PMI:Performance_monitoring_interrupts
334.67 ± 9% +59.2% 532.83 ± 26% interrupts.CPU93.RES:Rescheduling_interrupts
7834 -100.0% 0.00 interrupts.CPU94.NMI:Non-maskable_interrupts
7834 -100.0% 0.00 interrupts.CPU94.PMI:Performance_monitoring_interrupts
7834 -100.0% 0.00 interrupts.CPU95.NMI:Non-maskable_interrupts
7834 -100.0% 0.00 interrupts.CPU95.PMI:Performance_monitoring_interrupts
7834 -100.0% 0.00 interrupts.CPU96.NMI:Non-maskable_interrupts
7834 -100.0% 0.00 interrupts.CPU96.PMI:Performance_monitoring_interrupts
326.33 ± 2% +205.5% 996.83 ±103% interrupts.CPU96.RES:Rescheduling_interrupts
7833 -100.0% 0.00 interrupts.CPU97.NMI:Non-maskable_interrupts
7833 -100.0% 0.00 interrupts.CPU97.PMI:Performance_monitoring_interrupts
7834 -100.0% 0.00 interrupts.CPU98.NMI:Non-maskable_interrupts
7834 -100.0% 0.00 interrupts.CPU98.PMI:Performance_monitoring_interrupts
7833 -100.0% 0.00 interrupts.CPU99.NMI:Non-maskable_interrupts
7833 -100.0% 0.00 interrupts.CPU99.PMI:Performance_monitoring_interrupts
192.67 ± 4% -100.0% 0.00 interrupts.IWI:IRQ_work_interrupts
729455 ± 4% -100.0% 4.00 ± 70% interrupts.NMI:Non-maskable_interrupts
729455 ± 4% -100.0% 4.00 ± 70% interrupts.PMI:Performance_monitoring_interrupts
51460 ± 7% +86.2% 95806 ± 2% interrupts.RES:Rescheduling_interrupts
314.50 ± 3% -28.1% 226.17 ± 30% interrupts.TLB:TLB_shootdowns***************************************************************************************************
lkp-ivb-2ep1: 48 threads Intel(R) Xeon(R) CPU E5-2697 v2 @ 2.70GHz with 112G memory
=========================================================================================
compiler/cpufreq_governor/kconfig/mode/nr_task/rootfs/tbox_group/test/testcase/ucode:
gcc-9/performance/x86_64-rhel-8.3/thread/100%/debian-10.4-x86_64-20200603.cgz/lkp-ivb-2ep1/poll1/will-it-scale/0x42e
commit:
dde38d5b80 ("kernel: don't include PF_IO_WORKERs as part of same_thread_group()") 43b2a76b1a ("proc: don't show PF_IO_WORKER threads as threads in /proc/<pid>/task/")
dde38d5b805358c7 43b2a76b1a5abcc9833b463bef1
---------------- ---------------------------
%stddev %change %stddev
\ | \
59738282 -2.2% 58422246 will-it-scale.48.threads
1244547 -2.2% 1217129 will-it-scale.per_thread_ops
59738282 -2.2% 58422246 will-it-scale.workload
2.54 +3.5 6.04 ± 2% mpstat.cpu.all.idle%
0.02 ± 14% +0.0 0.04 mpstat.cpu.all.soft%
1379034 +56.6% 2159868 vmstat.memory.cache
1272 ± 2% +50.3% 1912 vmstat.system.cs
328771 ± 6% +75.5% 577027 ± 20% numa-numastat.node0.local_node
364262 ± 5% +65.8% 603798 ± 21% numa-numastat.node0.numa_hit
356834 ± 5% +68.2% 600293 ± 20% numa-numastat.node1.local_node
364821 ± 5% +69.1% 616999 ± 21% numa-numastat.node1.numa_hit
101943 +765.8% 882625 ± 4% meminfo.Active
101943 +765.8% 882625 ± 4% meminfo.Active(anon)
1297433 +60.0% 2075764 meminfo.Cached
815318 +100.9% 1638128 ± 2% meminfo.Committed_AS
2185803 +43.6% 3138095 meminfo.Memused
116458 +668.3% 894794 ± 4% meminfo.Shmem
2409774 +30.2% 3138133 meminfo.max_used_kB
13355 ± 41% +822.5% 123204 ± 15% numa-vmstat.node0.nr_active_anon
161959 ± 3% +67.0% 270513 ± 7% numa-vmstat.node0.nr_file_pages
14956 ± 39% +736.5% 125114 ± 15% numa-vmstat.node0.nr_shmem
13355 ± 41% +822.5% 123204 ± 15% numa-vmstat.node0.nr_zone_active_anon
12128 ± 46% +704.6% 97588 ± 11% numa-vmstat.node1.nr_active_anon
162378 ± 3% +53.1% 248564 ± 5% numa-vmstat.node1.nr_file_pages
14137 ± 42% +598.3% 98721 ± 11% numa-vmstat.node1.nr_shmem
12128 ± 46% +704.6% 97588 ± 11% numa-vmstat.node1.nr_zone_active_anon
5488 ± 3% +33.7% 7337 ± 3% slabinfo.kmalloc-256.active_objs
5529 ± 3% +33.7% 7392 ± 3% slabinfo.kmalloc-256.num_objs
9068 ± 2% -9.6% 8198 slabinfo.proc_inode_cache.active_objs
9068 ± 2% -9.6% 8198 slabinfo.proc_inode_cache.num_objs
17898 +15.2% 20618 slabinfo.radix_tree_node.active_objs
638.83 +15.2% 736.00 slabinfo.radix_tree_node.active_slabs
17898 +15.2% 20618 slabinfo.radix_tree_node.num_objs
638.83 +15.2% 736.00 slabinfo.radix_tree_node.num_slabs
53322 ± 41% +823.0% 492188 ± 15% numa-meminfo.node0.Active
53322 ± 41% +823.0% 492188 ± 15% numa-meminfo.node0.Active(anon)
647688 ± 3% +67.0% 1081422 ± 7% numa-meminfo.node0.FilePages
1128100 ± 3% +42.7% 1609970 ± 5% numa-meminfo.node0.MemUsed
59674 ± 39% +737.6% 499822 ± 15% numa-meminfo.node0.Shmem
48465 ± 46% +704.4% 389842 ± 11% numa-meminfo.node1.Active
48465 ± 46% +704.4% 389842 ± 11% numa-meminfo.node1.Active(anon)
649545 ± 3% +53.0% 993748 ± 5% numa-meminfo.node1.FilePages
1057395 ± 3% +44.5% 1527564 ± 4% numa-meminfo.node1.MemUsed
56584 ± 42% +597.0% 394377 ± 11% numa-meminfo.node1.Shmem
9204 ± 93% +100.9% 18488 ± 64% sched_debug.cfs_rq:/.load.stddev
1403 ± 4% +39.6% 1958 ± 2% sched_debug.cfs_rq:/.runnable_avg.max
104.09 ± 5% +127.0% 236.30 ± 3% sched_debug.cfs_rq:/.runnable_avg.stddev
1112 ± 2% +19.1% 1325 ± 2% sched_debug.cfs_rq:/.util_avg.max
835.58 -28.3% 598.81 ± 5% sched_debug.cfs_rq:/.util_avg.min
61.94 ± 5% +87.5% 116.16 ± 5% sched_debug.cfs_rq:/.util_avg.stddev
1022 +60.9% 1645 ± 12% sched_debug.cfs_rq:/.util_est_enqueued.max
105.35 ± 65% +112.0% 223.36 ± 12% sched_debug.cfs_rq:/.util_est_enqueued.stddev
707908 ± 9% +24.2% 879248 ± 5% sched_debug.cpu.avg_idle.avg
1063850 ± 13% +37.8% 1465737 ± 6% sched_debug.cpu.avg_idle.max
126226 ± 11% +26.1% 159118 ± 15% sched_debug.cpu.avg_idle.stddev
0.00 ± 20% -18.6% 0.00 ± 2% sched_debug.cpu.next_balance.stddev
0.23 ± 5% +28.7% 0.29 ± 4% sched_debug.cpu.nr_running.stddev
6079 ± 2% +32.7% 8069 sched_debug.cpu.nr_switches.avg
20560 ± 5% -25.0% 15421 ± 7% sched_debug.cpu.nr_switches.max
2039 ± 17% +95.6% 3988 ± 13% sched_debug.cpu.nr_switches.min
4387 ± 3% -42.5% 2520 ± 19% sched_debug.cpu.nr_switches.stddev
25464 +766.1% 220557 ± 4% proc-vmstat.nr_active_anon
63374 +2.9% 65240 proc-vmstat.nr_anon_pages
324325 +60.0% 518843 proc-vmstat.nr_file_pages
66898 +1.8% 68101 proc-vmstat.nr_inactive_anon
9338 -5.6% 8819 proc-vmstat.nr_mapped
1084 +9.0% 1182 proc-vmstat.nr_page_table_pages
29081 +668.9% 223601 ± 4% proc-vmstat.nr_shmem
25464 +766.1% 220557 ± 4% proc-vmstat.nr_zone_active_anon
66898 +1.8% 68101 proc-vmstat.nr_zone_inactive_anon
10303 ± 28% -56.9% 4446 ± 52% proc-vmstat.numa_hint_faults_local
760864 +64.3% 1249911 ± 2% proc-vmstat.numa_hit
717323 +68.2% 1206367 ± 2% proc-vmstat.numa_local
145574 ± 13% -42.7% 83394 ± 28% proc-vmstat.numa_pte_updates
37598 -20.5% 29891 ± 4% proc-vmstat.pgactivate
816352 +71.9% 1403351 ± 2% proc-vmstat.pgalloc_normal
893418 -6.9% 831535 proc-vmstat.pgfault
762113 +14.3% 871330 ± 2% proc-vmstat.pgfree
50605 -6.2% 47486 ± 3% proc-vmstat.pgreuse
98.56 -98.6 0.00 perf-profile.calltrace.cycles-pp.__poll
51.59 -51.6 0.00 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__poll
32.22 -32.2 0.00 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_safe_stack.__poll
26.60 -26.6 0.00 perf-profile.calltrace.cycles-pp.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe.__poll
23.39 -23.4 0.00 perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__poll
20.60 -20.6 0.00 perf-profile.calltrace.cycles-pp.__x64_sys_poll.do_syscall_64.entry_SYSCALL_64_after_hwframe.__poll
17.01 -17.0 0.00 perf-profile.calltrace.cycles-pp.do_sys_poll.__x64_sys_poll.do_syscall_64.entry_SYSCALL_64_after_hwframe.__poll
14.54 -14.5 0.00 perf-profile.calltrace.cycles-pp.syscall_return_via_sysret.__poll
13.78 -13.8 0.00 perf-profile.calltrace.cycles-pp.__entry_text_start.__poll
98.80 -98.8 0.00 perf-profile.children.cycles-pp.__poll
51.95 -52.0 0.00 perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
27.71 -27.7 0.00 perf-profile.children.cycles-pp.__entry_text_start
26.72 -26.7 0.00 perf-profile.children.cycles-pp.syscall_exit_to_user_mode
23.59 -23.6 0.00 perf-profile.children.cycles-pp.do_syscall_64
20.71 -20.7 0.00 perf-profile.children.cycles-pp.__x64_sys_poll
17.47 -17.5 0.00 perf-profile.children.cycles-pp.do_sys_poll
16.54 -16.5 0.00 perf-profile.children.cycles-pp.entry_SYSCALL_64_safe_stack
16.32 -16.3 0.00 perf-profile.children.cycles-pp.syscall_return_via_sysret
25.93 -25.9 0.00 perf-profile.self.cycles-pp.__entry_text_start
25.01 -25.0 0.00 perf-profile.self.cycles-pp.syscall_exit_to_user_mode
16.28 -16.3 0.00 perf-profile.self.cycles-pp.syscall_return_via_sysret
6.99 -7.0 0.00 perf-profile.self.cycles-pp.do_sys_poll
40603 ± 14% +119.9% 89296 ± 21% softirqs.CPU0.RCU
39894 ± 14% +72.5% 68812 ± 16% softirqs.CPU1.RCU
35093 ± 15% +80.3% 63288 ± 14% softirqs.CPU10.RCU
35614 ± 15% +90.9% 67979 ± 11% softirqs.CPU11.RCU
35000 ± 12% +79.5% 62812 ± 7% softirqs.CPU12.RCU
31579 ± 11% +98.5% 62671 ± 12% softirqs.CPU13.RCU
32174 ± 10% +114.7% 69079 ± 13% softirqs.CPU14.RCU
31104 ± 11% +109.4% 65136 ± 9% softirqs.CPU15.RCU
36318 ± 14% +92.3% 69837 ± 7% softirqs.CPU16.RCU
32951 ± 18% +105.5% 67709 ± 13% softirqs.CPU17.RCU
36326 ± 14% +90.5% 69212 ± 6% softirqs.CPU18.RCU
34802 ± 13% +89.3% 65899 ± 15% softirqs.CPU19.RCU
38578 ± 20% +90.1% 73339 ± 19% softirqs.CPU2.RCU
35302 ± 14% +87.9% 66341 ± 7% softirqs.CPU20.RCU
35113 ± 16% +93.3% 67861 softirqs.CPU21.RCU
35292 ± 15% +90.3% 67170 ± 3% softirqs.CPU22.RCU
35612 ± 16% +82.5% 64992 ± 5% softirqs.CPU23.RCU
31301 ± 16% +149.5% 78086 ± 21% softirqs.CPU24.RCU
34412 ± 14% +87.9% 64657 ± 20% softirqs.CPU25.RCU
32752 ± 14% +101.7% 66067 ± 15% softirqs.CPU26.RCU
34903 ± 16% +79.1% 62519 ± 15% softirqs.CPU27.RCU
34158 ± 16% +76.9% 60417 ± 9% softirqs.CPU28.RCU
33748 ± 17% +89.0% 63791 ± 20% softirqs.CPU29.RCU
38876 ± 15% +73.5% 67453 ± 9% softirqs.CPU3.RCU
33417 ± 16% +113.8% 71458 ± 17% softirqs.CPU30.RCU
33281 ± 16% +83.8% 61177 ± 15% softirqs.CPU31.RCU
29887 ± 16% +80.3% 53893 ± 12% softirqs.CPU32.RCU
30644 ± 16% +94.5% 59592 ± 6% softirqs.CPU33.RCU
31044 ± 20% +69.1% 52510 ± 18% softirqs.CPU34.RCU
29665 ± 14% +86.4% 55307 ± 12% softirqs.CPU35.RCU
35628 ± 16% +83.5% 65376 ± 12% softirqs.CPU36.RCU
32107 ± 18% +103.8% 65444 ± 10% softirqs.CPU37.RCU
34464 ± 14% +100.6% 69129 ± 7% softirqs.CPU38.RCU
34575 ± 15% +92.1% 66419 ± 13% softirqs.CPU39.RCU
37352 ± 16% +75.6% 65588 ± 9% softirqs.CPU4.RCU
38060 ± 15% +87.5% 71346 ± 4% softirqs.CPU40.RCU
32929 ± 23% +101.2% 66264 ± 17% softirqs.CPU41.RCU
37962 ± 13% +90.0% 72112 ± 5% softirqs.CPU42.RCU
36473 ± 14% +88.1% 68587 ± 9% softirqs.CPU43.RCU
36698 ± 14% +81.1% 66468 ± 3% softirqs.CPU44.RCU
36159 ± 16% +85.3% 67014 ± 3% softirqs.CPU45.RCU
36154 ± 16% +86.9% 67582 ± 5% softirqs.CPU46.RCU
36546 ± 16% +76.5% 64495 ± 4% softirqs.CPU47.RCU
36519 ± 15% +91.3% 69871 ± 22% softirqs.CPU5.RCU
36807 ± 17% +92.5% 70871 ± 13% softirqs.CPU6.RCU
35242 ± 16% +83.1% 64543 ± 13% softirqs.CPU7.RCU
36761 ± 16% +72.9% 63562 ± 7% softirqs.CPU8.RCU
37277 ± 11% +92.7% 71838 ± 9% softirqs.CPU9.RCU
1677186 ± 14% +90.5% 3194899 softirqs.RCU
33514 ± 3% +52.1% 50986 softirqs.TIMER
0.09 ± 10% +204.1% 0.26 ± 6% perf-stat.i.MPKI
8.85e+09 +3.4% 9.151e+09 perf-stat.i.branch-instructions
63173486 +4.0% 65686220 perf-stat.i.branch-misses
7.05 +2.4 9.48 ± 9% perf-stat.i.cache-miss-rate%
246018 ± 2% +332.2% 1063300 ± 7% perf-stat.i.cache-misses
2755412 ± 3% +299.4% 11004931 ± 6% perf-stat.i.cache-references
1244 ± 2% +51.6% 1886 perf-stat.i.context-switches
3.31 -3.3% 3.20 perf-stat.i.cpi
111.57 ± 6% +37.7% 153.66 ± 2% perf-stat.i.cpu-migrations
955562 ± 3% -85.0% 143699 ± 8% perf-stat.i.cycles-between-cache-misses
1.17 -0.0 1.13 perf-stat.i.dTLB-load-miss-rate%
1.108e+10 +2.5% 1.136e+10 perf-stat.i.dTLB-loads
1.14 -0.0 1.11 perf-stat.i.dTLB-store-miss-rate%
1.323e+08 -1.9% 1.298e+08 perf-stat.i.dTLB-store-misses
4.299e+10 +3.4% 4.445e+10 perf-stat.i.instructions
0.30 +3.5% 0.31 perf-stat.i.ipc
0.78 ± 25% -64.3% 0.28 ± 47% perf-stat.i.major-faults
1.20 ± 7% -83.2% 0.20 ± 52% perf-stat.i.metric.K/sec
655.44 +2.2% 669.80 perf-stat.i.metric.M/sec
2858 -7.1% 2654 perf-stat.i.minor-faults
46.94 +1.3 48.26 perf-stat.i.node-load-miss-rate%
93609 ± 2% +365.9% 436129 ± 11% perf-stat.i.node-load-misses
116813 ± 2% +301.2% 468642 ± 11% perf-stat.i.node-loads
33.38 ± 6% +9.2 42.59 perf-stat.i.node-store-miss-rate%
50561 ± 5% +725.2% 417237 ± 7% perf-stat.i.node-store-misses
116603 ± 11% +383.2% 563385 ± 6% perf-stat.i.node-stores
2859 -7.2% 2654 perf-stat.i.page-faults
0.06 ± 3% +285.5% 0.25 ± 6% perf-stat.overall.MPKI
3.31 -3.3% 3.21 perf-stat.overall.cpi
576612 ± 2% -76.6% 134705 ± 7% perf-stat.overall.cycles-between-cache-misses
1.17 -0.0 1.14 perf-stat.overall.dTLB-load-miss-rate%
1.15 -0.0 1.12 perf-stat.overall.dTLB-store-miss-rate%
0.30 +3.4% 0.31 perf-stat.overall.ipc
44.41 +3.8 48.20 perf-stat.overall.node-load-miss-rate%
30.38 ± 8% +12.1 42.53 perf-stat.overall.node-store-miss-rate%
216892 +5.7% 229166 perf-stat.overall.path-length
8.821e+09 +3.4% 9.12e+09 perf-stat.ps.branch-instructions
62993650 +3.9% 65475101 perf-stat.ps.branch-misses
246469 ± 2% +330.2% 1060339 ± 7% perf-stat.ps.cache-misses
2753111 ± 3% +298.5% 10970294 ± 6% perf-stat.ps.cache-references
1240 ± 2% +51.5% 1880 perf-stat.ps.context-switches
111.34 ± 6% +37.6% 153.15 ± 2% perf-stat.ps.cpu-migrations
1.105e+10 +2.5% 1.132e+10 perf-stat.ps.dTLB-loads
1.319e+08 -1.9% 1.293e+08 perf-stat.ps.dTLB-store-misses
4.285e+10 +3.4% 4.43e+10 perf-stat.ps.instructions
0.78 ± 25% -64.3% 0.28 ± 47% perf-stat.ps.major-faults
2851 -7.2% 2646 perf-stat.ps.minor-faults
93481 ± 2% +365.0% 434677 ± 11% perf-stat.ps.node-load-misses
117037 ± 2% +299.1% 467127 ± 11% perf-stat.ps.node-loads
50493 ± 5% +723.7% 415907 ± 7% perf-stat.ps.node-store-misses
116604 ± 11% +381.7% 561730 ± 6% perf-stat.ps.node-stores
2852 -7.2% 2646 perf-stat.ps.page-faults
1.296e+13 +3.3% 1.339e+13 perf-stat.total.instructions
128311 -24.4% 96996 ± 2% interrupts.CAL:Function_call_interrupts
7980 ± 7% -100.0% 0.00 interrupts.CPU0.NMI:Non-maskable_interrupts
7980 ± 7% -100.0% 0.00 interrupts.CPU0.PMI:Performance_monitoring_interrupts
5151 ± 26% -59.6% 2080 ± 11% interrupts.CPU1.CAL:Function_call_interrupts
5489 ± 27% -100.0% 0.00 interrupts.CPU1.NMI:Non-maskable_interrupts
5489 ± 27% -100.0% 0.00 interrupts.CPU1.PMI:Performance_monitoring_interrupts
5906 ± 31% -100.0% 0.00 interrupts.CPU10.NMI:Non-maskable_interrupts
5906 ± 31% -100.0% 0.00 interrupts.CPU10.PMI:Performance_monitoring_interrupts
6677 ± 27% -100.0% 0.00 interrupts.CPU11.NMI:Non-maskable_interrupts
6677 ± 27% -100.0% 0.00 interrupts.CPU11.PMI:Performance_monitoring_interrupts
372.50 ± 11% +139.5% 892.17 ± 47% interrupts.CPU11.RES:Rescheduling_interrupts
3246 ± 6% -35.5% 2093 ± 12% interrupts.CPU12.CAL:Function_call_interrupts
5948 ± 25% -100.0% 0.00 interrupts.CPU12.NMI:Non-maskable_interrupts
5948 ± 25% -100.0% 0.00 interrupts.CPU12.PMI:Performance_monitoring_interrupts
5726 ± 26% -64.9% 2011 ± 5% interrupts.CPU13.CAL:Function_call_interrupts
7394 ± 20% -100.0% 0.00 interrupts.CPU13.NMI:Non-maskable_interrupts
7394 ± 20% -100.0% 0.00 interrupts.CPU13.PMI:Performance_monitoring_interrupts
7596 ± 15% -75.0% 1901 ± 3% interrupts.CPU14.CAL:Function_call_interrupts
5949 ± 31% -100.0% 0.00 interrupts.CPU14.NMI:Non-maskable_interrupts
5949 ± 31% -100.0% 0.00 interrupts.CPU14.PMI:Performance_monitoring_interrupts
1372 ± 41% -57.9% 578.33 ± 22% interrupts.CPU14.RES:Rescheduling_interrupts
3586 ± 32% -46.9% 1903 ± 6% interrupts.CPU15.CAL:Function_call_interrupts
6700 ± 27% -100.0% 0.00 interrupts.CPU15.NMI:Non-maskable_interrupts
6700 ± 27% -100.0% 0.00 interrupts.CPU15.PMI:Performance_monitoring_interrupts
5647 ± 27% -100.0% 0.00 interrupts.CPU16.NMI:Non-maskable_interrupts
5647 ± 27% -100.0% 0.00 interrupts.CPU16.PMI:Performance_monitoring_interrupts
6007 ± 31% -100.0% 0.00 interrupts.CPU17.NMI:Non-maskable_interrupts
6007 ± 31% -100.0% 0.00 interrupts.CPU17.PMI:Performance_monitoring_interrupts
6206 ± 33% -100.0% 0.00 interrupts.CPU18.NMI:Non-maskable_interrupts
6206 ± 33% -100.0% 0.00 interrupts.CPU18.PMI:Performance_monitoring_interrupts
413.67 ± 17% +96.3% 812.17 ± 24% interrupts.CPU18.RES:Rescheduling_interrupts
5518 ± 35% -100.0% 0.00 interrupts.CPU19.NMI:Non-maskable_interrupts
5518 ± 35% -100.0% 0.00 interrupts.CPU19.PMI:Performance_monitoring_interrupts
399.33 ± 24% +92.7% 769.33 ± 37% interrupts.CPU19.RES:Rescheduling_interrupts
6808 ± 13% -70.0% 2044 ± 11% interrupts.CPU2.CAL:Function_call_interrupts
5454 ± 28% -100.0% 0.00 interrupts.CPU2.NMI:Non-maskable_interrupts
5454 ± 28% -100.0% 0.00 interrupts.CPU2.PMI:Performance_monitoring_interrupts
1100 ± 4% -9.1% 1000 ± 3% interrupts.CPU2.TLB:TLB_shootdowns
7581 ± 20% -100.0% 0.00 interrupts.CPU20.NMI:Non-maskable_interrupts
7581 ± 20% -100.0% 0.00 interrupts.CPU20.PMI:Performance_monitoring_interrupts
7109 ± 21% -100.0% 0.00 interrupts.CPU21.NMI:Non-maskable_interrupts
7109 ± 21% -100.0% 0.00 interrupts.CPU21.PMI:Performance_monitoring_interrupts
7036 ± 21% -100.0% 0.00 interrupts.CPU22.NMI:Non-maskable_interrupts
7036 ± 21% -100.0% 0.00 interrupts.CPU22.PMI:Performance_monitoring_interrupts
414.33 ± 27% +135.4% 975.50 ± 38% interrupts.CPU22.RES:Rescheduling_interrupts
7434 ± 20% -100.0% 0.00 interrupts.CPU23.NMI:Non-maskable_interrupts
7434 ± 20% -100.0% 0.00 interrupts.CPU23.PMI:Performance_monitoring_interrupts
333.00 ± 4% +132.5% 774.33 ± 59% interrupts.CPU23.RES:Rescheduling_interrupts
1966 ± 4% -9.9% 1770 ± 5% interrupts.CPU24.CAL:Function_call_interrupts
6889 ± 28% -100.0% 0.00 interrupts.CPU24.NMI:Non-maskable_interrupts
6889 ± 28% -100.0% 0.00 interrupts.CPU24.PMI:Performance_monitoring_interrupts
1082 ± 4% -15.5% 914.83 ± 6% interrupts.CPU24.TLB:TLB_shootdowns
2259 ± 9% -13.3% 1957 ± 6% interrupts.CPU25.CAL:Function_call_interrupts
7215 ± 22% -100.0% 0.00 interrupts.CPU25.NMI:Non-maskable_interrupts
7215 ± 22% -100.0% 0.00 interrupts.CPU25.PMI:Performance_monitoring_interrupts
2705 ± 14% -27.5% 1960 ± 9% interrupts.CPU26.CAL:Function_call_interrupts
7554 ± 18% -100.0% 1.17 ±223% interrupts.CPU26.NMI:Non-maskable_interrupts
7554 ± 18% -100.0% 1.17 ±223% interrupts.CPU26.PMI:Performance_monitoring_interrupts
6887 ± 28% -100.0% 0.00 interrupts.CPU27.NMI:Non-maskable_interrupts
6887 ± 28% -100.0% 0.00 interrupts.CPU27.PMI:Performance_monitoring_interrupts
7579 ± 20% -100.0% 0.00 interrupts.CPU28.NMI:Non-maskable_interrupts
7579 ± 20% -100.0% 0.00 interrupts.CPU28.PMI:Performance_monitoring_interrupts
6761 ± 27% -100.0% 0.00 interrupts.CPU29.NMI:Non-maskable_interrupts
6761 ± 27% -100.0% 0.00 interrupts.CPU29.PMI:Performance_monitoring_interrupts
5819 ± 17% -66.0% 1978 ± 8% interrupts.CPU3.CAL:Function_call_interrupts
6654 ± 27% -100.0% 0.00 interrupts.CPU3.NMI:Non-maskable_interrupts
6654 ± 27% -100.0% 0.00 interrupts.CPU3.PMI:Performance_monitoring_interrupts
1105 ± 3% -10.9% 985.50 ± 6% interrupts.CPU3.TLB:TLB_shootdowns
7517 ± 19% -100.0% 0.00 interrupts.CPU30.NMI:Non-maskable_interrupts
7517 ± 19% -100.0% 0.00 interrupts.CPU30.PMI:Performance_monitoring_interrupts
5579 ± 34% -100.0% 0.00 interrupts.CPU31.NMI:Non-maskable_interrupts
5579 ± 34% -100.0% 0.00 interrupts.CPU31.PMI:Performance_monitoring_interrupts
378.50 ± 20% +448.7% 2077 ± 62% interrupts.CPU31.RES:Rescheduling_interrupts
5652 ± 33% -100.0% 0.00 interrupts.CPU32.NMI:Non-maskable_interrupts
5652 ± 33% -100.0% 0.00 interrupts.CPU32.PMI:Performance_monitoring_interrupts
450.50 ± 54% +251.9% 1585 ± 86% interrupts.CPU32.RES:Rescheduling_interrupts
1092 ± 4% -8.6% 998.33 ± 4% interrupts.CPU32.TLB:TLB_shootdowns
6900 ± 28% -100.0% 0.00 interrupts.CPU33.NMI:Non-maskable_interrupts
6900 ± 28% -100.0% 0.00 interrupts.CPU33.PMI:Performance_monitoring_interrupts
1097 -10.9% 978.50 ± 7% interrupts.CPU33.TLB:TLB_shootdowns
6518 ± 28% -100.0% 0.00 interrupts.CPU34.NMI:Non-maskable_interrupts
6518 ± 28% -100.0% 0.00 interrupts.CPU34.PMI:Performance_monitoring_interrupts
366.33 ± 14% +303.8% 1479 ± 35% interrupts.CPU34.RES:Rescheduling_interrupts
6948 ± 27% -100.0% 0.00 interrupts.CPU35.NMI:Non-maskable_interrupts
6948 ± 27% -100.0% 0.00 interrupts.CPU35.PMI:Performance_monitoring_interrupts
496.17 ± 65% +218.7% 1581 ± 68% interrupts.CPU35.RES:Rescheduling_interrupts
7540 ± 20% -100.0% 0.00 interrupts.CPU36.NMI:Non-maskable_interrupts
7540 ± 20% -100.0% 0.00 interrupts.CPU36.PMI:Performance_monitoring_interrupts
401.33 ± 14% +243.2% 1377 ± 71% interrupts.CPU36.RES:Rescheduling_interrupts
1107 ± 2% -11.6% 979.17 ± 5% interrupts.CPU36.TLB:TLB_shootdowns
3479 ± 30% -44.0% 1947 ± 12% interrupts.CPU37.CAL:Function_call_interrupts
6836 ± 27% -100.0% 0.00 interrupts.CPU37.NMI:Non-maskable_interrupts
6836 ± 27% -100.0% 0.00 interrupts.CPU37.PMI:Performance_monitoring_interrupts
1093 ± 4% -8.4% 1001 ± 4% interrupts.CPU37.TLB:TLB_shootdowns
6880 ± 27% -100.0% 0.00 interrupts.CPU38.NMI:Non-maskable_interrupts
6880 ± 27% -100.0% 0.00 interrupts.CPU38.PMI:Performance_monitoring_interrupts
407.17 ± 14% +130.5% 938.67 ± 53% interrupts.CPU38.RES:Rescheduling_interrupts
6901 ± 28% -100.0% 0.00 interrupts.CPU39.NMI:Non-maskable_interrupts
6901 ± 28% -100.0% 0.00 interrupts.CPU39.PMI:Performance_monitoring_interrupts
3726 ± 21% -48.3% 1928 ± 3% interrupts.CPU4.CAL:Function_call_interrupts
6048 ± 32% -100.0% 0.00 interrupts.CPU4.NMI:Non-maskable_interrupts
6048 ± 32% -100.0% 0.00 interrupts.CPU4.PMI:Performance_monitoring_interrupts
6901 ± 28% -100.0% 0.00 interrupts.CPU40.NMI:Non-maskable_interrupts
6901 ± 28% -100.0% 0.00 interrupts.CPU40.PMI:Performance_monitoring_interrupts
6895 ± 27% -100.0% 0.00 interrupts.CPU41.NMI:Non-maskable_interrupts
6895 ± 27% -100.0% 0.00 interrupts.CPU41.PMI:Performance_monitoring_interrupts
7580 ± 20% -100.0% 0.00 interrupts.CPU42.NMI:Non-maskable_interrupts
7580 ± 20% -100.0% 0.00 interrupts.CPU42.PMI:Performance_monitoring_interrupts
349.83 ± 8% +188.9% 1010 ± 40% interrupts.CPU42.RES:Rescheduling_interrupts
7660 ± 17% -100.0% 0.00 interrupts.CPU43.NMI:Non-maskable_interrupts
7660 ± 17% -100.0% 0.00 interrupts.CPU43.PMI:Performance_monitoring_interrupts
6208 ± 33% -100.0% 0.00 interrupts.CPU44.NMI:Non-maskable_interrupts
6208 ± 33% -100.0% 0.00 interrupts.CPU44.PMI:Performance_monitoring_interrupts
395.50 ± 15% +223.7% 1280 ± 57% interrupts.CPU44.RES:Rescheduling_interrupts
1104 ± 3% -8.9% 1005 ± 4% interrupts.CPU44.TLB:TLB_shootdowns
5515 ± 35% -100.0% 0.00 interrupts.CPU45.NMI:Non-maskable_interrupts
5515 ± 35% -100.0% 0.00 interrupts.CPU45.PMI:Performance_monitoring_interrupts
374.17 ± 10% +227.8% 1226 ± 72% interrupts.CPU45.RES:Rescheduling_interrupts
7557 ± 20% -100.0% 0.00 interrupts.CPU46.NMI:Non-maskable_interrupts
7557 ± 20% -100.0% 0.00 interrupts.CPU46.PMI:Performance_monitoring_interrupts
346.33 ± 7% +251.3% 1216 ± 71% interrupts.CPU46.RES:Rescheduling_interrupts
6123 ± 32% -100.0% 0.00 interrupts.CPU47.NMI:Non-maskable_interrupts
6123 ± 32% -100.0% 0.00 interrupts.CPU47.PMI:Performance_monitoring_interrupts
360.67 ± 5% +183.0% 1020 ± 79% interrupts.CPU47.RES:Rescheduling_interrupts
4732 ± 28% -100.0% 0.00 interrupts.CPU5.NMI:Non-maskable_interrupts
4732 ± 28% -100.0% 0.00 interrupts.CPU5.PMI:Performance_monitoring_interrupts
5158 ± 30% -100.0% 0.00 interrupts.CPU6.NMI:Non-maskable_interrupts
5158 ± 30% -100.0% 0.00 interrupts.CPU6.PMI:Performance_monitoring_interrupts
4821 ± 31% -100.0% 0.00 interrupts.CPU7.NMI:Non-maskable_interrupts
4821 ± 31% -100.0% 0.00 interrupts.CPU7.PMI:Performance_monitoring_interrupts
7091 ± 20% -100.0% 0.00 interrupts.CPU8.NMI:Non-maskable_interrupts
7091 ± 20% -100.0% 0.00 interrupts.CPU8.PMI:Performance_monitoring_interrupts
5671 ± 28% -100.0% 0.00 interrupts.CPU9.NMI:Non-maskable_interrupts
5671 ± 28% -100.0% 0.00 interrupts.CPU9.PMI:Performance_monitoring_interrupts
383.50 ± 9% +133.6% 895.67 ± 51% interrupts.CPU9.RES:Rescheduling_interrupts
314820 ± 7% -100.0% 1.17 ±223% interrupts.NMI:Non-maskable_interrupts
314820 ± 7% -100.0% 1.17 ±223% interrupts.PMI:Performance_monitoring_interrupts
27370 ± 6% +104.9% 56082 ± 2% interrupts.RES:Rescheduling_interrupts
0.02 ± 4% -100.0% 0.00 perf-sched.sch_delay.avg.ms.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown]
2.03 ±101% -100.0% 0.00 perf-sched.sch_delay.avg.ms.devkmsg_read.vfs_read.ksys_read.do_syscall_64
0.02 ± 2% -100.0% 0.00 perf-sched.sch_delay.avg.ms.do_nanosleep.hrtimer_nanosleep.__x64_sys_nanosleep.do_syscall_64
3.23 ±120% -100.0% 0.00 perf-sched.sch_delay.avg.ms.do_syslog.part.0.kmsg_read.vfs_read
2.68 ± 16% -100.0% 0.00 perf-sched.sch_delay.avg.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
0.02 ± 2% -100.0% 0.00 perf-sched.sch_delay.avg.ms.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64
0.19 ± 14% -100.0% 0.00 perf-sched.sch_delay.avg.ms.exit_to_user_mode_prepare.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe.[unknown]
0.03 ± 22% -100.0% 0.00 perf-sched.sch_delay.avg.ms.futex_wait_queue_me.futex_wait.do_futex.__x64_sys_futex
0.02 ± 10% -100.0% 0.00 perf-sched.sch_delay.avg.ms.pipe_read.new_sync_read.vfs_read.ksys_read
10.55 ± 11% -100.0% 0.00 perf-sched.sch_delay.avg.ms.preempt_schedule_common.__cond_resched.stop_one_cpu.sched_exec.bprm_execve
0.08 ±155% -100.0% 0.00 perf-sched.sch_delay.avg.ms.rcu_gp_kthread.kthread.ret_from_fork
0.05 ± 28% -100.0% 0.00 perf-sched.sch_delay.avg.ms.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait.__x64_sys_epoll_wait
0.02 ± 11% -100.0% 0.00 perf-sched.sch_delay.avg.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_select
0.00 ± 14% -100.0% 0.00 perf-sched.sch_delay.avg.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_sys_poll
0.02 ± 2% -100.0% 0.00 perf-sched.sch_delay.avg.ms.schedule_timeout.kcompactd.kthread.ret_from_fork
0.02 ± 4% -100.0% 0.00 perf-sched.sch_delay.avg.ms.schedule_timeout.rcu_gp_kthread.kthread.ret_from_fork
0.03 ±126% -100.0% 0.00 perf-sched.sch_delay.avg.ms.smpboot_thread_fn.kthread.ret_from_fork
8.47 ± 7% -100.0% 0.00 perf-sched.sch_delay.avg.ms.wait_for_partner.fifo_open.do_dentry_open.do_open.isra
0.06 ± 43% -100.0% 0.00 perf-sched.sch_delay.avg.ms.worker_thread.kthread.ret_from_fork
0.02 ± 9% -100.0% 0.00 perf-sched.sch_delay.max.ms.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown]
10.11 ±101% -100.0% 0.00 perf-sched.sch_delay.max.ms.devkmsg_read.vfs_read.ksys_read.do_syscall_64
0.03 ± 18% -100.0% 0.00 perf-sched.sch_delay.max.ms.do_nanosleep.hrtimer_nanosleep.__x64_sys_nanosleep.do_syscall_64
11.64 ± 94% -100.0% 0.00 perf-sched.sch_delay.max.ms.do_syslog.part.0.kmsg_read.vfs_read
19.20 ± 8% -100.0% 0.00 perf-sched.sch_delay.max.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
0.05 ± 49% -100.0% 0.00 perf-sched.sch_delay.max.ms.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64
18.89 ± 11% -100.0% 0.00 perf-sched.sch_delay.max.ms.exit_to_user_mode_prepare.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe.[unknown]
0.04 ± 28% -100.0% 0.00 perf-sched.sch_delay.max.ms.futex_wait_queue_me.futex_wait.do_futex.__x64_sys_futex
0.33 ± 28% -100.0% 0.00 perf-sched.sch_delay.max.ms.pipe_read.new_sync_read.vfs_read.ksys_read
0.03 ± 5% -100.0% 0.00 perf-sched.sch_delay.max.ms.preempt_schedule_common.__cond_resched.stop_one_cpu.affine_move_task.__set_cpus_allowed_ptr
35.38 ±105% -100.0% 0.00 perf-sched.sch_delay.max.ms.preempt_schedule_common.__cond_resched.stop_one_cpu.sched_exec.bprm_execve
5.07 ±199% -100.0% 0.00 perf-sched.sch_delay.max.ms.rcu_gp_kthread.kthread.ret_from_fork
0.42 ± 4% -100.0% 0.00 perf-sched.sch_delay.max.ms.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait.__x64_sys_epoll_wait
0.02 ± 14% -100.0% 0.00 perf-sched.sch_delay.max.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_select
0.02 ± 25% -100.0% 0.00 perf-sched.sch_delay.max.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_sys_poll
0.02 ± 11% -100.0% 0.00 perf-sched.sch_delay.max.ms.schedule_timeout.kcompactd.kthread.ret_from_fork
0.21 ± 84% -100.0% 0.00 perf-sched.sch_delay.max.ms.schedule_timeout.rcu_gp_kthread.kthread.ret_from_fork
12.35 ±217% -100.0% 0.00 perf-sched.sch_delay.max.ms.smpboot_thread_fn.kthread.ret_from_fork
20.33 ± 4% -100.0% 0.00 perf-sched.sch_delay.max.ms.wait_for_partner.fifo_open.do_dentry_open.do_open.isra
15.52 ± 54% -100.0% 0.00 perf-sched.sch_delay.max.ms.worker_thread.kthread.ret_from_fork
0.21 ± 8% -100.0% 0.00 perf-sched.total_sch_delay.average.ms
48.71 ± 73% -100.0% 0.00 perf-sched.total_sch_delay.max.ms
98.46 ± 9% -100.0% 0.00 perf-sched.total_wait_and_delay.average.ms
10953 ± 10% -100.0% 0.00 perf-sched.total_wait_and_delay.count.ms
7483 ± 5% -100.0% 0.00 perf-sched.total_wait_and_delay.max.ms
98.25 ± 9% -100.0% 0.00 perf-sched.total_wait_time.average.ms
7483 ± 5% -100.0% 0.00 perf-sched.total_wait_time.max.ms
882.95 ± 4% -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown]
487.24 ± 60% -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.devkmsg_read.vfs_read.ksys_read.do_syscall_64
821.03 ± 4% -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.do_nanosleep.hrtimer_nanosleep.__x64_sys_nanosleep.do_syscall_64
506.66 ± 77% -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.do_syslog.part.0.kmsg_read.vfs_read
268.27 ± 2% -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
99.97 ± 13% -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64
0.37 ± 71% -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt.[unknown]
0.22 ± 21% -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_call_function_single.[unknown]
0.18 ± 57% -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_reschedule_ipi.[unknown]
0.41 ± 11% -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.exit_to_user_mode_prepare.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe.[unknown]
83.78 ± 18% -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.pipe_read.new_sync_read.vfs_read.ksys_read
18.86 -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.preempt_schedule_common.__cond_resched.stop_one_cpu.affine_move_task.__set_cpus_allowed_ptr
284.95 ± 40% -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait.__x64_sys_epoll_wait
239.54 ± 9% -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_sys_poll
478.81 -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.schedule_timeout.kcompactd.kthread.ret_from_fork
6.28 ± 15% -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.schedule_timeout.rcu_gp_kthread.kthread.ret_from_fork
653.01 -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.smpboot_thread_fn.kthread.ret_from_fork
8.87 ± 5% -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.wait_for_partner.fifo_open.do_dentry_open.do_open.isra
565.29 ± 5% -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.worker_thread.kthread.ret_from_fork
10.00 -100.0% 0.00 perf-sched.wait_and_delay.count.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown]
5.50 ± 9% -100.0% 0.00 perf-sched.wait_and_delay.count.devkmsg_read.vfs_read.ksys_read.do_syscall_64
22.00 ± 4% -100.0% 0.00 perf-sched.wait_and_delay.count.do_nanosleep.hrtimer_nanosleep.__x64_sys_nanosleep.do_syscall_64
5.67 ± 8% -100.0% 0.00 perf-sched.wait_and_delay.count.do_syslog.part.0.kmsg_read.vfs_read
228.00 -100.0% 0.00 perf-sched.wait_and_delay.count.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
264.00 ± 2% -100.0% 0.00 perf-sched.wait_and_delay.count.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64
1517 ± 13% -100.0% 0.00 perf-sched.wait_and_delay.count.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt.[unknown]
178.17 ± 10% -100.0% 0.00 perf-sched.wait_and_delay.count.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_call_function_single.[unknown]
104.50 ± 12% -100.0% 0.00 perf-sched.wait_and_delay.count.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_reschedule_ipi.[unknown]
3237 ± 14% -100.0% 0.00 perf-sched.wait_and_delay.count.exit_to_user_mode_prepare.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe.[unknown]
1581 ± 22% -100.0% 0.00 perf-sched.wait_and_delay.count.pipe_read.new_sync_read.vfs_read.ksys_read
576.00 -100.0% 0.00 perf-sched.wait_and_delay.count.preempt_schedule_common.__cond_resched.stop_one_cpu.affine_move_task.__set_cpus_allowed_ptr
23.33 ± 46% -100.0% 0.00 perf-sched.wait_and_delay.count.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait.__x64_sys_epoll_wait
105.50 ± 7% -100.0% 0.00 perf-sched.wait_and_delay.count.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_sys_poll
40.00 -100.0% 0.00 perf-sched.wait_and_delay.count.schedule_timeout.kcompactd.kthread.ret_from_fork
1598 ± 21% -100.0% 0.00 perf-sched.wait_and_delay.count.schedule_timeout.rcu_gp_kthread.kthread.ret_from_fork
749.00 -100.0% 0.00 perf-sched.wait_and_delay.count.smpboot_thread_fn.kthread.ret_from_fork
66.17 -100.0% 0.00 perf-sched.wait_and_delay.count.wait_for_partner.fifo_open.do_dentry_open.do_open.isra
436.17 ± 2% -100.0% 0.00 perf-sched.wait_and_delay.count.worker_thread.kthread.ret_from_fork
999.61 -100.0% 0.00 perf-sched.wait_and_delay.max.ms.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown]
2337 ± 50% -100.0% 0.00 perf-sched.wait_and_delay.max.ms.devkmsg_read.vfs_read.ksys_read.do_syscall_64
1000 -100.0% 0.00 perf-sched.wait_and_delay.max.ms.do_nanosleep.hrtimer_nanosleep.__x64_sys_nanosleep.do_syscall_64
2368 ± 49% -100.0% 0.00 perf-sched.wait_and_delay.max.ms.do_syslog.part.0.kmsg_read.vfs_read
1989 ± 84% -100.0% 0.00 perf-sched.wait_and_delay.max.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
1022 -100.0% 0.00 perf-sched.wait_and_delay.max.ms.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64
193.25 ±186% -100.0% 0.00 perf-sched.wait_and_delay.max.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt.[unknown]
9.91 ± 74% -100.0% 0.00 perf-sched.wait_and_delay.max.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_call_function_single.[unknown]
9.67 ±113% -100.0% 0.00 perf-sched.wait_and_delay.max.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_reschedule_ipi.[unknown]
28.27 ± 19% -100.0% 0.00 perf-sched.wait_and_delay.max.ms.exit_to_user_mode_prepare.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe.[unknown]
2338 ± 50% -100.0% 0.00 perf-sched.wait_and_delay.max.ms.pipe_read.new_sync_read.vfs_read.ksys_read
1849 -100.0% 0.00 perf-sched.wait_and_delay.max.ms.preempt_schedule_common.__cond_resched.stop_one_cpu.affine_move_task.__set_cpus_allowed_ptr
1999 ± 57% -100.0% 0.00 perf-sched.wait_and_delay.max.ms.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait.__x64_sys_epoll_wait
5216 ± 36% -100.0% 0.00 perf-sched.wait_and_delay.max.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_sys_poll
505.01 -100.0% 0.00 perf-sched.wait_and_delay.max.ms.schedule_timeout.kcompactd.kthread.ret_from_fork
254.85 ± 29% -100.0% 0.00 perf-sched.wait_and_delay.max.ms.schedule_timeout.rcu_gp_kthread.kthread.ret_from_fork
5585 ± 20% -100.0% 0.00 perf-sched.wait_and_delay.max.ms.smpboot_thread_fn.kthread.ret_from_fork
20.33 ± 4% -100.0% 0.00 perf-sched.wait_and_delay.max.ms.wait_for_partner.fifo_open.do_dentry_open.do_open.isra
7413 ± 6% -100.0% 0.00 perf-sched.wait_and_delay.max.ms.worker_thread.kthread.ret_from_fork
882.94 ± 4% -100.0% 0.00 perf-sched.wait_time.avg.ms.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown]
485.21 ± 60% -100.0% 0.00 perf-sched.wait_time.avg.ms.devkmsg_read.vfs_read.ksys_read.do_syscall_64
821.01 ± 4% -100.0% 0.00 perf-sched.wait_time.avg.ms.do_nanosleep.hrtimer_nanosleep.__x64_sys_nanosleep.do_syscall_64
503.43 ± 77% -100.0% 0.00 perf-sched.wait_time.avg.ms.do_syslog.part.0.kmsg_read.vfs_read
265.59 ± 2% -100.0% 0.00 perf-sched.wait_time.avg.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
99.95 ± 13% -100.0% 0.00 perf-sched.wait_time.avg.ms.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64
0.36 ± 71% -100.0% 0.00 perf-sched.wait_time.avg.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt.[unknown]
0.21 ± 19% -100.0% 0.00 perf-sched.wait_time.avg.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_call_function_single.[unknown]
1.02 ±216% -100.0% 0.00 perf-sched.wait_time.avg.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_irq_work.[unknown]
0.17 ± 62% -100.0% 0.00 perf-sched.wait_time.avg.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_reschedule_ipi.[unknown]
0.22 ± 9% -100.0% 0.00 perf-sched.wait_time.avg.ms.exit_to_user_mode_prepare.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe.[unknown]
300.42 ±204% -100.0% 0.00 perf-sched.wait_time.avg.ms.futex_wait_queue_me.futex_wait.do_futex.__x64_sys_futex
83.76 ± 18% -100.0% 0.00 perf-sched.wait_time.avg.ms.pipe_read.new_sync_read.vfs_read.ksys_read
10.87 ± 11% -100.0% 0.00 perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.dput.__fput.task_work_run
741.75 ±104% -100.0% 0.00 perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.generic_perform_write.__generic_file_write_iter.generic_file_write_iter
3.83 ± 19% -100.0% 0.00 perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.khugepaged.kthread.ret_from_fork
18.86 -100.0% 0.00 perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.stop_one_cpu.affine_move_task.__set_cpus_allowed_ptr
2.77 ± 13% -100.0% 0.00 perf-sched.wait_time.avg.ms.rcu_gp_kthread.kthread.ret_from_fork
284.90 ± 40% -100.0% 0.00 perf-sched.wait_time.avg.ms.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait.__x64_sys_epoll_wait
239.53 ± 9% -100.0% 0.00 perf-sched.wait_time.avg.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_sys_poll
478.79 -100.0% 0.00 perf-sched.wait_time.avg.ms.schedule_timeout.kcompactd.kthread.ret_from_fork
6.71 ± 12% -100.0% 0.00 perf-sched.wait_time.avg.ms.schedule_timeout.khugepaged.kthread.ret_from_fork
6.26 ± 15% -100.0% 0.00 perf-sched.wait_time.avg.ms.schedule_timeout.rcu_gp_kthread.kthread.ret_from_fork
652.98 -100.0% 0.00 perf-sched.wait_time.avg.ms.smpboot_thread_fn.kthread.ret_from_fork
0.40 ± 53% -100.0% 0.00 perf-sched.wait_time.avg.ms.wait_for_partner.fifo_open.do_dentry_open.do_open.isra
565.24 ± 5% -100.0% 0.00 perf-sched.wait_time.avg.ms.worker_thread.kthread.ret_from_fork
999.59 -100.0% 0.00 perf-sched.wait_time.max.ms.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown]
2337 ± 50% -100.0% 0.00 perf-sched.wait_time.max.ms.devkmsg_read.vfs_read.ksys_read.do_syscall_64
1000 -100.0% 0.00 perf-sched.wait_time.max.ms.do_nanosleep.hrtimer_nanosleep.__x64_sys_nanosleep.do_syscall_64
2357 ± 49% -100.0% 0.00 perf-sched.wait_time.max.ms.do_syslog.part.0.kmsg_read.vfs_read
1986 ± 84% -100.0% 0.00 perf-sched.wait_time.max.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
1022 -100.0% 0.00 perf-sched.wait_time.max.ms.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64
193.25 ±186% -100.0% 0.00 perf-sched.wait_time.max.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt.[unknown]
7.96 ± 94% -100.0% 0.00 perf-sched.wait_time.max.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_call_function_single.[unknown]
10.10 ±219% -100.0% 0.00 perf-sched.wait_time.max.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_irq_work.[unknown]
9.53 ±115% -100.0% 0.00 perf-sched.wait_time.max.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_reschedule_ipi.[unknown]
27.50 ± 25% -100.0% 0.00 perf-sched.wait_time.max.ms.exit_to_user_mode_prepare.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe.[unknown]
869.52 ±211% -100.0% 0.00 perf-sched.wait_time.max.ms.futex_wait_queue_me.futex_wait.do_futex.__x64_sys_futex
2338 ± 50% -100.0% 0.00 perf-sched.wait_time.max.ms.pipe_read.new_sync_read.vfs_read.ksys_read
12.88 ± 2% -100.0% 0.00 perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.dput.__fput.task_work_run
3266 ± 99% -100.0% 0.00 perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.generic_perform_write.__generic_file_write_iter.generic_file_write_iter
6.44 ± 21% -100.0% 0.00 perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.khugepaged.kthread.ret_from_fork
1849 -100.0% 0.00 perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.stop_one_cpu.affine_move_task.__set_cpus_allowed_ptr
10.00 ±116% -100.0% 0.00 perf-sched.wait_time.max.ms.rcu_gp_kthread.kthread.ret_from_fork
1999 ± 57% -100.0% 0.00 perf-sched.wait_time.max.ms.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait.__x64_sys_epoll_wait
5216 ± 36% -100.0% 0.00 perf-sched.wait_time.max.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_sys_poll
504.99 -100.0% 0.00 perf-sched.wait_time.max.ms.schedule_timeout.kcompactd.kthread.ret_from_fork
6.71 ± 12% -100.0% 0.00 perf-sched.wait_time.max.ms.schedule_timeout.khugepaged.kthread.ret_from_fork
254.82 ± 29% -100.0% 0.00 perf-sched.wait_time.max.ms.schedule_timeout.rcu_gp_kthread.kthread.ret_from_fork
5585 ± 20% -100.0% 0.00 perf-sched.wait_time.max.ms.smpboot_thread_fn.kthread.ret_from_fork
7.41 ± 59% -100.0% 0.00 perf-sched.wait_time.max.ms.wait_for_partner.fifo_open.do_dentry_open.do_open.isra
7413 ± 6% -100.0% 0.00 perf-sched.wait_time.max.ms.worker_thread.kthread.ret_from_fork***************************************************************************************************
lkp-ivb-2ep1: 48 threads Intel(R) Xeon(R) CPU E5-2697 v2 @ 2.70GHz with 112G memory
=========================================================================================
compiler/cpufreq_governor/kconfig/mode/nr_task/rootfs/tbox_group/test/testcase/ucode:
gcc-9/performance/x86_64-rhel-8.3/process/50%/debian-10.4-x86_64-20200603.cgz/lkp-ivb-2ep1/pread1/will-it-scale/0x42e
commit:
dde38d5b80 ("kernel: don't include PF_IO_WORKERs as part of same_thread_group()") 43b2a76b1a ("proc: don't show PF_IO_WORKER threads as threads in /proc/<pid>/task/")
dde38d5b805358c7 43b2a76b1a5abcc9833b463bef1
---------------- ---------------------------
%stddev %change %stddev
\ | \
32174247 -2.6% 31345253 will-it-scale.24.processes
49.34 -8.3% 45.25 will-it-scale.24.processes_idle
1340593 -2.6% 1306051 will-it-scale.per_process_ops
32174247 -2.6% 31345253 will-it-scale.workload
0.03 +0.0 0.04 ± 3% mpstat.cpu.all.soft%
327098 ± 5% +118.6% 715075 ± 5% numa-numastat.node0.local_node
357493 ± 2% +106.8% 739292 ± 4% numa-numastat.node0.numa_hit
372736 ± 4% +93.8% 722461 ± 3% numa-numastat.node1.local_node
385821 +92.2% 741716 numa-numastat.node1.numa_hit
49.00 -8.2% 45.00 vmstat.cpu.id
1337175 +92.9% 2579543 vmstat.memory.cache
1225 ± 2% +47.6% 1808 ± 2% vmstat.system.cs
97244 -9.5% 87974 ± 20% vmstat.system.in
61834 +2002.0% 1299755 meminfo.Active
61834 +2002.0% 1299755 meminfo.Active(anon)
1255800 +98.5% 2492662 meminfo.Cached
441192 +290.3% 1722149 meminfo.Committed_AS
2148004 +67.0% 3586556 meminfo.Memused
74825 +1653.0% 1311691 meminfo.Shmem
2339292 +53.3% 3586556 meminfo.max_used_kB
0.53 +8.7% 0.58 ± 2% sched_debug.cfs_rq:/.nr_running.avg
225.94 ± 2% +15.3% 260.59 ± 3% sched_debug.cfs_rq:/.util_est_enqueued.avg
584.12 ± 4% +70.5% 995.71 ± 2% sched_debug.cfs_rq:/.util_est_enqueued.max
231.12 ± 2% +13.1% 261.49 sched_debug.cfs_rq:/.util_est_enqueued.stddev
611.20 ± 3% +12.9% 689.89 ± 7% sched_debug.cpu.clock_task.stddev
5469 ± 7% +26.7% 6932 ± 6% sched_debug.cpu.nr_switches.avg
15262 ± 15% +72.5% 26320 ± 12% sched_debug.cpu.nr_switches.max
3393 ± 13% +56.6% 5313 ± 7% sched_debug.cpu.nr_switches.stddev
14380 ± 3% +17.7% 16924 ± 4% slabinfo.filp.active_objs
14621 ± 2% +17.5% 17182 ± 4% slabinfo.filp.num_objs
5668 ± 2% +11.9% 6341 ± 2% slabinfo.kmalloc-256.active_objs
5756 ± 2% +13.0% 6503 ± 2% slabinfo.kmalloc-256.num_objs
9145 -9.5% 8277 ± 2% slabinfo.proc_inode_cache.active_objs
9145 -9.5% 8277 ± 2% slabinfo.proc_inode_cache.num_objs
17859 +26.4% 22573 slabinfo.radix_tree_node.active_objs
637.33 +26.4% 805.83 slabinfo.radix_tree_node.active_slabs
17859 +26.4% 22573 slabinfo.radix_tree_node.num_objs
637.33 +26.4% 805.83 slabinfo.radix_tree_node.num_slabs
4183 ± 76% +16432.3% 691573 ± 18% numa-meminfo.node0.Active
4183 ± 76% +16432.3% 691573 ± 18% numa-meminfo.node0.Active(anon)
594660 ± 2% +117.1% 1290807 ± 9% numa-meminfo.node0.FilePages
1089205 ± 7% +74.9% 1905557 ± 4% numa-meminfo.node0.MemUsed
2271 ± 15% +19.0% 2702 ± 6% numa-meminfo.node0.PageTables
10495 ± 55% +6560.5% 699019 ± 18% numa-meminfo.node0.Shmem
57656 ± 5% +958.4% 610240 ± 18% numa-meminfo.node1.Active
57656 ± 5% +958.4% 610240 ± 18% numa-meminfo.node1.Active(anon)
661155 ± 2% +82.1% 1203912 ± 9% numa-meminfo.node1.FilePages
1058311 ± 7% +59.0% 1683051 ± 4% numa-meminfo.node1.MemUsed
64345 ± 8% +855.4% 614730 ± 18% numa-meminfo.node1.Shmem
1045 ± 76% +16398.7% 172521 ± 18% numa-vmstat.node0.nr_active_anon
148663 ± 2% +116.8% 322329 ± 9% numa-vmstat.node0.nr_file_pages
567.50 ± 16% +19.2% 676.33 ± 6% numa-vmstat.node0.nr_page_table_pages
2621 ± 55% +6551.1% 174381 ± 18% numa-vmstat.node0.nr_shmem
1045 ± 76% +16398.7% 172521 ± 18% numa-vmstat.node0.nr_zone_active_anon
663612 ± 11% +34.6% 893276 ± 4% numa-vmstat.node0.numa_hit
630366 ± 10% +37.7% 867909 ± 5% numa-vmstat.node0.numa_local
14426 ± 5% +955.1% 152209 ± 18% numa-vmstat.node1.nr_active_anon
165264 ± 2% +81.9% 300627 ± 9% numa-vmstat.node1.nr_file_pages
16062 ± 8% +854.6% 153332 ± 18% numa-vmstat.node1.nr_shmem
14426 ± 5% +955.1% 152208 ± 18% numa-vmstat.node1.nr_zone_active_anon
721032 ± 10% +19.7% 862855 ± 6% numa-vmstat.node1.numa_hit
15452 +2002.0% 324798 proc-vmstat.nr_active_anon
59520 +4.1% 61985 proc-vmstat.nr_anon_pages
2812461 -1.3% 2776554 proc-vmstat.nr_dirty_background_threshold
5631801 -1.3% 5559898 proc-vmstat.nr_dirty_threshold
313924 +98.5% 623025 proc-vmstat.nr_file_pages
28307278 -1.3% 27947675 proc-vmstat.nr_free_pages
62640 +3.5% 64853 proc-vmstat.nr_inactive_anon
1155 +8.2% 1250 proc-vmstat.nr_page_table_pages
18681 +1654.6% 327783 proc-vmstat.nr_shmem
20318 +1.9% 20696 proc-vmstat.nr_slab_reclaimable
25125 +2.8% 25818 proc-vmstat.nr_slab_unreclaimable
15452 +2002.0% 324798 proc-vmstat.nr_zone_active_anon
62640 +3.5% 64853 proc-vmstat.nr_zone_inactive_anon
771109 +95.3% 1505616 proc-vmstat.numa_hit
727606 +100.9% 1462123 proc-vmstat.numa_local
62333 ± 25% -44.5% 34565 ± 48% proc-vmstat.numa_pte_updates
23012 +88.6% 43401 proc-vmstat.pgactivate
819919 +104.4% 1676106 ± 2% proc-vmstat.pgalloc_normal
901629 -3.4% 870576 proc-vmstat.pgfault
783570 +19.4% 935666 ± 3% proc-vmstat.pgfree
72.90 ± 2% -72.9 0.00 perf-profile.calltrace.cycles-pp.__libc_pread
53.42 ± 2% -53.4 0.00 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__libc_pread
35.64 ± 2% -35.6 0.00 perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_pread
33.46 ± 2% -33.5 0.00 perf-profile.calltrace.cycles-pp.ksys_pread64.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_pread
31.79 ± 2% -31.8 0.00 perf-profile.calltrace.cycles-pp.vfs_read.ksys_pread64.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_pread
26.04 ± 5% -26.0 0.00 perf-profile.calltrace.cycles-pp.secondary_startup_64_no_verify
25.98 ± 5% -26.0 0.00 perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry
25.65 ± 2% -25.7 0.00 perf-profile.calltrace.cycles-pp.new_sync_read.vfs_read.ksys_pread64.do_syscall_64.entry_SYSCALL_64_after_hwframe
25.08 ± 5% -25.1 0.00 perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
25.08 ± 5% -25.1 0.00 perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
25.08 ± 5% -25.1 0.00 perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64_no_verify
25.06 ± 5% -25.1 0.00 perf-profile.calltrace.cycles-pp.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
25.06 ± 5% -25.1 0.00 perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary
24.49 ± 2% -24.5 0.00 perf-profile.calltrace.cycles-pp.shmem_file_read_iter.new_sync_read.vfs_read.ksys_pread64.do_syscall_64
16.81 -16.8 0.00 perf-profile.calltrace.cycles-pp.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe.__libc_pread
11.46 -11.5 0.00 perf-profile.calltrace.cycles-pp.copy_page_to_iter.shmem_file_read_iter.new_sync_read.vfs_read.ksys_pread64
9.80 -9.8 0.00 perf-profile.calltrace.cycles-pp.copyout.copy_page_to_iter.shmem_file_read_iter.new_sync_read.vfs_read
9.66 -9.7 0.00 perf-profile.calltrace.cycles-pp.copy_user_enhanced_fast_string.copyout.copy_page_to_iter.shmem_file_read_iter.new_sync_read
9.16 ± 2% -9.2 0.00 perf-profile.calltrace.cycles-pp.__entry_text_start.__libc_pread
8.47 -8.5 0.00 perf-profile.calltrace.cycles-pp.syscall_return_via_sysret.__libc_pread
6.27 ± 2% -6.3 0.00 perf-profile.calltrace.cycles-pp.shmem_getpage_gfp.shmem_file_read_iter.new_sync_read.vfs_read.ksys_pread64
73.26 ± 2% -73.3 0.00 perf-profile.children.cycles-pp.__libc_pread
53.67 ± 2% -53.7 0.00 perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
35.82 ± 2% -35.8 0.00 perf-profile.children.cycles-pp.do_syscall_64
33.53 ± 2% -33.5 0.00 perf-profile.children.cycles-pp.ksys_pread64
31.94 ± 2% -31.9 0.00 perf-profile.children.cycles-pp.vfs_read
26.04 ± 5% -26.0 0.00 perf-profile.children.cycles-pp.secondary_startup_64_no_verify
26.04 ± 5% -26.0 0.00 perf-profile.children.cycles-pp.cpu_startup_entry
26.04 ± 5% -26.0 0.00 perf-profile.children.cycles-pp.do_idle
26.02 ± 5% -26.0 0.00 perf-profile.children.cycles-pp.cpuidle_enter
26.02 ± 5% -26.0 0.00 perf-profile.children.cycles-pp.cpuidle_enter_state
25.99 ± 5% -26.0 0.00 perf-profile.children.cycles-pp.intel_idle
25.69 ± 2% -25.7 0.00 perf-profile.children.cycles-pp.new_sync_read
25.08 ± 5% -25.1 0.00 perf-profile.children.cycles-pp.start_secondary
24.59 ± 2% -24.6 0.00 perf-profile.children.cycles-pp.shmem_file_read_iter
16.90 -16.9 0.00 perf-profile.children.cycles-pp.syscall_exit_to_user_mode
11.56 -11.6 0.00 perf-profile.children.cycles-pp.copy_page_to_iter
9.87 -9.9 0.00 perf-profile.children.cycles-pp.copyout
9.71 -9.7 0.00 perf-profile.children.cycles-pp.copy_user_enhanced_fast_string
9.52 ± 2% -9.5 0.00 perf-profile.children.cycles-pp.syscall_return_via_sysret
8.98 -9.0 0.00 perf-profile.children.cycles-pp.__entry_text_start
6.29 ± 2% -6.3 0.00 perf-profile.children.cycles-pp.shmem_getpage_gfp
25.99 ± 5% -26.0 0.00 perf-profile.self.cycles-pp.intel_idle
16.30 -16.3 0.00 perf-profile.self.cycles-pp.syscall_exit_to_user_mode
9.60 -9.6 0.00 perf-profile.self.cycles-pp.copy_user_enhanced_fast_string
9.48 ± 2% -9.5 0.00 perf-profile.self.cycles-pp.syscall_return_via_sysret
7.91 ± 2% -7.9 0.00 perf-profile.self.cycles-pp.__entry_text_start
19897 ± 14% +948.1% 208538 ±107% softirqs.CPU0.RCU
22322 ± 25% +435.8% 119597 ±145% softirqs.CPU1.RCU
25408 ± 22% +86.4% 47368 ± 14% softirqs.CPU10.RCU
21039 ± 19% +90.9% 40158 ± 18% softirqs.CPU11.RCU
15984 ± 17% +706.3% 128875 ±137% softirqs.CPU12.RCU
20208 ± 8% +705.7% 162828 ±117% softirqs.CPU13.RCU
20333 ± 22% +64.4% 33427 ± 13% softirqs.CPU14.RCU
20027 ± 19% +63.8% 32798 ± 24% softirqs.CPU15.RCU
23328 ± 19% +79.4% 41842 ± 24% softirqs.CPU16.RCU
19957 ± 22% +106.7% 41245 ± 24% softirqs.CPU17.RCU
18919 ± 31% +108.0% 39358 ± 26% softirqs.CPU18.RCU
21789 ± 18% +67.6% 36521 ± 16% softirqs.CPU19.RCU
24863 ± 29% +82.5% 45387 ± 24% softirqs.CPU2.RCU
20013 ± 18% +98.5% 39732 ± 14% softirqs.CPU20.RCU
20732 ± 18% +112.8% 44129 ± 16% softirqs.CPU21.RCU
19935 ± 17% +501.2% 119845 ±149% softirqs.CPU22.RCU
19639 ± 13% +138.7% 46875 ± 16% softirqs.CPU23.RCU
22718 ± 16% +78.6% 40573 ± 14% softirqs.CPU24.RCU
16177 ± 28% +131.3% 37422 ± 15% softirqs.CPU25.RCU
17400 ± 18% +97.5% 34370 ± 19% softirqs.CPU26.RCU
16930 ± 13% +588.6% 116579 ±153% softirqs.CPU27.RCU
16811 ± 21% +117.9% 36632 ± 18% softirqs.CPU28.RCU
18904 ± 21% +537.8% 120568 ±149% softirqs.CPU29.RCU
23340 ± 24% +88.6% 44009 ± 9% softirqs.CPU3.RCU
16236 ± 19% +100.2% 32511 ± 26% softirqs.CPU30.RCU
17583 ± 22% +87.9% 33031 ± 18% softirqs.CPU31.RCU
15133 ± 17% +97.0% 29817 ± 10% softirqs.CPU32.RCU
14155 ± 6% +142.6% 34335 ± 19% softirqs.CPU33.RCU
14907 ± 8% +105.2% 30596 ± 17% softirqs.CPU34.RCU
17290 ± 26% +106.8% 35749 ± 18% softirqs.CPU35.RCU
17657 ± 26% +89.5% 33455 ± 13% softirqs.CPU36.RCU
15973 ± 12% +123.0% 35617 ± 18% softirqs.CPU37.RCU
16076 ± 7% +625.1% 116569 ±156% softirqs.CPU38.RCU
17260 ± 13% +123.5% 38585 ± 12% softirqs.CPU39.RCU
22100 ± 24% +116.2% 47774 ± 33% softirqs.CPU4.RCU
17436 ± 15% +128.0% 39762 ± 17% softirqs.CPU40.RCU
18958 ± 30% +538.8% 121115 ±145% softirqs.CPU41.RCU
19060 ± 14% +110.9% 40191 ± 17% softirqs.CPU42.RCU
17989 ± 14% +123.7% 40249 ± 21% softirqs.CPU43.RCU
18994 ± 16% +106.2% 39168 ± 22% softirqs.CPU44.RCU
18178 ± 11% +104.4% 37160 ± 23% softirqs.CPU45.RCU
19523 ± 12% +90.3% 37155 ± 11% softirqs.CPU46.RCU
19627 ± 20% +61.9% 31774 ± 15% softirqs.CPU47.RCU
20905 ± 17% +104.9% 42825 ± 19% softirqs.CPU5.RCU
21597 ± 24% +90.4% 41126 ± 27% softirqs.CPU6.RCU
21684 ± 19% +365.1% 100860 ±117% softirqs.CPU7.RCU
22081 ± 14% +112.4% 46911 ± 22% softirqs.CPU8.RCU
933853 ± 8% +198.4% 2786674 ± 6% softirqs.RCU
25875 ± 5% +66.9% 43189 ± 4% softirqs.TIMER
0.24 ± 15% +108.0% 0.51 ± 6% perf-stat.i.MPKI
1.079e+10 +4.6% 1.128e+10 perf-stat.i.branch-instructions
0.94 -0.0 0.91 perf-stat.i.branch-miss-rate%
1.003e+08 +2.0% 1.023e+08 perf-stat.i.branch-misses
1.90 ± 13% +10.3 12.23 ± 9% perf-stat.i.cache-miss-rate%
271081 ± 3% +1125.1% 3321051 ± 8% perf-stat.i.cache-misses
12121406 ± 15% +124.5% 27215732 ± 7% perf-stat.i.cache-references
1200 ± 2% +48.6% 1783 ± 2% perf-stat.i.context-switches
1.38 +3.2% 1.42 perf-stat.i.cpi
7.223e+10 +8.0% 7.801e+10 perf-stat.i.cpu-cycles
71.77 -2.8% 69.73 perf-stat.i.cpu-migrations
391619 ± 2% -92.1% 30968 ± 50% perf-stat.i.cycles-between-cache-misses
0.34 -0.0 0.33 perf-stat.i.dTLB-store-miss-rate%
64281882 -1.2% 63510875 perf-stat.i.dTLB-store-misses
33566536 -4.4% 32101494 perf-stat.i.iTLB-load-misses
5.247e+10 +4.6% 5.489e+10 perf-stat.i.instructions
1577 +9.5% 1726 perf-stat.i.instructions-per-iTLB-miss
0.73 -3.1% 0.70 perf-stat.i.ipc
0.40 ± 4% -78.5% 0.09 ± 34% perf-stat.i.major-faults
1.50 +8.0% 1.63 perf-stat.i.metric.GHz
0.72 ± 9% -78.5% 0.15 perf-stat.i.metric.K/sec
1123 +1.4% 1138 perf-stat.i.metric.M/sec
2891 -3.6% 2786 perf-stat.i.minor-faults
47.36 +1.6 48.98 perf-stat.i.node-load-miss-rate%
107976 ± 3% +1660.0% 1900404 ± 11% perf-stat.i.node-load-misses
130609 +1396.4% 1954489 ± 11% perf-stat.i.node-loads
34.59 ± 6% +11.5 46.05 ± 4% perf-stat.i.node-store-miss-rate%
59311 ± 7% +2382.6% 1472493 ± 27% perf-stat.i.node-store-misses
119688 +1282.0% 1654044 ± 25% perf-stat.i.node-stores
2891 -3.6% 2786 perf-stat.i.page-faults
0.23 ± 15% +114.6% 0.50 ± 7% perf-stat.overall.MPKI
0.93 -0.0 0.91 perf-stat.overall.branch-miss-rate%
2.29 ± 15% +10.0 12.25 ± 9% perf-stat.overall.cache-miss-rate%
1.38 +3.2% 1.42 perf-stat.overall.cpi
266141 ± 3% -91.1% 23675 ± 8% perf-stat.overall.cycles-between-cache-misses
0.34 -0.0 0.33 perf-stat.overall.dTLB-store-miss-rate%
1563 +9.4% 1710 perf-stat.overall.instructions-per-iTLB-miss
0.73 -3.1% 0.70 perf-stat.overall.ipc
45.24 +4.1 49.29 perf-stat.overall.node-load-miss-rate%
33.09 ± 4% +13.9 46.98 perf-stat.overall.node-store-miss-rate%
490908 +7.4% 527414 perf-stat.overall.path-length
1.075e+10 +4.6% 1.124e+10 perf-stat.ps.branch-instructions
99957025 +2.0% 1.02e+08 perf-stat.ps.branch-misses
270839 ± 3% +1122.1% 3309953 ± 8% perf-stat.ps.cache-misses
12084255 ± 15% +124.5% 27126455 ± 7% perf-stat.ps.cache-references
1196 ± 2% +48.5% 1777 ± 2% perf-stat.ps.context-switches
7.199e+10 +8.0% 7.774e+10 perf-stat.ps.cpu-cycles
71.54 -2.8% 69.51 perf-stat.ps.cpu-migrations
64064667 -1.2% 63294229 perf-stat.ps.dTLB-store-misses
33453056 -4.4% 31991765 perf-stat.ps.iTLB-load-misses
5.229e+10 +4.6% 5.471e+10 perf-stat.ps.instructions
0.40 ± 4% -78.4% 0.09 ± 34% perf-stat.ps.major-faults
2882 -3.6% 2777 perf-stat.ps.minor-faults
107680 ± 3% +1658.7% 1893794 ± 11% perf-stat.ps.node-load-misses
130298 +1394.8% 1947735 ± 11% perf-stat.ps.node-loads
59232 ± 7% +2377.4% 1467437 ± 27% perf-stat.ps.node-store-misses
119549 +1278.9% 1648476 ± 25% perf-stat.ps.node-stores
2882 -3.7% 2777 perf-stat.ps.page-faults
1.579e+13 +4.7% 1.653e+13 perf-stat.total.instructions
57963 -6.5% 54206 interrupts.CAL:Function_call_interrupts
2708 ± 34% -100.0% 0.00 interrupts.CPU0.NMI:Non-maskable_interrupts
2708 ± 34% -100.0% 0.00 interrupts.CPU0.PMI:Performance_monitoring_interrupts
3773 ± 61% -100.0% 0.00 interrupts.CPU1.NMI:Non-maskable_interrupts
3773 ± 61% -100.0% 0.00 interrupts.CPU1.PMI:Performance_monitoring_interrupts
5117 ± 48% -100.0% 0.00 interrupts.CPU10.NMI:Non-maskable_interrupts
5117 ± 48% -100.0% 0.00 interrupts.CPU10.PMI:Performance_monitoring_interrupts
3932 ± 31% -100.0% 0.00 interrupts.CPU11.NMI:Non-maskable_interrupts
3932 ± 31% -100.0% 0.00 interrupts.CPU11.PMI:Performance_monitoring_interrupts
4655 ± 60% -100.0% 0.00 interrupts.CPU12.NMI:Non-maskable_interrupts
4655 ± 60% -100.0% 0.00 interrupts.CPU12.PMI:Performance_monitoring_interrupts
4903 ± 51% -100.0% 0.00 interrupts.CPU13.NMI:Non-maskable_interrupts
4903 ± 51% -100.0% 0.00 interrupts.CPU13.PMI:Performance_monitoring_interrupts
4380 ± 64% -100.0% 0.00 interrupts.CPU14.NMI:Non-maskable_interrupts
4380 ± 64% -100.0% 0.00 interrupts.CPU14.PMI:Performance_monitoring_interrupts
3342 ± 54% -100.0% 0.00 interrupts.CPU15.NMI:Non-maskable_interrupts
3342 ± 54% -100.0% 0.00 interrupts.CPU15.PMI:Performance_monitoring_interrupts
4933 ± 50% -100.0% 0.00 interrupts.CPU16.NMI:Non-maskable_interrupts
4933 ± 50% -100.0% 0.00 interrupts.CPU16.PMI:Performance_monitoring_interrupts
4272 ± 67% -100.0% 0.00 interrupts.CPU17.NMI:Non-maskable_interrupts
4272 ± 67% -100.0% 0.00 interrupts.CPU17.PMI:Performance_monitoring_interrupts
4640 ± 37% -100.0% 0.00 interrupts.CPU18.NMI:Non-maskable_interrupts
4640 ± 37% -100.0% 0.00 interrupts.CPU18.PMI:Performance_monitoring_interrupts
3997 ± 28% -100.0% 0.00 interrupts.CPU19.NMI:Non-maskable_interrupts
3997 ± 28% -100.0% 0.00 interrupts.CPU19.PMI:Performance_monitoring_interrupts
4229 ± 47% -100.0% 0.00 interrupts.CPU2.NMI:Non-maskable_interrupts
4229 ± 47% -100.0% 0.00 interrupts.CPU2.PMI:Performance_monitoring_interrupts
3888 ± 55% -100.0% 0.00 interrupts.CPU20.NMI:Non-maskable_interrupts
3888 ± 55% -100.0% 0.00 interrupts.CPU20.PMI:Performance_monitoring_interrupts
5098 ± 48% -100.0% 0.00 interrupts.CPU21.NMI:Non-maskable_interrupts
5098 ± 48% -100.0% 0.00 interrupts.CPU21.PMI:Performance_monitoring_interrupts
4788 ± 51% -100.0% 0.00 interrupts.CPU22.NMI:Non-maskable_interrupts
4788 ± 51% -100.0% 0.00 interrupts.CPU22.PMI:Performance_monitoring_interrupts
4535 ± 42% -100.0% 0.00 interrupts.CPU23.NMI:Non-maskable_interrupts
4535 ± 42% -100.0% 0.00 interrupts.CPU23.PMI:Performance_monitoring_interrupts
6687 ± 34% -100.0% 0.00 interrupts.CPU24.NMI:Non-maskable_interrupts
6687 ± 34% -100.0% 0.00 interrupts.CPU24.PMI:Performance_monitoring_interrupts
3988 ± 52% -100.0% 0.00 interrupts.CPU25.NMI:Non-maskable_interrupts
3988 ± 52% -100.0% 0.00 interrupts.CPU25.PMI:Performance_monitoring_interrupts
2905 ± 37% -100.0% 0.00 interrupts.CPU26.NMI:Non-maskable_interrupts
2905 ± 37% -100.0% 0.00 interrupts.CPU26.PMI:Performance_monitoring_interrupts
4086 ± 66% -100.0% 0.00 interrupts.CPU27.NMI:Non-maskable_interrupts
4086 ± 66% -100.0% 0.00 interrupts.CPU27.PMI:Performance_monitoring_interrupts
5137 ± 53% -100.0% 0.00 interrupts.CPU28.NMI:Non-maskable_interrupts
5137 ± 53% -100.0% 0.00 interrupts.CPU28.PMI:Performance_monitoring_interrupts
4698 ± 43% -100.0% 0.00 interrupts.CPU29.NMI:Non-maskable_interrupts
4698 ± 43% -100.0% 0.00 interrupts.CPU29.PMI:Performance_monitoring_interrupts
6014 ± 39% -100.0% 0.00 interrupts.CPU3.NMI:Non-maskable_interrupts
6014 ± 39% -100.0% 0.00 interrupts.CPU3.PMI:Performance_monitoring_interrupts
3091 ± 30% -100.0% 0.00 interrupts.CPU30.NMI:Non-maskable_interrupts
3091 ± 30% -100.0% 0.00 interrupts.CPU30.PMI:Performance_monitoring_interrupts
4042 ± 52% -100.0% 0.00 interrupts.CPU31.NMI:Non-maskable_interrupts
4042 ± 52% -100.0% 0.00 interrupts.CPU31.PMI:Performance_monitoring_interrupts
2411 ± 43% -100.0% 0.00 interrupts.CPU32.NMI:Non-maskable_interrupts
2411 ± 43% -100.0% 0.00 interrupts.CPU32.PMI:Performance_monitoring_interrupts
1840 ± 26% -37.6% 1148 ± 23% interrupts.CPU33.CAL:Function_call_interrupts
2570 ± 96% -100.0% 0.00 interrupts.CPU33.NMI:Non-maskable_interrupts
2570 ± 96% -100.0% 0.00 interrupts.CPU33.PMI:Performance_monitoring_interrupts
100.00 ± 16% +77.2% 177.17 ± 28% interrupts.CPU33.RES:Rescheduling_interrupts
2373 ± 42% -100.0% 0.00 interrupts.CPU34.NMI:Non-maskable_interrupts
2373 ± 42% -100.0% 0.00 interrupts.CPU34.PMI:Performance_monitoring_interrupts
3532 ± 73% -100.0% 0.00 interrupts.CPU35.NMI:Non-maskable_interrupts
3532 ± 73% -100.0% 0.00 interrupts.CPU35.PMI:Performance_monitoring_interrupts
3293 ± 74% -100.0% 0.00 interrupts.CPU36.NMI:Non-maskable_interrupts
3293 ± 74% -100.0% 0.00 interrupts.CPU36.PMI:Performance_monitoring_interrupts
2805 ± 42% -100.0% 0.00 interrupts.CPU37.NMI:Non-maskable_interrupts
2805 ± 42% -100.0% 0.00 interrupts.CPU37.PMI:Performance_monitoring_interrupts
4824 ± 42% -100.0% 0.00 interrupts.CPU38.NMI:Non-maskable_interrupts
4824 ± 42% -100.0% 0.00 interrupts.CPU38.PMI:Performance_monitoring_interrupts
120.00 ± 52% +89.4% 227.33 ± 23% interrupts.CPU38.RES:Rescheduling_interrupts
4038 ± 52% -100.0% 0.00 interrupts.CPU39.NMI:Non-maskable_interrupts
4038 ± 52% -100.0% 0.00 interrupts.CPU39.PMI:Performance_monitoring_interrupts
3067 ± 39% -100.0% 0.00 interrupts.CPU4.NMI:Non-maskable_interrupts
3067 ± 39% -100.0% 0.00 interrupts.CPU4.PMI:Performance_monitoring_interrupts
4436 ± 60% -100.0% 1.00 ±223% interrupts.CPU40.NMI:Non-maskable_interrupts
4436 ± 60% -100.0% 1.00 ±223% interrupts.CPU40.PMI:Performance_monitoring_interrupts
4425 ± 40% -100.0% 0.00 interrupts.CPU41.NMI:Non-maskable_interrupts
4425 ± 40% -100.0% 0.00 interrupts.CPU41.PMI:Performance_monitoring_interrupts
3077 ± 79% -100.0% 0.00 interrupts.CPU42.NMI:Non-maskable_interrupts
3077 ± 79% -100.0% 0.00 interrupts.CPU42.PMI:Performance_monitoring_interrupts
2626 ± 32% -100.0% 0.00 interrupts.CPU43.NMI:Non-maskable_interrupts
2626 ± 32% -100.0% 0.00 interrupts.CPU43.PMI:Performance_monitoring_interrupts
5302 ± 43% -100.0% 0.00 interrupts.CPU44.NMI:Non-maskable_interrupts
5302 ± 43% -100.0% 0.00 interrupts.CPU44.PMI:Performance_monitoring_interrupts
2654 ± 35% -100.0% 0.00 interrupts.CPU45.NMI:Non-maskable_interrupts
2654 ± 35% -100.0% 0.00 interrupts.CPU45.PMI:Performance_monitoring_interrupts
3091 ± 41% -100.0% 0.00 interrupts.CPU46.NMI:Non-maskable_interrupts
3091 ± 41% -100.0% 0.00 interrupts.CPU46.PMI:Performance_monitoring_interrupts
3730 ± 44% -100.0% 0.00 interrupts.CPU47.NMI:Non-maskable_interrupts
3730 ± 44% -100.0% 0.00 interrupts.CPU47.PMI:Performance_monitoring_interrupts
4877 ± 53% -100.0% 0.00 interrupts.CPU5.NMI:Non-maskable_interrupts
4877 ± 53% -100.0% 0.00 interrupts.CPU5.PMI:Performance_monitoring_interrupts
3268 ± 29% -100.0% 0.00 interrupts.CPU6.NMI:Non-maskable_interrupts
3268 ± 29% -100.0% 0.00 interrupts.CPU6.PMI:Performance_monitoring_interrupts
5376 ± 50% -100.0% 0.00 interrupts.CPU7.NMI:Non-maskable_interrupts
5376 ± 50% -100.0% 0.00 interrupts.CPU7.PMI:Performance_monitoring_interrupts
3774 ± 23% -100.0% 0.00 interrupts.CPU8.NMI:Non-maskable_interrupts
3774 ± 23% -100.0% 0.00 interrupts.CPU8.PMI:Performance_monitoring_interrupts
6783 ± 32% -100.0% 0.00 interrupts.CPU9.NMI:Non-maskable_interrupts
6783 ± 32% -100.0% 0.00 interrupts.CPU9.PMI:Performance_monitoring_interrupts
289.33 ± 5% -21.8% 226.17 ± 15% interrupts.CPU9.RES:Rescheduling_interrupts
196194 ± 3% -100.0% 1.00 ±223% interrupts.NMI:Non-maskable_interrupts
196194 ± 3% -100.0% 1.00 ±223% interrupts.PMI:Performance_monitoring_interrupts
0.01 ± 5% -100.0% 0.00 perf-sched.sch_delay.avg.ms.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown]
0.01 ± 7% -100.0% 0.00 perf-sched.sch_delay.avg.ms.devkmsg_read.vfs_read.ksys_read.do_syscall_64
0.01 ± 5% -100.0% 0.00 perf-sched.sch_delay.avg.ms.do_nanosleep.hrtimer_nanosleep.__x64_sys_nanosleep.do_syscall_64
0.01 ± 8% -100.0% 0.00 perf-sched.sch_delay.avg.ms.do_syslog.part.0.kmsg_read.vfs_read
0.01 -100.0% 0.00 perf-sched.sch_delay.avg.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
0.01 -100.0% 0.00 perf-sched.sch_delay.avg.ms.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64
0.01 ± 12% -100.0% 0.00 perf-sched.sch_delay.avg.ms.futex_wait_queue_me.futex_wait.do_futex.__x64_sys_futex
0.00 -100.0% 0.00 perf-sched.sch_delay.avg.ms.pipe_read.new_sync_read.vfs_read.ksys_read
0.01 ± 40% -100.0% 0.00 perf-sched.sch_delay.avg.ms.preempt_schedule_common.__cond_resched.stop_one_cpu.sched_exec.bprm_execve
0.01 ± 7% -100.0% 0.00 perf-sched.sch_delay.avg.ms.rcu_gp_kthread.kthread.ret_from_fork
0.01 ± 5% -100.0% 0.00 perf-sched.sch_delay.avg.ms.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait.__x64_sys_epoll_wait
0.01 ± 26% -100.0% 0.00 perf-sched.sch_delay.avg.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_select
0.00 ± 14% -100.0% 0.00 perf-sched.sch_delay.avg.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_sys_poll
0.01 ± 6% -100.0% 0.00 perf-sched.sch_delay.avg.ms.schedule_timeout.kcompactd.kthread.ret_from_fork
0.01 ± 7% -100.0% 0.00 perf-sched.sch_delay.avg.ms.schedule_timeout.rcu_gp_kthread.kthread.ret_from_fork
0.02 ± 40% -100.0% 0.00 perf-sched.sch_delay.avg.ms.smpboot_thread_fn.kthread.ret_from_fork
0.01 ± 8% -100.0% 0.00 perf-sched.sch_delay.avg.ms.wait_for_partner.fifo_open.do_dentry_open.do_open.isra
0.04 ± 4% -100.0% 0.00 perf-sched.sch_delay.avg.ms.worker_thread.kthread.ret_from_fork
0.01 ± 21% -100.0% 0.00 perf-sched.sch_delay.max.ms.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown]
0.02 ± 9% -100.0% 0.00 perf-sched.sch_delay.max.ms.devkmsg_read.vfs_read.ksys_read.do_syscall_64
0.02 ± 45% -100.0% 0.00 perf-sched.sch_delay.max.ms.do_nanosleep.hrtimer_nanosleep.__x64_sys_nanosleep.do_syscall_64
0.02 ± 17% -100.0% 0.00 perf-sched.sch_delay.max.ms.do_syslog.part.0.kmsg_read.vfs_read
0.03 ± 25% -100.0% 0.00 perf-sched.sch_delay.max.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
0.04 ± 44% -100.0% 0.00 perf-sched.sch_delay.max.ms.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64
0.01 ± 20% -100.0% 0.00 perf-sched.sch_delay.max.ms.exit_to_user_mode_prepare.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe.[unknown]
0.02 ± 21% -100.0% 0.00 perf-sched.sch_delay.max.ms.futex_wait_queue_me.futex_wait.do_futex.__x64_sys_futex
0.04 ± 14% -100.0% 0.00 perf-sched.sch_delay.max.ms.pipe_read.new_sync_read.vfs_read.ksys_read
0.02 ± 11% -100.0% 0.00 perf-sched.sch_delay.max.ms.preempt_schedule_common.__cond_resched.stop_one_cpu.affine_move_task.__set_cpus_allowed_ptr
0.02 ± 48% -100.0% 0.00 perf-sched.sch_delay.max.ms.preempt_schedule_common.__cond_resched.stop_one_cpu.sched_exec.bprm_execve
0.01 ± 30% -100.0% 0.00 perf-sched.sch_delay.max.ms.rcu_gp_kthread.kthread.ret_from_fork
0.02 ± 21% -100.0% 0.00 perf-sched.sch_delay.max.ms.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait.__x64_sys_epoll_wait
0.01 ± 23% -100.0% 0.00 perf-sched.sch_delay.max.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_select
0.01 ± 10% -100.0% 0.00 perf-sched.sch_delay.max.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_sys_poll
0.02 ± 41% -100.0% 0.00 perf-sched.sch_delay.max.ms.schedule_timeout.kcompactd.kthread.ret_from_fork
0.03 ± 31% -100.0% 0.00 perf-sched.sch_delay.max.ms.schedule_timeout.rcu_gp_kthread.kthread.ret_from_fork
6.02 ± 51% -100.0% 0.00 perf-sched.sch_delay.max.ms.smpboot_thread_fn.kthread.ret_from_fork
0.02 ± 7% -100.0% 0.00 perf-sched.sch_delay.max.ms.wait_for_partner.fifo_open.do_dentry_open.do_open.isra
12.45 -100.0% 0.00 perf-sched.sch_delay.max.ms.worker_thread.kthread.ret_from_fork
0.01 ± 9% -100.0% 0.00 perf-sched.total_sch_delay.average.ms
12.45 -100.0% 0.00 perf-sched.total_sch_delay.max.ms
162.59 ± 7% -100.0% 0.00 perf-sched.total_wait_and_delay.average.ms
6367 ± 9% -100.0% 0.00 perf-sched.total_wait_and_delay.count.ms
7735 ± 4% -100.0% 0.00 perf-sched.total_wait_and_delay.max.ms
162.58 ± 7% -100.0% 0.00 perf-sched.total_wait_time.average.ms
7735 ± 4% -100.0% 0.00 perf-sched.total_wait_time.max.ms
899.64 -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown]
469.21 ± 73% -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.devkmsg_read.vfs_read.ksys_read.do_syscall_64
808.95 ± 4% -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.do_nanosleep.hrtimer_nanosleep.__x64_sys_nanosleep.do_syscall_64
469.22 ± 73% -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.do_syslog.part.0.kmsg_read.vfs_read
272.98 ± 5% -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
1.43 -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64
0.09 ± 61% -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt.[unknown]
0.04 ± 16% -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.exit_to_user_mode_prepare.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe.[unknown]
71.33 ± 2% -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.pipe_read.new_sync_read.vfs_read.ksys_read
0.04 ± 22% -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.preempt_schedule_common.__cond_resched.pagecache_get_page.shmem_getpage_gfp.shmem_file_read_iter
17.39 -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.preempt_schedule_common.__cond_resched.stop_one_cpu.affine_move_task.__set_cpus_allowed_ptr
587.99 ± 9% -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_sys_poll
478.78 -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.schedule_timeout.kcompactd.kthread.ret_from_fork
9.25 ± 31% -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.schedule_timeout.rcu_gp_kthread.kthread.ret_from_fork
652.76 -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.smpboot_thread_fn.kthread.ret_from_fork
0.01 ± 8% -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.wait_for_partner.fifo_open.do_dentry_open.do_open.isra
528.39 ± 2% -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.worker_thread.kthread.ret_from_fork
10.00 -100.0% 0.00 perf-sched.wait_and_delay.count.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown]
7.33 ± 6% -100.0% 0.00 perf-sched.wait_and_delay.count.devkmsg_read.vfs_read.ksys_read.do_syscall_64
22.33 ± 4% -100.0% 0.00 perf-sched.wait_and_delay.count.do_nanosleep.hrtimer_nanosleep.__x64_sys_nanosleep.do_syscall_64
7.33 ± 6% -100.0% 0.00 perf-sched.wait_and_delay.count.do_syslog.part.0.kmsg_read.vfs_read
246.67 -100.0% 0.00 perf-sched.wait_and_delay.count.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
245.67 -100.0% 0.00 perf-sched.wait_and_delay.count.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64
47.83 ± 35% -100.0% 0.00 perf-sched.wait_and_delay.count.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt.[unknown]
185.00 ± 10% -100.0% 0.00 perf-sched.wait_and_delay.count.exit_to_user_mode_prepare.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe.[unknown]
2200 -100.0% 0.00 perf-sched.wait_and_delay.count.pipe_read.new_sync_read.vfs_read.ksys_read
113.33 ± 11% -100.0% 0.00 perf-sched.wait_and_delay.count.preempt_schedule_common.__cond_resched.pagecache_get_page.shmem_getpage_gfp.shmem_file_read_iter
576.00 -100.0% 0.00 perf-sched.wait_and_delay.count.preempt_schedule_common.__cond_resched.stop_one_cpu.affine_move_task.__set_cpus_allowed_ptr
50.67 ± 11% -100.0% 0.00 perf-sched.wait_and_delay.count.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_sys_poll
40.00 -100.0% 0.00 perf-sched.wait_and_delay.count.schedule_timeout.kcompactd.kthread.ret_from_fork
1261 ± 52% -100.0% 0.00 perf-sched.wait_and_delay.count.schedule_timeout.rcu_gp_kthread.kthread.ret_from_fork
680.33 -100.0% 0.00 perf-sched.wait_and_delay.count.smpboot_thread_fn.kthread.ret_from_fork
72.00 -100.0% 0.00 perf-sched.wait_and_delay.count.wait_for_partner.fifo_open.do_dentry_open.do_open.isra
456.50 -100.0% 0.00 perf-sched.wait_and_delay.count.worker_thread.kthread.ret_from_fork
999.63 -100.0% 0.00 perf-sched.wait_and_delay.max.ms.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown]
3039 ± 72% -100.0% 0.00 perf-sched.wait_and_delay.max.ms.devkmsg_read.vfs_read.ksys_read.do_syscall_64
1000 -100.0% 0.00 perf-sched.wait_and_delay.max.ms.do_nanosleep.hrtimer_nanosleep.__x64_sys_nanosleep.do_syscall_64
3039 ± 72% -100.0% 0.00 perf-sched.wait_and_delay.max.ms.do_syslog.part.0.kmsg_read.vfs_read
3228 ± 60% -100.0% 0.00 perf-sched.wait_and_delay.max.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
31.13 -100.0% 0.00 perf-sched.wait_and_delay.max.ms.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64
1.00 ±178% -100.0% 0.00 perf-sched.wait_and_delay.max.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt.[unknown]
0.58 ±115% -100.0% 0.00 perf-sched.wait_and_delay.max.ms.exit_to_user_mode_prepare.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe.[unknown]
3041 ± 72% -100.0% 0.00 perf-sched.wait_and_delay.max.ms.pipe_read.new_sync_read.vfs_read.ksys_read
0.17 ± 52% -100.0% 0.00 perf-sched.wait_and_delay.max.ms.preempt_schedule_common.__cond_resched.pagecache_get_page.shmem_getpage_gfp.shmem_file_read_iter
1029 ± 2% -100.0% 0.00 perf-sched.wait_and_delay.max.ms.preempt_schedule_common.__cond_resched.stop_one_cpu.affine_move_task.__set_cpus_allowed_ptr
6493 ± 16% -100.0% 0.00 perf-sched.wait_and_delay.max.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_sys_poll
504.51 -100.0% 0.00 perf-sched.wait_and_delay.max.ms.schedule_timeout.kcompactd.kthread.ret_from_fork
289.34 ± 25% -100.0% 0.00 perf-sched.wait_and_delay.max.ms.schedule_timeout.rcu_gp_kthread.kthread.ret_from_fork
2708 ± 61% -100.0% 0.00 perf-sched.wait_and_delay.max.ms.smpboot_thread_fn.kthread.ret_from_fork
0.02 ± 7% -100.0% 0.00 perf-sched.wait_and_delay.max.ms.wait_for_partner.fifo_open.do_dentry_open.do_open.isra
6444 ± 17% -100.0% 0.00 perf-sched.wait_and_delay.max.ms.worker_thread.kthread.ret_from_fork
899.63 -100.0% 0.00 perf-sched.wait_time.avg.ms.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown]
469.20 ± 73% -100.0% 0.00 perf-sched.wait_time.avg.ms.devkmsg_read.vfs_read.ksys_read.do_syscall_64
808.94 ± 4% -100.0% 0.00 perf-sched.wait_time.avg.ms.do_nanosleep.hrtimer_nanosleep.__x64_sys_nanosleep.do_syscall_64
469.20 ± 73% -100.0% 0.00 perf-sched.wait_time.avg.ms.do_syslog.part.0.kmsg_read.vfs_read
272.97 ± 5% -100.0% 0.00 perf-sched.wait_time.avg.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
1.41 -100.0% 0.00 perf-sched.wait_time.avg.ms.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64
0.09 ± 61% -100.0% 0.00 perf-sched.wait_time.avg.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt.[unknown]
0.06 ± 90% -100.0% 0.00 perf-sched.wait_time.avg.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_call_function_single.[unknown]
0.04 ± 21% -100.0% 0.00 perf-sched.wait_time.avg.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_reschedule_ipi.[unknown]
0.04 ± 16% -100.0% 0.00 perf-sched.wait_time.avg.ms.exit_to_user_mode_prepare.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe.[unknown]
537.84 ±134% -100.0% 0.00 perf-sched.wait_time.avg.ms.futex_wait_queue_me.futex_wait.do_futex.__x64_sys_futex
71.33 ± 2% -100.0% 0.00 perf-sched.wait_time.avg.ms.pipe_read.new_sync_read.vfs_read.ksys_read
0.03 -100.0% 0.00 perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.copy_page_to_iter.shmem_file_read_iter.new_sync_read
0.04 ± 22% -100.0% 0.00 perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.pagecache_get_page.shmem_getpage_gfp.shmem_file_read_iter
17.39 -100.0% 0.00 perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.stop_one_cpu.affine_move_task.__set_cpus_allowed_ptr
2.56 ± 13% -100.0% 0.00 perf-sched.wait_time.avg.ms.rcu_gp_kthread.kthread.ret_from_fork
365.76 ± 36% -100.0% 0.00 perf-sched.wait_time.avg.ms.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait.__x64_sys_epoll_wait
587.98 ± 9% -100.0% 0.00 perf-sched.wait_time.avg.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_sys_poll
478.78 -100.0% 0.00 perf-sched.wait_time.avg.ms.schedule_timeout.kcompactd.kthread.ret_from_fork
0.02 ± 39% -100.0% 0.00 perf-sched.wait_time.avg.ms.schedule_timeout.khugepaged.kthread.ret_from_fork
9.25 ± 31% -100.0% 0.00 perf-sched.wait_time.avg.ms.schedule_timeout.rcu_gp_kthread.kthread.ret_from_fork
652.75 -100.0% 0.00 perf-sched.wait_time.avg.ms.smpboot_thread_fn.kthread.ret_from_fork
528.35 ± 2% -100.0% 0.00 perf-sched.wait_time.avg.ms.worker_thread.kthread.ret_from_fork
999.62 -100.0% 0.00 perf-sched.wait_time.max.ms.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown]
3039 ± 72% -100.0% 0.00 perf-sched.wait_time.max.ms.devkmsg_read.vfs_read.ksys_read.do_syscall_64
1000 -100.0% 0.00 perf-sched.wait_time.max.ms.do_nanosleep.hrtimer_nanosleep.__x64_sys_nanosleep.do_syscall_64
3039 ± 72% -100.0% 0.00 perf-sched.wait_time.max.ms.do_syslog.part.0.kmsg_read.vfs_read
3228 ± 60% -100.0% 0.00 perf-sched.wait_time.max.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
31.12 -100.0% 0.00 perf-sched.wait_time.max.ms.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64
1.00 ±178% -100.0% 0.00 perf-sched.wait_time.max.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt.[unknown]
0.06 ± 90% -100.0% 0.00 perf-sched.wait_time.max.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_call_function_single.[unknown]
0.09 ± 44% -100.0% 0.00 perf-sched.wait_time.max.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_reschedule_ipi.[unknown]
0.58 ±115% -100.0% 0.00 perf-sched.wait_time.max.ms.exit_to_user_mode_prepare.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe.[unknown]
2081 ±138% -100.0% 0.00 perf-sched.wait_time.max.ms.futex_wait_queue_me.futex_wait.do_futex.__x64_sys_futex
3041 ± 72% -100.0% 0.00 perf-sched.wait_time.max.ms.pipe_read.new_sync_read.vfs_read.ksys_read
0.10 ± 41% -100.0% 0.00 perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.copy_page_to_iter.shmem_file_read_iter.new_sync_read
0.17 ± 52% -100.0% 0.00 perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.pagecache_get_page.shmem_getpage_gfp.shmem_file_read_iter
1029 ± 2% -100.0% 0.00 perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.stop_one_cpu.affine_move_task.__set_cpus_allowed_ptr
4.49 ± 11% -100.0% 0.00 perf-sched.wait_time.max.ms.rcu_gp_kthread.kthread.ret_from_fork
3794 ± 70% -100.0% 0.00 perf-sched.wait_time.max.ms.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait.__x64_sys_epoll_wait
6493 ± 16% -100.0% 0.00 perf-sched.wait_time.max.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_sys_poll
504.50 -100.0% 0.00 perf-sched.wait_time.max.ms.schedule_timeout.kcompactd.kthread.ret_from_fork
0.02 ± 39% -100.0% 0.00 perf-sched.wait_time.max.ms.schedule_timeout.khugepaged.kthread.ret_from_fork
289.33 ± 25% -100.0% 0.00 perf-sched.wait_time.max.ms.schedule_timeout.rcu_gp_kthread.kthread.ret_from_fork
2708 ± 61% -100.0% 0.00 perf-sched.wait_time.max.ms.smpboot_thread_fn.kthread.ret_from_fork
6444 ± 17% -100.0% 0.00 perf-sched.wait_time.max.ms.worker_thread.kthread.ret_from_fork***************************************************************************************************
lkp-skl-fpga01: 104 threads Skylake with 192G memory
=========================================================================================
compiler/cpufreq_governor/kconfig/mode/nr_task/rootfs/tbox_group/test/testcase/ucode:
gcc-9/performance/x86_64-rhel-8.3/process/50%/debian-10.4-x86_64-20200603.cgz/lkp-skl-fpga01/brk1/will-it-scale/0x2006a0a
commit:
dde38d5b80 ("kernel: don't include PF_IO_WORKERs as part of same_thread_group()") 43b2a76b1a ("proc: don't show PF_IO_WORKER threads as threads in /proc/<pid>/task/")
dde38d5b805358c7 43b2a76b1a5abcc9833b463bef1
---------------- ---------------------------
%stddev %change %stddev
\ | \
66531819 -15.3% 56377631 will-it-scale.52.processes
49.47 -3.8% 47.59 will-it-scale.52.processes_idle
1279457 -15.3% 1084184 will-it-scale.per_process_ops
9487 +0.9% 9569 will-it-scale.time.maximum_resident_set_size
66531819 -15.3% 56377631 will-it-scale.workload
52376 ± 7% +32.8% 69564 ± 27% cpuidle.POLL.time
0.04 ± 5% +0.0 0.05 ± 6% mpstat.cpu.all.soft%
27.94 +3.2 31.18 mpstat.cpu.all.sys%
20.15 -3.0 17.13 mpstat.cpu.all.usr%
534.28 ± 5% -41.2% 314.38 ± 6% sched_debug.cfs_rq:/.util_est_enqueued.avg
423.00 ± 2% -29.6% 297.65 ± 3% sched_debug.cfs_rq:/.util_est_enqueued.stddev
3100 ± 17% +42.9% 4430 ± 16% sched_debug.cpu.nr_switches.stddev
357400 ± 8% +116.1% 772492 ± 11% numa-numastat.node0.local_node
408607 +101.1% 821517 ± 8% numa-numastat.node0.numa_hit
431312 ± 6% +73.2% 747209 ± 7% numa-numastat.node1.local_node
473757 +67.1% 791872 ± 4% numa-numastat.node1.numa_hit
494.33 ± 9% +35972.6% 178318 ± 34% numa-vmstat.node0.nr_active_anon
151965 ± 2% +118.3% 331721 ± 18% numa-vmstat.node0.nr_file_pages
3549 ± 25% +5006.2% 181234 ± 33% numa-vmstat.node0.nr_shmem
494.33 ± 9% +35972.6% 178318 ± 34% numa-vmstat.node0.nr_zone_active_anon
994186 ± 6% +25.6% 1248383 ± 10% numa-vmstat.node0.numa_hit
50.00 -4.0% 48.00 vmstat.cpu.id
20.00 -11.7% 17.67 ± 2% vmstat.cpu.us
1433837 +79.8% 2577361 vmstat.memory.cache
1572 ± 5% +31.2% 2063 ± 4% vmstat.system.cs
211278 -1.8% 207503 vmstat.system.in
2088 ± 9% +34105.3% 714491 ± 34% numa-meminfo.node0.Active
1982 ± 9% +35929.8% 714170 ± 34% numa-meminfo.node0.Active(anon)
607865 ± 2% +118.4% 1327776 ± 18% numa-meminfo.node0.FilePages
1359425 ± 3% +65.5% 2249289 ± 12% numa-meminfo.node0.MemUsed
14202 ± 25% +5010.6% 725830 ± 33% numa-meminfo.node0.Shmem
1323966 ± 3% +50.8% 1996033 ± 12% numa-meminfo.node1.MemUsed
130438 ± 3% +876.0% 1273092 ± 3% meminfo.Active
130230 ± 3% +877.2% 1272567 ± 3% meminfo.Active(anon)
1334231 +85.6% 2476409 meminfo.Cached
971466 +123.4% 2170088 ± 2% meminfo.Committed_AS
2683686 +58.2% 4245240 meminfo.Memused
151555 ± 2% +751.5% 1290535 ± 3% meminfo.Shmem
3145259 +35.0% 4245265 meminfo.max_used_kB
8674 ± 3% +9.2% 9469 ± 2% slabinfo.kmalloc-256.active_objs
8722 ± 3% +9.7% 9564 ± 2% slabinfo.kmalloc-256.num_objs
959.83 ± 8% +15.3% 1107 ± 5% slabinfo.mnt_cache.active_objs
959.83 ± 8% +15.3% 1107 ± 5% slabinfo.mnt_cache.num_objs
14118 -36.3% 8989 slabinfo.proc_inode_cache.active_objs
293.50 -36.4% 186.67 slabinfo.proc_inode_cache.active_slabs
14118 -36.3% 8989 slabinfo.proc_inode_cache.num_objs
293.50 -36.4% 186.67 slabinfo.proc_inode_cache.num_slabs
22359 +19.9% 26804 slabinfo.radix_tree_node.active_objs
22359 +19.9% 26804 slabinfo.radix_tree_node.num_objs
32514 ± 3% +877.6% 317847 ± 3% proc-vmstat.nr_active_anon
65254 +8.2% 70573 proc-vmstat.nr_anon_pages
333544 +85.5% 618811 proc-vmstat.nr_file_pages
70493 +6.3% 74935 proc-vmstat.nr_inactive_anon
10402 -7.8% 9585 proc-vmstat.nr_mapped
1551 +7.4% 1665 proc-vmstat.nr_page_table_pages
37874 ± 2% +751.1% 322340 ± 3% proc-vmstat.nr_shmem
24876 -2.6% 24238 proc-vmstat.nr_slab_reclaimable
48365 +1.5% 49089 proc-vmstat.nr_slab_unreclaimable
32514 ± 3% +877.6% 317847 ± 3% proc-vmstat.nr_zone_active_anon
70493 +6.3% 74935 proc-vmstat.nr_zone_inactive_anon
7610 ± 40% -41.4% 4459 ± 53% proc-vmstat.numa_hint_faults
4133 ± 8% -48.5% 2129 ± 9% proc-vmstat.numa_hint_faults_local
912421 +80.1% 1643260 ± 2% proc-vmstat.numa_hit
818742 +89.3% 1549535 ± 2% proc-vmstat.numa_local
85725 ± 18% -59.3% 34864 ± 30% proc-vmstat.numa_pte_updates
49420 ± 2% -13.4% 42800 ± 3% proc-vmstat.pgactivate
969571 +91.4% 1856181 ± 4% proc-vmstat.pgalloc_normal
941570 -7.5% 871253 proc-vmstat.pgfault
898227 +14.9% 1031957 ± 6% proc-vmstat.pgfree
19059 ± 33% +287.5% 73861 ±128% softirqs.CPU100.RCU
25846 ± 36% -65.1% 9030 ± 79% softirqs.CPU16.SCHED
25015 ± 30% +51.9% 38003 ± 17% softirqs.CPU19.RCU
22651 ± 15% +36.8% 30997 ± 15% softirqs.CPU2.RCU
23715 ± 22% +69.0% 40085 ± 9% softirqs.CPU24.RCU
24337 ± 28% +74.8% 42545 ± 47% softirqs.CPU31.RCU
20850 ± 19% +66.3% 34668 ± 20% softirqs.CPU32.RCU
22721 ± 21% +57.4% 35756 ± 17% softirqs.CPU40.RCU
17821 ± 27% +87.9% 33480 ± 12% softirqs.CPU46.RCU
21298 ± 22% +49.7% 31874 ± 15% softirqs.CPU50.RCU
18539 ± 21% +58.8% 29435 ± 21% softirqs.CPU55.RCU
19198 ± 20% +48.3% 28475 ± 19% softirqs.CPU57.RCU
17888 ± 25% +61.7% 28930 ± 25% softirqs.CPU60.RCU
19030 ± 25% +66.1% 31604 ± 24% softirqs.CPU61.RCU
18768 ± 23% +60.9% 30204 ± 24% softirqs.CPU63.RCU
20254 ± 30% +431.3% 107621 ±162% softirqs.CPU64.RCU
19895 ± 20% +47.9% 29429 ± 20% softirqs.CPU65.RCU
18800 ± 23% +65.0% 31020 ± 21% softirqs.CPU66.RCU
19145 ± 22% +63.5% 31294 ± 25% softirqs.CPU74.RCU
18727 ± 25% +367.0% 87458 ±151% softirqs.CPU77.RCU
22560 ± 22% +51.6% 34208 ± 14% softirqs.CPU81.RCU
17142 ± 25% +65.2% 28315 ± 23% softirqs.CPU85.RCU
18137 ± 23% +58.5% 28747 ± 28% softirqs.CPU89.RCU
17197 ± 25% +83.1% 31484 ± 34% softirqs.CPU91.RCU
18710 ± 22% +61.9% 30283 ± 19% softirqs.CPU94.RCU
18735 ± 20% +33.9% 25080 ± 21% softirqs.CPU95.RCU
17769 ± 51% -60.3% 7047 ± 44% softirqs.NET_RX
2222678 ± 29% +80.1% 4003205 ± 12% softirqs.RCU
35500 ± 12% +44.8% 51399 ± 3% softirqs.TIMER
67.45 ± 9% -67.4 0.00 perf-profile.calltrace.cycles-pp.brk
48.31 ± 9% -48.3 0.00 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.brk
32.16 ± 20% -32.2 0.00 perf-profile.calltrace.cycles-pp.secondary_startup_64_no_verify
32.09 ± 20% -32.1 0.00 perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry
31.51 ± 20% -31.5 0.00 perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
31.51 ± 20% -31.5 0.00 perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
31.51 ± 20% -31.5 0.00 perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64_no_verify
31.51 ± 20% -31.5 0.00 perf-profile.calltrace.cycles-pp.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
31.51 ± 20% -31.5 0.00 perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary
31.32 ± 9% -31.3 0.00 perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.brk
30.75 ± 9% -30.8 0.00 perf-profile.calltrace.cycles-pp.__x64_sys_brk.do_syscall_64.entry_SYSCALL_64_after_hwframe.brk
16.21 ± 9% -16.2 0.00 perf-profile.calltrace.cycles-pp.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe.brk
15.04 ± 9% -15.0 0.00 perf-profile.calltrace.cycles-pp.do_brk_flags.__x64_sys_brk.do_syscall_64.entry_SYSCALL_64_after_hwframe.brk
11.50 ± 9% -11.5 0.00 perf-profile.calltrace.cycles-pp.__do_munmap.__x64_sys_brk.do_syscall_64.entry_SYSCALL_64_after_hwframe.brk
8.87 ± 9% -8.9 0.00 perf-profile.calltrace.cycles-pp.__entry_text_start.brk
7.98 ± 9% -8.0 0.00 perf-profile.calltrace.cycles-pp.syscall_return_via_sysret.brk
5.55 ± 9% -5.6 0.00 perf-profile.calltrace.cycles-pp.perf_event_mmap.do_brk_flags.__x64_sys_brk.do_syscall_64.entry_SYSCALL_64_after_hwframe
5.21 ± 9% -5.2 0.00 perf-profile.calltrace.cycles-pp.unmap_region.__do_munmap.__x64_sys_brk.do_syscall_64.entry_SYSCALL_64_after_hwframe
67.49 ± 9% -67.5 0.00 perf-profile.children.cycles-pp.brk
48.47 ± 9% -48.5 0.00 perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
32.16 ± 20% -32.2 0.00 perf-profile.children.cycles-pp.secondary_startup_64_no_verify
32.16 ± 20% -32.2 0.00 perf-profile.children.cycles-pp.cpu_startup_entry
32.16 ± 20% -32.2 0.00 perf-profile.children.cycles-pp.do_idle
32.16 ± 20% -32.2 0.00 perf-profile.children.cycles-pp.cpuidle_enter
32.16 ± 20% -32.2 0.00 perf-profile.children.cycles-pp.cpuidle_enter_state
32.16 ± 20% -32.2 0.00 perf-profile.children.cycles-pp.intel_idle
31.51 ± 20% -31.5 0.00 perf-profile.children.cycles-pp.start_secondary
31.43 ± 9% -31.4 0.00 perf-profile.children.cycles-pp.do_syscall_64
30.84 ± 9% -30.8 0.00 perf-profile.children.cycles-pp.__x64_sys_brk
16.29 ± 9% -16.3 0.00 perf-profile.children.cycles-pp.syscall_exit_to_user_mode
15.10 ± 9% -15.1 0.00 perf-profile.children.cycles-pp.do_brk_flags
11.55 ± 9% -11.6 0.00 perf-profile.children.cycles-pp.__do_munmap
9.08 ± 10% -9.1 0.00 perf-profile.children.cycles-pp.syscall_return_via_sysret
8.47 ± 9% -8.5 0.00 perf-profile.children.cycles-pp.__entry_text_start
5.63 ± 9% -5.6 0.00 perf-profile.children.cycles-pp.perf_event_mmap
5.29 ± 9% -5.3 0.00 perf-profile.children.cycles-pp.unmap_region
32.16 ± 20% -32.2 0.00 perf-profile.self.cycles-pp.intel_idle
15.96 ± 9% -16.0 0.00 perf-profile.self.cycles-pp.syscall_exit_to_user_mode
9.05 ± 9% -9.0 0.00 perf-profile.self.cycles-pp.syscall_return_via_sysret
7.34 ± 9% -7.3 0.00 perf-profile.self.cycles-pp.__entry_text_start
0.06 ± 23% +849.3% 0.52 ± 27% perf-stat.i.MPKI
2.817e+10 -7.5% 2.605e+10 perf-stat.i.branch-instructions
0.70 +0.1 0.79 perf-stat.i.branch-miss-rate%
1.965e+08 +4.2% 2.047e+08 perf-stat.i.branch-misses
1348212 ± 93% +927.5% 13853011 ± 20% perf-stat.i.cache-misses
6337510 ± 28% +909.9% 64003716 ± 27% perf-stat.i.cache-references
1541 ± 5% +31.9% 2033 ± 4% perf-stat.i.context-switches
1.08 +12.9% 1.22 perf-stat.i.cpi
1.448e+11 +4.4% 1.512e+11 perf-stat.i.cpu-cycles
132.05 -4.9% 125.55 ± 4% perf-stat.i.cpu-migrations
224930 ± 19% -94.6% 12223 ± 28% perf-stat.i.cycles-between-cache-misses
0.17 -0.0 0.16 perf-stat.i.dTLB-load-miss-rate%
66339784 -14.7% 56566077 perf-stat.i.dTLB-load-misses
3.941e+10 -8.0% 3.624e+10 perf-stat.i.dTLB-loads
52761 -11.0% 46938 perf-stat.i.dTLB-store-misses
2.163e+10 -7.8% 1.994e+10 perf-stat.i.dTLB-stores
68108407 -15.1% 57828873 ± 2% perf-stat.i.iTLB-load-misses
1.34e+11 -7.6% 1.239e+11 perf-stat.i.instructions
1997 +8.9% 2174 ± 2% perf-stat.i.instructions-per-iTLB-miss
0.93 -11.4% 0.82 perf-stat.i.ipc
0.72 -94.5% 0.04 ± 33% perf-stat.i.major-faults
1.39 +4.4% 1.45 perf-stat.i.metric.GHz
857.73 -7.7% 791.38 perf-stat.i.metric.M/sec
3021 -7.7% 2788 perf-stat.i.minor-faults
89.68 -3.9 85.77 perf-stat.i.node-load-miss-rate%
315756 ±104% +546.6% 2041712 ± 20% perf-stat.i.node-load-misses
55314 ±104% +530.4% 348680 ± 25% perf-stat.i.node-loads
34490 ± 45% +2319.4% 834464 ± 31% perf-stat.i.node-store-misses
8436 ± 5% +145.9% 20748 ± 6% perf-stat.i.node-stores
3021 -7.7% 2788 perf-stat.i.page-faults
0.05 ± 28% +987.0% 0.52 ± 27% perf-stat.overall.MPKI
0.70 +0.1 0.79 perf-stat.overall.branch-miss-rate%
1.08 +13.0% 1.22 perf-stat.overall.cpi
166572 ± 41% -93.1% 11460 ± 23% perf-stat.overall.cycles-between-cache-misses
0.17 -0.0 0.16 perf-stat.overall.dTLB-load-miss-rate%
0.00 -0.0 0.00 perf-stat.overall.dTLB-store-miss-rate%
1968 +8.9% 2143 perf-stat.overall.instructions-per-iTLB-miss
0.93 -11.5% 0.82 perf-stat.overall.ipc
78.34 ± 6% +18.6 96.97 ± 2% perf-stat.overall.node-store-miss-rate%
606198 +9.2% 661776 perf-stat.overall.path-length
2.807e+10 -7.5% 2.597e+10 perf-stat.ps.branch-instructions
1.959e+08 +4.1% 2.04e+08 perf-stat.ps.branch-misses
1346058 ± 93% +925.6% 13805032 ± 20% perf-stat.ps.cache-misses
6345091 ± 28% +905.4% 63791617 ± 27% perf-stat.ps.cache-references
1536 ± 5% +31.9% 2026 ± 4% perf-stat.ps.context-switches
1.443e+11 +4.4% 1.507e+11 perf-stat.ps.cpu-cycles
132.04 -5.2% 125.17 ± 4% perf-stat.ps.cpu-migrations
66096851 -14.7% 56375461 perf-stat.ps.dTLB-load-misses
3.927e+10 -8.0% 3.612e+10 perf-stat.ps.dTLB-loads
52611 -11.1% 46790 perf-stat.ps.dTLB-store-misses
2.155e+10 -7.8% 1.987e+10 perf-stat.ps.dTLB-stores
67859804 -15.1% 57635424 ± 2% perf-stat.ps.iTLB-load-misses
1.336e+11 -7.5% 1.235e+11 perf-stat.ps.instructions
0.72 -94.4% 0.04 ± 33% perf-stat.ps.major-faults
3011 -7.7% 2779 perf-stat.ps.minor-faults
314425 ±104% +547.1% 2034667 ± 20% perf-stat.ps.node-load-misses
55599 ±103% +525.0% 347495 ± 25% perf-stat.ps.node-loads
34364 ± 45% +2319.6% 831507 ± 31% perf-stat.ps.node-store-misses
8502 ± 5% +143.5% 20703 ± 6% perf-stat.ps.node-stores
3012 -7.7% 2779 perf-stat.ps.page-faults
4.033e+13 -7.5% 3.731e+13 perf-stat.total.instructions
0.01 ± 9% -100.0% 0.00 perf-sched.sch_delay.avg.ms.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown]
0.01 ± 11% -100.0% 0.00 perf-sched.sch_delay.avg.ms.devkmsg_read.vfs_read.ksys_read.do_syscall_64
0.01 ± 6% -100.0% 0.00 perf-sched.sch_delay.avg.ms.do_nanosleep.hrtimer_nanosleep.__x64_sys_nanosleep.do_syscall_64
0.01 ± 13% -100.0% 0.00 perf-sched.sch_delay.avg.ms.do_syslog.part.0.kmsg_read.vfs_read
0.01 ± 5% -100.0% 0.00 perf-sched.sch_delay.avg.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
0.01 ± 3% -100.0% 0.00 perf-sched.sch_delay.avg.ms.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64
0.01 ± 20% -100.0% 0.00 perf-sched.sch_delay.avg.ms.futex_wait_queue_me.futex_wait.do_futex.__x64_sys_futex
0.00 -100.0% 0.00 perf-sched.sch_delay.avg.ms.pipe_read.new_sync_read.vfs_read.ksys_read
0.01 ± 7% -100.0% 0.00 perf-sched.sch_delay.avg.ms.rcu_gp_kthread.kthread.ret_from_fork
0.01 ± 15% -100.0% 0.00 perf-sched.sch_delay.avg.ms.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait.__x64_sys_epoll_wait
0.01 ± 21% -100.0% 0.00 perf-sched.sch_delay.avg.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_select
0.00 ± 28% -100.0% 0.00 perf-sched.sch_delay.avg.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_sys_poll
0.01 -100.0% 0.00 perf-sched.sch_delay.avg.ms.schedule_timeout.kcompactd.kthread.ret_from_fork
0.01 -100.0% 0.00 perf-sched.sch_delay.avg.ms.schedule_timeout.rcu_gp_kthread.kthread.ret_from_fork
0.01 ± 11% -100.0% 0.00 perf-sched.sch_delay.avg.ms.schedule_timeout.wait_for_completion.__flush_work.lru_add_drain_all
0.01 -100.0% 0.00 perf-sched.sch_delay.avg.ms.smpboot_thread_fn.kthread.ret_from_fork
0.01 ± 6% -100.0% 0.00 perf-sched.sch_delay.avg.ms.wait_for_partner.fifo_open.do_dentry_open.do_open.isra
0.02 ± 16% -100.0% 0.00 perf-sched.sch_delay.avg.ms.worker_thread.kthread.ret_from_fork
0.01 ± 53% -100.0% 0.00 perf-sched.sch_delay.max.ms.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown]
0.01 ± 23% -100.0% 0.00 perf-sched.sch_delay.max.ms.devkmsg_read.vfs_read.ksys_read.do_syscall_64
0.01 ± 14% -100.0% 0.00 perf-sched.sch_delay.max.ms.do_nanosleep.hrtimer_nanosleep.__x64_sys_nanosleep.do_syscall_64
0.02 ± 27% -100.0% 0.00 perf-sched.sch_delay.max.ms.do_syslog.part.0.kmsg_read.vfs_read
0.03 ± 64% -100.0% 0.00 perf-sched.sch_delay.max.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
0.02 ± 12% -100.0% 0.00 perf-sched.sch_delay.max.ms.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64
0.01 ± 19% -100.0% 0.00 perf-sched.sch_delay.max.ms.exit_to_user_mode_prepare.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe.[unknown]
0.01 ± 18% -100.0% 0.00 perf-sched.sch_delay.max.ms.futex_wait_queue_me.futex_wait.do_futex.__x64_sys_futex
0.03 ± 10% -100.0% 0.00 perf-sched.sch_delay.max.ms.pipe_read.new_sync_read.vfs_read.ksys_read
0.02 ± 15% -100.0% 0.00 perf-sched.sch_delay.max.ms.preempt_schedule_common.__cond_resched.stop_one_cpu.affine_move_task.__set_cpus_allowed_ptr
0.01 ± 19% -100.0% 0.00 perf-sched.sch_delay.max.ms.rcu_gp_kthread.kthread.ret_from_fork
0.03 ± 78% -100.0% 0.00 perf-sched.sch_delay.max.ms.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait.__x64_sys_epoll_wait
0.01 ± 29% -100.0% 0.00 perf-sched.sch_delay.max.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_select
0.01 ± 51% -100.0% 0.00 perf-sched.sch_delay.max.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_sys_poll
0.01 ± 42% -100.0% 0.00 perf-sched.sch_delay.max.ms.schedule_timeout.kcompactd.kthread.ret_from_fork
0.02 ± 26% -100.0% 0.00 perf-sched.sch_delay.max.ms.schedule_timeout.rcu_gp_kthread.kthread.ret_from_fork
0.01 ± 11% -100.0% 0.00 perf-sched.sch_delay.max.ms.schedule_timeout.wait_for_completion.__flush_work.lru_add_drain_all
0.02 ± 22% -100.0% 0.00 perf-sched.sch_delay.max.ms.smpboot_thread_fn.kthread.ret_from_fork
0.02 ± 66% -100.0% 0.00 perf-sched.sch_delay.max.ms.wait_for_partner.fifo_open.do_dentry_open.do_open.isra
5.62 ± 2% -100.0% 0.00 perf-sched.sch_delay.max.ms.worker_thread.kthread.ret_from_fork
0.01 -100.0% 0.00 perf-sched.total_sch_delay.average.ms
5.62 ± 2% -100.0% 0.00 perf-sched.total_sch_delay.max.ms
195.80 ± 6% -100.0% 0.00 perf-sched.total_wait_and_delay.average.ms
8759 ± 6% -100.0% 0.00 perf-sched.total_wait_and_delay.count.ms
8233 ± 6% -100.0% 0.00 perf-sched.total_wait_and_delay.max.ms
195.80 ± 6% -100.0% 0.00 perf-sched.total_wait_time.average.ms
8233 ± 6% -100.0% 0.00 perf-sched.total_wait_time.max.ms
899.53 -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown]
1969 -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.devkmsg_read.vfs_read.ksys_read.do_syscall_64
809.06 ± 4% -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.do_nanosleep.hrtimer_nanosleep.__x64_sys_nanosleep.do_syscall_64
1969 -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.do_syslog.part.0.kmsg_read.vfs_read
269.47 ± 4% -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
1.43 -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64
0.04 ± 18% -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt.[unknown]
0.03 ± 5% -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.exit_to_user_mode_prepare.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe.[unknown]
64.05 -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.pipe_read.new_sync_read.vfs_read.ksys_read
0.04 ± 30% -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.preempt_schedule_common.__cond_resched.down_write_killable.__x64_sys_brk.do_syscall_64
0.04 ± 16% -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.preempt_schedule_common.__cond_resched.kmem_cache_alloc.vm_area_alloc.do_brk_flags
7.87 ± 6% -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.preempt_schedule_common.__cond_resched.stop_one_cpu.affine_move_task.__set_cpus_allowed_ptr
0.03 ± 6% -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.preempt_schedule_common.__cond_resched.unmap_vmas.unmap_region.__do_munmap
2.56 ± 8% -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.rcu_gp_kthread.kthread.ret_from_fork
595.03 ± 12% -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait.__x64_sys_epoll_wait
326.53 ± 27% -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_sys_poll
478.79 -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.schedule_timeout.kcompactd.kthread.ret_from_fork
8.32 ± 25% -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.schedule_timeout.rcu_gp_kthread.kthread.ret_from_fork
676.61 ± 2% -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.smpboot_thread_fn.kthread.ret_from_fork
0.01 ± 6% -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.wait_for_partner.fifo_open.do_dentry_open.do_open.isra
623.29 ± 4% -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.worker_thread.kthread.ret_from_fork
10.00 -100.0% 0.00 perf-sched.wait_and_delay.count.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown]
4.00 -100.0% 0.00 perf-sched.wait_and_delay.count.devkmsg_read.vfs_read.ksys_read.do_syscall_64
22.33 ± 4% -100.0% 0.00 perf-sched.wait_and_delay.count.do_nanosleep.hrtimer_nanosleep.__x64_sys_nanosleep.do_syscall_64
4.00 -100.0% 0.00 perf-sched.wait_and_delay.count.do_syslog.part.0.kmsg_read.vfs_read
247.00 -100.0% 0.00 perf-sched.wait_and_delay.count.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
246.00 -100.0% 0.00 perf-sched.wait_and_delay.count.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64
118.83 ± 38% -100.0% 0.00 perf-sched.wait_and_delay.count.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt.[unknown]
342.83 ± 14% -100.0% 0.00 perf-sched.wait_and_delay.count.exit_to_user_mode_prepare.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe.[unknown]
2518 -100.0% 0.00 perf-sched.wait_and_delay.count.pipe_read.new_sync_read.vfs_read.ksys_read
115.33 ± 6% -100.0% 0.00 perf-sched.wait_and_delay.count.preempt_schedule_common.__cond_resched.down_write_killable.__x64_sys_brk.do_syscall_64
61.83 ± 7% -100.0% 0.00 perf-sched.wait_and_delay.count.preempt_schedule_common.__cond_resched.kmem_cache_alloc.vm_area_alloc.do_brk_flags
1248 -100.0% 0.00 perf-sched.wait_and_delay.count.preempt_schedule_common.__cond_resched.stop_one_cpu.affine_move_task.__set_cpus_allowed_ptr
54.17 ± 12% -100.0% 0.00 perf-sched.wait_and_delay.count.preempt_schedule_common.__cond_resched.unmap_vmas.unmap_region.__do_munmap
143.67 ± 55% -100.0% 0.00 perf-sched.wait_and_delay.count.rcu_gp_kthread.kthread.ret_from_fork
23.67 ± 45% -100.0% 0.00 perf-sched.wait_and_delay.count.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait.__x64_sys_epoll_wait
83.83 ± 25% -100.0% 0.00 perf-sched.wait_and_delay.count.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_sys_poll
40.00 -100.0% 0.00 perf-sched.wait_and_delay.count.schedule_timeout.kcompactd.kthread.ret_from_fork
1272 ± 37% -100.0% 0.00 perf-sched.wait_and_delay.count.schedule_timeout.rcu_gp_kthread.kthread.ret_from_fork
1464 -100.0% 0.00 perf-sched.wait_and_delay.count.smpboot_thread_fn.kthread.ret_from_fork
72.00 -100.0% 0.00 perf-sched.wait_and_delay.count.wait_for_partner.fifo_open.do_dentry_open.do_open.isra
575.00 ± 8% -100.0% 0.00 perf-sched.wait_and_delay.count.worker_thread.kthread.ret_from_fork
999.50 -100.0% 0.00 perf-sched.wait_and_delay.max.ms.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown]
7815 -100.0% 0.00 perf-sched.wait_and_delay.max.ms.devkmsg_read.vfs_read.ksys_read.do_syscall_64
1000 -100.0% 0.00 perf-sched.wait_and_delay.max.ms.do_nanosleep.hrtimer_nanosleep.__x64_sys_nanosleep.do_syscall_64
7815 -100.0% 0.00 perf-sched.wait_and_delay.max.ms.do_syslog.part.0.kmsg_read.vfs_read
3095 ± 56% -100.0% 0.00 perf-sched.wait_and_delay.max.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
33.25 ± 2% -100.0% 0.00 perf-sched.wait_and_delay.max.ms.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64
0.88 ±146% -100.0% 0.00 perf-sched.wait_and_delay.max.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt.[unknown]
0.19 ±106% -100.0% 0.00 perf-sched.wait_and_delay.max.ms.exit_to_user_mode_prepare.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe.[unknown]
7819 -100.0% 0.00 perf-sched.wait_and_delay.max.ms.pipe_read.new_sync_read.vfs_read.ksys_read
0.78 ±149% -100.0% 0.00 perf-sched.wait_and_delay.max.ms.preempt_schedule_common.__cond_resched.down_write_killable.__x64_sys_brk.do_syscall_64
0.25 ±144% -100.0% 0.00 perf-sched.wait_and_delay.max.ms.preempt_schedule_common.__cond_resched.kmem_cache_alloc.vm_area_alloc.do_brk_flags
1172 ± 32% -100.0% 0.00 perf-sched.wait_and_delay.max.ms.preempt_schedule_common.__cond_resched.stop_one_cpu.affine_move_task.__set_cpus_allowed_ptr
0.06 ± 27% -100.0% 0.00 perf-sched.wait_and_delay.max.ms.preempt_schedule_common.__cond_resched.unmap_vmas.unmap_region.__do_munmap
5.00 -100.0% 0.00 perf-sched.wait_and_delay.max.ms.rcu_gp_kthread.kthread.ret_from_fork
7407 -100.0% 0.00 perf-sched.wait_and_delay.max.ms.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait.__x64_sys_epoll_wait
5250 ± 36% -100.0% 0.00 perf-sched.wait_and_delay.max.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_sys_poll
504.84 -100.0% 0.00 perf-sched.wait_and_delay.max.ms.schedule_timeout.kcompactd.kthread.ret_from_fork
243.33 ± 50% -100.0% 0.00 perf-sched.wait_and_delay.max.ms.schedule_timeout.rcu_gp_kthread.kthread.ret_from_fork
2729 ± 47% -100.0% 0.00 perf-sched.wait_and_delay.max.ms.smpboot_thread_fn.kthread.ret_from_fork
0.03 ± 63% -100.0% 0.00 perf-sched.wait_and_delay.max.ms.wait_for_partner.fifo_open.do_dentry_open.do_open.isra
7699 ± 14% -100.0% 0.00 perf-sched.wait_and_delay.max.ms.worker_thread.kthread.ret_from_fork
899.53 -100.0% 0.00 perf-sched.wait_time.avg.ms.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown]
1969 -100.0% 0.00 perf-sched.wait_time.avg.ms.devkmsg_read.vfs_read.ksys_read.do_syscall_64
809.06 ± 4% -100.0% 0.00 perf-sched.wait_time.avg.ms.do_nanosleep.hrtimer_nanosleep.__x64_sys_nanosleep.do_syscall_64
1969 -100.0% 0.00 perf-sched.wait_time.avg.ms.do_syslog.part.0.kmsg_read.vfs_read
269.46 ± 4% -100.0% 0.00 perf-sched.wait_time.avg.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
1.42 -100.0% 0.00 perf-sched.wait_time.avg.ms.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64
0.04 ± 18% -100.0% 0.00 perf-sched.wait_time.avg.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt.[unknown]
0.03 ± 29% -100.0% 0.00 perf-sched.wait_time.avg.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_reschedule_ipi.[unknown]
0.03 ± 4% -100.0% 0.00 perf-sched.wait_time.avg.ms.exit_to_user_mode_prepare.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe.[unknown]
28.97 ± 27% -100.0% 0.00 perf-sched.wait_time.avg.ms.futex_wait_queue_me.futex_wait.do_futex.__x64_sys_futex
64.05 -100.0% 0.00 perf-sched.wait_time.avg.ms.pipe_read.new_sync_read.vfs_read.ksys_read
0.04 ± 30% -100.0% 0.00 perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.down_write_killable.__x64_sys_brk.do_syscall_64
0.04 ± 16% -100.0% 0.00 perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.kmem_cache_alloc.vm_area_alloc.do_brk_flags
0.03 ± 8% -100.0% 0.00 perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.remove_vma.__do_munmap.__x64_sys_brk
7.87 ± 6% -100.0% 0.00 perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.stop_one_cpu.affine_move_task.__set_cpus_allowed_ptr
0.04 ± 6% -100.0% 0.00 perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.unmap_page_range.unmap_vmas.unmap_region
0.03 ± 6% -100.0% 0.00 perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.unmap_vmas.unmap_region.__do_munmap
2.56 ± 8% -100.0% 0.00 perf-sched.wait_time.avg.ms.rcu_gp_kthread.kthread.ret_from_fork
595.02 ± 12% -100.0% 0.00 perf-sched.wait_time.avg.ms.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait.__x64_sys_epoll_wait
326.53 ± 27% -100.0% 0.00 perf-sched.wait_time.avg.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_sys_poll
478.79 -100.0% 0.00 perf-sched.wait_time.avg.ms.schedule_timeout.kcompactd.kthread.ret_from_fork
0.01 ± 63% -100.0% 0.00 perf-sched.wait_time.avg.ms.schedule_timeout.khugepaged.kthread.ret_from_fork
8.31 ± 25% -100.0% 0.00 perf-sched.wait_time.avg.ms.schedule_timeout.rcu_gp_kthread.kthread.ret_from_fork
676.60 ± 2% -100.0% 0.00 perf-sched.wait_time.avg.ms.smpboot_thread_fn.kthread.ret_from_fork
623.27 ± 4% -100.0% 0.00 perf-sched.wait_time.avg.ms.worker_thread.kthread.ret_from_fork
999.50 -100.0% 0.00 perf-sched.wait_time.max.ms.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown]
7815 -100.0% 0.00 perf-sched.wait_time.max.ms.devkmsg_read.vfs_read.ksys_read.do_syscall_64
1000 -100.0% 0.00 perf-sched.wait_time.max.ms.do_nanosleep.hrtimer_nanosleep.__x64_sys_nanosleep.do_syscall_64
7815 -100.0% 0.00 perf-sched.wait_time.max.ms.do_syslog.part.0.kmsg_read.vfs_read
3095 ± 56% -100.0% 0.00 perf-sched.wait_time.max.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
33.24 ± 2% -100.0% 0.00 perf-sched.wait_time.max.ms.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64
0.88 ±146% -100.0% 0.00 perf-sched.wait_time.max.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt.[unknown]
0.10 ±106% -100.0% 0.00 perf-sched.wait_time.max.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_reschedule_ipi.[unknown]
0.19 ±106% -100.0% 0.00 perf-sched.wait_time.max.ms.exit_to_user_mode_prepare.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe.[unknown]
57.94 ± 27% -100.0% 0.00 perf-sched.wait_time.max.ms.futex_wait_queue_me.futex_wait.do_futex.__x64_sys_futex
7819 -100.0% 0.00 perf-sched.wait_time.max.ms.pipe_read.new_sync_read.vfs_read.ksys_read
0.78 ±149% -100.0% 0.00 perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.down_write_killable.__x64_sys_brk.do_syscall_64
0.25 ±144% -100.0% 0.00 perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.kmem_cache_alloc.vm_area_alloc.do_brk_flags
0.05 -100.0% 0.00 perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.remove_vma.__do_munmap.__x64_sys_brk
1172 ± 32% -100.0% 0.00 perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.stop_one_cpu.affine_move_task.__set_cpus_allowed_ptr
0.09 ± 37% -100.0% 0.00 perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.unmap_page_range.unmap_vmas.unmap_region
0.06 ± 27% -100.0% 0.00 perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.unmap_vmas.unmap_region.__do_munmap
4.99 -100.0% 0.00 perf-sched.wait_time.max.ms.rcu_gp_kthread.kthread.ret_from_fork
7407 -100.0% 0.00 perf-sched.wait_time.max.ms.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait.__x64_sys_epoll_wait
5250 ± 36% -100.0% 0.00 perf-sched.wait_time.max.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_sys_poll
504.83 -100.0% 0.00 perf-sched.wait_time.max.ms.schedule_timeout.kcompactd.kthread.ret_from_fork
0.01 ± 63% -100.0% 0.00 perf-sched.wait_time.max.ms.schedule_timeout.khugepaged.kthread.ret_from_fork
243.32 ± 50% -100.0% 0.00 perf-sched.wait_time.max.ms.schedule_timeout.rcu_gp_kthread.kthread.ret_from_fork
2729 ± 47% -100.0% 0.00 perf-sched.wait_time.max.ms.smpboot_thread_fn.kthread.ret_from_fork
7699 ± 14% -100.0% 0.00 perf-sched.wait_time.max.ms.worker_thread.kthread.ret_from_fork
76758 ± 2% -8.6% 70119 ± 2% interrupts.CAL:Function_call_interrupts
4021 ± 30% -100.0% 0.00 interrupts.CPU0.NMI:Non-maskable_interrupts
4021 ± 30% -100.0% 0.00 interrupts.CPU0.PMI:Performance_monitoring_interrupts
5976 ± 37% -100.0% 0.00 interrupts.CPU1.NMI:Non-maskable_interrupts
5976 ± 37% -100.0% 0.00 interrupts.CPU1.PMI:Performance_monitoring_interrupts
5375 ± 45% -100.0% 0.00 interrupts.CPU10.NMI:Non-maskable_interrupts
5375 ± 45% -100.0% 0.00 interrupts.CPU10.PMI:Performance_monitoring_interrupts
4174 ± 47% -100.0% 0.00 interrupts.CPU100.NMI:Non-maskable_interrupts
4174 ± 47% -100.0% 0.00 interrupts.CPU100.PMI:Performance_monitoring_interrupts
5327 ± 42% -100.0% 0.00 interrupts.CPU101.NMI:Non-maskable_interrupts
5327 ± 42% -100.0% 0.00 interrupts.CPU101.PMI:Performance_monitoring_interrupts
4581 ± 38% -100.0% 0.00 interrupts.CPU102.NMI:Non-maskable_interrupts
4581 ± 38% -100.0% 0.00 interrupts.CPU102.PMI:Performance_monitoring_interrupts
5337 ± 26% -100.0% 0.00 interrupts.CPU103.NMI:Non-maskable_interrupts
5337 ± 26% -100.0% 0.00 interrupts.CPU103.PMI:Performance_monitoring_interrupts
3615 ± 45% -100.0% 0.00 interrupts.CPU11.NMI:Non-maskable_interrupts
3615 ± 45% -100.0% 0.00 interrupts.CPU11.PMI:Performance_monitoring_interrupts
4389 ± 57% -100.0% 0.00 interrupts.CPU12.NMI:Non-maskable_interrupts
4389 ± 57% -100.0% 0.00 interrupts.CPU12.PMI:Performance_monitoring_interrupts
5230 ± 44% -100.0% 0.00 interrupts.CPU13.NMI:Non-maskable_interrupts
5230 ± 44% -100.0% 0.00 interrupts.CPU13.PMI:Performance_monitoring_interrupts
4857 ± 37% -100.0% 0.00 interrupts.CPU14.NMI:Non-maskable_interrupts
4857 ± 37% -100.0% 0.00 interrupts.CPU14.PMI:Performance_monitoring_interrupts
4571 ± 51% -100.0% 0.00 interrupts.CPU15.NMI:Non-maskable_interrupts
4571 ± 51% -100.0% 0.00 interrupts.CPU15.PMI:Performance_monitoring_interrupts
769.50 ± 20% -28.1% 553.00 ± 8% interrupts.CPU16.CAL:Function_call_interrupts
3950 ± 54% -100.0% 0.00 interrupts.CPU16.NMI:Non-maskable_interrupts
3950 ± 54% -100.0% 0.00 interrupts.CPU16.PMI:Performance_monitoring_interrupts
4485 ± 30% -100.0% 0.00 interrupts.CPU17.NMI:Non-maskable_interrupts
4485 ± 30% -100.0% 0.00 interrupts.CPU17.PMI:Performance_monitoring_interrupts
5139 ± 35% -100.0% 0.00 interrupts.CPU18.NMI:Non-maskable_interrupts
5139 ± 35% -100.0% 0.00 interrupts.CPU18.PMI:Performance_monitoring_interrupts
5525 ± 45% -100.0% 0.00 interrupts.CPU19.NMI:Non-maskable_interrupts
5525 ± 45% -100.0% 0.00 interrupts.CPU19.PMI:Performance_monitoring_interrupts
6327 ± 30% -100.0% 0.00 interrupts.CPU2.NMI:Non-maskable_interrupts
6327 ± 30% -100.0% 0.00 interrupts.CPU2.PMI:Performance_monitoring_interrupts
732.67 ± 24% -21.5% 575.17 ± 12% interrupts.CPU20.CAL:Function_call_interrupts
3597 ± 54% -100.0% 0.00 interrupts.CPU20.NMI:Non-maskable_interrupts
3597 ± 54% -100.0% 0.00 interrupts.CPU20.PMI:Performance_monitoring_interrupts
4856 ± 42% -100.0% 0.00 interrupts.CPU21.NMI:Non-maskable_interrupts
4856 ± 42% -100.0% 0.00 interrupts.CPU21.PMI:Performance_monitoring_interrupts
6101 ± 38% -100.0% 0.00 interrupts.CPU22.NMI:Non-maskable_interrupts
6101 ± 38% -100.0% 0.00 interrupts.CPU22.PMI:Performance_monitoring_interrupts
4139 ± 57% -100.0% 0.00 interrupts.CPU23.NMI:Non-maskable_interrupts
4139 ± 57% -100.0% 0.00 interrupts.CPU23.PMI:Performance_monitoring_interrupts
647.00 ± 14% -15.8% 544.67 ± 3% interrupts.CPU24.CAL:Function_call_interrupts
5617 ± 37% -100.0% 0.00 interrupts.CPU24.NMI:Non-maskable_interrupts
5617 ± 37% -100.0% 0.00 interrupts.CPU24.PMI:Performance_monitoring_interrupts
6282 ± 27% -100.0% 0.00 interrupts.CPU25.NMI:Non-maskable_interrupts
6282 ± 27% -100.0% 0.00 interrupts.CPU25.PMI:Performance_monitoring_interrupts
5144 ± 54% -100.0% 0.00 interrupts.CPU26.NMI:Non-maskable_interrupts
5144 ± 54% -100.0% 0.00 interrupts.CPU26.PMI:Performance_monitoring_interrupts
6256 ± 27% -100.0% 0.00 interrupts.CPU27.NMI:Non-maskable_interrupts
6256 ± 27% -100.0% 0.00 interrupts.CPU27.PMI:Performance_monitoring_interrupts
5805 ± 38% -100.0% 0.00 interrupts.CPU28.NMI:Non-maskable_interrupts
5805 ± 38% -100.0% 0.00 interrupts.CPU28.PMI:Performance_monitoring_interrupts
3944 ± 55% -100.0% 0.00 interrupts.CPU29.NMI:Non-maskable_interrupts
3944 ± 55% -100.0% 0.00 interrupts.CPU29.PMI:Performance_monitoring_interrupts
5923 ± 33% -100.0% 0.00 interrupts.CPU3.NMI:Non-maskable_interrupts
5923 ± 33% -100.0% 0.00 interrupts.CPU3.PMI:Performance_monitoring_interrupts
4233 ± 47% -100.0% 0.00 interrupts.CPU30.NMI:Non-maskable_interrupts
4233 ± 47% -100.0% 0.00 interrupts.CPU30.PMI:Performance_monitoring_interrupts
7332 ± 9% -100.0% 0.00 interrupts.CPU31.NMI:Non-maskable_interrupts
7332 ± 9% -100.0% 0.00 interrupts.CPU31.PMI:Performance_monitoring_interrupts
4973 ± 44% -100.0% 0.00 interrupts.CPU32.NMI:Non-maskable_interrupts
4973 ± 44% -100.0% 0.00 interrupts.CPU32.PMI:Performance_monitoring_interrupts
7439 ± 10% -100.0% 0.00 interrupts.CPU33.NMI:Non-maskable_interrupts
7439 ± 10% -100.0% 0.00 interrupts.CPU33.PMI:Performance_monitoring_interrupts
7900 -100.0% 0.00 interrupts.CPU34.NMI:Non-maskable_interrupts
7900 -100.0% 0.00 interrupts.CPU34.PMI:Performance_monitoring_interrupts
6148 ± 31% -100.0% 0.00 interrupts.CPU35.NMI:Non-maskable_interrupts
6148 ± 31% -100.0% 0.00 interrupts.CPU35.PMI:Performance_monitoring_interrupts
6697 ± 28% -100.0% 0.00 interrupts.CPU36.NMI:Non-maskable_interrupts
6697 ± 28% -100.0% 0.00 interrupts.CPU36.PMI:Performance_monitoring_interrupts
6696 ± 28% -100.0% 0.00 interrupts.CPU37.NMI:Non-maskable_interrupts
6696 ± 28% -100.0% 0.00 interrupts.CPU37.PMI:Performance_monitoring_interrupts
7901 -100.0% 0.00 interrupts.CPU38.NMI:Non-maskable_interrupts
7901 -100.0% 0.00 interrupts.CPU38.PMI:Performance_monitoring_interrupts
7061 ± 26% -100.0% 0.00 interrupts.CPU39.NMI:Non-maskable_interrupts
7061 ± 26% -100.0% 0.00 interrupts.CPU39.PMI:Performance_monitoring_interrupts
5894 ± 24% -100.0% 0.00 interrupts.CPU4.NMI:Non-maskable_interrupts
5894 ± 24% -100.0% 0.00 interrupts.CPU4.PMI:Performance_monitoring_interrupts
6487 ± 30% -100.0% 0.00 interrupts.CPU40.NMI:Non-maskable_interrupts
6487 ± 30% -100.0% 0.00 interrupts.CPU40.PMI:Performance_monitoring_interrupts
6538 ± 29% -100.0% 0.00 interrupts.CPU41.NMI:Non-maskable_interrupts
6538 ± 29% -100.0% 0.00 interrupts.CPU41.PMI:Performance_monitoring_interrupts
6221 ± 38% -100.0% 0.00 interrupts.CPU42.NMI:Non-maskable_interrupts
6221 ± 38% -100.0% 0.00 interrupts.CPU42.PMI:Performance_monitoring_interrupts
6657 ± 28% -100.0% 0.00 interrupts.CPU43.NMI:Non-maskable_interrupts
6657 ± 28% -100.0% 0.00 interrupts.CPU43.PMI:Performance_monitoring_interrupts
6069 ± 32% -100.0% 0.00 interrupts.CPU44.NMI:Non-maskable_interrupts
6069 ± 32% -100.0% 0.00 interrupts.CPU44.PMI:Performance_monitoring_interrupts
7011 ± 26% -100.0% 0.00 interrupts.CPU45.NMI:Non-maskable_interrupts
7011 ± 26% -100.0% 0.00 interrupts.CPU45.PMI:Performance_monitoring_interrupts
5375 ± 35% -100.0% 0.00 interrupts.CPU46.NMI:Non-maskable_interrupts
5375 ± 35% -100.0% 0.00 interrupts.CPU46.PMI:Performance_monitoring_interrupts
5413 ± 35% -100.0% 0.00 interrupts.CPU47.NMI:Non-maskable_interrupts
5413 ± 35% -100.0% 0.00 interrupts.CPU47.PMI:Performance_monitoring_interrupts
7061 ± 26% -100.0% 0.00 interrupts.CPU48.NMI:Non-maskable_interrupts
7061 ± 26% -100.0% 0.00 interrupts.CPU48.PMI:Performance_monitoring_interrupts
5134 ± 33% -100.0% 0.00 interrupts.CPU49.NMI:Non-maskable_interrupts
5134 ± 33% -100.0% 0.00 interrupts.CPU49.PMI:Performance_monitoring_interrupts
5876 ± 36% -100.0% 0.00 interrupts.CPU5.NMI:Non-maskable_interrupts
5876 ± 36% -100.0% 0.00 interrupts.CPU5.PMI:Performance_monitoring_interrupts
6389 ± 35% -100.0% 0.00 interrupts.CPU50.NMI:Non-maskable_interrupts
6389 ± 35% -100.0% 0.00 interrupts.CPU50.PMI:Performance_monitoring_interrupts
5871 ± 26% -100.0% 0.00 interrupts.CPU51.NMI:Non-maskable_interrupts
5871 ± 26% -100.0% 0.00 interrupts.CPU51.PMI:Performance_monitoring_interrupts
6226 ± 27% -100.0% 0.00 interrupts.CPU52.NMI:Non-maskable_interrupts
6226 ± 27% -100.0% 0.00 interrupts.CPU52.PMI:Performance_monitoring_interrupts
4904 ± 33% -100.0% 0.00 interrupts.CPU53.NMI:Non-maskable_interrupts
4904 ± 33% -100.0% 0.00 interrupts.CPU53.PMI:Performance_monitoring_interrupts
3705 ± 42% -100.0% 0.00 interrupts.CPU54.NMI:Non-maskable_interrupts
3705 ± 42% -100.0% 0.00 interrupts.CPU54.PMI:Performance_monitoring_interrupts
4316 ± 42% -100.0% 0.00 interrupts.CPU55.NMI:Non-maskable_interrupts
4316 ± 42% -100.0% 0.00 interrupts.CPU55.PMI:Performance_monitoring_interrupts
4496 ± 50% -100.0% 0.00 interrupts.CPU56.NMI:Non-maskable_interrupts
4496 ± 50% -100.0% 0.00 interrupts.CPU56.PMI:Performance_monitoring_interrupts
5172 ± 50% -100.0% 0.00 interrupts.CPU57.NMI:Non-maskable_interrupts
5172 ± 50% -100.0% 0.00 interrupts.CPU57.PMI:Performance_monitoring_interrupts
4153 ± 45% -100.0% 1.00 ±223% interrupts.CPU58.NMI:Non-maskable_interrupts
4153 ± 45% -100.0% 1.00 ±223% interrupts.CPU58.PMI:Performance_monitoring_interrupts
4949 ± 44% -100.0% 0.00 interrupts.CPU59.NMI:Non-maskable_interrupts
4949 ± 44% -100.0% 0.00 interrupts.CPU59.PMI:Performance_monitoring_interrupts
6086 ± 28% -100.0% 0.00 interrupts.CPU6.NMI:Non-maskable_interrupts
6086 ± 28% -100.0% 0.00 interrupts.CPU6.PMI:Performance_monitoring_interrupts
4156 ± 45% -100.0% 0.00 interrupts.CPU60.NMI:Non-maskable_interrupts
4156 ± 45% -100.0% 0.00 interrupts.CPU60.PMI:Performance_monitoring_interrupts
4486 ± 50% -100.0% 0.00 interrupts.CPU61.NMI:Non-maskable_interrupts
4486 ± 50% -100.0% 0.00 interrupts.CPU61.PMI:Performance_monitoring_interrupts
1459 ±121% -62.1% 553.67 ± 11% interrupts.CPU62.CAL:Function_call_interrupts
4623 ± 40% -100.0% 0.00 interrupts.CPU62.NMI:Non-maskable_interrupts
4623 ± 40% -100.0% 0.00 interrupts.CPU62.PMI:Performance_monitoring_interrupts
5745 ± 32% -100.0% 0.00 interrupts.CPU63.NMI:Non-maskable_interrupts
5745 ± 32% -100.0% 0.00 interrupts.CPU63.PMI:Performance_monitoring_interrupts
5320 ± 35% -100.0% 0.00 interrupts.CPU64.NMI:Non-maskable_interrupts
5320 ± 35% -100.0% 0.00 interrupts.CPU64.PMI:Performance_monitoring_interrupts
4127 ± 36% -100.0% 0.00 interrupts.CPU65.NMI:Non-maskable_interrupts
4127 ± 36% -100.0% 0.00 interrupts.CPU65.PMI:Performance_monitoring_interrupts
4024 ± 51% -100.0% 0.00 interrupts.CPU66.NMI:Non-maskable_interrupts
4024 ± 51% -100.0% 0.00 interrupts.CPU66.PMI:Performance_monitoring_interrupts
4786 ± 36% -100.0% 0.00 interrupts.CPU67.NMI:Non-maskable_interrupts
4786 ± 36% -100.0% 0.00 interrupts.CPU67.PMI:Performance_monitoring_interrupts
6864 ± 26% -100.0% 0.00 interrupts.CPU68.NMI:Non-maskable_interrupts
6864 ± 26% -100.0% 0.00 interrupts.CPU68.PMI:Performance_monitoring_interrupts
183.00 ± 39% -65.5% 63.17 ± 80% interrupts.CPU68.RES:Rescheduling_interrupts
5280 ± 41% -100.0% 0.00 interrupts.CPU69.NMI:Non-maskable_interrupts
5280 ± 41% -100.0% 0.00 interrupts.CPU69.PMI:Performance_monitoring_interrupts
5936 ± 33% -100.0% 0.00 interrupts.CPU7.NMI:Non-maskable_interrupts
5936 ± 33% -100.0% 0.00 interrupts.CPU7.PMI:Performance_monitoring_interrupts
3813 ± 35% -100.0% 1.00 ±223% interrupts.CPU70.NMI:Non-maskable_interrupts
3813 ± 35% -100.0% 1.00 ±223% interrupts.CPU70.PMI:Performance_monitoring_interrupts
5284 ± 44% -100.0% 0.00 interrupts.CPU71.NMI:Non-maskable_interrupts
5284 ± 44% -100.0% 0.00 interrupts.CPU71.PMI:Performance_monitoring_interrupts
6574 ± 27% -100.0% 0.00 interrupts.CPU72.NMI:Non-maskable_interrupts
6574 ± 27% -100.0% 0.00 interrupts.CPU72.PMI:Performance_monitoring_interrupts
4859 ± 35% -100.0% 0.00 interrupts.CPU73.NMI:Non-maskable_interrupts
4859 ± 35% -100.0% 0.00 interrupts.CPU73.PMI:Performance_monitoring_interrupts
4462 ± 36% -100.0% 0.00 interrupts.CPU74.NMI:Non-maskable_interrupts
4462 ± 36% -100.0% 0.00 interrupts.CPU74.PMI:Performance_monitoring_interrupts
6674 ± 26% -100.0% 0.00 interrupts.CPU75.NMI:Non-maskable_interrupts
6674 ± 26% -100.0% 0.00 interrupts.CPU75.PMI:Performance_monitoring_interrupts
4385 ± 49% -100.0% 0.00 interrupts.CPU76.NMI:Non-maskable_interrupts
4385 ± 49% -100.0% 0.00 interrupts.CPU76.PMI:Performance_monitoring_interrupts
730.50 ± 17% -21.4% 574.50 ± 6% interrupts.CPU77.CAL:Function_call_interrupts
5248 ± 45% -100.0% 0.00 interrupts.CPU77.NMI:Non-maskable_interrupts
5248 ± 45% -100.0% 0.00 interrupts.CPU77.PMI:Performance_monitoring_interrupts
6331 ± 28% -100.0% 0.00 interrupts.CPU78.NMI:Non-maskable_interrupts
6331 ± 28% -100.0% 0.00 interrupts.CPU78.PMI:Performance_monitoring_interrupts
4801 ± 38% -100.0% 0.00 interrupts.CPU79.NMI:Non-maskable_interrupts
4801 ± 38% -100.0% 0.00 interrupts.CPU79.PMI:Performance_monitoring_interrupts
5608 ± 37% -100.0% 0.00 interrupts.CPU8.NMI:Non-maskable_interrupts
5608 ± 37% -100.0% 0.00 interrupts.CPU8.PMI:Performance_monitoring_interrupts
5432 ± 45% -100.0% 1.00 ±223% interrupts.CPU80.NMI:Non-maskable_interrupts
5432 ± 45% -100.0% 1.00 ±223% interrupts.CPU80.PMI:Performance_monitoring_interrupts
6875 ± 22% -100.0% 1.00 ±223% interrupts.CPU81.NMI:Non-maskable_interrupts
6875 ± 22% -100.0% 1.00 ±223% interrupts.CPU81.PMI:Performance_monitoring_interrupts
6375 ± 34% -100.0% 0.00 interrupts.CPU82.NMI:Non-maskable_interrupts
6375 ± 34% -100.0% 0.00 interrupts.CPU82.PMI:Performance_monitoring_interrupts
3909 ± 43% -100.0% 0.00 interrupts.CPU83.NMI:Non-maskable_interrupts
3909 ± 43% -100.0% 0.00 interrupts.CPU83.PMI:Performance_monitoring_interrupts
6750 ± 26% -100.0% 0.00 interrupts.CPU84.NMI:Non-maskable_interrupts
6750 ± 26% -100.0% 0.00 interrupts.CPU84.PMI:Performance_monitoring_interrupts
4277 ± 47% -100.0% 0.00 interrupts.CPU85.NMI:Non-maskable_interrupts
4277 ± 47% -100.0% 0.00 interrupts.CPU85.PMI:Performance_monitoring_interrupts
3813 ± 35% -100.0% 0.00 interrupts.CPU86.NMI:Non-maskable_interrupts
3813 ± 35% -100.0% 0.00 interrupts.CPU86.PMI:Performance_monitoring_interrupts
4906 ± 43% -100.0% 1.00 ±223% interrupts.CPU87.NMI:Non-maskable_interrupts
4906 ± 43% -100.0% 1.00 ±223% interrupts.CPU87.PMI:Performance_monitoring_interrupts
5015 ± 45% -100.0% 0.00 interrupts.CPU88.NMI:Non-maskable_interrupts
5015 ± 45% -100.0% 0.00 interrupts.CPU88.PMI:Performance_monitoring_interrupts
4539 ± 52% -100.0% 0.00 interrupts.CPU89.NMI:Non-maskable_interrupts
4539 ± 52% -100.0% 0.00 interrupts.CPU89.PMI:Performance_monitoring_interrupts
4803 ± 43% -100.0% 0.00 interrupts.CPU9.NMI:Non-maskable_interrupts
4803 ± 43% -100.0% 0.00 interrupts.CPU9.PMI:Performance_monitoring_interrupts
3339 ± 31% -100.0% 0.00 interrupts.CPU90.NMI:Non-maskable_interrupts
3339 ± 31% -100.0% 0.00 interrupts.CPU90.PMI:Performance_monitoring_interrupts
3704 ± 50% -100.0% 0.00 interrupts.CPU91.NMI:Non-maskable_interrupts
3704 ± 50% -100.0% 0.00 interrupts.CPU91.PMI:Performance_monitoring_interrupts
4157 ± 44% -100.0% 0.00 interrupts.CPU92.NMI:Non-maskable_interrupts
4157 ± 44% -100.0% 0.00 interrupts.CPU92.PMI:Performance_monitoring_interrupts
3429 ± 37% -100.0% 0.00 interrupts.CPU93.NMI:Non-maskable_interrupts
3429 ± 37% -100.0% 0.00 interrupts.CPU93.PMI:Performance_monitoring_interrupts
4539 ± 52% -100.0% 0.00 interrupts.CPU94.NMI:Non-maskable_interrupts
4539 ± 52% -100.0% 0.00 interrupts.CPU94.PMI:Performance_monitoring_interrupts
4583 ± 41% -100.0% 0.00 interrupts.CPU95.NMI:Non-maskable_interrupts
4583 ± 41% -100.0% 0.00 interrupts.CPU95.PMI:Performance_monitoring_interrupts
5170 ± 44% -100.0% 0.00 interrupts.CPU96.NMI:Non-maskable_interrupts
5170 ± 44% -100.0% 0.00 interrupts.CPU96.PMI:Performance_monitoring_interrupts
4205 ± 46% -100.0% 1.00 ±223% interrupts.CPU97.NMI:Non-maskable_interrupts
4205 ± 46% -100.0% 1.00 ±223% interrupts.CPU97.PMI:Performance_monitoring_interrupts
5019 ± 39% -100.0% 0.00 interrupts.CPU98.NMI:Non-maskable_interrupts
5019 ± 39% -100.0% 0.00 interrupts.CPU98.PMI:Performance_monitoring_interrupts
4982 ± 38% -100.0% 0.00 interrupts.CPU99.NMI:Non-maskable_interrupts
4982 ± 38% -100.0% 0.00 interrupts.CPU99.PMI:Performance_monitoring_interrupts
107.67 ± 16% -100.0% 0.00 interrupts.IWI:IRQ_work_interrupts
550173 ± 9% -100.0% 6.00 interrupts.NMI:Non-maskable_interrupts
550173 ± 9% -100.0% 6.00 interrupts.PMI:Performance_monitoring_interrupts***************************************************************************************************
lkp-hsw-4ex1: 144 threads Intel(R) Xeon(R) CPU E7-8890 v3 @ 2.50GHz with 512G memory
=========================================================================================
compiler/cpufreq_governor/kconfig/mode/nr_task/rootfs/tbox_group/test/testcase/ucode:
gcc-9/performance/x86_64-rhel-8.3/thread/16/debian-10.4-x86_64-20200603.cgz/lkp-hsw-4ex1/pipe1/will-it-scale/0x16
commit:
dde38d5b80 ("kernel: don't include PF_IO_WORKERs as part of same_thread_group()") 43b2a76b1a ("proc: don't show PF_IO_WORKER threads as threads in /proc/<pid>/task/")
dde38d5b805358c7 43b2a76b1a5abcc9833b463bef1
---------------- ---------------------------
%stddev %change %stddev
\ | \
10347715 -7.1% 9613244 will-it-scale.16.threads
646731 -7.1% 600827 will-it-scale.per_thread_ops
10347715 -7.1% 9613244 will-it-scale.workload
2161039 ±140% -89.9% 217227 ± 11% cpuidle.POLL.time
778537 ± 94% -90.1% 76755 ± 15% cpuidle.POLL.usage
2.35 ± 5% -0.6 1.75 ± 3% mpstat.cpu.all.irq%
0.09 ± 3% -0.0 0.08 ± 2% mpstat.cpu.all.soft%
5.16 +0.5 5.67 mpstat.cpu.all.sys%
126150 ± 67% +455.9% 701316 ± 13% numa-numastat.node2.local_node
190120 ± 44% +302.2% 764691 ± 11% numa-numastat.node2.numa_hit
343721 ± 40% +124.8% 772821 ± 19% numa-numastat.node3.local_node
411537 ± 35% +104.5% 841447 ± 16% numa-numastat.node3.numa_hit
15406 ± 9% -19.7% 12379 ± 13% sched_debug.cfs_rq:/.load.avg
1538 ± 9% -28.6% 1098 ± 10% sched_debug.cpu.clock_task.stddev
3291 +20.7% 3971 ± 5% sched_debug.cpu.nr_switches.avg
-16.72 -28.2% -12.00 sched_debug.cpu.nr_uninterruptible.min
86.17 -1.4% 85.00 vmstat.cpu.id
1367031 +179.8% 3824534 vmstat.memory.cache
16.00 +12.5% 18.00 vmstat.procs.r
1773 +41.5% 2508 vmstat.system.cs
49509 ± 3% +4948.7% 2499569 ± 2% meminfo.Active
49509 ± 3% +4948.7% 2499569 ± 2% meminfo.Active(anon)
1253240 +195.4% 3702564 meminfo.Cached
532084 +471.8% 3042485 meminfo.Committed_AS
3151775 +94.6% 6133656 meminfo.Memused
4357 +12.6% 4906 meminfo.PageTables
71968 ± 2% +3403.3% 2521291 ± 2% meminfo.Shmem
3803846 +61.3% 6134154 meminfo.max_used_kB
54196 ± 4% +25.7% 68114 ± 4% slabinfo.filp.active_objs
855.67 ± 4% +26.0% 1078 ± 4% slabinfo.filp.active_slabs
54795 ± 4% +26.0% 69029 ± 4% slabinfo.filp.num_objs
855.67 ± 4% +26.0% 1078 ± 4% slabinfo.filp.num_slabs
19864 -29.1% 14091 slabinfo.proc_inode_cache.active_objs
413.17 -28.8% 294.17 slabinfo.proc_inode_cache.active_slabs
19865 -28.8% 14140 slabinfo.proc_inode_cache.num_objs
413.17 -28.8% 294.17 slabinfo.proc_inode_cache.num_slabs
24290 +36.2% 33073 slabinfo.radix_tree_node.active_objs
433.50 +36.2% 590.33 slabinfo.radix_tree_node.active_slabs
24290 +36.2% 33073 slabinfo.radix_tree_node.num_objs
433.50 +36.2% 590.33 slabinfo.radix_tree_node.num_slabs
243.00 ± 67% +12967.1% 31753 ± 95% numa-vmstat.node0.nr_active_anon
1329 ±107% +2462.5% 34073 ± 87% numa-vmstat.node0.nr_shmem
243.00 ± 67% +12967.1% 31753 ± 95% numa-vmstat.node0.nr_zone_active_anon
390.50 ± 70% +51012.8% 199595 ± 33% numa-vmstat.node1.nr_active_anon
74888 ± 3% +267.5% 275184 ± 24% numa-vmstat.node1.nr_file_pages
2116 ± 81% +9380.3% 200665 ± 33% numa-vmstat.node1.nr_shmem
390.50 ± 70% +51012.8% 199595 ± 33% numa-vmstat.node1.nr_zone_active_anon
219.50 ± 74% +87889.5% 193137 ± 42% numa-vmstat.node2.nr_active_anon
75475 ± 5% +250.7% 264662 ± 31% numa-vmstat.node2.nr_file_pages
1579 ± 57% +12147.1% 193483 ± 42% numa-vmstat.node2.nr_shmem
219.50 ± 74% +87889.5% 193137 ± 42% numa-vmstat.node2.nr_zone_active_anon
11535 ± 4% +1630.4% 199614 ± 60% numa-vmstat.node3.nr_active_anon
85408 ± 3% +226.2% 278565 ± 42% numa-vmstat.node3.nr_file_pages
12935 ± 14% +1456.2% 201307 ± 59% numa-vmstat.node3.nr_shmem
11535 ± 4% +1630.4% 199614 ± 60% numa-vmstat.node3.nr_zone_active_anon
575672 ± 16% +57.8% 908500 ± 8% numa-vmstat.node3.numa_hit
439401 ± 23% +70.1% 747636 ± 11% numa-vmstat.node3.numa_local
12360 ± 3% +4953.6% 624669 ± 2% proc-vmstat.nr_active_anon
64392 +8.8% 70043 proc-vmstat.nr_anon_pages
313288 +195.4% 925417 proc-vmstat.nr_file_pages
69914 +7.7% 75279 proc-vmstat.nr_inactive_anon
1091 +12.4% 1226 proc-vmstat.nr_page_table_pages
17970 ± 2% +3406.4% 630099 ± 2% proc-vmstat.nr_shmem
53352 +4.7% 55880 proc-vmstat.nr_slab_unreclaimable
12360 ± 3% +4953.6% 624669 ± 2% proc-vmstat.nr_zone_active_anon
69914 +7.7% 75279 proc-vmstat.nr_zone_inactive_anon
1201307 +109.1% 2512032 proc-vmstat.numa_hit
968119 +135.4% 2278868 proc-vmstat.numa_local
13054 ± 49% -95.5% 588.67 ± 34% proc-vmstat.numa_pages_migrated
17335 ± 4% +382.6% 83662 ± 2% proc-vmstat.pgactivate
1269551 +125.3% 2860508 ± 2% proc-vmstat.pgalloc_normal
1172329 -2.8% 1138971 proc-vmstat.pgfault
1183638 +10.8% 1311710 ± 5% proc-vmstat.pgfree
13054 ± 49% -95.5% 588.67 ± 34% proc-vmstat.pgmigrate_success
38556 ± 13% +163.9% 101756 ± 30% syscalls.sys_close.max
568.17 -22.5% 440.50 ± 6% syscalls.sys_close.med
17718871 ± 11% -4.3e+06 13405659 ± 21% syscalls.sys_close.noise.2%
17644152 ± 11% -4.3e+06 13344394 ± 22% syscalls.sys_close.noise.5%
103226 ± 13% +267.9% 379798 ± 12% syscalls.sys_openat.max
3284 +72.8% 5675 ± 6% syscalls.sys_openat.med
99825845 ± 3% -4.1e+07 58578210 ± 13% syscalls.sys_openat.noise.100%
1.404e+08 ± 3% -3.3e+07 1.078e+08 ± 13% syscalls.sys_openat.noise.2%
1.332e+08 ± 3% -4.5e+07 87987720 ± 18% syscalls.sys_openat.noise.25%
1.4e+08 ± 3% -3.4e+07 1.065e+08 ± 14% syscalls.sys_openat.noise.5%
1.183e+08 ± 3% -4.4e+07 74351118 ± 24% syscalls.sys_openat.noise.50%
1.066e+08 ± 3% -3.9e+07 67567568 ± 24% syscalls.sys_openat.noise.75%
1.074e+11 ± 6% -3.8e+10 6.928e+10 ± 19% syscalls.sys_read.noise.100%
1.074e+11 ± 6% -3.8e+10 6.928e+10 ± 19% syscalls.sys_read.noise.2%
1.074e+11 ± 6% -3.8e+10 6.928e+10 ± 19% syscalls.sys_read.noise.25%
1.074e+11 ± 6% -3.8e+10 6.928e+10 ± 19% syscalls.sys_read.noise.5%
1.074e+11 ± 6% -3.8e+10 6.928e+10 ± 19% syscalls.sys_read.noise.50%
1.074e+11 ± 6% -3.8e+10 6.928e+10 ± 19% syscalls.sys_read.noise.75%
973.33 ± 67% +12957.0% 127087 ± 95% numa-meminfo.node0.Active
973.33 ± 67% +12957.0% 127087 ± 95% numa-meminfo.node0.Active(anon)
973600 ± 8% +18.3% 1152023 ± 8% numa-meminfo.node0.MemUsed
5320 ±107% +2463.2% 136369 ± 87% numa-meminfo.node0.Shmem
1563 ± 70% +51094.1% 800163 ± 33% numa-meminfo.node1.Active
1563 ± 70% +51094.1% 800163 ± 33% numa-meminfo.node1.Active(anon)
299552 ± 3% +268.1% 1102521 ± 24% numa-meminfo.node1.FilePages
693218 ± 6% +133.1% 1615690 ± 16% numa-meminfo.node1.MemUsed
8467 ± 81% +9400.8% 804445 ± 33% numa-meminfo.node1.Shmem
879.67 ± 73% +87936.5% 774427 ± 42% numa-meminfo.node2.Active
879.67 ± 73% +87936.5% 774427 ± 42% numa-meminfo.node2.Active(anon)
301902 ± 5% +251.3% 1060528 ± 31% numa-meminfo.node2.FilePages
728495 ± 15% +119.9% 1601674 ± 20% numa-meminfo.node2.MemUsed
6321 ± 57% +12173.6% 775813 ± 42% numa-meminfo.node2.Shmem
46166 ± 4% +1633.7% 800379 ± 60% numa-meminfo.node3.Active
46166 ± 4% +1633.7% 800379 ± 60% numa-meminfo.node3.Active(anon)
341668 ± 3% +226.7% 1116179 ± 42% numa-meminfo.node3.FilePages
756726 ± 7% +133.5% 1766769 ± 28% numa-meminfo.node3.MemUsed
51779 ± 14% +1458.8% 807147 ± 59% numa-meminfo.node3.Shmem
36.03 ± 18% -36.0 0.00 perf-profile.calltrace.cycles-pp.secondary_startup_64_no_verify
34.77 ± 16% -34.8 0.00 perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
34.77 ± 16% -34.8 0.00 perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64_no_verify
34.76 ± 16% -34.8 0.00 perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
33.91 ± 16% -33.9 0.00 perf-profile.calltrace.cycles-pp.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
33.74 ± 17% -33.7 0.00 perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary
32.55 ± 21% -32.5 0.00 perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry
31.89 ± 10% -31.9 0.00 perf-profile.calltrace.cycles-pp.__libc_read
31.50 ± 10% -31.5 0.00 perf-profile.calltrace.cycles-pp.__libc_write
24.59 ± 10% -24.6 0.00 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__libc_read
24.14 ± 10% -24.1 0.00 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__libc_write
12.22 ± 10% -12.2 0.00 perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_read
12.04 ± 10% -12.0 0.00 perf-profile.calltrace.cycles-pp.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe.__libc_write
11.97 ± 10% -12.0 0.00 perf-profile.calltrace.cycles-pp.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe.__libc_read
11.73 ± 10% -11.7 0.00 perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_write
8.45 ± 10% -8.4 0.00 perf-profile.calltrace.cycles-pp.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_read
7.88 ± 9% -7.9 0.00 perf-profile.calltrace.cycles-pp.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_write
7.20 ± 10% -7.2 0.00 perf-profile.calltrace.cycles-pp.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_read
6.64 ± 9% -6.6 0.00 perf-profile.calltrace.cycles-pp.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_write
5.12 ± 10% -5.1 0.00 perf-profile.calltrace.cycles-pp.new_sync_read.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe
4.97 ± 10% -5.0 0.00 perf-profile.calltrace.cycles-pp.new_sync_write.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
48.91 ± 10% -48.9 0.00 perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
36.03 ± 18% -36.0 0.00 perf-profile.children.cycles-pp.secondary_startup_64_no_verify
36.03 ± 18% -36.0 0.00 perf-profile.children.cycles-pp.cpu_startup_entry
36.03 ± 18% -36.0 0.00 perf-profile.children.cycles-pp.do_idle
35.17 ± 19% -35.2 0.00 perf-profile.children.cycles-pp.cpuidle_enter
35.16 ± 19% -35.2 0.00 perf-profile.children.cycles-pp.cpuidle_enter_state
34.77 ± 16% -34.8 0.00 perf-profile.children.cycles-pp.start_secondary
32.55 ± 21% -32.5 0.00 perf-profile.children.cycles-pp.intel_idle
32.38 ± 10% -32.4 0.00 perf-profile.children.cycles-pp.__libc_read
31.78 ± 10% -31.8 0.00 perf-profile.children.cycles-pp.__libc_write
24.08 ± 10% -24.1 0.00 perf-profile.children.cycles-pp.syscall_exit_to_user_mode
24.06 ± 10% -24.1 0.00 perf-profile.children.cycles-pp.do_syscall_64
8.49 ± 10% -8.5 0.00 perf-profile.children.cycles-pp.ksys_read
7.92 ± 9% -7.9 0.00 perf-profile.children.cycles-pp.ksys_write
7.27 ± 10% -7.3 0.00 perf-profile.children.cycles-pp.vfs_read
6.70 ± 9% -6.7 0.00 perf-profile.children.cycles-pp.vfs_write
6.68 ± 10% -6.7 0.00 perf-profile.children.cycles-pp.syscall_return_via_sysret
6.38 ± 10% -6.4 0.00 perf-profile.children.cycles-pp.syscall_trace_enter
6.34 ± 10% -6.3 0.00 perf-profile.children.cycles-pp.__entry_text_start
6.15 ± 10% -6.1 0.00 perf-profile.children.cycles-pp.trace_buffer_lock_reserve
5.96 ± 10% -6.0 0.00 perf-profile.children.cycles-pp.ftrace_syscall_enter
5.66 ± 10% -5.7 0.00 perf-profile.children.cycles-pp.syscall_exit_to_user_mode_prepare
5.42 ± 11% -5.4 0.00 perf-profile.children.cycles-pp.ring_buffer_lock_reserve
5.29 ± 10% -5.3 0.00 perf-profile.children.cycles-pp.ftrace_syscall_exit
5.16 ± 11% -5.2 0.00 perf-profile.children.cycles-pp.new_sync_read
5.03 ± 9% -5.0 0.00 perf-profile.children.cycles-pp.new_sync_write
4.69 ± 10% -4.7 0.00 perf-profile.children.cycles-pp.pipe_read
32.55 ± 21% -32.5 0.00 perf-profile.self.cycles-pp.intel_idle
18.11 ± 10% -18.1 0.00 perf-profile.self.cycles-pp.syscall_exit_to_user_mode
6.65 ± 10% -6.7 0.00 perf-profile.self.cycles-pp.syscall_return_via_sysret
5.45 ± 10% -5.5 0.00 perf-profile.self.cycles-pp.__entry_text_start
1.86 +9.5% 2.04 perf-stat.i.MPKI
7.787e+09 +13.4% 8.828e+09 perf-stat.i.branch-instructions
1.27 -0.1 1.14 perf-stat.i.branch-miss-rate%
0.34 ± 9% +5.1 5.46 ± 13% perf-stat.i.cache-miss-rate%
245601 ± 9% +1822.1% 4720745 ± 13% perf-stat.i.cache-misses
70035252 +24.3% 87072170 perf-stat.i.cache-references
1724 +42.7% 2461 perf-stat.i.context-switches
5.293e+10 +12.2% 5.936e+10 perf-stat.i.cpu-cycles
251199 ± 10% -88.3% 29314 ± 26% perf-stat.i.cycles-between-cache-misses
0.21 -0.0 0.19 perf-stat.i.dTLB-load-miss-rate%
1.139e+10 +9.2% 1.244e+10 perf-stat.i.dTLB-loads
0.26 -0.0 0.23 perf-stat.i.dTLB-store-miss-rate%
8.301e+09 +7.0% 8.884e+09 perf-stat.i.dTLB-stores
86.29 -5.0 81.30 ± 2% perf-stat.i.iTLB-load-miss-rate%
3599988 ± 13% +44.8% 5213290 ± 14% perf-stat.i.iTLB-loads
3.804e+10 +13.4% 4.312e+10 perf-stat.i.instructions
1686 ± 2% +13.5% 1913 ± 4% perf-stat.i.instructions-per-iTLB-miss
1.02 ± 2% -94.3% 0.06 ± 44% perf-stat.i.major-faults
0.37 +12.2% 0.41 perf-stat.i.metric.GHz
1.09 ± 10% -24.1% 0.83 ± 14% perf-stat.i.metric.K/sec
191.47 +9.8% 210.19 perf-stat.i.metric.M/sec
3770 -3.1% 3653 perf-stat.i.minor-faults
94.62 +3.2 97.82 perf-stat.i.node-load-miss-rate%
155456 ± 10% +1936.0% 3165118 ± 16% perf-stat.i.node-load-misses
13817 ± 8% +243.8% 47505 ± 43% perf-stat.i.node-loads
46.19 ± 12% +39.2 85.35 ± 2% perf-stat.i.node-store-miss-rate%
32493 ± 13% +4121.7% 1371781 ± 11% perf-stat.i.node-store-misses
40563 ± 14% +220.5% 130023 ± 8% perf-stat.i.node-stores
3771 -3.1% 3653 perf-stat.i.page-faults
1.84 +9.7% 2.02 perf-stat.overall.MPKI
1.26 -0.1 1.13 perf-stat.overall.branch-miss-rate%
0.35 ± 9% +5.1 5.42 ± 13% perf-stat.overall.cache-miss-rate%
216967 ± 8% -94.1% 12875 ± 16% perf-stat.overall.cycles-between-cache-misses
0.21 -0.0 0.19 perf-stat.overall.dTLB-load-miss-rate%
0.26 -0.0 0.23 perf-stat.overall.dTLB-store-miss-rate%
86.34 -5.0 81.34 ± 2% perf-stat.overall.iTLB-load-miss-rate%
1677 ± 2% +13.5% 1904 ± 4% perf-stat.overall.instructions-per-iTLB-miss
91.77 +6.7 98.44 perf-stat.overall.node-load-miss-rate%
44.57 ± 11% +46.7 91.25 perf-stat.overall.node-store-miss-rate%
1106216 +22.1% 1350408 perf-stat.overall.path-length
7.76e+09 +13.4% 8.798e+09 perf-stat.ps.branch-instructions
245116 ± 9% +1819.1% 4704109 ± 13% perf-stat.ps.cache-misses
69795445 +24.3% 86773526 perf-stat.ps.cache-references
1719 +42.7% 2452 perf-stat.ps.context-switches
5.275e+10 +12.2% 5.916e+10 perf-stat.ps.cpu-cycles
1.135e+10 +9.2% 1.24e+10 perf-stat.ps.dTLB-loads
8.273e+09 +7.0% 8.853e+09 perf-stat.ps.dTLB-stores
3587022 ± 13% +44.8% 5195656 ± 14% perf-stat.ps.iTLB-loads
3.792e+10 +13.3% 4.298e+10 perf-stat.ps.instructions
1.01 ± 2% -94.3% 0.06 ± 44% perf-stat.ps.major-faults
3757 -3.1% 3640 perf-stat.ps.minor-faults
155084 ± 10% +1933.7% 3153975 ± 16% perf-stat.ps.node-load-misses
13831 ± 8% +242.2% 47330 ± 43% perf-stat.ps.node-loads
32417 ± 13% +4116.6% 1366944 ± 11% perf-stat.ps.node-store-misses
40467 ± 14% +220.2% 129580 ± 8% perf-stat.ps.node-stores
3758 -3.1% 3640 perf-stat.ps.page-faults
1.145e+13 +13.4% 1.298e+13 perf-stat.total.instructions
0.02 ± 23% -100.0% 0.00 perf-sched.sch_delay.avg.ms.__traceiter_sched_switch.__traceiter_sched_switch.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe
0.04 ± 18% -100.0% 0.00 perf-sched.sch_delay.avg.ms.__traceiter_sched_switch.__traceiter_sched_switch.devkmsg_read.vfs_read.ksys_read
0.01 ± 21% -100.0% 0.00 perf-sched.sch_delay.avg.ms.__traceiter_sched_switch.__traceiter_sched_switch.do_nanosleep.hrtimer_nanosleep.__x64_sys_nanosleep
0.03 ± 32% -100.0% 0.00 perf-sched.sch_delay.avg.ms.__traceiter_sched_switch.__traceiter_sched_switch.do_syslog.part.0
0.01 ± 7% -100.0% 0.00 perf-sched.sch_delay.avg.ms.__traceiter_sched_switch.__traceiter_sched_switch.do_task_dead.do_exit.do_group_exit
0.01 ± 19% -100.0% 0.00 perf-sched.sch_delay.avg.ms.__traceiter_sched_switch.__traceiter_sched_switch.do_wait.kernel_wait4.__do_sys_wait4
0.03 ± 37% -100.0% 0.00 perf-sched.sch_delay.avg.ms.__traceiter_sched_switch.__traceiter_sched_switch.futex_wait_queue_me.futex_wait.do_futex
0.00 ± 14% -100.0% 0.00 perf-sched.sch_delay.avg.ms.__traceiter_sched_switch.__traceiter_sched_switch.pipe_read.new_sync_read.vfs_read
0.05 ± 34% -100.0% 0.00 perf-sched.sch_delay.avg.ms.__traceiter_sched_switch.__traceiter_sched_switch.rcu_gp_kthread.kthread.ret_from_fork
0.02 ± 24% -100.0% 0.00 perf-sched.sch_delay.avg.ms.__traceiter_sched_switch.__traceiter_sched_switch.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait
0.01 ± 20% -100.0% 0.00 perf-sched.sch_delay.avg.ms.__traceiter_sched_switch.__traceiter_sched_switch.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop
0.02 ± 31% -100.0% 0.00 perf-sched.sch_delay.avg.ms.__traceiter_sched_switch.__traceiter_sched_switch.schedule_timeout.io_schedule_timeout.wait_for_completion_io
0.02 ± 61% -100.0% 0.00 perf-sched.sch_delay.avg.ms.__traceiter_sched_switch.__traceiter_sched_switch.schedule_timeout.kcompactd.kthread
0.04 ± 43% -100.0% 0.00 perf-sched.sch_delay.avg.ms.__traceiter_sched_switch.__traceiter_sched_switch.schedule_timeout.rcu_gp_kthread.kthread
0.02 ± 40% -100.0% 0.00 perf-sched.sch_delay.avg.ms.__traceiter_sched_switch.__traceiter_sched_switch.schedule_timeout.wait_for_completion.__flush_work
0.01 ± 4% -100.0% 0.00 perf-sched.sch_delay.avg.ms.__traceiter_sched_switch.__traceiter_sched_switch.smpboot_thread_fn.kthread.ret_from_fork
0.00 ± 14% -100.0% 0.00 perf-sched.sch_delay.avg.ms.__traceiter_sched_switch.__traceiter_sched_switch.wait_for_partner.fifo_open.do_dentry_open
1.91 ±218% -100.0% 0.00 perf-sched.sch_delay.avg.ms.__traceiter_sched_switch.__traceiter_sched_switch.worker_thread.kthread.ret_from_fork
0.03 ± 42% -100.0% 0.00 perf-sched.sch_delay.max.ms.__traceiter_sched_switch.__traceiter_sched_switch.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe
0.06 ± 19% -100.0% 0.00 perf-sched.sch_delay.max.ms.__traceiter_sched_switch.__traceiter_sched_switch.devkmsg_read.vfs_read.ksys_read
0.03 ± 49% -100.0% 0.00 perf-sched.sch_delay.max.ms.__traceiter_sched_switch.__traceiter_sched_switch.do_nanosleep.hrtimer_nanosleep.__x64_sys_nanosleep
0.05 ± 23% -100.0% 0.00 perf-sched.sch_delay.max.ms.__traceiter_sched_switch.__traceiter_sched_switch.do_syslog.part.0
0.05 ± 3% -100.0% 0.00 perf-sched.sch_delay.max.ms.__traceiter_sched_switch.__traceiter_sched_switch.do_task_dead.do_exit.do_group_exit
0.06 ± 9% -100.0% 0.00 perf-sched.sch_delay.max.ms.__traceiter_sched_switch.__traceiter_sched_switch.do_wait.kernel_wait4.__do_sys_wait4
0.03 ± 47% -100.0% 0.00 perf-sched.sch_delay.max.ms.__traceiter_sched_switch.__traceiter_sched_switch.exit_to_user_mode_prepare.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe
0.05 ± 33% -100.0% 0.00 perf-sched.sch_delay.max.ms.__traceiter_sched_switch.__traceiter_sched_switch.futex_wait_queue_me.futex_wait.do_futex
0.06 ± 8% -100.0% 0.00 perf-sched.sch_delay.max.ms.__traceiter_sched_switch.__traceiter_sched_switch.pipe_read.new_sync_read.vfs_read
0.04 ± 12% -100.0% 0.00 perf-sched.sch_delay.max.ms.__traceiter_sched_switch.__traceiter_sched_switch.preempt_schedule_common.__cond_resched.stop_one_cpu
0.07 ± 30% -100.0% 0.00 perf-sched.sch_delay.max.ms.__traceiter_sched_switch.__traceiter_sched_switch.rcu_gp_kthread.kthread.ret_from_fork
0.05 ± 20% -100.0% 0.00 perf-sched.sch_delay.max.ms.__traceiter_sched_switch.__traceiter_sched_switch.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait
0.04 ± 29% -100.0% 0.00 perf-sched.sch_delay.max.ms.__traceiter_sched_switch.__traceiter_sched_switch.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop
0.02 ± 31% -100.0% 0.00 perf-sched.sch_delay.max.ms.__traceiter_sched_switch.__traceiter_sched_switch.schedule_timeout.io_schedule_timeout.wait_for_completion_io
0.55 ±208% -100.0% 0.00 perf-sched.sch_delay.max.ms.__traceiter_sched_switch.__traceiter_sched_switch.schedule_timeout.kcompactd.kthread
6.73 ±140% -100.0% 0.00 perf-sched.sch_delay.max.ms.__traceiter_sched_switch.__traceiter_sched_switch.schedule_timeout.rcu_gp_kthread.kthread
0.02 ± 51% -100.0% 0.00 perf-sched.sch_delay.max.ms.__traceiter_sched_switch.__traceiter_sched_switch.schedule_timeout.wait_for_completion.__flush_work
6.77 ± 9% -100.0% 0.00 perf-sched.sch_delay.max.ms.__traceiter_sched_switch.__traceiter_sched_switch.smpboot_thread_fn.kthread.ret_from_fork
0.03 ± 28% -100.0% 0.00 perf-sched.sch_delay.max.ms.__traceiter_sched_switch.__traceiter_sched_switch.wait_for_partner.fifo_open.do_dentry_open
298.90 ±213% -100.0% 0.00 perf-sched.sch_delay.max.ms.__traceiter_sched_switch.__traceiter_sched_switch.worker_thread.kthread.ret_from_fork
0.14 ±206% -100.0% 0.00 perf-sched.total_sch_delay.average.ms
300.24 ±212% -100.0% 0.00 perf-sched.total_sch_delay.max.ms
215.33 -100.0% 0.00 perf-sched.total_wait_and_delay.average.ms
9201 -100.0% 0.00 perf-sched.total_wait_and_delay.count.ms
8311 ± 8% -100.0% 0.00 perf-sched.total_wait_and_delay.max.ms
215.19 -100.0% 0.00 perf-sched.total_wait_time.average.ms
8311 ± 8% -100.0% 0.00 perf-sched.total_wait_time.max.ms
899.66 -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.__traceiter_sched_switch.__traceiter_sched_switch.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe
259.95 ± 13% -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.__traceiter_sched_switch.__traceiter_sched_switch.devkmsg_read.vfs_read.ksys_read
781.14 ± 5% -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.__traceiter_sched_switch.__traceiter_sched_switch.do_nanosleep.hrtimer_nanosleep.__x64_sys_nanosleep
259.95 ± 13% -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.__traceiter_sched_switch.__traceiter_sched_switch.do_syslog.part.0
211.48 ± 5% -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.__traceiter_sched_switch.__traceiter_sched_switch.do_task_dead.do_exit.do_group_exit
0.88 -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.__traceiter_sched_switch.__traceiter_sched_switch.do_wait.kernel_wait4.__do_sys_wait4
0.05 ± 26% -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.__traceiter_sched_switch.__traceiter_sched_switch.exit_to_user_mode_prepare.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe
53.04 -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.__traceiter_sched_switch.__traceiter_sched_switch.pipe_read.new_sync_read.vfs_read
0.05 ± 16% -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.__traceiter_sched_switch.__traceiter_sched_switch.preempt_schedule_common.__cond_resched.mutex_lock
5.67 ± 3% -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.__traceiter_sched_switch.__traceiter_sched_switch.preempt_schedule_common.__cond_resched.stop_one_cpu
2.82 ± 4% -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.__traceiter_sched_switch.__traceiter_sched_switch.rcu_gp_kthread.kthread.ret_from_fork
580.11 ± 6% -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.__traceiter_sched_switch.__traceiter_sched_switch.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop
677.37 ± 30% -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.__traceiter_sched_switch.__traceiter_sched_switch.schedule_timeout.io_schedule_timeout.wait_for_completion_io
478.71 -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.__traceiter_sched_switch.__traceiter_sched_switch.schedule_timeout.kcompactd.kthread
12.51 ± 3% -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.__traceiter_sched_switch.__traceiter_sched_switch.schedule_timeout.rcu_gp_kthread.kthread
687.31 ± 2% -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.__traceiter_sched_switch.__traceiter_sched_switch.smpboot_thread_fn.kthread.ret_from_fork
0.00 ± 14% -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.__traceiter_sched_switch.__traceiter_sched_switch.wait_for_partner.fifo_open.do_dentry_open
438.87 ± 11% -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.__traceiter_sched_switch.__traceiter_sched_switch.worker_thread.kthread.ret_from_fork
10.00 -100.0% 0.00 perf-sched.wait_and_delay.count.__traceiter_sched_switch.__traceiter_sched_switch.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe
6.33 ± 11% -100.0% 0.00 perf-sched.wait_and_delay.count.__traceiter_sched_switch.__traceiter_sched_switch.devkmsg_read.vfs_read.ksys_read
23.17 ± 5% -100.0% 0.00 perf-sched.wait_and_delay.count.__traceiter_sched_switch.__traceiter_sched_switch.do_nanosleep.hrtimer_nanosleep.__x64_sys_nanosleep
6.33 ± 11% -100.0% 0.00 perf-sched.wait_and_delay.count.__traceiter_sched_switch.__traceiter_sched_switch.do_syslog.part.0
313.50 -100.0% 0.00 perf-sched.wait_and_delay.count.__traceiter_sched_switch.__traceiter_sched_switch.do_task_dead.do_exit.do_group_exit
310.83 -100.0% 0.00 perf-sched.wait_and_delay.count.__traceiter_sched_switch.__traceiter_sched_switch.do_wait.kernel_wait4.__do_sys_wait4
69.00 ± 43% -100.0% 0.00 perf-sched.wait_and_delay.count.__traceiter_sched_switch.__traceiter_sched_switch.exit_to_user_mode_prepare.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe
2916 -100.0% 0.00 perf-sched.wait_and_delay.count.__traceiter_sched_switch.__traceiter_sched_switch.pipe_read.new_sync_read.vfs_read
82.83 ± 13% -100.0% 0.00 perf-sched.wait_and_delay.count.__traceiter_sched_switch.__traceiter_sched_switch.preempt_schedule_common.__cond_resched.mutex_lock
1655 ± 4% -100.0% 0.00 perf-sched.wait_and_delay.count.__traceiter_sched_switch.__traceiter_sched_switch.preempt_schedule_common.__cond_resched.stop_one_cpu
83.50 ± 7% -100.0% 0.00 perf-sched.wait_and_delay.count.__traceiter_sched_switch.__traceiter_sched_switch.rcu_gp_kthread.kthread.ret_from_fork
59.50 ± 6% -100.0% 0.00 perf-sched.wait_and_delay.count.__traceiter_sched_switch.__traceiter_sched_switch.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop
4.83 ± 7% -100.0% 0.00 perf-sched.wait_and_delay.count.__traceiter_sched_switch.__traceiter_sched_switch.schedule_timeout.io_schedule_timeout.wait_for_completion_io
79.67 -100.0% 0.00 perf-sched.wait_and_delay.count.__traceiter_sched_switch.__traceiter_sched_switch.schedule_timeout.kcompactd.kthread
782.17 ± 3% -100.0% 0.00 perf-sched.wait_and_delay.count.__traceiter_sched_switch.__traceiter_sched_switch.schedule_timeout.rcu_gp_kthread.kthread
1947 ± 3% -100.0% 0.00 perf-sched.wait_and_delay.count.__traceiter_sched_switch.__traceiter_sched_switch.smpboot_thread_fn.kthread.ret_from_fork
73.00 -100.0% 0.00 perf-sched.wait_and_delay.count.__traceiter_sched_switch.__traceiter_sched_switch.wait_for_partner.fifo_open.do_dentry_open
658.50 ± 2% -100.0% 0.00 perf-sched.wait_and_delay.count.__traceiter_sched_switch.__traceiter_sched_switch.worker_thread.kthread.ret_from_fork
999.71 -100.0% 0.00 perf-sched.wait_and_delay.max.ms.__traceiter_sched_switch.__traceiter_sched_switch.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe
1537 ± 5% -100.0% 0.00 perf-sched.wait_and_delay.max.ms.__traceiter_sched_switch.__traceiter_sched_switch.devkmsg_read.vfs_read.ksys_read
1000 -100.0% 0.00 perf-sched.wait_and_delay.max.ms.__traceiter_sched_switch.__traceiter_sched_switch.do_nanosleep.hrtimer_nanosleep.__x64_sys_nanosleep
1537 ± 5% -100.0% 0.00 perf-sched.wait_and_delay.max.ms.__traceiter_sched_switch.__traceiter_sched_switch.do_syslog.part.0
1966 ± 59% -100.0% 0.00 perf-sched.wait_and_delay.max.ms.__traceiter_sched_switch.__traceiter_sched_switch.do_task_dead.do_exit.do_group_exit
20.30 -100.0% 0.00 perf-sched.wait_and_delay.max.ms.__traceiter_sched_switch.__traceiter_sched_switch.do_wait.kernel_wait4.__do_sys_wait4
0.48 ±164% -100.0% 0.00 perf-sched.wait_and_delay.max.ms.__traceiter_sched_switch.__traceiter_sched_switch.exit_to_user_mode_prepare.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe
1540 ± 5% -100.0% 0.00 perf-sched.wait_and_delay.max.ms.__traceiter_sched_switch.__traceiter_sched_switch.pipe_read.new_sync_read.vfs_read
0.12 ± 46% -100.0% 0.00 perf-sched.wait_and_delay.max.ms.__traceiter_sched_switch.__traceiter_sched_switch.preempt_schedule_common.__cond_resched.mutex_lock
1071 ± 14% -100.0% 0.00 perf-sched.wait_and_delay.max.ms.__traceiter_sched_switch.__traceiter_sched_switch.preempt_schedule_common.__cond_resched.stop_one_cpu
4.54 ± 10% -100.0% 0.00 perf-sched.wait_and_delay.max.ms.__traceiter_sched_switch.__traceiter_sched_switch.rcu_gp_kthread.kthread.ret_from_fork
7489 -100.0% 0.00 perf-sched.wait_and_delay.max.ms.__traceiter_sched_switch.__traceiter_sched_switch.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop
1151 ± 23% -100.0% 0.00 perf-sched.wait_and_delay.max.ms.__traceiter_sched_switch.__traceiter_sched_switch.schedule_timeout.io_schedule_timeout.wait_for_completion_io
505.63 -100.0% 0.00 perf-sched.wait_and_delay.max.ms.__traceiter_sched_switch.__traceiter_sched_switch.schedule_timeout.kcompactd.kthread
376.36 ± 10% -100.0% 0.00 perf-sched.wait_and_delay.max.ms.__traceiter_sched_switch.__traceiter_sched_switch.schedule_timeout.rcu_gp_kthread.kthread
1691 ± 88% -100.0% 0.00 perf-sched.wait_and_delay.max.ms.__traceiter_sched_switch.__traceiter_sched_switch.smpboot_thread_fn.kthread.ret_from_fork
0.03 ± 28% -100.0% 0.00 perf-sched.wait_and_delay.max.ms.__traceiter_sched_switch.__traceiter_sched_switch.wait_for_partner.fifo_open.do_dentry_open
7711 ± 21% -100.0% 0.00 perf-sched.wait_and_delay.max.ms.__traceiter_sched_switch.__traceiter_sched_switch.worker_thread.kthread.ret_from_fork
899.64 -100.0% 0.00 perf-sched.wait_time.avg.ms.__traceiter_sched_switch.__traceiter_sched_switch.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe
259.91 ± 13% -100.0% 0.00 perf-sched.wait_time.avg.ms.__traceiter_sched_switch.__traceiter_sched_switch.devkmsg_read.vfs_read.ksys_read
781.13 ± 5% -100.0% 0.00 perf-sched.wait_time.avg.ms.__traceiter_sched_switch.__traceiter_sched_switch.do_nanosleep.hrtimer_nanosleep.__x64_sys_nanosleep
259.91 ± 13% -100.0% 0.00 perf-sched.wait_time.avg.ms.__traceiter_sched_switch.__traceiter_sched_switch.do_syslog.part.0
211.47 ± 5% -100.0% 0.00 perf-sched.wait_time.avg.ms.__traceiter_sched_switch.__traceiter_sched_switch.do_task_dead.do_exit.do_group_exit
0.86 -100.0% 0.00 perf-sched.wait_time.avg.ms.__traceiter_sched_switch.__traceiter_sched_switch.do_wait.kernel_wait4.__do_sys_wait4
0.06 ± 49% -100.0% 0.00 perf-sched.wait_time.avg.ms.__traceiter_sched_switch.__traceiter_sched_switch.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt
0.04 ± 11% -100.0% 0.00 perf-sched.wait_time.avg.ms.__traceiter_sched_switch.__traceiter_sched_switch.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_irq_work
0.06 ± 32% -100.0% 0.00 perf-sched.wait_time.avg.ms.__traceiter_sched_switch.__traceiter_sched_switch.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_reschedule_ipi
0.05 ± 26% -100.0% 0.00 perf-sched.wait_time.avg.ms.__traceiter_sched_switch.__traceiter_sched_switch.exit_to_user_mode_prepare.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe
208.50 ±196% -100.0% 0.00 perf-sched.wait_time.avg.ms.__traceiter_sched_switch.__traceiter_sched_switch.futex_wait_queue_me.futex_wait.do_futex
53.03 -100.0% 0.00 perf-sched.wait_time.avg.ms.__traceiter_sched_switch.__traceiter_sched_switch.pipe_read.new_sync_read.vfs_read
0.05 ± 23% -100.0% 0.00 perf-sched.wait_time.avg.ms.__traceiter_sched_switch.__traceiter_sched_switch.preempt_schedule_common.__cond_resched.copy_page_from_iter
0.06 ± 47% -100.0% 0.00 perf-sched.wait_time.avg.ms.__traceiter_sched_switch.__traceiter_sched_switch.preempt_schedule_common.__cond_resched.copy_page_to_iter
0.05 ± 16% -100.0% 0.00 perf-sched.wait_time.avg.ms.__traceiter_sched_switch.__traceiter_sched_switch.preempt_schedule_common.__cond_resched.mutex_lock
5.67 ± 3% -100.0% 0.00 perf-sched.wait_time.avg.ms.__traceiter_sched_switch.__traceiter_sched_switch.preempt_schedule_common.__cond_resched.stop_one_cpu
2.77 ± 4% -100.0% 0.00 perf-sched.wait_time.avg.ms.__traceiter_sched_switch.__traceiter_sched_switch.rcu_gp_kthread.kthread.ret_from_fork
247.85 ± 48% -100.0% 0.00 perf-sched.wait_time.avg.ms.__traceiter_sched_switch.__traceiter_sched_switch.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait
580.11 ± 6% -100.0% 0.00 perf-sched.wait_time.avg.ms.__traceiter_sched_switch.__traceiter_sched_switch.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop
677.35 ± 30% -100.0% 0.00 perf-sched.wait_time.avg.ms.__traceiter_sched_switch.__traceiter_sched_switch.schedule_timeout.io_schedule_timeout.wait_for_completion_io
478.69 -100.0% 0.00 perf-sched.wait_time.avg.ms.__traceiter_sched_switch.__traceiter_sched_switch.schedule_timeout.kcompactd.kthread
0.01 ±110% -100.0% 0.00 perf-sched.wait_time.avg.ms.__traceiter_sched_switch.__traceiter_sched_switch.schedule_timeout.khugepaged.kthread
12.47 ± 3% -100.0% 0.00 perf-sched.wait_time.avg.ms.__traceiter_sched_switch.__traceiter_sched_switch.schedule_timeout.rcu_gp_kthread.kthread
687.30 ± 2% -100.0% 0.00 perf-sched.wait_time.avg.ms.__traceiter_sched_switch.__traceiter_sched_switch.smpboot_thread_fn.kthread.ret_from_fork
436.96 ± 11% -100.0% 0.00 perf-sched.wait_time.avg.ms.__traceiter_sched_switch.__traceiter_sched_switch.worker_thread.kthread.ret_from_fork
999.69 -100.0% 0.00 perf-sched.wait_time.max.ms.__traceiter_sched_switch.__traceiter_sched_switch.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe
1537 ± 5% -100.0% 0.00 perf-sched.wait_time.max.ms.__traceiter_sched_switch.__traceiter_sched_switch.devkmsg_read.vfs_read.ksys_read
1000 -100.0% 0.00 perf-sched.wait_time.max.ms.__traceiter_sched_switch.__traceiter_sched_switch.do_nanosleep.hrtimer_nanosleep.__x64_sys_nanosleep
1537 ± 5% -100.0% 0.00 perf-sched.wait_time.max.ms.__traceiter_sched_switch.__traceiter_sched_switch.do_syslog.part.0
1966 ± 59% -100.0% 0.00 perf-sched.wait_time.max.ms.__traceiter_sched_switch.__traceiter_sched_switch.do_task_dead.do_exit.do_group_exit
20.30 -100.0% 0.00 perf-sched.wait_time.max.ms.__traceiter_sched_switch.__traceiter_sched_switch.do_wait.kernel_wait4.__do_sys_wait4
0.46 ±162% -100.0% 0.00 perf-sched.wait_time.max.ms.__traceiter_sched_switch.__traceiter_sched_switch.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt
0.05 ± 13% -100.0% 0.00 perf-sched.wait_time.max.ms.__traceiter_sched_switch.__traceiter_sched_switch.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_irq_work
0.13 ±115% -100.0% 0.00 perf-sched.wait_time.max.ms.__traceiter_sched_switch.__traceiter_sched_switch.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_reschedule_ipi
0.48 ±164% -100.0% 0.00 perf-sched.wait_time.max.ms.__traceiter_sched_switch.__traceiter_sched_switch.exit_to_user_mode_prepare.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe
1152 ±213% -100.0% 0.00 perf-sched.wait_time.max.ms.__traceiter_sched_switch.__traceiter_sched_switch.futex_wait_queue_me.futex_wait.do_futex
1540 ± 5% -100.0% 0.00 perf-sched.wait_time.max.ms.__traceiter_sched_switch.__traceiter_sched_switch.pipe_read.new_sync_read.vfs_read
0.08 ± 71% -100.0% 0.00 perf-sched.wait_time.max.ms.__traceiter_sched_switch.__traceiter_sched_switch.preempt_schedule_common.__cond_resched.copy_page_from_iter
0.09 ± 73% -100.0% 0.00 perf-sched.wait_time.max.ms.__traceiter_sched_switch.__traceiter_sched_switch.preempt_schedule_common.__cond_resched.copy_page_to_iter
0.12 ± 46% -100.0% 0.00 perf-sched.wait_time.max.ms.__traceiter_sched_switch.__traceiter_sched_switch.preempt_schedule_common.__cond_resched.mutex_lock
1070 ± 14% -100.0% 0.00 perf-sched.wait_time.max.ms.__traceiter_sched_switch.__traceiter_sched_switch.preempt_schedule_common.__cond_resched.stop_one_cpu
4.50 ± 10% -100.0% 0.00 perf-sched.wait_time.max.ms.__traceiter_sched_switch.__traceiter_sched_switch.rcu_gp_kthread.kthread.ret_from_fork
2024 ± 94% -100.0% 0.00 perf-sched.wait_time.max.ms.__traceiter_sched_switch.__traceiter_sched_switch.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait
7489 -100.0% 0.00 perf-sched.wait_time.max.ms.__traceiter_sched_switch.__traceiter_sched_switch.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop
1151 ± 23% -100.0% 0.00 perf-sched.wait_time.max.ms.__traceiter_sched_switch.__traceiter_sched_switch.schedule_timeout.io_schedule_timeout.wait_for_completion_io
505.62 -100.0% 0.00 perf-sched.wait_time.max.ms.__traceiter_sched_switch.__traceiter_sched_switch.schedule_timeout.kcompactd.kthread
0.01 ±110% -100.0% 0.00 perf-sched.wait_time.max.ms.__traceiter_sched_switch.__traceiter_sched_switch.schedule_timeout.khugepaged.kthread
376.31 ± 10% -100.0% 0.00 perf-sched.wait_time.max.ms.__traceiter_sched_switch.__traceiter_sched_switch.schedule_timeout.rcu_gp_kthread.kthread
1691 ± 88% -100.0% 0.00 perf-sched.wait_time.max.ms.__traceiter_sched_switch.__traceiter_sched_switch.smpboot_thread_fn.kthread.ret_from_fork
7711 ± 21% -100.0% 0.00 perf-sched.wait_time.max.ms.__traceiter_sched_switch.__traceiter_sched_switch.worker_thread.kthread.ret_from_fork
19423 ± 16% +181.4% 54663 ± 9% softirqs.CPU0.RCU
17368 ± 14% +203.7% 52743 ± 7% softirqs.CPU1.RCU
17971 ± 17% +220.1% 57519 ± 4% softirqs.CPU10.RCU
14509 ± 18% +188.4% 41847 ± 18% softirqs.CPU100.RCU
16450 ± 21% +192.8% 48170 ± 9% softirqs.CPU101.RCU
16134 ± 22% +191.2% 46984 ± 12% softirqs.CPU102.RCU
15942 ± 23% +210.5% 49505 ± 15% softirqs.CPU103.RCU
14345 ± 18% +222.9% 46315 ± 13% softirqs.CPU104.RCU
15933 ± 22% +194.7% 46949 ± 6% softirqs.CPU105.RCU
16305 ± 22% +205.6% 49821 ± 12% softirqs.CPU106.RCU
15880 ± 24% +195.4% 46907 ± 6% softirqs.CPU107.RCU
10021 ± 14% +802.9% 90484 ± 21% softirqs.CPU108.RCU
9881 ± 11% +346.2% 44092 ± 30% softirqs.CPU109.RCU
18504 ± 11% +229.5% 60977 ± 4% softirqs.CPU11.RCU
8863 ± 8% +273.6% 33113 ± 23% softirqs.CPU110.RCU
9235 ± 7% +231.7% 30633 ± 18% softirqs.CPU111.RCU
12750 ± 17% +283.6% 48906 ± 25% softirqs.CPU112.RCU
12897 ± 17% +240.0% 43852 ± 11% softirqs.CPU113.RCU
8779 ± 5% +231.7% 29116 ± 13% softirqs.CPU114.RCU
12658 ± 26% +248.9% 44164 ± 22% softirqs.CPU115.RCU
11441 ± 11% +247.2% 39726 ± 10% softirqs.CPU116.RCU
13127 ± 15% +233.2% 43736 ± 12% softirqs.CPU117.RCU
15022 ± 18% +222.7% 48483 ± 6% softirqs.CPU118.RCU
16019 ± 21% +222.8% 51715 ± 9% softirqs.CPU119.RCU
18445 ± 13% +229.4% 60762 ± 5% softirqs.CPU12.RCU
16411 ± 22% +272.3% 61102 ± 22% softirqs.CPU120.RCU
15637 ± 21% +249.0% 54573 ± 15% softirqs.CPU121.RCU
13994 ± 20% +245.8% 48393 ± 13% softirqs.CPU122.RCU
16481 ± 17% +240.0% 56032 ± 8% softirqs.CPU123.RCU
16313 ± 22% +244.0% 56119 ± 11% softirqs.CPU124.RCU
15193 ± 19% +238.4% 51414 ± 15% softirqs.CPU125.RCU
9449 ± 22% +684.6% 74140 ± 29% softirqs.CPU126.RCU
8026 ± 7% +341.6% 35447 ± 32% softirqs.CPU127.RCU
9925 ± 12% +316.8% 41374 ± 26% softirqs.CPU128.RCU
10407 ± 10% +284.2% 39989 ± 17% softirqs.CPU129.RCU
18780 ± 8% +211.9% 58578 ± 8% softirqs.CPU13.RCU
12368 ± 14% +234.7% 41402 ± 8% softirqs.CPU130.RCU
13023 ± 13% +212.5% 40692 ± 12% softirqs.CPU131.RCU
12325 ± 16% +210.6% 38280 ± 13% softirqs.CPU132.RCU
12901 ± 17% +328.4% 55266 ± 25% softirqs.CPU133.RCU
11583 ± 18% +274.1% 43338 ± 14% softirqs.CPU134.RCU
12702 ± 14% +269.0% 46874 ± 13% softirqs.CPU135.RCU
14088 ± 18% +234.1% 47075 ± 9% softirqs.CPU136.RCU
15252 ± 19% +254.4% 54060 ± 17% softirqs.CPU137.RCU
14751 ± 21% +247.2% 51225 ± 13% softirqs.CPU138.RCU
14324 ± 20% +241.1% 48856 ± 9% softirqs.CPU139.RCU
16718 ± 24% +256.5% 59604 ± 4% softirqs.CPU14.RCU
13748 ± 16% +249.7% 48071 ± 9% softirqs.CPU140.RCU
13603 ± 22% +276.9% 51270 ± 7% softirqs.CPU141.RCU
15426 ± 19% +231.4% 51124 ± 8% softirqs.CPU142.RCU
14208 ± 17% +246.0% 49156 ± 9% softirqs.CPU143.RCU
18660 ± 10% +208.6% 57578 ± 7% softirqs.CPU15.RCU
7546 ± 10% +338.4% 33085 ± 56% softirqs.CPU16.RCU
8489 ± 7% +304.3% 34320 ± 55% softirqs.CPU17.RCU
15143 ± 21% +1246.1% 203848 ± 12% softirqs.CPU18.RCU
40325 ± 3% -21.5% 31673 ± 7% softirqs.CPU18.SCHED
12313 ± 12% +435.6% 65947 ± 21% softirqs.CPU19.RCU
17395 ± 9% +207.3% 53452 ± 9% softirqs.CPU2.RCU
10912 ± 12% +233.8% 36423 ± 9% softirqs.CPU20.RCU
11902 ± 9% +246.3% 41217 ± 15% softirqs.CPU21.RCU
14465 ± 18% +275.2% 54269 ± 46% softirqs.CPU22.RCU
14245 ± 18% +261.1% 51444 ± 24% softirqs.CPU23.RCU
13959 ± 21% +162.6% 36661 ± 14% softirqs.CPU24.RCU
14336 ± 18% +177.9% 39840 ± 12% softirqs.CPU25.RCU
12796 ± 15% +273.2% 47753 ± 27% softirqs.CPU26.RCU
14604 ± 20% +248.6% 50910 ± 34% softirqs.CPU27.RCU
15773 ± 19% +181.0% 44330 ± 11% softirqs.CPU28.RCU
16353 ± 20% +225.3% 53200 ± 15% softirqs.CPU29.RCU
16735 ± 17% +243.5% 57482 ± 4% softirqs.CPU3.RCU
16013 ± 24% +190.4% 46509 ± 6% softirqs.CPU30.RCU
16103 ± 21% +242.3% 55121 ± 36% softirqs.CPU31.RCU
9610 ± 11% +349.1% 43162 ± 50% softirqs.CPU32.RCU
11333 ± 12% +233.9% 37847 ± 15% softirqs.CPU33.RCU
11324 ± 14% +247.2% 39320 ± 9% softirqs.CPU34.RCU
11168 ± 15% +210.2% 34641 ± 20% softirqs.CPU35.RCU
14767 ± 20% +1293.9% 205842 ± 19% softirqs.CPU36.RCU
12741 ± 15% +449.9% 70065 ± 13% softirqs.CPU37.RCU
11457 ± 12% +379.8% 54980 ± 31% softirqs.CPU38.RCU
12440 ± 15% +307.9% 50748 ± 16% softirqs.CPU39.RCU
17911 ± 9% +184.9% 51035 ± 19% softirqs.CPU4.RCU
13898 ± 19% +301.5% 55802 ± 21% softirqs.CPU40.RCU
14268 ± 15% +244.9% 49204 ± 20% softirqs.CPU41.RCU
9776 ± 4% +260.8% 35272 ± 24% softirqs.CPU42.RCU
12653 ± 13% +275.9% 47564 ± 20% softirqs.CPU43.RCU
12635 ± 15% +267.8% 46472 ± 11% softirqs.CPU44.RCU
14269 ± 17% +234.5% 47725 ± 15% softirqs.CPU45.RCU
14850 ± 17% +229.5% 48938 ± 14% softirqs.CPU46.RCU
14489 ± 20% +235.0% 48536 ± 14% softirqs.CPU47.RCU
12510 ± 16% +294.0% 49293 ± 20% softirqs.CPU48.RCU
11967 ± 9% +237.0% 40330 ± 14% softirqs.CPU49.RCU
18541 ± 14% +195.2% 54740 ± 6% softirqs.CPU5.RCU
10043 ± 8% +296.6% 39831 ± 32% softirqs.CPU50.RCU
12549 ± 17% +243.7% 43135 ± 8% softirqs.CPU51.RCU
13016 ± 16% +231.6% 43156 ± 15% softirqs.CPU52.RCU
11202 ± 13% +244.3% 38567 ± 19% softirqs.CPU53.RCU
11751 ± 24% +1443.5% 181385 ± 11% softirqs.CPU54.RCU
40026 ± 2% -17.4% 33063 ± 6% softirqs.CPU54.SCHED
10619 ± 13% +557.8% 69857 ± 30% softirqs.CPU55.RCU
11473 ± 20% +328.5% 49168 ± 29% softirqs.CPU56.RCU
10539 ± 11% +337.0% 46052 ± 19% softirqs.CPU57.RCU
12270 ± 13% +256.0% 43680 ± 16% softirqs.CPU58.RCU
12315 ± 14% +246.1% 42625 ± 18% softirqs.CPU59.RCU
17153 ± 13% +216.0% 54205 ± 7% softirqs.CPU6.RCU
11927 ± 13% +289.7% 46480 ± 24% softirqs.CPU60.RCU
12446 ± 17% +315.3% 51691 ± 48% softirqs.CPU61.RCU
11852 ± 17% +267.2% 43518 ± 23% softirqs.CPU62.RCU
12387 ± 13% +251.1% 43496 ± 11% softirqs.CPU63.RCU
10996 ± 14% +232.5% 36563 ± 18% softirqs.CPU64.RCU
12143 ± 23% +262.1% 43967 ± 16% softirqs.CPU65.RCU
12599 ± 19% +274.8% 47226 ± 22% softirqs.CPU66.RCU
11427 ± 16% +285.8% 44090 ± 24% softirqs.CPU67.RCU
10829 ± 11% +257.9% 38760 ± 25% softirqs.CPU68.RCU
12273 ± 14% +262.0% 44435 ± 9% softirqs.CPU69.RCU
17499 ± 8% +227.2% 57257 ± 6% softirqs.CPU7.RCU
13200 ± 17% +232.9% 43942 ± 7% softirqs.CPU70.RCU
10909 ± 11% +240.2% 37115 ± 10% softirqs.CPU71.RCU
16827 ± 21% +144.9% 41204 softirqs.CPU71.SCHED
13148 ± 5% +222.0% 42332 ± 4% softirqs.CPU72.RCU
12524 ± 17% +233.6% 41781 ± 3% softirqs.CPU73.RCU
11992 ± 9% +228.0% 39332 ± 4% softirqs.CPU74.RCU
12262 ± 10% +224.1% 39743 ± 6% softirqs.CPU75.RCU
12221 ± 8% +215.6% 38564 ± 5% softirqs.CPU76.RCU
12596 ± 8% +213.9% 39538 ± 7% softirqs.CPU77.RCU
12097 ± 7% +228.9% 39794 ± 4% softirqs.CPU78.RCU
11903 ± 7% +218.4% 37895 ± 6% softirqs.CPU79.RCU
17361 ± 12% +221.1% 55751 ± 7% softirqs.CPU8.RCU
12445 ± 8% +221.8% 40052 ± 7% softirqs.CPU80.RCU
12004 ± 7% +225.0% 39015 ± 4% softirqs.CPU81.RCU
12000 ± 6% +229.3% 39517 ± 8% softirqs.CPU82.RCU
12504 ± 9% +219.9% 40003 ± 6% softirqs.CPU83.RCU
12440 ± 8% +216.1% 39328 ± 4% softirqs.CPU84.RCU
12262 ± 8% +230.4% 40513 ± 7% softirqs.CPU85.RCU
12229 ± 9% +225.8% 39845 ± 8% softirqs.CPU86.RCU
11983 ± 7% +234.5% 40084 ± 5% softirqs.CPU87.RCU
7843 ± 8% +252.7% 27661 ± 27% softirqs.CPU88.RCU
8306 ± 12% +243.3% 28516 ± 23% softirqs.CPU89.RCU
17622 ± 11% +230.0% 58148 ± 4% softirqs.CPU9.RCU
12073 ± 23% +619.4% 86857 ± 23% softirqs.CPU90.RCU
10332 ± 19% +290.2% 40318 ± 27% softirqs.CPU91.RCU
8983 ± 8% +221.7% 28896 ± 17% softirqs.CPU92.RCU
9696 ± 10% +239.8% 32944 ± 15% softirqs.CPU93.RCU
10886 ± 12% +245.6% 37620 ± 21% softirqs.CPU94.RCU
10441 ± 15% +214.6% 32849 ± 16% softirqs.CPU95.RCU
12834 ± 20% +158.0% 33113 ± 9% softirqs.CPU96.RCU
13146 ± 17% +159.9% 34160 ± 8% softirqs.CPU97.RCU
10393 ± 8% +267.4% 38183 ± 19% softirqs.CPU98.RCU
12492 ± 17% +277.9% 47207 ± 26% softirqs.CPU99.RCU
1898417 ± 11% +276.1% 7139397 softirqs.RCU
24540 ± 2% +96.5% 48221 softirqs.TIMER
4440 ± 54% -100.0% 0.00 interrupts.CPU0.NMI:Non-maskable_interrupts
4440 ± 54% -100.0% 0.00 interrupts.CPU0.PMI:Performance_monitoring_interrupts
2777 ±126% -62.7% 1037 ± 6% interrupts.CPU1.CAL:Function_call_interrupts
6995 ± 26% -100.0% 0.00 interrupts.CPU1.NMI:Non-maskable_interrupts
6995 ± 26% -100.0% 0.00 interrupts.CPU1.PMI:Performance_monitoring_interrupts
5665 ± 38% -100.0% 0.00 interrupts.CPU10.NMI:Non-maskable_interrupts
5665 ± 38% -100.0% 0.00 interrupts.CPU10.PMI:Performance_monitoring_interrupts
149.50 ± 35% -100.0% 0.00 interrupts.CPU100.NMI:Non-maskable_interrupts
149.50 ± 35% -100.0% 0.00 interrupts.CPU100.PMI:Performance_monitoring_interrupts
118.17 ± 35% -100.0% 0.00 interrupts.CPU101.NMI:Non-maskable_interrupts
118.17 ± 35% -100.0% 0.00 interrupts.CPU101.PMI:Performance_monitoring_interrupts
162.17 ± 31% -100.0% 0.00 interrupts.CPU102.NMI:Non-maskable_interrupts
162.17 ± 31% -100.0% 0.00 interrupts.CPU102.PMI:Performance_monitoring_interrupts
141.33 ± 32% -100.0% 0.00 interrupts.CPU103.NMI:Non-maskable_interrupts
141.33 ± 32% -100.0% 0.00 interrupts.CPU103.PMI:Performance_monitoring_interrupts
152.50 ± 37% -100.0% 0.00 interrupts.CPU104.NMI:Non-maskable_interrupts
152.50 ± 37% -100.0% 0.00 interrupts.CPU104.PMI:Performance_monitoring_interrupts
151.67 ± 37% -100.0% 0.00 interrupts.CPU105.NMI:Non-maskable_interrupts
151.67 ± 37% -100.0% 0.00 interrupts.CPU105.PMI:Performance_monitoring_interrupts
152.67 ± 37% -100.0% 0.00 interrupts.CPU106.NMI:Non-maskable_interrupts
152.67 ± 37% -100.0% 0.00 interrupts.CPU106.PMI:Performance_monitoring_interrupts
203.50 ± 34% -100.0% 0.00 interrupts.CPU107.NMI:Non-maskable_interrupts
203.50 ± 34% -100.0% 0.00 interrupts.CPU107.PMI:Performance_monitoring_interrupts
223.83 ± 40% -100.0% 0.00 interrupts.CPU108.NMI:Non-maskable_interrupts
223.83 ± 40% -100.0% 0.00 interrupts.CPU108.PMI:Performance_monitoring_interrupts
174.50 ± 29% -99.4% 1.00 ±223% interrupts.CPU109.NMI:Non-maskable_interrupts
174.50 ± 29% -99.4% 1.00 ±223% interrupts.CPU109.PMI:Performance_monitoring_interrupts
4690 ± 55% -100.0% 0.00 interrupts.CPU11.NMI:Non-maskable_interrupts
4690 ± 55% -100.0% 0.00 interrupts.CPU11.PMI:Performance_monitoring_interrupts
147.67 ± 28% -100.0% 0.00 interrupts.CPU110.NMI:Non-maskable_interrupts
147.67 ± 28% -100.0% 0.00 interrupts.CPU110.PMI:Performance_monitoring_interrupts
149.67 ± 38% -100.0% 0.00 interrupts.CPU111.NMI:Non-maskable_interrupts
149.67 ± 38% -100.0% 0.00 interrupts.CPU111.PMI:Performance_monitoring_interrupts
133.17 ± 36% -100.0% 0.00 interrupts.CPU112.NMI:Non-maskable_interrupts
133.17 ± 36% -100.0% 0.00 interrupts.CPU112.PMI:Performance_monitoring_interrupts
160.83 ± 26% -100.0% 0.00 interrupts.CPU113.NMI:Non-maskable_interrupts
160.83 ± 26% -100.0% 0.00 interrupts.CPU113.PMI:Performance_monitoring_interrupts
129.33 ± 38% -100.0% 0.00 interrupts.CPU114.NMI:Non-maskable_interrupts
129.33 ± 38% -100.0% 0.00 interrupts.CPU114.PMI:Performance_monitoring_interrupts
147.00 ± 37% -100.0% 0.00 interrupts.CPU115.NMI:Non-maskable_interrupts
147.00 ± 37% -100.0% 0.00 interrupts.CPU115.PMI:Performance_monitoring_interrupts
112.00 ± 36% -100.0% 0.00 interrupts.CPU116.NMI:Non-maskable_interrupts
112.00 ± 36% -100.0% 0.00 interrupts.CPU116.PMI:Performance_monitoring_interrupts
133.00 ± 39% -100.0% 0.00 interrupts.CPU117.NMI:Non-maskable_interrupts
133.00 ± 39% -100.0% 0.00 interrupts.CPU117.PMI:Performance_monitoring_interrupts
98.00 ± 8% -100.0% 0.00 interrupts.CPU118.NMI:Non-maskable_interrupts
98.00 ± 8% -100.0% 0.00 interrupts.CPU118.PMI:Performance_monitoring_interrupts
131.17 ± 30% -100.0% 0.00 interrupts.CPU119.NMI:Non-maskable_interrupts
131.17 ± 30% -100.0% 0.00 interrupts.CPU119.PMI:Performance_monitoring_interrupts
5913 ± 32% -100.0% 0.00 interrupts.CPU12.NMI:Non-maskable_interrupts
5913 ± 32% -100.0% 0.00 interrupts.CPU12.PMI:Performance_monitoring_interrupts
116.83 ± 35% -100.0% 0.00 interrupts.CPU120.NMI:Non-maskable_interrupts
116.83 ± 35% -100.0% 0.00 interrupts.CPU120.PMI:Performance_monitoring_interrupts
113.00 ± 21% -100.0% 0.00 interrupts.CPU121.NMI:Non-maskable_interrupts
113.00 ± 21% -100.0% 0.00 interrupts.CPU121.PMI:Performance_monitoring_interrupts
98.83 ± 6% -100.0% 0.00 interrupts.CPU122.NMI:Non-maskable_interrupts
98.83 ± 6% -100.0% 0.00 interrupts.CPU122.PMI:Performance_monitoring_interrupts
142.00 ± 61% -100.0% 0.00 interrupts.CPU123.NMI:Non-maskable_interrupts
142.00 ± 61% -100.0% 0.00 interrupts.CPU123.PMI:Performance_monitoring_interrupts
119.17 ± 39% -100.0% 0.00 interrupts.CPU124.NMI:Non-maskable_interrupts
119.17 ± 39% -100.0% 0.00 interrupts.CPU124.PMI:Performance_monitoring_interrupts
133.67 ± 38% -100.0% 0.00 interrupts.CPU125.NMI:Non-maskable_interrupts
133.67 ± 38% -100.0% 0.00 interrupts.CPU125.PMI:Performance_monitoring_interrupts
173.17 ± 48% -100.0% 0.00 interrupts.CPU126.NMI:Non-maskable_interrupts
173.17 ± 48% -100.0% 0.00 interrupts.CPU126.PMI:Performance_monitoring_interrupts
137.00 ± 32% -100.0% 0.00 interrupts.CPU127.NMI:Non-maskable_interrupts
137.00 ± 32% -100.0% 0.00 interrupts.CPU127.PMI:Performance_monitoring_interrupts
117.83 ± 32% -100.0% 0.00 interrupts.CPU128.NMI:Non-maskable_interrupts
117.83 ± 32% -100.0% 0.00 interrupts.CPU128.PMI:Performance_monitoring_interrupts
122.33 ± 34% -100.0% 0.00 interrupts.CPU129.NMI:Non-maskable_interrupts
122.33 ± 34% -100.0% 0.00 interrupts.CPU129.PMI:Performance_monitoring_interrupts
5466 ± 50% -100.0% 0.00 interrupts.CPU13.NMI:Non-maskable_interrupts
5466 ± 50% -100.0% 0.00 interrupts.CPU13.PMI:Performance_monitoring_interrupts
101.50 ± 8% -100.0% 0.00 interrupts.CPU130.NMI:Non-maskable_interrupts
101.50 ± 8% -100.0% 0.00 interrupts.CPU130.PMI:Performance_monitoring_interrupts
116.17 ± 29% -100.0% 0.00 interrupts.CPU131.NMI:Non-maskable_interrupts
116.17 ± 29% -100.0% 0.00 interrupts.CPU131.PMI:Performance_monitoring_interrupts
215.17 ± 98% -100.0% 0.00 interrupts.CPU132.NMI:Non-maskable_interrupts
215.17 ± 98% -100.0% 0.00 interrupts.CPU132.PMI:Performance_monitoring_interrupts
114.67 ± 32% -100.0% 0.00 interrupts.CPU133.NMI:Non-maskable_interrupts
114.67 ± 32% -100.0% 0.00 interrupts.CPU133.PMI:Performance_monitoring_interrupts
164.33 ± 27% -100.0% 0.00 interrupts.CPU134.NMI:Non-maskable_interrupts
164.33 ± 27% -100.0% 0.00 interrupts.CPU134.PMI:Performance_monitoring_interrupts
118.67 ± 34% -100.0% 0.00 interrupts.CPU135.NMI:Non-maskable_interrupts
118.67 ± 34% -100.0% 0.00 interrupts.CPU135.PMI:Performance_monitoring_interrupts
131.83 ± 30% -100.0% 0.00 interrupts.CPU136.NMI:Non-maskable_interrupts
131.83 ± 30% -100.0% 0.00 interrupts.CPU136.PMI:Performance_monitoring_interrupts
118.67 ± 33% -100.0% 0.00 interrupts.CPU137.NMI:Non-maskable_interrupts
118.67 ± 33% -100.0% 0.00 interrupts.CPU137.PMI:Performance_monitoring_interrupts
122.50 ± 31% -100.0% 0.00 interrupts.CPU138.NMI:Non-maskable_interrupts
122.50 ± 31% -100.0% 0.00 interrupts.CPU138.PMI:Performance_monitoring_interrupts
100.67 ± 7% -100.0% 0.00 interrupts.CPU139.NMI:Non-maskable_interrupts
100.67 ± 7% -100.0% 0.00 interrupts.CPU139.PMI:Performance_monitoring_interrupts
5232 ± 48% -100.0% 0.00 interrupts.CPU14.NMI:Non-maskable_interrupts
5232 ± 48% -100.0% 0.00 interrupts.CPU14.PMI:Performance_monitoring_interrupts
118.33 ± 36% -100.0% 0.00 interrupts.CPU140.NMI:Non-maskable_interrupts
118.33 ± 36% -100.0% 0.00 interrupts.CPU140.PMI:Performance_monitoring_interrupts
118.00 ± 31% -100.0% 0.00 interrupts.CPU141.NMI:Non-maskable_interrupts
118.00 ± 31% -100.0% 0.00 interrupts.CPU141.PMI:Performance_monitoring_interrupts
144.83 ± 45% -100.0% 0.00 interrupts.CPU142.NMI:Non-maskable_interrupts
144.83 ± 45% -100.0% 0.00 interrupts.CPU142.PMI:Performance_monitoring_interrupts
361.83 ± 30% -100.0% 0.00 interrupts.CPU143.NMI:Non-maskable_interrupts
361.83 ± 30% -100.0% 0.00 interrupts.CPU143.PMI:Performance_monitoring_interrupts
6247 ± 35% -100.0% 0.00 interrupts.CPU15.NMI:Non-maskable_interrupts
6247 ± 35% -100.0% 0.00 interrupts.CPU15.PMI:Performance_monitoring_interrupts
425.50 ± 14% -23.4% 325.83 ± 14% interrupts.CPU15.TLB:TLB_shootdowns
172.17 ± 41% -100.0% 0.00 interrupts.CPU16.NMI:Non-maskable_interrupts
172.17 ± 41% -100.0% 0.00 interrupts.CPU16.PMI:Performance_monitoring_interrupts
164.50 ± 30% -100.0% 0.00 interrupts.CPU17.NMI:Non-maskable_interrupts
164.50 ± 30% -100.0% 0.00 interrupts.CPU17.PMI:Performance_monitoring_interrupts
181.00 ± 20% -100.0% 0.00 interrupts.CPU18.NMI:Non-maskable_interrupts
181.00 ± 20% -100.0% 0.00 interrupts.CPU18.PMI:Performance_monitoring_interrupts
159.17 ± 33% -100.0% 0.00 interrupts.CPU19.NMI:Non-maskable_interrupts
159.17 ± 33% -100.0% 0.00 interrupts.CPU19.PMI:Performance_monitoring_interrupts
6587 ± 33% -100.0% 0.00 interrupts.CPU2.NMI:Non-maskable_interrupts
6587 ± 33% -100.0% 0.00 interrupts.CPU2.PMI:Performance_monitoring_interrupts
147.50 ± 28% -100.0% 0.00 interrupts.CPU20.NMI:Non-maskable_interrupts
147.50 ± 28% -100.0% 0.00 interrupts.CPU20.PMI:Performance_monitoring_interrupts
160.67 ± 23% -100.0% 0.00 interrupts.CPU21.NMI:Non-maskable_interrupts
160.67 ± 23% -100.0% 0.00 interrupts.CPU21.PMI:Performance_monitoring_interrupts
183.33 ± 17% -100.0% 0.00 interrupts.CPU22.NMI:Non-maskable_interrupts
183.33 ± 17% -100.0% 0.00 interrupts.CPU22.PMI:Performance_monitoring_interrupts
156.67 ± 38% -100.0% 0.00 interrupts.CPU23.NMI:Non-maskable_interrupts
156.67 ± 38% -100.0% 0.00 interrupts.CPU23.PMI:Performance_monitoring_interrupts
164.67 ± 26% -100.0% 0.00 interrupts.CPU24.NMI:Non-maskable_interrupts
164.67 ± 26% -100.0% 0.00 interrupts.CPU24.PMI:Performance_monitoring_interrupts
176.50 ± 26% -100.0% 0.00 interrupts.CPU25.NMI:Non-maskable_interrupts
176.50 ± 26% -100.0% 0.00 interrupts.CPU25.PMI:Performance_monitoring_interrupts
153.50 ± 33% -100.0% 0.00 interrupts.CPU26.NMI:Non-maskable_interrupts
153.50 ± 33% -100.0% 0.00 interrupts.CPU26.PMI:Performance_monitoring_interrupts
144.50 ± 26% -100.0% 0.00 interrupts.CPU27.NMI:Non-maskable_interrupts
144.50 ± 26% -100.0% 0.00 interrupts.CPU27.PMI:Performance_monitoring_interrupts
135.00 ± 37% -100.0% 0.00 interrupts.CPU28.NMI:Non-maskable_interrupts
135.00 ± 37% -100.0% 0.00 interrupts.CPU28.PMI:Performance_monitoring_interrupts
131.17 ± 29% -100.0% 0.00 interrupts.CPU29.NMI:Non-maskable_interrupts
131.17 ± 29% -100.0% 0.00 interrupts.CPU29.PMI:Performance_monitoring_interrupts
6274 ± 42% -100.0% 0.00 interrupts.CPU3.NMI:Non-maskable_interrupts
6274 ± 42% -100.0% 0.00 interrupts.CPU3.PMI:Performance_monitoring_interrupts
147.33 ± 29% -100.0% 0.00 interrupts.CPU30.NMI:Non-maskable_interrupts
147.33 ± 29% -100.0% 0.00 interrupts.CPU30.PMI:Performance_monitoring_interrupts
118.00 ± 21% -100.0% 0.00 interrupts.CPU31.NMI:Non-maskable_interrupts
118.00 ± 21% -100.0% 0.00 interrupts.CPU31.PMI:Performance_monitoring_interrupts
137.83 ± 34% -100.0% 0.00 interrupts.CPU32.NMI:Non-maskable_interrupts
137.83 ± 34% -100.0% 0.00 interrupts.CPU32.PMI:Performance_monitoring_interrupts
131.67 ± 31% -100.0% 0.00 interrupts.CPU33.NMI:Non-maskable_interrupts
131.67 ± 31% -100.0% 0.00 interrupts.CPU33.PMI:Performance_monitoring_interrupts
152.33 ± 30% -100.0% 0.00 interrupts.CPU34.NMI:Non-maskable_interrupts
152.33 ± 30% -100.0% 0.00 interrupts.CPU34.PMI:Performance_monitoring_interrupts
177.33 ± 54% -100.0% 0.00 interrupts.CPU35.NMI:Non-maskable_interrupts
177.33 ± 54% -100.0% 0.00 interrupts.CPU35.PMI:Performance_monitoring_interrupts
173.33 ± 45% -100.0% 0.00 interrupts.CPU36.NMI:Non-maskable_interrupts
173.33 ± 45% -100.0% 0.00 interrupts.CPU36.PMI:Performance_monitoring_interrupts
143.50 ± 38% -100.0% 0.00 interrupts.CPU37.NMI:Non-maskable_interrupts
143.50 ± 38% -100.0% 0.00 interrupts.CPU37.PMI:Performance_monitoring_interrupts
124.33 ± 27% -100.0% 0.00 interrupts.CPU38.NMI:Non-maskable_interrupts
124.33 ± 27% -100.0% 0.00 interrupts.CPU38.PMI:Performance_monitoring_interrupts
145.67 ± 29% -100.0% 0.00 interrupts.CPU39.NMI:Non-maskable_interrupts
145.67 ± 29% -100.0% 0.00 interrupts.CPU39.PMI:Performance_monitoring_interrupts
5122 ± 38% -100.0% 0.00 interrupts.CPU4.NMI:Non-maskable_interrupts
5122 ± 38% -100.0% 0.00 interrupts.CPU4.PMI:Performance_monitoring_interrupts
142.67 ± 29% -99.3% 1.00 ±223% interrupts.CPU40.NMI:Non-maskable_interrupts
142.67 ± 29% -99.3% 1.00 ±223% interrupts.CPU40.PMI:Performance_monitoring_interrupts
151.50 ± 26% -100.0% 0.00 interrupts.CPU41.NMI:Non-maskable_interrupts
151.50 ± 26% -100.0% 0.00 interrupts.CPU41.PMI:Performance_monitoring_interrupts
150.67 ± 29% -100.0% 0.00 interrupts.CPU42.NMI:Non-maskable_interrupts
150.67 ± 29% -100.0% 0.00 interrupts.CPU42.PMI:Performance_monitoring_interrupts
143.50 ± 30% -100.0% 0.00 interrupts.CPU43.NMI:Non-maskable_interrupts
143.50 ± 30% -100.0% 0.00 interrupts.CPU43.PMI:Performance_monitoring_interrupts
160.67 ± 26% -100.0% 0.00 interrupts.CPU44.NMI:Non-maskable_interrupts
160.67 ± 26% -100.0% 0.00 interrupts.CPU44.PMI:Performance_monitoring_interrupts
122.67 ± 30% -100.0% 0.00 interrupts.CPU45.NMI:Non-maskable_interrupts
122.67 ± 30% -100.0% 0.00 interrupts.CPU45.PMI:Performance_monitoring_interrupts
177.17 ± 20% -100.0% 0.00 interrupts.CPU46.NMI:Non-maskable_interrupts
177.17 ± 20% -100.0% 0.00 interrupts.CPU46.PMI:Performance_monitoring_interrupts
186.83 ± 24% -100.0% 0.00 interrupts.CPU47.NMI:Non-maskable_interrupts
186.83 ± 24% -100.0% 0.00 interrupts.CPU47.PMI:Performance_monitoring_interrupts
141.00 ± 23% -100.0% 0.00 interrupts.CPU48.NMI:Non-maskable_interrupts
141.00 ± 23% -100.0% 0.00 interrupts.CPU48.PMI:Performance_monitoring_interrupts
170.50 ± 30% -100.0% 0.00 interrupts.CPU49.NMI:Non-maskable_interrupts
170.50 ± 30% -100.0% 0.00 interrupts.CPU49.PMI:Performance_monitoring_interrupts
5397 ± 51% -100.0% 0.00 interrupts.CPU5.NMI:Non-maskable_interrupts
5397 ± 51% -100.0% 0.00 interrupts.CPU5.PMI:Performance_monitoring_interrupts
169.17 ± 23% -100.0% 0.00 interrupts.CPU50.NMI:Non-maskable_interrupts
169.17 ± 23% -100.0% 0.00 interrupts.CPU50.PMI:Performance_monitoring_interrupts
181.83 ± 19% -100.0% 0.00 interrupts.CPU51.NMI:Non-maskable_interrupts
181.83 ± 19% -100.0% 0.00 interrupts.CPU51.PMI:Performance_monitoring_interrupts
136.67 ± 22% -100.0% 0.00 interrupts.CPU52.NMI:Non-maskable_interrupts
136.67 ± 22% -100.0% 0.00 interrupts.CPU52.PMI:Performance_monitoring_interrupts
183.17 ± 19% -100.0% 0.00 interrupts.CPU53.NMI:Non-maskable_interrupts
183.17 ± 19% -100.0% 0.00 interrupts.CPU53.PMI:Performance_monitoring_interrupts
197.50 ± 47% -100.0% 0.00 interrupts.CPU54.NMI:Non-maskable_interrupts
197.50 ± 47% -100.0% 0.00 interrupts.CPU54.PMI:Performance_monitoring_interrupts
173.00 ± 32% -100.0% 0.00 interrupts.CPU55.NMI:Non-maskable_interrupts
173.00 ± 32% -100.0% 0.00 interrupts.CPU55.PMI:Performance_monitoring_interrupts
162.17 ± 22% -100.0% 0.00 interrupts.CPU56.NMI:Non-maskable_interrupts
162.17 ± 22% -100.0% 0.00 interrupts.CPU56.PMI:Performance_monitoring_interrupts
155.00 ± 27% -100.0% 0.00 interrupts.CPU57.NMI:Non-maskable_interrupts
155.00 ± 27% -100.0% 0.00 interrupts.CPU57.PMI:Performance_monitoring_interrupts
171.50 ± 19% -100.0% 0.00 interrupts.CPU58.NMI:Non-maskable_interrupts
171.50 ± 19% -100.0% 0.00 interrupts.CPU58.PMI:Performance_monitoring_interrupts
182.33 ± 16% -100.0% 0.00 interrupts.CPU59.NMI:Non-maskable_interrupts
182.33 ± 16% -100.0% 0.00 interrupts.CPU59.PMI:Performance_monitoring_interrupts
5701 ± 45% -100.0% 1.00 ±223% interrupts.CPU6.NMI:Non-maskable_interrupts
5701 ± 45% -100.0% 1.00 ±223% interrupts.CPU6.PMI:Performance_monitoring_interrupts
207.50 ± 44% -100.0% 0.00 interrupts.CPU60.NMI:Non-maskable_interrupts
207.50 ± 44% -100.0% 0.00 interrupts.CPU60.PMI:Performance_monitoring_interrupts
175.00 ± 25% -100.0% 0.00 interrupts.CPU61.NMI:Non-maskable_interrupts
175.00 ± 25% -100.0% 0.00 interrupts.CPU61.PMI:Performance_monitoring_interrupts
160.17 ± 25% -100.0% 0.00 interrupts.CPU62.NMI:Non-maskable_interrupts
160.17 ± 25% -100.0% 0.00 interrupts.CPU62.PMI:Performance_monitoring_interrupts
175.00 ± 18% -100.0% 0.00 interrupts.CPU63.NMI:Non-maskable_interrupts
175.00 ± 18% -100.0% 0.00 interrupts.CPU63.PMI:Performance_monitoring_interrupts
133.33 ± 34% -100.0% 0.00 interrupts.CPU64.NMI:Non-maskable_interrupts
133.33 ± 34% -100.0% 0.00 interrupts.CPU64.PMI:Performance_monitoring_interrupts
160.00 ± 23% -100.0% 0.00 interrupts.CPU65.NMI:Non-maskable_interrupts
160.00 ± 23% -100.0% 0.00 interrupts.CPU65.PMI:Performance_monitoring_interrupts
171.67 ± 24% -100.0% 0.00 interrupts.CPU66.NMI:Non-maskable_interrupts
171.67 ± 24% -100.0% 0.00 interrupts.CPU66.PMI:Performance_monitoring_interrupts
197.00 ± 19% -100.0% 0.00 interrupts.CPU67.NMI:Non-maskable_interrupts
197.00 ± 19% -100.0% 0.00 interrupts.CPU67.PMI:Performance_monitoring_interrupts
181.67 ± 19% -100.0% 0.00 interrupts.CPU68.NMI:Non-maskable_interrupts
181.67 ± 19% -100.0% 0.00 interrupts.CPU68.PMI:Performance_monitoring_interrupts
189.67 ± 23% -100.0% 0.00 interrupts.CPU69.NMI:Non-maskable_interrupts
189.67 ± 23% -100.0% 0.00 interrupts.CPU69.PMI:Performance_monitoring_interrupts
5634 ± 32% -100.0% 0.00 interrupts.CPU7.NMI:Non-maskable_interrupts
5634 ± 32% -100.0% 0.00 interrupts.CPU7.PMI:Performance_monitoring_interrupts
168.67 ± 26% -100.0% 0.00 interrupts.CPU70.NMI:Non-maskable_interrupts
168.67 ± 26% -100.0% 0.00 interrupts.CPU70.PMI:Performance_monitoring_interrupts
351.83 ± 29% -100.0% 0.00 interrupts.CPU71.NMI:Non-maskable_interrupts
351.83 ± 29% -100.0% 0.00 interrupts.CPU71.PMI:Performance_monitoring_interrupts
5467 ± 50% -100.0% 0.00 interrupts.CPU72.NMI:Non-maskable_interrupts
5467 ± 50% -100.0% 0.00 interrupts.CPU72.PMI:Performance_monitoring_interrupts
3198 ± 44% -100.0% 0.00 interrupts.CPU73.NMI:Non-maskable_interrupts
3198 ± 44% -100.0% 0.00 interrupts.CPU73.PMI:Performance_monitoring_interrupts
3548 ± 30% -100.0% 0.00 interrupts.CPU74.NMI:Non-maskable_interrupts
3548 ± 30% -100.0% 0.00 interrupts.CPU74.PMI:Performance_monitoring_interrupts
2786 ± 38% -100.0% 0.00 interrupts.CPU75.NMI:Non-maskable_interrupts
2786 ± 38% -100.0% 0.00 interrupts.CPU75.PMI:Performance_monitoring_interrupts
3942 ± 61% -100.0% 0.00 interrupts.CPU76.NMI:Non-maskable_interrupts
3942 ± 61% -100.0% 0.00 interrupts.CPU76.PMI:Performance_monitoring_interrupts
5265 ± 45% -100.0% 1.00 ±223% interrupts.CPU77.NMI:Non-maskable_interrupts
5265 ± 45% -100.0% 1.00 ±223% interrupts.CPU77.PMI:Performance_monitoring_interrupts
3939 ± 58% -100.0% 0.00 interrupts.CPU78.NMI:Non-maskable_interrupts
3939 ± 58% -100.0% 0.00 interrupts.CPU78.PMI:Performance_monitoring_interrupts
4860 ± 54% -100.0% 0.00 interrupts.CPU79.NMI:Non-maskable_interrupts
4860 ± 54% -100.0% 0.00 interrupts.CPU79.PMI:Performance_monitoring_interrupts
4825 ± 35% -100.0% 0.00 interrupts.CPU8.NMI:Non-maskable_interrupts
4825 ± 35% -100.0% 0.00 interrupts.CPU8.PMI:Performance_monitoring_interrupts
4414 ± 51% -100.0% 0.00 interrupts.CPU80.NMI:Non-maskable_interrupts
4414 ± 51% -100.0% 0.00 interrupts.CPU80.PMI:Performance_monitoring_interrupts44 lines not shown · raw message
17,039 lines not shown · raw message
Part of this message is not shown here. View the raw message.
Quoted 13 lines · Jens Axboe
Hi,
Stefan reports that attaching to a task with
io_uringwill leave gdb very confused and just repeatedly attempting to attach to the IO threads, even though it receives an-EPERMevery time. This patchset proposes to skipPF_IO_WORKERthreads assame_thread_group(), except for accounting purposes which we still desire.We also skip listing the IO threads in /proc/<pid>/task/ so that gdb doesn't think it should stop and attach to them. This makes us consistent with earlier kernels, where these async threads were not related to the ring owning task, and hence gdb (and others) ignored them anyway.
Quoted 7 lines
Seems to me that this is the right approach, but open to comments on if others agree with this. Oleg, I did see your messages as well on SIGSTOP, and as was discussed with Eric as well, this is something we most certainly can revisit. I do think that the visibility of these threads is a separate issue. Even with SIGSTOP implemented (which I did try as well), we're never going to allow ptrace attach and hence gdb would still be broken. Hence I'd rather treat them as separate issues to attack.
A quick skim shows that these threads are not showing up anywhere in proc which appears to be a problem, as it hides them from top.
Sysadmins need the ability to dig into a system and find out where all their cpu usage or io's have gone when there is a problem. I general I think this argues that these threads should show up as threads of the process so I am not even certain this is the right fix to deal with gdb.
Eric
Quoted 8 lines · Eric W. Biederman
A quick skim shows that these threads are not showing up anywhere in proc which appears to be a problem, as it hides them from top.
Sysadmins need the ability to dig into a system and find out where all their cpu usage or io's have gone when there is a problem. I general I think this argues that these threads should show up as threads of the process so I am not even certain this is the right fix to deal with gdb.
Yeah, I do think that hiding them is the wrong model, because it also hides them from "ps" etc, which is very wrong.
I don't know what the gdb logic is, but maybe there's some other option that makes gdb not react to them?
Linus
Jens Axboe
maintainerQuoted 6 lines · Linus Torvalds
Quoted 4 lines · Eric W. Biederman
A quick skim shows that these threads are not showing up anywhere in proc which appears to be a problem, as it hides them from top.
Quoted 7 lines
Quoted 4 lines
Sysadmins need the ability to dig into a system and find out where all their cpu usage or io's have gone when there is a problem. I general I think this argues that these threads should show up as threads of the process so I am not even certain this is the right fix to deal with gdb.
Yeah, I do think that hiding them is the wrong model, because it also hides them from "ps" etc, which is very wrong.
Totally agree.
Quoted 2 lines
I don't know what the gdb logic is, but maybe there's some other option that makes gdb not react to them?
Guess it's time to dig out the gdb source... I'll take a look.
Signature · 1 line
Jens Axboe
Quoted 3 lines · Linus Torvalds
I don't know what the gdb logic is, but maybe there's some other option that makes gdb not react to them?
.. maybe we could have a different name for them under the task/ subdirectory, for example (not just the pid)? Although that probably messes up 'ps' too..
Linus
Jens Axboe
maintainerQuoted 9 lines · Linus Torvalds
Quoted 3 lines · Linus Torvalds
I don't know what the gdb logic is, but maybe there's some other option that makes gdb not react to them?
.. maybe we could have a different name for them under the task/ subdirectory, for example (not just the pid)? Although that probably messes up 'ps' too..
Heh, I can try, but my guess is that it would mess up _something_, if not ps/top.
Signature · 1 line
Jens Axboe
Quoted 7 lines · Jens Axboe
Quoted 6 lines · Linus Torvalds
Quoted 3 lines · Linus Torvalds
I don't know what the gdb logic is, but maybe there's some other option that makes gdb not react to them?
Quoted 6 lines
Quoted 3 lines
.. maybe we could have a different name for them under the task/ subdirectory, for example (not just the pid)? Although that probably messes up 'ps' too..
Heh, I can try, but my guess is that it would mess up _something_, if not ps/top.
Hmm.
So looking quickly the flip side of the coin is gdb (and other debuggers) needs a way to know these threads are special, so it can know not to attach.
I suspect getting -EPERM (or possibly a different error code) when attempting attach is the right was to know that a thread is not available to be debugged.
Eric
Oleg Nesterov
maintainerQuoted 4 lines · Eric W. Biederman
So looking quickly the flip side of the coin is gdb (and other debuggers) needs a way to know these threads are special, so it can know not to attach.
may be,
Quoted 3 lines
I suspect getting
-EPERM(or possibly a different error code) when attempting attach is the right was to know that a thread is not available to be debugged.
may be.
But I don't think we can blame gdb. The kernel changed the rules, and this broke gdb. IOW, I don't agree this is gdb bug.
Oleg.
Jens Axboe
maintainerQuoted 8 lines · Oleg Nesterov
Quoted 4 lines · Eric W. Biederman
So looking quickly the flip side of the coin is gdb (and other debuggers) needs a way to know these threads are special, so it can know not to attach.
may be,
Quoted 8 lines
Quoted 3 lines
I suspect getting
-EPERM(or possibly a different error code) when attempting attach is the right was to know that a thread is not available to be debugged.may be.
But I don't think we can blame gdb. The kernel changed the rules, and this broke gdb. IOW, I don't agree this is gdb bug.
Right, that's what I was getting at too - and it's likely not just gdb. We have to ensure that we don't break this use case, which seems to imply that we:
1) Just make it work, or
2) Make them hidden in such a way that gdb doesn't see them, but regular tooling does
#2 seems fraught with peril, and maybe not even possible.
Signature · 1 line
Jens Axboe
Quoted 8 lines · Oleg Nesterov
Quoted 4 lines · Eric W. Biederman
So looking quickly the flip side of the coin is gdb (and other debuggers) needs a way to know these threads are special, so it can know not to attach.
may be,
Quoted 8 lines
Quoted 3 lines
I suspect getting
-EPERM(or possibly a different error code) when attempting attach is the right was to know that a thread is not available to be debugged.may be.
But I don't think we can blame gdb. The kernel changed the rules, and this broke gdb. IOW, I don't agree this is gdb bug.
My point would be it is not strictly a regression either. It is gdb not handling new functionality.
If we can be backwards compatible and make ptrace_attach work that is preferable. If we can't saying the handful of ptrace using applications need an upgrade to support processes that use io_uring may be acceptable.
I don't see any easy to implement path that is guaranteed to work.
Eric
Jens Axboe
maintainerQuoted 18 lines · Eric W. Biederman
Quoted 7 lines · Jens Axboe
Quoted 6 lines · Linus Torvalds
Quoted 3 lines · Linus Torvalds
I don't know what the gdb logic is, but maybe there's some other option that makes gdb not react to them?
Quoted 6 lines
Quoted 3 lines
.. maybe we could have a different name for them under the task/ subdirectory, for example (not just the pid)? Although that probably messes up 'ps' too..
Heh, I can try, but my guess is that it would mess up _something_, if not ps/top.
Hmm.
Quoted 7 lines
So looking quickly the flip side of the coin is gdb (and other debuggers) needs a way to know these threads are special, so it can know not to attach.
I suspect getting
-EPERM(or possibly a different error code) when attempting attach is the right was to know that a thread is not available to be debugged.
But that's what's being returned right now, and gdb seemingly doesn't really handle that. And even if it was just a gdb issue that could be fixed it (it definitely is), I'd still greatly prefer not having to do that. It just takes too long for packages to get updated in distros, and it'd be years until it got fixed widely. Secondly, I'm even more worried about cases that we haven't seen yet. I doubt that gdb is the only thing that'd fall over, not expecting threads in there that it cannot attach to.
Signature · 1 line
Jens Axboe
Quoted 10 lines · Linus Torvalds
Quoted 3 lines · Linus Torvalds
I don't know what the gdb logic is, but maybe there's some other option that makes gdb not react to them?
.. maybe we could have a different name for them under the task/ subdirectory, for example (not just the pid)? Although that probably messes up 'ps' too..
Actually, maybe the right model is to simply make all the io threads take signals, and get rid of all the special cases.
Sure, the signals will never be delivered to user space, but if we
- just made the thread loop do "get_signal()" when there are pending signals
- allowed ptrace_attach on them
they'd look pretty much like regular threads that just never do the user-space part of signal handling.
The whole "signals are very special for IO threads" thing has caused so many problems, that maybe the solution is simply to _not_ make them special?
Linus
Jens Axboe
maintainerQuoted 20 lines · Linus Torvalds
Quoted 10 lines · Linus Torvalds
Quoted 3 lines · Linus Torvalds
I don't know what the gdb logic is, but maybe there's some other option that makes gdb not react to them?
.. maybe we could have a different name for them under the task/ subdirectory, for example (not just the pid)? Although that probably messes up 'ps' too..
Actually, maybe the right model is to simply make all the io threads take signals, and get rid of all the special cases.
Sure, the signals will never be delivered to user space, but if we
- just made the thread loop do "
get_signal()" when there are pending signals
Quoted 8 lines
- allowed
ptrace_attachon themthey'd look pretty much like regular threads that just never do the user-space part of signal handling.
The whole "signals are very special for IO threads" thing has caused so many problems, that maybe the solution is simply to _not_ make them special?
Just to wrap up the previous one, yes it broke all sorts of things to make the 'tid' directory different. They just end up being hidden anyway through that, for both ps and top.
Yes, I do think that maybe it's better to just embrace maybe just embrace the signals, and have everything just work by default. It's better than continually trying to make the threads special. I'll see if there are some demons lurking down that path.
Signature · 1 line
Jens Axboe
Jens Axboe
maintainerQuoted 30 lines · Jens Axboe
Quoted 20 lines · Linus Torvalds
Quoted 10 lines · Linus Torvalds
Quoted 3 lines · Linus Torvalds
I don't know what the gdb logic is, but maybe there's some other option that makes gdb not react to them?
.. maybe we could have a different name for them under the task/ subdirectory, for example (not just the pid)? Although that probably messes up 'ps' too..
Actually, maybe the right model is to simply make all the io threads take signals, and get rid of all the special cases.
Sure, the signals will never be delivered to user space, but if we
- just made the thread loop do "
get_signal()" when there are pending signalsQuoted 8 lines
- allowed
ptrace_attachon themthey'd look pretty much like regular threads that just never do the user-space part of signal handling.
The whole "signals are very special for IO threads" thing has caused so many problems, that maybe the solution is simply to _not_ make them special?
Quoted 8 lines
Just to wrap up the previous one, yes it broke all sorts of things to make the 'tid' directory different. They just end up being hidden anyway through that, for both ps and top.
Yes, I do think that maybe it's better to just embrace maybe just embrace the signals, and have everything just work by default. It's better than continually trying to make the threads special. I'll see if there are some demons lurking down that path.
In the spirit of "let's just try it", I ran with the below patch. With that, I can gdb attach just fine to a test case that creates an io_uring and a regular thread with pthread_create(). The regular thread uses the ring, so you end up with two iou-mgr threads. Attach:
[root@archlinux ~]# gdb -p 360
[snip gdb noise]
Attaching to process 360
[New LWP 361]
[New LWP 362]
[New LWP 363]
warning: Selected architecture i386:x86-64 is not compatible with reported target architecture i386
warning: Architecture rejected target-supplied description
Error while reading shared library symbols for /usr/lib/libpthread.so.0:
Cannot find user-level thread for LWP 363: generic error 0x00007f7aa526e125 in clock_nanosleep@GLIBC_2.2.5 () from /usr/lib/libc.so.6 (gdb) info threads
Id Target Id Frame * 1 LWP 360 "io_uring" 0x00007f7aa526e125 in clock_nanosleep@GLIBC_2.2.5 () from /usr/lib/libc.so.6 2 LWP 361 "iou-mgr-360" 0x0000000000000000 in ?? () 3 LWP 362 "io_uring" 0x00007f7aa52a0a9d in syscall () from /usr/lib/libc.so.6 4 LWP 363 "iou-mgr-362" 0x0000000000000000 in ?? ()
(gdb) thread 2
[Switching to thread 2 (LWP 361)]
#0 0x0000000000000000 in ?? ()
(gdb) bt
#0 0x0000000000000000 in ?? ()
Backtrace stopped: Cannot access memory at address 0x0
(gdb) cont
Continuing.
^C
Thread 1 "io_uring" received signal SIGINT, Interrupt. [Switching to LWP 360]0x00007f7aa526e125 in clock_nanosleep@GLIBC_2.2.5 () from /usr/lib/libc.so.6 (gdb) q
A debugging session is active.
Inferior 1 [process 360] will be detached.
Quit anyway? (y or n) y
Detaching from program: /root/git/fio/t/io_uring, process 360 [Inferior 1 (process 360) detached]
The iou-mgr-x threads are stopped just fine, gdb obviously can't get any real info out of them. But it works... Regular test cases work fine too, just a sanity check. Didn't expect them not to.
Only thing that I dislike a bit, but I guess that's just a Linuxism, is that if can now kill an io_uring owning task by sending a signal to one of its IO thread workers.
fs/io-wq.c +17 -4
@@ -505,8 +505,14 @@ static int io_wqe_worker(void *data) ret =schedule_timeout (WORKER_IDLE_TIMEOUT );if (try_to_freeze () || ret)continue ; -if (fatal_signal_pending (current)) -break ; +if (signal_pending (current)) { +struct ksignal ksig; + +if (fatal_signal_pending (current)) +break ; +get_signal (&ksig); +continue ; + }/* timed out, exit unless we're the fixed worker */ if (test_bit (IO_WQ_BIT_EXIT , &wq->state) || !(worker->flags &IO_WORKER_F_FIXED ))
@@ -715,8 +721,15 @@ static int io_wq_manager(void *data)io_wq_check_workers (wq);schedule_timeout (HZ );try_to_freeze (); -if (fatal_signal_pending (current)) -set_bit (IO_WQ_BIT_EXIT , &wq->state); +if (signal_pending (current)) { +struct ksignal ksig; + +if (fatal_signal_pending (current)) +set_bit (IO_WQ_BIT_EXIT , &wq->state); +else +get_signal (&ksig); +continue ; + } }while (!test_bit (IO_WQ_BIT_EXIT , &wq->state));io_wq_check_workers (wq);
fs/io_uring.c +8 -2
@@ -6765,8 +6765,14 @@ static int io_sq_thread(void *data) timeout = jiffies + sqd->sq_thread_idle;continue ; } -if (fatal_signal_pending (current)) -break ; +if (signal_pending (current)) { +struct ksignal ksig; + +if (fatal_signal_pending (current)) +break ; +get_signal (&ksig); +continue ; + } sqt_spin = false; cap_entries = !list_is_singular (&sqd->ctx_list);list_for_each_entry (ctx, &sqd->ctx_list, sqd_list) {
kernel/fork.c +1 -0
@@ -2436,6 +2436,7 @@ struct task_struct *create_io_thread(int (*fn)(void *), void *arg, int node)if (!IS_ERR (tsk)) {sigfillset (&tsk->blocked);sigdelsetmask (&tsk->blocked,sigmask (SIGKILL )); +sigdelsetmask (&tsk->blocked,sigmask (SIGSTOP )); }return tsk; }
kernel/ptrace.c +1 -1
@@ -375,7 +375,7 @@ static int ptrace_attach(struct task_struct *task, long request,audit_ptrace (task); retval = -EPERM ; -if (unlikely (task->flags & (PF_KTHREAD |PF_IO_WORKER ))) +if (unlikely (task->flags &PF_KTHREAD )) goto out;if (same_thread_group (task, current)) goto out;
kernel/signal.c +2 -6
@@ -91,7 +91,7 @@ static bool sig_task_ignored(struct task_struct *t, int sig, bool force)return true;/* Only allow kernel generated signals to this kthread */ -if (unlikely ((t->flags & (PF_KTHREAD |PF_IO_WORKER )) && +if (unlikely ((t->flags &PF_KTHREAD ) && (handler ==SIG_KTHREAD_KERNEL ) && !force))return true;
@@ -288,8 +288,7 @@ bool task_set_jobctl_pending(struct task_struct *task, unsigned long mask) JOBCTL_STOP_SIGMASK | JOBCTL_TRAPPING));BUG_ON ((mask &JOBCTL_TRAPPING ) && !(mask &JOBCTL_PENDING_MASK )); -if (unlikely (fatal_signal_pending (task) || - (task->flags & (PF_EXITING |PF_IO_WORKER )))) +if (unlikely (fatal_signal_pending (task) || task->flags &PF_EXITING ))return false;if (mask &JOBCTL_STOP_SIGMASK )
@@ -834,9 +833,6 @@ static int check_kill_permission(int sig, struct kernel_siginfo *info,if (!valid_signal (sig))return -EINVAL ; -/* PF_IO_WORKER threads don't take any signals */ -if (t->flags &PF_IO_WORKER ) -return -ESRCH ;if (!si_fromuser (info))return 0 ;
Signature · 1 line
Jens Axboe
Am 25.03.21 um 22:44 schrieb Jens Axboe:
Quoted 82 lines · Jens Axboe
Quoted 30 lines · Linus Torvalds
Quoted 20 lines
Quoted 10 lines · Linus Torvalds
Quoted 3 lines · Linus Torvalds
I don't know what the gdb logic is, but maybe there's some other option that makes gdb not react to them?
.. maybe we could have a different name for them under the task/ subdirectory, for example (not just the pid)? Although that probably messes up 'ps' too..
Actually, maybe the right model is to simply make all the io threads take signals, and get rid of all the special cases.
Sure, the signals will never be delivered to user space, but if we
- just made the thread loop do "
get_signal()" when there are pending signalsQuoted 8 lines
- allowed
ptrace_attachon themthey'd look pretty much like regular threads that just never do the user-space part of signal handling.
The whole "signals are very special for IO threads" thing has caused so many problems, that maybe the solution is simply to _not_ make them special?
Quoted 8 lines
Just to wrap up the previous one, yes it broke all sorts of things to make the 'tid' directory different. They just end up being hidden anyway through that, for both ps and top.
Yes, I do think that maybe it's better to just embrace maybe just embrace the signals, and have everything just work by default. It's better than continually trying to make the threads special. I'll see if there are some demons lurking down that path.
In the spirit of "let's just try it", I ran with the below patch. With that, I can gdb attach just fine to a test case that creates an
io_uringand a regular thread withpthread_create(). The regular thread uses the ring, so you end up with two iou-mgr threads. Attach:[root@archlinux ~]# gdb -p 360
[snip gdb noise]
Attaching to process 360
[New LWP 361]
[New LWP 362]
[New LWP 363]warning: Selected architecture i386:x86-64 is not compatible with reported target architecture i386
warning: Architecture rejected target-supplied description
Error while reading shared library symbols for/usr/lib/libpthread.so.0:
Cannot find user-level thread for LWP 363: generic error0x00007f7aa526e125in clock_nanosleep@GLIBC_2.2.5 () from/usr/lib/libc.so.6(gdb) info threadsId Target Id Frame * 1 LWP 360 "io_uring" 0x00007f7aa526e125 in clock_nanosleep@GLIBC_2.2.5 () from /usr/lib/libc.so.6 2 LWP 361 "iou-mgr-360" 0x0000000000000000 in ?? () 3 LWP 362 "io_uring" 0x00007f7aa52a0a9d in syscall () from /usr/lib/libc.so.6 4 LWP 363 "iou-mgr-362" 0x0000000000000000 in ?? ()(gdb) thread 2
[Switching to thread 2 (LWP 361)]
#00x0000000000000000in ?? ()
(gdb) bt
#00x0000000000000000in ?? ()
Backtrace stopped: Cannot access memory at address0x0
(gdb) cont
Continuing.
^C
Thread 1 "io_uring" received signal SIGINT, Interrupt. [Switching to LWP 360]0x00007f7aa526e125in clock_nanosleep@GLIBC_2.2.5 () from/usr/lib/libc.so.6(gdb) q
A debugging session is active.Inferior 1 [process 360] will be detached.
Quoted 7 lines
Quit anyway? (y or n) y
Detaching from program:/root/git/fio/t/io_uring, process 360 [Inferior 1 (process 360) detached]The iou-mgr-x threads are stopped just fine, gdb obviously can't get any real info out of them. But it works... Regular test cases work fine too, just a sanity check. Didn't expect them not to.
I guess that's basically what I tried to describe when I said they should look like a userspace process that is blocked in a syscall forever.
Quoted 3 lines
Only thing that I dislike a bit, but I guess that's just a Linuxism, is that if can now kill an
io_uringowning task by sending a signal to one of its IO thread workers.
Can't we just only allow SIGSTOP, which will be only delivered to the iothread itself? And also SIGKILL should not be allowed from userspace.
And /proc/$iothread/ should be read only and owned by root with "cmdline" and "exe" being empty.
Thanks!
metze
Jens Axboe
maintainerQuoted 88 lines · Stefan Metzmacher
Am 25.03.21 um 22:44 schrieb Jens Axboe:
Quoted 86 lines · Jens Axboe
Quoted 30 lines · Linus Torvalds
Quoted 20 lines
Quoted 10 lines · Linus Torvalds
Quoted 3 lines · Linus Torvalds
I don't know what the gdb logic is, but maybe there's some other option that makes gdb not react to them?
.. maybe we could have a different name for them under the task/ subdirectory, for example (not just the pid)? Although that probably messes up 'ps' too..
Actually, maybe the right model is to simply make all the io threads take signals, and get rid of all the special cases.
Sure, the signals will never be delivered to user space, but if we
- just made the thread loop do "
get_signal()" when there are pending signalsQuoted 8 lines
- allowed
ptrace_attachon themthey'd look pretty much like regular threads that just never do the user-space part of signal handling.
The whole "signals are very special for IO threads" thing has caused so many problems, that maybe the solution is simply to _not_ make them special?
Quoted 8 lines
Just to wrap up the previous one, yes it broke all sorts of things to make the 'tid' directory different. They just end up being hidden anyway through that, for both ps and top.
Yes, I do think that maybe it's better to just embrace maybe just embrace the signals, and have everything just work by default. It's better than continually trying to make the threads special. I'll see if there are some demons lurking down that path.
In the spirit of "let's just try it", I ran with the below patch. With that, I can gdb attach just fine to a test case that creates an
io_uringand a regular thread withpthread_create(). The regular thread uses the ring, so you end up with two iou-mgr threads. Attach:[root@archlinux ~]# gdb -p 360
[snip gdb noise]
Attaching to process 360
[New LWP 361]
[New LWP 362]
[New LWP 363]warning: Selected architecture i386:x86-64 is not compatible with reported target architecture i386
warning: Architecture rejected target-supplied description
Error while reading shared library symbols for/usr/lib/libpthread.so.0:
Cannot find user-level thread for LWP 363: generic error0x00007f7aa526e125in clock_nanosleep@GLIBC_2.2.5 () from/usr/lib/libc.so.6(gdb) info threadsId Target Id Frame * 1 LWP 360 "io_uring" 0x00007f7aa526e125 in clock_nanosleep@GLIBC_2.2.5 () from /usr/lib/libc.so.6 2 LWP 361 "iou-mgr-360" 0x0000000000000000 in ?? () 3 LWP 362 "io_uring" 0x00007f7aa52a0a9d in syscall () from /usr/lib/libc.so.6 4 LWP 363 "iou-mgr-362" 0x0000000000000000 in ?? ()(gdb) thread 2
[Switching to thread 2 (LWP 361)]
#00x0000000000000000in ?? ()
(gdb) bt
#00x0000000000000000in ?? ()
Backtrace stopped: Cannot access memory at address0x0
(gdb) cont
Continuing.
^C
Thread 1 "io_uring" received signal SIGINT, Interrupt. [Switching to LWP 360]0x00007f7aa526e125in clock_nanosleep@GLIBC_2.2.5 () from/usr/lib/libc.so.6(gdb) q
A debugging session is active.Inferior 1 [process 360] will be detached.
Quit anyway? (y or n) y
Detaching from program:/root/git/fio/t/io_uring, process 360 [Inferior 1 (process 360) detached]
Quoted 7 lines
Quoted 3 lines
The iou-mgr-x threads are stopped just fine, gdb obviously can't get any real info out of them. But it works... Regular test cases work fine too, just a sanity check. Didn't expect them not to.
I guess that's basically what I tried to describe when I said they should look like a userspace process that is blocked in a syscall forever.
Right, that's almost what they look like, in practice that is what they look like.
Quoted 6 lines
Quoted 3 lines
Only thing that I dislike a bit, but I guess that's just a Linuxism, is that if can now kill an
io_uringowning task by sending a signal to one of its IO thread workers.Can't we just only allow SIGSTOP, which will be only delivered to the iothread itself? And also SIGKILL should not be allowed from userspace.
I don't think we can sanely block them, and we to cleanup and teardown normally regardless of who gets the signal (owner or one of the threads). So I'm not _too_ hung up on the "io thread gets signal goes to owner" as that is what happens with normal threads too, though I would prefer if that wasn't the case. But overall I feel better just embracing the thread model, rather than having something that kinda sorta looks like a thread, but differs in odd ways.
Quoted 2 lines
And
/proc/$iothread/should be read only and owned by root with "cmdline" and "exe" being empty.
I know you brought this one up as part of your series, not sure I get why you want it owned by root and read-only? cmdline and exe, yeah those could be hidden, but is there really any point?
Maybe I'm missing something here, if so, do clue me in!
Signature · 1 line
Jens Axboe
Hi Jens,
Quoted 8 lines
Quoted 2 lines
And
/proc/$iothread/should be read only and owned by root with "cmdline" and "exe" being empty.I know you brought this one up as part of your series, not sure I get why you want it owned by root and read-only? cmdline and exe, yeah those could be hidden, but is there really any point?
Maybe I'm missing something here, if so, do clue me in!
I looked through /proc and I think it's mostly similar to the unshare() case, if userspace wants to do stupid things like changing "comm" of iothreads, it gets what was asked for.
But the "cmdline" hiding would be very useful.
While most tools use "comm", by default.
ps -eLf or 'iotop' use "cmdline".
Some processes use setproctitle to change "cmdline" in order to identify the process better, without the 15 chars comm restriction, that's why I very often press 'c' in 'top' to see the cmdline, in that case it would be very helpful to see '[iou-wrk-1234]' instead of the seeing the cmdline.
So I'd very much prefer if this could be applied:
https://lore.kernel.org/io-uring/d4487f959c778d0b1d4c5738b75bcff17d21df5b.1616197787.git.metze@samba.org/T/#u
If you want I can add a comment and a more verbose commit message...
metze
Hi Jens,
Quoted 22 lines
Quoted 5 lines
I know you brought this one up as part of your series, not sure I get why you want it owned by root and read-only? cmdline and exe, yeah those could be hidden, but is there really any point?
Maybe I'm missing something here, if so, do clue me in!
I looked through /proc and I think it's mostly similar to the
unshare()case, if userspace wants to do stupid things like changing "comm" of iothreads, it gets what was asked for.But the "cmdline" hiding would be very useful.
While most tools use "comm", by default.
ps -eLf or 'iotop' use "cmdline".
Some processes use setproctitle to change "cmdline" in order to identify the process better, without the 15 chars comm restriction, that's why I very often press 'c' in 'top' to see the cmdline, in that case it would be very helpful to see '[iou-wrk-1234]' instead of the seeing the cmdline.
Quoted 4 lines
So I'd very much prefer if this could be applied:
https://lore.kernel.org/io-uring/d4487f959c778d0b1d4c5738b75bcff17d21df5b.1616197787.git.metze@samba.org/T/#uIf you want I can add a comment and a more verbose commit message...
I noticed that 'iotop' actually appends ' [iou-wrk-1234]' to the cmdline value, so that leaves us with 'ps -eLf' and 'top' (with 'c').
pstree -a -t -p is also fine:
│ └─io_uring-cp,1315 /root/kernel/linux-image-5.12.0-rc2+-dbg_5.12.0-rc2+-5_amd64.deb file
│ ├─{iou-mgr-1315},1316
│ ├─{iou-wrk-1315},1317
│ ├─{iou-wrk-1315},1318
│ ├─{iou-wrk-1315},1319
│ ├─{iou-wrk-1315},1320In the spirit of "avoid special PF_IO_WORKER checks" I guess it's ok to leave of as is...
metze
Quoted 6 lines · Jens Axboe
In the spirit of "let's just try it", I ran with the below patch. With that, I can gdb attach just fine to a test case that creates an
io_uringand a regular thread withpthread_create(). The regular thread uses the ring, so you end up with two iou-mgr threads. Attach:
Quoted 6 lines
[root@archlinux ~]# gdb -p 360
[snip gdb noise]
Attaching to process 360
[New LWP 361]
[New LWP 362]
[New LWP 363]
[..]
Looks fairly sane to me.
I think this ends up being the right approach - just the final part (famous last words) of "io_uring threads act like normal threads".
Doing it for VM and FS got rid of all the special cases there, and now doing it for signal handling gets rid of all these ptrace etc issues.
And the fact that a noticeable part of the patch was removing the PF_IO_WORKER tests again looks like a very good sign to me.
In fact, I think you could now remove all the freezer hacks too - because get_signal() will now do the proper try_to_freeze(), so all those freezer things are stale as well.
Yeah, it's still going to be different in that there's no real user space return, and so it will never look _entirely_ like a normal thread, but on the whole I really like how this does seem to get rid of another batch of special cases.
Linus
Jens Axboe
maintainerQuoted 17 lines · Linus Torvalds
Quoted 6 lines · Jens Axboe
In the spirit of "let's just try it", I ran with the below patch. With that, I can gdb attach just fine to a test case that creates an
io_uringand a regular thread withpthread_create(). The regular thread uses the ring, so you end up with two iou-mgr threads. Attach:Quoted 6 lines
[root@archlinux ~]# gdb -p 360
[snip gdb noise]
Attaching to process 360
[New LWP 361]
[New LWP 362]
[New LWP 363][..]
Looks fairly sane to me.
Quoted 8 lines
I think this ends up being the right approach - just the final part (famous last words) of "
io_uringthreads act like normal threads".Doing it for VM and FS got rid of all the special cases there, and now doing it for signal handling gets rid of all these ptrace etc issues.
And the fact that a noticeable part of the patch was removing the
PF_IO_WORKERtests again looks like a very good sign to me.
I agree, and in fact there are more PF_IO_WORKER checks that can go too. The patch is just the bare minimum.
Quoted 3 lines
In fact, I think you could now remove all the freezer hacks too - because
get_signal()will now do the propertry_to_freeze(), so all those freezer things are stale as well.
Yep
Quoted 4 lines
Yeah, it's still going to be different in that there's no real user space return, and so it will never look _entirely_ like a normal thread, but on the whole I really like how this does seem to get rid of another batch of special cases.
That's what makes me feel better too. I think was so hung up on the "never take signals" that it just didn't occur to me to go this route instead.
I'll send out a clean series.
Signature · 1 line
Jens Axboe
Oleg Nesterov
maintainerQuoted 4 lines · Linus Torvalds
The whole "signals are very special for IO threads" thing has caused so many problems, that maybe the solution is simply to _not_ make them special?
Or may be IO threads should not abuse CLONE_THREAD?
Why does create_io_thread() abuse CLONE_THREAD ?
One reason (I think) is that this implies SIGKILL when the process exits/execs, anything else?
Oleg.
Quoted 6 lines · Oleg Nesterov
Quoted 4 lines · Linus Torvalds
The whole "signals are very special for IO threads" thing has caused so many problems, that maybe the solution is simply to _not_ make them special?
Quoted 6 lines
Or may be IO threads should not abuse
CLONE_THREAD?Why does
create_io_thread()abuseCLONE_THREAD?One reason (I think) is that this implies SIGKILL when the process exits/execs, anything else?
A lot.
The io workers perform work on behave of the ordinary userspace threads. Some of that work is opening files. For things like rlimits to work properly you need to share the signal_struct. But odds are if you find anything in signal_struct (not counting signals) there will be an io_uring code path that can exercise it as io_uring can traverse the filesystem, open files and read/write files. So io_uring can exercise all of proc.
Using create_io_thread with CLONE_THREAD is the least problematic way (including all of the signal and ptrace problems we are looking at right now) to implement the io worker threads.
They _really_ are threads of the process that just never execute any code in userspace.
Eric
Am 25.03.21 um 21:55 schrieb Eric W. Biederman:
Quoted 25 lines
Quoted 6 lines · Oleg Nesterov
Quoted 4 lines · Linus Torvalds
The whole "signals are very special for IO threads" thing has caused so many problems, that maybe the solution is simply to _not_ make them special?
Quoted 6 lines
Or may be IO threads should not abuse
CLONE_THREAD?Why does
create_io_thread()abuseCLONE_THREAD?One reason (I think) is that this implies SIGKILL when the process exits/execs, anything else?
A lot.
The io workers perform work on behave of the ordinary userspace threads. Some of that work is opening files. For things like rlimits to work properly you need to share the
signal_struct. But odds are if you find anything insignal_struct(not counting signals) there will be anio_uringcode path that can exercise it asio_uringcan traverse the filesystem, open files and read/write files. Soio_uringcan exercise all of proc.
Quoted 6 lines
Using
create_io_threadwithCLONE_THREADis the least problematic way (including all of the signal and ptrace problems we are looking at right now) to implement the io worker threads.They _really_ are threads of the process that just never execute any code in userspace.
So they should look like a userspace thread sitting in something like epoll_pwait() with all signals blocked, which will never return to userspace again?
I think that would be useful, but I also think that userspace should see:
- /proc/$tidofiothread/cmdline as empty (in order to let ps and top use [iou-wrk-$tidofuserspacethread])
- /proc/$tidofiothread/exe as symlink to that not exists
- all of /proc/$tidofiothread/ shows root.root as owner and group
and things which still allow write access to /proc/$tidofiothread/comm similar things with rw permissions should still disallow modifications:
For the other kernel threads e.g. "[cryptd]" I see the following:
LANG=C ls -l /proc/653 | grep rw
ls: cannot read symbolic link '/proc/653/exe': No such file or directory
-rw-r--r-- 1 root root 0 Mar 25 22:09 autogroup -rw-r--r-- 1 root root 0 Mar 25 22:09 comm -rw-r--r-- 1 root root 0 Mar 25 22:09 coredump_filter lrwxrwxrwx 1 root root 0 Mar 25 22:09 cwd -> / lrwxrwxrwx 1 root root 0 Mar 25 22:09 exe -rw-r--r-- 1 root root 0 Mar 25 22:09 gid_map -rw-r--r-- 1 root root 0 Mar 25 22:09 loginuid -rw------- 1 root root 0 Mar 25 22:09 mem -rw-r--r-- 1 root root 0 Mar 25 22:09 oom_adj -rw-r--r-- 1 root root 0 Mar 25 22:09 oom_score_adj -rw-r--r-- 1 root root 0 Mar 25 22:09 projid_map lrwxrwxrwx 1 root root 0 Mar 25 22:09 root -> / -rw-r--r-- 1 root root 0 Mar 25 22:09 sched -rw-r--r-- 1 root root 0 Mar 25 22:09 setgroups -rw-r--r-- 1 root root 0 Mar 25 22:09 timens_offsets -rw-rw-rw- 1 root root 0 Mar 25 22:09 timerslack_ns -rw-r--r-- 1 root root 0 Mar 25 22:09 uid_map
And this:
LANG=C echo "bla" > /proc/653/comm
-bash: echo: write error: Invalid argument
LANG=C echo "bla" > /proc/653/gid_map
-bash: echo: write error: Operation not permitted
Can't we do the same for iothreads regarding /proc? Just make things read only there and empty "cmdline"/"exe"?
Maybe I'm too naive, but that what I'd assume as a userspace developer/admin.
Does at least parts of it make any sense?
metze
Am 25.03.21 um 22:20 schrieb Stefan Metzmacher:
Quoted 31 lines
Am 25.03.21 um 21:55 schrieb Eric W. Biederman:
Quoted 29 lines
Quoted 6 lines · Oleg Nesterov
Quoted 4 lines · Linus Torvalds
The whole "signals are very special for IO threads" thing has caused so many problems, that maybe the solution is simply to _not_ make them special?
Quoted 6 lines
Or may be IO threads should not abuse
CLONE_THREAD?Why does
create_io_thread()abuseCLONE_THREAD?One reason (I think) is that this implies SIGKILL when the process exits/execs, anything else?
A lot.
The io workers perform work on behave of the ordinary userspace threads. Some of that work is opening files. For things like rlimits to work properly you need to share the
signal_struct. But odds are if you find anything insignal_struct(not counting signals) there will be anio_uringcode path that can exercise it asio_uringcan traverse the filesystem, open files and read/write files. Soio_uringcan exercise all of proc.Using
create_io_threadwithCLONE_THREADis the least problematic way (including all of the signal and ptrace problems we are looking at right now) to implement the io worker threads.
Quoted 5 lines
Quoted 2 lines
They _really_ are threads of the process that just never execute any code in userspace.
So they should look like a userspace thread sitting in something like
epoll_pwait()with all signals blocked, which will never return to userspace again?
Would gdb work with that?
The question is what backtrace gdb would show for that thread.
Is it possible to block SIGSTOP/SIGCONT?
I also think that all signals to an iothread should not be delivered to other threads and it may only react on a direct SIGSTOP/SIGCONT. I guess even SIGKILL should be ignored as the shutdown should happen via the exit path of the iothread parent only.
Quoted 37 lines
I think that would be useful, but I also think that userspace should see:
-/proc/$tidofiothread/cmdlineas empty (in order to let ps and top use [iou-wrk-$tidofuserspacethread])
-/proc/$tidofiothread/exeas symlink to that not exists
- all of/proc/$tidofiothread/shows root.root as owner and group
and things which still allow write access to/proc/$tidofiothread/commsimilar things with rw permissions should still disallow modifications:For the other kernel threads e.g. "[cryptd]" I see the following:
LANG=C ls -l
/proc/653| grep rw
ls: cannot read symbolic link '/proc/653/exe': No such file or directory(quoted diff) +0
-3-rw-r--r-- 1 root root 0 Mar 25 22:09 autogroup -rw-r--r-- 1 root root 0 Mar 25 22:09 comm -rw-r--r-- 1 root root 0 Mar 25 22:09 coredump_filterlrwxrwxrwx 1 root root 0 Mar 25 22:09 cwd -> /
lrwxrwxrwx 1 root root 0 Mar 25 22:09 exe(quoted diff) +0
-6-rw-r--r-- 1 root root 0 Mar 25 22:09 gid_map -rw-r--r-- 1 root root 0 Mar 25 22:09 loginuid -rw------- 1 root root 0 Mar 25 22:09 mem -rw-r--r-- 1 root root 0 Mar 25 22:09 oom_adj -rw-r--r-- 1 root root 0 Mar 25 22:09 oom_score_adj -rw-r--r-- 1 root root 0 Mar 25 22:09 projid_maplrwxrwxrwx 1 root root 0 Mar 25 22:09 root -> /
(quoted diff) +0
-5-rw-r--r-- 1 root root 0 Mar 25 22:09 sched -rw-r--r-- 1 root root 0 Mar 25 22:09 setgroups -rw-r--r-- 1 root root 0 Mar 25 22:09 timens_offsets -rw-rw-rw- 1 root root 0 Mar 25 22:09 timerslack_ns -rw-r--r-- 1 root root 0 Mar 25 22:09 uid_mapAnd this:
LANG=C echo "bla" >
/proc/653/comm
-bash: echo: write error: Invalid argumentLANG=C echo "bla" >
/proc/653/gid_map
-bash: echo: write error: Operation not permitted
Quoted 6 lines
Can't we do the same for iothreads regarding /proc? Just make things read only there and empty "cmdline"/"exe"?
Maybe I'm too naive, but that what I'd assume as a userspace developer/admin.
Does at least parts of it make any sense?
I think the strange glibc setuid() behavior should also be tests here, I guess we don't want that to reset the credentials of an iothread!
Another idea would be to have the iothreads as a child process with it's threads, but again I'm only looking as an admin to what I'd except to see under /proc via ps and top.
metze
Quoted 20 lines · Linus Torvalds
Quoted 10 lines · Linus Torvalds
Quoted 3 lines · Linus Torvalds
I don't know what the gdb logic is, but maybe there's some other option that makes gdb not react to them?
.. maybe we could have a different name for them under the task/ subdirectory, for example (not just the pid)? Although that probably messes up 'ps' too..
Actually, maybe the right model is to simply make all the io threads take signals, and get rid of all the special cases.
Sure, the signals will never be delivered to user space, but if we
- just made the thread loop do "
get_signal()" when there are pending signals
Quoted 8 lines
- allowed
ptrace_attachon themthey'd look pretty much like regular threads that just never do the user-space part of signal handling.
The whole "signals are very special for IO threads" thing has caused so many problems, that maybe the solution is simply to _not_ make them special?
The special case in check_kill_permission is certainly unnecessary. Having the signal blocked is enough to prevent signal_pending() from being true.
The most straight forward thing I can see is to allow ptrace_attach and to modify ptrace_check_attach to always return -ESRCH for io workers unless ignore_state is set causing none of the other ptrace operations to work.
That is what a long running in-kernel thread would do today so user-space aka gdb may actually cope with it.
We might be able to support if io workers start supporting SIGSTOP but I am not at all certain.
Eric
Jens Axboe
maintainerQuoted 41 lines · Eric W. Biederman
Quoted 20 lines · Linus Torvalds
Quoted 10 lines · Linus Torvalds
Quoted 3 lines · Linus Torvalds
I don't know what the gdb logic is, but maybe there's some other option that makes gdb not react to them?
.. maybe we could have a different name for them under the task/ subdirectory, for example (not just the pid)? Although that probably messes up 'ps' too..
Actually, maybe the right model is to simply make all the io threads take signals, and get rid of all the special cases.
Sure, the signals will never be delivered to user space, but if we
- just made the thread loop do "
get_signal()" when there are pending signalsQuoted 8 lines
- allowed
ptrace_attachon themthey'd look pretty much like regular threads that just never do the user-space part of signal handling.
The whole "signals are very special for IO threads" thing has caused so many problems, that maybe the solution is simply to _not_ make them special?
The special case in
check_kill_permissionis certainly unnecessary. Having the signal blocked is enough to preventsignal_pending()from being true.The most straight forward thing I can see is to allow
ptrace_attachand to modifyptrace_check_attachto always return-ESRCHfor io workers unlessignore_stateis set causing none of the other ptrace operations to work.
Quoted 6 lines
That is what a long running in-kernel thread would do today so user-space aka gdb may actually cope with it.
We might be able to support if io workers start supporting SIGSTOP but I am not at all certain.
See patch just send out as a POC, mostly, not fully sanitized yet. But I did try to return -ESRCH from ptrace_check_attach() if it's an IO thread and ignore_state isn't set:
if (!ignore_state && child->flags & PF_IO_WORKER)
return -ESRCH;
and that causes gdb to abort at that thread. For the same test case as in the previous email, you get:
Attaching to process 358
[New LWP 359]
[New LWP 360]
[New LWP 361]
Couldn't get CS register: No such process.
(gdb) 0x00007ffa58537125 in ?? ()
(gdb) bt
#0 0x00007ffa58537125 in ?? ()
#1 0x0000000000000000 in ?? ()
(gdb) info threads
Id Target Id Frame * 1 LWP 358 "io_uring" 0x00007ffa58537125 in ?? () 2 LWP 359 "iou-mgr-358" Couldn't get registers: No such process.
(gdb) q
A debugging session is active.
Inferior 1 [process 358] will be detached.
Quit anyway? (y or n) y
Couldn't write debug register: No such process.
where 360 here is a regular pthread created thread, and 361 is another iou-mgr-x task. While gdb behaves better in this case, it does still prevent you from inspecting thread 3 which would be totally valid.
Signature · 1 line
Jens Axboe
Jens Axboe
maintainerQuoted 23 lines · Eric W. Biederman
Quoted 13 lines · Jens Axboe
Hi,
Stefan reports that attaching to a task with
io_uringwill leave gdb very confused and just repeatedly attempting to attach to the IO threads, even though it receives an-EPERMevery time. This patchset proposes to skipPF_IO_WORKERthreads assame_thread_group(), except for accounting purposes which we still desire.We also skip listing the IO threads in /proc/<pid>/task/ so that gdb doesn't think it should stop and attach to them. This makes us consistent with earlier kernels, where these async threads were not related to the ring owning task, and hence gdb (and others) ignored them anyway.
Quoted 7 lines
Seems to me that this is the right approach, but open to comments on if others agree with this. Oleg, I did see your messages as well on SIGSTOP, and as was discussed with Eric as well, this is something we most certainly can revisit. I do think that the visibility of these threads is a separate issue. Even with SIGSTOP implemented (which I did try as well), we're never going to allow ptrace attach and hence gdb would still be broken. Hence I'd rather treat them as separate issues to attack.
Quoted 7 lines
A quick skim shows that these threads are not showing up anywhere in proc which appears to be a problem, as it hides them from top.
Sysadmins need the ability to dig into a system and find out where all their cpu usage or io's have gone when there is a problem. I general I think this argues that these threads should show up as threads of the process so I am not even certain this is the right fix to deal with gdb.
That's a good point, overall hiding was not really what I desired, just getting them out of gdb's hands. And arguably it _is_ a gdb bug, but...
Signature · 1 line
Jens Axboe
Oleg Nesterov
maintainerI didn't even try to read this series yet, will try tomorrow.
But sorry, I can't resist...
Quoted 9 lines · Jens Axboe
Quoted 7 lines · Eric W. Biederman
Quoted 5 lines · Jens Axboe
Hi,
Stefan reports that attaching to a task with
io_uringwill leave gdb very confused and just repeatedly attempting to attach to the IO threads, even though it receives an-EPERMevery time.
Heh. As expected :/
Quoted 1 line
And arguably it _is_ a gdb bug, but...
Why do you think so?
Oleg.