Skip to contentSkip to conversation

8cf0c45999: fio.read_bw_MBps 51.1% regression

linux-next:master kernel test robot 2 participants 3 messages Dec 27, 2024 – Dec 31, 2024 io_uring

lore
Share & tools
lore

kernel test robot

author build bot
#0 lore raw

Hello,

kernel test robot noticed a 51.1% regression of fio.read_bw_MBps on:

commit: 8cf0c459993ee2911f4f01fba21b1987b102c887 ("io_uring/rw: Allocate async data through helper")
https://git.kernel.org/cgit/linux/kernel/git/next/linux-next.git master

[test failed on linux-next/master 8155b4ef3466f0e289e8fcc9e6e62f3f4dceeac2]

testcase: fio-basic
config: x86_64-rhel-9.4
compiler: gcc-12
test machine: 96 threads 2 sockets Intel(R) Xeon(R) Platinum 8260L CPU @ 2.40GHz (Cascade Lake) with 128G memory parameters:

	runtime: 300s
	nr_task: 8t
	disk: 1SSD
	fs: xfs
	rw: read
	bs: 2M
	ioengine: io_uring
	test_size: 256g
	cpufreq_governor: performance

If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-lkp/[email protected]

Details are as below:
-------------------------------------------------------------------------------------------------->

The kernel config and materials to reproduce are available at:
https://download.01.org/0day-ci/archive/20241227/[email protected]

=========================================================================================
bs/compiler/cpufreq_governor/disk/fs/ioengine/kconfig/nr_task/rootfs/runtime/rw/tbox_group/test_size/testcase:
2M/gcc-12/performance/1SSD/xfs/io_uring/x86_64-rhel-9.4/8t/debian-12-x86_64-20240206.cgz/300s/read/lkp-csl-2sp3/256g/fio-basic

commit:
23d91035ca ("io_uring/net: Allocate msghdr async data through helper") 8cf0c45999 ("io_uring/rw: Allocate async data through helper")

23d91035cafa30d1 8cf0c459993ee2911f4f01fba21
---------------- ---------------------------
%stddev %change %stddev
\ | \
1.169e+10 -97.9% 2.464e+08 ± 4% cpuidle..time
4187672 ± 5% -98.3% 69610 cpuidle..usage

    490.00           -24.5%     370.16        uptime.boot
     45002           -23.4%      34483        uptime.idle
     91.67            +3.9%      95.24        iostat.cpu.idle
      6.67           -83.6%       1.09 ±  2%  iostat.cpu.iowait
      0.15 ±  3%   +1569.3%       2.51 ±  5%  iostat.cpu.user
      6.77 ±  2%      -5.2        1.52 ±  4%  mpstat.cpu.all.iowait%
      0.03 ± 22%      +0.0        0.05 ± 18%  mpstat.cpu.all.soft%
      0.15 ±  4%      +4.1        4.23 ±  6%  mpstat.cpu.all.usr%
   5982113 ± 42%     -97.3%     160513 ± 30%  numa-numastat.node0.local_node
   6032931 ± 41%     -96.5%     209090 ± 13%  numa-numastat.node0.numa_hit
   2662250 ±124%     -94.1%     155780 ± 31%  numa-numastat.node1.local_node
   2710984 ±121%     -92.4%     206648 ± 13%  numa-numastat.node1.numa_hit
   2142706           -87.8%     262066        vmstat.io.bi
     62.03         +2998.4%       1922        vmstat.io.bo
      6.47 ±  3%    -100.0%       0.00        vmstat.procs.b
      2.69 ± 12%    +173.1%       7.33 ±  8%  vmstat.procs.r
     19637           -46.7%      10473 ±  4%  vmstat.system.cs
     34417 ±  3%     -10.8%      30715 ±  2%  vmstat.system.in
      2.37 ± 49%      -2.1        0.26 ±103%  fio.latency_100ms%
     81.78 ±  3%     -56.2       25.56 ± 29%  fio.latency_250ms%
      0.34 ± 66%      -0.3        0.00        fio.latency_50ms%
      2171           -51.1%       1060 ±  3%  fio.read_bw_MBps
 4.194e+08           -14.2%    3.6e+08 ±  5%  fio.read_clat_99%_ns
    121.24           -99.2%       0.96 ±  6%  fio.time.elapsed_time
    121.24           -99.2%       0.96 ±  6%  fio.time.elapsed_time.max
 5.369e+08           -99.6%    2096533        fio.time.file_system_inputs
      8592 ±  3%     -10.2%       7716        fio.time.minor_page_faults
     89.00 ±  4%    +587.6%     612.00 ±  6%  fio.time.percent_of_cpu_this_job_got
    102.38 ±  4%     -99.6%       0.44 ±  2%  fio.time.system_time
   1101885           -99.6%       4454        fio.time.voluntary_context_switches
    131072           -99.6%     510.83        fio.workload
   6857403 ± 71%    -100.0%       0.00        numa-meminfo.node0.Active(file)
  49781569 ± 10%     -98.0%     971701 ±119%  numa-meminfo.node0.FilePages
  42007683 ±  4%     -99.4%     232792 ± 70%  numa-meminfo.node0.Inactive
  41773509 ±  4%     -99.4%     232792 ± 70%  numa-meminfo.node0.Inactive(file)
    181744 ± 10%     -80.2%      36054 ± 50%  numa-meminfo.node0.KReclaimable
  11216992 ± 48%    +440.7%   60655275        numa-meminfo.node0.MemFree
  54456267 ±  9%     -90.8%    5017984 ± 22%  numa-meminfo.node0.MemUsed
    181744 ± 10%     -80.2%      36054 ± 50%  numa-meminfo.node0.SReclaimable
     11353 ± 48%     -50.3%       5640 ±  9%  numa-meminfo.node0.Shmem
  46742560 ± 10%     -93.7%    2968006 ± 39%  numa-meminfo.node1.FilePages
  44279309 ± 12%     -99.7%     116761 ±141%  numa-meminfo.node1.Inactive
  44279309 ± 12%     -99.7%     116761 ±141%  numa-meminfo.node1.Inactive(file)
    222588 ±  9%     -71.5%      63492 ± 29%  numa-meminfo.node1.KReclaimable
  17706281 ± 29%    +248.8%   61765996        numa-meminfo.node1.MemFree
  48294686 ± 10%     -91.2%    4234972 ± 26%  numa-meminfo.node1.MemUsed
    222588 ±  9%     -71.5%      63492 ± 29%  numa-meminfo.node1.SReclaimable
     19553 ± 30%     -91.4%       1688 ± 29%  numa-meminfo.node1.Shmem
    484021 ± 22%     -54.2%     221920 ± 54%  numa-meminfo.node1.Slab
   8292851 ± 60%     -91.7%     685299        meminfo.Active
    993491 ± 17%     -31.0%     685299        meminfo.Active(anon)
   7299359 ± 70%    -100.0%       0.00        meminfo.Active(file)
   1059116 ±  3%     -57.9%     446122 ± 20%  meminfo.AnonHugePages
   1199023           -43.1%     682106        meminfo.AnonPages
  96504886           -95.9%    3939272        meminfo.Cached
   1771531           -52.2%     846516 ±  3%  meminfo.Committed_AS
  85824229 ±  5%     -99.6%     349553        meminfo.Inactive
  85591704 ±  5%     -99.6%     349553        meminfo.Inactive(file)
    404111           -75.5%      99205        meminfo.KReclaimable
     82199           -26.3%      60586        meminfo.Mapped
  28942581 ±  2%    +323.0%  1.224e+08        meminfo.MemFree
 1.027e+08           -91.0%    9249647        meminfo.Memused
      2432           -62.3%     917.33 ± 79%  meminfo.Mlocked
    404111           -75.5%      99205        meminfo.SReclaimable
    600546           -30.5%     417425        meminfo.SUnreclaim
     30852 ±  2%     -77.6%       6905 ±  5%  meminfo.Shmem
   1004658           -48.6%     516630        meminfo.Slab
 1.028e+08           -91.0%    9249647        meminfo.max_used_kB
   1780344 ± 71%    -100.0%       0.00        numa-vmstat.node0.nr_active_file
  12441416 ± 10%     -98.0%     242939 ±119%  numa-vmstat.node0.nr_file_pages
   2808184 ± 48%    +440.0%   15163781        numa-vmstat.node0.nr_free_pages
  10373418 ±  5%     -99.4%      58198 ± 70%  numa-vmstat.node0.nr_inactive_file
      2830 ± 48%     -49.7%       1424 ±  8%  numa-vmstat.node0.nr_shmem
     45390 ± 10%     -80.1%       9015 ± 50%  numa-vmstat.node0.nr_slab_reclaimable
   1779467 ± 71%    -100.0%       0.00        numa-vmstat.node0.nr_zone_active_file
  10374301 ±  5%     -99.4%      58198 ± 70%  numa-vmstat.node0.nr_zone_inactive_file
   6032577 ± 41%     -96.5%     209022 ± 13%  numa-vmstat.node0.numa_hit
   5981759 ± 42%     -97.3%     160445 ± 30%  numa-vmstat.node0.numa_local
     60865 ± 46%    -100.0%       0.00        numa-vmstat.node0.workingset_nodes
  11683973 ± 10%     -93.6%     742001 ± 39%  numa-vmstat.node1.nr_file_pages
   4428033 ± 29%    +248.7%   15441242        numa-vmstat.node1.nr_free_pages
  11068153 ± 12%     -99.7%      29190 ±141%  numa-vmstat.node1.nr_inactive_file
      4895 ± 30%     -91.4%     422.28 ± 29%  numa-vmstat.node1.nr_shmem
     55624 ±  9%     -71.5%      15873 ± 29%  numa-vmstat.node1.nr_slab_reclaimable
  11068150 ± 12%     -99.7%      29190 ±141%  numa-vmstat.node1.nr_zone_inactive_file
   2710384 ±121%     -92.4%     205957 ± 13%  numa-vmstat.node1.numa_hit
   2661650 ±124%     -94.2%     155089 ± 31%  numa-vmstat.node1.numa_local
     96303 ± 31%    -100.0%       0.00        numa-vmstat.node1.workingset_nodes
      9.70 ± 66%      -9.4        0.32 ±223%  perf-profile.calltrace.cycles-pp.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.write
      9.09 ± 54%      -9.1        0.00        perf-profile.calltrace.cycles-pp.record__pushfn.perf_mmap__push.record__mmap_read_evlist.__cmd_record.cmd_record
      9.09 ± 54%      -9.1        0.00        perf-profile.calltrace.cycles-pp.write.writen.record__pushfn.perf_mmap__push.record__mmap_read_evlist
      9.09 ± 54%      -9.1        0.00        perf-profile.calltrace.cycles-pp.writen.record__pushfn.perf_mmap__push.record__mmap_read_evlist.__cmd_record
     13.49 ± 32%      -8.7        4.75 ± 38%  perf-profile.calltrace.cycles-pp.handle_internal_command.main
     13.49 ± 32%      -8.7        4.75 ± 38%  perf-profile.calltrace.cycles-pp.main
     13.49 ± 32%      -8.7        4.75 ± 38%  perf-profile.calltrace.cycles-pp.run_builtin.handle_internal_command.main
      8.57 ± 57%      -8.6        0.00        perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.write.writen.record__pushfn
      8.57 ± 57%      -8.6        0.00        perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.write.writen.record__pushfn.perf_mmap__push
      8.57 ± 57%      -8.6        0.00        perf-profile.calltrace.cycles-pp.generic_perform_write.shmem_file_write_iter.vfs_write.ksys_write.do_syscall_64
      8.57 ± 57%      -8.6        0.00        perf-profile.calltrace.cycles-pp.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.write.writen
      8.57 ± 57%      -8.6        0.00        perf-profile.calltrace.cycles-pp.shmem_file_write_iter.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
      9.34 ± 22%      -6.9        2.45 ± 67%  perf-profile.calltrace.cycles-pp.__cmd_record.cmd_record.run_builtin.handle_internal_command.main
      9.34 ± 22%      -6.9        2.45 ± 67%  perf-profile.calltrace.cycles-pp.cmd_record.run_builtin.handle_internal_command.main
      7.20 ± 25%      -5.5        1.70 ± 79%  perf-profile.calltrace.cycles-pp.perf_mmap__push.record__mmap_read_evlist.__cmd_record.cmd_record.run_builtin
      7.20 ± 25%      -5.5        1.70 ± 79%  perf-profile.calltrace.cycles-pp.record__mmap_read_evlist.__cmd_record.cmd_record.run_builtin.handle_internal_command
     10.62 ± 57%      -9.4        1.17 ±142%  perf-profile.children.cycles-pp.write
      9.69 ± 66%      -9.4        0.32 ±223%  perf-profile.children.cycles-pp.ksys_write
      9.69 ± 66%      -9.4        0.32 ±223%  perf-profile.children.cycles-pp.vfs_write
     13.49 ± 32%      -9.1        4.41 ± 47%  perf-profile.children.cycles-pp.__cmd_record
     13.49 ± 32%      -9.1        4.41 ± 47%  perf-profile.children.cycles-pp.cmd_record
      9.09 ± 54%      -8.8        0.32 ±223%  perf-profile.children.cycles-pp.writen
      9.09 ± 54%      -8.7        0.34 ±223%  perf-profile.children.cycles-pp.record__pushfn
     13.49 ± 32%      -8.7        4.75 ± 38%  perf-profile.children.cycles-pp.handle_internal_command
     13.49 ± 32%      -8.7        4.75 ± 38%  perf-profile.children.cycles-pp.main
     13.49 ± 32%      -8.7        4.75 ± 38%  perf-profile.children.cycles-pp.run_builtin
      8.56 ± 57%      -8.6        0.00        perf-profile.children.cycles-pp.generic_perform_write
      8.56 ± 57%      -8.6        0.00        perf-profile.children.cycles-pp.shmem_file_write_iter
      9.94 ± 51%      -8.2        1.70 ± 79%  perf-profile.children.cycles-pp.perf_mmap__push
      9.94 ± 51%      -8.2        1.70 ± 79%  perf-profile.children.cycles-pp.record__mmap_read_evlist
     21.85 ± 22%     -95.7%       0.93 ±  9%  perf-stat.i.MPKI
 7.833e+08 ± 36%    +248.8%  2.733e+09        perf-stat.i.branch-instructions
      1.20 ± 21%      +1.4        2.60        perf-stat.i.branch-miss-rate%
  10294722 ±  8%    +588.8%   70910805        perf-stat.i.branch-misses
     71.04 ±  7%     -45.3       25.71 ±  6%  perf-stat.i.cache-miss-rate%
  67179322 ± 23%     -82.0%   12088775 ±  8%  perf-stat.i.cache-misses
  92053950 ± 16%     -49.0%   46958614 ±  2%  perf-stat.i.cache-references
     19893           -57.6%       8444 ±  6%  perf-stat.i.context-switches
      1.42 ± 18%     +47.0%       2.09 ±  6%  perf-stat.i.cpi
     96008            +3.1%      99012        perf-stat.i.cpu-clock
 5.166e+09 ± 27%    +426.7%  2.721e+10 ±  5%  perf-stat.i.cpu-cycles
    109.64          +297.0%     435.28 ± 26%  perf-stat.i.cpu-migrations
     95.02 ± 12%   +2287.2%       2268 ± 10%  perf-stat.i.cycles-between-cache-misses
 4.488e+09 ± 40%    +190.0%  1.301e+10        perf-stat.i.instructions
      0.78 ± 18%     -38.5%       0.48 ±  6%  perf-stat.i.ipc
      3257         +2386.4%      81003 ±  2%  perf-stat.i.minor-faults
      3257         +2386.3%      81001 ±  2%  perf-stat.i.page-faults
     96008            +3.1%      99013        perf-stat.i.task-clock
     16.77 ± 26%     -94.5%       0.93 ±  9%  perf-stat.overall.MPKI
      1.53 ± 39%      +1.1        2.60        perf-stat.overall.branch-miss-rate%
     72.01 ±  8%     -46.3       25.71 ±  6%  perf-stat.overall.cache-miss-rate%
      1.26 ± 22%     +66.0%       2.09 ±  6%  perf-stat.overall.cpi
     76.07 ±  5%   +2881.9%       2268 ± 10%  perf-stat.overall.cycles-between-cache-misses
      0.83 ± 18%     -42.0%       0.48 ±  6%  perf-stat.overall.ipc
   4201457 ± 40%    +507.2%   25510793        perf-stat.overall.path-length
 7.771e+08 ± 36%     +72.4%   1.34e+09        perf-stat.ps.branch-instructions
  10227839 ±  8%    +239.9%   34759364        perf-stat.ps.branch-misses
  66623351 ± 23%     -91.1%    5924306 ±  8%  perf-stat.ps.cache-misses
  91293692 ± 16%     -74.8%   23018351 ±  2%  perf-stat.ps.cache-references
     19729           -79.0%       4140 ±  6%  perf-stat.ps.context-switches
     95218           -49.0%      48533        perf-stat.ps.cpu-clock
 5.123e+09 ± 27%    +160.3%  1.334e+10 ±  5%  perf-stat.ps.cpu-cycles
    108.79           +95.9%     213.12 ± 25%  perf-stat.ps.cpu-migrations
 4.452e+09 ± 40%     +43.3%   6.38e+09        perf-stat.ps.instructions
      3230         +1129.5%      39716 ±  3%  perf-stat.ps.minor-faults
      3230         +1129.5%      39715 ±  3%  perf-stat.ps.page-faults
     95218           -49.0%      48533        perf-stat.ps.task-clock
 5.507e+11 ± 40%     -97.6%  1.303e+10        perf-stat.total.instructions
    428.50 ±113%    -100.0%       0.00        proc-vmstat.kswapd_high_wmark_hit_quickly
      1663 ± 68%    -100.0%       0.00        proc-vmstat.kswapd_low_wmark_hit_quickly
    248125 ± 17%     -30.9%     171528        proc-vmstat.nr_active_anon
   1740791 ± 71%    -100.0%       0.00        proc-vmstat.nr_active_file
    299833           -43.1%     170711        proc-vmstat.nr_anon_pages
    517.25 ±  3%     -57.9%     217.83 ± 20%  proc-vmstat.nr_anon_transparent_hugepages
  24125371           -95.9%     984856        proc-vmstat.nr_file_pages
   7236331          +322.9%   30605541        proc-vmstat.nr_free_pages
  21481117 ±  5%     -99.6%      87388        proc-vmstat.nr_inactive_file
     20818           -25.3%      15557        proc-vmstat.nr_mapped
    608.02           -62.3%     229.33 ± 79%  proc-vmstat.nr_mlock
      7717           -77.2%       1762 ±  4%  proc-vmstat.nr_shmem
    101012           -75.4%      24886        proc-vmstat.nr_slab_reclaimable
    150134           -30.4%     104433        proc-vmstat.nr_slab_unreclaimable
    248125 ± 17%     -30.9%     171528        proc-vmstat.nr_zone_active_anon
   1740790 ± 71%    -100.0%       0.00        proc-vmstat.nr_zone_active_file
  21481124 ±  5%     -99.6%      87388        proc-vmstat.nr_zone_inactive_file
   9755834 ± 25%    -100.0%       0.00        proc-vmstat.numa_foreign
    995.00 ± 26%     -95.4%      45.33 ±164%  proc-vmstat.numa_hint_faults
    939.00 ± 28%     -96.3%      34.67 ±204%  proc-vmstat.numa_hint_faults_local
   8745252 ± 28%     -95.2%     416538        proc-vmstat.numa_hit
   8645700 ± 28%     -96.3%     317091        proc-vmstat.numa_local
   9755622 ± 25%    -100.0%       0.00        proc-vmstat.numa_miss
   9856110 ± 25%     -99.0%      99445        proc-vmstat.numa_other
    386923           -97.4%       9959 ±222%  proc-vmstat.numa_pte_updates
      2224 ± 32%    -100.0%       0.00        proc-vmstat.pageoutrun
    423905 ±  3%    -100.0%       0.00        proc-vmstat.pgalloc_dma32
  68828649           -98.9%     752946        proc-vmstat.pgalloc_normal
    498127           -61.7%     190797        proc-vmstat.pgfault
  38943033           -99.2%     297228        proc-vmstat.pgfree
 2.685e+08           -99.6%    1048266        proc-vmstat.pgpgin
     22286 ±  3%     -58.9%       9168 ±  6%  proc-vmstat.pgreuse
  36664920          -100.0%       0.00        proc-vmstat.pgscan_file
  36664920          -100.0%       0.00        proc-vmstat.pgscan_kswapd
  36664893          -100.0%       0.00        proc-vmstat.pgsteal_file
  36664893          -100.0%       0.00        proc-vmstat.pgsteal_kswapd
     30510 ±  9%    -100.0%       0.00        proc-vmstat.slabs_scanned
    157121 ±  3%    -100.0%       0.00        proc-vmstat.workingset_nodes
      8736 ±  9%     -37.6%       5455 ±  7%  sched_debug.cfs_rq:/.avg_vruntime.min
      0.00 ± 56%    +407.7%       0.02 ± 58%  sched_debug.cfs_rq:/.h_nr_delayed.avg
      0.31 ± 48%    +227.3%       1.00        sched_debug.cfs_rq:/.h_nr_delayed.max
      0.03 ± 49%    +295.2%       0.13 ± 27%  sched_debug.cfs_rq:/.h_nr_delayed.stddev
      0.09 ± 11%     +84.1%       0.17 ±  9%  sched_debug.cfs_rq:/.h_nr_running.avg
      0.28 ±  6%     +43.8%       0.40 ± 11%  sched_debug.cfs_rq:/.h_nr_running.stddev
     48.57 ± 18%     +84.9%      89.82 ± 18%  sched_debug.cfs_rq:/.load_avg.avg
    947.69 ± 14%     +46.5%       1388 ± 33%  sched_debug.cfs_rq:/.load_avg.max
    166.24 ± 14%     +62.3%     269.79 ± 16%  sched_debug.cfs_rq:/.load_avg.stddev
      8736 ±  9%     -37.6%       5452 ±  7%  sched_debug.cfs_rq:/.min_vruntime.min
      0.09 ± 11%     +88.9%       0.18 ±  9%  sched_debug.cfs_rq:/.nr_running.avg
      1.36 ±  4%     +46.9%       2.00        sched_debug.cfs_rq:/.nr_running.max
      0.29 ±  6%     +50.1%       0.43 ±  9%  sched_debug.cfs_rq:/.nr_running.stddev
     17.33 ± 38%    +147.9%      42.98 ± 29%  sched_debug.cfs_rq:/.removed.load_avg.avg
    483.56 ± 31%    +111.8%       1024        sched_debug.cfs_rq:/.removed.load_avg.max
     85.81 ± 31%    +132.6%     199.60 ± 14%  sched_debug.cfs_rq:/.removed.load_avg.stddev
      6.41 ± 42%    +176.3%      17.71 ± 38%  sched_debug.cfs_rq:/.removed.runnable_avg.avg
    239.17 ± 37%    +119.8%     525.67 ±  2%  sched_debug.cfs_rq:/.removed.runnable_avg.max
     34.33 ± 35%    +152.5%      86.68 ± 22%  sched_debug.cfs_rq:/.removed.runnable_avg.stddev
      6.41 ± 42%    +176.4%      17.71 ± 38%  sched_debug.cfs_rq:/.removed.util_avg.avg
    238.94 ± 37%    +120.0%     525.67 ±  2%  sched_debug.cfs_rq:/.removed.util_avg.max
     34.31 ± 35%    +152.7%      86.68 ± 22%  sched_debug.cfs_rq:/.removed.util_avg.stddev
    126.39 ± 12%    +132.4%     293.67 ±  3%  sched_debug.cfs_rq:/.util_avg.avg
    992.22 ±  8%     +30.8%       1298 ± 16%  sched_debug.cfs_rq:/.util_avg.max
    188.05 ±  8%     +70.1%     319.91 ±  7%  sched_debug.cfs_rq:/.util_avg.stddev
     18.55 ± 20%     +78.8%      33.17 ± 14%  sched_debug.cfs_rq:/.util_est.avg
    614.94 ± 29%     +62.9%       1002 ±  3%  sched_debug.cfs_rq:/.util_est.max
     83.77 ± 24%     +64.2%     137.53 ±  4%  sched_debug.cfs_rq:/.util_est.stddev
    196044 ± 15%     -98.6%       2808 ± 15%  sched_debug.cpu.avg_idle.min
    163689 ±  3%     +38.6%     226871 ±  4%  sched_debug.cpu.avg_idle.stddev
    421612 ±  2%     -12.7%     367991        sched_debug.cpu.clock.avg
    421617 ±  2%     -12.7%     367996        sched_debug.cpu.clock.max
    421588 ±  2%     -12.7%     367986        sched_debug.cpu.clock.min
      4.22 ± 27%     -38.7%       2.58 ±  4%  sched_debug.cpu.clock.stddev
    421249 ±  2%     -12.7%     367704        sched_debug.cpu.clock_task.avg
    421583 ±  2%     -12.7%     367963        sched_debug.cpu.clock_task.max
    411856 ±  2%     -12.8%     359277        sched_debug.cpu.clock_task.min
      1029 ±  6%     -11.9%     907.72 ±  3%  sched_debug.cpu.clock_task.stddev
    252.14 ± 13%     +61.7%     407.60 ±  6%  sched_debug.cpu.curr->pid.avg
      5177 ±  6%     -42.7%       2967        sched_debug.cpu.curr->pid.max
      0.00 ±  9%     +38.2%       0.00 ± 17%  sched_debug.cpu.next_balance.stddev
      0.09 ± 12%     +96.6%       0.17 ±  9%  sched_debug.cpu.nr_running.avg
      0.26 ±  6%     +50.6%       0.40 ± 11%  sched_debug.cpu.nr_running.stddev
     35646 ±  7%     -28.2%      25593 ±  2%  sched_debug.cpu.nr_switches.avg
    711.47 ± 10%     -44.3%     396.17 ± 11%  sched_debug.cpu.nr_switches.min
     52034 ±  7%     -16.0%      43686 ±  6%  sched_debug.cpu.nr_switches.stddev
      0.01 ± 21%    +100.0%       0.01 ± 31%  sched_debug.cpu.nr_uninterruptible.avg
    421610 ±  2%     -12.7%     367987        sched_debug.cpu_clk
    421050 ±  2%     -12.7%     367426        sched_debug.ktime
    422192 ±  2%     -12.7%     368564        sched_debug.sched_clk
      0.01 ± 71%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.__cond_resched.filemap_read.xfs_file_buffered_read.xfs_file_read_iter.__io_read
      0.06 ± 10%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.__cond_resched.shrink_folio_list.evict_folios.try_to_shrink_lruvec.shrink_one
      0.04 ± 49%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.__cond_resched.try_to_shrink_lruvec.shrink_one.shrink_many.shrink_node
      0.03 ± 23%     -93.6%       0.00 ±223%  perf-sched.sch_delay.avg.ms.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown]
      0.06 ± 33%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.devkmsg_read.vfs_read.ksys_read.do_syscall_64
      0.01 ± 20%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.io_cqring_wait.__do_sys_io_uring_enter.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.02 ± 70%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.irq_thread.kthread.ret_from_fork.ret_from_fork_asm
      0.04 ± 31%     -91.0%       0.00 ±170%  perf-sched.sch_delay.avg.ms.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt.[unknown].[unknown]
      0.03 ± 52%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.rcu_gp_kthread.kthread.ret_from_fork.ret_from_fork_asm
      0.02 ± 20%     -74.2%       0.01 ± 42%  perf-sched.sch_delay.avg.ms.schedule_hrtimeout_range.do_poll.constprop.0.do_sys_poll
      0.02 ± 33%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.schedule_hrtimeout_range.do_select.core_sys_select.do_pselect.constprop
      0.04 ± 51%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.schedule_timeout.__wait_for_common.__flush_work.__lru_add_drain_all
      0.01 ± 44%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.schedule_timeout.__wait_for_common.wait_for_completion_state.kernel_clone
      0.01 ± 43%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.schedule_timeout.io_schedule_timeout.__wait_for_common.blk_execute_rq
      0.05 ± 27%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.schedule_timeout.kcompactd.kthread.ret_from_fork
      0.01 ± 35%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.schedule_timeout.khugepaged_wait_work.khugepaged.kthread
      0.04 ± 25%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.schedule_timeout.kswapd_try_to_sleep.kswapd.kthread
      0.03 ± 18%     -69.6%       0.01 ± 92%  perf-sched.sch_delay.avg.ms.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown]
      0.02 ± 20%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.wait_for_partner.fifo_open.do_dentry_open.vfs_open
      0.03 ± 75%    -100.0%       0.00        perf-sched.sch_delay.max.ms.__cond_resched.filemap_read.xfs_file_buffered_read.xfs_file_read_iter.__io_read
      0.08 ±  4%    -100.0%       0.00        perf-sched.sch_delay.max.ms.__cond_resched.shrink_folio_list.evict_folios.try_to_shrink_lruvec.shrink_one
      0.07 ± 21%    -100.0%       0.00        perf-sched.sch_delay.max.ms.__cond_resched.try_to_shrink_lruvec.shrink_one.shrink_many.shrink_node
      0.04 ±  6%     -95.2%       0.00 ±223%  perf-sched.sch_delay.max.ms.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown]
      0.06 ± 34%    -100.0%       0.00        perf-sched.sch_delay.max.ms.devkmsg_read.vfs_read.ksys_read.do_syscall_64
      3.27 ±147%    -100.0%       0.00        perf-sched.sch_delay.max.ms.io_cqring_wait.__do_sys_io_uring_enter.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.02 ± 89%    -100.0%       0.00        perf-sched.sch_delay.max.ms.irq_thread.kthread.ret_from_fork.ret_from_fork_asm
      0.07 ± 18%     -89.3%       0.01 ±192%  perf-sched.sch_delay.max.ms.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt.[unknown].[unknown]
      0.08 ± 15%    -100.0%       0.00        perf-sched.sch_delay.max.ms.rcu_gp_kthread.kthread.ret_from_fork.ret_from_fork_asm
      0.10 ± 23%     -92.1%       0.01 ± 26%  perf-sched.sch_delay.max.ms.schedule_hrtimeout_range.do_poll.constprop.0.do_sys_poll
      0.04 ± 38%    -100.0%       0.00        perf-sched.sch_delay.max.ms.schedule_hrtimeout_range.do_select.core_sys_select.do_pselect.constprop
      0.07 ±  4%    -100.0%       0.00        perf-sched.sch_delay.max.ms.schedule_timeout.__wait_for_common.__flush_work.__lru_add_drain_all
      0.09 ± 30%    -100.0%       0.00        perf-sched.sch_delay.max.ms.schedule_timeout.__wait_for_common.wait_for_completion_state.kernel_clone
      0.02 ± 54%    -100.0%       0.00        perf-sched.sch_delay.max.ms.schedule_timeout.io_schedule_timeout.__wait_for_common.blk_execute_rq
      0.09 ± 12%    -100.0%       0.00        perf-sched.sch_delay.max.ms.schedule_timeout.kcompactd.kthread.ret_from_fork
      0.01 ± 35%    -100.0%       0.00        perf-sched.sch_delay.max.ms.schedule_timeout.khugepaged_wait_work.khugepaged.kthread
      0.12 ± 26%    -100.0%       0.00        perf-sched.sch_delay.max.ms.schedule_timeout.kswapd_try_to_sleep.kswapd.kthread
      0.07 ± 27%     -62.3%       0.03 ± 94%  perf-sched.sch_delay.max.ms.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown]
      0.07 ± 15%    -100.0%       0.00        perf-sched.sch_delay.max.ms.wait_for_partner.fifo_open.do_dentry_open.vfs_open
      0.01 ± 17%    +593.2%       0.09 ± 74%  perf-sched.total_sch_delay.average.ms
     15.98 ±  2%     -85.5%       2.32 ± 48%  perf-sched.total_wait_and_delay.average.ms
     52651           -98.0%       1045 ± 30%  perf-sched.total_wait_and_delay.count.ms
      4255 ± 13%     -98.3%      73.23 ± 15%  perf-sched.total_wait_and_delay.max.ms
     15.97 ±  2%     -86.0%       2.24 ± 53%  perf-sched.total_wait_time.average.ms
      4255 ± 13%     -98.9%      45.89 ±  7%  perf-sched.total_wait_time.max.ms
      7.81 ±  7%     -97.5%       0.19 ± 62%  perf-sched.wait_and_delay.avg.ms.__cond_resched.__wait_for_common.affine_move_task.__set_cpus_allowed_ptr.__sched_setaffinity
     69.50 ± 29%    -100.0%       0.00        perf-sched.wait_and_delay.avg.ms.__cond_resched.shrink_folio_list.evict_folios.try_to_shrink_lruvec.shrink_one
    689.13 ± 14%    -100.0%       0.00        perf-sched.wait_and_delay.avg.ms.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown]
     17.56 ±  4%    -100.0%       0.00        perf-sched.wait_and_delay.avg.ms.do_nanosleep.hrtimer_nanosleep.common_nsleep.__x64_sys_clock_nanosleep
     32.71           -99.4%       0.21 ± 50%  perf-sched.wait_and_delay.avg.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.x64_sys_call
      0.78          -100.0%       0.00        perf-sched.wait_and_delay.avg.ms.io_cqring_wait.__do_sys_io_uring_enter.do_syscall_64.entry_SYSCALL_64_after_hwframe
     29.64 ±  3%     -97.5%       0.73 ±  7%  perf-sched.wait_and_delay.avg.ms.pipe_read.vfs_read.ksys_read.do_syscall_64
    403.38 ±  7%     -99.4%       2.51 ±223%  perf-sched.wait_and_delay.avg.ms.schedule_hrtimeout_range.do_poll.constprop.0.do_sys_poll
    158.27          -100.0%       0.00        perf-sched.wait_and_delay.avg.ms.schedule_hrtimeout_range.do_select.core_sys_select.do_pselect.constprop
    112.37 ± 23%     -96.3%       4.12 ±103%  perf-sched.wait_and_delay.avg.ms.schedule_hrtimeout_range.ep_poll.do_epoll_wait.__x64_sys_epoll_wait
    285.75 ± 73%    -100.0%       0.00        perf-sched.wait_and_delay.avg.ms.schedule_timeout.io_schedule_timeout.__wait_for_common.blk_execute_rq
    453.18          -100.0%       0.00        perf-sched.wait_and_delay.avg.ms.schedule_timeout.kcompactd.kthread.ret_from_fork
     47.62 ± 62%    -100.0%       0.00        perf-sched.wait_and_delay.avg.ms.schedule_timeout.kswapd_try_to_sleep.kswapd.kthread
    500.57 ±  7%     -98.9%       5.42 ± 59%  perf-sched.wait_and_delay.avg.ms.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm
    561.24 ±  5%     -99.5%       3.01 ± 50%  perf-sched.wait_and_delay.avg.ms.worker_thread.kthread.ret_from_fork.ret_from_fork_asm
    577.00           -40.4%     343.83 ± 23%  perf-sched.wait_and_delay.count.__cond_resched.__wait_for_common.affine_move_task.__set_cpus_allowed_ptr.__sched_setaffinity
      6.00 ± 31%    -100.0%       0.00        perf-sched.wait_and_delay.count.__cond_resched.shrink_folio_list.evict_folios.try_to_shrink_lruvec.shrink_one
     10.17 ±  3%    -100.0%       0.00        perf-sched.wait_and_delay.count.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown]
    501.83          -100.0%       0.00        perf-sched.wait_and_delay.count.do_nanosleep.hrtimer_nanosleep.common_nsleep.__x64_sys_clock_nanosleep
     45905          -100.0%       0.00        perf-sched.wait_and_delay.count.io_cqring_wait.__do_sys_io_uring_enter.do_syscall_64.entry_SYSCALL_64_after_hwframe
      2426 ±  2%     -97.9%      51.67 ± 61%  perf-sched.wait_and_delay.count.pipe_read.vfs_read.ksys_read.do_syscall_64
     37.83 ±  5%     -98.7%       0.50 ±223%  perf-sched.wait_and_delay.count.schedule_hrtimeout_range.do_poll.constprop.0.do_sys_poll
     30.00          -100.0%       0.00        perf-sched.wait_and_delay.count.schedule_hrtimeout_range.do_select.core_sys_select.do_pselect.constprop
      2.33 ± 20%    -100.0%       0.00        perf-sched.wait_and_delay.count.schedule_timeout.io_schedule_timeout.__wait_for_common.blk_execute_rq
     19.83          -100.0%       0.00        perf-sched.wait_and_delay.count.schedule_timeout.kcompactd.kthread.ret_from_fork
    134.67 ± 21%    -100.0%       0.00        perf-sched.wait_and_delay.count.schedule_timeout.kswapd_try_to_sleep.kswapd.kthread
      1195 ±  3%     -99.1%      10.17 ± 54%  perf-sched.wait_and_delay.count.schedule_timeout.rcu_gp_fqs_loop.rcu_gp_kthread.kthread
    925.67 ±  7%     -58.9%     380.00 ± 24%  perf-sched.wait_and_delay.count.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm
    362.33 ±  5%     -79.0%      76.17 ± 58%  perf-sched.wait_and_delay.count.worker_thread.kthread.ret_from_fork.ret_from_fork_asm
      1013 ±  2%     -94.8%      53.21 ± 65%  perf-sched.wait_and_delay.max.ms.__cond_resched.__wait_for_common.affine_move_task.__set_cpus_allowed_ptr.__sched_setaffinity
     89.90 ± 17%    -100.0%       0.00        perf-sched.wait_and_delay.max.ms.__cond_resched.shrink_folio_list.evict_folios.try_to_shrink_lruvec.shrink_one
    999.82          -100.0%       0.00        perf-sched.wait_and_delay.max.ms.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown]
      1000          -100.0%       0.00        perf-sched.wait_and_delay.max.ms.do_nanosleep.hrtimer_nanosleep.common_nsleep.__x64_sys_clock_nanosleep
    890.88 ±  8%     -99.4%       5.21 ±101%  perf-sched.wait_and_delay.max.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.x64_sys_call
     13.58 ±106%    -100.0%       0.00        perf-sched.wait_and_delay.max.ms.io_cqring_wait.__do_sys_io_uring_enter.do_syscall_64.entry_SYSCALL_64_after_hwframe
      1013           -99.8%       1.97 ± 28%  perf-sched.wait_and_delay.max.ms.pipe_read.vfs_read.ksys_read.do_syscall_64
      1464 ± 16%     -99.5%       7.52 ±223%  perf-sched.wait_and_delay.max.ms.schedule_hrtimeout_range.do_poll.constprop.0.do_sys_poll
    250.08          -100.0%       0.00        perf-sched.wait_and_delay.max.ms.schedule_hrtimeout_range.do_select.core_sys_select.do_pselect.constprop
    334.30 ± 23%     -95.4%      15.29 ±103%  perf-sched.wait_and_delay.max.ms.schedule_hrtimeout_range.ep_poll.do_epoll_wait.__x64_sys_epoll_wait
    355.60 ± 68%    -100.0%       0.00        perf-sched.wait_and_delay.max.ms.schedule_timeout.io_schedule_timeout.__wait_for_common.blk_execute_rq
    504.88          -100.0%       0.00        perf-sched.wait_and_delay.max.ms.schedule_timeout.kcompactd.kthread.ret_from_fork
    111.51 ±  7%    -100.0%       0.00        perf-sched.wait_and_delay.max.ms.schedule_timeout.kswapd_try_to_sleep.kswapd.kthread
      6.34 ± 18%     -28.2%       4.55 ± 11%  perf-sched.wait_and_delay.max.ms.schedule_timeout.rcu_gp_fqs_loop.rcu_gp_kthread.kthread
      3150 ± 21%     -98.5%      45.76 ±  7%  perf-sched.wait_and_delay.max.ms.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm
      4235 ± 14%     -99.3%      29.44 ± 43%  perf-sched.wait_and_delay.max.ms.worker_thread.kthread.ret_from_fork.ret_from_fork_asm
      7.76 ±  7%     -98.8%       0.10 ± 61%  perf-sched.wait_time.avg.ms.__cond_resched.__wait_for_common.affine_move_task.__set_cpus_allowed_ptr.__sched_setaffinity
      0.96 ± 56%    -100.0%       0.00        perf-sched.wait_time.avg.ms.__cond_resched.filemap_read.xfs_file_buffered_read.xfs_file_read_iter.__io_read
     69.43 ± 29%    -100.0%       0.00        perf-sched.wait_time.avg.ms.__cond_resched.shrink_folio_list.evict_folios.try_to_shrink_lruvec.shrink_one
     45.25 ± 92%    -100.0%       0.00        perf-sched.wait_time.avg.ms.__cond_resched.try_to_shrink_lruvec.shrink_one.shrink_many.shrink_node
    689.10 ± 14%    -100.0%       0.00        perf-sched.wait_time.avg.ms.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown]
      1.38 ±  9%    -100.0%       0.00        perf-sched.wait_time.avg.ms.devkmsg_read.vfs_read.ksys_read.do_syscall_64
     17.55 ±  4%    -100.0%       0.00        perf-sched.wait_time.avg.ms.do_nanosleep.hrtimer_nanosleep.common_nsleep.__x64_sys_clock_nanosleep
     32.71           -99.4%       0.21 ± 50%  perf-sched.wait_time.avg.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.x64_sys_call
      0.41 ±  4%     -14.0%       0.35 ±  3%  perf-sched.wait_time.avg.ms.do_wait.kernel_wait4.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.77          -100.0%       0.00        perf-sched.wait_time.avg.ms.io_cqring_wait.__do_sys_io_uring_enter.do_syscall_64.entry_SYSCALL_64_after_hwframe
    194.51 ±103%    -100.0%       0.00 ±223%  perf-sched.wait_time.avg.ms.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt.[unknown].[unknown]
     29.63 ±  3%     -97.6%       0.71 ±  8%  perf-sched.wait_time.avg.ms.pipe_read.vfs_read.ksys_read.do_syscall_64
      4.01 ±  7%    -100.0%       0.00        perf-sched.wait_time.avg.ms.rcu_gp_kthread.kthread.ret_from_fork.ret_from_fork_asm
    403.36 ±  7%     -99.3%       2.82 ±193%  perf-sched.wait_time.avg.ms.schedule_hrtimeout_range.do_poll.constprop.0.do_sys_poll
    158.25          -100.0%       0.00        perf-sched.wait_time.avg.ms.schedule_hrtimeout_range.do_select.core_sys_select.do_pselect.constprop
    112.32 ± 23%     -96.5%       3.93 ±106%  perf-sched.wait_time.avg.ms.schedule_hrtimeout_range.ep_poll.do_epoll_wait.__x64_sys_epoll_wait
      0.66 ±  4%    -100.0%       0.00        perf-sched.wait_time.avg.ms.schedule_timeout.__wait_for_common.wait_for_completion_state.kernel_clone
    285.74 ± 73%    -100.0%       0.00        perf-sched.wait_time.avg.ms.schedule_timeout.io_schedule_timeout.__wait_for_common.blk_execute_rq
    453.14          -100.0%       0.00        perf-sched.wait_time.avg.ms.schedule_timeout.kcompactd.kthread.ret_from_fork
      0.07 ± 83%    -100.0%       0.00        perf-sched.wait_time.avg.ms.schedule_timeout.khugepaged_wait_work.khugepaged.kthread
     47.57 ± 62%    -100.0%       0.00        perf-sched.wait_time.avg.ms.schedule_timeout.kswapd_try_to_sleep.kswapd.kthread
      3.86 ±  6%     -24.9%       2.90 ± 23%  perf-sched.wait_time.avg.ms.schedule_timeout.rcu_gp_fqs_loop.rcu_gp_kthread.kthread
    500.56 ±  7%     -98.9%       5.38 ± 60%  perf-sched.wait_time.avg.ms.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm
    226.30 ± 64%    -100.0%       0.01 ±107%  perf-sched.wait_time.avg.ms.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown]
    561.18 ±  5%     -99.6%       2.50 ± 79%  perf-sched.wait_time.avg.ms.worker_thread.kthread.ret_from_fork.ret_from_fork_asm
      1013 ±  2%     -97.4%      26.61 ± 65%  perf-sched.wait_time.max.ms.__cond_resched.__wait_for_common.affine_move_task.__set_cpus_allowed_ptr.__sched_setaffinity
      1.65 ± 66%    -100.0%       0.00        perf-sched.wait_time.max.ms.__cond_resched.filemap_read.xfs_file_buffered_read.xfs_file_read_iter.__io_read
     89.83 ± 17%    -100.0%       0.00        perf-sched.wait_time.max.ms.__cond_resched.shrink_folio_list.evict_folios.try_to_shrink_lruvec.shrink_one
     91.01 ± 14%    -100.0%       0.00        perf-sched.wait_time.max.ms.__cond_resched.try_to_shrink_lruvec.shrink_one.shrink_many.shrink_node
    999.79          -100.0%       0.00        perf-sched.wait_time.max.ms.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown]
      2.76 ±  9%    -100.0%       0.00        perf-sched.wait_time.max.ms.devkmsg_read.vfs_read.ksys_read.do_syscall_64
      1000          -100.0%       0.00        perf-sched.wait_time.max.ms.do_nanosleep.hrtimer_nanosleep.common_nsleep.__x64_sys_clock_nanosleep
    890.88 ±  8%     -99.4%       5.21 ±101%  perf-sched.wait_time.max.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.x64_sys_call
     15.72 ±  2%     -93.4%       1.04 ± 28%  perf-sched.wait_time.max.ms.do_wait.kernel_wait4.do_syscall_64.entry_SYSCALL_64_after_hwframe
     13.23 ±110%    -100.0%       0.00        perf-sched.wait_time.max.ms.io_cqring_wait.__do_sys_io_uring_enter.do_syscall_64.entry_SYSCALL_64_after_hwframe
    500.20 ± 99%    -100.0%       0.01 ±223%  perf-sched.wait_time.max.ms.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt.[unknown].[unknown]
      1013           -99.8%       1.96 ± 29%  perf-sched.wait_time.max.ms.pipe_read.vfs_read.ksys_read.do_syscall_64
      4.85 ±  6%    -100.0%       0.00        perf-sched.wait_time.max.ms.rcu_gp_kthread.kthread.ret_from_fork.ret_from_fork_asm
      1464 ± 16%     -99.4%       8.15 ±202%  perf-sched.wait_time.max.ms.schedule_hrtimeout_range.do_poll.constprop.0.do_sys_poll
    250.06          -100.0%       0.00        perf-sched.wait_time.max.ms.schedule_hrtimeout_range.do_select.core_sys_select.do_pselect.constprop
    334.27 ± 23%     -95.4%      15.28 ±103%  perf-sched.wait_time.max.ms.schedule_hrtimeout_range.ep_poll.do_epoll_wait.__x64_sys_epoll_wait
      2.04 ±  7%    -100.0%       0.00        perf-sched.wait_time.max.ms.schedule_timeout.__wait_for_common.wait_for_completion_state.kernel_clone
    355.59 ± 68%    -100.0%       0.00        perf-sched.wait_time.max.ms.schedule_timeout.io_schedule_timeout.__wait_for_common.blk_execute_rq
    504.84          -100.0%       0.00        perf-sched.wait_time.max.ms.schedule_timeout.kcompactd.kthread.ret_from_fork
      0.07 ± 83%    -100.0%       0.00        perf-sched.wait_time.max.ms.schedule_timeout.khugepaged_wait_work.khugepaged.kthread
    111.44 ±  7%    -100.0%       0.00        perf-sched.wait_time.max.ms.schedule_timeout.kswapd_try_to_sleep.kswapd.kthread
      3150 ± 21%     -98.7%      39.96 ± 40%  perf-sched.wait_time.max.ms.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm
    843.01 ± 41%    -100.0%       0.03 ± 94%  perf-sched.wait_time.max.ms.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown]
      4235 ± 14%     -99.5%      21.72 ± 79%  perf-sched.wait_time.max.ms.worker_thread.kthread.ret_from_fork.ret_from_fork_asm

Disclaimer:
Results have been estimated based on internal Intel analysis and are provided for informational purposes only. Any difference in system hardware or software design or configuration may affect actual performance.

Signature · 2 lines
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki
maintainer
#1 lore raw
Quoted 11 lines · kernel test robot

Hello,

kernel test robot noticed a 51.1% regression of fio.read_bw_MBps on:

commit: 8cf0c459993ee2911f4f01fba21b1987b102c887 ("io_uring/rw: Allocate async data through helper")
https://git.kernel.org/cgit/linux/kernel/git/next/linux-next.git master

[test failed on linux-next/master 8155b4ef3466f0e289e8fcc9e6e62f3f4dceeac2]

Took a quick look, and I think that patch forgets to clear ->bytes_done when we don't have a free_iovec. That looks like a mistake, it should always get cleared, it has no dependence on ->free_iovec.

io_uring/rw.c +1 -1
index 75f70935ccf4..ca1b19d3d142 100644
@@ -228,8 +228,8 @@ static int io_rw_alloc_async(struct io_kiocb *req)
 		kasan_mempool_unpoison_object(rw->free_iovec,
 					      rw->free_iov_nr * sizeof(struct iovec));
 		req->flags |= REQ_F_NEED_CLEANUP;
-		rw->bytes_done = 0;
 	}
+	rw->bytes_done = 0;
 	return 0;
 }
 
Signature · 1 line
Jens Axboe
author Tested-by
#2 lore raw

hi, Jens Axboe,

Quoted 8 lines · Jens Axboe
Quoted 7 lines · kernel test robot

Hello,

kernel test robot noticed a 51.1% regression of fio.read_bw_MBps on:

Quoted 8 lines
Quoted 4 lines

commit: 8cf0c459993ee2911f4f01fba21b1987b102c887 ("io_uring/rw: Allocate async data through helper")
https://git.kernel.org/cgit/linux/kernel/git/next/linux-next.git master

[test failed on linux-next/master 8155b4ef3466f0e289e8fcc9e6e62f3f4dceeac2]

Took a quick look, and I think that patch forgets to clear ->bytes_done when we don't have a free_iovec. That looks like a mistake, it should always get cleared, it has no dependence on ->free_iovec.

below patch recover the performance fully. thanks!

=========================================================================================
bs/compiler/cpufreq_governor/disk/fs/ioengine/kconfig/nr_task/rootfs/runtime/rw/tbox_group/test_size/testcase:
2M/gcc-12/performance/1SSD/xfs/io_uring/x86_64-rhel-9.4/8t/debian-12-x86_64-20240206.cgz/300s/read/lkp-csl-2sp3/256g/fio-basic

commit:
23d91035ca ("io_uring/net: Allocate msghdr async data through helper") 8cf0c45999 ("io_uring/rw: Allocate async data through helper") 605f6d311e <---- your patch

23d91035cafa30d1 8cf0c459993ee2911f4f01fba21 605f6d311ea47324304d60dd32a ---------------- --------------------------- ---------------------------

         %stddev     %change         %stddev     %change         %stddev
             \          |                \          |                \
      2171           -51.1%       1060 ±  3%      +0.2%       2174        fio.read_bw_MBps

full comparison as below [1] FYI

Quoted 18 lines
io_uring/rw.c +1 -1
index 75f70935ccf4..ca1b19d3d142 100644
@@ -228,8 +228,8 @@ static int io_rw_alloc_async(struct io_kiocb *req)
 		kasan_mempool_unpoison_object(rw->free_iovec,
 					      rw->free_iov_nr * sizeof(struct iovec));
 		req->flags |= REQ_F_NEED_CLEANUP;
-		rw->bytes_done = 0;
 	}
+	rw->bytes_done = 0;
 	return 0;
 }
 
Signature · 2 lines
-- 
Jens Axboe

[1]
=========================================================================================
bs/compiler/cpufreq_governor/disk/fs/ioengine/kconfig/nr_task/rootfs/runtime/rw/tbox_group/test_size/testcase:
2M/gcc-12/performance/1SSD/xfs/io_uring/x86_64-rhel-9.4/8t/debian-12-x86_64-20240206.cgz/300s/read/lkp-csl-2sp3/256g/fio-basic

commit:
23d91035ca ("io_uring/net: Allocate msghdr async data through helper") 8cf0c45999 ("io_uring/rw: Allocate async data through helper") 605f6d311e <---- your patch

23d91035cafa30d1 8cf0c459993ee2911f4f01fba21 605f6d311ea47324304d60dd32a ---------------- --------------------------- ---------------------------
%stddev %change %stddev %change %stddev
\ | \ | \
1.169e+10 -97.9% 2.464e+08 ± 4% -0.9% 1.158e+10 cpuidle..time
4187672 ± 5% -98.3% 69610 +1.4% 4247402 ± 3% cpuidle..usage

    490.00           -24.5%     370.16            +0.2%     491.13        uptime.boot
     45002           -23.4%      34483            +0.2%      45104        uptime.idle
     91.67            +3.9%      95.24            -0.1%      91.56        iostat.cpu.idle
      6.67           -83.6%       1.09 ±  2%      +0.6%       6.71        iostat.cpu.iowait
      0.15 ±  3%   +1569.3%       2.51 ±  5%      +3.2%       0.16 ±  2%  iostat.cpu.user
      6.77 ±  2%      -5.2        1.52 ±  4%      +0.0        6.81        mpstat.cpu.all.iowait%
      0.03 ± 22%      +0.0        0.05 ± 18%      -0.0        0.03 ± 24%  mpstat.cpu.all.soft%
      0.15 ±  4%      +4.1        4.23 ±  6%      +0.0        0.15 ±  2%  mpstat.cpu.all.usr%
   5982113 ± 42%     -97.3%     160513 ± 30%      -4.7%    5699677 ± 47%  numa-numastat.node0.local_node
   6032931 ± 41%     -96.5%     209090 ± 13%      -4.6%    5755952 ± 47%  numa-numastat.node0.numa_hit
   2662250 ±124%     -94.1%     155780 ± 31%     -33.0%    1783404 ±162%  numa-numastat.node1.local_node
   2710984 ±121%     -92.4%     206648 ± 13%     -32.5%    1829467 ±159%  numa-numastat.node1.numa_hit
   2142706           -87.8%     262066            +0.6%    2156601        vmstat.io.bi
     62.03         +2998.4%       1922            +0.0%      62.04        vmstat.io.bo
      6.47 ±  3%    -100.0%       0.00            +0.3%       6.49        vmstat.procs.b
      2.69 ± 12%    +173.1%       7.33 ±  8%      +2.8%       2.76 ±  9%  vmstat.procs.r
     19637           -46.7%      10473 ±  4%      +0.3%      19702        vmstat.system.cs
     34417 ±  3%     -10.8%      30715 ±  2%      +1.8%      35043 ±  3%  vmstat.system.in
    121.24           -99.2%       0.96 ±  6%      -0.2%     120.98        time.elapsed_time
    121.24           -99.2%       0.96 ±  6%      -0.2%     120.98        time.elapsed_time.max
 5.369e+08           -99.6%    2096533            +0.0%   5.37e+08        time.file_system_inputs
    340.33 ±  6%     -53.7%     157.67 ±  9%      +2.3%     348.20 ±  7%  time.involuntary_context_switches
      8592 ±  3%     -10.2%       7716            +0.1%       8598 ±  4%  time.minor_page_faults
     89.00 ±  4%    +587.6%     612.00 ±  6%      -2.8%      86.50 ±  3%  time.percent_of_cpu_this_job_got
    102.38 ±  4%     -99.6%       0.44 ±  2%      -3.3%      98.96 ±  3%  time.system_time
   1101885           -99.6%       4454            -0.3%    1098618        time.voluntary_context_switches
      2.37 ± 49%      -2.1        0.26 ±103%      +0.8        3.16 ± 26%  fio.latency_100ms%
     81.78 ±  3%     -56.2       25.56 ± 29%      -0.5       81.27 ±  5%  fio.latency_250ms%
      0.34 ± 66%      -0.3        0.00            +0.6        0.94 ± 54%  fio.latency_50ms%
      2171           -51.1%       1060 ±  3%      +0.2%       2174        fio.read_bw_MBps
 4.194e+08           -14.2%    3.6e+08 ±  5%      +8.0%   4.53e+08 ± 11%  fio.read_clat_99%_ns
    121.24           -99.2%       0.96 ±  6%      -0.2%     120.98        fio.time.elapsed_time
    121.24           -99.2%       0.96 ±  6%      -0.2%     120.98        fio.time.elapsed_time.max
 5.369e+08           -99.6%    2096533            +0.0%   5.37e+08        fio.time.file_system_inputs
      8592 ±  3%     -10.2%       7716            +0.1%       8598 ±  4%  fio.time.minor_page_faults
     89.00 ±  4%    +587.6%     612.00 ±  6%      -2.8%      86.50 ±  3%  fio.time.percent_of_cpu_this_job_got
    102.38 ±  4%     -99.6%       0.44 ±  2%      -3.3%      98.96 ±  3%  fio.time.system_time
   1101885           -99.6%       4454            -0.3%    1098618        fio.time.voluntary_context_switches
    131072           -99.6%     510.83            +0.0%     131072        fio.workload
   6857403 ± 71%    -100.0%       0.00           +19.9%    8220508 ± 50%  numa-meminfo.node0.Active(file)
  49781569 ± 10%     -98.0%     971701 ±119%      +2.0%   50763968 ± 10%  numa-meminfo.node0.FilePages
  42007683 ±  4%     -99.4%     232792 ± 70%      -2.9%   40784060 ±  5%  numa-meminfo.node0.Inactive
  41773509 ±  4%     -99.4%     232792 ± 70%      -2.9%   40541517 ±  5%  numa-meminfo.node0.Inactive(file)
    181744 ± 10%     -80.2%      36054 ± 50%      +5.2%     191237 ± 11%  numa-meminfo.node0.KReclaimable
  11216992 ± 48%    +440.7%   60655275           -10.4%   10049266 ± 55%  numa-meminfo.node0.MemFree
  54456267 ±  9%     -90.8%    5017984 ± 22%      +2.1%   55623993 ±  9%  numa-meminfo.node0.MemUsed
    181744 ± 10%     -80.2%      36054 ± 50%      +5.2%     191237 ± 11%  numa-meminfo.node0.SReclaimable
     11353 ± 48%     -50.3%       5640 ±  9%      -9.8%      10245 ± 60%  numa-meminfo.node0.Shmem
  46742560 ± 10%     -93.7%    2968006 ± 39%      -2.4%   45597696 ± 11%  numa-meminfo.node1.FilePages
  44279309 ± 12%     -99.7%     116761 ±141%      -0.7%   43985860 ± 12%  numa-meminfo.node1.Inactive
  44279309 ± 12%     -99.7%     116761 ±141%      -0.7%   43985860 ± 12%  numa-meminfo.node1.Inactive(file)
    222588 ±  9%     -71.5%      63492 ± 29%      -5.5%     210385 ± 10%  numa-meminfo.node1.KReclaimable
  17706281 ± 29%    +248.8%   61765996            +7.6%   19054511 ± 28%  numa-meminfo.node1.MemFree
  48294686 ± 10%     -91.2%    4234972 ± 26%      -2.8%   46946456 ± 11%  numa-meminfo.node1.MemUsed
    222588 ±  9%     -71.5%      63492 ± 29%      -5.5%     210385 ± 10%  numa-meminfo.node1.SReclaimable
     19553 ± 30%     -91.4%       1688 ± 29%      +4.2%      20377 ± 31%  numa-meminfo.node1.Shmem
    484021 ± 22%     -54.2%     221920 ± 54%     -15.5%     408872 ± 16%  numa-meminfo.node1.Slab
   8292851 ± 60%     -91.7%     685299            +9.9%    9110413 ± 44%  meminfo.Active
    993491 ± 17%     -31.0%     685299            -1.3%     980308 ± 15%  meminfo.Active(anon)
   7299359 ± 70%    -100.0%       0.00           +11.4%    8130104 ± 51%  meminfo.Active(file)
   1059116 ±  3%     -57.9%     446122 ± 20%      -9.0%     963510 ± 10%  meminfo.AnonHugePages
   1199023           -43.1%     682106            -0.3%    1194992        meminfo.AnonPages
  96504886           -95.9%    3939272            -0.2%   96340575        meminfo.Cached
   1771531           -52.2%     846516 ±  3%      +0.2%    1774404        meminfo.Committed_AS
  85824229 ±  5%     -99.6%     349553            -1.1%   84838187 ±  4%  meminfo.Inactive
  85591704 ±  5%     -99.6%     349553            -1.2%   84596818 ±  4%  meminfo.Inactive(file)
    404111           -75.5%      99205            -0.7%     401394        meminfo.KReclaimable
     82199           -26.3%      60586            +0.4%      82511        meminfo.Mapped
  28942581 ±  2%    +323.0%  1.224e+08            +0.6%   29124721        meminfo.MemFree
 1.027e+08           -91.0%    9249647            -0.2%  1.025e+08        meminfo.Memused
      2432           -62.3%     917.33 ± 79%      +0.0%       2432 ±  3%  meminfo.Mlocked
    404111           -75.5%      99205            -0.7%     401394        meminfo.SReclaimable
    600546           -30.5%     417425            -0.1%     600233        meminfo.SUnreclaim
     30852 ±  2%     -77.6%       6905 ±  5%      -1.0%      30536        meminfo.Shmem
   1004658           -48.6%     516630            -0.3%    1001628        meminfo.Slab
 1.028e+08           -91.0%    9249647            -0.2%  1.026e+08        meminfo.max_used_kB
   1780344 ± 71%    -100.0%       0.00           +25.0%    2225468 ± 50%  numa-vmstat.node0.nr_active_file
  12441416 ± 10%     -98.0%     242939 ±119%      +2.0%   12690240 ± 10%  numa-vmstat.node0.nr_file_pages
   2808184 ± 48%    +440.0%   15163781           -10.5%    2513104 ± 55%  numa-vmstat.node0.nr_free_pages
  10373418 ±  5%     -99.4%      58198 ± 70%      -3.9%    9964288 ±  3%  numa-vmstat.node0.nr_inactive_file
      2830 ± 48%     -49.7%       1424 ±  8%      -9.6%       2559 ± 60%  numa-vmstat.node0.nr_shmem
     45390 ± 10%     -80.1%       9015 ± 50%      +5.3%      47791 ± 11%  numa-vmstat.node0.nr_slab_reclaimable
   1779467 ± 71%    -100.0%       0.00           +25.0%    2224367 ± 50%  numa-vmstat.node0.nr_zone_active_file
  10374301 ±  5%     -99.4%      58198 ± 70%      -3.9%    9965396 ±  3%  numa-vmstat.node0.nr_zone_inactive_file
   6032577 ± 41%     -96.5%     209022 ± 13%      -4.6%    5755362 ± 47%  numa-vmstat.node0.numa_hit
   5981759 ± 42%     -97.3%     160445 ± 30%      -4.7%    5699087 ± 47%  numa-vmstat.node0.numa_local
     60865 ± 46%    -100.0%       0.00           -16.8%      50609 ± 41%  numa-vmstat.node0.workingset_nodes
  11683973 ± 10%     -93.6%     742001 ± 39%      -2.4%   11401511 ± 11%  numa-vmstat.node1.nr_file_pages
   4428033 ± 29%    +248.7%   15441242            +7.5%    4761384 ± 28%  numa-vmstat.node1.nr_free_pages
  11068153 ± 12%     -99.7%      29190 ±141%      -0.6%   10998539 ± 12%  numa-vmstat.node1.nr_inactive_file
      4895 ± 30%     -91.4%     422.28 ± 29%      +4.3%       5107 ± 31%  numa-vmstat.node1.nr_shmem
     55624 ±  9%     -71.5%      15873 ± 29%      -5.4%      52619 ± 10%  numa-vmstat.node1.nr_slab_reclaimable
  11068150 ± 12%     -99.7%      29190 ±141%      -0.6%   10998535 ± 12%  numa-vmstat.node1.nr_zone_inactive_file
   2710384 ±121%     -92.4%     205957 ± 13%     -32.5%    1828924 ±159%  numa-vmstat.node1.numa_hit
   2661650 ±124%     -94.2%     155089 ± 31%     -33.0%    1782861 ±162%  numa-vmstat.node1.numa_local
     96303 ± 31%    -100.0%       0.00            +7.8%     103794 ± 21%  numa-vmstat.node1.workingset_nodes
     21.85 ± 22%     -95.7%       0.93 ±  9%      -6.2%      20.49 ± 21%  perf-stat.i.MPKI
 7.833e+08 ± 36%    +248.8%  2.733e+09           +11.1%  8.702e+08 ± 28%  perf-stat.i.branch-instructions
      1.20 ± 21%      +1.4        2.60            -0.1        1.11 ± 23%  perf-stat.i.branch-miss-rate%
  10294722 ±  8%    +588.8%   70910805            +1.9%   10489353 ±  5%  perf-stat.i.branch-misses
     71.04 ±  7%     -45.3       25.71 ±  6%      +1.9       72.95 ±  7%  perf-stat.i.cache-miss-rate%
  67179322 ± 23%     -82.0%   12088775 ±  8%      +7.8%   72424185 ± 18%  perf-stat.i.cache-misses
  92053950 ± 16%     -49.0%   46958614 ±  2%      +5.7%   97306974 ± 12%  perf-stat.i.cache-references
     19893           -57.6%       8444 ±  6%      +0.5%      19982        perf-stat.i.context-switches
      1.42 ± 18%     +47.0%       2.09 ±  6%      -3.7%       1.37 ± 21%  perf-stat.i.cpi
     96008            +3.1%      99012            -0.0%      96007        perf-stat.i.cpu-clock
 5.166e+09 ± 27%    +426.7%  2.721e+10 ±  5%      +9.2%  5.642e+09 ± 19%  perf-stat.i.cpu-cycles
    109.64          +297.0%     435.28 ± 26%      -0.2%     109.44        perf-stat.i.cpu-migrations
     95.02 ± 12%   +2287.2%       2268 ± 10%      -1.4%      93.64 ±  4%  perf-stat.i.cycles-between-cache-misses
 4.488e+09 ± 40%    +190.0%  1.301e+10           +12.3%  5.042e+09 ± 30%  perf-stat.i.instructions
      0.78 ± 18%     -38.5%       0.48 ±  6%      +4.8%       0.82 ± 17%  perf-stat.i.ipc
      3257         +2386.4%      81003 ±  2%      -0.3%       3248        perf-stat.i.minor-faults
      3257         +2386.3%      81001 ±  2%      -0.3%       3248        perf-stat.i.page-faults
     96008            +3.1%      99013            -0.0%      96007        perf-stat.i.task-clock
     16.77 ± 26%     -94.5%       0.93 ±  9%      -7.2%      15.57 ± 25%  perf-stat.overall.MPKI
      1.53 ± 39%      +1.1        2.60            -0.2        1.36 ± 39%  perf-stat.overall.branch-miss-rate%
     72.01 ±  8%     -46.3       25.71 ±  6%      +1.7       73.69 ±  8%  perf-stat.overall.cache-miss-rate%
      1.26 ± 22%     +66.0%       2.09 ±  6%      -4.2%       1.21 ± 23%  perf-stat.overall.cpi
     76.07 ±  5%   +2881.9%       2268 ± 10%      +2.2%      77.75        perf-stat.overall.cycles-between-cache-misses
      0.83 ± 18%     -42.0%       0.48 ±  6%      +4.4%       0.86 ± 17%  perf-stat.overall.ipc
   4201457 ± 40%    +507.2%   25510793           +10.9%    4661229 ± 30%  perf-stat.overall.path-length
 7.771e+08 ± 36%     +72.4%   1.34e+09           +11.1%  8.631e+08 ± 28%  perf-stat.ps.branch-instructions
  10227839 ±  8%    +239.9%   34759364            +1.8%   10413922 ±  5%  perf-stat.ps.branch-misses
  66623351 ± 23%     -91.1%    5924306 ±  8%      +7.8%   71821853 ± 18%  perf-stat.ps.cache-misses
  91293692 ± 16%     -74.8%   23018351 ±  2%      +5.7%   96498110 ± 12%  perf-stat.ps.cache-references
     19729           -79.0%       4140 ±  6%      +0.4%      19817        perf-stat.ps.context-switches
     95218           -49.0%      48533            -0.0%      95211        perf-stat.ps.cpu-clock
 5.123e+09 ± 27%    +160.3%  1.334e+10 ±  5%      +9.2%  5.594e+09 ± 19%  perf-stat.ps.cpu-cycles
    108.79           +95.9%     213.12 ± 25%      -0.2%     108.56        perf-stat.ps.cpu-migrations
 4.452e+09 ± 40%     +43.3%   6.38e+09           +12.3%      5e+09 ± 30%  perf-stat.ps.instructions
      3230         +1129.5%      39716 ±  3%      -0.4%       3218        perf-stat.ps.minor-faults
      3230         +1129.5%      39715 ±  3%      -0.4%       3218        perf-stat.ps.page-faults
     95218           -49.0%      48533            -0.0%      95211        perf-stat.ps.task-clock
 5.507e+11 ± 40%     -97.6%  1.303e+10           +10.9%   6.11e+11 ± 30%  perf-stat.total.instructions
    428.50 ±113%    -100.0%       0.00           -32.7%     288.40 ±132%  proc-vmstat.kswapd_high_wmark_hit_quickly
      1663 ± 68%    -100.0%       0.00           +20.0%       1995 ± 47%  proc-vmstat.kswapd_low_wmark_hit_quickly
    248125 ± 17%     -30.9%     171528            -1.3%     244986 ± 15%  proc-vmstat.nr_active_anon
   1740791 ± 71%    -100.0%       0.00           +16.8%    2033072 ± 51%  proc-vmstat.nr_active_file
    299833           -43.1%     170711            -0.4%     298782        proc-vmstat.nr_anon_pages
    517.25 ±  3%     -57.9%     217.83 ± 20%      -9.0%     470.49 ± 10%  proc-vmstat.nr_anon_transparent_hugepages
  24125371           -95.9%     984856            -0.2%   24081001        proc-vmstat.nr_file_pages
   7236331          +322.9%   30605541            +0.7%    7285216        proc-vmstat.nr_free_pages
  21481117 ±  5%     -99.6%      87388            -1.6%   21144500 ±  4%  proc-vmstat.nr_inactive_file
     20818           -25.3%      15557            +0.4%      20894        proc-vmstat.nr_mapped
    608.02           -62.3%     229.33 ± 79%      +0.0%     608.17 ±  3%  proc-vmstat.nr_mlock
      7717           -77.2%       1762 ±  4%      -0.9%       7646        proc-vmstat.nr_shmem
    101012           -75.4%      24886            -0.7%     100304        proc-vmstat.nr_slab_reclaimable
    150134           -30.4%     104433            -0.1%     150047        proc-vmstat.nr_slab_unreclaimable
    248125 ± 17%     -30.9%     171528            -1.3%     244986 ± 15%  proc-vmstat.nr_zone_active_anon
   1740790 ± 71%    -100.0%       0.00           +16.8%    2033071 ± 51%  proc-vmstat.nr_zone_active_file
  21481124 ±  5%     -99.6%      87388            -1.6%   21144511 ±  4%  proc-vmstat.nr_zone_inactive_file
   9755834 ± 25%    -100.0%       0.00           +12.1%   10940500 ±  3%  proc-vmstat.numa_foreign
    995.00 ± 26%     -95.4%      45.33 ±164%     +35.7%       1350 ± 82%  proc-vmstat.numa_hint_faults
    939.00 ± 28%     -96.3%      34.67 ±204%     +30.4%       1224 ± 91%  proc-vmstat.numa_hint_faults_local
   8745252 ± 28%     -95.2%     416538           -13.3%    7584794 ±  3%  proc-vmstat.numa_hit
   8645700 ± 28%     -96.3%     317091           -13.5%    7482456 ±  3%  proc-vmstat.numa_local
   9755622 ± 25%    -100.0%       0.00           +12.1%   10939557 ±  3%  proc-vmstat.numa_miss
   9856110 ± 25%     -99.0%      99445           +12.0%   11040265 ±  3%  proc-vmstat.numa_other
    386923           -97.4%       9959 ±222%      -0.6%     384707        proc-vmstat.numa_pte_updates
      2224 ± 32%    -100.0%       0.00            +9.4%       2433 ± 25%  proc-vmstat.pageoutrun
    423905 ±  3%    -100.0%       0.00            +1.5%     430262 ±  3%  proc-vmstat.pgalloc_dma32
  68828649           -98.9%     752946            +0.1%   68875748        proc-vmstat.pgalloc_normal
    498127           -61.7%     190797            -0.8%     494373        proc-vmstat.pgfault
  38943033           -99.2%     297228            +0.4%   39088883        proc-vmstat.pgfree
 2.685e+08           -99.6%    1048266            -0.0%  2.684e+08        proc-vmstat.pgpgin
     22286 ±  3%     -58.9%       9168 ±  6%      +1.3%      22573        proc-vmstat.pgreuse
  36664920          -100.0%       0.00            -0.0%   36656051        proc-vmstat.pgscan_file
  36664920          -100.0%       0.00            -0.0%   36656051        proc-vmstat.pgscan_kswapd
  36664893          -100.0%       0.00            -0.0%   36655993        proc-vmstat.pgsteal_file
  36664893          -100.0%       0.00            -0.0%   36655993        proc-vmstat.pgsteal_kswapd
     30510 ±  9%    -100.0%       0.00            +0.7%      30728 ±  8%  proc-vmstat.slabs_scanned
    157121 ±  3%    -100.0%       0.00            -2.1%     153839        proc-vmstat.workingset_nodes
      9.70 ± 66%      -9.4        0.32 ±223%      -0.9        8.78 ± 35%  perf-profile.calltrace.cycles-pp.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.write
      9.09 ± 54%      -9.1        0.00            -0.4        8.72 ± 25%  perf-profile.calltrace.cycles-pp.record__pushfn.perf_mmap__push.record__mmap_read_evlist.__cmd_record.cmd_record
      9.09 ± 54%      -9.1        0.00            -0.4        8.72 ± 25%  perf-profile.calltrace.cycles-pp.write.writen.record__pushfn.perf_mmap__push.record__mmap_read_evlist
      9.09 ± 54%      -9.1        0.00            -0.4        8.72 ± 25%  perf-profile.calltrace.cycles-pp.writen.record__pushfn.perf_mmap__push.record__mmap_read_evlist.__cmd_record
     13.49 ± 32%      -8.7        4.75 ± 38%      -1.1       12.42 ± 13%  perf-profile.calltrace.cycles-pp.handle_internal_command.main
     13.49 ± 32%      -8.7        4.75 ± 38%      -1.1       12.42 ± 13%  perf-profile.calltrace.cycles-pp.main
     13.49 ± 32%      -8.7        4.75 ± 38%      -1.1       12.42 ± 13%  perf-profile.calltrace.cycles-pp.run_builtin.handle_internal_command.main
      8.57 ± 57%      -8.6        0.00            -0.3        8.24 ± 25%  perf-profile.calltrace.cycles-pp.generic_perform_write.shmem_file_write_iter.vfs_write.ksys_write.do_syscall_64
      8.57 ± 57%      -8.6        0.00            -0.3        8.24 ± 25%  perf-profile.calltrace.cycles-pp.shmem_file_write_iter.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
      8.57 ± 57%      -8.6        0.00            -0.2        8.40 ± 27%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.write.writen.record__pushfn
      8.57 ± 57%      -8.6        0.00            -0.2        8.40 ± 27%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.write.writen.record__pushfn.perf_mmap__push
      8.57 ± 57%      -8.6        0.00            -0.2        8.40 ± 27%  perf-profile.calltrace.cycles-pp.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.write.writen
      9.34 ± 22%      -6.9        2.45 ± 67%      +0.0        9.35 ± 25%  perf-profile.calltrace.cycles-pp.__cmd_record.cmd_record.run_builtin.handle_internal_command.main
      9.34 ± 22%      -6.9        2.45 ± 67%      +0.0        9.35 ± 25%  perf-profile.calltrace.cycles-pp.cmd_record.run_builtin.handle_internal_command.main
      7.20 ± 25%      -5.5        1.70 ± 79%      +1.3        8.52 ± 28%  perf-profile.calltrace.cycles-pp.perf_mmap__push.record__mmap_read_evlist.__cmd_record.cmd_record.run_builtin
      7.20 ± 25%      -5.5        1.70 ± 79%      +1.3        8.52 ± 28%  perf-profile.calltrace.cycles-pp.record__mmap_read_evlist.__cmd_record.cmd_record.run_builtin.handle_internal_command
      0.37 ±142%      +1.0        1.38 ± 79%      +1.2        1.61 ± 25%  perf-profile.calltrace.cycles-pp.__tlb_batch_free_encoded_pages.tlb_finish_mmu.exit_mmap.__mmput.exit_mm
      0.37 ±142%      +1.0        1.38 ± 79%      +1.2        1.61 ± 25%  perf-profile.calltrace.cycles-pp.free_pages_and_swap_cache.__tlb_batch_free_encoded_pages.tlb_finish_mmu.exit_mmap.__mmput
      0.37 ±142%      +1.0        1.38 ± 79%      +1.2        1.61 ± 25%  perf-profile.calltrace.cycles-pp.tlb_finish_mmu.exit_mmap.__mmput.exit_mm.do_exit
      1.21 ±112%      +2.2        3.41 ± 70%      +3.1        4.30 ± 38%  perf-profile.calltrace.cycles-pp.zap_pmd_range.unmap_page_range.unmap_vmas.exit_mmap.__mmput
      1.21 ±112%      +2.2        3.41 ± 70%      +3.1        4.30 ± 38%  perf-profile.calltrace.cycles-pp.zap_pte_range.zap_pmd_range.unmap_page_range.unmap_vmas.exit_mmap
      1.21 ±112%      +2.6        3.85 ± 75%      +3.2        4.38 ± 42%  perf-profile.calltrace.cycles-pp.unmap_vmas.exit_mmap.__mmput.exit_mm.do_exit
      1.01 ±141%      +2.8        3.85 ± 75%      +3.2        4.24 ± 40%  perf-profile.calltrace.cycles-pp.unmap_page_range.unmap_vmas.exit_mmap.__mmput.exit_mm
      2.20 ±101%      +3.5        5.73 ± 58%      +5.4        7.65 ± 37%  perf-profile.calltrace.cycles-pp.exit_mmap.__mmput.exit_mm.do_exit.do_group_exit
     10.62 ± 57%      -9.4        1.17 ±142%      -1.2        9.37 ± 31%  perf-profile.children.cycles-pp.write
      9.69 ± 66%      -9.4        0.32 ±223%      -0.9        8.78 ± 35%  perf-profile.children.cycles-pp.ksys_write
      9.69 ± 66%      -9.4        0.32 ±223%      -0.9        8.78 ± 35%  perf-profile.children.cycles-pp.vfs_write
     13.49 ± 32%      -9.1        4.41 ± 47%      -1.1       12.42 ± 13%  perf-profile.children.cycles-pp.__cmd_record
     13.49 ± 32%      -9.1        4.41 ± 47%      -1.1       12.42 ± 13%  perf-profile.children.cycles-pp.cmd_record
      9.09 ± 54%      -8.8        0.32 ±223%      -0.1        9.03 ± 23%  perf-profile.children.cycles-pp.writen
      9.09 ± 54%      -8.7        0.34 ±223%      -0.3        8.83 ± 23%  perf-profile.children.cycles-pp.record__pushfn
     13.49 ± 32%      -8.7        4.75 ± 38%      -1.1       12.42 ± 13%  perf-profile.children.cycles-pp.handle_internal_command
     13.49 ± 32%      -8.7        4.75 ± 38%      -1.1       12.42 ± 13%  perf-profile.children.cycles-pp.main
     13.49 ± 32%      -8.7        4.75 ± 38%      -1.1       12.42 ± 13%  perf-profile.children.cycles-pp.run_builtin
      8.56 ± 57%      -8.6        0.00            -0.3        8.24 ± 25%  perf-profile.children.cycles-pp.generic_perform_write
      8.56 ± 57%      -8.6        0.00            -0.3        8.24 ± 25%  perf-profile.children.cycles-pp.shmem_file_write_iter
      9.94 ± 51%      -8.2        1.70 ± 79%      -0.3        9.60 ± 23%  perf-profile.children.cycles-pp.perf_mmap__push
      9.94 ± 51%      -8.2        1.70 ± 79%      -0.3        9.60 ± 23%  perf-profile.children.cycles-pp.record__mmap_read_evlist
      0.85 ± 82%      +0.5        1.38 ± 79%      +1.5        2.33 ± 28%  perf-profile.children.cycles-pp.__tlb_batch_free_encoded_pages
      0.85 ± 82%      +0.5        1.38 ± 79%      +1.5        2.33 ± 28%  perf-profile.children.cycles-pp.free_pages_and_swap_cache
      0.56 ±100%      +0.8        1.38 ± 79%      +1.1        1.66 ± 21%  perf-profile.children.cycles-pp.tlb_finish_mmu
      0.96 ±115%      +2.0        2.98 ± 76%      +2.4        3.41 ± 30%  perf-profile.children.cycles-pp.zap_present_ptes
      1.38 ±116%      +2.0        3.41 ± 70%      +3.0        4.34 ± 37%  perf-profile.children.cycles-pp.zap_pmd_range
      1.38 ±116%      +2.0        3.41 ± 70%      +3.0        4.34 ± 37%  perf-profile.children.cycles-pp.zap_pte_range
      1.58 ± 94%      +2.3        3.85 ± 75%      +2.9        4.49 ± 40%  perf-profile.children.cycles-pp.unmap_vmas
      1.38 ±116%      +2.5        3.85 ± 75%      +3.0        4.34 ± 37%  perf-profile.children.cycles-pp.unmap_page_range
      2.20 ±101%      +3.5        5.73 ± 58%      +5.6        7.77 ± 36%  perf-profile.children.cycles-pp.exit_mm
      2.40 ± 86%      +3.5        5.95 ± 55%      +5.4        7.86 ± 39%  perf-profile.children.cycles-pp.exit_mmap
      2.40 ± 86%      +3.5        5.95 ± 55%      +5.6        7.98 ± 38%  perf-profile.children.cycles-pp.__mmput
      8736 ±  9%     -37.6%       5455 ±  7%      +8.3%       9460 ± 23%  sched_debug.cfs_rq:/.avg_vruntime.min
      0.00 ± 56%    +407.7%       0.02 ± 58%      +1.5%       0.00 ± 56%  sched_debug.cfs_rq:/.h_nr_delayed.avg
      0.31 ± 48%    +227.3%       1.00            +9.1%       0.33 ± 54%  sched_debug.cfs_rq:/.h_nr_delayed.max
      0.03 ± 49%    +295.2%       0.13 ± 27%      +5.7%       0.04 ± 53%  sched_debug.cfs_rq:/.h_nr_delayed.stddev
      0.09 ± 11%     +84.1%       0.17 ±  9%      +5.5%       0.10 ± 17%  sched_debug.cfs_rq:/.h_nr_running.avg
      0.28 ±  6%     +43.8%       0.40 ± 11%      +0.9%       0.28 ±  9%  sched_debug.cfs_rq:/.h_nr_running.stddev
     48.57 ± 18%     +84.9%      89.82 ± 18%     +75.3%      85.12 ±102%  sched_debug.cfs_rq:/.load_avg.avg
    947.69 ± 14%     +46.5%       1388 ± 33%    +297.9%       3771 ±221%  sched_debug.cfs_rq:/.load_avg.max
    166.24 ± 14%     +62.3%     269.79 ± 16%    +175.8%     458.43 ±180%  sched_debug.cfs_rq:/.load_avg.stddev
      8736 ±  9%     -37.6%       5452 ±  7%      +8.3%       9460 ± 23%  sched_debug.cfs_rq:/.min_vruntime.min
      0.09 ± 11%     +88.9%       0.18 ±  9%      +5.7%       0.10 ± 18%  sched_debug.cfs_rq:/.nr_running.avg
      1.36 ±  4%     +46.9%       2.00            -2.0%       1.33 ± 13%  sched_debug.cfs_rq:/.nr_running.max
      0.29 ±  6%     +50.1%       0.43 ±  9%      +1.0%       0.29 ± 10%  sched_debug.cfs_rq:/.nr_running.stddev
     17.33 ± 38%    +147.9%      42.98 ± 29%     +44.5%      25.05 ± 43%  sched_debug.cfs_rq:/.removed.load_avg.avg
    483.56 ± 31%    +111.8%       1024           +37.6%     665.60 ± 40%  sched_debug.cfs_rq:/.removed.load_avg.max
     85.81 ± 31%    +132.6%     199.60 ± 14%     +38.4%     118.74 ± 36%  sched_debug.cfs_rq:/.removed.load_avg.stddev
      6.41 ± 42%    +176.3%      17.71 ± 38%     +50.5%       9.65 ± 51%  sched_debug.cfs_rq:/.removed.runnable_avg.avg
    239.17 ± 37%    +119.8%     525.67 ±  2%     +37.4%     328.52 ± 42%  sched_debug.cfs_rq:/.removed.runnable_avg.max
     34.33 ± 35%    +152.5%      86.68 ± 22%     +45.2%      49.85 ± 44%  sched_debug.cfs_rq:/.removed.runnable_avg.stddev
      6.41 ± 42%    +176.4%      17.71 ± 38%     +50.4%       9.64 ± 51%  sched_debug.cfs_rq:/.removed.util_avg.avg
    238.94 ± 37%    +120.0%     525.67 ±  2%     +37.5%     328.52 ± 42%  sched_debug.cfs_rq:/.removed.util_avg.max
     34.31 ± 35%    +152.7%      86.68 ± 22%     +45.2%      49.82 ± 44%  sched_debug.cfs_rq:/.removed.util_avg.stddev
    126.39 ± 12%    +132.4%     293.67 ±  3%     +12.1%     141.62 ± 16%  sched_debug.cfs_rq:/.util_avg.avg
    992.22 ±  8%     +30.8%       1298 ± 16%      +9.1%       1082 ± 23%  sched_debug.cfs_rq:/.util_avg.max
    188.05 ±  8%     +70.1%     319.91 ±  7%      +8.7%     204.36 ± 12%  sched_debug.cfs_rq:/.util_avg.stddev
     18.55 ± 20%     +78.8%      33.17 ± 14%      +7.6%      19.97 ± 21%  sched_debug.cfs_rq:/.util_est.avg
    614.94 ± 29%     +62.9%       1002 ±  3%     +10.9%     682.00 ± 24%  sched_debug.cfs_rq:/.util_est.max
     83.77 ± 24%     +64.2%     137.53 ±  4%      +9.7%      91.87 ± 22%  sched_debug.cfs_rq:/.util_est.stddev
    196044 ± 15%     -98.6%       2808 ± 15%     -17.2%     162396 ± 48%  sched_debug.cpu.avg_idle.min
    163689 ±  3%     +38.6%     226871 ±  4%      -0.3%     163181 ±  8%  sched_debug.cpu.avg_idle.stddev
    421612 ±  2%     -12.7%     367991            -1.1%     416790 ±  4%  sched_debug.cpu.clock.avg
    421617 ±  2%     -12.7%     367996            -1.1%     416795 ±  4%  sched_debug.cpu.clock.max
    421588 ±  2%     -12.7%     367986            -1.1%     416772 ±  4%  sched_debug.cpu.clock.min
      4.22 ± 27%     -38.7%       2.58 ±  4%     -15.7%       3.55 ± 37%  sched_debug.cpu.clock.stddev
    421249 ±  2%     -12.7%     367704            -1.1%     416439 ±  4%  sched_debug.cpu.clock_task.avg
    421583 ±  2%     -12.7%     367963            -1.1%     416760 ±  4%  sched_debug.cpu.clock_task.max
    411856 ±  2%     -12.8%     359277            -1.2%     406861 ±  4%  sched_debug.cpu.clock_task.min
      1029 ±  6%     -11.9%     907.72 ±  3%      +1.1%       1041 ±  8%  sched_debug.cpu.clock_task.stddev
    252.14 ± 13%     +61.7%     407.60 ±  6%      +6.5%     268.49 ±  9%  sched_debug.cpu.curr->pid.avg
      5177 ±  6%     -42.7%       2967            -4.5%       4944 ±  9%  sched_debug.cpu.curr->pid.max
      0.00 ±  9%     +38.2%       0.00 ± 17%      -3.3%       0.00 ± 18%  sched_debug.cpu.next_balance.stddev
      0.09 ± 12%     +96.6%       0.17 ±  9%      +8.0%       0.09 ± 16%  sched_debug.cpu.nr_running.avg
      0.26 ±  6%     +50.6%       0.40 ± 11%      +2.1%       0.27 ± 10%  sched_debug.cpu.nr_running.stddev
     35646 ±  7%     -28.2%      25593 ±  2%      -3.2%      34501 ± 12%  sched_debug.cpu.nr_switches.avg
    711.47 ± 10%     -44.3%     396.17 ± 11%      -7.2%     659.97 ± 19%  sched_debug.cpu.nr_switches.min
     52034 ±  7%     -16.0%      43686 ±  6%      -0.6%      51709 ± 16%  sched_debug.cpu.nr_switches.stddev
      0.01 ± 21%    +100.0%       0.01 ± 31%      -5.7%       0.01 ± 54%  sched_debug.cpu.nr_uninterruptible.avg
    421610 ±  2%     -12.7%     367987            -1.1%     416787 ±  4%  sched_debug.cpu_clk
    421050 ±  2%     -12.7%     367426            -1.1%     416227 ±  4%  sched_debug.ktime
    422192 ±  2%     -12.7%     368564            -1.1%     417374 ±  4%  sched_debug.sched_clk
      0.01 ± 71%    -100.0%       0.00           +27.1%       0.02 ± 91%  perf-sched.sch_delay.avg.ms.__cond_resched.filemap_read.xfs_file_buffered_read.xfs_file_read_iter.__io_read
      0.06 ± 10%    -100.0%       0.00            -4.4%       0.06 ± 22%  perf-sched.sch_delay.avg.ms.__cond_resched.shrink_folio_list.evict_folios.try_to_shrink_lruvec.shrink_one
      0.04 ± 49%    -100.0%       0.00            -1.1%       0.04 ± 50%  perf-sched.sch_delay.avg.ms.__cond_resched.try_to_shrink_lruvec.shrink_one.shrink_many.shrink_node
      0.03 ± 23%     -93.6%       0.00 ±223%      +5.9%       0.03 ± 18%  perf-sched.sch_delay.avg.ms.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown]
      0.06 ± 33%    -100.0%       0.00           -20.4%       0.05 ± 63%  perf-sched.sch_delay.avg.ms.devkmsg_read.vfs_read.ksys_read.do_syscall_64
      0.01 ± 20%    -100.0%       0.00            -8.2%       0.01 ± 19%  perf-sched.sch_delay.avg.ms.io_cqring_wait.__do_sys_io_uring_enter.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.02 ± 70%    -100.0%       0.00           -32.0%       0.01 ± 73%  perf-sched.sch_delay.avg.ms.irq_thread.kthread.ret_from_fork.ret_from_fork_asm
      0.04 ± 31%     -91.0%       0.00 ±170%     -12.6%       0.04 ± 55%  perf-sched.sch_delay.avg.ms.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt.[unknown].[unknown]
      0.03 ± 52%    -100.0%       0.00           +71.6%       0.06 ± 19%  perf-sched.sch_delay.avg.ms.rcu_gp_kthread.kthread.ret_from_fork.ret_from_fork_asm
      0.02 ± 20%     -74.2%       0.01 ± 42%     +18.2%       0.03 ± 36%  perf-sched.sch_delay.avg.ms.schedule_hrtimeout_range.do_poll.constprop.0.do_sys_poll
      0.02 ± 33%    -100.0%       0.00           +11.4%       0.02 ± 36%  perf-sched.sch_delay.avg.ms.schedule_hrtimeout_range.do_select.core_sys_select.do_pselect.constprop
      0.04 ± 51%    -100.0%       0.00           -47.4%       0.02 ±113%  perf-sched.sch_delay.avg.ms.schedule_timeout.__wait_for_common.__flush_work.__lru_add_drain_all
      0.01 ± 44%    -100.0%       0.00            +7.8%       0.01 ± 19%  perf-sched.sch_delay.avg.ms.schedule_timeout.__wait_for_common.wait_for_completion_state.kernel_clone
      0.01 ± 43%    -100.0%       0.00           -23.7%       0.01 ± 67%  perf-sched.sch_delay.avg.ms.schedule_timeout.io_schedule_timeout.__wait_for_common.blk_execute_rq
      0.05 ± 27%    -100.0%       0.00            +7.9%       0.05 ± 26%  perf-sched.sch_delay.avg.ms.schedule_timeout.kcompactd.kthread.ret_from_fork
      0.01 ± 35%    -100.0%       0.00           -45.3%       0.00 ±108%  perf-sched.sch_delay.avg.ms.schedule_timeout.khugepaged_wait_work.khugepaged.kthread
      0.04 ± 25%    -100.0%       0.00           -16.5%       0.04 ± 13%  perf-sched.sch_delay.avg.ms.schedule_timeout.kswapd_try_to_sleep.kswapd.kthread
      0.03 ± 18%     -69.6%       0.01 ± 92%     +36.7%       0.04 ± 46%  perf-sched.sch_delay.avg.ms.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown]
      0.02 ± 20%    -100.0%       0.00            +2.3%       0.02 ± 19%  perf-sched.sch_delay.avg.ms.wait_for_partner.fifo_open.do_dentry_open.vfs_open
      0.03 ± 75%    -100.0%       0.00            +7.1%       0.03 ± 90%  perf-sched.sch_delay.max.ms.__cond_resched.filemap_read.xfs_file_buffered_read.xfs_file_read_iter.__io_read
      0.08 ±  4%    -100.0%       0.00            +7.7%       0.08 ± 20%  perf-sched.sch_delay.max.ms.__cond_resched.shrink_folio_list.evict_folios.try_to_shrink_lruvec.shrink_one
      0.07 ± 21%    -100.0%       0.00           +10.9%       0.07 ± 18%  perf-sched.sch_delay.max.ms.__cond_resched.try_to_shrink_lruvec.shrink_one.shrink_many.shrink_node
      0.04 ±  6%     -95.2%       0.00 ±223%      +8.3%       0.04 ± 19%  perf-sched.sch_delay.max.ms.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown]
      0.06 ± 34%    -100.0%       0.00           -18.8%       0.05 ± 65%  perf-sched.sch_delay.max.ms.devkmsg_read.vfs_read.ksys_read.do_syscall_64
      3.27 ±147%    -100.0%       0.00           -93.3%       0.22 ± 62%  perf-sched.sch_delay.max.ms.io_cqring_wait.__do_sys_io_uring_enter.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.02 ± 89%    -100.0%       0.00           -41.1%       0.01 ± 76%  perf-sched.sch_delay.max.ms.irq_thread.kthread.ret_from_fork.ret_from_fork_asm
      0.07 ± 18%     -89.3%       0.01 ±192%     -12.4%       0.06 ± 53%  perf-sched.sch_delay.max.ms.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt.[unknown].[unknown]
      0.08 ± 15%    -100.0%       0.00           +22.4%       0.10 ± 11%  perf-sched.sch_delay.max.ms.rcu_gp_kthread.kthread.ret_from_fork.ret_from_fork_asm
      0.10 ± 23%     -92.1%       0.01 ± 26%    +115.2%       0.22 ±161%  perf-sched.sch_delay.max.ms.schedule_hrtimeout_range.do_poll.constprop.0.do_sys_poll
      0.04 ± 38%    -100.0%       0.00            -4.4%       0.03 ± 38%  perf-sched.sch_delay.max.ms.schedule_hrtimeout_range.do_select.core_sys_select.do_pselect.constprop
      0.07 ±  4%    -100.0%       0.00           -42.8%       0.04 ± 84%  perf-sched.sch_delay.max.ms.schedule_timeout.__wait_for_common.__flush_work.__lru_add_drain_all
      0.09 ± 30%    -100.0%       0.00            +0.5%       0.09 ± 17%  perf-sched.sch_delay.max.ms.schedule_timeout.__wait_for_common.wait_for_completion_state.kernel_clone
      0.02 ± 54%    -100.0%       0.00            +6.4%       0.02 ±105%  perf-sched.sch_delay.max.ms.schedule_timeout.io_schedule_timeout.__wait_for_common.blk_execute_rq
      0.09 ± 12%    -100.0%       0.00            +0.7%       0.09 ± 15%  perf-sched.sch_delay.max.ms.schedule_timeout.kcompactd.kthread.ret_from_fork
      0.01 ± 35%    -100.0%       0.00           -45.3%       0.00 ±108%  perf-sched.sch_delay.max.ms.schedule_timeout.khugepaged_wait_work.khugepaged.kthread
      0.12 ± 26%    -100.0%       0.00            +7.4%       0.13 ± 19%  perf-sched.sch_delay.max.ms.schedule_timeout.kswapd_try_to_sleep.kswapd.kthread
      0.07 ± 27%     -62.3%       0.03 ± 94%      +7.3%       0.07 ± 37%  perf-sched.sch_delay.max.ms.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown]
      0.07 ± 15%    -100.0%       0.00            -0.1%       0.07 ± 16%  perf-sched.sch_delay.max.ms.wait_for_partner.fifo_open.do_dentry_open.vfs_open
      0.01 ± 17%    +593.2%       0.09 ± 74%      -5.1%       0.01 ± 15%  perf-sched.total_sch_delay.average.ms
     15.98 ±  2%     -85.5%       2.32 ± 48%      +0.3%      16.03 ±  2%  perf-sched.total_wait_and_delay.average.ms
     52651           -98.0%       1045 ± 30%      -0.1%      52620        perf-sched.total_wait_and_delay.count.ms
      4255 ± 13%     -98.3%      73.23 ± 15%      -9.1%       3867 ± 15%  perf-sched.total_wait_and_delay.max.ms
     15.97 ±  2%     -86.0%       2.24 ± 53%      +0.3%      16.02 ±  2%  perf-sched.total_wait_time.average.ms
      4255 ± 13%     -98.9%      45.89 ±  7%      -9.1%       3866 ± 15%  perf-sched.total_wait_time.max.ms
      7.81 ±  7%     -97.5%       0.19 ± 62%      -3.0%       7.58 ±  7%  perf-sched.wait_and_delay.avg.ms.__cond_resched.__wait_for_common.affine_move_task.__set_cpus_allowed_ptr.__sched_setaffinity
     69.50 ± 29%    -100.0%       0.00            -2.9%      67.48 ± 26%  perf-sched.wait_and_delay.avg.ms.__cond_resched.shrink_folio_list.evict_folios.try_to_shrink_lruvec.shrink_one
    689.13 ± 14%    -100.0%       0.00           +10.2%     759.73 ± 12%  perf-sched.wait_and_delay.avg.ms.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown]
     17.56 ±  4%    -100.0%       0.00            +0.6%      17.67 ±  3%  perf-sched.wait_and_delay.avg.ms.do_nanosleep.hrtimer_nanosleep.common_nsleep.__x64_sys_clock_nanosleep
     32.71           -99.4%       0.21 ± 50%      -1.6%      32.18        perf-sched.wait_and_delay.avg.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.x64_sys_call
      0.78          -100.0%       0.00            +0.5%       0.79        perf-sched.wait_and_delay.avg.ms.io_cqring_wait.__do_sys_io_uring_enter.do_syscall_64.entry_SYSCALL_64_after_hwframe
     29.64 ±  3%     -97.5%       0.73 ±  7%      -1.8%      29.10 ±  4%  perf-sched.wait_and_delay.avg.ms.pipe_read.vfs_read.ksys_read.do_syscall_64
    403.38 ±  7%     -99.4%       2.51 ±223%      +2.1%     411.67 ±  4%  perf-sched.wait_and_delay.avg.ms.schedule_hrtimeout_range.do_poll.constprop.0.do_sys_poll
    158.27          -100.0%       0.00            -2.8%     153.90 ±  3%  perf-sched.wait_and_delay.avg.ms.schedule_hrtimeout_range.do_select.core_sys_select.do_pselect.constprop
    112.37 ± 23%     -96.3%       4.12 ±103%      +6.5%     119.69 ± 20%  perf-sched.wait_and_delay.avg.ms.schedule_hrtimeout_range.ep_poll.do_epoll_wait.__x64_sys_epoll_wait
    285.75 ± 73%    -100.0%       0.00           -47.6%     149.67 ± 84%  perf-sched.wait_and_delay.avg.ms.schedule_timeout.io_schedule_timeout.__wait_for_common.blk_execute_rq
    453.18          -100.0%       0.00            -0.0%     453.09        perf-sched.wait_and_delay.avg.ms.schedule_timeout.kcompactd.kthread.ret_from_fork
     47.62 ± 62%    -100.0%       0.00           -20.9%      37.65 ± 63%  perf-sched.wait_and_delay.avg.ms.schedule_timeout.kswapd_try_to_sleep.kswapd.kthread
    500.57 ±  7%     -98.9%       5.42 ± 59%      +4.0%     520.58 ±  2%  perf-sched.wait_and_delay.avg.ms.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm
    561.24 ±  5%     -99.5%       3.01 ± 50%      +0.9%     566.29 ±  4%  perf-sched.wait_and_delay.avg.ms.worker_thread.kthread.ret_from_fork.ret_from_fork_asm
    577.00           -40.4%     343.83 ± 23%      -0.1%     576.70        perf-sched.wait_and_delay.count.__cond_resched.__wait_for_common.affine_move_task.__set_cpus_allowed_ptr.__sched_setaffinity
      6.00 ± 31%    -100.0%       0.00           -15.0%       5.10 ± 51%  perf-sched.wait_and_delay.count.__cond_resched.shrink_folio_list.evict_folios.try_to_shrink_lruvec.shrink_one
     10.17 ±  3%    -100.0%       0.00            -1.6%      10.00        perf-sched.wait_and_delay.count.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown]
    501.83          -100.0%       0.00            +0.2%     502.70        perf-sched.wait_and_delay.count.do_nanosleep.hrtimer_nanosleep.common_nsleep.__x64_sys_clock_nanosleep
     45905          -100.0%       0.00            -0.1%      45851        perf-sched.wait_and_delay.count.io_cqring_wait.__do_sys_io_uring_enter.do_syscall_64.entry_SYSCALL_64_after_hwframe
      2426 ±  2%     -97.9%      51.67 ± 61%      +1.0%       2450 ±  3%  perf-sched.wait_and_delay.count.pipe_read.vfs_read.ksys_read.do_syscall_64
     37.83 ±  5%     -98.7%       0.50 ±223%      +1.8%      38.50 ±  4%  perf-sched.wait_and_delay.count.schedule_hrtimeout_range.do_poll.constprop.0.do_sys_poll
     30.00          -100.0%       0.00            +0.3%      30.10        perf-sched.wait_and_delay.count.schedule_hrtimeout_range.do_select.core_sys_select.do_pselect.constprop
      2.33 ± 20%    -100.0%       0.00           +15.7%       2.70 ± 16%  perf-sched.wait_and_delay.count.schedule_timeout.io_schedule_timeout.__wait_for_common.blk_execute_rq
     19.83          -100.0%       0.00            -0.2%      19.80 ±  2%  perf-sched.wait_and_delay.count.schedule_timeout.kcompactd.kthread.ret_from_fork
    134.67 ± 21%    -100.0%       0.00            +8.6%     146.20 ± 18%  perf-sched.wait_and_delay.count.schedule_timeout.kswapd_try_to_sleep.kswapd.kthread
      1195 ±  3%     -99.1%      10.17 ± 54%      +1.4%       1212 ±  3%  perf-sched.wait_and_delay.count.schedule_timeout.rcu_gp_fqs_loop.rcu_gp_kthread.kthread
    925.67 ±  7%     -58.9%     380.00 ± 24%      -3.7%     891.10 ±  2%  perf-sched.wait_and_delay.count.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm
    362.33 ±  5%     -79.0%      76.17 ± 58%      -1.1%     358.50 ±  6%  perf-sched.wait_and_delay.count.worker_thread.kthread.ret_from_fork.ret_from_fork_asm
      1013 ±  2%     -94.8%      53.21 ± 65%      -0.1%       1012 ±  3%  perf-sched.wait_and_delay.max.ms.__cond_resched.__wait_for_common.affine_move_task.__set_cpus_allowed_ptr.__sched_setaffinity
     89.90 ± 17%    -100.0%       0.00            -0.2%      89.68 ± 26%  perf-sched.wait_and_delay.max.ms.__cond_resched.shrink_folio_list.evict_folios.try_to_shrink_lruvec.shrink_one
    999.82          -100.0%       0.00            +0.0%     999.82        perf-sched.wait_and_delay.max.ms.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown]
      1000          -100.0%       0.00            -0.0%       1000        perf-sched.wait_and_delay.max.ms.do_nanosleep.hrtimer_nanosleep.common_nsleep.__x64_sys_clock_nanosleep
    890.88 ±  8%     -99.4%       5.21 ±101%      -0.7%     884.99 ±  6%  perf-sched.wait_and_delay.max.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.x64_sys_call
     13.58 ±106%    -100.0%       0.00           +14.8%      15.59 ± 77%  perf-sched.wait_and_delay.max.ms.io_cqring_wait.__do_sys_io_uring_enter.do_syscall_64.entry_SYSCALL_64_after_hwframe
      1013           -99.8%       1.97 ± 28%      -0.0%       1013        perf-sched.wait_and_delay.max.ms.pipe_read.vfs_read.ksys_read.do_syscall_64
      1464 ± 16%     -99.5%       7.52 ±223%      +0.4%       1471 ± 17%  perf-sched.wait_and_delay.max.ms.schedule_hrtimeout_range.do_poll.constprop.0.do_sys_poll
    250.08          -100.0%       0.00            +0.0%     250.09        perf-sched.wait_and_delay.max.ms.schedule_hrtimeout_range.do_select.core_sys_select.do_pselect.constprop
    334.30 ± 23%     -95.4%      15.29 ±103%      +6.6%     356.31 ± 20%  perf-sched.wait_and_delay.max.ms.schedule_hrtimeout_range.ep_poll.do_epoll_wait.__x64_sys_epoll_wait
    355.60 ± 68%    -100.0%       0.00           -43.2%     201.89 ± 66%  perf-sched.wait_and_delay.max.ms.schedule_timeout.io_schedule_timeout.__wait_for_common.blk_execute_rq
    504.88          -100.0%       0.00            -0.0%     504.71        perf-sched.wait_and_delay.max.ms.schedule_timeout.kcompactd.kthread.ret_from_fork
    111.51 ±  7%    -100.0%       0.00            -6.7%     104.09 ±  8%  perf-sched.wait_and_delay.max.ms.schedule_timeout.kswapd_try_to_sleep.kswapd.kthread
      6.34 ± 18%     -28.2%       4.55 ± 11%      +7.5%       6.81 ± 21%  perf-sched.wait_and_delay.max.ms.schedule_timeout.rcu_gp_fqs_loop.rcu_gp_kthread.kthread
      3150 ± 21%     -98.5%      45.76 ±  7%     -16.1%       2642 ± 17%  perf-sched.wait_and_delay.max.ms.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm
      4235 ± 14%     -99.3%      29.44 ± 43%      -8.7%       3867 ± 15%  perf-sched.wait_and_delay.max.ms.worker_thread.kthread.ret_from_fork.ret_from_fork_asm
      7.76 ±  7%     -98.8%       0.10 ± 61%      -3.0%       7.53 ±  7%  perf-sched.wait_time.avg.ms.__cond_resched.__wait_for_common.affine_move_task.__set_cpus_allowed_ptr.__sched_setaffinity
      0.96 ± 56%    -100.0%       0.00            +3.6%       0.99 ± 68%  perf-sched.wait_time.avg.ms.__cond_resched.filemap_read.xfs_file_buffered_read.xfs_file_read_iter.__io_read
     69.43 ± 29%    -100.0%       0.00            -2.9%      67.42 ± 26%  perf-sched.wait_time.avg.ms.__cond_resched.shrink_folio_list.evict_folios.try_to_shrink_lruvec.shrink_one
     45.25 ± 92%    -100.0%       0.00           -30.3%      31.53 ±119%  perf-sched.wait_time.avg.ms.__cond_resched.try_to_shrink_lruvec.shrink_one.shrink_many.shrink_node
    689.10 ± 14%    -100.0%       0.00           +10.2%     759.70 ± 12%  perf-sched.wait_time.avg.ms.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown]
      1.38 ±  9%    -100.0%       0.00            -0.3%       1.38 ± 10%  perf-sched.wait_time.avg.ms.devkmsg_read.vfs_read.ksys_read.do_syscall_64
     17.55 ±  4%    -100.0%       0.00            +0.7%      17.67 ±  3%  perf-sched.wait_time.avg.ms.do_nanosleep.hrtimer_nanosleep.common_nsleep.__x64_sys_clock_nanosleep
     32.71           -99.4%       0.21 ± 50%      -1.6%      32.18        perf-sched.wait_time.avg.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.x64_sys_call
      0.41 ±  4%     -14.0%       0.35 ±  3%      +2.2%       0.42 ±  4%  perf-sched.wait_time.avg.ms.do_wait.kernel_wait4.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.77          -100.0%       0.00            +0.7%       0.78        perf-sched.wait_time.avg.ms.io_cqring_wait.__do_sys_io_uring_enter.do_syscall_64.entry_SYSCALL_64_after_hwframe
    194.51 ±103%    -100.0%       0.00 ±223%     -13.3%     168.55 ±177%  perf-sched.wait_time.avg.ms.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt.[unknown].[unknown]
     29.63 ±  3%     -97.6%       0.71 ±  8%      -1.8%      29.10 ±  4%  perf-sched.wait_time.avg.ms.pipe_read.vfs_read.ksys_read.do_syscall_64
      4.01 ±  7%    -100.0%       0.00            -4.7%       3.83 ±  2%  perf-sched.wait_time.avg.ms.rcu_gp_kthread.kthread.ret_from_fork.ret_from_fork_asm
    403.36 ±  7%     -99.3%       2.82 ±193%      +2.1%     411.65 ±  4%  perf-sched.wait_time.avg.ms.schedule_hrtimeout_range.do_poll.constprop.0.do_sys_poll
    158.25          -100.0%       0.00            -2.8%     153.89 ±  3%  perf-sched.wait_time.avg.ms.schedule_hrtimeout_range.do_select.core_sys_select.do_pselect.constprop
    112.32 ± 23%     -96.5%       3.93 ±106%      +6.5%     119.64 ± 20%  perf-sched.wait_time.avg.ms.schedule_hrtimeout_range.ep_poll.do_epoll_wait.__x64_sys_epoll_wait
      0.66 ±  4%    -100.0%       0.00            +1.0%       0.67 ±  4%  perf-sched.wait_time.avg.ms.schedule_timeout.__wait_for_common.wait_for_completion_state.kernel_clone
    285.74 ± 73%    -100.0%       0.00           -47.6%     149.66 ± 84%  perf-sched.wait_time.avg.ms.schedule_timeout.io_schedule_timeout.__wait_for_common.blk_execute_rq
    453.14          -100.0%       0.00            -0.0%     453.04        perf-sched.wait_time.avg.ms.schedule_timeout.kcompactd.kthread.ret_from_fork
      0.07 ± 83%    -100.0%       0.00           -43.0%       0.04 ±160%  perf-sched.wait_time.avg.ms.schedule_timeout.khugepaged_wait_work.khugepaged.kthread
     47.57 ± 62%    -100.0%       0.00           -20.9%      37.61 ± 63%  perf-sched.wait_time.avg.ms.schedule_timeout.kswapd_try_to_sleep.kswapd.kthread
      3.86 ±  6%     -24.9%       2.90 ± 23%      -1.9%       3.79 ±  5%  perf-sched.wait_time.avg.ms.schedule_timeout.rcu_gp_fqs_loop.rcu_gp_kthread.kthread
    500.56 ±  7%     -98.9%       5.38 ± 60%      +4.0%     520.57 ±  2%  perf-sched.wait_time.avg.ms.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm
    226.30 ± 64%    -100.0%       0.01 ±107%     -18.7%     184.02 ±120%  perf-sched.wait_time.avg.ms.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown]
    561.18 ±  5%     -99.6%       2.50 ± 79%      +0.9%     566.24 ±  4%  perf-sched.wait_time.avg.ms.worker_thread.kthread.ret_from_fork.ret_from_fork_asm
      1013 ±  2%     -97.4%      26.61 ± 65%      -0.1%       1012 ±  3%  perf-sched.wait_time.max.ms.__cond_resched.__wait_for_common.affine_move_task.__set_cpus_allowed_ptr.__sched_setaffinity
      1.65 ± 66%    -100.0%       0.00            +7.2%       1.77 ± 48%  perf-sched.wait_time.max.ms.__cond_resched.filemap_read.xfs_file_buffered_read.xfs_file_read_iter.__io_read
     89.83 ± 17%    -100.0%       0.00            -0.2%      89.61 ± 26%  perf-sched.wait_time.max.ms.__cond_resched.shrink_folio_list.evict_folios.try_to_shrink_lruvec.shrink_one
     91.01 ± 14%    -100.0%       0.00           -19.8%      73.00 ± 37%  perf-sched.wait_time.max.ms.__cond_resched.try_to_shrink_lruvec.shrink_one.shrink_many.shrink_node
    999.79          -100.0%       0.00            +0.0%     999.79        perf-sched.wait_time.max.ms.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown]
      2.76 ±  9%    -100.0%       0.00            -0.3%       2.76 ± 10%  perf-sched.wait_time.max.ms.devkmsg_read.vfs_read.ksys_read.do_syscall_64
      1000          -100.0%       0.00            -0.0%       1000        perf-sched.wait_time.max.ms.do_nanosleep.hrtimer_nanosleep.common_nsleep.__x64_sys_clock_nanosleep
    890.88 ±  8%     -99.4%       5.21 ±101%      -0.7%     884.99 ±  6%  perf-sched.wait_time.max.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.x64_sys_call
     15.72 ±  2%     -93.4%       1.04 ± 28%      +3.2%      16.22 ±  4%  perf-sched.wait_time.max.ms.do_wait.kernel_wait4.do_syscall_64.entry_SYSCALL_64_after_hwframe
     13.23 ±110%    -100.0%       0.00           +17.7%      15.57 ± 77%  perf-sched.wait_time.max.ms.io_cqring_wait.__do_sys_io_uring_enter.do_syscall_64.entry_SYSCALL_64_after_hwframe
    500.20 ± 99%    -100.0%       0.01 ±223%     -48.4%     258.00 ±148%  perf-sched.wait_time.max.ms.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt.[unknown].[unknown]
      1013           -99.8%       1.96 ± 29%      -0.0%       1013        perf-sched.wait_time.max.ms.pipe_read.vfs_read.ksys_read.do_syscall_64
      4.85 ±  6%    -100.0%       0.00            -5.0%       4.61 ± 10%  perf-sched.wait_time.max.ms.rcu_gp_kthread.kthread.ret_from_fork.ret_from_fork_asm
      1464 ± 16%     -99.4%       8.15 ±202%      +0.4%       1471 ± 17%  perf-sched.wait_time.max.ms.schedule_hrtimeout_range.do_poll.constprop.0.do_sys_poll
    250.06          -100.0%       0.00            +0.0%     250.06        perf-sched.wait_time.max.ms.schedule_hrtimeout_range.do_select.core_sys_select.do_pselect.constprop
    334.27 ± 23%     -95.4%      15.28 ±103%      +6.6%     356.28 ± 20%  perf-sched.wait_time.max.ms.schedule_hrtimeout_range.ep_poll.do_epoll_wait.__x64_sys_epoll_wait
      2.04 ±  7%    -100.0%       0.00            -1.8%       2.01 ±  6%  perf-sched.wait_time.max.ms.schedule_timeout.__wait_for_common.wait_for_completion_state.kernel_clone
    355.59 ± 68%    -100.0%       0.00           -43.2%     201.88 ± 66%  perf-sched.wait_time.max.ms.schedule_timeout.io_schedule_timeout.__wait_for_common.blk_execute_rq
    504.84          -100.0%       0.00            -0.0%     504.68        perf-sched.wait_time.max.ms.schedule_timeout.kcompactd.kthread.ret_from_fork
      0.07 ± 83%    -100.0%       0.00           -43.0%       0.04 ±160%  perf-sched.wait_time.max.ms.schedule_timeout.khugepaged_wait_work.khugepaged.kthread
    111.44 ±  7%    -100.0%       0.00            -6.7%     104.03 ±  8%  perf-sched.wait_time.max.ms.schedule_timeout.kswapd_try_to_sleep.kswapd.kthread
      3150 ± 21%     -98.7%      39.96 ± 40%     -16.1%       2642 ± 17%  perf-sched.wait_time.max.ms.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm
    843.01 ± 41%    -100.0%       0.03 ± 94%     -50.4%     417.72 ±114%  perf-sched.wait_time.max.ms.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown]
      4235 ± 14%     -99.5%      21.72 ± 79%      -8.7%       3866 ± 15%  perf-sched.wait_time.max.ms.worker_thread.kthread.ret_from_fork.ret_from_fork_asm