Skip to content

nvmet: add buffered_io support for bdev-ns

Chaitanya Kulkarni linux-nvme 3 versions – Applied

Status from this message: applied to nvme-5.18.

Get series with b4
b4 am [email protected]

Versions newest first · linked by author and title

  1. v3 Latest2 patches 1 replyApplied
  2. v22 patches 2 repliesSuperseded
  3. v13 patches 7 repliesSuperseded

Changes in v3 from the cover letter

1. Add a comment in the nvmet_bdev_ns_enable() when returning early for buffered_io case. Feel free to edit the comment at the time of applying patch.

Patches in v3 Applied

#Subject and filesSizeRepliesApplied
1/2nvmet: use i_size_read() to set size for file-nsdrivers/nvme/target/io-cmd-file.c drivers/nvme/target/nvmet.h+5 -140 replies
2/2nvmet: allow bdev in buffered_io modedrivers/nvme/target/io-cmd-bdev.c+8 -00 replies

Reviews trailers per patch

No Reviewed-by, Acked-by or Tested-by trailers for v3 yet.

Read the v3 thread

Cover letter [PATCH V3 0/2] nvmet: add buffered_io support for bdev-ns

From: Chaitanya Kulkarni <[email protected]>

Hi,

In current implementation like NVMeOF target file backed namespace we don't support buffered_io for block device backed namesapce. This allows user to configure block device backend in buffered_io mode so user can get the benefits of caching.

Please note that using this combination host needs to periodically sync between cache and storage or set the system cache parameters accordingly. NVMeOF target is not responsible for data consistency between cache and storage.

fio data verification, rand read/write performance numbers :-

1. NVMeOF Target ns device_path=/dev/nvme0n1 & buffered_io=0 :-

fio Verify job :-
----------------------
read: IOPS=24.3k, BW=94.9MiB/s (99.5MB/s)(506MiB/5335msec)
write: IOPS=27.9k, BW=109MiB/s (114MB/s)(800MiB/7353msec);

fio Random Write :-
----------------------
write: IOPS=68.2k, BW=266MiB/s (279MB/s)(15.6GiB/60002msec);
write: IOPS=68.6k, BW=268MiB/s (281MB/s)(15.7GiB/60003msec);
write: IOPS=68.2k, BW=266MiB/s (279MB/s)(15.6GiB/60003msec);
write: IOPS=68.3k, BW=267MiB/s (280MB/s)(15.6GiB/60003msec);
write: IOPS=69.0k, BW=269MiB/s (283MB/s)(15.8GiB/60002msec);
write: IOPS=69.6k, BW=272MiB/s (285MB/s)(15.9GiB/60003msec);
write: IOPS=69.8k, BW=272MiB/s (286MB/s)(16.0GiB/60003msec);
write: IOPS=69.8k, BW=273MiB/s (286MB/s)(16.0GiB/60003msec);
write: IOPS=71.5k, BW=279MiB/s (293MB/s)(16.4GiB/60003msec);
write: IOPS=71.0k, BW=277MiB/s (291MB/s)(16.2GiB/60003msec);

fio Random Read :-
----------------------
read: IOPS=72.8k, BW=284MiB/s (298MB/s)(16.7GiB/60037msec)
read: IOPS=71.5k, BW=279MiB/s (293MB/s)(16.4GiB/60037msec)
read: IOPS=67.5k, BW=264MiB/s (277MB/s)(15.5GiB/60033msec)
read: IOPS=74.5k, BW=291MiB/s (305MB/s)(17.1GiB/60035msec)
read: IOPS=72.4k, BW=283MiB/s (296MB/s)(16.6GiB/60036msec)
read: IOPS=71.8k, BW=281MiB/s (294MB/s)(16.5GiB/60036msec)
read: IOPS=73.7k, BW=288MiB/s (302MB/s)(16.9GiB/60035msec)
read: IOPS=72.3k, BW=283MiB/s (296MB/s)(16.6GiB/60039msec)
read: IOPS=72.8k, BW=285MiB/s (298MB/s)(16.7GiB/60037msec)
read: IOPS=74.6k, BW=291MiB/s (306MB/s)(17.1GiB/60036msec)

2. NVMeOF Target ns device_path=/dev/nvme0n1 & buffered_io=1 :-

fio Verify job :-
----------------------
read: IOPS=195k, BW=763MiB/s (800MB/s)(506MiB/663msec)
write: IOPS=123k, BW=481MiB/s (505MB/s)(800MiB/1662msec);

fio Random Write :-
----------------------
write: IOPS=4248k, BW=16.2GiB/s (17.4GB/s)(972GiB/60002msec);
write: IOPS=4157k, BW=15.9GiB/s (17.0GB/s)(951GiB/60002msec);
write: IOPS=4203k, BW=16.0GiB/s (17.2GB/s)(962GiB/60002msec);
write: IOPS=4130k, BW=15.8GiB/s (16.9GB/s)(945GiB/60002msec);
write: IOPS=4170k, BW=15.9GiB/s (17.1GB/s)(955GiB/60002msec);
write: IOPS=4203k, BW=16.0GiB/s (17.2GB/s)(962GiB/60002msec);
write: IOPS=4166k, BW=15.9GiB/s (17.1GB/s)(954GiB/60002msec);
write: IOPS=4106k, BW=15.7GiB/s (16.8GB/s)(940GiB/60002msec);
write: IOPS=4181k, BW=15.9GiB/s (17.1GB/s)(957GiB/60001msec);
write: IOPS=4153k, BW=15.8GiB/s (17.0GB/s)(951GiB/60002msec);

fio Random Read :-
----------------------
read: IOPS=4942k, BW=18.9GiB/s (20.2GB/s)(1131GiB/60002msec)
read: IOPS=4940k, BW=18.8GiB/s (20.2GB/s)(1131GiB/60002msec)
read: IOPS=5041k, BW=19.2GiB/s (20.6GB/s)(1154GiB/60002msec)
read: IOPS=4989k, BW=19.0GiB/s (20.4GB/s)(1142GiB/60001msec)
read: IOPS=4982k, BW=19.0GiB/s (20.4GB/s)(1140GiB/60001msec)
read: IOPS=5031k, BW=19.2GiB/s (20.6GB/s)(1152GiB/60001msec)
read: IOPS=4948k, BW=18.9GiB/s (20.3GB/s)(1133GiB/60002msec)
read: IOPS=4720k, BW=18.0GiB/s (19.3GB/s)(1080GiB/60002msec)
read: IOPS=4830k, BW=18.4GiB/s (19.8GB/s)(1106GiB/60002msec)
read: IOPS=4915k, BW=18.7GiB/s (20.1GB/s)(1125GiB/60002msec)

-ck

changes from V2:
1. Add a comment in the nvmet_bdev_ns_enable() when returning early for buffered_io case. Feel free to edit the comment at the time of applying patch.

changes from v1:
1. Remove first two patches and don't check for stat mode for block device.
2. use i_size_read() to get the size of the file.

Chaitanya Kulkarni (2):
nvmet: use i_size_read() to set size for file-ns
nvmet: allow bdev in buffered_io mode

 drivers/nvme/target/io-cmd-bdev.c |  8 ++++++++
 drivers/nvme/target/io-cmd-file.c | 17 ++++-------------
 drivers/nvme/target/nvmet.h       |  2 +-
 3 files changed, 13 insertions(+), 14 deletions(-)

root@dev nvme (nvme-5.17) # ./bdev-buffered-io
bdev-buffered-io/ bdev-buffered-io-test.sh
root@dev nvme (nvme-5.17) # ./bdev-buffered-io-test.sh
+ NQN=testnqn
+ nvme disconnect -n testnqn
NQN:testnqn disconnected 1 controller(s)

real 0m0.350s
user 0m0.003s
sys 0m0.002s
+ for i in /sys/kernel/config/nvmet/subsystems/${NQN}/namespaces/*
+ echo 0
+ rmdir /sys/kernel/config/nvmet/subsystems/testnqn/namespaces/1
+ rm -fr /sys/kernel/config/nvmet/ports/1/subsystems/testnqn
+ rmdir /sys/kernel/config/nvmet/ports/1
+ rmdir /sys/kernel/config/nvmet/subsystems/testnqn
+ rmdir 'config/nullb/nullb*'
rmdir: failed to remove 'config/nullb/nullb*': No such file or directory
+ umount /mnt/nvme0n1
umount: /mnt/nvme0n1: no mount point specified.
+ umount /mnt/backend
umount: /mnt/backend: not mounted.
+ modprobe -r nvme_loop
+ modprobe -r nvme_fabrics
+ modprobe -r nvmet
+ modprobe -r nvme
+ modprobe -r null_blk
+ tree /sys/kernel/config
/sys/kernel/config

0 directories, 0 files
+ umount /mnt/nvme0n1
+ clear_dmesg
./compile_nvme.sh: line 3: clear_dmesg: command not found
umount: /mnt/nvme0n1: no mount point specified.
+ ./delete.sh 1
+ NQN=testnqn
+ nvme disconnect -n testnqn
Failed to disconnect by NQN: testnqn

real 0m0.001s
user 0m0.000s
sys 0m0.001s
+ for i in /sys/kernel/config/nvmet/subsystems/${NQN}/namespaces/*
+ echo 0
./delete.sh: line 9: /sys/kernel/config/nvmet/subsystems/testnqn/namespaces/*/enable: No such file or directory
+ rmdir '/sys/kernel/config/nvmet/subsystems/testnqn/namespaces/*'
rmdir: failed to remove '/sys/kernel/config/nvmet/subsystems/testnqn/namespaces/*': No such file or directory
+ rm -fr /sys/kernel/config/nvmet/ports/1/subsystems/testnqn
+ rmdir /sys/kernel/config/nvmet/ports/1
rmdir: failed to remove '/sys/kernel/config/nvmet/ports/1': No such file or directory
+ rmdir /sys/kernel/config/nvmet/subsystems/testnqn
rmdir: failed to remove '/sys/kernel/config/nvmet/subsystems/testnqn': No such file or directory
+ rmdir 'config/nullb/nullb*'
rmdir: failed to remove 'config/nullb/nullb*': No such file or directory
+ umount /mnt/nvme0n1
umount: /mnt/nvme0n1: no mount point specified.
+ umount /mnt/backend
umount: /mnt/backend: not mounted.
+ modprobe -r nvme_loop
+ modprobe -r nvme_fabrics
+ modprobe -r nvmet
+ modprobe -r nvme
+ modprobe -r null_blk
+ tree /sys/kernel/config
/sys/kernel/config

0 directories, 0 files
+ modprobe -r nvme-fabrics
+ modprobe -r nvme_loop
+ modprobe -r nvmet
+ modprobe -r nvme
+ sleep 1
+ modprobe -r nvme-core
+ lsmod
+ grep nvme
+ sleep 1
+ git diff
+ sleep 1
++ nproc
+ make -j 48 M=drivers/nvme/ modules
+ HOST=drivers/nvme/host
+ TARGET=drivers/nvme/target
++ uname -r
+ HOST_DEST=/lib/modules/5.17.0-rc1nvme+/kernel/drivers/nvme/host/
++ uname -r
+ TARGET_DEST=/lib/modules/5.17.0-rc1nvme+/kernel/drivers/nvme/target/
+ cp drivers/nvme/host/nvme-core.ko drivers/nvme/host/nvme-fabrics.ko drivers/nvme/host/nvme-fc.ko drivers/nvme/host/nvme.ko drivers/nvme/host/nvme-rdma.ko drivers/nvme/host/nvme-tcp.ko /lib/modules/5.17.0-rc1nvme+/kernel/drivers/nvme/host//
+ cp drivers/nvme/target/nvme-fcloop.ko drivers/nvme/target/nvme-loop.ko drivers/nvme/target/nvmet-fc.ko drivers/nvme/target/nvmet.ko drivers/nvme/target/nvmet-rdma.ko drivers/nvme/target/nvmet-tcp.ko /lib/modules/5.17.0-rc1nvme+/kernel/drivers/nvme/target//
+ ls -lrth /lib/modules/5.17.0-rc1nvme+/kernel/drivers/nvme/host/ /lib/modules/5.17.0-rc1nvme+/kernel/drivers/nvme/target//
/lib/modules/5.17.0-rc1nvme+/kernel/drivers/nvme/host/:
total 6.7M

-rw-r--r--. 1 root root  2.8M Feb  1 23:42 nvme-core.ko
-rw-r--r--. 1 root root  448K Feb  1 23:42 nvme-fabrics.ko
-rw-r--r--. 1 root root 1015K Feb  1 23:42 nvme-fc.ko
-rw-r--r--. 1 root root  745K Feb  1 23:42 nvme.ko
-rw-r--r--. 1 root root  958K Feb  1 23:42 nvme-rdma.ko
-rw-r--r--. 1 root root  902K Feb  1 23:42 nvme-tcp.ko

/lib/modules/5.17.0-rc1nvme+/kernel/drivers/nvme/target//:
total 6.7M

-rw-r--r--. 1 root root 498K Feb  1 23:42 nvme-fcloop.ko
-rw-r--r--. 1 root root 439K Feb  1 23:42 nvme-loop.ko
-rw-r--r--. 1 root root 768K Feb  1 23:42 nvmet-fc.ko
-rw-r--r--. 1 root root 3.3M Feb  1 23:42 nvmet.ko
-rw-r--r--. 1 root root 929K Feb  1 23:42 nvmet-rdma.ko
-rw-r--r--. 1 root root 775K Feb  1 23:42 nvmet-tcp.ko

+ modprobe nvme
+ NQN=testnqn
+ nvme disconnect -n testnqn
NQN:testnqn disconnected 0 controller(s)

real 0m0.001s
user 0m0.001s
sys 0m0.001s
+ for i in /sys/kernel/config/nvmet/subsystems/${NQN}/namespaces/*
+ echo 0
./delete.sh: line 9: /sys/kernel/config/nvmet/subsystems/testnqn/namespaces/*/enable: No such file or directory
+ rmdir '/sys/kernel/config/nvmet/subsystems/testnqn/namespaces/*'
rmdir: failed to remove '/sys/kernel/config/nvmet/subsystems/testnqn/namespaces/*': No such file or directory
+ rm -fr /sys/kernel/config/nvmet/ports/1/subsystems/testnqn
+ rmdir /sys/kernel/config/nvmet/ports/1
rmdir: failed to remove '/sys/kernel/config/nvmet/ports/1': No such file or directory
+ rmdir /sys/kernel/config/nvmet/subsystems/testnqn
rmdir: failed to remove '/sys/kernel/config/nvmet/subsystems/testnqn': No such file or directory
+ rmdir 'config/nullb/nullb*'
rmdir: failed to remove 'config/nullb/nullb*': No such file or directory
+ umount /mnt/nvme0n1
umount: /mnt/nvme0n1: no mount point specified.
+ umount /mnt/backend
umount: /mnt/backend: not mounted.
+ modprobe -r nvme_loop
+ modprobe -r nvme_fabrics
+ modprobe -r nvmet
+ modprobe -r nvme
+ modprobe -r null_blk
+ tree /sys/kernel/config
/sys/kernel/config

0 directories, 0 files
+++ NN=1
+++ NQN=testnqn
+++ let NR_DEVICES=NN+1
+++ modprobe -r null_blk
+++ modprobe -r nvme
+++ modprobe null_blk nr_devices=0
+++ modprobe nvme
+++ modprobe nvme-fabrics
+++ modprobe nvmet
+++ modprobe nvme-loop
+++ dmesg -c
+++ sleep 2
+++ tree /sys/kernel/config
/sys/kernel/config

├── nullb
│   └── features
└── nvmet
    ├── hosts
    ├── ports
    └── subsystems

5 directories, 1 file
+++ mkdir /sys/kernel/config/nvmet/subsystems/testnqn
+++ mkdir /sys/kernel/config/nvmet/subsystems/testnqn/namespaces/1
+++ echo -n /dev/nvme0n1
+++ cat /sys/kernel/config/nvmet/subsystems/testnqn/namespaces/1/device_path /dev/nvme0n1
+++ echo 1
+++ cat /sys/kernel/config/nvmet/subsystems/testnqn/namespaces/1/buffered_io
1
+++ echo 1
+++ mkdir /sys/kernel/config/nvmet/ports/1/
+++ echo -n loop
+++ echo -n 1
+++ ln -s /sys/kernel/config/nvmet/subsystems/testnqn /sys/kernel/config/nvmet/ports/1/subsystems/ +++ sleep 1
+++ echo transport=loop,nqn=testnqn
+++ sleep 1
+++ mount
+++ column -t
+++ grep nvme
+++ dmesg -c

[ 2348.622209] nvme nvme0: 48/0/0 default/read/poll queues
[ 2350.605991] nvmet: adding nsid 1 to subsystem testnqn
[ 2351.616737] nvmet: creating nvm controller 1 for subsystem testnqn for NQN nqn.2014-08.org.nvmexpress:uuid:bb4f2c46-9ac7-4283-902a-420883b62f3c.
[ 2351.616857] nvme nvme1: creating 48 I/O queues.
[ 2351.621168] nvme nvme1: new ctrl: "testnqn"

write-and-verify: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=4 fio-3.27
Starting 1 process
Jobs: 1 (f=1)
write-and-verify: (groupid=0, jobs=1): err= 0: pid=20757: Tue Feb 1 23:42:57 2022
read: IOPS=195k, BW=763MiB/s (800MB/s)(506MiB/663msec)

    slat (usec): min=3, max=422, avg= 3.96, stdev= 1.90
    clat (nsec): min=1072, max=483948, avg=15610.29, stdev=5064.35
     lat (usec): min=4, max=494, avg=19.64, stdev= 6.20
    clat percentiles (nsec):
     |  1.00th=[13248],  5.00th=[13376], 10.00th=[13504], 20.00th=[13632],
     | 30.00th=[13760], 40.00th=[13888], 50.00th=[14016], 60.00th=[14144],
     | 70.00th=[14272], 80.00th=[14528], 90.00th=[23424], 95.00th=[26752],
     | 99.00th=[28544], 99.50th=[32384], 99.90th=[42752], 99.95th=[47360],
     | 99.99th=[82432]
  write: IOPS=123k, BW=481MiB/s (505MB/s)(800MiB/1662msec);
    slat (usec): min=4, max=221, avg= 7.36, stdev= 2.87
    clat (nsec): min=841, max=222422, avg=24669.34, stdev=8735.58
     lat (usec): min=6, max=263, avg=32.11, stdev=11.14
    clat percentiles (usec):
     |  1.00th=[   18],  5.00th=[   19], 10.00th=[   19], 20.00th=[   19],
     | 30.00th=[   20], 40.00th=[   20], 50.00th=[   20], 60.00th=[   21],
     | 70.00th=[   28], 80.00th=[   36], 90.00th=[   38], 95.00th=[   40],
     | 99.00th=[   51], 99.50th=[   58], 99.90th=[   70], 99.95th=[   81],
     | 99.99th=[  155]
   bw (  KiB/s): min=150160, max=525424, per=83.10%, avg=409600.00, stdev=174551.65, samples=4
   iops        : min=37540, max=131356, avg=102400.00, stdev=43637.91, samples=4
  lat (nsec)   : 1000=0.01%
  lat (usec)   : 2=0.01%, 10=0.01%, 20=64.99%, 50=34.34%, 100=0.65%
  lat (usec)   : 250=0.02%, 500=0.01%
  cpu          : usr=27.97%, sys=32.66%, ctx=333157, majf=0, minf=3053
  IO depths    : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=129572,204800,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4

Run status group 0 (all jobs):
READ: bw=763MiB/s (800MB/s), 763MiB/s-763MiB/s (800MB/s-800MB/s), io=506MiB (531MB), run=663-663msec
WRITE: bw=481MiB/s (505MB/s), 481MiB/s-481MiB/s (505MB/s-505MB/s), io=800MiB (839MB), run=1662-1662msec

Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
RANDREAD: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=8
...
fio-3.27
Starting 32 processes
Jobs: 32 (f=32): [w(32)][100.0%][w=16.2GiB/s][w=4236k IOPS][eta 00m:00s]
RANDREAD: (groupid=0, jobs=32): err= 0: pid=20761: Tue Feb 1 23:43:57 2022

  write: IOPS=4248k, BW=16.2GiB/s (17.4GB/s)(972GiB/60002msec);
    slat (nsec): min=1112, max=24135k, avg=6779.81, stdev=5320.08
    clat (nsec): min=721, max=24222k, avg=53033.78, stdev=21465.11
     lat (usec): min=6, max=24230, avg=59.89, stdev=23.52
    clat percentiles (usec):
     |  1.00th=[   41],  5.00th=[   42], 10.00th=[   42], 20.00th=[   43],
     | 30.00th=[   43], 40.00th=[   44], 50.00th=[   46], 60.00th=[   57],
     | 70.00th=[   58], 80.00th=[   60], 90.00th=[   63], 95.00th=[   88],
     | 99.00th=[  126], 99.50th=[  133], 99.90th=[  186], 99.95th=[  223],
     | 99.99th=[  277]
   bw (  MiB/s): min=11452, max=20170, per=100.00%, avg=16608.41, stdev=84.92, samples=3808
   iops        : min=2931936, max=5163618, avg=4251753.76, stdev=21739.07, samples=3808
  lat (nsec)   : 750=0.01%, 1000=0.01%
  lat (usec)   : 2=0.01%, 10=0.01%, 20=0.01%, 50=54.82%, 100=42.95%
  lat (usec)   : 250=2.21%, 500=0.02%, 750=0.01%, 1000=0.01%
  lat (msec)   : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
  cpu          : usr=11.32%, sys=36.88%, ctx=254865246, majf=0, minf=439
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,254859948,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
WRITE: bw=16.2GiB/s (17.4GB/s), 16.2GiB/s-16.2GiB/s (17.4GB/s-17.4GB/s), io=972GiB (1044GB), run=60002-60002msec

Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
RANDREAD: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=8
...
fio-3.27
Starting 32 processes
Jobs: 32 (f=32): [w(32)][100.0%][w=15.3GiB/s][w=4015k IOPS][eta 00m:00s]
RANDREAD: (groupid=0, jobs=32): err= 0: pid=20878: Tue Feb 1 23:44:57 2022

  write: IOPS=4157k, BW=15.9GiB/s (17.0GB/s)(951GiB/60002msec);
    slat (nsec): min=1773, max=23957k, avg=6930.59, stdev=5690.00
    clat (nsec): min=672, max=24058k, avg=54190.75, stdev=22713.63
     lat (usec): min=6, max=24086, avg=61.20, stdev=24.89
    clat percentiles (usec):
     |  1.00th=[   41],  5.00th=[   42], 10.00th=[   42], 20.00th=[   43],
     | 30.00th=[   44], 40.00th=[   45], 50.00th=[   47], 60.00th=[   57],
     | 70.00th=[   59], 80.00th=[   61], 90.00th=[   67], 95.00th=[   94],
     | 99.00th=[  129], 99.50th=[  137], 99.90th=[  192], 99.95th=[  225],
     | 99.99th=[  269]
   bw (  MiB/s): min=11359, max=20085, per=100.00%, avg=16257.07, stdev=84.39, samples=3808
   iops        : min=2907927, max=5141948, avg=4161809.03, stdev=21605.08, samples=3808
  lat (nsec)   : 750=0.01%, 1000=0.01%
  lat (usec)   : 2=0.01%, 10=0.01%, 20=0.01%, 50=54.02%, 100=42.99%
  lat (usec)   : 250=2.97%, 500=0.02%, 750=0.01%, 1000=0.01%
  lat (msec)   : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
  cpu          : usr=11.06%, sys=36.43%, ctx=249428122, majf=0, minf=464
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,249423456,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
WRITE: bw=15.9GiB/s (17.0GB/s), 15.9GiB/s-15.9GiB/s (17.0GB/s-17.0GB/s), io=951GiB (1022GB), run=60002-60002msec

Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
RANDREAD: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=8
...
fio-3.27
Starting 32 processes
Jobs: 32 (f=32): [w(32)][100.0%][w=16.3GiB/s][w=4283k IOPS][eta 00m:00s]
RANDREAD: (groupid=0, jobs=32): err= 0: pid=20954: Tue Feb 1 23:45:57 2022

  write: IOPS=4203k, BW=16.0GiB/s (17.2GB/s)(962GiB/60002msec);
    slat (nsec): min=1783, max=23922k, avg=6853.24, stdev=4821.07
    clat (nsec): min=711, max=24147k, avg=53603.75, stdev=19747.55
     lat (usec): min=5, max=24155, avg=60.53, stdev=21.64
    clat percentiles (usec):
     |  1.00th=[   41],  5.00th=[   42], 10.00th=[   42], 20.00th=[   43],
     | 30.00th=[   44], 40.00th=[   45], 50.00th=[   48], 60.00th=[   57],
     | 70.00th=[   59], 80.00th=[   60], 90.00th=[   65], 95.00th=[   85],
     | 99.00th=[  118], 99.50th=[  133], 99.90th=[  167], 99.95th=[  190],
     | 99.99th=[  251]
   bw (  MiB/s): min=11876, max=20042, per=100.00%, avg=16429.89, stdev=81.89, samples=3808
   iops        : min=3040376, max=5130956, avg=4206051.45, stdev=20963.19, samples=3808
  lat (nsec)   : 750=0.01%, 1000=0.01%
  lat (usec)   : 2=0.01%, 10=0.01%, 20=0.01%, 50=52.72%, 100=45.12%
  lat (usec)   : 250=2.14%, 500=0.01%, 750=0.01%, 1000=0.01%
  lat (msec)   : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
  cpu          : usr=11.02%, sys=37.05%, ctx=252169523, majf=0, minf=473
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,252164251,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
WRITE: bw=16.0GiB/s (17.2GB/s), 16.0GiB/s-16.0GiB/s (17.2GB/s-17.2GB/s), io=962GiB (1033GB), run=60002-60002msec

Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
RANDREAD: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=8
...
fio-3.27
Starting 32 processes
Jobs: 32 (f=32): [w(32)][100.0%][w=14.1GiB/s][w=3695k IOPS][eta 00m:00s]
RANDREAD: (groupid=0, jobs=32): err= 0: pid=21089: Tue Feb 1 23:46:58 2022

  write: IOPS=4130k, BW=15.8GiB/s (16.9GB/s)(945GiB/60002msec);
    slat (nsec): min=1222, max=12690k, avg=6987.00, stdev=5499.44
    clat (nsec): min=711, max=16691k, avg=54537.79, stdev=23234.80
     lat (usec): min=5, max=16701, avg=61.61, stdev=25.56
    clat percentiles (usec):
     |  1.00th=[   41],  5.00th=[   42], 10.00th=[   42], 20.00th=[   43],
     | 30.00th=[   44], 40.00th=[   45], 50.00th=[   48], 60.00th=[   57],
     | 70.00th=[   59], 80.00th=[   61], 90.00th=[   69], 95.00th=[   93],
     | 99.00th=[  131], 99.50th=[  141], 99.90th=[  184], 99.95th=[  204],
     | 99.99th=[  260]
   bw (  MiB/s): min=11551, max=20105, per=100.00%, avg=16156.07, stdev=74.80, samples=3808
   iops        : min=2957123, max=5146990, avg=4135953.66, stdev=19148.37, samples=3808
  lat (nsec)   : 750=0.01%, 1000=0.01%
  lat (usec)   : 2=0.01%, 10=0.01%, 20=0.01%, 50=52.25%, 100=44.71%
  lat (usec)   : 250=3.03%, 500=0.01%, 750=0.01%, 1000=0.01%
  lat (msec)   : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%
  cpu          : usr=10.94%, sys=36.18%, ctx=247812660, majf=0, minf=472
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,247805833,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
WRITE: bw=15.8GiB/s (16.9GB/s), 15.8GiB/s-15.8GiB/s (16.9GB/s-16.9GB/s), io=945GiB (1015GB), run=60002-60002msec

Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
RANDREAD: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=8
...
fio-3.27
Starting 32 processes
Jobs: 32 (f=32): [w(32)][100.0%][w=16.3GiB/s][w=4268k IOPS][eta 00m:00s]
RANDREAD: (groupid=0, jobs=32): err= 0: pid=22043: Tue Feb 1 23:47:58 2022

  write: IOPS=4170k, BW=15.9GiB/s (17.1GB/s)(955GiB/60002msec);
    slat (nsec): min=1763, max=17410k, avg=6908.89, stdev=5766.44
    clat (nsec): min=712, max=17505k, avg=54011.25, stdev=23339.67
     lat (usec): min=5, max=17513, avg=61.00, stdev=25.50
    clat percentiles (usec):
     |  1.00th=[   41],  5.00th=[   42], 10.00th=[   42], 20.00th=[   43],
     | 30.00th=[   44], 40.00th=[   45], 50.00th=[   48], 60.00th=[   57],
     | 70.00th=[   59], 80.00th=[   61], 90.00th=[   68], 95.00th=[   87],
     | 99.00th=[  121], 99.50th=[  135], 99.90th=[  178], 99.95th=[  204],
     | 99.99th=[  310]
   bw (  MiB/s): min=11707, max=19992, per=100.00%, avg=16303.82, stdev=73.70, samples=3808
   iops        : min=2997012, max=5118104, avg=4173776.76, stdev=18867.69, samples=3808
  lat (nsec)   : 750=0.01%, 1000=0.01%
  lat (usec)   : 2=0.01%, 10=0.01%, 20=0.01%, 50=52.47%, 100=45.42%
  lat (usec)   : 250=2.10%, 500=0.01%, 750=0.01%, 1000=0.01%
  lat (msec)   : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%
  cpu          : usr=11.04%, sys=36.68%, ctx=250242780, majf=0, minf=461
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,250236670,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
WRITE: bw=15.9GiB/s (17.1GB/s), 15.9GiB/s-15.9GiB/s (17.1GB/s-17.1GB/s), io=955GiB (1025GB), run=60002-60002msec

Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
RANDREAD: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=8
...
fio-3.27
Starting 32 processes
Jobs: 32 (f=32): [w(32)][100.0%][w=16.0GiB/s][w=4186k IOPS][eta 00m:00s]
RANDREAD: (groupid=0, jobs=32): err= 0: pid=22119: Tue Feb 1 23:48:58 2022

  write: IOPS=4203k, BW=16.0GiB/s (17.2GB/s)(962GiB/60002msec);
    slat (nsec): min=1162, max=13647k, avg=6841.84, stdev=4990.36
    clat (nsec): min=832, max=17776k, avg=53604.58, stdev=21052.59
     lat (usec): min=6, max=17783, avg=60.53, stdev=22.89
    clat percentiles (usec):
     |  1.00th=[   41],  5.00th=[   42], 10.00th=[   42], 20.00th=[   43],
     | 30.00th=[   44], 40.00th=[   45], 50.00th=[   50], 60.00th=[   58],
     | 70.00th=[   59], 80.00th=[   61], 90.00th=[   65], 95.00th=[   79],
     | 99.00th=[  103], 99.50th=[  121], 99.90th=[  153], 99.95th=[  176],
     | 99.99th=[  306]
   bw (  MiB/s): min=11133, max=20082, per=100.00%, avg=16434.73, stdev=79.43, samples=3808
   iops        : min=2850202, max=5141134, avg=4207290.84, stdev=20334.84, samples=3808
  lat (nsec)   : 1000=0.01%
  lat (usec)   : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=50.62%
  lat (usec)   : 100=48.12%, 250=1.25%, 500=0.01%, 750=0.01%, 1000=0.01%
  lat (msec)   : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%
  cpu          : usr=11.77%, sys=36.68%, ctx=252168793, majf=0, minf=462
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,252163391,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
WRITE: bw=16.0GiB/s (17.2GB/s), 16.0GiB/s-16.0GiB/s (17.2GB/s-17.2GB/s), io=962GiB (1033GB), run=60002-60002msec

Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
RANDREAD: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=8
...
fio-3.27
Starting 32 processes
Jobs: 32 (f=32): [w(32)][100.0%][w=16.5GiB/s][w=4326k IOPS][eta 00m:00s]
RANDREAD: (groupid=0, jobs=32): err= 0: pid=22221: Tue Feb 1 23:49:58 2022

  write: IOPS=4166k, BW=15.9GiB/s (17.1GB/s)(954GiB/60002msec);
    slat (nsec): min=1804, max=16555k, avg=6911.03, stdev=4216.56
    clat (nsec): min=731, max=16639k, avg=54075.05, stdev=17110.09
     lat (usec): min=5, max=16647, avg=61.07, stdev=18.98
    clat percentiles (usec):
     |  1.00th=[   41],  5.00th=[   42], 10.00th=[   42], 20.00th=[   43],
     | 30.00th=[   44], 40.00th=[   45], 50.00th=[   49], 60.00th=[   58],
     | 70.00th=[   60], 80.00th=[   62], 90.00th=[   67], 95.00th=[   81],
     | 99.00th=[  123], 99.50th=[  135], 99.90th=[  165], 99.95th=[  184],
     | 99.99th=[  210]
   bw (  MiB/s): min=11319, max=20001, per=100.00%, avg=16285.86, stdev=86.40, samples=3808
   iops        : min=2897836, max=5120368, avg=4169180.79, stdev=22118.90, samples=3808
  lat (nsec)   : 750=0.01%, 1000=0.01%
  lat (usec)   : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=51.04%
  lat (usec)   : 100=47.21%, 250=1.75%, 500=0.01%, 750=0.01%, 1000=0.01%
  lat (msec)   : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%
  cpu          : usr=11.43%, sys=36.52%, ctx=249967922, majf=0, minf=464
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,249962350,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
WRITE: bw=15.9GiB/s (17.1GB/s), 15.9GiB/s-15.9GiB/s (17.1GB/s-17.1GB/s), io=954GiB (1024GB), run=60002-60002msec

Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
RANDREAD: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=8
...
fio-3.27
Starting 32 processes
Jobs: 32 (f=32): [w(32)][100.0%][w=16.1GiB/s][w=4224k IOPS][eta 00m:00s]
RANDREAD: (groupid=0, jobs=32): err= 0: pid=22298: Tue Feb 1 23:50:59 2022

  write: IOPS=4106k, BW=15.7GiB/s (16.8GB/s)(940GiB/60002msec);
    slat (nsec): min=1372, max=18974k, avg=7033.73, stdev=4335.83
    clat (nsec): min=721, max=19073k, avg=54854.61, stdev=19806.39
     lat (usec): min=6, max=19082, avg=61.97, stdev=22.12
    clat percentiles (usec):
     |  1.00th=[   41],  5.00th=[   42], 10.00th=[   42], 20.00th=[   43],
     | 30.00th=[   44], 40.00th=[   45], 50.00th=[   47], 60.00th=[   57],
     | 70.00th=[   59], 80.00th=[   61], 90.00th=[   74], 95.00th=[   95],
     | 99.00th=[  133], 99.50th=[  147], 99.90th=[  184], 99.95th=[  196],
     | 99.99th=[  219]
   bw (  MiB/s): min=11060, max=20108, per=100.00%, avg=16051.21, stdev=92.86, samples=3808
   iops        : min=2831464, max=5147868, avg=4109110.77, stdev=23772.60, samples=3808
  lat (nsec)   : 750=0.01%, 1000=0.01%
  lat (usec)   : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=53.51%
  lat (usec)   : 100=42.99%, 250=3.49%, 500=0.01%, 750=0.01%, 1000=0.01%
  lat (msec)   : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%
  cpu          : usr=10.98%, sys=35.65%, ctx=246371950, majf=0, minf=458
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,246367496,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
WRITE: bw=15.7GiB/s (16.8GB/s), 15.7GiB/s-15.7GiB/s (16.8GB/s-16.8GB/s), io=940GiB (1009GB), run=60002-60002msec

Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
RANDREAD: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=8
...
fio-3.27
Starting 32 processes
Jobs: 32 (f=32): [w(32)][100.0%][w=14.9GiB/s][w=3895k IOPS][eta 00m:00s]
RANDREAD: (groupid=0, jobs=32): err= 0: pid=22372: Tue Feb 1 23:51:59 2022

  write: IOPS=4181k, BW=15.9GiB/s (17.1GB/s)(957GiB/60001msec);
    slat (nsec): min=1623, max=16114k, avg=6882.50, stdev=4431.39
    clat (nsec): min=712, max=16232k, avg=53879.81, stdev=16614.37
     lat (usec): min=6, max=16241, avg=60.84, stdev=18.29
    clat percentiles (usec):
     |  1.00th=[   41],  5.00th=[   42], 10.00th=[   42], 20.00th=[   43],
     | 30.00th=[   44], 40.00th=[   45], 50.00th=[   51], 60.00th=[   58],
     | 70.00th=[   60], 80.00th=[   61], 90.00th=[   67], 95.00th=[   80],
     | 99.00th=[  104], 99.50th=[  129], 99.90th=[  161], 99.95th=[  182],
     | 99.99th=[  219]
   bw (  MiB/s): min=12006, max=19952, per=100.00%, avg=16360.11, stdev=79.02, samples=3808
   iops        : min=3073692, max=5107912, avg=4188189.50, stdev=20228.57, samples=3808
  lat (nsec)   : 750=0.01%, 1000=0.01%
  lat (usec)   : 2=0.01%, 10=0.01%, 20=0.01%, 50=49.59%, 100=49.27%
  lat (usec)   : 250=1.14%, 500=0.01%, 750=0.01%, 1000=0.01%
  lat (msec)   : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%
  cpu          : usr=11.16%, sys=37.05%, ctx=250877181, majf=0, minf=498
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,250866850,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
WRITE: bw=15.9GiB/s (17.1GB/s), 15.9GiB/s-15.9GiB/s (17.1GB/s-17.1GB/s), io=957GiB (1028GB), run=60001-60001msec

Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
RANDREAD: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=8
...
fio-3.27
Starting 32 processes
Jobs: 32 (f=32): [w(32)][100.0%][w=16.1GiB/s][w=4230k IOPS][eta 00m:00s]
RANDREAD: (groupid=0, jobs=32): err= 0: pid=23453: Tue Feb 1 23:52:59 2022

  write: IOPS=4153k, BW=15.8GiB/s (17.0GB/s)(951GiB/60002msec);
    slat (nsec): min=1824, max=12568k, avg=6939.60, stdev=3999.18
    clat (nsec): min=701, max=16670k, avg=54238.72, stdev=17869.32
     lat (usec): min=6, max=16677, avg=61.26, stdev=19.85
    clat percentiles (usec):
     |  1.00th=[   41],  5.00th=[   42], 10.00th=[   42], 20.00th=[   43],
     | 30.00th=[   44], 40.00th=[   45], 50.00th=[   48], 60.00th=[   57],
     | 70.00th=[   59], 80.00th=[   61], 90.00th=[   69], 95.00th=[   84],
     | 99.00th=[  129], 99.50th=[  141], 99.90th=[  176], 99.95th=[  194],
     | 99.99th=[  217]
   bw (  MiB/s): min=11198, max=20030, per=100.00%, avg=16238.98, stdev=82.50, samples=3808
   iops        : min=2866772, max=5127914, avg=4157179.99, stdev=21119.50, samples=3808
  lat (nsec)   : 750=0.01%, 1000=0.01%
  lat (usec)   : 2=0.01%, 10=0.01%, 20=0.01%, 50=52.21%, 100=45.55%
  lat (usec)   : 250=2.23%, 500=0.01%, 750=0.01%, 1000=0.01%
  lat (msec)   : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%
  cpu          : usr=10.99%, sys=36.59%, ctx=249190314, majf=0, minf=490
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,249183943,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
WRITE: bw=15.8GiB/s (17.0GB/s), 15.8GiB/s-15.8GiB/s (17.0GB/s-17.0GB/s), io=951GiB (1021GB), run=60002-60002msec

Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
RANDREAD: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
...
fio-3.27
Starting 48 processes
Jobs: 48 (f=48): [r(48)][100.0%][r=18.9GiB/s][r=4960k IOPS][eta 00m:00s]
RANDREAD: (groupid=0, jobs=48): err= 0: pid=23530: Tue Feb 1 23:53:59 2022
read: IOPS=4942k, BW=18.9GiB/s (20.2GB/s)(1131GiB/60002msec)

    slat (nsec): min=1142, max=49607k, avg=8828.54, stdev=15569.98
    clat (nsec): min=441, max=243040k, avg=612190.41, stdev=339908.18
     lat (usec): min=6, max=243043, avg=621.12, stdev=343.94
    clat percentiles (usec):
     |  1.00th=[  404],  5.00th=[  412], 10.00th=[  420], 20.00th=[  523],
     | 30.00th=[  529], 40.00th=[  537], 50.00th=[  537], 60.00th=[  537],
     | 70.00th=[  545], 80.00th=[  553], 90.00th=[ 1139], 95.00th=[ 1188],
     | 99.00th=[ 1237], 99.50th=[ 1254], 99.90th=[ 1926], 99.95th=[ 4817],
     | 99.99th=[11469]
   bw (  MiB/s): min= 1691, max=27419, per=100.00%, avg=19320.99, stdev=89.34, samples=5712
   iops        : min=433098, max=7019516, avg=4946173.24, stdev=22869.85, samples=5712
  lat (nsec)   : 500=0.01%, 750=0.01%, 1000=0.01%
  lat (usec)   : 2=0.01%, 10=0.01%, 20=0.01%, 50=0.01%, 100=0.01%
  lat (usec)   : 250=0.01%, 500=15.40%, 750=69.25%, 1000=4.03%
  lat (msec)   : 2=11.24%, 4=0.03%, 10=0.04%, 20=0.01%, 50=0.01%
  lat (msec)   : 100=0.01%, 250=0.01%
  cpu          : usr=9.29%, sys=35.84%, ctx=296529230, majf=0, minf=3836
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=296546544,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
READ: bw=18.9GiB/s (20.2GB/s), 18.9GiB/s-18.9GiB/s (20.2GB/s-20.2GB/s), io=1131GiB (1215GB), run=60002-60002msec

Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
RANDREAD: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
...
fio-3.27
Starting 48 processes
Jobs: 48 (f=48): [r(48)][100.0%][r=18.4GiB/s][r=4834k IOPS][eta 00m:00s]
RANDREAD: (groupid=0, jobs=48): err= 0: pid=23882: Tue Feb 1 23:55:00 2022
read: IOPS=4940k, BW=18.8GiB/s (20.2GB/s)(1131GiB/60002msec)

    slat (usec): min=2, max=16996, avg= 8.84, stdev= 6.11
    clat (nsec): min=911, max=21566k, avg=612493.08, stdev=234517.40
     lat (usec): min=4, max=21574, avg=621.43, stdev=238.08
    clat percentiles (usec):
     |  1.00th=[  404],  5.00th=[  412], 10.00th=[  420], 20.00th=[  523],
     | 30.00th=[  529], 40.00th=[  537], 50.00th=[  537], 60.00th=[  537],
     | 70.00th=[  545], 80.00th=[  553], 90.00th=[ 1123], 95.00th=[ 1172],
     | 99.00th=[ 1237], 99.50th=[ 1254], 99.90th=[ 1549], 99.95th=[ 2024],
     | 99.99th=[ 3261]
   bw (  MiB/s): min=12590, max=26589, per=100.00%, avg=19314.78, stdev=65.21, samples=5712
   iops        : min=3223108, max=6806844, avg=4944582.49, stdev=16693.22, samples=5712
  lat (nsec)   : 1000=0.01%
  lat (usec)   : 2=0.01%, 10=0.01%, 20=0.01%, 50=0.01%, 100=0.01%
  lat (usec)   : 250=0.01%, 500=14.27%, 750=69.32%, 1000=5.38%
  lat (msec)   : 2=10.98%, 4=0.05%, 10=0.01%, 20=0.01%, 50=0.01%
  cpu          : usr=9.29%, sys=36.21%, ctx=296412664, majf=0, minf=3763
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=296403321,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
READ: bw=18.8GiB/s (20.2GB/s), 18.8GiB/s-18.8GiB/s (20.2GB/s-20.2GB/s), io=1131GiB (1214GB), run=60002-60002msec

Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
RANDREAD: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
...
fio-3.27
Starting 48 processes
Jobs: 48 (f=48): [r(48)][100.0%][r=18.6GiB/s][r=4885k IOPS][eta 00m:00s]
RANDREAD: (groupid=0, jobs=48): err= 0: pid=24112: Tue Feb 1 23:56:00 2022
read: IOPS=5041k, BW=19.2GiB/s (20.6GB/s)(1154GiB/60002msec)

    slat (usec): min=2, max=22808, avg= 8.65, stdev= 6.86
    clat (nsec): min=912, max=26105k, avg=600221.79, stdev=221404.88
     lat (usec): min=5, max=26112, avg=608.96, stdev=224.71
    clat percentiles (usec):
     |  1.00th=[  404],  5.00th=[  412], 10.00th=[  424], 20.00th=[  523],
     | 30.00th=[  529], 40.00th=[  529], 50.00th=[  537], 60.00th=[  537],
     | 70.00th=[  545], 80.00th=[  553], 90.00th=[ 1106], 95.00th=[ 1188],
     | 99.00th=[ 1237], 99.50th=[ 1254], 99.90th=[ 1287], 99.95th=[ 1319],
     | 99.99th=[ 2057]
   bw (  MiB/s): min=16300, max=26946, per=100.00%, avg=19713.39, stdev=67.57, samples=5712
   iops        : min=4173052, max=6898318, avg=5046626.47, stdev=17299.17, samples=5712
  lat (nsec)   : 1000=0.01%
  lat (usec)   : 2=0.01%, 10=0.01%, 20=0.01%, 50=0.01%, 100=0.01%
  lat (usec)   : 250=0.01%, 500=12.91%, 750=73.53%, 1000=3.26%
  lat (msec)   : 2=10.28%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
  cpu          : usr=9.37%, sys=36.74%, ctx=302479289, majf=0, minf=3758
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=302468143,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
READ: bw=19.2GiB/s (20.6GB/s), 19.2GiB/s-19.2GiB/s (20.6GB/s-20.6GB/s), io=1154GiB (1239GB), run=60002-60002msec

Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
RANDREAD: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
...
fio-3.27
Starting 48 processes
Jobs: 48 (f=48): [r(48)][100.0%][r=17.9GiB/s][r=4683k IOPS][eta 00m:00s]
RANDREAD: (groupid=0, jobs=48): err= 0: pid=24212: Tue Feb 1 23:57:00 2022
read: IOPS=4989k, BW=19.0GiB/s (20.4GB/s)(1142GiB/60001msec)

    slat (nsec): min=1953, max=52925k, avg=8751.89, stdev=11303.93
    clat (nsec): min=912, max=56316k, avg=606515.90, stdev=246434.14
     lat (usec): min=5, max=56326, avg=615.36, stdev=249.97
    clat percentiles (usec):
     |  1.00th=[  400],  5.00th=[  412], 10.00th=[  420], 20.00th=[  519],
     | 30.00th=[  529], 40.00th=[  529], 50.00th=[  537], 60.00th=[  537],
     | 70.00th=[  545], 80.00th=[  553], 90.00th=[ 1123], 95.00th=[ 1188],
     | 99.00th=[ 1237], 99.50th=[ 1270], 99.90th=[ 1745], 99.95th=[ 1827],
     | 99.99th=[ 4752]
   bw (  MiB/s): min=13300, max=27673, per=100.00%, avg=19511.23, stdev=82.41, samples=5712
   iops        : min=3405011, max=7084445, avg=4994872.61, stdev=21096.46, samples=5712
  lat (nsec)   : 1000=0.01%
  lat (usec)   : 2=0.01%, 10=0.01%, 20=0.01%, 50=0.01%, 100=0.01%
  lat (usec)   : 250=0.01%, 500=14.24%, 750=70.95%, 1000=3.71%
  lat (msec)   : 2=11.08%, 4=0.02%, 10=0.01%, 20=0.01%, 50=0.01%
  lat (msec)   : 100=0.01%
  cpu          : usr=9.31%, sys=36.30%, ctx=299345988, majf=0, minf=3730
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=299326708,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
READ: bw=19.0GiB/s (20.4GB/s), 19.0GiB/s-19.0GiB/s (20.4GB/s-20.4GB/s), io=1142GiB (1226GB), run=60001-60001msec

Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
RANDREAD: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
...
fio-3.27
Starting 48 processes
Jobs: 48 (f=48): [r(48)][100.0%][r=17.8GiB/s][r=4674k IOPS][eta 00m:00s]
RANDREAD: (groupid=0, jobs=48): err= 0: pid=25485: Tue Feb 1 23:58:00 2022
read: IOPS=4982k, BW=19.0GiB/s (20.4GB/s)(1140GiB/60001msec)

    slat (nsec): min=1473, max=41255k, avg=8761.08, stdev=7916.28
    clat (nsec): min=871, max=45519k, avg=607322.90, stdev=229101.48
     lat (usec): min=5, max=45526, avg=616.18, stdev=232.51
    clat percentiles (usec):
     |  1.00th=[  400],  5.00th=[  412], 10.00th=[  420], 20.00th=[  519],
     | 30.00th=[  529], 40.00th=[  529], 50.00th=[  537], 60.00th=[  545],
     | 70.00th=[  545], 80.00th=[  553], 90.00th=[ 1106], 95.00th=[ 1172],
     | 99.00th=[ 1221], 99.50th=[ 1237], 99.90th=[ 1287], 99.95th=[ 1303],
     | 99.99th=[ 1745]
   bw (  MiB/s): min=15677, max=27351, per=100.00%, avg=19485.07, stdev=73.52, samples=5712
   iops        : min=4013328, max=7002048, avg=4988177.24, stdev=18821.75, samples=5712
  lat (nsec)   : 1000=0.01%
  lat (usec)   : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
  lat (usec)   : 100=0.01%, 250=0.01%, 500=14.54%, 750=69.53%, 1000=5.29%
  lat (msec)   : 2=10.62%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
  cpu          : usr=9.33%, sys=36.13%, ctx=298932803, majf=0, minf=3800
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=298924191,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
READ: bw=19.0GiB/s (20.4GB/s), 19.0GiB/s-19.0GiB/s (20.4GB/s-20.4GB/s), io=1140GiB (1224GB), run=60001-60001msec

Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
RANDREAD: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
...
fio-3.27
Starting 48 processes
Jobs: 48 (f=48): [r(48)][100.0%][r=18.2GiB/s][r=4763k IOPS][eta 00m:00s]
RANDREAD: (groupid=0, jobs=48): err= 0: pid=25572: Tue Feb 1 23:59:01 2022
read: IOPS=5031k, BW=19.2GiB/s (20.6GB/s)(1152GiB/60001msec)

    slat (nsec): min=1985, max=21488k, avg=8673.54, stdev=5859.90
    clat (nsec): min=902, max=22056k, avg=601397.58, stdev=214885.23
     lat (usec): min=5, max=22064, avg=610.17, stdev=218.14
    clat percentiles (usec):
     |  1.00th=[  400],  5.00th=[  412], 10.00th=[  424], 20.00th=[  523],
     | 30.00th=[  529], 40.00th=[  537], 50.00th=[  537], 60.00th=[  545],
     | 70.00th=[  553], 80.00th=[  553], 90.00th=[  971], 95.00th=[ 1172],
     | 99.00th=[ 1221], 99.50th=[ 1237], 99.90th=[ 1270], 99.95th=[ 1287],
     | 99.99th=[ 1631]
   bw (  MiB/s): min=14781, max=27237, per=100.00%, avg=19676.38, stdev=75.29, samples=5712
   iops        : min=3784135, max=6972871, avg=5037153.12, stdev=19275.07, samples=5712
  lat (nsec)   : 1000=0.01%
  lat (usec)   : 2=0.01%, 10=0.01%, 20=0.01%, 50=0.01%, 100=0.01%
  lat (usec)   : 250=0.01%, 500=13.17%, 750=72.94%, 1000=4.22%
  lat (msec)   : 2=9.67%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
  cpu          : usr=9.32%, sys=36.93%, ctx=301882480, majf=0, minf=3751
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=301872061,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
READ: bw=19.2GiB/s (20.6GB/s), 19.2GiB/s-19.2GiB/s (20.6GB/s-20.6GB/s), io=1152GiB (1236GB), run=60001-60001msec

Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
RANDREAD: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
...
fio-3.27
Starting 48 processes
Jobs: 48 (f=48): [r(48)][100.0%][r=17.6GiB/s][r=4607k IOPS][eta 00m:00s]
RANDREAD: (groupid=0, jobs=48): err= 0: pid=25664: Wed Feb 2 00:00:01 2022
read: IOPS=4948k, BW=18.9GiB/s (20.3GB/s)(1133GiB/60002msec)

    slat (usec): min=2, max=21123, avg= 8.83, stdev= 5.96
    clat (nsec): min=942, max=22372k, avg=611470.57, stdev=236265.95
     lat (usec): min=6, max=22391, avg=620.40, stdev=239.84
    clat percentiles (usec):
     |  1.00th=[  396],  5.00th=[  408], 10.00th=[  416], 20.00th=[  515],
     | 30.00th=[  523], 40.00th=[  529], 50.00th=[  537], 60.00th=[  545],
     | 70.00th=[  545], 80.00th=[  562], 90.00th=[ 1139], 95.00th=[ 1188],
     | 99.00th=[ 1221], 99.50th=[ 1237], 99.90th=[ 1287], 99.95th=[ 1303],
     | 99.99th=[ 1795]
   bw (  MiB/s): min=15222, max=27543, per=100.00%, avg=19354.61, stdev=79.33, samples=5712
   iops        : min=3896862, max=7051174, avg=4954779.41, stdev=20309.68, samples=5712
  lat (nsec)   : 1000=0.01%
  lat (usec)   : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
  lat (usec)   : 100=0.01%, 250=0.01%, 500=16.87%, 750=66.30%, 1000=4.54%
  lat (msec)   : 2=12.28%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
  cpu          : usr=9.19%, sys=35.87%, ctx=296904832, majf=0, minf=3743
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=296894931,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
READ: bw=18.9GiB/s (20.3GB/s), 18.9GiB/s-18.9GiB/s (20.3GB/s-20.3GB/s), io=1133GiB (1216GB), run=60002-60002msec

Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
RANDREAD: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
...
fio-3.27
Starting 48 processes
Jobs: 48 (f=48): [r(48)][100.0%][r=17.4GiB/s][r=4554k IOPS][eta 00m:00s]
RANDREAD: (groupid=0, jobs=48): err= 0: pid=25785: Wed Feb 2 00:01:01 2022
read: IOPS=4720k, BW=18.0GiB/s (19.3GB/s)(1080GiB/60002msec)

    slat (usec): min=2, max=28533, avg= 9.30, stdev= 7.03
    clat (nsec): min=921, max=29089k, avg=640985.06, stdev=258045.84
     lat (usec): min=5, max=29096, avg=650.39, stdev=261.95
    clat percentiles (usec):
     |  1.00th=[  392],  5.00th=[  404], 10.00th=[  412], 20.00th=[  510],
     | 30.00th=[  523], 40.00th=[  529], 50.00th=[  537], 60.00th=[  545],
     | 70.00th=[  553], 80.00th=[  906], 90.00th=[ 1156], 95.00th=[ 1188],
     | 99.00th=[ 1237], 99.50th=[ 1254], 99.90th=[ 1303], 99.95th=[ 1401],
     | 99.99th=[ 1876]
   bw (  MiB/s): min=13222, max=27706, per=100.00%, avg=18458.01, stdev=98.55, samples=5712
   iops        : min=3384868, max=7092862, avg=4725251.34, stdev=25229.04, samples=5712
  lat (nsec)   : 1000=0.01%
  lat (usec)   : 2=0.01%, 10=0.01%, 20=0.01%, 50=0.01%, 100=0.01%
  lat (usec)   : 250=0.01%, 500=18.90%, 750=56.44%, 1000=10.71%
  lat (msec)   : 2=13.94%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
  cpu          : usr=8.95%, sys=33.65%, ctx=283232675, majf=0, minf=3746
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=283217843,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
READ: bw=18.0GiB/s (19.3GB/s), 18.0GiB/s-18.0GiB/s (19.3GB/s-19.3GB/s), io=1080GiB (1160GB), run=60002-60002msec

Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
RANDREAD: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
...
fio-3.27
Starting 48 processes
Jobs: 48 (f=48): [r(48)][100.0%][r=18.2GiB/s][r=4774k IOPS][eta 00m:00s]
RANDREAD: (groupid=0, jobs=48): err= 0: pid=25897: Wed Feb 2 00:02:01 2022
read: IOPS=4830k, BW=18.4GiB/s (19.8GB/s)(1106GiB/60002msec)

    slat (usec): min=2, max=37297, avg= 9.06, stdev=11.53
    clat (nsec): min=892, max=37911k, avg=626423.87, stdev=281337.79
     lat (usec): min=5, max=37919, avg=635.59, stdev=285.46
    clat percentiles (usec):
     |  1.00th=[  400],  5.00th=[  412], 10.00th=[  429], 20.00th=[  515],
     | 30.00th=[  523], 40.00th=[  529], 50.00th=[  537], 60.00th=[  537],
     | 70.00th=[  545], 80.00th=[  562], 90.00th=[ 1156], 95.00th=[ 1205],
     | 99.00th=[ 1549], 99.50th=[ 1680], 99.90th=[ 2835], 99.95th=[ 3392],
     | 99.99th=[ 5407]
   bw (  MiB/s): min= 6580, max=27466, per=100.00%, avg=18885.17, stdev=100.72, samples=5712
   iops        : min=1684599, max=7031464, avg=4834602.87, stdev=25783.81, samples=5712
  lat (nsec)   : 1000=0.01%
  lat (usec)   : 2=0.01%, 10=0.01%, 20=0.01%, 50=0.01%, 100=0.01%
  lat (usec)   : 250=0.01%, 500=12.90%, 750=70.01%, 1000=4.60%
  lat (msec)   : 2=12.35%, 4=0.13%, 10=0.02%, 20=0.01%, 50=0.01%
  cpu          : usr=9.17%, sys=37.36%, ctx=289835159, majf=0, minf=3741
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=289804479,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
READ: bw=18.4GiB/s (19.8GB/s), 18.4GiB/s-18.4GiB/s (19.8GB/s-19.8GB/s), io=1106GiB (1187GB), run=60002-60002msec

Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
RANDREAD: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
...
fio-3.27
Starting 48 processes
Jobs: 48 (f=48): [r(48)][100.0%][r=18.3GiB/s][r=4805k IOPS][eta 00m:00s]
RANDREAD: (groupid=0, jobs=48): err= 0: pid=26489: Wed Feb 2 00:03:02 2022
read: IOPS=4915k, BW=18.7GiB/s (20.1GB/s)(1125GiB/60002msec)

    slat (nsec): min=1714, max=56074k, avg=8893.16, stdev=12899.01
    clat (nsec): min=922, max=57329k, avg=615610.19, stdev=258832.65
     lat (usec): min=5, max=57337, avg=624.60, stdev=262.51
    clat percentiles (usec):
     |  1.00th=[  400],  5.00th=[  408], 10.00th=[  420], 20.00th=[  515],
     | 30.00th=[  523], 40.00th=[  529], 50.00th=[  537], 60.00th=[  545],
     | 70.00th=[  545], 80.00th=[  562], 90.00th=[ 1139], 95.00th=[ 1188],
     | 99.00th=[ 1237], 99.50th=[ 1254], 99.90th=[ 1418], 99.95th=[ 1811],
     | 99.99th=[ 4948]
   bw (  MiB/s): min=14198, max=26839, per=100.00%, avg=19217.89, stdev=63.10, samples=5712
   iops        : min=3634881, max=6870838, avg=4919778.50, stdev=16154.49, samples=5712
  lat (nsec)   : 1000=0.01%
  lat (usec)   : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
  lat (usec)   : 100=0.01%, 250=0.01%, 500=15.84%, 750=67.55%, 1000=3.63%
  lat (msec)   : 2=12.94%, 4=0.03%, 10=0.01%, 20=0.01%, 50=0.01%
  lat (msec)   : 100=0.01%
  cpu          : usr=9.32%, sys=35.56%, ctx=294925914, majf=0, minf=3702
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=294900613,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
READ: bw=18.7GiB/s (20.1GB/s), 18.7GiB/s-18.7GiB/s (20.1GB/s-20.1GB/s), io=1125GiB (1208GB), run=60002-60002msec

Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
+ NQN=testnqn
+ nvme disconnect -n testnqn
NQN:testnqn disconnected 1 controller(s)

real 0m0.349s
user 0m0.000s
sys 0m0.003s
+ for i in /sys/kernel/config/nvmet/subsystems/${NQN}/namespaces/*
+ echo 0
+ rmdir /sys/kernel/config/nvmet/subsystems/testnqn/namespaces/1
+ rm -fr /sys/kernel/config/nvmet/ports/1/subsystems/testnqn
+ rmdir /sys/kernel/config/nvmet/ports/1
+ rmdir /sys/kernel/config/nvmet/subsystems/testnqn
+ rmdir 'config/nullb/nullb*'
rmdir: failed to remove 'config/nullb/nullb*': No such file or directory
+ umount /mnt/nvme0n1
umount: /mnt/nvme0n1: no mount point specified.
+ umount /mnt/backend
umount: /mnt/backend: not mounted.
+ modprobe -r nvme_loop
+ modprobe -r nvme_fabrics
+ modprobe -r nvmet
+ modprobe -r nvme
+ modprobe -r null_blk
+ tree /sys/kernel/config
/sys/kernel/config

0 directories, 0 files
+++ NN=1
+++ NQN=testnqn
+++ let NR_DEVICES=NN+1
+++ modprobe -r null_blk
+++ modprobe -r nvme
+++ modprobe null_blk nr_devices=0
+++ modprobe nvme
+++ modprobe nvme-fabrics
+++ modprobe nvmet
+++ modprobe nvme-loop
+++ dmesg -c
+++ sleep 2
+++ tree /sys/kernel/config
/sys/kernel/config

├── nullb
│   └── features
└── nvmet
    ├── hosts
    ├── ports
    └── subsystems

5 directories, 1 file
+++ mkdir /sys/kernel/config/nvmet/subsystems/testnqn
+++ mkdir /sys/kernel/config/nvmet/subsystems/testnqn/namespaces/1
+++ echo -n /dev/nvme0n1
+++ cat /sys/kernel/config/nvmet/subsystems/testnqn/namespaces/1/device_path /dev/nvme0n1
+++ echo 0
+++ cat /sys/kernel/config/nvmet/subsystems/testnqn/namespaces/1/buffered_io
0
+++ echo 1
+++ mkdir /sys/kernel/config/nvmet/ports/1/
+++ echo -n loop
+++ echo -n 1
+++ ln -s /sys/kernel/config/nvmet/subsystems/testnqn /sys/kernel/config/nvmet/ports/1/subsystems/ +++ sleep 1
+++ echo transport=loop,nqn=testnqn
+++ sleep 1
+++ mount
+++ column -t
+++ grep nvme
+++ dmesg -c

[ 3560.886773] nvme nvme0: 48/0/0 default/read/poll queues
[ 3562.872451] nvmet: adding nsid 1 to subsystem testnqn
[ 3563.883217] nvmet: creating nvm controller 1 for subsystem testnqn for NQN nqn.2014-08.org.nvmexpress:uuid:bc654437-83be-4628-91bd-a439b8e830d5.
[ 3563.883353] nvme nvme1: creating 48 I/O queues.
[ 3563.887931] nvme nvme1: new ctrl: "testnqn"

write-and-verify: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=4 fio-3.27
Starting 1 process
Jobs: 1 (f=1): [V(1)][80.0%][r=99.2MiB/s][r=25.4k IOPS][eta 00m:03s]
write-and-verify: (groupid=0, jobs=1): err= 0: pid=27715: Wed Feb 2 00:03:19 2022
read: IOPS=24.3k, BW=94.9MiB/s (99.5MB/s)(506MiB/5335msec)

    slat (nsec): min=1032, max=1447.4k, avg=8870.69, stdev=27575.11
    clat (usec): min=11, max=4669, avg=153.54, stdev=131.98
     lat (usec): min=47, max=4672, avg=162.54, stdev=137.66
    clat percentiles (usec):
     |  1.00th=[   62],  5.00th=[   86], 10.00th=[   96], 20.00th=[  106],
     | 30.00th=[  113], 40.00th=[  116], 50.00th=[  120], 60.00th=[  125],
     | 70.00th=[  135], 80.00th=[  198], 90.00th=[  221], 95.00th=[  251],
     | 99.00th=[  709], 99.50th=[  947], 99.90th=[ 1663], 99.95th=[ 2114],
     | 99.99th=[ 2933]
  write: IOPS=27.9k, BW=109MiB/s (114MB/s)(800MiB/7353msec);
    slat (usec): min=2, max=3897, avg=13.26, stdev=21.42
    clat (usec): min=15, max=5720, avg=129.39, stdev=79.93
     lat (usec): min=37, max=6857, avg=142.77, stdev=84.05
    clat percentiles (usec):
     |  1.00th=[   63],  5.00th=[   77], 10.00th=[   84], 20.00th=[   96],
     | 30.00th=[  105], 40.00th=[  115], 50.00th=[  123], 60.00th=[  129],
     | 70.00th=[  137], 80.00th=[  151], 90.00th=[  169], 95.00th=[  215],
     | 99.00th=[  273], 99.50th=[  310], 99.90th=[ 1057], 99.95th=[ 1696],
     | 99.99th=[ 2868]
   bw (  KiB/s): min=82496, max=143312, per=98.04%, avg=109226.13, stdev=16354.90, samples=15
   iops        : min=20624, max=35828, avg=27306.67, stdev=4088.95, samples=15
  lat (usec)   : 20=0.01%, 50=0.28%, 100=19.29%, 250=77.38%, 500=2.09%
  lat (usec)   : 750=0.52%, 1000=0.20%
  lat (msec)   : 2=0.19%, 4=0.05%, 10=0.01%
  cpu          : usr=18.65%, sys=16.14%, ctx=141033, majf=0, minf=3053
  IO depths    : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=129572,204800,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4

Run status group 0 (all jobs):
READ: bw=94.9MiB/s (99.5MB/s), 94.9MiB/s-94.9MiB/s (99.5MB/s-99.5MB/s), io=506MiB (531MB), run=5335-5335msec
WRITE: bw=109MiB/s (114MB/s), 109MiB/s-109MiB/s (114MB/s-114MB/s), io=800MiB (839MB), run=7353-7353msec

Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
RANDREAD: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=8
...
fio-3.27
Starting 32 processes
Jobs: 32 (f=32): [w(32)][100.0%][w=253MiB/s][w=64.8k IOPS][eta 00m:00s]
RANDREAD: (groupid=0, jobs=32): err= 0: pid=27744: Wed Feb 2 00:04:20 2022

  write: IOPS=68.2k, BW=266MiB/s (279MB/s)(15.6GiB/60002msec);
    slat (nsec): min=1072, max=42904k, avg=162051.61, stdev=467637.32
    clat (nsec): min=752, max=72679k, avg=3588602.81, stdev=2243011.38
     lat (usec): min=52, max=72681, avg=3750.87, stdev=2268.70
    clat percentiles (usec):
     |  1.00th=[  562],  5.00th=[ 1029], 10.00th=[ 1369], 20.00th=[ 1860],
     | 30.00th=[ 2278], 40.00th=[ 2671], 50.00th=[ 3097], 60.00th=[ 3589],
     | 70.00th=[ 4178], 80.00th=[ 5014], 90.00th=[ 6325], 95.00th=[ 7701],
     | 99.00th=[11076], 99.50th=[12780], 99.90th=[18220], 99.95th=[21890],
     | 99.99th=[30278]
   bw (  KiB/s): min=193029, max=335346, per=100.00%, avg=273269.63, stdev=877.09, samples=3808
   iops        : min=48248, max=83826, avg=68310.92, stdev=219.29, samples=3808
  lat (nsec)   : 1000=0.01%
  lat (usec)   : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.16%
  lat (usec)   : 100=0.04%, 250=0.09%, 500=0.46%, 750=1.38%, 1000=2.46%
  lat (msec)   : 2=18.84%, 4=43.56%, 10=31.38%, 20=1.55%, 50=0.07%
  lat (msec)   : 100=0.01%
  cpu          : usr=0.63%, sys=1.57%, ctx=1935887, majf=0, minf=485
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,4093399,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
WRITE: bw=266MiB/s (279MB/s), 266MiB/s-266MiB/s (279MB/s-279MB/s), io=15.6GiB (16.8GB), run=60002-60002msec

Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
RANDREAD: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=8
...
fio-3.27
Starting 32 processes
Jobs: 32 (f=32): [w(32)][100.0%][w=251MiB/s][w=64.3k IOPS][eta 00m:00s]
RANDREAD: (groupid=0, jobs=32): err= 0: pid=27840: Wed Feb 2 00:05:20 2022

  write: IOPS=68.6k, BW=268MiB/s (281MB/s)(15.7GiB/60003msec);
    slat (nsec): min=1052, max=52671k, avg=146009.86, stdev=449344.58
    clat (nsec): min=641, max=64251k, avg=3581258.82, stdev=2231979.80
     lat (usec): min=55, max=64976, avg=3727.49, stdev=2255.52
    clat percentiles (usec):
     |  1.00th=[  570],  5.00th=[ 1037], 10.00th=[ 1369], 20.00th=[ 1844],
     | 30.00th=[ 2278], 40.00th=[ 2671], 50.00th=[ 3097], 60.00th=[ 3589],
     | 70.00th=[ 4178], 80.00th=[ 5014], 90.00th=[ 6325], 95.00th=[ 7701],
     | 99.00th=[10945], 99.50th=[12649], 99.90th=[18220], 99.95th=[21627],
     | 99.99th=[30016]
   bw (  KiB/s): min=203172, max=338441, per=100.00%, avg=275054.71, stdev=839.49, samples=3808
   iops        : min=50788, max=84602, avg=68758.55, stdev=209.87, samples=3808
  lat (nsec)   : 750=0.01%, 1000=0.01%
  lat (usec)   : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.15%
  lat (usec)   : 100=0.03%, 250=0.08%, 500=0.45%, 750=1.40%, 1000=2.44%
  lat (msec)   : 2=18.96%, 4=43.62%, 10=31.29%, 20=1.51%, 50=0.07%
  lat (msec)   : 100=0.01%
  cpu          : usr=0.63%, sys=1.56%, ctx=1832905, majf=0, minf=488
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,4119099,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
WRITE: bw=268MiB/s (281MB/s), 268MiB/s-268MiB/s (281MB/s-281MB/s), io=15.7GiB (16.9GB), run=60003-60003msec

Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
RANDREAD: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=8
...
fio-3.27
Starting 32 processes
Jobs: 32 (f=32): [w(32)][100.0%][w=270MiB/s][w=69.1k IOPS][eta 00m:00s]
RANDREAD: (groupid=0, jobs=32): err= 0: pid=27949: Wed Feb 2 00:06:20 2022

  write: IOPS=68.2k, BW=266MiB/s (279MB/s)(15.6GiB/60003msec);
    slat (nsec): min=1082, max=41371k, avg=222396.07, stdev=539362.31
    clat (nsec): min=631, max=67998k, avg=3529440.50, stdev=2240119.90
     lat (usec): min=54, max=69361, avg=3752.10, stdev=2276.78
    clat percentiles (usec):
     |  1.00th=[  529],  5.00th=[ 1012], 10.00th=[ 1352], 20.00th=[ 1827],
     | 30.00th=[ 2245], 40.00th=[ 2638], 50.00th=[ 3064], 60.00th=[ 3523],
     | 70.00th=[ 4113], 80.00th=[ 4883], 90.00th=[ 6194], 95.00th=[ 7570],
     | 99.00th=[11076], 99.50th=[12780], 99.90th=[20055], 99.95th=[23987],
     | 99.99th=[31327]
   bw (  KiB/s): min=195270, max=336343, per=100.00%, avg=273029.60, stdev=880.99, samples=3808
   iops        : min=48813, max=84075, avg=68253.36, stdev=220.24, samples=3808
  lat (nsec)   : 750=0.01%, 1000=0.01%
  lat (usec)   : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.20%
  lat (usec)   : 100=0.05%, 250=0.13%, 500=0.52%, 750=1.45%, 1000=2.48%
  lat (msec)   : 2=19.07%, 4=44.45%, 10=30.09%, 20=1.45%, 50=0.10%
  lat (msec)   : 100=0.01%
  cpu          : usr=0.64%, sys=1.69%, ctx=2408175, majf=0, minf=508
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,4091921,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
WRITE: bw=266MiB/s (279MB/s), 266MiB/s-266MiB/s (279MB/s-279MB/s), io=15.6GiB (16.8GB), run=60003-60003msec

Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
RANDREAD: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=8
...
fio-3.27
Starting 32 processes
Jobs: 32 (f=32): [w(32)][100.0%][w=268MiB/s][w=68.6k IOPS][eta 00m:00s]
RANDREAD: (groupid=0, jobs=32): err= 0: pid=28031: Wed Feb 2 00:07:20 2022

  write: IOPS=68.3k, BW=267MiB/s (280MB/s)(15.6GiB/60003msec);
    slat (nsec): min=1062, max=50611k, avg=206709.24, stdev=515844.58
    clat (nsec): min=491, max=58217k, avg=3538847.51, stdev=2211954.38
     lat (usec): min=46, max=58793, avg=3745.81, stdev=2241.07
    clat percentiles (usec):
     |  1.00th=[  529],  5.00th=[ 1020], 10.00th=[ 1352], 20.00th=[ 1844],
     | 30.00th=[ 2245], 40.00th=[ 2671], 50.00th=[ 3064], 60.00th=[ 3556],
     | 70.00th=[ 4146], 80.00th=[ 4883], 90.00th=[ 6259], 95.00th=[ 7570],
     | 99.00th=[10945], 99.50th=[12649], 99.90th=[18744], 99.95th=[22414],
     | 99.99th=[30540]
   bw (  KiB/s): min=205666, max=333724, per=100.00%, avg=273520.75, stdev=810.61, samples=3808
   iops        : min=51408, max=83423, avg=68374.75, stdev=202.66, samples=3808
  lat (nsec)   : 500=0.01%, 750=0.01%, 1000=0.01%
  lat (usec)   : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.21%
  lat (usec)   : 100=0.05%, 250=0.12%, 500=0.50%, 750=1.42%, 1000=2.45%
  lat (msec)   : 2=18.99%, 4=44.21%, 10=30.54%, 20=1.43%, 50=0.08%
  lat (msec)   : 100=0.01%
  cpu          : usr=0.66%, sys=1.67%, ctx=2291683, majf=0, minf=481
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,4098881,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
WRITE: bw=267MiB/s (280MB/s), 267MiB/s-267MiB/s (280MB/s-280MB/s), io=15.6GiB (16.8GB), run=60003-60003msec

Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
RANDREAD: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=8
...
fio-3.27
Starting 32 processes
Jobs: 32 (f=32): [w(32)][100.0%][w=268MiB/s][w=68.6k IOPS][eta 00m:00s]
RANDREAD: (groupid=0, jobs=32): err= 0: pid=28274: Wed Feb 2 00:08:21 2022

  write: IOPS=69.0k, BW=269MiB/s (283MB/s)(15.8GiB/60002msec);
    slat (nsec): min=1062, max=68056k, avg=176848.85, stdev=488062.07
    clat (nsec): min=692, max=77359k, avg=3532992.98, stdev=2215070.84
     lat (usec): min=42, max=78568, avg=3710.06, stdev=2245.78
    clat percentiles (usec):
     |  1.00th=[  545],  5.00th=[ 1012], 10.00th=[ 1352], 20.00th=[ 1827],
     | 30.00th=[ 2245], 40.00th=[ 2638], 50.00th=[ 3064], 60.00th=[ 3556],
     | 70.00th=[ 4146], 80.00th=[ 4948], 90.00th=[ 6259], 95.00th=[ 7570],
     | 99.00th=[10814], 99.50th=[12518], 99.90th=[18482], 99.95th=[22414],
     | 99.99th=[30540]
   bw (  KiB/s): min=201233, max=338661, per=100.00%, avg=276170.71, stdev=819.64, samples=3808
   iops        : min=50302, max=84658, avg=69037.91, stdev=204.91, samples=3808
  lat (nsec)   : 750=0.01%, 1000=0.01%
  lat (usec)   : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.16%
  lat (usec)   : 100=0.04%, 250=0.11%, 500=0.50%, 750=1.46%, 1000=2.53%
  lat (msec)   : 2=19.26%, 4=43.92%, 10=30.52%, 20=1.41%, 50=0.08%
  lat (msec)   : 100=0.01%
  cpu          : usr=0.64%, sys=1.61%, ctx=2102892, majf=0, minf=494
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,4138405,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
WRITE: bw=269MiB/s (283MB/s), 269MiB/s-269MiB/s (283MB/s-283MB/s), io=15.8GiB (16.9GB), run=60002-60002msec

Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
RANDREAD: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=8
...
fio-3.27
Starting 32 processes
Jobs: 32 (f=0): [f(32)][100.0%][w=277MiB/s][w=71.0k IOPS][eta 00m:00s]
RANDREAD: (groupid=0, jobs=32): err= 0: pid=29446: Wed Feb 2 00:09:21 2022

  write: IOPS=69.6k, BW=272MiB/s (285MB/s)(15.9GiB/60003msec);
    slat (nsec): min=1052, max=50297k, avg=138125.49, stdev=435715.13
    clat (nsec): min=521, max=59414k, avg=3537010.84, stdev=2183018.58
     lat (usec): min=51, max=61090, avg=3675.34, stdev=2211.08
    clat percentiles (usec):
     |  1.00th=[  570],  5.00th=[ 1029], 10.00th=[ 1352], 20.00th=[ 1827],
     | 30.00th=[ 2245], 40.00th=[ 2671], 50.00th=[ 3064], 60.00th=[ 3556],
     | 70.00th=[ 4146], 80.00th=[ 4948], 90.00th=[ 6259], 95.00th=[ 7570],
     | 99.00th=[10683], 99.50th=[12256], 99.90th=[17433], 99.95th=[21365],
     | 99.99th=[30540]
   bw (  KiB/s): min=206519, max=341031, per=100.00%, avg=278664.74, stdev=817.73, samples=3808
   iops        : min=51622, max=85253, avg=69660.15, stdev=204.42, samples=3808
  lat (nsec)   : 750=0.01%, 1000=0.01%
  lat (usec)   : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.12%
  lat (usec)   : 100=0.03%, 250=0.08%, 500=0.47%, 750=1.43%, 1000=2.51%
  lat (msec)   : 2=19.26%, 4=43.83%, 10=30.86%, 20=1.34%, 50=0.06%
  lat (msec)   : 100=0.01%
  cpu          : usr=0.64%, sys=1.55%, ctx=1803573, majf=0, minf=455
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,4177602,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
WRITE: bw=272MiB/s (285MB/s), 272MiB/s-272MiB/s (285MB/s-285MB/s), io=15.9GiB (17.1GB), run=60003-60003msec

Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
RANDREAD: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=8
...
fio-3.27
Starting 32 processes
Jobs: 32 (f=0): [f(32)][100.0%][w=272MiB/s][w=69.7k IOPS][eta 00m:00s]
RANDREAD: (groupid=0, jobs=32): err= 0: pid=29556: Wed Feb 2 00:10:21 2022

  write: IOPS=69.8k, BW=272MiB/s (286MB/s)(16.0GiB/60003msec);
    slat (nsec): min=1052, max=35550k, avg=79344.28, stdev=323613.65
    clat (nsec): min=771, max=53581k, avg=3588937.03, stdev=2176963.25
     lat (usec): min=58, max=53842, avg=3668.46, stdev=2181.80
    clat percentiles (usec):
     |  1.00th=[  586],  5.00th=[ 1037], 10.00th=[ 1369], 20.00th=[ 1844],
     | 30.00th=[ 2278], 40.00th=[ 2704], 50.00th=[ 3130], 60.00th=[ 3621],
     | 70.00th=[ 4228], 80.00th=[ 5080], 90.00th=[ 6390], 95.00th=[ 7701],
     | 99.00th=[10683], 99.50th=[12125], 99.90th=[16057], 99.95th=[18744],
     | 99.99th=[27395]
   bw (  KiB/s): min=204774, max=339176, per=100.00%, avg=279266.54, stdev=802.82, samples=3808
   iops        : min=51191, max=84786, avg=69810.42, stdev=200.71, samples=3808
  lat (nsec)   : 1000=0.01%
  lat (usec)   : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.13%
  lat (usec)   : 100=0.03%, 250=0.06%, 500=0.44%, 750=1.37%, 1000=2.49%
  lat (msec)   : 2=18.96%, 4=43.01%, 10=32.04%, 20=1.42%, 50=0.04%
  lat (msec)   : 100=0.01%
  cpu          : usr=0.62%, sys=1.45%, ctx=1356660, majf=0, minf=493
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,4185468,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
WRITE: bw=272MiB/s (286MB/s), 272MiB/s-272MiB/s (286MB/s-286MB/s), io=16.0GiB (17.1GB), run=60003-60003msec

Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
RANDREAD: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=8
...
fio-3.27
Starting 32 processes
Jobs: 32 (f=32): [w(32)][98.4%][w=280MiB/s][w=71.7k IOPS][eta 00m:01s]
Jobs: 32 (f=32): [w(32)][100.0%][w=277MiB/s][w=70.9k IOPS][eta 00m:00s]
RANDREAD: (groupid=0, jobs=32): err= 0: pid=29643: Wed Feb 2 00:11:21 2022

  write: IOPS=69.8k, BW=273MiB/s (286MB/s)(16.0GiB/60003msec);
    slat (nsec): min=1052, max=34378k, avg=59061.75, stdev=277238.20
    clat (nsec): min=742, max=47279k, avg=3608224.03, stdev=2171757.46
     lat (usec): min=48, max=47679, avg=3667.45, stdev=2172.85
    clat percentiles (usec):
     |  1.00th=[  594],  5.00th=[ 1045], 10.00th=[ 1369], 20.00th=[ 1860],
     | 30.00th=[ 2278], 40.00th=[ 2704], 50.00th=[ 3130], 60.00th=[ 3654],
     | 70.00th=[ 4293], 80.00th=[ 5080], 90.00th=[ 6390], 95.00th=[ 7701],
     | 99.00th=[10683], 99.50th=[12125], 99.90th=[15664], 99.95th=[17433],
     | 99.99th=[23725]
   bw (  KiB/s): min=201486, max=341296, per=100.00%, avg=279357.72, stdev=825.73, samples=3808
   iops        : min=50369, max=85319, avg=69833.71, stdev=206.44, samples=3808
  lat (nsec)   : 750=0.01%, 1000=0.01%
  lat (usec)   : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.11%
  lat (usec)   : 100=0.03%, 250=0.05%, 500=0.42%, 750=1.37%, 1000=2.47%
  lat (msec)   : 2=18.87%, 4=42.63%, 10=32.55%, 20=1.47%, 50=0.02%
  cpu          : usr=0.60%, sys=1.42%, ctx=1195286, majf=0, minf=481
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,4186642,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
WRITE: bw=273MiB/s (286MB/s), 273MiB/s-273MiB/s (286MB/s-286MB/s), io=16.0GiB (17.1GB), run=60003-60003msec

Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
RANDREAD: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=8
...
fio-3.27
Starting 32 processes
Jobs: 32 (f=0): [f(32)][100.0%][w=272MiB/s][w=69.6k IOPS][eta 00m:00s]
RANDREAD: (groupid=0, jobs=32): err= 0: pid=29748: Wed Feb 2 00:12:22 2022

  write: IOPS=71.5k, BW=279MiB/s (293MB/s)(16.4GiB/60003msec);
    slat (nsec): min=1072, max=25141k, avg=14918.87, stdev=115673.29
    clat (nsec): min=752, max=37983k, avg=3564960.33, stdev=2091487.49
     lat (usec): min=57, max=37985, avg=3580.03, stdev=2089.00
    clat percentiles (usec):
     |  1.00th=[  635],  5.00th=[ 1057], 10.00th=[ 1369], 20.00th=[ 1860],
     | 30.00th=[ 2278], 40.00th=[ 2671], 50.00th=[ 3130], 60.00th=[ 3621],
     | 70.00th=[ 4228], 80.00th=[ 5014], 90.00th=[ 6325], 95.00th=[ 7570],
     | 99.00th=[10421], 99.50th=[11600], 99.90th=[14615], 99.95th=[15926],
     | 99.99th=[18744]
   bw (  KiB/s): min=223459, max=340302, per=100.00%, avg=286170.92, stdev=703.56, samples=3808
   iops        : min=55856, max=85068, avg=71536.67, stdev=175.90, samples=3808
  lat (nsec)   : 1000=0.01%
  lat (usec)   : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.04%
  lat (usec)   : 100=0.02%, 250=0.02%, 500=0.37%, 750=1.33%, 1000=2.48%
  lat (msec)   : 2=19.22%, 4=43.12%, 10=32.14%, 20=1.26%, 50=0.01%
  cpu          : usr=0.61%, sys=1.38%, ctx=872668, majf=0, minf=472
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,4288951,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
WRITE: bw=279MiB/s (293MB/s), 279MiB/s-279MiB/s (293MB/s-293MB/s), io=16.4GiB (17.6GB), run=60003-60003msec

Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
RANDREAD: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=8
...
fio-3.27
Starting 32 processes
Jobs: 2 (f=0): [_(12),f(2),_(18)][100.0%][w=266MiB/s][w=68.1k IOPS][eta 00m:00s]
RANDREAD: (groupid=0, jobs=32): err= 0: pid=29827: Wed Feb 2 00:13:22 2022

  write: IOPS=71.0k, BW=277MiB/s (291MB/s)(16.2GiB/60003msec);
    slat (nsec): min=1062, max=32432k, avg=19937.63, stdev=148066.81
    clat (nsec): min=811, max=59246k, avg=3585601.00, stdev=2115468.91
     lat (usec): min=52, max=59254, avg=3605.70, stdev=2113.21
    clat percentiles (usec):
     |  1.00th=[  635],  5.00th=[ 1074], 10.00th=[ 1385], 20.00th=[ 1860],
     | 30.00th=[ 2278], 40.00th=[ 2704], 50.00th=[ 3130], 60.00th=[ 3621],
     | 70.00th=[ 4228], 80.00th=[ 5014], 90.00th=[ 6390], 95.00th=[ 7635],
     | 99.00th=[10552], 99.50th=[11731], 99.90th=[14877], 99.95th=[16188],
     | 99.99th=[19792]
   bw (  KiB/s): min=222503, max=339437, per=100.00%, avg=284148.08, stdev=711.74, samples=3808
   iops        : min=55619, max=84852, avg=71031.03, stdev=177.94, samples=3808
  lat (nsec)   : 1000=0.01%
  lat (usec)   : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.05%
  lat (usec)   : 100=0.02%, 250=0.02%, 500=0.36%, 750=1.29%, 1000=2.44%
  lat (msec)   : 2=18.96%, 4=43.25%, 10=32.27%, 20=1.33%, 50=0.01%
  lat (msec)   : 100=0.01%
  cpu          : usr=0.61%, sys=1.42%, ctx=918578, majf=0, minf=478
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,4258348,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
WRITE: bw=277MiB/s (291MB/s), 277MiB/s-277MiB/s (291MB/s-291MB/s), io=16.2GiB (17.4GB), run=60003-60003msec

Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
RANDREAD: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
...
fio-3.27
Starting 48 processes
Jobs: 48 (f=48): [r(48)][100.0%][r=295MiB/s][r=75.5k IOPS][eta 00m:00s]
RANDREAD: (groupid=0, jobs=48): err= 0: pid=30964: Wed Feb 2 00:14:22 2022
read: IOPS=72.8k, BW=284MiB/s (298MB/s)(16.7GiB/60037msec)

    slat (nsec): min=971, max=173108k, avg=39325.53, stdev=826441.69
    clat (nsec): min=1894, max=316249k, avg=42166512.42, stdev=14952004.11
     lat (usec): min=417, max=316251, avg=42205.96, stdev=15032.61
    clat percentiles (msec):
     |  1.00th=[    9],  5.00th=[   24], 10.00th=[   28], 20.00th=[   31],
     | 30.00th=[   34], 40.00th=[   41], 50.00th=[   43], 60.00th=[   45],
     | 70.00th=[   47], 80.00th=[   50], 90.00th=[   59], 95.00th=[   68],
     | 99.00th=[   91], 99.50th=[   99], 99.90th=[  121], 99.95th=[  140],
     | 99.99th=[  201]
   bw (  KiB/s): min=142106, max=403311, per=100.00%, avg=291270.21, stdev=902.19, samples=5664
   iops        : min=35512, max=100811, avg=72801.07, stdev=225.55, samples=5664
  lat (usec)   : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
  lat (usec)   : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
  lat (msec)   : 2=0.05%, 4=0.22%, 10=1.02%, 20=3.07%, 50=76.23%
  lat (msec)   : 100=18.97%, 250=0.42%, 500=0.01%
  cpu          : usr=0.24%, sys=0.55%, ctx=261137, majf=0, minf=3757
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=4368373,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
READ: bw=284MiB/s (298MB/s), 284MiB/s-284MiB/s (298MB/s-298MB/s), io=16.7GiB (17.9GB), run=60037-60037msec

Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
RANDREAD: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
...
fio-3.27
Starting 48 processes
Jobs: 48 (f=48): [r(48)][100.0%][r=300MiB/s][r=76.9k IOPS][eta 00m:00s]
RANDREAD: (groupid=0, jobs=48): err= 0: pid=31076: Wed Feb 2 00:15:23 2022
read: IOPS=71.5k, BW=279MiB/s (293MB/s)(16.4GiB/60037msec)

    slat (nsec): min=961, max=88836k, avg=17391.79, stdev=551047.29
    clat (usec): min=22, max=221605, avg=42951.57, stdev=12630.75
     lat (usec): min=383, max=222644, avg=42969.08, stdev=12673.43
    clat percentiles (msec):
     |  1.00th=[   14],  5.00th=[   27], 10.00th=[   29], 20.00th=[   32],
     | 30.00th=[   39], 40.00th=[   42], 50.00th=[   44], 60.00th=[   45],
     | 70.00th=[   47], 80.00th=[   51], 90.00th=[   58], 95.00th=[   63],
     | 99.00th=[   84], 99.50th=[   93], 99.90th=[  112], 99.95th=[  123],
     | 99.99th=[  150]
   bw (  KiB/s): min=170354, max=369108, per=100.00%, avg=286058.08, stdev=686.26, samples=5664
   iops        : min=42573, max=92262, avg=71497.39, stdev=171.56, samples=5664
  lat (usec)   : 50=0.01%, 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%
  lat (usec)   : 1000=0.01%
  lat (msec)   : 2=0.02%, 4=0.06%, 10=0.36%, 20=1.78%, 50=77.37%
  lat (msec)   : 100=20.13%, 250=0.27%
  cpu          : usr=0.22%, sys=0.53%, ctx=218767, majf=0, minf=3743
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=4290648,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
READ: bw=279MiB/s (293MB/s), 279MiB/s-279MiB/s (293MB/s-293MB/s), io=16.4GiB (17.6GB), run=60037-60037msec

Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
RANDREAD: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
...
fio-3.27
Starting 48 processes
Jobs: 48 (f=48): [r(48)][100.0%][r=299MiB/s][r=76.4k IOPS][eta 00m:00s]
RANDREAD: (groupid=0, jobs=48): err= 0: pid=31174: Wed Feb 2 00:16:23 2022
read: IOPS=67.5k, BW=264MiB/s (277MB/s)(15.5GiB/60033msec)

    slat (nsec): min=941, max=113553k, avg=20605.09, stdev=608598.15
    clat (usec): min=20, max=244312, avg=45449.46, stdev=13495.74
     lat (usec): min=78, max=244315, avg=45470.18, stdev=13549.72
    clat percentiles (msec):
     |  1.00th=[   14],  5.00th=[   28], 10.00th=[   30], 20.00th=[   34],
     | 30.00th=[   42], 40.00th=[   44], 50.00th=[   45], 60.00th=[   47],
     | 70.00th=[   50], 80.00th=[   55], 90.00th=[   61], 95.00th=[   66],
     | 99.00th=[   92], 99.50th=[  101], 99.90th=[  120], 99.95th=[  129],
     | 99.99th=[  163]
   bw (  KiB/s): min=147119, max=347537, per=100.00%, avg=270244.42, stdev=628.01, samples=5664
   iops        : min=36762, max=86866, avg=67544.04, stdev=157.00, samples=5664
  lat (usec)   : 50=0.01%, 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%
  lat (usec)   : 1000=0.01%
  lat (msec)   : 2=0.02%, 4=0.08%, 10=0.40%, 20=1.57%, 50=70.55%
  lat (msec)   : 100=26.85%, 250=0.52%
  cpu          : usr=0.21%, sys=0.51%, ctx=217032, majf=0, minf=3833
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=4054701,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
READ: bw=264MiB/s (277MB/s), 264MiB/s-264MiB/s (277MB/s-277MB/s), io=15.5GiB (16.6GB), run=60033-60033msec

Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
RANDREAD: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
...
fio-3.27
Starting 48 processes
Jobs: 48 (f=48): [r(48)][100.0%][r=296MiB/s][r=75.8k IOPS][eta 00m:00s]
RANDREAD: (groupid=0, jobs=48): err= 0: pid=31271: Wed Feb 2 00:17:23 2022
read: IOPS=74.5k, BW=291MiB/s (305MB/s)(17.1GiB/60035msec)

    slat (nsec): min=952, max=94814k, avg=15742.78, stdev=457046.68
    clat (usec): min=38, max=202717, avg=41217.82, stdev=11676.36
     lat (usec): min=65, max=202721, avg=41233.69, stdev=11708.54
    clat percentiles (msec):
     |  1.00th=[   15],  5.00th=[   27], 10.00th=[   29], 20.00th=[   31],
     | 30.00th=[   34], 40.00th=[   41], 50.00th=[   43], 60.00th=[   44],
     | 70.00th=[   46], 80.00th=[   48], 90.00th=[   56], 95.00th=[   61],
     | 99.00th=[   79], 99.50th=[   88], 99.90th=[  105], 99.95th=[  112],
     | 99.99th=[  140]
   bw (  KiB/s): min=182011, max=362829, per=100.00%, avg=298134.45, stdev=603.05, samples=5664
   iops        : min=45487, max=90694, avg=74515.97, stdev=150.76, samples=5664
  lat (usec)   : 50=0.01%, 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%
  lat (usec)   : 1000=0.01%
  lat (msec)   : 2=0.01%, 4=0.06%, 10=0.30%, 20=1.80%, 50=82.63%
  lat (msec)   : 100=15.05%, 250=0.14%
  cpu          : usr=0.23%, sys=0.55%, ctx=232008, majf=0, minf=3740
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=4471696,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
READ: bw=291MiB/s (305MB/s), 291MiB/s-291MiB/s (305MB/s-305MB/s), io=17.1GiB (18.3GB), run=60035-60035msec

Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
RANDREAD: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
...
fio-3.27
Starting 48 processes
Jobs: 48 (f=48): [r(48)][100.0%][r=294MiB/s][r=75.2k IOPS][eta 00m:00s]
RANDREAD: (groupid=0, jobs=48): err= 0: pid=31366: Wed Feb 2 00:18:23 2022
read: IOPS=72.4k, BW=283MiB/s (296MB/s)(16.6GiB/60036msec)

    slat (nsec): min=952, max=131203k, avg=21974.73, stdev=596139.49
    clat (usec): min=3, max=270950, avg=42409.20, stdev=13195.67
     lat (usec): min=80, max=270961, avg=42431.29, stdev=13247.16
    clat percentiles (msec):
     |  1.00th=[   12],  5.00th=[   27], 10.00th=[   28], 20.00th=[   31],
     | 30.00th=[   36], 40.00th=[   42], 50.00th=[   43], 60.00th=[   45],
     | 70.00th=[   47], 80.00th=[   50], 90.00th=[   58], 95.00th=[   64],
     | 99.00th=[   86], 99.50th=[   93], 99.90th=[  109], 99.95th=[  120],
     | 99.99th=[  161]
   bw (  KiB/s): min=163161, max=388414, per=100.00%, avg=289753.58, stdev=752.52, samples=5664
   iops        : min=40771, max=97088, avg=72421.87, stdev=188.12, samples=5664
  lat (usec)   : 4=0.01%, 50=0.01%, 100=0.01%, 250=0.01%, 500=0.01%
  lat (usec)   : 750=0.01%, 1000=0.01%
  lat (msec)   : 2=0.03%, 4=0.14%, 10=0.67%, 20=2.25%, 50=77.40%
  lat (msec)   : 100=19.26%, 250=0.25%, 500=0.01%
  cpu          : usr=0.23%, sys=0.53%, ctx=232559, majf=0, minf=3758
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=4345429,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
READ: bw=283MiB/s (296MB/s), 283MiB/s-283MiB/s (296MB/s-296MB/s), io=16.6GiB (17.8GB), run=60036-60036msec

Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
RANDREAD: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
...
fio-3.27
Starting 48 processes
Jobs: 48 (f=48): [r(48)][100.0%][r=288MiB/s][r=73.8k IOPS][eta 00m:00s]
RANDREAD: (groupid=0, jobs=48): err= 0: pid=32520: Wed Feb 2 00:19:24 2022
read: IOPS=71.8k, BW=281MiB/s (294MB/s)(16.5GiB/60036msec)

    slat (nsec): min=961, max=155112k, avg=23329.58, stdev=600906.79
    clat (usec): min=18, max=269118, avg=42721.27, stdev=13343.53
     lat (usec): min=109, max=269121, avg=42744.72, stdev=13395.21
    clat percentiles (msec):
     |  1.00th=[   13],  5.00th=[   27], 10.00th=[   29], 20.00th=[   31],
     | 30.00th=[   36], 40.00th=[   42], 50.00th=[   44], 60.00th=[   45],
     | 70.00th=[   47], 80.00th=[   51], 90.00th=[   58], 95.00th=[   64],
     | 99.00th=[   88], 99.50th=[   96], 99.90th=[  113], 99.95th=[  125],
     | 99.99th=[  176]
   bw (  KiB/s): min=155002, max=377376, per=100.00%, avg=287593.81, stdev=736.00, samples=5664
   iops        : min=38739, max=94329, avg=71881.32, stdev=184.00, samples=5664
  lat (usec)   : 20=0.01%, 50=0.01%, 100=0.01%, 250=0.01%, 500=0.01%
  lat (usec)   : 750=0.01%, 1000=0.01%
  lat (msec)   : 2=0.02%, 4=0.10%, 10=0.54%, 20=2.09%, 50=77.36%
  lat (msec)   : 100=19.56%, 250=0.33%, 500=0.01%
  cpu          : usr=0.22%, sys=0.54%, ctx=235545, majf=0, minf=3694
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=4313415,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
READ: bw=281MiB/s (294MB/s), 281MiB/s-281MiB/s (294MB/s-294MB/s), io=16.5GiB (17.7GB), run=60036-60036msec

Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
RANDREAD: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
...
fio-3.27
Starting 48 processes
Jobs: 48 (f=48): [r(48)][100.0%][r=291MiB/s][r=74.6k IOPS][eta 00m:00s]
RANDREAD: (groupid=0, jobs=48): err= 0: pid=32630: Wed Feb 2 00:20:24 2022
read: IOPS=73.7k, BW=288MiB/s (302MB/s)(16.9GiB/60035msec)

    slat (nsec): min=962, max=177671k, avg=38019.32, stdev=644096.49
    clat (usec): min=29, max=287028, avg=41615.50, stdev=13238.93
     lat (usec): min=94, max=319545, avg=41653.64, stdev=13291.04
    clat percentiles (msec):
     |  1.00th=[   12],  5.00th=[   26], 10.00th=[   28], 20.00th=[   31],
     | 30.00th=[   34], 40.00th=[   41], 50.00th=[   43], 60.00th=[   45],
     | 70.00th=[   46], 80.00th=[   49], 90.00th=[   57], 95.00th=[   63],
     | 99.00th=[   88], 99.50th=[   95], 99.90th=[  112], 99.95th=[  123],
     | 99.99th=[  176]
   bw (  KiB/s): min=156946, max=373773, per=100.00%, avg=295110.41, stdev=718.25, samples=5664
   iops        : min=39217, max=93429, avg=73760.01, stdev=179.56, samples=5664
  lat (usec)   : 50=0.01%, 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%
  lat (usec)   : 1000=0.01%
  lat (msec)   : 2=0.02%, 4=0.11%, 10=0.60%, 20=2.37%, 50=80.04%
  lat (msec)   : 100=16.54%, 250=0.31%, 500=0.01%
  cpu          : usr=0.23%, sys=0.56%, ctx=327850, majf=0, minf=3771
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=4426283,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
READ: bw=288MiB/s (302MB/s), 288MiB/s-288MiB/s (302MB/s-302MB/s), io=16.9GiB (18.1GB), run=60035-60035msec

Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
RANDREAD: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
...
fio-3.27
Starting 48 processes
Jobs: 48 (f=48): [r(48)][100.0%][r=292MiB/s][r=74.6k IOPS][eta 00m:00s]
RANDREAD: (groupid=0, jobs=48): err= 0: pid=32752: Wed Feb 2 00:21:24 2022
read: IOPS=72.3k, BW=283MiB/s (296MB/s)(16.6GiB/60039msec)

    slat (nsec): min=972, max=125387k, avg=20090.72, stdev=496977.06
    clat (nsec): min=1563, max=210556k, avg=42435419.14, stdev=12289751.96
     lat (usec): min=123, max=210559, avg=42455.63, stdev=12326.24
    clat percentiles (msec):
     |  1.00th=[   14],  5.00th=[   27], 10.00th=[   29], 20.00th=[   31],
     | 30.00th=[   36], 40.00th=[   42], 50.00th=[   44], 60.00th=[   45],
     | 70.00th=[   47], 80.00th=[   50], 90.00th=[   58], 95.00th=[   63],
     | 99.00th=[   82], 99.50th=[   91], 99.90th=[  108], 99.95th=[  116],
     | 99.99th=[  146]
   bw (  KiB/s): min=169789, max=367924, per=100.00%, avg=289626.85, stdev=669.34, samples=5664
   iops        : min=42433, max=91970, avg=72388.99, stdev=167.32, samples=5664
  lat (usec)   : 2=0.01%, 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%
  lat (usec)   : 1000=0.01%
  lat (msec)   : 2=0.02%, 4=0.07%, 10=0.34%, 20=1.74%, 50=79.17%
  lat (msec)   : 100=18.44%, 250=0.22%
  cpu          : usr=0.23%, sys=0.52%, ctx=248980, majf=0, minf=3751
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=4343055,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
READ: bw=283MiB/s (296MB/s), 283MiB/s-283MiB/s (296MB/s-296MB/s), io=16.6GiB (17.8GB), run=60039-60039msec

Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
RANDREAD: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
...
fio-3.27
Starting 48 processes
Jobs: 48 (f=48): [r(48)][100.0%][r=294MiB/s][r=75.3k IOPS][eta 00m:00s]
RANDREAD: (groupid=0, jobs=48): err= 0: pid=32851: Wed Feb 2 00:22:25 2022
read: IOPS=72.8k, BW=285MiB/s (298MB/s)(16.7GiB/60037msec)

    slat (nsec): min=961, max=88402k, avg=16580.98, stdev=499828.64
    clat (usec): min=16, max=190979, avg=42147.65, stdev=12291.43
     lat (usec): min=76, max=200180, avg=42164.34, stdev=12331.85
    clat percentiles (msec):
     |  1.00th=[   14],  5.00th=[   27], 10.00th=[   29], 20.00th=[   31],
     | 30.00th=[   36], 40.00th=[   42], 50.00th=[   43], 60.00th=[   45],
     | 70.00th=[   46], 80.00th=[   50], 90.00th=[   57], 95.00th=[   62],
     | 99.00th=[   83], 99.50th=[   92], 99.90th=[  108], 99.95th=[  116],
     | 99.99th=[  142]
   bw (  KiB/s): min=170567, max=369399, per=100.00%, avg=291617.48, stdev=663.74, samples=5664
   iops        : min=42628, max=92333, avg=72887.66, stdev=165.93, samples=5664
  lat (usec)   : 20=0.01%, 50=0.01%, 100=0.01%, 250=0.01%, 500=0.01%
  lat (usec)   : 750=0.01%, 1000=0.01%
  lat (msec)   : 2=0.01%, 4=0.07%, 10=0.39%, 20=1.84%, 50=79.83%
  lat (msec)   : 100=17.63%, 250=0.22%
  cpu          : usr=0.23%, sys=0.53%, ctx=222791, majf=0, minf=3753
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=4372949,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
READ: bw=285MiB/s (298MB/s), 285MiB/s-285MiB/s (298MB/s-298MB/s), io=16.7GiB (17.9GB), run=60037-60037msec

Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
RANDREAD: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
...
fio-3.27
Starting 48 processes
Jobs: 48 (f=48): [r(48)][100.0%][r=296MiB/s][r=75.7k IOPS][eta 00m:00s]
RANDREAD: (groupid=0, jobs=48): err= 0: pid=32948: Wed Feb 2 00:23:25 2022
read: IOPS=74.6k, BW=291MiB/s (306MB/s)(17.1GiB/60036msec)

    slat (nsec): min=982, max=143449k, avg=14518.04, stdev=453275.74
    clat (usec): min=39, max=211504, avg=41151.34, stdev=11534.04
     lat (usec): min=227, max=230931, avg=41165.97, stdev=11567.45
    clat percentiles (msec):
     |  1.00th=[   15],  5.00th=[   27], 10.00th=[   29], 20.00th=[   31],
     | 30.00th=[   34], 40.00th=[   41], 50.00th=[   43], 60.00th=[   44],
     | 70.00th=[   46], 80.00th=[   48], 90.00th=[   55], 95.00th=[   60],
     | 99.00th=[   79], 99.50th=[   87], 99.90th=[  103], 99.95th=[  112],
     | 99.99th=[  148]
   bw (  KiB/s): min=182147, max=368337, per=100.00%, avg=298660.44, stdev=576.72, samples=5664
   iops        : min=45520, max=92066, avg=74647.02, stdev=144.17, samples=5664
  lat (usec)   : 50=0.01%, 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%
  lat (usec)   : 1000=0.01%
  lat (msec)   : 2=0.01%, 4=0.07%, 10=0.33%, 20=1.75%, 50=83.09%
  lat (msec)   : 100=14.62%, 250=0.13%
  cpu          : usr=0.23%, sys=0.54%, ctx=233134, majf=0, minf=3732
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=4479122,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
READ: bw=291MiB/s (306MB/s), 291MiB/s-291MiB/s (306MB/s-306MB/s), io=17.1GiB (18.3GB), run=60036-60036msec

Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%

Signature · 1 line
2.29.0