nvmet: add buffered_io support for bdev-ns
Subsystems (2)
v2 cover nvmet: add buffered_io support for bdev-ns
From: Chaitanya Kulkarni <[email protected]>
Hi,
In current implementation like NVMeOF target file backed namespace we don't support buffered_io for block device backed namesapce. This allows user to configure block device backend in buffered_io mode so user can get the benefits of caching.
Please note that using this combination host needs to periodically sync between cache and storage or set the system cache parameters accordingly. NVMeOF target is not responsible for data consistency between cache and storage.
fio data verification, rand read/write performance numbers :-
1. NVMeOF Target ns device_path=/dev/nvme0n1 & buffered_io=0 :-
fio Verify job :-
----------------------
read: IOPS=27.2k, BW=106MiB/s (111MB/s)(506MiB/4762msec)
write: IOPS=24.9k, BW=97.3MiB/s (102MB/s)(800MiB/8223msec); 0 zone resets
fio Random Write :-
----------------------
write: IOPS=70.6k, BW=276MiB/s (289MB/s)(16.2GiB/60003msec); 0 zone resets
write: IOPS=70.5k, BW=275MiB/s (289MB/s)(16.1GiB/60003msec); 0 zone resets
write: IOPS=70.8k, BW=277MiB/s (290MB/s)(16.2GiB/60003msec); 0 zone resets
write: IOPS=70.8k, BW=277MiB/s (290MB/s)(16.2GiB/60003msec); 0 zone resets
write: IOPS=70.8k, BW=276MiB/s (290MB/s)(16.2GiB/60003msec); 0 zone resets
write: IOPS=70.9k, BW=277MiB/s (291MB/s)(16.2GiB/60003msec); 0 zone resets
write: IOPS=70.4k, BW=275MiB/s (288MB/s)(16.1GiB/60003msec); 0 zone resets
write: IOPS=70.6k, BW=276MiB/s (289MB/s)(16.2GiB/60003msec); 0 zone resets
write: IOPS=70.4k, BW=275MiB/s (288MB/s)(16.1GiB/60003msec); 0 zone resets
write: IOPS=70.5k, BW=275MiB/s (289MB/s)(16.1GiB/60002msec); 0 zone resets
fio Random Read :-
----------------------
read: IOPS=70.7k, BW=276MiB/s (290MB/s)(16.2GiB/60035msec)
read: IOPS=71.6k, BW=280MiB/s (293MB/s)(16.4GiB/60039msec)
read: IOPS=68.4k, BW=267MiB/s (280MB/s)(15.7GiB/60037msec)
read: IOPS=67.1k, BW=262MiB/s (275MB/s)(15.4GiB/60036msec)
read: IOPS=72.2k, BW=282MiB/s (296MB/s)(16.5GiB/60036msec)
read: IOPS=64.9k, BW=254MiB/s (266MB/s)(14.9GiB/60041msec)
read: IOPS=64.9k, BW=254MiB/s (266MB/s)(14.9GiB/60041msec)
read: IOPS=64.9k, BW=254MiB/s (266MB/s)(14.9GiB/60045msec)
read: IOPS=69.3k, BW=271MiB/s (284MB/s)(15.9GiB/60037msec)
read: IOPS=64.7k, BW=253MiB/s (265MB/s)(14.8GiB/60041msec)
2. NVMeOF Target ns device_path=/dev/nvme0n1 & buffered_io=1 :-
fio Verify job :-
----------------------
read: IOPS=197k, BW=770MiB/s (808MB/s)(506MiB/657msec)
write: IOPS=137k, BW=536MiB/s (562MB/s)(800MiB/1493msec); 0 zone resets
fio Random Write :-
----------------------
write: IOPS=4239k, BW=16.2GiB/s (17.4GB/s)(970GiB/60002msec); 0 zone resets
write: IOPS=4245k, BW=16.2GiB/s (17.4GB/s)(972GiB/60002msec); 0 zone resets
write: IOPS=4202k, BW=16.0GiB/s (17.2GB/s)(962GiB/60002msec); 0 zone resets
write: IOPS=4183k, BW=16.0GiB/s (17.1GB/s)(957GiB/60002msec); 0 zone resets
write: IOPS=4194k, BW=16.0GiB/s (17.2GB/s)(960GiB/60002msec); 0 zone resets
write: IOPS=4209k, BW=16.1GiB/s (17.2GB/s)(963GiB/60002msec); 0 zone resets
write: IOPS=4164k, BW=15.9GiB/s (17.1GB/s)(953GiB/60002msec); 0 zone resets
write: IOPS=4204k, BW=16.0GiB/s (17.2GB/s)(962GiB/60002msec); 0 zone resets
write: IOPS=4206k, BW=16.0GiB/s (17.2GB/s)(963GiB/60002msec); 0 zone resets
write: IOPS=4185k, BW=16.0GiB/s (17.1GB/s)(958GiB/60002msec); 0 zone resets
fio Random Read :-
----------------------
read: IOPS=4849k, BW=18.5GiB/s (19.9GB/s)(1110GiB/60002msec)
read: IOPS=4951k, BW=18.9GiB/s (20.3GB/s)(1133GiB/60002msec)
read: IOPS=5165k, BW=19.7GiB/s (21.2GB/s)(1182GiB/60001msec)
read: IOPS=5100k, BW=19.5GiB/s (20.9GB/s)(1167GiB/60001msec)
read: IOPS=5294k, BW=20.2GiB/s (21.7GB/s)(1212GiB/60002msec)
read: IOPS=5044k, BW=19.2GiB/s (20.7GB/s)(1155GiB/60002msec)
read: IOPS=5084k, BW=19.4GiB/s (20.8GB/s)(1164GiB/60002msec)
read: IOPS=4916k, BW=18.8GiB/s (20.1GB/s)(1125GiB/60001msec)
read: IOPS=5178k, BW=19.8GiB/s (21.2GB/s)(1185GiB/60002msec)
read: IOPS=5027k, BW=19.2GiB/s (20.6GB/s)(1151GiB/60001msec)
-ck
changes from v1:
1. Remove first two patches and don't check for stat mode for block device.
2. use i_size_read() to get the size of the file.
Chaitanya Kulkarni (2):
nvmet: use i_size_read() to set size for file-ns
nvmet: allow bdev in buffered_io mode
drivers/nvme/target/io-cmd-bdev.c | 3 +++ drivers/nvme/target/io-cmd-file.c | 17 ++++------------- drivers/nvme/target/nvmet.h | 2 +- 3 files changed, 8 insertions(+), 14 deletions(-)
root@dev nvme (nvme-5.17) # ./bdev-buffered-io-test.sh
+ NQN=testnqn
+ nvme disconnect -n testnqn
NQN:testnqn disconnected 1 controller(s)
real 0m0.334s
user 0m0.000s
sys 0m0.006s
+ for i in /sys/kernel/config/nvmet/subsystems/${NQN}/namespaces/*
+ echo 0
+ rmdir /sys/kernel/config/nvmet/subsystems/testnqn/namespaces/1
+ rm -fr /sys/kernel/config/nvmet/ports/1/subsystems/testnqn
+ rmdir /sys/kernel/config/nvmet/ports/1
+ rmdir /sys/kernel/config/nvmet/subsystems/testnqn
+ rmdir 'config/nullb/nullb*'
rmdir: failed to remove 'config/nullb/nullb*': No such file or directory
+ umount /mnt/nvme0n1
umount: /mnt/nvme0n1: no mount point specified.
+ umount /mnt/backend
umount: /mnt/backend: not mounted.
+ modprobe -r nvme_loop
+ modprobe -r nvme_fabrics
+ modprobe -r nvmet
+ modprobe -r nvme
+ modprobe -r null_blk
+ tree /sys/kernel/config/sys/kernel/config
0 directories, 0 files
+ umount /mnt/nvme0n1
+ clear_dmesg
./compile_nvme.sh: line 3: clear_dmesg: command not found
umount: /mnt/nvme0n1: no mount point specified.
+ ./delete.sh 1
+ NQN=testnqn
+ nvme disconnect -n testnqn
Failed to disconnect by NQN: testnqn
real 0m0.001s
user 0m0.000s
sys 0m0.001s
+ for i in /sys/kernel/config/nvmet/subsystems/${NQN}/namespaces/*
+ echo 0
./delete.sh: line 9: /sys/kernel/config/nvmet/subsystems/testnqn/namespaces/*/enable: No such file or directory
+ rmdir '/sys/kernel/config/nvmet/subsystems/testnqn/namespaces/*'
rmdir: failed to remove '/sys/kernel/config/nvmet/subsystems/testnqn/namespaces/*': No such file or directory
+ rm -fr /sys/kernel/config/nvmet/ports/1/subsystems/testnqn
+ rmdir /sys/kernel/config/nvmet/ports/1
rmdir: failed to remove '/sys/kernel/config/nvmet/ports/1': No such file or directory
+ rmdir /sys/kernel/config/nvmet/subsystems/testnqn
rmdir: failed to remove '/sys/kernel/config/nvmet/subsystems/testnqn': No such file or directory
+ rmdir 'config/nullb/nullb*'
rmdir: failed to remove 'config/nullb/nullb*': No such file or directory
+ umount /mnt/nvme0n1
umount: /mnt/nvme0n1: no mount point specified.
+ umount /mnt/backend
umount: /mnt/backend: not mounted.
+ modprobe -r nvme_loop
+ modprobe -r nvme_fabrics
+ modprobe -r nvmet
+ modprobe -r nvme
+ modprobe -r null_blk
+ tree /sys/kernel/config/sys/kernel/config
0 directories, 0 files
+ modprobe -r nvme-fabrics
+ modprobe -r nvme_loop
+ modprobe -r nvmet
+ modprobe -r nvme
+ sleep 1
+ modprobe -r nvme-core
+ lsmod
+ grep nvme
+ sleep 1
+ git diff
+ sleep 1
++ nproc
+ make -j 48 M=drivers/nvme/ modules
CC [M] drivers/nvme/host/core.o CC [M] drivers/nvme/target/core.o CC [M] drivers/nvme/target/configfs.o CC [M] drivers/nvme/target/admin-cmd.o CC [M] drivers/nvme/target/io-cmd-file.o CC [M] drivers/nvme/target/io-cmd-bdev.o LD [M] drivers/nvme/target/nvmet.o LD [M] drivers/nvme/host/nvme-core.o
MODPOST drivers/nvme/Module.symvers
CC [M] drivers/nvme/host/nvme-core.mod.o
LD [M] drivers/nvme/target/nvmet.ko
BTF [M] drivers/nvme/target/nvmet.ko
LD [M] drivers/nvme/host/nvme-core.ko
BTF [M] drivers/nvme/host/nvme-core.ko
+ HOST=drivers/nvme/host
+ TARGET=drivers/nvme/target
++ uname -r
+ HOST_DEST=/lib/modules/5.17.0-rc1nvme+/kernel/drivers/nvme/host/
++ uname -r
+ TARGET_DEST=/lib/modules/5.17.0-rc1nvme+/kernel/drivers/nvme/target/
+ cp drivers/nvme/host/nvme-core.ko drivers/nvme/host/nvme-fabrics.ko drivers/nvme/host/nvme-fc.ko drivers/nvme/host/nvme.ko drivers/nvme/host/nvme-rdma.ko drivers/nvme/host/nvme-tcp.ko /lib/modules/5.17.0-rc1nvme+/kernel/drivers/nvme/host//
+ cp drivers/nvme/target/nvme-fcloop.ko drivers/nvme/target/nvme-loop.ko drivers/nvme/target/nvmet-fc.ko drivers/nvme/target/nvmet.ko drivers/nvme/target/nvmet-rdma.ko drivers/nvme/target/nvmet-tcp.ko /lib/modules/5.17.0-rc1nvme+/kernel/drivers/nvme/target//
+ ls -lrth /lib/modules/5.17.0-rc1nvme+/kernel/drivers/nvme/host/ /lib/modules/5.17.0-rc1nvme+/kernel/drivers/nvme/target///lib/modules/5.17.0-rc1nvme+/kernel/drivers/nvme/host/:
total 6.7M
-rw-r--r--. 1 root root 2.8M Jan 27 17:52 nvme-core.ko
-rw-r--r--. 1 root root 448K Jan 27 17:52 nvme-fabrics.ko
-rw-r--r--. 1 root root 1015K Jan 27 17:52 nvme-fc.ko
-rw-r--r--. 1 root root 745K Jan 27 17:52 nvme.ko -rw-r--r--. 1 root root 958K Jan 27 17:52 nvme-rdma.ko -rw-r--r--. 1 root root 902K Jan 27 17:52 nvme-tcp.ko
/lib/modules/5.17.0-rc1nvme+/kernel/drivers/nvme/target//:
total 6.7M
-rw-r--r--. 1 root root 498K Jan 27 17:52 nvme-fcloop.ko -rw-r--r--. 1 root root 439K Jan 27 17:52 nvme-loop.ko -rw-r--r--. 1 root root 768K Jan 27 17:52 nvmet-fc.ko -rw-r--r--. 1 root root 3.3M Jan 27 17:52 nvmet.ko -rw-r--r--. 1 root root 929K Jan 27 17:52 nvmet-rdma.ko -rw-r--r--. 1 root root 775K Jan 27 17:52 nvmet-tcp.ko
+ modprobe nvme
+ NQN=testnqn
+ nvme disconnect -n testnqn
NQN:testnqn disconnected 0 controller(s)
real 0m0.001s
user 0m0.000s
sys 0m0.001s
+ for i in /sys/kernel/config/nvmet/subsystems/${NQN}/namespaces/*
+ echo 0
./delete.sh: line 9: /sys/kernel/config/nvmet/subsystems/testnqn/namespaces/*/enable: No such file or directory
+ rmdir '/sys/kernel/config/nvmet/subsystems/testnqn/namespaces/*'
rmdir: failed to remove '/sys/kernel/config/nvmet/subsystems/testnqn/namespaces/*': No such file or directory
+ rm -fr /sys/kernel/config/nvmet/ports/1/subsystems/testnqn
+ rmdir /sys/kernel/config/nvmet/ports/1
rmdir: failed to remove '/sys/kernel/config/nvmet/ports/1': No such file or directory
+ rmdir /sys/kernel/config/nvmet/subsystems/testnqn
rmdir: failed to remove '/sys/kernel/config/nvmet/subsystems/testnqn': No such file or directory
+ rmdir 'config/nullb/nullb*'
rmdir: failed to remove 'config/nullb/nullb*': No such file or directory
+ umount /mnt/nvme0n1
umount: /mnt/nvme0n1: no mount point specified.
+ umount /mnt/backend
umount: /mnt/backend: not mounted.
+ modprobe -r nvme_loop
+ modprobe -r nvme_fabrics
+ modprobe -r nvmet
+ modprobe -r nvme
+ modprobe -r null_blk
+ tree /sys/kernel/config/sys/kernel/config
0 directories, 0 files
+++ NN=1
+++ NQN=testnqn
+++ let NR_DEVICES=NN+1
+++ modprobe -r null_blk
+++ modprobe -r nvme
+++ modprobe null_blk nr_devices=0
+++ modprobe nvme
+++ modprobe nvme-fabrics
+++ modprobe nvmet
+++ modprobe nvme-loop
+++ dmesg -c
+++ sleep 2
+++ tree /sys/kernel/config/sys/kernel/config
├── nullb
│ └── features
└── nvmet
├── hosts
├── ports
└── subsystems5 directories, 1 file
+++ mkdir /sys/kernel/config/nvmet/subsystems/testnqn
+++ mkdir /sys/kernel/config/nvmet/subsystems/testnqn/namespaces/1
+++ echo -n /dev/nvme0n1
+++ cat /sys/kernel/config/nvmet/subsystems/testnqn/namespaces/1/device_path /dev/nvme0n1
+++ echo 1
+++ cat /sys/kernel/config/nvmet/subsystems/testnqn/namespaces/1/buffered_io
1
+++ echo 1
+++ mkdir /sys/kernel/config/nvmet/ports/1/
+++ echo -n loop
+++ echo -n 1
+++ ln -s /sys/kernel/config/nvmet/subsystems/testnqn /sys/kernel/config/nvmet/ports/1/subsystems/ +++ sleep 1
+++ echo transport=loop,nqn=testnqn
+++ sleep 1
+++ mount
+++ column -t
+++ grep nvme
+++ dmesg -c
[ 659.631342] nvme nvme0: 48/0/0 default/read/poll queues [ 661.614910] nvmet: adding nsid 1 to subsystem testnqn [ 662.625653] nvmet: creating nvm controller 1 for subsystem testnqn for NQN nqn.2014-08.org.nvmexpress:uuid:8443e545-5a0b-4738-bf56-fcbf0cd13320. [ 662.625773] nvme nvme1: creating 48 I/O queues. [ 662.630443] nvme nvme1: new ctrl: "testnqn"
write-and-verify: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=4 fio-3.27
Starting 1 process
Jobs: 1 (f=1)
write-and-verify: (groupid=0, jobs=1): err= 0: pid=5346: Thu Jan 27 17:52:48 2022
read: IOPS=197k, BW=770MiB/s (808MB/s)(506MiB/657msec)
slat (usec): min=3, max=209, avg= 3.92, stdev= 1.39
clat (nsec): min=1092, max=231500, avg=15464.34, stdev=4454.50
lat (usec): min=4, max=236, avg=19.45, stdev= 5.50
clat percentiles (nsec):
| 1.00th=[13120], 5.00th=[13248], 10.00th=[13376], 20.00th=[13504],
| 30.00th=[13632], 40.00th=[13760], 50.00th=[13760], 60.00th=[13888],
| 70.00th=[14016], 80.00th=[14400], 90.00th=[24192], 95.00th=[26240],
| 99.00th=[29312], 99.50th=[33024], 99.90th=[39680], 99.95th=[42752],
| 99.99th=[72192]
write: IOPS=137k, BW=536MiB/s (562MB/s)(800MiB/1493msec); 0 zone resets
slat (usec): min=4, max=430, avg= 6.59, stdev= 2.37
clat (nsec): min=781, max=504027, avg=22173.67, stdev=6641.70
lat (usec): min=6, max=516, avg=28.83, stdev= 8.39
clat percentiles (nsec):
| 1.00th=[17536], 5.00th=[17792], 10.00th=[18048], 20.00th=[18560],
| 30.00th=[19072], 40.00th=[19328], 50.00th=[19584], 60.00th=[20096],
| 70.00th=[20608], 80.00th=[22656], 90.00th=[34048], 95.00th=[36608],
| 99.00th=[43776], 99.50th=[48384], 99.90th=[56064], 99.95th=[62720],
| 99.99th=[98816]
bw ( KiB/s): min=510648, max=585552, per=99.53%, avg=546133.33, stdev=37606.59, samples=3
iops : min=127664, max=146386, avg=136533.33, stdev=9399.66, samples=3
lat (nsec) : 1000=0.01%
lat (usec) : 2=0.01%, 10=0.01%, 20=69.00%, 50=30.78%, 100=0.21%
lat (usec) : 250=0.01%, 500=0.01%, 750=0.01%
cpu : usr=27.41%, sys=32.90%, ctx=333175, majf=0, minf=3054
IO depths : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=129572,204800,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4Run status group 0 (all jobs):
READ: bw=770MiB/s (808MB/s), 770MiB/s-770MiB/s (808MB/s-808MB/s), io=506MiB (531MB), run=657-657msec
WRITE: bw=536MiB/s (562MB/s), 536MiB/s-536MiB/s (562MB/s-562MB/s), io=800MiB (839MB), run=1493-1493msec
Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
RANDREAD: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=8
...
fio-3.27
Starting 32 processes
Jobs: 32 (f=32): [w(32)][100.0%][w=16.0GiB/s][w=4204k IOPS][eta 00m:00s]
RANDREAD: (groupid=0, jobs=32): err= 0: pid=5350: Thu Jan 27 17:53:48 2022
write: IOPS=4239k, BW=16.2GiB/s (17.4GB/s)(970GiB/60002msec); 0 zone resets
slat (nsec): min=1052, max=6298.9k, avg=6783.36, stdev=3723.70
clat (nsec): min=681, max=6390.7k, avg=53142.21, stdev=17451.81
lat (usec): min=5, max=6400, avg=60.00, stdev=19.41
clat percentiles (usec):
| 1.00th=[ 41], 5.00th=[ 41], 10.00th=[ 42], 20.00th=[ 43],
| 30.00th=[ 43], 40.00th=[ 44], 50.00th=[ 47], 60.00th=[ 57],
| 70.00th=[ 58], 80.00th=[ 60], 90.00th=[ 67], 95.00th=[ 82],
| 99.00th=[ 128], 99.50th=[ 139], 99.90th=[ 178], 99.95th=[ 202],
| 99.99th=[ 247]
bw ( MiB/s): min=11727, max=20114, per=100.00%, avg=16575.43, stdev=79.65, samples=3808
iops : min=3002119, max=5149382, avg=4243308.64, stdev=20391.17, samples=3808
lat (nsec) : 750=0.01%, 1000=0.01%
lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=52.95%
lat (usec) : 100=44.99%, 250=2.05%, 500=0.01%, 750=0.01%, 1000=0.01%
lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%
cpu : usr=11.11%, sys=36.70%, ctx=254333246, majf=0, minf=458
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,254332005,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8Run status group 0 (all jobs):
WRITE: bw=16.2GiB/s (17.4GB/s), 16.2GiB/s-16.2GiB/s (17.4GB/s-17.4GB/s), io=970GiB (1042GB), run=60002-60002msec
Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
RANDREAD: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=8
...
fio-3.27
Starting 32 processes
Jobs: 32 (f=32): [w(32)][100.0%][w=16.3GiB/s][w=4273k IOPS][eta 00m:00s]
RANDREAD: (groupid=0, jobs=32): err= 0: pid=5427: Thu Jan 27 17:54:48 2022
write: IOPS=4245k, BW=16.2GiB/s (17.4GB/s)(972GiB/60002msec); 0 zone resets
slat (nsec): min=1703, max=9173.2k, avg=6757.54, stdev=3629.49
clat (nsec): min=662, max=9238.6k, avg=53063.30, stdev=15488.04
lat (usec): min=5, max=9247, avg=59.90, stdev=17.02
clat percentiles (usec):
| 1.00th=[ 41], 5.00th=[ 42], 10.00th=[ 42], 20.00th=[ 43],
| 30.00th=[ 44], 40.00th=[ 45], 50.00th=[ 49], 60.00th=[ 57],
| 70.00th=[ 59], 80.00th=[ 61], 90.00th=[ 65], 95.00th=[ 78],
| 99.00th=[ 98], 99.50th=[ 125], 99.90th=[ 153], 99.95th=[ 169],
| 99.99th=[ 217]
bw ( MiB/s): min=12078, max=19996, per=100.00%, avg=16599.68, stdev=76.90, samples=3808
iops : min=3092171, max=5119032, avg=4249517.34, stdev=19685.78, samples=3808
lat (nsec) : 750=0.01%, 1000=0.01%
lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=50.67%
lat (usec) : 100=48.44%, 250=0.88%, 500=0.01%, 750=0.01%, 1000=0.01%
lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%
cpu : usr=11.51%, sys=36.96%, ctx=254724648, majf=0, minf=457
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,254721978,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8Run status group 0 (all jobs):
WRITE: bw=16.2GiB/s (17.4GB/s), 16.2GiB/s-16.2GiB/s (17.4GB/s-17.4GB/s), io=972GiB (1043GB), run=60002-60002msec
Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
RANDREAD: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=8
...
fio-3.27
Starting 32 processes
Jobs: 32 (f=32): [w(32)][100.0%][w=16.3GiB/s][w=4263k IOPS][eta 00m:00s]
RANDREAD: (groupid=0, jobs=32): err= 0: pid=5463: Thu Jan 27 17:55:48 2022
write: IOPS=4202k, BW=16.0GiB/s (17.2GB/s)(962GiB/60002msec); 0 zone resets
slat (nsec): min=1473, max=8141.0k, avg=6835.36, stdev=4208.33
clat (nsec): min=762, max=8205.4k, avg=53600.59, stdev=18081.09
lat (usec): min=6, max=8214, avg=60.51, stdev=19.88
clat percentiles (usec):
| 1.00th=[ 41], 5.00th=[ 42], 10.00th=[ 42], 20.00th=[ 43],
| 30.00th=[ 44], 40.00th=[ 45], 50.00th=[ 50], 60.00th=[ 57],
| 70.00th=[ 59], 80.00th=[ 61], 90.00th=[ 67], 95.00th=[ 79],
| 99.00th=[ 118], 99.50th=[ 133], 99.90th=[ 163], 99.95th=[ 186],
| 99.99th=[ 227]
bw ( MiB/s): min=11930, max=19982, per=100.00%, avg=16430.36, stdev=83.37, samples=3808
iops : min=3054264, max=5115568, avg=4206171.98, stdev=21342.34, samples=3808
lat (nsec) : 1000=0.01%
lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=50.62%
lat (usec) : 100=47.91%, 250=1.46%, 500=0.01%, 750=0.01%, 1000=0.01%
lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%
cpu : usr=11.29%, sys=36.70%, ctx=252160334, majf=0, minf=468
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,252156450,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8Run status group 0 (all jobs):
WRITE: bw=16.0GiB/s (17.2GB/s), 16.0GiB/s-16.0GiB/s (17.2GB/s-17.2GB/s), io=962GiB (1033GB), run=60002-60002msec
Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
RANDREAD: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=8
...
fio-3.27
Starting 32 processes
Jobs: 32 (f=32): [w(32)][100.0%][w=16.2GiB/s][w=4249k IOPS][eta 00m:00s]
RANDREAD: (groupid=0, jobs=32): err= 0: pid=5514: Thu Jan 27 17:56:49 2022
write: IOPS=4183k, BW=16.0GiB/s (17.1GB/s)(957GiB/60002msec); 0 zone resets
slat (nsec): min=1924, max=17060k, avg=6871.81, stdev=4168.70
clat (nsec): min=732, max=17171k, avg=53851.87, stdev=19003.86
lat (usec): min=6, max=17179, avg=60.80, stdev=21.03
clat percentiles (usec):
| 1.00th=[ 41], 5.00th=[ 42], 10.00th=[ 42], 20.00th=[ 43],
| 30.00th=[ 44], 40.00th=[ 45], 50.00th=[ 48], 60.00th=[ 57],
| 70.00th=[ 59], 80.00th=[ 61], 90.00th=[ 68], 95.00th=[ 84],
| 99.00th=[ 128], 99.50th=[ 137], 99.90th=[ 176], 99.95th=[ 200],
| 99.99th=[ 239]
bw ( MiB/s): min=11373, max=20061, per=100.00%, avg=16353.68, stdev=88.86, samples=3808
iops : min=2911622, max=5135864, avg=4186540.81, stdev=22748.13, samples=3808
lat (nsec) : 750=0.01%, 1000=0.01%
lat (usec) : 2=0.01%, 10=0.01%, 20=0.01%, 50=52.48%, 100=45.18%
lat (usec) : 250=2.33%, 500=0.01%, 750=0.01%, 1000=0.01%
lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%
cpu : usr=11.24%, sys=36.38%, ctx=250990537, majf=0, minf=467
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,250987092,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8Run status group 0 (all jobs):
WRITE: bw=16.0GiB/s (17.1GB/s), 16.0GiB/s-16.0GiB/s (17.1GB/s-17.1GB/s), io=957GiB (1028GB), run=60002-60002msec
Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
RANDREAD: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=8
...
fio-3.27
Starting 32 processes
Jobs: 32 (f=32): [w(32)][100.0%][w=14.6GiB/s][w=3828k IOPS][eta 00m:00s]
RANDREAD: (groupid=0, jobs=32): err= 0: pid=5550: Thu Jan 27 17:57:49 2022
write: IOPS=4194k, BW=16.0GiB/s (17.2GB/s)(960GiB/60002msec); 0 zone resets
slat (nsec): min=1322, max=28060k, avg=6843.41, stdev=4820.43
clat (nsec): min=671, max=28139k, avg=53708.73, stdev=19454.27
lat (usec): min=5, max=28147, avg=60.63, stdev=21.33
clat percentiles (usec):
| 1.00th=[ 41], 5.00th=[ 42], 10.00th=[ 42], 20.00th=[ 43],
| 30.00th=[ 44], 40.00th=[ 45], 50.00th=[ 49], 60.00th=[ 57],
| 70.00th=[ 59], 80.00th=[ 61], 90.00th=[ 68], 95.00th=[ 79],
| 99.00th=[ 124], 99.50th=[ 137], 99.90th=[ 169], 99.95th=[ 188],
| 99.99th=[ 229]
bw ( MiB/s): min=11105, max=19937, per=100.00%, avg=16409.68, stdev=84.52, samples=3808
iops : min=2843058, max=5103984, avg=4200878.39, stdev=21637.18, samples=3808
lat (nsec) : 750=0.01%, 1000=0.01%
lat (usec) : 2=0.01%, 10=0.01%, 20=0.01%, 50=50.82%, 100=47.64%
lat (usec) : 250=1.54%, 500=0.01%, 750=0.01%, 1000=0.01%
lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
cpu : usr=11.23%, sys=36.86%, ctx=251667327, majf=0, minf=462
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,251664333,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8Run status group 0 (all jobs):
WRITE: bw=16.0GiB/s (17.2GB/s), 16.0GiB/s-16.0GiB/s (17.2GB/s-17.2GB/s), io=960GiB (1031GB), run=60002-60002msec
Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
RANDREAD: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=8
...
fio-3.27
Starting 32 processes
Jobs: 32 (f=32): [w(32)][100.0%][w=16.0GiB/s][w=4198k IOPS][eta 00m:00s]
RANDREAD: (groupid=0, jobs=32): err= 0: pid=5590: Thu Jan 27 17:58:49 2022
write: IOPS=4209k, BW=16.1GiB/s (17.2GB/s)(963GiB/60002msec); 0 zone resets
slat (nsec): min=1152, max=9032.6k, avg=6818.07, stdev=4518.03
clat (nsec): min=731, max=9134.7k, avg=53517.89, stdev=19533.23
lat (usec): min=5, max=9143, avg=60.41, stdev=21.38
clat percentiles (usec):
| 1.00th=[ 41], 5.00th=[ 42], 10.00th=[ 42], 20.00th=[ 43],
| 30.00th=[ 44], 40.00th=[ 45], 50.00th=[ 49], 60.00th=[ 57],
| 70.00th=[ 59], 80.00th=[ 61], 90.00th=[ 67], 95.00th=[ 81],
| 99.00th=[ 108], 99.50th=[ 130], 99.90th=[ 163], 99.95th=[ 184],
| 99.99th=[ 247]
bw ( MiB/s): min=11087, max=19994, per=100.00%, avg=16459.38, stdev=85.78, samples=3808
iops : min=2838462, max=5118598, avg=4213601.45, stdev=21959.20, samples=3808
lat (nsec) : 750=0.01%, 1000=0.01%
lat (usec) : 2=0.01%, 10=0.01%, 20=0.01%, 50=51.47%, 100=47.09%
lat (usec) : 250=1.43%, 500=0.01%, 750=0.01%, 1000=0.01%
lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%
cpu : usr=11.29%, sys=36.79%, ctx=252571826, majf=0, minf=452
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,252569484,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8Run status group 0 (all jobs):
WRITE: bw=16.1GiB/s (17.2GB/s), 16.1GiB/s-16.1GiB/s (17.2GB/s-17.2GB/s), io=963GiB (1035GB), run=60002-60002msec
Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
RANDREAD: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=8
...
fio-3.27
Starting 32 processes
Jobs: 32 (f=32): [w(32)][100.0%][w=16.3GiB/s][w=4264k IOPS][eta 00m:00s]
RANDREAD: (groupid=0, jobs=32): err= 0: pid=5656: Thu Jan 27 17:59:49 2022
write: IOPS=4164k, BW=15.9GiB/s (17.1GB/s)(953GiB/60002msec); 0 zone resets
slat (nsec): min=1944, max=6038.1k, avg=6904.06, stdev=4726.09
clat (nsec): min=702, max=6151.0k, avg=54088.57, stdev=21333.47
lat (usec): min=5, max=6160, avg=61.07, stdev=23.45
clat percentiles (usec):
| 1.00th=[ 41], 5.00th=[ 42], 10.00th=[ 42], 20.00th=[ 43],
| 30.00th=[ 44], 40.00th=[ 45], 50.00th=[ 48], 60.00th=[ 57],
| 70.00th=[ 59], 80.00th=[ 61], 90.00th=[ 68], 95.00th=[ 82],
| 99.00th=[ 129], 99.50th=[ 141], 99.90th=[ 182], 99.95th=[ 200],
| 99.99th=[ 273]
bw ( MiB/s): min= 9379, max=19988, per=100.00%, avg=16279.09, stdev=90.88, samples=3808
iops : min=2401269, max=5117100, avg=4167447.44, stdev=23265.01, samples=3808
lat (nsec) : 750=0.01%, 1000=0.01%
lat (usec) : 2=0.01%, 10=0.01%, 20=0.01%, 50=51.86%, 100=45.88%
lat (usec) : 250=2.25%, 500=0.01%, 750=0.01%, 1000=0.01%
lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%
cpu : usr=11.46%, sys=36.02%, ctx=249868343, majf=0, minf=468
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,249864894,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8Run status group 0 (all jobs):
WRITE: bw=15.9GiB/s (17.1GB/s), 15.9GiB/s-15.9GiB/s (17.1GB/s-17.1GB/s), io=953GiB (1023GB), run=60002-60002msec
Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
RANDREAD: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=8
...
fio-3.27
Starting 32 processes
Jobs: 32 (f=32): [w(32)][100.0%][w=16.6GiB/s][w=4344k IOPS][eta 00m:00s]
RANDREAD: (groupid=0, jobs=32): err= 0: pid=5691: Thu Jan 27 18:00:50 2022
write: IOPS=4204k, BW=16.0GiB/s (17.2GB/s)(962GiB/60002msec); 0 zone resets
slat (nsec): min=1793, max=4070.8k, avg=6830.69, stdev=3956.65
clat (nsec): min=722, max=4800.9k, avg=53585.28, stdev=16879.04
lat (usec): min=6, max=4808, avg=60.49, stdev=18.53
clat percentiles (usec):
| 1.00th=[ 41], 5.00th=[ 42], 10.00th=[ 42], 20.00th=[ 43],
| 30.00th=[ 44], 40.00th=[ 45], 50.00th=[ 50], 60.00th=[ 58],
| 70.00th=[ 59], 80.00th=[ 61], 90.00th=[ 67], 95.00th=[ 79],
| 99.00th=[ 103], 99.50th=[ 128], 99.90th=[ 159], 99.95th=[ 178],
| 99.99th=[ 215]
bw ( MiB/s): min=12007, max=19928, per=100.00%, avg=16435.02, stdev=78.86, samples=3808
iops : min=3074024, max=5101580, avg=4207364.55, stdev=20188.09, samples=3808
lat (nsec) : 750=0.01%, 1000=0.01%
lat (usec) : 2=0.01%, 10=0.01%, 20=0.01%, 50=50.36%, 100=48.55%
lat (usec) : 250=1.09%, 500=0.01%, 750=0.01%, 1000=0.01%
lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%
cpu : usr=11.24%, sys=37.00%, ctx=252252647, majf=0, minf=459
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,252249340,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8Run status group 0 (all jobs):
WRITE: bw=16.0GiB/s (17.2GB/s), 16.0GiB/s-16.0GiB/s (17.2GB/s-17.2GB/s), io=962GiB (1033GB), run=60002-60002msec
Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
RANDREAD: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=8
...
fio-3.27
Starting 32 processes
Jobs: 32 (f=32): [w(32)][100.0%][w=16.1GiB/s][w=4228k IOPS][eta 00m:00s]
RANDREAD: (groupid=0, jobs=32): err= 0: pid=5728: Thu Jan 27 18:01:50 2022
write: IOPS=4206k, BW=16.0GiB/s (17.2GB/s)(963GiB/60002msec); 0 zone resets
slat (nsec): min=1893, max=5884.2k, avg=6827.20, stdev=3323.00
clat (nsec): min=721, max=5947.0k, avg=53554.56, stdev=15149.53
lat (usec): min=6, max=5955, avg=60.46, stdev=16.85
clat percentiles (usec):
| 1.00th=[ 41], 5.00th=[ 42], 10.00th=[ 42], 20.00th=[ 43],
| 30.00th=[ 44], 40.00th=[ 45], 50.00th=[ 49], 60.00th=[ 57],
| 70.00th=[ 59], 80.00th=[ 61], 90.00th=[ 66], 95.00th=[ 80],
| 99.00th=[ 114], 99.50th=[ 131], 99.90th=[ 163], 99.95th=[ 186],
| 99.99th=[ 233]
bw ( MiB/s): min=11577, max=19959, per=100.00%, avg=16446.23, stdev=82.53, samples=3808
iops : min=2963773, max=5109736, avg=4210235.39, stdev=21126.75, samples=3808
lat (nsec) : 750=0.01%, 1000=0.01%
lat (usec) : 2=0.01%, 10=0.01%, 20=0.01%, 50=51.43%, 100=46.90%
lat (usec) : 250=1.67%, 500=0.01%, 750=0.01%, 1000=0.01%
lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%
cpu : usr=11.31%, sys=36.74%, ctx=252382497, majf=0, minf=458
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,252378162,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8Run status group 0 (all jobs):
WRITE: bw=16.0GiB/s (17.2GB/s), 16.0GiB/s-16.0GiB/s (17.2GB/s-17.2GB/s), io=963GiB (1034GB), run=60002-60002msec
Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
RANDREAD: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=8
...
fio-3.27
Starting 32 processes
Jobs: 32 (f=32): [w(32)][100.0%][w=15.2GiB/s][w=3991k IOPS][eta 00m:00s]
RANDREAD: (groupid=0, jobs=32): err= 0: pid=5764: Thu Jan 27 18:02:50 2022
write: IOPS=4185k, BW=16.0GiB/s (17.1GB/s)(958GiB/60002msec); 0 zone resets
slat (nsec): min=1703, max=22107k, avg=6866.93, stdev=3619.78
clat (nsec): min=721, max=22216k, avg=53822.24, stdev=16496.57
lat (usec): min=6, max=22225, avg=60.77, stdev=18.38
clat percentiles (usec):
| 1.00th=[ 41], 5.00th=[ 42], 10.00th=[ 42], 20.00th=[ 43],
| 30.00th=[ 44], 40.00th=[ 45], 50.00th=[ 49], 60.00th=[ 58],
| 70.00th=[ 59], 80.00th=[ 61], 90.00th=[ 67], 95.00th=[ 81],
| 99.00th=[ 126], 99.50th=[ 137], 99.90th=[ 167], 99.95th=[ 188],
| 99.99th=[ 215]
bw ( MiB/s): min=11586, max=20072, per=100.00%, avg=16363.86, stdev=79.91, samples=3808
iops : min=2966081, max=5138584, avg=4189148.29, stdev=20458.03, samples=3808
lat (nsec) : 750=0.01%, 1000=0.01%
lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=51.63%
lat (usec) : 100=46.38%, 250=2.00%, 500=0.01%, 750=0.01%, 1000=0.01%
lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 50=0.01%
cpu : usr=11.68%, sys=35.90%, ctx=251114127, majf=0, minf=487
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,251110660,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8Run status group 0 (all jobs):
WRITE: bw=16.0GiB/s (17.1GB/s), 16.0GiB/s-16.0GiB/s (17.1GB/s-17.1GB/s), io=958GiB (1029GB), run=60002-60002msec
Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
RANDREAD: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
...
fio-3.27
Starting 48 processes
Jobs: 48 (f=48): [r(48)][100.0%][r=17.1GiB/s][r=4489k IOPS][eta 00m:00s]
RANDREAD: (groupid=0, jobs=48): err= 0: pid=5804: Thu Jan 27 18:03:50 2022
read: IOPS=4849k, BW=18.5GiB/s (19.9GB/s)(1110GiB/60002msec)
slat (nsec): min=1132, max=159145k, avg=8981.10, stdev=19376.45
clat (nsec): min=450, max=179342k, avg=623965.09, stdev=331422.19
lat (usec): min=5, max=179344, avg=633.04, stdev=335.77
clat percentiles (usec):
| 1.00th=[ 400], 5.00th=[ 412], 10.00th=[ 420], 20.00th=[ 519],
| 30.00th=[ 523], 40.00th=[ 529], 50.00th=[ 537], 60.00th=[ 537],
| 70.00th=[ 545], 80.00th=[ 570], 90.00th=[ 1123], 95.00th=[ 1172],
| 99.00th=[ 1221], 99.50th=[ 1237], 99.90th=[ 1385], 99.95th=[ 4948],
| 99.99th=[11469]
bw ( MiB/s): min= 1701, max=26666, per=100.00%, avg=18969.39, stdev=72.63, samples=5712
iops : min=435502, max=6826592, avg=4856162.51, stdev=18593.47, samples=5712
lat (nsec) : 500=0.01%, 750=0.01%, 1000=0.01%
lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
lat (usec) : 100=0.01%, 250=0.01%, 500=14.96%, 750=65.56%, 1000=7.84%
lat (msec) : 2=11.55%, 4=0.02%, 10=0.05%, 20=0.01%, 50=0.01%
lat (msec) : 100=0.01%, 250=0.01%
cpu : usr=9.59%, sys=34.72%, ctx=290926854, majf=0, minf=3800
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=290951231,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64Run status group 0 (all jobs):
READ: bw=18.5GiB/s (19.9GB/s), 18.5GiB/s-18.5GiB/s (19.9GB/s-19.9GB/s), io=1110GiB (1192GB), run=60002-60002msec
Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
RANDREAD: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
...
fio-3.27
Starting 48 processes
Jobs: 48 (f=48): [r(48)][100.0%][r=17.7GiB/s][r=4652k IOPS][eta 00m:00s]
RANDREAD: (groupid=0, jobs=48): err= 0: pid=6122: Thu Jan 27 18:04:51 2022
read: IOPS=4951k, BW=18.9GiB/s (20.3GB/s)(1133GiB/60002msec)
slat (usec): min=2, max=14554, avg= 8.81, stdev= 5.43
clat (nsec): min=942, max=15180k, avg=611092.58, stdev=228636.31
lat (usec): min=5, max=15187, avg=620.00, stdev=232.12
clat percentiles (usec):
| 1.00th=[ 400], 5.00th=[ 408], 10.00th=[ 416], 20.00th=[ 519],
| 30.00th=[ 523], 40.00th=[ 529], 50.00th=[ 529], 60.00th=[ 537],
| 70.00th=[ 545], 80.00th=[ 553], 90.00th=[ 1123], 95.00th=[ 1172],
| 99.00th=[ 1221], 99.50th=[ 1237], 99.90th=[ 1270], 99.95th=[ 1287],
| 99.99th=[ 1434]
bw ( MiB/s): min=15970, max=26850, per=100.00%, avg=19365.18, stdev=74.95, samples=5712
iops : min=4088368, max=6873808, avg=4957485.39, stdev=19187.62, samples=5712
lat (nsec) : 1000=0.01%
lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
lat (usec) : 100=0.01%, 250=0.01%, 500=15.33%, 750=66.81%, 1000=6.49%
lat (msec) : 2=11.36%, 4=0.01%, 10=0.01%, 20=0.01%
cpu : usr=9.55%, sys=35.55%, ctx=297084580, majf=0, minf=3746
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=297077529,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64Run status group 0 (all jobs):
READ: bw=18.9GiB/s (20.3GB/s), 18.9GiB/s-18.9GiB/s (20.3GB/s-20.3GB/s), io=1133GiB (1217GB), run=60002-60002msec
Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
RANDREAD: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
...
fio-3.27
Starting 48 processes
Jobs: 48 (f=48): [r(48)][100.0%][r=18.8GiB/s][r=4916k IOPS][eta 00m:00s]
RANDREAD: (groupid=0, jobs=48): err= 0: pid=6172: Thu Jan 27 18:05:51 2022
read: IOPS=5165k, BW=19.7GiB/s (21.2GB/s)(1182GiB/60001msec)
slat (usec): min=2, max=17729, avg= 8.40, stdev= 5.56
clat (nsec): min=892, max=18656k, avg=585778.22, stdev=198124.01
lat (usec): min=5, max=18662, avg=594.27, stdev=201.11
clat percentiles (usec):
| 1.00th=[ 404], 5.00th=[ 416], 10.00th=[ 429], 20.00th=[ 523],
| 30.00th=[ 529], 40.00th=[ 529], 50.00th=[ 537], 60.00th=[ 537],
| 70.00th=[ 537], 80.00th=[ 545], 90.00th=[ 922], 95.00th=[ 1156],
| 99.00th=[ 1205], 99.50th=[ 1221], 99.90th=[ 1270], 99.95th=[ 1287],
| 99.99th=[ 1958]
bw ( MiB/s): min=16761, max=27172, per=100.00%, avg=20200.70, stdev=70.43, samples=5712
iops : min=4290928, max=6956172, avg=5171378.75, stdev=18029.98, samples=5712
lat (nsec) : 1000=0.01%
lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
lat (usec) : 100=0.01%, 250=0.01%, 500=11.46%, 750=77.26%, 1000=2.56%
lat (msec) : 2=8.71%, 4=0.01%, 10=0.01%, 20=0.01%
cpu : usr=9.81%, sys=37.61%, ctx=309932159, majf=0, minf=3754
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=309924108,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64Run status group 0 (all jobs):
READ: bw=19.7GiB/s (21.2GB/s), 19.7GiB/s-19.7GiB/s (21.2GB/s-21.2GB/s), io=1182GiB (1269GB), run=60001-60001msec
Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
RANDREAD: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
...
fio-3.27
Starting 48 processes
Jobs: 48 (f=48): [r(48)][100.0%][r=18.4GiB/s][r=4823k IOPS][eta 00m:00s]
RANDREAD: (groupid=0, jobs=48): err= 0: pid=6223: Thu Jan 27 18:06:51 2022
read: IOPS=5100k, BW=19.5GiB/s (20.9GB/s)(1167GiB/60001msec)
slat (nsec): min=1783, max=12540k, avg=8508.96, stdev=4346.12
clat (nsec): min=902, max=13091k, avg=593248.59, stdev=203310.48
lat (usec): min=5, max=13098, avg=601.85, stdev=206.43
clat percentiles (usec):
| 1.00th=[ 404], 5.00th=[ 416], 10.00th=[ 424], 20.00th=[ 523],
| 30.00th=[ 529], 40.00th=[ 529], 50.00th=[ 537], 60.00th=[ 537],
| 70.00th=[ 545], 80.00th=[ 545], 90.00th=[ 955], 95.00th=[ 1156],
| 99.00th=[ 1205], 99.50th=[ 1221], 99.90th=[ 1254], 99.95th=[ 1270],
| 99.99th=[ 1319]
bw ( MiB/s): min=16307, max=26858, per=100.00%, avg=19946.86, stdev=68.81, samples=5712
iops : min=4174673, max=6875752, avg=5106394.77, stdev=17615.47, samples=5712
lat (nsec) : 1000=0.01%
lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
lat (usec) : 100=0.01%, 250=0.01%, 500=12.52%, 750=74.51%, 1000=3.63%
lat (msec) : 2=9.34%, 4=0.01%, 10=0.01%, 20=0.01%
cpu : usr=9.82%, sys=37.09%, ctx=306024761, majf=0, minf=3752
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=306018256,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64Run status group 0 (all jobs):
READ: bw=19.5GiB/s (20.9GB/s), 19.5GiB/s-19.5GiB/s (20.9GB/s-20.9GB/s), io=1167GiB (1253GB), run=60001-60001msec
Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
RANDREAD: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
...
fio-3.27
Starting 48 processes
Jobs: 48 (f=48): [r(48)][100.0%][r=19.4GiB/s][r=5091k IOPS][eta 00m:00s]
RANDREAD: (groupid=0, jobs=48): err= 0: pid=6296: Thu Jan 27 18:07:51 2022
read: IOPS=5294k, BW=20.2GiB/s (21.7GB/s)(1212GiB/60002msec)
slat (usec): min=2, max=15196, avg= 8.17, stdev= 4.41
clat (nsec): min=882, max=22326k, avg=571595.01, stdev=165345.43
lat (usec): min=5, max=24612, avg=579.86, stdev=167.86
clat percentiles (usec):
| 1.00th=[ 408], 5.00th=[ 424], 10.00th=[ 519], 20.00th=[ 529],
| 30.00th=[ 529], 40.00th=[ 529], 50.00th=[ 537], 60.00th=[ 537],
| 70.00th=[ 537], 80.00th=[ 545], 90.00th=[ 553], 95.00th=[ 1123],
| 99.00th=[ 1205], 99.50th=[ 1221], 99.90th=[ 1270], 99.95th=[ 1287],
| 99.99th=[ 1598]
bw ( MiB/s): min=17335, max=26843, per=100.00%, avg=20700.27, stdev=67.10, samples=5712
iops : min=4437902, max=6871860, avg=5299267.64, stdev=17176.46, samples=5712
lat (nsec) : 1000=0.01%
lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
lat (usec) : 100=0.01%, 250=0.01%, 500=7.44%, 750=84.36%, 1000=2.47%
lat (msec) : 2=5.72%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
cpu : usr=9.89%, sys=38.86%, ctx=317629057, majf=0, minf=3793
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=317620780,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64Run status group 0 (all jobs):
READ: bw=20.2GiB/s (21.7GB/s), 20.2GiB/s-20.2GiB/s (21.7GB/s-21.7GB/s), io=1212GiB (1301GB), run=60002-60002msec
Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
RANDREAD: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
...
fio-3.27
Starting 48 processes
Jobs: 48 (f=48): [r(48)][100.0%][r=18.3GiB/s][r=4798k IOPS][eta 00m:00s]
RANDREAD: (groupid=0, jobs=48): err= 0: pid=6365: Thu Jan 27 18:08:52 2022
read: IOPS=5044k, BW=19.2GiB/s (20.7GB/s)(1155GiB/60002msec)
slat (usec): min=2, max=13061, avg= 8.62, stdev= 4.52
clat (nsec): min=921, max=13609k, avg=599818.01, stdev=216963.17
lat (usec): min=5, max=13617, avg=608.53, stdev=220.30
clat percentiles (usec):
| 1.00th=[ 404], 5.00th=[ 412], 10.00th=[ 420], 20.00th=[ 523],
| 30.00th=[ 529], 40.00th=[ 529], 50.00th=[ 537], 60.00th=[ 537],
| 70.00th=[ 537], 80.00th=[ 545], 90.00th=[ 1106], 95.00th=[ 1172],
| 99.00th=[ 1221], 99.50th=[ 1237], 99.90th=[ 1270], 99.95th=[ 1287],
| 99.99th=[ 1352]
bw ( MiB/s): min=16173, max=26964, per=100.00%, avg=19726.76, stdev=74.27, samples=5712
iops : min=4140321, max=6902906, avg=5050050.54, stdev=19012.08, samples=5712
lat (nsec) : 1000=0.01%
lat (usec) : 2=0.01%, 10=0.01%, 20=0.01%, 50=0.01%, 100=0.01%
lat (usec) : 250=0.01%, 500=14.23%, 750=71.40%, 1000=3.68%
lat (msec) : 2=10.69%, 4=0.01%, 10=0.01%, 20=0.01%
cpu : usr=9.68%, sys=36.60%, ctx=302656317, majf=0, minf=3792
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=302651095,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64Run status group 0 (all jobs):
READ: bw=19.2GiB/s (20.7GB/s), 19.2GiB/s-19.2GiB/s (20.7GB/s-20.7GB/s), io=1155GiB (1240GB), run=60002-60002msec
Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
RANDREAD: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
...
fio-3.27
Starting 48 processes
Jobs: 48 (f=48): [r(48)][100.0%][r=18.4GiB/s][r=4829k IOPS][eta 00m:00s]
RANDREAD: (groupid=0, jobs=48): err= 0: pid=6415: Thu Jan 27 18:09:52 2022
read: IOPS=5084k, BW=19.4GiB/s (20.8GB/s)(1164GiB/60002msec)
slat (nsec): min=1833, max=13078k, avg=8547.11, stdev=4680.01
clat (nsec): min=901, max=16594k, avg=595160.42, stdev=216066.92
lat (usec): min=5, max=16602, avg=603.80, stdev=219.37
clat percentiles (usec):
| 1.00th=[ 404], 5.00th=[ 412], 10.00th=[ 420], 20.00th=[ 523],
| 30.00th=[ 529], 40.00th=[ 529], 50.00th=[ 529], 60.00th=[ 537],
| 70.00th=[ 537], 80.00th=[ 545], 90.00th=[ 1123], 95.00th=[ 1172],
| 99.00th=[ 1221], 99.50th=[ 1237], 99.90th=[ 1270], 99.95th=[ 1287],
| 99.99th=[ 1336]
bw ( MiB/s): min=16019, max=27450, per=100.00%, avg=19883.77, stdev=74.22, samples=5712
iops : min=4100900, max=7027312, avg=5090245.61, stdev=18999.42, samples=5712
lat (nsec) : 1000=0.01%
lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
lat (usec) : 100=0.01%, 250=0.01%, 500=14.78%, 750=71.94%, 1000=2.25%
lat (msec) : 2=11.03%, 4=0.01%, 10=0.01%, 20=0.01%
cpu : usr=9.71%, sys=36.85%, ctx=305045338, majf=0, minf=3705
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=305037748,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64Run status group 0 (all jobs):
READ: bw=19.4GiB/s (20.8GB/s), 19.4GiB/s-19.4GiB/s (20.8GB/s-20.8GB/s), io=1164GiB (1249GB), run=60002-60002msec
Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
RANDREAD: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
...
fio-3.27
Starting 48 processes
Jobs: 48 (f=48): [r(48)][100.0%][r=17.8GiB/s][r=4664k IOPS][eta 00m:00s]
RANDREAD: (groupid=0, jobs=48): err= 0: pid=6466: Thu Jan 27 18:10:52 2022
read: IOPS=4916k, BW=18.8GiB/s (20.1GB/s)(1125GiB/60001msec)
slat (nsec): min=1934, max=13123k, avg=8874.39, stdev=4727.74
clat (nsec): min=852, max=13751k, avg=615357.93, stdev=231967.01
lat (usec): min=5, max=13757, avg=624.33, stdev=235.53
clat percentiles (usec):
| 1.00th=[ 400], 5.00th=[ 408], 10.00th=[ 416], 20.00th=[ 519],
| 30.00th=[ 529], 40.00th=[ 529], 50.00th=[ 529], 60.00th=[ 537],
| 70.00th=[ 545], 80.00th=[ 553], 90.00th=[ 1123], 95.00th=[ 1172],
| 99.00th=[ 1221], 99.50th=[ 1237], 99.90th=[ 1270], 99.95th=[ 1287],
| 99.99th=[ 1369]
bw ( MiB/s): min=15963, max=26966, per=100.00%, avg=19226.37, stdev=73.18, samples=5712
iops : min=4086652, max=6903490, avg=4921951.14, stdev=18733.45, samples=5712
lat (nsec) : 1000=0.01%
lat (usec) : 2=0.01%, 10=0.01%, 20=0.01%, 50=0.01%, 100=0.01%
lat (usec) : 250=0.01%, 500=16.23%, 750=64.97%, 1000=6.63%
lat (msec) : 2=12.17%, 4=0.01%, 10=0.01%, 20=0.01%
cpu : usr=9.56%, sys=35.15%, ctx=295000587, majf=0, minf=3760
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=294994610,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64Run status group 0 (all jobs):
READ: bw=18.8GiB/s (20.1GB/s), 18.8GiB/s-18.8GiB/s (20.1GB/s-20.1GB/s), io=1125GiB (1208GB), run=60001-60001msec
Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
RANDREAD: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
...
fio-3.27
Starting 48 processes
Jobs: 48 (f=48): [r(48)][100.0%][r=17.7GiB/s][r=4644k IOPS][eta 00m:00s]
RANDREAD: (groupid=0, jobs=48): err= 0: pid=6527: Thu Jan 27 18:11:52 2022
read: IOPS=5178k, BW=19.8GiB/s (21.2GB/s)(1185GiB/60002msec)
slat (usec): min=2, max=28107, avg= 8.37, stdev= 5.52
clat (nsec): min=852, max=31436k, avg=584283.93, stdev=204899.19
lat (usec): min=6, max=31444, avg=592.74, stdev=208.01
clat percentiles (usec):
| 1.00th=[ 404], 5.00th=[ 416], 10.00th=[ 433], 20.00th=[ 523],
| 30.00th=[ 529], 40.00th=[ 529], 50.00th=[ 537], 60.00th=[ 537],
| 70.00th=[ 545], 80.00th=[ 545], 90.00th=[ 873], 95.00th=[ 1156],
| 99.00th=[ 1221], 99.50th=[ 1696], 99.90th=[ 1844], 99.95th=[ 1876],
| 99.99th=[ 1975]
bw ( MiB/s): min=15730, max=27345, per=100.00%, avg=20259.01, stdev=78.17, samples=5712
iops : min=4026896, max=7000570, avg=5186307.07, stdev=20012.58, samples=5712
lat (nsec) : 1000=0.01%
lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
lat (usec) : 100=0.01%, 250=0.01%, 500=10.95%, 750=78.86%, 1000=2.55%
lat (msec) : 2=7.63%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
cpu : usr=9.79%, sys=37.84%, ctx=310725626, majf=0, minf=3759
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=310718002,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64Run status group 0 (all jobs):
READ: bw=19.8GiB/s (21.2GB/s), 19.8GiB/s-19.8GiB/s (21.2GB/s-21.2GB/s), io=1185GiB (1273GB), run=60002-60002msec
Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
RANDREAD: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
...
fio-3.27
Starting 48 processes
Jobs: 48 (f=48): [r(48)][100.0%][r=17.9GiB/s][r=4694k IOPS][eta 00m:00s]
RANDREAD: (groupid=0, jobs=48): err= 0: pid=6583: Thu Jan 27 18:12:53 2022
read: IOPS=5027k, BW=19.2GiB/s (20.6GB/s)(1151GiB/60001msec)
slat (usec): min=2, max=13076, avg= 8.65, stdev= 5.02
clat (nsec): min=922, max=17165k, avg=601847.41, stdev=215517.78
lat (usec): min=5, max=17240, avg=610.59, stdev=218.81
clat percentiles (usec):
| 1.00th=[ 404], 5.00th=[ 412], 10.00th=[ 424], 20.00th=[ 523],
| 30.00th=[ 529], 40.00th=[ 529], 50.00th=[ 537], 60.00th=[ 537],
| 70.00th=[ 545], 80.00th=[ 553], 90.00th=[ 1090], 95.00th=[ 1172],
| 99.00th=[ 1221], 99.50th=[ 1237], 99.90th=[ 1287], 99.95th=[ 1303],
| 99.99th=[ 1352]
bw ( MiB/s): min=16418, max=26693, per=100.00%, avg=19663.30, stdev=64.58, samples=5712
iops : min=4203136, max=6833668, avg=5033804.61, stdev=16531.95, samples=5712
lat (nsec) : 1000=0.01%
lat (usec) : 2=0.01%, 10=0.01%, 20=0.01%, 50=0.01%, 100=0.01%
lat (usec) : 250=0.01%, 500=13.51%, 750=71.59%, 1000=4.67%
lat (msec) : 2=10.24%, 4=0.01%, 10=0.01%, 20=0.01%
cpu : usr=9.66%, sys=36.48%, ctx=301651235, majf=0, minf=3785
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=301643983,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64Run status group 0 (all jobs):
READ: bw=19.2GiB/s (20.6GB/s), 19.2GiB/s-19.2GiB/s (20.6GB/s-20.6GB/s), io=1151GiB (1236GB), run=60001-60001msec
Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
+ NQN=testnqn
+ nvme disconnect -n testnqn
NQN:testnqn disconnected 1 controller(s)
real 0m0.370s
user 0m0.000s
sys 0m0.003s
+ for i in /sys/kernel/config/nvmet/subsystems/${NQN}/namespaces/*
+ echo 0
+ rmdir /sys/kernel/config/nvmet/subsystems/testnqn/namespaces/1
+ rm -fr /sys/kernel/config/nvmet/ports/1/subsystems/testnqn
+ rmdir /sys/kernel/config/nvmet/ports/1
+ rmdir /sys/kernel/config/nvmet/subsystems/testnqn
+ rmdir 'config/nullb/nullb*'
rmdir: failed to remove 'config/nullb/nullb*': No such file or directory
+ umount /mnt/nvme0n1
umount: /mnt/nvme0n1: no mount point specified.
+ umount /mnt/backend
umount: /mnt/backend: not mounted.
+ modprobe -r nvme_loop
+ modprobe -r nvme_fabrics
+ modprobe -r nvmet
+ modprobe -r nvme
+ modprobe -r null_blk
+ tree /sys/kernel/config/sys/kernel/config
0 directories, 0 files
+++ NN=1
+++ NQN=testnqn
+++ let NR_DEVICES=NN+1
+++ modprobe -r null_blk
+++ modprobe -r nvme
+++ modprobe null_blk nr_devices=0
+++ modprobe nvme
+++ modprobe nvme-fabrics
+++ modprobe nvmet
+++ modprobe nvme-loop
+++ dmesg -c
+++ sleep 2
+++ tree /sys/kernel/config/sys/kernel/config
├── nullb
│ └── features
└── nvmet
├── hosts
├── ports
└── subsystems5 directories, 1 file
+++ mkdir /sys/kernel/config/nvmet/subsystems/testnqn
+++ mkdir /sys/kernel/config/nvmet/subsystems/testnqn/namespaces/1
+++ echo -n /dev/nvme0n1
+++ cat /sys/kernel/config/nvmet/subsystems/testnqn/namespaces/1/device_path /dev/nvme0n1
+++ echo 0
+++ cat /sys/kernel/config/nvmet/subsystems/testnqn/namespaces/1/buffered_io
0
+++ echo 1
+++ mkdir /sys/kernel/config/nvmet/ports/1/
+++ echo -n loop
+++ echo -n 1
+++ ln -s /sys/kernel/config/nvmet/subsystems/testnqn /sys/kernel/config/nvmet/ports/1/subsystems/ +++ sleep 1
+++ echo transport=loop,nqn=testnqn
+++ sleep 1
+++ mount
+++ column -t
+++ grep nvme
+++ dmesg -c
[ 1871.786270] nvme nvme0: 48/0/0 default/read/poll queues [ 1873.775725] nvmet: adding nsid 1 to subsystem testnqn [ 1874.785877] nvmet: creating nvm controller 1 for subsystem testnqn for NQN nqn.2014-08.org.nvmexpress:uuid:80e5be7e-7d04-4c0b-807c-59aa6436392f. [ 1874.786002] nvme nvme1: creating 48 I/O queues. [ 1874.790499] nvme nvme1: new ctrl: "testnqn"
write-and-verify: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=4 fio-3.27
Starting 1 process
Jobs: 1 (f=1): [V(1)][81.2%][r=116MiB/s][r=29.8k IOPS][eta 00m:03s]
write-and-verify: (groupid=0, jobs=1): err= 0: pid=6693: Thu Jan 27 18:13:11 2022
read: IOPS=27.2k, BW=106MiB/s (111MB/s)(506MiB/4762msec)
slat (nsec): min=1032, max=516771, avg=4595.85, stdev=8646.19
clat (usec): min=16, max=3043, avg=140.01, stdev=76.43
lat (usec): min=45, max=3045, avg=144.74, stdev=76.35
clat percentiles (usec):
| 1.00th=[ 62], 5.00th=[ 78], 10.00th=[ 93], 20.00th=[ 103],
| 30.00th=[ 111], 40.00th=[ 116], 50.00th=[ 125], 60.00th=[ 139],
| 70.00th=[ 149], 80.00th=[ 169], 90.00th=[ 206], 95.00th=[ 221],
| 99.00th=[ 318], 99.50th=[ 498], 99.90th=[ 676], 99.95th=[ 1565],
| 99.99th=[ 2868]
write: IOPS=24.9k, BW=97.3MiB/s (102MB/s)(800MiB/8223msec); 0 zone resets
slat (usec): min=2, max=3278, avg=12.85, stdev=20.71
clat (usec): min=17, max=3726, avg=146.60, stdev=86.33
lat (usec): min=58, max=3731, avg=159.59, stdev=87.97
clat percentiles (usec):
| 1.00th=[ 82], 5.00th=[ 98], 10.00th=[ 104], 20.00th=[ 117],
| 30.00th=[ 125], 40.00th=[ 130], 50.00th=[ 137], 60.00th=[ 145],
| 70.00th=[ 155], 80.00th=[ 165], 90.00th=[ 178], 95.00th=[ 202],
| 99.00th=[ 437], 99.50th=[ 570], 99.90th=[ 717], 99.95th=[ 2704],
| 99.99th=[ 2966]
bw ( KiB/s): min=41904, max=106024, per=96.74%, avg=96376.47, stdev=14628.41, samples=17
iops : min=10476, max=26506, avg=24094.12, stdev=3657.10, samples=17
lat (usec) : 20=0.01%, 50=0.12%, 100=9.84%, 250=87.51%, 500=1.80%
lat (usec) : 750=0.64%, 1000=0.01%
lat (msec) : 2=0.01%, 4=0.05%
cpu : usr=20.26%, sys=17.67%, ctx=112192, majf=0, minf=3053
IO depths : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=129572,204800,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4Run status group 0 (all jobs):
READ: bw=106MiB/s (111MB/s), 106MiB/s-106MiB/s (111MB/s-111MB/s), io=506MiB (531MB), run=4762-4762msec
WRITE: bw=97.3MiB/s (102MB/s), 97.3MiB/s-97.3MiB/s (102MB/s-102MB/s), io=800MiB (839MB), run=8223-8223msec
Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
RANDREAD: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=8
...
fio-3.27
Starting 32 processes
Jobs: 32 (f=32): [w(32)][100.0%][w=278MiB/s][w=71.2k IOPS][eta 00m:00s]
RANDREAD: (groupid=0, jobs=32): err= 0: pid=6701: Thu Jan 27 18:14:11 2022
write: IOPS=70.6k, BW=276MiB/s (289MB/s)(16.2GiB/60003msec); 0 zone resets
slat (nsec): min=1062, max=10953k, avg=8479.24, stdev=68796.36
clat (nsec): min=782, max=21915k, avg=3616335.82, stdev=2120821.29
lat (usec): min=52, max=21918, avg=3624.96, stdev=2118.24
clat percentiles (usec):
| 1.00th=[ 644], 5.00th=[ 1074], 10.00th=[ 1385], 20.00th=[ 1876],
| 30.00th=[ 2311], 40.00th=[ 2737], 50.00th=[ 3163], 60.00th=[ 3687],
| 70.00th=[ 4293], 80.00th=[ 5080], 90.00th=[ 6390], 95.00th=[ 7701],
| 99.00th=[10552], 99.50th=[11731], 99.90th=[14615], 99.95th=[15926],
| 99.99th=[18744]
bw ( KiB/s): min=227255, max=337609, per=100.00%, avg=282652.32, stdev=684.49, samples=3808
iops : min=56808, max=84394, avg=70657.12, stdev=171.14, samples=3808
lat (nsec) : 1000=0.01%
lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.03%
lat (usec) : 100=0.02%, 250=0.02%, 500=0.35%, 750=1.27%, 1000=2.39%
lat (msec) : 2=18.82%, 4=42.74%, 10=33.01%, 20=1.35%, 50=0.01%
cpu : usr=0.61%, sys=1.37%, ctx=810168, majf=0, minf=460
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,4235752,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8Run status group 0 (all jobs):
WRITE: bw=276MiB/s (289MB/s), 276MiB/s-276MiB/s (289MB/s-289MB/s), io=16.2GiB (17.3GB), run=60003-60003msec
Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
RANDREAD: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=8
...
fio-3.27
Starting 32 processes
Jobs: 32 (f=32): [w(32)][100.0%][w=278MiB/s][w=71.1k IOPS][eta 00m:00s]
RANDREAD: (groupid=0, jobs=32): err= 0: pid=6736: Thu Jan 27 18:15:11 2022
write: IOPS=70.5k, BW=275MiB/s (289MB/s)(16.1GiB/60003msec); 0 zone resets
slat (nsec): min=1032, max=18765k, avg=9164.66, stdev=78099.99
clat (nsec): min=1132, max=28371k, avg=3619069.12, stdev=2128877.06
lat (usec): min=58, max=28387, avg=3628.38, stdev=2126.28
clat percentiles (usec):
| 1.00th=[ 635], 5.00th=[ 1074], 10.00th=[ 1385], 20.00th=[ 1876],
| 30.00th=[ 2311], 40.00th=[ 2737], 50.00th=[ 3163], 60.00th=[ 3687],
| 70.00th=[ 4293], 80.00th=[ 5080], 90.00th=[ 6390], 95.00th=[ 7701],
| 99.00th=[10552], 99.50th=[11863], 99.90th=[14877], 99.95th=[16450],
| 99.99th=[19268]
bw ( KiB/s): min=220950, max=338088, per=100.00%, avg=282364.13, stdev=715.72, samples=3808
iops : min=55232, max=84515, avg=70584.97, stdev=178.95, samples=3808
lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.03%
lat (usec) : 100=0.02%, 250=0.02%, 500=0.36%, 750=1.28%, 1000=2.40%
lat (msec) : 2=18.82%, 4=42.65%, 10=33.06%, 20=1.37%, 50=0.01%
cpu : usr=0.60%, sys=1.37%, ctx=814074, majf=0, minf=465
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,4231716,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8Run status group 0 (all jobs):
WRITE: bw=275MiB/s (289MB/s), 275MiB/s-275MiB/s (289MB/s-289MB/s), io=16.1GiB (17.3GB), run=60003-60003msec
Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
RANDREAD: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=8
...
fio-3.27
Starting 32 processes
Jobs: 32 (f=32): [w(32)][100.0%][w=279MiB/s][w=71.4k IOPS][eta 00m:00s]
RANDREAD: (groupid=0, jobs=32): err= 0: pid=6874: Thu Jan 27 18:16:11 2022
write: IOPS=70.8k, BW=277MiB/s (290MB/s)(16.2GiB/60003msec); 0 zone resets
slat (nsec): min=1052, max=15214k, avg=8336.55, stdev=70464.37
clat (nsec): min=933, max=27162k, avg=3603651.62, stdev=2113870.32
lat (usec): min=53, max=27165, avg=3612.13, stdev=2111.38
clat percentiles (usec):
| 1.00th=[ 644], 5.00th=[ 1074], 10.00th=[ 1385], 20.00th=[ 1876],
| 30.00th=[ 2278], 40.00th=[ 2704], 50.00th=[ 3163], 60.00th=[ 3654],
| 70.00th=[ 4293], 80.00th=[ 5080], 90.00th=[ 6390], 95.00th=[ 7635],
| 99.00th=[10552], 99.50th=[11731], 99.90th=[14746], 99.95th=[15926],
| 99.99th=[18744]
bw ( KiB/s): min=221940, max=338378, per=100.00%, avg=283664.92, stdev=696.08, samples=3808
iops : min=55477, max=84585, avg=70909.98, stdev=174.04, samples=3808
lat (nsec) : 1000=0.01%
lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.03%
lat (usec) : 100=0.02%, 250=0.02%, 500=0.35%, 750=1.28%, 1000=2.43%
lat (msec) : 2=18.90%, 4=42.89%, 10=32.75%, 20=1.32%, 50=0.01%
cpu : usr=0.60%, sys=1.37%, ctx=811420, majf=0, minf=460
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,4250762,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8Run status group 0 (all jobs):
WRITE: bw=277MiB/s (290MB/s), 277MiB/s-277MiB/s (290MB/s-290MB/s), io=16.2GiB (17.4GB), run=60003-60003msec
Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
RANDREAD: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=8
...
fio-3.27
Starting 32 processes
Jobs: 32 (f=32): [w(32)][100.0%][w=278MiB/s][w=71.3k IOPS][eta 00m:00s]
RANDREAD: (groupid=0, jobs=32): err= 0: pid=6909: Thu Jan 27 18:17:12 2022
write: IOPS=70.8k, BW=277MiB/s (290MB/s)(16.2GiB/60003msec); 0 zone resets
slat (nsec): min=1042, max=14703k, avg=7972.25, stdev=65737.59
clat (nsec): min=892, max=30708k, avg=3605587.06, stdev=2119441.89
lat (usec): min=55, max=30714, avg=3613.70, stdev=2117.11
clat percentiles (usec):
| 1.00th=[ 635], 5.00th=[ 1074], 10.00th=[ 1385], 20.00th=[ 1860],
| 30.00th=[ 2278], 40.00th=[ 2704], 50.00th=[ 3163], 60.00th=[ 3654],
| 70.00th=[ 4293], 80.00th=[ 5080], 90.00th=[ 6390], 95.00th=[ 7635],
| 99.00th=[10552], 99.50th=[11731], 99.90th=[14746], 99.95th=[16057],
| 99.99th=[18744]
bw ( KiB/s): min=218037, max=337309, per=100.00%, avg=283483.15, stdev=695.32, samples=3808
iops : min=54507, max=84319, avg=70864.50, stdev=173.84, samples=3808
lat (nsec) : 1000=0.01%
lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.02%
lat (usec) : 100=0.02%, 250=0.02%, 500=0.36%, 750=1.31%, 1000=2.43%
lat (msec) : 2=18.97%, 4=42.75%, 10=32.78%, 20=1.35%, 50=0.01%
cpu : usr=0.60%, sys=1.36%, ctx=810930, majf=0, minf=463
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,4248890,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8Run status group 0 (all jobs):
WRITE: bw=277MiB/s (290MB/s), 277MiB/s-277MiB/s (290MB/s-290MB/s), io=16.2GiB (17.4GB), run=60003-60003msec
Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
RANDREAD: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=8
...
fio-3.27
Starting 32 processes
Jobs: 10 (f=0): [_(1),f(1),_(4),f(3),_(1),f(1),_(1),f(2),_(4),E(1),_(5),f(2),_(4),f(1),_(1)][100.0%][w=274MiB/s][w=70.1k IOPS][eta 00m:00s]
RANDREAD: (groupid=0, jobs=32): err= 0: pid=6947: Thu Jan 27 18:18:12 2022
write: IOPS=70.8k, BW=276MiB/s (290MB/s)(16.2GiB/60003msec); 0 zone resets
slat (nsec): min=1052, max=12322k, avg=8564.84, stdev=70836.46
clat (nsec): min=832, max=26513k, avg=3606722.49, stdev=2118981.37
lat (usec): min=44, max=26516, avg=3615.43, stdev=2116.31
clat percentiles (usec):
| 1.00th=[ 644], 5.00th=[ 1057], 10.00th=[ 1385], 20.00th=[ 1860],
| 30.00th=[ 2278], 40.00th=[ 2704], 50.00th=[ 3163], 60.00th=[ 3654],
| 70.00th=[ 4293], 80.00th=[ 5080], 90.00th=[ 6390], 95.00th=[ 7635],
| 99.00th=[10552], 99.50th=[11731], 99.90th=[14615], 99.95th=[15926],
| 99.99th=[18744]
bw ( KiB/s): min=224303, max=338155, per=100.00%, avg=283361.52, stdev=693.75, samples=3808
iops : min=56070, max=84529, avg=70833.75, stdev=173.45, samples=3808
lat (nsec) : 1000=0.01%
lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.03%
lat (usec) : 100=0.02%, 250=0.02%, 500=0.35%, 750=1.31%, 1000=2.46%
lat (msec) : 2=18.95%, 4=42.66%, 10=32.86%, 20=1.34%, 50=0.01%
cpu : usr=0.61%, sys=1.35%, ctx=813770, majf=0, minf=444
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,4246897,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8Run status group 0 (all jobs):
WRITE: bw=276MiB/s (290MB/s), 276MiB/s-276MiB/s (290MB/s-290MB/s), io=16.2GiB (17.4GB), run=60003-60003msec
Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
RANDREAD: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=8
...
fio-3.27
Starting 32 processes
Jobs: 32 (f=32): [w(32)][100.0%][w=283MiB/s][w=72.4k IOPS][eta 00m:00s]
RANDREAD: (groupid=0, jobs=32): err= 0: pid=6982: Thu Jan 27 18:19:12 2022
write: IOPS=70.9k, BW=277MiB/s (291MB/s)(16.2GiB/60003msec); 0 zone resets
slat (nsec): min=1042, max=14867k, avg=8020.33, stdev=67623.07
clat (nsec): min=901, max=25692k, avg=3599694.38, stdev=2111664.88
lat (usec): min=49, max=25695, avg=3607.86, stdev=2109.25
clat percentiles (usec):
| 1.00th=[ 635], 5.00th=[ 1057], 10.00th=[ 1385], 20.00th=[ 1860],
| 30.00th=[ 2278], 40.00th=[ 2704], 50.00th=[ 3163], 60.00th=[ 3654],
| 70.00th=[ 4293], 80.00th=[ 5080], 90.00th=[ 6390], 95.00th=[ 7635],
| 99.00th=[10421], 99.50th=[11731], 99.90th=[14746], 99.95th=[15926],
| 99.99th=[18482]
bw ( KiB/s): min=223339, max=340257, per=100.00%, avg=283985.42, stdev=696.63, samples=3808
iops : min=55832, max=85057, avg=70991.25, stdev=174.17, samples=3808
lat (nsec) : 1000=0.01%
lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.02%
lat (usec) : 100=0.02%, 250=0.02%, 500=0.36%, 750=1.33%, 1000=2.47%
lat (msec) : 2=18.90%, 4=42.78%, 10=32.77%, 20=1.32%, 50=0.01%
cpu : usr=0.61%, sys=1.36%, ctx=812512, majf=0, minf=492
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,4255806,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8Run status group 0 (all jobs):
WRITE: bw=277MiB/s (291MB/s), 277MiB/s-277MiB/s (291MB/s-291MB/s), io=16.2GiB (17.4GB), run=60003-60003msec
Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
RANDREAD: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=8
...
fio-3.27
Starting 32 processes
Jobs: 31 (f=0): [f(24),E(1),f(7)][100.0%][w=274MiB/s][w=70.1k IOPS][eta 00m:00s]
RANDREAD: (groupid=0, jobs=32): err= 0: pid=7018: Thu Jan 27 18:20:12 2022
write: IOPS=70.4k, BW=275MiB/s (288MB/s)(16.1GiB/60003msec); 0 zone resets
slat (nsec): min=1052, max=14536k, avg=8133.28, stdev=68038.63
clat (nsec): min=821, max=27597k, avg=3626388.02, stdev=2136516.22
lat (usec): min=62, max=27600, avg=3634.66, stdev=2134.10
clat percentiles (usec):
| 1.00th=[ 644], 5.00th=[ 1074], 10.00th=[ 1385], 20.00th=[ 1876],
| 30.00th=[ 2311], 40.00th=[ 2737], 50.00th=[ 3163], 60.00th=[ 3687],
| 70.00th=[ 4293], 80.00th=[ 5145], 90.00th=[ 6456], 95.00th=[ 7701],
| 99.00th=[10552], 99.50th=[11863], 99.90th=[14877], 99.95th=[16188],
| 99.99th=[18744]
bw ( KiB/s): min=208143, max=334884, per=100.00%, avg=281852.13, stdev=742.26, samples=3808
iops : min=52028, max=83715, avg=70457.33, stdev=185.58, samples=3808
lat (nsec) : 1000=0.01%
lat (usec) : 2=0.01%, 4=0.01%, 20=0.01%, 50=0.02%, 100=0.02%
lat (usec) : 250=0.02%, 500=0.35%, 750=1.29%, 1000=2.44%
lat (msec) : 2=18.82%, 4=42.45%, 10=33.20%, 20=1.39%, 50=0.01%
cpu : usr=0.61%, sys=1.33%, ctx=807486, majf=0, minf=475
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,4224406,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8Run status group 0 (all jobs):
WRITE: bw=275MiB/s (288MB/s), 275MiB/s-275MiB/s (288MB/s-288MB/s), io=16.1GiB (17.3GB), run=60003-60003msec
Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
RANDREAD: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=8
...
fio-3.27
Starting 32 processes
Jobs: 32 (f=32): [w(32)][100.0%][w=275MiB/s][w=70.4k IOPS][eta 00m:00s]
RANDREAD: (groupid=0, jobs=32): err= 0: pid=7055: Thu Jan 27 18:21:13 2022
write: IOPS=70.6k, BW=276MiB/s (289MB/s)(16.2GiB/60003msec); 0 zone resets
slat (nsec): min=1032, max=20412k, avg=8158.81, stdev=69737.89
clat (nsec): min=962, max=24941k, avg=3617807.93, stdev=2115567.18
lat (usec): min=52, max=24943, avg=3626.11, stdev=2113.19
clat percentiles (usec):
| 1.00th=[ 652], 5.00th=[ 1074], 10.00th=[ 1401], 20.00th=[ 1876],
| 30.00th=[ 2311], 40.00th=[ 2737], 50.00th=[ 3163], 60.00th=[ 3687],
| 70.00th=[ 4293], 80.00th=[ 5080], 90.00th=[ 6390], 95.00th=[ 7701],
| 99.00th=[10552], 99.50th=[11731], 99.90th=[14746], 99.95th=[15926],
| 99.99th=[18482]
bw ( KiB/s): min=217283, max=336948, per=100.00%, avg=282493.76, stdev=715.13, samples=3808
iops : min=54314, max=84232, avg=70617.30, stdev=178.80, samples=3808
lat (nsec) : 1000=0.01%
lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.02%
lat (usec) : 100=0.01%, 250=0.02%, 500=0.34%, 750=1.24%, 1000=2.37%
lat (msec) : 2=18.68%, 4=42.98%, 10=32.99%, 20=1.34%, 50=0.01%
cpu : usr=0.62%, sys=1.36%, ctx=806027, majf=0, minf=472
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,4234361,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8Run status group 0 (all jobs):
WRITE: bw=276MiB/s (289MB/s), 276MiB/s-276MiB/s (289MB/s-289MB/s), io=16.2GiB (17.3GB), run=60003-60003msec
Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
RANDREAD: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=8
...
fio-3.27
Starting 32 processes
Jobs: 32 (f=32): [w(32)][100.0%][w=276MiB/s][w=70.7k IOPS][eta 00m:00s]
RANDREAD: (groupid=0, jobs=32): err= 0: pid=7090: Thu Jan 27 18:22:13 2022
write: IOPS=70.4k, BW=275MiB/s (288MB/s)(16.1GiB/60003msec); 0 zone resets
slat (nsec): min=1042, max=17861k, avg=8286.99, stdev=68542.10
clat (nsec): min=761, max=27072k, avg=3626493.56, stdev=2113252.93
lat (usec): min=59, max=27075, avg=3634.92, stdev=2110.79
clat percentiles (usec):
| 1.00th=[ 652], 5.00th=[ 1090], 10.00th=[ 1401], 20.00th=[ 1893],
| 30.00th=[ 2311], 40.00th=[ 2737], 50.00th=[ 3195], 60.00th=[ 3687],
| 70.00th=[ 4293], 80.00th=[ 5080], 90.00th=[ 6390], 95.00th=[ 7701],
| 99.00th=[10552], 99.50th=[11731], 99.90th=[14615], 99.95th=[15795],
| 99.99th=[18744]
bw ( KiB/s): min=219915, max=337142, per=100.00%, avg=281891.44, stdev=699.72, samples=3808
iops : min=54970, max=84277, avg=70466.40, stdev=174.95, samples=3808
lat (nsec) : 1000=0.01%
lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.02%
lat (usec) : 100=0.01%, 250=0.02%, 500=0.34%, 750=1.21%, 1000=2.34%
lat (msec) : 2=18.55%, 4=43.04%, 10=33.13%, 20=1.34%, 50=0.01%
cpu : usr=0.62%, sys=1.36%, ctx=804216, majf=0, minf=431
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,4224035,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8Run status group 0 (all jobs):
WRITE: bw=275MiB/s (288MB/s), 275MiB/s-275MiB/s (288MB/s-288MB/s), io=16.1GiB (17.3GB), run=60003-60003msec
Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
RANDREAD: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=8
...
fio-3.27
Starting 32 processes
Jobs: 32 (f=32): [w(32)][100.0%][w=271MiB/s][w=69.5k IOPS][eta 00m:00s]
RANDREAD: (groupid=0, jobs=32): err= 0: pid=7129: Thu Jan 27 18:23:13 2022
write: IOPS=70.5k, BW=275MiB/s (289MB/s)(16.1GiB/60002msec); 0 zone resets
slat (nsec): min=1052, max=14893k, avg=8317.42, stdev=68537.42
clat (nsec): min=992, max=32268k, avg=3621884.28, stdev=2123143.47
lat (usec): min=47, max=32271, avg=3630.35, stdev=2120.64
clat percentiles (usec):
| 1.00th=[ 652], 5.00th=[ 1074], 10.00th=[ 1401], 20.00th=[ 1893],
| 30.00th=[ 2311], 40.00th=[ 2737], 50.00th=[ 3163], 60.00th=[ 3687],
| 70.00th=[ 4293], 80.00th=[ 5080], 90.00th=[ 6390], 95.00th=[ 7701],
| 99.00th=[10552], 99.50th=[11863], 99.90th=[14746], 99.95th=[16188],
| 99.99th=[18744]
bw ( KiB/s): min=222694, max=337437, per=100.00%, avg=282249.68, stdev=701.69, samples=3808
iops : min=55666, max=84355, avg=70556.92, stdev=175.44, samples=3808
lat (nsec) : 1000=0.01%
lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.02%
lat (usec) : 100=0.02%, 250=0.02%, 500=0.34%, 750=1.25%, 1000=2.36%
lat (msec) : 2=18.69%, 4=42.93%, 10=33.00%, 20=1.37%, 50=0.01%
cpu : usr=0.63%, sys=1.35%, ctx=806199, majf=0, minf=453
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,4229414,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8Run status group 0 (all jobs):
WRITE: bw=275MiB/s (289MB/s), 275MiB/s-275MiB/s (289MB/s-289MB/s), io=16.1GiB (17.3GB), run=60002-60002msec
Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
RANDREAD: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
...
fio-3.27
Starting 48 processes
Jobs: 48 (f=48): [r(48)][100.0%][r=291MiB/s][r=74.6k IOPS][eta 00m:00s]
RANDREAD: (groupid=0, jobs=48): err= 0: pid=7165: Thu Jan 27 18:24:13 2022
read: IOPS=70.7k, BW=276MiB/s (290MB/s)(16.2GiB/60035msec)
slat (nsec): min=952, max=58117k, avg=4520.11, stdev=161503.14
clat (usec): min=65, max=150510, avg=43420.07, stdev=10497.47
lat (usec): min=958, max=174048, avg=43424.71, stdev=10500.87
clat percentiles (msec):
| 1.00th=[ 25], 5.00th=[ 28], 10.00th=[ 30], 20.00th=[ 33],
| 30.00th=[ 41], 40.00th=[ 43], 50.00th=[ 44], 60.00th=[ 46],
| 70.00th=[ 47], 80.00th=[ 51], 90.00th=[ 58], 95.00th=[ 62],
| 99.00th=[ 69], 99.50th=[ 74], 99.90th=[ 90], 99.95th=[ 100],
| 99.99th=[ 118]
bw ( KiB/s): min=211427, max=336387, per=100.00%, avg=283065.16, stdev=517.97, samples=5664
iops : min=52844, max=84081, avg=70748.95, stdev=129.47, samples=5664
lat (usec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
lat (msec) : 2=0.01%, 4=0.01%, 10=0.04%, 20=0.70%, 50=78.41%
lat (msec) : 100=20.78%, 250=0.05%
cpu : usr=0.21%, sys=0.51%, ctx=207589, majf=0, minf=3775
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=4245864,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64Run status group 0 (all jobs):
READ: bw=276MiB/s (290MB/s), 276MiB/s-276MiB/s (290MB/s-290MB/s), io=16.2GiB (17.4GB), run=60035-60035msec
Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
RANDREAD: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
...
fio-3.27
Starting 48 processes
Jobs: 48 (f=48): [r(48)][100.0%][r=290MiB/s][r=74.4k IOPS][eta 00m:00s]
RANDREAD: (groupid=0, jobs=48): err= 0: pid=7215: Thu Jan 27 18:25:14 2022
read: IOPS=71.6k, BW=280MiB/s (293MB/s)(16.4GiB/60039msec)
slat (nsec): min=961, max=92120k, avg=4780.46, stdev=198912.93
clat (usec): min=137, max=207359, avg=42871.61, stdev=10469.73
lat (usec): min=1036, max=207361, avg=42876.49, stdev=10474.88
clat percentiles (msec):
| 1.00th=[ 24], 5.00th=[ 28], 10.00th=[ 29], 20.00th=[ 32],
| 30.00th=[ 40], 40.00th=[ 43], 50.00th=[ 44], 60.00th=[ 45],
| 70.00th=[ 47], 80.00th=[ 50], 90.00th=[ 58], 95.00th=[ 61],
| 99.00th=[ 69], 99.50th=[ 73], 99.90th=[ 91], 99.95th=[ 103],
| 99.99th=[ 129]
bw ( KiB/s): min=207038, max=338144, per=100.00%, avg=286710.62, stdev=517.12, samples=5664
iops : min=51742, max=84521, avg=71660.22, stdev=129.27, samples=5664
lat (usec) : 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
lat (msec) : 2=0.01%, 4=0.01%, 10=0.05%, 20=0.75%, 50=80.14%
lat (msec) : 100=18.99%, 250=0.05%
cpu : usr=0.22%, sys=0.51%, ctx=209650, majf=0, minf=3741
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=4300515,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64Run status group 0 (all jobs):
READ: bw=280MiB/s (293MB/s), 280MiB/s-280MiB/s (293MB/s-293MB/s), io=16.4GiB (17.6GB), run=60039-60039msec
Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
RANDREAD: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
...
fio-3.27
Starting 48 processes
Jobs: 48 (f=48): [r(48)][100.0%][r=293MiB/s][r=75.1k IOPS][eta 00m:00s]
RANDREAD: (groupid=0, jobs=48): err= 0: pid=7266: Thu Jan 27 18:26:14 2022
read: IOPS=68.4k, BW=267MiB/s (280MB/s)(15.7GiB/60037msec)
slat (nsec): min=961, max=94194k, avg=8621.80, stdev=342824.46
clat (usec): min=42, max=216758, avg=44869.36, stdev=11485.17
lat (usec): min=694, max=216762, avg=44878.09, stdev=11505.95
clat percentiles (msec):
| 1.00th=[ 22], 5.00th=[ 28], 10.00th=[ 30], 20.00th=[ 34],
| 30.00th=[ 42], 40.00th=[ 44], 50.00th=[ 45], 60.00th=[ 47],
| 70.00th=[ 49], 80.00th=[ 54], 90.00th=[ 60], 95.00th=[ 63],
| 99.00th=[ 77], 99.50th=[ 88], 99.90th=[ 107], 99.95th=[ 113],
| 99.99th=[ 138]
bw ( KiB/s): min=184285, max=333915, per=100.00%, avg=273874.23, stdev=543.14, samples=5664
iops : min=46054, max=83457, avg=68452.00, stdev=135.77, samples=5664
lat (usec) : 50=0.01%, 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%
lat (usec) : 1000=0.01%
lat (msec) : 2=0.01%, 4=0.01%, 10=0.10%, 20=0.83%, 50=73.84%
lat (msec) : 100=25.03%, 250=0.19%
cpu : usr=0.22%, sys=0.50%, ctx=206236, majf=0, minf=3699
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=4108428,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64Run status group 0 (all jobs):
READ: bw=267MiB/s (280MB/s), 267MiB/s-267MiB/s (280MB/s-280MB/s), io=15.7GiB (16.8GB), run=60037-60037msec
Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
RANDREAD: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
...
fio-3.27
Starting 48 processes
Jobs: 48 (f=48): [r(48)][100.0%][r=290MiB/s][r=74.3k IOPS][eta 00m:00s]
RANDREAD: (groupid=0, jobs=48): err= 0: pid=7316: Thu Jan 27 18:27:14 2022
read: IOPS=67.1k, BW=262MiB/s (275MB/s)(15.4GiB/60036msec)
slat (nsec): min=951, max=55729k, avg=4035.52, stdev=113426.11
clat (usec): min=210, max=124444, avg=45794.93, stdev=10681.33
lat (usec): min=569, max=160416, avg=45799.08, stdev=10682.15
clat percentiles (msec):
| 1.00th=[ 26], 5.00th=[ 29], 10.00th=[ 31], 20.00th=[ 38],
| 30.00th=[ 43], 40.00th=[ 44], 50.00th=[ 46], 60.00th=[ 47],
| 70.00th=[ 50], 80.00th=[ 56], 90.00th=[ 61], 95.00th=[ 64],
| 99.00th=[ 72], 99.50th=[ 77], 99.90th=[ 86], 99.95th=[ 92],
| 99.99th=[ 105]
bw ( KiB/s): min=215248, max=327081, per=100.00%, avg=268405.62, stdev=452.79, samples=5664
iops : min=53792, max=81752, avg=67085.35, stdev=113.19, samples=5664
lat (usec) : 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
lat (msec) : 2=0.01%, 4=0.01%, 10=0.04%, 20=0.48%, 50=70.86%
lat (msec) : 100=28.60%, 250=0.02%
cpu : usr=0.21%, sys=0.48%, ctx=205998, majf=0, minf=3682
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=4026005,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64Run status group 0 (all jobs):
READ: bw=262MiB/s (275MB/s), 262MiB/s-262MiB/s (275MB/s-275MB/s), io=15.4GiB (16.5GB), run=60036-60036msec
Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
RANDREAD: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
...
fio-3.27
Starting 48 processes
Jobs: 48 (f=48): [r(48)][100.0%][r=293MiB/s][r=75.1k IOPS][eta 00m:00s]
RANDREAD: (groupid=0, jobs=48): err= 0: pid=7369: Thu Jan 27 18:28:15 2022
read: IOPS=72.2k, BW=282MiB/s (296MB/s)(16.5GiB/60036msec)
slat (nsec): min=951, max=130905k, avg=13129.16, stdev=481989.11
clat (usec): min=39, max=213166, avg=42523.90, stdev=11610.80
lat (usec): min=112, max=213169, avg=42537.15, stdev=11642.39
clat percentiles (msec):
| 1.00th=[ 15], 5.00th=[ 28], 10.00th=[ 29], 20.00th=[ 32],
| 30.00th=[ 39], 40.00th=[ 42], 50.00th=[ 44], 60.00th=[ 45],
| 70.00th=[ 47], 80.00th=[ 50], 90.00th=[ 58], 95.00th=[ 62],
| 99.00th=[ 78], 99.50th=[ 85], 99.90th=[ 101], 99.95th=[ 110],
| 99.99th=[ 140]
bw ( KiB/s): min=186271, max=362505, per=100.00%, avg=289027.55, stdev=639.85, samples=5664
iops : min=46554, max=90618, avg=72239.86, stdev=159.96, samples=5664
lat (usec) : 50=0.01%, 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%
lat (usec) : 1000=0.01%
lat (msec) : 2=0.01%, 4=0.03%, 10=0.19%, 20=1.48%, 50=79.68%
lat (msec) : 100=18.52%, 250=0.10%
cpu : usr=0.23%, sys=0.51%, ctx=209830, majf=0, minf=3679
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=4334765,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64Run status group 0 (all jobs):
READ: bw=282MiB/s (296MB/s), 282MiB/s-282MiB/s (296MB/s-296MB/s), io=16.5GiB (17.8GB), run=60036-60036msec
Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
RANDREAD: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
...
fio-3.27
Starting 48 processes
Jobs: 48 (f=48): [r(48)][100.0%][r=252MiB/s][r=64.5k IOPS][eta 00m:00s]
RANDREAD: (groupid=0, jobs=48): err= 0: pid=7420: Thu Jan 27 18:29:15 2022
read: IOPS=64.9k, BW=254MiB/s (266MB/s)(14.9GiB/60041msec)
slat (nsec): min=962, max=140336k, avg=8685.85, stdev=361027.29
clat (usec): min=22, max=236662, avg=47307.13, stdev=11546.15
lat (usec): min=158, max=236664, avg=47315.97, stdev=11566.36
clat percentiles (msec):
| 1.00th=[ 25], 5.00th=[ 29], 10.00th=[ 32], 20.00th=[ 41],
| 30.00th=[ 44], 40.00th=[ 45], 50.00th=[ 47], 60.00th=[ 48],
| 70.00th=[ 52], 80.00th=[ 58], 90.00th=[ 62], 95.00th=[ 65],
| 99.00th=[ 79], 99.50th=[ 89], 99.90th=[ 109], 99.95th=[ 115],
| 99.99th=[ 138]
bw ( KiB/s): min=184485, max=305435, per=100.00%, avg=259804.55, stdev=387.19, samples=5664
iops : min=46101, max=76340, avg=64934.75, stdev=96.82, samples=5664
lat (usec) : 50=0.01%, 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%
lat (usec) : 1000=0.01%
lat (msec) : 2=0.01%, 4=0.02%, 10=0.10%, 20=0.61%, 50=66.06%
lat (msec) : 100=32.99%, 250=0.22%
cpu : usr=0.22%, sys=0.47%, ctx=204069, majf=0, minf=3728
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=3896888,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64Run status group 0 (all jobs):
READ: bw=254MiB/s (266MB/s), 254MiB/s-254MiB/s (266MB/s-266MB/s), io=14.9GiB (16.0GB), run=60041-60041msec
Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
RANDREAD: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
...
fio-3.27
Starting 48 processes
Jobs: 48 (f=48): [r(48)][100.0%][r=253MiB/s][r=64.9k IOPS][eta 00m:00s]
RANDREAD: (groupid=0, jobs=48): err= 0: pid=7470: Thu Jan 27 18:30:15 2022
read: IOPS=64.9k, BW=254MiB/s (266MB/s)(14.9GiB/60041msec)
slat (nsec): min=962, max=47238k, avg=3900.68, stdev=102522.05
clat (usec): min=211, max=139068, avg=47297.25, stdev=10569.79
lat (usec): min=1274, max=147957, avg=47301.26, stdev=10569.35
clat percentiles (msec):
| 1.00th=[ 27], 5.00th=[ 30], 10.00th=[ 32], 20.00th=[ 41],
| 30.00th=[ 44], 40.00th=[ 45], 50.00th=[ 47], 60.00th=[ 48],
| 70.00th=[ 52], 80.00th=[ 58], 90.00th=[ 62], 95.00th=[ 65],
| 99.00th=[ 73], 99.50th=[ 77], 99.90th=[ 86], 99.95th=[ 93],
| 99.99th=[ 106]
bw ( KiB/s): min=217716, max=298751, per=100.00%, avg=259905.72, stdev=306.13, samples=5664
iops : min=54409, max=74668, avg=64959.92, stdev=76.55, samples=5664
lat (usec) : 250=0.01%, 500=0.01%, 1000=0.01%
lat (msec) : 2=0.01%, 4=0.01%, 10=0.03%, 20=0.30%, 50=66.07%
lat (msec) : 100=33.57%, 250=0.02%
cpu : usr=0.21%, sys=0.47%, ctx=204669, majf=0, minf=3815
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=3898180,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64Run status group 0 (all jobs):
READ: bw=254MiB/s (266MB/s), 254MiB/s-254MiB/s (266MB/s-266MB/s), io=14.9GiB (16.0GB), run=60041-60041msec
Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
RANDREAD: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
...
fio-3.27
Starting 48 processes
Jobs: 48 (f=48): [r(48)][100.0%][r=252MiB/s][r=64.6k IOPS][eta 00m:00s]
RANDREAD: (groupid=0, jobs=48): err= 0: pid=7520: Thu Jan 27 18:31:16 2022
read: IOPS=64.9k, BW=254MiB/s (266MB/s)(14.9GiB/60045msec)
slat (nsec): min=952, max=62060k, avg=5386.66, stdev=203889.44
clat (usec): min=55, max=152052, avg=47291.79, stdev=10952.81
lat (usec): min=656, max=172623, avg=47297.29, stdev=10959.29
clat percentiles (msec):
| 1.00th=[ 27], 5.00th=[ 30], 10.00th=[ 32], 20.00th=[ 41],
| 30.00th=[ 44], 40.00th=[ 45], 50.00th=[ 47], 60.00th=[ 48],
| 70.00th=[ 52], 80.00th=[ 58], 90.00th=[ 62], 95.00th=[ 65],
| 99.00th=[ 74], 99.50th=[ 80], 99.90th=[ 102], 99.95th=[ 110],
| 99.99th=[ 125]
bw ( KiB/s): min=196644, max=301084, per=100.00%, avg=259941.97, stdev=341.90, samples=5664
iops : min=49145, max=75254, avg=64969.79, stdev=85.50, samples=5664
lat (usec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
lat (msec) : 2=0.01%, 4=0.01%, 10=0.06%, 20=0.42%, 50=66.06%
lat (msec) : 100=33.34%, 250=0.11%
cpu : usr=0.21%, sys=0.47%, ctx=204667, majf=0, minf=3732
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=3898567,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64Run status group 0 (all jobs):
READ: bw=254MiB/s (266MB/s), 254MiB/s-254MiB/s (266MB/s-266MB/s), io=14.9GiB (16.0GB), run=60045-60045msec
Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
RANDREAD: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
...
fio-3.27
Starting 48 processes
Jobs: 48 (f=48): [r(48)][100.0%][r=291MiB/s][r=74.5k IOPS][eta 00m:00s]
RANDREAD: (groupid=0, jobs=48): err= 0: pid=7573: Thu Jan 27 18:32:16 2022
read: IOPS=69.3k, BW=271MiB/s (284MB/s)(15.9GiB/60037msec)
slat (nsec): min=962, max=119889k, avg=6570.81, stdev=271570.85
clat (nsec): min=1784, max=265702k, avg=44292797.24, stdev=11045197.81
lat (usec): min=937, max=265705, avg=44299.48, stdev=11056.26
clat percentiles (msec):
| 1.00th=[ 24], 5.00th=[ 28], 10.00th=[ 30], 20.00th=[ 34],
| 30.00th=[ 42], 40.00th=[ 43], 50.00th=[ 45], 60.00th=[ 46],
| 70.00th=[ 48], 80.00th=[ 53], 90.00th=[ 59], 95.00th=[ 63],
| 99.00th=[ 73], 99.50th=[ 79], 99.90th=[ 99], 99.95th=[ 107],
| 99.99th=[ 127]
bw ( KiB/s): min=196140, max=336434, per=100.00%, avg=277459.75, stdev=548.59, samples=5664
iops : min=49018, max=84092, avg=69348.58, stdev=137.14, samples=5664
lat (usec) : 2=0.01%, 50=0.01%, 100=0.01%, 250=0.01%, 500=0.01%
lat (usec) : 750=0.01%, 1000=0.01%
lat (msec) : 2=0.01%, 4=0.01%, 10=0.07%, 20=0.77%, 50=75.42%
lat (msec) : 100=23.63%, 250=0.09%, 500=0.01%
cpu : usr=0.21%, sys=0.49%, ctx=207302, majf=0, minf=3740
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=4162006,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64Run status group 0 (all jobs):
READ: bw=271MiB/s (284MB/s), 271MiB/s-271MiB/s (284MB/s-284MB/s), io=15.9GiB (17.0GB), run=60037-60037msec
Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
RANDREAD: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
...
fio-3.27
Starting 48 processes
Jobs: 48 (f=48): [r(48)][100.0%][r=255MiB/s][r=65.2k IOPS][eta 00m:00s]
RANDREAD: (groupid=0, jobs=48): err= 0: pid=7627: Thu Jan 27 18:33:16 2022
read: IOPS=64.7k, BW=253MiB/s (265MB/s)(14.8GiB/60041msec)
slat (nsec): min=941, max=90920k, avg=7000.17, stdev=307653.58
clat (usec): min=80, max=231950, avg=47434.66, stdev=11193.75
lat (usec): min=580, max=231953, avg=47441.78, stdev=11207.19
clat percentiles (msec):
| 1.00th=[ 26], 5.00th=[ 30], 10.00th=[ 32], 20.00th=[ 41],
| 30.00th=[ 44], 40.00th=[ 45], 50.00th=[ 47], 60.00th=[ 49],
| 70.00th=[ 52], 80.00th=[ 58], 90.00th=[ 62], 95.00th=[ 65],
| 99.00th=[ 77], 99.50th=[ 83], 99.90th=[ 105], 99.95th=[ 113],
| 99.99th=[ 146]
bw ( KiB/s): min=187097, max=307908, per=100.00%, avg=259134.59, stdev=371.45, samples=5664
iops : min=46756, max=76958, avg=64767.39, stdev=92.88, samples=5664
lat (usec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
lat (msec) : 2=0.01%, 4=0.01%, 10=0.07%, 20=0.50%, 50=65.49%
lat (msec) : 100=33.78%, 250=0.14%
cpu : usr=0.20%, sys=0.47%, ctx=204191, majf=0, minf=3734
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=3886625,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64Run status group 0 (all jobs):
READ: bw=253MiB/s (265MB/s), 253MiB/s-253MiB/s (265MB/s-265MB/s), io=14.8GiB (15.9GB), run=60041-60041msec
Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
Signature · 1 line
2.29.0
PATCH v2 1/2 nvmet: use i_size_read() to set size for file-ns
From: Chaitanya Kulkarni <[email protected]>
Instead of calling vfs_getattr() use i_size_read() to read the size of file so we can read the size of not only file type but also block type with one call. This is needed to implement buffered_io support for the NVMeOF block device backend.
We also change return type of function nvmet_file_ns_revalidate() from int to void, since this function does not return any meaning value.
--- drivers/nvme/target/io-cmd-file.c | 17 ++++------------- drivers/nvme/target/nvmet.h | 2 +- 2 files changed, 5 insertions(+), 14 deletions(-)
drivers/nvme/target/io-cmd-file.c +4 -13
@@ -14,16 +14,9 @@#define NVMET_MAX_MPOOL_BVEC 16#define NVMET_MIN_MPOOL_OBJ 16 -int nvmet_file_ns_revalidate (struct nvmet_ns *ns) +void nvmet_file_ns_revalidate (struct nvmet_ns *ns) { -struct kstat stat; -int ret; - - ret =vfs_getattr (&ns->file->f_path, &stat,STATX_SIZE , -AT_STATX_FORCE_SYNC ); -if (!ret) - ns->size = stat.size; -return ret; + ns->size =i_size_read (ns->file->f_mapping->host); }void nvmet_file_ns_disable (struct nvmet_ns *ns)
@@ -43,7 +36,7 @@ void nvmet_file_ns_disable(struct nvmet_ns *ns)int nvmet_file_ns_enable (struct nvmet_ns *ns) {int flags =O_RDWR |O_LARGEFILE ; -int ret; +int ret =0 ;if (!ns->buffered_io) flags |=O_DIRECT ;
@@ -57,9 +50,7 @@ int nvmet_file_ns_enable(struct nvmet_ns *ns)return ret; } - ret =nvmet_file_ns_revalidate (ns); -if (ret) - goto err; +nvmet_file_ns_revalidate (ns);/* * i_blkbits can be greater than the universally accepted upper bound,
drivers/nvme/target/nvmet.h +1 -1
@@ -541,7 +541,7 @@ u16 nvmet_bdev_flush(struct nvmet_req *req);u16 nvmet_file_flush (struct nvmet_req *req);void nvmet_ns_changed (struct nvmet_subsys *subsys,u32 nsid);void nvmet_bdev_ns_revalidate (struct nvmet_ns *ns); -int nvmet_file_ns_revalidate (struct nvmet_ns *ns); +void nvmet_file_ns_revalidate (struct nvmet_ns *ns);void nvmet_ns_revalidate (struct nvmet_ns *ns);u16 blk_to_nvme_status (struct nvmet_req *req,blk_status_t blk_sts);
Signature · 1 line
2.29.0
PATCH v2 2/2 nvmet: allow bdev in buffered_io mode
In reply to Chaitanya KulkarniFrom: Chaitanya Kulkarni <[email protected]>
Allow block device to be configured in the buffered I/O mode by using the file backend. In this way now we can use cache for the block device namespace which shows significant performance improvement.
We update the block device ns enable function and return early when buffered_io flag is set.
--- drivers/nvme/target/io-cmd-bdev.c | 3 +++ 1 file changed, 3 insertions(+)
drivers/nvme/target/io-cmd-bdev.c +3 -0
@@ -76,6 +76,9 @@ int nvmet_bdev_ns_enable(struct nvmet_ns *ns) {int ret; +if (ns->buffered_io) +return -ENOTBLK ; + ns->bdev =blkdev_get_by_path (ns->device_path,FMODE_READ |FMODE_WRITE ,NULL );if (IS_ERR (ns->bdev)) {
Signature · 1 line
2.29.0
Christoph Hellwig
maintainerRe: [PATCH V2 2/2] nvmet: allow bdev in buffered_io mode
drivers/nvme/target/io-cmd-bdev.c int nvmet_bdev_ns_enable(struct nvmet_ns *ns) · line 80
Quoted 2 lines · Chaitanya Kulkarni
drivers/nvme/target/io-cmd-bdev.c +2
-0+if (ns->buffered_io) +return -ENOTBLK ;
Maybe add a comment here that we want to force a fallback to the file backend for buffered I/O here as this check on it's own isn't totally obvious.
Otherwise the series looks good to me.
Re: [PATCH V2 2/2] nvmet: allow bdev in buffered_io mode
Quoted 4 lines · Christoph Hellwig
External email: Use caution opening links or attachments
On Fri, Jan 28, 2022 at 12:51:48AM -0800, Chaitanya Kulkarni wrote:
Quoted 9 lines
drivers/nvme/target/io-cmd-bdev.c int nvmet_bdev_ns_enable(struct nvmet_ns *ns) · line 80
Quoted 2 lines
drivers/nvme/target/io-cmd-bdev.c +2
-0+if (ns->buffered_io) +return -ENOTBLK ;Maybe add a comment here that we want to force a fallback to the file backend for buffered I/O here as this check on it's own isn't totally obvious.
Otherwise the series looks good to me.
Okay will sendout a v3 with added comment, I guess had it in V1 forgot in V2.