Skip to contentSkip to conversation

nvmet: add buffered_io support for bdev-ns

PATCH Superseded Chaitanya Kulkarni 3 participants 11 messages Jan 19, 2022 – Jan 27, 2022

Subsystems (2)

v1 of 3 Superseded Latest is v3 →

lore
Share & tools
lore
#0 lore raw

cover nvmet: add buffered_io support for bdev-ns

From: Chaitanya Kulkarni <[email protected]>

Hi,

In current implementation like NVMeOF target file backed namespace we don't support buffered_io for block device backed namesapce. This allows user to configure block device backend in buffered_io mode so user can get the benefits of caching.

Please note that using this combination host needs to periodically sync between cache and storage or set the system cache parameters accordingly. NVMeOF target is not responsible for data consistency between cache and storage.

First two patches create helper functions which are used in the third patch to avoid code duplication.

Following is the summary of fio performance numbers, see complete log at the end of this cover-letter.

Data verification and fio performance numbers :-

1. NVMeOF Target ns device_path=/dev/nvme0n1 & buffered_io=0 :-

fio Verify job :-
----------------------
read: IOPS=32.1k, BW=125MiB/s (131MB/s)(506MiB/4042msec)
write: IOPS=27.3k, BW=107MiB/s (112MB/s)(800MiB/7489msec)

fio Random Write :-
----------------------
write: IOPS=69.9k, BW=273MiB/s (286MB/s)(16.0GiB/60003msec)
write: IOPS=69.6k, BW=272MiB/s (285MB/s)(15.9GiB/60003msec)
write: IOPS=70.1k, BW=274MiB/s (287MB/s)(16.1GiB/60002msec)
write: IOPS=69.9k, BW=273MiB/s (286MB/s)(16.0GiB/60003msec)
write: IOPS=70.1k, BW=274MiB/s (287MB/s)(16.1GiB/60004msec)
write: IOPS=69.9k, BW=273MiB/s (286MB/s)(16.0GiB/60003msec)
write: IOPS=70.0k, BW=273MiB/s (287MB/s)(16.0GiB/60003msec)
write: IOPS=69.9k, BW=273MiB/s (287MB/s)(16.0GiB/60003msec)
write: IOPS=70.3k, BW=274MiB/s (288MB/s)(16.1GiB/60003msec)
write: IOPS=70.1k, BW=274MiB/s (287MB/s)(16.1GiB/60003msec)

fio Random Read :-
----------------------
read: IOPS=74.6k, BW=292MiB/s (306MB/s)(17.1GiB/60034msec)
read: IOPS=67.4k, BW=263MiB/s (276MB/s)(15.4GiB/60035msec)
read: IOPS=74.2k, BW=290MiB/s (304MB/s)(17.0GiB/60033msec)
read: IOPS=73.4k, BW=287MiB/s (301MB/s)(16.8GiB/60034msec)
read: IOPS=75.0k, BW=293MiB/s (307MB/s)(17.2GiB/60036msec)
read: IOPS=75.1k, BW=293MiB/s (308MB/s)(17.2GiB/60034msec)
read: IOPS=73.7k, BW=288MiB/s (302MB/s)(16.9GiB/60036msec)
read: IOPS=74.2k, BW=290MiB/s (304MB/s)(17.0GiB/60033msec)
read: IOPS=75.9k, BW=296MiB/s (311MB/s)(17.4GiB/60035msec)
read: IOPS=72.3k, BW=282MiB/s (296MB/s)(16.6GiB/60037msec)

2. NVMeOF Target ns device_path=/dev/nvme0n1 & buffered_io=1 :-

fio Verify job :-
----------------------
read: IOPS=163k, BW=638MiB/s (669MB/s)(506MiB/793msec)
write: IOPS=138k, BW=540MiB/s (566MB/s)(800MiB/1481msec);

fio Random Write :-
----------------------
write: IOPS=4129k, BW=15.7GiB/s (16.9GB/s)(945GiB/60001msec)
write: IOPS=4010k, BW=15.3GiB/s (16.4GB/s)(918GiB/60002msec)
write: IOPS=4170k, BW=15.9GiB/s (17.1GB/s)(955GiB/60001msec)
write: IOPS=4064k, BW=15.5GiB/s (16.6GB/s)(930GiB/60002msec)
write: IOPS=4078k, BW=15.6GiB/s (16.7GB/s)(933GiB/60002msec)
write: IOPS=4078k, BW=15.6GiB/s (16.7GB/s)(933GiB/60002msec)
write: IOPS=4136k, BW=15.8GiB/s (16.9GB/s)(947GiB/60002msec)
write: IOPS=3885k, BW=14.8GiB/s (15.9GB/s)(889GiB/60002msec)
write: IOPS=4020k, BW=15.3GiB/s (16.5GB/s)(920GiB/60002msec)
write: IOPS=4136k, BW=15.8GiB/s (16.9GB/s)(947GiB/60002msec)

fio Random Read :-
----------------------
read: IOPS=4859k, BW=18.5GiB/s (19.9GB/s)(1112GiB/60001msec)
read: IOPS=4862k, BW=18.5GiB/s (19.9GB/s)(1113GiB/60001msec)
read: IOPS=5134k, BW=19.6GiB/s (21.0GB/s)(1175GiB/60002msec)
read: IOPS=4860k, BW=18.5GiB/s (19.9GB/s)(1112GiB/60002msec)
read: IOPS=4756k, BW=18.1GiB/s (19.5GB/s)(1089GiB/60002msec)
read: IOPS=4706k, BW=18.0GiB/s (19.3GB/s)(1077GiB/60001msec)
read: IOPS=5039k, BW=19.2GiB/s (20.6GB/s)(1153GiB/60002msec)
read: IOPS=4670k, BW=17.8GiB/s (19.1GB/s)(1069GiB/60002msec)
read: IOPS=4882k, BW=18.6GiB/s (20.0GB/s)(1117GiB/60002msec)
read: IOPS=4861k, BW=18.5GiB/s (19.9GB/s)(1113GiB/60002msec)

-ck

Chaitanya Kulkarni (3):
nvmet: add helper to set ns->blkshift value
nvmet: add helper to open block device
nvmet: allow bdev in buffered_io mode

 drivers/nvme/target/io-cmd-bdev.c | 22 ++++++++++++++++--
 drivers/nvme/target/io-cmd-file.c | 38 ++++++++++++++++++++++++-------
 drivers/nvme/target/nvmet.h       |  1 +
 3 files changed, 51 insertions(+), 10 deletions(-)

* FIO Verify, Random Read. Random write log :-
----------------------------------------------

1. Verbose log Block Device with buffered_io = 0 :-
---------------------------------------------------

root@dev nvme (nvme-5.17) # sh bdev-buffered-io-test.sh
+ nvme disconnect -n testnqn
NQN:testnqn disconnected 0 controller(s)

real 0m0.001s
user 0m0.000s
sys 0m0.001s
+ for i in /sys/kernel/config/nvmet/subsystems/${NQN}/namespaces/*
+ echo 0
./delete.sh: line 9: /sys/kernel/config/nvmet/subsystems/testnqn/namespaces/*/enable: No such file or directory
+ rmdir '/sys/kernel/config/nvmet/subsystems/testnqn/namespaces/*'
rmdir: failed to remove '/sys/kernel/config/nvmet/subsystems/testnqn/namespaces/*': No such file or directory
+ rm -fr /sys/kernel/config/nvmet/ports/1/subsystems/testnqn
+ rmdir /sys/kernel/config/nvmet/ports/1
rmdir: failed to remove '/sys/kernel/config/nvmet/ports/1': No such file or directory
+ rmdir /sys/kernel/config/nvmet/subsystems/testnqn
rmdir: failed to remove '/sys/kernel/config/nvmet/subsystems/testnqn': No such file or directory
+ rmdir 'config/nullb/nullb*'
rmdir: failed to remove 'config/nullb/nullb*': No such file or directory
+ umount /mnt/nvme0n1
umount: /mnt/nvme0n1: no mount point specified.
+ umount /mnt/backend
umount: /mnt/backend: not mounted.
+ modprobe -r nvme_loop
+ modprobe -r nvme_fabrics
+ modprobe -r nvmet
+ modprobe -r nvme
+ modprobe -r null_blk
+ tree /sys/kernel/config
/sys/kernel/config

0 directories, 0 files
+ umount /mnt/nvme0n1
+ clear_dmesg
./compile_nvme.sh: line 3: clear_dmesg: command not found
umount: /mnt/nvme0n1: no mount point specified.
+ modprobe -r nvme-fabrics
+ modprobe -r nvme_loop
+ modprobe -r nvmet
+ modprobe -r nvme
+ sleep 1
+ modprobe -r nvme-core
+ lsmod
+ grep nvme
+ sleep 1
+ git diff
+ sleep 1
++ nproc
+ make -j 48 M=drivers/nvme/ modules
+ HOST=drivers/nvme/host
+ TARGET=drivers/nvme/target
++ uname -r
+ HOST_DEST=/lib/modules/5.16.0-rc3nvme+/kernel/drivers/nvme/host/
++ uname -r
+ TARGET_DEST=/lib/modules/5.16.0-rc3nvme+/kernel/drivers/nvme/target/
+ cp drivers/nvme/host/nvme-core.ko drivers/nvme/host/nvme-fabrics.ko drivers/nvme/host/nvme-fc.ko drivers/nvme/host/nvme.ko drivers/nvme/host/nvme-rdma.ko drivers/nvme/host/nvme-tcp.ko /lib/modules/5.16.0-rc3nvme+/kernel/drivers/nvme/host//
+ cp drivers/nvme/target/nvme-fcloop.ko drivers/nvme/target/nvme-loop.ko drivers/nvme/target/nvmet-fc.ko drivers/nvme/target/nvmet.ko drivers/nvme/target/nvmet-rdma.ko drivers/nvme/target/nvmet-tcp.ko /lib/modules/5.16.0-rc3nvme+/kernel/drivers/nvme/target//
+ ls -lrth /lib/modules/5.16.0-rc3nvme+/kernel/drivers/nvme/host/ /lib/modules/5.16.0-rc3nvme+/kernel/drivers/nvme/target//
/lib/modules/5.16.0-rc3nvme+/kernel/drivers/nvme/host/:
total 6.6M

-rw-r--r--. 1 root root  2.7M Jan 18 13:22 nvme-core.ko
-rw-r--r--. 1 root root  447K Jan 18 13:22 nvme-fabrics.ko
-rw-r--r--. 1 root root  965K Jan 18 13:22 nvme-fc.ko
-rw-r--r--. 1 root root  732K Jan 18 13:22 nvme.ko

-rw-r--r--. 1 root root 1021K Jan 18 13:22 nvme-rdma.ko -rw-r--r--. 1 root root 897K Jan 18 13:22 nvme-tcp.ko

/lib/modules/5.16.0-rc3nvme+/kernel/drivers/nvme/target//:
total 6.7M
-rw-r--r--. 1 root root 498K Jan 18 13:22 nvme-fcloop.ko -rw-r--r--. 1 root root 438K Jan 18 13:22 nvme-loop.ko -rw-r--r--. 1 root root 767K Jan 18 13:22 nvmet-fc.ko -rw-r--r--. 1 root root 3.3M Jan 18 13:22 nvmet.ko -rw-r--r--. 1 root root 992K Jan 18 13:22 nvmet-rdma.ko -rw-r--r--. 1 root root 768K Jan 18 13:22 nvmet-tcp.ko
+ modprobe nvme
+ nvme disconnect -n testnqn
NQN:testnqn disconnected 0 controller(s)

real 0m0.001s
user 0m0.000s
sys 0m0.001s
+ for i in /sys/kernel/config/nvmet/subsystems/${NQN}/namespaces/*
+ echo 0
./delete.sh: line 9: /sys/kernel/config/nvmet/subsystems/testnqn/namespaces/*/enable: No such file or directory
+ rmdir '/sys/kernel/config/nvmet/subsystems/testnqn/namespaces/*'
rmdir: failed to remove '/sys/kernel/config/nvmet/subsystems/testnqn/namespaces/*': No such file or directory
+ rm -fr /sys/kernel/config/nvmet/ports/1/subsystems/testnqn
+ rmdir /sys/kernel/config/nvmet/ports/1
rmdir: failed to remove '/sys/kernel/config/nvmet/ports/1': No such file or directory
+ rmdir /sys/kernel/config/nvmet/subsystems/testnqn
rmdir: failed to remove '/sys/kernel/config/nvmet/subsystems/testnqn': No such file or directory
+ rmdir 'config/nullb/nullb*'
rmdir: failed to remove 'config/nullb/nullb*': No such file or directory
+ umount /mnt/nvme0n1
umount: /mnt/nvme0n1: no mount point specified.
+ umount /mnt/backend
umount: /mnt/backend: not mounted.
+ modprobe -r nvme_loop
+ modprobe -r nvme_fabrics
+ modprobe -r nvmet
+ modprobe -r nvme
+ modprobe -r null_blk
+ tree /sys/kernel/config
/sys/kernel/config

0 directories, 0 files
++ NN=1
++ NQN=testnqn
++ let NR_DEVICES=NN+1
++ modprobe -r null_blk
++ modprobe -r nvme
++ modprobe null_blk nr_devices=0
++ modprobe nvme
++ modprobe nvme-fabrics
++ modprobe nvmet
++ modprobe nvme-loop
++ dmesg -c
++ sleep 2
++ tree /sys/kernel/config
/sys/kernel/config

├── nullb
│   └── features
└── nvmet
    ├── hosts
    ├── ports
    └── subsystems

5 directories, 1 file
++ mkdir /sys/kernel/config/nvmet/subsystems/testnqn
++ mkdir /sys/kernel/config/nvmet/subsystems/testnqn/namespaces/1
++ echo -n /dev/nvme0n1
++ cat /sys/kernel/config/nvmet/subsystems/testnqn/namespaces/1/device_path /dev/nvme0n1
++ echo 0
++ cat /sys/kernel/config/nvmet/subsystems/testnqn/namespaces/1/buffered_io
0
++ echo 1
++ mkdir /sys/kernel/config/nvmet/ports/1/
++ echo -n loop
++ echo -n 1
++ ln -s /sys/kernel/config/nvmet/subsystems/testnqn /sys/kernel/config/nvmet/ports/1/subsystems/ ++ sleep 1
++ echo transport=loop,nqn=testnqn
++ sleep 1
++ mount
++ column -t
++ grep nvme
++ '[' 1 ']'
+++ wc -l
+++ ls -l /dev/nvme1 /dev/nvme1n1
++ cnt=2
++ echo 2
2
++ '[' 2 -gt 1 ']'
++ break
++ dmesg -c

[55445.819019] nvme nvme0: 48/0/0 default/read/poll queues
[55447.805257] nvmet: adding nsid 1 to subsystem testnqn
[55448.815307] nvmet: creating nvm controller 1 for subsystem testnqn for NQN nqn.2014-08.org.nvmexpress:uuid:3aed9b58-95ea-477b-ac51-d87bc7d10326.
[55448.815470] nvme nvme1: creating 48 I/O queues.
[55448.819432] nvme nvme1: new ctrl: "testnqn"

++ sleep 5
write-and-verify: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=4 fio-3.27
Starting 1 process
Jobs: 1 (f=1): [V(1)][86.7%][r=102MiB/s][r=26.2k IOPS][eta 00m:02s]
write-and-verify: (groupid=0, jobs=1): err= 0: pid=11490: Tue Jan 18 13:22:52 2022
read: IOPS=27.7k, BW=108MiB/s (113MB/s)(506MiB/4680msec)

    slat (nsec): min=1052, max=1431.4k, avg=4089.33, stdev=9877.17
    clat (usec): min=12, max=3429, avg=138.18, stdev=75.13
     lat (usec): min=44, max=3433, avg=142.39, stdev=75.17
    clat percentiles (usec):
     |  1.00th=[   60],  5.00th=[   78], 10.00th=[   88], 20.00th=[  101],
     | 30.00th=[  110], 40.00th=[  114], 50.00th=[  117], 60.00th=[  122],
     | 70.00th=[  139], 80.00th=[  196], 90.00th=[  212], 95.00th=[  223],
     | 99.00th=[  343], 99.50th=[  457], 99.90th=[  955], 99.95th=[ 1614],
     | 99.99th=[ 1991]
  write: IOPS=24.8k, BW=96.7MiB/s (101MB/s)(800MiB/8273msec); 0 zone resets
    slat (usec): min=2, max=2817, avg=10.17, stdev=12.87
    clat (usec): min=6, max=3742, avg=150.46, stdev=85.68
     lat (usec): min=49, max=3748, avg=160.74, stdev=85.53
    clat percentiles (usec):
     |  1.00th=[   59],  5.00th=[   79], 10.00th=[   90], 20.00th=[  108],
     | 30.00th=[  119], 40.00th=[  127], 50.00th=[  133], 60.00th=[  143],
     | 70.00th=[  159], 80.00th=[  186], 90.00th=[  229], 95.00th=[  253],
     | 99.00th=[  408], 99.50th=[  529], 99.90th=[  766], 99.95th=[ 1598],
     | 99.99th=[ 2999]
   bw (  KiB/s): min=54560, max=141576, per=97.33%, avg=96376.47, stdev=19561.14, samples=17
   iops        : min=13640, max=35394, avg=24094.12, stdev=4890.28, samples=17
  lat (usec)   : 10=0.01%, 20=0.02%, 50=0.29%, 100=16.34%, 250=79.47%
  lat (usec)   : 500=3.41%, 750=0.36%, 1000=0.02%
  lat (msec)   : 2=0.07%, 4=0.02%
  cpu          : usr=19.11%, sys=15.27%, ctx=122606, majf=0, minf=3054
  IO depths    : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=129572,204800,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4

Run status group 0 (all jobs):
READ: bw=108MiB/s (113MB/s), 108MiB/s-108MiB/s (113MB/s-113MB/s), io=506MiB (531MB), run=4680-4680msec
WRITE: bw=96.7MiB/s (101MB/s), 96.7MiB/s-96.7MiB/s (101MB/s-101MB/s), io=800MiB (839MB), run=8273-8273msec

Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
RANDREAD: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=8
...
fio-3.27
Starting 32 processes
Jobs: 25 (f=0): [E(2),f(5),E(1),f(3),E(1),f(3),E(1),f(7),E(1),f(2),E(1),f(5)][100.0%][w=259MiB/s][w=66.4k IOPS][eta 00m:00s]
RANDREAD: (groupid=0, jobs=32): err= 0: pid=12227: Tue Jan 18 13:23:52 2022
write: IOPS=68.8k, BW=269MiB/s (282MB/s)(15.7GiB/60003msec); 0 zone resets

    slat (nsec): min=1042, max=29831k, avg=12162.75, stdev=103918.07
    clat (nsec): min=511, max=38271k, avg=3708415.18, stdev=2189164.56
     lat (usec): min=47, max=38800, avg=3720.72, stdev=2186.96
    clat percentiles (usec):
     |  1.00th=[  652],  5.00th=[ 1090], 10.00th=[ 1418], 20.00th=[ 1926],
     | 30.00th=[ 2343], 40.00th=[ 2802], 50.00th=[ 3228], 60.00th=[ 3752],
     | 70.00th=[ 4359], 80.00th=[ 5211], 90.00th=[ 6587], 95.00th=[ 7898],
     | 99.00th=[10814], 99.50th=[12256], 99.90th=[15401], 99.95th=[16909],
     | 99.99th=[19530]
   bw (  KiB/s): min=217922, max=328001, per=100.00%, avg=275365.13, stdev=666.91, samples=3808
   iops        : min=54470, max=81992, avg=68834.13, stdev=166.73, samples=3808
  lat (nsec)   : 750=0.01%, 1000=0.01%
  lat (usec)   : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.03%
  lat (usec)   : 100=0.01%, 250=0.02%, 500=0.33%, 750=1.22%, 1000=2.24%
  lat (msec)   : 2=17.95%, 4=42.33%, 10=34.25%, 20=1.60%, 50=0.01%
  cpu          : usr=0.59%, sys=1.35%, ctx=825570, majf=0, minf=480
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,4126796,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
WRITE: bw=269MiB/s (282MB/s), 269MiB/s-269MiB/s (282MB/s-282MB/s), io=15.7GiB (16.9GB), run=60003-60003msec

Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
RANDREAD: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=8
...
fio-3.27
Starting 32 processes
Jobs: 32 (f=0): [f(32)][100.0%][w=271MiB/s][w=69.4k IOPS][eta 00m:00s]
RANDREAD: (groupid=0, jobs=32): err= 0: pid=12784: Tue Jan 18 13:24:53 2022
write: IOPS=69.3k, BW=271MiB/s (284MB/s)(15.9GiB/60003msec); 0 zone resets

    slat (nsec): min=1052, max=18635k, avg=8508.06, stdev=73079.65
    clat (nsec): min=761, max=33431k, avg=3684368.79, stdev=2159290.61
     lat (usec): min=45, max=33436, avg=3693.02, stdev=2156.87
    clat percentiles (usec):
     |  1.00th=[  652],  5.00th=[ 1090], 10.00th=[ 1418], 20.00th=[ 1909],
     | 30.00th=[ 2343], 40.00th=[ 2769], 50.00th=[ 3228], 60.00th=[ 3752],
     | 70.00th=[ 4359], 80.00th=[ 5211], 90.00th=[ 6521], 95.00th=[ 7832],
     | 99.00th=[10814], 99.50th=[11994], 99.90th=[15008], 99.95th=[16319],
     | 99.99th=[19006]
   bw (  KiB/s): min=218885, max=330648, per=100.00%, avg=277393.20, stdev=661.31, samples=3808
   iops        : min=54714, max=82657, avg=69342.65, stdev=165.34, samples=3808
  lat (nsec)   : 1000=0.01%
  lat (usec)   : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.02%
  lat (usec)   : 100=0.01%, 250=0.01%, 500=0.33%, 750=1.20%, 1000=2.28%
  lat (msec)   : 2=18.09%, 4=42.50%, 10=34.04%, 20=1.50%, 50=0.01%
  cpu          : usr=0.59%, sys=1.34%, ctx=799313, majf=0, minf=473
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,4157753,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
WRITE: bw=271MiB/s (284MB/s), 271MiB/s-271MiB/s (284MB/s-284MB/s), io=15.9GiB (17.0GB), run=60003-60003msec

Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
RANDREAD: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=8
...
fio-3.27
Starting 32 processes
Jobs: 32 (f=32): [w(32)][100.0%][w=268MiB/s][w=68.6k IOPS][eta 00m:00s]
RANDREAD: (groupid=0, jobs=32): err= 0: pid=12998: Tue Jan 18 13:25:53 2022
write: IOPS=65.0k, BW=254MiB/s (266MB/s)(14.9GiB/60003msec); 0 zone resets

    slat (nsec): min=1042, max=43641k, avg=109484.94, stdev=394081.46
    clat (nsec): min=632, max=55539k, avg=3829004.51, stdev=2354852.14
     lat (usec): min=51, max=55541, avg=3938.69, stdev=2356.96
    clat percentiles (usec):
     |  1.00th=[  594],  5.00th=[ 1090], 10.00th=[ 1434], 20.00th=[ 1958],
     | 30.00th=[ 2409], 40.00th=[ 2868], 50.00th=[ 3326], 60.00th=[ 3851],
     | 70.00th=[ 4490], 80.00th=[ 5407], 90.00th=[ 6849], 95.00th=[ 8291],
     | 99.00th=[11600], 99.50th=[13173], 99.90th=[17695], 99.95th=[20579],
     | 99.99th=[26870]
   bw (  KiB/s): min=193792, max=318358, per=100.00%, avg=260028.15, stdev=771.99, samples=3808
   iops        : min=48444, max=79586, avg=65002.81, stdev=192.98, samples=3808
  lat (nsec)   : 750=0.01%, 1000=0.01%
  lat (usec)   : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.17%
  lat (usec)   : 100=0.04%, 250=0.06%, 500=0.39%, 750=1.20%, 1000=2.13%
  lat (msec)   : 2=16.86%, 4=41.57%, 10=35.41%, 20=2.09%, 50=0.06%
  lat (msec)   : 100=0.01%
  cpu          : usr=0.59%, sys=1.42%, ctx=1428856, majf=0, minf=477
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,3898434,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
WRITE: bw=254MiB/s (266MB/s), 254MiB/s-254MiB/s (266MB/s-266MB/s), io=14.9GiB (16.0GB), run=60003-60003msec

Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
RANDREAD: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
...
fio-3.27
Starting 48 processes
Jobs: 48 (f=48): [r(48)][100.0%][r=291MiB/s][r=74.5k IOPS][eta 00m:00s]
RANDREAD: (groupid=0, jobs=48): err= 0: pid=13629: Tue Jan 18 13:26:53 2022
read: IOPS=73.8k, BW=288MiB/s (302MB/s)(16.9GiB/60034msec)

    slat (nsec): min=971, max=92291k, avg=7601.08, stdev=294169.16
    clat (nsec): min=881, max=172495k, avg=41609264.60, stdev=10573216.33
     lat (usec): min=73, max=186206, avg=41616.97, stdev=10588.91
    clat percentiles (msec):
     |  1.00th=[   17],  5.00th=[   28], 10.00th=[   29], 20.00th=[   31],
     | 30.00th=[   36], 40.00th=[   42], 50.00th=[   43], 60.00th=[   44],
     | 70.00th=[   46], 80.00th=[   48], 90.00th=[   56], 95.00th=[   60],
     | 99.00th=[   69], 99.50th=[   79], 99.90th=[   96], 99.95th=[  104],
     | 99.99th=[  127]
   bw (  KiB/s): min=197554, max=351693, per=100.00%, avg=295385.18, stdev=487.78, samples=5664
   iops        : min=49373, max=87909, avg=73828.39, stdev=121.94, samples=5664
  lat (nsec)   : 1000=0.01%
  lat (usec)   : 2=0.01%, 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%
  lat (usec)   : 1000=0.01%
  lat (msec)   : 2=0.01%, 4=0.02%, 10=0.12%, 20=1.19%, 50=83.06%
  lat (msec)   : 100=15.54%, 250=0.06%
  cpu          : usr=0.22%, sys=0.54%, ctx=211466, majf=0, minf=3783
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=4430082,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
READ: bw=288MiB/s (302MB/s), 288MiB/s-288MiB/s (302MB/s-302MB/s), io=16.9GiB (18.1GB), run=60034-60034msec

Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
RANDREAD: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
...
fio-3.27
Starting 48 processes
Jobs: 48 (f=48): [r(48)][100.0%][r=290MiB/s][r=74.3k IOPS][eta 00m:00s]
RANDREAD: (groupid=0, jobs=48): err= 0: pid=13758: Tue Jan 18 13:27:54 2022
read: IOPS=74.7k, BW=292MiB/s (306MB/s)(17.1GiB/60033msec)

    slat (nsec): min=961, max=67006k, avg=5644.52, stdev=214199.20
    clat (usec): min=54, max=161646, avg=41099.12, stdev=9934.03
     lat (usec): min=748, max=161649, avg=41104.87, stdev=9941.10
    clat percentiles (msec):
     |  1.00th=[   18],  5.00th=[   28], 10.00th=[   29], 20.00th=[   31],
     | 30.00th=[   35], 40.00th=[   41], 50.00th=[   43], 60.00th=[   44],
     | 70.00th=[   46], 80.00th=[   48], 90.00th=[   54], 95.00th=[   59],
     | 99.00th=[   65], 99.50th=[   70], 99.90th=[   86], 99.95th=[   92],
     | 99.99th=[  109]
   bw (  KiB/s): min=223837, max=345279, per=100.00%, avg=299116.64, stdev=387.45, samples=5664
   iops        : min=55941, max=86304, avg=74761.37, stdev=96.85, samples=5664
  lat (usec)   : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
  lat (msec)   : 2=0.01%, 4=0.01%, 10=0.07%, 20=1.08%, 50=84.88%
  lat (msec)   : 100=13.93%, 250=0.02%
  cpu          : usr=0.22%, sys=0.55%, ctx=212760, majf=0, minf=3830
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=4485508,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
READ: bw=292MiB/s (306MB/s), 292MiB/s-292MiB/s (306MB/s-306MB/s), io=17.1GiB (18.4GB), run=60033-60033msec

Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
RANDREAD: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
...
fio-3.27
Starting 48 processes
Jobs: 48 (f=48): [r(48)][100.0%][r=295MiB/s][r=75.5k IOPS][eta 00m:00s]
RANDREAD: (groupid=0, jobs=48): err= 0: pid=13832: Tue Jan 18 13:28:54 2022
read: IOPS=74.7k, BW=292MiB/s (306MB/s)(17.1GiB/60033msec)

    slat (nsec): min=972, max=88341k, avg=5140.64, stdev=192310.77
    clat (usec): min=62, max=175937, avg=41091.41, stdev=9929.06
     lat (usec): min=566, max=228627, avg=41096.66, stdev=9935.15
    clat percentiles (msec):
     |  1.00th=[   18],  5.00th=[   28], 10.00th=[   29], 20.00th=[   31],
     | 30.00th=[   35], 40.00th=[   41], 50.00th=[   43], 60.00th=[   44],
     | 70.00th=[   46], 80.00th=[   48], 90.00th=[   54], 95.00th=[   59],
     | 99.00th=[   65], 99.50th=[   70], 99.90th=[   87], 99.95th=[   94],
     | 99.99th=[  110]
   bw (  KiB/s): min=223518, max=345146, per=100.00%, avg=299138.52, stdev=394.76, samples=5664
   iops        : min=55860, max=86264, avg=74766.90, stdev=98.69, samples=5664
  lat (usec)   : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
  lat (msec)   : 2=0.01%, 4=0.01%, 10=0.06%, 20=1.06%, 50=85.06%
  lat (msec)   : 100=13.77%, 250=0.02%
  cpu          : usr=0.23%, sys=0.54%, ctx=211810, majf=0, minf=3779
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=4486204,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
READ: bw=292MiB/s (306MB/s), 292MiB/s-292MiB/s (306MB/s-306MB/s), io=17.1GiB (18.4GB), run=60033-60033msec

Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
+ nvme disconnect -n testnqn
NQN:testnqn disconnected 1 controller(s)

real 0m0.358s
user 0m0.000s
sys 0m0.004s
+ for i in /sys/kernel/config/nvmet/subsystems/${NQN}/namespaces/*
+ echo 0
+ rmdir /sys/kernel/config/nvmet/subsystems/testnqn/namespaces/1
+ rm -fr /sys/kernel/config/nvmet/ports/1/subsystems/testnqn
+ rmdir /sys/kernel/config/nvmet/ports/1
+ rmdir /sys/kernel/config/nvmet/subsystems/testnqn
+ rmdir 'config/nullb/nullb*'
rmdir: failed to remove 'config/nullb/nullb*': No such file or directory
+ umount /mnt/nvme0n1
umount: /mnt/nvme0n1: no mount point specified.
+ umount /mnt/backend
umount: /mnt/backend: not mounted.
+ modprobe -r nvme_loop
+ modprobe -r nvme_fabrics
+ modprobe -r nvmet
+ modprobe -r nvme
+ modprobe -r null_blk
+ tree /sys/kernel/config
/sys/kernel/config

2. Verbose log Block Device with buffered_io = 1 :-
---------------------------------------------------

0 directories, 0 files
++ NN=1
++ NQN=testnqn
++ let NR_DEVICES=NN+1
++ modprobe -r null_blk
++ modprobe -r nvme
++ modprobe null_blk nr_devices=0
++ modprobe nvme
++ modprobe nvme-fabrics
++ modprobe nvmet
++ modprobe nvme-loop
++ dmesg -c
++ sleep 2
++ tree /sys/kernel/config
/sys/kernel/config

├── nullb
│   └── features
└── nvmet
    ├── hosts
    ├── ports
    └── subsystems

5 directories, 1 file
++ mkdir /sys/kernel/config/nvmet/subsystems/testnqn
++ mkdir /sys/kernel/config/nvmet/subsystems/testnqn/namespaces/1
++ echo -n /dev/nvme0n1
++ cat /sys/kernel/config/nvmet/subsystems/testnqn/namespaces/1/device_path /dev/nvme0n1
++ echo 1
++ cat /sys/kernel/config/nvmet/subsystems/testnqn/namespaces/1/buffered_io
1
++ echo 1
++ mkdir /sys/kernel/config/nvmet/ports/1/
++ echo -n loop
++ echo -n 1
++ ln -s /sys/kernel/config/nvmet/subsystems/testnqn /sys/kernel/config/nvmet/ports/1/subsystems/ ++ sleep 1
++ echo transport=loop,nqn=testnqn
++ sleep 1
++ mount
++ column -t
++ grep nvme
++ '[' 1 ']'
+++ wc -l
+++ ls -l /dev/nvme1 /dev/nvme1n1
++ cnt=2
++ echo 2
2
++ '[' 2 -gt 1 ']'
++ break
++ dmesg -c

[55830.215441] nvme nvme0: 48/0/0 default/read/poll queues
[55832.213819] nvmet: adding nsid 1 to subsystem testnqn
[55833.223744] nvmet: creating nvm controller 1 for subsystem testnqn for NQN nqn.2014-08.org.nvmexpress:uuid:2ba90169-7cb8-47b3-b20f-9c2bbcb8b092.
[55833.223872] nvme nvme1: creating 48 I/O queues.
[55833.228124] nvme nvme1: new ctrl: "testnqn"

++ sleep 5
write-and-verify: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=4 fio-3.27
Starting 1 process
Jobs: 1 (f=1)
write-and-verify: (groupid=0, jobs=1): err= 0: pid=13967: Tue Jan 18 13:29:06 2022
read: IOPS=180k, BW=703MiB/s (737MB/s)(506MiB/720msec)

    slat (usec): min=2, max=118, avg= 4.20, stdev= 1.61
    clat (nsec): min=1113, max=224236, avg=17031.32, stdev=5565.81
     lat (usec): min=4, max=342, avg=21.31, stdev= 6.80
    clat percentiles (usec):
     |  1.00th=[   14],  5.00th=[   14], 10.00th=[   14], 20.00th=[   14],
     | 30.00th=[   15], 40.00th=[   15], 50.00th=[   15], 60.00th=[   15],
     | 70.00th=[   15], 80.00th=[   22], 90.00th=[   27], 95.00th=[   28],
     | 99.00th=[   35], 99.50th=[   38], 99.90th=[   45], 99.95th=[   50],
     | 99.99th=[  113]
  write: IOPS=125k, BW=489MiB/s (513MB/s)(800MiB/1635msec); 0 zone resets
    slat (usec): min=2, max=248, avg= 6.78, stdev= 2.59
    clat (nsec): min=661, max=202274, avg=24724.98, stdev=8660.56
     lat (usec): min=6, max=293, avg=31.59, stdev=10.27
    clat percentiles (usec):
     |  1.00th=[   18],  5.00th=[   19], 10.00th=[   19], 20.00th=[   20],
     | 30.00th=[   20], 40.00th=[   20], 50.00th=[   21], 60.00th=[   21],
     | 70.00th=[   25], 80.00th=[   35], 90.00th=[   38], 95.00th=[   42],
     | 99.00th=[   51], 99.50th=[   55], 99.90th=[   65], 99.95th=[   71],
     | 99.99th=[  121]
   bw (  KiB/s): min=94024, max=525016, per=81.75%, avg=409600.00, stdev=210508.70, samples=4
   iops        : min=23506, max=131254, avg=102400.00, stdev=52627.18, samples=4
  lat (nsec)   : 750=0.01%
  lat (usec)   : 2=0.01%, 10=0.01%, 20=57.13%, 50=42.10%, 100=0.76%
  lat (usec)   : 250=0.02%
  cpu          : usr=28.33%, sys=32.75%, ctx=319653, majf=0, minf=3055
  IO depths    : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=129572,204800,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4

Run status group 0 (all jobs):
READ: bw=703MiB/s (737MB/s), 703MiB/s-703MiB/s (737MB/s-737MB/s), io=506MiB (531MB), run=720-720msec
WRITE: bw=489MiB/s (513MB/s), 489MiB/s-489MiB/s (513MB/s-513MB/s), io=800MiB (839MB), run=1635-1635msec

Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
RANDREAD: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=8
...
fio-3.27
Starting 32 processes
Jobs: 32 (f=32): [w(32)][100.0%][w=15.7GiB/s][w=4108k IOPS][eta 00m:00s]
RANDREAD: (groupid=0, jobs=32): err= 0: pid=13970: Tue Jan 18 13:30:06 2022
write: IOPS=4140k, BW=15.8GiB/s (17.0GB/s)(948GiB/60002msec); 0 zone resets

    slat (nsec): min=1143, max=23104k, avg=6968.02, stdev=4184.92
    clat (nsec): min=731, max=23218k, avg=54413.34, stdev=20272.07
     lat (usec): min=5, max=23227, avg=61.46, stdev=22.72
    clat percentiles (usec):
     |  1.00th=[   41],  5.00th=[   41], 10.00th=[   42], 20.00th=[   43],
     | 30.00th=[   43], 40.00th=[   44], 50.00th=[   46], 60.00th=[   56],
     | 70.00th=[   58], 80.00th=[   60], 90.00th=[   75], 95.00th=[   97],
     | 99.00th=[  135], 99.50th=[  149], 99.90th=[  188], 99.95th=[  198],
     | 99.99th=[  231]
   bw (  MiB/s): min=11988, max=19957, per=100.00%, avg=16186.94, stdev=76.54, samples=3808
   iops        : min=3069065, max=5108996, avg=4143856.81, stdev=19593.23, samples=3808
  lat (nsec)   : 750=0.01%, 1000=0.01%
  lat (usec)   : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=54.78%
  lat (usec)   : 100=41.05%, 250=4.16%, 500=0.01%, 750=0.01%, 1000=0.01%
  lat (msec)   : 2=0.01%, 4=0.01%, 10=0.01%, 50=0.01%
  cpu          : usr=10.96%, sys=35.89%, ctx=248415838, majf=0, minf=446
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,248414657,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
WRITE: bw=15.8GiB/s (17.0GB/s), 15.8GiB/s-15.8GiB/s (17.0GB/s-17.0GB/s), io=948GiB (1018GB), run=60002-60002msec

Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
RANDREAD: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=8
...
fio-3.27
Starting 32 processes
Jobs: 32 (f=32): [w(32)][100.0%][w=16.1GiB/s][w=4215k IOPS][eta 00m:00s]
RANDREAD: (groupid=0, jobs=32): err= 0: pid=14054: Tue Jan 18 13:31:06 2022
write: IOPS=4232k, BW=16.1GiB/s (17.3GB/s)(969GiB/60002msec); 0 zone resets

    slat (nsec): min=1312, max=5006.7k, avg=6792.48, stdev=3245.46
    clat (nsec): min=721, max=23279k, avg=53238.95, stdev=14126.26
     lat (usec): min=5, max=23285, avg=60.11, stdev=15.65
    clat percentiles (usec):
     |  1.00th=[   41],  5.00th=[   42], 10.00th=[   42], 20.00th=[   43],
     | 30.00th=[   44], 40.00th=[   45], 50.00th=[   51], 60.00th=[   57],
     | 70.00th=[   58], 80.00th=[   60], 90.00th=[   66], 95.00th=[   79],
     | 99.00th=[  102], 99.50th=[  126], 99.90th=[  155], 99.95th=[  172],
     | 99.99th=[  206]
   bw (  MiB/s): min=12424, max=19988, per=100.00%, avg=16547.88, stdev=76.36, samples=3808
   iops        : min=3180626, max=5117092, avg=4236256.72, stdev=19547.40, samples=3808
  lat (nsec)   : 750=0.01%, 1000=0.01%
  lat (usec)   : 2=0.01%, 10=0.01%, 20=0.01%, 50=49.68%, 100=49.24%
  lat (usec)   : 250=1.07%, 500=0.01%, 750=0.01%, 1000=0.01%
  lat (msec)   : 2=0.01%, 4=0.01%, 10=0.01%, 50=0.01%
  cpu          : usr=11.28%, sys=37.25%, ctx=253938868, majf=0, minf=462
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,253934166,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
WRITE: bw=16.1GiB/s (17.3GB/s), 16.1GiB/s-16.1GiB/s (17.3GB/s-17.3GB/s), io=969GiB (1040GB), run=60002-60002msec

Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
RANDREAD: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=8
...
fio-3.27
Starting 32 processes
Jobs: 32 (f=32): [w(32)][100.0%][w=15.7GiB/s][w=4119k IOPS][eta 00m:00s]
RANDREAD: (groupid=0, jobs=32): err= 0: pid=14113: Tue Jan 18 13:32:07 2022
write: IOPS=4125k, BW=15.7GiB/s (16.9GB/s)(944GiB/60002msec); 0 zone resets

    slat (nsec): min=1282, max=5173.6k, avg=6989.07, stdev=3699.41
    clat (nsec): min=741, max=5243.0k, avg=54604.96, stdev=18594.18
     lat (usec): min=5, max=5251, avg=61.67, stdev=20.85
    clat percentiles (usec):
     |  1.00th=[   41],  5.00th=[   42], 10.00th=[   42], 20.00th=[   43],
     | 30.00th=[   44], 40.00th=[   45], 50.00th=[   48], 60.00th=[   57],
     | 70.00th=[   58], 80.00th=[   61], 90.00th=[   72], 95.00th=[   93],
     | 99.00th=[  133], 99.50th=[  145], 99.90th=[  186], 99.95th=[  198],
     | 99.99th=[  258]
   bw (  MiB/s): min=10106, max=20077, per=100.00%, avg=16129.59, stdev=91.59, samples=3808
   iops        : min=2587208, max=5139896, avg=4129175.92, stdev=23445.99, samples=3808
  lat (nsec)   : 750=0.01%, 1000=0.01%
  lat (usec)   : 2=0.01%, 10=0.01%, 20=0.01%, 50=52.41%, 100=44.26%
  lat (usec)   : 250=3.32%, 500=0.01%, 750=0.01%, 1000=0.01%
  lat (msec)   : 2=0.01%, 4=0.01%, 10=0.01%
  cpu          : usr=11.27%, sys=35.70%, ctx=247529856, majf=0, minf=488
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,247525302,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
WRITE: bw=15.7GiB/s (16.9GB/s), 15.7GiB/s-15.7GiB/s (16.9GB/s-16.9GB/s), io=944GiB (1014GB), run=60002-60002msec

Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
RANDREAD: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
...
fio-3.27
Starting 48 processes
Jobs: 48 (f=48): [r(48)][100.0%][r=17.5GiB/s][r=4580k IOPS][eta 00m:00s]
RANDREAD: (groupid=0, jobs=48): err= 0: pid=14172: Tue Jan 18 13:33:07 2022
read: IOPS=4731k, BW=18.0GiB/s (19.4GB/s)(1083GiB/60002msec)

    slat (nsec): min=1001, max=69146k, avg=9264.71, stdev=16039.18
    clat (nsec): min=481, max=83414k, avg=639474.44, stdev=334536.92
     lat (usec): min=5, max=83416, avg=648.84, stdev=338.99
    clat percentiles (usec):
     |  1.00th=[  404],  5.00th=[  412], 10.00th=[  424], 20.00th=[  510],
     | 30.00th=[  523], 40.00th=[  529], 50.00th=[  537], 60.00th=[  545],
     | 70.00th=[  553], 80.00th=[  914], 90.00th=[ 1139], 95.00th=[ 1188],
     | 99.00th=[ 1237], 99.50th=[ 1270], 99.90th=[ 1483], 99.95th=[ 4883],
     | 99.99th=[11076]
   bw (  MiB/s): min= 1806, max=26193, per=100.00%, avg=18499.19, stdev=75.70, samples=5712
   iops        : min=462371, max=6705466, avg=4735791.68, stdev=19380.33, samples=5712
  lat (nsec)   : 500=0.01%, 750=0.01%, 1000=0.01%
  lat (usec)   : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
  lat (usec)   : 100=0.01%, 250=0.01%, 500=17.63%, 750=60.60%, 1000=8.11%
  lat (msec)   : 2=13.58%, 4=0.03%, 10=0.05%, 20=0.01%, 50=0.01%
  lat (msec)   : 100=0.01%
  cpu          : usr=8.84%, sys=34.58%, ctx=283862866, majf=0, minf=3785
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=283879844,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
READ: bw=18.0GiB/s (19.4GB/s), 18.0GiB/s-18.0GiB/s (19.4GB/s-19.4GB/s), io=1083GiB (1163GB), run=60002-60002msec

Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
RANDREAD: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
...
fio-3.27
Starting 48 processes
Jobs: 48 (f=48): [r(48)][100.0%][r=17.2GiB/s][r=4506k IOPS][eta 00m:00s]
RANDREAD: (groupid=0, jobs=48): err= 0: pid=14451: Tue Jan 18 13:34:07 2022
read: IOPS=4790k, BW=18.3GiB/s (19.6GB/s)(1096GiB/60002msec)

    slat (usec): min=2, max=9735, avg= 9.15, stdev= 4.82
    clat (nsec): min=842, max=10334k, avg=631631.33, stdev=249341.17
     lat (usec): min=5, max=10342, avg=640.88, stdev=253.17
    clat percentiles (usec):
     |  1.00th=[  400],  5.00th=[  408], 10.00th=[  420], 20.00th=[  510],
     | 30.00th=[  523], 40.00th=[  529], 50.00th=[  537], 60.00th=[  545],
     | 70.00th=[  553], 80.00th=[  906], 90.00th=[ 1156], 95.00th=[ 1188],
     | 99.00th=[ 1237], 99.50th=[ 1254], 99.90th=[ 1287], 99.95th=[ 1319],
     | 99.99th=[ 1434]
   bw (  MiB/s): min=15109, max=25642, per=100.00%, avg=18732.45, stdev=71.99, samples=5712
   iops        : min=3868088, max=6564526, avg=4795506.90, stdev=18429.43, samples=5712
  lat (nsec)   : 1000=0.01%
  lat (usec)   : 2=0.01%, 10=0.01%, 20=0.01%, 50=0.01%, 100=0.01%
  lat (usec)   : 250=0.01%, 500=18.55%, 750=60.35%, 1000=7.37%
  lat (msec)   : 2=13.73%, 4=0.01%, 10=0.01%, 20=0.01%
  cpu          : usr=9.18%, sys=34.74%, ctx=287421571, majf=0, minf=3694
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=287412242,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
READ: bw=18.3GiB/s (19.6GB/s), 18.3GiB/s-18.3GiB/s (19.6GB/s-19.6GB/s), io=1096GiB (1177GB), run=60002-60002msec

Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
RANDREAD: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
...
fio-3.27
Starting 48 processes
Jobs: 48 (f=48): [r(48)][100.0%][r=17.4GiB/s][r=4562k IOPS][eta 00m:00s]
RANDREAD: (groupid=0, jobs=48): err= 0: pid=14522: Tue Jan 18 13:35:08 2022
read: IOPS=4925k, BW=18.8GiB/s (20.2GB/s)(1127GiB/60002msec)

    slat (usec): min=3, max=22231, avg= 8.88, stdev= 4.65
    clat (nsec): min=892, max=24497k, avg=614341.36, stdev=226405.16
     lat (usec): min=6, max=24528, avg=623.32, stdev=229.88
    clat percentiles (usec):
     |  1.00th=[  404],  5.00th=[  416], 10.00th=[  429], 20.00th=[  519],
     | 30.00th=[  529], 40.00th=[  529], 50.00th=[  537], 60.00th=[  545],
     | 70.00th=[  553], 80.00th=[  562], 90.00th=[ 1123], 95.00th=[ 1188],
     | 99.00th=[ 1237], 99.50th=[ 1254], 99.90th=[ 1303], 99.95th=[ 1319],
     | 99.99th=[ 1418]
   bw (  MiB/s): min=15698, max=26142, per=100.00%, avg=19262.82, stdev=68.27, samples=5712
   iops        : min=4018694, max=6692394, avg=4931281.76, stdev=17477.36, samples=5712
  lat (nsec)   : 1000=0.01%
  lat (usec)   : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
  lat (usec)   : 100=0.01%, 250=0.01%, 500=14.73%, 750=68.64%, 1000=5.40%
  lat (msec)   : 2=11.22%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
  cpu          : usr=9.25%, sys=36.21%, ctx=295513468, majf=0, minf=3792
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=295503180,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
READ: bw=18.8GiB/s (20.2GB/s), 18.8GiB/s-18.8GiB/s (20.2GB/s-20.2GB/s), io=1127GiB (1210GB), run=60002-60002msec

Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
root@dev nvme (nvme-5.17) #

Signature · 1 line
2.29.0
#1 lore raw

PATCH 1/3 nvmet: add helper to set ns->blkshift value

From: Chaitanya Kulkarni <[email protected]>

In order to support the block device in the buffered I/O mode, we use file backend. Calculating the blksize_shift value for the block device backend is different from the file backend.

Move existing code for calculating the namespace's blksize_shift value into the helper for file backend. The last patch in this series will add the code for block device blksize_shift calculation in the same helper.

---
 drivers/nvme/target/io-cmd-file.c | 21 ++++++++++++++-------
 1 file changed, 14 insertions(+), 7 deletions(-)
drivers/nvme/target/io-cmd-file.c +14 -7
index 6be6e59d273b..7aa4cdcb80ce 100644
@@ -26,6 +26,16 @@ int nvmet_file_ns_revalidate(struct nvmet_ns *ns)
 	return ret;
 }
 
+static int nvmet_file_ns_set_blkshift(struct nvmet_ns *ns)
+{
+	/*
+	 * i_blkbits can be greater than the universally accepted upper bound,
+	 * so make sure we export a sane namespace lba_shift.
+	 */
+	ns->blksize_shift = min_t(u8, file_inode(ns->file)->i_blkbits, 12);
+	return 0;
+}
+
 void nvmet_file_ns_disable(struct nvmet_ns *ns)
 {
 	if (ns->file) {
@@ -57,17 +67,14 @@ int nvmet_file_ns_enable(struct nvmet_ns *ns)
 		return ret;
 	}
 
+	ret = nvmet_file_ns_set_blkshift(ns);
+	if (ret)
+		return ret;
+
 	ret = nvmet_file_ns_revalidate(ns);
 	if (ret)
 		goto err;
 
-	/*
-	 * i_blkbits can be greater than the universally accepted upper bound,
-	 * so make sure we export a sane namespace lba_shift.
-	 */
-	ns->blksize_shift = min_t(u8,
-			file_inode(ns->file)->i_blkbits, 12);
-
 	ns->bvec_cache = kmem_cache_create("nvmet-bvec",
 			NVMET_MAX_MPOOL_BVEC * sizeof(struct bio_vec),
 			0, SLAB_HWCACHE_ALIGN, NULL);
Signature · 1 line
2.29.0
maintainer
#2 lore raw

Re: [PATCH 1/3] nvmet: add helper to set ns->blkshift value

Quoted 13 lines
  void nvmet_file_ns_disable(struct nvmet_ns *ns)
  {
  	if (ns->file) {
drivers/nvme/target/io-cmd-file.c +4 -0
@@ -57,17 +67,14 @@ int nvmet_file_ns_enable(struct nvmet_ns *ns)
  		return ret;
  	}
 
+	ret = nvmet_file_ns_set_blkshift(ns);
+	if (ret)
+		return ret;
+
  	ret = nvmet_file_ns_revalidate(ns);

drivers/nvme/target/io-cmd-file.c int nvmet_file_ns_enable(struct nvmet_ns *ns) · line 70

Quoted 10 lines
drivers/nvme/target/io-cmd-file.c +0 -7
  	if (ret)
  		goto err;
 
-	/*
-	 * i_blkbits can be greater than the universally accepted upper bound,
-	 * so make sure we export a sane namespace lba_shift.
-	 */
-	ns->blksize_shift = min_t(u8,
-			file_inode(ns->file)->i_blkbits, 12);
-

Any particular reason for the change in location?

#3 lore raw

Re: [PATCH 1/3] nvmet: add helper to set ns->blkshift value

Sagi,

Quoted 18 lines · Sagi Grimberg

External email: Use caution opening links or attachments

Quoted 15 lines

  void nvmet_file_ns_disable(struct nvmet_ns *ns)
  {
      if (ns->file) {

drivers/nvme/target/io-cmd-file.c +4 -0
@@ -57,17 +67,14 @@ int nvmet_file_ns_enable(struct nvmet_ns *ns)
               return ret;
       }
 
+     ret = nvmet_file_ns_set_blkshift(ns);
+     if (ret)
+             return ret;
+
       ret = nvmet_file_ns_revalidate(ns);
       if (ret)
               goto err;
 
Quoted 10 lines

drivers/nvme/target/io-cmd-file.c int nvmet_file_ns_enable(struct nvmet_ns *ns) · line 70

Quoted 8 lines
drivers/nvme/target/io-cmd-file.c +0 -7
-     /*
-      * i_blkbits can be greater than the universally accepted upper 
 bound,
-      * so make sure we export a sane namespace lba_shift.
-      */
-     ns->blksize_shift = min_t(u8,
-                     file_inode(ns->file)->i_blkbits, 12);
-

Any particular reason for the change in location?

So third patch in this series sets ns->bdev if file type is block device in nvmet_file_ns_set_blkshift(). Based on ns->bdev is NULL or not NULL that is used to determine how we calculate ns->size in the nvmet_file_ns_revalidate(), see last patch.

Hence move blksize_shift setting before nvmet_file_ns_revalidate().

-ck

#4 lore raw

PATCH 2/3 nvmet: add helper to open block device

In reply to Chaitanya Kulkarni

From: Chaitanya Kulkarni <[email protected]>

In order to support block device backend in the buffered I/O mode we keep the ns->bdev handle open so we can reavalidate the ns->size in nvmet_file_ns_revalidate() using bdev_nr_bytes(). Also, instead of duplicating the code for opening block device we move the existing code into the nvme_bdev_open() helper and call from nvmet_bdev_ns_enable().

This newly added helper will be called in the next patch to open and set the ns->size and ns->blksize_shift values when block device is configured with buffere I/O mode from file backend code.

---
 drivers/nvme/target/io-cmd-bdev.c | 14 ++++++++++++--
 drivers/nvme/target/nvmet.h       |  1 +
 2 files changed, 13 insertions(+), 2 deletions(-)
drivers/nvme/target/io-cmd-bdev.c +12 -2
index 70ca9dfc1771..58907304587e 100644
@@ -72,9 +72,9 @@ static void nvmet_bdev_ns_enable_integrity(struct nvmet_ns *ns)
 	}
 }
 
-int nvmet_bdev_ns_enable(struct nvmet_ns *ns)
+int nvmet_bdev_ns_open(struct nvmet_ns *ns)
 {
-	int ret;
+	int ret = 0;
 
 	ns->bdev = blkdev_get_by_path(ns->device_path,
 			FMODE_READ | FMODE_WRITE, NULL);
@@ -89,6 +89,16 @@ int nvmet_bdev_ns_enable(struct nvmet_ns *ns)
 	}
 	ns->size = bdev_nr_bytes(ns->bdev);
 	ns->blksize_shift = blksize_bits(bdev_logical_block_size(ns->bdev));
+	return ret;
+}
+
+int nvmet_bdev_ns_enable(struct nvmet_ns *ns)
+{
+	int ret;
+
+	ret = nvmet_bdev_ns_open(ns);
+	if (ret)
+		return ret;
 
 	ns->pi_type = 0;
 	ns->metadata_size = 0;
drivers/nvme/target/nvmet.h +1 -0
index af193423c10b..534fe2b32e99 100644
@@ -533,6 +533,7 @@ extern struct rw_semaphore nvmet_ana_sem;
 
 bool nvmet_host_allowed(struct nvmet_subsys *subsys, const char *hostnqn);
 
+int nvmet_bdev_ns_open(struct nvmet_ns *ns);
 int nvmet_bdev_ns_enable(struct nvmet_ns *ns);
 int nvmet_file_ns_enable(struct nvmet_ns *ns);
 void nvmet_bdev_ns_disable(struct nvmet_ns *ns);
Signature · 1 line
2.29.0
#5 lore raw

PATCH 3/3 nvmet: allow bdev in buffered_io mode

In reply to Chaitanya Kulkarni

From: Chaitanya Kulkarni <[email protected]>

Allow block device to be configured in the buffered I/O mode by using the file backend. In this way now we can use cache for the block device namespace which shows significant performance improvement.

We update the block device ns enable function and return early when buffered_io flag is set, when revalidating the file backed namespace open block device handle and consider block device size, use previously exported block device API to set the namespace size and blksize_shift value, and call nvmet_bdev_ns_disable() when disabling file backed namespace.

---
 drivers/nvme/target/io-cmd-bdev.c |  8 ++++++++
 drivers/nvme/target/io-cmd-file.c | 17 ++++++++++++++++-
 2 files changed, 24 insertions(+), 1 deletion(-)
drivers/nvme/target/io-cmd-bdev.c +8 -0
index 58907304587e..5e5eb49a0255 100644
@@ -96,6 +96,14 @@ int nvmet_bdev_ns_enable(struct nvmet_ns *ns)
 {
 	int ret;
 
+	/*
+	 * When buffered_io namespace attribute is enabled that means user want
+	 * this block device to be used as a file, so block device can take
+	 * an advantage of cache.
+	 */
+	if (ns->buffered_io)
+		return -ENOTBLK;
+
 	ret = nvmet_bdev_ns_open(ns);
 	if (ret)
 		return ret;
drivers/nvme/target/io-cmd-file.c +16 -1
index 7aa4cdcb80ce..067c45b23bba 100644
@@ -22,12 +22,25 @@ int nvmet_file_ns_revalidate(struct nvmet_ns *ns)
 	ret = vfs_getattr(&ns->file->f_path, &stat, STATX_SIZE,
 			  AT_STATX_FORCE_SYNC);
 	if (!ret)
-		ns->size = stat.size;
+		ns->size = ns->bdev ? bdev_nr_bytes(ns->bdev) : stat.size;
+
 	return ret;
 }
 
 static int nvmet_file_ns_set_blkshift(struct nvmet_ns *ns)
 {
+	struct kstat stat;
+	int ret;
+
+	ret = vfs_getattr(&ns->file->f_path, &stat, STATX_MODE,
+			  AT_STATX_FORCE_SYNC);
+	if (ret)
+		return ret;
+
+	/* nvmet_bdev_ns_open() sets size & blksize_shift values for ns->bdev */
+	if (S_ISBLK(stat.mode) && ns->buffered_io)
+		return nvmet_bdev_ns_open(ns);
+
 	/*
 	 * i_blkbits can be greater than the universally accepted upper bound,
 	 * so make sure we export a sane namespace lba_shift.
@@ -45,6 +58,8 @@ void nvmet_file_ns_disable(struct nvmet_ns *ns)
 		ns->bvec_pool = NULL;
 		kmem_cache_destroy(ns->bvec_cache);
 		ns->bvec_cache = NULL;
+		/* if block device is configured with buffered_io */
+		nvmet_bdev_ns_disable(ns);
 		fput(ns->file);
 		ns->file = NULL;
 	}
Signature · 1 line
2.29.0
#6 lore raw
In reply to Chaitanya Kulkarni

Shouldn't something like the patch below be all we need to use buffered I/O on a block device?

drivers/nvme/target/io-cmd-bdev.c +4 -0
index 70ca9dfc1771a..7d9da2f77b83a 100644
@@ -76,6 +76,10 @@ int nvmet_bdev_ns_enable(struct nvmet_ns *ns)
 {
 	int ret;
 
+	/* force fallback to the file backend for buffered I/O mode */
+	if (ns->buffered_io)
+		return -ENOTBLK;
+
 	ns->bdev = blkdev_get_by_path(ns->device_path,
 			FMODE_READ | FMODE_WRITE, NULL);
 	if (IS_ERR(ns->bdev)) {
#7 lore raw

Christoph,

Quoted 12 lines · Christoph Hellwig

External email: Use caution opening links or attachments

Shouldn't something like the patch below be all we need to use buffered I/O on a block device?

drivers/nvme/target/io-cmd-bdev.c +0 -0
index 70ca9dfc1771a..7d9da2f77b83a 100644

drivers/nvme/target/io-cmd-bdev.c int nvmet_bdev_ns_enable(struct nvmet_ns *ns) · line 85

Quoted 10 lines
drivers/nvme/target/io-cmd-bdev.c +4 -0
@@ -76,6 +76,10 @@ int nvmet_bdev_ns_enable(struct nvmet_ns *ns)
  {
         int ret;
 
+       /* force fallback to the file backend for buffered I/O mode */
+       if (ns->buffered_io)
+               return -ENOTBLK;
+
         ns->bdev = blkdev_get_by_path(ns->device_path,
                         FMODE_READ | FMODE_WRITE, NULL);
         if (IS_ERR(ns->bdev)) {

This is exactly how I stared this patch-series, but above doesn't work since it errors out in the nvme_identify_ns() due to ncap value set to 0 (set from ns->size in id-ns admin on target side) see [1], since ns->size value we get is 0, that is set using call to vfs_getattr(STATX_SIZE) in the nvmet_file_ns_revalidate() for file backend.

[1]

[  579.444478] nvme nvme0: 48/0/0 default/read/poll queues
[  581.430805] nvmet: adding nsid 1 to subsystem testnqn
[  582.439296] nvmet: creating nvm controller 1 for subsystem testnqn 
for NQN 
nqn.2014-08.org.nvmexpress:uuid:23f0992c-7ef7-4fbd-80f7-209cda6d872a.
[  582.439429] nvme nvme1: creating 48 I/O queues.
[  582.444036] nvme nvme1: new ctrl: "testnqn"

**[ 582.444095] nvmet: nvmet_execute_identify_ns 520 req->ns->size 0 >> req->ns->blksize_shift 12 = 0**
**[ 582.444103] ERROR nvme_identify_ns 1473**
++ sleep 5

Debug diff :-

root@dev nvme (nvme-5.17) # git diff

drivers/nvme/host/core.c +0 -0
index dd18861f77c0..5a33b3f73b3b 100644
@@ -1469,8 +1469,10 @@ static int nvme_identify_ns(struct nvme_ctrl 

*ctrl, unsigned nsid,

         }

         error = NVME_SC_INVALID_NS | NVME_SC_DNR;
-       if ((*id)->ncap == 0) /* namespace not allocated or attached */
+       if ((*id)->ncap == 0) { /* namespace not allocated or attached */
+               pr_info(" ERROR %s %d\n", __func__, __LINE__);
                 goto out_free_id;

+ }

         if (ctrl->vs >= NVME_VS(1, 1, 0) &&
             !memchr_inv(ids->eui64, 0, sizeof(ids->eui64)))
(quoted diff) +0 -0

b/drivers/nvme/target/admin-cmd.c
index 6fb24746de06..94f158aa7bfa 100644

drivers/nvme/target/admin-cmd.c +0 -0
@@ -517,6 +517,9 @@ static void nvmet_execute_identify_ns(struct 

nvmet_req *req)
* nuse = ncap = nsze isn't always true, but we have no way to find
* that out from the underlying device.
*/
+ pr_info("%s %d req->ns->size %u >> req->ns->blksize_shift %u =
%u\n",
+ __func__, __LINE__, req->ns->size,
req->ns->blksize_shift,

+                       req->ns->size >> req->ns->blksize_shift);
         id->ncap = id->nsze =
                 cpu_to_le64(req->ns->size >> req->ns->blksize_shift);
         switch (req->port->ana_state[req->ns->anagrpid]) {
(quoted diff) +0 -0

b/drivers/nvme/target/io-cmd-bdev.c
index 70ca9dfc1771..7d9da2f77b83 100644

drivers/nvme/target/io-cmd-bdev.c +4 -0
@@ -76,6 +76,10 @@ int nvmet_bdev_ns_enable(struct nvmet_ns *ns)
  {
         int ret;
 
+       /* force fallback to the file backend for buffered I/O mode */
+       if (ns->buffered_io)
+               return -ENOTBLK;
+
         ns->bdev = blkdev_get_by_path(ns->device_path,
                         FMODE_READ | FMODE_WRITE, NULL);
         if (IS_ERR(ns->bdev)) {

root@dev nvme (nvme-5.17) #

#8 lore raw
Quoted 17 lines · Chaitanya Kulkarni

Christoph,

Quoted 14 lines · Christoph Hellwig

External email: Use caution opening links or attachments

Shouldn't something like the patch below be all we need to use buffered I/O on a block device?

drivers/nvme/target/io-cmd-bdev.c +0 -0
index 70ca9dfc1771a..7d9da2f77b83a 100644
@@ -76,6 +76,10 @@ int nvmet_bdev_ns_enable(struct nvmet_ns *ns)
   {
          int ret;
 
Quoted 17 lines

drivers/nvme/target/io-cmd-bdev.c int nvmet_bdev_ns_enable(struct nvmet_ns *ns) · line 85

Quoted 8 lines
drivers/nvme/target/io-cmd-bdev.c +4 -0
+       /* force fallback to the file backend for buffered I/O mode */
+       if (ns->buffered_io)
+               return -ENOTBLK;
+
          ns->bdev = blkdev_get_by_path(ns->device_path,
                          FMODE_READ | FMODE_WRITE, NULL);
          if (IS_ERR(ns->bdev)) {

This is exactly how I stared this patch-series, but above doesn't work since it errors out in the nvme_identify_ns() due to ncap value set to 0 (set from ns->size in id-ns admin on target side) see [1], since ns->size value we get is 0, that is set using call to vfs_getattr(STATX_SIZE) in the nvmet_file_ns_revalidate() for file backend.

Also, I believe we need to make sure ns->device_path is block device type with vfs_getattr(STATX_MODE).

#9 lore raw
Quoted 2 lines · Chaitanya Kulkarni

Also, I believe we need to make sure ns->device_path is block device type with vfs_getattr(STATX_MODE).

I don't think we care about the type of filp_open works.

#10 lore raw
In reply to Chaitanya Kulkarni
Quoted 6 lines · Chaitanya Kulkarni

This is exactly how I stared this patch-series, but above doesn't work since it errors out in the nvme_identify_ns() due to ncap value set to 0 (set from ns->size in id-ns admin on target side) see [1], since ns->size value we get is 0, that is set using call to vfs_getattr(STATX_SIZE) in the nvmet_file_ns_revalidate() for file backend.

Something like this should solve that issue:

drivers/nvme/target/io-cmd-file.c +2 -8
index 6be6e59d273bb..fb7f4dd72daf6 100644
@@ -16,14 +16,8 @@
 
 int nvmet_file_ns_revalidate(struct nvmet_ns *ns)
 {
-	struct kstat stat;
-	int ret;
-
-	ret = vfs_getattr(&ns->file->f_path, &stat, STATX_SIZE,
-			  AT_STATX_FORCE_SYNC);
-	if (!ret)
-		ns->size = stat.size;
-	return ret;
+	ns->size = i_size_read(ns->file->f_mapping->host);
+	return 0;
 }
 
 void nvmet_file_ns_disable(struct nvmet_ns *ns)