Evenly distribute jobs and iodepth over a 1 TiB device so that every byte is written to in parallel
Hello,
I have a 1 TiB NVMe namespace from a NetApp connected via two distinct direct links to a Linux system over NVMe/TCP. I would like to generate read and write I/O using multiple jobs/iodepth so that every byte of the device is being written to in parallel with the maximum number of available parallel inflight I/Os. The NetApp does deduplication and compression by default so I want to generate random data. Because I think if I don't do refill_buffers, the NetApp gets that the same data is used over and over again and dedups it. I tried:
fio --ioengine=libaio --refill_buffers --filesize=25G --ramp_time=2s \
--runtime=1m --numjobs=40 --direct=1 --verify=0 --randrepeat=0 \
--group_reporting --filename=/dev/nvme0n1 --name=1mhqd --blocksize=1m \
--iodepth=1638 --readwrite=write
So I'm on a Linux system with 40 hyperthreads and a mellanox 2x 25 Gbit/s card hooked up to the NetApp:
(live) [~] ip -br a s
...
eth6 UP 192.168.0.100/24 eth7 UP 192.168.1.100/24
(live) [~] nvme list-subsys /dev/nvme0n1
nvme-subsys0 - NQN=nqn.1992-08.com.netapp:sn.e0a0273a60b711f09deed039ead647e8:subsystem.svm1_subsystem_553
hostnqn=nqn.2014-08.org.nvmexpress:uuid:20f011e6-9ab8-584f-abb0-a260d2d685c4 \ +- nvme0 tcp traddr=192.168.0.2,trsvcid=4420,src_addr=192.168.0.100 live optimized +- nvme1 tcp traddr=192.168.1.2,trsvcid=4420,src_addr=192.168.1.100 live optimized
na2501::*> network interface show
Logical Status Network Current Current Is
Vserver Interface Admin/Oper Address/Mask Node Port Home
----------- ---------- ---------- ------------------ ------------- ------- ----
...
svm1
lif_svm1_2660 up/up 192.168.1.2/24 na2501-02 e4c true
lif_svm1_9354 up/up 192.168.0.2/24 na2501-01 e4c trueSo when I run the above command the NetApp only reports a few hundered GiB of physically allocated space:
na2501::*> aggr show -fields physical-used
aggregate physical-used
-------------- -------------dataFA_4_p0_i1 169.5GB
So, I ran:
(live) [~] pv < /dev/urandom > /dev/nvme0n1 1.00TiB 0:59:14 [ 294MiB/s] [======================>] 100%
And afterwards more physical space was used:
na2501::*> aggr show -fields physical-used
aggregate physical-used
-------------- -------------dataFA_4_p0_i1 1.15TB
So, what is the best way to use fio to write random data to every byte of this 1 TiB device in parallel?
- Is there a command line parameter?
- Or should I create 40 25.6 GiB (1024/40) partitions and give them as colon separated list to fio?
I also would like to determine the number of queues and queue depth? Is there a command available. When I run:
fio --ioengine=libaio --refill_buffers --filesize=8G --ramp_time=2s \
--runtime=1m --numjobs=40 --direct=1 --verify=0 --randrepeat=0 \
--group_reporting --filename=/dev/nvme0n1 --name=4khqd --blocksize=4k \
--iodepth=1638 --readwrite=randwrite
And also watch 'iostat -xm 2' I can see aqu-sz is 194.87 per path and 391.96 for the multipathed device nvme0n1. So I kind of know it but would like to have a command on Linux that shows me the available queues and queue depths.
avg-cpu: %user %nice %system %iowait %steal %idle
1.67 0.00 6.48 85.13 0.00 6.72
Device r/s rMB/s rrqm/s %rrqm r_await rareq-sz w/s wMB/s wrqm/s %wrqm w_await wareq-sz d/s dMB/s drqm/s %drqm d_await dareq-sz f/s f_await aqu-sz %util nvme0c0n1 0.00 0.00 0.00 0.00 0.00 0.00 60855.00 237.71 0.00 0.00 3.20 4.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 194.87 100.00 nvme0c1n1 0.00 0.00 0.00 0.00 0.00 0.00 58719.00 229.37 0.00 0.00 3.32 4.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 194.94 100.00
nvme0n1 0.00 0.00 0.00 0.00 0.00 0.00 119570.50 467.07 0.00 0.00 3.28 4.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 391.96 100.00
Cheers,
Thomas
Hello Thomas,
Quoted 9 lines · Thomas Glanzmann
I have a 1 TiB NVMe namespace from a NetApp connected via two distinct direct links to a Linux system over NVMe/TCP. I would like to generate read and write I/O using multiple jobs/iodepth so that every byte of the device is being written to in parallel with the maximum number of available parallel inflight I/Os. The NetApp does deduplication and compression by default so I want to generate random data. Because I think if I don't do
refill_buffers, the NetApp gets that the same data is used over and over again and dedups it. I tried:
It depends on just how clever it is. By default fio uses scramble_buffers
(https://fio.readthedocs.io/en/latest/fio_doc.html#cmdoption-arg-scramble_buffers
) which may be enough but you would have to check. The trick would be to see what happens with a single stream as that's easier to reason about.
Quoted 4 lines
fio --ioengine=libaio --refill_buffers --filesize=25G --ramp_time=2s \
--runtime=1m --numjobs=40 --direct=1 --verify=0 --randrepeat=0 \
--group_reporting --filename=/dev/nvme0n1 --name=1mhqd --blocksize=1m \
--iodepth=1638 --readwrite=write
If I'm reading this correctly you have 40 jobs all sequentially writing the same 25G of the device at the same time. This is problematic because if you send two or more write I/Os for the same area to storage at the same time then something in your data path could legitimately just throw all but one of them away and say "I'm done" (because one I/O overwrites the others) - you're essentially saying "I don't care about the data" and in certain setups such behaviour is undefined. One change you could make is to have each of the 40 jobs write to a different region to the others e.g. by using offset_increment
https://fio.readthedocs.io/en/latest/fio_doc.html#cmdoption-arg-offset_increment
and size https://fio.readthedocs.io/en/latest/fio_doc.html#cmdoption-arg-size
.
<snip>
Quoted 11 lines
So, I ran:
(live) [~] pv <
/dev/urandom>/dev/nvme0n11.00TiB 0:59:14 [ 294MiB/s] [======================>] 100%And afterwards more physical space was used:
na2501::*> aggr show -fields physical-used
aggregate physical-used
-------------- -------------dataFA_4_p0_i11.15TB
Do you get similar results in terms of space used with a single fio stream? Start small and then work your way up!
Quoted 6 lines
So, what is the best way to use fio to write random data to every byte of this 1 TiB device in parallel?
- Is there a command line parameter?
- Or should I create 40 25.6 GiB (1024/40) partitions and give them as colon separated list to fio?
See above.
Quoted 2 lines
I also would like to determine the number of queues and queue depth? Is there a command available. When I run:
fio does report statistics about what queue depth each job internally reached but these may be different to what your device sees for a variety of reasons (e.g. splitting a coalescing done by the block layer). See IO depths/IO submit/IO complete over on
https://fio.readthedocs.io/en/latest/fio_doc.html#interpreting-the-output
. But perhaps you're thinking of device queue depths?
Quoted 4 lines
fio --ioengine=libaio --refill_buffers --filesize=8G --ramp_time=2s \
--runtime=1m --numjobs=40 --direct=1 --verify=0 --randrepeat=0 \
--group_reporting --filename=/dev/nvme0n1 --name=4khqd --blocksize=4k \
--iodepth=1638 --readwrite=randwrite
Given you're running 40 jobs I'd be surprised if you can hit a depth of over 1000 per job (that would be over 65000 I/Os in total) without some serious tuning. You may want to look at /sys/block/[disk]/queue/nr_requests (see
https://www.kernel.org/doc/Documentation/block/queue-sysfs.rst ) and /sys/block/[disk]/device/queue_depth but you may also find you run into libaio limits...
Quoted 4 lines
And also watch 'iostat -xm 2' I can see aqu-sz is 194.87 per path and 391.96 for the multipathed device nvme0n1. So I kind of know it but would like to have a command on Linux that shows me the available queues and queue depths.
I'm fairly sure iostat is the right way to go (unless you wanted to write some BPF tracing).
Signature · 1 line
Sitsofe
Hello Sitsofe,
Quoted 4 lines · Sitsofe Wheeler
(https://fio.readthedocs.io/en/latest/fio_doc.html#cmdoption-arg-scramble_buffers)
which may be enough but you would have to check. The trick would be to see what happens with a single stream as that's easier to reason about.
I see, I stuck with refill_buffers because it was good enough.
Quoted 2 lines
https://fio.readthedocs.io/en/latest/fio_doc.html#cmdoption-arg-offset_increment
and size https://fio.readthedocs.io/en/latest/fio_doc.html#cmdoption-arg-size
Thank you that resolved my problem. I used the following command line:
fio --ioengine=libaio --refill_buffers --offset=0 --offset_increment=26G \
--size=25G --ramp_time=2s --numjobs=80 --direct=1 --verify=0 \
--randrepeat=0 --group_reporting --filename /dev/nvme0n1 --name=1mhqd \
--blocksize=1m --iodepth=3 --readwrite=writeQuoted 2 lines
Do you get similar results in terms of space used with a single fio stream? Start small and then work your way up!
Yes, that works, but I also wanted to benchmark parallel performacne. Also the single fio stream takes an hour while the parallel one only takes 8,5 minutes.
Quoted 1 line
But perhaps you're thinking of device queue depths?
Yeah, that I was looking for Keith Bush answered me. The commands I was looking for, are:
# How many IO queues are there:
ls -1 /sys/block/nvme0n1/mq/ | wc -l
# How large is each IO queue:
cat /sys/block/nvme0n1/queue/nr_requests
Quoted 7 lines
Given you're running 40 jobs I'd be surprised if you can hit a depth of over 1000 per job (that would be over 65000 I/Os in total) without some serious tuning. You may want to look at /sys/block/[disk]/queue/nr_requests (see
https://www.kernel.org/doc/Documentation/block/queue-sysfs.rst ) and /sys/block/[disk]/device/queue_depth but you may also find you run into libaio limits...
I could not. The netapp only has 8 * 128 queue depth, so now that I knew the exact values I adopted the same.
Cheers,
Thomas