Number of data and admin queues in use
Hello,
I have Linux system hooked up over two dedicated links to a NetApp using NVMe/TCP. I would like to find out how many data and admin queues there are and their queue depth? How can I find out the same?
So far, I found out:
(live) [~] nvme netapp ontapdevices /dev/nvme0n1/dev/nvme0n1, Vserver svm1, Subsystem svm1_subsystem_553, Namespace Path rx3082_1, NSID 1, UUID 7f6be93b-60cb-11f0-866f-d039ead647e8, 1.10TB (live) [~] nvme list-subsys /dev/nvme0n1
nvme-subsys0 - NQN=nqn.1992-08.com.netapp:sn.e0a0273a60b711f09deed039ead647e8:subsystem.svm1_subsystem_553
hostnqn=nqn.2014-08.org.nvmexpress:uuid:20f011e6-9ab8-584f-abb0-a260d2d685c4 \ +- nvme0 tcp traddr=192.168.0.2,trsvcid=4420,src_addr=192.168.0.100 live optimized +- nvme1 tcp traddr=192.168.1.2,trsvcid=4420,src_addr=192.168.1.100 live optimized
(live) [~] nvme list
Node Generic SN Model Namespace Usage Format FW Rev
--------------------- --------------------- -------------------- ---------------------------------------- ---------- -------------------------- ---------------- --------/dev/nvme0n1 /dev/ng0n1 824nlJYbYm5ZAAAAAAAB NetApp ONTAP Controller 0x1 8.62 GB / 1.10 TB 4 KiB + 0 B 9.16.1
(live) [~] sudo nvme get-feature /dev/nvme0n1 --feature-id=7 -H
get-feature:0x07 (Number of Queues), Current value:0x00010001
Number of IO Completion Queues Allocated (NCQA): 2
Number of IO Submission Queues Allocated (NSQA): 2
I also have some local NVMe where I would like to find out the same:
(infra) [~] sudo nvme get-feature /dev/nvme0n1 --feature-id=7 -H
get-feature:0x07 (Number of Queues), Current value:0x007f007f
Number of IO Completion Queues Allocated (NCQA): 128
Number of IO Submission Queues Allocated (NSQA): 128
(infra) [~] nvme list
Node Generic SN Model Namespace Usage Format FW Rev
--------------------- --------------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- --------
/dev/nvme1n1 /dev/ng1n1 50026B7685E14353 KINGSTON SKC3000D2048G 1 2.05 TB / 2.05 TB 512 B + 0 B EIFK31.6 /dev/nvme0n1 /dev/ng0n1 50026B7685E1439E KINGSTON SKC3000D2048G 1 2.05 TB / 2.05 TB 512 B + 0 B EIFK31.6
(infra) [~] nvme list-subsys /dev/nvme0n1
nvme-subsys0 - NQN=nqn.2020-04.com.kingston:nvme:nvm-subsystem-sn-50026B7685E1439E \
+- nvme0 pcie 0000:01:00.0 live
I one heard that the Linux kernel allocates one queue per processor (core or hyperthread). I can see that using /proc/interrupts but only on physical devices, but not on NVME/TCP systems.
Cheers,
Thomas
Keith Busch
maintainerQuoted 3 lines · Thomas Glanzmann
I one heard that the Linux kernel allocates one queue per processor (core or hyperthread). I can see that using
/proc/interruptsbut only on physical devices, but not on NVME/TCP systems.
For PCI, the driver automatically handles the queue and interrupt setup, and cpu assignment.
For TCP (and all fabrics transports), you have to specificy how many connections you want to make ("nr_io_queues=X") when you're setting up your initial fabrics connection.
If you want to see what you've ended up with, you can consult the namespaces' sysfs entries:
How many IO queues are there:
# ls -1 /sys/block/nvme0n1/mq/ | wc -l 64
How large is each IO queue:
# cat /sys/block/nvme0n1/queue/nr_requests 1023
Hello Keith,
Quoted 12 lines · Keith Busch
For PCI, the driver automatically handles the queue and interrupt setup, and cpu assignment.
For TCP (and all fabrics transports), you have to specificy how many connections you want to make ("nr_io_queues=X") when you're setting up your initial fabrics connection.
If you want to see what you've ended up with, you can consult the namespaces' sysfs entries:
How many IO queues are there:
Quoted 7 lines
# ls -1 /sys/block/nvme0n1/mq/ | wc -l 64How large is each IO queue:
# cat /sys/block/nvme0n1/queue/nr_requests 1023
thank you for taking the time to answer me. I was looking multiple years for an answer to this. nr_requests I stumbled on before, but /sys/block/nvme0n1/mq/ was new to me. The maximum that the NetApp appears to support is:
na2501::*> vserver nvme show-host-priority
Node Protocol Priority I/O Queue Count I/O Queue Depth
--------------------- --------- -------- --------------- ---------------
na2501-01 fc-nvme
regular 4 32
high 6 32 nvme-tcp
regular 2 128
high 4 128
na2501-02 fc-nvme
regular 4 32
high 6 32 nvme-tcp
regular 2 128
high 4 128
(live) [~] ls -1 /sys/block/nvme0c0n1/mq/ | wc -l 4
(live) [~] ls -1 /sys/block/nvme0c1n1/mq/ | wc -l 4
(live) [~] cat /sys/block/nvme0c0n1/queue/nr_requests 127
With ext4 and fio I get:
fio --ioengine=libaio --refill_buffers --filesize=4G --ramp_time=2s --numjobs=40 --direct=1 --verify=0 --randrepeat=0 --group_reporting --directory /mnt --name=4khqd --blocksize=4k --iodepth=50 --readwrite=write
write: IOPS=159k, BW=620MiB/s (651MB/s)(159GiB/261872msec); 0 zone resets
fio --ioengine=libaio --refill_buffers --filesize=4G --ramp_time=2s --numjobs=40 --direct=1 --verify=0 --randrepeat=0 --group_reporting --directory /mnt --name=4khqd --blocksize=4k --iodepth=50 --readwrite=read
read: IOPS=449k, BW=1752MiB/s (1838MB/s)(157GiB/91645msec)
fio --ioengine=libaio --refill_buffers --filesize=4G --ramp_time=2s --numjobs=40 --direct=1 --verify=0 --randrepeat=0 --group_reporting --directory /mnt --name=1mhqd --blocksize=1m --iodepth=50 --readwrite=write
write: IOPS=1965, BW=1970MiB/s (2066MB/s)(157GiB/81434msec); 0 zone resets
fio --ioengine=libaio --refill_buffers --filesize=4G --ramp_time=2s --numjobs=40 --direct=1 --verify=0 --randrepeat=0 --group_reporting --directory /mnt --name=1mhqd --blocksize=1m --iodepth=50 --readwrite=read
read: IOPS=4034, BW=4044MiB/s (4241MB/s)(153GiB/38682msec)
Using 'iostat -xm 2' I can see that is utilizes the queue depth by watching aqu-sz.
Cheers,
Thomas
Quoted 16 lines · Keith Busch
Quoted 3 lines · Thomas Glanzmann
I one heard that the Linux kernel allocates one queue per processor (core or hyperthread). I can see that using
/proc/interruptsbut only on physical devices, but not on NVME/TCP systems.For PCI, the driver automatically handles the queue and interrupt setup, and cpu assignment.
For TCP (and all fabrics transports), you have to specificy how many connections you want to make ("nr_io_queues=X") when you're setting up your initial fabrics connection.
If you want to see what you've ended up with, you can consult the namespaces' sysfs entries:
How many IO queues are there:
Quoted 8 lines
# ls -1 /sys/block/nvme0n1/mq/ | wc -l 64How large is each IO queue:
# cat /sys/block/nvme0n1/queue/nr_requests 1023
For block layer queue allocation which will happen when you issue connect command from host to target which will create controller and populate block and char devices see below [1].
From what I can see you are getting number of queues for both tcp and pcie
NVMe controller, what is your question ?
Another way to dig into controller side fields or queue depth you can read the CAP space see this from spec
Figure 36: Offset 0h: CAP – Controller Capabilities :-
"Maximum Queue Entries Supported (MQES): This field indicates the maximum individual queue size that the controller supports. For NVMe over PCIe implementations, this value applies to the I/O Submission Queues and I/O Completion Queues that the host creates. For NVMe over Fabrics implementations,
this value applies to only the I/O Submission Queues that the host creates. This is
a 0’s based value. The minimum value is 1h, indicating two entries."
-ck
[1]
For fabrics transport (TCP) number are queues are calculated using nvmf_nr_io_queue() to make sure we don't create more read/defult queues than CPUs available same check is also applicable for write and poll queues.nvme_set_queue_count adjusts the queue count based on controller capabilities which cal also clamp the queue count.
nvmf_set_io_queues() set queue count for each queue type read, default, poll. then nvmf_map_queues() maps them into blk-mq structure so that default/read/poll and each gets attached to blk_mq context.
On My machine I've 48 CPUs so when I create tcp target I get :-
[ 1196.058440] nvme nvme1: creating 48 I/O queues. [ 1196.062370] nvme nvme1: mapped 48/0/0 default/read/poll queues.
you should be able to see this into debug messages that is coming from queue allocation helpers respectively that also has controller device name "nvme1" :-
nvme_tcp_alloc_io_queues()nvmf_map_queues()
Hope this helps.
Hello Chaitanya,
Quoted 2 lines
From what I can see you are getting number of queues for both tcp and pcie NVMe controller, what is your question?
My question was how to see the number and size of NVMe IO queues but Keith already answered that. I just thanked him and added some stats from the NetApp.
Quoted 34 lines
Another way to dig into controller side fields or queue depth you can read the CAP space see this from spec
Figure 36: Offset 0h: CAP – Controller Capabilities :-
"Maximum Queue Entries Supported (MQES): This field indicates the maximum individual queue size that the controller supports. For NVMe over PCIe implementations, this value applies to the I/O Submission Queues and I/O Completion Queues that the host creates. For NVMe over Fabrics implementations, this value applies to only the I/O Submission Queues that the host creates. This is a 0’s based value. The minimum value is 1h, indicating two entries."
-ck
[1]
For fabrics transport (TCP) number are queues are calculated using
nvmf_nr_io_queue()to make sure we don't create more read/defult queues than CPUs available same check is also applicable for write and poll queues.nvme_set_queue_countadjusts the queue count based on controller capabilities which cal also clamp the queue count.
nvmf_set_io_queues()set queue count for each queue type read, default, poll. thennvmf_map_queues()maps them into blk-mq structure so that default/read/poll and each gets attached toblk_mqcontext.On My machine I've 48 CPUs so when I create tcp target I get :-
[ 1196.058440] nvme nvme1: creating 48 I/O queues. [ 1196.062370] nvme nvme1: mapped 48/0/0 default/read/poll queues.
Quoted 6 lines
you should be able to see this into debug messages that is coming from queue allocation helpers respectively that also has controller device name "nvme1" :-
nvme_tcp_alloc_io_queues()nvmf_map_queues()
Tomorrow I'll setup a NVMe/TCP target on Linux and do some benchmarking. I'll also hookup the NetApp to 64 Gbit/s FC and do some benchmarking with FC and FC/NVMe.
Thank you for the additional insight, I never paid attention to this but, I did now:
[ 3730.402432] nvme nvme0: queue_size 128 > ctrl sqsize 32, clamping down [ 3795.115084] subsysnqn nqn.1992-08.com.netapp:sn.e0a0273a60b711f09deed039ead647e8:subsystem.svm1_subsystem_553 iopolicy changed from numa to queue-depth [ 3795.154560] nvme nvme0: creating 2 I/O queues. [ 3795.156535] nvme nvme0: mapped 2/0/0 default/read/poll queues. [ 3801.004641] nvme nvme1: creating 2 I/O queues. [ 3801.006541] nvme nvme1: mapped 2/0/0 default/read/poll queues.
Than I bumped the queues and queue size on the NetApp and got:
[98114.846603] nvme nvme0: queue_size 128 > ctrl sqsize 32, clamping down [98727.596158] subsysnqn nqn.1992-08.com.netapp:sn.e0a0273a60b711f09deed039ead647e8:subsystem.svm1_subsystem_553 iopolicy changed from numa to queue-depth [98727.635617] nvme nvme0: creating 4 I/O queues. [98727.638218] nvme nvme0: mapped 4/0/0 default/read/poll queues. [98741.459565] nvme nvme1: creating 4 I/O queues. [98741.462227] nvme nvme1: mapped 4/0/0 default/read/poll queues.
Cheers,
Thomas