nvme-pic: improve max I/O queue handling
cover nvme-pic: improve max I/O queue handling
Hi,
while searching for a bug around zPCI + NVMe IRQ handling on a distro kernel, I got confused around handling of the maximum number of I/O queues in the NVMe driver.
I think I groked it in the end but would like to propose the following improvements, that said I'm quite new to this code.
I tested both patches on s390x (with a debug config) and x86_64 so with both data center and consumer NVMes.
For the second patch, since I don't own a device with the quirk, I tried always returning 1 from nvme_max_io_queues() and confirmed that on my Evo 970 Pro this resulted in about half the performance in a fio test but did not otherwise break things. I couldn't find a reason why allocating only the I/O queues we actually use would be problematic in the code either but I might have missed something of course.
Best regards,
Niklas Schnelle
Niklas Schnelle (2):
nvme-pci: drop min() from nr_io_queues assignment
nvme-pci: don't allocate unused I/O queues
drivers/nvme/host/pci.c | 17 +++++++---------- 1 file changed, 7 insertions(+), 10 deletions(-)
Signature · 1 line
2.17.1
PATCH 1/2 nvme-pci: drop min() from nr_io_queues assignment
in nvme_setup_io_queues() the number of I/O queues is set to either 1 in case of a quirky Apple device or to the min of nvme_max_io_queues() or dev->nr_allocated_queues - 1.
This is unnecessarily complicated as dev->nr_allocated_queues is only assigned once and is nvme_max_io_queues() + 1.
--- drivers/nvme/host/pci.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-)
drivers/nvme/host/pci.c +1 -2
@@ -2114,8 +2114,7 @@ static int nvme_setup_io_queues(struct nvme_dev *dev)if (dev->ctrl.quirks &NVME_QUIRK_SHARED_TAGS ) nr_io_queues =1 ;else - nr_io_queues =min (nvme_max_io_queues (dev), - dev->nr_allocated_queues -1 ); + nr_io_queues = dev->nr_allocated_queues -1 ; result =nvme_set_queue_count (&dev->ctrl, &nr_io_queues);if (result <0 )
Signature · 1 line
2.17.1
Re: [PATCH 1/2] nvme-pci: drop min() from nr_io_queues assignment
Thanks,
applied both patches to nvme-5.11.
_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
PATCH 2/2 nvme-pci: don't allocate unused I/O queues
In reply to Niklas Schnellecurrently the NVME_QUIRK_SHARED_TAGS quirk for Apple devices is handled during the assignment of nr_io_queues in nvme_setup_io_queues(). This however means that for these devices nvme_max_io_queues() will actually not return the supported maximum which is confusing and unexpected and also means that in nvme_probe() we are allocating for I/O queues that will never be used.
Fix this by moving the quirk handling into nvme_max_io_queues().
--- drivers/nvme/host/pci.c | 16 +++++++--------- 1 file changed, 7 insertions(+), 9 deletions(-)
drivers/nvme/host/pci.c +7 -9
@@ -2089,6 +2089,12 @@ static void nvme_disable_io_queues(struct nvme_dev *dev)static unsigned int nvme_max_io_queues (struct nvme_dev *dev) { +/* +* If tags are shared with admin queue (Apple bug), then +* make sure we only use one IO queue. +*/ +if (dev->ctrl.quirks &NVME_QUIRK_SHARED_TAGS ) +return 1 ;return num_possible_cpus () + dev->nr_write_queues + dev->nr_poll_queues; }
@@ -2107,15 +2113,7 @@ static int nvme_setup_io_queues(struct nvme_dev *dev) dev->nr_write_queues = write_queues; dev->nr_poll_queues = poll_queues; -/* -* If tags are shared with admin queue (Apple bug), then -* make sure we only use one IO queue. -*/ -if (dev->ctrl.quirks &NVME_QUIRK_SHARED_TAGS ) - nr_io_queues =1 ; -else - nr_io_queues = dev->nr_allocated_queues -1 ; - + nr_io_queues = dev->nr_allocated_queues -1 ; result =nvme_set_queue_count (&dev->ctrl, &nr_io_queues);if (result <0 )return result;
Signature · 1 line
2.17.1
Quoted 5 lines · Niklas Schnelle
while searching for a bug around zPCI + NVMe IRQ handling on a distro kernel, I got confused around handling of the maximum number of I/O queues in the NVMe driver.
I think I groked it in the end but would like to propose the following improvements, that said I'm quite new to this code.
Quoted 8 lines
I tested both patches on s390x (with a debug config) and
x86_64so with both data center and consumer NVMes.
For the second patch, since I don't own a device with the quirk, I tried always returning 1 fromnvme_max_io_queues()and confirmed that on my Evo 970 Pro this resulted in about half the performance in a fio test but did not otherwise break things. I couldn't find a reason why allocating only the I/O queues we actually use would be problematic in the code either but I might have missed something of course.
I don't think you missed anything, and the series looks like a reasonable cleanup. I suspect the code was left over from a time when we didn't allocate the possible queues up-front.
Quoted 15 lines · Keith Busch
Quoted 5 lines · Niklas Schnelle
while searching for a bug around zPCI + NVMe IRQ handling on a distro kernel, I got confused around handling of the maximum number of I/O queues in the NVMe driver.
I think I groked it in the end but would like to propose the following improvements, that said I'm quite new to this code.Quoted 8 lines
I tested both patches on s390x (with a debug config) and
x86_64so with both data center and consumer NVMes.
For the second patch, since I don't own a device with the quirk, I tried always returning 1 fromnvme_max_io_queues()and confirmed that on my Evo 970 Pro this resulted in about half the performance in a fio test but did not otherwise break things. I couldn't find a reason why allocating only the I/O queues we actually use would be problematic in the code either but I might have missed something of course.
Quoted 6 lines
I don't think you missed anything, and the series looks like a reasonable cleanup. I suspect the code was left over from a time when we didn't allocate the possible queues up-front.
Reviewed-by Keith Busch <[email protected]>
You got to get something wrong, I hope in this case it's just the subject of the cover letter :D
Thanks for the review, I appreciate it. Might be getting ahead of myself but I'm curious who would take this change through their tree if accepted?
Keith Busch
maintainerQuoted 2 lines · Niklas Schnelle
You got to get something wrong, I hope in this case it's just the subject of the cover letter :D
I suppose the change logs could be worded a little better :)
Quoted 3 lines
Thanks for the review, I appreciate it. Might be getting ahead of myself but I'm curious who would take this change through their tree if accepted?
The linux-nvme tree is located here:
http://git.infradead.org/nvme.git
Christoph is currently handling patch commits there.
Quoted 5 lines · Keith Busch
Quoted 2 lines · Niklas Schnelle
You got to get something wrong, I hope in this case it's just the subject of the cover letter :D
I suppose the change logs could be worded a little better :)
Do you think I should send a v2 with an improved message? I just realized I'm pretty alone (and wrong) in starting the commit message body lower case too.
Christoph Hellwig
maintainerQuoted 4 lines · Niklas Schnelle
Quoted 1 line · Keith Busch
On Thu, Nov 12, 2020 at 04:45:35PM +0100, Niklas Schnelle wrote:
Quoted 8 lines
Quoted 4 lines
Quoted 2 lines
You got to get something wrong, I hope in this case it's just the subject of the cover letter :D
I suppose the change logs could be worded a little better :)
Do you think I should send a v2 with an improved message? I just realized I'm pretty alone (and wrong) in starting the commit message body lower case too.
I'll pull in soon, no need to resend.
Quoted 8 lines · Christoph Hellwig
Quoted 7 lines · Niklas Schnelle
Quoted 4 lines · Keith Busch
Quoted 2 lines · Niklas Schnelle
You got to get something wrong, I hope in this case it's just the subject of the cover letter :D
Quoted 8 lines
Quoted 5 lines
Quoted 1 line
I suppose the change logs could be worded a little better :)
Do you think I should send a v2 with an improved message? I just realized I'm pretty alone (and wrong) in starting the commit message body lower case too.
I'll pull in soon, no need to resend.
Thanks, appreciate your effort!