Skip to contentSkip to conversation

nvme-pic: improve max I/O queue handling

PATCH Applied Niklas Schnelle 3 participants 10 messages Nov 12, 2020 – Nov 14, 2020 NVMe

lore
Share & tools
lore
#3 lore raw

PATCH 2/2 nvme-pci: don't allocate unused I/O queues

In reply to Niklas Schnelle

currently the NVME_QUIRK_SHARED_TAGS quirk for Apple devices is handled during the assignment of nr_io_queues in nvme_setup_io_queues(). This however means that for these devices nvme_max_io_queues() will actually not return the supported maximum which is confusing and unexpected and also means that in nvme_probe() we are allocating for I/O queues that will never be used.
Fix this by moving the quirk handling into nvme_max_io_queues().

---
 drivers/nvme/host/pci.c | 16 +++++++---------
 1 file changed, 7 insertions(+), 9 deletions(-)
drivers/nvme/host/pci.c +7 -9
index b56250b83bdf..0f8cea5b30eb 100644
@@ -2089,6 +2089,12 @@ static void nvme_disable_io_queues(struct nvme_dev *dev)
 
 static unsigned int nvme_max_io_queues(struct nvme_dev *dev)
 {
+	/*
+	 * If tags are shared with admin queue (Apple bug), then
+	 * make sure we only use one IO queue.
+	 */
+	if (dev->ctrl.quirks & NVME_QUIRK_SHARED_TAGS)
+		return 1;
 	return num_possible_cpus() + dev->nr_write_queues + dev->nr_poll_queues;
 }
 
@@ -2107,15 +2113,7 @@ static int nvme_setup_io_queues(struct nvme_dev *dev)
 	dev->nr_write_queues = write_queues;
 	dev->nr_poll_queues = poll_queues;
 
-	/*
-	 * If tags are shared with admin queue (Apple bug), then
-	 * make sure we only use one IO queue.
-	 */
-	if (dev->ctrl.quirks & NVME_QUIRK_SHARED_TAGS)
-		nr_io_queues = 1;
-	else
-		nr_io_queues = dev->nr_allocated_queues - 1;
-
+	nr_io_queues = dev->nr_allocated_queues - 1;
 	result = nvme_set_queue_count(&dev->ctrl, &nr_io_queues);
 	if (result < 0)
 		return result;
Signature · 1 line
2.17.1
#1 lore raw

PATCH 1/2 nvme-pci: drop min() from nr_io_queues assignment

In reply to Niklas Schnelle

in nvme_setup_io_queues() the number of I/O queues is set to either 1 in case of a quirky Apple device or to the min of nvme_max_io_queues() or dev->nr_allocated_queues - 1.
This is unnecessarily complicated as dev->nr_allocated_queues is only assigned once and is nvme_max_io_queues() + 1.

---
 drivers/nvme/host/pci.c | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)
drivers/nvme/host/pci.c +1 -2
index 0578ff253c47..b56250b83bdf 100644
@@ -2114,8 +2114,7 @@ static int nvme_setup_io_queues(struct nvme_dev *dev)
 	if (dev->ctrl.quirks & NVME_QUIRK_SHARED_TAGS)
 		nr_io_queues = 1;
 	else
-		nr_io_queues = min(nvme_max_io_queues(dev),
-				   dev->nr_allocated_queues - 1);
+		nr_io_queues = dev->nr_allocated_queues - 1;
 
 	result = nvme_set_queue_count(&dev->ctrl, &nr_io_queues);
 	if (result < 0)
Signature · 1 line
2.17.1
#0 lore raw

cover nvme-pic: improve max I/O queue handling

Hi,

while searching for a bug around zPCI + NVMe IRQ handling on a distro kernel, I got confused around handling of the maximum number of I/O queues in the NVMe driver.
I think I groked it in the end but would like to propose the following improvements, that said I'm quite new to this code.
I tested both patches on s390x (with a debug config) and x86_64 so with both data center and consumer NVMes.
For the second patch, since I don't own a device with the quirk, I tried always returning 1 from nvme_max_io_queues() and confirmed that on my Evo 970 Pro this resulted in about half the performance in a fio test but did not otherwise break things. I couldn't find a reason why allocating only the I/O queues we actually use would be problematic in the code either but I might have missed something of course.

Best regards,
Niklas Schnelle

Niklas Schnelle (2):
nvme-pci: drop min() from nr_io_queues assignment
nvme-pci: don't allocate unused I/O queues

 drivers/nvme/host/pci.c | 17 +++++++----------
 1 file changed, 7 insertions(+), 10 deletions(-)
Signature · 1 line
2.17.1
maintainer Reviewed-by on the v1 series
#4 lore raw
Quoted 5 lines · Niklas Schnelle

while searching for a bug around zPCI + NVMe IRQ handling on a distro kernel, I got confused around handling of the maximum number of I/O queues in the NVMe driver.
I think I groked it in the end but would like to propose the following improvements, that said I'm quite new to this code.

Quoted 8 lines

I tested both patches on s390x (with a debug config) and x86_64 so with both data center and consumer NVMes.
For the second patch, since I don't own a device with the quirk, I tried always returning 1 from nvme_max_io_queues() and confirmed that on my Evo 970 Pro this resulted in about half the performance in a fio test but did not otherwise break things. I couldn't find a reason why allocating only the I/O queues we actually use would be problematic in the code either but I might have missed something of course.

I don't think you missed anything, and the series looks like a reasonable cleanup. I suspect the code was left over from a time when we didn't allocate the possible queues up-front.

#5 lore raw
Quoted 15 lines · Keith Busch
Quoted 5 lines · Niklas Schnelle

while searching for a bug around zPCI + NVMe IRQ handling on a distro kernel, I got confused around handling of the maximum number of I/O queues in the NVMe driver.
I think I groked it in the end but would like to propose the following improvements, that said I'm quite new to this code.

Quoted 8 lines

I tested both patches on s390x (with a debug config) and x86_64 so with both data center and consumer NVMes.
For the second patch, since I don't own a device with the quirk, I tried always returning 1 from nvme_max_io_queues() and confirmed that on my Evo 970 Pro this resulted in about half the performance in a fio test but did not otherwise break things. I couldn't find a reason why allocating only the I/O queues we actually use would be problematic in the code either but I might have missed something of course.

Quoted 6 lines

I don't think you missed anything, and the series looks like a reasonable cleanup. I suspect the code was left over from a time when we didn't allocate the possible queues up-front.

You got to get something wrong, I hope in this case it's just the subject of the cover letter :D
Thanks for the review, I appreciate it. Might be getting ahead of myself but I'm curious who would take this change through their tree if accepted?

maintainer
#6 lore raw
Quoted 2 lines · Niklas Schnelle

You got to get something wrong, I hope in this case it's just the subject of the cover letter :D

I suppose the change logs could be worded a little better :)

Quoted 3 lines

Thanks for the review, I appreciate it. Might be getting ahead of myself but I'm curious who would take this change through their tree if accepted?

The linux-nvme tree is located here:

http://git.infradead.org/nvme.git

Christoph is currently handling patch commits there.

#7 lore raw
Quoted 5 lines · Keith Busch
Quoted 2 lines · Niklas Schnelle

You got to get something wrong, I hope in this case it's just the subject of the cover letter :D

I suppose the change logs could be worded a little better :)

Do you think I should send a v2 with an improved message? I just realized I'm pretty alone (and wrong) in starting the commit message body lower case too.

#8 lore raw
Quoted 4 lines · Niklas Schnelle
Quoted 1 line · Keith Busch

On Thu, Nov 12, 2020 at 04:45:35PM +0100, Niklas Schnelle wrote:

Quoted 8 lines
Quoted 4 lines
Quoted 2 lines

You got to get something wrong, I hope in this case it's just the subject of the cover letter :D

I suppose the change logs could be worded a little better :)

Do you think I should send a v2 with an improved message? I just realized I'm pretty alone (and wrong) in starting the commit message body lower case too.

I'll pull in soon, no need to resend.

#9 lore raw
Quoted 8 lines · Christoph Hellwig
Quoted 7 lines · Niklas Schnelle
Quoted 4 lines · Keith Busch
Quoted 2 lines · Niklas Schnelle

You got to get something wrong, I hope in this case it's just the subject of the cover letter :D

Quoted 8 lines
Quoted 5 lines
Quoted 1 line

I suppose the change logs could be worded a little better :)

Do you think I should send a v2 with an improved message? I just realized I'm pretty alone (and wrong) in starting the commit message body lower case too.

I'll pull in soon, no need to resend.

Thanks, appreciate your effort!