Skip to contentSkip to conversation

nvme: Add error checking in nvme_create_io_queues

PATCH New Jon Derrick 3 participants 7 messages Sep 24, 2015 – Sep 28, 2015

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 nvme: Add error checking in nvme_create_io_queues

allow nvme_create_io_queues to return an error if the first io queue fails to be allocated or initialized

Signed-off-by Jon Derrick <jonathan.derrick at intel.com>
---
 drivers/block/nvme-core.c | 20 ++++++++++++++++----
 1 file changed, 16 insertions(+), 4 deletions(-)
drivers/block/nvme-core.c +16 -4
index b97fc3f..e0ec139 100644
@@ -2186,17 +2186,25 @@ static void nvme_alloc_ns(struct nvme_dev *dev, unsigned nsid)
 	kfree(ns);
 }
 
-static void nvme_create_io_queues(struct nvme_dev *dev)
+static int nvme_create_io_queues(struct nvme_dev *dev)
 {
 	unsigned i;
 
 	for (i = dev->queue_count; i <= dev->max_qid; i++)
-		if (!nvme_alloc_queue(dev, i, dev->q_depth))
+		if (!nvme_alloc_queue(dev, i, dev->q_depth)) {
+			if (dev->queue_count == 1)
+				return -ENODEV;
 			break;
+		}
 
 	for (i = dev->online_queues; i <= dev->queue_count - 1; i++)
-		if (nvme_create_queue(dev->queues[i], i))
+		if (nvme_create_queue(dev->queues[i], i)) {
+			if (dev->online_queues == 1)
+				return -ENODEV;
 			break;
+		}
+
+	return 0;
 }
 
 static int set_queue_count(struct nvme_dev *dev, int count)
@@ -2350,7 +2358,11 @@ static int nvme_setup_io_queues(struct nvme_dev *dev)
 
 	/* Free previously allocated queues that are no longer usable */
 	nvme_free_queues(dev, nr_io_queues + 1);
-	nvme_create_io_queues(dev);
+	result = nvme_create_io_queues(dev);
+	if (result) {
+		dev_warn(dev->dev, "Failed to create io queues\n");
+		goto free_queues;
+	}
 
 	return 0;
 
Signature · 1 line
2.5.0
#1 lore raw

PATCH 1/1 nvme: Add error checking in nvme_create_io_queues

Quoted 2 lines · Jon Derrick

allow nvme_create_io_queues to return an error if the first io queue fails to be allocated or initialized

If the driver gets far enough to attempt creating IO queues, the device can be managed so I don't think we want to error out and bail on the device in this scenario, even though it is not IO capable.

Quoted 4 lines
Signed-off-by Jon Derrick <jonathan.derrick at intel.com>

---
drivers/block/nvme-core.c | 20 ++++++++++++++++----

1 file changed, 16 insertions(+), 4 deletions(-)
#2 lore raw

PATCH 1/1 nvme: Add error checking in nvme_create_io_queues

Quoted 8 lines · Keith Busch
Quoted 2 lines · Jon Derrick

allow nvme_create_io_queues to return an error if the first io queue fails to be allocated or initialized

If the driver gets far enough to attempt creating IO queues, the device can be managed so I don't think we want to error out and bail on the device in this scenario, even though it is not IO capable.

Very good point.

I think just the dev_warn should suffice then, without the teardown. After testing I have not seeing any issues with only leaving the controller handle up.

Quoted 4 lines
Quoted 4 lines
Signed-off-by Jon Derrick <jonathan.derrick at intel.com>

---
drivers/block/nvme-core.c | 20 ++++++++++++++++----

1 file changed, 16 insertions(+), 4 deletions(-)
#3 lore raw

PATCH 1/1 nvme: Add error checking in nvme_create_io_queues

In reply to Keith Busch
Quoted 7 lines · Keith Busch
Quoted 2 lines · Jon Derrick

allow nvme_create_io_queues to return an error if the first io queue fails to be allocated or initialized

If the driver gets far enough to attempt creating IO queues, the device can be managed so I don't think we want to error out and bail on the device in this scenario, even though it is not IO capable.

But it's left in a somewhat odd state, which is the cleaned up via nvme_dev_resume and the odd and misnamed nvme_remove_disks.

I think nvme_create_io_queues should at least make sure any partial created I/O queues are properly cleaned up on a failure of nvme_alloc_queue or nvme_create_queue.

Keith, do you remember why nvme_remove_disks is called through a workqueue? I'd much rather do the work in line and in the proper place, queues should never leave nvme_create_io_queues half-setup, and namespaces should probably be removed by nvme_dev_start if we don't have any working I/O queues after a resume or reset.

While we're at it: can we rename nvme_dev_remove to nvme_remove_namespaces? Also dev->namespaces seems to lack proper synchronization.

#4 lore raw

PATCH 1/1 nvme: Add error checking in nvme_create_io_queues

Quoted 5 lines · Christoph Hellwig

Keith, do you remember why nvme_remove_disks is called through a workqueue? I'd much rather do the work in line and in the proper place, queues should never leave nvme_create_io_queues half-setup, and namespaces should probably be removed by nvme_dev_start if we don't have any working I/O queues after a resume or reset.

A resume from S3/S4 will deadlock if we try to delete inaccessible disks from the resume context, so using the work queue instead. There is probably a better way to do this now that we probe asynchronously, we can resume the same way.

#5 lore raw

PATCH 1/1 nvme: Add error checking in nvme_create_io_queues

Quoted 11 lines · Keith Busch
Quoted 5 lines · Christoph Hellwig

Keith, do you remember why nvme_remove_disks is called through a workqueue? I'd much rather do the work in line and in the proper place, queues should never leave nvme_create_io_queues half-setup, and namespaces should probably be removed by nvme_dev_start if we don't have any working I/O queues after a resume or reset.

A resume from S3/S4 will deadlock if we try to delete inaccessible disks from the resume context, so using the work queue instead. There is probably a better way to do this now that we probe asynchronously, we can resume the same way.

Is this still the case with the nvme_io_incapable check in place? What are the symptoms? Unfortunately I can't test PM easily on my test systems with NVMe cards as they are all remotely managed server class systems.

Anyway, I have some pending patches for restructuring the reset path in the NVMe driver and I will keep this in mind, but careful testing by someone who has PM capable hardware would be really appreciated.

#6 lore raw

PATCH 1/1 nvme: Add error checking in nvme_create_io_queues

Quoted 10 lines · Christoph Hellwig
Quoted 4 lines · Keith Busch

A resume from S3/S4 will deadlock if we try to delete inaccessible disks from the resume context, so using the work queue instead. There is probably a better way to do this now that we probe asynchronously, we can resume the same way.

Is this still the case with the nvme_io_incapable check in place? What are the symptoms? Unfortunately I can't test PM easily on my test systems with NVMe cards as they are all remotely managed server class systems.

Good point, I think that will work. I don't recall the exact sympotoms (it's been two years), but I believe related to the same deadlock. I'll try this out on a desktop system.