nvme: Add error checking in nvme_create_io_queues
Subsystems (2)
PATCH 1/1 nvme: Add error checking in nvme_create_io_queues
allow nvme_create_io_queues to return an error if the first io queue fails to be allocated or initialized
--- drivers/block/nvme-core.c | 20 ++++++++++++++++---- 1 file changed, 16 insertions(+), 4 deletions(-)
drivers/block/nvme-core.c +16 -4
@@ -2186,17 +2186,25 @@ static void nvme_alloc_ns(struct nvme_dev *dev, unsigned nsid)kfree (ns); } -static void nvme_create_io_queues (struct nvme_dev *dev) +static int nvme_create_io_queues (struct nvme_dev *dev) {unsigned i;for (i = dev->queue_count; i <= dev->max_qid; i++) -if (!nvme_alloc_queue (dev, i, dev->q_depth)) +if (!nvme_alloc_queue (dev, i, dev->q_depth)) { +if (dev->queue_count ==1 ) +return -ENODEV ;break ; + }for (i = dev->online_queues; i <= dev->queue_count -1 ; i++) -if (nvme_create_queue (dev->queues[i], i)) +if (nvme_create_queue (dev->queues[i], i)) { +if (dev->online_queues ==1 ) +return -ENODEV ;break ; + } + +return 0 ; }static int set_queue_count (struct nvme_dev *dev,int count)
@@ -2350,7 +2358,11 @@ static int nvme_setup_io_queues(struct nvme_dev *dev)/* Free previously allocated queues that are no longer usable */ nvme_free_queues (dev, nr_io_queues +1 ); -nvme_create_io_queues (dev); + result =nvme_create_io_queues (dev); +if (result) { +dev_warn (dev->dev,"Failed to create io queues\n" ); + goto free_queues; + }return 0 ;
Signature · 1 line
2.5.0
PATCH 1/1 nvme: Add error checking in nvme_create_io_queues
Quoted 2 lines · Jon Derrick
allow
nvme_create_io_queuesto return an error if the first io queue fails to be allocated or initialized
If the driver gets far enough to attempt creating IO queues, the device can be managed so I don't think we want to error out and bail on the device in this scenario, even though it is not IO capable.
Quoted 4 lines
Signed-off-by Jon Derrick <jonathan.derrick at intel.com>---
drivers/block/nvme-core.c| 20 ++++++++++++++++----1 file changed, 16 insertions(+), 4 deletions(-)
PATCH 1/1 nvme: Add error checking in nvme_create_io_queues
Quoted 8 lines · Keith Busch
Quoted 2 lines · Jon Derrick
allow
nvme_create_io_queuesto return an error if the first io queue fails to be allocated or initializedIf the driver gets far enough to attempt creating IO queues, the device can be managed so I don't think we want to error out and bail on the device in this scenario, even though it is not IO capable.
Very good point.
I think just the dev_warn should suffice then, without the teardown. After testing I have not seeing any issues with only leaving the controller handle up.
Quoted 4 lines
Quoted 4 lines
Signed-off-by Jon Derrick <jonathan.derrick at intel.com>---
drivers/block/nvme-core.c| 20 ++++++++++++++++----1 file changed, 16 insertions(+), 4 deletions(-)
PATCH 1/1 nvme: Add error checking in nvme_create_io_queues
In reply to Keith BuschQuoted 7 lines · Keith Busch
Quoted 2 lines · Jon Derrick
allow
nvme_create_io_queuesto return an error if the first io queue fails to be allocated or initializedIf the driver gets far enough to attempt creating IO queues, the device can be managed so I don't think we want to error out and bail on the device in this scenario, even though it is not IO capable.
But it's left in a somewhat odd state, which is the cleaned up via nvme_dev_resume and the odd and misnamed nvme_remove_disks.
I think nvme_create_io_queues should at least make sure any partial created I/O queues are properly cleaned up on a failure of nvme_alloc_queue or nvme_create_queue.
Keith, do you remember why nvme_remove_disks is called through a workqueue? I'd much rather do the work in line and in the proper place, queues should never leave nvme_create_io_queues half-setup, and namespaces should probably be removed by nvme_dev_start if we don't have any working I/O queues after a resume or reset.
While we're at it: can we rename nvme_dev_remove to nvme_remove_namespaces? Also dev->namespaces seems to lack proper synchronization.
PATCH 1/1 nvme: Add error checking in nvme_create_io_queues
Quoted 5 lines · Christoph Hellwig
Keith, do you remember why
nvme_remove_disksis called through a workqueue? I'd much rather do the work in line and in the proper place, queues should never leavenvme_create_io_queueshalf-setup, and namespaces should probably be removed bynvme_dev_startif we don't have any working I/O queues after a resume or reset.
A resume from S3/S4 will deadlock if we try to delete inaccessible disks from the resume context, so using the work queue instead. There is probably a better way to do this now that we probe asynchronously, we can resume the same way.
PATCH 1/1 nvme: Add error checking in nvme_create_io_queues
Quoted 11 lines · Keith Busch
Quoted 5 lines · Christoph Hellwig
Keith, do you remember why
nvme_remove_disksis called through a workqueue? I'd much rather do the work in line and in the proper place, queues should never leavenvme_create_io_queueshalf-setup, and namespaces should probably be removed bynvme_dev_startif we don't have any working I/O queues after a resume or reset.A resume from S3/S4 will deadlock if we try to delete inaccessible disks from the resume context, so using the work queue instead. There is probably a better way to do this now that we probe asynchronously, we can resume the same way.
Is this still the case with the nvme_io_incapable check in place? What are the symptoms? Unfortunately I can't test PM easily on my test systems with NVMe cards as they are all remotely managed server class systems.
Anyway, I have some pending patches for restructuring the reset path in the NVMe driver and I will keep this in mind, but careful testing by someone who has PM capable hardware would be really appreciated.
PATCH 1/1 nvme: Add error checking in nvme_create_io_queues
Quoted 10 lines · Christoph Hellwig
Quoted 4 lines · Keith Busch
A resume from S3/S4 will deadlock if we try to delete inaccessible disks from the resume context, so using the work queue instead. There is probably a better way to do this now that we probe asynchronously, we can resume the same way.
Is this still the case with the
nvme_io_incapablecheck in place? What are the symptoms? Unfortunately I can't test PM easily on my test systems with NVMe cards as they are all remotely managed server class systems.
Good point, I think that will work. I don't recall the exact sympotoms (it's been two years), but I believe related to the same deadlock. I'll try this out on a desktop system.