nvme/host/pci: Fix a race in controller removal
PATCH 1/2 nvme/host/pci: Fix a race in controller removal
This race is hard to hit in general, now that we have the shutdown_lock in both nvme_reset_work() and nvme_dev_disable()
The real issue is that after doing all the setup work in nvme_reset_work(), when get another timeout (nvme_timeout()), then we proceed to disable the controller. This causes the reset work to only partially progress and then fail.
Depending on the progress made, we call into nvme_remove_dead_ctrl(), which does another nvme_dev_disable() freezing the block-mq queues.
I've noticed a race with udevd with udevd trying to re-read the partition table, it ends up with the bd_mutex held and it ends up waiting in blk_queue_enter(), since we froze the queues in nvme_dev_disable(). nvme_kill_queues() calls revalidate_disk() and ends up waiting on the bd_mutex resulting in a deadlock.
Allow the hung tasks a chance by unfreezing the queues after setting dying bits on the queue, then call revalidate_disk() to update the disk size.
NOTE: I've seen this race when the controller does not respond to IOs or abort requests, but responds to other commands and even signals it's ready after its reset, but still drops IO. I've tested this by emulating the behaviour in the driver.
--- drivers/nvme/host/core.c | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-)
drivers/nvme/host/core.c +6 -1
@@ -103,10 +103,15 @@ static void nvme_set_queue_dying(struct nvme_ns *ns)*/ if (!ns->disk ||test_and_set_bit (NVME_NS_DEAD , &ns->flags))return ; -revalidate_disk (ns->disk);blk_set_queue_dying (ns->queue); +/* +* Allow any pending udevd commands to be unblocked +* so that revalidate_disk can then get bd_mutex +*/ +blk_mq_unfreeze_queue (ns->queue);/* Forcibly unquiesce queues to avoid blocking dispatch */ blk_mq_unquiesce_queue (ns->queue); +revalidate_disk (ns->disk); }static void nvme_queue_scan (struct nvme_ctrl *ctrl)
Signature · 7 lines
2.16.5 _______________________________________________ Linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme
PATCH 2/2 nvme/host/core: Allow overriding of wait_ready timeout
Largely for debugging purposes where controllers with large timeouts get stuck during reset.
--- drivers/nvme/host/core.c | 8 ++++++++ drivers/nvme/host/nvme.h | 3 +++ 2 files changed, 11 insertions(+)
drivers/nvme/host/core.c +8 -0
@@ -40,6 +40,11 @@ module_param_named(io_timeout, nvme_io_timeout, uint, 0644);MODULE_PARM_DESC (io_timeout,"timeout in seconds for I/O" );EXPORT_SYMBOL_GPL (nvme_io_timeout); +unsigned int nvme_wait_ready_timeout =0 ; +module_param_named (wait_ready_timeout, nvme_wait_ready_timeout, uint,0644 ); +MODULE_PARM_DESC (wait_ready_timeout,"timeout in seconds for wait ready on reset" ); +EXPORT_SYMBOL_GPL (nvme_wait_ready_timeout); +static unsigned char shutdown_timeout =5 ;module_param (shutdown_timeout, byte,0644 );MODULE_PARM_DESC (shutdown_timeout,"timeout in seconds for controller shutdown" );
@@ -1937,6 +1942,9 @@ static int nvme_wait_ready(struct nvme_ctrl *ctrl, u64 cap, bool enabled)u32 csts, bit = enabled ?NVME_CSTS_RDY :0 ;int ret; +if (nvme_wait_ready_timeout) + timeout =NVME_WAIT_READY_TIMEOUT + jiffies; +while ((ret = ctrl->ops->reg_read32 (ctrl,NVME_REG_CSTS , &csts)) ==0 ) {if (csts == ~0 )return -ENODEV ;
drivers/nvme/host/nvme.h +3 -0
@@ -21,6 +21,9 @@extern unsigned int nvme_io_timeout;#define NVME_IO_TIMEOUT (nvme_io_timeout * HZ) +extern unsigned int nvme_wait_ready_timeout; +#define NVME_WAIT_READY_TIMEOUT (nvme_wait_ready_timeout * HZ) +extern unsigned int admin_timeout;#define ADMIN_TIMEOUT (admin_timeout * HZ)
Signature · 7 lines
2.16.5 _______________________________________________ Linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme
Keith Busch
maintainerRe: [PATCH 2/2] nvme/host/core: Allow overriding of wait_ready timeout
Quoted 14 lines · Balbir Singh
Largely for debugging purposes where controllers with large timeouts get stuck during reset.
Signed-off-by Balbir Singh <[email protected]>--- drivers/nvme/host/core.c | 8 ++++++++ drivers/nvme/host/nvme.h | 3 +++ 2 files changed, 11 insertions(+)drivers/nvme/host/core.c +0
-0
drivers/nvme/host/core.c module_param_named(io_timeout, nvme_io_timeout, uint, 0644); · line 46
Quoted 7 lines
drivers/nvme/host/core.c +4
-0@@ -40,6 +40,11 @@ module_param_named(io_timeout, nvme_io_timeout, uint, 0644);MODULE_PARM_DESC (io_timeout,"timeout in seconds for I/O" );EXPORT_SYMBOL_GPL (nvme_io_timeout); +unsigned int nvme_wait_ready_timeout =0 ; +module_param_named (wait_ready_timeout, nvme_wait_ready_timeout, uint,0644 ); +MODULE_PARM_DESC (wait_ready_timeout,"timeout in seconds for wait ready on reset" ); +EXPORT_SYMBOL_GPL (nvme_wait_ready_timeout);
There's no need to export this symbol.
_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Re: [PATCH 2/2] nvme/host/core: Allow overriding of wait_ready timeout
Quoted 19 lines · Keith Busch
Quoted 18 lines · Balbir Singh
Largely for debugging purposes where controllers with large timeouts get stuck during reset.
Signed-off-by Balbir Singh <[email protected]>--- drivers/nvme/host/core.c | 8 ++++++++ drivers/nvme/host/nvme.h | 3 +++ 2 files changed, 11 insertions(+)drivers/nvme/host/core.c +0
-0@@ -40,6 +40,11 @@ module_param_named(io_timeout, nvme_io_timeout, uint, 0644);MODULE_PARM_DESC (io_timeout,"timeout in seconds for I/O" );EXPORT_SYMBOL_GPL (nvme_io_timeout);
Quoted 8 lines
drivers/nvme/host/core.c module_param_named(io_timeout, nvme_io_timeout, · line 49
Quoted 6 lines
drivers/nvme/host/core.c +4
-0+unsigned int nvme_wait_ready_timeout =0 ; +module_param_named (wait_ready_timeout, nvme_wait_ready_timeout, uint,0644 ); +MODULE_PARM_DESC(wait_ready_timeout, "timeout in seconds for wait ready on reset"); +EXPORT_SYMBOL_GPL(nvme_wait_ready_timeout);There's no need to export this symbol.
Good point, I'll repost without the export
Balbir Singh.
_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Keith Busch
maintainerQuoted 3 lines · Balbir Singh
Allow the hung tasks a chance by unfreezing the queues after setting dying bits on the queue, then call
revalidate_disk()to update the disk size.
Isn't it enough to just set the queue to dying? That should unblock everything calling blk_queue_enter().
_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Quoted 7 lines · Keith Busch
Quoted 3 lines · Balbir Singh
Allow the hung tasks a chance by unfreezing the queues after setting dying bits on the queue, then call
revalidate_disk()to update the disk size.Isn't it enough to just set the queue to dying? That should unblock everything calling
blk_queue_enter().
The real issue is that by then udevd has already called into blk_queue_enter() and is waiting on wait_event(q->mq_freeze_wq,...), so we do need to unfreeze and wake_all waiting on the event.
Balbir Singh.
_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Quoted 3 lines · Balbir
The real issue is that by then udevd has already called into
blk_queue_enter()and is waiting onwait_event(q->mq_freeze_wq,...), so we do need to unfreeze andwake_allwaiting on the event.
I don't think that's correct. blk_set_queue_dying() wakes up blk_queue_enter() and causes it to return -ENODEV.
Bart.
_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Quoted 8 lines · Bart Van Assche
Quoted 3 lines · Balbir
The real issue is that by then udevd has already called into
blk_queue_enter()and is waiting onwait_event(q->mq_freeze_wq,...), so we do need to unfreeze andwake_allwaiting on the event.I don't think that's correct.
blk_set_queue_dying()wakes upblk_queue_enter()and causes it to return-ENODEV.
Fair enough.. I presume looking at the name, it was twiddling bits. I am going to test with just the re-ordering and check.
Quoted 1 line
Bart.
_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Quoted 8 lines · Balbir Singh
Quoted 7 lines · Bart Van Assche
Quoted 5 lines · Balbir
The real issue is that by then udevd has already called into
blk_queue_enter()
and is waiting onwait_event(q->mq_freeze_wq,...), so we do need to unfreeze
andwake_allwaiting on the event.
Quoted 7 lines
Quoted 3 lines
I don't think that's correct.
blk_set_queue_dying()wakes upblk_queue_enter()and causes it to return-ENODEV.Fair enough.. I presume looking at the name, it was twiddling bits. I am going to test with just the re-ordering and check.
Upon further check, blk_set_queue_dying() tries to freeze the queue again, but then it does do an unconditional wakeup. I can confirm that just re-ordering seems to help and blk_set_queue_dying() seems to have the right bits.
Respinning for comments, thanks for the reviews Bart and Keith
Balbir Singh.
Quoted 1 line
Quoted 1 line
Bart.
_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme