nvme: Ignore timeouts while a PCIe reset is pending
RFC PATCH 1/1 nvme: Ignore timeouts while a PCIe reset is pending
If an admin command timeout occurs while a PCIe reset (FLR) is pending, the CSTS bits may not be valid which could result in the controller being removed.
[372337.996566] nvme nvme0: I/O 0 QID 0 timeout, reset controller [372339.984662] nvme 0000:1c:00.0: enabling device (0000 -> 0002) [372339.984951] nvme nvme0: Removing after probe failure status: -19
--- drivers/nvme/host/pci.c | 13 +++++++++++++ 1 file changed, 13 insertions(+)
drivers/nvme/host/pci.c +13 -0
@@ -118,6 +118,8 @@ struct nvme_dev {struct nvme_ctrl ctrl;mempool_t *iod_mempool; +unsigned long flags; +#define NVME_CTRL_PCI_RESET_PENDING 0/* shadow doorbell buffer support: */ u32 *dbbuf_dbs;
@@ -1250,6 +1252,11 @@ static void nvme_warn_reset(struct nvme_dev *dev, u32 csts) csts, result); } +static bool nvme_pci_reset_pending (const struct nvme_dev *dev) +{ +return !!test_bit (NVME_CTRL_PCI_RESET_PENDING , &dev->flags); +} +static enum blk_eh_timer_return nvme_timeout (struct request *req,bool reserved) {struct nvme_iod *iod =blk_mq_rq_to_pdu (req);
@@ -1267,6 +1274,10 @@ static enum blk_eh_timer_return nvme_timeout(struct request *req, bool reserved)if (pci_channel_offline (to_pci_dev (dev->dev)))return BLK_EH_RESET_TIMER ; +/* If a PCIe reset (FLR) is pending, wait for it to complete */ +if (nvme_pci_reset_pending (dev)) +return BLK_EH_RESET_TIMER ; +/* * Reset immediately if the controller is failed */
@@ -2780,12 +2791,14 @@ static int nvme_probe(struct pci_dev *pdev, const struct pci_device_id *id)static void nvme_reset_prepare (struct pci_dev *pdev) {struct nvme_dev *dev =pci_get_drvdata (pdev); +set_bit (NVME_CTRL_PCI_RESET_PENDING , &dev->flags);nvme_dev_disable (dev, false); }static void nvme_reset_done (struct pci_dev *pdev) {struct nvme_dev *dev =pci_get_drvdata (pdev); +clear_bit (NVME_CTRL_PCI_RESET_PENDING , &dev->flags);nvme_reset_ctrl_sync (&dev->ctrl); }
Signature · 1 line
2.17.1
Keith Busch
maintainerRFC PATCH 1/1 nvme: Ignore timeouts while a PCIe reset is pending
Quoted 7 lines · Kenneth Heitke
If an admin command timeout occurs while a PCIe reset (FLR) is pending, the CSTS bits may not be valid which could result in the controller being removed.
[372337.996566] nvme nvme0: I/O 0 QID 0 timeout, reset controller [372339.984662] nvme 0000:1c:00.0: enabling device (0000 -> 0002) [372339.984951] nvme nvme0: Removing after probe failure status: -19
The disable reclaims all commands, including the ones it dispatches, so it sounds like you're talking about a race between the ones it dispatched and its timeout work. If so, we can just make sure commands sent during nvme_dev_disable never timeout, which are just the delete queue commands:
---
drivers/nvme/host/pci.c +1 -1
@@ -2210,7 +2210,7 @@ static int nvme_delete_queue(struct nvme_queue *nvmeq, u8 opcode)if (IS_ERR (req))return PTR_ERR (req); - req->timeout =ADMIN_TIMEOUT ; + req->timeout =UINT_MAX ; req->end_io_data = nvmeq;init_completion (&nvmeq->delete_done);
--
Keith Busch
maintainerRFC PATCH 1/1 nvme: Ignore timeouts while a PCIe reset is pending
Quoted 11 lines · Keith Busch
The disable reclaims all commands, including the ones it dispatches, so it sounds like you're talking about a race between the ones it dispatched and its timeout work. If so, we can just make sure commands sent during
nvme_dev_disablenever timeout, which are just the delete queue commands:---
drivers/nvme/host/pci.c +0
-0
drivers/nvme/host/pci.c static int nvme_delete_queue(struct nvme_queue *nvmeq, u8 opcode)
Quoted 9 lines
drivers/nvme/host/pci.c +1
-2@@ -2210,7 +2210,7 @@ static int nvme_delete_queue(struct nvme_queue *nvmeq, u8 opcode)if (IS_ERR (req))return PTR_ERR (req); - req->timeout =ADMIN_TIMEOUT ; + req->timeout =UINT_MAX ; req->end_io_data = nvmeq;init_completion (&nvmeq->delete_done); --
I think we should do the above anyway, but it isn't going to help if commands dispatched outside disabling timeout. This should fix that. Note, we never needed to have a sync'ed reset on reset_done(), but this makes it necessary.
---
drivers/nvme/host/pci.c +4 -2
@@ -1257,13 +1257,14 @@ static enum blk_eh_timer_return nvme_timeout(struct request *req, bool reserved)struct nvme_dev *dev = nvmeq->dev;struct request *abort_req;struct nvme_command cmd; +struct pci_dev *pdev =to_pci_dev (dev->dev);u32 csts =readl (dev->bar +NVME_REG_CSTS );/* If PCI error recovery process is happening, we cannot reset or * the recovery mechanism will surely fail. */ mb (); -if (pci_channel_offline (to_pci_dev (dev->dev))) +if (pci_channel_offline (pdev) || pdev->block_cfg_access)return BLK_EH_RESET_TIMER ;/*
@@ -2782,12 +2783,13 @@ static void nvme_reset_prepare(struct pci_dev *pdev) {struct nvme_dev *dev =pci_get_drvdata (pdev);nvme_dev_disable (dev, false); +nvme_sync_queues (&dev->ctrl); }static void nvme_reset_done (struct pci_dev *pdev) {struct nvme_dev *dev =pci_get_drvdata (pdev); -nvme_reset_ctrl_sync (&dev->ctrl); +nvme_reset_ctrl (&dev->ctrl); }static void nvme_shutdown (struct pci_dev *pdev)
--
RFC PATCH 1/1 nvme: Ignore timeouts while a PCIe reset is pending
Quoted 37 lines · Keith Busch
Quoted 11 lines · Keith Busch
The disable reclaims all commands, including the ones it dispatches, so it sounds like you're talking about a race between the ones it dispatched and its timeout work. If so, we can just make sure commands sent during
nvme_dev_disablenever timeout, which are just the delete queue commands:---
drivers/nvme/host/pci.c +0
-0drivers/nvme/host/pci.c static int nvme_delete_queue(struct nvme_queue *nvmeq, u8 opcode)
Quoted 9 lines
drivers/nvme/host/pci.c +1
-2@@ -2210,7 +2210,7 @@ static int nvme_delete_queue(struct nvme_queue *nvmeq, u8 opcode)if (IS_ERR (req))return PTR_ERR (req); - req->timeout =ADMIN_TIMEOUT ; + req->timeout =UINT_MAX ; req->end_io_data = nvmeq;init_completion (&nvmeq->delete_done); --I think we should do the above anyway, but it isn't going to help if commands dispatched outside disabling timeout. This should fix that. Note, we never needed to have a sync'ed reset on
reset_done(), but this makes it necessary.---
drivers/nvme/host/pci.c +1
-0@@ -1257,13 +1257,14 @@ static enum blk_eh_timer_return nvme_timeout(struct request *req, bool reserved)struct nvme_dev *dev = nvmeq->dev;struct request *abort_req;struct nvme_command cmd; +struct pci_dev *pdev =to_pci_dev (dev->dev);
drivers/nvme/host/pci.c static enum blk_eh_timer_return nvme_timeout(struct request *req, bool reserved) · line 1269
Quoted 10 lines
drivers/nvme/host/pci.c +1
-1u32 csts =readl (dev->bar +NVME_REG_CSTS );/* If PCI error recovery process is happening, we cannot reset or * the recovery mechanism will surely fail. */ mb (); -if (pci_channel_offline (to_pci_dev (dev->dev))) +if (pci_channel_offline (pdev) || pdev->block_cfg_access)return BLK_EH_RESET_TIMER ;
Thanks Keith. The block_cfg_access is exactly what I was looking for.
The use case that I have is NVMe format which can run a long time. If the FLR occurs while the format command is pending, it will be held off while the nvme_dev_disable() waits for the queues to quiesce (which doesn't happen until the command completes or times out).
Quoted 8 lines
drivers/nvme/host/pci.c +1
-0/* @@ -2782,12 +2783,13 @@ static void nvme_reset_prepare(struct pci_dev *pdev) {struct nvme_dev *dev =pci_get_drvdata (pdev);nvme_dev_disable (dev, false); +nvme_sync_queues (&dev->ctrl); }
drivers/nvme/host/pci.c static void nvme_reset_prepare(struct pci_dev *pdev)
Quoted 10 lines
drivers/nvme/host/pci.c +1
-2static void nvme_reset_done (struct pci_dev *pdev) {struct nvme_dev *dev =pci_get_drvdata (pdev); -nvme_reset_ctrl_sync (&dev->ctrl); +nvme_reset_ctrl (&dev->ctrl); }static void nvme_shutdown (struct pci_dev *pdev) --
For my specific case, is the sync_queues and reset_ctrl change needed as well?
Keith Busch
maintainerRFC PATCH 1/1 nvme: Ignore timeouts while a PCIe reset is pending
Quoted 11 lines · Kenneth
Quoted 10 lines · Keith Busch
drivers/nvme/host/pci.c +1
-1@@ -2782,12 +2783,13 @@ static void nvme_reset_prepare(struct pci_dev *pdev) {struct nvme_dev *dev =pci_get_drvdata (pdev);nvme_dev_disable (dev, false); +nvme_sync_queues (&dev->ctrl); }static void nvme_reset_done (struct pci_dev *pdev) {struct nvme_dev *dev =pci_get_drvdata (pdev); -nvme_reset_ctrl_sync (&dev->ctrl);
Quoted 8 lines
drivers/nvme/host/pci.c static void nvme_reset_prepare(struct pci_dev *pdev)
Quoted 5 lines
drivers/nvme/host/pci.c +1
-1+nvme_reset_ctrl (&dev->ctrl); }static void nvme_shutdown (struct pci_dev *pdev) --For my specific case, is the
sync_queuesandreset_ctrlchange needed as well?
I shouldn't have included the sync_queue part.
Definitely need the nvme_reset_ctrl change as blk_cfg_access is still set here so need to reset asynchronously to unblock new timeouts.
Sagi Grimberg
maintainerRFC PATCH 1/1 nvme: Ignore timeouts while a PCIe reset is pending
In reply to Keith BuschKeith,
Quoted 4 lines
I think we should do the above anyway, but it isn't going to help if commands dispatched outside disabling timeout. This should fix that. Note, we never needed to have a sync'ed reset on
reset_done(), but this makes it necessary.
With async reset on reset_done() what guarantees that nvme_dev_disable does not run concurrently with another context of nvme_reset_work? both mangle with the same queues assuming that they are not running concurrently.
quick archive browse got me to:
http://lists.infradead.org/pipermail/linux-nvme/2017-December/014599.html
discussion on my patch, but I think that it was a side effect from ming's tests..
Keith Busch
maintainerRFC PATCH 1/1 nvme: Ignore timeouts while a PCIe reset is pending
Quoted 12 lines · Sagi Grimberg
Keith,
Quoted 4 lines
I think we should do the above anyway, but it isn't going to help if commands dispatched outside disabling timeout. This should fix that. Note, we never needed to have a sync'ed reset on
reset_done(), but this makes it necessary.With async reset on
reset_done()what guarantees thatnvme_dev_disabledoes not run concurrently with another context ofnvme_reset_work? both mangle with the same queues assuming that they are not running concurrently.
Quoted 5 lines
quick archive browse got me to:
http://lists.infradead.org/pipermail/linux-nvme/2017-December/014599.htmldiscussion on my patch, but I think that it was a side effect from ming's tests..
Oh, you're right. I think Ming must have been writing to the pc reset repeatedly, in which case this proposal will have a problem coordinating.