Skip to contentSkip to conversation

nvme: Ignore timeouts while a PCIe reset is pending

RFC PATCH New Kenneth Heitke 3 participants 7 messages May 22, 2019 – May 24, 2019 NVMe

lore
Share & tools
lore
#0 lore raw

RFC PATCH 1/1 nvme: Ignore timeouts while a PCIe reset is pending

If an admin command timeout occurs while a PCIe reset (FLR) is pending, the CSTS bits may not be valid which could result in the controller being removed.

[372337.996566] nvme nvme0: I/O 0 QID 0 timeout, reset controller
[372339.984662] nvme 0000:1c:00.0: enabling device (0000 -> 0002)
[372339.984951] nvme nvme0: Removing after probe failure status: -19
Signed-off-by Kenneth Heitke <kenneth.heitke at intel.com>
---
 drivers/nvme/host/pci.c | 13 +++++++++++++
 1 file changed, 13 insertions(+)
drivers/nvme/host/pci.c +13 -0
index 2a8708c9ac18..aa9ea64a8b53 100644
@@ -118,6 +118,8 @@ struct nvme_dev {
 	struct nvme_ctrl ctrl;
 
 	mempool_t *iod_mempool;
+	unsigned long flags;
+#define NVME_CTRL_PCI_RESET_PENDING	0
 
 	/* shadow doorbell buffer support: */
 	u32 *dbbuf_dbs;
@@ -1250,6 +1252,11 @@ static void nvme_warn_reset(struct nvme_dev *dev, u32 csts)
 			 csts, result);
 }
 
+static bool nvme_pci_reset_pending(const struct nvme_dev *dev)
+{
+	return !!test_bit(NVME_CTRL_PCI_RESET_PENDING, &dev->flags);
+}
+
 static enum blk_eh_timer_return nvme_timeout(struct request *req, bool reserved)
 {
 	struct nvme_iod *iod = blk_mq_rq_to_pdu(req);
@@ -1267,6 +1274,10 @@ static enum blk_eh_timer_return nvme_timeout(struct request *req, bool reserved)
 	if (pci_channel_offline(to_pci_dev(dev->dev)))
 		return BLK_EH_RESET_TIMER;
 
+	/* If a PCIe reset (FLR) is pending, wait for it to complete */
+	if (nvme_pci_reset_pending(dev))
+		return BLK_EH_RESET_TIMER;
+
 	/*
 	 * Reset immediately if the controller is failed
 	 */
@@ -2780,12 +2791,14 @@ static int nvme_probe(struct pci_dev *pdev, const struct pci_device_id *id)
 static void nvme_reset_prepare(struct pci_dev *pdev)
 {
 	struct nvme_dev *dev = pci_get_drvdata(pdev);
+	set_bit(NVME_CTRL_PCI_RESET_PENDING, &dev->flags);
 	nvme_dev_disable(dev, false);
 }
 
 static void nvme_reset_done(struct pci_dev *pdev)
 {
 	struct nvme_dev *dev = pci_get_drvdata(pdev);
+	clear_bit(NVME_CTRL_PCI_RESET_PENDING, &dev->flags);
 	nvme_reset_ctrl_sync(&dev->ctrl);
 }
 
Signature · 1 line
2.17.1
maintainer
#1 lore raw

RFC PATCH 1/1 nvme: Ignore timeouts while a PCIe reset is pending

Quoted 7 lines · Kenneth Heitke

If an admin command timeout occurs while a PCIe reset (FLR) is pending, the CSTS bits may not be valid which could result in the controller being removed.

[372337.996566] nvme nvme0: I/O 0 QID 0 timeout, reset controller
[372339.984662] nvme 0000:1c:00.0: enabling device (0000 -> 0002)
[372339.984951] nvme nvme0: Removing after probe failure status: -19

The disable reclaims all commands, including the ones it dispatches, so it sounds like you're talking about a race between the ones it dispatched and its timeout work. If so, we can just make sure commands sent during nvme_dev_disable never timeout, which are just the delete queue commands:

---

drivers/nvme/host/pci.c +1 -1
index f562154551ce..4678704c2138 100644
@@ -2210,7 +2210,7 @@ static int nvme_delete_queue(struct nvme_queue *nvmeq, u8 opcode)
 	if (IS_ERR(req))
 		return PTR_ERR(req);
 
-	req->timeout = ADMIN_TIMEOUT;
+	req->timeout = UINT_MAX;
 	req->end_io_data = nvmeq;
 
 	init_completion(&nvmeq->delete_done);

--

maintainer
#2 lore raw

RFC PATCH 1/1 nvme: Ignore timeouts while a PCIe reset is pending

Quoted 11 lines · Keith Busch

The disable reclaims all commands, including the ones it dispatches, so it sounds like you're talking about a race between the ones it dispatched and its timeout work. If so, we can just make sure commands sent during nvme_dev_disable never timeout, which are just the delete queue commands:

---

drivers/nvme/host/pci.c +0 -0
index f562154551ce..4678704c2138 100644

drivers/nvme/host/pci.c static int nvme_delete_queue(struct nvme_queue *nvmeq, u8 opcode)

Quoted 9 lines
drivers/nvme/host/pci.c +1 -2
@@ -2210,7 +2210,7 @@ static int nvme_delete_queue(struct nvme_queue *nvmeq, u8 opcode)
 	if (IS_ERR(req))
 		return PTR_ERR(req);
 
-	req->timeout = ADMIN_TIMEOUT;
+	req->timeout = UINT_MAX;
 	req->end_io_data = nvmeq;
 
 	init_completion(&nvmeq->delete_done);
--

I think we should do the above anyway, but it isn't going to help if commands dispatched outside disabling timeout. This should fix that. Note, we never needed to have a sync'ed reset on reset_done(), but this makes it necessary.

---

drivers/nvme/host/pci.c +4 -2
index f562154551ce..3edb9d098eb8 100644
@@ -1257,13 +1257,14 @@ static enum blk_eh_timer_return nvme_timeout(struct request *req, bool reserved)
 	struct nvme_dev *dev = nvmeq->dev;
 	struct request *abort_req;
 	struct nvme_command cmd;
+	struct pci_dev *pdev = to_pci_dev(dev->dev);
 	u32 csts = readl(dev->bar + NVME_REG_CSTS);
 
 	/* If PCI error recovery process is happening, we cannot reset or
 	 * the recovery mechanism will surely fail.
 	 */
 	mb();
-	if (pci_channel_offline(to_pci_dev(dev->dev)))
+	if (pci_channel_offline(pdev) || pdev->block_cfg_access)
 		return BLK_EH_RESET_TIMER;
 
 	/*
@@ -2782,12 +2783,13 @@ static void nvme_reset_prepare(struct pci_dev *pdev)
 {
 	struct nvme_dev *dev = pci_get_drvdata(pdev);
 	nvme_dev_disable(dev, false);
+	nvme_sync_queues(&dev->ctrl);
 }
 
 static void nvme_reset_done(struct pci_dev *pdev)
 {
 	struct nvme_dev *dev = pci_get_drvdata(pdev);
-	nvme_reset_ctrl_sync(&dev->ctrl);
+	nvme_reset_ctrl(&dev->ctrl);
 }
 
 static void nvme_shutdown(struct pci_dev *pdev)

--

#3 lore raw

RFC PATCH 1/1 nvme: Ignore timeouts while a PCIe reset is pending

Quoted 37 lines · Keith Busch
Quoted 11 lines · Keith Busch

The disable reclaims all commands, including the ones it dispatches, so it sounds like you're talking about a race between the ones it dispatched and its timeout work. If so, we can just make sure commands sent during nvme_dev_disable never timeout, which are just the delete queue commands:

---

drivers/nvme/host/pci.c +0 -0
index f562154551ce..4678704c2138 100644

drivers/nvme/host/pci.c static int nvme_delete_queue(struct nvme_queue *nvmeq, u8 opcode)

Quoted 9 lines
drivers/nvme/host/pci.c +1 -2
@@ -2210,7 +2210,7 @@ static int nvme_delete_queue(struct nvme_queue *nvmeq, u8 opcode)
  	if (IS_ERR(req))
  		return PTR_ERR(req);
 
-	req->timeout = ADMIN_TIMEOUT;
+	req->timeout = UINT_MAX;
  	req->end_io_data = nvmeq;
 
  	init_completion(&nvmeq->delete_done);
--

I think we should do the above anyway, but it isn't going to help if commands dispatched outside disabling timeout. This should fix that. Note, we never needed to have a sync'ed reset on reset_done(), but this makes it necessary.

---

drivers/nvme/host/pci.c +1 -0
index f562154551ce..3edb9d098eb8 100644
@@ -1257,13 +1257,14 @@ static enum blk_eh_timer_return nvme_timeout(struct request *req, bool reserved)
  	struct nvme_dev *dev = nvmeq->dev;
  	struct request *abort_req;
  	struct nvme_command cmd;
+	struct pci_dev *pdev = to_pci_dev(dev->dev);

drivers/nvme/host/pci.c static enum blk_eh_timer_return nvme_timeout(struct request *req, bool reserved) · line 1269

Quoted 10 lines
drivers/nvme/host/pci.c +1 -1
  	u32 csts = readl(dev->bar + NVME_REG_CSTS);
 
  	/* If PCI error recovery process is happening, we cannot reset or
  	 * the recovery mechanism will surely fail.
  	 */
  	mb();
-	if (pci_channel_offline(to_pci_dev(dev->dev)))
+	if (pci_channel_offline(pdev) || pdev->block_cfg_access)
  		return BLK_EH_RESET_TIMER;
 

Thanks Keith. The block_cfg_access is exactly what I was looking for.

The use case that I have is NVMe format which can run a long time. If the FLR occurs while the format command is pending, it will be held off while the nvme_dev_disable() waits for the queues to quiesce (which doesn't happen until the command completes or times out).

Quoted 8 lines
drivers/nvme/host/pci.c +1 -0
  	/*
@@ -2782,12 +2783,13 @@ static void nvme_reset_prepare(struct pci_dev *pdev)
  {
  	struct nvme_dev *dev = pci_get_drvdata(pdev);
  	nvme_dev_disable(dev, false);
+	nvme_sync_queues(&dev->ctrl);
  }

drivers/nvme/host/pci.c static void nvme_reset_prepare(struct pci_dev *pdev)

Quoted 10 lines
drivers/nvme/host/pci.c +1 -2
 
  static void nvme_reset_done(struct pci_dev *pdev)
  {
  	struct nvme_dev *dev = pci_get_drvdata(pdev);
-	nvme_reset_ctrl_sync(&dev->ctrl);
+	nvme_reset_ctrl(&dev->ctrl);
  }
 
  static void nvme_shutdown(struct pci_dev *pdev)
--

For my specific case, is the sync_queues and reset_ctrl change needed as well?

maintainer
#4 lore raw

RFC PATCH 1/1 nvme: Ignore timeouts while a PCIe reset is pending

Quoted 11 lines · Kenneth
Quoted 10 lines · Keith Busch
drivers/nvme/host/pci.c +1 -1
@@ -2782,12 +2783,13 @@ static void nvme_reset_prepare(struct pci_dev *pdev)
  {
  	struct nvme_dev *dev = pci_get_drvdata(pdev);
  	nvme_dev_disable(dev, false);
+	nvme_sync_queues(&dev->ctrl);
  }
  static void nvme_reset_done(struct pci_dev *pdev)
  {
  	struct nvme_dev *dev = pci_get_drvdata(pdev);
-	nvme_reset_ctrl_sync(&dev->ctrl);
Quoted 8 lines

drivers/nvme/host/pci.c static void nvme_reset_prepare(struct pci_dev *pdev)

Quoted 5 lines
drivers/nvme/host/pci.c +1 -1
+	nvme_reset_ctrl(&dev->ctrl);
  }
  static void nvme_shutdown(struct pci_dev *pdev)
--
 

For my specific case, is the sync_queues and reset_ctrl change needed as well?

I shouldn't have included the sync_queue part.

Definitely need the nvme_reset_ctrl change as blk_cfg_access is still set here so need to reset asynchronously to unblock new timeouts.

maintainer
#5 lore raw

RFC PATCH 1/1 nvme: Ignore timeouts while a PCIe reset is pending

In reply to Keith Busch

Keith,

Quoted 4 lines

I think we should do the above anyway, but it isn't going to help if commands dispatched outside disabling timeout. This should fix that. Note, we never needed to have a sync'ed reset on reset_done(), but this makes it necessary.

With async reset on reset_done() what guarantees that nvme_dev_disable does not run concurrently with another context of nvme_reset_work? both mangle with the same queues assuming that they are not running concurrently.

quick archive browse got me to:
http://lists.infradead.org/pipermail/linux-nvme/2017-December/014599.html

discussion on my patch, but I think that it was a side effect from ming's tests..

maintainer
#6 lore raw

RFC PATCH 1/1 nvme: Ignore timeouts while a PCIe reset is pending

Quoted 12 lines · Sagi Grimberg

Keith,

Quoted 4 lines

I think we should do the above anyway, but it isn't going to help if commands dispatched outside disabling timeout. This should fix that. Note, we never needed to have a sync'ed reset on reset_done(), but this makes it necessary.

With async reset on reset_done() what guarantees that nvme_dev_disable does not run concurrently with another context of nvme_reset_work? both mangle with the same queues assuming that they are not running concurrently.

Quoted 5 lines

quick archive browse got me to:
http://lists.infradead.org/pipermail/linux-nvme/2017-December/014599.html

discussion on my patch, but I think that it was a side effect from ming's tests..

Oh, you're right. I think Ming must have been writing to the pc reset repeatedly, in which case this proposal will have a problem coordinating.