Skip to contentSkip to conversation

nvme: Fix problem when booting from NVMe drive was leading to a hang.

PATCH In review Michael Kropaczek 2 participants 3 messages Mar 11, 2024 – Mar 12, 2024 NVMe

lore
Share & tools
lore
#0 lore raw

v3 cover nvme: Fix problem when booting from NVMe drive was leading to a hang.

Description:

During endurance test, when a system was rebooted from NVMe drive, boot process hung occasionally. The number of reboot cycles was set to 1000, with interval of 120s. Hang occurred after ~300 reboot cycles. After investigating the cause, it was established that NVMe driver did not disable host memory during shutdown leaving NVMe controller in a state preventing proper initialization in BIOS pre-boot stage. Adding of the call to nvme_set_host_mem(dev, 0), when in shutdown, fixed the issue.
In this version redundant comments were removed.

Michael Kropaczek (1):
nvme: Fix problem when booting from NVMe drive was leading to a hang.

 drivers/nvme/host/pci.c | 3 +++
 1 file changed, 3 insertions(+)

base-commit: 8d30528a170905ede9ab6ab81f229e441808590b

Signature · 23 lines
2.34.1

From 054a4e846d967b0b37d0f398166c1f56ef536a72 Mon Sep 17 00:00:00 2001
From: Michael Kropaczek <[email protected]>
Date: Thu, 7 Mar 2024 14:51:30 -0800
Subject: [PATCH v3 1/1] nvme: Fix problem when booting from NVMe drive was
 leading to a hang.
To: [email protected]
Cc: Keith Busch <[email protected]>,
    Jens Axboe <[email protected]>,
    Christoph Hellwig <[email protected]>,
    Sagi Grimberg <[email protected]>,
    Michael Kropaczek <[email protected]>

On certain host architectures/HW, DRAM was keeping memory contents over reboot
cycles. Certain NVMe controllers were accessing host memory after startup which
led to undefined state, preventing proper initialization in BIOS boot stage.
Freeing host memory during host's shutdown prevents the problem from occurring.

Signed-off-by: Michael Kropaczek <[email protected]>
---
 drivers/nvme/host/pci.c | 3 +++
 1 file changed, 3 insertions(+)
drivers/nvme/host/pci.c +3 -0
index e6267a6aa380..bda7cf5ff674 100644
@@ -2593,6 +2593,9 @@ static void nvme_dev_disable(struct nvme_dev *dev, bool shutdown)
 			nvme_wait_freeze_timeout(&dev->ctrl, NVME_IO_TIMEOUT);
 	}
 
+	if (shutdown && dev->hmb)
+		nvme_set_host_mem(dev, 0);
+
 	nvme_quiesce_io_queues(&dev->ctrl);
 
 	if (!dead && dev->ctrl.queue_count > 0) {
Signature · 1 line
2.34.1
maintainer
#1 lore raw

drivers/nvme/host/pci.c static void nvme_dev_disable(struct nvme_dev *dev, bool shutdown) · line 2598

Quoted 11 lines · Michael Kropaczek
drivers/nvme/host/pci.c +3 -0
index e6267a6aa380..bda7cf5ff674 100644
@@ -2593,6 +2593,9 @@ static void nvme_dev_disable(struct nvme_dev *dev, bool shutdown)
 			nvme_wait_freeze_timeout(&dev->ctrl, NVME_IO_TIMEOUT);
 	}
 
+	if (shutdown && dev->hmb)
+		nvme_set_host_mem(dev, 0);
+

As I said last time, this deadlocks if the controller doesn't respond to the command. While that shouldn't happen with a properly functioning controller, we can't count on that to prevent a deadlock.

Quoted 4 lines
drivers/nvme/host/pci.c +0 -0
 	nvme_quiesce_io_queues(&dev->ctrl);
 
 	if (!dead && dev->ctrl.queue_count > 0) {
#2 lore raw

Thank you, Keith,

Understood,

Michael

-----Original Message-----
From: Keith Busch <[email protected]>
Sent: Tuesday, March 12, 2024 7:39 AM
To: Michael Kropaczek <[email protected]>
Cc: [email protected]; Jens Axboe <[email protected]>; Christoph Hellwig <[email protected]>; Sagi Grimberg <[email protected]>
Subject: Re: [PATCH v3 0/1] nvme: Fix problem when booting from NVMe drive was leading to a hang.

Caution: External Email

drivers/nvme/host/pci.c static void nvme_dev_disable(struct nvme_dev *dev, bool shutdown) · line 2598

Quoted 11 lines · Michael Kropaczek
drivers/nvme/host/pci.c index renamed from drivers/nvme/host/pci.c +0 -0

e6267a6aa380..bda7cf5ff674 100644

drivers/nvme/host/pci.c +3 -0
@@ -2593,6 +2593,9 @@ static void nvme_dev_disable(struct nvme_dev *dev, bool shutdown)
                      nvme_wait_freeze_timeout(&dev->ctrl, NVME_IO_TIMEOUT);
      }
 
+     if (shutdown && dev->hmb)
+             nvme_set_host_mem(dev, 0);
+

As I said last time, this deadlocks if the controller doesn't respond to the command. While that shouldn't happen with a properly functioning controller, we can't count on that to prevent a deadlock.

Quoted 4 lines
drivers/nvme/host/pci.c +0 -1
      nvme_quiesce_io_queues(&dev->ctrl);
 
      if (!dead && dev->ctrl.queue_count > 0) {
--