nvme-pci: Harden drive presence detect in nvme_dev_disable()
PATCH 1/1 nvme-pci: Harden drive presence detect in nvme_dev_disable()
On our ZynqMP system we observe, that a NVMe drive that resets itself while doing a firmware update causes a Kernel crash like this:
[ 67.720772] pcieport 0000:02:02.0: pciehp: Slot(2): Link Down [ 67.720783] pcieport 0000:02:02.0: pciehp: Slot(2): Card not present [ 67.720795] nvme 0000:04:00.0: PME# disabled [ 67.720849] Internal error: synchronous external abort: 96000010 [#1] PREEMPT SMP [ 67.720853] nwl-pcie fd0e0000.pcie: Slave error
Analysis: When nvme_dev_disable() is called because of this PCIe hotplug event, pci_is_enabled() is still true. And accessing the NVMe drive which is currently not available as it's in reboot process causes this "synchronous external abort" on this ARM64 platform.
This patch adds the pci_device_is_present() check as well, which returns false in this "Card not present" hot-plug case. With this change, the NVMe driver does not try to access the NVMe registers any more and the FW update finishes without any problems.
--- drivers/nvme/host/pci.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/nvme/host/pci.c +1 -1
@@ -2473,7 +2473,7 @@ static void nvme_dev_disable(struct nvme_dev *dev, bool shutdown)struct pci_dev *pdev =to_pci_dev (dev->dev);mutex_lock (&dev->shutdown_lock); -if (pci_is_enabled (pdev)) { +if (pci_device_is_present (pdev) &&pci_is_enabled (pdev)) {u32 csts =readl (dev->bar +NVME_REG_CSTS ); if (dev->ctrl.state == NVME_CTRL_LIVE ||
Signature · 1 line
2.36.0
Christoph Hellwig
AppliedThanks,
applied to nvme-5.19.
Quoted 32 lines · Stefan Roese
On our ZynqMP system we observe, that a NVMe drive that resets itself while doing a firmware update causes a Kernel crash like this:
[ 67.720772] pcieport 0000:02:02.0: pciehp: Slot(2): Link Down [ 67.720783] pcieport 0000:02:02.0: pciehp: Slot(2): Card not present [ 67.720795] nvme 0000:04:00.0: PME# disabled [ 67.720849] Internal error: synchronous external abort: 96000010 [#1] PREEMPT SMP [ 67.720853] nwl-pcie fd0e0000.pcie: Slave errorAnalysis: When
nvme_dev_disable()is called because of this PCIe hotplug event,pci_is_enabled()is still true. And accessing the NVMe drive which is currently not available as it's in reboot process causes this "synchronous external abort" on this ARM64 platform.This patch adds the
pci_device_is_present()check as well, which returns false in this "Card not present" hot-plug case. With this change, the NVMe driver does not try to access the NVMe registers any more and the FW update finishes without any problems.Signed-off-by Stefan Roese <[email protected]>Cc Keith Busch <[email protected]>Cc Jens Axboe <[email protected]>Cc Christoph Hellwig <[email protected]>--- drivers/nvme/host/pci.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)drivers/nvme/host/pci.c +0
-0
drivers/nvme/host/pci.c static void nvme_dev_disable(struct nvme_dev *dev, bool shutdown) · line 2476
Quoted 5 lines
drivers/nvme/host/pci.c +1
-1@@ -2473,7 +2473,7 @@ static void nvme_dev_disable(struct nvme_dev *dev, bool shutdown)struct pci_dev *pdev =to_pci_dev (dev->dev);mutex_lock (&dev->shutdown_lock); -if (pci_is_enabled (pdev)) { +if (pci_device_is_present (pdev) &&pci_is_enabled (pdev)) {
sound weird to me that pci device can be enabled but not present.
Also why we get hot unplug event for FW upgrade ?
is this the correct behavior ?
Quoted 3 lines
drivers/nvme/host/pci.c +0
-0u32 csts =readl (dev->bar +NVME_REG_CSTS ); if (dev->ctrl.state == NVME_CTRL_LIVE ||
Keith Busch
maintainerQuoted 28 lines · Max Gurtovoy
Quoted 27 lines · Stefan Roese
On our ZynqMP system we observe, that a NVMe drive that resets itself while doing a firmware update causes a Kernel crash like this:
[ 67.720772] pcieport 0000:02:02.0: pciehp: Slot(2): Link Down [ 67.720783] pcieport 0000:02:02.0: pciehp: Slot(2): Card not present [ 67.720795] nvme 0000:04:00.0: PME# disabled [ 67.720849] Internal error: synchronous external abort: 96000010 [#1] PREEMPT SMP [ 67.720853] nwl-pcie fd0e0000.pcie: Slave errorAnalysis: When
nvme_dev_disable()is called because of this PCIe hotplug event,pci_is_enabled()is still true. And accessing the NVMe drive which is currently not available as it's in reboot process causes this "synchronous external abort" on this ARM64 platform.This patch adds the
pci_device_is_present()check as well, which returns false in this "Card not present" hot-plug case. With this change, the NVMe driver does not try to access the NVMe registers any more and the FW update finishes without any problems.Signed-off-by Stefan Roese <[email protected]>Cc Keith Busch <[email protected]>Cc Jens Axboe <[email protected]>Cc Christoph Hellwig <[email protected]>--- drivers/nvme/host/pci.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
Quoted 11 lines
drivers/nvme/host/pci.c static void nvme_dev_disable(struct nvme_dev *dev, bool shutdown) · line 2475
Quoted 9 lines
drivers/nvme/host/pci.c +1
-1@@ -2473,7 +2473,7 @@ static void nvme_dev_disable(struct nvme_dev *dev, bool shutdown)struct pci_dev *pdev =to_pci_dev (dev->dev);mutex_lock (&dev->shutdown_lock); -if (pci_is_enabled (pdev)) { +if (pci_device_is_present (pdev) &&pci_is_enabled (pdev)) {sound weird to me that pci device can be enabled but not present.
pci_is_enabled() just checks a driver reference count. It doesn't actually query the physical status of the device's link it references, but pci_device_is_present() will do that.
Quoted 1 line
Also why we get hot unplug event for FW upgrade ?
It might happen if the device requires a Subsystem Reset to activate the new firmware.
Quoted 1 line
is this the correct behavior ?
I don't know if the "synchronous external abort" is normal, or how severe of a problem it indicates, but we normally expect accessing registers on an inaccessible device to be mostly harmless.
Quoted 2 lines
Quoted 2 lines
drivers/nvme/host/pci.c +0
-0u32 csts =readl (dev->bar +NVME_REG_CSTS ); if (dev->ctrl.state == NVME_CTRL_LIVE ||
Quoted 39 lines · Keith Busch
Quoted 28 lines · Max Gurtovoy
Quoted 27 lines · Stefan Roese
On our ZynqMP system we observe, that a NVMe drive that resets itself while doing a firmware update causes a Kernel crash like this:
[ 67.720772] pcieport 0000:02:02.0: pciehp: Slot(2): Link Down [ 67.720783] pcieport 0000:02:02.0: pciehp: Slot(2): Card not present [ 67.720795] nvme 0000:04:00.0: PME# disabled [ 67.720849] Internal error: synchronous external abort: 96000010 [#1] PREEMPT SMP [ 67.720853] nwl-pcie fd0e0000.pcie: Slave errorAnalysis: When
nvme_dev_disable()is called because of this PCIe hotplug event,pci_is_enabled()is still true. And accessing the NVMe drive which is currently not available as it's in reboot process causes this "synchronous external abort" on this ARM64 platform.This patch adds the
pci_device_is_present()check as well, which returns false in this "Card not present" hot-plug case. With this change, the NVMe driver does not try to access the NVMe registers any more and the FW update finishes without any problems.Signed-off-by Stefan Roese <[email protected]>Cc Keith Busch <[email protected]>Cc Jens Axboe <[email protected]>Cc Christoph Hellwig <[email protected]>--- drivers/nvme/host/pci.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)Quoted 10 lines
drivers/nvme/host/pci.c static void nvme_dev_disable(struct nvme_dev *dev, bool shutdown) · line 2475
Quoted 9 lines
drivers/nvme/host/pci.c +1
-1@@ -2473,7 +2473,7 @@ static void nvme_dev_disable(struct nvme_dev *dev, bool shutdown)struct pci_dev *pdev =to_pci_dev (dev->dev);mutex_lock (&dev->shutdown_lock); -if (pci_is_enabled (pdev)) { +if (pci_device_is_present (pdev) &&pci_is_enabled (pdev)) {sound weird to me that pci device can be enabled but not present.
Quoted 7 lines
pci_is_enabled()just checks a driver reference count. It doesn't actually query the physical status of the device's link it references, butpci_device_is_present()will do that.Quoted 1 line
Also why we get hot unplug event for FW upgrade ?
It might happen if the device requires a Subsystem Reset to activate the new firmware.
but isn't the reset something that the admin should do ?
some power-cycle or cold-reboot or other reset.
In the reported case the device resets itself. I'm not sure it's expected.
BTW, for sure the fix is good for the hot unplug case but FW reset shouldn't cause this scenario IMO.
Quoted 8 lines
Quoted 2 lines
is this the correct behavior ?
I don't know if the "synchronous external abort" is normal, or how severe of a problem it indicates, but we normally expect accessing registers on an inaccessible device to be mostly harmless.
Quoted 2 lines
Quoted 2 lines
drivers/nvme/host/pci.c +0
-0u32 csts =readl (dev->bar +NVME_REG_CSTS ); if (dev->ctrl.state == NVME_CTRL_LIVE ||
Keith Busch
maintainerQuoted 9 lines · Max Gurtovoy
Quoted 2 lines · Keith Busch
It might happen if the device requires a Subsystem Reset to activate the new firmware.
but isn't the reset something that the admin should do ?
some power-cycle or cold-reboot or other reset.
In the reported case the device resets itself. I'm not sure it's expected.
I'm not familiar with this device or the procedure used to update the firmware, but I'm aware many vendors still provide their firmware bundled within their own tooling, so simply running that utility could do all sorts of things including a device link reset.
If the device is resetting itself without a user or driver app initiating it, though, that would be bad behavior outside the spec. But if this harmless patch improves interop regardless of device behavior, then it should be okay to include.
Quoted 2 lines
BTW, for sure the fix is good for the hot unplug case but FW reset shouldn't cause this scenario IMO.
Yeah, the patch is fine as far as I'm concerned, though it is impossible to prevent all register reads executing concurrently with any random link-down event. The commit log indicates reading registers during such an event causes a kernel crash, so a more complete fix probably needs to come from the platform level.
Quoted 7 lines · Keith Busch
Quoted 6 lines · Max Gurtovoy
Quoted 2 lines · Keith Busch
It might happen if the device requires a Subsystem Reset to activate the new firmware.
but isn't the reset something that the admin should do ?
Quoted 8 lines
Quoted 3 lines
some power-cycle or cold-reboot or other reset.
In the reported case the device resets itself. I'm not sure it's expected.
I'm not familiar with this device or the procedure used to update the firmware, but I'm aware many vendors still provide their firmware bundled within their own tooling, so simply running that utility could do all sorts of things including a device link reset.
We are using standard nvme-cli tools for the FW update process, like:
nvme fw-download ...
nvme fw-commit ...
Quoted 4 lines
If the device is resetting itself without a user or driver app initiating it, though, that would be bad behavior outside the spec. But if this harmless patch improves interop regardless of device behavior, then it should be okay to include.
Yes. The device resetting itself seems to be outside of the spec. Still we somehow need to handle this situation.
Quoted 8 lines
Quoted 2 lines
BTW, for sure the fix is good for the hot unplug case but FW reset shouldn't cause this scenario IMO.
Yeah, the patch is fine as far as I'm concerned, though it is impossible to prevent all register reads executing concurrently with any random link-down event. The commit log indicates reading registers during such an event causes a kernel crash, so a more complete fix probably needs to come from the platform level.
Agreed. Even with this patch applied, we have still seen Kernel crashes after the "nvme fw-commit", when the drive reset'ed itself twice (IIRC). We were able to work around these issues by removing the nvme modules exactly when the PCIe device was removed (via HP event) for now.
Thanks,
Stefan