nvme-pci: let platform handle subsystem reset fallout
PATCH 1/1 nvme-pci: let platform handle subsystem reset fallout
From: Keith Busch <[email protected]>
Scheduling reset_work after a nvme subsystem reset is expected to fail, but this also prevents potential handling the platform may provide from successfully recovering the link without re-enumeration. Provide a pci specific operation that safely initiates a subsystem reset, and instead of scheduling reset work, read back the status register to trigger a pcie read error.
--- drivers/nvme/host/nvme.h | 3 +++ drivers/nvme/host/pci.c | 32 ++++++++++++++++++++++++++++++++ 2 files changed, 35 insertions(+)
drivers/nvme/host/nvme.h +3 -0
@@ -551,6 +551,7 @@ struct nvme_ctrl_ops {int (*reg_read64)(struct nvme_ctrl *ctrl,u32 off,u64 *val);void (*free_ctrl)(struct nvme_ctrl *ctrl);void (*submit_async_event)(struct nvme_ctrl *ctrl); +int (*subsystem_reset)(struct nvme_ctrl *ctrl);void (*delete_ctrl)(struct nvme_ctrl *ctrl);void (*stop_ctrl)(struct nvme_ctrl *ctrl);int (*get_address)(struct nvme_ctrl *ctrl,char *buf,int size);
@@ -653,6 +654,8 @@ static inline int nvme_reset_subsystem(struct nvme_ctrl *ctrl)if (!ctrl->subsystem)return -ENOTTY ; +if (ctrl->ops->subsystem_reset) +return ctrl->ops->subsystem_reset (ctrl);if (!nvme_wait_reset (ctrl))return -EBUSY ;
drivers/nvme/host/pci.c +32 -0
@@ -1143,6 +1143,37 @@ static void nvme_pci_submit_async_event(struct nvme_ctrl *ctrl)spin_unlock (&nvmeq->sq_lock); } +static int nvme_pci_subsystem_reset (struct nvme_ctrl *ctrl) +{ +struct nvme_dev *dev =to_nvme_dev (ctrl); +int ret =0 ; + +/* +* Taking the shutdown_lock ensures the iomap is not being altered by +* reset_work. Holding this lock before the RESETTING state change, if +* successful, also ensures nvme_remove won't be able to proceed to +* iounmap until we're done. +*/ +mutex_lock (&dev->shutdown_lock); +if (!nvme_change_ctrl_state (ctrl,NVME_CTRL_RESETTING )) { + ret = -EBUSY ; + goto unlock; + } + +if (!dev->bar_mapped_size) { + ret = -ENODEV ; + goto unlock; + } + +writel (0x4E564D65 , dev->bar +NVME_REG_NSSR ); + +/* Read back to trigger platform error handling, if any */ +readl (dev->bar +NVME_REG_CSTS ); +unlock: +mutex_unlock (&dev->shutdown_lock); +return ret; +} +static int adapter_delete_queue (struct nvme_dev *dev,u8 opcode,u16 id) {struct nvme_command c = { };
@@ -2859,6 +2890,7 @@ static const struct nvme_ctrl_ops nvme_pci_ctrl_ops = { .reg_read64 = nvme_pci_reg_read64, .free_ctrl = nvme_pci_free_ctrl, .submit_async_event = nvme_pci_submit_async_event, + .subsystem_reset = nvme_pci_subsystem_reset, .get_address = nvme_pci_get_address, .print_device_info = nvme_pci_print_device_info, .supports_pci_p2pdma = nvme_pci_supports_pci_p2pdma,
Signature · 1 line
2.43.0
Christoph Hellwig
maintainerQuoted 8 lines · Keith Busch
From: Keith Busch <[email protected]>
Scheduling
reset_workafter a nvme subsystem reset is expected to fail, but this also prevents potential handling the platform may provide from successfully recovering the link without re-enumeration. Provide a pci specific operation that safely initiates a subsystem reset, and instead of scheduling reset work, read back the status register to trigger a pcie read error.
What does platform mean here?
drivers/nvme/host/nvme.h static inline int nvme_reset_subsystem(struct nvme_ctrl *ctrl) · line 660
Quoted 8 lines
drivers/nvme/host/nvme.h +2
-0@@ -653,6 +654,8 @@ static inline int nvme_reset_subsystem(struct nvme_ctrl *ctrl)if (!ctrl->subsystem)return -ENOTTY ; +if (ctrl->ops->subsystem_reset) +return ctrl->ops->subsystem_reset (ctrl);if (!nvme_wait_reset (ctrl))return -EBUSY ;
Branching out into a method but having the default inline here without any comment explaining it feels weird.
If apple nvme devices supported subsystems resets, the PCIe version would probably the right thing to do for them as well, but my guess is they don't anyway. So maybe return -ENOTTY if no method is weird up, and turn the old generic one into nvmf_subsystem_reset in fabrics.c?
drivers/nvme/host/pci.c static void nvme_pci_submit_async_event(struct nvme_ctrl *ctrl) · line 1152
Quoted 2 lines
drivers/nvme/host/pci.c +2
-0+/* +* Taking the shutdown_lock ensures the iomap is not being altered by
s/iomap/BAR mapping/ ?
drivers/nvme/host/pci.c static void nvme_pci_submit_async_event(struct nvme_ctrl *ctrl) · line 1168
Quoted 1 line
drivers/nvme/host/pci.c +1
-0+writel (0x4E564D65 , dev->bar +NVME_REG_NSSR );
And now that we're duplicating this constant it could really use a symbolic name.
drivers/nvme/host/pci.c static void nvme_pci_submit_async_event(struct nvme_ctrl *ctrl) · line 1171
Quoted 2 lines
drivers/nvme/host/pci.c +2
-0+/* Read back to trigger platform error handling, if any */ +readl (dev->bar +NVME_REG_CSTS );
.. also to flush the posted write above.
Keith Busch
maintainerQuoted 11 lines · Christoph Hellwig
Quoted 8 lines · Keith Busch
From: Keith Busch <[email protected]>
Scheduling
reset_workafter a nvme subsystem reset is expected to fail, but this also prevents potential handling the platform may provide from successfully recovering the link without re-enumeration. Provide a pci specific operation that safely initiates a subsystem reset, and instead of scheduling reset work, read back the status register to trigger a pcie read error.What does platform mean here?
I'm thinking of pcie port serices: AER, DPC, or hotplug. Nilay's power platfrom has their own proprietary mechanism called EEH.
If you're platform has none of these, you'll have to wait for an IO timeout for the driver to try to do anything to recover. Before this patch, the driver would just quickly unbind.