Skip to contentSkip to conversation

New: pciehp: nvme not visible after re-insert to tbt port]

[email protected]: Bjorn Helgaas 3 participants 4 messages Mar 27, 2023

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

Forwarding to NVMe folks, lists for visibility.

----- Forwarded message from [email protected] -----

https://bugzilla.kernel.org/show_bug.cgi?id=217251
...

Created attachment 304031
--> https://bugzilla.kernel.org/attachment.cgi?id=304031&action=edit
the tracing of nvme_pci_enable() during re-insertion

Hi,

There is a JHL7540-based device that may host a NVMe device. After the first insertion a nvme drive is properly discovered and handled by the relevant modules. Once disconnected any further attempts are not successful. The device is visible on a PCI bus, but nvme_pci_enable() ends up calling pci_disable_device() every time; the runtime PM status of the device is "suspended", the power status of the 04:01.0 PCI bridge is D3. Preventing the device from being power managed ("on" -> /sys/devices/../power/control) combined with device removal and pci rescan changes nothing. A host reboot restores the initial state.

I would appreciate any suggestions how to debug it further.

maintainer
#1 lore raw
Quoted 23 lines · Bjorn Helgaas

Forwarding to NVMe folks, lists for visibility.

----- Forwarded message from [email protected] -----

https://bugzilla.kernel.org/show_bug.cgi?id=217251
...

Created attachment 304031
--> https://bugzilla.kernel.org/attachment.cgi?id=304031&action=edit
the tracing of nvme_pci_enable() during re-insertion

Hi,

There is a JHL7540-based device that may host a NVMe device. After the first insertion a nvme drive is properly discovered and handled by the relevant modules. Once disconnected any further attempts are not successful. The device is visible on a PCI bus, but nvme_pci_enable() ends up calling pci_disable_device() every time; the runtime PM status of the device is "suspended", the power status of the 04:01.0 PCI bridge is D3. Preventing the device from being power managed ("on" -> /sys/devices/../power/control) combined with device removal and pci rescan changes nothing. A host reboot restores the initial state.

Quoted 1 line

I would appreciate any suggestions how to debug it further.

Sounds the same as this report:

http://lists.infradead.org/pipermail/linux-nvme/2023-March/038259.html

The driver is bailing on the device because we can't read it's status register out of the remapped BAR. There's nothing we can do about that from the nvme driver level. Memory mapped IO has to work in order to proceed.

#2 lore raw
Quoted 26 lines · Keith Busch
Quoted 23 lines · Bjorn Helgaas

Forwarding to NVMe folks, lists for visibility.

----- Forwarded message from [email protected] -----

https://bugzilla.kernel.org/show_bug.cgi?id=217251
...

Created attachment 304031
--> https://bugzilla.kernel.org/attachment.cgi?id=304031&action=edit
the tracing of nvme_pci_enable() during re-insertion

Hi,

There is a JHL7540-based device that may host a NVMe device. After the first insertion a nvme drive is properly discovered and handled by the relevant modules. Once disconnected any further attempts are not successful. The device is visible on a PCI bus, but nvme_pci_enable() ends up calling pci_disable_device() every time; the runtime PM status of the device is "suspended", the power status of the 04:01.0 PCI bridge is D3. Preventing the device from being power managed ("on" -> /sys/devices/../power/control) combined with device removal and pci rescan changes nothing. A host reboot restores the initial state.

Quoted 1 line

I would appreciate any suggestions how to debug it further.

Quoted 8 lines

Sounds the same as this report:

http://lists.infradead.org/pipermail/linux-nvme/2023-March/038259.html

The driver is bailing on the device because we can't read it's status register out of the remapped BAR. There's nothing we can do about that from the nvme driver level. Memory mapped IO has to work in order to proceed.

Thanks. I can confirm it is the same problem:

a) the platform is Intel Alderlake
b) readl(dev->bar + NVME_REG_CSTS) in nvme_pci_enable() fails
c) reading BAR0 via setpci gives 0x00000004

--
at

maintainer
#3 lore raw
Quoted 37 lines · Aleksander Trofimowicz
Quoted 26 lines · Keith Busch
Quoted 23 lines · Bjorn Helgaas

Forwarding to NVMe folks, lists for visibility.

----- Forwarded message from [email protected] -----

https://bugzilla.kernel.org/show_bug.cgi?id=217251
...

Created attachment 304031
--> https://bugzilla.kernel.org/attachment.cgi?id=304031&action=edit
the tracing of nvme_pci_enable() during re-insertion

Hi,

There is a JHL7540-based device that may host a NVMe device. After the first insertion a nvme drive is properly discovered and handled by the relevant modules. Once disconnected any further attempts are not successful. The device is visible on a PCI bus, but nvme_pci_enable() ends up calling pci_disable_device() every time; the runtime PM status of the device is "suspended", the power status of the 04:01.0 PCI bridge is D3. Preventing the device from being power managed ("on" -> /sys/devices/../power/control) combined with device removal and pci rescan changes nothing. A host reboot restores the initial state.

Quoted 1 line

I would appreciate any suggestions how to debug it further.

Quoted 8 lines

Sounds the same as this report:

http://lists.infradead.org/pipermail/linux-nvme/2023-March/038259.html

The driver is bailing on the device because we can't read it's status register out of the remapped BAR. There's nothing we can do about that from the nvme driver level. Memory mapped IO has to work in order to proceed.

Quoted 5 lines

Thanks. I can confirm it is the same problem:

a) the platform is Intel Alderlake
b) readl(dev->bar + NVME_REG_CSTS) in nvme_pci_enable() fails
c) reading BAR0 via setpci gives 0x00000004

It's strange too. In your example, kernel says:

0000:05:00.0: BAR 0: assigned [mem 0x54000000-0x54003fff 64bit]

There is a check right after that message that ensures the kernel reads back what it wrote. No failures reported means the device really did have the expected BAR value at one point.