ASPM powersupersave change NVMe SSD Samsung 960 PRO capacity to 0 and read-only

I have a Samsung 960 PRO NVMe SSD (Non-Volatile memory controller:
Samsung Electronics Co Ltd NVMe SSD Controller SM961/PM961). It works fine until I enable powersupersave via /sys/module/pcie_aspm/parameters/policy

ASPM is enabled in BIOS and works fine for all devices and in powersave mode. I'm able to reproduce this always at any time while the system is up and running via:

$> echo powersupersave > /sys/module/pcie_aspm/parameters/policy

The Linux kernel is 4.14.4 and APST for my device is working with powersave. As soon as I enable powersupersave I get:

[11535.142755] dpc 0000:00:10.0:pcie010: DPC containment event, status:0x1f09 source:0x0000
[11535.142760] dpc 0000:00:10.0:pcie010: DPC unmasked uncorrectable error detected, remove downstream devices
[11535.159999] nvme0n1: detected capacity change from 1024209543168 to 0

...

It looks like APST feature cannot be set anymore after enabling powersupersave. Also the PCIe device disappears completely from lspci output.

Any idea why the device is failing with powersupersave and how to avoid it? Especially how to enable it but skip certain broken devices as this is my boot device.

[+cc Rajat, Keith, linux-kernel]

Quoted 11 lines · Maik Broemme

I have a Samsung 960 PRO NVMe SSD (Non-Volatile memory controller:
Samsung Electronics Co Ltd NVMe SSD Controller SM961/PM961). It works fine until I enable powersupersave via /sys/module/pcie_aspm/parameters/policy

ASPM is enabled in BIOS and works fine for all devices and in powersave mode. I'm able to reproduce this always at any time while the system is up and running via:

$> echo powersupersave > /sys/module/pcie_aspm/parameters/policy

Quoted 7 lines

The Linux kernel is 4.14.4 and APST for my device is working with powersave. As soon as I enable powersupersave I get:

[11535.142755] dpc 0000:00:10.0:pcie010: DPC containment event, status:0x1f09 source:0x0000
[11535.142760] dpc 0000:00:10.0:pcie010: DPC unmasked uncorrectable error detected, remove downstream devices
[11535.159999] nvme0n1: detected capacity change from 1024209543168 to 0

...

Can you start by opening a bug report at https://bugzilla.kernel.org,
category Drivers/PCI, and attaching the complete "lspci -vv" output (as root) and the complete dmesg log? Make sure you have a new enough lspci to decode the ASPM L1 Substates capability and the LTR bits. Source is at git://git.kernel.org/pub/scm/utils/pciutils/pciutils.git

powersupersave enables ASPM L1 Substates. Rajat, do you have any ideas about this or how we might debug it?

Keith, is this really all the information about the event that we can get out of DPC? Is there some AER logging we might be able to get via "lspci -vv"? Sounds like this is the boot disk, so Maik may not be able to run lspci after the DPC event. If there *is* any AER info, can we connect up the DPC event so we can print the AER info from the kernel?

[+cc Rajat, Keith, linux-kernel]

Quoted 9 lines · Maik Broemme

I have a Samsung 960 PRO NVMe SSD (Non-Volatile memory controller:
Samsung Electronics Co Ltd NVMe SSD Controller SM961/PM961). It works fine until I enable powersupersave via /sys/module/pcie_aspm/parameters/policy

ASPM is enabled in BIOS and works fine for all devices and in powersave mode. I'm able to reproduce this always at any time while the system is up and running via:

Quoted 7 lines

$> echo powersupersave > /sys/module/pcie_aspm/parameters/policy

The Linux kernel is 4.14.4 and APST for my device is working with powersave. As soon as I enable powersupersave I get:

[11535.142755] dpc 0000:00:10.0:pcie010: DPC containment event,

status:0x1f09 source:0x0000

Quoted 1 line

[11535.142760] dpc 0000:00:10.0:pcie010: DPC unmasked uncorrectable

error detected, remove downstream devices

Quoted 2 lines

[11535.159999] nvme0n1: detected capacity change from 1024209543168 to 0

...

Can you start by opening a bug report at https://bugzilla.kernel.org,
category Drivers/PCI, and attaching the complete "lspci -vv" output (as root) and the complete dmesg log? Make sure you have a new enough lspci to decode the ASPM L1 Substates capability and the LTR bits. Source is at git://git.kernel.org/pub/scm/utils/pciutils/pciutils.git

powersupersave enables ASPM L1 Substates. Rajat, do you have any ideas about this or how we might debug it?

Keith, is this really all the information about the event that we can get out of DPC? Is there some AER logging we might be able to get via "lspci -vv"? Sounds like this is the boot disk, so Maik may not be able to run lspci after the DPC event. If there *is* any AER info, can we connect up the DPC event so we can print the AER info from the kernel?

I wonder if there's some way improper L1 Substate configuration could cause a DPC event. There are lots of knobs there that seem to depend on devices, and I'm not sure we have them all correct yet.

There are some recent changes in that area that are in linux-next:

PCI/ASPM: Enable Latency Tolerance Reporting when supported PCI/ASPM: Calculate LTR_L1.2_THRESHOLD from device characteristics PCI/ASPM: Use correct capability pointer to program LTR_L1.2_THRESHOLD PCI/ASPM: Account for downstream device's Port Common_Mode_Restore_Time

It's conceivable that they could have some bearing on this problem. If you could give this a whirl on linux-next, that would be interesting. If you do this, please also collect the "lspci -vv" output there so we can compare it with the v4.14 configuration.

Quoted 3 lines

It looks like APST feature cannot be set anymore after enabling powersupersave. Also the PCIe device disappears completely from lspci output.

My guess is this is to be expected after the DPC event. That basically disconnects the PCIe device from the system.

Quoted 3 lines

Any idea why the device is failing with powersupersave and how to avoid it? Especially how to enable it but skip certain broken devices as this is my boot device.

We could conceivably add a quirk if we find that L1SS is broken on this particular device. But L1SS is so new that I'd be more suspicious of the Linux code than the device.

Bjorn

[+cc Rajat, Keith, linux-kernel]

Quoted 11 lines · Maik Broemme

I have a Samsung 960 PRO NVMe SSD (Non-Volatile memory controller:
Samsung Electronics Co Ltd NVMe SSD Controller SM961/PM961). It works fine until I enable powersupersave via /sys/module/pcie_aspm/parameters/policy

ASPM is enabled in BIOS and works fine for all devices and in powersave mode. I'm able to reproduce this always at any time while the system is up and running via:

$> echo powersupersave > /sys/module/pcie_aspm/parameters/policy

Quoted 7 lines

The Linux kernel is 4.14.4 and APST for my device is working with powersave. As soon as I enable powersupersave I get:

[11535.142755] dpc 0000:00:10.0:pcie010: DPC containment event, status:0x1f09 source:0x0000
[11535.142760] dpc 0000:00:10.0:pcie010: DPC unmasked uncorrectable error detected, remove downstream devices
[11535.159999] nvme0n1: detected capacity change from 1024209543168 to 0

...

Can you start by opening a bug report at https://bugzilla.kernel.org,
category Drivers/PCI, and attaching the complete "lspci -vv" output (as root) and the complete dmesg log? Make sure you have a new enough lspci to decode the ASPM L1 Substates capability and the LTR bits. Source is at git://git.kernel.org/pub/scm/utils/pciutils/pciutils.git

powersupersave enables ASPM L1 Substates. Rajat, do you have any ideas about this or how we might debug it?

Keith, is this really all the information about the event that we can get out of DPC? Is there some AER logging we might be able to get via "lspci -vv"? Sounds like this is the boot disk, so Maik may not be able to run lspci after the DPC event. If there *is* any AER info, can we connect up the DPC event so we can print the AER info from the kernel?

I wonder if there's some way improper L1 Substate configuration could cause a DPC event. There are lots of knobs there that seem to depend on devices, and I'm not sure we have them all correct yet.

There are some recent changes in that area that are in linux-next:

PCI/ASPM: Enable Latency Tolerance Reporting when supported PCI/ASPM: Calculate LTR_L1.2_THRESHOLD from device characteristics PCI/ASPM: Use correct capability pointer to program LTR_L1.2_THRESHOLD PCI/ASPM: Account for downstream device's Port Common_Mode_Restore_Time

It's conceivable that they could have some bearing on this problem. If you could give this a whirl on linux-next, that would be interesting. If you do this, please also collect the "lspci -vv" output there so we can compare it with the v4.14 configuration.

Quoted 3 lines

It looks like APST feature cannot be set anymore after enabling powersupersave. Also the PCIe device disappears completely from lspci output.

My guess is this is to be expected after the DPC event. That basically disconnects the PCIe device from the system.

Quoted 3 lines

Any idea why the device is failing with powersupersave and how to avoid it? Especially how to enable it but skip certain broken devices as this is my boot device.

We could conceivably add a quirk if we find that L1SS is broken on this particular device. But L1SS is so new that I'd be more suspicious of the Linux code than the device.

Bjorn

Quoted 22 lines · Bjorn Helgaas

[+cc Rajat, Keith, linux-kernel]

Quoted 11 lines · Maik Broemme

I have a Samsung 960 PRO NVMe SSD (Non-Volatile memory controller:
Samsung Electronics Co Ltd NVMe SSD Controller SM961/PM961). It works fine until I enable powersupersave via /sys/module/pcie_aspm/parameters/policy

ASPM is enabled in BIOS and works fine for all devices and in powersave mode. I'm able to reproduce this always at any time while the system is up and running via:

$> echo powersupersave > /sys/module/pcie_aspm/parameters/policy

Quoted 7 lines

The Linux kernel is 4.14.4 and APST for my device is working with powersave. As soon as I enable powersupersave I get:

[11535.142755] dpc 0000:00:10.0:pcie010: DPC containment event, status:0x1f09 source:0x0000
[11535.142760] dpc 0000:00:10.0:pcie010: DPC unmasked uncorrectable error detected, remove downstream devices
[11535.159999] nvme0n1: detected capacity change from 1024209543168 to 0

...

Quoted 8 lines

Can you start by opening a bug report at https://bugzilla.kernel.org,
category Drivers/PCI, and attaching the complete "lspci -vv" output (as root) and the complete dmesg log? Make sure you have a new enough lspci to decode the ASPM L1 Substates capability and the LTR bits. Source is at git://git.kernel.org/pub/scm/utils/pciutils/pciutils.git

powersupersave enables ASPM L1 Substates. Rajat, do you have any ideas about this or how we might debug it?

I know Maik mentioned that this is the boot device. Maik, is it possible to boot off something else so that we can do some more experiments on this port? If so,
- can you try to see if the device comes back if you switch the ASPM policy back from "powersupersave" -> powersave, and potentially do a rescan (echo 1 > /sys/bus/pci/rescan)?

- It would be good to get the complete lspci -vv for the root port (assuming device is connected to root port i.e. no switch). Specifically what does the Link status show?
- Also, do you know if your root port provides any debug registers that could tell the current L1 substate of the link (My system's root port had such register).
- I had usually resorted to a PCIe analyzer to peak at the packets when I was debugging it. Not sure if that is an option here.

I don't see any debug prints in aspm.c that we could enable. Even if I provide a patch, I suspect that the problem will start at the last step of the pcie_config_aspm_l1ss() i.e. as soon as we really enable it in HW. Maik, would you be open to take a debug patch that adds some debug prints and try it out (compile your kernel with that patch)?

Quoted 35 lines

Keith, is this really all the information about the event that we can get out of DPC? Is there some AER logging we might be able to get via "lspci -vv"? Sounds like this is the boot disk, so Maik may not be able to run lspci after the DPC event. If there *is* any AER info, can we connect up the DPC event so we can print the AER info from the kernel?

I wonder if there's some way improper L1 Substate configuration could cause a DPC event. There are lots of knobs there that seem to depend on devices, and I'm not sure we have them all correct yet.

There are some recent changes in that area that are in linux-next:

PCI/ASPM: Enable Latency Tolerance Reporting when supported PCI/ASPM: Calculate LTR_L1.2_THRESHOLD from device characteristics PCI/ASPM: Use correct capability pointer to program LTR_L1.2_THRESHOLD PCI/ASPM: Account for downstream device's Port Common_Mode_Restore_Time

It's conceivable that they could have some bearing on this problem. If you could give this a whirl on linux-next, that would be interesting. If you do this, please also collect the "lspci -vv" output there so we can compare it with the v4.14 configuration.

Quoted 3 lines

It looks like APST feature cannot be set anymore after enabling powersupersave. Also the PCIe device disappears completely from lspci output.

My guess is this is to be expected after the DPC event. That basically disconnects the PCIe device from the system.

Quoted 3 lines

Any idea why the device is failing with powersupersave and how to avoid it? Especially how to enable it but skip certain broken devices as this is my boot device.

Quoted 5 lines

We could conceivably add a quirk if we find that L1SS is broken on this particular device. But L1SS is so new that I'd be more suspicious of the Linux code than the device.

Bjorn

Hi Rajat,

Quoted 33 lines · Rajat Jain

Quoted 30 lines · Bjorn Helgaas

[+cc Rajat, Keith, linux-kernel]

Quoted 11 lines · Maik Broemme

I have a Samsung 960 PRO NVMe SSD (Non-Volatile memory controller:
Samsung Electronics Co Ltd NVMe SSD Controller SM961/PM961). It works fine until I enable powersupersave via /sys/module/pcie_aspm/parameters/policy

ASPM is enabled in BIOS and works fine for all devices and in powersave mode. I'm able to reproduce this always at any time while the system is up and running via:

$> echo powersupersave > /sys/module/pcie_aspm/parameters/policy

Quoted 7 lines

The Linux kernel is 4.14.4 and APST for my device is working with powersave. As soon as I enable powersupersave I get:

[11535.142755] dpc 0000:00:10.0:pcie010: DPC containment event, status:0x1f09 source:0x0000
[11535.142760] dpc 0000:00:10.0:pcie010: DPC unmasked uncorrectable error detected, remove downstream devices
[11535.159999] nvme0n1: detected capacity change from 1024209543168 to 0

...

Can you start by opening a bug report at https://bugzilla.kernel.org,
category Drivers/PCI, and attaching the complete "lspci -vv" output (as root) and the complete dmesg log? Make sure you have a new enough lspci to decode the ASPM L1 Substates capability and the LTR bits. Source is at git://git.kernel.org/pub/scm/utils/pciutils/pciutils.git

powersupersave enables ASPM L1 Substates. Rajat, do you have any ideas about this or how we might debug it?

Quoted 6 lines

I know Maik mentioned that this is the boot device. Maik, is it possible to boot off something else so that we can do some more experiments on this port? If so,
- can you try to see if the device comes back if you switch the ASPM policy back from "powersupersave" -> powersave, and potentially do a rescan (echo 1 > /sys/bus/pci/rescan)?

Yes it is possible, will do later today.

Quoted 9 lines

- It would be good to get the complete lspci -vv for the root port (assuming device is connected to root port i.e. no switch). Specifically what does the Link status show?
- Also, do you know if your root port provides any debug registers that could tell the current L1 substate of the link (My system's root port had such register).
- I had usually resorted to a PCIe analyzer to peak at the packets when I was debugging it. Not sure if that is an option here.

Quoted 6 lines

I don't see any debug prints in aspm.c that we could enable. Even if I provide a patch, I suspect that the problem will start at the last step of the pcie_config_aspm_l1ss() i.e. as soon as we really enable it in HW. Maik, would you be open to take a debug patch that adds some debug prints and try it out (compile your kernel with that patch)?

Sure that is fine. I will also re-run later today with 4.15rc3.

Quoted 35 lines

Quoted 35 lines

Keith, is this really all the information about the event that we can get out of DPC? Is there some AER logging we might be able to get via "lspci -vv"? Sounds like this is the boot disk, so Maik may not be able to run lspci after the DPC event. If there *is* any AER info, can we connect up the DPC event so we can print the AER info from the kernel?

I wonder if there's some way improper L1 Substate configuration could cause a DPC event. There are lots of knobs there that seem to depend on devices, and I'm not sure we have them all correct yet.

There are some recent changes in that area that are in linux-next:

PCI/ASPM: Enable Latency Tolerance Reporting when supported PCI/ASPM: Calculate LTR_L1.2_THRESHOLD from device characteristics PCI/ASPM: Use correct capability pointer to program LTR_L1.2_THRESHOLD PCI/ASPM: Account for downstream device's Port Common_Mode_Restore_Time

It's conceivable that they could have some bearing on this problem. If you could give this a whirl on linux-next, that would be interesting. If you do this, please also collect the "lspci -vv" output there so we can compare it with the v4.14 configuration.

Quoted 3 lines

It looks like APST feature cannot be set anymore after enabling powersupersave. Also the PCIe device disappears completely from lspci output.

My guess is this is to be expected after the DPC event. That basically disconnects the PCIe device from the system.

Quoted 3 lines

Any idea why the device is failing with powersupersave and how to avoid it? Especially how to enable it but skip certain broken devices as this is my boot device.

Quoted 6 lines

Quoted 5 lines

We could conceivably add a quirk if we find that L1SS is broken on this particular device. But L1SS is so new that I'd be more suspicious of the Linux code than the device.

Bjorn

--Maik

I've re-run the test with 4.15rc7.r111.g5f615b97cdea and the following patches from Keith:

[PATCH 1/4] PCI/AER: Return approrpiate value when AER is not supported [PATCH 2/4] PCI/AER: Provide API for getting AER information [PATCH 3/4] PCI/DPC: Enable DPC in conjuction with AER [PATCH 4/4] PCI/DPC: Print AER status in DPC event handling

The issue is still the same. Additionally to the output before I see now:

Jan 11 18:34:45 server.theraso.int kernel: dpc 0000:00:10.0:pcie010: DPC containment event, status:0x1f09 source:0x0000
Jan 11 18:34:45 server.theraso.int kernel: dpc 0000:00:10.0:pcie010: DPC unmasked uncorrectable error detected, remove downstream devices
Jan 11 18:34:45 server.theraso.int kernel: pcieport 0000:00:10.0: PCIe Bus Error: severity=Uncorrected (Non-Fatal), type=Transaction Layer, id=0080(Receiver ID)
Jan 11 18:34:45 server.theraso.int kernel: pcieport 0000:00:10.0:   device [8086:19aa] error status/mask=00000020/00000000
Jan 11 18:34:45 server.theraso.int kernel: pcieport 0000:00:10.0:    [ 5] Surprise Down Error    (First)
Jan 11 18:34:46 server.theraso.int kernel: nvme0n1: detected capacity change from 1024209543168 to 0

Quoted 8 lines · Maik Broemme

I've re-run the test with 4.15rc7.r111.g5f615b97cdea and the following patches from Keith:

[PATCH 1/4] PCI/AER: Return approrpiate value when AER is not supported [PATCH 2/4] PCI/AER: Provide API for getting AER information [PATCH 3/4] PCI/DPC: Enable DPC in conjuction with AER [PATCH 4/4] PCI/DPC: Print AER status in DPC event handling

Quoted 8 lines

The issue is still the same. Additionally to the output before I see now:

Jan 11 18:34:45 server.theraso.int kernel: dpc 0000:00:10.0:pcie010: DPC containment event, status:0x1f09 source:0x0000
Jan 11 18:34:45 server.theraso.int kernel: dpc 0000:00:10.0:pcie010: DPC unmasked uncorrectable error detected, remove downstream devices
Jan 11 18:34:45 server.theraso.int kernel: pcieport 0000:00:10.0: PCIe Bus Error: severity=Uncorrected (Non-Fatal), type=Transaction Layer, id=0080(Receiver ID)
Jan 11 18:34:45 server.theraso.int kernel: pcieport 0000:00:10.0:   device [8086:19aa] error status/mask=00000020/00000000
Jan 11 18:34:45 server.theraso.int kernel: pcieport 0000:00:10.0:    [ 5] Surprise Down Error    (First)
Jan 11 18:34:46 server.theraso.int kernel: nvme0n1: detected capacity change from 1024209543168 to 0

Okay, so that series wasn't going to fix anything, but at least it gets some visibility into what's happened. The DPC was triggered due to a Surprise Down uncorrectable error, so the power settting is causing the link to fail.

The NVMe driver has quirks specifically for this vendor's devices to fence off NVMe specific automated power settings. Your observations appear to align with the same issues.