Skip to contentSkip to conversation

New: "iommu/vt-d: Enable PCI ACS for platform opt in hint" makes NVMe config space not accessible after S3]

[email protected]: Bjorn Helgaas 5 participants 9 messages Sep 23, 2020 – Sep 25, 2020

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

[+cc IOMMU and NVMe folks]

Sorry, I forgot to forward this to linux-pci when it was first reported.

Apparently this happens with v5.9-rc3, and may be related to 50310600ebda ("iommu/vt-d: Enable PCI ACS for platform opt in hint"), which appeared in v5.8-rc3.

There are several dmesg logs and proposed patches in the bugzilla, but no analysis yet of what the problem is. From the first dmesg attachment (https://bugzilla.kernel.org/attachment.cgi?id=292327):

  [   50.434945] PM: suspend entry (deep)
  [   50.802086] nvme 0000:01:00.0: saving config space at offset 0x0 (reading 0x11e0f)
  [   50.842775] ACPI: Preparing to enter system sleep state S3
  [   50.858922] ACPI: Waking up from system sleep state S3
  [   50.883622] nvme 0000:01:00.0: can't change power state from D3hot to D0 (config space inaccessible)
  [   50.947352] nvme 0000:01:00.0: restoring config space at offset 0x0 (was 0xffffffff, writing 0x11e0f)
  [   50.947816] pcieport 0000:00:1b.0: DPC: containment event, status:0x1f01 source:0x0000
  [   50.947817] pcieport 0000:00:1b.0: DPC: unmasked uncorrectable error detected
  [   50.947829] pcieport 0000:00:1b.0: PCIe Bus Error: severity=Uncorrected (Non-Fatal), type=Transaction Layer, (Receiver ID)
  [   50.947830] pcieport 0000:00:1b.0:   device [8086:06ac] error status/mask=00200000/00010000
  [   50.947831] pcieport 0000:00:1b.0:    [21] ACSViol                (First)
  [   50.947841] pcieport 0000:00:1b.0: AER: broadcast error_detected message
  [   50.947843] nvme nvme0: frozen state error detected, reset controller

I suspect the nvme "can't change power state" and restore config space errors are a consequence of the DPC event. If DPC disables the link, the device is inaccessible.

I don't know what caused the ACS Violation. The AER TLP Header Log might have a clue, but unfortunately we didn't print it.

Tangent:

The fact that we didn't print the AER TLP Header log looks like a bug in itself. PCIe r5.0, sec 6.2.7, table 6-5, says many errors, including ACS Violation, should log the TLP header. But aer_get_device_error_info() only reads the log for error bits in AER_LOG_TLP_MASKS, which doesn't include PCI_ERR_UNC_ACSV.

I don't think there's a "TLP Header Log Valid" bit, and it's ugly to have to update AER_LOG_TLP_MASKS if new errors are added. I think maybe we should always print the header log.

----- Forwarded message from [email protected] -----

Date: Fri, 04 Sep 2020 14:31:20 +0000
From: [email protected]
To: [email protected]
Subject: [Bug 209149] New: "iommu/vt-d: Enable PCI ACS for platform opt in
hint" makes NVMe config space not accessible after S3
Message-ID: <[email protected]/>

https://bugzilla.kernel.org/show_bug.cgi?id=209149

Bug ID: 209149
Summary: "iommu/vt-d: Enable PCI ACS for platform opt in hint"
makes NVMe config space not accessible after S3
Product: Drivers
Version: 2.5
Kernel Version: mainline
Hardware: All
OS: Linux
Tree: Mainline
Status: NEW
Severity: normal
Priority: P1
Component: PCI
Assignee: [email protected]
Reporter: [email protected]
Regression: No

Here's the error:

[   50.947816] pcieport 0000:00:1b.0: DPC: containment event, status:0x1f01
source:0x0000
[   50.947817] pcieport 0000:00:1b.0: DPC: unmasked uncorrectable error
detected
[   50.947829] pcieport 0000:00:1b.0: PCIe Bus Error: severity=Uncorrected
(Non-Fatal), type=Transaction Layer, (Receiver ID)
[   50.947830] pcieport 0000:00:1b.0:   device [8086:06ac] error
status/mask=00200000/00010000
[   50.947831] pcieport 0000:00:1b.0:    [21] ACSViol                (First)
[   50.947841] pcieport 0000:00:1b.0: AER: broadcast error_detected message
[   50.947843] nvme nvme0: frozen state error detected, reset controller
Signature · 4 lines
You are receiving this mail because:
You are watching the assignee of the bug.

----- End forwarded message -----
#1 lore raw

Hi Bjorn

Quoted 5 lines · Bjorn Helgaas

[+cc IOMMU and NVMe folks]

Sorry, I forgot to forward this to linux-pci when it was first reported.

Quoted 7 lines

Apparently this happens with v5.9-rc3, and may be related to 50310600ebda ("iommu/vt-d: Enable PCI ACS for platform opt in hint"), which appeared in v5.8-rc3.

There are several dmesg logs and proposed patches in the bugzilla, but no analysis yet of what the problem is. From the first dmesg attachment (https://bugzilla.kernel.org/attachment.cgi?id=292327):

We have been investigating this internally as well. It appears maybe the specupdate for Cometlake is missing the errata documention. The offsets were wrong in some of them, and if its the same issue its likely cause.

Will nudge the hw folks to hunt that down :-(.

Cheers,
Ashok

#2 lore raw
Quoted 14 lines · Ashok

Hi Bjorn

Quoted 9 lines · Bjorn Helgaas

[+cc IOMMU and NVMe folks]

Sorry, I forgot to forward this to linux-pci when it was first reported.

Apparently this happens with v5.9-rc3, and may be related to 50310600ebda ("iommu/vt-d: Enable PCI ACS for platform opt in hint"), which appeared in v5.8-rc3.

Quoted 7 lines
Quoted 3 lines

There are several dmesg logs and proposed patches in the bugzilla, but no analysis yet of what the problem is. From the first dmesg attachment (https://bugzilla.kernel.org/attachment.cgi?id=292327):

We have been investigating this internally as well. It appears maybe the specupdate for Cometlake is missing the errata documention. The offsets were wrong in some of them, and if its the same issue its likely cause.

Can you please also confirm if errata applies to Tigerlake ?

Thanks,

Rajat

Quoted 5 lines

Will nudge the hw folks to hunt that down :-(.

Cheers,
Ashok

#3 lore raw
Quoted 19 lines · Rajat Jain
Quoted 18 lines · Ashok

Hi Bjorn

Quoted 12 lines · Bjorn Helgaas

[+cc IOMMU and NVMe folks]

Sorry, I forgot to forward this to linux-pci when it was first reported.

Apparently this happens with v5.9-rc3, and may be related to 50310600ebda ("iommu/vt-d: Enable PCI ACS for platform opt in hint"), which appeared in v5.8-rc3.

There are several dmesg logs and proposed patches in the bugzilla, but no analysis yet of what the problem is. From the first dmesg attachment (https://bugzilla.kernel.org/attachment.cgi?id=292327):

Quoted 6 lines
Quoted 3 lines

We have been investigating this internally as well. It appears maybe the specupdate for Cometlake is missing the errata documention. The offsets were wrong in some of them, and if its the same issue its likely cause.

Can you please also confirm if errata applies to Tigerlake ?

We confirmed ICL/TGL isn't affected.

_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu

#4 lore raw
In reply to Bjorn Helgaas

[+Cc Christoph]

Quoted 7 lines · Bjorn Helgaas

[+cc IOMMU and NVMe folks]

Sorry, I forgot to forward this to linux-pci when it was first reported.

Quoted 7 lines

Apparently this happens with v5.9-rc3, and may be related to 50310600ebda ("iommu/vt-d: Enable PCI ACS for platform opt in hint"), which appeared in v5.8-rc3.

There are several dmesg logs and proposed patches in the bugzilla, but no analysis yet of what the problem is. From the first dmesg attachment (https://bugzilla.kernel.org/attachment.cgi?id=292327):

AFAIK Intel is working on it internally.
Comet Lake probably needs ACS quirk like older generation chips.

Quoted 30 lines
 [   50.434945] PM: suspend entry (deep)
 [   50.802086] nvme 0000:01:00.0: saving config space at offset 0x0 (reading 0x11e0f)
 [   50.842775] ACPI: Preparing to enter system sleep state S3
 [   50.858922] ACPI: Waking up from system sleep state S3
 [   50.883622] nvme 0000:01:00.0: can't change power state from D3hot to D0 (config space inaccessible)
 [   50.947352] nvme 0000:01:00.0: restoring config space at offset 0x0 (was 0xffffffff, writing 0x11e0f)
 [   50.947816] pcieport 0000:00:1b.0: DPC: containment event, status:0x1f01 source:0x0000
 [   50.947817] pcieport 0000:00:1b.0: DPC: unmasked uncorrectable error detected
 [   50.947829] pcieport 0000:00:1b.0: PCIe Bus Error: severity=Uncorrected (Non-Fatal), type=Transaction Layer, (Receiver ID)
 [   50.947830] pcieport 0000:00:1b.0:   device [8086:06ac] error status/mask=00200000/00010000
 [   50.947831] pcieport 0000:00:1b.0:    [21] ACSViol                (First)
 [   50.947841] pcieport 0000:00:1b.0: AER: broadcast error_detected message
 [   50.947843] nvme nvme0: frozen state error detected, reset controller

I suspect the nvme "can't change power state" and restore config space errors are a consequence of the DPC event. If DPC disables the link, the device is inaccessible.

I don't know what caused the ACS Violation. The AER TLP Header Log might have a clue, but unfortunately we didn't print it.

Tangent:

The fact that we didn't print the AER TLP Header log looks like a bug in itself. PCIe r5.0, sec 6.2.7, table 6-5, says many errors, including ACS Violation, should log the TLP header. But aer_get_device_error_info() only reads the log for error bits in AER_LOG_TLP_MASKS, which doesn't include PCI_ERR_UNC_ACSV.

Quoted 3 lines

I don't think there's a "TLP Header Log Valid" bit, and it's ugly to have to update AER_LOG_TLP_MASKS if new errors are added. I think maybe we should always print the header log.

I can attach TLP Header if there's a patch...

Kai-Heng

Quoted 47 lines

----- Forwarded message from [email protected] -----

Date: Fri, 04 Sep 2020 14:31:20 +0000
From: [email protected]
To: [email protected]
Subject: [Bug 209149] New: "iommu/vt-d: Enable PCI ACS for platform opt in
hint" makes NVMe config space not accessible after S3
Message-ID: <[email protected]/>

https://bugzilla.kernel.org/show_bug.cgi?id=209149

Bug ID: 209149
Summary: "iommu/vt-d: Enable PCI ACS for platform opt in hint"
makes NVMe config space not accessible after S3
Product: Drivers
Version: 2.5
Kernel Version: mainline
Hardware: All
OS: Linux
Tree: Mainline
Status: NEW
Severity: normal
Priority: P1
Component: PCI
Assignee: [email protected]
Reporter: [email protected]
Regression: No

Here's the error:

[   50.947816] pcieport 0000:00:1b.0: DPC: containment event, status:0x1f01
source:0x0000
[   50.947817] pcieport 0000:00:1b.0: DPC: unmasked uncorrectable error
detected
[   50.947829] pcieport 0000:00:1b.0: PCIe Bus Error: severity=Uncorrected
(Non-Fatal), type=Transaction Layer, (Receiver ID)
[   50.947830] pcieport 0000:00:1b.0:   device [8086:06ac] error
status/mask=00200000/00010000
[   50.947831] pcieport 0000:00:1b.0:    [21] ACSViol                (First)
[   50.947841] pcieport 0000:00:1b.0: AER: broadcast error_detected message
[   50.947843] nvme nvme0: frozen state error detected, reset controller
Signature · 4 lines
You are receiving this mail because:
You are watching the assignee of the bug.

----- End forwarded message -----
#5 lore raw

Hi Kai

+ Alex, since he had some of the early quirks authored.

Quoted 13 lines · Kai-Heng Feng

[+Cc Christoph]

Quoted 11 lines · Bjorn Helgaas

[+cc IOMMU and NVMe folks]

Sorry, I forgot to forward this to linux-pci when it was first reported.

Apparently this happens with v5.9-rc3, and may be related to 50310600ebda ("iommu/vt-d: Enable PCI ACS for platform opt in hint"), which appeared in v5.8-rc3.

Quoted 6 lines
Quoted 3 lines

There are several dmesg logs and proposed patches in the bugzilla, but no analysis yet of what the problem is. From the first dmesg attachment (https://bugzilla.kernel.org/attachment.cgi?id=292327):

AFAIK Intel is working on it internally.
Comet Lake probably needs ACS quirk like older generation chips.

I have confirmed with Internal documentation that the problem exists on Comet Lake. But its fixed ICL and TGL generations.

Unfortunately I do not see if the public specupdate documents are for these generation chipsets to makes sure all root port id's can be captured.

There is also another entry in bugzilla that was forwarded that referred to Request Redirect Capability to be always disabled as well. This same workaround also seems to be turning off RR for the root port. I believe it should fix it as well. But i saw another patch attached.

Can you tell how you reproduce this? just doing a

#echo mem > /sys/power/state

is sufficient with an attached NVMe drive?

Quoted 16 lines
Quoted 16 lines

 [   50.434945] PM: suspend entry (deep)
 [   50.802086] nvme 0000:01:00.0: saving config space at offset 0x0 (reading 0x11e0f)
 [   50.842775] ACPI: Preparing to enter system sleep state S3
 [   50.858922] ACPI: Waking up from system sleep state S3
 [   50.883622] nvme 0000:01:00.0: can't change power state from D3hot to D0 (config space inaccessible)
 [   50.947352] nvme 0000:01:00.0: restoring config space at offset 0x0 (was 0xffffffff, writing 0x11e0f)
 [   50.947816] pcieport 0000:00:1b.0: DPC: containment event, status:0x1f01 source:0x0000
 [   50.947817] pcieport 0000:00:1b.0: DPC: unmasked uncorrectable error detected
 [   50.947829] pcieport 0000:00:1b.0: PCIe Bus Error: severity=Uncorrected (Non-Fatal), type=Transaction Layer, (Receiver ID)
 [   50.947830] pcieport 0000:00:1b.0:   device [8086:06ac] error status/mask=00200000/00010000
 [   50.947831] pcieport 0000:00:1b.0:    [21] ACSViol                (First)
 [   50.947841] pcieport 0000:00:1b.0: AER: broadcast error_detected message
 [   50.947843] nvme nvme0: frozen state error detected, reset controller
Quoted 7 lines
Quoted 7 lines

I suspect the nvme "can't change power state" and restore config space errors are a consequence of the DPC event. If DPC disables the link, the device is inaccessible.

I don't know what caused the ACS Violation. The AER TLP Header Log might have a clue, but unfortunately we didn't print it.

Apparently it also requires to disable RR, and I'm not able to confirm if CML requires that as well.

pci_quirk_disable_intel_spt_pch_acs_redir() also seems to consult the same table, so i'm not sure why we need the other patch in bugzilla is required.

Cheers,
Ashok

#6 lore raw
Quoted 66 lines · Ashok

Hi Kai

+ Alex, since he had some of the early quirks authored.

Quoted 13 lines · Kai-Heng Feng

[+Cc Christoph]

Quoted 11 lines · Bjorn Helgaas

[+cc IOMMU and NVMe folks]

Sorry, I forgot to forward this to linux-pci when it was first reported.

Apparently this happens with v5.9-rc3, and may be related to 50310600ebda ("iommu/vt-d: Enable PCI ACS for platform opt in hint"), which appeared in v5.8-rc3.

Quoted 6 lines
Quoted 3 lines

There are several dmesg logs and proposed patches in the bugzilla, but no analysis yet of what the problem is. From the first dmesg attachment (https://bugzilla.kernel.org/attachment.cgi?id=292327):

AFAIK Intel is working on it internally.
Comet Lake probably needs ACS quirk like older generation chips.

I have confirmed with Internal documentation that the problem exists on Comet Lake. But its fixed ICL and TGL generations.

Unfortunately I do not see if the public specupdate documents are for these generation chipsets to makes sure all root port id's can be captured.

There is also another entry in bugzilla that was forwarded that referred to Request Redirect Capability to be always disabled as well. This same workaround also seems to be turning off RR for the root port. I believe it should fix it as well. But i saw another patch attached.

Can you tell how you reproduce this? just doing a

#echo mem > /sys/power/state

is sufficient with an attached NVMe drive?

Quoted 16 lines
Quoted 16 lines
  

 [   50.434945] PM: suspend entry (deep)
 [   50.802086] nvme 0000:01:00.0: saving config space at offset 0x0 (reading 0x11e0f)
 [   50.842775] ACPI: Preparing to enter system sleep state S3
 [   50.858922] ACPI: Waking up from system sleep state S3
 [   50.883622] nvme 0000:01:00.0: can't change power state from D3hot to D0 (config space inaccessible)
 [   50.947352] nvme 0000:01:00.0: restoring config space at offset 0x0 (was 0xffffffff, writing 0x11e0f)
 [   50.947816] pcieport 0000:00:1b.0: DPC: containment event, status:0x1f01 source:0x0000
 [   50.947817] pcieport 0000:00:1b.0: DPC: unmasked uncorrectable error detected
 [   50.947829] pcieport 0000:00:1b.0: PCIe Bus Error: severity=Uncorrected (Non-Fatal), type=Transaction Layer, (Receiver ID)
 [   50.947830] pcieport 0000:00:1b.0:   device [8086:06ac] error status/mask=00200000/00010000
 [   50.947831] pcieport 0000:00:1b.0:    [21] ACSViol                (First)
 [   50.947841] pcieport 0000:00:1b.0: AER: broadcast error_detected message
 [   50.947843] nvme nvme0: frozen state error detected, reset controller
Quoted 7 lines
Quoted 7 lines

I suspect the nvme "can't change power state" and restore config space errors are a consequence of the DPC event. If DPC disables the link, the device is inaccessible.

I don't know what caused the ACS Violation. The AER TLP Header Log might have a clue, but unfortunately we didn't print it.

Quoted 5 lines

Apparently it also requires to disable RR, and I'm not able to confirm if CML requires that as well.

pci_quirk_disable_intel_spt_pch_acs_redir() also seems to consult the same table, so i'm not sure why we need the other patch in bugzilla is required.

If we're talking about the Intel bug where PCH root ports implement the ACS capability and control registers as dword rather than word registers, then how is ACS getting enabled in order to generate an ACS violation? The standard ACS code would write to the control register word at offset 6, which is still the read-only capability register on those devices. Thanks,

Alex

#7 lore raw

Hi Alex

Quoted 6 lines
Quoted 5 lines

Apparently it also requires to disable RR, and I'm not able to confirm if CML requires that as well.

pci_quirk_disable_intel_spt_pch_acs_redir() also seems to consult the same table, so i'm not sure why we need the other patch in bugzilla is required.

Quoted 6 lines

If we're talking about the Intel bug where PCH root ports implement the ACS capability and control registers as dword rather than word registers, then how is ACS getting enabled in order to generate an ACS violation? The standard ACS code would write to the control register word at offset 6, which is still the read-only capability register on those devices. Thanks,

Right... Maybe we need header log to figure out what exatly is happening.

#8 lore raw

Raj,

Quoted 18 lines · Ashok

Hi Alex

Quoted 6 lines
Quoted 5 lines

Apparently it also requires to disable RR, and I'm not able to confirm if CML requires that as well.

pci_quirk_disable_intel_spt_pch_acs_redir() also seems to consult the same table, so i'm not sure why we need the other patch in bugzilla is required.

Quoted 6 lines

If we're talking about the Intel bug where PCH root ports implement the ACS capability and control registers as dword rather than word registers, then how is ACS getting enabled in order to generate an ACS violation? The standard ACS code would write to the control register word at offset 6, which is still the read-only capability register on those devices. Thanks,

Quoted 2 lines

Right... Maybe we need header log to figure out what exatly is happening.

Please let me know what logs you need.

As Bjorn mentioned earlier, there's currently no way to dump TLP header log?

Kai-Heng