Skip to contentSkip to conversation

Are AER corrected errors worrying?

DISCUSSION Samuel Thibault 2 participants 8 messages Jan 1, 2021 – Jan 6, 2021

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

Hello,

Our lab has bought a new Dell Latitude 5410 laptop, I installed debian bullseye on it with kernel 5.9.0-5-amd64, but it is spitting these errors now and then (sometimes a dozen per a minute):

Jan  1 23:30:53 begin kernel: [   46.675818] pcieport 0000:00:1d.0: AER: Corrected error received: 0000:02:00.0
Jan  1 23:30:53 begin kernel: [   46.675933] nvme 0000:02:00.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, (Receiver ID)
Jan  1 23:30:53 begin kernel: [   46.676048] nvme 0000:02:00.0:   device [15b7:5006] error status/mask=00000001/0000e000
Jan  1 23:30:53 begin kernel: [   46.676140] nvme 0000:02:00.0:    [ 0] RxErr

Since it's corrected it's not actually an issue, but how worrying is it to see such errors on new hardware? Documentation/PCI/pcieaer-howto.rst is not commenting whether we are really supposed to see some of them. I see forums telling to use pci=noaer to stop the error logging, but is that really something to do?

Samuel

_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme

maintainer
#1 lore raw
Quoted 11 lines · Samuel Thibault

Hello,

Our lab has bought a new Dell Latitude 5410 laptop, I installed debian bullseye on it with kernel 5.9.0-5-amd64, but it is spitting these errors now and then (sometimes a dozen per a minute):

Jan  1 23:30:53 begin kernel: [   46.675818] pcieport 0000:00:1d.0: AER: Corrected error received: 0000:02:00.0
Jan  1 23:30:53 begin kernel: [   46.675933] nvme 0000:02:00.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, (Receiver ID)
Jan  1 23:30:53 begin kernel: [   46.676048] nvme 0000:02:00.0:   device [15b7:5006] error status/mask=00000001/0000e000
Jan  1 23:30:53 begin kernel: [   46.676140] nvme 0000:02:00.0:    [ 0] RxErr
Quoted 5 lines

Since it's corrected it's not actually an issue, but how worrying is it to see such errors on new hardware? Documentation/PCI/pcieaer-howto.rst is not commenting whether we are really supposed to see some of them. I see forums telling to use pci=noaer to stop the error logging, but is that really something to do?

Additional work has to happen to correct a receiver error, so it's possible you're getting degraded performance. You may not notice worse performance if these are infrequent enough, though.

Sometimes these types of errors occur from low power settings, so you can try disabling the automatic management of these (assuming the hardware supports it). To disable nvme specific power state transitions, the kernel parameter is "nvme_core.default_ps_max_latency_us=0". PCI also has automatic link power savings that you can disable with parameter "pcie_aspm=off". It might be worth seeing if either of those changes your observation.

_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme

#2 lore raw

Hello,

Quoted 2 lines · Vidya Sagar

Since this is a laptop, I'm suspecting that ASPM states might have been enabled which could be causing these errors.

Keith Busch, le lun. 04 janv. 2021 10:44:35 -0800, a ecrit:

Quoted 4 lines

Sometimes these types of errors occur from low power settings, so you can try disabling the automatic management of these (assuming the hardware supports it). To disable nvme specific power state transitions, the kernel parameter is "nvme_core.default_ps_max_latency_us=0".

I have tried to add it, and this one line changed in lspci -vv:

02:00.0 Non-Volatile memory controller: Sandisk Corp WD Black SN750 / PC SN730 NVMe SSD (prog-if 02 [NVM Express]) [...]
Capabilities: [c0] Express (v2) Endpoint, MSI 00
[...]
DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend-
turned to
DevSta: CorrErr+ NonFatalErr- FatalErr- UnsupReq+ AuxPwr- TransPend-

that last value happens to be what I was seeing for that line with the manufacturer-provided ubuntu linux kernel.

So far (30m uptime) no corrected error report, I'll watch in the coming hours/days to see if that avoided the issue. I wasn't able to trigger such corrected errors by loading the machine, so possibly that's indeed the converse that I should have been trying: letting it go low power :)

Quoted 2 lines

PCI also has automatic link power savings that you can disable with parameter "pcie_aspm=off".

I'll try that if I still see errors with the nvme_core parameter.

Samuel

_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme

#3 lore raw

Samuel Thibault, le lun. 04 janv. 2021 21:12:47 +0100, a ecrit:

Quoted 10 lines
Quoted 2 lines · Vidya Sagar

Since this is a laptop, I'm suspecting that ASPM states might have been enabled which could be causing these errors.

Keith Busch, le lun. 04 janv. 2021 10:44:35 -0800, a ecrit:

Quoted 4 lines

Sometimes these types of errors occur from low power settings, so you can try disabling the automatic management of these (assuming the hardware supports it). To disable nvme specific power state transitions, the kernel parameter is "nvme_core.default_ps_max_latency_us=0".

Quoted 4 lines

I have tried to add it,

I'll watch in the coming
hours/days to see if that avoided the issue.

I did get one

Jan  4 22:34:53 begin kernel: [ 7165.207562] pcieport 0000:00:1d.0: AER: Corrected error received: 0000:02:00.0
Jan  4 22:34:53 begin kernel: [ 7165.213891] nvme 0000:02:00.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, (Receiver ID)
Jan  4 22:34:53 begin kernel: [ 7165.216949] nvme 0000:02:00.0:   device [15b7:5006] error status/mask=00000001/0000e000
Jan  4 22:34:53 begin kernel: [ 7165.219995] nvme 0000:02:00.0:    [ 0] RxErr
Quoted 4 lines
Quoted 2 lines

PCI also has automatic link power savings that you can disable with parameter "pcie_aspm=off".

I'll try that if I still see errors with the nvme_core parameter.

I'm on it.

Samuel

_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme

#4 lore raw

Samuel Thibault, le lun. 04 janv. 2021 22:36:48 +0100, a ecrit:

Quoted 8 lines

Samuel Thibault, le lun. 04 janv. 2021 21:12:47 +0100, a ecrit:

Quoted 5 lines

Keith Busch, le lun. 04 janv. 2021 10:44:35 -0800, a ecrit:

Quoted 2 lines

PCI also has automatic link power savings that you can disable with parameter "pcie_aspm=off".

I'll try that if I still see errors with the nvme_core parameter.

I'm on it.

(FTR It switched these lines)

02:00.0 Non-Volatile memory controller: Sandisk Corp WD Black SN750 / PC SN730 NVMe SSD (prog-if 02 [NVM Express])

	[...]
	Capabilities: [c0] Express (v2) Endpoint, MSI 00
		[...]
		DevCtl: CorrErr+ NonFatalErr+ FatalErr+ UnsupReq+
	->	DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq-
		DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend-
	->	DevSta: CorrErr+ NonFatalErr- FatalErr- UnsupReq+ AuxPwr- TransPend-
	[...]
	Capabilities: [100 v2] Advanced Error Reporting
		CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr-
	->	CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
	[...]
	Capabilities: [900 v1] L1 PM Substates
		[...]
		L1SubCtl1: PCI-PM_L1.2+ PCI-PM_L1.1- ASPM_L1.2+ ASPM_L1.1-
			T_CommonMode=0us LTR1.2_Threshold=163840ns
		->	T_CommonMode=0us LTR1.2_Threshold=81920ns

Samuel

_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme

#5 lore raw
In reply to Samuel Thibault

Samuel Thibault, le lun. 04 janv. 2021 22:36:48 +0100, a ecrit:

Quoted 23 lines

Samuel Thibault, le lun. 04 janv. 2021 21:12:47 +0100, a ecrit:

Quoted 10 lines
Quoted 2 lines · Vidya Sagar

Since this is a laptop, I'm suspecting that ASPM states might have been enabled which could be causing these errors.

Keith Busch, le lun. 04 janv. 2021 10:44:35 -0800, a ecrit:

Quoted 4 lines

Sometimes these types of errors occur from low power settings, so you can try disabling the automatic management of these (assuming the hardware supports it). To disable nvme specific power state transitions, the kernel parameter is "nvme_core.default_ps_max_latency_us=0".

Quoted 4 lines

I have tried to add it,

I'll watch in the coming
hours/days to see if that avoided the issue.

I did get one

Jan  4 22:34:53 begin kernel: [ 7165.207562] pcieport 0000:00:1d.0: AER: Corrected error received: 0000:02:00.0
Jan  4 22:34:53 begin kernel: [ 7165.213891] nvme 0000:02:00.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, (Receiver ID)
Jan  4 22:34:53 begin kernel: [ 7165.216949] nvme 0000:02:00.0:   device [15b7:5006] error status/mask=00000001/0000e000
Jan  4 22:34:53 begin kernel: [ 7165.219995] nvme 0000:02:00.0:    [ 0] RxErr
Quoted 6 lines
Quoted 4 lines
Quoted 2 lines

PCI also has automatic link power savings that you can disable with parameter "pcie_aspm=off".

I'll try that if I still see errors with the nvme_core parameter.

I'm on it.

I tried to make the machine only run apt-get update every 10m for 24h.

With pcie_aspm=off, I didn't get any corrected error Without it I got 39 corrected errors

So that seems very relevant :)

Is there more I can provide to investigate if that can somehow be fixed in the driver? I guess I can safely use the system with pcie_aspm=off? (the energy saving seems neglectible)

Samuel

maintainer
#6 lore raw
Quoted 33 lines · Samuel Thibault

Samuel Thibault, le lun. 04 janv. 2021 22:36:48 +0100, a ecrit:

Quoted 23 lines

Samuel Thibault, le lun. 04 janv. 2021 21:12:47 +0100, a ecrit:

Quoted 10 lines
Quoted 2 lines · Vidya Sagar

Since this is a laptop, I'm suspecting that ASPM states might have been enabled which could be causing these errors.

Keith Busch, le lun. 04 janv. 2021 10:44:35 -0800, a ecrit:

Quoted 4 lines

Sometimes these types of errors occur from low power settings, so you can try disabling the automatic management of these (assuming the hardware supports it). To disable nvme specific power state transitions, the kernel parameter is "nvme_core.default_ps_max_latency_us=0".

Quoted 4 lines

I have tried to add it,

I'll watch in the coming
hours/days to see if that avoided the issue.

I did get one

Jan  4 22:34:53 begin kernel: [ 7165.207562] pcieport 0000:00:1d.0: AER: Corrected error received: 0000:02:00.0
Jan  4 22:34:53 begin kernel: [ 7165.213891] nvme 0000:02:00.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, (Receiver ID)
Jan  4 22:34:53 begin kernel: [ 7165.216949] nvme 0000:02:00.0:   device [15b7:5006] error status/mask=00000001/0000e000
Jan  4 22:34:53 begin kernel: [ 7165.219995] nvme 0000:02:00.0:    [ 0] RxErr
Quoted 6 lines
Quoted 4 lines
Quoted 2 lines

PCI also has automatic link power savings that you can disable with parameter "pcie_aspm=off".

I'll try that if I still see errors with the nvme_core parameter.

I'm on it.

I tried to make the machine only run apt-get update every 10m for 24h.

Quoted 8 lines

With pcie_aspm=off, I didn't get any corrected error Without it I got 39 corrected errors

So that seems very relevant :)

Is there more I can provide to investigate if that can somehow be fixed in the driver? I guess I can safely use the system with pcie_aspm=off? (the energy saving seems neglectible)

I don't think there's more to do from the kernel or driver beyond disabling usage of the problematic feature. I think a proper fix would have to come from the hardware vendor.

_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme

#7 lore raw

Keith Busch, le mer. 06 janv. 2021 13:48:08 -0800, a ecrit:

Quoted 8 lines · Samuel Thibault
Quoted 3 lines

Is there more I can provide to investigate if that can somehow be fixed in the driver? I guess I can safely use the system with pcie_aspm=off? (the energy saving seems neglectible)

I don't think there's more to do from the kernel or driver beyond disabling usage of the problematic feature. I think a proper fix would have to come from the hardware vendor.

Ok, thanks!

Samuel

_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme