Skip to contentSkip to conversation

PCI/AER: Configure ECRC only AER is native

PATCH Superseded Vidya Sagar 3 participants 12 messages Jan 11, 2023 – Jan 12, 2023

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 PCI/AER: Configure ECRC only AER is native

As the ECRC configuration bits are part of AER registers, configure ECRC only if AER is natively owned by the kernel.

---
 drivers/pci/pcie/aer.c | 3 +++
 1 file changed, 3 insertions(+)
drivers/pci/pcie/aer.c +3 -0
index e2d8a74f83c3..730b47bdcdef 100644
@@ -184,6 +184,9 @@ static int disable_ecrc_checking(struct pci_dev *dev)
  */
 void pcie_set_ecrc_checking(struct pci_dev *dev)
 {
+	if (!pcie_aer_is_native(dev))
+		return;
+
 	switch (ecrc_policy) {
 	case ECRC_POLICY_DEFAULT:
 		return;
Signature · 1 line
2.17.1
#1 lore raw
Quoted 2 lines · Vidya Sagar

As the ECRC configuration bits are part of AER registers, configure ECRC only if AER is natively owned by the kernel.

ecrc command line option takes "bios/on/off" as possible options. It does not clarify whether "on/off" choices can only be used if AER is owned by OS or it can override the ownership of ECRC configuration similar to pcie_ports=native option. Maybe that needs to be clarified.

Quoted 20 lines
---
 drivers/pci/pcie/aer.c | 3 +++
 1 file changed, 3 insertions(+)
drivers/pci/pcie/aer.c +3 -0
index e2d8a74f83c3..730b47bdcdef 100644
@@ -184,6 +184,9 @@ static int disable_ecrc_checking(struct pci_dev *dev)
  */
 void pcie_set_ecrc_checking(struct pci_dev *dev)
 {
+	if (!pcie_aer_is_native(dev))
+		return;
+
 	switch (ecrc_policy) {
 	case ECRC_POLICY_DEFAULT:
 		return;
Signature · 2 lines
Sathyanarayanan Kuppuswamy
Linux Kernel Developer
#2 lore raw
Quoted 8 lines · Sathyanarayanan Kuppuswamy
Quoted 2 lines · Vidya Sagar

As the ECRC configuration bits are part of AER registers, configure ECRC only if AER is natively owned by the kernel.

ecrc command line option takes "bios/on/off" as possible options. It does not clarify whether "on/off" choices can only be used if AER is owned by OS or it can override the ownership of ECRC configuration similar to pcie_ports=native option. Maybe that needs to be clarified.

Good point, what do you think of an update like this:

Documentation/admin-guide/kernel-parameters.txt +3 -1
index 6cfa6e3996cf..f7b40a439194 100644
@@ -4296,7 +4296,9 @@
 				specified, e.g., 12@pci:8086:9c22:103c:198f
 				for 4096-byte alignment.
 		ecrc=		Enable/disable PCIe ECRC (transaction layer
-				end-to-end CRC checking).
+				end-to-end CRC checking).  Only effective
+				if OS has native AER control (either granted by
+				ACPI _OSC or forced via "pcie_ports=native").
 				bios: Use BIOS/firmware settings. This is the
 				the default.
 				off: Turn ECRC off

I don't know whether the "ecrc=" parameter is really needed. If we were adding it today, I would ask "why not enable ECRC wherever it is supported?" If there are devices where it's broken, we could always add quirks to disable it on a case-by-case basis.

But I think the patch below is the right thing to do for now. Vidya, did you trip over an issue because of this, e.g., a conflict between firmware use of AER and Linux use of it? If so, maybe we could mention a symptom on the commit log. But my guess is you probably found this by inspection.

Bjorn

Quoted 23 lines
Quoted 19 lines
---
 drivers/pci/pcie/aer.c | 3 +++
 1 file changed, 3 insertions(+)
drivers/pci/pcie/aer.c +3 -0
index e2d8a74f83c3..730b47bdcdef 100644
@@ -184,6 +184,9 @@ static int disable_ecrc_checking(struct pci_dev *dev)
  */
 void pcie_set_ecrc_checking(struct pci_dev *dev)
 {
+	if (!pcie_aer_is_native(dev))
+		return;
+
 	switch (ecrc_policy) {
 	case ECRC_POLICY_DEFAULT:
 		return;
Signature · 2 lines
Sathyanarayanan Kuppuswamy
Linux Kernel Developer
#3 lore raw

Hi,

Quoted 17 lines · Bjorn Helgaas
Quoted 8 lines · Sathyanarayanan Kuppuswamy
Quoted 2 lines · Vidya Sagar

As the ECRC configuration bits are part of AER registers, configure ECRC only if AER is natively owned by the kernel.

ecrc command line option takes "bios/on/off" as possible options. It does not clarify whether "on/off" choices can only be used if AER is owned by OS or it can override the ownership of ECRC configuration similar to pcie_ports=native option. Maybe that needs to be clarified.

Good point, what do you think of an update like this:

Documentation/admin-guide/kernel-parameters.txt +0 -0
index 6cfa6e3996cf..f7b40a439194 100644

Documentation/admin-guide/kernel-parameters.txt line 4304

Quoted 10 lines
Documentation/admin-guide/kernel-parameters.txt +3 -1
@@ -4296,7 +4296,9 @@
 				specified, e.g., 12@pci:8086:9c22:103c:198f
 				for 4096-byte alignment.
 		ecrc=		Enable/disable PCIe ECRC (transaction layer
-				end-to-end CRC checking).
+				end-to-end CRC checking).  Only effective
+				if OS has native AER control (either granted by
+				ACPI _OSC or forced via "pcie_ports=native").
 				bios: Use BIOS/firmware settings. This is the
 				the default.
 				off: Turn ECRC off

Looks fine. But do we even need "bios" option? Since it is the default value, I am not sure why we need to list that as an option again. IMO this could be removed.

Quoted 5 lines

I don't know whether the "ecrc=" parameter is really needed. If we were adding it today, I would ask "why not enable ECRC wherever it is supported?" If there are devices where it's broken, we could always add quirks to disable it on a case-by-case basis.

Checking the original patch which added it, it looks like the intention is to give option to boost performance over integrity.

commit 43c16408842b0eeb367c23a6fa540ce69f99e347

Author: Andrew Patterson <[email protected]>
Date:   Wed Apr 22 16:52:09 2009 -0600

PCI: Add support for turning PCIe ECRC on or off

Adds support for PCI Express transaction layer end-to-end CRC checking (ECRC). This patch will enable/disable ECRC checking by setting/clearing the ECRC Check Enable and/or ECRC Generation Enable bits for devices that support ECRC.

The ECRC setting is controlled by the "pci=ecrc=<policy>" command-line option. If this option is not set or is set to 'bios", the enable and generation bits are left in whatever state that firmware/BIOS set them to. The "off" setting turns them off, and the "on" option turns them on (if the device supports it).

Turning ECRC on or off can be a data integrity versus performance tradeoff. In theory, turning it on will catch more data errors, turning it off means possibly better performance since CRC does not need to be calculated by the PCIe hardware and packet sizes are reduced.

Quoted 2 lines

But I think the patch below is the right thing to do for now. Vidya,

Agree.

Quoted 30 lines

did you trip over an issue because of this, e.g., a conflict between firmware use of AER and Linux use of it? If so, maybe we could mention a symptom on the commit log. But my guess is you probably found this by inspection.

Bjorn

Quoted 23 lines
Quoted 19 lines
---
 drivers/pci/pcie/aer.c | 3 +++
 1 file changed, 3 insertions(+)
drivers/pci/pcie/aer.c +3 -0
index e2d8a74f83c3..730b47bdcdef 100644
@@ -184,6 +184,9 @@ static int disable_ecrc_checking(struct pci_dev *dev)
  */
 void pcie_set_ecrc_checking(struct pci_dev *dev)
 {
+	if (!pcie_aer_is_native(dev))
+		return;
+
 	switch (ecrc_policy) {
 	case ECRC_POLICY_DEFAULT:
 		return;
Signature · 2 lines
Sathyanarayanan Kuppuswamy
Linux Kernel Developer
Signature · 2 lines
Sathyanarayanan Kuppuswamy
Linux Kernel Developer
#4 lore raw
Quoted 18 lines · Sathyanarayanan Kuppuswamy

External email: Use caution opening links or attachments

Hi,

Quoted 12 lines · Bjorn Helgaas
Quoted 8 lines · Sathyanarayanan Kuppuswamy
Quoted 2 lines · Vidya Sagar

As the ECRC configuration bits are part of AER registers, configure ECRC only if AER is natively owned by the kernel.

ecrc command line option takes "bios/on/off" as possible options. It does not clarify whether "on/off" choices can only be used if AER is owned by OS or it can override the ownership of ECRC configuration similar to pcie_ports=native option. Maybe that needs to be clarified.

Good point, what do you think of an update like this:

Quoted 15 lines
Quoted 5 lines
Documentation/admin-guide/kernel-parameters.txt +0 -0
index 6cfa6e3996cf..f7b40a439194 100644

Documentation/admin-guide/kernel-parameters.txt line 4304

Quoted 10 lines
Documentation/admin-guide/kernel-parameters.txt +3 -1
@@ -4296,7 +4296,9 @@
                               specified, e.g., 12@pci:8086:9c22:103c:198f
                               for 4096-byte alignment.
               ecrc=           Enable/disable PCIe ECRC (transaction layer
-                             end-to-end CRC checking).
+                             end-to-end CRC checking).  Only effective
+                             if OS has native AER control (either granted by
+                             ACPI _OSC or forced via "pcie_ports=native").
                               bios: Use BIOS/firmware settings. This is the
                               the default.
                               off: Turn ECRC off

I'm also fine with this change. I'll take it in V2.

Quoted 4 lines

Looks fine. But do we even need "bios" option? Since it is the default value, I am not sure why we need to list that as an option again. IMO this could be removed.

I think we still need bios option. For example, consider a system where BIOS needs to keep ECRC enabled for integrity reasons but if kernel doesn't want it for perf reasons, then, kernel can always use 'ecrc=off' option.

Quoted 34 lines
Quoted 6 lines

I don't know whether the "ecrc=" parameter is really needed. If we were adding it today, I would ask "why not enable ECRC wherever it is supported?" If there are devices where it's broken, we could always add quirks to disable it on a case-by-case basis.

Checking the original patch which added it, it looks like the intention is to give option to boost performance over integrity.

commit 43c16408842b0eeb367c23a6fa540ce69f99e347

Author: Andrew Patterson <[email protected]>
Date:   Wed Apr 22 16:52:09 2009 -0600

PCI: Add support for turning PCIe ECRC on or off

Adds support for PCI Express transaction layer end-to-end CRC checking (ECRC). This patch will enable/disable ECRC checking by setting/clearing the ECRC Check Enable and/or ECRC Generation Enable bits for devices that support ECRC.

The ECRC setting is controlled by the "pci=ecrc=<policy>" command-line option. If this option is not set or is set to 'bios", the enable and generation bits are left in whatever state that firmware/BIOS set them to. The "off" setting turns them off, and the "on" option turns them on (if the device supports it).

Turning ECRC on or off can be a data integrity versus performance tradeoff. In theory, turning it on will catch more data errors, turning it off means possibly better performance since CRC does not need to be calculated by the PCIe hardware and packet sizes are reduced.

Quoted 1 line
Quoted 8 lines
Quoted 1 line

But I think the patch below is the right thing to do for now. Vidya,

Agree.

Quoted 4 lines

did you trip over an issue because of this, e.g., a conflict between firmware use of AER and Linux use of it? If so, maybe we could mention a symptom on the commit log. But my guess is you probably found this by inspection.

Not really. I was just checking when does kernel touch ECRC settings and happened to find this where it configures ECRC irrespective of its ownership of AER registers.

Quoted 30 lines
Quoted 26 lines

Bjorn

Quoted 23 lines
Quoted 10 lines
---
  drivers/pci/pcie/aer.c | 3 +++
  1 file changed, 3 insertions(+)
drivers/pci/pcie/aer.c +0 -0
index e2d8a74f83c3..730b47bdcdef 100644

drivers/pci/pcie/aer.c static int disable_ecrc_checking(struct pci_dev *dev) · line 192

Quoted 9 lines
drivers/pci/pcie/aer.c +3 -0
@@ -184,6 +184,9 @@ static int disable_ecrc_checking(struct pci_dev *dev)
   */
  void pcie_set_ecrc_checking(struct pci_dev *dev)
  {
+   if (!pcie_aer_is_native(dev))
+           return;
+
     switch (ecrc_policy) {
     case ECRC_POLICY_DEFAULT:
             return;

--
Sathyanarayanan Kuppuswamy
Linux Kernel Developer

--
Sathyanarayanan Kuppuswamy
Linux Kernel Developer

#5 lore raw
Quoted 1 line · Vidya Sagar

I think we still need bios option. For example, consider a system where BIOS needs to keep ECRC enabled for integrity reasons but if kernel doesn't want it for perf reasons, then, kernel can always use 'ecrc=off' option.

I agree that "on" and "off" option makes sense. Since the kernel defaults ecrc setting to "bios", why again allow it as a command line option?

Signature · 2 lines
Sathyanarayanan Kuppuswamy
Linux Kernel Developer
#6 lore raw
Quoted 7 lines · Sathyanarayanan Kuppuswamy

External email: Use caution opening links or attachments

Quoted 1 line · Vidya Sagar

I think we still need bios option. For example, consider a system where BIOS needs to keep ECRC enabled for integrity reasons but if kernel doesn't want it for perf reasons, then, kernel can always use 'ecrc=off' option.

I agree that "on" and "off" option makes sense. Since the kernel defaults ecrc setting to "bios", why again allow it as a command line option?

Agree. "on" and "off" are fine but "default" is redundant. Do you want me to push a change to remove that as part of this patch itself? I think it is more like a cleanup and should go separately.

Quoted 4 lines

--
Sathyanarayanan Kuppuswamy
Linux Kernel Developer

#7 lore raw
Quoted 6 lines · Vidya Sagar
Quoted 3 lines · Sathyanarayanan Kuppuswamy

External email: Use caution opening links or attachments

Quoted 7 lines
Quoted 4 lines · Vidya Sagar
Quoted 1 line

I think we still need bios option. For example, consider a system where BIOS needs to keep ECRC enabled for integrity reasons but if kernel doesn't want it for perf reasons, then, kernel can always use 'ecrc=off' option.

I agree that "on" and "off" option makes sense. Since the kernel defaults ecrc setting to "bios", why again allow it as a command line option?

Agree. "on" and "off" are fine but "default" is redundant. Do you want me to push a change to remove that as part of this patch itself? I think it is more like a cleanup and should go separately.

IMO, the "bios" option cleanup and command line update from Bjorn can be in one patch, and your change could be a separate patch. But it is up to you and Bjorn.

Quoted 5 lines
Quoted 5 lines
Signature · 3 lines
-- 
Sathyanarayanan Kuppuswamy
Linux Kernel Developer
Signature · 2 lines
Sathyanarayanan Kuppuswamy
Linux Kernel Developer
#8 lore raw
Quoted 13 lines · Sathyanarayanan Kuppuswamy

External email: Use caution opening links or attachments

Quoted 9 lines · Vidya Sagar
Quoted 6 lines · Sathyanarayanan Kuppuswamy

External email: Use caution opening links or attachments

Quoted 1 line · Vidya Sagar

I think we still need bios option. For example, consider a system where BIOS needs to keep ECRC enabled for integrity reasons but if kernel doesn't want it for perf reasons, then, kernel can always use 'ecrc=off' option.

Quoted 7 lines
Quoted 4 lines
Quoted 1 line

I agree that "on" and "off" option makes sense. Since the kernel defaults ecrc setting to "bios", why again allow it as a command line option?

Agree. "on" and "off" are fine but "default" is redundant. Do you want me to push a change to remove that as part of this patch itself? I think it is more like a cleanup and should go separately.

IMO, the "bios" option cleanup and command line update from Bjorn can be in one patch, and your change could be a separate patch. But it is up to you and Bjorn.

I think Bjorn's command line suggestion should go along with my patch otherwise the ECRC control through command line doesn't work if OS doesn't own the AER. So, it helps to make it explicit that the 'ecrc=' option works only if either kernel has native AER control or 'pcie_ports' is set to 'native'

Quoted 10 lines
Quoted 6 lines
Quoted 6 lines

--
Sathyanarayanan Kuppuswamy
Linux Kernel Developer

--
Sathyanarayanan Kuppuswamy
Linux Kernel Developer

#9 lore raw
In reply to Sathyanarayanan Kuppuswamy
Quoted 24 lines · Sathyanarayanan Kuppuswamy
Quoted 23 lines · Bjorn Helgaas
Quoted 8 lines · Sathyanarayanan Kuppuswamy
Quoted 2 lines · Vidya Sagar

As the ECRC configuration bits are part of AER registers, configure ECRC only if AER is natively owned by the kernel.

ecrc command line option takes "bios/on/off" as possible options. It does not clarify whether "on/off" choices can only be used if AER is owned by OS or it can override the ownership of ECRC configuration similar to pcie_ports=native option. Maybe that needs to be clarified.

Good point, what do you think of an update like this:

Documentation/admin-guide/kernel-parameters.txt +2 -1
index 6cfa6e3996cf..f7b40a439194 100644
@@ -4296,7 +4296,9 @@
 				specified, e.g., 12@pci:8086:9c22:103c:198f
 				for 4096-byte alignment.
 		ecrc=		Enable/disable PCIe ECRC (transaction layer
-				end-to-end CRC checking).
+				end-to-end CRC checking).  Only effective
+				if OS has native AER control (either granted by
Quoted 8 lines

Documentation/admin-guide/kernel-parameters.txt line 4304

Quoted 4 lines
Documentation/admin-guide/kernel-parameters.txt +1 -0
+				ACPI _OSC or forced via "pcie_ports=native").
 				bios: Use BIOS/firmware settings. This is the
 				the default.
 				off: Turn ECRC off

Looks fine. But do we even need "bios" option? Since it is the default value, I am not sure why we need to list that as an option again. IMO this could be removed.

I agree, it seems pointless.

Quoted 25 lines
Quoted 4 lines

I don't know whether the "ecrc=" parameter is really needed. If we were adding it today, I would ask "why not enable ECRC wherever it is supported?" If there are devices where it's broken, we could always add quirks to disable it on a case-by-case basis.

Checking the original patch which added it, it looks like the intention is to give option to boost performance over integrity.

commit 43c16408842b0eeb367c23a6fa540ce69f99e347

Author: Andrew Patterson <[email protected]>
Date:   Wed Apr 22 16:52:09 2009 -0600

PCI: Add support for turning PCIe ECRC on or off

Adds support for PCI Express transaction layer end-to-end CRC checking (ECRC). This patch will enable/disable ECRC checking by setting/clearing the ECRC Check Enable and/or ECRC Generation Enable bits for devices that support ECRC.

The ECRC setting is controlled by the "pci=ecrc=<policy>" command-line option. If this option is not set or is set to 'bios", the enable and generation bits are left in whatever state that firmware/BIOS set them to. The "off" setting turns them off, and the "on" option turns them on (if the device supports it).

Quoted 4 lines

Turning ECRC on or off can be a data integrity versus performance tradeoff. In theory, turning it on will catch more data errors, turning it off means possibly better performance since CRC does not need to be calculated by the PCIe hardware and packet sizes are reduced.

Ah, right, and I think I was even part of the conversation when this was added :)

I'm not sure I would make the same choice today, though. IMHO it's kind of hard to defend choosing performance over data integrity.

If a platform really wants to sacrifice integrity for performance, it could retain control of AER, and after Vidya's patch, Linux will leave the ECRC configuration alone.

Straw-man: If Linux owns AER and ECRC is supported, enable ECRC by default. Retain "ecrc=off" to turn it off, but drop a note in dmesg and taint the kernel.

Bjorn

#10 lore raw

PATCH v2 1/1 PCI/AER: Configure ECRC only AER is native

In reply to Vidya Sagar

As the ECRC configuration bits are part of AER registers, configure ECRC only if AER is natively owned by the kernel.

---
v2:
* Updated kernel-parameters.txt document based on Bjorn's suggestion

 Documentation/admin-guide/kernel-parameters.txt | 4 +++-
 drivers/pci/pcie/aer.c                          | 3 +++
 2 files changed, 6 insertions(+), 1 deletion(-)
Documentation/admin-guide/kernel-parameters.txt +3 -1
index 426fa892d311..8f85a1230525 100644
@@ -4242,7 +4242,9 @@
 				specified, e.g., 12@pci:8086:9c22:103c:198f
 				for 4096-byte alignment.
 		ecrc=		Enable/disable PCIe ECRC (transaction layer
-				end-to-end CRC checking).
+				end-to-end CRC checking). Only effective if
+				OS has native AER control (either granted by
+				ACPI _OSC or forced via "pcie_ports=native")
 				bios: Use BIOS/firmware settings. This is the
 				the default.
 				off: Turn ECRC off
drivers/pci/pcie/aer.c +3 -0
index e2d8a74f83c3..730b47bdcdef 100644
@@ -184,6 +184,9 @@ static int disable_ecrc_checking(struct pci_dev *dev)
  */
 void pcie_set_ecrc_checking(struct pci_dev *dev)
 {
+	if (!pcie_aer_is_native(dev))
+		return;
+
 	switch (ecrc_policy) {
 	case ECRC_POLICY_DEFAULT:
 		return;
Signature · 1 line
2.17.1
#11 lore raw
Quoted 4 lines · Vidya Sagar

As the ECRC configuration bits are part of AER registers, configure ECRC only if AER is natively owned by the kernel.

Applied to pci/aer for v6.3, thanks!

Quoted 40 lines

---
v2:
* Updated kernel-parameters.txt document based on Bjorn's suggestion

 Documentation/admin-guide/kernel-parameters.txt | 4 +++-
 drivers/pci/pcie/aer.c                          | 3 +++
 2 files changed, 6 insertions(+), 1 deletion(-)
Documentation/admin-guide/kernel-parameters.txt +3 -1
index 426fa892d311..8f85a1230525 100644
@@ -4242,7 +4242,9 @@
 				specified, e.g., 12@pci:8086:9c22:103c:198f
 				for 4096-byte alignment.
 		ecrc=		Enable/disable PCIe ECRC (transaction layer
-				end-to-end CRC checking).
+				end-to-end CRC checking). Only effective if
+				OS has native AER control (either granted by
+				ACPI _OSC or forced via "pcie_ports=native")
 				bios: Use BIOS/firmware settings. This is the
 				the default.
 				off: Turn ECRC off
drivers/pci/pcie/aer.c +3 -0
index e2d8a74f83c3..730b47bdcdef 100644
@@ -184,6 +184,9 @@ static int disable_ecrc_checking(struct pci_dev *dev)
  */
 void pcie_set_ecrc_checking(struct pci_dev *dev)
 {
+	if (!pcie_aer_is_native(dev))
+		return;
+
 	switch (ecrc_policy) {
 	case ECRC_POLICY_DEFAULT:
 		return;
Signature · 1 line
2.17.1