Skip to contentSkip to conversation

PCI: Clear errors logged in Secondary Status Register

PATCH Superseded Vidya Sagar 2 participants 10 messages Jan 4, 2024 – Apr 23, 2024 PCI

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 PCI: Clear errors logged in Secondary Status Register

If a downstream port has a PCIe switch connected to it, the enumeration process leaves the 'Received Master Abort' bit set in the Secondary Status Register of the downstream port because of the Unsupported Requests (URs) take place in the downstream hierarchy. Since the ownership of Secondary Status Register always lies with the OS including systems with Firmware-First approach for error handling[1], clear the error status bits in the Secondary Status Register post enumeration.

[1] https://lore.kernel.org/all/[email protected]/T/

---
 drivers/pci/probe.c | 3 +++
 1 file changed, 3 insertions(+)
drivers/pci/probe.c +3 -0
index 43159965e09e..edf8202465d8 100644
@@ -1470,6 +1470,9 @@ static int pci_scan_bridge_extend(struct pci_bus *bus, struct pci_dev *dev,
 	}
 
 out:
+	/* Clear errors in the Secondary Status Register */
+	pci_write_config_word(dev, PCI_SEC_STATUS, 0xffff);
+
 	pci_write_config_word(dev, PCI_BRIDGE_CONTROL, bctl);
 
 	pm_runtime_put(&dev->dev);
Signature · 1 line
2.25.1
#1 lore raw

Hi Bjorn,
Do you have any comments for this patch?

Thanks,
Vidya Sagar

Quoted 29 lines · Vidya Sagar

If a downstream port has a PCIe switch connected to it, the enumeration process leaves the 'Received Master Abort' bit set in the Secondary Status Register of the downstream port because of the Unsupported Requests (URs) take place in the downstream hierarchy. Since the ownership of Secondary Status Register always lies with the OS including systems with Firmware-First approach for error handling[1], clear the error status bits in the Secondary Status Register post enumeration.

[1] https://lore.kernel.org/all/[email protected]/T/

---
  drivers/pci/probe.c | 3 +++
  1 file changed, 3 insertions(+)
drivers/pci/probe.c +3 -0
index 43159965e09e..edf8202465d8 100644
@@ -1470,6 +1470,9 @@ static int pci_scan_bridge_extend(struct pci_bus *bus, struct pci_dev *dev,
  	}
 
  out:
+	/* Clear errors in the Secondary Status Register */
+	pci_write_config_word(dev, PCI_SEC_STATUS, 0xffff);
+
  	pci_write_config_word(dev, PCI_BRIDGE_CONTROL, bctl);
 
  	pm_runtime_put(&dev->dev);
#2 lore raw
In reply to Vidya Sagar
Quoted 7 lines · Vidya Sagar

If a downstream port has a PCIe switch connected to it, the enumeration process leaves the 'Received Master Abort' bit set in the Secondary Status Register of the downstream port because of the Unsupported Requests (URs) take place in the downstream hierarchy. Since the ownership of Secondary Status Register always lies with the OS including systems with Firmware-First approach for error handling[1], clear the error status bits in the Secondary Status Register post enumeration.

I would expect these URs to happen when enumerating below *all* PCIe Root Ports (not just when switches are present), and Master Aborts should happen in conventional PCI.

Similarly, I don't think Firmware-First is relevant here. Only the fact that the OS owns PCI_SEC_STATUS because there's no mechanism to negotiate for platform ownership of it.

We're in the merge window right now, so we'll start merging v6.9 material after v6.8-rc1 is tagged.

Quoted 24 lines

[1] https://lore.kernel.org/all/[email protected]/T/

---
 drivers/pci/probe.c | 3 +++
 1 file changed, 3 insertions(+)
drivers/pci/probe.c +3 -0
index 43159965e09e..edf8202465d8 100644
@@ -1470,6 +1470,9 @@ static int pci_scan_bridge_extend(struct pci_bus *bus, struct pci_dev *dev,
 	}
 
 out:
+	/* Clear errors in the Secondary Status Register */
+	pci_write_config_word(dev, PCI_SEC_STATUS, 0xffff);
+
 	pci_write_config_word(dev, PCI_BRIDGE_CONTROL, bctl);
 
 	pm_runtime_put(&dev->dev);
Signature · 1 line
2.25.1
#3 lore raw
Quoted 12 lines · Bjorn Helgaas

External email: Use caution opening links or attachments

Quoted 7 lines · Vidya Sagar

If a downstream port has a PCIe switch connected to it, the enumeration process leaves the 'Received Master Abort' bit set in the Secondary Status Register of the downstream port because of the Unsupported Requests (URs) take place in the downstream hierarchy. Since the ownership of Secondary Status Register always lies with the OS including systems with Firmware-First approach for error handling[1], clear the error status bits in the Secondary Status Register post enumeration.

Quoted 3 lines

I would expect these URs to happen when enumerating below *all* PCIe Root Ports (not just when switches are present), and Master Aborts should happen in conventional PCI.

Agree.
There was a misunderstanding from my side because of which I had said that the 'Received Master Abort' bit gets set only if there is a PCIe switch connected downstream. I'll correct it in my next patch.

Quoted 4 lines

Similarly, I don't think Firmware-First is relevant here. Only the fact that the OS owns PCI_SEC_STATUS because there's no mechanism to negotiate for platform ownership of it.

I mentioned about Firmware-First as a continuation to the discussion we had in [1]. But, agree that, this being a standalone patch, there is no need to mentioned about Firmware-First flow.

Quoted 28 lines

We're in the merge window right now, so we'll start merging v6.9 material after v6.8-rc1 is tagged.

Quoted 24 lines

[1] https://lore.kernel.org/all/[email protected]/T/

---
  drivers/pci/probe.c | 3 +++
  1 file changed, 3 insertions(+)
drivers/pci/probe.c +3 -1
index 43159965e09e..edf8202465d8 100644
@@ -1470,6 +1470,9 @@ static int pci_scan_bridge_extend(struct pci_bus *bus, struct pci_dev *dev,
       }
 
  out:
+     /* Clear errors in the Secondary Status Register */
+     pci_write_config_word(dev, PCI_SEC_STATUS, 0xffff);
+
       pci_write_config_word(dev, PCI_BRIDGE_CONTROL, bctl);
 
       pm_runtime_put(&dev->dev);
--

2.25.1

#4 lore raw

PATCH v2 1/1 PCI: Clear errors logged in Secondary Status Register

In reply to Vidya Sagar

The enumeration process leaves the 'Received Master Abort' bit set in the Secondary Status Register of the downstream port in the following scenarios.

(1) The device connected to the downstream port has ARI capability and that makes the kernel set the 'ARI Forwarding Enable' bit in the Device Control 2 Register of the downstream port. This effectively makes the downstream port forward the configuration requests targeting the devices downstream of it, even though they don't exist in reality. It causes the downstream devices return completions with UR set in the status in turn causing 'Received Master Abort' bit set.

In contrast, if the downstream device doesn't have ARI capability, the 'ARI Forwarding Enable' bit in the downstream port is not set and any configuration requests targeting the downstream devices that don't exist are terminated (section 6.13 of PCI Express Base 6.0 spec) in the downstream port itself resulting in no change of the 'Received Master Abort' bit.

(2) A PCIe switch is connected to the downstream port and when the enumeration flow tries to explore the presence of devices that don't really exist downstream of the switch, the downstream port receives the completions with UR set causing the 'Received Master Abort' bit set.

Clear 'Received Master Abort' bit to keep the bridge device in a clean state post enumeration.

---
V2:
* Changed commit message based on Bjorn's feedback

 drivers/pci/probe.c | 3 +++
 1 file changed, 3 insertions(+)
drivers/pci/probe.c +3 -0
index 795534589b98..640d2871b061 100644
@@ -1470,6 +1470,9 @@ static int pci_scan_bridge_extend(struct pci_bus *bus, struct pci_dev *dev,
 	}
 
 out:
+	/* Clear errors in the Secondary Status Register */
+	pci_write_config_word(dev, PCI_SEC_STATUS, 0xffff);
+
 	pci_write_config_word(dev, PCI_BRIDGE_CONTROL, bctl);
 
 	pm_runtime_put(&dev->dev);
Signature · 1 line
2.25.1
#5 lore raw
Quoted 20 lines · Vidya Sagar

The enumeration process leaves the 'Received Master Abort' bit set in the Secondary Status Register of the downstream port in the following scenarios.

(1) The device connected to the downstream port has ARI capability and that makes the kernel set the 'ARI Forwarding Enable' bit in the Device Control 2 Register of the downstream port. This effectively makes the downstream port forward the configuration requests targeting the devices downstream of it, even though they don't exist in reality. It causes the downstream devices return completions with UR set in the status in turn causing 'Received Master Abort' bit set.

In contrast, if the downstream device doesn't have ARI capability, the 'ARI Forwarding Enable' bit in the downstream port is not set and any configuration requests targeting the downstream devices that don't exist are terminated (section 6.13 of PCI Express Base 6.0 spec) in the downstream port itself resulting in no change of the 'Received Master Abort' bit.

Quoted 5 lines

(2) A PCIe switch is connected to the downstream port and when the enumeration flow tries to explore the presence of devices that don't really exist downstream of the switch, the downstream port receives the completions with UR set causing the 'Received Master Abort' bit set.

Are these the only possible ways this error is logged? I expected them to be logged when we enumerate below a Root Port that has nothing attached, for example.

Does clearing them in pci_scan_bridge_extend() cover all ways this error might be logged during enumeration? I can't remember whether all enumeration goes through this path.

Quoted 28 lines

Clear 'Received Master Abort' bit to keep the bridge device in a clean state post enumeration.

---
V2:
* Changed commit message based on Bjorn's feedback

 drivers/pci/probe.c | 3 +++
 1 file changed, 3 insertions(+)
drivers/pci/probe.c +3 -0
index 795534589b98..640d2871b061 100644
@@ -1470,6 +1470,9 @@ static int pci_scan_bridge_extend(struct pci_bus *bus, struct pci_dev *dev,
 	}
 
 out:
+	/* Clear errors in the Secondary Status Register */
+	pci_write_config_word(dev, PCI_SEC_STATUS, 0xffff);
+
 	pci_write_config_word(dev, PCI_BRIDGE_CONTROL, bctl);
 
 	pm_runtime_put(&dev->dev);
Signature · 1 line
2.25.1
#6 lore raw
Quoted 24 lines · Bjorn Helgaas

External email: Use caution opening links or attachments

Quoted 20 lines · Vidya Sagar

The enumeration process leaves the 'Received Master Abort' bit set in the Secondary Status Register of the downstream port in the following scenarios.

(1) The device connected to the downstream port has ARI capability and that makes the kernel set the 'ARI Forwarding Enable' bit in the Device Control 2 Register of the downstream port. This effectively makes the downstream port forward the configuration requests targeting the devices downstream of it, even though they don't exist in reality. It causes the downstream devices return completions with UR set in the status in turn causing 'Received Master Abort' bit set.

In contrast, if the downstream device doesn't have ARI capability, the 'ARI Forwarding Enable' bit in the downstream port is not set and any configuration requests targeting the downstream devices that don't exist are terminated (section 6.13 of PCI Express Base 6.0 spec) in the downstream port itself resulting in no change of the 'Received Master Abort' bit.

Quoted 8 lines
Quoted 5 lines

(2) A PCIe switch is connected to the downstream port and when the enumeration flow tries to explore the presence of devices that don't really exist downstream of the switch, the downstream port receives the completions with UR set causing the 'Received Master Abort' bit set.

Are these the only possible ways this error is logged? I expected them to be logged when we enumerate below a Root Port that has nothing attached, for example.

In this case, there won't be any TLP sent downstream. I talked about this scenario in the second paragraph of point (1) above.

Quoted 3 lines

Does clearing them in pci_scan_bridge_extend() cover all ways this error might be logged during enumeration? I can't remember whether all enumeration goes through this path.

So far in my testing, clearing it in pci_scan_bridge_extend() covers all the cases.

Quoted 28 lines
Quoted 28 lines

Clear 'Received Master Abort' bit to keep the bridge device in a clean state post enumeration.

---
V2:
* Changed commit message based on Bjorn's feedback

  drivers/pci/probe.c | 3 +++
  1 file changed, 3 insertions(+)
drivers/pci/probe.c +3 -1
index 795534589b98..640d2871b061 100644
@@ -1470,6 +1470,9 @@ static int pci_scan_bridge_extend(struct pci_bus *bus, struct pci_dev *dev,
       }
 
  out:
+     /* Clear errors in the Secondary Status Register */
+     pci_write_config_word(dev, PCI_SEC_STATUS, 0xffff);
+
       pci_write_config_word(dev, PCI_BRIDGE_CONTROL, bctl);
 
       pm_runtime_put(&dev->dev);
--

2.25.1

#7 lore raw

Hi Bjorn,
Just checking on this thread.
Is there anything else you want me to clarify on?

Thanks,
Vidya Sagar

Quoted 72 lines · Vidya Sagar
Quoted 24 lines · Bjorn Helgaas

External email: Use caution opening links or attachments

Quoted 20 lines · Vidya Sagar

The enumeration process leaves the 'Received Master Abort' bit set in the Secondary Status Register of the downstream port in the following scenarios.

(1) The device connected to the downstream port has ARI capability and that makes the kernel set the 'ARI Forwarding Enable' bit in the Device Control 2 Register of the downstream port. This effectively makes the downstream port forward the configuration requests targeting the devices downstream of it, even though they don't exist in reality. It causes the downstream devices return completions with UR set in the status in turn causing 'Received Master Abort' bit set.

     In contrast, if the downstream device doesn't have ARI capability, the 'ARI Forwarding Enable' bit in the downstream port is not set and any configuration requests targeting the downstream devices that don't exist are terminated (section 6.13 of PCI Express Base 6.0 spec) in the downstream port itself resulting in no change of the 'Received Master Abort' bit.

Quoted 8 lines
Quoted 5 lines

(2) A PCIe switch is connected to the downstream port and when the enumeration flow tries to explore the presence of devices that don't really exist downstream of the switch, the downstream port receives the completions with UR set causing the 'Received Master Abort' bit set.

Are these the only possible ways this error is logged?  I expected them to be logged when we enumerate below a Root Port that has nothing attached, for example.

In this case, there won't be any TLP sent downstream. I talked about this scenario in the second paragraph of point (1) above.

Quoted 3 lines

Does clearing them in pci_scan_bridge_extend() cover all ways this error might be logged during enumeration?  I can't remember whether all enumeration goes through this path.

So far in my testing, clearing it in pci_scan_bridge_extend() covers all the cases.

Quoted 29 lines
Quoted 29 lines

Clear 'Received Master Abort' bit to keep the bridge device in a clean state post enumeration.

---
V2:
* Changed commit message based on Bjorn's feedback

  drivers/pci/probe.c | 3 +++

  1 file changed, 3 insertions(+)
drivers/pci/probe.c +0 -0
index 795534589b98..640d2871b061 100644
@@ -1470,6 +1470,9 @@ static int pci_scan_bridge_extend(struct 
 pci_bus *bus, struct pci_dev *dev,
        }
 

  out:
+     /* Clear errors in the Secondary Status Register */ +     pci_write_config_word(dev, PCI_SEC_STATUS, 0xffff); +
       pci_write_config_word(dev, PCI_BRIDGE_CONTROL, bctl);

       pm_runtime_put(&dev->dev);

Signature · 1 line
2.25.1
#8 lore raw

Hi Bjorn,
Sorry to bug you.
Is this change good to be accepted?

Thanks,
Vidya Sagar

Quoted 81 lines · Vidya Sagar

Hi Bjorn,
Just checking on this thread.
Is there anything else you want me to clarify on?

Thanks,
Vidya Sagar

Quoted 72 lines · Vidya Sagar
Quoted 24 lines · Bjorn Helgaas

External email: Use caution opening links or attachments

Quoted 20 lines · Vidya Sagar

The enumeration process leaves the 'Received Master Abort' bit set in the Secondary Status Register of the downstream port in the following scenarios.

(1) The device connected to the downstream port has ARI capability and that makes the kernel set the 'ARI Forwarding Enable' bit in the Device Control 2 Register of the downstream port. This effectively makes the downstream port forward the configuration requests targeting the devices downstream of it, even though they don't exist in reality. It causes the downstream devices return completions with UR set in the status in turn causing 'Received Master Abort' bit set.

     In contrast, if the downstream device doesn't have ARI capability, the 'ARI Forwarding Enable' bit in the downstream port is not set and any configuration requests targeting the downstream devices that don't exist are terminated (section 6.13 of PCI Express Base 6.0 spec) in the downstream port itself resulting in no change of the 'Received Master Abort' bit.

Quoted 8 lines
Quoted 5 lines

(2) A PCIe switch is connected to the downstream port and when the enumeration flow tries to explore the presence of devices that don't really exist downstream of the switch, the downstream port receives the completions with UR set causing the 'Received Master Abort' bit set.

Are these the only possible ways this error is logged?  I expected them to be logged when we enumerate below a Root Port that has nothing attached, for example.

In this case, there won't be any TLP sent downstream. I talked about this scenario in the second paragraph of point (1) above.

Quoted 3 lines

Does clearing them in pci_scan_bridge_extend() cover all ways this error might be logged during enumeration?  I can't remember whether all enumeration goes through this path.

So far in my testing, clearing it in pci_scan_bridge_extend() covers all the cases.

Quoted 29 lines
Quoted 29 lines

Clear 'Received Master Abort' bit to keep the bridge device in a clean state post enumeration.

---
V2:
* Changed commit message based on Bjorn's feedback

  drivers/pci/probe.c | 3 +++

  1 file changed, 3 insertions(+)
drivers/pci/probe.c +0 -0
index 795534589b98..640d2871b061 100644
@@ -1470,6 +1470,9 @@ static int pci_scan_bridge_extend(struct pci_bus 
 *bus, struct pci_dev *dev,
        }
 

  out:
+     /* Clear errors in the Secondary Status Register */ +     pci_write_config_word(dev, PCI_SEC_STATUS, 0xffff); +
       pci_write_config_word(dev, PCI_BRIDGE_CONTROL, bctl);

       pm_runtime_put(&dev->dev);

Signature · 1 line
2.25.1
#9 lore raw
In reply to Vidya Sagar
Quoted 26 lines · Vidya Sagar

The enumeration process leaves the 'Received Master Abort' bit set in the Secondary Status Register of the downstream port in the following scenarios.

(1) The device connected to the downstream port has ARI capability and that makes the kernel set the 'ARI Forwarding Enable' bit in the Device Control 2 Register of the downstream port. This effectively makes the downstream port forward the configuration requests targeting the devices downstream of it, even though they don't exist in reality. It causes the downstream devices return completions with UR set in the status in turn causing 'Received Master Abort' bit set.

In contrast, if the downstream device doesn't have ARI capability, the 'ARI Forwarding Enable' bit in the downstream port is not set and any configuration requests targeting the downstream devices that don't exist are terminated (section 6.13 of PCI Express Base 6.0 spec) in the downstream port itself resulting in no change of the 'Received Master Abort' bit.

(2) A PCIe switch is connected to the downstream port and when the enumeration flow tries to explore the presence of devices that don't really exist downstream of the switch, the downstream port receives the completions with UR set causing the 'Received Master Abort' bit set.

Quoted 4 lines

Clear 'Received Master Abort' bit to keep the bridge device in a clean state post enumeration.

Applied to pci/enumeration for v6.10, thanks!

I shortened the commit log because I think this happens all the time, not just in the specific cases you mentioned above:

PCI: Clear Secondary Status errors after enumeration

We enumerate devices by attempting config reads to the Vendor ID of each possible device. On conventional PCI, if no device responds, the read terminates with a Master Abort (PCI r3.0, sec 6.1). On PCIe, the config read is terminated as an Unsupported Request (PCIe r6.0, sec 2.3.2, 7.5.1.3.7). In either case, if the read addressed a device below a bridge, it is logged by setting "Received Master Abort" in the bridge Secondary Status register.

Clear any errors logged in the Secondary Status register after enumeration.

Quoted 24 lines

---
V2:
* Changed commit message based on Bjorn's feedback

 drivers/pci/probe.c | 3 +++
 1 file changed, 3 insertions(+)
drivers/pci/probe.c +3 -0
index 795534589b98..640d2871b061 100644
@@ -1470,6 +1470,9 @@ static int pci_scan_bridge_extend(struct pci_bus *bus, struct pci_dev *dev,
 	}
 
 out:
+	/* Clear errors in the Secondary Status Register */
+	pci_write_config_word(dev, PCI_SEC_STATUS, 0xffff);
+
 	pci_write_config_word(dev, PCI_BRIDGE_CONTROL, bctl);
 
 	pm_runtime_put(&dev->dev);
Signature · 1 line
2.25.1