Skip to contentSkip to conversation

PCI: Check for PCIe downtraining conditions

PATCH Superseded Alexandru Gagniuc 3 participants 7 messages Jun 1, 2018 PCI

v2 of 6 Superseded Latest is v6 →

lore
Share & tools
lore
#0 lore raw

PATCH v2 1/1 PCI: Check for PCIe downtraining conditions

PCIe downtraining happens when both the device and PCIe port are capable of a larger bus width or higher speed than negotiated. Downtraining might be indicative of other problems in the system, and identifying this from userspace is neither intuitive, nor straigh forward.

The easiest way to detect this is with pcie_print_link_status(), since the bottleneck is usually the link that is downtrained. It's not a perfect solution, but it works extremely well in most cases.

---
 drivers/pci/probe.c | 22 ++++++++++++++++++++++
 1 file changed, 22 insertions(+)
drivers/pci/probe.c +22 -0
index ac91b6fd0bcd..e8e158046cab 100644
@@ -2146,6 +2146,25 @@ static struct pci_dev *pci_scan_device(struct pci_bus *bus, int devfn)
 	return dev;
 }
 
+static void pcie_check_upstream_link(struct pci_dev *dev)
+{
+
+	if (!pci_is_pcie(dev))
+		return;
+
+	/* Look from the device up to avoid downstream ports with no devices. */
+	if ((pci_pcie_type(dev) != PCI_EXP_TYPE_ENDPOINT) &&
+	    (pci_pcie_type(dev) != PCI_EXP_TYPE_LEG_END) &&
+	    (pci_pcie_type(dev) != PCI_EXP_TYPE_UPSTREAM))
+		return;
+
+	/* Multi-function PCIe share the same link/status. */
+	if (PCI_FUNC(dev->devfn) != 0)
+		return;
+
+	pcie_print_link_status(dev);
+}
+
 static void pci_init_capabilities(struct pci_dev *dev)
 {
 	/* Enhanced Allocation */
@@ -2181,6 +2200,9 @@ static void pci_init_capabilities(struct pci_dev *dev)
 	/* Advanced Error Reporting */
 	pci_aer_init(dev);
 
+	/* Check link and detect downtrain errors */
+	pcie_check_upstream_link(dev);
+
 	if (pci_probe_reset_function(dev) == 0)
 		dev->reset_fn = 1;
 }
Signature · 1 line
2.14.3
#1 lore raw

drivers/pci/probe.c static struct pci_dev *pci_scan_device(struct pci_bus *bus, int devfn) · line 2163

Quoted 3 lines · Alexandru Gagniuc
drivers/pci/probe.c +3 -0
+	/* Multi-function PCIe share the same link/status. */
+	if (PCI_FUNC(dev->devfn) != 0)
+		return;

How about virtual functions?

Signature · 3 lines
Sinan Kaya
Qualcomm Datacenter Technologies, Inc. as an affiliate of Qualcomm Technologies, Inc.
Qualcomm Technologies, Inc. is a member of the Code Aurora Forum, a Linux Foundation Collaborative Project.
author
#2 lore raw
Quoted 6 lines · Sinan Kaya

drivers/pci/probe.c static struct pci_dev *pci_scan_device(struct pci_bus *bus, int devfn) · line 2163

Quoted 3 lines · Alexandru Gagniuc
drivers/pci/probe.c +3 -0
+	/* Multi-function PCIe share the same link/status. */
+	if (PCI_FUNC(dev->devfn) != 0)
+		return;

How about virtual functions?

I have almost no clue about those. Is your concern that we might end up printing more than our fair share of link statuses?

Alex

#3 lore raw
Quoted 10 lines · Alex G.
Quoted 6 lines · Sinan Kaya

drivers/pci/probe.c static struct pci_dev *pci_scan_device(struct pci_bus *bus, int devfn) · line 2163

Quoted 3 lines · Alexandru Gagniuc
drivers/pci/probe.c +3 -0
+	/* Multi-function PCIe share the same link/status. */
+	if (PCI_FUNC(dev->devfn) != 0)
+		return;

How about virtual functions?

I have almost no clue about those. Is your concern that we might end up printing more than our fair share of link statuses?

Yes, struct pci_dev also has a flag called is_virtfn that you should bail out here too.

Quoted 4 lines

Alex

Signature · 3 lines
Sinan Kaya
Qualcomm Datacenter Technologies, Inc. as an affiliate of Qualcomm Technologies, Inc.
Qualcomm Technologies, Inc. is a member of the Code Aurora Forum, a Linux Foundation Collaborative Project.
author
#4 lore raw
Quoted 9 lines · Sinan Kaya
Quoted 8 lines · Alex G.
Quoted 6 lines · Sinan Kaya

drivers/pci/probe.c static struct pci_dev *pci_scan_device(struct pci_bus *bus, int devfn) · line 2163

Quoted 3 lines · Alexandru Gagniuc
drivers/pci/probe.c +3 -0
+	/* Multi-function PCIe share the same link/status. */
+	if (PCI_FUNC(dev->devfn) != 0)
+		return;

How about virtual functions?

Quoted 5 lines
Quoted 2 lines

I have almost no clue about those. Is your concern that we might end up printing more than our fair share of link statuses?

Yes, struct pci_dev also has a flag called is_virtfn that you should bail out here too.

Will be fixed in v3.

Thanks,
Alex

Quoted 6 lines
Quoted 4 lines

Alex

#5 lore raw
In reply to Alexandru Gagniuc
Quoted 10 lines · Alexandru Gagniuc

PCIe downtraining happens when both the device and PCIe port are capable of a larger bus width or higher speed than negotiated. Downtraining might be indicative of other problems in the system, and identifying this from userspace is neither intuitive, nor straigh forward.

The easiest way to detect this is with pcie_print_link_status(), since the bottleneck is usually the link that is downtrained. It's not a perfect solution, but it works extremely well in most cases.

drivers/pci/probe.c static struct pci_dev *pci_scan_device(struct pci_bus *bus, int devfn) · line 2151

Quoted 4 lines
drivers/pci/probe.c +3 -0
+static void pcie_check_upstream_link(struct pci_dev *dev)
+{
 
+

This is redundant, but...

drivers/pci/probe.c static struct pci_dev *pci_scan_device(struct pci_bus *bus, int devfn) · line 2159

Quoted 8 lines
drivers/pci/probe.c +8 -0
+       if (!pci_is_pcie(dev))
+               return;
+
+       /* Look from the device up to avoid downstream ports with no devices. */
+       if ((pci_pcie_type(dev) != PCI_EXP_TYPE_ENDPOINT) &&
+           (pci_pcie_type(dev) != PCI_EXP_TYPE_LEG_END) &&
+           (pci_pcie_type(dev) != PCI_EXP_TYPE_UPSTREAM))
+               return;

...wouldn't be better

int type = pci_pcie_type(dev);

?

But also possible, looking at existing code,

static inline bool pci_is_pcie_type(dev, type)
{
return pci_is_pcie(dev) ? pci_pcie_type(dev) == type : false;
}

Signature · 2 lines
With Best Regards,
Andy Shevchenko
author
#6 lore raw
Quoted 11 lines · Andy Shevchenko
Quoted 9 lines · Alexandru Gagniuc

PCIe downtraining happens when both the device and PCIe port are capable of a larger bus width or higher speed than negotiated. Downtraining might be indicative of other problems in the system, and identifying this from userspace is neither intuitive, nor straigh forward.

The easiest way to detect this is with pcie_print_link_status(), since the bottleneck is usually the link that is downtrained. It's not a perfect solution, but it works extremely well in most cases.

Quoted 6 lines

drivers/pci/probe.c static struct pci_dev *pci_scan_device(struct pci_bus *bus, int devfn) · line 2151

Quoted 4 lines
drivers/pci/probe.c +3 -0
+static void pcie_check_upstream_link(struct pci_dev *dev)
+{
 
+

This is redundant, but...

Hmm. I thought it's not safe to call pci_pcie_type() on non-pcie devices.

I see the pci_is_pcie() check followed by pci_pcie_type() is not uncommon. I didn't think it would be an issue, as long as it's consistent with the rest of the code.

Quoted 9 lines

drivers/pci/probe.c static struct pci_dev *pci_scan_device(struct pci_bus *bus, int devfn) · line 2159

Quoted 8 lines
drivers/pci/probe.c +8 -0
+       if (!pci_is_pcie(dev))
+               return;
+
+       /* Look from the device up to avoid downstream ports with no devices. */
+       if ((pci_pcie_type(dev) != PCI_EXP_TYPE_ENDPOINT) &&
+           (pci_pcie_type(dev) != PCI_EXP_TYPE_LEG_END) &&
+           (pci_pcie_type(dev) != PCI_EXP_TYPE_UPSTREAM))
+               return;
Quoted 5 lines

...wouldn't be better

int type = pci_pcie_type(dev);

?

An extra local variable when the compiler knows how to optimize it out? To me, it doesn't seem like it would improve readability, but it would make the code longer.

Quoted 6 lines

But also possible, looking at existing code,

static inline bool pci_is_pcie_type(dev, type)
{
return pci_is_pcie(dev) ? pci_pcie_type(dev) == type : false;
}

	return pci_is_pcie(dev) && (pci_pcie_type(dev) == type);

seems cleaner. Although this sort of cleanup is beyond the scope of this change.

Thanks,
Alex