Patch "PCI: Set Read Completion Boundary to 128 iff Root Port supports it (_HPX)" has been added to the 4.8-stable tree

PATCH [email protected] 3 participants 5 messages Dec 4, 2016 – Dec 5, 2016

Subsystems (2)

Stable branch PCI

lore

Share & tools

lore

[email protected]

author maintainer Applied

2016-12-04 11:52 UTC #0 lore raw

This is a note to let you know that I've just added the patch titled

PCI: Set Read Completion Boundary to 128 iff Root Port supports it (_HPX)

to the 4.8-stable tree which can be found at:
http://www.kernel.org/git/?p=linux/kernel/git/stable/stable-queue.git;a=summary

The filename of the patch is:
pci-set-read-completion-boundary-to-128-iff-root-port-supports-it-_hpx.patch
and it can be found in the queue-4.8 subdirectory.

If you, or anyone else, feels it should not be added to the stable tree, please let <[email protected]> know about it.

Quoted 1 line

From e42010d8207f9d15a605ceb8e321bcd9648071b0 Mon Sep 17 00:00:00 2001

From: Johannes Thumshirn <[email protected]>
Date: Wed, 23 Nov 2016 10:56:28 -0600
Subject: PCI: Set Read Completion Boundary to 128 iff Root Port supports it (_HPX)
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

From: Johannes Thumshirn <[email protected]>

commit e42010d8207f9d15a605ceb8e321bcd9648071b0 upstream.

Per PCIe spec r3.0, sec 2.3.1.1, the Read Completion Boundary (RCB) determines the naturally aligned address boundaries on which a Read Request may be serviced with multiple Completions:

- For a Root Complex, RCB is 64 bytes or 128 bytes This value is reported in the Link Control Register

Note: Bridges and Endpoints may implement a corresponding command bit which may be set by system software to indicate the RCB value for the Root Complex, allowing the Bridge/Endpoint to optimize its behavior when the Root Complex’s RCB is 128 bytes.

- For all other system elements, RCB is 128 bytes

Per sec 7.8.7, if a Root Port only supports a 64-byte RCB, the RCB of all downstream devices must be clear, indicating an RCB of 64 bytes. If the Root Port supports a 128-byte RCB, we may optionally set the RCB of downstream devices so they know they can generate larger Completions.

Some BIOSes supply an _HPX that tells us to set RCB, even though the Root Port doesn't have RCB set, which may lead to Malformed TLP errors if the Endpoint generates completions larger than the Root Port can handle.

The IBM x3850 X6 with BIOS version -[A8E120CUS-1.30]- 08/22/2016 supplies such an _HPX and a Mellanox MT27500 ConnectX-3 device fails to initialize:

mlx4_core 0000:41:00.0: command 0xfff timed out (go bit not cleared) mlx4_core 0000:41:00.0: device is going to be reset mlx4_core 0000:41:00.0: Failed to obtain HW semaphore, aborting mlx4_core 0000:41:00.0: Fail to reset HCA

  ------------[ cut here ]------------
  kernel BUG at drivers/net/ethernet/mellanox/mlx4/catas.c:193!

After 6cd33649fa83 ("PCI: Add pci_configure_device() during enumeration") and 7a1562d4f2d0 ("PCI: Apply _HPX Link Control settings to all devices with a link"), we apply _HPX settings to *all* devices, not just those hot-added after boot.

Before 7a1562d4f2d0, we didn't touch the Mellanox RCB, and the device worked. After 7a1562d4f2d0, we set its RCB to 128, and it failed.

Set the RCB to 128 iff the Root Port supports a 128-byte RCB. Otherwise, set RCB to 64 bytes. This effectively ignores what _HPX tells us about RCB.

Note that this change only affects _HPX handling. If we have no _HPX, this does nothing with RCB.

[bhelgaas: changelog, clear RCB if not set for Root Port]

Fixes 6cd33649fa83 ("PCI: Add pci_configure_device() during enumeration")

Fixes 7a1562d4f2d0 ("PCI: Apply _HPX Link Control settings to all devices with a link")

Link https://bugzilla.kernel.org/show_bug.cgi?id=187781

Tested-by Frank Danapfel <[email protected]>

Signed-off-by Johannes Thumshirn <[email protected]>

Signed-off-by Bjorn Helgaas <[email protected]>

Acked-by Myron Stowe <[email protected]>

Signed-off-by Greg Kroah-Hartman <[email protected]>

---
 drivers/pci/probe.c |   28 +++++++++++++++++++++++++++-
 1 file changed, 27 insertions(+), 1 deletion(-)

drivers/pci/probe.c +27 -1

@@ -1439,6 +1439,21 @@ static void program_hpp_type1(struct pci
 		dev_warn(&dev->dev, "PCI-X settings not supported\n");
 }
 
+static bool pcie_root_rcb_set(struct pci_dev *dev)
+{
+	struct pci_dev *rp = pcie_find_root_port(dev);
+	u16 lnkctl;
+
+	if (!rp)
+		return false;
+
+	pcie_capability_read_word(rp, PCI_EXP_LNKCTL, &lnkctl);
+	if (lnkctl & PCI_EXP_LNKCTL_RCB)
+		return true;
+
+	return false;
+}
+
 static void program_hpp_type2(struct pci_dev *dev, struct hpp_type2 *hpp)
 {
 	int pos;

@@ -1468,9 +1483,20 @@ static void program_hpp_type2(struct pci
 			~hpp->pci_exp_devctl_and, hpp->pci_exp_devctl_or);
 
 	/* Initialize Link Control Register */
-	if (pcie_cap_has_lnkctl(dev))
+	if (pcie_cap_has_lnkctl(dev)) {
+
+		/*
+		 * If the Root Port supports Read Completion Boundary of
+		 * 128, set RCB to 128.  Otherwise, clear it.
+		 */
+		hpp->pci_exp_lnkctl_and |= PCI_EXP_LNKCTL_RCB;
+		hpp->pci_exp_lnkctl_or &= ~PCI_EXP_LNKCTL_RCB;
+		if (pcie_root_rcb_set(dev))
+			hpp->pci_exp_lnkctl_or |= PCI_EXP_LNKCTL_RCB;
+
 		pcie_capability_clear_and_set_word(dev, PCI_EXP_LNKCTL,
 			~hpp->pci_exp_lnkctl_and, hpp->pci_exp_lnkctl_or);
+	}
 
 	/* Find Advanced Error Reporting Enhanced Capability */
 	pos = pci_find_ext_capability(dev, PCI_EXT_CAP_ID_ERR);

Patches currently in stable-queue which might be from [email protected] are

queue-4.8/scsi-libfc-fix-seconds_since_last_reset-miscalculation.patch queue-4.8/pci-set-read-completion-boundary-to-128-iff-root-port-supports-it-_hpx.patch

Stefan Lippers-Hollmann

2016-12-05 01:37 UTC #1 lore raw

Adding this commit

Quoted 23 lines · On 2016-12-04, [email protected] wrote:

This is a note to let you know that I've just added the patch titled
PCI: Set Read Completion Boundary to 128 iff Root Port supports it (_HPX)
to the 4.8-stable tree which can be found at:
http://www.kernel.org/git/?p=linux/kernel/git/stable/stable-queue.git;a=summary
The filename of the patch is:
pci-set-read-completion-boundary-to-128-iff-root-port-supports-it-_hpx.patch
and it can be found in the queue-4.8 subdirectory.
If you, or anyone else, feels it should not be added to the stable tree, please let <[email protected]> know about it.
From e42010d8207f9d15a605ceb8e321bcd9648071b0 Mon Sep 17 00:00:00 2001
From: Johannes Thumshirn <[email protected]>
Date: Wed, 23 Nov 2016 10:56:28 -0600
Subject: PCI: Set Read Completion Boundary to 128 iff Root Port supports it (_HPX)
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

Quoted 3 lines

From: Johannes Thumshirn <[email protected]>
commit e42010d8207f9d15a605ceb8e321bcd9648071b0 upstream.

breaks building kernel 4.8.12+queue-4.8 with

drivers/pci/probe.c: In function 'pcie_root_rcb_set':
drivers/pci/probe.c:1444:23: error: implicit declaration of function 'pcie_find_root_port' [-Werror=implicit-function-declaration]

  struct pci_dev *rp = pcie_find_root_port(dev);
                       ^~~~~~~~~~~~~~~~~~~

drivers/pci/probe.c:1444:23: warning: initialization makes pointer from integer without a cast [-Wint-conversion]

Regards
Stefan Lippers-Hollmann

Johannes Thumshirn

2016-12-05 07:55 UTC #2 lore raw

Quoted 32 lines · Stefan Lippers-Hollmann

Hi
Adding this commit
Quoted 23 lines · On 2016-12-04, [email protected] wrote:
This is a note to let you know that I've just added the patch titled
PCI: Set Read Completion Boundary to 128 iff Root Port supports it (_HPX)
to the 4.8-stable tree which can be found at:
http://www.kernel.org/git/?p=linux/kernel/git/stable/stable-queue.git;a=summary
The filename of the patch is:
pci-set-read-completion-boundary-to-128-iff-root-port-supports-it-_hpx.patch
and it can be found in the queue-4.8 subdirectory.
If you, or anyone else, feels it should not be added to the stable tree, please let <[email protected]> know about it.
From e42010d8207f9d15a605ceb8e321bcd9648071b0 Mon Sep 17 00:00:00 2001
From: Johannes Thumshirn <[email protected]>
Date: Wed, 23 Nov 2016 10:56:28 -0600
Subject: PCI: Set Read Completion Boundary to 128 iff Root Port supports it (_HPX)
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
Quoted 3 lines
From: Johannes Thumshirn <[email protected]>
commit e42010d8207f9d15a605ceb8e321bcd9648071b0 upstream.

Quoted 7 lines

breaks building kernel 4.8.12+queue-4.8 with
drivers/pci/probe.c: In function 'pcie_root_rcb_set':
drivers/pci/probe.c:1444:23: error: implicit declaration of function 'pcie_find_root_port' [-Werror=implicit-function-declaration]
  struct pci_dev *rp = pcie_find_root_port(dev);
                       ^~~~~~~~~~~~~~~~~~~
drivers/pci/probe.c:1444:23: warning: initialization makes pointer from integer without a cast [-Wint-conversion]

Hi,
you'll need comit e784930bd ('PCI: Export pcie_find_root_port') as well to have the definition of pcie_find_root_port() outside of aer_inject.

Byte,
Johannes

Signature · 6 lines

Johannes Thumshirn                                          Storage
[email protected]                                +49 911 74053 689
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nï¿½rnberg
GF: Felix Imendï¿½rffer, Jane Smithard, Graham Norton
HRB 21284 (AG Nï¿½rnberg)
Key fingerprint = EC38 9CAB C2C4 F25D 8600 D0D0 0393 969D 2D76 0850

Greg KH

author maintainer

2016-12-05 12:54 UTC #3 lore raw

Quoted 41 lines · Johannes Thumshirn

Quoted 32 lines · Stefan Lippers-Hollmann
Hi
Adding this commit
Quoted 23 lines · On 2016-12-04, [email protected] wrote:
This is a note to let you know that I've just added the patch titled
PCI: Set Read Completion Boundary to 128 iff Root Port supports it (_HPX)
to the 4.8-stable tree which can be found at:
http://www.kernel.org/git/?p=linux/kernel/git/stable/stable-queue.git;a=summary
The filename of the patch is:
pci-set-read-completion-boundary-to-128-iff-root-port-supports-it-_hpx.patch
and it can be found in the queue-4.8 subdirectory.
If you, or anyone else, feels it should not be added to the stable tree, please let <[email protected]> know about it.
From e42010d8207f9d15a605ceb8e321bcd9648071b0 Mon Sep 17 00:00:00 2001
From: Johannes Thumshirn <[email protected]>
Date: Wed, 23 Nov 2016 10:56:28 -0600
Subject: PCI: Set Read Completion Boundary to 128 iff Root Port supports it (_HPX)
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
Quoted 3 lines
From: Johannes Thumshirn <[email protected]>
commit e42010d8207f9d15a605ceb8e321bcd9648071b0 upstream.
Quoted 7 lines
breaks building kernel 4.8.12+queue-4.8 with
drivers/pci/probe.c: In function 'pcie_root_rcb_set':
drivers/pci/probe.c:1444:23: error: implicit declaration of function 'pcie_find_root_port' [-Werror=implicit-function-declaration]
  struct pci_dev *rp = pcie_find_root_port(dev);
                       ^~~~~~~~~~~~~~~~~~~
drivers/pci/probe.c:1444:23: warning: initialization makes pointer from integer without a cast [-Wint-conversion]

Quoted 3 lines

Hi,
you'll need comit e784930bd ('PCI: Export pcie_find_root_port') as well to have the definition of pcie_find_root_port() outside of aer_inject.

Thanks for the fix, now queued up.

greg k-h

Greg KH

author maintainer

2016-12-05 12:55 UTC #4 lore raw

In reply to Stefan Lippers-Hollmann

Quoted 32 lines · Stefan Lippers-Hollmann

Hi
Adding this commit
Quoted 23 lines · On 2016-12-04, [email protected] wrote:
This is a note to let you know that I've just added the patch titled
PCI: Set Read Completion Boundary to 128 iff Root Port supports it (_HPX)
to the 4.8-stable tree which can be found at:
http://www.kernel.org/git/?p=linux/kernel/git/stable/stable-queue.git;a=summary
The filename of the patch is:
pci-set-read-completion-boundary-to-128-iff-root-port-supports-it-_hpx.patch
and it can be found in the queue-4.8 subdirectory.
If you, or anyone else, feels it should not be added to the stable tree, please let <[email protected]> know about it.
From e42010d8207f9d15a605ceb8e321bcd9648071b0 Mon Sep 17 00:00:00 2001
From: Johannes Thumshirn <[email protected]>
Date: Wed, 23 Nov 2016 10:56:28 -0600
Subject: PCI: Set Read Completion Boundary to 128 iff Root Port supports it (_HPX)
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
Quoted 3 lines
From: Johannes Thumshirn <[email protected]>
commit e42010d8207f9d15a605ceb8e321bcd9648071b0 upstream.

Quoted 7 lines

breaks building kernel 4.8.12+queue-4.8 with
drivers/pci/probe.c: In function 'pcie_root_rcb_set':
drivers/pci/probe.c:1444:23: error: implicit declaration of function 'pcie_find_root_port' [-Werror=implicit-function-declaration]
  struct pci_dev *rp = pcie_find_root_port(dev);
                       ^~~~~~~~~~~~~~~~~~~
drivers/pci/probe.c:1444:23: warning: initialization makes pointer from integer without a cast [-Wint-conversion]

Thanks for the notification, my build tests failed but I was trying to ignore them and enjoy a Sunday off :)

Should be fixed soon...

greg k-h