Skip to contentSkip to conversation

PCI: PCI-E MPS bug fixes

PATCH In review Jon Mason 4 participants 8 messages Sep 8, 2011 – Sep 13, 2011 PCI

lore
Share & tools
lore

Jon Mason

author
#0 lore raw

cover PCI: PCI-E MPS bug fixes

Please include these bug fixes to PCI in 3.1-rc6

#1 lore raw

Jesse,
what's the status of this? It does seem like our current PCIe
behavior is rather unsafe.

Should I apply the patches, should I wait for a pull request from you, should I ignore them because of xyz?

Linus

Quoted 3 lines · Jon Mason

Please include these bug fixes to PCI in 3.1-rc6

#2 lore raw
Quoted 4 lines · Linus Torvalds

Jesse,
what's the status of this? It does seem like our current PCIe
behavior is rather unsafe.

Quoted 9 lines

Should I apply the patches, should I wait for a pull request from you, should I ignore them because of xyz?

Linus

Quoted 3 lines · Jon Mason

Please include these bug fixes to PCI in 3.1-rc6

You should apply them. I would queue them, and but I don't have another PCI tree anywhere for you to pull them.

In fact, I'll be out for the next few weeks; I'll ask Bjorn to cover for me.

Thanks,
Jesse

Signature · 1 line
Jesse Barnes, Intel Open Source Technology Center
#3 lore raw
Quoted 7 lines

You should apply them. I would queue them, and but I don't have another PCI tree anywhere for you to pull them.

In fact, I'll be out for the next few weeks; I'll ask Bjorn to cover for me.

Note that I might be missing some patches, but what's currently in Linus tree is rather wrong.

The comment is right, but the implementation is bogus:

static void pcie_write_mps(struct pci_dev *dev, int mps)

{
	int rc, dev_mpss;

	dev_mpss = 128 << dev->pcie_mpss;

	if (pcie_bus_config == PCIE_BUS_PERFORMANCE) {
		if (dev->bus->self) {
			dev_dbg(&dev->bus->dev, "Bus MPSS %d\n",
				128 << dev->bus->self->pcie_mpss);

			/* For "MPS Force Max", the assumption is made that
			 * downstream communication will never be larger than
			 * the MRRS.  So, the MPS only needs to be configured
			 * for the upstream communication.  This being the case,
			 * walk from the top down and set the MPS of the child
			 * to that of the parent bus.
			 */
			mps = 128 << dev->bus->self->pcie_mpss;

So here, we decide to use as an mps, the mpss of the bridge. That's already not quite right. The bridge might itself be clamped down (because it's parent has a smaller MPSS, well, your other bugs below will prevent that but that's how it should be)

So we should use the MPS of the bridge not the MPSS (ie the configured MPS of the bridge, not the max MPS of the bridge).

That does mean that we do need to configure the bridge first before we configure the bridge children, I don't remember if that's what happens with the walk function so we need to dbl check (I have to run, no time to check right now)

if (mps > dev_mpss)
dev_warn(&dev->dev, "MPS configured higher than"
" maximum supported by the device. If" " a bus issue occurs, try running with" " pci=pcie_bus_safe.\n");

Yeah right. Instead of warning, we should actually -clamp- it. IE If the bridge can do 4096 bytes and the device can only do 128 bytes, it's crazy to try to set the device to 4096 bytes.

The idea is that we set the device to the higher it can up to the size supported by the brigde, not the exact size of the brigde regardless of whether that's bigger than what the device can handle.

It -never- makes any sense to configure a device to an MPS larger than it's own maxium.

		}

		dev->pcie_mpss = ffs(mps) - 8;

And here I'm baffled. Why on earth would we -ever- want to change dev->pcie_mpss ?

This is the the HARD maximum supported by the device, as read from the capability register. Changing our idea of what the device supports as a maximum isn't going to help us... all it will do is allow the subsequent:

	}

	rc = pcie_set_mps(dev, mps);
	if (rc)
		dev_err(&dev->dev, "Failed attempting to set the MPS\n");

To actually try to program the bogus mps value we just calculated (ie the device larger than what it supports).

}

The right thing to do here for the "performance" case is to use the min() of the bus mps and the dev_mpss basically.

Ie. Going top down:

- If !parent, mps = mpss else mps = min(mpss, parent->mps)
- Iterate children

That's it.

So of course, the current implementation -will- break things. It might even explain some of the radeon problems you've been observing, ie, might have nothing to do with MRRS...

As to whether we should default to "safe" or "performance", well, it really depends on the platform.

I know for a fact on power that my host bridges are never going to generate a downstream request with a TLP larger than 128 bytes, and we never do PCIe <-> PCIe transfers, so "performance" is good for me.

x86 with funky DMA engines in the CPU etc... might be capable of generating larger TLPs, in which case it must switch to "safe".

Note that this whole business is really a mis-design of PCIe. The 'performance' mode is a way to try to work around it and not end up with everything clamped down to the minimum MPS (128 bytes) all the time, but it's a bit risky. Ideally the spec should have separated at least the MPS of packets generated vs. the MPS of packets received at the root complex level.

Now, I am giving some courses/training today and may not have time to propose a patch, but if you don't get to it by tomorrow I'll see what I can do.

Cheers,
Ben.

#4 lore raw
Quoted 11 lines · Benjamin Herrenschmidt
Quoted 6 lines

You should apply them. I would queue them, and but I don't have another PCI tree anywhere for you to pull them.

In fact, I'll be out for the next few weeks; I'll ask Bjorn to cover for me.

Note that I might be missing some patches, but what's currently in Linus tree is rather wrong.

The comment is right, but the implementation is bogus:

Oh and another one (again, I never saw the patches I'm responding to for some odd reason, only email 0/2, so you might have already fixed that):

When using "safe" mode, I get a crash in pcie_find_smpss(). Patch below.

Quoted 1 line

From 1b542d690323fab332e61099193d598c5c515d13 Mon Sep 17 00:00:00 2001

From: Benjamin Herrenschmidt <[email protected]>
Date: Tue, 13 Sep 2011 15:16:33 -0300
Subject: [PATCH] pci: Don't crash when reading mpss from root complex

In pcie_find_smpss(), we have the following statement:

if (dev->is_hotplug_bridge && (!list_is_singular(&dev->bus->devices) ||

	    dev->bus->self->pcie_type != PCI_EXP_TYPE_ROOT_PORT))

The problem is that at least on my machine, this gets called for the root complex (virtual P2P bridge), and dev->bus->self is NULL since the parent bus for this is not itself anchor to a PCI device.

This adds the necessary NULL check.

---
 drivers/pci/probe.c |    3 ++-
 1 files changed, 2 insertions(+), 1 deletions(-)
drivers/pci/probe.c +2 -1
index 8473727..09d1c0f 100644
@@ -1351,7 +1351,8 @@ static int pcie_find_smpss(struct pci_dev *dev, void *data)
 	 * will occur as normal.
 	 */
 	if (dev->is_hotplug_bridge && (!list_is_singular(&dev->bus->devices) ||
-	    dev->bus->self->pcie_type != PCI_EXP_TYPE_ROOT_PORT))
+	     (dev->bus->self &&
+	      dev->bus->self->pcie_type != PCI_EXP_TYPE_ROOT_PORT)))
 		*smpss = 0;
 
 	if (*smpss > dev->pcie_mpss)
Signature · 1 line
1.7.4.1
author Acked-by
#5 lore raw
Quoted 8 lines · Benjamin Herrenschmidt
Quoted 7 lines · Benjamin Herrenschmidt
Quoted 6 lines

You should apply them. I would queue them, and but I don't have another PCI tree anywhere for you to pull them.

In fact, I'll be out for the next few weeks; I'll ask Bjorn to cover for me.

Quoted 7 lines
Quoted 4 lines

Note that I might be missing some patches, but what's currently in Linus tree is rather wrong.

The comment is right, but the implementation is bogus:

Oh and another one (again, I never saw the patches I'm responding to for some odd reason, only email 0/2, so you might have already fixed that):

DNS issue with kernel.org when the patches went out caused it to not get out to any of the mailing lists.

In Linux' github tree
Patch 1/2 is commit 5307f6d5fb12fd01f9f321bc4a8fd77e74858647 Patch 2/2 is commit ed2888e906b56769b4ffabb9c577190438aa68b8

Quoted 12 lines

When using "safe" mode, I get a crash in pcie_find_smpss(). Patch below.

From 1b542d690323fab332e61099193d598c5c515d13 Mon Sep 17 00:00:00 2001
From: Benjamin Herrenschmidt <[email protected]>
Date: Tue, 13 Sep 2011 15:16:33 -0300
Subject: [PATCH] pci: Don't crash when reading mpss from root complex

In pcie_find_smpss(), we have the following statement:

       if (dev->is_hotplug_bridge && (!list_is_singular(&dev->bus->devices) || dev->bus->self->pcie_type != PCI_EXP_TYPE_ROOT_PORT))

Quoted 7 lines

The problem is that at least on my machine, this gets called for the root complex (virtual P2P bridge), and dev->bus->self is NULL since the parent bus for this is not itself anchor to a PCI device.

This adds the necessary NULL check.

Good catch.

Quoted 27 lines

---
 drivers/pci/probe.c |    3 ++-

 1 files changed, 2 insertions(+), 1 deletions(-)
drivers/pci/probe.c +0 -0
index 8473727..09d1c0f 100644
@@ -1351,7 +1351,8 @@ static int pcie_find_smpss(struct pci_dev *dev, void *data)
         * will occur as normal.
         */

       if (dev->is_hotplug_bridge && (!list_is_singular(&dev->bus->devices) ||
-           dev->bus->self->pcie_type != PCI_EXP_TYPE_ROOT_PORT))
+            (dev->bus->self &&
+             dev->bus->self->pcie_type != PCI_EXP_TYPE_ROOT_PORT))) *smpss = 0;

       if (*smpss > dev->pcie_mpss)
--
1.7.4.1

#6 lore raw
Quoted 24 lines

DNS issue with kernel.org when the patches went out caused it to not get out to any of the mailing lists.

In Linux' github tree
Patch 1/2 is commit 5307f6d5fb12fd01f9f321bc4a8fd77e74858647 Patch 2/2 is commit ed2888e906b56769b4ffabb9c577190438aa68b8

Quoted 16 lines

When using "safe" mode, I get a crash in pcie_find_smpss(). Patch below.

From 1b542d690323fab332e61099193d598c5c515d13 Mon Sep 17 00:00:00 2001
From: Benjamin Herrenschmidt <[email protected]>
Date: Tue, 13 Sep 2011 15:16:33 -0300
Subject: [PATCH] pci: Don't crash when reading mpss from root complex

In pcie_find_smpss(), we have the following statement:

       if (dev->is_hotplug_bridge && (!list_is_singular(&dev->bus->devices) ||
           dev->bus->self->pcie_type != PCI_EXP_TYPE_ROOT_PORT))

The problem is that at least on my machine, this gets called for the root complex (virtual P2P bridge), and dev->bus->self is NULL since the parent bus for this is not itself anchor to a PCI device.

Quoted 7 lines
Quoted 3 lines

This adds the necessary NULL check.

Good catch.

Linus, please apply.

Cheers
Ben.

Quoted 31 lines
Quoted 19 lines
---
 drivers/pci/probe.c |    3 ++-
 1 files changed, 2 insertions(+), 1 deletions(-)
drivers/pci/probe.c +2 -2
index 8473727..09d1c0f 100644
@@ -1351,7 +1351,8 @@ static int pcie_find_smpss(struct pci_dev *dev, void *data)
        * will occur as normal.
        */
       if (dev->is_hotplug_bridge && (!list_is_singular(&dev->bus->devices) ||
-           dev->bus->self->pcie_type != PCI_EXP_TYPE_ROOT_PORT))
+            (dev->bus->self &&
+             dev->bus->self->pcie_type != PCI_EXP_TYPE_ROOT_PORT)))
               *smpss = 0;
 
       if (*smpss > dev->pcie_mpss)
--
Quoted 8 lines

1.7.4.1

--
To unsubscribe from this list: send the line "unsubscribe linux-pci" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

author
#7 lore raw
In reply to Benjamin Herrenschmidt
Quoted 10 lines · Benjamin Herrenschmidt
Quoted 7 lines

You should apply them. I would queue them, and but I don't have another PCI tree anywhere for you to pull them.

In fact, I'll be out for the next few weeks; I'll ask Bjorn to cover for me.

Note that I might be missing some patches, but what's currently in Linus tree is rather wrong.

It is worth noting that with my last patch, this is not the default behavior. Whether it is faulty is open for debate.

Quoted 23 lines

The comment is right, but the implementation is bogus:

static void pcie_write_mps(struct pci_dev *dev, int mps) {
       int rc, dev_mpss;

       dev_mpss = 128 << dev->pcie_mpss;

       if (pcie_bus_config == PCIE_BUS_PERFORMANCE) { if (dev->bus->self) { dev_dbg(&dev->bus->dev, "Bus MPSS %d\n", 128 << dev->bus->self->pcie_mpss);

                       /* For "MPS Force Max", the assumption is made that
                        * downstream communication will never be larger than
                        * the MRRS.  So, the MPS only needs to be configured
                        * for the upstream communication.  This being the case,
                        * walk from the top down and set the MPS of the child
                        * to that of the parent bus.
                        */
                       mps = 128 << dev->bus->self->pcie_mpss;

Quoted 4 lines

So here, we decide to use as an mps, the mpss of the bridge. That's already not quite right. The bridge might itself be clamped down (because it's parent has a smaller MPSS, well, your other bugs below will prevent that but that's how it should be)

This is started from the root complex. So, it should be able to handle the maximum setting of the bridge chip.

Quoted 7 lines

So we should use the MPS of the bridge not the MPSS (ie the configured MPS of the bridge, not the max MPS of the bridge).

That does mean that we do need to configure the bridge first before we configure the bridge children, I don't remember if that's what happens with the walk function so we need to dbl check (I have to run, no time to check right now)

Yes, that is the behavior.

Quoted 10 lines

                       if (mps > dev_mpss)
                               dev_warn(&dev->dev, "MPS configured higher than" " maximum supported by the device.  If" " a bus issue occurs, try running with" " pci=pcie_bus_safe.\n");

Yeah right. Instead of warning, we should actually -clamp- it. IE If the bridge can do 4096 bytes and the device can only do 128 bytes, it's crazy to try to set the device to 4096 bytes.

You are correct. I was looking at this code when the bugs started coming in and realized this was erroneous. A simple mps=min(mps, dev_mpss) should fix it.

Quoted 4 lines

The idea is that we set the device to the higher it can up to the size supported by the brigde, not the exact size of the brigde regardless of whether that's bigger than what the device can handle.

Quoted 8 lines

It -never- makes any sense to configure a device to an MPS larger than it's own maxium.

               }

               dev->pcie_mpss = ffs(mps) - 8;

And here I'm baffled. Why on earth would we -ever- want to change dev->pcie_mpss ?

I was being overly clever here and using it as a way to actual MPS of the bridges. A comment here would be helpful, but in hindsight I think it is a bad idea to do it this way.

Quoted 21 lines

This is the the HARD maximum supported by the device, as read from the capability register. Changing our idea of what the device supports as a maximum isn't going to help us... all it will do is allow the subsequent:

       }

       rc = pcie_set_mps(dev, mps);
       if (rc)
               dev_err(&dev->dev, "Failed attempting to set the MPS\n");

To actually try to program the bogus mps value we just calculated (ie the device larger than what it supports).

}

The right thing to do here for the "performance" case is to use the min() of the bus mps and the dev_mpss basically.

Ie. Going top down:

Quoted 8 lines

 - If !parent, mps = mpss else mps = min(mpss, parent->mps)
 - Iterate children

That's it.

So of course, the current implementation -will- break things. It might even explain some of the radeon problems you've been observing, ie, might have nothing to do with MRRS...

My patch before that removed MRRS setting from the "performance" option solved a majority of the problems seen, but as you said in response to that patch, it was buggy.

Quoted 10 lines

As to whether we should default to "safe" or "performance", well, it really depends on the platform.

I know for a fact on power that my host bridges are never going to generate a downstream request with a TLP larger than 128 bytes, and we never do PCIe  <-> PCIe transfers, so "performance" is good for me.

x86 with funky DMA engines in the CPU etc... might be capable of generating larger TLPs, in which case it must switch to "safe".

Only is it is not observing the MRRS value.

Quoted 6 lines

Note that this whole business is really a mis-design of PCIe. The 'performance' mode is a way to try to work around it and not end up with everything clamped down to the minimum MPS (128 bytes) all the time, but it's a bit risky. Ideally the spec should have separated at least the MPS of packets generated vs. the MPS of packets received at the root complex level.

And a fix for poorly written BIOS (the original cause of this patch).

Quoted 3 lines

Now, I am giving some courses/training today and may not have time to propose a patch, but if you don't get to it by tomorrow I'll see what I can do.

I'll make a pass at this shortly and send it to linux-pci. Thanks for going over the code.

Quoted 5 lines

Cheers,
Ben.