Skip to contentSkip to conversation

pci: only release that resource index is less than 3

PATCH In review Yinghai Lu 3 participants 9 messages Oct 24, 2009 – Oct 28, 2009

Subsystems (2)
lore
Share & tools
lore

Yinghai Lu

author
#0 lore raw

PATCH 1/1 pci: only release that resource index is less than 3

after

| commit 308cf8e13f42f476dfd6552aeff58fdc0788e566
|
|    PCI: get larger bridge ranges when space is available

found one of resource of peer root bus (0x00) get released from root resource. later one hotplug device can not get big range anymore. other peer root buses is ok.

it turns out it is from transparent path.

those resources will be used for pci bridge BAR updated. so need to limit it to 3.

---
 drivers/pci/setup-bus.c |    9 +++++++--
 1 file changed, 7 insertions(+), 2 deletions(-)
drivers/pci/setup-bus.c +7 -2
Index: linux-2.6/drivers/pci/setup-bus.c
===================================================================
@@ -344,9 +344,14 @@ static struct resource *find_free_bus_re
 			 * if there is no child under that, we should release
 			 * and use it. don't need to reset it, pbus_size_* will
 			 * set it again
+			 * need to be less 3, otherwise can not write it to
+			 * bridge, also need to avoid releasing it from
+			 * transparent bus path
 			 */
-			if (!r->child && !release_resource(r))
-				return r;
+			if (i < 3 && !r->child) {
+				if (!release_resource(r))
+					return r;
+			}
 		}
 	}
 	return NULL;
#1 lore raw
Quoted 29 lines · Yinghai Lu

after

| commit 308cf8e13f42f476dfd6552aeff58fdc0788e566
|
|    PCI: get larger bridge ranges when space is available

found one of resource of peer root bus (0x00) get released from root resource. later one hotplug device can not get big range anymore. other peer root buses is ok.

it turns out it is from transparent path.

those resources will be used for pci bridge BAR updated. so need to limit it to 3.

---
 drivers/pci/setup-bus.c |    9 +++++++--
 1 file changed, 7 insertions(+), 2 deletions(-)

Index: linux-2.6/drivers/pci/setup-bus.c
===================================================================

drivers/pci/setup-bus.c +0 -0
@@ -344,9 +344,14 @@ static struct resource *find_free_bus_re
 			 * if there is no child under that, we should release
 			 * and use it. don't need to reset it, pbus_size_* will
 			 * set it again

drivers/pci/setup-bus.c static struct resource *find_free_bus_re · line 354

Quoted 10 lines
drivers/pci/setup-bus.c +7 -2
+			 * need to be less 3, otherwise can not write it to
+			 * bridge, also need to avoid releasing it from
+			 * transparent bus path
 			 */
-			if (!r->child && !release_resource(r))
-				return r;
+			if (i < 3 && !r->child) {
+				if (!release_resource(r))
+					return r;
+			}

I am bewildered.

308cf8e13f42f added the release_resource() call here in find_free_bus_resource(). I don't understand why the release should be there in the first place -- it doesn't seem to logically fit there. A "release" should be connected to an event, maybe a hot-remove or a move of a device from one place to another. It shouldn't be something we do as a side-effect of searching for a free resource.

Now you're adding the magic number "3", which seems even less related to the job of "finding an available bus resource." I'm guessing the "3" is related to PCI_BRIDGE_RESOURCES or something, but that should be made explicit, and I really don't think it belongs in this function.

Bjorn

#2 lore raw
Quoted 55 lines · On Mon, 26 Oct 2009 10:32:57 -0600 Bjorn Helgaas <[email protected]> wrote:
Quoted 35 lines · Yinghai Lu

after

| commit 308cf8e13f42f476dfd6552aeff58fdc0788e566
|
|    PCI: get larger bridge ranges when space is available

found one of resource of peer root bus (0x00) get released from root resource. later one hotplug device can not get big range anymore. other peer root buses is ok.

it turns out it is from transparent path.

those resources will be used for pci bridge BAR updated. so need to limit it to 3.

---
 drivers/pci/setup-bus.c |    9 +++++++--
 1 file changed, 7 insertions(+), 2 deletions(-)

Index: linux-2.6/drivers/pci/setup-bus.c
===================================================================

drivers/pci/setup-bus.c +0 -0
@@ -344,9 +344,14 @@ static struct resource *find_free_bus_re
 			 * if there is no child under that, we

should release
* and use it. don't need to reset it,
pbus_size_* will

drivers/pci/setup-bus.c +1 -0
 			 * set it again
+			 * need to be less 3, otherwise can not

write it to

drivers/pci/setup-bus.c +1 -0
+			 * bridge, also need to avoid releasing it

from

drivers/pci/setup-bus.c static struct resource *find_free_bus_re · line 354

Quoted 8 lines
drivers/pci/setup-bus.c +5 -2
+			 * transparent bus path
 			 */
-			if (!r->child && !release_resource(r))
-				return r;
+			if (i < 3 && !r->child) {
+				if (!release_resource(r))
+					return r;
+			}

I am bewildered.

308cf8e13f42f added the release_resource() call here in find_free_bus_resource(). I don't understand why the release should be there in the first place -- it doesn't seem to logically fit there. A "release" should be connected to an event, maybe a hot-remove or a move of a device from one place to another. It shouldn't be something we do as a side-effect of searching for a free resource.

Quoted 5 lines

Now you're adding the magic number "3", which seems even less related to the job of "finding an available bus resource." I'm guessing the "3" is related to PCI_BRIDGE_RESOURCES or something, but that should be made explicit, and I really don't think it belongs in this function.

I agree, the 3 is a bit of magic. The free could probably be shuffled around a bit too, though it really is related to finding bus resources. I was a bit ambivalent about putting it in find_free_bus_resources but when I started a reply to Yinghai and looked for a better place it made some sense to leave it as-is.

But if we're going to be adding more logic here, we should probably figure out a better way of doing it, maybe by adding a new pass to do the freeing? I know we've avoided modifying bus resources too much in the past, but I think that'll be harder and harder to avoid as Windows moves to that model and platforms begin to expect it.

Signature · 1 line
Jesse Barnes, Intel Open Source Technology Center
author
#8 lore raw
In reply to Bjorn Helgaas
Quoted 43 lines · Bjorn Helgaas
Quoted 29 lines · Yinghai Lu

after

| commit 308cf8e13f42f476dfd6552aeff58fdc0788e566
|
|    PCI: get larger bridge ranges when space is available

found one of resource of peer root bus (0x00) get released from root resource. later one hotplug device can not get big range anymore. other peer root buses is ok.

it turns out it is from transparent path.

those resources will be used for pci bridge BAR updated. so need to limit it to 3.

---
 drivers/pci/setup-bus.c |    9 +++++++--
 1 file changed, 7 insertions(+), 2 deletions(-)

Index: linux-2.6/drivers/pci/setup-bus.c
===================================================================

drivers/pci/setup-bus.c +0 -0
@@ -344,9 +344,14 @@ static struct resource *find_free_bus_re
 			 * if there is no child under that, we should release
 			 * and use it. don't need to reset it, pbus_size_* will
 			 * set it again

drivers/pci/setup-bus.c static struct resource *find_free_bus_re · line 354

Quoted 10 lines
drivers/pci/setup-bus.c +7 -2
+			 * need to be less 3, otherwise can not write it to
+			 * bridge, also need to avoid releasing it from
+			 * transparent bus path
 			 */
-			if (!r->child && !release_resource(r))
-				return r;
+			if (i < 3 && !r->child) {
+				if (!release_resource(r))
+					return r;
+			}

I am bewildered.

Quoted 7 lines

308cf8e13f42f added the release_resource() call here in find_free_bus_resource(). I don't understand why the release should be there in the first place -- it doesn't seem to logically fit there. A "release" should be connected to an event, maybe a hot-remove or a move of a device from one place to another. It shouldn't be something we do as a side-effect of searching for a free resource.

system with several IOHs, and one pci bridge under one IOH,

one of device under that bridge request big range 256M, and the bridge can not find range it, so the device BAR is not initialized. and the bridge already some range preallocated ( about 8M ) for future hotplug support.

so need to release the resource used by that bridge, so it could get new one.

Quoted 6 lines

Now you're adding the magic number "3", which seems even less related to the job of "finding an available bus resource." I'm guessing the "3" is related to PCI_BRIDGE_RESOURCES or something, but that should be made explicit, and I really don't think it belongs in this function.

find_free_bus_resource is only needed by pbus_size_io and bus_size_mem with pci bridge (not card bus) to find position of bridge BAR, so later could update corresponding BAR.

YH

author
#3 lore raw
In reply to Jesse Barnes
Quoted 69 lines · Jesse Barnes
Quoted 54 lines · On Mon, 26 Oct 2009 10:32:57 -0600 Bjorn Helgaas <[email protected]> wrote:
Quoted 35 lines · Yinghai Lu

after

| commit 308cf8e13f42f476dfd6552aeff58fdc0788e566
|
|    PCI: get larger bridge ranges when space is available

found one of resource of peer root bus (0x00) get released from root resource. later one hotplug device can not get big range anymore. other peer root buses is ok.

it turns out it is from transparent path.

those resources will be used for pci bridge BAR updated. so need to limit it to 3.

---
 drivers/pci/setup-bus.c |    9 +++++++--
 1 file changed, 7 insertions(+), 2 deletions(-)

Index: linux-2.6/drivers/pci/setup-bus.c
===================================================================

drivers/pci/setup-bus.c +0 -0
@@ -344,9 +344,14 @@ static struct resource *find_free_bus_re
 			 * if there is no child under that, we

should release
* and use it. don't need to reset it,
pbus_size_* will

drivers/pci/setup-bus.c +1 -0
 			 * set it again
+			 * need to be less 3, otherwise can not

write it to

drivers/pci/setup-bus.c +1 -0
+			 * bridge, also need to avoid releasing it

from

drivers/pci/setup-bus.c

Quoted 8 lines
drivers/pci/setup-bus.c +5 -2
+			 * transparent bus path
 			 */
-			if (!r->child && !release_resource(r))
-				return r;
+			if (i < 3 && !r->child) {
+				if (!release_resource(r))
+					return r;
+			}

I am bewildered.

308cf8e13f42f added the release_resource() call here in find_free_bus_resource(). I don't understand why the release should be there in the first place -- it doesn't seem to logically fit there. A "release" should be connected to an event, maybe a hot-remove or a move of a device from one place to another. It shouldn't be something we do as a side-effect of searching for a free resource.

Quoted 5 lines

Now you're adding the magic number "3", which seems even less related to the job of "finding an available bus resource." I'm guessing the "3" is related to PCI_BRIDGE_RESOURCES or something, but that should be made explicit, and I really don't think it belongs in this function.

I agree, the 3 is a bit of magic. The free could probably be shuffled around a bit too, though it really is related to finding bus resources. I was a bit ambivalent about putting it in find_free_bus_resources but when I started a reply to Yinghai and looked for a better place it made some sense to leave it as-is.

Quoted 5 lines

But if we're going to be adding more logic here, we should probably figure out a better way of doing it, maybe by adding a new pass to do the freeing? I know we've avoided modifying bus resources too much in the past, but I think that'll be harder and harder to avoid as Windows moves to that model and platforms begin to expect it.

not sure if you like this version. it need more lines changes.

[PATCH] pci: only release that resource index is less than 3 -v2

after

| commit 308cf8e13f42f476dfd6552aeff58fdc0788e566
|
|    PCI: get larger bridge ranges when space is available

found one of resource of peer root bus (0x00) get released from root resource. later one hotplug device can not get big range anymore. other peer root buses is ok.

it turns out it is from transparent path.

those resources will be used for pci bridge BAR updated. so need to limit it to 3.

v2: Jesse doesn't like it is in find_free_bus_resource...
try to move out of pci_bus_size_bridges loop.
need to apply after:
[PATCH] pci: pciehp update the slot bridge res to get big range for pcie devices - v3

---
 drivers/pci/hotplug/pciehp_pci.c |    1 
 drivers/pci/setup-bus.c          |   83 +++++++++++++++++++++++++++++++++------
 include/linux/pci.h              |    1 
 3 files changed, 74 insertions(+), 11 deletions(-)
drivers/pci/setup-bus.c +72 -11
Index: linux-2.6/drivers/pci/setup-bus.c
===================================================================
@@ -322,33 +322,54 @@ static void pci_bridge_check_ranges(stru
 	}
 }
 
-/* Helper function for sizing routines: find first available
-   bus resource of a given type. Note: we intentionally skip
-   the bus resources which have already been assigned (that is,
-   have non-NULL parent resource). */
-static struct resource *find_free_bus_resource(struct pci_bus *bus, unsigned long type)
+void pci_bridge_release_not_used_res(struct pci_bus *bus)
 {
 	int i;
 	struct resource *r;
 	unsigned long type_mask = IORESOURCE_IO | IORESOURCE_MEM |
 				  IORESOURCE_PREFETCH;
 
-	for (i = 0; i < PCI_BUS_NUM_RESOURCES; i++) {
+	/* for pci bridges res only */
+	for (i = 0; i < 3; i++) {
 		r = bus->resource[i];
-		if (r == &ioport_resource || r == &iomem_resource)
-			continue;
-		if (r && (r->flags & type_mask) == type) {
+		if (r && (r->flags & type_mask)) {
 			if (!r->parent)
-				return r;
+				continue;
 			/*
 			 * if there is no child under that, we should release
 			 * and use it. don't need to reset it, pbus_size_* will
 			 * set it again
 			 */
 			if (!r->child && !release_resource(r))
-				return r;
+				dev_printk(KERN_DEBUG, &bus->dev,
+					   "resource %d %s %pR released\n", i,
+					   (r->flags & IORESOURCE_IO) ? "io: " :
+					    ((r->flags & IORESOURCE_PREFETCH)?
+						 "pref mem":"mem:"),
+					   r);
 		}
 	}
+}
+EXPORT_SYMBOL(pci_bridge_release_not_used_res);
+
+/* Helper function for sizing routines: find first available
+   bus resource of a given type. Note: we intentionally skip
+   the bus resources which have already been assigned (that is,
+   have non-NULL parent resource). */
+static struct resource *find_free_bus_resource(struct pci_bus *bus, unsigned long type)
+{
+	int i;
+	struct resource *r;
+	unsigned long type_mask = IORESOURCE_IO | IORESOURCE_MEM |
+				  IORESOURCE_PREFETCH;
+
+	for (i = 0; i < PCI_BUS_NUM_RESOURCES; i++) {
+		r = bus->resource[i];
+		if (r == &ioport_resource || r == &iomem_resource)
+			continue;
+		if (r && (r->flags & type_mask) == type && !r->parent)
+			return r;
+	}
 	return NULL;
 }
 
@@ -588,6 +609,41 @@ void __ref pci_bus_size_bridges(struct p
 }
 EXPORT_SYMBOL(pci_bus_size_bridges);
 
+static void pci_bus_release_bridges_not_used_res(struct pci_bus *bus)
+{
+	struct pci_dev *dev;
+
+	list_for_each_entry(dev, &bus->devices, bus_list) {
+		struct pci_bus *b = dev->subordinate;
+		if (!b)
+			continue;
+
+		switch (dev->class >> 8) {
+		case PCI_CLASS_BRIDGE_CARDBUS:
+			break;
+
+		case PCI_CLASS_BRIDGE_PCI:
+		default:
+			pci_bus_release_bridges_not_used_res(b);
+			break;
+		}
+	}
+
+	/* The root bus? */
+	if (!bus->self)
+		return;
+
+	switch (bus->self->class >> 8) {
+	case PCI_CLASS_BRIDGE_CARDBUS:
+		break;
+
+	case PCI_CLASS_BRIDGE_PCI:
+	default:
+		pci_bridge_release_not_used_res(bus);
+		break;
+	}
+}
+
 void __ref pci_bridge_assign_resources(const struct pci_dev *bridge)
 {
 	struct pci_bus *b;
@@ -687,6 +743,11 @@ pci_assign_unassigned_resources(void)
 {
 	struct pci_bus *bus;
 
+	/* Try to release bridge resources, that there is not child under it */
+	list_for_each_entry(bus, &pci_root_buses, node) {
+		pci_bus_release_bridges_not_used_res(bus);
+	}
+
 	/* Depth first, calculate sizes and alignments of all
 	   subordinate buses. */
 	list_for_each_entry(bus, &pci_root_buses, node) {
drivers/pci/hotplug/pciehp_pci.c +1 -0
Index: linux-2.6/drivers/pci/hotplug/pciehp_pci.c
===================================================================
@@ -171,6 +171,7 @@ int pciehp_unconfigure_device(struct slo
 		}
 		pci_dev_put(temp);
 	}
+	pci_bridge_release_not_used_res(parent);
 
 	return rc;
 }
include/linux/pci.h +1 -0
Index: linux-2.6/include/linux/pci.h
===================================================================
@@ -759,6 +759,7 @@ int pci_vpd_truncate(struct pci_dev *dev
 void pci_bridge_assign_resources(const struct pci_dev *bridge);
 void pci_bus_assign_resources(const struct pci_bus *bus);
 void pci_bus_size_bridges(struct pci_bus *bus);
+void pci_bridge_release_not_used_res(struct pci_bus *bus);
 int pci_claim_resource(struct pci_dev *, int);
 void pci_assign_unassigned_resources(void);
 void pdev_enable_device(struct pci_dev *);
#4 lore raw
Quoted 12 lines · Yinghai Lu

===================================================================

drivers/pci/setup-bus.c +0 -5
@@ -322,33 +322,54 @@ static void pci_bridge_check_ranges(stru
 	}
 }
 
-/* Helper function for sizing routines: find first available
-   bus resource of a given type. Note: we intentionally skip
-   the bus resources which have already been assigned (that is,
-   have non-NULL parent resource). */
-static struct resource *find_free_bus_resource(struct pci_bus *bus, unsigned long type)

drivers/pci/setup-bus.c static void pci_bridge_check_ranges(stru · line 333

Quoted 10 lines
drivers/pci/setup-bus.c +3 -1
+void pci_bridge_release_not_used_res(struct pci_bus *bus)
 {
 	int i;
 	struct resource *r;
 	unsigned long type_mask = IORESOURCE_IO | IORESOURCE_MEM |
 				  IORESOURCE_PREFETCH;
 
-	for (i = 0; i < PCI_BUS_NUM_RESOURCES; i++) {
+	/* for pci bridges res only */
+	for (i = 0; i < 3; i++) {

I think the "i = 0; i < 3" part is to check the bridge apertures, i.e., the I/O base/limit, the memory base/limit, and the prefetchable memory base/limit. Right? We need some way to indicate that more clearly than using a hard-coded "3".

Quoted 4 lines
drivers/pci/setup-bus.c +0 -3
 		r = bus->resource[i];
-		if (r == &ioport_resource || r == &iomem_resource)
-			continue;
-		if (r && (r->flags & type_mask) == type) {

drivers/pci/setup-bus.c static void pci_bridge_check_ranges(stru · line 343

Quoted 10 lines
drivers/pci/setup-bus.c +2 -1
+		if (r && (r->flags & type_mask)) {
 			if (!r->parent)
-				return r;
+				continue;
 			/*
 			 * if there is no child under that, we should release
 			 * and use it. don't need to reset it, pbus_size_* will
 			 * set it again
 			 */
 			if (!r->child && !release_resource(r))

We got this resource pointer out of a struct pci_bus, and we release it here. We must have previously done a request_resource(), allocate_resource(), or similar. Where does that happen? Are the requests and releases nested correctly?

I would think that somewhere, we would be doing a request_resource() and assigning the resource to pci_bus->resource[x]. But there are very few assignments to the pci_bus resources:
setup_resource (only for "pci=use_crs") pci_read_bridge_bases (just a copy from upstream bus resources) pci_alloc_child_bus (copy from upstream bridge resources) pci_create_bus (set to &ioport_resource or &iomem_resource)

My guess is that this release_resource() releases something we copied from the bridge in pci_alloc_child_bus(). But that doesn't seem right, because we aren't changing the bridge programming here.

Maybe I'm being confused by the fact that we find the resource pointer from the pci_bus table, but that's only a pointer to the actual struct resource that lives in the pci_dev of the upstream bridge.

Bjorn

Quoted 111 lines
drivers/pci/setup-bus.c +67 -1
-				return r;
+				dev_printk(KERN_DEBUG, &bus->dev,
+					   "resource %d %s %pR released\n", i,
+					   (r->flags & IORESOURCE_IO) ? "io: " :
+					    ((r->flags & IORESOURCE_PREFETCH)?
+						 "pref mem":"mem:"),
+					   r);
 		}
 	}
+}
+EXPORT_SYMBOL(pci_bridge_release_not_used_res);
+
+/* Helper function for sizing routines: find first available
+   bus resource of a given type. Note: we intentionally skip
+   the bus resources which have already been assigned (that is,
+   have non-NULL parent resource). */
+static struct resource *find_free_bus_resource(struct pci_bus *bus, unsigned long type)
+{
+	int i;
+	struct resource *r;
+	unsigned long type_mask = IORESOURCE_IO | IORESOURCE_MEM |
+				  IORESOURCE_PREFETCH;
+
+	for (i = 0; i < PCI_BUS_NUM_RESOURCES; i++) {
+		r = bus->resource[i];
+		if (r == &ioport_resource || r == &iomem_resource)
+			continue;
+		if (r && (r->flags & type_mask) == type && !r->parent)
+			return r;
+	}
 	return NULL;
 }
 
@@ -588,6 +609,41 @@ void __ref pci_bus_size_bridges(struct p
 }
 EXPORT_SYMBOL(pci_bus_size_bridges);
 
+static void pci_bus_release_bridges_not_used_res(struct pci_bus *bus)
+{
+	struct pci_dev *dev;
+
+	list_for_each_entry(dev, &bus->devices, bus_list) {
+		struct pci_bus *b = dev->subordinate;
+		if (!b)
+			continue;
+
+		switch (dev->class >> 8) {
+		case PCI_CLASS_BRIDGE_CARDBUS:
+			break;
+
+		case PCI_CLASS_BRIDGE_PCI:
+		default:
+			pci_bus_release_bridges_not_used_res(b);
+			break;
+		}
+	}
+
+	/* The root bus? */
+	if (!bus->self)
+		return;
+
+	switch (bus->self->class >> 8) {
+	case PCI_CLASS_BRIDGE_CARDBUS:
+		break;
+
+	case PCI_CLASS_BRIDGE_PCI:
+	default:
+		pci_bridge_release_not_used_res(bus);
+		break;
+	}
+}
+
 void __ref pci_bridge_assign_resources(const struct pci_dev *bridge)
 {
 	struct pci_bus *b;
@@ -687,6 +743,11 @@ pci_assign_unassigned_resources(void)
 {
 	struct pci_bus *bus;
 
+	/* Try to release bridge resources, that there is not child under it */
+	list_for_each_entry(bus, &pci_root_buses, node) {
+		pci_bus_release_bridges_not_used_res(bus);
+	}
+
 	/* Depth first, calculate sizes and alignments of all
 	   subordinate buses. */
 	list_for_each_entry(bus, &pci_root_buses, node) {

Index: linux-2.6/drivers/pci/hotplug/pciehp_pci.c
===================================================================

drivers/pci/hotplug/pciehp_pci.c +1 -0
@@ -171,6 +171,7 @@ int pciehp_unconfigure_device(struct slo
 		}
 		pci_dev_put(temp);
 	}
+	pci_bridge_release_not_used_res(parent);
 
 	return rc;
 }

Index: linux-2.6/include/linux/pci.h
===================================================================

include/linux/pci.h +1 -0
@@ -759,6 +759,7 @@ int pci_vpd_truncate(struct pci_dev *dev
 void pci_bridge_assign_resources(const struct pci_dev *bridge);
 void pci_bus_assign_resources(const struct pci_bus *bus);
 void pci_bus_size_bridges(struct pci_bus *bus);
+void pci_bridge_release_not_used_res(struct pci_bus *bus);
 int pci_claim_resource(struct pci_dev *, int);
 void pci_assign_unassigned_resources(void);
 void pdev_enable_device(struct pci_dev *);
author
#5 lore raw
Quoted 21 lines · Bjorn Helgaas
Quoted 19 lines · Yinghai Lu

===================================================================

drivers/pci/setup-bus.c +1 -5
@@ -322,33 +322,54 @@ static void pci_bridge_check_ranges(stru
 	}
 }
 
-/* Helper function for sizing routines: find first available
-   bus resource of a given type. Note: we intentionally skip
-   the bus resources which have already been assigned (that is,
-   have non-NULL parent resource). */
-static struct resource *find_free_bus_resource(struct pci_bus *bus, unsigned long type)
+void pci_bridge_release_not_used_res(struct pci_bus *bus)
 {
 	int i;
 	struct resource *r;
 	unsigned long type_mask = IORESOURCE_IO | IORESOURCE_MEM |
 				  IORESOURCE_PREFETCH;
 
Quoted 8 lines

drivers/pci/setup-bus.c static void pci_bridge_check_ranges(stru · line 333

Quoted 3 lines
drivers/pci/setup-bus.c +2 -1
-	for (i = 0; i < PCI_BUS_NUM_RESOURCES; i++) {
+	/* for pci bridges res only */
+	for (i = 0; i < 3; i++) {

I think the "i = 0; i < 3" part is to check the bridge apertures, i.e., the I/O base/limit, the memory base/limit, and the prefetchable memory base/limit. Right? We need some way to indicate that more clearly than using a hard-coded "3".

yes. the 0x1c, 0x20, and 0x28

Quoted 29 lines
Quoted 5 lines
drivers/pci/setup-bus.c +0 -3
 		r = bus->resource[i];
-		if (r == &ioport_resource || r == &iomem_resource)
-			continue;
-		if (r && (r->flags & type_mask) == type) {

drivers/pci/setup-bus.c static void pci_bridge_check_ranges(stru · line 343

Quoted 10 lines
drivers/pci/setup-bus.c +2 -1
+		if (r && (r->flags & type_mask)) {
 			if (!r->parent)
-				return r;
+				continue;
 			/*
 			 * if there is no child under that, we should release
 			 * and use it. don't need to reset it, pbus_size_* will
 			 * set it again
 			 */
 			if (!r->child && !release_resource(r))

We got this resource pointer out of a struct pci_bus, and we release it here. We must have previously done a request_resource(), allocate_resource(), or similar. Where does that happen? Are the requests and releases nested correctly?

I would think that somewhere, we would be doing a request_resource() and assigning the resource to pci_bus->resource[x]. But there are very few assignments to the pci_bus resources:
setup_resource (only for "pci=use_crs") pci_read_bridge_bases (just a copy from upstream bus resources) pci_alloc_child_bus (copy from upstream bridge resources) pci_create_bus (set to &ioport_resource or &iomem_resource)

Quoted 3 lines

My guess is that this release_resource() releases something we copied from the bridge in pci_alloc_child_bus(). But that doesn't seem right, because we aren't changing the bridge programming here.

in arch/x86/pci/i386.c::
pcibios_resource_survey() ==> pcibios_allocate_bus_resources()

YH

#6 lore raw
Quoted 31 lines · Yinghai Lu
Quoted 21 lines · Bjorn Helgaas
Quoted 19 lines · Yinghai Lu

===================================================================

drivers/pci/setup-bus.c +1 -5
@@ -322,33 +322,54 @@ static void pci_bridge_check_ranges(stru
 	}
 }
 
-/* Helper function for sizing routines: find first available
-   bus resource of a given type. Note: we intentionally skip
-   the bus resources which have already been assigned (that is,
-   have non-NULL parent resource). */
-static struct resource *find_free_bus_resource(struct pci_bus *bus, unsigned long type)
+void pci_bridge_release_not_used_res(struct pci_bus *bus)
 {
 	int i;
 	struct resource *r;
 	unsigned long type_mask = IORESOURCE_IO | IORESOURCE_MEM |
 				  IORESOURCE_PREFETCH;
 
Quoted 8 lines

drivers/pci/setup-bus.c static void pci_bridge_check_ranges(stru · line 333

Quoted 3 lines
drivers/pci/setup-bus.c +2 -1
-	for (i = 0; i < PCI_BUS_NUM_RESOURCES; i++) {
+	/* for pci bridges res only */
+	for (i = 0; i < 3; i++) {

I think the "i = 0; i < 3" part is to check the bridge apertures, i.e., the I/O base/limit, the memory base/limit, and the prefetchable memory base/limit. Right? We need some way to indicate that more clearly than using a hard-coded "3".

Quoted 1 line

yes. the 0x1c, 0x20, and 0x28

Do you have a proposal for something other than "3"? The magic number "3" does appear other places, e.g., pci_read_bridge_bases() but I don't think we should add new uses of it. Make up a descriptive #define if nothing else.

Quoted 28 lines
Quoted 28 lines
Quoted 4 lines
drivers/pci/setup-bus.c +0 -3
 		r = bus->resource[i];
-		if (r == &ioport_resource || r == &iomem_resource)
-			continue;
-		if (r && (r->flags & type_mask) == type) {

drivers/pci/setup-bus.c static void pci_bridge_check_ranges(stru · line 343

Quoted 10 lines
drivers/pci/setup-bus.c +2 -1
+		if (r && (r->flags & type_mask)) {
 			if (!r->parent)
-				return r;
+				continue;
 			/*
 			 * if there is no child under that, we should release
 			 * and use it. don't need to reset it, pbus_size_* will
 			 * set it again
 			 */
 			if (!r->child && !release_resource(r))

We got this resource pointer out of a struct pci_bus, and we release it here. We must have previously done a request_resource(), allocate_resource(), or similar. Where does that happen? Are the requests and releases nested correctly?

I would think that somewhere, we would be doing a request_resource() and assigning the resource to pci_bus->resource[x]. But there are very few assignments to the pci_bus resources:
setup_resource (only for "pci=use_crs") pci_read_bridge_bases (just a copy from upstream bus resources) pci_alloc_child_bus (copy from upstream bridge resources) pci_create_bus (set to &ioport_resource or &iomem_resource)

Quoted 6 lines
Quoted 3 lines

My guess is that this release_resource() releases something we copied from the bridge in pci_alloc_child_bus(). But that doesn't seem right, because we aren't changing the bridge programming here.

in arch/x86/pci/i386.c::
pcibios_resource_survey() ==> pcibios_allocate_bus_resources()

The asymmetry here bothers me. The pci_claim_resource() in pcibios_allocate_bus_resources() is done on a *device*, while the release_resource() you added is done on a *bus*. And the claim is done in arch-specific code, while the release is done in generic code. And there's an interval where the bridge device resources don't match the hardware apertures, and it's hard to figure out when they are in sync again.

This might all work fine, but it's harder to understand than it should be.

Bjorn

author
#7 lore raw

[PATCH 3/4] pci: only release that resource index is less than 3 -v4

after

| commit 308cf8e13f42f476dfd6552aeff58fdc0788e566
|
|    PCI: get larger bridge ranges when space is available

found one of resource of peer root bus (0x00) get released from root resource. later one hotplug device can not get big range anymore. other peer root buses is ok.

it turns out it is from transparent path.

those resources will be used for pci bridge BAR updated. so need to limit it to 3.

v2: Jesse doesn't like it is in find_free_bus_resource...
try to move out of pci_bus_size_bridges loop.
need to apply after:
[PATCH] pci: pciehp update the slot bridge res to get big range for pcie devices - v4
v3: add pci_setup_bridge calling after pci_bridge_release_not_used_res.
only clear release those res for x86.
v4: Bjorn want to release use dev instead of bus.

---
 arch/x86/pci/i386.c              |    6 +++
 drivers/pci/hotplug/pciehp_pci.c |    2 +
 drivers/pci/setup-bus.c          |   75 ++++++++++++++++++++++++++++++++++++++-
 include/linux/pci.h              |    2 +
 4 files changed, 84 insertions(+), 1 deletion(-)
drivers/pci/setup-bus.c +74 -1
Index: linux-2.6/drivers/pci/setup-bus.c
===================================================================
@@ -319,6 +319,42 @@ static void pci_bridge_check_ranges(stru
 	}
 }
 
+void pci_bridge_release_not_used_res(struct pci_bus *bus)
+{
+	int idx;
+	bool changed = false;
+	struct pci_dev *dev;
+	struct resource *r;
+	unsigned long type_mask = IORESOURCE_IO | IORESOURCE_MEM |
+				  IORESOURCE_PREFETCH;
+
+	/* for pci bridges res only */
+	dev = bus->self;
+	for (idx = PCI_BRIDGE_RESOURCES; idx < PCI_BRIDGE_RESOURCES + 3;
+	    idx++) {
+		r = &dev->resource[idx];
+		if (r->flags & type_mask) {
+			if (!r->parent)
+				continue;
+			/*
+			 * if there is no child under that, we should release
+			 * and use it.
+			 */
+			if (!r->child && !release_resource(r)) {
+				dev_info(&dev->dev,
+					 "resource %d %pRt released\n",
+					 idx, r);
+				r->flags = 0;
+				changed = true;
+			}
+		}
+	}
+
+	if (changed)
+		pci_setup_bridge(bus);
+}
+EXPORT_SYMBOL(pci_bridge_release_not_used_res);
+
 /* Helper function for sizing routines: find first available
    bus resource of a given type. Note: we intentionally skip
    the bus resources which have already been assigned (that is,
@@ -576,6 +612,42 @@ void __ref pci_bus_size_bridges(struct p
 }
 EXPORT_SYMBOL(pci_bus_size_bridges);
 
+void __ref pci_bus_release_bridges_not_used_res(struct pci_bus *bus)
+{
+	struct pci_dev *dev;
+
+	list_for_each_entry(dev, &bus->devices, bus_list) {
+		struct pci_bus *b = dev->subordinate;
+		if (!b)
+			continue;
+
+		switch (dev->class >> 8) {
+		case PCI_CLASS_BRIDGE_CARDBUS:
+			break;
+
+		case PCI_CLASS_BRIDGE_PCI:
+		default:
+			pci_bus_release_bridges_not_used_res(b);
+			break;
+		}
+	}
+
+	/* The root bus? */
+	if (!bus->self)
+		return;
+
+	switch (bus->self->class >> 8) {
+	case PCI_CLASS_BRIDGE_CARDBUS:
+		break;
+
+	case PCI_CLASS_BRIDGE_PCI:
+	default:
+		pci_bridge_release_not_used_res(bus);
+		break;
+	}
+}
+EXPORT_SYMBOL(pci_bus_release_bridges_not_used_res);
+
 void __ref pci_bridge_assign_resources(const struct pci_dev *bridge)
 {
 	struct pci_bus *b;
@@ -644,7 +716,8 @@ static void pci_bus_dump_res(struct pci_
 
         for (i = 0; i < PCI_BUS_NUM_RESOURCES; i++) {
                 struct resource *res = bus->resource[i];
-                if (!res || !res->end)
+
+		if (!res || !res->end || !res->flags)
                         continue;
 
 		dev_printk(KERN_DEBUG, &bus->dev, "resource %d %pRt\n", i, res);
drivers/pci/hotplug/pciehp_pci.c +2 -0
Index: linux-2.6/drivers/pci/hotplug/pciehp_pci.c
===================================================================
@@ -98,6 +98,7 @@ int pciehp_configure_device(struct slot
 		pci_dev_put(dev);
 	}
 
+	pci_bridge_release_not_used_res(parent);
 	pci_bus_size_bridges(parent);
 	pci_clear_master(bridge);
 	pci_bridge_assign_resources(bridge);
@@ -171,6 +172,7 @@ int pciehp_unconfigure_device(struct slo
 		}
 		pci_dev_put(temp);
 	}
+	pci_bridge_release_not_used_res(parent);
 
 	return rc;
 }
include/linux/pci.h +2 -0
Index: linux-2.6/include/linux/pci.h
===================================================================
@@ -759,6 +759,8 @@ int pci_vpd_truncate(struct pci_dev *dev
 void pci_bridge_assign_resources(const struct pci_dev *bridge);
 void pci_bus_assign_resources(const struct pci_bus *bus);
 void pci_bus_size_bridges(struct pci_bus *bus);
+void pci_bus_release_bridges_not_used_res(struct pci_bus *bus);
+void pci_bridge_release_not_used_res(struct pci_bus *bus);
 int pci_claim_resource(struct pci_dev *, int);
 void pci_assign_unassigned_resources(void);
 void pdev_enable_device(struct pci_dev *);
arch/x86/pci/i386.c +6 -0
Index: linux-2.6/arch/x86/pci/i386.c
===================================================================
@@ -194,6 +194,7 @@ static void __init pcibios_allocate_reso
 static int __init pcibios_assign_resources(void)
 {
 	struct pci_dev *dev = NULL;
+	struct pci_bus *bus;
 	struct resource *r;
 
 	if (!(pci_probe & PCI_ASSIGN_ROMS)) {
@@ -213,6 +214,11 @@ static int __init pcibios_assign_resourc
 		}
 	}
 
+	/* Try to release bridge resources, that there is not child under it */
+	list_for_each_entry(bus, &pci_root_buses, node) {
+		pci_bus_release_bridges_not_used_res(bus);
+	}
+
 	pci_assign_unassigned_resources();
 
 	return 0;