PCI: Resource fitting/assignment fixes and cleanups

PATCH Applied Ilpo Järvinen 7 participants 70 messages Dec 16, 2024 – Jun 26, 2025 PCI

lore

Share & tools

lore

Ilpo Järvinen

author

2024-12-16 17:56 UTC #0 lore raw

cover PCI: Resource fitting/assignment fixes and cleanups

Hi all,

This series focuses on PCI resource fitting and assignment algorithms. I've further changes in works to enable handling resizable BARs better during resource fitting built on top of these, but that's still WIP and this series seems way too large as is to have more stuff included.

First there are small tweaks and fixes to the relaxed tail alignment code and applying the lessons learned to other similar cases. They are sort of independent of the rest. Then a large set of pure cleanups and refactoring that are not intended to make any functional changes. Finally, starting from "PCI: Extend enable to check for any optional resource" are again patches that aim to make behavioral changes to fix bridge window sizing to consider expansion ROM as an optional resource (to fix a remove/rescan cycle issue) and improve resource fitting algorithm in general.

The series includes one of the change from Michał Winiarski <[email protected]> as these changes also touch the same IOV checks.

Please let me know if you'd prefer me to order the changes differently or split it into smaller chunks.

I've extensively tested this series over the hosts in our lab which have quite heterogeneous PCI setup each. There were no losses of any important resource. Without pci=realloc, there's some churn in which of the disabled expansion ROMs gets a scarce memory space assigned (with pci=realloc, they are all assigned large enough bridge window).

Ilpo Järvinen (24):
PCI: Remove add_align overwrite unrelated to size0
PCI: size0 is unrelated to add_align
PCI: Simplify size1 assignment logic
PCI: Optional bridge window size too may need relaxing
PCI: Fix old_size lower bound in calculate_iosize() too
PCI: Use SZ_* instead of literals in setup-bus.c
PCI: resource_set_range/size() conversions
PCI: Check resource_size() separately
PCI: Add pci_resource_num() helper
PCI: Add dev & res local variables to resource assignment funcs
PCI: Converge return paths in __assign_resources_sorted()
PCI: Refactor pdev_sort_resources() & __dev_sort_resources()
PCI: Use while loop and break instead of gotos
PCI: Rename retval to ret
PCI: Consolidate assignment loop next round preparation
PCI: Remove wrong comment from pci_reassign_resource()
PCI: Add restore_dev_resource()
PCI: Extend enable to check for any optional resource
PCI: Always have realloc_head in __assign_resources_sorted()
PCI: Indicate optional resource assignment failures
PCI: Add debug print when releasing resources before retry
PCI: Use res->parent to check is resource is assigned
PCI: Perform reset_resource() and build fail list in sync
PCI: Rework optional resource handling

Michał Winiarski (1):
PCI: Add a helper to identify IOV resources

 drivers/pci/pci.h       |  44 +++-
 drivers/pci/setup-bus.c | 566 +++++++++++++++++++++++-----------------
 drivers/pci/setup-res.c |   8 +-
 3 files changed, 364 insertions(+), 254 deletions(-)

Signature · 1 line

2.39.5

Ilpo Järvinen

author

2024-12-16 17:56 UTC #1 lore raw

PATCH 1/25 PCI: Remove add_align overwrite unrelated to size0

The commit 566f1dd52816 ("PCI: Relax bridge window tail sizing rules") relaxed bridge window tail alignment rule for the non-optional part (size0, no add_size/add_align). The change, however, also overwrite add_align that is only related to case where optional size1 related entry is added into realloc head.

Correct this by removing the add_align overwrite.

Fixes 566f1dd52816 ("PCI: Relax bridge window tail sizing rules")

Signed-off-by Ilpo Järvinen <[email protected]>

---
 drivers/pci/setup-bus.c | 1 -
 1 file changed, 1 deletion(-)

drivers/pci/setup-bus.c +0 -1

index 5e00cecf1f1a..d92832e1f4a1 100644
@@ -1149,7 +1149,6 @@ static int pbus_size_mem(struct pci_bus *bus, unsigned long mask,
 		min_align = 1ULL << (max_order + __ffs(SZ_1M));
 		min_align = max(min_align, win_align);
 		size0 = calculate_memsize(size, min_size, 0, 0, resource_size(b_res), win_align);
-		add_align = win_align;
 		pci_info(bus->self, "bridge window %pR to %pR requires relaxed alignment rules\n",
 			 b_res, &bus->busn_res);
 	}

Signature · 1 line

2.39.5

Ilpo Järvinen

author

2024-12-16 17:57 UTC #2 lore raw

PATCH 2/25 PCI: size0 is unrelated to add_align

In reply to Ilpo Järvinen

The commit 566f1dd52816 ("PCI: Relax bridge window tail sizing rules") relaxed bridge window tail alignment rule for the non-optional part (size0, no add_size/add_align). The required alignment given for pbus_upstream_space_available(), however, was add_align which relates only to size1 alignment.

As pbus_upstream_space_available() only selects between normal and relaxed tail alignment of the bridge window, the different alignment only makes relaxed tail alignment to be used more often than what was intended which should be harmless because relaxed tail alignment itself should work in all cases.

For consistency, change pbus_upstream_space_available() call to use min_align which is the alignment that is going to be used for the bridge window in the case where size0 sized allocation is attempted.

Signed-off-by Ilpo Järvinen <[email protected]>

---
 drivers/pci/setup-bus.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

drivers/pci/setup-bus.c +1 -1

index d92832e1f4a1..09c275f8d088 100644
@@ -1145,7 +1145,7 @@ static int pbus_size_mem(struct pci_bus *bus, unsigned long mask,
 
 	if (bus->self && size0 &&
 	    !pbus_upstream_space_available(bus, mask | IORESOURCE_PREFETCH, type,
-					   size0, add_align)) {
+					   size0, min_align)) {
 		min_align = 1ULL << (max_order + __ffs(SZ_1M));
 		min_align = max(min_align, win_align);
 		size0 = calculate_memsize(size, min_size, 0, 0, resource_size(b_res), win_align);

Signature · 1 line

2.39.5

Ilpo Järvinen

author

2024-12-16 17:57 UTC #3 lore raw

PATCH 3/25 PCI: Simplify size1 assignment logic

In reply to Ilpo Järvinen

In pbus_size_io() and pbus_size_mem(), a complex ?: operation is performed to set size1 which becomes easier to read when decomposed.

In the case of pbus_size_mem(), simply initializing size1 to zero ensures the size1 checks work as expected.

Signed-off-by Ilpo Järvinen <[email protected]>

---
 drivers/pci/setup-bus.c | 21 ++++++++++++++-------
 1 file changed, 14 insertions(+), 7 deletions(-)

drivers/pci/setup-bus.c +14 -7

index 09c275f8d088..7f4680a23c13 100644
@@ -927,9 +927,14 @@ static void pbus_size_io(struct pci_bus *bus, resource_size_t min_size,
 
 	size0 = calculate_iosize(size, min_size, size1, 0, 0,
 			resource_size(b_res), min_align);
-	size1 = (!realloc_head || (realloc_head && !add_size && !children_add_size)) ? size0 :
-		calculate_iosize(size, min_size, size1, add_size, children_add_size,
-			resource_size(b_res), min_align);
+
+	size1 = size0;
+	if (realloc_head && (add_size > 0 || children_add_size > 0)) {
+		size1 = calculate_iosize(size, min_size, size1, add_size,
+					 children_add_size, resource_size(b_res),
+					 min_align);
+	}
+
 	if (!size0 && !size1) {
 		if (bus->self && (b_res->start || b_res->end))
 			pci_info(bus->self, "disabling bridge window %pR to %pR (unused)\n",

@@ -1058,7 +1063,7 @@ static int pbus_size_mem(struct pci_bus *bus, unsigned long mask,
 			 struct list_head *realloc_head)
 {
 	struct pci_dev *dev;
-	resource_size_t min_align, win_align, align, size, size0, size1;
+	resource_size_t min_align, win_align, align, size, size0, size1 = 0;
 	resource_size_t aligns[24]; /* Alignments from 1MB to 8TB */
 	int order, max_order;
 	struct resource *b_res = find_bus_resource_of_type(bus,

@@ -1153,9 +1158,11 @@ static int pbus_size_mem(struct pci_bus *bus, unsigned long mask,
 			 b_res, &bus->busn_res);
 	}
 
-	size1 = (!realloc_head || (realloc_head && !add_size && !children_add_size)) ? size0 :
-		calculate_memsize(size, min_size, add_size, children_add_size,
-				resource_size(b_res), add_align);
+	if (realloc_head && (add_size > 0 || children_add_size > 0)) {
+		size1 = calculate_memsize(size, min_size, add_size, children_add_size,
+					  resource_size(b_res), add_align);
+	}
+
 	if (!size0 && !size1) {
 		if (bus->self && (b_res->start || b_res->end))
 			pci_info(bus->self, "disabling bridge window %pR to %pR (unused)\n",

Signature · 1 line

2.39.5

Ilpo Järvinen

author

2024-12-16 17:57 UTC #4 lore raw

PATCH 4/25 PCI: Optional bridge window size too may need relaxing

In reply to Ilpo Järvinen

The commit 566f1dd52816 ("PCI: Relax bridge window tail sizing rules") relaxed the bridge window requirements for non-optional size (size0) but pbus_size_mem() also handles optional sizes (IOV resources) using size1. This can manifest, e.g., as a failure to resize a BAR back to its original size after it was first shrunk when device has a VF BAR resource because the bridge window (size1) is enlarged beyond what is strictly required to fit the downstream resources.

Allow using relaxed bridge window tail sizing rules also with the optional resources (size1) so that the remove/realloc cycle during BAR resize (smaller and back to the original size) does not fail unexpectedly due to sudden increase in bridge window size demand.

Move also add_align calculation into more logical place next to size1 assignment as they are strongly related to each other.

Fixes 566f1dd52816 ("PCI: Relax bridge window tail sizing rules")

Reported-by Michał Winiarski <[email protected]>

Signed-off-by Ilpo Järvinen <[email protected]>

---
 drivers/pci/setup-bus.c | 14 +++++++++++++-
 1 file changed, 13 insertions(+), 1 deletion(-)

drivers/pci/setup-bus.c +13 -1

index 7f4680a23c13..31f051cdac68 100644
@@ -1146,7 +1146,6 @@ static int pbus_size_mem(struct pci_bus *bus, unsigned long mask,
 	min_align = calculate_mem_align(aligns, max_order);
 	min_align = max(min_align, win_align);
 	size0 = calculate_memsize(size, min_size, 0, 0, resource_size(b_res), min_align);
-	add_align = max(min_align, add_align);
 
 	if (bus->self && size0 &&
 	    !pbus_upstream_space_available(bus, mask | IORESOURCE_PREFETCH, type,

@@ -1159,8 +1158,21 @@ static int pbus_size_mem(struct pci_bus *bus, unsigned long mask,
 	}
 
 	if (realloc_head && (add_size > 0 || children_add_size > 0)) {
+		add_align = max(min_align, add_align);
 		size1 = calculate_memsize(size, min_size, add_size, children_add_size,
 					  resource_size(b_res), add_align);
+
+		if (bus->self && size1 &&
+		    !pbus_upstream_space_available(bus, mask | IORESOURCE_PREFETCH, type,
+						   size1, add_align)) {
+			min_align = 1ULL << (max_order + __ffs(SZ_1M));
+			min_align = max(min_align, win_align);
+			size1 = calculate_memsize(size, min_size, add_size, children_add_size,
+						  resource_size(b_res), win_align);
+			pci_info(bus->self,
+				 "bridge window %pR to %pR requires relaxed alignment rules\n",
+				 b_res, &bus->busn_res);
+		}
 	}
 
 	if (!size0 && !size1) {

Signature · 1 line

2.39.5

Ilpo Järvinen

author

2024-12-16 17:57 UTC #5 lore raw

PATCH 5/25 PCI: Fix old_size lower bound in calculate_iosize() too

In reply to Ilpo Järvinen

The commit 903534fa7d30 ("PCI: Fix resource double counting on remove & rescan") fixed double counting of mem resources because of old_size being applied too early.

Fix a similar counting bug on the io resource side.

Signed-off-by Ilpo Järvinen <[email protected]>

---
 drivers/pci/setup-bus.c | 6 ++----
 1 file changed, 2 insertions(+), 4 deletions(-)

drivers/pci/setup-bus.c +2 -4

index 31f051cdac68..ca544fb83700 100644
@@ -814,11 +814,9 @@ static resource_size_t calculate_iosize(resource_size_t size,
 	size = (size & 0xff) + ((size & ~0xffUL) << 2);
 #endif
 	size = size + size1;
-	if (size < old_size)
-		size = old_size;
 
-	size = ALIGN(max(size, add_size) + children_add_size, align);
-	return size;
+	size = max(size, add_size) + children_add_size;
+	return ALIGN(max(size, old_size), align);
 }
 
 static resource_size_t calculate_memsize(resource_size_t size,

Signature · 1 line

2.39.5

Ilpo Järvinen

author

2024-12-16 17:57 UTC #6 lore raw

PATCH 6/25 PCI: Use SZ_* instead of literals in setup-bus.c

In reply to Ilpo Järvinen

Convert literals in setup-bus.c to SZ_* defines that make the size more human readable.

As the code is now self-explanatory, the comments about the size can be eliminated.

Signed-off-by Ilpo Järvinen <[email protected]>

---
 drivers/pci/setup-bus.c | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

drivers/pci/setup-bus.c +4 -4

index ca544fb83700..303c4fbf2d14 100644
@@ -841,9 +841,9 @@ resource_size_t __weak pcibios_window_alignment(struct pci_bus *bus,
 	return 1;
 }
 
-#define PCI_P2P_DEFAULT_MEM_ALIGN	0x100000	/* 1MiB */
-#define PCI_P2P_DEFAULT_IO_ALIGN	0x1000		/* 4KiB */
-#define PCI_P2P_DEFAULT_IO_ALIGN_1K	0x400		/* 1KiB */
+#define PCI_P2P_DEFAULT_MEM_ALIGN	SZ_1M
+#define PCI_P2P_DEFAULT_IO_ALIGN	SZ_4K
+#define PCI_P2P_DEFAULT_IO_ALIGN_1K	SZ_1K
 
 static resource_size_t window_alignment(struct pci_bus *bus, unsigned long type)
 {

@@ -908,7 +908,7 @@ static void pbus_size_io(struct pci_bus *bus, resource_size_t min_size,
 				continue;
 			r_size = resource_size(r);
 
-			if (r_size < 0x400)
+			if (r_size < SZ_1K)
 				/* Might be re-aligned for ISA */
 				size += r_size;
 			else

Signature · 1 line

2.39.5

Ilpo Järvinen

author

2024-12-16 17:57 UTC #7 lore raw

PATCH 7/25 PCI: resource_set_range/size() conversions

In reply to Ilpo Järvinen

A few sites that could use resource_set_range/size() in setup-bus.c were not picked up earlier due to them no matching the usual pattern. Convert them now.

Signed-off-by Ilpo Järvinen <[email protected]>

---
 drivers/pci/setup-bus.c | 15 ++++++---------
 1 file changed, 6 insertions(+), 9 deletions(-)

drivers/pci/setup-bus.c +6 -9

index 303c4fbf2d14..dd9b06947621 100644
@@ -413,10 +413,8 @@ static void __assign_resources_sorted(struct list_head *head,
 		 * consistent.
 		 */
 		if (add_align > dev_res->res->start) {
-			resource_size_t r_size = resource_size(dev_res->res);
-
-			dev_res->res->start = add_align;
-			dev_res->res->end = add_align + r_size - 1;
+			resource_set_range(dev_res->res, add_align,
+					   resource_size(dev_res->res));
 
 			list_for_each_entry(dev_res2, head, list) {
 				align = pci_resource_alignment(dev_res2->dev,

@@ -1100,7 +1098,7 @@ static int pbus_size_mem(struct pci_bus *bus, unsigned long mask,
 			if (realloc_head && i >= PCI_IOV_RESOURCES &&
 					i <= PCI_IOV_RESOURCE_END) {
 				add_align = max(pci_resource_alignment(dev, r), add_align);
-				r->end = r->start - 1;
+				resource_set_size(r, 0);
 				add_to_list(realloc_head, dev, r, r_size, 0 /* Don't care */);
 				children_add_size += r_size;
 				continue;

@@ -1180,8 +1178,8 @@ static int pbus_size_mem(struct pci_bus *bus, unsigned long mask,
 		b_res->flags = 0;
 		return 0;
 	}
-	b_res->start = min_align;
-	b_res->end = size0 + min_align - 1;
+
+	resource_set_range(b_res, min_align, size0);
 	b_res->flags |= IORESOURCE_STARTALIGN;
 	if (bus->self && size1 > size0 && realloc_head) {
 		add_to_list(realloc_head, bus->self, b_res, size1-size0, add_align);

@@ -1656,8 +1654,7 @@ static void pci_bridge_release_resources(struct pci_bus *bus,
 		pci_info(dev, "resource %d %pR released\n",
 			 PCI_BRIDGE_RESOURCES + idx, r);
 		/* Keep the old size */
-		r->end = resource_size(r) - 1;
-		r->start = 0;
+		resource_set_range(r, 0, resource_size(r));
 		r->flags = 0;
 
 		/* Avoiding touch the one without PREF */

Signature · 1 line

2.39.5

Ilpo Järvinen

author

2024-12-16 17:57 UTC #8 lore raw

PATCH 8/25 PCI: Add a helper to identify IOV resources

In reply to Ilpo Järvinen

From: Michał Winiarski <[email protected]>

There are multiple places where special handling is required for IOV resources.

Extract it to pci_resource_is_iov() helper and drop a few ifdefs.

Signed-off-by Michał Winiarski <[email protected]>

Reviewed-by Ilpo Järvinen <[email protected]>

Reviewed-by Christian König <[email protected]>

---
 drivers/pci/pci.h       | 19 +++++++++++++++----
 drivers/pci/setup-bus.c |  7 +++----
 drivers/pci/setup-res.c |  4 +---
 3 files changed, 19 insertions(+), 11 deletions(-)

drivers/pci/pci.h +15 -4

index 2e40fc63ba31..25bae4bfebea 100644
@@ -621,6 +621,10 @@ void pci_iov_update_resource(struct pci_dev *dev, int resno);
 resource_size_t pci_sriov_resource_alignment(struct pci_dev *dev, int resno);
 void pci_restore_iov_state(struct pci_dev *dev);
 int pci_iov_bus_range(struct pci_bus *bus);
+static inline bool pci_resource_is_iov(int resno)
+{
+	return resno >= PCI_IOV_RESOURCES && resno <= PCI_IOV_RESOURCE_END;
+}
 extern const struct attribute_group sriov_pf_dev_attr_group;
 extern const struct attribute_group sriov_vf_dev_attr_group;
 #else

@@ -630,12 +634,21 @@ static inline int pci_iov_init(struct pci_dev *dev)
 }
 static inline void pci_iov_release(struct pci_dev *dev) { }
 static inline void pci_iov_remove(struct pci_dev *dev) { }
+static inline void pci_iov_update_resource(struct pci_dev *dev, int resno) { }
+static inline resource_size_t pci_sriov_resource_alignment(struct pci_dev *dev,
+							   int resno)
+{
+	return 0;
+}
 static inline void pci_restore_iov_state(struct pci_dev *dev) { }
 static inline int pci_iov_bus_range(struct pci_bus *bus)
 {
 	return 0;
 }
-
+static inline bool pci_resource_is_iov(int resno)
+{
+	return false;
+}
 #endif /* CONFIG_PCI_IOV */
 
 #ifdef CONFIG_PCIE_TPH

@@ -669,12 +682,10 @@ unsigned long pci_cardbus_resource_alignment(struct resource *);
 static inline resource_size_t pci_resource_alignment(struct pci_dev *dev,
 						     struct resource *res)
 {
-#ifdef CONFIG_PCI_IOV
 	int resno = res - dev->resource;
 
-	if (resno >= PCI_IOV_RESOURCES && resno <= PCI_IOV_RESOURCE_END)
+	if (pci_resource_is_iov(resno))
 		return pci_sriov_resource_alignment(dev, resno);
-#endif
 	if (dev->class >> 8 == PCI_CLASS_BRIDGE_CARDBUS)
 		return pci_cardbus_resource_alignment(res);
 	return resource_alignment(res);

drivers/pci/setup-bus.c +3 -4

index dd9b06947621..3907930da00d 100644
@@ -1093,17 +1093,16 @@ static int pbus_size_mem(struct pci_bus *bus, unsigned long mask,
 			     (r->flags & mask) != type3))
 				continue;
 			r_size = resource_size(r);
-#ifdef CONFIG_PCI_IOV
+
 			/* Put SRIOV requested res to the optional list */
-			if (realloc_head && i >= PCI_IOV_RESOURCES &&
-					i <= PCI_IOV_RESOURCE_END) {
+			if (realloc_head && pci_resource_is_iov(i)) {
 				add_align = max(pci_resource_alignment(dev, r), add_align);
 				resource_set_size(r, 0);
 				add_to_list(realloc_head, dev, r, r_size, 0 /* Don't care */);
 				children_add_size += r_size;
 				continue;
 			}
-#endif
+
 			/*
 			 * aligns[0] is for 1MB (since bridge memory
 			 * windows are always at least 1MB aligned), so

drivers/pci/setup-res.c +1 -3

index ca14576bf2bf..79c7ef349856 100644
@@ -127,10 +127,8 @@ void pci_update_resource(struct pci_dev *dev, int resno)
 {
 	if (resno <= PCI_ROM_RESOURCE)
 		pci_std_update_resource(dev, resno);
-#ifdef CONFIG_PCI_IOV
-	else if (resno >= PCI_IOV_RESOURCES && resno <= PCI_IOV_RESOURCE_END)
+	else if (pci_resource_is_iov(resno))
 		pci_iov_update_resource(dev, resno);
-#endif
 }
 
 int pci_claim_resource(struct pci_dev *dev, int resource)

Signature · 1 line

2.39.5

Ilpo Järvinen

author

2024-12-16 17:58 UTC #9 lore raw

PATCH 9/25 PCI: Check resource_size() separately

In reply to Ilpo Järvinen

Instead of chaining logic inside if () condition so that multiple lines are required, make !resource_size() a separate check and use continue.

Signed-off-by Ilpo Järvinen <[email protected]>

---
 drivers/pci/setup-bus.c | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

drivers/pci/setup-bus.c +5 -2

index 3907930da00d..63c134b087d5 100644
@@ -285,8 +285,11 @@ static void assign_requested_resources_sorted(struct list_head *head,
 	list_for_each_entry(dev_res, head, list) {
 		res = dev_res->res;
 		idx = res - &dev_res->dev->resource[0];
-		if (resource_size(res) &&
-		    pci_assign_resource(dev_res->dev, idx)) {
+
+		if (!resource_size(res))
+			continue;
+
+		if (pci_assign_resource(dev_res->dev, idx)) {
 			if (fail_head) {
 				/*
 				 * If the failed resource is a ROM BAR and

Signature · 1 line

2.39.5

Ilpo Järvinen

author

2024-12-16 17:58 UTC #10 lore raw

PATCH 10/25 PCI: Add pci_resource_num() helper

In reply to Ilpo Järvinen

A few places in PCI code, mainly in setup-bus.c, need to reverse lookup the index of a resource in pci_dev's resource array. Create pci_resource_num() helper to avoid repeating the pointer arithmetic trick used to calculate the index.

Signed-off-by Ilpo Järvinen <[email protected]>

---
 drivers/pci/pci.h       | 24 +++++++++++++++++++++++-
 drivers/pci/setup-bus.c | 10 +++++-----
 2 files changed, 28 insertions(+), 6 deletions(-)

drivers/pci/pci.h +23 -1

index 25bae4bfebea..0b722d158b6a 100644
@@ -330,6 +330,28 @@ void pci_walk_bus_locked(struct pci_bus *top,
 
 const char *pci_resource_name(struct pci_dev *dev, unsigned int i);
 
+/**
+ * pci_resource_num - Reverse lookup resource number from device resources
+ * @dev: PCI device
+ * @res: Resource to lookup index for (MUST be a @dev's resource)
+ *
+ * Perform reverse lookup to determine the resource number for @res within
+ * @dev resource array. NOTE: The caller is responsible for ensuring @res is
+ * among @dev's resources!
+ *
+ * Returns: resource number.
+ */
+static inline int pci_resource_num(const struct pci_dev *dev,
+				   const struct resource *res)
+{
+	int resno = res - &dev->resource[0];
+
+	/* Passing a resource that is not among dev's resources? */
+	WARN_ON_ONCE(resno >= PCI_NUM_RESOURCES);
+
+	return resno;
+}
+
 void pci_reassigndev_resource_alignment(struct pci_dev *dev);
 void pci_disable_bridge_window(struct pci_dev *dev);
 struct pci_bus *pci_bus_get(struct pci_bus *bus);

@@ -682,7 +704,7 @@ unsigned long pci_cardbus_resource_alignment(struct resource *);
 static inline resource_size_t pci_resource_alignment(struct pci_dev *dev,
 						     struct resource *res)
 {
-	int resno = res - dev->resource;
+	int resno = pci_resource_num(dev, res);
 
 	if (pci_resource_is_iov(resno))
 		return pci_sriov_resource_alignment(dev, resno);

drivers/pci/setup-bus.c +5 -5

index 63c134b087d5..8831365418d6 100644
@@ -242,7 +242,7 @@ static void reassign_resources_sorted(struct list_head *realloc_head,
 		if (!found_match) /* Just skip */
 			continue;
 
-		idx = res - &add_res->dev->resource[0];
+		idx = pci_resource_num(add_res->dev, res);
 		res_name = pci_resource_name(add_res->dev, idx);
 		add_size = add_res->add_size;
 		align = add_res->min_align;

@@ -284,7 +284,7 @@ static void assign_requested_resources_sorted(struct list_head *head,
 
 	list_for_each_entry(dev_res, head, list) {
 		res = dev_res->res;
-		idx = res - &dev_res->dev->resource[0];
+		idx = pci_resource_num(dev_res->dev, res);
 
 		if (!resource_size(res))
 			continue;

@@ -2211,7 +2211,7 @@ void pci_assign_unassigned_root_bus_resources(struct pci_bus *bus)
 		res->flags = fail_res->flags;
 
 		if (pci_is_bridge(fail_res->dev)) {
-			idx = res - &fail_res->dev->resource[0];
+			idx = pci_resource_num(fail_res->dev, res);
 			if (idx >= PCI_BRIDGE_RESOURCES &&
 			    idx <= PCI_BRIDGE_RESOURCE_END)
 				res->flags = 0;

@@ -2295,7 +2295,7 @@ void pci_assign_unassigned_bridge_resources(struct pci_dev *bridge)
 		res->flags = fail_res->flags;
 
 		if (pci_is_bridge(fail_res->dev)) {
-			idx = res - &fail_res->dev->resource[0];
+			idx = pci_resource_num(fail_res->dev, res);
 			if (idx >= PCI_BRIDGE_RESOURCES &&
 			    idx <= PCI_BRIDGE_RESOURCE_END)
 				res->flags = 0;

@@ -2402,7 +2402,7 @@ int pci_reassign_bridge_resources(struct pci_dev *bridge, unsigned long type)
 		struct resource *res = dev_res->res;
 
 		bridge = dev_res->dev;
-		i = res - bridge->resource;
+		i = pci_resource_num(bridge, res);
 
 		res->start = dev_res->start;
 		res->end = dev_res->end;

Signature · 1 line

2.39.5

Ilpo Järvinen

author

2024-12-16 17:58 UTC #11 lore raw

PATCH 11/25 PCI: Add dev & res local variables to resource assignment funcs

In reply to Ilpo Järvinen

Many PCI resource allocation related functions process struct pci_dev_resource items which hold the struct pci_dev and resource pointers. Reduce the number of lines that need indirection by adding 'dev' and 'res' local variable to hold the pointers.

Signed-off-by Ilpo Järvinen <[email protected]>

---
 drivers/pci/setup-bus.c | 66 +++++++++++++++++++++++------------------
 1 file changed, 37 insertions(+), 29 deletions(-)

drivers/pci/setup-bus.c +37 ~~-29~~

index 8831365418d6..6b4318da1147 100644
@@ -216,10 +216,11 @@ static inline void reset_resource(struct resource *res)
 static void reassign_resources_sorted(struct list_head *realloc_head,
 				      struct list_head *head)
 {
-	struct resource *res;
-	const char *res_name;
 	struct pci_dev_resource *add_res, *tmp;
 	struct pci_dev_resource *dev_res;
+	struct pci_dev *dev;
+	struct resource *res;
+	const char *res_name;
 	resource_size_t add_size, align;
 	int idx;

@@ -227,6 +228,7 @@ static void reassign_resources_sorted(struct list_head *realloc_head,
 		bool found_match = false;
 
 		res = add_res->res;
+		dev = add_res->dev;
 
 		/* Skip resource that has been reset */
 		if (!res->flags)

@@ -242,20 +244,19 @@ static void reassign_resources_sorted(struct list_head *realloc_head,
 		if (!found_match) /* Just skip */
 			continue;
 
-		idx = pci_resource_num(add_res->dev, res);
-		res_name = pci_resource_name(add_res->dev, idx);
+		idx = pci_resource_num(dev, res);
+		res_name = pci_resource_name(dev, idx);
 		add_size = add_res->add_size;
 		align = add_res->min_align;
 		if (!resource_size(res)) {
 			resource_set_range(res, align, add_size);
-			if (pci_assign_resource(add_res->dev, idx))
+			if (pci_assign_resource(dev, idx))
 				reset_resource(res);
 		} else {
 			res->flags |= add_res->flags &
 				 (IORESOURCE_STARTALIGN|IORESOURCE_SIZEALIGN);
-			if (pci_reassign_resource(add_res->dev, idx,
-						  add_size, align))
-				pci_info(add_res->dev, "%s %pR: failed to add %llx\n",
+			if (pci_reassign_resource(dev, idx, add_size, align))
+				pci_info(dev, "%s %pR: failed to add %llx\n",
 					 res_name, res,
 					 (unsigned long long) add_size);
 		}

@@ -278,18 +279,20 @@ static void reassign_resources_sorted(struct list_head *realloc_head,
 static void assign_requested_resources_sorted(struct list_head *head,
 				 struct list_head *fail_head)
 {
-	struct resource *res;
 	struct pci_dev_resource *dev_res;
+	struct resource *res;
+	struct pci_dev *dev;
 	int idx;
 
 	list_for_each_entry(dev_res, head, list) {
 		res = dev_res->res;
-		idx = pci_resource_num(dev_res->dev, res);
+		dev = dev_res->dev;
+		idx = pci_resource_num(dev, res);
 
 		if (!resource_size(res))
 			continue;
 
-		if (pci_assign_resource(dev_res->dev, idx)) {
+		if (pci_assign_resource(dev, idx)) {
 			if (fail_head) {
 				/*
 				 * If the failed resource is a ROM BAR and

@@ -298,8 +301,7 @@ static void assign_requested_resources_sorted(struct list_head *head,
 				 */
 				if (!((idx == PCI_ROM_RESOURCE) &&
 				      (!(res->flags & IORESOURCE_ROM_ENABLE))))
-					add_to_list(fail_head,
-						    dev_res->dev, res,
+					add_to_list(fail_head, dev, res,
 						    0 /* don't care */,
 						    0 /* don't care */);
 			}

@@ -377,6 +379,7 @@ static void __assign_resources_sorted(struct list_head *head,
 	LIST_HEAD(local_fail_head);
 	struct pci_dev_resource *save_res;
 	struct pci_dev_resource *dev_res, *tmp_res, *dev_res2;
+	struct resource *res;
 	unsigned long fail_type;
 	resource_size_t add_align, align;

@@ -394,8 +397,9 @@ static void __assign_resources_sorted(struct list_head *head,
 
 	/* Update res in head list with add_size in realloc_head list */
 	list_for_each_entry_safe(dev_res, tmp_res, head, list) {
-		dev_res->res->end += get_res_add_size(realloc_head,
-							dev_res->res);
+		res = dev_res->res;
+
+		res->end += get_res_add_size(realloc_head, res);
 
 		/*
 		 * There are two kinds of additional resources in the list:

@@ -403,10 +407,10 @@ static void __assign_resources_sorted(struct list_head *head,
 		 * 2. SR-IOV resource  -- IORESOURCE_SIZEALIGN
 		 * Here just fix the additional alignment for bridge
 		 */
-		if (!(dev_res->res->flags & IORESOURCE_STARTALIGN))
+		if (!(res->flags & IORESOURCE_STARTALIGN))
 			continue;
 
-		add_align = get_res_add_align(realloc_head, dev_res->res);
+		add_align = get_res_add_align(realloc_head, res);
 
 		/*
 		 * The "head" list is sorted by alignment so resources with

@@ -415,9 +419,8 @@ static void __assign_resources_sorted(struct list_head *head,
 		 * need to reorder the list by alignment to make it
 		 * consistent.
 		 */
-		if (add_align > dev_res->res->start) {
-			resource_set_range(dev_res->res, add_align,
-					   resource_size(dev_res->res));
+		if (add_align > res->start) {
+			resource_set_range(res, add_align, resource_size(res));
 
 			list_for_each_entry(dev_res2, head, list) {
 				align = pci_resource_alignment(dev_res2->dev,

@@ -448,24 +451,29 @@ static void __assign_resources_sorted(struct list_head *head,
 	/* Check failed type */
 	fail_type = pci_fail_res_type_mask(&local_fail_head);
 	/* Remove not need to be released assigned res from head list etc */
-	list_for_each_entry_safe(dev_res, tmp_res, head, list)
-		if (dev_res->res->parent &&
-		    !pci_need_to_release(fail_type, dev_res->res)) {
+	list_for_each_entry_safe(dev_res, tmp_res, head, list) {
+		res = dev_res->res;
+
+		if (res->parent && !pci_need_to_release(fail_type, res)) {
 			/* Remove it from realloc_head list */
-			remove_from_list(realloc_head, dev_res->res);
-			remove_from_list(&save_head, dev_res->res);
+			remove_from_list(realloc_head, res);
+			remove_from_list(&save_head, res);
 			list_del(&dev_res->list);
 			kfree(dev_res);
 		}
+	}
 
 	free_list(&local_fail_head);
 	/* Release assigned resource */
-	list_for_each_entry(dev_res, head, list)
-		if (dev_res->res->parent)
-			release_resource(dev_res->res);
+	list_for_each_entry(dev_res, head, list) {
+		res = dev_res->res;
+
+		if (res->parent)
+			release_resource(res);
+	}
 	/* Restore start/end/flags from saved list */
 	list_for_each_entry(save_res, &save_head, list) {
-		struct resource *res = save_res->res;
+		res = save_res->res;
 
 		res->start = save_res->start;
 		res->end = save_res->end;

Signature · 1 line

2.39.5

Ilpo Järvinen

author

2024-12-16 17:58 UTC #12 lore raw

PATCH 12/25 PCI: Converge return paths in __assign_resources_sorted()

In reply to Ilpo Järvinen

All return paths want to free head list in __assign_resources_sorted() so add a label and use goto.

Signed-off-by Ilpo Järvinen <[email protected]>

---
 drivers/pci/setup-bus.c | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

drivers/pci/setup-bus.c +3 -2

index 6b4318da1147..ad7bc6166b23 100644
@@ -444,8 +444,7 @@ static void __assign_resources_sorted(struct list_head *head,
 		list_for_each_entry(dev_res, head, list)
 			remove_from_list(realloc_head, dev_res->res);
 		free_list(&save_head);
-		free_list(head);
-		return;
+		goto out;
 	}
 
 	/* Check failed type */

@@ -488,6 +487,8 @@ static void __assign_resources_sorted(struct list_head *head,
 	/* Try to satisfy any additional optional resource requests */
 	if (realloc_head)
 		reassign_resources_sorted(realloc_head, head);
+
+out:
 	free_list(head);
 }

Signature · 1 line

2.39.5

Ilpo Järvinen

author

2024-12-16 17:58 UTC #13 lore raw

PATCH 13/25 PCI: Refactor pdev_sort_resources() & __dev_sort_resources()

In reply to Ilpo Järvinen

Reduce level of call nesting by calling pdev_sort_resources() directly and by moving the tests done inside __dev_sort_resources() into pdev_resources_assignable() helper.

Signed-off-by Ilpo Järvinen <[email protected]>

---
 drivers/pci/setup-bus.c | 44 +++++++++++++++++++++--------------------
 1 file changed, 23 insertions(+), 21 deletions(-)

drivers/pci/setup-bus.c +23 ~~-21~~

index ad7bc6166b23..ba935a050be3 100644
@@ -127,12 +127,33 @@ static resource_size_t get_res_add_align(struct list_head *head,
 	return dev_res ? dev_res->min_align : 0;
 }
 
+static bool pdev_resources_assignable(struct pci_dev *dev)
+{
+	u16 class = dev->class >> 8, command;
+
+	/* Don't touch classless devices or host bridges or IOAPICs */
+	if (class == PCI_CLASS_NOT_DEFINED || class == PCI_CLASS_BRIDGE_HOST)
+		return false;
+
+	/* Don't touch IOAPIC devices already enabled by firmware */
+	if (class == PCI_CLASS_SYSTEM_PIC) {
+		pci_read_config_word(dev, PCI_COMMAND, &command);
+		if (command & (PCI_COMMAND_IO | PCI_COMMAND_MEMORY))
+			return false;
+	}
+
+	return true;
+}
+
 /* Sort resources by alignment */
 static void pdev_sort_resources(struct pci_dev *dev, struct list_head *head)
 {
 	struct resource *r;
 	int i;
 
+	if (!pdev_resources_assignable(dev))
+		return;
+
 	pci_dev_for_each_resource(dev, r, i) {
 		const char *r_name = pci_resource_name(dev, i);
 		struct pci_dev_resource *dev_res, *tmp;

@@ -176,25 +197,6 @@ static void pdev_sort_resources(struct pci_dev *dev, struct list_head *head)
 	}
 }
 
-static void __dev_sort_resources(struct pci_dev *dev, struct list_head *head)
-{
-	u16 class = dev->class >> 8;
-
-	/* Don't touch classless devices or host bridges or IOAPICs */
-	if (class == PCI_CLASS_NOT_DEFINED || class == PCI_CLASS_BRIDGE_HOST)
-		return;
-
-	/* Don't touch IOAPIC devices already enabled by firmware */
-	if (class == PCI_CLASS_SYSTEM_PIC) {
-		u16 command;
-		pci_read_config_word(dev, PCI_COMMAND, &command);
-		if (command & (PCI_COMMAND_IO | PCI_COMMAND_MEMORY))
-			return;
-	}
-
-	pdev_sort_resources(dev, head);
-}
-
 static inline void reset_resource(struct resource *res)
 {
 	res->start = 0;

@@ -498,7 +500,7 @@ static void pdev_assign_resources_sorted(struct pci_dev *dev,
 {
 	LIST_HEAD(head);
 
-	__dev_sort_resources(dev, &head);
+	pdev_sort_resources(dev, &head);
 	__assign_resources_sorted(&head, add_head, fail_head);
 
 }

@@ -511,7 +513,7 @@ static void pbus_assign_resources_sorted(const struct pci_bus *bus,
 	LIST_HEAD(head);
 
 	list_for_each_entry(dev, &bus->devices, bus_list)
-		__dev_sort_resources(dev, &head);
+		pdev_sort_resources(dev, &head);
 
 	__assign_resources_sorted(&head, realloc_head, fail_head);
 }

Signature · 1 line

2.39.5

Ilpo Järvinen

author

2024-12-16 17:58 UTC #14 lore raw

PATCH 14/25 PCI: Use while loop and break instead of gotos

In reply to Ilpo Järvinen

pci_assign_unassigned_root_bus_resources() and pci_assign_unassigned_bridge_resources() contain ad-hoc loops using backwards goto and gotos out of the loop. Replace them with while loops and break statements.

While reindenting the loop bodies, do a few coding style tweaks (add braces & remove parenthesis).

Signed-off-by Ilpo Järvinen <[email protected]>

---
 drivers/pci/setup-bus.c | 212 ++++++++++++++++++++--------------------
 1 file changed, 106 insertions(+), 106 deletions(-)

drivers/pci/setup-bus.c +106 ~~-106~~

index ba935a050be3..bbe3472cfba9 100644
@@ -2162,78 +2162,79 @@ void pci_assign_unassigned_root_bus_resources(struct pci_bus *bus)
 			 max_depth, pci_try_num);
 	}
 
-again:
-	/*
-	 * Last try will use add_list, otherwise will try good to have as must
-	 * have, so can realloc parent bridge resource
-	 */
-	if (tried_times + 1 == pci_try_num)
-		add_list = &realloc_head;
-	/*
-	 * Depth first, calculate sizes and alignments of all subordinate buses.
-	 */
-	__pci_bus_size_bridges(bus, add_list);
+	while (1) {
+		/*
+		 * Last try will use add_list, otherwise will try good to
+		 * have as must have, so can realloc parent bridge resource
+		 */
+		if (tried_times + 1 == pci_try_num)
+			add_list = &realloc_head;
+		/*
+		 * Depth first, calculate sizes and alignments of all
+		 * subordinate buses.
+		 */
+		__pci_bus_size_bridges(bus, add_list);
 
-	pci_root_bus_distribute_available_resources(bus, add_list);
+		pci_root_bus_distribute_available_resources(bus, add_list);
 
-	/* Depth last, allocate resources and update the hardware. */
-	__pci_bus_assign_resources(bus, add_list, &fail_head);
-	if (add_list)
-		BUG_ON(!list_empty(add_list));
-	tried_times++;
+		/* Depth last, allocate resources and update the hardware. */
+		__pci_bus_assign_resources(bus, add_list, &fail_head);
+		if (add_list)
+			BUG_ON(!list_empty(add_list));
+		tried_times++;
 
-	/* Any device complain? */
-	if (list_empty(&fail_head))
-		goto dump;
+		/* Any device complain? */
+		if (list_empty(&fail_head))
+			break;
 
-	if (tried_times >= pci_try_num) {
-		if (enable_local == undefined)
-			dev_info(&bus->dev, "Some PCI device resources are unassigned, try booting with pci=realloc\n");
-		else if (enable_local == auto_enabled)
-			dev_info(&bus->dev, "Automatically enabled pci realloc, if you have problem, try booting with pci=realloc=off\n");
+		if (tried_times >= pci_try_num) {
+			if (enable_local == undefined) {
+				dev_info(&bus->dev,
+					 "Some PCI device resources are unassigned, try booting with pci=realloc\n");
+			} else if (enable_local == auto_enabled) {
+				dev_info(&bus->dev,
+					 "Automatically enabled pci realloc, if you have problem, try booting with pci=realloc=off\n");
+			}
+			free_list(&fail_head);
+			break;
+		}
 
-		free_list(&fail_head);
-		goto dump;
-	}
+		dev_info(&bus->dev, "No. %d try to assign unassigned res\n",
+			 tried_times + 1);
 
-	dev_info(&bus->dev, "No. %d try to assign unassigned res\n",
-		 tried_times + 1);
+		/* Third times and later will not check if it is leaf */
+		if (tried_times + 1 > 2)
+			rel_type = whole_subtree;
+
+		/*
+		 * Try to release leaf bridge's resources that doesn't fit
+		 * resource of child device under that bridge.
+		 */
+		list_for_each_entry(fail_res, &fail_head, list) {
+			pci_bus_release_bridge_resources(fail_res->dev->bus,
+							 fail_res->flags & PCI_RES_TYPE_MASK,
+							 rel_type);
+		}
 
-	/* Third times and later will not check if it is leaf */
-	if ((tried_times + 1) > 2)
-		rel_type = whole_subtree;
+		/* Restore size and flags */
+		list_for_each_entry(fail_res, &fail_head, list) {
+			struct resource *res = fail_res->res;
+			int idx;
 
-	/*
-	 * Try to release leaf bridge's resources that doesn't fit resource of
-	 * child device under that bridge.
-	 */
-	list_for_each_entry(fail_res, &fail_head, list)
-		pci_bus_release_bridge_resources(fail_res->dev->bus,
-						 fail_res->flags & PCI_RES_TYPE_MASK,
-						 rel_type);
+			res->start = fail_res->start;
+			res->end = fail_res->end;
+			res->flags = fail_res->flags;
 
-	/* Restore size and flags */
-	list_for_each_entry(fail_res, &fail_head, list) {
-		struct resource *res = fail_res->res;
-		int idx;
-
-		res->start = fail_res->start;
-		res->end = fail_res->end;
-		res->flags = fail_res->flags;
-
-		if (pci_is_bridge(fail_res->dev)) {
-			idx = pci_resource_num(fail_res->dev, res);
-			if (idx >= PCI_BRIDGE_RESOURCES &&
-			    idx <= PCI_BRIDGE_RESOURCE_END)
-				res->flags = 0;
+			if (pci_is_bridge(fail_res->dev)) {
+				idx = pci_resource_num(fail_res->dev, res);
+				if (idx >= PCI_BRIDGE_RESOURCES &&
+				    idx <= PCI_BRIDGE_RESOURCE_END)
+					res->flags = 0;
+			}
 		}
+		free_list(&fail_head);
 	}
-	free_list(&fail_head);
 
-	goto again;
-
-dump:
-	/* Dump the resource on buses */
 	pci_bus_dump_resources(bus);
 }

@@ -2261,62 +2262,61 @@ void pci_assign_unassigned_bridge_resources(struct pci_dev *bridge)
 	struct pci_dev_resource *fail_res;
 	int retval;
 
-again:
-	__pci_bus_size_bridges(parent, &add_list);
+	while (1) {
+		__pci_bus_size_bridges(parent, &add_list);
 
-	/*
-	 * Distribute remaining resources (if any) equally between hotplug
-	 * bridges below.  This makes it possible to extend the hierarchy
-	 * later without running out of resources.
-	 */
-	pci_bridge_distribute_available_resources(bridge, &add_list);
+		/*
+		 * Distribute remaining resources (if any) equally between
+		 * hotplug bridges below. This makes it possible to extend
+		 * the hierarchy later without running out of resources.
+		 */
+		pci_bridge_distribute_available_resources(bridge, &add_list);
 
-	__pci_bridge_assign_resources(bridge, &add_list, &fail_head);
-	BUG_ON(!list_empty(&add_list));
-	tried_times++;
+		__pci_bridge_assign_resources(bridge, &add_list, &fail_head);
+		BUG_ON(!list_empty(&add_list));
+		tried_times++;
 
-	if (list_empty(&fail_head))
-		goto enable_all;
+		if (list_empty(&fail_head))
+			break;
 
-	if (tried_times >= 2) {
-		/* Still fail, don't need to try more */
-		free_list(&fail_head);
-		goto enable_all;
-	}
+		if (tried_times >= 2) {
+			/* Still fail, don't need to try more */
+			free_list(&fail_head);
+			break;
+		}
 
-	printk(KERN_DEBUG "PCI: No. %d try to assign unassigned res\n",
-			 tried_times + 1);
+		printk(KERN_DEBUG "PCI: No. %d try to assign unassigned res\n",
+				 tried_times + 1);
 
-	/*
-	 * Try to release leaf bridge's resources that aren't big enough
-	 * to contain child device resources.
-	 */
-	list_for_each_entry(fail_res, &fail_head, list)
-		pci_bus_release_bridge_resources(fail_res->dev->bus,
-						 fail_res->flags & PCI_RES_TYPE_MASK,
-						 whole_subtree);
+		/*
+		 * Try to release leaf bridge's resources that aren't big
+		 * enough to contain child device resources.
+		 */
+		list_for_each_entry(fail_res, &fail_head, list) {
+			pci_bus_release_bridge_resources(fail_res->dev->bus,
+							 fail_res->flags & PCI_RES_TYPE_MASK,
+							 whole_subtree);
+		}
 
-	/* Restore size and flags */
-	list_for_each_entry(fail_res, &fail_head, list) {
-		struct resource *res = fail_res->res;
-		int idx;
-
-		res->start = fail_res->start;
-		res->end = fail_res->end;
-		res->flags = fail_res->flags;
-
-		if (pci_is_bridge(fail_res->dev)) {
-			idx = pci_resource_num(fail_res->dev, res);
-			if (idx >= PCI_BRIDGE_RESOURCES &&
-			    idx <= PCI_BRIDGE_RESOURCE_END)
-				res->flags = 0;
+		/* Restore size and flags */
+		list_for_each_entry(fail_res, &fail_head, list) {
+			struct resource *res = fail_res->res;
+			int idx;
+
+			res->start = fail_res->start;
+			res->end = fail_res->end;
+			res->flags = fail_res->flags;
+
+			if (pci_is_bridge(fail_res->dev)) {
+				idx = pci_resource_num(fail_res->dev, res);
+				if (idx >= PCI_BRIDGE_RESOURCES &&
+				    idx <= PCI_BRIDGE_RESOURCE_END)
+					res->flags = 0;
+			}
 		}
+		free_list(&fail_head);
 	}
-	free_list(&fail_head);
-
-	goto again;
 
-enable_all:
 	retval = pci_reenable_device(bridge);
 	if (retval)
 		pci_err(bridge, "Error reenabling bridge (%d)\n", retval);

Signature · 1 line

2.39.5

Ilpo Järvinen

author

2024-12-16 17:58 UTC #15 lore raw

PATCH 15/25 PCI: Rename retval to ret

In reply to Ilpo Järvinen

Rename 'retval' to 'ret' in pci_assign_unassigned_bridge_resources().

Signed-off-by Ilpo Järvinen <[email protected]>

---
 drivers/pci/setup-bus.c | 9 ++++-----
 1 file changed, 4 insertions(+), 5 deletions(-)

drivers/pci/setup-bus.c +4 -5

index bbe3472cfba9..38dbe8b99910 100644
@@ -2256,11 +2256,10 @@ void pci_assign_unassigned_bridge_resources(struct pci_dev *bridge)
 	struct pci_bus *parent = bridge->subordinate;
 	/* List of resources that want additional resources */
 	LIST_HEAD(add_list);
-
 	int tried_times = 0;
 	LIST_HEAD(fail_head);
 	struct pci_dev_resource *fail_res;
-	int retval;
+	int ret;
 
 	while (1) {
 		__pci_bus_size_bridges(parent, &add_list);

@@ -2317,9 +2316,9 @@ void pci_assign_unassigned_bridge_resources(struct pci_dev *bridge)
 		free_list(&fail_head);
 	}
 
-	retval = pci_reenable_device(bridge);
-	if (retval)
-		pci_err(bridge, "Error reenabling bridge (%d)\n", retval);
+	ret = pci_reenable_device(bridge);
+	if (ret)
+		pci_err(bridge, "Error reenabling bridge (%d)\n", ret);
 	pci_set_master(bridge);
 }
 EXPORT_SYMBOL_GPL(pci_assign_unassigned_bridge_resources);

Signature · 1 line

2.39.5

Ilpo Järvinen

author

2024-12-16 17:58 UTC #16 lore raw

PATCH 16/25 PCI: Consolidate assignment loop next round preparation

In reply to Ilpo Järvinen

pci_assign_unassigned_root_bus_resources() and pci_assign_unassigned_bridge_resources() have a loop that performs a number rounds to assign resources. The code to prepare for the next round is identical.

Consolidate the code that prepares for the next assignment round into pci_prepare_next_assign_round().

Signed-off-by Ilpo Järvinen <[email protected]>

---
 drivers/pci/setup-bus.c | 104 ++++++++++++++++------------------------
 1 file changed, 42 insertions(+), 62 deletions(-)

drivers/pci/setup-bus.c +42 ~~-62~~

index 38dbe8b99910..7e5985543734 100644
@@ -2135,6 +2135,45 @@ pci_root_bus_distribute_available_resources(struct pci_bus *bus,
 	}
 }
 
+static void pci_prepare_next_assign_round(struct list_head *fail_head,
+					  int tried_times,
+					  enum release_type rel_type)
+{
+	struct pci_dev_resource *fail_res;
+
+	pr_info("PCI: No. %d try to assign unassigned res\n", tried_times + 1);
+
+	/*
+	 * Try to release leaf bridge's resources that aren't big
+	 * enough to contain child device resources.
+	 */
+	list_for_each_entry(fail_res, fail_head, list) {
+		pci_bus_release_bridge_resources(fail_res->dev->bus,
+						 fail_res->flags & PCI_RES_TYPE_MASK,
+						 rel_type);
+	}
+
+	/* Restore size and flags */
+	list_for_each_entry(fail_res, fail_head, list) {
+		struct resource *res = fail_res->res;
+		struct pci_dev *dev = fail_res->dev;
+		int idx = pci_resource_num(dev, res);
+
+		res->start = fail_res->start;
+		res->end = fail_res->end;
+		res->flags = fail_res->flags;
+
+		if (!pci_is_bridge(dev))
+			continue;
+
+		if (idx >= PCI_BRIDGE_RESOURCES &&
+		    idx <= PCI_BRIDGE_RESOURCE_END)
+			res->flags = 0;
+	}
+
+	free_list(fail_head);
+}
+
 /*
  * First try will not touch PCI bridge res.
  * Second and later try will clear small leaf bridge res.

@@ -2148,7 +2187,6 @@ void pci_assign_unassigned_root_bus_resources(struct pci_bus *bus)
 	int tried_times = 0;
 	enum release_type rel_type = leaf_only;
 	LIST_HEAD(fail_head);
-	struct pci_dev_resource *fail_res;
 	int pci_try_num = 1;
 	enum enable_type enable_local;

@@ -2199,40 +2237,11 @@ void pci_assign_unassigned_root_bus_resources(struct pci_bus *bus)
 			break;
 		}
 
-		dev_info(&bus->dev, "No. %d try to assign unassigned res\n",
-			 tried_times + 1);
-
 		/* Third times and later will not check if it is leaf */
 		if (tried_times + 1 > 2)
 			rel_type = whole_subtree;
 
-		/*
-		 * Try to release leaf bridge's resources that doesn't fit
-		 * resource of child device under that bridge.
-		 */
-		list_for_each_entry(fail_res, &fail_head, list) {
-			pci_bus_release_bridge_resources(fail_res->dev->bus,
-							 fail_res->flags & PCI_RES_TYPE_MASK,
-							 rel_type);
-		}
-
-		/* Restore size and flags */
-		list_for_each_entry(fail_res, &fail_head, list) {
-			struct resource *res = fail_res->res;
-			int idx;
-
-			res->start = fail_res->start;
-			res->end = fail_res->end;
-			res->flags = fail_res->flags;
-
-			if (pci_is_bridge(fail_res->dev)) {
-				idx = pci_resource_num(fail_res->dev, res);
-				if (idx >= PCI_BRIDGE_RESOURCES &&
-				    idx <= PCI_BRIDGE_RESOURCE_END)
-					res->flags = 0;
-			}
-		}
-		free_list(&fail_head);
+		pci_prepare_next_assign_round(&fail_head, tried_times, rel_type);
 	}
 
 	pci_bus_dump_resources(bus);

@@ -2258,7 +2267,6 @@ void pci_assign_unassigned_bridge_resources(struct pci_dev *bridge)
 	LIST_HEAD(add_list);
 	int tried_times = 0;
 	LIST_HEAD(fail_head);
-	struct pci_dev_resource *fail_res;
 	int ret;
 
 	while (1) {

@@ -2284,36 +2292,8 @@ void pci_assign_unassigned_bridge_resources(struct pci_dev *bridge)
 			break;
 		}
 
-		printk(KERN_DEBUG "PCI: No. %d try to assign unassigned res\n",
-				 tried_times + 1);
-
-		/*
-		 * Try to release leaf bridge's resources that aren't big
-		 * enough to contain child device resources.
-		 */
-		list_for_each_entry(fail_res, &fail_head, list) {
-			pci_bus_release_bridge_resources(fail_res->dev->bus,
-							 fail_res->flags & PCI_RES_TYPE_MASK,
-							 whole_subtree);
-		}
-
-		/* Restore size and flags */
-		list_for_each_entry(fail_res, &fail_head, list) {
-			struct resource *res = fail_res->res;
-			int idx;
-
-			res->start = fail_res->start;
-			res->end = fail_res->end;
-			res->flags = fail_res->flags;
-
-			if (pci_is_bridge(fail_res->dev)) {
-				idx = pci_resource_num(fail_res->dev, res);
-				if (idx >= PCI_BRIDGE_RESOURCES &&
-				    idx <= PCI_BRIDGE_RESOURCE_END)
-					res->flags = 0;
-			}
-		}
-		free_list(&fail_head);
+		pci_prepare_next_assign_round(&fail_head, tried_times,
+					      whole_subtree);
 	}
 
 	ret = pci_reenable_device(bridge);

Signature · 1 line

2.39.5

Ilpo Järvinen

author

2024-12-16 17:59 UTC #17 lore raw

PATCH 17/25 PCI: Remove wrong comment from pci_reassign_resource()

In reply to Ilpo Järvinen

The commit a4ac9fea016f ("PCI : Calculate right add_size") removed including min_align into new_size in pci_reassign_resource() which is correct thing to do. However, it also added a snakeoil comment that the resource would already be aligned with min_align which is incorrect.

A resource that is assigned earlier is aligned with the old alignment, NOT with the new requested alignment (min_align) until later deep within the reassignment callchain. Thus, remove the incorrect comment.

Signed-off-by Ilpo Järvinen <[email protected]>

Cc Yinghai Lu <[email protected]>

---
 drivers/pci/setup-res.c | 1 -
 1 file changed, 1 deletion(-)

drivers/pci/setup-res.c +0 -1

index 79c7ef349856..a862b5d769dc 100644
@@ -387,7 +387,6 @@ int pci_reassign_resource(struct pci_dev *dev, int resno,
 		return -EINVAL;
 	}
 
-	/* already aligned with min_align */
 	new_size = resource_size(res) + addsize;
 	ret = _pci_assign_resource(dev, resno, new_size, min_align);
 	if (ret) {

Signature · 1 line

2.39.5

Ilpo Järvinen

author

2024-12-16 17:59 UTC #18 lore raw

PATCH 18/25 PCI: Add restore_dev_resource()

In reply to Ilpo Järvinen

Resource fitting needs to restore the saved dev resources in a few places. Add a restore_dev_resource() helper for that.

Signed-off-by Ilpo Järvinen <[email protected]>

---
 drivers/pci/setup-bus.c | 35 +++++++++++++++--------------------
 1 file changed, 15 insertions(+), 20 deletions(-)

drivers/pci/setup-bus.c +15 ~~-20~~

index 7e5985543734..d30e8a5a0311 100644
@@ -127,6 +127,15 @@ static resource_size_t get_res_add_align(struct list_head *head,
 	return dev_res ? dev_res->min_align : 0;
 }
 
+static void restore_dev_resource(struct pci_dev_resource *dev_res)
+{
+	struct resource *res = dev_res->res;
+
+	res->start = dev_res->start;
+	res->end = dev_res->end;
+	res->flags = dev_res->flags;
+}
+
 static bool pdev_resources_assignable(struct pci_dev *dev)
 {
 	u16 class = dev->class >> 8, command;

@@ -473,13 +482,8 @@ static void __assign_resources_sorted(struct list_head *head,
 			release_resource(res);
 	}
 	/* Restore start/end/flags from saved list */
-	list_for_each_entry(save_res, &save_head, list) {
-		res = save_res->res;
-
-		res->start = save_res->start;
-		res->end = save_res->end;
-		res->flags = save_res->flags;
-	}
+	list_for_each_entry(save_res, &save_head, list)
+		restore_dev_resource(save_res);
 	free_list(&save_head);
 
 requested_and_reassign:

@@ -2159,9 +2163,7 @@ static void pci_prepare_next_assign_round(struct list_head *fail_head,
 		struct pci_dev *dev = fail_res->dev;
 		int idx = pci_resource_num(dev, res);
 
-		res->start = fail_res->start;
-		res->end = fail_res->end;
-		res->flags = fail_res->flags;
+		restore_dev_resource(fail_res);
 
 		if (!pci_is_bridge(dev))
 			continue;

@@ -2378,13 +2380,8 @@ int pci_reassign_bridge_resources(struct pci_dev *bridge, unsigned long type)
 
 cleanup:
 	/* Restore size and flags */
-	list_for_each_entry(dev_res, &failed, list) {
-		struct resource *res = dev_res->res;
-
-		res->start = dev_res->start;
-		res->end = dev_res->end;
-		res->flags = dev_res->flags;
-	}
+	list_for_each_entry(dev_res, &failed, list)
+		restore_dev_resource(dev_res);
 	free_list(&failed);
 
 	/* Revert to the old configuration */

@@ -2394,9 +2391,7 @@ int pci_reassign_bridge_resources(struct pci_dev *bridge, unsigned long type)
 		bridge = dev_res->dev;
 		i = pci_resource_num(bridge, res);
 
-		res->start = dev_res->start;
-		res->end = dev_res->end;
-		res->flags = dev_res->flags;
+		restore_dev_resource(dev_res);
 
 		pci_claim_resource(bridge, i);
 		pci_setup_bridge(bridge->subordinate);

Signature · 1 line

2.39.5

Ilpo Järvinen

author

2024-12-16 17:59 UTC #19 lore raw

PATCH 19/25 PCI: Extend enable to check for any optional resource

In reply to Ilpo Järvinen

pci_enable_resources() checks if device's io and mem resources are all assigned and disallows enable if any resource failed to assign (*) but makes an exception for the case of disabled extension ROM. There are other optional resources, however.

Add pci_resource_is_optional() and use it instead of pci_resource_is_disabled_rom() to cover also IOV resources that are also optional as per pbus_size_mem().

As there will be more users of pci_resource_is_optional() inside setup-bus.c in changes coming up after this one, the function is placed there.

(*) In practice, resource fitting code calls reset_resource() for any resource it fails to assign which clears resource's ->flags causing pci_enable_resources() to never detect failed resource assignments. This seems undesirable internal logic inconsistency, effectively reset_resource() prevents pci_enable_resources() from functioning as intended. This is one step of many that will be needed towards removing reset_resource().

Signed-off-by Ilpo Järvinen <[email protected]>

---
 drivers/pci/pci.h       |  1 +
 drivers/pci/setup-bus.c | 12 ++++++++++++
 drivers/pci/setup-res.c |  3 +--
 3 files changed, 14 insertions(+), 2 deletions(-)

drivers/pci/pci.h +1 -0

index 0b722d158b6a..efdb9af8e256 100644
@@ -329,6 +329,7 @@ void pci_walk_bus_locked(struct pci_bus *top,
 			 void *userdata);
 
 const char *pci_resource_name(struct pci_dev *dev, unsigned int i);
+bool pci_resource_is_optional(const struct pci_dev *dev, int resno);
 
 /**
  * pci_resource_num - Reverse lookup resource number from device resources

drivers/pci/setup-bus.c +12 -0

index d30e8a5a0311..a4e40916e2fc 100644
@@ -206,6 +206,18 @@ static void pdev_sort_resources(struct pci_dev *dev, struct list_head *head)
 	}
 }
 
+bool pci_resource_is_optional(const struct pci_dev *dev, int resno)
+{
+	const struct resource *res = pci_resource_n(dev, resno);
+
+	if (pci_resource_is_iov(resno))
+		return true;
+	if (resno == PCI_ROM_RESOURCE && !(res->flags & IORESOURCE_ROM_ENABLE))
+		return true;
+
+	return false;
+}
+
 static inline void reset_resource(struct resource *res)
 {
 	res->start = 0;

drivers/pci/setup-res.c +1 -2

index a862b5d769dc..041c881c3c9c 100644
@@ -494,8 +494,7 @@ int pci_enable_resources(struct pci_dev *dev, int mask)
 
 		if (!(r->flags & (IORESOURCE_IO | IORESOURCE_MEM)))
 			continue;
-		if ((i == PCI_ROM_RESOURCE) &&
-				(!(r->flags & IORESOURCE_ROM_ENABLE)))
+		if (pci_resource_is_optional(dev, i))
 			continue;
 
 		if (r->flags & IORESOURCE_UNSET) {

Signature · 1 line

2.39.5

Ilpo Järvinen

author

2024-12-16 17:59 UTC #20 lore raw

PATCH 20/25 PCI: Always have realloc_head in __assign_resources_sorted()

In reply to Ilpo Järvinen

Add a dummy list to always have a non-NULL realloc head in __assign_resources_sorted() as it allows only checking list_empty().

In future, it would be good to ensure all callers provide a valid realloc_head but that is relatively complex to do in practice and not necessary for the subsequent optional resource handling fix.

Signed-off-by Ilpo Järvinen <[email protected]>

---
 drivers/pci/setup-bus.c | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)

drivers/pci/setup-bus.c +6 -2

index a4e40916e2fc..a10acf4671ef 100644
@@ -400,14 +400,18 @@ static void __assign_resources_sorted(struct list_head *head,
 	 */
 	LIST_HEAD(save_head);
 	LIST_HEAD(local_fail_head);
+	LIST_HEAD(dummy_head);
 	struct pci_dev_resource *save_res;
 	struct pci_dev_resource *dev_res, *tmp_res, *dev_res2;
 	struct resource *res;
 	unsigned long fail_type;
 	resource_size_t add_align, align;
 
+	if (!realloc_head)
+		realloc_head = &dummy_head;
+
 	/* Check if optional add_size is there */
-	if (!realloc_head || list_empty(realloc_head))
+	if (list_empty(realloc_head))
 		goto requested_and_reassign;
 
 	/* Save original start, end, flags etc at first */

@@ -503,7 +507,7 @@ static void __assign_resources_sorted(struct list_head *head,
 	assign_requested_resources_sorted(head, fail_head);
 
 	/* Try to satisfy any additional optional resource requests */
-	if (realloc_head)
+	if (!list_empty(realloc_head))
 		reassign_resources_sorted(realloc_head, head);
 
 out:

Signature · 1 line

2.39.5

Ilpo Järvinen

author

2024-12-16 17:59 UTC #21 lore raw

PATCH 21/25 PCI: Indicate optional resource assignment failures

In reply to Ilpo Järvinen

Add pci_dbg() to tell the an assignment failure was for an optional resource and reword existing one about resource resize to tell the change was optional.

Signed-off-by Ilpo Järvinen <[email protected]>

---
 drivers/pci/setup-bus.c | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)

drivers/pci/setup-bus.c +6 -2

index a10acf4671ef..500652eef17b 100644
@@ -273,13 +273,17 @@ static void reassign_resources_sorted(struct list_head *realloc_head,
 		align = add_res->min_align;
 		if (!resource_size(res)) {
 			resource_set_range(res, align, add_size);
-			if (pci_assign_resource(dev, idx))
+			if (pci_assign_resource(dev, idx)) {
+				pci_dbg(dev,
+					"%s %pR: ignoring failure in optional allocation\n",
+					res_name, res);
 				reset_resource(res);
+			}
 		} else {
 			res->flags |= add_res->flags &
 				 (IORESOURCE_STARTALIGN|IORESOURCE_SIZEALIGN);
 			if (pci_reassign_resource(dev, idx, add_size, align))
-				pci_info(dev, "%s %pR: failed to add %llx\n",
+				pci_info(dev, "%s %pR: failed to add optional %llx\n",
 					 res_name, res,
 					 (unsigned long long) add_size);
 		}

Signature · 1 line

2.39.5

Ilpo Järvinen

author

2024-12-16 17:59 UTC #22 lore raw

PATCH 22/25 PCI: Add debug print when releasing resources before retry

In reply to Ilpo Järvinen

PCI resource fitting is somewhat hard to track because it performs many actions without logging them. In the case inside __assign_resources_sorted(), the resources are released before resource assignment is going to be retried in a different order. That is just one level of retries the resource fitting performs overall so tracking it through repeated assignments or failures of a resource gets messy rather quickly.

Simply make the release announced explicitly using pci_dbg() so it is clear what is going on with each resource.

Signed-off-by Ilpo Järvinen <[email protected]>

---
 drivers/pci/setup-bus.c | 14 ++++++++++++--
 1 file changed, 12 insertions(+), 2 deletions(-)

drivers/pci/setup-bus.c +12 -2

index 500652eef17b..5a3b320f1511 100644
@@ -408,6 +408,9 @@ static void __assign_resources_sorted(struct list_head *head,
 	struct pci_dev_resource *save_res;
 	struct pci_dev_resource *dev_res, *tmp_res, *dev_res2;
 	struct resource *res;
+	struct pci_dev *dev;
+	const char *res_name;
+	int idx;
 	unsigned long fail_type;
 	resource_size_t add_align, align;

@@ -497,9 +500,16 @@ static void __assign_resources_sorted(struct list_head *head,
 	/* Release assigned resource */
 	list_for_each_entry(dev_res, head, list) {
 		res = dev_res->res;
+		dev = dev_res->dev;
+
+		if (!res->parent)
+			continue;
+
+		idx = pci_resource_num(dev, res);
+		res_name = pci_resource_name(dev, idx);
+		pci_dbg(dev, "%s %pR: releasing\n", res_name, res);
 
-		if (res->parent)
-			release_resource(res);
+		release_resource(res);
 	}
 	/* Restore start/end/flags from saved list */
 	list_for_each_entry(save_res, &save_head, list)

Signature · 1 line

2.39.5

Ilpo Järvinen

author

2024-12-16 17:59 UTC #23 lore raw

PATCH 23/25 PCI: Use res->parent to check is resource is assigned

In reply to Ilpo Järvinen

reassign_resources_sorted() uses resource_size() to select between pci_assign_resource() and pci_reassign_resource(). Due to twisted way bridge window sizing in pbus_size_mem() sets resource sizes to 0, it works to match into IOV resources but that is going to be changed by an upcoming change.

Replace resource_size() check with res->parent check that is the true dividing line in between whether assign or reassign function should be used for the resource.

Signed-off-by Ilpo Järvinen <[email protected]>

---
 drivers/pci/setup-bus.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

drivers/pci/setup-bus.c +1 -1

index 5a3b320f1511..fec7d68fb971 100644
@@ -271,7 +271,7 @@ static void reassign_resources_sorted(struct list_head *realloc_head,
 		res_name = pci_resource_name(dev, idx);
 		add_size = add_res->add_size;
 		align = add_res->min_align;
-		if (!resource_size(res)) {
+		if (!res->parent) {
 			resource_set_range(res, align, add_size);
 			if (pci_assign_resource(dev, idx)) {
 				pci_dbg(dev,

Signature · 1 line

2.39.5

Ilpo Järvinen

author

2024-12-16 18:00 UTC #24 lore raw

PATCH 24/25 PCI: Perform reset_resource() and build fail list in sync

In reply to Ilpo Järvinen

Resetting resource is problematic as it prevent attempting to allocate the resource later, unless something in between restores the resource. Similarly, if fail_head does not contain all resources that were reset, those resource cannot be restored later.

The entire reset/restore cycle adds complexity and leaving resources into reseted state causes issues to other code such as for checks done in pci_enable_resources(). Take a small step towards not resetting resources by delaying reset until the end of resource assignment and build failure list (fail_head) in sync with the reset to avoid leaving behind resources that cannot be restored (for the case where the caller provides fail_head in the first place to allow restore somewhere in the callchain, as is not all callers pass non-NULL fail_head).

The Expansion ROM check is temporarily left in place while building the failure list until the upcoming change which reworks optional resource handling.

Ideally, whole resource reset could be removed but doing that in a big step would make the impact non-tractable due to complexity of all related code.

Signed-off-by Ilpo Järvinen <[email protected]>

---
 drivers/pci/setup-bus.c | 35 +++++++++++++++++++++++++++++------
 1 file changed, 29 insertions(+), 6 deletions(-)

drivers/pci/setup-bus.c +29 -6

index fec7d68fb971..b61f24a5cfa5 100644
@@ -252,9 +252,14 @@ static void reassign_resources_sorted(struct list_head *realloc_head,
 
 		res = add_res->res;
 		dev = add_res->dev;
+		idx = pci_resource_num(dev, res);
 
-		/* Skip resource that has been reset */
-		if (!res->flags)
+		/*
+		 * Skip resource that failed the earlier assignment and is
+		 * not optional as it would just fail again.
+		 */
+		if (!res->parent && resource_size(res) &&
+		    !pci_resource_is_optional(dev, idx))
 			goto out;
 
 		/* Skip this resource if not found in head list */

@@ -267,7 +272,6 @@ static void reassign_resources_sorted(struct list_head *realloc_head,
 		if (!found_match) /* Just skip */
 			continue;
 
-		idx = pci_resource_num(dev, res);
 		res_name = pci_resource_name(dev, idx);
 		add_size = add_res->add_size;
 		align = add_res->min_align;

@@ -277,7 +281,6 @@ static void reassign_resources_sorted(struct list_head *realloc_head,
 				pci_dbg(dev,
 					"%s %pR: ignoring failure in optional allocation\n",
 					res_name, res);
-				reset_resource(res);
 			}
 		} else {
 			res->flags |= add_res->flags &

@@ -332,7 +335,6 @@ static void assign_requested_resources_sorted(struct list_head *head,
 						    0 /* don't care */,
 						    0 /* don't care */);
 			}
-			reset_resource(res);
 		}
 	}
 }

@@ -518,13 +520,34 @@ static void __assign_resources_sorted(struct list_head *head,
 
 requested_and_reassign:
 	/* Satisfy the must-have resource requests */
-	assign_requested_resources_sorted(head, fail_head);
+	assign_requested_resources_sorted(head, NULL);
 
 	/* Try to satisfy any additional optional resource requests */
 	if (!list_empty(realloc_head))
 		reassign_resources_sorted(realloc_head, head);
 
 out:
+	/* Reset any failed resource, cannot use fail_head as it can be NULL. */
+	list_for_each_entry(dev_res, head, list) {
+		res = dev_res->res;
+		dev = dev_res->dev;
+
+		if (res->parent)
+			continue;
+
+		/*
+		 * If the failed resource is a ROM BAR and it will
+		 * be enabled later, don't add it to the list.
+		 */
+		if (fail_head && !pci_resource_is_disabled_rom(res, idx)) {
+			add_to_list(fail_head, dev, res,
+				    0 /* don't care */,
+				    0 /* don't care */);
+		}
+
+		reset_resource(res);
+	}
+
 	free_list(head);
 }

Signature · 1 line

2.39.5

Guenter Roeck

2025-04-01 02:35 UTC #25 lore raw

Re: [PATCH 24/25] PCI: Perform reset_resource() and build fail list in sync

Hi,

Quoted 18 lines · Ilpo Järvinen

Resetting resource is problematic as it prevent attempting to allocate the resource later, unless something in between restores the resource. Similarly, if fail_head does not contain all resources that were reset, those resource cannot be restored later.
The entire reset/restore cycle adds complexity and leaving resources into reseted state causes issues to other code such as for checks done in pci_enable_resources(). Take a small step towards not resetting resources by delaying reset until the end of resource assignment and build failure list (fail_head) in sync with the reset to avoid leaving behind resources that cannot be restored (for the case where the caller provides fail_head in the first place to allow restore somewhere in the callchain, as is not all callers pass non-NULL fail_head).
The Expansion ROM check is temporarily left in place while building the failure list until the upcoming change which reworks optional resource handling.

Quoted 5 lines

Ideally, whole resource reset could be removed but doing that in a big step would make the impact non-tractable due to complexity of all related code.
Signed-off-by Ilpo Järvinen <[email protected]>

With this patch in the mainline kernel, all mips:boston qemu emulations fail when running a 64-bit little endian configuration (64r6el_defconfig).

The problem is that the PCI based IDE/ATA controller is not initialized. There are a number of pci error messages.

pci_bus 0002:01: extended config space not accessible
pci 0002:01:00.0: [8086:2922] type 00 class 0x010601 conventional PCI endpoint pci 0002:01:00.0: BAR 4 [io 0x0000-0x001f]
pci 0002:01:00.0: BAR 5 [mem 0x00000000-0x00000fff]
pci 0002:00:00.0: PCI bridge to [bus 01-ff]
pci_bus 0002:01: busn_res: [bus 01-ff] end is updated to 01
pci 0002:00:00.0: bridge window [mem 0x16000000-0x160fffff]: assigned
pci 0002:00:00.0: bridge window [mem size 0x00100000 64bit pref]: can't assign; no space pci 0002:00:00.0: bridge window [mem size 0x00100000 64bit pref]: failed to assign pci 0002:00:00.0: bridge window [io size 0x1000]: can't assign; no space pci 0002:00:00.0: bridge window [io size 0x1000]: failed to assign
pci 0002:00:00.0: bridge window [mem size 0x00100000]: can't assign; bogus alignment pci 0002:00:00.0: bridge window [mem 0x16000000-0x160fffff 64bit pref]: assigned pci 0002:00:00.0: bridge window [io size 0x1000]: can't assign; no space pci 0002:00:00.0: bridge window [io size 0x1000]: failed to assign
pci 0002:01:00.0: BAR 5 [mem size 0x00001000]: can't assign; no space
pci 0002:01:00.0: BAR 5 [mem size 0x00001000]: failed to assign
pci 0002:01:00.0: BAR 4 [io size 0x0020]: can't assign; no space
pci 0002:01:00.0: BAR 4 [io size 0x0020]: failed to assign
pci 0002:01:00.0: BAR 5 [mem size 0x00001000]: can't assign; no space
pci 0002:01:00.0: BAR 5 [mem size 0x00001000]: failed to assign
pci 0002:01:00.0: BAR 4 [io size 0x0020]: can't assign; no space
pci 0002:01:00.0: BAR 4 [io size 0x0020]: failed to assign
pci 0002:00:00.0: PCI bridge to [bus 01]
pci 0002:00:00.0: bridge window [mem 0x16000000-0x160fffff 64bit pref]
pci_bus 0002:00: Some PCI device resources are unassigned, try booting with pci=realloc pci_bus 0002:00: resource 4 [mem 0x16000000-0x160fffff]
pci_bus 0002:01: resource 2 [mem 0x16000000-0x160fffff 64bit pref]
...
pci 0002:00:00.0: enabling device (0000 -> 0002)
ahci 0002:01:00.0: probe with driver ahci failed with error -12

Bisect points to this patch. Reverting it together with "PCI: Rework optional resource handling" fixes the problem. For comparison, after reverting the offending patches, the log messages are as follows.

pci_bus 0002:00: root bus resource [bus 00-ff]
pci_bus 0002:00: root bus resource [mem 0x16000000-0x160fffff]
pci 0002:00:00.0: [10ee:7021] type 01 class 0x060400 PCIe Root Complex Integrated Endpoint pci 0002:00:00.0: PCI bridge to [bus 00]

pci 0002:00:00.0:   bridge window [io  0x0000-0x0fff]
pci 0002:00:00.0:   bridge window [mem 0x00000000-0x000fffff]
pci 0002:00:00.0:   bridge window [mem 0x00000000-0x000fffff 64bit pref]

pci 0002:00:00.0: enabling Extended Tags
pci 0002:00:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring pci_bus 0002:01: extended config space not accessible
pci 0002:01:00.0: [8086:2922] type 00 class 0x010601 conventional PCI endpoint pci 0002:01:00.0: BAR 4 [io 0x0000-0x001f]
pci 0002:01:00.0: BAR 5 [mem 0x00000000-0x00000fff]
pci 0002:00:00.0: PCI bridge to [bus 01-ff]
pci_bus 0002:01: busn_res: [bus 01-ff] end is updated to 01
pci 0002:00:00.0: bridge window [mem 0x16000000-0x160fffff]: assigned
pci 0002:00:00.0: bridge window [mem size 0x00100000 64bit pref]: can't assign; no space pci 0002:00:00.0: bridge window [mem size 0x00100000 64bit pref]: failed to assign pci 0002:00:00.0: bridge window [io size 0x1000]: can't assign; no space pci 0002:00:00.0: bridge window [io size 0x1000]: failed to assign
pci 0002:01:00.0: BAR 5 [mem 0x16000000-0x16000fff]: assigned
pci 0002:01:00.0: BAR 4 [io size 0x0020]: can't assign; no space
pci 0002:01:00.0: BAR 4 [io size 0x0020]: failed to assign
pci 0002:00:00.0: PCI bridge to [bus 01]
pci 0002:00:00.0: bridge window [mem 0x16000000-0x160fffff]
pci_bus 0002:00: Some PCI device resources are unassigned, try booting with pci=realloc pci_bus 0002:00: resource 4 [mem 0x16000000-0x160fffff]
pci_bus 0002:01: resource 1 [mem 0x16000000-0x160fffff]
...
pci 0002:00:00.0: enabling device (0000 -> 0002)
ahci 0002:01:00.0: enabling device (0000 -> 0002)
ahci 0002:01:00.0: AHCI vers 0001.0000, 32 command slots, 1.5 Gbps, SATA mode ahci 0002:01:00.0: 6/6 ports implemented (port mask 0x3f)
ahci 0002:01:00.0: flags: 64bit ncq only

Bisect log is attached for reference.

Guenter
---
# bad: [609706855d90bcab6080ba2cd030b9af322a1f0c] Merge tag 'trace-latency-v6.15-3' of git://git.kernel.org/pub/scm/linux/kernel/git/trace/linux-trace
# good: [acb4f33713b9f6cadb6143f211714c343465411c] Merge tag 'm68knommu-for-v6.15' of git://git.kernel.org/pub/scm/linux/kernel/git/gerg/m68knommu
git bisect start 'HEAD' 'acb4f33713b9'
# good: [cf05922d63e2ae6a9b1b52ff5236a44c3b29f78c] Merge tag 'drm-intel-gt-next-2025-03-12' of https://gitlab.freedesktop.org/drm/i915/kernel into drm-next git bisect good cf05922d63e2ae6a9b1b52ff5236a44c3b29f78c
# bad: [93d52288679e29aaa44a6f12d5a02e8a90e742c5] Merge tag 'backlight-next-6.15' of git://git.kernel.org/pub/scm/linux/kernel/git/lee/backlight
git bisect bad 93d52288679e29aaa44a6f12d5a02e8a90e742c5
# bad: [e5e0e6bebef3a21081fd1057c40468d4cff1a60d] Merge tag 'v6.15-p1' of git://git.kernel.org/pub/scm/linux/kernel/git/herbert/crypto-2.6
git bisect bad e5e0e6bebef3a21081fd1057c40468d4cff1a60d
# bad: [dea140198b846f7432d78566b7b0b83979c72c2b] Merge branch 'pci/misc'
git bisect bad dea140198b846f7432d78566b7b0b83979c72c2b
# bad: [a113afb84ae63ec4c893bc3204945ef6f3bb89f7] Merge branch 'pci/endpoint'
git bisect bad a113afb84ae63ec4c893bc3204945ef6f3bb89f7
# good: [a7a8e7996c1c114b50df5599229b1e7be38be3db] Merge branch 'pci/reset'
git bisect good a7a8e7996c1c114b50df5599229b1e7be38be3db
# bad: [7d4bcc0f2631e4ee10b5bcfff24a423d1c3c02a3] PCI: Move resource reassignment func declarations into pci/pci.h
git bisect bad 7d4bcc0f2631e4ee10b5bcfff24a423d1c3c02a3
# good: [acba174d2e754346c07578ad2220258706a203e2] PCI: Use while loop and break instead of gotos
git bisect good acba174d2e754346c07578ad2220258706a203e2
# good: [8884b5637b794ae541e8d6fb72102b1d8dba2b8d] PCI: Add debug print when releasing resources before retry
git bisect good 8884b5637b794ae541e8d6fb72102b1d8dba2b8d
# bad: [5af473941b56189423a7d16c05efabaf77299847] PCI: Increase Resizable BAR support from 512 GB to 128 TB
git bisect bad 5af473941b56189423a7d16c05efabaf77299847
# bad: [96336ec702643aec2addb3b1cdb81d687fe362f0] PCI: Perform reset_resource() and build fail list in sync
git bisect bad 96336ec702643aec2addb3b1cdb81d687fe362f0
# good: [e89df6d2beae847e931d84a190b192dfac41eba7] PCI: Use res->parent to check if resource is assigned
git bisect good e89df6d2beae847e931d84a190b192dfac41eba7

# first bad commit: [96336ec702643aec2addb3b1cdb81d687fe362f0] PCI: Perform reset_resource() and build fail list in sync

Ilpo Järvinen

author

2025-04-01 10:18 UTC #26 lore raw

Re: [PATCH 24/25] PCI: Perform reset_resource() and build fail list in sync

Quoted 93 lines · Guenter Roeck

Quoted 18 lines · Ilpo Järvinen
Resetting resource is problematic as it prevent attempting to allocate the resource later, unless something in between restores the resource. Similarly, if fail_head does not contain all resources that were reset, those resource cannot be restored later.
The entire reset/restore cycle adds complexity and leaving resources into reseted state causes issues to other code such as for checks done in pci_enable_resources(). Take a small step towards not resetting resources by delaying reset until the end of resource assignment and build failure list (fail_head) in sync with the reset to avoid leaving behind resources that cannot be restored (for the case where the caller provides fail_head in the first place to allow restore somewhere in the callchain, as is not all callers pass non-NULL fail_head).
The Expansion ROM check is temporarily left in place while building the failure list until the upcoming change which reworks optional resource handling.
Quoted 5 lines
Ideally, whole resource reset could be removed but doing that in a big step would make the impact non-tractable due to complexity of all related code.
Signed-off-by Ilpo Järvinen <[email protected]>
With this patch in the mainline kernel, all mips:boston qemu emulations fail when running a 64-bit little endian configuration (64r6el_defconfig).
The problem is that the PCI based IDE/ATA controller is not initialized. There are a number of pci error messages.
pci_bus 0002:01: extended config space not accessible
pci 0002:01:00.0: [8086:2922] type 00 class 0x010601 conventional PCI endpoint pci 0002:01:00.0: BAR 4 [io 0x0000-0x001f]
pci 0002:01:00.0: BAR 5 [mem 0x00000000-0x00000fff]
pci 0002:00:00.0: PCI bridge to [bus 01-ff]
pci_bus 0002:01: busn_res: [bus 01-ff] end is updated to 01
pci 0002:00:00.0: bridge window [mem 0x16000000-0x160fffff]: assigned
pci 0002:00:00.0: bridge window [mem size 0x00100000 64bit pref]: can't assign; no space pci 0002:00:00.0: bridge window [mem size 0x00100000 64bit pref]: failed to assign pci 0002:00:00.0: bridge window [io size 0x1000]: can't assign; no space pci 0002:00:00.0: bridge window [io size 0x1000]: failed to assign
pci 0002:00:00.0: bridge window [mem size 0x00100000]: can't assign; bogus alignment pci 0002:00:00.0: bridge window [mem 0x16000000-0x160fffff 64bit pref]: assigned pci 0002:00:00.0: bridge window [io size 0x1000]: can't assign; no space pci 0002:00:00.0: bridge window [io size 0x1000]: failed to assign
pci 0002:01:00.0: BAR 5 [mem size 0x00001000]: can't assign; no space
pci 0002:01:00.0: BAR 5 [mem size 0x00001000]: failed to assign
pci 0002:01:00.0: BAR 4 [io size 0x0020]: can't assign; no space
pci 0002:01:00.0: BAR 4 [io size 0x0020]: failed to assign
pci 0002:01:00.0: BAR 5 [mem size 0x00001000]: can't assign; no space
pci 0002:01:00.0: BAR 5 [mem size 0x00001000]: failed to assign
pci 0002:01:00.0: BAR 4 [io size 0x0020]: can't assign; no space
pci 0002:01:00.0: BAR 4 [io size 0x0020]: failed to assign
pci 0002:00:00.0: PCI bridge to [bus 01]
pci 0002:00:00.0: bridge window [mem 0x16000000-0x160fffff 64bit pref]
pci_bus 0002:00: Some PCI device resources are unassigned, try booting with pci=realloc pci_bus 0002:00: resource 4 [mem 0x16000000-0x160fffff]
pci_bus 0002:01: resource 2 [mem 0x16000000-0x160fffff 64bit pref]
...
pci 0002:00:00.0: enabling device (0000 -> 0002)
ahci 0002:01:00.0: probe with driver ahci failed with error -12
Bisect points to this patch. Reverting it together with "PCI: Rework optional resource handling" fixes the problem. For comparison, after reverting the offending patches, the log messages are as follows.
pci_bus 0002:00: root bus resource [bus 00-ff]
pci_bus 0002:00: root bus resource [mem 0x16000000-0x160fffff]
pci 0002:00:00.0: [10ee:7021] type 01 class 0x060400 PCIe Root Complex Integrated Endpoint pci 0002:00:00.0: PCI bridge to [bus 00]
pci 0002:00:00.0:   bridge window [io  0x0000-0x0fff]
pci 0002:00:00.0:   bridge window [mem 0x00000000-0x000fffff]
pci 0002:00:00.0:   bridge window [mem 0x00000000-0x000fffff 64bit pref]
pci 0002:00:00.0: enabling Extended Tags
pci 0002:00:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring pci_bus 0002:01: extended config space not accessible
pci 0002:01:00.0: [8086:2922] type 00 class 0x010601 conventional PCI endpoint pci 0002:01:00.0: BAR 4 [io 0x0000-0x001f]
pci 0002:01:00.0: BAR 5 [mem 0x00000000-0x00000fff]
pci 0002:00:00.0: PCI bridge to [bus 01-ff]
pci_bus 0002:01: busn_res: [bus 01-ff] end is updated to 01
pci 0002:00:00.0: bridge window [mem 0x16000000-0x160fffff]: assigned
pci 0002:00:00.0: bridge window [mem size 0x00100000 64bit pref]: can't assign; no space pci 0002:00:00.0: bridge window [mem size 0x00100000 64bit pref]: failed to assign pci 0002:00:00.0: bridge window [io size 0x1000]: can't assign; no space pci 0002:00:00.0: bridge window [io size 0x1000]: failed to assign
pci 0002:01:00.0: BAR 5 [mem 0x16000000-0x16000fff]: assigned
pci 0002:01:00.0: BAR 4 [io size 0x0020]: can't assign; no space
pci 0002:01:00.0: BAR 4 [io size 0x0020]: failed to assign
pci 0002:00:00.0: PCI bridge to [bus 01]
pci 0002:00:00.0: bridge window [mem 0x16000000-0x160fffff]
pci_bus 0002:00: Some PCI device resources are unassigned, try booting with pci=realloc

Quoted 8 lines

pci_bus 0002:00: resource 4 [mem 0x16000000-0x160fffff]
pci_bus 0002:01: resource 1 [mem 0x16000000-0x160fffff]
...
pci 0002:00:00.0: enabling device (0000 -> 0002)
ahci 0002:01:00.0: enabling device (0000 -> 0002)
ahci 0002:01:00.0: AHCI vers 0001.0000, 32 command slots, 1.5 Gbps, SATA mode ahci 0002:01:00.0: 6/6 ports implemented (port mask 0x3f)
ahci 0002:01:00.0: flags: 64bit ncq only

Hi,

Thanks for reporting. Please add this to the command line to get the resource releasing between the steps to show:

dyndbg="file drivers/pci/setup-bus.c +p"

Also, the log snippet just shows it fails but it is impossible to know from it why the resource assigments do not fit so could you please provide a complete dmesg logs. Also providing the contents of /proc/iomem from the working case would save me quite a bit of decoding the iomem layout from the dmesgs.

Quoted 33 lines

Bisect log is attached for reference.
Guenter
---
# bad: [609706855d90bcab6080ba2cd030b9af322a1f0c] Merge tag 'trace-latency-v6.15-3' of git://git.kernel.org/pub/scm/linux/kernel/git/trace/linux-trace
# good: [acb4f33713b9f6cadb6143f211714c343465411c] Merge tag 'm68knommu-for-v6.15' of git://git.kernel.org/pub/scm/linux/kernel/git/gerg/m68knommu
git bisect start 'HEAD' 'acb4f33713b9'
# good: [cf05922d63e2ae6a9b1b52ff5236a44c3b29f78c] Merge tag 'drm-intel-gt-next-2025-03-12' of https://gitlab.freedesktop.org/drm/i915/kernel into drm-next git bisect good cf05922d63e2ae6a9b1b52ff5236a44c3b29f78c
# bad: [93d52288679e29aaa44a6f12d5a02e8a90e742c5] Merge tag 'backlight-next-6.15' of git://git.kernel.org/pub/scm/linux/kernel/git/lee/backlight
git bisect bad 93d52288679e29aaa44a6f12d5a02e8a90e742c5
# bad: [e5e0e6bebef3a21081fd1057c40468d4cff1a60d] Merge tag 'v6.15-p1' of git://git.kernel.org/pub/scm/linux/kernel/git/herbert/crypto-2.6
git bisect bad e5e0e6bebef3a21081fd1057c40468d4cff1a60d
# bad: [dea140198b846f7432d78566b7b0b83979c72c2b] Merge branch 'pci/misc'
git bisect bad dea140198b846f7432d78566b7b0b83979c72c2b
# bad: [a113afb84ae63ec4c893bc3204945ef6f3bb89f7] Merge branch 'pci/endpoint'
git bisect bad a113afb84ae63ec4c893bc3204945ef6f3bb89f7
# good: [a7a8e7996c1c114b50df5599229b1e7be38be3db] Merge branch 'pci/reset'
git bisect good a7a8e7996c1c114b50df5599229b1e7be38be3db
# bad: [7d4bcc0f2631e4ee10b5bcfff24a423d1c3c02a3] PCI: Move resource reassignment func declarations into pci/pci.h
git bisect bad 7d4bcc0f2631e4ee10b5bcfff24a423d1c3c02a3
# good: [acba174d2e754346c07578ad2220258706a203e2] PCI: Use while loop and break instead of gotos
git bisect good acba174d2e754346c07578ad2220258706a203e2
# good: [8884b5637b794ae541e8d6fb72102b1d8dba2b8d] PCI: Add debug print when releasing resources before retry
git bisect good 8884b5637b794ae541e8d6fb72102b1d8dba2b8d
# bad: [5af473941b56189423a7d16c05efabaf77299847] PCI: Increase Resizable BAR support from 512 GB to 128 TB
git bisect bad 5af473941b56189423a7d16c05efabaf77299847
# bad: [96336ec702643aec2addb3b1cdb81d687fe362f0] PCI: Perform reset_resource() and build fail list in sync
git bisect bad 96336ec702643aec2addb3b1cdb81d687fe362f0
# good: [e89df6d2beae847e931d84a190b192dfac41eba7] PCI: Use res->parent to check if resource is assigned
git bisect good e89df6d2beae847e931d84a190b192dfac41eba7
# first bad commit: [96336ec702643aec2addb3b1cdb81d687fe362f0] PCI: Perform reset_resource() and build fail list in sync

Signature · 1 line

i.

Ilpo Järvinen

author

2025-04-01 12:07 UTC #27 lore raw

Re: [PATCH 24/25] PCI: Perform reset_resource() and build fail list in sync

Quoted 108 lines · Ilpo Järvinen

Quoted 93 lines · Guenter Roeck
Quoted 18 lines · Ilpo Järvinen
Resetting resource is problematic as it prevent attempting to allocate the resource later, unless something in between restores the resource. Similarly, if fail_head does not contain all resources that were reset, those resource cannot be restored later.
The entire reset/restore cycle adds complexity and leaving resources into reseted state causes issues to other code such as for checks done in pci_enable_resources(). Take a small step towards not resetting resources by delaying reset until the end of resource assignment and build failure list (fail_head) in sync with the reset to avoid leaving behind resources that cannot be restored (for the case where the caller provides fail_head in the first place to allow restore somewhere in the callchain, as is not all callers pass non-NULL fail_head).
The Expansion ROM check is temporarily left in place while building the failure list until the upcoming change which reworks optional resource handling.
Quoted 5 lines
Ideally, whole resource reset could be removed but doing that in a big step would make the impact non-tractable due to complexity of all related code.
Signed-off-by Ilpo Järvinen <[email protected]>
With this patch in the mainline kernel, all mips:boston qemu emulations fail when running a 64-bit little endian configuration (64r6el_defconfig).
The problem is that the PCI based IDE/ATA controller is not initialized. There are a number of pci error messages.
pci_bus 0002:01: extended config space not accessible
pci 0002:01:00.0: [8086:2922] type 00 class 0x010601 conventional PCI endpoint pci 0002:01:00.0: BAR 4 [io 0x0000-0x001f]
pci 0002:01:00.0: BAR 5 [mem 0x00000000-0x00000fff]
pci 0002:00:00.0: PCI bridge to [bus 01-ff]
pci_bus 0002:01: busn_res: [bus 01-ff] end is updated to 01
pci 0002:00:00.0: bridge window [mem 0x16000000-0x160fffff]: assigned
pci 0002:00:00.0: bridge window [mem size 0x00100000 64bit pref]: can't assign; no space pci 0002:00:00.0: bridge window [mem size 0x00100000 64bit pref]: failed to assign pci 0002:00:00.0: bridge window [io size 0x1000]: can't assign; no space pci 0002:00:00.0: bridge window [io size 0x1000]: failed to assign
pci 0002:00:00.0: bridge window [mem size 0x00100000]: can't assign; bogus alignment pci 0002:00:00.0: bridge window [mem 0x16000000-0x160fffff 64bit pref]: assigned pci 0002:00:00.0: bridge window [io size 0x1000]: can't assign; no space pci 0002:00:00.0: bridge window [io size 0x1000]: failed to assign
pci 0002:01:00.0: BAR 5 [mem size 0x00001000]: can't assign; no space
pci 0002:01:00.0: BAR 5 [mem size 0x00001000]: failed to assign
pci 0002:01:00.0: BAR 4 [io size 0x0020]: can't assign; no space
pci 0002:01:00.0: BAR 4 [io size 0x0020]: failed to assign
pci 0002:01:00.0: BAR 5 [mem size 0x00001000]: can't assign; no space
pci 0002:01:00.0: BAR 5 [mem size 0x00001000]: failed to assign
pci 0002:01:00.0: BAR 4 [io size 0x0020]: can't assign; no space
pci 0002:01:00.0: BAR 4 [io size 0x0020]: failed to assign
pci 0002:00:00.0: PCI bridge to [bus 01]
pci 0002:00:00.0: bridge window [mem 0x16000000-0x160fffff 64bit pref]
pci_bus 0002:00: Some PCI device resources are unassigned, try booting with pci=realloc pci_bus 0002:00: resource 4 [mem 0x16000000-0x160fffff]
pci_bus 0002:01: resource 2 [mem 0x16000000-0x160fffff 64bit pref]
...
pci 0002:00:00.0: enabling device (0000 -> 0002)
ahci 0002:01:00.0: probe with driver ahci failed with error -12
Bisect points to this patch. Reverting it together with "PCI: Rework optional resource handling" fixes the problem. For comparison, after reverting the offending patches, the log messages are as follows.
pci_bus 0002:00: root bus resource [bus 00-ff]
pci_bus 0002:00: root bus resource [mem 0x16000000-0x160fffff]
pci 0002:00:00.0: [10ee:7021] type 01 class 0x060400 PCIe Root Complex Integrated Endpoint pci 0002:00:00.0: PCI bridge to [bus 00]
pci 0002:00:00.0:   bridge window [io  0x0000-0x0fff]
pci 0002:00:00.0:   bridge window [mem 0x00000000-0x000fffff]
pci 0002:00:00.0:   bridge window [mem 0x00000000-0x000fffff 64bit pref]
pci 0002:00:00.0: enabling Extended Tags
pci 0002:00:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring pci_bus 0002:01: extended config space not accessible
pci 0002:01:00.0: [8086:2922] type 00 class 0x010601 conventional PCI endpoint pci 0002:01:00.0: BAR 4 [io 0x0000-0x001f]
pci 0002:01:00.0: BAR 5 [mem 0x00000000-0x00000fff]
pci 0002:00:00.0: PCI bridge to [bus 01-ff]
pci_bus 0002:01: busn_res: [bus 01-ff] end is updated to 01
pci 0002:00:00.0: bridge window [mem 0x16000000-0x160fffff]: assigned
pci 0002:00:00.0: bridge window [mem size 0x00100000 64bit pref]: can't assign; no space pci 0002:00:00.0: bridge window [mem size 0x00100000 64bit pref]: failed to assign pci 0002:00:00.0: bridge window [io size 0x1000]: can't assign; no space pci 0002:00:00.0: bridge window [io size 0x1000]: failed to assign
pci 0002:01:00.0: BAR 5 [mem 0x16000000-0x16000fff]: assigned
pci 0002:01:00.0: BAR 4 [io size 0x0020]: can't assign; no space
pci 0002:01:00.0: BAR 4 [io size 0x0020]: failed to assign
pci 0002:00:00.0: PCI bridge to [bus 01]
pci 0002:00:00.0: bridge window [mem 0x16000000-0x160fffff]
pci_bus 0002:00: Some PCI device resources are unassigned, try booting with pci=realloc
Quoted 8 lines
pci_bus 0002:00: resource 4 [mem 0x16000000-0x160fffff]
pci_bus 0002:01: resource 1 [mem 0x16000000-0x160fffff]
...
pci 0002:00:00.0: enabling device (0000 -> 0002)
ahci 0002:01:00.0: enabling device (0000 -> 0002)
ahci 0002:01:00.0: AHCI vers 0001.0000, 32 command slots, 1.5 Gbps, SATA mode ahci 0002:01:00.0: 6/6 ports implemented (port mask 0x3f)
ahci 0002:01:00.0: flags: 64bit ncq only
Hi,
Thanks for reporting. Please add this to the command line to get the resource releasing between the steps to show:

Quoted 7 lines

dyndbg="file drivers/pci/setup-bus.c +p"
Also, the log snippet just shows it fails but it is impossible to know from it why the resource assigments do not fit so could you please provide a complete dmesg logs. Also providing the contents of /proc/iomem from the working case would save me quite a bit of decoding the iomem layout from the dmesgs.

Hi again,

If you could kindly include this patch into the test with pci_dbg() enabled so the resource reset/restore is better tracked.

From e7cb98904ab2ee235bbc13b3e981332e944dd476 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Ilpo=20J=C3=A4rvinen?= <[email protected]>
Date: Tue, 1 Apr 2025 15:05:13 +0300
Subject: [PATCH 1/1] PCI: Log reset and restore of resources
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

PCI resource fitting and assignment is complicated to track because it performs many actions without any logging. One of these is resource reset (zeroing the resource) and the restore during the multi-passed resource fitting algorithm.

Resource reset does not play well with the other PCI code if the code later wants to reattempt assignment of that resource, knowing that a resource was left into the reset state without a pairing restore is useful for understanding issues that show up as resource assignment failures.

Add pci_dbg() to both reset and restore to be better able to track what's going on within the resource fitting algorithm.

The resource fitting and assignment tracking should eventually be convert to use trace events to cover all changes made into the resources fully but that requires significant effort. In the meantime, logging resource reset and restore with pci_dbg() seems low-hanging fruit.

Signed-off-by Ilpo Järvinen <[email protected]>

---
 drivers/pci/setup-bus.c | 16 ++++++++++++++--
 1 file changed, 14 insertions(+), 2 deletions(-)

drivers/pci/setup-bus.c +14 -2

index 54d6f4fa3ce1..e67af19cb876 100644
@@ -130,10 +130,15 @@ static resource_size_t get_res_add_align(struct list_head *head,
 static void restore_dev_resource(struct pci_dev_resource *dev_res)
 {
 	struct resource *res = dev_res->res;
+	struct pci_dev *dev = dev_res->dev;
+	int idx = pci_resource_num(dev, res);
+	const char *res_name = pci_resource_name(dev, idx);
 
 	res->start = dev_res->start;
 	res->end = dev_res->end;
 	res->flags = dev_res->flags;
+
+	pci_dbg(dev_res->dev, "%s %pR: resource restored\n", res_name, res);
 }
 
 static bool pdev_resources_assignable(struct pci_dev *dev)

@@ -218,8 +223,15 @@ bool pci_resource_is_optional(const struct pci_dev *dev, int resno)
 	return false;
 }
 
-static inline void reset_resource(struct resource *res)
+static void reset_resource(struct pci_dev_resource *dev_res)
 {
+	struct resource *res = dev_res->res;
+	struct pci_dev *dev = dev_res->dev;
+	int idx = pci_resource_num(dev, res);
+	const char *res_name = pci_resource_name(dev, idx);
+
+	pci_dbg(dev_res->dev, "%s %pR: resetting resource\n", res_name, res);
+
 	res->start = 0;
 	res->end = 0;
 	res->flags = 0;

@@ -573,7 +585,7 @@ static void __assign_resources_sorted(struct list_head *head,
 				    0 /* don't care */);
 		}
 
-		reset_resource(res);
+		reset_resource(dev_res);
 	}
 
 	free_list(head);

base-commit: 7d06015d936c861160803e020f68f413b5c3cd9d

Signature · 1 line

2.39.5

Guenter Roeck

2025-04-01 14:15 UTC #28 lore raw

Re: [PATCH 24/25] PCI: Perform reset_resource() and build fail list in sync

Quoted 118 lines · Ilpo Järvinen

Quoted 108 lines · Ilpo Järvinen
Quoted 93 lines · Guenter Roeck
Quoted 18 lines · Ilpo Järvinen
Resetting resource is problematic as it prevent attempting to allocate the resource later, unless something in between restores the resource. Similarly, if fail_head does not contain all resources that were reset, those resource cannot be restored later.
The entire reset/restore cycle adds complexity and leaving resources into reseted state causes issues to other code such as for checks done in pci_enable_resources(). Take a small step towards not resetting resources by delaying reset until the end of resource assignment and build failure list (fail_head) in sync with the reset to avoid leaving behind resources that cannot be restored (for the case where the caller provides fail_head in the first place to allow restore somewhere in the callchain, as is not all callers pass non-NULL fail_head).
The Expansion ROM check is temporarily left in place while building the failure list until the upcoming change which reworks optional resource handling.
Quoted 5 lines
Ideally, whole resource reset could be removed but doing that in a big step would make the impact non-tractable due to complexity of all related code.
Signed-off-by Ilpo Järvinen <[email protected]>
With this patch in the mainline kernel, all mips:boston qemu emulations fail when running a 64-bit little endian configuration (64r6el_defconfig).
The problem is that the PCI based IDE/ATA controller is not initialized. There are a number of pci error messages.
pci_bus 0002:01: extended config space not accessible
pci 0002:01:00.0: [8086:2922] type 00 class 0x010601 conventional PCI endpoint pci 0002:01:00.0: BAR 4 [io 0x0000-0x001f]
pci 0002:01:00.0: BAR 5 [mem 0x00000000-0x00000fff]
pci 0002:00:00.0: PCI bridge to [bus 01-ff]
pci_bus 0002:01: busn_res: [bus 01-ff] end is updated to 01
pci 0002:00:00.0: bridge window [mem 0x16000000-0x160fffff]: assigned
pci 0002:00:00.0: bridge window [mem size 0x00100000 64bit pref]: can't assign; no space pci 0002:00:00.0: bridge window [mem size 0x00100000 64bit pref]: failed to assign pci 0002:00:00.0: bridge window [io size 0x1000]: can't assign; no space pci 0002:00:00.0: bridge window [io size 0x1000]: failed to assign
pci 0002:00:00.0: bridge window [mem size 0x00100000]: can't assign; bogus alignment pci 0002:00:00.0: bridge window [mem 0x16000000-0x160fffff 64bit pref]: assigned pci 0002:00:00.0: bridge window [io size 0x1000]: can't assign; no space pci 0002:00:00.0: bridge window [io size 0x1000]: failed to assign
pci 0002:01:00.0: BAR 5 [mem size 0x00001000]: can't assign; no space
pci 0002:01:00.0: BAR 5 [mem size 0x00001000]: failed to assign
pci 0002:01:00.0: BAR 4 [io size 0x0020]: can't assign; no space
pci 0002:01:00.0: BAR 4 [io size 0x0020]: failed to assign
pci 0002:01:00.0: BAR 5 [mem size 0x00001000]: can't assign; no space
pci 0002:01:00.0: BAR 5 [mem size 0x00001000]: failed to assign
pci 0002:01:00.0: BAR 4 [io size 0x0020]: can't assign; no space
pci 0002:01:00.0: BAR 4 [io size 0x0020]: failed to assign
pci 0002:00:00.0: PCI bridge to [bus 01]
pci 0002:00:00.0: bridge window [mem 0x16000000-0x160fffff 64bit pref]
pci_bus 0002:00: Some PCI device resources are unassigned, try booting with pci=realloc pci_bus 0002:00: resource 4 [mem 0x16000000-0x160fffff]
pci_bus 0002:01: resource 2 [mem 0x16000000-0x160fffff 64bit pref]
...
pci 0002:00:00.0: enabling device (0000 -> 0002)
ahci 0002:01:00.0: probe with driver ahci failed with error -12
Bisect points to this patch. Reverting it together with "PCI: Rework optional resource handling" fixes the problem. For comparison, after reverting the offending patches, the log messages are as follows.
pci_bus 0002:00: root bus resource [bus 00-ff]
pci_bus 0002:00: root bus resource [mem 0x16000000-0x160fffff]
pci 0002:00:00.0: [10ee:7021] type 01 class 0x060400 PCIe Root Complex Integrated Endpoint pci 0002:00:00.0: PCI bridge to [bus 00]
pci 0002:00:00.0:   bridge window [io  0x0000-0x0fff]
pci 0002:00:00.0:   bridge window [mem 0x00000000-0x000fffff]
pci 0002:00:00.0:   bridge window [mem 0x00000000-0x000fffff 64bit pref]
pci 0002:00:00.0: enabling Extended Tags
pci 0002:00:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring pci_bus 0002:01: extended config space not accessible
pci 0002:01:00.0: [8086:2922] type 00 class 0x010601 conventional PCI endpoint pci 0002:01:00.0: BAR 4 [io 0x0000-0x001f]
pci 0002:01:00.0: BAR 5 [mem 0x00000000-0x00000fff]
pci 0002:00:00.0: PCI bridge to [bus 01-ff]
pci_bus 0002:01: busn_res: [bus 01-ff] end is updated to 01
pci 0002:00:00.0: bridge window [mem 0x16000000-0x160fffff]: assigned
pci 0002:00:00.0: bridge window [mem size 0x00100000 64bit pref]: can't assign; no space pci 0002:00:00.0: bridge window [mem size 0x00100000 64bit pref]: failed to assign pci 0002:00:00.0: bridge window [io size 0x1000]: can't assign; no space pci 0002:00:00.0: bridge window [io size 0x1000]: failed to assign
pci 0002:01:00.0: BAR 5 [mem 0x16000000-0x16000fff]: assigned
pci 0002:01:00.0: BAR 4 [io size 0x0020]: can't assign; no space
pci 0002:01:00.0: BAR 4 [io size 0x0020]: failed to assign
pci 0002:00:00.0: PCI bridge to [bus 01]
pci 0002:00:00.0: bridge window [mem 0x16000000-0x160fffff]
pci_bus 0002:00: Some PCI device resources are unassigned, try booting with pci=realloc
Quoted 8 lines
pci_bus 0002:00: resource 4 [mem 0x16000000-0x160fffff]
pci_bus 0002:01: resource 1 [mem 0x16000000-0x160fffff]
...
pci 0002:00:00.0: enabling device (0000 -> 0002)
ahci 0002:01:00.0: enabling device (0000 -> 0002)
ahci 0002:01:00.0: AHCI vers 0001.0000, 32 command slots, 1.5 Gbps, SATA mode ahci 0002:01:00.0: 6/6 ports implemented (port mask 0x3f)
ahci 0002:01:00.0: flags: 64bit ncq only
Hi,
Thanks for reporting. Please add this to the command line to get the resource releasing between the steps to show:
Quoted 7 lines
dyndbg="file drivers/pci/setup-bus.c +p"
Also, the log snippet just shows it fails but it is impossible to know from it why the resource assigments do not fit so could you please provide a complete dmesg logs. Also providing the contents of /proc/iomem from the working case would save me quite a bit of decoding the iomem layout from the dmesgs.

Quoted 5 lines

Hi again,
If you could kindly include this patch into the test with pci_dbg() enabled so the resource reset/restore is better tracked.

Same link as before (http://server.roeck-us.net/qemu/mipsel64/).
The log with the patch applied is in bad-extra.log.

Hope this helps,
Guenter

Ilpo Järvinen

author

2025-04-01 17:38 UTC #29 lore raw

Re: [PATCH 24/25] PCI: Perform reset_resource() and build fail list in sync

Quoted 140 lines · Guenter Roeck

Quoted 139 lines · Ilpo Järvinen
Quoted 136 lines · Ilpo Järvinen
Quoted 114 lines · Guenter Roeck
Quoted 22 lines · Ilpo Järvinen
Resetting resource is problematic as it prevent attempting to allocate the resource later, unless something in between restores the resource. Similarly, if fail_head does not contain all resources that were reset,
those resource cannot be restored later.
The entire reset/restore cycle adds complexity and leaving resources into reseted state causes issues to other code such as for checks done in pci_enable_resources(). Take a small step towards not resetting resources by delaying reset until the end of resource assignment and build failure list (fail_head) in sync with the reset to avoid leaving behind resources that cannot be restored (for the case where the caller
provides fail_head in the first place to allow restore somewhere in the
callchain, as is not all callers pass non-NULL fail_head).
The Expansion ROM check is temporarily left in place while building the
failure list until the upcoming change which reworks optional resource handling.
Quoted 5 lines
Ideally, whole resource reset could be removed but doing that in a big step would make the impact non-tractable due to complexity of all related code.
Signed-off-by Ilpo Järvinen <[email protected]>
With this patch in the mainline kernel, all mips:boston qemu emulations fail when running a 64-bit little endian configuration (64r6el_defconfig).
The problem is that the PCI based IDE/ATA controller is not initialized. There are a number of pci error messages.
pci_bus 0002:01: extended config space not accessible
pci 0002:01:00.0: [8086:2922] type 00 class 0x010601 conventional PCI endpoint
pci 0002:01:00.0: BAR 4 [io 0x0000-0x001f]
pci 0002:01:00.0: BAR 5 [mem 0x00000000-0x00000fff]
pci 0002:00:00.0: PCI bridge to [bus 01-ff]
pci_bus 0002:01: busn_res: [bus 01-ff] end is updated to 01 pci 0002:00:00.0: bridge window [mem 0x16000000-0x160fffff]: assigned pci 0002:00:00.0: bridge window [mem size 0x00100000 64bit pref]: can't assign; no space
pci 0002:00:00.0: bridge window [mem size 0x00100000 64bit pref]: failed to assign
pci 0002:00:00.0: bridge window [io size 0x1000]: can't assign; no space
pci 0002:00:00.0: bridge window [io size 0x1000]: failed to assign pci 0002:00:00.0: bridge window [mem size 0x00100000]: can't assign; bogus alignment
pci 0002:00:00.0: bridge window [mem 0x16000000-0x160fffff 64bit pref]:
assigned
pci 0002:00:00.0: bridge window [io size 0x1000]: can't assign; no space
pci 0002:00:00.0: bridge window [io size 0x1000]: failed to assign pci 0002:01:00.0: BAR 5 [mem size 0x00001000]: can't assign; no space pci 0002:01:00.0: BAR 5 [mem size 0x00001000]: failed to assign pci 0002:01:00.0: BAR 4 [io size 0x0020]: can't assign; no space pci 0002:01:00.0: BAR 4 [io size 0x0020]: failed to assign pci 0002:01:00.0: BAR 5 [mem size 0x00001000]: can't assign; no space pci 0002:01:00.0: BAR 5 [mem size 0x00001000]: failed to assign pci 0002:01:00.0: BAR 4 [io size 0x0020]: can't assign; no space pci 0002:01:00.0: BAR 4 [io size 0x0020]: failed to assign pci 0002:00:00.0: PCI bridge to [bus 01]
pci 0002:00:00.0: bridge window [mem 0x16000000-0x160fffff 64bit pref]
pci_bus 0002:00: Some PCI device resources are unassigned, try booting with pci=realloc
pci_bus 0002:00: resource 4 [mem 0x16000000-0x160fffff] pci_bus 0002:01: resource 2 [mem 0x16000000-0x160fffff 64bit pref]
...
pci 0002:00:00.0: enabling device (0000 -> 0002)
ahci 0002:01:00.0: probe with driver ahci failed with error -12
Bisect points to this patch. Reverting it together with "PCI: Rework optional resource handling" fixes the problem. For comparison, after reverting the offending patches, the log messages are as follows.
pci_bus 0002:00: root bus resource [bus 00-ff]
pci_bus 0002:00: root bus resource [mem 0x16000000-0x160fffff] pci 0002:00:00.0: [10ee:7021] type 01 class 0x060400 PCIe Root Complex Integrated Endpoint
pci 0002:00:00.0: PCI bridge to [bus 00]
pci 0002:00:00.0:   bridge window [io  0x0000-0x0fff]
pci 0002:00:00.0:   bridge window [mem 0x00000000-0x000fffff]
pci 0002:00:00.0:   bridge window [mem 0x00000000-0x000fffff 64bit pref]
pci 0002:00:00.0: enabling Extended Tags
pci 0002:00:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring
pci_bus 0002:01: extended config space not accessible
pci 0002:01:00.0: [8086:2922] type 00 class 0x010601 conventional PCI endpoint
pci 0002:01:00.0: BAR 4 [io 0x0000-0x001f]
pci 0002:01:00.0: BAR 5 [mem 0x00000000-0x00000fff]
pci 0002:00:00.0: PCI bridge to [bus 01-ff]
pci_bus 0002:01: busn_res: [bus 01-ff] end is updated to 01 pci 0002:00:00.0: bridge window [mem 0x16000000-0x160fffff]: assigned pci 0002:00:00.0: bridge window [mem size 0x00100000 64bit pref]: can't assign; no space
pci 0002:00:00.0: bridge window [mem size 0x00100000 64bit pref]: failed to assign
pci 0002:00:00.0: bridge window [io size 0x1000]: can't assign; no space
pci 0002:00:00.0: bridge window [io size 0x1000]: failed to assign pci 0002:01:00.0: BAR 5 [mem 0x16000000-0x16000fff]: assigned pci 0002:01:00.0: BAR 4 [io size 0x0020]: can't assign; no space pci 0002:01:00.0: BAR 4 [io size 0x0020]: failed to assign pci 0002:00:00.0: PCI bridge to [bus 01]
pci 0002:00:00.0: bridge window [mem 0x16000000-0x160fffff]
pci_bus 0002:00: Some PCI device resources are unassigned, try booting with pci=realloc
pci_bus 0002:00: resource 4 [mem 0x16000000-0x160fffff]
Quoted 8 lines
pci_bus 0002:01: resource 1 [mem 0x16000000-0x160fffff]
...
pci 0002:00:00.0: enabling device (0000 -> 0002)
ahci 0002:01:00.0: enabling device (0000 -> 0002)
ahci 0002:01:00.0: AHCI vers 0001.0000, 32 command slots, 1.5 Gbps, SATA mode
ahci 0002:01:00.0: 6/6 ports implemented (port mask 0x3f) ahci 0002:01:00.0: flags: 64bit ncq only
Hi,
Thanks for reporting. Please add this to the command line to get the resource releasing between the steps to show:
dyndbg="file drivers/pci/setup-bus.c +p"
Also, the log snippet just shows it fails but it is impossible to know from it why the resource assigments do not fit so could you please provide a complete dmesg logs. Also providing the contents of /proc/iomem from the working case would save me quite a bit of decoding the iomem layout from the dmesgs.

Quoted 8 lines

Quoted 5 lines
Hi again,
If you could kindly include this patch into the test with pci_dbg() enabled so the resource reset/restore is better tracked.
Same link as before (http://server.roeck-us.net/qemu/mipsel64/).
The log with the patch applied is in bad-extra.log.

That log wasn't taken from a bad case but it doesn't matter anymore as I could test this with qemu myself, thanks for providing enough to make it easy to reproduce & test it locally :-).

The problem is caused by assign+release cycle being destructive on start aligned resources because successful assigment overwrites the start field. I'll send a patch to fix the problem once I've given it a bit more thought as this resource fitting code is somewhat complex.

Signature · 1 line

i.

Ondřej Jirman

2025-04-11 19:37 UTC #30 lore raw

Re: [PATCH 24/25] PCI: Perform reset_resource() and build fail list in sync

Hello Ilpo,

Quoted 8 lines · Ilpo Järvinen

That log wasn't taken from a bad case but it doesn't matter anymore as I could test this with qemu myself, thanks for providing enough to make it easy to reproduce & test it locally :-).
The problem is caused by assign+release cycle being destructive on start aligned resources because successful assigment overwrites the start field. I'll send a patch to fix the problem once I've given it a bit more thought as this resource fitting code is somewhat complex.

BTW, same thing here on a different SoC:

https://lore.kernel.org/lkml/hrcsm2bo4ysqj2ggejndlou32cdc7yiknnm5nrlcoz4d64wall@7te4dfqsoe3y/T/#u

There are kernel logs there, too, although I don't have dynamic debug enabled in the kernel.

Interestingly, bisect pointed me initially to a different commit. Reverting it helped, but just on one board (QuartzPro64).

And this is iomem:

0010f000-0010f0ff : 10f000.sram sram@10f000 00200000-e2bbffff : System RAM
02010000-0474ffff : Kernel code 04750000-0498ffff : reserved 04990000-0508ffff : Kernel data daa00000-e29fffff : reserved
e2bc0000-ecbbffff : reserved
e2bc0000-ecbbffff : reserved
ecbc0000-efffffff : System RAM
ecbc7000-ecbdffff : reserved
f0000000-f00fffff : a40000000.pcie config f0200000-f0ffffff : pcie@fe150000
f0200000-f020ffff : 0000:00:00.0 f0300000-f03fffff : PCI Bus 0000:01

    f0300000-f0303fff : 0000:01:00.0
      f0300000-f0303fff : nvme
    f0304000-f03040ff : 0000:01:00.0
      f0304000-f03040ff : nvme
f2000000-f20fffff : a40800000.pcie config
f2200000-f2ffffff : pcie@fe170000
  f2200000-f27fffff : PCI Bus 0002:21
    f2200000-f220ffff : 0002:21:00.0
    f2400000-f27fffff : 0002:21:00.0

f2800000-f280ffff : 0002:20:00.0
f3000000-f30fffff : a40c00000.pcie config
f3200000-f3ffffff : pcie@fe180000
f3200000-f320ffff : 0003:30:00.0
f3300000-f33fffff : PCI Bus 0003:31

    f3300000-f3303fff : 0003:31:00.0
    f3304000-f3304fff : 0003:31:00.0
      f3304000-f3304fff : r8169

fb000000-fb1fffff : fb000000.gpu gpu@fb000000 fc00c100-fc3fffff : fc000000.usb usb@fc000000 fc400000-fc407fff : usb@fc400000
fc400000-fc407fff : xhci-hcd.10.auto usb@fc400000
fc40c100-fc7fffff : fc400000.usb usb@fc400000 fc800000-fc83ffff : fc800000.usb usb@fc800000 fc840000-fc87ffff : fc840000.usb usb@fc840000 fc880000-fc8bffff : fc880000.usb usb@fc880000 fc8c0000-fc8fffff : fc8c0000.usb usb@fc8c0000 fc900000-fc900dff : fc900000.iommu
fc910000-fc910dff : fc900000.iommu
fd600000-fd6fffff : fd600000.sram sram@fd600000 fd8a0000-fd8a00ff : fd8a0000.gpio gpio@fd8a0000 fdb50000-fdb507ff : fdb50000.video-codec video-codec@fdb50000 fdb50800-fdb5083f : fdb50800.iommu iommu@fdb50800 fdb80000-fdb8017f : fdb80000.rga rga@fdb80000 fdba0000-fdba07ff : fdba0000.video-codec video-codec@fdba0000 fdba0800-fdba083f : fdba0800.iommu iommu@fdba0800 fdba4800-fdba483f : fdba4800.iommu iommu@fdba4800 fdba8800-fdba883f : fdba8800.iommu iommu@fdba8800 fdbac800-fdbac83f : fdbac800.iommu iommu@fdbac800 fdc70000-fdc707ff : fdc70000.video-codec video-codec@fdc70000 fdd90000-fdd941ff : fdd90000.vop vop
fdd95000-fdd95fff : fdd90000.vop gamma-lut fdd97e00-fdd97eff : fdd97e00.iommu iommu@fdd97e00 fdd97f00-fdd97fff : fdd97e00.iommu iommu@fdd97e00 fddf0000-fddf0fff : fddf0000.i2s i2s@fddf0000 fddf4000-fddf4fff : fddf4000.i2s i2s@fddf4000 fde80000-fde9ffff : fde80000.hdmi hdmi@fde80000 fdea0000-fdebffff : fdea0000.hdmi hdmi@fdea0000 fdee0000-fdee5fff : fdee0000.hdmi_receiver hdmi_receiver@fdee0000 fe060000-fe06ffff : fe060000.dfi dfi@fe060000 fe150000-fe15ffff : a40000000.pcie apb
fe170000-fe17ffff : a40800000.pcie apb
fe180000-fe18ffff : a40c00000.pcie apb
fe1b0000-fe1bffff : fe1b0000.ethernet ethernet@fe1b0000 fe210000-fe210fff : fe210000.sata sata@fe210000 fe2c0000-fe2c3fff : fe2c0000.mmc mmc@fe2c0000 fe2e0000-fe2effff : fe2e0000.mmc mmc@fe2e0000 fe470000-fe470fff : fe470000.i2s i2s@fe470000 fe600000-fe60ffff : GICD
fe680000-fe77ffff : GICR
fea10000-fea13fff : dma-controller@fea10000
fea10000-fea13fff : fea10000.dma-controller dma-controller@fea10000
fea30000-fea33fff : dma-controller@fea30000
fea30000-fea33fff : fea30000.dma-controller dma-controller@fea30000
feaa0000-feaa0fff : feaa0000.i2c i2c@feaa0000 feaf0000-feaf00ff : feaf0000.watchdog watchdog@feaf0000 feb20000-feb20fff : feb20000.spi spi@feb20000 feb50000-feb500ff : serial
fec00000-fec003ff : fec00000.tsadc tsadc@fec00000 fec10000-fec1ffff : fec10000.adc adc@fec10000 fec20000-fec200ff : fec20000.gpio gpio@fec20000 fec30000-fec300ff : fec30000.gpio gpio@fec30000 fec40000-fec400ff : fec40000.gpio gpio@fec40000 fec50000-fec500ff : fec50000.gpio gpio@fec50000 fec90000-fec90fff : fec90000.i2c i2c@fec90000 fed10000-fed13fff : dma-controller@fed10000
fed10000-fed13fff : fed10000.dma-controller dma-controller@fed10000
fed60000-fed61fff : fed60000.phy phy@fed60000 fed70000-fed71fff : fed70000.phy phy@fed70000 fed80000-fed8ffff : fed80000.phy phy@fed80000 fed90000-fed9ffff : fed90000.phy phy@fed90000 fee00000-fee000ff : fee00000.phy phy@fee00000 fee10000-fee100ff : fee10000.phy phy@fee10000 fee20000-fee200ff : fee20000.phy phy@fee20000 fee80000-fee9ffff : fee80000.phy phy@fee80000 ff001000-ff0effff : ff001000.sram sram@ff001000 100000000-3fbffffff : System RAM
3ec000000-3fbffffff : reserved
3fc500000-3ffefffff : System RAM
4f0000000-4ffffffff : System RAM
4fc611000-4fc6d0fff : reserved
4fc6d1000-4fded1fff : reserved
4fded2000-4fdf91fff : reserved
4fdf93000-4fdf96fff : reserved
4fdf97000-4fdfabfff : reserved
4fdfac000-4fe051fff : reserved
4fe052000-4ffffffff : reserved
900000000-93fffffff : pcie@fe150000
900000000-93fffffff : 0000:00:00.0
980000000-9bfffffff : pcie@fe170000
9c0000000-9ffffffff : pcie@fe180000
a40000000-a403fffff : a40000000.pcie dbi a40800000-a40bfffff : a40800000.pcie dbi a40c00000-a40ffffff : a40c00000.pcie dbi

Thank you,

o.

Quoted 2 lines

Signature · 1 line
 i.

Ilpo Järvinen

author

2025-04-14 09:52 UTC #31 lore raw

Re: [PATCH 24/25] PCI: Perform reset_resource() and build fail list in sync

Quoted 14 lines · Ondřej Jirman

Hello Ilpo,
Quoted 8 lines · Ilpo Järvinen
That log wasn't taken from a bad case but it doesn't matter anymore as I could test this with qemu myself, thanks for providing enough to make it easy to reproduce & test it locally :-).
The problem is caused by assign+release cycle being destructive on start aligned resources because successful assigment overwrites the start field. I'll send a patch to fix the problem once I've given it a bit more thought as this resource fitting code is somewhat complex.
BTW, same thing here on a different SoC:

Quoted 7 lines

https://lore.kernel.org/lkml/hrcsm2bo4ysqj2ggejndlou32cdc7yiknnm5nrlcoz4d64wall@7te4dfqsoe3y/T/#u
There are kernel logs there, too, although I don't have dynamic debug enabled in the kernel.
Interestingly, bisect pointed me initially to a different commit. Reverting it helped, but just on one board (QuartzPro64).

Hi,

Since you didn't mention it, I guess you haven't tried the fix:

https://patchwork.kernel.org/project/linux-pci/patch/[email protected]/

Signature · 1 line

i.

Quoted 126 lines

And this is iomem:
0010f000-0010f0ff : 10f000.sram sram@10f000 00200000-e2bbffff : System RAM
02010000-0474ffff : Kernel code 04750000-0498ffff : reserved 04990000-0508ffff : Kernel data daa00000-e29fffff : reserved
e2bc0000-ecbbffff : reserved
e2bc0000-ecbbffff : reserved
ecbc0000-efffffff : System RAM
ecbc7000-ecbdffff : reserved
f0000000-f00fffff : a40000000.pcie config f0200000-f0ffffff : pcie@fe150000
f0200000-f020ffff : 0000:00:00.0 f0300000-f03fffff : PCI Bus 0000:01
    f0300000-f0303fff : 0000:01:00.0
      f0300000-f0303fff : nvme
    f0304000-f03040ff : 0000:01:00.0
      f0304000-f03040ff : nvme
f2000000-f20fffff : a40800000.pcie config
f2200000-f2ffffff : pcie@fe170000
  f2200000-f27fffff : PCI Bus 0002:21
    f2200000-f220ffff : 0002:21:00.0
    f2400000-f27fffff : 0002:21:00.0
f2800000-f280ffff : 0002:20:00.0
f3000000-f30fffff : a40c00000.pcie config
f3200000-f3ffffff : pcie@fe180000
f3200000-f320ffff : 0003:30:00.0
f3300000-f33fffff : PCI Bus 0003:31
    f3300000-f3303fff : 0003:31:00.0
    f3304000-f3304fff : 0003:31:00.0
      f3304000-f3304fff : r8169
fb000000-fb1fffff : fb000000.gpu gpu@fb000000 fc00c100-fc3fffff : fc000000.usb usb@fc000000 fc400000-fc407fff : usb@fc400000
fc400000-fc407fff : xhci-hcd.10.auto usb@fc400000
fc40c100-fc7fffff : fc400000.usb usb@fc400000 fc800000-fc83ffff : fc800000.usb usb@fc800000 fc840000-fc87ffff : fc840000.usb usb@fc840000 fc880000-fc8bffff : fc880000.usb usb@fc880000 fc8c0000-fc8fffff : fc8c0000.usb usb@fc8c0000 fc900000-fc900dff : fc900000.iommu
fc910000-fc910dff : fc900000.iommu
fd600000-fd6fffff : fd600000.sram sram@fd600000 fd8a0000-fd8a00ff : fd8a0000.gpio gpio@fd8a0000 fdb50000-fdb507ff : fdb50000.video-codec video-codec@fdb50000 fdb50800-fdb5083f : fdb50800.iommu iommu@fdb50800 fdb80000-fdb8017f : fdb80000.rga rga@fdb80000 fdba0000-fdba07ff : fdba0000.video-codec video-codec@fdba0000 fdba0800-fdba083f : fdba0800.iommu iommu@fdba0800 fdba4800-fdba483f : fdba4800.iommu iommu@fdba4800 fdba8800-fdba883f : fdba8800.iommu iommu@fdba8800 fdbac800-fdbac83f : fdbac800.iommu iommu@fdbac800 fdc70000-fdc707ff : fdc70000.video-codec video-codec@fdc70000 fdd90000-fdd941ff : fdd90000.vop vop
fdd95000-fdd95fff : fdd90000.vop gamma-lut fdd97e00-fdd97eff : fdd97e00.iommu iommu@fdd97e00 fdd97f00-fdd97fff : fdd97e00.iommu iommu@fdd97e00 fddf0000-fddf0fff : fddf0000.i2s i2s@fddf0000 fddf4000-fddf4fff : fddf4000.i2s i2s@fddf4000 fde80000-fde9ffff : fde80000.hdmi hdmi@fde80000 fdea0000-fdebffff : fdea0000.hdmi hdmi@fdea0000 fdee0000-fdee5fff : fdee0000.hdmi_receiver hdmi_receiver@fdee0000 fe060000-fe06ffff : fe060000.dfi dfi@fe060000 fe150000-fe15ffff : a40000000.pcie apb
fe170000-fe17ffff : a40800000.pcie apb
fe180000-fe18ffff : a40c00000.pcie apb
fe1b0000-fe1bffff : fe1b0000.ethernet ethernet@fe1b0000 fe210000-fe210fff : fe210000.sata sata@fe210000 fe2c0000-fe2c3fff : fe2c0000.mmc mmc@fe2c0000 fe2e0000-fe2effff : fe2e0000.mmc mmc@fe2e0000 fe470000-fe470fff : fe470000.i2s i2s@fe470000 fe600000-fe60ffff : GICD
fe680000-fe77ffff : GICR
fea10000-fea13fff : dma-controller@fea10000
fea10000-fea13fff : fea10000.dma-controller dma-controller@fea10000
fea30000-fea33fff : dma-controller@fea30000
fea30000-fea33fff : fea30000.dma-controller dma-controller@fea30000
feaa0000-feaa0fff : feaa0000.i2c i2c@feaa0000 feaf0000-feaf00ff : feaf0000.watchdog watchdog@feaf0000 feb20000-feb20fff : feb20000.spi spi@feb20000 feb50000-feb500ff : serial
fec00000-fec003ff : fec00000.tsadc tsadc@fec00000 fec10000-fec1ffff : fec10000.adc adc@fec10000 fec20000-fec200ff : fec20000.gpio gpio@fec20000 fec30000-fec300ff : fec30000.gpio gpio@fec30000 fec40000-fec400ff : fec40000.gpio gpio@fec40000 fec50000-fec500ff : fec50000.gpio gpio@fec50000 fec90000-fec90fff : fec90000.i2c i2c@fec90000 fed10000-fed13fff : dma-controller@fed10000
fed10000-fed13fff : fed10000.dma-controller dma-controller@fed10000
fed60000-fed61fff : fed60000.phy phy@fed60000 fed70000-fed71fff : fed70000.phy phy@fed70000 fed80000-fed8ffff : fed80000.phy phy@fed80000 fed90000-fed9ffff : fed90000.phy phy@fed90000 fee00000-fee000ff : fee00000.phy phy@fee00000 fee10000-fee100ff : fee10000.phy phy@fee10000 fee20000-fee200ff : fee20000.phy phy@fee20000 fee80000-fee9ffff : fee80000.phy phy@fee80000 ff001000-ff0effff : ff001000.sram sram@ff001000 100000000-3fbffffff : System RAM
3ec000000-3fbffffff : reserved
3fc500000-3ffefffff : System RAM
4f0000000-4ffffffff : System RAM
4fc611000-4fc6d0fff : reserved
4fc6d1000-4fded1fff : reserved
4fded2000-4fdf91fff : reserved
4fdf93000-4fdf96fff : reserved
4fdf97000-4fdfabfff : reserved
4fdfac000-4fe051fff : reserved
4fe052000-4ffffffff : reserved
900000000-93fffffff : pcie@fe150000
900000000-93fffffff : 0000:00:00.0
980000000-9bfffffff : pcie@fe170000
9c0000000-9ffffffff : pcie@fe180000
a40000000-a403fffff : a40000000.pcie dbi a40800000-a40bfffff : a40800000.pcie dbi a40c00000-a40ffffff : a40c00000.pcie dbi
Thank you,
	o.
Quoted 2 lines
Signature · 1 line
 i.

Ondřej Jirman

2025-04-14 12:19 UTC #32 lore raw

Re: [PATCH 24/25] PCI: Perform reset_resource() and build fail list in sync

Hello Ilpo,

Quoted 21 lines · Ilpo Järvinen

Quoted 19 lines · Ondřej Jirman
Hello Ilpo,
Quoted 8 lines · Ilpo Järvinen
That log wasn't taken from a bad case but it doesn't matter anymore as I could test this with qemu myself, thanks for providing enough to make it easy to reproduce & test it locally :-).
The problem is caused by assign+release cycle being destructive on start aligned resources because successful assigment overwrites the start field. I'll send a patch to fix the problem once I've given it a bit more thought as this resource fitting code is somewhat complex.
BTW, same thing here on a different SoC:
https://lore.kernel.org/lkml/hrcsm2bo4ysqj2ggejndlou32cdc7yiknnm5nrlcoz4d64wall@7te4dfqsoe3y/T/#u
There are kernel logs there, too, although I don't have dynamic debug enabled in the kernel.

Quoted 8 lines

Quoted 2 lines
Interestingly, bisect pointed me initially to a different commit. Reverting it helped, but just on one board (QuartzPro64).
Hi,
Since you didn't mention it, I guess you haven't tried the fix:
https://patchwork.kernel.org/project/linux-pci/patch/[email protected]/

This patch works. Thank you. One difference compared to 6.14 that I'm noticing in the kernel log is that "save config" sequences now are printed twice for unpopulated port. Not sure if it's related to your patches. Previously it was printed just once.

Kind regards,

o.

rockchip-dw-pcie a40800000.pcie: PCI host bridge to bus 0002:20
pci_bus 0002:20: root bus resource [bus 20-2f]
pci_bus 0002:20: root bus resource [io 0x300000-0x3fffff] (bus address [0xf2100000-0xf21fffff]) pci_bus 0002:20: root bus resource [mem 0xf2200000-0xf2ffffff]
pci_bus 0002:20: root bus resource [mem 0x980000000-0x9bfffffff] (bus address [0x40000000-0x7fffffff]) pci_bus 0002:20: scanning bus
pci 0002:20:00.0: [1d87:3588] type 01 class 0x060400 PCIe Root Port
pci 0002:20:00.0: ROM [mem 0x00000000-0x0000ffff pref]
pci 0002:20:00.0: PCI bridge to [bus 01-ff]

pci 0002:20:00.0:   bridge window [io  0x0000-0x0fff]
pci 0002:20:00.0:   bridge window [mem 0x00000000-0x000fffff]
pci 0002:20:00.0:   bridge window [mem 0x00000000-0x000fffff 64bit pref]

pci 0002:20:00.0: supports D1 D2
pci 0002:20:00.0: PME# supported from D0 D1 D3hot
pci 0002:20:00.0: PME# disabled
pci 0002:20:00.0: Adding to iommu group 9
pci 0002:20:00.0: vgaarb: pci_notify
pci_bus 0002:20: fixups for bus
pci 0002:20:00.0: scanning [bus 01-ff] behind bridge, pass 0 pci 0002:20:00.0: Primary bus is hard wired to 0
pci 0002:20:00.0: bridge configuration invalid ([bus 01-ff]), reconfiguring pci 0002:20:00.0: scanning [bus 00-00] behind bridge, pass 1 pci_bus 0002:20: bus scan returning with max=21
pci 0002:20:00.0: ROM [mem 0xf2200000-0xf220ffff pref]: assigned pci 0002:20:00.0: PCI bridge to [bus 21]
pci_bus 0002:20: resource 4 [io 0x300000-0x3fffff]
pci_bus 0002:20: resource 5 [mem 0xf2200000-0xf2ffffff]
pci_bus 0002:20: resource 6 [mem 0x980000000-0x9bfffffff] pcieport 0002:20:00.0: vgaarb: pci_notify
pcieport 0002:20:00.0: assign IRQ: got 148
pcieport 0002:20:00.0: PME: Signaling with IRQ 157
pcieport 0002:20:00.0: AER: enabled with IRQ 158
pcieport 0002:20:00.0: bwctrl: enabled with IRQ 157
pcieport 0002:20:00.0: save config 0x00: 0x35881d87
pcieport 0002:20:00.0: save config 0x04: 0x00100507
pcieport 0002:20:00.0: save config 0x08: 0x06040001
pcieport 0002:20:00.0: save config 0x0c: 0x00010000
pcieport 0002:20:00.0: save config 0x10: 0x00000000
pcieport 0002:20:00.0: save config 0x14: 0x00000000
pcieport 0002:20:00.0: save config 0x18: 0x00212120
pcieport 0002:20:00.0: save config 0x1c: 0x000000f0
pcieport 0002:20:00.0: save config 0x20: 0x0000fff0
pcieport 0002:20:00.0: save config 0x24: 0x0001fff1
pcieport 0002:20:00.0: save config 0x28: 0x00000000
pcieport 0002:20:00.0: save config 0x2c: 0x00000000
pcieport 0002:20:00.0: save config 0x30: 0x00000000
pcieport 0002:20:00.0: save config 0x34: 0x00000040
pcieport 0002:20:00.0: save config 0x38: 0x00000000
pcieport 0002:20:00.0: save config 0x3c: 0x00020194
pcieport 0002:20:00.0: vgaarb: pci_notify
pcieport 0002:20:00.0: save config 0x00: 0x35881d87
pcieport 0002:20:00.0: save config 0x04: 0x00100507
pcieport 0002:20:00.0: save config 0x08: 0x06040001
pcieport 0002:20:00.0: save config 0x0c: 0x00010000
pcieport 0002:20:00.0: save config 0x10: 0x00000000
pcieport 0002:20:00.0: save config 0x14: 0x00000000
pcieport 0002:20:00.0: save config 0x18: 0x00212120
pcieport 0002:20:00.0: save config 0x1c: 0x000000f0
pcieport 0002:20:00.0: save config 0x20: 0x0000fff0
pcieport 0002:20:00.0: save config 0x24: 0x0001fff1
pcieport 0002:20:00.0: save config 0x28: 0x00000000
pcieport 0002:20:00.0: save config 0x2c: 0x00000000
pcieport 0002:20:00.0: save config 0x30: 0x00000000
pcieport 0002:20:00.0: save config 0x34: 0x00000040
pcieport 0002:20:00.0: save config 0x38: 0x00000000
pcieport 0002:20:00.0: save config 0x3c: 0x00020194
pcieport 0002:20:00.0: PME# enabled

Quoted 129 lines

Signature · 1 line
 i.
Quoted 126 lines
And this is iomem:
0010f000-0010f0ff : 10f000.sram sram@10f000 00200000-e2bbffff : System RAM
02010000-0474ffff : Kernel code 04750000-0498ffff : reserved 04990000-0508ffff : Kernel data daa00000-e29fffff : reserved
e2bc0000-ecbbffff : reserved
e2bc0000-ecbbffff : reserved
ecbc0000-efffffff : System RAM
ecbc7000-ecbdffff : reserved
f0000000-f00fffff : a40000000.pcie config f0200000-f0ffffff : pcie@fe150000
f0200000-f020ffff : 0000:00:00.0 f0300000-f03fffff : PCI Bus 0000:01
    f0300000-f0303fff : 0000:01:00.0
      f0300000-f0303fff : nvme
    f0304000-f03040ff : 0000:01:00.0
      f0304000-f03040ff : nvme
f2000000-f20fffff : a40800000.pcie config
f2200000-f2ffffff : pcie@fe170000
  f2200000-f27fffff : PCI Bus 0002:21
    f2200000-f220ffff : 0002:21:00.0
    f2400000-f27fffff : 0002:21:00.0
f2800000-f280ffff : 0002:20:00.0
f3000000-f30fffff : a40c00000.pcie config
f3200000-f3ffffff : pcie@fe180000
f3200000-f320ffff : 0003:30:00.0
f3300000-f33fffff : PCI Bus 0003:31
    f3300000-f3303fff : 0003:31:00.0
    f3304000-f3304fff : 0003:31:00.0
      f3304000-f3304fff : r8169
fb000000-fb1fffff : fb000000.gpu gpu@fb000000 fc00c100-fc3fffff : fc000000.usb usb@fc000000 fc400000-fc407fff : usb@fc400000
fc400000-fc407fff : xhci-hcd.10.auto usb@fc400000
fc40c100-fc7fffff : fc400000.usb usb@fc400000 fc800000-fc83ffff : fc800000.usb usb@fc800000 fc840000-fc87ffff : fc840000.usb usb@fc840000 fc880000-fc8bffff : fc880000.usb usb@fc880000 fc8c0000-fc8fffff : fc8c0000.usb usb@fc8c0000 fc900000-fc900dff : fc900000.iommu
fc910000-fc910dff : fc900000.iommu
fd600000-fd6fffff : fd600000.sram sram@fd600000 fd8a0000-fd8a00ff : fd8a0000.gpio gpio@fd8a0000 fdb50000-fdb507ff : fdb50000.video-codec video-codec@fdb50000 fdb50800-fdb5083f : fdb50800.iommu iommu@fdb50800 fdb80000-fdb8017f : fdb80000.rga rga@fdb80000 fdba0000-fdba07ff : fdba0000.video-codec video-codec@fdba0000 fdba0800-fdba083f : fdba0800.iommu iommu@fdba0800 fdba4800-fdba483f : fdba4800.iommu iommu@fdba4800 fdba8800-fdba883f : fdba8800.iommu iommu@fdba8800 fdbac800-fdbac83f : fdbac800.iommu iommu@fdbac800 fdc70000-fdc707ff : fdc70000.video-codec video-codec@fdc70000 fdd90000-fdd941ff : fdd90000.vop vop
fdd95000-fdd95fff : fdd90000.vop gamma-lut fdd97e00-fdd97eff : fdd97e00.iommu iommu@fdd97e00 fdd97f00-fdd97fff : fdd97e00.iommu iommu@fdd97e00 fddf0000-fddf0fff : fddf0000.i2s i2s@fddf0000 fddf4000-fddf4fff : fddf4000.i2s i2s@fddf4000 fde80000-fde9ffff : fde80000.hdmi hdmi@fde80000 fdea0000-fdebffff : fdea0000.hdmi hdmi@fdea0000 fdee0000-fdee5fff : fdee0000.hdmi_receiver hdmi_receiver@fdee0000 fe060000-fe06ffff : fe060000.dfi dfi@fe060000 fe150000-fe15ffff : a40000000.pcie apb
fe170000-fe17ffff : a40800000.pcie apb
fe180000-fe18ffff : a40c00000.pcie apb
fe1b0000-fe1bffff : fe1b0000.ethernet ethernet@fe1b0000 fe210000-fe210fff : fe210000.sata sata@fe210000 fe2c0000-fe2c3fff : fe2c0000.mmc mmc@fe2c0000 fe2e0000-fe2effff : fe2e0000.mmc mmc@fe2e0000 fe470000-fe470fff : fe470000.i2s i2s@fe470000 fe600000-fe60ffff : GICD
fe680000-fe77ffff : GICR
fea10000-fea13fff : dma-controller@fea10000
fea10000-fea13fff : fea10000.dma-controller dma-controller@fea10000
fea30000-fea33fff : dma-controller@fea30000
fea30000-fea33fff : fea30000.dma-controller dma-controller@fea30000
feaa0000-feaa0fff : feaa0000.i2c i2c@feaa0000 feaf0000-feaf00ff : feaf0000.watchdog watchdog@feaf0000 feb20000-feb20fff : feb20000.spi spi@feb20000 feb50000-feb500ff : serial
fec00000-fec003ff : fec00000.tsadc tsadc@fec00000 fec10000-fec1ffff : fec10000.adc adc@fec10000 fec20000-fec200ff : fec20000.gpio gpio@fec20000 fec30000-fec300ff : fec30000.gpio gpio@fec30000 fec40000-fec400ff : fec40000.gpio gpio@fec40000 fec50000-fec500ff : fec50000.gpio gpio@fec50000 fec90000-fec90fff : fec90000.i2c i2c@fec90000 fed10000-fed13fff : dma-controller@fed10000
fed10000-fed13fff : fed10000.dma-controller dma-controller@fed10000
fed60000-fed61fff : fed60000.phy phy@fed60000 fed70000-fed71fff : fed70000.phy phy@fed70000 fed80000-fed8ffff : fed80000.phy phy@fed80000 fed90000-fed9ffff : fed90000.phy phy@fed90000 fee00000-fee000ff : fee00000.phy phy@fee00000 fee10000-fee100ff : fee10000.phy phy@fee10000 fee20000-fee200ff : fee20000.phy phy@fee20000 fee80000-fee9ffff : fee80000.phy phy@fee80000 ff001000-ff0effff : ff001000.sram sram@ff001000 100000000-3fbffffff : System RAM
3ec000000-3fbffffff : reserved
3fc500000-3ffefffff : System RAM
4f0000000-4ffffffff : System RAM
4fc611000-4fc6d0fff : reserved
4fc6d1000-4fded1fff : reserved
4fded2000-4fdf91fff : reserved
4fdf93000-4fdf96fff : reserved
4fdf97000-4fdfabfff : reserved
4fdfac000-4fe051fff : reserved
4fe052000-4ffffffff : reserved
900000000-93fffffff : pcie@fe150000
900000000-93fffffff : 0000:00:00.0
980000000-9bfffffff : pcie@fe170000
9c0000000-9ffffffff : pcie@fe180000
a40000000-a403fffff : a40000000.pcie dbi a40800000-a40bfffff : a40800000.pcie dbi a40c00000-a40ffffff : a40c00000.pcie dbi
Thank you,
	o.
Quoted 2 lines
Signature · 1 line
 i.

Ilpo Järvinen

author

2025-04-14 13:15 UTC #33 lore raw

Re: [PATCH 24/25] PCI: Perform reset_resource() and build fail list in sync

Quoted 25 lines · Ondřej Jirman

Quoted 24 lines · Ilpo Järvinen
Quoted 21 lines · Ondřej Jirman
Hello Ilpo,
Quoted 8 lines · Ilpo Järvinen
That log wasn't taken from a bad case but it doesn't matter anymore as I could test this with qemu myself, thanks for providing enough to make it easy to reproduce & test it locally :-).
The problem is caused by assign+release cycle being destructive on start aligned resources because successful assigment overwrites the start field. I'll send a patch to fix the problem once I've given it a bit more thought as this resource fitting code is somewhat complex.
BTW, same thing here on a different SoC:
https://lore.kernel.org/lkml/hrcsm2bo4ysqj2ggejndlou32cdc7yiknnm5nrlcoz4d64wall@7te4dfqsoe3y/T/#u
There are kernel logs there, too, although I don't have dynamic debug enabled in the kernel.
Interestingly, bisect pointed me initially to a different commit. Reverting it helped, but just on one board (QuartzPro64).

Quoted 7 lines

Quoted 5 lines
Hi,
Since you didn't mention it, I guess you haven't tried the fix:
https://patchwork.kernel.org/project/linux-pci/patch/[email protected]/
This patch works. Thank you.

Thanks for testing. If you feel confident enough, please send your Tested-by tag to its thread (not this one).

Quoted 4 lines

One difference compared to 6.14 that I'm noticing
in the kernel log is that "save config" sequences now are printed twice for unpopulated port. Not sure if it's related to your patches. Previously it was printed just once.

I don't think it is related. The resource fitting/assignment changes should not impact PCI save/restore AFAIK (except by preventing device from working in the first place if necessary resources do not get assigned).

PCI state saving has always seemed like that in most logs I've seen with dyndbg enabled, that is, the state is seemingly saved multiple times, often right after the previous save too. So TBH, I'm not convinced it's even something new. If you have backtraces to show those places that invoke pci_save_state() I can take a look but I don't expect much to come out of that.

Signature · 1 line

i.

Quoted 202 lines

Kind regards,
	o.
rockchip-dw-pcie a40800000.pcie: PCI host bridge to bus 0002:20
pci_bus 0002:20: root bus resource [bus 20-2f]
pci_bus 0002:20: root bus resource [io 0x300000-0x3fffff] (bus address [0xf2100000-0xf21fffff]) pci_bus 0002:20: root bus resource [mem 0xf2200000-0xf2ffffff]
pci_bus 0002:20: root bus resource [mem 0x980000000-0x9bfffffff] (bus address [0x40000000-0x7fffffff]) pci_bus 0002:20: scanning bus
pci 0002:20:00.0: [1d87:3588] type 01 class 0x060400 PCIe Root Port
pci 0002:20:00.0: ROM [mem 0x00000000-0x0000ffff pref]
pci 0002:20:00.0: PCI bridge to [bus 01-ff]
pci 0002:20:00.0:   bridge window [io  0x0000-0x0fff]
pci 0002:20:00.0:   bridge window [mem 0x00000000-0x000fffff]
pci 0002:20:00.0:   bridge window [mem 0x00000000-0x000fffff 64bit pref]
pci 0002:20:00.0: supports D1 D2
pci 0002:20:00.0: PME# supported from D0 D1 D3hot
pci 0002:20:00.0: PME# disabled
pci 0002:20:00.0: Adding to iommu group 9
pci 0002:20:00.0: vgaarb: pci_notify
pci_bus 0002:20: fixups for bus
pci 0002:20:00.0: scanning [bus 01-ff] behind bridge, pass 0 pci 0002:20:00.0: Primary bus is hard wired to 0
pci 0002:20:00.0: bridge configuration invalid ([bus 01-ff]), reconfiguring pci 0002:20:00.0: scanning [bus 00-00] behind bridge, pass 1 pci_bus 0002:20: bus scan returning with max=21
pci 0002:20:00.0: ROM [mem 0xf2200000-0xf220ffff pref]: assigned pci 0002:20:00.0: PCI bridge to [bus 21]
pci_bus 0002:20: resource 4 [io 0x300000-0x3fffff]
pci_bus 0002:20: resource 5 [mem 0xf2200000-0xf2ffffff]
pci_bus 0002:20: resource 6 [mem 0x980000000-0x9bfffffff] pcieport 0002:20:00.0: vgaarb: pci_notify
pcieport 0002:20:00.0: assign IRQ: got 148
pcieport 0002:20:00.0: PME: Signaling with IRQ 157
pcieport 0002:20:00.0: AER: enabled with IRQ 158
pcieport 0002:20:00.0: bwctrl: enabled with IRQ 157
pcieport 0002:20:00.0: save config 0x00: 0x35881d87
pcieport 0002:20:00.0: save config 0x04: 0x00100507
pcieport 0002:20:00.0: save config 0x08: 0x06040001
pcieport 0002:20:00.0: save config 0x0c: 0x00010000
pcieport 0002:20:00.0: save config 0x10: 0x00000000
pcieport 0002:20:00.0: save config 0x14: 0x00000000
pcieport 0002:20:00.0: save config 0x18: 0x00212120
pcieport 0002:20:00.0: save config 0x1c: 0x000000f0
pcieport 0002:20:00.0: save config 0x20: 0x0000fff0
pcieport 0002:20:00.0: save config 0x24: 0x0001fff1
pcieport 0002:20:00.0: save config 0x28: 0x00000000
pcieport 0002:20:00.0: save config 0x2c: 0x00000000
pcieport 0002:20:00.0: save config 0x30: 0x00000000
pcieport 0002:20:00.0: save config 0x34: 0x00000040
pcieport 0002:20:00.0: save config 0x38: 0x00000000
pcieport 0002:20:00.0: save config 0x3c: 0x00020194
pcieport 0002:20:00.0: vgaarb: pci_notify
pcieport 0002:20:00.0: save config 0x00: 0x35881d87
pcieport 0002:20:00.0: save config 0x04: 0x00100507
pcieport 0002:20:00.0: save config 0x08: 0x06040001
pcieport 0002:20:00.0: save config 0x0c: 0x00010000
pcieport 0002:20:00.0: save config 0x10: 0x00000000
pcieport 0002:20:00.0: save config 0x14: 0x00000000
pcieport 0002:20:00.0: save config 0x18: 0x00212120
pcieport 0002:20:00.0: save config 0x1c: 0x000000f0
pcieport 0002:20:00.0: save config 0x20: 0x0000fff0
pcieport 0002:20:00.0: save config 0x24: 0x0001fff1
pcieport 0002:20:00.0: save config 0x28: 0x00000000
pcieport 0002:20:00.0: save config 0x2c: 0x00000000
pcieport 0002:20:00.0: save config 0x30: 0x00000000
pcieport 0002:20:00.0: save config 0x34: 0x00000040
pcieport 0002:20:00.0: save config 0x38: 0x00000000
pcieport 0002:20:00.0: save config 0x3c: 0x00020194
pcieport 0002:20:00.0: PME# enabled
Quoted 129 lines
Signature · 1 line
 i.
Quoted 126 lines
And this is iomem:
0010f000-0010f0ff : 10f000.sram sram@10f000 00200000-e2bbffff : System RAM
02010000-0474ffff : Kernel code 04750000-0498ffff : reserved 04990000-0508ffff : Kernel data daa00000-e29fffff : reserved
e2bc0000-ecbbffff : reserved
e2bc0000-ecbbffff : reserved
ecbc0000-efffffff : System RAM
ecbc7000-ecbdffff : reserved
f0000000-f00fffff : a40000000.pcie config f0200000-f0ffffff : pcie@fe150000
f0200000-f020ffff : 0000:00:00.0 f0300000-f03fffff : PCI Bus 0000:01
    f0300000-f0303fff : 0000:01:00.0
      f0300000-f0303fff : nvme
    f0304000-f03040ff : 0000:01:00.0
      f0304000-f03040ff : nvme
f2000000-f20fffff : a40800000.pcie config
f2200000-f2ffffff : pcie@fe170000
  f2200000-f27fffff : PCI Bus 0002:21
    f2200000-f220ffff : 0002:21:00.0
    f2400000-f27fffff : 0002:21:00.0
f2800000-f280ffff : 0002:20:00.0
f3000000-f30fffff : a40c00000.pcie config
f3200000-f3ffffff : pcie@fe180000
f3200000-f320ffff : 0003:30:00.0
f3300000-f33fffff : PCI Bus 0003:31
    f3300000-f3303fff : 0003:31:00.0
    f3304000-f3304fff : 0003:31:00.0
      f3304000-f3304fff : r8169
fb000000-fb1fffff : fb000000.gpu gpu@fb000000 fc00c100-fc3fffff : fc000000.usb usb@fc000000 fc400000-fc407fff : usb@fc400000
fc400000-fc407fff : xhci-hcd.10.auto usb@fc400000
fc40c100-fc7fffff : fc400000.usb usb@fc400000 fc800000-fc83ffff : fc800000.usb usb@fc800000 fc840000-fc87ffff : fc840000.usb usb@fc840000 fc880000-fc8bffff : fc880000.usb usb@fc880000 fc8c0000-fc8fffff : fc8c0000.usb usb@fc8c0000 fc900000-fc900dff : fc900000.iommu
fc910000-fc910dff : fc900000.iommu
fd600000-fd6fffff : fd600000.sram sram@fd600000 fd8a0000-fd8a00ff : fd8a0000.gpio gpio@fd8a0000 fdb50000-fdb507ff : fdb50000.video-codec video-codec@fdb50000 fdb50800-fdb5083f : fdb50800.iommu iommu@fdb50800 fdb80000-fdb8017f : fdb80000.rga rga@fdb80000 fdba0000-fdba07ff : fdba0000.video-codec video-codec@fdba0000 fdba0800-fdba083f : fdba0800.iommu iommu@fdba0800 fdba4800-fdba483f : fdba4800.iommu iommu@fdba4800 fdba8800-fdba883f : fdba8800.iommu iommu@fdba8800 fdbac800-fdbac83f : fdbac800.iommu iommu@fdbac800 fdc70000-fdc707ff : fdc70000.video-codec video-codec@fdc70000 fdd90000-fdd941ff : fdd90000.vop vop
fdd95000-fdd95fff : fdd90000.vop gamma-lut fdd97e00-fdd97eff : fdd97e00.iommu iommu@fdd97e00 fdd97f00-fdd97fff : fdd97e00.iommu iommu@fdd97e00 fddf0000-fddf0fff : fddf0000.i2s i2s@fddf0000 fddf4000-fddf4fff : fddf4000.i2s i2s@fddf4000 fde80000-fde9ffff : fde80000.hdmi hdmi@fde80000 fdea0000-fdebffff : fdea0000.hdmi hdmi@fdea0000 fdee0000-fdee5fff : fdee0000.hdmi_receiver hdmi_receiver@fdee0000 fe060000-fe06ffff : fe060000.dfi dfi@fe060000 fe150000-fe15ffff : a40000000.pcie apb
fe170000-fe17ffff : a40800000.pcie apb
fe180000-fe18ffff : a40c00000.pcie apb
fe1b0000-fe1bffff : fe1b0000.ethernet ethernet@fe1b0000 fe210000-fe210fff : fe210000.sata sata@fe210000 fe2c0000-fe2c3fff : fe2c0000.mmc mmc@fe2c0000 fe2e0000-fe2effff : fe2e0000.mmc mmc@fe2e0000 fe470000-fe470fff : fe470000.i2s i2s@fe470000 fe600000-fe60ffff : GICD
fe680000-fe77ffff : GICR
fea10000-fea13fff : dma-controller@fea10000
fea10000-fea13fff : fea10000.dma-controller dma-controller@fea10000
fea30000-fea33fff : dma-controller@fea30000
fea30000-fea33fff : fea30000.dma-controller dma-controller@fea30000
feaa0000-feaa0fff : feaa0000.i2c i2c@feaa0000 feaf0000-feaf00ff : feaf0000.watchdog watchdog@feaf0000 feb20000-feb20fff : feb20000.spi spi@feb20000 feb50000-feb500ff : serial
fec00000-fec003ff : fec00000.tsadc tsadc@fec00000 fec10000-fec1ffff : fec10000.adc adc@fec10000 fec20000-fec200ff : fec20000.gpio gpio@fec20000 fec30000-fec300ff : fec30000.gpio gpio@fec30000 fec40000-fec400ff : fec40000.gpio gpio@fec40000 fec50000-fec500ff : fec50000.gpio gpio@fec50000 fec90000-fec90fff : fec90000.i2c i2c@fec90000 fed10000-fed13fff : dma-controller@fed10000
fed10000-fed13fff : fed10000.dma-controller dma-controller@fed10000
fed60000-fed61fff : fed60000.phy phy@fed60000 fed70000-fed71fff : fed70000.phy phy@fed70000 fed80000-fed8ffff : fed80000.phy phy@fed80000 fed90000-fed9ffff : fed90000.phy phy@fed90000 fee00000-fee000ff : fee00000.phy phy@fee00000 fee10000-fee100ff : fee10000.phy phy@fee10000 fee20000-fee200ff : fee20000.phy phy@fee20000 fee80000-fee9ffff : fee80000.phy phy@fee80000 ff001000-ff0effff : ff001000.sram sram@ff001000 100000000-3fbffffff : System RAM
3ec000000-3fbffffff : reserved
3fc500000-3ffefffff : System RAM
4f0000000-4ffffffff : System RAM
4fc611000-4fc6d0fff : reserved
4fc6d1000-4fded1fff : reserved
4fded2000-4fdf91fff : reserved
4fdf93000-4fdf96fff : reserved
4fdf97000-4fdfabfff : reserved
4fdfac000-4fe051fff : reserved
4fe052000-4ffffffff : reserved
900000000-93fffffff : pcie@fe150000
900000000-93fffffff : 0000:00:00.0
980000000-9bfffffff : pcie@fe170000
9c0000000-9ffffffff : pcie@fe180000
a40000000-a403fffff : a40000000.pcie dbi a40800000-a40bfffff : a40800000.pcie dbi a40c00000-a40ffffff : a40c00000.pcie dbi
Thank you,
	o.
Quoted 2 lines
Signature · 1 line
 i.

Ondřej Jirman

2025-04-14 13:43 UTC #34 lore raw

Re: [PATCH 24/25] PCI: Perform reset_resource() and build fail list in sync

Quoted 46 lines · Ilpo Järvinen

Quoted 25 lines · Ondřej Jirman
Quoted 24 lines · Ilpo Järvinen
Quoted 21 lines · Ondřej Jirman
Hello Ilpo,
Quoted 8 lines · Ilpo Järvinen
That log wasn't taken from a bad case but it doesn't matter anymore as I could test this with qemu myself, thanks for providing enough to make it easy to reproduce & test it locally :-).
The problem is caused by assign+release cycle being destructive on start aligned resources because successful assigment overwrites the start field. I'll send a patch to fix the problem once I've given it a bit more thought as this resource fitting code is somewhat complex.
BTW, same thing here on a different SoC:
https://lore.kernel.org/lkml/hrcsm2bo4ysqj2ggejndlou32cdc7yiknnm5nrlcoz4d64wall@7te4dfqsoe3y/T/#u
There are kernel logs there, too, although I don't have dynamic debug enabled in the kernel.
Interestingly, bisect pointed me initially to a different commit. Reverting it helped, but just on one board (QuartzPro64).
Quoted 7 lines
Quoted 5 lines
Hi,
Since you didn't mention it, I guess you haven't tried the fix:
https://patchwork.kernel.org/project/linux-pci/patch/[email protected]/
This patch works. Thank you.
Thanks for testing. If you feel confident enough, please send your Tested-by tag to its thread (not this one).
Quoted 4 lines
One difference compared to 6.14 that I'm noticing
in the kernel log is that "save config" sequences now are printed twice for unpopulated port. Not sure if it's related to your patches. Previously it was printed just once.
I don't think it is related. The resource fitting/assignment changes should not impact PCI save/restore AFAIK (except by preventing device from working in the first place if necessary resources do not get assigned).

Quoted 6 lines

PCI state saving has always seemed like that in most logs I've seen with dyndbg enabled, that is, the state is seemingly saved multiple times, often right after the previous save too. So TBH, I'm not convinced it's even something new. If you have backtraces to show those places that invoke pci_save_state() I can take a look but I don't expect much to come out of that.

You're right. It's unrelated. The traces:

[    1.878732]  show_stack+0x28/0x80 (C)
[    1.878751]  dump_stack_lvl+0x58/0x74
[    1.878762]  dump_stack+0x14/0x1c
[    1.878772]  pci_save_state+0x34/0x1e8
[    1.878783]  pcie_portdrv_probe+0x330/0x6a8
[    1.878794]  local_pci_probe+0x3c/0xa0
[    1.878804]  pci_device_probe+0xac/0x194
[    1.878813]  really_probe+0xbc/0x388
[    1.878825]  __driver_probe_device+0x78/0x144
[    1.878837]  driver_probe_device+0x38/0x110
[    1.878850]  __device_attach_driver+0xb0/0x150
[    1.878862]  bus_for_each_drv+0x6c/0xb0
[    1.878873]  __device_attach+0x98/0x1a0
[    1.878886]  device_attach+0x10/0x20
[    1.878897]  pci_bus_add_device+0x84/0x12c
[    1.878911]  pci_bus_add_devices+0x40/0x90
[    1.878924]  pci_host_probe+0x88/0xe4
[    1.878933]  dw_pcie_host_init+0x258/0x714
[    1.878945]  rockchip_pcie_probe+0x2f0/0x3f8
[    1.878956]  platform_probe+0x64/0xcc
[    1.878965]  really_probe+0xbc/0x388
[    1.878977]  __driver_probe_device+0x78/0x144
[    1.878989]  driver_probe_device+0x38/0x110
[    1.879001]  __device_attach_driver+0xb0/0x150
[    1.879014]  bus_for_each_drv+0x6c/0xb0
[    1.879025]  __device_attach+0x98/0x1a0
[    1.879037]  device_initial_probe+0x10/0x20
[    1.879049]  bus_probe_device+0xa8/0xb8
[    1.879061]  deferred_probe_work_func+0xa4/0xf0
[    1.879072]  process_one_work+0x13c/0x2bc
[    1.879084]  worker_thread+0x284/0x460
[    1.879094]  kthread+0xe4/0x1a0
[    1.879107]  ret_from_fork+0x10/0x20
[    1.879121] pcieport 0002:20:00.0: save config 0x00: 0x35881d87
[    1.879158] pcieport 0002:20:00.0: save config 0x04: 0x00100507
[    1.879168] pcieport 0002:20:00.0: save config 0x08: 0x06040001
[    1.879177] pcieport 0002:20:00.0: save config 0x0c: 0x00010000
[    1.879221] pcieport 0002:20:00.0: save config 0x10: 0x00000000
[    1.879232] pcieport 0002:20:00.0: save config 0x14: 0x00000000
[    1.879242] pcieport 0002:20:00.0: save config 0x18: 0x00212120
[    1.879251] pcieport 0002:20:00.0: save config 0x1c: 0x000000f0
[    1.879260] pcieport 0002:20:00.0: save config 0x20: 0x0000fff0
[    1.879270] pcieport 0002:20:00.0: save config 0x24: 0x0001fff1
[    1.879279] pcieport 0002:20:00.0: save config 0x28: 0x00000000
[    1.879289] pcieport 0002:20:00.0: save config 0x2c: 0x00000000
[    1.879298] pcieport 0002:20:00.0: save config 0x30: 0x00000000
[    1.879307] pcieport 0002:20:00.0: save config 0x34: 0x00000040
[    1.879316] pcieport 0002:20:00.0: save config 0x38: 0x00000000
[    1.879325] pcieport 0002:20:00.0: save config 0x3c: 0x00020194               
[    1.879376] pcieport 0002:20:00.0: vgaarb: pci_notify

second time it's from:

[    2.004478] Workqueue: pm pm_runtime_work
[    2.004488] Call trace:
[    2.004491]  show_stack+0x28/0x80 (C)
[    2.004500]  dump_stack_lvl+0x58/0x74
[    2.004506]  dump_stack+0x14/0x1c
[    2.004511]  pci_save_state+0x34/0x1e8
[    2.004516]  pci_pm_runtime_suspend+0xa8/0x1e0
[    2.004521]  __rpm_callback+0x3c/0x220
[    2.004527]  rpm_callback+0x6c/0x80
[    2.004532]  rpm_suspend+0xec/0x674
[    2.004537]  pm_runtime_work+0x104/0x114
[    2.004542]  process_one_work+0x13c/0x2bc
[    2.004548]  worker_thread+0x284/0x460
[    2.004552]  kthread+0xe4/0x1a0
[    2.004559]  ret_from_fork+0x10/0x20
[    2.004567] pcieport 0002:20:00.0: save config 0x00: 0x35881d87
[    2.004578] pcieport 0002:20:00.0: save config 0x04: 0x00100507
[    2.004583] pcieport 0002:20:00.0: save config 0x08: 0x06040001
[    2.004587] pcieport 0002:20:00.0: save config 0x0c: 0x00010000
[    2.004591] pcieport 0002:20:00.0: save config 0x10: 0x00000000
[    2.004596] pcieport 0002:20:00.0: save config 0x14: 0x00000000
[    2.004600] pcieport 0002:20:00.0: save config 0x18: 0x00212120
[    2.004604] pcieport 0002:20:00.0: save config 0x1c: 0x000000f0
[    2.004608] pcieport 0002:20:00.0: save config 0x20: 0x0000fff0
[    2.004613] pcieport 0002:20:00.0: save config 0x24: 0x0001fff1
[    2.004617] pcieport 0002:20:00.0: save config 0x28: 0x00000000
[    2.004621] pcieport 0002:20:00.0: save config 0x2c: 0x00000000
[    2.004625] pcieport 0002:20:00.0: save config 0x30: 0x00000000
[    2.004629] pcieport 0002:20:00.0: save config 0x34: 0x00000040
[    2.004633] pcieport 0002:20:00.0: save config 0x38: 0x00000000
[    2.004638] pcieport 0002:20:00.0: save config 0x3c: 0x00020194
[    2.004662] pcieport 0002:20:00.0: PME# enabled

on 6.15-rc:

cat /sys/class/pci_bus/0002:20/device/0002:20:00.0/power/runtime_status OK
suspended

on 6.14:

cat /sys/class/pci_bus/0002:20/device/0002:20:00.0/power/runtime_status OK
active

So some other unrelated change in 6.15-rc just enabled RPM to suspend the unused port.

Thnak you,

o.

Quoted 205 lines

Signature · 1 line
 i.
Quoted 202 lines
Kind regards,
	o.
rockchip-dw-pcie a40800000.pcie: PCI host bridge to bus 0002:20
pci_bus 0002:20: root bus resource [bus 20-2f]
pci_bus 0002:20: root bus resource [io 0x300000-0x3fffff] (bus address [0xf2100000-0xf21fffff]) pci_bus 0002:20: root bus resource [mem 0xf2200000-0xf2ffffff]
pci_bus 0002:20: root bus resource [mem 0x980000000-0x9bfffffff] (bus address [0x40000000-0x7fffffff]) pci_bus 0002:20: scanning bus
pci 0002:20:00.0: [1d87:3588] type 01 class 0x060400 PCIe Root Port
pci 0002:20:00.0: ROM [mem 0x00000000-0x0000ffff pref]
pci 0002:20:00.0: PCI bridge to [bus 01-ff]
pci 0002:20:00.0:   bridge window [io  0x0000-0x0fff]
pci 0002:20:00.0:   bridge window [mem 0x00000000-0x000fffff]
pci 0002:20:00.0:   bridge window [mem 0x00000000-0x000fffff 64bit pref]
pci 0002:20:00.0: supports D1 D2
pci 0002:20:00.0: PME# supported from D0 D1 D3hot
pci 0002:20:00.0: PME# disabled
pci 0002:20:00.0: Adding to iommu group 9
pci 0002:20:00.0: vgaarb: pci_notify
pci_bus 0002:20: fixups for bus
pci 0002:20:00.0: scanning [bus 01-ff] behind bridge, pass 0 pci 0002:20:00.0: Primary bus is hard wired to 0
pci 0002:20:00.0: bridge configuration invalid ([bus 01-ff]), reconfiguring pci 0002:20:00.0: scanning [bus 00-00] behind bridge, pass 1 pci_bus 0002:20: bus scan returning with max=21
pci 0002:20:00.0: ROM [mem 0xf2200000-0xf220ffff pref]: assigned pci 0002:20:00.0: PCI bridge to [bus 21]
pci_bus 0002:20: resource 4 [io 0x300000-0x3fffff]
pci_bus 0002:20: resource 5 [mem 0xf2200000-0xf2ffffff]
pci_bus 0002:20: resource 6 [mem 0x980000000-0x9bfffffff] pcieport 0002:20:00.0: vgaarb: pci_notify
pcieport 0002:20:00.0: assign IRQ: got 148
pcieport 0002:20:00.0: PME: Signaling with IRQ 157
pcieport 0002:20:00.0: AER: enabled with IRQ 158
pcieport 0002:20:00.0: bwctrl: enabled with IRQ 157
pcieport 0002:20:00.0: save config 0x00: 0x35881d87
pcieport 0002:20:00.0: save config 0x04: 0x00100507
pcieport 0002:20:00.0: save config 0x08: 0x06040001
pcieport 0002:20:00.0: save config 0x0c: 0x00010000
pcieport 0002:20:00.0: save config 0x10: 0x00000000
pcieport 0002:20:00.0: save config 0x14: 0x00000000
pcieport 0002:20:00.0: save config 0x18: 0x00212120
pcieport 0002:20:00.0: save config 0x1c: 0x000000f0
pcieport 0002:20:00.0: save config 0x20: 0x0000fff0
pcieport 0002:20:00.0: save config 0x24: 0x0001fff1
pcieport 0002:20:00.0: save config 0x28: 0x00000000
pcieport 0002:20:00.0: save config 0x2c: 0x00000000
pcieport 0002:20:00.0: save config 0x30: 0x00000000
pcieport 0002:20:00.0: save config 0x34: 0x00000040
pcieport 0002:20:00.0: save config 0x38: 0x00000000
pcieport 0002:20:00.0: save config 0x3c: 0x00020194
pcieport 0002:20:00.0: vgaarb: pci_notify
pcieport 0002:20:00.0: save config 0x00: 0x35881d87
pcieport 0002:20:00.0: save config 0x04: 0x00100507
pcieport 0002:20:00.0: save config 0x08: 0x06040001
pcieport 0002:20:00.0: save config 0x0c: 0x00010000
pcieport 0002:20:00.0: save config 0x10: 0x00000000
pcieport 0002:20:00.0: save config 0x14: 0x00000000
pcieport 0002:20:00.0: save config 0x18: 0x00212120
pcieport 0002:20:00.0: save config 0x1c: 0x000000f0
pcieport 0002:20:00.0: save config 0x20: 0x0000fff0
pcieport 0002:20:00.0: save config 0x24: 0x0001fff1
pcieport 0002:20:00.0: save config 0x28: 0x00000000
pcieport 0002:20:00.0: save config 0x2c: 0x00000000
pcieport 0002:20:00.0: save config 0x30: 0x00000000
pcieport 0002:20:00.0: save config 0x34: 0x00000040
pcieport 0002:20:00.0: save config 0x38: 0x00000000
pcieport 0002:20:00.0: save config 0x3c: 0x00020194
pcieport 0002:20:00.0: PME# enabled
Quoted 129 lines
Signature · 1 line
 i.
Quoted 126 lines
And this is iomem:
0010f000-0010f0ff : 10f000.sram sram@10f000 00200000-e2bbffff : System RAM
02010000-0474ffff : Kernel code 04750000-0498ffff : reserved 04990000-0508ffff : Kernel data daa00000-e29fffff : reserved
e2bc0000-ecbbffff : reserved
e2bc0000-ecbbffff : reserved
ecbc0000-efffffff : System RAM
ecbc7000-ecbdffff : reserved
f0000000-f00fffff : a40000000.pcie config f0200000-f0ffffff : pcie@fe150000
f0200000-f020ffff : 0000:00:00.0 f0300000-f03fffff : PCI Bus 0000:01
    f0300000-f0303fff : 0000:01:00.0
      f0300000-f0303fff : nvme
    f0304000-f03040ff : 0000:01:00.0
      f0304000-f03040ff : nvme
f2000000-f20fffff : a40800000.pcie config
f2200000-f2ffffff : pcie@fe170000
  f2200000-f27fffff : PCI Bus 0002:21
    f2200000-f220ffff : 0002:21:00.0
    f2400000-f27fffff : 0002:21:00.0
f2800000-f280ffff : 0002:20:00.0
f3000000-f30fffff : a40c00000.pcie config
f3200000-f3ffffff : pcie@fe180000
f3200000-f320ffff : 0003:30:00.0
f3300000-f33fffff : PCI Bus 0003:31
    f3300000-f3303fff : 0003:31:00.0
    f3304000-f3304fff : 0003:31:00.0
      f3304000-f3304fff : r8169
fb000000-fb1fffff : fb000000.gpu gpu@fb000000 fc00c100-fc3fffff : fc000000.usb usb@fc000000 fc400000-fc407fff : usb@fc400000
fc400000-fc407fff : xhci-hcd.10.auto usb@fc400000
fc40c100-fc7fffff : fc400000.usb usb@fc400000 fc800000-fc83ffff : fc800000.usb usb@fc800000 fc840000-fc87ffff : fc840000.usb usb@fc840000 fc880000-fc8bffff : fc880000.usb usb@fc880000 fc8c0000-fc8fffff : fc8c0000.usb usb@fc8c0000 fc900000-fc900dff : fc900000.iommu
fc910000-fc910dff : fc900000.iommu
fd600000-fd6fffff : fd600000.sram sram@fd600000 fd8a0000-fd8a00ff : fd8a0000.gpio gpio@fd8a0000 fdb50000-fdb507ff : fdb50000.video-codec video-codec@fdb50000 fdb50800-fdb5083f : fdb50800.iommu iommu@fdb50800 fdb80000-fdb8017f : fdb80000.rga rga@fdb80000 fdba0000-fdba07ff : fdba0000.video-codec video-codec@fdba0000 fdba0800-fdba083f : fdba0800.iommu iommu@fdba0800 fdba4800-fdba483f : fdba4800.iommu iommu@fdba4800 fdba8800-fdba883f : fdba8800.iommu iommu@fdba8800 fdbac800-fdbac83f : fdbac800.iommu iommu@fdbac800 fdc70000-fdc707ff : fdc70000.video-codec video-codec@fdc70000 fdd90000-fdd941ff : fdd90000.vop vop
fdd95000-fdd95fff : fdd90000.vop gamma-lut fdd97e00-fdd97eff : fdd97e00.iommu iommu@fdd97e00 fdd97f00-fdd97fff : fdd97e00.iommu iommu@fdd97e00 fddf0000-fddf0fff : fddf0000.i2s i2s@fddf0000 fddf4000-fddf4fff : fddf4000.i2s i2s@fddf4000 fde80000-fde9ffff : fde80000.hdmi hdmi@fde80000 fdea0000-fdebffff : fdea0000.hdmi hdmi@fdea0000 fdee0000-fdee5fff : fdee0000.hdmi_receiver hdmi_receiver@fdee0000 fe060000-fe06ffff : fe060000.dfi dfi@fe060000 fe150000-fe15ffff : a40000000.pcie apb
fe170000-fe17ffff : a40800000.pcie apb
fe180000-fe18ffff : a40c00000.pcie apb
fe1b0000-fe1bffff : fe1b0000.ethernet ethernet@fe1b0000 fe210000-fe210fff : fe210000.sata sata@fe210000 fe2c0000-fe2c3fff : fe2c0000.mmc mmc@fe2c0000 fe2e0000-fe2effff : fe2e0000.mmc mmc@fe2e0000 fe470000-fe470fff : fe470000.i2s i2s@fe470000 fe600000-fe60ffff : GICD
fe680000-fe77ffff : GICR
fea10000-fea13fff : dma-controller@fea10000
fea10000-fea13fff : fea10000.dma-controller dma-controller@fea10000
fea30000-fea33fff : dma-controller@fea30000
fea30000-fea33fff : fea30000.dma-controller dma-controller@fea30000
feaa0000-feaa0fff : feaa0000.i2c i2c@feaa0000 feaf0000-feaf00ff : feaf0000.watchdog watchdog@feaf0000 feb20000-feb20fff : feb20000.spi spi@feb20000 feb50000-feb500ff : serial
fec00000-fec003ff : fec00000.tsadc tsadc@fec00000 fec10000-fec1ffff : fec10000.adc adc@fec10000 fec20000-fec200ff : fec20000.gpio gpio@fec20000 fec30000-fec300ff : fec30000.gpio gpio@fec30000 fec40000-fec400ff : fec40000.gpio gpio@fec40000 fec50000-fec500ff : fec50000.gpio gpio@fec50000 fec90000-fec90fff : fec90000.i2c i2c@fec90000 fed10000-fed13fff : dma-controller@fed10000
fed10000-fed13fff : fed10000.dma-controller dma-controller@fed10000
fed60000-fed61fff : fed60000.phy phy@fed60000 fed70000-fed71fff : fed70000.phy phy@fed70000 fed80000-fed8ffff : fed80000.phy phy@fed80000 fed90000-fed9ffff : fed90000.phy phy@fed90000 fee00000-fee000ff : fee00000.phy phy@fee00000 fee10000-fee100ff : fee10000.phy phy@fee10000 fee20000-fee200ff : fee20000.phy phy@fee20000 fee80000-fee9ffff : fee80000.phy phy@fee80000 ff001000-ff0effff : ff001000.sram sram@ff001000 100000000-3fbffffff : System RAM
3ec000000-3fbffffff : reserved
3fc500000-3ffefffff : System RAM
4f0000000-4ffffffff : System RAM
4fc611000-4fc6d0fff : reserved
4fc6d1000-4fded1fff : reserved
4fded2000-4fdf91fff : reserved
4fdf93000-4fdf96fff : reserved
4fdf97000-4fdfabfff : reserved
4fdfac000-4fe051fff : reserved
4fe052000-4ffffffff : reserved
900000000-93fffffff : pcie@fe150000
900000000-93fffffff : 0000:00:00.0
980000000-9bfffffff : pcie@fe170000
9c0000000-9ffffffff : pcie@fe180000
a40000000-a403fffff : a40000000.pcie dbi a40800000-a40bfffff : a40800000.pcie dbi a40c00000-a40ffffff : a40c00000.pcie dbi
Thank you,
	o.
Quoted 2 lines
Signature · 1 line
 i.

Ilpo Järvinen

author

2025-04-14 13:52 UTC #35 lore raw

Re: [PATCH 24/25] PCI: Perform reset_resource() and build fail list in sync

Quoted 147 lines · Ondřej Jirman

Quoted 46 lines · Ilpo Järvinen
Quoted 25 lines · Ondřej Jirman
Quoted 24 lines · Ilpo Järvinen
Quoted 21 lines · Ondřej Jirman
Hello Ilpo,
Quoted 8 lines · Ilpo Järvinen
That log wasn't taken from a bad case but it doesn't matter anymore as I could test this with qemu myself, thanks for providing enough to make it easy to reproduce & test it locally :-).
The problem is caused by assign+release cycle being destructive on start aligned resources because successful assigment overwrites the start field. I'll send a patch to fix the problem once I've given it a bit more thought as this resource fitting code is somewhat complex.
BTW, same thing here on a different SoC:
https://lore.kernel.org/lkml/hrcsm2bo4ysqj2ggejndlou32cdc7yiknnm5nrlcoz4d64wall@7te4dfqsoe3y/T/#u
There are kernel logs there, too, although I don't have dynamic debug enabled in the kernel.
Interestingly, bisect pointed me initially to a different commit. Reverting it helped, but just on one board (QuartzPro64).
Quoted 7 lines
Quoted 5 lines
Hi,
Since you didn't mention it, I guess you haven't tried the fix:
https://patchwork.kernel.org/project/linux-pci/patch/[email protected]/
This patch works. Thank you.
Thanks for testing. If you feel confident enough, please send your Tested-by tag to its thread (not this one).
Quoted 4 lines
One difference compared to 6.14 that I'm noticing
in the kernel log is that "save config" sequences now are printed twice for unpopulated port. Not sure if it's related to your patches. Previously it was printed just once.
I don't think it is related. The resource fitting/assignment changes should not impact PCI save/restore AFAIK (except by preventing device from working in the first place if necessary resources do not get assigned).
Quoted 6 lines
PCI state saving has always seemed like that in most logs I've seen with dyndbg enabled, that is, the state is seemingly saved multiple times, often right after the previous save too. So TBH, I'm not convinced it's even something new. If you have backtraces to show those places that invoke pci_save_state() I can take a look but I don't expect much to come out of that.
You're right. It's unrelated. The traces:
[    1.878732]  show_stack+0x28/0x80 (C)
[    1.878751]  dump_stack_lvl+0x58/0x74
[    1.878762]  dump_stack+0x14/0x1c
[    1.878772]  pci_save_state+0x34/0x1e8
[    1.878783]  pcie_portdrv_probe+0x330/0x6a8
[    1.878794]  local_pci_probe+0x3c/0xa0
[    1.878804]  pci_device_probe+0xac/0x194
[    1.878813]  really_probe+0xbc/0x388
[    1.878825]  __driver_probe_device+0x78/0x144
[    1.878837]  driver_probe_device+0x38/0x110
[    1.878850]  __device_attach_driver+0xb0/0x150
[    1.878862]  bus_for_each_drv+0x6c/0xb0
[    1.878873]  __device_attach+0x98/0x1a0
[    1.878886]  device_attach+0x10/0x20
[    1.878897]  pci_bus_add_device+0x84/0x12c
[    1.878911]  pci_bus_add_devices+0x40/0x90
[    1.878924]  pci_host_probe+0x88/0xe4
[    1.878933]  dw_pcie_host_init+0x258/0x714
[    1.878945]  rockchip_pcie_probe+0x2f0/0x3f8
[    1.878956]  platform_probe+0x64/0xcc
[    1.878965]  really_probe+0xbc/0x388
[    1.878977]  __driver_probe_device+0x78/0x144
[    1.878989]  driver_probe_device+0x38/0x110
[    1.879001]  __device_attach_driver+0xb0/0x150
[    1.879014]  bus_for_each_drv+0x6c/0xb0
[    1.879025]  __device_attach+0x98/0x1a0
[    1.879037]  device_initial_probe+0x10/0x20
[    1.879049]  bus_probe_device+0xa8/0xb8
[    1.879061]  deferred_probe_work_func+0xa4/0xf0
[    1.879072]  process_one_work+0x13c/0x2bc
[    1.879084]  worker_thread+0x284/0x460
[    1.879094]  kthread+0xe4/0x1a0
[    1.879107]  ret_from_fork+0x10/0x20
[    1.879121] pcieport 0002:20:00.0: save config 0x00: 0x35881d87
[    1.879158] pcieport 0002:20:00.0: save config 0x04: 0x00100507
[    1.879168] pcieport 0002:20:00.0: save config 0x08: 0x06040001
[    1.879177] pcieport 0002:20:00.0: save config 0x0c: 0x00010000
[    1.879221] pcieport 0002:20:00.0: save config 0x10: 0x00000000
[    1.879232] pcieport 0002:20:00.0: save config 0x14: 0x00000000
[    1.879242] pcieport 0002:20:00.0: save config 0x18: 0x00212120
[    1.879251] pcieport 0002:20:00.0: save config 0x1c: 0x000000f0
[    1.879260] pcieport 0002:20:00.0: save config 0x20: 0x0000fff0
[    1.879270] pcieport 0002:20:00.0: save config 0x24: 0x0001fff1
[    1.879279] pcieport 0002:20:00.0: save config 0x28: 0x00000000
[    1.879289] pcieport 0002:20:00.0: save config 0x2c: 0x00000000
[    1.879298] pcieport 0002:20:00.0: save config 0x30: 0x00000000
[    1.879307] pcieport 0002:20:00.0: save config 0x34: 0x00000040
[    1.879316] pcieport 0002:20:00.0: save config 0x38: 0x00000000
[    1.879325] pcieport 0002:20:00.0: save config 0x3c: 0x00020194               
[    1.879376] pcieport 0002:20:00.0: vgaarb: pci_notify
second time it's from:
[    2.004478] Workqueue: pm pm_runtime_work
[    2.004488] Call trace:
[    2.004491]  show_stack+0x28/0x80 (C)
[    2.004500]  dump_stack_lvl+0x58/0x74
[    2.004506]  dump_stack+0x14/0x1c
[    2.004511]  pci_save_state+0x34/0x1e8
[    2.004516]  pci_pm_runtime_suspend+0xa8/0x1e0
[    2.004521]  __rpm_callback+0x3c/0x220
[    2.004527]  rpm_callback+0x6c/0x80
[    2.004532]  rpm_suspend+0xec/0x674
[    2.004537]  pm_runtime_work+0x104/0x114
[    2.004542]  process_one_work+0x13c/0x2bc
[    2.004548]  worker_thread+0x284/0x460
[    2.004552]  kthread+0xe4/0x1a0
[    2.004559]  ret_from_fork+0x10/0x20
[    2.004567] pcieport 0002:20:00.0: save config 0x00: 0x35881d87
[    2.004578] pcieport 0002:20:00.0: save config 0x04: 0x00100507
[    2.004583] pcieport 0002:20:00.0: save config 0x08: 0x06040001
[    2.004587] pcieport 0002:20:00.0: save config 0x0c: 0x00010000
[    2.004591] pcieport 0002:20:00.0: save config 0x10: 0x00000000
[    2.004596] pcieport 0002:20:00.0: save config 0x14: 0x00000000
[    2.004600] pcieport 0002:20:00.0: save config 0x18: 0x00212120
[    2.004604] pcieport 0002:20:00.0: save config 0x1c: 0x000000f0
[    2.004608] pcieport 0002:20:00.0: save config 0x20: 0x0000fff0
[    2.004613] pcieport 0002:20:00.0: save config 0x24: 0x0001fff1
[    2.004617] pcieport 0002:20:00.0: save config 0x28: 0x00000000
[    2.004621] pcieport 0002:20:00.0: save config 0x2c: 0x00000000
[    2.004625] pcieport 0002:20:00.0: save config 0x30: 0x00000000
[    2.004629] pcieport 0002:20:00.0: save config 0x34: 0x00000040
[    2.004633] pcieport 0002:20:00.0: save config 0x38: 0x00000000
[    2.004638] pcieport 0002:20:00.0: save config 0x3c: 0x00020194
[    2.004662] pcieport 0002:20:00.0: PME# enabled
on 6.15-rc:
cat /sys/class/pci_bus/0002:20/device/0002:20:00.0/power/runtime_status OK
suspended

Quoted 7 lines

on 6.14:
cat /sys/class/pci_bus/0002:20/device/0002:20:00.0/power/runtime_status OK
active
So some other unrelated change in 6.15-rc just enabled RPM to suspend the unused port.

Ah right, makes sense, now that you mention I recall D3 became allowed for PCI bridge on non-x86 archs but I just couldn't make the connection in my mind.

Signature · 1 line

i.

Guenter Roeck

2025-04-01 13:28 UTC #36 lore raw

Re: [PATCH 24/25] PCI: Perform reset_resource() and build fail list in sync

In reply to Ilpo Järvinen

Quoted 108 lines · Ilpo Järvinen

Quoted 93 lines · Guenter Roeck
Quoted 18 lines · Ilpo Järvinen
Resetting resource is problematic as it prevent attempting to allocate the resource later, unless something in between restores the resource. Similarly, if fail_head does not contain all resources that were reset, those resource cannot be restored later.
The entire reset/restore cycle adds complexity and leaving resources into reseted state causes issues to other code such as for checks done in pci_enable_resources(). Take a small step towards not resetting resources by delaying reset until the end of resource assignment and build failure list (fail_head) in sync with the reset to avoid leaving behind resources that cannot be restored (for the case where the caller provides fail_head in the first place to allow restore somewhere in the callchain, as is not all callers pass non-NULL fail_head).
The Expansion ROM check is temporarily left in place while building the failure list until the upcoming change which reworks optional resource handling.
Quoted 5 lines
Ideally, whole resource reset could be removed but doing that in a big step would make the impact non-tractable due to complexity of all related code.
Signed-off-by Ilpo Järvinen <[email protected]>
With this patch in the mainline kernel, all mips:boston qemu emulations fail when running a 64-bit little endian configuration (64r6el_defconfig).
The problem is that the PCI based IDE/ATA controller is not initialized. There are a number of pci error messages.
pci_bus 0002:01: extended config space not accessible
pci 0002:01:00.0: [8086:2922] type 00 class 0x010601 conventional PCI endpoint pci 0002:01:00.0: BAR 4 [io 0x0000-0x001f]
pci 0002:01:00.0: BAR 5 [mem 0x00000000-0x00000fff]
pci 0002:00:00.0: PCI bridge to [bus 01-ff]
pci_bus 0002:01: busn_res: [bus 01-ff] end is updated to 01
pci 0002:00:00.0: bridge window [mem 0x16000000-0x160fffff]: assigned
pci 0002:00:00.0: bridge window [mem size 0x00100000 64bit pref]: can't assign; no space pci 0002:00:00.0: bridge window [mem size 0x00100000 64bit pref]: failed to assign pci 0002:00:00.0: bridge window [io size 0x1000]: can't assign; no space pci 0002:00:00.0: bridge window [io size 0x1000]: failed to assign
pci 0002:00:00.0: bridge window [mem size 0x00100000]: can't assign; bogus alignment pci 0002:00:00.0: bridge window [mem 0x16000000-0x160fffff 64bit pref]: assigned pci 0002:00:00.0: bridge window [io size 0x1000]: can't assign; no space pci 0002:00:00.0: bridge window [io size 0x1000]: failed to assign
pci 0002:01:00.0: BAR 5 [mem size 0x00001000]: can't assign; no space
pci 0002:01:00.0: BAR 5 [mem size 0x00001000]: failed to assign
pci 0002:01:00.0: BAR 4 [io size 0x0020]: can't assign; no space
pci 0002:01:00.0: BAR 4 [io size 0x0020]: failed to assign
pci 0002:01:00.0: BAR 5 [mem size 0x00001000]: can't assign; no space
pci 0002:01:00.0: BAR 5 [mem size 0x00001000]: failed to assign
pci 0002:01:00.0: BAR 4 [io size 0x0020]: can't assign; no space
pci 0002:01:00.0: BAR 4 [io size 0x0020]: failed to assign
pci 0002:00:00.0: PCI bridge to [bus 01]
pci 0002:00:00.0: bridge window [mem 0x16000000-0x160fffff 64bit pref]
pci_bus 0002:00: Some PCI device resources are unassigned, try booting with pci=realloc pci_bus 0002:00: resource 4 [mem 0x16000000-0x160fffff]
pci_bus 0002:01: resource 2 [mem 0x16000000-0x160fffff 64bit pref]
...
pci 0002:00:00.0: enabling device (0000 -> 0002)
ahci 0002:01:00.0: probe with driver ahci failed with error -12
Bisect points to this patch. Reverting it together with "PCI: Rework optional resource handling" fixes the problem. For comparison, after reverting the offending patches, the log messages are as follows.
pci_bus 0002:00: root bus resource [bus 00-ff]
pci_bus 0002:00: root bus resource [mem 0x16000000-0x160fffff]
pci 0002:00:00.0: [10ee:7021] type 01 class 0x060400 PCIe Root Complex Integrated Endpoint pci 0002:00:00.0: PCI bridge to [bus 00]
pci 0002:00:00.0:   bridge window [io  0x0000-0x0fff]
pci 0002:00:00.0:   bridge window [mem 0x00000000-0x000fffff]
pci 0002:00:00.0:   bridge window [mem 0x00000000-0x000fffff 64bit pref]
pci 0002:00:00.0: enabling Extended Tags
pci 0002:00:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring pci_bus 0002:01: extended config space not accessible
pci 0002:01:00.0: [8086:2922] type 00 class 0x010601 conventional PCI endpoint pci 0002:01:00.0: BAR 4 [io 0x0000-0x001f]
pci 0002:01:00.0: BAR 5 [mem 0x00000000-0x00000fff]
pci 0002:00:00.0: PCI bridge to [bus 01-ff]
pci_bus 0002:01: busn_res: [bus 01-ff] end is updated to 01
pci 0002:00:00.0: bridge window [mem 0x16000000-0x160fffff]: assigned
pci 0002:00:00.0: bridge window [mem size 0x00100000 64bit pref]: can't assign; no space pci 0002:00:00.0: bridge window [mem size 0x00100000 64bit pref]: failed to assign pci 0002:00:00.0: bridge window [io size 0x1000]: can't assign; no space pci 0002:00:00.0: bridge window [io size 0x1000]: failed to assign
pci 0002:01:00.0: BAR 5 [mem 0x16000000-0x16000fff]: assigned
pci 0002:01:00.0: BAR 4 [io size 0x0020]: can't assign; no space
pci 0002:01:00.0: BAR 4 [io size 0x0020]: failed to assign
pci 0002:00:00.0: PCI bridge to [bus 01]
pci 0002:00:00.0: bridge window [mem 0x16000000-0x160fffff]
pci_bus 0002:00: Some PCI device resources are unassigned, try booting with pci=realloc
Quoted 8 lines
pci_bus 0002:00: resource 4 [mem 0x16000000-0x160fffff]
pci_bus 0002:01: resource 1 [mem 0x16000000-0x160fffff]
...
pci 0002:00:00.0: enabling device (0000 -> 0002)
ahci 0002:01:00.0: enabling device (0000 -> 0002)
ahci 0002:01:00.0: AHCI vers 0001.0000, 32 command slots, 1.5 Gbps, SATA mode ahci 0002:01:00.0: 6/6 ports implemented (port mask 0x3f)
ahci 0002:01:00.0: flags: 64bit ncq only
Hi,
Thanks for reporting. Please add this to the command line to get the resource releasing between the steps to show:

Quoted 8 lines

dyndbg="file drivers/pci/setup-bus.c +p"
Also, the log snippet just shows it fails but it is impossible to know from it why the resource assigments do not fit so could you please provide a complete dmesg logs. Also providing the contents of /proc/iomem from the working case would save me quite a bit of decoding the iomem layout from the dmesgs.

You should find everything you need at http://server.roeck-us.net/qemu/mipsel64/.
Please let me know if you need anything else.

Thanks,
Guenter

Tudor Ambarus

2025-05-06 15:03 UTC #37 lore raw

Re: [PATCH 24/25] PCI: Perform reset_resource() and build fail list in sync

In reply to Ilpo Järvinen

Hi!

Quoted 18 lines · Ilpo Järvinen

Resetting resource is problematic as it prevent attempting to allocate the resource later, unless something in between restores the resource. Similarly, if fail_head does not contain all resources that were reset, those resource cannot be restored later.
The entire reset/restore cycle adds complexity and leaving resources into reseted state causes issues to other code such as for checks done in pci_enable_resources(). Take a small step towards not resetting resources by delaying reset until the end of resource assignment and build failure list (fail_head) in sync with the reset to avoid leaving behind resources that cannot be restored (for the case where the caller provides fail_head in the first place to allow restore somewhere in the callchain, as is not all callers pass non-NULL fail_head).
The Expansion ROM check is temporarily left in place while building the failure list until the upcoming change which reworks optional resource handling.

Quoted 5 lines

Ideally, whole resource reset could be removed but doing that in a big step would make the impact non-tractable due to complexity of all related code.
Signed-off-by Ilpo Järvinen <[email protected]>

I'm hitting the BUG_ON(!list_empty(&add_list)); in pci_assign_unassigned_bus_resources() [1] with 6.15-rc5 and the the pixel6 downstream pcie driver.

I saw the thread where "a34d74877c66 PCI: Restore assigned resources fully after release" fixes things for some other cases, but it's not the case here.

Reverting the following patches fixes the problem:
a34d74877c66 PCI: Restore assigned resources fully after release 2499f5348431 PCI: Rework optional resource handling 96336ec70264 PCI: Perform reset_resource() and build fail list in sync

In the working case the add_list list is empty throughout the entire body of pci_assign_unassigned_bus_resources().

In the failing case __pci_bus_size_bridges() leaves the add_list not empty and __pci_bus_assign_resources() does not consume the list, thus the BUG_ON. The failing case contains an extra print that's not shown when reverting the blamed commits:

[   13.951185][ T1101] pcieport 0000:00:00.0: bridge window [mem

0x00100000-0x001fffff] to [bus 01-ff] add_size 100000 add_align 100000

I've added some prints trying to describe the code path, see
https://paste.ofcode.org/Aeu2YBpLztc49ZDw3uUJmd#

Failing case:

[   13.944231][ T1101] pci 0000:01:00.0: [144d:a5a5] type 00 class
0x000000 PCIe Endpoint
[   13.944412][ T1101] pci 0000:01:00.0: BAR 0 [mem
0x00000000-0x000fffff 64bit]
[   13.944532][ T1101] pci 0000:01:00.0: ROM [mem 0x00000000-0x0000ffff
pref]
[   13.944649][ T1101] pci 0000:01:00.0: enabling Extended Tags
[   13.944844][ T1101] pci 0000:01:00.0: PME# supported from D0 D3hot D3cold
[   13.945015][ T1101] pci 0000:01:00.0: 15.752 Gb/s available PCIe
bandwidth, limited by 8.0 GT/s PCIe x2 link at 0000:00:00.0 (capable of
31.506 Gb/s with 16.0 GT/s PCIe x2 link)
[   13.950616][ T1101] __pci_bus_size_bridges: before pbus_size_mem.
list empty? 1
[   13.950784][ T1101] pbus_size_mem: 2. list empty? 1
[   13.950886][ T1101] pbus_size_mem: 1 list empty? 0
[   13.950982][ T1101] pbus_size_mem: 3. list empty? 0
[   13.951082][ T1101] pbus_size_mem: 4. list empty? 0
[   13.951185][ T1101] pcieport 0000:00:00.0: bridge window [mem
0x00100000-0x001fffff] to [bus 01-ff] add_size 100000 add_align 100000
[   13.951448][ T1101] __pci_bus_size_bridges: after pbus_size_mem. list
empty? 0
[   13.951643][ T1101] pci_assign_unassigned_bus_resources: before
__pci_bus_assign_resources -> list empty? 0
[   13.951924][ T1101] pcieport 0000:00:00.0: bridge window [mem
0x40000000-0x401fffff]: assigned
[   13.952248][ T1101] pci_assign_unassigned_bus_resources: after
__pci_bus_assign_resources -> list empty? 0
[   13.952634][ T1101] ------------[ cut here ]------------
[   13.952818][ T1101] kernel BUG at drivers/pci/setup-bus.c:2514!
[   13.953045][ T1101] Internal error: Oops - BUG: 00000000f2000800 [#1]
 SMP
...
[   13.976086][ T1101] Call trace:
[   13.976206][ T1101]  pci_assign_unassigned_bus_resources+0x110/0x114 (P)
[   13.976462][ T1101]  pci_rescan_bus+0x28/0x48
[   13.976628][ T1101]  exynos_pcie_rc_poweron

Working case:

[   13.786961][ T1120] pci 0000:01:00.0: [144d:a5a5] type 00 class
0x000000 PCIe Endpoint
[   13.787136][ T1120] pci 0000:01:00.0: BAR 0 [mem
0x00000000-0x000fffff 64bit]
[   13.787280][ T1120] pci 0000:01:00.0: ROM [mem 0x00000000-0x0000ffff
pref]
[   13.787541][ T1120] pci 0000:01:00.0: enabling Extended Tags
[   13.787808][ T1120] pci 0000:01:00.0: PME# supported from D0 D3hot D3cold
[   13.787988][ T1120] pci 0000:01:00.0: 15.752 Gb/s available PCIe
bandwidth, limited by 8.0 GT/s PCIe x2 link at 0000:00:00.0 (capable of
31.506 Gb/s with 16.0 GT/s PCIe x2 link)
[   13.795279][ T1120] __pci_bus_size_bridges: before pbus_size_mem.
list empty? 1
[   13.795408][ T1120] pbus_size_mem: 2. list empty? 1
[   13.795495][ T1120] pbus_size_mem: 2. list empty? 1
[   13.795577][ T1120] __pci_bus_size_bridges: after pbus_size_mem. list
empty? 1
[   13.795692][ T1120] pci_assign_unassigned_bus_resources: before
__pci_bus_assign_resources -> list empty? 1
[   13.795849][ T1120] pcieport 0000:00:00.0: bridge window [mem
0x40000000-0x401fffff]: assigned
[   13.796072][ T1120] pci_assign_unassigned_bus_resources: after
__pci_bus_assign_resources -> list empty? 1
[   13.796662][ T1120] cpif: s5100_poweron_pcie: DBG: MSI sfr not set
up, yet(s5100_pdev is NULL)
[   13.796666][ T1120] cpif: register_pcie: s51xx_pcie_init start

Any hints are welcomed. Thanks,
ta

[1]
https://web.git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/pci/setup-bus.c?h=v6.15-rc5#n2500

Ilpo Järvinen

author

2025-05-06 15:54 UTC #38 lore raw

Re: [PATCH 24/25] PCI: Perform reset_resource() and build fail list in sync

Quoted 31 lines · Tudor Ambarus

Hi!
Quoted 18 lines · Ilpo Järvinen
Resetting resource is problematic as it prevent attempting to allocate the resource later, unless something in between restores the resource. Similarly, if fail_head does not contain all resources that were reset, those resource cannot be restored later.
The entire reset/restore cycle adds complexity and leaving resources into reseted state causes issues to other code such as for checks done in pci_enable_resources(). Take a small step towards not resetting resources by delaying reset until the end of resource assignment and build failure list (fail_head) in sync with the reset to avoid leaving behind resources that cannot be restored (for the case where the caller provides fail_head in the first place to allow restore somewhere in the callchain, as is not all callers pass non-NULL fail_head).
The Expansion ROM check is temporarily left in place while building the failure list until the upcoming change which reworks optional resource handling.
Quoted 5 lines
Ideally, whole resource reset could be removed but doing that in a big step would make the impact non-tractable due to complexity of all related code.
Signed-off-by Ilpo Järvinen <[email protected]>
I'm hitting the BUG_ON(!list_empty(&add_list)); in pci_assign_unassigned_bus_resources() [1] with 6.15-rc5 and the the pixel6 downstream pcie driver.

Quoted 8 lines

I saw the thread where "a34d74877c66 PCI: Restore assigned resources fully after release" fixes things for some other cases, but it's not the case here.
Reverting the following patches fixes the problem:
a34d74877c66 PCI: Restore assigned resources fully after release 2499f5348431 PCI: Rework optional resource handling 96336ec70264 PCI: Perform reset_resource() and build fail list in sync

So it's confirmed that you needed to revert also this last commit 96336ec70264, not just the rework change?

Quoted 80 lines

In the working case the add_list list is empty throughout the entire body of pci_assign_unassigned_bus_resources().

[   13.951185][ T1101] pcieport 0000:00:00.0: bridge window [mem

0x00100000-0x001fffff] to [bus 01-ff] add_size 100000 add_align 100000

I've added some prints trying to describe the code path, see
https://paste.ofcode.org/Aeu2YBpLztc49ZDw3uUJmd#

Failing case:

[   13.944231][ T1101] pci 0000:01:00.0: [144d:a5a5] type 00 class
0x000000 PCIe Endpoint
[   13.944412][ T1101] pci 0000:01:00.0: BAR 0 [mem
0x00000000-0x000fffff 64bit]
[   13.944532][ T1101] pci 0000:01:00.0: ROM [mem 0x00000000-0x0000ffff
pref]
[   13.944649][ T1101] pci 0000:01:00.0: enabling Extended Tags
[   13.944844][ T1101] pci 0000:01:00.0: PME# supported from D0 D3hot D3cold
[   13.945015][ T1101] pci 0000:01:00.0: 15.752 Gb/s available PCIe
bandwidth, limited by 8.0 GT/s PCIe x2 link at 0000:00:00.0 (capable of
31.506 Gb/s with 16.0 GT/s PCIe x2 link)
[   13.950616][ T1101] __pci_bus_size_bridges: before pbus_size_mem.
list empty? 1
[   13.950784][ T1101] pbus_size_mem: 2. list empty? 1
[   13.950886][ T1101] pbus_size_mem: 1 list empty? 0
[   13.950982][ T1101] pbus_size_mem: 3. list empty? 0
[   13.951082][ T1101] pbus_size_mem: 4. list empty? 0
[   13.951185][ T1101] pcieport 0000:00:00.0: bridge window [mem
0x00100000-0x001fffff] to [bus 01-ff] add_size 100000 add_align 100000
[   13.951448][ T1101] __pci_bus_size_bridges: after pbus_size_mem. list
empty? 0
[   13.951643][ T1101] pci_assign_unassigned_bus_resources: before
__pci_bus_assign_resources -> list empty? 0
[   13.951924][ T1101] pcieport 0000:00:00.0: bridge window [mem
0x40000000-0x401fffff]: assigned
[   13.952248][ T1101] pci_assign_unassigned_bus_resources: after
__pci_bus_assign_resources -> list empty? 0
[   13.952634][ T1101] ------------[ cut here ]------------
[   13.952818][ T1101] kernel BUG at drivers/pci/setup-bus.c:2514!
[   13.953045][ T1101] Internal error: Oops - BUG: 00000000f2000800 [#1]
 SMP
...
[   13.976086][ T1101] Call trace:
[   13.976206][ T1101]  pci_assign_unassigned_bus_resources+0x110/0x114 (P)
[   13.976462][ T1101]  pci_rescan_bus+0x28/0x48
[   13.976628][ T1101]  exynos_pcie_rc_poweron

Working case:

[   13.786961][ T1120] pci 0000:01:00.0: [144d:a5a5] type 00 class
0x000000 PCIe Endpoint
[   13.787136][ T1120] pci 0000:01:00.0: BAR 0 [mem
0x00000000-0x000fffff 64bit]
[   13.787280][ T1120] pci 0000:01:00.0: ROM [mem 0x00000000-0x0000ffff
pref]
[   13.787541][ T1120] pci 0000:01:00.0: enabling Extended Tags
[   13.787808][ T1120] pci 0000:01:00.0: PME# supported from D0 D3hot D3cold
[   13.787988][ T1120] pci 0000:01:00.0: 15.752 Gb/s available PCIe
bandwidth, limited by 8.0 GT/s PCIe x2 link at 0000:00:00.0 (capable of
31.506 Gb/s with 16.0 GT/s PCIe x2 link)
[   13.795279][ T1120] __pci_bus_size_bridges: before pbus_size_mem.
list empty? 1
[   13.795408][ T1120] pbus_size_mem: 2. list empty? 1
[   13.795495][ T1120] pbus_size_mem: 2. list empty? 1
[   13.795577][ T1120] __pci_bus_size_bridges: after pbus_size_mem. list
empty? 1
[   13.795692][ T1120] pci_assign_unassigned_bus_resources: before
__pci_bus_assign_resources -> list empty? 1
[   13.795849][ T1120] pcieport 0000:00:00.0: bridge window [mem
0x40000000-0x401fffff]: assigned
[   13.796072][ T1120] pci_assign_unassigned_bus_resources: after
__pci_bus_assign_resources -> list empty? 1
[   13.796662][ T1120] cpif: s5100_poweron_pcie: DBG: MSI sfr not set
up, yet(s5100_pdev is NULL)
[   13.796666][ T1120] cpif: register_pcie: s51xx_pcie_init start

Quoted 2 lines

Any hints are welcomed. Thanks,
ta

Hi and thanks for the report.

The interesting part occurs inside reassign_resources_sorted() where most items are eliminated from realloc_head by the list_del().

My guess is that somehow, the change in 96336ec70264 from !res->flags to the more complicated check somehow causes this. If the new check doesn't match and subsequently, no match is found from the head list, the loop will do continue and not remove the entry from realloc_head.

But it's hard to confirm without knowing what that resources realloc_head contains. Perhaps if you print the resources that are processed around that part of the code in reassign_resources_sorted(), comparing the log from the reverted code with the non-working case might help to understand what is different there and why. To understand better what is in the head list, it would be also useful to know from which device the resources were added into the head list in pdev_sort_resources().

In any case, that BUG_ON() seems a bit drastic action for what might be just a single resource allocation failure so it should be downgraded to:

if (WARN_ON(!list_empty(&add_list))

	free_list(&add_list);

... or WARN_ON_ONCE().

Signature · 1 line

i.

Tudor Ambarus

2025-05-28 11:22 UTC #39 lore raw

Re: [PATCH 24/25] PCI: Perform reset_resource() and build fail list in sync

Quoted 37 lines · Ilpo Järvinen

Quoted 35 lines · Tudor Ambarus
Hi!
Quoted 18 lines · Ilpo Järvinen
Resetting resource is problematic as it prevent attempting to allocate the resource later, unless something in between restores the resource. Similarly, if fail_head does not contain all resources that were reset, those resource cannot be restored later.
The entire reset/restore cycle adds complexity and leaving resources into reseted state causes issues to other code such as for checks done in pci_enable_resources(). Take a small step towards not resetting resources by delaying reset until the end of resource assignment and build failure list (fail_head) in sync with the reset to avoid leaving behind resources that cannot be restored (for the case where the caller provides fail_head in the first place to allow restore somewhere in the callchain, as is not all callers pass non-NULL fail_head).
The Expansion ROM check is temporarily left in place while building the failure list until the upcoming change which reworks optional resource handling.
Quoted 5 lines
Ideally, whole resource reset could be removed but doing that in a big step would make the impact non-tractable due to complexity of all related code.
Signed-off-by Ilpo Järvinen <[email protected]>
I'm hitting the BUG_ON(!list_empty(&add_list)); in pci_assign_unassigned_bus_resources() [1] with 6.15-rc5 and the the pixel6 downstream pcie driver.
I saw the thread where "a34d74877c66 PCI: Restore assigned resources fully after release" fixes things for some other cases, but it's not the case here.

Quoted 7 lines

Quoted 4 lines
Reverting the following patches fixes the problem:
a34d74877c66 PCI: Restore assigned resources fully after release 2499f5348431 PCI: Rework optional resource handling 96336ec70264 PCI: Perform reset_resource() and build fail list in sync
So it's confirmed that you needed to revert also this last commit 96336ec70264, not just the rework change?

I needed to revert 96336ec70264 as well otherwise the build fails.

Quoted 81 lines

In the working case the add_list list is empty throughout the entire body of pci_assign_unassigned_bus_resources().

[   13.951185][ T1101] pcieport 0000:00:00.0: bridge window [mem

0x00100000-0x001fffff] to [bus 01-ff] add_size 100000 add_align 100000

I've added some prints trying to describe the code path, see
https://paste.ofcode.org/Aeu2YBpLztc49ZDw3uUJmd#

Failing case:

[   13.944231][ T1101] pci 0000:01:00.0: [144d:a5a5] type 00 class
0x000000 PCIe Endpoint
[   13.944412][ T1101] pci 0000:01:00.0: BAR 0 [mem
0x00000000-0x000fffff 64bit]
[   13.944532][ T1101] pci 0000:01:00.0: ROM [mem 0x00000000-0x0000ffff
pref]
[   13.944649][ T1101] pci 0000:01:00.0: enabling Extended Tags
[   13.944844][ T1101] pci 0000:01:00.0: PME# supported from D0 D3hot D3cold
[   13.945015][ T1101] pci 0000:01:00.0: 15.752 Gb/s available PCIe
bandwidth, limited by 8.0 GT/s PCIe x2 link at 0000:00:00.0 (capable of
31.506 Gb/s with 16.0 GT/s PCIe x2 link)
[   13.950616][ T1101] __pci_bus_size_bridges: before pbus_size_mem.
list empty? 1
[   13.950784][ T1101] pbus_size_mem: 2. list empty? 1
[   13.950886][ T1101] pbus_size_mem: 1 list empty? 0
[   13.950982][ T1101] pbus_size_mem: 3. list empty? 0
[   13.951082][ T1101] pbus_size_mem: 4. list empty? 0
[   13.951185][ T1101] pcieport 0000:00:00.0: bridge window [mem
0x00100000-0x001fffff] to [bus 01-ff] add_size 100000 add_align 100000
[   13.951448][ T1101] __pci_bus_size_bridges: after pbus_size_mem. list
empty? 0
[   13.951643][ T1101] pci_assign_unassigned_bus_resources: before
__pci_bus_assign_resources -> list empty? 0
[   13.951924][ T1101] pcieport 0000:00:00.0: bridge window [mem
0x40000000-0x401fffff]: assigned
[   13.952248][ T1101] pci_assign_unassigned_bus_resources: after
__pci_bus_assign_resources -> list empty? 0
[   13.952634][ T1101] ------------[ cut here ]------------
[   13.952818][ T1101] kernel BUG at drivers/pci/setup-bus.c:2514!
[   13.953045][ T1101] Internal error: Oops - BUG: 00000000f2000800 [#1]
 SMP
...
[   13.976086][ T1101] Call trace:
[   13.976206][ T1101]  pci_assign_unassigned_bus_resources+0x110/0x114 (P)
[   13.976462][ T1101]  pci_rescan_bus+0x28/0x48
[   13.976628][ T1101]  exynos_pcie_rc_poweron

Working case:

[   13.786961][ T1120] pci 0000:01:00.0: [144d:a5a5] type 00 class
0x000000 PCIe Endpoint
[   13.787136][ T1120] pci 0000:01:00.0: BAR 0 [mem
0x00000000-0x000fffff 64bit]
[   13.787280][ T1120] pci 0000:01:00.0: ROM [mem 0x00000000-0x0000ffff
pref]
[   13.787541][ T1120] pci 0000:01:00.0: enabling Extended Tags
[   13.787808][ T1120] pci 0000:01:00.0: PME# supported from D0 D3hot D3cold
[   13.787988][ T1120] pci 0000:01:00.0: 15.752 Gb/s available PCIe
bandwidth, limited by 8.0 GT/s PCIe x2 link at 0000:00:00.0 (capable of
31.506 Gb/s with 16.0 GT/s PCIe x2 link)
[   13.795279][ T1120] __pci_bus_size_bridges: before pbus_size_mem.
list empty? 1
[   13.795408][ T1120] pbus_size_mem: 2. list empty? 1
[   13.795495][ T1120] pbus_size_mem: 2. list empty? 1
[   13.795577][ T1120] __pci_bus_size_bridges: after pbus_size_mem. list
empty? 1
[   13.795692][ T1120] pci_assign_unassigned_bus_resources: before
__pci_bus_assign_resources -> list empty? 1
[   13.795849][ T1120] pcieport 0000:00:00.0: bridge window [mem
0x40000000-0x401fffff]: assigned
[   13.796072][ T1120] pci_assign_unassigned_bus_resources: after
__pci_bus_assign_resources -> list empty? 1
[   13.796662][ T1120] cpif: s5100_poweron_pcie: DBG: MSI sfr not set
up, yet(s5100_pdev is NULL)
[   13.796666][ T1120] cpif: register_pcie: s51xx_pcie_init start

Quoted 4 lines

Quoted 2 lines
Any hints are welcomed. Thanks,
ta
Hi and thanks for the report.

Hi! Thanks for the help. I've been out of office for the last 2 weeks, sorry for the delayed reply.

Quoted 8 lines

The interesting part occurs inside reassign_resources_sorted() where most items are eliminated from realloc_head by the list_del().
My guess is that somehow, the change in 96336ec70264 from !res->flags to the more complicated check somehow causes this. If the new check doesn't match and subsequently, no match is found from the head list, the loop will do continue and not remove the entry from realloc_head.

I added a print right there and it seems it's something else. See below.

Quoted 9 lines

But it's hard to confirm without knowing what that resources realloc_head contains. Perhaps if you print the resources that are processed around that part of the code in reassign_resources_sorted(), comparing the log from the reverted code with the non-working case might help to understand what is different there and why. To understand better what is in the head list, it would be also useful to know from which device the resources were added into the head list in pdev_sort_resources().

I added the suggested prints
(https://paste.ofcode.org/DgmZGGgS6D36nWEzmfCqMm) on top of v6.15 with the downstream PCIe pixel driver and I obtain the following. Note that all added prints contain "tudor" for differentiation.

[   15.211179][ T1107] pci 0001:01:00.0: [144d:a5a5] type 00 class
0x000000 PCIe Endpoint
[   15.212248][ T1107] pci 0001:01:00.0: BAR 0 [mem
0x00000000-0x000fffff 64bit]
[   15.212775][ T1107] pci 0001:01:00.0: ROM [mem 0x00000000-0x0000ffff
pref]
[   15.213195][ T1107] pci 0001:01:00.0: enabling Extended Tags
[   15.213720][ T1107] pci 0001:01:00.0: PME# supported from D0 D3hot
D3cold
[   15.214035][ T1107] pci 0001:01:00.0: 15.752 Gb/s available PCIe
bandwidth, limited by 8.0 GT/s PCIe x2 link at 0001:00:00.0 (capable of
31.506 Gb/s with 16.0 GT/s PCIe x2 link)
[   15.222286][ T1107] pci 0001:01:00.0: tudor: 1: pbus_size_mem: BAR 0
[mem 0x00000000-0x000fffff 64bit] list empty? 1
[   15.222813][ T1107] pci 0001:01:00.0: tudor: 1: pbus_size_mem: ROM
[mem 0x00000000-0x0000ffff pref] list empty? 1
[   15.224429][ T1107] pci 0001:01:00.0: tudor: 2: pbus_size_mem: ROM
[mem 0x00000000-0x0000ffff pref] list empty? 0
[   15.224750][ T1107] pcieport 0001:00:00.0: bridge window [mem

0x00100000-0x001fffff] to [bus 01-ff] add_size 100000 add_align 100000

[   15.225393][ T1107] tudor : pci_assign_unassigned_bus_resources:
before __pci_bus_assign_resources -> list empty? 0
[   15.225594][ T1107] pcieport 0001:00:00.0: tudor:
pdev_sort_resources: bridge window [mem 0x00100000-0x001fffff] resource
added in head list
[   15.226078][ T1107] pcieport 0001:00:00.0: bridge window [mem
0x40000000-0x401fffff]: assigned
[   15.226419][ T1107] tudor : pci_assign_unassigned_bus_resources:
after __pci_bus_assign_resources -> list empty? 0
[   15.226442][ T1107] ------------[ cut here ]------------
[   15.227587][ T1107] kernel BUG at drivers/pci/setup-bus.c:2522!
[   15.227813][ T1107] Internal error: Oops - BUG: 00000000f2000800 [#1]
 SMP
...
[   15.251570][ T1107] Call trace:
[   15.251690][ T1107]  pci_assign_unassigned_bus_resources+0x110/0x114 (P)
[   15.251945][ T1107]  pci_rescan_bus+0x28/0x48

I obtain the following output when using the same prints adapted
(https://paste.ofcode.org/37w7RnKkPaCxyNhi5yhZPbZ) and with the blamed commits reverted:
a34d74877c66 PCI: Restore assigned resources fully after release 2499f5348431 PCI: Rework optional resource handling 96336ec70264 PCI: Perform reset_resource() and build fail list in sync

[   15.200456][ T1102] pci 0000:01:00.0: [144d:a5a5] type 00 class
0x000000 PCIe Endpoint
[   15.200632][ T1102] pci 0000:01:00.0: BAR 0 [mem
0x00000000-0x000fffff 64bit]
[   15.200755][ T1102] pci 0000:01:00.0: ROM [mem 0x00000000-0x0000ffff
pref]
[   15.200876][ T1102] pci 0000:01:00.0: enabling Extended Tags
[   15.201075][ T1102] pci 0000:01:00.0: PME# supported from D0 D3hot D3cold
[   15.201254][ T1102] pci 0000:01:00.0: 15.752 Gb/s available PCIe
bandwidth, limited by 8.0 GT/s PCIe x2 link at 0000:00:00.0 (capable of
31.506 Gb/s with 16.0 GT/s PCIe x2 link)
[   15.206555][ T1102] pci 0000:01:00.0: tudor: 1: pbus_size_mem: BAR 0
[mem 0x00000000-0x000fffff 64bit] list empty? 1
[   15.206737][ T1102] pci 0000:01:00.0: tudor: 1: pbus_size_mem: ROM
[mem 0x00000000-0x0000ffff pref] list empty? 1
[   15.206901][ T1102] tudor : pci_assign_unassigned_bus_resources:
before __pci_bus_assign_resources -> list empty? 1
[   15.207072][ T1102] pcieport 0000:00:00.0: tudor:
pdev_sort_resources: bridge window [mem 0x00100000-0x002fffff] resource
added in head list
[   15.207396][ T1102] pcieport 0000:00:00.0: bridge window [mem
0x40000000-0x401fffff]: assigned
[   15.208165][ T1102] tudor : pci_assign_unassigned_bus_resources:
after __pci_bus_assign_resources -> list empty? 1
[   15.208783][ T1102] cpif: s5100_poweron_pcie: DBG: MSI sfr not set
up, yet(s5100_pdev is NULL)
[   15.208786][ T1102] cpif: register_pcie: s51xx_pcie_init start

Quoted 7 lines

In any case, that BUG_ON() seems a bit drastic action for what might be just a single resource allocation failure so it should be downgraded to:
if (WARN_ON(!list_empty(&add_list))
	free_list(&add_list);
... or WARN_ON_ONCE().

I saw your patch doing this, the phone now boots, but obviously I still see the WARN, so maybe there's still something to be fixed.

Thanks!
ta

Tudor Ambarus

2025-05-28 11:39 UTC #40 lore raw

Re: [PATCH 24/25] PCI: Perform reset_resource() and build fail list in sync

Quoted 217 lines · Tudor Ambarus

Quoted 36 lines · Ilpo Järvinen
Quoted 34 lines · Tudor Ambarus
Hi!
Quoted 18 lines · Ilpo Järvinen
Resetting resource is problematic as it prevent attempting to allocate the resource later, unless something in between restores the resource. Similarly, if fail_head does not contain all resources that were reset, those resource cannot be restored later.
The entire reset/restore cycle adds complexity and leaving resources into reseted state causes issues to other code such as for checks done in pci_enable_resources(). Take a small step towards not resetting resources by delaying reset until the end of resource assignment and build failure list (fail_head) in sync with the reset to avoid leaving behind resources that cannot be restored (for the case where the caller provides fail_head in the first place to allow restore somewhere in the callchain, as is not all callers pass non-NULL fail_head).
The Expansion ROM check is temporarily left in place while building the failure list until the upcoming change which reworks optional resource handling.
Quoted 5 lines
Ideally, whole resource reset could be removed but doing that in a big step would make the impact non-tractable due to complexity of all related code.
Signed-off-by Ilpo Järvinen <[email protected]>
I'm hitting the BUG_ON(!list_empty(&add_list)); in pci_assign_unassigned_bus_resources() [1] with 6.15-rc5 and the the pixel6 downstream pcie driver.
I saw the thread where "a34d74877c66 PCI: Restore assigned resources fully after release" fixes things for some other cases, but it's not the case here.
Quoted 6 lines
Quoted 4 lines
Reverting the following patches fixes the problem:
a34d74877c66 PCI: Restore assigned resources fully after release 2499f5348431 PCI: Rework optional resource handling 96336ec70264 PCI: Perform reset_resource() and build fail list in sync
So it's confirmed that you needed to revert also this last commit 96336ec70264, not just the rework change?
I needed to revert 96336ec70264 as well otherwise the build fails.
Quoted 80 lines
Quoted 80 lines
In the working case the add_list list is empty throughout the entire body of pci_assign_unassigned_bus_resources().
In the failing case __pci_bus_size_bridges() leaves the add_list not empty and __pci_bus_assign_resources() does not consume the list, thus the BUG_ON. The failing case contains an extra print that's not shown when reverting the blamed commits:
[   13.951185][ T1101] pcieport 0000:00:00.0: bridge window [mem
0x00100000-0x001fffff] to [bus 01-ff] add_size 100000 add_align 100000
I've added some prints trying to describe the code path, see
https://paste.ofcode.org/Aeu2YBpLztc49ZDw3uUJmd#
Failing case:
[   13.944231][ T1101] pci 0000:01:00.0: [144d:a5a5] type 00 class
0x000000 PCIe Endpoint
[   13.944412][ T1101] pci 0000:01:00.0: BAR 0 [mem
0x00000000-0x000fffff 64bit]
[   13.944532][ T1101] pci 0000:01:00.0: ROM [mem 0x00000000-0x0000ffff
pref]
[   13.944649][ T1101] pci 0000:01:00.0: enabling Extended Tags
[   13.944844][ T1101] pci 0000:01:00.0: PME# supported from D0 D3hot D3cold
[   13.945015][ T1101] pci 0000:01:00.0: 15.752 Gb/s available PCIe
bandwidth, limited by 8.0 GT/s PCIe x2 link at 0000:00:00.0 (capable of
31.506 Gb/s with 16.0 GT/s PCIe x2 link)
[   13.950616][ T1101] __pci_bus_size_bridges: before pbus_size_mem.
list empty? 1
[   13.950784][ T1101] pbus_size_mem: 2. list empty? 1
[   13.950886][ T1101] pbus_size_mem: 1 list empty? 0
[   13.950982][ T1101] pbus_size_mem: 3. list empty? 0
[   13.951082][ T1101] pbus_size_mem: 4. list empty? 0
[   13.951185][ T1101] pcieport 0000:00:00.0: bridge window [mem
0x00100000-0x001fffff] to [bus 01-ff] add_size 100000 add_align 100000
[   13.951448][ T1101] __pci_bus_size_bridges: after pbus_size_mem. list
empty? 0
[   13.951643][ T1101] pci_assign_unassigned_bus_resources: before
__pci_bus_assign_resources -> list empty? 0
[   13.951924][ T1101] pcieport 0000:00:00.0: bridge window [mem
0x40000000-0x401fffff]: assigned
[   13.952248][ T1101] pci_assign_unassigned_bus_resources: after
__pci_bus_assign_resources -> list empty? 0
[   13.952634][ T1101] ------------[ cut here ]------------
[   13.952818][ T1101] kernel BUG at drivers/pci/setup-bus.c:2514!
[   13.953045][ T1101] Internal error: Oops - BUG: 00000000f2000800 [#1]
 SMP
...
[   13.976086][ T1101] Call trace:
[   13.976206][ T1101]  pci_assign_unassigned_bus_resources+0x110/0x114 (P)
[   13.976462][ T1101]  pci_rescan_bus+0x28/0x48
[   13.976628][ T1101]  exynos_pcie_rc_poweron
Working case:
[   13.786961][ T1120] pci 0000:01:00.0: [144d:a5a5] type 00 class
0x000000 PCIe Endpoint
[   13.787136][ T1120] pci 0000:01:00.0: BAR 0 [mem
0x00000000-0x000fffff 64bit]
[   13.787280][ T1120] pci 0000:01:00.0: ROM [mem 0x00000000-0x0000ffff
pref]
[   13.787541][ T1120] pci 0000:01:00.0: enabling Extended Tags
[   13.787808][ T1120] pci 0000:01:00.0: PME# supported from D0 D3hot D3cold
[   13.787988][ T1120] pci 0000:01:00.0: 15.752 Gb/s available PCIe
bandwidth, limited by 8.0 GT/s PCIe x2 link at 0000:00:00.0 (capable of
31.506 Gb/s with 16.0 GT/s PCIe x2 link)
[   13.795279][ T1120] __pci_bus_size_bridges: before pbus_size_mem.
list empty? 1
[   13.795408][ T1120] pbus_size_mem: 2. list empty? 1
[   13.795495][ T1120] pbus_size_mem: 2. list empty? 1
[   13.795577][ T1120] __pci_bus_size_bridges: after pbus_size_mem. list
empty? 1
[   13.795692][ T1120] pci_assign_unassigned_bus_resources: before
__pci_bus_assign_resources -> list empty? 1
[   13.795849][ T1120] pcieport 0000:00:00.0: bridge window [mem
0x40000000-0x401fffff]: assigned
[   13.796072][ T1120] pci_assign_unassigned_bus_resources: after
__pci_bus_assign_resources -> list empty? 1
[   13.796662][ T1120] cpif: s5100_poweron_pcie: DBG: MSI sfr not set
up, yet(s5100_pdev is NULL)
[   13.796666][ T1120] cpif: register_pcie: s51xx_pcie_init start
Quoted 3 lines
Quoted 2 lines
Any hints are welcomed. Thanks,
ta
Hi and thanks for the report.
Hi! Thanks for the help. I've been out of office for the last 2 weeks, sorry for the delayed reply.
Quoted 7 lines
The interesting part occurs inside reassign_resources_sorted() where most items are eliminated from realloc_head by the list_del().
My guess is that somehow, the change in 96336ec70264 from !res->flags to the more complicated check somehow causes this. If the new check doesn't match and subsequently, no match is found from the head list, the loop will do continue and not remove the entry from realloc_head.
I added a print right there and it seems it's something else. See below.
Quoted 8 lines
But it's hard to confirm without knowing what that resources realloc_head contains. Perhaps if you print the resources that are processed around that part of the code in reassign_resources_sorted(), comparing the log from the reverted code with the non-working case might help to understand what is different there and why. To understand better what is in the head list, it would be also useful to know from which device the resources were added into the head list in pdev_sort_resources().
I added the suggested prints
(https://paste.ofcode.org/DgmZGGgS6D36nWEzmfCqMm) on top of v6.15 with the downstream PCIe pixel driver and I obtain the following. Note that all added prints contain "tudor" for differentiation.
[   15.211179][ T1107] pci 0001:01:00.0: [144d:a5a5] type 00 class
0x000000 PCIe Endpoint
[   15.212248][ T1107] pci 0001:01:00.0: BAR 0 [mem
0x00000000-0x000fffff 64bit]
[   15.212775][ T1107] pci 0001:01:00.0: ROM [mem 0x00000000-0x0000ffff
pref]
[   15.213195][ T1107] pci 0001:01:00.0: enabling Extended Tags
[   15.213720][ T1107] pci 0001:01:00.0: PME# supported from D0 D3hot
D3cold
[   15.214035][ T1107] pci 0001:01:00.0: 15.752 Gb/s available PCIe
bandwidth, limited by 8.0 GT/s PCIe x2 link at 0001:00:00.0 (capable of
31.506 Gb/s with 16.0 GT/s PCIe x2 link)
[   15.222286][ T1107] pci 0001:01:00.0: tudor: 1: pbus_size_mem: BAR 0
[mem 0x00000000-0x000fffff 64bit] list empty? 1
[   15.222813][ T1107] pci 0001:01:00.0: tudor: 1: pbus_size_mem: ROM
[mem 0x00000000-0x0000ffff pref] list empty? 1
[   15.224429][ T1107] pci 0001:01:00.0: tudor: 2: pbus_size_mem: ROM
[mem 0x00000000-0x0000ffff pref] list empty? 0
[   15.224750][ T1107] pcieport 0001:00:00.0: bridge window [mem
0x00100000-0x001fffff] to [bus 01-ff] add_size 100000 add_align 100000
[   15.225393][ T1107] tudor : pci_assign_unassigned_bus_resources:
before __pci_bus_assign_resources -> list empty? 0
[   15.225594][ T1107] pcieport 0001:00:00.0: tudor:
pdev_sort_resources: bridge window [mem 0x00100000-0x001fffff] resource
added in head list
[   15.226078][ T1107] pcieport 0001:00:00.0: bridge window [mem
0x40000000-0x401fffff]: assigned
[   15.226419][ T1107] tudor : pci_assign_unassigned_bus_resources:
after __pci_bus_assign_resources -> list empty? 0
[   15.226442][ T1107] ------------[ cut here ]------------
[   15.227587][ T1107] kernel BUG at drivers/pci/setup-bus.c:2522!
[   15.227813][ T1107] Internal error: Oops - BUG: 00000000f2000800 [#1]
 SMP
...
[   15.251570][ T1107] Call trace:
[   15.251690][ T1107]  pci_assign_unassigned_bus_resources+0x110/0x114 (P)
[   15.251945][ T1107]  pci_rescan_bus+0x28/0x48
I obtain the following output when using the same prints adapted
(https://paste.ofcode.org/37w7RnKkPaCxyNhi5yhZPbZ) and with the blamed commits reverted:
a34d74877c66 PCI: Restore assigned resources fully after release 2499f5348431 PCI: Rework optional resource handling 96336ec70264 PCI: Perform reset_resource() and build fail list in sync
[   15.200456][ T1102] pci 0000:01:00.0: [144d:a5a5] type 00 class
0x000000 PCIe Endpoint
[   15.200632][ T1102] pci 0000:01:00.0: BAR 0 [mem
0x00000000-0x000fffff 64bit]
[   15.200755][ T1102] pci 0000:01:00.0: ROM [mem 0x00000000-0x0000ffff
pref]
[   15.200876][ T1102] pci 0000:01:00.0: enabling Extended Tags
[   15.201075][ T1102] pci 0000:01:00.0: PME# supported from D0 D3hot D3cold
[   15.201254][ T1102] pci 0000:01:00.0: 15.752 Gb/s available PCIe
bandwidth, limited by 8.0 GT/s PCIe x2 link at 0000:00:00.0 (capable of
31.506 Gb/s with 16.0 GT/s PCIe x2 link)
[   15.206555][ T1102] pci 0000:01:00.0: tudor: 1: pbus_size_mem: BAR 0
[mem 0x00000000-0x000fffff 64bit] list empty? 1
[   15.206737][ T1102] pci 0000:01:00.0: tudor: 1: pbus_size_mem: ROM
[mem 0x00000000-0x0000ffff pref] list empty? 1
[   15.206901][ T1102] tudor : pci_assign_unassigned_bus_resources:
before __pci_bus_assign_resources -> list empty? 1
[   15.207072][ T1102] pcieport 0000:00:00.0: tudor:
pdev_sort_resources: bridge window [mem 0x00100000-0x002fffff] resource

Quoted 8 lines

added in head list

[   15.207396][ T1102] pcieport 0000:00:00.0: bridge window [mem
0x40000000-0x401fffff]: assigned
[   15.208165][ T1102] tudor : pci_assign_unassigned_bus_resources:
after __pci_bus_assign_resources -> list empty? 1
[   15.208783][ T1102] cpif: s5100_poweron_pcie: DBG: MSI sfr not set
up, yet(s5100_pdev is NULL)
[   15.208786][ T1102] cpif: register_pcie: s51xx_pcie_init start

I see my email client split the lines for the prints making the output very hard to read. Added the output here too:
https://paste.ofcode.org/AEfjASQW8Z2jbMak5VkmpJ

Tudor Ambarus

2025-05-28 13:10 UTC #41 lore raw

Re: [PATCH 24/25] PCI: Perform reset_resource() and build fail list in sync

Quoted 229 lines · Tudor Ambarus

Quoted 217 lines · Tudor Ambarus
Quoted 36 lines · Ilpo Järvinen
Quoted 34 lines · Tudor Ambarus
Hi!
Quoted 18 lines · Ilpo Järvinen
Resetting resource is problematic as it prevent attempting to allocate the resource later, unless something in between restores the resource. Similarly, if fail_head does not contain all resources that were reset, those resource cannot be restored later.
The entire reset/restore cycle adds complexity and leaving resources into reseted state causes issues to other code such as for checks done in pci_enable_resources(). Take a small step towards not resetting resources by delaying reset until the end of resource assignment and build failure list (fail_head) in sync with the reset to avoid leaving behind resources that cannot be restored (for the case where the caller provides fail_head in the first place to allow restore somewhere in the callchain, as is not all callers pass non-NULL fail_head).
The Expansion ROM check is temporarily left in place while building the failure list until the upcoming change which reworks optional resource handling.
Quoted 5 lines
Ideally, whole resource reset could be removed but doing that in a big step would make the impact non-tractable due to complexity of all related code.
Signed-off-by Ilpo Järvinen <[email protected]>
I'm hitting the BUG_ON(!list_empty(&add_list)); in pci_assign_unassigned_bus_resources() [1] with 6.15-rc5 and the the pixel6 downstream pcie driver.
I saw the thread where "a34d74877c66 PCI: Restore assigned resources fully after release" fixes things for some other cases, but it's not the case here.
Quoted 6 lines
Quoted 4 lines
Reverting the following patches fixes the problem:
a34d74877c66 PCI: Restore assigned resources fully after release 2499f5348431 PCI: Rework optional resource handling 96336ec70264 PCI: Perform reset_resource() and build fail list in sync
So it's confirmed that you needed to revert also this last commit 96336ec70264, not just the rework change?
I needed to revert 96336ec70264 as well otherwise the build fails.
Quoted 80 lines
Quoted 80 lines
In the working case the add_list list is empty throughout the entire body of pci_assign_unassigned_bus_resources().
In the failing case __pci_bus_size_bridges() leaves the add_list not empty and __pci_bus_assign_resources() does not consume the list, thus the BUG_ON. The failing case contains an extra print that's not shown when reverting the blamed commits:
[   13.951185][ T1101] pcieport 0000:00:00.0: bridge window [mem
0x00100000-0x001fffff] to [bus 01-ff] add_size 100000 add_align 100000
I've added some prints trying to describe the code path, see
https://paste.ofcode.org/Aeu2YBpLztc49ZDw3uUJmd#
Failing case:
[   13.944231][ T1101] pci 0000:01:00.0: [144d:a5a5] type 00 class
0x000000 PCIe Endpoint
[   13.944412][ T1101] pci 0000:01:00.0: BAR 0 [mem
0x00000000-0x000fffff 64bit]
[   13.944532][ T1101] pci 0000:01:00.0: ROM [mem 0x00000000-0x0000ffff
pref]
[   13.944649][ T1101] pci 0000:01:00.0: enabling Extended Tags
[   13.944844][ T1101] pci 0000:01:00.0: PME# supported from D0 D3hot D3cold
[   13.945015][ T1101] pci 0000:01:00.0: 15.752 Gb/s available PCIe
bandwidth, limited by 8.0 GT/s PCIe x2 link at 0000:00:00.0 (capable of
31.506 Gb/s with 16.0 GT/s PCIe x2 link)
[   13.950616][ T1101] __pci_bus_size_bridges: before pbus_size_mem.
list empty? 1
[   13.950784][ T1101] pbus_size_mem: 2. list empty? 1
[   13.950886][ T1101] pbus_size_mem: 1 list empty? 0
[   13.950982][ T1101] pbus_size_mem: 3. list empty? 0
[   13.951082][ T1101] pbus_size_mem: 4. list empty? 0
[   13.951185][ T1101] pcieport 0000:00:00.0: bridge window [mem
0x00100000-0x001fffff] to [bus 01-ff] add_size 100000 add_align 100000
[   13.951448][ T1101] __pci_bus_size_bridges: after pbus_size_mem. list
empty? 0
[   13.951643][ T1101] pci_assign_unassigned_bus_resources: before
__pci_bus_assign_resources -> list empty? 0
[   13.951924][ T1101] pcieport 0000:00:00.0: bridge window [mem
0x40000000-0x401fffff]: assigned
[   13.952248][ T1101] pci_assign_unassigned_bus_resources: after
__pci_bus_assign_resources -> list empty? 0
[   13.952634][ T1101] ------------[ cut here ]------------
[   13.952818][ T1101] kernel BUG at drivers/pci/setup-bus.c:2514!
[   13.953045][ T1101] Internal error: Oops - BUG: 00000000f2000800 [#1]
 SMP
...
[   13.976086][ T1101] Call trace:
[   13.976206][ T1101]  pci_assign_unassigned_bus_resources+0x110/0x114 (P)
[   13.976462][ T1101]  pci_rescan_bus+0x28/0x48
[   13.976628][ T1101]  exynos_pcie_rc_poweron
Working case:
[   13.786961][ T1120] pci 0000:01:00.0: [144d:a5a5] type 00 class
0x000000 PCIe Endpoint
[   13.787136][ T1120] pci 0000:01:00.0: BAR 0 [mem
0x00000000-0x000fffff 64bit]
[   13.787280][ T1120] pci 0000:01:00.0: ROM [mem 0x00000000-0x0000ffff
pref]
[   13.787541][ T1120] pci 0000:01:00.0: enabling Extended Tags
[   13.787808][ T1120] pci 0000:01:00.0: PME# supported from D0 D3hot D3cold
[   13.787988][ T1120] pci 0000:01:00.0: 15.752 Gb/s available PCIe
bandwidth, limited by 8.0 GT/s PCIe x2 link at 0000:00:00.0 (capable of
31.506 Gb/s with 16.0 GT/s PCIe x2 link)
[   13.795279][ T1120] __pci_bus_size_bridges: before pbus_size_mem.
list empty? 1
[   13.795408][ T1120] pbus_size_mem: 2. list empty? 1
[   13.795495][ T1120] pbus_size_mem: 2. list empty? 1
[   13.795577][ T1120] __pci_bus_size_bridges: after pbus_size_mem. list
empty? 1
[   13.795692][ T1120] pci_assign_unassigned_bus_resources: before
__pci_bus_assign_resources -> list empty? 1
[   13.795849][ T1120] pcieport 0000:00:00.0: bridge window [mem
0x40000000-0x401fffff]: assigned
[   13.796072][ T1120] pci_assign_unassigned_bus_resources: after
__pci_bus_assign_resources -> list empty? 1
[   13.796662][ T1120] cpif: s5100_poweron_pcie: DBG: MSI sfr not set
up, yet(s5100_pdev is NULL)
[   13.796666][ T1120] cpif: register_pcie: s51xx_pcie_init start
Quoted 3 lines
Quoted 2 lines
Any hints are welcomed. Thanks,
ta
Hi and thanks for the report.
Hi! Thanks for the help. I've been out of office for the last 2 weeks, sorry for the delayed reply.
Quoted 7 lines
The interesting part occurs inside reassign_resources_sorted() where most items are eliminated from realloc_head by the list_del().
My guess is that somehow, the change in 96336ec70264 from !res->flags to the more complicated check somehow causes this. If the new check doesn't match and subsequently, no match is found from the head list, the loop will do continue and not remove the entry from realloc_head.
I added a print right there and it seems it's something else. See below.
Quoted 8 lines
But it's hard to confirm without knowing what that resources realloc_head contains. Perhaps if you print the resources that are processed around that part of the code in reassign_resources_sorted(), comparing the log from the reverted code with the non-working case might help to understand what is different there and why. To understand better what is in the head list, it would be also useful to know from which device the resources were added into the head list in pdev_sort_resources().
I added the suggested prints
(https://paste.ofcode.org/DgmZGGgS6D36nWEzmfCqMm) on top of v6.15 with the downstream PCIe pixel driver and I obtain the following. Note that all added prints contain "tudor" for differentiation.
[   15.211179][ T1107] pci 0001:01:00.0: [144d:a5a5] type 00 class
0x000000 PCIe Endpoint
[   15.212248][ T1107] pci 0001:01:00.0: BAR 0 [mem
0x00000000-0x000fffff 64bit]
[   15.212775][ T1107] pci 0001:01:00.0: ROM [mem 0x00000000-0x0000ffff
pref]
[   15.213195][ T1107] pci 0001:01:00.0: enabling Extended Tags
[   15.213720][ T1107] pci 0001:01:00.0: PME# supported from D0 D3hot
D3cold
[   15.214035][ T1107] pci 0001:01:00.0: 15.752 Gb/s available PCIe
bandwidth, limited by 8.0 GT/s PCIe x2 link at 0001:00:00.0 (capable of
31.506 Gb/s with 16.0 GT/s PCIe x2 link)
[   15.222286][ T1107] pci 0001:01:00.0: tudor: 1: pbus_size_mem: BAR 0
[mem 0x00000000-0x000fffff 64bit] list empty? 1
[   15.222813][ T1107] pci 0001:01:00.0: tudor: 1: pbus_size_mem: ROM
[mem 0x00000000-0x0000ffff pref] list empty? 1
[   15.224429][ T1107] pci 0001:01:00.0: tudor: 2: pbus_size_mem: ROM
[mem 0x00000000-0x0000ffff pref] list empty? 0
[   15.224750][ T1107] pcieport 0001:00:00.0: bridge window [mem
0x00100000-0x001fffff] to [bus 01-ff] add_size 100000 add_align 100000
[   15.225393][ T1107] tudor : pci_assign_unassigned_bus_resources:
before __pci_bus_assign_resources -> list empty? 0
[   15.225594][ T1107] pcieport 0001:00:00.0: tudor:
pdev_sort_resources: bridge window [mem 0x00100000-0x001fffff] resource
added in head list
[   15.226078][ T1107] pcieport 0001:00:00.0: bridge window [mem
0x40000000-0x401fffff]: assigned
[   15.226419][ T1107] tudor : pci_assign_unassigned_bus_resources:
after __pci_bus_assign_resources -> list empty? 0
[   15.226442][ T1107] ------------[ cut here ]------------
[   15.227587][ T1107] kernel BUG at drivers/pci/setup-bus.c:2522!
[   15.227813][ T1107] Internal error: Oops - BUG: 00000000f2000800 [#1]
 SMP
...
[   15.251570][ T1107] Call trace:
[   15.251690][ T1107]  pci_assign_unassigned_bus_resources+0x110/0x114 (P)
[   15.251945][ T1107]  pci_rescan_bus+0x28/0x48
I obtain the following output when using the same prints adapted
(https://paste.ofcode.org/37w7RnKkPaCxyNhi5yhZPbZ) and with the blamed commits reverted:
a34d74877c66 PCI: Restore assigned resources fully after release 2499f5348431 PCI: Rework optional resource handling 96336ec70264 PCI: Perform reset_resource() and build fail list in sync
[   15.200456][ T1102] pci 0000:01:00.0: [144d:a5a5] type 00 class
0x000000 PCIe Endpoint
[   15.200632][ T1102] pci 0000:01:00.0: BAR 0 [mem
0x00000000-0x000fffff 64bit]
[   15.200755][ T1102] pci 0000:01:00.0: ROM [mem 0x00000000-0x0000ffff
pref]
[   15.200876][ T1102] pci 0000:01:00.0: enabling Extended Tags
[   15.201075][ T1102] pci 0000:01:00.0: PME# supported from D0 D3hot D3cold
[   15.201254][ T1102] pci 0000:01:00.0: 15.752 Gb/s available PCIe
bandwidth, limited by 8.0 GT/s PCIe x2 link at 0000:00:00.0 (capable of
31.506 Gb/s with 16.0 GT/s PCIe x2 link)
[   15.206555][ T1102] pci 0000:01:00.0: tudor: 1: pbus_size_mem: BAR 0
[mem 0x00000000-0x000fffff 64bit] list empty? 1
[   15.206737][ T1102] pci 0000:01:00.0: tudor: 1: pbus_size_mem: ROM
[mem 0x00000000-0x0000ffff pref] list empty? 1
[   15.206901][ T1102] tudor : pci_assign_unassigned_bus_resources:
before __pci_bus_assign_resources -> list empty? 1
[   15.207072][ T1102] pcieport 0000:00:00.0: tudor:
pdev_sort_resources: bridge window [mem 0x00100000-0x002fffff] resource
Quoted 8 lines
added in head list
[   15.207396][ T1102] pcieport 0000:00:00.0: bridge window [mem
0x40000000-0x401fffff]: assigned
[   15.208165][ T1102] tudor : pci_assign_unassigned_bus_resources:
after __pci_bus_assign_resources -> list empty? 1
[   15.208783][ T1102] cpif: s5100_poweron_pcie: DBG: MSI sfr not set
up, yet(s5100_pdev is NULL)
[   15.208786][ T1102] cpif: register_pcie: s51xx_pcie_init start

Quoted 3 lines

I see my email client split the lines for the prints making the output very hard to read. Added the output here too:
https://paste.ofcode.org/AEfjASQW8Z2jbMak5VkmpJ

With the following change things get back to how they were before 2499f5348431:

drivers/pci/setup-bus.c +0 -0

index 5247370010aa..1589dd8afa69 100644
@@ -1214,9 +1214,10 @@ static int pbus_size_mem(struct pci_bus *bus,

unsigned long mask,
__func__, r_name, r, list_empty(realloc_head));

/* Put SRIOV requested res to the optional list */
- if (realloc_head && pci_resource_is_optional(dev, i)) {
+ if (realloc_head && pci_resource_is_iov(i)) {
add_align = max(pci_resource_alignment(dev, r), add_align);
- add_to_list(realloc_head, dev, r, 0, 0 /* Don't care */);
+ resource_set_size(r, 0);
+ add_to_list(realloc_head, dev, r, r_size, 0 /* Don't care */);
children_add_size += r_size;
pci_info(dev, "tudor: 2: %s: %s %pR list empty? %d\n",
__func__, r_name, r, list_empty(realloc_head));

Signature · 1 line

2.49.0.1238.gf8c92423fb-goog

Ilpo Järvinen

author

2025-05-30 06:55 UTC #42 lore raw

Re: [PATCH 24/25] PCI: Perform reset_resource() and build fail list in sync

Quoted 251 lines · Tudor Ambarus

Quoted 226 lines · Tudor Ambarus
Quoted 216 lines · Tudor Ambarus
Quoted 36 lines · Ilpo Järvinen
Quoted 34 lines · Tudor Ambarus
Hi!
Quoted 18 lines · Ilpo Järvinen
Resetting resource is problematic as it prevent attempting to allocate the resource later, unless something in between restores the resource. Similarly, if fail_head does not contain all resources that were reset, those resource cannot be restored later.
The entire reset/restore cycle adds complexity and leaving resources into reseted state causes issues to other code such as for checks done in pci_enable_resources(). Take a small step towards not resetting resources by delaying reset until the end of resource assignment and build failure list (fail_head) in sync with the reset to avoid leaving behind resources that cannot be restored (for the case where the caller provides fail_head in the first place to allow restore somewhere in the callchain, as is not all callers pass non-NULL fail_head).
The Expansion ROM check is temporarily left in place while building the failure list until the upcoming change which reworks optional resource handling.
Quoted 5 lines
Ideally, whole resource reset could be removed but doing that in a big step would make the impact non-tractable due to complexity of all related code.
Signed-off-by Ilpo Järvinen <[email protected]>
I'm hitting the BUG_ON(!list_empty(&add_list)); in pci_assign_unassigned_bus_resources() [1] with 6.15-rc5 and the the pixel6 downstream pcie driver.
I saw the thread where "a34d74877c66 PCI: Restore assigned resources fully after release" fixes things for some other cases, but it's not the case here.
Quoted 6 lines
Quoted 4 lines
Reverting the following patches fixes the problem:
a34d74877c66 PCI: Restore assigned resources fully after release 2499f5348431 PCI: Rework optional resource handling 96336ec70264 PCI: Perform reset_resource() and build fail list in sync
So it's confirmed that you needed to revert also this last commit 96336ec70264, not just the rework change?
I needed to revert 96336ec70264 as well otherwise the build fails.
Quoted 80 lines
Quoted 80 lines
In the working case the add_list list is empty throughout the entire body of pci_assign_unassigned_bus_resources().
In the failing case __pci_bus_size_bridges() leaves the add_list not empty and __pci_bus_assign_resources() does not consume the list, thus the BUG_ON. The failing case contains an extra print that's not shown when reverting the blamed commits:
[   13.951185][ T1101] pcieport 0000:00:00.0: bridge window [mem
0x00100000-0x001fffff] to [bus 01-ff] add_size 100000 add_align 100000
I've added some prints trying to describe the code path, see
https://paste.ofcode.org/Aeu2YBpLztc49ZDw3uUJmd#
Failing case:
[   13.944231][ T1101] pci 0000:01:00.0: [144d:a5a5] type 00 class
0x000000 PCIe Endpoint
[   13.944412][ T1101] pci 0000:01:00.0: BAR 0 [mem
0x00000000-0x000fffff 64bit]
[   13.944532][ T1101] pci 0000:01:00.0: ROM [mem 0x00000000-0x0000ffff
pref]
[   13.944649][ T1101] pci 0000:01:00.0: enabling Extended Tags
[   13.944844][ T1101] pci 0000:01:00.0: PME# supported from D0 D3hot D3cold
[   13.945015][ T1101] pci 0000:01:00.0: 15.752 Gb/s available PCIe
bandwidth, limited by 8.0 GT/s PCIe x2 link at 0000:00:00.0 (capable of
31.506 Gb/s with 16.0 GT/s PCIe x2 link)
[   13.950616][ T1101] __pci_bus_size_bridges: before pbus_size_mem.
list empty? 1
[   13.950784][ T1101] pbus_size_mem: 2. list empty? 1
[   13.950886][ T1101] pbus_size_mem: 1 list empty? 0
[   13.950982][ T1101] pbus_size_mem: 3. list empty? 0
[   13.951082][ T1101] pbus_size_mem: 4. list empty? 0
[   13.951185][ T1101] pcieport 0000:00:00.0: bridge window [mem
0x00100000-0x001fffff] to [bus 01-ff] add_size 100000 add_align 100000
[   13.951448][ T1101] __pci_bus_size_bridges: after pbus_size_mem. list
empty? 0
[   13.951643][ T1101] pci_assign_unassigned_bus_resources: before
__pci_bus_assign_resources -> list empty? 0
[   13.951924][ T1101] pcieport 0000:00:00.0: bridge window [mem
0x40000000-0x401fffff]: assigned
[   13.952248][ T1101] pci_assign_unassigned_bus_resources: after
__pci_bus_assign_resources -> list empty? 0
[   13.952634][ T1101] ------------[ cut here ]------------
[   13.952818][ T1101] kernel BUG at drivers/pci/setup-bus.c:2514!
[   13.953045][ T1101] Internal error: Oops - BUG: 00000000f2000800 [#1]
 SMP
...
[   13.976086][ T1101] Call trace:
[   13.976206][ T1101]  pci_assign_unassigned_bus_resources+0x110/0x114 (P)
[   13.976462][ T1101]  pci_rescan_bus+0x28/0x48
[   13.976628][ T1101]  exynos_pcie_rc_poweron
Working case:
[   13.786961][ T1120] pci 0000:01:00.0: [144d:a5a5] type 00 class
0x000000 PCIe Endpoint
[   13.787136][ T1120] pci 0000:01:00.0: BAR 0 [mem
0x00000000-0x000fffff 64bit]
[   13.787280][ T1120] pci 0000:01:00.0: ROM [mem 0x00000000-0x0000ffff
pref]
[   13.787541][ T1120] pci 0000:01:00.0: enabling Extended Tags
[   13.787808][ T1120] pci 0000:01:00.0: PME# supported from D0 D3hot D3cold
[   13.787988][ T1120] pci 0000:01:00.0: 15.752 Gb/s available PCIe
bandwidth, limited by 8.0 GT/s PCIe x2 link at 0000:00:00.0 (capable of
31.506 Gb/s with 16.0 GT/s PCIe x2 link)
[   13.795279][ T1120] __pci_bus_size_bridges: before pbus_size_mem.
list empty? 1
[   13.795408][ T1120] pbus_size_mem: 2. list empty? 1
[   13.795495][ T1120] pbus_size_mem: 2. list empty? 1
[   13.795577][ T1120] __pci_bus_size_bridges: after pbus_size_mem. list
empty? 1
[   13.795692][ T1120] pci_assign_unassigned_bus_resources: before
__pci_bus_assign_resources -> list empty? 1
[   13.795849][ T1120] pcieport 0000:00:00.0: bridge window [mem
0x40000000-0x401fffff]: assigned
[   13.796072][ T1120] pci_assign_unassigned_bus_resources: after
__pci_bus_assign_resources -> list empty? 1
[   13.796662][ T1120] cpif: s5100_poweron_pcie: DBG: MSI sfr not set
up, yet(s5100_pdev is NULL)
[   13.796666][ T1120] cpif: register_pcie: s51xx_pcie_init start
Quoted 3 lines
Quoted 2 lines
Any hints are welcomed. Thanks,
ta
Hi and thanks for the report.
Hi! Thanks for the help. I've been out of office for the last 2 weeks, sorry for the delayed reply.
Quoted 7 lines
The interesting part occurs inside reassign_resources_sorted() where most items are eliminated from realloc_head by the list_del().
My guess is that somehow, the change in 96336ec70264 from !res->flags to the more complicated check somehow causes this. If the new check doesn't match and subsequently, no match is found from the head list, the loop will do continue and not remove the entry from realloc_head.
I added a print right there and it seems it's something else. See below.
Quoted 8 lines
But it's hard to confirm without knowing what that resources realloc_head contains. Perhaps if you print the resources that are processed around that part of the code in reassign_resources_sorted(), comparing the log from the reverted code with the non-working case might help to understand what is different there and why. To understand better what is in the head list, it would be also useful to know from which device the resources were added into the head list in pdev_sort_resources().
I added the suggested prints
(https://paste.ofcode.org/DgmZGGgS6D36nWEzmfCqMm) on top of v6.15 with the downstream PCIe pixel driver and I obtain the following. Note that all added prints contain "tudor" for differentiation.
[   15.211179][ T1107] pci 0001:01:00.0: [144d:a5a5] type 00 class
0x000000 PCIe Endpoint
[   15.212248][ T1107] pci 0001:01:00.0: BAR 0 [mem
0x00000000-0x000fffff 64bit]
[   15.212775][ T1107] pci 0001:01:00.0: ROM [mem 0x00000000-0x0000ffff
pref]
[   15.213195][ T1107] pci 0001:01:00.0: enabling Extended Tags
[   15.213720][ T1107] pci 0001:01:00.0: PME# supported from D0 D3hot
D3cold
[   15.214035][ T1107] pci 0001:01:00.0: 15.752 Gb/s available PCIe
bandwidth, limited by 8.0 GT/s PCIe x2 link at 0001:00:00.0 (capable of
31.506 Gb/s with 16.0 GT/s PCIe x2 link)
[   15.222286][ T1107] pci 0001:01:00.0: tudor: 1: pbus_size_mem: BAR 0
[mem 0x00000000-0x000fffff 64bit] list empty? 1
[   15.222813][ T1107] pci 0001:01:00.0: tudor: 1: pbus_size_mem: ROM
[mem 0x00000000-0x0000ffff pref] list empty? 1
[   15.224429][ T1107] pci 0001:01:00.0: tudor: 2: pbus_size_mem: ROM
[mem 0x00000000-0x0000ffff pref] list empty? 0
[   15.224750][ T1107] pcieport 0001:00:00.0: bridge window [mem
0x00100000-0x001fffff] to [bus 01-ff] add_size 100000 add_align 100000
[   15.225393][ T1107] tudor : pci_assign_unassigned_bus_resources:
before __pci_bus_assign_resources -> list empty? 0
[   15.225594][ T1107] pcieport 0001:00:00.0: tudor:
pdev_sort_resources: bridge window [mem 0x00100000-0x001fffff] resource
added in head list
[   15.226078][ T1107] pcieport 0001:00:00.0: bridge window [mem
0x40000000-0x401fffff]: assigned
[   15.226419][ T1107] tudor : pci_assign_unassigned_bus_resources:
after __pci_bus_assign_resources -> list empty? 0
[   15.226442][ T1107] ------------[ cut here ]------------
[   15.227587][ T1107] kernel BUG at drivers/pci/setup-bus.c:2522!
[   15.227813][ T1107] Internal error: Oops - BUG: 00000000f2000800 [#1]
 SMP
...
[   15.251570][ T1107] Call trace:
[   15.251690][ T1107]  pci_assign_unassigned_bus_resources+0x110/0x114 (P)
[   15.251945][ T1107]  pci_rescan_bus+0x28/0x48
I obtain the following output when using the same prints adapted
(https://paste.ofcode.org/37w7RnKkPaCxyNhi5yhZPbZ) and with the blamed commits reverted:
a34d74877c66 PCI: Restore assigned resources fully after release 2499f5348431 PCI: Rework optional resource handling 96336ec70264 PCI: Perform reset_resource() and build fail list in sync
[   15.200456][ T1102] pci 0000:01:00.0: [144d:a5a5] type 00 class
0x000000 PCIe Endpoint
[   15.200632][ T1102] pci 0000:01:00.0: BAR 0 [mem
0x00000000-0x000fffff 64bit]
[   15.200755][ T1102] pci 0000:01:00.0: ROM [mem 0x00000000-0x0000ffff
pref]
[   15.200876][ T1102] pci 0000:01:00.0: enabling Extended Tags
[   15.201075][ T1102] pci 0000:01:00.0: PME# supported from D0 D3hot D3cold
[   15.201254][ T1102] pci 0000:01:00.0: 15.752 Gb/s available PCIe
bandwidth, limited by 8.0 GT/s PCIe x2 link at 0000:00:00.0 (capable of
31.506 Gb/s with 16.0 GT/s PCIe x2 link)
[   15.206555][ T1102] pci 0000:01:00.0: tudor: 1: pbus_size_mem: BAR 0
[mem 0x00000000-0x000fffff 64bit] list empty? 1
[   15.206737][ T1102] pci 0000:01:00.0: tudor: 1: pbus_size_mem: ROM
[mem 0x00000000-0x0000ffff pref] list empty? 1
[   15.206901][ T1102] tudor : pci_assign_unassigned_bus_resources:
before __pci_bus_assign_resources -> list empty? 1
[   15.207072][ T1102] pcieport 0000:00:00.0: tudor:
pdev_sort_resources: bridge window [mem 0x00100000-0x002fffff] resource
Quoted 8 lines
added in head list
[   15.207396][ T1102] pcieport 0000:00:00.0: bridge window [mem
0x40000000-0x401fffff]: assigned
[   15.208165][ T1102] tudor : pci_assign_unassigned_bus_resources:
after __pci_bus_assign_resources -> list empty? 1
[   15.208783][ T1102] cpif: s5100_poweron_pcie: DBG: MSI sfr not set
up, yet(s5100_pdev is NULL)
[   15.208786][ T1102] cpif: register_pcie: s51xx_pcie_init start
Quoted 3 lines
I see my email client split the lines for the prints making the output very hard to read. Added the output here too:
https://paste.ofcode.org/AEfjASQW8Z2jbMak5VkmpJ
With the following change things get back to how they were before 2499f5348431:
drivers/pci/setup-bus.c +3 -2
index 5247370010aa..1589dd8afa69 100644
@@ -1214,9 +1214,10 @@ static int pbus_size_mem(struct pci_bus *bus,
 unsigned long mask,
 				__func__, r_name, r, list_empty(realloc_head));
 
 			/* Put SRIOV requested res to the optional list */
-			if (realloc_head && pci_resource_is_optional(dev, i)) {
+			if (realloc_head && pci_resource_is_iov(i)) {
 				add_align = max(pci_resource_alignment(dev, r), add_align);
-				add_to_list(realloc_head, dev, r, 0, 0 /* Don't care */);
+				resource_set_size(r, 0);
+				add_to_list(realloc_head, dev, r, r_size, 0 /* Don't care */);
 				children_add_size += r_size;
 				pci_info(dev, "tudor: 2: %s: %s %pR list empty? %d\n",

Quoted 1 line

__func__, r_name, r, list_empty(realloc_head));

Okay, thanks for the additional data point, but I don't think the problem is at this site. That removal of that resource_set_size(r, 0) is crucial piece in my changes (and as is, it even doesn't extend far enough and we can end up lose resources in some situations when their sizes are reset like that).

This entire resource fitting and assignment code as whole is very complex spanning many functions. The steps are roughly these:

1) calculate sizing in advance (which the above code relates to)
2) collect all resources we want to assign in a PCI bus subtree
3) opportunistically assign with the optional sizes
4) if everything was assigned, skip steps 5-7.
5) release some/all from step 2 assignments
6) assign only required resources
7) reassign optional parts (enlargen to include the optional sizes) what we can
8) As a final sanity check, realloc_head is checked to have been fully consumed by the earlier steps (~ no unchecked work todo remains)

In some cases, kernel iterates the entire process a few times (but that mostly occurs when pci=realloc is given on the kernel cmdline).

Now, as discussed in my other replay, it looks at some point the algorithm assigned the bridge window resource successfully, but it still remained in the realloc_head for some reason if I parsed all the information from your debug patch right. The resource should have been removed from realloc_head before __assign_resources_sorted() exits if it got assigned, if it wasn't, that's the bug we're looking for. (Alternatively, it could have been that the assignment was never tried for the particular resource, but it now looks that might not be the case here).

Signature · 1 line

i.

Ilpo Järvinen

author

2025-05-30 06:38 UTC #43 lore raw

Re: [PATCH 24/25] PCI: Perform reset_resource() and build fail list in sync

In reply to Tudor Ambarus

Quoted 40 lines · Tudor Ambarus

Quoted 39 lines · Ilpo Järvinen
Quoted 37 lines · Tudor Ambarus
Quoted 18 lines · Ilpo Järvinen
Resetting resource is problematic as it prevent attempting to allocate the resource later, unless something in between restores the resource. Similarly, if fail_head does not contain all resources that were reset, those resource cannot be restored later.
The entire reset/restore cycle adds complexity and leaving resources into reseted state causes issues to other code such as for checks done in pci_enable_resources(). Take a small step towards not resetting resources by delaying reset until the end of resource assignment and build failure list (fail_head) in sync with the reset to avoid leaving behind resources that cannot be restored (for the case where the caller provides fail_head in the first place to allow restore somewhere in the callchain, as is not all callers pass non-NULL fail_head).
The Expansion ROM check is temporarily left in place while building the failure list until the upcoming change which reworks optional resource handling.
Quoted 5 lines
Ideally, whole resource reset could be removed but doing that in a big step would make the impact non-tractable due to complexity of all related code.
Signed-off-by Ilpo Järvinen <[email protected]>
I'm hitting the BUG_ON(!list_empty(&add_list)); in pci_assign_unassigned_bus_resources() [1] with 6.15-rc5 and the the pixel6 downstream pcie driver.
I saw the thread where "a34d74877c66 PCI: Restore assigned resources fully after release" fixes things for some other cases, but it's not the case here.
Reverting the following patches fixes the problem:
a34d74877c66 PCI: Restore assigned resources fully after release 2499f5348431 PCI: Rework optional resource handling 96336ec70264 PCI: Perform reset_resource() and build fail list in sync

Quoted 4 lines

Quoted 2 lines
So it's confirmed that you needed to revert also this last commit 96336ec70264, not just the rework change?
I needed to revert 96336ec70264 as well otherwise the build fails.

Hi again,

That's news to me... I seem to have botched the resource assignment rework series at some point when I reordered patches and dropped that helper as a result. And it seems the intermediate build fail wasn't caught by LKP :-(. (Pretty annoying as I intentionally separated these two to make them bisectable but not it isn't without amends.)

The missing helper is basically this:

static bool pci_resource_is_disabled_rom(const struct pci_dev *dev, int resno)

{
	const struct resource *res = pci_resource_n(dev, resno);

	return resno == PCI_ROM_RESOURCE && !(res->flags & IORESOURCE_ROM_ENABLE)
}

(I didn't build test that.)

Because of this, the actual culprit could be in 2499f5348431, not it 96336ec70264 (which would make more sense as it does significant rework on the assignment algorithm).

Quoted 80 lines

In the working case the add_list list is empty throughout the entire body of pci_assign_unassigned_bus_resources().

[   13.951185][ T1101] pcieport 0000:00:00.0: bridge window [mem

0x00100000-0x001fffff] to [bus 01-ff] add_size 100000 add_align 100000

I've added some prints trying to describe the code path, see
https://paste.ofcode.org/Aeu2YBpLztc49ZDw3uUJmd#

Failing case:

[   13.944231][ T1101] pci 0000:01:00.0: [144d:a5a5] type 00 class
0x000000 PCIe Endpoint
[   13.944412][ T1101] pci 0000:01:00.0: BAR 0 [mem
0x00000000-0x000fffff 64bit]
[   13.944532][ T1101] pci 0000:01:00.0: ROM [mem 0x00000000-0x0000ffff
pref]
[   13.944649][ T1101] pci 0000:01:00.0: enabling Extended Tags
[   13.944844][ T1101] pci 0000:01:00.0: PME# supported from D0 D3hot D3cold
[   13.945015][ T1101] pci 0000:01:00.0: 15.752 Gb/s available PCIe
bandwidth, limited by 8.0 GT/s PCIe x2 link at 0000:00:00.0 (capable of
31.506 Gb/s with 16.0 GT/s PCIe x2 link)
[   13.950616][ T1101] __pci_bus_size_bridges: before pbus_size_mem.
list empty? 1
[   13.950784][ T1101] pbus_size_mem: 2. list empty? 1
[   13.950886][ T1101] pbus_size_mem: 1 list empty? 0
[   13.950982][ T1101] pbus_size_mem: 3. list empty? 0
[   13.951082][ T1101] pbus_size_mem: 4. list empty? 0
[   13.951185][ T1101] pcieport 0000:00:00.0: bridge window [mem
0x00100000-0x001fffff] to [bus 01-ff] add_size 100000 add_align 100000
[   13.951448][ T1101] __pci_bus_size_bridges: after pbus_size_mem. list
empty? 0
[   13.951643][ T1101] pci_assign_unassigned_bus_resources: before
__pci_bus_assign_resources -> list empty? 0
[   13.951924][ T1101] pcieport 0000:00:00.0: bridge window [mem
0x40000000-0x401fffff]: assigned
[   13.952248][ T1101] pci_assign_unassigned_bus_resources: after
__pci_bus_assign_resources -> list empty? 0
[   13.952634][ T1101] ------------[ cut here ]------------
[   13.952818][ T1101] kernel BUG at drivers/pci/setup-bus.c:2514!
[   13.953045][ T1101] Internal error: Oops - BUG: 00000000f2000800 [#1]
 SMP
...
[   13.976086][ T1101] Call trace:
[   13.976206][ T1101]  pci_assign_unassigned_bus_resources+0x110/0x114 (P)
[   13.976462][ T1101]  pci_rescan_bus+0x28/0x48
[   13.976628][ T1101]  exynos_pcie_rc_poweron

Working case:

[   13.786961][ T1120] pci 0000:01:00.0: [144d:a5a5] type 00 class
0x000000 PCIe Endpoint
[   13.787136][ T1120] pci 0000:01:00.0: BAR 0 [mem
0x00000000-0x000fffff 64bit]
[   13.787280][ T1120] pci 0000:01:00.0: ROM [mem 0x00000000-0x0000ffff
pref]
[   13.787541][ T1120] pci 0000:01:00.0: enabling Extended Tags
[   13.787808][ T1120] pci 0000:01:00.0: PME# supported from D0 D3hot D3cold
[   13.787988][ T1120] pci 0000:01:00.0: 15.752 Gb/s available PCIe
bandwidth, limited by 8.0 GT/s PCIe x2 link at 0000:00:00.0 (capable of
31.506 Gb/s with 16.0 GT/s PCIe x2 link)
[   13.795279][ T1120] __pci_bus_size_bridges: before pbus_size_mem.
list empty? 1
[   13.795408][ T1120] pbus_size_mem: 2. list empty? 1
[   13.795495][ T1120] pbus_size_mem: 2. list empty? 1
[   13.795577][ T1120] __pci_bus_size_bridges: after pbus_size_mem. list
empty? 1
[   13.795692][ T1120] pci_assign_unassigned_bus_resources: before
__pci_bus_assign_resources -> list empty? 1
[   13.795849][ T1120] pcieport 0000:00:00.0: bridge window [mem
0x40000000-0x401fffff]: assigned
[   13.796072][ T1120] pci_assign_unassigned_bus_resources: after
__pci_bus_assign_resources -> list empty? 1
[   13.796662][ T1120] cpif: s5100_poweron_pcie: DBG: MSI sfr not set
up, yet(s5100_pdev is NULL)
[   13.796666][ T1120] cpif: register_pcie: s51xx_pcie_init start

Quoted 7 lines

Quoted 4 lines
Quoted 2 lines
Any hints are welcomed. Thanks,
ta
Hi and thanks for the report.
Hi! Thanks for the help. I've been out of office for the last 2 weeks, sorry for the delayed reply.

Np.

Quoted 45 lines

Quoted 7 lines
The interesting part occurs inside reassign_resources_sorted() where most items are eliminated from realloc_head by the list_del().
My guess is that somehow, the change in 96336ec70264 from !res->flags to the more complicated check somehow causes this. If the new check doesn't match and subsequently, no match is found from the head list, the loop will do continue and not remove the entry from realloc_head.
I added a print right there and it seems it's something else. See below.
Quoted 9 lines
But it's hard to confirm without knowing what that resources realloc_head contains. Perhaps if you print the resources that are processed around that part of the code in reassign_resources_sorted(), comparing the log from the reverted code with the non-working case might help to understand what is different there and why. To understand better what is in the head list, it would be also useful to know from which device the resources were added into the head list in pdev_sort_resources().
I added the suggested prints
(https://paste.ofcode.org/DgmZGGgS6D36nWEzmfCqMm) on top of v6.15 with the downstream PCIe pixel driver and I obtain the following. Note that all added prints contain "tudor" for differentiation.
[   15.211179][ T1107] pci 0001:01:00.0: [144d:a5a5] type 00 class
0x000000 PCIe Endpoint
[   15.212248][ T1107] pci 0001:01:00.0: BAR 0 [mem
0x00000000-0x000fffff 64bit]
[   15.212775][ T1107] pci 0001:01:00.0: ROM [mem 0x00000000-0x0000ffff
pref]
[   15.213195][ T1107] pci 0001:01:00.0: enabling Extended Tags
[   15.213720][ T1107] pci 0001:01:00.0: PME# supported from D0 D3hot
D3cold
[   15.214035][ T1107] pci 0001:01:00.0: 15.752 Gb/s available PCIe
bandwidth, limited by 8.0 GT/s PCIe x2 link at 0001:00:00.0 (capable of
31.506 Gb/s with 16.0 GT/s PCIe x2 link)
[   15.222286][ T1107] pci 0001:01:00.0: tudor: 1: pbus_size_mem: BAR 0
[mem 0x00000000-0x000fffff 64bit] list empty? 1
[   15.222813][ T1107] pci 0001:01:00.0: tudor: 1: pbus_size_mem: ROM
[mem 0x00000000-0x0000ffff pref] list empty? 1
[   15.224429][ T1107] pci 0001:01:00.0: tudor: 2: pbus_size_mem: ROM
[mem 0x00000000-0x0000ffff pref] list empty? 0
[   15.224750][ T1107] pcieport 0001:00:00.0: bridge window [mem
0x00100000-0x001fffff] to [bus 01-ff] add_size 100000 add_align 100000

Quoted 7 lines

[   15.225393][ T1107] tudor : pci_assign_unassigned_bus_resources:
before __pci_bus_assign_resources -> list empty? 0
[   15.225594][ T1107] pcieport 0001:00:00.0: tudor:
pdev_sort_resources: bridge window [mem 0x00100000-0x001fffff] resource
added in head list
[   15.226078][ T1107] pcieport 0001:00:00.0: bridge window [mem

0x40000000-0x401fffff]: assigned

So here it ends up assigning the resource here I think.

That print isn't one of yours in reassign_resources_sorted() so the assignment must have been made in assign_requested_resources_sorted(). But then nothing is printed out from reassign_resources_sorted() so I suspect __assign_resources_sorted() has short-circuited.

We know that realloc_head is not empty, so that leaves the goto out from if (list_empty(&local_fail_head)), which kind of makes sense, all entries on the head list were assigned. But the code there tries to remove all head list resources from realloc_head so why it doesn't get removed is still a mystery. assign_requested_resources_sorted() doesn't seem to remove anything from the head list so that resource should still be on the head list AFAICT so it should call that remove_from_list(realloc_head, dev_res->res) for it.

So can you see if that theory holds water and it short-circuits without removing the entry from realloc_head?

Quoted 49 lines

[   15.226419][ T1107] tudor : pci_assign_unassigned_bus_resources:
after __pci_bus_assign_resources -> list empty? 0
[   15.226442][ T1107] ------------[ cut here ]------------
[   15.227587][ T1107] kernel BUG at drivers/pci/setup-bus.c:2522!
[   15.227813][ T1107] Internal error: Oops - BUG: 00000000f2000800 [#1]
 SMP
...
[   15.251570][ T1107] Call trace:
[   15.251690][ T1107]  pci_assign_unassigned_bus_resources+0x110/0x114 (P)
[   15.251945][ T1107]  pci_rescan_bus+0x28/0x48

[   15.200456][ T1102] pci 0000:01:00.0: [144d:a5a5] type 00 class
0x000000 PCIe Endpoint
[   15.200632][ T1102] pci 0000:01:00.0: BAR 0 [mem
0x00000000-0x000fffff 64bit]
[   15.200755][ T1102] pci 0000:01:00.0: ROM [mem 0x00000000-0x0000ffff
pref]
[   15.200876][ T1102] pci 0000:01:00.0: enabling Extended Tags
[   15.201075][ T1102] pci 0000:01:00.0: PME# supported from D0 D3hot D3cold
[   15.201254][ T1102] pci 0000:01:00.0: 15.752 Gb/s available PCIe
bandwidth, limited by 8.0 GT/s PCIe x2 link at 0000:00:00.0 (capable of
31.506 Gb/s with 16.0 GT/s PCIe x2 link)
[   15.206555][ T1102] pci 0000:01:00.0: tudor: 1: pbus_size_mem: BAR 0
[mem 0x00000000-0x000fffff 64bit] list empty? 1
[   15.206737][ T1102] pci 0000:01:00.0: tudor: 1: pbus_size_mem: ROM
[mem 0x00000000-0x0000ffff pref] list empty? 1
[   15.206901][ T1102] tudor : pci_assign_unassigned_bus_resources:
before __pci_bus_assign_resources -> list empty? 1
[   15.207072][ T1102] pcieport 0000:00:00.0: tudor:
pdev_sort_resources: bridge window [mem 0x00100000-0x002fffff] resource
added in head list
[   15.207396][ T1102] pcieport 0000:00:00.0: bridge window [mem
0x40000000-0x401fffff]: assigned
[   15.208165][ T1102] tudor : pci_assign_unassigned_bus_resources:
after __pci_bus_assign_resources -> list empty? 1
[   15.208783][ T1102] cpif: s5100_poweron_pcie: DBG: MSI sfr not set
up, yet(s5100_pdev is NULL)
[   15.208786][ T1102] cpif: register_pcie: s51xx_pcie_init start

Quoted 3 lines

In any case, that BUG_ON() seems a bit drastic action for what might be just a single resource allocation failure so it should be downgraded to:

Quoted 7 lines

Quoted 4 lines
if (WARN_ON(!list_empty(&add_list))
	free_list(&add_list);
... or WARN_ON_ONCE().
I saw your patch doing this, the phone now boots, but obviously I still see the WARN, so maybe there's still something to be fixed.

Yes, I don't expect BUG_ON() -> WARN "fix" anything, it just downgrades the severity so that the system can still try to boot, which can often succeed as this tends to be non-critical failure in many cases so it's useful change to have regardless despite the splat.

Now that it boots, can you please check if /proc/iomem is the same both in the non-working and working config. If that resource got assigned successfully, it might well be there is no actual differences in the assigned resources (which again doesn't mean there wouldn't be a bug in the logic as discussed above).

Signature · 1 line

i.

Ilpo Järvinen

author

2025-05-30 14:48 UTC #44 lore raw

Re: [PATCH 24/25] PCI: Perform reset_resource() and build fail list in sync

Quoted 229 lines · Ilpo Järvinen

Quoted 40 lines · Tudor Ambarus
Quoted 39 lines · Ilpo Järvinen
Quoted 37 lines · Tudor Ambarus
Quoted 18 lines · Ilpo Järvinen
Resetting resource is problematic as it prevent attempting to allocate the resource later, unless something in between restores the resource. Similarly, if fail_head does not contain all resources that were reset, those resource cannot be restored later.
The entire reset/restore cycle adds complexity and leaving resources into reseted state causes issues to other code such as for checks done in pci_enable_resources(). Take a small step towards not resetting resources by delaying reset until the end of resource assignment and build failure list (fail_head) in sync with the reset to avoid leaving behind resources that cannot be restored (for the case where the caller provides fail_head in the first place to allow restore somewhere in the callchain, as is not all callers pass non-NULL fail_head).
The Expansion ROM check is temporarily left in place while building the failure list until the upcoming change which reworks optional resource handling.
Quoted 5 lines
Ideally, whole resource reset could be removed but doing that in a big step would make the impact non-tractable due to complexity of all related code.
Signed-off-by Ilpo Järvinen <[email protected]>
I'm hitting the BUG_ON(!list_empty(&add_list)); in pci_assign_unassigned_bus_resources() [1] with 6.15-rc5 and the the pixel6 downstream pcie driver.
I saw the thread where "a34d74877c66 PCI: Restore assigned resources fully after release" fixes things for some other cases, but it's not the case here.
Reverting the following patches fixes the problem:
a34d74877c66 PCI: Restore assigned resources fully after release 2499f5348431 PCI: Rework optional resource handling 96336ec70264 PCI: Perform reset_resource() and build fail list in sync
Quoted 4 lines
Quoted 2 lines
So it's confirmed that you needed to revert also this last commit 96336ec70264, not just the rework change?
I needed to revert 96336ec70264 as well otherwise the build fails.
Hi again,
That's news to me... I seem to have botched the resource assignment rework series at some point when I reordered patches and dropped that helper as a result. And it seems the intermediate build fail wasn't caught by LKP :-(. (Pretty annoying as I intentionally separated these two to make them bisectable but not it isn't without amends.)
The missing helper is basically this:
static bool pci_resource_is_disabled_rom(const struct pci_dev *dev, int resno)
{
	const struct resource *res = pci_resource_n(dev, resno);

	return resno == PCI_ROM_RESOURCE && !(res->flags & IORESOURCE_ROM_ENABLE)
}
(I didn't build test that.)
Because of this, the actual culprit could be in 2499f5348431, not it 96336ec70264 (which would make more sense as it does significant rework on the assignment algorithm).
Quoted 80 lines
Quoted 80 lines
Quoted 80 lines
In the working case the add_list list is empty throughout the entire body of pci_assign_unassigned_bus_resources().
In the failing case __pci_bus_size_bridges() leaves the add_list not empty and __pci_bus_assign_resources() does not consume the list, thus the BUG_ON. The failing case contains an extra print that's not shown when reverting the blamed commits:
[   13.951185][ T1101] pcieport 0000:00:00.0: bridge window [mem
0x00100000-0x001fffff] to [bus 01-ff] add_size 100000 add_align 100000
I've added some prints trying to describe the code path, see
https://paste.ofcode.org/Aeu2YBpLztc49ZDw3uUJmd#
Failing case:
[   13.944231][ T1101] pci 0000:01:00.0: [144d:a5a5] type 00 class
0x000000 PCIe Endpoint
[   13.944412][ T1101] pci 0000:01:00.0: BAR 0 [mem
0x00000000-0x000fffff 64bit]
[   13.944532][ T1101] pci 0000:01:00.0: ROM [mem 0x00000000-0x0000ffff
pref]
[   13.944649][ T1101] pci 0000:01:00.0: enabling Extended Tags
[   13.944844][ T1101] pci 0000:01:00.0: PME# supported from D0 D3hot D3cold
[   13.945015][ T1101] pci 0000:01:00.0: 15.752 Gb/s available PCIe
bandwidth, limited by 8.0 GT/s PCIe x2 link at 0000:00:00.0 (capable of
31.506 Gb/s with 16.0 GT/s PCIe x2 link)
[   13.950616][ T1101] __pci_bus_size_bridges: before pbus_size_mem.
list empty? 1
[   13.950784][ T1101] pbus_size_mem: 2. list empty? 1
[   13.950886][ T1101] pbus_size_mem: 1 list empty? 0
[   13.950982][ T1101] pbus_size_mem: 3. list empty? 0
[   13.951082][ T1101] pbus_size_mem: 4. list empty? 0
[   13.951185][ T1101] pcieport 0000:00:00.0: bridge window [mem
0x00100000-0x001fffff] to [bus 01-ff] add_size 100000 add_align 100000
[   13.951448][ T1101] __pci_bus_size_bridges: after pbus_size_mem. list
empty? 0
[   13.951643][ T1101] pci_assign_unassigned_bus_resources: before
__pci_bus_assign_resources -> list empty? 0
[   13.951924][ T1101] pcieport 0000:00:00.0: bridge window [mem
0x40000000-0x401fffff]: assigned
[   13.952248][ T1101] pci_assign_unassigned_bus_resources: after
__pci_bus_assign_resources -> list empty? 0
[   13.952634][ T1101] ------------[ cut here ]------------
[   13.952818][ T1101] kernel BUG at drivers/pci/setup-bus.c:2514!
[   13.953045][ T1101] Internal error: Oops - BUG: 00000000f2000800 [#1]
 SMP
...
[   13.976086][ T1101] Call trace:
[   13.976206][ T1101]  pci_assign_unassigned_bus_resources+0x110/0x114 (P)
[   13.976462][ T1101]  pci_rescan_bus+0x28/0x48
[   13.976628][ T1101]  exynos_pcie_rc_poweron
Working case:
[   13.786961][ T1120] pci 0000:01:00.0: [144d:a5a5] type 00 class
0x000000 PCIe Endpoint
[   13.787136][ T1120] pci 0000:01:00.0: BAR 0 [mem
0x00000000-0x000fffff 64bit]
[   13.787280][ T1120] pci 0000:01:00.0: ROM [mem 0x00000000-0x0000ffff
pref]
[   13.787541][ T1120] pci 0000:01:00.0: enabling Extended Tags
[   13.787808][ T1120] pci 0000:01:00.0: PME# supported from D0 D3hot D3cold
[   13.787988][ T1120] pci 0000:01:00.0: 15.752 Gb/s available PCIe
bandwidth, limited by 8.0 GT/s PCIe x2 link at 0000:00:00.0 (capable of
31.506 Gb/s with 16.0 GT/s PCIe x2 link)
[   13.795279][ T1120] __pci_bus_size_bridges: before pbus_size_mem.
list empty? 1
[   13.795408][ T1120] pbus_size_mem: 2. list empty? 1
[   13.795495][ T1120] pbus_size_mem: 2. list empty? 1
[   13.795577][ T1120] __pci_bus_size_bridges: after pbus_size_mem. list
empty? 1
[   13.795692][ T1120] pci_assign_unassigned_bus_resources: before
__pci_bus_assign_resources -> list empty? 1
[   13.795849][ T1120] pcieport 0000:00:00.0: bridge window [mem
0x40000000-0x401fffff]: assigned
[   13.796072][ T1120] pci_assign_unassigned_bus_resources: after
__pci_bus_assign_resources -> list empty? 1
[   13.796662][ T1120] cpif: s5100_poweron_pcie: DBG: MSI sfr not set
up, yet(s5100_pdev is NULL)
[   13.796666][ T1120] cpif: register_pcie: s51xx_pcie_init start
Quoted 7 lines
Quoted 4 lines
Quoted 2 lines
Any hints are welcomed. Thanks,
ta
Hi and thanks for the report.
Hi! Thanks for the help. I've been out of office for the last 2 weeks, sorry for the delayed reply.
Np.
Quoted 45 lines
Quoted 7 lines
The interesting part occurs inside reassign_resources_sorted() where most items are eliminated from realloc_head by the list_del().
My guess is that somehow, the change in 96336ec70264 from !res->flags to the more complicated check somehow causes this. If the new check doesn't match and subsequently, no match is found from the head list, the loop will do continue and not remove the entry from realloc_head.
I added a print right there and it seems it's something else. See below.
Quoted 9 lines
But it's hard to confirm without knowing what that resources realloc_head contains. Perhaps if you print the resources that are processed around that part of the code in reassign_resources_sorted(), comparing the log from the reverted code with the non-working case might help to understand what is different there and why. To understand better what is in the head list, it would be also useful to know from which device the resources were added into the head list in pdev_sort_resources().
I added the suggested prints
(https://paste.ofcode.org/DgmZGGgS6D36nWEzmfCqMm) on top of v6.15 with the downstream PCIe pixel driver and I obtain the following. Note that all added prints contain "tudor" for differentiation.
[   15.211179][ T1107] pci 0001:01:00.0: [144d:a5a5] type 00 class
0x000000 PCIe Endpoint
[   15.212248][ T1107] pci 0001:01:00.0: BAR 0 [mem
0x00000000-0x000fffff 64bit]
[   15.212775][ T1107] pci 0001:01:00.0: ROM [mem 0x00000000-0x0000ffff
pref]
[   15.213195][ T1107] pci 0001:01:00.0: enabling Extended Tags
[   15.213720][ T1107] pci 0001:01:00.0: PME# supported from D0 D3hot
D3cold
[   15.214035][ T1107] pci 0001:01:00.0: 15.752 Gb/s available PCIe
bandwidth, limited by 8.0 GT/s PCIe x2 link at 0001:00:00.0 (capable of
31.506 Gb/s with 16.0 GT/s PCIe x2 link)
[   15.222286][ T1107] pci 0001:01:00.0: tudor: 1: pbus_size_mem: BAR 0
[mem 0x00000000-0x000fffff 64bit] list empty? 1
[   15.222813][ T1107] pci 0001:01:00.0: tudor: 1: pbus_size_mem: ROM
[mem 0x00000000-0x0000ffff pref] list empty? 1
[   15.224429][ T1107] pci 0001:01:00.0: tudor: 2: pbus_size_mem: ROM
[mem 0x00000000-0x0000ffff pref] list empty? 0
[   15.224750][ T1107] pcieport 0001:00:00.0: bridge window [mem
0x00100000-0x001fffff] to [bus 01-ff] add_size 100000 add_align 100000
Quoted 7 lines
[   15.225393][ T1107] tudor : pci_assign_unassigned_bus_resources:
before __pci_bus_assign_resources -> list empty? 0
[   15.225594][ T1107] pcieport 0001:00:00.0: tudor:
pdev_sort_resources: bridge window [mem 0x00100000-0x001fffff] resource
added in head list
[   15.226078][ T1107] pcieport 0001:00:00.0: bridge window [mem
0x40000000-0x401fffff]: assigned
So here it ends up assigning the resource here I think.
That print isn't one of yours in reassign_resources_sorted() so the assignment must have been made in assign_requested_resources_sorted(). But then nothing is printed out from reassign_resources_sorted() so I suspect __assign_resources_sorted() has short-circuited.
We know that realloc_head is not empty, so that leaves the goto out from if (list_empty(&local_fail_head)), which kind of makes sense, all entries on the head list were assigned. But the code there tries to remove all head list resources from realloc_head so why it doesn't get removed is still a mystery. assign_requested_resources_sorted() doesn't seem to remove anything from the head list so that resource should still be on the head list AFAICT so it should call that remove_from_list(realloc_head, dev_res->res) for it.

Quoted 2 lines

So can you see if that theory holds water and it short-circuits without removing the entry from realloc_head?

I think I figured out more about the reason. It's not related to that bridge window resource.

pbus_size_mem() will add also that ROM resource into realloc_head as it is considered (intentionally) optional after the optional change (as per "tudor: 2:" line). And that resource is never assigned because pdev_sort_resources() didn't pick it up into the head list. The next question is why the ROM resource isn't in the head list.

While it is not necessarily related to issue, I think the bridge sizing functions too should consider pdev_resources_assignable() so that it won't ever add resources from such devices onto the realloc_head. This is yet another small inconsistency within all this fitting/assignment logic.

pbus_size_mem() seems to consider IORESOURCE_PCI_FIXED so that cannot explain it as the ROM resource wouldn't be on the realloc_head list in that case.

Just wanted to let you know early even if I don't fully understand everything so you can hopefully avoid unnecessary debugging.

Quoted 70 lines

Quoted 49 lines
[   15.226419][ T1107] tudor : pci_assign_unassigned_bus_resources:
after __pci_bus_assign_resources -> list empty? 0
[   15.226442][ T1107] ------------[ cut here ]------------
[   15.227587][ T1107] kernel BUG at drivers/pci/setup-bus.c:2522!
[   15.227813][ T1107] Internal error: Oops - BUG: 00000000f2000800 [#1]
 SMP
...
[   15.251570][ T1107] Call trace:
[   15.251690][ T1107]  pci_assign_unassigned_bus_resources+0x110/0x114 (P)
[   15.251945][ T1107]  pci_rescan_bus+0x28/0x48
I obtain the following output when using the same prints adapted
(https://paste.ofcode.org/37w7RnKkPaCxyNhi5yhZPbZ) and with the blamed commits reverted:
a34d74877c66 PCI: Restore assigned resources fully after release 2499f5348431 PCI: Rework optional resource handling 96336ec70264 PCI: Perform reset_resource() and build fail list in sync
[   15.200456][ T1102] pci 0000:01:00.0: [144d:a5a5] type 00 class
0x000000 PCIe Endpoint
[   15.200632][ T1102] pci 0000:01:00.0: BAR 0 [mem
0x00000000-0x000fffff 64bit]
[   15.200755][ T1102] pci 0000:01:00.0: ROM [mem 0x00000000-0x0000ffff
pref]
[   15.200876][ T1102] pci 0000:01:00.0: enabling Extended Tags
[   15.201075][ T1102] pci 0000:01:00.0: PME# supported from D0 D3hot D3cold
[   15.201254][ T1102] pci 0000:01:00.0: 15.752 Gb/s available PCIe
bandwidth, limited by 8.0 GT/s PCIe x2 link at 0000:00:00.0 (capable of
31.506 Gb/s with 16.0 GT/s PCIe x2 link)
[   15.206555][ T1102] pci 0000:01:00.0: tudor: 1: pbus_size_mem: BAR 0
[mem 0x00000000-0x000fffff 64bit] list empty? 1
[   15.206737][ T1102] pci 0000:01:00.0: tudor: 1: pbus_size_mem: ROM
[mem 0x00000000-0x0000ffff pref] list empty? 1
[   15.206901][ T1102] tudor : pci_assign_unassigned_bus_resources:
before __pci_bus_assign_resources -> list empty? 1
[   15.207072][ T1102] pcieport 0000:00:00.0: tudor:
pdev_sort_resources: bridge window [mem 0x00100000-0x002fffff] resource
added in head list
[   15.207396][ T1102] pcieport 0000:00:00.0: bridge window [mem
0x40000000-0x401fffff]: assigned
[   15.208165][ T1102] tudor : pci_assign_unassigned_bus_resources:
after __pci_bus_assign_resources -> list empty? 1
[   15.208783][ T1102] cpif: s5100_poweron_pcie: DBG: MSI sfr not set
up, yet(s5100_pdev is NULL)
[   15.208786][ T1102] cpif: register_pcie: s51xx_pcie_init start
Quoted 3 lines
In any case, that BUG_ON() seems a bit drastic action for what might be just a single resource allocation failure so it should be downgraded to:
Quoted 7 lines
Quoted 4 lines
if (WARN_ON(!list_empty(&add_list))
	free_list(&add_list);
... or WARN_ON_ONCE().
I saw your patch doing this, the phone now boots, but obviously I still see the WARN, so maybe there's still something to be fixed.
Yes, I don't expect BUG_ON() -> WARN "fix" anything, it just downgrades the severity so that the system can still try to boot, which can often succeed as this tends to be non-critical failure in many cases so it's useful change to have regardless despite the splat.
Now that it boots, can you please check if /proc/iomem is the same both in the non-working and working config. If that resource got assigned successfully, it might well be there is no actual differences in the assigned resources (which again doesn't mean there wouldn't be a bug in the logic as discussed above).

Signature · 1 line

i.

Tudor Ambarus

2025-06-02 14:40 UTC #45 lore raw

Re: [PATCH 24/25] PCI: Perform reset_resource() and build fail list in sync

Quoted 50 lines · Ilpo Järvinen

Quoted 50 lines
Quoted 26 lines
I added the suggested prints
(https://paste.ofcode.org/DgmZGGgS6D36nWEzmfCqMm) on top of v6.15 with the downstream PCIe pixel driver and I obtain the following. Note that all added prints contain "tudor" for differentiation.
[   15.211179][ T1107] pci 0001:01:00.0: [144d:a5a5] type 00 class
0x000000 PCIe Endpoint
[   15.212248][ T1107] pci 0001:01:00.0: BAR 0 [mem
0x00000000-0x000fffff 64bit]
[   15.212775][ T1107] pci 0001:01:00.0: ROM [mem 0x00000000-0x0000ffff
pref]
[   15.213195][ T1107] pci 0001:01:00.0: enabling Extended Tags
[   15.213720][ T1107] pci 0001:01:00.0: PME# supported from D0 D3hot
D3cold
[   15.214035][ T1107] pci 0001:01:00.0: 15.752 Gb/s available PCIe
bandwidth, limited by 8.0 GT/s PCIe x2 link at 0001:00:00.0 (capable of
31.506 Gb/s with 16.0 GT/s PCIe x2 link)
[   15.222286][ T1107] pci 0001:01:00.0: tudor: 1: pbus_size_mem: BAR 0
[mem 0x00000000-0x000fffff 64bit] list empty? 1
[   15.222813][ T1107] pci 0001:01:00.0: tudor: 1: pbus_size_mem: ROM
[mem 0x00000000-0x0000ffff pref] list empty? 1
[   15.224429][ T1107] pci 0001:01:00.0: tudor: 2: pbus_size_mem: ROM
[mem 0x00000000-0x0000ffff pref] list empty? 0
[   15.224750][ T1107] pcieport 0001:00:00.0: bridge window [mem
0x00100000-0x001fffff] to [bus 01-ff] add_size 100000 add_align 100000
Quoted 7 lines
[   15.225393][ T1107] tudor : pci_assign_unassigned_bus_resources:
before __pci_bus_assign_resources -> list empty? 0
[   15.225594][ T1107] pcieport 0001:00:00.0: tudor:
pdev_sort_resources: bridge window [mem 0x00100000-0x001fffff] resource
added in head list
[   15.226078][ T1107] pcieport 0001:00:00.0: bridge window [mem
0x40000000-0x401fffff]: assigned
So here it ends up assigning the resource here I think.
That print isn't one of yours in reassign_resources_sorted() so the assignment must have been made in assign_requested_resources_sorted(). But then nothing is printed out from reassign_resources_sorted() so I suspect __assign_resources_sorted() has short-circuited.
We know that realloc_head is not empty, so that leaves the goto out from if (list_empty(&local_fail_head)), which kind of makes sense, all entries on the head list were assigned. But the code there tries to remove all head list resources from realloc_head so why it doesn't get removed is still a mystery. assign_requested_resources_sorted() doesn't seem to remove anything from the head list so that resource should still be on the head list AFAICT so it should call that remove_from_list(realloc_head, dev_res->res) for it.

Quoted 8 lines

Quoted 2 lines
So can you see if that theory holds water and it short-circuits without removing the entry from realloc_head?
I think I figured out more about the reason. It's not related to that bridge window resource.
pbus_size_mem() will add also that ROM resource into realloc_head as it is considered (intentionally) optional after the optional change (as per "tudor: 2:" line). And that resource is never assigned because

right, the ROM resource is added into realloc_head here:
https://web.git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/tree/drivers/pci/setup-bus.c#n1202

Then in the failing case, and extra resource is added:

[   15.224750][ T1107] pcieport 0001:00:00.0: bridge window [mem

0x00100000-0x001fffff] to [bus 01-ff] add_size 100000 add_align 100000

The above extra print happens just in the failing case. Here's where the extra resource is added:
https://web.git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/tree/drivers/pci/setup-bus.c#n1285

It seems that in the failing case 2 resources are added into realloc_head at the pbus_size_mem() time, whereas with the patch reverted - none.

Also, in the failing case a smaller resource is added into the list:
pdev_sort_resources: bridge window [mem 0x00100000-0x001fffff] compared to the working case:
pdev_sort_resources: bridge window [mem 0x00100000-0x002fffff]

Can this make a difference?

Quoted 3 lines

pdev_sort_resources() didn't pick it up into the head list. The next question is why the ROM resource isn't in the head list.

It seems the ROM resource is skipped at:
https://web.git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/tree/drivers/pci/setup-bus.c#n175

tudor: pdev_sort_resources: ROM [??? 0x00000000 flags 0x0] resource skipped due to !(r->flags) || r->parent

Quoted 6 lines

While it is not necessarily related to issue, I think the bridge sizing functions too should consider pdev_resources_assignable() so that it won't ever add resources from such devices onto the realloc_head. This is yet another small inconsistency within all this fitting/assignment logic.

Quoted 7 lines

pbus_size_mem() seems to consider IORESOURCE_PCI_FIXED so that cannot explain it as the ROM resource wouldn't be on the realloc_head list in that case.
Just wanted to let you know early even if I don't fully understand everything so you can hopefully avoid unnecessary debugging.

Thanks! Would adding some prints in pbus_size_mem() to describe the code paths in the working and non-working case help?

Cheers,
ta

Ilpo Järvinen

author

2025-06-02 15:11 UTC #46 lore raw

Re: [PATCH 24/25] PCI: Perform reset_resource() and build fail list in sync

Quoted 74 lines · Tudor Ambarus

Quoted 50 lines · Ilpo Järvinen
Quoted 50 lines
Quoted 26 lines
I added the suggested prints
(https://paste.ofcode.org/DgmZGGgS6D36nWEzmfCqMm) on top of v6.15 with the downstream PCIe pixel driver and I obtain the following. Note that all added prints contain "tudor" for differentiation.
[   15.211179][ T1107] pci 0001:01:00.0: [144d:a5a5] type 00 class
0x000000 PCIe Endpoint
[   15.212248][ T1107] pci 0001:01:00.0: BAR 0 [mem
0x00000000-0x000fffff 64bit]
[   15.212775][ T1107] pci 0001:01:00.0: ROM [mem 0x00000000-0x0000ffff
pref]
[   15.213195][ T1107] pci 0001:01:00.0: enabling Extended Tags
[   15.213720][ T1107] pci 0001:01:00.0: PME# supported from D0 D3hot
D3cold
[   15.214035][ T1107] pci 0001:01:00.0: 15.752 Gb/s available PCIe
bandwidth, limited by 8.0 GT/s PCIe x2 link at 0001:00:00.0 (capable of
31.506 Gb/s with 16.0 GT/s PCIe x2 link)
[   15.222286][ T1107] pci 0001:01:00.0: tudor: 1: pbus_size_mem: BAR 0
[mem 0x00000000-0x000fffff 64bit] list empty? 1
[   15.222813][ T1107] pci 0001:01:00.0: tudor: 1: pbus_size_mem: ROM
[mem 0x00000000-0x0000ffff pref] list empty? 1
[   15.224429][ T1107] pci 0001:01:00.0: tudor: 2: pbus_size_mem: ROM
[mem 0x00000000-0x0000ffff pref] list empty? 0
[   15.224750][ T1107] pcieport 0001:00:00.0: bridge window [mem
0x00100000-0x001fffff] to [bus 01-ff] add_size 100000 add_align 100000
Quoted 7 lines
[   15.225393][ T1107] tudor : pci_assign_unassigned_bus_resources:
before __pci_bus_assign_resources -> list empty? 0
[   15.225594][ T1107] pcieport 0001:00:00.0: tudor:
pdev_sort_resources: bridge window [mem 0x00100000-0x001fffff] resource
added in head list
[   15.226078][ T1107] pcieport 0001:00:00.0: bridge window [mem
0x40000000-0x401fffff]: assigned
So here it ends up assigning the resource here I think.
That print isn't one of yours in reassign_resources_sorted() so the assignment must have been made in assign_requested_resources_sorted(). But then nothing is printed out from reassign_resources_sorted() so I suspect __assign_resources_sorted() has short-circuited.
We know that realloc_head is not empty, so that leaves the goto out from if (list_empty(&local_fail_head)), which kind of makes sense, all entries on the head list were assigned. But the code there tries to remove all head list resources from realloc_head so why it doesn't get removed is still a mystery. assign_requested_resources_sorted() doesn't seem to remove anything from the head list so that resource should still be on the head list AFAICT so it should call that remove_from_list(realloc_head, dev_res->res) for it.
Quoted 8 lines
Quoted 2 lines
So can you see if that theory holds water and it short-circuits without removing the entry from realloc_head?
I think I figured out more about the reason. It's not related to that bridge window resource.
pbus_size_mem() will add also that ROM resource into realloc_head as it is considered (intentionally) optional after the optional change (as per "tudor: 2:" line). And that resource is never assigned because
right, the ROM resource is added into realloc_head here:
https://web.git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/tree/drivers/pci/setup-bus.c#n1202
Then in the failing case, and extra resource is added:
[   15.224750][ T1107] pcieport 0001:00:00.0: bridge window [mem
0x00100000-0x001fffff] to [bus 01-ff] add_size 100000 add_align 100000 The above extra print happens just in the failing case. Here's where the extra resource is added:
https://web.git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/tree/drivers/pci/setup-bus.c#n1285
It seems that in the failing case 2 resources are added into realloc_head at the pbus_size_mem() time, whereas with the patch reverted - none.

Quoted 6 lines

Also, in the failing case a smaller resource is added into the list:
pdev_sort_resources: bridge window [mem 0x00100000-0x001fffff] compared to the working case:
pdev_sort_resources: bridge window [mem 0x00100000-0x002fffff]
Can this make a difference?

I don't think the bridge window resource different matters, that bridge resource got assigned just fine so it is a red herring. The code just calculates optional sizes in different place after the rework than before it. The successful assignment itself is for the same size so there's nothing wrong there AFAICT.

Quoted 9 lines

Quoted 3 lines
pdev_sort_resources() didn't pick it up into the head list. The next question is why the ROM resource isn't in the head list.
It seems the ROM resource is skipped at:
https://web.git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/tree/drivers/pci/setup-bus.c#n175
tudor: pdev_sort_resources: ROM [??? 0x00000000 flags 0x0] resource skipped due to !(r->flags) || r->parent

I don't see the device in this print, hope it is for the same device.

In any case, I don't understand what reset resource's flags in between pbus_size_mem() and pdev_sort_resources(), or alternative, why type checking in pbus_size_mem() matches if flags == 0 at that point.

Those two functions should work on the same resources, if one skips something, the other should too. Disparity between them can cause issues, but despite reading the code multiple times, I couldn't figure out how that disparity occurs (except for the !pdev_resources_assignable() case).

Quoted 10 lines

Quoted 10 lines
While it is not necessarily related to issue, I think the bridge sizing functions too should consider pdev_resources_assignable() so that it won't ever add resources from such devices onto the realloc_head. This is yet another small inconsistency within all this fitting/assignment logic.
pbus_size_mem() seems to consider IORESOURCE_PCI_FIXED so that cannot explain it as the ROM resource wouldn't be on the realloc_head list in that case.

Quoted 5 lines

Quoted 2 lines
Just wanted to let you know early even if I don't fully understand everything so you can hopefully avoid unnecessary debugging.
Thanks! Would adding some prints in pbus_size_mem() to describe the code paths in the working and non-working case help?

It is of interest to know why the same resource is treated differently. So what were the resource flags, type* args when it's processed by pbus_size_mem()? If resource's flags are zero at that point but it matches one of the types, that would be a bug.

Signature · 1 line

i.

Tudor Ambarus

2025-06-02 18:43 UTC #47 lore raw

Re: [PATCH 24/25] PCI: Perform reset_resource() and build fail list in sync

Quoted 6 lines · Ilpo Järvinen

Quoted 6 lines
Quoted 6 lines
I think I figured out more about the reason. It's not related to that bridge window resource.
pbus_size_mem() will add also that ROM resource into realloc_head as it is considered (intentionally) optional after the optional change (as per "tudor: 2:" line). And that resource is never assigned because

cut

Quoted 19 lines

Quoted 9 lines
Quoted 3 lines
pdev_sort_resources() didn't pick it up into the head list. The next question is why the ROM resource isn't in the head list.
It seems the ROM resource is skipped at:
https://web.git.kernel.org/pub/scm/linux/kernel/git/next/linux-
next.git/tree/drivers/pci/setup-bus.c#n175
tudor: pdev_sort_resources: ROM [??? 0x00000000 flags 0x0] resource skipped due to !(r->flags) || r->parent
I don't see the device in this print, hope it is for the same device.
In any case, I don't understand what reset resource's flags in between pbus_size_mem() and pdev_sort_resources(), or alternative, why type checking in pbus_size_mem() matches if flags == 0 at that point.
Those two functions should work on the same resources, if one skips something, the other should too. Disparity between them can cause issues, but despite reading the code multiple times, I couldn't figure out how that disparity occurs (except for the !pdev_resources_assignable() case).

cut

Quoted 3 lines

It is of interest to know why the same resource is treated differently. So what were the resource flags, type* args when it's processed by pbus_size_mem()? If resource's flags are zero at that point but it matches

This is the full output: https://termbin.com/mn1x
for the following prints: https://termbin.com/q57h

It seems ROM resource is of type 2 at pbus_size_mem() time.

Quoted 1 line

one of the types, that would be a bug.

I'll give another try tomorrow. Thanks,
ta

Ilpo Järvinen

author

2025-06-03 08:13 UTC #48 lore raw

Re: [PATCH 24/25] PCI: Perform reset_resource() and build fail list in sync

Quoted 36 lines · Tudor Ambarus

Quoted 6 lines · Ilpo Järvinen
Quoted 6 lines
Quoted 6 lines
I think I figured out more about the reason. It's not related to that bridge window resource.
pbus_size_mem() will add also that ROM resource into realloc_head as it is considered (intentionally) optional after the optional change (as per "tudor: 2:" line). And that resource is never assigned because
cut
Quoted 19 lines
Quoted 9 lines
Quoted 3 lines
pdev_sort_resources() didn't pick it up into the head list. The next question is why the ROM resource isn't in the head list.
It seems the ROM resource is skipped at:
https://web.git.kernel.org/pub/scm/linux/kernel/git/next/linux-
next.git/tree/drivers/pci/setup-bus.c#n175
tudor: pdev_sort_resources: ROM [??? 0x00000000 flags 0x0] resource skipped due to !(r->flags) || r->parent
I don't see the device in this print, hope it is for the same device.
In any case, I don't understand what reset resource's flags in between pbus_size_mem() and pdev_sort_resources(), or alternative, why type checking in pbus_size_mem() matches if flags == 0 at that point.
Those two functions should work on the same resources, if one skips something, the other should too. Disparity between them can cause issues, but despite reading the code multiple times, I couldn't figure out how that disparity occurs (except for the !pdev_resources_assignable() case).
cut
Quoted 3 lines
It is of interest to know why the same resource is treated differently. So what were the resource flags, type* args when it's processed by pbus_size_mem()? If resource's flags are zero at that point but it matches

Quoted 8 lines

This is the full output: https://termbin.com/mn1x
for the following prints: https://termbin.com/q57h
It seems ROM resource is of type 2 at pbus_size_mem() time.
Quoted 1 line
one of the types, that would be a bug.
I'll give another try tomorrow. Thanks,

Those are not the same device, so not the same resource either:

[   16.262745][ T1113] pci 0001:01:00.0: tudor: 2: pbus_size_mem: ROM [mem 0x00000000-0x0000ffff pref] list empty? 0

[   16.267736][ T1113] pcieport 0001:00:00.0: tudor: pdev_sort_resources: ROM [??? 0x00000000 flags 0x0] resource skipped due to !(r->flags) || r->parent

0001:01:00.0
vs
0001:00:00.0

And the resources for 0001:01:00.0 were never processed by __pci_bus_assign_resources(). But __pci_bus_assign_resources() should recurse to subordinate busses.

And, it seems this boils down to the inconsistency I noticed earlier:

[   16.253464][ T1113] pci 0001:01:00.0: [144d:a5a5] type 00 class 0x000000 PCIe Endpoint

include/linux/pci_ids.h:#define PCI_CLASS_NOT_DEFINED 0x0000

pdev_resources_assignable() checks if class == PCI_CLASS_NOT_DEFINED and pdev_sort_resources() bails out without processing those resources.

So please test if this patch solves your problem:

From 00e440f505a79568cf5203dce265488cb3f66941 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Ilpo=20J=C3=A4rvinen?= <[email protected]>
Date: Tue, 3 Jun 2025 11:07:38 +0300
Subject: [PATCH 1/1] PCI: Fix pdev_resources_assignable() disparity
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

pdev_sort_resources() uses pdev_resources_assignable() helper to decide if device's resources cannot be assigned. pbus_size_mem(), on the other hand, does not do the same check. This could lead into a situation where a resource ends up on realloc_head list but is not on the head list, which is turn prevents emptying the resource from the realloc_head list in __assign_resources_sorted().

A non-empty realloc_head is unacceptable because it triggers an internal sanity check as show in this log with a device that has class 0 (PCI_CLASS_NOT_DEFINED):

pci 0001:01:00.0: [144d:a5a5] type 00 class 0x000000 PCIe Endpoint
pci 0001:01:00.0: BAR 0 [mem 0x00000000-0x000fffff 64bit]
pci 0001:01:00.0: ROM [mem 0x00000000-0x0000ffff pref]
pci 0001:01:00.0: enabling Extended Tags
pci 0001:01:00.0: PME# supported from D0 D3hot D3cold
pci 0001:01:00.0: 15.752 Gb/s available PCIe bandwidth, limited by 8.0 GT/s PCIe x2 link at 0001:00:00.0 (capable of 31.506 Gb/s with 16.0 GT/s PCIe x2 link) pcieport 0001:00:00.0: bridge window [mem 0x00100000-0x001fffff] to [bus 01-ff] add_size 100000 add_align 100000
pcieport 0001:00:00.0: bridge window [mem 0x40000000-0x401fffff]: assigned

------------[ cut here ]------------
kernel BUG at drivers/pci/setup-bus.c:2532!
Internal error: Oops - BUG: 00000000f2000800 [#1]  SMP
...
Call trace:
 pci_assign_unassigned_bus_resources+0x110/0x114 (P)
 pci_rescan_bus+0x28/0x48

Use pdev_resources_assignable() also within pbus_size_mem() to skip processing of non-assignable resources which removes the disparity in between what resources pdev_sort_resources() and pbus_size_mem() consider. As non-assignable resources are no longer processed, they are not added to the realloc_head list, thus the sanity check no longer triggers.

This disparity problem is very old but only now became apparent after the commit 2499f5348431 ("PCI: Rework optional resource handling") that made the ROM resources optional when calculating bridge window sizes which required adding the resource to the realloc_head list. Previously, bridge windows were just sized larger than necessary.

Fixes 2499f5348431 ("PCI: Rework optional resource handling")

Cc [email protected]

Signed-off-by Ilpo Järvinen <[email protected]>

---
 drivers/pci/setup-bus.c | 1 +
 1 file changed, 1 insertion(+)

drivers/pci/setup-bus.c +1 -0

index 54d6f4fa3ce1..da084251df43 100644
@@ -1187,6 +1187,7 @@ static int pbus_size_mem(struct pci_bus *bus, unsigned long mask,
 			resource_size_t r_size;
 
 			if (r->parent || (r->flags & IORESOURCE_PCI_FIXED) ||
+			    !pdev_resources_assignable(dev) ||
 			    ((r->flags & mask) != type &&
 			     (r->flags & mask) != type2 &&
 			     (r->flags & mask) != type3))

base-commit: 0af2f6be1b4281385b618cb86ad946eded089ac8

Signature · 1 line

2.39.5

Tudor Ambarus

2025-06-03 10:36 UTC #49 lore raw

Re: [PATCH 24/25] PCI: Perform reset_resource() and build fail list in sync

Quoted 1 line · Ilpo Järvinen

So please test if this patch solves your problem:

It fails in a different way, the bridge window resource never gets assigned with the proposed patch.

With the patch applied: https://termbin.com/h3w0
With the blamed commit reverted: https://termbin.com/3rh6

Tudor Ambarus

2025-06-03 10:48 UTC #50 lore raw

Re: [PATCH 24/25] PCI: Perform reset_resource() and build fail list in sync

Quoted 9 lines · Tudor Ambarus

Quoted 1 line · Ilpo Järvinen
So please test if this patch solves your problem:
It fails in a different way, the bridge window resource never gets assigned with the proposed patch.
With the patch applied: https://termbin.com/h3w0

above is no revert and with the proposed fix. It also contains the prints https://termbin.com/g4zn

It seems the prints in pbus_size_mem are not longer hit, likely because of the new condition added: ``!pdev_resources_assignable(dev) ||``, pci_dev_for_each_resource() finishes without doing anything.

Quoted 1 line

With the blamed commit reverted: https://termbin.com/3rh6

Tudor Ambarus

2025-06-03 11:43 UTC #51 lore raw

Re: [PATCH 24/25] PCI: Perform reset_resource() and build fail list in sync

Quoted 11 lines · Tudor Ambarus

Quoted 8 lines · Tudor Ambarus
Quoted 1 line · Ilpo Järvinen
So please test if this patch solves your problem:
It fails in a different way, the bridge window resource never gets assigned with the proposed patch.

Quoted 11 lines

Quoted 1 line
With the patch applied: https://termbin.com/h3w0
above is no revert and with the proposed fix. It also contains the prints https://termbin.com/g4zn
It seems the prints in pbus_size_mem are not longer hit, likely because of the new condition added: ``!pdev_resources_assignable(dev) ||``, pci_dev_for_each_resource() finishes without doing anything.
Quoted 1 line
With the blamed commit reverted: https://termbin.com/3rh6

I think I found the inconsistency.

__pci_bus_assign_resources()

	pbus_assign_resources_sorted()
		pdev_sort_resources(dev, &head);

But pdev_sort_resources() is called with a newly LIST_HEAD(head), not with realloc_head, thus the resources never get sorted.

pdev_sort_resources() exits early at

	``if (!pdev_resources_assignable(dev))``

Ilpo Järvinen

author

2025-06-03 14:23 UTC #52 lore raw

Re: [PATCH 24/25] PCI: Perform reset_resource() and build fail list in sync

Quoted 20 lines · Tudor Ambarus

Quoted 7 lines · Tudor Ambarus
Quoted 6 lines · Tudor Ambarus
Quoted 1 line · Ilpo Järvinen
So please test if this patch solves your problem:
It fails in a different way, the bridge window resource never gets assigned with the proposed patch.
Quoted 11 lines
Quoted 1 line
With the patch applied: https://termbin.com/h3w0
above is no revert and with the proposed fix. It also contains the prints https://termbin.com/g4zn
It seems the prints in pbus_size_mem are not longer hit, likely because of the new condition added: ``!pdev_resources_assignable(dev) ||``, pci_dev_for_each_resource() finishes without doing anything.
Quoted 1 line
With the blamed commit reverted: https://termbin.com/3rh6

Quoted 8 lines

I think I found the inconsistency.
__pci_bus_assign_resources()
	pbus_assign_resources_sorted()
		pdev_sort_resources(dev, &head);
But pdev_sort_resources() is called with a newly LIST_HEAD(head), not with realloc_head, thus the resources never get sorted.

pdev_sort_resources() is not supposed to add resources into realloc_head but just collects all the relevant resources in the descending order by size.

There are two main lists here. The head list contains all relevant resources we're going to process and realloc_head keeps track which of them are optional (or optional in part, that is, some resources have the base size and the optional size).

__assign_resources_sorted() will apply the optional sizes from realloc_head and re-sorts the head list while changing the sizes. If not all resources can be assigned, rollback happens and base sizes are assigned first, and then reassign_resources_sorted() handles the realloc_head ones afterwards for as many resources as possible.

Quoted 2 lines

pdev_sort_resources() exits early at
	``if (!pdev_resources_assignable(dev))``

Yes it does, for 0001:01:00.0.

Signature · 1 line

i.

Ilpo Järvinen

author

2025-06-03 14:43 UTC #53 lore raw

Re: [PATCH 24/25] PCI: Perform reset_resource() and build fail list in sync

Quoted 27 lines · Ilpo Järvinen

Quoted 26 lines · Tudor Ambarus
Quoted 7 lines · Tudor Ambarus
Quoted 6 lines · Tudor Ambarus
Quoted 1 line · Ilpo Järvinen
So please test if this patch solves your problem:
It fails in a different way, the bridge window resource never gets assigned with the proposed patch.
Quoted 11 lines
Quoted 1 line
With the patch applied: https://termbin.com/h3w0
above is no revert and with the proposed fix. It also contains the prints https://termbin.com/g4zn
It seems the prints in pbus_size_mem are not longer hit, likely because of the new condition added: ``!pdev_resources_assignable(dev) ||``, pci_dev_for_each_resource() finishes without doing anything.
Quoted 1 line
With the blamed commit reverted: https://termbin.com/3rh6
I think I found the inconsistency.
__pci_bus_assign_resources()
	pbus_assign_resources_sorted()
		pdev_sort_resources(dev, &head);

Quoted 6 lines

Quoted 2 lines
But pdev_sort_resources() is called with a newly LIST_HEAD(head), not with realloc_head, thus the resources never get sorted.
pdev_sort_resources() is not supposed to add resources into realloc_head but just collects all the relevant resources in the descending order by size.

Small correction, they're ordered by alignment, not by size. For other than iov resources and bridge window, alignment order effectively the same as size order.

Quoted 17 lines

There are two main lists here. The head list contains all relevant resources we're going to process and realloc_head keeps track which of them are optional (or optional in part, that is, some resources have the base size and the optional size).
__assign_resources_sorted() will apply the optional sizes from realloc_head and re-sorts the head list while changing the sizes. If not all resources can be assigned, rollback happens and base sizes are assigned first, and then reassign_resources_sorted() handles the realloc_head ones afterwards for as many resources as possible.
Quoted 2 lines
pdev_sort_resources() exits early at
	``if (!pdev_resources_assignable(dev))``
Yes it does, for 0001:01:00.0.

Signature · 1 line

i.

Ilpo Järvinen

author

2025-06-03 14:13 UTC #54 lore raw

Re: [PATCH 24/25] PCI: Perform reset_resource() and build fail list in sync

In reply to Tudor Ambarus

Quoted 5 lines · Tudor Ambarus

Quoted 1 line · Ilpo Järvinen
So please test if this patch solves your problem:
It fails in a different way, the bridge window resource never gets assigned with the proposed patch.

Is that a failure? I was expecting that to occur. It didn't assign any resources into that bridge window.

If there's nothing to be assigned into the bridge window, the bridge window itself is not created, that is the expected behavior (working as designed). So you're comparing to the bridge window that was made too large due to the disparity (and left unused, AFAICT).

It would be possible to put the condition inside the block which adds the resource to the realloc_head, I initially put it there but then decided to remove the disparity completely because why keep it if no resource is going to be placed into the bridge window.

What's that class 0 device anyway? Why it has class 0?

Quoted 2 lines

With the patch applied: https://termbin.com/h3w0
With the blamed commit reverted: https://termbin.com/3rh6

Signature · 1 line

i.

Tudor Ambarus

2025-06-03 15:25 UTC #55 lore raw

Re: [PATCH 24/25] PCI: Perform reset_resource() and build fail list in sync

Quoted 9 lines · Ilpo Järvinen

Quoted 5 lines · Tudor Ambarus
Quoted 1 line · Ilpo Järvinen
So please test if this patch solves your problem:
It fails in a different way, the bridge window resource never gets assigned with the proposed patch.
Is that a failure? I was expecting that to occur. It didn't assign any resources into that bridge window.

It leads to a watchdog interrupt on my pixel6. Last print I see on my console is related to the modem booting status. My wild guess is that that modem accesses something from the unassigned bridge window.

In the working case I see the bridge window printed:

[   15.457310][ T1083] pcieport 0000:00:00.0: [s51xx_pcie_probe] BAR 14:

tmp rsc : [mem 0x40000000-0x401fffff]

[   15.457683][ T1083] cpif: s51xx_pcie_probe: Set Doorbell register

address.

In the failing case I see:

[   15.623270][ T1113] pcieport 0000:00:00.0: [s51xx_pcie_probe] BAR 14:

tmp rsc : [??? 0x00000000 flags 0x0]

[   15.623638][ T1113] cpif: s51xx_pcie_probe: Set Doorbell register

address.

Quoted 11 lines

If there's nothing to be assigned into the bridge window, the bridge window itself is not created, that is the expected behavior (working as designed). So you're comparing to the bridge window that was made too large due to the disparity (and left unused, AFAICT).
It would be possible to put the condition inside the block which adds the resource to the realloc_head, I initially put it there but then decided to remove the disparity completely because why keep it if no resource is going to be placed into the bridge window.

Thanks for the educative answers.

Quoted 2 lines

What's that class 0 device anyway? Why it has class 0?

I don't know yet, it's the first time I'm dealing with a PCI driver. Any idea where is the class typically assigned?

Quoted 3 lines

Quoted 2 lines
With the patch applied: https://termbin.com/h3w0
With the blamed commit reverted: https://termbin.com/3rh6

Ilpo Järvinen

author

2025-06-03 17:03 UTC #56 lore raw

Re: [PATCH 24/25] PCI: Perform reset_resource() and build fail list in sync

Quoted 8 lines · Tudor Ambarus

Quoted 7 lines · Ilpo Järvinen
Quoted 5 lines · Tudor Ambarus
Quoted 1 line · Ilpo Järvinen
So please test if this patch solves your problem:
It fails in a different way, the bridge window resource never gets assigned with the proposed patch.

Quoted 6 lines

Quoted 2 lines
Is that a failure? I was expecting that to occur. It didn't assign any resources into that bridge window.
It leads to a watchdog interrupt on my pixel6. Last print I see on my console is related to the modem booting status. My wild guess is that that modem accesses something from the unassigned bridge window.

The bridge window is not for the bridge device itself. It's as the name says, a window into where subordinate busses can assign their resources. The bridge knows it must forward that window address range to the subordinate bus.

Quoted 5 lines

In the working case I see the bridge window printed:
[   15.457310][ T1083] pcieport 0000:00:00.0: [s51xx_pcie_probe] BAR 14:
tmp rsc : [mem 0x40000000-0x401fffff]
[   15.457683][ T1083] cpif: s51xx_pcie_probe: Set Doorbell register

Quoted 8 lines

address.
In the failing case I see:
[   15.623270][ T1113] pcieport 0000:00:00.0: [s51xx_pcie_probe] BAR 14:
tmp rsc : [??? 0x00000000 flags 0x0]
[   15.623638][ T1113] cpif: s51xx_pcie_probe: Set Doorbell register
address.

Oh, is it this one?

https://github.com/oberdfr/google-modules_radio_samsung_s5300/blob/11a10f955a267a45a1997f65671d7054adf1a33a/s51xx_pcie.c#L366

There are number of crazy things going on there... Probe shouldn't be messing resources like that. If it wants to change resources, a quirk would be more appropriate place I guess but I'm very unsure what that even tries to achieve with all that craziness ("Disable BAR resources" by assigning them :-/).

But yes, it seems to take the bridge window's address and assumes something is there (which isn't there as we know). So this driver code is plain wrong.

Perhaps it would want to use the address of some endpoint device resource instead of the bridge window address (e.g., that device with class 0?).

Quoted 10 lines

Quoted 10 lines
If there's nothing to be assigned into the bridge window, the bridge window itself is not created, that is the expected behavior (working as designed). So you're comparing to the bridge window that was made too large due to the disparity (and left unused, AFAICT).
It would be possible to put the condition inside the block which adds the resource to the realloc_head, I initially put it there but then decided to remove the disparity completely because why keep it if no resource is going to be placed into the bridge window.

Quoted 6 lines

Thanks for the educative answers.
Quoted 2 lines
What's that class 0 device anyway? Why it has class 0?
I don't know yet, it's the first time I'm dealing with a PCI driver. Any idea where is the class typically assigned?

https://pcisig.com/sites/default/files/files/PCI_Code-ID_r_1_12__v9_Jan_2020.pdf

Perhaps try a quirk which changes the class of the device underneath the bridge to something else than 0, it should make the resource fitting and allocation to assign its resources.

But honestly, that s51xx_pcie_probe() has more than one thing wrong.

Quoted 4 lines

Quoted 3 lines
Quoted 2 lines
With the patch applied: https://termbin.com/h3w0
With the blamed commit reverted: https://termbin.com/3rh6

Signature · 1 line

i.

Ilpo Järvinen

author

2025-06-03 17:09 UTC #57 lore raw

Re: [PATCH 24/25] PCI: Perform reset_resource() and build fail list in sync

Quoted 37 lines · Ilpo Järvinen

Quoted 8 lines · Tudor Ambarus
Quoted 7 lines · Ilpo Järvinen
Quoted 5 lines · Tudor Ambarus
Quoted 1 line · Ilpo Järvinen
So please test if this patch solves your problem:
It fails in a different way, the bridge window resource never gets assigned with the proposed patch.
Quoted 6 lines
Quoted 2 lines
Is that a failure? I was expecting that to occur. It didn't assign any resources into that bridge window.
It leads to a watchdog interrupt on my pixel6. Last print I see on my console is related to the modem booting status. My wild guess is that that modem accesses something from the unassigned bridge window.
The bridge window is not for the bridge device itself. It's as the name says, a window into where subordinate busses can assign their resources. The bridge knows it must forward that window address range to the subordinate bus.
Quoted 5 lines
In the working case I see the bridge window printed:
[   15.457310][ T1083] pcieport 0000:00:00.0: [s51xx_pcie_probe] BAR 14:
tmp rsc : [mem 0x40000000-0x401fffff]
[   15.457683][ T1083] cpif: s51xx_pcie_probe: Set Doorbell register
Quoted 8 lines
address.
In the failing case I see:
[   15.623270][ T1113] pcieport 0000:00:00.0: [s51xx_pcie_probe] BAR 14:
tmp rsc : [??? 0x00000000 flags 0x0]
[   15.623638][ T1113] cpif: s51xx_pcie_probe: Set Doorbell register
address.
Oh, is it this one?

Quoted 7 lines

https://github.com/oberdfr/google-modules_radio_samsung_s5300/blob/11a10f955a267a45a1997f65671d7054adf1a33a/s51xx_pcie.c#L366
There are number of crazy things going on there... Probe shouldn't be messing resources like that. If it wants to change resources, a quirk would be more appropriate place I guess but I'm very unsure what that even tries to achieve with all that craziness ("Disable BAR resources" by assigning them :-/).

Or maybe DT, I'm not very familiar with DT things.

--
i.

Quoted 38 lines

But yes, it seems to take the bridge window's address and assumes something is there (which isn't there as we know). So this driver code is plain wrong.
Perhaps it would want to use the address of some endpoint device resource instead of the bridge window address (e.g., that device with class 0?).
Quoted 10 lines
Quoted 10 lines
If there's nothing to be assigned into the bridge window, the bridge window itself is not created, that is the expected behavior (working as designed). So you're comparing to the bridge window that was made too large due to the disparity (and left unused, AFAICT).
It would be possible to put the condition inside the block which adds the resource to the realloc_head, I initially put it there but then decided to remove the disparity completely because why keep it if no resource is going to be placed into the bridge window.
Quoted 6 lines
Thanks for the educative answers.
Quoted 2 lines
What's that class 0 device anyway? Why it has class 0?
I don't know yet, it's the first time I'm dealing with a PCI driver. Any idea where is the class typically assigned?
https://pcisig.com/sites/default/files/files/PCI_Code-ID_r_1_12__v9_Jan_2020.pdf
Perhaps try a quirk which changes the class of the device underneath the bridge to something else than 0, it should make the resource fitting and allocation to assign its resources.
But honestly, that s51xx_pcie_probe() has more than one thing wrong.
Quoted 4 lines
Quoted 3 lines
Quoted 2 lines
With the patch applied: https://termbin.com/h3w0
With the blamed commit reverted: https://termbin.com/3rh6

Tudor Ambarus

2025-06-02 12:32 UTC #58 lore raw

Re: [PATCH 24/25] PCI: Perform reset_resource() and build fail list in sync

In reply to Ilpo Järvinen

On 5/30/25 7:38 AM, Ilpo Järvinen wrote:

cut

Quoted 11 lines

Quoted 9 lines
Quoted 7 lines
Quoted 4 lines
Reverting the following patches fixes the problem:
a34d74877c66 PCI: Restore assigned resources fully after release 2499f5348431 PCI: Rework optional resource handling 96336ec70264 PCI: Perform reset_resource() and build fail list in sync
So it's confirmed that you needed to revert also this last commit 96336ec70264, not just the rework change?
I needed to revert 96336ec70264 as well otherwise the build fails.
Hi again,

Hi!

cut

Quoted 2 lines

The missing helper is basically this:

cut
I used the following:

+static bool pci_resource_is_disabled_rom(const struct resource *res, int resno)
+{
+ return resno == PCI_ROM_RESOURCE && !(res->flags &
IORESOURCE_ROM_ENABLE);
+}

Quoted 4 lines

Because of this, the actual culprit could be in 2499f5348431, not it 96336ec70264 (which would make more sense as it does significant rework on the assignment algorithm).

I confirm with the above that the problem is in 2499f5348431 indeed.

cut

Quoted 51 lines

Quoted 26 lines
I added the suggested prints
(https://paste.ofcode.org/DgmZGGgS6D36nWEzmfCqMm) on top of v6.15 with the downstream PCIe pixel driver and I obtain the following. Note that all added prints contain "tudor" for differentiation.
[   15.211179][ T1107] pci 0001:01:00.0: [144d:a5a5] type 00 class
0x000000 PCIe Endpoint
[   15.212248][ T1107] pci 0001:01:00.0: BAR 0 [mem
0x00000000-0x000fffff 64bit]
[   15.212775][ T1107] pci 0001:01:00.0: ROM [mem 0x00000000-0x0000ffff
pref]
[   15.213195][ T1107] pci 0001:01:00.0: enabling Extended Tags
[   15.213720][ T1107] pci 0001:01:00.0: PME# supported from D0 D3hot
D3cold
[   15.214035][ T1107] pci 0001:01:00.0: 15.752 Gb/s available PCIe
bandwidth, limited by 8.0 GT/s PCIe x2 link at 0001:00:00.0 (capable of
31.506 Gb/s with 16.0 GT/s PCIe x2 link)
[   15.222286][ T1107] pci 0001:01:00.0: tudor: 1: pbus_size_mem: BAR 0
[mem 0x00000000-0x000fffff 64bit] list empty? 1
[   15.222813][ T1107] pci 0001:01:00.0: tudor: 1: pbus_size_mem: ROM
[mem 0x00000000-0x0000ffff pref] list empty? 1
[   15.224429][ T1107] pci 0001:01:00.0: tudor: 2: pbus_size_mem: ROM
[mem 0x00000000-0x0000ffff pref] list empty? 0
[   15.224750][ T1107] pcieport 0001:00:00.0: bridge window [mem
0x00100000-0x001fffff] to [bus 01-ff] add_size 100000 add_align 100000
Quoted 7 lines
[   15.225393][ T1107] tudor : pci_assign_unassigned_bus_resources:
before __pci_bus_assign_resources -> list empty? 0
[   15.225594][ T1107] pcieport 0001:00:00.0: tudor:
pdev_sort_resources: bridge window [mem 0x00100000-0x001fffff] resource
added in head list
[   15.226078][ T1107] pcieport 0001:00:00.0: bridge window [mem
0x40000000-0x401fffff]: assigned
So here it ends up assigning the resource here I think.
That print isn't one of yours in reassign_resources_sorted() so the assignment must have been made in assign_requested_resources_sorted(). But then nothing is printed out from reassign_resources_sorted() so I suspect __assign_resources_sorted() has short-circuited.
We know that realloc_head is not empty, so that leaves the goto out from if (list_empty(&local_fail_head)), which kind of makes sense, all entries on the head list were assigned. But the code there tries to remove all head list resources from realloc_head so why it doesn't get removed is still a mystery. assign_requested_resources_sorted() doesn't seem to remove anything from the head list so that resource should still be on the head list AFAICT so it should call that remove_from_list(realloc_head, dev_res->res) for it.

Quoted 3 lines

So can you see if that theory holds water and it short-circuits without removing the entry from realloc_head?

cut. I saw your other reply. Will check a bit both and respond there directly.

Quoted 12 lines

Quoted 11 lines
Quoted 8 lines
In any case, that BUG_ON() seems a bit drastic action for what might be just a single resource allocation failure so it should be downgraded to:
if (WARN_ON(!list_empty(&add_list))
	free_list(&add_list);
... or WARN_ON_ONCE().
I saw your patch doing this, the phone now boots, but obviously I still see the WARN, so maybe there's still something to be fixed.

cut

Quoted 5 lines

Now that it boots, can you please check if /proc/iomem is the same both in the non-working and working config. If that resource got assigned successfully, it might well be there is no actual differences in the assigned resources (which again doesn't mean there wouldn't be a bug in the logic as discussed above).

I confirm /proc/iomem is identical when comparing the no revert and the WARN_ON_ONCE() case, and when reverting the blamed commit case.

D Scott Phillips

2025-06-19 00:30 UTC #59 lore raw

Re: [PATCH 24/25] PCI: Perform reset_resource() and build fail list in sync

In reply to Ilpo Järvinen

Quoted 18 lines · Ilpo Järvinen

Resetting resource is problematic as it prevent attempting to allocate the resource later, unless something in between restores the resource. Similarly, if fail_head does not contain all resources that were reset, those resource cannot be restored later.
The entire reset/restore cycle adds complexity and leaving resources into reseted state causes issues to other code such as for checks done in pci_enable_resources(). Take a small step towards not resetting resources by delaying reset until the end of resource assignment and build failure list (fail_head) in sync with the reset to avoid leaving behind resources that cannot be restored (for the case where the caller provides fail_head in the first place to allow restore somewhere in the callchain, as is not all callers pass non-NULL fail_head).
The Expansion ROM check is temporarily left in place while building the failure list until the upcoming change which reworks optional resource handling.

Quoted 5 lines

Ideally, whole resource reset could be removed but doing that in a big step would make the impact non-tractable due to complexity of all related code.
Signed-off-by Ilpo Järvinen <[email protected]>

Hi Ilpo, I'm seeing a crash on arm64 at boot that I bisected to this change. I don't think it's the same as the other issues reported. I've confirmed the crash is still there after your follow up patches. The crash itself is below[1].

It looks like the problem begins when:

amdgpu_device_resize_fb_bar()
pci_resize_resource()
pci_reassign_bridge_resources()
__pci_bus_size_bridges()

adds pci_hotplug_io_size to realloc_head. The io resource allocation has failed earlier because the root port doesn't have an io window[2].

Then with this patch, pci_reassign_bridge_resources()'s call to __pci_bridge_assign_resources() now returns the io added space for hotplug in the failed list where the old code dropped it and did not.

That sends pci_reassign_bridge_resources() into the cleanup: path, where I think the cleanup code doesn't properly release the resources that were assigned by __pci_bridge_assign_resources() and there's a conflict reported in pci_claim_resource() where a restored resource is found as conflicting with itself:

Quoted 1 line

pcieport 000d:00:01.0: bridge window [mem 0x340000000000-0x340017ffffff 64bit pref]: can't claim; address conflict with PCI Bus 000d:01 [mem 0x340000000000-0x340017ffffff 64bit pref]

Setting pci=hpiosize=0 avoids this crash, as does this change:

drivers/pci/setup-bus.c +11 -7

index 16d5d390599a..59ece11702da 100644
@@ -2442,7 +2442,7 @@ int pci_reassign_bridge_resources(struct pci_dev *bridge, unsigned long type)
 	LIST_HEAD(saved);
 	LIST_HEAD(added);
 	LIST_HEAD(failed);
-	unsigned int i;
+	unsigned int i, relevant_fails;
 	int ret;
 
 	down_read(&pci_bus_sem);

@@ -2490,7 +2490,16 @@ int pci_reassign_bridge_resources(struct pci_dev *bridge, unsigned long type)
 	__pci_bridge_assign_resources(bridge, &added, &failed);
 	BUG_ON(!list_empty(&added));
 
-	if (!list_empty(&failed)) {
+	relevant_fails = 0;
+	list_for_each_entry(dev_res, &failed, list) {
+		restore_dev_resource(dev_res);
+		if (((dev_res->res->flags ^ type) & PCI_RES_TYPE_MASK) == 0)
+			relevant_fails++;
+	}
+	free_list(&failed);
+
+	/* Cleanup if we had failures in resources of interest */
+	if (relevant_fails != 0) {
 		ret = -ENOSPC;
 		goto cleanup;
 	}

@@ -2509,11 +2518,6 @@ int pci_reassign_bridge_resources(struct pci_dev *bridge, unsigned long type)
 	return 0;
 
 cleanup:
-	/* Restore size and flags */
-	list_for_each_entry(dev_res, &failed, list)
-		restore_dev_resource(dev_res);
-	free_list(&failed);
-
 	/* Revert to the old configuration */
 	list_for_each_entry(dev_res, &saved, list) {
 		struct resource *res = dev_res->res;

I don't know this code well enough to know if that changes is completely bonkers or what. Maybe a change that gets zero as pci_hotplug_io_size for root ports that don't have an io window would be better? Any other ideas, or other information about the crash that I could provide? Thanks,

Scott

[1]: Crash:

Quoted 66 lines

SError Interrupt on CPU0, code 0x00000000be000411 -- SError

CPU: 0 UID: 0 PID: 9 Comm: kworker/0:0 Not tainted 6.15.2+ #16 PREEMPT(undef)
Hardware name: Adlink Ampere Altra Developer Platform/Ampere Altra Developer Platform, BIOS TianoCore 2.09.100.00 (SYS: 2.10.20221028) 04/30/2
Workqueue: events work_for_cpu_fn
pstate: 204000c9 (nzCv daIF +PAN -UAO -TCO -DIT -SSBS BTYPE=--)
pc : _raw_spin_lock_irqsave+0x34/0xb0
lr : __wake_up+0x30/0x80

sp : ffff800080003e20
x29: ffff800080003e20 x28: ffff07ff808b0000 x27: 0000000000050260
x26: 0000000000000001 x25: ffffcc5decad6bd8 x24: ffffcc5decc8b5b8
x23: ffff080138a10000 x22: ffff080138a00000 x21: 0000000000000003
x20: ffff080138a14dc8 x19: 0000000000000000 x18: 006808018e775f03
x17: ffff3bc1330d2000 x16: ffffcc5e3a520ed8 x15: 8daad055c6e77021
x14: f231631cf9328575 x13: 0e51168d06a6cf91 x12: f5db8c23b764520c
x11: 0000000000000040 x10: 0000000000000000 x9 : ffffcc5e3a520f08 x8 : 0000000000002113 x7 : 0000000000000000 x6 : 0000000000000000 x5 : 0000000000000000 x4 : ffff800080003e08 x3 : 00000000000000c0 x2 : 0000000000000001 x1 : 0000000000000000 x0 : ffff080138a14dc8

Kernel panic - not syncing: Asynchronous SError Interrupt
CPU: 0 UID: 0 PID: 9 Comm: kworker/0:0 Not tainted 6.15.2+ #16 PREEMPT(undef)
Hardware name: Adlink Ampere Altra Developer Platform/Ampere Altra Developer Platform, BIOS TianoCore 2.09.100.00 (SYS: 2.10.20221028) 04/30/2
Workqueue: events work_for_cpu_fn
Call trace:
 show_stack+0x30/0x98 (C)
 dump_stack_lvl+0x7c/0xa0
 dump_stack+0x18/0x2c
 panic+0x184/0x3c0
 nmi_panic+0x90/0xa0
 arm64_serror_panic+0x6c/0x90
 do_serror+0x58/0x60
 el1h_64_error_handler+0x38/0x60
 el1h_64_error+0x84/0x88
 _raw_spin_lock_irqsave+0x34/0xb0 (P)
 amdgpu_ih_process+0xf0/0x150 [amdgpu]
 amdgpu_irq_handler+0x34/0xa0 [amdgpu]
 __handle_irq_event_percpu+0x60/0x248
 handle_irq_event+0x4c/0xc0
 handle_fasteoi_irq+0xa0/0x1c8
 handle_irq_desc+0x3c/0x68
 generic_handle_domain_irq+0x24/0x40
 __gic_handle_irq_from_irqson.isra.0+0x15c/0x260
 gic_handle_irq+0x28/0x80
 call_on_irq_stack+0x24/0x30
 do_interrupt_handler+0x88/0xa0
 el1_interrupt+0x44/0xd0
 el1h_64_irq_handler+0x18/0x28
 el1h_64_irq+0x84/0x88
 amdgpu_device_rreg.part.0+0x4c/0x190 [amdgpu] (P)
 amdgpu_device_rreg+0x24/0x40 [amdgpu]
 psp_wait_for+0x88/0xd8 [amdgpu]
 psp_v11_0_bootloader_load_component+0x164/0x1b0 [amdgpu]
 psp_v11_0_bootloader_load_kdb+0x20/0x40 [amdgpu]
 psp_hw_start+0x5c/0x580 [amdgpu]
 psp_load_fw+0x9c/0x280 [amdgpu]
 psp_hw_init+0x44/0xa0 [amdgpu]
 amdgpu_device_fw_loading+0xf8/0x358 [amdgpu]
 amdgpu_device_ip_init+0x684/0x990 [amdgpu]
 amdgpu_device_init+0xba4/0x1038 [amdgpu]
 amdgpu_driver_load_kms+0x20/0xb8 [amdgpu]
 amdgpu_pci_probe+0x1f8/0x7f8 [amdgpu]
 local_pci_probe+0x44/0xb0
 work_for_cpu_fn+0x24/0x40
 process_one_work+0x17c/0x410
 worker_thread+0x254/0x388
 kthread+0x10c/0x128
 ret_from_fork+0x10/0x20

Quoted 5 lines

Kernel Offset: 0x4c5dba380000 from 0xffff800080000000
PHYS_OFFSET: 0x80000000
CPU features: 0x0800,000042e0,01202650,8241720b Memory Limit: none
---[ end Kernel panic - not syncing: Asynchronous SError Interrupt ]---

[2]: boot snippet

Quoted 27 lines

ACPI: PCI Root Bridge [PCI1] (domain 000d [bus 00-ff])
acpi PNP0A08:01: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI EDR HPX-Type3] acpi PNP0A08:01: _OSC: platform does not support [PCIeHotplug PME LTR DPC]
acpi PNP0A08:01: _OSC: OS now controls [PCIeCapability]
acpi PNP0A08:01: FADT indicates ASPM is unsupported, using BIOS configuration
acpi PNP0A08:01: MCFG quirk: ECAM at [mem 0x37fff0000000-0x37ffffffffff] for [bus 00-ff] with pci_32b_read_ops acpi PNP0A08:01: ECAM area [mem 0x37fff0000000-0x37ffffffffff] reserved by PNP0C02:00
acpi PNP0A08:01: ECAM at [mem 0x37fff0000000-0x37ffffffffff] for [bus 00-ff]
PCI host bridge to bus 000d:00
pci_bus 000d:00: root bus resource [mem 0x50000000-0x5fffffff window]
pci_bus 000d:00: root bus resource [mem 0x340000000000-0x37ffdfffffff window]
pci_bus 000d:00: root bus resource [bus 00-ff]
pci 000d:00:00.0: [1def:e100] type 00 class 0x060000 conventional PCI endpoint
pci 000d:00:01.0: [1def:e101] type 01 class 0x060400 PCIe Root Port
pci 000d:00:01.0: PCI bridge to [bus 01-03]
pci 000d:00:01.0:   bridge window [io  0xe000-0xefff]
pci 000d:00:01.0:   bridge window [mem 0x50000000-0x502fffff]
pci 000d:00:01.0:   bridge window [mem 0x340000000000-0x3400101fffff 64bit pref]
pci 000d:00:01.0: supports D1 D2
pci 000d:00:01.0: PME# supported from D0 D1 D3hot
...
pci 000d:00:01.0: bridge window [mem 0x340000000000-0x340017ffffff 64bit pref]: assigned pci 000d:00:01.0: bridge window [mem 0x50000000-0x502fffff]: assigned
pci 000d:00:01.0: bridge window [io  size 0x1000]: can't assign; no space
pci 000d:00:01.0: bridge window [io  size 0x1000]: failed to assign
pci 000d:00:01.0: bridge window [io  size 0x1000]: can't assign; no space
pci 000d:00:01.0: bridge window [io  size 0x1000]: failed to assign

Ilpo Järvinen

author

2025-06-24 12:48 UTC #60 lore raw

Re: [PATCH 24/25] PCI: Perform reset_resource() and build fail list in sync

Quoted 24 lines · D Scott Phillips

Quoted 22 lines · Ilpo Järvinen
Resetting resource is problematic as it prevent attempting to allocate the resource later, unless something in between restores the resource. Similarly, if fail_head does not contain all resources that were reset, those resource cannot be restored later.
The entire reset/restore cycle adds complexity and leaving resources into reseted state causes issues to other code such as for checks done in pci_enable_resources(). Take a small step towards not resetting resources by delaying reset until the end of resource assignment and build failure list (fail_head) in sync with the reset to avoid leaving behind resources that cannot be restored (for the case where the caller provides fail_head in the first place to allow restore somewhere in the callchain, as is not all callers pass non-NULL fail_head).
The Expansion ROM check is temporarily left in place while building the failure list until the upcoming change which reworks optional resource handling.
Ideally, whole resource reset could be removed but doing that in a big step would make the impact non-tractable due to complexity of all related code.

Quoted 6 lines

Quoted 1 line
Signed-off-by Ilpo Järvinen <[email protected]>
Hi Ilpo, I'm seeing a crash on arm64 at boot that I bisected to this change. I don't think it's the same as the other issues reported. I've confirmed the crash is still there after your follow up patches. The crash itself is below[1].

Thanks for the report. I'm sorry about the delay, I was away a few days (and hoping all the gremlins in that resource change would already have been finally solved but it seems my hopes were premature).

Quoted 68 lines

It looks like the problem begins when:
amdgpu_device_resize_fb_bar()
pci_resize_resource()
pci_reassign_bridge_resources()
__pci_bus_size_bridges()
adds pci_hotplug_io_size to realloc_head. The io resource allocation has failed earlier because the root port doesn't have an io window[2].
Then with this patch, pci_reassign_bridge_resources()'s call to __pci_bridge_assign_resources() now returns the io added space for hotplug in the failed list where the old code dropped it and did not.
That sends pci_reassign_bridge_resources() into the cleanup: path, where I think the cleanup code doesn't properly release the resources that were assigned by __pci_bridge_assign_resources() and there's a conflict reported in pci_claim_resource() where a restored resource is found as conflicting with itself:
Quoted 1 line
pcieport 000d:00:01.0: bridge window [mem 0x340000000000-0x340017ffffff 64bit pref]: can't claim; address conflict with PCI Bus 000d:01 [mem 0x340000000000-0x340017ffffff 64bit pref]
Setting pci=hpiosize=0 avoids this crash, as does this change:
drivers/pci/setup-bus.c +11 -7
index 16d5d390599a..59ece11702da 100644
@@ -2442,7 +2442,7 @@ int pci_reassign_bridge_resources(struct pci_dev *bridge, unsigned long type)
 	LIST_HEAD(saved);
 	LIST_HEAD(added);
 	LIST_HEAD(failed);
-	unsigned int i;
+	unsigned int i, relevant_fails;
 	int ret;
 
 	down_read(&pci_bus_sem);
@@ -2490,7 +2490,16 @@ int pci_reassign_bridge_resources(struct pci_dev *bridge, unsigned long type)
 	__pci_bridge_assign_resources(bridge, &added, &failed);
 	BUG_ON(!list_empty(&added));
 
-	if (!list_empty(&failed)) {
+	relevant_fails = 0;
+	list_for_each_entry(dev_res, &failed, list) {
+		restore_dev_resource(dev_res);
+		if (((dev_res->res->flags ^ type) & PCI_RES_TYPE_MASK) == 0)
+			relevant_fails++;
+	}
+	free_list(&failed);
+
+	/* Cleanup if we had failures in resources of interest */
+	if (relevant_fails != 0) {
 		ret = -ENOSPC;
 		goto cleanup;
 	}
@@ -2509,11 +2518,6 @@ int pci_reassign_bridge_resources(struct pci_dev *bridge, unsigned long type)
 	return 0;
 
 cleanup:
-	/* Restore size and flags */
-	list_for_each_entry(dev_res, &failed, list)
-		restore_dev_resource(dev_res);
-	free_list(&failed);
-
 	/* Revert to the old configuration */
 	list_for_each_entry(dev_res, &saved, list) {
 		struct resource *res = dev_res->res;

Quoted 2 lines

I don't know this code well enough to know if that changes is completely bonkers or what.

It will take me a bit of time to assimilate all the information too, this code has astronomical complexity and a failure due to resize is yet another angle to it. I'll get back to you once I've figured more out.

Quoted 2 lines

Maybe a change that gets zero as pci_hotplug_io_size for root ports that don't have an io window would be better?

Similar to this, I personally don't like having a non-zero IO window for the cases where there is no io ports required for any devices. On dual socket systems here, it always leads to exhausting the io port space and the later probed devices that would actually have io port resources will not get them. While those failures are not critical/important in my case, it seems pointless to allocate IO window just for that sake of potential hotplug, hotplug could just get it when there's a true need for the IO window (and nothing bogusly reserved all io port space).

So IMO, a patch to change the default hotplug IO window to zero would make sense which effectively matches what you provided on the cmdline. But this is said before fully understanding what goes wrong on the resize path, it may need corrections too. But regardless of that, I think zeroing HP IO window is worthwhile on its own right to avoid exhausting the IO port space for nothing.

Quoted 2 lines

Any other
ideas, or other information about the crash that I could provide?

I suppose the actual crash itself is because the driver does make some invalid assumption about the resources without checking the resources first (I've not had time to locate and check the driver code yet).

Quoted 102 lines

[1]: Crash:
Quoted 66 lines
SError Interrupt on CPU0, code 0x00000000be000411 -- SError
CPU: 0 UID: 0 PID: 9 Comm: kworker/0:0 Not tainted 6.15.2+ #16 PREEMPT(undef)
Hardware name: Adlink Ampere Altra Developer Platform/Ampere Altra Developer Platform, BIOS TianoCore 2.09.100.00 (SYS: 2.10.20221028) 04/30/2
Workqueue: events work_for_cpu_fn
pstate: 204000c9 (nzCv daIF +PAN -UAO -TCO -DIT -SSBS BTYPE=--)
pc : _raw_spin_lock_irqsave+0x34/0xb0
lr : __wake_up+0x30/0x80
sp : ffff800080003e20
x29: ffff800080003e20 x28: ffff07ff808b0000 x27: 0000000000050260
x26: 0000000000000001 x25: ffffcc5decad6bd8 x24: ffffcc5decc8b5b8
x23: ffff080138a10000 x22: ffff080138a00000 x21: 0000000000000003
x20: ffff080138a14dc8 x19: 0000000000000000 x18: 006808018e775f03
x17: ffff3bc1330d2000 x16: ffffcc5e3a520ed8 x15: 8daad055c6e77021
x14: f231631cf9328575 x13: 0e51168d06a6cf91 x12: f5db8c23b764520c
x11: 0000000000000040 x10: 0000000000000000 x9 : ffffcc5e3a520f08 x8 : 0000000000002113 x7 : 0000000000000000 x6 : 0000000000000000 x5 : 0000000000000000 x4 : ffff800080003e08 x3 : 00000000000000c0 x2 : 0000000000000001 x1 : 0000000000000000 x0 : ffff080138a14dc8
Kernel panic - not syncing: Asynchronous SError Interrupt
CPU: 0 UID: 0 PID: 9 Comm: kworker/0:0 Not tainted 6.15.2+ #16 PREEMPT(undef)
Hardware name: Adlink Ampere Altra Developer Platform/Ampere Altra Developer Platform, BIOS TianoCore 2.09.100.00 (SYS: 2.10.20221028) 04/30/2
Workqueue: events work_for_cpu_fn
Call trace:
 show_stack+0x30/0x98 (C)
 dump_stack_lvl+0x7c/0xa0
 dump_stack+0x18/0x2c
 panic+0x184/0x3c0
 nmi_panic+0x90/0xa0
 arm64_serror_panic+0x6c/0x90
 do_serror+0x58/0x60
 el1h_64_error_handler+0x38/0x60
 el1h_64_error+0x84/0x88
 _raw_spin_lock_irqsave+0x34/0xb0 (P)
 amdgpu_ih_process+0xf0/0x150 [amdgpu]
 amdgpu_irq_handler+0x34/0xa0 [amdgpu]
 __handle_irq_event_percpu+0x60/0x248
 handle_irq_event+0x4c/0xc0
 handle_fasteoi_irq+0xa0/0x1c8
 handle_irq_desc+0x3c/0x68
 generic_handle_domain_irq+0x24/0x40
 __gic_handle_irq_from_irqson.isra.0+0x15c/0x260
 gic_handle_irq+0x28/0x80
 call_on_irq_stack+0x24/0x30
 do_interrupt_handler+0x88/0xa0
 el1_interrupt+0x44/0xd0
 el1h_64_irq_handler+0x18/0x28
 el1h_64_irq+0x84/0x88
 amdgpu_device_rreg.part.0+0x4c/0x190 [amdgpu] (P)
 amdgpu_device_rreg+0x24/0x40 [amdgpu]
 psp_wait_for+0x88/0xd8 [amdgpu]
 psp_v11_0_bootloader_load_component+0x164/0x1b0 [amdgpu]
 psp_v11_0_bootloader_load_kdb+0x20/0x40 [amdgpu]
 psp_hw_start+0x5c/0x580 [amdgpu]
 psp_load_fw+0x9c/0x280 [amdgpu]
 psp_hw_init+0x44/0xa0 [amdgpu]
 amdgpu_device_fw_loading+0xf8/0x358 [amdgpu]
 amdgpu_device_ip_init+0x684/0x990 [amdgpu]
 amdgpu_device_init+0xba4/0x1038 [amdgpu]
 amdgpu_driver_load_kms+0x20/0xb8 [amdgpu]
 amdgpu_pci_probe+0x1f8/0x7f8 [amdgpu]
 local_pci_probe+0x44/0xb0
 work_for_cpu_fn+0x24/0x40
 process_one_work+0x17c/0x410
 worker_thread+0x254/0x388
 kthread+0x10c/0x128
 ret_from_fork+0x10/0x20
Quoted 5 lines
Kernel Offset: 0x4c5dba380000 from 0xffff800080000000
PHYS_OFFSET: 0x80000000
CPU features: 0x0800,000042e0,01202650,8241720b Memory Limit: none
---[ end Kernel panic - not syncing: Asynchronous SError Interrupt ]---
[2]: boot snippet
Quoted 27 lines
ACPI: PCI Root Bridge [PCI1] (domain 000d [bus 00-ff])
acpi PNP0A08:01: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI EDR HPX-Type3] acpi PNP0A08:01: _OSC: platform does not support [PCIeHotplug PME LTR DPC]
acpi PNP0A08:01: _OSC: OS now controls [PCIeCapability]
acpi PNP0A08:01: FADT indicates ASPM is unsupported, using BIOS configuration
acpi PNP0A08:01: MCFG quirk: ECAM at [mem 0x37fff0000000-0x37ffffffffff] for [bus 00-ff] with pci_32b_read_ops acpi PNP0A08:01: ECAM area [mem 0x37fff0000000-0x37ffffffffff] reserved by PNP0C02:00
acpi PNP0A08:01: ECAM at [mem 0x37fff0000000-0x37ffffffffff] for [bus 00-ff]
PCI host bridge to bus 000d:00
pci_bus 000d:00: root bus resource [mem 0x50000000-0x5fffffff window]
pci_bus 000d:00: root bus resource [mem 0x340000000000-0x37ffdfffffff window]
pci_bus 000d:00: root bus resource [bus 00-ff]
pci 000d:00:00.0: [1def:e100] type 00 class 0x060000 conventional PCI endpoint
pci 000d:00:01.0: [1def:e101] type 01 class 0x060400 PCIe Root Port
pci 000d:00:01.0: PCI bridge to [bus 01-03]
pci 000d:00:01.0:   bridge window [io  0xe000-0xefff]
pci 000d:00:01.0:   bridge window [mem 0x50000000-0x502fffff]
pci 000d:00:01.0:   bridge window [mem 0x340000000000-0x3400101fffff 64bit pref]
pci 000d:00:01.0: supports D1 D2
pci 000d:00:01.0: PME# supported from D0 D1 D3hot
...
pci 000d:00:01.0: bridge window [mem 0x340000000000-0x340017ffffff 64bit pref]: assigned pci 000d:00:01.0: bridge window [mem 0x50000000-0x502fffff]: assigned
pci 000d:00:01.0: bridge window [io  size 0x1000]: can't assign; no space
pci 000d:00:01.0: bridge window [io  size 0x1000]: failed to assign
pci 000d:00:01.0: bridge window [io  size 0x1000]: can't assign; no space
pci 000d:00:01.0: bridge window [io  size 0x1000]: failed to assign

Signature · 1 line

i.

Ilpo Järvinen

author

2025-06-25 17:45 UTC #61 lore raw

Re: [PATCH 24/25] PCI: Perform reset_resource() and build fail list in sync

In reply to D Scott Phillips

Quoted 99 lines · D Scott Phillips

Quoted 18 lines · Ilpo Järvinen
Resetting resource is problematic as it prevent attempting to allocate the resource later, unless something in between restores the resource. Similarly, if fail_head does not contain all resources that were reset, those resource cannot be restored later.
The entire reset/restore cycle adds complexity and leaving resources into reseted state causes issues to other code such as for checks done in pci_enable_resources(). Take a small step towards not resetting resources by delaying reset until the end of resource assignment and build failure list (fail_head) in sync with the reset to avoid leaving behind resources that cannot be restored (for the case where the caller provides fail_head in the first place to allow restore somewhere in the callchain, as is not all callers pass non-NULL fail_head).
The Expansion ROM check is temporarily left in place while building the failure list until the upcoming change which reworks optional resource handling.
Quoted 5 lines
Ideally, whole resource reset could be removed but doing that in a big step would make the impact non-tractable due to complexity of all related code.
Signed-off-by Ilpo Järvinen <[email protected]>
Hi Ilpo, I'm seeing a crash on arm64 at boot that I bisected to this change. I don't think it's the same as the other issues reported. I've confirmed the crash is still there after your follow up patches. The crash itself is below[1].
It looks like the problem begins when:
amdgpu_device_resize_fb_bar()
pci_resize_resource()
pci_reassign_bridge_resources()
__pci_bus_size_bridges()
adds pci_hotplug_io_size to realloc_head. The io resource allocation has failed earlier because the root port doesn't have an io window[2].
Then with this patch, pci_reassign_bridge_resources()'s call to __pci_bridge_assign_resources() now returns the io added space for hotplug in the failed list where the old code dropped it and did not.
That sends pci_reassign_bridge_resources() into the cleanup: path, where I think the cleanup code doesn't properly release the resources that were assigned by __pci_bridge_assign_resources() and there's a conflict reported in pci_claim_resource() where a restored resource is found as conflicting with itself:
Quoted 1 line
pcieport 000d:00:01.0: bridge window [mem 0x340000000000-0x340017ffffff 64bit pref]: can't claim; address conflict with PCI Bus 000d:01 [mem 0x340000000000-0x340017ffffff 64bit pref]
Setting pci=hpiosize=0 avoids this crash, as does this change:
drivers/pci/setup-bus.c +11 -7
index 16d5d390599a..59ece11702da 100644
@@ -2442,7 +2442,7 @@ int pci_reassign_bridge_resources(struct pci_dev *bridge, unsigned long type)
 	LIST_HEAD(saved);
 	LIST_HEAD(added);
 	LIST_HEAD(failed);
-	unsigned int i;
+	unsigned int i, relevant_fails;
 	int ret;
 
 	down_read(&pci_bus_sem);
@@ -2490,7 +2490,16 @@ int pci_reassign_bridge_resources(struct pci_dev *bridge, unsigned long type)
 	__pci_bridge_assign_resources(bridge, &added, &failed);
 	BUG_ON(!list_empty(&added));
 
-	if (!list_empty(&failed)) {
+	relevant_fails = 0;
+	list_for_each_entry(dev_res, &failed, list) {
+		restore_dev_resource(dev_res);
+		if (((dev_res->res->flags ^ type) & PCI_RES_TYPE_MASK) == 0)
+			relevant_fails++;
+	}
+	free_list(&failed);
+
+	/* Cleanup if we had failures in resources of interest */
+	if (relevant_fails != 0) {
 		ret = -ENOSPC;
 		goto cleanup;
 	}
@@ -2509,11 +2518,6 @@ int pci_reassign_bridge_resources(struct pci_dev *bridge, unsigned long type)
 	return 0;
 
 cleanup:
-	/* Restore size and flags */
-	list_for_each_entry(dev_res, &failed, list)
-		restore_dev_resource(dev_res);
-	free_list(&failed);
-
 	/* Revert to the old configuration */
 	list_for_each_entry(dev_res, &saved, list) {
 		struct resource *res = dev_res->res;

Quoted 2 lines

I don't know this code well enough to know if that changes is completely bonkers or what.

Hi again,

Thanks for all the details what you think went wrong, it was really useful. I think you have it towards the right direction but a more targetted seems enough to address this (this needs to be confirmed, please test the patch below).

The most correct solution would be to make all the resource fitting code to focus on the resources that match the type filter. However, that looks way too scary change at the moment to implement, and especially, let it end up into stable (to fix this issue). So it looks this somewhat band-aid solution similar to your attempt might be better as a fix for now.

In medium term, I'd want to avoid using type as a filter and base all such decisions on matching the bridge window resource the dev resource belongs to. I've some work towards that direction already which removes lots of complexity in which bridge window is going to be selected as there will be a single place to make always the same decision. That change is also going to simplify the internal interfaces between functions very noticably (but the change require more testing before I've enough confidence to submit it). That work doesn't cover this resize side yet but it should be extended there as well.

So please test this somewhat band-aid patch:

From 971686ed85e341e7234f8fe8b666140187f63ad1 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Ilpo=20J=C3=A4rvinen?= <[email protected]>
Date: Wed, 25 Jun 2025 20:30:43 +0300
Subject: [PATCH 1/1] PCI: Fix failure detectiong during resource resize
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

Since the commit 96336ec70264 ("PCI: Perform reset_resource() and build fail list in sync") the failed list is always built and returned to let the caller decide if what to do with the failures. The caller may want to retry resource fitting and assignment and before that can happen, the resources should be restored to their original state (a reset effectively clears the struct resource), which requires returning them on the failed list so that the original state remains stored in the associated struct pci_dev_resource.

Resource resizing is different from the ordinary resource fitting and assignment in that it only considers part of the resources. This means failures for other resource types are not relevant at all and should be ignored. As resize doesn't unassign such unrelated resources, those resource ending up into the failed list implies assignment of that resource must have failed before resize too. The check in pci_reassign_bridge_resources() to decide if the whole assignment is successful, however, is based on list emptiness which may cause false negatives when the failed list resources with unrelated type.

If the failed list is not empty, call pci_required_resource_failed() and extend it to be able to filter on specific resource types too (if provided).

Calling pci_required_resource_failed() at this point is slightly problematic because the resource itself is reset when the failed list is constructed in __assign_resources_sorted(). As a result, pci_resource_is_optional() does not have access to the original resource flags. This could be worked around by restoring and re-reseting the resource around the call to pci_resource_is_optional(), however, it shouldn't cause issue as resource resizing is meant for 64-bit prefetchable resources according to Christian König (see the Link which unfortunately doesn't point directly to Christian's reply because lore didn't store that email at all).

Link https://lore.kernel.org/all/[email protected]/ open here

Reported-by D Scott Phillips <[email protected]>

Signed-off-by Ilpo Järvinen <[email protected]>

Cc Christian König <[email protected]>

---
 drivers/pci/setup-bus.c | 26 ++++++++++++++++++--------
 1 file changed, 18 insertions(+), 8 deletions(-)

drivers/pci/setup-bus.c +18 -8

index 07c3d021a47e..8284bbdc44b4 100644
@@ -28,6 +28,10 @@
 #include <linux/acpi.h>
 #include "pci.h"
 
+#define PCI_RES_TYPE_MASK \
+	(IORESOURCE_IO | IORESOURCE_MEM | IORESOURCE_PREFETCH |\
+	 IORESOURCE_MEM_64)
+
 unsigned int pci_flags;
 EXPORT_SYMBOL_GPL(pci_flags);

@@ -384,13 +388,19 @@ static bool pci_need_to_release(unsigned long mask, struct resource *res)
 }
 
 /* Return: @true if assignment of a required resource failed. */
-static bool pci_required_resource_failed(struct list_head *fail_head)
+static bool pci_required_resource_failed(struct list_head *fail_head,
+					 unsigned long type)
 {
 	struct pci_dev_resource *fail_res;
 
+	type &= ~PCI_RES_TYPE_MASK;
+
 	list_for_each_entry(fail_res, fail_head, list) {
 		int idx = pci_resource_num(fail_res->dev, fail_res->res);
 
+		if (type && (fail_res->flags & PCI_RES_TYPE_MASK) != type)
+			continue;
+
 		if (!pci_resource_is_optional(fail_res->dev, idx))
 			return true;
 	}

@@ -504,7 +514,7 @@ static void __assign_resources_sorted(struct list_head *head,
 	}
 
 	/* Without realloc_head and only optional fails, nothing more to do. */
-	if (!pci_required_resource_failed(&local_fail_head) &&
+	if (!pci_required_resource_failed(&local_fail_head, 0) &&
 	    list_empty(realloc_head)) {
 		list_for_each_entry(save_res, &save_head, list) {
 			struct resource *res = save_res->res;

@@ -1704,10 +1714,6 @@ static void __pci_bridge_assign_resources(const struct pci_dev *bridge,
 	}
 }
 
-#define PCI_RES_TYPE_MASK \
-	(IORESOURCE_IO | IORESOURCE_MEM | IORESOURCE_PREFETCH |\
-	 IORESOURCE_MEM_64)
-
 static void pci_bridge_release_resources(struct pci_bus *bus,
 					 unsigned long type)
 {

@@ -2445,8 +2451,12 @@ int pci_reassign_bridge_resources(struct pci_dev *bridge, unsigned long type)
 		free_list(&added);
 
 	if (!list_empty(&failed)) {
-		ret = -ENOSPC;
-		goto cleanup;
+		if (pci_required_resource_failed(&failed, type)) {
+			ret = -ENOSPC;
+			goto cleanup;
+		}
+		/* Only resources with unrelated types failed (again) */
+		free_list(&failed);
 	}
 
 	list_for_each_entry(dev_res, &saved, list) {

base-commit: 19272b37aa4f83ca52bdf9c16d5d81bdd1354494

Signature · 1 line

2.39.5

D Scott Phillips

2025-06-25 20:43 UTC #62 lore raw

Re: [PATCH 24/25] PCI: Perform reset_resource() and build fail list in sync

Quoted 127 lines · Ilpo Järvinen

Quoted 99 lines · D Scott Phillips
Quoted 18 lines · Ilpo Järvinen
Resetting resource is problematic as it prevent attempting to allocate the resource later, unless something in between restores the resource. Similarly, if fail_head does not contain all resources that were reset, those resource cannot be restored later.
The entire reset/restore cycle adds complexity and leaving resources into reseted state causes issues to other code such as for checks done in pci_enable_resources(). Take a small step towards not resetting resources by delaying reset until the end of resource assignment and build failure list (fail_head) in sync with the reset to avoid leaving behind resources that cannot be restored (for the case where the caller provides fail_head in the first place to allow restore somewhere in the callchain, as is not all callers pass non-NULL fail_head).
The Expansion ROM check is temporarily left in place while building the failure list until the upcoming change which reworks optional resource handling.
Quoted 5 lines
Ideally, whole resource reset could be removed but doing that in a big step would make the impact non-tractable due to complexity of all related code.
Signed-off-by Ilpo Järvinen <[email protected]>
Hi Ilpo, I'm seeing a crash on arm64 at boot that I bisected to this change. I don't think it's the same as the other issues reported. I've confirmed the crash is still there after your follow up patches. The crash itself is below[1].
It looks like the problem begins when:
amdgpu_device_resize_fb_bar()
pci_resize_resource()
pci_reassign_bridge_resources()
__pci_bus_size_bridges()
adds pci_hotplug_io_size to realloc_head. The io resource allocation has failed earlier because the root port doesn't have an io window[2].
Then with this patch, pci_reassign_bridge_resources()'s call to __pci_bridge_assign_resources() now returns the io added space for hotplug in the failed list where the old code dropped it and did not.
That sends pci_reassign_bridge_resources() into the cleanup: path, where I think the cleanup code doesn't properly release the resources that were assigned by __pci_bridge_assign_resources() and there's a conflict reported in pci_claim_resource() where a restored resource is found as conflicting with itself:
Quoted 1 line
pcieport 000d:00:01.0: bridge window [mem 0x340000000000-0x340017ffffff 64bit pref]: can't claim; address conflict with PCI Bus 000d:01 [mem 0x340000000000-0x340017ffffff 64bit pref]
Setting pci=hpiosize=0 avoids this crash, as does this change:
drivers/pci/setup-bus.c +11 -7
index 16d5d390599a..59ece11702da 100644
@@ -2442,7 +2442,7 @@ int pci_reassign_bridge_resources(struct pci_dev *bridge, unsigned long type)
 	LIST_HEAD(saved);
 	LIST_HEAD(added);
 	LIST_HEAD(failed);
-	unsigned int i;
+	unsigned int i, relevant_fails;
 	int ret;
 
 	down_read(&pci_bus_sem);
@@ -2490,7 +2490,16 @@ int pci_reassign_bridge_resources(struct pci_dev *bridge, unsigned long type)
 	__pci_bridge_assign_resources(bridge, &added, &failed);
 	BUG_ON(!list_empty(&added));
 
-	if (!list_empty(&failed)) {
+	relevant_fails = 0;
+	list_for_each_entry(dev_res, &failed, list) {
+		restore_dev_resource(dev_res);
+		if (((dev_res->res->flags ^ type) & PCI_RES_TYPE_MASK) == 0)
+			relevant_fails++;
+	}
+	free_list(&failed);
+
+	/* Cleanup if we had failures in resources of interest */
+	if (relevant_fails != 0) {
 		ret = -ENOSPC;
 		goto cleanup;
 	}
@@ -2509,11 +2518,6 @@ int pci_reassign_bridge_resources(struct pci_dev *bridge, unsigned long type)
 	return 0;
 
 cleanup:
-	/* Restore size and flags */
-	list_for_each_entry(dev_res, &failed, list)
-		restore_dev_resource(dev_res);
-	free_list(&failed);
-
 	/* Revert to the old configuration */
 	list_for_each_entry(dev_res, &saved, list) {
 		struct resource *res = dev_res->res;
Quoted 2 lines
I don't know this code well enough to know if that changes is completely bonkers or what.
Hi again,
Thanks for all the details what you think went wrong, it was really useful. I think you have it towards the right direction but a more targetted seems enough to address this (this needs to be confirmed, please test the patch below).
The most correct solution would be to make all the resource fitting code to focus on the resources that match the type filter. However, that looks way too scary change at the moment to implement, and especially, let it end up into stable (to fix this issue). So it looks this somewhat band-aid solution similar to your attempt might be better as a fix for now.
In medium term, I'd want to avoid using type as a filter and base all such decisions on matching the bridge window resource the dev resource belongs to. I've some work towards that direction already which removes lots of complexity in which bridge window is going to be selected as there will be a single place to make always the same decision. That change is also going to simplify the internal interfaces between functions very noticably (but the change require more testing before I've enough confidence to submit it). That work doesn't cover this resize side yet but it should be extended there as well.

Quoted 6 lines

So please test this somewhat band-aid patch:
From 971686ed85e341e7234f8fe8b666140187f63ad1 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Ilpo=20J=C3=A4rvinen?= <[email protected]>
Date: Wed, 25 Jun 2025 20:30:43 +0300
Subject: [PATCH 1/1] PCI: Fix failure detectiong during resource resize

detection

Quoted 62 lines

MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
Since the commit 96336ec70264 ("PCI: Perform reset_resource() and build fail list in sync") the failed list is always built and returned to let the caller decide if what to do with the failures. The caller may want to retry resource fitting and assignment and before that can happen, the resources should be restored to their original state (a reset effectively clears the struct resource), which requires returning them on the failed list so that the original state remains stored in the associated struct pci_dev_resource.
Resource resizing is different from the ordinary resource fitting and assignment in that it only considers part of the resources. This means failures for other resource types are not relevant at all and should be ignored. As resize doesn't unassign such unrelated resources, those resource ending up into the failed list implies assignment of that resource must have failed before resize too. The check in pci_reassign_bridge_resources() to decide if the whole assignment is successful, however, is based on list emptiness which may cause false negatives when the failed list resources with unrelated type.
If the failed list is not empty, call pci_required_resource_failed() and extend it to be able to filter on specific resource types too (if provided).
Calling pci_required_resource_failed() at this point is slightly problematic because the resource itself is reset when the failed list is constructed in __assign_resources_sorted(). As a result, pci_resource_is_optional() does not have access to the original resource flags. This could be worked around by restoring and re-reseting the resource around the call to pci_resource_is_optional(), however, it shouldn't cause issue as resource resizing is meant for 64-bit prefetchable resources according to Christian König (see the Link which unfortunately doesn't point directly to Christian's reply because lore didn't store that email at all).
Link https://lore.kernel.org/all/[email protected]/ open here
Reported-by D Scott Phillips <[email protected]>
Signed-off-by Ilpo Järvinen <[email protected]>
Cc Christian König <[email protected]>
---
 drivers/pci/setup-bus.c | 26 ++++++++++++++++++--------
 1 file changed, 18 insertions(+), 8 deletions(-)
drivers/pci/setup-bus.c +4 -0
index 07c3d021a47e..8284bbdc44b4 100644
@@ -28,6 +28,10 @@
 #include <linux/acpi.h>
 #include "pci.h"
 
+#define PCI_RES_TYPE_MASK \
+	(IORESOURCE_IO | IORESOURCE_MEM | IORESOURCE_PREFETCH |\
+	 IORESOURCE_MEM_64)
+
 unsigned int pci_flags;
 EXPORT_SYMBOL_GPL(pci_flags);
 

drivers/pci/setup-bus.c static bool pci_need_to_release(unsigned long mask, struct resource *res) · line 396

Quoted 10 lines

drivers/pci/setup-bus.c +3 -1

@@ -384,13 +388,19 @@ static bool pci_need_to_release(unsigned long mask, struct resource *res)
 }
 
 /* Return: @true if assignment of a required resource failed. */
-static bool pci_required_resource_failed(struct list_head *fail_head)
+static bool pci_required_resource_failed(struct list_head *fail_head,
+					 unsigned long type)
 {
 	struct pci_dev_resource *fail_res;
 
+	type &= ~PCI_RES_TYPE_MASK;

Is this meant to be type &= PCI_RES_TYPE_MASK? If not, then I think the new if check below is effectively just if (type).

FWIW, the patch in the current state is fixing the problem that I've been hitting.

Quoted 49 lines

drivers/pci/setup-bus.c +11 -7

+
 	list_for_each_entry(fail_res, fail_head, list) {
 		int idx = pci_resource_num(fail_res->dev, fail_res->res);
 
+		if (type && (fail_res->flags & PCI_RES_TYPE_MASK) != type)
+			continue;
+
 		if (!pci_resource_is_optional(fail_res->dev, idx))
 			return true;
 	}

@@ -504,7 +514,7 @@ static void __assign_resources_sorted(struct list_head *head,
 	}
 
 	/* Without realloc_head and only optional fails, nothing more to do. */
-	if (!pci_required_resource_failed(&local_fail_head) &&
+	if (!pci_required_resource_failed(&local_fail_head, 0) &&
 	    list_empty(realloc_head)) {
 		list_for_each_entry(save_res, &save_head, list) {
 			struct resource *res = save_res->res;

@@ -1704,10 +1714,6 @@ static void __pci_bridge_assign_resources(const struct pci_dev *bridge,
 	}
 }
 
-#define PCI_RES_TYPE_MASK \
-	(IORESOURCE_IO | IORESOURCE_MEM | IORESOURCE_PREFETCH |\
-	 IORESOURCE_MEM_64)
-
 static void pci_bridge_release_resources(struct pci_bus *bus,
 					 unsigned long type)
 {

@@ -2445,8 +2451,12 @@ int pci_reassign_bridge_resources(struct pci_dev *bridge, unsigned long type)
 		free_list(&added);
 
 	if (!list_empty(&failed)) {
-		ret = -ENOSPC;
-		goto cleanup;
+		if (pci_required_resource_failed(&failed, type)) {
+			ret = -ENOSPC;
+			goto cleanup;
+		}
+		/* Only resources with unrelated types failed (again) */
+		free_list(&failed);
 	}
 
 	list_for_each_entry(dev_res, &saved, list) {

base-commit: 19272b37aa4f83ca52bdf9c16d5d81bdd1354494

Signature · 1 line

2.39.5

Ilpo Järvinen

author

2025-06-26 09:22 UTC #63 lore raw

Re: [PATCH 24/25] PCI: Perform reset_resource() and build fail list in sync

Quoted 209 lines · D Scott Phillips

Quoted 127 lines · Ilpo Järvinen
Quoted 99 lines · D Scott Phillips
Quoted 18 lines · Ilpo Järvinen
Resetting resource is problematic as it prevent attempting to allocate the resource later, unless something in between restores the resource. Similarly, if fail_head does not contain all resources that were reset, those resource cannot be restored later.
The entire reset/restore cycle adds complexity and leaving resources into reseted state causes issues to other code such as for checks done in pci_enable_resources(). Take a small step towards not resetting resources by delaying reset until the end of resource assignment and build failure list (fail_head) in sync with the reset to avoid leaving behind resources that cannot be restored (for the case where the caller provides fail_head in the first place to allow restore somewhere in the callchain, as is not all callers pass non-NULL fail_head).
The Expansion ROM check is temporarily left in place while building the failure list until the upcoming change which reworks optional resource handling.
Quoted 5 lines
Ideally, whole resource reset could be removed but doing that in a big step would make the impact non-tractable due to complexity of all related code.
Signed-off-by Ilpo Järvinen <[email protected]>
Hi Ilpo, I'm seeing a crash on arm64 at boot that I bisected to this change. I don't think it's the same as the other issues reported. I've confirmed the crash is still there after your follow up patches. The crash itself is below[1].
It looks like the problem begins when:
amdgpu_device_resize_fb_bar()
pci_resize_resource()
pci_reassign_bridge_resources()
__pci_bus_size_bridges()
adds pci_hotplug_io_size to realloc_head. The io resource allocation has failed earlier because the root port doesn't have an io window[2].
Then with this patch, pci_reassign_bridge_resources()'s call to __pci_bridge_assign_resources() now returns the io added space for hotplug in the failed list where the old code dropped it and did not.
That sends pci_reassign_bridge_resources() into the cleanup: path, where I think the cleanup code doesn't properly release the resources that were assigned by __pci_bridge_assign_resources() and there's a conflict reported in pci_claim_resource() where a restored resource is found as conflicting with itself:
Quoted 1 line
pcieport 000d:00:01.0: bridge window [mem 0x340000000000-0x340017ffffff 64bit pref]: can't claim; address conflict with PCI Bus 000d:01 [mem 0x340000000000-0x340017ffffff 64bit pref]
Setting pci=hpiosize=0 avoids this crash, as does this change:
drivers/pci/setup-bus.c +11 -7
index 16d5d390599a..59ece11702da 100644
@@ -2442,7 +2442,7 @@ int pci_reassign_bridge_resources(struct pci_dev *bridge, unsigned long type)
 	LIST_HEAD(saved);
 	LIST_HEAD(added);
 	LIST_HEAD(failed);
-	unsigned int i;
+	unsigned int i, relevant_fails;
 	int ret;
 
 	down_read(&pci_bus_sem);
@@ -2490,7 +2490,16 @@ int pci_reassign_bridge_resources(struct pci_dev *bridge, unsigned long type)
 	__pci_bridge_assign_resources(bridge, &added, &failed);
 	BUG_ON(!list_empty(&added));
 
-	if (!list_empty(&failed)) {
+	relevant_fails = 0;
+	list_for_each_entry(dev_res, &failed, list) {
+		restore_dev_resource(dev_res);
+		if (((dev_res->res->flags ^ type) & PCI_RES_TYPE_MASK) == 0)
+			relevant_fails++;
+	}
+	free_list(&failed);
+
+	/* Cleanup if we had failures in resources of interest */
+	if (relevant_fails != 0) {
 		ret = -ENOSPC;
 		goto cleanup;
 	}
@@ -2509,11 +2518,6 @@ int pci_reassign_bridge_resources(struct pci_dev *bridge, unsigned long type)
 	return 0;
 
 cleanup:
-	/* Restore size and flags */
-	list_for_each_entry(dev_res, &failed, list)
-		restore_dev_resource(dev_res);
-	free_list(&failed);
-
 	/* Revert to the old configuration */
 	list_for_each_entry(dev_res, &saved, list) {
 		struct resource *res = dev_res->res;
Quoted 2 lines
I don't know this code well enough to know if that changes is completely bonkers or what.
Hi again,
Thanks for all the details what you think went wrong, it was really useful. I think you have it towards the right direction but a more targetted seems enough to address this (this needs to be confirmed, please test the patch below).
The most correct solution would be to make all the resource fitting code to focus on the resources that match the type filter. However, that looks way too scary change at the moment to implement, and especially, let it end up into stable (to fix this issue). So it looks this somewhat band-aid solution similar to your attempt might be better as a fix for now.
In medium term, I'd want to avoid using type as a filter and base all such decisions on matching the bridge window resource the dev resource belongs to. I've some work towards that direction already which removes lots of complexity in which bridge window is going to be selected as there will be a single place to make always the same decision. That change is also going to simplify the internal interfaces between functions very noticably (but the change require more testing before I've enough confidence to submit it). That work doesn't cover this resize side yet but it should be extended there as well.
Quoted 6 lines
So please test this somewhat band-aid patch:
From 971686ed85e341e7234f8fe8b666140187f63ad1 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Ilpo=20J=C3=A4rvinen?= <[email protected]>
Date: Wed, 25 Jun 2025 20:30:43 +0300
Subject: [PATCH 1/1] PCI: Fix failure detectiong during resource resize
detection
Quoted 71 lines
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
Since the commit 96336ec70264 ("PCI: Perform reset_resource() and build fail list in sync") the failed list is always built and returned to let the caller decide if what to do with the failures. The caller may want to retry resource fitting and assignment and before that can happen, the resources should be restored to their original state (a reset effectively clears the struct resource), which requires returning them on the failed list so that the original state remains stored in the associated struct pci_dev_resource.
Resource resizing is different from the ordinary resource fitting and assignment in that it only considers part of the resources. This means failures for other resource types are not relevant at all and should be ignored. As resize doesn't unassign such unrelated resources, those resource ending up into the failed list implies assignment of that resource must have failed before resize too. The check in pci_reassign_bridge_resources() to decide if the whole assignment is successful, however, is based on list emptiness which may cause false negatives when the failed list resources with unrelated type.
If the failed list is not empty, call pci_required_resource_failed() and extend it to be able to filter on specific resource types too (if provided).
Calling pci_required_resource_failed() at this point is slightly problematic because the resource itself is reset when the failed list is constructed in __assign_resources_sorted(). As a result, pci_resource_is_optional() does not have access to the original resource flags. This could be worked around by restoring and re-reseting the resource around the call to pci_resource_is_optional(), however, it shouldn't cause issue as resource resizing is meant for 64-bit prefetchable resources according to Christian König (see the Link which unfortunately doesn't point directly to Christian's reply because lore didn't store that email at all).
Link https://lore.kernel.org/all/[email protected]/ open here
Reported-by D Scott Phillips <[email protected]>
Signed-off-by Ilpo Järvinen <[email protected]>
Cc Christian König <[email protected]>
---
 drivers/pci/setup-bus.c | 26 ++++++++++++++++++--------
 1 file changed, 18 insertions(+), 8 deletions(-)
drivers/pci/setup-bus.c +6 -1
index 07c3d021a47e..8284bbdc44b4 100644
@@ -28,6 +28,10 @@
 #include <linux/acpi.h>
 #include "pci.h"
 
+#define PCI_RES_TYPE_MASK \
+	(IORESOURCE_IO | IORESOURCE_MEM | IORESOURCE_PREFETCH |\
+	 IORESOURCE_MEM_64)
+
 unsigned int pci_flags;
 EXPORT_SYMBOL_GPL(pci_flags);
 
@@ -384,13 +388,19 @@ static bool pci_need_to_release(unsigned long mask, struct resource *res)
 }
 
 /* Return: @true if assignment of a required resource failed. */
-static bool pci_required_resource_failed(struct list_head *fail_head)
+static bool pci_required_resource_failed(struct list_head *fail_head,
+					 unsigned long type)
 {
 	struct pci_dev_resource *fail_res;
 

Quoted 4 lines

drivers/pci/setup-bus.c static bool pci_need_to_release(unsigned long mask, struct resource *res) · line 396
Quoted 1 line
drivers/pci/setup-bus.c +1 -0
+	type &= ~PCI_RES_TYPE_MASK;
Is this meant to be type &= PCI_RES_TYPE_MASK? If not, then I think the new if check below is effectively just if (type).

Yes, it should have been without that ~. Can you test the change with that changed? I'm sorry about the extra trouble.

Quoted 53 lines

FWIW, the patch in the current state is fixing the problem that I've been hitting.

Quoted 49 lines

drivers/pci/setup-bus.c +11 -7

+
 	list_for_each_entry(fail_res, fail_head, list) {
 		int idx = pci_resource_num(fail_res->dev, fail_res->res);
 
+		if (type && (fail_res->flags & PCI_RES_TYPE_MASK) != type)
+			continue;
+
 		if (!pci_resource_is_optional(fail_res->dev, idx))
 			return true;
 	}

@@ -504,7 +514,7 @@ static void __assign_resources_sorted(struct list_head *head,
 	}
 
 	/* Without realloc_head and only optional fails, nothing more to do. */
-	if (!pci_required_resource_failed(&local_fail_head) &&
+	if (!pci_required_resource_failed(&local_fail_head, 0) &&
 	    list_empty(realloc_head)) {
 		list_for_each_entry(save_res, &save_head, list) {
 			struct resource *res = save_res->res;

@@ -1704,10 +1714,6 @@ static void __pci_bridge_assign_resources(const struct pci_dev *bridge,
 	}
 }
 
-#define PCI_RES_TYPE_MASK \
-	(IORESOURCE_IO | IORESOURCE_MEM | IORESOURCE_PREFETCH |\
-	 IORESOURCE_MEM_64)
-
 static void pci_bridge_release_resources(struct pci_bus *bus,
 					 unsigned long type)
 {

@@ -2445,8 +2451,12 @@ int pci_reassign_bridge_resources(struct pci_dev *bridge, unsigned long type)
 		free_list(&added);
 
 	if (!list_empty(&failed)) {
-		ret = -ENOSPC;
-		goto cleanup;
+		if (pci_required_resource_failed(&failed, type)) {
+			ret = -ENOSPC;
+			goto cleanup;
+		}
+		/* Only resources with unrelated types failed (again) */
+		free_list(&failed);
 	}
 
 	list_for_each_entry(dev_res, &saved, list) {

base-commit: 19272b37aa4f83ca52bdf9c16d5d81bdd1354494

Signature · 1 line

2.39.5

Signature · 1 line

i.

D Scott Phillips

Tested-by Reviewed-by

2025-06-26 14:53 UTC #64 lore raw

Re: [PATCH 24/25] PCI: Perform reset_resource() and build fail list in sync

Quoted 211 lines · Ilpo Järvinen

Quoted 209 lines · D Scott Phillips
Quoted 127 lines · Ilpo Järvinen
Quoted 99 lines · D Scott Phillips
Quoted 18 lines · Ilpo Järvinen
Resetting resource is problematic as it prevent attempting to allocate the resource later, unless something in between restores the resource. Similarly, if fail_head does not contain all resources that were reset, those resource cannot be restored later.
The entire reset/restore cycle adds complexity and leaving resources into reseted state causes issues to other code such as for checks done in pci_enable_resources(). Take a small step towards not resetting resources by delaying reset until the end of resource assignment and build failure list (fail_head) in sync with the reset to avoid leaving behind resources that cannot be restored (for the case where the caller provides fail_head in the first place to allow restore somewhere in the callchain, as is not all callers pass non-NULL fail_head).
The Expansion ROM check is temporarily left in place while building the failure list until the upcoming change which reworks optional resource handling.
Quoted 5 lines
Ideally, whole resource reset could be removed but doing that in a big step would make the impact non-tractable due to complexity of all related code.
Signed-off-by Ilpo Järvinen <[email protected]>
Hi Ilpo, I'm seeing a crash on arm64 at boot that I bisected to this change. I don't think it's the same as the other issues reported. I've confirmed the crash is still there after your follow up patches. The crash itself is below[1].
It looks like the problem begins when:
amdgpu_device_resize_fb_bar()
pci_resize_resource()
pci_reassign_bridge_resources()
__pci_bus_size_bridges()
adds pci_hotplug_io_size to realloc_head. The io resource allocation has failed earlier because the root port doesn't have an io window[2].
Then with this patch, pci_reassign_bridge_resources()'s call to __pci_bridge_assign_resources() now returns the io added space for hotplug in the failed list where the old code dropped it and did not.
That sends pci_reassign_bridge_resources() into the cleanup: path, where I think the cleanup code doesn't properly release the resources that were assigned by __pci_bridge_assign_resources() and there's a conflict reported in pci_claim_resource() where a restored resource is found as conflicting with itself:
Quoted 1 line
pcieport 000d:00:01.0: bridge window [mem 0x340000000000-0x340017ffffff 64bit pref]: can't claim; address conflict with PCI Bus 000d:01 [mem 0x340000000000-0x340017ffffff 64bit pref]
Setting pci=hpiosize=0 avoids this crash, as does this change:
drivers/pci/setup-bus.c +11 -7
index 16d5d390599a..59ece11702da 100644
@@ -2442,7 +2442,7 @@ int pci_reassign_bridge_resources(struct pci_dev *bridge, unsigned long type)
 	LIST_HEAD(saved);
 	LIST_HEAD(added);
 	LIST_HEAD(failed);
-	unsigned int i;
+	unsigned int i, relevant_fails;
 	int ret;
 
 	down_read(&pci_bus_sem);
@@ -2490,7 +2490,16 @@ int pci_reassign_bridge_resources(struct pci_dev *bridge, unsigned long type)
 	__pci_bridge_assign_resources(bridge, &added, &failed);
 	BUG_ON(!list_empty(&added));
 
-	if (!list_empty(&failed)) {
+	relevant_fails = 0;
+	list_for_each_entry(dev_res, &failed, list) {
+		restore_dev_resource(dev_res);
+		if (((dev_res->res->flags ^ type) & PCI_RES_TYPE_MASK) == 0)
+			relevant_fails++;
+	}
+	free_list(&failed);
+
+	/* Cleanup if we had failures in resources of interest */
+	if (relevant_fails != 0) {
 		ret = -ENOSPC;
 		goto cleanup;
 	}
@@ -2509,11 +2518,6 @@ int pci_reassign_bridge_resources(struct pci_dev *bridge, unsigned long type)
 	return 0;
 
 cleanup:
-	/* Restore size and flags */
-	list_for_each_entry(dev_res, &failed, list)
-		restore_dev_resource(dev_res);
-	free_list(&failed);
-
 	/* Revert to the old configuration */
 	list_for_each_entry(dev_res, &saved, list) {
 		struct resource *res = dev_res->res;
Quoted 2 lines
I don't know this code well enough to know if that changes is completely bonkers or what.
Hi again,
Thanks for all the details what you think went wrong, it was really useful. I think you have it towards the right direction but a more targetted seems enough to address this (this needs to be confirmed, please test the patch below).
The most correct solution would be to make all the resource fitting code to focus on the resources that match the type filter. However, that looks way too scary change at the moment to implement, and especially, let it end up into stable (to fix this issue). So it looks this somewhat band-aid solution similar to your attempt might be better as a fix for now.
In medium term, I'd want to avoid using type as a filter and base all such decisions on matching the bridge window resource the dev resource belongs to. I've some work towards that direction already which removes lots of complexity in which bridge window is going to be selected as there will be a single place to make always the same decision. That change is also going to simplify the internal interfaces between functions very noticably (but the change require more testing before I've enough confidence to submit it). That work doesn't cover this resize side yet but it should be extended there as well.
Quoted 6 lines
So please test this somewhat band-aid patch:
From 971686ed85e341e7234f8fe8b666140187f63ad1 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Ilpo=20J=C3=A4rvinen?= <[email protected]>
Date: Wed, 25 Jun 2025 20:30:43 +0300
Subject: [PATCH 1/1] PCI: Fix failure detectiong during resource resize
detection
Quoted 71 lines
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
Since the commit 96336ec70264 ("PCI: Perform reset_resource() and build fail list in sync") the failed list is always built and returned to let the caller decide if what to do with the failures. The caller may want to retry resource fitting and assignment and before that can happen, the resources should be restored to their original state (a reset effectively clears the struct resource), which requires returning them on the failed list so that the original state remains stored in the associated struct pci_dev_resource.
Resource resizing is different from the ordinary resource fitting and assignment in that it only considers part of the resources. This means failures for other resource types are not relevant at all and should be ignored. As resize doesn't unassign such unrelated resources, those resource ending up into the failed list implies assignment of that resource must have failed before resize too. The check in pci_reassign_bridge_resources() to decide if the whole assignment is successful, however, is based on list emptiness which may cause false negatives when the failed list resources with unrelated type.
If the failed list is not empty, call pci_required_resource_failed() and extend it to be able to filter on specific resource types too (if provided).
Calling pci_required_resource_failed() at this point is slightly problematic because the resource itself is reset when the failed list is constructed in __assign_resources_sorted(). As a result, pci_resource_is_optional() does not have access to the original resource flags. This could be worked around by restoring and re-reseting the resource around the call to pci_resource_is_optional(), however, it shouldn't cause issue as resource resizing is meant for 64-bit prefetchable resources according to Christian König (see the Link which unfortunately doesn't point directly to Christian's reply because lore didn't store that email at all).
Link https://lore.kernel.org/all/[email protected]/ open here
Reported-by D Scott Phillips <[email protected]>
Signed-off-by Ilpo Järvinen <[email protected]>
Cc Christian König <[email protected]>
---
 drivers/pci/setup-bus.c | 26 ++++++++++++++++++--------
 1 file changed, 18 insertions(+), 8 deletions(-)
drivers/pci/setup-bus.c +6 -1
index 07c3d021a47e..8284bbdc44b4 100644
@@ -28,6 +28,10 @@
 #include <linux/acpi.h>
 #include "pci.h"
 
+#define PCI_RES_TYPE_MASK \
+	(IORESOURCE_IO | IORESOURCE_MEM | IORESOURCE_PREFETCH |\
+	 IORESOURCE_MEM_64)
+
 unsigned int pci_flags;
 EXPORT_SYMBOL_GPL(pci_flags);
 
@@ -384,13 +388,19 @@ static bool pci_need_to_release(unsigned long mask, struct resource *res)
 }
 
 /* Return: @true if assignment of a required resource failed. */
-static bool pci_required_resource_failed(struct list_head *fail_head)
+static bool pci_required_resource_failed(struct list_head *fail_head,
+					 unsigned long type)
 {
 	struct pci_dev_resource *fail_res;
 

Quoted 7 lines

Quoted 4 lines
drivers/pci/setup-bus.c static bool pci_need_to_release(unsigned long mask, struct resource *res) · line 396
Quoted 1 line
drivers/pci/setup-bus.c +1 -0
+	type &= ~PCI_RES_TYPE_MASK;
Is this meant to be type &= PCI_RES_TYPE_MASK? If not, then I think the new if check below is effectively just if (type).
Yes, it should have been without that ~. Can you test the change with that changed? I'm sorry about the extra trouble.

Hi Ilpo, no trouble at all, and thanks for your effort in fixing this case. With that change to ~, the patch keeps working for my case.

Tested-by D Scott Phillips <[email protected]>

Reviewed-by D Scott Phillips <[email protected]>

Ilpo Järvinen

author

2024-12-16 18:00 UTC #65 lore raw

PATCH 25/25 PCI: Rework optional resource handling

In reply to Ilpo Järvinen

Remove and rescan cycle can result in not failure to assign a bridge windows if it becomes larger than before the remove. The bridge window size will include space for disabled Expansion ROM, which can causes the bridge window to not fit anymore into the same address space slot on rescan if the Expansion ROM resource was not assigned before the remove. In addition, the optional resource handling is not internally consistent.

The resource fitting logic supports three main types of optional resources:

- IOV BARs
- Expansion ROMs
- Bridge window size variation due to optional resources

In addition to the above, resizable BARs beyond their current size will require handling optional variation in resource sizes within the resource fitting algorithm (not yet done by the resource fitting code).

There are multiple inconsistencies related to optional resource handling:

a) The allocation failure of disabled expansion ROM requires special case inside assign_requested_resources_sorted().

b) The optionality of disabled expansion ROM is not considered during bridge window sizing in pbus_size_mem().

c) Setting resource size to zero for optional resource in pbus_size_mem() is problematic because it makes also the alignment invalid, which is checked by pdev_sort_resources().

Optional IOV resources have their size set to zero by pbus_size_mem() but the information about size is stored externally into the struct pci_sriov and complex call-chain trickery in pci_resource_alignment() ensures IOV resources return a valid alignment despite having zero resource size. A solution that is specific to IOV resources makes it hard to use the same solution for other types of resources such as expansion ROM.

Simply changing only pbus_size_mem() is not sufficient to fully address the main issue because it would introduce disparity between bridge window sizing and resource allocation. Due size based ordering of resource list during assignment loop, an Expansion ROM resource could steal space from some other resource and make the other resource not fit if the Expansion ROM is larger than the other resource. Thus, the resource assignment functions need to be changed as well.

Make optional resource handling more straightforwards. Use pci_resource_is_optional() to determine if a resource is optional or not in both bridge window sizing and assignment failure classification to ensure they always align. Indicate with a parameter to assign_requested_resources_sorted() whether it should attempt to allocate optional resources or not.

Always try first to assign all resources (also when realloc_head is not provided). This is required for calls from pci_assign_unassigned_root_bus_resources() that provides realloc_head only with some of its iterations.

Non-bridge-window optional resources in realloc_head now have add_size
0. This condition has to be detected in reassign_resources_sorted() before reassigning them (which would fail as there is no size change). Removing add_size=0 optional resources entirely from realloc_head might eventually be doable but further rework in __assign_resources_sorted() is needed first to support such a change.

Reported-by Jia Yao <[email protected]>

Tested-by Jia Yao <[email protected]>

Signed-off-by Ilpo Järvinen <[email protected]>

---
 drivers/pci/setup-bus.c | 89 +++++++++++++++++++++++++++--------------
 1 file changed, 58 insertions(+), 31 deletions(-)

drivers/pci/setup-bus.c +58 ~~-31~~

index b61f24a5cfa5..802643a4de47 100644
@@ -276,13 +276,14 @@ static void reassign_resources_sorted(struct list_head *realloc_head,
 		add_size = add_res->add_size;
 		align = add_res->min_align;
 		if (!res->parent) {
-			resource_set_range(res, align, add_size);
+			resource_set_range(res, align,
+					   resource_size(res) + add_size);
 			if (pci_assign_resource(dev, idx)) {
 				pci_dbg(dev,
 					"%s %pR: ignoring failure in optional allocation\n",
 					res_name, res);
 			}
-		} else {
+		} else if (add_size > 0) {
 			res->flags |= add_res->flags &
 				 (IORESOURCE_STARTALIGN|IORESOURCE_SIZEALIGN);
 			if (pci_reassign_resource(dev, idx, add_size, align))

@@ -302,38 +303,38 @@ static void reassign_resources_sorted(struct list_head *realloc_head,
  * @head:	Head of the list tracking requests for resources
  * @fail_head:	Head of the list tracking requests that could not be
  *		allocated
+ * @optional:	Assign also optional resources
  *
  * Satisfy resource requests of each element in the list.  Add requests that
  * could not be satisfied to the failed_list.
  */
 static void assign_requested_resources_sorted(struct list_head *head,
-				 struct list_head *fail_head)
+					      struct list_head *fail_head,
+					      bool optional)
 {
 	struct pci_dev_resource *dev_res;
 	struct resource *res;
 	struct pci_dev *dev;
+	bool optional_res;
 	int idx;
 
 	list_for_each_entry(dev_res, head, list) {
 		res = dev_res->res;
 		dev = dev_res->dev;
 		idx = pci_resource_num(dev, res);
+		optional_res = pci_resource_is_optional(dev, idx);
 
 		if (!resource_size(res))
 			continue;
 
+		if (!optional && optional_res)
+			continue;
+
 		if (pci_assign_resource(dev, idx)) {
 			if (fail_head) {
-				/*
-				 * If the failed resource is a ROM BAR and
-				 * it will be enabled later, don't add it
-				 * to the list.
-				 */
-				if (!((idx == PCI_ROM_RESOURCE) &&
-				      (!(res->flags & IORESOURCE_ROM_ENABLE))))
-					add_to_list(fail_head, dev, res,
-						    0 /* don't care */,
-						    0 /* don't care */);
+				add_to_list(fail_head, dev, res,
+					    0 /* don't care */,
+					    0 /* don't care */);
 			}
 		}
 	}

@@ -379,6 +380,20 @@ static bool pci_need_to_release(unsigned long mask, struct resource *res)
 	return false;	/* Should not get here */
 }
 
+/* Return: @true if assignment of a required resource failed. */
+static bool pci_required_resource_failed(struct list_head *fail_head)
+{
+	struct pci_dev_resource *fail_res;
+
+	list_for_each_entry(fail_res, fail_head, list) {
+		int idx = pci_resource_num(fail_res->dev, fail_res->res);
+
+		if (!pci_resource_is_optional(fail_res->dev, idx))
+			return true;
+	}
+	return false;
+}
+
 static void __assign_resources_sorted(struct list_head *head,
 				      struct list_head *realloc_head,
 				      struct list_head *fail_head)

@@ -388,9 +403,11 @@ static void __assign_resources_sorted(struct list_head *head,
 	 * adjacent, so later reassign can not reallocate them one by one in
 	 * parent resource window.
 	 *
-	 * Try to assign requested + add_size at beginning.  If could do that,
-	 * could get out early.  If could not do that, we still try to assign
-	 * requested at first, then try to reassign add_size for some resources.
+	 * Try to assign required and any optional resources at beginning
+	 * (add_size included). If all required resources were successfully
+	 * assigned, get out early. If could not do that, we still try to
+	 * assign required at first, then try to reassign some optional
+	 * resources.
 	 *
 	 * Separate three resource type checking if we need to release
 	 * assigned resource after requested + add_size try.

@@ -421,13 +438,13 @@ static void __assign_resources_sorted(struct list_head *head,
 
 	/* Check if optional add_size is there */
 	if (list_empty(realloc_head))
-		goto requested_and_reassign;
+		goto assign;
 
 	/* Save original start, end, flags etc at first */
 	list_for_each_entry(dev_res, head, list) {
 		if (add_to_list(&save_head, dev_res->dev, dev_res->res, 0, 0)) {
 			free_list(&save_head);
-			goto requested_and_reassign;
+			goto assign;
 		}
 	}

@@ -471,10 +488,10 @@ static void __assign_resources_sorted(struct list_head *head,
 
 	}
 
-	/* Try updated head list with add_size added */
-	assign_requested_resources_sorted(head, &local_fail_head);
+assign:
+	assign_requested_resources_sorted(head, &local_fail_head, true);
 
-	/* All assigned with add_size? */
+	/* All non-optional resources assigned? */
 	if (list_empty(&local_fail_head)) {
 		/* Remove head list from realloc_head list */
 		list_for_each_entry(dev_res, head, list)

@@ -483,6 +500,22 @@ static void __assign_resources_sorted(struct list_head *head,
 		goto out;
 	}
 
+	/* Without realloc_head and only optional fails, nothing more to do. */
+	if (!pci_required_resource_failed(&local_fail_head) &&
+	    list_empty(realloc_head)) {
+		list_for_each_entry(save_res, &save_head, list) {
+			struct resource *res = save_res->res;
+
+			if (res->parent)
+				continue;
+
+			restore_dev_resource(save_res);
+		}
+		free_list(&local_fail_head);
+		free_list(&save_head);
+		goto out;
+	}
+
 	/* Check failed type */
 	fail_type = pci_fail_res_type_mask(&local_fail_head);
 	/* Remove not need to be released assigned res from head list etc */

@@ -518,9 +551,8 @@ static void __assign_resources_sorted(struct list_head *head,
 		restore_dev_resource(save_res);
 	free_list(&save_head);
 
-requested_and_reassign:
 	/* Satisfy the must-have resource requests */
-	assign_requested_resources_sorted(head, NULL);
+	assign_requested_resources_sorted(head, NULL, false);
 
 	/* Try to satisfy any additional optional resource requests */
 	if (!list_empty(realloc_head))

@@ -535,11 +567,7 @@ static void __assign_resources_sorted(struct list_head *head,
 		if (res->parent)
 			continue;
 
-		/*
-		 * If the failed resource is a ROM BAR and it will
-		 * be enabled later, don't add it to the list.
-		 */
-		if (fail_head && !pci_resource_is_disabled_rom(res, idx)) {
+		if (fail_head) {
 			add_to_list(fail_head, dev, res,
 				    0 /* don't care */,
 				    0 /* don't care */);

@@ -1166,10 +1194,9 @@ static int pbus_size_mem(struct pci_bus *bus, unsigned long mask,
 			r_size = resource_size(r);
 
 			/* Put SRIOV requested res to the optional list */
-			if (realloc_head && pci_resource_is_iov(i)) {
+			if (realloc_head && pci_resource_is_optional(dev, i)) {
 				add_align = max(pci_resource_alignment(dev, r), add_align);
-				resource_set_size(r, 0);
-				add_to_list(realloc_head, dev, r, r_size, 0 /* Don't care */);
+				add_to_list(realloc_head, dev, r, 0, 0 /* Don't care */);
 				children_add_size += r_size;
 				continue;
 			}

Signature · 1 line

2.39.5

Bjorn Helgaas

Applied

2025-02-13 21:46 UTC #66 lore raw

In reply to Ilpo Järvinen

Quoted 58 lines · Ilpo Järvinen

Hi all,
This series focuses on PCI resource fitting and assignment algorithms. I've further changes in works to enable handling resizable BARs better during resource fitting built on top of these, but that's still WIP and this series seems way too large as is to have more stuff included.
First there are small tweaks and fixes to the relaxed tail alignment code and applying the lessons learned to other similar cases. They are sort of independent of the rest. Then a large set of pure cleanups and refactoring that are not intended to make any functional changes. Finally, starting from "PCI: Extend enable to check for any optional resource" are again patches that aim to make behavioral changes to fix bridge window sizing to consider expansion ROM as an optional resource (to fix a remove/rescan cycle issue) and improve resource fitting algorithm in general.
The series includes one of the change from Michał Winiarski <[email protected]> as these changes also touch the same IOV checks.
Please let me know if you'd prefer me to order the changes differently or split it into smaller chunks.
I've extensively tested this series over the hosts in our lab which have quite heterogeneous PCI setup each. There were no losses of any important resource. Without pci=realloc, there's some churn in which of the disabled expansion ROMs gets a scarce memory space assigned (with pci=realloc, they are all assigned large enough bridge window).
Ilpo Järvinen (24):
PCI: Remove add_align overwrite unrelated to size0
PCI: size0 is unrelated to add_align
PCI: Simplify size1 assignment logic
PCI: Optional bridge window size too may need relaxing
PCI: Fix old_size lower bound in calculate_iosize() too
PCI: Use SZ_* instead of literals in setup-bus.c
PCI: resource_set_range/size() conversions
PCI: Check resource_size() separately
PCI: Add pci_resource_num() helper
PCI: Add dev & res local variables to resource assignment funcs
PCI: Converge return paths in __assign_resources_sorted()
PCI: Refactor pdev_sort_resources() & __dev_sort_resources()
PCI: Use while loop and break instead of gotos
PCI: Rename retval to ret
PCI: Consolidate assignment loop next round preparation
PCI: Remove wrong comment from pci_reassign_resource()
PCI: Add restore_dev_resource()
PCI: Extend enable to check for any optional resource
PCI: Always have realloc_head in __assign_resources_sorted()
PCI: Indicate optional resource assignment failures
PCI: Add debug print when releasing resources before retry
PCI: Use res->parent to check is resource is assigned
PCI: Perform reset_resource() and build fail list in sync
PCI: Rework optional resource handling

Quoted 7 lines

Michał Winiarski (1):
PCI: Add a helper to identify IOV resources

 drivers/pci/pci.h       |  44 +++-
 drivers/pci/setup-bus.c | 566 +++++++++++++++++++++++-----------------
 drivers/pci/setup-res.c |   8 +-
 3 files changed, 364 insertions(+), 254 deletions(-)

Applied to pci/resource for v6.15, thanks, Ilpo!

Xiaochun XC17 Li | 李小春 Xavier

Tested-by on the v1 series

2025-02-14 08:18 UTC #67 lore raw

In reply to Ilpo Järvinen

Quoted 58 lines · On Mon, Dec 16, 2024 at 7:56:45PM +0200 Ilpo Järvinen…

From: Ilpo Järvinen <[email protected]>
Hi all,
This series focuses on PCI resource fitting and assignment algorithms. I've further changes in works to enable handling resizable BARs better during resource fitting built on top of these, but that's still WIP and this series seems way too large as is to have more stuff included.
First there are small tweaks and fixes to the relaxed tail alignment code and applying the lessons learned to other similar cases. They are sort of independent of the rest. Then a large set of pure cleanups and refactoring that are not intended to make any functional changes.
Finally, starting from "PCI: Extend enable to check for any optional resource" are again patches that aim to make behavioral changes to fix bridge window sizing to consider expansion ROM as an optional resource (to fix a remove/rescan cycle issue) and improve resource fitting algorithm in general.
The series includes one of the change from Michał Winiarski <[email protected]> as these changes also touch the same IOV checks.
Please let me know if you'd prefer me to order the changes differently or split it into smaller chunks.
I've extensively tested this series over the hosts in our lab which have quite heterogeneous PCI setup each. There were no losses of any important resource. Without pci=realloc, there's some churn in which of the disabled expansion ROMs gets a scarce memory space assigned (with pci=realloc, they are all assigned large enough bridge window).
Ilpo Järvinen (24):
PCI: Remove add_align overwrite unrelated to size0
PCI: size0 is unrelated to add_align
PCI: Simplify size1 assignment logic
PCI: Optional bridge window size too may need relaxing
PCI: Fix old_size lower bound in calculate_iosize() too
PCI: Use SZ_* instead of literals in setup-bus.c
PCI: resource_set_range/size() conversions
PCI: Check resource_size() separately
PCI: Add pci_resource_num() helper
PCI: Add dev & res local variables to resource assignment funcs
PCI: Converge return paths in __assign_resources_sorted()
PCI: Refactor pdev_sort_resources() & __dev_sort_resources()
PCI: Use while loop and break instead of gotos
PCI: Rename retval to ret
PCI: Consolidate assignment loop next round preparation
PCI: Remove wrong comment from pci_reassign_resource()
PCI: Add restore_dev_resource()
PCI: Extend enable to check for any optional resource
PCI: Always have realloc_head in __assign_resources_sorted()
PCI: Indicate optional resource assignment failures
PCI: Add debug print when releasing resources before retry
PCI: Use res->parent to check is resource is assigned
PCI: Perform reset_resource() and build fail list in sync
PCI: Rework optional resource handling

Quoted 8 lines

Michał Winiarski (1):
PCI: Add a helper to identify IOV resources

 drivers/pci/pci.h       |  44 +++-
 drivers/pci/setup-bus.c | 566 +++++++++++++++++++++++-----------------
 drivers/pci/setup-res.c |   8 +-
 3 files changed, 364 insertions(+), 254 deletions(-)

Hi, all
This series has undergone testing on the following configurations:
- Lenovo ThinkSystem SR630 V4 equipped with Intel Granite Rapids CPUs.
- The latest upstream kernel v6.14-rc2 and the stable kernel 6.13.2.
- With "pci=realloc" appended to the kernel command line.
- Red Hat Enterprise Linux 10.0 Beta.

Test results:
- All patches were applied cleanly and the build process was successful.
- The assignment for the ROM BAR of downstream devices was successful.
- The bridge window has been adjusted to fit the downstream resources.

Tested-by Xiaochun Lee <[email protected]>

Quoted 3 lines

--
2.39.5

Ilpo Järvinen

author

2025-02-14 11:53 UTC #68 lore raw

Hi Bjorn,

Can you please add to the last patch of the series:

Closes https://bugzilla.kernel.org/show_bug.cgi?id=219547

...And also Xiaochun's tested by tag from below to the series.

Quoted 48 lines · On Fri, 14 Feb 2025, Xiaochun XC17 Li | 李小春 Xavier wrote:

Quoted 47 lines · On Mon, Dec 16, 2024 at 7:56:45PM +0200 Ilpo Järvinen…
From: Ilpo Järvinen <[email protected]>
Hi all,
This series focuses on PCI resource fitting and assignment algorithms. I've further changes in works to enable handling resizable BARs better during resource fitting built on top of these, but that's still WIP and this series seems way too large as is to have more stuff included.
First there are small tweaks and fixes to the relaxed tail alignment code and applying the lessons learned to other similar cases. They are sort of independent of the rest. Then a large set of pure cleanups and refactoring that are not intended to make any functional changes.
Finally, starting from "PCI: Extend enable to check for any optional resource" are again patches that aim to make behavioral changes to fix bridge window sizing to consider expansion ROM as an optional resource (to fix a remove/rescan cycle issue) and improve resource fitting algorithm in general.
The series includes one of the change from Michał Winiarski <[email protected]> as these changes also touch the same IOV checks.
Please let me know if you'd prefer me to order the changes differently or split it into smaller chunks.
I've extensively tested this series over the hosts in our lab which have quite heterogeneous PCI setup each. There were no losses of any important resource. Without pci=realloc, there's some churn in which of the disabled expansion ROMs gets a scarce memory space assigned (with pci=realloc, they are all assigned large enough bridge window).
Ilpo Järvinen (24):
PCI: Remove add_align overwrite unrelated to size0
PCI: size0 is unrelated to add_align
PCI: Simplify size1 assignment logic
PCI: Optional bridge window size too may need relaxing
PCI: Fix old_size lower bound in calculate_iosize() too
PCI: Use SZ_* instead of literals in setup-bus.c
PCI: resource_set_range/size() conversions
PCI: Check resource_size() separately
PCI: Add pci_resource_num() helper
PCI: Add dev & res local variables to resource assignment funcs
PCI: Converge return paths in __assign_resources_sorted()
PCI: Refactor pdev_sort_resources() & __dev_sort_resources()
PCI: Use while loop and break instead of gotos
PCI: Rename retval to ret

Quoted 8 lines

Quoted 8 lines
PCI: Consolidate assignment loop next round preparation
PCI: Remove wrong comment from pci_reassign_resource()
PCI: Add restore_dev_resource()
PCI: Extend enable to check for any optional resource
PCI: Always have realloc_head in __assign_resources_sorted()
PCI: Indicate optional resource assignment failures
PCI: Add debug print when releasing resources before retry
PCI: Use res->parent to check is resource is assigned

I also noticed there's a typo here, should be "if resource is assigned"

My apologies for the extra work.

Quoted 24 lines

Quoted 11 lines
PCI: Perform reset_resource() and build fail list in sync
PCI: Rework optional resource handling
Michał Winiarski (1):
PCI: Add a helper to identify IOV resources
 drivers/pci/pci.h       |  44 +++-
 drivers/pci/setup-bus.c | 566 +++++++++++++++++++++++-----------------
 drivers/pci/setup-res.c |   8 +-
 3 files changed, 364 insertions(+), 254 deletions(-)
Hi, all
This series has undergone testing on the following configurations:
- Lenovo ThinkSystem SR630 V4 equipped with Intel Granite Rapids CPUs.
- The latest upstream kernel v6.14-rc2 and the stable kernel 6.13.2.
- With "pci=realloc" appended to the kernel command line.
- Red Hat Enterprise Linux 10.0 Beta.
Test results:
- All patches were applied cleanly and the build process was successful.
- The assignment for the ROM BAR of downstream devices was successful.
- The bridge window has been adjusted to fit the downstream resources.
Tested-by Xiaochun Lee <[email protected]>

Signature · 1 line

i.

Bjorn Helgaas

2025-02-14 21:28 UTC #69 lore raw

Quoted 7 lines · Ilpo Järvinen

Hi Bjorn,
Can you please add to the last patch of the series:
Closes https://bugzilla.kernel.org/show_bug.cgi?id=219547
...And also Xiaochun's tested by tag from below to the series.

Done! How exciting to have somebody step up to work on this resource assignment mess, thank you!

Bjorn