Skip to contentSkip to conversation

PCI: BAR resizing fix/rework

PATCH Superseded Ilpo Järvinen 6 participants 29 messages Oct 28, 2025 – Nov 13, 2025

Subsystems (5)

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore
#0 lore raw

cover PCI: BAR resizing fix/rework

Simon and Alex, could you please test if this series eliminates the claim conflicts and makes the BAR resize either succeed or not break things while rolling back resource changes? It should be tested without other fix patches (from me; if you need some random unrelated fix, that's okay).

Hi all,

Thanks to issue reports from Simon Richter and Alex Bennée, I discovered BAR resize rollback can corrupt the resource tree. As fixing corruption requires avoiding overlapping resource assignments, the correct fix can unfortunately results in worse user experience, what appeared to be "working" previously might no longer do so. Thus, I had to do a larger rework to pci_resize_resource() in order to properly restore resource states as it was prior to BAR resize.

This rework has been on my TODO list anyway but it wasn't the highest prio item until pci_resize_resource() started to cause regressions due to other resource assignment algorithm changes.

BAR resize rollback does not always restore BAR resources as they were before the resize operation was started. Currently, when pci_resize_resource() call is made by a driver, the driver must release device resource prior to the call. This is a design flaw in pci_resize_resource() API as PCI core cannot then save the state of those resources from what it was prior to release so it could restore them later if the BAR size change has to be rolled back.

PCI core's BAR resize operation doesn't even attempt to restore the device resources currently when rolling back BAR resize operation. If the normal resource assignment algorithm assigned those resources, then device resources might be assigned after pci_resize_resource() call but that could also trigger the resource tree corruption issue so what appeared to an user as "working" might be a corrupted state.

With the new pci_resize_resource() interface, the driver calling pci_resize_resource() should no longer release the device resources.

I've added WARN_ON_ONCE() to pick up similar bugs that cause resource tree corruption. At least in my tests all looked clear on that front after this series.

I was a bit on the edge how to split this series. Between patches 1 and 5-8, there might be cases where user experience is made worse if only part of the series are applied. But at the same time I was hesitant to merge all those changes together either as the changes way easier to understand when split properly. Personally I think BAR resize rollback code has not really functioned okay prior to series at all because touching an assigned resource on the rollback path is a bug, plain and simple. If that got things "working" it's still a bad bug (that one can get lucky and corruption results in non-corrupted numbers doesn't make it any better). If those patches need to be merged into one, just let me know and I can rearrange the patch order to make it easier.

This series will conflict what's in pci/rebar and likely with some xe changes from Lucas De Marchi that might also be rendered in part unnecessary due to pci_resize_resource() API change. My suggestion is that this series takes precedence over what's in pci/rebar to make things easier for stable people (I can rebase the pci/rebar patches on top of these so feel free to drop those other patches, if needed).

Ilpo Järvinen (9):
PCI: Prevent resource tree corruption when BAR resize fails PCI/IOV: Adjust ->barsz[] when changing BAR size
PCI: Change pci_dev variable from 'bridge' to 'dev'
PCI: Try BAR resize even when no window was released
PCI: Fix restoring BARs on BAR resize rollback path drm/xe: Remove driver side BAR release before resize drm/i915: Remove driver side BAR release before resize drm/amdgpu: Remove driver side BAR release before resize
PCI: Prevent restoring assigned resources

 drivers/gpu/drm/amd/amdgpu/amdgpu_device.c  |   8 +-
 drivers/gpu/drm/i915/gt/intel_region_lmem.c |  12 --
 drivers/gpu/drm/xe/xe_vram.c                |   3 -
 drivers/pci/iov.c                           |  15 +--
 drivers/pci/pci-sysfs.c                     |  15 +--
 drivers/pci/pci.c                           |   4 +
 drivers/pci/pci.h                           |   8 +-
 drivers/pci/setup-bus.c                     | 119 ++++++++++++++------
 drivers/pci/setup-res.c                     |  30 ++---
 9 files changed, 108 insertions(+), 106 deletions(-)

base-commit: 3a8660878839faadb4f1a6dd72c3179c1df56787

Signature · 1 line
2.39.5
#1 lore raw

PATCH 1/9 PCI: Prevent resource tree corruption when BAR resize fails

pbus_reassign_bridge_resources() saves bridge windows into the saved list before attempting to adjust resource assignments to perform a BAR resize operation. If resource adjustments cannot be completed fully, rollback is attempted by restoring the resource from the saved list.

The rollback, however, does not check whether the resources it restores were assigned by the partial resize attempt. If restore changes addresses of the resource, it can result in corrupting the resource tree.

An example of a corrupted resource tree with overlapping addresses:

6200000000000-6203fbfffffff : pciex@620c3c0000000
6200000000000-6203fbff0ffff : PCI Bus 0030:01

    6200020000000-62000207fffff : 0030:01:00.0
    6200000000000-6203fbff0ffff : PCI Bus 0030:02

A resource that are assigned into the resource tree must remain unchanged. Thus, release such a resource before attempting to restore and claim it back.

For simplicity, always do the release and claim back for the resource even in the cases where it is restored to the same address range.

Note: this fix may "break" some cases where devices "worked" because the resource tree corruption allowed address space double counting to fit more resource than what can now be assigned without double counting. The upcoming changes to BAR resizing should address those scenarios (to the extent possible).

Fixes 8bb705e3e79d ("PCI: Add pci_resize_resource() for resizing BARs")
---
 drivers/pci/setup-bus.c | 5 +++++
 1 file changed, 5 insertions(+)
drivers/pci/setup-bus.c +5 -0
index 4a8735b275e4..e6984bb530ae 100644
@@ -2504,6 +2504,11 @@ int pbus_reassign_bridge_resources(struct pci_bus *bus, struct resource *res)
 		bridge = dev_res->dev;
 		i = pci_resource_num(bridge, res);
 
+		if (res->parent) {
+			release_child_resources(res);
+			pci_release_resource(bridge, i);
+		}
+
 		restore_dev_resource(dev_res);
 
 		pci_claim_resource(bridge, i);
Signature · 1 line
2.39.5
#2 lore raw

Re: [PATCH 1/9] PCI: Prevent resource tree corruption when BAR resize fails

Quoted 8 lines · Ilpo Järvinen

pbus_reassign_bridge_resources() saves bridge windows into the saved list before attempting to adjust resource assignments to perform a BAR resize operation. If resource adjustments cannot be completed fully, rollback is attempted by restoring the resource from the saved list.

Fixes 8bb705e3e79d ("PCI: Add pci_resize_resource() for resizing BARs")

If these reports were public, can we include lore URLs for them?

Same question for [PATCH 5/9] PCI: Fix restoring BARs on BAR resize rollback path.

I put these all on pci/resource for build testing. I assume we'll tweak these based on testing reports and sorting out the pci/rebar conflicts.

Bjorn

#3 lore raw

Re: [PATCH 1/9] PCI: Prevent resource tree corruption when BAR resize fails

Quoted 10 lines · Bjorn Helgaas
Quoted 8 lines · Ilpo Järvinen

pbus_reassign_bridge_resources() saves bridge windows into the saved list before attempting to adjust resource assignments to perform a BAR resize operation. If resource adjustments cannot be completed fully, rollback is attempted by restoring the resource from the saved list.

Fixes 8bb705e3e79d ("PCI: Add pci_resize_resource() for resizing BARs")
Quoted 8 lines

If these reports were public, can we include lore URLs for them?

Same question for [PATCH 5/9] PCI: Fix restoring BARs on BAR resize rollback path.

I put these all on pci/resource for build testing. I assume we'll tweak these based on testing reports and sorting out the pci/rebar conflicts.

Thanks, the links will come in v2 along with fixing a few things found by more extensive tests by LKP. E.g., it seems clang thinks guard() cannot be used here because goto jumps over it (auto variable initialization gets skipped so it's kind of understandable limitation).

Signature · 1 line
 i.
#4 lore raw

Re: [PATCH 1/9] PCI: Prevent resource tree corruption when BAR resize fails

Quoted 17 lines · Ilpo Järvinen
Quoted 15 lines · Bjorn Helgaas
Quoted 8 lines · Ilpo Järvinen

pbus_reassign_bridge_resources() saves bridge windows into the saved list before attempting to adjust resource assignments to perform a BAR resize operation. If resource adjustments cannot be completed fully, rollback is attempted by restoring the resource from the saved list.

Fixes 8bb705e3e79d ("PCI: Add pci_resize_resource() for resizing BARs")

If these reports were public, can we include lore URLs for them?

Same question for [PATCH 5/9] PCI: Fix restoring BARs on BAR resize rollback path.

Quoted 8 lines
Quoted 3 lines

I put these all on pci/resource for build testing. I assume we'll tweak these based on testing reports and sorting out the pci/rebar conflicts.

Thanks, the links will come in v2 along with fixing a few things found by more extensive tests by LKP. E.g., it seems clang thinks guard() cannot be used here because goto jumps over it (auto variable initialization gets skipped so it's kind of understandable limitation).

Just a ping on this. The lkp robot did build this fine:
https://lore.kernel.org/r/[email protected]

I'm happy to put it in pci/next as-is, especially if the amdgpu and i915 folks are ok with it.

#5 lore raw

PATCH 2/9 PCI/IOV: Adjust ->barsz[] when changing BAR size

In reply to Ilpo Järvinen

pci_rebar_set_size() adjusts BAR size for both normal and IOV BARs. The struct pci_srvio keeps a cached copy of BAR size in unit of resource_size_t in ->barsz[] which is not adjusted by pci_rebar_set_size() but by pci_iov_resource_set_size(). pci_iov_resource_set_size() is called also from pci_resize_resource_set_size().

The current arrangement is problematic once BAR resize algorithm starts to roll back changes properly in case of a failure. The normal resource fitting algorithm rolls back resource size using the struct pci_dev_resource easily but having to call also pci_resize_resource_set_size() or pci_iov_resource_set_size() to roll back BAR size would be an extra burden, whereas combining ->barsz[] update with pci_rebar_set_size() naturally rolls back it when restoring the old BAR size on a different layer of the BAR resize operation.

Thus, rework pci_rebar_set_size() to also update ->barsz[].

---
 drivers/pci/iov.c       | 15 ++++-----------
 drivers/pci/pci.c       |  4 ++++
 drivers/pci/pci.h       |  5 ++---
 drivers/pci/setup-res.c | 10 ++++------
 4 files changed, 14 insertions(+), 20 deletions(-)
drivers/pci/iov.c +4 -11
index 77dee43b7858..04b675e90963 100644
@@ -158,8 +158,7 @@ resource_size_t pci_iov_resource_size(struct pci_dev *dev, int resno)
 	return dev->sriov->barsz[pci_resource_num_to_vf_bar(resno)];
 }
 
-void pci_iov_resource_set_size(struct pci_dev *dev, int resno,
-			       resource_size_t size)
+void pci_iov_resource_set_size(struct pci_dev *dev, int resno, int size)
 {
 	if (!pci_resource_is_iov(resno)) {
 		pci_warn(dev, "%s is not an IOV resource\n",
@@ -167,7 +166,8 @@ void pci_iov_resource_set_size(struct pci_dev *dev, int resno,
 		return;
 	}
 
-	dev->sriov->barsz[pci_resource_num_to_vf_bar(resno)] = size;
+	resno = pci_resource_num_to_vf_bar(resno);
+	dev->sriov->barsz[resno] = pci_rebar_size_to_bytes(size);
 }
 
 bool pci_iov_is_memory_decoding_enabled(struct pci_dev *dev)
@@ -1340,7 +1340,6 @@ EXPORT_SYMBOL_GPL(pci_sriov_configure_simple);
 int pci_iov_vf_bar_set_size(struct pci_dev *dev, int resno, int size)
 {
 	u32 sizes;
-	int ret;
 
 	if (!pci_resource_is_iov(resno))
 		return -EINVAL;
@@ -1355,13 +1354,7 @@ int pci_iov_vf_bar_set_size(struct pci_dev *dev, int resno, int size)
 	if (!(sizes & BIT(size)))
 		return -EINVAL;
 
-	ret = pci_rebar_set_size(dev, resno, size);
-	if (ret)
-		return ret;
-
-	pci_iov_resource_set_size(dev, resno, pci_rebar_size_to_bytes(size));
-
-	return 0;
+	return pci_rebar_set_size(dev, resno, size);
 }
 EXPORT_SYMBOL_GPL(pci_iov_vf_bar_set_size);
 
drivers/pci/pci.c +4 -0
index b14dd064006c..7dfc58b0e55e 100644
@@ -3803,6 +3803,10 @@ int pci_rebar_set_size(struct pci_dev *pdev, int bar, int size)
 	ctrl &= ~PCI_REBAR_CTRL_BAR_SIZE;
 	ctrl |= FIELD_PREP(PCI_REBAR_CTRL_BAR_SIZE, size);
 	pci_write_config_dword(pdev, pos + PCI_REBAR_CTRL, ctrl);
+
+	if (pci_resource_is_iov(bar))
+		pci_iov_resource_set_size(pdev, bar, size);
+
 	return 0;
 }
 
drivers/pci/pci.h +2 -3
index 4492b809094b..bf1a577e9623 100644
@@ -808,8 +808,7 @@ void pci_iov_update_resource(struct pci_dev *dev, int resno);
 resource_size_t pci_sriov_resource_alignment(struct pci_dev *dev, int resno);
 void pci_restore_iov_state(struct pci_dev *dev);
 int pci_iov_bus_range(struct pci_bus *bus);
-void pci_iov_resource_set_size(struct pci_dev *dev, int resno,
-			       resource_size_t size);
+void pci_iov_resource_set_size(struct pci_dev *dev, int resno, int size);
 bool pci_iov_is_memory_decoding_enabled(struct pci_dev *dev);
 static inline u16 pci_iov_vf_rebar_cap(struct pci_dev *dev)
 {
@@ -851,7 +850,7 @@ static inline int pci_iov_bus_range(struct pci_bus *bus)
 	return 0;
 }
 static inline void pci_iov_resource_set_size(struct pci_dev *dev, int resno,
-					     resource_size_t size) { }
+					     int size) { }
 static inline bool pci_iov_is_memory_decoding_enabled(struct pci_dev *dev)
 {
 	return false;
drivers/pci/setup-res.c +4 -6
index c3ba4ccecd43..3d0b0b3f60c4 100644
@@ -450,12 +450,10 @@ static void pci_resize_resource_set_size(struct pci_dev *dev, int resno,
 	resource_size_t res_size = pci_rebar_size_to_bytes(size);
 	struct resource *res = pci_resource_n(dev, resno);
 
-	if (!pci_resource_is_iov(resno)) {
-		resource_set_size(res, res_size);
-	} else {
-		resource_set_size(res, res_size * pci_sriov_get_totalvfs(dev));
-		pci_iov_resource_set_size(dev, resno, res_size);
-	}
+	if (pci_resource_is_iov(resno))
+		res_size *= pci_sriov_get_totalvfs(dev);
+
+	resource_set_size(res, res_size);
 }
 
 int pci_resize_resource(struct pci_dev *dev, int resno, int size)
Signature · 1 line
2.39.5
#6 lore raw

Re: [PATCH 2/9] PCI/IOV: Adjust ->barsz[] when changing BAR size

Quoted 3 lines · Ilpo Järvinen

pci_rebar_set_size() adjusts BAR size for both normal and IOV BARs. The struct pci_srvio keeps a cached copy of BAR size in unit of resource_size_t in ->barsz[] ...

Nit: s/pci_srvio/pci/sriov/ (fixed locally, FYI in case you post a v2)

I'm not sure what "unit of resource_size_t" adds here, maybe could be removed to just say this?

struct pci_srvio keeps a cached copy of BAR size in ->barsz[] ...

#7 lore raw

Re: [PATCH 2/9] PCI/IOV: Adjust ->barsz[] when changing BAR size

Quoted 6 lines · Bjorn Helgaas
Quoted 3 lines · Ilpo Järvinen

pci_rebar_set_size() adjusts BAR size for both normal and IOV BARs. The struct pci_srvio keeps a cached copy of BAR size in unit of resource_size_t in ->barsz[] ...

Nit: s/pci_srvio/pci/sriov/ (fixed locally, FYI in case you post a v2)

I just posted v2 without seeing this first. :-(

I seem to never learn to type those letters in the correct order, I don't know why I always keep typing them wrong.

Quoted 4 lines

I'm not sure what "unit of resource_size_t" adds here, maybe could be removed to just say this?

struct pci_srvio keeps a cached copy of BAR size in ->barsz[] ...

Seems okay with me. I just had it there to differentiate from "BAR size" which happens to often be the format directly compatible with field in the capability.

Signature · 1 line
 i.
#8 lore raw

Re: [PATCH 2/9] PCI/IOV: Adjust ->barsz[] when changing BAR size

Quoted 9 lines · Ilpo Järvinen
Quoted 6 lines · Bjorn Helgaas
Quoted 3 lines · Ilpo Järvinen

pci_rebar_set_size() adjusts BAR size for both normal and IOV BARs. The struct pci_srvio keeps a cached copy of BAR size in unit of resource_size_t in ->barsz[] ...

Nit: s/pci_srvio/pci/sriov/ (fixed locally, FYI in case you post a v2)

I just posted v2 without seeing this first. :-(

Perfect, we crossed in the mail! I'll tweak this locally.

#9 lore raw

Re: [PATCH 2/9] PCI/IOV: Adjust ->barsz[] when changing BAR size

In reply to Ilpo Järvinen
Quoted 7 lines · Ilpo Järvinen
Quoted 4 lines · Bjorn Helgaas
Quoted 3 lines · Ilpo Järvinen

pci_rebar_set_size() adjusts BAR size for both normal and IOV BARs. The struct pci_srvio keeps a cached copy of BAR size in unit of resource_size_t in ->barsz[] ...

Quoted 8 lines
Quoted 4 lines

I'm not sure what "unit of resource_size_t" adds here, maybe could be removed to just say this?

struct pci_srvio keeps a cached copy of BAR size in ->barsz[] ...

Seems okay with me. I just had it there to differentiate from "BAR size" which happens to often be the format directly compatible with field in the capability.

Ah, I see now, "size" used to be in bytes, but now it's the encoded size.

#10 lore raw

PATCH 3/9 PCI: Change pci_dev variable from 'bridge' to 'dev'

In reply to Ilpo Järvinen

Upcoming fix to BAR resize will store also device BAR resource into the saved list. Change the pci_dev variable in the loop from 'bridge' to 'dev' as the former would be misleading with non-bridges in the list.

This is in a separate change to reduce churn in the upcoming BAR resize fix.

While it appears that the logic in the loop doing pci_setup_bridge() is altered as 'bridge' variable is no longer updated, a bridge should never appear more than once in the saved list so the if check can only match to the first entry. As such, the code with two distinct pci_dev variables better represent the intention of the check compared with the old code where bridge variable was reused for different purpose.

---
 drivers/pci/setup-bus.c | 17 +++++++++--------
 1 file changed, 9 insertions(+), 8 deletions(-)
drivers/pci/setup-bus.c +9 -8
index e6984bb530ae..d58f025aeaff 100644
@@ -2479,12 +2479,13 @@ int pbus_reassign_bridge_resources(struct pci_bus *bus, struct resource *res)
 	}
 
 	list_for_each_entry(dev_res, &saved, list) {
+		struct pci_dev *dev = dev_res->dev;
+
 		/* Skip the bridge we just assigned resources for */
-		if (bridge == dev_res->dev)
+		if (bridge == dev)
 			continue;
 
-		bridge = dev_res->dev;
-		pci_setup_bridge(bridge->subordinate);
+		pci_setup_bridge(dev->subordinate);
 	}
 
 	free_list(&saved);
@@ -2500,19 +2501,19 @@ int pbus_reassign_bridge_resources(struct pci_bus *bus, struct resource *res)
 	/* Revert to the old configuration */
 	list_for_each_entry(dev_res, &saved, list) {
 		struct resource *res = dev_res->res;
+		struct pci_dev *dev = dev_res->dev;
 
-		bridge = dev_res->dev;
-		i = pci_resource_num(bridge, res);
+		i = pci_resource_num(dev, res);
 
 		if (res->parent) {
 			release_child_resources(res);
-			pci_release_resource(bridge, i);
+			pci_release_resource(dev, i);
 		}
 
 		restore_dev_resource(dev_res);
 
-		pci_claim_resource(bridge, i);
-		pci_setup_bridge(bridge->subordinate);
+		pci_claim_resource(dev, i);
+		pci_setup_bridge(dev->subordinate);
 	}
 	free_list(&saved);
 	up_read(&pci_bus_sem);
Signature · 1 line
2.39.5
#11 lore raw

PATCH 4/9 PCI: Try BAR resize even when no window was released

In reply to Ilpo Järvinen

Usually, resizing BARs requires releasing bridge windows in order to resize it to fit a larged BAR into the window. This is not always the case, however, FW could have made the window large enough to accomodate larger BAR as is, or the user might prefer to shrink a BAR to make more space for another Resizable BAR.

Thus, replace the check that requires that at least one bridge window was released with a check that simply ensures bridge is not NULL.

---
 drivers/pci/setup-bus.c | 4 +---
 1 file changed, 1 insertion(+), 3 deletions(-)
drivers/pci/setup-bus.c +1 -3
index d58f025aeaff..76a4259ab076 100644
@@ -2459,10 +2459,8 @@ int pbus_reassign_bridge_resources(struct pci_bus *bus, struct resource *res)
 		bus = bus->parent;
 	}
 
-	if (list_empty(&saved)) {
-		up_read(&pci_bus_sem);
+	if (!bridge)
 		return -ENOENT;
-	}
 
 	__pci_bus_size_bridges(bridge->subordinate, &added);
 	__pci_bridge_assign_resources(bridge, &added, &failed);
Signature · 1 line
2.39.5
#12 lore raw

PATCH 5/9 PCI: Fix restoring BARs on BAR resize rollback path

In reply to Ilpo Järvinen

BAR resize operation is implemented in the pci_resize_resource() and pbus_reassign_bridge_resources() functions. pci_resize_resource() can be called either from __resource_resize_store() from sysfs or directly by the driver for the Endpoint Device.

The pci_resize_resource() requires that caller has released the device resources that share the bridge window with the BAR to be resized as otherwise the bridge window is pinned in place and cannot be changed.

pbus_reassign_bridge_resources() implement rollback of the resources if the resize operation fails, but rollback is performed only for the bridge windows. Because releasing the device resources are done by the caller of the BAR resize interface, these functions performing the BAR resize do not have access to the device resources as they were before the resize.

pbus_reassign_bridge_resources() could try to __pci_bridge_assign_resources() after rolling back the bridge windows as they were, however, it will not guarantee the resource are assigned due to differences how FW and the kernel may want to assign the resources (alignment of the start address and tail).

In order to perform rollback robustly, the BAR resize interface has to be altered to also release the device resources that share the bridge window with the BAR to be resized.

Also, remove restoring from the entries failed list as saved list should now contain both the bridge windows and device resources so the extra restore is duplicated work.

This change (together with the driver side changes) is to counter the resource releases that had to be done to prevent resource tree corruption in the ("PCI: Release assigned resource before restoring them") change. As such, it likely restores functionality in cases where device resources were released to avoid resource tree conflicts which appeared to be "working" when such conflicts were not correctly detected by the kernel.

---
 drivers/pci/pci-sysfs.c | 15 +------
 drivers/pci/pci.h       |  3 +-
 drivers/pci/setup-bus.c | 95 ++++++++++++++++++++++++++++++-----------
 drivers/pci/setup-res.c | 20 ++-------
 4 files changed, 77 insertions(+), 56 deletions(-)
drivers/pci/pci-sysfs.c +1 -14
index 9d6f74bd95f8..caffd20abb9f 100644
@@ -1599,18 +1599,13 @@ static ssize_t __resource_resize_store(struct device *dev, int n,
 {
 	struct pci_dev *pdev = to_pci_dev(dev);
 	struct pci_bus *bus = pdev->bus;
-	struct resource *b_win, *res;
 	unsigned long size;
-	int ret, i;
+	int ret;
 	u16 cmd;
 
 	if (kstrtoul(buf, 0, &size) < 0)
 		return -EINVAL;
 
-	b_win = pbus_select_window(bus, pci_resource_n(pdev, n));
-	if (!b_win)
-		return -EINVAL;
-
 	device_lock(dev);
 	if (dev->driver || pci_num_vf(pdev)) {
 		ret = -EBUSY;
@@ -1632,14 +1627,6 @@ static ssize_t __resource_resize_store(struct device *dev, int n,
 
 	pci_remove_resource_files(pdev);
 
-	pci_dev_for_each_resource(pdev, res, i) {
-		if (i >= PCI_BRIDGE_RESOURCES)
-			break;
-
-		if (b_win == pbus_select_window(bus, res))
-			pci_release_resource(pdev, i);
-	}
-
 	ret = pci_resize_resource(pdev, n, size);
 
 	pci_assign_unassigned_bus_resources(bus);
drivers/pci/pci.h +2 -1
index bf1a577e9623..d22e595b3891 100644
@@ -421,8 +421,9 @@ enum pci_bar_type {
 struct device *pci_get_host_bridge_device(struct pci_dev *dev);
 void pci_put_host_bridge_device(struct device *dev);
 
+void pci_resize_resource_set_size(struct pci_dev *dev, int resno, int size);
+int pci_do_resource_release_and_resize(struct pci_dev *dev, int resno, int size);
 unsigned int pci_rescan_bus_bridge_resize(struct pci_dev *bridge);
-int pbus_reassign_bridge_resources(struct pci_bus *bus, struct resource *res);
 int __must_check pci_reassign_resource(struct pci_dev *dev, int i, resource_size_t add_size, resource_size_t align);
 
 int pci_configure_extended_tags(struct pci_dev *dev, void *ign);
drivers/pci/setup-bus.c +70 -25

144 lines not on this page · raw message

drivers/pci/setup-res.c +4 -16
index 3d0b0b3f60c4..93c70f8a8552 100644
@@ -444,8 +444,7 @@ static bool pci_resize_is_memory_decoding_enabled(struct pci_dev *dev,
 	return cmd & PCI_COMMAND_MEMORY;
 }
 
-static void pci_resize_resource_set_size(struct pci_dev *dev, int resno,
-					 int size)
+void pci_resize_resource_set_size(struct pci_dev *dev, int resno, int size)
 {
 	resource_size_t res_size = pci_rebar_size_to_bytes(size);
 	struct resource *res = pci_resource_n(dev, resno);
@@ -458,7 +457,6 @@ static void pci_resize_resource_set_size(struct pci_dev *dev, int resno,
 
 int pci_resize_resource(struct pci_dev *dev, int resno, int size)
 {
-	struct resource *res = pci_resource_n(dev, resno);
 	struct pci_host_bridge *host;
 	int old, ret;
 	u32 sizes;
@@ -468,10 +466,6 @@ int pci_resize_resource(struct pci_dev *dev, int resno, int size)
 	if (host->preserve_config)
 		return -ENOTSUPP;
 
-	/* Make sure the resource isn't assigned before resizing it. */
-	if (!(res->flags & IORESOURCE_UNSET))
-		return -EBUSY;
-
 	if (pci_resize_is_memory_decoding_enabled(dev, resno))
 		return -EBUSY;
 
@@ -490,19 +484,13 @@ int pci_resize_resource(struct pci_dev *dev, int resno, int size)
 	if (ret)
 		return ret;
 
-	pci_resize_resource_set_size(dev, resno, size);
-
-	/* Check if the new config works by trying to assign everything. */
-	if (dev->bus->self) {
-		ret = pbus_reassign_bridge_resources(dev->bus, res);
-		if (ret)
-			goto error_resize;
-	}
+	ret = pci_do_resource_release_and_resize(dev, resno, size);
+	if (ret)
+		goto error_resize;
 	return 0;
 
 error_resize:
 	pci_rebar_set_size(dev, resno, old);
-	pci_resize_resource_set_size(dev, resno, old);
 	return ret;
 }
 EXPORT_SYMBOL(pci_resize_resource);
Signature · 1 line
2.39.5
#13 lore raw

PATCH 6/9 drm/xe: Remove driver side BAR release before resize

In reply to Ilpo Järvinen

PCI core handles releasing device's resources and their rollback in case of failure of a BAR resizing operation. Releasing resource prior to calling pci_resize_resource() prevents PCI core from restoring the BARs as they were.

Remove driver-side release of BARs from the xe driver.

---
 drivers/gpu/drm/xe/xe_vram.c | 3 ---
 1 file changed, 3 deletions(-)
drivers/gpu/drm/xe/xe_vram.c +0 -3
index b44ebf50fedb..929412f0d131 100644
@@ -33,9 +33,6 @@ _resize_bar(struct xe_device *xe, int resno, resource_size_t size)
 	int bar_size = pci_rebar_bytes_to_size(size);
 	int ret;
 
-	if (pci_resource_len(pdev, resno))
-		pci_release_resource(pdev, resno);
-
 	ret = pci_resize_resource(pdev, resno, bar_size);
 	if (ret) {
 		drm_info(&xe->drm, "Failed to resize BAR%d to %dM (%pe). Consider enabling 'Resizable BAR' support in your BIOS\n",
Signature · 1 line
2.39.5
#14 lore raw

Re: [PATCH 6/9] drm/xe: Remove driver side BAR release before resize

Quoted 18 lines · Ilpo Järvinen

PCI core handles releasing device's resources and their rollback in case of failure of a BAR resizing operation. Releasing resource prior to calling pci_resize_resource() prevents PCI core from restoring the BARs as they were.

Remove driver-side release of BARs from the xe driver.

---
drivers/gpu/drm/xe/xe_vram.c | 3 ---

1 file changed, 3 deletions(-)
drivers/gpu/drm/xe/xe_vram.c +0 -0
index b44ebf50fedb..929412f0d131 100644

drivers/gpu/drm/xe/xe_vram.c _resize_bar(struct xe_device *xe, int resno, resource_size_t size) · line 38

Quoted 6 lines
drivers/gpu/drm/xe/xe_vram.c +0 -3
@@ -33,9 +33,6 @@ _resize_bar(struct xe_device *xe, int resno, resource_size_t size)
 	int bar_size = pci_rebar_bytes_to_size(size);
 	int ret;
 
-	if (pci_resource_len(pdev, resno))
-		pci_release_resource(pdev, resno);
-

conflict with drm-xe-next:

++<<<<<<< ours
+ release_bars(pdev);
+
++=======
++>>>>>>> theirs

if we don't need to release the BARs anymore to call pci_resize_resource(), then the resolution is simply to drop the function release_bars() function.

I'm sending that to our CI for coverage:
https://lore.kernel.org/intel-xe/[email protected]/T/#u

thanks
Lucas De Marchi

Quoted 6 lines
drivers/gpu/drm/xe/xe_vram.c +0 -0
 	ret = pci_resize_resource(pdev, resno, bar_size);
 	if (ret) {
 		drm_info(&xe->drm, "Failed to resize BAR%d to %dM (%pe). Consider enabling 'Resizable BAR' support in your BIOS\n",
Signature · 1 line
2.39.5
#15 lore raw

Re: [PATCH 6/9] drm/xe: Remove driver side BAR release before resize

Quoted 34 lines · Lucas De Marchi
Quoted 18 lines · Ilpo Järvinen

PCI core handles releasing device's resources and their rollback in case of failure of a BAR resizing operation. Releasing resource prior to calling pci_resize_resource() prevents PCI core from restoring the BARs as they were.

Remove driver-side release of BARs from the xe driver.

---
drivers/gpu/drm/xe/xe_vram.c | 3 ---

1 file changed, 3 deletions(-)
drivers/gpu/drm/xe/xe_vram.c +0 -0
index b44ebf50fedb..929412f0d131 100644

drivers/gpu/drm/xe/xe_vram.c _resize_bar(struct xe_device *xe, int resno, resource_size_t size) · line 38

Quoted 6 lines
drivers/gpu/drm/xe/xe_vram.c +0 -3
@@ -33,9 +33,6 @@ _resize_bar(struct xe_device *xe, int resno, resource_size_t size)
 	int bar_size = pci_rebar_bytes_to_size(size);
 	int ret;
 
-	if (pci_resource_len(pdev, resno))
-		pci_release_resource(pdev, resno);
-

conflict with drm-xe-next:

++<<<<<<< ours

(quoted diff) +4 -0
+      release_bars(pdev);
+
++=======
++>>>>>>> theirs
Quoted 6 lines

if we don't need to release the BARs anymore to call pci_resize_resource(), then the resolution is simply to drop the function release_bars() function.

I'm sending that to our CI for coverage:
https://lore.kernel.org/intel-xe/[email protected]/T/#u

CI came back clean. Looks good from xe side:

thanks
Lucas De Marchi

Quoted 10 lines

thanks
Lucas De Marchi

Quoted 6 lines
drivers/gpu/drm/xe/xe_vram.c +0 -0
 	ret = pci_resize_resource(pdev, resno, bar_size);
 	if (ret) {
 		drm_info(&xe->drm, "Failed to resize BAR%d to %dM (%pe). Consider enabling 'Resizable BAR' support in your BIOS\n",
Signature · 1 line
2.39.5
#16 lore raw

PATCH 7/9 drm/i915: Remove driver side BAR release before resize

In reply to Ilpo Järvinen

PCI core handles releasing device's resources and their rollback in case of failure of a BAR resizing operation. Releasing resource prior to calling pci_resize_resource() prevents PCI core from restoring the BARs as they were.

Remove driver-side release of BARs from the i915 driver.

---
 drivers/gpu/drm/i915/gt/intel_region_lmem.c | 12 ------------
 1 file changed, 12 deletions(-)
drivers/gpu/drm/i915/gt/intel_region_lmem.c +0 -12
index 51bb27e10a4f..ca3de61451a3 100644
@@ -18,16 +18,6 @@
 #include "gt/intel_gt_regs.h"
 
 #ifdef CONFIG_64BIT
-static void _release_bars(struct pci_dev *pdev)
-{
-	int resno;
-
-	for (resno = PCI_STD_RESOURCES; resno < PCI_STD_RESOURCE_END; resno++) {
-		if (pci_resource_len(pdev, resno))
-			pci_release_resource(pdev, resno);
-	}
-}
-
 static void
 _resize_bar(struct drm_i915_private *i915, int resno, resource_size_t size)
 {
@@ -35,8 +25,6 @@ _resize_bar(struct drm_i915_private *i915, int resno, resource_size_t size)
 	int bar_size = pci_rebar_bytes_to_size(size);
 	int ret;
 
-	_release_bars(pdev);
-
 	ret = pci_resize_resource(pdev, resno, bar_size);
 	if (ret) {
 		drm_info(&i915->drm, "Failed to resize BAR%d to %dM (%pe)\n",
Signature · 1 line
2.39.5
#17 lore raw

Re: [PATCH 7/9] drm/i915: Remove driver side BAR release before resize

i915 folks, any objection to this?

Quoted 45 lines · Ilpo Järvinen

PCI core handles releasing device's resources and their rollback in case of failure of a BAR resizing operation. Releasing resource prior to calling pci_resize_resource() prevents PCI core from restoring the BARs as they were.

Remove driver-side release of BARs from the i915 driver.

---
 drivers/gpu/drm/i915/gt/intel_region_lmem.c | 12 ------------
 1 file changed, 12 deletions(-)
drivers/gpu/drm/i915/gt/intel_region_lmem.c +0 -12
index 51bb27e10a4f..ca3de61451a3 100644
@@ -18,16 +18,6 @@
 #include "gt/intel_gt_regs.h"
 
 #ifdef CONFIG_64BIT
-static void _release_bars(struct pci_dev *pdev)
-{
-	int resno;
-
-	for (resno = PCI_STD_RESOURCES; resno < PCI_STD_RESOURCE_END; resno++) {
-		if (pci_resource_len(pdev, resno))
-			pci_release_resource(pdev, resno);
-	}
-}
-
 static void
 _resize_bar(struct drm_i915_private *i915, int resno, resource_size_t size)
 {
@@ -35,8 +25,6 @@ _resize_bar(struct drm_i915_private *i915, int resno, resource_size_t size)
 	int bar_size = pci_rebar_bytes_to_size(size);
 	int ret;
 
-	_release_bars(pdev);
-
 	ret = pci_resize_resource(pdev, resno, bar_size);
 	if (ret) {
 		drm_info(&i915->drm, "Failed to resize BAR%d to %dM (%pe)\n",
Signature · 1 line
2.39.5
#18 lore raw

PATCH 8/9 drm/amdgpu: Remove driver side BAR release before resize

In reply to Ilpo Järvinen

PCI core handles releasing device's resources and their rollback in case of failure of a BAR resizing operation. Releasing resource prior to calling pci_resize_resource() prevents PCI core from restoring the BARs as they were.

Remove driver-side release of BARs from the amdgpu driver.

Also remove the driver initiated assignment as pci_resize_resource() should try to assign as much as possible. If the driver side call manages to get more required resources assigned in some scenario, such a problem should be fixed inside pci_resize_resource() instead.

---
 drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 8 +-------
 1 file changed, 1 insertion(+), 7 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +1 -7
index 7a899fb4de29..65474d365229 100644
@@ -1729,12 +1729,8 @@ int amdgpu_device_resize_fb_bar(struct amdgpu_device *adev)
 	pci_write_config_word(adev->pdev, PCI_COMMAND,
 			      cmd & ~PCI_COMMAND_MEMORY);
 
-	/* Free the VRAM and doorbell BAR, we most likely need to move both. */
+	/* Tear down doorbell as resizing will release BARs */
 	amdgpu_doorbell_fini(adev);
-	if (adev->asic_type >= CHIP_BONAIRE)
-		pci_release_resource(adev->pdev, 2);
-
-	pci_release_resource(adev->pdev, 0);
 
 	r = pci_resize_resource(adev->pdev, 0, rbar_size);
 	if (r == -ENOSPC)
@@ -1743,8 +1739,6 @@ int amdgpu_device_resize_fb_bar(struct amdgpu_device *adev)
 	else if (r && r != -ENOTSUPP)
 		dev_err(adev->dev, "Problem resizing BAR0 (%d).", r);
 
-	pci_assign_unassigned_bus_resources(adev->pdev->bus);
-
 	/* When the doorbell or fb BAR isn't available we have no chance of
 	 * using the device.
 	 */
Signature · 1 line
2.39.5
#19 lore raw

Re: [PATCH 8/9] drm/amdgpu: Remove driver side BAR release before resize

amdgpu folks, any objection to this?

Quoted 47 lines · Ilpo Järvinen

PCI core handles releasing device's resources and their rollback in case of failure of a BAR resizing operation. Releasing resource prior to calling pci_resize_resource() prevents PCI core from restoring the BARs as they were.

Remove driver-side release of BARs from the amdgpu driver.

Also remove the driver initiated assignment as pci_resize_resource() should try to assign as much as possible. If the driver side call manages to get more required resources assigned in some scenario, such a problem should be fixed inside pci_resize_resource() instead.

---
 drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 8 +-------
 1 file changed, 1 insertion(+), 7 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +1 -7
index 7a899fb4de29..65474d365229 100644
@@ -1729,12 +1729,8 @@ int amdgpu_device_resize_fb_bar(struct amdgpu_device *adev)
 	pci_write_config_word(adev->pdev, PCI_COMMAND,
 			      cmd & ~PCI_COMMAND_MEMORY);
 
-	/* Free the VRAM and doorbell BAR, we most likely need to move both. */
+	/* Tear down doorbell as resizing will release BARs */
 	amdgpu_doorbell_fini(adev);
-	if (adev->asic_type >= CHIP_BONAIRE)
-		pci_release_resource(adev->pdev, 2);
-
-	pci_release_resource(adev->pdev, 0);
 
 	r = pci_resize_resource(adev->pdev, 0, rbar_size);
 	if (r == -ENOSPC)
@@ -1743,8 +1739,6 @@ int amdgpu_device_resize_fb_bar(struct amdgpu_device *adev)
 	else if (r && r != -ENOTSUPP)
 		dev_err(adev->dev, "Problem resizing BAR0 (%d).", r);
 
-	pci_assign_unassigned_bus_resources(adev->pdev->bus);
-
 	/* When the doorbell or fb BAR isn't available we have no chance of
 	 * using the device.
 	 */
Signature · 1 line
2.39.5
maintainer
#20 lore raw

Re: [PATCH 8/9] drm/amdgpu: Remove driver side BAR release before resize

In reply to Ilpo Järvinen

Sorry for the late reply I'm really busy at the moment.

Quoted 4 lines · Ilpo Järvinen

PCI core handles releasing device's resources and their rollback in case of failure of a BAR resizing operation. Releasing resource prior to calling pci_resize_resource() prevents PCI core from restoring the BARs as they were.

I've intentionally didn't do it this way because at least on AMD HW we could only release the VRAM and doorbell BAR (both 64bit), but not the register BAR (32bit only).

This patch set looks like the right thing in general, but which BARs are now released by pci_resize_resource()?

If we avoid releasing the 32bit BAR as well then that should work, otherwise we will probably cause problems.

Regards,
Christian.

Quoted 40 lines

Remove driver-side release of BARs from the amdgpu driver.

Also remove the driver initiated assignment as pci_resize_resource() should try to assign as much as possible. If the driver side call manages to get more required resources assigned in some scenario, such a problem should be fixed inside pci_resize_resource() instead.

---
 drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 8 +-------
 1 file changed, 1 insertion(+), 7 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +1 -7
index 7a899fb4de29..65474d365229 100644
@@ -1729,12 +1729,8 @@ int amdgpu_device_resize_fb_bar(struct amdgpu_device *adev)
 	pci_write_config_word(adev->pdev, PCI_COMMAND,
 			      cmd & ~PCI_COMMAND_MEMORY);
 
-	/* Free the VRAM and doorbell BAR, we most likely need to move both. */
+	/* Tear down doorbell as resizing will release BARs */
 	amdgpu_doorbell_fini(adev);
-	if (adev->asic_type >= CHIP_BONAIRE)
-		pci_release_resource(adev->pdev, 2);
-
-	pci_release_resource(adev->pdev, 0);
 
 	r = pci_resize_resource(adev->pdev, 0, rbar_size);
 	if (r == -ENOSPC)
@@ -1743,8 +1739,6 @@ int amdgpu_device_resize_fb_bar(struct amdgpu_device *adev)
 	else if (r && r != -ENOTSUPP)
 		dev_err(adev->dev, "Problem resizing BAR0 (%d).", r);
 
-	pci_assign_unassigned_bus_resources(adev->pdev->bus);
-
 	/* When the doorbell or fb BAR isn't available we have no chance of
 	 * using the device.
 	 */
#21 lore raw

Re: [PATCH 8/9] drm/amdgpu: Remove driver side BAR release before resize

Quoted 12 lines · Christian König

Sorry for the late reply I'm really busy at the moment.

Quoted 4 lines · Ilpo Järvinen

PCI core handles releasing device's resources and their rollback in case of failure of a BAR resizing operation. Releasing resource prior to calling pci_resize_resource() prevents PCI core from restoring the BARs as they were.

I've intentionally didn't do it this way because at least on AMD HW we could only release the VRAM and doorbell BAR (both 64bit), but not the register BAR (32bit only).

Quoted 5 lines

This patch set looks like the right thing in general, but which BARs are now released by pci_resize_resource()?

If we avoid releasing the 32bit BAR as well then that should work, otherwise we will probably cause problems.

After these changes, pci_resize_resource() releases BARs that share the bridge window with the BAR to be resized. So the answer depends on the upstream bridge.

However, amdgpu_device_resize_fb_bar() also checks that root bus has a resource with a 64-bit address. That won't tell what the nearest bridge has though. Maybe that check should be converted to check the resources of the nearest bus instead? It would make it impossible to have the 32-bit resource share the bridge window with the 64-bit resources so the resize would be safe.

Thanks a lot for checking this out!

Signature · 1 line
 i.
Quoted 44 lines

Regards,
Christian.

Quoted 40 lines

Remove driver-side release of BARs from the amdgpu driver.

Also remove the driver initiated assignment as pci_resize_resource() should try to assign as much as possible. If the driver side call manages to get more required resources assigned in some scenario, such a problem should be fixed inside pci_resize_resource() instead.

---
 drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 8 +-------
 1 file changed, 1 insertion(+), 7 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +1 -7
index 7a899fb4de29..65474d365229 100644
@@ -1729,12 +1729,8 @@ int amdgpu_device_resize_fb_bar(struct amdgpu_device *adev)
 	pci_write_config_word(adev->pdev, PCI_COMMAND,
 			      cmd & ~PCI_COMMAND_MEMORY);
 
-	/* Free the VRAM and doorbell BAR, we most likely need to move both. */
+	/* Tear down doorbell as resizing will release BARs */
 	amdgpu_doorbell_fini(adev);
-	if (adev->asic_type >= CHIP_BONAIRE)
-		pci_release_resource(adev->pdev, 2);
-
-	pci_release_resource(adev->pdev, 0);
 
 	r = pci_resize_resource(adev->pdev, 0, rbar_size);
 	if (r == -ENOSPC)
@@ -1743,8 +1739,6 @@ int amdgpu_device_resize_fb_bar(struct amdgpu_device *adev)
 	else if (r && r != -ENOTSUPP)
 		dev_err(adev->dev, "Problem resizing BAR0 (%d).", r);
 
-	pci_assign_unassigned_bus_resources(adev->pdev->bus);
-
 	/* When the doorbell or fb BAR isn't available we have no chance of
 	 * using the device.
 	 */
maintainer
#22 lore raw

Re: [PATCH 8/9] drm/amdgpu: Remove driver side BAR release before resize

Quoted 24 lines · Ilpo Järvinen
Quoted 12 lines · Christian König

Sorry for the late reply I'm really busy at the moment.

Quoted 4 lines · Ilpo Järvinen

PCI core handles releasing device's resources and their rollback in case of failure of a BAR resizing operation. Releasing resource prior to calling pci_resize_resource() prevents PCI core from restoring the BARs as they were.

I've intentionally didn't do it this way because at least on AMD HW we could only release the VRAM and doorbell BAR (both 64bit), but not the register BAR (32bit only).

Quoted 5 lines

This patch set looks like the right thing in general, but which BARs are now released by pci_resize_resource()?

If we avoid releasing the 32bit BAR as well then that should work, otherwise we will probably cause problems.

After these changes, pci_resize_resource() releases BARs that share the bridge window with the BAR to be resized. So the answer depends on the upstream bridge.

Quoted 6 lines

However, amdgpu_device_resize_fb_bar() also checks that root bus has a resource with a 64-bit address. That won't tell what the nearest bridge has though. Maybe that check should be converted to check the resources of the nearest bus instead? It would make it impossible to have the 32-bit resource share the bridge window with the 64-bit resources so the resize would be safe.

Mhm, I don't think that will work.

I've added the check for the root bus to avoid a couple of issues during resize, but checking the nearest bridge would block a whole bunch of use cases and isn't even 100% save.

See one use case of this is that all the BARs of the device start in the same 32bit bridge window (or a mixture of 64bit and 32bit window).

What we have is that BAR 0 and 2 are 64bit BARs which can (after some preparation) move around freely. But IIRC BAR 4 are the legacy I/O ports and BAR 5 is the 32bit MMIO registers (don't nail me on that, could be just the other way around).

Especially that 32bit MMIO BAR *can't* move! Not only because it is 32bit, but also because the amdgpu driver as well as the HW itself through the VGA emulation, as well as the EFI/VESA/VBIOS code might reference its absolute address.

Could we give pci_resize_resource() a mask of BARs which are save to release? Or maybe a flag to indicate that it can only free up 64bit BARs?

Regards,
Christian.

Quoted 3 lines

Thanks a lot for checking this out!

#23 lore raw

Re: [PATCH 8/9] drm/amdgpu: Remove driver side BAR release before resize

Quoted 35 lines · Christian König
Quoted 24 lines · Ilpo Järvinen
Quoted 12 lines · Christian König

Sorry for the late reply I'm really busy at the moment.

Quoted 4 lines · Ilpo Järvinen

PCI core handles releasing device's resources and their rollback in case of failure of a BAR resizing operation. Releasing resource prior to calling pci_resize_resource() prevents PCI core from restoring the BARs as they were.

I've intentionally didn't do it this way because at least on AMD HW we could only release the VRAM and doorbell BAR (both 64bit), but not the register BAR (32bit only).

Quoted 5 lines

This patch set looks like the right thing in general, but which BARs are now released by pci_resize_resource()?

If we avoid releasing the 32bit BAR as well then that should work, otherwise we will probably cause problems.

After these changes, pci_resize_resource() releases BARs that share the bridge window with the BAR to be resized. So the answer depends on the upstream bridge.

Quoted 6 lines

However, amdgpu_device_resize_fb_bar() also checks that root bus has a resource with a 64-bit address. That won't tell what the nearest bridge has though. Maybe that check should be converted to check the resources of the nearest bus instead? It would make it impossible to have the 32-bit resource share the bridge window with the 64-bit resources so the resize would be safe.

Mhm, I don't think that will work.

Quoted 6 lines

I've added the check for the root bus to avoid a couple of issues during resize, but checking the nearest bridge would block a whole bunch of use cases and isn't even 100% save.

See one use case of this is that all the BARs of the device start in the same 32bit bridge window (or a mixture of 64bit and 32bit window).

"32bit bridge window" is ambiguous. There are non-prefetchable and prefetchable bridge windows, out of which the latter can be 64-bit as well. Which one you're talking about?

If a 64-bit prefetchable window exists, pbus_size_mem() nor __pci_assign_resource() would not have produced such a configuration where they're put into the same bridge window, even before the commit ae88d0b9c57f ("PCI: Use pbus_select_window_for_type() during mem window sizing") (I think). Now pbus_size_mem() certainly doesn't.

Quoted 9 lines

What we have is that BAR 0 and 2 are 64bit BARs which can (after some preparation) move around freely. But IIRC BAR 4 are the legacy I/O ports and BAR 5 is the 32bit MMIO registers (don't nail me on that, could be just the other way around).

Especially that 32bit MMIO BAR *can't* move! Not only because it is 32bit, but also because the amdgpu driver as well as the HW itself through the VGA emulation, as well as the EFI/VESA/VBIOS code might reference its absolute address.

So if the 64-bit check is replaced with this:

+       /* Check if the parent bridge has a 64-bit (pref) memory resource */
+       res = pci_resource_n(adev->pdev, 0)->parent;
+       /* Trying to resize is pointless without a window above 4GB */
+       if (!(res->flags & IORESOURCE_MEM_64))
		return 0;

...I don't think it's possible for 32-bit resource to share that window under _any_ circumstance.

If you say that ->parent somehow points to a non-IORESOURCE_MEM_64 window at this point, you're implying allocation for the 64-bit prefetchable window was tried and failed, and __pci_assign_resource() then used one of its fallbacks.

Are you saying that "some preparation" includes making room for that 64-bit prefetchable window that failed to assign earlier as I cannot see how else it would ever get assigned so that the 64-bit BARs could be moved there?

Quoted 2 lines

Could we give pci_resize_resource() a mask of BARs which are save to release?

It is possible.

Quoted 9 lines

Or maybe a flag to indicate that it can only free up 64bit BARs?

Regards,
Christian.

Quoted 3 lines

Thanks a lot for checking this out!

Signature · 1 line
 i.
maintainer
#24 lore raw

Re: [PATCH 8/9] drm/amdgpu: Remove driver side BAR release before resize

Quoted 41 lines · Ilpo Järvinen
Quoted 39 lines · Christian König
Quoted 24 lines · Ilpo Järvinen
Quoted 12 lines · Christian König

Sorry for the late reply I'm really busy at the moment.

Quoted 4 lines · Ilpo Järvinen

PCI core handles releasing device's resources and their rollback in case of failure of a BAR resizing operation. Releasing resource prior to calling pci_resize_resource() prevents PCI core from restoring the BARs as they were.

I've intentionally didn't do it this way because at least on AMD HW we could only release the VRAM and doorbell BAR (both 64bit), but not the register BAR (32bit only).

Quoted 5 lines

This patch set looks like the right thing in general, but which BARs are now released by pci_resize_resource()?

If we avoid releasing the 32bit BAR as well then that should work, otherwise we will probably cause problems.

After these changes, pci_resize_resource() releases BARs that share the bridge window with the BAR to be resized. So the answer depends on the upstream bridge.

Quoted 6 lines

However, amdgpu_device_resize_fb_bar() also checks that root bus has a resource with a 64-bit address. That won't tell what the nearest bridge has though. Maybe that check should be converted to check the resources of the nearest bus instead? It would make it impossible to have the 32-bit resource share the bridge window with the 64-bit resources so the resize would be safe.

Mhm, I don't think that will work.

I've added the check for the root bus to avoid a couple of issues during resize, but checking the nearest bridge would block a whole bunch of use cases and isn't even 100% save.

Quoted 6 lines
Quoted 2 lines

See one use case of this is that all the BARs of the device start in the same 32bit bridge window (or a mixture of 64bit and 32bit window).

"32bit bridge window" is ambiguous. There are non-prefetchable and prefetchable bridge windows, out of which the latter can be 64-bit as well. Which one you're talking about?

The non-prefetchable 32bit window.

Quoted 5 lines

If a 64-bit prefetchable window exists, pbus_size_mem() nor __pci_assign_resource() would not have produced such a configuration where they're put into the same bridge window, even before the commit ae88d0b9c57f ("PCI: Use pbus_select_window_for_type() during mem window sizing") (I think). Now pbus_size_mem() certainly doesn't.

I need to double check, but if I'm not completely mistaken that is assigned by the BIOS.

Here is an example of a "good" configuration where both VRAM (BAR0) and doorbell (BAR2) is in the prefetchable window and MMIO in the non-prefetchable:

Device:

	Region 0: Memory at 80000000 (64-bit, prefetchable) [size=256M]
	Region 2: Memory at 90000000 (64-bit, prefetchable) [size=2M]
	Region 4: I/O ports at 3000 [size=256]
	Region 5: Memory at 9f300000 (32-bit, non-prefetchable) [size=1M]

Bridge:

	Memory behind bridge: 9f300000-9f4fffff [size=2M] [32-bit]
	Prefetchable memory behind bridge: 80000000-901fffff [size=258M] [32-bit]

And here is an example of another system where things are mixed up:

Device:

	Region 0: Memory at 2c00000000 (64-bit, prefetchable) [size=256M]
	Region 2: Memory at 94000000 (64-bit, prefetchable) [size=2M]
	Region 4: I/O ports at 1000 [size=256]
	Region 5: Memory at 94600000 (32-bit, non-prefetchable) [size=512K]

Bridge:

	Memory behind bridge: 94000000-946fffff [size=7M] [32-bit]
	Prefetchable memory behind bridge: 2c00000000-2c107fffff [size=264M] [32-bit]

In that example the doorbell ended up in the non-prefetchable window for some reason. And that config comes in all possible variations.

On AMD GPUs both BAR0 and BAR2 are resizeable.

So far we have only implemented resizing of BAR0, but essentially we want to have both for some use cases.

Quoted 18 lines
Quoted 9 lines

What we have is that BAR 0 and 2 are 64bit BARs which can (after some preparation) move around freely. But IIRC BAR 4 are the legacy I/O ports and BAR 5 is the 32bit MMIO registers (don't nail me on that, could be just the other way around).

Especially that 32bit MMIO BAR *can't* move! Not only because it is 32bit, but also because the amdgpu driver as well as the HW itself through the VGA emulation, as well as the EFI/VESA/VBIOS code might reference its absolute address.

So if the 64-bit check is replaced with this:

(quoted diff) +4 -0
+       /* Check if the parent bridge has a 64-bit (pref) memory resource */
+       res = pci_resource_n(adev->pdev, 0)->parent;
+       /* Trying to resize is pointless without a window above 4GB */
+       if (!(res->flags & IORESOURCE_MEM_64))
		return 0;
Quoted 2 lines

...I don't think it's possible for 32-bit resource to share that window under _any_ circumstance.

Well see the example above. I have SSH access to a system where exactly that is the configuration.

Quoted 4 lines

If you say that ->parent somehow points to a non-IORESOURCE_MEM_64 window at this point, you're implying allocation for the 64-bit prefetchable window was tried and failed, and __pci_assign_resource() then used one of its fallbacks.

No, as I said that comes from the BIOS.

Quoted 4 lines

Are you saying that "some preparation" includes making room for that 64-bit prefetchable window that failed to assign earlier as I cannot see how else it would ever get assigned so that the 64-bit BARs could be moved there?

No, at least from the amdgpu driver side we don't touch the resource allocation at all.

In this case preparation means disabling the VGA emulation, cause otherwise trying to resize the BAR can just cause a spontaneous system reboot for some reason.

Quoted 4 lines
Quoted 2 lines

Could we give pci_resize_resource() a mask of BARs which are save to release?

It is possible.

Then let us solve this issue by this somehow.

Regards,
Christian.

Quoted 11 lines
Quoted 10 lines

Or maybe a flag to indicate that it can only free up 64bit BARs?

Regards,
Christian.

Quoted 3 lines

Thanks a lot for checking this out!

#25 lore raw

Re: [PATCH 8/9] drm/amdgpu: Remove driver side BAR release before resize

Quoted 79 lines · Christian König
Quoted 41 lines · Ilpo Järvinen
Quoted 39 lines · Christian König
Quoted 24 lines · Ilpo Järvinen
Quoted 12 lines · Christian König

Sorry for the late reply I'm really busy at the moment.

Quoted 4 lines · Ilpo Järvinen

PCI core handles releasing device's resources and their rollback in case of failure of a BAR resizing operation. Releasing resource prior to calling pci_resize_resource() prevents PCI core from restoring the BARs as they were.

I've intentionally didn't do it this way because at least on AMD HW we could only release the VRAM and doorbell BAR (both 64bit), but not the register BAR (32bit only).

Quoted 5 lines

This patch set looks like the right thing in general, but which BARs are now released by pci_resize_resource()?

If we avoid releasing the 32bit BAR as well then that should work, otherwise we will probably cause problems.

After these changes, pci_resize_resource() releases BARs that share the bridge window with the BAR to be resized. So the answer depends on the upstream bridge.

Quoted 6 lines

However, amdgpu_device_resize_fb_bar() also checks that root bus has a resource with a 64-bit address. That won't tell what the nearest bridge has though. Maybe that check should be converted to check the resources of the nearest bus instead? It would make it impossible to have the 32-bit resource share the bridge window with the 64-bit resources so the resize would be safe.

Mhm, I don't think that will work.

I've added the check for the root bus to avoid a couple of issues during resize, but checking the nearest bridge would block a whole bunch of use cases and isn't even 100% save.

Quoted 6 lines
Quoted 2 lines

See one use case of this is that all the BARs of the device start in the same 32bit bridge window (or a mixture of 64bit and 32bit window).

"32bit bridge window" is ambiguous. There are non-prefetchable and prefetchable bridge windows, out of which the latter can be 64-bit as well. Which one you're talking about?

The non-prefetchable 32bit window.

Quoted 5 lines

If a 64-bit prefetchable window exists, pbus_size_mem() nor __pci_assign_resource() would not have produced such a configuration where they're put into the same bridge window, even before the commit ae88d0b9c57f ("PCI: Use pbus_select_window_for_type() during mem window sizing") (I think). Now pbus_size_mem() certainly doesn't.

I need to double check, but if I'm not completely mistaken that is assigned by the BIOS.

Here is an example of a "good" configuration where both VRAM (BAR0) and doorbell (BAR2) is in the prefetchable window and MMIO in the non-prefetchable:

Device:

	Region 0: Memory at 80000000 (64-bit, prefetchable) [size=256M]
	Region 2: Memory at 90000000 (64-bit, prefetchable) [size=2M]
	Region 4: I/O ports at 3000 [size=256]
	Region 5: Memory at 9f300000 (32-bit, non-prefetchable) [size=1M]

Bridge:

	Memory behind bridge: 9f300000-9f4fffff [size=2M] [32-bit]
	Prefetchable memory behind bridge: 80000000-901fffff [size=258M] [32-bit]

And here is an example of another system where things are mixed up:

Device:

	Region 0: Memory at 2c00000000 (64-bit, prefetchable) [size=256M]
	Region 2: Memory at 94000000 (64-bit, prefetchable) [size=2M]
	Region 4: I/O ports at 1000 [size=256]
	Region 5: Memory at 94600000 (32-bit, non-prefetchable) [size=512K]
Quoted 6 lines

Bridge:

	Memory behind bridge: 94000000-946fffff [size=7M] [32-bit]
	Prefetchable memory behind bridge: 2c00000000-2c107fffff [size=264M] [32-bit]

In that example the doorbell ended up in the non-prefetchable window for some reason. And that config comes in all possible variations.

The really odd thing is that there seems to be even room in that prefetchable window for a 2MB BAR.

(Unless it's ppc which I heard is placing small BARs in a weird way.)

Quoted 4 lines

On AMD GPUs both BAR0 and BAR2 are resizeable.

So far we have only implemented resizing of BAR0, but essentially we want to have both for some use cases.

Okay. My plan is anyway to change the resource fitting logic so it will leave enough space to fit as large resources as possible at where the resizable BARs is at (once I get that far). Then BAR resize itself can be mostly done in-place without need to release the bridge windows at all.

Quoted 18 lines
Quoted 18 lines
Quoted 9 lines

What we have is that BAR 0 and 2 are 64bit BARs which can (after some preparation) move around freely. But IIRC BAR 4 are the legacy I/O ports and BAR 5 is the 32bit MMIO registers (don't nail me on that, could be just the other way around).

Especially that 32bit MMIO BAR *can't* move! Not only because it is 32bit, but also because the amdgpu driver as well as the HW itself through the VGA emulation, as well as the EFI/VESA/VBIOS code might reference its absolute address.

So if the 64-bit check is replaced with this:

(quoted diff) +4 -0
+       /* Check if the parent bridge has a 64-bit (pref) memory resource */
+       res = pci_resource_n(adev->pdev, 0)->parent;
+       /* Trying to resize is pointless without a window above 4GB */
+       if (!(res->flags & IORESOURCE_MEM_64))
		return 0;
Quoted 4 lines
Quoted 2 lines

...I don't think it's possible for 32-bit resource to share that window under _any_ circumstance.

Well see the example above.

For the record, BAR0 would pass that 64-bit check above and could be resized safely too.

But I hear your point that this kind of mixed config seems possible for some reason.

Quoted 8 lines

I have SSH access to a system where exactly that is the configuration.

Quoted 4 lines

If you say that ->parent somehow points to a non-IORESOURCE_MEM_64 window at this point, you're implying allocation for the 64-bit prefetchable window was tried and failed, and __pci_assign_resource() then used one of its fallbacks.

No, as I said that comes from the BIOS.

Normally we don't abide the BIOS allocations for normal BARs, only bridge windows are claimed using pci_claim_resource(). Only if preserve_config is set for the host bridge, also dev resources are claimed as they were discovered.

The normal BARs are normally added into the resource tree using pci_assign_resource() which will not end up using the resource address of the resource itself in determining where to place the resource (AFAICT from the code in __pci_assign_resource() -> pci_bus_alloc_resource() -> pci_bus_alloc_from_region() -> allocate_resource() -> __find_resource_space()).

Quoted 12 lines
Quoted 4 lines

Are you saying that "some preparation" includes making room for that 64-bit prefetchable window that failed to assign earlier as I cannot see how else it would ever get assigned so that the 64-bit BARs could be moved there?

No, at least from the amdgpu driver side we don't touch the resource allocation at all.

In this case preparation means disabling the VGA emulation, cause otherwise trying to resize the BAR can just cause a spontaneous system reboot for some reason.

Quoted 6 lines
Quoted 4 lines
Quoted 2 lines

Could we give pci_resize_resource() a mask of BARs which are save to release?

It is possible.

Then let us solve this issue by this somehow.

I've added exclude_bars parameters to pci_resize_resource() and made amdgpu to pass 1 << 5 to it, will send v2 in a day or two with that. If you've better idea than using a literal like that, please let me know.

Signature · 1 line
 i.
#26 lore raw

RE: [PATCH 8/9] drm/amdgpu: Remove driver side BAR release before resize

In reply to Christian König

[AMD Official Use Only - AMD Internal Distribution Only]

Hi @Shi, Lianjie

Do we think this patch series impact our use case ?

Monk Liu | Cloud GPU & Virtualization | AMD

-----Original Message-----
From: amd-gfx <[email protected]> On Behalf Of Christian König
Sent: Tuesday, November 11, 2025 5:09 PM
To: Ilpo Järvinen <[email protected]>; Alex Bennée <[email protected]>; Simon Richter <[email protected]>; Lucas De Marchi <[email protected]>; Deucher, Alexander <[email protected]>; [email protected]; Bjorn Helgaas <[email protected]>; David Airlie <[email protected]>; [email protected]; [email protected]; [email protected]; Jani Nikula <[email protected]>; Joonas Lahtinen <[email protected]>; [email protected]; Rodrigo Vivi <[email protected]>; Simona Vetter <[email protected]>; Tvrtko Ursulin <[email protected]>; Thomas Hellström <[email protected]>; Michał Winiarski <[email protected]>; [email protected]
Subject: Re: [PATCH 8/9] drm/amdgpu: Remove driver side BAR release before resize

Sorry for the late reply I'm really busy at the moment.

Quoted 4 lines · Ilpo Järvinen

PCI core handles releasing device's resources and their rollback in case of failure of a BAR resizing operation. Releasing resource prior to calling pci_resize_resource() prevents PCI core from restoring the BARs as they were.

I've intentionally didn't do it this way because at least on AMD HW we could only release the VRAM and doorbell BAR (both 64bit), but not the register BAR (32bit only).

This patch set looks like the right thing in general, but which BARs are now released by pci_resize_resource()?

If we avoid releasing the 32bit BAR as well then that should work, otherwise we will probably cause problems.

Regards,
Christian.

Quoted 41 lines

Remove driver-side release of BARs from the amdgpu driver.

Also remove the driver initiated assignment as pci_resize_resource() should try to assign as much as possible. If the driver side call manages to get more required resources assigned in some scenario, such a problem should be fixed inside pci_resize_resource() instead.

---
 drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 8 +-------
 1 file changed, 1 insertion(+), 7 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +1 -7
index 7a899fb4de29..65474d365229 100644
@@ -1729,12 +1729,8 @@ int amdgpu_device_resize_fb_bar(struct amdgpu_device *adev)
      pci_write_config_word(adev->pdev, PCI_COMMAND,
                            cmd & ~PCI_COMMAND_MEMORY);
 
-     /* Free the VRAM and doorbell BAR, we most likely need to move both. */
+     /* Tear down doorbell as resizing will release BARs */
      amdgpu_doorbell_fini(adev);
-     if (adev->asic_type >= CHIP_BONAIRE)
-             pci_release_resource(adev->pdev, 2);
-
-     pci_release_resource(adev->pdev, 0);
 
      r = pci_resize_resource(adev->pdev, 0, rbar_size);
      if (r == -ENOSPC)
@@ -1743,8 +1739,6 @@ int amdgpu_device_resize_fb_bar(struct amdgpu_device *adev)
      else if (r && r != -ENOTSUPP)
              dev_err(adev->dev, "Problem resizing BAR0 (%d).", r);
 
-     pci_assign_unassigned_bus_resources(adev->pdev->bus);
-
      /* When the doorbell or fb BAR isn't available we have no chance of
       * using the device.
       */
#27 lore raw

PATCH 9/9 PCI: Prevent restoring assigned resources

In reply to Ilpo Järvinen

restore_dev_resource() copies saved addresses and flags from the struct pci_dev_resource back to the struct resource, typically, during rollback from a failure or in preparation for a retry attempt.

If the resource is within resource tree, the resource must not be modified as the resource tree could be corrupted. Thus, it's a bug to call restore_dev_resource() for assigned resources (which did happen due to logic flaws in the BAR resize rollback).

Add WARN_ON_ONCE() into restore_dev_resource() to detect such bugs easily and return without altering the resource to prevent corruption.

---
 drivers/pci/setup-bus.c | 4 ++++
 1 file changed, 4 insertions(+)
drivers/pci/setup-bus.c +4 -0
index 8da83b612c59..28d6ae822c0b 100644
@@ -15,6 +15,7 @@
  */
 
 #include <linux/bitops.h>
+#include <linux/bug.h>
 #include <linux/init.h>
 #include <linux/kernel.h>
 #include <linux/module.h>
@@ -135,6 +136,9 @@ static void restore_dev_resource(struct pci_dev_resource *dev_res)
 {
 	struct resource *res = dev_res->res;
 
+	if (WARN_ON_ONCE(res->parent))
+		return;
+
 	res->start = dev_res->start;
 	res->end = dev_res->end;
 	res->flags = dev_res->flags;
Signature · 1 line
2.39.5
#28 lore raw

✗ Fi.CI.BUILD: failure for PCI: BAR resizing fix/rework

In reply to Ilpo Järvinen
== Series Details ==

== Series Details ==

Series: PCI: BAR resizing fix/rework
URL : https://patchwork.freedesktop.org/series/156675/
State : failure

== Summary ==

Error: patch https://patchwork.freedesktop.org/api/1.0/series/156675/revisions/1/mbox/ not applied
Applying: PCI: Prevent resource tree corruption when BAR resize fails
Applying: PCI/IOV: Adjust ->barsz[] when changing BAR size
Applying: PCI: Change pci_dev variable from 'bridge' to 'dev'
Applying: PCI: Try BAR resize even when no window was released
Applying: PCI: Fix restoring BARs on BAR resize rollback path
Applying: drm/xe: Remove driver side BAR release before resize
Using index info to reconstruct a base tree...
M drivers/gpu/drm/xe/xe_vram.c
Falling back to patching base and 3-way merge...
Auto-merging drivers/gpu/drm/xe/xe_vram.c
CONFLICT (content): Merge conflict in drivers/gpu/drm/xe/xe_vram.c
error: Failed to merge in the changes.
hint: Use 'git am --show-current-patch=diff' to see the failed patch
Patch failed at 0006 drm/xe: Remove driver side BAR release before resize
When you have resolved this problem, run "git am --continue".
If you prefer to skip this patch, run "git am --skip" instead.
To restore the original branch and stop patching, run "git am --abort".
Build failed, no error log produced