Skip to contentSkip to conversation

PCI: Sort resources by size as secondary key

RFC PATCH In review Ding Hui 3 participants 5 messages Jun 18 – Jun 22 PCI

lore
Share & tools
lore

Ding Hui

author
#0 lore raw

RFC PATCH 1/1 PCI: Sort resources by size as secondary key

We encountered an issue on BCM57414 NIC where function 1 failed to enable SR-IOV after remove & rescan. Investigation revealed this is caused by BAR allocation failure during rescan.

Simplified topology:

 +-[0000:30]-+- ...
 |           +-02.0-[31]--+-00.0  Broadcom Inc. and subsidiaries BCM57414 NetXtreme-E 10Gb/25Gb RDMA Ethernet Controller [14e4:16d7]
 |           |            \-00.1  Broadcom Inc. and subsidiaries BCM57414 NetXtreme-E 10Gb/25Gb RDMA Ethernet Controller [14e4:16d7]

iomem layout after init bootup:

22fffec00000-22ffffefffff : PCI Bus 0000:31 [Window size=19M]

    22fffec00000-22ffff3fffff : 0000:31:00.1  [align=1M  size=8M   BAR 9 (VF BAR 2)]
    22ffff400000-22ffffbfffff : 0000:31:00.0  [align=1M  size=8M   BAR 9 (VF BAR 2)]
    22ffffc00000-22ffffcfffff : 0000:31:00.1  [align=1M  size=1M   BAR 2]
    22ffffd00000-22ffffdfffff : 0000:31:00.0  [align=1M  size=1M   BAR 2]
    22ffffe00000-22ffffe0ffff : 0000:31:00.1  [align=64K size=64K  BAR 0]
    22ffffe10000-22ffffe1ffff : 0000:31:00.0  [align=64K size=64K  BAR 0]
    22ffffe20000-22ffffe3ffff : 0000:31:00.1  [align=16K size=128K BAR 11(VF BAR 4)]
    22ffffe40000-22ffffe5ffff : 0000:31:00.1  [align=16K size=128K BAR 7 (VF BAR 0)]
    22ffffe60000-22ffffe7ffff : 0000:31:00.0  [align=16K size=128K BAR 11(VF BAR 4)]
    22ffffe80000-22ffffe9ffff : 0000:31:00.0  [align=16K size=128K BAR 7 (VF BAR 0)]
    22ffffea0000-22ffffea1fff : 0000:31:00.1  [align=8K  size=8K   BAR 4]
    22ffffea2000-22ffffea3fff : 0000:31:00.0  [align=8K  size=8K   BAR 4]

iomem layout after remove function 1 by
echo "1" > /sys/bus/pci/devices/0000:31:00.1/remove

22fffec00000-22ffffefffff : PCI Bus 0000:31 [Window size=19M]

    22ffff400000-22ffffbfffff : 0000:31:00.0  [align=1M  size=8M   BAR 9 (VF BAR 2)]
    22ffffd00000-22ffffdfffff : 0000:31:00.0  [align=1M  size=1M   BAR 2]
    22ffffe10000-22ffffe1ffff : 0000:31:00.0  [align=64K size=64K  BAR 0]
    22ffffe60000-22ffffe7ffff : 0000:31:00.0  [align=16K size=128K BAR 11(VF BAR 4)]
    22ffffe80000-22ffffe9ffff : 0000:31:00.0  [align=16K size=128K BAR 7 (VF BAR 0)]
    22ffffea2000-22ffffea3fff : 0000:31:00.0  [align=8K  size=8K   BAR 4]

Rescan logs triggered by
echo "1" > /sys/bus/pci/devices/0000:30:02.0/rescan

[   90.585067] pci 0000:31:00.1: [14e4:16d7] type 00 class 0x020000 PCIe Endpoint
[   90.585107] pci 0000:31:00.1: BAR 0 [mem 0x22ffffe00000-0x22ffffe0ffff 64bit pref]
[   90.585113] pci 0000:31:00.1: BAR 2 [mem 0x22ffffc00000-0x22ffffcfffff 64bit pref]
[   90.585116] pci 0000:31:00.1: BAR 4 [mem 0x22ffffea0000-0x22ffffea1fff 64bit pref]
[   90.585119] pci 0000:31:00.1: ROM [mem 0xb0e00000-0xb0e7ffff pref]
[   90.585216] pci 0000:31:00.1: PME# supported from D0 D3hot D3cold
[   90.585253] pci 0000:31:00.1: VF BAR 0 [mem 0x22ffffe40000-0x22ffffe43fff 64bit pref]
[   90.585255] pci 0000:31:00.1: VF BAR 0 [mem 0x22ffffe40000-0x22ffffe5ffff 64bit pref]: contains BAR 0 for 8 VFs
[   90.585258] pci 0000:31:00.1: VF BAR 2 [mem 0x22fffec00000-0x22fffecfffff 64bit pref]
[   90.585260] pci 0000:31:00.1: VF BAR 2 [mem 0x22fffec00000-0x22ffff3fffff 64bit pref]: contains BAR 2 for 8 VFs
[   90.585263] pci 0000:31:00.1: VF BAR 4 [mem 0x22ffffe20000-0x22ffffe23fff 64bit pref]
[   90.585265] pci 0000:31:00.1: VF BAR 4 [mem 0x22ffffe20000-0x22ffffe3ffff 64bit pref]: contains BAR 4 for 8 VFs
[   90.585534] pci 0000:31:00.1: Adding to iommu group 11
[   90.585575] pci 0000:31:00.1: BAR 2 [mem 0x22fffec00000-0x22fffecfffff 64bit pref]: assigned
[   90.585585] pci 0000:31:00.1: VF BAR 2 [mem size 0x00800000 64bit pref]: can't assign; no space
[   90.585587] pci 0000:31:00.1: VF BAR 2 [mem size 0x00800000 64bit pref]: failed to assign
[   90.585589] pci 0000:31:00.1: ROM [mem 0xb0e00000-0xb0e7ffff pref]: assigned
[   90.585591] pci 0000:31:00.1: BAR 0 [mem 0x22fffed00000-0x22fffed0ffff 64bit pref]: assigned
[   90.585599] pci 0000:31:00.1: VF BAR 0 [mem 0x22fffed10000-0x22fffed2ffff 64bit pref]: assigned
[   90.585603] pci 0000:31:00.1: VF BAR 4 [mem 0x22fffed30000-0x22fffed4ffff 64bit pref]: assigned
[   90.585606] pci 0000:31:00.1: BAR 4 [mem 0x22fffed50000-0x22fffed51fff 64bit pref]: assigned

Enable sriov failed logs triggered by
echo 2 > /sys/bus/pci/devices/0000:31:00.1/sriov_numvfs

[ 1666.918432] bnxt_en 0000:31:00.1: not enough MMIO resources for SR-IOV
[ 1666.918442] bnxt_en 0000:31:00.1 eth5: pci_enable_sriov failed : -12

The resource allocation process during rescan is as follows:

dev_rescan_store
pci_rescan_bus
pci_assign_unassigned_bus_resources
__pci_bus_assign_resources
pbus_assign_resources_sorted
pdev_sort_resources
__assign_resources_sorted
assign_requested_resources_sorted
pci_assign_resource

We noticed that current sort algorithm is only by alignment. The BAR 2 (align=1M size=1M) is located before BAR 9 (VF BAR 2 align=1M size=8M), so the 8M cannot be satisfied.

If we keep alignment as primary sorting key, but use size as secondary key, all resource can be satisfied when remove & rescan.

Does this approach only solve current specific case as a workaround, or does it also benefit general PCI resource allocation?

I think it may help reduce allocation failures due to fragmentation theoretically, but I'm not sure.

Appreciate any comment and suggestion, thanks.

---
 drivers/pci/setup-bus.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
drivers/pci/setup-bus.c +2 -1
index 4cf120ebe5ad..63f224f0c6be 100644
@@ -367,7 +367,8 @@ static void pdev_sort_resources(struct pci_dev *dev, struct list_head *head)
 			align = pci_resource_alignment(dev_res->dev,
 							 dev_res->res);
 
-			if (r_align > align) {
+			if (r_align > align ||
+			    (r_align == align && resource_size(r) > resource_size(dev_res->res))) {
 				n = &dev_res->list;
 				break;
 			}
Signature · 1 line
2.17.1
#1 lore raw
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [High] Incomplete fix: `__assign_resources_sorted()` fails to maintain the new size-based secondary sort…

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] Incomplete fix: __assign_resources_sorted() fails to maintain the new size-based secondary sort key when bridge windows are dynamically resized, breaking the sorting invariant. --

commit b576a58abc2576977d567e69cb6a8b9e9d867d9b
Author: Ding Hui <[email protected]>
Subject: PCI: Sort resources by size as secondary key

The commit updates pdev_sort_resources() to use resource size as a secondary sorting key to reduce fragmentation. This resolves an issue on the BCM57414 NIC where function 1 failed to enable SR-IOV after remove and rescan due to BAR allocation failures.

drivers/pci/setup-bus.c static void pdev_sort_resources(struct pci_dev *dev, struct list_head *head) · line 371

Quoted 11 lines
drivers/pci/setup-bus.c +2 -1
index 4cf120ebe5adf..63f224f0c6bee 100644
@@ -367,7 +367,8 @@ static void pdev_sort_resources(struct pci_dev *dev, struct list_head *head)
 			align = pci_resource_alignment(dev_res->dev,
 							 dev_res->res);
 
-			if (r_align > align) {
+			if (r_align > align ||
+			    (r_align == align && resource_size(r) > resource_size(dev_res->res))) {

[Severity: High]
Does this patch also need to update the re-sorting logic in __assign_resources_sorted() to maintain the size-based secondary sort key?

When a bridge window requests additional size via realloc_head in __assign_resources_sorted(), its size is increased:

drivers/pci/setup-bus.c:__assign_resources_sorted() {
    ...
    res->end += addsize_res->add_size;
    ...
    if (addsize_res->min_align <= res->start)
        continue;

If the alignment does not increase, it skips re-sorting entirely, leaving the enlarged resource in its original position based on its older, smaller size.

If the alignment does increase, it is re-inserted into the sorted list, but the loop only compares alignments:

    list_for_each_entry(dev_res2, head, list) {
        align = pci_resource_alignment(dev_res2->dev, dev_res2->res);
        if (addsize_res->min_align > align) {
            list_move_tail(&dev_res->list, &dev_res2->list);
            break;
        }
    }

This skips over elements with equal alignment, placing the newly enlarged resource at the end of its alignment group and disregarding its size. Because bridge windows are frequently dynamically resized, failing to sort them by size when enlarged might place them behind smaller resources of the same alignment, potentially re-introducing the fragmentation and allocation failures this patch intends to solve.

Quoted 3 lines
drivers/pci/setup-bus.c +0 -0
 				n = &dev_res->list;
 				break;
 			}
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
#2 lore raw
In reply to Ding Hui
Quoted 1 line · Ding Hui

We encountered an issue on BCM57414 NIC where function 1 failed to

Don't or "We" (or "I") in changelog sentences. Use imperative tone. Here you can start just with:

BCM57414 NIC function 1 fails to ...

Quoted 52 lines

enable SR-IOV after remove & rescan. Investigation revealed this is caused by BAR allocation failure during rescan.

Simplified topology:

 +-[0000:30]-+- ...
 |           +-02.0-[31]--+-00.0  Broadcom Inc. and subsidiaries BCM57414 NetXtreme-E 10Gb/25Gb RDMA Ethernet Controller [14e4:16d7]
 |           |            \-00.1  Broadcom Inc. and subsidiaries BCM57414 NetXtreme-E 10Gb/25Gb RDMA Ethernet Controller [14e4:16d7]

iomem layout after init bootup:

22fffec00000-22ffffefffff : PCI Bus 0000:31 [Window size=19M]

    22fffec00000-22ffff3fffff : 0000:31:00.1  [align=1M  size=8M   BAR 9 (VF BAR 2)]
    22ffff400000-22ffffbfffff : 0000:31:00.0  [align=1M  size=8M   BAR 9 (VF BAR 2)]
    22ffffc00000-22ffffcfffff : 0000:31:00.1  [align=1M  size=1M   BAR 2]
    22ffffd00000-22ffffdfffff : 0000:31:00.0  [align=1M  size=1M   BAR 2]
    22ffffe00000-22ffffe0ffff : 0000:31:00.1  [align=64K size=64K  BAR 0]
    22ffffe10000-22ffffe1ffff : 0000:31:00.0  [align=64K size=64K  BAR 0]
    22ffffe20000-22ffffe3ffff : 0000:31:00.1  [align=16K size=128K BAR 11(VF BAR 4)]
    22ffffe40000-22ffffe5ffff : 0000:31:00.1  [align=16K size=128K BAR 7 (VF BAR 0)]
    22ffffe60000-22ffffe7ffff : 0000:31:00.0  [align=16K size=128K BAR 11(VF BAR 4)]
    22ffffe80000-22ffffe9ffff : 0000:31:00.0  [align=16K size=128K BAR 7 (VF BAR 0)]
    22ffffea0000-22ffffea1fff : 0000:31:00.1  [align=8K  size=8K   BAR 4]
    22ffffea2000-22ffffea3fff : 0000:31:00.0  [align=8K  size=8K   BAR 4]

iomem layout after remove function 1 by
echo "1" > /sys/bus/pci/devices/0000:31:00.1/remove

22fffec00000-22ffffefffff : PCI Bus 0000:31 [Window size=19M]

    22ffff400000-22ffffbfffff : 0000:31:00.0  [align=1M  size=8M   BAR 9 (VF BAR 2)]
    22ffffd00000-22ffffdfffff : 0000:31:00.0  [align=1M  size=1M   BAR 2]
    22ffffe10000-22ffffe1ffff : 0000:31:00.0  [align=64K size=64K  BAR 0]
    22ffffe60000-22ffffe7ffff : 0000:31:00.0  [align=16K size=128K BAR 11(VF BAR 4)]
    22ffffe80000-22ffffe9ffff : 0000:31:00.0  [align=16K size=128K BAR 7 (VF BAR 0)]
    22ffffea2000-22ffffea3fff : 0000:31:00.0  [align=8K  size=8K   BAR 4]

Rescan logs triggered by
echo "1" > /sys/bus/pci/devices/0000:30:02.0/rescan

[   90.585067] pci 0000:31:00.1: [14e4:16d7] type 00 class 0x020000 PCIe Endpoint
[   90.585107] pci 0000:31:00.1: BAR 0 [mem 0x22ffffe00000-0x22ffffe0ffff 64bit pref]
[   90.585113] pci 0000:31:00.1: BAR 2 [mem 0x22ffffc00000-0x22ffffcfffff 64bit pref]
[   90.585116] pci 0000:31:00.1: BAR 4 [mem 0x22ffffea0000-0x22ffffea1fff 64bit pref]
[   90.585119] pci 0000:31:00.1: ROM [mem 0xb0e00000-0xb0e7ffff pref]
[   90.585216] pci 0000:31:00.1: PME# supported from D0 D3hot D3cold
[   90.585253] pci 0000:31:00.1: VF BAR 0 [mem 0x22ffffe40000-0x22ffffe43fff 64bit pref]
[   90.585255] pci 0000:31:00.1: VF BAR 0 [mem 0x22ffffe40000-0x22ffffe5ffff 64bit pref]: contains BAR 0 for 8 VFs
[   90.585258] pci 0000:31:00.1: VF BAR 2 [mem 0x22fffec00000-0x22fffecfffff 64bit pref]
[   90.585260] pci 0000:31:00.1: VF BAR 2 [mem 0x22fffec00000-0x22ffff3fffff 64bit pref]: contains BAR 2 for 8 VFs
[   90.585263] pci 0000:31:00.1: VF BAR 4 [mem 0x22ffffe20000-0x22ffffe23fff 64bit pref]
[   90.585265] pci 0000:31:00.1: VF BAR 4 [mem 0x22ffffe20000-0x22ffffe3ffff 64bit pref]: contains BAR 4 for 8 VFs
[   90.585534] pci 0000:31:00.1: Adding to iommu group 11
Quoted 8 lines
[   90.585575] pci 0000:31:00.1: BAR 2 [mem 0x22fffec00000-0x22fffecfffff 64bit pref]: assigned
[   90.585585] pci 0000:31:00.1: VF BAR 2 [mem size 0x00800000 64bit pref]: can't assign; no space
[   90.585587] pci 0000:31:00.1: VF BAR 2 [mem size 0x00800000 64bit pref]: failed to assign
[   90.585589] pci 0000:31:00.1: ROM [mem 0xb0e00000-0xb0e7ffff pref]: assigned
[   90.585591] pci 0000:31:00.1: BAR 0 [mem 0x22fffed00000-0x22fffed0ffff 64bit pref]: assigned
[   90.585599] pci 0000:31:00.1: VF BAR 0 [mem 0x22fffed10000-0x22fffed2ffff 64bit pref]: assigned
[   90.585603] pci 0000:31:00.1: VF BAR 4 [mem 0x22fffed30000-0x22fffed4ffff 64bit pref]: assigned
[   90.585606] pci 0000:31:00.1: BAR 4 [mem 0x22fffed50000-0x22fffed51fff 64bit pref]: assigned

Timestamps are irrelevant noise to the problem and should be removed.

Quoted 19 lines

Enable sriov failed logs triggered by
echo 2 > /sys/bus/pci/devices/0000:31:00.1/sriov_numvfs

[ 1666.918432] bnxt_en 0000:31:00.1: not enough MMIO resources for SR-IOV
[ 1666.918442] bnxt_en 0000:31:00.1 eth5: pci_enable_sriov failed : -12

The resource allocation process during rescan is as follows:

dev_rescan_store
pci_rescan_bus
pci_assign_unassigned_bus_resources
__pci_bus_assign_resources
pbus_assign_resources_sorted
pdev_sort_resources
__assign_resources_sorted
assign_requested_resources_sorted
pci_assign_resource

Quoted 3 lines

We noticed that current sort algorithm is only by alignment. The BAR 2 (align=1M size=1M) is located before BAR 9 (VF BAR 2 align=1M size=8M), so the 8M cannot be satisfied.

I think you have a typo (located vs allocated)? The difference changes meaning significantly. (located implies before in address, allocated implies before in the order of made allocations).

Quoted 8 lines

If we keep alignment as primary sorting key, but use size as secondary key, all resource can be satisfied when remove & rescan.

Does this approach only solve current specific case as a workaround, or does it also benefit general PCI resource allocation?

I think it may help reduce allocation failures due to fragmentation theoretically, but I'm not sure.

I suppose trying the largest first does generally increase the chances of success in cases where the resource sizes are very heterogeneous like in your case, not just in this case.

You should really rewrite the changelog text though. Try more to focus on how the other allocations from the sibling make it possible to fit some of the resource(s) only into a single place within the window. And therefore largest one should be requested first. I had to figure that bit myself as you didn't clearly state why it fails but only talked vaguely about the order of (al)location(s).

There will still be some cases this greedy approach will not get right, such as align=2M,size=2M & align=1M,size=8M. This algorithm is not really designed for filling gaps in a window but the entire window from scratch, which is why it cannot handle all cases.

Quoted 12 lines

Appreciate any comment and suggestion, thanks.

---
 drivers/pci/setup-bus.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
drivers/pci/setup-bus.c +0 -0
index 4cf120ebe5ad..63f224f0c6be 100644

drivers/pci/setup-bus.c static void pdev_sort_resources(struct pci_dev *dev, struct list_head *head) · line 371

Quoted 6 lines
drivers/pci/setup-bus.c +2 -1
@@ -367,7 +367,8 @@ static void pdev_sort_resources(struct pci_dev *dev, struct list_head *head)
 			align = pci_resource_alignment(dev_res->dev,
 							 dev_res->res);
 
-			if (r_align > align) {
+			if (r_align > align ||
+			    (r_align == align && resource_size(r) > resource_size(dev_res->res))) {

This is not the only place where the algorithm does sorting.

(And I also know one restore place is lacking restoring ordering.)

I was planning to move to rbtree for storing the resources that need to be in a certain order as doing it everywhere results in small variations which is error prone.

So maybe it would be time to consider moving to that so we could do the sort order in one place.

Signature · 1 line
 i.
author
#3 lore raw
Quoted 91 lines · Ilpo Järvinen
Quoted 1 line · Ding Hui

We encountered an issue on BCM57414 NIC where function 1 failed to

Don't or "We" (or "I") in changelog sentences. Use imperative tone. Here you can start just with:

BCM57414 NIC function 1 fails to ...

Quoted 52 lines

enable SR-IOV after remove & rescan. Investigation revealed this is caused by BAR allocation failure during rescan.

Simplified topology:

  +-[0000:30]-+- ...
  |           +-02.0-[31]--+-00.0  Broadcom Inc. and subsidiaries BCM57414 NetXtreme-E 10Gb/25Gb RDMA Ethernet Controller [14e4:16d7]
  |           |            \-00.1  Broadcom Inc. and subsidiaries BCM57414 NetXtreme-E 10Gb/25Gb RDMA Ethernet Controller [14e4:16d7]

iomem layout after init bootup:

   22fffec00000-22ffffefffff : PCI Bus 0000:31 [Window size=19M]
     22fffec00000-22ffff3fffff : 0000:31:00.1  [align=1M  size=8M   BAR 9 (VF BAR 2)]
     22ffff400000-22ffffbfffff : 0000:31:00.0  [align=1M  size=8M   BAR 9 (VF BAR 2)]
     22ffffc00000-22ffffcfffff : 0000:31:00.1  [align=1M  size=1M   BAR 2]
     22ffffd00000-22ffffdfffff : 0000:31:00.0  [align=1M  size=1M   BAR 2]
     22ffffe00000-22ffffe0ffff : 0000:31:00.1  [align=64K size=64K  BAR 0]
     22ffffe10000-22ffffe1ffff : 0000:31:00.0  [align=64K size=64K  BAR 0]
     22ffffe20000-22ffffe3ffff : 0000:31:00.1  [align=16K size=128K BAR 11(VF BAR 4)]
     22ffffe40000-22ffffe5ffff : 0000:31:00.1  [align=16K size=128K BAR 7 (VF BAR 0)]
     22ffffe60000-22ffffe7ffff : 0000:31:00.0  [align=16K size=128K BAR 11(VF BAR 4)]
     22ffffe80000-22ffffe9ffff : 0000:31:00.0  [align=16K size=128K BAR 7 (VF BAR 0)]
     22ffffea0000-22ffffea1fff : 0000:31:00.1  [align=8K  size=8K   BAR 4]
     22ffffea2000-22ffffea3fff : 0000:31:00.0  [align=8K  size=8K   BAR 4]

iomem layout after remove function 1 by
echo "1" > /sys/bus/pci/devices/0000:31:00.1/remove

   22fffec00000-22ffffefffff : PCI Bus 0000:31 [Window size=19M]
     22ffff400000-22ffffbfffff : 0000:31:00.0  [align=1M  size=8M   BAR 9 (VF BAR 2)]
     22ffffd00000-22ffffdfffff : 0000:31:00.0  [align=1M  size=1M   BAR 2]
     22ffffe10000-22ffffe1ffff : 0000:31:00.0  [align=64K size=64K  BAR 0]
     22ffffe60000-22ffffe7ffff : 0000:31:00.0  [align=16K size=128K BAR 11(VF BAR 4)]
     22ffffe80000-22ffffe9ffff : 0000:31:00.0  [align=16K size=128K BAR 7 (VF BAR 0)]
     22ffffea2000-22ffffea3fff : 0000:31:00.0  [align=8K  size=8K   BAR 4]

Rescan logs triggered by
echo "1" > /sys/bus/pci/devices/0000:30:02.0/rescan

[   90.585067] pci 0000:31:00.1: [14e4:16d7] type 00 class 0x020000 PCIe Endpoint
[   90.585107] pci 0000:31:00.1: BAR 0 [mem 0x22ffffe00000-0x22ffffe0ffff 64bit pref]
[   90.585113] pci 0000:31:00.1: BAR 2 [mem 0x22ffffc00000-0x22ffffcfffff 64bit pref]
[   90.585116] pci 0000:31:00.1: BAR 4 [mem 0x22ffffea0000-0x22ffffea1fff 64bit pref]
[   90.585119] pci 0000:31:00.1: ROM [mem 0xb0e00000-0xb0e7ffff pref]
[   90.585216] pci 0000:31:00.1: PME# supported from D0 D3hot D3cold
[   90.585253] pci 0000:31:00.1: VF BAR 0 [mem 0x22ffffe40000-0x22ffffe43fff 64bit pref]
[   90.585255] pci 0000:31:00.1: VF BAR 0 [mem 0x22ffffe40000-0x22ffffe5ffff 64bit pref]: contains BAR 0 for 8 VFs
[   90.585258] pci 0000:31:00.1: VF BAR 2 [mem 0x22fffec00000-0x22fffecfffff 64bit pref]
[   90.585260] pci 0000:31:00.1: VF BAR 2 [mem 0x22fffec00000-0x22ffff3fffff 64bit pref]: contains BAR 2 for 8 VFs
[   90.585263] pci 0000:31:00.1: VF BAR 4 [mem 0x22ffffe20000-0x22ffffe23fff 64bit pref]
[   90.585265] pci 0000:31:00.1: VF BAR 4 [mem 0x22ffffe20000-0x22ffffe3ffff 64bit pref]: contains BAR 4 for 8 VFs
[   90.585534] pci 0000:31:00.1: Adding to iommu group 11
Quoted 8 lines
[   90.585575] pci 0000:31:00.1: BAR 2 [mem 0x22fffec00000-0x22fffecfffff 64bit pref]: assigned
[   90.585585] pci 0000:31:00.1: VF BAR 2 [mem size 0x00800000 64bit pref]: can't assign; no space
[   90.585587] pci 0000:31:00.1: VF BAR 2 [mem size 0x00800000 64bit pref]: failed to assign
[   90.585589] pci 0000:31:00.1: ROM [mem 0xb0e00000-0xb0e7ffff pref]: assigned
[   90.585591] pci 0000:31:00.1: BAR 0 [mem 0x22fffed00000-0x22fffed0ffff 64bit pref]: assigned
[   90.585599] pci 0000:31:00.1: VF BAR 0 [mem 0x22fffed10000-0x22fffed2ffff 64bit pref]: assigned
[   90.585603] pci 0000:31:00.1: VF BAR 4 [mem 0x22fffed30000-0x22fffed4ffff 64bit pref]: assigned
[   90.585606] pci 0000:31:00.1: BAR 4 [mem 0x22fffed50000-0x22fffed51fff 64bit pref]: assigned

Timestamps are irrelevant noise to the problem and should be removed.

Quoted 19 lines

Enable sriov failed logs triggered by
echo 2 > /sys/bus/pci/devices/0000:31:00.1/sriov_numvfs

[ 1666.918432] bnxt_en 0000:31:00.1: not enough MMIO resources for SR-IOV
[ 1666.918442] bnxt_en 0000:31:00.1 eth5: pci_enable_sriov failed : -12

The resource allocation process during rescan is as follows:

dev_rescan_store
pci_rescan_bus
pci_assign_unassigned_bus_resources
__pci_bus_assign_resources
pbus_assign_resources_sorted
pdev_sort_resources
__assign_resources_sorted
assign_requested_resources_sorted
pci_assign_resource

Quoted 8 lines
Quoted 3 lines

We noticed that current sort algorithm is only by alignment. The BAR 2 (align=1M size=1M) is located before BAR 9 (VF BAR 2 align=1M size=8M), so the 8M cannot be satisfied.

I think you have a typo (located vs allocated)? The difference changes meaning significantly. (located implies before in address, allocated implies before in the order of made allocations).

My description could indeed be misleading. My intention was that BAR 2 is before BAR 9 in the sorted list, and therefore it is allocated before BAR 9.

Quoted 6 lines
Quoted 6 lines

If we keep alignment as primary sorting key, but use size as secondary key, all resource can be satisfied when remove & rescan.

Does this approach only solve current specific case as a workaround, or does it also benefit general PCI resource allocation?

Quoted 7 lines
Quoted 2 lines

I think it may help reduce allocation failures due to fragmentation theoretically, but I'm not sure.

I suppose trying the largest first does generally increase the chances of success in cases where the resource sizes are very heterogeneous like in your case, not just in this case.

Thanks for agreeing with this point.

Quoted 7 lines

You should really rewrite the changelog text though. Try more to focus on how the other allocations from the sibling make it possible to fit some of the resource(s) only into a single place within the window. And therefore largest one should be requested first. I had to figure that bit myself as you didn't clearly state why it fails but only talked vaguely about the order of (al)location(s).

After .1 BAR 2 assigned, the iomem layout (deduce should be):

22fffec00000-22ffffefffff : PCI Bus 0000:31 [Window size=19M]

    22fffec00000-22fffecfffff : 0000:31:00.1  [align=1M  size=1M   BAR 2]
                                              [gap=7M]
    22ffff400000-22ffffbfffff : 0000:31:00.0  [align=1M  size=8M   BAR 9 (VF BAR 2)]
                                              [gap=1M]
    22ffffd00000-22ffffdfffff : 0000:31:00.0  [align=1M  size=1M   BAR 2]
                                              [gap=64K]
    22ffffe10000-22ffffe1ffff : 0000:31:00.0  [align=64K size=64K  BAR 0]
                                              [gap=256K]
    22ffffe60000-22ffffe7ffff : 0000:31:00.0  [align=16K size=128K BAR 11(VF BAR 4)]
    22ffffe80000-22ffffe9ffff : 0000:31:00.0  [align=16K size=128K BAR 7 (VF BAR 0)]
                                              [gap=8K]
    22ffffea2000-22ffffea3fff : 0000:31:00.0  [align=8K  size=8K   BAR 4]
                                              [gap=368K]

And then there is no suitable space available to satisfy both align=1M and size=8M (BAR 9), that lead to "0000:31:00.1: VF BAR 2 [mem size 0x00800000 64bit pref]: can't assign; no space".

Quoted 5 lines

There will still be some cases this greedy approach will not get right, such as align=2M,size=2M & align=1M,size=8M. This algorithm is not really designed for filling gaps in a window but the entire window from scratch, which is why it cannot handle all cases.

Does the "remove & rescan" is still a corner case for kernel, especially that only removing a single function rather than the entire device for a multi-function device, even after you've fixed multiple issues in this scenario?

Does the community resent the issues caused by this scenario?

Quoted 15 lines
Quoted 15 lines

Appreciate any comment and suggestion, thanks.

---
  drivers/pci/setup-bus.c | 3 ++-
  1 file changed, 2 insertions(+), 1 deletion(-)
drivers/pci/setup-bus.c +0 -0
index 4cf120ebe5ad..63f224f0c6be 100644
@@ -367,7 +367,8 @@ static void pdev_sort_resources(struct pci_dev *dev, struct list_head *head)
  			align = pci_resource_alignment(dev_res->dev,
  							 dev_res->res);
 
Quoted 8 lines

drivers/pci/setup-bus.c static void pdev_sort_resources(struct pci_dev *dev, struct list_head *head) · line 371

Quoted 3 lines
drivers/pci/setup-bus.c +2 -1
-			if (r_align > align) {
+			if (r_align > align ||
+			    (r_align == align && resource_size(r) > resource_size(dev_res->res))) {

This is not the only place where the algorithm does sorting.

(And I also know one restore place is lacking restoring ordering.)

Do you mean in __assign_resources_sorted(), retry normal assign after add_size assign failed?

I noticed this function after being replied by Sashiko AI review.

Quoted 7 lines

I was planning to move to rbtree for storing the resources that need to be in a certain order as doing it everywhere results in small variations which is error prone.

So maybe it would be time to consider moving to that so we could do the sort order in one place.

Thank you for your guidance.

Signature · 2 lines
Thanks,
- Ding Hui
#4 lore raw
Quoted 125 lines · Ding Hui
Quoted 124 lines · Ilpo Järvinen
Quoted 1 line · Ding Hui

We encountered an issue on BCM57414 NIC where function 1 failed to

Don't or "We" (or "I") in changelog sentences. Use imperative tone. Here you can start just with:

BCM57414 NIC function 1 fails to ...

Quoted 88 lines

enable SR-IOV after remove & rescan. Investigation revealed this is caused by BAR allocation failure during rescan.

Simplified topology:

  +-[0000:30]-+- ...
  |           +-02.0-[31]--+-00.0  Broadcom Inc. and subsidiaries BCM57414

NetXtreme-E 10Gb/25Gb RDMA Ethernet Controller [14e4:16d7]
| | \-00.1 Broadcom Inc. and subsidiaries BCM57414
NetXtreme-E 10Gb/25Gb RDMA Ethernet Controller [14e4:16d7]

iomem layout after init bootup:

   22fffec00000-22ffffefffff : PCI Bus 0000:31 [Window size=19M]
     22fffec00000-22ffff3fffff : 0000:31:00.1  [align=1M  size=8M   BAR 9
(VF BAR 2)]
     22ffff400000-22ffffbfffff : 0000:31:00.0  [align=1M  size=8M   BAR 9
(VF BAR 2)]
     22ffffc00000-22ffffcfffff : 0000:31:00.1  [align=1M  size=1M   BAR 2]
     22ffffd00000-22ffffdfffff : 0000:31:00.0  [align=1M  size=1M   BAR 2]
     22ffffe00000-22ffffe0ffff : 0000:31:00.1  [align=64K size=64K  BAR 0]
     22ffffe10000-22ffffe1ffff : 0000:31:00.0  [align=64K size=64K  BAR 0]
     22ffffe20000-22ffffe3ffff : 0000:31:00.1  [align=16K size=128K BAR
11(VF BAR 4)]
     22ffffe40000-22ffffe5ffff : 0000:31:00.1  [align=16K size=128K BAR 7
(VF BAR 0)]
     22ffffe60000-22ffffe7ffff : 0000:31:00.0  [align=16K size=128K BAR
11(VF BAR 4)]
     22ffffe80000-22ffffe9ffff : 0000:31:00.0  [align=16K size=128K BAR 7
(VF BAR 0)]
     22ffffea0000-22ffffea1fff : 0000:31:00.1  [align=8K  size=8K   BAR 4]
     22ffffea2000-22ffffea3fff : 0000:31:00.0  [align=8K  size=8K   BAR 4]

iomem layout after remove function 1 by
echo "1" > /sys/bus/pci/devices/0000:31:00.1/remove

   22fffec00000-22ffffefffff : PCI Bus 0000:31 [Window size=19M]
     22ffff400000-22ffffbfffff : 0000:31:00.0  [align=1M  size=8M   BAR 9
(VF BAR 2)]
     22ffffd00000-22ffffdfffff : 0000:31:00.0  [align=1M  size=1M   BAR 2]
     22ffffe10000-22ffffe1ffff : 0000:31:00.0  [align=64K size=64K  BAR 0]
     22ffffe60000-22ffffe7ffff : 0000:31:00.0  [align=16K size=128K BAR
11(VF BAR 4)]
     22ffffe80000-22ffffe9ffff : 0000:31:00.0  [align=16K size=128K BAR 7
(VF BAR 0)]
     22ffffea2000-22ffffea3fff : 0000:31:00.0  [align=8K  size=8K   BAR 4]

Rescan logs triggered by
echo "1" > /sys/bus/pci/devices/0000:30:02.0/rescan

[   90.585067] pci 0000:31:00.1: [14e4:16d7] type 00 class 0x020000 PCIe
Endpoint
[   90.585107] pci 0000:31:00.1: BAR 0 [mem 0x22ffffe00000-0x22ffffe0ffff
64bit pref]
[   90.585113] pci 0000:31:00.1: BAR 2 [mem 0x22ffffc00000-0x22ffffcfffff
64bit pref]
[   90.585116] pci 0000:31:00.1: BAR 4 [mem 0x22ffffea0000-0x22ffffea1fff
64bit pref]
[   90.585119] pci 0000:31:00.1: ROM [mem 0xb0e00000-0xb0e7ffff pref]
[   90.585216] pci 0000:31:00.1: PME# supported from D0 D3hot D3cold
[   90.585253] pci 0000:31:00.1: VF BAR 0 [mem
0x22ffffe40000-0x22ffffe43fff 64bit pref]
[   90.585255] pci 0000:31:00.1: VF BAR 0 [mem
0x22ffffe40000-0x22ffffe5ffff 64bit pref]: contains BAR 0 for 8 VFs
[   90.585258] pci 0000:31:00.1: VF BAR 2 [mem
0x22fffec00000-0x22fffecfffff 64bit pref]
[   90.585260] pci 0000:31:00.1: VF BAR 2 [mem
0x22fffec00000-0x22ffff3fffff 64bit pref]: contains BAR 2 for 8 VFs
[   90.585263] pci 0000:31:00.1: VF BAR 4 [mem
0x22ffffe20000-0x22ffffe23fff 64bit pref]
[   90.585265] pci 0000:31:00.1: VF BAR 4 [mem
0x22ffffe20000-0x22ffffe3ffff 64bit pref]: contains BAR 4 for 8 VFs
[   90.585534] pci 0000:31:00.1: Adding to iommu group 11
[   90.585575] pci 0000:31:00.1: BAR 2 [mem 0x22fffec00000-0x22fffecfffff
64bit pref]: assigned
[   90.585585] pci 0000:31:00.1: VF BAR 2 [mem size 0x00800000 64bit
pref]: can't assign; no space
[   90.585587] pci 0000:31:00.1: VF BAR 2 [mem size 0x00800000 64bit
pref]: failed to assign
[   90.585589] pci 0000:31:00.1: ROM [mem 0xb0e00000-0xb0e7ffff pref]:
assigned
[   90.585591] pci 0000:31:00.1: BAR 0 [mem 0x22fffed00000-0x22fffed0ffff
64bit pref]: assigned
[   90.585599] pci 0000:31:00.1: VF BAR 0 [mem
0x22fffed10000-0x22fffed2ffff 64bit pref]: assigned
[   90.585603] pci 0000:31:00.1: VF BAR 4 [mem
0x22fffed30000-0x22fffed4ffff 64bit pref]: assigned
[   90.585606] pci 0000:31:00.1: BAR 4 [mem 0x22fffed50000-0x22fffed51fff
Quoted 1 line

64bit pref]: assigned

Timestamps are irrelevant noise to the problem and should be removed.

Quoted 22 lines

Enable sriov failed logs triggered by
echo 2 > /sys/bus/pci/devices/0000:31:00.1/sriov_numvfs

[ 1666.918432] bnxt_en 0000:31:00.1: not enough MMIO resources for SR-IOV
[ 1666.918442] bnxt_en 0000:31:00.1 eth5: pci_enable_sriov failed : -12

The resource allocation process during rescan is as follows:

dev_rescan_store
pci_rescan_bus
pci_assign_unassigned_bus_resources
__pci_bus_assign_resources
pbus_assign_resources_sorted
pdev_sort_resources
__assign_resources_sorted
assign_requested_resources_sorted
pci_assign_resource

We noticed that current sort algorithm is only by alignment. The BAR 2 (align=1M size=1M) is located before BAR 9 (VF BAR 2 align=1M size=8M), so the 8M cannot be satisfied.

Quoted 7 lines
Quoted 4 lines

I think you have a typo (located vs allocated)? The difference changes meaning significantly. (located implies before in address, allocated implies before in the order of made allocations).

My description could indeed be misleading. My intention was that BAR 2 is before BAR 9 in the sorted list, and therefore it is allocated before BAR 9.

Yes. First I misunderstood you but realized later what was your meaning after I managed to decipher what is the story your logs told.

I'd prefer the changelog is written such that logs only prove things happened the way you description they did. That is, even if all lines would be deleted, the person looking at your patch should understand what's going wrong.

Quoted 45 lines
Quoted 6 lines
Quoted 6 lines

If we keep alignment as primary sorting key, but use size as secondary key, all resource can be satisfied when remove & rescan.

Does this approach only solve current specific case as a workaround, or does it also benefit general PCI resource allocation?

Quoted 8 lines
Quoted 2 lines

I think it may help reduce allocation failures due to fragmentation theoretically, but I'm not sure.

I suppose trying the largest first does generally increase the chances of success in cases where the resource sizes are very heterogeneous like in your
case, not just in this case.

Thanks for agreeing with this point.

Quoted 7 lines

You should really rewrite the changelog text though. Try more to focus on how the other allocations from the sibling make it possible to fit some of the resource(s) only into a single place within the window. And therefore largest one should be requested first. I had to figure that bit myself as you didn't clearly state why it fails but only talked vaguely about the order of (al)location(s).

After .1 BAR 2 assigned, the iomem layout (deduce should be):

22fffec00000-22ffffefffff : PCI Bus 0000:31 [Window size=19M]

    22fffec00000-22fffecfffff : 0000:31:00.1  [align=1M  size=1M   BAR 2]
                                              [gap=7M]
    22ffff400000-22ffffbfffff : 0000:31:00.0  [align=1M  size=8M   BAR 9 (VF
BAR 2)]
                                              [gap=1M]
    22ffffd00000-22ffffdfffff : 0000:31:00.0  [align=1M  size=1M   BAR 2]
                                              [gap=64K]
    22ffffe10000-22ffffe1ffff : 0000:31:00.0  [align=64K size=64K  BAR 0]
                                              [gap=256K]
    22ffffe60000-22ffffe7ffff : 0000:31:00.0  [align=16K size=128K BAR 11(VF
BAR 4)]
    22ffffe80000-22ffffe9ffff : 0000:31:00.0  [align=16K size=128K BAR 7 (VF
BAR 0)]
                                              [gap=8K]
    22ffffea2000-22ffffea3fff : 0000:31:00.0  [align=8K  size=8K   BAR 4]
                                              [gap=368K]
Quoted 4 lines

And then there is no suitable space available to satisfy both align=1M and size=8M (BAR 9),
that lead to "0000:31:00.1: VF BAR 2 [mem size 0x00800000 64bit pref]: can't assign; no space".

I figured that out myself by deciphering your logs (which has quite high cost from reviewer point of view). It would have been much easier to tell expliciltly that resource tree ends up into this intermediate state while doing the assignments.

That is, explicitly state before the log snippets there are 1M and 8M empty gaps in the bridge window, and the greedy approach places size=1M,align=1M one first into 8M gap, leaving no space for the size=8M,align=1M resource.

Quoted 10 lines
Quoted 5 lines

There will still be some cases this greedy approach will not get right, such as align=2M,size=2M & align=1M,size=8M. This algorithm is not really designed for filling gaps in a window but the entire window from scratch, which is why it cannot handle all cases.

Does the "remove & rescan" is still a corner case for kernel, especially that only removing a single function rather than the entire device for a multi-function
device, even after you've fixed multiple issues in this scenario?

I didn't say it's a corner case but just pointed up there will always be some cases with which a greedy approach will end up in tears. How common they are, I don't know (but suspect they're rare).

I'd prefer remove + rescan for the same device to work at least as good as before remove.

To go beyond that, tracking things is hard with the current algorith. The complexity comes from the disjoint nature of sizing and assignment. We cannot easily retry sizing after learning there's an assignment failure because we lack of persisting structure where to store the fitting information (struct pci_dev_resource that would last across resource fitting rounds/passes). Those challenges make it a bit harder to come up better fitting strategies.

Quoted 1 line

Does the community resent the issues caused by this scenario?

I suspect those how encounter such issues feel powerless to make any change to it, even if some would resent their failing cases. Also, changing one thing easily breaks another scenario risking revert and return to status quo. This algorithm is not exactly easy to approach to and contains very much decades old code (largely unexplained, of course).

Quoted 22 lines
Quoted 22 lines
Quoted 12 lines

Appreciate any comment and suggestion, thanks.

---
  drivers/pci/setup-bus.c | 3 ++-
  1 file changed, 2 insertions(+), 1 deletion(-)
drivers/pci/setup-bus.c +0 -0
index 4cf120ebe5ad..63f224f0c6be 100644

drivers/pci/setup-bus.c static void pdev_sort_resources(struct pci_dev *dev, · line 373

Quoted 7 lines
drivers/pci/setup-bus.c +2 -0
@@ -367,7 +367,8 @@ static void pdev_sort_resources(struct pci_dev *dev,
 struct list_head *head)
  			align = pci_resource_alignment(dev_res->dev,
  							 dev_res->res);
  -			if (r_align > align) {
+			if (r_align > align ||
+			    (r_align == align && resource_size(r) >
 resource_size(dev_res->res))) {

This is not the only place where the algorithm does sorting.

Quoted 5 lines
Quoted 2 lines

(And I also know one restore place is lacking restoring ordering.)

Do you mean in __assign_resources_sorted(), retry normal assign after add_size assign failed?

Yes.

Quoted 11 lines

I noticed this function after being replied by Sashiko AI review.

Quoted 7 lines

I was planning to move to rbtree for storing the resources that need to be in a certain order as doing it everywhere results in small variations which is error prone.

So maybe it would be time to consider moving to that so we could do the sort order in one place.

Thank you for your guidance.

Signature · 1 line
 i.