Skip to contentSkip to conversation

pci: acpiphp: try to reassign resources on bridge if necessary

PATCH Superseded Igor Mammedov 4 participants 9 messages Apr 18, 2023 – Apr 24, 2023

Subsystems (2)

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 pci: acpiphp: try to reassign resources on bridge if necessary

When using ACPI PCI hotplug, hotplugging a device with large BARs may fail if bridge windows programmed by firmware are not large enough.

Reproducer:

  $ qemu-kvm -monitor stdio -M q35  -m 4G \
      -global ICH9-LPC.acpi-pci-hotplug-with-bridge-support=on \
      -device id=rp1,pcie-root-port,bus=pcie.0,chassis=4 \
      disk_image

wait till linux guest boots, then hotplug device
(qemu) device_add qxl,bus=rp1

hotplug on guest side fails with:
pci 0000:01:00.0: [1b36:0100] type 00 class 0x038000 pci 0000:01:00.0: reg 0x10: [mem 0x00000000-0x03ffffff] pci 0000:01:00.0: reg 0x14: [mem 0x00000000-0x03ffffff] pci 0000:01:00.0: reg 0x18: [mem 0x00000000-0x00001fff] pci 0000:01:00.0: reg 0x1c: [io 0x0000-0x001f]
pci 0000:01:00.0: BAR 0: no space for [mem size 0x04000000] pci 0000:01:00.0: BAR 0: failed to assign [mem size 0x04000000] pci 0000:01:00.0: BAR 1: no space for [mem size 0x04000000] pci 0000:01:00.0: BAR 1: failed to assign [mem size 0x04000000] pci 0000:01:00.0: BAR 2: assigned [mem 0xfe800000-0xfe801fff] pci 0000:01:00.0: BAR 3: assigned [io 0x1000-0x101f] qxl 0000:01:00.0: enabling device (0000 -> 0003) Unable to create vram_mapping
qxl: probe of 0000:01:00.0 failed with error -12

However when using native PCIe hotplug
'-global ICH9-LPC.acpi-pci-hotplug-with-bridge-support=off'
it works fine, since kernel attempts to reassign unused resources. Use the same machinery as native PCIe hotplug to (re)assign resources.

---
tested in QEMU with Q35 machine on PCIE root port and also with nested conventional bridge attached to root port.

---
 drivers/pci/hotplug/acpiphp_glue.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/pci/hotplug/acpiphp_glue.c +1 -1
index 5b1f271c6034..9aebde28a92f 100644
@@ -517,7 +517,7 @@ static void enable_slot(struct acpiphp_slot *slot, bool bridge)
 				}
 			}
 		}
-		__pci_bus_assign_resources(bus, &add_list, NULL);
+		pci_assign_unassigned_bridge_resources(bus->self);
 	}
 
 	acpiphp_sanitize_bus(bus);
Signature · 1 line
2.39.1
Acked-by on v1 1/1
#1 lore raw
Quoted 29 lines · Igor Mammedov

When using ACPI PCI hotplug, hotplugging a device with large BARs may fail if bridge windows programmed by firmware are not large enough.

Reproducer:

  $ qemu-kvm -monitor stdio -M q35  -m 4G \
      -global ICH9-LPC.acpi-pci-hotplug-with-bridge-support=on \
      -device id=rp1,pcie-root-port,bus=pcie.0,chassis=4 \
      disk_image

wait till linux guest boots, then hotplug device
(qemu) device_add qxl,bus=rp1

hotplug on guest side fails with:

   pci 0000:01:00.0: [1b36:0100] type 00 class 0x038000
   pci 0000:01:00.0: reg 0x10: [mem 0x00000000-0x03ffffff]
   pci 0000:01:00.0: reg 0x14: [mem 0x00000000-0x03ffffff]
   pci 0000:01:00.0: reg 0x18: [mem 0x00000000-0x00001fff]
   pci 0000:01:00.0: reg 0x1c: [io  0x0000-0x001f]
   pci 0000:01:00.0: BAR 0: no space for [mem size 0x04000000]
   pci 0000:01:00.0: BAR 0: failed to assign [mem size 0x04000000]
   pci 0000:01:00.0: BAR 1: no space for [mem size 0x04000000]
   pci 0000:01:00.0: BAR 1: failed to assign [mem size 0x04000000]
   pci 0000:01:00.0: BAR 2: assigned [mem 0xfe800000-0xfe801fff]
   pci 0000:01:00.0: BAR 3: assigned [io  0x1000-0x101f]
   qxl 0000:01:00.0: enabling device (0000 -> 0003)
   Unable to create vram_mapping
   qxl: probe of 0000:01:00.0 failed with error -12
Quoted 6 lines

However when using native PCIe hotplug
'-global ICH9-LPC.acpi-pci-hotplug-with-bridge-support=off'
it works fine, since kernel attempts to reassign unused resources. Use the same machinery as native PCIe hotplug to (re)assign resources.

And I think:

Fixes d66ecb7220a7 ("PCI / ACPI: Use boot-time resource allocation rules during hotplug")
Quoted 22 lines

---
tested in QEMU with Q35 machine on PCIE root port and also with nested conventional bridge attached to root port.

---
 drivers/pci/hotplug/acpiphp_glue.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/pci/hotplug/acpiphp_glue.c +1 -1
index 5b1f271c6034..9aebde28a92f 100644
@@ -517,7 +517,7 @@ static void enable_slot(struct acpiphp_slot *slot, bool bridge)
 				}
 			}
 		}
-		__pci_bus_assign_resources(bus, &add_list, NULL);
+		pci_assign_unassigned_bridge_resources(bus->self);
 	}
 
 	acpiphp_sanitize_bus(bus);
Signature · 1 line
2.39.1
#2 lore raw
Quoted 35 lines · On Tue, 18 Apr 2023 07:08:09 -0400 "Michael S. Tsirkin" <[email protected]> wrote:
Quoted 34 lines · Igor Mammedov

When using ACPI PCI hotplug, hotplugging a device with large BARs may fail if bridge windows programmed by firmware are not large enough.

Reproducer:

  $ qemu-kvm -monitor stdio -M q35  -m 4G \
      -global ICH9-LPC.acpi-pci-hotplug-with-bridge-support=on \
      -device id=rp1,pcie-root-port,bus=pcie.0,chassis=4 \
      disk_image

wait till linux guest boots, then hotplug device
(qemu) device_add qxl,bus=rp1

hotplug on guest side fails with:

   pci 0000:01:00.0: [1b36:0100] type 00 class 0x038000
   pci 0000:01:00.0: reg 0x10: [mem 0x00000000-0x03ffffff]
   pci 0000:01:00.0: reg 0x14: [mem 0x00000000-0x03ffffff]
   pci 0000:01:00.0: reg 0x18: [mem 0x00000000-0x00001fff]
   pci 0000:01:00.0: reg 0x1c: [io  0x0000-0x001f]
   pci 0000:01:00.0: BAR 0: no space for [mem size 0x04000000]
   pci 0000:01:00.0: BAR 0: failed to assign [mem size 0x04000000]
   pci 0000:01:00.0: BAR 1: no space for [mem size 0x04000000]
   pci 0000:01:00.0: BAR 1: failed to assign [mem size 0x04000000]
   pci 0000:01:00.0: BAR 2: assigned [mem 0xfe800000-0xfe801fff]
   pci 0000:01:00.0: BAR 3: assigned [io  0x1000-0x101f]
   qxl 0000:01:00.0: enabling device (0000 -> 0003)
   Unable to create vram_mapping
   qxl: probe of 0000:01:00.0 failed with error -12

However when using native PCIe hotplug
'-global ICH9-LPC.acpi-pci-hotplug-with-bridge-support=off'
it works fine, since kernel attempts to reassign unused resources. Use the same machinery as native PCIe hotplug to (re)assign resources.

Quoted 7 lines
Quoted 1 line

And I think:

Fixes d66ecb7220a7 ("PCI / ACPI: Use boot-time resource allocation rules during hotplug")

Probably not, this commit basically added pcibios_resource_survey_bus() and nothing else important. Looking through history it was always broken this way.

Quoted 24 lines
Quoted 23 lines

---
tested in QEMU with Q35 machine on PCIE root port and also with nested conventional bridge attached to root port.

---
 drivers/pci/hotplug/acpiphp_glue.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/pci/hotplug/acpiphp_glue.c +1 -1
index 5b1f271c6034..9aebde28a92f 100644
@@ -517,7 +517,7 @@ static void enable_slot(struct acpiphp_slot *slot, bool bridge)
 				}
 			}
 		}
-		__pci_bus_assign_resources(bus, &add_list, NULL);
+		pci_assign_unassigned_bridge_resources(bus->self);
 	}
 
 	acpiphp_sanitize_bus(bus);
Signature · 1 line
2.39.1  
maintainer Acked-by on v1 1/1
#3 lore raw
In reply to Igor Mammedov
Quoted 30 lines · Igor Mammedov

When using ACPI PCI hotplug, hotplugging a device with large BARs may fail if bridge windows programmed by firmware are not large enough.

Reproducer:

  $ qemu-kvm -monitor stdio -M q35  -m 4G \
      -global ICH9-LPC.acpi-pci-hotplug-with-bridge-support=on \
      -device id=rp1,pcie-root-port,bus=pcie.0,chassis=4 \
      disk_image

wait till linux guest boots, then hotplug device
(qemu) device_add qxl,bus=rp1

hotplug on guest side fails with:

   pci 0000:01:00.0: [1b36:0100] type 00 class 0x038000
   pci 0000:01:00.0: reg 0x10: [mem 0x00000000-0x03ffffff]
   pci 0000:01:00.0: reg 0x14: [mem 0x00000000-0x03ffffff]
   pci 0000:01:00.0: reg 0x18: [mem 0x00000000-0x00001fff]
   pci 0000:01:00.0: reg 0x1c: [io  0x0000-0x001f]
   pci 0000:01:00.0: BAR 0: no space for [mem size 0x04000000]
   pci 0000:01:00.0: BAR 0: failed to assign [mem size 0x04000000]
   pci 0000:01:00.0: BAR 1: no space for [mem size 0x04000000]
   pci 0000:01:00.0: BAR 1: failed to assign [mem size 0x04000000]
   pci 0000:01:00.0: BAR 2: assigned [mem 0xfe800000-0xfe801fff]
   pci 0000:01:00.0: BAR 3: assigned [io  0x1000-0x101f]
   qxl 0000:01:00.0: enabling device (0000 -> 0003)
   Unable to create vram_mapping
   qxl: probe of 0000:01:00.0 failed with error -12
Quoted 6 lines

However when using native PCIe hotplug
'-global ICH9-LPC.acpi-pci-hotplug-with-bridge-support=off'
it works fine, since kernel attempts to reassign unused resources. Use the same machinery as native PCIe hotplug to (re)assign resources.

or please let me know if you want me to pick this up.

Quoted 23 lines

---
tested in QEMU with Q35 machine on PCIE root port and also with nested conventional bridge attached to root port.

---
 drivers/pci/hotplug/acpiphp_glue.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/pci/hotplug/acpiphp_glue.c +1 -2
index 5b1f271c6034..9aebde28a92f 100644
@@ -517,7 +517,7 @@ static void enable_slot(struct acpiphp_slot *slot, bool bridge)
                                }
                        }
                }
-               __pci_bus_assign_resources(bus, &add_list, NULL);
+               pci_assign_unassigned_bridge_resources(bus->self);
        }
 
        acpiphp_sanitize_bus(bus);
--

2.39.1

#4 lore raw
Quoted 36 lines · On Tue, 18 Apr 2023 14:55:29 +0200 "Rafael J. Wysocki" <[email protected]>…
Quoted 35 lines · Igor Mammedov

When using ACPI PCI hotplug, hotplugging a device with large BARs may fail if bridge windows programmed by firmware are not large enough.

Reproducer:

  $ qemu-kvm -monitor stdio -M q35  -m 4G \
      -global ICH9-LPC.acpi-pci-hotplug-with-bridge-support=on \
      -device id=rp1,pcie-root-port,bus=pcie.0,chassis=4 \
      disk_image

wait till linux guest boots, then hotplug device
(qemu) device_add qxl,bus=rp1

hotplug on guest side fails with:

   pci 0000:01:00.0: [1b36:0100] type 00 class 0x038000
   pci 0000:01:00.0: reg 0x10: [mem 0x00000000-0x03ffffff]
   pci 0000:01:00.0: reg 0x14: [mem 0x00000000-0x03ffffff]
   pci 0000:01:00.0: reg 0x18: [mem 0x00000000-0x00001fff]
   pci 0000:01:00.0: reg 0x1c: [io  0x0000-0x001f]
   pci 0000:01:00.0: BAR 0: no space for [mem size 0x04000000]
   pci 0000:01:00.0: BAR 0: failed to assign [mem size 0x04000000]
   pci 0000:01:00.0: BAR 1: no space for [mem size 0x04000000]
   pci 0000:01:00.0: BAR 1: failed to assign [mem size 0x04000000]
   pci 0000:01:00.0: BAR 2: assigned [mem 0xfe800000-0xfe801fff]
   pci 0000:01:00.0: BAR 3: assigned [io  0x1000-0x101f]
   qxl 0000:01:00.0: enabling device (0000 -> 0003)
   Unable to create vram_mapping
   qxl: probe of 0000:01:00.0 failed with error -12

However when using native PCIe hotplug
'-global ICH9-LPC.acpi-pci-hotplug-with-bridge-support=off'
it works fine, since kernel attempts to reassign unused resources. Use the same machinery as native PCIe hotplug to (re)assign resources.

Quoted 5 lines
Quoted 1 line

or please let me know if you want me to pick this up.

It would be nice if you could pick it up.
Thanks!

Quoted 25 lines
Quoted 24 lines

---
tested in QEMU with Q35 machine on PCIE root port and also with nested conventional bridge attached to root port.

---
 drivers/pci/hotplug/acpiphp_glue.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/pci/hotplug/acpiphp_glue.c +1 -2
index 5b1f271c6034..9aebde28a92f 100644
@@ -517,7 +517,7 @@ static void enable_slot(struct acpiphp_slot *slot, bool bridge)
                                }
                        }
                }
-               __pci_bus_assign_resources(bus, &add_list, NULL);
+               pci_assign_unassigned_bridge_resources(bus->self);
        }
 
        acpiphp_sanitize_bus(bus);
--

2.39.1

#5 lore raw
Quoted 40 lines · Igor Mammedov
Quoted 36 lines · On Tue, 18 Apr 2023 14:55:29 +0200 "Rafael J. Wysocki" <[email protected]>…
Quoted 35 lines · Igor Mammedov

When using ACPI PCI hotplug, hotplugging a device with large BARs may fail if bridge windows programmed by firmware are not large enough.

Reproducer:

  $ qemu-kvm -monitor stdio -M q35  -m 4G \
      -global ICH9-LPC.acpi-pci-hotplug-with-bridge-support=on \
      -device id=rp1,pcie-root-port,bus=pcie.0,chassis=4 \
      disk_image

wait till linux guest boots, then hotplug device
(qemu) device_add qxl,bus=rp1

hotplug on guest side fails with:

   pci 0000:01:00.0: [1b36:0100] type 00 class 0x038000
   pci 0000:01:00.0: reg 0x10: [mem 0x00000000-0x03ffffff]
   pci 0000:01:00.0: reg 0x14: [mem 0x00000000-0x03ffffff]
   pci 0000:01:00.0: reg 0x18: [mem 0x00000000-0x00001fff]
   pci 0000:01:00.0: reg 0x1c: [io  0x0000-0x001f]
   pci 0000:01:00.0: BAR 0: no space for [mem size 0x04000000]
   pci 0000:01:00.0: BAR 0: failed to assign [mem size 0x04000000]
   pci 0000:01:00.0: BAR 1: no space for [mem size 0x04000000]
   pci 0000:01:00.0: BAR 1: failed to assign [mem size 0x04000000]
   pci 0000:01:00.0: BAR 2: assigned [mem 0xfe800000-0xfe801fff]
   pci 0000:01:00.0: BAR 3: assigned [io  0x1000-0x101f]
   qxl 0000:01:00.0: enabling device (0000 -> 0003)
   Unable to create vram_mapping
   qxl: probe of 0000:01:00.0 failed with error -12

However when using native PCIe hotplug
'-global ICH9-LPC.acpi-pci-hotplug-with-bridge-support=off'
it works fine, since kernel attempts to reassign unused resources. Use the same machinery as native PCIe hotplug to (re)assign resources.

Quoted 7 lines
Quoted 5 lines
Quoted 1 line

or please let me know if you want me to pick this up.

It would be nice if you could pick it up.

OK, I'll do that unless Bjorn tells me that he prefers to take it via the PCI tree.

Thanks!

Quoted 26 lines
Quoted 25 lines
Quoted 24 lines

---
tested in QEMU with Q35 machine on PCIE root port and also with nested conventional bridge attached to root port.

---
 drivers/pci/hotplug/acpiphp_glue.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/pci/hotplug/acpiphp_glue.c +1 -2
index 5b1f271c6034..9aebde28a92f 100644
@@ -517,7 +517,7 @@ static void enable_slot(struct acpiphp_slot *slot, bool bridge)
                                }
                        }
                }
-               __pci_bus_assign_resources(bus, &add_list, NULL);
+               pci_assign_unassigned_bridge_resources(bus->self);
        }
 
        acpiphp_sanitize_bus(bus);
--

2.39.1

#6 lore raw

[+cc Mika, who made previous changes in this area]

Quoted 34 lines · Rafael J. Wysocki
Quoted 33 lines · Igor Mammedov
Quoted 31 lines · On Tue, 18 Apr 2023 14:55:29 +0200 "Rafael J. Wysocki" <[email protected]>…
Quoted 30 lines · Igor Mammedov

When using ACPI PCI hotplug, hotplugging a device with large BARs may fail if bridge windows programmed by firmware are not large enough.

Reproducer:

  $ qemu-kvm -monitor stdio -M q35  -m 4G \
      -global ICH9-LPC.acpi-pci-hotplug-with-bridge-support=on \
      -device id=rp1,pcie-root-port,bus=pcie.0,chassis=4 \
      disk_image

wait till linux guest boots, then hotplug device
(qemu) device_add qxl,bus=rp1

hotplug on guest side fails with:

   pci 0000:01:00.0: [1b36:0100] type 00 class 0x038000
   pci 0000:01:00.0: reg 0x10: [mem 0x00000000-0x03ffffff]
   pci 0000:01:00.0: reg 0x14: [mem 0x00000000-0x03ffffff]
   pci 0000:01:00.0: reg 0x18: [mem 0x00000000-0x00001fff]
   pci 0000:01:00.0: reg 0x1c: [io  0x0000-0x001f]
   pci 0000:01:00.0: BAR 0: no space for [mem size 0x04000000]
   pci 0000:01:00.0: BAR 0: failed to assign [mem size 0x04000000]
   pci 0000:01:00.0: BAR 1: no space for [mem size 0x04000000]
   pci 0000:01:00.0: BAR 1: failed to assign [mem size 0x04000000]
   pci 0000:01:00.0: BAR 2: assigned [mem 0xfe800000-0xfe801fff]
   pci 0000:01:00.0: BAR 3: assigned [io  0x1000-0x101f]
   qxl 0000:01:00.0: enabling device (0000 -> 0003)
   Unable to create vram_mapping
   qxl: probe of 0000:01:00.0 failed with error -12
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

However when using native PCIe hotplug
'-global ICH9-LPC.acpi-pci-hotplug-with-bridge-support=off'
it works fine, since kernel attempts to reassign unused resources. Use the same machinery as native PCIe hotplug to (re)assign resources.

Thanks for the nice reproducer and logs!

Quoted 10 lines
Quoted 7 lines
Quoted 5 lines
Quoted 1 line

or please let me know if you want me to pick this up.

It would be nice if you could pick it up.

OK, I'll do that unless Bjorn tells me that he prefers to take it via the PCI tree.

It's OK with me if you pick this up, but please update the subject to use the style of previous commits, e.g.,

PCI: acpiphp: Reassign resources on bridge if necessary

Previous changes involving pci_assign_unassigned_bridge_resources() in enable_slot() (these are from Mika, so I cc'd him in case he wants to comment):

84c8b58ed3ad ("ACPI / hotplug / PCI: Don't scan bridges managed by native hotplug") 77adf9355304 ("ACPI / hotplug / PCI: Allocate resources directly under the non-hotplug bridge")

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

---
tested in QEMU with Q35 machine on PCIE root port and also with nested conventional bridge attached to root port.

---
 drivers/pci/hotplug/acpiphp_glue.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

drivers/pci/hotplug/acpiphp_glue.c static void enable_slot(struct acpiphp_slot *slot, bool bridge)

Quoted 5 lines
drivers/pci/hotplug/acpiphp_glue.c +0 -0
index 5b1f271c6034..9aebde28a92f 100644
@@ -517,7 +517,7 @@ static void enable_slot(struct acpiphp_slot *slot, bool bridge)

Previous context:

                                             __pci_bus_size_bridges(dev->subordinate,
                                                                    &add_list);
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

drivers/pci/hotplug/acpiphp_glue.c static void enable_slot(struct acpiphp_slot *slot, bool bridge) · line 520

Quoted 5 lines
drivers/pci/hotplug/acpiphp_glue.c +1 -1
                                }
                        }
                }
-               __pci_bus_assign_resources(bus, &add_list, NULL);
+               pci_assign_unassigned_bridge_resources(bus->self);

"add_list" is now used only for __pci_bus_size_bridges(), which *looks* unnecessary unless there's some obscure side-effect of that path when that parameter is non-NULL.

If "add_list" is unnecessary, you would probably use pci_bus_size_bridges() above instead of __pci_bus_size_bridges().

After this patch, we have:

  if (bridge && bus->self && hotplug_is_native(bus->self)) {
    for_each_pci_bridge(dev, bus)
      acpiphp_native_scan_bridge(dev);
  } else {
    ...
    pci_assign_unassigned_bridge_resources(bus->self);
  }

We do not do pci_assign_unassigned_bridge_resources() in the "then" part of the "if". Per the comment, that case may be used for adding Thunderbolt controllers. Is there a reason we do not want pci_assign_unassigned_bridge_resources() in that path, or should it be in both cases?

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
drivers/pci/hotplug/acpiphp_glue.c +0 -0
        }
 
        acpiphp_sanitize_bus(bus);
#7 lore raw
Quoted 91 lines · On Tue, 18 Apr 2023 11:31:14 -0500 Bjorn Helgaas <[email protected]> wrote:

[+cc Mika, who made previous changes in this area]

Quoted 34 lines · Rafael J. Wysocki
Quoted 33 lines · Igor Mammedov
Quoted 31 lines · On Tue, 18 Apr 2023 14:55:29 +0200 "Rafael J. Wysocki" <[email protected]>…
Quoted 30 lines · Igor Mammedov

When using ACPI PCI hotplug, hotplugging a device with large BARs may fail if bridge windows programmed by firmware are not large enough.

Reproducer:

  $ qemu-kvm -monitor stdio -M q35  -m 4G \
      -global ICH9-LPC.acpi-pci-hotplug-with-bridge-support=on \
      -device id=rp1,pcie-root-port,bus=pcie.0,chassis=4 \
      disk_image

wait till linux guest boots, then hotplug device
(qemu) device_add qxl,bus=rp1

hotplug on guest side fails with:

   pci 0000:01:00.0: [1b36:0100] type 00 class 0x038000
   pci 0000:01:00.0: reg 0x10: [mem 0x00000000-0x03ffffff]
   pci 0000:01:00.0: reg 0x14: [mem 0x00000000-0x03ffffff]
   pci 0000:01:00.0: reg 0x18: [mem 0x00000000-0x00001fff]
   pci 0000:01:00.0: reg 0x1c: [io  0x0000-0x001f]
   pci 0000:01:00.0: BAR 0: no space for [mem size 0x04000000]
   pci 0000:01:00.0: BAR 0: failed to assign [mem size 0x04000000]
   pci 0000:01:00.0: BAR 1: no space for [mem size 0x04000000]
   pci 0000:01:00.0: BAR 1: failed to assign [mem size 0x04000000]
   pci 0000:01:00.0: BAR 2: assigned [mem 0xfe800000-0xfe801fff]
   pci 0000:01:00.0: BAR 3: assigned [io  0x1000-0x101f]
   qxl 0000:01:00.0: enabling device (0000 -> 0003)
   Unable to create vram_mapping
   qxl: probe of 0000:01:00.0 failed with error -12
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

However when using native PCIe hotplug
'-global ICH9-LPC.acpi-pci-hotplug-with-bridge-support=off'
it works fine, since kernel attempts to reassign unused resources. Use the same machinery as native PCIe hotplug to (re)assign resources.

Thanks for the nice reproducer and logs!

Quoted 10 lines
Quoted 7 lines
Quoted 5 lines
Quoted 1 line

or please let me know if you want me to pick this up.

It would be nice if you could pick it up.

OK, I'll do that unless Bjorn tells me that he prefers to take it via the PCI tree.

It's OK with me if you pick this up, but please update the subject to use the style of previous commits, e.g.,

PCI: acpiphp: Reassign resources on bridge if necessary

Previous changes involving pci_assign_unassigned_bridge_resources() in enable_slot() (these are from Mika, so I cc'd him in case he wants to comment):

84c8b58ed3ad ("ACPI / hotplug / PCI: Don't scan bridges managed by native hotplug") 77adf9355304 ("ACPI / hotplug / PCI: Allocate resources directly under the non-hotplug bridge")

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

---
tested in QEMU with Q35 machine on PCIE root port and also with nested conventional bridge attached to root port.

---
 drivers/pci/hotplug/acpiphp_glue.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

drivers/pci/hotplug/acpiphp_glue.c static void enable_slot(struct acpiphp_slot *slot, bool bridge)

Quoted 5 lines
drivers/pci/hotplug/acpiphp_glue.c +0 -0
index 5b1f271c6034..9aebde28a92f 100644
@@ -517,7 +517,7 @@ static void enable_slot(struct acpiphp_slot *slot, bool bridge)  

Previous context:

                                             __pci_bus_size_bridges(dev->subordinate,
                                                                    &add_list);
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

drivers/pci/hotplug/acpiphp_glue.c static void enable_slot(struct acpiphp_slot *slot, bool bridge) · line 520

Quoted 5 lines
drivers/pci/hotplug/acpiphp_glue.c +1 -1
                                }
                        }
                }
-               __pci_bus_assign_resources(bus, &add_list, NULL);
+               pci_assign_unassigned_bridge_resources(bus->self);  
Quoted 6 lines

"add_list" is now used only for __pci_bus_size_bridges(), which *looks* unnecessary unless there's some obscure side-effect of that path when that parameter is non-NULL.

If "add_list" is unnecessary, you would probably use pci_bus_size_bridges() above instead of __pci_bus_size_bridges().

pci_assign_unassigned_bridge_resources() calls __pci_bus_size_bridges() so original one is not needed anymore, in addition it might leak entries added to add_list.
I'll remove __pci_bus_size_bridges() and respin patch (incl. updated subject)

Quoted 10 lines

After this patch, we have:

  if (bridge && bus->self && hotplug_is_native(bus->self)) {
    for_each_pci_bridge(dev, bus)
      acpiphp_native_scan_bridge(dev);
  } else {
    ...
    pci_assign_unassigned_bridge_resources(bus->self);
  }
Quoted 6 lines

We do not do pci_assign_unassigned_bridge_resources() in the "then" part of the "if". Per the comment, that case may be used for adding Thunderbolt controllers. Is there a reason we do not want pci_assign_unassigned_bridge_resources() in that path,
or should it be
in both cases?

acpiphp_native_scan_bridge() looks very similar to 'else' branch modulo skip native hp bridge condition. Otherwise both branches look similar,
but I don't have means to test that usecase,
so I'd avoid touching something nobody complains about.

Quoted 5 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
drivers/pci/hotplug/acpiphp_glue.c +0 -0
        }
 
        acpiphp_sanitize_bus(bus);  
#8 lore raw
Quoted 123 lines · On Mon, 24 Apr 2023 20:50:14 +0200 Igor Mammedov <[email protected]> wrote:
Quoted 91 lines · On Tue, 18 Apr 2023 11:31:14 -0500 Bjorn Helgaas <[email protected]> wrote:

[+cc Mika, who made previous changes in this area]

Quoted 34 lines · Rafael J. Wysocki
Quoted 33 lines · Igor Mammedov
Quoted 31 lines · On Tue, 18 Apr 2023 14:55:29 +0200 "Rafael J. Wysocki" <[email protected]>…
Quoted 30 lines · Igor Mammedov

When using ACPI PCI hotplug, hotplugging a device with large BARs may fail if bridge windows programmed by firmware are not large enough.

Reproducer:

  $ qemu-kvm -monitor stdio -M q35  -m 4G \
      -global ICH9-LPC.acpi-pci-hotplug-with-bridge-support=on \
      -device id=rp1,pcie-root-port,bus=pcie.0,chassis=4 \
      disk_image

wait till linux guest boots, then hotplug device
(qemu) device_add qxl,bus=rp1

hotplug on guest side fails with:

   pci 0000:01:00.0: [1b36:0100] type 00 class 0x038000
   pci 0000:01:00.0: reg 0x10: [mem 0x00000000-0x03ffffff]
   pci 0000:01:00.0: reg 0x14: [mem 0x00000000-0x03ffffff]
   pci 0000:01:00.0: reg 0x18: [mem 0x00000000-0x00001fff]
   pci 0000:01:00.0: reg 0x1c: [io  0x0000-0x001f]
   pci 0000:01:00.0: BAR 0: no space for [mem size 0x04000000]
   pci 0000:01:00.0: BAR 0: failed to assign [mem size 0x04000000]
   pci 0000:01:00.0: BAR 1: no space for [mem size 0x04000000]
   pci 0000:01:00.0: BAR 1: failed to assign [mem size 0x04000000]
   pci 0000:01:00.0: BAR 2: assigned [mem 0xfe800000-0xfe801fff]
   pci 0000:01:00.0: BAR 3: assigned [io  0x1000-0x101f]
   qxl 0000:01:00.0: enabling device (0000 -> 0003)
   Unable to create vram_mapping
   qxl: probe of 0000:01:00.0 failed with error -12
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

However when using native PCIe hotplug
'-global ICH9-LPC.acpi-pci-hotplug-with-bridge-support=off'
it works fine, since kernel attempts to reassign unused resources. Use the same machinery as native PCIe hotplug to (re)assign resources.

Thanks for the nice reproducer and logs!

Quoted 10 lines
Quoted 7 lines
Quoted 5 lines
Quoted 1 line

or please let me know if you want me to pick this up.

It would be nice if you could pick it up.

OK, I'll do that unless Bjorn tells me that he prefers to take it via the PCI tree.

It's OK with me if you pick this up, but please update the subject to use the style of previous commits, e.g.,

PCI: acpiphp: Reassign resources on bridge if necessary

Previous changes involving pci_assign_unassigned_bridge_resources() in enable_slot() (these are from Mika, so I cc'd him in case he wants to comment):

84c8b58ed3ad ("ACPI / hotplug / PCI: Don't scan bridges managed by native hotplug") 77adf9355304 ("ACPI / hotplug / PCI: Allocate resources directly under the non-hotplug bridge")

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

---
tested in QEMU with Q35 machine on PCIE root port and also with nested conventional bridge attached to root port.

---
 drivers/pci/hotplug/acpiphp_glue.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

drivers/pci/hotplug/acpiphp_glue.c static void enable_slot(struct acpiphp_slot *slot, bool bridge)

Quoted 5 lines
drivers/pci/hotplug/acpiphp_glue.c +0 -0
index 5b1f271c6034..9aebde28a92f 100644
@@ -517,7 +517,7 @@ static void enable_slot(struct acpiphp_slot *slot, bool bridge)    

Previous context:

                                             __pci_bus_size_bridges(dev->subordinate,
                                                                    &add_list);
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

drivers/pci/hotplug/acpiphp_glue.c static void enable_slot(struct acpiphp_slot *slot, bool bridge) · line 520

Quoted 5 lines
drivers/pci/hotplug/acpiphp_glue.c +1 -1
                                }
                        }
                }
-               __pci_bus_assign_resources(bus, &add_list, NULL);
+               pci_assign_unassigned_bridge_resources(bus->self);    
Quoted 6 lines

"add_list" is now used only for __pci_bus_size_bridges(), which *looks* unnecessary unless there's some obscure side-effect of that path when that parameter is non-NULL.

If "add_list" is unnecessary, you would probably use pci_bus_size_bridges() above instead of __pci_bus_size_bridges().

pci_assign_unassigned_bridge_resources() calls __pci_bus_size_bridges() so original one is not needed anymore, in addition it might leak entries added to add_list.
I'll remove __pci_bus_size_bridges() and respin patch (incl. updated subject)

Quoted 10 lines

After this patch, we have:

  if (bridge && bus->self && hotplug_is_native(bus->self)) {
    for_each_pci_bridge(dev, bus)
      acpiphp_native_scan_bridge(dev);
  } else {
    ...
    pci_assign_unassigned_bridge_resources(bus->self);
  }
Quoted 6 lines

We do not do pci_assign_unassigned_bridge_resources() in the "then" part of the "if". Per the comment, that case may be used for adding Thunderbolt controllers. Is there a reason we do not want pci_assign_unassigned_bridge_resources() in that path,
or should it be
in both cases?

Quoted 5 lines

acpiphp_native_scan_bridge() looks very similar to 'else' branch modulo skip native hp bridge condition. Otherwise both branches look similar,
but I don't have means to test that usecase,
so I'd avoid touching something nobody complains about.

I gave some more testing to the case with hotplugged sub-bridges, and this patch doesn't help much with that, meaning that pci_assign_unassigned_bridge_resources() when re-sizing goes only to parent for extra resources. So case:
1. hotplug SHPC bridge first & then hotplug a device into hotplugged SHPC bridge, may still fail as SHPC calling its own pci_assign_unassigned_bridge_resources(), will reach only to its parent (root port) which in turn might not have enough resources.
2. hotplugging compound bridge+device attached to it in one go, works as expected since pci_assign_unassigned_bridge_resources() on root port accounts for all needed resources and asks for them from host-bridge.

So basically patch fixes reassignment in case of hotplug into root port (it brings acpiphp on root port on par with native hotplug).

It doesn't work for nested bridges, but the same applies to native hotplug as well.
Perhaps we should make pci_assign_unassigned_bridge_resources() ask for more resources all the way down to host bridge (crude but might be sufficient).

PS:
I've found an attempt to make reassignment work properly (dating to 2012)
https://git.kernel.org/pub/scm/linux/kernel/git/yinghai/linux-yinghai.git/log/
but it doesn't look like it's been merged.

Quoted 5 lines
Quoted 4 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
drivers/pci/hotplug/acpiphp_glue.c +0 -0
        }
 
        acpiphp_sanitize_bus(bus);