Skip to contentSkip to conversation

hw/arm/bcm2838_pcie: make PCI device enumeration work

PATCH In review Marcelo Manzo 2 participants 4 messages Jul 25 – Aug 11

Subsystems (2)
lore
Share & tools
lore
#2 lore raw

PATCH 2/2 tests/functional/aarch64: add raspi4b PCIe enumeration test

In reply to Marcelo Manzo

Add test_arm_raspi4_pcie, exercising the fix in the previous patch:
attach a qemu-xhci controller to the PCIe root port, confirm the guest's driver binds it, and confirm both the root port (00:00.0) and the xHCI controller (01:00.0) show up in /sys/bus/pci/devices/. Before the previous patch this would find no devices at all.

Attaches with msi=off,msix=off: the BCM2838 PCIe root complex's own MSI controller is not implemented, so a device relying on MSI/MSI-X for interrupts never signals its driver. INTx works and is what this test (and any real device behind this root port, for now) relies on.

---
 tests/functional/aarch64/test_raspi4.py | 37 +++++++++++++++++++++++++
 1 file changed, 37 insertions(+)
tests/functional/aarch64/test_raspi4.py +37 -0
index 0e254a20..32542b84 100755
@@ -122,5 +122,42 @@ def test_arm_raspi4_genet(self):
         exec_command_and_wait_for_pattern(self, 'halt', 'reboot: System halted')
 
 
+    def test_arm_raspi4_pcie(self):
+        kernel_path = self.archive_extract(self.ASSET_KERNEL_20190215,
+                                           member='boot/kernel8.img')
+        dtb_path = self.archive_extract(self.ASSET_KERNEL_20190215,
+                                        member='boot/bcm2711-rpi-4-b.dtb')
+        initrd_path = self.uncompress(self.ASSET_INITRD)
+
+        self.set_machine('raspi4b')
+        self.vm.set_console()
+        kernel_command_line = (self.KERNEL_COMMON_COMMAND_LINE +
+                               'earlycon=pl011,mmio32,0xfe201000 ' +
+                               'console=ttyAMA0,115200 ' +
+                               'panic=-1 noreboot ' +
+                               'dwc_otg.fiq_fsm_enable=0')
+        self.vm.add_args('-kernel', kernel_path,
+                         '-dtb', dtb_path,
+                         '-initrd', initrd_path,
+                         '-append', kernel_command_line,
+                         '-no-reboot',
+                         # msi=off,msix=off: the BCM2838 PCIe root complex's
+                         # MSI controller is not implemented, so a device
+                         # relying on MSI/MSI-X for interrupts never signals
+                         # and its driver never probes. INTx works.
+                         '-device',
+                         'qemu-xhci,bus=pcie.1,id=xhci,msi=off,msix=off')
+        self.vm.launch()
+        self.wait_for_console_pattern(
+            'xhci_hcd 0000:01:00.0: xHCI Host Controller')
+        self.wait_for_console_pattern('Boot successful.')
+
+        exec_command_and_wait_for_pattern(self, 'ls /sys/bus/pci/devices/',
+                                                '0000:00:00.0')
+        exec_command_and_wait_for_pattern(self, 'ls /sys/bus/pci/devices/',
+                                                '0000:01:00.0')
+        exec_command_and_wait_for_pattern(self, 'halt', 'reboot: System halted')
+
+
 if __name__ == '__main__':
     LinuxKernelTest.main()
Signature · 1 line
2.47.1
#0 lore raw

cover hw/arm/bcm2838_pcie: make PCI device enumeration work

Follow-up to "hw/arm/raspi4b: working PCIe and GENET (real networking)" [1]. That series got the PCIe Root Complex link up and GENET (a plain sysbus device, not PCI) working, but nothing in it actually required a guest to enumerate a device *on* the PCIe bus -- so a real gap went unnoticed until now: attach anything to raspi4b's PCIe port and "lspci"/"/sys/bus/pci/devices/" come back completely empty, even though the link itself is up.

Root causes, all on the same host-bridge config-space path in hw/arm/bcm2838_pcie.c:

1. bcm2838_pcie_host_read/write computed
"offset - PCIE_CONFIG_SPACE_SIZE" on an unsigned hwaddr, which underflows for every offset below 4KB -- i.e. the root port's own config space, including vendor/device ID at offset 0. Every such access fell into the "out of range" branch and read back as all-ones ("no device present"), so the guest's PCI core gave up immediately.

2. pcie_host_mmcfg_init() was never called, even though both accessors dereference the MemoryRegion it sets up to service EXT_CFG_DATA. Fixing (1) let the guest reach that path for the first time, which then dereferenced an uninitialised region.

3. The root port overrode PCIDeviceClass::config_read/config_write with plain pci_default_*_config(), bypassing pci_bridge_write_config(). Bridge memory windows were therefore never actually programmed, so BARs behind the root port stayed unreachable however the guest wrote them.

4. PCIE_MMIO_OFFSET (0xc0000000) was defined in hw/arm/bcm2838_peripherals.c for exactly this purpose and never used -- grep count of 1, the definition itself. The PCI MMIO window was mapped straight onto the ARM address with no translation, even though the DTB declares CPU 0x600000000 as mapping to PCI 0xc0000000. The guest ended up reading PCI address 0 where a device's BAR should have been.

Patch 1 fixes all four. I want to flag why they're one patch rather than four: I tried splitting them during development, and every subset short of all four either changes nothing (dormant bug) or produces a kernel panic that doesn't happen on unfixed master (wrong BAR content, alignment fault in a driver's MMIO access) -- worse than the current silent "no devices enumerated". None of the four are a coherent, independently-safe change on their own.

With all four, a PCIe device is enumerated and its driver binds:

xhci_hcd 0000:01:00.0: xHCI Host Controller
xhci_hcd 0000:01:00.0: new USB bus registered, assigned bus number 1 xhci_hcd 0000:01:00.0: Host supports USB 3.0 SuperSpeed

This is not a complete PCIe implementation. The BCM2711's own MSI controller is not implemented -- the guest driver programs it correctly (visible as BRCM STB PCIe MSI vectors in /proc/interrupts, permanently at 0 count) but QEMU never delivers anything, so a device that only signals via MSI/MSI-X never actually gets its interrupt. INTx works. Patch 1 documents this in docs/system/arm/raspi.rst; patch 2's functional test attaches its xHCI controller with "msi=off,msix=off" for the same reason. I plan to look at the MSI controller separately; wanted this enumeration fix to stand on its own rather than block on it, since it already turns "no PCI device is usable at all" into "PCI devices using INTx work".

Tested against real hardware, not just the emulated xHCI: with a USB serial device passed through to the guest via QEMU's usb-serial device pointed at the real /dev/cu.usbmodem* port, a real serial peripheral connected to my Mac was reachable from inside the guest, byte-for-byte identical to reading it directly on the host.

checkpatch.pl is clean (0 errors, 0 warnings) across both patches. All 4 tests in tests/functional/aarch64/test_raspi4.py pass, including the 3 pre-existing ones (no regression) and the new one added here.

[1] https://patchwork.kernel.org/project/qemu-devel/cover/[email protected]/

Marcelo Manzo (2):
hw/arm/bcm2838_pcie: make PCI device enumeration actually work tests/functional/aarch64: add raspi4b PCIe enumeration test

 docs/system/arm/raspi.rst               |  2 ++
 hw/arm/bcm2838_pcie.c                   | 45 +++++++++++++++----------
 hw/arm/bcm2838_peripherals.c            | 16 +++++++--
 include/hw/arm/bcm2838_peripherals.h    |  1 +
 tests/functional/aarch64/test_raspi4.py | 37 ++++++++++++++++++++
 5 files changed, 82 insertions(+), 19 deletions(-)
Signature · 1 line
2.47.1
#1 lore raw

PATCH 1/2 hw/arm/bcm2838_pcie: make PCI device enumeration actually work

raspi4b's PCIe root complex brought the link up but no device was ever
enumerated: "lspci" was empty and /sys/bus/pci/devices/ had no entries. Nothing exercised this before, since GENET is a sysbus device rather than a PCI one.

Four separate defects, all on the same path:

1. bcm2838_pcie_host_read/write computed
"offset - PCIE_CONFIG_SPACE_SIZE" on an unsigned hwaddr, which underflows for every offset below 4KB. The root port's own config space -- including vendor/device ID at offset 0 -- therefore always fell into the out-of-range branch and read back as all-ones, i.e. "no device present", so the guest stopped scanning immediately.

2. pcie_host_mmcfg_init() was never called, yet both accessors dereference pcie_hb->mmio.ops to service EXT_CFG_DATA. Once (1) was fixed and the guest actually reached that path, it touched an uninitialised MemoryRegion.

3. The root port overrode PCIDeviceClass::config_read/config_write with plain pci_default_*_config(). That bypasses rp_write_config() -> pci_bridge_write_config(), so programming the bridge's memory window never updated the bridge's address space and BARs behind the root port stayed unreachable. Drop the overrides and dispatch through pci_host_config_{read,write}_common() so the device's real handlers run.

4. The PCI MMIO window was mapped at the ARM base address with no address translation, even though PCIE_MMIO_OFFSET was defined for exactly that purpose (and otherwise unused). The DTB declares CPU 0x600000000 as mapping to PCI 0xc0000000, so the guest was reading PCI address 0 where it expected a device BAR. Map an alias at the correct PCI offset instead.

With these, a PCIe device is enumerated and its driver binds:

xhci_hcd 0000:01:00.0: xHCI Host Controller
xhci_hcd 0000:01:00.0: new USB bus registered, assigned bus number 1 xhci_hcd 0000:01:00.0: Host supports USB 3.0 SuperSpeed

Note this is not yet complete: devices attached behind the xHCI are still not enumerated by the guest, which looks like a separate interrupt-delivery problem rather than an enumeration one.

---
 docs/system/arm/raspi.rst            |  2 ++
 hw/arm/bcm2838_pcie.c                | 45 +++++++++++++++++-----------
 hw/arm/bcm2838_peripherals.c         | 16 ++++++++--
 include/hw/arm/bcm2838_peripherals.h |  1 +
 4 files changed, 45 insertions(+), 19 deletions(-)
docs/system/arm/raspi.rst +2 -0
index a4f83d0a..a8fc37e9 100644
@@ -43,3 +43,5 @@ Missing devices
 ---------------
 
  * Pulse Width Modulation (PWM)
+ * PCIe MSI/MSI-X interrupt delivery (raspi4b) -- INTx works; a PCIe
+   device that only signals via MSI/MSI-X needs ``msi=off,msix=off``
hw/arm/bcm2838_pcie.c +28 -17

84 lines not on this page · raw message

hw/arm/bcm2838_peripherals.c +14 -2
index de49f48b..012a4567 100644
@@ -204,10 +204,22 @@ static void bcm2838_peripherals_realize(DeviceState *dev, Error **errp)
     /* RC registers region */
     regs_mr = sysbus_mmio_get_region(SYS_BUS_DEVICE(&s->pcie_host), 0);
     memory_region_add_subregion(&s->peri_low_mr, PCIE_RC_OFFSET, regs_mr);
-    /* MMIO region */
+    /*
+     * MMIO region.
+     *
+     * The BCM2711 PCIe controller translates addresses between the ARM and
+     * PCI address spaces: the DTB declares CPU 0x600000000 as mapping to PCI
+     * 0xc0000000. Map an alias of the PCI window starting at that PCI offset
+     * so accesses land on the right addresses; mapping the window directly
+     * would expose PCI address 0 at the ARM base instead, and every BAR
+     * behind the root port would be read at the wrong address.
+     */
     mmio_mr = sysbus_mmio_get_region(SYS_BUS_DEVICE(&s->pcie_host), 1);
+    memory_region_init_alias(&s->pcie_mmio_alias, OBJECT(s),
+                             "bcm2838_pcie_mmio_alias", mmio_mr,
+                             PCIE_MMIO_OFFSET, PCIE_MMIO_SIZE);
     memory_region_add_subregion(get_system_memory(), PCIE_MMIO_ARM_OFFSET,
-                                mmio_mr);
+                                &s->pcie_mmio_alias);
 
     /* Gigabit Ethernet */
     if (!sysbus_realize(SYS_BUS_DEVICE(&s->genet), errp)) {
include/hw/arm/bcm2838_peripherals.h +1 -0
index 5da340f2..c05709f4 100644
@@ -70,6 +70,7 @@ struct BCM2838PeripheralState {
 
     SDHCIState emmc2;
     BCM2838PcieHostState pcie_host;
+    MemoryRegion pcie_mmio_alias;
     BCM2838GenetState genet;
     BCM2838GpioState gpio;
 
Signature · 1 line
2.47.1
#3 lore raw
In reply to Marcelo Manzo
Quoted 11 lines · Marcelo Manzo

Follow-up to "hw/arm/raspi4b: working PCIe and GENET (real networking)" [1]. That series got the PCIe Root Complex link up and GENET (a plain sysbus device, not PCI) working, but nothing in it actually required a guest to enumerate a device *on* the PCIe bus -- so a real gap went unnoticed until now: attach anything to raspi4b's PCIe port and "lspci"/"/sys/bus/pci/devices/" come back completely empty, even though the link itself is up.

Marcelo Manzo (2):
hw/arm/bcm2838_pcie: make PCI device enumeration actually work

Since bcm2838_pcie isn't in upstream yet, I think we should fold this fix into that patchset, rather than introducing it in a broken state that we then fix with a later patchset.

thanks
-- PMM