hw/arm/bcm2838_pcie: make PCI device enumeration work
Subsystems (2)
cover hw/arm/bcm2838_pcie: make PCI device enumeration work
Follow-up to "hw/arm/raspi4b: working PCIe and GENET (real networking)" [1]. That series got the PCIe Root Complex link up and GENET (a plain sysbus device, not PCI) working, but nothing in it actually required a guest to enumerate a device *on* the PCIe bus -- so a real gap went unnoticed until now: attach anything to raspi4b's PCIe port and "lspci"/"/sys/bus/pci/devices/" come back completely empty, even though the link itself is up.
Root causes, all on the same host-bridge config-space path in hw/arm/bcm2838_pcie.c:
1. bcm2838_pcie_host_read/write computed
"offset - PCIE_CONFIG_SPACE_SIZE" on an unsigned hwaddr, which underflows for every offset below 4KB -- i.e. the root port's own config space, including vendor/device ID at offset 0. Every such access fell into the "out of range" branch and read back as all-ones ("no device present"), so the guest's PCI core gave up immediately.
2. pcie_host_mmcfg_init() was never called, even though both accessors dereference the MemoryRegion it sets up to service EXT_CFG_DATA. Fixing (1) let the guest reach that path for the first time, which then dereferenced an uninitialised region.
3. The root port overrode PCIDeviceClass::config_read/config_write with plain pci_default_*_config(), bypassing pci_bridge_write_config(). Bridge memory windows were therefore never actually programmed, so BARs behind the root port stayed unreachable however the guest wrote them.
4. PCIE_MMIO_OFFSET (0xc0000000) was defined in hw/arm/bcm2838_peripherals.c for exactly this purpose and never used -- grep count of 1, the definition itself. The PCI MMIO window was mapped straight onto the ARM address with no translation, even though the DTB declares CPU 0x600000000 as mapping to PCI 0xc0000000. The guest ended up reading PCI address 0 where a device's BAR should have been.
Patch 1 fixes all four. I want to flag why they're one patch rather than four: I tried splitting them during development, and every subset short of all four either changes nothing (dormant bug) or produces a kernel panic that doesn't happen on unfixed master (wrong BAR content, alignment fault in a driver's MMIO access) -- worse than the current silent "no devices enumerated". None of the four are a coherent, independently-safe change on their own.
With all four, a PCIe device is enumerated and its driver binds:
xhci_hcd 0000:01:00.0: xHCI Host Controller
xhci_hcd 0000:01:00.0: new USB bus registered, assigned bus number 1 xhci_hcd 0000:01:00.0: Host supports USB 3.0 SuperSpeed
This is not a complete PCIe implementation. The BCM2711's own MSI controller is not implemented -- the guest driver programs it correctly (visible as BRCM STB PCIe MSI vectors in /proc/interrupts, permanently at 0 count) but QEMU never delivers anything, so a device that only signals via MSI/MSI-X never actually gets its interrupt. INTx works. Patch 1 documents this in docs/system/arm/raspi.rst; patch 2's functional test attaches its xHCI controller with "msi=off,msix=off" for the same reason. I plan to look at the MSI controller separately; wanted this enumeration fix to stand on its own rather than block on it, since it already turns "no PCI device is usable at all" into "PCI devices using INTx work".
Tested against real hardware, not just the emulated xHCI: with a USB serial device passed through to the guest via QEMU's usb-serial device pointed at the real /dev/cu.usbmodem* port, a real serial peripheral connected to my Mac was reachable from inside the guest, byte-for-byte identical to reading it directly on the host.
checkpatch.pl is clean (0 errors, 0 warnings) across both patches. All 4 tests in tests/functional/aarch64/test_raspi4.py pass, including the 3 pre-existing ones (no regression) and the new one added here.
[1] https://patchwork.kernel.org/project/qemu-devel/cover/[email protected]/
Marcelo Manzo (2):
hw/arm/bcm2838_pcie: make PCI device enumeration actually work tests/functional/aarch64: add raspi4b PCIe enumeration test
docs/system/arm/raspi.rst | 2 ++ hw/arm/bcm2838_pcie.c | 45 +++++++++++++++---------- hw/arm/bcm2838_peripherals.c | 16 +++++++-- include/hw/arm/bcm2838_peripherals.h | 1 + tests/functional/aarch64/test_raspi4.py | 37 ++++++++++++++++++++ 5 files changed, 82 insertions(+), 19 deletions(-)
Signature · 1 line
2.47.1
PATCH 1/2 hw/arm/bcm2838_pcie: make PCI device enumeration actually work
raspi4b's PCIe root complex brought the link up but no device was ever
enumerated: "lspci" was empty and /sys/bus/pci/devices/ had no entries. Nothing exercised this before, since GENET is a sysbus device rather than a PCI one.
Four separate defects, all on the same path:
1. bcm2838_pcie_host_read/write computed
"offset - PCIE_CONFIG_SPACE_SIZE" on an unsigned hwaddr, which underflows for every offset below 4KB. The root port's own config space -- including vendor/device ID at offset 0 -- therefore always fell into the out-of-range branch and read back as all-ones, i.e. "no device present", so the guest stopped scanning immediately.
2. pcie_host_mmcfg_init() was never called, yet both accessors dereference pcie_hb->mmio.ops to service EXT_CFG_DATA. Once (1) was fixed and the guest actually reached that path, it touched an uninitialised MemoryRegion.
3. The root port overrode PCIDeviceClass::config_read/config_write with plain pci_default_*_config(). That bypasses rp_write_config() -> pci_bridge_write_config(), so programming the bridge's memory window never updated the bridge's address space and BARs behind the root port stayed unreachable. Drop the overrides and dispatch through pci_host_config_{read,write}_common() so the device's real handlers run.
4. The PCI MMIO window was mapped at the ARM base address with no address translation, even though PCIE_MMIO_OFFSET was defined for exactly that purpose (and otherwise unused). The DTB declares CPU 0x600000000 as mapping to PCI 0xc0000000, so the guest was reading PCI address 0 where it expected a device BAR. Map an alias at the correct PCI offset instead.
With these, a PCIe device is enumerated and its driver binds:
xhci_hcd 0000:01:00.0: xHCI Host Controller
xhci_hcd 0000:01:00.0: new USB bus registered, assigned bus number 1 xhci_hcd 0000:01:00.0: Host supports USB 3.0 SuperSpeed
Note this is not yet complete: devices attached behind the xHCI are still not enumerated by the guest, which looks like a separate interrupt-delivery problem rather than an enumeration one.
--- docs/system/arm/raspi.rst | 2 ++ hw/arm/bcm2838_pcie.c | 45 +++++++++++++++++----------- hw/arm/bcm2838_peripherals.c | 16 ++++++++-- include/hw/arm/bcm2838_peripherals.h | 1 + 4 files changed, 45 insertions(+), 19 deletions(-)
docs/system/arm/raspi.rst +2 -0
@@ -43,3 +43,5 @@ Missing devices --------------- * Pulse Width Modulation (PWM) + * PCIe MSI/MSI-X interrupt delivery (raspi4b) -- INTx works; a PCIe + device that only signals via MSI/MSI-X needs ``msi=off,msix=off``
hw/arm/bcm2838_pcie.c +28 -17
84 lines not on this page · raw message
hw/arm/bcm2838_peripherals.c +14 -2
@@ -204,10 +204,22 @@ static void bcm2838_peripherals_realize(DeviceState *dev, Error **errp)/* RC registers region */ regs_mr =sysbus_mmio_get_region (SYS_BUS_DEVICE (&s->pcie_host),0 );memory_region_add_subregion (&s->peri_low_mr,PCIE_RC_OFFSET , regs_mr); -/* MMIO region */ +/* +* MMIO region. +* +* The BCM2711 PCIe controller translates addresses between the ARM and +* PCI address spaces: the DTB declares CPU 0x600000000 as mapping to PCI +* 0xc0000000. Map an alias of the PCI window starting at that PCI offset +* so accesses land on the right addresses; mapping the window directly +* would expose PCI address 0 at the ARM base instead, and every BAR +* behind the root port would be read at the wrong address. +*/ mmio_mr =sysbus_mmio_get_region (SYS_BUS_DEVICE (&s->pcie_host),1 ); +memory_region_init_alias (&s->pcie_mmio_alias,OBJECT (s), +"bcm2838_pcie_mmio_alias" , mmio_mr, +PCIE_MMIO_OFFSET ,PCIE_MMIO_SIZE );memory_region_add_subregion (get_system_memory (),PCIE_MMIO_ARM_OFFSET , - mmio_mr); + &s->pcie_mmio_alias);/* Gigabit Ethernet */ if (!sysbus_realize (SYS_BUS_DEVICE (&s->genet), errp)) {
include/hw/arm/bcm2838_peripherals.h +1 -0
@@ -70,6 +70,7 @@ struct BCM2838PeripheralState {SDHCIState emmc2;BCM2838PcieHostState pcie_host; +MemoryRegion pcie_mmio_alias;BCM2838GenetState genet;BCM2838GpioState gpio;
Signature · 1 line
2.47.1
PATCH 2/2 tests/functional/aarch64: add raspi4b PCIe enumeration test
In reply to Marcelo ManzoAdd test_arm_raspi4_pcie, exercising the fix in the previous patch:
attach a qemu-xhci controller to the PCIe root port, confirm the guest's driver binds it, and confirm both the root port (00:00.0) and the xHCI controller (01:00.0) show up in /sys/bus/pci/devices/. Before the previous patch this would find no devices at all.
Attaches with msi=off,msix=off: the BCM2838 PCIe root complex's own MSI controller is not implemented, so a device relying on MSI/MSI-X for interrupts never signals its driver. INTx works and is what this test (and any real device behind this root port, for now) relies on.
--- tests/functional/aarch64/test_raspi4.py | 37 +++++++++++++++++++++++++ 1 file changed, 37 insertions(+)
tests/functional/aarch64/test_raspi4.py +37 -0
@@ -122,5 +122,42 @@ def test_arm_raspi4_genet(self): exec_command_and_wait_for_pattern(self, 'halt', 'reboot: System halted') + def test_arm_raspi4_pcie(self): + kernel_path = self.archive_extract(self.ASSET_KERNEL_20190215, + member='boot/kernel8.img') + dtb_path = self.archive_extract(self.ASSET_KERNEL_20190215, + member='boot/bcm2711-rpi-4-b.dtb') + initrd_path = self.uncompress(self.ASSET_INITRD) + + self.set_machine('raspi4b') + self.vm.set_console() + kernel_command_line = (self.KERNEL_COMMON_COMMAND_LINE + + 'earlycon=pl011,mmio32,0xfe201000 ' + + 'console=ttyAMA0,115200 ' + + 'panic=-1 noreboot ' + + 'dwc_otg.fiq_fsm_enable=0') + self.vm.add_args('-kernel', kernel_path, + '-dtb', dtb_path, + '-initrd', initrd_path, + '-append', kernel_command_line, + '-no-reboot', + # msi=off,msix=off: the BCM2838 PCIe root complex's + # MSI controller is not implemented, so a device + # relying on MSI/MSI-X for interrupts never signals + # and its driver never probes. INTx works. + '-device', + 'qemu-xhci,bus=pcie.1,id=xhci,msi=off,msix=off') + self.vm.launch() + self.wait_for_console_pattern( + 'xhci_hcd 0000:01:00.0: xHCI Host Controller') + self.wait_for_console_pattern('Boot successful.') + + exec_command_and_wait_for_pattern(self, 'ls /sys/bus/pci/devices/', + '0000:00:00.0') + exec_command_and_wait_for_pattern(self, 'ls /sys/bus/pci/devices/', + '0000:01:00.0') + exec_command_and_wait_for_pattern(self, 'halt', 'reboot: System halted') + + if __name__ == '__main__': LinuxKernelTest.main()
Signature · 1 line
2.47.1
Quoted 11 lines · Marcelo Manzo
Follow-up to "hw/arm/raspi4b: working PCIe and GENET (real networking)" [1]. That series got the PCIe Root Complex link up and GENET (a plain sysbus device, not PCI) working, but nothing in it actually required a guest to enumerate a device *on* the PCIe bus -- so a real gap went unnoticed until now: attach anything to raspi4b's PCIe port and "lspci"/"/sys/bus/pci/devices/" come back completely empty, even though the link itself is up.
Marcelo Manzo (2):
hw/arm/bcm2838_pcie: make PCI device enumeration actually work
Since bcm2838_pcie isn't in upstream yet, I think we should fold this fix into that patchset, rather than introducing it in a broken state that we then fix with a later patchset.
thanks
-- PMM