Bad ROM contents on pass-through card
I started trying to pass through a PCI-E card to a KVM guest early this year. Other hardware can be passed through successfully, but the guest sees the wrong option ROM for this card.
My system is an AMD CPU and GA-970A-D3 motherboard. There are currently two Radeon HD 3470 cards installed, partially so that I could read their ROMs into a file.
The host OS is CentOS 6.3, and I've updated seabios and qemu to current versions.
My guest is started with:
/usr/local/bin/qemu-system-x86_64 ... -vga cirrus \
-device pci-assign,host=05:00.0,id=hostdev1,configfd=27,\
bus=pci.0,addr=0x5,romfile=/var/lib/libvirt/images/Radeon3470-5.rom
However, when the guest tries to read the ROM for the ATI card, it's seeing the ROM for the emulated cirrus card instead:
# strings /sys/devices/pci0000:00/0000:00:05.0/rom | head -4 Plex86/Bochs VGABios (PCI) current-cvs 23 Sep 2012 (C) 2008 the LGPL VGABios developers Team
If I don't pass an emulated VGA device (and modify seabios to not run the VGA ROM), the guess won't read anything from the rom at all.
Can anyone point me at which parts of the code I might review to see where things are getting mapped and what's going wrong?
On the host system, lspci describes the device:
05:00.0 VGA compatible controller: Advanced Micro Devices [AMD] nee ATI RV620 PRO [Radeon HD 3470] (prog-if 00 [VGA controller])
Subsystem: Dell Device 3243
Flags: fast devsel, IRQ 26
Memory at c0000000 (64-bit, prefetchable) [size=256M]
Memory at fd9f0000 (64-bit, non-prefetchable) [size=64K]
I/O ports at ae00 [size=256]
Expansion ROM at fd900000 [disabled] [size=128K]
Capabilities: [50] Power Management version 3
Capabilities: [58] Express Legacy Endpoint, MSI 00
Capabilities: [a0] MSI: Enable+ Count=1/1 Maskable- 64bit+
Capabilities: [100] Vendor Specific Information <?>
Kernel driver in use: pci-stub
Kernel modules: radeon
and in the guest (Fedora 17):
00:05.0 VGA compatible controller: ATI Technologies Inc Mobility Radeon HD 3470 (prog-if 00 [VGA controller])
Subsystem: Dell Device 3243
Physical Slot: 5
Flags: fast devsel, IRQ 11
Memory at e0000000 (32-bit, non-prefetchable) [size=256M]
Memory at f2030000 (32-bit, non-prefetchable) [size=64K]
I/O ports at c100 [size=256]
Expansion ROM at f2040000 [disabled] [size=64K]
Capabilities: [58] Express Legacy Endpoint, MSI 00
Capabilities: [50] Power Management version 3
Capabilities: [a0] MSI: Enable- Count=1/1 Maskable- 64bit-
Quoted 25 lines · Gordon Messmer
I started trying to pass through a PCI-E card to a KVM guest early this year. Other hardware can be passed through successfully, but the guest sees the wrong option ROM for this card.
My system is an AMD CPU and GA-970A-D3 motherboard. There are currently two Radeon HD 3470 cards installed, partially so that I could read their ROMs into a file.
The host OS is CentOS 6.3, and I've updated seabios and qemu to current versions.
My guest is started with:
/usr/local/bin/qemu-system-x86_64... -vga cirrus \
-device pci-assign,host=05:00.0,id=hostdev1,configfd=27,\
bus=pci.0,addr=0x5,romfile=/var/lib/libvirt/images/Radeon3470-5.romHowever, when the guest tries to read the ROM for the ATI card, it's seeing the ROM for the emulated cirrus card instead:
# strings /sys/devices/pci0000:00/0000:00:05.0/rom | head -4 Plex86/Bochs VGABios (PCI) current-cvs 23 Sep 2012 (C) 2008 the LGPL VGABios developers Team
Quoted 5 lines
If I don't pass an emulated VGA device (and modify seabios to not run the VGA ROM), the guess won't read anything from the rom at all.
Can anyone point me at which parts of the code I might review to see where things are getting mapped and what's going wrong?
The comments in linux:/drivers/pci/rom.c:pci_map_rom are probably relevant. Linux decides based on the VGA enable bit whether to return the PCI option ROM or legacy VGA ROM. In your case, it's giving you the legacy ROM from the cirrus device. The next step is probably looking at generating a PCI topology in the guest similar to a physical system when multiple VGA devices are present. That means bridges and/or root ports with VGA routing control.
Quoted 16 lines
On the host system, lspci describes the device:
05:00.0 VGA compatible controller: Advanced Micro Devices [AMD] nee ATI RV620 PRO [Radeon HD 3470] (prog-if 00 [VGA controller])
Subsystem: Dell Device 3243 Flags: fast devsel, IRQ 26 Memory at c0000000 (64-bit, prefetchable) [size=256M] Memory at fd9f0000 (64-bit, non-prefetchable) [size=64K] I/O ports at ae00 [size=256] Expansion ROM at fd900000 [disabled] [size=128K]and in the guest (Fedora 17):
00:05.0 VGA compatible controller: ATI Technologies Inc Mobility Radeon HD 3470 (prog-if 00 [VGA controller])
Quoted 7 lines
Subsystem: Dell Device 3243 Physical Slot: 5 Flags: fast devsel, IRQ 11 Memory at e0000000 (32-bit, non-prefetchable) [size=256M] Memory at f2030000 (32-bit, non-prefetchable) [size=64K] I/O ports at c100 [size=256] Expansion ROM at f2040000 [disabled] [size=64K]
What's the size of Radeon3470-5.rom? When using romfile the size of the option ROM BAR is made to be big enough to contain the ROM. Without that, we match the physical device and expose the BAR as potentially being bigger than the ROM it contains. Thanks,
Alex
Quoted 5 lines · Alex Williamson
Quoted 4 lines · Gordon Messmer
If I don't pass an emulated VGA device (and modify seabios to not run the VGA ROM), the guess won't read anything from the rom at all.
Quoted 7 lines
Quoted 2 lines
Can anyone point me at which parts of the code I might review to see where things are getting mapped and what's going wrong?
The comments in linux:/drivers/pci/rom.c:pci_map_rom are probably relevant. Linux decides based on the VGA enable bit whether to return the PCI option ROM or legacy VGA ROM. In your case, it's giving you the legacy ROM from the cirrus device.
Interesting. There are two things that seem odd about that. First, the Linux guest doesn't seem to read the correct ROM either. During boot, dmesg prints:
[ 1.047559] [drm] radeon defaulting to kernel modesetting. [ 1.047561] [drm] radeon kernel modesetting enabled. [ 1.047595] checking generic (f0000000 160000) vs hw (e0000000 10000000) [ 1.047832] ACPI: PCI Interrupt Link [LNKA] enabled at IRQ 11 [ 1.048270] [drm] initializing kernel modesetting (RV620 0x1002:0x95C0 0x1028:0x3243). [ 1.048288] [drm] register mmio base: 0xF2030000 [ 1.048289] [drm] register mmio size: 65536 [ 1.048395] radeon 0000:00:05.0: Expecting atombios for R600 GPU [ 1.048924] radeon 0000:00:05.0: Fatal error during GPU init [ 1.048925] [drm] radeon: finishing device. [ 1.048927] [TTM] Memory type 2 has not been initialized
Second, if I don't provide a -vga argument, the guest will hang during boot. If I modify seabios not to run the VGA ROM, the guest boots but the radeon driver output is the same and I can't read any ROM at all from /sys
Quoted 4 lines
The next step is probably looking at
generating a PCI topology in the guest similar to a physical system when multiple VGA devices are present. That means bridges and/or root ports with VGA routing control.
Can you point me toward the documentation? I don't see anything that looks relevant in the man page.
Quoted 4 lines
What's the size of Radeon3470-5.rom? When using romfile the size of the option ROM BAR is made to be big enough to contain the ROM. Without that, we match the physical device and expose the BAR as potentially being bigger than the ROM it contains. Thanks,
It's 64512 bytes.
Thanks, Alex. I really appreciate your time.