poor performance of passed-through VGA
Hello,
I am one of those early tinkerers with VGA passthrough. This is very exciting feature and as soon as i saw it i thought to myself that i want that "95% speed of bare metal" as various sources claimed. So i invested into some harware only to find out that actual 3d performance is far lower. I am not sure why it is the case so maybe someone could either clarify if it is known defect or something to be expected. Or if this is actually a bug and someone someone cared enough to tackle it i would gladly be a test-monkey.
Hardware:
Motherboard: SABERTOOTH 990FX R2.0 (bios v2104)
CPU: AMD FX(tm)-8350
GPU1 (host): GeForce GTX 550 Ti
GPU2 (guest): Radeon R9 270X
QEMU 1.7.90 (from git)
qemu command line:
Quoted 4 lines
/usr/local/bin/qemu-system-x86_64\(quoted diff) +0
-3-enable-kvm -m 4096 -cpu qemu64 -machine q35,accel=kvm \ -smp 8,sockets=1,cores=8,threads=1 \ -drive file=/dev/sdc,if=none,id=virtio-disk0,format=raw,cache=none,aio=native \
Review
Quoted 10 lines
(quoted diff) +0
-10-device virtio-blk-pci,scsi=off,drive=virtio-disk0,id=disk0 \ -drive file=/dev/sdd,if=none,id=virtio-disk1,format=raw,cache=none,aio=native \ -device virtio-blk-pci,scsi=off,drive=virtio-disk1,id=disk1 \ -device e1000,netdev=vnet0,mac=40:01:23:ff:9a:00 -netdev tap,id=vnet0 \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=06:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=00:13.0,bus=pcie.0 \ -device vfio-pci,host=00:13.2,bus=pcie.0 \ -bios /home/novist/opt/src/seabios/out/bios.bin \ -vga none
Kernel: 3.14 with kvm patches for 3.15 (i was hoping they would make a difference but sadly they dont). Additional kvm options: iommu=1 vfio_iommu_type1.allow_unsafe_interrupts=1 kvm.ignore_msrs=1
Quoted 4 lines
00:13.0 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB OHCI0 Controller 00:13.2 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB EHCI Controller 06:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Curacao XT [Radeon R9 270X] 06:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Cape Verde/Pitcairn HDMI Audio [Radeon HD 7700/7800 Series]
Actual problem is that GPU seems to be underused. For example in passmark benchmark i score only 15 FPS in directx9 complex test while on bare metal its 100 FPS. That is quite a gap. With kernels prior 3.14 it was even worse - i could get only 10 FPS.
Another rather naive test i performed was checking gpu load with process hacker while playing actual recent game (arma3 in this case). GPU utilization sits at about 20-30%, rarely 40% spikes. And it is lagy of course. Low settings make it somewhat playable. On bare metal though very high settings run smooth. Its worth noting that directx10/11 bencmarks of passmark perform much better than directx9 ones.
Here are benchmark results and GPU load graphs:
GPU load: http://imgur.com/M70CnIV
Benchmark: http://imgur.com/IeYh4Zc
As you see while directx11 performs at about acceptable rate (considering others preach glorious 95% of bare metal performance) directx10 test is quite slower and directx9 performs terribly.
Not sure what other information i could provide. If there is any - i would be happy to look it up. So any idea where this mythical "95% of bare metal performance" can be found? Cant wait for some insight from a professional.
P.S. i am also adding 2d benchmark results if anyone cares. GPU load: http://imgur.com/28x5DN8
Benchmark: http://imgur.com/562PJYl
Quoted 57 lines · rndbit
Hello,
I am one of those early tinkerers with VGA passthrough. This is very exciting feature and as soon as i saw it i thought to myself that i want that "95% speed of bare metal" as various sources claimed. So i invested into some harware only to find out that actual 3d performance is far lower. I am not sure why it is the case so maybe someone could either clarify if it is known defect or something to be expected. Or if this is actually a bug and someone someone cared enough to tackle it i would gladly be a test-monkey.Hardware:
Motherboard: SABERTOOTH 990FX R2.0 (bios v2104)
CPU: AMD FX(tm)-8350
GPU1 (host): GeForce GTX 550 Ti
GPU2 (guest): Radeon R9 270X
QEMU 1.7.90 (from git)qemu command line:
Quoted 4 lines
/usr/local/bin/qemu-system-x86_64\(quoted diff) +0
-3-enable-kvm -m 4096 -cpu qemu64 -machine q35,accel=kvm \ -smp 8,sockets=1,cores=8,threads=1 \ -drive file=/dev/sdc,if=none,id=virtio-disk0,format=raw,cache=none,aio=native \Review
Quoted 10 lines
(quoted diff) +0
-10-device virtio-blk-pci,scsi=off,drive=virtio-disk0,id=disk0 \ -drive file=/dev/sdd,if=none,id=virtio-disk1,format=raw,cache=none,aio=native \ -device virtio-blk-pci,scsi=off,drive=virtio-disk1,id=disk1 \ -device e1000,netdev=vnet0,mac=40:01:23:ff:9a:00 -netdev tap,id=vnet0 \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=06:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=00:13.0,bus=pcie.0 \ -device vfio-pci,host=00:13.2,bus=pcie.0 \ -bios /home/novist/opt/src/seabios/out/bios.bin \ -vga noneKernel: 3.14 with kvm patches for 3.15 (i was hoping they would make a difference but sadly they dont). Additional kvm options: iommu=1 vfio_iommu_type1.allow_unsafe_interrupts=1 kvm.ignore_msrs=1
Quoted 4 lines
00:13.0 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB OHCI0 Controller 00:13.2 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB EHCI Controller 06:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Curacao XT [Radeon R9 270X] 06:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Cape Verde/Pitcairn HDMI Audio [Radeon HD 7700/7800 Series]
Actual problem is that GPU seems to be underused. For example in passmark benchmark i score only 15 FPS in directx9 complex test while on bare metal its 100 FPS. That is quite a gap. With kernels prior 3.14 it was even worse - i could get only 10 FPS.
Another rather naive test i performed was checking gpu load with process hacker while playing actual recent game (arma3 in this case). GPU utilization sits at about 20-30%, rarely 40% spikes. And it is lagy of course. Low settings make it somewhat playable. On bare metal though very high settings run smooth. Its worth noting that directx10/11 bencmarks of passmark perform much better than directx9 ones.
Here are benchmark results and GPU load graphs:
GPU load: http://imgur.com/M70CnIV
Benchmark: http://imgur.com/IeYh4ZcAs you see while directx11 performs at about acceptable rate (considering others preach glorious 95% of bare metal performance) directx10 test is quite slower and directx9 performs terribly.
Quoted 7 lines
Not sure what other information i could provide. If there is any - i would be happy to look it up. So any idea where this mythical "95% of bare metal performance" can be found? Cant wait for some insight from a professional.
P.S. i am also adding 2d benchmark results if anyone cares. GPU load: http://imgur.com/28x5DN8
Benchmark: http://imgur.com/562PJYl
Have you tried assigning the Nvidia GPU? If you doubt that good performance is possible, see these images of the same tests:
This is with an Nvidia Quadro K4000 assigned as a secondary device, without VGA quirks on an Intel host. It's entirely possible that VGA quirks are getting in the way in your test and may still with the GTX550Ti using VGA quirks. I guarantee that if the quirks are hit in any sort of hot path, performance will plummet.
It's also important to tune the setup, the above tests use hv-time on the vCPU and a kernel that supports it, as well as large pages for the guest and vCPU pinning.
There's potentially still work to do, the above tests show some significant hits in the 2D test, but 3D is quite good.
If you'd like to help improve the situation, you can enable debug when compiling qemu vfio, uncomment hw/misc/vfio.c:#define DEBUG_VFIO. This will produce a lot of output, especially on guest boot, but it should settle down as the drivers get loaded and direct access is used. If it doesn't settle down and it's through the quirks, we may never be able to achieve good performance since we don't own the driver. If it does settle down, you can use tools like perf and trace to analyze where time is being spent. Thanks,
Alex
Quoted 93 lines · Alex Williamson
Quoted 57 lines · rndbit
Hello,
I am one of those early tinkerers with VGA passthrough. This is very exciting feature and as soon as i saw it i thought to myself that i want that "95% speed of bare metal" as various sources claimed. So i invested into some harware only to find out that actual 3d performance is far lower. I am not sure why it is the case so maybe someone could either clarify if it is known defect or something to be expected. Or if this is actually a bug and someone someone cared enough to tackle it i would gladly be a test-monkey.Hardware:
Motherboard: SABERTOOTH 990FX R2.0 (bios v2104)
CPU: AMD FX(tm)-8350
GPU1 (host): GeForce GTX 550 Ti
GPU2 (guest): Radeon R9 270X
QEMU 1.7.90 (from git)qemu command line:
Quoted 4 lines
/usr/local/bin/qemu-system-x86_64\(quoted diff) +0
-3-enable-kvm -m 4096 -cpu qemu64 -machine q35,accel=kvm \ -smp 8,sockets=1,cores=8,threads=1 \ -drive file=/dev/sdc,if=none,id=virtio-disk0,format=raw,cache=none,aio=native \Review
Quoted 10 lines
(quoted diff) +0
-10-device virtio-blk-pci,scsi=off,drive=virtio-disk0,id=disk0 \ -drive file=/dev/sdd,if=none,id=virtio-disk1,format=raw,cache=none,aio=native \ -device virtio-blk-pci,scsi=off,drive=virtio-disk1,id=disk1 \ -device e1000,netdev=vnet0,mac=40:01:23:ff:9a:00 -netdev tap,id=vnet0 \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=06:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=00:13.0,bus=pcie.0 \ -device vfio-pci,host=00:13.2,bus=pcie.0 \ -bios /home/novist/opt/src/seabios/out/bios.bin \ -vga noneKernel: 3.14 with kvm patches for 3.15 (i was hoping they would make a difference but sadly they dont). Additional kvm options: iommu=1 vfio_iommu_type1.allow_unsafe_interrupts=1 kvm.ignore_msrs=1
Quoted 4 lines
00:13.0 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB OHCI0 Controller 00:13.2 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB EHCI Controller 06:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Curacao XT [Radeon R9 270X] 06:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Cape Verde/Pitcairn HDMI Audio [Radeon HD 7700/7800 Series]
Actual problem is that GPU seems to be underused. For example in passmark benchmark i score only 15 FPS in directx9 complex test while on bare metal its 100 FPS. That is quite a gap. With kernels prior 3.14 it was even worse - i could get only 10 FPS.
Another rather naive test i performed was checking gpu load with process hacker while playing actual recent game (arma3 in this case). GPU utilization sits at about 20-30%, rarely 40% spikes. And it is lagy of course. Low settings make it somewhat playable. On bare metal though very high settings run smooth. Its worth noting that directx10/11 bencmarks of passmark perform much better than directx9 ones.
Here are benchmark results and GPU load graphs:
GPU load: http://imgur.com/M70CnIV
Benchmark: http://imgur.com/IeYh4ZcAs you see while directx11 performs at about acceptable rate (considering others preach glorious 95% of bare metal performance) directx10 test is quite slower and directx9 performs terribly.
Quoted 7 lines
Not sure what other information i could provide. If there is any - i would be happy to look it up. So any idea where this mythical "95% of bare metal performance" can be found? Cant wait for some insight from a professional.
P.S. i am also adding 2d benchmark results if anyone cares. GPU load: http://imgur.com/28x5DN8
Benchmark: http://imgur.com/562PJYlHave you tried assigning the Nvidia GPU? If you doubt that good performance is possible, see these images of the same tests:
This is with an Nvidia Quadro K4000 assigned as a secondary device, without VGA quirks on an Intel host. It's entirely possible that VGA quirks are getting in the way in your test and may still with the GTX550Ti using VGA quirks. I guarantee that if the quirks are hit in any sort of hot path, performance will plummet.
It's also important to tune the setup, the above tests use hv-time on the vCPU and a kernel that supports it, as well as large pages for the guest and vCPU pinning.
There's potentially still work to do, the above tests show some significant hits in the 2D test, but 3D is quite good.
If you'd like to help improve the situation, you can enable debug when compiling qemu vfio, uncomment hw/misc/vfio.c:#define
DEBUG_VFIO. This will produce a lot of output, especially on guest boot, but it should settle down as the drivers get loaded and direct access is used. If it doesn't settle down and it's through the quirks, we may never be able to achieve good performance since we don't own the driver. If it does settle down, you can use tools like perf and trace to analyze where time is being spent. Thanks,
Quoted 3 lines
Alex
I have not tried passing-through nvidia gpu as i have read some scary tales of nvidia gpus being problematic. Besides now it would be a little bit pointless (unless its only for test) as i got more powerful GPU for use in VM. I will definitely try figuring out where time is spent. Some of your mentioned stuff is really all new to me and might take bit of time to figure out.
Im not 100% sure what vga quirks are so please forgive for possibly stupid question but are they enabled by x-vga=on? Something at the back of my head is telling me that you are probably not using that flag since your card is passed-through as >secondary< device. I also tried to achieve same thing with no luck. What i tried was basically -vga std and removing x-vga=on bit. Result was device manager indicating non-working device. I believe error was 12 (not enough resources) or something very similar. If you have ideas regarding this bit i would love to hear them. In the meantime ill snoop around stuff you mentioned, thank you for hints.
Quoted 98 lines · rndbit
Quoted 93 lines · Alex Williamson
Quoted 57 lines · rndbit
Hello,
I am one of those early tinkerers with VGA passthrough. This is very exciting feature and as soon as i saw it i thought to myself that i want that "95% speed of bare metal" as various sources claimed. So i invested into some harware only to find out that actual 3d performance is far lower. I am not sure why it is the case so maybe someone could either clarify if it is known defect or something to be expected. Or if this is actually a bug and someone someone cared enough to tackle it i would gladly be a test-monkey.Hardware:
Motherboard: SABERTOOTH 990FX R2.0 (bios v2104)
CPU: AMD FX(tm)-8350
GPU1 (host): GeForce GTX 550 Ti
GPU2 (guest): Radeon R9 270X
QEMU 1.7.90 (from git)qemu command line:
Quoted 4 lines
/usr/local/bin/qemu-system-x86_64\(quoted diff) +0
-3-enable-kvm -m 4096 -cpu qemu64 -machine q35,accel=kvm \ -smp 8,sockets=1,cores=8,threads=1 \ -drive file=/dev/sdc,if=none,id=virtio-disk0,format=raw,cache=none,aio=native \Review
Quoted 10 lines
(quoted diff) +0
-10-device virtio-blk-pci,scsi=off,drive=virtio-disk0,id=disk0 \ -drive file=/dev/sdd,if=none,id=virtio-disk1,format=raw,cache=none,aio=native \ -device virtio-blk-pci,scsi=off,drive=virtio-disk1,id=disk1 \ -device e1000,netdev=vnet0,mac=40:01:23:ff:9a:00 -netdev tap,id=vnet0 \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=06:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=00:13.0,bus=pcie.0 \ -device vfio-pci,host=00:13.2,bus=pcie.0 \ -bios /home/novist/opt/src/seabios/out/bios.bin \ -vga noneKernel: 3.14 with kvm patches for 3.15 (i was hoping they would make a difference but sadly they dont). Additional kvm options: iommu=1 vfio_iommu_type1.allow_unsafe_interrupts=1 kvm.ignore_msrs=1
Quoted 4 lines
00:13.0 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB OHCI0 Controller 00:13.2 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB EHCI Controller 06:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Curacao XT [Radeon R9 270X] 06:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Cape Verde/Pitcairn HDMI Audio [Radeon HD 7700/7800 Series]
Actual problem is that GPU seems to be underused. For example in passmark benchmark i score only 15 FPS in directx9 complex test while on bare metal its 100 FPS. That is quite a gap. With kernels prior 3.14 it was even worse - i could get only 10 FPS.
Another rather naive test i performed was checking gpu load with process hacker while playing actual recent game (arma3 in this case). GPU utilization sits at about 20-30%, rarely 40% spikes. And it is lagy of course. Low settings make it somewhat playable. On bare metal though very high settings run smooth. Its worth noting that directx10/11 bencmarks of passmark perform much better than directx9 ones.
Here are benchmark results and GPU load graphs:
GPU load: http://imgur.com/M70CnIV
Benchmark: http://imgur.com/IeYh4ZcAs you see while directx11 performs at about acceptable rate (considering others preach glorious 95% of bare metal performance) directx10 test is quite slower and directx9 performs terribly.
Quoted 7 lines
Not sure what other information i could provide. If there is any - i would be happy to look it up. So any idea where this mythical "95% of bare metal performance" can be found? Cant wait for some insight from a professional.
P.S. i am also adding 2d benchmark results if anyone cares. GPU load: http://imgur.com/28x5DN8
Benchmark: http://imgur.com/562PJYlHave you tried assigning the Nvidia GPU? If you doubt that good performance is possible, see these images of the same tests:
This is with an Nvidia Quadro K4000 assigned as a secondary device, without VGA quirks on an Intel host. It's entirely possible that VGA quirks are getting in the way in your test and may still with the GTX550Ti using VGA quirks. I guarantee that if the quirks are hit in any sort of hot path, performance will plummet.
It's also important to tune the setup, the above tests use hv-time on the vCPU and a kernel that supports it, as well as large pages for the guest and vCPU pinning.
There's potentially still work to do, the above tests show some significant hits in the 2D test, but 3D is quite good.
If you'd like to help improve the situation, you can enable debug when compiling qemu vfio, uncomment hw/misc/vfio.c:#define
DEBUG_VFIO. This will produce a lot of output, especially on guest boot, but it should settle down as the drivers get loaded and direct access is used. If it doesn't settle down and it's through the quirks, we may never be able to achieve good performance since we don't own the driver. If it does settle down, you can use tools like perf and trace to analyze where time is being spent. Thanks,Quoted 3 lines
Alex
Quoted 6 lines
I have not tried passing-through nvidia gpu as i have read some scary tales of nvidia gpus being problematic. Besides now it would be a little bit pointless (unless its only for test) as i got more powerful GPU for use in VM. I will definitely try figuring out where time is spent. Some of your mentioned stuff is really all new to me and might take bit of time to figure out.
IME, Nvidia actually works better than AMD. The Nvidia quirks mostly come from the reverse engineering that the nouveau project has done while the AMD quirks come solely from my reverse engineering. Nvidia is also interested in supporting the professional class cards in VMs whereas I haven't seen any interest in that from AMD. Note that the professional class cards are the Quadro, GRID, and Tesla boards and they're supported in a secondary display configuration. Using x-vga=on with consumer Geforce or Radeon boards is equally not supported by either vendor.
Quoted 9 lines
Im not 100% sure what vga quirks are so please forgive for possibly stupid question but are they enabled by x-vga=on? Something at the back of my head is telling me that you are probably not using that flag since your card is passed-through as >secondary< device. I also tried to achieve same thing with no luck. What i tried was basically -vga std and removing x-vga=on bit. Result was device manager indicating non-working device. I believe error was 12 (not enough resources) or something very similar. If you have ideas regarding this bit i would love to hear them. In the meantime ill snoop around stuff you mentioned, thank you for hints.
Yes, secondary mode is to provide the guest with a -vga device in addition to the assigned GPU, and not use the x-vga=on option. Whether this works often depends on the driver in the guest.
The quirks that I'm talking about are fixes for hardware backdoors. For instance PCI config space is often mirrored in MMIO config space on graphics cards. Running in a virtual machine requires that we emulate portions of config space, so we need to trap accesses to the MMIO mirror. If the driver uses the mirror, or anything within PAGE_SIZE of the mirror, for performance critical access, the traps required for the quirk will seriously affect the performance.
The Q35 QEMU model could also have something to do with the performance issues you see. There have been unsubstantiated claims that VMs are slower with Q35. At the same time, it provides a topology that looks a lot more like a modern system than the default machine. The numbers I reported were using the standard 440FX machine type. This is another case where the driver in the guest may be more or less accommodating of what we expose in the VM, so YMMV with consumer cards and different drivers. Thanks,
Alex
Quoted 147 lines · Alex Williamson
Quoted 98 lines · rndbit
Quoted 93 lines · Alex Williamson
Quoted 57 lines · rndbit
Hello,
I am one of those early tinkerers with VGA passthrough. This is very exciting feature and as soon as i saw it i thought to myself that i want that "95% speed of bare metal" as various sources claimed. So i invested into some harware only to find out that actual 3d performance is far lower. I am not sure why it is the case so maybe someone could either clarify if it is known defect or something to be expected. Or if this is actually a bug and someone someone cared enough to tackle it i would gladly be a test-monkey.Hardware:
Motherboard: SABERTOOTH 990FX R2.0 (bios v2104)
CPU: AMD FX(tm)-8350
GPU1 (host): GeForce GTX 550 Ti
GPU2 (guest): Radeon R9 270X
QEMU 1.7.90 (from git)qemu command line:
Quoted 4 lines
/usr/local/bin/qemu-system-x86_64\(quoted diff) +0
-3-enable-kvm -m 4096 -cpu qemu64 -machine q35,accel=kvm \ -smp 8,sockets=1,cores=8,threads=1 \ -drive file=/dev/sdc,if=none,id=virtio-disk0,format=raw,cache=none,aio=native \Review
Quoted 10 lines
(quoted diff) +0
-10-device virtio-blk-pci,scsi=off,drive=virtio-disk0,id=disk0 \ -drive file=/dev/sdd,if=none,id=virtio-disk1,format=raw,cache=none,aio=native \ -device virtio-blk-pci,scsi=off,drive=virtio-disk1,id=disk1 \ -device e1000,netdev=vnet0,mac=40:01:23:ff:9a:00 -netdev tap,id=vnet0 \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=06:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=00:13.0,bus=pcie.0 \ -device vfio-pci,host=00:13.2,bus=pcie.0 \ -bios /home/novist/opt/src/seabios/out/bios.bin \ -vga noneKernel: 3.14 with kvm patches for 3.15 (i was hoping they would make a difference but sadly they dont). Additional kvm options: iommu=1 vfio_iommu_type1.allow_unsafe_interrupts=1 kvm.ignore_msrs=1
Quoted 4 lines
00:13.0 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB OHCI0 Controller 00:13.2 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB EHCI Controller 06:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Curacao XT [Radeon R9 270X] 06:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Cape Verde/Pitcairn HDMI Audio [Radeon HD 7700/7800 Series]
Actual problem is that GPU seems to be underused. For example in passmark benchmark i score only 15 FPS in directx9 complex test while on bare metal its 100 FPS. That is quite a gap. With kernels prior 3.14 it was even worse - i could get only 10 FPS.
Another rather naive test i performed was checking gpu load with process hacker while playing actual recent game (arma3 in this case). GPU utilization sits at about 20-30%, rarely 40% spikes. And it is lagy of course. Low settings make it somewhat playable. On bare metal though very high settings run smooth. Its worth noting that directx10/11 bencmarks of passmark perform much better than directx9 ones.
Here are benchmark results and GPU load graphs:
GPU load: http://imgur.com/M70CnIV
Benchmark: http://imgur.com/IeYh4ZcAs you see while directx11 performs at about acceptable rate (considering others preach glorious 95% of bare metal performance) directx10 test is quite slower and directx9 performs terribly.
Quoted 7 lines
Not sure what other information i could provide. If there is any - i would be happy to look it up. So any idea where this mythical "95% of bare metal performance" can be found? Cant wait for some insight from a professional.
P.S. i am also adding 2d benchmark results if anyone cares. GPU load: http://imgur.com/28x5DN8
Benchmark: http://imgur.com/562PJYlHave you tried assigning the Nvidia GPU? If you doubt that good performance is possible, see these images of the same tests:
This is with an Nvidia Quadro K4000 assigned as a secondary device, without VGA quirks on an Intel host. It's entirely possible that VGA quirks are getting in the way in your test and may still with the GTX550Ti using VGA quirks. I guarantee that if the quirks are hit in any sort of hot path, performance will plummet.
It's also important to tune the setup, the above tests use hv-time on the vCPU and a kernel that supports it, as well as large pages for the guest and vCPU pinning.
There's potentially still work to do, the above tests show some significant hits in the 2D test, but 3D is quite good.
If you'd like to help improve the situation, you can enable debug when compiling qemu vfio, uncomment hw/misc/vfio.c:#define
DEBUG_VFIO. This will produce a lot of output, especially on guest boot, but it should settle down as the drivers get loaded and direct access is used. If it doesn't settle down and it's through the quirks, we may never be able to achieve good performance since we don't own the driver. If it does settle down, you can use tools like perf and trace to analyze where time is being spent. Thanks,Quoted 3 lines
Alex
Quoted 6 lines
I have not tried passing-through nvidia gpu as i have read some scary tales of nvidia gpus being problematic. Besides now it would be a little bit pointless (unless its only for test) as i got more powerful GPU for use in VM. I will definitely try figuring out where time is spent. Some of your mentioned stuff is really all new to me and might take bit of time to figure out.
IME, Nvidia actually works better than AMD. The Nvidia quirks mostly come from the reverse engineering that the nouveau project has done while the AMD quirks come solely from my reverse engineering. Nvidia is also interested in supporting the professional class cards in VMs whereas I haven't seen any interest in that from AMD. Note that the professional class cards are the Quadro, GRID, and Tesla boards and they're supported in a secondary display configuration. Using x-vga=on with consumer Geforce or Radeon boards is equally not supported by either vendor.
Quoted 9 lines
Im not 100% sure what vga quirks are so please forgive for possibly stupid question but are they enabled by x-vga=on? Something at the back of my head is telling me that you are probably not using that flag since your card is passed-through as >secondary< device. I also tried to achieve same thing with no luck. What i tried was basically -vga std and removing x-vga=on bit. Result was device manager indicating non-working device. I believe error was 12 (not enough resources) or something very similar. If you have ideas regarding this bit i would love to hear them. In the meantime ill snoop around stuff you mentioned, thank you for hints.
Yes, secondary mode is to provide the guest with a -vga device in addition to the assigned GPU, and not use the x-vga=on option. Whether this works often depends on the driver in the guest.
The quirks that I'm talking about are fixes for hardware backdoors. For instance PCI config space is often mirrored in MMIO config space on graphics cards. Running in a virtual machine requires that we emulate portions of config space, so we need to trap accesses to the MMIO mirror. If the driver uses the mirror, or anything within
PAGE_SIZEof the mirror, for performance critical access, the traps required for the quirk will seriously affect the performance.The Q35 QEMU model could also have something to do with the performance issues you see. There have been unsubstantiated claims that VMs are slower with Q35. At the same time, it provides a topology that looks a lot more like a modern system than the default machine. The numbers I reported were using the standard 440FX machine type. This is another case where the driver in the guest may be more or less accommodating of what we expose in the VM, so YMMV with consumer cards and different drivers. Thanks,
Quoted 3 lines
Alex
I see. From what you were saying i got an impression that trying 440fx machine type would help. I actually managed to do it by simply defining vm on libvirt. It passed through both GPU and USB ports just fine. On boot VM properly switches to physical GPU and emulated one is disabled by windows. So far so good. However there was no performance increase whatsoever. At best it was just the same. I tried qemu with DEBUG_VFIO enabled too. After OS is done booting log gets appended only with log lines like these:
Quoted 2 lines
vfio:
vfio_bar_read(0000:00:13.2:BAR0+0x24, 4)=0x4000
vfio:vfio_bar_write(0000:00:13.2:BAR0+0x28, 0x3f, 4)
00:13.2 is USB port. I also tried running dx9 complex test and did not see any more spam relating to GPU (06:00.0). I also checked vfio output for config i posted earlier (with addition of hv-time) but result is identical to libvirt 440fx test. Not sure what to make of this.
I also had not really successful attempt to enable hugepages but i think it really should not be the case of 3d slowdown so i choose to ignore this failure for now.
Tinkering around cpu pinning also resulted in few fps more or few less in dx9 complex benchmark so really not significant to be the real culprit here.