New: 4.2 breaks PCI passthrough in QEMU/KVM
https://bugzilla.kernel.org/show_bug.cgi?id=107561
Bug ID: 107561
Summary: 4.2 breaks PCI passthrough in QEMU/KVM
Product: Virtualization
Version: unspecified
Kernel Version: 4.2
Hardware: All
OS: Linux
Tree: Mainline
Status: NEW
Severity: normal
Priority: P1
Component: kvm
Assignee: [email protected]
Reporter: [email protected]
Regression: No
QEMU/KVM VM's using PCI pass-through are taking a long time to boot, or never booting. The issue is only present when the VM has more then ~2.5G of ram. I was able to get a VM with 6G to boot, but I had to let it run overnight.
More info can be found in this thread:
https://bbs.archlinux.org/viewtopic.php?id=203240
Someone in the forum seems to have narrowed it down to a a set of commits regarding kvm and mtrr in 4.1-rc2 that are in the mainline kernel as of 4.2.
Specifically commit fa61213746a706dd975661151c35795ca4dd82c2 KVM: MTRR:
simplify kvm_mtrr_get_guest_memory_type.
Signature · 2 lines
You are receiving this mail because: You are watching the assignee of the bug.
[Bug 107561] 4.2 breaks PCI passthrough in QEMU/KVM
https://bugzilla.kernel.org/show_bug.cgi?id=107561
John Brooks <[email protected]> changed:
What |Removed |Added
----------------------------------------------------------------------------
CC| |[email protected]
--- Comment #1 from John Brooks <[email protected]> ---
I can confirm this bug. As soon as I enabled PCI passthrough on my VM while using kernel 4.2.2, my VM ran extremely slowly, and increasing the amount of RAM made the problem worse. Switching to kernel 4.1.12 fixed the problem.
Signature · 2 lines
You are receiving this mail because: You are watching the assignee of the bug.
[Bug 107561] 4.2 breaks PCI passthrough in QEMU/KVM
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=107561
Aaron Campbell <[email protected]> changed:
What |Removed |Added
----------------------------------------------------------------------------
CC| |[email protected]
--- Comment #2 from Aaron Campbell <[email protected]> ---
I can also confirm this bug. After upgrading the kernel on a Fedora box from 4.0.4 to 4.2.5, my Windows 8.1 VM hung on start. The VM would only start if the GPU passthrough configuration was removed, or if the VM's memory size was reduced to 2GB.
Signature · 2 lines
You are receiving this mail because: You are watching the assignee of the bug.
[Bug 107561] 4.2 breaks PCI passthrough in QEMU/KVM
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=107561
[email protected] changed:
What |Removed |Added
----------------------------------------------------------------------------
CC| |[email protected]
--- Comment #3 from [email protected] ---
Issue is still present in 4.4-rc1 and same conditions exist. After lowering it below 2.5G of ram, the VM boots normally. Anything over 2.5G will cause it to boot really slow and utilize a lot of cpu.
Signature · 2 lines
You are receiving this mail because: You are watching the assignee of the bug.
[Bug 107561] 4.2 breaks PCI passthrough in QEMU/KVM
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=107561
[email protected] changed:
What |Removed |Added
----------------------------------------------------------------------------
CC| |[email protected]
--- Comment #4 from [email protected] ---
I narrowed it down earlier to "simplify kvm_mtrr_get_guest_memory_type". Actually this was the commit to break it, but it is only a small part of the feauture set implementing vMTRR. To my knowledge 4.1.x versions didn't have this code set at all. It was introduced in 4.2.
What kvm_mtrr_get_guest_memory_type() does is basically decides what cache type to use (Uncachable, Write-Back, or Write-Through) based on the function mtrr_for_each_mem_type(). This kvm_mtrr_get_guest_memory_type function is only called by vmx_get_mt_mask. This is all related to how the hardware is set up regarding IOMMU, MMIO, VT-d and EPT andress translation.
For experimenting purposes I simply modified kvm_mtrr_get_guest_memory_type to always return MTRR_TYPE_WRBACK and that fixed the issue. Guest boots correctly. Although I know, that hardcoding Write-back mode is not a viable long-term solution. It seems as the new function (in 4.2 and onward) does not return the proper cache type in case of pci passthrough configurations.
I would also like to point out, that
https://bugzilla.kernel.org/show_bug.cgi?id=107921 may be related. The other bug mentions very similar hardware to mine, and issues started the same time (upgrading to 4.2.x) with a pci passthrough configuration.
Signature · 2 lines
You are receiving this mail because: You are watching the assignee of the bug.
[Bug 107561] 4.2 breaks PCI passthrough in QEMU/KVM
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=107561
Paolo Bonzini <[email protected]> changed:
What |Removed |Added
----------------------------------------------------------------------------
CC| |[email protected]
--- Comment #5 from Paolo Bonzini <[email protected]> ---
Please capture a trace according to http://www.linux-kvm.org/page/Tracing, so that I can see your MTRR settings and figure out why that patch breaks things.
If possible, capture a trace with 2 GB of memory and one with 3 GB.
Signature · 2 lines
You are receiving this mail because: You are watching the assignee of the bug.
[Bug 107561] 4.2 breaks PCI passthrough in QEMU/KVM
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=107561
--- Comment #6 from Paolo Bonzini <[email protected]> ---
Ah, the traces are huge. You can xz them and send them through private email.
Signature · 2 lines
You are receiving this mail because: You are watching the assignee of the bug.
[Bug 107561] 4.2 breaks PCI passthrough in QEMU/KVM
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=107561
Paolo Bonzini <[email protected]> changed:
What |Removed |Added
----------------------------------------------------------------------------
CC| |[email protected]
--- Comment #7 from Paolo Bonzini <[email protected]> --- *** Bug 107921 has been marked as a duplicate of this bug. ***
Signature · 2 lines
You are receiving this mail because: You are watching the assignee of the bug.
[Bug 107561] 4.2 breaks PCI passthrough in QEMU/KVM
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=107561
--- Comment #8 from [email protected] ---
I captured the traces, they take more than half a GB uncompressed. They are available in xz under
https://drive.google.com/folderview?id=0B8ebX_WjVHnGNlN4eTEzU2xtMEk&usp=sharing
To make things clear: I have two hosts. HostB is a testing machine. VGA passthrough with EDKII worked out of the box on 4.1 kernels. It broke with with several 4.2 versions and also 4.3. Tried several repo versions and untouched kernel.org versions, that was back about a month ago. Than I tested again back a few days, and it works again with 4.2.6-301.fc23 fedora repo kernel.
HostA is the problematic one. I did the trace as requested, with 2G and 3G guest ram. It worked with 4.1 kernels, but still doesn't work with 4.2.6-301.fc23. I can make it work with either lowering RAM to under 2.5 GB or with my beforementioned modification (kvm_mtrr_get_guest_memory_type to always return MTRR_TYPE_WRBACK). Of course I made the traces with unmodified kernel, and only the 2G guest actually booted.
The exact symptom is (when I say it doesn't work), that the guest is extremly slow. I tried booting live Linux guest, after about 15 minutes a saw messages, but even after two hours I still couldn't get to a shell. Windows guest only shows the white dots under the logo circling around very slowly. Once I got a blue screen, and letters came up one-by-one, like if the error message was written with a typewriter. Sometimes the guest just shuts down, qemu terminates. UEFI shell, UEFI setup in guest works perfectly at all conditions, slowness starts when booting an OS.
I can also provide the traces with working kernel version on HostA and also on HostB, if requested for comparison.
In the shared folders you can find MTRR, PAT, and lspci -vvv info for each host, along with traces for HostA as requested (2GB and 3GB). One of the members on the original Arch Linux thread suggested I put a printk in the problematic function. The dmesg files in each folder show the arguments of vmx_get_mt_mask and what kvm_mtrr_get_guest_memory_type returns to it. The added line was (just before the return statement in vmx_get_mt_mask):
printk(KERN_INFO "vmx_get_mt_mask got the following: cpu=%d, vcpu=%d, gfn=%x, MMIO=%d, cache=%x", vcpu->cpu, vcpu->vcpu_id, gfn, is_mmio, cache);
It is visible from dmesg files that in case of large guest ram, the function doesn't even get called for vcpus other than 0. On the other hand it is called for all in case of small memory.
The traces are NOT from the same run as the dmesg files, as they have been created before your post about tracing.
Please ask if any more info, traces or dumps are needed. I would be glad to provide any help.
Signature · 2 lines
You are receiving this mail because: You are watching the assignee of the bug.
[Bug 107561] 4.2 breaks PCI passthrough in QEMU/KVM
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=107561
--- Comment #9 from Paolo Bonzini <[email protected]> --- Can you try the following on HostA:
1) providing *host* dmesg
2) disabling each MTRR starting from the first, until things are fast again. Then reboot and try disabling only the MTRR that made things fast again.
Signature · 2 lines
You are receiving this mail because: You are watching the assignee of the bug.
[Bug 107561] 4.2 breaks PCI passthrough in QEMU/KVM
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=107561
--- Comment #10 from Paolo Bonzini <[email protected]> ---
Another test on the 3GB machine: make it fast by nuking kvm_mtrr_get_guest_memory_type (return MTRR_TYPE_WRBACK), then cat /proc/mtrr from the guest.
Signature · 2 lines
You are receiving this mail because: You are watching the assignee of the bug.
[Bug 107561] 4.2 breaks PCI passthrough in QEMU/KVM
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=107561
--- Comment #11 from Paolo Bonzini <[email protected]> --- Created attachment 197071
--> https://bugzilla.kernel.org/attachment.cgi?id=197071&action=edit
OVMF log
Quoted 1 line
1) providing host dmesg
Of course this makes no sense, you already did. And from the traces I suspect that /proc/mtrr is the same on the two guests. The culprit seems to be this MTRR:
qemu-system-x86-2228 [002] 127.763684: kvm_msr: msr_read 201 = 0xff80000800 qemu-system-x86-2228 [002] 127.763685: kvm_msr: msr_read 200 = 0x80000000
which is set up by OVMF. I'm attaching the OVMF output for reference (it can be extracted from the traces with "trace-cmd report trace-3GBmem.dat |grep 0x402 | awk --non-decimal-data '{printf "%c", 0+$NF}' > OVMF.log").
Signature · 2 lines
You are receiving this mail because: You are watching the assignee of the bug.
[Bug 107561] 4.2 breaks PCI passthrough in QEMU/KVM
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=107561
--- Comment #12 from Paolo Bonzini <[email protected]> --- Created attachment 197081
--> https://bugzilla.kernel.org/attachment.cgi?id=197081&action=edit
patch to be tested
The bug happens when the VM is created with a maxphyaddr that doesn't match the host. This patch should fix your bug.
Signature · 2 lines
You are receiving this mail because: You are watching the assignee of the bug.
[Bug 107561] 4.2 breaks PCI passthrough in QEMU/KVM
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=107561
--- Comment #13 from [email protected] ---
Sorry for the delay. I couldn't check sooner.
Unfortunately the guest doesn't start with the patch applied. Nothing displayed (no sync signal from passthrough card). Host dmesg doesn't show any errors. Usually, even with slow guest I get the following on host dmesg when starting VM:
...
VFIO - User Level meta-driver version: 0.3
kernel: vgaarb: device changed decodes:
PCI:0000:01:00.0,olddecodes=io+mem,decodes=io+mem:owns=io+mem
kernel: vfio_ecap_init: 0000:01:00.0 hiding ecap 0x1e@0x258
kernel: vfio_ecap_init: 0000:01:00.0 hiding ecap 0x19@0x900
kernel: pmd_set_huge: Cannot satisfy [mem 0xe0000000-0xe0200000] with a huge-page mapping due to MTRR override.
- than some USB resets for keyboard passthrough -
kernel: kvm: zapping shadow pages for mmio generation wraparound ...and so on, the guest screen displays UEFI startup by now
With the patch applied it simly stops at the pmd_set_huge line, and nothing further, no usb, no guest start, no error messages.
I also did follow up on the other tests.
Disabling host mtrrs one-by-one did not yield any results. Guest is still slow even if disabling all the hosts mtrrs.
Tried nukeing kvm_mtrr_get_guest_memory_type as requested and started a live distro. Guest did boot, but did not have a /proc/mtrr file. (Graphics scrolling was slow, I guess on account of lack of cacheing, but otherwise guest was speedy as normal) I saved some info from the guest, like dmesg, lspci -vvv, and some sysfs files. You can find them in the same share as before.
https://drive.google.com/folderview?id=0B8ebX_WjVHnGNlN4eTEzU2xtMEk&usp=sharing
Signature · 2 lines
You are receiving this mail because: You are watching the assignee of the bug.
[Bug 107561] 4.2 breaks PCI passthrough in QEMU/KVM
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=107561
--- Comment #14 from Jasen Borisov <[email protected]> ---
Can confirm. I am also hit by this bug, and my virtual machine doesn't start either.
I tried applying the patch to both 4.3.0 and linux-next, and neither of those worked.
Signature · 2 lines
You are receiving this mail because: You are watching the assignee of the bug.
[Bug 107561] 4.2 breaks PCI passthrough in QEMU/KVM
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=107561
Paolo Bonzini <[email protected]> changed:
What |Removed |Added
----------------------------------------------------------------------------
Attachment #197081|0 |1
is obsolete| |
Status|NEW |NEEDINFO
Assignee|virtualization_kvm@kernel-b |[email protected]
|ugs.osdl.org |--- Comment #15 from Paolo Bonzini <[email protected]> --- Created attachment 197381
--> https://bugzilla.kernel.org/attachment.cgi?id=197381&action=edit
patch to be tested v2
There is indeed a two-character mistake in the patch. Can you guys please test this one instead?
Signature · 2 lines
You are receiving this mail because: You are watching the assignee of the bug.