New: NPT page attribute support causes extreme slowdown

https://bugzilla.kernel.org/show_bug.cgi?id=103321

            Bug ID: 103321
           Summary: NPT page attribute support causes extreme slowdown
           Product: Virtualization
           Version: unspecified
    Kernel Version: 4.2-rc3+
          Hardware: x86-64
                OS: Linux
              Tree: Mainline
            Status: NEW
          Severity: normal
          Priority: P1
         Component: kvm
          Assignee: [email protected]
          Reporter: [email protected]
        Regression: No

Created attachment 185501
--> https://bugzilla.kernel.org/attachment.cgi?id=185501&action=edit
dmesg

For some reason the new NPT attribute support that has been introduced with 4.3-RC3 causes my VM to run awfully slow, I managed to track this issue to commit 3c2e7f7de3 ("KVM: SVM: use NPT page attributes").

The VM runs Win7 64bit via Qemu and uses PCI passthrough with several devices (primary GPU, USB, audio) on a AMD 970/SB950 based Asrock 970 EXTREME3 2.0 motherboard and an AMD Phenom II X6 CPU.

There are no error/warning messages whatsoever, booting the VM takes several minutes just to get past the first boot screen.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=103321

--- Comment #1 from Sebastian Schütte <[email protected]> --- Created attachment 185511
--> https://bugzilla.kernel.org/attachment.cgi?id=185511&action=edit
cpuinfo

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=103321

--- Comment #2 from Sebastian Schütte <[email protected]> --- Created attachment 185521
--> https://bugzilla.kernel.org/attachment.cgi?id=185521&action=edit
lspci

You are receiving this mail because:
You are watching the assignee of the bug.

No, it seems to be in the BIOS. Please try this:

index 74d825716f4f..f8f69cffecbf 100644
@@ -882,7 +882,8 @@ static u8 fallback_mtrr_type(int mtrr)
 	case MTRR_TYPE_WRTHROUGH:
 		return MTRR_TYPE_UNCACHABLE;
 	case MTRR_TYPE_WRPROT:
-		return MTRR_TYPE_UC_MINUS;
+		/* Assume it's backed by RAM in the host.  */
+		return MTRR_TYPE_WRBACK;
 	default:
 		BUG();
 	}

@@ -3054,6 +3055,7 @@ static int cr_interception(struct vcpu_svm *svm)
 	if (cr >= 16) { /* mov to cr */
 		cr -= 16;
 		val = kvm_register_read(&svm->vcpu, reg);
+		trace_kvm_cr_write(cr, val);
 		switch (cr) {
 		case 0:
 			if (!check_selective_cr0_intercepted(svm, val))

This is a bit of a hack, but it can be enough for a test. For a real patch one would make mtrr2protval bidimensional, accessing it like "return mtrr2protval[is_mmio][mtrr]" in svm_get_mt_mask. Then you can return MTRR_TYPE_UC_MINUS if is_mmio, and MTRR_TYPE_WRBACK if !is_mmio.

Paolo

Still no difference

I guess the trace_kvm_cr_write() call in that patch was supposed to trigger kvm_cr entries while tracing? I couldn't find any, though, the only entries containing "cr" within the output of "trace-cmd report" were kvm_exit ones that looked quite similar to the previous dump.

If you still think it's worth it I'll send you the whole ~10MB compressed trace.

I inserted some printk() lines into init_vmcb() around the call of svm_set_guest_pat() to print out the g_pat value as well as svm->vcpu.vcpu_id and noticed that something was off:

Initially, the PATs of all VCPUs are set to 0x0606060606060606. However, after attaching some devices (vfio-pci enabling device and vfio_ecap_init lines are being printed) init_vmcb() is only called again for vcpu_id > 0. Unless g_pat is changed somewhere else, VCPU #0 remains set to 0x0606060606060606 (according to comments in svm_set_guest_pat() this is bad for assigned devices) while all other VCPUs use 0x0007040600070406.

I'd guess that could explain the slowdown.

Hi, sorry for the delay. If you change KVM like this

index 27f57fc05bc7..3ce878c5fde8 100644
@@ -1155,10 +1155,7 @@ static void svm_set_guest_pat(struct vcpu_svm *svm, u64 *g_pat)
 	 * have assigned devices, however, we cannot force WB for RAM
 	 * pages only, so use the guest PAT directly.
 	 */
-	if (!kvm_arch_has_assigned_device(vcpu->kvm))
-		*g_pat = 0x0606060606060606;
-	else
-		*g_pat = vcpu->arch.pat;
+	*g_pat = vcpu->arch.pat;
 }
 
 static u64 svm_get_mt_mask(struct kvm_vcpu *vcpu, gfn_t gfn, bool is_mmio)

(obviously not a valid patch for upstream patch, but good enough for testing) do you get normal speed?

Paolo

Nope, still slow... I somehow missed that svm_set_msr() calls svm_set_guest_pat() as well, it does seem to properly change g_pat. The only thing that catches my eyes is that later on all WT entries in g_pat are replaced by WC (0x0007040600070406 -> 0x0007010600070106), but that doesn't seem to make a difference.

If it wasn't for some frustrating and badly timed libvirt update breaking my old VM config some days ago I would have noticed that earlier :/

https://bugzilla.kernel.org/show_bug.cgi?id=103321

Jimi <[email protected]> changed:

What |Removed |Added
----------------------------------------------------------------------------
CC| |[email protected]

--- Comment #3 from Jimi <[email protected]> ---
Before discovering this bug page, I started a thread on the Arch forums before coming across a bug that's either the same one as this or something very similar. We've done some extra investigation over there. Here's a link:
https://bbs.archlinux.org/viewtopic.php?pid=1574465

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=103321

--- Comment #4 from Jimi <[email protected]> --- (In reply to Jimi from comment #3)

Er, that is, by "very similar" I mean the behavior. Obviously, they could have wildly different causes.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=103321

Sebastian Schütte <[email protected]> changed:

What |Removed |Added
----------------------------------------------------------------------------

             Status|NEW                         |RESOLVED
         Resolution|---                         |CODE_FIX

--- Comment #5 from Sebastian Schütte <[email protected]> ---
As far as I am concerned, RC4 fixes the issue (should've marked it as fixed earlier) by reverting said commit. This change has also been backported to 4.2 stable about 2 weeks ago, AFAIK.

Note that this bug is actually AMD specific and only affects CPUs with NPT support.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=103321

--- Comment #6 from Jimi <[email protected]> ---
Oh. Definitely not the same bug, then. Ours still isn't fixed and affects more than just AMD.

You are receiving this mail because:
You are watching the assignee of the bug.