KVM: How does is PAT emulation supposed to work?

When hardware supports the g_pat VMCB field, we can use it for emulating the PAT configuration that the guest configures by writing to the corresponding MSR.

Signed-off-by Jan Kiszka <[email protected]>

---

When hardware supports the g_pat VMCB field, we can use it for emulating the PAT configuration that the guest configures by writing to the corresponding MSR.

Signed-off-by Jan Kiszka <[email protected]>

---

RFC because it is only compile-tested.

Tested-by Radim Krčmář <[email protected]>

2015-04-20 18:14+0200, Radim Krčmář:

Quoted 1 line

Tested-by Radim Krčmář <[email protected]>

Uncached accesses were roughly 20x slower.
In case anyone wanted to reproduce, I used this as a kvm-unit-test:

---
#include "processor.h"

#define NR_TOP_LOOPS 24
#define NR_MEM_LOOPS 10
#define MEM_ELEMENTS 1024

static volatile u64 pat_test_memory[MEM_ELEMENTS];

static void flush_tlb(void)

{
	write_cr3(read_cr3());
}

static void set_pat(u64 val)

{
	wrmsr(0x277, val);
	flush_tlb();

}

static u64 time_memory_accesses(void)

{
	u64 tsc_before = rdtsc();

	for (unsigned loop = 0; loop < NR_MEM_LOOPS; loop++)
		for (unsigned i = 0; i < MEM_ELEMENTS; i++)
			pat_test_memory[i]++;

	return rdtsc() - tsc_before;
}

int main(int argc, char **argv)

{
	unsigned error = 0;

	for (unsigned loop = 0; loop < NR_TOP_LOOPS; loop++) {
		u64 time_uc, time_wb;

		set_pat(0);
		time_uc = time_memory_accesses();

		set_pat(0x0606060606060606ULL);
		time_wb = time_memory_accesses();

		if (time_uc < time_wb * 4)
			error++;

		printf("%02d uc: %10lld wb: %8lld\n", loop, time_uc, time_wb);
	}

	report("guest PAT", !error);

	return report_summary();
}

Quoted 7 lines

2015-04-20 18:14+0200, Radim Krčmář:

Quoted 1 line

Tested-by Radim Krčmář <[email protected]>

Uncached accesses were roughly 20x slower.
In case anyone wanted to reproduce, I used this as a kvm-unit-test:

---

Great, thanks. Will you push it to the unit tests? Could raise motivations to fix the !NPT/EPT case.

2015-04-20 19:21+0200, Jan Kiszka:

Quoted 7 lines · Radim Krčmář

Quoted 6 lines

2015-04-20 18:14+0200, Radim Krčmář:

Quoted 1 line

Tested-by Radim Krčmář <[email protected]>

Uncached accesses were roughly 20x slower.
In case anyone wanted to reproduce, I used this as a kvm-unit-test:

Quoted 1 line

Quoted 1 line

---

| [code]

Quoted 3 lines

Great, thanks. Will you push it to the unit tests? Could raise motivations to fix the !NPT/EPT case.

It can't be included in run_tests.sh, because we intenionally ignore PAT for normal RAM on VMX and the test does "fail" ...

I'll think how to make the test use fool-proof first, and also look how to fix the !NPT/EPT without affecting the case we care about too much. (And if we can do a similar trick with NPT.)

Quoted 11 lines · Radim Krčmář

2015-04-20 19:21+0200, Jan Kiszka:

Quoted 8 lines · Radim Krčmář

Quoted 7 lines

2015-04-20 18:14+0200, Radim Krčmář:

Quoted 1 line

Tested-by Radim Krčmář <[email protected]>

Uncached accesses were roughly 20x slower.
In case anyone wanted to reproduce, I used this as a kvm-unit-test:

---

| [code]

Quoted 1 line

Quoted 5 lines

Quoted 2 lines

Great, thanks. Will you push it to the unit tests? Could raise motivations to fix the !NPT/EPT case.

It can't be included in run_tests.sh, because we intenionally ignore PAT for normal RAM on VMX and the test does "fail" ...

That ignoring is encoded into the EPT? Hmm... Maybe we can create a ivshmem device and use that as test target.

Quoted 12 lines · Radim Krčmář

Quoted 11 lines

2015-04-20 19:21+0200, Jan Kiszka:

Quoted 8 lines · Radim Krčmář

Quoted 7 lines

2015-04-20 18:14+0200, Radim Krčmář:

Quoted 1 line

Tested-by Radim Krčmář <[email protected]>

Uncached accesses were roughly 20x slower.
In case anyone wanted to reproduce, I used this as a kvm-unit-test:

---

| [code]

Quoted 1 line

Quoted 7 lines

Quoted 5 lines

Quoted 2 lines

Great, thanks. Will you push it to the unit tests? Could raise motivations to fix the !NPT/EPT case.

It can't be included in run_tests.sh, because we intenionally ignore PAT for normal RAM on VMX and the test does "fail" ...

That ignoring is encoded into the EPT?

And do you also know why is it ignored on Intel? Side effects on the host?

Quoted 2 lines

Hmm... Maybe we can create a
ivshmem device and use that as test target.

2015-04-20 19:45+0200, Jan Kiszka:

Quoted 13 lines · Jan Kiszka

Quoted 12 lines · Radim Krčmář

Quoted 11 lines

2015-04-20 19:21+0200, Jan Kiszka:

Quoted 8 lines · Radim Krčmář

Quoted 7 lines

2015-04-20 18:14+0200, Radim Krčmář:

Quoted 1 line

Tested-by Radim Krčmář <[email protected]>

Uncached accesses were roughly 20x slower.
In case anyone wanted to reproduce, I used this as a kvm-unit-test:

---

| [code]

Quoted 1 line

Quoted 7 lines

Quoted 7 lines

Quoted 5 lines

Quoted 2 lines

Great, thanks. Will you push it to the unit tests? Could raise motivations to fix the !NPT/EPT case.

It can't be included in run_tests.sh, because we intenionally ignore PAT for normal RAM on VMX and the test does "fail" ...

That ignoring is encoded into the EPT?

Yes, it's the VMX_EPT_IPAT_BIT.

Quoted 1 line

And do you also know why is it ignored on Intel? Side effects on the host?

I think it is an optimization exclusive to Intel.
We know that the other side is not real hardware, which could avoid CPU caches when accessing memory, so there is little reason to slow the guest down.

Quoted 26 lines

2015-04-20 19:45+0200, Jan Kiszka:

Quoted 13 lines · Jan Kiszka

Quoted 12 lines · Radim Krčmář

Quoted 11 lines

2015-04-20 19:21+0200, Jan Kiszka:

Quoted 8 lines · Radim Krčmář

Quoted 7 lines

2015-04-20 18:14+0200, Radim Krčmář:

Quoted 1 line

Tested-by Radim Krčmář <[email protected]>

Uncached accesses were roughly 20x slower.
In case anyone wanted to reproduce, I used this as a kvm-unit-test:

---

| [code]

Quoted 1 line

Quoted 7 lines

Quoted 7 lines

Quoted 5 lines

Quoted 2 lines

Great, thanks. Will you push it to the unit tests? Could raise motivations to fix the !NPT/EPT case.

It can't be included in run_tests.sh, because we intenionally ignore PAT for normal RAM on VMX and the test does "fail" ...

That ignoring is encoded into the EPT?

Yes, it's the VMX_EPT_IPAT_BIT.

Quoted 1 line

And do you also know why is it ignored on Intel? Side effects on the host?

Quoted 4 lines

I think it is an optimization exclusive to Intel.
We know that the other side is not real hardware, which could avoid CPU caches when accessing memory, so there is little reason to slow the guest down.

If the guest pushes data for DMA into RAM, it may assume that it lands there directly, without the need for explicit flushes, because it has caching disabled - no?

If the guest pushes data for DMA into RAM, it may assume that it lands there directly, without the need for explicit flushes, because it has caching disabled - no?

Uncached accesses were roughly 20x slower.

When hardware supports the g_pat VMCB field, we can use it for emulating the PAT configuration that the guest configures by writing to the corresponding MSR.

Signed-off-by Jan Kiszka <[email protected]>

---

Changes in v2:
- add mark_dirty as found missing by Radim

When hardware supports the g_pat VMCB field, we can use it for emulating the PAT configuration that the guest configures by writing to the corresponding MSR.

Signed-off-by Jan Kiszka <[email protected]>

---

Changes in v2:
- add mark_dirty as found missing by Radim

 arch/x86/kvm/svm.c | 10 ++++++++++
 1 file changed, 10 insertions(+)

arch/x86/kvm/svm.c +10 -0

index ce741b8..68fdddc 100644
@@ -3245,6 +3245,16 @@ static int svm_set_msr(struct kvm_vcpu *vcpu, struct msr_data *msr)
 	case MSR_VM_IGNNE:
 		vcpu_unimpl(vcpu, "unimplemented wrmsr: 0x%x data 0x%llx\n", ecx, data);
 		break;
+	case MSR_IA32_CR_PAT:
+		if (npt_enabled) {
+			if (!kvm_mtrr_valid(vcpu, MSR_IA32_CR_PAT, data))
+				return 1;
+			svm->vmcb->save.g_pat = data;
+			mark_dirty(svm->vmcb, VMCB_NPT);
+			vcpu->arch.pat = data;
+			break;
+		}
+		/* fall through */
 	default:
 		return kvm_set_msr_common(vcpu, msr);
 	}

Quoted 4 lines · Jan Kiszka

When hardware supports the g_pat VMCB field, we can use it for emulating the PAT configuration that the guest configures by writing to the corresponding MSR.

Quoted 1 line

Signed-off-by Jan Kiszka <[email protected]>

I'm not sure about this. The problem is that, unlike Intel, AMD has no way for the host to force its PAT value and ignore the guest's. I'm worried about potential performance problems in the guest.

This is not as bad as on ARM, because the guest cannot disable the cache

snooping protocol and thus cache coherency is guaranteed (see tables 7-10 and 15-20 in the AMD docs), but still I think I'd prefer having some knob (module parameter) to enable/disable gPAT. It's okay to make it enabled by default.

Quoted 7 lines · Paolo Bonzini

Quoted 4 lines · Jan Kiszka

When hardware supports the g_pat VMCB field, we can use it for emulating the PAT configuration that the guest configures by writing to the corresponding MSR.

Quoted 5 lines

Quoted 1 line

Signed-off-by Jan Kiszka <[email protected]>

I'm not sure about this. The problem is that, unlike Intel, AMD has no way for the host to force its PAT value and ignore the guest's. I'm worried about potential performance problems in the guest.

I think the guest needs to get what it requests - see my remark in
http://thread.gmane.org/gmane.comp.emulators.kvm.devel/135271.

Quoted 2 lines

This is not as bad as on ARM, because the guest cannot disable the cache

You mean AMD, I guess.

Quoted 4 lines

snooping protocol and thus cache coherency is guaranteed (see tables 7-10 and 15-20 in the AMD docs), but still I think I'd prefer having some knob (module parameter) to enable/disable gPAT. It's okay to make it enabled by default.

I still don't get the scenario where we want to override the guest settings. Maybe you can help out - would be valuable for the reasoning in code or commit logs as well.

Basically it's an optimization. The guest can set the UC memory type on PCI BARs that are actually backed by RAM in QEMU, and then accesses to these BARs will be unnecessarily slow. It would be particularly bad if, for example, access to ivshmem were slowed down because the guest PAT says the memory is uncacheable.

Quoted 5 lines

Basically it's an optimization. The guest can set the UC memory type on PCI BARs that are actually backed by RAM in QEMU, and then accesses to these BARs will be unnecessarily slow. It would be particularly bad if, for example, access to ivshmem were slowed down because the guest PAT says the memory is uncacheable.

ivshmem is pv anyway - why shouldn't the guest driver take this room for optimization into account and ask for a cached mapping?

Is that that only use case?

Quoted 4 lines

When hardware supports the g_pat VMCB field, we can use it for emulating the PAT configuration that the guest configures by writing to the corresponding MSR.

Quoted 1 line

Signed-off-by Jan Kiszka <[email protected]>

I'm not sure about this. The problem is that, unlike Intel, AMD has no way for the host to force its PAT value and ignore the guest's. I'm worried about potential performance problems in the guest.

2015-04-21 13:09+0200, Paolo Bonzini:

Quoted 9 lines · Jan Kiszka

Quoted 5 lines

When hardware supports the g_pat VMCB field, we can use it for emulating the PAT configuration that the guest configures by writing to the corresponding MSR.

Signed-off-by Jan Kiszka <[email protected]>

Quoted 3 lines

I'm not sure about this. The problem is that, unlike Intel, AMD has no way for the host to force its PAT value and ignore the guest's. I'm worried about potential performance problems in the guest.

We already set g_pat to 0x0007040600070406ULL in init_vmcb(). This patch uses caching that the guest expects, which might improve performance as well. I think it's a step in right direction even if we somehow optimize cache coherent cases later.

Quoted 14 lines · Radim Krčmář

2015-04-21 13:09+0200, Paolo Bonzini:

Quoted 12 lines · Jan Kiszka

Quoted 5 lines

When hardware supports the g_pat VMCB field, we can use it for emulating the PAT configuration that the guest configures by writing to the corresponding MSR.

Signed-off-by Jan Kiszka <[email protected]>

I'm not sure about this. The problem is that, unlike Intel, AMD has no way for the host to force its PAT value and ignore the guest's. I'm worried about potential performance problems in the guest.

Quoted 4 lines

We already set g_pat to 0x0007040600070406ULL in init_vmcb(). This patch uses caching that the guest expects, which might improve performance as well. I think it's a step in right direction even if we somehow optimize cache coherent cases later.

This topic is still open - and the patch still applies.

Hi all,

while digging into the PAT topic for Jailhouse, I also wondered how KVM deals with it. And I'm still not getting it complete - or there is a bug:

KVM intercepts all guest writes to the PAT MSR and instead keeps the guest value in vcpu->arch.pat. But, besides returning that value back on read accesses, arch.pat has no other purpose.

On Intel, we only seem to have proper emulation - through hardware - when VMX supports PAT switching (see vmx_set_msr). On AMD, the situation is even worse as the g_pat save field is not updated at all on PAT writes. That seems to be a low hanging fruit to bring svm on the same support level as vmx.

Or am I missing something?

PS: If someone has a good idea for a simple test case on machines without IOMMU (like my current boxes), thus without a chance to use device pass-through to stress guest PAT settings, I would be all ears.

2015-04-13 07:16+0200, Jan Kiszka:

Quoted 3 lines

PS: If someone has a good idea for a simple test case on machines without IOMMU (like my current boxes), thus without a chance to use device pass-through to stress guest PAT settings, I would be all ears.

Not a good one: KVM sets VMX_EPT_IPAT_BIT for RAM unless kvm_arch_has_noncoherent_dma(). You can comment the line in vmx_get_mt_mask(), or call kvm_arch_register_noncoherent_dma(), for guest PAT to work on normal memory.

Quoted 5 lines

2015-04-13 07:16+0200, Jan Kiszka:

Quoted 3 lines

PS: If someone has a good idea for a simple test case on machines without IOMMU (like my current boxes), thus without a chance to use device pass-through to stress guest PAT settings, I would be all ears.

Quoted 4 lines

Not a good one: KVM sets VMX_EPT_IPAT_BIT for RAM unless kvm_arch_has_noncoherent_dma(). You can comment the line in vmx_get_mt_mask(), or call kvm_arch_register_noncoherent_dma(), for guest PAT to work on normal memory.

That's for VMX (where I do have IOMMUs), but I would need something for AMD. :)