Skip to contentSkip to conversation

kvm/vmx: EPTP switching test

PATCH In review Michael S. Tsirkin 6 participants 10 messages Nov 15, 2015 – Nov 17, 2015

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 kvm/vmx: EPTP switching test

This patch adds a new parameter: eptp_switching_test, which enables testing EPT switching on VMX if supported by hardware. All EPT entries are initialized to the same value so this adds no useful functionality by itself, but can be used to test VMFUNC performance, and serve as a basis for future features based on EPTP switching.

Support for nested virt is not enabled.

This was tested using the following code within guest:

	#define VMX_VMFUNC ".byte 0x0f,0x01,0xd4"
	static void vmfunc(unsigned int nr, unsigned int ept)
	{
		asm volatile(VMX_VMFUNC
			     :
			     : "a"(nr), "c"(ept)
			     : "memory");
	}

VMFUNC instruction cost was measured at ~122 cycles. (Note: recent versions of gnu toolchain support
the vmfunc instruction - removing the need for writing the bytecode manually).

---

I think I'd like to put this upstream so future eptp switching work can be implemented on top. Comments?

 arch/x86/include/asm/vmx.h |  7 ++++
 arch/x86/kvm/vmx.c         | 84 ++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 91 insertions(+)
arch/x86/include/asm/vmx.h +7 -0
index 448b7ca..ceb68d9 100644
@@ -69,10 +69,13 @@
 #define SECONDARY_EXEC_VIRTUAL_INTR_DELIVERY    0x00000200
 #define SECONDARY_EXEC_PAUSE_LOOP_EXITING	0x00000400
 #define SECONDARY_EXEC_ENABLE_INVPCID		0x00001000
+#define SECONDARY_EXEC_ENABLE_VM_FUNCTIONS	0x00002000
 #define SECONDARY_EXEC_SHADOW_VMCS              0x00004000
 #define SECONDARY_EXEC_ENABLE_PML               0x00020000
 #define SECONDARY_EXEC_XSAVES			0x00100000
 
+/* Definitions for VM-function controls */
+#define VM_FUNCTION_EPTP_SWITCHING		0x00000001
 
 #define PIN_BASED_EXT_INTR_MASK                 0x00000001
 #define PIN_BASED_NMI_EXITING                   0x00000008
@@ -153,6 +156,8 @@ enum vmcs_field {
 	APIC_ACCESS_ADDR_HIGH		= 0x00002015,
 	POSTED_INTR_DESC_ADDR           = 0x00002016,
 	POSTED_INTR_DESC_ADDR_HIGH      = 0x00002017,
+	VM_FUNCTION_CTRL                = 0x00002018,
+	VM_FUNCTION_CTRL_HIGH           = 0x00002019,
 	EPT_POINTER                     = 0x0000201a,
 	EPT_POINTER_HIGH                = 0x0000201b,
 	EOI_EXIT_BITMAP0                = 0x0000201c,
@@ -163,6 +168,8 @@ enum vmcs_field {
 	EOI_EXIT_BITMAP2_HIGH           = 0x00002021,
 	EOI_EXIT_BITMAP3                = 0x00002022,
 	EOI_EXIT_BITMAP3_HIGH           = 0x00002023,
+	EPTP_LIST_ADDRESS               = 0x00002024,
+	EPTP_LIST_ADDRESS_HIGH          = 0x00002025,
 	VMREAD_BITMAP                   = 0x00002026,
 	VMWRITE_BITMAP                  = 0x00002028,
 	XSS_EXIT_BITMAP                 = 0x0000202C,
arch/x86/kvm/vmx.c +84 -0

152 lines not on this page · raw message

Signature · 1 line
MST
#1 lore raw

2015-11-15 18:00+0200, Michael S. Tsirkin:

Quoted 24 lines

This patch adds a new parameter: eptp_switching_test, which enables

testing EPT switching on VMX if supported by hardware. All EPT entries are initialized to the same value so this adds no useful functionality by itself, but can be used to test VMFUNC performance, and serve as a basis for future features based on EPTP switching.

Support for nested virt is not enabled.

This was tested using the following code within guest:

	#define VMX_VMFUNC ".byte 0x0f,0x01,0xd4"
	static void vmfunc(unsigned int nr, unsigned int ept)
	{
		asm volatile(VMX_VMFUNC
			     :
			     : "a"(nr), "c"(ept)
			     : "memory");
	}

VMFUNC instruction cost was measured at ~122 cycles. (Note: recent versions of gnu toolchain support
the vmfunc instruction - removing the need for writing the bytecode manually).

Quoted 5 lines

---

I think I'd like to put this upstream so future eptp switching work can be implemented on top. Comments?

I'd wait for the future. Patch is already on the list so people interested in benchmarking VMFUNC can quickly compile a kernel and developers will need to overwrite the code anyway.

(And I think that eptp switching is expected to be used in conjuction
with #VE, so it'd then make sense to implement a nop for it as well.)

arch/x86/include/asm/vmx.h static __init int setup_vmcs_config(struct vmcs_config *vmcs_conf) · line 3038

Quoted 6 lines
arch/x86/include/asm/vmx.h +1 -0
@@ -3011,6 +3035,7 @@ static __init int setup_vmcs_config(struct vmcs_config *vmcs_conf)
 			SECONDARY_EXEC_PAUSE_LOOP_EXITING |
 			SECONDARY_EXEC_RDTSCP |
 			SECONDARY_EXEC_ENABLE_INVPCID |
+			SECONDARY_EXEC_ENABLE_VM_FUNCTIONS |

The VMFUNC vmexit should be handled to prevent guests from triggering a WARN_ON on the host. (VMFUNC did just #UD before this patch.)

After that, it's ok for KVM.

#2 lore raw
Quoted 28 lines · On Mon, Nov 16, 2015 at 06:51:06PM +0100…

2015-11-15 18:00+0200, Michael S. Tsirkin:

Quoted 27 lines

This patch adds a new parameter: eptp_switching_test, which enables

testing EPT switching on VMX if supported by hardware. All EPT entries are initialized to the same value so this adds no useful functionality by itself, but can be used to test VMFUNC performance, and serve as a basis for future features based on EPTP switching.

Support for nested virt is not enabled.

This was tested using the following code within guest:

	#define VMX_VMFUNC ".byte 0x0f,0x01,0xd4"
	static void vmfunc(unsigned int nr, unsigned int ept)
	{
		asm volatile(VMX_VMFUNC
			     :
			     : "a"(nr), "c"(ept)
			     : "memory");
	}

VMFUNC instruction cost was measured at ~122 cycles. (Note: recent versions of gnu toolchain support
the vmfunc instruction - removing the need for writing the bytecode manually).

---

Quoted 6 lines
Quoted 2 lines

I think I'd like to put this upstream so future eptp switching work can be implemented on top. Comments?

I'd wait for the future. Patch is already on the list so people interested in benchmarking VMFUNC can quickly compile a kernel and developers will need to overwrite the code anyway.

It'll bitrot though. But I'll let Paolo decide that.

Quoted 2 lines

(And I think that eptp switching is expected to be used in conjuction
with #VE, so it'd then make sense to implement a nop for it as well.)

No idea how would I even test it, so I'm not interested in #VE at this point. If you are - go ahead and post a patch for that on top though, why not.

Quoted 9 lines

arch/x86/include/asm/vmx.h static __init int setup_vmcs_config(struct vmcs_config *vmcs_conf) · line 3038

Quoted 6 lines
arch/x86/include/asm/vmx.h +1 -0
@@ -3011,6 +3035,7 @@ static __init int setup_vmcs_config(struct vmcs_config *vmcs_conf)
 			SECONDARY_EXEC_PAUSE_LOOP_EXITING |
 			SECONDARY_EXEC_RDTSCP |
 			SECONDARY_EXEC_ENABLE_INVPCID |
+			SECONDARY_EXEC_ENABLE_VM_FUNCTIONS |

The VMFUNC vmexit should be handled to prevent guests from triggering a WARN_ON on the host. (VMFUNC did just #UD before this patch.)

Do you mean VMFUNC other than EPTP switch 0? True, thanks!

Quoted 2 lines

After that, it's ok for KVM.

#3 lore raw

2015-11-16 19:59+0200, Michael S. Tsirkin:

Quoted 8 lines · On Mon, Nov 16, 2015 at 06:51:06PM +0100…
Quoted 3 lines

2015-11-15 18:00+0200, Michael S. Tsirkin:
(And I think that eptp switching is expected to be used in conjuction
with #VE, so it'd then make sense to implement a nop for it as well.)

No idea how would I even test it, so I'm not interested in #VE at this point. If you are - go ahead and post a patch for that on top though, why not.

I thought that it's going to be simpler to provide functionality (that utilizes eptp switching) to the guest through #VE, which probably isn't true as I think more about it. (Not interested in implementing it :])

#4 lore raw
Quoted 6 lines · On Mon, Nov 16, 2015 at 10:18 AM, =?UTF-8?q?Radim=20Kr=C4=8Dm=C3=A1=C5=99?=…

2015-11-16 19:59+0200, Michael S. Tsirkin:

Quoted 5 lines · On Mon, Nov 16, 2015 at 06:51:06PM +0100…
Quoted 3 lines

2015-11-15 18:00+0200, Michael S. Tsirkin:
(And I think that eptp switching is expected to be used in conjuction
with #VE, so it'd then make sense to implement a nop for it as well.)

Quoted 7 lines
Quoted 3 lines

No idea how would I even test it, so I'm not interested in #VE at this point. If you are - go ahead and post a patch for that on top though, why not.

I thought that it's going to be simpler to provide functionality (that utilizes eptp switching) to the guest through #VE, which probably isn't true as I think more about it. (Not interested in implementing it :])

I'm willing to review a #VE stub. I'm not likely to implement it.

--Andy

maintainer
#5 lore raw
In reply to =?UTF-8?q?Radim=20Kr=C4=8Dm=C3=A1=C5=99?=
Quoted 6 lines · On 16/11/2015 19:18, =?UTF-8?q?Radim=20Kr=C4=8Dm=C3=A1=C5=99?= wrote:
Quoted 3 lines
Quoted 3 lines

No idea how would I even test it, so I'm not interested in #VE at this point. If you are - go ahead and post a patch for that on top though, why not.

I thought that it's going to be simpler to provide functionality (that utilizes eptp switching) to the guest through #VE, which probably isn't true as I think more about it. (Not interested in implementing it :])

#VE and EPTP switching are distinct features, one does not imply the other.

Unfortunately, EPTP switching is designed for a very specific use case where the hypervisor is effectively part of the kernel, and the kernel is trusted to some extent. Examples include antivirus software and virtual machines. Antiviruses do use VMFUNC, that's as far as I know the only current use case of it
(https://embedded.communities.intel.com/community/en/applications/blog/2013/06/13/roving-reporter-enhancing-retail-security-and-manageability-with-4th-generation-intel-core-processors).

So I'm against this patch, but only because I'm not sure why KVM would ever use EPTP switching in its current incarnation. The guest kernel is absolutely not trusted by KVM.

Paolo

#6 lore raw
In reply to Michael S. Tsirkin

Michael S. Tsirkin wrote on 2015-11-16:

Quoted 22 lines

This patch adds a new parameter: eptp_switching_test, which enables testing EPT switching on VMX if supported by hardware. All EPT entries are initialized to the same value so this adds no useful functionality by itself, but can be used to test VMFUNC performance, and serve as a basis for future features based on EPTP switching.

Support for nested virt is not enabled.

This was tested using the following code within guest:

	#define VMX_VMFUNC ".byte 0x0f,0x01,0xd4"
	static void vmfunc(unsigned int nr, unsigned int ept)
	{
		asm volatile(VMX_VMFUNC
			     :
			     : "a"(nr), "c"(ept)
			     : "memory");
	}

VMFUNC instruction cost was measured at ~122 cycles. (Note: recent versions of gnu toolchain support the vmfunc instruction - removing the need for writing the bytecode manually).

Quoted 5 lines

---

I think I'd like to put this upstream so future eptp switching work can be implemented on top. Comments?

We have a different version in hand which is using separate EPTP. As you known, the patch will be more complex if using separate EPTP. And there are still lots of thing need to do in our version. We will send out for comments soon.

Best regards,
Yang

maintainer
#7 lore raw
Quoted 1 line · Yang Z

We have a different version in hand which is using separate EPTP.

Can you say in advance what you are using EPTP switching for? Offlist if necessary.

Paolo

maintainer
#9 lore raw
Quoted 5 lines · Wei W
Quoted 4 lines · Paolo Bonzini
Quoted 2 lines · Yang Z

We have a different version in hand which is using separate EPTP.

Quoted 8 lines
Quoted 2 lines

Can you say in advance what you are using EPTP switching for? Offlist if necessary.

Hi Paolo,

We are using EPTP switching for a protected inter-VM communication design, as shown in the slides (#23) here:
http://events.linuxfoundation.org/sites/events/files/slides/Jun_Nakajima_NFV_KVM%202015_final.pdf

[offlist, adding [email protected]]

If the EPTP switch is only adding extra data pages (e.g. mapping another guest's memory inside a PCI BAR), this can work.

However, slides 24 and 25 suggest that the executable pages change between the two EPTP views, similar to Jun's KVM Forum 2014 presentation. Michael and I explained in Seattle that this only works if the guest is trusted. I am a bit disappointed that Intel continued developing this feature without contacting us or without urging us to present our issues more formally.

I think I should make this very clear: I am not going to accept in KVM a feature that requires the guest to be trusted. A trusted guest kernel may make sense for other applications of VMFUNC (e.g. McAfee memory scan) but not for virtualization; if the guest is trusted, you don't have virtualization anymore.

Michael and I are going to present our findings to Intel soon. This will hopefully clarify why the guest has to be trusted. We will also present possible extensions to VMFUNC that enable its usage with untrusted guests, albeit only at CPL=0.

Asit Mallick is going to contact Jun about this so we can organize the meeting. Unfortunately it is going to be hard for everyone to attend since we have people in Europe, US and China, but we will share the slides.

Thanks,

Paolo