Skip to contentSkip to conversation

Nested virtualization for VMX

PATCH In review Qing He 5 participants 68 messages Sep 8, 2010 – Sep 20, 2010

lore
Share & tools
lore

Qing He

author
#0 lore raw

cover Nested virtualization for VMX

This patch set is the upgraded version of nested virtualization for VMX, that allows a VMX guest (L1) to run other VMX guests (L2).

The nested virtualization for vmx is built on homogeneous L1 and L2 for better performance and minimal emulation, the common code involved is small and contained in patch 03/16, for two flags, one for feature availability, the other for indicating current mode.

The userspace components (xend/xm/xl) is not included since Christopher's userspace patch has similar coverage. vEPT is not included as well because it's still WIP.

Major changes to last version:
- address Tim's comments on error handling and others
- split context switch into smaller pieces with certain restructure for better readability
- update interrupt handling, rewrite comments
- move cpuid into userspace
- etc.

The patch set includes the following patches.

[PATCH 01/16] vmx: nest: rename host_vmcs
[PATCH 02/16] vmx: nest: wrapper for control update [PATCH 03/16] vmx: nest: nested availability and status flags [PATCH 04/16] vmx: nest: nested control structure [PATCH 05/16] vmx: nest: virtual vmcs layout
[PATCH 06/16] vmx: nest: handling VMX instruction exits [PATCH 07/16] vmx: nest: switch current vmcs
[PATCH 08/16] vmx: nest: vmresume/vmlaunch
[PATCH 09/16] vmx: nest: shadow controls
[PATCH 10/16] vmx: nest: L1 <-> L2 context switch [PATCH 11/16] vmx: nest: interrupt handling
[PATCH 12/16] vmx: nest: VMExit handler in L2 [PATCH 13/16] vmx: nest: L2 tsc
[PATCH 14/16] vmx: nest: CR0.TS and #NM
[PATCH 15/16] vmx: nest: capability reporting MSRs [PATCH 16/16] vmx: nest: expose cpuid and CR4.VMXE

Thanks,
Qing He

author
#1 lore raw

PATCH 1/16 vmx: nest: rename host_vmcs

the VMCS region used for vmxon is named host_vmcs, which is somewhat misleading in nested virtualization context, rename it to vmxon_vmcs.

---

xen/arch/x86/hvm/vmx/vmcs.c +7 -7
diff -r d6a8d49f3526 xen/arch/x86/hvm/vmx/vmcs.c
@@ -67,7 +67,7 @@
 u64 vmx_ept_vpid_cap __read_mostly;
 bool_t cpu_has_vmx_ins_outs_instr_info __read_mostly;
 
-static DEFINE_PER_CPU_READ_MOSTLY(struct vmcs_struct *, host_vmcs);
+static DEFINE_PER_CPU_READ_MOSTLY(struct vmcs_struct *, vmxon_vmcs);
 static DEFINE_PER_CPU(struct vmcs_struct *, current_vmcs);
 static DEFINE_PER_CPU(struct list_head, active_vmcs_list);
 
@@ -427,11 +427,11 @@
 
 int vmx_cpu_up_prepare(unsigned int cpu)
 {
-    if ( per_cpu(host_vmcs, cpu) != NULL )
+    if ( per_cpu(vmxon_vmcs, cpu) != NULL )
         return 0;
 
-    per_cpu(host_vmcs, cpu) = vmx_alloc_vmcs();
-    if ( per_cpu(host_vmcs, cpu) != NULL )
+    per_cpu(vmxon_vmcs, cpu) = vmx_alloc_vmcs();
+    if ( per_cpu(vmxon_vmcs, cpu) != NULL )
         return 0;
 
     printk("CPU%d: Could not allocate host VMCS\n", cpu);
@@ -440,8 +440,8 @@
 
 void vmx_cpu_dead(unsigned int cpu)
 {
-    vmx_free_vmcs(per_cpu(host_vmcs, cpu));
-    per_cpu(host_vmcs, cpu) = NULL;
+    vmx_free_vmcs(per_cpu(vmxon_vmcs, cpu));
+    per_cpu(vmxon_vmcs, cpu) = NULL;
 }
 
 int vmx_cpu_up(void)
@@ -498,7 +498,7 @@
     if ( (rc = vmx_cpu_up_prepare(cpu)) != 0 )
         return rc;
 
-    switch ( __vmxon(virt_to_maddr(this_cpu(host_vmcs))) )
+    switch ( __vmxon(virt_to_maddr(this_cpu(vmxon_vmcs))) )
     {
     case -2: /* #UD or #GP */
         if ( bios_locked &&
Acked-by on v1 1/16
#2 lore raw

Re: [PATCH 01/16] vmx: nest: rename host_vmcs

Quoted 6 lines · On Wednesday 08 September 2010 17:22:09 Qing He wrote:

the VMCS region used for vmxon is named host_vmcs, which is somewhat misleading in nested virtualization context, rename it to vmxon_vmcs.

Quoted 54 lines

---

diff -r d6a8d49f3526 xen/arch/x86/hvm/vmx/vmcs.c

xen/arch/x86/hvm/vmx/vmcs.c +7 -7
@@ -67,7 +67,7 @@
 u64 vmx_ept_vpid_cap __read_mostly;
 bool_t cpu_has_vmx_ins_outs_instr_info __read_mostly;
 
-static DEFINE_PER_CPU_READ_MOSTLY(struct vmcs_struct *, host_vmcs);
+static DEFINE_PER_CPU_READ_MOSTLY(struct vmcs_struct *, vmxon_vmcs);
 static DEFINE_PER_CPU(struct vmcs_struct *, current_vmcs);
 static DEFINE_PER_CPU(struct list_head, active_vmcs_list);
 
@@ -427,11 +427,11 @@
 
 int vmx_cpu_up_prepare(unsigned int cpu)
 {
-    if ( per_cpu(host_vmcs, cpu) != NULL )
+    if ( per_cpu(vmxon_vmcs, cpu) != NULL )
         return 0;
 
-    per_cpu(host_vmcs, cpu) = vmx_alloc_vmcs();
-    if ( per_cpu(host_vmcs, cpu) != NULL )
+    per_cpu(vmxon_vmcs, cpu) = vmx_alloc_vmcs();
+    if ( per_cpu(vmxon_vmcs, cpu) != NULL )
         return 0;
 
     printk("CPU%d: Could not allocate host VMCS\n", cpu);
@@ -440,8 +440,8 @@
 
 void vmx_cpu_dead(unsigned int cpu)
 {
-    vmx_free_vmcs(per_cpu(host_vmcs, cpu));
-    per_cpu(host_vmcs, cpu) = NULL;
+    vmx_free_vmcs(per_cpu(vmxon_vmcs, cpu));
+    per_cpu(vmxon_vmcs, cpu) = NULL;
 }
 
 int vmx_cpu_up(void)
@@ -498,7 +498,7 @@
     if ( (rc = vmx_cpu_up_prepare(cpu)) != 0 )
         return rc;
 
-    switch ( __vmxon(virt_to_maddr(this_cpu(host_vmcs))) )
+    switch ( __vmxon(virt_to_maddr(this_cpu(vmxon_vmcs))) )
     {
     case -2: /* #UD or #GP */
         if ( bios_locked &&

_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel

Signature · 6 lines
---to satisfy European Law for business letters:
Advanced Micro Devices GmbH
Einsteinring 24, 85609 Dornach b. Muenchen
Geschaeftsfuehrer: Alberto Bozzo, Andrew Bowd
Sitz: Dornach, Gemeinde Aschheim, Landkreis Muenchen
Registergericht Muenchen, HRB Nr. 43632
author
#3 lore raw

PATCH 2/16 vmx: nest: wrapper for control update

In reply to Qing He

In nested virtualization, the L0 controls may not be the same with controls in physical VMCS.
Explict maintain guest controls in variables and use wrappers for control update, do not rely on physical control value.

---

xen/arch/x86/hvm/vmx/intr.c +2 -2
diff -r 905ca9cc0596 xen/arch/x86/hvm/vmx/intr.c
@@ -106,7 +106,7 @@
     if ( !(*cpu_exec_control & ctl) )
     {
         *cpu_exec_control |= ctl;
-        __vmwrite(CPU_BASED_VM_EXEC_CONTROL, *cpu_exec_control);
+        vmx_update_cpu_exec_control(v);
     }
 }
 
@@ -121,7 +121,7 @@
     if ( unlikely(v->arch.hvm_vcpu.single_step) )
     {
         v->arch.hvm_vmx.exec_control |= CPU_BASED_MONITOR_TRAP_FLAG;
-        __vmwrite(CPU_BASED_VM_EXEC_CONTROL, v->arch.hvm_vmx.exec_control);
+        vmx_update_cpu_exec_control(v);
         return;
     }
 
xen/arch/x86/hvm/vmx/vmcs.c +3 -3
diff -r 905ca9cc0596 xen/arch/x86/hvm/vmx/vmcs.c
@@ -839,10 +839,10 @@
     __vmwrite(VMCS_LINK_POINTER_HIGH, ~0UL);
 #endif
 
-    __vmwrite(EXCEPTION_BITMAP,
-              HVM_TRAP_MASK
+    v->arch.hvm_vmx.exception_bitmap = HVM_TRAP_MASK
               | (paging_mode_hap(d) ? 0 : (1U << TRAP_page_fault))
-              | (1U << TRAP_no_device));
+              | (1U << TRAP_no_device);
+    vmx_update_exception_bitmap(v);
 
     v->arch.hvm_vcpu.guest_cr[0] = X86_CR0_PE | X86_CR0_ET;
     hvm_update_guest_cr(v, 0);
xen/arch/x86/hvm/vmx/vmx.c +43 -29

183 lines not on this page · raw message

xen/include/asm-x86/hvm/vmx/vmcs.h +1 -0
diff -r 905ca9cc0596 xen/include/asm-x86/hvm/vmx/vmcs.h
@@ -97,6 +97,7 @@
     /* Cache of cpu execution control. */
     u32                  exec_control;
     u32                  secondary_exec_control;
+    u32                  exception_bitmap;
 
 #ifdef __x86_64__
     struct vmx_msr_state msr_state;
xen/include/asm-x86/hvm/vmx/vmx.h +3 -0
diff -r 905ca9cc0596 xen/include/asm-x86/hvm/vmx/vmx.h
@@ -60,6 +60,9 @@
 void vmx_vlapic_msr_changed(struct vcpu *v);
 void vmx_realmode(struct cpu_user_regs *regs);
 void vmx_update_debug_state(struct vcpu *v);
+void vmx_update_cpu_exec_control(struct vcpu *v);
+void vmx_update_secondary_exec_control(struct vcpu *v);
+void vmx_update_exception_bitmap(struct vcpu *v);
 
 /*
  * Exit Reasons
Acked-by on v1 2/16
#4 lore raw

Re: [PATCH 02/16] vmx: nest: wrapper for control update

Quoted 7 lines · On Wednesday 08 September 2010 17:22:10 Qing He wrote:

In nested virtualization, the L0 controls may not be the same with controls in physical VMCS.
Explict maintain guest controls in variables and use wrappers for control update, do not rely on physical control value.

Quoted 276 lines

---

diff -r 905ca9cc0596 xen/arch/x86/hvm/vmx/intr.c

xen/arch/x86/hvm/vmx/intr.c Thu Aug 05 15:32:24 2010 +0800 renamed from xen/arch/x86/hvm/vmx/intr.c Wed Aug 04 16:30:40 2010 +0800 +1 -2
@@ -106,7 +106,7 @@
     if ( !(*cpu_exec_control & ctl) )
     {
         *cpu_exec_control |= ctl;
-        __vmwrite(CPU_BASED_VM_EXEC_CONTROL, *cpu_exec_control);
+        vmx_update_cpu_exec_control(v);
     }
 }
 
@@ -121,7 +121,7 @@
     if ( unlikely(v->arch.hvm_vcpu.single_step) )
     {
         v->arch.hvm_vmx.exec_control |= CPU_BASED_MONITOR_TRAP_FLAG;
-        __vmwrite(CPU_BASED_VM_EXEC_CONTROL,
 v->arch.hvm_vmx.exec_control); +        vmx_update_cpu_exec_control(v);
         return;
     }

diff -r 905ca9cc0596 xen/arch/x86/hvm/vmx/vmcs.c

xen/arch/x86/hvm/vmx/vmcs.c Thu Aug 05 15:32:24 2010 +0800 renamed from xen/arch/x86/hvm/vmx/vmcs.c Wed Aug 04 16:30:40 2010 +0800 +3 -3
@@ -839,10 +839,10 @@
     __vmwrite(VMCS_LINK_POINTER_HIGH, ~0UL);
 #endif
 
-    __vmwrite(EXCEPTION_BITMAP,
-              HVM_TRAP_MASK
+    v->arch.hvm_vmx.exception_bitmap = HVM_TRAP_MASK
 
               | (paging_mode_hap(d) ? 0 : (1U << TRAP_page_fault))
 
-              | (1U << TRAP_no_device));
+              | (1U << TRAP_no_device);
+    vmx_update_exception_bitmap(v);
 
     v->arch.hvm_vcpu.guest_cr[0] = X86_CR0_PE | X86_CR0_ET;
     hvm_update_guest_cr(v, 0);

diff -r 905ca9cc0596 xen/arch/x86/hvm/vmx/vmx.c

xen/arch/x86/hvm/vmx/vmx.c Thu Aug 05 15:32:24 2010 +0800 renamed from xen/arch/x86/hvm/vmx/vmx.c Wed Aug 04 16:30:40 2010 +0800 +41 -29
@@ -385,6 +385,22 @@
 
 #endif /* __i386__ */
 
+void vmx_update_cpu_exec_control(struct vcpu *v)
+{
+    __vmwrite(CPU_BASED_VM_EXEC_CONTROL, v->arch.hvm_vmx.exec_control);
+}
+
+void vmx_update_secondary_exec_control(struct vcpu *v)
+{
+    __vmwrite(SECONDARY_VM_EXEC_CONTROL,
+              v->arch.hvm_vmx.secondary_exec_control);
+}
+
+void vmx_update_exception_bitmap(struct vcpu *v)
+{
+    __vmwrite(EXCEPTION_BITMAP, v->arch.hvm_vmx.exception_bitmap);
+}
+
 static int vmx_guest_x86_mode(struct vcpu *v)
 {
     unsigned int cs_ar_bytes;
@@ -408,7 +424,7 @@
     /* Clear the DR dirty flag and re-enable intercepts for DR accesses.
 */ v->arch.hvm_vcpu.flag_dr_dirty = 0;
     v->arch.hvm_vmx.exec_control |= CPU_BASED_MOV_DR_EXITING;
-    __vmwrite(CPU_BASED_VM_EXEC_CONTROL, v->arch.hvm_vmx.exec_control);
+    vmx_update_cpu_exec_control(v);
 
     v->arch.guest_context.debugreg[0] = read_debugreg(0);
     v->arch.guest_context.debugreg[1] = read_debugreg(1);
@@ -622,7 +638,8 @@
 static void vmx_fpu_enter(struct vcpu *v)
 {
     setup_fpu(v);
-    __vm_clear_bit(EXCEPTION_BITMAP, TRAP_no_device);
+    v->arch.hvm_vmx.exception_bitmap &= ~(1u << TRAP_no_device);
+    vmx_update_exception_bitmap(v);
     v->arch.hvm_vmx.host_cr0 &= ~X86_CR0_TS;
     __vmwrite(HOST_CR0, v->arch.hvm_vmx.host_cr0);
 }
@@ -648,7 +665,8 @@
     {
         v->arch.hvm_vcpu.hw_cr[0] |= X86_CR0_TS;
         __vmwrite(GUEST_CR0, v->arch.hvm_vcpu.hw_cr[0]);
-        __vm_set_bit(EXCEPTION_BITMAP, TRAP_no_device);
+        v->arch.hvm_vmx.exception_bitmap |= (1u << TRAP_no_device);
+        vmx_update_exception_bitmap(v);
     }
 }
 
@@ -954,7 +972,7 @@
     v->arch.hvm_vmx.exec_control &= ~CPU_BASED_RDTSC_EXITING;
     if ( enable )
         v->arch.hvm_vmx.exec_control |= CPU_BASED_RDTSC_EXITING;
-    __vmwrite(CPU_BASED_VM_EXEC_CONTROL, v->arch.hvm_vmx.exec_control);
+    vmx_update_cpu_exec_control(v);
     vmx_vmcs_exit(v);
 }
 
@@ -1047,7 +1065,7 @@
 
 void vmx_update_debug_state(struct vcpu *v)
 {
-    unsigned long intercepts, mask;
+    unsigned long mask;
 
     ASSERT(v == current);
 
@@ -1055,12 +1073,11 @@
     if ( !cpu_has_monitor_trap_flag )
         mask |= 1u << TRAP_debug;
 
-    intercepts = __vmread(EXCEPTION_BITMAP);
     if ( v->arch.hvm_vcpu.debug_state_latch )
-        intercepts |= mask;
+        v->arch.hvm_vmx.exception_bitmap |= mask;
     else
-        intercepts &= ~mask;
-    __vmwrite(EXCEPTION_BITMAP, intercepts);
+        v->arch.hvm_vmx.exception_bitmap &= ~mask;
+    vmx_update_exception_bitmap(v);
 }
 
 static void vmx_update_guest_cr(struct vcpu *v, unsigned int cr)
@@ -1087,7 +1104,7 @@
             v->arch.hvm_vmx.exec_control &= ~cr3_ctls;
             if ( !hvm_paging_enabled(v) )
                 v->arch.hvm_vmx.exec_control |= cr3_ctls;
-            __vmwrite(CPU_BASED_VM_EXEC_CONTROL,
 v->arch.hvm_vmx.exec_control); +            vmx_update_cpu_exec_control(v);
 
             /* Changing CR0.PE can change some bits in real CR4. */
             vmx_update_guest_cr(v, 4);
@@ -1122,7 +1139,8 @@
                     vmx_set_segment_register(v, s, &reg[s]);
                 v->arch.hvm_vcpu.hw_cr[4] |= X86_CR4_VME;
                 __vmwrite(GUEST_CR4, v->arch.hvm_vcpu.hw_cr[4]);
-                __vmwrite(EXCEPTION_BITMAP, 0xffffffff);
+                v->arch.hvm_vmx.exception_bitmap = 0xffffffff;
+                vmx_update_exception_bitmap(v);
             }
             else
             {
@@ -1134,11 +1152,11 @@
                     ((v->arch.hvm_vcpu.hw_cr[4] & ~X86_CR4_VME)
 
                      |(v->arch.hvm_vcpu.guest_cr[4] & X86_CR4_VME));
 
                 __vmwrite(GUEST_CR4, v->arch.hvm_vcpu.hw_cr[4]);
-                __vmwrite(EXCEPTION_BITMAP,
-                          HVM_TRAP_MASK
+                v->arch.hvm_vmx.exception_bitmap = HVM_TRAP_MASK
 
                           | (paging_mode_hap(v->domain) ?
 
                              0 : (1U << TRAP_page_fault))
-                          | (1U << TRAP_no_device));
+                          | (1U << TRAP_no_device);
+                vmx_update_exception_bitmap(v);
                 vmx_update_debug_state(v);
             }
         }
@@ -1544,7 +1562,7 @@
 
     /* Allow guest direct access to DR registers */
     v->arch.hvm_vmx.exec_control &= ~CPU_BASED_MOV_DR_EXITING;
-    __vmwrite(CPU_BASED_VM_EXEC_CONTROL, v->arch.hvm_vmx.exec_control);
+    vmx_update_cpu_exec_control(v);
 }
 
 static void vmx_invlpg_intercept(unsigned long vaddr)
@@ -1928,18 +1946,18 @@
 void vmx_vlapic_msr_changed(struct vcpu *v)
 {
     struct vlapic *vlapic = vcpu_vlapic(v);
-    uint32_t ctl;
 
     if ( !cpu_has_vmx_virtualize_apic_accesses )
         return;
 
     vmx_vmcs_enter(v);
-    ctl  = __vmread(SECONDARY_VM_EXEC_CONTROL);
-    ctl &= ~SECONDARY_EXEC_VIRTUALIZE_APIC_ACCESSES;
+    v->arch.hvm_vmx.secondary_exec_control
+        &= ~SECONDARY_EXEC_VIRTUALIZE_APIC_ACCESSES;
     if ( !vlapic_hw_disabled(vlapic) &&
          (vlapic_base_address(vlapic) == APIC_DEFAULT_PHYS_BASE) )
-        ctl |= SECONDARY_EXEC_VIRTUALIZE_APIC_ACCESSES;
-    __vmwrite(SECONDARY_VM_EXEC_CONTROL, ctl);
+        v->arch.hvm_vmx.secondary_exec_control
+            |= SECONDARY_EXEC_VIRTUALIZE_APIC_ACCESSES;
+    vmx_update_secondary_exec_control(v);
     vmx_vmcs_exit(v);
 }
 
@@ -2469,14 +2487,12 @@
     case EXIT_REASON_PENDING_VIRT_INTR:
         /* Disable the interrupt window. */
         v->arch.hvm_vmx.exec_control &= ~CPU_BASED_VIRTUAL_INTR_PENDING;
-        __vmwrite(CPU_BASED_VM_EXEC_CONTROL,
-                  v->arch.hvm_vmx.exec_control);
+        vmx_update_cpu_exec_control(v);
         break;
     case EXIT_REASON_PENDING_VIRT_NMI:
         /* Disable the NMI window. */
         v->arch.hvm_vmx.exec_control &= ~CPU_BASED_VIRTUAL_NMI_PENDING;
-        __vmwrite(CPU_BASED_VM_EXEC_CONTROL,
-                  v->arch.hvm_vmx.exec_control);
+        vmx_update_cpu_exec_control(v);
         break;
     case EXIT_REASON_TASK_SWITCH: {
         const enum hvm_task_switch_reason reasons[] = {
@@ -2627,7 +2643,7 @@
 
     case EXIT_REASON_MONITOR_TRAP_FLAG:
         v->arch.hvm_vmx.exec_control &= ~CPU_BASED_MONITOR_TRAP_FLAG;
-        __vmwrite(CPU_BASED_VM_EXEC_CONTROL,
 v->arch.hvm_vmx.exec_control); +        vmx_update_cpu_exec_control(v);
         if ( v->domain->debugger_attached && v->arch.hvm_vcpu.single_step
 ) domain_pause_for_debugger();
         break;
@@ -2677,16 +2693,14 @@
             /* VPID was disabled: now enabled. */
             curr->arch.hvm_vmx.secondary_exec_control |=
                 SECONDARY_EXEC_ENABLE_VPID;
-            __vmwrite(SECONDARY_VM_EXEC_CONTROL,
-                      curr->arch.hvm_vmx.secondary_exec_control);
+            vmx_update_secondary_exec_control(curr);
         }
         else if ( old_asid && !new_asid )
         {
             /* VPID was enabled: now disabled. */
             curr->arch.hvm_vmx.secondary_exec_control &=
                 ~SECONDARY_EXEC_ENABLE_VPID;
-            __vmwrite(SECONDARY_VM_EXEC_CONTROL,
-                      curr->arch.hvm_vmx.secondary_exec_control);
+            vmx_update_secondary_exec_control(curr);
         }
     }
 

diff -r 905ca9cc0596 xen/include/asm-x86/hvm/vmx/vmcs.h
--- a/xen/include/asm-x86/hvm/vmx/vmcs.h Wed Aug 04 16:30:40 2010
+0800 +++ b/xen/include/asm-x86/hvm/vmx/vmcs.h Thu Aug 05 15:32:24
2010 +0800 @@ -97,6 +97,7 @@

     /* Cache of cpu execution control. */
     u32                  exec_control;
     u32                  secondary_exec_control;

+ u32 exception_bitmap;

 #ifdef __x86_64__
     struct vmx_msr_state msr_state;

diff -r 905ca9cc0596 xen/include/asm-x86/hvm/vmx/vmx.h

xen/include/asm-x86/hvm/vmx/vmx.h Thu Aug 05 15:32:24 2010 +0800 renamed from xen/include/asm-x86/hvm/vmx/vmx.h Wed Aug 04 16:30:40 2010 +0800 +3 -0
@@ -60,6 +60,9 @@
 void vmx_vlapic_msr_changed(struct vcpu *v);
 void vmx_realmode(struct cpu_user_regs *regs);
 void vmx_update_debug_state(struct vcpu *v);
+void vmx_update_cpu_exec_control(struct vcpu *v);
+void vmx_update_secondary_exec_control(struct vcpu *v);
+void vmx_update_exception_bitmap(struct vcpu *v);
 
 /*
  * Exit Reasons

_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel

Signature · 6 lines
---to satisfy European Law for business letters:
Advanced Micro Devices GmbH
Einsteinring 24, 85609 Dornach b. Muenchen
Geschaeftsfuehrer: Alberto Bozzo, Andrew Bowd
Sitz: Dornach, Gemeinde Aschheim, Landkreis Muenchen
Registergericht Muenchen, HRB Nr. 43632
author
#5 lore raw

PATCH 3/16 vmx: nest: nested availability and status flags

In reply to Qing He

These are the vendor neutral availability and status flags of nested virtualization.

The availability hvm parameter can be used to disable all reporting and functions of nested, improving guest security in certain circumstances.

The per vcpu flag in_nesting is used to indicate fundamental status:
the current mode.

---

xen/include/asm-x86/hvm/hvm.h +4 -0
diff -r 11c98ab76326 xen/include/asm-x86/hvm/hvm.h
@@ -250,6 +250,10 @@
 #define is_viridian_domain(_d)                                             \
  (is_hvm_domain(_d) && ((_d)->arch.hvm_domain.params[HVM_PARAM_VIRIDIAN]))
 
+#define is_nested_avail(_d)                                                \
+ (is_hvm_domain(_d) && ((_d)->arch.hvm_domain.params[HVM_PARAM_NESTEDHVM]))
+
+
 void hvm_cpuid(unsigned int input, unsigned int *eax, unsigned int *ebx,
                                    unsigned int *ecx, unsigned int *edx);
 void hvm_migrate_timers(struct vcpu *v);
xen/include/asm-x86/hvm/vcpu.h +2 -0
diff -r 11c98ab76326 xen/include/asm-x86/hvm/vcpu.h
@@ -71,6 +71,8 @@
     bool_t              debug_state_latch;
     bool_t              single_step;
 
+    bool_t              in_nesting;
+
     u64                 asid_generation;
     u32                 asid;
 
xen/include/public/hvm/params.h +4 -1
diff -r 11c98ab76326 xen/include/public/hvm/params.h
@@ -113,6 +113,9 @@
 #define HVM_PARAM_CONSOLE_PFN    17
 #define HVM_PARAM_CONSOLE_EVTCHN 18
 
-#define HVM_NR_PARAMS          19
+/* Boolean: Enable nested virtualization (hvm only) */
+#define HVM_PARAM_NESTEDHVM    19
+
+#define HVM_NR_PARAMS          20
 
 #endif /* __XEN_PUBLIC_HVM_PARAMS_H__ */
#6 lore raw

Re: [PATCH 03/16] vmx: nest: nested availability and status flags

Quoted 21 lines · On Wednesday 08 September 2010 17:22:11 Qing He wrote:

These are the vendor neutral availability and status flags of nested virtualization.

The availability hvm parameter can be used to disable all reporting and functions of nested, improving guest security in certain circumstances.

The per vcpu flag in_nesting is used to indicate fundamental status:
the current mode.

---
diff -r 11c98ab76326 xen/include/asm-x86/hvm/hvm.h

xen/include/asm-x86/hvm/hvm.h +1 -0
@@ -250,6 +250,10 @@
 #define is_viridian_domain(_d)                                            
 \ (is_hvm_domain(_d) && ((_d)->arch.hvm_domain.params[HVM_PARAM_VIRIDIAN]))
 
+#define is_nested_avail(_d)                                               
Quoted 3 lines

\ + (is_hvm_domain(_d) && ((_d)->arch.hvm_domain.params[HVM_PARAM_NESTEDHVM])) + +

That is functional equal to nestedhvm_enabled() in my patch series. The is_hvm_domain() check is not necessary. The tools patch check that nestedhvm is for hvm guests only.

xen/include/asm-x86/hvm/vcpu.h line 74

Quoted 11 lines

void hvm_cpuid(unsigned int input, unsigned int *eax, unsigned int *ebx,

                                    unsigned int *ecx, unsigned int *edx);

void hvm_migrate_timers(struct vcpu *v);
diff -r 11c98ab76326 xen/include/asm-x86/hvm/vcpu.h

xen/include/asm-x86/hvm/vcpu.h +1 -0
@@ -71,6 +71,8 @@
     bool_t              debug_state_latch;
     bool_t              single_step;
 
+    bool_t              in_nesting;

This is functional equal to nestedhvm_vcpu_in_guestmode() in my patch series.

Quoted 8 lines
xen/include/asm-x86/hvm/vcpu.h +1 -0
+
     u64                 asid_generation;
     u32                 asid;
 

diff -r 11c98ab76326 xen/include/public/hvm/params.h

xen/include/public/hvm/params.h line 121

Quoted 10 lines
xen/include/public/hvm/params.h +4 -1
@@ -113,6 +113,9 @@
 #define HVM_PARAM_CONSOLE_PFN    17
 #define HVM_PARAM_CONSOLE_EVTCHN 18
 
-#define HVM_NR_PARAMS          19
+/* Boolean: Enable nested virtualization (hvm only) */
+#define HVM_PARAM_NESTEDHVM    19
+
+#define HVM_NR_PARAMS          20
 
 #endif /* __XEN_PUBLIC_HVM_PARAMS_H__ */

I already have this part in my tools patch.

Christoph

Signature · 6 lines
---to satisfy European Law for business letters:
Advanced Micro Devices GmbH
Einsteinring 24, 85609 Dornach b. Muenchen
Geschaeftsfuehrer: Alberto Bozzo, Andrew Bowd
Sitz: Dornach, Gemeinde Aschheim, Landkreis Muenchen
Registergericht Muenchen, HRB Nr. 43632
#7 lore raw

RE: [PATCH 03/16] vmx: nest: nested availability and status flags

Quoted 61 lines · Christoph Egger
Quoted 21 lines · On Wednesday 08 September 2010 17:22:11 Qing He wrote:

These are the vendor neutral availability and status flags of nested virtualization.

The availability hvm parameter can be used to disable all reporting and functions of nested, improving guest security in certain circumstances.

The per vcpu flag in_nesting is used to indicate fundamental status:
the current mode.

---
diff -r 11c98ab76326 xen/include/asm-x86/hvm/hvm.h

xen/include/asm-x86/hvm/hvm.h +0 -0
 @@ -250,6 +250,10 @@ #define is_viridian_domain(_d)

\ (is_hvm_domain(_d) && ((_d)->arch.hvm_domain.params[HVM_PARAM_VIRIDIAN]))

Quoted 4 lines

+#define is_nested_avail(_d)
\ + (is_hvm_domain(_d) && ((_d)->arch.hvm_domain.params[HVM_PARAM_NESTEDHVM])) + +

That is functional equal to nestedhvm_enabled() in my patch series. The is_hvm_domain() check is not necessary. The tools patch check that nestedhvm is for hvm guests only.

xen/include/asm-x86/hvm/vcpu.h line 74

Quoted 10 lines

void hvm_cpuid(unsigned int input, unsigned int *eax, unsigned int
*ebx, unsigned int *ecx,
unsigned int *edx); void hvm_migrate_timers(struct vcpu *v);
diff -r 11c98ab76326 xen/include/asm-x86/hvm/vcpu.h

xen/include/asm-x86/hvm/vcpu.h +1 -0
     @@ -71,6 +71,8 @@ bool_t              debug_state_latch;
     bool_t              single_step;
 
+    bool_t              in_nesting;

This is functional equal to nestedhvm_vcpu_in_guestmode() in my patch series.

Quoted 16 lines

+

     u64                 asid_generation;
     u32                 asid;

diff -r 11c98ab76326 xen/include/public/hvm/params.h

xen/include/public/hvm/params.h +4 -1
 @@ -113,6 +113,9 @@ #define HVM_PARAM_CONSOLE_PFN    17
 #define HVM_PARAM_CONSOLE_EVTCHN 18
 
-#define HVM_NR_PARAMS          19
+/* Boolean: Enable nested virtualization (hvm only) */
+#define HVM_PARAM_NESTEDHVM    19
+
+#define HVM_NR_PARAMS          20
Quoted 5 lines
Quoted 1 line

#endif /* __XEN_PUBLIC_HVM_PARAMS_H__ */

I already have this part in my tools patch.

Christoph

This part is one of the consense one, we can merge one day in future.

author
#8 lore raw

PATCH 4/16 vmx: nest: nested control structure

In reply to Qing He

v->arch.hvm_vmx.nest as control structure

---

xen/include/asm-x86/hvm/vmx/nest.h new file +45 -0
diff -r fc4de5eedd1d xen/include/asm-x86/hvm/vmx/nest.h
@@ -0,0 +1,45 @@
+/*
+ * nest.h: nested virtualization for VMX.
+ *
+ * Copyright (c) 2010, Intel Corporation.
+ * Author: Qing He <[email protected]>
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms and conditions of the GNU General Public License,
+ * version 2, as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope it will be useful, but WITHOUT
+ * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
+ * FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License for
+ * more details.
+ *
+ * You should have received a copy of the GNU General Public License along with
+ * this program; if not, write to the Free Software Foundation, Inc., 59 Temple
+ * Place - Suite 330, Boston, MA 02111-1307 USA.
+ *
+ */
+#ifndef __ASM_X86_HVM_NEST_H__
+#define __ASM_X86_HVM_NEST_H__
+
+struct vmcs_struct;
+
+struct vmx_nest_struct {
+    paddr_t              guest_vmxon_pa;
+
+    /* Saved host vmcs for vcpu itself */
+    struct vmcs_struct  *hvmcs;
+
+    /*
+     * Guest's `current vmcs' of vcpu
+     *  - gvmcs_pa: guest VMCS region physical address
+     *  - vvmcs:    (guest) virtual vmcs
+     *  - svmcs:    effective vmcs for the guest of this vcpu
+     *  - valid:    launch state: invalid on clear, valid on ld
+     */
+    paddr_t              gvmcs_pa;
+    void                *vvmcs;
+    struct vmcs_struct  *svmcs;
+    int                  vmcs_valid;
+};
+
+#endif /* __ASM_X86_HVM_NEST_H__ */
xen/include/asm-x86/hvm/vmx/vmcs.h +4 -0
diff -r fc4de5eedd1d xen/include/asm-x86/hvm/vmx/vmcs.h
@@ -22,6 +22,7 @@
 #include <asm/config.h>
 #include <asm/hvm/io.h>
 #include <asm/hvm/vpmu.h>
+#include <asm/hvm/vmx/nest.h>
 
 extern void vmcs_dump_vcpu(struct vcpu *v);
 extern void setup_vmcs_dump(void);
@@ -99,6 +100,9 @@
     u32                  secondary_exec_control;
     u32                  exception_bitmap;
 
+    /* nested virtualization */
+    struct vmx_nest_struct nest;
+
 #ifdef __x86_64__
     struct vmx_msr_state msr_state;
     unsigned long        shadow_gs;
#9 lore raw

RE: [PATCH 04/16] vmx: nest: nested control structure

Quoted 77 lines · Qing He

v->arch.hvm_vmx.nest as control structure

---
diff -r fc4de5eedd1d xen/include/asm-x86/hvm/vmx/nest.h

xen/include/asm-x86/hvm/vmx/nest.h new file +8 -0
+0800 @@ -0,0 +1,45 @@
+/*
+ * nest.h: nested virtualization for VMX.
+ *
+ * Copyright (c) 2010, Intel Corporation.
+ * Author: Qing He <[email protected]>
+ *
+ * This program is free software; you can redistribute it and/or

modify it + * under the terms and conditions of the GNU General Public License, + * version 2, as published by the Free Software Foundation. + *
+ * This program is distributed in the hope it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details.
+ *
+ * You should have received a copy of the GNU General Public License along with + * this program; if not, write to the Free Software Foundation, Inc., 59 Temple + * Place - Suite 330, Boston, MA 02111-1307 USA. + *
+ */

xen/include/asm-x86/hvm/vmx/nest.h new file +25 -0
+#ifndef __ASM_X86_HVM_NEST_H__
+#define __ASM_X86_HVM_NEST_H__
+
+struct vmcs_struct;
+
+struct vmx_nest_struct {
+    paddr_t              guest_vmxon_pa;
+
+    /* Saved host vmcs for vcpu itself */
+    struct vmcs_struct  *hvmcs;
+
+    /*
+     * Guest's `current vmcs' of vcpu
+     *  - gvmcs_pa: guest VMCS region physical address
+     *  - vvmcs:    (guest) virtual vmcs
+     *  - svmcs:    effective vmcs for the guest of this vcpu
+     *  - valid:    launch state: invalid on clear, valid on ld
+     */
+    paddr_t              gvmcs_pa;
+    void                *vvmcs;
+    struct vmcs_struct  *svmcs;
+    int                  vmcs_valid;
+};
+
+#endif /* __ASM_X86_HVM_NEST_H__ */

diff -r fc4de5eedd1d xen/include/asm-x86/hvm/vmx/vmcs.h
--- a/xen/include/asm-x86/hvm/vmx/vmcs.h Wed Sep 08 21:00:00 2010
+0800 +++ b/xen/include/asm-x86/hvm/vmx/vmcs.h Wed Sep 08 21:03:41
2010 +0800 @@ -22,6 +22,7 @@
#include <asm/config.h>
#include <asm/hvm/io.h>
#include <asm/hvm/vpmu.h>

xen/include/asm-x86/hvm/vmx/nest.h new file +4 -0
+#include <asm/hvm/vmx/nest.h>
 
 extern void vmcs_dump_vcpu(struct vcpu *v);
 extern void setup_vmcs_dump(void);
@@ -99,6 +100,9 @@
     u32                  secondary_exec_control;
     u32                  exception_bitmap;
 
+    /* nested virtualization */
+    struct vmx_nest_struct nest;
+
 #ifdef __x86_64__
     struct vmx_msr_state msr_state;
     unsigned long        shadow_gs;
Quoted 4 lines

_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel

Hi,

This one remind me something when I was reviewing other VMM side patch.
The terminology here is pretty hard. One naming solution from nested virtualization layer point of view is to use subscript (IBM guys uses this in their to-be-coming OSDI paper as well). Here is an example:

L2 guest
L1 guest (VMM)
L0 VMM

vmcs01 is the VMCS that L0 VMM uses for L1 guest, like vmcs in single layer virtualization. (hvmcs in current patch)
vmcs12 is the VMCS that L1 VMM uses for L2 guest. But this vmcs12 is visible to L0 VMM. (vvmcs in current patch)
vmcs02 is the VMCS that L0 VMM uses for L2 guest when L2 guest is executing. It is a kind of shadow of vmcs12. (svmcs in current patch)

Will this be more clear? Appreciate for your comments (except Qing and me).

Thx, Eddie

#10 lore raw

Re: [PATCH 04/16] vmx: nest: nested control structure

In reply to Qing He
Quoted 30 lines · On Wednesday 08 September 2010 17:22:12 Qing He wrote:

v->arch.hvm_vmx.nest as control structure

---
diff -r fc4de5eedd1d xen/include/asm-x86/hvm/vmx/nest.h

xen/include/asm-x86/hvm/vmx/nest.h new file +13 -0
@@ -0,0 +1,45 @@
+/*
+ * nest.h: nested virtualization for VMX.
+ *
+ * Copyright (c) 2010, Intel Corporation.
+ * Author: Qing He <[email protected]>
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms and conditions of the GNU General Public License,
+ * version 2, as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope it will be useful, but WITHOUT
+ * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
+ * FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License

for + * more details.
+ *
+ * You should have received a copy of the GNU General Public License along with + * this program; if not, write to the Free Software Foundation, Inc., 59 Temple + * Place - Suite 330, Boston, MA 02111-1307 USA.
+ *
+ */

xen/include/asm-x86/hvm/vmx/nest.h line 26

Quoted 6 lines
xen/include/asm-x86/hvm/vmx/nest.h new file +6 -0
+#ifndef __ASM_X86_HVM_NEST_H__
+#define __ASM_X86_HVM_NEST_H__
+
+struct vmcs_struct;
+
+struct vmx_nest_struct {

Is it ok to name it 'struct nestedvmx' ?

Quoted 31 lines
xen/include/asm-x86/hvm/vmx/nest.h new file +19 -0
+    paddr_t              guest_vmxon_pa;
+
+    /* Saved host vmcs for vcpu itself */
+    struct vmcs_struct  *hvmcs;
+
+    /*
+     * Guest's `current vmcs' of vcpu
+     *  - gvmcs_pa: guest VMCS region physical address
+     *  - vvmcs:    (guest) virtual vmcs
+     *  - svmcs:    effective vmcs for the guest of this vcpu
+     *  - valid:    launch state: invalid on clear, valid on ld
+     */
+    paddr_t              gvmcs_pa;
+    void                *vvmcs;
+    struct vmcs_struct  *svmcs;
+    int                  vmcs_valid;
+};
+
+#endif /* __ASM_X86_HVM_NEST_H__ */

diff -r fc4de5eedd1d xen/include/asm-x86/hvm/vmx/vmcs.h

xen/include/asm-x86/hvm/vmx/vmcs.h +1 -0
@@ -22,6 +22,7 @@
 #include <asm/config.h>
 #include <asm/hvm/io.h>
 #include <asm/hvm/vpmu.h>
+#include <asm/hvm/vmx/nest.h>
 
 extern void vmcs_dump_vcpu(struct vcpu *v);
 extern void setup_vmcs_dump(void);

xen/include/asm-x86/hvm/vmx/vmcs.h line 108

Quoted 9 lines
xen/include/asm-x86/hvm/vmx/vmcs.h +3 -0
@@ -99,6 +100,9 @@
     u32                  secondary_exec_control;
     u32                  exception_bitmap;
 
+    /* nested virtualization */
+    struct vmx_nest_struct nest;
+
 #ifdef __x86_64__
     struct vmx_msr_state msr_state;
     unsigned long        shadow_gs;

I think, the structure should be allocated in the nestedhvm_vcpu_initialise() function hook and assigned to the nh_arch pointer in struct nestedhvm.

Christoph

Signature · 6 lines
---to satisfy European Law for business letters:
Advanced Micro Devices GmbH
Einsteinring 24, 85609 Dornach b. Muenchen
Geschaeftsfuehrer: Alberto Bozzo, Andrew Bowd
Sitz: Dornach, Gemeinde Aschheim, Landkreis Muenchen
Registergericht Muenchen, HRB Nr. 43632
#11 lore raw

RE: [PATCH 04/16] vmx: nest: nested control structure

Quoted 27 lines · Christoph Egger
Quoted 26 lines · On Wednesday 08 September 2010 17:22:12 Qing He wrote:

v->arch.hvm_vmx.nest as control structure

---
diff -r fc4de5eedd1d xen/include/asm-x86/hvm/vmx/nest.h

xen/include/asm-x86/hvm/vmx/nest.h new file +7 -0
+0800 @@ -0,0 +1,45 @@ +/*
+ * nest.h: nested virtualization for VMX.
+ *
+ * Copyright (c) 2010, Intel Corporation.
+ * Author: Qing He <[email protected]>
+ *
+ * This program is free software; you can redistribute it and/or

modify it + * under the terms and conditions of the GNU General Public License, + * version 2, as published by the Free Software Foundation. + * + * This program is distributed in the hope it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + *
+ * You should have received a copy of the GNU General Public License along with + * this program; if not, write to the Free Software Foundation, Inc., 59 Temple + * Place - Suite 330, Boston, MA 02111-1307 USA. + * + */

Quoted 8 lines

xen/include/asm-x86/hvm/vmx/nest.h line 26

Quoted 6 lines
xen/include/asm-x86/hvm/vmx/nest.h new file +6 -0
+#ifndef __ASM_X86_HVM_NEST_H__
+#define __ASM_X86_HVM_NEST_H__
+
+struct vmcs_struct;
+
+struct vmx_nest_struct {

Is it ok to name it 'struct nestedvmx' ?

Fine, renamed to nested_vmx.

Quoted 33 lines
Quoted 33 lines
xen/include/asm-x86/hvm/vmx/nest.h new file +11 -0
+    paddr_t              guest_vmxon_pa;
+
+    /* Saved host vmcs for vcpu itself */
+    struct vmcs_struct  *hvmcs;
+
+    /*
+     * Guest's `current vmcs' of vcpu
+     *  - gvmcs_pa: guest VMCS region physical address
+     *  - vvmcs:    (guest) virtual vmcs
+     *  - svmcs:    effective vmcs for the guest of this vcpu
+     *  - valid:    launch state: invalid on clear, valid on ld +  

*/ + paddr_t gvmcs_pa;

xen/include/asm-x86/hvm/vmx/nest.h new file +6 -0
+    void                *vvmcs;
+    struct vmcs_struct  *svmcs;
+    int                  vmcs_valid;
+};
+
+#endif /* __ASM_X86_HVM_NEST_H__ */

diff -r fc4de5eedd1d xen/include/asm-x86/hvm/vmx/vmcs.h
--- a/xen/include/asm-x86/hvm/vmx/vmcs.h Wed Sep 08 21:00:00 2010
+0800 +++ b/xen/include/asm-x86/hvm/vmx/vmcs.h Wed Sep 08 21:03:41
2010 +0800 @@ -22,6 +22,7 @@ #include <asm/config.h> #include <asm/hvm/io.h>
#include <asm/hvm/vpmu.h>

xen/include/asm-x86/hvm/vmx/nest.h new file +1 -0
+#include <asm/hvm/vmx/nest.h>
 
 extern void vmcs_dump_vcpu(struct vcpu *v);
 extern void setup_vmcs_dump(void);
@@ -99,6 +100,9 @@
     u32                  secondary_exec_control;
     u32                  exception_bitmap;
 
Quoted 11 lines

xen/include/asm-x86/hvm/vmx/nest.h line 108

Quoted 6 lines
xen/include/asm-x86/hvm/vmx/nest.h new file +3 -0
+    /* nested virtualization */
+    struct vmx_nest_struct nest;
+
 #ifdef __x86_64__
     struct vmx_msr_state msr_state;
     unsigned long        shadow_gs;

I think, the structure should be allocated in the nestedhvm_vcpu_initialise() function hook and assigned to the nh_arch pointer in struct nestedhvm.

Well, the structure itself is pretty small, so dynamic allocation is really not a good idea. Instead, the internal field such as vvmcs/svmcs are big and thus we use a pointer, but they are allocated on demand. This follows the style of arch_vmx_struct in vcpu data structure.

I am fine with your nestedhvm_vcpu_initialise "design", but VMX doesn't need to use the wrapper so far.

Thx, Eddie

#12 lore raw

Re: [PATCH 04/16] vmx: nest: nested control structure

Quoted 82 lines · Eddie
Quoted 27 lines · Christoph Egger
Quoted 26 lines · On Wednesday 08 September 2010 17:22:12 Qing He wrote:

v->arch.hvm_vmx.nest as control structure

---
diff -r fc4de5eedd1d xen/include/asm-x86/hvm/vmx/nest.h

xen/include/asm-x86/hvm/vmx/nest.h new file +7 -0
+0800 @@ -0,0 +1,45 @@ +/*
+ * nest.h: nested virtualization for VMX.
+ *
+ * Copyright (c) 2010, Intel Corporation.
+ * Author: Qing He <[email protected]>
+ *
+ * This program is free software; you can redistribute it and/or

modify it + * under the terms and conditions of the GNU General Public License, + * version 2, as published by the Free Software Foundation. + * + * This program is distributed in the hope it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + *
+ * You should have received a copy of the GNU General Public License along with + * this program; if not, write to the Free Software Foundation, Inc., 59 Temple + * Place - Suite 330, Boston, MA 02111-1307 USA. + * + */

Quoted 8 lines

xen/include/asm-x86/hvm/vmx/nest.h line 26

Quoted 6 lines
xen/include/asm-x86/hvm/vmx/nest.h new file +6 -0
+#ifndef __ASM_X86_HVM_NEST_H__
+#define __ASM_X86_HVM_NEST_H__
+
+struct vmcs_struct;
+
+struct vmx_nest_struct {

Is it ok to name it 'struct nestedvmx' ?

Fine, renamed to nested_vmx.

Quoted 32 lines
Quoted 32 lines
xen/include/asm-x86/hvm/vmx/nest.h new file +11 -0
+    paddr_t              guest_vmxon_pa;
+
+    /* Saved host vmcs for vcpu itself */
+    struct vmcs_struct  *hvmcs;
+
+    /*
+     * Guest's `current vmcs' of vcpu
+     *  - gvmcs_pa: guest VMCS region physical address
+     *  - vvmcs:    (guest) virtual vmcs
+     *  - svmcs:    effective vmcs for the guest of this vcpu
+     *  - valid:    launch state: invalid on clear, valid on ld +

*/ + paddr_t gvmcs_pa;

xen/include/asm-x86/hvm/vmx/nest.h new file +6 -0
+    void                *vvmcs;
+    struct vmcs_struct  *svmcs;
+    int                  vmcs_valid;
+};
+
+#endif /* __ASM_X86_HVM_NEST_H__ */

diff -r fc4de5eedd1d xen/include/asm-x86/hvm/vmx/vmcs.h
--- a/xen/include/asm-x86/hvm/vmx/vmcs.h Wed Sep 08 21:00:00 2010
+0800 +++ b/xen/include/asm-x86/hvm/vmx/vmcs.h Wed Sep 08 21:03:41
2010 +0800 @@ -22,6 +22,7 @@ #include <asm/config.h> #include <asm/hvm/io.h>
#include <asm/hvm/vpmu.h>

xen/include/asm-x86/hvm/vmx/nest.h new file +1 -0
+#include <asm/hvm/vmx/nest.h>
 
 extern void vmcs_dump_vcpu(struct vcpu *v);
 extern void setup_vmcs_dump(void);
@@ -99,6 +100,9 @@
     u32                  secondary_exec_control;
     u32                  exception_bitmap;
 
Quoted 10 lines

xen/include/asm-x86/hvm/vmx/nest.h line 108

Quoted 6 lines
xen/include/asm-x86/hvm/vmx/nest.h new file +3 -0
+    /* nested virtualization */
+    struct vmx_nest_struct nest;
+
 #ifdef __x86_64__
     struct vmx_msr_state msr_state;
     unsigned long        shadow_gs;

I think, the structure should be allocated in the nestedhvm_vcpu_initialise() function hook and assigned to the nh_arch pointer in struct nestedhvm.

Quoted 7 lines

Well, the structure itself is pretty small, so dynamic allocation is really not a good idea. Instead, the internal field such as vvmcs/svmcs are big and thus we use a pointer, but they are allocated on demand. This follows the style of arch_vmx_struct in vcpu data structure.

I am fine with your nestedhvm_vcpu_initialise "design", but VMX doesn't need to use the wrapper so far.

It should be implemented good enough that vcpu creation doesn't fail.

Christoph

Signature · 6 lines
---to satisfy European Law for business letters:
Advanced Micro Devices GmbH
Einsteinring 24, 85609 Dornach b. Muenchen
Geschaeftsfuehrer: Alberto Bozzo, Andrew Bowd
Sitz: Dornach, Gemeinde Aschheim, Landkreis Muenchen
Registergericht Muenchen, HRB Nr. 43632
#13 lore raw

Re: [PATCH 04/16] vmx: nest: nested control structure

Quoted 83 lines · On Wednesday 15 September 2010 15:17:42 Christoph Egger wrote:
Quoted 82 lines · Eddie
Quoted 27 lines · Christoph Egger
Quoted 26 lines · On Wednesday 08 September 2010 17:22:12 Qing He wrote:

v->arch.hvm_vmx.nest as control structure

---
diff -r fc4de5eedd1d xen/include/asm-x86/hvm/vmx/nest.h

xen/include/asm-x86/hvm/vmx/nest.h new file +7 -0
+0800 @@ -0,0 +1,45 @@ +/*
+ * nest.h: nested virtualization for VMX.
+ *
+ * Copyright (c) 2010, Intel Corporation.
+ * Author: Qing He <[email protected]>
+ *
+ * This program is free software; you can redistribute it and/or

modify it + * under the terms and conditions of the GNU General Public License, + * version 2, as published by the Free Software Foundation. + * + * This program is distributed in the hope it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + *
+ * You should have received a copy of the GNU General Public License along with + * this program; if not, write to the Free Software Foundation, Inc., 59 Temple + * Place - Suite 330, Boston, MA 02111-1307 USA. + * + */

Quoted 8 lines

xen/include/asm-x86/hvm/vmx/nest.h line 26

Quoted 6 lines
xen/include/asm-x86/hvm/vmx/nest.h new file +6 -0
+#ifndef __ASM_X86_HVM_NEST_H__
+#define __ASM_X86_HVM_NEST_H__
+
+struct vmcs_struct;
+
+struct vmx_nest_struct {

Is it ok to name it 'struct nestedvmx' ?

Fine, renamed to nested_vmx.

Quoted 42 lines
Quoted 29 lines
xen/include/asm-x86/hvm/vmx/nest.h new file +11 -0
+    paddr_t              guest_vmxon_pa;
+
+    /* Saved host vmcs for vcpu itself */
+    struct vmcs_struct  *hvmcs;
+
+    /*
+     * Guest's `current vmcs' of vcpu
+     *  - gvmcs_pa: guest VMCS region physical address
+     *  - vvmcs:    (guest) virtual vmcs
+     *  - svmcs:    effective vmcs for the guest of this vcpu
+     *  - valid:    launch state: invalid on clear, valid on ld +

*/ + paddr_t gvmcs_pa;

xen/include/asm-x86/hvm/vmx/nest.h new file +6 -0
+    void                *vvmcs;
+    struct vmcs_struct  *svmcs;
+    int                  vmcs_valid;
+};
+
+#endif /* __ASM_X86_HVM_NEST_H__ */

diff -r fc4de5eedd1d xen/include/asm-x86/hvm/vmx/vmcs.h
--- a/xen/include/asm-x86/hvm/vmx/vmcs.h Wed Sep 08 21:00:00 2010
+0800 +++ b/xen/include/asm-x86/hvm/vmx/vmcs.h Wed Sep 08 21:03:41
2010 +0800 @@ -22,6 +22,7 @@ #include <asm/config.h> #include <asm/hvm/io.h>
#include <asm/hvm/vpmu.h>

xen/include/asm-x86/hvm/vmx/nest.h new file +1 -0
+#include <asm/hvm/vmx/nest.h>
 
 extern void vmcs_dump_vcpu(struct vcpu *v);
 extern void setup_vmcs_dump(void);

xen/include/asm-x86/hvm/vmx/nest.h line 108

Quoted 9 lines
xen/include/asm-x86/hvm/vmx/nest.h new file +3 -0
@@ -99,6 +100,9 @@
     u32                  secondary_exec_control;
     u32                  exception_bitmap;
 
+    /* nested virtualization */
+    struct vmx_nest_struct nest;
+
 #ifdef __x86_64__
     struct vmx_msr_state msr_state;
     unsigned long        shadow_gs;

I think, the structure should be allocated in the nestedhvm_vcpu_initialise() function hook and assigned to the nh_arch pointer in struct nestedhvm.

Quoted 2 lines
Quoted 2 lines

Well, the structure itself is pretty small, so dynamic allocation is really not a good idea.

It's not a question of size. The point is that it is opaque to non-vmx code.

Quoted 3 lines
Quoted 3 lines

Instead, the internal field such as vvmcs/svmcs
are big and thus we use a pointer, but they are allocated on demand. This follows the style of arch_vmx_struct in vcpu data structure.

That's fine though. However you can reuse some fields of struct nestedhvm:

gvmcs_pa is functional the same as nh_vmaddr svmcs is functional the same as nh_vm

Quoted 6 lines
Quoted 2 lines

I am fine with your nestedhvm_vcpu_initialise "design", but VMX doesn't need to use the wrapper so far.

It should be implemented good enough that vcpu creation doesn't fail.

Christoph

Signature · 6 lines
---to satisfy European Law for business letters:
Advanced Micro Devices GmbH
Einsteinring 24, 85609 Dornach b. Muenchen
Geschaeftsfuehrer: Alberto Bozzo, Andrew Bowd
Sitz: Dornach, Gemeinde Aschheim, Landkreis Muenchen
Registergericht Muenchen, HRB Nr. 43632
#14 lore raw

RE: [PATCH 04/16] vmx: nest: nested control structure

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 8 lines
Quoted 4 lines
 #ifdef __x86_64__
     struct vmx_msr_state msr_state;
     unsigned long        shadow_gs;

I think, the structure should be allocated in the nestedhvm_vcpu_initialise() function hook and assigned to the nh_arch pointer in struct nestedhvm.

Quoted 5 lines
Quoted 2 lines
Quoted 2 lines

Well, the structure itself is pretty small, so dynamic allocation is really not a good idea.

It's not a question of size. The point is that it is opaque to non-vmx code.

I think it is anyway recycling again, that is not necessary.
Outside of VMX should not access those VMX specific code in theory. If it needs some parameters, vcpu can stand for all of them.

Thx, Eddie

#15 lore raw

Re: [PATCH 04/16] vmx: nest: nested control structure

Quoted 8 lines · Eddie
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 7 lines
Quoted 3 lines
 #ifdef __x86_64__
     struct vmx_msr_state msr_state;
     unsigned long        shadow_gs;

I think, the structure should be allocated in the nestedhvm_vcpu_initialise() function hook and assigned to the nh_arch pointer in struct nestedhvm.

Quoted 8 lines
Quoted 5 lines
Quoted 2 lines
Quoted 2 lines

Well, the structure itself is pretty small, so dynamic allocation is really not a good idea.

It's not a question of size. The point is that it is opaque to non-vmx code.

I think it is anyway recycling again, that is not necessary. Outside of VMX should not access those VMX specific code in theory.

Also practically. The point is if that mistake happens then gcc will complain when the data structure is opaque to non-vmx code.

I already discussed exactly that point with Keir when he suggested me to go your way but then agreed.
See:
http://lists.xensource.com/archives/html/xen-devel/2010-08/msg01024.html
http://lists.xensource.com/archives/html/xen-devel/2010-08/msg01028.html

Christoph

Signature · 6 lines
---to satisfy European Law for business letters:
Advanced Micro Devices GmbH
Einsteinring 24, 85609 Dornach b. Muenchen
Geschaeftsfuehrer: Alberto Bozzo, Andrew Bowd
Sitz: Dornach, Gemeinde Aschheim, Landkreis Muenchen
Registergericht Muenchen, HRB Nr. 43632
author
#16 lore raw

PATCH 5/16 vmx: nest: virtual vmcs layout

In reply to Qing He

Since physical vmcs is transparent, a customized virtual vmcs (vvmcs) is introduced. It converts the vmcs encoding to an offset into vvmcs page.

---

xen/include/asm-x86/hvm/vmx/vvmcs.h new file +154 -0
diff -r 5935027e5e70 xen/include/asm-x86/hvm/vmx/vvmcs.h
@@ -0,0 +1,154 @@
+/*
+ * vvmcs.h: virtual VMCS access for nested virtualization.
+ *
+ * Copyright (c) 2010, Intel Corporation.
+ * Author: Qing He <[email protected]>
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms and conditions of the GNU General Public License,
+ * version 2, as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope it will be useful, but WITHOUT
+ * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
+ * FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License for
+ * more details.
+ *
+ * You should have received a copy of the GNU General Public License along with
+ * this program; if not, write to the Free Software Foundation, Inc., 59 Temple
+ * Place - Suite 330, Boston, MA 02111-1307 USA.
+ *
+ */
+
+#include <xen/config.h>
+#include <asm/types.h>
+
+/*
+ * Virtual VMCS layout
+ *
+ * Since physical VMCS layout is unknown, a custom layout is used
+ * for virtual VMCS seen by guest. It occupies a 4k page, and the
+ * field is offset by an 9-bit offset into u64[], The offset is as
+ * follow, which means every <width, type> pair has a max of 32
+ * fields available.
+ *
+ *             9       7      5               0
+ *             --------------------------------
+ *     offset: | width | type |     index     |
+ *             --------------------------------
+ *
+ * Also, since the lower range <width=0, type={0,1}> has only one
+ * field: VPID, it is moved to a higher offset (63), and leaves the
+ * lower range to non-indexed field like VMCS revision.
+ *
+ */
+
+#define VVMCS_REVISION 0x40000001u
+
+struct vvmcs_header {
+    u32 revision;
+    u32 abort;
+};
+
+union vmcs_encoding {
+    struct {
+        u32 access_type : 1;
+        u32 index : 9;
+        u32 type : 2;
+        u32 rsv1 : 1;
+        u32 width : 2;
+        u32 rsv2 : 17;
+    };
+    u32 word;
+};
+
+enum vvmcs_encoding_width {
+    VVMCS_WIDTH_16 = 0,
+    VVMCS_WIDTH_64,
+    VVMCS_WIDTH_32,
+    VVMCS_WIDTH_NATURAL,
+};
+
+enum vvmcs_encoding_type {
+    VVMCS_TYPE_CONTROL = 0,
+    VVMCS_TYPE_RO,
+    VVMCS_TYPE_GSTATE,
+    VVMCS_TYPE_HSTATE,
+};
+
+static inline int vvmcs_offset(u32 width, u32 type, u32 index)
+{
+    int offset;
+
+    offset = (index & 0x1f) | type << 5 | width << 7;
+
+    if ( offset == 0 )    /* vpid */
+        offset = 0x3f;
+
+    return offset;
+}
+
+static inline u64 __get_vvmcs(void *vvmcs, u32 vmcs_encoding)
+{
+    union vmcs_encoding enc;
+    u64 *content = (u64 *) vvmcs;
+    int offset;
+    u64 res;
+
+    enc.word = vmcs_encoding;
+    offset = vvmcs_offset(enc.width, enc.type, enc.index);
+    res = content[offset];
+
+    switch ( enc.width ) {
+    case VVMCS_WIDTH_16:
+        res &= 0xffff;
+        break;
+    case VVMCS_WIDTH_64:
+        if ( enc.access_type )
+            res >>= 32;
+        break;
+    case VVMCS_WIDTH_32:
+        res &= 0xffffffff;
+        break;
+    case VVMCS_WIDTH_NATURAL:
+    default:
+        break;
+    }
+
+    return res;
+}
+
+static inline void __set_vvmcs(void *vvmcs, u32 vmcs_encoding, u64 val)
+{
+    union vmcs_encoding enc;
+    u64 *content = (u64 *) vvmcs;
+    int offset;
+    u64 res;
+
+    enc.word = vmcs_encoding;
+    offset = vvmcs_offset(enc.width, enc.type, enc.index);
+    res = content[offset];
+
+    switch ( enc.width ) {
+    case VVMCS_WIDTH_16:
+        res = val & 0xffff;
+        break;
+    case VVMCS_WIDTH_64:
+        if ( enc.access_type )
+        {
+            res &= 0xffffffff;
+            res |= val << 32;
+        }
+        else
+            res = val;
+        break;
+    case VVMCS_WIDTH_32:
+        res = val & 0xffffffff;
+        break;
+    case VVMCS_WIDTH_NATURAL:
+    default:
+        res = val;
+        break;
+    }
+
+    content[offset] = res;
+}
#17 lore raw

Re: [PATCH 05/16] vmx: nest: virtual vmcs layout

Quoted 5 lines · Qing He

+ * Since physical VMCS layout is unknown, a custom layout is used
+ * for virtual VMCS seen by guest. It occupies a 4k page, and the
+ * field is offset by an 9-bit offset into u64[], The offset is as
+ * follow, which means every <width, type> pair has a max of 32
+ * fields available.

A question - is this layout likely to have to change often in future? (I guess the answer's no but you'll know best.) We'll have to carry compatibility code indefinitely so that VMs can be migrated to newer Xens without breaking.

Quoted 10 lines
+ *             9       7      5               0
+ *             --------------------------------
+ *     offset: | width | type |     index     |
+ *             --------------------------------

+ *
+ * Also, since the lower range <width=0, type={0,1}> has only one
+ * field: VPID, it is moved to a higher offset (63), and leaves the
+ * lower range to non-indexed field like VMCS revision.
+ *
+ */

xen/include/asm-x86/hvm/vmx/vvmcs.h line 50

Quoted 7 lines
xen/include/asm-x86/hvm/vmx/vvmcs.h new file +7 -0
+
+#define VVMCS_REVISION 0x40000001u
+
+struct vvmcs_header {
+    u32 revision;
+    u32 abort;
+};

This structure isn't used anywhere in the rest of the patch series. Oversight?

Cheers,

Tim.

Signature · 3 lines
Tim Deegan <[email protected]>
Principal Software Engineer, XenServer Engineering
Citrix Systems UK Ltd.  (Company #02937203, SL9 0BG)
author
#18 lore raw

PATCH 6/16 vmx: nest: handling VMX instruction exits

In reply to Qing He

add a VMX instruction decoder and handle simple VMX instructions except vmlaunch/vmresume and invept

---

xen/arch/x86/hvm/vmx/Makefile +1 -0
diff -r f1c1d3077337 xen/arch/x86/hvm/vmx/Makefile
@@ -4,3 +4,4 @@
 obj-y += vmcs.o
 obj-y += vmx.o
 obj-y += vpmu_core2.o
+obj-y += nest.o
xen/arch/x86/hvm/vmx/nest.c new file +635 -0

635 lines not on this page · raw message

xen/arch/x86/hvm/vmx/vmx.c +36 -7

53 lines not on this page · raw message

xen/include/asm-x86/hvm/vmx/nest.h +10 -0
diff -r f1c1d3077337 xen/include/asm-x86/hvm/vmx/nest.h
@@ -42,4 +42,14 @@
     int                  vmcs_valid;
 };
 
+int vmx_nest_handle_vmxon(struct cpu_user_regs *regs);
+int vmx_nest_handle_vmxoff(struct cpu_user_regs *regs);
+
+int vmx_nest_handle_vmptrld(struct cpu_user_regs *regs);
+int vmx_nest_handle_vmptrst(struct cpu_user_regs *regs);
+int vmx_nest_handle_vmclear(struct cpu_user_regs *regs);
+
+int vmx_nest_handle_vmread(struct cpu_user_regs *regs);
+int vmx_nest_handle_vmwrite(struct cpu_user_regs *regs);
+
 #endif /* __ASM_X86_HVM_NEST_H__ */
#19 lore raw

RE: [PATCH 06/16] vmx: nest: handling VMX instruction exits

Quoted 9 lines · Qing He

add a VMX instruction decoder and handle simple VMX instructions except vmlaunch/vmresume and invept

---

xen/arch/x86/hvm/vmx/nest.c

Quoted 9 lines
xen/arch/x86/hvm/vmx/nest.c new file +9 -0
+static int __clear_current_vvmcs(struct vmx_nest_struct *nest)
+{
+    int rc;
+
+    if ( nest->svmcs )
+        __vmpclear(virt_to_maddr(nest->svmcs));
+
+#if !CONFIG_VVMCS_MAPPING
+    rc = hvm_copy_to_guest_phys(nest->gvmcs_pa, nest->vvmcs,

Qing:

	Why this may be failure? The only reason may be nest->gvmcs_pa, but I guess we already verified the address.

Thx, Eddie

Quoted 10 lines

PAGE_SIZE); + if ( rc != HVMCOPY_okay )

xen/arch/x86/hvm/vmx/nest.c new file +9 -0
+        return X86EMUL_EXCEPTION;
+#endif
+
+    nest->vmcs_valid = 0;
+
+    return X86EMUL_OKAY;
+}
+
+/*
#20 lore raw

Re: [PATCH 06/16] vmx: nest: handling VMX instruction exits

Quoted 13 lines · Eddie

xen/arch/x86/hvm/vmx/nest.c

Quoted 9 lines · Qing He
xen/arch/x86/hvm/vmx/nest.c new file +9 -0
+static int __clear_current_vvmcs(struct vmx_nest_struct *nest)
+{
+    int rc;
+
+    if ( nest->svmcs )
+        __vmpclear(virt_to_maddr(nest->svmcs));
+
+#if !CONFIG_VVMCS_MAPPING
+    rc = hvm_copy_to_guest_phys(nest->gvmcs_pa, nest->vvmcs,

Qing:

Quoted 2 lines
	Why this may be failure? The only reason may be nest->gvmcs_pa, but I guess we already verified the address.

It was verified at load time, but the guest could have ballooned it out in the meantime.

Cheers,

Tim

Signature · 3 lines
Tim Deegan <[email protected]>
Principal Software Engineer, XenServer Engineering
Citrix Systems UK Ltd.  (Company #02937203, SL9 0BG)
#21 lore raw

RE: [PATCH 06/16] vmx: nest: handling VMX instruction exits

Quoted 17 lines · Tim Deegan
Quoted 12 lines · Eddie

xen/arch/x86/hvm/vmx/nest.c

Quoted 8 lines · Qing He
xen/arch/x86/hvm/vmx/nest.c new file +8 -0
+static int __clear_current_vvmcs(struct vmx_nest_struct *nest) +{
+    int rc;
+
+    if ( nest->svmcs )
+        __vmpclear(virt_to_maddr(nest->svmcs));
+
+#if !CONFIG_VVMCS_MAPPING
+    rc = hvm_copy_to_guest_phys(nest->gvmcs_pa, nest->vvmcs,

Qing:

Quoted 3 lines
	Why this may be failure? The only reason may be nest->gvmcs_pa, but

I guess we already verified the address.

Quoted 2 lines

It was verified at load time, but the guest could have ballooned it out in the meantime.

If the L1 guest allocated that GPA as VMCS memory, it can't balloon them out. If L1 is a mallicious guest and ballooned the VMCS memory out, it is worthy to do. Not?

Quoted 3 lines

Cheers,

Thx, Eddie

#22 lore raw

Re: [PATCH 06/16] vmx: nest: handling VMX instruction exits

Quoted 18 lines · Eddie
Quoted 17 lines · Tim Deegan
Quoted 15 lines · Eddie

xen/arch/x86/hvm/vmx/nest.c

Quoted 8 lines · Qing He
xen/arch/x86/hvm/vmx/nest.c new file +8 -0
+static int __clear_current_vvmcs(struct vmx_nest_struct *nest) +{
+    int rc;
+
+    if ( nest->svmcs )
+        __vmpclear(virt_to_maddr(nest->svmcs));
+
+#if !CONFIG_VVMCS_MAPPING
+    rc = hvm_copy_to_guest_phys(nest->gvmcs_pa, nest->vvmcs,

Qing:

	Why this may be failure? The only reason may be nest->gvmcs_pa, but

I guess we already verified the address.

Quoted 6 lines
Quoted 2 lines

It was verified at load time, but the guest could have ballooned it out in the meantime.

If the L1 guest allocated that GPA as VMCS memory, it can't balloon them out. If L1 is a mallicious guest and ballooned the VMCS memory out, it is worthy to do. Not?

Yes, in this case it looks like there's probably no harm in ignoring a failure, but it seems reasonable to handle it.

Tim.

Signature · 3 lines
Tim Deegan <[email protected]>
Principal Software Engineer, XenServer Engineering
Citrix Systems UK Ltd.  (Company #02937203, SL9 0BG)
#23 lore raw

Re: [PATCH 06/16] vmx: nest: handling VMX instruction exits

In reply to Qing He
Quoted 33 lines · Qing He

diff -r f1c1d3077337 xen/arch/x86/hvm/vmx/nest.c

xen/arch/x86/hvm/vmx/nest.c new file +32 -0
+/*
+ * VMX instructions support functions
+ */
+
+enum vmx_regs_enc {
+    VMX_REG_RAX,
+    VMX_REG_RCX,
+    VMX_REG_RDX,
+    VMX_REG_RBX,
+    VMX_REG_RSP,
+    VMX_REG_RBP,
+    VMX_REG_RSI,
+    VMX_REG_RDI,
+#ifdef CONFIG_X86_64
+    VMX_REG_R8,
+    VMX_REG_R9,
+    VMX_REG_R10,
+    VMX_REG_R11,
+    VMX_REG_R12,
+    VMX_REG_R13,
+    VMX_REG_R14,
+    VMX_REG_R15,
+#endif
+};
+
+enum vmx_sregs_enc {
+    VMX_SREG_ES,
+    VMX_SREG_CS,
+    VMX_SREG_SS,
+    VMX_SREG_DS,
+    VMX_SREG_FS,
+    VMX_SREG_GS,

xen/arch/x86/hvm/vmx/nest.c line 73

Quoted 10 lines
xen/arch/x86/hvm/vmx/nest.c new file +10 -0
+};
+
+enum x86_segment sreg_to_index[] = {
+    [VMX_SREG_ES] = x86_seg_es,
+    [VMX_SREG_CS] = x86_seg_cs,
+    [VMX_SREG_SS] = x86_seg_ss,
+    [VMX_SREG_DS] = x86_seg_ds,
+    [VMX_SREG_FS] = x86_seg_fs,
+    [VMX_SREG_GS] = x86_seg_gs,
+};

Since you dislike adding new namespaces and translations, I'm sure you can get rid of these. :) It might even simplify some of the macros below.

Quoted 114 lines
xen/arch/x86/hvm/vmx/nest.c new file +114 -0
+union vmx_inst_info {
+    struct {
+        unsigned int scaling           :2; /* bit 0-1 */
+        unsigned int __rsvd0           :1; /* bit 2 */
+        unsigned int reg1              :4; /* bit 3-6 */
+        unsigned int addr_size         :3; /* bit 7-9 */
+        unsigned int memreg            :1; /* bit 10 */
+        unsigned int __rsvd1           :4; /* bit 11-14 */
+        unsigned int segment           :3; /* bit 15-17 */
+        unsigned int index_reg         :4; /* bit 18-21 */
+        unsigned int index_reg_invalid :1; /* bit 22 */
+        unsigned int base_reg          :4; /* bit 23-26 */
+        unsigned int base_reg_invalid  :1; /* bit 27 */
+        unsigned int reg2              :4; /* bit 28-31 */
+    } fields;
+    u32 word;
+};
+
+struct vmx_inst_decoded {
+#define VMX_INST_MEMREG_TYPE_MEMORY 0
+#define VMX_INST_MEMREG_TYPE_REG    1
+    int type;
+    union {
+        struct {
+            unsigned long mem;
+            unsigned int  len;
+        };
+        enum vmx_regs_enc reg1;
+    };
+
+    enum vmx_regs_enc reg2;
+};
+
+enum vmx_ops_result {
+    VMSUCCEED,
+    VMFAIL_VALID,
+    VMFAIL_INVALID,
+};
+
+#define CASE_SET_REG(REG, reg)      \
+    case VMX_REG_ ## REG: regs->reg = value; break
+#define CASE_GET_REG(REG, reg)      \
+    case VMX_REG_ ## REG: value = regs->reg; break
+
+#define CASE_EXTEND_SET_REG         \
+    CASE_EXTEND_REG(S)
+#define CASE_EXTEND_GET_REG         \
+    CASE_EXTEND_REG(G)
+
+#ifdef __i386__
+#define CASE_EXTEND_REG(T)
+#else
+#define CASE_EXTEND_REG(T)          \
+    CASE_ ## T ## ET_REG(R8, r8);   \
+    CASE_ ## T ## ET_REG(R9, r9);   \
+    CASE_ ## T ## ET_REG(R10, r10); \
+    CASE_ ## T ## ET_REG(R11, r11); \
+    CASE_ ## T ## ET_REG(R12, r12); \
+    CASE_ ## T ## ET_REG(R13, r13); \
+    CASE_ ## T ## ET_REG(R14, r14); \
+    CASE_ ## T ## ET_REG(R15, r15)
+#endif
+
+static unsigned long reg_read(struct cpu_user_regs *regs,
+                              enum vmx_regs_enc index)
+{
+    unsigned long value = 0;
+
+    switch ( index ) {
+    CASE_GET_REG(RAX, eax);
+    CASE_GET_REG(RCX, ecx);
+    CASE_GET_REG(RDX, edx);
+    CASE_GET_REG(RBX, ebx);
+    CASE_GET_REG(RBP, ebp);
+    CASE_GET_REG(RSI, esi);
+    CASE_GET_REG(RDI, edi);
+    CASE_GET_REG(RSP, esp);
+    CASE_EXTEND_GET_REG;
+    default:
+        break;
+    }
+
+    return value;
+}
+
+static void reg_write(struct cpu_user_regs *regs,
+                      enum vmx_regs_enc index,
+                      unsigned long value)
+{
+    switch ( index ) {
+    CASE_SET_REG(RAX, eax);
+    CASE_SET_REG(RCX, ecx);
+    CASE_SET_REG(RDX, edx);
+    CASE_SET_REG(RBX, ebx);
+    CASE_SET_REG(RBP, ebp);
+    CASE_SET_REG(RSI, esi);
+    CASE_SET_REG(RDI, edi);
+    CASE_SET_REG(RSP, esp);
+    CASE_EXTEND_SET_REG;
+    default:
+        break;
+    }
+}
+
+static int decode_vmx_inst(struct cpu_user_regs *regs,
+                           struct vmx_inst_decoded *decode)
+{
+    struct vcpu *v = current;
+    union vmx_inst_info info;
+    struct segment_register seg;
+    unsigned long base, index, seg_base, disp, offset;
+    int scale;
+
+    info.word = __vmread(VMX_INSTRUCTION_INFO);

xen/arch/x86/hvm/vmx/nest.c line 198

Quoted 10 lines
xen/arch/x86/hvm/vmx/nest.c new file +10 -0
+
+    if ( info.fields.memreg ) {
+        decode->type = VMX_INST_MEMREG_TYPE_REG;
+        decode->reg1 = info.fields.reg1;
+    }
+    else
+    {
+        decode->type = VMX_INST_MEMREG_TYPE_MEMORY;
+        hvm_get_segment_register(v, sreg_to_index[info.fields.segment], &seg);
+        /* TODO: segment type check */

Indeed. :)

Quoted 4 lines
xen/arch/x86/hvm/vmx/nest.c new file +4 -0
+        seg_base = seg.base;
+
+        base = info.fields.base_reg_invalid ? 0 :
+            reg_read(regs, info.fields.base_reg);

xen/arch/x86/hvm/vmx/nest.c line 212

Quoted 10 lines
xen/arch/x86/hvm/vmx/nest.c new file +10 -0
+
+        index = info.fields.index_reg_invalid ? 0 :
+            reg_read(regs, info.fields.index_reg);
+
+        scale = 1 << info.fields.scaling;
+
+        disp = __vmread(EXIT_QUALIFICATION);
+
+        offset = base + index * scale + disp;
+        if ( offset > seg.limit )

DYM if ( offset + len > limit )?

Would it be worth trying to fold this into the existing x86_emulate code, which already has careful memory access checks?

Quoted 92 lines
xen/arch/x86/hvm/vmx/nest.c new file +92 -0
+            goto gp_fault;
+
+        decode->mem = seg_base + base + index * scale + disp;
+        decode->len = 1 << (info.fields.addr_size + 1);
+    }
+
+    decode->reg2 = info.fields.reg2;
+
+    return X86EMUL_OKAY;
+
+gp_fault:
+    hvm_inject_exception(TRAP_gp_fault, 0, 0);
+    return X86EMUL_EXCEPTION;
+}
+
+static int vmx_inst_check_privilege(struct cpu_user_regs *regs)
+{
+    struct vcpu *v = current;
+    struct segment_register cs;
+
+    hvm_get_segment_register(v, x86_seg_cs, &cs);
+
+    if ( !(v->arch.hvm_vcpu.guest_cr[0] & X86_CR0_PE) ||
+         !(v->arch.hvm_vcpu.guest_cr[4] & X86_CR4_VMXE) ||
+         (regs->eflags & X86_EFLAGS_VM) ||
+         (hvm_long_mode_enabled(v) && cs.attr.fields.l == 0) )
+        goto invalid_op;
+
+    if ( (cs.sel & 3) > 0 )
+        goto gp_fault;
+
+    return X86EMUL_OKAY;
+
+invalid_op:
+    hvm_inject_exception(TRAP_invalid_op, 0, 0);
+    return X86EMUL_EXCEPTION;
+
+gp_fault:
+    hvm_inject_exception(TRAP_gp_fault, 0, 0);
+    return X86EMUL_EXCEPTION;
+}
+
+static void vmreturn(struct cpu_user_regs *regs, enum vmx_ops_result res)
+{
+    unsigned long eflags = regs->eflags;
+    unsigned long mask = X86_EFLAGS_CF | X86_EFLAGS_PF | X86_EFLAGS_AF |
+                         X86_EFLAGS_ZF | X86_EFLAGS_SF | X86_EFLAGS_OF;
+
+    eflags &= ~mask;
+
+    switch ( res ) {
+    case VMSUCCEED:
+        break;
+    case VMFAIL_VALID:
+        /* TODO: error number, useful for guest VMM debugging */
+        eflags |= X86_EFLAGS_ZF;
+        break;
+    case VMFAIL_INVALID:
+    default:
+        eflags |= X86_EFLAGS_CF;
+        break;
+    }
+
+    regs->eflags = eflags;
+}
+
+static int __clear_current_vvmcs(struct vmx_nest_struct *nest)
+{
+    int rc;
+
+    if ( nest->svmcs )
+        __vmpclear(virt_to_maddr(nest->svmcs));
+
+#if !CONFIG_VVMCS_MAPPING
+    rc = hvm_copy_to_guest_phys(nest->gvmcs_pa, nest->vvmcs, PAGE_SIZE);
+    if ( rc != HVMCOPY_okay )
+        return X86EMUL_EXCEPTION;
+#endif
+
+    nest->vmcs_valid = 0;
+
+    return X86EMUL_OKAY;
+}
+
+/*
+ * VMX instructions handling
+ */
+
+int vmx_nest_handle_vmxon(struct cpu_user_regs *regs)
+{
+    struct vcpu *v = current;
+    struct vmx_nest_struct *nest = &v->arch.hvm_vmx.nest;

xen/arch/x86/hvm/vmx/nest.c line 314

Quoted 10 lines
xen/arch/x86/hvm/vmx/nest.c new file +10 -0
+    struct vmx_inst_decoded decode;
+    unsigned long gpa = 0;
+    int rc;
+
+    if ( !is_nested_avail(v->domain) )
+        goto invalid_op;
+ 
+    rc = vmx_inst_check_privilege(regs);
+    if ( rc != X86EMUL_OKAY )
+        return rc;

I think you could fold these checks and the goto target into decode_vmx_inst(), just to avoid repeating them in every vmx_nest_handle_* function.

Quoted 60 lines
xen/arch/x86/hvm/vmx/nest.c new file +60 -0
+    rc = decode_vmx_inst(regs, &decode);
+    if ( rc != X86EMUL_OKAY )
+        return rc;
+
+    ASSERT(decode.type == VMX_INST_MEMREG_TYPE_MEMORY);
+    rc = hvm_copy_from_guest_virt(&gpa, decode.mem, decode.len, 0);
+    if ( rc != HVMCOPY_okay )
+        return X86EMUL_EXCEPTION;
+
+    nest->guest_vmxon_pa = gpa;
+    nest->gvmcs_pa = 0;
+    nest->vmcs_valid = 0;
+#if !CONFIG_VVMCS_MAPPING
+    nest->vvmcs = alloc_xenheap_page();
+    if ( !nest->vvmcs )
+    {
+        gdprintk(XENLOG_ERR, "nest: allocation for virtual vmcs failed\n");
+        vmreturn(regs, VMFAIL_INVALID);
+        goto out;
+    }
+#endif
+    nest->svmcs = alloc_xenheap_page();
+    if ( !nest->svmcs )
+    {
+        gdprintk(XENLOG_ERR, "nest: allocation for shadow vmcs failed\n");
+        free_xenheap_page(nest->vvmcs);
+        vmreturn(regs, VMFAIL_INVALID);
+        goto out;
+    }
+
+    /*
+     * `fork' the host vmcs to shadow_vmcs
+     * vmcs_lock is not needed since we are on current
+     */
+    nest->hvmcs = v->arch.hvm_vmx.vmcs;
+    __vmpclear(virt_to_maddr(nest->hvmcs));
+    memcpy(nest->svmcs, nest->hvmcs, PAGE_SIZE);
+    __vmptrld(virt_to_maddr(nest->hvmcs));
+    v->arch.hvm_vmx.launched = 0;
+
+    vmreturn(regs, VMSUCCEED);
+
+out:
+    return X86EMUL_OKAY;
+
+invalid_op:
+    hvm_inject_exception(TRAP_invalid_op, 0, 0);
+    return X86EMUL_EXCEPTION;
+}
+
+int vmx_nest_handle_vmxoff(struct cpu_user_regs *regs)
+{
+    struct vcpu *v = current;
+    struct vmx_nest_struct *nest = &v->arch.hvm_vmx.nest;
+    int rc;
+
+    if ( unlikely(!nest->guest_vmxon_pa) )
+        goto invalid_op;
+
+    rc = vmx_inst_check_privilege(regs);

xen/arch/x86/hvm/vmx/nest.c line 385

Quoted 10 lines
xen/arch/x86/hvm/vmx/nest.c new file +10 -0
+    if ( rc != X86EMUL_OKAY )
+        return rc;
+
+    nest->guest_vmxon_pa = 0;
+    __vmpclear(virt_to_maddr(nest->svmcs));
+
+#if !CONFIG_VVMCS_MAPPING
+    free_xenheap_page(nest->vvmcs);
+#endif
+    free_xenheap_page(nest->svmcs);

These also need to be freed on domain teardown.

Quoted 124 lines
xen/arch/x86/hvm/vmx/nest.c new file +124 -0
+    vmreturn(regs, VMSUCCEED);
+    return X86EMUL_OKAY;
+
+invalid_op:
+    hvm_inject_exception(TRAP_invalid_op, 0, 0);
+    return X86EMUL_EXCEPTION;
+}
+
+int vmx_nest_handle_vmptrld(struct cpu_user_regs *regs)
+{
+    struct vcpu *v = current;
+    struct vmx_inst_decoded decode;
+    struct vmx_nest_struct *nest = &v->arch.hvm_vmx.nest;
+    unsigned long gpa = 0;
+    int rc;
+
+    if ( unlikely(!nest->guest_vmxon_pa) )
+        goto invalid_op;
+
+    rc = vmx_inst_check_privilege(regs);
+    if ( rc != X86EMUL_OKAY )
+        return rc;
+
+    rc = decode_vmx_inst(regs, &decode);
+    if ( rc != X86EMUL_OKAY )
+        return rc;
+
+    ASSERT(decode.type == VMX_INST_MEMREG_TYPE_MEMORY);
+    rc = hvm_copy_from_guest_virt(&gpa, decode.mem, decode.len, 0);
+    if ( rc != HVMCOPY_okay )
+        return X86EMUL_EXCEPTION;
+
+    if ( gpa == nest->guest_vmxon_pa || gpa & 0xfff )
+    {
+        vmreturn(regs, VMFAIL_INVALID);
+        goto out;
+    }
+
+    if ( nest->gvmcs_pa != gpa )
+    {
+        if ( nest->vmcs_valid )
+        {
+            rc = __clear_current_vvmcs(nest);
+            if ( rc != X86EMUL_OKAY )
+                return rc;
+        }
+        nest->gvmcs_pa = gpa;
+        ASSERT(nest->vmcs_valid == 0);
+    }
+
+
+    if ( !nest->vmcs_valid )
+    {
+#if CONFIG_VVMCS_MAPPING
+        unsigned long mfn;
+        p2m_type_t p2mt;
+
+        mfn = mfn_x(gfn_to_mfn(p2m_get_hostp2m(v->domain),
+                               nest->gvmcs_pa >> PAGE_SHIFT, &p2mt));
+        nest->vvmcs = map_domain_page_global(mfn);
+#else
+        rc = hvm_copy_from_guest_phys(nest->vvmcs, nest->gvmcs_pa, PAGE_SIZE);
+        if ( rc != HVMCOPY_okay )
+            return X86EMUL_EXCEPTION;
+#endif
+        nest->vmcs_valid = 1;
+    }
+
+    vmreturn(regs, VMSUCCEED);
+
+out:
+    return X86EMUL_OKAY;
+
+invalid_op:
+    hvm_inject_exception(TRAP_invalid_op, 0, 0);
+    return X86EMUL_EXCEPTION;
+}
+
+int vmx_nest_handle_vmptrst(struct cpu_user_regs *regs)
+{
+    struct vcpu *v = current;
+    struct vmx_inst_decoded decode;
+    struct vmx_nest_struct *nest = &v->arch.hvm_vmx.nest;
+    unsigned long gpa = 0;
+    int rc;
+
+    if ( unlikely(!nest->guest_vmxon_pa) )
+        goto invalid_op;
+
+    rc = vmx_inst_check_privilege(regs);
+    if ( rc != X86EMUL_OKAY )
+        return rc;
+
+    rc = decode_vmx_inst(regs, &decode);
+    if ( rc != X86EMUL_OKAY )
+        return rc;
+
+    ASSERT(decode.type == VMX_INST_MEMREG_TYPE_MEMORY);
+
+    gpa = nest->gvmcs_pa;
+
+    rc = hvm_copy_to_guest_virt(decode.mem, &gpa, decode.len, 0);
+    if ( rc != HVMCOPY_okay )
+        return X86EMUL_EXCEPTION;
+
+    vmreturn(regs, VMSUCCEED);
+    return X86EMUL_OKAY;
+
+invalid_op:
+    hvm_inject_exception(TRAP_invalid_op, 0, 0);
+    return X86EMUL_EXCEPTION;
+}
+
+int vmx_nest_handle_vmclear(struct cpu_user_regs *regs)
+{
+    struct vcpu *v = current;
+    struct vmx_inst_decoded decode;
+    struct vmx_nest_struct *nest = &v->arch.hvm_vmx.nest;
+    unsigned long gpa = 0;
+    int rc;
+
+    if ( unlikely(!nest->guest_vmxon_pa) )
+        goto invalid_op;
+

xen/arch/x86/hvm/vmx/nest.c line 520

Quoted 10 lines
xen/arch/x86/hvm/vmx/nest.c new file +10 -0
+    rc = vmx_inst_check_privilege(regs);
+    if ( rc != X86EMUL_OKAY )
+        return rc;
+
+    rc = decode_vmx_inst(regs, &decode);
+    if ( rc != X86EMUL_OKAY )
+        return rc;
+
+    ASSERT(decode.type == VMX_INST_MEMREG_TYPE_MEMORY);
+    rc = hvm_copy_from_guest_virt(&gpa, decode.mem, decode.len, 0);

Is it guaranteed that decode.len is always <= sizeof gpa here, even with a malicious guest?

Cheers,

Tim.

Signature · 3 lines
Tim Deegan <[email protected]>
Principal Software Engineer, XenServer Engineering
Citrix Systems UK Ltd.  (Company #02937203, SL9 0BG)
#24 lore raw

RE: [PATCH 06/16] vmx: nest: handling VMX instruction exits

Quoted 40 lines · Tim Deegan
Quoted 39 lines · Qing He

diff -r f1c1d3077337 xen/arch/x86/hvm/vmx/nest.c

xen/arch/x86/hvm/vmx/nest.c new file +38 -0
+/*
+ * VMX instructions support functions
+ */
+
+enum vmx_regs_enc {
+    VMX_REG_RAX,
+    VMX_REG_RCX,
+    VMX_REG_RDX,
+    VMX_REG_RBX,
+    VMX_REG_RSP,
+    VMX_REG_RBP,
+    VMX_REG_RSI,
+    VMX_REG_RDI,
+#ifdef CONFIG_X86_64
+    VMX_REG_R8,
+    VMX_REG_R9,
+    VMX_REG_R10,
+    VMX_REG_R11,
+    VMX_REG_R12,
+    VMX_REG_R13,
+    VMX_REG_R14,
+    VMX_REG_R15,
+#endif
+};
+
+enum vmx_sregs_enc {
+    VMX_SREG_ES,
+    VMX_SREG_CS,
+    VMX_SREG_SS,
+    VMX_SREG_DS,
+    VMX_SREG_FS,
+    VMX_SREG_GS,
+};
+
+enum x86_segment sreg_to_index[] = {
+    [VMX_SREG_ES] = x86_seg_es,
+    [VMX_SREG_CS] = x86_seg_cs,
+    [VMX_SREG_SS] = x86_seg_ss,
Quoted 8 lines

xen/arch/x86/hvm/vmx/nest.c line 73

Quoted 4 lines
xen/arch/x86/hvm/vmx/nest.c new file +4 -0
+    [VMX_SREG_DS] = x86_seg_ds,
+    [VMX_SREG_FS] = x86_seg_fs,
+    [VMX_SREG_GS] = x86_seg_gs,
+};

Since you dislike adding new namespaces and translations, I'm sure you can get rid of these. :) It might even simplify some of the macros below.

True, some dupcation here. Regarding following definition in x86_emulate.c, we can reuse.

static enum x86_segment
decode_segment(uint8_t modrm_reg)

{
    switch ( modrm_reg )
    {
    case 0: return x86_seg_es;
    case 1: return x86_seg_cs;
    case 2: return x86_seg_ss;
    case 3: return x86_seg_ds;
    case 4: return x86_seg_fs;
    case 5: return x86_seg_gs;
    default: break;
    }
    return decode_segment_failed;
}

BTW, should we use MACROs rather than digital # here? and can we modify x86_segment structure to use same naming space?

Quoted 140 lines
Quoted 119 lines
xen/arch/x86/hvm/vmx/nest.c new file +14 -0
+union vmx_inst_info {
+    struct {
+        unsigned int scaling           :2; /* bit 0-1 */
+        unsigned int __rsvd0           :1; /* bit 2 */
+        unsigned int reg1              :4; /* bit 3-6 */
+        unsigned int addr_size         :3; /* bit 7-9 */
+        unsigned int memreg            :1; /* bit 10 */
+        unsigned int __rsvd1           :4; /* bit 11-14 */
+        unsigned int segment           :3; /* bit 15-17 */
+        unsigned int index_reg         :4; /* bit 18-21 */
+        unsigned int index_reg_invalid :1; /* bit 22 */
+        unsigned int base_reg          :4; /* bit 23-26 */
+        unsigned int base_reg_invalid  :1; /* bit 27 */
+        unsigned int reg2              :4; /* bit 28-31 */ +    }

fields; + u32 word;

xen/arch/x86/hvm/vmx/nest.c new file +92 -0
+};
+
+struct vmx_inst_decoded {
+#define VMX_INST_MEMREG_TYPE_MEMORY 0
+#define VMX_INST_MEMREG_TYPE_REG    1
+    int type;
+    union {
+        struct {
+            unsigned long mem;
+            unsigned int  len;
+        };
+        enum vmx_regs_enc reg1;
+    };
+
+    enum vmx_regs_enc reg2;
+};
+
+enum vmx_ops_result {
+    VMSUCCEED,
+    VMFAIL_VALID,
+    VMFAIL_INVALID,
+};
+
+#define CASE_SET_REG(REG, reg)      \
+    case VMX_REG_ ## REG: regs->reg = value; break
+#define CASE_GET_REG(REG, reg)      \
+    case VMX_REG_ ## REG: value = regs->reg; break +
+#define CASE_EXTEND_SET_REG         \
+    CASE_EXTEND_REG(S)
+#define CASE_EXTEND_GET_REG         \
+    CASE_EXTEND_REG(G)
+
+#ifdef __i386__
+#define CASE_EXTEND_REG(T)
+#else
+#define CASE_EXTEND_REG(T)          \
+    CASE_ ## T ## ET_REG(R8, r8);   \
+    CASE_ ## T ## ET_REG(R9, r9);   \
+    CASE_ ## T ## ET_REG(R10, r10); \
+    CASE_ ## T ## ET_REG(R11, r11); \
+    CASE_ ## T ## ET_REG(R12, r12); \
+    CASE_ ## T ## ET_REG(R13, r13); \
+    CASE_ ## T ## ET_REG(R14, r14); \
+    CASE_ ## T ## ET_REG(R15, r15)
+#endif
+
+static unsigned long reg_read(struct cpu_user_regs *regs,
+                              enum vmx_regs_enc index) +{
+    unsigned long value = 0;
+
+    switch ( index ) {
+    CASE_GET_REG(RAX, eax);
+    CASE_GET_REG(RCX, ecx);
+    CASE_GET_REG(RDX, edx);
+    CASE_GET_REG(RBX, ebx);
+    CASE_GET_REG(RBP, ebp);
+    CASE_GET_REG(RSI, esi);
+    CASE_GET_REG(RDI, edi);
+    CASE_GET_REG(RSP, esp);
+    CASE_EXTEND_GET_REG;
+    default:
+        break;
+    }
+
+    return value;
+}
+
+static void reg_write(struct cpu_user_regs *regs,
+                      enum vmx_regs_enc index,
+                      unsigned long value)
+{
+    switch ( index ) {
+    CASE_SET_REG(RAX, eax);
+    CASE_SET_REG(RCX, ecx);
+    CASE_SET_REG(RDX, edx);
+    CASE_SET_REG(RBX, ebx);
+    CASE_SET_REG(RBP, ebp);
+    CASE_SET_REG(RSI, esi);
+    CASE_SET_REG(RDI, edi);
+    CASE_SET_REG(RSP, esp);
+    CASE_EXTEND_SET_REG;
+    default:
+        break;
+    }
+}
+
+static int decode_vmx_inst(struct cpu_user_regs *regs,
+                           struct vmx_inst_decoded *decode) +{
+    struct vcpu *v = current;
+    union vmx_inst_info info;
+    struct segment_register seg;
+    unsigned long base, index, seg_base, disp, offset; +    int

scale; +

xen/arch/x86/hvm/vmx/nest.c new file +10 -0
+    info.word = __vmread(VMX_INSTRUCTION_INFO);
+
+    if ( info.fields.memreg ) {
+        decode->type = VMX_INST_MEMREG_TYPE_REG;
+        decode->reg1 = info.fields.reg1;
+    }
+    else
+    {
+        decode->type = VMX_INST_MEMREG_TYPE_MEMORY;
+        hvm_get_segment_register(v,
Quoted 2 lines

sreg_to_index[info.fields.segment], &seg); + /* TODO: segment
type check */

Indeed. :)

xen/arch/x86/hvm/vmx/nest.c line 212

Quoted 13 lines
xen/arch/x86/hvm/vmx/nest.c new file +13 -0
+        seg_base = seg.base;
+
+        base = info.fields.base_reg_invalid ? 0 :
+            reg_read(regs, info.fields.base_reg);
+
+        index = info.fields.index_reg_invalid ? 0 :
+            reg_read(regs, info.fields.index_reg); +
+        scale = 1 << info.fields.scaling;
+
+        disp = __vmread(EXIT_QUALIFICATION);
+
+        offset = base + index * scale + disp;
+        if ( offset > seg.limit )

DYM if ( offset + len > limit )?

Quoted 2 lines

Would it be worth trying to fold this into the existing x86_emulate code, which already has careful memory access checks?

Can you give more details? Re-construct hvm_virtual_to_linear_addr?

Quoted 291 lines
Quoted 86 lines
xen/arch/x86/hvm/vmx/nest.c new file +24 -0
+            goto gp_fault;
+
+        decode->mem = seg_base + base + index * scale + disp;
+        decode->len = 1 << (info.fields.addr_size + 1); +    }
+
+    decode->reg2 = info.fields.reg2;
+
+    return X86EMUL_OKAY;
+
+gp_fault:
+    hvm_inject_exception(TRAP_gp_fault, 0, 0);
+    return X86EMUL_EXCEPTION;
+}
+
+static int vmx_inst_check_privilege(struct cpu_user_regs *regs) +{
+    struct vcpu *v = current;
+    struct segment_register cs;
+
+    hvm_get_segment_register(v, x86_seg_cs, &cs);
+
+    if ( !(v->arch.hvm_vcpu.guest_cr[0] & X86_CR0_PE) ||
+         !(v->arch.hvm_vcpu.guest_cr[4] & X86_CR4_VMXE) ||
+         (regs->eflags & X86_EFLAGS_VM) ||
+         (hvm_long_mode_enabled(v) && cs.attr.fields.l == 0) ) +

goto invalid_op; +

xen/arch/x86/hvm/vmx/nest.c new file +15 -0
+    if ( (cs.sel & 3) > 0 )
+        goto gp_fault;
+
+    return X86EMUL_OKAY;
+
+invalid_op:
+    hvm_inject_exception(TRAP_invalid_op, 0, 0);
+    return X86EMUL_EXCEPTION;
+
+gp_fault:
+    hvm_inject_exception(TRAP_gp_fault, 0, 0);
+    return X86EMUL_EXCEPTION;
+}
+
+static void vmreturn(struct cpu_user_regs *regs, enum

vmx_ops_result res) +{ + unsigned long eflags = regs->eflags;
+ unsigned long mask = X86_EFLAGS_CF | X86_EFLAGS_PF |
X86_EFLAGS_AF | + X86_EFLAGS_ZF |
X86_EFLAGS_SF | X86_EFLAGS_OF; + + eflags &= ~mask;

xen/arch/x86/hvm/vmx/nest.c new file +25 -0
+
+    switch ( res ) {
+    case VMSUCCEED:
+        break;
+    case VMFAIL_VALID:
+        /* TODO: error number, useful for guest VMM debugging */
+        eflags |= X86_EFLAGS_ZF;
+        break;
+    case VMFAIL_INVALID:
+    default:
+        eflags |= X86_EFLAGS_CF;
+        break;
+    }
+
+    regs->eflags = eflags;
+}
+
+static int __clear_current_vvmcs(struct vmx_nest_struct *nest) +{
+    int rc;
+
+    if ( nest->svmcs )
+        __vmpclear(virt_to_maddr(nest->svmcs));
+
+#if !CONFIG_VVMCS_MAPPING
+    rc = hvm_copy_to_guest_phys(nest->gvmcs_pa, nest->vvmcs,

PAGE_SIZE); + if ( rc != HVMCOPY_okay )

xen/arch/x86/hvm/vmx/nest.c new file +15 -0
+        return X86EMUL_EXCEPTION;
+#endif
+
+    nest->vmcs_valid = 0;
+
+    return X86EMUL_OKAY;
+}
+
+/*
+ * VMX instructions handling
+ */
+
+int vmx_nest_handle_vmxon(struct cpu_user_regs *regs) +{
+    struct vcpu *v = current;
+    struct vmx_nest_struct *nest = &v->arch.hvm_vmx.nest;

xen/arch/x86/hvm/vmx/nest.c line 314

Quoted 10 lines
xen/arch/x86/hvm/vmx/nest.c new file +10 -0
+    struct vmx_inst_decoded decode;
+    unsigned long gpa = 0;
+    int rc;
+
+    if ( !is_nested_avail(v->domain) )
+        goto invalid_op;
+
+    rc = vmx_inst_check_privilege(regs);
+    if ( rc != X86EMUL_OKAY )
+        return rc;

I think you could fold these checks and the goto target into decode_vmx_inst(), just to avoid repeating them in every vmx_nest_handle_* function.

Quoted 57 lines
xen/arch/x86/hvm/vmx/nest.c new file +17 -0
+    rc = decode_vmx_inst(regs, &decode);
+    if ( rc != X86EMUL_OKAY )
+        return rc;
+
+    ASSERT(decode.type == VMX_INST_MEMREG_TYPE_MEMORY);
+    rc = hvm_copy_from_guest_virt(&gpa, decode.mem, decode.len, 0);
+    if ( rc != HVMCOPY_okay )
+        return X86EMUL_EXCEPTION;
+
+    nest->guest_vmxon_pa = gpa;
+    nest->gvmcs_pa = 0;
+    nest->vmcs_valid = 0;
+#if !CONFIG_VVMCS_MAPPING
+    nest->vvmcs = alloc_xenheap_page();
+    if ( !nest->vvmcs )
+    {
+        gdprintk(XENLOG_ERR, "nest: allocation for virtual vmcs

failed\n"); + vmreturn(regs, VMFAIL_INVALID);

xen/arch/x86/hvm/vmx/nest.c new file +7 -0
+        goto out;
+    }
+#endif
+    nest->svmcs = alloc_xenheap_page();
+    if ( !nest->svmcs )
+    {
+        gdprintk(XENLOG_ERR, "nest: allocation for shadow vmcs

failed\n"); + free_xenheap_page(nest->vvmcs);

xen/arch/x86/hvm/vmx/nest.c new file +26 -0
+        vmreturn(regs, VMFAIL_INVALID);
+        goto out;
+    }
+
+    /*
+     * `fork' the host vmcs to shadow_vmcs
+     * vmcs_lock is not needed since we are on current +     */
+    nest->hvmcs = v->arch.hvm_vmx.vmcs;
+    __vmpclear(virt_to_maddr(nest->hvmcs));
+    memcpy(nest->svmcs, nest->hvmcs, PAGE_SIZE);
+    __vmptrld(virt_to_maddr(nest->hvmcs));
+    v->arch.hvm_vmx.launched = 0;
+
+    vmreturn(regs, VMSUCCEED);
+
+out:
+    return X86EMUL_OKAY;
+
+invalid_op:
+    hvm_inject_exception(TRAP_invalid_op, 0, 0);
+    return X86EMUL_EXCEPTION;
+}
+
+int vmx_nest_handle_vmxoff(struct cpu_user_regs *regs) +{
+    struct vcpu *v = current;
+    struct vmx_nest_struct *nest = &v->arch.hvm_vmx.nest; +    int

rc; +

xen/arch/x86/hvm/vmx/nest.c new file +4 -0
+    if ( unlikely(!nest->guest_vmxon_pa) )
+        goto invalid_op;
+
+    rc = vmx_inst_check_privilege(regs);

xen/arch/x86/hvm/vmx/nest.c line 385

Quoted 10 lines
xen/arch/x86/hvm/vmx/nest.c new file +10 -0
+    if ( rc != X86EMUL_OKAY )
+        return rc;
+
+    nest->guest_vmxon_pa = 0;
+    __vmpclear(virt_to_maddr(nest->svmcs));
+
+#if !CONFIG_VVMCS_MAPPING
+    free_xenheap_page(nest->vvmcs);
+#endif
+    free_xenheap_page(nest->svmcs);

These also need to be freed on domain teardown.

Quoted 120 lines
xen/arch/x86/hvm/vmx/nest.c new file +12 -0
+    vmreturn(regs, VMSUCCEED);
+    return X86EMUL_OKAY;
+
+invalid_op:
+    hvm_inject_exception(TRAP_invalid_op, 0, 0);
+    return X86EMUL_EXCEPTION;
+}
+
+int vmx_nest_handle_vmptrld(struct cpu_user_regs *regs) +{
+    struct vcpu *v = current;
+    struct vmx_inst_decoded decode;
+    struct vmx_nest_struct *nest = &v->arch.hvm_vmx.nest; +

unsigned long gpa = 0; + int rc;

xen/arch/x86/hvm/vmx/nest.c new file +43 -0
+
+    if ( unlikely(!nest->guest_vmxon_pa) )
+        goto invalid_op;
+
+    rc = vmx_inst_check_privilege(regs);
+    if ( rc != X86EMUL_OKAY )
+        return rc;
+
+    rc = decode_vmx_inst(regs, &decode);
+    if ( rc != X86EMUL_OKAY )
+        return rc;
+
+    ASSERT(decode.type == VMX_INST_MEMREG_TYPE_MEMORY);
+    rc = hvm_copy_from_guest_virt(&gpa, decode.mem, decode.len, 0);
+    if ( rc != HVMCOPY_okay )
+        return X86EMUL_EXCEPTION;
+
+    if ( gpa == nest->guest_vmxon_pa || gpa & 0xfff ) +    {
+        vmreturn(regs, VMFAIL_INVALID);
+        goto out;
+    }
+
+    if ( nest->gvmcs_pa != gpa )
+    {
+        if ( nest->vmcs_valid )
+        {
+            rc = __clear_current_vvmcs(nest);
+            if ( rc != X86EMUL_OKAY )
+                return rc;
+        }
+        nest->gvmcs_pa = gpa;
+        ASSERT(nest->vmcs_valid == 0);
+    }
+
+
+    if ( !nest->vmcs_valid )
+    {
+#if CONFIG_VVMCS_MAPPING
+        unsigned long mfn;
+        p2m_type_t p2mt;
+
+        mfn = mfn_x(gfn_to_mfn(p2m_get_hostp2m(v->domain),
+                               nest->gvmcs_pa >> PAGE_SHIFT,

&p2mt)); + nest->vvmcs = map_domain_page_global(mfn); +#else
+ rc = hvm_copy_from_guest_phys(nest->vvmcs, nest->gvmcs_pa,
PAGE_SIZE); + if ( rc != HVMCOPY_okay )

xen/arch/x86/hvm/vmx/nest.c new file +19 -0
+            return X86EMUL_EXCEPTION;
+#endif
+        nest->vmcs_valid = 1;
+    }
+
+    vmreturn(regs, VMSUCCEED);
+
+out:
+    return X86EMUL_OKAY;
+
+invalid_op:
+    hvm_inject_exception(TRAP_invalid_op, 0, 0);
+    return X86EMUL_EXCEPTION;
+}
+
+int vmx_nest_handle_vmptrst(struct cpu_user_regs *regs) +{
+    struct vcpu *v = current;
+    struct vmx_inst_decoded decode;
+    struct vmx_nest_struct *nest = &v->arch.hvm_vmx.nest; +

unsigned long gpa = 0; + int rc;

xen/arch/x86/hvm/vmx/nest.c new file +31 -0
+
+    if ( unlikely(!nest->guest_vmxon_pa) )
+        goto invalid_op;
+
+    rc = vmx_inst_check_privilege(regs);
+    if ( rc != X86EMUL_OKAY )
+        return rc;
+
+    rc = decode_vmx_inst(regs, &decode);
+    if ( rc != X86EMUL_OKAY )
+        return rc;
+
+    ASSERT(decode.type == VMX_INST_MEMREG_TYPE_MEMORY); +
+    gpa = nest->gvmcs_pa;
+
+    rc = hvm_copy_to_guest_virt(decode.mem, &gpa, decode.len, 0);
+    if ( rc != HVMCOPY_okay )
+        return X86EMUL_EXCEPTION;
+
+    vmreturn(regs, VMSUCCEED);
+    return X86EMUL_OKAY;
+
+invalid_op:
+    hvm_inject_exception(TRAP_invalid_op, 0, 0);
+    return X86EMUL_EXCEPTION;
+}
+
+int vmx_nest_handle_vmclear(struct cpu_user_regs *regs) +{
+    struct vcpu *v = current;
+    struct vmx_inst_decoded decode;
+    struct vmx_nest_struct *nest = &v->arch.hvm_vmx.nest; +

unsigned long gpa = 0; + int rc;

xen/arch/x86/hvm/vmx/nest.c new file +9 -0
+
+    if ( unlikely(!nest->guest_vmxon_pa) )
+        goto invalid_op;
+
+    rc = vmx_inst_check_privilege(regs);
+    if ( rc != X86EMUL_OKAY )
+        return rc;
+
+    rc = decode_vmx_inst(regs, &decode);
Quoted 8 lines

xen/arch/x86/hvm/vmx/nest.c line 321

Quoted 5 lines
xen/arch/x86/hvm/vmx/nest.c new file +5 -0
+    if ( rc != X86EMUL_OKAY )
+        return rc;
+
+    ASSERT(decode.type == VMX_INST_MEMREG_TYPE_MEMORY);
+    rc = hvm_copy_from_guest_virt(&gpa, decode.mem, decode.len, 0);

Is it guaranteed that decode.len is always <= sizeof gpa here, even with a malicious guest?

Reusing hvm_virtual_to_linear_addr w/ consideration of last byte may be the best choice :)

Thx, Eddie

#25 lore raw

Re: [PATCH 06/16] vmx: nest: handling VMX instruction exits

Quoted 13 lines · Eddie
Quoted 4 lines
Quoted 4 lines
xen/arch/x86/hvm/vmx/nest.c new file +4 -0
+enum x86_segment sreg_to_index[] = {
+    [VMX_SREG_ES] = x86_seg_es,
+    [VMX_SREG_CS] = x86_seg_cs,
+    [VMX_SREG_SS] = x86_seg_ss,
Quoted 8 lines

xen/arch/x86/hvm/vmx/nest.c line 73

Quoted 4 lines
xen/arch/x86/hvm/vmx/nest.c new file +4 -0
+    [VMX_SREG_DS] = x86_seg_ds,
+    [VMX_SREG_FS] = x86_seg_fs,
+    [VMX_SREG_GS] = x86_seg_gs,
+};

Since you dislike adding new namespaces and translations, I'm sure you can get rid of these. :) It might even simplify some of the macros below.

Quoted 2 lines

True, some dupcation here. Regarding following definition in x86_emulate.c, we can reuse.

AFAICS if you must have your own extra instruction decoder, a few register translation definitions and arrays is the least of it really. I'd rather keep x86_emulate clean and separate rather than become intertwined with another emulator.

What is wrong with simply extending x86_emulate to handle these VMX-related instructions? We've dealt with emulators provided by Intel guys in the past and frankly they were full of holes.

-- Keir

#26 lore raw

RE: [PATCH 06/16] vmx: nest: handling VMX instruction exits

Quoted 18 lines · Keir Fraser
Quoted 13 lines · Eddie
Quoted 4 lines
Quoted 4 lines
xen/arch/x86/hvm/vmx/nest.c new file +4 -0
+enum x86_segment sreg_to_index[] = {
+    [VMX_SREG_ES] = x86_seg_es,
+    [VMX_SREG_CS] = x86_seg_cs,
+    [VMX_SREG_SS] = x86_seg_ss,
Quoted 8 lines

xen/arch/x86/hvm/vmx/nest.c line 73

Quoted 4 lines
xen/arch/x86/hvm/vmx/nest.c new file +4 -0
+    [VMX_SREG_DS] = x86_seg_ds,
+    [VMX_SREG_FS] = x86_seg_fs,
+    [VMX_SREG_GS] = x86_seg_gs,
+};

Since you dislike adding new namespaces and translations, I'm sure you can get rid of these. :) It might even simplify some of the macros below.

Quoted 2 lines

True, some dupcation here. Regarding following definition in x86_emulate.c, we can reuse.

Quoted 9 lines

AFAICS if you must have your own extra instruction decoder, a few register translation definitions and arrays is the least of it really. I'd rather keep x86_emulate clean and separate rather than become intertwined with another emulator.

What is wrong with simply extending x86_emulate to handle these VMX-related instructions? We've dealt with emulators provided by Intel guys in the past and frankly they were full of holes.

Certainly fine to move those VMX instruction emulation to hvm/emulate.c as if you don't think that is VMX specific :)

Will do.

Thx, Eddie

#27 lore raw

Re: [PATCH 06/16] vmx: nest: handling VMX instruction exits

Quoted 6 lines · Eddie
Quoted 4 lines

What is wrong with simply extending x86_emulate to handle these VMX-related instructions? We've dealt with emulators provided by Intel guys in the past and frankly they were full of holes.

Certainly fine to move those VMX instruction emulation to hvm/emulate.c as if you don't think that is VMX specific :)

It's the right place to put all instruction emulation, if at all possible. You will then presumably require at least one or two call-back hooks to caller context, at least to read/write VMCS, and that would be the place to determine whether these VMX instructions are executable. For example, SVM and PV emulation contexts would either leave the VMX callback hooks as NULL, and/or there will be checks for is-nested-VMX-guest in the VMX callback hooks, injecting #UD otherwise.

The main trick with x86_emulate extensions is determining the correct neat small set of callback hooks to add, which is somewhat driven by deciding what should be emulated within x86_emulate and what should be left without for implementation in the caller's context.

-- Keir

#28 lore raw

Re: [PATCH 06/16] vmx: nest: handling VMX instruction exits

Quoted 16 lines · On Wednesday 15 September 2010 09:31:13 Keir Fraser wrote:
Quoted 6 lines · Eddie
Quoted 3 lines

What is wrong with simply extending x86_emulate to handle these VMX-related instructions? We've dealt with emulators provided by Intel guys in the past and frankly they were full of holes.

Certainly fine to move those VMX instruction emulation to hvm/emulate.c as if you don't think that is VMX specific :)

It's the right place to put all instruction emulation, if at all possible. You will then presumably require at least one or two call-back hooks to caller context, at least to read/write VMCS, and that would be the place to determine whether these VMX instructions are executable. For example, SVM and PV emulation contexts would either leave the VMX callback hooks as NULL, and/or there will be checks for is-nested-VMX-guest in the VMX callback hooks, injecting #UD otherwise.

Quoted 4 lines

The main trick with x86_emulate extensions is determining the correct neat small set of callback hooks to add, which is somewhat driven by deciding what should be emulated within x86_emulate and what should be left without for implementation in the caller's context.

There is a case where the host must emulate an instruction of the l2 guest when the l1 guest doesn't intercept it.

When the vcpu is in guest mode, the fields in struct hvm_vcpu and guest_cpu_user_regs() represent the l2 guest state in my patch series.

That way the instruction emulator works out-of-the box. You need to add instructions to the emulator that are missing there.

Christoph

Signature · 6 lines
---to satisfy European Law for business letters:
Advanced Micro Devices GmbH
Einsteinring 24, 85609 Dornach b. Muenchen
Geschaeftsfuehrer: Alberto Bozzo, Andrew Bowd
Sitz: Dornach, Gemeinde Aschheim, Landkreis Muenchen
Registergericht Muenchen, HRB Nr. 43632
#29 lore raw

Re: [PATCH 06/16] vmx: nest: handling VMX instruction exits

Quoted 5 lines · Christoph Egger
Quoted 4 lines

The main trick with x86_emulate extensions is determining the correct neat small set of callback hooks to add, which is somewhat driven by deciding what should be emulated within x86_emulate and what should be left without for implementation in the caller's context.

Quoted 7 lines

There is a case where the host must emulate an instruction of the l2 guest when the l1 guest doesn't intercept it.

When the vcpu is in guest mode, the fields in struct hvm_vcpu and guest_cpu_user_regs() represent the l2 guest state in my patch series.

That way the instruction emulator works out-of-the box.

Well in this specific case, all VMX-related instructions executed by an L2 guest would properly cause vmexit to the L1 guest for emulation there. We wouldn't want to emulate in Xen.

But yes I can see that emulation of L2 guest instructions is needed in some other cases. Like instructions performing I/O in areas which L1 thinks it has given L2 direct unmediated access to, but which Xen is actually filtering or emulating.

-- Keir

Quoted 1 line

You need to add instructions to the emulator that are missing there.

#30 lore raw

RE: [PATCH 06/16] vmx: nest: handling VMX instruction exits

Quoted 16 lines · Keir Fraser
Quoted 13 lines · Christoph Egger
Quoted 5 lines

The main trick with x86_emulate extensions is determining the correct neat small set of callback hooks to add, which is somewhat driven by deciding what should be emulated within x86_emulate and what should be left without for implementation in the caller's context.

There is a case where the host must emulate an instruction of the l2 guest when the l1 guest doesn't intercept it.

When the vcpu is in guest mode, the fields in struct hvm_vcpu and guest_cpu_user_regs() represent the l2 guest state in my patch series.

Quoted 5 lines
Quoted 1 line

That way the instruction emulator works out-of-the box.

Well in this specific case, all VMX-related instructions executed by an L2 guest would properly cause vmexit to the L1 guest for emulation there. We wouldn't want to emulate in Xen.

Yes, in nested VMX side, we L0 VMM won't emulate L2 VMX instruction.

Quoted 5 lines

But yes I can see that emulation of L2 guest instructions is needed in some other cases. Like instructions performing I/O in areas which L1 thinks it has given L2 direct unmediated access to, but which Xen is actually filtering or emulating.

That may be an issue in plan when we support virtual VT-d for nested I/O performance. But not now :) I suggest we leave that to future at least for nested VMX side where L0 VMM doesn't directly emulate L0 VMM instruction.

We can see if there are other needs for that kind of case.

Thx, Eddie

#31 lore raw

Re: [PATCH 06/16] vmx: nest: handling VMX instruction exits

Quoted 7 lines · Eddie
Quoted 4 lines

But yes I can see that emulation of L2 guest instructions is needed in some other cases. Like instructions performing I/O in areas which L1 thinks it has given L2 direct unmediated access to, but which Xen is actually filtering or emulating.

That may be an issue in plan when we support virtual VT-d for nested I/O performance. But not now :)

Actually it is an issue now. This has nothing to do with VT-d (ie. IOMMU, irq remapping, etc) but with basic core VMX functionality -- per I/O port direct execute versus vmexit; per virtual-address page direct access versus #PF vmexit; per physical-frame direct access versus nexted-paging vmexit. In any of these cases the L1 may think it has given direct unfettered access to the L2, but L0 (Xen) is actually blocking it. In this case any resulting required instruction emulations have to be performed by Xen on behalf of the L2, without L1's help or knowledge. Consider that even in Xen we give all HVM guests direct access to thinks like port 0x80 for perf reasons. Maybe a VMM running in L1 would give direct access to more even than that -- in such cases Xen must be able to emulate those 'direct' accesses.

Now this shouldn't be hard to arrange anyhow. When you vmexit to Xen from running L2 guest, the saved general-purpose CPU state will be the L2's state, and that is what you would want x86_emulate to see. But it does require some thought, and it is merely not an extension to be dealt with later. It is core VMX stuff and hence core nested VMX stuff.

-- Keir

Quoted 4 lines

I suggest we leave that to future at least for nested VMX side where L0 VMM doesn't directly emulate L0 VMM instruction.

We can see if there are other needs for that kind of case.

#32 lore raw

RE: [PATCH 06/16] vmx: nest: handling VMX instruction exits

Quoted 7 lines · Keir Fraser
Quoted 5 lines · Eddie
Quoted 4 lines

But yes I can see that emulation of L2 guest instructions is needed in some other cases. Like instructions performing I/O in areas which L1 thinks it has given L2 direct unmediated access to, but which Xen is actually filtering or emulating.

Quoted 6 lines
Quoted 2 lines

That may be an issue in plan when we support virtual VT-d for nested I/O performance. But not now :)

Actually it is an issue now. This has nothing to do with VT-d (ie. IOMMU, irq remapping, etc) but with basic core VMX functionality -- per I/O port direct execute versus vmexit; per virtual-address page

I see, for the I/O port, right now we are letting L1 handle it though it doesn't expect to :( How about to remove the capability of CPU_BASED_ACTIVATE_IO_BITMAP in L1 VMM for now to focus on framework?

Quoted 2 lines

direct access versus #PF vmexit; per physical-frame direct access versus nexted-paging vmexit. In any of these cases the L1 may think

Didn't quit catch. The memory direct access is always guarded by L0 shadow or nested EPT/NPT. Missing something?

Quoted 7 lines

it has given direct unfettered access to the L2, but L0 (Xen) is actually blocking it. In this case any resulting required instruction emulations have to be performed by Xen on behalf of the L2, without L1's help or knowledge. Consider that even in Xen we give all HVM guests direct access to thinks like port 0x80 for perf reasons. Maybe a VMM running in L1 would give direct access to more even than that -- in such cases Xen must be able to emulate those 'direct' accesses.

Agree!

Quoted 7 lines

Now this shouldn't be hard to arrange anyhow. When you vmexit to Xen from running L2 guest, the saved general-purpose CPU state will be the L2's state, and that is what you would want x86_emulate to see. But it does require some thought, and it is merely not an extension to be dealt with later. It is core VMX stuff and hence core nested VMX stuff.

Yes. For I/O issue, emulating by L0 should be fine.

Thx, Eddie

#33 lore raw

Re: [PATCH 06/16] vmx: nest: handling VMX instruction exits

Quoted 8 lines · Eddie
Quoted 3 lines

Actually it is an issue now. This has nothing to do with VT-d (ie. IOMMU, irq remapping, etc) but with basic core VMX functionality -- per I/O port direct execute versus vmexit; per virtual-address page

I see, for the I/O port, right now we are letting L1 handle it though it doesn't expect to :(
How about to remove the capability of CPU_BASED_ACTIVATE_IO_BITMAP in L1 VMM for now to focus on framework?

Well. It'd be better if just worked really, wouldn't it? :-) How hard can it be?

Quoted 5 lines
Quoted 2 lines

direct access versus #PF vmexit; per physical-frame direct access versus nexted-paging vmexit. In any of these cases the L1 may think

Didn't quit catch. The memory direct access is always guarded by L0 shadow or nested EPT/NPT. Missing something?

L1 gives L2 direct access to, say, HPET (memory-mapped IO) which is actually (unknown to L1) a virtual HPET emulated by Xen? Yeah, okay, that may be more unlikely to happen in practice but it *is* allowable by the architecture and it *should* be supported.

I would be inclined to add test cases for nestedhvm to hvmloader (we already test various other tricky things in there) to test these kinds of cases. Broadly speaking it's just a case of walking VVMCS structures to check IO_BITMAP, or shadow pagetables, or EPT, and jump to the emulator with L2 state if the L1 would have permitted execution. It's really a core bit of logic in properly doing nested VMX. The unfortunate thing is that the necessary checks will slow down nested-hvm further, I guess, but perhaps it's not too bad?

-- Keir

#34 lore raw

RE: [PATCH 06/16] vmx: nest: handling VMX instruction exits

Quoted 6 lines · Keir Fraser
Quoted 4 lines · Eddie
Quoted 3 lines

Actually it is an issue now. This has nothing to do with VT-d (ie. IOMMU, irq remapping, etc) but with basic core VMX functionality -- per I/O port direct execute versus vmexit; per virtual-address page

Quoted 6 lines
Quoted 3 lines

I see, for the I/O port, right now we are letting L1 handle it though it doesn't expect to :( How about to remove the capability of CPU_BASED_ACTIVATE_IO_BITMAP in L1 VMM for now to focus on framework?

Well. It'd be better if just worked really, wouldn't it? :-) How hard can it be?

You are right. It is easy to do, but we have dillemma to either write-protect guest I/O bitmap page, or have to create the shadow I/O bitmap at each vmresume of L2 guest.

Currently we are injecting to L1 guest, but may be not correct in theory. For now, VMX can trap L2 guest I/O and emulate them in L0, we can revisit some time later to see if we need write-protection of guest I/O bitmap page :)

But, yes, L0 VMM needs to emulate L2 instruction here :)

MSR bitmap will have similar situation. Currently VMX removed MSR bitmap features, but we may do like I/O bitmap to write-protect the page, though it is slightly complicated.

Quoted 11 lines
Quoted 6 lines
Quoted 3 lines

direct access versus #PF vmexit; per physical-frame direct access versus nexted-paging vmexit. In any of these cases the L1 may think

Didn't quit catch. The memory direct access is always guarded by L0 shadow or nested EPT/NPT. Missing something?

L1 gives L2 direct access to, say, HPET (memory-mapped IO) which is actually (unknown to L1) a virtual HPET emulated by Xen? Yeah, okay, that may be more unlikely to happen in practice but it *is* allowable by the architecture and it *should* be supported.

Agree, thanks!

Quoted 9 lines

I would be inclined to add test cases for nestedhvm to hvmloader (we already test various other tricky things in there) to test these kinds of cases. Broadly speaking it's just a case of walking VVMCS structures to check IO_BITMAP, or shadow pagetables, or EPT, and jump to the emulator with L2 state if the L1 would have permitted execution. It's really a core bit of logic in properly doing nested VMX. The unfortunate thing is that the necessary checks will slow down nested-hvm further, I guess, but perhaps it's not too bad?

Agree. Thanks.
Need write-protection, otherwise generating shadow bitmap is expansive. Checking bitmap at I/O exit is fine.

Eddie

#35 lore raw

Re: [PATCH 06/16] vmx: nest: handling VMX instruction exits

Quoted 8 lines · Eddie
Quoted 8 lines
Quoted 7 lines
Quoted 3 lines

Actually it is an issue now. This has nothing to do with VT-d (ie. IOMMU, irq remapping, etc) but with basic core VMX functionality -- per I/O port direct execute versus vmexit; per virtual-address page

I see, for the I/O port, right now we are letting L1 handle it though it doesn't expect to :( How about to remove the capability of CPU_BASED_ACTIVATE_IO_BITMAP in L1 VMM for now to focus on framework?

Quoted 6 lines
Quoted 2 lines

Well. It'd be better if just worked really, wouldn't it? :-) How hard can it be?

You are right. It is easy to do, but we have dillemma to either write-protect guest I/O bitmap page, or have to create the shadow I/O bitmap at each vmresume of L2 guest.

You need that anyway don't you, regardless of whether you are accurately deciding whether to inject-to-L1 or emulate-L2 on vmexit to L0? Whether you inject or emulate, ports that L1 has disallowed for L2 must be properly represented in the shadow I/O bitmap page.

Quoted 3 lines

Currently we are injecting to L1 guest, but may be not correct in theory. For now, VMX can trap L2 guest I/O and emulate them in L0, we can revisit some time later to see if we need write-protection of guest I/O bitmap page :)

Are you suggesting to always emulate instead of always inject-to-L1? That's still not accurate virtualisation of this VMX feature.

Hmm... Are you currently setting up to always vmexit on I/O port accesses by L2? Even if you are, that doesn't stop you looking at the virtual I/O bitmap from in your L0 vmexit handler, and doing the right thing (emulate versus inject-to-L1).

-- Keir

Quoted 1 line

But, yes, L0 VMM needs to emulate L2 instruction here :)

#36 lore raw

RE: [PATCH 06/16] vmx: nest: handling VMX instruction exits

Quoted 15 lines · Keir Fraser
Quoted 13 lines · Eddie
Quoted 12 lines
Quoted 9 lines
Quoted 4 lines

Actually it is an issue now. This has nothing to do with VT-d (ie. IOMMU, irq remapping, etc) but with basic core VMX functionality -- per I/O port direct execute versus vmexit; per virtual-address page

I see, for the I/O port, right now we are letting L1 handle it though it doesn't expect to :( How about to remove the capability of CPU_BASED_ACTIVATE_IO_BITMAP in L1 VMM for now to focus on framework?

Well. It'd be better if just worked really, wouldn't it? :-) How hard can it be?

Quoted 8 lines
Quoted 3 lines

You are right. It is easy to do, but we have dillemma to either write-protect guest I/O bitmap page, or have to create the shadow I/O bitmap at each vmresume of L2 guest.

You need that anyway don't you, regardless of whether you are accurately deciding whether to inject-to-L1 or emulate-L2 on vmexit to L0? Whether you inject or emulate, ports that L1 has disallowed for L2 must be properly represented in the shadow I/O bitmap page.

VMX has a feature "always exit" for PIO which doesn't use I/O bitmap.

Quoted 8 lines
Quoted 5 lines

Currently we are injecting to L1 guest, but may be not correct in theory. For now, VMX can trap L2 guest I/O and emulate them in L0, we can revisit some time later to see if we need write-protection of guest I/O bitmap page :)

Are you suggesting to always emulate instead of always inject-to-L1? That's still not accurate virtualisation of this VMX feature.

L2 PIO is always exiting to L0. So we wither inject to L1, or emulate it in L0, base on L1 I/O exiting and bitmap setting.

Quoted 3 lines

Hmm... Are you currently setting up to always vmexit on I/O port accesses by L2? Even if you are, that doesn't stop you looking at the

Yes.

Quoted 1 line

virtual I/O bitmap from in your L0 vmexit handler, and doing the

No, we checked the L1 settings.

Quoted 2 lines

right thing (emulate versus inject-to-L1).

BTW, does SVM side already implemented the write-protection of I/O bitmap & MSR bitmap. it seems not.

Thx, Eddie

#37 lore raw

Re: [PATCH 06/16] vmx: nest: handling VMX instruction exits

Quoted 7 lines · Eddie
Quoted 2 lines

Are you suggesting to always emulate instead of always inject-to-L1? That's still not accurate virtualisation of this VMX feature.

L2 PIO is always exiting to L0. So we wither inject to L1, or emulate it in L0, base on L1 I/O exiting and bitmap setting.

Quoted 1 line
Quoted 8 lines
Quoted 2 lines

Hmm... Are you currently setting up to always vmexit on I/O port accesses by L2? Even if you are, that doesn't stop you looking at the

Yes.

Quoted 1 line

virtual I/O bitmap from in your L0 vmexit handler, and doing the

No, we checked the L1 settings.

Right, okay, I think this issue about write-protecting the I/O bitmap and maintaining an accurate shadow bitmap is entirely orthogonal to the issue of emulation versus injection. The former issue is a performance optimisation only afaics, and I wouldn't care about that in a first checkin of nestedhvm.

-- Keir

#38 lore raw

RE: [PATCH 06/16] vmx: nest: handling VMX instruction exits

Quoted 17 lines · Keir Fraser
Quoted 15 lines · Eddie
Quoted 2 lines

Are you suggesting to always emulate instead of always inject-to-L1? That's still not accurate virtualisation of this VMX feature.

L2 PIO is always exiting to L0. So we wither inject to L1, or emulate it in L0, base on L1 I/O exiting and bitmap setting.

Quoted 4 lines

Hmm... Are you currently setting up to always vmexit on I/O port accesses by L2? Even if you are, that doesn't stop you looking at the

Yes.

Quoted 1 line

virtual I/O bitmap from in your L0 vmexit handler, and doing the

Quoted 7 lines
Quoted 1 line

No, we checked the L1 settings.

Right, okay, I think this issue about write-protecting the I/O bitmap and maintaining an accurate shadow bitmap is entirely orthogonal to the issue of emulation versus injection. The former issue is a performance optimisation only afaics, and I wouldn't care about that in a first checkin of nestedhvm.

Yes, that is what I mean. I fixed it locally w/ emulation in L0 + "always exit", and we may enhance by maintaining an accurate bitmap.

Eddie

#39 lore raw

Re: [PATCH 06/16] vmx: nest: handling VMX instruction exits

In reply to Keir Fraser
Quoted 12 lines · On Monday 20 September 2010 10:08:02 Keir Fraser wrote:
Quoted 11 lines · Eddie
Quoted 10 lines
Quoted 7 lines
Quoted 3 lines

Actually it is an issue now. This has nothing to do with VT-d (ie. IOMMU, irq remapping, etc) but with basic core VMX functionality -- per I/O port direct execute versus vmexit; per virtual-address page

I see, for the I/O port, right now we are letting L1 handle it though it doesn't expect to :( How about to remove the capability of CPU_BASED_ACTIVATE_IO_BITMAP in L1 VMM for now to focus on framework?

Well. It'd be better if just worked really, wouldn't it? :-) How hard can it be?

Quoted 8 lines
Quoted 3 lines

You are right. It is easy to do, but we have dillemma to either write-protect guest I/O bitmap page, or have to create the shadow I/O bitmap at each vmresume of L2 guest.

You need that anyway don't you, regardless of whether you are accurately deciding whether to inject-to-L1 or emulate-L2 on vmexit to L0? Whether you inject or emulate, ports that L1 has disallowed for L2 must be properly represented in the shadow I/O bitmap page.

You need to do additional range-checking to determine if the guest actually touched the IO bitmap page in case Xen uses a super page.

Quoted 17 lines
Quoted 5 lines

Currently we are injecting to L1 guest, but may be not correct in theory. For now, VMX can trap L2 guest I/O and emulate them in L0, we can revisit some time later to see if we need write-protection of guest I/O bitmap page :)

Are you suggesting to always emulate instead of always inject-to-L1? That's still not accurate virtualisation of this VMX feature.

Hmm... Are you currently setting up to always vmexit on I/O port accesses by L2? Even if you are, that doesn't stop you looking at the virtual I/O bitmap from in your L0 vmexit handler, and doing the right thing (emulate versus inject-to-L1).

-- Keir

Quoted 1 line

But, yes, L0 VMM needs to emulate L2 instruction here :)

Signature · 6 lines
---to satisfy European Law for business letters:
Advanced Micro Devices GmbH
Einsteinring 24, 85609 Dornach b. Muenchen
Geschaeftsfuehrer: Alberto Bozzo, Andrew Bowd
Sitz: Dornach, Gemeinde Aschheim, Landkreis Muenchen
Registergericht Muenchen, HRB Nr. 43632
#40 lore raw

RE: [PATCH 06/16] vmx: nest: handling VMX instruction exits

Quoted 19 lines · Christoph Egger
Quoted 18 lines · On Monday 20 September 2010 10:08:02 Keir Fraser wrote:
Quoted 16 lines · Eddie
Quoted 5 lines
Quoted 5 lines
Quoted 4 lines

Actually it is an issue now. This has nothing to do with VT-d (ie. IOMMU, irq remapping, etc) but with basic core VMX functionality -- per I/O port direct execute versus vmexit; per virtual-address page

Quoted 7 lines
Quoted 4 lines

I see, for the I/O port, right now we are letting L1 handle it though it doesn't expect to :( How about to remove the capability of CPU_BASED_ACTIVATE_IO_BITMAP in L1 VMM for now to focus on framework?

Well. It'd be better if just worked really, wouldn't it? :-) How hard can it be?

You are right. It is easy to do, but we have dillemma to either write-protect guest I/O bitmap page, or have to create the shadow I/O bitmap at each vmresume of L2 guest.

Quoted 8 lines
Quoted 4 lines

You need that anyway don't you, regardless of whether you are accurately deciding whether to inject-to-L1 or emulate-L2 on vmexit to L0? Whether you inject or emulate, ports that L1 has disallowed for L2 must be properly represented in the shadow I/O bitmap page.

You need to do additional range-checking to determine if the guest actually touched the IO bitmap page in case Xen uses a super page.

We may have many alternatives to this. If we treat this address space as MMIO, we can hook handler for MMIO emulation.

Eddie

author
#41 lore raw

Re: [PATCH 06/16] vmx: nest: handling VMX instruction exits

In reply to Keir Fraser
Quoted 18 lines · Keir Fraser
Quoted 13 lines · Eddie
Quoted 4 lines
Quoted 4 lines
xen/arch/x86/hvm/vmx/nest.c new file +4 -0
+enum x86_segment sreg_to_index[] = {
+    [VMX_SREG_ES] = x86_seg_es,
+    [VMX_SREG_CS] = x86_seg_cs,
+    [VMX_SREG_SS] = x86_seg_ss,
Quoted 8 lines

xen/arch/x86/hvm/vmx/nest.c line 73

Quoted 4 lines
xen/arch/x86/hvm/vmx/nest.c new file +4 -0
+    [VMX_SREG_DS] = x86_seg_ds,
+    [VMX_SREG_FS] = x86_seg_fs,
+    [VMX_SREG_GS] = x86_seg_gs,
+};

Since you dislike adding new namespaces and translations, I'm sure you can get rid of these. :) It might even simplify some of the macros below.

Quoted 2 lines

True, some dupcation here. Regarding following definition in x86_emulate.c, we can reuse.

Quoted 8 lines

AFAICS if you must have your own extra instruction decoder, a few register translation definitions and arrays is the least of it really. I'd rather keep x86_emulate clean and separate rather than become intertwined with another emulator.

What is wrong with simply extending x86_emulate to handle these VMX-related instructions? We've dealt with emulators provided by Intel guys in the past and frankly they were full of holes.

That needs additional callback when handling vmcs and state change, doesn't it? I'm a little worried that it's too vmx-specific to get into x86_emulate, and that's why we used a separate decoder in the first place (I know it's ugly, though).

And if we are to use x86_emulate, is it possible to avoid redecoding the opcode, because exit reason is already there?

Thanks,
Qing

#42 lore raw

Re: [PATCH 06/16] vmx: nest: handling VMX instruction exits

Quoted 8 lines · Qing He
Quoted 3 lines

What is wrong with simply extending x86_emulate to handle these VMX-related instructions? We've dealt with emulators provided by Intel guys in the past and frankly they were full of holes.

That needs additional callback when handling vmcs and state change, doesn't it? I'm a little worried that it's too vmx-specific to get into x86_emulate, and that's why we used a separate decoder in the first place (I know it's ugly, though).

A few VMX-specific callbacks would be fine. Extra callbacks are cheap. Just focus on making the callback interface clean and tidy. I'd *much* rather have VMX-specific callbacks than an extra emulator.

Quoted 2 lines

And if we are to use x86_emulate, is it possible to avoid redecoding the opcode, because exit reason is already there?

If vmexit reason fully decodes the instruction for you then I would agree that skipping x86_emulate could make sense. And then your instruction emulator would be really simple and fast -- vmx_io_instruction() is a good example of this. If you still need to parse the instruction to decode ModRM and the like, then I don't see that the partial decode from vmexit reason saves you much at all, and you might as well go the whole hog and do full decode. I don't see much saving from a hacky middle-ground.

-- Keir

#43 lore raw

RE: [PATCH 06/16] vmx: nest: handling VMX instruction exits

Quoted 11 lines · Keir Fraser
Quoted 8 lines · Qing He
Quoted 3 lines

What is wrong with simply extending x86_emulate to handle these VMX-related instructions? We've dealt with emulators provided by Intel guys in the past and frankly they were full of holes.

That needs additional callback when handling vmcs and state change, doesn't it? I'm a little worried that it's too vmx-specific to get into x86_emulate, and that's why we used a separate decoder in the first place (I know it's ugly, though).

Quoted 8 lines

A few VMX-specific callbacks would be fine. Extra callbacks are cheap. Just focus on making the callback interface clean and tidy. I'd *much* rather have VMX-specific callbacks than an extra emulator.

Quoted 2 lines

And if we are to use x86_emulate, is it possible to avoid redecoding the opcode, because exit reason is already there?

If vmexit reason fully decodes the instruction for you then I would

Yes, VM exit reason + VMX_INSTRUCTION_INFO includes everything we needs :) It is mainly because VMX instruction is simple.

Quoted 3 lines

agree that skipping x86_emulate could make sense. And then your instruction emulator would be really simple and fast -- vmx_io_instruction() is a good example of this. If you still need to

Sure, the emulation is very simple and fast, I cab put all privilege check into one function as well.

Quoted 1 line

parse the instruction to decode ModRM and the like, then I don't see

No need to decode ModRM.

Quoted 3 lines

that the partial decode from vmexit reason saves you much at all, and you might as well go the whole hog and do full decode. I don't see much saving from a hacky middle-ground.

So how about we reuse some functions in x86 emulate like this one?

static enum x86_segment
decode_segment(uint8_t modrm_reg)

{
    switch ( modrm_reg )
    {
    case 0: return x86_seg_es;
    case 1: return x86_seg_cs;
    case 2: return x86_seg_ss;
    case 3: return x86_seg_ds;
    case 4: return x86_seg_fs;
    case 5: return x86_seg_gs;
    default: break;
    }
    return decode_segment_failed;
}

Thx, Eddie

#44 lore raw

Re: [PATCH 06/16] vmx: nest: handling VMX instruction exits

Quoted 5 lines · Eddie
Quoted 3 lines

that the partial decode from vmexit reason saves you much at all, and you might as well go the whole hog and do full decode. I don't see much saving from a hacky middle-ground.

So how about we reuse some functions in x86 emulate like this one?

Ah, well, now I look at your patch 06/16 properly, I think it's clear and self-contained as it is. Your private enumerations within nest.c simply serve to document the format of the decoded instruction provided to you via fields in the VMCS. I wouldn't be inclined to change it at all, unless Tim really has strong objections about it. It's not like you're defining namespaces for new abstractions you have conjured from thin air -- they correspond directly to a hardware-defined decode format. Defining enumerations on top of that is *good*, imo. I would take 06/16 as it stands.

-- Keir

Quoted 17 lines

static enum x86_segment
decode_segment(uint8_t modrm_reg)

{
    switch ( modrm_reg )
    {
    case 0: return x86_seg_es;
    case 1: return x86_seg_cs;
    case 2: return x86_seg_ss;
    case 3: return x86_seg_ds;
    case 4: return x86_seg_fs;
    case 5: return x86_seg_gs;
    default: break;
    }
    return decode_segment_failed;
}

Thx, Eddie

#45 lore raw

Re: [PATCH 06/16] vmx: nest: handling VMX instruction exits

Quoted 6 lines · Keir Fraser
Quoted 4 lines · Eddie
Quoted 3 lines

that the partial decode from vmexit reason saves you much at all, and you might as well go the whole hog and do full decode. I don't see much saving from a hacky middle-ground.

Quoted 7 lines
Quoted 1 line

So how about we reuse some functions in x86 emulate like this one?

Ah, well, now I look at your patch 06/16 properly, I think it's clear and self-contained as it is. Your private enumerations within nest.c simply serve to document the format of the decoded instruction provided to you via fields in the VMCS. I wouldn't be inclined to change it at all, unless Tim really has strong objections about it.

No, that's OK.

Quoted 4 lines

It's not like you're defining
namespaces for new abstractions you have conjured from thin air -- they correspond directly to a hardware-defined decode format. Defining enumerations on top of that is *good*, imo. I would take 06/16 as it stands.

Fair enough, but I'd like the memory leak fixed too (svmcs and vvmcs are only freed if the N1 guest executes VMXOFF).

Cheers,

Tim.

Quoted 19 lines
Quoted 17 lines

static enum x86_segment
decode_segment(uint8_t modrm_reg)

{
    switch ( modrm_reg )
    {
    case 0: return x86_seg_es;
    case 1: return x86_seg_cs;
    case 2: return x86_seg_ss;
    case 3: return x86_seg_ds;
    case 4: return x86_seg_fs;
    case 5: return x86_seg_gs;
    default: break;
    }
    return decode_segment_failed;
}

Thx, Eddie

Signature · 3 lines
Tim Deegan <[email protected]>
Principal Software Engineer, XenServer Engineering
Citrix Systems UK Ltd.  (Company #02937203, SL9 0BG)
#46 lore raw

RE: [PATCH 06/16] vmx: nest: handling VMX instruction exits

Quoted 10 lines
Quoted 6 lines

It's not like you're defining
namespaces for new abstractions you have conjured from thin air -- they correspond directly to a hardware-defined decode format. Defining enumerations on top of that is *good*, imo. I would take 06/16 as it stands.

Fair enough, but I'd like the memory leak fixed too (svmcs and vvmcs are only freed if the N1 guest executes VMXOFF).

Sure. Fixed it locally at vmx_destroy_vmcs.

BTW, how do you like CONFIG_VVMCS_MAPPING ? I feel a little bit more complicated.

And how about rename vvmcs to vmcs12 (VMCS used by L1 VMM for L2 guest), and rename svmcs as vmcs02 (VMCS used by L0 VMM for L2 guest).
Of course hvmcs becomes vmcs01 then, (VMCS used by L0 VMM for L1 guest).

Thx, Eddie

#47 lore raw

Re: [PATCH 06/16] vmx: nest: handling VMX instruction exits

Quoted 7 lines · Eddie
Quoted 3 lines

Fair enough, but I'd like the memory leak fixed too (svmcs and vvmcs are only freed if the N1 guest executes VMXOFF).

Sure. Fixed it locally at vmx_destroy_vmcs.

BTW, how do you like CONFIG_VVMCS_MAPPING ? I feel a little bit more complicated.

Ah yes, presumably you will be picking one or the other and getting rid of ifdefs in a future spin of this patchset? I don't personally care whether you map or copy, though the former should be faster I guess? Anyway I think your logic for mapping is too short and simplistic -- look at hvm_map_entry() to see how it uses gfn_to_mfn_unshare (necessary if you will be modifying the page) and handles the various return codes from that. You could even factor out a common helper routine from hvm_map_entry() that you could then use rather than open-coding very similar logic.

Quoted 3 lines

And how about rename vvmcs to vmcs12 (VMCS used by L1 VMM for L2 guest), and rename svmcs as vmcs02 (VMCS used by L0 VMM for L2 guest).
Of course hvmcs becomes vmcs01 then, (VMCS used by L0 VMM for L1 guest).

No opinion myself. :-)

-- Keir

author
#48 lore raw

PATCH 7/16 vmx: nest: switch current vmcs

In reply to Qing He

facility to switch between host vmcs and shadow vmcs

---

xen/arch/x86/hvm/vmx/vmcs.c +35 -0
diff -r e638812d8f46 xen/arch/x86/hvm/vmx/vmcs.c
@@ -642,6 +642,35 @@
               (unsigned long)&get_cpu_info()->guest_cpu_user_regs.error_code);
 }
 
+void vmx_vmcs_switch_current(struct vcpu *v,
+                             struct vmcs_struct *from,
+                             struct vmcs_struct *to)
+{
+    /* no foreign access */
+    if ( unlikely(v != current) )
+        return;
+
+    if ( unlikely(current->arch.hvm_vmx.vmcs != from) )
+        return;
+
+    spin_lock(&v->arch.hvm_vmx.vmcs_lock);
+
+    __vmpclear(virt_to_maddr(from));
+    __vmptrld(virt_to_maddr(to));
+
+    v->arch.hvm_vmx.vmcs = to;
+    v->arch.hvm_vmx.launched = 0;
+    this_cpu(current_vmcs) = to;
+
+    if ( v->arch.hvm_vmx.vmcs_host_updated )
+    {
+        v->arch.hvm_vmx.vmcs_host_updated = 0;
+        vmx_set_host_env(v);
+    }
+
+    spin_unlock(&v->arch.hvm_vmx.vmcs_lock);
+}
+
 void vmx_disable_intercept_for_msr(struct vcpu *v, u32 msr)
 {
     unsigned long *msr_bitmap = v->arch.hvm_vmx.msr_bitmap;
@@ -1080,6 +1109,12 @@
         hvm_migrate_pirqs(v);
         vmx_set_host_env(v);
         hvm_asid_flush_vcpu(v);
+
+        /*
+         * nesting: we need to do additional host env sync if we have other
+         * VMCS's. Currently this only works with only one active sVMCS.
+         */
+        v->arch.hvm_vmx.vmcs_host_updated = 1;
     }
 
     debug_state = v->domain->debugger_attached;
xen/include/asm-x86/hvm/vmx/vmcs.h +4 -0
diff -r e638812d8f46 xen/include/asm-x86/hvm/vmx/vmcs.h
@@ -102,6 +102,7 @@
 
     /* nested virtualization */
     struct vmx_nest_struct nest;
+    int                  vmcs_host_updated;
 
 #ifdef __x86_64__
     struct vmx_msr_state msr_state;
@@ -389,6 +390,9 @@
 int vmx_write_guest_msr(u32 msr, u64 val);
 int vmx_add_guest_msr(u32 msr);
 int vmx_add_host_load_msr(u32 msr);
+void vmx_vmcs_switch_current(struct vcpu *v,
+                             struct vmcs_struct *from,
+                             struct vmcs_struct *to);
 
 #endif /* ASM_X86_HVM_VMX_VMCS_H__ */
author
#49 lore raw

PATCH 8/16 vmx: nest: vmresume/vmlaunch

In reply to Qing He

vmresume and vmlaunch instructions and transitional states

---

xen/arch/x86/hvm/vmx/nest.c +30 -0
diff -r e828d55c10bb xen/arch/x86/hvm/vmx/nest.c
@@ -633,3 +633,33 @@
     hvm_inject_exception(TRAP_invalid_op, 0, 0);
     return X86EMUL_EXCEPTION;
 }
+
+int vmx_nest_handle_vmresume(struct cpu_user_regs *regs)
+{
+    struct vcpu *v = current;
+    struct vmx_nest_struct *nest = &v->arch.hvm_vmx.nest;
+    int rc;
+
+    if ( unlikely(!nest->guest_vmxon_pa) )
+        goto invalid_op;
+
+    rc = vmx_inst_check_privilege(regs);
+    if ( rc != X86EMUL_OKAY )
+        return rc;
+
+    if ( nest->vmcs_valid == 1 )
+        nest->vmresume_pending = 1;
+    else
+        vmreturn(regs, VMFAIL_INVALID);
+
+    return X86EMUL_OKAY;
+
+invalid_op:
+    hvm_inject_exception(TRAP_invalid_op, 0, 0);
+    return X86EMUL_EXCEPTION;
+}
+
+int vmx_nest_handle_vmlaunch(struct cpu_user_regs *regs)
+{
+    return vmx_nest_handle_vmresume(regs);
+}
xen/arch/x86/hvm/vmx/vmx.c +15 -2
diff -r e828d55c10bb xen/arch/x86/hvm/vmx/vmx.c
@@ -2321,6 +2321,11 @@
     /* Now enable interrupts so it's safe to take locks. */
     local_irq_enable();
 
+    /* XXX: This looks ugly, but we need a mechanism to ensure
+     * any pending vmresume has really happened
+     */
+    v->arch.hvm_vmx.nest.vmresume_in_progress = 0;
+
     if ( unlikely(exit_reason & VMX_EXIT_REASONS_FAILED_VMENTRY) )
         return vmx_failed_vmentry(exit_reason, regs);
 
@@ -2592,6 +2597,11 @@
         if ( vmx_nest_handle_vmclear(regs) == X86EMUL_OKAY )
             __update_guest_eip(inst_len);
         break;
+    case EXIT_REASON_VMLAUNCH:
+        inst_len = __get_instruction_length();
+        if ( vmx_nest_handle_vmlaunch(regs) == X86EMUL_OKAY )
+            __update_guest_eip(inst_len);
+        break;
     case EXIT_REASON_VMPTRLD:
         inst_len = __get_instruction_length();
         if ( vmx_nest_handle_vmptrld(regs) == X86EMUL_OKAY )
@@ -2607,6 +2617,11 @@
         if ( vmx_nest_handle_vmread(regs) == X86EMUL_OKAY )
             __update_guest_eip(inst_len);
         break;
+    case EXIT_REASON_VMRESUME:
+        inst_len = __get_instruction_length();
+        if ( vmx_nest_handle_vmresume(regs) == X86EMUL_OKAY )
+            __update_guest_eip(inst_len);
+        break;
     case EXIT_REASON_VMWRITE:
         inst_len = __get_instruction_length();
         if ( vmx_nest_handle_vmwrite(regs) == X86EMUL_OKAY )
@@ -2625,8 +2640,6 @@
 
     case EXIT_REASON_MWAIT_INSTRUCTION:
     case EXIT_REASON_MONITOR_INSTRUCTION:
-    case EXIT_REASON_VMLAUNCH:
-    case EXIT_REASON_VMRESUME:
         vmx_inject_hw_exception(TRAP_invalid_op, HVM_DELIVER_NO_ERROR_CODE);
         break;
 
xen/include/asm-x86/hvm/vmx/nest.h +17 -0
diff -r e828d55c10bb xen/include/asm-x86/hvm/vmx/nest.h
@@ -40,6 +40,20 @@
     void                *vvmcs;
     struct vmcs_struct  *svmcs;
     int                  vmcs_valid;
+
+    /*
+     * vmexit_pending and vmresume_pending is to mark pending
+     * switches, they are cleared when physical vmcs is changed.
+     */
+    int                  vmexit_pending;
+    int                  vmresume_pending;
+
+    /*
+     * upon L1->L2, there is a window between context switch and
+     * the physical vmentry of the shadow vmcs, protect against it
+     * with vmresume_in_progress
+     */
+    int                  vmresume_in_progress;
 };
 
 int vmx_nest_handle_vmxon(struct cpu_user_regs *regs);
@@ -52,4 +66,7 @@
 int vmx_nest_handle_vmread(struct cpu_user_regs *regs);
 int vmx_nest_handle_vmwrite(struct cpu_user_regs *regs);
 
+int vmx_nest_handle_vmresume(struct cpu_user_regs *regs);
+int vmx_nest_handle_vmlaunch(struct cpu_user_regs *regs);
+
 #endif /* __ASM_X86_HVM_NEST_H__ */
#50 lore raw

Re: [PATCH 08/16] vmx: nest: vmresume/vmlaunch

Quoted 96 lines · On Wednesday 08 September 2010 17:22:16 Qing He wrote:

vmresume and vmlaunch instructions and transitional states

---

diff -r e828d55c10bb xen/arch/x86/hvm/vmx/nest.c

xen/arch/x86/hvm/vmx/nest.c +30 -0
@@ -633,3 +633,33 @@
     hvm_inject_exception(TRAP_invalid_op, 0, 0);
     return X86EMUL_EXCEPTION;
 }
+
+int vmx_nest_handle_vmresume(struct cpu_user_regs *regs)
+{
+    struct vcpu *v = current;
+    struct vmx_nest_struct *nest = &v->arch.hvm_vmx.nest;
+    int rc;
+
+    if ( unlikely(!nest->guest_vmxon_pa) )
+        goto invalid_op;
+
+    rc = vmx_inst_check_privilege(regs);
+    if ( rc != X86EMUL_OKAY )
+        return rc;
+
+    if ( nest->vmcs_valid == 1 )
+        nest->vmresume_pending = 1;
+    else
+        vmreturn(regs, VMFAIL_INVALID);
+
+    return X86EMUL_OKAY;
+
+invalid_op:
+    hvm_inject_exception(TRAP_invalid_op, 0, 0);
+    return X86EMUL_EXCEPTION;
+}
+
+int vmx_nest_handle_vmlaunch(struct cpu_user_regs *regs)
+{
+    return vmx_nest_handle_vmresume(regs);
+}

diff -r e828d55c10bb xen/arch/x86/hvm/vmx/vmx.c

xen/arch/x86/hvm/vmx/vmx.c +15 -2
@@ -2321,6 +2321,11 @@
     /* Now enable interrupts so it's safe to take locks. */
     local_irq_enable();
 
+    /* XXX: This looks ugly, but we need a mechanism to ensure
+     * any pending vmresume has really happened
+     */
+    v->arch.hvm_vmx.nest.vmresume_in_progress = 0;
+
     if ( unlikely(exit_reason & VMX_EXIT_REASONS_FAILED_VMENTRY) )
         return vmx_failed_vmentry(exit_reason, regs);
 
@@ -2592,6 +2597,11 @@
         if ( vmx_nest_handle_vmclear(regs) == X86EMUL_OKAY )
             __update_guest_eip(inst_len);
         break;
+    case EXIT_REASON_VMLAUNCH:
+        inst_len = __get_instruction_length();
+        if ( vmx_nest_handle_vmlaunch(regs) == X86EMUL_OKAY )
+            __update_guest_eip(inst_len);
+        break;
     case EXIT_REASON_VMPTRLD:
         inst_len = __get_instruction_length();
         if ( vmx_nest_handle_vmptrld(regs) == X86EMUL_OKAY )
@@ -2607,6 +2617,11 @@
         if ( vmx_nest_handle_vmread(regs) == X86EMUL_OKAY )
             __update_guest_eip(inst_len);
         break;
+    case EXIT_REASON_VMRESUME:
+        inst_len = __get_instruction_length();
+        if ( vmx_nest_handle_vmresume(regs) == X86EMUL_OKAY )
+            __update_guest_eip(inst_len);
+        break;
     case EXIT_REASON_VMWRITE:
         inst_len = __get_instruction_length();
         if ( vmx_nest_handle_vmwrite(regs) == X86EMUL_OKAY )
@@ -2625,8 +2640,6 @@
 
     case EXIT_REASON_MWAIT_INSTRUCTION:
     case EXIT_REASON_MONITOR_INSTRUCTION:
-    case EXIT_REASON_VMLAUNCH:
-    case EXIT_REASON_VMRESUME:
         vmx_inject_hw_exception(TRAP_invalid_op,
 HVM_DELIVER_NO_ERROR_CODE); break;
 

diff -r e828d55c10bb xen/include/asm-x86/hvm/vmx/nest.h

xen/include/asm-x86/hvm/vmx/nest.h line 49

Quoted 10 lines
xen/include/asm-x86/hvm/vmx/nest.h +7 -0
@@ -40,6 +40,20 @@
     void                *vvmcs;
     struct vmcs_struct  *svmcs;
     int                  vmcs_valid;
+
+    /*
+     * vmexit_pending and vmresume_pending is to mark pending
+     * switches, they are cleared when physical vmcs is changed.
+     */
+    int                  vmexit_pending;
+    int                  vmresume_pending;

This is functional equal to the vmentry flag in struct nestedhvm.

xen/include/asm-x86/hvm/vmx/nest.h line 57

Quoted 7 lines
xen/include/asm-x86/hvm/vmx/nest.h +6 -0
+    /*
+     * upon L1->L2, there is a window between context switch and
+     * the physical vmentry of the shadow vmcs, protect against it
+     * with vmresume_in_progress
+     */
+    int                  vmresume_in_progress;
 };

What is the window you describe in the comment ? And which problem does it arise?

You use this to check if you can inject interrupts into the l2 guest. To merge the interrupt code as Tim said in
http://lists.xensource.com/archives/html/xen-devel/2010-09/msg00747.html
you can probably use the nh_gif field for this.

Christoph

Quoted 14 lines
xen/include/asm-x86/hvm/vmx/nest.h +3 -0
 int vmx_nest_handle_vmxon(struct cpu_user_regs *regs);
@@ -52,4 +66,7 @@
 int vmx_nest_handle_vmread(struct cpu_user_regs *regs);
 int vmx_nest_handle_vmwrite(struct cpu_user_regs *regs);
 
+int vmx_nest_handle_vmresume(struct cpu_user_regs *regs);
+int vmx_nest_handle_vmlaunch(struct cpu_user_regs *regs);
+
 #endif /* __ASM_X86_HVM_NEST_H__ */

_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel

Signature · 6 lines
---to satisfy European Law for business letters:
Advanced Micro Devices GmbH
Einsteinring 24, 85609 Dornach b. Muenchen
Geschaeftsfuehrer: Alberto Bozzo, Andrew Bowd
Sitz: Dornach, Gemeinde Aschheim, Landkreis Muenchen
Registergericht Muenchen, HRB Nr. 43632
#51 lore raw

Re: [PATCH 08/16] vmx: nest: vmresume/vmlaunch

Quoted 93 lines · On Wednesday 15 September 2010 11:52:26 Christoph Egger wrote:
Quoted 92 lines · On Wednesday 08 September 2010 17:22:16 Qing He wrote:

vmresume and vmlaunch instructions and transitional states

---

diff -r e828d55c10bb xen/arch/x86/hvm/vmx/nest.c

xen/arch/x86/hvm/vmx/nest.c +30 -0
@@ -633,3 +633,33 @@
     hvm_inject_exception(TRAP_invalid_op, 0, 0);
     return X86EMUL_EXCEPTION;
 }
+
+int vmx_nest_handle_vmresume(struct cpu_user_regs *regs)
+{
+    struct vcpu *v = current;
+    struct vmx_nest_struct *nest = &v->arch.hvm_vmx.nest;
+    int rc;
+
+    if ( unlikely(!nest->guest_vmxon_pa) )
+        goto invalid_op;
+
+    rc = vmx_inst_check_privilege(regs);
+    if ( rc != X86EMUL_OKAY )
+        return rc;
+
+    if ( nest->vmcs_valid == 1 )
+        nest->vmresume_pending = 1;
+    else
+        vmreturn(regs, VMFAIL_INVALID);
+
+    return X86EMUL_OKAY;
+
+invalid_op:
+    hvm_inject_exception(TRAP_invalid_op, 0, 0);
+    return X86EMUL_EXCEPTION;
+}
+
+int vmx_nest_handle_vmlaunch(struct cpu_user_regs *regs)
+{
+    return vmx_nest_handle_vmresume(regs);
+}

diff -r e828d55c10bb xen/arch/x86/hvm/vmx/vmx.c

xen/arch/x86/hvm/vmx/vmx.c +15 -2
@@ -2321,6 +2321,11 @@
     /* Now enable interrupts so it's safe to take locks. */
     local_irq_enable();
 
+    /* XXX: This looks ugly, but we need a mechanism to ensure
+     * any pending vmresume has really happened
+     */
+    v->arch.hvm_vmx.nest.vmresume_in_progress = 0;
+
     if ( unlikely(exit_reason & VMX_EXIT_REASONS_FAILED_VMENTRY) )
         return vmx_failed_vmentry(exit_reason, regs);
 
@@ -2592,6 +2597,11 @@
         if ( vmx_nest_handle_vmclear(regs) == X86EMUL_OKAY )
             __update_guest_eip(inst_len);
         break;
+    case EXIT_REASON_VMLAUNCH:
+        inst_len = __get_instruction_length();
+        if ( vmx_nest_handle_vmlaunch(regs) == X86EMUL_OKAY )
+            __update_guest_eip(inst_len);
+        break;
     case EXIT_REASON_VMPTRLD:
         inst_len = __get_instruction_length();
         if ( vmx_nest_handle_vmptrld(regs) == X86EMUL_OKAY )
@@ -2607,6 +2617,11 @@
         if ( vmx_nest_handle_vmread(regs) == X86EMUL_OKAY )
             __update_guest_eip(inst_len);
         break;
+    case EXIT_REASON_VMRESUME:
+        inst_len = __get_instruction_length();
+        if ( vmx_nest_handle_vmresume(regs) == X86EMUL_OKAY )
+            __update_guest_eip(inst_len);
+        break;
     case EXIT_REASON_VMWRITE:
         inst_len = __get_instruction_length();
         if ( vmx_nest_handle_vmwrite(regs) == X86EMUL_OKAY )
@@ -2625,8 +2640,6 @@
 
     case EXIT_REASON_MWAIT_INSTRUCTION:
     case EXIT_REASON_MONITOR_INSTRUCTION:
-    case EXIT_REASON_VMLAUNCH:
-    case EXIT_REASON_VMRESUME:
         vmx_inject_hw_exception(TRAP_invalid_op,
 HVM_DELIVER_NO_ERROR_CODE); break;
 
Quoted 16 lines
Quoted 4 lines

diff -r e828d55c10bb xen/include/asm-x86/hvm/vmx/nest.h

xen/include/asm-x86/hvm/vmx/nest.h line 49

Quoted 10 lines
xen/include/asm-x86/hvm/vmx/nest.h +7 -0
@@ -40,6 +40,20 @@
     void                *vvmcs;
     struct vmcs_struct  *svmcs;
     int                  vmcs_valid;
+
+    /*
+     * vmexit_pending and vmresume_pending is to mark pending
+     * switches, they are cleared when physical vmcs is changed.
+     */
+    int                  vmexit_pending;
+    int                  vmresume_pending;

This is functional equal to the vmentry flag in struct nestedhvm.

Is it possible to have both vmexit and vmresume pending at the same time ?

Christoph

Quoted 34 lines

xen/include/asm-x86/hvm/vmx/nest.h line 57

Quoted 8 lines
xen/include/asm-x86/hvm/vmx/nest.h +6 -0
+    /*
+     * upon L1->L2, there is a window between context switch and
+     * the physical vmentry of the shadow vmcs, protect against it
+     * with vmresume_in_progress
+     */
+    int                  vmresume_in_progress;
 };

What is the window you describe in the comment ? And which problem does it arise?

You use this to check if you can inject interrupts into the l2 guest. To merge the interrupt code as Tim said in
http://lists.xensource.com/archives/html/xen-devel/2010-09/msg00747.html
you can probably use the nh_gif field for this.

Christoph

Quoted 14 lines
xen/include/asm-x86/hvm/vmx/nest.h +3 -0
 int vmx_nest_handle_vmxon(struct cpu_user_regs *regs);
@@ -52,4 +66,7 @@
 int vmx_nest_handle_vmread(struct cpu_user_regs *regs);
 int vmx_nest_handle_vmwrite(struct cpu_user_regs *regs);
 
+int vmx_nest_handle_vmresume(struct cpu_user_regs *regs);
+int vmx_nest_handle_vmlaunch(struct cpu_user_regs *regs);
+
 #endif /* __ASM_X86_HVM_NEST_H__ */

_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel

Signature · 6 lines
---to satisfy European Law for business letters:
Advanced Micro Devices GmbH
Einsteinring 24, 85609 Dornach b. Muenchen
Geschaeftsfuehrer: Alberto Bozzo, Andrew Bowd
Sitz: Dornach, Gemeinde Aschheim, Landkreis Muenchen
Registergericht Muenchen, HRB Nr. 43632
#52 lore raw

RE: [PATCH 08/16] vmx: nest: vmresume/vmlaunch

Quoted 102 lines · Christoph Egger
Quoted 86 lines · On Wednesday 15 September 2010 11:52:26 Christoph Egger wrote:
Quoted 85 lines · On Wednesday 08 September 2010 17:22:16 Qing He wrote:

vmresume and vmlaunch instructions and transitional states

---

diff -r e828d55c10bb xen/arch/x86/hvm/vmx/nest.c

xen/arch/x86/hvm/vmx/nest.c +4 -0
     -633,3 +633,33 @@ hvm_inject_exception(TRAP_invalid_op, 0, 0);
     return X86EMUL_EXCEPTION;
 }
+
+int vmx_nest_handle_vmresume(struct cpu_user_regs *regs) +{
+    struct vcpu *v = current;
+    struct vmx_nest_struct *nest = &v->arch.hvm_vmx.nest; +    int

rc; +

xen/arch/x86/hvm/vmx/nest.c +22 -0
+    if ( unlikely(!nest->guest_vmxon_pa) )
+        goto invalid_op;
+
+    rc = vmx_inst_check_privilege(regs);
+    if ( rc != X86EMUL_OKAY )
+        return rc;
+
+    if ( nest->vmcs_valid == 1 )
+        nest->vmresume_pending = 1;
+    else
+        vmreturn(regs, VMFAIL_INVALID);
+
+    return X86EMUL_OKAY;
+
+invalid_op:
+    hvm_inject_exception(TRAP_invalid_op, 0, 0);
+    return X86EMUL_EXCEPTION;
+}
+
+int vmx_nest_handle_vmlaunch(struct cpu_user_regs *regs) +{
+    return vmx_nest_handle_vmresume(regs);
+}

diff -r e828d55c10bb xen/arch/x86/hvm/vmx/vmx.c

xen/arch/x86/hvm/vmx/vmx.c +0 -0
     -2321,6 +2321,11 @@ /* Now enable interrupts so it's safe to

take locks. */ local_irq_enable();

xen/arch/x86/hvm/vmx/vmx.c +9 -0
+    /* XXX: This looks ugly, but we need a mechanism to ensure
+     * any pending vmresume has really happened
+     */
+    v->arch.hvm_vmx.nest.vmresume_in_progress = 0; +
     if ( unlikely(exit_reason & VMX_EXIT_REASONS_FAILED_VMENTRY) )
         return vmx_failed_vmentry(exit_reason, regs);
@@ -2592,6 +2597,11 @@
         if ( vmx_nest_handle_vmclear(regs) == X86EMUL_OKAY )
             __update_guest_eip(inst_len);
         break;
+    case EXIT_REASON_VMLAUNCH:
+        inst_len = __get_instruction_length();
+        if ( vmx_nest_handle_vmlaunch(regs) == X86EMUL_OKAY )
+            __update_guest_eip(inst_len);
+        break;
     case EXIT_REASON_VMPTRLD:
         inst_len = __get_instruction_length();
         if ( vmx_nest_handle_vmptrld(regs) == X86EMUL_OKAY ) @@
         -2607,6 +2617,11 @@ if ( vmx_nest_handle_vmread(regs) ==
             X86EMUL_OKAY ) __update_guest_eip(inst_len);
         break;
xen/arch/x86/hvm/vmx/vmx.c +5 -3
+    case EXIT_REASON_VMRESUME:
+        inst_len = __get_instruction_length();
+        if ( vmx_nest_handle_vmresume(regs) == X86EMUL_OKAY )
+            __update_guest_eip(inst_len);
+        break;
     case EXIT_REASON_VMWRITE:
         inst_len = __get_instruction_length();
         if ( vmx_nest_handle_vmwrite(regs) == X86EMUL_OKAY ) @@
-2625,8 +2640,6 @@ 
 
     case EXIT_REASON_MWAIT_INSTRUCTION:
     case EXIT_REASON_MONITOR_INSTRUCTION:
-    case EXIT_REASON_VMLAUNCH:
-    case EXIT_REASON_VMRESUME:
         vmx_inject_hw_exception(TRAP_invalid_op,

HVM_DELIVER_NO_ERROR_CODE); break;

Quoted 14 lines
Quoted 10 lines

diff -r e828d55c10bb xen/include/asm-x86/hvm/vmx/nest.h
--- a/xen/include/asm-x86/hvm/vmx/nest.h Wed Sep 08 21:42:10 2010
+0800 +++ b/xen/include/asm-x86/hvm/vmx/nest.h Wed Sep 08 22:04:16

     2010 +0800 @@ -40,6 +40,20 @@ void                *vvmcs;
     struct vmcs_struct  *svmcs;
     int                  vmcs_valid;
xen/arch/x86/hvm/vmx/vmx.c +4 -0
+
+    /*
+     * vmexit_pending and vmresume_pending is to mark pending
+     * switches, they are cleared when physical vmcs is changed. +
Quoted 2 lines

*/ + int vmexit_pending;
+ int vmresume_pending;

This is functional equal to the vmentry flag in struct nestedhvm.

Quoted 3 lines

Is it possible to have both vmexit and vmresume pending at the same time ?

Should not be. We may be able to use single variable for multiple state.

author
#53 lore raw

PATCH 9/16 vmx: nest: shadow controls

In reply to Qing He

automatically compute controls according to current mode

---

xen/arch/x86/hvm/vmx/nest.c +32 -0
diff -r 625f74a9bb36 xen/arch/x86/hvm/vmx/nest.c
@@ -663,3 +663,35 @@
 {
     return vmx_nest_handle_vmresume(regs);
 }
+
+static void set_shadow_control(struct vmx_nest_struct *nest,
+                               unsigned int field,
+                               u32 host_value)
+{
+    u32 value;
+
+    value = (u32) __get_vvmcs(nest->vvmcs, field) | host_value;
+    __vmwrite(field, value);
+}
+
+void vmx_nest_update_exec_control(struct vcpu *v, unsigned long value)
+{
+    struct vmx_nest_struct *nest = &v->arch.hvm_vmx.nest;
+
+    set_shadow_control(nest, CPU_BASED_VM_EXEC_CONTROL, value);
+}
+
+void vmx_nest_update_secondary_exec_control(struct vcpu *v,
+                                            unsigned long value)
+{
+    struct vmx_nest_struct *nest = &v->arch.hvm_vmx.nest;
+
+    set_shadow_control(nest, SECONDARY_VM_EXEC_CONTROL, value);
+}
+
+void vmx_nest_update_exception_bitmap(struct vcpu *v, unsigned long value)
+{
+    struct vmx_nest_struct *nest = &v->arch.hvm_vmx.nest;
+
+    set_shadow_control(nest, EXCEPTION_BITMAP, value);
+}
xen/arch/x86/hvm/vmx/vmx.c +14 -4
diff -r 625f74a9bb36 xen/arch/x86/hvm/vmx/vmx.c
@@ -387,18 +387,28 @@
 
 void vmx_update_cpu_exec_control(struct vcpu *v)
 {
-    __vmwrite(CPU_BASED_VM_EXEC_CONTROL, v->arch.hvm_vmx.exec_control);
+    if ( v->arch.hvm_vcpu.in_nesting )
+        vmx_nest_update_exec_control(v, v->arch.hvm_vmx.exec_control);
+    else
+        __vmwrite(CPU_BASED_VM_EXEC_CONTROL, v->arch.hvm_vmx.exec_control);
 }
 
 void vmx_update_secondary_exec_control(struct vcpu *v)
 {
-    __vmwrite(SECONDARY_VM_EXEC_CONTROL,
-              v->arch.hvm_vmx.secondary_exec_control);
+    if ( v->arch.hvm_vcpu.in_nesting )
+        vmx_nest_update_secondary_exec_control(v,
+            v->arch.hvm_vmx.secondary_exec_control);
+    else
+        __vmwrite(SECONDARY_VM_EXEC_CONTROL,
+                  v->arch.hvm_vmx.secondary_exec_control);
 }
 
 void vmx_update_exception_bitmap(struct vcpu *v)
 {
-    __vmwrite(EXCEPTION_BITMAP, v->arch.hvm_vmx.exception_bitmap);
+    if ( v->arch.hvm_vcpu.in_nesting )
+        vmx_nest_update_exception_bitmap(v, v->arch.hvm_vmx.exception_bitmap);
+    else
+        __vmwrite(EXCEPTION_BITMAP, v->arch.hvm_vmx.exception_bitmap);
 }
 
 static int vmx_guest_x86_mode(struct vcpu *v)
xen/include/asm-x86/hvm/vmx/nest.h +5 -0
diff -r 625f74a9bb36 xen/include/asm-x86/hvm/vmx/nest.h
@@ -69,4 +69,9 @@
 int vmx_nest_handle_vmresume(struct cpu_user_regs *regs);
 int vmx_nest_handle_vmlaunch(struct cpu_user_regs *regs);
 
+void vmx_nest_update_exec_control(struct vcpu *v, unsigned long value);
+void vmx_nest_update_secondary_exec_control(struct vcpu *v,
+                                            unsigned long value);
+void vmx_nest_update_exception_bitmap(struct vcpu *v, unsigned long value);
+
 #endif /* __ASM_X86_HVM_NEST_H__ */
author
#54 lore raw

PATCH 10/16 vmx: nest: L1 <-> L2 context switch

In reply to Qing He

This patch adds mode switch between L1 and L2

---

xen/arch/x86/hvm/vmx/entry.S +1 -0
diff -r 86c36f5c38f2 xen/arch/x86/hvm/vmx/entry.S
@@ -119,6 +119,7 @@
 .globl vmx_asm_do_vmentry
 vmx_asm_do_vmentry:
         call vmx_intr_assist
+        call vmx_nest_switch_mode
 
         get_current(bx)
         cli
xen/arch/x86/hvm/vmx/nest.c +348 -0

357 lines not on this page · raw message

xen/include/asm-x86/hvm/vmx/nest.h +2 -0
diff -r 86c36f5c38f2 xen/include/asm-x86/hvm/vmx/nest.h
@@ -56,6 +56,8 @@
     int                  vmresume_in_progress;
 };
 
+asmlinkage void vmx_nest_switch_mode(void);
+
 int vmx_nest_handle_vmxon(struct cpu_user_regs *regs);
 int vmx_nest_handle_vmxoff(struct cpu_user_regs *regs);
author
#55 lore raw

PATCH 11/16 vmx: nest: interrupt handling

In reply to Qing He

This patch adds interrupt handling for nested, mainly includes:
- virtual interrupt when running in nested mode,
- idtv handling in L2.
- interrupt blocking handling in L2

---

xen/arch/x86/hvm/vmx/intr.c +101 -0
diff -r 4934d8db96bf xen/arch/x86/hvm/vmx/intr.c
@@ -33,6 +33,7 @@
 #include <asm/hvm/support.h>
 #include <asm/hvm/vmx/vmx.h>
 #include <asm/hvm/vmx/vmcs.h>
+#include <asm/hvm/vmx/vvmcs.h>
 #include <asm/hvm/vpic.h>
 #include <asm/hvm/vlapic.h>
 #include <public/hvm/ioreq.h>
@@ -110,6 +111,103 @@
     }
 }
 
+/*
+ * Injecting interrupts for nested virtualization
+ *
+ *  When injecting virtual interrupts (originated from L0), there are
+ *  two major possibilities, within L1 context and within L2 context
+ *   1. L1 context (in_nesting == 0)
+ *     Everything is the same as without nested, check RFLAGS.IF to
+ *     see if the injection can be done, using VMCS to inject the
+ *     interrupt
+ *
+ *   2. L2 context (in_nesting == 1)
+ *     Causes a virtual VMExit, RFLAGS.IF is ignored, whether to ack
+ *     irq according to intr_ack_on_exit, shouldn't block normally,
+ *     except for:
+ *    a. context transition
+ *     interrupt needs to be blocked at virtual VMEntry time
+ *    b. L2 idtv reinjection
+ *     if L2 idtv is handled within L0 (e.g. L0 shadow page fault),
+ *     it needs to be reinjected without exiting to L1, interrupt
+ *     injection should be blocked as well at this point.
+ *
+ *  Unfortunately, interrupt blocking in L2 won't work with simple
+ *  intr_window_open (which depends on L2's IF). To solve this,
+ *  the following algorithm can be used:
+ *   v->arch.hvm_vmx.exec_control.VIRTUAL_INTR_PENDING now denotes
+ *   only L0 control, physical control may be different from it.
+ *       - if in L1, it behaves normally, intr window is written
+ *         to physical control as it is
+ *       - if in L2, replace it to MTF (or NMI window) if possible
+ *       - if MTF/NMI window is not used, intr window can still be
+ *         used but may have negative impact on interrupt performance.
+ */
+
+static int nest_intr_blocked(struct vcpu *v, struct hvm_intack intack)
+{
+    int r = 0;
+
+    if ( !v->arch.hvm_vcpu.in_nesting &&
+         v->arch.hvm_vmx.nest.vmresume_pending )
+        r = 1;
+
+    if ( v->arch.hvm_vcpu.in_nesting )
+    {
+        if ( v->arch.hvm_vmx.nest.vmexit_pending ||
+             v->arch.hvm_vmx.nest.vmresume_in_progress ||
+             (__vmread(VM_ENTRY_INTR_INFO) & INTR_INFO_VALID_MASK) )
+            r = 1;
+    }
+
+    return r;
+}
+
+static int vmx_nest_intr_intercept(struct vcpu *v, struct hvm_intack intack)
+{
+    u32 exit_ctrl;
+
+    /*
+     * TODO:
+     *   - if L1 intr-window exiting == 0
+     *   - vNMI
+     */
+
+    if ( nest_intr_blocked(v, intack) )
+    {
+        enable_intr_window(v, intack);
+        return 1;
+    }
+
+    if ( v->arch.hvm_vcpu.in_nesting )
+    {
+        if ( intack.source == hvm_intsrc_pic ||
+                 intack.source == hvm_intsrc_lapic )
+        {
+            vmx_inject_extint(intack.vector);
+
+            exit_ctrl = __get_vvmcs(v->arch.hvm_vmx.nest.vvmcs,
+                            VM_EXIT_CONTROLS);
+            if ( exit_ctrl & VM_EXIT_ACK_INTR_ON_EXIT )
+            {
+                /* for now, duplicate the ack path in vmx_intr_assist */
+                hvm_vcpu_ack_pending_irq(v, intack);
+                pt_intr_post(v, intack);
+
+                intack = hvm_vcpu_has_pending_irq(v);
+                if ( unlikely(intack.source != hvm_intsrc_none) )
+                    enable_intr_window(v, intack);
+            }
+            else
+                enable_intr_window(v, intack);
+
+            return 1;
+        }
+    }
+
+    return 0;
+}
+
 asmlinkage void vmx_intr_assist(void)
 {
     struct hvm_intack intack;
@@ -133,6 +231,9 @@
         if ( likely(intack.source == hvm_intsrc_none) )
             goto out;
 
+        if ( unlikely(vmx_nest_intr_intercept(v, intack)) )
+            goto out;
+
         intblk = hvm_interrupt_blocked(v, intack);
         if ( intblk == hvm_intblk_tpr )
         {
xen/arch/x86/hvm/vmx/nest.c +66 -0

87 lines not on this page · raw message

xen/arch/x86/hvm/vmx/vmx.c +52 -26

122 lines not on this page · raw message

xen/include/asm-x86/hvm/vmx/nest.h +5 -0
diff -r 4934d8db96bf xen/include/asm-x86/hvm/vmx/nest.h
@@ -54,6 +54,9 @@
      * with vmresume_in_progress
      */
     int                  vmresume_in_progress;
+
+    unsigned long        intr_info;
+    unsigned long        error_code;
 };
 
 asmlinkage void vmx_nest_switch_mode(void);
@@ -76,4 +79,6 @@
                                             unsigned long value);
 void vmx_nest_update_exception_bitmap(struct vcpu *v, unsigned long value);
 
+void vmx_nest_idtv_handling(void);
+
 #endif /* __ASM_X86_HVM_NEST_H__ */
author
#56 lore raw

PATCH 12/16 vmx: nest: VMExit handler in L2

In reply to Qing He

handles VMExits happened in L2

---

xen/arch/x86/hvm/vmx/nest.c +221 -0
diff -r 7a9edf7654ad xen/arch/x86/hvm/vmx/nest.c
@@ -1109,3 +1109,224 @@
 
     /* TODO: NMI */
 }
+
+/*
+ * L2 VMExit handling
+ */
+
+int vmx_nest_l2_vmexit_handler(struct cpu_user_regs *regs,
+                               unsigned int exit_reason)
+{
+    struct vcpu *v = current;
+    struct vmx_nest_struct *nest = &v->arch.hvm_vmx.nest;
+    u32 ctrl;
+    int bypass_l0 = 0;
+
+    nest->vmexit_pending = 0;
+    nest->intr_info = 0;
+    nest->error_code = 0;
+
+    switch (exit_reason) {
+    case EXIT_REASON_EXCEPTION_NMI:
+    {
+        u32 intr_info = __vmread(VM_EXIT_INTR_INFO);
+        u32 valid_mask = (X86_EVENTTYPE_HW_EXCEPTION << 8) |
+                         INTR_INFO_VALID_MASK;
+        u64 exec_bitmap;
+        int vector = intr_info & INTR_INFO_VECTOR_MASK;
+
+        /*
+         * decided by L0 and L1 exception bitmap, if the vetor is set by
+         * both, L0 has priority on #PF, L1 has priority on others
+         */
+        if ( vector == TRAP_page_fault )
+        {
+            if ( paging_mode_hap(v->domain) )
+                nest->vmexit_pending = 1;
+        }
+        else if ( (intr_info & valid_mask) == valid_mask )
+        {
+            exec_bitmap =__get_vvmcs(nest->vvmcs, EXCEPTION_BITMAP);
+
+            if ( exec_bitmap & (1 << vector) )
+                nest->vmexit_pending = 1;
+        }
+        break;
+    }
+
+    case EXIT_REASON_WBINVD:
+    case EXIT_REASON_EPT_VIOLATION:
+    case EXIT_REASON_EPT_MISCONFIG:
+    case EXIT_REASON_EXTERNAL_INTERRUPT:
+        /* pass to L0 handler */
+        break;
+
+    case VMX_EXIT_REASONS_FAILED_VMENTRY:
+    case EXIT_REASON_TRIPLE_FAULT:
+    case EXIT_REASON_TASK_SWITCH:
+    case EXIT_REASON_IO_INSTRUCTION:
+    case EXIT_REASON_CPUID:
+    case EXIT_REASON_MSR_READ:
+    case EXIT_REASON_MSR_WRITE:
+    case EXIT_REASON_VMCALL:
+    case EXIT_REASON_VMCLEAR:
+    case EXIT_REASON_VMLAUNCH:
+    case EXIT_REASON_VMPTRLD:
+    case EXIT_REASON_VMPTRST:
+    case EXIT_REASON_VMREAD:
+    case EXIT_REASON_VMRESUME:
+    case EXIT_REASON_VMWRITE:
+    case EXIT_REASON_VMXOFF:
+    case EXIT_REASON_VMXON:
+    case EXIT_REASON_INVEPT:
+        /* inject to L1 */
+        nest->vmexit_pending = 1;
+        break;
+
+    case EXIT_REASON_PENDING_VIRT_INTR:
+    {
+        ctrl = v->arch.hvm_vmx.exec_control;
+
+        /*
+         * if both open intr/nmi window, L0 has priority.
+         *
+         * Note that this is not strictly correct, in L2 context,
+         * L0's intr/nmi window flag should be replaced to MTF,
+         * causing an imediate VMExit, but MTF may not be available
+         * on all hardware.
+         */
+        if ( !(ctrl & CPU_BASED_VIRTUAL_INTR_PENDING) )
+            nest->vmexit_pending = 1;
+
+        break;
+    }
+    case EXIT_REASON_PENDING_VIRT_NMI:
+    {
+        ctrl = v->arch.hvm_vmx.exec_control;
+
+        if ( !(ctrl & CPU_BASED_VIRTUAL_NMI_PENDING) )
+            nest->vmexit_pending = 1;
+
+        break;
+    }
+
+    /* L1 has priority handling several other types of exits */
+    case EXIT_REASON_HLT:
+    {
+        ctrl = __get_vvmcs(nest->vvmcs, CPU_BASED_VM_EXEC_CONTROL);
+
+        if ( ctrl & CPU_BASED_HLT_EXITING )
+            nest->vmexit_pending = 1;
+
+        break;
+    }
+
+    case EXIT_REASON_RDTSC:
+    {
+        ctrl = __get_vvmcs(nest->vvmcs, CPU_BASED_VM_EXEC_CONTROL);
+
+        if ( ctrl & CPU_BASED_RDTSC_EXITING )
+            nest->vmexit_pending = 1;
+
+        break;
+    }
+
+    case EXIT_REASON_RDPMC:
+    {
+        ctrl = __get_vvmcs(nest->vvmcs, CPU_BASED_VM_EXEC_CONTROL);
+
+        if ( ctrl & CPU_BASED_RDPMC_EXITING )
+            nest->vmexit_pending = 1;
+
+        break;
+    }
+
+    case EXIT_REASON_MWAIT_INSTRUCTION:
+    {
+        ctrl = __get_vvmcs(nest->vvmcs, CPU_BASED_VM_EXEC_CONTROL);
+
+        if ( ctrl & CPU_BASED_MWAIT_EXITING )
+            nest->vmexit_pending = 1;
+
+        break;
+    }
+
+    case EXIT_REASON_PAUSE_INSTRUCTION:
+    {
+        ctrl = __get_vvmcs(nest->vvmcs, CPU_BASED_VM_EXEC_CONTROL);
+
+        if ( ctrl & CPU_BASED_PAUSE_EXITING )
+            nest->vmexit_pending = 1;
+
+        break;
+    }
+
+    case EXIT_REASON_MONITOR_INSTRUCTION:
+    {
+        ctrl = __get_vvmcs(nest->vvmcs, CPU_BASED_VM_EXEC_CONTROL);
+
+        if ( ctrl & CPU_BASED_MONITOR_EXITING )
+            nest->vmexit_pending = 1;
+
+        break;
+    }
+
+    case EXIT_REASON_DR_ACCESS:
+    {
+        ctrl = __get_vvmcs(nest->vvmcs, CPU_BASED_VM_EXEC_CONTROL);
+
+        if ( ctrl & CPU_BASED_MOV_DR_EXITING )
+            nest->vmexit_pending = 1;
+
+        break;
+    }
+
+    case EXIT_REASON_INVLPG:
+    {
+        ctrl = __get_vvmcs(nest->vvmcs, CPU_BASED_VM_EXEC_CONTROL);
+
+        if ( ctrl & CPU_BASED_INVLPG_EXITING )
+            nest->vmexit_pending = 1;
+
+        break;
+    }
+
+    case EXIT_REASON_CR_ACCESS:
+    {
+        u64 exit_qualification = __vmread(EXIT_QUALIFICATION);
+        int cr = exit_qualification & 15;
+        int write = (exit_qualification >> 4) & 3;
+        u32 mask = 0;
+
+        /* also according to guest exec_control */
+        ctrl = __get_vvmcs(nest->vvmcs, CPU_BASED_VM_EXEC_CONTROL);
+
+        if ( cr == 3 )
+        {
+            mask = write? CPU_BASED_CR3_STORE_EXITING:
+                          CPU_BASED_CR3_LOAD_EXITING;
+            if ( ctrl & mask )
+                nest->vmexit_pending = 1;
+        }
+        else if ( cr == 8 )
+        {
+            mask = write? CPU_BASED_CR8_STORE_EXITING:
+                          CPU_BASED_CR8_LOAD_EXITING;
+            if ( ctrl & mask )
+                nest->vmexit_pending = 1;
+        }
+        else  /* CR0, CR4, CLTS, LMSW */
+            nest->vmexit_pending = 1;
+
+        break;
+    }
+    default:
+        gdprintk(XENLOG_WARNING, "Unknown nested vmexit reason %x.\n",
+                 exit_reason);
+    }
+
+    if ( nest->vmexit_pending )
+        bypass_l0 = 1;
+
+    return bypass_l0;
+}
xen/arch/x86/hvm/vmx/vmx.c +6 -0
diff -r 7a9edf7654ad xen/arch/x86/hvm/vmx/vmx.c
@@ -2373,6 +2373,11 @@
      * any pending vmresume has really happened
      */
     v->arch.hvm_vmx.nest.vmresume_in_progress = 0;
+    if ( v->arch.hvm_vcpu.in_nesting )
+    {
+        if ( vmx_nest_l2_vmexit_handler(regs, exit_reason) )
+            goto out;
+    }
 
     if ( unlikely(exit_reason & VMX_EXIT_REASONS_FAILED_VMENTRY) )
         return vmx_failed_vmentry(exit_reason, regs);
@@ -2745,6 +2750,7 @@
         break;
     }
 
+out:
     if ( v->arch.hvm_vcpu.in_nesting )
         vmx_nest_idtv_handling();
 }
xen/include/asm-x86/hvm/vmx/nest.h +3 -0
diff -r 7a9edf7654ad xen/include/asm-x86/hvm/vmx/nest.h
@@ -81,4 +81,7 @@
 
 void vmx_nest_idtv_handling(void);
 
+int vmx_nest_l2_vmexit_handler(struct cpu_user_regs *regs,
+                               unsigned int exit_reason);
+
 #endif /* __ASM_X86_HVM_NEST_H__ */
xen/include/asm-x86/hvm/vmx/vmx.h +1 -0
diff -r 7a9edf7654ad xen/include/asm-x86/hvm/vmx/vmx.h
@@ -112,6 +112,7 @@
 #define EXIT_REASON_APIC_ACCESS         44
 #define EXIT_REASON_EPT_VIOLATION       48
 #define EXIT_REASON_EPT_MISCONFIG       49
+#define EXIT_REASON_INVEPT              50
 #define EXIT_REASON_RDTSCP              51
 #define EXIT_REASON_WBINVD              54
 #define EXIT_REASON_XSETBV              55
author
#57 lore raw

PATCH 13/16 vmx: nest: L2 tsc

In reply to Qing He

L2 TSC needs special handling, either rdtsc exiting is turned on or off

---

xen/arch/x86/hvm/vmx/nest.c +33 -0
diff -r 0f6400481299 xen/arch/x86/hvm/vmx/nest.c
@@ -647,6 +647,18 @@
  * Nested VMX context switch
  */
 
+u64 vmx_nest_get_tsc_offset(struct vcpu *v)
+{
+    u64 offset = 0;
+    struct vmx_nest_struct *nest = &v->arch.hvm_vmx.nest;
+
+    if ( __get_vvmcs(nest->vvmcs, CPU_BASED_VM_EXEC_CONTROL) &
+         CPU_BASED_USE_TSC_OFFSETING )
+        offset = __get_vvmcs(nest->vvmcs, TSC_OFFSET);
+
+    return offset;
+}
+
 static unsigned long vmcs_gstate_field[] = {
     /* 16 BITS */
     GUEST_ES_SELECTOR,
@@ -818,6 +830,7 @@
 
 static void load_vvmcs_guest_state(struct vmx_nest_struct *nest)
 {
+    struct vcpu *v = current;
     int i;
 
     /* vvmcs.gstate to svmcs.gstate */
@@ -828,6 +841,8 @@
     hvm_set_cr4(__get_vvmcs(nest->vvmcs, GUEST_CR4));
     hvm_set_cr3(__get_vvmcs(nest->vvmcs, GUEST_CR3));
 
+    hvm_funcs.set_tsc_offset(v, v->arch.hvm_vcpu.cache_tsc_offset);
+
     vvmcs_to_shadow(nest->vvmcs, VM_ENTRY_INTR_INFO);
     vvmcs_to_shadow(nest->vvmcs, VM_ENTRY_EXCEPTION_ERROR_CODE);
     vvmcs_to_shadow(nest->vvmcs, VM_ENTRY_INSTRUCTION_LEN);
@@ -936,6 +951,7 @@
 
 static void load_vvmcs_host_state(struct vmx_nest_struct *nest)
 {
+    struct vcpu *v = current;
     int i;
     u64 r;
 
@@ -949,6 +965,8 @@
     hvm_set_cr4(__get_vvmcs(nest->vvmcs, HOST_CR4));
     hvm_set_cr3(__get_vvmcs(nest->vvmcs, HOST_CR3));
 
+    hvm_funcs.set_tsc_offset(v, v->arch.hvm_vcpu.cache_tsc_offset);
+
     __set_vvmcs(nest->vvmcs, VM_ENTRY_INTR_INFO, 0);
 }
 
@@ -1205,6 +1223,21 @@
 
         if ( ctrl & CPU_BASED_RDTSC_EXITING )
             nest->vmexit_pending = 1;
+        else
+        {
+            uint64_t tsc;
+
+            /*
+             * special handler is needed if L1 doesn't intercept rdtsc,
+             * avoiding changing guest_tsc and messing up timekeeping in L1
+             */
+            tsc = hvm_get_guest_tsc(v);
+            tsc += __get_vvmcs(nest->vvmcs, TSC_OFFSET);
+            regs->eax = (uint32_t)tsc;
+            regs->edx = (uint32_t)(tsc >> 32);
+
+            bypass_l0 = 1;
+        }
 
         break;
     }
xen/arch/x86/hvm/vmx/vmx.c +4 -0
diff -r 0f6400481299 xen/arch/x86/hvm/vmx/vmx.c
@@ -969,6 +969,10 @@
 static void vmx_set_tsc_offset(struct vcpu *v, u64 offset)
 {
     vmx_vmcs_enter(v);
+
+    if ( v->arch.hvm_vcpu.in_nesting )
+        offset += vmx_nest_get_tsc_offset(v);
+
     __vmwrite(TSC_OFFSET, offset);
 #if defined (__i386__)
     __vmwrite(TSC_OFFSET_HIGH, offset >> 32);
xen/include/asm-x86/hvm/vmx/nest.h +2 -0
diff -r 0f6400481299 xen/include/asm-x86/hvm/vmx/nest.h
@@ -69,6 +69,8 @@
                                             unsigned long value);
 void vmx_nest_update_exception_bitmap(struct vcpu *v, unsigned long value);
 
+u64 vmx_nest_get_tsc_offset(struct vcpu *v);
+
 void vmx_nest_idtv_handling(void);
 
 int vmx_nest_l2_vmexit_handler(struct cpu_user_regs *regs,
author
#58 lore raw

PATCH 14/16 vmx: nest: CR0.TS and #NM

In reply to Qing He

1. #NM exits from L2 should be handled by L0 if it wants.
2. HOST_CR0.TS may need to be updated after L1<->L2 switch

---

xen/arch/x86/hvm/vmx/nest.c +12 -1
diff -r a5f6653f4c5a xen/arch/x86/hvm/vmx/nest.c
@@ -791,6 +791,9 @@
     regs->rsp = __get_vvmcs(nest->vvmcs, GUEST_RSP);
     regs->rflags = __get_vvmcs(nest->vvmcs, GUEST_RFLAGS);
 
+    /* updating host cr0 to sync TS bit */
+    __vmwrite(HOST_CR0, v->arch.hvm_vmx.host_cr0);
+
     /* TODO: EPT_POINTER */
 }
 
@@ -927,6 +930,9 @@
     regs->rsp = __get_vvmcs(nest->vvmcs, HOST_RSP);
     regs->rflags = __vmread(GUEST_RFLAGS);
 
+    /* updating host cr0 to sync TS bit */
+    __vmwrite(HOST_CR0, v->arch.hvm_vmx.host_cr0);
+
     vmreturn(regs, VMSUCCEED);
 }
 
@@ -1036,13 +1042,18 @@
 
         /*
          * decided by L0 and L1 exception bitmap, if the vetor is set by
-         * both, L0 has priority on #PF, L1 has priority on others
+         * both, L0 has priority on #PF and #NM, L1 has priority on others
          */
         if ( vector == TRAP_page_fault )
         {
             if ( paging_mode_hap(v->domain) )
                 nest->vmexit_pending = 1;
         }
+        else if ( vector == TRAP_no_device )
+        {
+            if ( v->fpu_dirtied )
+                nest->vmexit_pending = 1;
+        }
         else if ( (intr_info & valid_mask) == valid_mask )
         {
             exec_bitmap =__get_vvmcs(nest->vvmcs, EXCEPTION_BITMAP);
author
#59 lore raw

PATCH 15/16 vmx: nest: capability reporting MSRs

In reply to Qing He

handles VMX capability reporting MSRs.
Some features are masked so L1 would see a rather
simple configuration

---

xen/arch/x86/hvm/vmx/nest.c +88 -0
diff -r 694dcf6c3f06 xen/arch/x86/hvm/vmx/nest.c
@@ -1352,3 +1352,91 @@
 
     return bypass_l0;
 }
+
+/*
+ * Capability reporting
+ */
+int vmx_nest_msr_read_intercept(unsigned int msr, u64 *msr_content)
+{
+    u32 eax, edx;
+    u64 data = 0;
+    int r = 1;
+    u32 mask = 0;
+
+    if ( !is_nested_avail(current->domain) )
+        return 0;
+
+    switch (msr) {
+    case MSR_IA32_VMX_BASIC:
+        rdmsr(msr, eax, edx);
+        data = edx;
+        data = (data & ~0x1fff) | 0x1000;     /* request 4KB for guest VMCS */
+        data &= ~(1 << 23);                   /* disable TRUE_xxx_CTLS */
+        data = (data << 32) | VVMCS_REVISION; /* VVMCS revision */
+        break;
+    case MSR_IA32_VMX_PINBASED_CTLS:
+#define REMOVED_PIN_CONTROL_CAP (PIN_BASED_PREEMPT_TIMER)
+        rdmsr(msr, eax, edx);
+        data = edx;
+        data = (data << 32) | eax;
+        break;
+    case MSR_IA32_VMX_PROCBASED_CTLS:
+        rdmsr(msr, eax, edx);
+#define REMOVED_EXEC_CONTROL_CAP (CPU_BASED_TPR_SHADOW \
+            | CPU_BASED_ACTIVATE_MSR_BITMAP            \
+            | CPU_BASED_ACTIVATE_SECONDARY_CONTROLS)
+        data = edx & ~REMOVED_EXEC_CONTROL_CAP;
+        data = (data << 32) | eax;
+        break;
+    case MSR_IA32_VMX_EXIT_CTLS:
+        rdmsr(msr, eax, edx);
+#define REMOVED_EXIT_CONTROL_CAP (VM_EXIT_SAVE_GUEST_PAT \
+            | VM_EXIT_LOAD_HOST_PAT                      \
+            | VM_EXIT_SAVE_GUEST_EFER                    \
+            | VM_EXIT_LOAD_HOST_EFER                     \
+            | VM_EXIT_SAVE_PREEMPT_TIMER)
+        data = edx & ~REMOVED_EXIT_CONTROL_CAP;
+        data = (data << 32) | eax;
+        break;
+    case MSR_IA32_VMX_ENTRY_CTLS:
+        rdmsr(msr, eax, edx);
+#define REMOVED_ENTRY_CONTROL_CAP (VM_ENTRY_LOAD_GUEST_PAT \
+            | VM_ENTRY_LOAD_GUEST_EFER)
+        data = edx & ~REMOVED_ENTRY_CONTROL_CAP;
+        data = (data << 32) | eax;
+        break;
+    case MSR_IA32_VMX_PROCBASED_CTLS2:
+        mask = 0;
+
+        rdmsr(msr, eax, edx);
+        data = edx & mask;
+        data = (data << 32) | eax;
+        break;
+
+    /* pass through MSRs */
+    case IA32_FEATURE_CONTROL_MSR:
+    case MSR_IA32_VMX_MISC:
+    case MSR_IA32_VMX_CR0_FIXED0:
+    case MSR_IA32_VMX_CR0_FIXED1:
+    case MSR_IA32_VMX_CR4_FIXED0:
+    case MSR_IA32_VMX_CR4_FIXED1:
+    case MSR_IA32_VMX_VMCS_ENUM:
+        rdmsr(msr, eax, edx);
+        data = edx;
+        data = (data << 32) | eax;
+        break;
+
+    default:
+        r = 0;
+        break;
+    }
+
+    *msr_content = data;
+    return r;
+}
+
+int vmx_nest_msr_write_intercept(unsigned int msr, u64 msr_content)
+{
+    /* silently ignore for now */
+    return 1;
+}
xen/arch/x86/hvm/vmx/vmx.c +10 -4
diff -r 694dcf6c3f06 xen/arch/x86/hvm/vmx/vmx.c
@@ -1877,8 +1877,11 @@
         *msr_content |= (u64)__vmread(GUEST_IA32_DEBUGCTL_HIGH) << 32;
 #endif
         break;
-    case MSR_IA32_VMX_BASIC...MSR_IA32_VMX_PROCBASED_CTLS2:
-        goto gp_fault;
+    case IA32_FEATURE_CONTROL_MSR:
+    case MSR_IA32_VMX_BASIC...MSR_IA32_VMX_TRUE_ENTRY_CTLS:
+        if ( !vmx_nest_msr_read_intercept(msr, msr_content) )
+            goto gp_fault;
+        break;
     case MSR_IA32_MISC_ENABLE:
         rdmsrl(MSR_IA32_MISC_ENABLE, *msr_content);
         /* Debug Trace Store is not supported. */
@@ -2043,8 +2046,11 @@
 
         break;
     }
-    case MSR_IA32_VMX_BASIC...MSR_IA32_VMX_PROCBASED_CTLS2:
-        goto gp_fault;
+    case IA32_FEATURE_CONTROL_MSR:
+    case MSR_IA32_VMX_BASIC...MSR_IA32_VMX_TRUE_ENTRY_CTLS:
+        if ( !vmx_nest_msr_write_intercept(msr, msr_content) )
+            goto gp_fault;
+        break;
     default:
         if ( vpmu_do_wrmsr(msr, msr_content) )
             return X86EMUL_OKAY;
xen/include/asm-x86/hvm/vmx/nest.h +5 -0
diff -r 694dcf6c3f06 xen/include/asm-x86/hvm/vmx/nest.h
@@ -76,4 +76,9 @@
 int vmx_nest_l2_vmexit_handler(struct cpu_user_regs *regs,
                                unsigned int exit_reason);
 
+int vmx_nest_msr_read_intercept(unsigned int msr,
+                                u64 *msr_content);
+int vmx_nest_msr_write_intercept(unsigned int msr,
+                                 u64 msr_content);
+
 #endif /* __ASM_X86_HVM_NEST_H__ */
xen/include/asm-x86/hvm/vmx/vmcs.h +5 -0
diff -r 694dcf6c3f06 xen/include/asm-x86/hvm/vmx/vmcs.h
@@ -161,18 +161,23 @@
 #define PIN_BASED_EXT_INTR_MASK         0x00000001
 #define PIN_BASED_NMI_EXITING           0x00000008
 #define PIN_BASED_VIRTUAL_NMIS          0x00000020
+#define PIN_BASED_PREEMPT_TIMER         0x00000040
 extern u32 vmx_pin_based_exec_control;
 
 #define VM_EXIT_IA32E_MODE              0x00000200
 #define VM_EXIT_ACK_INTR_ON_EXIT        0x00008000
 #define VM_EXIT_SAVE_GUEST_PAT          0x00040000
 #define VM_EXIT_LOAD_HOST_PAT           0x00080000
+#define VM_EXIT_SAVE_GUEST_EFER         0x00100000
+#define VM_EXIT_LOAD_HOST_EFER          0x00200000
+#define VM_EXIT_SAVE_PREEMPT_TIMER      0x00400000
 extern u32 vmx_vmexit_control;
 
 #define VM_ENTRY_IA32E_MODE             0x00000200
 #define VM_ENTRY_SMM                    0x00000400
 #define VM_ENTRY_DEACT_DUAL_MONITOR     0x00000800
 #define VM_ENTRY_LOAD_GUEST_PAT         0x00004000
+#define VM_ENTRY_LOAD_GUEST_EFER        0x00008000
 extern u32 vmx_vmentry_control;
 
 #define SECONDARY_EXEC_VIRTUALIZE_APIC_ACCESSES 0x00000001
xen/include/asm-x86/msr-index.h +1 -0
diff -r 694dcf6c3f06 xen/include/asm-x86/msr-index.h
@@ -172,6 +172,7 @@
 #define MSR_IA32_VMX_CR0_FIXED1                 0x487
 #define MSR_IA32_VMX_CR4_FIXED0                 0x488
 #define MSR_IA32_VMX_CR4_FIXED1                 0x489
+#define MSR_IA32_VMX_VMCS_ENUM                  0x48a
 #define MSR_IA32_VMX_PROCBASED_CTLS2            0x48b
 #define MSR_IA32_VMX_EPT_VPID_CAP               0x48c
 #define MSR_IA32_VMX_TRUE_PINBASED_CTLS         0x48d
#60 lore raw

Re: [PATCH 15/16] vmx: nest: capability reporting MSRs

Quoted 3 lines · Qing He

handles VMX capability reporting MSRs.
Some features are masked so L1 would see a rather
simple configuration

As I said last time, would it be better to whitelist features that we know are safely virtualized?

Quoted 22 lines

---

diff -r 694dcf6c3f06 xen/arch/x86/hvm/vmx/nest.c

xen/arch/x86/hvm/vmx/nest.c +10 -0
@@ -1352,3 +1352,91 @@
 
     return bypass_l0;
 }
+
+/*
+ * Capability reporting
+ */
+int vmx_nest_msr_read_intercept(unsigned int msr, u64 *msr_content)
+{
+    u32 eax, edx;
+    u64 data = 0;
+    int r = 1;
+    u32 mask = 0;

xen/arch/x86/hvm/vmx/nest.c line 1374

Quoted 10 lines
xen/arch/x86/hvm/vmx/nest.c +10 -0
+
+    if ( !is_nested_avail(current->domain) )
+        return 0;
+
+    switch (msr) {
+    case MSR_IA32_VMX_BASIC:
+        rdmsr(msr, eax, edx);
+        data = edx;
+        data = (data & ~0x1fff) | 0x1000;     /* request 4KB for guest VMCS */
+        data &= ~(1 << 23);                   /* disable TRUE_xxx_CTLS */

Magic number - please use a macro to define it.

xen/arch/x86/hvm/vmx/nest.c line 1378

Quoted 4 lines
xen/arch/x86/hvm/vmx/nest.c +4 -0
+        data = (data << 32) | VVMCS_REVISION; /* VVMCS revision */
+        break;
+    case MSR_IA32_VMX_PINBASED_CTLS:
+#define REMOVED_PIN_CONTROL_CAP (PIN_BASED_PREEMPT_TIMER)

You define this mask but don't actually mask anything with it.

Cheers,

Tim.

Signature · 3 lines
Tim Deegan <[email protected]>
Principal Software Engineer, XenServer Engineering
Citrix Systems UK Ltd.  (Company #02937203, SL9 0BG)
#61 lore raw

Re: [PATCH 15/16] vmx: nest: capability reporting MSRs

In reply to Qing He
Quoted 6 lines · On Wednesday 08 September 2010 17:22:23 Qing He wrote:

handles VMX capability reporting MSRs.
Some features are masked so L1 would see a rather
simple configuration

Are there any vmx capability features that are read out via cpuid ? If yes, then that code belongs into the tools patch.

In SVM the nestedhvm_vcpu_features hook is empty and for MSRs there are already two msr hooks namely msr_read_intercept and msr_write_intercept. I assume the functions below all called from there directly or indirectly.

That renders the nestedhvm_vcpu_features hook useless and will remove it.

Christoph

Quoted 184 lines

---

diff -r 694dcf6c3f06 xen/arch/x86/hvm/vmx/nest.c

xen/arch/x86/hvm/vmx/nest.c +19 -0
@@ -1352,3 +1352,91 @@
 
     return bypass_l0;
 }
+
+/*
+ * Capability reporting
+ */
+int vmx_nest_msr_read_intercept(unsigned int msr, u64 *msr_content)
+{
+    u32 eax, edx;
+    u64 data = 0;
+    int r = 1;
+    u32 mask = 0;
+
+    if ( !is_nested_avail(current->domain) )
+        return 0;
+
+    switch (msr) {
+    case MSR_IA32_VMX_BASIC:
+        rdmsr(msr, eax, edx);
+        data = edx;
+        data = (data & ~0x1fff) | 0x1000;     /* request 4KB for guest

VMCS */ + data &= ~(1 << 23); /* disable
TRUE_xxx_CTLS */ + data = (data << 32) | VVMCS_REVISION; /* VVMCS
revision */ + break;

xen/arch/x86/hvm/vmx/nest.c +66 -0
+    case MSR_IA32_VMX_PINBASED_CTLS:
+#define REMOVED_PIN_CONTROL_CAP (PIN_BASED_PREEMPT_TIMER)
+        rdmsr(msr, eax, edx);
+        data = edx;
+        data = (data << 32) | eax;
+        break;
+    case MSR_IA32_VMX_PROCBASED_CTLS:
+        rdmsr(msr, eax, edx);
+#define REMOVED_EXEC_CONTROL_CAP (CPU_BASED_TPR_SHADOW \
+            | CPU_BASED_ACTIVATE_MSR_BITMAP            \
+            | CPU_BASED_ACTIVATE_SECONDARY_CONTROLS)
+        data = edx & ~REMOVED_EXEC_CONTROL_CAP;
+        data = (data << 32) | eax;
+        break;
+    case MSR_IA32_VMX_EXIT_CTLS:
+        rdmsr(msr, eax, edx);
+#define REMOVED_EXIT_CONTROL_CAP (VM_EXIT_SAVE_GUEST_PAT \
+            | VM_EXIT_LOAD_HOST_PAT                      \
+            | VM_EXIT_SAVE_GUEST_EFER                    \
+            | VM_EXIT_LOAD_HOST_EFER                     \
+            | VM_EXIT_SAVE_PREEMPT_TIMER)
+        data = edx & ~REMOVED_EXIT_CONTROL_CAP;
+        data = (data << 32) | eax;
+        break;
+    case MSR_IA32_VMX_ENTRY_CTLS:
+        rdmsr(msr, eax, edx);
+#define REMOVED_ENTRY_CONTROL_CAP (VM_ENTRY_LOAD_GUEST_PAT \
+            | VM_ENTRY_LOAD_GUEST_EFER)
+        data = edx & ~REMOVED_ENTRY_CONTROL_CAP;
+        data = (data << 32) | eax;
+        break;
+    case MSR_IA32_VMX_PROCBASED_CTLS2:
+        mask = 0;
+
+        rdmsr(msr, eax, edx);
+        data = edx & mask;
+        data = (data << 32) | eax;
+        break;
+
+    /* pass through MSRs */
+    case IA32_FEATURE_CONTROL_MSR:
+    case MSR_IA32_VMX_MISC:
+    case MSR_IA32_VMX_CR0_FIXED0:
+    case MSR_IA32_VMX_CR0_FIXED1:
+    case MSR_IA32_VMX_CR4_FIXED0:
+    case MSR_IA32_VMX_CR4_FIXED1:
+    case MSR_IA32_VMX_VMCS_ENUM:
+        rdmsr(msr, eax, edx);
+        data = edx;
+        data = (data << 32) | eax;
+        break;
+
+    default:
+        r = 0;
+        break;
+    }
+
+    *msr_content = data;
+    return r;
+}
+
+int vmx_nest_msr_write_intercept(unsigned int msr, u64 msr_content)
+{
+    /* silently ignore for now */
+    return 1;
+}

diff -r 694dcf6c3f06 xen/arch/x86/hvm/vmx/vmx.c

xen/arch/x86/hvm/vmx/vmx.c +10 -4
@@ -1877,8 +1877,11 @@
         *msr_content |= (u64)__vmread(GUEST_IA32_DEBUGCTL_HIGH) << 32;
 #endif
         break;
-    case MSR_IA32_VMX_BASIC...MSR_IA32_VMX_PROCBASED_CTLS2:
-        goto gp_fault;
+    case IA32_FEATURE_CONTROL_MSR:
+    case MSR_IA32_VMX_BASIC...MSR_IA32_VMX_TRUE_ENTRY_CTLS:
+        if ( !vmx_nest_msr_read_intercept(msr, msr_content) )
+            goto gp_fault;
+        break;
     case MSR_IA32_MISC_ENABLE:
         rdmsrl(MSR_IA32_MISC_ENABLE, *msr_content);
         /* Debug Trace Store is not supported. */
@@ -2043,8 +2046,11 @@
 
         break;
     }
-    case MSR_IA32_VMX_BASIC...MSR_IA32_VMX_PROCBASED_CTLS2:
-        goto gp_fault;
+    case IA32_FEATURE_CONTROL_MSR:
+    case MSR_IA32_VMX_BASIC...MSR_IA32_VMX_TRUE_ENTRY_CTLS:
+        if ( !vmx_nest_msr_write_intercept(msr, msr_content) )
+            goto gp_fault;
+        break;
     default:
         if ( vpmu_do_wrmsr(msr, msr_content) )
             return X86EMUL_OKAY;

diff -r 694dcf6c3f06 xen/include/asm-x86/hvm/vmx/nest.h

xen/include/asm-x86/hvm/vmx/nest.h +5 -0
@@ -76,4 +76,9 @@
 int vmx_nest_l2_vmexit_handler(struct cpu_user_regs *regs,
                                unsigned int exit_reason);
 
+int vmx_nest_msr_read_intercept(unsigned int msr,
+                                u64 *msr_content);
+int vmx_nest_msr_write_intercept(unsigned int msr,
+                                 u64 msr_content);
+
 #endif /* __ASM_X86_HVM_NEST_H__ */

diff -r 694dcf6c3f06 xen/include/asm-x86/hvm/vmx/vmcs.h

xen/include/asm-x86/hvm/vmx/vmcs.h +5 -0
@@ -161,18 +161,23 @@
 #define PIN_BASED_EXT_INTR_MASK         0x00000001
 #define PIN_BASED_NMI_EXITING           0x00000008
 #define PIN_BASED_VIRTUAL_NMIS          0x00000020
+#define PIN_BASED_PREEMPT_TIMER         0x00000040
 extern u32 vmx_pin_based_exec_control;
 
 #define VM_EXIT_IA32E_MODE              0x00000200
 #define VM_EXIT_ACK_INTR_ON_EXIT        0x00008000
 #define VM_EXIT_SAVE_GUEST_PAT          0x00040000
 #define VM_EXIT_LOAD_HOST_PAT           0x00080000
+#define VM_EXIT_SAVE_GUEST_EFER         0x00100000
+#define VM_EXIT_LOAD_HOST_EFER          0x00200000
+#define VM_EXIT_SAVE_PREEMPT_TIMER      0x00400000
 extern u32 vmx_vmexit_control;
 
 #define VM_ENTRY_IA32E_MODE             0x00000200
 #define VM_ENTRY_SMM                    0x00000400
 #define VM_ENTRY_DEACT_DUAL_MONITOR     0x00000800
 #define VM_ENTRY_LOAD_GUEST_PAT         0x00004000
+#define VM_ENTRY_LOAD_GUEST_EFER        0x00008000
 extern u32 vmx_vmentry_control;
 
 #define SECONDARY_EXEC_VIRTUALIZE_APIC_ACCESSES 0x00000001

diff -r 694dcf6c3f06 xen/include/asm-x86/msr-index.h

xen/include/asm-x86/msr-index.h +1 -0
@@ -172,6 +172,7 @@
 #define MSR_IA32_VMX_CR0_FIXED1                 0x487
 #define MSR_IA32_VMX_CR4_FIXED0                 0x488
 #define MSR_IA32_VMX_CR4_FIXED1                 0x489
+#define MSR_IA32_VMX_VMCS_ENUM                  0x48a
 #define MSR_IA32_VMX_PROCBASED_CTLS2            0x48b
 #define MSR_IA32_VMX_EPT_VPID_CAP               0x48c
 #define MSR_IA32_VMX_TRUE_PINBASED_CTLS         0x48d

_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel

Signature · 6 lines
---to satisfy European Law for business letters:
Advanced Micro Devices GmbH
Einsteinring 24, 85609 Dornach b. Muenchen
Geschaeftsfuehrer: Alberto Bozzo, Andrew Bowd
Sitz: Dornach, Gemeinde Aschheim, Landkreis Muenchen
Registergericht Muenchen, HRB Nr. 43632
#62 lore raw

RE: [PATCH 15/16] vmx: nest: capability reporting MSRs

Quoted 11 lines · Christoph Egger
Quoted 6 lines · On Wednesday 08 September 2010 17:22:23 Qing He wrote:

handles VMX capability reporting MSRs.
Some features are masked so L1 would see a rather
simple configuration

Are there any vmx capability features that are read out via cpuid ? If yes, then that code belongs into the tools patch.

That is in 16.txt, you can include in your code. Once you fixed the MAX LEAF issue, I can ack that patch.

Quoted 5 lines

In SVM the nestedhvm_vcpu_features hook is empty and for MSRs there are already two msr hooks namely msr_read_intercept and msr_write_intercept. I assume the functions below all called from there directly or indirectly.

No. this interception is for L1 guest.

Quoted 3 lines

That renders the nestedhvm_vcpu_features hook useless and will remove it.

A step toward my wish of light weight wrapper, glad to see! The more you removed, the more I can ack, untill the point only those necessary APIs were left, such as nested EPT/NPT, heavily revisited interrupt injection API after removing the new namespace.

Thx, Eddie

#63 lore raw

Re: [PATCH 15/16] vmx: nest: capability reporting MSRs

Quoted 6 lines · Eddie
Quoted 5 lines · Christoph Egger
Quoted 4 lines · On Wednesday 08 September 2010 17:22:23 Qing He wrote:

handles VMX capability reporting MSRs.
Some features are masked so L1 would see a rather
simple configuration

Quoted 8 lines
Quoted 5 lines
Quoted 2 lines

Are there any vmx capability features that are read out via cpuid ? If yes, then that code belongs into the tools patch.

That is in 16.txt, you can include in your code. Once you fixed the MAX LEAF issue, I can ack that patch.

Andre tried you to explain why the MAXLEAF change is not a problem. Is there an other problem ?

Quoted 7 lines
Quoted 5 lines

In SVM the nestedhvm_vcpu_features hook is empty and for MSRs there are already two msr hooks namely msr_read_intercept and msr_write_intercept. I assume the functions below all called from there directly or indirectly.

No. this interception is for L1 guest.

Yes, that is how I understood.
What I mean is that you can call the functions from vmx_msr_read_intercept/vmx_msr_write_intercept.

Quoted 5 lines
Quoted 3 lines

That renders the nestedhvm_vcpu_features hook useless and will remove it.

A step toward my wish of light weight wrapper, glad to see!

I would have removed it earlier, if you were telling me what I said above which is what I want to know.

Quoted 3 lines

The more you removed, the more I can ack, untill the point only those necessary APIs were left, such as nested EPT/NPT, heavily revisited interrupt injection API after removing the new namespace.

When you tell me which adjustments you exactly need I will do the changes w/o breaking SVM, of course.

Christoph

Signature · 6 lines
---to satisfy European Law for business letters:
Advanced Micro Devices GmbH
Einsteinring 24, 85609 Dornach b. Muenchen
Geschaeftsfuehrer: Alberto Bozzo, Andrew Bowd
Sitz: Dornach, Gemeinde Aschheim, Landkreis Muenchen
Registergericht Muenchen, HRB Nr. 43632
#64 lore raw

RE: [PATCH 15/16] vmx: nest: capability reporting MSRs

Quoted 10 lines · Christoph Egger
Quoted 9 lines · Eddie
Quoted 8 lines · Christoph Egger
Quoted 6 lines · On Wednesday 08 September 2010 17:22:23 Qing He wrote:

handles VMX capability reporting MSRs.
Some features are masked so L1 would see a rather
simple configuration

Quoted 9 lines
Quoted 5 lines
Quoted 2 lines

Are there any vmx capability features that are read out via cpuid ? If yes, then that code belongs into the tools patch.

That is in 16.txt, you can include in your code. Once you fixed the MAX LEAF issue, I can ack that patch.

Andre tried you to explain why the MAXLEAF change is not a problem. Is there an other problem ?

I replied to him and figured out the problem.

That is assuming Intel future processor won't implement more than 0x8...8 leaf, that is not good thing to do. And the name of MAX_LEAF is then wrong to Intel CPU.

author
#65 lore raw

PATCH 16/16 vmx: nest: expose cpuid and CR4.VMXE

In reply to Qing He

expose VMX cpuid and allow guest to enable VMX.

---

tools/libxc/xc_cpuid_x86.c +9 -0
diff -r 3f40a1f79cf8 tools/libxc/xc_cpuid_x86.c
@@ -128,8 +128,17 @@
     const unsigned int *input, unsigned int *regs,
     int is_pae)
 {
+    unsigned long nest;
+
     switch ( input[0] )
     {
+    case 0x00000001:
+        /* ECX[5] is availability of VMX */
+        xc_get_hvm_param(xch, domid, HVM_PARAM_NESTEDHVM, &nest);
+        if (nest)
+            regs[2] |= 0x20;
+        break;
+
     case 0x00000004:
         /*
          * EAX[31:26] is Maximum Cores Per Package (minus one).
xen/include/asm-x86/hvm/hvm.h +2 -1
diff -r 3f40a1f79cf8 xen/include/asm-x86/hvm/hvm.h
@@ -295,7 +295,8 @@
         X86_CR4_DE  | X86_CR4_PSE | X86_CR4_PAE |       \
         X86_CR4_MCE | X86_CR4_PGE | X86_CR4_PCE |       \
         X86_CR4_OSFXSR | X86_CR4_OSXMMEXCPT |           \
-        (cpu_has_xsave ? X86_CR4_OSXSAVE : 0))))
+        (cpu_has_xsave ? X86_CR4_OSXSAVE : 0)   |       \
+        X86_CR4_VMXE)))
 
 /* These exceptions must always be intercepted. */
 #define HVM_TRAP_MASK ((1U << TRAP_machine_check) | (1U << TRAP_invalid_op))
#66 lore raw

Re: [PATCH 16/16] vmx: nest: expose cpuid and CR4.VMXE

Quoted 15 lines · On Wednesday 08 September 2010 17:22:24 Qing He wrote:

expose VMX cpuid and allow guest to enable VMX.

---

diff -r 3f40a1f79cf8 tools/libxc/xc_cpuid_x86.c

tools/libxc/xc_cpuid_x86.c +1 -0
@@ -128,8 +128,17 @@
     const unsigned int *input, unsigned int *regs,
     int is_pae)
 {
+    unsigned long nest;

tools/libxc/xc_cpuid_x86.c line 141

Quoted 10 lines
tools/libxc/xc_cpuid_x86.c +8 -0
+
     switch ( input[0] )
     {
+    case 0x00000001:
+        /* ECX[5] is availability of VMX */
+        xc_get_hvm_param(xch, domid, HVM_PARAM_NESTEDHVM, &nest);
+        if (nest)
+            regs[2] |= 0x20;
+        break;
+

I merged this part into my tools patch this way:

              /* ECX[5] is availability of VMX */
              if (is_nstedhvm)
                  set_bit(X86_FEATURE_VMXE, regs[2]);
              break;

xen/include/asm-x86/hvm/hvm.h line 299

Quoted 13 lines
tools/libxc/xc_cpuid_x86.c +0 -0
     case 0x00000004:
         /*
          * EAX[31:26] is Maximum Cores Per Package (minus one).

diff -r 3f40a1f79cf8 xen/include/asm-x86/hvm/hvm.h

xen/include/asm-x86/hvm/hvm.h +2 -1
@@ -295,7 +295,8 @@
         X86_CR4_DE  | X86_CR4_PSE | X86_CR4_PAE |       \
         X86_CR4_MCE | X86_CR4_PGE | X86_CR4_PCE |       \
         X86_CR4_OSFXSR | X86_CR4_OSXMMEXCPT |           \
-        (cpu_has_xsave ? X86_CR4_OSXSAVE : 0))))
+        (cpu_has_xsave ? X86_CR4_OSXSAVE : 0)   |       \
+        X86_CR4_VMXE)))

I changed this to

         (cpu_has_vmx ? X86_CR4_VMXE : 0))))

where cpu_has vmx is in <asm/cpufeature.h> defined as

#define cpu_has_vmx boot_cpu_has(X86_FEATURE_VMXE)

Christoph

Quoted 4 lines
xen/include/asm-x86/hvm/hvm.h +0 -0
 /* These exceptions must always be intercepted. */
 #define HVM_TRAP_MASK ((1U << TRAP_machine_check) | (1U <<
 TRAP_invalid_op))
Signature · 6 lines
---to satisfy European Law for business letters:
Advanced Micro Devices GmbH
Einsteinring 24, 85609 Dornach b. Muenchen
Geschaeftsfuehrer: Alberto Bozzo, Andrew Bowd
Sitz: Dornach, Gemeinde Aschheim, Landkreis Muenchen
Registergericht Muenchen, HRB Nr. 43632
#67 lore raw
In reply to Qing He

Hi,

Thanks for this updated patchset, and for addressing a lot of what I mentioned last time. I've made a few comments on individual patches; since the nested EPT changes aren't in the series this time I don't think any of it needs an explicit ack from me with my x86-mm-maintainer hat on.

Apart from that I'll say again that some common code should be found between these and the SVM patches, if that's possible. Not necessarily the interface that Christoph is proposing, but since he's gone to the effort it seems like a good starting point.

I would certainly like to see common interrupt-routing changes next time these patches are posted, even if we're still discussing the vmexit path.

Cheers,

Tim.

Quoted 48 lines · Qing He

This patch set is the upgraded version of nested virtualization for VMX, that allows a VMX guest (L1) to run other VMX guests (L2).

The nested virtualization for vmx is built on homogeneous L1 and L2 for better performance and minimal emulation, the common code involved is small and contained in patch 03/16, for two flags, one for feature availability, the other for indicating current mode.

The userspace components (xend/xm/xl) is not included since Christopher's userspace patch has similar coverage. vEPT is not included as well because it's still WIP.

Major changes to last version:
- address Tim's comments on error handling and others
- split context switch into smaller pieces with certain restructure for better readability
- update interrupt handling, rewrite comments
- move cpuid into userspace
- etc.

The patch set includes the following patches.

[PATCH 01/16] vmx: nest: rename host_vmcs
[PATCH 02/16] vmx: nest: wrapper for control update [PATCH 03/16] vmx: nest: nested availability and status flags [PATCH 04/16] vmx: nest: nested control structure [PATCH 05/16] vmx: nest: virtual vmcs layout
[PATCH 06/16] vmx: nest: handling VMX instruction exits [PATCH 07/16] vmx: nest: switch current vmcs
[PATCH 08/16] vmx: nest: vmresume/vmlaunch
[PATCH 09/16] vmx: nest: shadow controls
[PATCH 10/16] vmx: nest: L1 <-> L2 context switch [PATCH 11/16] vmx: nest: interrupt handling
[PATCH 12/16] vmx: nest: VMExit handler in L2 [PATCH 13/16] vmx: nest: L2 tsc
[PATCH 14/16] vmx: nest: CR0.TS and #NM
[PATCH 15/16] vmx: nest: capability reporting MSRs [PATCH 16/16] vmx: nest: expose cpuid and CR4.VMXE

Thanks,
Qing He

_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel

Signature · 3 lines
Tim Deegan <[email protected]>
Principal Software Engineer, XenServer Engineering
Citrix Systems UK Ltd.  (Company #02937203, SL9 0BG)