gfxboot VMX workaround v2

Hi,

this is an improved version of the patch I sent several weeks ago to this list. Functionally nothing changed; it still hacks into gfxboot and patches it to work on Intel CPUs on the fly. The big difference is that this version is cleaned up and should work with every future CPU available.

Please do _not_ apply this patch. I send it to the list only for interested people, who would like to have a working version of KVM for their systems right now. It is neither a proper fix nor the right approach to deal with this issue. It is merely a hack that works for me and maybe for others too.

Alex

------------------------------------------------------------------------

-------------------------------------------------------------------------
This SF.net email is sponsored by the 2008 JavaOne(SM) Conference Register now and save $200. Hurry, offer ends at 11:59 p.m., Monday, April 7! Use priority code J8TLD2.
http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone
------------------------------------------------------------------------

_______________________________________________
kvm-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/kvm-devel

Quoted 10 lines · Alexander Graf

Hi,

this is an improved version of the patch I sent several weeks ago to this list. Functionally nothing changed; it still hacks into gfxboot and
patches it to work on Intel CPUs on the fly. The big difference is that
this version is cleaned up and should work with every future CPU available.

Quoted 8 lines

Please do _not_ apply this patch. I send it to the list only for interested people, who would like to have a working version of KVM for
their systems right now. It is neither a proper fix nor the right approach to deal with this issue. It is merely a hack that works for me
and maybe for others too.

Perhaps a viable way to fix this upstream would be to catch the vmentry failure, look to see if SS.CPL != CS.CPL, and if so, invoke x86_emulate() in a loop until SS.CPL == CS.CPL.

There are very few instructions in gfxboot that would need to be added to x86_emulate (if they aren't already there).

Quoted 15 lines · Anthony Liguori

Quoted 13 lines · Alexander Graf

Hi,

this is an improved version of the patch I sent several weeks ago to this list. Functionally nothing changed; it still hacks into gfxboot and patches it to work on Intel CPUs on the fly. The big difference is that this version is cleaned up and should work with every future CPU available.

Please do _not_ apply this patch. I send it to the list only for interested people, who would like to have a working version of KVM for their systems right now. It is neither a proper fix nor the right approach to deal with this issue. It is merely a hack that works for me and maybe for others too.

Quoted 3 lines

Perhaps a viable way to fix this upstream would be to catch the vmentry failure, look to see if SS.CPL != CS.CPL, and if so, invoke x86_emulate() in a loop until SS.CPL == CS.CPL.

There are very few instructions in gfxboot that would need to be added to x86_emulate (if they aren't already there).

In a previous thread Avi already explained a quite reasonable way to approach this problem, which I believe is a really good approach. He wanted to x86_emulate until the environment is "VMX friendly" again, thus resolving big real mode problems as well.

I personally agree that the real approach is way superior to my patch. I just won't have the time to do it in the near future and not being able to boot intuitively hurts KVM users unnecessarily ;-).

Regards,

Alex

Quoted 28 lines · Alexander Graf

Quoted 25 lines · Anthony Liguori

Quoted 11 lines · Alexander Graf

Hi,

this is an improved version of the patch I sent several weeks ago to
this list. Functionally nothing changed; it still hacks into gfxboot and
patches it to work on Intel CPUs on the fly. The big difference is that
this version is cleaned up and should work with every future CPU available.

Quoted 8 lines

Please do _not_ apply this patch. I send it to the list only for interested people, who would like to have a working version of KVM for
their systems right now. It is neither a proper fix nor the right approach to deal with this issue. It is merely a hack that works for me
and maybe for others too.

Perhaps a viable way to fix this upstream would be to catch the vmentry failure, look to see if SS.CPL != CS.CPL, and if so, invoke x86_emulate() in a loop until SS.CPL == CS.CPL.

Quoted 7 lines

Quoted 2 lines

There are very few instructions in gfxboot that would need to be added to x86_emulate (if they aren't already there).

In a previous thread Avi already explained a quite reasonable way to approach this problem, which I believe is a really good approach. He wanted to x86_emulate until the environment is "VMX friendly" again, thus resolving big real mode problems as well.

I've got a slightly lamer approach than what Avi probably wants. I lost interest in updating x86_emulate once I realized how far xen's copy has gotten. To get GFXBOOT 3.3.28 working just requires adding far jmp to x86_emulate. The sequence should look like:

      jmp pm_seg.prog_c32:switch_to_pm_20

switch_to_pm_20:

      bits 32

      mov ax,pm_seg.prog_d16
      mov ds,ax
      mov eax,ss

Which means we'll get 3 vmentry failures. The two moves should already be supported by x86_emulate but I haven't confirmed. It's not a complete solution to our real mode woes but I think it's a reasonable first step.

Regards,

Anthony Liguori

Quoted 4 lines

I personally agree that the real approach is way superior to my patch. I just won't have the time to do it in the near future and not being able to boot intuitively hurts KVM users unnecessarily ;-).

Quoted 3 lines

Regards,

Alex

<x86_emulate.patch>

Quoted 50 lines · Anthony Liguori

Quoted 27 lines · Alexander Graf

Quoted 24 lines · Anthony Liguori

Quoted 10 lines · Alexander Graf

Hi,

this is an improved version of the patch I sent several weeks ago to this list. Functionally nothing changed; it still hacks into gfxboot and
patches it to work on Intel CPUs on the fly. The big difference is that
this version is cleaned up and should work with every future CPU available.

Quoted 8 lines

Please do _not_ apply this patch. I send it to the list only for interested people, who would like to have a working version of KVM for
their systems right now. It is neither a proper fix nor the right approach to deal with this issue. It is merely a hack that works for me
and maybe for others too.

Perhaps a viable way to fix this upstream would be to catch the vmentry failure, look to see if SS.CPL != CS.CPL, and if so, invoke x86_emulate() in a loop until SS.CPL == CS.CPL.

Quoted 7 lines

Quoted 2 lines

There are very few instructions in gfxboot that would need to be added to x86_emulate (if they aren't already there).

In a previous thread Avi already explained a quite reasonable way to approach this problem, which I believe is a really good approach. He wanted to x86_emulate until the environment is "VMX friendly" again, thus resolving big real mode problems as well.

I've got a slightly lamer approach than what Avi probably wants. I lost interest in updating x86_emulate once I realized how far xen's copy has gotten. To get GFXBOOT 3.3.28 working just requires adding far jmp to x86_emulate. The sequence should look like:

      jmp pm_seg.prog_c32:switch_to_pm_20

switch_to_pm_20:

      bits 32

      mov ax,pm_seg.prog_d16
      mov ds,ax
      mov eax,ss

Quoted 4 lines

Which means we'll get 3 vmentry failures. The two moves should already be supported by x86_emulate but I haven't confirmed. It's not a complete solution to our real mode woes but I think it's a reasonable first step.

Right, this was exactly the approach I wanted to go at first. I just backed off it as I saw how much ljmp actually does, as normal x86 CPUs switch to PM not on cr0 setting, but on the ljmp after that.

So is that simple implementation you have written actually doing the right thing?

Regards,

Alex

Quoted 14 lines

Regards,

Anthony Liguori

Quoted 7 lines

I personally agree that the real approach is way superior to my patch. I just won't have the time to do it in the near future and not being able to boot intuitively hurts KVM users unnecessarily ;-).

Regards,

Alex

<x86_emulate.patch>

Yes, but it won't compile as KVM does not have a proper load_seg() function like the Xen's x86_emulate.

Perhaps a viable way to fix this upstream would be to catch the vmentry failure, look to see if SS.CPL != CS.CPL, and if so, invoke x86_emulate() in a loop until SS.CPL == CS.CPL.

Quoted 4 lines · On Mon, 07 Apr 2008 11:05:06 -0500 Anthony Liguori <[email protected]>…

Perhaps a viable way to fix this upstream would be to catch the vmentry failure, look to see if SS.CPL != CS.CPL, and if so, invoke x86_emulate() in a loop until SS.CPL == CS.CPL.

I tried this solution some time ago but unfortunately x86_emulate() failed. I suspected a problem with guest EIP that could different between the vmentry catch and the emulation. I will rebase my patch and post them on the mailing list.

Regards,
Guillaume

Quoted 9 lines · Guillaume Thouvenin

Quoted 4 lines · On Mon, 07 Apr 2008 11:05:06 -0500 Anthony Liguori <[email protected]>…

Perhaps a viable way to fix this upstream would be to catch the vmentry failure, look to see if SS.CPL != CS.CPL, and if so, invoke x86_emulate() in a loop until SS.CPL == CS.CPL.

Quoted 5 lines

I tried this solution some time ago but unfortunately x86_emulate() failed. I suspected a problem with guest EIP that could different between the vmentry catch and the emulation. I will rebase my patch and post them on the mailing list.

x86 emulate is missing support for jmp far which is used to switch into protected mode. It just needs to be added.

Quoted 5 lines

Quoted 2 lines

x86 emulate is missing support for jmp far which is used to switch into protected mode. It just needs to be added.

Ok I see. I understand now why you said in a previous email that KVM needs to have a proper load_seg() function like the Xen's x86_emulate. This function is used to load the segment in a far jmp. I will look how it is done in Xen and I will try to copy the stuff like you did.

Perhaps a viable way to fix this upstream would be to catch the vmentry failure, look to see if SS.CPL != CS.CPL, and if so, invoke x86_emulate() in a loop until SS.CPL == CS.CPL.

There are very few instructions in gfxboot that would need to be added to x86_emulate (if they aren't already there).

Quoted 7 lines · On Mon, 07 Apr 2008 11:05:06 -0500 Anthony Liguori <[email protected]>…

Perhaps a viable way to fix this upstream would be to catch the vmentry failure, look to see if SS.CPL != CS.CPL, and if so, invoke x86_emulate() in a loop until SS.CPL == CS.CPL.

There are very few instructions in gfxboot that would need to be added to x86_emulate (if they aren't already there).

So to see if I'm on the good way here is an attempt to implement the solution. It doesn't work yet.

I'm trying to:
- Disable the code that modifies SS value in order to detect VM entry
failure
- Add the handler that catches the VM entry failure
- Emulate the instruction until we recover a friendly VMX state

     => add the jmp far (opcode 0xea) instruction in the emulation.

With the patch, the VM entry failure is caught but the jmp far instruction seems to fail. I've got the following dmesg:

...
handle_vmentry_failure: invalid guest state
handle_vmentry_failure: start emulation
handle_vmentry_failure: emulation failed

...

Regards,
Guillaume

---

arch/x86/kvm/vmx.c +0 -0

index 8e5d664..a56bd83 100644

arch/x86/kvm/vmx.c +32 -0

+#if 0
 	vmcs_write16(GUEST_SS_SELECTOR, 0);
 	vmcs_write32(GUEST_SS_AR_BYTES, 0x93);
+#endif
 
 	vmcs_write16(GUEST_CS_SELECTOR,
 		     vmcs_read16(GUEST_CS_SELECTOR) & ~SELECTOR_RPL_MASK);

@@ -1952,6 +1955,33 @@ static int handle_rmode_exception(struct kvm_vcpu *vcpu,
 	return 0;
 }
 
+static int handle_vmentry_failure(u32 exit_reason, struct kvm_vcpu *vcpu)
+{
+	unsigned int basic_exit_reason = (uint16_t) exit_reason;
+	unsigned int cs_rpl = vmcs_read16(GUEST_CS_SELECTOR) & SELECTOR_RPL_MASK;
+	unsigned int ss_rpl = vmcs_read16(GUEST_SS_SELECTOR) & SELECTOR_RPL_MASK;
+
+	switch (basic_exit_reason) {
+		case EXIT_REASON_INVALID_GUEST_STATE:
+			printk(KERN_INFO "%s: invalid guest state\n", __FUNCTION__);
+			printk(KERN_INFO "%s: start emulation \n", __FUNCTION__);
+			while (cs_rpl != ss_rpl) {
+				if (emulate_instruction(vcpu, NULL, 0, 0, 0) == EMULATE_FAIL) {
+					printk(KERN_INFO "%s: emulation failed\n", __FUNCTION__);
+					return 0;
+				}
+				cs_rpl = vmcs_read16(GUEST_CS_SELECTOR) & SELECTOR_RPL_MASK;
+				ss_rpl = vmcs_read16(GUEST_SS_SELECTOR) & SELECTOR_RPL_MASK;
+			}
+			printk(KERN_INFO "%s: VMX friendly state recovered\n", __FUNCTION__);
+			break;
+		default:
+			printk(KERN_INFO "VM-entry failure due to unkown reason\n");
+			return 0;
+	}
+	return 1;
+}
+
 static int handle_exception(struct kvm_vcpu *vcpu, struct kvm_run *kvm_run)
 {
 	struct vcpu_vmx *vmx = to_vmx(vcpu);

@@ -2364,6 +2394,9 @@ static int kvm_handle_exit(struct kvm_run *kvm_run, struct kvm_vcpu *vcpu)
 	KVMTRACE_3D(VMEXIT, vcpu, exit_reason, (u32)vmcs_readl(GUEST_RIP),
 		    (u32)((u64)vmcs_readl(GUEST_RIP) >> 32), entryexit);
 
+	if (unlikely(exit_reason & VMX_EXIT_REASONS_FAILED_VMENTRY))
+		return handle_vmentry_failure(exit_reason, vcpu);
+
 	if (unlikely(vmx->fail)) {
 		kvm_run->exit_reason = KVM_EXIT_FAIL_ENTRY;
 		kvm_run->fail_entry.hardware_entry_failure_reason

arch/x86/kvm/vmx.h +3 -0

index 5dff460..200c0f8 100644
@@ -223,7 +223,10 @@ enum vmcs_field {
 #define EXIT_REASON_IO_INSTRUCTION      30
 #define EXIT_REASON_MSR_READ            31
 #define EXIT_REASON_MSR_WRITE           32
+#define EXIT_REASON_INVALID_GUEST_STATE 33
+#define EXIT_REASON_MSR_LOADING         34
 #define EXIT_REASON_MWAIT_INSTRUCTION   36
+#define EXIT_REASON_MACHINE_CHECK       41
 #define EXIT_REASON_TPR_BELOW_THRESHOLD 43
 #define EXIT_REASON_APIC_ACCESS         44
 #define EXIT_REASON_WBINVD		54

arch/x86/kvm/x86.c +1 -1

index 0ce5563..b38065d 100644
@@ -3267,7 +3267,7 @@ static int load_segment_descriptor_to_kvm_desct(struct kvm_vcpu *vcpu,
 	return 0;
 }
 
-static int load_segment_descriptor(struct kvm_vcpu *vcpu, u16 selector,
+int load_segment_descriptor(struct kvm_vcpu *vcpu, u16 selector,
 				   int type_bits, int seg)
 {
 	struct kvm_segment kvm_seg;

arch/x86/kvm/x86_emulate.c +11 -1

index 2ca0838..4bc3c80 100644
@@ -168,7 +168,7 @@ static u16 opcode_table[256] = {
 	/* 0xE0 - 0xE7 */
 	0, 0, 0, 0, 0, 0, 0, 0,
 	/* 0xE8 - 0xEF */
-	ImplicitOps | Stack, SrcImm|ImplicitOps, 0, SrcImmByte|ImplicitOps,
+	ImplicitOps | Stack, SrcImm|ImplicitOps, ImplicitOps, SrcImmByte|ImplicitOps,
 	0, 0, 0, 0,
 	/* 0xF0 - 0xF7 */
 	0, 0, 0, 0,

@@ -1156,6 +1156,9 @@ static inline int emulate_grp45(struct x86_emulate_ctxt *ctxt,
 	case 4: /* jmp abs */
 		c->eip = c->src.val;
 		break;
+	case 5: /* jmp far */
+		printk(KERN_INFO "Jmp far need to be implemented\n");
+		break;
 	case 6:	/* push */
 		emulate_push(ctxt);
 		break;

@@ -1657,6 +1660,24 @@ special_insn:
 		break;
 	}
 	case 0xe9: /* jmp rel */
+		jmp_rel(c, c->src.val);
+		c->dst.type = OP_NONE; /* Disable writeback. */
+		break;
+	case 0xea: /* jmp far */ {
+		struct kvm_segment kvm_seg;
+		int ret;
+

Quoted 5 lines

...
handle_vmentry_failure: invalid guest state
handle_vmentry_failure: start emulation
handle_vmentry_failure: emulation failed

What instruction failed, exactly?

You need to load rip as well.

I added the code do dump the instruction and it seems that it's the emulation of 0xe6 (== out imm8, al) that failed. I made modifications to emulate it (see below) and now I have another problem in kvm userspace with the following message (and the emulation doesn't work):

enterprise:~ $ kvm_run: Operation not permitted
enterprise:~ $ kvm_run returned -1

Quoted 9 lines · On Tue, 15 Apr 2008 16:06:43 +0300 Avi Kivity <[email protected]> wrote:

Quoted 6 lines

...
handle_vmentry_failure: invalid guest state
handle_vmentry_failure: start emulation
handle_vmentry_failure: emulation failed

What instruction failed, exactly?

I added the code do dump the instruction and it seems that it's the emulation of 0xe6 (== out imm8, al) that failed. I made modifications to emulate it (see below) and now I have another problem in kvm userspace with the following message (and the emulation doesn't work):

enterprise:~ $ kvm_run: Operation not permitted
enterprise:~ $ kvm_run returned -1

Quoted 2 lines

You need to load rip as well.

Ooops, yes. So jump far emulation is now like:

(quoted diff) +28 -0

+       case 0xea: /* jmp far */ {
+               struct kvm_segment kvm_seg;
+               long int eip;
+               int ret;
+
+               kvm_x86_ops->get_segment(ctxt->vcpu, &kvm_seg, VCPU_SREG_CS); 
+
+               ret = load_segment_descriptor(ctxt->vcpu, kvm_seg.selector, 9, VCPU_SREG_CS);
+               if (ret < 0){
+                       printk(KERN_INFO "%s: Failed to load CS descriptor\n", __FUNCTION__);
+                       goto cannot_emulate;
+               }
+
+               switch (c->op_bytes) {
+               case 2:
+                       eip = insn_fetch(s16, 2, c->eip);
+                       break;
+               case 4:
+                       eip = insn_fetch(s32, 4, c->eip);
+                       break;
+               default:
+                       DPRINTF("jmp far: Invalid op_bytes\n");
+                       goto cannot_emulate;
+               }
+               printk(KERN_INFO "eip == 0x%lx\n", eip);
+               c->eip = eip;
+               break;
+       }

It seems that the jump to cs:eip works and now I have the following error:

[18535.446917] handle_vmentry_failure: invalid guest state
[18535.449519] handle_vmentry_failure: start emulation
[18535.457519] eip == 0x6e18
[18535.467685] handle_vmentry_failure: emulation of 0xe6 failed

For the emulation of 0xe6 I used the following one that I found in nitin's tree:

(quoted diff) +10 -0

+       case 0xe6: /* out imm8, al */
+       case 0xe7: /* out imm8, ax/eax */ {
+               struct kvm_io_device *pio_dev;
+               
+               pio_dev = vcpu_find_pio_dev(ctxt->vcpu, c->src.val);
+               kvm_iodevice_write(pio_dev, c->src.val,
+                               (c->d & ByteOp) ? 1 : c->op_bytes,
+                               &c->regs[VCPU_REGS_RAX]);
+               }
+               break;

I will look closer where is the problem and as you suggested, I will display the instruction to be emulated and the register state before and after, and compare with the expected state.

Thanks for your help,
Regards,
Guillaume

-------------------------------------------------------------------------
This SF.net email is sponsored by the 2008 JavaOne(SM) Conference Don't miss this year's exciting event. There's still time to save $100. Use priority code J8TL2D2.
http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone
_______________________________________________
kvm-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/kvm-devel

This doesn't seem right. You should have been able to break out of the emulator long before encountering an out instruction. The next instruction you encounter should be a mov instruction. Are you sure you're updating eip correctly?

Quoted 5 lines · On Fri, 18 Apr 2008 08:23:07 -0500 Anthony Liguori <[email protected]>…

This doesn't seem right. You should have been able to break out of the emulator long before encountering an out instruction. The next instruction you encounter should be a mov instruction. Are you sure you're updating eip correctly?

I think that eip is updated correctly but you're right, I think that the condition to stop emulation is not well implemented. I emulate a lot of mov instructions and I remain blocked in the emulation loop until I reach the "out" instruction. The loop is the following:

  [...]
  cs_rpl = vmcs_read16(GUEST_CS_SELECTOR) & SELECTOR_RPL_MASK;
  ss_rpl = vmcs_read16(GUEST_SS_SELECTOR) & SELECTOR_RPL_MASK;

  while (cs_rpl != ss_rpl) {
      if (emulate_instruction(vcpu, NULL, 0,0, 0) == EMULATE_FAIL) {
          printk(KERN_INFO "%s: emulation of 0x%x failed\n",
                           __FUNCTION__,
                           vcpu->arch.emulate_ctxt.decode.b);
          return -1;
       }
       cs_rpl = vmcs_read16(GUEST_CS_SELECTOR) & SELECTOR_RPL_MASK;
       ss_rpl = vmcs_read16(GUEST_SS_SELECTOR) & SELECTOR_RPL_MASK;
  }
  printk(KERN_INFO "%s: VMX friendly state recovered\n", __FUNCTION__);
  // I never reach this point

Maybe CS and SS selector are not well updated. I will add trace to see their values before and after the emulation.

Regards,
Guillaume

I'd prefer you not do an emulate_instruction loop at all. Just emulate one instruction on vmentry failure and let VT tell you what instructions you need to emulate.

It's only four instructions so I don't think the performance is going to matter. Take a look at the patch I posted previously.

I'd prefer you not do an emulate_instruction loop at all. Just emulate one instruction on vmentry failure and let VT tell you what instructions you need to emulate.

It's only four instructions so I don't think the performance is going to matter. Take a look at the patch I posted previously.