Etherboot problems

DISCUSSION Simon Kagstrom 3 participants 9 messages Nov 9, 2006 – Nov 12, 2006 7 messages not in the archive KVM

Share & tools

Simon Kagstrom

author

2006-11-09 16:05 UTC #0 lore raw

Hello!

I'm trying to get etherboot running with KVM (version 2 from today), but have been unsucessful so far. It works well in plain qemu, but not with KVM.

I'm running etherboot 5.4.2, ROM rtl8139:rtl8139 [0x10ec,0x8139] built as a "ISO bootable image without legacy floppy emulation (.iso)" on
http://rom-o-matic.net/5.4.2/. When I run it I get

   $ qemu-system-x86_64 -cdrom ~/eb-rtl8139.iso
   ...
   unhandled vm exit:  0x2
   rax 0000000060000011 rbx 0000000000000e51 rcx 0000000000000000 rdx 0000000000000000
   rsi 0000000000106347 rdi 0000000000016347 rsp 0000000000008fbe rbp 0000000000098fbe
   r8  0000000000000000 r9  0000000000000000 r10 0000000000000000 r11 0000000000000000
   r12 0000000000000000 r13 0000000000000000 r14 0000000000000000 r15 0000000000000000
   rip 0000000000000e8a rflags 00010006
   Aborted

. From browsing the Xen sources, I think the VM exit 0x2 is a triple-fault. I've disassembled the .iso with hte, and the code that runs last is

...

   00000e75 660fb7ec                       movzx       ebp, sp
   00000e79 6601c5                         add         ebp, eax            
   00000e7c 660f01975200                   lgdt        single ptr [bx+0x52]
   00000e82 0f20c0                         mov         eax, cr0
   00000e85 0c01                           or          al, 0x1
   00000e87 0f22c0                         mov         cr0, eax
   00000e8a 66ffaf6a00                     jmp         far ptr [bx+0x6a]

...

I'm not an expert on Intel-syntax 16-bit assembly, but I guess the last instruction is something like

jmp *0x6a(%bx)

. One thing which I'm sure of is if the vm exit was caused by the move to %cr0 or the jump.

Any takes? I can supply the .iso if someone would like to play with it

// Simon

-------------------------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642

A message that is not in the archive (sent off-list or not yet received): 874pt8ip14.wl%[email protected]

Avi Kivity

2006-11-09 16:17 UTC #2 lore raw

Quoted 37 lines · Simon Kagstrom

Hello!
I'm trying to get etherboot running with KVM (version 2 from today), but have been unsucessful so far. It works well in plain qemu, but not with KVM.
I'm running etherboot 5.4.2, ROM rtl8139:rtl8139 [0x10ec,0x8139] built as a "ISO bootable image without legacy floppy emulation (.iso)" on
http://rom-o-matic.net/5.4.2/. When I run it I get
   $ qemu-system-x86_64 -cdrom ~/eb-rtl8139.iso
   ...
   unhandled vm exit:  0x2
   rax 0000000060000011 rbx 0000000000000e51 rcx 0000000000000000 rdx 0000000000000000
   rsi 0000000000106347 rdi 0000000000016347 rsp 0000000000008fbe rbp 0000000000098fbe
   r8  0000000000000000 r9  0000000000000000 r10 0000000000000000 r11 0000000000000000
   r12 0000000000000000 r13 0000000000000000 r14 0000000000000000 r15 0000000000000000
   rip 0000000000000e8a rflags 00010006
   Aborted
. From browsing the Xen sources, I think the VM exit 0x2 is a triple-fault. I've disassembled the .iso with hte, and the code that runs last is
   ...
   00000e75 660fb7ec                       movzx       ebp, sp
   00000e79 6601c5                         add         ebp, eax            
   00000e7c 660f01975200                   lgdt        single ptr [bx+0x52]
   00000e82 0f20c0                         mov         eax, cr0
   00000e85 0c01                           or          al, 0x1
   00000e87 0f22c0                         mov         cr0, eax
   00000e8a 66ffaf6a00                     jmp         far ptr [bx+0x6a]
   ...
I'm not an expert on Intel-syntax 16-bit assembly, but I guess the last instruction is something like

Quoted 7 lines

   jmp  *0x6a(%bx)
. One thing which I'm sure of is if the vm exit was caused by the move to %cr0 or the jump.
Any takes? I can supply the .iso if someone would like to play with it

I've generated the .iso (nice interface) and will try to reproduce it.

Thanks for testing -- we really need those wierd guests to test the corner cases.

Signature · 8 lines

error compiling committee.c: too many arguments to function


-------------------------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642

A message that is not in the archive (sent off-list or not yet received): 455354AB.8000908-atKUWr5tajBWk0Htik3J/[email protected]

Avi Kivity

2006-11-09 17:49 UTC #4 lore raw

Quoted 18 lines · Avi Kivity

Quoted 18 lines

   00000e75 660fb7ec                       movzx       ebp, sp
   00000e79 6601c5                         add         ebp, 
eax               00000e7c 660f01975200                   lgdt        
single ptr [bx+0x52]
   00000e82 0f20c0                         mov         eax, cr0
   00000e85 0c01                           or          al, 0x1
   00000e87 0f22c0                         mov         cr0, eax
   00000e8a 66ffaf6a00                     jmp         far ptr [bx+0x6a]

...

I'm not an expert on Intel-syntax 16-bit assembly, but I guess the last instruction is something like

   jmp  *0x6a(%bx)

. One thing which I'm sure of is if the vm exit was caused by the move to %cr0 or the jump.

Quoted 8 lines

Quoted 2 lines
Any takes? I can supply the .iso if someone would like to play with it
I've generated the .iso (nice interface) and will try to reproduce it.
Thanks for testing -- we really need those wierd guests to test the corner cases.

Very likely this is due to VT's inability to virtualize real mode correctly. Is anyone in the mood to extend x86_emulate.c to support real mode?

Signature · 8 lines

error compiling committee.c: too many arguments to function


-------------------------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642

A message that is not in the archive (sent off-list or not yet received): 45536A11.6050808-atKUWr5tajBWk0Htik3J/[email protected]

Simon Kagstrom

author

2006-11-09 18:11 UTC #6 lore raw

At Thu, 09 Nov 2006 19:49:05 +0200,

Quoted 10 lines · Avi Kivity

Quoted 8 lines · Avi Kivity

Quoted 8 lines

   00000e75 660fb7ec                       movzx       ebp, sp
   00000e79 6601c5                         add         ebp,  eax              
   00000e7c 660f01975200                   lgdt        single ptr [bx+0x52]
   00000e82 0f20c0                         mov         eax, cr0
   00000e85 0c01                           or          al, 0x1
   00000e87 0f22c0                         mov         cr0, eax
   00000e8a 66ffaf6a00                     jmp         far ptr [bx+0x6a]

...

Quoted 8 lines

Quoted 4 lines
I've generated the .iso (nice interface) and will try to reproduce it.
Thanks for testing -- we really need those wierd guests to test the corner cases.
Very likely this is due to VT's inability to virtualize real mode correctly. Is anyone in the mood to extend x86_emulate.c to support real mode?

I forgot to add that I've tested it in Xen as well, and it works there. Maybe someone has an AMD at hand to test it on?

But I also suspected that this was the reason why it failed. I ran a diff between x86_emulate.c in Xen and KVM but couldn't immediately see any differences which would explain this.

One thing though: when KVM says

Quoted 6 lines

   unhandled vm exit:  0x2
   rax 0000000060000011 rbx 0000000000000e51 rcx 0000000000000000 rdx 0000000000000000
   rsi 0000000000106347 rdi 0000000000016347 rsp 0000000000008fbe rbp 0000000000098fbe
   r8  0000000000000000 r9  0000000000000000 r10 0000000000000000 r11 0000000000000000
   r12 0000000000000000 r13 0000000000000000 r14 0000000000000000 r15 0000000000000000
   rip 0000000000000e8a rflags 00010006

, I suppose this mean that the "jmp far ptr ..." instruction couldn't be executed (maybe because of the GDT setup before).

I've tried to debug the program with gdb (qemu -s -S), but I cannot examine the code from gdb (either "raw" or KVM). Well, well.

// Simon

A message that is not in the archive (sent off-list or not yet received): 873b8sij7k.wl%[email protected]

Anthony Liguori

2006-11-10 02:32 UTC #8 lore raw

Quoted 10 lines · Simon Kagstrom

Quoted 3 lines
Very likely this is due to VT's inability to virtualize real mode correctly. Is anyone in the mood to extend x86_emulate.c to support real mode?
I forgot to add that I've tested it in Xen as well, and it works there. Maybe someone has an AMD at hand to test it on?
But I also suspected that this was the reason why it failed. I ran a diff between x86_emulate.c in Xen and KVM but couldn't immediately see any differences which would explain this.

Xen doesn't use x86_emulate.c for real mode. It uses vmxassist which uses vm86 mode (yes, there are many many emulators in Xen).

Xen 3.0.4 is planning on dropping vmxassist in favor of CPU state hand off between QEMU and Xen. This would be much easier in KVM ATM as it doesn't require state synchronization for hardware emulation.

Regards,

Anthony Liguori

Simon Kagstrom

author

2006-11-10 15:35 UTC #9 lore raw

In reply to a message not in the archive

At Thu, 09 Nov 2006 19:49:05 +0200,

Quoted 3 lines · Avi Kivity

Very likely this is due to VT's inability to virtualize real mode correctly. Is anyone in the mood to extend x86_emulate.c to support real mode?

I found what caused the problems for the etherboot startup and it now works fine (well, I've not tested downloading anything over the network, but at least it gets that far). I've started grub as well, and that works as before even with this fix.

However, I'm not sure what the correct solution to the problem is.

The crash occurs after etherboot has switched to protected mode and does a long jump to the new code segment:

	/* Load GDT */

1: DATA32 lgdt %ds:(r2p_gdt-1b)(%bx) /* Load GDT */

	/* Switch to protected mode */

2: movl %cr0, %eax
3: orb $CR0_PE, %al
4: movl %eax, %cr0

	/* flush prefetch queue, and reload %cs:%eip */
   5:	DATA32 ljmp %ds:*(r2p_paddr-1b)(%bx)
	.code32

The problem is that KVM zeroes all segment registers directly on a switch to protected mode (i.e., on instruction 4 above) - and %ds being zero causes a crash on the ljmp above.

What I've done is to remove the zeroing of all segment registers except %ss, which is shown in the trivial patch below. I don't remember enough of transfer-to-protected-mode assembly to know the correct solution to this (the %ss zeroing needs to be kept for example, not sure why).

// Simon

kernel/kvm_main.c +0 -4

@@ -714,10 +714,6 @@
 	}
 
 	FIX_PMODE_DATASEG(SS, vcpu->rmode.ss);
-	FIX_PMODE_DATASEG(ES, vcpu->rmode.es);
-	FIX_PMODE_DATASEG(DS, vcpu->rmode.ds);
-	FIX_PMODE_DATASEG(GS, vcpu->rmode.gs);
-	FIX_PMODE_DATASEG(FS, vcpu->rmode.fs);
 
 	vmcs_write16(GUEST_CS_SELECTOR,
 		     vmcs_read16(GUEST_CS_SELECTOR) & ~SELECTOR_RPL_MASK);

_______________________________________________
kvm-devel mailing list [email protected]
https://lists.sourceforge.net/lists/listinfo/kvm-devel

A message that is not in the archive (sent off-list or not yet received): 87odrfgvqf.wl%[email protected]

Avi Kivity

2006-11-10 16:11 UTC #11 lore raw

Quoted 34 lines · Simon Kagstrom

At Thu, 09 Nov 2006 19:49:05 +0200,
Quoted 4 lines · Avi Kivity
Very likely this is due to VT's inability to virtualize real mode correctly. Is anyone in the mood to extend x86_emulate.c to support real mode?
I found what caused the problems for the etherboot startup and it now works fine (well, I've not tested downloading anything over the network, but at least it gets that far). I've started grub as well, and that works as before even with this fix.
However, I'm not sure what the correct solution to the problem is.
The crash occurs after etherboot has switched to protected mode and does a long jump to the new code segment:
	/* Load GDT */
1: DATA32 lgdt %ds:(r2p_gdt-1b)(%bx) /* Load GDT */
	/* Switch to protected mode */
2: movl %cr0, %eax
3: orb $CR0_PE, %al
4: movl %eax, %cr0
	/* flush prefetch queue, and reload %cs:%eip */
   5:	DATA32 ljmp %ds:*(r2p_paddr-1b)(%bx)
	.code32
The problem is that KVM zeroes all segment registers directly on a switch to protected mode (i.e., on instruction 4 above) - and %ds being zero causes a crash on the ljmp above.

Quoted 6 lines

What I've done is to remove the zeroing of all segment registers except %ss, which is shown in the trivial patch below. I don't remember enough of transfer-to-protected-mode assembly to know the correct solution to this (the %ss zeroing needs to be kept for example, not sure why).

VT does a lot of checks when entering guest mode. These checks are stricter than what the processor can handle, and stricter than what some software expects. The upshot is that virtualizing real mode and the transition between modes can't be made to work 100%.

The long term solution is to emulate rather than virtualize while in these states. Short term, I'll see if there's some workaround we can apply here (your patch probably makes other guests fail; I'll check this though).

Signature · 8 lines

Do not meddle in the internals of kernels, for they are subtle and quick to panic.


-------------------------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642

A message that is not in the archive (sent off-list or not yet received): 4554A4BF.5020409-atKUWr5tajBWk0Htik3J/[email protected]

Simon Kagstrom

author

2006-11-11 13:27 UTC #13 lore raw

At Fri, 10 Nov 2006 18:11:43 +0200,

Quoted 8 lines · Avi Kivity

Quoted 5 lines · Simon Kagstrom
What I've done is to remove the zeroing of all segment registers except %ss, which is shown in the trivial patch below. I don't remember enough of transfer-to-protected-mode assembly to know the correct solution to this (the %ss zeroing needs to be kept for example, not sure why).

Quoted 4 lines

VT does a lot of checks when entering guest mode. These checks are stricter than what the processor can handle, and stricter than what some software expects. The upshot is that virtualizing real mode and the transition between modes can't be made to work 100%.

OK. Just out of curiosity: What is the reason for the zeroing of the segment selectors? I've been looking a bit in the developers manual (volume 3a), and section 9.9.1 says

7. After entering protected mode, the segment registers continue to hold the contents they had in real-address mode. The JMP or CALL instruction in step 4 resets the CS register.

and thereafter the segment registers should be reloaded. It seems to me that KVM does this directly when the PE flag is set (and enter_pmode is called), which causes etherboot to not work.

Quoted 4 lines

The long term solution is to emulate rather than virtualize while in these states. Short term, I'll see if there's some workaround we can apply here (your patch probably makes other guests fail; I'll check this though).

I see. Grub and Linux starts with the patch, but I understand that it's not a real solution to the problem (but since I need etherboot, I'm attempted to continue using it for a while :-)).

// Simon

A message that is not in the archive (sent off-list or not yet received): 87r6wa9kr2.wl%[email protected]

Avi Kivity

2006-11-12 08:20 UTC #15 lore raw

Quoted 21 lines · Simon Kagstrom

At Fri, 10 Nov 2006 18:11:43 +0200,
Quoted 8 lines · Avi Kivity
Quoted 6 lines · Simon Kagstrom
What I've done is to remove the zeroing of all segment registers except %ss, which is shown in the trivial patch below. I don't remember enough of transfer-to-protected-mode assembly to know the correct solution to this (the %ss zeroing needs to be kept for example, not sure why).
Quoted 5 lines
VT does a lot of checks when entering guest mode. These checks are stricter than what the processor can handle, and stricter than what some software expects. The upshot is that virtualizing real mode and the transition between modes can't be made to work 100%.
OK. Just out of curiosity: What is the reason for the zeroing of the segment selectors? I've been looking a bit in the developers manual (volume 3a), and section 9.9.1 says

Quoted 9 lines

7. After entering protected mode, the segment registers continue to hold the contents they had in real-address mode. The JMP or CALL instruction in step 4 resets the CS register.
and thereafter the segment registers should be reloaded. It seems to me that KVM does this directly when the PE flag is set (and enter_pmode is called), which causes etherboot to not work.

The problem is 22.3.1.2, "Checks on Guest Segment Registers". When transitioning to _real_ mode, we have to hack the segment registers so that VT will accept them. When we switch back, the hacked segments are no longer suitable for protected mode.

The best way forward is likely to remember the segments during a real->protected or protected->real mode switch, and on the switch back, if a segment selector has not been changed, reload it from the saved state.

Long term, we need not to use VT for real mode at all.

Quoted 10 lines

Quoted 5 lines
The long term solution is to emulate rather than virtualize while in these states. Short term, I'll see if there's some workaround we can apply here (your patch probably makes other guests fail; I'll check this though).
I see. Grub and Linux starts with the patch, but I understand that it's not a real solution to the problem (but since I need etherboot, I'm attempted to continue using it for a while :-)).

Can you check Windows? I'd apply the patch if it doesn't break that.

Signature · 8 lines

error compiling committee.c: too many arguments to function


-------------------------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642