Linux in VirtualPC in KVM fails

Hi,

let's start with the scenario I tried to use: I have two levels of virtualization. On the physical hardware I run a Linux with KVM. The KVM guest is a Win2k3 VM which runs VirtualPC. In VirtualPC I try to run a Linux again (openSUSE 11.1 to be specific, but that shouldn't matter).

The boot menu comes up nicely and so on, but early in the kernel boot it crashes:

EIP is at kvm_deferred_mmu_op+0x46/0xbf

Call Trace:
 [<c0117f79>] kvm_mmu_write+0x59/0x61
 [<c011bad9>] set_pte_vaddr+0x95/0xec
 [<c011b3b2>] __native_set_fixmap+0x1d/0x24
 [<c054ae5b>] test_wp_bit+0x24/0x6c
 [<c054b6b1>] mem_init+0x295/0x2b8
 [<c053a8a3>] start_kernel+0x262/0x31f

Now obviously this is a KVM function where there should be none. The problem seems to be that VirtualPC doesn't intercept cpuid and thus the VirtualPC guest sees the KVM cpuid values where it better wouldn't. Consequently, it turns on the paravirt support for KVM which is exactly wrong and leads to the crash on the first hypercall.

The guest has no chance to detect correctly if it's running directly on KVM or if there is another virtualization layer which can't emulate cpuid. So the fix must involve the mechanism itself. Alex has suggested to change the interface to use a KVM-specific MSR instead of cpuid as these should be handled by any virtualization software. I'm copying him so he can take over for the details, I just want to get the discussion started.

So... Comments? Suggestions? Patches? ;-)

Quoted 33 lines · Kevin Wolf

Hi,

let's start with the scenario I tried to use: I have two levels of virtualization. On the physical hardware I run a Linux with KVM. The KVM guest is a Win2k3 VM which runs VirtualPC. In VirtualPC I try to run a Linux again (openSUSE 11.1 to be specific, but that shouldn't matter).

The boot menu comes up nicely and so on, but early in the kernel boot it crashes:

EIP is at kvm_deferred_mmu_op+0x46/0xbf

Call Trace:
[<c0117f79>] kvm_mmu_write+0x59/0x61
[<c011bad9>] set_pte_vaddr+0x95/0xec
[<c011b3b2>] __native_set_fixmap+0x1d/0x24
[<c054ae5b>] test_wp_bit+0x24/0x6c
[<c054b6b1>] mem_init+0x295/0x2b8
[<c053a8a3>] start_kernel+0x262/0x31f

Now obviously this is a KVM function where there should be none. The problem seems to be that VirtualPC doesn't intercept cpuid and thus the VirtualPC guest sees the KVM cpuid values where it better wouldn't. Consequently, it turns on the paravirt support for KVM which is exactly wrong and leads to the crash on the first hypercall.

The guest has no chance to detect correctly if it's running directly on KVM or if there is another virtualization layer which can't emulate cpuid. So the fix must involve the mechanism itself. Alex has suggested to change the interface to use a KVM-specific MSR instead of cpuid as these should be handled by any virtualization software. I'm copying him so he can take over for the details, I just want to get the discussion started.

Quoted 2 lines

So... Comments? Suggestions? Patches? ;-)

Gaa. Looks like cpuid is totally broken by first-generation virtualization products.

We're in a nice compatibility mess. We can't just switch paravirt detection methods since that will break older guests _and_ older hosts on non-nested virtualization (which is the common case).

So I suggest:
- add a new cpuid bit, for 'paravirt verification'
- if the new bit is present, the guest verifies that paravirt is truly enabled by reading an msr (and checking a signature).

Older guests or newer guests running on older hosts will still be broken, but there's a command line parameter for disabling paravirt on the guest, and we can add one for the host as well.

Signature · 2 lines

I have a truly marvellous patch that fixes the bug which this
signature is too narrow to contain.

Shouldn't it be ok to push patches to linux-stable to use the CPUID and MSR information and simply not expose the CPUID identification in newer KVM versions? That way older guests on newer KVM don't use KVM paravirt (which should still be ok), but everything else runs as smoothly as possible.

Alex

Quoted 11 lines

So I suggest:
- add a new cpuid bit, for 'paravirt verification'
- if the new bit is present, the guest verifies that paravirt is truly enabled by reading an msr (and checking a signature).

Older guests or newer guests running on older hosts will still be broken, but there's a command line parameter for disabling paravirt on the guest, and we can add one for the host as well.

Signature · 2 lines

I have a truly marvellous patch that fixes the bug which this
signature is too narrow to contain.

Signature · 3 lines

To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Quoted 2 lines · Alexander Graf

Shouldn't it be ok to push patches to linux-stable to use the CPUID and MSR information and simply not expose the CPUID identification in newer KVM versions? That way older guests on newer KVM don't use KVM paravirt (which should still be ok), but everything else runs as smoothly as possible.

I'd say that VirtualPC is broken and leave it at that.

All modern hypervisors use CPUID to expose themselves to the guests. VirtualPC is totally capable of rewriting CPUID instructions.

Shame on it for not doing so. I don't think we should change anything.

Quoted 7 lines · Anthony Liguori

Quoted 2 lines · Alexander Graf

Shouldn't it be ok to push patches to linux-stable to use the CPUID and MSR information and simply not expose the CPUID identification in newer KVM versions? That way older guests on newer KVM don't use KVM paravirt (which should still be ok), but everything else runs as smoothly as possible.

I'd say that VirtualPC is broken and leave it at that.

All modern hypervisors use CPUID to expose themselves to the guests. VirtualPC is totally capable of rewriting CPUID instructions.

Apparently it isn't. I know of one other fullvirt product that does not jit kernel code. Or maybe they wanted to preserve consistency between kernel cpuid and host cpuid.

Quoted 2 lines

Apparently it isn't. I know of one other fullvirt product that does not jit kernel code. Or maybe they wanted to preserve consistency between kernel cpuid and host cpuid.

How can you get away with not jitting kernel code?

Quoted 1 line

We're in a nice compatibility mess. We can't just switch paravirt detection methods since that will break older guests _and_ older hosts on non-nested virtualization (which is the common case).

Shouldn't it be ok to push patches to linux-stable to use the CPUID and MSR information and simply not expose the CPUID identification in newer KVM versions? That way older guests on newer KVM don't use KVM paravirt (which should still be ok), but everything else runs as smoothly as possible.

Quoted 33 lines · Kevin Wolf

Hi,

let's start with the scenario I tried to use: I have two levels of virtualization. On the physical hardware I run a Linux with KVM. The KVM guest is a Win2k3 VM which runs VirtualPC. In VirtualPC I try to run a Linux again (openSUSE 11.1 to be specific, but that shouldn't matter).

The boot menu comes up nicely and so on, but early in the kernel boot it crashes:

EIP is at kvm_deferred_mmu_op+0x46/0xbf

Call Trace:
 [<c0117f79>] kvm_mmu_write+0x59/0x61
 [<c011bad9>] set_pte_vaddr+0x95/0xec
 [<c011b3b2>] __native_set_fixmap+0x1d/0x24
 [<c054ae5b>] test_wp_bit+0x24/0x6c
 [<c054b6b1>] mem_init+0x295/0x2b8
 [<c053a8a3>] start_kernel+0x262/0x31f

Now obviously this is a KVM function where there should be none. The problem seems to be that VirtualPC doesn't intercept cpuid and thus the VirtualPC guest sees the KVM cpuid values where it better wouldn't. Consequently, it turns on the paravirt support for KVM which is exactly wrong and leads to the crash on the first hypercall.

The guest has no chance to detect correctly if it's running directly on KVM or if there is another virtualization layer which can't emulate cpuid. So the fix must involve the mechanism itself. Alex has suggested to change the interface to use a KVM-specific MSR instead of cpuid as these should be handled by any virtualization software. I'm copying him so he can take over for the details, I just want to get the discussion started.

Quoted 2 lines

So... Comments? Suggestions? Patches? ;-)

Gaa. Looks like cpuid is totally broken by first-generation virtualization products.

Quoted 34 lines · Avi Kivity

Quoted 33 lines · Kevin Wolf

Hi,

let's start with the scenario I tried to use: I have two levels of virtualization. On the physical hardware I run a Linux with KVM. The KVM guest is a Win2k3 VM which runs VirtualPC. In VirtualPC I try to run a Linux again (openSUSE 11.1 to be specific, but that shouldn't matter).

The boot menu comes up nicely and so on, but early in the kernel boot it crashes:

EIP is at kvm_deferred_mmu_op+0x46/0xbf

Call Trace:
[<c0117f79>] kvm_mmu_write+0x59/0x61
[<c011bad9>] set_pte_vaddr+0x95/0xec
[<c011b3b2>] __native_set_fixmap+0x1d/0x24
[<c054ae5b>] test_wp_bit+0x24/0x6c
[<c054b6b1>] mem_init+0x295/0x2b8
[<c053a8a3>] start_kernel+0x262/0x31f

Now obviously this is a KVM function where there should be none. The problem seems to be that VirtualPC doesn't intercept cpuid and thus the VirtualPC guest sees the KVM cpuid values where it better wouldn't. Consequently, it turns on the paravirt support for KVM which is exactly wrong and leads to the crash on the first hypercall.

The guest has no chance to detect correctly if it's running directly on KVM or if there is another virtualization layer which can't emulate cpuid. So the fix must involve the mechanism itself. Alex has suggested to change the interface to use a KVM-specific MSR instead of cpuid as these should be handled by any virtualization software. I'm copying him so he can take over for the details, I just want to get the discussion started.

Quoted 4 lines

Quoted 2 lines

So... Comments? Suggestions? Patches? ;-)

Gaa. Looks like cpuid is totally broken by first-generation virtualization products.

Not at all. There's no reason that a JIT'ing virtualization product can't rewrite CPUID to a function call and then mask off unsupported bits. It's a bug in the virtualization product if it doesn't do this.

Isn't one of the great things about virtualization the fact that you can do things you can on real hardware in the virtual machine? While I'm not exactly a fan of VirtualPC, I would still like it to work in KVM, as that's what real hardware is capable of.

Alex

Quoted 2 lines · Alexander Graf

Isn't one of the great things about virtualization the fact that you can do things you can on real hardware in the virtual machine? While I'm not exactly a fan of VirtualPC, I would still like it to work in KVM, as that's what real hardware is capable of.

Paravirtualization support is advertised in userspace, just add a flag to QEMU to disable KVM paravirt support.