Kernel GPF in vmx_save_host_state()

I accidentally tried to run a 64-bit guest on a 32-bit host. Even though this isn't supported, it shouldn't crash my kernel. :-)

CPU: Intel Core 2 Quad
KVM: kernel kvm-84-640-g967f619, userspace kvm-84-304-g2ced1d8. Also occurs with vanilla kvm-84. Host kernel: Vanilla 2.6.28
Host arch: i386
Guest: Debian Etch x86_64, distributor 2.6.18 kernel
Commandline: qemu-system-x86_64 -hda debian-etch-x86_64 -m 512
Occurs with: -no-kvm-irqchip, -no-kvm-pit
Does not occur with: -no-kvm

As soon as the guest kernel starts, the host produces:

general protection fault: 0000 [#1] PREEMPT SMP 

last sysfs file: /sys/kernel/uevent_seqnum

Dumping ftrace buffer:

(ftrace buffer empty)

Modules linked in: kvm_intel kvm 

Pid: 6570, comm: qemu-system-x86 Not tainted (2.6.28-686 #4) Precision WorkStat
EIP: 0060:[<f8cf3d97>] EFLAGS: 00010246 CPU: 2

EIP is at vmx_save_host_state+0x193/0x1a6 [kvm_intel]

EAX: 00000100 EBX: 00000000 ECX: c0000080 EDX: 00000000

ESI: c0000080 EDI: 00000000 EBP: f462ae80 ESP: f462ae58

DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068

Process qemu-system-x86 (pid: 6570, ti=f462a000 task=f43fe460 task.ti=f462a000)

Stack: 

c0406379 f7c5467d 00000100 00000000 ee020020 00000000 00000000
ee020020
00000001 00000000 f462aeb0 f7c58b4d f47b3000 ee020020 00000000
c0406469
c0403ede 7ffbfeff fffffffe 0000ae80 f43a8730 00000000 f462af18
f7c5467d

Call Trace: 

[<c0406379>] ? _spin_unlock+0x2c/0x41

[<f7c5467d>] ? kvm_vcpu_ioctl+0xf4/0x40f [kvm]

[<f7c58b4d>] ? kvm_arch_vcpu_ioctl_run+0x444/0x918 [kvm]

[<c0406469>] ? _spin_unlock_irqrestore+0x59/0x5d

[<c0403ede>] ? preempt_schedule+0x30/0x3f

[<f7c5467d>] ? kvm_vcpu_ioctl+0xf4/0x40f [kvm]

[<c013ae80>] ? up_read+0x1b/0x2f

[<c0148144>] ? futex_wake+0xd0/0xdb

[<c0148e4c>] ? do_futex+0x81/0x6c9

[<f7c54589>] ? kvm_vcpu_ioctl+0x0/0x40f [kvm]

[<c018e345>] ? vfs_ioctl+0x27/0x6c

[<c018e7ec>] ? do_vfs_ioctl+0x394/0x3d8

[<c0184c88>] ? fget_light+0xc8/0xe4

[<c018e84c>] ? sys_ioctl+0x1c/0x5f

[<c011e06f>] ? sub_preempt_count+0x9d/0xab

[<c018e875>] ? sys_ioctl+0x45/0x5f

[<c0102e25>] ? sysenter_do_call+0x12/0x35

Code: ec 81 e1 01 08 00 00 31 db 89 f2 09 ca 89 55 e0 89 f8 09 d8 89 45 e4 be 8
EIP: [<f8cf3d97>] vmx_save_host_state+0x193/0x1a6 [kvm_intel] SS:ESP 0068:f462a8
---[ end trace b07f1e77e8b208d3 ]--- 

--Benjamin Gilbert

From ECX, it looks like it's trying to access EFER and enable LM.

Please provide your /proc/cpuinfo.

And this line, in full.

error compiling committee.c: too many arguments to function

processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 15
model name : Intel(R) Core(TM)2 Quad CPU @ 2.66GHz
stepping : 7
cpu MHz : 2660.345
cache size : 4096 KB
physical id : 0
siblings : 4
core id : 0
cpu cores : 4
apicid : 0
initial apicid : 0
fdiv_bug : no
hlt_bug : no
f00f_bug : no
coma_bug : no
fpu : yes
fpu_exception : yes
cpuid level : 10
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe nx lm constant_tsc arch_perfmon pebs bts pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm lahf_lm tpr_shadow
bogomips : 5320.69
clflush size : 64
power management:

processor : 1
vendor_id : GenuineIntel
cpu family : 6
model : 15
model name : Intel(R) Core(TM)2 Quad CPU @ 2.66GHz
stepping : 7
cpu MHz : 2660.345
cache size : 4096 KB
physical id : 0
siblings : 4
core id : 1
cpu cores : 4
apicid : 1
initial apicid : 1
fdiv_bug : no
hlt_bug : no
f00f_bug : no
coma_bug : no
fpu : yes
fpu_exception : yes
cpuid level : 10
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe nx lm constant_tsc arch_perfmon pebs bts pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm lahf_lm tpr_shadow
bogomips : 5320.23
clflush size : 64
power management:

processor : 2
vendor_id : GenuineIntel
cpu family : 6
model : 15
model name : Intel(R) Core(TM)2 Quad CPU @ 2.66GHz
stepping : 7
cpu MHz : 2660.345
cache size : 4096 KB
physical id : 0
siblings : 4
core id : 2
cpu cores : 4
apicid : 2
initial apicid : 2
fdiv_bug : no
hlt_bug : no
f00f_bug : no
coma_bug : no
fpu : yes
fpu_exception : yes
cpuid level : 10
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe nx lm constant_tsc arch_perfmon pebs bts pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm lahf_lm tpr_shadow
bogomips : 5320.28
clflush size : 64
power management:

processor : 3
vendor_id : GenuineIntel
cpu family : 6
model : 15
model name : Intel(R) Core(TM)2 Quad CPU @ 2.66GHz
stepping : 7
cpu MHz : 2660.345
cache size : 4096 KB
physical id : 0
siblings : 4
core id : 3
cpu cores : 4
apicid : 3
initial apicid : 3
fdiv_bug : no
hlt_bug : no
f00f_bug : no
coma_bug : no
fpu : yes
fpu_exception : yes
cpuid level : 10
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe nx lm constant_tsc arch_perfmon pebs bts pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm lahf_lm tpr_shadow
bogomips : 5320.24
clflush size : 64
power management:

Whoops.

Code: ec 81 e1 01 08 00 00 31 db 89 f2 09 ca 89 55 e0 89 f8 09 d8 89 45 e4 be 80 00 00 c0 8b 55 e0 8b 4d e4 89 ca 31 c9 89 f1 8b 45 e0 <0f> 30 8b 5d e8 ff 83 ec 00 00 00 83 c4 1c 5b 5e 5f 5d c3 55 89

--Benjamin Gilbert

Well, that's certainly the wrmsr instruction. But I don't see how this can happen.

Can you patch set_efer() in x86.c to print the value of the efer argument and of efer_reserved_bits?

error compiling committee.c: too many arguments to function

Yes, but apparently set_efer() is never called. To verify, I patched kvm_set_msr_common() to print the msr parameter:

kvm_set_msr_common: 0xc0010117
kvm_set_msr_common: 0x250
kvm_set_msr_common: 0x258
kvm_set_msr_common: 0x259
kvm_set_msr_common: 0x268
kvm_set_msr_common: 0x269
kvm_set_msr_common: 0x26a
kvm_set_msr_common: 0x26b
kvm_set_msr_common: 0x26c
kvm_set_msr_common: 0x26d
kvm_set_msr_common: 0x26e
kvm_set_msr_common: 0x26f
kvm_set_msr_common: 0x200
kvm_set_msr_common: 0x201
kvm_set_msr_common: 0x2ff

general protection fault: 0000 [#1] PREEMPT SMP

[...]

--Benjamin Gilbert

On 32-bit, we might actually reach the default: label of the switch in vmx_set_msr(). Can you add a printk() there? print both msr_index, and, if msr is not NULL, msr->index and msr->data.

error compiling committee.c: too many arguments to function

Sure:

vmx_set_msr: msr_index 0xc0000081 msr->index 0xc0000081 msr->data 0x0
vmx_set_msr: msr_index 0xc0010117
vmx_set_msr: msr_index 0x250
vmx_set_msr: msr_index 0x258
vmx_set_msr: msr_index 0x259
vmx_set_msr: msr_index 0x268
vmx_set_msr: msr_index 0x269
vmx_set_msr: msr_index 0x26a
vmx_set_msr: msr_index 0x26b
vmx_set_msr: msr_index 0x26c
vmx_set_msr: msr_index 0x26d
vmx_set_msr: msr_index 0x26e
vmx_set_msr: msr_index 0x26f
vmx_set_msr: msr_index 0x200
vmx_set_msr: msr_index 0x201
vmx_set_msr: msr_index 0x2ff
vmx_set_msr: msr_index 0xc0000080 msr->index 0xc0000080 msr->data 0x100

general protection fault: 0000 [#1] PREEMPT SMP

[...]

The printk is after msr->data is set to the value of the data parameter.

--Benjamin Gilbert

How did that get in there?!

Please add a dump_stack() after that printk().

I have a truly marvellous patch that fixes the bug which this
signature is too narrow to contain.

Pid: 2381, comm: qemu-system-x86 Not tainted 2.6.28-686 #4

Call Trace:
 [<f8cf2fdc>] vmx_set_msr+0x150/0x178 [kvm_intel]
 [<f8cf325a>] handle_wrmsr+0x71/0x9d [kvm_intel]
 [<f8cf4fb0>] kvm_handle_exit+0x1c8/0x1e5 [kvm_intel]
 [<f7c58e34>] kvm_arch_vcpu_ioctl_run+0x6f2/0x918 [kvm]
 [<c0406469>] ? _spin_unlock_irqrestore+0x59/0x5d
 [<c0403ede>] ? preempt_schedule+0x30/0x3f
 [<f7c5467d>] kvm_vcpu_ioctl+0xf4/0x40f [kvm]
 [<c013ae80>] ? up_read+0x1b/0x2f
 [<c0148144>] ? futex_wake+0xd0/0xdb
 [<c0148e4c>] ? do_futex+0x81/0x6c9
 [<f7c54589>] ? kvm_vcpu_ioctl+0x0/0x40f [kvm]
 [<c018e345>] vfs_ioctl+0x27/0x6c
 [<c018e7ec>] do_vfs_ioctl+0x394/0x3d8
 [<c0184c88>] ? fget_light+0xc8/0xe4
 [<c018e84c>] ? sys_ioctl+0x1c/0x5f
 [<c011e06f>] ? sub_preempt_count+0x9d/0xab
 [<c018e875>] sys_ioctl+0x45/0x5f
 [<c0102e25>] sysenter_do_call+0x12/0x35

--Benjamin Gilbert

Duh, I noted this hole in a previous email.

Attached patch should fix.

I have a truly marvellous patch that fixes the bug which this
signature is too narrow to contain.

index 88ef094..da6461d 100644
@@ -942,11 +942,11 @@ static int vmx_set_msr(struct kvm_vcpu *vcpu, u32 msr_index, u64 data)
 	int ret = 0;
 
 	switch (msr_index) {
-#ifdef CONFIG_X86_64
 	case MSR_EFER:
 		vmx_load_host_state(vmx);
 		ret = kvm_set_msr_common(vcpu, msr_index, data);
 		break;
+#ifdef CONFIG_X86_64
 	case MSR_FS_BASE:
 		vmcs_writel(GUEST_FS_BASE, data);
 		break;

It does, thanks.

--Benjamin Gilbert

Glauber had a patch that fixes this; it needs a refresh now though:

http://marc.info/?l=qemu-devel&m=123324996718919&w=2

Amit

That won't fix the kernel crash.

error compiling committee.c: too many arguments to function

Right.

What action should the kernel take, though? If lm is advertised and we don't let the guest enter long mode -- exit to userspace and close the VM?

Since it should never happen, it doesn't really matter. #GP will be the natural action (setting reserved bit); userspace lied to the guest, and we can't recover from that.

error compiling committee.c: too many arguments to function