vmx_handle_exit: unexpected, valid vectoring info (0x80000b0e)

3.5.0 BUG Fengguang Wu 3 participants 8 messages Sep 12, 2012 – Oct 17, 2012 KVM

lore

Share & tools

lore

Fengguang Wu

author

2012-09-12 04:40 UTC #0 lore raw

Hi,

3 of my test boxes running v3.5 kernel become unaccessible and I find two of them kept emitting this dmesg:

vmx_handle_exit: unexpected, valid vectoring info (0x80000b0e) and exit reason is 0x31

The other one has froze and the above lines are the last dmesg. Any ideas?

Thanks,
Fengguang

Third, it's really unexpected. It seems the guest was attempting to deliver a page fault exception (0x0e) but encountered an mmio page during delivery (in the IDT, TSS, stack, or page tables). Is this reproducible? If so it's easy to patch kvm to halt in that case and allow examining the guest via qemu.

Maybe we should do so regardless (return a KVM_EXIT_INTERNAL_ERROR).

Signature · 1 line

error compiling committee.c: too many arguments to function

Fengguang Wu

author

2012-09-12 08:25 UTC #2 lore raw

Quoted 23 lines · Avi Kivity

Quoted 9 lines · Fengguang Wu
Hi,
3 of my test boxes running v3.5 kernel become unaccessible and I find two of them kept emitting this dmesg:
vmx_handle_exit: unexpected, valid vectoring info (0x80000b0e) and exit reason is 0x31
The other one has froze and the above lines are the last dmesg. Any ideas?
First, that printk should be rate-limited.
Second, we should add EXIT_REASON_EPT_MISCONFIG (0x31) to
	if ((vectoring_info & VECTORING_INFO_VALID_MASK) &&
			(exit_reason != EXIT_REASON_EXCEPTION_NMI &&
			exit_reason != EXIT_REASON_EPT_VIOLATION &&
			exit_reason != EXIT_REASON_TASK_SWITCH))
		printk(KERN_WARNING "%s: unexpected, valid vectoring info "
		       "(0x%x) and exit reason is 0x%x\n",
		       __func__, vectoring_info, exit_reason);

Quoted 3 lines

since it's easily caused by the guest.
Third, it's really unexpected. It seems the guest was attempting to deliver a page fault exception (0x0e) but encountered an mmio page during delivery (in the IDT, TSS, stack, or page tables). Is this reproducible? If so it's easy to patch kvm to halt in that case and allow examining the guest via qemu.

It's the first time I see such errors. For now I've upgraded the host kernel to 3.6-rc5. Let's check whether it will happen again.

Quoted 1 line

Maybe we should do so regardless (return a KVM_EXIT_INTERNAL_ERROR).

I can test your changes either way.

Thanks,
Fengguang

Xiao Guangrong

2012-09-14 05:57 UTC #3 lore raw

In reply to Avi Kivity

Quoted 23 lines · Avi Kivity

Quoted 9 lines · Fengguang Wu
Hi,
3 of my test boxes running v3.5 kernel become unaccessible and I find two of them kept emitting this dmesg:
vmx_handle_exit: unexpected, valid vectoring info (0x80000b0e) and exit reason is 0x31
The other one has froze and the above lines are the last dmesg. Any ideas?
First, that printk should be rate-limited.
Second, we should add EXIT_REASON_EPT_MISCONFIG (0x31) to
	if ((vectoring_info & VECTORING_INFO_VALID_MASK) &&
			(exit_reason != EXIT_REASON_EXCEPTION_NMI &&
			exit_reason != EXIT_REASON_EPT_VIOLATION &&
			exit_reason != EXIT_REASON_TASK_SWITCH))
		printk(KERN_WARNING "%s: unexpected, valid vectoring info "
		       "(0x%x) and exit reason is 0x%x\n",
		       __func__, vectoring_info, exit_reason);

Quoted 1 line

since it's easily caused by the guest.

Yes, i will do these.

Quoted 3 lines

Third, it's really unexpected. It seems the guest was attempting to deliver a page fault exception (0x0e) but encountered an mmio page during delivery (in the IDT, TSS, stack, or page tables). Is this reproducible? If so it's easy to patch kvm to halt in that case and allow examining the guest via qemu.

Have no idea yet why the box was frozen under this case, will try to write a test case, hope it can help me to find the reason out.

Quoted 1 line

Maybe we should do so regardless (return a KVM_EXIT_INTERNAL_ERROR).

I think this is reasonable.

Thanks!

Xiao Guangrong

2012-10-17 06:26 UTC #4 lore raw

Quoted 26 lines · Xiao Guangrong

Quoted 23 lines · Avi Kivity
Quoted 9 lines · Fengguang Wu
Hi,
3 of my test boxes running v3.5 kernel become unaccessible and I find two of them kept emitting this dmesg:
vmx_handle_exit: unexpected, valid vectoring info (0x80000b0e) and exit reason is 0x31
The other one has froze and the above lines are the last dmesg. Any ideas?
First, that printk should be rate-limited.
Second, we should add EXIT_REASON_EPT_MISCONFIG (0x31) to
	if ((vectoring_info & VECTORING_INFO_VALID_MASK) &&
			(exit_reason != EXIT_REASON_EXCEPTION_NMI &&
			exit_reason != EXIT_REASON_EPT_VIOLATION &&
			exit_reason != EXIT_REASON_TASK_SWITCH))
		printk(KERN_WARNING "%s: unexpected, valid vectoring info "
		       "(0x%x) and exit reason is 0x%x\n",
		       __func__, vectoring_info, exit_reason);
Quoted 1 line
since it's easily caused by the guest.

Quoted 9 lines

Yes, i will do these.
Quoted 3 lines
Third, it's really unexpected. It seems the guest was attempting to deliver a page fault exception (0x0e) but encountered an mmio page during delivery (in the IDT, TSS, stack, or page tables). Is this reproducible? If so it's easy to patch kvm to halt in that case and allow examining the guest via qemu.
Have no idea yet why the box was frozen under this case, will try to write a test case, hope it can help me to find the reason out.

Still did not know why linux kernel triggered it. I have posted a patchset to report an internal error for this case, hoping Fengguang can reproduce it after the patchset and Qemu's dump can help us to find the reason out.

I will keep working on it.

Fengguang Wu

author

2012-10-17 06:43 UTC #5 lore raw

Quoted 37 lines · Xiao Guangrong

Quoted 26 lines · Xiao Guangrong
Quoted 23 lines · Avi Kivity
Quoted 9 lines · Fengguang Wu
Hi,
3 of my test boxes running v3.5 kernel become unaccessible and I find two of them kept emitting this dmesg:
vmx_handle_exit: unexpected, valid vectoring info (0x80000b0e) and exit reason is 0x31
The other one has froze and the above lines are the last dmesg. Any ideas?
First, that printk should be rate-limited.
Second, we should add EXIT_REASON_EPT_MISCONFIG (0x31) to
	if ((vectoring_info & VECTORING_INFO_VALID_MASK) &&
			(exit_reason != EXIT_REASON_EXCEPTION_NMI &&
			exit_reason != EXIT_REASON_EPT_VIOLATION &&
			exit_reason != EXIT_REASON_TASK_SWITCH))
		printk(KERN_WARNING "%s: unexpected, valid vectoring info "
		       "(0x%x) and exit reason is 0x%x\n",
		       __func__, vectoring_info, exit_reason);
Quoted 1 line
since it's easily caused by the guest.
Quoted 9 lines
Yes, i will do these.
Quoted 3 lines
Third, it's really unexpected. It seems the guest was attempting to deliver a page fault exception (0x0e) but encountered an mmio page during delivery (in the IDT, TSS, stack, or page tables). Is this reproducible? If so it's easy to patch kvm to halt in that case and allow examining the guest via qemu.
Have no idea yet why the box was frozen under this case, will try to write a test case, hope it can help me to find the reason out.

Quoted 6 lines

Still did not know why linux kernel triggered it. I have posted a patchset to report an internal error for this case, hoping Fengguang can reproduce it after the patchset and Qemu's dump can help us to find the reason out.
I will keep working on it.

Thanks! Shall I run some patched kernel, or just 3.6.0?

Another problem I sometimes run into is, dmesg no longer works in the test boxes that run lots of KVMs. It aborts with an error message:

dmesg: klogctl failed: Bad address

Thanks,
Fengguang

Xiao Guangrong

2012-10-17 07:04 UTC #6 lore raw

Quoted 38 lines · Fengguang Wu

Quoted 37 lines · Xiao Guangrong
Quoted 35 lines · Xiao Guangrong
Quoted 23 lines · Avi Kivity
Quoted 9 lines · Fengguang Wu
Hi,
3 of my test boxes running v3.5 kernel become unaccessible and I find two of them kept emitting this dmesg:
vmx_handle_exit: unexpected, valid vectoring info (0x80000b0e) and exit reason is 0x31
The other one has froze and the above lines are the last dmesg. Any ideas?
First, that printk should be rate-limited.
Second, we should add EXIT_REASON_EPT_MISCONFIG (0x31) to
	if ((vectoring_info & VECTORING_INFO_VALID_MASK) &&
			(exit_reason != EXIT_REASON_EXCEPTION_NMI &&
			exit_reason != EXIT_REASON_EPT_VIOLATION &&
			exit_reason != EXIT_REASON_TASK_SWITCH))
		printk(KERN_WARNING "%s: unexpected, valid vectoring info "
		       "(0x%x) and exit reason is 0x%x\n",
		       __func__, vectoring_info, exit_reason);
Quoted 1 line
since it's easily caused by the guest.
Yes, i will do these.
Quoted 3 lines
Third, it's really unexpected. It seems the guest was attempting to deliver a page fault exception (0x0e) but encountered an mmio page during delivery (in the IDT, TSS, stack, or page tables). Is this reproducible? If so it's easy to patch kvm to halt in that case and allow examining the guest via qemu.
Have no idea yet why the box was frozen under this case, will try to write a test case, hope it can help me to find the reason out.

Quoted 8 lines

Quoted 6 lines
Still did not know why linux kernel triggered it. I have posted a patchset to report an internal error for this case, hoping Fengguang can reproduce it after the patchset and Qemu's dump can help us to find the reason out.
I will keep working on it.
Thanks! Shall I run some patched kernel, or just 3.6.0?

The patchset is under review. Can be found at:
https://lkml.org/lkml/2012/10/17/31

Quoted 5 lines

Another problem I sometimes run into is, dmesg no longer works in the test boxes that run lots of KVMs. It aborts with an error message:
dmesg: klogctl failed: Bad address

Interesting, will fight for it. :)

Fengguang Wu

author

2012-10-17 09:15 UTC #7 lore raw

Quoted 44 lines · Xiao Guangrong

Quoted 43 lines · Fengguang Wu
Quoted 42 lines · Xiao Guangrong
Quoted 26 lines · Xiao Guangrong
Quoted 23 lines · Avi Kivity
Quoted 9 lines · Fengguang Wu
Hi,
3 of my test boxes running v3.5 kernel become unaccessible and I find two of them kept emitting this dmesg:
vmx_handle_exit: unexpected, valid vectoring info (0x80000b0e) and exit reason is 0x31
The other one has froze and the above lines are the last dmesg. Any ideas?
First, that printk should be rate-limited.
Second, we should add EXIT_REASON_EPT_MISCONFIG (0x31) to
	if ((vectoring_info & VECTORING_INFO_VALID_MASK) &&
			(exit_reason != EXIT_REASON_EXCEPTION_NMI &&
			exit_reason != EXIT_REASON_EPT_VIOLATION &&
			exit_reason != EXIT_REASON_TASK_SWITCH))
		printk(KERN_WARNING "%s: unexpected, valid vectoring info "
		       "(0x%x) and exit reason is 0x%x\n",
		       __func__, vectoring_info, exit_reason);
Quoted 1 line
since it's easily caused by the guest.
Quoted 9 lines
Yes, i will do these.
Quoted 3 lines
Third, it's really unexpected. It seems the guest was attempting to deliver a page fault exception (0x0e) but encountered an mmio page during delivery (in the IDT, TSS, stack, or page tables). Is this reproducible? If so it's easy to patch kvm to halt in that case and allow examining the guest via qemu.
Have no idea yet why the box was frozen under this case, will try to write a test case, hope it can help me to find the reason out.
Still did not know why linux kernel triggered it. I have posted a patchset to report an internal error for this case, hoping Fengguang can reproduce it after the patchset and Qemu's dump can help us to find the reason out.

Quoted 6 lines

Quoted 3 lines
Quoted 1 line
I will keep working on it.
Thanks! Shall I run some patched kernel, or just 3.6.0?
The patchset is under review. Can be found at:
https://lkml.org/lkml/2012/10/17/31

Thanks, I'll try it.

Thanks,
Fengguang