Skip to contentSkip to conversation

kvm: WARNING in vmx_handle_exit

PATCH In review Dmitry Vyukov 2 participants 5 messages Jan 2, 2017 – Mar 14, 2017

Subsystems (3)
x86 KVM KVM x86
lore
Share & tools
lore

Dmitry Vyukov

author
#0 lore raw

Hello,

I've got the following warning while running syzkaller fuzzer:

WARNING: CPU: 2 PID: 13257 at arch/x86/kvm/vmx.c:8633
vmx_handle_exit+0x262b/0x38b0 arch/x86/kvm/vmx.c:8633
vmx: unexpected exit reason 0xb
CPU: 2 PID: 13257 Comm: syz-executor7 Not tainted 4.10.0-rc1+ #118
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011
Call Trace:
 __dump_stack lib/dump_stack.c:15 [inline]
 dump_stack+0x292/0x3a2 lib/dump_stack.c:51
 panic+0x1cb/0x3a9 kernel/panic.c:179
 __warn+0x1c4/0x1e0 kernel/panic.c:539
 warn_slowpath_fmt+0xc5/0x100 kernel/panic.c:562
 vmx_handle_exit+0x262b/0x38b0 arch/x86/kvm/vmx.c:8633
 vcpu_enter_guest arch/x86/kvm/x86.c:6884 [inline]
 vcpu_run arch/x86/kvm/x86.c:6943 [inline]
 kvm_arch_vcpu_ioctl_run+0xf3d/0x45f0 arch/x86/kvm/x86.c:7101
 kvm_vcpu_ioctl+0x673/0x1120 arch/x86/kvm/../../../virt/kvm/kvm_main.c:2569
 vfs_ioctl fs/ioctl.c:43 [inline]
 do_vfs_ioctl+0x1bf/0x1780 fs/ioctl.c:683
 SYSC_ioctl fs/ioctl.c:698 [inline]
 SyS_ioctl+0x8f/0xc0 fs/ioctl.c:689
 entry_SYSCALL_64_fastpath+0x1f/0xc2
RIP: 0033:0x4438a9
RSP: 002b:00007f935aa2fb58 EFLAGS: 00000286 ORIG_RAX: 0000000000000010
RAX: ffffffffffffffda RBX: 0000000000000015 RCX: 00000000004438a9
RDX: 0000000000000000 RSI: 000000000000ae80 RDI: 0000000000000015
RBP: 00000000006ddb30 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000286 R12: 0000000000700000
R13: 0000000000000006 R14: 0000000000000015 R15: 0000000020014000

Dumping ftrace buffer:
(ftrace buffer empty)
Kernel Offset: disabled

Exit reason 0xb is GETSEC instruction. This does not look harmful as it is handled as #UD. But I think we should print a single line message regarding non-emulated instruction as in other cases, just to not scare cloud admins and to make syzkaller ignore it.

FTR, a raw reproducer is here:
https://gist.githubusercontent.com/dvyukov/c762f6ea04ebbba49cdee0a6caca31b4/raw/f21deb04cdc70ae74100c12447d71bb0cd2025c7/gistfile1.txt

#1 lore raw

PATCH 1/1 KVM: VMX: downgrade warning on unexpected exit code

2017-01-02 11:23+0100, Dmitry Vyukov:

Quoted 36 lines

Hello,

I've got the following warning while running syzkaller fuzzer:

WARNING: CPU: 2 PID: 13257 at arch/x86/kvm/vmx.c:8633
vmx_handle_exit+0x262b/0x38b0 arch/x86/kvm/vmx.c:8633
vmx: unexpected exit reason 0xb
CPU: 2 PID: 13257 Comm: syz-executor7 Not tainted 4.10.0-rc1+ #118
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011
Call Trace:
 __dump_stack lib/dump_stack.c:15 [inline]
 dump_stack+0x292/0x3a2 lib/dump_stack.c:51
 panic+0x1cb/0x3a9 kernel/panic.c:179
 __warn+0x1c4/0x1e0 kernel/panic.c:539
 warn_slowpath_fmt+0xc5/0x100 kernel/panic.c:562
 vmx_handle_exit+0x262b/0x38b0 arch/x86/kvm/vmx.c:8633
 vcpu_enter_guest arch/x86/kvm/x86.c:6884 [inline]
 vcpu_run arch/x86/kvm/x86.c:6943 [inline]
 kvm_arch_vcpu_ioctl_run+0xf3d/0x45f0 arch/x86/kvm/x86.c:7101
 kvm_vcpu_ioctl+0x673/0x1120 arch/x86/kvm/../../../virt/kvm/kvm_main.c:2569
 vfs_ioctl fs/ioctl.c:43 [inline]
 do_vfs_ioctl+0x1bf/0x1780 fs/ioctl.c:683
 SYSC_ioctl fs/ioctl.c:698 [inline]
 SyS_ioctl+0x8f/0xc0 fs/ioctl.c:689
 entry_SYSCALL_64_fastpath+0x1f/0xc2
RIP: 0033:0x4438a9
RSP: 002b:00007f935aa2fb58 EFLAGS: 00000286 ORIG_RAX: 0000000000000010
RAX: ffffffffffffffda RBX: 0000000000000015 RCX: 00000000004438a9
RDX: 0000000000000000 RSI: 000000000000ae80 RDI: 0000000000000015
RBP: 00000000006ddb30 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000286 R12: 0000000000700000
R13: 0000000000000006 R14: 0000000000000015 R15: 0000000020014000

Dumping ftrace buffer:
(ftrace buffer empty)
Kernel Offset: disabled

Quoted 7 lines

Exit reason 0xb is GETSEC instruction. This does not look harmful as it is handled as #UD. But I think we should print a single line message regarding non-emulated instruction as in other cases, just to not scare cloud admins and to make syzkaller ignore it.

FTR, a raw reproducer is here:
https://gist.githubusercontent.com/dvyukov/c762f6ea04ebbba49cdee0a6caca31b4/raw/f21deb04cdc70ae74100c12447d71bb0cd2025c7/gistfile1.txt

I can't reproduce -- maybe a nested bug, which will take a while to figure out. Still, host dump at that point is useless, so the change makes sense.

A guest dump would be useful (e.g. I think that GETSEC should not exit if guest CR4.SMX is disabled), but just giving the error is about as good and we want something short and rate-limited if the message can be trigerred by a guest in production ...

---8<---
We never needed the call trace and we better rate-limit if it can be triggered by a guest.

Signed-off-by Radim Krčmář <[email protected]>
---
 arch/x86/kvm/vmx.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
arch/x86/kvm/vmx.c +2 -1
index a236decb81e4..7cd606daa01e 100644
@@ -8595,7 +8595,8 @@ static int vmx_handle_exit(struct kvm_vcpu *vcpu)
 	    && kvm_vmx_exit_handlers[exit_reason])
 		return kvm_vmx_exit_handlers[exit_reason](vcpu);
 	else {
-		WARN_ONCE(1, "vmx: unexpected exit reason 0x%x\n", exit_reason);
+		vcpu_unimpl(vcpu, "vmx: unexpected exit reason 0x%x\n",
+				exit_reason);
 		kvm_queue_exception(vcpu, UD_VECTOR);
 		return 1;
 	}
Signature · 1 line
2.11.0
author
#2 lore raw

Re: [PATCH] KVM: VMX: downgrade warning on unexpected exit code

Quoted 42 lines · Radim Krčmář

2017-01-02 11:23+0100, Dmitry Vyukov:

Quoted 41 lines

Hello,

I've got the following warning while running syzkaller fuzzer:

WARNING: CPU: 2 PID: 13257 at arch/x86/kvm/vmx.c:8633
vmx_handle_exit+0x262b/0x38b0 arch/x86/kvm/vmx.c:8633
vmx: unexpected exit reason 0xb
CPU: 2 PID: 13257 Comm: syz-executor7 Not tainted 4.10.0-rc1+ #118
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011
Call Trace:
 __dump_stack lib/dump_stack.c:15 [inline]
 dump_stack+0x292/0x3a2 lib/dump_stack.c:51
 panic+0x1cb/0x3a9 kernel/panic.c:179
 __warn+0x1c4/0x1e0 kernel/panic.c:539
 warn_slowpath_fmt+0xc5/0x100 kernel/panic.c:562
 vmx_handle_exit+0x262b/0x38b0 arch/x86/kvm/vmx.c:8633
 vcpu_enter_guest arch/x86/kvm/x86.c:6884 [inline]
 vcpu_run arch/x86/kvm/x86.c:6943 [inline]
 kvm_arch_vcpu_ioctl_run+0xf3d/0x45f0 arch/x86/kvm/x86.c:7101
 kvm_vcpu_ioctl+0x673/0x1120 arch/x86/kvm/../../../virt/kvm/kvm_main.c:2569
 vfs_ioctl fs/ioctl.c:43 [inline]
 do_vfs_ioctl+0x1bf/0x1780 fs/ioctl.c:683
 SYSC_ioctl fs/ioctl.c:698 [inline]
 SyS_ioctl+0x8f/0xc0 fs/ioctl.c:689
 entry_SYSCALL_64_fastpath+0x1f/0xc2
RIP: 0033:0x4438a9
RSP: 002b:00007f935aa2fb58 EFLAGS: 00000286 ORIG_RAX: 0000000000000010
RAX: ffffffffffffffda RBX: 0000000000000015 RCX: 00000000004438a9
RDX: 0000000000000000 RSI: 000000000000ae80 RDI: 0000000000000015
RBP: 00000000006ddb30 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000286 R12: 0000000000700000
R13: 0000000000000006 R14: 0000000000000015 R15: 0000000020014000

Dumping ftrace buffer:
(ftrace buffer empty)
Kernel Offset: disabled

Exit reason 0xb is GETSEC instruction. This does not look harmful as it is handled as #UD. But I think we should print a single line message regarding non-emulated instruction as in other cases, just to not scare cloud admins and to make syzkaller ignore it.

Quoted 8 lines
Quoted 2 lines

FTR, a raw reproducer is here:
https://gist.githubusercontent.com/dvyukov/c762f6ea04ebbba49cdee0a6caca31b4/raw/f21deb04cdc70ae74100c12447d71bb0cd2025c7/gistfile1.txt

I can't reproduce -- maybe a nested bug, which will take a while to figure out. Still, host dump at that point is useless, so the change makes sense.

A guest dump would be useful (e.g. I think that GETSEC should not exit

Thanks!

I had to disable dump_vmcs locally because it constantly produces tons of output even when no bugs happen.
If you really-really need it, I can enable dump_vmcs and re-reproduce the bug.

Quoted 29 lines

if guest CR4.SMX is disabled), but just giving the error is about as good and we want something short and rate-limited if the message can be trigerred by a guest in production ...
---8<---
We never needed the call trace and we better rate-limit if it can be triggered by a guest.

Signed-off-by Radim Krčmář <[email protected]>
---
 arch/x86/kvm/vmx.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
arch/x86/kvm/vmx.c +2 -2
index a236decb81e4..7cd606daa01e 100644
@@ -8595,7 +8595,8 @@ static int vmx_handle_exit(struct kvm_vcpu *vcpu)
            && kvm_vmx_exit_handlers[exit_reason])
                return kvm_vmx_exit_handlers[exit_reason](vcpu);
        else {
-               WARN_ONCE(1, "vmx: unexpected exit reason 0x%x\n", exit_reason);
+               vcpu_unimpl(vcpu, "vmx: unexpected exit reason 0x%x\n",
+                               exit_reason);
                kvm_queue_exception(vcpu, UD_VECTOR);
                return 1;
        }
--

2.11.0

author
#3 lore raw

Re: [PATCH] KVM: VMX: downgrade warning on unexpected exit code

Quoted 79 lines · Dmitry Vyukov
Quoted 42 lines

2017-01-02 11:23+0100, Dmitry Vyukov:

Quoted 41 lines

Hello,

I've got the following warning while running syzkaller fuzzer:

WARNING: CPU: 2 PID: 13257 at arch/x86/kvm/vmx.c:8633
vmx_handle_exit+0x262b/0x38b0 arch/x86/kvm/vmx.c:8633
vmx: unexpected exit reason 0xb
CPU: 2 PID: 13257 Comm: syz-executor7 Not tainted 4.10.0-rc1+ #118
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011
Call Trace:
 __dump_stack lib/dump_stack.c:15 [inline]
 dump_stack+0x292/0x3a2 lib/dump_stack.c:51
 panic+0x1cb/0x3a9 kernel/panic.c:179
 __warn+0x1c4/0x1e0 kernel/panic.c:539
 warn_slowpath_fmt+0xc5/0x100 kernel/panic.c:562
 vmx_handle_exit+0x262b/0x38b0 arch/x86/kvm/vmx.c:8633
 vcpu_enter_guest arch/x86/kvm/x86.c:6884 [inline]
 vcpu_run arch/x86/kvm/x86.c:6943 [inline]
 kvm_arch_vcpu_ioctl_run+0xf3d/0x45f0 arch/x86/kvm/x86.c:7101
 kvm_vcpu_ioctl+0x673/0x1120 arch/x86/kvm/../../../virt/kvm/kvm_main.c:2569
 vfs_ioctl fs/ioctl.c:43 [inline]
 do_vfs_ioctl+0x1bf/0x1780 fs/ioctl.c:683
 SYSC_ioctl fs/ioctl.c:698 [inline]
 SyS_ioctl+0x8f/0xc0 fs/ioctl.c:689
 entry_SYSCALL_64_fastpath+0x1f/0xc2
RIP: 0033:0x4438a9
RSP: 002b:00007f935aa2fb58 EFLAGS: 00000286 ORIG_RAX: 0000000000000010
RAX: ffffffffffffffda RBX: 0000000000000015 RCX: 00000000004438a9
RDX: 0000000000000000 RSI: 000000000000ae80 RDI: 0000000000000015
RBP: 00000000006ddb30 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000286 R12: 0000000000700000
R13: 0000000000000006 R14: 0000000000000015 R15: 0000000020014000

Dumping ftrace buffer:
(ftrace buffer empty)
Kernel Offset: disabled

Exit reason 0xb is GETSEC instruction. This does not look harmful as it is handled as #UD. But I think we should print a single line message regarding non-emulated instruction as in other cases, just to not scare cloud admins and to make syzkaller ignore it.

Quoted 8 lines
Quoted 2 lines

FTR, a raw reproducer is here:
https://gist.githubusercontent.com/dvyukov/c762f6ea04ebbba49cdee0a6caca31b4/raw/f21deb04cdc70ae74100c12447d71bb0cd2025c7/gistfile1.txt

I can't reproduce -- maybe a nested bug, which will take a while to figure out. Still, host dump at that point is useless, so the change makes sense.

A guest dump would be useful (e.g. I think that GETSEC should not exit

Thanks!

I had to disable dump_vmcs locally because it constantly produces tons of output even when no bugs happen.
If you really-really need it, I can enable dump_vmcs and re-reproduce the bug.

Quoted 21 lines

if guest CR4.SMX is disabled), but just giving the error is about as good and we want something short and rate-limited if the message can be trigerred by a guest in production ...
---8<---
We never needed the call trace and we better rate-limit if it can be triggered by a guest.

Signed-off-by Radim Krčmář <[email protected]>
---
 arch/x86/kvm/vmx.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
arch/x86/kvm/vmx.c +0 -1
index a236decb81e4..7cd606daa01e 100644
@@ -8595,7 +8595,8 @@ static int vmx_handle_exit(struct kvm_vcpu *vcpu)
            && kvm_vmx_exit_handlers[exit_reason])
                return kvm_vmx_exit_handlers[exit_reason](vcpu);
        else {
-               WARN_ONCE(1, "vmx: unexpected exit reason 0x%x\n", exit_reason);
Quoted 8 lines
Quoted 8 lines
arch/x86/kvm/vmx.c +2 -1
+               vcpu_unimpl(vcpu, "vmx: unexpected exit reason 0x%x\n",
+                               exit_reason);
                kvm_queue_exception(vcpu, UD_VECTOR);
                return 1;
        }
--

2.11.0

Was this ever merged? Still happens for me.

#4 lore raw

Re: [PATCH] KVM: VMX: downgrade warning on unexpected exit code

2017-03-10 19:56+0100, Dmitry Vyukov:

Quoted 82 lines · Dmitry Vyukov
Quoted 81 lines
Quoted 42 lines

2017-01-02 11:23+0100, Dmitry Vyukov:

Quoted 41 lines

Hello,

I've got the following warning while running syzkaller fuzzer:

WARNING: CPU: 2 PID: 13257 at arch/x86/kvm/vmx.c:8633
vmx_handle_exit+0x262b/0x38b0 arch/x86/kvm/vmx.c:8633
vmx: unexpected exit reason 0xb
CPU: 2 PID: 13257 Comm: syz-executor7 Not tainted 4.10.0-rc1+ #118
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011
Call Trace:
 __dump_stack lib/dump_stack.c:15 [inline]
 dump_stack+0x292/0x3a2 lib/dump_stack.c:51
 panic+0x1cb/0x3a9 kernel/panic.c:179
 __warn+0x1c4/0x1e0 kernel/panic.c:539
 warn_slowpath_fmt+0xc5/0x100 kernel/panic.c:562
 vmx_handle_exit+0x262b/0x38b0 arch/x86/kvm/vmx.c:8633
 vcpu_enter_guest arch/x86/kvm/x86.c:6884 [inline]
 vcpu_run arch/x86/kvm/x86.c:6943 [inline]
 kvm_arch_vcpu_ioctl_run+0xf3d/0x45f0 arch/x86/kvm/x86.c:7101
 kvm_vcpu_ioctl+0x673/0x1120 arch/x86/kvm/../../../virt/kvm/kvm_main.c:2569
 vfs_ioctl fs/ioctl.c:43 [inline]
 do_vfs_ioctl+0x1bf/0x1780 fs/ioctl.c:683
 SYSC_ioctl fs/ioctl.c:698 [inline]
 SyS_ioctl+0x8f/0xc0 fs/ioctl.c:689
 entry_SYSCALL_64_fastpath+0x1f/0xc2
RIP: 0033:0x4438a9
RSP: 002b:00007f935aa2fb58 EFLAGS: 00000286 ORIG_RAX: 0000000000000010
RAX: ffffffffffffffda RBX: 0000000000000015 RCX: 00000000004438a9
RDX: 0000000000000000 RSI: 000000000000ae80 RDI: 0000000000000015
RBP: 00000000006ddb30 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000286 R12: 0000000000700000
R13: 0000000000000006 R14: 0000000000000015 R15: 0000000020014000

Dumping ftrace buffer:
(ftrace buffer empty)
Kernel Offset: disabled

Exit reason 0xb is GETSEC instruction. This does not look harmful as it is handled as #UD. But I think we should print a single line message regarding non-emulated instruction as in other cases, just to not scare cloud admins and to make syzkaller ignore it.

Quoted 8 lines
Quoted 2 lines

FTR, a raw reproducer is here:
https://gist.githubusercontent.com/dvyukov/c762f6ea04ebbba49cdee0a6caca31b4/raw/f21deb04cdc70ae74100c12447d71bb0cd2025c7/gistfile1.txt

I can't reproduce -- maybe a nested bug, which will take a while to figure out. Still, host dump at that point is useless, so the change makes sense.

A guest dump would be useful (e.g. I think that GETSEC should not exit

Thanks!

I had to disable dump_vmcs locally because it constantly produces tons of output even when no bugs happen.
If you really-really need it, I can enable dump_vmcs and re-reproduce the bug.

Quoted 23 lines

if guest CR4.SMX is disabled), but just giving the error is about as good and we want something short and rate-limited if the message can be trigerred by a guest in production ...
---8<---
We never needed the call trace and we better rate-limit if it can be triggered by a guest.

Signed-off-by Radim Krčmář <[email protected]>
---
 arch/x86/kvm/vmx.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
arch/x86/kvm/vmx.c +2 -1
index a236decb81e4..7cd606daa01e 100644
@@ -8595,7 +8595,8 @@ static int vmx_handle_exit(struct kvm_vcpu *vcpu)
            && kvm_vmx_exit_handlers[exit_reason])
                return kvm_vmx_exit_handlers[exit_reason](vcpu);
        else {
-               WARN_ONCE(1, "vmx: unexpected exit reason 0x%x\n", exit_reason);
+               vcpu_unimpl(vcpu, "vmx: unexpected exit reason 0x%x\n",
+                               exit_reason);
Quoted 8 lines
Quoted 6 lines
Quoted 6 lines
arch/x86/kvm/vmx.c +0 -1
                kvm_queue_exception(vcpu, UD_VECTOR);
                return 1;
        }
--

2.11.0

Was this ever merged? Still happens for me.

No, it was forgotten ... queued now,

Thanks