With 3.20.0-0.rc0.git5.1 on L0, booting L2 guest results in L1 *rebooting*

I can observe this only one of the Intel Xeon machines (which has 48 CPUs and 1TB memory), but very reliably reproducible.

Reproducer:

- Just ensure physical host (L0) and guest hypervisor (L1) are running 3.20.0-0.rc0.git5.1 Kernel (I used from Fedora's Rawhide). Preferably on an Intel Xeon machine - as that's where I could reproduce this issue, not on a Haswell machine
- Boot an L2 guest: Run qemu-sanity-check --accel=kvm in L1 (or your own preferred method to boot an L2 KVM guest).
- On a different terminal, which has serial console for L1: observe L1 reboot

The only thing I notice in demsg (on L0) is this trace. _However_ this trace does not occur when an L1 reboot is triggered while you watch dmesg -w (to wait for new messages) as I boot an L2 guest -- which means, the below trace is not the root cause of L1 being rebooted. When the L2 gets rebooted, what you observe is just one of these messages "vcpu0 unhandled rdmsr: 0x1a6" below

. . .
[Feb16 13:44] ------------[ cut here ]------------

[  +0.004632] WARNING: CPU: 4 PID: 1837 at arch/x86/kvm/vmx.c:9190 nested_vmx_vmexit+0x96e/0xb00 [kvm_intel]()
[  +0.009835] Modules linked in: xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack tun bridge stp llc ip6table_filter ip6_tables cfg80211 rfkill iTCO_wdt iTCO_vendor_support ipmi_devintf gpio_ich dcdbas coretemp kvm_intel kvm crc32c_intel ipmi_ssif serio_raw acpi_power_meter ipmi_si tpm_tis ipmi_msghandler tpm lpc_ich i7core_edac mfd_core edac_core acpi_cpufreq shpchp wmi mgag200 i2c_algo_bit drm_kms_helper ttm ata_generic drm pata_acpi megaraid_sas bnx2
[  +0.050289] CPU: 4 PID: 1837 Comm: qemu-system-x86 Not tainted 3.20.0-0.rc0.git5.1.fc23.x86_64 #1
[  +0.008902] Hardware name: Dell Inc. PowerEdge R910/0P658H, BIOS 2.8.2 10/25/2012
[  +0.007469]  0000000000000000 00000000ee6c0c54 ffff88bf60bf7c18 ffffffff818760f7
[  +0.007542]  0000000000000000 0000000000000000 ffff88bf60bf7c58 ffffffff810ab80a
[  +0.007519]  ffff88ff625b8000 ffff883f55f9b000 0000000000000000 0000000000000014
[  +0.007489] Call Trace:
[  +0.002471]  [<ffffffff818760f7>] dump_stack+0x4c/0x65
[  +0.005152]  [<ffffffff810ab80a>] warn_slowpath_common+0x8a/0xc0
[  +0.006020]  [<ffffffff810ab93a>] warn_slowpath_null+0x1a/0x20
[  +0.005851]  [<ffffffffa130957e>] nested_vmx_vmexit+0x96e/0xb00 [kvm_intel]
[  +0.006974]  [<ffffffffa130c5f7>] ? vmx_handle_exit+0x1e7/0xcb2 [kvm_intel]
[  +0.006999]  [<ffffffffa02ca972>] ? kvm_arch_vcpu_ioctl_run+0x6d2/0x1b50 [kvm]
[  +0.007239]  [<ffffffffa130992a>] vmx_queue_exception+0x10a/0x150 [kvm_intel]
[  +0.007136]  [<ffffffffa02cb30b>] kvm_arch_vcpu_ioctl_run+0x106b/0x1b50 [kvm]
[  +0.007162]  [<ffffffffa02ca972>] ? kvm_arch_vcpu_ioctl_run+0x6d2/0x1b50 [kvm]
[  +0.007241]  [<ffffffff8110760d>] ? trace_hardirqs_on+0xd/0x10
[  +0.005864]  [<ffffffffa02b2df6>] ? vcpu_load+0x26/0x70 [kvm]
[  +0.005761]  [<ffffffff81103c0f>] ? lock_release_holdtime.part.29+0xf/0x200
[  +0.006979]  [<ffffffffa02c5f88>] ? kvm_arch_vcpu_load+0x58/0x210 [kvm]
[  +0.006634]  [<ffffffffa02b3203>] kvm_vcpu_ioctl+0x383/0x7e0 [kvm]
[  +0.006197]  [<ffffffff81027b9d>] ? native_sched_clock+0x2d/0xa0
[  +0.006026]  [<ffffffff810d5fc6>] ? creds_are_invalid.part.1+0x16/0x50
[  +0.006537]  [<ffffffff810d6021>] ? creds_are_invalid+0x21/0x30
[  +0.005930]  [<ffffffff813a61da>] ? inode_has_perm.isra.48+0x2a/0xa0
[  +0.006365]  [<ffffffff8128c7b8>] do_vfs_ioctl+0x2e8/0x530
[  +0.005496]  [<ffffffff8128ca81>] SyS_ioctl+0x81/0xa0
[  +0.005065]  [<ffffffff8187f8e9>] system_call_fastpath+0x12/0x17
[  +0.006014] ---[ end trace 2f24e0820b44f686 ]---
[  +5.870886] kvm [1783]: vcpu0 unhandled rdmsr: 0x1c9
[  +0.004991] kvm [1783]: vcpu0 unhandled rdmsr: 0x1a6
[  +0.005020] kvm [1783]: vcpu0 unhandled rdmsr: 0x3f6

[Feb16 14:18] kvm [1783]: vcpu0 unhandled rdmsr: 0x1c9 [ +0.005020] kvm [1783]: vcpu0 unhandled rdmsr: 0x1a6 [ +0.004998] kvm [1783]: vcpu0 unhandled rdmsr: 0x3f6 . . .

Version
-------

Exact below versions were used on L0 and L1:

  $ uname -r; rpm -q qemu-system-x86
  3.20.0-0.rc0.git5.1.fc23.x86_64
  qemu-system-x86-2.2.0-5.fc22.x86_64

Other info
----------

- Unpacking the kernel-3.20.0-0.rc0.git5.1.fc23.src.rpm and looking at this file, arch/x86/kvm/vmx.c, line 9190 is below, with contextual code:

   [. . .]
   9178  * Emulate an exit from nested guest (L2) to L1, i.e., prepare to run L1
   9179  * and modify vmcs12 to make it see what it would expect to see there if
   9180  * L2 was its real guest. Must only be called when in L2 (is_guest_mode())
   9181  */
   9182 static void nested_vmx_vmexit(struct kvm_vcpu *vcpu, u32 exit_reason,
   9183                               u32 exit_intr_info,
   9184                               unsigned long exit_qualification)
   9185 {
   9186         struct vcpu_vmx *vmx = to_vmx(vcpu);
   9187         struct vmcs12 *vmcs12 = get_vmcs12(vcpu);
   9188 
   9189         /* trying to cancel vmlaunch/vmresume is a bug */
   9190         WARN_ON_ONCE(vmx->nested.nested_run_pending);
   9191 
   9192         leave_guest_mode(vcpu);
   9193         prepare_vmcs12(vcpu, vmcs12, exit_reason, exit_intr_info,
   9194                        exit_qualification);
   9195 
   9196         vmx_load_vmcs01(vcpu);
   9197 
   9198         if ((exit_reason == EXIT_REASON_EXTERNAL_INTERRUPT)
   9199             && nested_exit_intr_ack_set(vcpu)) {
   9200                 int irq = kvm_cpu_get_interrupt(vcpu);
   9201                 WARN_ON(irq < 0);
   9202                 vmcs12->vm_exit_intr_info = irq |
   9203                         INTR_INFO_VALID_MASK | INTR_TYPE_EXT_INTR;
   9204         }

- The above line 9190 was introduced in this commt:

  $ git log -S'WARN_ON_ONCE(vmx->nested.nested_run_pending)' \
      -- ./arch/x86/kvm/vmx.c
  commit 5f3d5799974b89100268ba813cec8db7bd0693fb
  Author: Jan Kiszka <[email protected]>
  Date:   Sun Apr 14 12:12:46 2013 +0200

KVM: nVMX: Rework event injection and recovery

The basic idea is to always transfer the pending event injection on vmexit into the architectural state of the VCPU and then drop it from there if it turns out that we left L2 to enter L1, i.e. if we enter prepare_vmcs12.

vmcs12_save_pending_events takes care to transfer pending L0 events into the queue of L1. That is mandatory as L1 may decide to switch the guest state completely, invalidating or preserving the pending events for later injection (including on a different node, once we support migration).

This concept is based on the rule that a pending vmlaunch/vmresume is not canceled. Otherwise, we would risk to lose injected events or leak them into the wrong queues. Encode this rule via a WARN_ON_ONCE at the entry of nested_vmx_vmexit.

Signed-off-by Jan Kiszka <[email protected]>

Signed-off-by Gleb Natapov <[email protected]>

- dmesg, dmidecode, x86info -a details of L0 and L1 here

https://kashyapc.fedorapeople.org/virt/Info-L0-Intel-Xeon-and-L1-nVMX-test/

Quoted 142 lines · Kashyap Chamarthy

I can observe this only one of the Intel Xeon machines (which has 48 CPUs and 1TB memory), but very reliably reproducible.

Reproducer:

- Just ensure physical host (L0) and guest hypervisor (L1) are running 3.20.0-0.rc0.git5.1 Kernel (I used from Fedora's Rawhide). Preferably on an Intel Xeon machine - as that's where I could reproduce this issue, not on a Haswell machine
- Boot an L2 guest: Run qemu-sanity-check --accel=kvm in L1 (or your own preferred method to boot an L2 KVM guest).
- On a different terminal, which has serial console for L1: observe L1 reboot

The only thing I notice in demsg (on L0) is this trace. _However_ this trace does not occur when an L1 reboot is triggered while you watch dmesg -w (to wait for new messages) as I boot an L2 guest -- which means, the below trace is not the root cause of L1 being rebooted. When the L2 gets rebooted, what you observe is just one of these messages "vcpu0 unhandled rdmsr: 0x1a6" below

. . .
[Feb16 13:44] ------------[ cut here ]------------

[  +0.004632] WARNING: CPU: 4 PID: 1837 at arch/x86/kvm/vmx.c:9190 nested_vmx_vmexit+0x96e/0xb00 [kvm_intel]()
[  +0.009835] Modules linked in: xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack tun bridge stp llc ip6table_filter ip6_tables cfg80211 rfkill iTCO_wdt iTCO_vendor_support ipmi_devintf gpio_ich dcdbas coretemp kvm_intel kvm crc32c_intel ipmi_ssif serio_raw acpi_power_meter ipmi_si tpm_tis ipmi_msghandler tpm lpc_ich i7core_edac mfd_core edac_core acpi_cpufreq shpchp wmi mgag200 i2c_algo_bit drm_kms_helper ttm ata_generic drm pata_acpi megaraid_sas bnx2
[  +0.050289] CPU: 4 PID: 1837 Comm: qemu-system-x86 Not tainted 3.20.0-0.rc0.git5.1.fc23.x86_64 #1
[  +0.008902] Hardware name: Dell Inc. PowerEdge R910/0P658H, BIOS 2.8.2 10/25/2012
[  +0.007469]  0000000000000000 00000000ee6c0c54 ffff88bf60bf7c18 ffffffff818760f7
[  +0.007542]  0000000000000000 0000000000000000 ffff88bf60bf7c58 ffffffff810ab80a
[  +0.007519]  ffff88ff625b8000 ffff883f55f9b000 0000000000000000 0000000000000014
[  +0.007489] Call Trace:
[  +0.002471]  [<ffffffff818760f7>] dump_stack+0x4c/0x65
[  +0.005152]  [<ffffffff810ab80a>] warn_slowpath_common+0x8a/0xc0
[  +0.006020]  [<ffffffff810ab93a>] warn_slowpath_null+0x1a/0x20
[  +0.005851]  [<ffffffffa130957e>] nested_vmx_vmexit+0x96e/0xb00 [kvm_intel]
[  +0.006974]  [<ffffffffa130c5f7>] ? vmx_handle_exit+0x1e7/0xcb2 [kvm_intel]
[  +0.006999]  [<ffffffffa02ca972>] ? kvm_arch_vcpu_ioctl_run+0x6d2/0x1b50 [kvm]
[  +0.007239]  [<ffffffffa130992a>] vmx_queue_exception+0x10a/0x150 [kvm_intel]
[  +0.007136]  [<ffffffffa02cb30b>] kvm_arch_vcpu_ioctl_run+0x106b/0x1b50 [kvm]
[  +0.007162]  [<ffffffffa02ca972>] ? kvm_arch_vcpu_ioctl_run+0x6d2/0x1b50 [kvm]
[  +0.007241]  [<ffffffff8110760d>] ? trace_hardirqs_on+0xd/0x10
[  +0.005864]  [<ffffffffa02b2df6>] ? vcpu_load+0x26/0x70 [kvm]
[  +0.005761]  [<ffffffff81103c0f>] ? lock_release_holdtime.part.29+0xf/0x200
[  +0.006979]  [<ffffffffa02c5f88>] ? kvm_arch_vcpu_load+0x58/0x210 [kvm]
[  +0.006634]  [<ffffffffa02b3203>] kvm_vcpu_ioctl+0x383/0x7e0 [kvm]
[  +0.006197]  [<ffffffff81027b9d>] ? native_sched_clock+0x2d/0xa0
[  +0.006026]  [<ffffffff810d5fc6>] ? creds_are_invalid.part.1+0x16/0x50
[  +0.006537]  [<ffffffff810d6021>] ? creds_are_invalid+0x21/0x30
[  +0.005930]  [<ffffffff813a61da>] ? inode_has_perm.isra.48+0x2a/0xa0
[  +0.006365]  [<ffffffff8128c7b8>] do_vfs_ioctl+0x2e8/0x530
[  +0.005496]  [<ffffffff8128ca81>] SyS_ioctl+0x81/0xa0
[  +0.005065]  [<ffffffff8187f8e9>] system_call_fastpath+0x12/0x17
[  +0.006014] ---[ end trace 2f24e0820b44f686 ]---
[  +5.870886] kvm [1783]: vcpu0 unhandled rdmsr: 0x1c9
[  +0.004991] kvm [1783]: vcpu0 unhandled rdmsr: 0x1a6
[  +0.005020] kvm [1783]: vcpu0 unhandled rdmsr: 0x3f6

[Feb16 14:18] kvm [1783]: vcpu0 unhandled rdmsr: 0x1c9 [ +0.005020] kvm [1783]: vcpu0 unhandled rdmsr: 0x1a6 [ +0.004998] kvm [1783]: vcpu0 unhandled rdmsr: 0x3f6 . . .

Version
-------

Exact below versions were used on L0 and L1:

  $ uname -r; rpm -q qemu-system-x86
  3.20.0-0.rc0.git5.1.fc23.x86_64
  qemu-system-x86-2.2.0-5.fc22.x86_64

Other info
----------

- Unpacking the kernel-3.20.0-0.rc0.git5.1.fc23.src.rpm and looking at this file, arch/x86/kvm/vmx.c, line 9190 is below, with contextual code:

   [. . .]
   9178  * Emulate an exit from nested guest (L2) to L1, i.e., prepare to run L1
   9179  * and modify vmcs12 to make it see what it would expect to see there if
   9180  * L2 was its real guest. Must only be called when in L2 (is_guest_mode())
   9181  */
   9182 static void nested_vmx_vmexit(struct kvm_vcpu *vcpu, u32 exit_reason,
   9183                               u32 exit_intr_info,
   9184                               unsigned long exit_qualification)
   9185 {
   9186         struct vcpu_vmx *vmx = to_vmx(vcpu);
   9187         struct vmcs12 *vmcs12 = get_vmcs12(vcpu);
   9188 
   9189         /* trying to cancel vmlaunch/vmresume is a bug */
   9190         WARN_ON_ONCE(vmx->nested.nested_run_pending);
   9191 
   9192         leave_guest_mode(vcpu);
   9193         prepare_vmcs12(vcpu, vmcs12, exit_reason, exit_intr_info,
   9194                        exit_qualification);
   9195 
   9196         vmx_load_vmcs01(vcpu);
   9197 
   9198         if ((exit_reason == EXIT_REASON_EXTERNAL_INTERRUPT)
   9199             && nested_exit_intr_ack_set(vcpu)) {
   9200                 int irq = kvm_cpu_get_interrupt(vcpu);
   9201                 WARN_ON(irq < 0);
   9202                 vmcs12->vm_exit_intr_info = irq |
   9203                         INTR_INFO_VALID_MASK | INTR_TYPE_EXT_INTR;
   9204         }

- The above line 9190 was introduced in this commt:

  $ git log -S'WARN_ON_ONCE(vmx->nested.nested_run_pending)' \
      -- ./arch/x86/kvm/vmx.c
  commit 5f3d5799974b89100268ba813cec8db7bd0693fb
  Author: Jan Kiszka <[email protected]>
  Date:   Sun Apr 14 12:12:46 2013 +0200

KVM: nVMX: Rework event injection and recovery

The basic idea is to always transfer the pending event injection on vmexit into the architectural state of the VCPU and then drop it from there if it turns out that we left L2 to enter L1, i.e. if we enter prepare_vmcs12.

vmcs12_save_pending_events takes care to transfer pending L0 events into the queue of L1. That is mandatory as L1 may decide to switch the guest state completely, invalidating or preserving the pending events for later injection (including on a different node, once we support migration).

This concept is based on the rule that a pending vmlaunch/vmresume is not canceled. Otherwise, we would risk to lose injected events or leak them into the wrong queues. Encode this rule via a WARN_ON_ONCE at the entry of nested_vmx_vmexit.

Signed-off-by Jan Kiszka <[email protected]>

Signed-off-by Gleb Natapov <[email protected]>

Quoted 4 lines

- dmesg, dmidecode, x86info -a details of L0 and L1 here

https://kashyapc.fedorapeople.org/virt/Info-L0-Intel-Xeon-and-L1-nVMX-test/

Does enable_apicv make a difference?

Is this a regression caused by the commit, or do you only see it with very recent kvm.git?

Quoted 2 lines

Does enable_apicv make a difference?

Actually, I did perform a test (on Paolo's suggestion on IRC) with enable_apicv=0 on physical host, and it didn't make any difference:

$ cat /proc/cmdline 
BOOT_IMAGE=/vmlinuz-3.20.0-0.rc0.git5.1.fc23.x86_64 root=/dev/mapper/fedora--server_dell--per910--02-root ro console=ttyS1,115200n81 rd.lvm.lv=fedora-server_dell-per910-02/swap rd.lvm.lv=fedora-server_dell-per910-02/root LANG=en_US.UTF-8 enable_apicv=0

Quoted 2 lines

Is this a regression caused by the commit, or do you only see it with very recent kvm.git?

Afraid, I didn't bisect it, but I just wanted to note that the above specific WARN was introduced in the above commit.

I'm sure this Kernel (on L0) does not exhibit the problem:
kernel-3.17.4-301.fc21.x86_64. But, if I had either of these two Kernels on the physical host, then the said problem manifests (L1 reboots):
3.19.0-1.fc22 or kernel-3.20.0-0.rc0.git5.1.fc23

Kashyap Chamarthy <[email protected]> writes:
..

Quoted 3 lines

Quoted 2 lines

Does enable_apicv make a difference?

Quoted 5 lines

Actually, I did perform a test (on Paolo's suggestion on IRC) with enable_apicv=0 on physical host, and it didn't make any difference:

$ cat /proc/cmdline 
BOOT_IMAGE=/vmlinuz-3.20.0-0.rc0.git5.1.fc23.x86_64 root=/dev/mapper/fedora--server_dell--per910--02-root ro console=ttyS1,115200n81 rd.lvm.lv=fedora-server_dell-per910-02/swap rd.lvm.lv=fedora-server_dell-per910-02/root LANG=en_US.UTF-8 enable_apicv=0

I am not sure if this works ? enable_apicv is a kvm_intel module parameter

Quoted 8 lines · Bandan Das

Kashyap Chamarthy <[email protected]> writes:
..

Quoted 6 lines

Quoted 2 lines

Does enable_apicv make a difference?

Actually, I did perform a test (on Paolo's suggestion on IRC) with enable_apicv=0 on physical host, and it didn't make any difference:

Quoted 4 lines

Quoted 2 lines

$ cat /proc/cmdline 
BOOT_IMAGE=/vmlinuz-3.20.0-0.rc0.git5.1.fc23.x86_64 root=/dev/mapper/fedora--server_dell--per910--02-root ro console=ttyS1,115200n81 rd.lvm.lv=fedora-server_dell-per910-02/swap rd.lvm.lv=fedora-server_dell-per910-02/root LANG=en_US.UTF-8 enable_apicv=0

I am not sure if this works ? enable_apicv is a kvm_intel module parameter

Good point. Has to be kvm_intel.enable_apicv=0 (if the module is built in).

Afraid, I didn't bisect it, but I just wanted to note that the above specific WARN was introduced in the above commit.

I'm sure this Kernel (on L0) does not exhibit the problem:
kernel-3.17.4-301.fc21.x86_64. But, if I had either of these two Kernels on the physical host, then the said problem manifests (L1 reboots):
3.19.0-1.fc22 or kernel-3.20.0-0.rc0.git5.1.fc23

Quoted 3 lines · Kashyap Chamarthy

Afraid, I didn't bisect it, but I just wanted to note that the above specific WARN was introduced in the above commit.

Quoted 4 lines

I'm sure this Kernel (on L0) does not exhibit the problem:
kernel-3.17.4-301.fc21.x86_64. But, if I had either of these two Kernels on the physical host, then the said problem manifests (L1 reboots):
3.19.0-1.fc22 or kernel-3.20.0-0.rc0.git5.1.fc23

Nested APICv is not part of 3.19, so it cannot be the culprit.

Can you try 3.18?

Just did two tests with 3.18:

(1) Kernel 3.18 on L0 and 3.20 on L1

Result: Booting L2 guest causes L1 to reboot, and the same[*] stack
trace on L0 (mentioned on this thread previously).

But, annoyingly enough, when I did test (2) below, and then switched back to test (1), I don't notice the said stack trace in L0's dmesg however many times I boot an L2 guest.

(2) Kernel 3.18 on both L0 and L1

Result: Booting L2 guest causes L1 to reboot, but *no* stack trace
on L0

2015-02-19 13:07+0100, Kashyap Chamarthy:
5f3d5799974b8 KVM: nVMX: Rework event injection and recovery:
This concept is based on the rule that a pending vmlaunch/vmresume is not canceled. Otherwise, we would risk to lose injected events or leak them into the wrong queues. Encode this rule via a WARN_ON_ONCE at the entry of nested_vmx_vmexit.

I wonder if we have broken the invariant since 3.9 ...

Fixes e011c663b9c7 ("Check all exceptions for intercept during delivery to L2")

2015-02-19 16:01+0100, Radim Krčmář:

Quoted 8 lines

2015-02-19 13:07+0100, Kashyap Chamarthy:
5f3d5799974b8 KVM: nVMX: Rework event injection and recovery:
This concept is based on the rule that a pending vmlaunch/vmresume is not canceled. Otherwise, we would risk to lose injected events or leak them into the wrong queues. Encode this rule via a WARN_ON_ONCE at the entry of nested_vmx_vmexit.

I wonder if we have broken the invariant since 3.9 ...

e011c663b9c786d115c0f45e5b0bfae0c39428d4
KVM: nVMX: Check all exceptions for intercept during delivery to L2

All exceptions should be checked for intercept during delivery to L2, but we check only #PF currently. Drop nested_run_pending while we are at it since exception cannot be injected during vmentry anyway.

The last sentence is not true.

Can you try if the following patch works?

(I know little about nested, so it might be introducing another bug.)

Thanks.

---8<---
KVM: nVMX: fix L2 to L1 interrupt leak

When vmx->nested.nested_run_pending is set, we aren't expected to exit to L1, but nested_vmx_check_exception() could, since e011c663b9c7. Prevent that.

Fixes e011c663b9c7 ("Check all exceptions for intercept during delivery to L2")

Signed-off-by Radim Krčmář <[email protected]>

---
 arch/x86/kvm/vmx.c | 3 +++
 1 file changed, 3 insertions(+)

arch/x86/kvm/vmx.c +3 -0

index 3f73bfad0349..389166a1b79a 100644
@@ -2098,6 +2098,9 @@ static int nested_vmx_check_exception(struct kvm_vcpu *vcpu, unsigned nr)
 {
 	struct vmcs12 *vmcs12 = get_vmcs12(vcpu);
 
+	if (to_vmx(vcpu)->nested.nested_run_pending)
+		return 0;
+
 	if (!(vmcs12->exception_bitmap & (1u << nr)))
 		return 0;

On Thu, Feb 19, 2015 at 05:02:22PM +0100, Radim Krčmář wrote:

Quoted 1 line

Can you try if the following patch works?

Sure, will test a Kernel built with the below patch and report back.

Quoted 1 line · Radim Krčmář

[. . .]

Quoted 3 lines

Quoted 1 line

Can you try if the following patch works?

Sure, will test a Kernel built with the below patch and report back.

Hmm, I'm stuck with a meta issue.

I checked out the KVM tree[1] on L0, applied your patch and built[*] the Kernel, and booted into it. Boot fails and drops into a dracut shell on because:

. . .
dracut-initqueue[3045]: Warning: Cancelling resume operation. Device not found. [ TIME ] Timed out waiting for device dev-ma...per910\x2d\x2d02\x2droot.device.
[DEPEND] Dependency failed for /sysroot.
[DEPEND] Dependency failed for Initrd Root File SyWarning:
/dev/disk/by-uuid/4ccddb2d-4d63-4fce-b4d4-9b2f119a30cc does not exist . . .

I saved the report from /run/initramfs/rdsosreport.txt here[2].

Then, I did another test:

- Rebooted into Kernel 3.20.0-0.rc0.git5.1.fc23.x86_64 on physical

    host (L0).

- In L1, checked out the KVM tree, applied your patch and built Kernel[*] from the current KVM tree and booted into the newly built one, here too, I'm thrown into a dracut shell

[1] git://git.kernel.org/pub/scm/virt/kvm/kvm.git
[2] https://kashyapc.fedorapeople.org/temp/kernel-boot-failure.txt

[*] Exactly, I built it this way:

# Clone the tree

  $ git://git.kernel.org/pub/scm/virt/kvm/kvm.git

# Make a new branch:

  $ git checkout -b nvmx_test
  $ git describe
  warning: tag 'for-linus' is really 'kvm-3.19-1' here
  for-linus-14459-g49776d5

# Make a config file

  $ make defconfig

  # Compile
  $ make -j4 && make bzImage && make modules
  
  # Install
  $ sudo -i
  $ make modules_install && make install

Signature · 1 line

/kashyap

2015-02-19 23:28+0100, Kashyap Chamarthy:

Quoted 2 lines · Kashyap Chamarthy

Quoted 1 line · Radim Krčmář

Quoted 7 lines

Then, I did another test:

- Rebooted into Kernel 3.20.0-0.rc0.git5.1.fc23.x86_64 on physical

    host (L0).

- In L1, checked out the KVM tree, applied your patch and built Kernel[*] from the current KVM tree and booted into the newly built one, here too, I'm thrown into a dracut shell

Weird, but considering that boot fails on L0 as well, I think it that basing off a different commit could help ...

Quoted 10 lines

[*] Exactly, I built it this way:

# Clone the tree

  $ git://git.kernel.org/pub/scm/virt/kvm/kvm.git

# Make a new branch:

  $ git checkout -b nvmx_test
  $ git describe
  warning: tag 'for-linus' is really 'kvm-3.19-1' here
  for-linus-14459-g49776d5

Hm, it should say v3.19 -- does it stay the same if you do git fetch && git checkout origin/master?

If it still does, please try to apply it on top of git checkout v3.18. (The one that one failed too.)

Quoted 2 lines

# Make a config file

  $ make defconfig

It would be safer to copy the fedora config (from /boot) to .config and do make olddefconfig.

Radim,

I just tested with your patch[1] in this thread. I built a Fedora Kernel[2] with it, and installed (and booted into) it on both L0 and L1.

Result: I don't have good news, I'm afraid: L1 *still* reboots when an
L2 guest is booted. And, L0 throws the stack trace that was previously noted on this thread:

. . .
[< 57.747345>] ------------[ cut here ]------------
[< 0.004638>] WARNING: CPU: 5 PID: 50206 at arch/x86/kvm/vmx.c:8962 nested_vmx_vmexit+0x7ee/0x880 [kvm_intel]()
[< 0.060404>] CPU: 5 PID: 50206 Comm: qemu-system-x86 Not tainted 3.18.7-200.fc21.x86_64 #1

[  +0.006055]  [<ffffffff810992ea>] warn_slowpath_null+0x1a/0x20
[  +0.005889]  [<ffffffffa02f00ee>] nested_vmx_vmexit+0x7ee/0x880 [kvm_intel]
[  +0.007014]  [<ffffffffa02f05af>] ? vmx_handle_exit+0x1bf/0xaa0 [kvm_intel]
[  +0.007015]  [<ffffffffa02f039c>] vmx_queue_exception+0xfc/0x150 [kvm_intel]
[  +0.007130]  [<ffffffffa028cdfd>] kvm_arch_vcpu_ioctl_run+0xd9d/0x1290 [kvm]

[  +0.007111]  [<ffffffffa0288528>] ? kvm_arch_vcpu_load+0x58/0x220 [kvm]
[  +0.006670]  [<ffffffffa0274cbc>] kvm_vcpu_ioctl+0x32c/0x5c0 [kvm]

[1] http://article.gmane.org/gmane.comp.emulators.kvm.devel/132937
[2] http://koji.fedoraproject.org/koji/taskinfo?taskID=9004708

2015-02-22 16:46+0100, Kashyap Chamarthy:

Quoted 5 lines

Radim,

I just tested with your patch[1] in this thread. I built a Fedora Kernel[2] with it, and installed (and booted into) it on both L0 and L1.

Quoted 3 lines

Result: I don't have good news, I'm afraid: L1 *still* reboots when an
L2 guest is booted. And, L0 throws the stack trace that was previously noted on this thread:

Thanks, I'm puzzled though ... isn't it possible that a wrong kernel sneaked into grub?

Quoted 4 lines

. . .
[< 57.747345>] ------------[ cut here ]------------
[< 0.004638>] WARNING: CPU: 5 PID: 50206 at arch/x86/kvm/vmx.c:8962 nested_vmx_vmexit+0x7ee/0x880 [kvm_intel]()
[< 0.060404>] CPU: 5 PID: 50206 Comm: qemu-system-x86 Not tainted 3.18.7-200.fc21.x86_64 #1

This looks like a new backtrace, but the kernel is not [2].

Quoted 5 lines

[  +0.006055]  [<ffffffff810992ea>] warn_slowpath_null+0x1a/0x20
[  +0.005889]  [<ffffffffa02f00ee>] nested_vmx_vmexit+0x7ee/0x880 [kvm_intel]
[  +0.007014]  [<ffffffffa02f05af>] ? vmx_handle_exit+0x1bf/0xaa0 [kvm_intel]
[  +0.007015]  [<ffffffffa02f039c>] vmx_queue_exception+0xfc/0x150 [kvm_intel]
[  +0.007130]  [<ffffffffa028cdfd>] kvm_arch_vcpu_ioctl_run+0xd9d/0x1290 [kvm]

(There is only one execution path and unless there is a race, it would
be prevented by [1].)

Quoted 2 lines

[  +0.007111]  [<ffffffffa0288528>] ? kvm_arch_vcpu_load+0x58/0x220 [kvm]
[  +0.006670]  [<ffffffffa0274cbc>] kvm_vcpu_ioctl+0x32c/0x5c0 [kvm]

[...]

Quoted 2 lines

[1] http://article.gmane.org/gmane.comp.emulators.kvm.devel/132937
[2] http://koji.fedoraproject.org/koji/taskinfo?taskID=9004708

Quoted 10 lines · Radim Krčmář

2015-02-22 16:46+0100, Kashyap Chamarthy:

Quoted 8 lines

Radim,

I just tested with your patch[1] in this thread. I built a Fedora Kernel[2] with it, and installed (and booted into) it on both L0 and L1.

Result: I don't have good news, I'm afraid: L1 *still* reboots when an
L2 guest is booted. And, L0 throws the stack trace that was previously noted on this thread:

Quoted 2 lines

Thanks, I'm puzzled though ... isn't it possible that a wrong kernel sneaked into grub?

Hmm, unlikely - I just double-confirmed that I'm running the same patched Kernel (3.20.0-0.rc0.git9.1.fc23.x86_64) on both L0 and L1.

Tested with the _correct_ Kernel[1] (that has Radim's patch) now -- applied it on both L0 and L1.

Result: Same as before -- Booting L2 causes L1 to reboot. However, the
stack trace from dmesg on L0 is took slightly different path than before -- it's using MSR handling:

2015-02-23 19:05+0100, Kashyap Chamarthy:

Quoted 6 lines

Tested with the _correct_ Kernel[1] (that has Radim's patch) now -- applied it on both L0 and L1.

Result: Same as before -- Booting L2 causes L1 to reboot. However, the
stack trace from dmesg on L0 is took slightly different path than before -- it's using MSR handling:

Thanks, the problem was deeper ... L1 enabled unrestricted mode while L0 had it disabled. L1 could then vmrun a L2 state that L0 would have to emulate, but that doesn't work. There are at least these solutions:

1) don't expose unrestricted_guest when L0 doesn't have it

2) fix unrestricted mode emulation code
3) handle the failure a without killing L1

I'd do just (1) -- emulating unrestricted mode is a loss.

I have done initial testing and at least qemu-sanity-check works now:

---8<---
If EPT was enabled, unrestricted_guest was allowed in L1 regardless of L0. L1 triple faulted when running L2 guest that required emulation.

Another side effect was 'WARN_ON_ONCE(vmx->nested.nested_run_pending)' in L0's dmesg:

  WARNING: CPU: 0 PID: 0 at arch/x86/kvm/vmx.c:9190 nested_vmx_vmexit+0x96e/0xb00 [kvm_intel] ()

Prevent this scenario by masking SECONDARY_EXEC_UNRESTRICTED_GUEST when the host doesn't have it enabled.

Fixes 78051e3b7e35 ("KVM: nVMX: Disable unrestricted mode if ept=0")

Signed-off-by Radim Krčmář <[email protected]>

---
 arch/x86/kvm/vmx.c | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

arch/x86/kvm/vmx.c +5 -2

index f7b20b417a3a..dbabea21357b 100644
@@ -2476,8 +2476,7 @@ static void nested_vmx_setup_ctls_msrs(struct vcpu_vmx *vmx)
 	if (enable_ept) {
 		/* nested EPT: emulate EPT also to L1 */
 		vmx->nested.nested_vmx_secondary_ctls_high |=
-			SECONDARY_EXEC_ENABLE_EPT |
-			SECONDARY_EXEC_UNRESTRICTED_GUEST;
+			SECONDARY_EXEC_ENABLE_EPT;
 		vmx->nested.nested_vmx_ept_caps = VMX_EPT_PAGE_WALK_4_BIT |
 			 VMX_EPTP_WB_BIT | VMX_EPT_2MB_PAGE_BIT |
 			 VMX_EPT_INVEPT_BIT;

@@ -2491,6 +2490,10 @@ static void nested_vmx_setup_ctls_msrs(struct vcpu_vmx *vmx)
 	} else
 		vmx->nested.nested_vmx_ept_caps = 0;
 
+	if (enable_unrestricted_guest)
+		vmx->nested.nested_vmx_secondary_ctls_high |=
+			SECONDARY_EXEC_UNRESTRICTED_GUEST;
+
 	/* miscellaneous data */
 	rdmsr(MSR_IA32_VMX_MISC,
 		vmx->nested.nested_vmx_misc_low,

Quoted 8 lines · Radim Krčmář

2015-02-23 19:05+0100, Kashyap Chamarthy:

Quoted 6 lines

Tested with the _correct_ Kernel[1] (that has Radim's patch) now -- applied it on both L0 and L1.

Result: Same as before -- Booting L2 causes L1 to reboot. However, the
stack trace from dmesg on L0 is took slightly different path than before -- it's using MSR handling:

Quoted 5 lines

Thanks, the problem was deeper ... L1 enabled unrestricted mode while L0 had it disabled. L1 could then vmrun a L2 state that L0 would have to emulate, but that doesn't work. There are at least these solutions:

1) don't expose unrestricted_guest when L0 doesn't have it

Reminds me of a patch called "KVM: nVMX: Disable unrestricted mode if ept=0" by Bandan. I thought that would have caught it - apparently not.

Quoted 4 lines

2) fix unrestricted mode emulation code
3) handle the failure a without killing L1

I'd do just (1) -- emulating unrestricted mode is a loss.

Agreed.

Jan

Quoted 16 lines

I have done initial testing and at least qemu-sanity-check works now:

---8<---
If EPT was enabled, unrestricted_guest was allowed in L1 regardless of L0. L1 triple faulted when running L2 guest that required emulation.

Another side effect was 'WARN_ON_ONCE(vmx->nested.nested_run_pending)' in L0's dmesg:

  WARNING: CPU: 0 PID: 0 at arch/x86/kvm/vmx.c:9190 nested_vmx_vmexit+0x96e/0xb00 [kvm_intel] ()

Prevent this scenario by masking SECONDARY_EXEC_UNRESTRICTED_GUEST when the host doesn't have it enabled.

Fixes 78051e3b7e35 ("KVM: nVMX: Disable unrestricted mode if ept=0")

Signed-off-by Radim Krčmář <[email protected]>

Quoted 30 lines

---
 arch/x86/kvm/vmx.c | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

arch/x86/kvm/vmx.c +5 -2

index f7b20b417a3a..dbabea21357b 100644
@@ -2476,8 +2476,7 @@ static void nested_vmx_setup_ctls_msrs(struct vcpu_vmx *vmx)
 	if (enable_ept) {
 		/* nested EPT: emulate EPT also to L1 */
 		vmx->nested.nested_vmx_secondary_ctls_high |=
-			SECONDARY_EXEC_ENABLE_EPT |
-			SECONDARY_EXEC_UNRESTRICTED_GUEST;
+			SECONDARY_EXEC_ENABLE_EPT;
 		vmx->nested.nested_vmx_ept_caps = VMX_EPT_PAGE_WALK_4_BIT |
 			 VMX_EPTP_WB_BIT | VMX_EPT_2MB_PAGE_BIT |
 			 VMX_EPT_INVEPT_BIT;

@@ -2491,6 +2490,10 @@ static void nested_vmx_setup_ctls_msrs(struct vcpu_vmx *vmx)
 	} else
 		vmx->nested.nested_vmx_ept_caps = 0;
 
+	if (enable_unrestricted_guest)
+		vmx->nested.nested_vmx_secondary_ctls_high |=
+			SECONDARY_EXEC_UNRESTRICTED_GUEST;
+
 	/* miscellaneous data */
 	rdmsr(MSR_IA32_VMX_MISC,
 		vmx->nested.nested_vmx_misc_low,

2015-02-23 19:05+0100, Kashyap Chamarthy:

Quoted 6 lines

Tested with the _correct_ Kernel[1] (that has Radim's patch) now -- applied it on both L0 and L1.

Result: Same as before -- Booting L2 causes L1 to reboot. However, the
stack trace from dmesg on L0 is took slightly different path than before -- it's using MSR handling:

Thanks, the problem was deeper ... L1 enabled unrestricted mode while L0 had it disabled. L1 could then vmrun a L2 state that L0 would have to emulate, but that doesn't work. There are at least these solutions:

1) don't expose unrestricted_guest when L0 doesn't have it
2) fix unrestricted mode emulation code
3) handle the failure a without killing L1

I'd do just (1) -- emulating unrestricted mode is a loss.

I have done initial testing and at least qemu-sanity-check works now:

---8<---
If EPT was enabled, unrestricted_guest was allowed in L1 regardless of L0. L1 triple faulted when running L2 guest that required emulation.

Another side effect was 'WARN_ON_ONCE(vmx->nested.nested_run_pending)' in L0's dmesg:

  WARNING: CPU: 0 PID: 0 at arch/x86/kvm/vmx.c:9190 nested_vmx_vmexit+0x96e/0xb00 [kvm_intel] ()

Prevent this scenario by masking SECONDARY_EXEC_UNRESTRICTED_GUEST when the host doesn't have it enabled.

Fixes 78051e3b7e35 ("KVM: nVMX: Disable unrestricted mode if ept=0")

Signed-off-by Radim Krčmář <[email protected]>

---
 arch/x86/kvm/vmx.c | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

arch/x86/kvm/vmx.c +5 -2

index f7b20b417a3a..dbabea21357b 100644
@@ -2476,8 +2476,7 @@ static void nested_vmx_setup_ctls_msrs(struct vcpu_vmx *vmx)
 	if (enable_ept) {
 		/* nested EPT: emulate EPT also to L1 */
 		vmx->nested.nested_vmx_secondary_ctls_high |=
-			SECONDARY_EXEC_ENABLE_EPT |
-			SECONDARY_EXEC_UNRESTRICTED_GUEST;
+			SECONDARY_EXEC_ENABLE_EPT;
 		vmx->nested.nested_vmx_ept_caps = VMX_EPT_PAGE_WALK_4_BIT |
 			 VMX_EPTP_WB_BIT | VMX_EPT_2MB_PAGE_BIT |
 			 VMX_EPT_INVEPT_BIT;

@@ -2491,6 +2490,10 @@ static void nested_vmx_setup_ctls_msrs(struct vcpu_vmx *vmx)
 	} else
 		vmx->nested.nested_vmx_ept_caps = 0;
 
+	if (enable_unrestricted_guest)
+		vmx->nested.nested_vmx_secondary_ctls_high |=
+			SECONDARY_EXEC_UNRESTRICTED_GUEST;
+
 	/* miscellaneous data */
 	rdmsr(MSR_IA32_VMX_MISC,
 		vmx->nested.nested_vmx_misc_low,