2nd level lockups using VMX nesting on 3.11 based host kernel

With current 3.11 kernels we got reports of nested qemu failing in weird ways. I believe 3.10 also had issues before. Not sure whether those were the same. With 3.8 based kernels (close to current stable) I found no such issues. It is possible to reproduce things with the following setup:

Host 64bit user-space, kernel 3.8 or 3.11 based, 64bit hypervisor,
Haswell CPU masked to core2duo for 1rst, 4-core, 8G memory
1rst 32 or 64bit user-space, kernel 3.8 or 3.11 based,
virtio net and block device, swap, 64bit hypervisor, 2-vcpu, 2G memory
2nd 32bit user-space, kernel 3.8 based
user network stack virtio, virtio block device, 2-vcpu, 1G memory

Test is basically to start the 2nd level guest from a base raw image file and perform some package updates and install some new packages through ssh inside.

With a 3.8 kernel running on the host the host logs some attempted (and likely ignored) MSR accesses (caused by masking the vcpu to core2duo) but the install in the 2nd level succeeds, except when the 1rst level runs a 32bit userspace. In that case I could observe the 1rst level qemu process to use a lot of cpu time but the 2nd level showed signs of soft-lockup. Maybe at least one of the second level vcpus not getting scheduled anymore?

Switching the host kernel to 3.11 (about -rc4) the 2nd level install fails with various symptoms. For a 32bit user-space it looks like the previously described lockup. Though I observed NMI reason 21 and 31 messages as well. With a 3.8 kernel and 64bit user-space in 1rst level there were the NMI messages again but a double fault crash in 2nd level which seemed to have cmos_interrupt function on the start of the stack.
Using a 3.11-rc4 64bit user-space in 1rst level only had the double fault without the NMI messages.

The symptoms could vary but the ones described above were the most likely with a given combination. I also tried 3.11 with 64bit userspace on host and 1rst level while not doing any cpu masking (so 1rst level sees a qemu 64bit vcpu). This got rid of the msr messages but otherwise would make the 2nd level get stuck without any messages. Sometimes with 1rst level busy sometimes not. Though, except for very rare cases where things really went bad and in one case took down the host, the 2nd level guest can be killed by ctrl-c from the 1rst level guest.

Now I am not sure which way to debug this better. Has anybody seem similar things or can help me with some advice on how to get more information?

Thanks,
Stefan

Please cc me on replies as I am not subscribed.

Try to bisect it.

--
Gleb.

It took a while to bisect. Though I am not sure this helps much. Starting from v3.9, the first broken commit is:

commit 5f3d5799974b89100268ba813cec8db7bd0693fb
KVM: nVMX: Rework event injection and recovery

This sounds reasonable as this changes event injection between nested levels. However starting with this patch I am unable to start any second level guest. Very soon after the second level guest starts, the first (and by that the second level as well) lock up completely without any visible messages.

This goes on until

commit 5a2892ce72e010e3cb96b438d7cdddce0c88e0e6
KVM: nVMX: Skip PF interception check when queuing during nested run

In between there was also a period where first level did not lock up but would either seem not to schedule the second level guest or displayed internal error messages from starting the second level.

Given that it sounds like the current double faults in second level might be one of the issues introduced by the injection rework that remains until now while other issues were fixed from the second commit on.

I am not really deeply familiar with the nVMX code, just trying to make sense of observations. The double fault always seems to originate from the cmos_interrupt function in the second level guest. It is not immediate and sometimes took several repeated runs to trigger (during bisect I would require 10 successful test runs before marking it good). So could it maybe be some event / interrupt (cmos related?) that accidentally gets injected into the wrong guest level? Or maybe the same event taking place at the same time for more than one level and messing up things?

-Stefan

Il 10/09/2013 09:52, Stefan Bader ha scritto:

I'm not sure I'm seeing the same issue as you, but it is similar enough to point out.

Nested virtualization is completely broken with shadow paging on the host even before commit 5f3d5799974b89100268ba813cec8db7bd0693fb. Whether it works probably depends on the combination of host and guest kernels; I am constantly using 3.10 in the guest. It is very reproducible, my testcase is x86/realmode.flat from kvm-unit-tests.

There are several problems, some of which were fixed along the way. I bisected while doing this:

- apply patch 63fbf59 (nVMX: reset rflags register cache during nested vmentry., 2013-07-28)

- use the emulate_invalid_guest_state=0 argument to kvm-intel. This is fixed somewhere between commit 5f3d579 and commit 205befd (KVM:
nVMX: correctly set tr base on nested vmexit emulation, 2013-08-04); I haven't bisected it fully, but it should not be necessary.

The resulting faulty patch is the same as yours. The symptoms are the same for all three cases:

- commit 5f3d579 + patch 63fbf59 + emulate_invalid_guest_state=0

- commit 205befd + emulate_invalid_guest_state=0

- commit 205befd + emulate_invalid_guest_state=1

My first impression is that a pagefault is injected erroneously, will look more at it tomorrow.

Paolo