kvm-84: Nested virtualization, crashes, and kvm binary name

Hello,

I'm currently writing my bachelor's thesis on the KVM and ran a series
of benchmarks, partly "home grown" on it. This was a few months ago. I used the kvm package from the Debian sid repository (kvm-72) with kernel 2.6.26, which at the time were both almost up-to-date :-) (upstream KVM was at 79 IIRC). Recently I found out that meanwhile KVM is at 84, and has some exciting features like nested virtualization, which I would like to test.

So I took the upstream KVM source from sf.net and compiled it myself.
All went OK, except that I wasn't able to find a KVM binary named 'kvm', which was what it always used to be called, at least under Debian. Also no 'kvm-qemu', 'qemu' or the like.

The only thing even remotely close to what I need seems to be a binary
called 'qemu-system-x86_64'. Except that I actually have a 32 bit system! The CPU is 64-bit of course (see below), but I have a 32 bit PAE kernel with all 32 bit userland running on it.

So question #1: Is this the right thing to start, and if yes, what's
the story behind that name? I ran across some qemu-system-i386 on google, but my compile did not produce such a binary.

So even though that looked quite strange to me, I ran
'qemu-system-x86_64' instead of Debian's 'kvm'. That seemed to work, except for two things:

First off, it crashed with an abort right on the first run, when I
started it with -m 2047M (this used to work OK with Debian's kvm-72). Details see below, short description is:

*** glibc detected *** qemu-system-x86_64: corrupted double-linked list

I experimented with different parameters to -m, and using a "binary
search" approach came to the conclusion that values up to 475M work fine, from 476M upwards glibc aborts with that linked list error. In particular, omitting -m altogether also works fine, since it defaults to 128M IIRC.

So I booted my VM with -m 450M and it ran fine. However, no nested
virtualization seems to be supported. The guest's /proc/cpuinfo does not list the 'svm' flag, and installation of the 'kvm' package complains about the CPU not supporting virtualization extensions. And yes, the kvm-amd.ko module on the host was inserted with the explicit parameters "npt=1 nested=1" (though those should be the defaults anyway).

So question #2: Any hints to what I may be doing wrong?

Thanks for any answers!

Mike

====================================================================

The technical stuff:

Machine is a Dell Optiplex, don't know exact model number. AMD Phenom
9550 @ 2.2 GHz, 4GB RAM, NVidia chipset. Host is Debian sid. Here's a sample console log:

# uname -a
Linux <hostname> 2.6.26-1-686-bigmem #1 SMP Sat Jan 10 19:13:22 UTC 2009
i686 GNU/Linux

# qemu-system-x86_64 | grep version
QEMU PC emulator version 0.9.1 (kvm-84), Copyright (c) 2003-2008 Fabrice
Bellard

# modinfo kvm
filename:       /lib/modules/2.6.26-1-686-bigmem/extra/kvm.ko
license:        GPL
author:         Qumranet
version:        kvm-84
srcversion:     D964574B5665D21B64CD65A
depends:
vermagic:       2.6.26-1-686-bigmem SMP mod_unload modversions 686
parm:           oos_shadow:bool
parm:           msi2intx:bool

# modinfo kvm-amd
filename:       /lib/modules/2.6.26-1-686-bigmem/extra/kvm-amd.ko
license:        GPL
author:         Qumranet
version:        kvm-84
srcversion:     9A79BE920E710D34A514FA5
depends:        kvm
vermagic:       2.6.26-1-686-bigmem SMP mod_unload modversions 686
parm:           npt:int
parm:           nested:int

# modprobe -rv kvm-amd
rmmod /lib/modules/2.6.26-1-686-bigmem/extra/kvm-amd.ko
rmmod /lib/modules/2.6.26-1-686-bigmem/extra/kvm.ko
# modprobe -v kvm-amd npt=1 nested=1

insmod /lib/modules/2.6.26-1-686-bigmem/extra/kvm.ko
insmod /lib/modules/2.6.26-1-686-bigmem/extra/kvm-amd.ko npt=1 nested=1

(Note KVM modules being loaded from the extra/ tree, instead of the stock modules under kernel/arch/x86/kvm.)

# qemu-system-x86_64 debian-lenny.qcow2 -m 475M
# echo $?
0
# qemu-system-x86_64 debian-lenny.qcow2 -m 476M
*** glibc detected *** qemu-system-x86_64: corrupted double-linked list:
0x0941d6b8 ***
======= Backtrace: =========
/lib/i686/cmov/libc.so.6[0xb7c6dc5f]
/lib/i686/cmov/libc.so.6[0xb7c6f76d]
/lib/i686/cmov/libc.so.6(__libc_malloc+0x95)[0xb7c715a5]
qemu-system-x86_64[0x80b8a71]
qemu-system-x86_64[0x80bdb70]
qemu-system-x86_64[0x80be301]
qemu-system-x86_64[0x8053d13]
/lib/i686/cmov/libc.so.6(__libc_start_main+0xe5)[0xb7c14775]
qemu-system-x86_64[0x804cfa1]
======= Memory map: ========
08048000-081f5000 r-xp 00000000 08:02 472453
/usr/local/bin/qemu-system-x86_64
081f5000-081f8000 rw-p 001ad000 08:02 472453
/usr/local/bin/qemu-system-x86_64
081f8000-0840a000 rw-p 081f8000 00:00 0
0941d000-09699000 rw-p 0941d000 00:00 0          [heap]
97000000-97021000 rw-p 97000000 00:00 0
97021000-97100000 ---p 97021000 00:00 0
971ed000-971f9000 r-xp 00000000 08:02 2312929    /lib/libgcc_s.so.1
971f9000-971fa000 rw-p 0000c000 08:02 2312929    /lib/libgcc_s.so.1
971fa000-97361000 rw-p 971fa000 00:00 0
973e3000-973eb000 r-xp 00000000 08:02 458339
/usr/lib/libXcursor.so.1.0.2
973eb000-973ec000 rw-p 00007000 08:02 458339
/usr/lib/libXcursor.so.1.0.2
973fc000-975fc000 r--p 00000000 08:02 1286759
/usr/lib/locale/locale-archive
975fc000-97602000 r-xp 00000000 08:02 1938625    /usr/lib/libXrandr.so.2.2.0
97602000-97603000 rw-p 00006000 08:02 1938625    /usr/lib/libXrandr.so.2.2.0
97603000-9760b000 r-xp 00000000 08:02 456660
/usr/lib/libXrender.so.1.3.0
9760b000-9760c000 rw-p 00007000 08:02 456660
/usr/lib/libXrender.so.1.3.0
9760c000-97619000 r-xp 00000000 08:02 456482     /usr/lib/libXext.so.6.4.0
97619000-9761a000 rw-p 0000c000 08:02 456482     /usr/lib/libXext.so.6.4.0
9761a000-97632000 r-xp 00000000 08:02 460955     /usr/lib/libxcb.so.1.1.0
97632000-97633000 rw-p 00017000 08:02 460955     /usr/lib/libxcb.so.1.1.0
97633000-9774d000 r-xp 00000000 08:02 460953     /usr/lib/libX11.so.6.2.0
9774d000-97751000 rw-p 00119000 08:02 460953     /usr/lib/libX11.so.6.2.0
9775a000-97761000 r--s 00000000 08:02 460199
/usr/lib/gconv/gconv-modules.cache
97761000-977c4000 rw-p 97761000 00:00 0
977c4000-977c5000 ---p 977c4000 00:00 0
977c5000-98153000 rw-p 977c5000 00:00 0
98153000-98154000 ---p 98153000 00:00 0
98154000-98997000 rw-p 98154000 00:00 0
98997000-b799a000 rw-p 98997000 00:00 0
b799a000-b799b000 rw-p b799a000 00:00 0
b799b000-b79a5000 r-xp 00000000 08:02 2345666
/lib/i686/cmov/libnss_files-2.9.so
b79a5000-b79a6000 r--p 00009000 08:02 2345666
/lib/i686/cmov/libnss_files-2.9.so
b79a6000-b79a7000 rw-p 0000a000 08:02 2345666
/lib/i686/cmov/libnss_files-2.9.so
b79a7000-b79a9000 rw-p b79a7000 00:00 0
b79a9000-b79ab000 r-xp 00000000 08:02 2312902    /lib/libx86.so.1
b79ab000-b79ac000 rw-p 00001000 08:02 2312902    /lib/libx86.so.1
b79ac000-b79fd000 r-xp 00000000 08:02 465563     /usr/lib/libvga.so.1.4.3
b79fd000-b7a04000 rw-p 00050000 08:02 465563     /usr/lib/libvga.so.1.4.3
b7a04000-b7a0e000 rw-p b7a04000 00:00 0
b7a0e000-b7a24000 r-xp 00000000 08:02 1938461
/usr/lib/libdirect-1.2.so.0.7.0
b7a24000-b7a25000 rw-p 00016000 08:02 1938461
/usr/lib/libdirect-1.2.so.0.7.0
b7a25000-b7a2d000 r-xp 00000000 08:02 1938463
/usr/lib/libfusion-1.2.so.0.7.0
b7a2d000-b7a2e000 rw-p 00007000 08:02 1938463
/usr/lib/libfusion-1.2.so.0.7.0
b7a2e000-b7aa4000 r-xp 00000000 08:02 1938460
/usr/lib/libdirectfb-1.2.so.0.7.0
b7aa4000-b7aa7000 rw-p 00075000 08:02 1938460
/usr/lib/libdirectfb-1.2.so.0.7.0
b7aa7000-b7aa9000 r-xp 00000000 08:02 2345676    /lib/i686/cmov/libdl-2.9.so
b7aa9000-b7aaa000 r--p 00001000 08:02 2345676    /lib/i686/cmov/libdl-2.9.so
b7aaa000-b7aab000 rw-p 00002000 08:02 2345676    /lib/i686/cmov/libdl-2.9.so
b7aab000-b7b6f000 r-xp 00000000 08:02 1938445    /usr/lib/libasound.so.2.0.0
b7b6f000-b7b73000 rw-p 000c4000 08:02 1938445    /usr/lib/libasound.so.2.0.0
b7b73000-b7b74000 rw-p b7b73000 00:00 0
b7b74000-b7b77000 r-xp 00000000 08:02 1938628
/usr/lib/libgpg-error.so.0.4.0
b7b77000-b7b78000 rw-p 00002000 08:02 1938628
/usr/lib/libgpg-error.so.0.4.0
b7b78000-b7beb000 r-xp 00000000 08:02 457003
/usr/lib/libgcrypt.so.11.5.2
b7beb000-b7bee000 rw-p 00072000 08:02 457003
/usr/lib/libgcrypt.so.11.5.2
b7bee000-b7bfd000 r-xp 00000000 08:02 457010     /usr/lib/libtasn1.so.3.1.2
b7bfd000-b7bfe000 rw-p 0000e000 08:02 457010     /usr/lib/libtasn1.so.3.1.2
b7bfe000-b7d58000 r-xp 00000000 08:02 2345649    /lib/i686/cmov/libc-2.9.so
b7d58000-b7d59000 ---p 0015a000 08:02 2345649    /lib/i686/cmov/libc-2.9.so
b7d59000-b7d5b000 r--p 0015a000 08:02 2345649    /lib/i686/cmov/libc-2.9.so
b7d5b000-b7d5c000 rw-p 0015c000 08:02 2345649    /lib/i686/cmov/libc-2.9.so
b7d5c000-b7d5f000 rw-p b7d5c000 00:00 0
b7d5f000-b7d63000 r-xp 00000000 08:02 1938669
/usr/lib/libvdeplug.so.2.1.0
b7d63000-b7d64000 rw-p 00003000 08:02 1938669
/usr/lib/libvdeplug.so.2.1.0
b7d64000-b7d94000 r-xp 00000000 08:02 2317221    /lib/libncurses.so.5.7
b7d94000-b7d97000 rw-p 0002f000 08:02 2317221    /lib/libncurses.so.5.7
b7d97000-b7d98000 rw-p b7d97000 00:00 0
b7d98000-b7e02000 r-xp 00000000 08:02 1938526
/usr/lib/libSDL-1.2.so.0.11.2
b7e02000-b7e04000 rw-p 00069000 08:02 1938526
/usr/lib/libSDL-1.2.so.0.11.2
b7e04000-b7e4f000 rw-p b7e04000 00:00 0
b7e4f000-b7e51000 r-xp 00000000 08:02 2345670
/lib/i686/cmov/libutil-2.9.so
b7e51000-b7e52000 r--p 00001000 08:02 2345670
/lib/i686/cmov/libutil-2.9.so
b7e52000-b7e53000 rw-p 00002000 08:02 2345670
/lib/i686/cmov/libutil-2.9.so
b7e53000-b7e5a000 r-xp 00000000 08:02 2345671    /lib/i686/cmov/librt-2.9.so
b7e5a000-b7e5b000 r--p 00006000 08:02 2345671    /lib/i686/cmov/librt-2.9.so
b7e5b000-b7e5c000 rw-p 00007000 08:02 2345671    /lib/i686/cmov/librt-2.9.so
b7e5c000-b7e71000 r-xp 00000000 08:02 2345674
/lib/i686/cmov/libpthread-2.9.so
b7e71000-b7e72000 r--p 00014000 08:02 2345674
/lib/i686/cmov/libpthread-2.9.so
b7e72000-b7e73000 rw-p 00015000 08:02 2345674
/lib/i686/cmov/libpthread-2.9.so
b7e73000-b7e75000 rw-p b7e73000 00:00 0
b7e75000-b7f0d000 r-xp 00000000 08:02 458215
/usr/lib/libgnutls.so.26.11.5
b7f0d000-b7f13000 rw-p 00097000 08:02 458215
/usr/lib/libgnutls.so.26.11.5
b7f13000-b7f14000 rw-p b7f13000 00:00 0
b7f14000-b7f28000 r-xp 00000000 08:02 466046     /usr/lib/libz.so.1.2.3.3
b7f28000-b7f29000 rw-p 00013000 08:02 466046     /usr/lib/libz.so.1.2.3.3
b7f29000-b7f4d000 r-xp 00000000 08:02 2345655    /lib/i686/cmov/libm-2.9.so
b7f4d000-b7f4e000 r--p 00023000 08:02 2345655    /lib/i686/cmov/libm-2.9.so
b7f4e000-b7f4f000 rw-p 00024000 08:02 2345655    /lib/i686/cmov/libm-2.9.so
b7f4f000-b7f53000 r-xp 00000000 08:02 464977     /usr/lib/libXfixes.so.3.1.0
b7f53000-b7f54000 rw-p 00003000 08:02 464977     /usr/lib/libXfixes.so.3.1.0
b7f54000-b7f58000 r-xp 00000000 08:02 462851     /usr/lib/libXdmcp.so.6.0.0
b7f58000-b7f59000 rw-p 00003000 08:02 462851     /usr/lib/libXdmcp.so.6.0.0
b7f59000-b7f5b000 r-xp 00000000 08:02 461523     /usr/lib/libXau.so.6.0.0
b7f5b000-b7f5c000 rw-p 00001000 08:02 461523     /usr/lib/libXau.so.6.0.0
b7f5c000-b7f5f000 rw-s 00000000 00:07 13         anon_inode:kvm-vcpu
b7f5f000-b7f61000 rw-p b7f5f000 00:00 0
b7f61000-b7f62000 r-xp b7f61000 00:00 0          [vdso]
b7f62000-b7f7e000 r-xp 00000000 08:02 2316512    /lib/ld-2.9.so
b7f7e000-b7f7f000 r--p 0001b000 08:02 2316512    /lib/ld-2.9.so
b7f7f000-b7f80000 rw-p 0001c000 08:02 2316512    /lib/ld-2.9.so
bfe6b000-bfe80000 rw-p bffeb000 00:00 0          [stack]
Aborted
# echo $?
134

The local KVM was compiled with default settings:

# ./configure
Install prefix    /usr/local
BIOS directory    /usr/local/share/qemu
binary directory  /usr/local/bin
Manual directory  /usr/local/share/man
ELF interp prefix /usr/gnemul/qemu-%M
Source path       /home/mike/ba/kvm-84/qemu
C compiler        gcc
Host C compiler   gcc
ARCH_CFLAGS       -m32
make              make
install           install
host CPU          i386
host big endian   no
target list       x86_64-softmmu
gprof enabled     no
sparse enabled    no
profiler          no
static build      no
-Werror enabled   no
SDL support       yes
SDL static link   yes
curses support    yes
mingw32 support   no
Audio drivers     oss
Extra audio cards ac97 es1370 sb16
Mixer emulation   no
VNC TLS support   yes
    TLS CFLAGS
    TLS LIBS      -lgnutls
kqemu support     no
kvm support       yes
CPU emulation     yes
brlapi support    no
Documentation     no
NPTL support      yes
vde support       yes
AIO support       yes
Install blobs     yes
KVM support       yes
fdt support       no

What bothers me a bit is this "target list x86_64-softmmu". The thing
is supposed to have NPT, why need softmmu?! And why x86_64?

Hi Mihail,

Yes, the binary is called "qemu-system-x86_64". The story behind all this is somewhere in the archive of this ML. This will probably go away when qemu and kvm-userspace merge some day.

I guess filing a bug on the kvm sourceforge bug tracker is the best thing to do here.

You need to load the module with nested=1 (you did that) to enable nested virtualization support in the kvm kernel module. Now the second thing you need to do is pass -enable-nesting to qemu-system-x86_64 in order to enable nesting support in the userspace part too.

I agree that this is not exactly obvious, but I wanted to guard the code as heavily as possible :-).

Also, keep in mind that for now only KVM in KVM works for me. Getting for example Xen running should be definitely doable - it just didn't work for me last time I tried. Speed is not exactly great yet either.

Please tell me how it works for you!

Thanks,

Alex

Hi Alex,

Yeah, I had already checked that out before I posted, but what I found
only seems to clarify the difference between qemu-<arch> and qemu-system-<arch>. What baffles me most is why the heck does it generate a x86_64 target on a i686 system?!

Or has KVM dropped support for 32-bit x86 as a guest platform
altogether, relying on x86_64's legacy/compatibility mode? That would make sense, but on the other hand it could mislead into thinking that you can run a 64-bit guest on it, which wouldn't work when the host is actually 32-bit (like in my case). Or do I miss something here?

I actually thought that had already happened, after I read the
changelog for kvm-84. Now I see that I must have interpreted "merge qemu-svn" the other way round as it was meant :-)

Makes sense. Done.

Yeah, later I stumbled across your posting with the patches for that
feature, and it says the same there. Sorry for "asking before reading".

Guard it? Is that still experimental? I thought this was enabled by
default...

Actually it would have been more obvious if there was a manpage
documenting that. However, my build did not produce any manpages, although the configure script said "Manual directory /usr/local/share/man".

Even if so, I think that needing to set the module parameter
explicitly is enough of a safeguard. Apropos parameter, I'd rather name it something more self-explanatory like "nested_virt" or "nested_svm" or something. Most people, including me, usually associate "nested" with NPT, when it comes to virtualization.

And while we are at parameters, kvm-intel.ko's enable/disable
parameters are bool, which actually makes more sense. I think it wouldn't hurt to make that consistent across modules, i.e. either turn amd's to bool, or intel's to int.

After I "discovered" the -enable-nesting thing, I tried it and it
worked fine. More precisely, I was able to start another guest within the guest, and it ran OK in text mode. As soon as it booted gdm, it hanged. I used the cirrus vga for the nested guest. The (outer) guest was a standard Debian Lenny, so the nested guest used the kvm version shipped with that, which is kvm-72. That actually shouldn't matter, but who knows...

At that point, I somehow managed to shoot off the sshd, since I was
doing that over SSH to my university machine. Now the remote machine seems to run, but the SSH daemon is down. Thus, I cannot reconnect to make further tests. Due to Easter holidays, I have no physical access to the machine right now, so I will be able to follow up on the matter from Tuesday onwards.

Btw, SDL performance over SSH X-forwarding just plain sucks, even on a
100MBit network. VNC is a bit better, but there I have an issue with a "double mouse pointer", having the local one as a black dot and the remote one the usual way, and they get terribly out of sync all the time. It's an enormous PITA! Do you experience that too? It happens here no matter how I set the "render mouse pointer locally" setting in xvnc.

Sure, as soon as I get my hands back on my university machine.

And until then, Happy Easter :-)

Greets,

Mike

Try specifying '-usbdevice tablet' in the KVM command line. Regular mice report relative coordinates, and that supposedly messes up with VNC. Tablet devices report absolute coordinates so they work well with VNC.

Yeah, I read that too and tried it out, but it somehow brought no
change. The exact same behavior as before. But I'll try it out again on Tuesday.

Besides, what's the problem with relative coordinates? When I connect
to a normal X session via VNC, they're relative too, and I have no problems. There's actually no black dot per default, I must enable it explicitly, and even then the remote mouse pointer moves quite simultaneously with the dot (maybe 50-100 ms lag?). With KVM, I cannot disable the dot at all (which would itself be already a big improvement).

Thanks for your hints,

Mike

Please don't question the usefulness of the decision. It is called "qemu-system-x86_64" for both i386 and x86_64 alike.

kvm-userspace merges in upstream qemu changes from time to time. Also some developers try to push their changes in kvm-userspace to upstream qemu. The process is not finished yet though, as you still have 2 distinct trees.

Yes, it's experimental.

I was imagining a world where you would always have the feature enabled and choose its activation via -cpu. Per default -cpu is set to "compatible" - a CPU that can be migrated even across intel and AMD platforms. If you choose -cpu barcelona or so you get SVM features in the guest.

For now it's the way you see it though and will be until I find time to make the code work perfectly ;-).

Yep - there was a discussion on qemu-devel about this.

This is because you have several conversion layers here. Normal VNC just sets the mouse pointer on the remote display. With a VM you emulate a normal mouse though, that only knows things like "move left by 20". That means you're stuck with a relative input device, which will always bring you the "double mouse cursor" effect.

If you want to have an absolute input device, try to use "vmmouse" instead of "mouse" as input driver in your xorg.conf.

PS: I'm offline for 2 weeks now, so don't expect any reply for me in the meantime :-).

Alex