Testing nested virtualization

DISCUSSION Daniel Bareiro 4 participants 13 messages Jan 1, 2010 – Jan 4, 2010 KVM

lore

Share & tools

lore

Daniel Bareiro

author

2010-01-01 20:20 UTC #0 lore raw

Hi, all!

I'm testing nested virtualization with Linux 2.6.32.2 and qemu-kvm-0.12.1.1 in the host.

I was looking for if in the KVM official site there is something documented, but I didn't find information so any reference will be welcome.

What I got to do until the moment is to load the module in host of the following way:

# modprobe kvm_amd nested=1

Then, to boot guest with the parameter -enable-nesting:

# /usr/local/qemu-kvm/bin/qemu-system-x86_64 -hda /dev/vm/test-disk -m \
  1024 -boot c -net nic,vlan=0,macaddr=00:16:3e:00:00:34 -net tap \
  -daemonize -vnc :3 -k es -localtime -serial \
  telnet:localhost:5000,server,nowait -monitor \
  telnet:localhost:5001,server,nowait -enable-nesting

In this guest I'm using Linux 2.6.32, with userspace kvm-85 of backports. For the VM booted within of 'test', I am using a qcow2 file. But when connecting via VNC, the boot process is hung in "Press for F12 boot menu". Sometimes it gets to pass this instance and I see the Debian install menu, but while I cross the options, it is hung.

With these options I boot the guest within test:

# kvm -hda disk01.img -m 512 -boot d -cdrom debian-500-amd64-netinst.iso \
  -net nic,vlan=0,macaddr=00:16:3e:00:00:33 -net tap -daemonize -vnc :2 \
  -k es -localtime

Which can be the problem?

Thanks in advance for your replies.

Regards,
Daniel

Signature · 2 lines

Fingerprint: BFB3 08D6 B4D1 31B2 72B9  29CE 6696 BF1B 14E6 1D37
Powered by Debian GNU/Linux Lenny - Linux user #188.598

Daniel Bareiro

author

2010-01-01 20:56 UTC #1 lore raw

Quoted 5 lines · Daniel Bareiro

In this guest I'm using Linux 2.6.32, with userspace kvm-85 of backports. For the VM booted within of 'test', I am using a qcow2 file. But when connecting via VNC, the boot process is hung in "Press for F12 boot menu". Sometimes it gets to pass this instance and I see the Debian install menu, but while I cross the options, it is hung.

Also I was trying to use qemu-kvm-0.12.1.1 with to Linux 2.6.32 in guest within 'test'. And here it happens something similar. Sometimes I get to select the option of the menu of the installer, but after boot, the installation is hung again.

Regards,
Daniel

Signature · 2 lines

Fingerprint: BFB3 08D6 B4D1 31B2 72B9  29CE 6696 BF1B 14E6 1D37
Powered by Debian GNU/Linux Lenny - Linux user #188.598

Alexander Graf

2010-01-02 13:11 UTC #2 lore raw

Quoted 9 lines · Daniel Bareiro

Quoted 5 lines · Daniel Bareiro
In this guest I'm using Linux 2.6.32, with userspace kvm-85 of backports. For the VM booted within of 'test', I am using a qcow2 file. But when connecting via VNC, the boot process is hung in "Press for F12 boot menu". Sometimes it gets to pass this instance and I see the Debian install menu, but while I cross the options, it is hung.

Quoted 4 lines

Also I was trying to use qemu-kvm-0.12.1.1 with to Linux 2.6.32 in guest within 'test'. And here it happens something similar. Sometimes I get to select the option of the menu of the installer, but after boot, the installation is hung again.

Can your host do nested paging? Shadow shadow paging can be a bit flaky at times :-(.

Alex

Daniel Bareiro

author

2010-01-02 13:45 UTC #3 lore raw

Hi, Alexander.

Quoted 7 lines · Alexander Graf

Quoted 6 lines
Quoted 6 lines
In this guest I'm using Linux 2.6.32, with userspace kvm-85 of backports. For the VM booted within of 'test', I am using a qcow2 file. But when connecting via VNC, the boot process is hung in "Press for F12 boot menu". Sometimes it gets to pass this instance and I see the Debian install menu, but while I cross the options, it is hung.

Quoted 7 lines

Quoted 4 lines
Also I was trying to use qemu-kvm-0.12.1.1 with to Linux 2.6.32 in guest within 'test'. And here it happens something similar. Sometimes I get to select the option of the menu of the installer, but after boot, the installation is hung again.
Can your host do nested paging? Shadow shadow paging can be a bit flaky at times :-(.

Hmmm... is there some way to verify that?

The CPUs model is AMD Athlon 64 X2 Dual Core 3800+. Perhaps some flag related? In such case, these are the available ones in the host:

flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge
mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt rdtscp lm 3dnowext 3dnow rep_good pni cx16 lahf_lm cmp_legacy svm extapic cr8_legacy

When loading the module with "kvm_amd nested=1", /var/log/messages registers the following thing:

kvm: Nested Virtualization enabled

And these are flags in test, booting with -enable-nesting:

flags : fpu de pse tsc msr pae mce cx8 apic sep mtrr pge mca
cmov pat pse36 clflush mmx fxsr sse sse2 syscall nx lm up pni cx16 hypervisor lahf_lm svm

Thanks for your reply.

Regards,
Daniel

Signature · 2 lines

Fingerprint: BFB3 08D6 B4D1 31B2 72B9  29CE 6696 BF1B 14E6 1D37
Powered by Debian GNU/Linux Lenny - Linux user #188.598

Alexander Graf

2010-01-02 13:48 UTC #4 lore raw

Quoted 31 lines · Daniel Bareiro

Hi, Alexander.
Quoted 7 lines · Alexander Graf
Quoted 6 lines
Quoted 6 lines
In this guest I'm using Linux 2.6.32, with userspace kvm-85 of backports. For the VM booted within of 'test', I am using a qcow2 file. But when connecting via VNC, the boot process is hung in "Press for F12 boot menu". Sometimes it gets to pass this instance and I see the Debian install menu, but while I cross the options, it is hung.
Quoted 7 lines
Quoted 4 lines
Also I was trying to use qemu-kvm-0.12.1.1 with to Linux 2.6.32 in guest within 'test'. And here it happens something similar. Sometimes I get to select the option of the menu of the installer, but after boot, the installation is hung again.
Can your host do nested paging? Shadow shadow paging can be a bit flaky at times :-(.
Hmmm... is there some way to verify that?
The CPUs model is AMD Athlon 64 X2 Dual Core 3800+. Perhaps some flag related? In such case, these are the available ones in the host:
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge
mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt rdtscp lm 3dnowext 3dnow rep_good pni cx16 lahf_lm cmp_legacy svm extapic cr8_legacy

Quoted 4 lines

When loading the module with "kvm_amd nested=1", /var/log/messages registers the following thing:
kvm: Nested Virtualization enabled

With nested paging it would show something like this in dmesg:

kvm: Nested Paging enabled

All family 10 AMD CPUs have nested paging (read: quad-core and above).

So I guess you found something going wrong with shadow-shadow paging. CC'ing Jörg. He's more into memory management than me :-).

Alex

Alexander Graf

2010-01-02 13:49 UTC #5 lore raw

Quoted 39 lines · Alexander Graf

Quoted 31 lines · Daniel Bareiro
Hi, Alexander.
Quoted 7 lines · Alexander Graf
Quoted 6 lines
Quoted 6 lines
In this guest I'm using Linux 2.6.32, with userspace kvm-85 of backports. For the VM booted within of 'test', I am using a qcow2 file. But when connecting via VNC, the boot process is hung in "Press for F12 boot menu". Sometimes it gets to pass this instance and I see the Debian install menu, but while I cross the options, it is hung.
Quoted 7 lines
Quoted 4 lines
Also I was trying to use qemu-kvm-0.12.1.1 with to Linux 2.6.32 in guest within 'test'. And here it happens something similar. Sometimes I get to select the option of the menu of the installer, but after boot, the installation is hung again.
Can your host do nested paging? Shadow shadow paging can be a bit flaky at times :-(.
Hmmm... is there some way to verify that?
The CPUs model is AMD Athlon 64 X2 Dual Core 3800+. Perhaps some flag related? In such case, these are the available ones in the host:
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge
mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt rdtscp lm 3dnowext 3dnow rep_good pni cx16 lahf_lm cmp_legacy svm extapic cr8_legacy
Quoted 4 lines
When loading the module with "kvm_amd nested=1", /var/log/messages registers the following thing:
kvm: Nested Virtualization enabled

Quoted 8 lines

With nested paging it would show something like this in dmesg:
kvm: Nested Paging enabled
All family 10 AMD CPUs have nested paging (read: quad-core and above).
So I guess you found something going wrong with shadow-shadow paging. CC'ing Jörg. He's more into memory management than me :-).

Eh - CC'ing :-).

Alex

Joerg Roedel

2010-01-04 11:12 UTC #6 lore raw

Quoted 4 lines

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Also I was trying to use qemu-kvm-0.12.1.1 with to Linux 2.6.32 in guest within 'test'. And here it happens something similar. Sometimes I get to select the option of the menu of the installer, but after boot, the installation is hung again.

The problem has probably to do with the TSC bugs I fixed lately for nested SVM. You can try to disable KVM-Clock for the L1 and the L2 guest or use the latest 2.6.32.x kernel on the Host. Does one of this fix the issues for you?

Joerg

Daniel Bareiro

2010-01-04 14:21 UTC #7 lore raw

Hi, Joerg.

Quoted 9 lines · Joerg Roedel

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Also I was trying to use qemu-kvm-0.12.1.1 with to Linux 2.6.32 in guest within 'test'. And here it happens something similar. Sometimes I get to select the option of the menu of the installer, but after boot, the installation is hung again.
The problem has probably to do with the TSC bugs I fixed lately for nested SVM. You can try to disable KVM-Clock for the L1 and the L2 guest or use the latest 2.6.32.x kernel on the Host. Does one of this fix the issues for you?

I'm using Linux 2.6.32.2 with qemu-kvm-0.12.1.1 in the host, so I will try to disable KVM-Clock for L1 and L2 guest. How I can get it?

Alexander said in another mail that AMD family 10 have nested paging (read quad-core and above) but my processor is dual-core. The problem can be related to that?

Thanks for your reply.

Regards,
Daniel

Signature · 3 lines

Daniel Bareiro - GNU/Linux registered user #188.598
Proudly running Debian GNU/Linux with uptime:
10:52:11 up 19:37, 11 users,  load average: 0.12, 0.17, 0.11

Alexander Graf

2010-01-04 15:07 UTC #8 lore raw

Quoted 15 lines · Daniel Bareiro

Hi, Joerg.
Quoted 9 lines · Joerg Roedel
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Also I was trying to use qemu-kvm-0.12.1.1 with to Linux 2.6.32 in guest within 'test'. And here it happens something similar. Sometimes I get to select the option of the menu of the installer, but after boot, the installation is hung again.
The problem has probably to do with the TSC bugs I fixed lately for nested SVM. You can try to disable KVM-Clock for the L1 and the L2 guest or use the latest 2.6.32.x kernel on the Host. Does one of this fix the issues for you?

Quoted 6 lines

I'm using Linux 2.6.32.2 with qemu-kvm-0.12.1.1 in the host, so I will try to disable KVM-Clock for L1 and L2 guest. How I can get it?
Alexander said in another mail that AMD family 10 have nested paging (read quad-core and above) but my processor is dual-core. The problem can be related to that?

It just means you're using code paths that I usually don't use, since my main development machine can do nested paging. It should work nevertheless, just be horribly slow.

Alex

Daniel Bareiro

2010-01-04 15:52 UTC #9 lore raw

Hi, Alex.

Quoted 13 lines · Alexander Graf

Quoted 13 lines
Quoted 9 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Also I was trying to use qemu-kvm-0.12.1.1 with to Linux 2.6.32 in guest within 'test'. And here it happens something similar. Sometimes I get to select the option of the menu of the installer, but after boot, the installation is hung again.
The problem has probably to do with the TSC bugs I fixed lately for nested SVM. You can try to disable KVM-Clock for the L1 and the L2 guest or use the latest 2.6.32.x kernel on the Host. Does one of this fix the issues for you?
I'm using Linux 2.6.32.2 with qemu-kvm-0.12.1.1 in the host, so I will try to disable KVM-Clock for L1 and L2 guest. How I can get it?

Quoted 7 lines

Quoted 3 lines
Alexander said in another mail that AMD family 10 have nested paging (read quad-core and above) but my processor is dual-core. The problem can be related to that?
It just means you're using code paths that I usually don't use, since my main development machine can do nested paging. It should work nevertheless, just be horribly slow.

Then what it would seem it hangs can in fact be a horribly slow operation?

What it draws attention to me is that, as I said in another mail, this happens at different moments. Can this be related to TSC bug mentioned by Joerg? The 'Clocksource tsc unstable' in dmesg is due to this bug?

In 'test' host I've the following thing:

test:~# cat /sys/devices/system/clocksource/clocksource0/current_clocksource
kvm-clock

test:~# cat /sys/devices/system/clocksource/clocksource0/available_clocksource
kvm-clock hpet acpi_pm

Thanks for your reply.

Regards,
Daniel

Signature · 3 lines

Daniel Bareiro - GNU/Linux registered user #188.598
Proudly running Debian GNU/Linux with uptime:
12:21:25 up 21:06, 11 users,  load average: 0.08, 0.11, 0.09

Joerg Roedel

2010-01-04 16:05 UTC #10 lore raw

Hi Daniel,

Quoted 5 lines · Daniel Bareiro

test:~# cat /sys/devices/system/clocksource/clocksource0/current_clocksource
kvm-clock

test:~# cat /sys/devices/system/clocksource/clocksource0/available_clocksource
kvm-clock hpet acpi_pm

Can you try to boot L1 and L2 guest with the 'no-kvmclock' kernel parameter? This disables the kvm-clock.

Joerg

Daniel Bareiro

2010-01-04 23:57 UTC #11 lore raw

Quoted 1 line · Joerg Roedel

Hi Daniel,

Hi, Joerg.

Quoted 8 lines

Quoted 5 lines
test:~# cat /sys/devices/system/clocksource/clocksource0/current_clocksource
kvm-clock
test:~# cat /sys/devices/system/clocksource/clocksource0/available_clocksource
kvm-clock hpet acpi_pm
Can you try to boot L1 and L2 guest with the 'no-kvmclock' kernel parameter? This disables the kvm-clock.

Question: what is L1 and L2? Because it sounds to cache levels, but I suppose that against this background one will talk about another thing.

Beyond that, using no-kvmclock kernel parameter en 'test', now I have the following thing:

test:~# cat /sys/devices/system/clocksource/clocksource0/current_clocksource
hpet

test:~# cat /sys/devices/system/clocksource/clocksource0/available_clocksource
hpet acpi_pm

But the same happens. The Debian installer hangs in the initial screen.

Thanks for your reply.

Regards,
Daniel

Signature · 3 lines

Daniel Bareiro - GNU/Linux registered user #188.598
Proudly running Debian GNU/Linux with uptime:
20:43:21 up 1 day,  5:28, 11 users,  load average: 0.21, 0.20, 0.12

Daniel Bareiro

author

2010-01-02 14:18 UTC #12 lore raw

In reply to Alexander Graf

Hi, Alex.

Quoted 26 lines · Alexander Graf

Quoted 26 lines
Quoted 7 lines
Quoted 6 lines
Quoted 6 lines
In this guest I'm using Linux 2.6.32, with userspace kvm-85 of backports. For the VM booted within of 'test', I am using a qcow2 file. But when connecting via VNC, the boot process is hung in "Press for F12 boot menu". Sometimes it gets to pass this instance and I see the Debian install menu, but while I cross the options, it is hung.
Quoted 7 lines
Quoted 4 lines
Also I was trying to use qemu-kvm-0.12.1.1 with to Linux 2.6.32 in guest within 'test'. And here it happens something similar. Sometimes I get to select the option of the menu of the installer, but after boot, the installation is hung again.
Can your host do nested paging? Shadow shadow paging can be a bit flaky at times :-(.
Hmmm... is there some way to verify that?
The CPUs model is AMD Athlon 64 X2 Dual Core 3800+. Perhaps some flag related? In such case, these are the available ones in the host:
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge
mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt rdtscp lm 3dnowext 3dnow rep_good pni cx16 lahf_lm cmp_legacy svm extapic cr8_legacy

Quoted 8 lines

Quoted 4 lines
When loading the module with "kvm_amd nested=1", /var/log/messages registers the following thing:
kvm: Nested Virtualization enabled
With nested paging it would show something like this in dmesg:
kvm: Nested Paging enabled

Hmmm... dmesg shows just like /var/log/messages:

kvm: Nested Virtualization enabled

"Nested Paging enabled" would have to be a complementary message?

Quoted 1 line

All family 10 AMD CPUs have nested paging (read: quad-core and above).

But this model is dual-Core :-(. That would mean that it cannot do nested paging?

Quoted 2 lines

So I guess you found something going wrong with shadow-shadow paging. CC'ing Jörg. He's more into memory management than me :-).

We hope then that Jörg can illuminate a little more to us :-).

Thanks for your reply.

Regards,
Daniel

Signature · 2 lines

Fingerprint: BFB3 08D6 B4D1 31B2 72B9  29CE 6696 BF1B 14E6 1D37
Powered by Debian GNU/Linux Lenny - Linux user #188.598