Skip to contentSkip to conversation

3.13 - Nested KVM (vmx) totally broken?

fedora-virt Ian Pilcher 5 participants 12 messages Mar 4, 2014 – Mar 14, 2014 KVM

lore
Share & tools
lore
#0 lore raw

Is this a known problem? I just tried using nested vmx for the first time since upgrading my system from F19 (3.12.?? at the time) to F20, and I cannot start any L2 guests. The L2 guest appears to hang almost immediately after starting, consuming 100% of one of the L1 guest's VCPUs.

If I reboot with kernel-3.12.10-300.fc20.x86_64, the problem does not occur.

Any known workaround? (Other than using 3.12.10?)

Thanks!

Signature · 9 lines
========================================================================
Ian Pilcher                                         [email protected]
           Sent from the cloud -- where it's already tomorrow
========================================================================

_______________________________________________
virt mailing list
[email protected]
https://admin.fedoraproject.org/mailman/listinfo/virt
maintainer
#1 lore raw

Il 04/03/2014 03:40, Ian Pilcher ha scritto:

Quoted 10 lines

Is this a known problem? I just tried using nested vmx for the first time since upgrading my system from F19 (3.12.?? at the time) to F20, and I cannot start any L2 guests. The L2 guest appears to hang almost immediately after starting, consuming 100% of one of the L1 guest's VCPUs.

If I reboot with kernel-3.12.10-300.fc20.x86_64, the problem does not occur.

Any known workaround? (Other than using 3.12.10?)

There is a fix on the way to the 3.13 kernel.

You can open a Fedora bug and ask them to include http://article.gmane.org/gmane.linux.kernel.stable/82043/raw in the kernel.

Paolo
_______________________________________________
virt mailing list
[email protected]
https://admin.fedoraproject.org/mailman/listinfo/virt

#2 lore raw
Quoted 11 lines · Paolo Bonzini

Il 04/03/2014 03:40, Ian Pilcher ha scritto:

Quoted 10 lines

Is this a known problem? I just tried using nested vmx for the first time since upgrading my system from F19 (3.12.?? at the time) to F20, and I cannot start any L2 guests. The L2 guest appears to hang almost immediately after starting, consuming 100% of one of the L1 guest's VCPUs.

If I reboot with kernel-3.12.10-300.fc20.x86_64, the problem does not occur.

Any known workaround? (Other than using 3.12.10?)

If you want to try, I made a Fedora Kernel scratch build (i.e. not official) with fix Paolo pointed to below and this works for me:

http://koji.fedoraproject.org/koji/taskinfo?taskID=6577700

(NOTE: Fedora Scratch build URLs won't last more than 10 days or so)

Quoted 12 lines

There is a fix on the way to the 3.13 kernel.

You can open a Fedora bug and ask them to include
http://article.gmane.org/gmane.linux.kernel.stable/82043/raw in the kernel.

Paolo
--
To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Signature · 1 line
/kashyap
#3 lore raw
Quoted 10 lines · Kashyap Chamarthy
Quoted 9 lines · Paolo Bonzini

Il 04/03/2014 03:40, Ian Pilcher ha scritto:

Quoted 8 lines

Is this a known problem? I just tried using nested vmx for the first time since upgrading my system from F19 (3.12.?? at the time) to F20, and I cannot start any L2 guests. The L2 guest appears to hang almost immediately after starting, consuming 100% of one of the L1 guest's VCPUs.

If I reboot with kernel-3.12.10-300.fc20.x86_64, the problem does not occur.

Err, I missed to read this. Sorry about that.

Quoted 25 lines
Quoted 2 lines
Quoted 2 lines

Any known workaround? (Other than using 3.12.10?)

If you want to try, I made a Fedora Kernel scratch build (i.e. not official) with fix Paolo pointed to below and this works for me:

http://koji.fedoraproject.org/koji/taskinfo?taskID=6577700

(NOTE: Fedora Scratch build URLs won't last more than 10 days or so)

Quoted 12 lines

There is a fix on the way to the 3.13 kernel.

You can open a Fedora bug and ask them to include
http://article.gmane.org/gmane.linux.kernel.stable/82043/raw in the kernel.

Paolo
--
To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Signature · 1 line
/kashyap
Signature · 1 line
/kashyap
#4 lore raw
In reply to Kashyap Chamarthy
Quoted 4 lines · Kashyap Chamarthy

If you want to try, I made a Fedora Kernel scratch build (i.e. not official) with fix Paolo pointed to below and this works for me:

http://koji.fedoraproject.org/koji/taskinfo?taskID=6577700

Works here. Thanks!

Signature · 4 lines
========================================================================
Ian Pilcher                                         [email protected]
           Sent from the cloud -- where it's already tomorrow
========================================================================
#5 lore raw
In reply to Paolo Bonzini
Quoted 10 lines · Paolo Bonzini

Il 04/03/2014 03:40, Ian Pilcher ha scritto:

Quoted 9 lines

Is this a known problem? I just tried using nested vmx for the first time since upgrading my system from F19 (3.12.?? at the time) to F20, and I cannot start any L2 guests. The L2 guest appears to hang almost immediately after starting, consuming 100% of one of the L1 guest's VCPUs.

If I reboot with kernel-3.12.10-300.fc20.x86_64, the problem does not occur.

Quoted 7 lines
Quoted 1 line

Any known workaround? (Other than using 3.12.10?)

There is a fix on the way to the 3.13 kernel.

You can open a Fedora bug and ask them to include
http://article.gmane.org/gmane.linux.kernel.stable/82043/raw in the kernel.

Thanks for fixing this. It affects a lot of libguestfs users too.

I opened this bug:

https://bugzilla.redhat.com/show_bug.cgi?id=1073663

Rich.

Signature · 8 lines
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
virt-top is 'top' for virtual machines.  Tiny program with many
powerful monitoring features, net stats, disk stats, logging, etc.
http://people.redhat.com/~rjones/virt-top
_______________________________________________
virt mailing list
[email protected]
https://admin.fedoraproject.org/mailman/listinfo/virt
#6 lore raw

2014-03-07 1:59 GMT+04:00 Richard W.M. Jones <[email protected]>:

Quoted 19 lines · Paolo Bonzini
Quoted 10 lines

Il 04/03/2014 03:40, Ian Pilcher ha scritto:

Quoted 9 lines

Is this a known problem? I just tried using nested vmx for the first time since upgrading my system from F19 (3.12.?? at the time) to F20, and I cannot start any L2 guests. The L2 guest appears to hang almost immediately after starting, consuming 100% of one of the L1 guest's VCPUs.

If I reboot with kernel-3.12.10-300.fc20.x86_64, the problem does not occur.

Quoted 7 lines
Quoted 1 line

Any known workaround? (Other than using 3.12.10?)

There is a fix on the way to the 3.13 kernel.

You can open a Fedora bug and ask them to include
http://article.gmane.org/gmane.linux.kernel.stable/82043/raw in the kernel.

Quoted 7 lines

Thanks for fixing this. It affects a lot of libguestfs users too.

I opened this bug:

https://bugzilla.redhat.com/show_bug.cgi?id=1073663

Rich.

If i use 3.13.6 kernel that have alredy this patch, but sometimes i get kernel panic, what can i do?
P.S. I'm using nested virt, fault from L2

[   10.942007] PANIC: double fault, error_code: 0x0
[   10.942007] CPU: 0 PID: 182 Comm: systemd-journal Not tainted 3.13.6 #3
[   10.942007] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011
[   10.942007] task: ffff88001cc08000 ti: ffff88001d70e000 task.ti:
ffff88001d70e000
[   10.942007] RIP: 0033:[<00007fe61b2fce8a>]  [<00007fe61b2fce8a>]
0x7fe61b2fce8a
[   10.942007] RSP: 002b:00007fffee7468d8  EFLAGS: 00010286
[   10.942007] RAX: 0000000000000000 RBX: 000000000043344e RCX: 0000000000430a70
[   10.942007] RDX: 0000000000000010 RSI: 00000000ffffffff RDI: 0000000000430a70
[   10.942007] RBP: 00007fffee747130 R08: 0000000000000003 R09: 00007fe61be81780
[   10.942007] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000001
[   10.942007] R13: 0000000001c9c380 R14: 0000000000000003 R15: 00007fffee747148
[   10.942007] FS:  00007fe61be81780(0000) GS:ffff88001f800000(0000)
knlGS:0000000000000000
[   10.942007] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   10.942007] CR2: 0000000000000000 CR3: 000000001e1a2000 CR4: 00000000000006f0
[   10.942007]
[   10.942007] Kernel panic - not syncing: Machine halted.
Signature · 3 lines
Vasiliy Tolstov,
e-mail: [email protected]
jabber: [email protected]
#7 lore raw
Quoted 3 lines · Vasiliy Tolstov

If i use 3.13.6 kernel that have alredy this patch, but sometimes i get kernel panic, what can i do?
P.S. I'm using nested virt, fault from L2

It could be there is another, less frequent, bug in nested KVM. I'm assuming this is on Intel hardware?

From the libguestfs point of view what you can do is to force TCG:

export LIBGUESTFS_BACKEND_SETTINGS=force_tcg

Unfortunately this only has an effect in libguestfs >= 1.25.24. We're going to have the new version in Fedora 20 real soon -- probably before the end of this month. Or you can compile the Rawhide version on F20.

Rich.

Signature · 8 lines
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
virt-df lists disk usage of guests without needing to install any
software inside the virtual machine.  Supports Linux and Windows.
http://people.redhat.com/~rjones/virt-df/
_______________________________________________
virt mailing list
[email protected]
https://admin.fedoraproject.org/mailman/listinfo/virt
#8 lore raw

2014-03-14 15:58 GMT+04:00 Richard W.M. Jones <[email protected]>:

Quoted 11 lines

It could be there is another, less frequent, bug in nested KVM. I'm assuming this is on Intel hardware?

From the libguestfs point of view what you can do is to force TCG:

export LIBGUESTFS_BACKEND_SETTINGS=force_tcg

Unfortunately this only has an effect in libguestfs >= 1.25.24. We're going to have the new version in Fedora 20 real soon -- probably before the end of this month. Or you can compile the Rawhide version on F20.

Thanks for answer. I'm not using libguestfs. I'm try tun vm inside vm via libvirt.
is that possible to debug this issue ? How can i help? P.S. Yes i'm using intel hardware.

Signature · 3 lines
Vasiliy Tolstov,
e-mail: [email protected]
jabber: [email protected]
#9 lore raw
Quoted 8 lines · Vasiliy Tolstov

2014-03-14 15:58 GMT+04:00 Richard W.M. Jones <[email protected]>:

Quoted 7 lines

It could be there is another, less frequent, bug in nested KVM. I'm assuming this is on Intel hardware?

From the libguestfs point of view what you can do is to force TCG:

export LIBGUESTFS_BACKEND_SETTINGS=force_tcg

Quoted 8 lines
Quoted 4 lines

Unfortunately this only has an effect in libguestfs >= 1.25.24. We're going to have the new version in Fedora 20 real soon -- probably before the end of this month. Or you can compile the Rawhide version on F20.

Thanks for answer. I'm not using libguestfs. I'm try tun vm inside vm via libvirt.

You can set the VM <domain type="qemu">. Of course it'll run quite slowly.

Quoted 1 line

is that possible to debug this issue ? How can i help?

Complete logs from the guest.
Any messages from qemu or the host.
& put all of that into a full bug report.

Rich.

Signature · 7 lines
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
libguestfs lets you edit virtual machines.  Supports shell scripting,
bindings from many languages.  http://libguestfs.org
_______________________________________________
virt mailing list
[email protected]
https://admin.fedoraproject.org/mailman/listinfo/virt
#10 lore raw

2014-03-14 16:16 GMT+04:00 Richard W.M. Jones <[email protected]>:

Quoted 8 lines

You can set the VM <domain type="qemu">. Of course it'll run quite slowly.

Quoted 1 line

is that possible to debug this issue ? How can i help?

Complete logs from the guest.
Any messages from qemu or the host.
& put all of that into a full bug report.

Where i can find submission form for bug report? (I'm using exherbo linux, but it does not have like debian or sles personal patches and using only upstream)

Signature · 3 lines
Vasiliy Tolstov,
e-mail: [email protected]
jabber: [email protected]
#11 lore raw
Quoted 6 lines · Vasiliy Tolstov

2014-03-14 16:16 GMT+04:00 Richard W.M. Jones <[email protected]>:

Quoted 5 lines

You can set the VM <domain type="qemu">. Of course it'll run quite slowly.

Quoted 1 line

is that possible to debug this issue ? How can i help?

Quoted 8 lines
Quoted 3 lines

Complete logs from the guest.
Any messages from qemu or the host.
& put all of that into a full bug report.

Where i can find submission form for bug report? (I'm using exherbo linux, but it does not have like debian or sles personal patches and using only upstream)

I suspect this is going to be a kernel bug, in which case:

https://bugzilla.kernel.org/

For libvirt bugs it would be:

https://bugzilla.redhat.com/enter_bug.cgi?component=libvirt&product=Virtualization+Tools

Rich.

Signature · 8 lines
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
virt-top is 'top' for virtual machines.  Tiny program with many
powerful monitoring features, net stats, disk stats, logging, etc.
http://people.redhat.com/~rjones/virt-top
_______________________________________________
virt mailing list
[email protected]
https://admin.fedoraproject.org/mailman/listinfo/virt