New: nVMX: Support live migration of whole L1 guest
https://bugzilla.kernel.org/show_bug.cgi?id=53851
Summary: nVMX: Support live migration of whole L1 guest
Product: Virtualization
Version: unspecified
Platform: All
OS/Version: Linux
Tree: Mainline
Status: NEW
Severity: low
Priority: P1
Component: kvm
AssignedTo: [email protected]
ReportedBy: [email protected]
Regression: No
We need to support live migration of a guest, even if it is running a hypervisor with sub-guests.
I.e., this issue is about supporting live migration of a whole L1 guest hypervisor, together with its L2 guests.
One option is to add ioctls for saving and restoring a currently-running L2 guest's state (vmcs01, vcpu_vmx.nested, and perhaps more things?).
A different option is to force an exit from L2 to L1 during live migration. We can use a spurious external interrupt exit (hoping L1 will ignore it and immediately return to L2), or hide this exit from L1 - add ioctl for L2->L1 exit (if in L2) and another ioctl for reentry used on the receiving hypervisor (it will need to know to run it - i.e., is_guest_mode(vcpu) also needs to be migrated). Note how the exit from L2 to L1 includes writing all the L2's state from host memory (vmcs02) to guest memory (vmcs12), where it will be migrated together with the rest of the guest.
See discussion in http://www.mail-archive.com/[email protected]/msg54257.html
on comparing these two options, and how one approach should be chosen for both nVMX and nSVM.
Note that additionally, we need to also save/restore vmx->nested.vmxon (whether the migrated L1 guest ever did VMXON), and also vmx->nested.current_vmptr (last VMPTRLDed VMCS).
Signature · 3 lines
Configure bugmail: https://bugzilla.kernel.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching the assignee of the bug.
[Bug 53851] nVMX: Support live migration of whole L1 guest
https://bugzilla.kernel.org/show_bug.cgi?id=53851
Nadav Har'El <[email protected]> changed:
What |Removed |Added
----------------------------------------------------------------------------
Blocks| |53601
Signature · 3 lines
Configure bugmail: https://bugzilla.kernel.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching the assignee of the bug.
[Bug 53851] nVMX: Support live migration of whole L1 guest
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=53851
--- Comment #1 from Nadav Har'El <[email protected]> 2013-02-26 15:27:02 --- Another issue to consider for live migration: right now we only kunmap() and release_page_dirty() the vmcs12 page after a vmclear (or vmxof, etc.). However, we may need to do this also on exit to user space (KVM_VCPU_RUN ioctl completes), so that live migration knows this page is dirty and needs to be copied.
Signature · 3 lines
Configure bugmail: https://bugzilla.kernel.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching the assignee of the bug.
[Bug 53851] nVMX: Support live migration of whole L1 guest
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=53851
Bandan Das <[email protected]> changed:
What |Removed |Added
----------------------------------------------------------------------------
Blocks| |94971
Signature · 2 lines
You are receiving this mail because: You are watching the assignee of the bug.
[Bug 53851] nVMX: Support live migration of whole L1 guest
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=53851
Dennis ([email protected]) changed:
What |Removed |Added
----------------------------------------------------------------------------
Blocks| |198621
Referenced Bugs:
https://bugzilla.kernel.org/show_bug.cgi?id=198621
[Bug 198621] KVM Guest panics when running own guest, while being live migrated
Signature · 2 lines
You are receiving this mail because: You are watching the assignee of the bug.
[Bug 53851] nVMX: Support live migration of whole L1 guest
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=53851
--- Comment #2 from [email protected] --- Sorry for replying to this rather old bug - I was pointed to this via
https://www.linux-kvm.org/page/Nested_Guests#Limitations
If I may ask, is this really the last state of discussion and work on this issue?
Looking at i.e.
* https://github.com/qemu/qemu/commit/ebbfef2f34cfc749c045a4569dedb4f748ec024a
*
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=039aeb9deb9291f3b19c375a8bc6fa7f768996cc
there have been commits for the kernel as well as QEMU to support migration of nested VMs.
Signature · 4 lines
You may reply to this email to add a comment. You are receiving this mail because: You are watching the assignee of the bug.
Re: [Bug 53851] nVMX: Support live migration of whole L1 guest
Quoted 11 lines · On Wed, 2021-06-09 at 08:55 +0000, [email protected] wrote:
https://bugzilla.kernel.org/show_bug.cgi?id=53851
--- Comment #2 from [email protected] --- Sorry for replying to this rather old bug - I was pointed to this via
https://www.linux-kvm.org/page/Nested_Guests#LimitationsIf I may ask, is this really the last state of discussion and work on this issue?
Looking at i.e.
Quoted 8 lines
* https://github.com/qemu/qemu/commit/ebbfef2f34cfc749c045a4569dedb4f748ec024a
*
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=039aeb9deb9291f3b19c375a8bc6fa7f768996ccthere have been commits for the kernel as well as QEMU to support migration of nested VMs.
AFAIK, running nested guests and migration while nested guest is running should work on both Intel and AMD, but there were lots of fixes in this area recently so a very new kernel should be used.
Plus in some cases if the nested guest is 32 bit, the migration still can fail, on Intel at least, last time I checked. On AMD I just recently fixed such issue for 32 bit guest and it seems to work for me.
I also know that if the nested guest is hyper-v enabled (which is a bit overkill as this brings us to a double nesting), then it crashes once in a whileafter lots of migration cycles.
So there are still bugs, but overall it works.
Best regards,
Maxim Levitsky
[Bug 53851] nVMX: Support live migration of whole L1 guest
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=53851
--- Comment #3 from [email protected] ---
Quoted 15 lines · On Wed, 2021-06-09 at 08:55 +0000, [email protected] wrote:
https://bugzilla.kernel.org/show_bug.cgi?id=53851
--- Comment #2 from [email protected] --- Sorry for replying to this rather old bug - I was pointed to this via
https://www.linux-kvm.org/page/Nested_Guests#LimitationsIf I may ask, is this really the last state of discussion and work on this issue?
Looking at i.e.*
https://github.com/qemu/qemu/commit/ebbfef2f34cfc749c045a4569dedb4f748ec024a
*
Quoted 7 lines
there have been commits for the kernel as well as QEMU to support migration of
nested VMs.
AFAIK, running nested guests and migration while nested guest is running should work on both Intel and AMD, but there were lots of fixes in this area recently so a very new kernel should be used.
Plus in some cases if the nested guest is 32 bit, the migration still can fail, on Intel at least, last time I checked. On AMD I just recently fixed such issue for 32 bit guest and it seems to work for me.
I also know that if the nested guest is hyper-v enabled (which is a bit overkill as
this brings us to a double nesting), then it crashes once in a whileafter lots of migration
cycles.
So there are still bugs, but overall it works.
Best regards,
Maxim Levitsky
Signature · 4 lines
You may reply to this email to add a comment. You are receiving this mail because: You are watching the assignee of the bug.