Nested SVM and migration

Perhaps I am misunderstanding, but I don't see how nested SVM instances can be properly migrated. How does one extract and rebuild the nested hsave control block?

If it isn't done already, one possible way to add it as an extension might be to represent the data as additional MSRs which are saved and restored with migration.

Actually, looking deeper, there doesn't even appear to be any way to export the nested CPU data at all, meaning basic features like suspending and restoring the VM are not possible. Is there any plan to make it work in the near future? I'm not complaining; if my understanding is correct, this actually makes my current task easier.

Quoted 5 lines

Perhaps I am misunderstanding, but I don't see how nested SVM instances can be properly migrated. How does one extract and rebuild the nested hsave control block?

Migrating guests which run in nested mode could not be migrated in a save way currently but there are plans to fix that :-)
The first step is to save the l1 cpu state in the guest supplied hsave area. But that is not sufficient because this does not work for all l1 state.

Quoted 4 lines

If it isn't done already, one possible way to add it as an extension might be to represent the data as additional MSRs which are saved and restored with migration.

This sounds complicated.

Quoted 7 lines

Actually, looking deeper, there doesn't even appear to be any way to export the nested CPU data at all, meaning basic features like suspending and restoring the VM are not possible. Is there any plan to make it work in the near future? I'm not complaining; if my understanding is correct, this actually makes my current task easier.

I think we should introduce a flag to indicate userspace if a vcpu is in a state that could be migrated in a save way together with a way for userspace to request that the vcpu enters a migratable state. In the kernel we could do something like that:

nested_svm_vmrun(...)

{
	/* ... */
	kvm_migration_disable(vcpu);
	/* ... */
}

nested_svm_vmexit(...)

{
	/* ... */
	kvm_migration_enable(vcpu);
	/* ... */
}

and somewhere in the vcpu_run loop:

if (vcpu->arch.migration_win_req)

	nested_svm_vmexit(INTR);

This might be helpful in other situations too. Thoughts?

Quoted 27 lines · Joerg Roedel

Quoted 6 lines

Actually, looking deeper, there doesn't even appear to be any way to export the nested CPU data at all, meaning basic features like suspending and restoring the VM are not possible. Is there any plan to make it work in the near future? I'm not complaining; if my understanding is correct, this actually makes my current task easier.

I think we should introduce a flag to indicate userspace if a vcpu is in a state that could be migrated in a save way together with a way for userspace to request that the vcpu enters a migratable state. In the kernel we could do something like that:

nested_svm_vmrun(...)

{
	/* ... */
	kvm_migration_disable(vcpu);
	/* ... */
}

nested_svm_vmexit(...)

{
	/* ... */
	kvm_migration_enable(vcpu);
	/* ... */
}

and somewhere in the vcpu_run loop:

Quoted 4 lines

if (vcpu->arch.migration_win_req)

	nested_svm_vmexit(INTR);

This might be helpful in other situations too. Thoughts?

This doesn't work if the guest disables INTR intercepts, or if the guest checks that an interrupt was actually received. Of course no sane guest does this.

Quoted 33 lines · Avi Kivity

Quoted 31 lines · Joerg Roedel

Quoted 7 lines

Actually, looking deeper, there doesn't even appear to be any way to export the nested CPU data at all, meaning basic features like suspending and restoring the VM are not possible. Is there any plan to make it work in the near future? I'm not complaining; if my understanding is correct, this actually makes my current task easier.

I think we should introduce a flag to indicate userspace if a vcpu is in a state that could be migrated in a save way together with a way for userspace to request that the vcpu enters a migratable state. In the kernel we could do something like that:

nested_svm_vmrun(...)

{
	/* ... */
	kvm_migration_disable(vcpu);
	/* ... */
}

nested_svm_vmexit(...)

{
	/* ... */
	kvm_migration_enable(vcpu);
	/* ... */
}

and somewhere in the vcpu_run loop:

if (vcpu->arch.migration_win_req)

	nested_svm_vmexit(INTR);

Quoted 7 lines

Quoted 2 lines

This might be helpful in other situations too. Thoughts?

This doesn't work if the guest disables INTR intercepts, or if the guest checks that an interrupt was actually received. Of course no sane guest does this.

Malicious guest may do this on purpose just to prevent migration.

Relying on vcpu state controllable by a guest for migration is not a good idea.

The infrastructure is already there to import / export and migrate MSR settings. MSRs are also 64-bit, and hold "model-specific" settings, so if you don't mind thinking of the nested feature as a model-specific feature of the KVM-SVM CPU, it's even somewhat well defined in terms of the architecture.

Mostly the problem is figuring out what chunk of MSR space to use.

Quoted 24 lines · Joerg Roedel

I think we should introduce a flag to indicate userspace if a vcpu is in a state that could be migrated in a save way together with a way for userspace to request that the vcpu enters a migratable state. In the kernel we could do something like that:

nested_svm_vmrun(...)

{
	/* ... */
	kvm_migration_disable(vcpu);
	/* ... */
}

nested_svm_vmexit(...)

{
	/* ... */
	kvm_migration_enable(vcpu);
	/* ... */
}

and somewhere in the vcpu_run loop:

if (vcpu->arch.migration_win_req)

	nested_svm_vmexit(INTR);

Quoted 2 lines

This might be helpful in other situations too. Thoughts?

This doesn't work if the guest disables INTR intercepts, or if the guest checks that an interrupt was actually received. Of course no sane guest does this.

Quoted 6 lines · Zachary Amsden

The infrastructure is already there to import / export and migrate MSR settings. MSRs are also 64-bit, and hold "model-specific" settings, so if you don't mind thinking of the nested feature as a model-specific feature of the KVM-SVM CPU, it's even somewhat well defined in terms of the architecture.

There is a lot of additional state to migrate if the vcpu is running nested. To be architecturally correct you need to transfer 6kb of data through MSRs only for the msr permission bitmap.

The rest comes down to
the nested intercept masks

and some small bits like the global interrupt
flag and the nested vmcb address.

It is doable but I still think its
complicated to get this right. The simplest approach would be to disallow migration when the vcpu is running in guest mode.

Quoted 8 lines · Joerg Roedel

Quoted 6 lines · Zachary Amsden

The infrastructure is already there to import / export and migrate MSR settings. MSRs are also 64-bit, and hold "model-specific" settings, so if you don't mind thinking of the nested feature as a model-specific feature of the KVM-SVM CPU, it's even somewhat well defined in terms of the architecture.

Quoted 3 lines

There is a lot of additional state to migrate if the vcpu is running nested. To be architecturally correct you need to transfer 6kb of data through MSRs only for the msr permission bitmap.

The msr permission bitmap is in guest memory, so it is already migrated.

Quoted 2 lines

The rest comes down to
the nested intercept masks

These are in the vmcb, which is in guest memory.

Quoted 4 lines

It is doable but I still think its
complicated to get this right. The simplest approach would be to disallow migration when the vcpu is running in guest mode.

Agree, though I dislike the need to introduce a "force vmexit" ioctl.

Quoted 16 lines · Avi Kivity

Quoted 14 lines · Joerg Roedel

Quoted 7 lines · Zachary Amsden

The infrastructure is already there to import / export and migrate MSR settings. MSRs are also 64-bit, and hold "model-specific" settings, so if you don't mind thinking of the nested feature as a model-specific feature of the KVM-SVM CPU, it's even somewhat well defined in terms of the architecture.

There is a lot of additional state to migrate if the vcpu is running nested. To be architecturally correct you need to transfer 6kb of data through MSRs only for the msr permission bitmap.

Quoted 2 lines

The msr permission bitmap is in guest memory, so it is already migrated.

This will work almost always but its not architecturally correct because the memory contents may have changed since the last vmrun instruction. On the other hand we already have this problem with the current nested msr intercept handling...

Quoted 5 lines

Quoted 3 lines

The rest comes down to
the nested intercept masks

These are in the vmcb, which is in guest memory.

Same as with the MSR permission map here.

Quoted 7 lines

Quoted 5 lines

It is doable but I still think its
complicated to get this right. The simplest approach would be to disallow migration when the vcpu is running in guest mode.

Agree, though I dislike the need to introduce a "force vmexit" ioctl.

Yes, this has possible issues too. If we reconstruct the nested state from the nested vmcb there is not much state left which needs migration. But we should keep in mind that this is not how real hardware works.

So, if some other cpu (or the guest itself, with appropriate permissions) modifies the msr permission bitmap, svm will not notice this? svm loads the bitmap during entry?

I don't think you can tell, unless the host cpu modifying the vmcb is synchronized with the guest (or the guest modifies its own vmcb). But this is all academic.

Quoted 5 lines

I don't think you can tell, unless the host cpu modifying the vmcb is synchronized with the guest (or the guest modifies its own vmcb). But this is all academic.

Hmm, another thing comes to mind. We would need some redesign of the nested_svm code to allow userspace to put a vcpu directly into nested state. With the MSR approach, all userspace does is to write MSRs into the vcpu before the first run?

Quoted 6 lines · Zachary Amsden

The infrastructure is already there to import / export and migrate MSR settings. MSRs are also 64-bit, and hold "model-specific" settings, so if you don't mind thinking of the nested feature as a model-specific feature of the KVM-SVM CPU, it's even somewhat well defined in terms of the architecture.

There is a lot of additional state to migrate if the vcpu is running nested. To be architecturally correct you need to transfer 6kb of data through MSRs only for the msr permission bitmap. The rest comes down to the nested intercept masks and some small bits like the global interrupt flag and the nested vmcb address. It is doable but I still think its complicated to get this right. The simplest approach would be to disallow migration when the vcpu is running in guest mode.

Quoted 3 lines

Mostly the problem is figuring out what chunk of MSR space to use.

And hoping that this MSR space is not used by real hardware in the future ;-)

Quoted 3 lines · Joerg Roedel

It is doable but I still think its
complicated to get this right. The simplest approach would be to disallow migration when the vcpu is running in guest mode.

Agree, though I dislike the need to introduce a "force vmexit" ioctl.

Quoted 8 lines · Avi Kivity

Quoted 3 lines · Joerg Roedel

It is doable but I still think its
complicated to get this right. The simplest approach would be to disallow migration when the vcpu is running in guest mode.

Agree, though I dislike the need to introduce a "force vmexit" ioctl.

How can this possibly work for guests which never exit SVM? They can never be migrated.

Quoted 4 lines · Avi Kivity

So, if some other cpu (or the guest itself, with appropriate permissions) modifies the msr permission bitmap, svm will not notice this? svm loads the bitmap during entry?

Yes.

Quoted 4 lines

I don't think you can tell, unless the host cpu modifying the vmcb is synchronized with the guest (or the guest modifies its own vmcb). But this is all academic.

Hmm, another thing comes to mind. We would need some redesign of the nested_svm code to allow userspace to put a vcpu directly into nested state. With the MSR approach, all userspace does is to write MSRs into the vcpu before the first run?

Quoted 5 lines · Avi Kivity

Quoted 3 lines · Avi Kivity

(Either synthetic msrs, or an new state ioctl). The state would contain a bit that says whether the guest is in guest or host mode.

Quoted 9 lines

Quoted 4 lines

However, since we're breaking the architecture one way or another, let's just go with the synthetic INTR intercept.

On the other hand, there is no equivalent intercept in vmx. The "external interrupt" exit can be configured to run an INTACK cycle and capture the vector in a vmcs field, and there's no vector we can insert there (Xen for example uses this).

Difficult. We could use an instruction intercept which has no side effect on guest state (invlpg for example). But thats a lot more dangerous than an INTR intercept. What about PENDING_INTERRUPT? Are there hypervisors that may get confused getting this intercept without asking for it?

Quoted 14 lines · Avi Kivity

Quoted 3 lines · Joerg Roedel

Difficult. We could use an instruction intercept which has no side effect on guest state (invlpg for example).

Especially as the guest might disable it and BUG() if npt is enabled.

Quoted 6 lines

But thats a lot more
dangerous than an INTR intercept. What about PENDING_INTERRUPT? Are there hypervisors that may get confused getting this intercept without asking for it?

Quoted 4 lines

That will likely confuse kvm, it means interrupts are not blocked so it is okay to inject one, and if the guest is running with interrupts disabled then the next entry will fail.

Completly different idea:

1. Give userspace an ioctl to freeze the complete VM (all vcpus must be stopped afterwards)
2. The freeze routine does the following:

freeze_vm(...)

{
	stop_all_vcpus();
	for_each_vcpu(vcpu)
		kvm_x86_ops->freeze_vcpu(vcpu);
}

The module specific freeze_vcpu function can save the nested state inside the guests memory (vmcs area for nested-vmx and hsave area for nested-svm). When no vcpu is running anymore this is save.

After migration the information is restored from there before any vcpu is started again.

unfreeze_vm(...)

{
	for_each_vcpu(vcpu)
		kvm_x86_ops->unfreeze_vcpu(vcpu);
	restart_all_vcpus();
}

In the SVM case this still leaves the problem that the MSR bitmap must be read again from guests memory on unfreeze but that is not a real problem.

Quoted 4 lines

In the SVM case this still leaves the problem that the MSR bitmap must be read again from guests memory on unfreeze but that is not a real problem.

This is easily solved by shadowing the MSR bitmap. Too bad we have to do it, but at least the state isn't hidden in a non-extractable field inside the processor.

Quoted 7 lines · Zachary Amsden

Quoted 6 lines · Avi Kivity

Quoted 3 lines · Joerg Roedel

It is doable but I still think its
complicated to get this right. The simplest approach would be to disallow migration when the vcpu is running in guest mode.

Quoted 4 lines

Quoted 2 lines

Agree, though I dislike the need to introduce a "force vmexit" ioctl.

How can this possibly work for guests which never exit SVM? They can never be migrated.

The force vmexit would generate an INTR #vmexit even if the INTR intercept was disabled and even if no INTR is pending. However this was shot down since there was no equivalent vmx exit reason that we can except the guest to reasonably handle.

Quoted 1 line

The force vmexit would generate an INTR #vmexit even if the INTR intercept was disabled and even if no INTR is pending. However this was shot down since there was no equivalent vmx exit reason that we can except the guest to reasonably handle.

While true, my point is more precisely - how can this possibly work for guests which MUST never exit SVM? As in, the hypervisor is broken or deliberately disabled from taking exits, and in fact, may no longer even exist in memory?

Quoted 4 lines · Zachary Amsden

Quoted 1 line

The force vmexit would generate an INTR #vmexit even if the INTR intercept was disabled and even if no INTR is pending. However this was shot down since there was no equivalent vmx exit reason that we can except the guest to reasonably handle.

While true, my point is more precisely - how can this possibly work for guests which MUST never exit SVM? As in, the hypervisor is broken or deliberately disabled from taking exits, and in fact, may no longer even exist in memory?

These guests will be broken. My assumption was that only malicious guests will disable INTR intercepts (though I can imagine a Luvalley-like system that disables INTR intercepts when running dom0).

Quoted 5 lines · Joerg Roedel

Quoted 3 lines · Avi Kivity

So, if some other cpu (or the guest itself, with appropriate permissions) modifies the msr permission bitmap, svm will not notice this? svm loads the bitmap during entry?

Yes.

Ugh. So we have non-reversible architectural state all over again. There are ways around this problem, all ugly, but the easiest is shadowing the MSR permission bitmap.

Quoted 3 lines

Quoted 3 lines

I don't think you can tell, unless the host cpu modifying the vmcb is synchronized with the guest (or the guest modifies its own vmcb). But this is all academic.

Quoted 4 lines

Hmm, another thing comes to mind. We would need some redesign of the nested_svm code to allow userspace to put a vcpu directly into nested state. With the MSR approach, all userspace does is to write MSRs into the vcpu before the first run?

How does MSR_KVM_NESTED_SVM_ACTIVE not solve this problem?

Quoted 7 lines · Avi Kivity

Quoted 4 lines · Zachary Amsden

Quoted 1 line

The force vmexit would generate an INTR #vmexit even if the INTR intercept was disabled and even if no INTR is pending. However this was shot down since there was no equivalent vmx exit reason that we can except the guest to reasonably handle.

While true, my point is more precisely - how can this possibly work for guests which MUST never exit SVM? As in, the hypervisor is broken or deliberately disabled from taking exits, and in fact, may no longer even exist in memory?

These guests will be broken. My assumption was that only malicious guests will disable INTR intercepts (though I can imagine a Luvalley-like system that disables INTR intercepts when running dom0).

Not an SVM expert, but can't you pass through INTR in SVM and leave a fully functioning guest which technically runs under SVM but requires no hypervisor?

Is that what the Luvalley system does?

Quoted 5 lines · Zachary Amsden

Quoted 4 lines · Avi Kivity

Quoted 3 lines · Zachary Amsden

Quoted 1 line

The force vmexit would generate an INTR #vmexit even if the INTR intercept was disabled and even if no INTR is pending. However this was shot down since there was no equivalent vmx exit reason that we can except the guest to reasonably handle.

Quoted 5 lines

Quoted 3 lines

Quoted 1 line

While true, my point is more precisely - how can this possibly work for guests which MUST never exit SVM? As in, the hypervisor is broken or deliberately disabled from taking exits, and in fact, may no longer even exist in memory?

These guests will be broken. My assumption was that only malicious guests will disable INTR intercepts (though I can imagine a Luvalley-like system that disables INTR intercepts when running dom0).

Not an SVM expert, but can't you pass through INTR in SVM and leave a fully functioning guest which technically runs under SVM but requires no hypervisor?

You could, but without trapping INTR, you can't reliably multiplex guests or have a hypervisor-controlled network interface. That means you're likely a blue pill thing.

Quoted 2 lines

Is that what the Luvalley system does?

Luvalley is vmx only at the moment, but it certainly could let its dom0 handle interrupts (since the scheduler and all device drivers are in dom0). Once it switches to a different guest, it needs to enable INTR.

Quoted 3 lines · Avi Kivity

So, if some other cpu (or the guest itself, with appropriate permissions) modifies the msr permission bitmap, svm will not notice this? svm loads the bitmap during entry?

Yes.

Quoted 8 lines · Joerg Roedel

Quoted 4 lines · Avi Kivity

So, if some other cpu (or the guest itself, with appropriate permissions) modifies the msr permission bitmap, svm will not notice this? svm loads the bitmap during entry?

Yes.

Talked again with a few people about it and the architectural answer is a bit different. The right answer to that is:

The cpu is allowed to load the msr permission map to internal state, but not required to do it by the architecture. The result of changing the msr permission map of a running vmcb is undefined.

So we are fine with the way nested svm does it. And we don't need to care about that situation at all and just migrate whats in guest memory.