KVM/arm64: Guest ABI changes do not appear rollback-safe

Hey folks,

Ricardo and I were discussing hypercall support in KVM/arm64 and something seems to be a bit problematic. I do not see anywhere that KVM requires opt-in from the VMM to expose new hypercalls to the guest. To name a few, the TRNG and KVM PTP hypercalls are unconditionally provided to the guest.

Exposing new hypercalls to guests in this manner seems very unsafe to me. Suppose an operator is trying to upgrade from kernel N to kernel N+1, which brings in the new 'widget' hypercall. Guests are live migrated onto the N+1 kernel, but the operator finds a defect that warrants a kernel rollback. VMs are then migrated from kernel N+1 -> N. Any guests that discovered the 'widget' hypercall are likely going to get fussy _very_ quickly on the old kernel.

Really, we need to ensure that the exposed guest ABI is backwards-compatible. Running a VMM that is blissfully unaware of the 'widget' hypercall should not implicitly expose it to its guest on a new kernel.

This conversation was in the context of devising a new UAPI that allows VMMs to trap hypercalls to userspace. While such an interface would easily work around the issue, the onus of ABI compatibility lies with the kernel.

So, this is all a long-winded way of asking: how do we dig ourselves out of this situation, and how to we avoid it happening again in the future? I believe the answer to both is to have new VM capabilities for sets of hypercalls exposed to the guest. Unfortunately, the unconditional exposure of TRNG and PTP hypercalls is ABI now, so we'd have to provide an opt-out at this point. For anything new, require opt-in from the VMM before we give it to the guest.

Have I missed something blatantly obvious, or do others see this as an issue as well? I'll reply with an example of adding opt-out for PTP. I'm sure other hypercalls could be handled similarly.

Quoted 7 lines

Exposing new hypercalls to guests in this manner seems very unsafe to me. Suppose an operator is trying to upgrade from kernel N to kernel N+1, which brings in the new 'widget' hypercall. Guests are live migrated onto the N+1 kernel, but the operator finds a defect that warrants a kernel rollback. VMs are then migrated from kernel N+1 -> N. Any guests that discovered the 'widget' hypercall are likely going to get fussy _very_ quickly on the old kernel.

This goes against what we decided to support for the *only* publicly available VMM that cares about save/restore, which is that we only move forward and don't rollback.

Hypercalls are the least of your
worries, and there is a whole range of other architectural features that will have also appeared/disappeared (your own CNTPOFF series is a glaring example of this).

Quoted 3 lines

Have I missed something blatantly obvious, or do others see this as an issue as well? I'll reply with an example of adding opt-out for PTP. I'm sure other hypercalls could be handled similarly.

Why do we need this? For future hypercalls, we could have some buy-in capabilities. For existing ones, it is too late, and negative features are just too horrible.

For KVM-specific hypercalls, we could get the VMM to save/restore the bitmap of supported functions. That would be "less horrible". This could be implemented using extra "firmware pseudo-registers" such as the ones described in Documentation/virt/kvm/arm/psci.rst.

Thanks,

        M.

--
Without deviation from the norm, progress is not possible.

Quoted 11 lines · Marc Zyngier

Quoted 7 lines

Exposing new hypercalls to guests in this manner seems very unsafe to me. Suppose an operator is trying to upgrade from kernel N to kernel N+1, which brings in the new 'widget' hypercall. Guests are live migrated onto the N+1 kernel, but the operator finds a defect that warrants a kernel rollback. VMs are then migrated from kernel N+1 -> N. Any guests that discovered the 'widget' hypercall are likely going to get fussy _very_ quickly on the old kernel.

This goes against what we decided to support for the *only* publicly available VMM that cares about save/restore, which is that we only move forward and don't rollback.

Ah, I was definitely missing this context. Current behavior makes much more sense then.

Quoted 4 lines

Hypercalls are the least of your
worries, and there is a whole range of other architectural features that will have also appeared/disappeared (your own CNTPOFF series is a glaring example of this).

Isn't that a tad bit different though? I'll admit, I'm just as guilty with my own series forgetting to add a KVM_CAP (oops), but it is in my queue to kick out with the fix for nVHE/ptimer. Nonetheless, if a user takes up a new KVM UAPI, it is up to the user to run on a new kernel.

My concerns are explicitly with the 'under the nose' changes, where KVM modifies the guest feature set without userspace opting in. Based on your comment, though, it would appear that other parts of KVM are affected too.

simply be a hypercall which an operator doesn't want to give its guests, and it needs a way to tell KVM to hide it.

Quoted 7 lines

Quoted 3 lines

Have I missed something blatantly obvious, or do others see this as an issue as well? I'll reply with an example of adding opt-out for PTP. I'm sure other hypercalls could be handled similarly.

Why do we need this? For future hypercalls, we could have some buy-in capabilities. For existing ones, it is too late, and negative features are just too horrible.

Oh, agreed on the nastiness. Lazy hack to realize the intended functional change..

Quoted 4 lines

For KVM-specific hypercalls, we could get the VMM to save/restore the bitmap of supported functions. That would be "less horrible". This could be implemented using extra "firmware pseudo-registers" such as the ones described in Documentation/virt/kvm/arm/psci.rst.

This seems more reasonable, especially since we do this for migrating the guest's PSCI version.

Alternatively, I had thought about using a VM attribute, given the fact that it is non-architectural information and we avoid ABI issues in KVM_GET_REG_LIST without buy-in through a KVM_CAP.

Quoted 22 lines · Oliver Upton

Quoted 11 lines · Marc Zyngier

Quoted 7 lines

Exposing new hypercalls to guests in this manner seems very unsafe to me. Suppose an operator is trying to upgrade from kernel N to kernel N+1, which brings in the new 'widget' hypercall. Guests are live migrated onto the N+1 kernel, but the operator finds a defect that warrants a kernel rollback. VMs are then migrated from kernel N+1 -> N. Any guests that discovered the 'widget' hypercall are likely going to get fussy _very_ quickly on the old kernel.

This goes against what we decided to support for the *only* publicly available VMM that cares about save/restore, which is that we only move forward and don't rollback.

Ah, I was definitely missing this context. Current behavior makes much more sense then.

Quoted 4 lines

Hypercalls are the least of your
worries, and there is a whole range of other architectural features that will have also appeared/disappeared (your own CNTPOFF series is a glaring example of this).

Quoted 4 lines

Isn't that a tad bit different though? I'll admit, I'm just as guilty with my own series forgetting to add a KVM_CAP (oops), but it is in my queue to kick out with the fix for nVHE/ptimer. Nonetheless, if a user takes up a new KVM UAPI, it is up to the user to run on a new kernel.

The two are linked. Exposing a new register to userspace and/or guest result in the same thing: you can't rollback. That's specially true in the QEMU case, which *learns* from the kernel what registers are available, and doesn't maintain a fixed list.

Quoted 4 lines

My concerns are explicitly with the 'under the nose' changes, where KVM modifies the guest feature set without userspace opting in. Based on your comment, though, it would appear that other parts of KVM are affected too.

Any new system register that is exposed by a new kernel feature breaks rollback. And so far, we only consider it a bug if the set of exposed registers reduces. Anything can be added safely (as checked by one of the selftests added by Drew).

< It doesn't have to be rollback safety, either. There may

Quoted 2 lines

simply be a hypercall which an operator doesn't want to give its guests, and it needs a way to tell KVM to hide it.

Fair enough. But this has to be done in a scalable way, which individual capability cannot provide.

Quoted 10 lines

Quoted 7 lines

Quoted 3 lines

Have I missed something blatantly obvious, or do others see this as an issue as well? I'll reply with an example of adding opt-out for PTP. I'm sure other hypercalls could be handled similarly.

Why do we need this? For future hypercalls, we could have some buy-in capabilities. For existing ones, it is too late, and negative features are just too horrible.

Oh, agreed on the nastiness. Lazy hack to realize the intended functional change..

Well, you definitely achieved your goal of attracting my attention :).

Quoted 11 lines

Quoted 4 lines

For KVM-specific hypercalls, we could get the VMM to save/restore the bitmap of supported functions. That would be "less horrible". This could be implemented using extra "firmware pseudo-registers" such as the ones described in Documentation/virt/kvm/arm/psci.rst.

This seems more reasonable, especially since we do this for migrating the guest's PSCI version.

Alternatively, I had thought about using a VM attribute, given the fact that it is non-architectural information and we avoid ABI issues in KVM_GET_REG_LIST without buy-in through a KVM_CAP.

The whole point is that these settings get exposed by KVM_GET_REG_LIST, as this is QEMU's way to dump a VM state. Given that we already have this for things like the spectre management state, we can just as well expose the bitmaps that deal with the KVM-specific hypercalls. After all, this falls into the realm of "KVM as VM firmware".

For ARM-architected hypercalls (TRNG, stolen time), we may need a similar extension.

Quoted 83 lines · Marc Zyngier

Quoted 22 lines · Oliver Upton

Quoted 11 lines · Marc Zyngier

Quoted 7 lines

Exposing new hypercalls to guests in this manner seems very unsafe to me. Suppose an operator is trying to upgrade from kernel N to kernel N+1, which brings in the new 'widget' hypercall. Guests are live migrated onto the N+1 kernel, but the operator finds a defect that warrants a kernel rollback. VMs are then migrated from kernel N+1 -> N. Any guests that discovered the 'widget' hypercall are likely going to get fussy _very_ quickly on the old kernel.

This goes against what we decided to support for the *only* publicly available VMM that cares about save/restore, which is that we only move forward and don't rollback.

Ah, I was definitely missing this context. Current behavior makes much more sense then.

Quoted 4 lines

Hypercalls are the least of your
worries, and there is a whole range of other architectural features that will have also appeared/disappeared (your own CNTPOFF series is a glaring example of this).

Quoted 4 lines

Isn't that a tad bit different though? I'll admit, I'm just as guilty with my own series forgetting to add a KVM_CAP (oops), but it is in my queue to kick out with the fix for nVHE/ptimer. Nonetheless, if a user takes up a new KVM UAPI, it is up to the user to run on a new kernel.

The two are linked. Exposing a new register to userspace and/or guest result in the same thing: you can't rollback. That's specially true in the QEMU case, which *learns* from the kernel what registers are available, and doesn't maintain a fixed list.

Quoted 4 lines

My concerns are explicitly with the 'under the nose' changes, where KVM modifies the guest feature set without userspace opting in. Based on your comment, though, it would appear that other parts of KVM are affected too.

Any new system register that is exposed by a new kernel feature breaks rollback. And so far, we only consider it a bug if the set of exposed registers reduces. Anything can be added safely (as checked by one of the selftests added by Drew).

< It doesn't have to be rollback safety, either. There may

Quoted 2 lines

simply be a hypercall which an operator doesn't want to give its guests, and it needs a way to tell KVM to hide it.

Fair enough. But this has to be done in a scalable way, which individual capability cannot provide.

Quoted 10 lines

Quoted 7 lines

Quoted 3 lines

Have I missed something blatantly obvious, or do others see this as an issue as well? I'll reply with an example of adding opt-out for PTP. I'm sure other hypercalls could be handled similarly.

Why do we need this? For future hypercalls, we could have some buy-in capabilities. For existing ones, it is too late, and negative features are just too horrible.

Oh, agreed on the nastiness. Lazy hack to realize the intended functional change..

Well, you definitely achieved your goal of attracting my attention :).

Quoted 11 lines

Quoted 4 lines

For KVM-specific hypercalls, we could get the VMM to save/restore the bitmap of supported functions. That would be "less horrible". This could be implemented using extra "firmware pseudo-registers" such as the ones described in Documentation/virt/kvm/arm/psci.rst.

This seems more reasonable, especially since we do this for migrating the guest's PSCI version.

Alternatively, I had thought about using a VM attribute, given the fact that it is non-architectural information and we avoid ABI issues in KVM_GET_REG_LIST without buy-in through a KVM_CAP.

The whole point is that these settings get exposed by KVM_GET_REG_LIST, as this is QEMU's way to dump a VM state. Given that we already have this for things like the spectre management state, we can just as well expose the bitmaps that deal with the KVM-specific hypercalls. After all, this falls into the realm of "KVM as VM firmware".

Quoted 3 lines

For ARM-architected hypercalls (TRNG, stolen time), we may need a similar extension.

Thanks for including me Marc. I think you've mentioned all the examples of why we don't generally expect N+1 -> N migrations to work that I can think of. While some of the examples like get-reg-list could eventually be eliminated if we had CPU models to tighten our machine type state, I think N+1 -> N migrations will always be best effort at most.

I agree with giving userspace control over the exposer of the hypercalls though. Using pseudo-registers for that purpose rather than a pile of CAPs also seems reasonable to me.

And, while I don't think this patch is going to proceed, I thought I'd point out that the opt-out approach doesn't help much with expanding our migration support unless we require the VMM to be upgraded first.

And, even then, the (N_kern, N+1_vmm) -> (N+1_kern, N_vmm) case won't work as expected, since the source enforce opt-out, but the destination won't.

Also, since the VMM doesn't key off the kernel version, for the most part N+1 VMMs won't know when they're supposed to opt-out or not, leaving it to the user to ensure they consider everything. opt-in usually only needs the user to consider what machine type they want to launch.

Thanks,
drew

On Wed, Aug 25, 2021 at 8:07 AM Andrew Jones <[email protected]> wrote:

Quoted 10 lines

Thanks for including me Marc. I think you've mentioned all the examples of why we don't generally expect N+1 -> N migrations to work that I can think of. While some of the examples like get-reg-list could eventually be eliminated if we had CPU models to tighten our machine type state, I think N+1 -> N migrations will always be best effort at most.

I agree with giving userspace control over the exposer of the hypercalls though. Using pseudo-registers for that purpose rather than a pile of CAPs also seems reasonable to me.

Quoted 7 lines

And, while I don't think this patch is going to proceed, I thought I'd point out that the opt-out approach doesn't help much with expanding our migration support unless we require the VMM to be upgraded first.

And, even then, the (N_kern, N+1_vmm) -> (N+1_kern, N_vmm) case won't work as expected, since the source enforce opt-out, but the destination won't.

Right, there's going to need to be a fence in both kernel and VMM versions. Before the fence, you can't rollback with either component. Once on the other side of the fence, the user may freely migrate between kernel + VMM combinations.

Quoted 5 lines

Also, since the VMM doesn't key off the kernel version, for the most part N+1 VMMs won't know when they're supposed to opt-out or not, leaving it to the user to ensure they consider everything. opt-in usually only needs the user to consider what machine type they want to launch.

Going the register route will implicitly require opt-out for all old hypercalls. We exposed them unconditionally to the guest before, and we must uphold that behavior. The default value for the bitmap will have those features set. Any hypercalls added after that register interface will then require explicit opt-in from userspace.

With regards to the pseudoregister interface, how would a VMM discover new bits? From my perspective, you need to have two bitmaps that the VMM can get at: the set of supported feature bits and the active bitmap of features for a running guest.

Quoted 2 lines · Oliver Upton

On Wed, Aug 25, 2021 at 8:07 AM Andrew Jones <[email protected]> wrote:

[...]

Quoted 29 lines

Quoted 10 lines

Thanks for including me Marc. I think you've mentioned all the examples of why we don't generally expect N+1 -> N migrations to work that I can think of. While some of the examples like get-reg-list could eventually be eliminated if we had CPU models to tighten our machine type state, I think N+1 -> N migrations will always be best effort at most.

I agree with giving userspace control over the exposer of the hypercalls though. Using pseudo-registers for that purpose rather than a pile of CAPs also seems reasonable to me.

Quoted 7 lines

And, while I don't think this patch is going to proceed, I thought I'd point out that the opt-out approach doesn't help much with expanding our migration support unless we require the VMM to be upgraded first.

And, even then, the (N_kern, N+1_vmm) -> (N+1_kern, N_vmm) case won't work as expected, since the source enforce opt-out, but the destination won't.

Right, there's going to need to be a fence in both kernel and VMM versions. Before the fence, you can't rollback with either component. Once on the other side of the fence, the user may freely migrate between kernel + VMM combinations.

Quoted 5 lines

Also, since the VMM doesn't key off the kernel version, for the most part N+1 VMMs won't know when they're supposed to opt-out or not, leaving it to the user to ensure they consider everything. opt-in usually only needs the user to consider what machine type they want to launch.

Quoted 5 lines

Going the register route will implicitly require opt-out for all old hypercalls. We exposed them unconditionally to the guest before, and we must uphold that behavior. The default value for the bitmap will have those features set. Any hypercalls added after that register interface will then require explicit opt-in from userspace.

I disagree here. This makes the ABI inconsistent, and means that no feature can be implemented without changing userspace. If you can deal with the existing features, you should be able to deal with the next lot.

Quoted 4 lines

With regards to the pseudoregister interface, how would a VMM discover new bits? From my perspective, you need to have two bitmaps that the VMM can get at: the set of supported feature bits and the active bitmap of features for a running guest.

My proposal is that we have a single pseudo-register exposing the list of implemented by the kernel. Clear the bits you don't want, and write back the result. As long as you haven't written anything, you have the full feature set. That's pretty similar to the virtio feature negotiation.

Quoted 52 lines · Marc Zyngier

Quoted 2 lines · Oliver Upton

On Wed, Aug 25, 2021 at 8:07 AM Andrew Jones <[email protected]> wrote:

[...]

Quoted 29 lines

Quoted 10 lines

Thanks for including me Marc. I think you've mentioned all the examples of why we don't generally expect N+1 -> N migrations to work that I can think of. While some of the examples like get-reg-list could eventually be eliminated if we had CPU models to tighten our machine type state, I think N+1 -> N migrations will always be best effort at most.

I agree with giving userspace control over the exposer of the hypercalls though. Using pseudo-registers for that purpose rather than a pile of CAPs also seems reasonable to me.

Quoted 7 lines

And, while I don't think this patch is going to proceed, I thought I'd point out that the opt-out approach doesn't help much with expanding our migration support unless we require the VMM to be upgraded first.

And, even then, the (N_kern, N+1_vmm) -> (N+1_kern, N_vmm) case won't work as expected, since the source enforce opt-out, but the destination won't.

Right, there's going to need to be a fence in both kernel and VMM versions. Before the fence, you can't rollback with either component. Once on the other side of the fence, the user may freely migrate between kernel + VMM combinations.

Quoted 5 lines

Also, since the VMM doesn't key off the kernel version, for the most part N+1 VMMs won't know when they're supposed to opt-out or not, leaving it to the user to ensure they consider everything. opt-in usually only needs the user to consider what machine type they want to launch.

Quoted 5 lines

Going the register route will implicitly require opt-out for all old hypercalls. We exposed them unconditionally to the guest before, and we must uphold that behavior. The default value for the bitmap will have those features set. Any hypercalls added after that register interface will then require explicit opt-in from userspace.

I disagree here. This makes the ABI inconsistent, and means that no feature can be implemented without changing userspace. If you can deal with the existing features, you should be able to deal with the next lot.

Quoted 4 lines

With regards to the pseudoregister interface, how would a VMM discover new bits? From my perspective, you need to have two bitmaps that the VMM can get at: the set of supported feature bits and the active bitmap of features for a running guest.

Quoted 5 lines

My proposal is that we have a single pseudo-register exposing the list of implemented by the kernel. Clear the bits you don't want, and write back the result. As long as you haven't written anything, you have the full feature set. That's pretty similar to the virtio feature negotiation.

Ah, yes I agree. Thinking about it more we will not need something similar to KVM_GET_SUPPORTED_CPUID.

So then, for any register where userspace/KVM need to negotiate features, the default value will return the maximum feature set that is supported. If userspace wants to constrain features, read out the register, make sure everything you want is there, and write it back blowing away the superfluous bits. Given this should we enforce ordering on feature registers, such that a VMM can only write to the registers before a VM is started?

Also, Reiji is working on making the identity registers writable for the sake of feature restriction. The suggested negotiation interface would be applicable there too, IMO.

Many thanks to both you and Drew for working this out with me.

Quoted 62 lines · Oliver Upton

Quoted 52 lines · Marc Zyngier

Quoted 2 lines · Oliver Upton

On Wed, Aug 25, 2021 at 8:07 AM Andrew Jones <[email protected]> wrote:

[...]

Quoted 29 lines

Quoted 10 lines

Thanks for including me Marc. I think you've mentioned all the examples of why we don't generally expect N+1 -> N migrations to work that I can think of. While some of the examples like get-reg-list could eventually be eliminated if we had CPU models to tighten our machine type state, I think N+1 -> N migrations will always be best effort at most.

I agree with giving userspace control over the exposer of the hypercalls though. Using pseudo-registers for that purpose rather than a pile of CAPs also seems reasonable to me.

Quoted 7 lines

And, while I don't think this patch is going to proceed, I thought I'd point out that the opt-out approach doesn't help much with expanding our migration support unless we require the VMM to be upgraded first.

And, even then, the (N_kern, N+1_vmm) -> (N+1_kern, N_vmm) case won't work as expected, since the source enforce opt-out, but the destination won't.

Right, there's going to need to be a fence in both kernel and VMM versions. Before the fence, you can't rollback with either component. Once on the other side of the fence, the user may freely migrate between kernel + VMM combinations.

Quoted 5 lines

Also, since the VMM doesn't key off the kernel version, for the most part N+1 VMMs won't know when they're supposed to opt-out or not, leaving it to the user to ensure they consider everything. opt-in usually only needs the user to consider what machine type they want to launch.

Quoted 5 lines

Going the register route will implicitly require opt-out for all old hypercalls. We exposed them unconditionally to the guest before, and we must uphold that behavior. The default value for the bitmap will have those features set. Any hypercalls added after that register interface will then require explicit opt-in from userspace.

I disagree here. This makes the ABI inconsistent, and means that no feature can be implemented without changing userspace. If you can deal with the existing features, you should be able to deal with the next lot.

Quoted 4 lines

With regards to the pseudoregister interface, how would a VMM discover new bits? From my perspective, you need to have two bitmaps that the VMM can get at: the set of supported feature bits and the active bitmap of features for a running guest.

Quoted 5 lines

My proposal is that we have a single pseudo-register exposing the list of implemented by the kernel. Clear the bits you don't want, and write back the result. As long as you haven't written anything, you have the full feature set. That's pretty similar to the virtio feature negotiation.

Ah, yes I agree. Thinking about it more we will not need something similar to KVM_GET_SUPPORTED_CPUID.

Quoted 7 lines

So then, for any register where userspace/KVM need to negotiate features, the default value will return the maximum feature set that is supported. If userspace wants to constrain features, read out the register, make sure everything you want is there, and write it back blowing away the superfluous bits. Given this should we enforce ordering on feature registers, such that a VMM can only write to the registers before a VM is started?

That's a good idea. KVM_REG_ARM64_SVE_VLS has this type of constraint so we can model the feature register control off that.

Quoted 4 lines

Also, Reiji is working on making the identity registers writable for the sake of feature restriction. The suggested negotiation interface would be applicable there too, IMO.

This this interesting news. I'll look forward to the posting.

Quoted 3 lines

Many thanks to both you and Drew for working this out with me.

Thanks,
drew

I have some lingering thoughts on this subject since we last spoke and wanted to discuss.

I'm having a hard time figuring out how a VMM should handle a new hypercall identity register introduced on a newer kernel. In order to maintain guest ABI, the VMM would need to know about that register and zero it when restoring an older guest.

Perhaps instead we could reserve a range of firmware registers as the 'hypercall identity' registers. Implement all of them as RAZ/WI by default, encouraging userspace to zero these registers away for older VMs but still allowing an old userspace to pick up new KVM features. Doing so would align the hypercall identity registers with the feature ID registers from the architecture.

Hi Oliver,

Quoted 8 lines · Oliver Upton

I have some lingering thoughts on this subject since we last spoke and wanted to discuss.

I'm having a hard time figuring out how a VMM should handle a new hypercall identity register introduced on a newer kernel. In order to maintain guest ABI, the VMM would need to know about that register and zero it when restoring an older guest.

Just as it would need to be able to discover any new system register exposed by default, as it happens at all times. Which is why we have a way to discover all the registers, architected or not.

Quoted 6 lines

Perhaps instead we could reserve a range of firmware registers as the 'hypercall identity' registers. Implement all of them as RAZ/WI by default, encouraging userspace to zero these registers away for older VMs but still allowing an old userspace to pick up new KVM features. Doing so would align the hypercall identity registers with the feature ID registers from the architecture.

The range already exists in the form of the "coprocessor" 0x14. I don't see the need to expose it as RAZ/WI, however. If userspace doesn't know about how this pseudo-register works, it won't be able to program it anyway.

I don't buy the parallel with the ID-regs either. They are RAZ/WI by default so that they don't UNDEF at runtime. The meaning of a RAZ id-register is also well defined (feature not implemented), and the CPU cannot write to them. In a way, the ID-regs *are* the enumeration mechanism.

Our firmware registers don't follow the same rules. Userspace can write to them, and there is no such "not implemented" rule (case in point, PSCI). We also have a separate enumeration mechanism (GET_ONE_REG), which is (more or less) designed for userspace to find what is implemented.

For these reasons, I don't immediately see the point of advertising a set of registers ahead of time, before userspace grows an understanding of what these registers mean.

Hey Marc,

Quoted 42 lines · Marc Zyngier

Hi Oliver,

Quoted 8 lines · Oliver Upton

I have some lingering thoughts on this subject since we last spoke and wanted to discuss.

I'm having a hard time figuring out how a VMM should handle a new hypercall identity register introduced on a newer kernel. In order to maintain guest ABI, the VMM would need to know about that register and zero it when restoring an older guest.

Just as it would need to be able to discover any new system register exposed by default, as it happens at all times. Which is why we have a way to discover all the registers, architected or not.

Quoted 6 lines

Perhaps instead we could reserve a range of firmware registers as the 'hypercall identity' registers. Implement all of them as RAZ/WI by default, encouraging userspace to zero these registers away for older VMs but still allowing an old userspace to pick up new KVM features. Doing so would align the hypercall identity registers with the feature ID registers from the architecture.

The range already exists in the form of the "coprocessor" 0x14. I don't see the need to expose it as RAZ/WI, however. If userspace doesn't know about how this pseudo-register works, it won't be able to program it anyway.

I don't buy the parallel with the ID-regs either. They are RAZ/WI by default so that they don't UNDEF at runtime. The meaning of a RAZ id-register is also well defined (feature not implemented), and the CPU cannot write to them. In a way, the ID-regs *are* the enumeration mechanism.

Our firmware registers don't follow the same rules. Userspace can write to them, and there is no such "not implemented" rule (case in point, PSCI). We also have a separate enumeration mechanism (GET_ONE_REG), which is (more or less) designed for userspace to find what is implemented.

Quoted 3 lines

For these reasons, I don't immediately see the point of advertising a set of registers ahead of time, before userspace grows an understanding of what these registers mean.

Supposing we don't preallocate some hypercall ID registers, how can we safely migrate a guest from an older kernel to newer kernel? Ideally, we would preserve the hypercall feature set across the migration which could work for a while with the first set of registers that get defined, but whenever a new hypercall firmware register comes along then the VMM will be clueless to the new ABI.

Fundamentally, I don't think userspace should need a patch to preserve ABI on a newer kernel. Despite that, it would seem that userspace will need to learn of any firmware registers that control hypercall features which come after the initial set that gets proposed. If KVM_GET_REG_LIST were to disambiguate between ID registers (hypercall, architectural feature ID registers) from other parts of the vCPU state, it would be clear to what registers to zero on a newer kernel. Apologies if it is distracting to mention the feature ID registers here, but both are on my mind currently and want to make sure there is some consistency in how features get handled on newer kernels, architected or not.

Quoted 50 lines · Oliver Upton

Hey Marc,

Quoted 42 lines · Marc Zyngier

Hi Oliver,

Quoted 8 lines · Oliver Upton

I have some lingering thoughts on this subject since we last spoke and wanted to discuss.

I'm having a hard time figuring out how a VMM should handle a new hypercall identity register introduced on a newer kernel. In order to maintain guest ABI, the VMM would need to know about that register and zero it when restoring an older guest.

Just as it would need to be able to discover any new system register exposed by default, as it happens at all times. Which is why we have a way to discover all the registers, architected or not.

Quoted 6 lines

Perhaps instead we could reserve a range of firmware registers as the 'hypercall identity' registers. Implement all of them as RAZ/WI by default, encouraging userspace to zero these registers away for older VMs but still allowing an old userspace to pick up new KVM features. Doing so would align the hypercall identity registers with the feature ID registers from the architecture.

The range already exists in the form of the "coprocessor" 0x14. I don't see the need to expose it as RAZ/WI, however. If userspace doesn't know about how this pseudo-register works, it won't be able to program it anyway.

I don't buy the parallel with the ID-regs either. They are RAZ/WI by default so that they don't UNDEF at runtime. The meaning of a RAZ id-register is also well defined (feature not implemented), and the CPU cannot write to them. In a way, the ID-regs *are* the enumeration mechanism.

Our firmware registers don't follow the same rules. Userspace can write to them, and there is no such "not implemented" rule (case in point, PSCI). We also have a separate enumeration mechanism (GET_ONE_REG), which is (more or less) designed for userspace to find what is implemented.

Quoted 3 lines

For these reasons, I don't immediately see the point of advertising a set of registers ahead of time, before userspace grows an understanding of what these registers mean.

Quoted 6 lines

Supposing we don't preallocate some hypercall ID registers, how can we safely migrate a guest from an older kernel to newer kernel? Ideally, we would preserve the hypercall feature set across the migration which could work for a while with the first set of registers that get defined, but whenever a new hypercall firmware register comes along then the VMM will be clueless to the new ABI.

My expectations were that whenever userspace writes a set of firmware register, all the defaults are invalidated. For example say that host-A know about a single hypercall register, while host-B knows about two. Upon writing to the first register, the host would clear the set of available services in the second one. If userspace eventually writes there, the value would stick if valid.

Also, remember that pseudo-registers don't have to be 64bit. We could define a new class of hypercall-specific registers that would be much wider, and thus have a larger write granularity (KVM supports anything from 8 to 2048 bits). This would make it pretty easy to implement the above.

Quoted 11 lines

Fundamentally, I don't think userspace should need a patch to preserve ABI on a newer kernel. Despite that, it would seem that userspace will need to learn of any firmware registers that control hypercall features which come after the initial set that gets proposed. If KVM_GET_REG_LIST were to disambiguate between ID registers (hypercall, architectural feature ID registers) from other parts of the vCPU state, it would be clear to what registers to zero on a newer kernel. Apologies if it is distracting to mention the feature ID registers here, but both are on my mind currently and want to make sure there is some consistency in how features get handled on newer kernels, architected or not.

The problem I see is that we will always need to grow the HC register space one way or another, no matter how many we reserve. Your approach only works if we don't exceed that particular range. Maybe it will never be a problem, but it remains that this is not scalable.

If we wanted to be safe, we'd reserve the whole of the possible space as defined by the SMCCC spec. Given that we currently have two HC spaces (the ARM-controlled one, and the KVM-specific one), the function space being 16bits in each case, that's 16kB worth of zeroes that userspace has to save/restore at all times... I'm not overly enthusiastic.

Quoted 90 lines · Marc Zyngier

Quoted 50 lines · Oliver Upton

Hey Marc,

Quoted 42 lines · Marc Zyngier

Hi Oliver,

Quoted 8 lines · Oliver Upton

I have some lingering thoughts on this subject since we last spoke and wanted to discuss.

I'm having a hard time figuring out how a VMM should handle a new hypercall identity register introduced on a newer kernel. In order to maintain guest ABI, the VMM would need to know about that register and zero it when restoring an older guest.

Just as it would need to be able to discover any new system register exposed by default, as it happens at all times. Which is why we have a way to discover all the registers, architected or not.

Quoted 6 lines

Perhaps instead we could reserve a range of firmware registers as the 'hypercall identity' registers. Implement all of them as RAZ/WI by default, encouraging userspace to zero these registers away for older VMs but still allowing an old userspace to pick up new KVM features. Doing so would align the hypercall identity registers with the feature ID registers from the architecture.

The range already exists in the form of the "coprocessor" 0x14. I don't see the need to expose it as RAZ/WI, however. If userspace doesn't know about how this pseudo-register works, it won't be able to program it anyway.

I don't buy the parallel with the ID-regs either. They are RAZ/WI by default so that they don't UNDEF at runtime. The meaning of a RAZ id-register is also well defined (feature not implemented), and the CPU cannot write to them. In a way, the ID-regs *are* the enumeration mechanism.

Our firmware registers don't follow the same rules. Userspace can write to them, and there is no such "not implemented" rule (case in point, PSCI). We also have a separate enumeration mechanism (GET_ONE_REG), which is (more or less) designed for userspace to find what is implemented.

Quoted 3 lines

For these reasons, I don't immediately see the point of advertising a set of registers ahead of time, before userspace grows an understanding of what these registers mean.

Quoted 6 lines

Supposing we don't preallocate some hypercall ID registers, how can we safely migrate a guest from an older kernel to newer kernel? Ideally, we would preserve the hypercall feature set across the migration which could work for a while with the first set of registers that get defined, but whenever a new hypercall firmware register comes along then the VMM will be clueless to the new ABI.

My expectations were that whenever userspace writes a set of firmware register, all the defaults are invalidated. For example say that host-A know about a single hypercall register, while host-B knows about two. Upon writing to the first register, the host would clear the set of available services in the second one. If userspace eventually writes there, the value would stick if valid.

Also, remember that pseudo-registers don't have to be 64bit. We could define a new class of hypercall-specific registers that would be much wider, and thus have a larger write granularity (KVM supports anything from 8 to 2048 bits). This would make it pretty easy to implement the above.

Quoted 11 lines

Fundamentally, I don't think userspace should need a patch to preserve ABI on a newer kernel. Despite that, it would seem that userspace will need to learn of any firmware registers that control hypercall features which come after the initial set that gets proposed. If KVM_GET_REG_LIST were to disambiguate between ID registers (hypercall, architectural feature ID registers) from other parts of the vCPU state, it would be clear to what registers to zero on a newer kernel. Apologies if it is distracting to mention the feature ID registers here, but both are on my mind currently and want to make sure there is some consistency in how features get handled on newer kernels, architected or not.

The problem I see is that we will always need to grow the HC register space one way or another, no matter how many we reserve. Your approach only works if we don't exceed that particular range. Maybe it will never be a problem, but it remains that this is not scalable.

Quoted 6 lines

If we wanted to be safe, we'd reserve the whole of the possible space as defined by the SMCCC spec. Given that we currently have two HC spaces (the ARM-controlled one, and the KVM-specific one), the function space being 16bits in each case, that's 16kB worth of zeroes that userspace has to save/restore at all times... I'm not overly enthusiastic.

Yeah, that's definitely overkill.

I agree with your position; we can solve the issue altogether by enforcing ordering on the hypercall registers. Userspace should read all hypercall registers to discover features, then write them. Given your suggestion, I don't know if there is much need for the guesswork to conclude on an appropriate register size for the hypercall bitmaps. 64 bits seems fine to me..

--
Thanks,
Oliver

Hello all,

Based on the recent discussion on the patch '[RFC PATCH v3 01/11] KVM:
Capture VM start' [1], Reiji, I (and others in the team) were wondering, since the hypercall feature-map is technically per-VM and not per-vCPU, would it make more sense to present it as a kvm_device, rather than vCPU psuedo-registers to the userspace?

If I understand correctly, the original motivation for going with pseudo-registers was to comply with QEMU, which uses KVM_GET_REG_LIST and KVM_[GET|SET]_ONE_REG interface, but I'm guessing the VMMs doing save/restore across migration might write the same values for every vCPU.

Granted that we would be diverging from the existing implementation (psci versioning and spectre WA registers), but this can be a cleaner way forward for extending hypercall support.

The kvm_device interface
can be made backward compatible with the way hypercalls are exposed today, it can have the same registers/features discovery mechanisms that we discussed above, and majorly the userspace has to configure it only once per-VM. We can probably make the feature selection immutable just before any vCPU is created.

Please let me know your thoughts or any disadvantages that I'm overlooking.

Quoted 5 lines

If I understand correctly, the original motivation for going with pseudo-registers was to comply with QEMU, which uses KVM_GET_REG_LIST and KVM_[GET|SET]_ONE_REG interface, but I'm guessing the VMMs doing save/restore across migration might write the same values for every vCPU.

KVM currently restricts the vcpu features to be unified across vcpus, but that's only an implementation choice.

The ARM architecture doesn't
mandate that these registers are all the same, and it isn't impossible that we'd allow for the feature set to become per-vcpu at some point in time. So this argument doesn't really hold.

Furthermore, compatibility with QEMU's save/restore model is essential, and AFAICT, there is no open source alternative.

A device means yet another configuration and migration API. Don't you think we have enough of those? The complexity of KVM/arm64 userspace API is already insane, and extremely fragile. Adding to it will be a validation nightmare (it already is, and I don't see anyone actively helping with it).

Hi Marc,

Quoted 6 lines · Marc Zyngier

Quoted 5 lines

If I understand correctly, the original motivation for going with pseudo-registers was to comply with QEMU, which uses KVM_GET_REG_LIST and KVM_[GET|SET]_ONE_REG interface, but I'm guessing the VMMs doing save/restore across migration might write the same values for every vCPU.

Quoted 2 lines

KVM currently restricts the vcpu features to be unified across vcpus, but that's only an implementation choice.

But that implementation choice has become ABI, no? How could support for asymmetry be added without requiring userspace opt-in or breaking existing VMMs that depend on feature unification?

Quoted 5 lines

The ARM architecture doesn't
mandate that these registers are all the same, and it isn't impossible that we'd allow for the feature set to become per-vcpu at some point in time. So this argument doesn't really hold.

Accessing per-VM state N times is bound to increase VM blackout time during migrations ~linearly as the number of vCPUs in a VM increases, since a VM scoped lock is necessary to serialize guest accesses. It could be tolerable at present scale, but seems like in the future it could become a real problem.

Quoted 3 lines

Furthermore, compatibility with QEMU's save/restore model is essential, and AFAICT, there is no open source alternative.

Agree fundamentally, but I believe it is entirely reasonable to require a userspace change to adopt a new KVM feature. Otherwise, we may be trying to shoehorn new features into existing UAPI that may not be a precise fit..

In order to cure the serialization mentioned above, two options are top of mind: accessing the VM state with the VM FD or informing userspace that a set of registers need only be written once for an entire VM. If we add support for asymmetry later down the road, that would become an opt-in such that userspace will do the access per-vCPU.

Quoted 6 lines

A device means yet another configuration and migration API. Don't you think we have enough of those? The complexity of KVM/arm64 userspace API is already insane, and extremely fragile. Adding to it will be a validation nightmare (it already is, and I don't see anyone actively helping with it).

It seems equally fragile to introduce VM-wide serialization to vCPU UAPI that we know is in the live migration critical path for _any_ VMM. Without requiring userspace changes for all the new widgets under discussion we're effectively forcing VMMs to do something suboptimal.

Quoted 6 lines

Quoted 2 lines

KVM currently restricts the vcpu features to be unified across vcpus, but that's only an implementation choice.

But that implementation choice has become ABI, no? How could support for asymmetry be added without requiring userspace opt-in or breaking existing VMMs that depend on feature unification?

Of course, you'd need some sort of advertising of this new behaviour.

One thing I would like to add to the current state of thing is an indication of whether the effects of a sysreg being written from userspace are global or local to a vcpu. You'd need a new capability, and an extra flag added to the encoding of each register.

Quoted 6 lines

Quoted 5 lines

A device means yet another configuration and migration API. Don't you think we have enough of those? The complexity of KVM/arm64 userspace API is already insane, and extremely fragile. Adding to it will be a validation nightmare (it already is, and I don't see anyone actively helping with it).

Quoted 4 lines

It seems equally fragile to introduce VM-wide serialization to vCPU UAPI that we know is in the live migration critical path for _any_ VMM. Without requiring userspace changes for all the new widgets under discussion we're effectively forcing VMMs to do something suboptimal.

I'm perfectly happy with suboptimal to start with, and find ways to make it better once we have a clear path forward. I just don't want to conflate the two.

Hi Marc,

Quoted 7 lines · Marc Zyngier

Quoted 6 lines

Quoted 2 lines

KVM currently restricts the vcpu features to be unified across vcpus, but that's only an implementation choice.

But that implementation choice has become ABI, no? How could support for asymmetry be added without requiring userspace opt-in or breaking existing VMMs that depend on feature unification?

Quoted 6 lines

Of course, you'd need some sort of advertising of this new behaviour.

One thing I would like to add to the current state of thing is an indication of whether the effects of a sysreg being written from userspace are global or local to a vcpu. You'd need a new capability, and an extra flag added to the encoding of each register.

Ah. I think that is a much more reasonable fit then. VMMs unaware of this can continue to migrate new bits (albeit at the cost of potentially higher lock contention for the per-VM stuff), and those that do can reap the benefits of writing such attributes exactly once.

Quoted 17 lines · Oliver Upton

Hi Marc,

Quoted 7 lines · Marc Zyngier

Quoted 6 lines

Quoted 2 lines

KVM currently restricts the vcpu features to be unified across vcpus, but that's only an implementation choice.

But that implementation choice has become ABI, no? How could support for asymmetry be added without requiring userspace opt-in or breaking existing VMMs that depend on feature unification?

Quoted 6 lines

Of course, you'd need some sort of advertising of this new behaviour.

One thing I would like to add to the current state of thing is an indication of whether the effects of a sysreg being written from userspace are global or local to a vcpu. You'd need a new capability, and an extra flag added to the encoding of each register.

Quoted 4 lines

Ah. I think that is a much more reasonable fit then. VMMs unaware of this can continue to migrate new bits (albeit at the cost of potentially higher lock contention for the per-VM stuff), and those that do can reap the benefits of writing such attributes exactly once.

But the "proper" usage is no different than adding support for VM-scoped variants of KVM_{G,S}ET_ONE_REG and friends, and a VM-scoped variant is conceptually a lot cleaner IMO. And making them truly VM-scoped means KVM can do things like support sysregs that are immutable after vCPUs are created.

So long as KVM defaults to '0' for all such registers, lack of migration support in userspace that isn't aware of the new API, i.e. doesn't do KVM_GET_REG_LIST at a VM-scope, is a nop because said userspace also won't modify the registers in the first place.

Quoted 99 lines · Andrew Jones

Quoted 83 lines · Marc Zyngier

Quoted 22 lines · Oliver Upton

Quoted 11 lines · Marc Zyngier

Quoted 7 lines

Exposing new hypercalls to guests in this manner seems very unsafe to me. Suppose an operator is trying to upgrade from kernel N to kernel N+1, which brings in the new 'widget' hypercall. Guests are live migrated onto the N+1 kernel, but the operator finds a defect that warrants a kernel rollback. VMs are then migrated from kernel N+1 -> N. Any guests that discovered the 'widget' hypercall are likely going to get fussy _very_ quickly on the old kernel.

This goes against what we decided to support for the *only* publicly available VMM that cares about save/restore, which is that we only move forward and don't rollback.

Ah, I was definitely missing this context. Current behavior makes much more sense then.

Quoted 4 lines

Hypercalls are the least of your
worries, and there is a whole range of other architectural features that will have also appeared/disappeared (your own CNTPOFF series is a glaring example of this).

Quoted 4 lines

Isn't that a tad bit different though? I'll admit, I'm just as guilty with my own series forgetting to add a KVM_CAP (oops), but it is in my queue to kick out with the fix for nVHE/ptimer. Nonetheless, if a user takes up a new KVM UAPI, it is up to the user to run on a new kernel.

The two are linked. Exposing a new register to userspace and/or guest result in the same thing: you can't rollback. That's specially true in the QEMU case, which *learns* from the kernel what registers are available, and doesn't maintain a fixed list.

Quoted 4 lines

My concerns are explicitly with the 'under the nose' changes, where KVM modifies the guest feature set without userspace opting in. Based on your comment, though, it would appear that other parts of KVM are affected too.

Any new system register that is exposed by a new kernel feature breaks rollback. And so far, we only consider it a bug if the set of exposed registers reduces. Anything can be added safely (as checked by one of the selftests added by Drew).

< It doesn't have to be rollback safety, either. There may

Quoted 2 lines

simply be a hypercall which an operator doesn't want to give its guests, and it needs a way to tell KVM to hide it.

Fair enough. But this has to be done in a scalable way, which individual capability cannot provide.

Quoted 10 lines

Quoted 7 lines

Quoted 3 lines

Have I missed something blatantly obvious, or do others see this as an issue as well? I'll reply with an example of adding opt-out for PTP. I'm sure other hypercalls could be handled similarly.

Why do we need this? For future hypercalls, we could have some buy-in capabilities. For existing ones, it is too late, and negative features are just too horrible.

Oh, agreed on the nastiness. Lazy hack to realize the intended functional change..

Well, you definitely achieved your goal of attracting my attention :).

Quoted 11 lines

Quoted 4 lines

For KVM-specific hypercalls, we could get the VMM to save/restore the bitmap of supported functions. That would be "less horrible". This could be implemented using extra "firmware pseudo-registers" such as the ones described in Documentation/virt/kvm/arm/psci.rst.

This seems more reasonable, especially since we do this for migrating the guest's PSCI version.

Alternatively, I had thought about using a VM attribute, given the fact that it is non-architectural information and we avoid ABI issues in KVM_GET_REG_LIST without buy-in through a KVM_CAP.

The whole point is that these settings get exposed by KVM_GET_REG_LIST, as this is QEMU's way to dump a VM state. Given that we already have this for things like the spectre management state, we can just as well expose the bitmaps that deal with the KVM-specific hypercalls. After all, this falls into the realm of "KVM as VM firmware".

Quoted 3 lines

For ARM-architected hypercalls (TRNG, stolen time), we may need a similar extension.

Thanks for including me Marc. I think you've mentioned all the examples of why we don't generally expect N+1 -> N migrations to work that I can think of. While some of the examples like get-reg-list could eventually be eliminated if we had CPU models to tighten our machine type state, I think N+1 -> N migrations will always be best effort at most.

I agree with giving userspace control over the exposer of the hypercalls though. Using pseudo-registers for that purpose rather than a pile of CAPs also seems reasonable to me.

Quoted 7 lines

And, while I don't think this patch is going to proceed, I thought I'd point out that the opt-out approach doesn't help much with expanding our migration support unless we require the VMM to be upgraded first.

And, even then, the (N_kern, N+1_vmm) -> (N+1_kern, N_vmm) case won't work as expected, since the source enforce opt-out, but the destination won't.

Right, there's going to need to be a fence in both kernel and VMM versions. Before the fence, you can't rollback with either component. Once on the other side of the fence, the user may freely migrate between kernel + VMM combinations.

Quoted 5 lines

Also, since the VMM doesn't key off the kernel version, for the most part N+1 VMMs won't know when they're supposed to opt-out or not, leaving it to the user to ensure they consider everything. opt-in usually only needs the user to consider what machine type they want to launch.

Going the register route will implicitly require opt-out for all old hypercalls. We exposed them unconditionally to the guest before, and we must uphold that behavior. The default value for the bitmap will have those features set. Any hypercalls added after that register interface will then require explicit opt-in from userspace.

With regards to the pseudoregister interface, how would a VMM discover new bits? From my perspective, you need to have two bitmaps that the VMM can get at: the set of supported feature bits and the active bitmap of features for a running guest.

Quoted 125 lines · Oliver Upton

Quoted 99 lines · Andrew Jones

Quoted 83 lines · Marc Zyngier

Quoted 22 lines · Oliver Upton

Quoted 11 lines · Marc Zyngier

Quoted 7 lines

Exposing new hypercalls to guests in this manner seems very unsafe to me. Suppose an operator is trying to upgrade from kernel N to kernel N+1, which brings in the new 'widget' hypercall. Guests are live migrated onto the N+1 kernel, but the operator finds a defect that warrants a kernel rollback. VMs are then migrated from kernel N+1 -> N. Any guests that discovered the 'widget' hypercall are likely going to get fussy _very_ quickly on the old kernel.

This goes against what we decided to support for the *only* publicly available VMM that cares about save/restore, which is that we only move forward and don't rollback.

Ah, I was definitely missing this context. Current behavior makes much more sense then.

Quoted 4 lines

Hypercalls are the least of your
worries, and there is a whole range of other architectural features that will have also appeared/disappeared (your own CNTPOFF series is a glaring example of this).

Quoted 4 lines

Isn't that a tad bit different though? I'll admit, I'm just as guilty with my own series forgetting to add a KVM_CAP (oops), but it is in my queue to kick out with the fix for nVHE/ptimer. Nonetheless, if a user takes up a new KVM UAPI, it is up to the user to run on a new kernel.

The two are linked. Exposing a new register to userspace and/or guest result in the same thing: you can't rollback. That's specially true in the QEMU case, which *learns* from the kernel what registers are available, and doesn't maintain a fixed list.

Quoted 4 lines

My concerns are explicitly with the 'under the nose' changes, where KVM modifies the guest feature set without userspace opting in. Based on your comment, though, it would appear that other parts of KVM are affected too.

Any new system register that is exposed by a new kernel feature breaks rollback. And so far, we only consider it a bug if the set of exposed registers reduces. Anything can be added safely (as checked by one of the selftests added by Drew).

< It doesn't have to be rollback safety, either. There may

Quoted 2 lines

simply be a hypercall which an operator doesn't want to give its guests, and it needs a way to tell KVM to hide it.

Fair enough. But this has to be done in a scalable way, which individual capability cannot provide.

Quoted 10 lines

Quoted 7 lines

Quoted 3 lines

Have I missed something blatantly obvious, or do others see this as an issue as well? I'll reply with an example of adding opt-out for PTP. I'm sure other hypercalls could be handled similarly.

Why do we need this? For future hypercalls, we could have some buy-in capabilities. For existing ones, it is too late, and negative features are just too horrible.

Oh, agreed on the nastiness. Lazy hack to realize the intended functional change..

Well, you definitely achieved your goal of attracting my attention :).

Quoted 11 lines

Quoted 4 lines

For KVM-specific hypercalls, we could get the VMM to save/restore the bitmap of supported functions. That would be "less horrible". This could be implemented using extra "firmware pseudo-registers" such as the ones described in Documentation/virt/kvm/arm/psci.rst.

This seems more reasonable, especially since we do this for migrating the guest's PSCI version.

Alternatively, I had thought about using a VM attribute, given the fact that it is non-architectural information and we avoid ABI issues in KVM_GET_REG_LIST without buy-in through a KVM_CAP.

The whole point is that these settings get exposed by KVM_GET_REG_LIST, as this is QEMU's way to dump a VM state. Given that we already have this for things like the spectre management state, we can just as well expose the bitmaps that deal with the KVM-specific hypercalls. After all, this falls into the realm of "KVM as VM firmware".

Quoted 3 lines

For ARM-architected hypercalls (TRNG, stolen time), we may need a similar extension.

Thanks for including me Marc. I think you've mentioned all the examples of why we don't generally expect N+1 -> N migrations to work that I can think of. While some of the examples like get-reg-list could eventually be eliminated if we had CPU models to tighten our machine type state, I think N+1 -> N migrations will always be best effort at most.

I agree with giving userspace control over the exposer of the hypercalls though. Using pseudo-registers for that purpose rather than a pile of CAPs also seems reasonable to me.

Quoted 7 lines

And, while I don't think this patch is going to proceed, I thought I'd point out that the opt-out approach doesn't help much with expanding our migration support unless we require the VMM to be upgraded first.

And, even then, the (N_kern, N+1_vmm) -> (N+1_kern, N_vmm) case won't work as expected, since the source enforce opt-out, but the destination won't.

Right, there's going to need to be a fence in both kernel and VMM versions. Before the fence, you can't rollback with either component. Once on the other side of the fence, the user may freely migrate between kernel + VMM combinations.

Quoted 5 lines

Also, since the VMM doesn't key off the kernel version, for the most part N+1 VMMs won't know when they're supposed to opt-out or not, leaving it to the user to ensure they consider everything. opt-in usually only needs the user to consider what machine type they want to launch.

Going the register route will implicitly require opt-out for all old hypercalls. We exposed them unconditionally to the guest before, and we must uphold that behavior. The default value for the bitmap will have those features set. Any hypercalls added after that register interface will then require explicit opt-in from userspace.

Quoted 5 lines

With regards to the pseudoregister interface, how would a VMM discover new bits? From my perspective, you need to have two bitmaps that the VMM can get at: the set of supported feature bits and the active bitmap of features for a running guest.

I think we should model the pseudo-register approach off of x86's CPUID approach. x86 has specific get/set ioctls for CPUIDs (KVM_GET/SET_CPUID2), but I think we should get by with just get/set-one-reg. However, it might be nice/necessary to have something like x86's KVM_GET_SUPPORTED_CPUID which returns all the registers at once as a bitmap and the set bits would inform userspace of what's supported by the hardware and KVM.

So a new ioctl similar to KVM_GET_SUPPORTED_CPUID would be your first bitmap that shows what's supported and then userpace can determine what it wants to change and calculate the appropriate pseudo-registers to set/clear bits on with set-one-reg, as x86 would do with kvm-set-cpuid2.

I see Marc just replied stating we'll probably just have a single register. The KVM_GET_SUPPORTED_CPUID type of ioctl would be overkill in that case (get-one_reg is enough). Anyway it can always be added later if we expand into more registers.