KVM: arm64: Add support for hypercall services selection

Common hypercall firmware register handing is currently employed by psci.c. Since the upcoming patches add more of these registers, it's better to move the generic handling to hypercall.c for a cleaner presentation.

While we are at it, collect all the firmware registers under fw_reg_ids[] to help implement kvm_arm_get_fw_num_regs() and kvm_arm_copy_fw_reg_indices() in a generic way.

No functional change intended.

Signed-off-by Raghavendra Rao Ananta <[email protected]>

---
 arch/arm64/kvm/guest.c       |   2 +-
 arch/arm64/kvm/hypercalls.c  | 170 +++++++++++++++++++++++++++++++++++
 arch/arm64/kvm/psci.c        | 166 ----------------------------------
 include/kvm/arm_hypercalls.h |   7 ++
 include/kvm/arm_psci.h       |   7 --
 5 files changed, 178 insertions(+), 174 deletions(-)

arch/arm64/kvm/guest.c +1 -1

index 5ce26bedf23c..625f97f7b304 100644
@@ -18,7 +18,7 @@
 #include <linux/string.h>
 #include <linux/vmalloc.h>
 #include <linux/fs.h>
-#include <kvm/arm_psci.h>
+#include <kvm/arm_hypercalls.h>
 #include <asm/cputype.h>
 #include <linux/uaccess.h>
 #include <asm/fpsimd.h>

arch/arm64/kvm/hypercalls.c +8 -0

index 30da78f72b3b..9e136d91b470 100644
@@ -146,3 +146,173 @@ int kvm_hvc_call_handler(struct kvm_vcpu *vcpu)
 	smccc_set_retval(vcpu, val[0], val[1], val[2], val[3]);
 	return 1;
 }
+
+static const u64 fw_reg_ids[] = {
+	KVM_REG_ARM_PSCI_VERSION,
+	KVM_REG_ARM_SMCCC_ARCH_WORKAROUND_1,
+	KVM_REG_ARM_SMCCC_ARCH_WORKAROUND_2,
+};
+
+int kvm_arm_get_fw_num_regs(struct kvm_vcpu *vcpu)

Quoted 54 lines · Raghavendra Rao Ananta

Common hypercall firmware register handing is currently employed by psci.c. Since the upcoming patches add more of these registers, it's better to move the generic handling to hypercall.c for a cleaner presentation.

While we are at it, collect all the firmware registers under fw_reg_ids[] to help implement kvm_arm_get_fw_num_regs() and kvm_arm_copy_fw_reg_indices() in a generic way.

No functional change intended.

Signed-off-by Raghavendra Rao Ananta <[email protected]>

---
 arch/arm64/kvm/guest.c       |   2 +-
 arch/arm64/kvm/hypercalls.c  | 170 +++++++++++++++++++++++++++++++++++
 arch/arm64/kvm/psci.c        | 166 ----------------------------------
 include/kvm/arm_hypercalls.h |   7 ++
 include/kvm/arm_psci.h       |   7 --
 5 files changed, 178 insertions(+), 174 deletions(-)

arch/arm64/kvm/guest.c +1 -1

index 5ce26bedf23c..625f97f7b304 100644
@@ -18,7 +18,7 @@
 #include <linux/string.h>
 #include <linux/vmalloc.h>
 #include <linux/fs.h>
-#include <kvm/arm_psci.h>
+#include <kvm/arm_hypercalls.h>
 #include <asm/cputype.h>
 #include <linux/uaccess.h>
 #include <asm/fpsimd.h>

arch/arm64/kvm/hypercalls.c +13 -0

index 30da78f72b3b..9e136d91b470 100644
@@ -146,3 +146,173 @@ int kvm_hvc_call_handler(struct kvm_vcpu *vcpu)
      smccc_set_retval(vcpu, val[0], val[1], val[2], val[3]);
      return 1;
 }
+
+static const u64 fw_reg_ids[] = {
+     KVM_REG_ARM_PSCI_VERSION,
+     KVM_REG_ARM_SMCCC_ARCH_WORKAROUND_1,
+     KVM_REG_ARM_SMCCC_ARCH_WORKAROUND_2,
+};
+
+int kvm_arm_get_fw_num_regs(struct kvm_vcpu *vcpu)
+{
+     return ARRAY_SIZE(fw_reg_ids);
+}
+
+int kvm_arm_copy_fw_reg_indices(struct kvm_vcpu *vcpu, u64 __user *uindices)

arch/arm64/kvm/hypercalls.c int kvm_hvc_call_handler(struct kvm_vcpu *vcpu) · line 166

Quoted 5 lines

arch/arm64/kvm/hypercalls.c +5 -0

+{
+     int i;
+
+     for (i = 0; i < ARRAY_SIZE(fw_reg_ids); i++) {
+             if (put_user(fw_reg_ids[i], uindices))

This is missing the ++ on uindices, so it just writes the same offset three times.

arch/arm64/kvm/hypercalls.c int kvm_hvc_call_handler(struct kvm_vcpu *vcpu) · line 171

Quoted 5 lines

arch/arm64/kvm/hypercalls.c +5 -0

+                     return -EFAULT;
+     }
+
+     return 0;
+}

I assume the rest of the patch is just a cut+paste move of code.

Thanks,
drew

Architectures such as arm64 and riscv uses vcpu variables such as has_run_once and ran_atleast_once, respectively, to mark if the vCPU has started running. Since these are architecture agnostic variables, introduce kvm_vcpu_has_run_once() as a core kvm functionality and use this instead of the architecture defined variables.

No functional change intended.

Signed-off-by Raghavendra Rao Ananta <[email protected]>

Quoted 8 lines · Raghavendra Rao Ananta

Architectures such as arm64 and riscv uses vcpu variables such as has_run_once and ran_atleast_once, respectively, to mark if the vCPU has started running. Since these are architecture agnostic variables, introduce kvm_vcpu_has_run_once() as a core kvm functionality and use this instead of the architecture defined variables.

Quoted 3 lines

No functional change intended.

Signed-off-by Raghavendra Rao Ananta <[email protected]>

arm64 is moving away from this, see [1]. You also don't need any new state, as vcpu->pid gives you exactly what you need.

Happy to queue additional patches on top if you want to deal with riscv.

Thanks,

        M.

[1] https://lore.kernel.org/all/[email protected]/

--
Without deviation from the norm, progress is not possible.

The upcoming patches need a way to detect if the VM, as a whole, has started. Hence, unionize kvm_vcpu_has_run_once() of all the vcpus of the VM and build kvm_vm_has_run_once() to achieve the functionality.

No functional change intended.

Signed-off-by Raghavendra Rao Ananta <[email protected]>

---
 include/linux/kvm_host.h |  2 ++
 virt/kvm/kvm_main.c      | 17 +++++++++++++++++
 2 files changed, 19 insertions(+)

include/linux/kvm_host.h +2 -0

index b373929c71eb..102e00c0e21c 100644
@@ -1854,4 +1854,6 @@ static inline bool kvm_vcpu_has_run_once(struct kvm_vcpu *vcpu)
 	return vcpu->has_run_once;
 }
 
+bool kvm_vm_has_run_once(struct kvm *kvm);
+
 #endif

virt/kvm/kvm_main.c +6 -0

index 1ec8a8e959b2..3d8d96e8f61d 100644
@@ -4339,6 +4339,23 @@ static int kvm_vm_ioctl_get_stats_fd(struct kvm *kvm)
 	return fd;
 }
 
+bool kvm_vm_has_run_once(struct kvm *kvm)
+{
+	int i, ret = false;
+	struct kvm_vcpu *vcpu;
+
+	mutex_lock(&kvm->lock);

Quoted 46 lines · Raghavendra Rao Ananta

The upcoming patches need a way to detect if the VM, as a whole, has started. Hence, unionize kvm_vcpu_has_run_once() of all the vcpus of the VM and build kvm_vm_has_run_once() to achieve the functionality.

No functional change intended.

Signed-off-by Raghavendra Rao Ananta <[email protected]>

---
 include/linux/kvm_host.h |  2 ++
 virt/kvm/kvm_main.c      | 17 +++++++++++++++++
 2 files changed, 19 insertions(+)

include/linux/kvm_host.h +2 -0

index b373929c71eb..102e00c0e21c 100644
@@ -1854,4 +1854,6 @@ static inline bool kvm_vcpu_has_run_once(struct kvm_vcpu *vcpu)
      return vcpu->has_run_once;
 }
 
+bool kvm_vm_has_run_once(struct kvm *kvm);
+
 #endif

virt/kvm/kvm_main.c +13 -0

index 1ec8a8e959b2..3d8d96e8f61d 100644
@@ -4339,6 +4339,23 @@ static int kvm_vm_ioctl_get_stats_fd(struct kvm *kvm)
      return fd;
 }
 
+bool kvm_vm_has_run_once(struct kvm *kvm)
+{
+     int i, ret = false;
+     struct kvm_vcpu *vcpu;
+
+     mutex_lock(&kvm->lock);
+
+     kvm_for_each_vcpu(i, vcpu, kvm) {
+             ret = kvm_vcpu_has_run_once(vcpu);
+             if (ret)
+                     break;
+     }
+

virt/kvm/kvm_main.c static int kvm_vm_ioctl_get_stats_fd(struct kvm *kvm) · line 4357

Quoted 3 lines

virt/kvm/kvm_main.c +3 -0

+     mutex_unlock(&kvm->lock);
+     return ret;
+}

This is horribly racy. Nothing prevents a vcpu from running behind your back. If you want any sort of guarantee, look at what we do in kvm_vgic_create(). Alexandru has patches that extract it to make it generally available (at least for arm64).

        M.

--
Without deviation from the norm, progress is not possible.

KVM regularly introduces new hypercall services to the guests without any consent from the Virtual Machine Manager (VMM). This means, the guests can observe hypercall services in and out as they migrate across various host kernel versions. This could be a major problem if the guest discovered a hypercall, started using it, and after getting migrated to an older kernel realizes that it's no longer available. Depending on how the guest handles the change, there's a potential chance that the guest would just panic.

As a result, there's a need for the VMM to elect the services that it wishes the guest to discover. VMM can elect these services based on the kernels spread across its (migration) fleet. To remedy this, extend the existing firmware psuedo-registers, such as KVM_REG_ARM_PSCI_VERSION, for all the hypercall services available.

These firmware registers are categorized based on the service call owners, and unlike the existing firmware psuedo-registers, they hold the features supported in the form of a bitmap. During VM (vCPU) initialization, the registers shows an upper-limit of the features supported by the corresponding registers. The VMM can simply use GET_ONE_REG to discover the features. If it's unhappy with any of the features, it can simply write-back the desired feature bitmap using SET_ONE_REG.

KVM allows these modification only until a VM has started. KVM also assumes that the VMM is unaware of a register if a register remains unaccessed (read/write), and would simply clear all the bits of the registers such that the guest accidently doesn't get exposed to the features. Finally, the set of bitmaps from all the registers are the services that are exposed to the guest.

In order to provide backward compatibility with already existing VMMs, a new capability, KVM_CAP_ARM_HVC_FW_REG_BMAP, is introduced. To enable the bitmap firmware registers extension, the capability must be explicitly enabled. If not, the behavior is similar to the previous setup.

In this patch, the framework adds the register only for ARM's standard secure services (owner value 4). Currently, this includes support only for ARM True Random Number Generator (TRNG) service, with bit-0 of the register representing mandatory features of v1.0. Other services are momentarily added in the upcoming patches.

Signed-off-by Raghavendra Rao Ananta <[email protected]>

---
 arch/arm64/include/asm/kvm_host.h |  16 +++
 arch/arm64/include/uapi/asm/kvm.h |   4 +
 arch/arm64/kvm/arm.c              |  23 +++-
 arch/arm64/kvm/hypercalls.c       | 217 +++++++++++++++++++++++++++++-
 arch/arm64/kvm/trng.c             |   9 +-
 include/kvm/arm_hypercalls.h      |   7 +
 include/uapi/linux/kvm.h          |   1 +
 7 files changed, 262 insertions(+), 15 deletions(-)

arch/arm64/include/asm/kvm_host.h +2 -0

index 02dffe50a20c..1546a2f973ef 100644
@@ -102,6 +102,19 @@ struct kvm_s2_mmu {
 struct kvm_arch_memory_slot {
 };
 
+struct hvc_fw_reg_bmap {
+	bool accessed;

arch/arm64/include/asm/kvm_host.h +4 -0

+
 struct kvm_arch {
 	struct kvm_s2_mmu mmu;
 

@@ -137,6 +150,9 @@ struct kvm_arch {
 
 	/* Memory Tagging Extension enabled for the guest */
 	bool mte_enabled;
+
+	/* Hypercall firmware registers' descriptor */
+	struct hvc_reg_desc hvc_desc;
 };
 
 struct kvm_vcpu_fault_info {

arch/arm64/include/uapi/asm/kvm.h +4 -0

index b3edde68bc3e..d6e099ed14ef 100644
@@ -281,6 +281,10 @@ struct kvm_arm_copy_mte_tags {
 #define KVM_REG_ARM_SMCCC_ARCH_WORKAROUND_2_NOT_REQUIRED	3
 #define KVM_REG_ARM_SMCCC_ARCH_WORKAROUND_2_ENABLED     	(1U << 4)
 
+#define KVM_REG_ARM_STD_BMAP			KVM_REG_ARM_FW_REG(3)
+#define KVM_REG_ARM_STD_BIT_TRNG_V1_0		BIT(0)
+#define KVM_REG_ARM_STD_BMAP_BIT_MAX		0	/* Last valid bit */
+
 /* SVE registers */
 #define KVM_REG_ARM64_SVE		(0x15 << KVM_REG_ARM_COPROC_SHIFT)
 

arch/arm64/kvm/arm.c +3 -5

index 0cc148211b4e..f2099e4d1109 100644
@@ -81,26 +81,32 @@ int kvm_arch_check_processor_compat(void *opaque)
 int kvm_vm_ioctl_enable_cap(struct kvm *kvm,
 			    struct kvm_enable_cap *cap)
 {
-	int r;
+	int r = 0;
+	struct hvc_reg_desc *hvc_desc = &kvm->arch.hvc_desc;
 
 	if (cap->flags)
 		return -EINVAL;
 
 	switch (cap->cap) {
 	case KVM_CAP_ARM_NISV_TO_USER:
-		r = 0;
 		kvm->arch.return_nisv_io_abort_to_user = true;
 		break;
 	case KVM_CAP_ARM_MTE:
 		mutex_lock(&kvm->lock);
-		if (!system_supports_mte() || kvm->created_vcpus) {
+		if (!system_supports_mte() || kvm->created_vcpus)
 			r = -EINVAL;
-		} else {
-			r = 0;

arch/arm64/kvm/arm.c +6 -0

+		hvc_desc->fw_reg_bmap_enabled = true;
+		spin_unlock(&hvc_desc->lock);
+		break;
 	default:
 		r = -EINVAL;
 		break;

@@ -157,6 +163,8 @@ int kvm_arch_init_vm(struct kvm *kvm, unsigned long type)
 
 	set_default_spectre(kvm);
 
+	kvm_arm_init_hypercalls(kvm);
+
 	return ret;
 out_free_stage2_pgd:
 	kvm_free_stage2_pgd(&kvm->arch.mmu);

@@ -215,6 +223,7 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext)
 	case KVM_CAP_SET_GUEST_DEBUG:
 	case KVM_CAP_VCPU_ATTRIBUTES:
 	case KVM_CAP_PTP_KVM:
+	case KVM_CAP_ARM_HVC_FW_REG_BMAP:
 		r = 1;
 		break;
 	case KVM_CAP_SET_GUEST_DEBUG2:

arch/arm64/kvm/arm.c +1 -0

+
 	return ret;
 }
 

arch/arm64/kvm/hypercalls.c +23 -0

index 9e136d91b470..f5df7bc61146 100644
@@ -58,6 +58,41 @@ static void kvm_ptp_get_time(struct kvm_vcpu *vcpu, u64 *val)
 	val[3] = lower_32_bits(cycles);
 }
 
+static bool
+kvm_arm_fw_reg_feat_enabled(struct hvc_fw_reg_bmap *reg_bmap, u64 feat_bit)
+{
+	return reg_bmap->bmap & feat_bit;
+}
+
+bool kvm_hvc_call_supported(struct kvm_vcpu *vcpu, u32 func_id)
+{
+	struct hvc_reg_desc *hvc_desc = &vcpu->kvm->arch.hvc_desc;
+
+	/*
+	 * To ensure backward compatibility, support all the service calls,
+	 * including new additions, if the firmware registers holding the
+	 * feature bitmaps isn't explicitly enabled.
+	 */
+	if (!hvc_desc->fw_reg_bmap_enabled)
+		return true;
+
+	switch (func_id) {
+	case ARM_SMCCC_TRNG_VERSION:
+	case ARM_SMCCC_TRNG_FEATURES:
+	case ARM_SMCCC_TRNG_GET_UUID:
+	case ARM_SMCCC_TRNG_RND32:

arch/arm64/kvm/hypercalls.c +66 -0

+}
+
 int kvm_hvc_call_handler(struct kvm_vcpu *vcpu)
 {
 	u32 func_id = smccc_get_function(vcpu);

@@ -65,6 +100,9 @@ int kvm_hvc_call_handler(struct kvm_vcpu *vcpu)
 	u32 feature;
 	gpa_t gpa;
 
+	if (!kvm_hvc_call_supported(vcpu, func_id))
+		goto out;
+
 	switch (func_id) {
 	case ARM_SMCCC_VERSION_FUNC_ID:
 		val[0] = ARM_SMCCC_VERSION_1_1;

@@ -143,6 +181,7 @@ int kvm_hvc_call_handler(struct kvm_vcpu *vcpu)
 		return kvm_psci_call(vcpu);
 	}
 
+out:
 	smccc_set_retval(vcpu, val[0], val[1], val[2], val[3]);
 	return 1;
 }

@@ -153,17 +192,178 @@ static const u64 fw_reg_ids[] = {
 	KVM_REG_ARM_SMCCC_ARCH_WORKAROUND_2,
 };
 
+static const u64 fw_reg_bmap_ids[] = {
+	KVM_REG_ARM_STD_BMAP,
+};
+
+static void kvm_arm_fw_reg_init_hvc(struct hvc_reg_desc *hvc_desc,
+					struct hvc_fw_reg_bmap *fw_reg_bmap,
+					u64 reg_id, u64 default_map)
+{
+	fw_reg_bmap->reg_id = reg_id;
+	fw_reg_bmap->bmap = default_map;
+}
+
+void kvm_arm_init_hypercalls(struct kvm *kvm)
+{
+	struct hvc_reg_desc *hvc_desc = &kvm->arch.hvc_desc;
+
+	spin_lock_init(&hvc_desc->lock);
+
+	kvm_arm_fw_reg_init_hvc(hvc_desc, &hvc_desc->hvc_std_bmap,
+				KVM_REG_ARM_STD_BMAP, ARM_SMCCC_STD_FEATURES);
+}
+
+static void kvm_arm_fw_reg_sanitize(struct hvc_fw_reg_bmap *fw_reg_bmap)
+{
+	if (!fw_reg_bmap->accessed)
+		fw_reg_bmap->bmap = 0;
+}
+
+/*
+ * kvm_arm_sanitize_fw_regs: Sanitize the hypercall firmware registers
+ *
+ * Sanitization, in the case of hypercall firmware registers, is basically
+ * clearing out the feature bitmaps so that the guests are not exposed to
+ * the services corresponding to a particular register. The registers that
+ * needs sanitization is decided on two factors on the user-space part:
+ *	1. Enablement of KVM_CAP_ARM_HVC_FW_REG_BMAP:
+ *	   If the user-space hasn't enabled the capability, it either means
+ *	   that it's unaware of its existence, or it simply doesn't want to
+ *	   participate in the arrangement and is okay with the default settings.
+ *	   The former case is to ensure backward compatibility.
+ *
+ *	2. Has the user-space accessed (read/write) the register? :
+ *	   If yes, it means that the user-space is aware of the register's
+ *	   existence and can set the bits as it sees fit for the guest. A
+ *	   read-only access from user-space indicates that the user-space is
+ *	   happy with the default settings, and doesn't wish to change it.
+ *
+ * The logic for sanitizing a register will then be:
+ * ---------------------------------------------------------------------------
+ * | CAP enabled | Accessed reg | Clear reg | Comments                       |
+ * ---------------------------------------------------------------------------
+ * |      N      |       N      |     N     |                                |
+ * |      N      |       Y      |     N     | -ENOENT returned during access |
+ * |      Y      |       N      |     Y     |                                |
+ * |      Y      |       Y      |     N     |                                |
+ * ---------------------------------------------------------------------------
+ */
+void kvm_arm_sanitize_fw_regs(struct kvm *kvm)
+{
+	struct hvc_reg_desc *hvc_desc = &kvm->arch.hvc_desc;

arch/arm64/kvm/hypercalls.c +84 -2

+}
+
+static int kvm_arm_fw_reg_get_bmap(struct kvm *kvm,
+				struct hvc_fw_reg_bmap *fw_reg_bmap, u64 *val)
+{
+	int ret = 0;
+	struct hvc_reg_desc *hvc_desc = &kvm->arch.hvc_desc;
+
+	spin_lock(&hvc_desc->lock);
+
+	if (!hvc_desc->fw_reg_bmap_enabled) {
+		ret = -ENOENT;
+		goto out;
+	}
+
+	fw_reg_bmap->accessed = true;
+	*val = fw_reg_bmap->bmap;
+out:
+	spin_unlock(&hvc_desc->lock);
+	return ret;
+}
+
+static int kvm_arm_fw_reg_set_bmap(struct kvm *kvm,
+				struct hvc_fw_reg_bmap *fw_reg_bmap, u64 val)
+{
+	int ret = 0;
+	u64 fw_reg_features;
+	struct hvc_reg_desc *hvc_desc = &kvm->arch.hvc_desc;
+
+	spin_lock(&hvc_desc->lock);
+
+	if (!hvc_desc->fw_reg_bmap_enabled) {
+		ret = -ENOENT;
+		goto out;
+	}
+
+	if (fw_reg_bmap->bmap == val)
+		goto out;
+
+	if (kvm_vm_has_run_once(kvm)) {
+		ret = -EBUSY;
+		goto out;
+	}
+
+	switch (fw_reg_bmap->reg_id) {
+	case KVM_REG_ARM_STD_BMAP:
+		fw_reg_features = ARM_SMCCC_STD_FEATURES;
+		break;
+	default:
+		ret = -EINVAL;
+		goto out;
+	}
+
+	/* Check for unsupported feature bit */
+	if (val & ~fw_reg_features) {
+		ret = -EINVAL;
+		goto out;
+	}
+
+	fw_reg_bmap->accessed = true;
+	fw_reg_bmap->bmap = val;
+out:
+	spin_unlock(&hvc_desc->lock);
+	return ret;
+}
+
 int kvm_arm_get_fw_num_regs(struct kvm_vcpu *vcpu)
 {
-	return ARRAY_SIZE(fw_reg_ids);
+	struct hvc_reg_desc *hvc_desc = &vcpu->kvm->arch.hvc_desc;
+	int n_regs = ARRAY_SIZE(fw_reg_ids);
+
+	spin_lock(&hvc_desc->lock);
+
+	if (hvc_desc->fw_reg_bmap_enabled)
+		n_regs += ARRAY_SIZE(fw_reg_bmap_ids);
+
+	spin_unlock(&hvc_desc->lock);
+
+	return n_regs;
 }
 
 int kvm_arm_copy_fw_reg_indices(struct kvm_vcpu *vcpu, u64 __user *uindices)
 {
+	struct hvc_reg_desc *hvc_desc = &vcpu->kvm->arch.hvc_desc;
 	int i;
 
 	for (i = 0; i < ARRAY_SIZE(fw_reg_ids); i++) {
-		if (put_user(fw_reg_ids[i], uindices))
+		if (put_user(fw_reg_ids[i], uindices++))
+			return -EFAULT;
+	}
+
+	spin_lock(&hvc_desc->lock);
+

Quoted 63 lines · Raghavendra Rao Ananta

KVM regularly introduces new hypercall services to the guests without any consent from the Virtual Machine Manager (VMM). This means, the guests can observe hypercall services in and out as they migrate across various host kernel versions. This could be a major problem if the guest discovered a hypercall, started using it, and after getting migrated to an older kernel realizes that it's no longer available. Depending on how the guest handles the change, there's a potential chance that the guest would just panic.

As a result, there's a need for the VMM to elect the services that it wishes the guest to discover. VMM can elect these services based on the kernels spread across its (migration) fleet. To remedy this, extend the existing firmware psuedo-registers, such as KVM_REG_ARM_PSCI_VERSION, for all the hypercall services available.

These firmware registers are categorized based on the service call owners, and unlike the existing firmware psuedo-registers, they hold the features supported in the form of a bitmap. During VM (vCPU) initialization, the registers shows an upper-limit of the features supported by the corresponding registers. The VMM can simply use GET_ONE_REG to discover the features. If it's unhappy with any of the features, it can simply write-back the desired feature bitmap using SET_ONE_REG.

KVM allows these modification only until a VM has started. KVM also assumes that the VMM is unaware of a register if a register remains unaccessed (read/write), and would simply clear all the bits of the registers such that the guest accidently doesn't get exposed to the features. Finally, the set of bitmaps from all the registers are the services that are exposed to the guest.

In order to provide backward compatibility with already existing VMMs, a new capability, KVM_CAP_ARM_HVC_FW_REG_BMAP, is introduced. To enable the bitmap firmware registers extension, the capability must be explicitly enabled. If not, the behavior is similar to the previous setup.

In this patch, the framework adds the register only for ARM's standard secure services (owner value 4). Currently, this includes support only for ARM True Random Number Generator (TRNG) service, with bit-0 of the register representing mandatory features of v1.0. Other services are momentarily added in the upcoming patches.

Signed-off-by Raghavendra Rao Ananta <[email protected]>

---
 arch/arm64/include/asm/kvm_host.h |  16 +++
 arch/arm64/include/uapi/asm/kvm.h |   4 +
 arch/arm64/kvm/arm.c              |  23 +++-
 arch/arm64/kvm/hypercalls.c       | 217 +++++++++++++++++++++++++++++-
 arch/arm64/kvm/trng.c             |   9 +-
 include/kvm/arm_hypercalls.h      |   7 +
 include/uapi/linux/kvm.h          |   1 +
 7 files changed, 262 insertions(+), 15 deletions(-)

arch/arm64/include/asm/kvm_host.h +0 -0

index 02dffe50a20c..1546a2f973ef 100644
@@ -102,6 +102,19 @@ struct kvm_s2_mmu {
 struct kvm_arch_memory_slot {
 };
 

arch/arm64/include/asm/kvm_host.h struct kvm_s2_mmu { · line 116

Quoted 12 lines

arch/arm64/include/asm/kvm_host.h +12 -0

+struct hvc_fw_reg_bmap {
+     bool accessed;
+     u64 reg_id;
+     u64 bmap;
+};
+
+struct hvc_reg_desc {
+     spinlock_t lock;
+     bool fw_reg_bmap_enabled;
+
+     struct hvc_fw_reg_bmap hvc_std_bmap;
+};

Please document what these data structures track. Without any documentation, it is pretty difficult to build a mental picture of how this all fits together.

Quoted 62 lines

arch/arm64/include/asm/kvm_host.h +4 -0

+
 struct kvm_arch {
      struct kvm_s2_mmu mmu;
 

@@ -137,6 +150,9 @@ struct kvm_arch {
 
      /* Memory Tagging Extension enabled for the guest */
      bool mte_enabled;
+
+     /* Hypercall firmware registers' descriptor */
+     struct hvc_reg_desc hvc_desc;
 };
 
 struct kvm_vcpu_fault_info {

arch/arm64/include/uapi/asm/kvm.h +4 -0

index b3edde68bc3e..d6e099ed14ef 100644
@@ -281,6 +281,10 @@ struct kvm_arm_copy_mte_tags {
 #define KVM_REG_ARM_SMCCC_ARCH_WORKAROUND_2_NOT_REQUIRED     3
 #define KVM_REG_ARM_SMCCC_ARCH_WORKAROUND_2_ENABLED          (1U << 4)
 
+#define KVM_REG_ARM_STD_BMAP                 KVM_REG_ARM_FW_REG(3)
+#define KVM_REG_ARM_STD_BIT_TRNG_V1_0                BIT(0)
+#define KVM_REG_ARM_STD_BMAP_BIT_MAX         0       /* Last valid bit */
+
 /* SVE registers */
 #define KVM_REG_ARM64_SVE            (0x15 << KVM_REG_ARM_COPROC_SHIFT)
 

arch/arm64/kvm/arm.c +5 -6

index 0cc148211b4e..f2099e4d1109 100644
@@ -81,26 +81,32 @@ int kvm_arch_check_processor_compat(void *opaque)
 int kvm_vm_ioctl_enable_cap(struct kvm *kvm,
                          struct kvm_enable_cap *cap)
 {
-     int r;
+     int r = 0;
+     struct hvc_reg_desc *hvc_desc = &kvm->arch.hvc_desc;
 
      if (cap->flags)
              return -EINVAL;
 
      switch (cap->cap) {
      case KVM_CAP_ARM_NISV_TO_USER:
-             r = 0;
              kvm->arch.return_nisv_io_abort_to_user = true;
              break;
      case KVM_CAP_ARM_MTE:
              mutex_lock(&kvm->lock);
-             if (!system_supports_mte() || kvm->created_vcpus) {
+             if (!system_supports_mte() || kvm->created_vcpus)
                      r = -EINVAL;
-             } else {
-                     r = 0;
+             else
                      kvm->arch.mte_enabled = true;
-             }
              mutex_unlock(&kvm->lock);
              break;
+     case KVM_CAP_ARM_HVC_FW_REG_BMAP:

arch/arm64/kvm/arm.c int kvm_arch_check_processor_compat(void *opaque) · line 106

Quoted 4 lines

arch/arm64/kvm/arm.c +4 -0

+             if (kvm_vm_has_run_once(kvm))
+                     return -EBUSY;
+
+             spin_lock(&hvc_desc->lock);

Does this really need to be a spin-lock? Are you ever taking it on a context where you cannot sleep? And why does it need to be a new lock when we already have a plethora of them?

Quoted 27 lines

arch/arm64/kvm/arm.c +6 -0

+             hvc_desc->fw_reg_bmap_enabled = true;
+             spin_unlock(&hvc_desc->lock);
+             break;
      default:
              r = -EINVAL;
              break;

@@ -157,6 +163,8 @@ int kvm_arch_init_vm(struct kvm *kvm, unsigned long type)
 
      set_default_spectre(kvm);
 
+     kvm_arm_init_hypercalls(kvm);
+
      return ret;
 out_free_stage2_pgd:
      kvm_free_stage2_pgd(&kvm->arch.mmu);

@@ -215,6 +223,7 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext)
      case KVM_CAP_SET_GUEST_DEBUG:
      case KVM_CAP_VCPU_ATTRIBUTES:
      case KVM_CAP_PTP_KVM:
+     case KVM_CAP_ARM_HVC_FW_REG_BMAP:
              r = 1;
              break;
      case KVM_CAP_SET_GUEST_DEBUG2:

@@ -622,6 +631,8 @@ static int kvm_vcpu_first_run_init(struct kvm_vcpu *vcpu)
      if (kvm_vm_is_protected(kvm))
              kvm_call_hyp_nvhe(__pkvm_vcpu_init_traps, vcpu);
 

arch/arm64/kvm/arm.c static int kvm_vcpu_first_run_init(struct kvm_vcpu *vcpu) · line 634

Quoted 1 line

arch/arm64/kvm/arm.c +1 -0

+     kvm_arm_sanitize_fw_regs(kvm);

What is the rational for doing this on first run? Surely this could be done at the point where userspace performs the write, couldn't it?

My mental model is that the VMM reads some data, clears some bits if it really wants to, and writes it back. There shouldn't be anything to sanitise after the facts. Or at least that's my gut feeling so far.

/me reads on.

Quoted 39 lines

arch/arm64/kvm/arm.c +1 -0

+
      return ret;
 }
 

arch/arm64/kvm/hypercalls.c +27 -0

index 9e136d91b470..f5df7bc61146 100644
@@ -58,6 +58,41 @@ static void kvm_ptp_get_time(struct kvm_vcpu *vcpu, u64 *val)
      val[3] = lower_32_bits(cycles);
 }
 
+static bool
+kvm_arm_fw_reg_feat_enabled(struct hvc_fw_reg_bmap *reg_bmap, u64 feat_bit)
+{
+     return reg_bmap->bmap & feat_bit;
+}
+
+bool kvm_hvc_call_supported(struct kvm_vcpu *vcpu, u32 func_id)
+{
+     struct hvc_reg_desc *hvc_desc = &vcpu->kvm->arch.hvc_desc;
+
+     /*
+      * To ensure backward compatibility, support all the service calls,
+      * including new additions, if the firmware registers holding the
+      * feature bitmaps isn't explicitly enabled.
+      */
+     if (!hvc_desc->fw_reg_bmap_enabled)
+             return true;
+
+     switch (func_id) {
+     case ARM_SMCCC_TRNG_VERSION:
+     case ARM_SMCCC_TRNG_FEATURES:
+     case ARM_SMCCC_TRNG_GET_UUID:
+     case ARM_SMCCC_TRNG_RND32:
+     case ARM_SMCCC_TRNG_RND64:
+             return kvm_arm_fw_reg_feat_enabled(&hvc_desc->hvc_std_bmap,
+                                     KVM_REG_ARM_STD_BIT_TRNG_V1_0);
+     default:

arch/arm64/kvm/hypercalls.c static void kvm_ptp_get_time(struct kvm_vcpu *vcpu, u64 *val) · line 93

Quoted 6 lines

arch/arm64/kvm/hypercalls.c +6 -0

+             /* By default, allow the services that aren't listed here */
+             return true;
+     }
+
+     /* We shouldn't be reaching here */
+     return true;

So why have anything at all?

Quoted 87 lines

arch/arm64/kvm/hypercalls.c +66 -0

+}
+
 int kvm_hvc_call_handler(struct kvm_vcpu *vcpu)
 {
      u32 func_id = smccc_get_function(vcpu);

@@ -65,6 +100,9 @@ int kvm_hvc_call_handler(struct kvm_vcpu *vcpu)
      u32 feature;
      gpa_t gpa;
 
+     if (!kvm_hvc_call_supported(vcpu, func_id))
+             goto out;
+
      switch (func_id) {
      case ARM_SMCCC_VERSION_FUNC_ID:
              val[0] = ARM_SMCCC_VERSION_1_1;

@@ -143,6 +181,7 @@ int kvm_hvc_call_handler(struct kvm_vcpu *vcpu)
              return kvm_psci_call(vcpu);
      }
 
+out:
      smccc_set_retval(vcpu, val[0], val[1], val[2], val[3]);
      return 1;
 }

@@ -153,17 +192,178 @@ static const u64 fw_reg_ids[] = {
      KVM_REG_ARM_SMCCC_ARCH_WORKAROUND_2,
 };
 
+static const u64 fw_reg_bmap_ids[] = {
+     KVM_REG_ARM_STD_BMAP,
+};
+
+static void kvm_arm_fw_reg_init_hvc(struct hvc_reg_desc *hvc_desc,
+                                     struct hvc_fw_reg_bmap *fw_reg_bmap,
+                                     u64 reg_id, u64 default_map)
+{
+     fw_reg_bmap->reg_id = reg_id;
+     fw_reg_bmap->bmap = default_map;
+}
+
+void kvm_arm_init_hypercalls(struct kvm *kvm)
+{
+     struct hvc_reg_desc *hvc_desc = &kvm->arch.hvc_desc;
+
+     spin_lock_init(&hvc_desc->lock);
+
+     kvm_arm_fw_reg_init_hvc(hvc_desc, &hvc_desc->hvc_std_bmap,
+                             KVM_REG_ARM_STD_BMAP, ARM_SMCCC_STD_FEATURES);
+}
+
+static void kvm_arm_fw_reg_sanitize(struct hvc_fw_reg_bmap *fw_reg_bmap)
+{
+     if (!fw_reg_bmap->accessed)
+             fw_reg_bmap->bmap = 0;
+}
+
+/*
+ * kvm_arm_sanitize_fw_regs: Sanitize the hypercall firmware registers
+ *
+ * Sanitization, in the case of hypercall firmware registers, is basically
+ * clearing out the feature bitmaps so that the guests are not exposed to
+ * the services corresponding to a particular register. The registers that
+ * needs sanitization is decided on two factors on the user-space part:
+ *   1. Enablement of KVM_CAP_ARM_HVC_FW_REG_BMAP:
+ *      If the user-space hasn't enabled the capability, it either means
+ *      that it's unaware of its existence, or it simply doesn't want to
+ *      participate in the arrangement and is okay with the default settings.
+ *      The former case is to ensure backward compatibility.
+ *
+ *   2. Has the user-space accessed (read/write) the register? :
+ *      If yes, it means that the user-space is aware of the register's
+ *      existence and can set the bits as it sees fit for the guest. A
+ *      read-only access from user-space indicates that the user-space is
+ *      happy with the default settings, and doesn't wish to change it.
+ *
+ * The logic for sanitizing a register will then be:
+ * ---------------------------------------------------------------------------
+ * | CAP enabled | Accessed reg | Clear reg | Comments                       |
+ * ---------------------------------------------------------------------------
+ * |      N      |       N      |     N     |                                |
+ * |      N      |       Y      |     N     | -ENOENT returned during access |
+ * |      Y      |       N      |     Y     |                                |
+ * |      Y      |       Y      |     N     |                                |
+ * ---------------------------------------------------------------------------
+ */
+void kvm_arm_sanitize_fw_regs(struct kvm *kvm)
+{
+     struct hvc_reg_desc *hvc_desc = &kvm->arch.hvc_desc;

arch/arm64/kvm/hypercalls.c static const u64 fw_reg_ids[] = { · line 264

Quoted 10 lines

arch/arm64/kvm/hypercalls.c +10 -0

+
+     spin_lock(&hvc_desc->lock);
+
+     if (!hvc_desc->fw_reg_bmap_enabled)
+             goto out;
+
+     kvm_arm_fw_reg_sanitize(&hvc_desc->hvc_std_bmap);
+
+out:
+     spin_unlock(&hvc_desc->lock);

I keep being baffled by this. Why should we track the VMM accesses or the VMM writeback? This logic doesn't seem to bring anything useful as far as I can tell. All we need to ensure is that what is written to the pseudo-register is an acceptable subset of the previous value, and I cannot see why this can't be done at write-time.

If you want to hide this behind a capability, fine (although my guts feeling is that we don't need that either). But I really want to be convinced about all this tracking.

Quoted 101 lines

arch/arm64/kvm/hypercalls.c +90 -2

+}
+
+static int kvm_arm_fw_reg_get_bmap(struct kvm *kvm,
+                             struct hvc_fw_reg_bmap *fw_reg_bmap, u64 *val)
+{
+     int ret = 0;
+     struct hvc_reg_desc *hvc_desc = &kvm->arch.hvc_desc;
+
+     spin_lock(&hvc_desc->lock);
+
+     if (!hvc_desc->fw_reg_bmap_enabled) {
+             ret = -ENOENT;
+             goto out;
+     }
+
+     fw_reg_bmap->accessed = true;
+     *val = fw_reg_bmap->bmap;
+out:
+     spin_unlock(&hvc_desc->lock);
+     return ret;
+}
+
+static int kvm_arm_fw_reg_set_bmap(struct kvm *kvm,
+                             struct hvc_fw_reg_bmap *fw_reg_bmap, u64 val)
+{
+     int ret = 0;
+     u64 fw_reg_features;
+     struct hvc_reg_desc *hvc_desc = &kvm->arch.hvc_desc;
+
+     spin_lock(&hvc_desc->lock);
+
+     if (!hvc_desc->fw_reg_bmap_enabled) {
+             ret = -ENOENT;
+             goto out;
+     }
+
+     if (fw_reg_bmap->bmap == val)
+             goto out;
+
+     if (kvm_vm_has_run_once(kvm)) {
+             ret = -EBUSY;
+             goto out;
+     }
+
+     switch (fw_reg_bmap->reg_id) {
+     case KVM_REG_ARM_STD_BMAP:
+             fw_reg_features = ARM_SMCCC_STD_FEATURES;
+             break;
+     default:
+             ret = -EINVAL;
+             goto out;
+     }
+
+     /* Check for unsupported feature bit */
+     if (val & ~fw_reg_features) {
+             ret = -EINVAL;
+             goto out;
+     }
+
+     fw_reg_bmap->accessed = true;
+     fw_reg_bmap->bmap = val;
+out:
+     spin_unlock(&hvc_desc->lock);
+     return ret;
+}
+
 int kvm_arm_get_fw_num_regs(struct kvm_vcpu *vcpu)
 {
-     return ARRAY_SIZE(fw_reg_ids);
+     struct hvc_reg_desc *hvc_desc = &vcpu->kvm->arch.hvc_desc;
+     int n_regs = ARRAY_SIZE(fw_reg_ids);
+
+     spin_lock(&hvc_desc->lock);
+
+     if (hvc_desc->fw_reg_bmap_enabled)
+             n_regs += ARRAY_SIZE(fw_reg_bmap_ids);
+
+     spin_unlock(&hvc_desc->lock);
+
+     return n_regs;
 }
 
 int kvm_arm_copy_fw_reg_indices(struct kvm_vcpu *vcpu, u64 __user *uindices)
 {
+     struct hvc_reg_desc *hvc_desc = &vcpu->kvm->arch.hvc_desc;
      int i;
 
      for (i = 0; i < ARRAY_SIZE(fw_reg_ids); i++) {
-             if (put_user(fw_reg_ids[i], uindices))
+             if (put_user(fw_reg_ids[i], uindices++))
+                     return -EFAULT;
+     }
+
+     spin_lock(&hvc_desc->lock);
+
+     if (!hvc_desc->fw_reg_bmap_enabled) {
+             spin_unlock(&hvc_desc->lock);
+             return 0;
+     }
+
+     spin_unlock(&hvc_desc->lock);

arch/arm64/kvm/hypercalls.c static const u64 fw_reg_ids[] = { · line 367

Quoted 4 lines

arch/arm64/kvm/hypercalls.c +3 -0

+
+     for (i = 0; i < ARRAY_SIZE(fw_reg_bmap_ids); i++) {
+             if (put_user(fw_reg_bmap_ids[i], uindices++))
                      return -EFAULT;

I really don't get what you are trying to achieve with this locking. You guard the 'enabled' bit, but you still allow the kernel view to be copied to userspace while another thread is writing to it?

Thanks,

        M.

--
Without deviation from the norm, progress is not possible.

Quoted 10 lines · Marc Zyngier

I keep being baffled by this. Why should we track the VMM accesses or the VMM writeback? This logic doesn't seem to bring anything useful as far as I can tell. All we need to ensure is that what is written to the pseudo-register is an acceptable subset of the previous value, and I cannot see why this can't be done at write-time.

If you want to hide this behind a capability, fine (although my guts feeling is that we don't need that either). But I really want to be convinced about all this tracking.

The tracking of each owner register is necessary here to safe-guard the possibility that the user-space may not be aware of a newly introduced register, and hence, hasn't accessed it. If it had at least read the register, but not write-back, we assume that the user-space is happy with the configuration. But the fact that the register has not even been read would state that user-space is unaware of the existence of this new register. In such a case, if we don't sanitize (clear all the bits) this register, the features will be exposed unconditionally to the guest.

The capability is introduced here to make sure that this new infrastructure is backward compatible with old VMMs. If the VMMs don't enable this capability, they are probably unaware of this, and this will work as it always has- expose new services to the guest unconditionally as and when they are introduced.

Quoted 21 lines · Raghavendra Rao Ananta

Quoted 10 lines · Marc Zyngier

I keep being baffled by this. Why should we track the VMM accesses or the VMM writeback? This logic doesn't seem to bring anything useful as far as I can tell. All we need to ensure is that what is written to the pseudo-register is an acceptable subset of the previous value, and I cannot see why this can't be done at write-time.

If you want to hide this behind a capability, fine (although my guts feeling is that we don't need that either). But I really want to be convinced about all this tracking.

The tracking of each owner register is necessary here to safe-guard the possibility that the user-space may not be aware of a newly introduced register, and hence, hasn't accessed it. If it had at least read the register, but not write-back, we assume that the user-space is happy with the configuration. But the fact that the register has not even been read would state that user-space is unaware of the existence of this new register. In such a case, if we don't sanitize (clear all the bits) this register, the features will be exposed unconditionally to the guest.

Quoted 5 lines

The capability is introduced here to make sure that this new infrastructure is backward compatible with old VMMs. If the VMMs don't enable this capability, they are probably unaware of this, and this will work as it always has- expose new services to the guest unconditionally as and when they are introduced.

Hi Raghavendra,

I don't think we need a CAP that has to be enabled or to make any assumptions or policy decisions in the kernel. I think we just need to provide a bit more information to the VMM when it checks if KVM has the CAP. If KVM would tell the VMM how may pseudo registers there are, which can be done with the return value of the CAP, then the VMM code could be something like this

  r = check_cap(KVM_CAP_ARM_HVC_FW_REG_BMAP);
  if (r) {
    num_regs = r;

    for (idx = 0; idx < num_regs; ++idx) {
      reg = hvc_fw_reg(idx);

      if (idx > vmm_last_known_idx) {
        ...
      } else {
        ...
      }
    }

}

With this, the VMM is free to decide if it wants to clear all registers greater than the last index it was aware of or if it wants to let those registers just get exposed to the guest without knowing what's getting exposed. Along with documenting that by default everything gets exposed by KVM, which is the backwards compatible thing to do, then the VMM has been warned and given everything it needs to manage its guests.

Another thing that might be nice is giving userspace control of how many pseudo registers show up in get-reg-list. In order to migrate from a host with a more recent KVM to a host with an older KVM[*] we should only expose the number of pseudo registers that the older host is aware of. The VMM would zero these registers out anyway, in order to be compatible for migration, but that's not enough when they also show up in the list (at least not with QEMU that aborts migration when the destination expects less registers than what get-reg-list provides)

[*] This isn't a great idea, but it'd be nice if we can make it work, because users may want to rollback upgrades or, after migrating to a host with a newer kernel, they may want to migrate back to where they started.

Thanks,
drew

Quoted 51 lines · Andrew Jones

Quoted 21 lines · Raghavendra Rao Ananta

Quoted 10 lines · Marc Zyngier

I keep being baffled by this. Why should we track the VMM accesses or the VMM writeback? This logic doesn't seem to bring anything useful as far as I can tell. All we need to ensure is that what is written to the pseudo-register is an acceptable subset of the previous value, and I cannot see why this can't be done at write-time.

If you want to hide this behind a capability, fine (although my guts feeling is that we don't need that either). But I really want to be convinced about all this tracking.

The tracking of each owner register is necessary here to safe-guard the possibility that the user-space may not be aware of a newly introduced register, and hence, hasn't accessed it. If it had at least read the register, but not write-back, we assume that the user-space is happy with the configuration. But the fact that the register has not even been read would state that user-space is unaware of the existence of this new register. In such a case, if we don't sanitize (clear all the bits) this register, the features will be exposed unconditionally to the guest.

Quoted 5 lines

The capability is introduced here to make sure that this new infrastructure is backward compatible with old VMMs. If the VMMs don't enable this capability, they are probably unaware of this, and this will work as it always has- expose new services to the guest unconditionally as and when they are introduced.

Hi Raghavendra,

I don't think we need a CAP that has to be enabled or to make any assumptions or policy decisions in the kernel. I think we just need to provide a bit more information to the VMM when it checks if KVM has the CAP. If KVM would tell the VMM how may pseudo registers there are, which can be done with the return value of the CAP, then the VMM code could be something like this

  r = check_cap(KVM_CAP_ARM_HVC_FW_REG_BMAP);
  if (r) {
    num_regs = r;

    for (idx = 0; idx < num_regs; ++idx) {
      reg = hvc_fw_reg(idx);

      if (idx > vmm_last_known_idx) {
        ...
      } else {
        ...
      }
    }

Quoted 9 lines

}

With this, the VMM is free to decide if it wants to clear all registers greater than the last index it was aware of or if it wants to let those registers just get exposed to the guest without knowing what's getting exposed. Along with documenting that by default everything gets exposed by KVM, which is the backwards compatible thing to do, then the VMM has been warned and given everything it needs to manage its guests.

Hi Andrew,

Thanks for your comments and suggestions!

I like the idea of sharing info via a read of the CAP, and not having to explicitly sanitize/clear the registers before the guest begins to run.
However the handshake is done over an API doc, which is a little concerning. The user-space must remember and explicitly clear any new register that it doesn't want to expose to the guest, while the current approach does this automatically.
Any bug in VMM's implementation could be risky and unintentionally expose features to the guest. What do you think?

Quoted 10 lines

Another thing that might be nice is giving userspace control of how many pseudo registers show up in get-reg-list. In order to migrate from a host with a more recent KVM to a host with an older KVM[*] we should only expose the number of pseudo registers that the older host is aware of. The VMM would zero these registers out anyway, in order to be compatible for migration, but that's not enough when they also show up in the list (at least not with QEMU that aborts migration when the destination expects less registers than what get-reg-list provides)

Quoted 5 lines

[*] This isn't a great idea, but it'd be nice if we can make it work, because users may want to rollback upgrades or, after migrating to a host with a newer kernel, they may want to migrate back to where they started.

Good point. But IIUC, if the user-space is able to communicate the info that it's expecting a certain get-reg-list, do you think it can handle it at its end too, rather than relying on the kernel to send a list back?

My assumption was that VMM would statically maintain a known set of registers that it wants to work with and are to be modified by hand, rather than relying on get-reg-list. This could be the least common set of registers that are present in all the host kernels (higher or lower versions) of the migration fleet. This config doesn't change even with get-reg-list declaring a new register as the features exposed by it could still be untested. Although, migrating to a host with a missing register shouldn't be possible in this setting, but if it encounters the scenario, it should be able to avoid migration to the host (similar to QEMU).

Please correct me if you think it's a false assumption to proceed with.