Skip to contentSkip to conversation

KVM: arm64: mixed-width check should be skipped for uninitialized vCPUs

PATCH In review Reiji Watanabe 2 participants 8 messages Jan 18, 2022 – Feb 11, 2022

Subsystems (4)
lore
Share & tools
lore
#0 lore raw

PATCH v2 1/2 KVM: arm64: mixed-width check should be skipped for uninitialized vCPUs

KVM allows userspace to configure either all 32bit or 64bit vCPUs for a guest. At vCPU reset, vcpu_allowed_register_width() checks if the vcpu's register width is consistent with all other vCPUs'. Since the checking is done even against vCPUs that are not initialized (KVM_ARM_VCPU_INIT has not been done) yet, the uninitialized vCPUs are erroneously treated as 64bit vCPU, which causes the function to incorrectly detect a mixed-width VM.

Introduce a new flag (el1_reg_width) in kvm_arch to indicates that the guest needs to be configured with all 32bit or 64bit vCPUs, and initialize it at the first KVM_ARM_VCPU_INIT for the guest. Check vcpu's register width against the flag at the vcpu's KVM_ARM_VCPU_INIT (instead of against other vCPUs' register width).

Fixes 66e94d5cafd4 ("KVM: arm64: Prevent mixed-width VM creation")
---
 arch/arm64/include/asm/kvm_host.h | 13 +++++++++++++
 arch/arm64/kvm/arm.c              | 30 ++++++++++++++++++++++++++++++
 arch/arm64/kvm/reset.c            |  8 --------
 3 files changed, 43 insertions(+), 8 deletions(-)
arch/arm64/include/asm/kvm_host.h +13 -0
index 2a5f7f38006f..c02b7caf2c82 100644
@@ -102,6 +102,12 @@ struct kvm_s2_mmu {
 struct kvm_arch_memory_slot {
 };
 
+enum kvm_el1_reg_width {
+	EL1_WIDTH_UNINITIALIZED = 0,
+	EL1_32BIT,
+	EL1_64BIT,
+};
+
 struct kvm_arch {
 	struct kvm_s2_mmu mmu;
 
@@ -137,6 +143,13 @@ struct kvm_arch {
 
 	/* Memory Tagging Extension enabled for the guest */
 	bool mte_enabled;
+
+	/*
+	 * EL1 register width for the guest.
+	 * This is set at the first KVM_ARM_VCPU_INIT for the guest based
+	 * on whether the vcpu has KVM_ARM_VCPU_EL1_32BIT or not.
+	 */
+	enum kvm_el1_reg_width el1_reg_width;
 };
 
 struct kvm_vcpu_fault_info {
arch/arm64/kvm/arm.c +30 -0
index e4727dc771bf..54ae8bf9d187 100644
@@ -1058,6 +1058,32 @@ int kvm_vm_ioctl_irq_line(struct kvm *kvm, struct kvm_irq_level *irq_level,
 	return -EINVAL;
 }
 
+/*
+ * A guest can have either all 32bit or 64bit vcpus only.
+ * Either one the guest has is indicated in kvm->arch.el1_reg_width.
+ * Check if the vcpu's register width is consistent with
+ * kvm->arch.el1_reg_width.  If kvm->arch.el1_reg_width is not set yet,
+ * set it based on the vcpu's KVM_ARM_VCPU_EL1_32BIT configuration.
+ */
+static int kvm_register_width_check_or_init(struct kvm_vcpu *vcpu)
+{
+	bool is32bit;
+	bool allowed = true;
+	struct kvm *kvm = vcpu->kvm;
+
+	is32bit = vcpu_has_feature(vcpu, KVM_ARM_VCPU_EL1_32BIT);
+
+	mutex_lock(&kvm->lock);
+
+	if (kvm->arch.el1_reg_width == EL1_WIDTH_UNINITIALIZED)
+		kvm->arch.el1_reg_width = is32bit ? EL1_32BIT : EL1_64BIT;
+	else
+		allowed = (is32bit == (kvm->arch.el1_reg_width == EL1_32BIT));
+
+	mutex_unlock(&kvm->lock);
+	return allowed ? 0 : -EINVAL;
+}
+
 static int kvm_vcpu_set_target(struct kvm_vcpu *vcpu,
 			       const struct kvm_vcpu_init *init)
 {
@@ -1097,6 +1123,10 @@ static int kvm_vcpu_set_target(struct kvm_vcpu *vcpu,
 
 	/* Now we know what it is, we can reset it. */
 	ret = kvm_reset_vcpu(vcpu);
+
+	if (!ret)
+		ret = kvm_register_width_check_or_init(vcpu);
+
 	if (ret) {
 		vcpu->arch.target = -1;
 		bitmap_zero(vcpu->arch.features, KVM_VCPU_MAX_FEATURES);
arch/arm64/kvm/reset.c +0 -8
index 426bd7fbc3fd..dbf2939a6a96 100644
@@ -168,9 +168,7 @@ static int kvm_vcpu_enable_ptrauth(struct kvm_vcpu *vcpu)
 
 static bool vcpu_allowed_register_width(struct kvm_vcpu *vcpu)
 {
-	struct kvm_vcpu *tmp;
 	bool is32bit;
-	int i;
 
 	is32bit = vcpu_has_feature(vcpu, KVM_ARM_VCPU_EL1_32BIT);
 	if (!cpus_have_const_cap(ARM64_HAS_32BIT_EL1) && is32bit)
@@ -180,12 +178,6 @@ static bool vcpu_allowed_register_width(struct kvm_vcpu *vcpu)
 	if (kvm_has_mte(vcpu->kvm) && is32bit)
 		return false;
 
-	/* Check that the vcpus are either all 32bit or all 64bit */
-	kvm_for_each_vcpu(i, tmp, vcpu->kvm) {
-		if (vcpu_has_feature(tmp, KVM_ARM_VCPU_EL1_32BIT) != is32bit)
-			return false;
-	}
-
 	return true;
 }
 

base-commit: 37144b2c855f9311c72f292125061d4a52d02856

Signature · 1 line
2.34.1.703.g22d0c6ccf7-goog
#1 lore raw

PATCH v2 2/2 KVM: arm64: selftests: Introduce vcpu_width_config

Introduce a test for aarch64 that ensures non-mixed-width vCPUs (all 64bit vCPUs or all 32bit vcPUs) can be configured, and mixed-width vCPUs cannot be configured.

---
 tools/testing/selftests/kvm/.gitignore        |   1 +
 tools/testing/selftests/kvm/Makefile          |   1 +
 .../selftests/kvm/aarch64/vcpu_width_config.c | 125 ++++++++++++++++++
 3 files changed, 127 insertions(+)
 create mode 100644 tools/testing/selftests/kvm/aarch64/vcpu_width_config.c
tools/testing/selftests/kvm/.gitignore +1 -0
index 3cb5ac5da087..8795a83cc382 100644
@@ -3,6 +3,7 @@
 /aarch64/debug-exceptions
 /aarch64/get-reg-list
 /aarch64/psci_cpu_on_test
+/aarch64/vcpu_width_config
 /aarch64/vgic_init
 /s390x/memop
 /s390x/resets
tools/testing/selftests/kvm/Makefile +1 -0
index 17342b575e85..259e01d0735a 100644
@@ -95,6 +95,7 @@ TEST_GEN_PROGS_aarch64 += aarch64/arch_timer
 TEST_GEN_PROGS_aarch64 += aarch64/debug-exceptions
 TEST_GEN_PROGS_aarch64 += aarch64/get-reg-list
 TEST_GEN_PROGS_aarch64 += aarch64/psci_cpu_on_test
+TEST_GEN_PROGS_aarch64 += aarch64/vcpu_width_config
 TEST_GEN_PROGS_aarch64 += aarch64/vgic_init
 TEST_GEN_PROGS_aarch64 += demand_paging_test
 TEST_GEN_PROGS_aarch64 += dirty_log_test
tools/testing/selftests/kvm/aarch64/vcpu_width_config.c new file +125 -0

125 lines not on this page · raw message

Signature · 1 line
2.34.1.703.g22d0c6ccf7-goog
maintainer
#2 lore raw
In reply to Reiji Watanabe
Quoted 41 lines · Reiji Watanabe

KVM allows userspace to configure either all 32bit or 64bit vCPUs for a guest. At vCPU reset, vcpu_allowed_register_width() checks if the vcpu's register width is consistent with all other vCPUs'. Since the checking is done even against vCPUs that are not initialized (KVM_ARM_VCPU_INIT has not been done) yet, the uninitialized vCPUs are erroneously treated as 64bit vCPU, which causes the function to incorrectly detect a mixed-width VM.

Introduce a new flag (el1_reg_width) in kvm_arch to indicates that the guest needs to be configured with all 32bit or 64bit vCPUs, and initialize it at the first KVM_ARM_VCPU_INIT for the guest. Check vcpu's register width against the flag at the vcpu's KVM_ARM_VCPU_INIT (instead of against other vCPUs' register width).

Fixes 66e94d5cafd4 ("KVM: arm64: Prevent mixed-width VM creation")
---
 arch/arm64/include/asm/kvm_host.h | 13 +++++++++++++
 arch/arm64/kvm/arm.c              | 30 ++++++++++++++++++++++++++++++
 arch/arm64/kvm/reset.c            |  8 --------
 3 files changed, 43 insertions(+), 8 deletions(-)
arch/arm64/include/asm/kvm_host.h +6 -0
index 2a5f7f38006f..c02b7caf2c82 100644
@@ -102,6 +102,12 @@ struct kvm_s2_mmu {
 struct kvm_arch_memory_slot {
 };
 
+enum kvm_el1_reg_width {
+	EL1_WIDTH_UNINITIALIZED = 0,
+	EL1_32BIT,
+	EL1_64BIT,
+};
+
 struct kvm_arch {
 	struct kvm_s2_mmu mmu;
 

arch/arm64/include/asm/kvm_host.h struct kvm_arch { · line 152

Quoted 10 lines
arch/arm64/include/asm/kvm_host.h +7 -0
@@ -137,6 +143,13 @@ struct kvm_arch {
 
 	/* Memory Tagging Extension enabled for the guest */
 	bool mte_enabled;
+
+	/*
+	 * EL1 register width for the guest.
+	 * This is set at the first KVM_ARM_VCPU_INIT for the guest based
+	 * on whether the vcpu has KVM_ARM_VCPU_EL1_32BIT or not.
+	 */
+	enum kvm_el1_reg_width el1_reg_width;

I really don't like that we need to keep track of yet another bit of state on top of the existing one. Duplicating state is a source of bugs, because you always end up checking the wrong one at the wrong time (and I have scars to prove it).

arch/arm64/kvm/arm.c int kvm_vm_ioctl_irq_line(struct kvm *kvm, struct kvm_irq_level *irq_level, · line 1062

Quoted 13 lines
arch/arm64/include/asm/kvm_host.h +0 -0
 };
 
 struct kvm_vcpu_fault_info {
arch/arm64/kvm/arm.c +2 -0
index e4727dc771bf..54ae8bf9d187 100644
@@ -1058,6 +1058,32 @@ int kvm_vm_ioctl_irq_line(struct kvm *kvm, struct kvm_irq_level *irq_level,
 	return -EINVAL;
 }
 
+/*
+ * A guest can have either all 32bit or 64bit vcpus only.

That's not strictly true. All we are enforcing is that EL1 is either 32 or 64bit.

Quoted 53 lines
arch/arm64/kvm/arm.c +28 -0
+ * Either one the guest has is indicated in kvm->arch.el1_reg_width.
+ * Check if the vcpu's register width is consistent with
+ * kvm->arch.el1_reg_width.  If kvm->arch.el1_reg_width is not set yet,
+ * set it based on the vcpu's KVM_ARM_VCPU_EL1_32BIT configuration.
+ */
+static int kvm_register_width_check_or_init(struct kvm_vcpu *vcpu)
+{
+	bool is32bit;
+	bool allowed = true;
+	struct kvm *kvm = vcpu->kvm;
+
+	is32bit = vcpu_has_feature(vcpu, KVM_ARM_VCPU_EL1_32BIT);
+
+	mutex_lock(&kvm->lock);
+
+	if (kvm->arch.el1_reg_width == EL1_WIDTH_UNINITIALIZED)
+		kvm->arch.el1_reg_width = is32bit ? EL1_32BIT : EL1_64BIT;
+	else
+		allowed = (is32bit == (kvm->arch.el1_reg_width == EL1_32BIT));
+
+	mutex_unlock(&kvm->lock);
+	return allowed ? 0 : -EINVAL;
+}
+
 static int kvm_vcpu_set_target(struct kvm_vcpu *vcpu,
 			       const struct kvm_vcpu_init *init)
 {
@@ -1097,6 +1123,10 @@ static int kvm_vcpu_set_target(struct kvm_vcpu *vcpu,
 
 	/* Now we know what it is, we can reset it. */
 	ret = kvm_reset_vcpu(vcpu);
+
+	if (!ret)
+		ret = kvm_register_width_check_or_init(vcpu);
+
 	if (ret) {
 		vcpu->arch.target = -1;
 		bitmap_zero(vcpu->arch.features, KVM_VCPU_MAX_FEATURES);
arch/arm64/kvm/reset.c +0 -2
index 426bd7fbc3fd..dbf2939a6a96 100644
@@ -168,9 +168,7 @@ static int kvm_vcpu_enable_ptrauth(struct kvm_vcpu *vcpu)
 
 static bool vcpu_allowed_register_width(struct kvm_vcpu *vcpu)
 {
-	struct kvm_vcpu *tmp;
 	bool is32bit;
-	int i;
 
 	is32bit = vcpu_has_feature(vcpu, KVM_ARM_VCPU_EL1_32BIT);
 	if (!cpus_have_const_cap(ARM64_HAS_32BIT_EL1) && is32bit)

arch/arm64/kvm/reset.c static bool vcpu_allowed_register_width(struct kvm_vcpu *vcpu) · line 188

Quoted 9 lines
arch/arm64/kvm/reset.c +0 -6
@@ -180,12 +178,6 @@ static bool vcpu_allowed_register_width(struct kvm_vcpu *vcpu)
 	if (kvm_has_mte(vcpu->kvm) && is32bit)
 		return false;
 
-	/* Check that the vcpus are either all 32bit or all 64bit */
-	kvm_for_each_vcpu(i, tmp, vcpu->kvm) {
-		if (vcpu_has_feature(tmp, KVM_ARM_VCPU_EL1_32BIT) != is32bit)
-			return false;
-	}
-

In [1], I suggested another approach that didn't require extra state, and moved the existing checks under the kvm lock. What was wrong with that approach?

Thanks,

	M.

[1] https://lore.kernel.org/r/[email protected]

Signature · 1 line
Without deviation from the norm, progress is not possible.
#3 lore raw

Hi Marc,

Quoted 71 lines · Marc Zyngier
Quoted 41 lines · Reiji Watanabe

KVM allows userspace to configure either all 32bit or 64bit vCPUs for a guest. At vCPU reset, vcpu_allowed_register_width() checks if the vcpu's register width is consistent with all other vCPUs'. Since the checking is done even against vCPUs that are not initialized (KVM_ARM_VCPU_INIT has not been done) yet, the uninitialized vCPUs are erroneously treated as 64bit vCPU, which causes the function to incorrectly detect a mixed-width VM.

Introduce a new flag (el1_reg_width) in kvm_arch to indicates that the guest needs to be configured with all 32bit or 64bit vCPUs, and initialize it at the first KVM_ARM_VCPU_INIT for the guest. Check vcpu's register width against the flag at the vcpu's KVM_ARM_VCPU_INIT (instead of against other vCPUs' register width).

Fixes 66e94d5cafd4 ("KVM: arm64: Prevent mixed-width VM creation")
---
 arch/arm64/include/asm/kvm_host.h | 13 +++++++++++++
 arch/arm64/kvm/arm.c              | 30 ++++++++++++++++++++++++++++++
 arch/arm64/kvm/reset.c            |  8 --------
 3 files changed, 43 insertions(+), 8 deletions(-)
arch/arm64/include/asm/kvm_host.h +6 -0
index 2a5f7f38006f..c02b7caf2c82 100644
@@ -102,6 +102,12 @@ struct kvm_s2_mmu {
 struct kvm_arch_memory_slot {
 };
 
+enum kvm_el1_reg_width {
+     EL1_WIDTH_UNINITIALIZED = 0,
+     EL1_32BIT,
+     EL1_64BIT,
+};
+
 struct kvm_arch {
      struct kvm_s2_mmu mmu;
 

arch/arm64/include/asm/kvm_host.h struct kvm_arch { · line 152

Quoted 10 lines
arch/arm64/include/asm/kvm_host.h +7 -0
@@ -137,6 +143,13 @@ struct kvm_arch {
 
      /* Memory Tagging Extension enabled for the guest */
      bool mte_enabled;
+
+     /*
+      * EL1 register width for the guest.
+      * This is set at the first KVM_ARM_VCPU_INIT for the guest based
+      * on whether the vcpu has KVM_ARM_VCPU_EL1_32BIT or not.
+      */
+     enum kvm_el1_reg_width el1_reg_width;

I really don't like that we need to keep track of yet another bit of state on top of the existing one. Duplicating state is a source of bugs, because you always end up checking the wrong one at the wrong time (and I have scars to prove it).

Quoted 11 lines
arch/arm64/include/asm/kvm_host.h +0 -0
 };
 
 struct kvm_vcpu_fault_info {
arch/arm64/kvm/arm.c +0 -0
index e4727dc771bf..54ae8bf9d187 100644
@@ -1058,6 +1058,32 @@ int kvm_vm_ioctl_irq_line(struct kvm *kvm, struct kvm_irq_level *irq_level,
      return -EINVAL;
 }
 
Quoted 5 lines

arch/arm64/kvm/arm.c int kvm_vm_ioctl_irq_line(struct kvm *kvm, struct kvm_irq_level *irq_level, · line 1062

Quoted 2 lines
arch/arm64/kvm/arm.c +2 -0
+/*
+ * A guest can have either all 32bit or 64bit vcpus only.

That's not strictly true. All we are enforcing is that EL1 is either 32 or 64bit.

I will fix the comment.

Quoted 57 lines
Quoted 57 lines
arch/arm64/kvm/arm.c +28 -0
+ * Either one the guest has is indicated in kvm->arch.el1_reg_width.
+ * Check if the vcpu's register width is consistent with
+ * kvm->arch.el1_reg_width.  If kvm->arch.el1_reg_width is not set yet,
+ * set it based on the vcpu's KVM_ARM_VCPU_EL1_32BIT configuration.
+ */
+static int kvm_register_width_check_or_init(struct kvm_vcpu *vcpu)
+{
+     bool is32bit;
+     bool allowed = true;
+     struct kvm *kvm = vcpu->kvm;
+
+     is32bit = vcpu_has_feature(vcpu, KVM_ARM_VCPU_EL1_32BIT);
+
+     mutex_lock(&kvm->lock);
+
+     if (kvm->arch.el1_reg_width == EL1_WIDTH_UNINITIALIZED)
+             kvm->arch.el1_reg_width = is32bit ? EL1_32BIT : EL1_64BIT;
+     else
+             allowed = (is32bit == (kvm->arch.el1_reg_width == EL1_32BIT));
+
+     mutex_unlock(&kvm->lock);
+     return allowed ? 0 : -EINVAL;
+}
+
 static int kvm_vcpu_set_target(struct kvm_vcpu *vcpu,
                             const struct kvm_vcpu_init *init)
 {
@@ -1097,6 +1123,10 @@ static int kvm_vcpu_set_target(struct kvm_vcpu *vcpu,
 
      /* Now we know what it is, we can reset it. */
      ret = kvm_reset_vcpu(vcpu);
+
+     if (!ret)
+             ret = kvm_register_width_check_or_init(vcpu);
+
      if (ret) {
              vcpu->arch.target = -1;
              bitmap_zero(vcpu->arch.features, KVM_VCPU_MAX_FEATURES);
arch/arm64/kvm/reset.c +0 -2
index 426bd7fbc3fd..dbf2939a6a96 100644
@@ -168,9 +168,7 @@ static int kvm_vcpu_enable_ptrauth(struct kvm_vcpu *vcpu)
 
 static bool vcpu_allowed_register_width(struct kvm_vcpu *vcpu)
 {
-     struct kvm_vcpu *tmp;
      bool is32bit;
-     int i;
 
      is32bit = vcpu_has_feature(vcpu, KVM_ARM_VCPU_EL1_32BIT);
      if (!cpus_have_const_cap(ARM64_HAS_32BIT_EL1) && is32bit)
@@ -180,12 +178,6 @@ static bool vcpu_allowed_register_width(struct kvm_vcpu *vcpu)
      if (kvm_has_mte(vcpu->kvm) && is32bit)
              return false;
 
Quoted 10 lines

arch/arm64/kvm/reset.c static bool vcpu_allowed_register_width(struct kvm_vcpu *vcpu) · line 188

Quoted 6 lines
arch/arm64/kvm/reset.c +0 -6
-     /* Check that the vcpus are either all 32bit or all 64bit */
-     kvm_for_each_vcpu(i, tmp, vcpu->kvm) {
-             if (vcpu_has_feature(tmp, KVM_ARM_VCPU_EL1_32BIT) != is32bit)
-                     return false;
-     }
-

In [1], I suggested another approach that didn't require extra state, and moved the existing checks under the kvm lock. What was wrong with that approach?

With that approach, even for a vcpu that has a broken set of features, which leads kvm_reset_vcpu() to fail for the vcpu, the vcpu->arch.features are checked by other vCPUs' vcpu_allowed_register_width() until the vcpu->arch.target is set to -1.
Due to this, I would think some or possibly all vCPUs' kvm_reset_vcpu() may or may not fail (e.g. if userspace tries to configure vCPU#0 with 32bit EL1, and vCPU#1 and #2 with 64 bit EL1, KVM_ARM_VCPU_INIT for either vCPU#0, or both vCPU#1 and #2 should fail. But, with that approach, it doesn't always work that way. Instead, KVM_ARM_VCPU_INIT for all vCPUs could fail or KVM_ARM_VCPU_INIT for vCPU#0 and #1 could fail while the one for CPU#2 works).
Also, even after the first KVM_RUN for vCPUs are already done, (the first) KVM_ARM_VCPU_INIT for another vCPU could cause the kvm_reset_vcpu() for those vCPUs to fail.

I would think those behaviors are odd, and I wanted to avoid them.

Thanks,
Reiji

Quoted 9 lines

Thanks,

        M.

[1] https://lore.kernel.org/r/[email protected]

--
Without deviation from the norm, progress is not possible.

_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

maintainer
#4 lore raw

Hi Reiji,

Quoted 24 lines · Reiji Watanabe

Hi Marc,

Quoted 4 lines · Marc Zyngier

In [1], I suggested another approach that didn't require extra state, and moved the existing checks under the kvm lock. What was wrong with that approach?

With that approach, even for a vcpu that has a broken set of features, which leads kvm_reset_vcpu() to fail for the vcpu, the vcpu->arch.features are checked by other vCPUs' vcpu_allowed_register_width() until the vcpu->arch.target is set to -1.
Due to this, I would think some or possibly all vCPUs' kvm_reset_vcpu() may or may not fail (e.g. if userspace tries to configure vCPU#0 with 32bit EL1, and vCPU#1 and #2 with 64 bit EL1, KVM_ARM_VCPU_INIT for either vCPU#0, or both vCPU#1 and #2 should fail. But, with that approach, it doesn't always work that way. Instead, KVM_ARM_VCPU_INIT for all vCPUs could fail or KVM_ARM_VCPU_INIT for vCPU#0 and #1 could fail while the one for CPU#2 works).
Also, even after the first KVM_RUN for vCPUs are already done, (the first) KVM_ARM_VCPU_INIT for another vCPU could cause the kvm_reset_vcpu() for those vCPUs to fail.

Quoted 1 line

I would think those behaviors are odd, and I wanted to avoid them.

OK, fair enough. But then you need to remove most of the uses of KVM_ARM_VCPU_EL1_32BIT so that it is only used as a userspace interface and maybe not carried as part of the vcpu feature flag anymore.

Also, we really should turn all these various bits in the kvm struct into a set of flags. I have a patch posted there[1] for this, feel free to pick it up.

Thanks,

	M.

[1] https://lore.kernel.org/r/[email protected]

Signature · 5 lines
Without deviation from the norm, progress is not possible.
_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
#5 lore raw

Hi Marc,

Quoted 29 lines · Marc Zyngier

Hi Reiji,

Quoted 24 lines · Reiji Watanabe

Hi Marc,

Quoted 4 lines · Marc Zyngier

In [1], I suggested another approach that didn't require extra state, and moved the existing checks under the kvm lock. What was wrong with that approach?

With that approach, even for a vcpu that has a broken set of features, which leads kvm_reset_vcpu() to fail for the vcpu, the vcpu->arch.features are checked by other vCPUs' vcpu_allowed_register_width() until the vcpu->arch.target is set to -1.
Due to this, I would think some or possibly all vCPUs' kvm_reset_vcpu() may or may not fail (e.g. if userspace tries to configure vCPU#0 with 32bit EL1, and vCPU#1 and #2 with 64 bit EL1, KVM_ARM_VCPU_INIT for either vCPU#0, or both vCPU#1 and #2 should fail. But, with that approach, it doesn't always work that way. Instead, KVM_ARM_VCPU_INIT for all vCPUs could fail or KVM_ARM_VCPU_INIT for vCPU#0 and #1 could fail while the one for CPU#2 works).
Also, even after the first KVM_RUN for vCPUs are already done, (the first) KVM_ARM_VCPU_INIT for another vCPU could cause the kvm_reset_vcpu() for those vCPUs to fail.

Quoted 5 lines
Quoted 1 line

I would think those behaviors are odd, and I wanted to avoid them.

OK, fair enough. But then you need to remove most of the uses of KVM_ARM_VCPU_EL1_32BIT so that it is only used as a userspace interface and

Yes, I will.

Quoted 1 line

maybe not carried as part of the vcpu feature flag anymore.

At the first call of kvm_reset_vcpu() for the guest, the new kvm flag is not set yet. So, KVM_ARM_VCPU_EL1_32BIT will be needed by the function (unless we pass the flag as an argument for the function or by any other way).

Quoted 3 lines

Also, we really should turn all these various bits in the kvm struct into a set of flags. I have a patch posted there[1] for this, feel free to pick it up.

Thank you for the suggestion. But, kvm->arch.el1_reg_width is not a binary because it needs to indicate an uninitialized state. So, it won't fit perfectly with kvm->arch.flags, which is introduced by [1] as it is. Of course it's feasible by using 2 bits of the flags though...

Thanks,
Reiji

Quoted 9 lines

Thanks,

        M.

[1] https://lore.kernel.org/r/[email protected]

--
Without deviation from the norm, progress is not possible.

maintainer
#6 lore raw
Quoted 39 lines · Reiji Watanabe

Hi Marc,

Quoted 29 lines · Marc Zyngier

Hi Reiji,

Quoted 24 lines · Reiji Watanabe

Hi Marc,

Quoted 4 lines · Marc Zyngier

In [1], I suggested another approach that didn't require extra state, and moved the existing checks under the kvm lock. What was wrong with that approach?

With that approach, even for a vcpu that has a broken set of features, which leads kvm_reset_vcpu() to fail for the vcpu, the vcpu->arch.features are checked by other vCPUs' vcpu_allowed_register_width() until the vcpu->arch.target is set to -1.
Due to this, I would think some or possibly all vCPUs' kvm_reset_vcpu() may or may not fail (e.g. if userspace tries to configure vCPU#0 with 32bit EL1, and vCPU#1 and #2 with 64 bit EL1, KVM_ARM_VCPU_INIT for either vCPU#0, or both vCPU#1 and #2 should fail. But, with that approach, it doesn't always work that way. Instead, KVM_ARM_VCPU_INIT for all vCPUs could fail or KVM_ARM_VCPU_INIT for vCPU#0 and #1 could fail while the one for CPU#2 works).
Also, even after the first KVM_RUN for vCPUs are already done, (the first) KVM_ARM_VCPU_INIT for another vCPU could cause the kvm_reset_vcpu() for those vCPUs to fail.

Quoted 5 lines
Quoted 1 line

I would think those behaviors are odd, and I wanted to avoid them.

OK, fair enough. But then you need to remove most of the uses of KVM_ARM_VCPU_EL1_32BIT so that it is only used as a userspace interface and

Quoted 8 lines

Yes, I will.

Quoted 1 line

maybe not carried as part of the vcpu feature flag anymore.

At the first call of kvm_reset_vcpu() for the guest, the new kvm flag is not set yet. So, KVM_ARM_VCPU_EL1_32BIT will be needed by the function (unless we pass the flag as an argument for the function or by any other way).

Which is why I said 'maybe'. It's not a big deal if the flags stays, but I don't want it evaluated further down the line. It is also pretty similar to HCR_EL2.RW, which we already test with vcpu_el1_is_32bit().

Overall, we need to reduce that state to be as simple as possible.

Quoted 9 lines
Quoted 4 lines

Also, we really should turn all these various bits in the kvm struct into a set of flags. I have a patch posted there[1] for this, feel free to pick it up.

Thank you for the suggestion. But, kvm->arch.el1_reg_width is not a binary because it needs to indicate an uninitialized state. So, it won't fit perfectly with kvm->arch.flags, which is introduced by [1] as it is. Of course it's feasible by using 2 bits of the flags though...

2 bits is what I had in mind (one bit to indicate that it has already been initialised, another to carry the actual width).

Thanks,

	M.
Signature · 1 line
Without deviation from the norm, progress is not possible.
#7 lore raw

Hi Marc,

Quoted 11 lines
Quoted 8 lines
Quoted 3 lines

Also, we really should turn all these various bits in the kvm struct into a set of flags. I have a patch posted there[1] for this, feel free to pick it up.

Thank you for the suggestion. But, kvm->arch.el1_reg_width is not a binary because it needs to indicate an uninitialized state. So, it won't fit perfectly with kvm->arch.flags, which is introduced by [1] as it is. Of course it's feasible by using 2 bits of the flags though...

2 bits is what I had in mind (one bit to indicate that it has already been initialised, another to carry the actual width).

Understood. Then, I will take the patch and will work on v3. Thank you for all the comments!

Regards,
Reiji