Skip to contentSkip to conversation

KVM: arm64: Clear VM-wide state when kvm_setup_vcpu() fails

PATCH Applied Fuad Tabba 3 participants 8 messages Nov 2 – Nov 3

Subsystems (2)
lore
Share & tools
lore

Fuad Tabba

author reviewer
#0 lore raw

v4 cover KVM: arm64: Clear VM-wide state when kvm_setup_vcpu() fails

Hi folks,

Changes since v3 [1]:
- New patch 2, clearing the default PMU left set by a failed INIT.
- Reworded patch 1's commit message.
- Rebased on 7.4-rc1.

A KVM_ARM_VCPU_INIT whose kvm_setup_vcpu() fails doesn't undo its changes to VM-wide state. Patch 1 clears the feature bitmap, which the PMU attributes read before the vCPU is initialized. Patch 2 clears the default PMU, which a retry with KVM_ARM_VCPU_PMU_V3_STRICT would otherwise inherit.

Tested under QEMU on a host without FEAT_NV1 (an Apple M2 MIDR, for which has_nv1() returns false), with a reproducer that fails an INIT and then initializes the PMU, directly for patch 1 and after a STRICT retry for patch 2.

Based on Linux 7.4-rc1.

Cheers,
/fuad

[1] https://lore.kernel.org/all/[email protected]/

Fuad Tabba (2):
KVM: arm64: Clear the VM's feature bitmap when kvm_setup_vcpu() fails
KVM: arm64: Clear the default PMU when kvm_setup_vcpu() fails

 arch/arm64/kvm/arm.c | 10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)

base-commit: 6628e7d4c6b25c8d18f1345e3a20d21829aae7e4

Signature · 1 line
2.47.3
author reviewer
#1 lore raw

PATCH v4 1/2 KVM: arm64: Clear the VM's feature bitmap when kvm_setup_vcpu() fails

KVM_ARM_VCPU_INIT records the requested features in the VM-wide bitmap before kvm_setup_vcpu() runs, and leaves them there when setup fails (e.g., HAS_EL2_E2H0 on a host without FEAT_NV1). The PMU attributes read that bitmap before the vCPU is initialized, so after such a failure with PMU_V3 requested, KVM_SET_DEVICE_ATTR on the PMU group no longer returns -ENODEV.

Until an INIT has succeeded, every INIT that gets past the feature checks rewrites the bitmap. After that, kvm_vcpu_init_changed() requires the requested features to match it, so the copy is a no-op. Clear the bitmap on failure while KVM_ARCH_FLAG_VCPU_FEATURES_CONFIGURED is clear.

Fixes 427733579744e ("KVM: arm64: Select default PMU in KVM_ARM_VCPU_INIT handler")
---
 arch/arm64/kvm/arm.c | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)
arch/arm64/kvm/arm.c +6 -1
index 157bf4b812020..3e91e01b032c4 100644
@@ -1856,8 +1856,13 @@ static int __kvm_vcpu_set_target(struct kvm_vcpu *vcpu,
 	bitmap_copy(kvm->arch.vcpu_features, &features, KVM_VCPU_MAX_FEATURES);
 
 	ret = kvm_setup_vcpu(vcpu);
-	if (ret)
+	if (ret) {
+		/* Clear the bitmap if setup fails on the first vCPU to be initialized. */
+		if (!test_bit(KVM_ARCH_FLAG_VCPU_FEATURES_CONFIGURED, &kvm->arch.flags))
+			bitmap_zero(kvm->arch.vcpu_features, KVM_VCPU_MAX_FEATURES);
+
 		goto out_unlock;
+	}
 
 	/* Now we know what it is, we can reset it. */
 	kvm_reset_vcpu(vcpu);
Signature · 1 line
2.47.3
reviewer Reviewed-by on v4 1/2
#2 lore raw

Re: [PATCH v4 1/2] KVM: arm64: Clear the VM's feature bitmap when kvm_setup_vcpu() fails

Quoted 26 lines · Fuad Tabba

KVM_ARM_VCPU_INIT records the requested features in the VM-wide bitmap before kvm_setup_vcpu() runs, and leaves them there when setup fails (e.g., HAS_EL2_E2H0 on a host without FEAT_NV1). The PMU attributes read that bitmap before the vCPU is initialized, so after such a failure with PMU_V3 requested, KVM_SET_DEVICE_ATTR on the PMU group no longer returns -ENODEV.

Until an INIT has succeeded, every INIT that gets past the feature checks rewrites the bitmap. After that, kvm_vcpu_init_changed() requires the requested features to match it, so the copy is a no-op. Clear the bitmap on failure while KVM_ARCH_FLAG_VCPU_FEATURES_CONFIGURED is clear.

Fixes 427733579744e ("KVM: arm64: Select default PMU in KVM_ARM_VCPU_INIT handler")
---
  arch/arm64/kvm/arm.c | 7 ++++++-
  1 file changed, 6 insertions(+), 1 deletion(-)
arch/arm64/kvm/arm.c +0 -0
index 157bf4b812020..3e91e01b032c4 100644

arch/arm64/kvm/arm.c static int __kvm_vcpu_set_target(struct kvm_vcpu *vcpu, · line 1861

Quoted 7 lines
arch/arm64/kvm/arm.c +3 -1
@@ -1856,8 +1856,13 @@ static int __kvm_vcpu_set_target(struct kvm_vcpu *vcpu,
  	bitmap_copy(kvm->arch.vcpu_features, &features, KVM_VCPU_MAX_FEATURES);
 
  	ret = kvm_setup_vcpu(vcpu);
-	if (ret)
+	if (ret) {
+		/* Clear the bitmap if setup fails on the first vCPU to be initialized. */
+		if (!test_bit(KVM_ARCH_FLAG_VCPU_FEATURES_CONFIGURED, &kvm->arch.flags))

Is this check necessary ? We go into kvm_setup_vcpu() only if the bit is cleared and we do this under the config_lock mutex. The flag is only set after we have finished the kvm_setup_vcpu() successfully, later down.

Otherwise,

Quoted 7 lines
arch/arm64/kvm/arm.c +3 -0
+			bitmap_zero(kvm->arch.vcpu_features, KVM_VCPU_MAX_FEATURES);
+
  		goto out_unlock;
+	}
 
  	/* Now we know what it is, we can reset it. */
  	kvm_reset_vcpu(vcpu);
author reviewer
#3 lore raw

Re: [PATCH v4 1/2] KVM: arm64: Clear the VM's feature bitmap when kvm_setup_vcpu() fails

Hi Suzuki,

On Mon, 02 Nov 2026 09:24:15 +0000, Suzuki K Poulose <[email protected]> wrote:
[...]

Quoted 1 line
Quoted 1 line
arch/arm64/kvm/arm.c +0 -0

[...]

Quoted 12 lines

arch/arm64/kvm/arm.c static int __kvm_vcpu_set_target(struct kvm_vcpu *vcpu, · line 1861

Quoted 8 lines
arch/arm64/kvm/arm.c +3 -1
@@ -1856,8 +1856,13 @@ static int __kvm_vcpu_set_target(struct kvm_vcpu *vcpu,
         bitmap_copy(kvm->arch.vcpu_features, &features, KVM_VCPU_MAX_FEATURES);
 
         ret = kvm_setup_vcpu(vcpu);
-        if (ret)
+        if (ret) {
+                /* Clear the bitmap if setup fails on the first vCPU to be initialized. */
+                if (!test_bit(KVM_ARCH_FLAG_VCPU_FEATURES_CONFIGURED, &kvm->arch.flags))

Is this check necessary ? We go into kvm_setup_vcpu() only if the bit is cleared and we do this under the config_lock mutex. The flag is only set after we have finished the kvm_setup_vcpu() successfully, later down.

It is. An INIT on a second vCPU with the same features also calls kvm_setup_vcpu(), with the flag already set, and kvm_vcpu_init_nested() can return -ENOMEM there.

Thanks,
/fuad

reviewer Reviewed-by
#4 lore raw

Re: [PATCH v4 1/2] KVM: arm64: Clear the VM's feature bitmap when kvm_setup_vcpu() fails

Quoted 20 lines · Fuad Tabba

Hi Suzuki,

On Mon, 02 Nov 2026 09:24:15 +0000, Suzuki K Poulose <[email protected]> wrote:
[...]

Quoted 1 line
Quoted 1 line
arch/arm64/kvm/arm.c +0 -0

[...]

Quoted 12 lines

arch/arm64/kvm/arm.c static int __kvm_vcpu_set_target(struct kvm_vcpu *vcpu, · line 1861

Quoted 8 lines
arch/arm64/kvm/arm.c +3 -1
@@ -1856,8 +1856,13 @@ static int __kvm_vcpu_set_target(struct kvm_vcpu *vcpu,
          bitmap_copy(kvm->arch.vcpu_features, &features, KVM_VCPU_MAX_FEATURES);
 
          ret = kvm_setup_vcpu(vcpu);
-        if (ret)
+        if (ret) {
+                /* Clear the bitmap if setup fails on the first vCPU to be initialized. */
+                if (!test_bit(KVM_ARCH_FLAG_VCPU_FEATURES_CONFIGURED, &kvm->arch.flags))

Is this check necessary ? We go into kvm_setup_vcpu() only if the bit is cleared and we do this under the config_lock mutex. The flag is only set after we have finished the kvm_setup_vcpu() successfully, later down.

Quoted 3 lines

It is. An INIT on a second vCPU with the same features also calls kvm_setup_vcpu(), with the flag already set, and kvm_vcpu_init_nested() can return -ENOMEM there.

Sorry, my bad. I missed the second condition after the flag check in.

/me Where's coffee!

Quoted 3 lines

Thanks,
/fuad

author reviewer
#5 lore raw

PATCH v4 2/2 KVM: arm64: Clear the default PMU when kvm_setup_vcpu() fails

In reply to Fuad Tabba

Under KVM_ARM_VCPU_PMU_V3_STRICT, the VM has no PMU until userspace selects one, and initializing the PMU fails with -ENXIO until then. However, without STRICT, kvm_setup_vcpu() selects the default PMU before it prepares the vCPU for nested virtualization, and leaves it set when that preparation fails. After a retry with STRICT, initializing the PMU succeeds and the guest gets the PMU of the CPU the failed INIT ran on.

Drop the PMU and its number of counters along with the VM's feature bitmap. While KVM_ARCH_FLAG_VCPU_FEATURES_CONFIGURED is clear, no vCPU has been initialized and the PMU attributes return -ENODEV, so only such a failure can have set them. Once the flag is set, every INIT has the features of the first one that succeeded, which either selected a PMU or had none to select, so kvm_setup_vcpu() doesn't select another.

Fixes ad220e275c398 ("KVM: arm64: Add KVM_ARM_VCPU_PMU_V3_STRICT vCPU feature")
---
 arch/arm64/kvm/arm.c | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)
arch/arm64/kvm/arm.c +5 -2
index 3e91e01b032c4..65194e02cdbc2 100644
@@ -1857,9 +1857,12 @@ static int __kvm_vcpu_set_target(struct kvm_vcpu *vcpu,
 
 	ret = kvm_setup_vcpu(vcpu);
 	if (ret) {
-		/* Clear the bitmap if setup fails on the first vCPU to be initialized. */
-		if (!test_bit(KVM_ARCH_FLAG_VCPU_FEATURES_CONFIGURED, &kvm->arch.flags))
+		/* Clear the bitmap and PMU if setup fails on the first vCPU to be initialized. */
+		if (!test_bit(KVM_ARCH_FLAG_VCPU_FEATURES_CONFIGURED, &kvm->arch.flags)) {
 			bitmap_zero(kvm->arch.vcpu_features, KVM_VCPU_MAX_FEATURES);
+			kvm->arch.arm_pmu = NULL;
+			kvm->arch.nr_pmu_counters = 0;
+		}
 
 		goto out_unlock;
 	}
Signature · 1 line
2.47.3
reviewer Reviewed-by on v4 2/2
#6 lore raw

Re: [PATCH v4 2/2] KVM: arm64: Clear the default PMU when kvm_setup_vcpu() fails

Quoted 27 lines · Fuad Tabba

Under KVM_ARM_VCPU_PMU_V3_STRICT, the VM has no PMU until userspace selects one, and initializing the PMU fails with -ENXIO until then. However, without STRICT, kvm_setup_vcpu() selects the default PMU before it prepares the vCPU for nested virtualization, and leaves it set when that preparation fails. After a retry with STRICT, initializing the PMU succeeds and the guest gets the PMU of the CPU the failed INIT ran on.

Drop the PMU and its number of counters along with the VM's feature bitmap. While KVM_ARCH_FLAG_VCPU_FEATURES_CONFIGURED is clear, no vCPU has been initialized and the PMU attributes return -ENODEV, so only such a failure can have set them. Once the flag is set, every INIT has the features of the first one that succeeded, which either selected a PMU or had none to select, so kvm_setup_vcpu() doesn't select another.

Fixes ad220e275c398 ("KVM: arm64: Add KVM_ARM_VCPU_PMU_V3_STRICT vCPU feature")
---
  arch/arm64/kvm/arm.c | 7 +++++--
  1 file changed, 5 insertions(+), 2 deletions(-)
arch/arm64/kvm/arm.c +0 -0
index 3e91e01b032c4..65194e02cdbc2 100644
@@ -1857,9 +1857,12 @@ static int __kvm_vcpu_set_target(struct kvm_vcpu *vcpu,
 

arch/arm64/kvm/arm.c static int __kvm_vcpu_set_target(struct kvm_vcpu *vcpu, · line 1865

Quoted 10 lines
arch/arm64/kvm/arm.c +5 -2
  	ret = kvm_setup_vcpu(vcpu);
  	if (ret) {
-		/* Clear the bitmap if setup fails on the first vCPU to be initialized. */
-		if (!test_bit(KVM_ARCH_FLAG_VCPU_FEATURES_CONFIGURED, &kvm->arch.flags))
+		/* Clear the bitmap and PMU if setup fails on the first vCPU to be initialized. */
+		if (!test_bit(KVM_ARCH_FLAG_VCPU_FEATURES_CONFIGURED, &kvm->arch.flags)) {
  			bitmap_zero(kvm->arch.vcpu_features, KVM_VCPU_MAX_FEATURES);
+			kvm->arch.arm_pmu = NULL;
+			kvm->arch.nr_pmu_counters = 0;
+		}

minor nit: While the above works, it looks a bit too much extracting the PMU details to arm.c. Alternatively we could do something like this:

arch/arm64/kvm/pmu-emul.c +5 -0
index e2b28dbc0451..7a669cccf912 100644
@@ -1088,6 +1088,11 @@ int kvm_arm_set_default_pmu(struct kvm *kvm)
        return 0;
 }
 
+int kvm_arm_clear_default_pmu(struct kvm *kvm)
+{
+       kvm_arm_set_pmu(kvm, NULL);
+}
+
include/kvm/arm_pmu.h +1 -0
index 6b4a118d17ca..571c0ba8507c 100644
@@ -92,6 +92,7 @@ void kvm_vcpu_pmu_resync_el0(void);
 u8 kvm_arm_pmu_get_pmuver_limit(void);
 u64 kvm_pmu_evtyper_mask(struct kvm *kvm);
 int kvm_arm_set_default_pmu(struct kvm *kvm);
+int kvm_arm_clear_default_pmu(struct kvm *kvm);
 u8 kvm_arm_pmu_get_max_counters(struct kvm *kvm);
 
 u64 kvm_vcpu_read_pmcr(struct kvm_vcpu *vcpu);

Either ways,

maintainer Applied
#7 lore raw
In reply to Fuad Tabba
Quoted 5 lines · Fuad Tabba

Changes since v3 [1]:
- New patch 2, clearing the default PMU left set by a failed INIT.
- Reworded patch 1's commit message.
- Rebased on 7.4-rc1.

Quoted 7 lines

A KVM_ARM_VCPU_INIT whose kvm_setup_vcpu() fails doesn't undo its changes to VM-wide state. Patch 1 clears the feature bitmap, which the PMU attributes read before the vCPU is initialized. Patch 2 clears the default PMU, which a retry with KVM_ARM_VCPU_PMU_V3_STRICT would otherwise inherit.

[...]

Applied to fixes, thanks!

[1/2] KVM: arm64: Clear the VM's feature bitmap when kvm_setup_vcpu() fails

      commit: 2c77a4a3040da8095e052849e2a0a05f598c8601
[2/2] KVM: arm64: Clear the default PMU when kvm_setup_vcpu() fails
      commit: 4d3361e4d60a46781c8697ee1ab9532f625163f1

Cheers,

	M.
Signature · 1 line
Without deviation from the norm, progress is not possible.