KVM: arm64: Clear VM-wide state when kvm_setup_vcpu() fails
Fuad Tabba
reviewerv4 cover KVM: arm64: Clear VM-wide state when kvm_setup_vcpu() fails
Hi folks,
Changes since v3 [1]:
- New patch 2, clearing the default PMU left set by a failed INIT.
- Reworded patch 1's commit message.
- Rebased on 7.4-rc1.
A KVM_ARM_VCPU_INIT whose kvm_setup_vcpu() fails doesn't undo its changes to VM-wide state. Patch 1 clears the feature bitmap, which the PMU attributes read before the vCPU is initialized. Patch 2 clears the default PMU, which a retry with KVM_ARM_VCPU_PMU_V3_STRICT would otherwise inherit.
Tested under QEMU on a host without FEAT_NV1 (an Apple M2 MIDR, for which has_nv1() returns false), with a reproducer that fails an INIT and then initializes the PMU, directly for patch 1 and after a STRICT retry for patch 2.
Based on Linux 7.4-rc1.
Cheers,
/fuad
[1] https://lore.kernel.org/all/[email protected]/
Fuad Tabba (2):
KVM: arm64: Clear the VM's feature bitmap when kvm_setup_vcpu() fails
KVM: arm64: Clear the default PMU when kvm_setup_vcpu() fails
arch/arm64/kvm/arm.c | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-)
base-commit: 6628e7d4c6b25c8d18f1345e3a20d21829aae7e4
Signature · 1 line
2.47.3
Fuad Tabba
reviewerPATCH v4 1/2 KVM: arm64: Clear the VM's feature bitmap when kvm_setup_vcpu() fails
KVM_ARM_VCPU_INIT records the requested features in the VM-wide bitmap before kvm_setup_vcpu() runs, and leaves them there when setup fails (e.g., HAS_EL2_E2H0 on a host without FEAT_NV1). The PMU attributes read that bitmap before the vCPU is initialized, so after such a failure with PMU_V3 requested, KVM_SET_DEVICE_ATTR on the PMU group no longer returns -ENODEV.
Until an INIT has succeeded, every INIT that gets past the feature checks rewrites the bitmap. After that, kvm_vcpu_init_changed() requires the requested features to match it, so the copy is a no-op. Clear the bitmap on failure while KVM_ARCH_FLAG_VCPU_FEATURES_CONFIGURED is clear.
--- arch/arm64/kvm/arm.c | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-)
arch/arm64/kvm/arm.c +6 -1
@@ -1856,8 +1856,13 @@ static int __kvm_vcpu_set_target(struct kvm_vcpu *vcpu,bitmap_copy (kvm->arch.vcpu_features, &features,KVM_VCPU_MAX_FEATURES ); ret =kvm_setup_vcpu (vcpu); -if (ret) +if (ret) { +/* Clear the bitmap if setup fails on the first vCPU to be initialized. */ +if (!test_bit (KVM_ARCH_FLAG_VCPU_FEATURES_CONFIGURED , &kvm->arch.flags)) +bitmap_zero (kvm->arch.vcpu_features,KVM_VCPU_MAX_FEATURES ); + goto out_unlock; + }/* Now we know what it is, we can reset it. */ kvm_reset_vcpu (vcpu);
Signature · 1 line
2.47.3
Re: [PATCH v4 1/2] KVM: arm64: Clear the VM's feature bitmap when kvm_setup_vcpu() fails
Quoted 26 lines · Fuad Tabba
KVM_ARM_VCPU_INITrecords the requested features in the VM-wide bitmap beforekvm_setup_vcpu()runs, and leaves them there when setup fails (e.g.,HAS_EL2_E2H0on a host withoutFEAT_NV1). The PMU attributes read that bitmap before the vCPU is initialized, so after such a failure withPMU_V3requested,KVM_SET_DEVICE_ATTRon the PMU group no longer returns-ENODEV.Until an INIT has succeeded, every INIT that gets past the feature checks rewrites the bitmap. After that,
kvm_vcpu_init_changed()requires the requested features to match it, so the copy is a no-op. Clear the bitmap on failure whileKVM_ARCH_FLAG_VCPU_FEATURES_CONFIGUREDis clear.Fixes 427733579744e ("KVM: arm64: Select default PMU in KVM_ARM_VCPU_INIT handler")Suggested-by Oliver Upton <[email protected]>Reviewed-by Lorenzo Stoakes (ARM) <[email protected]>Signed-off-by Fuad Tabba <[email protected]>--- arch/arm64/kvm/arm.c | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-)arch/arm64/kvm/arm.c +0
-0
arch/arm64/kvm/arm.c static int __kvm_vcpu_set_target(struct kvm_vcpu *vcpu, · line 1861
Quoted 7 lines
arch/arm64/kvm/arm.c +3
-1@@ -1856,8 +1856,13 @@ static int __kvm_vcpu_set_target(struct kvm_vcpu *vcpu,bitmap_copy (kvm->arch.vcpu_features, &features,KVM_VCPU_MAX_FEATURES ); ret =kvm_setup_vcpu (vcpu); -if (ret) +if (ret) { +/* Clear the bitmap if setup fails on the first vCPU to be initialized. */ +if (!test_bit (KVM_ARCH_FLAG_VCPU_FEATURES_CONFIGURED , &kvm->arch.flags))
Is this check necessary ? We go into kvm_setup_vcpu() only if the bit is cleared and we do this under the config_lock mutex. The flag is only set after we have finished the kvm_setup_vcpu() successfully, later down.
Otherwise,
Quoted 7 lines
arch/arm64/kvm/arm.c +3
-0+bitmap_zero (kvm->arch.vcpu_features,KVM_VCPU_MAX_FEATURES ); + goto out_unlock; + }/* Now we know what it is, we can reset it. */ kvm_reset_vcpu (vcpu);
Fuad Tabba
reviewerRe: [PATCH v4 1/2] KVM: arm64: Clear the VM's feature bitmap when kvm_setup_vcpu() fails
Hi Suzuki,
On Mon, 02 Nov 2026 09:24:15 +0000, Suzuki K Poulose <[email protected]> wrote:
[...]
Quoted 1 line
Quoted 1 line
arch/arm64/kvm/arm.c +0
-0
[...]
Quoted 12 lines
arch/arm64/kvm/arm.c static int __kvm_vcpu_set_target(struct kvm_vcpu *vcpu, · line 1861
Quoted 8 lines
arch/arm64/kvm/arm.c +3
-1@@ -1856,8 +1856,13 @@ static int __kvm_vcpu_set_target(struct kvm_vcpu *vcpu,bitmap_copy (kvm->arch.vcpu_features, &features,KVM_VCPU_MAX_FEATURES ); ret =kvm_setup_vcpu (vcpu); -if (ret) +if (ret) { +/* Clear the bitmap if setup fails on the first vCPU to be initialized. */ +if (!test_bit (KVM_ARCH_FLAG_VCPU_FEATURES_CONFIGURED , &kvm->arch.flags))Is this check necessary ? We go into
kvm_setup_vcpu()only if the bit is cleared and we do this under theconfig_lockmutex. The flag is only set after we have finished thekvm_setup_vcpu()successfully, later down.
It is. An INIT on a second vCPU with the same features also calls kvm_setup_vcpu(), with the flag already set, and kvm_vcpu_init_nested() can return -ENOMEM there.
Thanks,
/fuad
Re: [PATCH v4 1/2] KVM: arm64: Clear the VM's feature bitmap when kvm_setup_vcpu() fails
Quoted 20 lines · Fuad Tabba
Hi Suzuki,
On Mon, 02 Nov 2026 09:24:15 +0000, Suzuki K Poulose <[email protected]> wrote:
[...]Quoted 1 line
Quoted 1 line
arch/arm64/kvm/arm.c +0
-0[...]
Quoted 12 lines
arch/arm64/kvm/arm.c static int __kvm_vcpu_set_target(struct kvm_vcpu *vcpu, · line 1861
Quoted 8 lines
arch/arm64/kvm/arm.c +3
-1@@ -1856,8 +1856,13 @@ static int __kvm_vcpu_set_target(struct kvm_vcpu *vcpu,bitmap_copy (kvm->arch.vcpu_features, &features,KVM_VCPU_MAX_FEATURES ); ret =kvm_setup_vcpu (vcpu); -if (ret) +if (ret) { +/* Clear the bitmap if setup fails on the first vCPU to be initialized. */ +if (!test_bit (KVM_ARCH_FLAG_VCPU_FEATURES_CONFIGURED , &kvm->arch.flags))Is this check necessary ? We go into
kvm_setup_vcpu()only if the bit is cleared and we do this under theconfig_lockmutex. The flag is only set after we have finished thekvm_setup_vcpu()successfully, later down.
Quoted 3 lines
It is. An INIT on a second vCPU with the same features also calls
kvm_setup_vcpu(), with the flag already set, andkvm_vcpu_init_nested()can return-ENOMEMthere.
Sorry, my bad. I missed the second condition after the flag check in.
/me Where's coffee!
Quoted 3 lines
Thanks,
/fuad
Fuad Tabba
reviewerPATCH v4 2/2 KVM: arm64: Clear the default PMU when kvm_setup_vcpu() fails
In reply to Fuad TabbaUnder KVM_ARM_VCPU_PMU_V3_STRICT, the VM has no PMU until userspace selects one, and initializing the PMU fails with -ENXIO until then. However, without STRICT, kvm_setup_vcpu() selects the default PMU before it prepares the vCPU for nested virtualization, and leaves it set when that preparation fails. After a retry with STRICT, initializing the PMU succeeds and the guest gets the PMU of the CPU the failed INIT ran on.
Drop the PMU and its number of counters along with the VM's feature bitmap. While KVM_ARCH_FLAG_VCPU_FEATURES_CONFIGURED is clear, no vCPU has been initialized and the PMU attributes return -ENODEV, so only such a failure can have set them. Once the flag is set, every INIT has the features of the first one that succeeded, which either selected a PMU or had none to select, so kvm_setup_vcpu() doesn't select another.
--- arch/arm64/kvm/arm.c | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-)
arch/arm64/kvm/arm.c +5 -2
@@ -1857,9 +1857,12 @@ static int __kvm_vcpu_set_target(struct kvm_vcpu *vcpu, ret =kvm_setup_vcpu (vcpu);if (ret) { -/* Clear the bitmap if setup fails on the first vCPU to be initialized. */ -if (!test_bit (KVM_ARCH_FLAG_VCPU_FEATURES_CONFIGURED , &kvm->arch.flags)) +/* Clear the bitmap and PMU if setup fails on the first vCPU to be initialized. */ +if (!test_bit (KVM_ARCH_FLAG_VCPU_FEATURES_CONFIGURED , &kvm->arch.flags)) {bitmap_zero (kvm->arch.vcpu_features,KVM_VCPU_MAX_FEATURES ); + kvm->arch.arm_pmu =NULL ; + kvm->arch.nr_pmu_counters =0 ; + } goto out_unlock; }
Signature · 1 line
2.47.3
Re: [PATCH v4 2/2] KVM: arm64: Clear the default PMU when kvm_setup_vcpu() fails
Quoted 27 lines · Fuad Tabba
Under
KVM_ARM_VCPU_PMU_V3_STRICT, the VM has no PMU until userspace selects one, and initializing the PMU fails with-ENXIOuntil then. However, without STRICT,kvm_setup_vcpu()selects the default PMU before it prepares the vCPU for nested virtualization, and leaves it set when that preparation fails. After a retry with STRICT, initializing the PMU succeeds and the guest gets the PMU of the CPU the failed INIT ran on.Drop the PMU and its number of counters along with the VM's feature bitmap. While
KVM_ARCH_FLAG_VCPU_FEATURES_CONFIGUREDis clear, no vCPU has been initialized and the PMU attributes return-ENODEV, so only such a failure can have set them. Once the flag is set, every INIT has the features of the first one that succeeded, which either selected a PMU or had none to select, sokvm_setup_vcpu()doesn't select another.Fixes ad220e275c398 ("KVM: arm64: Add KVM_ARM_VCPU_PMU_V3_STRICT vCPU feature")Signed-off-by Fuad Tabba <[email protected]>--- arch/arm64/kvm/arm.c | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-)arch/arm64/kvm/arm.c +0
-0@@ -1857,9 +1857,12 @@ static int __kvm_vcpu_set_target(struct kvm_vcpu *vcpu,
arch/arm64/kvm/arm.c static int __kvm_vcpu_set_target(struct kvm_vcpu *vcpu, · line 1865
Quoted 10 lines
arch/arm64/kvm/arm.c +5
-2ret =kvm_setup_vcpu (vcpu);if (ret) { -/* Clear the bitmap if setup fails on the first vCPU to be initialized. */ -if (!test_bit (KVM_ARCH_FLAG_VCPU_FEATURES_CONFIGURED , &kvm->arch.flags)) +/* Clear the bitmap and PMU if setup fails on the first vCPU to be initialized. */ +if (!test_bit (KVM_ARCH_FLAG_VCPU_FEATURES_CONFIGURED , &kvm->arch.flags)) {bitmap_zero (kvm->arch.vcpu_features,KVM_VCPU_MAX_FEATURES ); + kvm->arch.arm_pmu =NULL ; + kvm->arch.nr_pmu_counters =0 ; + }
minor nit: While the above works, it looks a bit too much extracting the PMU details to arm.c. Alternatively we could do something like this:
arch/arm64/kvm/pmu-emul.c +5 -0
@@ -1088,6 +1088,11 @@ int kvm_arm_set_default_pmu(struct kvm *kvm)return 0 ; } +int kvm_arm_clear_default_pmu (struct kvm *kvm) +{ +kvm_arm_set_pmu (kvm,NULL ); +} +
include/kvm/arm_pmu.h +1 -0
@@ -92,6 +92,7 @@ void kvm_vcpu_pmu_resync_el0(void);u8 kvm_arm_pmu_get_pmuver_limit (void );u64 kvm_pmu_evtyper_mask (struct kvm *kvm);int kvm_arm_set_default_pmu (struct kvm *kvm); +int kvm_arm_clear_default_pmu (struct kvm *kvm);u8 kvm_arm_pmu_get_max_counters (struct kvm *kvm);u64 kvm_vcpu_read_pmcr (struct kvm_vcpu *vcpu);
Either ways,
Quoted 5 lines · Fuad Tabba
Changes since v3 [1]:
- New patch 2, clearing the default PMU left set by a failed INIT.
- Reworded patch 1's commit message.
- Rebased on 7.4-rc1.
Quoted 7 lines
A
KVM_ARM_VCPU_INITwhosekvm_setup_vcpu()fails doesn't undo its changes to VM-wide state. Patch 1 clears the feature bitmap, which the PMU attributes read before the vCPU is initialized. Patch 2 clears the default PMU, which a retry withKVM_ARM_VCPU_PMU_V3_STRICTwould otherwise inherit.[...]
Applied to fixes, thanks!
[1/2] KVM: arm64: Clear the VM's feature bitmap when kvm_setup_vcpu() fails
commit: 2c77a4a3040da8095e052849e2a0a05f598c8601
[2/2] KVM: arm64: Clear the default PMU when kvm_setup_vcpu() fails
commit: 4d3361e4d60a46781c8697ee1ab9532f625163f1Cheers,
M.
Signature · 1 line
Without deviation from the norm, progress is not possible.