Skip to contentSkip to conversation

KVM: arm64: Hold kvm->mmu_lock while initialising vcpu->arch.vncr_tlb

PATCH In review Marc Zyngier 4 participants 9 messages Jun 8 – Jun 10

Subsystems (3)
lore
Share & tools
lore

Marc Zyngier

author maintainer
#0 lore raw

PATCH 1/1 KVM: arm64: Hold kvm->mmu_lock while initialising vcpu->arch.vncr_tlb

Sashiko reports that there is a race between initialising vncr_tlb and making use of it, as we don't hold the mmu_lock at this point.

Additionally, it identifies a memory leak, should userspace repeatedly invokes the KVM_RUN ioctl after a failure of kvm_arch_vcpu_run_pid_change(), as we assign vncr_tlb blindly on first run, irrespective of prior allocations.

Slap the two bugs in one go by taking the kvm->mmu_lock on assigning vncr_tlb, preventing the race for good, and by checking that vncr_tlb is indeed NULL prior to allocation.

---
 arch/arm64/kvm/nested.c | 10 ++++++++--
 1 file changed, 8 insertions(+), 2 deletions(-)
arch/arm64/kvm/nested.c +8 -2
index 690b8e8564166..d11e36b3cfcc2 100644
@@ -1253,8 +1253,14 @@ int kvm_vcpu_allocate_vncr_tlb(struct kvm_vcpu *vcpu)
 	if (!kvm_has_feat(vcpu->kvm, ID_AA64MMFR4_EL1, NV_frac, NV2_ONLY))
 		return 0;
 
-	vcpu->arch.vncr_tlb = kzalloc_obj(*vcpu->arch.vncr_tlb,
-					  GFP_KERNEL_ACCOUNT);
+	if (!vcpu->arch.vncr_tlb) {
+		struct vncr_tlb *vt = kzalloc_obj(*vcpu->arch.vncr_tlb,
+						  GFP_KERNEL_ACCOUNT);
+
+		scoped_guard(write_lock, &vcpu->kvm->mmu_lock)
+			vcpu->arch.vncr_tlb = vt;
+	}
+
 	if (!vcpu->arch.vncr_tlb)
 		return -ENOMEM;
 
Signature · 1 line
2.47.3
#1 lore raw
Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider:

Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider:

Pre-existing issues:
- [Critical] Missing NULL check for vcpu->arch.vncr_tlb in invalidate_vncr_va() causes a host kernel panic when iterating over uninitialized vCPUs.
- [High] The lazily allocated vncr_tlb structure leaves the cpu field zero-initialized, causing it to inadvertently unmap CPU 0's VNCR fixmap on the first fault.
- [High] Resource leak of struct page references on error paths in kvm_translate_vncr().
--

commit cf6d236759b9eb5ba69620ec32a536af7c15a9a2
Author: Marc Zyngier <[email protected]>

KVM: arm64: Hold kvm->mmu_lock while initialising vcpu->arch.vncr_tlb

This commit fixes a race condition and memory leak in the initialization of vncr_tlb by taking the mmu_lock during assignment and verifying it is NULL beforehand.

arch/arm64/kvm/nested.c int kvm_vcpu_allocate_vncr_tlb(struct kvm_vcpu *vcpu) · line 1253

Quoted 13 lines
arch/arm64/kvm/nested.c +3 -2
index 883b6c1008fbb..21c9d1f07d7a8 100644
@@ -1248,8 +1248,14 @@ int kvm_vcpu_allocate_vncr_tlb(struct kvm_vcpu *vcpu)
 	if (!kvm_has_feat(vcpu->kvm, ID_AA64MMFR4_EL1, NV_frac, NV2_ONLY))
 		return 0;
 
-	vcpu->arch.vncr_tlb = kzalloc_obj(*vcpu->arch.vncr_tlb,
-					  GFP_KERNEL_ACCOUNT);
+	if (!vcpu->arch.vncr_tlb) {
+		struct vncr_tlb *vt = kzalloc_obj(*vcpu->arch.vncr_tlb,
+						  GFP_KERNEL_ACCOUNT);

[Severity: High]
This is a pre-existing issue, but since we are allocating the vncr_tlb here using kzalloc_obj(), vt->cpu will be zero-initialized.

Looking at invalidate_vncr(), if a vCPU encounters its first VNCR fault, kvm_translate_vncr() calls invalidate_vncr(vt). Because vt->cpu is 0 instead of -1, the following check passes:

arch/arm64/kvm/nested.c:invalidate_vncr() {
        ...
        if (vt->cpu != -1)
                clear_fixmap(vncr_fixmap(vt->cpu));
        ...
}

Does this inadvertently unmap CPU 0's fixmap, potentially corrupting the translation state if another vCPU is actively mapped on CPU 0? Could we initialize vt->cpu to -1 right after this allocation?

arch/arm64/kvm/nested.c int kvm_vcpu_allocate_vncr_tlb(struct kvm_vcpu *vcpu) · line 1265

Quoted 7 lines
arch/arm64/kvm/nested.c +5 -0
+
+		scoped_guard(write_lock, &vcpu->kvm->mmu_lock)
+			vcpu->arch.vncr_tlb = vt;
+	}
+
 	if (!vcpu->arch.vncr_tlb)
 		return -ENOMEM;

[Severity: Critical]
This isn't a bug introduced by this patch, but while reviewing the vncr_tlb lifecycle, I noticed a potential issue in invalidate_vncr_va() when iterating over uninitialized vCPUs.

If an online vCPU triggers an EL2 TLBI instruction before another vCPU in the same VM has run, could this lead to a NULL pointer dereference?

arch/arm64/kvm/nested.c:invalidate_vncr_va() {
        ...
        kvm_for_each_vcpu(i, vcpu, kvm) {
                struct vncr_tlb *vt = vcpu->arch.vncr_tlb;

                if (!vt->valid)
                        continue;
        ...
}

Since vncr_tlb is allocated lazily on the first run, vt could be NULL here. Does this path need a NULL check before accessing vt->valid?

[Severity: High]
This is also a pre-existing issue, but I noticed a potential page reference leak in kvm_translate_vncr() on the error paths.

When __kvm_faultin_pfn() successfully pins a page, the early return for a write fault without a writable page bypasses kvm_release_faultin_page():

arch/arm64/kvm/nested.c:kvm_translate_vncr() {
        ...
        if (is_error_noslot_pfn(pfn) || (write_fault && !writable))
                return -EFAULT;
        ...
}

Similarly, the MMU retry check also returns directly:

arch/arm64/kvm/nested.c:kvm_translate_vncr() {
        ...
        if (mmu_invalidate_retry(vcpu->kvm, mmu_seq))
                return -EAGAIN;
        ...
}

Do these error paths leave the page pinned and permanently leak the struct page reference count?

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
author maintainer
#2 lore raw
Quoted 49 lines · On Mon, 08 Jun 2026 09:26:02 +0100, [email protected] wrote:

Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider:

Pre-existing issues:
- [Critical] Missing NULL check for vcpu->arch.vncr_tlb in invalidate_vncr_va() causes a host kernel panic when iterating over uninitialized vCPUs.
- [High] The lazily allocated vncr_tlb structure leaves the cpu field zero-initialized, causing it to inadvertently unmap CPU 0's VNCR fixmap on the first fault.
- [High] Resource leak of struct page references on error paths in kvm_translate_vncr().
--

commit cf6d236759b9eb5ba69620ec32a536af7c15a9a2
Author: Marc Zyngier <[email protected]>

KVM: arm64: Hold kvm->mmu_lock while initialising vcpu->arch.vncr_tlb

This commit fixes a race condition and memory leak in the initialization of vncr_tlb by taking the mmu_lock during assignment and verifying it is NULL beforehand.

arch/arm64/kvm/nested.c int kvm_vcpu_allocate_vncr_tlb(struct kvm_vcpu *vcpu) · line 1253

Quoted 13 lines
arch/arm64/kvm/nested.c +3 -2
index 883b6c1008fbb..21c9d1f07d7a8 100644
@@ -1248,8 +1248,14 @@ int kvm_vcpu_allocate_vncr_tlb(struct kvm_vcpu *vcpu)
 	if (!kvm_has_feat(vcpu->kvm, ID_AA64MMFR4_EL1, NV_frac, NV2_ONLY))
 		return 0;
 
-	vcpu->arch.vncr_tlb = kzalloc_obj(*vcpu->arch.vncr_tlb,
-					  GFP_KERNEL_ACCOUNT);
+	if (!vcpu->arch.vncr_tlb) {
+		struct vncr_tlb *vt = kzalloc_obj(*vcpu->arch.vncr_tlb,
+						  GFP_KERNEL_ACCOUNT);

[Severity: High]
This is a pre-existing issue, but since we are allocating the vncr_tlb here using kzalloc_obj(), vt->cpu will be zero-initialized.

Looking at invalidate_vncr(), if a vCPU encounters its first VNCR fault, kvm_translate_vncr() calls invalidate_vncr(vt). Because vt->cpu is 0 instead of -1, the following check passes:

arch/arm64/kvm/nested.c:invalidate_vncr() {
        ...
        if (vt->cpu != -1)
                clear_fixmap(vncr_fixmap(vt->cpu));
        ...
}
Quoted 3 lines

Does this inadvertently unmap CPU 0's fixmap, potentially corrupting the translation state if another vCPU is actively mapped on CPU 0? Could we initialize vt->cpu to -1 right after this allocation?

This is only evaluated when valid is true, and this is checked in the calling context. Since valid is initialised to false by virtue of the structure being 0-initialised, this path doesn't look possible.

Quoted 27 lines

arch/arm64/kvm/nested.c int kvm_vcpu_allocate_vncr_tlb(struct kvm_vcpu *vcpu) · line 1265

Quoted 8 lines
arch/arm64/kvm/nested.c +5 -0
+
+		scoped_guard(write_lock, &vcpu->kvm->mmu_lock)
+			vcpu->arch.vncr_tlb = vt;
+	}
+
 	if (!vcpu->arch.vncr_tlb)
 		return -ENOMEM;

[Severity: Critical]
This isn't a bug introduced by this patch, but while reviewing the vncr_tlb lifecycle, I noticed a potential issue in invalidate_vncr_va() when iterating over uninitialized vCPUs.

If an online vCPU triggers an EL2 TLBI instruction before another vCPU in the same VM has run, could this lead to a NULL pointer dereference?

arch/arm64/kvm/nested.c:invalidate_vncr_va() {
        ...
        kvm_for_each_vcpu(i, vcpu, kvm) {
                struct vncr_tlb *vt = vcpu->arch.vncr_tlb;

                if (!vt->valid)
                        continue;
        ...
}
Quoted 3 lines

Since vncr_tlb is allocated lazily on the first run, vt could be NULL here. Does this path need a NULL check before accessing vt->valid?

This is already fixed by a separate patch
(https://lore.kernel.org/all/[email protected]/)

Quoted 7 lines

[Severity: High]
This is also a pre-existing issue, but I noticed a potential page reference leak in kvm_translate_vncr() on the error paths.

When __kvm_faultin_pfn() successfully pins a page, the early return for a write fault without a writable page bypasses kvm_release_faultin_page():

Quoted 6 lines
arch/arm64/kvm/nested.c:kvm_translate_vncr() {
        ...
        if (is_error_noslot_pfn(pfn) || (write_fault && !writable))
                return -EFAULT;
        ...
}

This looks like a real issue indeed.

Quoted 8 lines

Similarly, the MMU retry check also returns directly:

arch/arm64/kvm/nested.c:kvm_translate_vncr() {
        ...
        if (mmu_invalidate_retry(vcpu->kvm, mmu_seq))
                return -EAGAIN;
        ...
}

This one has been fixed already:
https://patch.msgid.link/[email protected]

	M.
Signature · 1 line
Without deviation from the norm, progress is not possible.
maintainer Reviewed-by on v1 1/1
#3 lore raw
In reply to Marc Zyngier
Quoted 8 lines · Marc Zyngier

Sashiko reports that there is a race between initialising vncr_tlb and making use of it, as we don't hold the mmu_lock at this point.

Additionally, it identifies a memory leak, should userspace repeatedly invokes the KVM_RUN ioctl after a failure of kvm_arch_vcpu_run_pid_change(), as we assign vncr_tlb blindly on first run, irrespective of prior allocations.

Quoted 7 lines

Slap the two bugs in one go by taking the kvm->mmu_lock on assigning vncr_tlb, preventing the race for good, and by checking that vncr_tlb is indeed NULL prior to allocation.

Thanks,
Oliver

Quoted 28 lines
---
 arch/arm64/kvm/nested.c | 10 ++++++++--
 1 file changed, 8 insertions(+), 2 deletions(-)
arch/arm64/kvm/nested.c +8 -2
index 690b8e8564166..d11e36b3cfcc2 100644
@@ -1253,8 +1253,14 @@ int kvm_vcpu_allocate_vncr_tlb(struct kvm_vcpu *vcpu)
 	if (!kvm_has_feat(vcpu->kvm, ID_AA64MMFR4_EL1, NV_frac, NV2_ONLY))
 		return 0;
 
-	vcpu->arch.vncr_tlb = kzalloc_obj(*vcpu->arch.vncr_tlb,
-					  GFP_KERNEL_ACCOUNT);
+	if (!vcpu->arch.vncr_tlb) {
+		struct vncr_tlb *vt = kzalloc_obj(*vcpu->arch.vncr_tlb,
+						  GFP_KERNEL_ACCOUNT);
+
+		scoped_guard(write_lock, &vcpu->kvm->mmu_lock)
+			vcpu->arch.vncr_tlb = vt;
+	}
+
 	if (!vcpu->arch.vncr_tlb)
 		return -ENOMEM;
 
Signature · 1 line
2.47.3
#4 lore raw
In reply to Marc Zyngier
Quoted 26 lines · Marc Zyngier

Sashiko reports that there is a race between initialising vncr_tlb and making use of it, as we don't hold the mmu_lock at this point.

Additionally, it identifies a memory leak, should userspace repeatedly invokes the KVM_RUN ioctl after a failure of kvm_arch_vcpu_run_pid_change(), as we assign vncr_tlb blindly on first run, irrespective of prior allocations.

Slap the two bugs in one go by taking the kvm->mmu_lock on assigning vncr_tlb, preventing the race for good, and by checking that vncr_tlb is indeed NULL prior to allocation.

---
 arch/arm64/kvm/nested.c | 10 ++++++++--
 1 file changed, 8 insertions(+), 2 deletions(-)
arch/arm64/kvm/nested.c +0 -0
index 690b8e8564166..d11e36b3cfcc2 100644
@@ -1253,8 +1253,14 @@ int kvm_vcpu_allocate_vncr_tlb(struct kvm_vcpu *vcpu)
 	if (!kvm_has_feat(vcpu->kvm, ID_AA64MMFR4_EL1, NV_frac, NV2_ONLY))
 		return 0;

arch/arm64/kvm/nested.c int kvm_vcpu_allocate_vncr_tlb(struct kvm_vcpu *vcpu) · line 1262

Quoted 10 lines
arch/arm64/kvm/nested.c +7 -2
 
-	vcpu->arch.vncr_tlb = kzalloc_obj(*vcpu->arch.vncr_tlb,
-					  GFP_KERNEL_ACCOUNT);
+	if (!vcpu->arch.vncr_tlb) {
+		struct vncr_tlb *vt = kzalloc_obj(*vcpu->arch.vncr_tlb,
+						  GFP_KERNEL_ACCOUNT);
+
+		scoped_guard(write_lock, &vcpu->kvm->mmu_lock)
+			vcpu->arch.vncr_tlb = vt;
+	}

(I am not familiar with this code at all, so apologies in advance if I am making an idiot out of myself here)

IIUC, the point of holding the lock here is *not* to protect against concurrent initialization, as in this case the NULL check needs to be done under the lock.

Rather, the goal is to prevent re-ordering of zeroing from kzalloc and the assignment to vcpu->arch.vncr_tlb, by depending on the barriers provided by the lock. The lock is held by the readers so holding it here conviently means we do not need to add any barriers to the readers.

Is my understanding correct?

If yes, I think the code looks confusing, at least to a layman like myself. It initially seems like the lock protects against concurrent initializations, but then the NULL check is not done again under the lock. The goal of the lock is not clear without the original report.

Mayeb it's clearer to explicitly use barriers if the goal is preventing reordering?

author maintainer
#5 lore raw
Quoted 46 lines · Yosry Ahmed
Quoted 26 lines · Marc Zyngier

Sashiko reports that there is a race between initialising vncr_tlb and making use of it, as we don't hold the mmu_lock at this point.

Additionally, it identifies a memory leak, should userspace repeatedly invokes the KVM_RUN ioctl after a failure of kvm_arch_vcpu_run_pid_change(), as we assign vncr_tlb blindly on first run, irrespective of prior allocations.

Slap the two bugs in one go by taking the kvm->mmu_lock on assigning vncr_tlb, preventing the race for good, and by checking that vncr_tlb is indeed NULL prior to allocation.

---
 arch/arm64/kvm/nested.c | 10 ++++++++--
 1 file changed, 8 insertions(+), 2 deletions(-)
arch/arm64/kvm/nested.c +0 -0
index 690b8e8564166..d11e36b3cfcc2 100644
@@ -1253,8 +1253,14 @@ int kvm_vcpu_allocate_vncr_tlb(struct kvm_vcpu *vcpu)
 	if (!kvm_has_feat(vcpu->kvm, ID_AA64MMFR4_EL1, NV_frac, NV2_ONLY))
 		return 0;

arch/arm64/kvm/nested.c int kvm_vcpu_allocate_vncr_tlb(struct kvm_vcpu *vcpu) · line 1262

Quoted 10 lines
arch/arm64/kvm/nested.c +7 -2
 
-	vcpu->arch.vncr_tlb = kzalloc_obj(*vcpu->arch.vncr_tlb,
-					  GFP_KERNEL_ACCOUNT);
+	if (!vcpu->arch.vncr_tlb) {
+		struct vncr_tlb *vt = kzalloc_obj(*vcpu->arch.vncr_tlb,
+						  GFP_KERNEL_ACCOUNT);
+
+		scoped_guard(write_lock, &vcpu->kvm->mmu_lock)
+			vcpu->arch.vncr_tlb = vt;
+	}

(I am not familiar with this code at all, so apologies in advance if I am making an idiot out of myself here)

IIUC, the point of holding the lock here is *not* to protect against concurrent initialization, as in this case the NULL check needs to be done under the lock.

Quoted 6 lines

Rather, the goal is to prevent re-ordering of zeroing from kzalloc and the assignment to vcpu->arch.vncr_tlb, by depending on the barriers provided by the lock. The lock is held by the readers so holding it here conviently means we do not need to add any barriers to the readers.

Is my understanding correct?

It is correct.

Quoted 8 lines

If yes, I think the code looks confusing, at least to a layman like myself. It initially seems like the lock protects against concurrent initializations, but then the NULL check is not done again under the lock. The goal of the lock is not clear without the original report.

Mayeb it's clearer to explicitly use barriers if the goal is preventing reordering?

This would require both the initialisation of vncr_tlb to use a store release, *and* all the other call sites to use a load acquire.

I really don't think it is worth the churn, nor the (very small) burden on the readers.

Thanks,

	M.
Signature · 1 line
Without deviation from the norm, progress is not possible.
#6 lore raw
Quoted 5 lines
Quoted 5 lines

If yes, I think the code looks confusing, at least to a layman like myself. It initially seems like the lock protects against concurrent initializations, but then the NULL check is not done again under the lock. The goal of the lock is not clear without the original report.

Quoted 8 lines
Quoted 2 lines

Mayeb it's clearer to explicitly use barriers if the goal is preventing reordering?

This would require both the initialisation of vncr_tlb to use a store release, *and* all the other call sites to use a load acquire.

I really don't think it is worth the churn, nor the (very small) burden on the readers.

That's fair. I was mainly just pointing out my initial confusion and that others may share it. Avoiding the churn on the readers' side is understandable. Maybe a comment here would help explain why the lock needs to be held?

author maintainer
#7 lore raw
Quoted 12 lines · Yosry Ahmed
Quoted 12 lines
Quoted 8 lines

If yes, I think the code looks confusing, at least to a layman like myself. It initially seems like the lock protects against concurrent initializations, but then the NULL check is not done again under the lock. The goal of the lock is not clear without the original report.

Mayeb it's clearer to explicitly use barriers if the goal is preventing reordering?

This would require both the initialisation of vncr_tlb to use a store release, *and* all the other call sites to use a load acquire.

Quoted 7 lines
Quoted 2 lines

I really don't think it is worth the churn, nor the (very small) burden on the readers.

That's fair. I was mainly just pointing out my initial confusion and that others may share it. Avoiding the churn on the readers' side is understandable. Maybe a comment here would help explain why the lock needs to be held?

I have added this:

/*
* Taking the lock on assignment ensures that the TLB is
* seen as initialised when following the pointer (release
* semantics of the unlock), and avoids having acquires on
* each user which already take the lock.
*/

Thanks,

	M.
Signature · 1 line
Without deviation from the norm, progress is not possible.
#8 lore raw
Quoted 23 lines · Marc Zyngier
Quoted 12 lines · Yosry Ahmed
Quoted 12 lines
Quoted 8 lines

If yes, I think the code looks confusing, at least to a layman like myself. It initially seems like the lock protects against concurrent initializations, but then the NULL check is not done again under the lock. The goal of the lock is not clear without the original report.

Mayeb it's clearer to explicitly use barriers if the goal is preventing reordering?

This would require both the initialisation of vncr_tlb to use a store release, *and* all the other call sites to use a load acquire.

Quoted 7 lines
Quoted 2 lines

I really don't think it is worth the churn, nor the (very small) burden on the readers.

That's fair. I was mainly just pointing out my initial confusion and that others may share it. Avoiding the churn on the readers' side is understandable. Maybe a comment here would help explain why the lock needs to be held?

Quoted 8 lines

I have added this:

/*
* Taking the lock on assignment ensures that the TLB is
* seen as initialised when following the pointer (release
* semantics of the unlock), and avoids having acquires on
* each user which already take the lock.
*/

Looks good, thank you!