KVM pgtable fixes

Hi stable folks,

I noticed that these two KVM/arm64 pgtable fixes are missing from 6.6.y so I've done the backports. The second one is also needed in 6.1.y but it needs some tweaks so I'll post a separate backport for that.

Cheers,

Will

--->8

Will Deacon (2):
KVM: arm64: Don't defer TLB invalidation when zapping table entries
KVM: arm64: Don't pass a TLBI level hint when zapping table entries

 arch/arm64/kvm/hyp/pgtable.c | 12 ++++++++----
 1 file changed, 8 insertions(+), 4 deletions(-)

2.46.0.184.g6999bdac58-goog

commit f62d4c3eb687d87b616b4279acec7862553bda77 upstream.

Commit 7657ea920c54 ("KVM: arm64: Use TLBI range-based instructions for unmap") introduced deferred TLB invalidation for the stage-2 page-table so that range-based invalidation can be used for the accumulated addresses. This works fine if the structure of the page-tables remains unchanged, but if entire tables are zapped and subsequently freed then we transiently leave the hardware page-table walker with a reference to freed memory thanks to the translation walk caches. For example, stage2_unmap_walker() will free page-table pages:

	if (childp)
		mm_ops->put_page(childp);

and issue the TLB invalidation later in kvm_pgtable_stage2_unmap():

	if (stage2_unmap_defer_tlb_flush(pgt))
		/* Perform the deferred TLB invalidations */
		kvm_tlb_flush_vmid_range(pgt->mmu, addr, size);

For now, take the conservative approach and invalidate the TLB eagerly when we clear a table entry. Note, however, that the existing level hint passed to __kvm_tlb_flush_vmid_ipa() is incorrect and will be fixed in a subsequent patch.

---
 arch/arm64/kvm/hyp/pgtable.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

index 15aa9bad1c28..6692327fabe7 100644
@@ -861,9 +861,11 @@ static void stage2_unmap_put_pte(const struct kvm_pgtable_visit_ctx *ctx,
 	if (kvm_pte_valid(ctx->old)) {
 		kvm_clear_pte(ctx->ptep);
 
-		if (!stage2_unmap_defer_tlb_flush(pgt))
+		if (!stage2_unmap_defer_tlb_flush(pgt) ||
+		    kvm_pte_table(ctx->old, ctx->level)) {
 			kvm_call_hyp(__kvm_tlb_flush_vmid_ipa, mmu,
 					ctx->addr, ctx->level);
+		}
 	}
 
 	mm_ops->put_page(ctx->ptep);

2.46.0.184.g6999bdac58-goog

This is a note to let you know that I've just added the patch titled

KVM: arm64: Don't defer TLB invalidation when zapping table entries

to the 6.6-stable tree which can be found at:
http://www.kernel.org/git/?p=linux/kernel/git/stable/stable-queue.git;a=summary

The filename of the patch is:
kvm-arm64-don-t-defer-tlb-invalidation-when-zapping-table-entries.patch
and it can be found in the queue-6.6 subdirectory.

If you, or anyone else, feels it should not be added to the stable tree, please let <[email protected]> know about it.

From [email protected] Thu Aug 15 14:49:10 2024
From: Will Deacon <[email protected]>
Date: Thu, 15 Aug 2024 13:46:25 +0100
Subject: KVM: arm64: Don't defer TLB invalidation when zapping table entries
To: [email protected]
Cc: Will Deacon <[email protected]>, Marc Zyngier <[email protected]>, Oliver Upton <[email protected]>, [email protected], Raghavendra Rao Ananta <[email protected]>, Shaoqin Huang <[email protected]>
Message-ID: <[email protected]>

From: Will Deacon <[email protected]>

commit f62d4c3eb687d87b616b4279acec7862553bda77 upstream.

Commit 7657ea920c54 ("KVM: arm64: Use TLBI range-based instructions for unmap") introduced deferred TLB invalidation for the stage-2 page-table so that range-based invalidation can be used for the accumulated addresses. This works fine if the structure of the page-tables remains unchanged, but if entire tables are zapped and subsequently freed then we transiently leave the hardware page-table walker with a reference to freed memory thanks to the translation walk caches. For example, stage2_unmap_walker() will free page-table pages:

	if (childp)
		mm_ops->put_page(childp);

and issue the TLB invalidation later in kvm_pgtable_stage2_unmap():

	if (stage2_unmap_defer_tlb_flush(pgt))
		/* Perform the deferred TLB invalidations */
		kvm_tlb_flush_vmid_range(pgt->mmu, addr, size);

For now, take the conservative approach and invalidate the TLB eagerly when we clear a table entry. Note, however, that the existing level hint passed to __kvm_tlb_flush_vmid_ipa() is incorrect and will be fixed in a subsequent patch.

---
 arch/arm64/kvm/hyp/pgtable.c |    4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

@@ -861,9 +861,11 @@ static void stage2_unmap_put_pte(const s
 	if (kvm_pte_valid(ctx->old)) {
 		kvm_clear_pte(ctx->ptep);
 
-		if (!stage2_unmap_defer_tlb_flush(pgt))
+		if (!stage2_unmap_defer_tlb_flush(pgt) ||
+		    kvm_pte_table(ctx->old, ctx->level)) {
 			kvm_call_hyp(__kvm_tlb_flush_vmid_ipa, mmu,
 					ctx->addr, ctx->level);
+		}
 	}
 
 	mm_ops->put_page(ctx->ptep);

Patches currently in stable-queue which might be from [email protected] are

queue-6.6/kvm-arm64-don-t-defer-tlb-invalidation-when-zapping-table-entries.patch queue-6.6/kvm-arm64-don-t-pass-a-tlbi-level-hint-when-zapping-table-entries.patch

commit 36e008323926036650299cfbb2dca704c7aba849 upstream.

The TLBI level hints are for leaf entries only, so take care not to pass them incorrectly after clearing a table entry.

---
 arch/arm64/kvm/hyp/pgtable.c | 12 +++++++-----
 1 file changed, 7 insertions(+), 5 deletions(-)

index 6692327fabe7..ca0bf0b92ca0 100644
@@ -523,7 +523,7 @@ static int hyp_unmap_walker(const struct kvm_pgtable_visit_ctx *ctx,
 
 		kvm_clear_pte(ctx->ptep);
 		dsb(ishst);
-		__tlbi_level(vae2is, __TLBI_VADDR(ctx->addr, 0), ctx->level);
+		__tlbi_level(vae2is, __TLBI_VADDR(ctx->addr, 0), 0);
 	} else {
 		if (ctx->end - ctx->addr < granule)
 			return -EINVAL;

@@ -861,10 +861,12 @@ static void stage2_unmap_put_pte(const struct kvm_pgtable_visit_ctx *ctx,
 	if (kvm_pte_valid(ctx->old)) {
 		kvm_clear_pte(ctx->ptep);
 
-		if (!stage2_unmap_defer_tlb_flush(pgt) ||
-		    kvm_pte_table(ctx->old, ctx->level)) {
-			kvm_call_hyp(__kvm_tlb_flush_vmid_ipa, mmu,
-					ctx->addr, ctx->level);
+		if (kvm_pte_table(ctx->old, ctx->level)) {
+			kvm_call_hyp(__kvm_tlb_flush_vmid_ipa, mmu, ctx->addr,
+				     0);
+		} else if (!stage2_unmap_defer_tlb_flush(pgt)) {
+			kvm_call_hyp(__kvm_tlb_flush_vmid_ipa, mmu, ctx->addr,
+				     ctx->level);
 		}
 	}
 

2.46.0.184.g6999bdac58-goog

This is a note to let you know that I've just added the patch titled

KVM: arm64: Don't pass a TLBI level hint when zapping table entries

to the 6.6-stable tree which can be found at:
http://www.kernel.org/git/?p=linux/kernel/git/stable/stable-queue.git;a=summary

The filename of the patch is:
kvm-arm64-don-t-pass-a-tlbi-level-hint-when-zapping-table-entries.patch
and it can be found in the queue-6.6 subdirectory.

If you, or anyone else, feels it should not be added to the stable tree, please let <[email protected]> know about it.

From [email protected] Thu Aug 15 14:49:11 2024
From: Will Deacon <[email protected]>
Date: Thu, 15 Aug 2024 13:46:26 +0100
Subject: KVM: arm64: Don't pass a TLBI level hint when zapping table entries
To: [email protected]
Cc: Will Deacon <[email protected]>, Marc Zyngier <[email protected]>, Oliver Upton <[email protected]>, [email protected], Gavin Shan <[email protected]>, Quentin Perret <[email protected]>, Shaoqin Huang <[email protected]>
Message-ID: <[email protected]>

From: Will Deacon <[email protected]>

commit 36e008323926036650299cfbb2dca704c7aba849 upstream.

The TLBI level hints are for leaf entries only, so take care not to pass them incorrectly after clearing a table entry.

---
 arch/arm64/kvm/hyp/pgtable.c |   12 +++++++-----
 1 file changed, 7 insertions(+), 5 deletions(-)

@@ -523,7 +523,7 @@ static int hyp_unmap_walker(const struct
 
 		kvm_clear_pte(ctx->ptep);
 		dsb(ishst);
-		__tlbi_level(vae2is, __TLBI_VADDR(ctx->addr, 0), ctx->level);
+		__tlbi_level(vae2is, __TLBI_VADDR(ctx->addr, 0), 0);
 	} else {
 		if (ctx->end - ctx->addr < granule)
 			return -EINVAL;

@@ -861,10 +861,12 @@ static void stage2_unmap_put_pte(const s
 	if (kvm_pte_valid(ctx->old)) {
 		kvm_clear_pte(ctx->ptep);
 
-		if (!stage2_unmap_defer_tlb_flush(pgt) ||
-		    kvm_pte_table(ctx->old, ctx->level)) {
-			kvm_call_hyp(__kvm_tlb_flush_vmid_ipa, mmu,
-					ctx->addr, ctx->level);
+		if (kvm_pte_table(ctx->old, ctx->level)) {
+			kvm_call_hyp(__kvm_tlb_flush_vmid_ipa, mmu, ctx->addr,
+				     0);
+		} else if (!stage2_unmap_defer_tlb_flush(pgt)) {
+			kvm_call_hyp(__kvm_tlb_flush_vmid_ipa, mmu, ctx->addr,
+				     ctx->level);
 		}
 	}
 

Patches currently in stable-queue which might be from [email protected] are

queue-6.6/kvm-arm64-don-t-defer-tlb-invalidation-when-zapping-table-entries.patch queue-6.6/kvm-arm64-don-t-pass-a-tlbi-level-hint-when-zapping-table-entries.patch

Now queued up, thanks.

greg k-h