KVM: arm64: Always invalidate TLB for stage-2 permission faults

It is possible for multiple vCPUs to fault on the same IPA and attempt to resolve the fault. One of the page table walks will actually update the PTE and the rest will return -EAGAIN per our race detection scheme. KVM elides the TLB invalidation on the racing threads as the return value is nonzero.

Before commit a12ab1378a88 ("KVM: arm64: Use local TLBI on permission relaxation") KVM always used broadcast TLB invalidations when handling permission faults, which had the convenient property of making the stage-2 updates visible to all CPUs in the system. However now we do a local invalidation, and TLBI elision leads to vCPUs getting stuck in a permission fault loop. Remember that the architecture permits the TLB to cache translations that precipitate a permission fault.

Invalidate the TLB entry responsible for the permission fault if the stage-2 descriptor has been relaxed, regardless of which thread actually did the job.

Cc [email protected]

Fixes a12ab1378a88 ("KVM: arm64: Use local TLBI on permission relaxation")

Signed-off-by Oliver Upton <[email protected]>

---
 arch/arm64/kvm/hyp/pgtable.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

arch/arm64/kvm/hyp/pgtable.c +1 -1

index f155b8c9e98c..286888751793 100644
@@ -1314,7 +1314,7 @@ int kvm_pgtable_stage2_relax_perms(struct kvm_pgtable *pgt, u64 addr,
 	ret = stage2_update_leaf_attrs(pgt, addr, 1, set, clr, NULL, &level,
 				       KVM_PGTABLE_WALK_HANDLE_FAULT |
 				       KVM_PGTABLE_WALK_SHARED);
-	if (!ret)
+	if (!ret || ret == -EAGAIN)
 		kvm_call_hyp(__kvm_tlb_flush_vmid_ipa_nsh, pgt->mmu, addr, level);
 	return ret;
 }

base-commit: ce9ecca0238b140b88f43859b211c9fdfd8e5b70

Signature · 1 line

2.42.0.515.g380fc7ccd1-goog

Quoted 6 lines · Oliver Upton

It is possible for multiple vCPUs to fault on the same IPA and attempt to resolve the fault. One of the page table walks will actually update the PTE and the rest will return -EAGAIN per our race detection scheme. KVM elides the TLB invalidation on the racing threads as the return value is nonzero.

Quoted 7 lines

Before commit a12ab1378a88 ("KVM: arm64: Use local TLBI on permission relaxation") KVM always used broadcast TLB invalidations when handling permission faults, which had the convenient property of making the stage-2 updates visible to all CPUs in the system. However now we do a local invalidation, and TLBI elision leads to vCPUs getting stuck in a permission fault loop. Remember that the architecture permits the TLB to cache translations that precipitate a permission fault.

The effects of this are slightly overstated (got ahead of myself). EAGAIN only crops up if the cmpxchg() fails, we return 0 if the PTE didn't need to be updated.

On the subsequent permission fault we'll do the right thing and invalidate the TLB, so this change is purely an optimization rather than a correctness issue.

Quoted 20 lines · Oliver Upton

Quoted 6 lines · Oliver Upton

It is possible for multiple vCPUs to fault on the same IPA and attempt to resolve the fault. One of the page table walks will actually update the PTE and the rest will return -EAGAIN per our race detection scheme. KVM elides the TLB invalidation on the racing threads as the return value is nonzero.

Quoted 7 lines

Before commit a12ab1378a88 ("KVM: arm64: Use local TLBI on permission relaxation") KVM always used broadcast TLB invalidations when handling permission faults, which had the convenient property of making the stage-2 updates visible to all CPUs in the system. However now we do a local invalidation, and TLBI elision leads to vCPUs getting stuck in a permission fault loop. Remember that the architecture permits the TLB to cache translations that precipitate a permission fault.

The effects of this are slightly overstated (got ahead of myself). EAGAIN only crops up if the cmpxchg() fails, we return 0 if the PTE didn't need to be updated.

Quoted 3 lines

On the subsequent permission fault we'll do the right thing and invalidate the TLB, so this change is purely an optimization rather than a correctness issue.

Can you measure the actual effect of this change? In my (limited) experience, I had to actually trick the guest into doing this, and opportunistically invalidating TLBs didn't have any significant benefit.

Quoted 22 lines · Marc Zyngier

Quoted 20 lines · Oliver Upton

Quoted 6 lines · Oliver Upton

It is possible for multiple vCPUs to fault on the same IPA and attempt to resolve the fault. One of the page table walks will actually update the PTE and the rest will return -EAGAIN per our race detection scheme. KVM elides the TLB invalidation on the racing threads as the return value is nonzero.

Quoted 7 lines

Before commit a12ab1378a88 ("KVM: arm64: Use local TLBI on permission relaxation") KVM always used broadcast TLB invalidations when handling permission faults, which had the convenient property of making the stage-2 updates visible to all CPUs in the system. However now we do a local invalidation, and TLBI elision leads to vCPUs getting stuck in a permission fault loop. Remember that the architecture permits the TLB to cache translations that precipitate a permission fault.

The effects of this are slightly overstated (got ahead of myself). EAGAIN only crops up if the cmpxchg() fails, we return 0 if the PTE didn't need to be updated.

Quoted 8 lines

Quoted 3 lines

On the subsequent permission fault we'll do the right thing and invalidate the TLB, so this change is purely an optimization rather than a correctness issue.

Can you measure the actual effect of this change? In my (limited) experience, I had to actually trick the guest into doing this, and opportunistically invalidating TLBs didn't have any significant benefit.

Sure. We were debugging some issues of vCPU hangs during post-copy migration but that's more likely to be an issue with our VMM + out of tree code.

Marginal improvements be damned, I'm still somewhat keen on doing the TLB invalidation upon race detection anyway. Going back to the guest is pointless, since in all likelihood we will hit the TLB entry that led to the permission fault in the first place.

It is possible for multiple vCPUs to fault on the same IPA and attempt to resolve the fault. One of the page table walks will actually update the PTE and the rest will return -EAGAIN per our race detection scheme. KVM elides the TLB invalidation on the racing threads as the return value is nonzero.

Before commit a12ab1378a88 ("KVM: arm64: Use local TLBI on permission relaxation") KVM always used broadcast TLB invalidations when handling permission faults, which had the convenient property of making the stage-2 updates visible to all CPUs in the system. However now we do a local invalidation, and TLBI elision leads to vCPUs getting stuck in a permission fault loop. Remember that the architecture permits the TLB to cache translations that precipitate a permission fault.

[...]