Skip to contentSkip to conversation

KVM: x86/mmu: Fix wrong usages of range-based tlb flushing

PATCH Superseded Hou Wenlong 5 participants 25 messages Aug 24, 2022 – Sep 27, 2022

Subsystems (3)

v2 of 4 Superseded Latest is v4 →

lore
Share & tools
lore
#0 lore raw

v2 cover KVM: x86/mmu: Fix wrong usages of range-based tlb flushing

Commit c3134ce240eed
("KVM: Replace old tlb flush function with new one to flush a specified range.") replaces old tlb flush function with kvm_flush_remote_tlbs_with_address() to do tlb flushing. However, the gfn range of tlb flushing is wrong in some cases. E.g., when a spte is dropped, the start gfn of tlb flushing should be the gfn of spte not the base gfn of SP which contains the spte. Although, as Paolo said, Hyper-V may treat a 1-page flush the same if the address points to a huge page, and no fixes are reported so far. So it seems that it works well for Hyper-V. But it would be better to use the correct size for huge page. So this patchset would fix them and introduce some helper functions as David suggested to make the code clear.

Changed from v1:
- Align down gfn in kvm_set_pte_rmapp() instead of change iterator->gfn in rmap_walk_init_level() in Patch 2.
- Introduce some helper functions for common operations as David suggested.

v1: https://lore.kernel.org/kvm/[email protected]

Hou Wenlong (6):
KVM: x86/mmu: Fix wrong gfn range of tlb flushing in
validate_direct_spte()
KVM: x86/mmu: Fix wrong gfn range of tlb flushing in

    kvm_set_pte_rmapp()
  KVM: x86/mmu: Reduce gfn range of tlb flushing in
    tdp_mmu_map_handle_target_level()
  KVM: x86/mmu: Fix wrong start gfn of tlb flushing with range
  KVM: x86/mmu: Introduce helper function to do range-based flushing for
    given page
  KVM: x86/mmu: Use 1 as the size of gfn range for tlb flushing in
    FNAME(invlpg)()
 arch/x86/kvm/mmu/mmu.c          | 35 +++++++++++++++++++++++----------
 arch/x86/kvm/mmu/mmu_internal.h | 10 ++++++++++
 arch/x86/kvm/mmu/paging_tmpl.h  |  4 ++--
 arch/x86/kvm/mmu/tdp_mmu.c      |  6 ++----
 4 files changed, 39 insertions(+), 16 deletions(-)

--
2.31.1

#1 lore raw

PATCH v2 1/6 KVM: x86/mmu: Fix wrong gfn range of tlb flushing in validate_direct_spte()

The spte pointing to the children SP is dropped, so the whole gfn range covered by the children SP should be flushed. Although, Hyper-V may treat a 1-page flush the same if the address points to a huge page, it still would be better to use the correct size of huge page. Also introduce a helper function to do range-based flushing when a direct SP is dropped, which would help prevent future buggy use of kvm_flush_remote_tlbs_with_address() in such case.

Fixes c3134ce240eed ("KVM: Replace old tlb flush function with new one to flush a specified range.")
---
 arch/x86/kvm/mmu/mmu.c | 10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)
arch/x86/kvm/mmu/mmu.c +9 -1
index e418ef3ecfcb..a3578abd8bbc 100644
@@ -260,6 +260,14 @@ void kvm_flush_remote_tlbs_with_address(struct kvm *kvm,
 	kvm_flush_remote_tlbs_with_range(kvm, &range);
 }
 
+/* Flush all memory mapped by the given direct SP. */
+static void kvm_flush_remote_tlbs_direct_sp(struct kvm *kvm, struct kvm_mmu_page *sp)
+{
+	WARN_ON_ONCE(!sp->role.direct);
+	kvm_flush_remote_tlbs_with_address(kvm, sp->gfn,
+					   KVM_PAGES_PER_HPAGE(sp->role.level + 1));
+}
+
 static void mark_mmio_spte(struct kvm_vcpu *vcpu, u64 *sptep, u64 gfn,
 			   unsigned int access)
 {
@@ -2341,7 +2349,7 @@ static void validate_direct_spte(struct kvm_vcpu *vcpu, u64 *sptep,
 			return;
 
 		drop_parent_pte(child, sptep);
-		kvm_flush_remote_tlbs_with_address(vcpu->kvm, child->gfn, 1);
+		kvm_flush_remote_tlbs_direct_sp(vcpu->kvm, child);
 	}
 }
 
Signature · 1 line
2.31.1
#2 lore raw

Re: [PATCH v2 1/6] KVM: x86/mmu: Fix wrong gfn range of tlb flushing in validate_direct_spte()

Quoted 21 lines · Hou Wenlong

The spte pointing to the children SP is dropped, so the whole gfn range covered by the children SP should be flushed. Although, Hyper-V may treat a 1-page flush the same if the address points to a huge page, it still would be better to use the correct size of huge page. Also introduce a helper function to do range-based flushing when a direct SP is dropped, which would help prevent future buggy use of kvm_flush_remote_tlbs_with_address() in such case.

Fixes c3134ce240eed ("KVM: Replace old tlb flush function with new one to flush a specified range.")
---
 arch/x86/kvm/mmu/mmu.c | 10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)
arch/x86/kvm/mmu/mmu.c +0 -0
index e418ef3ecfcb..a3578abd8bbc 100644

arch/x86/kvm/mmu/mmu.c void kvm_flush_remote_tlbs_with_address(struct kvm *kvm, · line 268

Quoted 9 lines
arch/x86/kvm/mmu/mmu.c +6 -0
@@ -260,6 +260,14 @@ void kvm_flush_remote_tlbs_with_address(struct kvm *kvm,
 	kvm_flush_remote_tlbs_with_range(kvm, &range);
 }
 
+/* Flush all memory mapped by the given direct SP. */
+static void kvm_flush_remote_tlbs_direct_sp(struct kvm *kvm, struct kvm_mmu_page *sp)
+{
+	WARN_ON_ONCE(!sp->role.direct);
+	kvm_flush_remote_tlbs_with_address(kvm, sp->gfn,
+					   KVM_PAGES_PER_HPAGE(sp->role.level + 1));

nit: I think it would make sense to introduce kvm_flush_remote_tlbs_gfn() in this patch since you are going to eventually use it here anyway.

Quoted 17 lines
arch/x86/kvm/mmu/mmu.c +3 -1
+}
+
 static void mark_mmio_spte(struct kvm_vcpu *vcpu, u64 *sptep, u64 gfn,
 			   unsigned int access)
 {
@@ -2341,7 +2349,7 @@ static void validate_direct_spte(struct kvm_vcpu *vcpu, u64 *sptep,
 			return;
 
 		drop_parent_pte(child, sptep);
-		kvm_flush_remote_tlbs_with_address(vcpu->kvm, child->gfn, 1);
+		kvm_flush_remote_tlbs_direct_sp(vcpu->kvm, child);
 	}
 }
 
Signature · 1 line
2.31.1
#3 lore raw

Re: [PATCH v2 1/6] KVM: x86/mmu: Fix wrong gfn range of tlb flushing in validate_direct_spte()

Quoted 25 lines · David Matlack
Quoted 24 lines · Hou Wenlong

The spte pointing to the children SP is dropped, so the whole gfn range covered by the children SP should be flushed. Although, Hyper-V may treat a 1-page flush the same if the address points to a huge page, it still would be better to use the correct size of huge page. Also introduce a helper function to do range-based flushing when a direct SP is dropped, which would help prevent future buggy use of kvm_flush_remote_tlbs_with_address() in such case.

Fixes c3134ce240eed ("KVM: Replace old tlb flush function with new one to flush a specified range.")
---
 arch/x86/kvm/mmu/mmu.c | 10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)
arch/x86/kvm/mmu/mmu.c +0 -0
index e418ef3ecfcb..a3578abd8bbc 100644
@@ -260,6 +260,14 @@ void kvm_flush_remote_tlbs_with_address(struct kvm *kvm,
 	kvm_flush_remote_tlbs_with_range(kvm, &range);
 }
 
Quoted 11 lines

arch/x86/kvm/mmu/mmu.c void kvm_flush_remote_tlbs_with_address(struct kvm *kvm, · line 268

Quoted 6 lines
arch/x86/kvm/mmu/mmu.c +6 -0
+/* Flush all memory mapped by the given direct SP. */
+static void kvm_flush_remote_tlbs_direct_sp(struct kvm *kvm, struct kvm_mmu_page *sp)
+{
+	WARN_ON_ONCE(!sp->role.direct);
+	kvm_flush_remote_tlbs_with_address(kvm, sp->gfn,
+					   KVM_PAGES_PER_HPAGE(sp->role.level + 1));

nit: I think it would make sense to introduce kvm_flush_remote_tlbs_gfn() in this patch since you are going to eventually use it here anyway.

OK, I'll do it in the next version. Thanks!

Quoted 17 lines
Quoted 17 lines
arch/x86/kvm/mmu/mmu.c +3 -1
+}
+
 static void mark_mmio_spte(struct kvm_vcpu *vcpu, u64 *sptep, u64 gfn,
 			   unsigned int access)
 {
@@ -2341,7 +2349,7 @@ static void validate_direct_spte(struct kvm_vcpu *vcpu, u64 *sptep,
 			return;
 
 		drop_parent_pte(child, sptep);
-		kvm_flush_remote_tlbs_with_address(vcpu->kvm, child->gfn, 1);
+		kvm_flush_remote_tlbs_direct_sp(vcpu->kvm, child);
 	}
 }
 
Signature · 1 line
2.31.1
#4 lore raw

Re: [PATCH v2 1/6] KVM: x86/mmu: Fix wrong gfn range of tlb flushing in validate_direct_spte()

Quoted 27 lines · Hou Wenlong
Quoted 25 lines · David Matlack
Quoted 24 lines · Hou Wenlong

The spte pointing to the children SP is dropped, so the whole gfn range covered by the children SP should be flushed. Although, Hyper-V may treat a 1-page flush the same if the address points to a huge page, it still would be better to use the correct size of huge page. Also introduce a helper function to do range-based flushing when a direct SP is dropped, which would help prevent future buggy use of kvm_flush_remote_tlbs_with_address() in such case.

Fixes c3134ce240eed ("KVM: Replace old tlb flush function with new one to flush a specified range.")
---
 arch/x86/kvm/mmu/mmu.c | 10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)
arch/x86/kvm/mmu/mmu.c +0 -0
index e418ef3ecfcb..a3578abd8bbc 100644
@@ -260,6 +260,14 @@ void kvm_flush_remote_tlbs_with_address(struct kvm *kvm,
    kvm_flush_remote_tlbs_with_range(kvm, &range);
 }
 
Quoted 6 lines
Quoted 6 lines

arch/x86/kvm/mmu/mmu.c void kvm_flush_remote_tlbs_with_address(struct kvm *kvm, · line 268

Quoted 6 lines
arch/x86/kvm/mmu/mmu.c +6 -0
+/* Flush all memory mapped by the given direct SP. */
+static void kvm_flush_remote_tlbs_direct_sp(struct kvm *kvm, struct kvm_mmu_page *sp)
+{
+   WARN_ON_ONCE(!sp->role.direct);
+   kvm_flush_remote_tlbs_with_address(kvm, sp->gfn,
+                                      KVM_PAGES_PER_HPAGE(sp->role.level + 1));

Do we need "+1" here? sp->role.level=1 means 4k page.
I think here should be “KVM_PAGES_PER_HPAGE(sp->role.level)”

Quoted 24 lines
Quoted 5 lines

nit: I think it would make sense to introduce kvm_flush_remote_tlbs_gfn() in this patch since you are going to eventually use it here anyway.

OK, I'll do it in the next version. Thanks!

Quoted 17 lines
Quoted 17 lines
arch/x86/kvm/mmu/mmu.c +3 -2
+}
+
 static void mark_mmio_spte(struct kvm_vcpu *vcpu, u64 *sptep, u64 gfn,
                       unsigned int access)
 {
@@ -2341,7 +2349,7 @@ static void validate_direct_spte(struct kvm_vcpu *vcpu, u64 *sptep,
                    return;
 
            drop_parent_pte(child, sptep);
-           kvm_flush_remote_tlbs_with_address(vcpu->kvm, child->gfn, 1);
+           kvm_flush_remote_tlbs_direct_sp(vcpu->kvm, child);
    }
 }
 
--

2.31.1

#5 lore raw

Re: [PATCH v2 1/6] KVM: x86/mmu: Fix wrong gfn range of tlb flushing in validate_direct_spte()

Quoted 28 lines · Liam Ni
Quoted 27 lines · Hou Wenlong
Quoted 25 lines · David Matlack
Quoted 24 lines · Hou Wenlong

The spte pointing to the children SP is dropped, so the whole gfn range covered by the children SP should be flushed. Although, Hyper-V may treat a 1-page flush the same if the address points to a huge page, it still would be better to use the correct size of huge page. Also introduce a helper function to do range-based flushing when a direct SP is dropped, which would help prevent future buggy use of kvm_flush_remote_tlbs_with_address() in such case.

Fixes c3134ce240eed ("KVM: Replace old tlb flush function with new one to flush a specified range.")
---
 arch/x86/kvm/mmu/mmu.c | 10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)
arch/x86/kvm/mmu/mmu.c +0 -0
index e418ef3ecfcb..a3578abd8bbc 100644
@@ -260,6 +260,14 @@ void kvm_flush_remote_tlbs_with_address(struct kvm *kvm,
    kvm_flush_remote_tlbs_with_range(kvm, &range);
 }
 
Quoted 10 lines
Quoted 6 lines
Quoted 6 lines

arch/x86/kvm/mmu/mmu.c void kvm_flush_remote_tlbs_with_address(struct kvm *kvm, · line 268

Quoted 6 lines
arch/x86/kvm/mmu/mmu.c +6 -0
+/* Flush all memory mapped by the given direct SP. */
+static void kvm_flush_remote_tlbs_direct_sp(struct kvm *kvm, struct kvm_mmu_page *sp)
+{
+   WARN_ON_ONCE(!sp->role.direct);
+   kvm_flush_remote_tlbs_with_address(kvm, sp->gfn,
+                                      KVM_PAGES_PER_HPAGE(sp->role.level + 1));

Do we need "+1" here? sp->role.level=1 means 4k page.
I think here should be “KVM_PAGES_PER_HPAGE(sp->role.level)”

This helper function is used to flush all memory mapped by the given SP, not one spte in the SP. If sp->role.level=1, the size of memory mapped by the SP is 2M, but KVM_PAGES_PER_HPAGE(sp->role.level) is 1.

Quoted 24 lines
Quoted 24 lines
Quoted 5 lines

nit: I think it would make sense to introduce kvm_flush_remote_tlbs_gfn() in this patch since you are going to eventually use it here anyway.

OK, I'll do it in the next version. Thanks!

Quoted 17 lines
Quoted 17 lines
arch/x86/kvm/mmu/mmu.c +3 -2
+}
+
 static void mark_mmio_spte(struct kvm_vcpu *vcpu, u64 *sptep, u64 gfn,
                       unsigned int access)
 {
@@ -2341,7 +2349,7 @@ static void validate_direct_spte(struct kvm_vcpu *vcpu, u64 *sptep,
                    return;
 
            drop_parent_pte(child, sptep);
-           kvm_flush_remote_tlbs_with_address(vcpu->kvm, child->gfn, 1);
+           kvm_flush_remote_tlbs_direct_sp(vcpu->kvm, child);
    }
 }
 
--

2.31.1

#6 lore raw

Re: [PATCH v2 1/6] KVM: x86/mmu: Fix wrong gfn range of tlb flushing in validate_direct_spte()

In reply to Liam Ni
Quoted 29 lines · Liam Ni
Quoted 27 lines · Hou Wenlong
Quoted 25 lines · David Matlack
Quoted 24 lines · Hou Wenlong

The spte pointing to the children SP is dropped, so the whole gfn range covered by the children SP should be flushed. Although, Hyper-V may treat a 1-page flush the same if the address points to a huge page, it still would be better to use the correct size of huge page. Also introduce a helper function to do range-based flushing when a direct SP is dropped, which would help prevent future buggy use of kvm_flush_remote_tlbs_with_address() in such case.

Fixes c3134ce240eed ("KVM: Replace old tlb flush function with new one to flush a specified range.")
---
 arch/x86/kvm/mmu/mmu.c | 10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)
arch/x86/kvm/mmu/mmu.c +0 -0
index e418ef3ecfcb..a3578abd8bbc 100644
@@ -260,6 +260,14 @@ void kvm_flush_remote_tlbs_with_address(struct kvm *kvm,
    kvm_flush_remote_tlbs_with_range(kvm, &range);
 }
 
Quoted 9 lines
Quoted 6 lines
Quoted 6 lines

arch/x86/kvm/mmu/mmu.c void kvm_flush_remote_tlbs_with_address(struct kvm *kvm, · line 268

Quoted 6 lines
arch/x86/kvm/mmu/mmu.c +6 -0
+/* Flush all memory mapped by the given direct SP. */
+static void kvm_flush_remote_tlbs_direct_sp(struct kvm *kvm, struct kvm_mmu_page *sp)
+{
+   WARN_ON_ONCE(!sp->role.direct);
+   kvm_flush_remote_tlbs_with_address(kvm, sp->gfn,
+                                      KVM_PAGES_PER_HPAGE(sp->role.level + 1));

Do we need "+1" here? sp->role.level=1 means 4k page.
I think here should be “KVM_PAGES_PER_HPAGE(sp->role.level)”

Yes we need the "+ 1" here. kvm_flush_remote_tlbs_direct_sp() must flush all memory mapped by the shadow page, which is equivalent to the amount of memory mapped by a huge page at the next higher level. For example, a shadow page with role.level == PG_LEVEL_4K maps 2 MiB of the guest physical address space since 512 PTEs x 4KiB per PTE = 2MiB.

Quoted 25 lines
Quoted 25 lines
Quoted 6 lines

nit: I think it would make sense to introduce kvm_flush_remote_tlbs_gfn() in this patch since you are going to eventually use it here anyway.

OK, I'll do it in the next version. Thanks!

Quoted 17 lines
Quoted 17 lines
arch/x86/kvm/mmu/mmu.c +3 -2
+}
+
 static void mark_mmio_spte(struct kvm_vcpu *vcpu, u64 *sptep, u64 gfn,
                       unsigned int access)
 {
@@ -2341,7 +2349,7 @@ static void validate_direct_spte(struct kvm_vcpu *vcpu, u64 *sptep,
                    return;
 
            drop_parent_pte(child, sptep);
-           kvm_flush_remote_tlbs_with_address(vcpu->kvm, child->gfn, 1);
+           kvm_flush_remote_tlbs_direct_sp(vcpu->kvm, child);
    }
 }
 
--

2.31.1

#7 lore raw

Re: [PATCH v2 1/6] KVM: x86/mmu: Fix wrong gfn range of tlb flushing in validate_direct_spte()

In reply to Hou Wenlong
Quoted 22 lines · Hou Wenlong

The spte pointing to the children SP is dropped, so the whole gfn range covered by the children SP should be flushed. Although, Hyper-V may treat a 1-page flush the same if the address points to a huge page, it still would be better to use the correct size of huge page. Also introduce a helper function to do range-based flushing when a direct SP is dropped, which would help prevent future buggy use of kvm_flush_remote_tlbs_with_address() in such case.

Fixes c3134ce240eed ("KVM: Replace old tlb flush function with new

one to flush a specified range.")

---
 arch/x86/kvm/mmu/mmu.c | 10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)
arch/x86/kvm/mmu/mmu.c +0 -0
index e418ef3ecfcb..a3578abd8bbc 100644

arch/x86/kvm/mmu/mmu.c void kvm_flush_remote_tlbs_with_address(struct · line 268

Quoted 9 lines
arch/x86/kvm/mmu/mmu.c +4 -0
@@ -260,6 +260,14 @@ void kvm_flush_remote_tlbs_with_address(struct
 kvm *kvm,
 	kvm_flush_remote_tlbs_with_range(kvm, &range);
 }
 
+/* Flush all memory mapped by the given direct SP. */
+static void kvm_flush_remote_tlbs_direct_sp(struct kvm *kvm, struct
 kvm_mmu_page *sp)
+{
+	WARN_ON_ONCE(!sp->role.direct);

What if !sp->role.direct? Below flushing sp->gfn isn't expected? but still to do it. Is this operation harmless?

Quoted 20 lines
arch/x86/kvm/mmu/mmu.c +4 -0
+	kvm_flush_remote_tlbs_with_address(kvm, sp->gfn,
+					   KVM_PAGES_PER_HPAGE(sp-
 >role.level + 1));
+}
+

static void mark_mmio_spte(struct kvm_vcpu *vcpu, u64 *sptep, u64
gfn,

arch/x86/kvm/mmu/mmu.c +1 -1
 			   unsigned int access)
 {
@@ -2341,7 +2349,7 @@ static void validate_direct_spte(struct
 kvm_vcpu *vcpu, u64 *sptep,
 			return;
 
 		drop_parent_pte(child, sptep);
-		kvm_flush_remote_tlbs_with_address(vcpu->kvm, child-
 >gfn, 1);
+		kvm_flush_remote_tlbs_direct_sp(vcpu->kvm, child);
 	}

}

#8 lore raw

Re: [PATCH v2 1/6] KVM: x86/mmu: Fix wrong gfn range of tlb flushing in validate_direct_spte()

Quoted 27 lines · Robert Hoo
Quoted 26 lines · Hou Wenlong

The spte pointing to the children SP is dropped, so the whole gfn range covered by the children SP should be flushed. Although, Hyper-V may treat a 1-page flush the same if the address points to a huge page, it still would be better to use the correct size of huge page. Also introduce a helper function to do range-based flushing when a direct SP is dropped, which would help prevent future buggy use of kvm_flush_remote_tlbs_with_address() in such case.

Fixes c3134ce240eed ("KVM: Replace old tlb flush function with new

one to flush a specified range.")

---
 arch/x86/kvm/mmu/mmu.c | 10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)
arch/x86/kvm/mmu/mmu.c +0 -0
index e418ef3ecfcb..a3578abd8bbc 100644
@@ -260,6 +260,14 @@ void kvm_flush_remote_tlbs_with_address(struct
 kvm *kvm,
 	kvm_flush_remote_tlbs_with_range(kvm, &range);
 }
 
Quoted 8 lines

arch/x86/kvm/mmu/mmu.c void kvm_flush_remote_tlbs_with_address(struct kvm *kvm, · line 266

Quoted 5 lines
arch/x86/kvm/mmu/mmu.c +4 -0
+/* Flush all memory mapped by the given direct SP. */
+static void kvm_flush_remote_tlbs_direct_sp(struct kvm *kvm, struct
 kvm_mmu_page *sp)
+{
+	WARN_ON_ONCE(!sp->role.direct);

What if !sp->role.direct? Below flushing sp->gfn isn't expected? but still to do it. Is this operation harmless?

Flushing TLBs is always harmless because KVM cannot ever assume an entry is in the TLB. However, *not* (properly) flushing TLBs can be harmful. If KVM ever calls kvm_flush_remote_tlbs_direct_sp() with an indirect SP, that is a bug in KVM. The TLB flush here won't be harmful, as I explained, but KVM will miss a TLB flush.

That being said, I don't think any changes here are necessary. kvm_flush_remote_tlbs_direct_sp() only has one caller, validate_direct_spte(), which only operates on direct SPs. The name of the function also makes it obvious this should only be called with a direct SP. And if we ever mess this up in the future, we'll see the WARN_ON().

#9 lore raw

Re: [PATCH v2 1/6] KVM: x86/mmu: Fix wrong gfn range of tlb flushing in validate_direct_spte()

Quoted 44 lines · David Matlack
Quoted 27 lines · Robert Hoo
Quoted 26 lines · Hou Wenlong

The spte pointing to the children SP is dropped, so the whole gfn range covered by the children SP should be flushed. Although, Hyper-V may treat a 1-page flush the same if the address points to a huge page, it still would be better to use the correct size of huge page. Also introduce a helper function to do range-based flushing when a direct SP is dropped, which would help prevent future buggy use of kvm_flush_remote_tlbs_with_address() in such case.

Fixes c3134ce240eed ("KVM: Replace old tlb flush function with new

one to flush a specified range.")

---
 arch/x86/kvm/mmu/mmu.c | 10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)
arch/x86/kvm/mmu/mmu.c +0 -0
index e418ef3ecfcb..a3578abd8bbc 100644
@@ -260,6 +260,14 @@ void kvm_flush_remote_tlbs_with_address(struct
 kvm *kvm,
    kvm_flush_remote_tlbs_with_range(kvm, &range);
 }
 
Quoted 8 lines

arch/x86/kvm/mmu/mmu.c void kvm_flush_remote_tlbs_with_address(struct kvm *kvm, · line 266

Quoted 5 lines
arch/x86/kvm/mmu/mmu.c +4 -0
+/* Flush all memory mapped by the given direct SP. */
+static void kvm_flush_remote_tlbs_direct_sp(struct kvm *kvm, struct
 kvm_mmu_page *sp)
+{
+   WARN_ON_ONCE(!sp->role.direct);

What if !sp->role.direct? Below flushing sp->gfn isn't expected? but still to do it. Is this operation harmless?

Flushing TLBs is always harmless because KVM cannot ever assume an entry is in the TLB. However, *not* (properly) flushing TLBs can be harmful. If KVM ever calls kvm_flush_remote_tlbs_direct_sp() with an indirect SP, that is a bug in KVM. The TLB flush here won't be harmful, as I explained, but KVM will miss a TLB flush.

Quoted 5 lines

That being said, I don't think any changes here are necessary. kvm_flush_remote_tlbs_direct_sp() only has one caller, validate_direct_spte(), which only operates on direct SPs. The name of the function also makes it obvious this should only be called with a direct SP. And if we ever mess this up in the future, we'll see the WARN_ON().

That being said, we might as well replace the WARN_ON_ONCE() with KVM_BUG_ON(). That will still do a WARN_ON_ONCE() but has the added benefit of terminating the VM.

#10 lore raw

Re: [PATCH v2 1/6] KVM: x86/mmu: Fix wrong gfn range of tlb flushing in validate_direct_spte()

Quoted 44 lines · David Matlack
Quoted 42 lines · David Matlack
Quoted 39 lines · Robert Hoo
Quoted 32 lines · Hou Wenlong

The spte pointing to the children SP is dropped, so the whole gfn range covered by the children SP should be flushed. Although, Hyper-V may treat a 1-page flush the same if the address points to a huge page, it still would be better to use the correct size of huge page. Also introduce a helper function to do range-based flushing when a direct SP is dropped, which would help prevent future buggy use of kvm_flush_remote_tlbs_with_address() in such case.

Fixes c3134ce240eed ("KVM: Replace old tlb flush function with

new
one to flush a specified range.")

---
 arch/x86/kvm/mmu/mmu.c | 10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)
arch/x86/kvm/mmu/mmu.c +0 -0
index e418ef3ecfcb..a3578abd8bbc 100644
@@ -260,6 +260,14 @@ void

kvm_flush_remote_tlbs_with_address(struct
kvm *kvm,

    kvm_flush_remote_tlbs_with_range(kvm, &range);

}

arch/x86/kvm/mmu/mmu.c +2 -0
+/* Flush all memory mapped by the given direct SP. */
+static void kvm_flush_remote_tlbs_direct_sp(struct kvm *kvm,

struct
kvm_mmu_page *sp)

arch/x86/kvm/mmu/mmu.c void kvm_flush_remote_tlbs_with_address(struct kvm *kvm, · line 266

Quoted 2 lines
arch/x86/kvm/mmu/mmu.c +2 -0
+{
+   WARN_ON_ONCE(!sp->role.direct);

What if !sp->role.direct? Below flushing sp->gfn isn't expected? but
still to do it. Is this operation harmless?

Quoted 10 lines
Quoted 10 lines

Flushing TLBs is always harmless because KVM cannot ever assume an entry is
in the TLB. However, *not* (properly) flushing TLBs can be harmful. If KVM ever
calls kvm_flush_remote_tlbs_direct_sp() with an indirect SP, that is a bug in
KVM. The TLB flush here won't be harmful, as I explained, but KVM will miss a
TLB flush.

Yes, agree, not harmful, a cost of TLB miss, thanks.

Quoted 9 lines
Quoted 8 lines

That being said, I don't think any changes here are necessary. kvm_flush_remote_tlbs_direct_sp() only has one caller, validate_direct_spte(),
which only operates on direct SPs. The name of the function also makes it
obvious this should only be called with a direct SP. And if we ever mess this
up in the future, we'll see the WARN_ON().

Quoted 3 lines

That being said, we might as well replace the WARN_ON_ONCE() with KVM_BUG_ON(). That will still do a WARN_ON_ONCE() but has the added benefit of terminating the VM.

Yeah, here was my point, WARN_ON_ONCE() might not be warning obviously enough, as it usually for recoverable cases. But terminating VM is also over action I think. Just my 2 cents, the whole patch is good.

#11 lore raw

Re: [PATCH v2 1/6] KVM: x86/mmu: Fix wrong gfn range of tlb flushing in validate_direct_spte()

Quoted 8 lines · Robert Hoo
Quoted 3 lines · David Matlack

That being said, we might as well replace the WARN_ON_ONCE() with KVM_BUG_ON(). That will still do a WARN_ON_ONCE() but has the added benefit of terminating the VM.

Yeah, here was my point, WARN_ON_ONCE() might not be warning obviously enough, as it usually for recoverable cases. But terminating VM is also over action I think.

Agreed, from the helper's perspective, killing the VM is unnecessary, e.g. it can simply flush the entire TLB as a fallback.

	if (WARN_ON_ONCE(!sp->role.direct)) {
		kvm_flush_remote_tlbs(kvm);
		return;
	}

But looking at the series as a whole, I think the better option is to just not introduce this helper. There's exactly one user, and if that path encounters an indirect shadow page then KVM is deep in the weeds. But that's a moot point, because unless I'm misreading the flow, there's no need for the unique helper. kvm_flush_remote_tlbs_sptep() will do the right thing even if the target SP happens to be indirect.

If we really want to assert that the child is a direct shadow page, then validate_direct_spte() would be the right location for such an assert. IMO that's unnecessary paranoia. The odds of KVM reaching this point with a completely messed up shadow paging tree, and without already having hosed the guest and/or yelled loudly are very, very low.

In other words, IMO we're getting too fancy for this one and we should instead simply do:

		child = to_shadow_page(*sptep & SPTE_BASE_ADDR_MASK);
		if (child->role.access == direct_access)
			return;

		drop_parent_pte(child, sptep);
		kvm_flush_remote_tlbs_sptep(kvm, sptep);

That has the added benefit of flushing the same "thing" that is zapped, i.e. both operate on the parent SPTE, not the child.

Note, kvm_flush_remote_tlbs_sptep() operates on the _parent_, where the above snippets retrieves the child. This becomes much more obvious once spte_to_child_sp() comes along: https://lore.kernel.org/all/[email protected].

#12 lore raw

PATCH v2 2/6 KVM: x86/mmu: Fix wrong gfn range of tlb flushing in kvm_set_pte_rmapp()

In reply to Hou Wenlong

When the spte of hupe page is dropped in kvm_set_pte_rmapp(), the whole gfn range covered by the spte should be flushed. However, rmap_walk_init_level() doesn't align down the gfn for new level like tdp iterator does, then the gfn used in kvm_set_pte_rmapp() is not the base gfn of huge page. And the size of gfn range is wrong too for huge page. Use the base gfn of huge page and the size of huge page for flushing tlbs for huge page.

Fixes c3134ce240eed ("KVM: Replace old tlb flush function with new one to flush a specified range.")
---
 arch/x86/kvm/mmu/mmu.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
arch/x86/kvm/mmu/mmu.c +2 -1
index a3578abd8bbc..3bcff56df109 100644
@@ -1438,7 +1438,8 @@ static bool kvm_set_pte_rmap(struct kvm *kvm, struct kvm_rmap_head *rmap_head,
 	}
 
 	if (need_flush && kvm_available_flush_tlb_with_range()) {
-		kvm_flush_remote_tlbs_with_address(kvm, gfn, 1);
+		kvm_flush_remote_tlbs_with_address(kvm, gfn & -KVM_PAGES_PER_HPAGE(level),
+						   KVM_PAGES_PER_HPAGE(level));
 		return false;
 	}
 
Signature · 1 line
2.31.1
#13 lore raw

Re: [PATCH v2 2/6] KVM: x86/mmu: Fix wrong gfn range of tlb flushing in kvm_set_pte_rmapp()

Quoted 20 lines · Hou Wenlong

When the spte of hupe page is dropped in kvm_set_pte_rmapp(), the whole gfn range covered by the spte should be flushed. However, rmap_walk_init_level() doesn't align down the gfn for new level like tdp iterator does, then the gfn used in kvm_set_pte_rmapp() is not the base gfn of huge page. And the size of gfn range is wrong too for huge page. Use the base gfn of huge page and the size of huge page for flushing tlbs for huge page.

Fixes c3134ce240eed ("KVM: Replace old tlb flush function with new one to flush a specified range.")
---
 arch/x86/kvm/mmu/mmu.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
arch/x86/kvm/mmu/mmu.c +0 -0
index a3578abd8bbc..3bcff56df109 100644

arch/x86/kvm/mmu/mmu.c static bool kvm_set_pte_rmap(struct kvm *kvm, struct kvm_rmap_head *rmap_head, · line 1441

Quoted 5 lines
arch/x86/kvm/mmu/mmu.c +1 -1
@@ -1438,7 +1438,8 @@ static bool kvm_set_pte_rmap(struct kvm *kvm, struct kvm_rmap_head *rmap_head,
 	}
 
 	if (need_flush && kvm_available_flush_tlb_with_range()) {
-		kvm_flush_remote_tlbs_with_address(kvm, gfn, 1);
+		kvm_flush_remote_tlbs_with_address(kvm, gfn & -KVM_PAGES_PER_HPAGE(level),

Rounding down the GFN to a huge page size is a common pattern throughout KVM. Can you introduce a common way of doing this and clean up the other call sites?

arch/x86/kvm/mmu/mmu.c static bool kvm_set_pte_rmap(struct kvm *kvm, struct kvm_rmap_head *rmap_head, · line 1442

Quoted 1 line
arch/x86/kvm/mmu/mmu.c +1 -0
+						   KVM_PAGES_PER_HPAGE(level));

This eventually gets converted to kvm_flush_remote_tlbs_gfn() in a later patch; which is even more reason to introduce kvm_flush_remote_tlbs_gfn() in the previous patch.

Quoted 6 lines
arch/x86/kvm/mmu/mmu.c +0 -0
 		return false;
 	}
 
Signature · 1 line
2.31.1
#14 lore raw

PATCH v2 3/6 KVM: x86/mmu: Reduce gfn range of tlb flushing in tdp_mmu_map_handle_target_level()

In reply to Hou Wenlong

Since the children SP is zapped, the gfn range of tlb flushing should be the range covered by children SP not parent SP. Replace sp->gfn which is the base gfn of parent SP with iter->gfn and use the correct size of gfn range for children SP to reduce tlb flushing range.

---
 arch/x86/kvm/mmu/tdp_mmu.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
arch/x86/kvm/mmu/tdp_mmu.c +2 -2
index bf2ccf9debca..08b7932122ec 100644
@@ -1071,8 +1071,8 @@ static int tdp_mmu_map_handle_target_level(struct kvm_vcpu *vcpu,
 		return RET_PF_RETRY;
 	else if (is_shadow_present_pte(iter->old_spte) &&
 		 !is_last_spte(iter->old_spte, iter->level))
-		kvm_flush_remote_tlbs_with_address(vcpu->kvm, sp->gfn,
-						   KVM_PAGES_PER_HPAGE(iter->level + 1));
+		kvm_flush_remote_tlbs_with_address(vcpu->kvm, iter->gfn,
+						   KVM_PAGES_PER_HPAGE(iter->level));
 
 	/*
 	 * If the page fault was caused by a write but the page is write
Signature · 1 line
2.31.1
Reviewed-by on v2 3/6
#15 lore raw

Re: [PATCH v2 3/6] KVM: x86/mmu: Reduce gfn range of tlb flushing in tdp_mmu_map_handle_target_level()

Quoted 5 lines · Hou Wenlong

Since the children SP is zapped, the gfn range of tlb flushing should be the range covered by children SP not parent SP. Replace sp->gfn which is the base gfn of parent SP with iter->gfn and use the correct size of gfn range for children SP to reduce tlb flushing range.

Fixes bb95dfb9e2df ("KVM: x86/mmu: Defer TLB flush to caller when freeing TDP MMU shadow pages")
Quoted 1 line
Quoted 22 lines
---
 arch/x86/kvm/mmu/tdp_mmu.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
arch/x86/kvm/mmu/tdp_mmu.c +2 -2
index bf2ccf9debca..08b7932122ec 100644
@@ -1071,8 +1071,8 @@ static int tdp_mmu_map_handle_target_level(struct kvm_vcpu *vcpu,
 		return RET_PF_RETRY;
 	else if (is_shadow_present_pte(iter->old_spte) &&
 		 !is_last_spte(iter->old_spte, iter->level))
-		kvm_flush_remote_tlbs_with_address(vcpu->kvm, sp->gfn,
-						   KVM_PAGES_PER_HPAGE(iter->level + 1));
+		kvm_flush_remote_tlbs_with_address(vcpu->kvm, iter->gfn,
+						   KVM_PAGES_PER_HPAGE(iter->level));
 
 	/*
 	 * If the page fault was caused by a write but the page is write
Signature · 1 line
2.31.1
#16 lore raw

PATCH v2 4/6 KVM: x86/mmu: Fix wrong start gfn of tlb flushing with range

In reply to Hou Wenlong

When a spte is dropped, the start gfn of tlb flushing should be the gfn of spte not the base gfn of SP which contains the spte. Also introduce a helper function to do range-based flushing when a spte is dropped, which would help prevent future buggy use of kvm_flush_remote_tlbs_with_address() in such case.

Fixes c3134ce240eed ("KVM: Replace old tlb flush function with new one to flush a specified range.")
---
 arch/x86/kvm/mmu/mmu.c         | 20 +++++++++++++++-----
 arch/x86/kvm/mmu/paging_tmpl.h |  3 +--
 2 files changed, 16 insertions(+), 7 deletions(-)
arch/x86/kvm/mmu/mmu.c +15 -5
index 3bcff56df109..e0b9432b9491 100644
@@ -260,6 +260,18 @@ void kvm_flush_remote_tlbs_with_address(struct kvm *kvm,
 	kvm_flush_remote_tlbs_with_range(kvm, &range);
 }
 
+static gfn_t kvm_mmu_page_get_gfn(struct kvm_mmu_page *sp, int index);
+
+/* Flush the range of guest memory mapped by the given SPTE. */
+static void kvm_flush_remote_tlbs_sptep(struct kvm *kvm, u64 *sptep)
+{
+	struct kvm_mmu_page *sp = sptep_to_sp(sptep);
+	gfn_t gfn = kvm_mmu_page_get_gfn(sp, spte_index(sptep));
+
+	kvm_flush_remote_tlbs_with_address(kvm, gfn,
+					   KVM_PAGES_PER_HPAGE(sp->role.level));
+}
+
 /* Flush all memory mapped by the given direct SP. */
 static void kvm_flush_remote_tlbs_direct_sp(struct kvm *kvm, struct kvm_mmu_page *sp)
 {
@@ -1156,8 +1168,7 @@ static void drop_large_spte(struct kvm *kvm, u64 *sptep, bool flush)
 	drop_spte(kvm, sptep);
 
 	if (flush)
-		kvm_flush_remote_tlbs_with_address(kvm, sp->gfn,
-			KVM_PAGES_PER_HPAGE(sp->role.level));
+		kvm_flush_remote_tlbs_sptep(kvm, sptep);
 }
 
 /*
@@ -1608,7 +1619,7 @@ static void __rmap_add(struct kvm *kvm,
 	if (rmap_count > RMAP_RECYCLE_THRESHOLD) {
 		kvm_zap_all_rmap_sptes(kvm, rmap_head);
 		kvm_flush_remote_tlbs_with_address(
-				kvm, sp->gfn, KVM_PAGES_PER_HPAGE(sp->role.level));
+				kvm, gfn, KVM_PAGES_PER_HPAGE(sp->role.level));
 	}
 }
 
@@ -6402,8 +6413,7 @@ static bool kvm_mmu_zap_collapsible_spte(struct kvm *kvm,
 			kvm_zap_one_rmap_spte(kvm, rmap_head, sptep);
 
 			if (kvm_available_flush_tlb_with_range())
-				kvm_flush_remote_tlbs_with_address(kvm, sp->gfn,
-					KVM_PAGES_PER_HPAGE(sp->role.level));
+				kvm_flush_remote_tlbs_sptep(kvm, sptep);
 			else
 				need_tlb_flush = 1;
 
arch/x86/kvm/mmu/paging_tmpl.h +1 -2
index 39e0205e7300..04149c704d5b 100644
@@ -937,8 +937,7 @@ static void FNAME(invlpg)(struct kvm_vcpu *vcpu, gva_t gva, hpa_t root_hpa)
 
 			mmu_page_zap_pte(vcpu->kvm, sp, sptep, NULL);
 			if (is_shadow_present_pte(old_spte))
-				kvm_flush_remote_tlbs_with_address(vcpu->kvm,
-					sp->gfn, KVM_PAGES_PER_HPAGE(sp->role.level));
+				kvm_flush_remote_tlbs_sptep(vcpu->kvm, sptep);
 
 			if (!rmap_can_add(vcpu))
 				break;
Signature · 1 line
2.31.1
#17 lore raw

Re: [PATCH v2 4/6] KVM: x86/mmu: Fix wrong start gfn of tlb flushing with range

Quoted 23 lines · Hou Wenlong

When a spte is dropped, the start gfn of tlb flushing should be the gfn of spte not the base gfn of SP which contains the spte. Also introduce a helper function to do range-based flushing when a spte is dropped, which would help prevent future buggy use of kvm_flush_remote_tlbs_with_address() in such case.

Fixes c3134ce240eed ("KVM: Replace old tlb flush function with new one to flush a specified range.")
---
 arch/x86/kvm/mmu/mmu.c         | 20 +++++++++++++++-----
 arch/x86/kvm/mmu/paging_tmpl.h |  3 +--
 2 files changed, 16 insertions(+), 7 deletions(-)
arch/x86/kvm/mmu/mmu.c +0 -0
index 3bcff56df109..e0b9432b9491 100644
@@ -260,6 +260,18 @@ void kvm_flush_remote_tlbs_with_address(struct kvm *kvm,
 	kvm_flush_remote_tlbs_with_range(kvm, &range);
 }
 

arch/x86/kvm/mmu/mmu.c void kvm_flush_remote_tlbs_with_address(struct kvm *kvm, · line 272

Quoted 10 lines
arch/x86/kvm/mmu/mmu.c +10 -0
+static gfn_t kvm_mmu_page_get_gfn(struct kvm_mmu_page *sp, int index);
+
+/* Flush the range of guest memory mapped by the given SPTE. */
+static void kvm_flush_remote_tlbs_sptep(struct kvm *kvm, u64 *sptep)
+{
+	struct kvm_mmu_page *sp = sptep_to_sp(sptep);
+	gfn_t gfn = kvm_mmu_page_get_gfn(sp, spte_index(sptep));
+
+	kvm_flush_remote_tlbs_with_address(kvm, gfn,
+					   KVM_PAGES_PER_HPAGE(sp->role.level));

How is the range-based TLB flushing supposed to work with indirect MMUs? When KVM is using shadow paging, the gfn here is not part of the actual translation.

For example, when TDP is disabled, KVM's shadow page tables translate GVA to HPA. When Nested Virtualization is in use and running L2, KVM's shadow page tables translate nGPA to HPA.

Ah, I see x86_ops.tlb_remote_flush_with_range is only set when running on Hyper-V and TDP is enabled (VMX checks enable_ept and SVM checks npt_enabled). But it looks like the nested case might still be broken?

Quoted 51 lines
arch/x86/kvm/mmu/mmu.c +5 -5
+}
+
 /* Flush all memory mapped by the given direct SP. */
 static void kvm_flush_remote_tlbs_direct_sp(struct kvm *kvm, struct kvm_mmu_page *sp)
 {
@@ -1156,8 +1168,7 @@ static void drop_large_spte(struct kvm *kvm, u64 *sptep, bool flush)
 	drop_spte(kvm, sptep);
 
 	if (flush)
-		kvm_flush_remote_tlbs_with_address(kvm, sp->gfn,
-			KVM_PAGES_PER_HPAGE(sp->role.level));
+		kvm_flush_remote_tlbs_sptep(kvm, sptep);
 }
 
 /*
@@ -1608,7 +1619,7 @@ static void __rmap_add(struct kvm *kvm,
 	if (rmap_count > RMAP_RECYCLE_THRESHOLD) {
 		kvm_zap_all_rmap_sptes(kvm, rmap_head);
 		kvm_flush_remote_tlbs_with_address(
-				kvm, sp->gfn, KVM_PAGES_PER_HPAGE(sp->role.level));
+				kvm, gfn, KVM_PAGES_PER_HPAGE(sp->role.level));
 	}
 }
 
@@ -6402,8 +6413,7 @@ static bool kvm_mmu_zap_collapsible_spte(struct kvm *kvm,
 			kvm_zap_one_rmap_spte(kvm, rmap_head, sptep);
 
 			if (kvm_available_flush_tlb_with_range())
-				kvm_flush_remote_tlbs_with_address(kvm, sp->gfn,
-					KVM_PAGES_PER_HPAGE(sp->role.level));
+				kvm_flush_remote_tlbs_sptep(kvm, sptep);
 			else
 				need_tlb_flush = 1;
 
arch/x86/kvm/mmu/paging_tmpl.h +1 -2
index 39e0205e7300..04149c704d5b 100644
@@ -937,8 +937,7 @@ static void FNAME(invlpg)(struct kvm_vcpu *vcpu, gva_t gva, hpa_t root_hpa)
 
 			mmu_page_zap_pte(vcpu->kvm, sp, sptep, NULL);
 			if (is_shadow_present_pte(old_spte))
-				kvm_flush_remote_tlbs_with_address(vcpu->kvm,
-					sp->gfn, KVM_PAGES_PER_HPAGE(sp->role.level));
+				kvm_flush_remote_tlbs_sptep(vcpu->kvm, sptep);
 
 			if (!rmap_can_add(vcpu))
 				break;
Signature · 1 line
2.31.1
#18 lore raw

Re: [PATCH v2 4/6] KVM: x86/mmu: Fix wrong start gfn of tlb flushing with range

Quoted 39 lines · David Matlack
Quoted 23 lines · Hou Wenlong

When a spte is dropped, the start gfn of tlb flushing should be the gfn of spte not the base gfn of SP which contains the spte. Also introduce a helper function to do range-based flushing when a spte is dropped, which would help prevent future buggy use of kvm_flush_remote_tlbs_with_address() in such case.

Fixes c3134ce240eed ("KVM: Replace old tlb flush function with new one to flush a specified range.")
---
 arch/x86/kvm/mmu/mmu.c         | 20 +++++++++++++++-----
 arch/x86/kvm/mmu/paging_tmpl.h |  3 +--
 2 files changed, 16 insertions(+), 7 deletions(-)
arch/x86/kvm/mmu/mmu.c +0 -0
index 3bcff56df109..e0b9432b9491 100644
@@ -260,6 +260,18 @@ void kvm_flush_remote_tlbs_with_address(struct kvm *kvm,
 	kvm_flush_remote_tlbs_with_range(kvm, &range);
 }
 

arch/x86/kvm/mmu/mmu.c void kvm_flush_remote_tlbs_with_address(struct kvm *kvm, · line 272

Quoted 10 lines
arch/x86/kvm/mmu/mmu.c +10 -0
+static gfn_t kvm_mmu_page_get_gfn(struct kvm_mmu_page *sp, int index);
+
+/* Flush the range of guest memory mapped by the given SPTE. */
+static void kvm_flush_remote_tlbs_sptep(struct kvm *kvm, u64 *sptep)
+{
+	struct kvm_mmu_page *sp = sptep_to_sp(sptep);
+	gfn_t gfn = kvm_mmu_page_get_gfn(sp, spte_index(sptep));
+
+	kvm_flush_remote_tlbs_with_address(kvm, gfn,
+					   KVM_PAGES_PER_HPAGE(sp->role.level));

How is the range-based TLB flushing supposed to work with indirect MMUs? When KVM is using shadow paging, the gfn here is not part of the actual translation.

Quoted 8 lines

For example, when TDP is disabled, KVM's shadow page tables translate GVA to HPA. When Nested Virtualization is in use and running L2, KVM's shadow page tables translate nGPA to HPA.

Ah, I see x86_ops.tlb_remote_flush_with_range is only set when running on Hyper-V and TDP is enabled (VMX checks enable_ept and SVM checks npt_enabled). But it looks like the nested case might still be broken?

Yeah, range based tlb flushing is only used on Hyper-V as Paolo said. Actually, I don't know how Hyper-V implements the range based tlb flushing hypercall, since KVM on Hyper-V is already nested, so gfn here is already nGPA. It seems that the current used TDP root is passed in hypercall, so maybe Hyper-V could do nGPA to HPA translation by looking up TDP page table. Then for nested case, it chould work well?

Quoted 51 lines
Quoted 51 lines
arch/x86/kvm/mmu/mmu.c +5 -5
+}
+
 /* Flush all memory mapped by the given direct SP. */
 static void kvm_flush_remote_tlbs_direct_sp(struct kvm *kvm, struct kvm_mmu_page *sp)
 {
@@ -1156,8 +1168,7 @@ static void drop_large_spte(struct kvm *kvm, u64 *sptep, bool flush)
 	drop_spte(kvm, sptep);
 
 	if (flush)
-		kvm_flush_remote_tlbs_with_address(kvm, sp->gfn,
-			KVM_PAGES_PER_HPAGE(sp->role.level));
+		kvm_flush_remote_tlbs_sptep(kvm, sptep);
 }
 
 /*
@@ -1608,7 +1619,7 @@ static void __rmap_add(struct kvm *kvm,
 	if (rmap_count > RMAP_RECYCLE_THRESHOLD) {
 		kvm_zap_all_rmap_sptes(kvm, rmap_head);
 		kvm_flush_remote_tlbs_with_address(
-				kvm, sp->gfn, KVM_PAGES_PER_HPAGE(sp->role.level));
+				kvm, gfn, KVM_PAGES_PER_HPAGE(sp->role.level));
 	}
 }
 
@@ -6402,8 +6413,7 @@ static bool kvm_mmu_zap_collapsible_spte(struct kvm *kvm,
 			kvm_zap_one_rmap_spte(kvm, rmap_head, sptep);
 
 			if (kvm_available_flush_tlb_with_range())
-				kvm_flush_remote_tlbs_with_address(kvm, sp->gfn,
-					KVM_PAGES_PER_HPAGE(sp->role.level));
+				kvm_flush_remote_tlbs_sptep(kvm, sptep);
 			else
 				need_tlb_flush = 1;
 
arch/x86/kvm/mmu/paging_tmpl.h +1 -2
index 39e0205e7300..04149c704d5b 100644
@@ -937,8 +937,7 @@ static void FNAME(invlpg)(struct kvm_vcpu *vcpu, gva_t gva, hpa_t root_hpa)
 
 			mmu_page_zap_pte(vcpu->kvm, sp, sptep, NULL);
 			if (is_shadow_present_pte(old_spte))
-				kvm_flush_remote_tlbs_with_address(vcpu->kvm,
-					sp->gfn, KVM_PAGES_PER_HPAGE(sp->role.level));
+				kvm_flush_remote_tlbs_sptep(vcpu->kvm, sptep);
 
 			if (!rmap_can_add(vcpu))
 				break;
Signature · 1 line
2.31.1
#19 lore raw

PATCH v2 5/6 KVM: x86/mmu: Introduce helper function to do range-based flushing for given page

In reply to Hou Wenlong

Flushing tlb for one page (huge or not) is the main use case, so introduce a helper function for this common operation to make the code clear.

---
 arch/x86/kvm/mmu/mmu.c          | 16 ++++++----------
 arch/x86/kvm/mmu/mmu_internal.h | 10 ++++++++++
 arch/x86/kvm/mmu/tdp_mmu.c      |  6 ++----
 3 files changed, 18 insertions(+), 14 deletions(-)
arch/x86/kvm/mmu/mmu.c +6 -10
index e0b9432b9491..92ca76e11d96 100644
@@ -268,16 +268,14 @@ static void kvm_flush_remote_tlbs_sptep(struct kvm *kvm, u64 *sptep)
 	struct kvm_mmu_page *sp = sptep_to_sp(sptep);
 	gfn_t gfn = kvm_mmu_page_get_gfn(sp, spte_index(sptep));
 
-	kvm_flush_remote_tlbs_with_address(kvm, gfn,
-					   KVM_PAGES_PER_HPAGE(sp->role.level));
+	kvm_flush_remote_tlbs_gfn(kvm, gfn, sp->role.level);
 }
 
 /* Flush all memory mapped by the given direct SP. */
 static void kvm_flush_remote_tlbs_direct_sp(struct kvm *kvm, struct kvm_mmu_page *sp)
 {
 	WARN_ON_ONCE(!sp->role.direct);
-	kvm_flush_remote_tlbs_with_address(kvm, sp->gfn,
-					   KVM_PAGES_PER_HPAGE(sp->role.level + 1));
+	kvm_flush_remote_tlbs_gfn(kvm, sp->gfn, sp->role.level + 1);
 }
 
 static void mark_mmio_spte(struct kvm_vcpu *vcpu, u64 *sptep, u64 gfn,
@@ -1449,8 +1447,8 @@ static bool kvm_set_pte_rmap(struct kvm *kvm, struct kvm_rmap_head *rmap_head,
 	}
 
 	if (need_flush && kvm_available_flush_tlb_with_range()) {
-		kvm_flush_remote_tlbs_with_address(kvm, gfn & -KVM_PAGES_PER_HPAGE(level),
-						   KVM_PAGES_PER_HPAGE(level));
+		kvm_flush_remote_tlbs_gfn(kvm, gfn & -KVM_PAGES_PER_HPAGE(level),
+					  level);
 		return false;
 	}
 
@@ -1618,8 +1616,7 @@ static void __rmap_add(struct kvm *kvm,
 
 	if (rmap_count > RMAP_RECYCLE_THRESHOLD) {
 		kvm_zap_all_rmap_sptes(kvm, rmap_head);
-		kvm_flush_remote_tlbs_with_address(
-				kvm, gfn, KVM_PAGES_PER_HPAGE(sp->role.level));
+		kvm_flush_remote_tlbs_gfn(kvm, gfn, sp->role.level);
 	}
 }
 
@@ -2844,8 +2841,7 @@ static int mmu_set_spte(struct kvm_vcpu *vcpu, struct kvm_memory_slot *slot,
 	}
 
 	if (flush)
-		kvm_flush_remote_tlbs_with_address(vcpu->kvm, gfn,
-				KVM_PAGES_PER_HPAGE(level));
+		kvm_flush_remote_tlbs_gfn(vcpu->kvm, gfn, level);
 
 	pgprintk("%s: setting spte %llx\n", __func__, *sptep);
 
arch/x86/kvm/mmu/mmu_internal.h +10 -0
index 582def531d4d..6651c154f2e0 100644
@@ -163,8 +163,18 @@ void kvm_mmu_gfn_allow_lpage(const struct kvm_memory_slot *slot, gfn_t gfn);
 bool kvm_mmu_slot_gfn_write_protect(struct kvm *kvm,
 				    struct kvm_memory_slot *slot, u64 gfn,
 				    int min_level);
+
 void kvm_flush_remote_tlbs_with_address(struct kvm *kvm,
 					u64 start_gfn, u64 pages);
+
+/* Flush the given page (huge or not) of guest memory. */
+static inline void kvm_flush_remote_tlbs_gfn(struct kvm *kvm, gfn_t gfn, int level)
+{
+	u64 pages = KVM_PAGES_PER_HPAGE(level);
+
+	kvm_flush_remote_tlbs_with_address(kvm, gfn, pages);
+}
+
 unsigned int pte_list_count(struct kvm_rmap_head *rmap_head);
 
 extern int nx_huge_pages;
arch/x86/kvm/mmu/tdp_mmu.c +2 -4
index 08b7932122ec..567691440ab0 100644
@@ -673,8 +673,7 @@ static inline int tdp_mmu_zap_spte_atomic(struct kvm *kvm,
 	if (ret)
 		return ret;
 
-	kvm_flush_remote_tlbs_with_address(kvm, iter->gfn,
-					   KVM_PAGES_PER_HPAGE(iter->level));
+	kvm_flush_remote_tlbs_gfn(kvm, iter->gfn, iter->level);
 
 	/*
 	 * No other thread can overwrite the removed SPTE as they must either
@@ -1071,8 +1070,7 @@ static int tdp_mmu_map_handle_target_level(struct kvm_vcpu *vcpu,
 		return RET_PF_RETRY;
 	else if (is_shadow_present_pte(iter->old_spte) &&
 		 !is_last_spte(iter->old_spte, iter->level))
-		kvm_flush_remote_tlbs_with_address(vcpu->kvm, iter->gfn,
-						   KVM_PAGES_PER_HPAGE(iter->level));
+		kvm_flush_remote_tlbs_gfn(vcpu->kvm, iter->gfn, iter->level);
 
 	/*
 	 * If the page fault was caused by a write but the page is write

--
2.31.1

#20 lore raw

PATCH v2 6/6 KVM: x86/mmu: Use 1 as the size of gfn range for tlb flushing in FNAME(invlpg)()

In reply to Hou Wenlong

Only SP with PG_LEVLE_4K level could be unsync, so the size of gfn range must be 1.

---
 arch/x86/kvm/mmu/paging_tmpl.h | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
arch/x86/kvm/mmu/paging_tmpl.h +2 -1
index 04149c704d5b..486a3163b1e4 100644
@@ -937,7 +937,8 @@ static void FNAME(invlpg)(struct kvm_vcpu *vcpu, gva_t gva, hpa_t root_hpa)
 
 			mmu_page_zap_pte(vcpu->kvm, sp, sptep, NULL);
 			if (is_shadow_present_pte(old_spte))
-				kvm_flush_remote_tlbs_sptep(vcpu->kvm, sptep);
+				kvm_flush_remote_tlbs_gfn(vcpu->kvm,
+					kvm_mmu_page_get_gfn(sp, sptep - sp->spt), 1);
 
 			if (!rmap_can_add(vcpu))
 				break;
Signature · 1 line
2.31.1
#21 lore raw

Re: [PATCH v2 6/6] KVM: x86/mmu: Use 1 as the size of gfn range for tlb flushing in FNAME(invlpg)()

Quoted 13 lines · Hou Wenlong

Only SP with PG_LEVLE_4K level could be unsync, so the size of gfn range must be 1.

---
 arch/x86/kvm/mmu/paging_tmpl.h | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
arch/x86/kvm/mmu/paging_tmpl.h +0 -0
index 04149c704d5b..486a3163b1e4 100644

arch/x86/kvm/mmu/paging_tmpl.h static void FNAME(invlpg)(struct kvm_vcpu *vcpu, gva_t gva, hpa_t root_hpa) · line 941

Quoted 6 lines
arch/x86/kvm/mmu/paging_tmpl.h +2 -1
@@ -937,7 +937,8 @@ static void FNAME(invlpg)(struct kvm_vcpu *vcpu, gva_t gva, hpa_t root_hpa)
 
 			mmu_page_zap_pte(vcpu->kvm, sp, sptep, NULL);
 			if (is_shadow_present_pte(old_spte))
-				kvm_flush_remote_tlbs_sptep(vcpu->kvm, sptep);
+				kvm_flush_remote_tlbs_gfn(vcpu->kvm,
+					kvm_mmu_page_get_gfn(sp, sptep - sp->spt), 1);

The third argument to kvm_flush_remote_tlbs_gfn() is the level, not the number of pages. But that aside, I don't understand why this patch is necessary. kvm_flush_remote_tlbs_sptep() should already do the right thing.

Quoted 6 lines
arch/x86/kvm/mmu/paging_tmpl.h +0 -0
 			if (!rmap_can_add(vcpu))
 				break;
Signature · 1 line
2.31.1
#22 lore raw

Re: [PATCH v2 6/6] KVM: x86/mmu: Use 1 as the size of gfn range for tlb flushing in FNAME(invlpg)()

Quoted 15 lines · David Matlack
Quoted 14 lines · Hou Wenlong

Only SP with PG_LEVLE_4K level could be unsync, so the size of gfn range must be 1.

---
 arch/x86/kvm/mmu/paging_tmpl.h | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
arch/x86/kvm/mmu/paging_tmpl.h +0 -0
index 04149c704d5b..486a3163b1e4 100644
@@ -937,7 +937,8 @@ static void FNAME(invlpg)(struct kvm_vcpu *vcpu, gva_t gva, hpa_t root_hpa)
 
Quoted 11 lines

arch/x86/kvm/mmu/paging_tmpl.h static void FNAME(invlpg)(struct kvm_vcpu *vcpu, gva_t gva, hpa_t root_hpa) · line 941

Quoted 5 lines
arch/x86/kvm/mmu/paging_tmpl.h +2 -1
 			mmu_page_zap_pte(vcpu->kvm, sp, sptep, NULL);
 			if (is_shadow_present_pte(old_spte))
-				kvm_flush_remote_tlbs_sptep(vcpu->kvm, sptep);
+				kvm_flush_remote_tlbs_gfn(vcpu->kvm,
+					kvm_mmu_page_get_gfn(sp, sptep - sp->spt), 1);

The third argument to kvm_flush_remote_tlbs_gfn() is the level, not the number of pages. But that aside, I don't understand why this patch is necessary. kvm_flush_remote_tlbs_sptep() should already do the right thing.

Since only SP with PG_LEVEL_4K level could be unsync, so the level must be PG_LEVEL_4K, then sp->role.level access could be dropped. However, I'm not sure whether it is useful. I can drop it if it is useless.

Quoted 6 lines
Quoted 6 lines
arch/x86/kvm/mmu/paging_tmpl.h +0 -0
 			if (!rmap_can_add(vcpu))
 				break;
Signature · 1 line
2.31.1
#23 lore raw

Re: [PATCH v2 6/6] KVM: x86/mmu: Use 1 as the size of gfn range for tlb flushing in FNAME(invlpg)()

Quoted 28 lines · Hou Wenlong
Quoted 15 lines · David Matlack
Quoted 14 lines · Hou Wenlong

Only SP with PG_LEVLE_4K level could be unsync, so the size of gfn range must be 1.

---
 arch/x86/kvm/mmu/paging_tmpl.h | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
arch/x86/kvm/mmu/paging_tmpl.h +0 -0
index 04149c704d5b..486a3163b1e4 100644
@@ -937,7 +937,8 @@ static void FNAME(invlpg)(struct kvm_vcpu *vcpu, gva_t gva, hpa_t root_hpa)
 
Quoted 11 lines

arch/x86/kvm/mmu/paging_tmpl.h static void FNAME(invlpg)(struct kvm_vcpu *vcpu, gva_t gva, hpa_t root_hpa) · line 941

Quoted 5 lines
arch/x86/kvm/mmu/paging_tmpl.h +2 -1
                    mmu_page_zap_pte(vcpu->kvm, sp, sptep, NULL);
                    if (is_shadow_present_pte(old_spte))
-                           kvm_flush_remote_tlbs_sptep(vcpu->kvm, sptep);
+                           kvm_flush_remote_tlbs_gfn(vcpu->kvm,
+                                   kvm_mmu_page_get_gfn(sp, sptep - sp->spt), 1);

The third argument to kvm_flush_remote_tlbs_gfn() is the level, not the number of pages. But that aside, I don't understand why this patch is necessary. kvm_flush_remote_tlbs_sptep() should already do the right thing.

Quoted 3 lines

Since only SP with PG_LEVEL_4K level could be unsync, so the level must be PG_LEVEL_4K, then sp->role.level access could be dropped. However, I'm not sure whether it is useful. I can drop it if it is useless.

Ah, I see. I would be surprised if avoiding the read of sp->role.level has any noticeable impact on VM performance so I vote to drop this patch.

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
arch/x86/kvm/mmu/paging_tmpl.h +0 -1
                    if (!rmap_can_add(vcpu))
                            break;
--

2.31.1

#24 lore raw

Re: [PATCH v2 6/6] KVM: x86/mmu: Use 1 as the size of gfn range for tlb flushing in FNAME(invlpg)()

Quoted 29 lines · David Matlack
Quoted 28 lines · Hou Wenlong
Quoted 26 lines · David Matlack
Quoted 13 lines · Hou Wenlong

Only SP with PG_LEVLE_4K level could be unsync, so the size of gfn range must be 1.

---
 arch/x86/kvm/mmu/paging_tmpl.h | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
arch/x86/kvm/mmu/paging_tmpl.h +0 -0
index 04149c704d5b..486a3163b1e4 100644

arch/x86/kvm/mmu/paging_tmpl.h static void FNAME(invlpg)(struct kvm_vcpu *vcpu, gva_t gva, hpa_t root_hpa) · line 941

Quoted 6 lines
arch/x86/kvm/mmu/paging_tmpl.h +2 -1
@@ -937,7 +937,8 @@ static void FNAME(invlpg)(struct kvm_vcpu *vcpu, gva_t gva, hpa_t root_hpa)
 
                    mmu_page_zap_pte(vcpu->kvm, sp, sptep, NULL);
                    if (is_shadow_present_pte(old_spte))
-                           kvm_flush_remote_tlbs_sptep(vcpu->kvm, sptep);
+                           kvm_flush_remote_tlbs_gfn(vcpu->kvm,
+                                   kvm_mmu_page_get_gfn(sp, sptep - sp->spt), 1);

The third argument to kvm_flush_remote_tlbs_gfn() is the level, not the number of pages. But that aside, I don't understand why this patch is necessary. kvm_flush_remote_tlbs_sptep() should already do the right thing.

Quoted 6 lines
Quoted 3 lines

Since only SP with PG_LEVEL_4K level could be unsync, so the level must be PG_LEVEL_4K, then sp->role.level access could be dropped. However, I'm not sure whether it is useful. I can drop it if it is useless.

Ah, I see. I would be surprised if avoiding the read of sp->role.level has any noticeable impact on VM performance so I vote to drop this patch.

Agreed, the cost of the sp->role.level lookup is negligible in this case, and IMO using kvm_flush_remote_tlbs_sptep() is more intuitive.

If kvm_flush_remote_tlbs_sptep() didn't exist and this was open coding the use of kvm_flush_remote_tlbs_with_address() + KVM_PAGES_PER_HPAGE(), then I would be in favor of hardcoding '1', because at that point the use of KVM_PAGES_PER_HPAGE() is misleading in its own way.