Skip to contentSkip to conversation

KVM: arm64: Add support for FEAT_TLBIRANGE

PATCH Superseded Raghavendra Rao Ananta 6 participants 35 messages Aug 8, 2023 – Aug 11, 2023

Subsystems (9)
x86 KVM MIPS RISC-V +5 more subsystems

v8 of 9 Superseded Latest is v9 →

lore
Share & tools
lore
#0 lore raw

v8 cover KVM: arm64: Add support for FEAT_TLBIRANGE

In certain code paths, KVM/ARM currently invalidates the entire VM's page-tables instead of just invalidating a necessary range. For example, when collapsing a table PTE to a block PTE, instead of iterating over each PTE and flushing them, KVM uses 'vmalls12e1is' TLBI operation to flush all the entries. This is inefficient since the guest would have to refill the TLBs again, even for the addresses that aren't covered by the table entry. The performance impact would scale poorly if many addresses in the VM is going through this remapping.

For architectures that implement FEAT_TLBIRANGE, KVM can replace such inefficient paths by performing the invalidations only on the range of addresses that are in scope. This series tries to achieve the same in the areas of stage-2 map, unmap and write-protecting the pages.

As suggested by Oliver in the original v5 of the series [1], I'm posting the series by including v2 of David Matlack's 'KVM: Add a common API for range-based TLB invalidation' series [2].

Patches 1-6 includes David M.'s patches 1, 2, 6, and 7 from [2], with minor modifications as per upstream comments.

Patch-7 refactors the core arm64's __flush_tlb_range() to be used by other entities and patch-8 introduces a wrapper over it, __flush_s2_tlb_range_op(), more suited for stage-2 flushes.

Patch-9,10 adds a range-based TLBI mechanism for KVM (VHE and nVHE).

Patch-11 implements the kvm_arch_flush_remote_tlbs_range() for arm64.

Patch-12 aims to flush only the memslot that undergoes a write-protect, instead of the entire VM.

Patch-13 operates on stage2_try_break_pte() to use the range based TLBI instructions when collapsing a table entry. The map path is the immediate consumer of this when KVM remaps a table entry into a block.

Patch-14 modifies the stage-2 unmap path in which, if the system supports FEAT_TLBIRANGE, the TLB invalidations are skipped during the page-table. walk. Instead it's done in one go after the entire walk is finished.

The series is based off of upstream v6.5-rc1.

The performance evaluation was done on a hardware that supports FEAT_TLBIRANGE, on a VHE configuration, using a modified kvm_page_table_test.
The modified version updates the guest code in the ADJUST_MAPPINGS case to not only access this page but also to access up to 512 pages backwards for every new page it iterates through. This is done to test the effect of TLBI misses after KVM has handled a fault.

The series captures the impact in the map and unmap paths as described above.

$ kvm_page_table_test -m 2 -v 128 -s anonymous_hugetlb_2mb -b $i
+--------+------------------------------+------------------------------+
| mem_sz |    ADJUST_MAPPINGS (s)       |      Unmap VM (s)            |
|  (GB)  | Baseline | Baseline + series | Baseline | Baseline + series |
+--------+----------|-------------------+------------------------------+
|   1    |   3.33   |   3.22            | 0.009     | 0.005            |
|   2    |   7.39   |   7.32            | 0.012     | 0.006            |
|   4    |  13.49   |  10.50            | 0.017     | 0.008            |
|   8    |  21.60   |  21.50            | 0.027     | 0.011            |
|  16    |  57.02   |  43.63            | 0.046     | 0.018            |
|  32    |  95.92   |  83.26            | 0.087     | 0.030            |
|  64    | 199.57   | 165.14            | 0.146     | 0.055            |
| 128    | 423.65   | 349.37            | 0.280     | 0.100            |
+--------+----------+-------------------+----------+-------------------+
$ kvm_page_table_test -m 2 -b 128G -s anonymous_hugetlb_2mb -v $i
+--------+------------------------------+
| vCPUs  |    ADJUST_MAPPINGS (s)       |
|        | Baseline | Baseline + series |
+--------+----------|-------------------+
|   1    | 111.44   | 114.63            |
|   2    | 102.88   |  74.64            |
|   4    | 134.83   |  98.78            |
|   8    |  98.81   |  95.01            |
|  16    | 127.41   |  99.05            |
|  32    | 105.35   |  91.75            |
|  64    | 201.13   | 163.63            |
| 128    | 423.65   | 349.37            |   
+--------+----------+-------------------+

For the ADJUST_MAPPINGS cases, which maps back the 4K table entries to 2M hugepages, the series sees an average improvement of ~15%. For unmapping 2M hugepages, we see a gain of 2x to 3x.

$ kvm_page_table_test -m 2 -b $i
+--------+------------------------------+
| mem_sz |      Unmap VM (s)            |
|  (GB)  | Baseline | Baseline + series |
+--------+------------------------------+
|   1    |  0.54    |  0.13             |
|   2    |  1.07    |  0.25             |
|   4    |  2.10    |  0.47             |
|   8    |  4.19    |  0.92             |
|  16    |  8.35    |  1.92             |
|  32    | 16.66    |  3.61             |
|  64    | 32.36    |  7.62             |
| 128    | 64.65    | 14.39             |   
+--------+----------+-------------------+

The series sees an average gain of 4x when the guest backed by PAGE_SIZE (4K) pages.

Other testing:
- Booted on x86_64 and ran KVM selftests.
- Build tested for MIPS and RISCV architectures with malta_kvm_defconfig and defconfig, respectively.

Cc David Matlack <[email protected]>

v7:
https://lore.kernel.org/all/20230722022251.3446223-1-rananta at google.com/ Thanks, Marc and Sean for the reviews and suggestions
- Made the function declaration for kvm_arch_flush_remote_tlbs() global. (Marc, Sean)
- Rename 'pages' to 'nr_pages' in TLBI functions that were introduced in the series. (Sean)
- Define __flush_s2_tlb_range_op() as a wrapper over __flush_tlb_range_op(). (Marc)
- Correct/improve the comments as suggested throughout the series. (Marc)
- Get rid of WARN_ON() check and the 'struct stage2_unmap_data' as they were not necessary. (Marc)

v6:
https://lore.kernel.org/all/20230715005405.3689586-1-rananta at google.com/ Thank you, Philippe and Shaoqin for the reviews and suggestions
- Split the patch-2/11 to separate the removal of CONFIG_HAVE_KVM_ARCH_TLB_FLUSH_ALL and arm64 to switch to using kvm_arch_flush_remote_tlbs(). (Philippe)
- Align the 'pages' argument with 'kvm' in patch-3/11. (Shaoqin)
- Call __tlb_switch_to_guest() before __flush_tlb_range_op() in the VHE's implementation of __kvm_tlb_flush_vmid_range(). (Shaoqin)

v5 (RESEND):
https://lore.kernel.org/all/20230621175002.2832640-1-rananta at google.com/ Thanks, Gavin for the suggestions:
- Adjusted the comment on patch-2 to align with the code.
- Fixed checkpatch.pl warning on patch-5.

v5:
https://lore.kernel.org/all/20230606192858.3600174-1-rananta at google.com/ Thank you, Marc and Oliver for the comments
- Introduced a helper, kvm_tlb_flush_vmid_range(), to handle the decision of using range-based TLBI instructions or invalidating the entire VMID, rather than depending on __kvm_tlb_flush_vmid_range() for it.
- kvm_tlb_flush_vmid_range() splits the range-based invalidations if the requested range exceeds MAX_TLBI_RANGE_PAGES.
- All the users in need of invalidating the TLB upon a range now depends on kvm_tlb_flush_vmid_range() rather than directly on __kvm_tlb_flush_vmid_range().
- stage2_unmap_defer_tlb_flush() introduces a WARN_ON() to track if there's any change in TLBIRANGE or FWB support during the unmap process as the features are based on alternative patching and the TLBI operations solely depend
on this check.
- Corrected an incorrect hunk being present on v4's patch-3.
- Updated the patches changelog and code comments as per the suggestions.

v4:
https://lore.kernel.org/all/20230519005231.3027912-1-rananta at google.com/ Thanks again, Oliver for all the comments
- Updated the __kvm_tlb_flush_vmid_range() implementation for nVHE to adjust with the modfied __tlb_switch_to_guest() that accepts a new 'bool nsh' arg.
- Renamed stage2_put_pte() to stage2_unmap_put_pte() and removed the 'skip_flush' argument.
- Defined stage2_unmap_defer_tlb_flush() to check if the PTE flushes can be deferred during the unmap table walk. It's being called from stage2_unmap_put_pte() and kvm_pgtable_stage2_unmap().
- Got rid of the 'struct stage2_unmap_data'.

v3:
https://lore.kernel.org/all/20230414172922.812640-1-rananta at google.com/ Thanks, Oliver for all the suggestions.
- The core flush API (__kvm_tlb_flush_vmid_range()) now checks if the system support FEAT_TLBIRANGE or not, thus elimiating the redundancy in the upper layers.
- If FEAT_TLBIRANGE is not supported, the implementation falls back to invalidating all the TLB entries with the VMID, instead of doing an iterative flush for the range.
- The kvm_arch_flush_remote_tlbs_range() doesn't return -EOPNOTSUPP if the system doesn't implement FEAT_TLBIRANGE. It depends on __kvm_tlb_flush_vmid_range() to do take care of the decisions and return 0 regardless of the underlying feature support.
- __kvm_tlb_flush_vmid_range() doesn't take 'level' as input to calculate the 'stride'. Instead, it always assumes PAGE_SIZE.
- Fast unmap path is eliminated. Instead, the existing unmap walker is modified to skip the TLBIs during the walk, and do it all at once after the walk, using the range-based instructions.

v2:
https://lore.kernel.org/all/20230206172340.2639971-1-rananta at google.com/
- Rebased the series on top of David Matlack's series for common TLB invalidation API[1].
- Implement kvm_arch_flush_remote_tlbs_range() for arm64, by extending the support introduced by [1].
- Use kvm_flush_remote_tlbs_memslot() introduced by [1] to flush only the current memslot after write-protect.
- Modified the __kvm_tlb_flush_range() macro to accepts 'level' as an argument to calculate the 'stride' instead of just using PAGE_SIZE.
- Split the patch that introduces the range-based TLBI to KVM and the implementation of IPA-based invalidation into its own patches.
- Dropped the patch that tries to optimize the mmu notifiers paths.
- Rename the function kvm_table_pte_flush() to kvm_pgtable_stage2_flush_range(), and accept the range of addresses to flush. [Oliver]
- Drop the 'tlb_level' argument for stage2_try_break_pte() and directly pass '0' as 'tlb_level' to kvm_pgtable_stage2_flush_range(). [Oliver]

v1:
https://lore.kernel.org/all/20230109215347.3119271-1-rananta at google.com/

Thank you.
Raghavendra

David Matlack (3):
KVM: Rename kvm_arch_flush_remote_tlb() to
kvm_arch_flush_remote_tlbs()
KVM: Allow range-based TLB invalidation from common code
KVM: Move kvm_arch_flush_remote_tlbs_memslot() to common code

Raghavendra Rao Ananta (11):
KVM: Declare kvm_arch_flush_remote_tlbs() globally
KVM: arm64: Use kvm_arch_flush_remote_tlbs()
KVM: Remove CONFIG_HAVE_KVM_ARCH_TLB_FLUSH_ALL
arm64: tlb: Refactor the core flush algorithm of __flush_tlb_range
arm64: tlb: Implement __flush_s2_tlb_range_op()
KVM: arm64: Implement __kvm_tlb_flush_vmid_range()
KVM: arm64: Define kvm_tlb_flush_vmid_range()
KVM: arm64: Implement kvm_arch_flush_remote_tlbs_range()
KVM: arm64: Flush only the memslot after write-protect
KVM: arm64: Invalidate the table entries upon a range
KVM: arm64: Use TLBI range-based intructions for unmap

 arch/arm64/include/asm/kvm_asm.h     |   3 +
 arch/arm64/include/asm/kvm_host.h    |   4 +
 arch/arm64/include/asm/kvm_pgtable.h |  10 +++
 arch/arm64/include/asm/tlbflush.h    | 124 +++++++++++++++------------
 arch/arm64/kvm/Kconfig               |   1 -
 arch/arm64/kvm/arm.c                 |   6 --
 arch/arm64/kvm/hyp/nvhe/hyp-main.c   |  11 +++
 arch/arm64/kvm/hyp/nvhe/tlb.c        |  30 +++++++
 arch/arm64/kvm/hyp/pgtable.c         |  63 ++++++++++++--
 arch/arm64/kvm/hyp/vhe/tlb.c         |  28 ++++++
 arch/arm64/kvm/mmu.c                 |  16 +++-
 arch/mips/include/asm/kvm_host.h     |   3 +-
 arch/mips/kvm/mips.c                 |  12 +--
 arch/riscv/kvm/mmu.c                 |   6 --
 arch/x86/include/asm/kvm_host.h      |   6 +-
 arch/x86/kvm/mmu/mmu.c               |  24 ++----
 arch/x86/kvm/mmu/mmu_internal.h      |   3 -
 arch/x86/kvm/x86.c                   |   2 +-
 include/linux/kvm_host.h             |  25 ++++--
 virt/kvm/Kconfig                     |   3 -
 virt/kvm/kvm_main.c                  |  35 ++++++--
 21 files changed, 287 insertions(+), 128 deletions(-)
Signature · 1 line
2.41.0.640.ga95def55d0-goog
#1 lore raw

PATCH v8 1/14 KVM: Rename kvm_arch_flush_remote_tlb() to kvm_arch_flush_remote_tlbs()

From: David Matlack <[email protected]>

Rename kvm_arch_flush_remote_tlb() and the associated macro __KVM_HAVE_ARCH_FLUSH_REMOTE_TLB to kvm_arch_flush_remote_tlbs() and __KVM_HAVE_ARCH_FLUSH_REMOTE_TLBS respectively.

Making the name plural matches kvm_flush_remote_tlbs() and makes it more clear that this function can affect more than one remote TLB.

No functional change intended.

---
 arch/mips/include/asm/kvm_host.h | 4 ++--
 arch/mips/kvm/mips.c             | 2 +-
 arch/x86/include/asm/kvm_host.h  | 4 ++--
 include/linux/kvm_host.h         | 4 ++--
 virt/kvm/kvm_main.c              | 2 +-
 5 files changed, 8 insertions(+), 8 deletions(-)
arch/mips/include/asm/kvm_host.h +2 -2
index 04cedf9f88115..9b0ad8f3bf327 100644
@@ -896,7 +896,7 @@ static inline void kvm_arch_sched_in(struct kvm_vcpu *vcpu, int cpu) {}
 static inline void kvm_arch_vcpu_blocking(struct kvm_vcpu *vcpu) {}
 static inline void kvm_arch_vcpu_unblocking(struct kvm_vcpu *vcpu) {}
 
-#define __KVM_HAVE_ARCH_FLUSH_REMOTE_TLB
-int kvm_arch_flush_remote_tlb(struct kvm *kvm);
+#define __KVM_HAVE_ARCH_FLUSH_REMOTE_TLBS
+int kvm_arch_flush_remote_tlbs(struct kvm *kvm);
 
 #endif /* __MIPS_KVM_HOST_H__ */
arch/mips/kvm/mips.c +1 -1
index aa5583a7b05be..4b7bc39a41736 100644
@@ -981,7 +981,7 @@ void kvm_arch_sync_dirty_log(struct kvm *kvm, struct kvm_memory_slot *memslot)
 
 }
 
-int kvm_arch_flush_remote_tlb(struct kvm *kvm)
+int kvm_arch_flush_remote_tlbs(struct kvm *kvm)
 {
 	kvm_mips_callbacks->prepare_flush_shadow(kvm);
 	return 1;
arch/x86/include/asm/kvm_host.h +2 -2
index 28bd38303d704..a2d3cfc2eb75c 100644
@@ -1794,8 +1794,8 @@ static inline struct kvm *kvm_arch_alloc_vm(void)
 #define __KVM_HAVE_ARCH_VM_FREE
 void kvm_arch_free_vm(struct kvm *kvm);
 
-#define __KVM_HAVE_ARCH_FLUSH_REMOTE_TLB
-static inline int kvm_arch_flush_remote_tlb(struct kvm *kvm)
+#define __KVM_HAVE_ARCH_FLUSH_REMOTE_TLBS
+static inline int kvm_arch_flush_remote_tlbs(struct kvm *kvm)
 {
 	if (kvm_x86_ops.flush_remote_tlbs &&
 	    !static_call(kvm_x86_flush_remote_tlbs)(kvm))
include/linux/kvm_host.h +2 -2
index 9d3ac7720da9f..e3f968b38ae97 100644
@@ -1479,8 +1479,8 @@ static inline void kvm_arch_free_vm(struct kvm *kvm)
 }
 #endif
 
-#ifndef __KVM_HAVE_ARCH_FLUSH_REMOTE_TLB
-static inline int kvm_arch_flush_remote_tlb(struct kvm *kvm)
+#ifndef __KVM_HAVE_ARCH_FLUSH_REMOTE_TLBS
+static inline int kvm_arch_flush_remote_tlbs(struct kvm *kvm)
 {
 	return -ENOTSUPP;
 }
virt/kvm/kvm_main.c +1 -1
index dfbaafbe3a009..70e5479797ac3 100644
@@ -361,7 +361,7 @@ void kvm_flush_remote_tlbs(struct kvm *kvm)
 	 * kvm_make_all_cpus_request() reads vcpu->mode. We reuse that
 	 * barrier here.
 	 */
-	if (!kvm_arch_flush_remote_tlb(kvm)
+	if (!kvm_arch_flush_remote_tlbs(kvm)
 	    || kvm_make_all_cpus_request(kvm, KVM_REQ_TLB_FLUSH))
 		++kvm->stat.generic.remote_tlb_flush;
 }
Signature · 1 line
2.41.0.640.ga95def55d0-goog
#2 lore raw

PATCH v8 2/14 KVM: Declare kvm_arch_flush_remote_tlbs() globally

In reply to Raghavendra Rao Ananta

There's no reason for the architectures to declare kvm_arch_flush_remote_tlbs() in their own headers. Hence to avoid this duplication, make the declaration global, leaving the architectures to define only __KVM_HAVE_ARCH_FLUSH_REMOTE_TLBS as needed.

---
 arch/mips/include/asm/kvm_host.h | 1 -
 include/linux/kvm_host.h         | 2 ++
 2 files changed, 2 insertions(+), 1 deletion(-)
arch/mips/include/asm/kvm_host.h +0 -1
index 9b0ad8f3bf327..54a85f1d4f2c8 100644
@@ -897,6 +897,5 @@ static inline void kvm_arch_vcpu_blocking(struct kvm_vcpu *vcpu) {}
 static inline void kvm_arch_vcpu_unblocking(struct kvm_vcpu *vcpu) {}
 
 #define __KVM_HAVE_ARCH_FLUSH_REMOTE_TLBS
-int kvm_arch_flush_remote_tlbs(struct kvm *kvm);
 
 #endif /* __MIPS_KVM_HOST_H__ */
include/linux/kvm_host.h +2 -0
index e3f968b38ae97..ade5d4500c2ce 100644
@@ -1484,6 +1484,8 @@ static inline int kvm_arch_flush_remote_tlbs(struct kvm *kvm)
 {
 	return -ENOTSUPP;
 }
+#else
+int kvm_arch_flush_remote_tlbs(struct kvm *kvm);
 #endif
 
 #ifdef __KVM_HAVE_ARCH_NONCOHERENT_DMA
Signature · 1 line
2.41.0.640.ga95def55d0-goog
#3 lore raw

Re: [PATCH v8 02/14] KVM: Declare kvm_arch_flush_remote_tlbs() globally

Quoted 28 lines · Raghavendra Rao Ananta

There's no reason for the architectures to declare kvm_arch_flush_remote_tlbs() in their own headers. Hence to avoid this duplication, make the declaration global, leaving the architectures to define only __KVM_HAVE_ARCH_FLUSH_REMOTE_TLBS as needed.

---
  arch/mips/include/asm/kvm_host.h | 1 -
  include/linux/kvm_host.h         | 2 ++
  2 files changed, 2 insertions(+), 1 deletion(-)
arch/mips/include/asm/kvm_host.h +0 -1
index 9b0ad8f3bf327..54a85f1d4f2c8 100644
@@ -897,6 +897,5 @@ static inline void kvm_arch_vcpu_blocking(struct kvm_vcpu *vcpu) {}
  static inline void kvm_arch_vcpu_unblocking(struct kvm_vcpu *vcpu) {}
 
  #define __KVM_HAVE_ARCH_FLUSH_REMOTE_TLBS
-int kvm_arch_flush_remote_tlbs(struct kvm *kvm);
 
  #endif /* __MIPS_KVM_HOST_H__ */
include/linux/kvm_host.h +0 -0
index e3f968b38ae97..ade5d4500c2ce 100644

include/linux/kvm_host.h static inline int kvm_arch_flush_remote_tlbs(struct kvm *kvm) · line 1491

Quoted 8 lines
include/linux/kvm_host.h +2 -0
@@ -1484,6 +1484,8 @@ static inline int kvm_arch_flush_remote_tlbs(struct kvm *kvm)
  {
  	return -ENOTSUPP;
  }
+#else
+int kvm_arch_flush_remote_tlbs(struct kvm *kvm);
  #endif
 
  #ifdef __KVM_HAVE_ARCH_NONCOHERENT_DMA

Is the declaration inconsistent to that in arch/x86/include/asm/kvm_host.h? In order to keep them consistent, I guess we need move kvm_arch_flush_remote_tlbs() from x86's header file to arch/x86/kvm/mmu/mmu.c and 'inline' needs to be dropped.

Thanks,
Gavin

#4 lore raw

Re: [PATCH v8 02/14] KVM: Declare kvm_arch_flush_remote_tlbs() globally

Hi Gavin,

Quoted 38 lines · On Tue, Aug 8, 2023 at 9:00?PM Gavin Shan <[email protected]> wrote:
Quoted 35 lines · Raghavendra Rao Ananta

There's no reason for the architectures to declare kvm_arch_flush_remote_tlbs() in their own headers. Hence to avoid this duplication, make the declaration global, leaving the architectures to define only __KVM_HAVE_ARCH_FLUSH_REMOTE_TLBS as needed.

---
  arch/mips/include/asm/kvm_host.h | 1 -
  include/linux/kvm_host.h         | 2 ++
  2 files changed, 2 insertions(+), 1 deletion(-)
arch/mips/include/asm/kvm_host.h +0 -1
index 9b0ad8f3bf327..54a85f1d4f2c8 100644
@@ -897,6 +897,5 @@ static inline void kvm_arch_vcpu_blocking(struct kvm_vcpu *vcpu) {}
  static inline void kvm_arch_vcpu_unblocking(struct kvm_vcpu *vcpu) {}
 
  #define __KVM_HAVE_ARCH_FLUSH_REMOTE_TLBS
-int kvm_arch_flush_remote_tlbs(struct kvm *kvm);
 
  #endif /* __MIPS_KVM_HOST_H__ */
include/linux/kvm_host.h +2 -0
index e3f968b38ae97..ade5d4500c2ce 100644
@@ -1484,6 +1484,8 @@ static inline int kvm_arch_flush_remote_tlbs(struct kvm *kvm)
  {
      return -ENOTSUPP;
  }
+#else
+int kvm_arch_flush_remote_tlbs(struct kvm *kvm);
  #endif
 
Quoted 6 lines

include/linux/kvm_host.h static inline int kvm_arch_flush_remote_tlbs(struct kvm *kvm) · line 1491

Quoted 1 line
include/linux/kvm_host.h +0 -0
  #ifdef __KVM_HAVE_ARCH_NONCOHERENT_DMA

Is the declaration inconsistent to that in arch/x86/include/asm/kvm_host.h? In order to keep them consistent, I guess we need move kvm_arch_flush_remote_tlbs() from x86's header file to arch/x86/kvm/mmu/mmu.c and 'inline' needs to be dropped.

Unsure of the original intentions, I didn't want to disturb any existing arrangements. If more people agree to this refactoring, I'm happy to move.

Thank you.
Raghavendra

Quoted 3 lines

Thanks,
Gavin

#5 lore raw

Re: [PATCH v8 02/14] KVM: Declare kvm_arch_flush_remote_tlbs() globally

Quoted 47 lines · Raghavendra Rao Ananta

Hi Gavin,

Quoted 38 lines · On Tue, Aug 8, 2023 at 9:00?PM Gavin Shan <[email protected]> wrote:
Quoted 35 lines · Raghavendra Rao Ananta

There's no reason for the architectures to declare kvm_arch_flush_remote_tlbs() in their own headers. Hence to avoid this duplication, make the declaration global, leaving the architectures to define only __KVM_HAVE_ARCH_FLUSH_REMOTE_TLBS as needed.

---
   arch/mips/include/asm/kvm_host.h | 1 -
   include/linux/kvm_host.h         | 2 ++
   2 files changed, 2 insertions(+), 1 deletion(-)
arch/mips/include/asm/kvm_host.h +0 -1
index 9b0ad8f3bf327..54a85f1d4f2c8 100644
@@ -897,6 +897,5 @@ static inline void kvm_arch_vcpu_blocking(struct kvm_vcpu *vcpu) {}
   static inline void kvm_arch_vcpu_unblocking(struct kvm_vcpu *vcpu) {}
 
   #define __KVM_HAVE_ARCH_FLUSH_REMOTE_TLBS
-int kvm_arch_flush_remote_tlbs(struct kvm *kvm);
 
   #endif /* __MIPS_KVM_HOST_H__ */
include/linux/kvm_host.h +2 -0
index e3f968b38ae97..ade5d4500c2ce 100644
@@ -1484,6 +1484,8 @@ static inline int kvm_arch_flush_remote_tlbs(struct kvm *kvm)
   {
       return -ENOTSUPP;
   }
+#else
+int kvm_arch_flush_remote_tlbs(struct kvm *kvm);
   #endif
 
Quoted 6 lines

include/linux/kvm_host.h static inline int kvm_arch_flush_remote_tlbs(struct kvm *kvm) · line 1491

Quoted 1 line
include/linux/kvm_host.h +0 -0
   #ifdef __KVM_HAVE_ARCH_NONCOHERENT_DMA

Is the declaration inconsistent to that in arch/x86/include/asm/kvm_host.h? In order to keep them consistent, I guess we need move kvm_arch_flush_remote_tlbs() from x86's header file to arch/x86/kvm/mmu/mmu.c and 'inline' needs to be dropped.

Quoted 3 lines

Unsure of the original intentions, I didn't want to disturb any existing arrangements. If more people agree to this refactoring, I'm happy to move.

This is amazing to me. This change can be compiled without any error even if the declaration inconsistent between the kvm_host.h and x86's header file.

I'm curious which option make it possible?

Thanks,
Shaoqin

Quoted 7 lines

Thank you.
Raghavendra

Quoted 3 lines

Thanks,
Gavin

Signature · 1 line
Shaoqin
#6 lore raw

Re: [PATCH v8 02/14] KVM: Declare kvm_arch_flush_remote_tlbs() globally

Quoted 55 lines · On Thu, Aug 10, 2023 at 5:26?AM Shaoqin Huang <[email protected]> wrote:
Quoted 47 lines · Raghavendra Rao Ananta

Hi Gavin,

Quoted 38 lines · On Tue, Aug 8, 2023 at 9:00?PM Gavin Shan <[email protected]> wrote:
Quoted 35 lines · Raghavendra Rao Ananta

There's no reason for the architectures to declare kvm_arch_flush_remote_tlbs() in their own headers. Hence to avoid this duplication, make the declaration global, leaving the architectures to define only __KVM_HAVE_ARCH_FLUSH_REMOTE_TLBS as needed.

---
   arch/mips/include/asm/kvm_host.h | 1 -
   include/linux/kvm_host.h         | 2 ++
   2 files changed, 2 insertions(+), 1 deletion(-)
arch/mips/include/asm/kvm_host.h +0 -1
index 9b0ad8f3bf327..54a85f1d4f2c8 100644
@@ -897,6 +897,5 @@ static inline void kvm_arch_vcpu_blocking(struct kvm_vcpu *vcpu) {}
   static inline void kvm_arch_vcpu_unblocking(struct kvm_vcpu *vcpu) {}
 
   #define __KVM_HAVE_ARCH_FLUSH_REMOTE_TLBS
-int kvm_arch_flush_remote_tlbs(struct kvm *kvm);
 
   #endif /* __MIPS_KVM_HOST_H__ */
include/linux/kvm_host.h +2 -0
index e3f968b38ae97..ade5d4500c2ce 100644
@@ -1484,6 +1484,8 @@ static inline int kvm_arch_flush_remote_tlbs(struct kvm *kvm)
   {
       return -ENOTSUPP;
   }
+#else
+int kvm_arch_flush_remote_tlbs(struct kvm *kvm);
   #endif
 
Quoted 6 lines

include/linux/kvm_host.h static inline int kvm_arch_flush_remote_tlbs(struct kvm *kvm) · line 1491

Quoted 1 line
include/linux/kvm_host.h +0 -0
   #ifdef __KVM_HAVE_ARCH_NONCOHERENT_DMA

Is the declaration inconsistent to that in arch/x86/include/asm/kvm_host.h? In order to keep them consistent, I guess we need move kvm_arch_flush_remote_tlbs() from x86's header file to arch/x86/kvm/mmu/mmu.c and 'inline' needs to be dropped.

Quoted 3 lines

Unsure of the original intentions, I didn't want to disturb any existing arrangements. If more people agree to this refactoring, I'm happy to move.

Quoted 6 lines

This is amazing to me. This change can be compiled without any error even if the declaration inconsistent between the kvm_host.h and x86's header file.

I'm curious which option make it possible?

After doing some experiments, I think it works because of the order in which the inline-definition and the declaration are laid out. If the 'inline' part of the function comes first and then the declaration, we don't see any error. However if the positions were reversed, we would see an error. (I'm not sure what the technical reason for this is).

Just to be safe, I can move the definition to arch/x86/kvm/mmu/mmu.c as a non-inline function.

Thank you.
Raghavendra

Quoted 14 lines

Thanks,
Shaoqin

Quoted 7 lines

Thank you.
Raghavendra

Quoted 3 lines

Thanks,
Gavin

--
Shaoqin

#7 lore raw

Re: [PATCH v8 02/14] KVM: Declare kvm_arch_flush_remote_tlbs() globally

Quoted 30 lines · Raghavendra Rao Ananta
Quoted 23 lines · On Thu, Aug 10, 2023 at 5:26?AM Shaoqin Huang <[email protected]> wrote:
Quoted 18 lines · Raghavendra Rao Ananta
Quoted 12 lines
Quoted 12 lines
include/linux/kvm_host.h +2 -0
index e3f968b38ae97..ade5d4500c2ce 100644
@@ -1484,6 +1484,8 @@ static inline int kvm_arch_flush_remote_tlbs(struct kvm *kvm)
   {
       return -ENOTSUPP;
   }
+#else
+int kvm_arch_flush_remote_tlbs(struct kvm *kvm);
   #endif
 
Quoted 6 lines

include/linux/kvm_host.h static inline int kvm_arch_flush_remote_tlbs(struct kvm *kvm) · line 1491

Quoted 1 line
include/linux/kvm_host.h +0 -0
   #ifdef __KVM_HAVE_ARCH_NONCOHERENT_DMA

Is the declaration inconsistent to that in arch/x86/include/asm/kvm_host.h? In order to keep them consistent, I guess we need move kvm_arch_flush_remote_tlbs() from x86's header file to arch/x86/kvm/mmu/mmu.c and 'inline' needs to be dropped.

Quoted 3 lines

Unsure of the original intentions, I didn't want to disturb any existing arrangements. If more people agree to this refactoring, I'm happy to move.

Quoted 6 lines

This is amazing to me. This change can be compiled without any error even if the declaration inconsistent between the kvm_host.h and x86's header file.

I'm curious which option make it possible?

Quoted 8 lines

After doing some experiments, I think it works because of the order in which the inline-definition and the declaration are laid out. If the 'inline' part of the function comes first and then the declaration, we don't see any error. However if the positions were reversed, we would see an error. (I'm not sure what the technical reason for this is).

Just to be safe, I can move the definition to arch/x86/kvm/mmu/mmu.c as a non-inline function.

No need, asm/kvm_host.h _must_ be included before the declaration, otherwise the declaration wouldn't be made because __KVM_HAVE_ARCH_FLUSH_REMOTE_TLBS wouldn't be defined. I.e. we won't run into issues where the non-static declaration comes before the static inline definition.

C99 explicitly covers this case:

6.2.2 Linkages of identifiers

...

If the declaration of a file scope identifier for an object or a function contains the storage- class specifier static, the identifier has internal linkage.

For an identifier declared with the storage-class specifier extern in a scope in which a prior declaration of that identifier is visible if the prior declaration specifies internal or external linkage, the linkage of the identifier at the later declaration is the same as the linkage specified at the prior declaration. If no prior declaration is visible, or if the prior declaration specifies no linkage, then the identifier has external linkage.

In short, because the "static inline" declared internal linkage first, it wins.

#8 lore raw

Re: [PATCH v8 02/14] KVM: Declare kvm_arch_flush_remote_tlbs() globally

Quoted 55 lines · On Thu, Aug 10, 2023 at 3:20?PM Sean Christopherson <[email protected]> wrote:
Quoted 30 lines · Raghavendra Rao Ananta
Quoted 23 lines · On Thu, Aug 10, 2023 at 5:26?AM Shaoqin Huang <[email protected]> wrote:
Quoted 18 lines · Raghavendra Rao Ananta
Quoted 12 lines
Quoted 12 lines
include/linux/kvm_host.h +2 -0
index e3f968b38ae97..ade5d4500c2ce 100644
@@ -1484,6 +1484,8 @@ static inline int kvm_arch_flush_remote_tlbs(struct kvm *kvm)
   {
       return -ENOTSUPP;
   }
+#else
+int kvm_arch_flush_remote_tlbs(struct kvm *kvm);
   #endif
 
Quoted 6 lines

include/linux/kvm_host.h static inline int kvm_arch_flush_remote_tlbs(struct kvm *kvm) · line 1491

Quoted 1 line
include/linux/kvm_host.h +0 -0
   #ifdef __KVM_HAVE_ARCH_NONCOHERENT_DMA

Is the declaration inconsistent to that in arch/x86/include/asm/kvm_host.h? In order to keep them consistent, I guess we need move kvm_arch_flush_remote_tlbs() from x86's header file to arch/x86/kvm/mmu/mmu.c and 'inline' needs to be dropped.

Quoted 3 lines

Unsure of the original intentions, I didn't want to disturb any existing arrangements. If more people agree to this refactoring, I'm happy to move.

Quoted 6 lines

This is amazing to me. This change can be compiled without any error even if the declaration inconsistent between the kvm_host.h and x86's header file.

I'm curious which option make it possible?

Quoted 8 lines

After doing some experiments, I think it works because of the order in which the inline-definition and the declaration are laid out. If the 'inline' part of the function comes first and then the declaration, we don't see any error. However if the positions were reversed, we would see an error. (I'm not sure what the technical reason for this is).

Just to be safe, I can move the definition to arch/x86/kvm/mmu/mmu.c as a non-inline function.

No need, asm/kvm_host.h _must_ be included before the declaration, otherwise the declaration wouldn't be made because __KVM_HAVE_ARCH_FLUSH_REMOTE_TLBS wouldn't be defined. I.e. we won't run into issues where the non-static declaration comes before the static inline definition.

C99 explicitly covers this case:

6.2.2 Linkages of identifiers

...

If the declaration of a file scope identifier for an object or a function contains the storage- class specifier static, the identifier has internal linkage.

Quoted 7 lines

For an identifier declared with the storage-class specifier extern in a scope in which a prior declaration of that identifier is visible if the prior declaration specifies internal or external linkage, the linkage of the identifier at the later declaration is the same as the linkage specified at the prior declaration. If no prior declaration is visible, or if the prior declaration specifies no linkage, then the identifier has external linkage.

In short, because the "static inline" declared internal linkage first, it wins.

Thanks for sharing this! I can keep the 'static inline' definition as is then. However, since a later patch (patch-05/14) defines kvm_arch_flush_remote_tlbs_range() in arch/x86/kvm/mmu/mmu.c, do you think we can move this definition to the .c file as well for consistency?

Thank you.
Raghavendra

Raghavendra

#9 lore raw

Re: [PATCH v8 02/14] KVM: Declare kvm_arch_flush_remote_tlbs() globally

Quoted 31 lines · Raghavendra Rao Ananta
Quoted 30 lines · On Thu, Aug 10, 2023 at 3:20?PM Sean Christopherson <[email protected]> wrote:
Quoted 8 lines · Raghavendra Rao Ananta

After doing some experiments, I think it works because of the order in which the inline-definition and the declaration are laid out. If the 'inline' part of the function comes first and then the declaration, we don't see any error. However if the positions were reversed, we would see an error. (I'm not sure what the technical reason for this is).

Just to be safe, I can move the definition to arch/x86/kvm/mmu/mmu.c as a non-inline function.

No need, asm/kvm_host.h _must_ be included before the declaration, otherwise the declaration wouldn't be made because __KVM_HAVE_ARCH_FLUSH_REMOTE_TLBS wouldn't be defined. I.e. we won't run into issues where the non-static declaration comes before the static inline definition.

C99 explicitly covers this case:

6.2.2 Linkages of identifiers

...

If the declaration of a file scope identifier for an object or a function contains the storage- class specifier static, the identifier has internal linkage.

For an identifier declared with the storage-class specifier extern in a scope in which a prior declaration of that identifier is visible if the prior declaration specifies internal or external linkage, the linkage of the identifier at the later declaration is the same as the linkage specified at the prior declaration. If no prior declaration is visible, or if the prior declaration specifies no linkage, then the identifier has external linkage.

Quoted 6 lines
Quoted 1 line

In short, because the "static inline" declared internal linkage first, it wins.

Thanks for sharing this! I can keep the 'static inline' definition as is then. However, since a later patch (patch-05/14) defines kvm_arch_flush_remote_tlbs_range() in arch/x86/kvm/mmu/mmu.c, do you think we can move this definition to the .c file as well for consistency?

We "can", but I don't see any reason to do so. Trying to make helpers consistently inline or not is usually a fools errand. And in this case, I'd actually rather go the opposite direction and make the range variant an inline.

Ha! And I can justify that with minimal effort. The below makes the helper a straight passthrough for CONFIG_HYPERV=n builds, at which point I think it makes sense for it to be inline.

If it won't slow your series down even more, any objection to sliding the below patch in somewhere before patch 5? And then add a patch to inline the range-based helper?

Disclaimer: compile tested only.

---
From: Sean Christopherson <[email protected]>
Date: Thu, 10 Aug 2023 15:58:53 -0700
Subject: [PATCH] KVM: x86/mmu: Declare flush_remote_tlbs{_range}() hooks iff
HYPERV!=n

Declare the kvm_x86_ops hooks used to wire up paravirt TLB flushes when running under Hyper-V if and only if CONFIG_HYPERV!=n. Wrapping yet more code with IS_ENABLED(CONFIG_HYPERV) eliminates a handful of conditional branches, and makes it super obvious why the hooks *might* be valid.

---
 arch/x86/include/asm/kvm-x86-ops.h | 2 ++
 arch/x86/include/asm/kvm_host.h    | 4 ++++
 arch/x86/kvm/mmu/mmu.c             | 6 ++++++
 3 files changed, 12 insertions(+)
arch/x86/include/asm/kvm-x86-ops.h +2 -0
index 13bc212cd4bc..6bc1ab0627b7 100644
@@ -54,8 +54,10 @@ KVM_X86_OP(set_rflags)
 KVM_X86_OP(get_if_flag)
 KVM_X86_OP(flush_tlb_all)
 KVM_X86_OP(flush_tlb_current)
+#if IS_ENABLED(CONFIG_HYPERV)
 KVM_X86_OP_OPTIONAL(flush_remote_tlbs)
 KVM_X86_OP_OPTIONAL(flush_remote_tlbs_range)
+#endif
 KVM_X86_OP(flush_tlb_gva)
 KVM_X86_OP(flush_tlb_guest)
 KVM_X86_OP(vcpu_pre_run)
arch/x86/include/asm/kvm_host.h +4 -0
index 60d430b4650f..04fc80112dfe 100644
@@ -1604,9 +1604,11 @@ struct kvm_x86_ops {
 
 	void (*flush_tlb_all)(struct kvm_vcpu *vcpu);
 	void (*flush_tlb_current)(struct kvm_vcpu *vcpu);
+#if IS_ENABLED(CONFIG_HYPERV)
 	int  (*flush_remote_tlbs)(struct kvm *kvm);
 	int  (*flush_remote_tlbs_range)(struct kvm *kvm, gfn_t gfn,
 					gfn_t nr_pages);
+#endif
 
 	/*
 	 * Flush any TLB entries associated with the given GVA.
@@ -1814,6 +1816,7 @@ static inline struct kvm *kvm_arch_alloc_vm(void)
 #define __KVM_HAVE_ARCH_VM_FREE
 void kvm_arch_free_vm(struct kvm *kvm);
 
+#if IS_ENABLED(CONFIG_HYPERV)
 #define __KVM_HAVE_ARCH_FLUSH_REMOTE_TLB
 static inline int kvm_arch_flush_remote_tlb(struct kvm *kvm)
 {
@@ -1823,6 +1826,7 @@ static inline int kvm_arch_flush_remote_tlb(struct kvm *kvm)
 	else
 		return -ENOTSUPP;
 }
+#endif
 
 #define kvm_arch_pmi_in_guest(vcpu) \
 	((vcpu) && (vcpu)->arch.handling_intr_from_guest)
arch/x86/kvm/mmu/mmu.c +6 -0
index 9e4cd8b4a202..0189dfecce1f 100644
@@ -271,18 +271,24 @@ static inline unsigned long kvm_mmu_get_guest_pgd(struct kvm_vcpu *vcpu,
 
 static inline bool kvm_available_flush_remote_tlbs_range(void)
 {
+#if IS_ENABLED(CONFIG_HYPERV)
 	return kvm_x86_ops.flush_remote_tlbs_range;
+#else
+	return false;
+#endif
 }
 
 void kvm_flush_remote_tlbs_range(struct kvm *kvm, gfn_t start_gfn,
 				 gfn_t nr_pages)
 {
+#if IS_ENABLED(CONFIG_HYPERV)
 	int ret = -EOPNOTSUPP;
 
 	if (kvm_x86_ops.flush_remote_tlbs_range)
 		ret = static_call(kvm_x86_flush_remote_tlbs_range)(kvm, start_gfn,
 								   nr_pages);
 	if (ret)
+#endif
 		kvm_flush_remote_tlbs(kvm);
 }
 

base-commit: bc9e68820274c025840d3056d63f938d74ca35bb

#10 lore raw

Re: [PATCH v8 02/14] KVM: Declare kvm_arch_flush_remote_tlbs() globally

Quoted 44 lines · On Thu, Aug 10, 2023 at 4:04?PM Sean Christopherson <[email protected]> wrote:
Quoted 31 lines · Raghavendra Rao Ananta
Quoted 30 lines · On Thu, Aug 10, 2023 at 3:20?PM Sean Christopherson <[email protected]> wrote:
Quoted 8 lines · Raghavendra Rao Ananta

After doing some experiments, I think it works because of the order in which the inline-definition and the declaration are laid out. If the 'inline' part of the function comes first and then the declaration, we don't see any error. However if the positions were reversed, we would see an error. (I'm not sure what the technical reason for this is).

Just to be safe, I can move the definition to arch/x86/kvm/mmu/mmu.c as a non-inline function.

No need, asm/kvm_host.h _must_ be included before the declaration, otherwise the declaration wouldn't be made because __KVM_HAVE_ARCH_FLUSH_REMOTE_TLBS wouldn't be defined. I.e. we won't run into issues where the non-static declaration comes before the static inline definition.

C99 explicitly covers this case:

6.2.2 Linkages of identifiers

...

If the declaration of a file scope identifier for an object or a function contains the storage- class specifier static, the identifier has internal linkage.

For an identifier declared with the storage-class specifier extern in a scope in which a prior declaration of that identifier is visible if the prior declaration specifies internal or external linkage, the linkage of the identifier at the later declaration is the same as the linkage specified at the prior declaration. If no prior declaration is visible, or if the prior declaration specifies no linkage, then the identifier has external linkage.

Quoted 6 lines
Quoted 1 line

In short, because the "static inline" declared internal linkage first, it wins.

Thanks for sharing this! I can keep the 'static inline' definition as is then. However, since a later patch (patch-05/14) defines kvm_arch_flush_remote_tlbs_range() in arch/x86/kvm/mmu/mmu.c, do you think we can move this definition to the .c file as well for consistency?

We "can", but I don't see any reason to do so. Trying to make helpers consistently inline or not is usually a fools errand. And in this case, I'd actually rather go the opposite direction and make the range variant an inline.

Quoted 8 lines

Ha! And I can justify that with minimal effort. The below makes the helper a straight passthrough for CONFIG_HYPERV=n builds, at which point I think it makes sense for it to be inline.

If it won't slow your series down even more, any objection to sliding the below patch in somewhere before patch 5? And then add a patch to inline the range-based helper?

Since it is a little diverted from the rest of the series' goal (and yes, the slowing down part), do you mind if we send it as a separate series? :)
I'll keep the functions as we have on v8 for now.

Thank you.
Raghavendra

Quoted 100 lines

Disclaimer: compile tested only.

---
From: Sean Christopherson <[email protected]>
Date: Thu, 10 Aug 2023 15:58:53 -0700
Subject: [PATCH] KVM: x86/mmu: Declare flush_remote_tlbs{_range}() hooks iff
HYPERV!=n

Declare the kvm_x86_ops hooks used to wire up paravirt TLB flushes when running under Hyper-V if and only if CONFIG_HYPERV!=n. Wrapping yet more code with IS_ENABLED(CONFIG_HYPERV) eliminates a handful of conditional branches, and makes it super obvious why the hooks *might* be valid.

---
 arch/x86/include/asm/kvm-x86-ops.h | 2 ++
 arch/x86/include/asm/kvm_host.h    | 4 ++++
 arch/x86/kvm/mmu/mmu.c             | 6 ++++++
 3 files changed, 12 insertions(+)
arch/x86/include/asm/kvm-x86-ops.h +2 -0
index 13bc212cd4bc..6bc1ab0627b7 100644
@@ -54,8 +54,10 @@ KVM_X86_OP(set_rflags)
 KVM_X86_OP(get_if_flag)
 KVM_X86_OP(flush_tlb_all)
 KVM_X86_OP(flush_tlb_current)
+#if IS_ENABLED(CONFIG_HYPERV)
 KVM_X86_OP_OPTIONAL(flush_remote_tlbs)
 KVM_X86_OP_OPTIONAL(flush_remote_tlbs_range)
+#endif
 KVM_X86_OP(flush_tlb_gva)
 KVM_X86_OP(flush_tlb_guest)
 KVM_X86_OP(vcpu_pre_run)
arch/x86/include/asm/kvm_host.h +4 -0
index 60d430b4650f..04fc80112dfe 100644
@@ -1604,9 +1604,11 @@ struct kvm_x86_ops {
 
        void (*flush_tlb_all)(struct kvm_vcpu *vcpu);
        void (*flush_tlb_current)(struct kvm_vcpu *vcpu);
+#if IS_ENABLED(CONFIG_HYPERV)
        int  (*flush_remote_tlbs)(struct kvm *kvm);
        int  (*flush_remote_tlbs_range)(struct kvm *kvm, gfn_t gfn,
                                        gfn_t nr_pages);
+#endif
 
        /*
         * Flush any TLB entries associated with the given GVA.
@@ -1814,6 +1816,7 @@ static inline struct kvm *kvm_arch_alloc_vm(void)
 #define __KVM_HAVE_ARCH_VM_FREE
 void kvm_arch_free_vm(struct kvm *kvm);
 
+#if IS_ENABLED(CONFIG_HYPERV)
 #define __KVM_HAVE_ARCH_FLUSH_REMOTE_TLB
 static inline int kvm_arch_flush_remote_tlb(struct kvm *kvm)
 {
@@ -1823,6 +1826,7 @@ static inline int kvm_arch_flush_remote_tlb(struct kvm *kvm)
        else
                return -ENOTSUPP;
 }
+#endif
 
 #define kvm_arch_pmi_in_guest(vcpu) \
        ((vcpu) && (vcpu)->arch.handling_intr_from_guest)
arch/x86/kvm/mmu/mmu.c +6 -0
index 9e4cd8b4a202..0189dfecce1f 100644
@@ -271,18 +271,24 @@ static inline unsigned long kvm_mmu_get_guest_pgd(struct kvm_vcpu *vcpu,
 
 static inline bool kvm_available_flush_remote_tlbs_range(void)
 {
+#if IS_ENABLED(CONFIG_HYPERV)
        return kvm_x86_ops.flush_remote_tlbs_range;
+#else
+       return false;
+#endif
 }
 
 void kvm_flush_remote_tlbs_range(struct kvm *kvm, gfn_t start_gfn,
                                 gfn_t nr_pages)
 {
+#if IS_ENABLED(CONFIG_HYPERV)
        int ret = -EOPNOTSUPP;
 
        if (kvm_x86_ops.flush_remote_tlbs_range)
                ret = static_call(kvm_x86_flush_remote_tlbs_range)(kvm, start_gfn,
                                                                   nr_pages);
        if (ret)
+#endif
                kvm_flush_remote_tlbs(kvm);
 }
 

base-commit: bc9e68820274c025840d3056d63f938d74ca35bb --

Reviewed-by on v8 2/14
#11 lore raw

Re: [PATCH v8 02/14] KVM: Declare kvm_arch_flush_remote_tlbs() globally

In reply to Raghavendra Rao Ananta
Quoted 7 lines · Raghavendra Rao Ananta

There's no reason for the architectures to declare kvm_arch_flush_remote_tlbs() in their own headers. Hence to avoid this duplication, make the declaration global, leaving the architectures to define only __KVM_HAVE_ARCH_FLUSH_REMOTE_TLBS as needed.

Quoted 29 lines
---
  arch/mips/include/asm/kvm_host.h | 1 -
  include/linux/kvm_host.h         | 2 ++
  2 files changed, 2 insertions(+), 1 deletion(-)
arch/mips/include/asm/kvm_host.h +0 -1
index 9b0ad8f3bf327..54a85f1d4f2c8 100644
@@ -897,6 +897,5 @@ static inline void kvm_arch_vcpu_blocking(struct kvm_vcpu *vcpu) {}
  static inline void kvm_arch_vcpu_unblocking(struct kvm_vcpu *vcpu) {}
 
  #define __KVM_HAVE_ARCH_FLUSH_REMOTE_TLBS
-int kvm_arch_flush_remote_tlbs(struct kvm *kvm);
 
  #endif /* __MIPS_KVM_HOST_H__ */
include/linux/kvm_host.h +2 -0
index e3f968b38ae97..ade5d4500c2ce 100644
@@ -1484,6 +1484,8 @@ static inline int kvm_arch_flush_remote_tlbs(struct kvm *kvm)
  {
  	return -ENOTSUPP;
  }
+#else
+int kvm_arch_flush_remote_tlbs(struct kvm *kvm);
  #endif
 
  #ifdef __KVM_HAVE_ARCH_NONCOHERENT_DMA
Signature · 1 line
Shaoqin
#12 lore raw

PATCH v8 3/14 KVM: arm64: Use kvm_arch_flush_remote_tlbs()

In reply to Raghavendra Rao Ananta

Stop depending on CONFIG_HAVE_KVM_ARCH_TLB_FLUSH_ALL and opt to standardize on kvm_arch_flush_remote_tlbs() since it avoids duplicating the generic TLB stats across architectures that implement their own remote TLB flush.

This adds an extra function call to the ARM64 kvm_flush_remote_tlbs() path, but that is a small cost in comparison to flushing remote TLBs.

In addition, instead of just incrementing remote_tlb_flush_requests stat, the generic interface would also increment the remote_tlb_flush stat.

---
 arch/arm64/include/asm/kvm_host.h | 2 ++
 arch/arm64/kvm/Kconfig            | 1 -
 arch/arm64/kvm/mmu.c              | 6 +++---
 3 files changed, 5 insertions(+), 4 deletions(-)
arch/arm64/include/asm/kvm_host.h +2 -0
index 8b6096753740c..20f2ba149c70c 100644
@@ -1111,6 +1111,8 @@ int __init kvm_set_ipa_limit(void);
 #define __KVM_HAVE_ARCH_VM_ALLOC
 struct kvm *kvm_arch_alloc_vm(void);
 
+#define __KVM_HAVE_ARCH_FLUSH_REMOTE_TLBS
+
 static inline bool kvm_vm_is_protected(struct kvm *kvm)
 {
 	return false;
arch/arm64/kvm/Kconfig +0 -1
index f531da6b362e9..6b730fcfee379 100644
@@ -25,7 +25,6 @@ menuconfig KVM
 	select MMU_NOTIFIER
 	select PREEMPT_NOTIFIERS
 	select HAVE_KVM_CPU_RELAX_INTERCEPT
-	select HAVE_KVM_ARCH_TLB_FLUSH_ALL
 	select KVM_MMIO
 	select KVM_GENERIC_DIRTYLOG_READ_PROTECT
 	select KVM_XFER_TO_GUEST_WORK
arch/arm64/kvm/mmu.c +3 -3
index 6db9ef288ec38..0ac721fa27f18 100644
@@ -161,15 +161,15 @@ static bool memslot_is_logging(struct kvm_memory_slot *memslot)
 }
 
 /**
- * kvm_flush_remote_tlbs() - flush all VM TLB entries for v7/8
+ * kvm_arch_flush_remote_tlbs() - flush all VM TLB entries for v7/8
  * @kvm:	pointer to kvm structure.
  *
  * Interface to HYP function to flush all VM TLB entries
  */
-void kvm_flush_remote_tlbs(struct kvm *kvm)
+int kvm_arch_flush_remote_tlbs(struct kvm *kvm)
 {
-	++kvm->stat.generic.remote_tlb_flush_requests;
 	kvm_call_hyp(__kvm_tlb_flush_vmid, &kvm->arch.mmu);
+	return 0;
 }
 
 static bool kvm_is_device_pfn(unsigned long pfn)
Signature · 1 line
2.41.0.640.ga95def55d0-goog
Reviewed-by on v8 3/14
#13 lore raw

Re: [PATCH v8 03/14] KVM: arm64: Use kvm_arch_flush_remote_tlbs()

Quoted 12 lines · Raghavendra Rao Ananta

Stop depending on CONFIG_HAVE_KVM_ARCH_TLB_FLUSH_ALL and opt to standardize on kvm_arch_flush_remote_tlbs() since it avoids duplicating the generic TLB stats across architectures that implement their own remote TLB flush.

This adds an extra function call to the ARM64 kvm_flush_remote_tlbs() path, but that is a small cost in comparison to flushing remote TLBs.

In addition, instead of just incrementing remote_tlb_flush_requests stat, the generic interface would also increment the remote_tlb_flush stat.

Quoted 8 lines
---
  arch/arm64/include/asm/kvm_host.h | 2 ++
  arch/arm64/kvm/Kconfig            | 1 -
  arch/arm64/kvm/mmu.c              | 6 +++---
  3 files changed, 5 insertions(+), 4 deletions(-)
Quoted 48 lines
arch/arm64/include/asm/kvm_host.h +2 -0
index 8b6096753740c..20f2ba149c70c 100644
@@ -1111,6 +1111,8 @@ int __init kvm_set_ipa_limit(void);
  #define __KVM_HAVE_ARCH_VM_ALLOC
  struct kvm *kvm_arch_alloc_vm(void);
 
+#define __KVM_HAVE_ARCH_FLUSH_REMOTE_TLBS
+
  static inline bool kvm_vm_is_protected(struct kvm *kvm)
  {
  	return false;
arch/arm64/kvm/Kconfig +0 -1
index f531da6b362e9..6b730fcfee379 100644
@@ -25,7 +25,6 @@ menuconfig KVM
  	select MMU_NOTIFIER
  	select PREEMPT_NOTIFIERS
  	select HAVE_KVM_CPU_RELAX_INTERCEPT
-	select HAVE_KVM_ARCH_TLB_FLUSH_ALL
  	select KVM_MMIO
  	select KVM_GENERIC_DIRTYLOG_READ_PROTECT
  	select KVM_XFER_TO_GUEST_WORK
arch/arm64/kvm/mmu.c +3 -3
index 6db9ef288ec38..0ac721fa27f18 100644
@@ -161,15 +161,15 @@ static bool memslot_is_logging(struct kvm_memory_slot *memslot)
  }
 
  /**
- * kvm_flush_remote_tlbs() - flush all VM TLB entries for v7/8
+ * kvm_arch_flush_remote_tlbs() - flush all VM TLB entries for v7/8
   * @kvm:	pointer to kvm structure.
   *
   * Interface to HYP function to flush all VM TLB entries
   */
-void kvm_flush_remote_tlbs(struct kvm *kvm)
+int kvm_arch_flush_remote_tlbs(struct kvm *kvm)
  {
-	++kvm->stat.generic.remote_tlb_flush_requests;
  	kvm_call_hyp(__kvm_tlb_flush_vmid, &kvm->arch.mmu);
+	return 0;
  }
 
  static bool kvm_is_device_pfn(unsigned long pfn)
#14 lore raw

PATCH v8 4/14 KVM: Remove CONFIG_HAVE_KVM_ARCH_TLB_FLUSH_ALL

In reply to Raghavendra Rao Ananta

kvm_arch_flush_remote_tlbs() or CONFIG_HAVE_KVM_ARCH_TLB_FLUSH_ALL are two mechanisms to solve the same problem, allowing architecture-specific code to provide a non-IPI implementation of remote TLB flushing.

Dropping CONFIG_HAVE_KVM_ARCH_TLB_FLUSH_ALL allows KVM to standardize all architectures on kvm_arch_flush_remote_tlbs() instead of maintaining two mechanisms.

---
 virt/kvm/Kconfig    | 3 ---
 virt/kvm/kvm_main.c | 2 --
 2 files changed, 5 deletions(-)
virt/kvm/Kconfig +0 -3
index b74916de5183a..484d0873061ca 100644
@@ -62,9 +62,6 @@ config HAVE_KVM_CPU_RELAX_INTERCEPT
 config KVM_VFIO
        bool
 
-config HAVE_KVM_ARCH_TLB_FLUSH_ALL
-       bool
-
 config HAVE_KVM_INVALID_WAKEUPS
        bool
 
virt/kvm/kvm_main.c +0 -2
index 70e5479797ac3..d6b0507861550 100644
@@ -345,7 +345,6 @@ bool kvm_make_all_cpus_request(struct kvm *kvm, unsigned int req)
 }
 EXPORT_SYMBOL_GPL(kvm_make_all_cpus_request);
 
-#ifndef CONFIG_HAVE_KVM_ARCH_TLB_FLUSH_ALL
 void kvm_flush_remote_tlbs(struct kvm *kvm)
 {
 	++kvm->stat.generic.remote_tlb_flush_requests;
@@ -366,7 +365,6 @@ void kvm_flush_remote_tlbs(struct kvm *kvm)
 		++kvm->stat.generic.remote_tlb_flush;
 }
 EXPORT_SYMBOL_GPL(kvm_flush_remote_tlbs);
-#endif
 
 static void kvm_flush_shadow_all(struct kvm *kvm)
 {
Signature · 1 line
2.41.0.640.ga95def55d0-goog
Reviewed-by on v8 4/14
#15 lore raw

Re: [PATCH v8 04/14] KVM: Remove CONFIG_HAVE_KVM_ARCH_TLB_FLUSH_ALL

Quoted 9 lines · Raghavendra Rao Ananta

kvm_arch_flush_remote_tlbs() or CONFIG_HAVE_KVM_ARCH_TLB_FLUSH_ALL are two mechanisms to solve the same problem, allowing architecture-specific code to provide a non-IPI implementation of remote TLB flushing.

Dropping CONFIG_HAVE_KVM_ARCH_TLB_FLUSH_ALL allows KVM to standardize all architectures on kvm_arch_flush_remote_tlbs() instead of maintaining two mechanisms.

Quoted 7 lines
---
  virt/kvm/Kconfig    | 3 ---
  virt/kvm/kvm_main.c | 2 --
  2 files changed, 5 deletions(-)
Quoted 34 lines
virt/kvm/Kconfig +0 -3
index b74916de5183a..484d0873061ca 100644
@@ -62,9 +62,6 @@ config HAVE_KVM_CPU_RELAX_INTERCEPT
  config KVM_VFIO
         bool
 
-config HAVE_KVM_ARCH_TLB_FLUSH_ALL
-       bool
-
  config HAVE_KVM_INVALID_WAKEUPS
         bool
 
virt/kvm/kvm_main.c +0 -2
index 70e5479797ac3..d6b0507861550 100644
@@ -345,7 +345,6 @@ bool kvm_make_all_cpus_request(struct kvm *kvm, unsigned int req)
  }
  EXPORT_SYMBOL_GPL(kvm_make_all_cpus_request);
 
-#ifndef CONFIG_HAVE_KVM_ARCH_TLB_FLUSH_ALL
  void kvm_flush_remote_tlbs(struct kvm *kvm)
  {
  	++kvm->stat.generic.remote_tlb_flush_requests;
@@ -366,7 +365,6 @@ void kvm_flush_remote_tlbs(struct kvm *kvm)
  		++kvm->stat.generic.remote_tlb_flush;
  }
  EXPORT_SYMBOL_GPL(kvm_flush_remote_tlbs);
-#endif
 
  static void kvm_flush_shadow_all(struct kvm *kvm)
  {
Reviewed-by on v8 4/14
#16 lore raw

Re: [PATCH v8 04/14] KVM: Remove CONFIG_HAVE_KVM_ARCH_TLB_FLUSH_ALL

In reply to Raghavendra Rao Ananta
Quoted 9 lines · Raghavendra Rao Ananta

kvm_arch_flush_remote_tlbs() or CONFIG_HAVE_KVM_ARCH_TLB_FLUSH_ALL are two mechanisms to solve the same problem, allowing architecture-specific code to provide a non-IPI implementation of remote TLB flushing.

Dropping CONFIG_HAVE_KVM_ARCH_TLB_FLUSH_ALL allows KVM to standardize all architectures on kvm_arch_flush_remote_tlbs() instead of maintaining two mechanisms.

Quoted 6 lines
---
  virt/kvm/Kconfig    | 3 ---
  virt/kvm/kvm_main.c | 2 --
  2 files changed, 5 deletions(-)
#17 lore raw

PATCH v8 5/14 KVM: Allow range-based TLB invalidation from common code

In reply to Raghavendra Rao Ananta

From: David Matlack <[email protected]>

Make kvm_flush_remote_tlbs_range() visible in common code and create a default implementation that just invalidates the whole TLB.

This paves the way for several future features/cleanups:

- Introduction of range-based TLBI on ARM.
- Eliminating kvm_arch_flush_remote_tlbs_memslot()
- Moving the KVM/x86 TDP MMU to common code.

No functional change intended.

---
 arch/x86/include/asm/kvm_host.h |  2 ++
 arch/x86/kvm/mmu/mmu.c          |  8 ++++----
 arch/x86/kvm/mmu/mmu_internal.h |  3 ---
 include/linux/kvm_host.h        | 12 ++++++++++++
 virt/kvm/kvm_main.c             | 13 +++++++++++++
 5 files changed, 31 insertions(+), 7 deletions(-)
arch/x86/include/asm/kvm_host.h +2 -0
index a2d3cfc2eb75c..b547d17c58f63 100644
@@ -1804,6 +1804,8 @@ static inline int kvm_arch_flush_remote_tlbs(struct kvm *kvm)
 		return -ENOTSUPP;
 }
 
+#define __KVM_HAVE_ARCH_FLUSH_REMOTE_TLBS_RANGE
+
 #define kvm_arch_pmi_in_guest(vcpu) \
 	((vcpu) && (vcpu)->arch.handling_intr_from_guest)
 
arch/x86/kvm/mmu/mmu.c +4 -4
index ec169f5c7dce2..6adbe6c870982 100644
@@ -278,16 +278,16 @@ static inline bool kvm_available_flush_remote_tlbs_range(void)
 	return kvm_x86_ops.flush_remote_tlbs_range;
 }
 
-void kvm_flush_remote_tlbs_range(struct kvm *kvm, gfn_t start_gfn,
-				 gfn_t nr_pages)
+int kvm_arch_flush_remote_tlbs_range(struct kvm *kvm, gfn_t start_gfn,
+				      u64 nr_pages)
 {
 	int ret = -EOPNOTSUPP;
 
 	if (kvm_x86_ops.flush_remote_tlbs_range)
 		ret = static_call(kvm_x86_flush_remote_tlbs_range)(kvm, start_gfn,
 								   nr_pages);
-	if (ret)
-		kvm_flush_remote_tlbs(kvm);
+
+	return ret;
 }
 
 static gfn_t kvm_mmu_page_get_gfn(struct kvm_mmu_page *sp, int index);
arch/x86/kvm/mmu/mmu_internal.h +0 -3
index d39af5639ce97..86cb83bb34804 100644
@@ -170,9 +170,6 @@ bool kvm_mmu_slot_gfn_write_protect(struct kvm *kvm,
 				    struct kvm_memory_slot *slot, u64 gfn,
 				    int min_level);
 
-void kvm_flush_remote_tlbs_range(struct kvm *kvm, gfn_t start_gfn,
-				 gfn_t nr_pages);
-
 /* Flush the given page (huge or not) of guest memory. */
 static inline void kvm_flush_remote_tlbs_gfn(struct kvm *kvm, gfn_t gfn, int level)
 {
include/linux/kvm_host.h +12 -0
index ade5d4500c2ce..f0be5d9c37dd1 100644
@@ -1359,6 +1359,7 @@ int kvm_vcpu_yield_to(struct kvm_vcpu *target);
 void kvm_vcpu_on_spin(struct kvm_vcpu *vcpu, bool yield_to_kernel_mode);
 
 void kvm_flush_remote_tlbs(struct kvm *kvm);
+void kvm_flush_remote_tlbs_range(struct kvm *kvm, gfn_t gfn, u64 nr_pages);
 
 #ifdef KVM_ARCH_NR_OBJS_PER_MEMORY_CACHE
 int kvm_mmu_topup_memory_cache(struct kvm_mmu_memory_cache *mc, int min);
@@ -1488,6 +1489,17 @@ static inline int kvm_arch_flush_remote_tlbs(struct kvm *kvm)
 int kvm_arch_flush_remote_tlbs(struct kvm *kvm);
 #endif
 
+#ifndef __KVM_HAVE_ARCH_FLUSH_REMOTE_TLBS_RANGE
+static inline int kvm_arch_flush_remote_tlbs_range(struct kvm *kvm,
+						    gfn_t gfn, u64 nr_pages)
+{
+	return -EOPNOTSUPP;
+}
+#else
+int kvm_arch_flush_remote_tlbs_range(struct kvm *kvm,
+				      gfn_t gfn, u64 nr_pages);
+#endif
+
 #ifdef __KVM_HAVE_ARCH_NONCOHERENT_DMA
 void kvm_arch_register_noncoherent_dma(struct kvm *kvm);
 void kvm_arch_unregister_noncoherent_dma(struct kvm *kvm);
virt/kvm/kvm_main.c +13 -0
index d6b0507861550..26e91000f579d 100644
@@ -366,6 +366,19 @@ void kvm_flush_remote_tlbs(struct kvm *kvm)
 }
 EXPORT_SYMBOL_GPL(kvm_flush_remote_tlbs);
 
+void kvm_flush_remote_tlbs_range(struct kvm *kvm, gfn_t gfn, u64 nr_pages)
+{
+	if (!kvm_arch_flush_remote_tlbs_range(kvm, gfn, nr_pages))
+		return;
+
+	/*
+	 * Fall back to a flushing entire TLBs if the architecture range-based
+	 * TLB invalidation is unsupported or can't be performed for whatever
+	 * reason.
+	 */
+	kvm_flush_remote_tlbs(kvm);
+}
+
 static void kvm_flush_shadow_all(struct kvm *kvm)
 {
 	kvm_arch_flush_shadow_all(kvm);
Signature · 1 line
2.41.0.640.ga95def55d0-goog
#18 lore raw

Re: [PATCH v8 05/14] KVM: Allow range-based TLB invalidation from common code

Quoted 53 lines · Raghavendra Rao Ananta

From: David Matlack <[email protected]>

Make kvm_flush_remote_tlbs_range() visible in common code and create a default implementation that just invalidates the whole TLB.

This paves the way for several future features/cleanups:

- Introduction of range-based TLBI on ARM.
- Eliminating kvm_arch_flush_remote_tlbs_memslot()
- Moving the KVM/x86 TDP MMU to common code.

No functional change intended.

---
  arch/x86/include/asm/kvm_host.h |  2 ++
  arch/x86/kvm/mmu/mmu.c          |  8 ++++----
  arch/x86/kvm/mmu/mmu_internal.h |  3 ---
  include/linux/kvm_host.h        | 12 ++++++++++++
  virt/kvm/kvm_main.c             | 13 +++++++++++++
  5 files changed, 31 insertions(+), 7 deletions(-)
arch/x86/include/asm/kvm_host.h +2 -0
index a2d3cfc2eb75c..b547d17c58f63 100644
@@ -1804,6 +1804,8 @@ static inline int kvm_arch_flush_remote_tlbs(struct kvm *kvm)
  		return -ENOTSUPP;
  }
 
+#define __KVM_HAVE_ARCH_FLUSH_REMOTE_TLBS_RANGE
+
  #define kvm_arch_pmi_in_guest(vcpu) \
  	((vcpu) && (vcpu)->arch.handling_intr_from_guest)
 
arch/x86/kvm/mmu/mmu.c +2 -2
index ec169f5c7dce2..6adbe6c870982 100644
@@ -278,16 +278,16 @@ static inline bool kvm_available_flush_remote_tlbs_range(void)
  	return kvm_x86_ops.flush_remote_tlbs_range;
  }
 
-void kvm_flush_remote_tlbs_range(struct kvm *kvm, gfn_t start_gfn,
-				 gfn_t nr_pages)
+int kvm_arch_flush_remote_tlbs_range(struct kvm *kvm, gfn_t start_gfn,
+				      u64 nr_pages)
  {
  	int ret = -EOPNOTSUPP;

arch/x86/kvm/mmu/mmu.c static inline bool kvm_available_flush_remote_tlbs_range(void) · line 292

Quoted 10 lines
arch/x86/kvm/mmu/mmu.c +2 -2
 
  	if (kvm_x86_ops.flush_remote_tlbs_range)
  		ret = static_call(kvm_x86_flush_remote_tlbs_range)(kvm, start_gfn,
  								   nr_pages);
-	if (ret)
-		kvm_flush_remote_tlbs(kvm);
+
+	return ret;
  }
 

I guess @start_gfn can be renamed to @gfn, to be consistent with its declaration in include/linux/kvm_host.h and struct kvm_x86_ops::flush_remote_tlbs_range()

Quoted 59 lines
arch/x86/kvm/mmu/mmu.c +0 -0
  static gfn_t kvm_mmu_page_get_gfn(struct kvm_mmu_page *sp, int index);
arch/x86/kvm/mmu/mmu_internal.h +0 -3
index d39af5639ce97..86cb83bb34804 100644
@@ -170,9 +170,6 @@ bool kvm_mmu_slot_gfn_write_protect(struct kvm *kvm,
  				    struct kvm_memory_slot *slot, u64 gfn,
  				    int min_level);
 
-void kvm_flush_remote_tlbs_range(struct kvm *kvm, gfn_t start_gfn,
-				 gfn_t nr_pages);
-
  /* Flush the given page (huge or not) of guest memory. */
  static inline void kvm_flush_remote_tlbs_gfn(struct kvm *kvm, gfn_t gfn, int level)
  {
include/linux/kvm_host.h +12 -0
index ade5d4500c2ce..f0be5d9c37dd1 100644
@@ -1359,6 +1359,7 @@ int kvm_vcpu_yield_to(struct kvm_vcpu *target);
  void kvm_vcpu_on_spin(struct kvm_vcpu *vcpu, bool yield_to_kernel_mode);
 
  void kvm_flush_remote_tlbs(struct kvm *kvm);
+void kvm_flush_remote_tlbs_range(struct kvm *kvm, gfn_t gfn, u64 nr_pages);
 
  #ifdef KVM_ARCH_NR_OBJS_PER_MEMORY_CACHE
  int kvm_mmu_topup_memory_cache(struct kvm_mmu_memory_cache *mc, int min);
@@ -1488,6 +1489,17 @@ static inline int kvm_arch_flush_remote_tlbs(struct kvm *kvm)
  int kvm_arch_flush_remote_tlbs(struct kvm *kvm);
  #endif
 
+#ifndef __KVM_HAVE_ARCH_FLUSH_REMOTE_TLBS_RANGE
+static inline int kvm_arch_flush_remote_tlbs_range(struct kvm *kvm,
+						    gfn_t gfn, u64 nr_pages)
+{
+	return -EOPNOTSUPP;
+}
+#else
+int kvm_arch_flush_remote_tlbs_range(struct kvm *kvm,
+				      gfn_t gfn, u64 nr_pages);
+#endif
+
  #ifdef __KVM_HAVE_ARCH_NONCOHERENT_DMA
  void kvm_arch_register_noncoherent_dma(struct kvm *kvm);
  void kvm_arch_unregister_noncoherent_dma(struct kvm *kvm);
virt/kvm/kvm_main.c +6 -0
index d6b0507861550..26e91000f579d 100644
@@ -366,6 +366,19 @@ void kvm_flush_remote_tlbs(struct kvm *kvm)
  }
  EXPORT_SYMBOL_GPL(kvm_flush_remote_tlbs);
 
+void kvm_flush_remote_tlbs_range(struct kvm *kvm, gfn_t gfn, u64 nr_pages)
+{
+	if (!kvm_arch_flush_remote_tlbs_range(kvm, gfn, nr_pages))
+		return;
+
+	/*

virt/kvm/kvm_main.c void kvm_flush_remote_tlbs(struct kvm *kvm) · line 384

Quoted 10 lines
virt/kvm/kvm_main.c +7 -0
+	 * Fall back to a flushing entire TLBs if the architecture range-based
+	 * TLB invalidation is unsupported or can't be performed for whatever
+	 * reason.
+	 */
+	kvm_flush_remote_tlbs(kvm);
+}
+
  static void kvm_flush_shadow_all(struct kvm *kvm)
  {
  	kvm_arch_flush_shadow_all(kvm);

Thanks,
Gavin

#19 lore raw

Re: [PATCH v8 05/14] KVM: Allow range-based TLB invalidation from common code

Quoted 56 lines · On Tue, Aug 8, 2023 at 11:09?PM Gavin Shan <[email protected]> wrote:
Quoted 54 lines · Raghavendra Rao Ananta

From: David Matlack <[email protected]>

Make kvm_flush_remote_tlbs_range() visible in common code and create a default implementation that just invalidates the whole TLB.

This paves the way for several future features/cleanups:

- Introduction of range-based TLBI on ARM.
- Eliminating kvm_arch_flush_remote_tlbs_memslot()
- Moving the KVM/x86 TDP MMU to common code.

No functional change intended.

---
  arch/x86/include/asm/kvm_host.h |  2 ++
  arch/x86/kvm/mmu/mmu.c          |  8 ++++----
  arch/x86/kvm/mmu/mmu_internal.h |  3 ---
  include/linux/kvm_host.h        | 12 ++++++++++++
  virt/kvm/kvm_main.c             | 13 +++++++++++++
  5 files changed, 31 insertions(+), 7 deletions(-)
arch/x86/include/asm/kvm_host.h +2 -0
index a2d3cfc2eb75c..b547d17c58f63 100644
@@ -1804,6 +1804,8 @@ static inline int kvm_arch_flush_remote_tlbs(struct kvm *kvm)
              return -ENOTSUPP;
  }
 
+#define __KVM_HAVE_ARCH_FLUSH_REMOTE_TLBS_RANGE
+
  #define kvm_arch_pmi_in_guest(vcpu) \
      ((vcpu) && (vcpu)->arch.handling_intr_from_guest)
 
arch/x86/kvm/mmu/mmu.c +2 -2
index ec169f5c7dce2..6adbe6c870982 100644
@@ -278,16 +278,16 @@ static inline bool kvm_available_flush_remote_tlbs_range(void)
      return kvm_x86_ops.flush_remote_tlbs_range;
  }
 
-void kvm_flush_remote_tlbs_range(struct kvm *kvm, gfn_t start_gfn,
-                              gfn_t nr_pages)
+int kvm_arch_flush_remote_tlbs_range(struct kvm *kvm, gfn_t start_gfn,
+                                   u64 nr_pages)
  {
      int ret = -EOPNOTSUPP;
 
Quoted 13 lines

arch/x86/kvm/mmu/mmu.c static inline bool kvm_available_flush_remote_tlbs_range(void) · line 292

Quoted 9 lines
arch/x86/kvm/mmu/mmu.c +2 -2
      if (kvm_x86_ops.flush_remote_tlbs_range)
              ret = static_call(kvm_x86_flush_remote_tlbs_range)(kvm, start_gfn,
                                                                 nr_pages);
-     if (ret)
-             kvm_flush_remote_tlbs(kvm);
+
+     return ret;
  }
 

I guess @start_gfn can be renamed to @gfn, to be consistent with its declaration in include/linux/kvm_host.h and struct kvm_x86_ops::flush_remote_tlbs_range()

Oh, yes. I'll consider this change for v9 (if there are enough changes and we plan to push one).

Thank you.
Raghavendra

Quoted 73 lines
Quoted 59 lines
arch/x86/kvm/mmu/mmu.c +0 -0
  static gfn_t kvm_mmu_page_get_gfn(struct kvm_mmu_page *sp, int index);
arch/x86/kvm/mmu/mmu_internal.h +0 -3
index d39af5639ce97..86cb83bb34804 100644
@@ -170,9 +170,6 @@ bool kvm_mmu_slot_gfn_write_protect(struct kvm *kvm,
                                  struct kvm_memory_slot *slot, u64 gfn,
                                  int min_level);
 
-void kvm_flush_remote_tlbs_range(struct kvm *kvm, gfn_t start_gfn,
-                              gfn_t nr_pages);
-
  /* Flush the given page (huge or not) of guest memory. */
  static inline void kvm_flush_remote_tlbs_gfn(struct kvm *kvm, gfn_t gfn, int level)
  {
include/linux/kvm_host.h +12 -0
index ade5d4500c2ce..f0be5d9c37dd1 100644
@@ -1359,6 +1359,7 @@ int kvm_vcpu_yield_to(struct kvm_vcpu *target);
  void kvm_vcpu_on_spin(struct kvm_vcpu *vcpu, bool yield_to_kernel_mode);
 
  void kvm_flush_remote_tlbs(struct kvm *kvm);
+void kvm_flush_remote_tlbs_range(struct kvm *kvm, gfn_t gfn, u64 nr_pages);
 
  #ifdef KVM_ARCH_NR_OBJS_PER_MEMORY_CACHE
  int kvm_mmu_topup_memory_cache(struct kvm_mmu_memory_cache *mc, int min);
@@ -1488,6 +1489,17 @@ static inline int kvm_arch_flush_remote_tlbs(struct kvm *kvm)
  int kvm_arch_flush_remote_tlbs(struct kvm *kvm);
  #endif
 
+#ifndef __KVM_HAVE_ARCH_FLUSH_REMOTE_TLBS_RANGE
+static inline int kvm_arch_flush_remote_tlbs_range(struct kvm *kvm,
+                                                 gfn_t gfn, u64 nr_pages)
+{
+     return -EOPNOTSUPP;
+}
+#else
+int kvm_arch_flush_remote_tlbs_range(struct kvm *kvm,
+                                   gfn_t gfn, u64 nr_pages);
+#endif
+
  #ifdef __KVM_HAVE_ARCH_NONCOHERENT_DMA
  void kvm_arch_register_noncoherent_dma(struct kvm *kvm);
  void kvm_arch_unregister_noncoherent_dma(struct kvm *kvm);
virt/kvm/kvm_main.c +6 -0
index d6b0507861550..26e91000f579d 100644
@@ -366,6 +366,19 @@ void kvm_flush_remote_tlbs(struct kvm *kvm)
  }
  EXPORT_SYMBOL_GPL(kvm_flush_remote_tlbs);
 
+void kvm_flush_remote_tlbs_range(struct kvm *kvm, gfn_t gfn, u64 nr_pages)
+{
+     if (!kvm_arch_flush_remote_tlbs_range(kvm, gfn, nr_pages))
+             return;
+
+     /*

virt/kvm/kvm_main.c void kvm_flush_remote_tlbs(struct kvm *kvm) · line 384

Quoted 10 lines
virt/kvm/kvm_main.c +7 -0
+      * Fall back to a flushing entire TLBs if the architecture range-based
+      * TLB invalidation is unsupported or can't be performed for whatever
+      * reason.
+      */
+     kvm_flush_remote_tlbs(kvm);
+}
+
  static void kvm_flush_shadow_all(struct kvm *kvm)
  {
      kvm_arch_flush_shadow_all(kvm);

Thanks,
Gavin

#20 lore raw

PATCH v8 6/14 KVM: Move kvm_arch_flush_remote_tlbs_memslot() to common code

In reply to Raghavendra Rao Ananta

From: David Matlack <[email protected]>

Move kvm_arch_flush_remote_tlbs_memslot() to common code and drop "arch_" from the name. kvm_arch_flush_remote_tlbs_memslot() is just a range-based TLB invalidation where the range is defined by the memslot. Now that kvm_flush_remote_tlbs_range() can be called from common code we can just use that and drop a bunch of duplicate code from the arch directories.

Note this adds a lockdep assertion for slots_lock being held when calling kvm_flush_remote_tlbs_memslot(), which was previously only asserted on x86. MIPS has calls to kvm_flush_remote_tlbs_memslot(), but they all hold the slots_lock, so the lockdep assertion continues to hold true.

Also drop the CONFIG_KVM_GENERIC_DIRTYLOG_READ_PROTECT ifdef gating kvm_flush_remote_tlbs_memslot(), since it is no longer necessary.

---
 arch/arm64/kvm/arm.c     |  6 ------
 arch/mips/kvm/mips.c     | 10 ++--------
 arch/riscv/kvm/mmu.c     |  6 ------
 arch/x86/kvm/mmu/mmu.c   | 16 +---------------
 arch/x86/kvm/x86.c       |  2 +-
 include/linux/kvm_host.h |  7 +++----
 virt/kvm/kvm_main.c      | 18 ++++++++++++++++--
 7 files changed, 23 insertions(+), 42 deletions(-)
arch/arm64/kvm/arm.c +0 -6
index c2c14059f6a8c..ed7bef4d970b9 100644
@@ -1525,12 +1525,6 @@ void kvm_arch_sync_dirty_log(struct kvm *kvm, struct kvm_memory_slot *memslot)
 
 }
 
-void kvm_arch_flush_remote_tlbs_memslot(struct kvm *kvm,
-					const struct kvm_memory_slot *memslot)
-{
-	kvm_flush_remote_tlbs(kvm);
-}
-
 static int kvm_vm_ioctl_set_device_addr(struct kvm *kvm,
 					struct kvm_arm_device_addr *dev_addr)
 {
arch/mips/kvm/mips.c +2 -8
index 4b7bc39a41736..231ac052b506b 100644
@@ -199,7 +199,7 @@ void kvm_arch_flush_shadow_memslot(struct kvm *kvm,
 	/* Flush slot from GPA */
 	kvm_mips_flush_gpa_pt(kvm, slot->base_gfn,
 			      slot->base_gfn + slot->npages - 1);
-	kvm_arch_flush_remote_tlbs_memslot(kvm, slot);
+	kvm_flush_remote_tlbs_memslot(kvm, slot);
 	spin_unlock(&kvm->mmu_lock);
 }
 
@@ -235,7 +235,7 @@ void kvm_arch_commit_memory_region(struct kvm *kvm,
 		needs_flush = kvm_mips_mkclean_gpa_pt(kvm, new->base_gfn,
 					new->base_gfn + new->npages - 1);
 		if (needs_flush)
-			kvm_arch_flush_remote_tlbs_memslot(kvm, new);
+			kvm_flush_remote_tlbs_memslot(kvm, new);
 		spin_unlock(&kvm->mmu_lock);
 	}
 }
@@ -987,12 +987,6 @@ int kvm_arch_flush_remote_tlbs(struct kvm *kvm)
 	return 1;
 }
 
-void kvm_arch_flush_remote_tlbs_memslot(struct kvm *kvm,
-					const struct kvm_memory_slot *memslot)
-{
-	kvm_flush_remote_tlbs(kvm);
-}
-
 int kvm_arch_vm_ioctl(struct file *filp, unsigned int ioctl, unsigned long arg)
 {
 	int r;
arch/riscv/kvm/mmu.c +0 -6
index f2eb47925806b..97e129620686c 100644
@@ -406,12 +406,6 @@ void kvm_arch_sync_dirty_log(struct kvm *kvm, struct kvm_memory_slot *memslot)
 {
 }
 
-void kvm_arch_flush_remote_tlbs_memslot(struct kvm *kvm,
-					const struct kvm_memory_slot *memslot)
-{
-	kvm_flush_remote_tlbs(kvm);
-}
-
 void kvm_arch_free_memslot(struct kvm *kvm, struct kvm_memory_slot *free)
 {
 }
arch/x86/kvm/mmu/mmu.c +1 -15
index 6adbe6c870982..9e074b5f322de 100644
@@ -6670,7 +6670,7 @@ static void kvm_rmap_zap_collapsible_sptes(struct kvm *kvm,
 	 */
 	if (walk_slot_rmaps(kvm, slot, kvm_mmu_zap_collapsible_spte,
 			    PG_LEVEL_4K, KVM_MAX_HUGEPAGE_LEVEL - 1, true))
-		kvm_arch_flush_remote_tlbs_memslot(kvm, slot);
+		kvm_flush_remote_tlbs_memslot(kvm, slot);
 }
 
 void kvm_mmu_zap_collapsible_sptes(struct kvm *kvm,
@@ -6689,20 +6689,6 @@ void kvm_mmu_zap_collapsible_sptes(struct kvm *kvm,
 	}
 }
 
-void kvm_arch_flush_remote_tlbs_memslot(struct kvm *kvm,
-					const struct kvm_memory_slot *memslot)
-{
-	/*
-	 * All current use cases for flushing the TLBs for a specific memslot
-	 * related to dirty logging, and many do the TLB flush out of mmu_lock.
-	 * The interaction between the various operations on memslot must be
-	 * serialized by slots_locks to ensure the TLB flush from one operation
-	 * is observed by any other operation on the same memslot.
-	 */
-	lockdep_assert_held(&kvm->slots_lock);
-	kvm_flush_remote_tlbs_range(kvm, memslot->base_gfn, memslot->npages);
-}
-
 void kvm_mmu_slot_leaf_clear_dirty(struct kvm *kvm,
 				   const struct kvm_memory_slot *memslot)
 {
arch/x86/kvm/x86.c +1 -1
index a6b9bea62fb8a..faeb2e307b36a 100644
@@ -12751,7 +12751,7 @@ static void kvm_mmu_slot_apply_flags(struct kvm *kvm,
 		 * See is_writable_pte() for more details (the case involving
 		 * access-tracked SPTEs is particularly relevant).
 		 */
-		kvm_arch_flush_remote_tlbs_memslot(kvm, new);
+		kvm_flush_remote_tlbs_memslot(kvm, new);
 	}
 }
 
include/linux/kvm_host.h +3 -4
index f0be5d9c37dd1..49292befa97fb 100644
@@ -1360,6 +1360,8 @@ void kvm_vcpu_on_spin(struct kvm_vcpu *vcpu, bool yield_to_kernel_mode);
 
 void kvm_flush_remote_tlbs(struct kvm *kvm);
 void kvm_flush_remote_tlbs_range(struct kvm *kvm, gfn_t gfn, u64 nr_pages);
+void kvm_flush_remote_tlbs_memslot(struct kvm *kvm,
+				   const struct kvm_memory_slot *memslot);
 
 #ifdef KVM_ARCH_NR_OBJS_PER_MEMORY_CACHE
 int kvm_mmu_topup_memory_cache(struct kvm_mmu_memory_cache *mc, int min);
@@ -1388,10 +1390,7 @@ void kvm_arch_mmu_enable_log_dirty_pt_masked(struct kvm *kvm,
 					unsigned long mask);
 void kvm_arch_sync_dirty_log(struct kvm *kvm, struct kvm_memory_slot *memslot);
 
-#ifdef CONFIG_KVM_GENERIC_DIRTYLOG_READ_PROTECT
-void kvm_arch_flush_remote_tlbs_memslot(struct kvm *kvm,
-					const struct kvm_memory_slot *memslot);
-#else /* !CONFIG_KVM_GENERIC_DIRTYLOG_READ_PROTECT */
+#ifndef CONFIG_KVM_GENERIC_DIRTYLOG_READ_PROTECT
 int kvm_vm_ioctl_get_dirty_log(struct kvm *kvm, struct kvm_dirty_log *log);
 int kvm_get_dirty_log(struct kvm *kvm, struct kvm_dirty_log *log,
 		      int *is_dirty, struct kvm_memory_slot **memslot);
virt/kvm/kvm_main.c +16 -2
index 26e91000f579d..5d4d2e051aa09 100644
@@ -379,6 +379,20 @@ void kvm_flush_remote_tlbs_range(struct kvm *kvm, gfn_t gfn, u64 nr_pages)
 	kvm_flush_remote_tlbs(kvm);
 }
 
+void kvm_flush_remote_tlbs_memslot(struct kvm *kvm,
+				   const struct kvm_memory_slot *memslot)
+{
+	/*
+	 * All current use cases for flushing the TLBs for a specific memslot
+	 * are related to dirty logging, and many do the TLB flush out of
+	 * mmu_lock. The interaction between the various operations on memslot
+	 * must be serialized by slots_locks to ensure the TLB flush from one
+	 * operation is observed by any other operation on the same memslot.
+	 */
+	lockdep_assert_held(&kvm->slots_lock);
+	kvm_flush_remote_tlbs_range(kvm, memslot->base_gfn, memslot->npages);
+}
+
 static void kvm_flush_shadow_all(struct kvm *kvm)
 {
 	kvm_arch_flush_shadow_all(kvm);
@@ -2191,7 +2205,7 @@ static int kvm_get_dirty_log_protect(struct kvm *kvm, struct kvm_dirty_log *log)
 	}
 
 	if (flush)
-		kvm_arch_flush_remote_tlbs_memslot(kvm, memslot);
+		kvm_flush_remote_tlbs_memslot(kvm, memslot);
 
 	if (copy_to_user(log->dirty_bitmap, dirty_bitmap_buffer, n))
 		return -EFAULT;
@@ -2308,7 +2322,7 @@ static int kvm_clear_dirty_log_protect(struct kvm *kvm,
 	KVM_MMU_UNLOCK(kvm);
 
 	if (flush)
-		kvm_arch_flush_remote_tlbs_memslot(kvm, memslot);
+		kvm_flush_remote_tlbs_memslot(kvm, memslot);
 
 	return 0;
 }
Signature · 1 line
2.41.0.640.ga95def55d0-goog
#21 lore raw

PATCH v8 7/14 arm64: tlb: Refactor the core flush algorithm of __flush_tlb_range

In reply to Raghavendra Rao Ananta

Currently, the core TLB flush functionality of __flush_tlb_range() hardcodes vae1is (and variants) for the flush operation. In the upcoming patches, the KVM code reuses this core algorithm with ipas2e1is for range based TLB invalidations based on the IPA. Hence, extract the core flush functionality of __flush_tlb_range() into its own macro that accepts an 'op' argument to pass any TLBI operation, such that other callers (KVM) can benefit.

No functional changes intended.

---
 arch/arm64/include/asm/tlbflush.h | 121 +++++++++++++++++-------------
 1 file changed, 68 insertions(+), 53 deletions(-)
arch/arm64/include/asm/tlbflush.h +68 -53
index 412a3b9a3c25d..b9475a852d5be 100644
@@ -278,14 +278,74 @@ static inline void flush_tlb_page(struct vm_area_struct *vma,
  */
 #define MAX_TLBI_OPS	PTRS_PER_PTE
 
+/*
+ * __flush_tlb_range_op - Perform TLBI operation upon a range
+ *
+ * @op:	TLBI instruction that operates on a range (has 'r' prefix)
+ * @start:	The start address of the range
+ * @pages:	Range as the number of pages from 'start'
+ * @stride:	Flush granularity
+ * @asid:	The ASID of the task (0 for IPA instructions)
+ * @tlb_level:	Translation Table level hint, if known
+ * @tlbi_user:	If 'true', call an additional __tlbi_user()
+ *              (typically for user ASIDs). 'flase' for IPA instructions
+ *
+ * When the CPU does not support TLB range operations, flush the TLB
+ * entries one by one at the granularity of 'stride'. If the TLB
+ * range ops are supported, then:
+ *
+ * 1. If 'pages' is odd, flush the first page through non-range
+ *    operations;
+ *
+ * 2. For remaining pages: the minimum range granularity is decided
+ *    by 'scale', so multiple range TLBI operations may be required.
+ *    Start from scale = 0, flush the corresponding number of pages
+ *    ((num+1)*2^(5*scale+1) starting from 'addr'), then increase it
+ *    until no pages left.
+ *
+ * Note that certain ranges can be represented by either num = 31 and
+ * scale or num = 0 and scale + 1. The loop below favours the latter
+ * since num is limited to 30 by the __TLBI_RANGE_NUM() macro.
+ */
+#define __flush_tlb_range_op(op, start, pages, stride,			\
+				asid, tlb_level, tlbi_user)		\
+do {									\
+	int num = 0;							\
+	int scale = 0;							\
+	unsigned long addr;						\
+									\
+	while (pages > 0) {						\
+		if (!system_supports_tlb_range() ||			\
+		    pages % 2 == 1) {					\
+			addr = __TLBI_VADDR(start, asid);		\
+			__tlbi_level(op, addr, tlb_level);		\
+			if (tlbi_user)					\
+				__tlbi_user_level(op, addr, tlb_level);	\
+			start += stride;				\
+			pages -= stride >> PAGE_SHIFT;			\
+			continue;					\
+		}							\
+									\
+		num = __TLBI_RANGE_NUM(pages, scale);			\
+		if (num >= 0) {						\
+			addr = __TLBI_VADDR_RANGE(start, asid, scale,	\
+						  num, tlb_level);	\
+			__tlbi(r##op, addr);				\
+			if (tlbi_user)					\
+				__tlbi_user(r##op, addr);		\
+			start += __TLBI_RANGE_PAGES(num, scale) << PAGE_SHIFT; \
+			pages -= __TLBI_RANGE_PAGES(num, scale);	\
+		}							\
+		scale++;						\
+	}								\
+} while (0)
+
 static inline void __flush_tlb_range(struct vm_area_struct *vma,
 				     unsigned long start, unsigned long end,
 				     unsigned long stride, bool last_level,
 				     int tlb_level)
 {
-	int num = 0;
-	int scale = 0;
-	unsigned long asid, addr, pages;
+	unsigned long asid, pages;
 
 	start = round_down(start, stride);
 	end = round_up(end, stride);
@@ -307,56 +367,11 @@ static inline void __flush_tlb_range(struct vm_area_struct *vma,
 	dsb(ishst);
 	asid = ASID(vma->vm_mm);
 
-	/*
-	 * When the CPU does not support TLB range operations, flush the TLB
-	 * entries one by one at the granularity of 'stride'. If the TLB
-	 * range ops are supported, then:
-	 *
-	 * 1. If 'pages' is odd, flush the first page through non-range
-	 *    operations;
-	 *
-	 * 2. For remaining pages: the minimum range granularity is decided
-	 *    by 'scale', so multiple range TLBI operations may be required.
-	 *    Start from scale = 0, flush the corresponding number of pages
-	 *    ((num+1)*2^(5*scale+1) starting from 'addr'), then increase it
-	 *    until no pages left.
-	 *
-	 * Note that certain ranges can be represented by either num = 31 and
-	 * scale or num = 0 and scale + 1. The loop below favours the latter
-	 * since num is limited to 30 by the __TLBI_RANGE_NUM() macro.
-	 */
-	while (pages > 0) {
-		if (!system_supports_tlb_range() ||
-		    pages % 2 == 1) {
-			addr = __TLBI_VADDR(start, asid);
-			if (last_level) {
-				__tlbi_level(vale1is, addr, tlb_level);
-				__tlbi_user_level(vale1is, addr, tlb_level);
-			} else {
-				__tlbi_level(vae1is, addr, tlb_level);
-				__tlbi_user_level(vae1is, addr, tlb_level);
-			}
-			start += stride;
-			pages -= stride >> PAGE_SHIFT;
-			continue;
-		}
-
-		num = __TLBI_RANGE_NUM(pages, scale);
-		if (num >= 0) {
-			addr = __TLBI_VADDR_RANGE(start, asid, scale,
-						  num, tlb_level);
-			if (last_level) {
-				__tlbi(rvale1is, addr);
-				__tlbi_user(rvale1is, addr);
-			} else {
-				__tlbi(rvae1is, addr);
-				__tlbi_user(rvae1is, addr);
-			}
-			start += __TLBI_RANGE_PAGES(num, scale) << PAGE_SHIFT;
-			pages -= __TLBI_RANGE_PAGES(num, scale);
-		}
-		scale++;
-	}
+	if (last_level)
+		__flush_tlb_range_op(vale1is, start, pages, stride, asid, tlb_level, true);
+	else
+		__flush_tlb_range_op(vae1is, start, pages, stride, asid, tlb_level, true);
+
 	dsb(ish);
 }
 
Signature · 1 line
2.41.0.640.ga95def55d0-goog
#22 lore raw

PATCH v8 8/14 arm64: tlb: Implement __flush_s2_tlb_range_op()

In reply to Raghavendra Rao Ananta

Define __flush_s2_tlb_range_op(), as a wrapper over __flush_tlb_range_op(), for stage-2 specific range-based TLBI operations that doesn't necessarily have to deal with 'asid' and 'tlbi_user' arguments.

---
 arch/arm64/include/asm/tlbflush.h | 3 +++
 1 file changed, 3 insertions(+)
arch/arm64/include/asm/tlbflush.h +3 -0
index b9475a852d5be..93f4b397f9a12 100644
@@ -340,6 +340,9 @@ do {									\
 	}								\
 } while (0)
 
+#define __flush_s2_tlb_range_op(op, start, pages, stride, tlb_level) \
+	__flush_tlb_range_op(op, start, pages, stride, 0, tlb_level, false)
+
 static inline void __flush_tlb_range(struct vm_area_struct *vma,
 				     unsigned long start, unsigned long end,
 				     unsigned long stride, bool last_level,
Signature · 1 line
2.41.0.640.ga95def55d0-goog
Reviewed-by on v8 8/14
#23 lore raw

Re: [PATCH v8 08/14] arm64: tlb: Implement __flush_s2_tlb_range_op()

Quoted 6 lines · Raghavendra Rao Ananta

Define __flush_s2_tlb_range_op(), as a wrapper over __flush_tlb_range_op(), for stage-2 specific range-based TLBI operations that doesn't necessarily have to deal with 'asid' and 'tlbi_user' arguments.

Quoted 18 lines
---
  arch/arm64/include/asm/tlbflush.h | 3 +++
  1 file changed, 3 insertions(+)
arch/arm64/include/asm/tlbflush.h +3 -0
index b9475a852d5be..93f4b397f9a12 100644
@@ -340,6 +340,9 @@ do {									\
  	}								\
  } while (0)
 
+#define __flush_s2_tlb_range_op(op, start, pages, stride, tlb_level) \
+	__flush_tlb_range_op(op, start, pages, stride, 0, tlb_level, false)
+
  static inline void __flush_tlb_range(struct vm_area_struct *vma,
  				     unsigned long start, unsigned long end,
  				     unsigned long stride, bool last_level,
Signature · 1 line
Shaoqin
#24 lore raw

PATCH v8 9/14 KVM: arm64: Implement __kvm_tlb_flush_vmid_range()

In reply to Raghavendra Rao Ananta

Define __kvm_tlb_flush_vmid_range() (for VHE and nVHE) to flush a range of stage-2 page-tables using IPA in one go. If the system supports FEAT_TLBIRANGE, the following patches would conviniently replace global TLBI such as vmalls12e1is in the map, unmap, and dirty-logging paths with ripas2e1is instead.

---
 arch/arm64/include/asm/kvm_asm.h   |  3 +++
 arch/arm64/kvm/hyp/nvhe/hyp-main.c | 11 +++++++++++
 arch/arm64/kvm/hyp/nvhe/tlb.c      | 30 ++++++++++++++++++++++++++++++
 arch/arm64/kvm/hyp/vhe/tlb.c       | 28 ++++++++++++++++++++++++++++
 4 files changed, 72 insertions(+)
arch/arm64/include/asm/kvm_asm.h +3 -0
index 7d170aaa2db41..2c27cb8cf442d 100644
@@ -70,6 +70,7 @@ enum __kvm_host_smccc_func {
 	__KVM_HOST_SMCCC_FUNC___kvm_tlb_flush_vmid_ipa,
 	__KVM_HOST_SMCCC_FUNC___kvm_tlb_flush_vmid_ipa_nsh,
 	__KVM_HOST_SMCCC_FUNC___kvm_tlb_flush_vmid,
+	__KVM_HOST_SMCCC_FUNC___kvm_tlb_flush_vmid_range,
 	__KVM_HOST_SMCCC_FUNC___kvm_flush_cpu_context,
 	__KVM_HOST_SMCCC_FUNC___kvm_timer_set_cntvoff,
 	__KVM_HOST_SMCCC_FUNC___vgic_v3_read_vmcr,
@@ -229,6 +230,8 @@ extern void __kvm_tlb_flush_vmid_ipa(struct kvm_s2_mmu *mmu, phys_addr_t ipa,
 extern void __kvm_tlb_flush_vmid_ipa_nsh(struct kvm_s2_mmu *mmu,
 					 phys_addr_t ipa,
 					 int level);
+extern void __kvm_tlb_flush_vmid_range(struct kvm_s2_mmu *mmu,
+					phys_addr_t start, unsigned long pages);
 extern void __kvm_tlb_flush_vmid(struct kvm_s2_mmu *mmu);
 
 extern void __kvm_timer_set_cntvoff(u64 cntvoff);
arch/arm64/kvm/hyp/nvhe/hyp-main.c +11 -0
index a169c619db60b..857d9bc04fd48 100644
@@ -135,6 +135,16 @@ static void handle___kvm_tlb_flush_vmid_ipa_nsh(struct kvm_cpu_context *host_ctx
 	__kvm_tlb_flush_vmid_ipa_nsh(kern_hyp_va(mmu), ipa, level);
 }
 
+static void
+handle___kvm_tlb_flush_vmid_range(struct kvm_cpu_context *host_ctxt)
+{
+	DECLARE_REG(struct kvm_s2_mmu *, mmu, host_ctxt, 1);
+	DECLARE_REG(phys_addr_t, start, host_ctxt, 2);
+	DECLARE_REG(unsigned long, pages, host_ctxt, 3);
+
+	__kvm_tlb_flush_vmid_range(kern_hyp_va(mmu), start, pages);
+}
+
 static void handle___kvm_tlb_flush_vmid(struct kvm_cpu_context *host_ctxt)
 {
 	DECLARE_REG(struct kvm_s2_mmu *, mmu, host_ctxt, 1);
@@ -327,6 +337,7 @@ static const hcall_t host_hcall[] = {
 	HANDLE_FUNC(__kvm_tlb_flush_vmid_ipa),
 	HANDLE_FUNC(__kvm_tlb_flush_vmid_ipa_nsh),
 	HANDLE_FUNC(__kvm_tlb_flush_vmid),
+	HANDLE_FUNC(__kvm_tlb_flush_vmid_range),
 	HANDLE_FUNC(__kvm_flush_cpu_context),
 	HANDLE_FUNC(__kvm_timer_set_cntvoff),
 	HANDLE_FUNC(__vgic_v3_read_vmcr),
arch/arm64/kvm/hyp/nvhe/tlb.c +30 -0
index b9991bbd8e3fd..1b265713d6bed 100644
@@ -182,6 +182,36 @@ void __kvm_tlb_flush_vmid_ipa_nsh(struct kvm_s2_mmu *mmu,
 	__tlb_switch_to_host(&cxt);
 }
 
+void __kvm_tlb_flush_vmid_range(struct kvm_s2_mmu *mmu,
+				phys_addr_t start, unsigned long pages)
+{
+	struct tlb_inv_context cxt;
+	unsigned long stride;
+
+	/*
+	 * Since the range of addresses may not be mapped at
+	 * the same level, assume the worst case as PAGE_SIZE
+	 */
+	stride = PAGE_SIZE;
+	start = round_down(start, stride);
+
+	/* Switch to requested VMID */
+	__tlb_switch_to_guest(mmu, &cxt, false);
+
+	__flush_s2_tlb_range_op(ipas2e1is, start, pages, stride, 0);
+
+	dsb(ish);
+	__tlbi(vmalle1is);
+	dsb(ish);
+	isb();
+
+	/* See the comment in __kvm_tlb_flush_vmid_ipa() */
+	if (icache_is_vpipt())
+		icache_inval_all_pou();
+
+	__tlb_switch_to_host(&cxt);
+}
+
 void __kvm_tlb_flush_vmid(struct kvm_s2_mmu *mmu)
 {
 	struct tlb_inv_context cxt;
arch/arm64/kvm/hyp/vhe/tlb.c +28 -0
index e69da550cdc5b..46bd43f61d76f 100644
@@ -143,6 +143,34 @@ void __kvm_tlb_flush_vmid_ipa_nsh(struct kvm_s2_mmu *mmu,
 	__tlb_switch_to_host(&cxt);
 }
 
+void __kvm_tlb_flush_vmid_range(struct kvm_s2_mmu *mmu,
+				phys_addr_t start, unsigned long pages)
+{
+	struct tlb_inv_context cxt;
+	unsigned long stride;
+
+	/*
+	 * Since the range of addresses may not be mapped at
+	 * the same level, assume the worst case as PAGE_SIZE
+	 */
+	stride = PAGE_SIZE;
+	start = round_down(start, stride);
+
+	dsb(ishst);
+
+	/* Switch to requested VMID */
+	__tlb_switch_to_guest(mmu, &cxt);
+
+	__flush_s2_tlb_range_op(ipas2e1is, start, pages, stride, 0);
+
+	dsb(ish);
+	__tlbi(vmalle1is);
+	dsb(ish);
+	isb();
+
+	__tlb_switch_to_host(&cxt);
+}
+
 void __kvm_tlb_flush_vmid(struct kvm_s2_mmu *mmu)
 {
 	struct tlb_inv_context cxt;
Signature · 1 line
2.41.0.640.ga95def55d0-goog
#25 lore raw

Re: [PATCH v8 09/14] KVM: arm64: Implement __kvm_tlb_flush_vmid_range()

Quoted 4 lines · Raghavendra Rao Ananta

Define __kvm_tlb_flush_vmid_range() (for VHE and nVHE) to flush a range of stage-2 page-tables using IPA in one go. If the system supports FEAT_TLBIRANGE, the following patches would conviniently replace global TLBI such as vmalls12e1is

         ^^^^^^^^^^^^
         conveniently

Spotted by scripts/checkpatch.pl --codespell

Quoted 142 lines

in the map, unmap, and dirty-logging paths with ripas2e1is instead.

---
  arch/arm64/include/asm/kvm_asm.h   |  3 +++
  arch/arm64/kvm/hyp/nvhe/hyp-main.c | 11 +++++++++++
  arch/arm64/kvm/hyp/nvhe/tlb.c      | 30 ++++++++++++++++++++++++++++++
  arch/arm64/kvm/hyp/vhe/tlb.c       | 28 ++++++++++++++++++++++++++++
  4 files changed, 72 insertions(+)
arch/arm64/include/asm/kvm_asm.h +3 -0
index 7d170aaa2db41..2c27cb8cf442d 100644
@@ -70,6 +70,7 @@ enum __kvm_host_smccc_func {
  	__KVM_HOST_SMCCC_FUNC___kvm_tlb_flush_vmid_ipa,
  	__KVM_HOST_SMCCC_FUNC___kvm_tlb_flush_vmid_ipa_nsh,
  	__KVM_HOST_SMCCC_FUNC___kvm_tlb_flush_vmid,
+	__KVM_HOST_SMCCC_FUNC___kvm_tlb_flush_vmid_range,
  	__KVM_HOST_SMCCC_FUNC___kvm_flush_cpu_context,
  	__KVM_HOST_SMCCC_FUNC___kvm_timer_set_cntvoff,
  	__KVM_HOST_SMCCC_FUNC___vgic_v3_read_vmcr,
@@ -229,6 +230,8 @@ extern void __kvm_tlb_flush_vmid_ipa(struct kvm_s2_mmu *mmu, phys_addr_t ipa,
  extern void __kvm_tlb_flush_vmid_ipa_nsh(struct kvm_s2_mmu *mmu,
  					 phys_addr_t ipa,
  					 int level);
+extern void __kvm_tlb_flush_vmid_range(struct kvm_s2_mmu *mmu,
+					phys_addr_t start, unsigned long pages);
  extern void __kvm_tlb_flush_vmid(struct kvm_s2_mmu *mmu);
 
  extern void __kvm_timer_set_cntvoff(u64 cntvoff);
arch/arm64/kvm/hyp/nvhe/hyp-main.c +11 -0
index a169c619db60b..857d9bc04fd48 100644
@@ -135,6 +135,16 @@ static void handle___kvm_tlb_flush_vmid_ipa_nsh(struct kvm_cpu_context *host_ctx
  	__kvm_tlb_flush_vmid_ipa_nsh(kern_hyp_va(mmu), ipa, level);
  }
 
+static void
+handle___kvm_tlb_flush_vmid_range(struct kvm_cpu_context *host_ctxt)
+{
+	DECLARE_REG(struct kvm_s2_mmu *, mmu, host_ctxt, 1);
+	DECLARE_REG(phys_addr_t, start, host_ctxt, 2);
+	DECLARE_REG(unsigned long, pages, host_ctxt, 3);
+
+	__kvm_tlb_flush_vmid_range(kern_hyp_va(mmu), start, pages);
+}
+
  static void handle___kvm_tlb_flush_vmid(struct kvm_cpu_context *host_ctxt)
  {
  	DECLARE_REG(struct kvm_s2_mmu *, mmu, host_ctxt, 1);
@@ -327,6 +337,7 @@ static const hcall_t host_hcall[] = {
  	HANDLE_FUNC(__kvm_tlb_flush_vmid_ipa),
  	HANDLE_FUNC(__kvm_tlb_flush_vmid_ipa_nsh),
  	HANDLE_FUNC(__kvm_tlb_flush_vmid),
+	HANDLE_FUNC(__kvm_tlb_flush_vmid_range),
  	HANDLE_FUNC(__kvm_flush_cpu_context),
  	HANDLE_FUNC(__kvm_timer_set_cntvoff),
  	HANDLE_FUNC(__vgic_v3_read_vmcr),
arch/arm64/kvm/hyp/nvhe/tlb.c +30 -0
index b9991bbd8e3fd..1b265713d6bed 100644
@@ -182,6 +182,36 @@ void __kvm_tlb_flush_vmid_ipa_nsh(struct kvm_s2_mmu *mmu,
  	__tlb_switch_to_host(&cxt);
  }
 
+void __kvm_tlb_flush_vmid_range(struct kvm_s2_mmu *mmu,
+				phys_addr_t start, unsigned long pages)
+{
+	struct tlb_inv_context cxt;
+	unsigned long stride;
+
+	/*
+	 * Since the range of addresses may not be mapped at
+	 * the same level, assume the worst case as PAGE_SIZE
+	 */
+	stride = PAGE_SIZE;
+	start = round_down(start, stride);
+
+	/* Switch to requested VMID */
+	__tlb_switch_to_guest(mmu, &cxt, false);
+
+	__flush_s2_tlb_range_op(ipas2e1is, start, pages, stride, 0);
+
+	dsb(ish);
+	__tlbi(vmalle1is);
+	dsb(ish);
+	isb();
+
+	/* See the comment in __kvm_tlb_flush_vmid_ipa() */
+	if (icache_is_vpipt())
+		icache_inval_all_pou();
+
+	__tlb_switch_to_host(&cxt);
+}
+
  void __kvm_tlb_flush_vmid(struct kvm_s2_mmu *mmu)
  {
  	struct tlb_inv_context cxt;
arch/arm64/kvm/hyp/vhe/tlb.c +28 -0
index e69da550cdc5b..46bd43f61d76f 100644
@@ -143,6 +143,34 @@ void __kvm_tlb_flush_vmid_ipa_nsh(struct kvm_s2_mmu *mmu,
  	__tlb_switch_to_host(&cxt);
  }
 
+void __kvm_tlb_flush_vmid_range(struct kvm_s2_mmu *mmu,
+				phys_addr_t start, unsigned long pages)
+{
+	struct tlb_inv_context cxt;
+	unsigned long stride;
+
+	/*
+	 * Since the range of addresses may not be mapped at
+	 * the same level, assume the worst case as PAGE_SIZE
+	 */
+	stride = PAGE_SIZE;
+	start = round_down(start, stride);
+
+	dsb(ishst);
+
+	/* Switch to requested VMID */
+	__tlb_switch_to_guest(mmu, &cxt);
+
+	__flush_s2_tlb_range_op(ipas2e1is, start, pages, stride, 0);
+
+	dsb(ish);
+	__tlbi(vmalle1is);
+	dsb(ish);
+	isb();
+
+	__tlb_switch_to_host(&cxt);
+}
+
  void __kvm_tlb_flush_vmid(struct kvm_s2_mmu *mmu)
  {
  	struct tlb_inv_context cxt;
Reviewed-by on v8 9/14
#26 lore raw

Re: [PATCH v8 09/14] KVM: arm64: Implement __kvm_tlb_flush_vmid_range()

In reply to Raghavendra Rao Ananta
Quoted 9 lines · Raghavendra Rao Ananta

Define __kvm_tlb_flush_vmid_range() (for VHE and nVHE) to flush a range of stage-2 page-tables using IPA in one go. If the system supports FEAT_TLBIRANGE, the following patches would conviniently replace global TLBI such as vmalls12e1is in the map, unmap, and dirty-logging paths with ripas2e1is instead.

Quoted 137 lines
---
  arch/arm64/include/asm/kvm_asm.h   |  3 +++
  arch/arm64/kvm/hyp/nvhe/hyp-main.c | 11 +++++++++++
  arch/arm64/kvm/hyp/nvhe/tlb.c      | 30 ++++++++++++++++++++++++++++++
  arch/arm64/kvm/hyp/vhe/tlb.c       | 28 ++++++++++++++++++++++++++++
  4 files changed, 72 insertions(+)
arch/arm64/include/asm/kvm_asm.h +3 -0
index 7d170aaa2db41..2c27cb8cf442d 100644
@@ -70,6 +70,7 @@ enum __kvm_host_smccc_func {
  	__KVM_HOST_SMCCC_FUNC___kvm_tlb_flush_vmid_ipa,
  	__KVM_HOST_SMCCC_FUNC___kvm_tlb_flush_vmid_ipa_nsh,
  	__KVM_HOST_SMCCC_FUNC___kvm_tlb_flush_vmid,
+	__KVM_HOST_SMCCC_FUNC___kvm_tlb_flush_vmid_range,
  	__KVM_HOST_SMCCC_FUNC___kvm_flush_cpu_context,
  	__KVM_HOST_SMCCC_FUNC___kvm_timer_set_cntvoff,
  	__KVM_HOST_SMCCC_FUNC___vgic_v3_read_vmcr,
@@ -229,6 +230,8 @@ extern void __kvm_tlb_flush_vmid_ipa(struct kvm_s2_mmu *mmu, phys_addr_t ipa,
  extern void __kvm_tlb_flush_vmid_ipa_nsh(struct kvm_s2_mmu *mmu,
  					 phys_addr_t ipa,
  					 int level);
+extern void __kvm_tlb_flush_vmid_range(struct kvm_s2_mmu *mmu,
+					phys_addr_t start, unsigned long pages);
  extern void __kvm_tlb_flush_vmid(struct kvm_s2_mmu *mmu);
 
  extern void __kvm_timer_set_cntvoff(u64 cntvoff);
arch/arm64/kvm/hyp/nvhe/hyp-main.c +11 -0
index a169c619db60b..857d9bc04fd48 100644
@@ -135,6 +135,16 @@ static void handle___kvm_tlb_flush_vmid_ipa_nsh(struct kvm_cpu_context *host_ctx
  	__kvm_tlb_flush_vmid_ipa_nsh(kern_hyp_va(mmu), ipa, level);
  }
 
+static void
+handle___kvm_tlb_flush_vmid_range(struct kvm_cpu_context *host_ctxt)
+{
+	DECLARE_REG(struct kvm_s2_mmu *, mmu, host_ctxt, 1);
+	DECLARE_REG(phys_addr_t, start, host_ctxt, 2);
+	DECLARE_REG(unsigned long, pages, host_ctxt, 3);
+
+	__kvm_tlb_flush_vmid_range(kern_hyp_va(mmu), start, pages);
+}
+
  static void handle___kvm_tlb_flush_vmid(struct kvm_cpu_context *host_ctxt)
  {
  	DECLARE_REG(struct kvm_s2_mmu *, mmu, host_ctxt, 1);
@@ -327,6 +337,7 @@ static const hcall_t host_hcall[] = {
  	HANDLE_FUNC(__kvm_tlb_flush_vmid_ipa),
  	HANDLE_FUNC(__kvm_tlb_flush_vmid_ipa_nsh),
  	HANDLE_FUNC(__kvm_tlb_flush_vmid),
+	HANDLE_FUNC(__kvm_tlb_flush_vmid_range),
  	HANDLE_FUNC(__kvm_flush_cpu_context),
  	HANDLE_FUNC(__kvm_timer_set_cntvoff),
  	HANDLE_FUNC(__vgic_v3_read_vmcr),
arch/arm64/kvm/hyp/nvhe/tlb.c +30 -0
index b9991bbd8e3fd..1b265713d6bed 100644
@@ -182,6 +182,36 @@ void __kvm_tlb_flush_vmid_ipa_nsh(struct kvm_s2_mmu *mmu,
  	__tlb_switch_to_host(&cxt);
  }
 
+void __kvm_tlb_flush_vmid_range(struct kvm_s2_mmu *mmu,
+				phys_addr_t start, unsigned long pages)
+{
+	struct tlb_inv_context cxt;
+	unsigned long stride;
+
+	/*
+	 * Since the range of addresses may not be mapped at
+	 * the same level, assume the worst case as PAGE_SIZE
+	 */
+	stride = PAGE_SIZE;
+	start = round_down(start, stride);
+
+	/* Switch to requested VMID */
+	__tlb_switch_to_guest(mmu, &cxt, false);
+
+	__flush_s2_tlb_range_op(ipas2e1is, start, pages, stride, 0);
+
+	dsb(ish);
+	__tlbi(vmalle1is);
+	dsb(ish);
+	isb();
+
+	/* See the comment in __kvm_tlb_flush_vmid_ipa() */
+	if (icache_is_vpipt())
+		icache_inval_all_pou();
+
+	__tlb_switch_to_host(&cxt);
+}
+
  void __kvm_tlb_flush_vmid(struct kvm_s2_mmu *mmu)
  {
  	struct tlb_inv_context cxt;
arch/arm64/kvm/hyp/vhe/tlb.c +28 -0
index e69da550cdc5b..46bd43f61d76f 100644
@@ -143,6 +143,34 @@ void __kvm_tlb_flush_vmid_ipa_nsh(struct kvm_s2_mmu *mmu,
  	__tlb_switch_to_host(&cxt);
  }
 
+void __kvm_tlb_flush_vmid_range(struct kvm_s2_mmu *mmu,
+				phys_addr_t start, unsigned long pages)
+{
+	struct tlb_inv_context cxt;
+	unsigned long stride;
+
+	/*
+	 * Since the range of addresses may not be mapped at
+	 * the same level, assume the worst case as PAGE_SIZE
+	 */
+	stride = PAGE_SIZE;
+	start = round_down(start, stride);
+
+	dsb(ishst);
+
+	/* Switch to requested VMID */
+	__tlb_switch_to_guest(mmu, &cxt);
+
+	__flush_s2_tlb_range_op(ipas2e1is, start, pages, stride, 0);
+
+	dsb(ish);
+	__tlbi(vmalle1is);
+	dsb(ish);
+	isb();
+
+	__tlb_switch_to_host(&cxt);
+}
+
  void __kvm_tlb_flush_vmid(struct kvm_s2_mmu *mmu)
  {
  	struct tlb_inv_context cxt;
Signature · 1 line
Shaoqin
#27 lore raw

PATCH v8 10/14 KVM: arm64: Define kvm_tlb_flush_vmid_range()

In reply to Raghavendra Rao Ananta

Implement the helper kvm_tlb_flush_vmid_range() that acts as a wrapper for range-based TLB invalidations. For the given VMID, use the range-based TLBI instructions to do the job or fallback to invalidating all the TLB entries.

---
 arch/arm64/include/asm/kvm_pgtable.h | 10 ++++++++++
 arch/arm64/kvm/hyp/pgtable.c         | 20 ++++++++++++++++++++
 2 files changed, 30 insertions(+)
arch/arm64/include/asm/kvm_pgtable.h +10 -0
index 8294a9a7e566d..5e8b1ff07854b 100644
@@ -754,4 +754,14 @@ enum kvm_pgtable_prot kvm_pgtable_stage2_pte_prot(kvm_pte_t pte);
  *	   kvm_pgtable_prot format.
  */
 enum kvm_pgtable_prot kvm_pgtable_hyp_pte_prot(kvm_pte_t pte);
+
+/**
+ * kvm_tlb_flush_vmid_range() - Invalidate/flush a range of TLB entries
+ *
+ * @mmu:	Stage-2 KVM MMU struct
+ * @addr:	The base Intermediate physical address from which to invalidate
+ * @size:	Size of the range from the base to invalidate
+ */
+void kvm_tlb_flush_vmid_range(struct kvm_s2_mmu *mmu,
+				phys_addr_t addr, size_t size);
 #endif	/* __ARM64_KVM_PGTABLE_H__ */
arch/arm64/kvm/hyp/pgtable.c +20 -0
index aa740a974e024..5d14d5d5819a1 100644
@@ -670,6 +670,26 @@ static bool stage2_has_fwb(struct kvm_pgtable *pgt)
 	return !(pgt->flags & KVM_PGTABLE_S2_NOFWB);
 }
 
+void kvm_tlb_flush_vmid_range(struct kvm_s2_mmu *mmu,
+				phys_addr_t addr, size_t size)
+{
+	unsigned long pages, inval_pages;
+
+	if (!system_supports_tlb_range()) {
+		kvm_call_hyp(__kvm_tlb_flush_vmid, mmu);
+		return;
+	}
+
+	pages = size >> PAGE_SHIFT;
+	while (pages > 0) {
+		inval_pages = min(pages, MAX_TLBI_RANGE_PAGES);
+		kvm_call_hyp(__kvm_tlb_flush_vmid_range, mmu, addr, inval_pages);
+
+		addr += inval_pages << PAGE_SHIFT;
+		pages -= inval_pages;
+	}
+}
+
 #define KVM_S2_MEMATTR(pgt, attr) PAGE_S2_MEMATTR(attr, stage2_has_fwb(pgt))
 
 static int stage2_set_prot_attr(struct kvm_pgtable *pgt, enum kvm_pgtable_prot prot,
Signature · 1 line
2.41.0.640.ga95def55d0-goog
#28 lore raw

PATCH v8 11/14 KVM: arm64: Implement kvm_arch_flush_remote_tlbs_range()

In reply to Raghavendra Rao Ananta

Implement kvm_arch_flush_remote_tlbs_range() for arm64 to invalidate the given range in the TLB.

---
 arch/arm64/include/asm/kvm_host.h | 2 ++
 arch/arm64/kvm/mmu.c              | 8 ++++++++
 2 files changed, 10 insertions(+)
arch/arm64/include/asm/kvm_host.h +2 -0
index 20f2ba149c70c..8f2d99eaab036 100644
@@ -1113,6 +1113,8 @@ struct kvm *kvm_arch_alloc_vm(void);
 
 #define __KVM_HAVE_ARCH_FLUSH_REMOTE_TLBS
 
+#define __KVM_HAVE_ARCH_FLUSH_REMOTE_TLBS_RANGE
+
 static inline bool kvm_vm_is_protected(struct kvm *kvm)
 {
 	return false;
arch/arm64/kvm/mmu.c +8 -0
index 0ac721fa27f18..294078ce16349 100644
@@ -172,6 +172,14 @@ int kvm_arch_flush_remote_tlbs(struct kvm *kvm)
 	return 0;
 }
 
+int kvm_arch_flush_remote_tlbs_range(struct kvm *kvm,
+				      gfn_t gfn, u64 nr_pages)
+{
+	kvm_tlb_flush_vmid_range(&kvm->arch.mmu,
+				start_gfn << PAGE_SHIFT, nr_pages << PAGE_SHIFT);
+	return 0;
+}
+
 static bool kvm_is_device_pfn(unsigned long pfn)
 {
 	return !pfn_is_map_memory(pfn);
Signature · 1 line
2.41.0.640.ga95def55d0-goog
#29 lore raw

Re: [PATCH v8 11/14] KVM: arm64: Implement kvm_arch_flush_remote_tlbs_range()

kernel test robot noticed the following build errors:

Hi Raghavendra,

kernel test robot noticed the following build errors:

[auto build test ERROR on kvm/queue]
[also build test ERROR on kvmarm/next arm64/for-next/core linus/master v6.5-rc5 next-20230809] [cannot apply to kvm/linux-next]
[If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]

url:    https://github.com/intel-lab-lkp/linux/commits/Raghavendra-Rao-Ananta/KVM-Rename-kvm_arch_flush_remote_tlb-to-kvm_arch_flush_remote_tlbs/20230809-071643
base:   https://git.kernel.org/pub/scm/virt/kvm/kvm.git queue
patch link:    https://lore.kernel.org/r/20230808231330.3855936-12-rananta%40google.com
patch subject: [PATCH v8 11/14] KVM: arm64: Implement kvm_arch_flush_remote_tlbs_range()
config: arm64-randconfig-r023-20230809 (https://download.01.org/0day-ci/archive/20230810/202308100953.kGcDpe5z-lkp at intel.com/config)
compiler: clang version 15.0.7 (https://github.com/llvm/llvm-project.git 8dfdcc7b7bf66834a761bd8de445840ef68e4d1a)
reproduce: (https://download.01.org/0day-ci/archive/20230810/202308100953.kGcDpe5z-lkp at intel.com/reproduce)

If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/202308100953.kGcDpe5z-lkp at intel.com/

All errors (new ones prefixed by >>):

>> arch/arm64/kvm/mmu.c:179:5: error: use of undeclared identifier 'start_gfn'
                                   start_gfn << PAGE_SHIFT, nr_pages << PAGE_SHIFT);
                                   ^

1 error generated.

vim +/start_gfn +179 arch/arm64/kvm/mmu.c

174
175 int kvm_arch_flush_remote_tlbs_range(struct kvm *kvm,
176 gfn_t gfn, u64 nr_pages)
177 {
178 kvm_tlb_flush_vmid_range(&kvm->arch.mmu,

 > 179					start_gfn << PAGE_SHIFT, nr_pages << PAGE_SHIFT);

180 return 0;
181 }
182

Signature · 2 lines
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki
#30 lore raw

Re: [PATCH v8 11/14] KVM: arm64: Implement kvm_arch_flush_remote_tlbs_range()

In reply to Raghavendra Rao Ananta
Quoted 29 lines · Raghavendra Rao Ananta

Implement kvm_arch_flush_remote_tlbs_range() for arm64 to invalidate the given range in the TLB.

---
  arch/arm64/include/asm/kvm_host.h | 2 ++
  arch/arm64/kvm/mmu.c              | 8 ++++++++
  2 files changed, 10 insertions(+)
arch/arm64/include/asm/kvm_host.h +2 -0
index 20f2ba149c70c..8f2d99eaab036 100644
@@ -1113,6 +1113,8 @@ struct kvm *kvm_arch_alloc_vm(void);
 
  #define __KVM_HAVE_ARCH_FLUSH_REMOTE_TLBS
 
+#define __KVM_HAVE_ARCH_FLUSH_REMOTE_TLBS_RANGE
+
  static inline bool kvm_vm_is_protected(struct kvm *kvm)
  {
  	return false;
arch/arm64/kvm/mmu.c +0 -0
index 0ac721fa27f18..294078ce16349 100644

arch/arm64/kvm/mmu.c int kvm_arch_flush_remote_tlbs(struct kvm *kvm) · line 176

Quoted 5 lines
arch/arm64/kvm/mmu.c +2 -0
@@ -172,6 +172,14 @@ int kvm_arch_flush_remote_tlbs(struct kvm *kvm)
  	return 0;
  }
 
+int kvm_arch_flush_remote_tlbs_range(struct kvm *kvm,
+				      gfn_t gfn, u64 nr_pages)

Hi Raghavendra,

As the kernel test robot points out, the parameter gfn should change to start_gfn.

Thanks,
Shaoqin

Quoted 9 lines
arch/arm64/kvm/mmu.c +6 -0
+{
+	kvm_tlb_flush_vmid_range(&kvm->arch.mmu,
+				start_gfn << PAGE_SHIFT, nr_pages << PAGE_SHIFT);
+	return 0;
+}
+
  static bool kvm_is_device_pfn(unsigned long pfn)
  {
  	return !pfn_is_map_memory(pfn);
Signature · 1 line
Shaoqin
#31 lore raw

PATCH v8 12/14 KVM: arm64: Flush only the memslot after write-protect

In reply to Raghavendra Rao Ananta

After write-protecting the region, currently KVM invalidates the entire TLB entries using kvm_flush_remote_tlbs(). Instead, scope the invalidation only to the targeted memslot. If supported, the architecture would use the range-based TLBI instructions to flush the memslot or else fallback to flushing all of the TLBs.

---
 arch/arm64/kvm/mmu.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
arch/arm64/kvm/mmu.c +1 -1
index 294078ce16349..95ca2b86aa2cd 100644
@@ -1083,7 +1083,7 @@ static void kvm_mmu_wp_memory_region(struct kvm *kvm, int slot)
 	write_lock(&kvm->mmu_lock);
 	stage2_wp_range(&kvm->arch.mmu, start, end);
 	write_unlock(&kvm->mmu_lock);
-	kvm_flush_remote_tlbs(kvm);
+	kvm_flush_remote_tlbs_memslot(kvm, memslot);
 }
 
 /**
Signature · 1 line
2.41.0.640.ga95def55d0-goog
#32 lore raw

PATCH v8 13/14 KVM: arm64: Invalidate the table entries upon a range

In reply to Raghavendra Rao Ananta

Currently, during the operations such as a hugepage collapse, KVM would flush the entire VM's context using 'vmalls12e1is' TLBI operation. Specifically, if the VM is faulting on many hugepages (say after dirty-logging), it creates a performance penalty for the guest whose pages have already been faulted earlier as they would have to refill their TLBs again.

Instead, leverage kvm_tlb_flush_vmid_range() for table entries. If the system supports it, only the required range will be flushed. Else, it'll fallback to the previous mechanism.

---
 arch/arm64/kvm/hyp/pgtable.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
arch/arm64/kvm/hyp/pgtable.c +2 -1
index 5d14d5d5819a1..5ef098af17362 100644
@@ -806,7 +806,8 @@ static bool stage2_try_break_pte(const struct kvm_pgtable_visit_ctx *ctx,
 		 * evicted pte value (if any).
 		 */
 		if (kvm_pte_table(ctx->old, ctx->level))
-			kvm_call_hyp(__kvm_tlb_flush_vmid, mmu);
+			kvm_tlb_flush_vmid_range(mmu, ctx->addr,
+						kvm_granule_size(ctx->level));
 		else if (kvm_pte_valid(ctx->old))
 			kvm_call_hyp(__kvm_tlb_flush_vmid_ipa, mmu,
 				     ctx->addr, ctx->level);
Signature · 1 line
2.41.0.640.ga95def55d0-goog
#33 lore raw

PATCH v8 14/14 KVM: arm64: Use TLBI range-based intructions for unmap

In reply to Raghavendra Rao Ananta

The current implementation of the stage-2 unmap walker traverses the given range and, as a part of break-before-make, performs TLB invalidations with a DSB for every PTE. A multitude of this combination could cause a performance bottleneck on some systems.

Hence, if the system supports FEAT_TLBIRANGE, defer the TLB invalidations until the entire walk is finished, and then use range-based instructions to invalidate the TLBs in one go. Condition deferred TLB invalidation on the system supporting FWB, as the optimization is entirely pointless when the unmap walker needs to perform CMOs.

Rename stage2_put_pte() to stage2_unmap_put_pte() as the function now serves the stage-2 unmap walker specifically, rather than acting generic.

---
 arch/arm64/kvm/hyp/pgtable.c | 40 +++++++++++++++++++++++++++++-------
 1 file changed, 33 insertions(+), 7 deletions(-)
arch/arm64/kvm/hyp/pgtable.c +33 -7
index 5ef098af17362..eaaae76481fa9 100644
@@ -831,16 +831,36 @@ static void stage2_make_pte(const struct kvm_pgtable_visit_ctx *ctx, kvm_pte_t n
 	smp_store_release(ctx->ptep, new);
 }
 
-static void stage2_put_pte(const struct kvm_pgtable_visit_ctx *ctx, struct kvm_s2_mmu *mmu,
-			   struct kvm_pgtable_mm_ops *mm_ops)
+static bool stage2_unmap_defer_tlb_flush(struct kvm_pgtable *pgt)
 {
 	/*
-	 * Clear the existing PTE, and perform break-before-make with
-	 * TLB maintenance if it was valid.
+	 * If FEAT_TLBIRANGE is implemented, defer the individual
+	 * TLB invalidations until the entire walk is finished, and
+	 * then use the range-based TLBI instructions to do the
+	 * invalidations. Condition deferred TLB invalidation on the
+	 * system supporting FWB as the optimization is entirely
+	 * pointless when the unmap walker needs to perform CMOs.
+	 */
+	return system_supports_tlb_range() && stage2_has_fwb(pgt);
+}
+
+static void stage2_unmap_put_pte(const struct kvm_pgtable_visit_ctx *ctx,
+				struct kvm_s2_mmu *mmu,
+				struct kvm_pgtable_mm_ops *mm_ops)
+{
+	struct kvm_pgtable *pgt = ctx->arg;
+
+	/*
+	 * Clear the existing PTE, and perform break-before-make if it was
+	 * valid. Depending on the system support, defer the TLB maintenance
+	 * for the same until the entire unmap walk is completed.
 	 */
 	if (kvm_pte_valid(ctx->old)) {
 		kvm_clear_pte(ctx->ptep);
-		kvm_call_hyp(__kvm_tlb_flush_vmid_ipa, mmu, ctx->addr, ctx->level);
+
+		if (!stage2_unmap_defer_tlb_flush(pgt))
+			kvm_call_hyp(__kvm_tlb_flush_vmid_ipa, mmu,
+					ctx->addr, ctx->level);
 	}
 
 	mm_ops->put_page(ctx->ptep);
@@ -1098,7 +1118,7 @@ static int stage2_unmap_walker(const struct kvm_pgtable_visit_ctx *ctx,
 	 * block entry and rely on the remaining portions being faulted
 	 * back lazily.
 	 */
-	stage2_put_pte(ctx, mmu, mm_ops);
+	stage2_unmap_put_pte(ctx, mmu, mm_ops);
 
 	if (need_flush && mm_ops->dcache_clean_inval_poc)
 		mm_ops->dcache_clean_inval_poc(kvm_pte_follow(ctx->old, mm_ops),
@@ -1112,13 +1132,19 @@ static int stage2_unmap_walker(const struct kvm_pgtable_visit_ctx *ctx,
 
 int kvm_pgtable_stage2_unmap(struct kvm_pgtable *pgt, u64 addr, u64 size)
 {
+	int ret;
 	struct kvm_pgtable_walker walker = {
 		.cb	= stage2_unmap_walker,
 		.arg	= pgt,
 		.flags	= KVM_PGTABLE_WALK_LEAF | KVM_PGTABLE_WALK_TABLE_POST,
 	};
 
-	return kvm_pgtable_walk(pgt, addr, size, &walker);
+	ret = kvm_pgtable_walk(pgt, addr, size, &walker);
+	if (stage2_unmap_defer_tlb_flush(pgt))
+		/* Perform the deferred TLB invalidations */
+		kvm_tlb_flush_vmid_range(pgt->mmu, addr, size);
+
+	return ret;
 }
 
 struct stage2_attr_data {
Signature · 1 line
2.41.0.640.ga95def55d0-goog
Reviewed-by on v8 14/14
#34 lore raw

Re: [PATCH v8 14/14] KVM: arm64: Use TLBI range-based intructions for unmap

Quoted 12 lines · Raghavendra Rao Ananta

The current implementation of the stage-2 unmap walker traverses the given range and, as a part of break-before-make, performs TLB invalidations with a DSB for every PTE. A multitude of this combination could cause a performance bottleneck on some systems.

Hence, if the system supports FEAT_TLBIRANGE, defer the TLB invalidations until the entire walk is finished, and then use range-based instructions to invalidate the TLBs in one go. Condition deferred TLB invalidation on the system supporting FWB, as the optimization is entirely pointless when the unmap walker needs to perform CMOs.

Quoted 5 lines

Rename stage2_put_pte() to stage2_unmap_put_pte() as the function now serves the stage-2 unmap walker specifically, rather than acting generic.

Quoted 80 lines
---
  arch/arm64/kvm/hyp/pgtable.c | 40 +++++++++++++++++++++++++++++-------
  1 file changed, 33 insertions(+), 7 deletions(-)
arch/arm64/kvm/hyp/pgtable.c +33 -7
index 5ef098af17362..eaaae76481fa9 100644
@@ -831,16 +831,36 @@ static void stage2_make_pte(const struct kvm_pgtable_visit_ctx *ctx, kvm_pte_t n
  	smp_store_release(ctx->ptep, new);
  }
 
-static void stage2_put_pte(const struct kvm_pgtable_visit_ctx *ctx, struct kvm_s2_mmu *mmu,
-			   struct kvm_pgtable_mm_ops *mm_ops)
+static bool stage2_unmap_defer_tlb_flush(struct kvm_pgtable *pgt)
  {
  	/*
-	 * Clear the existing PTE, and perform break-before-make with
-	 * TLB maintenance if it was valid.
+	 * If FEAT_TLBIRANGE is implemented, defer the individual
+	 * TLB invalidations until the entire walk is finished, and
+	 * then use the range-based TLBI instructions to do the
+	 * invalidations. Condition deferred TLB invalidation on the
+	 * system supporting FWB as the optimization is entirely
+	 * pointless when the unmap walker needs to perform CMOs.
+	 */
+	return system_supports_tlb_range() && stage2_has_fwb(pgt);
+}
+
+static void stage2_unmap_put_pte(const struct kvm_pgtable_visit_ctx *ctx,
+				struct kvm_s2_mmu *mmu,
+				struct kvm_pgtable_mm_ops *mm_ops)
+{
+	struct kvm_pgtable *pgt = ctx->arg;
+
+	/*
+	 * Clear the existing PTE, and perform break-before-make if it was
+	 * valid. Depending on the system support, defer the TLB maintenance
+	 * for the same until the entire unmap walk is completed.
  	 */
  	if (kvm_pte_valid(ctx->old)) {
  		kvm_clear_pte(ctx->ptep);
-		kvm_call_hyp(__kvm_tlb_flush_vmid_ipa, mmu, ctx->addr, ctx->level);
+
+		if (!stage2_unmap_defer_tlb_flush(pgt))
+			kvm_call_hyp(__kvm_tlb_flush_vmid_ipa, mmu,
+					ctx->addr, ctx->level);
  	}
 
  	mm_ops->put_page(ctx->ptep);
@@ -1098,7 +1118,7 @@ static int stage2_unmap_walker(const struct kvm_pgtable_visit_ctx *ctx,
  	 * block entry and rely on the remaining portions being faulted
  	 * back lazily.
  	 */
-	stage2_put_pte(ctx, mmu, mm_ops);
+	stage2_unmap_put_pte(ctx, mmu, mm_ops);
 
  	if (need_flush && mm_ops->dcache_clean_inval_poc)
  		mm_ops->dcache_clean_inval_poc(kvm_pte_follow(ctx->old, mm_ops),
@@ -1112,13 +1132,19 @@ static int stage2_unmap_walker(const struct kvm_pgtable_visit_ctx *ctx,
 
  int kvm_pgtable_stage2_unmap(struct kvm_pgtable *pgt, u64 addr, u64 size)
  {
+	int ret;
  	struct kvm_pgtable_walker walker = {
  		.cb	= stage2_unmap_walker,
  		.arg	= pgt,
  		.flags	= KVM_PGTABLE_WALK_LEAF | KVM_PGTABLE_WALK_TABLE_POST,
  	};
 
-	return kvm_pgtable_walk(pgt, addr, size, &walker);
+	ret = kvm_pgtable_walk(pgt, addr, size, &walker);
+	if (stage2_unmap_defer_tlb_flush(pgt))
+		/* Perform the deferred TLB invalidations */
+		kvm_tlb_flush_vmid_range(pgt->mmu, addr, size);
+
+	return ret;
  }
 
  struct stage2_attr_data {
Signature · 1 line
Shaoqin