KVM: Support PUD hugepages at stage 2

This series is an update to the PUD hugepage support previously posted at [0]. This patchset adds support for PUD hugepages at stage
2. This feature is useful on cores that have support for large sized TLB mappings (e.g., 1GB for 4K granule).

The biggest change in this version is to replace repeated instances of walking the page tables to get to a leaf-entry with a function to return the appropriate entry. This was suggested by Suzuki and should help reduce the amount of churn resulting from future changes. It also address other feedback on the previous version.

Support is added to code that is shared between arm and arm64. Dummy helpers for arm are provided as the port does not support PUD hugepage sizes.

The patches have been tested on an A57 based system. The patchset is based on v4.18-rc4. The are a few conflicts with the support for 52 bit IPA[1] due to change in number of parameters for stage2_pmd_offset().

Thanks,
Punit

v4 -> v5:

* Patch 1 - Drop helper stage2_should_exec() and refactor the condition to decide if a page table entry should be marked executable
* Patch 4-6 - Introduce stage2_get_leaf_entry() and use it in this and latter patches
* Patch 7 - Use stage 2 accessors instead of using the page table helpers directly
* Patch 7 - Add a note to update the PUD hugepage support when number of levels of stage 2 tables differs from stage 1

v3 -> v4:
* Patch 1 and 7 - Don't put down hugepages pte if logging is enabled
* Patch 4-5 - Add PUD hugepage support for exec and access faults
* Patch 6 - PUD hugepage support for aging page table entries

v2 -> v3:
* Update vma_pagesize directly if THP [1/4]. Previsouly this was done indirectly via hugetlb
* Added review tag [4/4]

v1 -> v2:
* Create helper to check if the page should have exec permission [1/4]
* Fix broken condition to detect THP hugepage [1/4]
* Fix in-correct hunk resulting from a rebase [4/4]

[0] https://www.spinics.net/lists/arm-kernel/msg663562.html
[1] https://www.spinics.net/lists/kvm/msg171065.html

Punit Agrawal (7):
KVM: arm/arm64: Share common code in user_mem_abort()
KVM: arm/arm64: Introduce helpers to manupulate page table entries
KVM: arm64: Support dirty page tracking for PUD hugepages
KVM: arm64: Support PUD hugepage in stage2_is_exec()
KVM: arm64: Support handling access faults for PUD hugepages
KVM: arm64: Update age handlers to support PUD hugepages
KVM: arm64: Add support for creating PUD hugepages at stage 2

 arch/arm/include/asm/kvm_mmu.h         |  60 +++++
 arch/arm64/include/asm/kvm_mmu.h       |  47 ++++
 arch/arm64/include/asm/pgtable-hwdef.h |   4 +
 arch/arm64/include/asm/pgtable.h       |   9 +
 virt/kvm/arm/mmu.c                     | 289 ++++++++++++++++++-------
 5 files changed, 330 insertions(+), 79 deletions(-)

Introduce helpers to abstract architectural handling of the conversion of pfn to page table entries and marking a PMD page table entry as a block entry.

The helpers are introduced in preparation for supporting PUD hugepages at stage 2 - which are supported on arm64 but do not exist on arm.

Signed-off-by Punit Agrawal <[email protected]>

Acked-by Christoffer Dall <[email protected]>

Cc Marc Zyngier <[email protected]>

Cc Russell King <[email protected]>

Cc Catalin Marinas <[email protected]>

Cc Will Deacon <[email protected]>

---

Quoted 7 lines · Punit Agrawal

Introduce helpers to abstract architectural handling of the conversion of pfn to page table entries and marking a PMD page table entry as a block entry.

The helpers are introduced in preparation for supporting PUD hugepages at stage 2 - which are supported on arm64 but do not exist on arm.

Quoted 7 lines

Signed-off-by Punit Agrawal <[email protected]>

Acked-by Christoffer Dall <[email protected]>

Cc Marc Zyngier <[email protected]>

Cc Russell King <[email protected]>

Cc Catalin Marinas <[email protected]>

Cc Will Deacon <[email protected]>

---

Reviewed-by Suzuki K Poulose <[email protected]>

In preparation for creating PUD hugepages at stage 2, add support for write protecting PUD hugepages when they are encountered. Write protecting guest tables is used to track dirty pages when migrating VMs.

Also, provide trivial implementations of required kvm_s2pud_* helpers to allow sharing of code with arm32.

Signed-off-by Punit Agrawal <[email protected]>

Reviewed-by Christoffer Dall <[email protected]>

Cc Marc Zyngier <[email protected]>

Cc Russell King <[email protected]>

Cc Catalin Marinas <[email protected]>

Cc Will Deacon <[email protected]>

---
  arch/arm/include/asm/kvm_mmu.h   | 16 ++++++++++++++++
  arch/arm64/include/asm/kvm_mmu.h | 10 ++++++++++
  virt/kvm/arm/mmu.c               | 11 +++++++----
  3 files changed, 33 insertions(+), 4 deletions(-)

arch/arm/include/asm/kvm_mmu.h +0 -0

index d095c2d0b284..c23722f75d5c 100644

In preparation for creating PUD hugepages at stage 2, add support for detecting execute permissions on PUD page table entries. Faults due to lack of execute permissions on page table entries is used to perform i-cache invalidation on first execute.

Provide trivial implementations of arm32 helpers to allow sharing of code.

Signed-off-by Punit Agrawal <[email protected]>

Cc Christoffer Dall <[email protected]>

Cc Marc Zyngier <[email protected]>

Cc Russell King <[email protected]>

Cc Catalin Marinas <[email protected]>

Cc Will Deacon <[email protected]>

---
  arch/arm/include/asm/kvm_mmu.h         |  6 ++++
  arch/arm64/include/asm/kvm_mmu.h       |  5 +++
  arch/arm64/include/asm/pgtable-hwdef.h |  2 ++
  virt/kvm/arm/mmu.c                     | 49 +++++++++++++++++++++++---
  4 files changed, 57 insertions(+), 5 deletions(-)

arch/arm/include/asm/kvm_mmu.h +6 -0

index c23722f75d5c..d05c8986e495 100644
@@ -96,6 +96,12 @@ static inline bool kvm_s2pud_readonly(pud_t *pud)
  }
 
 
+static inline bool kvm_s2pud_exec(pud_t *pud)
+{
+	BUG();
+	return false;
+}
+
  static inline void kvm_set_pmd(pmd_t *pmd, pmd_t new_pmd)
  {
  	*pmd = new_pmd;

arch/arm64/include/asm/kvm_mmu.h +5 -0

index 84051930ddfe..15bc1be8f82f 100644
@@ -249,6 +249,11 @@ static inline bool kvm_s2pud_readonly(pud_t *pudp)
  	return kvm_s2pte_readonly((pte_t *)pudp);
  }
 
+static inline bool kvm_s2pud_exec(pud_t *pudp)
+{
+	return !(READ_ONCE(pud_val(*pudp)) & PUD_S2_XN);
+}
+
  static inline bool kvm_page_empty(void *ptr)
  {
  	struct page *ptr_page = virt_to_page(ptr);

arch/arm64/include/asm/pgtable-hwdef.h +2 -0

index fd208eac9f2a..10ae592b78b8 100644
@@ -193,6 +193,8 @@
  #define PMD_S2_RDWR		(_AT(pmdval_t, 3) << 6)   /* HAP[2:1] */
  #define PMD_S2_XN		(_AT(pmdval_t, 2) << 53)  /* XN[1:0] */
 
+#define PUD_S2_XN		(_AT(pudval_t, 2) << 53)  /* XN[1:0] */
+
  /*
   * Memory Attribute override for Stage-2 (MemAttr[3:0])
   */

virt/kvm/arm/mmu.c +2 -1

index ed8f8271c389..e73909a31e02 100644
@@ -1038,23 +1038,62 @@ static int stage2_set_pmd_huge(struct kvm *kvm, struct kvm_mmu_memory_cache
  	return 0;
  }
 
-static bool stage2_is_exec(struct kvm *kvm, phys_addr_t addr)
+/*
+ * stage2_get_leaf_entry - walk the stage2 VM page tables and return

virt/kvm/arm/mmu.c +26 -4

-	pmdp = stage2_get_pmd(kvm, NULL, addr);
+	pudp = stage2_get_pud(kvm, NULL, addr);
+	if (!pudp || pud_none(*pudp) || !pud_present(*pudp))
+		return false;
+
+	if (pud_huge(*pudp)) {
+		*pudpp = pudp;
+		return true;
+	}
+
+	pmdp = stage2_pmd_offset(pudp, addr);
  	if (!pmdp || pmd_none(*pmdp) || !pmd_present(*pmdp))
  		return false;
 
-	if (pmd_thp_or_huge(*pmdp))
-		return kvm_s2pmd_exec(pmdp);
+	if (pmd_thp_or_huge(*pmdp)) {
+		*pmdpp = pmdp;
+		return true;
+	}
 
  	ptep = pte_offset_kernel(pmdp, addr);
  	if (!ptep || pte_none(*ptep) || !pte_present(*ptep))
  		return false;
 
-	return kvm_s2pte_exec(ptep);
+	*ptepp = ptep;
+	return true;
+}
+
+static bool stage2_is_exec(struct kvm *kvm, phys_addr_t addr)
+{
+	pud_t *pudp = NULL;
+	pmd_t *pmdp = NULL;
+	pte_t *ptep = NULL;
+	bool found;
+
+	found = stage2_get_leaf_entry(kvm, addr, &pudp, &pmdp, &ptep);

Quoted 80 lines · Punit Agrawal

In preparation for creating PUD hugepages at stage 2, add support for detecting execute permissions on PUD page table entries. Faults due to lack of execute permissions on page table entries is used to perform i-cache invalidation on first execute.

Provide trivial implementations of arm32 helpers to allow sharing of code.

Signed-off-by Punit Agrawal <[email protected]>

Cc Christoffer Dall <[email protected]>

Cc Marc Zyngier <[email protected]>

Cc Russell King <[email protected]>

Cc Catalin Marinas <[email protected]>

Cc Will Deacon <[email protected]>

---
  arch/arm/include/asm/kvm_mmu.h         |  6 ++++
  arch/arm64/include/asm/kvm_mmu.h       |  5 +++
  arch/arm64/include/asm/pgtable-hwdef.h |  2 ++
  virt/kvm/arm/mmu.c                     | 49 +++++++++++++++++++++++---
  4 files changed, 57 insertions(+), 5 deletions(-)

arch/arm/include/asm/kvm_mmu.h +5 -0

index c23722f75d5c..d05c8986e495 100644
@@ -96,6 +96,12 @@ static inline bool kvm_s2pud_readonly(pud_t *pud)
  }
    +static inline bool kvm_s2pud_exec(pud_t *pud)
+{
+	BUG();
+	return false;
+}
+
  static inline void kvm_set_pmd(pmd_t *pmd, pmd_t new_pmd)
  {
  	*pmd = new_pmd;

arch/arm64/include/asm/kvm_mmu.h +4 -0

index 84051930ddfe..15bc1be8f82f 100644
@@ -249,6 +249,11 @@ static inline bool kvm_s2pud_readonly(pud_t *pudp)
  	return kvm_s2pte_readonly((pte_t *)pudp);
  }
  +static inline bool kvm_s2pud_exec(pud_t *pudp)
+{
+	return !(READ_ONCE(pud_val(*pudp)) & PUD_S2_XN);
+}
+
  static inline bool kvm_page_empty(void *ptr)
  {
  	struct page *ptr_page = virt_to_page(ptr);

arch/arm64/include/asm/pgtable-hwdef.h +1 -0

index fd208eac9f2a..10ae592b78b8 100644
@@ -193,6 +193,8 @@
  #define PMD_S2_RDWR		(_AT(pmdval_t, 3) << 6)   /* HAP[2:1] */
  #define PMD_S2_XN		(_AT(pmdval_t, 2) << 53)  /* XN[1:0] */
  +#define PUD_S2_XN		(_AT(pudval_t, 2) << 53)  /* XN[1:0]
 */
+
  /*
   * Memory Attribute override for Stage-2 (MemAttr[3:0])
   */

virt/kvm/arm/mmu.c +8 -0

index ed8f8271c389..e73909a31e02 100644
@@ -1038,23 +1038,62 @@ static int stage2_set_pmd_huge(struct kvm *kvm, struct kvm_mmu_memory_cache
  	return 0;
  }
  -static bool stage2_is_exec(struct kvm *kvm, phys_addr_t addr)
+/*
+ * stage2_get_leaf_entry - walk the stage2 VM page tables and return
+ * true if a valid and present leaf-entry is found. A pointer to the
+ * leaf-entry is returned in the appropriate level variable - pudpp,
+ * pmdpp, ptepp.
+ */
+static bool stage2_get_leaf_entry(struct kvm *kvm, phys_addr_t addr,
+				  pud_t **pudpp, pmd_t **pmdpp, pte_t **ptepp)

virt/kvm/arm/mmu.c static int stage2_set_pmd_huge(struct kvm *kvm, struct kvm_mmu_memory_cache · line 1052

Quoted 4 lines

virt/kvm/arm/mmu.c +1 -0

  {
+	pud_t *pudp;
  	pmd_t *pmdp;
  	pte_t *ptep;

nit: As mentioned in the other thread, you may initialize the reference pointers to NULL to make sure we start clean and avoid the initialization everywhere this is called.

Quoted 36 lines

virt/kvm/arm/mmu.c +26 -1

  -	pmdp = stage2_get_pmd(kvm, NULL, addr);
+	pudp = stage2_get_pud(kvm, NULL, addr);
+	if (!pudp || pud_none(*pudp) || !pud_present(*pudp))
+		return false;
+
+	if (pud_huge(*pudp)) {
+		*pudpp = pudp;
+		return true;
+	}
+
+	pmdp = stage2_pmd_offset(pudp, addr);
  	if (!pmdp || pmd_none(*pmdp) || !pmd_present(*pmdp))
  		return false;
  -	if (pmd_thp_or_huge(*pmdp))
-		return kvm_s2pmd_exec(pmdp);
+	if (pmd_thp_or_huge(*pmdp)) {
+		*pmdpp = pmdp;
+		return true;
+	}
    	ptep = pte_offset_kernel(pmdp, addr);
  	if (!ptep || pte_none(*ptep) || !pte_present(*ptep))
  		return false;
  -	return kvm_s2pte_exec(ptep);
+	*ptepp = ptep;
+	return true;
+}
+
+static bool stage2_is_exec(struct kvm *kvm, phys_addr_t addr)
+{
+	pud_t *pudp = NULL;
+	pmd_t *pmdp = NULL;
+	pte_t *ptep = NULL;
+	bool found;
+
+	found = stage2_get_leaf_entry(kvm, addr, &pudp, &pmdp, &ptep);

virt/kvm/arm/mmu.c static int stage2_set_pmd_huge(struct kvm *kvm, struct kvm_mmu_memory_cache · line 1097

Quoted 10 lines

virt/kvm/arm/mmu.c +9 -0

+	if (!found)
+		return false;
+
+	if (pudp)
+		return kvm_s2pud_exec(pudp);
+	else if (pmdp)
+		return kvm_s2pmd_exec(pmdp);
+	else
+		return kvm_s2pte_exec(ptep);
  }

Reviewed-by Suzuki K Poulose <[email protected]>

_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

In preparation for creating larger hugepages at Stage 2, extend the access fault handling at Stage 2 to support PUD hugepages when encountered.

Provide trivial helpers for arm32 to allow sharing of code.

Signed-off-by Punit Agrawal <[email protected]>

Cc Christoffer Dall <[email protected]>

Cc Marc Zyngier <[email protected]>

Cc Russell King <[email protected]>

Cc Catalin Marinas <[email protected]>

Cc Will Deacon <[email protected]>

---
  arch/arm/include/asm/kvm_mmu.h   |  8 ++++++++
  arch/arm64/include/asm/kvm_mmu.h |  7 +++++++
  arch/arm64/include/asm/pgtable.h |  6 ++++++
  virt/kvm/arm/mmu.c               | 29 ++++++++++++++++-------------
  4 files changed, 37 insertions(+), 13 deletions(-)

arch/arm/include/asm/kvm_mmu.h +0 -0

index d05c8986e495..a4298d429efc 100644

virt/kvm/arm/mmu.c +0 -0

index e73909a31e02..d2c705e31584 100644
@@ -1637,33 +1637,36 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa,
   */

virt/kvm/arm/mmu.c +0 -2

  	trace_kvm_access_fault(fault_ipa);
 
  	spin_lock(&vcpu->kvm->mmu_lock);
 
-	pmd = stage2_get_pmd(vcpu->kvm, NULL, fault_ipa);
-	if (!pmd || pmd_none(*pmd))	/* Nothing there */

virt/kvm/arm/mmu.c +5 -1

+	} else 	if (pmd) {	/* THP, HugeTLB */
  		*pmd = pmd_mkyoung(*pmd);
  		pfn = pmd_pfn(*pmd);
  		pfn_valid = true;
  		goto out;
+	} else {
+		*pte = pte_mkyoung(*pte);	/* Just a page... */
+		pfn = pte_pfn(*pte);
+		pfn_valid = true;
  	}
 
-	pte = pte_offset_kernel(pmd, fault_ipa);

In preparation for creating larger hugepages at Stage 2, add support to the age handling notifiers for PUD hugepages when encountered.

Provide trivial helpers for arm32 to allow sharing code.

Signed-off-by Punit Agrawal <[email protected]>

Cc Christoffer Dall <[email protected]>

Cc Marc Zyngier <[email protected]>

Cc Russell King <[email protected]>

Cc Catalin Marinas <[email protected]>

Cc Will Deacon <[email protected]>

---
  arch/arm/include/asm/kvm_mmu.h   |  6 ++++
  arch/arm64/include/asm/kvm_mmu.h |  5 ++++
  arch/arm64/include/asm/pgtable.h |  1 +
  virt/kvm/arm/mmu.c               | 51 ++++++++++++++++++--------------
  4 files changed, 40 insertions(+), 23 deletions(-)

arch/arm/include/asm/kvm_mmu.h +6 -0

index a4298d429efc..8e1e8aee229e 100644
@@ -110,6 +110,12 @@ static inline pud_t kvm_s2pud_mkyoung(pud_t pud)
  	return pud;
  }
 
+static inline bool kvm_s2pud_young(pud_t pud)
+{
+	BUG();
+	return false;
+}
+
  static inline void kvm_set_pmd(pmd_t *pmd, pmd_t new_pmd)
  {
  	*pmd = new_pmd;

arch/arm64/include/asm/kvm_mmu.h +5 -0

index 4d2780c588b0..c542052fb199 100644
@@ -261,6 +261,11 @@ static inline pud_t kvm_s2pud_mkyoung(pud_t pud)
  	return pud_mkyoung(pud);
  }
 
+static inline bool kvm_s2pud_young(pud_t pud)
+{
+	return pud_young(pud);
+}
+
  static inline bool kvm_page_empty(void *ptr)
  {
  	struct page *ptr_page = virt_to_page(ptr);

arch/arm64/include/asm/pgtable.h +1 -0

index a64a5c35beb1..4d9476e420d9 100644
@@ -385,6 +385,7 @@ static inline int pmd_protnone(pmd_t pmd)
  #define pfn_pmd(pfn,prot)	__pmd(__phys_to_pmd_val((phys_addr_t)(pfn) << PAGE_SHIFT) | pgprot_val(prot))
  #define mk_pmd(page,prot)	pfn_pmd(page_to_pfn(page),prot)
 
+#define pud_young(pud)		pte_young(pud_pte(pud))
  #define pud_mkyoung(pud)	pte_pud(pte_mkyoung(pud_pte(pud)))
  #define pud_write(pud)		pte_write(pud_pte(pud))
 

virt/kvm/arm/mmu.c +5 -0

index d2c705e31584..a6d3ac9d7c7a 100644
@@ -1172,6 +1172,11 @@ static int stage2_pmdp_test_and_clear_young(pmd_t *pmd)
  	return stage2_ptep_test_and_clear_young((pte_t *)pmd);
  }
 
+static int stage2_pudp_test_and_clear_young(pud_t *pud)
+{
+	return stage2_ptep_test_and_clear_young((pte_t *)pud);
+}
+
  /**
   * kvm_phys_addr_ioremap - map a device range to guest IPA
   *

virt/kvm/arm/mmu.c +2 -2

  static int kvm_test_age_hva_handler(struct kvm *kvm, gpa_t gpa, u64 size, void *data)
  {
-	pmd_t *pmd;
-	pte_t *pte;
+	pud_t *pud = NULL;
+	pmd_t *pmd = NULL;

KVM only supports PMD hugepages at stage 2. Now that the various page handling routines are updated, extend the stage 2 fault handling to map in PUD hugepages.

Addition of PUD hugepage support enables additional page sizes (e.g., 1G with 4K granule) which can be useful on cores that support mapping larger block sizes in the TLB entries.

Signed-off-by Punit Agrawal <[email protected]>

Cc Christoffer Dall <[email protected]>

Cc Marc Zyngier <[email protected]>

Cc Russell King <[email protected]>

Cc Catalin Marinas <[email protected]>

Cc Will Deacon <[email protected]>

---
  arch/arm/include/asm/kvm_mmu.h         | 19 +++++++
  arch/arm64/include/asm/kvm_mmu.h       | 15 +++++
  arch/arm64/include/asm/pgtable-hwdef.h |  2 +
  arch/arm64/include/asm/pgtable.h       |  2 +
  virt/kvm/arm/mmu.c                     | 78 ++++++++++++++++++++++++--
  5 files changed, 112 insertions(+), 4 deletions(-)

arch/arm/include/asm/kvm_mmu.h +19 -0

index 8e1e8aee229e..787baf9ec994 100644
@@ -77,10 +77,13 @@ void kvm_clear_hyp_idmap(void);
 
  #define kvm_pfn_pte(pfn, prot)	pfn_pte(pfn, prot)
  #define kvm_pfn_pmd(pfn, prot)	pfn_pmd(pfn, prot)
+#define kvm_pfn_pud(pfn, prot)	(__pud(0))
 
  #define kvm_pud_pfn(pud)	(((pud_val(pud) & PUD_MASK) & PHYS_MASK) >> PAGE_SHIFT)
 
  #define kvm_pmd_mkhuge(pmd)	pmd_mkhuge(pmd)
+/* No support for pud hugepages */
+#define kvm_pud_mkhuge(pud)	(pud)
 
  /*
   * The following kvm_*pud*() functionas are provided strictly to allow

@@ -97,6 +100,22 @@ static inline bool kvm_s2pud_readonly(pud_t *pud)
  	return false;
  }
 
+static inline void kvm_set_pud(pud_t *pud, pud_t new_pud)
+{
+	BUG();
+}
+
+static inline pud_t kvm_s2pud_mkwrite(pud_t pud)
+{
+	BUG();
+	return pud;
+}
+
+static inline pud_t kvm_s2pud_mkexec(pud_t pud)
+{
+	BUG();
+	return pud;
+}
 
  static inline bool kvm_s2pud_exec(pud_t *pud)
  {

arch/arm64/include/asm/kvm_mmu.h +15 -0

index c542052fb199..dd8a23159463 100644
@@ -171,13 +171,16 @@ void kvm_clear_hyp_idmap(void);
 
  #define	kvm_set_pte(ptep, pte)		set_pte(ptep, pte)
  #define	kvm_set_pmd(pmdp, pmd)		set_pmd(pmdp, pmd)
+#define kvm_set_pud(pudp, pud)		set_pud(pudp, pud)
 
  #define kvm_pfn_pte(pfn, prot)		pfn_pte(pfn, prot)
  #define kvm_pfn_pmd(pfn, prot)		pfn_pmd(pfn, prot)
+#define kvm_pfn_pud(pfn, prot)		pfn_pud(pfn, prot)
 
  #define kvm_pud_pfn(pud)		pud_pfn(pud)
 
  #define kvm_pmd_mkhuge(pmd)		pmd_mkhuge(pmd)
+#define kvm_pud_mkhuge(pud)		pud_mkhuge(pud)
 
  static inline pte_t kvm_s2pte_mkwrite(pte_t pte)
  {

@@ -191,6 +194,12 @@ static inline pmd_t kvm_s2pmd_mkwrite(pmd_t pmd)
  	return pmd;
  }
 
+static inline pud_t kvm_s2pud_mkwrite(pud_t pud)
+{
+	pud_val(pud) |= PUD_S2_RDWR;
+	return pud;
+}
+
  static inline pte_t kvm_s2pte_mkexec(pte_t pte)
  {
  	pte_val(pte) &= ~PTE_S2_XN;

@@ -203,6 +212,12 @@ static inline pmd_t kvm_s2pmd_mkexec(pmd_t pmd)
  	return pmd;
  }
 
+static inline pud_t kvm_s2pud_mkexec(pud_t pud)
+{
+	pud_val(pud) &= ~PUD_S2_XN;
+	return pud;
+}
+
  static inline void kvm_set_s2pte_readonly(pte_t *ptep)
  {
  	pteval_t old_pteval, pteval;

arch/arm64/include/asm/pgtable-hwdef.h +2 -0

index 10ae592b78b8..e327665e94d1 100644
@@ -193,6 +193,8 @@
  #define PMD_S2_RDWR		(_AT(pmdval_t, 3) << 6)   /* HAP[2:1] */
  #define PMD_S2_XN		(_AT(pmdval_t, 2) << 53)  /* XN[1:0] */
 
+#define PUD_S2_RDONLY		(_AT(pudval_t, 1) << 6)   /* HAP[2:1] */
+#define PUD_S2_RDWR		(_AT(pudval_t, 3) << 6)   /* HAP[2:1] */
  #define PUD_S2_XN		(_AT(pudval_t, 2) << 53)  /* XN[1:0] */
 
  /*

arch/arm64/include/asm/pgtable.h +2 -0

index 4d9476e420d9..0afc34f94ff5 100644
@@ -389,6 +389,8 @@ static inline int pmd_protnone(pmd_t pmd)
  #define pud_mkyoung(pud)	pte_pud(pte_mkyoung(pud_pte(pud)))
  #define pud_write(pud)		pte_write(pud_pte(pud))
 
+#define pud_mkhuge(pud)		(__pud(pud_val(pud) & ~PUD_TABLE_BIT))
+
  #define __pud_to_phys(pud)	__pte_to_phys(pud_pte(pud))
  #define __phys_to_pud_val(phys)	__phys_to_pte_val(phys)
  #define pud_pfn(pud)		((__pud_to_phys(pud) & PUD_MASK) >> PAGE_SHIFT)

virt/kvm/arm/mmu.c +43 -1

index a6d3ac9d7c7a..d8e2497e5353 100644
@@ -116,6 +116,25 @@ static void stage2_dissolve_pmd(struct kvm *kvm, phys_addr_t addr, pmd_t *pmd)
  	put_page(virt_to_page(pmd));
  }
 
+/**
+ * stage2_dissolve_pud() - clear and flush huge PUD entry
+ * @kvm:	pointer to kvm structure.
+ * @addr:	IPA
+ * @pud:	pud pointer for IPA
+ *
+ * Function clears a PUD entry, flushes addr 1st and 2nd stage TLBs. Marks all
+ * pages in the range dirty.
+ */
+static void stage2_dissolve_pud(struct kvm *kvm, phys_addr_t addr, pud_t *pudp)
+{
+	if (!stage2_pud_huge(*pudp))
+		return;
+
+	stage2_pud_clear(pudp);
+	kvm_tlb_flush_vmid_ipa(kvm, addr);
+	put_page(virt_to_page(pudp));
+}
+
  static int mmu_topup_memory_cache(struct kvm_mmu_memory_cache *cache,
  				  int min, int max)
  {

@@ -993,7 +1012,7 @@ static pmd_t *stage2_get_pmd(struct kvm *kvm, struct kvm_mmu_memory_cache *cache
  	pmd_t *pmd;
 
  	pud = stage2_get_pud(kvm, cache, addr);
-	if (!pud)
+	if (!pud || stage2_pud_huge(*pud))
  		return NULL;
 
  	if (stage2_pud_none(*pud)) {

@@ -1038,6 +1057,26 @@ static int stage2_set_pmd_huge(struct kvm *kvm, struct kvm_mmu_memory_cache
  	return 0;
  }
 
+static int stage2_set_pud_huge(struct kvm *kvm, struct kvm_mmu_memory_cache *cache,
+			       phys_addr_t addr, const pud_t *new_pudp)
+{
+	pud_t *pudp, old_pud;
+
+	pudp = stage2_get_pud(kvm, cache, addr);
+	VM_BUG_ON(!pudp);
+
+	old_pud = *pudp;
+	if (stage2_pud_present(old_pud)) {
+		stage2_pud_clear(pudp);
+		kvm_tlb_flush_vmid_ipa(kvm, addr);
+	} else {
+		get_page(virt_to_page(pudp));
+	}
+
+	kvm_set_pud(pudp, *new_pudp);
+	return 0;
+}
+
  /*
   * stage2_get_leaf_entry - walk the stage2 VM page tables and return
   * true if a valid and present leaf-entry is found. A pointer to the

@@ -1100,6 +1139,7 @@ static int stage2_set_pte(struct kvm *kvm, struct kvm_mmu_memory_cache *cache,
  			  phys_addr_t addr, const pte_t *new_pte,
  			  unsigned long flags)
  {
+	pud_t *pud;
  	pmd_t *pmd;
  	pte_t *pte, old_pte;
  	bool iomap = flags & KVM_S2PTE_FLAG_IS_IOMAP;

@@ -1108,6 +1148,22 @@ static int stage2_set_pte(struct kvm *kvm, struct kvm_mmu_memory_cache *cache,
  	VM_BUG_ON(logging_active && !cache);
 
  	/* Create stage-2 page table mapping - Levels 0 and 1 */
+	pud = stage2_get_pud(kvm, cache, addr);
+	if (!pud) {

Quoted 22 lines · Punit Agrawal

KVM only supports PMD hugepages at stage 2. Now that the various page handling routines are updated, extend the stage 2 fault handling to map in PUD hugepages.

Addition of PUD hugepage support enables additional page sizes (e.g., 1G with 4K granule) which can be useful on cores that support mapping larger block sizes in the TLB entries.

Signed-off-by Punit Agrawal <[email protected]>

Cc Christoffer Dall <[email protected]>

Cc Marc Zyngier <[email protected]>

Cc Russell King <[email protected]>

Cc Catalin Marinas <[email protected]>

Cc Will Deacon <[email protected]>

---
  arch/arm/include/asm/kvm_mmu.h         | 19 +++++++
  arch/arm64/include/asm/kvm_mmu.h       | 15 +++++
  arch/arm64/include/asm/pgtable-hwdef.h |  2 +
  arch/arm64/include/asm/pgtable.h       |  2 +
  virt/kvm/arm/mmu.c                     | 78 ++++++++++++++++++++++++--
  5 files changed, 112 insertions(+), 4 deletions(-)

Quoted 4 lines

virt/kvm/arm/mmu.c +0 -0

index a6d3ac9d7c7a..d8e2497e5353 100644

Quoted 13 lines

virt/kvm/arm/mmu.c +3 -0

@@ -1100,6 +1139,7 @@ static int stage2_set_pte(struct kvm *kvm, struct kvm_mmu_memory_cache *cache,
  			  phys_addr_t addr, const pte_t *new_pte,
  			  unsigned long flags)
  {
+	pud_t *pud;
  	pmd_t *pmd;
  	pte_t *pte, old_pte;
  	bool iomap = flags & KVM_S2PTE_FLAG_IS_IOMAP;

@@ -1108,6 +1148,22 @@ static int stage2_set_pte(struct kvm *kvm, struct kvm_mmu_memory_cache *cache,
  	VM_BUG_ON(logging_active && !cache);
    	/* Create stage-2 page table mapping - Levels 0 and 1 */
+	pud = stage2_get_pud(kvm, cache, addr);
+	if (!pud) {

virt/kvm/arm/mmu.c static int stage2_set_pte(struct kvm *kvm, struct kvm_mmu_memory_cache *cache, · line 1161

Quoted 10 lines

virt/kvm/arm/mmu.c +10 -0

+		/*
+		 * Ignore calls from kvm_set_spte_hva for unallocated
+		 * address ranges.
+		 */
+		return 0;
+	}
+
+	/*
+	 * While dirty page logging - dissolve huge PUD, then continue
+	 * on to allocate page.

Punit,

We don't seem to allocate a page here for the PUD entry, in case if it is dissolved or empty (i.e, stage2_pud_none(*pud) is true.).

virt/kvm/arm/mmu.c static int stage2_set_pte(struct kvm *kvm, struct kvm_mmu_memory_cache *cache, · line 1168

Quoted 6 lines

virt/kvm/arm/mmu.c +4 -0

+	 */
+	if (logging_active)
+		stage2_dissolve_pud(kvm, addr, pud);
+
  	pmd = stage2_get_pmd(kvm, cache, addr);
  	if (!pmd) {

And once you add an entry, pmd is just the matter of getting stage2_pmd_offset() from your pud. No need to start again from the top-level with stage2_get_pmd().

Cheers
Suzuki

_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

Quoted 23 lines · Suzuki K Poulose

Quoted 22 lines · Punit Agrawal

KVM only supports PMD hugepages at stage 2. Now that the various page handling routines are updated, extend the stage 2 fault handling to map in PUD hugepages.

Addition of PUD hugepage support enables additional page sizes (e.g., 1G with 4K granule) which can be useful on cores that support mapping larger block sizes in the TLB entries.

Signed-off-by Punit Agrawal <[email protected]>

Cc Christoffer Dall <[email protected]>

Cc Marc Zyngier <[email protected]>

Cc Russell King <[email protected]>

Cc Catalin Marinas <[email protected]>

Cc Will Deacon <[email protected]>

---
   arch/arm/include/asm/kvm_mmu.h         | 19 +++++++
   arch/arm64/include/asm/kvm_mmu.h       | 15 +++++
   arch/arm64/include/asm/pgtable-hwdef.h |  2 +
   arch/arm64/include/asm/pgtable.h       |  2 +
   virt/kvm/arm/mmu.c                     | 78 ++++++++++++++++++++++++--
   5 files changed, 112 insertions(+), 4 deletions(-)

[...]

Quoted 4 lines

Quoted 4 lines

virt/kvm/arm/mmu.c +0 -0

index a6d3ac9d7c7a..d8e2497e5353 100644

[...]

Quoted 26 lines

Quoted 13 lines

virt/kvm/arm/mmu.c +3 -0

@@ -1100,6 +1139,7 @@ static int stage2_set_pte(struct kvm *kvm, struct kvm_mmu_memory_cache *cache,
   			  phys_addr_t addr, const pte_t *new_pte,
   			  unsigned long flags)
   {
+	pud_t *pud;
   	pmd_t *pmd;
   	pte_t *pte, old_pte;
   	bool iomap = flags & KVM_S2PTE_FLAG_IS_IOMAP;

@@ -1108,6 +1148,22 @@ static int stage2_set_pte(struct kvm *kvm, struct kvm_mmu_memory_cache *cache,
   	VM_BUG_ON(logging_active && !cache);
     	/* Create stage-2 page table mapping - Levels 0 and 1 */
+	pud = stage2_get_pud(kvm, cache, addr);
+	if (!pud) {

virt/kvm/arm/mmu.c static int stage2_set_pte(struct kvm *kvm, struct kvm_mmu_memory_cache *cache, · line 1161

Quoted 10 lines

virt/kvm/arm/mmu.c +10 -0

+		/*
+		 * Ignore calls from kvm_set_spte_hva for unallocated
+		 * address ranges.
+		 */
+		return 0;
+	}
+
+	/*
+	 * While dirty page logging - dissolve huge PUD, then continue
+	 * on to allocate page.

Punit,

Quoted 2 lines

We don't seem to allocate a page here for the PUD entry, in case if it is dissolved or empty (i.e, stage2_pud_none(*pud) is true.).

I was trying to avoid duplicating the PUD allocation by reusing the functionality in stage2_get_pmd().

Does the below updated comment help?

	/*
	 * While dirty page logging - dissolve huge PUD, it'll be
	 * allocated in stage2_get_pmd().
	 */

The other option is to duplicate the stage2_pud_none() case from stage2_get_pmd() here.

Quoted 74 lines · Punit Agrawal

Quoted 23 lines · Suzuki K Poulose

Quoted 22 lines · Punit Agrawal

KVM only supports PMD hugepages at stage 2. Now that the various page handling routines are updated, extend the stage 2 fault handling to map in PUD hugepages.

Addition of PUD hugepage support enables additional page sizes (e.g., 1G with 4K granule) which can be useful on cores that support mapping larger block sizes in the TLB entries.

Signed-off-by Punit Agrawal <[email protected]>

Cc Christoffer Dall <[email protected]>

Cc Marc Zyngier <[email protected]>

Cc Russell King <[email protected]>

Cc Catalin Marinas <[email protected]>

Cc Will Deacon <[email protected]>

---
   arch/arm/include/asm/kvm_mmu.h         | 19 +++++++
   arch/arm64/include/asm/kvm_mmu.h       | 15 +++++
   arch/arm64/include/asm/pgtable-hwdef.h |  2 +
   arch/arm64/include/asm/pgtable.h       |  2 +
   virt/kvm/arm/mmu.c                     | 78 ++++++++++++++++++++++++--
   5 files changed, 112 insertions(+), 4 deletions(-)

[...]

Quoted 4 lines

Quoted 4 lines

virt/kvm/arm/mmu.c +0 -0

index a6d3ac9d7c7a..d8e2497e5353 100644

[...]

Quoted 26 lines

Quoted 13 lines

virt/kvm/arm/mmu.c +3 -0

@@ -1100,6 +1139,7 @@ static int stage2_set_pte(struct kvm *kvm, struct kvm_mmu_memory_cache *cache,
   			  phys_addr_t addr, const pte_t *new_pte,
   			  unsigned long flags)
   {
+	pud_t *pud;
   	pmd_t *pmd;
   	pte_t *pte, old_pte;
   	bool iomap = flags & KVM_S2PTE_FLAG_IS_IOMAP;

@@ -1108,6 +1148,22 @@ static int stage2_set_pte(struct kvm *kvm, struct kvm_mmu_memory_cache *cache,
   	VM_BUG_ON(logging_active && !cache);
     	/* Create stage-2 page table mapping - Levels 0 and 1 */
+	pud = stage2_get_pud(kvm, cache, addr);
+	if (!pud) {

virt/kvm/arm/mmu.c static int stage2_set_pte(struct kvm *kvm, struct kvm_mmu_memory_cache *cache, · line 1161

Quoted 10 lines

virt/kvm/arm/mmu.c +10 -0

+		/*
+		 * Ignore calls from kvm_set_spte_hva for unallocated
+		 * address ranges.
+		 */
+		return 0;
+	}
+
+	/*
+	 * While dirty page logging - dissolve huge PUD, then continue
+	 * on to allocate page.

Punit,

Quoted 2 lines

We don't seem to allocate a page here for the PUD entry, in case if it is dissolved or empty (i.e, stage2_pud_none(*pud) is true.).

I was trying to avoid duplicating the PUD allocation by reusing the functionality in stage2_get_pmd().

Does the below updated comment help?

	/*
	 * While dirty page logging - dissolve huge PUD, it'll be
	 * allocated in stage2_get_pmd().
	 */

Quoted 2 lines

The other option is to duplicate the stage2_pud_none() case from stage2_get_pmd() here.

I think the explicit check for stage2_pud_none() suits better here. That would make it explicit that we are tearing down the entries from top to bottom. Also, we may be able to short cut for case where we know we just allocated a PUD page and hence we need another PMD level page.

Also, you are missing the comment about the assumption that stage2 PUD level always exist with 4k fixed IPA.

Cheers
Suzuki
_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

The code for operations such as marking the pfn as dirty, and dcache/icache maintenance during stage 2 fault handling is duplicated between normal pages and PMD hugepages.

Instead of creating another copy of the operations when we introduce PUD hugepages, let's share them across the different pagesizes.

Signed-off-by Punit Agrawal <[email protected]>

Cc Christoffer Dall <[email protected]>

Cc Marc Zyngier <[email protected]>

---
  virt/kvm/arm/mmu.c | 67 ++++++++++++++++++++++++++--------------------
  1 file changed, 38 insertions(+), 29 deletions(-)