kvm: Use huge pages for DAX-backed files

This change allows KVM to map DAX-backed files made of huge pages with huge mappings in the EPT/TDP.

DAX pages are not PageTransCompound. The existing check is trying to determine if the mapping for the pfn is a huge mapping or not. For non-DAX maps, e.g. hugetlbfs, that means checking PageTransCompound. For DAX, we can check the page table itself.

Note that KVM already faulted in the page (or huge page) in the host's page table, and we hold the KVM mmu spinlock. We grabbed that lock in kvm_mmu_notifier_invalidate_range_end, before checking the mmu seq.

Signed-off-by Barret Rhoden <[email protected]>

---
 arch/x86/kvm/mmu/mmu.c | 36 ++++++++++++++++++++++++++++++++----
 1 file changed, 32 insertions(+), 4 deletions(-)

arch/x86/kvm/mmu/mmu.c +14 -0

index 6f92b40d798c..cd07bc4e595f 100644
@@ -3384,6 +3384,35 @@ static int kvm_handle_bad_page(struct kvm_vcpu *vcpu, gfn_t gfn, kvm_pfn_t pfn)
 	return -EFAULT;
 }
 
+static bool pfn_is_huge_mapped(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn)
+{
+	struct page *page = pfn_to_page(pfn);
+	unsigned long hva;
+
+	if (!is_zone_device_page(page))
+		return PageTransCompoundMap(page);
+
+	/*
+	 * DAX pages do not use compound pages.  The page should have already
+	 * been mapped into the host-side page table during try_async_pf(), so
+	 * we can check the page tables directly.
+	 */
+	hva = gfn_to_hva(kvm, gfn);

arch/x86/kvm/mmu/mmu.c +7 -2

+		return true;
+	}
+	return false;
+}
+
 static void transparent_hugepage_adjust(struct kvm_vcpu *vcpu,
 					gfn_t gfn, kvm_pfn_t *pfnp,
 					int *levelp)

@@ -3398,8 +3427,8 @@ static void transparent_hugepage_adjust(struct kvm_vcpu *vcpu,
 	 * here.
 	 */
 	if (!is_error_noslot_pfn(pfn) && !kvm_is_reserved_pfn(pfn) &&
-	    !kvm_is_zone_device_pfn(pfn) && level == PT_PAGE_TABLE_LEVEL &&
-	    PageTransCompoundMap(pfn_to_page(pfn)) &&
+	    level == PT_PAGE_TABLE_LEVEL &&
+	    pfn_is_huge_mapped(vcpu->kvm, gfn, pfn) &&
 	    !mmu_gfn_lpage_is_disallowed(vcpu, gfn, PT_DIRECTORY_LEVEL)) {
 		unsigned long mask;
 		/*

arch/x86/kvm/mmu/mmu.c static int kvm_handle_bad_page(struct kvm_vcpu *vcpu, gfn_t gfn, kvm_pfn_t pfn) · line 3410

Quoted 7 lines

arch/x86/kvm/mmu/mmu.c +7 -0

+	/*
+	 * Our caller grabbed the KVM mmu_lock with a successful
+	 * mmu_notifier_retry, so we're safe to walk the page table.
+	 */
+	switch (dev_pagemap_mapping_shift(hva, current->mm)) {
+	case PMD_SHIFT:
+	case PUD_SIZE:

Shouldn't this be PUD_SHIFT?

But I agree with Paolo, that this is simply

return dev_pagemap_mapping_shift(hva, current->mm) > PAGE_SHIFT;

This change allows KVM to map DAX-backed files made of huge pages with huge mappings in the EPT/TDP.

DAX pages are not PageTransCompound. The existing check is trying to determine if the mapping for the pfn is a huge mapping or not. For non-DAX maps, e.g. hugetlbfs, that means checking PageTransCompound. For DAX, we can check the page table itself.

Note that KVM already faulted in the page (or huge page) in the host's page table, and we hold the KVM mmu spinlock. We grabbed that lock in kvm_mmu_notifier_invalidate_range_end, before checking the mmu seq.

Signed-off-by Barret Rhoden <[email protected]>

---
arch/x86/kvm/mmu/mmu.c | 36 ++++++++++++++++++++++++++++++++----

1 file changed, 32 insertions(+), 4 deletions(-)

arch/x86/kvm/mmu/mmu.c +12 -0

index 6f92b40d798c..cd07bc4e595f 100644
@@ -3384,6 +3384,35 @@ static int kvm_handle_bad_page(struct kvm_vcpu *vcpu, gfn_t gfn, kvm_pfn_t pfn)
 	return -EFAULT;
 }
 
+static bool pfn_is_huge_mapped(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn)
+{
+	struct page *page = pfn_to_page(pfn);
+	unsigned long hva;
+
+	if (!is_zone_device_page(page))
+		return PageTransCompoundMap(page);
+
+	/*
+	 * DAX pages do not use compound pages.  The page should have already
+	 * been mapped into the host-side page table during try_async_pf(), so
+	 * we can check the page tables directly.

arch/x86/kvm/mmu/mmu.c +10 -4

+	case PMD_SHIFT:
+	case PUD_SIZE:
+		return true;
+	}
+	return false;
+}
+
 static void transparent_hugepage_adjust(struct kvm_vcpu *vcpu,
 					gfn_t gfn, kvm_pfn_t *pfnp,
 					int *levelp)

@@ -3398,8 +3427,8 @@ static void transparent_hugepage_adjust(struct kvm_vcpu *vcpu,
 	 * here.
 	 */
 	if (!is_error_noslot_pfn(pfn) && !kvm_is_reserved_pfn(pfn) &&
-	    !kvm_is_zone_device_pfn(pfn) && level == PT_PAGE_TABLE_LEVEL &&
-	    PageTransCompoundMap(pfn_to_page(pfn)) &&
+	    level == PT_PAGE_TABLE_LEVEL &&
+	    pfn_is_huge_mapped(vcpu->kvm, gfn, pfn) &&
 	    !mmu_gfn_lpage_is_disallowed(vcpu, gfn, PT_DIRECTORY_LEVEL)) {
 		unsigned long mask;
 		/*

@@ -6015,8 +6044,7 @@ static bool kvm_mmu_zap_collapsible_spte(struct kvm *kvm,
 		 * mapping if the indirect sp has level = 1.
 		 */
 		if (sp->role.direct && !kvm_is_reserved_pfn(pfn) &&
-		    !kvm_is_zone_device_pfn(pfn) &&
-		    PageTransCompoundMap(pfn_to_page(pfn))) {
+		    pfn_is_huge_mapped(kvm, sp->gfn, pfn)) {
 			pte_list_remove(rmap_head, sptep);
 
 			if (kvm_available_flush_tlb_with_range())

Signature · 1 line

2.24.0.525.g8f36a354ae-goog

Quoted 12 lines · Barret Rhoden

This change allows KVM to map DAX-backed files made of huge pages with huge mappings in the EPT/TDP.

DAX pages are not PageTransCompound. The existing check is trying to determine if the mapping for the pfn is a huge mapping or not. For non-DAX maps, e.g. hugetlbfs, that means checking PageTransCompound. For DAX, we can check the page table itself.

For hugetlbfs pages, tdp_page_fault() -> mapping_level() -> host_mapping_level() -> kvm_host_page_size() -> vma_kernel_pagesize() will return the page-size of the hugetlbfs without the need to parse the page-tables.
See vma->vm_ops->pagesize() callback implementation at hugetlb_vm_ops->pagesize()==hugetlb_vm_op_pagesize().

Only for pages that were originally mapped as small-pages and later merged to larger pages by THP, there is a need to check for PageTransCompound(). Again, instead of parsing page-tables.

Therefore, it seems more logical to me that:
(a) If DAX-backed files are mapped as large-pages to userspace, it should be reflected in vma->vm_ops->page_size() of that mapping. Causing kvm_host_page_size() to return the right size without the need to parse the page-tables.

(b) If DAX-backed files small-pages can be later merged to large-pages by THP, then the “struct page” of these pages should be modified as usual to make PageTransCompound() return true for them. I’m not highly familiar with this mechanism, but I would expect THP to be able to merge DAX-backed files small-pages to large-pages in case DAX provides “struct page” for the DAX pages.

Quoted 31 lines

Note that KVM already faulted in the page (or huge page) in the host's page table, and we hold the KVM mmu spinlock. We grabbed that lock in kvm_mmu_notifier_invalidate_range_end, before checking the mmu seq.

Signed-off-by Barret Rhoden <[email protected]>

---
arch/x86/kvm/mmu/mmu.c | 36 ++++++++++++++++++++++++++++++++----

1 file changed, 32 insertions(+), 4 deletions(-)

arch/x86/kvm/mmu/mmu.c +12 -0

index 6f92b40d798c..cd07bc4e595f 100644
@@ -3384,6 +3384,35 @@ static int kvm_handle_bad_page(struct kvm_vcpu *vcpu, gfn_t gfn, kvm_pfn_t pfn)
      return -EFAULT;
 }
 
+static bool pfn_is_huge_mapped(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn)
+{
+     struct page *page = pfn_to_page(pfn);
+     unsigned long hva;
+
+     if (!is_zone_device_page(page))
+             return PageTransCompoundMap(page);
+
+     /*
+      * DAX pages do not use compound pages.  The page should have already
+      * been mapped into the host-side page table during try_async_pf(), so
+      * we can check the page tables directly.

arch/x86/kvm/mmu/mmu.c static int kvm_handle_bad_page(struct kvm_vcpu *vcpu, gfn_t gfn, kvm_pfn_t pfn) · line 3408

Quoted 10 lines

arch/x86/kvm/mmu/mmu.c +10 -0

+      */
+     hva = gfn_to_hva(kvm, gfn);
+     if (kvm_is_error_hva(hva))
+             return false;
+
+     /*
+      * Our caller grabbed the KVM mmu_lock with a successful
+      * mmu_notifier_retry, so we're safe to walk the page table.
+      */
+     switch (dev_pagemap_mapping_shift(hva, current->mm)) {

Doesn’t dev_pagemap_mapping_shift() get “struct page” as first parameter? Was this changed by a commit I missed?

-Liran

Quoted 30 lines

arch/x86/kvm/mmu/mmu.c +10 -4

+     case PMD_SHIFT:
+     case PUD_SIZE:
+             return true;
+     }
+     return false;
+}
+
 static void transparent_hugepage_adjust(struct kvm_vcpu *vcpu,
                                      gfn_t gfn, kvm_pfn_t *pfnp,
                                      int *levelp)

@@ -3398,8 +3427,8 @@ static void transparent_hugepage_adjust(struct kvm_vcpu *vcpu,
       * here.
       */
      if (!is_error_noslot_pfn(pfn) && !kvm_is_reserved_pfn(pfn) &&
-         !kvm_is_zone_device_pfn(pfn) && level == PT_PAGE_TABLE_LEVEL &&
-         PageTransCompoundMap(pfn_to_page(pfn)) &&
+         level == PT_PAGE_TABLE_LEVEL &&
+         pfn_is_huge_mapped(vcpu->kvm, gfn, pfn) &&
          !mmu_gfn_lpage_is_disallowed(vcpu, gfn, PT_DIRECTORY_LEVEL)) {
              unsigned long mask;
              /*

@@ -6015,8 +6044,7 @@ static bool kvm_mmu_zap_collapsible_spte(struct kvm *kvm,
               * mapping if the indirect sp has level = 1.
               */
              if (sp->role.direct && !kvm_is_reserved_pfn(pfn) &&
-                 !kvm_is_zone_device_pfn(pfn) &&
-                 PageTransCompoundMap(pfn_to_page(pfn))) {
+                 pfn_is_huge_mapped(kvm, sp->gfn, pfn)) {
                      pte_list_remove(rmap_head, sptep);
 

Quoted 4 lines

arch/x86/kvm/mmu/mmu.c +0 -1

                      if (kvm_available_flush_tlb_with_range())
--

2.24.0.525.g8f36a354ae-goog

Quoted 17 lines · Liran Alon

Quoted 12 lines · Barret Rhoden

This change allows KVM to map DAX-backed files made of huge pages with huge mappings in the EPT/TDP.

DAX pages are not PageTransCompound. The existing check is trying to determine if the mapping for the pfn is a huge mapping or not. For non-DAX maps, e.g. hugetlbfs, that means checking PageTransCompound. For DAX, we can check the page table itself.

For hugetlbfs pages, tdp_page_fault() -> mapping_level() -> host_mapping_level() -> kvm_host_page_size() -> vma_kernel_pagesize() will return the page-size of the hugetlbfs without the need to parse the page-tables.
See vma->vm_ops->pagesize() callback implementation at hugetlb_vm_ops->pagesize()==hugetlb_vm_op_pagesize().

Quoted 4 lines

Only for pages that were originally mapped as small-pages and later merged to larger pages by THP, there is a need to check for PageTransCompound(). Again, instead of parsing page-tables.

Therefore, it seems more logical to me that:
(a) If DAX-backed files are mapped as large-pages to userspace, it should be reflected in vma->vm_ops->page_size() of that mapping. Causing kvm_host_page_size() to return the right size without the need to parse the page-tables.

A given dax-mapped vma may have mixed page sizes so ->page_size() can't be used reliably to enumerating the mapping size.

Quoted 2 lines

(b) If DAX-backed files small-pages can be later merged to large-pages by THP, then the “struct page” of these pages should be modified as usual to make PageTransCompound() return true for them. I’m not highly familiar with this mechanism, but I would expect THP to be able to merge DAX-backed files small-pages to large-pages in case DAX provides “struct page” for the DAX pages.

DAX pages do not participate in THP and do not have the PageTransCompound accounting. The only mechanism that records the mapping size for dax is the page tables themselves.

Quoted 79 lines

Quoted 33 lines

Note that KVM already faulted in the page (or huge page) in the host's page table, and we hold the KVM mmu spinlock. We grabbed that lock in kvm_mmu_notifier_invalidate_range_end, before checking the mmu seq.

Signed-off-by Barret Rhoden <[email protected]>

---
arch/x86/kvm/mmu/mmu.c | 36 ++++++++++++++++++++++++++++++++----

1 file changed, 32 insertions(+), 4 deletions(-)

arch/x86/kvm/mmu/mmu.c +12 -0

index 6f92b40d798c..cd07bc4e595f 100644
@@ -3384,6 +3384,35 @@ static int kvm_handle_bad_page(struct kvm_vcpu *vcpu, gfn_t gfn, kvm_pfn_t pfn)
     return -EFAULT;
 }
 
+static bool pfn_is_huge_mapped(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn)
+{
+     struct page *page = pfn_to_page(pfn);
+     unsigned long hva;
+
+     if (!is_zone_device_page(page))
+             return PageTransCompoundMap(page);
+
+     /*
+      * DAX pages do not use compound pages.  The page should have already
+      * been mapped into the host-side page table during try_async_pf(), so
+      * we can check the page tables directly.

arch/x86/kvm/mmu/mmu.c static int kvm_handle_bad_page(struct kvm_vcpu *vcpu, gfn_t gfn, kvm_pfn_t pfn) · line 3408

Quoted 10 lines

arch/x86/kvm/mmu/mmu.c +10 -0

+      */
+     hva = gfn_to_hva(kvm, gfn);
+     if (kvm_is_error_hva(hva))
+             return false;
+
+     /*
+      * Our caller grabbed the KVM mmu_lock with a successful
+      * mmu_notifier_retry, so we're safe to walk the page table.
+      */
+     switch (dev_pagemap_mapping_shift(hva, current->mm)) {

Doesn’t dev_pagemap_mapping_shift() get “struct page” as first parameter? Was this changed by a commit I missed?

-Liran

Quoted 30 lines

arch/x86/kvm/mmu/mmu.c +10 -4

+     case PMD_SHIFT:
+     case PUD_SIZE:
+             return true;
+     }
+     return false;
+}
+
 static void transparent_hugepage_adjust(struct kvm_vcpu *vcpu,
                                     gfn_t gfn, kvm_pfn_t *pfnp,
                                     int *levelp)

@@ -3398,8 +3427,8 @@ static void transparent_hugepage_adjust(struct kvm_vcpu *vcpu,
      * here.
      */
     if (!is_error_noslot_pfn(pfn) && !kvm_is_reserved_pfn(pfn) &&
-         !kvm_is_zone_device_pfn(pfn) && level == PT_PAGE_TABLE_LEVEL &&
-         PageTransCompoundMap(pfn_to_page(pfn)) &&
+         level == PT_PAGE_TABLE_LEVEL &&
+         pfn_is_huge_mapped(vcpu->kvm, gfn, pfn) &&
         !mmu_gfn_lpage_is_disallowed(vcpu, gfn, PT_DIRECTORY_LEVEL)) {
             unsigned long mask;
             /*

@@ -6015,8 +6044,7 @@ static bool kvm_mmu_zap_collapsible_spte(struct kvm *kvm,
              * mapping if the indirect sp has level = 1.
              */
             if (sp->role.direct && !kvm_is_reserved_pfn(pfn) &&
-                 !kvm_is_zone_device_pfn(pfn) &&
-                 PageTransCompoundMap(pfn_to_page(pfn))) {
+                 pfn_is_huge_mapped(kvm, sp->gfn, pfn)) {
                     pte_list_remove(rmap_head, sptep);
 

Quoted 5 lines

Quoted 4 lines

arch/x86/kvm/mmu/mmu.c +0 -1

                     if (kvm_available_flush_tlb_with_range())
--

2.24.0.525.g8f36a354ae-goog

Quoted 25 lines · Dan Williams

Quoted 19 lines · Liran Alon

Quoted 12 lines · Barret Rhoden

This change allows KVM to map DAX-backed files made of huge pages with huge mappings in the EPT/TDP.

DAX pages are not PageTransCompound. The existing check is trying to determine if the mapping for the pfn is a huge mapping or not. For non-DAX maps, e.g. hugetlbfs, that means checking PageTransCompound. For DAX, we can check the page table itself.

For hugetlbfs pages, tdp_page_fault() -> mapping_level() -> host_mapping_level() -> kvm_host_page_size() -> vma_kernel_pagesize() will return the page-size of the hugetlbfs without the need to parse the page-tables.
See vma->vm_ops->pagesize() callback implementation at hugetlb_vm_ops->pagesize()==hugetlb_vm_op_pagesize().

Only for pages that were originally mapped as small-pages and later merged to larger pages by THP, there is a need to check for PageTransCompound(). Again, instead of parsing page-tables.

Quoted 5 lines

Quoted 2 lines

Therefore, it seems more logical to me that:
(a) If DAX-backed files are mapped as large-pages to userspace, it should be reflected in vma->vm_ops->page_size() of that mapping. Causing kvm_host_page_size() to return the right size without the need to parse the page-tables.

A given dax-mapped vma may have mixed page sizes so ->page_size() can't be used reliably to enumerating the mapping size.

Naive question: Why don’t split the VMA in this case to multiple VMAs with different results for ->page_size()?

What you are describing sounds like DAX is breaking this callback semantics in an unpredictable manner.

Quoted 5 lines

Quoted 1 line

(b) If DAX-backed files small-pages can be later merged to large-pages by THP, then the “struct page” of these pages should be modified as usual to make PageTransCompound() return true for them. I’m not highly familiar with this mechanism, but I would expect THP to be able to merge DAX-backed files small-pages to large-pages in case DAX provides “struct page” for the DAX pages.

DAX pages do not participate in THP and do not have the PageTransCompound accounting. The only mechanism that records the mapping size for dax is the page tables themselves.

What is the rational behind this? Given that DAX pages can be described with “struct page” (i.e. ZONE_DEVICE), what prevents THP from manipulating page-tables to merge multiple DAX PFNs to a larger page?

Quoted 28 lines · Liran Alon

Quoted 28 lines · Dan Williams

Quoted 21 lines · Liran Alon

Quoted 12 lines · Barret Rhoden

This change allows KVM to map DAX-backed files made of huge pages with huge mappings in the EPT/TDP.

DAX pages are not PageTransCompound. The existing check is trying to determine if the mapping for the pfn is a huge mapping or not. For non-DAX maps, e.g. hugetlbfs, that means checking PageTransCompound. For DAX, we can check the page table itself.

For hugetlbfs pages, tdp_page_fault() -> mapping_level() -> host_mapping_level() -> kvm_host_page_size() -> vma_kernel_pagesize() will return the page-size of the hugetlbfs without the need to parse the page-tables.
See vma->vm_ops->pagesize() callback implementation at hugetlb_vm_ops->pagesize()==hugetlb_vm_op_pagesize().

Only for pages that were originally mapped as small-pages and later merged to larger pages by THP, there is a need to check for PageTransCompound(). Again, instead of parsing page-tables.

Therefore, it seems more logical to me that:
(a) If DAX-backed files are mapped as large-pages to userspace, it should be reflected in vma->vm_ops->page_size() of that mapping. Causing kvm_host_page_size() to return the right size without the need to parse the page-tables.

Quoted 4 lines

Quoted 2 lines

A given dax-mapped vma may have mixed page sizes so ->page_size() can't be used reliably to enumerating the mapping size.

Naive question: Why don’t split the VMA in this case to multiple VMAs with different results for ->page_size()?

Filesystems traditionally have not populated ->pagesize() in their vm_operations, there was no compelling reason to go add it and the complexity seems prohibitive.

Quoted 2 lines

What you are describing sounds like DAX is breaking this callback semantics in an unpredictable manner.

It's not unpredictable. vma_kernel_pagesize() returns PAGE_SIZE.

Huge
pages in the page cache has a similar issue.

Quoted 7 lines

Quoted 6 lines

Quoted 2 lines

(b) If DAX-backed files small-pages can be later merged to large-pages by THP, then the “struct page” of these pages should be modified as usual to make PageTransCompound() return true for them. I’m not highly familiar with this mechanism, but I would expect THP to be able to merge DAX-backed files small-pages to large-pages in case DAX provides “struct page” for the DAX pages.

DAX pages do not participate in THP and do not have the PageTransCompound accounting. The only mechanism that records the mapping size for dax is the page tables themselves.

Quoted 1 line

What is the rational behind this? Given that DAX pages can be described with “struct page” (i.e. ZONE_DEVICE), what prevents THP from manipulating page-tables to merge multiple DAX PFNs to a larger page?

THP accounting is a function of the page allocator. ZONE_DEVICE pages are excluded from the page allocator. ZONE_DEVICE is just enough infrastructure to support pfn_to_page(), page_address(), and get_user_pages(). Other page allocator services beyond that are not present.

arch/x86/kvm/mmu/mmu.c static int kvm_handle_bad_page(struct kvm_vcpu *vcpu, gfn_t gfn, kvm_pfn_t pfn) · line 3408

Quoted 5 lines

arch/x86/kvm/mmu/mmu.c +5 -0

+	/*
+	 * Our caller grabbed the KVM mmu_lock with a successful
+	 * mmu_notifier_retry, so we're safe to walk the page table.
+	 */
+	switch (dev_pagemap_mapping_shift(hva, current->mm)) {

Doesn’t dev_pagemap_mapping_shift() get “struct page” as first parameter? Was this changed by a commit I missed?

This change allows KVM to map DAX-backed files made of huge pages with huge mappings in the EPT/TDP.

DAX pages are not PageTransCompound. The existing check is trying to determine if the mapping for the pfn is a huge mapping or not. For non-DAX maps, e.g. hugetlbfs, that means checking PageTransCompound. For DAX, we can check the page table itself.

Note that KVM already faulted in the page (or huge page) in the host's page table, and we hold the KVM mmu spinlock. We grabbed that lock in kvm_mmu_notifier_invalidate_range_end, before checking the mmu seq.

Signed-off-by Barret Rhoden <[email protected]>

---
 arch/x86/kvm/mmu/mmu.c | 36 ++++++++++++++++++++++++++++++++----
 1 file changed, 32 insertions(+), 4 deletions(-)

arch/x86/kvm/mmu/mmu.c +14 -0

index 6f92b40d798c..cd07bc4e595f 100644
@@ -3384,6 +3384,35 @@ static int kvm_handle_bad_page(struct kvm_vcpu *vcpu, gfn_t gfn, kvm_pfn_t pfn)
 	return -EFAULT;
 }
 
+static bool pfn_is_huge_mapped(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn)
+{
+	struct page *page = pfn_to_page(pfn);
+	unsigned long hva;
+
+	if (!is_zone_device_page(page))
+		return PageTransCompoundMap(page);
+
+	/*
+	 * DAX pages do not use compound pages.  The page should have already
+	 * been mapped into the host-side page table during try_async_pf(), so
+	 * we can check the page tables directly.
+	 */
+	hva = gfn_to_hva(kvm, gfn);

arch/x86/kvm/mmu/mmu.c +8 -4

+		return true;
+	}
+	return false;
+}
+
 static void transparent_hugepage_adjust(struct kvm_vcpu *vcpu,
 					gfn_t gfn, kvm_pfn_t *pfnp,
 					int *levelp)

@@ -3398,8 +3427,8 @@ static void transparent_hugepage_adjust(struct kvm_vcpu *vcpu,
 	 * here.
 	 */
 	if (!is_error_noslot_pfn(pfn) && !kvm_is_reserved_pfn(pfn) &&
-	    !kvm_is_zone_device_pfn(pfn) && level == PT_PAGE_TABLE_LEVEL &&
-	    PageTransCompoundMap(pfn_to_page(pfn)) &&
+	    level == PT_PAGE_TABLE_LEVEL &&
+	    pfn_is_huge_mapped(vcpu->kvm, gfn, pfn) &&
 	    !mmu_gfn_lpage_is_disallowed(vcpu, gfn, PT_DIRECTORY_LEVEL)) {
 		unsigned long mask;
 		/*

@@ -6015,8 +6044,7 @@ static bool kvm_mmu_zap_collapsible_spte(struct kvm *kvm,
 		 * mapping if the indirect sp has level = 1.
 		 */
 		if (sp->role.direct && !kvm_is_reserved_pfn(pfn) &&
-		    !kvm_is_zone_device_pfn(pfn) &&
-		    PageTransCompoundMap(pfn_to_page(pfn))) {
+		    pfn_is_huge_mapped(kvm, sp->gfn, pfn)) {
 			pte_list_remove(rmap_head, sptep);
 
 			if (kvm_available_flush_tlb_with_range())

Signature · 1 line

2.24.0.525.g8f36a354ae-goog

Quoted 43 lines · Barret Rhoden

This change allows KVM to map DAX-backed files made of huge pages with huge mappings in the EPT/TDP.

DAX pages are not PageTransCompound. The existing check is trying to determine if the mapping for the pfn is a huge mapping or not. For non-DAX maps, e.g. hugetlbfs, that means checking PageTransCompound. For DAX, we can check the page table itself.

Note that KVM already faulted in the page (or huge page) in the host's page table, and we hold the KVM mmu spinlock. We grabbed that lock in kvm_mmu_notifier_invalidate_range_end, before checking the mmu seq.

Signed-off-by Barret Rhoden <[email protected]>

---
 arch/x86/kvm/mmu/mmu.c | 36 ++++++++++++++++++++++++++++++++----
 1 file changed, 32 insertions(+), 4 deletions(-)

arch/x86/kvm/mmu/mmu.c +18 -0

index 6f92b40d798c..cd07bc4e595f 100644
@@ -3384,6 +3384,35 @@ static int kvm_handle_bad_page(struct kvm_vcpu *vcpu, gfn_t gfn, kvm_pfn_t pfn)
      return -EFAULT;
 }
 
+static bool pfn_is_huge_mapped(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn)
+{
+     struct page *page = pfn_to_page(pfn);
+     unsigned long hva;
+
+     if (!is_zone_device_page(page))
+             return PageTransCompoundMap(page);
+
+     /*
+      * DAX pages do not use compound pages.  The page should have already
+      * been mapped into the host-side page table during try_async_pf(), so
+      * we can check the page tables directly.
+      */
+     hva = gfn_to_hva(kvm, gfn);
+     if (kvm_is_error_hva(hva))
+             return false;
+
+     /*

arch/x86/kvm/mmu/mmu.c static int kvm_handle_bad_page(struct kvm_vcpu *vcpu, gfn_t gfn, kvm_pfn_t pfn) · line 3410

Quoted 6 lines

arch/x86/kvm/mmu/mmu.c +6 -0

+      * Our caller grabbed the KVM mmu_lock with a successful
+      * mmu_notifier_retry, so we're safe to walk the page table.
+      */
+     switch (dev_pagemap_mapping_shift(hva, current->mm)) {
+     case PMD_SHIFT:
+     case PUD_SIZE:

I assume this means DAX can have 1GB pages?

I ask because KVM's THP logic
has historically relied on THP only supporting 2MB. I cleaned this up in a recent series[*], which is in kvm/queue, but I obviously didn't actually test whether or not KVM would correctly handle 1GB non-hugetlbfs pages.

Yeah, since device-dax is the only path to support longterm page pinning for vfio device assignment, testing with device-dax + 1GB pages would be a useful sanity check.

Quoted 3 lines · Dan Williams

Yeah, since device-dax is the only path to support longterm page pinning for vfio device assignment, testing with device-dax + 1GB pages would be a useful sanity check.

What are the issues with fs-dax and page pinning? Is that limitation something that is permanent and unfixable (by me or anyone)?

I'd like to put a lot more in a DAX/pmem region than just a guest's memory, and having a mountable filesystem would be extremely convenient.

Quoted 8 lines · Barret Rhoden

Quoted 3 lines · Dan Williams

Yeah, since device-dax is the only path to support longterm page pinning for vfio device assignment, testing with device-dax + 1GB pages would be a useful sanity check.

What are the issues with fs-dax and page pinning? Is that limitation something that is permanent and unfixable (by me or anyone)?

It's a surprisingly painful point of contention...

Quoted 2 lines

I'd like to put a lot more in a DAX/pmem region than just a guest's memory, and having a mountable filesystem would be extremely convenient.

Why would page pinning be involved in allowing the guest to mount a filesystem on guest-pmem? That already works today, it's just the device-passthrough that causes guest memory to be pinned indefinitely.

Quoted 46 lines · Barret Rhoden

This change allows KVM to map DAX-backed files made of huge pages with huge mappings in the EPT/TDP.

DAX pages are not PageTransCompound. The existing check is trying to determine if the mapping for the pfn is a huge mapping or not. For non-DAX maps, e.g. hugetlbfs, that means checking PageTransCompound. For DAX, we can check the page table itself.

Note that KVM already faulted in the page (or huge page) in the host's page table, and we hold the KVM mmu spinlock. We grabbed that lock in kvm_mmu_notifier_invalidate_range_end, before checking the mmu seq.

Signed-off-by Barret Rhoden <[email protected]>

---
arch/x86/kvm/mmu/mmu.c | 36 ++++++++++++++++++++++++++++++++----

1 file changed, 32 insertions(+), 4 deletions(-)

arch/x86/kvm/mmu/mmu.c +21 -0

index 6f92b40d798c..cd07bc4e595f 100644
@@ -3384,6 +3384,35 @@ static int kvm_handle_bad_page(struct kvm_vcpu *vcpu, gfn_t gfn, kvm_pfn_t pfn)
 	return -EFAULT;
 }
 
+static bool pfn_is_huge_mapped(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn)
+{
+	struct page *page = pfn_to_page(pfn);
+	unsigned long hva;
+
+	if (!is_zone_device_page(page))
+		return PageTransCompoundMap(page);
+
+	/*
+	 * DAX pages do not use compound pages.  The page should have already
+	 * been mapped into the host-side page table during try_async_pf(), so
+	 * we can check the page tables directly.
+	 */
+	hva = gfn_to_hva(kvm, gfn);
+	if (kvm_is_error_hva(hva))
+		return false;
+
+	/*
+	 * Our caller grabbed the KVM mmu_lock with a successful
+	 * mmu_notifier_retry, so we're safe to walk the page table.
+	 */

arch/x86/kvm/mmu/mmu.c static int kvm_handle_bad_page(struct kvm_vcpu *vcpu, gfn_t gfn, kvm_pfn_t pfn) · line 3410

Quoted 3 lines

arch/x86/kvm/mmu/mmu.c +3 -0

+	switch (dev_pagemap_mapping_shift(hva, current->mm)) {
+	case PMD_SHIFT:
+	case PUD_SIZE:

I assume this means DAX can have 1GB pages? I ask because KVM's THP logic has historically relied on THP only supporting 2MB. I cleaned this up in a recent series[*], which is in kvm/queue, but I obviously didn't actually test whether or not KVM would correctly handle 1GB non-hugetlbfs pages.

The easiest thing is probably to rebase on kvm/queue. You'll need to do that anyways, and I suspect doing so will help shake out any hiccups.

[*] https://urldefense.proofpoint.com/v2/url?u=https-3A__lkml.kernel.org_r_20191206235729.29263-2D1-2Dsean.j.christopherson-40intel.com&d=DwIBAg&c=RoP1YumCXCgaWHvlZYR8PZh8Bv7qIrMUB65eapI_JnE&r=Jk6Q8nNzkQ6LJ6g42qARkg6ryIDGQr-yKXPNGZbpTx0&m=Lk-PXE125WU3GWJOV4U4crsSEFx7f5AUmRJhkrfIeAE&s=BIo4tnL4OfswRQ2QKfTs9VYScLU5lBy2pwzePBnHow8&e=

Quoted 28 lines

arch/x86/kvm/mmu/mmu.c +8 -4

+		return true;
+	}
+	return false;
+}
+
 static void transparent_hugepage_adjust(struct kvm_vcpu *vcpu,
 					gfn_t gfn, kvm_pfn_t *pfnp,
 					int *levelp)

@@ -3398,8 +3427,8 @@ static void transparent_hugepage_adjust(struct kvm_vcpu *vcpu,
 	 * here.
 	 */
 	if (!is_error_noslot_pfn(pfn) && !kvm_is_reserved_pfn(pfn) &&
-	    !kvm_is_zone_device_pfn(pfn) && level == PT_PAGE_TABLE_LEVEL &&
-	    PageTransCompoundMap(pfn_to_page(pfn)) &&
+	    level == PT_PAGE_TABLE_LEVEL &&
+	    pfn_is_huge_mapped(vcpu->kvm, gfn, pfn) &&
 	    !mmu_gfn_lpage_is_disallowed(vcpu, gfn, PT_DIRECTORY_LEVEL)) {
 		unsigned long mask;
 		/*

@@ -6015,8 +6044,7 @@ static bool kvm_mmu_zap_collapsible_spte(struct kvm *kvm,
 		 * mapping if the indirect sp has level = 1.
 		 */
 		if (sp->role.direct && !kvm_is_reserved_pfn(pfn) &&
-		    !kvm_is_zone_device_pfn(pfn) &&
-		    PageTransCompoundMap(pfn_to_page(pfn))) {
+		    pfn_is_huge_mapped(kvm, sp->gfn, pfn)) {
 			pte_list_remove(rmap_head, sptep);
 

Quoted 4 lines

arch/x86/kvm/mmu/mmu.c +0 -0

 			if (kvm_available_flush_tlb_with_range())

Signature · 1 line

2.24.0.525.g8f36a354ae-goog

This patchset allows KVM to map huge pages for DAX-backed files.

I held previous versions in limbo while people were sorting out whether or not DAX pages were going to remain PageReserved and how that relates to KVM.

Now that that is sorted out (DAX pages are PageReserved, but they are not kvm_is_reserved_pfn(), and DAX pages are considered on a case-by-case basis for KVM), I can repost this.

v3 -> v4:
v3: https://lore.kernel.org/lkml/[email protected]/
- Rebased onto linus/master

v2 -> v3:
v2: https://lore.kernel.org/lkml/[email protected]/
- Updated Acks/Reviewed-by
- Rebased onto linux-next

v1 -> v2:
https://lore.kernel.org/lkml/[email protected]/
- Updated Acks/Reviewed-by
- Minor touchups
- Added patch to remove redundant PageReserved() check
- Rebased onto linux-next

RFC/discussion thread:
https://lore.kernel.org/lkml/[email protected]/

Barret Rhoden (2):
mm: make dev_pagemap_mapping_shift() externally visible
kvm: Use huge pages for DAX-backed files

 arch/x86/kvm/mmu/mmu.c | 36 ++++++++++++++++++++++++++++++++----
 include/linux/mm.h     |  3 +++
 mm/memory-failure.c    | 38 +++-----------------------------------
 mm/util.c              | 34 ++++++++++++++++++++++++++++++++++
 4 files changed, 72 insertions(+), 39 deletions(-)