hugetlb: introduce HugeTLB high-granularity mapping
Subsystems (26)
v2 cover hugetlb: introduce HugeTLB high-granularity mapping
This RFC v2 is a more complete and correct implementation of the original high-granularity mapping RFC[1]. For HGM background and motivation, please see the original RFC.
This series has changed quite significantly since its first version, so I've dropped all the Reviewed-bys that it picked up, and I am not including a full changelog here. Some notable changes:
1. mapcount rules have been simplified (now: the number of times a hugepage is referenced in page tables, still tracked on the head page).
2. Synchronizing page table collapsing is now done using the VMA lock that Mike introduced recently.
3. PTE splitting is only supported for blank PTEs, and it is done without needing to hold the VMA lock for writing. In many places, we explicitly check if a PTE has been split from under us.
4. The userspace API has changed slightly.
This series implements high-granularity mapping basics, enough to support PAGE_SIZE-aligned UFFDIO_CONTINUE operations and MADV_COLLAPSE for shared HugeTLB VMAs for x86. The main use case for this is post-copy for virtual machines, one of the important HGM use cases described in [1]. MADV_COLLAPSE was originally introduced for THPs[2], but it is now meaningful for HGM, and so I am co-opting the same API.
- Userspace API
There are two main ways userspace interacts with high-granularity mappings:
1. Create them with UFFDIO_CONTINUE in an apporiately configured userfaultfd VMA.
2. Collapse high-granularity mappings with MADV_COLLAPSE.
The userfaultfd bits of the userspace API have changed slightly since RFC v1. To configure a userfaultfd VMA to enable HGM, userspace must provide UFFD_FEATURE_MINOR_HUGETLBFS_HGM and UFFD_FEATURE_EXACT_ADDRESS in its call to UFFDIO_API.
- A Note About KVM
Normally KVM (as well as any other non-HugeTLB code that assumes that HugeTLB pages will always be mapped with huge PTEs) would need to be enlightened to do the correct thing with high-granularity-mapped HugeTLB pages. It turns out that the x86 TDP MMU already handles HGM mappings correctly, but other architectures' KVM MMUs, like arm64's, will need to be updated before HGM can be enabled for those architectures.
- How complete is this series?
I have tested this series with the self-tests that I have modified and added, and I have run real, large end-to-end migration tests. This series should be mostly stable, though I haven't tested DAMON and other pieces that were slightly changed by this series.
There is a bug in the current x86 TDP MMU that prevents MADV_COLLAPSE from having an effect. That is, the second-stage mappings will remain small. This will be fixed with [3], so unless you have [3] merged in your tree, you will see that MADV_COLLAPSE does not impact on virtual machine performance.
- Future Work
The main areas of future work are:
1) Support more architectures (arm64 support is mostly complete, but supporting it is not trivial, and to keep this RFC as short as possible, I will send the arm64 support series separately).
2) Improve performance. Right now we take two per-hpage locks in the hotpath for userfaultfd-based post-copy live migration, the page lock and the fault mutex. To improve post-copy performance as much as possible, we likely need to improve this locking strategy.
3) Support PAGE_SIZE poisoning of HugeTLB pages. To provide userspace with consistent poison behavior whether using MAP_PRIVATE or MAP_SHARED, more work is needed to implement basic HGM support for MAP_PRIVATE mappings.
- Patches
Patches 1-4: Cleanup.
Patches 5-6: Extend the HugeTLB shared VMA lock struct.
Patches 7-14: Create hugetlb_pte and implement HGM basics (PT walking,
enabling HGM).
Patches 15-30: Make existing routines compatible with HGM.
Patches 31-35: Extend userfaultfd to support high-granularity CONTINUEs.
Patch 36: Add HugeTLB HGM support to MADV_COLLAPSE.
Patches 37-40: Cleanup, add HGM stats, and enable HGM for x86.
Patches 41-47: Documentation and selftests.
This series is based on mm-everything-2022-10-20-00-43.
Finally, I will be on vacation next week (until Nov 2, unfortunate timing). I will try to respond before Nov 2; I wanted to get this series up ASAP.
[1] https://lore.kernel.org/linux-mm/[email protected]/
[2] commit 7d8faaf155454 ("mm/madvise: introduce MADV_COLLAPSE sync hugepage collapse")
[3] https://lore.kernel.org/kvm/[email protected]/
James Houghton (47):
hugetlb: don't set PageUptodate for UFFDIO_CONTINUE
hugetlb: remove mk_huge_pte; it is unused
hugetlb: remove redundant pte_mkhuge in migration path
hugetlb: only adjust address ranges when VMAs want PMD sharing
hugetlb: make hugetlb_vma_lock_alloc return its failure reason
hugetlb: extend vma lock for shared vmas
hugetlb: add CONFIG_HUGETLB_HIGH_GRANULARITY_MAPPING
hugetlb: add HGM enablement functions
hugetlb: make huge_pte_lockptr take an explicit shift argument.
hugetlb: add hugetlb_pte to track HugeTLB page table entries
hugetlb: add hugetlb_pmd_alloc and hugetlb_pte_alloc
hugetlb: add hugetlb_hgm_walk and hugetlb_walk_step
hugetlb: add make_huge_pte_with_shift
hugetlb: make default arch_make_huge_pte understand small mappings
hugetlbfs: for unmapping, treat HGM-mapped pages as potentially mapped
hugetlb: make unmapping compatible with high-granularity mappings
hugetlb: make hugetlb_change_protection compatible with HGM
hugetlb: enlighten follow_hugetlb_page to support HGM
hugetlb: make hugetlb_follow_page_mask HGM-enabled
hugetlb: use struct hugetlb_pte for walk_hugetlb_range
mm: rmap: provide pte_order in page_vma_mapped_walk
mm: rmap: make page_vma_mapped_walk callers use pte_order
rmap: update hugetlb lock comment for HGM
hugetlb: update page_vma_mapped to do high-granularity walks
hugetlb: add HGM support for copy_hugetlb_page_range
hugetlb: make move_hugetlb_page_tables compatible with HGM
hugetlb: add HGM support for hugetlb_fault and hugetlb_no_page
rmap: in try_to_{migrate,unmap}_one, check head page for page flags
hugetlb: add high-granularity migration support
hugetlb: add high-granularity check for hwpoison in fault path
hugetlb: sort hstates in hugetlb_init_hstates
hugetlb: add for_each_hgm_shift
userfaultfd: add UFFD_FEATURE_MINOR_HUGETLBFS_HGM
hugetlb: userfaultfd: add support for high-granularity UFFDIO_CONTINUE
userfaultfd: require UFFD_FEATURE_EXACT_ADDRESS when using HugeTLB HGM
hugetlb: add MADV_COLLAPSE for hugetlb
hugetlb: remove huge_pte_lock and huge_pte_lockptr
hugetlb: replace make_huge_pte with make_huge_pte_with_shift
mm: smaps: add stats for HugeTLB mapping size
hugetlb: x86: enable high-granularity mapping
docs: hugetlb: update hugetlb and userfaultfd admin-guides with HGM
info
docs: proc: include information about HugeTLB HGM selftests/vm: add HugeTLB HGM to userfaultfd selftest selftests/kvm: add HugeTLB HGM to KVM demand paging selftest selftests/vm: add anon and shared hugetlb to migration test selftests/vm: add hugetlb HGM test to migration selftest selftests/vm: add HGM UFFDIO_CONTINUE and hwpoison tests
Documentation/admin-guide/mm/hugetlbpage.rst | 4 + Documentation/admin-guide/mm/userfaultfd.rst | 16 +- Documentation/filesystems/proc.rst | 56 +- arch/powerpc/mm/pgtable.c | 3 +- arch/s390/include/asm/hugetlb.h | 5 - arch/s390/mm/gmap.c | 20 +- arch/x86/Kconfig | 1 + fs/Kconfig | 7 + fs/hugetlbfs/inode.c | 27 +- fs/proc/task_mmu.c | 184 ++- fs/userfaultfd.c | 56 +- include/asm-generic/hugetlb.h | 5 - include/asm-generic/tlb.h | 6 +- include/linux/huge_mm.h | 12 +- include/linux/hugetlb.h | 173 ++- include/linux/pagewalk.h | 11 +- include/linux/rmap.h | 5 + include/linux/swapops.h | 8 +- include/linux/userfaultfd_k.h | 7 + include/uapi/linux/userfaultfd.h | 2 + mm/damon/vaddr.c | 57 +- mm/debug_vm_pgtable.c | 2 +- mm/hmm.c | 21 +- mm/hugetlb.c | 1209 ++++++++++++++--- mm/khugepaged.c | 4 +- mm/madvise.c | 24 +- mm/memory-failure.c | 17 +- mm/mempolicy.c | 28 +- mm/migrate.c | 20 +- mm/mincore.c | 17 +- mm/mprotect.c | 18 +- mm/page_vma_mapped.c | 60 +- mm/pagewalk.c | 32 +- mm/rmap.c | 102 +- mm/userfaultfd.c | 46 +- .../selftests/kvm/demand_paging_test.c | 20 +- .../testing/selftests/kvm/include/test_util.h | 2 + tools/testing/selftests/kvm/lib/kvm_util.c | 2 +- tools/testing/selftests/kvm/lib/test_util.c | 14 + tools/testing/selftests/vm/Makefile | 1 + tools/testing/selftests/vm/hugetlb-hgm.c | 326 +++++ tools/testing/selftests/vm/migration.c | 222 ++- tools/testing/selftests/vm/userfaultfd.c | 90 +- 43 files changed, 2449 insertions(+), 493 deletions(-) create mode 100644 tools/testing/selftests/vm/hugetlb-hgm.c
Signature · 1 line
2.38.0.135.g90850a2211-goog
RFC PATCH v2 1/47 hugetlb: don't set PageUptodate for UFFDIO_CONTINUE
This is how it should have been to begin with. It would be very bad if we actually set PageUptodate with a UFFDIO_CONTINUE, as UFFDIO_CONTINUE doesn't actually set/update the contents of the page, so we would be exposing a non-zeroed page to the user.
The reason this change is being made now is because UFFDIO_CONTINUEs on subpages definitely shouldn't set this page flag on the head page.
--- mm/hugetlb.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-)
mm/hugetlb.c +4 -1
@@ -6097,7 +6097,10 @@ int hugetlb_mcopy_atomic_pte(struct mm_struct *dst_mm,* preceding stores to the page contents become visible before * the set_pte_at() write. */ -__SetPageUptodate (page); +if (!is_continue) +__SetPageUptodate (page); +else +VM_WARN_ON_ONCE_PAGE (!PageUptodate (page), page);/* Add shared, newly allocated pages to the page cache. */ if (vm_shared && !is_continue) {
Signature · 1 line
2.38.0.135.g90850a2211-goog
Peter Xu
reviewerRe: [RFC PATCH v2 01/47] hugetlb: don't set PageUptodate for UFFDIO_CONTINUE
Quoted 18 lines · James Houghton
This is how it should have been to begin with. It would be very bad if we actually set PageUptodate with a
UFFDIO_CONTINUE, asUFFDIO_CONTINUEdoesn't actually set/update the contents of the page, so we would be exposing a non-zeroed page to the user.The reason this change is being made now is because
UFFDIO_CONTINUEson subpages definitely shouldn't set this page flag on the head page.Signed-off-by James Houghton <[email protected]>--- mm/hugetlb.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-)mm/hugetlb.c +0
-0
mm/hugetlb.c int hugetlb_mcopy_atomic_pte(struct mm_struct *dst_mm, · line 6103
Quoted 8 lines
mm/hugetlb.c +4
-1@@ -6097,7 +6097,10 @@ int hugetlb_mcopy_atomic_pte(struct mm_struct *dst_mm,* preceding stores to the page contents become visible before * the set_pte_at() write. */ -__SetPageUptodate (page); +if (!is_continue) +__SetPageUptodate (page); +else +VM_WARN_ON_ONCE_PAGE (!PageUptodate (page), page);
Yeah the old code looks wrong, I'm just wondering whether we can 100% guarantee this for hugetlb. E.g. for shmem that won't hold when we uffd-continue on a not used page (e.g. by an over-sized fallocate()).
Another safer approach is simply fail the ioctl if !uptodate, but if you're certain then WARN_ON_ONCE sounds all good too. At least I did have a quick look on hugetlb fallocate() and pages will be uptodate immediately.
Quoted 7 lines
mm/hugetlb.c +0
-0/* Add shared, newly allocated pages to the page cache. */ if (vm_shared && !is_continue) {Signature · 1 line
2.38.0.135.g90850a2211-goog
Signature · 1 line
Peter Xu
Re: [RFC PATCH v2 01/47] hugetlb: don't set PageUptodate for UFFDIO_CONTINUE
Quoted 33 lines · Peter Xu
Quoted 18 lines · James Houghton
This is how it should have been to begin with. It would be very bad if we actually set PageUptodate with a
UFFDIO_CONTINUE, asUFFDIO_CONTINUEdoesn't actually set/update the contents of the page, so we would be exposing a non-zeroed page to the user.The reason this change is being made now is because
UFFDIO_CONTINUEson subpages definitely shouldn't set this page flag on the head page.Signed-off-by James Houghton <[email protected]>--- mm/hugetlb.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-)mm/hugetlb.c +0
-0mm/hugetlb.c int hugetlb_mcopy_atomic_pte(struct mm_struct *dst_mm, · line 6103
Quoted 8 lines
mm/hugetlb.c +4
-1@@ -6097,7 +6097,10 @@ int hugetlb_mcopy_atomic_pte(struct mm_struct *dst_mm,* preceding stores to the page contents become visible before * the set_pte_at() write. */ -__SetPageUptodate (page); +if (!is_continue) +__SetPageUptodate (page); +else +VM_WARN_ON_ONCE_PAGE (!PageUptodate (page), page);Yeah the old code looks wrong, I'm just wondering whether we can 100% guarantee this for hugetlb. E.g. for shmem that won't hold when we uffd-continue on a not used page (e.g. by an over-sized
fallocate()).
Quoted 3 lines
Another safer approach is simply fail the ioctl if !uptodate, but if you're certain then
WARN_ON_ONCEsounds all good too. At least I did have a quick look on hugetlbfallocate()and pages will be uptodate immediately.
Failing the ioctl sounds better than only WARNing. I'll do that and drop the WARN_ON_ONCE for v1. Thanks!
- James
Quoted 12 lines
Quoted 8 lines
mm/hugetlb.c +0
-1/* Add shared, newly allocated pages to the page cache. */ if (vm_shared && !is_continue) { --2.38.0.135.g90850a2211-goog
--
Peter Xu
Re: [RFC PATCH v2 01/47] hugetlb: don't set PageUptodate for UFFDIO_CONTINUE
Quoted 34 lines · James Houghton
Quoted 33 lines · Peter Xu
Quoted 18 lines · James Houghton
This is how it should have been to begin with. It would be very bad if we actually set PageUptodate with a
UFFDIO_CONTINUE, asUFFDIO_CONTINUEdoesn't actually set/update the contents of the page, so we would be exposing a non-zeroed page to the user.The reason this change is being made now is because
UFFDIO_CONTINUEson subpages definitely shouldn't set this page flag on the head page.Signed-off-by James Houghton <[email protected]>--- mm/hugetlb.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-)mm/hugetlb.c +0
-0mm/hugetlb.c int hugetlb_mcopy_atomic_pte(struct mm_struct *dst_mm, · line 6103
Quoted 8 lines
mm/hugetlb.c +4
-1@@ -6097,7 +6097,10 @@ int hugetlb_mcopy_atomic_pte(struct mm_struct *dst_mm,* preceding stores to the page contents become visible before * the set_pte_at() write. */ -__SetPageUptodate (page); +if (!is_continue) +__SetPageUptodate (page); +else +VM_WARN_ON_ONCE_PAGE (!PageUptodate (page), page);Yeah the old code looks wrong, I'm just wondering whether we can 100% guarantee this for hugetlb. E.g. for shmem that won't hold when we uffd-continue on a not used page (e.g. by an over-sized
fallocate()).
Quoted 7 lines
Quoted 3 lines
Another safer approach is simply fail the ioctl if !uptodate, but if you're certain then
WARN_ON_ONCEsounds all good too. At least I did have a quick look on hugetlbfallocate()and pages will be uptodate immediately.Failing the ioctl sounds better than only WARNing. I'll do that and drop the
WARN_ON_ONCEfor v1. Thanks!
Sorry for the VERY late reply ...
After checking all the code paths, I do not think it is possible for a !PageUptodate to be in the cache (target of continue).
ACK to failing the ioctl if not set, although I don't think it is possible in current code.
Signature · 1 line
Mike Kravetz
RFC PATCH v2 2/47 hugetlb: remove mk_huge_pte; it is unused
In reply to James Houghtonmk_huge_pte is unused and not necessary. pte_mkhuge is the appropriate function to call to create a HugeTLB PTE (see Documentation/mm/arch_pgtable_helpers.rst).
It is being removed now to avoid complicating the implementation of HugeTLB high-granularity mapping.
--- arch/s390/include/asm/hugetlb.h | 5 ----- include/asm-generic/hugetlb.h | 5 ----- mm/debug_vm_pgtable.c | 2 +- mm/hugetlb.c | 7 +++---- 4 files changed, 4 insertions(+), 15 deletions(-)
arch/s390/include/asm/hugetlb.h +0 -5
@@ -77,11 +77,6 @@ static inline void huge_ptep_set_wrprotect(struct mm_struct *mm,set_huge_pte_at (mm, addr, ptep,pte_wrprotect (pte)); } -static inline pte_t mk_huge_pte (struct page *page,pgprot_t pgprot) -{ -return mk_pte (page, pgprot); -} -static inline int huge_pte_none (pte_t pte) {return pte_none (pte);
include/asm-generic/hugetlb.h +0 -5
@@ -5,11 +5,6 @@#include <linux/swap.h> #include <linux/swapops.h> -static inline pte_t mk_huge_pte (struct page *page,pgprot_t pgprot) -{ -return mk_pte (page, pgprot); -} -static inline unsigned long huge_pte_write (pte_t pte) {return pte_write (pte);
mm/debug_vm_pgtable.c +1 -1
@@ -929,7 +929,7 @@ static void __init hugetlb_basic_tests(struct pgtable_debug_args *args)* as it was previously derived from a real kernel symbol. */ page =pfn_to_page (args->fixed_pmd_pfn); - pte =mk_huge_pte (page, args->page_prot); + pte =mk_pte (page, args->page_prot);WARN_ON (!huge_pte_dirty (huge_pte_mkdirty (pte)));WARN_ON (!huge_pte_write (huge_pte_mkwrite (huge_pte_wrprotect (pte))));
mm/hugetlb.c +3 -4
@@ -4728,11 +4728,10 @@ static pte_t make_huge_pte(struct vm_area_struct *vma, struct page *page,unsigned int shift =huge_page_shift (hstate_vma (vma));if (writable) { - entry =huge_pte_mkwrite (huge_pte_mkdirty (mk_huge_pte (page, - vma->vm_page_prot))); + entry =huge_pte_mkwrite (huge_pte_mkdirty (mk_pte (page, + vma->vm_page_prot))); }else { - entry =huge_pte_wrprotect (mk_huge_pte (page, - vma->vm_page_prot)); + entry =huge_pte_wrprotect (mk_pte (page, vma->vm_page_prot)); } entry =pte_mkyoung (entry); entry =arch_make_huge_pte (entry, shift, vma->vm_flags);
Signature · 1 line
2.38.0.135.g90850a2211-goog
Re: [RFC PATCH v2 02/47] hugetlb: remove mk_huge_pte; it is unused
Quoted 8 lines · James Houghton
mk_huge_pteis unused and not necessary.pte_mkhugeis the appropriate function to call to create a HugeTLB PTE (seeDocumentation/mm/arch_pgtable_helpers.rst).It is being removed now to avoid complicating the implementation of HugeTLB high-granularity mapping.
Signed-off-by James Houghton <[email protected]>
Signature · 1 line
Peter Xu
Re: [RFC PATCH v2 02/47] hugetlb: remove mk_huge_pte; it is unused
In reply to James HoughtonQuoted 9 lines · James Houghton
mk_huge_pteis unused and not necessary.pte_mkhugeis the appropriate function to call to create a HugeTLB PTE (seeDocumentation/mm/arch_pgtable_helpers.rst).It is being removed now to avoid complicating the implementation of HugeTLB high-granularity mapping.
Signed-off-by James Houghton <[email protected]>
Quoted 74 lines
--- arch/s390/include/asm/hugetlb.h | 5 ----- include/asm-generic/hugetlb.h | 5 ----- mm/debug_vm_pgtable.c | 2 +- mm/hugetlb.c | 7 +++---- 4 files changed, 4 insertions(+), 15 deletions(-)arch/s390/include/asm/hugetlb.h +0
-5@@ -77,11 +77,6 @@ static inline void huge_ptep_set_wrprotect(struct mm_struct *mm,set_huge_pte_at (mm, addr, ptep,pte_wrprotect (pte)); } -static inline pte_t mk_huge_pte (struct page *page,pgprot_t pgprot) -{ -return mk_pte (page, pgprot); -} -static inline int huge_pte_none (pte_t pte) {return pte_none (pte);include/asm-generic/hugetlb.h +0
-5@@ -5,11 +5,6 @@#include <linux/swap.h> #include <linux/swapops.h> -static inline pte_t mk_huge_pte (struct page *page,pgprot_t pgprot) -{ -return mk_pte (page, pgprot); -} -static inline unsigned long huge_pte_write (pte_t pte) {return pte_write (pte);mm/debug_vm_pgtable.c +1
-1@@ -929,7 +929,7 @@ static void __init hugetlb_basic_tests(struct pgtable_debug_args *args)* as it was previously derived from a real kernel symbol. */ page =pfn_to_page (args->fixed_pmd_pfn); - pte =mk_huge_pte (page, args->page_prot); + pte =mk_pte (page, args->page_prot);WARN_ON (!huge_pte_dirty (huge_pte_mkdirty (pte)));WARN_ON (!huge_pte_write (huge_pte_mkwrite (huge_pte_wrprotect (pte))));mm/hugetlb.c +3
-5@@ -4728,11 +4728,10 @@ static pte_t make_huge_pte(struct vm_area_struct *vma, struct page *page,unsigned int shift =huge_page_shift (hstate_vma (vma));if (writable) { - entry =huge_pte_mkwrite (huge_pte_mkdirty (mk_huge_pte (page, - vma->vm_page_prot))); + entry =huge_pte_mkwrite (huge_pte_mkdirty (mk_pte (page, + vma->vm_page_prot))); }else { - entry =huge_pte_wrprotect (mk_huge_pte (page, - vma->vm_page_prot)); + entry =huge_pte_wrprotect (mk_pte (page, vma->vm_page_prot)); } entry =pte_mkyoung (entry); entry =arch_make_huge_pte (entry, shift, vma->vm_flags); --2.38.0.135.g90850a2211-goog
Re: [RFC PATCH v2 02/47] hugetlb: remove mk_huge_pte; it is unused
In reply to James HoughtonQuoted 7 lines · James Houghton
mk_huge_pteis unused and not necessary.pte_mkhugeis the appropriate function to call to create a HugeTLB PTE (seeDocumentation/mm/arch_pgtable_helpers.rst).It is being removed now to avoid complicating the implementation of HugeTLB high-granularity mapping.
Quoted 7 lines
Signed-off-by James Houghton <[email protected]>--- arch/s390/include/asm/hugetlb.h | 5 ----- include/asm-generic/hugetlb.h | 5 ----- mm/debug_vm_pgtable.c | 2 +- mm/hugetlb.c | 7 +++---- 4 files changed, 4 insertions(+), 15 deletions(-)
Thanks!
I suspect there is more cleanup of 'hugetlb page table helpers' that could be done.
Signature · 1 line
Mike Kravetz
RFC PATCH v2 3/47 hugetlb: remove redundant pte_mkhuge in migration path
In reply to James Houghtonarch_make_huge_pte, which is called immediately following pte_mkhuge, already makes the necessary changes to the PTE that pte_mkhuge would have. The generic implementation of arch_make_huge_pte simply calls pte_mkhuge.
--- mm/migrate.c | 1 - 1 file changed, 1 deletion(-)
mm/migrate.c +0 -1
@@ -237,7 +237,6 @@ static bool remove_migration_pte(struct folio *folio,if (folio_test_hugetlb (folio)) {unsigned int shift =huge_page_shift (hstate_vma (vma)); - pte =pte_mkhuge (pte); pte =arch_make_huge_pte (pte, shift, vma->vm_flags);if (folio_test_anon (folio)) hugepage_add_anon_rmap(new, vma, pvmw.address,
Signature · 1 line
2.38.0.135.g90850a2211-goog
Re: [RFC PATCH v2 03/47] hugetlb: remove redundant pte_mkhuge in migration path
Quoted 6 lines · James Houghton
arch_make_huge_pte, which is called immediately followingpte_mkhuge, already makes the necessary changes to the PTE thatpte_mkhugewould have. The generic implementation ofarch_make_huge_ptesimply callspte_mkhuge.Signed-off-by James Houghton <[email protected]>
Signature · 1 line
Peter Xu
Re: [RFC PATCH v2 03/47] hugetlb: remove redundant pte_mkhuge in migration path
In reply to James HoughtonQuoted 7 lines · James Houghton
arch_make_huge_pte, which is called immediately followingpte_mkhuge, already makes the necessary changes to the PTE thatpte_mkhugewould have. The generic implementation ofarch_make_huge_ptesimply callspte_mkhuge.Signed-off-by James Houghton <[email protected]>
Quoted 19 lines
--- mm/migrate.c | 1 - 1 file changed, 1 deletion(-)mm/migrate.c +0
-2@@ -237,7 +237,6 @@ static bool remove_migration_pte(struct folio *folio,if (folio_test_hugetlb (folio)) {unsigned int shift =huge_page_shift (hstate_vma (vma)); - pte =pte_mkhuge (pte); pte =arch_make_huge_pte (pte, shift, vma->vm_flags);if (folio_test_anon (folio)) hugepage_add_anon_rmap(new, vma, pvmw.address, --2.38.0.135.g90850a2211-goog
Re: [RFC PATCH v2 03/47] hugetlb: remove redundant pte_mkhuge in migration path
In reply to James HoughtonQuoted 9 lines · James Houghton
arch_make_huge_pte, which is called immediately followingpte_mkhuge, already makes the necessary changes to the PTE thatpte_mkhugewould have. The generic implementation ofarch_make_huge_ptesimply callspte_mkhuge.Signed-off-by James Houghton <[email protected]>--- mm/migrate.c | 1 - 1 file changed, 1 deletion(-)
Thanks,
Signature · 1 line
Mike Kravetz
RFC PATCH v2 4/47 hugetlb: only adjust address ranges when VMAs want PMD sharing
In reply to James HoughtonCurrently this check is overly aggressive. For some userfaultfd VMAs, VMA sharing is disabled, yet we still widen the address range, which is used for flushing TLBs and sending MMU notifiers.
This is done now, as HGM VMAs also have sharing disabled, yet would still have flush ranges adjusted. Overaggressively flushing TLBs and triggering MMU notifiers is particularly harmful with lots of high-granularity operations.
--- mm/hugetlb.c | 21 +++++++++++++++------ 1 file changed, 15 insertions(+), 6 deletions(-)
mm/hugetlb.c +15 -6
@@ -6835,22 +6835,31 @@ static unsigned long page_table_shareable(struct vm_area_struct *svma,return saddr; } -bool want_pmd_share (struct vm_area_struct *vma,unsigned long addr) +static bool pmd_sharing_possible (struct vm_area_struct *vma) { -unsigned long start = addr &PUD_MASK ; -unsigned long end = start +PUD_SIZE ; -#ifdef CONFIG_USERFAULTFD if (uffd_disable_huge_pmd_share (vma))return false;#endif /* -* check on proper vm_flags and page table alignment +* Only shared VMAs can share PMDs. */ if (!(vma->vm_flags &VM_MAYSHARE ))return false;if (!vma->vm_private_data)/* vma lock required for sharing */ return false; +return true; +} + +bool want_pmd_share (struct vm_area_struct *vma,unsigned long addr) +{ +unsigned long start = addr &PUD_MASK ; +unsigned long end = start +PUD_SIZE ; +/* +* check on proper vm_flags and page table alignment +*/ +if (!pmd_sharing_possible (vma)) +return false;if (!range_in_vma (vma, start, end))return false;return true;
@@ -6871,7 +6880,7 @@ void adjust_range_if_pmd_sharing_possible(struct vm_area_struct *vma,* vma needs to span at least one aligned PUD size, and the range * must be at least partially within in. */ -if (!(vma->vm_flags &VM_MAYSHARE ) || !(v_end > v_start) || +if (!pmd_sharing_possible (vma) || !(v_end > v_start) || (*end <= v_start) || (*start >= v_end))return ;
Signature · 1 line
2.38.0.135.g90850a2211-goog
Re: [RFC PATCH v2 04/47] hugetlb: only adjust address ranges when VMAs want PMD sharing
Quoted 10 lines · James Houghton
Currently this check is overly aggressive. For some userfaultfd VMAs, VMA sharing is disabled, yet we still widen the address range, which is used for flushing TLBs and sending MMU notifiers.
This is done now, as HGM VMAs also have sharing disabled, yet would still have flush ranges adjusted. Overaggressively flushing TLBs and triggering MMU notifiers is particularly harmful with lots of high-granularity operations.
Signed-off-by James Houghton <[email protected]>
Signature · 1 line
Peter Xu
Re: [RFC PATCH v2 04/47] hugetlb: only adjust address ranges when VMAs want PMD sharing
In reply to James HoughtonQuoted 3 lines · James Houghton
Currently this check is overly aggressive. For some userfaultfd VMAs, VMA sharing is disabled, yet we still widen the address range, which is used for flushing TLBs and sending MMU notifiers.
Yes, the userfaultfd check is missing in the code today.
Quoted 9 lines
This is done now, as HGM VMAs also have sharing disabled, yet would still have flush ranges adjusted. Overaggressively flushing TLBs and triggering MMU notifiers is particularly harmful with lots of high-granularity operations.
Signed-off-by James Houghton <[email protected]>--- mm/hugetlb.c | 21 +++++++++++++++------ 1 file changed, 15 insertions(+), 6 deletions(-)
Thanks,
Signature · 1 line
Mike Kravetz
RFC PATCH v2 5/47 hugetlb: make hugetlb_vma_lock_alloc return its failure reason
In reply to James HoughtonCurrently hugetlb_vma_lock_alloc doesn't return anything, as there is no
need: if it fails, PMD sharing won't be enabled. However, HGM requires that the VMA lock exists, so we need to verify that hugetlb_vma_lock_alloc actually succeeded. If hugetlb_vma_lock_alloc fails, then we can pass that up to the caller that is attempting to enable HGM.
--- mm/hugetlb.c | 16 +++++++++------- 1 file changed, 9 insertions(+), 7 deletions(-)
mm/hugetlb.c +9 -7
@@ -92,7 +92,7 @@ struct mutex *hugetlb_fault_mutex_table ____cacheline_aligned_in_smp;/* Forward declaration */ static int hugetlb_acct_memory (struct hstate *h,long delta);static void hugetlb_vma_lock_free (struct vm_area_struct *vma); -static void hugetlb_vma_lock_alloc (struct vm_area_struct *vma); +static int hugetlb_vma_lock_alloc (struct vm_area_struct *vma);static void __hugetlb_vma_unlock_write_free (struct vm_area_struct *vma);static inline bool subpool_is_free (struct hugepage_subpool *spool)
@@ -7001,17 +7001,17 @@ static void hugetlb_vma_lock_free(struct vm_area_struct *vma) } } -static void hugetlb_vma_lock_alloc (struct vm_area_struct *vma) +static int hugetlb_vma_lock_alloc (struct vm_area_struct *vma) {struct hugetlb_vma_lock *vma_lock;/* Only establish in (flags) sharable vmas */ if (!vma || !(vma->vm_flags &VM_MAYSHARE )) -return ; +return -EINVAL ; -/* Should never get here with non-NULL vm_private_data */ +/* We've already allocated the lock. */ if (vma->vm_private_data) -return ; +return 0 ; vma_lock =kmalloc (sizeof (*vma_lock),GFP_KERNEL );if (!vma_lock) {
@@ -7026,13 +7026,14 @@ static void hugetlb_vma_lock_alloc(struct vm_area_struct *vma)* allocation failure. */ pr_warn_once ("HugeTLB: unable to allocate vma specific lock\n" ); -return ; +return -ENOMEM ; }kref_init (&vma_lock->refs);init_rwsem (&vma_lock->rw_sema); vma_lock->vma = vma; vma->vm_private_data = vma_lock; +return 0 ; }/*
@@ -7160,8 +7161,9 @@ static void hugetlb_vma_lock_free(struct vm_area_struct *vma) { } -static void hugetlb_vma_lock_alloc (struct vm_area_struct *vma) +static int hugetlb_vma_lock_alloc (struct vm_area_struct *vma) { +return 0 ; } pte_t *huge_pmd_share(struct mm_struct *mm, struct vm_area_struct *vma,
Signature · 1 line
2.38.0.135.g90850a2211-goog
Peter Xu
reviewerRe: [RFC PATCH v2 05/47] hugetlb: make hugetlb_vma_lock_alloc return its failure reason
Quoted 34 lines · James Houghton
Currently
hugetlb_vma_lock_allocdoesn't return anything, as there is no
need: if it fails, PMD sharing won't be enabled. However, HGM requires that the VMA lock exists, so we need to verify thathugetlb_vma_lock_allocactually succeeded. Ifhugetlb_vma_lock_allocfails, then we can pass that up to the caller that is attempting to enable HGM.Signed-off-by James Houghton <[email protected]>--- mm/hugetlb.c | 16 +++++++++------- 1 file changed, 9 insertions(+), 7 deletions(-)mm/hugetlb.c +2
-2@@ -92,7 +92,7 @@ struct mutex *hugetlb_fault_mutex_table ____cacheline_aligned_in_smp;/* Forward declaration */ static int hugetlb_acct_memory (struct hstate *h,long delta);static void hugetlb_vma_lock_free (struct vm_area_struct *vma); -static void hugetlb_vma_lock_alloc (struct vm_area_struct *vma); +static int hugetlb_vma_lock_alloc (struct vm_area_struct *vma);static void __hugetlb_vma_unlock_write_free (struct vm_area_struct *vma);static inline bool subpool_is_free (struct hugepage_subpool *spool)@@ -7001,17 +7001,17 @@ static void hugetlb_vma_lock_free(struct vm_area_struct *vma) } } -static void hugetlb_vma_lock_alloc (struct vm_area_struct *vma) +static int hugetlb_vma_lock_alloc (struct vm_area_struct *vma) {struct hugetlb_vma_lock *vma_lock;
mm/hugetlb.c static void hugetlb_vma_lock_free(struct vm_area_struct *vma) · line 7014
Quoted 10 lines
mm/hugetlb.c +3
-3/* Only establish in (flags) sharable vmas */ if (!vma || !(vma->vm_flags &VM_MAYSHARE )) -return ; +return -EINVAL ; -/* Should never get here with non-NULL vm_private_data */ +/* We've already allocated the lock. */ if (vma->vm_private_data) -return ; +return 0 ;
No objection on the patch itself, but I am just wondering what guarantees thread-safety for this function to not leak vm_private_data when two threads try to allocate at the same time.
I think it should be the write mmap lock. I saw that in your latest code base there's:
/* * We must hold the mmap lock for writing so that callers can rely on * hugetlb_hgm_enabled returning a consistent result while holding * the mmap lock for reading. */ mmap_assert_write_locked(vma->vm_mm); /* HugeTLB HGM requires the VMA lock to synchronize collapsing. */ ret = hugetlb_vma_data_alloc(vma); if (ret) return ret;
So that's covered there. The rest places are hugetlb_vm_op_open() and hugetlb_reserve_pages() and they all seem fine too: hugetlb_vm_op_open() is during mmap(), the latter has vma==NULL so allocation will be skipped.
I'm wondering whether it would make sense to move this assert to be inside of hugetlb_vma_data_alloc() after the !vma check, or just add the same assert too but for different reason.
Quoted 34 lines
mm/hugetlb.c +4
-2vma_lock =kmalloc (sizeof (*vma_lock),GFP_KERNEL );if (!vma_lock) {@@ -7026,13 +7026,14 @@ static void hugetlb_vma_lock_alloc(struct vm_area_struct *vma)* allocation failure. */ pr_warn_once ("HugeTLB: unable to allocate vma specific lock\n" ); -return ; +return -ENOMEM ; }kref_init (&vma_lock->refs);init_rwsem (&vma_lock->rw_sema); vma_lock->vma = vma; vma->vm_private_data = vma_lock; +return 0 ; }/* @@ -7160,8 +7161,9 @@ static void hugetlb_vma_lock_free(struct vm_area_struct *vma) { } -static void hugetlb_vma_lock_alloc (struct vm_area_struct *vma) +static int hugetlb_vma_lock_alloc (struct vm_area_struct *vma) { +return 0 ; } pte_t *huge_pmd_share(struct mm_struct *mm, struct vm_area_struct *vma,Signature · 1 line
2.38.0.135.g90850a2211-goog
Signature · 1 line
Peter Xu
Re: [RFC PATCH v2 05/47] hugetlb: make hugetlb_vma_lock_alloc return its failure reason
Quoted 20 lines · Peter Xu
No objection on the patch itself, but I am just wondering what guarantees thread-safety for this function to not leak
vm_private_datawhen two threads try to allocate at the same time.I think it should be the write mmap lock. I saw that in your latest code base there's:
/* * We must hold the mmap lock for writing so that callers can rely on * hugetlb_hgm_enabled returning a consistent result while holding * the mmap lock for reading. */ mmap_assert_write_locked(vma->vm_mm); /* HugeTLB HGM requires the VMA lock to synchronize collapsing. */ ret = hugetlb_vma_data_alloc(vma); if (ret) return ret;
Quoted 7 lines
So that's covered there. The rest places are
hugetlb_vm_op_open()andhugetlb_reserve_pages()and they all seem fine too:hugetlb_vm_op_open()is duringmmap(), the latter has vma==NULL so allocation will be skipped.I'm wondering whether it would make sense to move this assert to be inside of
hugetlb_vma_data_alloc()after the !vma check, or just add the same assert too but for different reason.
I think leaving the assert here and adding a new assert inside hugetlb_vma_data_alloc() makes sense. Thanks Peter.
- James
Quoted 39 lines
Quoted 32 lines
vma_lock = kmalloc(sizeof(*vma_lock), GFP_KERNEL); if (!vma_lock) {mm/hugetlb.c +4
-3@@ -7026,13 +7026,14 @@ static void hugetlb_vma_lock_alloc(struct vm_area_struct *vma)* allocation failure. */ pr_warn_once ("HugeTLB: unable to allocate vma specific lock\n" ); -return ; +return -ENOMEM ; }kref_init (&vma_lock->refs);init_rwsem (&vma_lock->rw_sema); vma_lock->vma = vma; vma->vm_private_data = vma_lock; +return 0 ; }/* @@ -7160,8 +7161,9 @@ static void hugetlb_vma_lock_free(struct vm_area_struct *vma) { } -static void hugetlb_vma_lock_alloc (struct vm_area_struct *vma) +static int hugetlb_vma_lock_alloc (struct vm_area_struct *vma) { +return 0 ; } pte_t *huge_pmd_share(struct mm_struct *mm, struct vm_area_struct *vma, --Quoted 3 lines
2.38.0.135.g90850a2211-goog
--
Peter Xu
Re: [RFC PATCH v2 05/47] hugetlb: make hugetlb_vma_lock_alloc return its failure reason
In reply to James HoughtonQuoted 35 lines · James Houghton
Currently
hugetlb_vma_lock_allocdoesn't return anything, as there is no
need: if it fails, PMD sharing won't be enabled. However, HGM requires that the VMA lock exists, so we need to verify thathugetlb_vma_lock_allocactually succeeded. Ifhugetlb_vma_lock_allocfails, then we can pass that up to the caller that is attempting to enable HGM.Signed-off-by James Houghton <[email protected]>--- mm/hugetlb.c | 16 +++++++++------- 1 file changed, 9 insertions(+), 7 deletions(-)mm/hugetlb.c +2
-2@@ -92,7 +92,7 @@ struct mutex *hugetlb_fault_mutex_table ____cacheline_aligned_in_smp;/* Forward declaration */ static int hugetlb_acct_memory (struct hstate *h,long delta);static void hugetlb_vma_lock_free (struct vm_area_struct *vma); -static void hugetlb_vma_lock_alloc (struct vm_area_struct *vma); +static int hugetlb_vma_lock_alloc (struct vm_area_struct *vma);static void __hugetlb_vma_unlock_write_free (struct vm_area_struct *vma);static inline bool subpool_is_free (struct hugepage_subpool *spool)@@ -7001,17 +7001,17 @@ static void hugetlb_vma_lock_free(struct vm_area_struct *vma) } } -static void hugetlb_vma_lock_alloc (struct vm_area_struct *vma) +static int hugetlb_vma_lock_alloc (struct vm_area_struct *vma) {struct hugetlb_vma_lock *vma_lock;
mm/hugetlb.c static void hugetlb_vma_lock_free(struct vm_area_struct *vma) · line 7014
Quoted 10 lines
mm/hugetlb.c +3
-3/* Only establish in (flags) sharable vmas */ if (!vma || !(vma->vm_flags &VM_MAYSHARE )) -return ; +return -EINVAL ; -/* Should never get here with non-NULL vm_private_data */ +/* We've already allocated the lock. */ if (vma->vm_private_data) -return ; +return 0 ;
I would have expected -EEXIST here.
Also even if the patch looks generally fine it's hard to provide Acked-by now. I need to look at the call site which is in another patch in the series. If there is an opportunity to squash changes to helpers and their call sites please do.
Quoted 33 lines
mm/hugetlb.c +4
-3vma_lock =kmalloc (sizeof (*vma_lock),GFP_KERNEL );if (!vma_lock) {@@ -7026,13 +7026,14 @@ static void hugetlb_vma_lock_alloc(struct vm_area_struct *vma)* allocation failure. */ pr_warn_once ("HugeTLB: unable to allocate vma specific lock\n" ); -return ; +return -ENOMEM ; }kref_init (&vma_lock->refs);init_rwsem (&vma_lock->rw_sema); vma_lock->vma = vma; vma->vm_private_data = vma_lock; +return 0 ; }/* @@ -7160,8 +7161,9 @@ static void hugetlb_vma_lock_free(struct vm_area_struct *vma) { } -static void hugetlb_vma_lock_alloc (struct vm_area_struct *vma) +static int hugetlb_vma_lock_alloc (struct vm_area_struct *vma) { +return 0 ; } pte_t *huge_pmd_share(struct mm_struct *mm, struct vm_area_struct *vma, --2.38.0.135.g90850a2211-goog
Re: [RFC PATCH v2 05/47] hugetlb: make hugetlb_vma_lock_alloc return its failure reason
In reply to James HoughtonQuoted 6 lines · James Houghton
Currently
hugetlb_vma_lock_allocdoesn't return anything, as there is no
need: if it fails, PMD sharing won't be enabled. However, HGM requires that the VMA lock exists, so we need to verify thathugetlb_vma_lock_allocactually succeeded. Ifhugetlb_vma_lock_allocfails, then we can pass that up to the caller that is attempting to enable HGM.
No serious objections to this change ...
However, there are currently only two places today where hugetlb_vma_lock_alloc is called: hugetlb_reserve_pages and hugetlb_vm_op_open. hugetlb_reserve_pages is not an issue. Since hugetlb_vm_op_open (as a defined vm_operation) returns void, I am not sure how you plan to pass up an allocation failure. Suspect this will become evident in subsequent patches.
Signature · 1 line
Mike Kravetz
RFC PATCH v2 6/47 hugetlb: extend vma lock for shared vmas
In reply to James HoughtonThis allows us to add more data into the shared structure, which we will use to store whether or not HGM is enabled for this VMA or not, as HGM is only available for shared mappings.
It may be better to include HGM as a VMA flag instead of extending the VMA lock structure.
--- include/linux/hugetlb.h | 4 +++ mm/hugetlb.c | 65 +++++++++++++++++++++-------------------- 2 files changed, 37 insertions(+), 32 deletions(-)
include/linux/hugetlb.h +4 -0
@@ -121,6 +121,10 @@ struct hugetlb_vma_lock {struct vm_area_struct *vma; }; +struct hugetlb_shared_vma_data { +struct hugetlb_vma_lock vma_lock; +}; +extern struct resv_map *resv_map_alloc (void );void resv_map_release (struct kref *ref);
mm/hugetlb.c +33 -32
184 lines not on this page · raw message
Signature · 1 line
2.38.0.135.g90850a2211-goog
Peter Xu
reviewerRe: [RFC PATCH v2 06/47] hugetlb: extend vma lock for shared vmas
Quoted 18 lines · James Houghton
This allows us to add more data into the shared structure, which we will use to store whether or not HGM is enabled for this VMA or not, as HGM is only available for shared mappings.
It may be better to include HGM as a VMA flag instead of extending the VMA lock structure.
Signed-off-by James Houghton <[email protected]>--- include/linux/hugetlb.h | 4 +++ mm/hugetlb.c | 65 +++++++++++++++++++++-------------------- 2 files changed, 37 insertions(+), 32 deletions(-)include/linux/hugetlb.h +0
-0
include/linux/hugetlb.h struct hugetlb_vma_lock { · line 126
Quoted 6 lines
include/linux/hugetlb.h +3
-0@@ -121,6 +121,10 @@ struct hugetlb_vma_lock {struct vm_area_struct *vma; }; +struct hugetlb_shared_vma_data { +struct hugetlb_vma_lock vma_lock; +};
How about add a comment above hugetlb_vma_lock showing how it should be used correctly? We lacked documents on the lock for pmd sharing protections, now if to reuse the same lock for HGM pgtables I think some doc will definitely help.
To summarize, I think so far it means:
- Read lock needed when one wants to stablize VM_SHARED pgtables (covers both pmd shared pgtables or hgm low-level pgtables)
- Write lock needed when one wants to release VM_SHARED pgtable pages (covers both pmd unshare or releasing hgm low-level pgtables)
Or something like that.
Signature · 1 line
Peter Xu
Re: [RFC PATCH v2 06/47] hugetlb: extend vma lock for shared vmas
Quoted 34 lines · Peter Xu
Quoted 18 lines · James Houghton
This allows us to add more data into the shared structure, which we will use to store whether or not HGM is enabled for this VMA or not, as HGM is only available for shared mappings.
It may be better to include HGM as a VMA flag instead of extending the VMA lock structure.
Signed-off-by James Houghton <[email protected]>--- include/linux/hugetlb.h | 4 +++ mm/hugetlb.c | 65 +++++++++++++++++++++-------------------- 2 files changed, 37 insertions(+), 32 deletions(-)include/linux/hugetlb.h +0
-0include/linux/hugetlb.h struct hugetlb_vma_lock { · line 126
Quoted 6 lines
include/linux/hugetlb.h +3
-0@@ -121,6 +121,10 @@ struct hugetlb_vma_lock {struct vm_area_struct *vma; }; +struct hugetlb_shared_vma_data { +struct hugetlb_vma_lock vma_lock; +};How about add a comment above
hugetlb_vma_lockshowing how it should be used correctly? We lacked documents on the lock for pmd sharing protections, now if to reuse the same lock for HGM pgtables I think some doc will definitely help.To summarize, I think so far it means:
Quoted 7 lines
- Read lock needed when one wants to stablize
VM_SHAREDpgtables (covers both pmd shared pgtables or hgm low-level pgtables)- Write lock needed when one wants to release
VM_SHAREDpgtable pages (covers both pmd unshare or releasing hgm low-level pgtables)Or something like that.
Will do. I'll make this change together with the rmap comment update ("rmap: update hugetlb lock comment for HGM").
- James
Quoted 4 lines
--
Peter Xu
Re: [RFC PATCH v2 06/47] hugetlb: extend vma lock for shared vmas
In reply to Peter XuQuoted 31 lines · Peter Xu
Quoted 18 lines · James Houghton
This allows us to add more data into the shared structure, which we will use to store whether or not HGM is enabled for this VMA or not, as HGM is only available for shared mappings.
It may be better to include HGM as a VMA flag instead of extending the VMA lock structure.
Signed-off-by James Houghton <[email protected]>--- include/linux/hugetlb.h | 4 +++ mm/hugetlb.c | 65 +++++++++++++++++++++-------------------- 2 files changed, 37 insertions(+), 32 deletions(-)include/linux/hugetlb.h +0
-0include/linux/hugetlb.h struct hugetlb_vma_lock { · line 126
Quoted 6 lines
include/linux/hugetlb.h +3
-0@@ -121,6 +121,10 @@ struct hugetlb_vma_lock {struct vm_area_struct *vma; }; +struct hugetlb_shared_vma_data { +struct hugetlb_vma_lock vma_lock; +};How about add a comment above
hugetlb_vma_lockshowing how it should be used correctly? We lacked documents on the lock for pmd sharing protections, now if to reuse the same lock for HGM pgtables I think some doc will definitely help.
Quoted 7 lines
To summarize, I think so far it means:
- Read lock needed when one wants to stablize
VM_SHAREDpgtables (covers both pmd shared pgtables or hgm low-level pgtables)- Write lock needed when one wants to release
VM_SHAREDpgtable pages (covers both pmd unshare or releasing hgm low-level pgtables)
Peter must have read ahead and knows that you plan to use the vma_lock for HGM.
The commit message implies that a you only need some type of indication (a flag for instance) that HGM is enabled for the vma.
No objections to expanding the structure as is done here.
If this is the direction we take, and someday this is extended to private mappings we could used the same scheme to expand the reserve map structure.
Signature · 1 line
Mike Kravetz
RFC PATCH v2 7/47 hugetlb: add CONFIG_HUGETLB_HIGH_GRANULARITY_MAPPING
In reply to James HoughtonThis adds the Kconfig to enable or disable high-granularity mapping. Each architecture must explicitly opt-in to it (via ARCH_WANT_HUGETLB_HIGH_GRANULARITY_MAPPING), but when opted in, HGM will be enabled by default if HUGETLB_PAGE is enabled.
--- fs/Kconfig | 7 +++++++ 1 file changed, 7 insertions(+)
fs/Kconfig +7 -0
@@ -267,6 +267,13 @@ config HUGETLB_PAGE_OPTIMIZE_VMEMMAP_DEFAULT_ON enable HVO by default. It can be disabled via hugetlb_free_vmemmap=off (boot command line) or hugetlb_optimize_vmemmap (sysctl). +config ARCH_WANT_HUGETLB_HIGH_GRANULARITY_MAPPING + bool + +config HUGETLB_HIGH_GRANULARITY_MAPPING + def_bool HUGETLB_PAGE + depends on ARCH_WANT_HUGETLB_HIGH_GRANULARITY_MAPPING + config MEMFD_CREATE def_bool TMPFS || HUGETLBFS
Signature · 1 line
2.38.0.135.g90850a2211-goog
Re: [RFC PATCH v2 07/47] hugetlb: add CONFIG_HUGETLB_HIGH_GRANULARITY_MAPPING
Quoted 15 lines · James Houghton
This adds the
Kconfigto enable or disable high-granularity mapping. Each architecture must explicitly opt-in to it (viaARCH_WANT_HUGETLB_HIGH_GRANULARITY_MAPPING), but when opted in, HGM will be enabled by default ifHUGETLB_PAGEis enabled.Signed-off-by James Houghton <[email protected]>--- fs/Kconfig | 7 +++++++ 1 file changed, 7 insertions(+)fs/Kconfig +0
-0
fs/Kconfig config HUGETLB_PAGE_OPTIMIZE_VMEMMAP_DEFAULT_ON · line 275
Quoted 9 lines
fs/Kconfig +6
-0@@ -267,6 +267,13 @@ config HUGETLB_PAGE_OPTIMIZE_VMEMMAP_DEFAULT_ON enable HVO by default. It can be disabled via hugetlb_free_vmemmap=off (boot command line) or hugetlb_optimize_vmemmap (sysctl). +config ARCH_WANT_HUGETLB_HIGH_GRANULARITY_MAPPING + bool + +config HUGETLB_HIGH_GRANULARITY_MAPPING + def_bool HUGETLB_PAGE + depends on ARCH_WANT_HUGETLB_HIGH_GRANULARITY_MAPPING
Might also need to make this depend on CONFIG_ARCH_WANT_HUGE_PMD_SHARE as the vma local allocation will only happen in this case.
Signature · 1 line
Mike Kravetz
Quoted 7 lines
fs/Kconfig +1
-0+ config MEMFD_CREATE def_bool TMPFS || HUGETLBFSSignature · 1 line
2.38.0.135.g90850a2211-goog
RFC PATCH v2 8/47 hugetlb: add HGM enablement functions
In reply to James HoughtonCurrently it is possible for all shared VMAs to use HGM, but it must be enabled first. This is because with HGM, we lose PMD sharing, and page table walks require additional synchronization (we need to take the VMA lock).
--- include/linux/hugetlb.h | 22 +++++++++++++ mm/hugetlb.c | 69 +++++++++++++++++++++++++++++++++++++++++ 2 files changed, 91 insertions(+)
include/linux/hugetlb.h +22 -0
@@ -123,6 +123,9 @@ struct hugetlb_vma_lock {struct hugetlb_shared_vma_data {struct hugetlb_vma_lock vma_lock; +#ifdef CONFIG_HUGETLB_HIGH_GRANULARITY_MAPPING +bool hgm_enabled; +#endif };extern struct resv_map *resv_map_alloc (void );
@@ -1179,6 +1182,25 @@ static inline void hugetlb_unregister_node(struct node *node) }#endif /* CONFIG_HUGETLB_PAGE */ +#ifdef CONFIG_HUGETLB_HIGH_GRANULARITY_MAPPING +bool hugetlb_hgm_enabled (struct vm_area_struct *vma); +bool hugetlb_hgm_eligible (struct vm_area_struct *vma); +int enable_hugetlb_hgm (struct vm_area_struct *vma); +#else +static inline bool hugetlb_hgm_enabled (struct vm_area_struct *vma) +{ +return false; +} +static inline bool hugetlb_hgm_eligible (struct vm_area_struct *vma) +{ +return false; +} +static inline int enable_hugetlb_hgm (struct vm_area_struct *vma) +{ +return -EINVAL ; +} +#endif +static inline spinlock_t *huge_pte_lock (struct hstate *h,struct mm_struct *mm,pte_t *pte) {
mm/hugetlb.c +69 -0
87 lines not on this page · raw message
Signature · 1 line
2.38.0.135.g90850a2211-goog
Peter Xu
reviewerRe: [RFC PATCH v2 08/47] hugetlb: add HGM enablement functions
Quoted 89 lines · James Houghton
Currently it is possible for all shared VMAs to use HGM, but it must be enabled first. This is because with HGM, we lose PMD sharing, and page table walks require additional synchronization (we need to take the VMA lock).
Signed-off-by James Houghton <[email protected]>--- include/linux/hugetlb.h | 22 +++++++++++++ mm/hugetlb.c | 69 +++++++++++++++++++++++++++++++++++++++++ 2 files changed, 91 insertions(+)include/linux/hugetlb.h +22
-0@@ -123,6 +123,9 @@ struct hugetlb_vma_lock {struct hugetlb_shared_vma_data {struct hugetlb_vma_lock vma_lock; +#ifdef CONFIG_HUGETLB_HIGH_GRANULARITY_MAPPING +bool hgm_enabled; +#endif };extern struct resv_map *resv_map_alloc (void );@@ -1179,6 +1182,25 @@ static inline void hugetlb_unregister_node(struct node *node) }#endif /* CONFIG_HUGETLB_PAGE */ +#ifdef CONFIG_HUGETLB_HIGH_GRANULARITY_MAPPING +bool hugetlb_hgm_enabled (struct vm_area_struct *vma); +bool hugetlb_hgm_eligible (struct vm_area_struct *vma); +int enable_hugetlb_hgm (struct vm_area_struct *vma); +#else +static inline bool hugetlb_hgm_enabled (struct vm_area_struct *vma) +{ +return false; +} +static inline bool hugetlb_hgm_eligible (struct vm_area_struct *vma) +{ +return false; +} +static inline int enable_hugetlb_hgm (struct vm_area_struct *vma) +{ +return -EINVAL ; +} +#endif +static inline spinlock_t *huge_pte_lock (struct hstate *h,struct mm_struct *mm,pte_t *pte) {mm/hugetlb.c +16
-0@@ -6840,6 +6840,10 @@ static bool pmd_sharing_possible(struct vm_area_struct *vma)#ifdef CONFIG_USERFAULTFD if (uffd_disable_huge_pmd_share (vma))return false; +#endif +#ifdef CONFIG_HUGETLB_HIGH_GRANULARITY_MAPPING +if (hugetlb_hgm_enabled (vma)) +return false;#endif /* * Only shared VMAs can share PMDs. @@ -7033,6 +7037,9 @@ static int hugetlb_vma_data_alloc(struct vm_area_struct *vma)kref_init (&data->vma_lock.refs);init_rwsem (&data->vma_lock.rw_sema); data->vma_lock.vma = vma; +#ifdef CONFIG_HUGETLB_HIGH_GRANULARITY_MAPPING + data->hgm_enabled = false; +#endif vma->vm_private_data = data;return 0 ; }@@ -7290,6 +7297,68 @@ __weak unsigned long hugetlb_mask_last_page(struct hstate *h)#endif /* CONFIG_ARCH_WANT_GENERAL_HUGETLB */ +#ifdef CONFIG_HUGETLB_HIGH_GRANULARITY_MAPPING +bool hugetlb_hgm_eligible (struct vm_area_struct *vma) +{ +/* +* All shared VMAs may have HGM. +* +* HGM requires using the VMA lock, which only exists for shared VMAs. +* To make HGM work for private VMAs, we would need to use another +* scheme to prevent collapsing/splitting from invalidating other
mm/hugetlb.c __weak unsigned long hugetlb_mask_last_page(struct hstate *h) · line 7318
Quoted 10 lines
mm/hugetlb.c +10
-0+* threads' page table walks. +*/ +return vma && (vma->vm_flags &VM_MAYSHARE ); +} +bool hugetlb_hgm_enabled (struct vm_area_struct *vma) +{ +struct hugetlb_shared_vma_data *data = vma->vm_private_data; + +if (!vma || !(vma->vm_flags &VM_MAYSHARE )) +return false;
Nit: smells like a open-coded hugetlb_hgm_eligible().
Quoted 25 lines
mm/hugetlb.c +25
-0+ +return data && data->hgm_enabled; +} + +/* +* Enable high-granularity mapping (HGM) for this VMA. Once enabled, HGM +* cannot be turned off. +* +* PMDs cannot be shared in HGM VMAs. +*/ +int enable_hugetlb_hgm (struct vm_area_struct *vma) +{ +int ret; +struct hugetlb_shared_vma_data *data; + +if (!hugetlb_hgm_eligible (vma)) +return -EINVAL ; + +if (hugetlb_hgm_enabled (vma)) +return 0 ; + +/* +* We must hold the mmap lock for writing so that callers can rely on +* hugetlb_hgm_enabled returning a consistent result while holding +* the mmap lock for reading.
mm/hugetlb.c __weak unsigned long hugetlb_mask_last_page(struct hstate *h) · line 7353
Quoted 10 lines
mm/hugetlb.c +10
-0+*/ +mmap_assert_write_locked (vma->vm_mm); + +/* HugeTLB HGM requires the VMA lock to synchronize collapsing. */ + ret =hugetlb_vma_data_alloc (vma); +if (ret) +return ret; + + data = vma->vm_private_data; +BUG_ON (!data);
Let's avoid BUG_ON() as afaiu it's mostly not welcomed unless extremely necessary. In this case it'll crash immediately in next dereference anyway with the whole stack dumped, so we won't miss anything important. :)
Quoted 14 lines
mm/hugetlb.c +8
-0+ data->hgm_enabled = true; + +/* We don't support PMD sharing with HGM. */ +hugetlb_unshare_all_pmds (vma); + return 0; +} +#endif /* CONFIG_HUGETLB_HIGH_GRANULARITY_MAPPING */ +/* * These functions are overwritable if your architecture needs its own * behavior. Signature · 1 line
2.38.0.135.g90850a2211-goog
Signature · 1 line
Peter Xu
Re: [RFC PATCH v2 08/47] hugetlb: add HGM enablement functions
In reply to James HoughtonQuoted 92 lines · James Houghton
Currently it is possible for all shared VMAs to use HGM, but it must be enabled first. This is because with HGM, we lose PMD sharing, and page table walks require additional synchronization (we need to take the VMA lock).
Signed-off-by James Houghton <[email protected]>--- include/linux/hugetlb.h | 22 +++++++++++++ mm/hugetlb.c | 69 +++++++++++++++++++++++++++++++++++++++++ 2 files changed, 91 insertions(+)include/linux/hugetlb.h +22
-0@@ -123,6 +123,9 @@ struct hugetlb_vma_lock {struct hugetlb_shared_vma_data {struct hugetlb_vma_lock vma_lock; +#ifdef CONFIG_HUGETLB_HIGH_GRANULARITY_MAPPING +bool hgm_enabled; +#endif };extern struct resv_map *resv_map_alloc (void );@@ -1179,6 +1182,25 @@ static inline void hugetlb_unregister_node(struct node *node) }#endif /* CONFIG_HUGETLB_PAGE */ +#ifdef CONFIG_HUGETLB_HIGH_GRANULARITY_MAPPING +bool hugetlb_hgm_enabled (struct vm_area_struct *vma); +bool hugetlb_hgm_eligible (struct vm_area_struct *vma); +int enable_hugetlb_hgm (struct vm_area_struct *vma); +#else +static inline bool hugetlb_hgm_enabled (struct vm_area_struct *vma) +{ +return false; +} +static inline bool hugetlb_hgm_eligible (struct vm_area_struct *vma) +{ +return false; +} +static inline int enable_hugetlb_hgm (struct vm_area_struct *vma) +{ +return -EINVAL ; +} +#endif +static inline spinlock_t *huge_pte_lock (struct hstate *h,struct mm_struct *mm,pte_t *pte) {mm/hugetlb.c +18
-0@@ -6840,6 +6840,10 @@ static bool pmd_sharing_possible(struct vm_area_struct *vma)#ifdef CONFIG_USERFAULTFD if (uffd_disable_huge_pmd_share (vma))return false; +#endif +#ifdef CONFIG_HUGETLB_HIGH_GRANULARITY_MAPPING +if (hugetlb_hgm_enabled (vma)) +return false;#endif /* * Only shared VMAs can share PMDs. @@ -7033,6 +7037,9 @@ static int hugetlb_vma_data_alloc(struct vm_area_struct *vma)kref_init (&data->vma_lock.refs);init_rwsem (&data->vma_lock.rw_sema); data->vma_lock.vma = vma; +#ifdef CONFIG_HUGETLB_HIGH_GRANULARITY_MAPPING + data->hgm_enabled = false; +#endif vma->vm_private_data = data;return 0 ; }@@ -7290,6 +7297,68 @@ __weak unsigned long hugetlb_mask_last_page(struct hstate *h)#endif /* CONFIG_ARCH_WANT_GENERAL_HUGETLB */ +#ifdef CONFIG_HUGETLB_HIGH_GRANULARITY_MAPPING +bool hugetlb_hgm_eligible (struct vm_area_struct *vma) +{ +/* +* All shared VMAs may have HGM. +* +* HGM requires using the VMA lock, which only exists for shared VMAs. +* To make HGM work for private VMAs, we would need to use another +* scheme to prevent collapsing/splitting from invalidating other +* threads' page table walks. +*/
mm/hugetlb.c __weak unsigned long hugetlb_mask_last_page(struct hstate *h) · line 7320
Quoted 10 lines
mm/hugetlb.c +10
-0+return vma && (vma->vm_flags &VM_MAYSHARE ); +} +bool hugetlb_hgm_enabled (struct vm_area_struct *vma) +{ +struct hugetlb_shared_vma_data *data = vma->vm_private_data; + +if (!vma || !(vma->vm_flags &VM_MAYSHARE )) +return false; + +return data && data->hgm_enabled;
Don't you need to lock data->vma_lock before you access data? Or did I misunderstand the locking? Or are you assuming this is safe before hgm_enabled can't be disabled?
Quoted 18 lines
mm/hugetlb.c +18
-0+} + +/* +* Enable high-granularity mapping (HGM) for this VMA. Once enabled, HGM +* cannot be turned off. +* +* PMDs cannot be shared in HGM VMAs. +*/ +int enable_hugetlb_hgm (struct vm_area_struct *vma) +{ +int ret; +struct hugetlb_shared_vma_data *data; + +if (!hugetlb_hgm_eligible (vma)) +return -EINVAL ; + +if (hugetlb_hgm_enabled (vma)) +return 0 ;
mm/hugetlb.c __weak unsigned long hugetlb_mask_last_page(struct hstate *h) · line 7348
Quoted 10 lines
mm/hugetlb.c +10
-0+ +/* +* We must hold the mmap lock for writing so that callers can rely on +* hugetlb_hgm_enabled returning a consistent result while holding +* the mmap lock for reading. +*/ +mmap_assert_write_locked (vma->vm_mm); + +/* HugeTLB HGM requires the VMA lock to synchronize collapsing. */ + ret =hugetlb_vma_data_alloc (vma);
Confused we need to vma_data_alloc() here. Shouldn't this be done by hugetlb_vm_op_open()?
Quoted 19 lines
mm/hugetlb.c +13
-1+if (ret) +return ret; + + data = vma->vm_private_data; +BUG_ON (!data); + data->hgm_enabled = true; + +/* We don't support PMD sharing with HGM. */ +hugetlb_unshare_all_pmds (vma); + return 0; +} +#endif /* CONFIG_HUGETLB_HIGH_GRANULARITY_MAPPING */ +/* * These functions are overwritable if your architecture needs its own * behavior. -- 2.38.0.135.g90850a2211-goog
Re: [RFC PATCH v2 08/47] hugetlb: add HGM enablement functions
Quoted 100 lines · Mina Almasry
Quoted 98 lines · James Houghton
Currently it is possible for all shared VMAs to use HGM, but it must be enabled first. This is because with HGM, we lose PMD sharing, and page table walks require additional synchronization (we need to take the VMA lock).
Signed-off-by James Houghton <[email protected]>--- include/linux/hugetlb.h | 22 +++++++++++++ mm/hugetlb.c | 69 +++++++++++++++++++++++++++++++++++++++++ 2 files changed, 91 insertions(+)include/linux/hugetlb.h +22
-0@@ -123,6 +123,9 @@ struct hugetlb_vma_lock {struct hugetlb_shared_vma_data {struct hugetlb_vma_lock vma_lock; +#ifdef CONFIG_HUGETLB_HIGH_GRANULARITY_MAPPING +bool hgm_enabled; +#endif };extern struct resv_map *resv_map_alloc (void );@@ -1179,6 +1182,25 @@ static inline void hugetlb_unregister_node(struct node *node) }#endif /* CONFIG_HUGETLB_PAGE */ +#ifdef CONFIG_HUGETLB_HIGH_GRANULARITY_MAPPING +bool hugetlb_hgm_enabled (struct vm_area_struct *vma); +bool hugetlb_hgm_eligible (struct vm_area_struct *vma); +int enable_hugetlb_hgm (struct vm_area_struct *vma); +#else +static inline bool hugetlb_hgm_enabled (struct vm_area_struct *vma) +{ +return false; +} +static inline bool hugetlb_hgm_eligible (struct vm_area_struct *vma) +{ +return false; +} +static inline int enable_hugetlb_hgm (struct vm_area_struct *vma) +{ +return -EINVAL ; +} +#endif +static inline spinlock_t *huge_pte_lock (struct hstate *h,struct mm_struct *mm,pte_t *pte) {mm/hugetlb.c +24
-0@@ -6840,6 +6840,10 @@ static bool pmd_sharing_possible(struct vm_area_struct *vma)#ifdef CONFIG_USERFAULTFD if (uffd_disable_huge_pmd_share (vma))return false; +#endif +#ifdef CONFIG_HUGETLB_HIGH_GRANULARITY_MAPPING +if (hugetlb_hgm_enabled (vma)) +return false;#endif /* * Only shared VMAs can share PMDs. @@ -7033,6 +7037,9 @@ static int hugetlb_vma_data_alloc(struct vm_area_struct *vma)kref_init (&data->vma_lock.refs);init_rwsem (&data->vma_lock.rw_sema); data->vma_lock.vma = vma; +#ifdef CONFIG_HUGETLB_HIGH_GRANULARITY_MAPPING + data->hgm_enabled = false; +#endif vma->vm_private_data = data;return 0 ; }@@ -7290,6 +7297,68 @@ __weak unsigned long hugetlb_mask_last_page(struct hstate *h)#endif /* CONFIG_ARCH_WANT_GENERAL_HUGETLB */ +#ifdef CONFIG_HUGETLB_HIGH_GRANULARITY_MAPPING +bool hugetlb_hgm_eligible (struct vm_area_struct *vma) +{ +/* +* All shared VMAs may have HGM. +* +* HGM requires using the VMA lock, which only exists for shared VMAs. +* To make HGM work for private VMAs, we would need to use another +* scheme to prevent collapsing/splitting from invalidating other +* threads' page table walks. +*/ +return vma && (vma->vm_flags &VM_MAYSHARE ); +} +bool hugetlb_hgm_enabled (struct vm_area_struct *vma) +{ +struct hugetlb_shared_vma_data *data = vma->vm_private_data; +
Quoted 8 lines
mm/hugetlb.c __weak unsigned long hugetlb_mask_last_page(struct hstate *h) · line 7320
Quoted 4 lines
mm/hugetlb.c +4
-0+if (!vma || !(vma->vm_flags &VM_MAYSHARE )) +return false; + +return data && data->hgm_enabled;Don't you need to lock
data->vma_lockbefore you access data? Or did I misunderstand the locking? Or are you assuming this is safe beforehgm_enabledcan't be disabled?
This should be protected by the mmap_lock (we must be holding it for at least reading here). data and data->hgm_enabled are only changed when holding the mmap_lock for writing.
Quoted 23 lines
Quoted 23 lines
mm/hugetlb.c +23
-0+} + +/* +* Enable high-granularity mapping (HGM) for this VMA. Once enabled, HGM +* cannot be turned off. +* +* PMDs cannot be shared in HGM VMAs. +*/ +int enable_hugetlb_hgm (struct vm_area_struct *vma) +{ +int ret; +struct hugetlb_shared_vma_data *data; + +if (!hugetlb_hgm_eligible (vma)) +return -EINVAL ; + +if (hugetlb_hgm_enabled (vma)) +return 0 ; + +/* +* We must hold the mmap lock for writing so that callers can rely on +* hugetlb_hgm_enabled returning a consistent result while holding +* the mmap lock for reading.
Quoted 8 lines
mm/hugetlb.c __weak unsigned long hugetlb_mask_last_page(struct hstate *h) · line 7348
Quoted 5 lines
mm/hugetlb.c +5
-0+*/ +mmap_assert_write_locked (vma->vm_mm); + +/* HugeTLB HGM requires the VMA lock to synchronize collapsing. */ + ret =hugetlb_vma_data_alloc (vma);Confused we need to
vma_data_alloc()here. Shouldn't this be done byhugetlb_vm_op_open()?
hugetlb_vma_data_alloc() can fail. In hugetlb_vm_op_open()/other places, it is allowed to fail, and so we call it again here and check that it succeeded so that we can rely on the VMA lock.
I think I need to be a little bit more careful with how I handle VMA splitting, though. It's possible for data not to be allocated after we split, but for some things to be mapped at high-granularity. The easiest solution here would be to disallow splitting when HGM is enabled; not sure what the best solution is though.
Thanks for the review, Mina!
Quoted 20 lines
Quoted 20 lines
mm/hugetlb.c +13
-1+if (ret) +return ret; + + data = vma->vm_private_data; +BUG_ON (!data); + data->hgm_enabled = true; + +/* We don't support PMD sharing with HGM. */ +hugetlb_unshare_all_pmds (vma); + return 0; +} +#endif /* CONFIG_HUGETLB_HIGH_GRANULARITY_MAPPING */ +/* * These functions are overwritable if your architecture needs its own * behavior. -- 2.38.0.135.g90850a2211-goog
Re: [RFC PATCH v2 08/47] hugetlb: add HGM enablement functions
In reply to James HoughtonQuoted 4 lines · James Houghton
Currently it is possible for all shared VMAs to use HGM, but it must be enabled first. This is because with HGM, we lose PMD sharing, and page table walks require additional synchronization (we need to take the VMA lock).
Not sure yet, but I expect Peter's series will help with locking for hugetlb specific page table walks.
Quoted 78 lines
Signed-off-by James Houghton <[email protected]>--- include/linux/hugetlb.h | 22 +++++++++++++ mm/hugetlb.c | 69 +++++++++++++++++++++++++++++++++++++++++ 2 files changed, 91 insertions(+)include/linux/hugetlb.h +22
-0@@ -123,6 +123,9 @@ struct hugetlb_vma_lock {struct hugetlb_shared_vma_data {struct hugetlb_vma_lock vma_lock; +#ifdef CONFIG_HUGETLB_HIGH_GRANULARITY_MAPPING +bool hgm_enabled; +#endif };extern struct resv_map *resv_map_alloc (void );@@ -1179,6 +1182,25 @@ static inline void hugetlb_unregister_node(struct node *node) }#endif /* CONFIG_HUGETLB_PAGE */ +#ifdef CONFIG_HUGETLB_HIGH_GRANULARITY_MAPPING +bool hugetlb_hgm_enabled (struct vm_area_struct *vma); +bool hugetlb_hgm_eligible (struct vm_area_struct *vma); +int enable_hugetlb_hgm (struct vm_area_struct *vma); +#else +static inline bool hugetlb_hgm_enabled (struct vm_area_struct *vma) +{ +return false; +} +static inline bool hugetlb_hgm_eligible (struct vm_area_struct *vma) +{ +return false; +} +static inline int enable_hugetlb_hgm (struct vm_area_struct *vma) +{ +return -EINVAL ; +} +#endif +static inline spinlock_t *huge_pte_lock (struct hstate *h,struct mm_struct *mm,pte_t *pte) {mm/hugetlb.c +9
-0@@ -6840,6 +6840,10 @@ static bool pmd_sharing_possible(struct vm_area_struct *vma)#ifdef CONFIG_USERFAULTFD if (uffd_disable_huge_pmd_share (vma))return false; +#endif +#ifdef CONFIG_HUGETLB_HIGH_GRANULARITY_MAPPING +if (hugetlb_hgm_enabled (vma)) +return false;#endif /* * Only shared VMAs can share PMDs. @@ -7033,6 +7037,9 @@ static int hugetlb_vma_data_alloc(struct vm_area_struct *vma)kref_init (&data->vma_lock.refs);init_rwsem (&data->vma_lock.rw_sema); data->vma_lock.vma = vma; +#ifdef CONFIG_HUGETLB_HIGH_GRANULARITY_MAPPING + data->hgm_enabled = false; +#endif vma->vm_private_data = data;return 0 ; }@@ -7290,6 +7297,68 @@ __weak unsigned long hugetlb_mask_last_page(struct hstate *h)#endif /* CONFIG_ARCH_WANT_GENERAL_HUGETLB */ +#ifdef CONFIG_HUGETLB_HIGH_GRANULARITY_MAPPING +bool hugetlb_hgm_eligible (struct vm_area_struct *vma)
mm/hugetlb.c __weak unsigned long hugetlb_mask_last_page(struct hstate *h) · line 7311
Quoted 10 lines
mm/hugetlb.c +10
-0+{ +/* +* All shared VMAs may have HGM. +* +* HGM requires using the VMA lock, which only exists for shared VMAs. +* To make HGM work for private VMAs, we would need to use another +* scheme to prevent collapsing/splitting from invalidating other +* threads' page table walks. +*/ +return vma && (vma->vm_flags &VM_MAYSHARE );
I am not yet 100% convinced you can/will take care of all possible code paths where hugetlb_vma_data allocation may fail. If not, then you should be checking vm_private_data here as well.
Quoted 32 lines
mm/hugetlb.c +32
-0+} +bool hugetlb_hgm_enabled (struct vm_area_struct *vma) +{ +struct hugetlb_shared_vma_data *data = vma->vm_private_data; + +if (!vma || !(vma->vm_flags &VM_MAYSHARE )) +return false; + +return data && data->hgm_enabled; +} + +/* +* Enable high-granularity mapping (HGM) for this VMA. Once enabled, HGM +* cannot be turned off. +* +* PMDs cannot be shared in HGM VMAs. +*/ +int enable_hugetlb_hgm (struct vm_area_struct *vma) +{ +int ret; +struct hugetlb_shared_vma_data *data; + +if (!hugetlb_hgm_eligible (vma)) +return -EINVAL ; + +if (hugetlb_hgm_enabled (vma)) +return 0 ; + +/* +* We must hold the mmap lock for writing so that callers can rely on +* hugetlb_hgm_enabled returning a consistent result while holding +* the mmap lock for reading.
mm/hugetlb.c __weak unsigned long hugetlb_mask_last_page(struct hstate *h) · line 7353
Quoted 10 lines
mm/hugetlb.c +10
-0+*/ +mmap_assert_write_locked (vma->vm_mm); + +/* HugeTLB HGM requires the VMA lock to synchronize collapsing. */ + ret =hugetlb_vma_data_alloc (vma); +if (ret) +return ret; + + data = vma->vm_private_data; +BUG_ON (!data);
Would rather have hugetlb_hgm_eligible check for vm_private_data as suggested above instead of the BUG here.
Signature · 1 line
Mike Kravetz
Quoted 14 lines
mm/hugetlb.c +8
-0+ data->hgm_enabled = true; + +/* We don't support PMD sharing with HGM. */ +hugetlb_unshare_all_pmds (vma); + return 0; +} +#endif /* CONFIG_HUGETLB_HIGH_GRANULARITY_MAPPING */ +/* * These functions are overwritable if your architecture needs its own * behavior. Signature · 1 line
2.38.0.135.g90850a2211-goog
Re: [RFC PATCH v2 08/47] hugetlb: add HGM enablement functions
Quoted 9 lines · Mike Kravetz
Quoted 4 lines · James Houghton
Currently it is possible for all shared VMAs to use HGM, but it must be enabled first. This is because with HGM, we lose PMD sharing, and page table walks require additional synchronization (we need to take the VMA lock).
Not sure yet, but I expect Peter's series will help with locking for hugetlb specific page table walks.
It should make things a little bit cleaner in this series; I'll rebase HGM on top of those patches this week (and hopefully get a v1 out soon).
I don't think it's possible to implement MADV_COLLAPSE with RCU alone (as implemented in Peter's series anyway); we still need the VMA lock.
Quoted 77 lines
Quoted 77 lines
Signed-off-by James Houghton <[email protected]>--- include/linux/hugetlb.h | 22 +++++++++++++ mm/hugetlb.c | 69 +++++++++++++++++++++++++++++++++++++++++ 2 files changed, 91 insertions(+)include/linux/hugetlb.h +22
-0@@ -123,6 +123,9 @@ struct hugetlb_vma_lock {struct hugetlb_shared_vma_data {struct hugetlb_vma_lock vma_lock; +#ifdef CONFIG_HUGETLB_HIGH_GRANULARITY_MAPPING +bool hgm_enabled; +#endif };extern struct resv_map *resv_map_alloc (void );@@ -1179,6 +1182,25 @@ static inline void hugetlb_unregister_node(struct node *node) }#endif /* CONFIG_HUGETLB_PAGE */ +#ifdef CONFIG_HUGETLB_HIGH_GRANULARITY_MAPPING +bool hugetlb_hgm_enabled (struct vm_area_struct *vma); +bool hugetlb_hgm_eligible (struct vm_area_struct *vma); +int enable_hugetlb_hgm (struct vm_area_struct *vma); +#else +static inline bool hugetlb_hgm_enabled (struct vm_area_struct *vma) +{ +return false; +} +static inline bool hugetlb_hgm_eligible (struct vm_area_struct *vma) +{ +return false; +} +static inline int enable_hugetlb_hgm (struct vm_area_struct *vma) +{ +return -EINVAL ; +} +#endif +static inline spinlock_t *huge_pte_lock (struct hstate *h,struct mm_struct *mm,pte_t *pte) {mm/hugetlb.c +7
-0@@ -6840,6 +6840,10 @@ static bool pmd_sharing_possible(struct vm_area_struct *vma)#ifdef CONFIG_USERFAULTFD if (uffd_disable_huge_pmd_share (vma))return false; +#endif +#ifdef CONFIG_HUGETLB_HIGH_GRANULARITY_MAPPING +if (hugetlb_hgm_enabled (vma)) +return false;#endif /* * Only shared VMAs can share PMDs. @@ -7033,6 +7037,9 @@ static int hugetlb_vma_data_alloc(struct vm_area_struct *vma)kref_init (&data->vma_lock.refs);init_rwsem (&data->vma_lock.rw_sema); data->vma_lock.vma = vma; +#ifdef CONFIG_HUGETLB_HIGH_GRANULARITY_MAPPING + data->hgm_enabled = false; +#endif vma->vm_private_data = data;return 0 ; }@@ -7290,6 +7297,68 @@ __weak unsigned long hugetlb_mask_last_page(struct hstate *h)#endif /* CONFIG_ARCH_WANT_GENERAL_HUGETLB */
Quoted 16 lines
mm/hugetlb.c __weak unsigned long hugetlb_mask_last_page(struct hstate *h) · line 7311
Quoted 12 lines
mm/hugetlb.c +12
-0+#ifdef CONFIG_HUGETLB_HIGH_GRANULARITY_MAPPING +bool hugetlb_hgm_eligible (struct vm_area_struct *vma) +{ +/* +* All shared VMAs may have HGM. +* +* HGM requires using the VMA lock, which only exists for shared VMAs. +* To make HGM work for private VMAs, we would need to use another +* scheme to prevent collapsing/splitting from invalidating other +* threads' page table walks. +*/ +return vma && (vma->vm_flags &VM_MAYSHARE );I am not yet 100% convinced you can/will take care of all possible code paths where
hugetlb_vma_dataallocation may fail. If not, then you should be checkingvm_private_datahere as well.
I think the check here makes sense -- if a VMA is shared, then it is eligible for HGM, but we might fail to enable it because we can't allocate the VMA lock. I'll reword the comment to clearly say this.
There is the problem of splitting, though: if we have high-granularity mapped PTEs in a VMA and that VMA gets split, we need to remember that the VMA had HGM enabled even if allocating the VMA lock fails, otherwise things get out of sync. How does PMD sharing handle the splitting case?
An easy way HGM could handle this is by disallowing splitting, but I think we can do better. If we fail to allocate the VMA lock, then we can no longer MADV_COLLAPSE safely, but everything else can proceed as normal, and so some "hugetlb_hgm_enabled" checks can be removed/changed. This should make things easier for when we have to handle (some bits of) HGM for private mappings, too. I'll make some improvements here for v1.
Quoted 38 lines
Quoted 38 lines
mm/hugetlb.c +37
-0+} +bool hugetlb_hgm_enabled (struct vm_area_struct *vma) +{ +struct hugetlb_shared_vma_data *data = vma->vm_private_data; + +if (!vma || !(vma->vm_flags &VM_MAYSHARE )) +return false; + +return data && data->hgm_enabled; +} + +/* +* Enable high-granularity mapping (HGM) for this VMA. Once enabled, HGM +* cannot be turned off. +* +* PMDs cannot be shared in HGM VMAs. +*/ +int enable_hugetlb_hgm (struct vm_area_struct *vma) +{ +int ret; +struct hugetlb_shared_vma_data *data; + +if (!hugetlb_hgm_eligible (vma)) +return -EINVAL ; + +if (hugetlb_hgm_enabled (vma)) +return 0 ; + +/* +* We must hold the mmap lock for writing so that callers can rely on +* hugetlb_hgm_enabled returning a consistent result while holding +* the mmap lock for reading. +*/ +mmap_assert_write_locked (vma->vm_mm); + +/* HugeTLB HGM requires the VMA lock to synchronize collapsing. */ + ret =hugetlb_vma_data_alloc (vma);
Quoted 8 lines
mm/hugetlb.c __weak unsigned long hugetlb_mask_last_page(struct hstate *h) · line 7353
Quoted 5 lines
mm/hugetlb.c +5
-0+if (ret) +return ret; + + data = vma->vm_private_data; +BUG_ON (!data);Would rather have
hugetlb_hgm_eligiblecheck forvm_private_dataas suggested above instead of the BUG here.
I don't think we'd ever actually BUG() here. Please correct me if I'm wrong, but if we are eligible for HGM, then hugetlb_vma_data_alloc() will only succeed if we actually allocated the VMA data/lock, so vma->vm_private_data should never be NULL (with the BUG_ON to inform the reader). Maybe I should just drop the BUG()?
Quoted 18 lines
--
Mike KravetzQuoted 14 lines
mm/hugetlb.c +8
-1+ data->hgm_enabled = true; + +/* We don't support PMD sharing with HGM. */ +hugetlb_unshare_all_pmds (vma); + return 0; +} +#endif /* CONFIG_HUGETLB_HIGH_GRANULARITY_MAPPING */ +/* * These functions are overwritable if your architecture needs its own * behavior. -- 2.38.0.135.g90850a2211-goog
Re: [RFC PATCH v2 08/47] hugetlb: add HGM enablement functions
Quoted 11 lines · James Houghton
Quoted 9 lines · Mike Kravetz
Quoted 4 lines · James Houghton
Currently it is possible for all shared VMAs to use HGM, but it must be enabled first. This is because with HGM, we lose PMD sharing, and page table walks require additional synchronization (we need to take the VMA lock).
Not sure yet, but I expect Peter's series will help with locking for hugetlb specific page table walks.
Quoted 6 lines
It should make things a little bit cleaner in this series; I'll rebase HGM on top of those patches this week (and hopefully get a v1 out soon).
I don't think it's possible to implement
MADV_COLLAPSEwith RCU alone (as implemented in Peter's series anyway); we still need the VMA lock.
As I continue going through the series, I realize that I am not exactly sure what synchronization by the vma lock is required by HGM. As you are aware, it was originally designed to protect against someone doing a pmd_unshare and effectively removing part of the page table. However, since pmd sharing is disabled for vmas with HGM enabled (I think?), then it might be a good idea to explicitly say somewhere the reason for using the lock.
Signature · 1 line
Mike Kravetz
Re: [RFC PATCH v2 08/47] hugetlb: add HGM enablement functions
Quoted 20 lines · Mike Kravetz
Quoted 11 lines · James Houghton
Quoted 9 lines · Mike Kravetz
Quoted 4 lines · James Houghton
Currently it is possible for all shared VMAs to use HGM, but it must be enabled first. This is because with HGM, we lose PMD sharing, and page table walks require additional synchronization (we need to take the VMA lock).
Not sure yet, but I expect Peter's series will help with locking for hugetlb specific page table walks.
Quoted 6 lines
It should make things a little bit cleaner in this series; I'll rebase HGM on top of those patches this week (and hopefully get a v1 out soon).
I don't think it's possible to implement
MADV_COLLAPSEwith RCU alone (as implemented in Peter's series anyway); we still need the VMA lock.
Quoted 7 lines
As I continue going through the series, I realize that I am not exactly sure what synchronization by the vma lock is required by HGM. As you are aware, it was originally designed to protect against someone doing a
pmd_unshareand effectively removing part of the page table. However, since pmd sharing is disabled for vmas with HGM enabled (I think?), then it might be a good idea to explicitly say somewhere the reason for using the lock.
It synchronizes MADV_COLLAPSE for hugetlb (hugetlb_collapse). MADV_COLLAPSE will take it for writing and free some page table pages, and high-granularity walks will generally take it for reading. I'll make this clear in a comment somewhere and in commit messages.
It might be easier if hugetlb_collapse() had the exact same synchronization as huge_pmd_unshare, where we not only take the VMA lock for writing, we also take the i_mmap_rw_sem for writing, so anywhere where hugetlb_walk() is safe, high-granularity walks are also safe. I think I should just do that for the sake of simplicity.
- James
Quoted 2 lines
--
Mike Kravetz
RFC PATCH v2 9/47 hugetlb: make huge_pte_lockptr take an explicit shift argument.
In reply to James HoughtonThis is needed to handle PTL locking with high-granularity mapping. We won't always be using the PMD-level PTL even if we're using the 2M hugepage hstate. It's possible that we're dealing with 4K PTEs, in which case, we need to lock the PTL for the 4K PTE.
--- arch/powerpc/mm/pgtable.c | 3 ++- include/linux/hugetlb.h | 9 ++++----- mm/hugetlb.c | 7 ++++--- mm/migrate.c | 3 ++- 4 files changed, 12 insertions(+), 10 deletions(-)
arch/powerpc/mm/pgtable.c +2 -1
@@ -261,7 +261,8 @@ int huge_ptep_set_access_flags(struct vm_area_struct *vma, psize =hstate_get_psize (h);#ifdef CONFIG_DEBUG_VM -assert_spin_locked (huge_pte_lockptr (h, vma->vm_mm, ptep)); +assert_spin_locked (huge_pte_lockptr (huge_page_shift (h), + vma->vm_mm, ptep));#endif #else
include/linux/hugetlb.h +4 -5
@@ -934,12 +934,11 @@ static inline gfp_t htlb_modify_alloc_mask(struct hstate *h, gfp_t gfp_mask)return modified_mask; } -static inline spinlock_t *huge_pte_lockptr (struct hstate *h, +static inline spinlock_t *huge_pte_lockptr (unsigned int shift,struct mm_struct *mm,pte_t *pte) { -if (huge_page_size (h) ==PMD_SIZE ) +if (shift ==PMD_SHIFT )return pmd_lockptr (mm, (pmd_t *) pte); -VM_BUG_ON (huge_page_size (h) ==PAGE_SIZE );return &mm->page_table_lock; }
@@ -1144,7 +1143,7 @@ static inline gfp_t htlb_modify_alloc_mask(struct hstate *h, gfp_t gfp_mask)return 0 ; } -static inline spinlock_t *huge_pte_lockptr (struct hstate *h, +static inline spinlock_t *huge_pte_lockptr (unsigned int shift,struct mm_struct *mm,pte_t *pte) {return &mm->page_table_lock;
@@ -1206,7 +1205,7 @@ static inline spinlock_t *huge_pte_lock(struct hstate *h, {spinlock_t *ptl; - ptl =huge_pte_lockptr (h, mm, pte); + ptl =huge_pte_lockptr (huge_page_shift (h), mm, pte);spin_lock (ptl);return ptl; }
mm/hugetlb.c +4 -3
@@ -4847,7 +4847,7 @@ int copy_hugetlb_page_range(struct mm_struct *dst, struct mm_struct *src, } dst_ptl =huge_pte_lock (h, dst, dst_pte); - src_ptl =huge_pte_lockptr (h, src, src_pte); + src_ptl =huge_pte_lockptr (huge_page_shift (h), src, src_pte);spin_lock_nested (src_ptl,SINGLE_DEPTH_NESTING ); entry =huge_ptep_get (src_pte); again:
@@ -4925,7 +4925,8 @@ int copy_hugetlb_page_range(struct mm_struct *dst, struct mm_struct *src,/* Install the new huge page if src pte stable */ dst_ptl =huge_pte_lock (h, dst, dst_pte); - src_ptl =huge_pte_lockptr (h, src, src_pte); + src_ptl =huge_pte_lockptr (huge_page_shift (h), + src, src_pte);spin_lock_nested (src_ptl,SINGLE_DEPTH_NESTING ); entry =huge_ptep_get (src_pte);if (!pte_same (src_pte_old, entry)) {
@@ -4979,7 +4980,7 @@ static void move_huge_pte(struct vm_area_struct *vma, unsigned long old_addr,pte_t pte; dst_ptl =huge_pte_lock (h, mm, dst_pte); - src_ptl =huge_pte_lockptr (h, mm, src_pte); + src_ptl =huge_pte_lockptr (huge_page_shift (h), mm, src_pte);/* * We don't have to worry about the ordering of src and dst ptlocks
mm/migrate.c +2 -1
@@ -334,7 +334,8 @@ void __migration_entry_wait_huge(pte_t *ptep, spinlock_t *ptl)void migration_entry_wait_huge (struct vm_area_struct *vma,pte_t *pte) { -spinlock_t *ptl =huge_pte_lockptr (hstate_vma (vma), vma->vm_mm, pte); +spinlock_t *ptl =huge_pte_lockptr (huge_page_shift (hstate_vma (vma)), + vma->vm_mm, pte);__migration_entry_wait_huge (pte, ptl); }
Signature · 1 line
2.38.0.135.g90850a2211-goog
Re: [RFC PATCH v2 09/47] hugetlb: make huge_pte_lockptr take an explicit shift argument.
Quoted 7 lines · James Houghton
This is needed to handle PTL locking with high-granularity mapping. We won't always be using the PMD-level PTL even if we're using the 2M hugepage hstate. It's possible that we're dealing with 4K PTEs, in which case, we need to lock the PTL for the 4K PTE.
Signed-off-by James Houghton <[email protected]>
Quoted 107 lines
--- arch/powerpc/mm/pgtable.c | 3 ++- include/linux/hugetlb.h | 9 ++++----- mm/hugetlb.c | 7 ++++--- mm/migrate.c | 3 ++- 4 files changed, 12 insertions(+), 10 deletions(-)arch/powerpc/mm/pgtable.c +2
-1@@ -261,7 +261,8 @@ int huge_ptep_set_access_flags(struct vm_area_struct *vma, psize =hstate_get_psize (h);#ifdef CONFIG_DEBUG_VM -assert_spin_locked (huge_pte_lockptr (h, vma->vm_mm, ptep)); +assert_spin_locked (huge_pte_lockptr (huge_page_shift (h), + vma->vm_mm, ptep));#endif #else include/linux/hugetlb.h +4
-5@@ -934,12 +934,11 @@ static inline gfp_t htlb_modify_alloc_mask(struct hstate *h, gfp_t gfp_mask)return modified_mask; } -static inline spinlock_t *huge_pte_lockptr (struct hstate *h, +static inline spinlock_t *huge_pte_lockptr (unsigned int shift,struct mm_struct *mm,pte_t *pte) { -if (huge_page_size (h) ==PMD_SIZE ) +if (shift ==PMD_SHIFT )return pmd_lockptr (mm, (pmd_t *) pte); -VM_BUG_ON (huge_page_size (h) ==PAGE_SIZE );return &mm->page_table_lock; }@@ -1144,7 +1143,7 @@ static inline gfp_t htlb_modify_alloc_mask(struct hstate *h, gfp_t gfp_mask)return 0 ; } -static inline spinlock_t *huge_pte_lockptr (struct hstate *h, +static inline spinlock_t *huge_pte_lockptr (unsigned int shift,struct mm_struct *mm,pte_t *pte) {return &mm->page_table_lock;@@ -1206,7 +1205,7 @@ static inline spinlock_t *huge_pte_lock(struct hstate *h, {spinlock_t *ptl; - ptl =huge_pte_lockptr (h, mm, pte); + ptl =huge_pte_lockptr (huge_page_shift (h), mm, pte);spin_lock (ptl);return ptl; }mm/hugetlb.c +4
-3@@ -4847,7 +4847,7 @@ int copy_hugetlb_page_range(struct mm_struct *dst, struct mm_struct *src, } dst_ptl =huge_pte_lock (h, dst, dst_pte); - src_ptl =huge_pte_lockptr (h, src, src_pte); + src_ptl =huge_pte_lockptr (huge_page_shift (h), src, src_pte);spin_lock_nested (src_ptl,SINGLE_DEPTH_NESTING ); entry =huge_ptep_get (src_pte); again:@@ -4925,7 +4925,8 @@ int copy_hugetlb_page_range(struct mm_struct *dst, struct mm_struct *src,/* Install the new huge page if src pte stable */ dst_ptl =huge_pte_lock (h, dst, dst_pte); - src_ptl =huge_pte_lockptr (h, src, src_pte); + src_ptl =huge_pte_lockptr (huge_page_shift (h), + src, src_pte);spin_lock_nested (src_ptl,SINGLE_DEPTH_NESTING ); entry =huge_ptep_get (src_pte);if (!pte_same (src_pte_old, entry)) {@@ -4979,7 +4980,7 @@ static void move_huge_pte(struct vm_area_struct *vma, unsigned long old_addr,pte_t pte; dst_ptl =huge_pte_lock (h, mm, dst_pte); - src_ptl =huge_pte_lockptr (h, mm, src_pte); + src_ptl =huge_pte_lockptr (huge_page_shift (h), mm, src_pte);/* * We don't have to worry about the ordering of src and dst ptlocks mm/migrate.c +2
-2@@ -334,7 +334,8 @@ void __migration_entry_wait_huge(pte_t *ptep, spinlock_t *ptl)void migration_entry_wait_huge (struct vm_area_struct *vma,pte_t *pte) { -spinlock_t *ptl =huge_pte_lockptr (hstate_vma (vma), vma->vm_mm, pte); +spinlock_t *ptl =huge_pte_lockptr (huge_page_shift (hstate_vma (vma)), + vma->vm_mm, pte);__migration_entry_wait_huge (pte, ptl); } --2.38.0.135.g90850a2211-goog
Re: [RFC PATCH v2 09/47] hugetlb: make huge_pte_lockptr take an explicit shift argument.
In reply to James HoughtonQuoted 5 lines · James Houghton
This is needed to handle PTL locking with high-granularity mapping. We won't always be using the PMD-level PTL even if we're using the 2M hugepage hstate. It's possible that we're dealing with 4K PTEs, in which case, we need to lock the PTL for the 4K PTE.
Quoted 7 lines
Signed-off-by James Houghton <[email protected]>--- arch/powerpc/mm/pgtable.c | 3 ++- include/linux/hugetlb.h | 9 ++++----- mm/hugetlb.c | 7 ++++--- mm/migrate.c | 3 ++- 4 files changed, 12 insertions(+), 10 deletions(-)
Straight forward substitution,
Signature · 1 line
Mike Kravetz
RFC PATCH v2 10/47 hugetlb: add hugetlb_pte to track HugeTLB page table entries
In reply to James HoughtonAfter high-granularity mapping, page table entries for HugeTLB pages can be of any size/type. (For example, we can have a 1G page mapped with a mix of PMDs and PTEs.) This struct is to help keep track of a HugeTLB PTE after we have done a page table walk.
Without this, we'd have to pass around the "size" of the PTE everywhere. We effectively did this before; it could be fetched from the hstate, which we pass around pretty much everywhere.
hugetlb_pte_present_leaf is included here as a helper function that will be used frequently later on.
--- include/linux/hugetlb.h | 88 +++++++++++++++++++++++++++++++++++++++++ mm/hugetlb.c | 29 ++++++++++++++ 2 files changed, 117 insertions(+)
include/linux/hugetlb.h +88 -0
@@ -50,6 +50,75 @@ enum { __NR_USED_SUBPAGE, }; +enum hugetlb_level { +HUGETLB_LEVEL_PTE =1 , +/* +* We always include PMD, PUD, and P4D in this enum definition so that, +* when logged as an integer, we can easily tell which level it is. +*/ +HUGETLB_LEVEL_PMD , +HUGETLB_LEVEL_PUD , +HUGETLB_LEVEL_P4D , +HUGETLB_LEVEL_PGD , +}; + +struct hugetlb_pte { +pte_t *ptep; +unsigned int shift; +enum hugetlb_level level; +spinlock_t *ptl; +}; + +static inline +void hugetlb_pte_populate (struct hugetlb_pte *hpte,pte_t *ptep, +unsigned int shift,enum hugetlb_level level) +{ +WARN_ON_ONCE (!ptep); + hpte->ptep = ptep; + hpte->shift = shift; + hpte->level = level; + hpte->ptl =NULL ; +} + +static inline +unsigned long hugetlb_pte_size (const struct hugetlb_pte *hpte) +{ +WARN_ON_ONCE (!hpte->ptep); +return 1UL << hpte->shift; +} + +static inline +unsigned long hugetlb_pte_mask (const struct hugetlb_pte *hpte) +{ +WARN_ON_ONCE (!hpte->ptep); +return ~(hugetlb_pte_size (hpte) -1 ); +} + +static inline +unsigned int hugetlb_pte_shift (const struct hugetlb_pte *hpte) +{ +WARN_ON_ONCE (!hpte->ptep); +return hpte->shift; +} + +static inline +enum hugetlb_level hugetlb_pte_level (const struct hugetlb_pte *hpte) +{ +WARN_ON_ONCE (!hpte->ptep); +return hpte->level; +} + +static inline +void hugetlb_pte_copy (struct hugetlb_pte *dest,const struct hugetlb_pte *src) +{ + dest->ptep = src->ptep; + dest->shift = src->shift; + dest->level = src->level; + dest->ptl = src->ptl; +} + +bool hugetlb_pte_present_leaf (const struct hugetlb_pte *hpte,pte_t pte); +struct hugepage_subpool {spinlock_t lock;long count;
@@ -1210,6 +1279,25 @@ static inline spinlock_t *huge_pte_lock(struct hstate *h,return ptl; } +static inline +spinlock_t *hugetlb_pte_lockptr (struct mm_struct *mm,struct hugetlb_pte *hpte) +{ + +BUG_ON (!hpte->ptep); +if (hpte->ptl) +return hpte->ptl; +return huge_pte_lockptr (hugetlb_pte_shift (hpte), mm, hpte->ptep); +} + +static inline +spinlock_t *hugetlb_pte_lock (struct mm_struct *mm,struct hugetlb_pte *hpte) +{ +spinlock_t *ptl =hugetlb_pte_lockptr (mm, hpte); + +spin_lock (ptl); +return ptl; +} +#if defined(CONFIG_HUGETLB_PAGE ) && defined(CONFIG_CMA )extern void __init hugetlb_cma_reserve (int order);#else
mm/hugetlb.c +29 -0
@@ -1127,6 +1127,35 @@ static bool vma_has_reserves(struct vm_area_struct *vma, long chg)return false; } +bool hugetlb_pte_present_leaf (const struct hugetlb_pte *hpte,pte_t pte) +{ +pgd_t pgd; +p4d_t p4d; +pud_t pud; +pmd_t pmd; + +WARN_ON_ONCE (!hpte->ptep); +switch (hugetlb_pte_level (hpte)) { +case HUGETLB_LEVEL_PGD : + pgd =__pgd (pte_val (pte)); +return pgd_present (pgd) &&pgd_leaf (pgd); +case HUGETLB_LEVEL_P4D : + p4d =__p4d (pte_val (pte)); +return p4d_present (p4d) &&p4d_leaf (p4d); +case HUGETLB_LEVEL_PUD : + pud =__pud (pte_val (pte)); +return pud_present (pud) &&pud_leaf (pud); +case HUGETLB_LEVEL_PMD : + pmd =__pmd (pte_val (pte)); +return pmd_present (pmd) &&pmd_leaf (pmd); +case HUGETLB_LEVEL_PTE : +return pte_present (pte); +default : +WARN_ON_ONCE (1 ); +return false; + } +} +static void enqueue_huge_page (struct hstate *h,struct page *page) {int nid =page_to_nid (page);
Signature · 1 line
2.38.0.135.g90850a2211-goog
Peter Xu
reviewerRe: [RFC PATCH v2 10/47] hugetlb: add hugetlb_pte to track HugeTLB page table entries
include/linux/hugetlb.h enum { · line 70
Quoted 6 lines · James Houghton
include/linux/hugetlb.h +6
-0+struct hugetlb_pte { +pte_t *ptep; +unsigned int shift; +enum hugetlb_level level; +spinlock_t *ptl; +};
Do we need both shift + level? Maybe it's only meaningful for ARM where the shift may not be directly calculcated from level?
I'm wondering whether we can just maintain "shift" then we calculate "level" realtime. It just reads a bit weird to have these two fields, also a burden to most of the call sites where shift and level exactly match..
include/linux/hugetlb.h enum { · line 74
Quoted 4 lines
include/linux/hugetlb.h +4
-0+ +static inline +void hugetlb_pte_populate (struct hugetlb_pte *hpte,pte_t *ptep, +unsigned int shift,enum hugetlb_level level)
I'd think it's nicer to replace "populate" with something else, as populate is definitely a meaningful word in vm world for "making something appear if it wasn't". Maybe hugetlb_pte_setup()?
Even one step back, on the naming of hugetlb_pte.. Sorry to comment on namings especially on this one, I really don't like to do that normally.. but here hugetlb_pte only walks the sub-page level of pgtables, meanwhile it's not really a pte but an iterator. How about hugetlb_hgm_iter? "hgm" tells that it only walks sub-level, and "iter" tells that it is an iterator, being updated for each stepping downwards.
Then hugetlb_pte_populate() can be hugetlb_hgm_iter_init().
Take these comments with a grain of salt, and it never hurts to wait for a 2nd opinion before anything.
Quoted 23 lines
include/linux/hugetlb.h +23
-0+{ +WARN_ON_ONCE (!ptep); + hpte->ptep = ptep; + hpte->shift = shift; + hpte->level = level; + hpte->ptl =NULL ; +} + +static inline +unsigned long hugetlb_pte_size (const struct hugetlb_pte *hpte) +{ +WARN_ON_ONCE (!hpte->ptep); +return 1UL << hpte->shift; +} + +static inline +unsigned long hugetlb_pte_mask (const struct hugetlb_pte *hpte) +{ +WARN_ON_ONCE (!hpte->ptep); +return ~(hugetlb_pte_size (hpte) -1 ); +} + +static inline
include/linux/hugetlb.h enum { · line 107
Quoted 10 lines
include/linux/hugetlb.h +10
-0+unsigned int hugetlb_pte_shift (const struct hugetlb_pte *hpte) +{ +WARN_ON_ONCE (!hpte->ptep); +return hpte->shift; +} + +static inline +enum hugetlb_level hugetlb_pte_level (const struct hugetlb_pte *hpte) +{ +WARN_ON_ONCE (!hpte->ptep);
There're definitely a bunch of hpte->ptep WARN_ON_ONCE()s.. AFAIK the hugetlb_pte* will be setup once with valid ptep and then it should always be. I rem someone commented on these helpers look not useful, which I must confess I had the same feeling. But besides that, I'd rather drop all these WARN_ON_ONCE()s but only check it when init() the iterator/pte.
Quoted 18 lines
include/linux/hugetlb.h +14
-0+return hpte->level; +} + +static inline +void hugetlb_pte_copy (struct hugetlb_pte *dest,const struct hugetlb_pte *src) +{ + dest->ptep = src->ptep; + dest->shift = src->shift; + dest->level = src->level; + dest->ptl = src->ptl; +} + +bool hugetlb_pte_present_leaf (const struct hugetlb_pte *hpte,pte_t pte); +struct hugepage_subpool {spinlock_t lock;long count;
include/linux/hugetlb.h static inline spinlock_t *huge_pte_lock(struct hstate *h, · line 1286
Quoted 8 lines
include/linux/hugetlb.h +5
-0@@ -1210,6 +1279,25 @@ static inline spinlock_t *huge_pte_lock(struct hstate *h,return ptl; } +static inline +spinlock_t *hugetlb_pte_lockptr (struct mm_struct *mm,struct hugetlb_pte *hpte) +{ + +BUG_ON (!hpte->ptep);
Another BUG_ON(); better be dropped too.
include/linux/hugetlb.h static inline spinlock_t *huge_pte_lock(struct hstate *h, · line 1289
Quoted 3 lines
include/linux/hugetlb.h +3
-0+if (hpte->ptl) +return hpte->ptl; +return huge_pte_lockptr (hugetlb_pte_shift (hpte), mm, hpte->ptep);
I'm curious whether we can always have hpte->ptl set for a valid hugetlb_pte. I think that means we'll need to also init the ptl in the init() fn of the iterator. Then it'll be clear on which lock to take for each valid hugetlb_pte.
Quoted 1 line
(quoted diff) +1
-0+}
Signature · 1 line
Peter Xu
Re: [RFC PATCH v2 10/47] hugetlb: add hugetlb_pte to track HugeTLB page table entries
Quoted 12 lines · Peter Xu
include/linux/hugetlb.h enum { · line 70
Quoted 6 lines · James Houghton
include/linux/hugetlb.h +6
-0+struct hugetlb_pte { +pte_t *ptep; +unsigned int shift; +enum hugetlb_level level; +spinlock_t *ptl; +};Do we need both shift + level? Maybe it's only meaningful for ARM where the shift may not be directly calculcated from level?
Quoted 3 lines
I'm wondering whether we can just maintain "shift" then we calculate "level" realtime. It just reads a bit weird to have these two fields, also a burden to most of the call sites where shift and level exactly match..
My main concern is interaction with folded levels. For example, if PUD_SIZE and PMD_SIZE are the same, we want to do something like this:
pud = pud_offset(p4d, addr)
pmd = pmd_offset(pud, addr) /* this is just pmd = (pmd_t *) pud */ pte = pte_offset(pmd, addr)
and I think we should avoid quietly skipping the folded level, which could happen:
pud = pud_offset(p4d, addr)
/* Each time, we go back to pte_t *, so if we stored PUD_SHIFT here, it is impossible to know that pud came from pud_offset and not pmd_offset. We must assume the deeper level so that we don't get stuck in a loop. */
pte = pte_offset(pud, addr) /* pud is cast from (pud_t * -> pte_t * -> pmd_t *) */
Quietly dropping p*d_offset for folded levels is safe; it's just a cast that we're doing anyway. If you think this is fine, then I can remove level. It might also be that this is a non-issue and that there will never be a folded level underneath a hugepage level.
We could also change ptep to a union eventually (to clean up "hugetlb casts everything to pte_t *" messiness), and having an explicit level as a tag for the union would be nice help. In the same way: I like having level explicitly so that we know for sure where ptep came from.
I can try to reduce the burden at the callsite while keeping level:hpage_size_to_level() is really annoying to have everywhere.
Quoted 17 lines
include/linux/hugetlb.h enum { · line 74
Quoted 5 lines
include/linux/hugetlb.h +4
-0+ +static inline +void hugetlb_pte_populate (struct hugetlb_pte *hpte,pte_t *ptep, +unsigned int shift,enum hugetlb_level level)I'd think it's nicer to replace "populate" with something else, as populate is definitely a meaningful word in vm world for "making something appear if it wasn't". Maybe
hugetlb_pte_setup()?Even one step back, on the naming of
hugetlb_pte.. Sorry to comment on namings especially on this one, I really don't like to do that normally.. but herehugetlb_pteonly walks the sub-page level of pgtables, meanwhile it's not really a pte but an iterator. How abouthugetlb_hgm_iter? "hgm" tells that it only walks sub-level, and "iter" tells that it is an iterator, being updated for each stepping downwards.
Quoted 4 lines
Then
hugetlb_pte_populate()can behugetlb_hgm_iter_init().Take these comments with a grain of salt, and it never hurts to wait for a 2nd opinion before anything.
I think this is a great idea. :) Thank you! I'll make this change for v1 unless someone has a better suggestion.
Quoted 32 lines
Quoted 32 lines
include/linux/hugetlb.h +31
-0+{ +WARN_ON_ONCE (!ptep); + hpte->ptep = ptep; + hpte->shift = shift; + hpte->level = level; + hpte->ptl =NULL ; +} + +static inline +unsigned long hugetlb_pte_size (const struct hugetlb_pte *hpte) +{ +WARN_ON_ONCE (!hpte->ptep); +return 1UL << hpte->shift; +} + +static inline +unsigned long hugetlb_pte_mask (const struct hugetlb_pte *hpte) +{ +WARN_ON_ONCE (!hpte->ptep); +return ~(hugetlb_pte_size (hpte) -1 ); +} + +static inline +unsigned int hugetlb_pte_shift (const struct hugetlb_pte *hpte) +{ +WARN_ON_ONCE (!hpte->ptep); +return hpte->shift; +} + +static inline +enum hugetlb_level hugetlb_pte_level (const struct hugetlb_pte *hpte)
Quoted 8 lines
Review
Quoted 2 lines
(quoted diff) +2
-0+{ + WARN_ON_ONCE(!hpte->ptep);There're definitely a bunch of
hpte->ptepWARN_ON_ONCE()s.. AFAIK the hugetlb_pte* will be setup once with valid ptep and then it should always be. I rem someone commented on these helpers look not useful, which I must confess I had the same feeling. But besides that, I'd rather drop all these WARN_ON_ONCE()s but only check it wheninit()the iterator/pte.
The idea with these WARN_ON_ONCE()s is that it WARNs for the case that hpte was never populated/initialized, but I realize that we can't even rely on hpte->ptep == NULL. I'll remove the WARN_ON_ONCE()s, and I'll drop hugetlb_pte_shift and hugetlb_pte_level entirely.
I'll keep the hugetlb_pte_{size,mask,copy,present_leaf} helpers as they are legitimately helpful.
Quoted 22 lines
Quoted 22 lines
include/linux/hugetlb.h +14
-0+return hpte->level; +} + +static inline +void hugetlb_pte_copy (struct hugetlb_pte *dest,const struct hugetlb_pte *src) +{ + dest->ptep = src->ptep; + dest->shift = src->shift; + dest->level = src->level; + dest->ptl = src->ptl; +} + +bool hugetlb_pte_present_leaf (const struct hugetlb_pte *hpte,pte_t pte); +struct hugepage_subpool {spinlock_t lock;long count;@@ -1210,6 +1279,25 @@ static inline spinlock_t *huge_pte_lock(struct hstate *h,return ptl; }
Quoted 7 lines
include/linux/hugetlb.h static inline spinlock_t *huge_pte_lock(struct hstate *h, · line 1286
Quoted 5 lines
include/linux/hugetlb.h +5
-0+static inline +spinlock_t *hugetlb_pte_lockptr (struct mm_struct *mm,struct hugetlb_pte *hpte) +{ + +BUG_ON (!hpte->ptep);Another
BUG_ON(); better be dropped too.
Can do.
Quoted 9 lines
include/linux/hugetlb.h static inline spinlock_t *huge_pte_lock(struct hstate *h, · line 1289
Quoted 4 lines
include/linux/hugetlb.h +3
-0+if (hpte->ptl) +return hpte->ptl; +return huge_pte_lockptr (hugetlb_pte_shift (hpte), mm, hpte->ptep);I'm curious whether we can always have
hpte->ptlset for a validhugetlb_pte. I think that means we'll need to also init the ptl in theinit()fn of the iterator. Then it'll be clear on which lock to take for each validhugetlb_pte.
I can work on this for v1. Right now it's not very good: for 4K PTEs, we manually set ->ptl while walking. I'll make it so that ->ptl is always populated so the code is easier to read.
- James
Quoted 7 lines
Review static inline spinlock_t *huge_pte_lock(struct hstate *h, · line 1290
Quoted 2 lines
(quoted diff) +1
-0+}--
Peter Xu
Peter Xu
reviewerRe: [RFC PATCH v2 10/47] hugetlb: add hugetlb_pte to track HugeTLB page table entries
Quoted 40 lines · James Houghton
Quoted 12 lines · Peter Xu
include/linux/hugetlb.h enum { · line 70
Quoted 6 lines · James Houghton
include/linux/hugetlb.h +6
-0+struct hugetlb_pte { +pte_t *ptep; +unsigned int shift; +enum hugetlb_level level; +spinlock_t *ptl; +};Do we need both shift + level? Maybe it's only meaningful for ARM where the shift may not be directly calculcated from level?
Quoted 3 lines
I'm wondering whether we can just maintain "shift" then we calculate "level" realtime. It just reads a bit weird to have these two fields, also a burden to most of the call sites where shift and level exactly match..
My main concern is interaction with folded levels. For example, if
PUD_SIZEandPMD_SIZEare the same, we want to do something like this:pud =
pud_offset(p4d, addr)
pmd =pmd_offset(pud, addr)/* this is just pmd = (pmd_t*) pud */ pte =pte_offset(pmd, addr)and I think we should avoid quietly skipping the folded level, which could happen:
pud =
pud_offset(p4d, addr)
/* Each time, we go back topte_t*, so if we storedPUD_SHIFThere, it is impossible to know thatpudcame frompud_offsetand notpmd_offset. We must assume the deeper level so that we don't get stuck in a loop. */
pte =pte_offset(pud, addr)/* pud is cast from (pud_t* ->pte_t* ->pmd_t*) */Quietly dropping p*d_offset for folded levels is safe; it's just a cast that we're doing anyway. If you think this is fine, then I can remove
level. It might also be that this is a non-issue and that there will never be a folded level underneath a hugepage level.
Quoted 5 lines
We could also change
ptepto a union eventually (to clean up "hugetlb casts everything topte_t*" messiness), and having an explicitlevelas a tag for the union would be nice help. In the same way: I like havinglevelexplicitly so that we know for sure whereptepcame from.
Makes sense.
Quoted 3 lines
I can try to reduce the burden at the callsite while keeping
level:hpage_size_to_level()is really annoying to have everywhere.
Yeah this would be nice.
Per what I see most callers are having paired level/shift, so maybe we can make hugetlb_hgm_iter_init() to only take one of them and calculate the other. Then there can also be the __hugetlb_hgm_iter_init() which takes both, only used in the few places where necessary to have explicit level/shift. hugetlb_hgm_iter_init() calls __hugetlb_hgm_iter_init().
Thanks,
Signature · 1 line
Peter Xu
Re: [RFC PATCH v2 10/47] hugetlb: add hugetlb_pte to track HugeTLB page table entries
In reply to James HoughtonQuoted 29 lines · James Houghton
After high-granularity mapping, page table entries for HugeTLB pages can be of any size/type. (For example, we can have a 1G page mapped with a mix of PMDs and PTEs.) This struct is to help keep track of a HugeTLB PTE after we have done a page table walk.
Without this, we'd have to pass around the "size" of the PTE everywhere. We effectively did this before; it could be fetched from the hstate, which we pass around pretty much everywhere.
hugetlb_pte_present_leafis included here as a helper function that will be used frequently later on.Signed-off-by James Houghton <[email protected]>--- include/linux/hugetlb.h | 88 +++++++++++++++++++++++++++++++++++++++++ mm/hugetlb.c | 29 ++++++++++++++ 2 files changed, 117 insertions(+)include/linux/hugetlb.h +2
-0@@ -50,6 +50,75 @@ enum { __NR_USED_SUBPAGE, }; +enum hugetlb_level { +HUGETLB_LEVEL_PTE =1 ,
include/linux/hugetlb.h enum { · line 64
Quoted 10 lines
include/linux/hugetlb.h +10
-0+/* +* We always include PMD, PUD, and P4D in this enum definition so that, +* when logged as an integer, we can easily tell which level it is. +*/ +HUGETLB_LEVEL_PMD , +HUGETLB_LEVEL_PUD , +HUGETLB_LEVEL_P4D , + HUGETLB_LEVEL_PGD, +}; +
Don't we need to support CONTIG_PTE/PMD levels here for ARM64?
include/linux/hugetlb.h enum { · line 68
Quoted 4 lines
include/linux/hugetlb.h +4
-0+struct hugetlb_pte { +pte_t *ptep; +unsigned int shift; +enum hugetlb_level level;
Is shift + level redundant? When would those diverge?
Quoted 57 lines
include/linux/hugetlb.h +53
-0+spinlock_t *ptl; +}; + +static inline +void hugetlb_pte_populate (struct hugetlb_pte *hpte,pte_t *ptep, +unsigned int shift,enum hugetlb_level level) +{ +WARN_ON_ONCE (!ptep); + hpte->ptep = ptep; + hpte->shift = shift; + hpte->level = level; + hpte->ptl =NULL ; +} + +static inline +unsigned long hugetlb_pte_size (const struct hugetlb_pte *hpte) +{ +WARN_ON_ONCE (!hpte->ptep); +return 1UL << hpte->shift; +} + +static inline +unsigned long hugetlb_pte_mask (const struct hugetlb_pte *hpte) +{ +WARN_ON_ONCE (!hpte->ptep); +return ~(hugetlb_pte_size (hpte) -1 ); +} + +static inline +unsigned int hugetlb_pte_shift (const struct hugetlb_pte *hpte) +{ +WARN_ON_ONCE (!hpte->ptep); +return hpte->shift; +} + +static inline +enum hugetlb_level hugetlb_pte_level (const struct hugetlb_pte *hpte) +{ +WARN_ON_ONCE (!hpte->ptep); +return hpte->level; +} + +static inline +void hugetlb_pte_copy (struct hugetlb_pte *dest,const struct hugetlb_pte *src) +{ + dest->ptep = src->ptep; + dest->shift = src->shift; + dest->level = src->level; + dest->ptl = src->ptl; +} + +bool hugetlb_pte_present_leaf (const struct hugetlb_pte *hpte,pte_t pte); +struct hugepage_subpool {spinlock_t lock;long count;
include/linux/hugetlb.h static inline spinlock_t *huge_pte_lock(struct hstate *h, · line 1286
Quoted 8 lines
include/linux/hugetlb.h +5
-0@@ -1210,6 +1279,25 @@ static inline spinlock_t *huge_pte_lock(struct hstate *h,return ptl; } +static inline +spinlock_t *hugetlb_pte_lockptr (struct mm_struct *mm,struct hugetlb_pte *hpte) +{ + +BUG_ON (!hpte->ptep);
I think BUG_ON()s will be frowned upon. This function also doesn't really need ptep. Maybe let hugetlb_pte_shift() decide to BUG_ON() if necessary.
include/linux/hugetlb.h static inline spinlock_t *huge_pte_lock(struct hstate *h, · line 1289
Quoted 3 lines
include/linux/hugetlb.h +3
-0+if (hpte->ptl) +return hpte->ptl; +return huge_pte_lockptr (hugetlb_pte_shift (hpte), mm, hpte->ptep);
I don't know if this fallback to huge_pte_lockptr() should be obivous to the reader. If not, a comment would help.
Quoted 19 lines
include/linux/hugetlb.h +11
-0+} + +static inline +spinlock_t *hugetlb_pte_lock (struct mm_struct *mm,struct hugetlb_pte *hpte) +{ +spinlock_t *ptl =hugetlb_pte_lockptr (mm, hpte); + +spin_lock (ptl); +return ptl; +} +#if defined(CONFIG_HUGETLB_PAGE ) && defined(CONFIG_CMA )extern void __init hugetlb_cma_reserve (int order);#else mm/hugetlb.c +0
-0
mm/hugetlb.c static bool vma_has_reserves(struct vm_area_struct *vma, long chg) · line 1130
Quoted 4 lines
mm/hugetlb.c +1
-0@@ -1127,6 +1127,35 @@ static bool vma_has_reserves(struct vm_area_struct *vma, long chg)return false; } +bool hugetlb_pte_present_leaf (const struct hugetlb_pte *hpte,pte_t pte)
I also don't know if this is obvious to other readers, but I'm quite confused that we pass both hugetlb_pte and pte_t here, especially when hpte has a pte_t inside of it. Maybe a comment would help.
Quoted 34 lines
mm/hugetlb.c +28
-1+{ +pgd_t pgd; +p4d_t p4d; +pud_t pud; +pmd_t pmd; + +WARN_ON_ONCE (!hpte->ptep); +switch (hugetlb_pte_level (hpte)) { +case HUGETLB_LEVEL_PGD : + pgd =__pgd (pte_val (pte)); +return pgd_present (pgd) &&pgd_leaf (pgd); +case HUGETLB_LEVEL_P4D : + p4d =__p4d (pte_val (pte)); +return p4d_present (p4d) &&p4d_leaf (p4d); +case HUGETLB_LEVEL_PUD : + pud =__pud (pte_val (pte)); +return pud_present (pud) &&pud_leaf (pud); +case HUGETLB_LEVEL_PMD : + pmd =__pmd (pte_val (pte)); +return pmd_present (pmd) &&pmd_leaf (pmd); +case HUGETLB_LEVEL_PTE : +return pte_present (pte); +default : +WARN_ON_ONCE (1 ); +return false; + } +} +static void enqueue_huge_page (struct hstate *h,struct page *page) {int nid =page_to_nid (page); --2.38.0.135.g90850a2211-goog
Re: [RFC PATCH v2 10/47] hugetlb: add hugetlb_pte to track HugeTLB page table entries
Quoted 29 lines · Mina Almasry
Quoted 27 lines · James Houghton
After high-granularity mapping, page table entries for HugeTLB pages can be of any size/type. (For example, we can have a 1G page mapped with a mix of PMDs and PTEs.) This struct is to help keep track of a HugeTLB PTE after we have done a page table walk.
Without this, we'd have to pass around the "size" of the PTE everywhere. We effectively did this before; it could be fetched from the hstate, which we pass around pretty much everywhere.
hugetlb_pte_present_leafis included here as a helper function that will be used frequently later on.Signed-off-by James Houghton <[email protected]>--- include/linux/hugetlb.h | 88 +++++++++++++++++++++++++++++++++++++++++ mm/hugetlb.c | 29 ++++++++++++++ 2 files changed, 117 insertions(+)include/linux/hugetlb.h +0
-0@@ -50,6 +50,75 @@ enum { __NR_USED_SUBPAGE, };
Quoted 14 lines
include/linux/hugetlb.h enum { · line 64
Quoted 12 lines
include/linux/hugetlb.h +12
-0+enum hugetlb_level { +HUGETLB_LEVEL_PTE =1 , +/* +* We always include PMD, PUD, and P4D in this enum definition so that, +* when logged as an integer, we can easily tell which level it is. +*/ +HUGETLB_LEVEL_PMD , +HUGETLB_LEVEL_PUD , +HUGETLB_LEVEL_P4D , +HUGETLB_LEVEL_PGD , +}; +Don't we need to support CONTIG_PTE/PMD levels here for ARM64?
Yeah, which is why shift and level aren't quite the same thing. Contiguous PMDs would be HUGETLB_LEVEL_PMD but have shift = CONT_PMD_SHIFT, whereas regular PMDs would have shift = PMD_SHIFT.
Quoted 7 lines
include/linux/hugetlb.h enum { · line 68
Quoted 5 lines
include/linux/hugetlb.h +4
-0+struct hugetlb_pte { +pte_t *ptep; +unsigned int shift; +enum hugetlb_level level;Is shift + level redundant? When would those diverge?
Peter asked a very similar question. shift can be used to determine level if no levels are being folded. In the case of folded levels, you might have a single shift that corresponds to multiple "levels". That isn't necessarily a problem, as folding a level just means casting your p?d_t* differently, but I think it's good to be able to *know* if the hugetlb_pte was populated with a pud_t* that we treat it like a pud_t* always.
If ptep was instead a union, then level would be the tag. Perhaps it should be written that way.
Quoted 61 lines
Quoted 61 lines
include/linux/hugetlb.h +53
-0+spinlock_t *ptl; +}; + +static inline +void hugetlb_pte_populate (struct hugetlb_pte *hpte,pte_t *ptep, +unsigned int shift,enum hugetlb_level level) +{ +WARN_ON_ONCE (!ptep); + hpte->ptep = ptep; + hpte->shift = shift; + hpte->level = level; + hpte->ptl =NULL ; +} + +static inline +unsigned long hugetlb_pte_size (const struct hugetlb_pte *hpte) +{ +WARN_ON_ONCE (!hpte->ptep); +return 1UL << hpte->shift; +} + +static inline +unsigned long hugetlb_pte_mask (const struct hugetlb_pte *hpte) +{ +WARN_ON_ONCE (!hpte->ptep); +return ~(hugetlb_pte_size (hpte) -1 ); +} + +static inline +unsigned int hugetlb_pte_shift (const struct hugetlb_pte *hpte) +{ +WARN_ON_ONCE (!hpte->ptep); +return hpte->shift; +} + +static inline +enum hugetlb_level hugetlb_pte_level (const struct hugetlb_pte *hpte) +{ +WARN_ON_ONCE (!hpte->ptep); +return hpte->level; +} + +static inline +void hugetlb_pte_copy (struct hugetlb_pte *dest,const struct hugetlb_pte *src) +{ + dest->ptep = src->ptep; + dest->shift = src->shift; + dest->level = src->level; + dest->ptl = src->ptl; +} + +bool hugetlb_pte_present_leaf (const struct hugetlb_pte *hpte,pte_t pte); +struct hugepage_subpool {spinlock_t lock;long count;@@ -1210,6 +1279,25 @@ static inline spinlock_t *huge_pte_lock(struct hstate *h,return ptl; }
Quoted 9 lines
include/linux/hugetlb.h static inline spinlock_t *huge_pte_lock(struct hstate *h, · line 1286
Quoted 5 lines
include/linux/hugetlb.h +5
-0+static inline +spinlock_t *hugetlb_pte_lockptr (struct mm_struct *mm,struct hugetlb_pte *hpte) +{ + +BUG_ON (!hpte->ptep);I think BUG_ON()s will be frowned upon. This function also doesn't really need ptep. Maybe let
hugetlb_pte_shift()decide toBUG_ON()if necessary.
Right. I'll remove this (and others that aren't really necessary). Peter's suggestion to just let the kernel take a #pf and crash (thereby logging more info) SGTM.
Quoted 8 lines
include/linux/hugetlb.h static inline spinlock_t *huge_pte_lock(struct hstate *h, · line 1289
Quoted 5 lines
include/linux/hugetlb.h +3
-0+if (hpte->ptl) +return hpte->ptl; +return huge_pte_lockptr (hugetlb_pte_shift (hpte), mm, hpte->ptep);I don't know if this fallback to
huge_pte_lockptr()should be obivous to the reader. If not, a comment would help.
I'll clean this up a little for the next version. If something like this branch stays, I'll add a comment.
Quoted 23 lines
Quoted 23 lines
include/linux/hugetlb.h +11
-0+} + +static inline +spinlock_t *hugetlb_pte_lock (struct mm_struct *mm,struct hugetlb_pte *hpte) +{ +spinlock_t *ptl =hugetlb_pte_lockptr (mm, hpte); + +spin_lock (ptl); +return ptl; +} +#if defined(CONFIG_HUGETLB_PAGE ) && defined(CONFIG_CMA )extern void __init hugetlb_cma_reserve (int order);#else mm/hugetlb.c +0
-0@@ -1127,6 +1127,35 @@ static bool vma_has_reserves(struct vm_area_struct *vma, long chg)return false; }
Quoted 5 lines
mm/hugetlb.c static bool vma_has_reserves(struct vm_area_struct *vma, long chg) · line 1130
Quoted 1 line
mm/hugetlb.c +1
-0+bool hugetlb_pte_present_leaf (const struct hugetlb_pte *hpte,pte_t pte)I also don't know if this is obvious to other readers, but I'm quite confused that we pass both
hugetlb_pteandpte_there, especially when hpte has apte_tinside of it. Maybe a comment would help.
It's possible for the value of the pte to change if we haven't locked the PTL; we only store a pte_t* in hugetlb_pte, not the value itself.
Thinking about this... we *do* store shift which technically depends on the value of the PTE. If the PTE is pte_none, the true shift of the PTE is ambiguous, and so we just provide what the user asked for. That could lead to a scenario where UFFDIO_CONTINUE(some 4K page) then UFFDIO_CONTINUE(CONT_PTE_SIZE range around that page) can both succeed because we merely check if the *first* PTE in the contiguous bunch is none/has changed.
So, in the case of a contiguous PTE where we *think* we're overwriting a bunch of none PTEs, we need to check that each PTE we're overwriting is still none while holding the PTL. That means that the PTL we use for cont PTEs and non-cont PTEs of the same level must be the same.
So for the next version, I'll:
- add some requirement that contiguous and non-contiguous PTEs on the same level must use the same PTL
- think up some kind of API like all_contig_ptes_none(), but it only really applies for arm64, so I think actually putting it in can wait. I'll at least put a comment in hugetlb_mcopy_atomic_pte and hugetlb_no_page (near the final huge_pte_none() and pte_same() checks).
Quoted 35 lines
Quoted 35 lines
mm/hugetlb.c +28
-1+{ +pgd_t pgd; +p4d_t p4d; +pud_t pud; +pmd_t pmd; + +WARN_ON_ONCE (!hpte->ptep); +switch (hugetlb_pte_level (hpte)) { +case HUGETLB_LEVEL_PGD : + pgd =__pgd (pte_val (pte)); +return pgd_present (pgd) &&pgd_leaf (pgd); +case HUGETLB_LEVEL_P4D : + p4d =__p4d (pte_val (pte)); +return p4d_present (p4d) &&p4d_leaf (p4d); +case HUGETLB_LEVEL_PUD : + pud =__pud (pte_val (pte)); +return pud_present (pud) &&pud_leaf (pud); +case HUGETLB_LEVEL_PMD : + pmd =__pmd (pte_val (pte)); +return pmd_present (pmd) &&pmd_leaf (pmd); +case HUGETLB_LEVEL_PTE : +return pte_present (pte); +default : +WARN_ON_ONCE (1 ); +return false; + } +} +static void enqueue_huge_page (struct hstate *h,struct page *page) {int nid =page_to_nid (page); --2.38.0.135.g90850a2211-goog
Re: [RFC PATCH v2 10/47] hugetlb: add hugetlb_pte to track HugeTLB page table entries
Quoted 11 lines · James Houghton
Quoted 7 lines · Mina Almasry
Quoted 2 lines · James Houghton
+bool
hugetlb_pte_present_leaf(const struct hugetlb_pte *hpte, pte_t pte)I also don't know if this is obvious to other readers, but I'm quite confused that we pass both
hugetlb_pteandpte_there, especially when hpte has apte_tinside of it. Maybe a comment would help.It's possible for the value of the pte to change if we haven't locked the PTL; we only store a pte_t* in
hugetlb_pte, not the value itself.
I had comments similar to Mina and Peter on other parts of this patch. Calling this without some type of locking is 'interesting'. I have not yet looked at callers (without locking), but I assume such callers can handle stale results.
Quoted 7 lines
Thinking about this... we *do* store
shiftwhich technically depends on the value of the PTE. If the PTE ispte_none, the trueshiftof the PTE is ambiguous, and so we just provide what the user asked for. That could lead to a scenario whereUFFDIO_CONTINUE(some 4K page)thenUFFDIO_CONTINUE(CONT_PTE_SIZE range around that page)can both succeed because we merely check if the *first* PTE in the contiguous bunch is none/has changed.
Right, Yuck!
Quoted 15 lines
So, in the case of a contiguous PTE where we *think* we're overwriting a bunch of none PTEs, we need to check that each PTE we're overwriting is still none while holding the PTL. That means that the PTL we use for cont PTEs and non-cont PTEs of the same level must be the same.
So for the next version, I'll:
- add some requirement that contiguous and non-contiguous PTEs on the same level must use the same PTL
- think up some kind of API likeall_contig_ptes_none(), but it only really applies for arm64, so I think actually putting it in can wait. I'll at least put a comment inhugetlb_mcopy_atomic_pteandhugetlb_no_page(near the finalhuge_pte_none()andpte_same()checks).
Signature · 1 line
Mike Kravetz
RFC PATCH v2 11/47 hugetlb: add hugetlb_pmd_alloc and hugetlb_pte_alloc
In reply to James HoughtonThese functions are used to allocate new PTEs below the hstate PTE. This will be used by hugetlb_walk_step, which implements stepping forwards in a HugeTLB high-granularity page table walk.
The reasons that we don't use the standard pmd_alloc/pte_alloc* functions are:
1) This prevents us from accidentally overwriting swap entries or
attempting to use swap entries as present non-leaf PTEs (see
pmd_alloc(); we assume that !pte_none means pte_present and
non-leaf). 2) Locking hugetlb PTEs can different than regular PTEs. (Although, as implemented right now, locking is the same.)
3) We can maintain compatibility with CONFIG_HIGHPTE. That is, HugeTLB HGM won't use HIGHPTE, but the kernel can still be built with it, and other mm code will use it.
When GENERAL_HUGETLB supports P4D-based hugepages, we will need to implement hugetlb_pud_alloc to implement hugetlb_walk_step.
--- include/linux/hugetlb.h | 5 +++ mm/hugetlb.c | 94 +++++++++++++++++++++++++++++++++++++++++ 2 files changed, 99 insertions(+)
include/linux/hugetlb.h +5 -0
@@ -119,6 +119,11 @@ void hugetlb_pte_copy(struct hugetlb_pte *dest, const struct hugetlb_pte *src)bool hugetlb_pte_present_leaf (const struct hugetlb_pte *hpte,pte_t pte); +pmd_t *hugetlb_pmd_alloc (struct mm_struct *mm,struct hugetlb_pte *hpte, +unsigned long addr); +pte_t *hugetlb_pte_alloc (struct mm_struct *mm,struct hugetlb_pte *hpte, +unsigned long addr); +struct hugepage_subpool {spinlock_t lock;long count;
mm/hugetlb.c +94 -0
100 lines not on this page · raw message
Signature · 1 line
2.38.0.135.g90850a2211-goog
Re: [RFC PATCH v2 11/47] hugetlb: add hugetlb_pmd_alloc and hugetlb_pte_alloc
Quoted 46 lines · James Houghton
These functions are used to allocate new PTEs below the hstate PTE. This will be used by
hugetlb_walk_step, which implements stepping forwards in a HugeTLB high-granularity page table walk.The reasons that we don't use the standard pmd_alloc/pte_alloc* functions are:
1) This prevents us from accidentally overwriting swap entries orattempting to use swap entries as present non-leaf PTEs (see pmd_alloc(); we assume that !pte_none means pte_present and non-leaf).2) Locking hugetlb PTEs can different than regular PTEs. (Although, as implemented right now, locking is the same.)
3) We can maintain compatibility withCONFIG_HIGHPTE. That is, HugeTLB HGM won't use HIGHPTE, but the kernel can still be built with it, and other mm code will use it.When
GENERAL_HUGETLBsupports P4D-based hugepages, we will need to implementhugetlb_pud_allocto implementhugetlb_walk_step.Signed-off-by James Houghton <[email protected]>--- include/linux/hugetlb.h | 5 +++ mm/hugetlb.c | 94 +++++++++++++++++++++++++++++++++++++++++ 2 files changed, 99 insertions(+)include/linux/hugetlb.h +5
-0@@ -119,6 +119,11 @@ void hugetlb_pte_copy(struct hugetlb_pte *dest, const struct hugetlb_pte *src)bool hugetlb_pte_present_leaf (const struct hugetlb_pte *hpte,pte_t pte); +pmd_t *hugetlb_pmd_alloc (struct mm_struct *mm,struct hugetlb_pte *hpte, +unsigned long addr); +pte_t *hugetlb_pte_alloc (struct mm_struct *mm,struct hugetlb_pte *hpte, +unsigned long addr); +struct hugepage_subpool {spinlock_t lock;long count;mm/hugetlb.c +0
-0
mm/hugetlb.c static bool has_same_uncharge_info(struct file_region *rg, · line 345
Quoted 5 lines
mm/hugetlb.c +2
-0@@ -341,6 +341,100 @@ static bool has_same_uncharge_info(struct file_region *rg,#endif } +pmd_t *hugetlb_pmd_alloc (struct mm_struct *mm,struct hugetlb_pte *hpte, +unsigned long addr)
A little confused as there are no users yet ... Is hpte the 'hstate PTE' that we are trying to allocate ptes under? For example, in the case of a hugetlb_pmd_alloc caller hpte would be a PUD or CONT_PMD size pte?
mm/hugetlb.c static bool has_same_uncharge_info(struct file_region *rg, · line 353
Quoted 8 lines
mm/hugetlb.c +8
-0+{ +spinlock_t *ptl =hugetlb_pte_lockptr (mm, hpte); +pmd_t *new; +pud_t *pudp; +pud_t pud; + +if (hpte->level !=HUGETLB_LEVEL_PUD ) +return ERR_PTR (-EINVAL );
Ah yes, it is PUD level. However, I guess CONT_PMD would also be valid on arm64?
mm/hugetlb.c static bool has_same_uncharge_info(struct file_region *rg, · line 357
Quoted 4 lines
mm/hugetlb.c +4
-0+ + pudp = (pud_t *)hpte->ptep; +retry: + pud = *pudp;
We might want to consider a READ_ONCE here. I am not an expert on such things, but recall a similar as pointed out in the now obsolete commit 27ceae9833843.
Signature · 1 line
Mike Kravetz
Quoted 86 lines
mm/hugetlb.c +80
-0+if (likely (pud_present (pud))) +return unlikely (pud_leaf (pud)) + ?ERR_PTR (-EEXIST ) + :pmd_offset (pudp, addr); +else if (!huge_pte_none (huge_ptep_get (hpte->ptep))) +/* +* Not present and not none means that a swap entry lives here, +* and we can't get rid of it. +*/ +return ERR_PTR (-EEXIST ); + + new =pmd_alloc_one (mm, addr); +if (!new) +return ERR_PTR (-ENOMEM ); + +spin_lock (ptl); +if (!pud_same (pud, *pudp)) { +spin_unlock (ptl); +pmd_free (mm, new); + goto retry; + } + +mm_inc_nr_pmds (mm); +smp_wmb ();/* See comment in pmd_install() */ +pud_populate (mm, pudp, new); +spin_unlock (ptl); +return pmd_offset (pudp, addr); +} + +pte_t *hugetlb_pte_alloc (struct mm_struct *mm,struct hugetlb_pte *hpte, +unsigned long addr) +{ +spinlock_t *ptl =hugetlb_pte_lockptr (mm, hpte); +pgtable_t new; +pmd_t *pmdp; +pmd_t pmd; + +if (hpte->level !=HUGETLB_LEVEL_PMD ) +return ERR_PTR (-EINVAL ); + + pmdp = (pmd_t *)hpte->ptep; +retry: + pmd = *pmdp; +if (likely (pmd_present (pmd))) +return unlikely (pmd_leaf (pmd)) + ?ERR_PTR (-EEXIST ) + :pte_offset_kernel (pmdp, addr); +else if (!huge_pte_none (huge_ptep_get (hpte->ptep))) +/* +* Not present and not none means that a swap entry lives here, +* and we can't get rid of it. +*/ +return ERR_PTR (-EEXIST ); + +/* +* With CONFIG_HIGHPTE, calling `pte_alloc_one` directly may result +* in page tables being allocated in high memory, needing a kmap to +* access. Instead, we call __pte_alloc_one directly with +* GFP_PGTABLE_USER to prevent these PTEs being allocated in high +* memory. +*/ + new =__pte_alloc_one (mm,GFP_PGTABLE_USER ); +if (!new) +return ERR_PTR (-ENOMEM ); + +spin_lock (ptl); +if (!pmd_same (pmd, *pmdp)) { +spin_unlock (ptl); +pgtable_pte_page_dtor (new); +__free_page (new); + goto retry; + } + +mm_inc_nr_ptes (mm); +smp_wmb ();/* See comment in pmd_install() */ +pmd_populate (mm, pmdp, new); +spin_unlock (ptl); +return pte_offset_kernel (pmdp, addr); +} +static void coalesce_file_region (struct resv_map *resv,struct file_region *rg) {struct file_region *nrg, *prg;Signature · 1 line
2.38.0.135.g90850a2211-goog
Re: [RFC PATCH v2 11/47] hugetlb: add hugetlb_pmd_alloc and hugetlb_pte_alloc
Quoted 51 lines · Mike Kravetz
Quoted 49 lines · James Houghton
These functions are used to allocate new PTEs below the hstate PTE. This will be used by
hugetlb_walk_step, which implements stepping forwards in a HugeTLB high-granularity page table walk.The reasons that we don't use the standard pmd_alloc/pte_alloc* functions are:
1) This prevents us from accidentally overwriting swap entries orattempting to use swap entries as present non-leaf PTEs (see pmd_alloc(); we assume that !pte_none means pte_present and non-leaf).2) Locking hugetlb PTEs can different than regular PTEs. (Although, as implemented right now, locking is the same.)
3) We can maintain compatibility withCONFIG_HIGHPTE. That is, HugeTLB HGM won't use HIGHPTE, but the kernel can still be built with it, and other mm code will use it.When
GENERAL_HUGETLBsupports P4D-based hugepages, we will need to implementhugetlb_pud_allocto implementhugetlb_walk_step.Signed-off-by James Houghton <[email protected]>--- include/linux/hugetlb.h | 5 +++ mm/hugetlb.c | 94 +++++++++++++++++++++++++++++++++++++++++ 2 files changed, 99 insertions(+)include/linux/hugetlb.h +5
-0@@ -119,6 +119,11 @@ void hugetlb_pte_copy(struct hugetlb_pte *dest, const struct hugetlb_pte *src)bool hugetlb_pte_present_leaf (const struct hugetlb_pte *hpte,pte_t pte); +pmd_t *hugetlb_pmd_alloc (struct mm_struct *mm,struct hugetlb_pte *hpte, +unsigned long addr); +pte_t *hugetlb_pte_alloc (struct mm_struct *mm,struct hugetlb_pte *hpte, +unsigned long addr); +struct hugepage_subpool {spinlock_t lock;long count;mm/hugetlb.c +0
-0@@ -341,6 +341,100 @@ static bool has_same_uncharge_info(struct file_region *rg,#endif }
Quoted 6 lines
mm/hugetlb.c static bool has_same_uncharge_info(struct file_region *rg, · line 345
Quoted 2 lines
mm/hugetlb.c +2
-0+pmd_t *hugetlb_pmd_alloc (struct mm_struct *mm,struct hugetlb_pte *hpte, +unsigned long addr)A little confused as there are no users yet ... Is hpte the 'hstate PTE' that we are trying to allocate ptes under? For example, in the case of a
hugetlb_pmd_alloccaller hpte would be a PUD orCONT_PMDsize pte?
The hpte is the level above the level we're trying to allocate (not necessarily the 'hstate PTE'). I'll make that clear in the comments for both functions.
So consider allocating 4K PTEs for a 1G HugeTLB page:
- With the hstate 'PTE' (PUD), we make a hugetlb_pte with that PUD (let's call it 'hpte')
- We call hugetlb_pmd_alloc(hpte) which will leave 'hpte' the same, but the pud_t that hpte->ptep points to is no longer a leaf.
- We call hugetlb_walk_step(hpte) to step down a level to get a PMD, changing hpte. The hpte->ptep is now pointing to a blank pmd_t.
- We call hugetlb_pte_alloc(hpte) to allocate a bunch of PTEs and populate the pmd_t.
- We call hugetlb_walk_step(hpte) to step down again.
This is basically what hugetlb_hgm_walk does (in the next patch). We only change 'hpte' when we do a step, and that is when we populate 'shift'. The 'sz' parameter for hugetlb_walk_step is what architectures can use to populate hpte->shift appropriately (ignored for x86).
For arm64, we can use 'sz' to populate hpte->shift with what the caller wants when we are free to choose (like if all the PTEs are none, we can do CONT_PTE_SHIFT). See [1]'s implementation of hugetlb_walk_step for what I *think* is correct for arm64.
[1] https://github.com/48ca/linux/commit/bf3b8742e95c58c2431c80c5bed5cb5cb95885af
Quoted 12 lines
mm/hugetlb.c static bool has_same_uncharge_info(struct file_region *rg, · line 353
Quoted 9 lines
mm/hugetlb.c +8
-0+{ +spinlock_t *ptl =hugetlb_pte_lockptr (mm, hpte); +pmd_t *new; +pud_t *pudp; +pud_t pud; + +if (hpte->level !=HUGETLB_LEVEL_PUD ) +return ERR_PTR (-EINVAL );Ah yes, it is PUD level. However, I guess
CONT_PMDwould also be valid on arm64?
The level is always PGD, P4D, PUD, PMD, or PTE. CONT_PTE is on HUGETLB_LEVEL_PTE, CONT_PMD is on HUGETLB_LEVEL_PMD.
These functions are supposed to be used for all architectures (in their implementations of 'hugetlb_walk_step'; that's why they're not static, actually. I'll make that clear in the commit description).
Quoted 9 lines
mm/hugetlb.c static bool has_same_uncharge_info(struct file_region *rg, · line 357
Quoted 5 lines
mm/hugetlb.c +4
-0+ + pudp = (pud_t *)hpte->ptep; +retry: + pud = *pudp;We might want to consider a
READ_ONCEhere. I am not an expert on such things, but recall a similar as pointed out in the now obsolete commit27ceae9833843.
Agreed. Will try to change all PTE reading to use READ_ONCE, though they can be easy to miss... :(
Thanks very much for the reviews so far, Mike!
- James
Quoted 4 lines
--
Mike Kravetz
Re: [RFC PATCH v2 11/47] hugetlb: add hugetlb_pmd_alloc and hugetlb_pte_alloc
Quoted 64 lines · James Houghton
Quoted 51 lines · Mike Kravetz
Quoted 49 lines · James Houghton
These functions are used to allocate new PTEs below the hstate PTE. This will be used by
hugetlb_walk_step, which implements stepping forwards in a HugeTLB high-granularity page table walk.The reasons that we don't use the standard pmd_alloc/pte_alloc* functions are:
1) This prevents us from accidentally overwriting swap entries orattempting to use swap entries as present non-leaf PTEs (see pmd_alloc(); we assume that !pte_none means pte_present and non-leaf).2) Locking hugetlb PTEs can different than regular PTEs. (Although, as implemented right now, locking is the same.)
3) We can maintain compatibility withCONFIG_HIGHPTE. That is, HugeTLB HGM won't use HIGHPTE, but the kernel can still be built with it, and other mm code will use it.When
GENERAL_HUGETLBsupports P4D-based hugepages, we will need to implementhugetlb_pud_allocto implementhugetlb_walk_step.Signed-off-by James Houghton <[email protected]>--- include/linux/hugetlb.h | 5 +++ mm/hugetlb.c | 94 +++++++++++++++++++++++++++++++++++++++++ 2 files changed, 99 insertions(+)include/linux/hugetlb.h +5
-0@@ -119,6 +119,11 @@ void hugetlb_pte_copy(struct hugetlb_pte *dest, const struct hugetlb_pte *src)bool hugetlb_pte_present_leaf (const struct hugetlb_pte *hpte,pte_t pte); +pmd_t *hugetlb_pmd_alloc (struct mm_struct *mm,struct hugetlb_pte *hpte, +unsigned long addr); +pte_t *hugetlb_pte_alloc (struct mm_struct *mm,struct hugetlb_pte *hpte, +unsigned long addr); +struct hugepage_subpool {spinlock_t lock;long count;mm/hugetlb.c +0
-0@@ -341,6 +341,100 @@ static bool has_same_uncharge_info(struct file_region *rg,#endif }Quoted 6 lines
mm/hugetlb.c static bool has_same_uncharge_info(struct file_region *rg, · line 345
Quoted 2 lines
mm/hugetlb.c +2
-0+pmd_t *hugetlb_pmd_alloc (struct mm_struct *mm,struct hugetlb_pte *hpte, +unsigned long addr)A little confused as there are no users yet ... Is hpte the 'hstate PTE' that we are trying to allocate ptes under? For example, in the case of a
hugetlb_pmd_alloccaller hpte would be a PUD orCONT_PMDsize pte?The hpte is the level above the level we're trying to allocate (not necessarily the 'hstate PTE'). I'll make that clear in the comments for both functions.
Quoted 10 lines
So consider allocating 4K PTEs for a 1G HugeTLB page:
- With the hstate 'PTE' (PUD), we make ahugetlb_ptewith that PUD (let's call it 'hpte')
- We callhugetlb_pmd_alloc(hpte)which will leave 'hpte' the same, but thepud_tthathpte->pteppoints to is no longer a leaf.
- We callhugetlb_walk_step(hpte)to step down a level to get a PMD, changing hpte. Thehpte->ptepis now pointing to a blankpmd_t.
- We callhugetlb_pte_alloc(hpte)to allocate a bunch of PTEs and populate thepmd_t.
- We callhugetlb_walk_step(hpte)to step down again.
Erm actually this isn't entirely accurate. The general flow is about right, but hugetlb_pmd_alloc/hugetlb_pte_alloc are actually part of hugetlb_walk_step. (See hugetlb_hgm_walk for the ground truth :P)
- James
Quoted 55 lines
This is basically what
hugetlb_hgm_walkdoes (in the next patch). We only change 'hpte' when we do a step, and that is when we populate 'shift'. The 'sz' parameter forhugetlb_walk_stepis what architectures can use to populatehpte->shiftappropriately (ignored for x86).For arm64, we can use 'sz' to populate
hpte->shiftwith what the caller wants when we are free to choose (like if all the PTEs are none, we can doCONT_PTE_SHIFT). See [1]'s implementation ofhugetlb_walk_stepfor what I *think* is correct for arm64.[1] https://github.com/48ca/linux/commit/bf3b8742e95c58c2431c80c5bed5cb5cb95885af
Quoted 12 lines
mm/hugetlb.c static bool has_same_uncharge_info(struct file_region *rg, · line 353
Quoted 9 lines
mm/hugetlb.c +8
-0+{ +spinlock_t *ptl =hugetlb_pte_lockptr (mm, hpte); +pmd_t *new; +pud_t *pudp; +pud_t pud; + +if (hpte->level !=HUGETLB_LEVEL_PUD ) +return ERR_PTR (-EINVAL );Ah yes, it is PUD level. However, I guess
CONT_PMDwould also be valid on arm64?The level is always PGD, P4D, PUD, PMD, or PTE.
CONT_PTEis onHUGETLB_LEVEL_PTE,CONT_PMDis onHUGETLB_LEVEL_PMD.These functions are supposed to be used for all architectures (in their implementations of '
hugetlb_walk_step'; that's why they're not static, actually. I'll make that clear in the commit description).Quoted 9 lines
mm/hugetlb.c static bool has_same_uncharge_info(struct file_region *rg, · line 357
Quoted 5 lines
mm/hugetlb.c +4
-0+ + pudp = (pud_t *)hpte->ptep; +retry: + pud = *pudp;We might want to consider a
READ_ONCEhere. I am not an expert on such things, but recall a similar as pointed out in the now obsolete commit27ceae9833843.Agreed. Will try to change all PTE reading to use
READ_ONCE, though they can be easy to miss... :(Thanks very much for the reviews so far, Mike!
- James
Quoted 4 lines
--
Mike Kravetz
RFC PATCH v2 12/47 hugetlb: add hugetlb_hgm_walk and hugetlb_walk_step
In reply to James Houghtonhugetlb_hgm_walk implements high-granularity page table walks for HugeTLB. It is safe to call on non-HGM enabled VMAs; it will return immediately.
hugetlb_walk_step implements how we step forwards in the walk. For architectures that don't use GENERAL_HUGETLB, they will need to provide their own implementation.
--- include/linux/hugetlb.h | 13 +++++ mm/hugetlb.c | 125 ++++++++++++++++++++++++++++++++++++++++ 2 files changed, 138 insertions(+)
include/linux/hugetlb.h +13 -0
@@ -276,6 +276,10 @@ u32 hugetlb_fault_mutex_hash(struct address_space *mapping, pgoff_t idx);pte_t *huge_pmd_share (struct mm_struct *mm,struct vm_area_struct *vma,unsigned long addr,pud_t *pud); +int hugetlb_hgm_walk (struct mm_struct *mm,struct vm_area_struct *vma, +struct hugetlb_pte *hpte,unsigned long addr, +unsigned long sz,bool stop_at_none); +struct address_space *hugetlb_page_mapping_lock_write (struct page *hpage);extern int sysctl_hugetlb_shm_group;
@@ -288,6 +292,8 @@ pte_t *huge_pte_alloc(struct mm_struct *mm, struct vm_area_struct *vma,pte_t *huge_pte_offset (struct mm_struct *mm,unsigned long addr,unsigned long sz);unsigned long hugetlb_mask_last_page (struct hstate *h); +int hugetlb_walk_step (struct mm_struct *mm,struct hugetlb_pte *hpte, +unsigned long addr,unsigned long sz);int huge_pmd_unshare (struct mm_struct *mm,struct vm_area_struct *vma,unsigned long addr,pte_t *ptep); void adjust_range_if_pmd_sharing_possible(struct vm_area_struct *vma,
@@ -1066,6 +1072,8 @@ void hugetlb_register_node(struct node *node);void hugetlb_unregister_node (struct node *node);#endif +enum hugetlb_level hpage_size_to_level (unsigned long sz); +#else /* CONFIG_HUGETLB_PAGE */ struct hstate {};
@@ -1253,6 +1261,11 @@ static inline void hugetlb_register_node(struct node *node)static inline void hugetlb_unregister_node (struct node *node) { } + +static inline enum hugetlb_level hpage_size_to_level (unsigned long sz) +{ +return HUGETLB_LEVEL_PTE ; +}#endif /* CONFIG_HUGETLB_PAGE */ #ifdef CONFIG_HUGETLB_HIGH_GRANULARITY_MAPPING
mm/hugetlb.c +125 -0
143 lines not on this page · raw message
Signature · 1 line
2.38.0.135.g90850a2211-goog
Peter Xu
reviewerRe: [RFC PATCH v2 12/47] hugetlb: add hugetlb_hgm_walk and hugetlb_walk_step
Quoted 9 lines · James Houghton
+/*
hugetlb_hgm_walk- walks a high-granularity HugeTLB page table to resolve
+ * the page table entry for @addr.
+ *
+ * @hpte must always be pointing at an hstate-level PTE (or deeper).
+ *
+ * This function will never walk further if it encounters a PTE of a size
+ * less than or equal to @sz.
+ *
+ * @stop_at_none determines what we do when we encounter an empty PTE.
IIUC it is not about empty PTE but swap-or-empty pte?
I'm not sure whether it'll be more straightforward to have "bool alloc" just to show whether the caller would like to allocate pgtables when walking the sub-level pgtable until the level specified.
In final version of the code I also think we should drop all the "/* stop_at_pte */" comments in the callers. Maybe that already means the meaning of the bool is confusing so we always need a hint.
Signature · 1 line
Peter Xu
Re: [RFC PATCH v2 12/47] hugetlb: add hugetlb_hgm_walk and hugetlb_walk_step
Quoted 12 lines · Peter Xu
Quoted 9 lines · James Houghton
+/*
hugetlb_hgm_walk- walks a high-granularity HugeTLB page table to resolve
+ * the page table entry for @addr.
+ *
+ * @hpte must always be pointing at an hstate-level PTE (or deeper).
+ *
+ * This function will never walk further if it encounters a PTE of a size
+ * less than or equal to @sz.
+ *
+ * @stop_at_none determines what we do when we encounter an empty PTE.
Quoted 5 lines
IIUC it is not about empty PTE but swap-or-empty pte?
I'm not sure whether it'll be more straightforward to have "bool alloc" just to show whether the caller would like to allocate pgtables when walking the sub-level pgtable until the level specified.
I think "bool alloc" is cleaner. I'll do that. Thanks for the suggestion.
Quoted 4 lines
In final version of the code I also think we should drop all the "/*
stop_at_pte*/" comments in the callers. Maybe that already means the meaning of the bool is confusing so we always need a hint.
I did that to hopefully make things easier to read. I'll remove it.
- James
Quoted 4 lines
--
Peter Xu
Re: [RFC PATCH v2 12/47] hugetlb: add hugetlb_hgm_walk and hugetlb_walk_step
In reply to James HoughtonQuoted 99 lines · James Houghton
hugetlb_hgm_walkimplements high-granularity page table walks for HugeTLB. It is safe to call on non-HGM enabled VMAs; it will return immediately.
hugetlb_walk_stepimplements how we step forwards in the walk. For architectures that don't useGENERAL_HUGETLB, they will need to provide their own implementation.Signed-off-by James Houghton <[email protected]>--- include/linux/hugetlb.h | 13 +++++ mm/hugetlb.c | 125 ++++++++++++++++++++++++++++++++++++++++ 2 files changed, 138 insertions(+)include/linux/hugetlb.h +13
-0@@ -276,6 +276,10 @@ u32 hugetlb_fault_mutex_hash(struct address_space *mapping, pgoff_t idx);pte_t *huge_pmd_share (struct mm_struct *mm,struct vm_area_struct *vma,unsigned long addr,pud_t *pud); +int hugetlb_hgm_walk (struct mm_struct *mm,struct vm_area_struct *vma, +struct hugetlb_pte *hpte,unsigned long addr, +unsigned long sz,bool stop_at_none); +struct address_space *hugetlb_page_mapping_lock_write (struct page *hpage);extern int sysctl_hugetlb_shm_group;@@ -288,6 +292,8 @@ pte_t *huge_pte_alloc(struct mm_struct *mm, struct vm_area_struct *vma,pte_t *huge_pte_offset (struct mm_struct *mm,unsigned long addr,unsigned long sz);unsigned long hugetlb_mask_last_page (struct hstate *h); +int hugetlb_walk_step (struct mm_struct *mm,struct hugetlb_pte *hpte, +unsigned long addr,unsigned long sz);int huge_pmd_unshare (struct mm_struct *mm,struct vm_area_struct *vma,unsigned long addr,pte_t *ptep); void adjust_range_if_pmd_sharing_possible(struct vm_area_struct *vma,@@ -1066,6 +1072,8 @@ void hugetlb_register_node(struct node *node);void hugetlb_unregister_node (struct node *node);#endif +enum hugetlb_level hpage_size_to_level (unsigned long sz); +#else /* CONFIG_HUGETLB_PAGE */ struct hstate {};@@ -1253,6 +1261,11 @@ static inline void hugetlb_register_node(struct node *node)static inline void hugetlb_unregister_node (struct node *node) { } + +static inline enum hugetlb_level hpage_size_to_level (unsigned long sz) +{ +return HUGETLB_LEVEL_PTE ; +}#endif /* CONFIG_HUGETLB_PAGE */ #ifdef CONFIG_HUGETLB_HIGH_GRANULARITY_MAPPING mm/hugetlb.c +25
-0@@ -95,6 +95,29 @@ static void hugetlb_vma_data_free(struct vm_area_struct *vma);static int hugetlb_vma_data_alloc (struct vm_area_struct *vma);static void __hugetlb_vma_unlock_write_free (struct vm_area_struct *vma); +/* +* hpage_size_to_level() - convert @sz to the corresponding page table level +* +* @sz must be less than or equal to a valid hugepage size. +*/ +enum hugetlb_level hpage_size_to_level (unsigned long sz) +{ +/* +* We order the conditionals from smallest to largest to pick the +* smallest level when multiple levels have the same size (i.e., +* when levels are folded). +*/ +if (sz <PMD_SIZE ) +return HUGETLB_LEVEL_PTE ; +if (sz <PUD_SIZE ) +return HUGETLB_LEVEL_PMD ; +if (sz <P4D_SIZE ) +return HUGETLB_LEVEL_PUD ; +if (sz <PGDIR_SIZE ) +return HUGETLB_LEVEL_P4D ; +return HUGETLB_LEVEL_PGD ; +} +static inline bool subpool_is_free (struct hugepage_subpool *spool) {if (spool->count)@@ -7321,6 +7344,70 @@ bool want_pmd_share(struct vm_area_struct *vma, unsigned long addr) }#endif /* CONFIG_ARCH_WANT_HUGE_PMD_SHARE */ +/* hugetlb_hgm_walk - walks a high-granularity HugeTLB page table to resolve +* the page table entry for @addr.
mm/hugetlb.c bool want_pmd_share(struct vm_area_struct *vma, unsigned long addr) · line 7358
Quoted 10 lines
mm/hugetlb.c +10
-0+* +* @hpte must always be pointing at an hstate-level PTE (or deeper). +* +* This function will never walk further if it encounters a PTE of a size +* less than or equal to @sz. +* +* @stop_at_none determines what we do when we encounter an empty PTE. If true, +* we return that PTE. If false and @sz is less than the current PTE's size, +* we make that PTE point to the next level down, going until @sz is the same +* as our current PTE.
I was a bit confused about 'we return that PTE' when the function is of type int. TBH, I am not a fan of the current scheme of passing in *hpte and having the hpte modified by the function.
Quoted 9 lines
mm/hugetlb.c +9
-0+* +* If @stop_at_none is true and @sz is PAGE_SIZE, this function will always +* succeed, but that does not guarantee that hugetlb_pte_size(hpte) is @sz. +* +* Return: +* -ENOMEM if we couldn't allocate new PTEs. +* -EEXIST if the caller wanted to walk further than a migration PTE, +* poison PTE, or a PTE marker. The caller needs to manually deal +* with this scenario.
mm/hugetlb.c bool want_pmd_share(struct vm_area_struct *vma, unsigned long addr) · line 7377
Quoted 10 lines
mm/hugetlb.c +10
-0+* -EINVAL if called with invalid arguments (@sz invalid, @hpte not +* initialized). +* 0 otherwise. +* +* Even if this function fails, @hpte is guaranteed to always remain +* valid. +*/ +int hugetlb_hgm_walk (struct mm_struct *mm,struct vm_area_struct *vma, +struct hugetlb_pte *hpte,unsigned long addr, +unsigned long sz,bool stop_at_none)
Since we are potentially populating lower level page tables, we may want a different function name. It may just be me, but I think of walk as a read only operation. I would suggest putting populate in the name, but as Peter pointed out elsewhere, that has other implications. Sorry, I can not think of something better right now.
Signature · 1 line
Mike Kravetz
Quoted 84 lines
mm/hugetlb.c +71
-0+{ +int ret =0 ; +pte_t pte; + +if (WARN_ON_ONCE (sz <PAGE_SIZE )) +return -EINVAL ; + +if (!hugetlb_hgm_enabled (vma)) { +if (stop_at_none) +return 0 ; +return sz ==huge_page_size (hstate_vma (vma)) ?0 : -EINVAL ; + } + +hugetlb_vma_assert_locked (vma); + +if (WARN_ON_ONCE (!hpte->ptep)) +return -EINVAL ; + +while (hugetlb_pte_size (hpte) > sz && !ret) { + pte =huge_ptep_get (hpte->ptep); +if (!pte_present (pte)) { +if (stop_at_none) +return 0 ; +if (unlikely (!huge_pte_none (pte))) +return -EEXIST ; + }else if (hugetlb_pte_present_leaf (hpte, pte)) +return 0 ; + ret =hugetlb_walk_step (mm, hpte, addr, sz); + } + +return ret; +} +#ifdef CONFIG_ARCH_WANT_GENERAL_HUGETLB pte_t *huge_pte_alloc (struct mm_struct *mm,struct vm_area_struct *vma,unsigned long addr,unsigned long sz)@@ -7388,6 +7475,44 @@ pte_t *huge_pte_offset(struct mm_struct *mm,return (pte_t *)pmd; } +/* +* hugetlb_walk_step() - Walk the page table one step to resolve the page +* (hugepage or subpage) entry at address @addr. +* +* @sz always points at the final target PTE size (e.g. PAGE_SIZE for the +* lowest level PTE). +* +* @hpte will always remain valid, even if this function fails. +*/ +int hugetlb_walk_step (struct mm_struct *mm,struct hugetlb_pte *hpte, +unsigned long addr,unsigned long sz) +{ +pte_t *ptep; +spinlock_t *ptl; + +switch (hpte->level) { +case HUGETLB_LEVEL_PUD : + ptep = (pte_t *)hugetlb_pmd_alloc (mm, hpte, addr); +if (IS_ERR (ptep)) +return PTR_ERR (ptep); +hugetlb_pte_populate (hpte, ptep,PMD_SHIFT ,HUGETLB_LEVEL_PMD ); +break ; +case HUGETLB_LEVEL_PMD : + ptep =hugetlb_pte_alloc (mm, hpte, addr); +if (IS_ERR (ptep)) +return PTR_ERR (ptep); + ptl =pte_lockptr (mm, (pmd_t *)hpte->ptep); +hugetlb_pte_populate (hpte, ptep,PAGE_SHIFT ,HUGETLB_LEVEL_PTE ); + hpte->ptl = ptl; +break ; +default : +WARN_ONCE (1 ,"%s: got invalid level: %d (shift: %d)\n" , + __func__, hpte->level, hpte->shift); +return -EINVAL ; + } +return 0 ; +} +/* * Return a mask that can be used to update an address to the last huge * page in a page table page mapping size. Used to skip non-present Signature · 1 line
2.38.0.135.g90850a2211-goog
Re: [RFC PATCH v2 12/47] hugetlb: add hugetlb_hgm_walk and hugetlb_walk_step
In reply to James HoughtonQuoted 4 lines
+ * @stop_at_none determines what we do when we encounter an empty PTE. If true,
+ * we return that PTE. If false and @sz is less than the current PTE's size,
+ * we make that PTE point to the next level down, going until @sz is the same
+ * as our current PTE.
[..]
Quoted 4 lines
mm/hugetlb.c +4
-0+int hugetlb_hgm_walk (struct mm_struct *mm,struct vm_area_struct *vma, +struct hugetlb_pte *hpte,unsigned long addr, +unsigned long sz,bool stop_at_none) +{
[..]
mm/hugetlb.c bool want_pmd_share(struct vm_area_struct *vma, unsigned long addr) · line 7402
Quoted 7 lines
mm/hugetlb.c +7
-0+while (hugetlb_pte_size (hpte) > sz && !ret) { + pte =huge_ptep_get (hpte->ptep); +if (!pte_present (pte)) { +if (stop_at_none) +return 0 ; +if (unlikely (!huge_pte_none (pte))) +return -EEXIST ;
If 'stop_at_none' means settling down on the just encountered empty PTE, should the above two "if" clauses switch order? I thought Peter has raised this question too, but I'm not seeing a response.
Regards,
-jane
Quoted 8 lines
mm/hugetlb.c +8
-0+ }else if (hugetlb_pte_present_leaf (hpte, pte)) +return 0 ; + ret =hugetlb_walk_step (mm, hpte, addr, sz); + } + +return ret; +} +
Re: [RFC PATCH v2 12/47] hugetlb: add hugetlb_hgm_walk and hugetlb_walk_step
Quoted 7 lines · Jane Chu
Quoted 7 lines
+ * @stop_at_none determines what we do when we encounter an empty PTE. If true,
+ * we return that PTE. If false and @sz is less than the current PTE's size,
+ * we make that PTE point to the next level down, going until @sz is the same
+ * as our current PTE.
Quoted 5 lines
[..]
mm/hugetlb.c bool want_pmd_share(struct vm_area_struct *vma, unsigned long addr) · line 7378
Quoted 4 lines
mm/hugetlb.c +4
-0+int hugetlb_hgm_walk (struct mm_struct *mm,struct vm_area_struct *vma, + struct hugetlb_pte *hpte, unsigned long addr, + unsigned long sz, bool stop_at_none) +{
Also here below, the way 'stop_at_none' is used when HGM isn't enabled is puzzling. Could you elaborate please?
+ if (!hugetlb_hgm_enabled(vma)) {
+ if (stop_at_none)
+ return 0;
+ return sz == huge_page_size(hstate_vma(vma)) ? 0 : -EINVAL;
+ }
Quoted 17 lines
[..]
Quoted 7 lines
+ while (
hugetlb_pte_size(hpte)> sz && !ret) { + pte =huge_ptep_get(hpte->ptep); + if (!pte_present(pte)) {
+ if (stop_at_none)
+ return 0;
+ if (unlikely(!huge_pte_none(pte))) + return-EEXIST;If '
stop_at_none' means settling down on the just encountered empty PTE, should the above two "if" clauses switch order? I thought Peter has raised this question too, but I'm not seeing a response.Regards,
-jane
Quoted 10 lines
mm/hugetlb.c bool want_pmd_share(struct vm_area_struct *vma, unsigned long addr) · line 7410
Quoted 8 lines
+ } else if (
hugetlb_pte_present_leaf(hpte, pte)) + return 0;
+ ret =hugetlb_walk_step(mm, hpte, addr, sz); + }mm/hugetlb.c +4
-0+ + return ret; +} +
thanks,
-jane
Re: [RFC PATCH v2 12/47] hugetlb: add hugetlb_hgm_walk and hugetlb_walk_step
In reply to Jane ChuQuoted 19 lines · Jane Chu
Quoted 5 lines
+ * @stop_at_none determines what we do when we encounter an empty PTE. If true,
+ * we return that PTE. If false and @sz is less than the current PTE's size,
+ * we make that PTE point to the next level down, going until @sz is the same
+ * as our current PTE.[..]
Quoted 4 lines
mm/hugetlb.c +4
-0+int hugetlb_hgm_walk (struct mm_struct *mm,struct vm_area_struct *vma, +struct hugetlb_pte *hpte,unsigned long addr, +unsigned long sz,bool stop_at_none) +{[..]
mm/hugetlb.c bool want_pmd_share(struct vm_area_struct *vma, unsigned long addr) · line 7402
Quoted 7 lines
mm/hugetlb.c +7
-0+while (hugetlb_pte_size (hpte) > sz && !ret) { + pte =huge_ptep_get (hpte->ptep); +if (!pte_present (pte)) { +if (stop_at_none) +return 0 ; +if (unlikely (!huge_pte_none (pte))) +return -EEXIST ;
Quoted 3 lines
If '
stop_at_none' means settling down on the just encountered empty PTE, should the above two "if" clauses switch order? I thought Peter has raised this question too, but I'm not seeing a response.
A better name for "stop_at_none" would be "dont_allocate"; it will be changed in the next version. The idea is that "stop_at_none" would simply do a walk, and the caller will deal with what it finds. If we can't continue the walk for any reason, just return 0. So in this case, if we land on a non-present, non-none PTE, we can't continue the walk, so just return 0.
Another way to justify this order: we want to ensure that calls to this function with stop_at_none=1 and sz=PAGE_SIZE will never fail, and that gives us the order that you see. (This requirement is documented in the comment above the definition of hugetlb_hgm_walk(). This guarantee makes it easier to write code that uses HGM walks.)
Quoted 8 lines
Also here below, the way '
stop_at_none' is used when HGM isn't enabled is puzzling. Could you elaborate please?Quoted 5 lines
mm/hugetlb.c +5
-0+if (!hugetlb_hgm_enabled (vma)) { +if (stop_at_none) +return 0 ; +return sz ==huge_page_size (hstate_vma (vma)) ?0 : -EINVAL ; + }
This is for the same reason; if "stop_at_none" is provided, we need to guarantee that this function won't fail. If "stop_at_none" is false and sz != huge_page_size(), then the caller is attempting to use HGM without having enabled it, hence -EINVAL.
Both of these bits will be cleaned up with the next version of this series. :)
Thanks!
- James
RFC PATCH v2 13/47 hugetlb: add make_huge_pte_with_shift
In reply to James HoughtonThis allows us to make huge PTEs at shifts other than the hstate shift, which will be necessary for high-granularity mappings.
--- mm/hugetlb.c | 14 +++++++++++--- 1 file changed, 11 insertions(+), 3 deletions(-)
mm/hugetlb.c +11 -3
@@ -4867,11 +4867,11 @@ const struct vm_operations_struct hugetlb_vm_ops = { .pagesize = hugetlb_vm_op_pagesize, }; -static pte_t make_huge_pte (struct vm_area_struct *vma,struct page *page, -int writable) +static pte_t make_huge_pte_with_shift (struct vm_area_struct *vma, +struct page *page,int writable, +int shift) {pte_t entry; -unsigned int shift =huge_page_shift (hstate_vma (vma));if (writable) { entry = huge_pte_mkwrite(huge_pte_mkdirty(mk_pte(page,
@@ -4885,6 +4885,14 @@ static pte_t make_huge_pte(struct vm_area_struct *vma, struct page *page,return entry; } +static pte_t make_huge_pte (struct vm_area_struct *vma,struct page *page, +int writable) +{ +unsigned int shift =huge_page_shift (hstate_vma (vma)); + +return make_huge_pte_with_shift (vma, page, writable, shift); +} +static void set_huge_ptep_writable (struct vm_area_struct *vma,unsigned long address,pte_t *ptep) {
Signature · 1 line
2.38.0.135.g90850a2211-goog
Re: [RFC PATCH v2 13/47] hugetlb: add make_huge_pte_with_shift
Quoted 7 lines · James Houghton
This allows us to make huge PTEs at shifts other than the hstate shift, which will be necessary for high-granularity mappings.
Signed-off-by James Houghton <[email protected]>--- mm/hugetlb.c | 14 +++++++++++--- 1 file changed, 11 insertions(+), 3 deletions(-)
Straight forward,
Signature · 1 line
Mike Kravetz
RFC PATCH v2 14/47 hugetlb: make default arch_make_huge_pte understand small mappings
In reply to James HoughtonThis is a simple change: don't create a "huge" PTE if we are making a regular, PAGE_SIZE PTE. All architectures that want to implement HGM likely need to be changed in a similar way if they implement their own version of arch_make_huge_pte.
--- include/linux/hugetlb.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
include/linux/hugetlb.h +1 -1
@@ -907,7 +907,7 @@ static inline void arch_clear_hugepage_flags(struct page *page) { }static inline pte_t arch_make_huge_pte (pte_t entry,unsigned int shift,vm_flags_t flags) { -return pte_mkhuge (entry); +return shift >PAGE_SHIFT ?pte_mkhuge (entry) : entry; }#endif
Signature · 1 line
2.38.0.135.g90850a2211-goog
Re: [RFC PATCH v2 14/47] hugetlb: make default arch_make_huge_pte understand small mappings
Quoted 4 lines · James Houghton
This is a simple change: don't create a "huge" PTE if we are making a regular,
PAGE_SIZEPTE. All architectures that want to implement HGM likely need to be changed in a similar way if they implement their own version ofarch_make_huge_pte.
Nothing wrong with this patch.
However, I wish there was some way we could flag this requirement in arch specific code. Just seems like something that would be easy to overlook.
Signature · 1 line
Mike Kravetz
Quoted 21 lines
Signed-off-by James Houghton <[email protected]>--- include/linux/hugetlb.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)include/linux/hugetlb.h +1
-1@@ -907,7 +907,7 @@ static inline void arch_clear_hugepage_flags(struct page *page) { }static inline pte_t arch_make_huge_pte (pte_t entry,unsigned int shift,vm_flags_t flags) { -return pte_mkhuge (entry); +return shift >PAGE_SHIFT ?pte_mkhuge (entry) : entry; }#endif Signature · 1 line
2.38.0.135.g90850a2211-goog
RFC PATCH v2 15/47 hugetlbfs: for unmapping, treat HGM-mapped pages as potentially mapped
In reply to James Houghtonhugetlb_vma_maps_page was mostly used as an optimization: if the VMA isn't mapping a page, then we don't have to attempt to unmap it again. We are still able to call the unmap routine if we need to.
For high-granularity mapped pages, we can't easily do a full walk to see if the page is actually mapped or not, so simply return that it might be.
--- fs/hugetlbfs/inode.c | 27 +++++++++++++++++++++------ 1 file changed, 21 insertions(+), 6 deletions(-)
fs/hugetlbfs/inode.c +21 -6
@@ -383,21 +383,34 @@ static void hugetlb_delete_from_page_cache(struct folio *folio)* mutex for the page in the mapping. So, we can not race with page being * faulted into the vma. */ -static bool hugetlb_vma_maps_page (struct vm_area_struct *vma, -unsigned long addr,struct page *page) +static bool hugetlb_vma_maybe_maps_page (struct vm_area_struct *vma, +unsigned long addr,struct page *page) {pte_t *ptep, pte; +struct hugetlb_pte hpte; +struct hstate *h =hstate_vma (vma); - ptep =huge_pte_offset (vma->vm_mm, addr, -huge_page_size (hstate_vma (vma))); + ptep =huge_pte_offset (vma->vm_mm, addr,huge_page_size (h));if (!ptep)return false; +hugetlb_pte_populate (&hpte, ptep,huge_page_shift (h), +hpage_size_to_level (huge_page_size (h))); + pte =huge_ptep_get (ptep);if (huge_pte_none (pte) || !pte_present (pte))return false; +if (!hugetlb_pte_present_leaf (&hpte, pte)) +/* +* The top-level PTE is not a leaf, so it's possible that a PTE +* under us is mapping the page. We aren't holding the VMA +* lock, so it is unsafe to continue the walk further. Instead, +* return true to indicate that we might be mapping the page. +*/ +return true; +if (pte_page (pte) == page)return true;
@@ -457,7 +470,8 @@ static void hugetlb_unmap_file_folio(struct hstate *h, v_start =vma_offset_start (vma, start); v_end =vma_offset_end (vma, end); -if (!hugetlb_vma_maps_page (vma, vma->vm_start + v_start, page)) +if (!hugetlb_vma_maybe_maps_page (vma, vma->vm_start + v_start, + page))continue ;if (!hugetlb_vma_trylock_write (vma)) {
@@ -507,7 +521,8 @@ static void hugetlb_unmap_file_folio(struct hstate *h,*/ v_start =vma_offset_start (vma, start); v_end =vma_offset_end (vma, end); -if (hugetlb_vma_maps_page (vma, vma->vm_start + v_start, page)) +if (hugetlb_vma_maybe_maps_page (vma, vma->vm_start + v_start, + page))unmap_hugepage_range (vma, vma->vm_start + v_start, v_end,NULL ,ZAP_FLAG_DROP_MARKER );
Signature · 1 line
2.38.0.135.g90850a2211-goog
Re: [RFC PATCH v2 15/47] hugetlbfs: for unmapping, treat HGM-mapped pages as potentially mapped
Quoted 30 lines · James Houghton
hugetlb_vma_maps_pagewas mostly used as an optimization: if the VMA isn't mapping a page, then we don't have to attempt to unmap it again. We are still able to call the unmap routine if we need to.For high-granularity mapped pages, we can't easily do a full walk to see if the page is actually mapped or not, so simply return that it might be.
Signed-off-by James Houghton <[email protected]>--- fs/hugetlbfs/inode.c | 27 +++++++++++++++++++++------ 1 file changed, 21 insertions(+), 6 deletions(-)fs/hugetlbfs/inode.c +4
-2@@ -383,21 +383,34 @@ static void hugetlb_delete_from_page_cache(struct folio *folio)* mutex for the page in the mapping. So, we can not race with page being * faulted into the vma. */ -static bool hugetlb_vma_maps_page (struct vm_area_struct *vma, -unsigned long addr,struct page *page) +static bool hugetlb_vma_maybe_maps_page (struct vm_area_struct *vma, +unsigned long addr,struct page *page) {pte_t *ptep, pte; +struct hugetlb_pte hpte; +struct hstate *h =hstate_vma (vma);
fs/hugetlbfs/inode.c static void hugetlb_delete_from_page_cache(struct folio *folio) · line 400
Quoted 10 lines
fs/hugetlbfs/inode.c +4
-2- ptep =huge_pte_offset (vma->vm_mm, addr, -huge_page_size (hstate_vma (vma))); + ptep =huge_pte_offset (vma->vm_mm, addr,huge_page_size (h));if (!ptep)return false; +hugetlb_pte_populate (&hpte, ptep,huge_page_shift (h), +hpage_size_to_level (huge_page_size (h))); +
Only a nit, the hugetlb_pte_populate call should probably be after the check below.
This makes sense, and as you mention hugetlb_vma_maybe_maps_page is mostly an optimization that will still work as designed for non HGM vmas.
Signature · 1 line
Mike Kravetz
Quoted 39 lines
fs/hugetlbfs/inode.c +13
-2pte =huge_ptep_get (ptep);if (huge_pte_none (pte) || !pte_present (pte))return false; +if (!hugetlb_pte_present_leaf (&hpte, pte)) +/* +* The top-level PTE is not a leaf, so it's possible that a PTE +* under us is mapping the page. We aren't holding the VMA +* lock, so it is unsafe to continue the walk further. Instead, +* return true to indicate that we might be mapping the page. +*/ +return true; +if (pte_page (pte) == page)return true;@@ -457,7 +470,8 @@ static void hugetlb_unmap_file_folio(struct hstate *h, v_start =vma_offset_start (vma, start); v_end =vma_offset_end (vma, end); -if (!hugetlb_vma_maps_page (vma, vma->vm_start + v_start, page)) +if (!hugetlb_vma_maybe_maps_page (vma, vma->vm_start + v_start, + page))continue ;if (!hugetlb_vma_trylock_write (vma)) {@@ -507,7 +521,8 @@ static void hugetlb_unmap_file_folio(struct hstate *h,*/ v_start =vma_offset_start (vma, start); v_end =vma_offset_end (vma, end); -if (hugetlb_vma_maps_page (vma, vma->vm_start + v_start, page)) +if (hugetlb_vma_maybe_maps_page (vma, vma->vm_start + v_start, + page))unmap_hugepage_range (vma, vma->vm_start + v_start, v_end,NULL ,ZAP_FLAG_DROP_MARKER );Signature · 1 line
2.38.0.135.g90850a2211-goog
RFC PATCH v2 16/47 hugetlb: make unmapping compatible with high-granularity mappings
In reply to James HoughtonEnlighten __unmap_hugepage_range to deal with high-granularity mappings. This doesn't change its API; it still must be called with hugepage alignment, but it will correctly unmap hugepages that have been mapped at high granularity.
The rules for mapcount and refcount here are:
1. Refcount and mapcount are tracked on the head page.
2. Each page table mapping into some of an hpage will increase that hpage's mapcount and refcount by 1.
Eventually, functionality here can be expanded to allow users to call MADV_DONTNEED on PAGE_SIZE-aligned sections of a hugepage, but that is not done here.
--- include/asm-generic/tlb.h | 6 ++-- mm/hugetlb.c | 76 +++++++++++++++++++++++++-------------- 2 files changed, 52 insertions(+), 30 deletions(-)
include/asm-generic/tlb.h +3 -3
@@ -566,9 +566,9 @@ static inline void tlb_flush_p4d_range(struct mmu_gather *tlb,__tlb_remove_tlb_entry (tlb, ptep, address); \ } while (0) -#define tlb_remove_huge_tlb_entry (h, tlb, ptep, address) \ +#define tlb_remove_huge_tlb_entry (tlb, hpte, address) \ do { \ - unsigned long _sz = huge_page_size(h); \ + unsigned long _sz = hugetlb_pte_size(&hpte); \ if (_sz >= P4D_SIZE) \ tlb_flush_p4d_range(tlb, address, _sz); \ else if (_sz >= PUD_SIZE) \
@@ -577,7 +577,7 @@ static inline void tlb_flush_p4d_range(struct mmu_gather *tlb,tlb_flush_pmd_range (tlb, address, _sz); \ else \tlb_flush_pte_range (tlb ,address ,_sz ); \ -__tlb_remove_tlb_entry (tlb, ptep, address); \ +__tlb_remove_tlb_entry (tlb, hpte.ptep, address);\ }while (0 )/**
mm/hugetlb.c +49 -27
148 lines not on this page · raw message
Signature · 1 line
2.38.0.135.g90850a2211-goog
Re: [RFC PATCH v2 16/47] hugetlb: make unmapping compatible with high-granularity mappings
Quoted 14 lines · James Houghton
Enlighten
__unmap_hugepage_rangeto deal with high-granularity mappings. This doesn't change its API; it still must be called with hugepage alignment, but it will correctly unmap hugepages that have been mapped at high granularity.The rules for mapcount and refcount here are:
1. Refcount and mapcount are tracked on the head page.
2. Each page table mapping into some of an hpage will increase that hpage's mapcount and refcount by 1.Eventually, functionality here can be expanded to allow users to call
MADV_DONTNEEDon PAGE_SIZE-aligned sections of a hugepage, but that is not done here.
Quoted 5 lines
Signed-off-by James Houghton <[email protected]>--- include/asm-generic/tlb.h | 6 ++-- mm/hugetlb.c | 76 +++++++++++++++++++++++++-------------- 2 files changed, 52 insertions(+), 30 deletions(-)
All looks reasonable, nothing stands out.
Signature · 1 line
Mike Kravetz
RFC PATCH v2 17/47 hugetlb: make hugetlb_change_protection compatible with HGM
In reply to James HoughtonThe main change here is to do a high-granularity walk and pulling the shift from the walk (not from the hstate).
--- mm/hugetlb.c | 65 ++++++++++++++++++++++++++++++++++++---------------- 1 file changed, 45 insertions(+), 20 deletions(-)
mm/hugetlb.c +45 -20
@@ -6654,15 +6654,15 @@ unsigned long hugetlb_change_protection(struct vm_area_struct *vma, {struct mm_struct *mm = vma->vm_mm;unsigned long start = address; -pte_t *ptep;pte_t pte;struct hstate *h =hstate_vma (vma); -unsigned long pages =0 , psize =huge_page_size (h); +unsigned long base_pages =0 , psize =huge_page_size (h);bool shared_pmd = false;struct mmu_notifier_range range;unsigned long last_addr_mask;bool uffd_wp = cp_flags &MM_CP_UFFD_WP ;bool uffd_wp_resolve = cp_flags &MM_CP_UFFD_WP_RESOLVE ; +struct hugetlb_pte hpte;/* * In the case of shared PMDs, the area to flush could be beyond
@@ -6680,31 +6680,38 @@ unsigned long hugetlb_change_protection(struct vm_area_struct *vma,hugetlb_vma_lock_write (vma);i_mmap_lock_write (vma->vm_file->f_mapping); last_addr_mask =hugetlb_mask_last_page (h); -for (; address < end; address += psize) { +while (address < end) {spinlock_t *ptl; - ptep =huge_pte_offset (mm, address, psize); +pte_t *ptep =huge_pte_offset (mm, address, psize); +if (!ptep) { address |= last_addr_mask; + address +=huge_page_size (h);continue ; } - ptl =huge_pte_lock (h, mm, ptep); -if (huge_pmd_unshare (mm, vma, address, ptep)) { +hugetlb_pte_populate (&hpte, ptep,huge_page_shift (h), +hpage_size_to_level (psize)); +hugetlb_hgm_walk (mm, vma, &hpte, address,PAGE_SIZE , +/*stop_at_none=*/ true); + + ptl =hugetlb_pte_lock (mm, &hpte); +if (huge_pmd_unshare (mm, vma, address, hpte.ptep)) {/* * When uffd-wp is enabled on the vma, unshare * shouldn't happen at all. Warn about it if it * happened due to some reason. */ WARN_ON_ONCE (uffd_wp || uffd_wp_resolve); - pages++; + base_pages +=hugetlb_pte_size (&hpte) /PAGE_SIZE ;spin_unlock (ptl); shared_pmd = true; address |= last_addr_mask; -continue ; + goto next_hpte; } - pte =huge_ptep_get (ptep); + pte =huge_ptep_get (hpte.ptep);if (unlikely (is_hugetlb_entry_hwpoisoned (pte))) {spin_unlock (ptl); -continue ; + goto next_hpte; }if (unlikely (is_hugetlb_entry_migration (pte))) {swp_entry_t entry =pte_to_swp_entry (pte);
@@ -6724,11 +6731,11 @@ unsigned long hugetlb_change_protection(struct vm_area_struct *vma, newpte =pte_swp_mkuffd_wp (newpte); else if (uffd_wp_resolve) newpte =pte_swp_clear_uffd_wp (newpte); -set_huge_pte_at (mm, address, ptep, newpte); - pages++; +set_huge_pte_at (mm, address, hpte.ptep, newpte); + base_pages +=hugetlb_pte_size (&hpte) /PAGE_SIZE ; }spin_unlock (ptl); -continue ; + goto next_hpte; }if (unlikely (pte_marker_uffd_wp (pte))) {/*
@@ -6736,21 +6743,37 @@ unsigned long hugetlb_change_protection(struct vm_area_struct *vma,* no need for huge_ptep_modify_prot_start/commit(). */ if (uffd_wp_resolve) -huge_pte_clear (mm, address, ptep, psize); +huge_pte_clear (mm, address, hpte.ptep, +hugetlb_pte_size (&hpte)); }if (!huge_pte_none (pte)) {pte_t old_pte; -unsigned int shift =huge_page_shift (hstate_vma (vma)); +unsigned int shift = hpte.shift; - old_pte =huge_ptep_modify_prot_start (vma, address, ptep); +/* +* Because we are holding the VMA lock for writing, pte +* will always be a leaf. WARN if it is not. +*/ +if (unlikely (!hugetlb_pte_present_leaf (&hpte, pte))) { +spin_unlock (ptl); +WARN_ONCE (1 ,"Unexpected non-leaf PTE: ptep:%p, address:0x%lx\n" , + hpte.ptep, address); +continue ; + } + + old_pte =huge_ptep_modify_prot_start ( + vma, address, hpte.ptep); pte =huge_pte_modify (old_pte, newprot); - pte =arch_make_huge_pte (pte, shift, vma->vm_flags); + pte =arch_make_huge_pte ( + pte, shift, vma->vm_flags);if (uffd_wp) pte =huge_pte_mkuffd_wp (huge_pte_wrprotect (pte));else if (uffd_wp_resolve) pte =huge_pte_clear_uffd_wp (pte); -huge_ptep_modify_prot_commit (vma, address, ptep, old_pte, pte); - pages++; +huge_ptep_modify_prot_commit ( + vma, address, hpte.ptep, + old_pte, pte); + base_pages +=hugetlb_pte_size (&hpte) /PAGE_SIZE ; }else {/* None pte */ if (unlikely (uffd_wp))
@@ -6759,6 +6782,8 @@ unsigned long hugetlb_change_protection(struct vm_area_struct *vma, make_pte_marker(PTE_MARKER_UFFD_WP)); }spin_unlock (ptl); +next_hpte: + address +=hugetlb_pte_size (&hpte); }/* * Must flush TLB before releasing i_mmap_rwsem: x86's huge_pmd_unshare
@@ -6781,7 +6806,7 @@ unsigned long hugetlb_change_protection(struct vm_area_struct *vma,hugetlb_vma_unlock_write (vma);mmu_notifier_invalidate_range_end (&range); -return pages << h->order; +return base_pages; }/* Return true if reservation was successful, false otherwise. */
Signature · 1 line
2.38.0.135.g90850a2211-goog
Re: [RFC PATCH v2 17/47] hugetlb: make hugetlb_change_protection compatible with HGM
Quoted 7 lines · James Houghton
The main change here is to do a high-granularity walk and pulling the shift from the walk (not from the hstate).
Signed-off-by James Houghton <[email protected]>--- mm/hugetlb.c | 65 ++++++++++++++++++++++++++++++++++++---------------- 1 file changed, 45 insertions(+), 20 deletions(-)
Nothing stands out. The more patches I look at, the more familiar I am becoming with the new data structures and interfaces.
Signature · 1 line
Mike Kravetz
RFC PATCH v2 18/47 hugetlb: enlighten follow_hugetlb_page to support HGM
In reply to James HoughtonThis enables high-granularity mapping support in GUP.
One important change here is that, before, we never needed to grab the VMA lock, but now, to prevent someone from collapsing the page tables out from under us, we grab it for reading when doing high-granularity PT walks.
In case it is confusing, pfn_offset is the offset (in PAGE_SIZE units) that vaddr points to within the subpage that hpte points to.
--- mm/hugetlb.c | 76 ++++++++++++++++++++++++++++++++++++---------------- 1 file changed, 53 insertions(+), 23 deletions(-)
mm/hugetlb.c +53 -23
@@ -6382,11 +6382,9 @@ static void record_subpages_vmas(struct page *page, struct vm_area_struct *vma, } } -static inline bool __follow_hugetlb_must_fault (unsigned int flags,pte_t *pte, +static inline bool __follow_hugetlb_must_fault (unsigned int flags,pte_t pteval,bool *unshare) { -pte_t pteval =huge_ptep_get (pte); - *unshare = false;if (is_swap_pte (pteval))return true;
@@ -6478,12 +6476,20 @@ long follow_hugetlb_page(struct mm_struct *mm, struct vm_area_struct *vma,struct hstate *h =hstate_vma (vma);int err = -EFAULT , refs; +/* +* Grab the VMA lock for reading now so no one can collapse the page +* table from under us. +*/ +hugetlb_vma_lock_read (vma); +while (vaddr < vma->vm_end && remainder) { -pte_t *pte; +pte_t *ptep, pte;spinlock_t *ptl =NULL ;bool unshare = false;int absent; -struct page *page; +unsigned long pages_per_hpte; +struct page *page, *subpage; +struct hugetlb_pte hpte;/* * If we have a pending SIGKILL, don't keep faulting pages and
@@ -6499,13 +6505,22 @@ long follow_hugetlb_page(struct mm_struct *mm, struct vm_area_struct *vma,* each hugepage. We have to make sure we get the * first, for the page indexing below to work. * -* Note that page table lock is not held when pte is null. +* Note that page table lock is not held when ptep is null. */ - pte =huge_pte_offset (mm, vaddr &huge_page_mask (h), -huge_page_size (h)); -if (pte) - ptl =huge_pte_lock (h, mm, pte); - absent = !pte ||huge_pte_none (huge_ptep_get (pte)); + ptep =huge_pte_offset (mm, vaddr &huge_page_mask (h), +huge_page_size (h)); +if (ptep) { +hugetlb_pte_populate (&hpte, ptep,huge_page_shift (h), +hpage_size_to_level (huge_page_size (h))); +hugetlb_hgm_walk (mm, vma, &hpte, vaddr, +PAGE_SIZE , +/*stop_at_none=*/ true); + ptl =hugetlb_pte_lock (mm, &hpte); + ptep = hpte.ptep; + pte =huge_ptep_get (ptep); + } + + absent = !ptep ||huge_pte_none (pte);/* * When coredumping, it suits get_dump_page if we just return
@@ -6516,12 +6531,19 @@ long follow_hugetlb_page(struct mm_struct *mm, struct vm_area_struct *vma,*/ if (absent && (flags &FOLL_DUMP ) && !hugetlbfs_pagecache_present (h, vma, vaddr)) { -if (pte) +if (ptep)spin_unlock (ptl); remainder =0 ;break ; } +if (!absent &&pte_present (pte) && + !hugetlb_pte_present_leaf (&hpte, pte)) { +/* We raced with someone splitting the PTE, so retry. */ +spin_unlock (ptl); +continue ; + } +/* * We need call hugetlb_fault for both hugepages under migration * (in which case hugetlb_fault waits for the migration,) and
@@ -6537,7 +6559,10 @@ long follow_hugetlb_page(struct mm_struct *mm, struct vm_area_struct *vma,vm_fault_t ret;unsigned int fault_flags =0 ; -if (pte) +/* Drop the lock before entering hugetlb_fault. */ +hugetlb_vma_unlock_read (vma); + +if (ptep)spin_unlock (ptl);if (flags &FOLL_WRITE ) fault_flags |=FAULT_FLAG_WRITE ;
@@ -6560,7 +6585,7 @@ long follow_hugetlb_page(struct mm_struct *mm, struct vm_area_struct *vma,if (ret &VM_FAULT_ERROR ) { err =vm_fault_to_errno (ret, flags); remainder =0 ; -break ; + goto out; }if (ret &VM_FAULT_RETRY ) { if (locked &&
@@ -6578,11 +6603,14 @@ long follow_hugetlb_page(struct mm_struct *mm, struct vm_area_struct *vma,*/ return i; } +hugetlb_vma_lock_read (vma);continue ; } - pfn_offset = (vaddr & ~huge_page_mask (h)) >>PAGE_SHIFT ; - page =pte_page (huge_ptep_get (pte)); + pfn_offset = (vaddr & ~hugetlb_pte_mask (&hpte)) >>PAGE_SHIFT ; + subpage =pte_page (pte); + pages_per_hpte =hugetlb_pte_size (&hpte) /PAGE_SIZE ; + page =compound_head (subpage);VM_BUG_ON_PAGE ((flags &FOLL_PIN ) &&PageAnon (page) && !PageAnonExclusive (page), page);
@@ -6592,21 +6620,21 @@ long follow_hugetlb_page(struct mm_struct *mm, struct vm_area_struct *vma,* and skip the same_page loop below. */ if (!pages && !vmas && !pfn_offset && - (vaddr +huge_page_size (h) < vma->vm_end) && - (remainder >=pages_per_huge_page (h))) { - vaddr +=huge_page_size (h); - remainder -=pages_per_huge_page (h); - i +=pages_per_huge_page (h); + (vaddr + pages_per_hpte < vma->vm_end) && + (remainder >= pages_per_hpte)) { + vaddr += pages_per_hpte; + remainder -= pages_per_hpte; + i += pages_per_hpte;spin_unlock (ptl);continue ; }/* vaddr may not be aligned to PAGE_SIZE */ - refs =min3 (pages_per_huge_page (h) - pfn_offset, remainder, + refs =min3 (pages_per_hpte - pfn_offset, remainder, (vma->vm_end -ALIGN_DOWN (vaddr,PAGE_SIZE )) >>PAGE_SHIFT );if (pages || vmas) -record_subpages_vmas (nth_page (page, pfn_offset), +record_subpages_vmas (nth_page (subpage, pfn_offset), vma, refs,likely (pages) ? pages + i :NULL , vmas ? vmas + i :NULL );
@@ -6637,6 +6665,8 @@ long follow_hugetlb_page(struct mm_struct *mm, struct vm_area_struct *vma,spin_unlock (ptl); } +hugetlb_vma_unlock_read (vma); +out: *nr_pages = remainder;/* * setting position is actually required only if remainder is
Signature · 1 line
2.38.0.135.g90850a2211-goog
Re: [RFC PATCH v2 18/47] hugetlb: enlighten follow_hugetlb_page to support HGM
Quoted 6 lines · James Houghton
This enables high-granularity mapping support in GUP.
One important change here is that, before, we never needed to grab the VMA lock, but now, to prevent someone from collapsing the page tables out from under us, we grab it for reading when doing high-granularity PT walks.
Once again, I think Peter's series will already take the vma lock here.
Quoted 28 lines
In case it is confusing,
pfn_offsetis the offset (inPAGE_SIZEunits) that vaddr points to within the subpage that hpte points to.Signed-off-by James Houghton <[email protected]>--- mm/hugetlb.c | 76 ++++++++++++++++++++++++++++++++++++---------------- 1 file changed, 53 insertions(+), 23 deletions(-)mm/hugetlb.c +1
-3@@ -6382,11 +6382,9 @@ static void record_subpages_vmas(struct page *page, struct vm_area_struct *vma, } } -static inline bool __follow_hugetlb_must_fault (unsigned int flags,pte_t *pte, +static inline bool __follow_hugetlb_must_fault (unsigned int flags,pte_t pteval,bool *unshare) { -pte_t pteval =huge_ptep_get (pte); - *unshare = false;if (is_swap_pte (pteval))return true;@@ -6478,12 +6476,20 @@ long follow_hugetlb_page(struct mm_struct *mm, struct vm_area_struct *vma,struct hstate *h =hstate_vma (vma);int err = -EFAULT , refs;
mm/hugetlb.c long follow_hugetlb_page(struct mm_struct *mm, struct vm_area_struct *vma, · line 6486
Quoted 10 lines
mm/hugetlb.c +7
-1+/* +* Grab the VMA lock for reading now so no one can collapse the page +* table from under us. +*/ +hugetlb_vma_lock_read (vma); +while (vaddr < vma->vm_end && remainder) { -pte_t *pte; +pte_t *ptep, pte;
Thanks, that really would be better as ptep in the existing code.
Quoted 26 lines
mm/hugetlb.c +8
-7spinlock_t *ptl =NULL ;bool unshare = false;int absent; -struct page *page; +unsigned long pages_per_hpte; +struct page *page, *subpage; +struct hugetlb_pte hpte;/* * If we have a pending SIGKILL, don't keep faulting pages and @@ -6499,13 +6505,22 @@ long follow_hugetlb_page(struct mm_struct *mm, struct vm_area_struct *vma,* each hugepage. We have to make sure we get the * first, for the page indexing below to work. * -* Note that page table lock is not held when pte is null. +* Note that page table lock is not held when ptep is null. */ - pte =huge_pte_offset (mm, vaddr &huge_page_mask (h), -huge_page_size (h)); -if (pte) - ptl =huge_pte_lock (h, mm, pte); - absent = !pte ||huge_pte_none (huge_ptep_get (pte)); + ptep =huge_pte_offset (mm, vaddr &huge_page_mask (h), +huge_page_size (h)); +if (ptep) { + hugetlb_pte_populate(&hpte, ptep, huge_page_shift(h),
mm/hugetlb.c long follow_hugetlb_page(struct mm_struct *mm, struct vm_area_struct *vma, · line 6523
Quoted 10 lines
mm/hugetlb.c +10
-0+ hpage_size_to_level(huge_page_size(h))); +hugetlb_hgm_walk (mm, vma, &hpte, vaddr, +PAGE_SIZE , +/*stop_at_none=*/ true); + ptl =hugetlb_pte_lock (mm, &hpte); + ptep = hpte.ptep; + pte =huge_ptep_get (ptep); + } + + absent = !ptep ||huge_pte_none (pte);
In Peter's series, huge_pte_offset calls are replaced with hugetlb_walk that takes a vma pointer. It might make sense now to consolidate the hugetlb page table walkers. I know that was discussed at some time. Just thinking we could possibly fold much of the above into hugetlb_walk.
Quoted 7 lines
mm/hugetlb.c +0
-0/* * When coredumping, it suits get_dump_page if we just return @@ -6516,12 +6531,19 @@ long follow_hugetlb_page(struct mm_struct *mm, struct vm_area_struct *vma,*/ if (absent && (flags &FOLL_DUMP ) && !hugetlbfs_pagecache_present (h, vma, vaddr)) {
mm/hugetlb.c long follow_hugetlb_page(struct mm_struct *mm, struct vm_area_struct *vma, · line 6542
Quoted 10 lines
mm/hugetlb.c +4
-1-if (pte) +if (ptep)spin_unlock (ptl); remainder =0 ;break ; } +if (!absent &&pte_present (pte) && + !hugetlb_pte_present_leaf (&hpte, pte)) { +/* We raced with someone splitting the PTE, so retry. */
I do not think I have got to the splitting code yet, but I am assuming we do not hold vma lock for write when splitting. We would of course hold page table lock.
Quoted 64 lines
mm/hugetlb.c +19
-9+spin_unlock (ptl); +continue ; + } +/* * We need call hugetlb_fault for both hugepages under migration * (in which case hugetlb_fault waits for the migration,) and @@ -6537,7 +6559,10 @@ long follow_hugetlb_page(struct mm_struct *mm, struct vm_area_struct *vma,vm_fault_t ret;unsigned int fault_flags =0 ; -if (pte) +/* Drop the lock before entering hugetlb_fault. */ +hugetlb_vma_unlock_read (vma); + +if (ptep)spin_unlock (ptl);if (flags &FOLL_WRITE ) fault_flags |=FAULT_FLAG_WRITE ;@@ -6560,7 +6585,7 @@ long follow_hugetlb_page(struct mm_struct *mm, struct vm_area_struct *vma,if (ret &VM_FAULT_ERROR ) { err =vm_fault_to_errno (ret, flags); remainder =0 ; -break ; + goto out; }if (ret &VM_FAULT_RETRY ) { if (locked &&@@ -6578,11 +6603,14 @@ long follow_hugetlb_page(struct mm_struct *mm, struct vm_area_struct *vma,*/ return i; } +hugetlb_vma_lock_read (vma);continue ; } - pfn_offset = (vaddr & ~huge_page_mask (h)) >>PAGE_SHIFT ; - page =pte_page (huge_ptep_get (pte)); + pfn_offset = (vaddr & ~hugetlb_pte_mask (&hpte)) >>PAGE_SHIFT ; + subpage =pte_page (pte); + pages_per_hpte =hugetlb_pte_size (&hpte) /PAGE_SIZE ; + page =compound_head (subpage);VM_BUG_ON_PAGE ((flags &FOLL_PIN ) &&PageAnon (page) && !PageAnonExclusive (page), page);@@ -6592,21 +6620,21 @@ long follow_hugetlb_page(struct mm_struct *mm, struct vm_area_struct *vma,* and skip the same_page loop below. */ if (!pages && !vmas && !pfn_offset && - (vaddr +huge_page_size (h) < vma->vm_end) && - (remainder >=pages_per_huge_page (h))) { - vaddr +=huge_page_size (h); - remainder -=pages_per_huge_page (h); - i +=pages_per_huge_page (h); + (vaddr + pages_per_hpte < vma->vm_end) && + (remainder >= pages_per_hpte)) { + vaddr += pages_per_hpte; + remainder -= pages_per_hpte; + i += pages_per_hpte;spin_unlock (ptl);continue ; }/* vaddr may not be aligned to PAGE_SIZE */
mm/hugetlb.c long follow_hugetlb_page(struct mm_struct *mm, struct vm_area_struct *vma, · line 6640
Quoted 10 lines
mm/hugetlb.c +2
-2- refs =min3 (pages_per_huge_page (h) - pfn_offset, remainder, + refs =min3 (pages_per_hpte - pfn_offset, remainder, (vma->vm_end -ALIGN_DOWN (vaddr,PAGE_SIZE )) >>PAGE_SHIFT );if (pages || vmas) -record_subpages_vmas (nth_page (page, pfn_offset), +record_subpages_vmas (nth_page (subpage, pfn_offset), vma, refs,likely (pages) ? pages + i :NULL , vmas ? vmas + i :NULL );
Not your fault, but all the above was difficult to follow before HGM. :( Did not notice any issues.
Signature · 1 line
Mike Kravetz
Quoted 12 lines
mm/hugetlb.c +2
-0@@ -6637,6 +6665,8 @@ long follow_hugetlb_page(struct mm_struct *mm, struct vm_area_struct *vma,spin_unlock (ptl); } +hugetlb_vma_unlock_read (vma); +out: *nr_pages = remainder;/* * setting position is actually required only if remainder is Signature · 1 line
2.38.0.135.g90850a2211-goog
RFC PATCH v2 19/47 hugetlb: make hugetlb_follow_page_mask HGM-enabled
In reply to James HoughtonThe change here is very simple: do a high-granularity walk.
--- mm/hugetlb.c | 16 +++++++++++++++- 1 file changed, 15 insertions(+), 1 deletion(-)
mm/hugetlb.c +15 -1
@@ -6408,6 +6408,7 @@ struct page *hugetlb_follow_page_mask(struct vm_area_struct *vma,struct page *page =NULL ;spinlock_t *ptl;pte_t *pte, entry; +struct hugetlb_pte hpte;/* * FOLL_PIN is not supported for follow_page(). Ordinary GUP goes via
@@ -6429,9 +6430,22 @@ struct page *hugetlb_follow_page_mask(struct vm_area_struct *vma,return NULL ; } - ptl =huge_pte_lock (h, mm, pte); +retry_walk: +hugetlb_pte_populate (&hpte, pte,huge_page_shift (h), +hpage_size_to_level (huge_page_size (h))); +hugetlb_hgm_walk (mm, vma, &hpte, address, +PAGE_SIZE , +/*stop_at_none=*/ true); + + ptl =hugetlb_pte_lock (mm, &hpte); entry =huge_ptep_get (pte);if (pte_present (entry)) { +if (unlikely (!hugetlb_pte_present_leaf (&hpte, entry))) { +/* We raced with someone splitting from under us. */ +spin_unlock (ptl); + goto retry_walk; + } + page =pte_page (entry) + ((address & ~huge_page_mask (h)) >>PAGE_SHIFT );/*
Signature · 1 line
2.38.0.135.g90850a2211-goog
Re: [RFC PATCH v2 19/47] hugetlb: make hugetlb_follow_page_mask HGM-enabled
Quoted 6 lines · James Houghton
The change here is very simple: do a high-granularity walk.
Signed-off-by James Houghton <[email protected]>--- mm/hugetlb.c | 16 +++++++++++++++- 1 file changed, 15 insertions(+), 1 deletion(-)
Looks fine
Signature · 1 line
Mike Kravetz
RFC PATCH v2 20/47 hugetlb: use struct hugetlb_pte for walk_hugetlb_range
In reply to James HoughtonThe main change in this commit is to walk_hugetlb_range to support walking HGM mappings, but all walk_hugetlb_range callers must be updated to use the new API and take the correct action.
Listing all the changes to the callers:
For s390 changes, we simply ignore HGM PTEs (we don't support s390 yet).
For smaps, shared_hugetlb (and private_hugetlb, although private mappings don't support HGM) may now not be divisible by the hugepage size. The appropriate changes have been made to support analyzing HGM PTEs.
For pagemap, we ignore non-leaf PTEs by treating that as if they were none PTEs. We can only end up with non-leaf PTEs if they had just been updated from a none PTE.
For show_numa_map, the challenge is that, if any of a hugepage is mapped, we have to count that entire page exactly once, as the results are given in units of hugepages. To support HGM mappings, we keep track of the last page that we looked it. If the hugepage we are currently looking at is the same as the last one, then we must be looking at an HGM-mapped page that has been mapped at high-granularity, and we've already accounted for it.
For DAMON, we treat non-leaf PTEs as if they were blank, for the same reason as pagemap.
For hwpoison, we proactively update the logic to support the case when hpte is pointing to a subpage within the poisoned hugepage.
For queue_pages_hugetlb/migration, we ignore all HGM-enabled VMAs for now.
For mincore, we ignore non-leaf PTEs for the same reason as pagemap.
For mprotect/prot_none_hugetlb_entry, we retry the walk when we get a non-leaf PTE.
--- arch/s390/mm/gmap.c | 20 ++++++++-- fs/proc/task_mmu.c | 83 +++++++++++++++++++++++++++++----------- include/linux/pagewalk.h | 11 ++++-- mm/damon/vaddr.c | 57 +++++++++++++++++---------- mm/hmm.c | 21 ++++++---- mm/memory-failure.c | 17 ++++---- mm/mempolicy.c | 12 ++++-- mm/mincore.c | 17 ++++++-- mm/mprotect.c | 18 ++++++--- mm/pagewalk.c | 32 +++++++++++++--- 10 files changed, 203 insertions(+), 85 deletions(-)
arch/s390/mm/gmap.c +16 -4
@@ -2622,13 +2622,25 @@ static int __s390_enable_skey_pmd(pmd_t *pmd, unsigned long addr,return 0 ; } -static int __s390_enable_skey_hugetlb (pte_t *pte,unsigned long addr, -unsigned long hmask,unsigned long next, +static int __s390_enable_skey_hugetlb (struct hugetlb_pte *hpte, +unsigned long addr,struct mm_walk *walk) { -pmd_t *pmd = (pmd_t *)pte; +struct hstate *h =hstate_vma (walk->vma); +pmd_t *pmd;unsigned long start, end; -struct page *page =pmd_page (*pmd); +struct page *page; + +if (huge_page_size (h) !=hugetlb_pte_size (hpte)) +/* Ignore high-granularity PTEs. */ +return 0 ; + +if (!pte_present (huge_ptep_get (hpte->ptep))) +/* Ignore non-present PTEs. */ +return 0 ; + + pmd = (pmd_t *)pte; + page =pmd_page (*pmd);/* * The write check makes sure we do not set a key on shared
fs/proc/task_mmu.c +61 -22
167 lines not on this page · raw message
include/linux/pagewalk.h +7 -4
@@ -3,6 +3,7 @@#define _LINUX_PAGEWALK_H#include <linux/mm.h> +#include <linux/hugetlb.h> struct mm_walk ;
@@ -21,7 +22,10 @@ struct mm_walk;* depth is -1 if not known, 0:PGD, 1:P4D, 2:PUD, 3:PMD. * Any folded depths (where PTRS_PER_P?D is equal to 1) * are skipped. -* @hugetlb_entry: if set, called for each hugetlb entry +* @hugetlb_entry: if set, called for each hugetlb entry. In the presence +* of high-granularity hugetlb entries, @hugetlb_entry is +* called only for leaf-level entries (i.e., hstate-level +* page table entries are ignored if they are not leaves). * @test_walk: caller specific callback function to determine whether * we walk over the current vma or not. Returning 0 means * "do page table walk over the current vma", returning
@@ -47,9 +51,8 @@ struct mm_walk_ops { unsigned long next, struct mm_walk *walk);int (*pte_hole)(unsigned long addr,unsigned long next,int depth,struct mm_walk *walk); -int (*hugetlb_entry)(pte_t *pte,unsigned long hmask, -unsigned long addr,unsigned long next, -struct mm_walk *walk); +int (*hugetlb_entry)(struct hugetlb_pte *hpte, +unsigned long addr,struct mm_walk *walk);int (*test_walk)(unsigned long addr,unsigned long next,struct mm_walk *walk); int (*pre_vma)(unsigned long start, unsigned long end,
mm/damon/vaddr.c +36 -21
115 lines not on this page · raw message
mm/hmm.c +14 -7
@@ -469,27 +469,34 @@ static int hmm_vma_walk_pud(pud_t *pudp, unsigned long start, unsigned long end,#endif #ifdef CONFIG_HUGETLB_PAGE -static int hmm_vma_walk_hugetlb_entry (pte_t *pte,unsigned long hmask, -unsigned long start,unsigned long end, +static int hmm_vma_walk_hugetlb_entry (struct hugetlb_pte *hpte, +unsigned long start,struct mm_walk *walk) {unsigned long addr = start, i, pfn;struct hmm_vma_walk *hmm_vma_walk = walk->private;struct hmm_range *range = hmm_vma_walk->range; -struct vm_area_struct *vma = walk->vma;unsigned int required_fault;unsigned long pfn_req_flags;unsigned long cpu_flags; +unsigned long hmask =hugetlb_pte_mask (hpte); +unsigned int order =hugetlb_pte_shift (hpte) -PAGE_SHIFT ; +unsigned long end = start +hugetlb_pte_size (hpte);spinlock_t *ptl;pte_t entry; - ptl =huge_pte_lock (hstate_vma (vma), walk->mm, pte); - entry =huge_ptep_get (pte); + ptl =hugetlb_pte_lock (walk->mm, hpte); + entry =huge_ptep_get (hpte->ptep); + +if (!hugetlb_pte_present_leaf (hpte, entry)) { +spin_unlock (ptl); +return -EAGAIN ; + } i = (start - range->start) >>PAGE_SHIFT ; pfn_req_flags = range->hmm_pfns[i]; cpu_flags =pte_to_hmm_pfn_flags (range, entry) | -hmm_pfn_flags_order (huge_page_order (hstate_vma (vma))); +hmm_pfn_flags_order (order); required_fault =hmm_pte_need_fault (hmm_vma_walk, pfn_req_flags, cpu_flags);if (required_fault) {
@@ -593,7 +600,7 @@ int hmm_range_fault(struct hmm_range *range)* in pfns. All entries < last in the pfn array are set to their * output, and all >= are still at their input values. */ - }while (ret == -EBUSY ); + }while (ret == -EBUSY || ret == -EAGAIN );return ret; }EXPORT_SYMBOL (hmm_range_fault);
mm/memory-failure.c +9 -8
@@ -628,6 +628,7 @@ static int check_hwpoisoned_entry(pte_t pte, unsigned long addr, short shift, unsigned long poisoned_pfn, struct to_kill *tk) { unsigned long pfn = 0; +unsigned long base_pages_poisoned = (1UL << shift) /PAGE_SIZE ;if (pte_present (pte)) { pfn =pte_pfn (pte);
@@ -638,7 +639,8 @@ static int check_hwpoisoned_entry(pte_t pte, unsigned long addr, short shift, pfn =swp_offset_pfn (swp); } -if (!pfn || pfn != poisoned_pfn) +if (!pfn || pfn < poisoned_pfn || + pfn >= poisoned_pfn + base_pages_poisoned)return 0 ;set_to_kill (tk, addr, shift);
@@ -704,16 +706,15 @@ static int hwpoison_pte_range(pmd_t *pmdp, unsigned long addr, }#ifdef CONFIG_HUGETLB_PAGE -static int hwpoison_hugetlb_range (pte_t *ptep,unsigned long hmask, -unsigned long addr,unsigned long end, -struct mm_walk *walk) +static int hwpoison_hugetlb_range (struct hugetlb_pte *hpte, +unsigned long addr, +struct mm_walk *walk) {struct hwp_walk *hwp = walk->private; -pte_t pte =huge_ptep_get (ptep); -struct hstate *h =hstate_vma (walk->vma); +pte_t pte =huge_ptep_get (hpte->ptep); -return check_hwpoisoned_entry (pte, addr,huge_page_shift (h), - hwp->pfn, &hwp->tk); +return check_hwpoisoned_entry (pte, addr &hugetlb_pte_mask (hpte), + hpte->shift, hwp->pfn, &hwp->tk); }#else #define hwpoison_hugetlb_range NULL
mm/mempolicy.c +8 -4
@@ -558,8 +558,8 @@ static int queue_pages_pte_range(pmd_t *pmd, unsigned long addr,return addr != end ? -EIO :0 ; } -static int queue_pages_hugetlb (pte_t *pte,unsigned long hmask, -unsigned long addr,unsigned long end, +static int queue_pages_hugetlb (struct hugetlb_pte *hpte, +unsigned long addr,struct mm_walk *walk) {int ret =0 ;
@@ -570,8 +570,12 @@ static int queue_pages_hugetlb(pte_t *pte, unsigned long hmask,spinlock_t *ptl;pte_t entry; - ptl =huge_pte_lock (hstate_vma (walk->vma), walk->mm, pte); - entry =huge_ptep_get (pte); +/* We don't migrate high-granularity HugeTLB mappings for now. */ +if (hugetlb_hgm_enabled (walk->vma)) +return -EINVAL ; + + ptl =hugetlb_pte_lock (walk->mm, hpte); + entry =huge_ptep_get (hpte->ptep);if (!pte_present (entry)) goto unlock; page =pte_page (entry);
mm/mincore.c +14 -3
@@ -22,18 +22,29 @@#include <linux/uaccess.h> #include "swap.h" -static int mincore_hugetlb (pte_t *pte,unsigned long hmask,unsigned long addr, -unsigned long end,struct mm_walk *walk) +static int mincore_hugetlb (struct hugetlb_pte *hpte,unsigned long addr, +struct mm_walk *walk) {#ifdef CONFIG_HUGETLB_PAGE unsigned char present; +unsigned long end = addr +hugetlb_pte_size (hpte);unsigned char *vec = walk->private; +pte_t pte =huge_ptep_get (hpte->ptep);/* * Hugepages under user process are always in RAM and never * swapped out, but theoretically it needs to be checked. */ - present = pte && !huge_pte_none (huge_ptep_get (pte)); + present = !huge_pte_none (pte); + +/* +* If the pte is present but not a leaf, we raced with someone +* splitting it. For someone to have split it, it must have been +* huge_pte_none before, so treat it as such. +*/ +if (pte_present (pte) && !hugetlb_pte_present_leaf (hpte, pte)) + present = false; +for (; addr != end; vec++, addr +=PAGE_SIZE ) *vec = present; walk->private = vec;
mm/mprotect.c +12 -6
@@ -524,12 +524,16 @@ static int prot_none_pte_entry(pte_t *pte, unsigned long addr, 0 : -EACCES; } -static int prot_none_hugetlb_entry (pte_t *pte,unsigned long hmask, -unsigned long addr,unsigned long next, +static int prot_none_hugetlb_entry (struct hugetlb_pte *hpte, +unsigned long addr,struct mm_walk *walk) { -return pfn_modify_allowed (pte_pfn (*pte), *(pgprot_t *)(walk->private)) ? -0 : -EACCES ; +pte_t pte =huge_ptep_get (hpte->ptep); + +if (!hugetlb_pte_present_leaf (hpte, pte)) +return -EAGAIN ; +return pfn_modify_allowed (pte_pfn (pte), + *(pgprot_t *)(walk->private)) ?0 : -EACCES ; } static int prot_none_test(unsigned long addr, unsigned long next,
@@ -572,8 +576,10 @@ mprotect_fixup(struct mmu_gather *tlb, struct vm_area_struct *vma, (newflags & VM_ACCESS_FLAGS) == 0) {pgprot_t new_pgprot =vm_get_page_prot (newflags); - error =walk_page_range (current->mm, start, end, - &prot_none_walk_ops, &new_pgprot); +do { + error =walk_page_range (current->mm, start, end, + &prot_none_walk_ops, &new_pgprot); + }while (error == -EAGAIN );if (error)return error; }
mm/pagewalk.c +26 -6
@@ -3,6 +3,7 @@#include <linux/highmem.h> #include <linux/sched.h> #include <linux/hugetlb.h> +#include <linux/minmax.h> /* * We want to know the real level where a entry is located ignoring any
@@ -301,20 +302,39 @@ static int walk_hugetlb_range(unsigned long addr, unsigned long end,pte_t *pte;const struct mm_walk_ops *ops = walk->ops;int err =0 ; +struct hugetlb_pte hpte; + +if (hugetlb_hgm_enabled (vma)) +/* +* We could potentially do high-granularity walks. Grab the +* VMA lock to prevent PTEs from becoming invalid. +*/ +hugetlb_vma_lock_read (vma);do { - next =hugetlb_entry_end (h, addr, end); pte =huge_pte_offset (walk->mm, addr & hmask, sz); - -if (pte) - err = ops->hugetlb_entry (pte, hmask, addr, next, walk); -else if (ops->pte_hole) - err = ops->pte_hole (addr, next,-1 , walk); +if (!pte) { + next =hugetlb_entry_end (h, addr, end); +if (ops->pte_hole) + err = ops->pte_hole (addr, next,-1 , walk); + }else { +hugetlb_pte_populate (&hpte, pte,huge_page_shift (h), +hpage_size_to_level (sz)); +hugetlb_hgm_walk (walk->mm, vma, &hpte, addr, +PAGE_SIZE , +/*stop_at_none=*/ true); + err = ops->hugetlb_entry ( + &hpte, addr, walk); + next =min (addr +hugetlb_pte_size (&hpte), end); + }if (err)break ; }while (addr = next, addr != end); +if (hugetlb_hgm_enabled (vma)) +hugetlb_vma_unlock_read (vma); +return err; }
Signature · 1 line
2.38.0.135.g90850a2211-goog
RFC PATCH v2 21/47 mm: rmap: provide pte_order in page_vma_mapped_walk
In reply to James Houghtonpage_vma_mapped_walk callers will need this information to know how HugeTLB pages are mapped. pte_order only applies if pte is not NULL.
--- include/linux/rmap.h | 1 + mm/page_vma_mapped.c | 3 +++ 2 files changed, 4 insertions(+)
include/linux/rmap.h +1 -0
@@ -378,6 +378,7 @@ struct page_vma_mapped_walk {pmd_t *pmd;pte_t *pte;spinlock_t *ptl; +unsigned int pte_order;unsigned int flags; };
mm/page_vma_mapped.c +3 -0
@@ -16,6 +16,7 @@ static inline bool not_found(struct page_vma_mapped_walk *pvmw)static bool map_pte (struct page_vma_mapped_walk *pvmw) { pvmw->pte =pte_offset_map (pvmw->pmd, pvmw->address); + pvmw->pte_order =0 ;if (!(pvmw->flags &PVMW_SYNC )) {if (pvmw->flags &PVMW_MIGRATION ) {if (!is_swap_pte (*pvmw->pte))
@@ -174,6 +175,7 @@ bool page_vma_mapped_walk(struct page_vma_mapped_walk *pvmw)if (!pvmw->pte)return false; + pvmw->pte_order =huge_page_order (hstate); pvmw->ptl =huge_pte_lock (hstate, mm, pvmw->pte);if (!check_pte (pvmw))return not_found (pvmw);
@@ -269,6 +271,7 @@ bool page_vma_mapped_walk(struct page_vma_mapped_walk *pvmw) }pte_unmap (pvmw->pte); pvmw->pte =NULL ; + pvmw->pte_order =0 ; goto restart; } pvmw->pte++;
Signature · 1 line
2.38.0.135.g90850a2211-goog
RFC PATCH v2 22/47 mm: rmap: make page_vma_mapped_walk callers use pte_order
In reply to James HoughtonThis also updates the callers' hugetlb mapcounting code to handle mapcount properly for subpage-mapped hugetlb pages.
--- mm/migrate.c | 2 +- mm/rmap.c | 17 +++++++++++++---- 2 files changed, 14 insertions(+), 5 deletions(-)
mm/migrate.c +1 -1
@@ -235,7 +235,7 @@ static bool remove_migration_pte(struct folio *folio,#ifdef CONFIG_HUGETLB_PAGE if (folio_test_hugetlb (folio)) { -unsigned int shift =huge_page_shift (hstate_vma (vma)); +unsigned int shift = pvmw.pte_order +PAGE_SHIFT ; pte =arch_make_huge_pte (pte, shift, vma->vm_flags);if (folio_test_anon (folio))
mm/rmap.c +13 -4
@@ -1626,7 +1626,7 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma,if (PageHWPoison (subpage) && !(flags &TTU_IGNORE_HWPOISON )) { pteval =swp_entry_to_pte (make_hwpoison_entry (subpage));if (folio_test_hugetlb (folio)) { -hugetlb_count_sub (folio_nr_pages (folio), mm); +hugetlb_count_sub (1UL << pvmw.pte_order, mm);set_huge_pte_at (mm, address, pvmw.pte, pteval); }else {dec_mm_counter (mm,mm_counter (&folio->page));
@@ -1785,7 +1785,11 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma,* * See Documentation/mm/mmu_notifier.rst */ -page_remove_rmap (subpage, vma,folio_test_hugetlb (folio)); +if (folio_test_hugetlb (folio)) +page_remove_rmap (&folio->page, vma, true); +else +page_remove_rmap (subpage, vma, false); +if (vma->vm_flags &VM_LOCKED )mlock_page_drain_local ();folio_put (folio);
@@ -2034,7 +2038,7 @@ static bool try_to_migrate_one(struct folio *folio, struct vm_area_struct *vma, }else if (PageHWPoison (subpage)) { pteval =swp_entry_to_pte (make_hwpoison_entry (subpage));if (folio_test_hugetlb (folio)) { -hugetlb_count_sub (folio_nr_pages (folio), mm); +hugetlb_count_sub (1L << pvmw.pte_order, mm);set_huge_pte_at (mm, address, pvmw.pte, pteval); }else {dec_mm_counter (mm,mm_counter (&folio->page));
@@ -2126,7 +2130,10 @@ static bool try_to_migrate_one(struct folio *folio, struct vm_area_struct *vma,* * See Documentation/mm/mmu_notifier.rst */ -page_remove_rmap (subpage, vma,folio_test_hugetlb (folio)); +if (folio_test_hugetlb (folio)) +page_remove_rmap (&folio->page, vma, true); +else +page_remove_rmap (subpage, vma, false);if (vma->vm_flags &VM_LOCKED )mlock_page_drain_local ();folio_put (folio);
@@ -2210,6 +2217,8 @@ static bool page_make_device_exclusive_one(struct folio *folio, args->owner);mmu_notifier_invalidate_range_start (&range); +VM_BUG_ON_FOLIO (folio_test_hugetlb (folio), folio); +while (page_vma_mapped_walk (&pvmw)) {/* Unexpected PMD-mapped THP? */ VM_BUG_ON_FOLIO (!pvmw.pte, folio);
Signature · 1 line
2.38.0.135.g90850a2211-goog
RFC PATCH v2 23/47 rmap: update hugetlb lock comment for HGM
In reply to James HoughtonThe VMA lock is used to prevent high-granularity HugeTLB mappings from being collapsed while other threads are doing high-granularity page table walks.
--- mm/rmap.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
mm/rmap.c +2 -1
@@ -47,7 +47,8 @@* * hugetlbfs PageHuge() take locks in this order: * hugetlb_fault_mutex (hugetlbfs specific page fault mutex) -* vma_lock (hugetlb specific lock for pmd_sharing) +* vma_lock (hugetlb specific lock for pmd_sharing and high-granularity +* mapping) * mapping->i_mmap_rwsem (also used for hugetlb pmd sharing) * page->flags PG_locked (lock_page) */
Signature · 1 line
2.38.0.135.g90850a2211-goog
RFC PATCH v2 24/47 hugetlb: update page_vma_mapped to do high-granularity walks
In reply to James HoughtonThis updates the HugeTLB logic to look a lot more like the PTE-mapped THP logic. When a user calls us in a loop, we will update pvmw->address to walk to each page table entry that could possibly map the hugepage containing pvmw->pfn.
This makes use of the new pte_order so callers know what size PTE they're getting.
--- include/linux/rmap.h | 4 +++ mm/page_vma_mapped.c | 59 ++++++++++++++++++++++++++++++++++++-------- mm/rmap.c | 48 +++++++++++++++++++++-------------- 3 files changed, 83 insertions(+), 28 deletions(-)
include/linux/rmap.h +4 -0
@@ -13,6 +13,7 @@#include <linux/highmem.h> #include <linux/pagemap.h> #include <linux/memremap.h> +#include <linux/hugetlb.h> /* * The anon_vma heads a list of private "related" vmas, to scan if
@@ -409,6 +410,9 @@ static inline void page_vma_mapped_walk_done(struct page_vma_mapped_walk *pvmw)pte_unmap (pvmw->pte);if (pvmw->ptl)spin_unlock (pvmw->ptl); +if (pvmw->pte &&is_vm_hugetlb_page (pvmw->vma) && +hugetlb_hgm_enabled (pvmw->vma)) +hugetlb_vma_unlock_read (pvmw->vma); }bool page_vma_mapped_walk (struct page_vma_mapped_walk *pvmw);
mm/page_vma_mapped.c +49 -10
75 lines not on this page · raw message
mm/rmap.c +30 -18
64 lines not on this page · raw message
Signature · 1 line
2.38.0.135.g90850a2211-goog
Re: [RFC PATCH v2 24/47] hugetlb: update page_vma_mapped to do high-granularity walks
Quoted 87 lines · James Houghton
This updates the HugeTLB logic to look a lot more like the PTE-mapped THP logic. When a user calls us in a loop, we will update
pvmw->addressto walk to each page table entry that could possibly map the hugepage containingpvmw->pfn.This makes use of the new
pte_orderso callers know what size PTE they're getting.Signed-off-by James Houghton <[email protected]>--- include/linux/rmap.h | 4 +++ mm/page_vma_mapped.c | 59 ++++++++++++++++++++++++++++++++++++-------- mm/rmap.c | 48 +++++++++++++++++++++-------------- 3 files changed, 83 insertions(+), 28 deletions(-)include/linux/rmap.h +4
-0@@ -13,6 +13,7 @@#include <linux/highmem.h> #include <linux/pagemap.h> #include <linux/memremap.h> +#include <linux/hugetlb.h> /* * The anon_vma heads a list of private "related" vmas, to scan if @@ -409,6 +410,9 @@ static inline void page_vma_mapped_walk_done(struct page_vma_mapped_walk *pvmw)pte_unmap (pvmw->pte);if (pvmw->ptl)spin_unlock (pvmw->ptl); +if (pvmw->pte &&is_vm_hugetlb_page (pvmw->vma) && +hugetlb_hgm_enabled (pvmw->vma)) +hugetlb_vma_unlock_read (pvmw->vma); }bool page_vma_mapped_walk (struct page_vma_mapped_walk *pvmw);mm/page_vma_mapped.c +20
-10@@ -133,7 +133,8 @@ static void step_forward(struct page_vma_mapped_walk *pvmw, unsigned long size)* * Returns true if the page is mapped in the vma. @pvmw->pmd and @pvmw->pte point * to relevant page table entries. @pvmw->ptl is locked. @pvmw->address is -* adjusted if needed (for PTE-mapped THPs). +* adjusted if needed (for PTE-mapped THPs and high-granularity--mapped HugeTLB +* pages). * * If @pvmw->pmd is set but @pvmw->pte is not, you have found PMD-mapped page * (usually THP). For PTE-mapped THP, you should run page_vma_mapped_walk() in @@ -166,19 +167,57 @@ bool page_vma_mapped_walk(struct page_vma_mapped_walk *pvmw)if (unlikely (is_vm_hugetlb_page (vma))) {struct hstate *hstate =hstate_vma (vma);unsigned long size =huge_page_size (hstate); -/* The only possible mapping was handled on last iteration */ -if (pvmw->pte) -return not_found (pvmw); +struct hugetlb_pte hpte; +pte_t *pte; +pte_t pteval; + + end = (pvmw->address &huge_page_mask (hstate)) + +huge_page_size (hstate);/* when pud is not present, pte will be NULL */ - pvmw->pte =huge_pte_offset (mm, pvmw->address, size); -if (!pvmw->pte) + pte =huge_pte_offset (mm, pvmw->address, size); +if (!pte)return false; - pvmw->pte_order =huge_page_order (hstate); - pvmw->ptl =huge_pte_lock (hstate, mm, pvmw->pte); -if (!check_pte (pvmw)) -return not_found (pvmw); + do { + hugetlb_pte_populate(&hpte, pte, huge_page_shift(hstate), + hpage_size_to_level(size)); + + /* + * Do a high granularity page table walk. The vma lock + * is grabbed to prevent the page table from being + * collapsed mid-walk. It is dropped in + * page_vma_mapped_walk_done(). + */
mm/page_vma_mapped.c bool page_vma_mapped_walk(struct page_vma_mapped_walk *pvmw) · line 201
Quoted 10 lines
mm/page_vma_mapped.c +10
-0+if (pvmw->pte) { +if (pvmw->ptl) +spin_unlock (pvmw->ptl); + pvmw->ptl =NULL ; + pvmw->address +=PAGE_SIZE << pvmw->pte_order; +if (pvmw->address >= end) +return not_found (pvmw); + }else if (hugetlb_hgm_enabled (vma)) +/* Only grab the lock once. */ +hugetlb_vma_lock_read (vma);
I realize that I can't do this -- we're already holding the i_mmap_rwsem, and we have to take the VMA lock first. It seems like we're always holding it for writing in this case, so if I make hugetlb_collapse taking the i_mmap_rwsem for reading, this will be safe.
Peter, you looked at this recently [1] -- do you know if we're always holding i_mmap_rwsem *for writing* here?
[1] https://lore.kernel.org/linux-mm/[email protected]/
Thanks!
- James
Quoted 95 lines
mm/page_vma_mapped.c +19
-0+ +retry_walk: +hugetlb_hgm_walk (mm, vma, &hpte, pvmw->address, +PAGE_SIZE ,/*stop_at_none=*/ true); + + pvmw->pte = hpte.ptep; + pvmw->pte_order = hpte.shift -PAGE_SHIFT ; + pvmw->ptl =hugetlb_pte_lock (mm, &hpte); + pteval =huge_ptep_get (hpte.ptep); +if (pte_present (pteval) && !hugetlb_pte_present_leaf ( + &hpte, pteval)) { +/* +* Someone split from under us, so keep +* walking. +*/ +spin_unlock (pvmw->ptl); + goto retry_walk; + } + }while (!check_pte (pvmw));return true; }mm/rmap.c +30
-19@@ -1552,17 +1552,23 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma,flush_cache_range (vma, range.start, range.end);/* -* To call huge_pmd_unshare, i_mmap_rwsem must be -* held in write mode. Caller needs to explicitly -* do this outside rmap routines. -* -* We also must hold hugetlb vma_lock in write mode. -* Lock order dictates acquiring vma_lock BEFORE -* i_mmap_rwsem. We can only try lock here and fail -* if unsuccessful. +* If HGM is enabled, we have already grabbed the VMA +* lock for reading, and we cannot safely release it. +* Because HGM-enabled VMAs have already unshared all +* PMDs, we can safely ignore PMD unsharing here. */ -if (!anon) { +if (!anon && !hugetlb_hgm_enabled (vma)) {VM_BUG_ON (!(flags &TTU_RMAP_LOCKED )); +/* +* To call huge_pmd_unshare, i_mmap_rwsem must +* be held in write mode. Caller needs to +* explicitly do this outside rmap routines. +* +* We also must hold hugetlb vma_lock in write +* mode. Lock order dictates acquiring vma_lock +* BEFORE i_mmap_rwsem. We can only try lock +* here and fail if unsuccessful. +*/ if (!hugetlb_vma_trylock_write (vma)) {page_vma_mapped_walk_done (&pvmw); ret = false;@@ -1946,17 +1952,23 @@ static bool try_to_migrate_one(struct folio *folio, struct vm_area_struct *vma,flush_cache_range (vma, range.start, range.end);/* -* To call huge_pmd_unshare, i_mmap_rwsem must be -* held in write mode. Caller needs to explicitly -* do this outside rmap routines. -* -* We also must hold hugetlb vma_lock in write mode. -* Lock order dictates acquiring vma_lock BEFORE -* i_mmap_rwsem. We can only try lock here and -* fail if unsuccessful. +* If HGM is enabled, we have already grabbed the VMA +* lock for reading, and we cannot safely release it. +* Because HGM-enabled VMAs have already unshared all +* PMDs, we can safely ignore PMD unsharing here. */ -if (!anon) { +if (!anon && !hugetlb_hgm_enabled (vma)) {VM_BUG_ON (!(flags &TTU_RMAP_LOCKED )); +/* +* To call huge_pmd_unshare, i_mmap_rwsem must +* be held in write mode. Caller needs to +* explicitly do this outside rmap routines. +* +* We also must hold hugetlb vma_lock in write +* mode. Lock order dictates acquiring vma_lock +* BEFORE i_mmap_rwsem. We can only try lock +* here and fail if unsuccessful. +*/ if (!hugetlb_vma_trylock_write (vma)) {page_vma_mapped_walk_done (&pvmw); ret = false; --2.38.0.135.g90850a2211-goog
Peter Xu
reviewerRe: [RFC PATCH v2 24/47] hugetlb: update page_vma_mapped to do high-granularity walks
Hi, James,
Quoted 1 line · James Houghton
Quoted 1 line
(quoted diff) +0
-0@@ -166,19 +167,57 @@ bool page_vma_mapped_walk(struct page_vma_mapped_walk *pvmw)
[...]
Quoted 10 lines
I realize that I can't do this -- we're already holding the
i_mmap_rwsem, and we have to take the VMA lock first. It seems like we're always holding it for writing in this case, so if I makehugetlb_collapsetaking thei_mmap_rwsemfor reading, this will be safe.Peter, you looked at this recently [1] -- do you know if we're always holding
i_mmap_rwsem*for writing* here?
I think so, an analysis is in previous v2 in one of my reply to John:
https://lore.kernel.org/all/Y5JjTPTxCWSklCan@x1n/
No hurt to double check, though.
Signature · 1 line
Peter Xu
RFC PATCH v2 25/47 hugetlb: add HGM support for copy_hugetlb_page_range
In reply to James HoughtonThis allows fork() to work with high-granularity mappings. The page table structure is copied such that partially mapped regions will remain partially mapped in the same way for the new process.
A page's reference count is incremented for *each* portion of it that is mapped in the page table. For example, if you have a PMD-mapped 1G page, the reference count and mapcount will be incremented by 512.
--- mm/hugetlb.c | 81 +++++++++++++++++++++++++++++++++++++++++----------- 1 file changed, 65 insertions(+), 16 deletions(-)
mm/hugetlb.c +65 -16
@@ -4946,7 +4946,8 @@ int copy_hugetlb_page_range(struct mm_struct *dst, struct mm_struct *src, struct vm_area_struct *src_vma) {pte_t *src_pte, *dst_pte, entry; -struct page *ptepage; +struct hugetlb_pte src_hpte, dst_hpte; +struct page *ptepage, *hpage;unsigned long addr;bool cow =is_cow_mapping (src_vma->vm_flags);struct hstate *h =hstate_vma (src_vma);
@@ -4956,6 +4957,16 @@ int copy_hugetlb_page_range(struct mm_struct *dst, struct mm_struct *src,unsigned long last_addr_mask;int ret =0 ; +if (hugetlb_hgm_enabled (src_vma)) { +/* +* src_vma might have high-granularity PTEs, and dst_vma will +* need to copy those. +*/ + ret =enable_hugetlb_hgm (dst_vma); +if (ret) +return ret; + } +if (cow) { mmu_notifier_range_init(&range, MMU_NOTIFY_CLEAR, 0, src_vma, src, src_vma->vm_start,
@@ -4967,18 +4978,22 @@ int copy_hugetlb_page_range(struct mm_struct *dst, struct mm_struct *src,/* * For shared mappings the vma lock must be held before * calling huge_pte_offset in the src vma. Otherwise, the -* returned ptep could go away if part of a shared pmd and -* another thread calls huge_pmd_unshare. +* returned ptep could go away if +* - part of a shared pmd and another thread calls +* huge_pmd_unshare, or +* - another thread collapses a high-granularity mapping. */ hugetlb_vma_lock_read (src_vma); } last_addr_mask =hugetlb_mask_last_page (h); -for (addr = src_vma->vm_start; addr < src_vma->vm_end; addr += sz) { + addr = src_vma->vm_start; +while (addr < src_vma->vm_end) {spinlock_t *src_ptl, *dst_ptl; +unsigned long hpte_sz; src_pte =huge_pte_offset (src, addr, sz);if (!src_pte) { - addr |= last_addr_mask; + addr = (addr | last_addr_mask) + sz;continue ; } dst_pte =huge_pte_alloc (dst, dst_vma, addr, sz);
@@ -4987,6 +5002,26 @@ int copy_hugetlb_page_range(struct mm_struct *dst, struct mm_struct *src,break ; } +hugetlb_pte_populate (&src_hpte, src_pte,huge_page_shift (h), +hpage_size_to_level (huge_page_size (h))); +hugetlb_pte_populate (&dst_hpte, dst_pte,huge_page_shift (h), +hpage_size_to_level (huge_page_size (h))); + +if (hugetlb_hgm_enabled (src_vma)) { +hugetlb_hgm_walk (src, src_vma, &src_hpte, addr, +PAGE_SIZE ,/*stop_at_none=*/ true); + ret =hugetlb_hgm_walk (dst, dst_vma, &dst_hpte, addr, +hugetlb_pte_size (&src_hpte), +/*stop_at_none=*/ false); +if (ret) +break ; + + src_pte = src_hpte.ptep; + dst_pte = dst_hpte.ptep; + } + + hpte_sz =hugetlb_pte_size (&src_hpte); +/* * If the pagetables are shared don't copy or take references. *
@@ -4996,12 +5031,12 @@ int copy_hugetlb_page_range(struct mm_struct *dst, struct mm_struct *src,* to reliably determine whether pte is shared. */ if (page_count (virt_to_page (dst_pte)) >1 ) { - addr |= last_addr_mask; + addr = (addr | last_addr_mask) + sz;continue ; } - dst_ptl =huge_pte_lock (h, dst, dst_pte); - src_ptl =huge_pte_lockptr (huge_page_shift (h), src, src_pte); + dst_ptl =hugetlb_pte_lock (dst, &dst_hpte); + src_ptl =hugetlb_pte_lockptr (src, &src_hpte);spin_lock_nested (src_ptl,SINGLE_DEPTH_NESTING ); entry =huge_ptep_get (src_pte); again:
@@ -5042,10 +5077,15 @@ int copy_hugetlb_page_range(struct mm_struct *dst, struct mm_struct *src,*/ if (userfaultfd_wp (dst_vma))set_huge_pte_at (dst, addr, dst_pte, entry); + }else if (!hugetlb_pte_present_leaf (&src_hpte, entry)) { +/* Retry the walk. */ +spin_unlock (src_ptl); +spin_unlock (dst_ptl); +continue ; }else { - entry =huge_ptep_get (src_pte); ptepage =pte_page (entry); -get_page (ptepage); + hpage =compound_head (ptepage); +get_page (hpage);/* * Failing to duplicate the anon rmap is a rare case
@@ -5058,24 +5098,29 @@ int copy_hugetlb_page_range(struct mm_struct *dst, struct mm_struct *src,* sleep during the process. */ if (!PageAnon (ptepage)) { -page_dup_file_rmap (ptepage, true); - }else if (page_try_dup_anon_rmap (ptepage, true, +page_dup_file_rmap (hpage, true); + }else if (page_try_dup_anon_rmap (hpage, true, src_vma)) {pte_t src_pte_old = entry;struct page *new; +if (hugetlb_hgm_enabled (src_vma)) { + ret = -EINVAL ; +break ; + } +spin_unlock (src_ptl);spin_unlock (dst_ptl);/* Do not use reserve as it's private owned */ new =alloc_huge_page (dst_vma, addr,1 );if (IS_ERR (new)) { -put_page (ptepage); +put_page (hpage); ret =PTR_ERR (new);break ; } -copy_user_huge_page (new, ptepage, addr, dst_vma, +copy_user_huge_page (new, hpage, addr, dst_vma, npages); -put_page (ptepage); +put_page (hpage);/* Install the new huge page if src pte stable */ dst_ptl =huge_pte_lock (h, dst, dst_pte);
@@ -5093,6 +5138,7 @@ int copy_hugetlb_page_range(struct mm_struct *dst, struct mm_struct *src,hugetlb_install_page (dst_vma, dst_pte, addr, new);spin_unlock (src_ptl);spin_unlock (dst_ptl); + addr +=hugetlb_pte_size (&src_hpte);continue ; }
@@ -5109,10 +5155,13 @@ int copy_hugetlb_page_range(struct mm_struct *dst, struct mm_struct *src, }set_huge_pte_at (dst, addr, dst_pte, entry); -hugetlb_count_add (npages, dst); +hugetlb_count_add ( +hugetlb_pte_size (&dst_hpte) /PAGE_SIZE , + dst); }spin_unlock (src_ptl);spin_unlock (dst_ptl); + addr +=hugetlb_pte_size (&src_hpte); }if (cow) {
Signature · 1 line
2.38.0.135.g90850a2211-goog
Peter Xu
reviewerRe: [RFC PATCH v2 25/47] hugetlb: add HGM support for copy_hugetlb_page_range
Quoted 9 lines · James Houghton
This allows
fork()to work with high-granularity mappings. The page table structure is copied such that partially mapped regions will remain partially mapped in the same way for the new process.A page's reference count is incremented for *each* portion of it that is mapped in the page table. For example, if you have a PMD-mapped 1G page, the reference count and mapcount will be incremented by 512.
Signed-off-by James Houghton <[email protected]>
I have a feeling that this path is not triggered. See:
bcd51a3c679d ("hugetlb: lazy page table copies in fork()", 2022-07-17)
It might be helpful to have it when exploring private mapping support of hgm on page poison in the future. But the thing is if we want this to be accepted we still need a way to test it. I just don't see how to test this without the private support being there..
Signature · 1 line
Peter Xu
Re: [RFC PATCH v2 25/47] hugetlb: add HGM support for copy_hugetlb_page_range
Quoted 12 lines · Peter Xu
Quoted 9 lines · James Houghton
This allows
fork()to work with high-granularity mappings. The page table structure is copied such that partially mapped regions will remain partially mapped in the same way for the new process.A page's reference count is incremented for *each* portion of it that is mapped in the page table. For example, if you have a PMD-mapped 1G page, the reference count and mapcount will be incremented by 512.
Signed-off-by James Houghton <[email protected]>
Quoted 8 lines
I have a feeling that this path is not triggered. See:
bcd51a3c679d("hugetlb: lazy page table copies infork()", 2022-07-17)It might be helpful to have it when exploring private mapping support of hgm on page poison in the future. But the thing is if we want this to be accepted we still need a way to test it. I just don't see how to test this without the private support being there..
We can trigger this behavior by registering the VMA with uffd-writeprotect. I didn't include any self-tests for this though; I'll make sure to actually test this path in v1.
- James
Quoted 4 lines
--
Peter Xu
RFC PATCH v2 26/47 hugetlb: make move_hugetlb_page_tables compatible with HGM
In reply to James HoughtonThis is very similar to the support that was added to copy_hugetlb_page_range. We simply do a high-granularity walk now, and most of the rest of the code stays the same.
--- mm/hugetlb.c | 47 ++++++++++++++++++++++++++++++++--------------- 1 file changed, 32 insertions(+), 15 deletions(-)
mm/hugetlb.c +32 -15
@@ -5174,16 +5174,16 @@ int copy_hugetlb_page_range(struct mm_struct *dst, struct mm_struct *src,return ret; } -static void move_huge_pte (struct vm_area_struct *vma,unsigned long old_addr, -unsigned long new_addr,pte_t *src_pte,pte_t *dst_pte) +static void move_hugetlb_pte (struct vm_area_struct *vma,unsigned long old_addr, +unsigned long new_addr,struct hugetlb_pte *src_hpte, +struct hugetlb_pte *dst_hpte) { -struct hstate *h =hstate_vma (vma);struct mm_struct *mm = vma->vm_mm;spinlock_t *src_ptl, *dst_ptl;pte_t pte; - dst_ptl =huge_pte_lock (h, mm, dst_pte); - src_ptl =huge_pte_lockptr (huge_page_shift (h), mm, src_pte); + dst_ptl =hugetlb_pte_lock (mm, dst_hpte); + src_ptl =hugetlb_pte_lockptr (mm, src_hpte);/* * We don't have to worry about the ordering of src and dst ptlocks
@@ -5192,8 +5192,8 @@ static void move_huge_pte(struct vm_area_struct *vma, unsigned long old_addr,if (src_ptl != dst_ptl)spin_lock_nested (src_ptl,SINGLE_DEPTH_NESTING ); - pte =huge_ptep_get_and_clear (mm, old_addr, src_pte); -set_huge_pte_at (mm, new_addr, dst_pte, pte); + pte =huge_ptep_get_and_clear (mm, old_addr, src_hpte->ptep); +set_huge_pte_at (mm, new_addr, dst_hpte->ptep, pte);if (src_ptl != dst_ptl)spin_unlock (src_ptl);
@@ -5214,6 +5214,7 @@ int move_hugetlb_page_tables(struct vm_area_struct *vma,pte_t *src_pte, *dst_pte;struct mmu_notifier_range range;bool shared_pmd = false; +struct hugetlb_pte src_hpte, dst_hpte;mmu_notifier_range_init (&range,MMU_NOTIFY_CLEAR ,0 , vma, mm, old_addr, old_end);
@@ -5229,20 +5230,28 @@ int move_hugetlb_page_tables(struct vm_area_struct *vma,/* Prevent race with file truncation */ hugetlb_vma_lock_write (vma);i_mmap_lock_write (mapping); -for (; old_addr < old_end; old_addr += sz, new_addr += sz) { +while (old_addr < old_end) { src_pte =huge_pte_offset (mm, old_addr, sz);if (!src_pte) { - old_addr |= last_addr_mask; - new_addr |= last_addr_mask; + old_addr = (old_addr | last_addr_mask) + sz; + new_addr = (new_addr | last_addr_mask) + sz;continue ; } -if (huge_pte_none (huge_ptep_get (src_pte))) + +hugetlb_pte_populate (&src_hpte, src_pte,huge_page_shift (h), +hpage_size_to_level (sz)); +hugetlb_hgm_walk (mm, vma, &src_hpte, old_addr, +PAGE_SIZE ,/*stop_at_none=*/ true); +if (huge_pte_none (huge_ptep_get (src_hpte.ptep))) { + old_addr +=hugetlb_pte_size (&src_hpte); + new_addr +=hugetlb_pte_size (&src_hpte);continue ; + } -if (huge_pmd_unshare (mm, vma, old_addr, src_pte)) { +if (huge_pmd_unshare (mm, vma, old_addr, src_hpte.ptep)) { shared_pmd = true; - old_addr |= last_addr_mask; - new_addr |= last_addr_mask; + old_addr = (old_addr | last_addr_mask) + sz; + new_addr = (new_addr | last_addr_mask) + sz;continue ; }
@@ -5250,7 +5259,15 @@ int move_hugetlb_page_tables(struct vm_area_struct *vma,if (!dst_pte)break ; -move_huge_pte (vma, old_addr, new_addr, src_pte, dst_pte); +hugetlb_pte_populate (&dst_hpte, dst_pte,huge_page_shift (h), +hpage_size_to_level (sz)); +if (hugetlb_hgm_walk (mm, vma, &dst_hpte, new_addr, +hugetlb_pte_size (&src_hpte), +/*stop_at_none=*/ false)) +break ; +move_hugetlb_pte (vma, old_addr, new_addr, &src_hpte, &dst_hpte); + old_addr +=hugetlb_pte_size (&src_hpte); + new_addr +=hugetlb_pte_size (&src_hpte); }if (shared_pmd)
Signature · 1 line
2.38.0.135.g90850a2211-goog
RFC PATCH v2 27/47 hugetlb: add HGM support for hugetlb_fault and hugetlb_no_page
In reply to James HoughtonUpdate the page fault handler to support high-granularity page faults. While handling a page fault on a partially-mapped HugeTLB page, if the PTE we find with hugetlb_pte_walk is none, then we will replace it with a leaf-level PTE to map the page. To give some examples:
1. For a completely unmapped 1G page, it will be mapped with a 1G PUD.
2. For a 1G page that has its first 512M mapped, any faults on the unmapped sections will result in 2M PMDs mapping each unmapped 2M section.
3. For a 1G page that has only its first 4K mapped, a page fault on its second 4K section will get a 4K PTE to map it.
Unless high-granularity mappings are created via UFFDIO_CONTINUE, it is impossible for hugetlb_fault to create high-granularity mappings.
This commit does not handle hugetlb_wp right now, and it doesn't handle HugeTLB page migration and swap entries.
--- mm/hugetlb.c | 90 +++++++++++++++++++++++++++++++++++++--------------- 1 file changed, 64 insertions(+), 26 deletions(-)
mm/hugetlb.c +64 -26
@@ -118,6 +118,18 @@ enum hugetlb_level hpage_size_to_level(unsigned long sz)return HUGETLB_LEVEL_PGD ; } +/* +* Find the subpage that corresponds to `addr` in `hpage`. +*/ +static struct page *hugetlb_find_subpage (struct hstate *h,struct page *hpage, +unsigned long addr) +{ +size_t idx = (addr & ~huge_page_mask (h))/PAGE_SIZE ; + +BUG_ON (idx >=pages_per_huge_page (h)); +return &hpage[idx]; +} +static inline bool subpool_is_free (struct hugepage_subpool *spool) {if (spool->count)
@@ -5810,13 +5822,13 @@ static inline vm_fault_t hugetlb_handle_userfault(struct vm_area_struct *vma,* false if pte changed or is changing. */ static bool hugetlb_pte_stable (struct hstate *h,struct mm_struct *mm, -pte_t *ptep,pte_t old_pte) +struct hugetlb_pte *hpte,pte_t old_pte) {spinlock_t *ptl;bool same; - ptl =huge_pte_lock (h, mm, ptep); - same =pte_same (huge_ptep_get (ptep), old_pte); + ptl =hugetlb_pte_lock (mm, hpte); + same =pte_same (huge_ptep_get (hpte->ptep), old_pte);spin_unlock (ptl);return same;
@@ -5825,17 +5837,18 @@ static bool hugetlb_pte_stable(struct hstate *h, struct mm_struct *mm,static vm_fault_t hugetlb_no_page (struct mm_struct *mm,struct vm_area_struct *vma,struct address_space *mapping,pgoff_t idx, -unsigned long address,pte_t *ptep, +unsigned long address,struct hugetlb_pte *hpte,pte_t old_pte,unsigned int flags) {struct hstate *h =hstate_vma (vma);vm_fault_t ret =VM_FAULT_SIGBUS ;int anon_rmap =0 ;unsigned long size; -struct page *page; +struct page *page, *subpage;pte_t new_pte;spinlock_t *ptl;unsigned long haddr = address &huge_page_mask (h); +unsigned long haddr_hgm = address &hugetlb_pte_mask (hpte);bool new_page, new_pagecache_page = false;u32 hash =hugetlb_fault_mutex_hash (mapping, idx);
@@ -5880,7 +5893,7 @@ static vm_fault_t hugetlb_no_page(struct mm_struct *mm,* never happen on the page after UFFDIO_COPY has * correctly installed the page and returned. */ -if (!hugetlb_pte_stable (h, mm, ptep, old_pte)) { +if (!hugetlb_pte_stable (h, mm, hpte, old_pte)) { ret =0 ; goto out; }
@@ -5904,7 +5917,7 @@ static vm_fault_t hugetlb_no_page(struct mm_struct *mm,* here. Before returning error, get ptl and make * sure there really is no pte entry. */ -if (hugetlb_pte_stable (h, mm, ptep, old_pte)) +if (hugetlb_pte_stable (h, mm, hpte, old_pte)) ret =vmf_error (PTR_ERR (page));else ret =0 ;
@@ -5954,7 +5967,7 @@ static vm_fault_t hugetlb_no_page(struct mm_struct *mm,unlock_page (page);put_page (page);/* See comment in userfaultfd_missing() block above */ -if (!hugetlb_pte_stable (h, mm, ptep, old_pte)) { +if (!hugetlb_pte_stable (h, mm, hpte, old_pte)) { ret =0 ; goto out; }
@@ -5979,10 +5992,10 @@ static vm_fault_t hugetlb_no_page(struct mm_struct *mm,vma_end_reservation (h, vma, haddr); } - ptl =huge_pte_lock (h, mm, ptep); + ptl =hugetlb_pte_lock (mm, hpte); ret =0 ;/* If pte changed from under us, retry */ -if (!pte_same (huge_ptep_get (ptep), old_pte)) +if (!pte_same (huge_ptep_get (hpte->ptep), old_pte)) goto backout;if (anon_rmap) {
@@ -5990,20 +6003,25 @@ static vm_fault_t hugetlb_no_page(struct mm_struct *mm,hugepage_add_new_anon_rmap (page, vma, haddr); }else page_dup_file_rmap (page, true); - new_pte =make_huge_pte (vma, page, ((vma->vm_flags &VM_WRITE ) - && (vma->vm_flags &VM_SHARED ))); + + subpage =hugetlb_find_subpage (h, page, haddr_hgm); + new_pte =make_huge_pte_with_shift (vma, subpage, + ((vma->vm_flags &VM_WRITE ) + && (vma->vm_flags &VM_SHARED )), + hpte->shift);/* * If this pte was previously wr-protected, keep it wr-protected even * if populated. */ if (unlikely (pte_marker_uffd_wp (old_pte))) new_pte =huge_pte_wrprotect (huge_pte_mkuffd_wp (new_pte)); -set_huge_pte_at (mm, haddr, ptep, new_pte); +set_huge_pte_at (mm, haddr_hgm, hpte->ptep, new_pte); -hugetlb_count_add (pages_per_huge_page (h), mm); +hugetlb_count_add (hugetlb_pte_size (hpte) /PAGE_SIZE , mm);if ((flags &FAULT_FLAG_WRITE ) && !(vma->vm_flags &VM_SHARED )) { +BUG_ON (hugetlb_pte_size (hpte) !=huge_page_size (h));/* Optimization, do the COW without a second fault */ - ret =hugetlb_wp (mm, vma, address, ptep, flags, page, ptl); + ret =hugetlb_wp (mm, vma, address, hpte->ptep, flags, page, ptl); }spin_unlock (ptl);
@@ -6066,11 +6084,14 @@ vm_fault_t hugetlb_fault(struct mm_struct *mm, struct vm_area_struct *vma,u32 hash;pgoff_t idx;struct page *page =NULL ; +struct page *subpage =NULL ;struct page *pagecache_page =NULL ;struct hstate *h =hstate_vma (vma);struct address_space *mapping;int need_wait_lock =0 ;unsigned long haddr = address &huge_page_mask (h); +unsigned long haddr_hgm; +struct hugetlb_pte hpte; ptep =huge_pte_offset (mm, haddr,huge_page_size (h));if (ptep) {
@@ -6115,15 +6136,22 @@ vm_fault_t hugetlb_fault(struct mm_struct *mm, struct vm_area_struct *vma,return VM_FAULT_OOM ; } - entry =huge_ptep_get (ptep); +hugetlb_pte_populate (&hpte, ptep,huge_page_shift (h), +hpage_size_to_level (huge_page_size (h))); +/* Do a high-granularity page table walk. */ +hugetlb_hgm_walk (mm, vma, &hpte, address,PAGE_SIZE , +/*stop_at_none=*/ true); + + entry =huge_ptep_get (hpte.ptep);/* PTE markers should be handled the same way as none pte */ -if (huge_pte_none_mostly (entry)) +if (huge_pte_none_mostly (entry)) {/* * hugetlb_no_page will drop vma lock and hugetlb fault * mutex internally, which make us return immediately. */ -return hugetlb_no_page (mm, vma, mapping, idx, address, ptep, +return hugetlb_no_page (mm, vma, mapping, idx, address, &hpte, entry, flags); + } ret =0 ;
@@ -6137,6 +6165,10 @@ vm_fault_t hugetlb_fault(struct mm_struct *mm, struct vm_area_struct *vma,if (!pte_present (entry)) goto out_mutex; +if (!hugetlb_pte_present_leaf (&hpte, entry)) +/* We raced with someone splitting the entry. */ + goto out_mutex; +/* * If we are going to COW/unshare the mapping later, we examine the * pending reservations for this page now. This will ensure that any
@@ -6156,14 +6188,17 @@ vm_fault_t hugetlb_fault(struct mm_struct *mm, struct vm_area_struct *vma, pagecache_page =find_lock_page (mapping, idx); } - ptl =huge_pte_lock (h, mm, ptep); + ptl =hugetlb_pte_lock (mm, &hpte);/* Check for a racing update before calling hugetlb_wp() */ -if (unlikely (!pte_same (entry,huge_ptep_get (ptep)))) +if (unlikely (!pte_same (entry,huge_ptep_get (hpte.ptep)))) goto out_ptl; +/* haddr_hgm is the base address of the region that hpte maps. */ + haddr_hgm = address &hugetlb_pte_mask (&hpte); +/* Handle userfault-wp first, before trying to lock more pages */ -if (userfaultfd_wp (vma) &&huge_pte_uffd_wp (huge_ptep_get (ptep)) && +if (userfaultfd_wp (vma) &&huge_pte_uffd_wp (entry) && (flags &FAULT_FLAG_WRITE ) && !huge_pte_write (entry)) {struct vm_fault vmf = { .vma = vma,
@@ -6187,7 +6222,8 @@ vm_fault_t hugetlb_fault(struct mm_struct *mm, struct vm_area_struct *vma,* pagecache_page, so here we need take the former one * when page != pagecache_page or !pagecache_page. */ - page =pte_page (entry); + subpage =pte_page (entry); + page =compound_head (subpage);if (page != pagecache_page)if (!trylock_page (page)) { need_wait_lock =1 ;
@@ -6198,7 +6234,8 @@ vm_fault_t hugetlb_fault(struct mm_struct *mm, struct vm_area_struct *vma,if (flags & (FAULT_FLAG_WRITE |FAULT_FLAG_UNSHARE )) {if (!huge_pte_write (entry)) { - ret =hugetlb_wp (mm, vma, address, ptep, flags, +BUG_ON (hugetlb_pte_size (&hpte) !=huge_page_size (h)); + ret =hugetlb_wp (mm, vma, address, hpte.ptep, flags, pagecache_page, ptl); goto out_put_page; }else if (likely (flags &FAULT_FLAG_WRITE )) {
@@ -6206,9 +6243,9 @@ vm_fault_t hugetlb_fault(struct mm_struct *mm, struct vm_area_struct *vma, } } entry =pte_mkyoung (entry); -if (huge_ptep_set_access_flags (vma, haddr, ptep, entry, +if (huge_ptep_set_access_flags (vma, haddr_hgm, hpte.ptep, entry, flags &FAULT_FLAG_WRITE )) -update_mmu_cache (vma, haddr, ptep); +update_mmu_cache (vma, haddr_hgm, hpte.ptep); out_put_page:if (page != pagecache_page)unlock_page (page);
@@ -7598,7 +7635,8 @@ pte_t *huge_pte_alloc(struct mm_struct *mm, struct vm_area_struct *vma, pte = (pte_t *)pmd_alloc (mm, pud, addr); } } -BUG_ON (pte &&pte_present (*pte) && !pte_huge (*pte)); +BUG_ON (pte &&pte_present (*pte) && !pte_huge (*pte) && + !hugetlb_hgm_enabled (vma));return pte; }
Signature · 1 line
2.38.0.135.g90850a2211-goog
RFC PATCH v2 28/47 rmap: in try_to_{migrate,unmap}_one, check head page for page flags
In reply to James HoughtonThe main complication here is that HugeTLB pages have their poison status stored in the head page as the HWPoison page flag. Because HugeTLB high-granularity mapping can create PTEs that point to subpages instead of always the head of a hugepage, we need to check the compound_head for page flags.
--- mm/rmap.c | 34 ++++++++++++++++++++++++++-------- 1 file changed, 26 insertions(+), 8 deletions(-)
mm/rmap.c +26 -8
@@ -1474,10 +1474,11 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma,struct mm_struct *mm = vma->vm_mm;DEFINE_FOLIO_VMA_WALK (pvmw, folio, vma, address,0 );pte_t pteval; -struct page *subpage; +struct page *subpage, *page_flags_page;bool anon_exclusive, ret = true;struct mmu_notifier_range range;enum ttu_flags flags = (enum ttu_flags )(long )arg; +bool page_poisoned;/* * When racing against e.g. zap_pte_range() on another cpu,
@@ -1530,9 +1531,17 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma, subpage =folio_page (folio,pte_pfn (*pvmw.pte) -folio_pfn (folio)); +/* +* We check the page flags of HugeTLB pages by checking the +* head page. +*/ + page_flags_page =folio_test_hugetlb (folio) + ? &folio->page + : subpage; + page_poisoned =PageHWPoison (page_flags_page); address = pvmw.address; anon_exclusive =folio_test_anon (folio) && -PageAnonExclusive (subpage); +PageAnonExclusive (page_flags_page);if (folio_test_hugetlb (folio)) {bool anon =folio_test_anon (folio);
@@ -1541,7 +1550,7 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma,* The try_to_unmap() is only passed a hugetlb page * in the case where the hugetlb page is poisoned. */ -VM_BUG_ON_PAGE (!PageHWPoison (subpage), subpage); +VM_BUG_ON_FOLIO (!page_poisoned, folio);/* * huge_pmd_unshare may unmap an entire PMD page. * There is no way of knowing exactly which PMDs may
@@ -1630,7 +1639,7 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma,/* Update high watermark before we lower rss */ update_hiwater_rss (mm); -if (PageHWPoison (subpage) && !(flags &TTU_IGNORE_HWPOISON )) { +if (page_poisoned && !(flags &TTU_IGNORE_HWPOISON )) { pteval =swp_entry_to_pte (make_hwpoison_entry (subpage));if (folio_test_hugetlb (folio)) {hugetlb_count_sub (1UL << pvmw.pte_order, mm);
@@ -1656,7 +1665,9 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma,mmu_notifier_invalidate_range (mm, address, address +PAGE_SIZE ); }else if (folio_test_anon (folio)) { -swp_entry_t entry = { .val =page_private (subpage) }; +swp_entry_t entry = { + .val =page_private (page_flags_page) + };pte_t swp_pte;/* * Store the swap location in the pte.
@@ -1855,7 +1866,7 @@ static bool try_to_migrate_one(struct folio *folio, struct vm_area_struct *vma,struct mm_struct *mm = vma->vm_mm;DEFINE_FOLIO_VMA_WALK (pvmw, folio, vma, address,0 );pte_t pteval; -struct page *subpage; +struct page *subpage, *page_flags_page;bool anon_exclusive, ret = true;struct mmu_notifier_range range;enum ttu_flags flags = (enum ttu_flags )(long )arg;
@@ -1935,9 +1946,16 @@ static bool try_to_migrate_one(struct folio *folio, struct vm_area_struct *vma, subpage =folio_page (folio,pte_pfn (*pvmw.pte) -folio_pfn (folio)); } +/* +* We check the page flags of HugeTLB pages by checking the +* head page. +*/ + page_flags_page =folio_test_hugetlb (folio) + ? &folio->page + : subpage; address = pvmw.address; anon_exclusive =folio_test_anon (folio) && -PageAnonExclusive (subpage); +PageAnonExclusive (page_flags_page);if (folio_test_hugetlb (folio)) {bool anon =folio_test_anon (folio);
@@ -2048,7 +2066,7 @@ static bool try_to_migrate_one(struct folio *folio, struct vm_area_struct *vma,* No need to invalidate here it will synchronize on * against the special swap migration pte. */ - }else if (PageHWPoison (subpage)) { + }else if (PageHWPoison (page_flags_page)) { pteval =swp_entry_to_pte (make_hwpoison_entry (subpage));if (folio_test_hugetlb (folio)) {hugetlb_count_sub (1L << pvmw.pte_order, mm);
Signature · 1 line
2.38.0.135.g90850a2211-goog
RFC PATCH v2 29/47 hugetlb: add high-granularity migration support
In reply to James HoughtonTo prevent queueing a hugepage for migration multiple times, we use last_page to keep track of the last page we saw in queue_pages_hugetlb, and if the page we're looking at is last_page, then we skip it.
For the non-hugetlb cases, last_page, although unused, is still updated so that it has a consistent meaning with the hugetlb case.
This commit adds a check in hugetlb_fault for high-granularity migration PTEs.
--- include/linux/swapops.h | 8 ++++++-- mm/hugetlb.c | 15 ++++++++++++++- mm/mempolicy.c | 24 +++++++++++++++++++----- mm/migrate.c | 18 +++++++++++------- 4 files changed, 50 insertions(+), 15 deletions(-)
include/linux/swapops.h +6 -2
@@ -66,6 +66,8 @@static inline bool is_pfn_swap_entry (swp_entry_t entry); +struct hugetlb_pte ; +/* Clear all flags but only keep swp_entry_t related information */ static inline pte_t pte_swp_clear_flags (pte_t pte) {
@@ -346,7 +348,8 @@ extern void migration_entry_wait(struct mm_struct *mm, pmd_t *pmd, unsigned long address);#ifdef CONFIG_HUGETLB_PAGE extern void __migration_entry_wait_huge (pte_t *ptep,spinlock_t *ptl); -extern void migration_entry_wait_huge (struct vm_area_struct *vma,pte_t *pte); +extern void migration_entry_wait_huge (struct vm_area_struct *vma, +struct hugetlb_pte *hpte);#endif /* CONFIG_HUGETLB_PAGE */ #else /* CONFIG_MIGRATION */ static inline swp_entry_t make_readable_migration_entry (pgoff_t offset)
@@ -375,7 +378,8 @@ static inline void migration_entry_wait(struct mm_struct *mm, pmd_t *pmd, unsigned long address) { }#ifdef CONFIG_HUGETLB_PAGE static inline void __migration_entry_wait_huge (pte_t *ptep,spinlock_t *ptl) { } -static inline void migration_entry_wait_huge (struct vm_area_struct *vma,pte_t *pte) { } +static inline void migration_entry_wait_huge (struct vm_area_struct *vma, +struct hugetlb_pte *hpte) { }#endif /* CONFIG_HUGETLB_PAGE */ static inline int is_writable_migration_entry (swp_entry_t entry) {
mm/hugetlb.c +14 -1
@@ -6100,9 +6100,11 @@ vm_fault_t hugetlb_fault(struct mm_struct *mm, struct vm_area_struct *vma,* OK as we are only making decisions based on content and * not actually modifying content here. */ +hugetlb_pte_populate (&hpte, ptep,huge_page_shift (h), +hpage_size_to_level (huge_page_size (h))); entry =huge_ptep_get (ptep);if (unlikely (is_hugetlb_entry_migration (entry))) { -migration_entry_wait_huge (vma, ptep); +migration_entry_wait_huge (vma, &hpte);return 0 ; }else if (unlikely (is_hugetlb_entry_hwpoisoned (entry))) return VM_FAULT_HWPOISON_LARGE |
@@ -6142,7 +6144,18 @@ vm_fault_t hugetlb_fault(struct mm_struct *mm, struct vm_area_struct *vma,hugetlb_hgm_walk (mm, vma, &hpte, address,PAGE_SIZE ,/*stop_at_none=*/ true); +/* +* Now that we have done a high-granularity walk, check again if we are +* looking at a migration entry. +*/ entry =huge_ptep_get (hpte.ptep); +if (unlikely (is_hugetlb_entry_migration (entry))) { +hugetlb_vma_unlock_read (vma); +mutex_unlock (&hugetlb_fault_mutex_table[hash]); +migration_entry_wait_huge (vma, &hpte); +return 0 ; + } +/* PTE markers should be handled the same way as none pte */ if (huge_pte_none_mostly (entry)) {/*
mm/mempolicy.c +19 -5
@@ -424,6 +424,7 @@ struct queue_pages {unsigned long start;unsigned long end;struct vm_area_struct *first; +struct page *last_page; };/*
@@ -475,6 +476,7 @@ static int queue_pages_pmd(pmd_t *pmd, spinlock_t *ptl, unsigned long addr, flags = qp->flags;/* go to thp migration */ if (flags & (MPOL_MF_MOVE |MPOL_MF_MOVE_ALL )) { + qp->last_page = page;if (!vma_migratable (walk->vma) ||migrate_page_add (page, qp->pagelist, flags)) { ret =1 ;
@@ -532,6 +534,7 @@ static int queue_pages_pte_range(pmd_t *pmd, unsigned long addr,continue ;if (!queue_pages_required (page, qp))continue ; +if (flags & (MPOL_MF_MOVE |MPOL_MF_MOVE_ALL )) {/* MPOL_MF_STRICT must be specified if we get here */ if (!vma_migratable (vma)) {
@@ -539,6 +542,8 @@ static int queue_pages_pte_range(pmd_t *pmd, unsigned long addr,break ; } + qp->last_page = page; +/* * Do not abort immediately since there may be * temporary off LRU pages in the range. Still
@@ -570,15 +575,22 @@ static int queue_pages_hugetlb(struct hugetlb_pte *hpte,spinlock_t *ptl;pte_t entry; -/* We don't migrate high-granularity HugeTLB mappings for now. */ -if (hugetlb_hgm_enabled (walk->vma)) -return -EINVAL ; - ptl =hugetlb_pte_lock (walk->mm, hpte); entry =huge_ptep_get (hpte->ptep);if (!pte_present (entry)) goto unlock; - page =pte_page (entry); + +if (!hugetlb_pte_present_leaf (hpte, entry)) { + ret = -EAGAIN ; + goto unlock; + } + + page =compound_head (pte_page (entry)); + +/* We already queued this page with another high-granularity PTE. */ +if (page == qp->last_page) + goto unlock; +if (!queue_pages_required (page, qp)) goto unlock;
@@ -605,6 +617,7 @@ static int queue_pages_hugetlb(struct hugetlb_pte *hpte,/* With MPOL_MF_MOVE, we migrate only unshared hugepage. */ if (flags & (MPOL_MF_MOVE_ALL ) || (flags &MPOL_MF_MOVE &&page_mapcount (page) ==1 )) { + qp->last_page = page;if (isolate_hugetlb (page, qp->pagelist) && (flags &MPOL_MF_STRICT ))/*
@@ -740,6 +753,7 @@ queue_pages_range(struct mm_struct *mm, unsigned long start, unsigned long end, .start = start, .end = end, .first = NULL, + .last_page = NULL, }; err =walk_page_range (mm, start, end, &queue_pages_walk_ops, &qp);
mm/migrate.c +11 -7
@@ -186,6 +186,9 @@ static bool remove_migration_pte(struct folio *folio,/* pgoff is invalid for ksm pages, but they are never large */ if (folio_test_large (folio) && !folio_test_hugetlb (folio)) idx =linear_page_index (vma, pvmw.address) - pvmw.pgoff; +else if (folio_test_hugetlb (folio)) + idx = (pvmw.address & ~huge_page_mask (hstate_vma (vma)))/ +PAGE_SIZE ; new =folio_page (folio, idx);#ifdef CONFIG_ARCH_ENABLE_THP_MIGRATION
@@ -235,14 +238,15 @@ static bool remove_migration_pte(struct folio *folio,#ifdef CONFIG_HUGETLB_PAGE if (folio_test_hugetlb (folio)) { +struct page *hpage =folio_page (folio,0 );unsigned int shift = pvmw.pte_order +PAGE_SHIFT ; pte =arch_make_huge_pte (pte, shift, vma->vm_flags);if (folio_test_anon (folio)) -hugepage_add_anon_rmap (new, vma, pvmw.address, +hugepage_add_anon_rmap (hpage, vma, pvmw.address, rmap_flags);else -page_dup_file_rmap (new, true); +page_dup_file_rmap (hpage, true);set_huge_pte_at (vma->vm_mm, pvmw.address, pvmw.pte, pte); } else#endif
@@ -258,7 +262,7 @@ static bool remove_migration_pte(struct folio *folio,mlock_page_drain_local ();trace_remove_migration_pte (pvmw.address,pte_val (pte), -compound_order (new)); + pvmw.pte_order);/* No need to invalidate - it was non-present before */ update_mmu_cache (vma, pvmw.address, pvmw.pte);
@@ -332,12 +336,12 @@ void __migration_entry_wait_huge(pte_t *ptep, spinlock_t *ptl) migration_entry_wait_on_locked(pte_to_swp_entry(pte), NULL, ptl); } -void migration_entry_wait_huge (struct vm_area_struct *vma,pte_t *pte) +void migration_entry_wait_huge (struct vm_area_struct *vma, +struct hugetlb_pte *hpte) { -spinlock_t *ptl =huge_pte_lockptr (huge_page_shift (hstate_vma (vma)), - vma->vm_mm, pte); +spinlock_t *ptl =hugetlb_pte_lockptr (vma->vm_mm, hpte); -__migration_entry_wait_huge (pte, ptl); +__migration_entry_wait_huge (hpte->ptep, ptl); }#endif
Signature · 1 line
2.38.0.135.g90850a2211-goog
RFC PATCH v2 30/47 hugetlb: add high-granularity check for hwpoison in fault path
In reply to James HoughtonBecause hwpoison swap entries may be placed beneath the hstate-level PTE, we need to check for it separately (on top of the hstate-level PTE check that remains).
--- mm/hugetlb.c | 5 +++++ 1 file changed, 5 insertions(+)
mm/hugetlb.c +5 -0
@@ -6154,6 +6154,11 @@ vm_fault_t hugetlb_fault(struct mm_struct *mm, struct vm_area_struct *vma,mutex_unlock (&hugetlb_fault_mutex_table[hash]);migration_entry_wait_huge (vma, &hpte);return 0 ; + }else if (unlikely (is_hugetlb_entry_hwpoisoned (entry))) { +hugetlb_vma_unlock_read (vma); +mutex_unlock (&hugetlb_fault_mutex_table[hash]); +return VM_FAULT_HWPOISON_LARGE | +VM_FAULT_SET_HINDEX (hstate_index (h)); }/* PTE markers should be handled the same way as none pte */
Signature · 1 line
2.38.0.135.g90850a2211-goog
RFC PATCH v2 31/47 hugetlb: sort hstates in hugetlb_init_hstates
In reply to James HoughtonWhen using HugeTLB high-granularity mapping, we need to go through the supported hugepage sizes in decreasing order so that we pick the largest size that works. Consider the case where we're faulting in a 1G hugepage for the first time: we want hugetlb_fault/hugetlb_no_page to map it with a PUD. By going through the sizes in decreasing order, we will find that PUD_SIZE works before finding out that PMD_SIZE or PAGE_SIZE work too.
This commit also changes bootmem hugepages from storing hstate pointers directly to storing the hstate sizes. The hstate pointers used for boot-time-allocated hugepages become invalid after we sort the hstates. gather_bootmem_prealloc, called after the hstates have been sorted, now converts the size to the correct hstate.
--- include/linux/hugetlb.h | 2 +- mm/hugetlb.c | 49 ++++++++++++++++++++++++++++++++--------- 2 files changed, 40 insertions(+), 11 deletions(-)
include/linux/hugetlb.h +1 -1
@@ -772,7 +772,7 @@ struct hstate {struct huge_bootmem_page {struct list_head list; -struct hstate *hstate; +unsigned long hstate_sz; };int isolate_or_dissolve_huge_page (struct page *page,struct list_head *list);
mm/hugetlb.c +39 -10
87 lines not on this page · raw message
Signature · 1 line
2.38.0.135.g90850a2211-goog
RFC PATCH v2 32/47 hugetlb: add for_each_hgm_shift
In reply to James HoughtonThis is a helper macro to loop through all the usable page sizes for a high-granularity-enabled HugeTLB VMA. Given the VMA's hstate, it will loop, in descending order, through the page sizes that HugeTLB supports for this architecture. It always includes PAGE_SIZE.
This is done by looping through the hstates; however, there is no hstate for PAGE_SIZE. To handle this case, the loop intentionally goes out of bounds, and the out-of-bounds pointer is mapped to PAGE_SIZE.
--- mm/hugetlb.c | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+)
mm/hugetlb.c +19 -0
@@ -7856,6 +7856,25 @@ int enable_hugetlb_hgm(struct vm_area_struct *vma)hugetlb_unshare_all_pmds (vma); return 0; } + +/* Should only be used by the for_each_hgm_shift macro. */ +static unsigned int __shift_for_hstate (struct hstate *h) +{ +/* If h is out of bounds, we have reached the end, so give PAGE_SIZE */ +if (h >= &hstates[hugetlb_max_hstate]) +return PAGE_SHIFT ; +return huge_page_shift (h); +} + +/* +* Intentionally go out of bounds. An out-of-bounds hstate will be converted to +* PAGE_SIZE. +*/ +#define for_each_hgm_shift (hstate, tmp_h, shift) \ + for ((tmp_h) = hstate; (shift) = __shift_for_hstate(tmp_h), \ + (tmp_h) <= &hstates[hugetlb_max_hstate]; \ + (tmp_h)++) +#endif /* CONFIG_HUGETLB_HIGH_GRANULARITY_MAPPING */ /*
Signature · 1 line
2.38.0.135.g90850a2211-goog
RFC PATCH v2 33/47 userfaultfd: add UFFD_FEATURE_MINOR_HUGETLBFS_HGM
In reply to James HoughtonUserspace must provide this new feature when it calls UFFDIO_API to enable HGM. Userspace can check if the feature exists in uffdio_api.features, and if it does not exist, the kernel does not support and therefore did not enable HGM.
--- fs/userfaultfd.c | 12 +++++++++++- include/linux/userfaultfd_k.h | 7 +++++++ include/uapi/linux/userfaultfd.h | 2 ++ 3 files changed, 20 insertions(+), 1 deletion(-)
fs/userfaultfd.c +11 -1
@@ -226,6 +226,11 @@ static inline struct uffd_msg userfault_msg(unsigned long address,return msg; } +bool uffd_ctx_has_hgm (struct vm_userfaultfd_ctx *ctx) +{ +return ctx->ctx->features &UFFD_FEATURE_MINOR_HUGETLBFS_HGM ; +} +#ifdef CONFIG_HUGETLB_PAGE /* * Same functionality as userfaultfd_must_wait below with modifications for
@@ -1954,10 +1959,15 @@ static int userfaultfd_api(struct userfaultfd_ctx *ctx, goto err_out;/* report all available features and ioctls to userland */ uffdio_api.features =UFFD_API_FEATURES ; +#ifndef CONFIG_HAVE_ARCH_USERFAULTFD_MINOR uffdio_api.features &= ~(UFFD_FEATURE_MINOR_HUGETLBFS |UFFD_FEATURE_MINOR_SHMEM ); -#endif +#ifndef CONFIG_HUGETLB_HIGH_GRANULARITY_MAPPING + uffdio_api.features &= ~UFFD_FEATURE_MINOR_HUGETLBFS_HGM ; +#endif /* CONFIG_HUGETLB_HIGH_GRANULARITY_MAPPING */ +#endif /* CONFIG_HAVE_ARCH_USERFAULTFD_MINOR */ +#ifndef CONFIG_HAVE_ARCH_USERFAULTFD_WP uffdio_api.features &= ~UFFD_FEATURE_PAGEFAULT_FLAG_WP ;#endif
include/linux/userfaultfd_k.h +7 -0
@@ -162,6 +162,8 @@ static inline bool vma_can_userfault(struct vm_area_struct *vma,vma_is_shmem (vma); } +extern bool uffd_ctx_has_hgm (struct vm_userfaultfd_ctx *); +extern int dup_userfaultfd (struct vm_area_struct *,struct list_head *);extern void dup_userfaultfd_complete (struct list_head *);
@@ -228,6 +230,11 @@ static inline bool userfaultfd_armed(struct vm_area_struct *vma)return false; } +static inline bool uffd_ctx_has_hgm (struct vm_userfaultfd_ctx *ctx) +{ +return false; +} +static inline int dup_userfaultfd (struct vm_area_struct *vma,struct list_head *l) {
include/uapi/linux/userfaultfd.h +2 -0
@@ -36,6 +36,7 @@UFFD_FEATURE_SIGBUS | \UFFD_FEATURE_THREAD_ID | \UFFD_FEATURE_MINOR_HUGETLBFS | \ +UFFD_FEATURE_MINOR_HUGETLBFS_HGM | \UFFD_FEATURE_MINOR_SHMEM | \UFFD_FEATURE_EXACT_ADDRESS | \ UFFD_FEATURE_WP_HUGETLBFS_SHMEM)
@@ -217,6 +218,7 @@ struct uffdio_api {#define UFFD_FEATURE_MINOR_SHMEM (1<<10)#define UFFD_FEATURE_EXACT_ADDRESS (1<<11)#define UFFD_FEATURE_WP_HUGETLBFS_SHMEM (1<<12) +#define UFFD_FEATURE_MINOR_HUGETLBFS_HGM (1<<13)__u64 features;__u64 ioctls;
Signature · 1 line
2.38.0.135.g90850a2211-goog
Peter Xu
reviewerRe: [RFC PATCH v2 33/47] userfaultfd: add UFFD_FEATURE_MINOR_HUGETLBFS_HGM
Quoted 6 lines · James Houghton
Userspace must provide this new feature when it calls
UFFDIO_APIto enable HGM. Userspace can check if the feature exists in uffdio_api.features, and if it does not exist, the kernel does not support and therefore did not enable HGM.Signed-off-by James Houghton <[email protected]>
It's still slightly a pity that this can only be enabled by an uffd context plus a minor fault, so generic hugetlb users cannot directly leverage this.
The patch itself looks good.
Signature · 1 line
Peter Xu
Re: [RFC PATCH v2 33/47] userfaultfd: add UFFD_FEATURE_MINOR_HUGETLBFS_HGM
Quoted 11 lines · Peter Xu
Quoted 6 lines · James Houghton
Userspace must provide this new feature when it calls
UFFDIO_APIto enable HGM. Userspace can check if the feature exists in uffdio_api.features, and if it does not exist, the kernel does not support and therefore did not enable HGM.Signed-off-by James Houghton <[email protected]>It's still slightly a pity that this can only be enabled by an uffd context plus a minor fault, so generic hugetlb users cannot directly leverage this.
The idea here is that, for applications that can conceivably benefit from HGM, we have a mechanism for enabling it for that application. So this patch creates that mechanism for userfaultfd/UFFDIO_CONTINUE. I prefer this approach over something more general like MADV_ENABLE_HGM or something.
For hwpoison, HGM will be automatically enabled, but that isn't implemented in this series. We could also extend MADV_DONTNEED to do high-granularity unmapping in some way, but that also isn't attempted here. I'm sure that if there are other cases where HGM may be useful, we can add/change some uapi to make it possible to take advantage HGM.
- James
Quoted 6 lines
The patch itself looks good.
--
Peter Xu
Peter Xu
reviewerRe: [RFC PATCH v2 33/47] userfaultfd: add UFFD_FEATURE_MINOR_HUGETLBFS_HGM
James,
Quoted 10 lines · James Houghton
Quoted 9 lines · Peter Xu
Quoted 6 lines · James Houghton
Userspace must provide this new feature when it calls
UFFDIO_APIto enable HGM. Userspace can check if the feature exists in uffdio_api.features, and if it does not exist, the kernel does not support and therefore did not enable HGM.Signed-off-by James Houghton <[email protected]>
Quoted 8 lines
Quoted 2 lines
It's still slightly a pity that this can only be enabled by an uffd context plus a minor fault, so generic hugetlb users cannot directly leverage this.
The idea here is that, for applications that can conceivably benefit from HGM, we have a mechanism for enabling it for that application. So this patch creates that mechanism for userfaultfd/UFFDIO_CONTINUE. I prefer this approach over something more general like
MADV_ENABLE_HGMor something.
Sorry to get back to this very late - I know this has been discussed since the very early stage of the feature, but is there any reasoning behind?
When I start to think seriously on applying this to process snapshot with uffd-wp I found that the minor mode trick won't easily play - normally that's a case where all the pages were there mapped huge, but when the app wants UFFDIO_WRITEPROTECT it may want to remap the huge pages into smaller pages, probably some size that the user can specify. It'll be non-trivial to enable HGM during that phase using MINOR mode because in that case the pages are all mapped.
For the long term, I am just still worried the current interface is still not as flexible.
Signature · 1 line
Peter Xu
Re: [RFC PATCH v2 33/47] userfaultfd: add UFFD_FEATURE_MINOR_HUGETLBFS_HGM
Quoted 34 lines · Peter Xu
James,
Quoted 10 lines · James Houghton
Quoted 9 lines · Peter Xu
Quoted 6 lines · James Houghton
Userspace must provide this new feature when it calls
UFFDIO_APIto enable HGM. Userspace can check if the feature exists in uffdio_api.features, and if it does not exist, the kernel does not support and therefore did not enable HGM.Signed-off-by James Houghton <[email protected]>Quoted 8 lines
Quoted 2 lines
It's still slightly a pity that this can only be enabled by an uffd context plus a minor fault, so generic hugetlb users cannot directly leverage this.
The idea here is that, for applications that can conceivably benefit from HGM, we have a mechanism for enabling it for that application. So this patch creates that mechanism for userfaultfd/UFFDIO_CONTINUE. I prefer this approach over something more general like
MADV_ENABLE_HGMor something.Sorry to get back to this very late - I know this has been discussed since the very early stage of the feature, but is there any reasoning behind?
When I start to think seriously on applying this to process snapshot with uffd-wp I found that the minor mode trick won't easily play - normally that's a case where all the pages were there mapped huge, but when the app wants
UFFDIO_WRITEPROTECTit may want to remap the huge pages into smaller pages, probably some size that the user can specify. It'll be non-trivial to enable HGM during that phase using MINOR mode because in that case the pages are all mapped.
Quoted 2 lines
For the long term, I am just still worried the current interface is still not as flexible.
Thanks for bringing this up, Peter. I think the main reason was:
having separate UFFD_FEATUREs clearly indicates to userspace what is and is not supported.
For UFFDIO_WRITEPROTECT, a user could remap huge pages into smaller pages by issuing a high-granularity UFFDIO_WRITEPROTECT. That isn't allowed as of this patch series, but it could be allowed in the future. To add support in the same way as this series, we would add another feature, say UFFD_FEATURE_WP_HUGETLBFS_HGM. I agree that having to add another feature isn't great; is this what you're concerned about?
Considering MADV_ENABLE_HUGETLB...
1. If a user provides this, then the contract becomes: "the kernel may allow UFFDIO_CONTINUE and UFFDIO_WRITEPROTECT for HugeTLB at high-granularities, provided the support exists", but it becomes unclear to userspace to know what's supported and what isn't.
2. We would then need to keep track if a user explicitly enabled it, or if it got enabled automatically in response to memory poison, for example. Not a big problem, just a complication. (Otherwise, if HGM got enabled for poison, suddenly userspace would be allowed to do things it wasn't allowed to do before.)
3. This API makes sense for enabling HGM for something outside of userfaultfd, like MADV_DONTNEED.
Maybe (1) is solvable if we provide a bit field that describes what's supported, or maybe (1) isn't even a problem.
Another possibility is to have a feature like UFFD_FEATURE_HUGETLB_HGM, which will enable the possibility of HGM for all relevant userfaultfd ioctls, but we have the same problem where it's unclear what's supported and what isn't.
I'm happy to change the API to whatever you think makes the most sense.
Thanks!
- James
Quoted 4 lines
--
Peter Xu
Re: [RFC PATCH v2 33/47] userfaultfd: add UFFD_FEATURE_MINOR_HUGETLBFS_HGM
Quoted 35 lines · James Houghton
Quoted 34 lines · Peter Xu
James,
Quoted 10 lines · James Houghton
Quoted 9 lines · Peter Xu
Quoted 6 lines · James Houghton
Userspace must provide this new feature when it calls
UFFDIO_APIto enable HGM. Userspace can check if the feature exists in uffdio_api.features, and if it does not exist, the kernel does not support and therefore did not enable HGM.Signed-off-by James Houghton <[email protected]>Quoted 8 lines
Quoted 2 lines
It's still slightly a pity that this can only be enabled by an uffd context plus a minor fault, so generic hugetlb users cannot directly leverage this.
The idea here is that, for applications that can conceivably benefit from HGM, we have a mechanism for enabling it for that application. So this patch creates that mechanism for userfaultfd/UFFDIO_CONTINUE. I prefer this approach over something more general like
MADV_ENABLE_HGMor something.Sorry to get back to this very late - I know this has been discussed since the very early stage of the feature, but is there any reasoning behind?
When I start to think seriously on applying this to process snapshot with uffd-wp I found that the minor mode trick won't easily play - normally that's a case where all the pages were there mapped huge, but when the app wants
UFFDIO_WRITEPROTECTit may want to remap the huge pages into smaller pages, probably some size that the user can specify. It'll be non-trivial to enable HGM during that phase using MINOR mode because in that case the pages are all mapped.
Quoted 6 lines
Quoted 2 lines
For the long term, I am just still worried the current interface is still not as flexible.
Thanks for bringing this up, Peter. I think the main reason was:
having separateUFFD_FEATUREsclearly indicates to userspace what is and is not supported.
IIRC, I think we wanted to initially limit the usage to the very specific use case (live migration). The idea is that we could then expand usage as more use cases came to light.
Another good thing is that userfaultfd has versioning built into the API. Thus a user can determine if HGM is enabled in their running kernel.
Quoted 8 lines
For
UFFDIO_WRITEPROTECT, a user could remap huge pages into smaller pages by issuing a high-granularityUFFDIO_WRITEPROTECT. That isn't allowed as of this patch series, but it could be allowed in the future. To add support in the same way as this series, we would add another feature, sayUFFD_FEATURE_WP_HUGETLBFS_HGM. I agree that having to add another feature isn't great; is this what you're concerned about?
Quoted 12 lines
Considering
MADV_ENABLE_HUGETLB...
1. If a user provides this, then the contract becomes: "the kernel may allowUFFDIO_CONTINUEandUFFDIO_WRITEPROTECTfor HugeTLB at high-granularities, provided the support exists", but it becomes unclear to userspace to know what's supported and what isn't.
2. We would then need to keep track if a user explicitly enabled it, or if it got enabled automatically in response to memory poison, for example. Not a big problem, just a complication. (Otherwise, if HGM got enabled for poison, suddenly userspace would be allowed to do things it wasn't allowed to do before.)
3. This API makes sense for enabling HGM for something outside of userfaultfd, likeMADV_DONTNEED.
I think #3 is key here. Once we start applying HGM to things outside userfaultfd, then more thought will be required on APIs. The API is somewhat limited by design until the basic functionality is in place.
Signature · 1 line
Mike Kravetz
Quoted 9 lines
Maybe (1) is solvable if we provide a bit field that describes what's supported, or maybe (1) isn't even a problem.
Another possibility is to have a feature like
UFFD_FEATURE_HUGETLB_HGM, which will enable the possibility of HGM for all relevant userfaultfd ioctls, but we have the same problem where it's unclear what's supported and what isn't.I'm happy to change the API to whatever you think makes the most sense.
Peter Xu
reviewerRe: [RFC PATCH v2 33/47] userfaultfd: add UFFD_FEATURE_MINOR_HUGETLBFS_HGM
Quoted 43 lines · Mike Kravetz
Quoted 35 lines · James Houghton
Quoted 34 lines · Peter Xu
James,
Quoted 10 lines · James Houghton
Quoted 9 lines · Peter Xu
Quoted 6 lines · James Houghton
Userspace must provide this new feature when it calls
UFFDIO_APIto enable HGM. Userspace can check if the feature exists in uffdio_api.features, and if it does not exist, the kernel does not support and therefore did not enable HGM.Signed-off-by James Houghton <[email protected]>Quoted 8 lines
Quoted 2 lines
It's still slightly a pity that this can only be enabled by an uffd context plus a minor fault, so generic hugetlb users cannot directly leverage this.
The idea here is that, for applications that can conceivably benefit from HGM, we have a mechanism for enabling it for that application. So this patch creates that mechanism for userfaultfd/UFFDIO_CONTINUE. I prefer this approach over something more general like
MADV_ENABLE_HGMor something.Sorry to get back to this very late - I know this has been discussed since the very early stage of the feature, but is there any reasoning behind?
When I start to think seriously on applying this to process snapshot with uffd-wp I found that the minor mode trick won't easily play - normally that's a case where all the pages were there mapped huge, but when the app wants
UFFDIO_WRITEPROTECTit may want to remap the huge pages into smaller pages, probably some size that the user can specify. It'll be non-trivial to enable HGM during that phase using MINOR mode because in that case the pages are all mapped.Quoted 6 lines
Quoted 2 lines
For the long term, I am just still worried the current interface is still not as flexible.
Thanks for bringing this up, Peter. I think the main reason was:
having separateUFFD_FEATUREsclearly indicates to userspace what is and is not supported.
Quoted 7 lines
IIRC, I think we wanted to initially limit the usage to the very specific use case (live migration). The idea is that we could then expand usage as more use cases came to light.
Another good thing is that userfaultfd has versioning built into the API. Thus a user can determine if HGM is enabled in their running kernel.
I don't worry much on this one, afaiu if we have any way to enable hgm then the user can just try enabling it on a test vma, just like when an app wants to detect whether a new madvise() is present on the current host OS.
Besides, I'm wondering whether something like /sys/kernel/vm/hugepages/hgm would work too.
Quoted 9 lines
Quoted 9 lines
For
UFFDIO_WRITEPROTECT, a user could remap huge pages into smaller pages by issuing a high-granularityUFFDIO_WRITEPROTECT. That isn't allowed as of this patch series, but it could be allowed in the future. To add support in the same way as this series, we would add another feature, sayUFFD_FEATURE_WP_HUGETLBFS_HGM. I agree that having to add another feature isn't great; is this what you're concerned about?
Quoted 10 lines
Quoted 10 lines
Considering
MADV_ENABLE_HUGETLB...
1. If a user provides this, then the contract becomes: "the kernel may allowUFFDIO_CONTINUEandUFFDIO_WRITEPROTECTfor HugeTLB at high-granularities, provided the support exists", but it becomes unclear to userspace to know what's supported and what isn't.
2. We would then need to keep track if a user explicitly enabled it, or if it got enabled automatically in response to memory poison, for example. Not a big problem, just a complication. (Otherwise, if HGM got enabled for poison, suddenly userspace would be allowed to do things it wasn't allowed to do before.)
We could alternatively have two flags for each vma: (a) hgm_advised and (b) hgm_enabled. (a) always sets (b) but not vice versa. We can limit poison to set (b) only. For this patchset, it can be all about (a).
Quoted 6 lines
Quoted 2 lines
3. This API makes sense for enabling HGM for something outside of userfaultfd, like
MADV_DONTNEED.I think #3 is key here. Once we start applying HGM to things outside userfaultfd, then more thought will be required on APIs. The API is somewhat limited by design until the basic functionality is in place.
Mike, could you elaborate what's the major concern of having hgm used outside uffd and live migration use cases?
I feel like I miss something here. I can understand we want to limit the usage only when the user specifies using hgm because we want to keep the old behavior intact. However if we want another way to enable hgm it'll still need one knob anyway even outside uffd, and I thought that'll service the same purpose, or maybe not?
Thanks,
Signature · 1 line
Peter Xu
Re: [RFC PATCH v2 33/47] userfaultfd: add UFFD_FEATURE_MINOR_HUGETLBFS_HGM
Quoted 90 lines · Peter Xu
Quoted 43 lines · Mike Kravetz
Quoted 35 lines · James Houghton
Quoted 34 lines · Peter Xu
James,
Quoted 10 lines · James Houghton
Quoted 9 lines · Peter Xu
Quoted 6 lines · James Houghton
Userspace must provide this new feature when it calls
UFFDIO_APIto enable HGM. Userspace can check if the feature exists in uffdio_api.features, and if it does not exist, the kernel does not support and therefore did not enable HGM.Signed-off-by James Houghton <[email protected]>Quoted 8 lines
Quoted 2 lines
It's still slightly a pity that this can only be enabled by an uffd context plus a minor fault, so generic hugetlb users cannot directly leverage this.
The idea here is that, for applications that can conceivably benefit from HGM, we have a mechanism for enabling it for that application. So this patch creates that mechanism for userfaultfd/UFFDIO_CONTINUE. I prefer this approach over something more general like
MADV_ENABLE_HGMor something.Sorry to get back to this very late - I know this has been discussed since the very early stage of the feature, but is there any reasoning behind?
When I start to think seriously on applying this to process snapshot with uffd-wp I found that the minor mode trick won't easily play - normally that's a case where all the pages were there mapped huge, but when the app wants
UFFDIO_WRITEPROTECTit may want to remap the huge pages into smaller pages, probably some size that the user can specify. It'll be non-trivial to enable HGM during that phase using MINOR mode because in that case the pages are all mapped.Quoted 6 lines
Quoted 2 lines
For the long term, I am just still worried the current interface is still not as flexible.
Thanks for bringing this up, Peter. I think the main reason was:
having separateUFFD_FEATUREsclearly indicates to userspace what is and is not supported.Quoted 7 lines
IIRC, I think we wanted to initially limit the usage to the very specific use case (live migration). The idea is that we could then expand usage as more use cases came to light.
Another good thing is that userfaultfd has versioning built into the API. Thus a user can determine if HGM is enabled in their running kernel.
I don't worry much on this one, afaiu if we have any way to enable hgm then the user can just try enabling it on a test vma, just like when an app wants to detect whether a new
madvise()is present on the current host OS.Besides, I'm wondering whether something like
/sys/kernel/vm/hugepages/hgmwould work too.Quoted 9 lines
Quoted 9 lines
For
UFFDIO_WRITEPROTECT, a user could remap huge pages into smaller pages by issuing a high-granularityUFFDIO_WRITEPROTECT. That isn't allowed as of this patch series, but it could be allowed in the future. To add support in the same way as this series, we would add another feature, sayUFFD_FEATURE_WP_HUGETLBFS_HGM. I agree that having to add another feature isn't great; is this what you're concerned about?Quoted 10 lines
Quoted 10 lines
Considering
MADV_ENABLE_HUGETLB...
1. If a user provides this, then the contract becomes: "the kernel may allowUFFDIO_CONTINUEandUFFDIO_WRITEPROTECTfor HugeTLB at high-granularities, provided the support exists", but it becomes unclear to userspace to know what's supported and what isn't.
2. We would then need to keep track if a user explicitly enabled it, or if it got enabled automatically in response to memory poison, for example. Not a big problem, just a complication. (Otherwise, if HGM got enabled for poison, suddenly userspace would be allowed to do things it wasn't allowed to do before.)We could alternatively have two flags for each vma: (a)
hgm_advisedand (b)hgm_enabled. (a) always sets (b) but not vice versa. We can limit poison to set (b) only. For this patchset, it can be all about (a).Quoted 6 lines
Quoted 2 lines
3. This API makes sense for enabling HGM for something outside of userfaultfd, like
MADV_DONTNEED.I think #3 is key here. Once we start applying HGM to things outside userfaultfd, then more thought will be required on APIs. The API is somewhat limited by design until the basic functionality is in place.
Quoted 8 lines
Mike, could you elaborate what's the major concern of having hgm used outside uffd and live migration use cases?
I feel like I miss something here. I can understand we want to limit the usage only when the user specifies using hgm because we want to keep the old behavior intact. However if we want another way to enable hgm it'll still need one knob anyway even outside uffd, and I thought that'll service the same purpose, or maybe not?
I am not opposed to using hgm outside the use cases targeted by this series.
It seems that when we were previously discussing the API we spent a bunch of time going around in circles trying to get the API correct. That is expected as it is more difficult to take all users/uses/abuses of the API into account.
Since the initial use case was fairly limited, it seemed like a good idea to limit the API to userfaultfd. In this way we could focus on the underlying code/implementation and then expand as needed. Of course, with an eye on anything that may be a limiting factor in the future.
I was not aware of the uffd-wp use case, and am more than happy to discuss expanding the API.
Signature · 1 line
Mike Kravetz
Re: [RFC PATCH v2 33/47] userfaultfd: add UFFD_FEATURE_MINOR_HUGETLBFS_HGM
Quoted 12 lines · Mike Kravetz
Quoted 10 lines · Peter Xu
Quoted 9 lines · Mike Kravetz
Quoted 3 lines · James Houghton
Thanks for bringing this up, Peter. I think the main reason was:
having separateUFFD_FEATUREsclearly indicates to userspace what is and is not supported.IIRC, I think we wanted to initially limit the usage to the very specific use case (live migration). The idea is that we could then expand usage as more use cases came to light.
Quoted 7 lines
Quoted 7 lines
Quoted 3 lines
Another good thing is that userfaultfd has versioning built into the API. Thus a user can determine if HGM is enabled in their running kernel.
I don't worry much on this one, afaiu if we have any way to enable hgm then the user can just try enabling it on a test vma, just like when an app wants to detect whether a new
madvise()is present on the current host OS.
That would be enough to test if HGM was merely present, but if specific features like 4K UFFDIO_CONTINUEs or 4K UFFDIO_WRITEPROTECTs were available. You could always check these by making a HugeTLB VMA and setting it up correctly for userfaultfd/etc., but that's a little messy.
Quoted 3 lines
Quoted 3 lines
Besides, I'm wondering whether something like
/sys/kernel/vm/hugepages/hgmwould work too.
I'm not opposed to this.
Quoted 21 lines
Quoted 21 lines
Quoted 20 lines
Quoted 20 lines
For
UFFDIO_WRITEPROTECT, a user could remap huge pages into smaller pages by issuing a high-granularityUFFDIO_WRITEPROTECT. That isn't allowed as of this patch series, but it could be allowed in the future. To add support in the same way as this series, we would add another feature, sayUFFD_FEATURE_WP_HUGETLBFS_HGM. I agree that having to add another feature isn't great; is this what you're concerned about?Considering
MADV_ENABLE_HUGETLB...
1. If a user provides this, then the contract becomes: "the kernel may allowUFFDIO_CONTINUEandUFFDIO_WRITEPROTECTfor HugeTLB at high-granularities, provided the support exists", but it becomes unclear to userspace to know what's supported and what isn't.
2. We would then need to keep track if a user explicitly enabled it, or if it got enabled automatically in response to memory poison, for example. Not a big problem, just a complication. (Otherwise, if HGM got enabled for poison, suddenly userspace would be allowed to do things it wasn't allowed to do before.)
Quoted 3 lines
Quoted 3 lines
We could alternatively have two flags for each vma: (a)
hgm_advisedand (b)hgm_enabled. (a) always sets (b) but not vice versa. We can limit poison to set (b) only. For this patchset, it can be all about (a).
My thoughts exactly. :)
Quoted 23 lines
Quoted 8 lines
Quoted 7 lines
Quoted 3 lines
3. This API makes sense for enabling HGM for something outside of userfaultfd, like
MADV_DONTNEED.I think #3 is key here. Once we start applying HGM to things outside userfaultfd, then more thought will be required on APIs. The API is somewhat limited by design until the basic functionality is in place.
Quoted 8 lines
Mike, could you elaborate what's the major concern of having hgm used outside uffd and live migration use cases?
I feel like I miss something here. I can understand we want to limit the usage only when the user specifies using hgm because we want to keep the old behavior intact. However if we want another way to enable hgm it'll still need one knob anyway even outside uffd, and I thought that'll service the same purpose, or maybe not?
I am not opposed to using hgm outside the use cases targeted by this series.
It seems that when we were previously discussing the API we spent a bunch of time going around in circles trying to get the API correct. That is expected as it is more difficult to take all users/uses/abuses of the API into account.
Quoted 7 lines
Since the initial use case was fairly limited, it seemed like a good idea to limit the API to userfaultfd. In this way we could focus on the underlying code/implementation and then expand as needed. Of course, with an eye on anything that may be a limiting factor in the future.
I was not aware of the uffd-wp use case, and am more than happy to discuss expanding the API.
So considering two API choices:
1. What we have now: UFFD_FEATURE_MINOR_HUGETLBFS_HGM for UFFDIO_CONTINUE, and later UFFD_FEATURE_WP_HUGETLBFS_HGM for UFFDIO_WRITEPROTECT. For MADV_DONTNEED, we could just suddenly start allowing high-granularity choices (not sure if this is bad; we started allowing it for HugeTLB recently with no other API change, AFAIA).
2. MADV_ENABLE_HGM or something similar. The changes to UFFDIO_CONTINUE/UFFDIO_WRITEPROTECT/MADV_DONTNEED come automatically, provided they are implemented.
I don't mind one way or the other. Peter, I assume you prefer #2. Mike, what about you? If we decide on something other than #1, I'll make the change before sending v1 out.
- James
Re: [RFC PATCH v2 33/47] userfaultfd: add UFFD_FEATURE_MINOR_HUGETLBFS_HGM
Quoted 91 lines · James Houghton
Quoted 12 lines · Mike Kravetz
Quoted 10 lines · Peter Xu
Quoted 9 lines · Mike Kravetz
Quoted 3 lines · James Houghton
Thanks for bringing this up, Peter. I think the main reason was:
having separateUFFD_FEATUREsclearly indicates to userspace what is and is not supported.IIRC, I think we wanted to initially limit the usage to the very specific use case (live migration). The idea is that we could then expand usage as more use cases came to light.
Quoted 7 lines
Quoted 7 lines
Quoted 3 lines
Another good thing is that userfaultfd has versioning built into the API. Thus a user can determine if HGM is enabled in their running kernel.
I don't worry much on this one, afaiu if we have any way to enable hgm then the user can just try enabling it on a test vma, just like when an app wants to detect whether a new
madvise()is present on the current host OS.That would be enough to test if HGM was merely present, but if specific features like 4K
UFFDIO_CONTINUEsor 4KUFFDIO_WRITEPROTECTswere available. You could always check these by making a HugeTLB VMA and setting it up correctly for userfaultfd/etc., but that's a little messy.Quoted 3 lines
Quoted 3 lines
Besides, I'm wondering whether something like
/sys/kernel/vm/hugepages/hgmwould work too.I'm not opposed to this.
Quoted 21 lines
Quoted 21 lines
Quoted 20 lines
Quoted 20 lines
For
UFFDIO_WRITEPROTECT, a user could remap huge pages into smaller pages by issuing a high-granularityUFFDIO_WRITEPROTECT. That isn't allowed as of this patch series, but it could be allowed in the future. To add support in the same way as this series, we would add another feature, sayUFFD_FEATURE_WP_HUGETLBFS_HGM. I agree that having to add another feature isn't great; is this what you're concerned about?Considering
MADV_ENABLE_HUGETLB...
1. If a user provides this, then the contract becomes: "the kernel may allowUFFDIO_CONTINUEandUFFDIO_WRITEPROTECTfor HugeTLB at high-granularities, provided the support exists", but it becomes unclear to userspace to know what's supported and what isn't.
2. We would then need to keep track if a user explicitly enabled it, or if it got enabled automatically in response to memory poison, for example. Not a big problem, just a complication. (Otherwise, if HGM got enabled for poison, suddenly userspace would be allowed to do things it wasn't allowed to do before.)Quoted 3 lines
Quoted 3 lines
We could alternatively have two flags for each vma: (a)
hgm_advisedand (b)hgm_enabled. (a) always sets (b) but not vice versa. We can limit poison to set (b) only. For this patchset, it can be all about (a).My thoughts exactly. :)
Quoted 23 lines
Quoted 8 lines
Quoted 7 lines
Quoted 3 lines
3. This API makes sense for enabling HGM for something outside of userfaultfd, like
MADV_DONTNEED.I think #3 is key here. Once we start applying HGM to things outside userfaultfd, then more thought will be required on APIs. The API is somewhat limited by design until the basic functionality is in place.
Quoted 8 lines
Mike, could you elaborate what's the major concern of having hgm used outside uffd and live migration use cases?
I feel like I miss something here. I can understand we want to limit the usage only when the user specifies using hgm because we want to keep the old behavior intact. However if we want another way to enable hgm it'll still need one knob anyway even outside uffd, and I thought that'll service the same purpose, or maybe not?
I am not opposed to using hgm outside the use cases targeted by this series.
It seems that when we were previously discussing the API we spent a bunch of time going around in circles trying to get the API correct. That is expected as it is more difficult to take all users/uses/abuses of the API into account.
Quoted 7 lines
Since the initial use case was fairly limited, it seemed like a good idea to limit the API to userfaultfd. In this way we could focus on the underlying code/implementation and then expand as needed. Of course, with an eye on anything that may be a limiting factor in the future.
I was not aware of the uffd-wp use case, and am more than happy to discuss expanding the API.
Quoted 7 lines
So considering two API choices:
1. What we have now:
UFFD_FEATURE_MINOR_HUGETLBFS_HGMforUFFDIO_CONTINUE, and laterUFFD_FEATURE_WP_HUGETLBFS_HGMforUFFDIO_WRITEPROTECT. ForMADV_DONTNEED, we could just suddenly start allowing high-granularity choices (not sure if this is bad; we started allowing it for HugeTLB recently with no other API change, AFAIA).
I don't think we can just start allowing HGM for MADV_DONTNEED without some type of user interaction/request. Otherwise, a user that passes in non-hugetlb page size requests may get unexpected results. And, one of the threads about MADV_DONTNEED points out a valid use cases where the caller may not know the mapping is hugetlb or not and is likely to pass in non-hugetlb page size requests.
Quoted 7 lines
2.
MADV_ENABLE_HGMor something similar. The changes to UFFDIO_CONTINUE/UFFDIO_WRITEPROTECT/MADV_DONTNEED come automatically, provided they are implemented.I don't mind one way or the other. Peter, I assume you prefer #2. Mike, what about you? If we decide on something other than #1, I'll make the change before sending v1 out.
Since I do not believe 1) is an option, MADV_ENABLE_HGM might be the way to go. Any thoughts about MADV_ENABLE_HGM? I'm thinking:
- Make it have same restrictions as other madvise hugetlb calls, . addr must be huge page aligned
. length is rounded down to a multiple of huge page size
- We split the vma as required
- Flags carrying HGM state reside in the hugetlb_shared_vma_data struct
Signature · 1 line
Mike Kravetz
Re: [RFC PATCH v2 33/47] userfaultfd: add UFFD_FEATURE_MINOR_HUGETLBFS_HGM
Quoted 23 lines
Quoted 7 lines
So considering two API choices:
1. What we have now:
UFFD_FEATURE_MINOR_HUGETLBFS_HGMforUFFDIO_CONTINUE, and laterUFFD_FEATURE_WP_HUGETLBFS_HGMforUFFDIO_WRITEPROTECT. ForMADV_DONTNEED, we could just suddenly start allowing high-granularity choices (not sure if this is bad; we started allowing it for HugeTLB recently with no other API change, AFAIA).I don't think we can just start allowing HGM for
MADV_DONTNEEDwithout some type of user interaction/request. Otherwise, a user that passes in non-hugetlb page size requests may get unexpected results. And, one of the threads aboutMADV_DONTNEEDpoints out a valid use cases where the caller may not know the mapping is hugetlb or not and is likely to pass in non-hugetlb page size requests.Quoted 7 lines
2.
MADV_ENABLE_HGMor something similar. The changes to UFFDIO_CONTINUE/UFFDIO_WRITEPROTECT/MADV_DONTNEED come automatically, provided they are implemented.I don't mind one way or the other. Peter, I assume you prefer #2. Mike, what about you? If we decide on something other than #1, I'll make the change before sending v1 out.
Quoted 6 lines
Since I do not believe 1) is an option,
MADV_ENABLE_HGMmight be the way to go. Any thoughts aboutMADV_ENABLE_HGM? I'm thinking:
- Make it have same restrictions as other madvise hugetlb calls, . addr must be huge page aligned
. length is rounded down to a multiple of huge page size
- We split the vma as required
I agree with these.
Quoted 1 line
- Flags carrying HGM state reside in the
hugetlb_shared_vma_datastruct
I actually changed this in v1 to storing HGM state as a VMA flag to avoid problems with splitting VMAs (like, when we split a VMA, it's possible the VMA data/lock struct doesn't get allocated). It seems better to me; I can change it back if you disagree.
Not sure what the best name for this flag is either. MADV_ENABLE_HGM sounds ok. MADV_HUGETLB_HGM or MADV_HUGETLB_SMALL_PAGES could work too. No need to figure it out now.
Thanks Mike and Peter :) I'll make this change for v1 and send it out sometime soon.
- James
Peter Xu
reviewerRe: [RFC PATCH v2 33/47] userfaultfd: add UFFD_FEATURE_MINOR_HUGETLBFS_HGM
Quoted 3 lines · James Houghton
Not sure what the best name for this flag is either.
MADV_ENABLE_HGMsounds ok.MADV_HUGETLB_HGMorMADV_HUGETLB_SMALL_PAGEScould work too. No need to figure it out now.
One more option to consider is MADV_SPLIT (hopefully to be more generic).
We already decided to reuse thp MADV_COLLAPSE, we can also introduce MADV_SPLIT and leave thp for later if it can be anything helpful (I remember we used to discuss this for thp split).
For hugetlb one SPLIT should enable hgm advise bit on the vma forever.
Thanks,
Signature · 1 line
Peter Xu
Re: [RFC PATCH v2 33/47] userfaultfd: add UFFD_FEATURE_MINOR_HUGETLBFS_HGM
Quoted 6 lines · Peter Xu
Quoted 3 lines · James Houghton
Not sure what the best name for this flag is either.
MADV_ENABLE_HGMsounds ok.MADV_HUGETLB_HGMorMADV_HUGETLB_SMALL_PAGEScould work too. No need to figure it out now.
Quoted 7 lines
One more option to consider is
MADV_SPLIT(hopefully to be more generic).We already decided to reuse thp
MADV_COLLAPSE, we can also introduceMADV_SPLITand leave thp for later if it can be anything helpful (I remember we used to discuss this for thp split).For hugetlb one SPLIT should enable hgm advise bit on the vma forever.
MADV_SPLIT sounds okay to me -- we'll see how it turns out when I send v1. However, there's an interesting API question regarding what address userfaultfd provides. We previously required UFFD_FEATURE_EXACT_ADDRESS when you specified UFFD_FEATURE_MINOR_HUGETLBFS_HGM so that there was no ambiguity. Now, we can do:
1. When MADV_SPLIT is given, userfaultfd will now round addresses to PAGE_SIZE instead of huge_page_size(hstate), and UFFD_FEATURE_EXACT_ADDRESS is not needed.
2. Don't change anything. A user must know to provide UFFD_FEATURE_EXACT_ADDRESS to get the real address, otherwise they get an (unusable) hugepage-aligned address.
I think #1 sounds fine; let me know if you disagree.
Thanks!
- James
Peter Xu
reviewerRe: [RFC PATCH v2 33/47] userfaultfd: add UFFD_FEATURE_MINOR_HUGETLBFS_HGM
Quoted 22 lines · James Houghton
Quoted 6 lines · Peter Xu
Quoted 3 lines · James Houghton
Not sure what the best name for this flag is either.
MADV_ENABLE_HGMsounds ok.MADV_HUGETLB_HGMorMADV_HUGETLB_SMALL_PAGEScould work too. No need to figure it out now.Quoted 7 lines
One more option to consider is
MADV_SPLIT(hopefully to be more generic).We already decided to reuse thp
MADV_COLLAPSE, we can also introduceMADV_SPLITand leave thp for later if it can be anything helpful (I remember we used to discuss this for thp split).For hugetlb one SPLIT should enable hgm advise bit on the vma forever.
MADV_SPLITsounds okay to me -- we'll see how it turns out when I send v1. However, there's an interesting API question regarding what address userfaultfd provides. We previously requiredUFFD_FEATURE_EXACT_ADDRESSwhen you specifiedUFFD_FEATURE_MINOR_HUGETLBFS_HGMso that there was no ambiguity. Now, we can do:
Quoted 8 lines
1. When
MADV_SPLITis given, userfaultfd will now round addresses toPAGE_SIZEinstead ofhuge_page_size(hstate), andUFFD_FEATURE_EXACT_ADDRESSis not needed.
2. Don't change anything. A user must know to provideUFFD_FEATURE_EXACT_ADDRESSto get the real address, otherwise they get an (unusable) hugepage-aligned address.I think #1 sounds fine; let me know if you disagree.
Sounds good to me, thanks!
Signature · 1 line
Peter Xu
RFC PATCH v2 34/47 hugetlb: userfaultfd: add support for high-granularity UFFDIO_CONTINUE
In reply to James HoughtonChanges here are similar to the changes made for hugetlb_no_page.
Pass vmf->real_address to userfaultfd_huge_must_wait because vmf->address is rounded down to the hugepage size, and a high-granularity page table walk would look up the wrong PTE. Also change the call to userfaultfd_must_wait in the same way for consistency.
This commit introduces hugetlb_alloc_largest_pte which is used to find the appropriate PTE size to map pages with UFFDIO_CONTINUE.
--- fs/userfaultfd.c | 33 +++++++++++++++--- include/linux/hugetlb.h | 14 +++++++- mm/hugetlb.c | 76 +++++++++++++++++++++++++++++++++-------- mm/userfaultfd.c | 46 +++++++++++++++++-------- 4 files changed, 135 insertions(+), 34 deletions(-)
fs/userfaultfd.c +29 -4
@@ -245,14 +245,22 @@ static inline bool userfaultfd_huge_must_wait(struct userfaultfd_ctx *ctx,struct mm_struct *mm = ctx->mm;pte_t *ptep, pte;bool ret = true; +struct hugetlb_pte hpte; +unsigned long sz =vma_mmu_pagesize (vma); +unsigned int shift =huge_page_shift (hstate_vma (vma));mmap_assert_locked (mm); - ptep =huge_pte_offset (mm, address,vma_mmu_pagesize (vma)); + ptep =huge_pte_offset (mm, address, sz);if (!ptep) goto out; +hugetlb_pte_populate (&hpte, ptep, shift,hpage_size_to_level (sz)); +hugetlb_hgm_walk (mm, vma, &hpte, address,PAGE_SIZE , +/*stop_at_none=*/ true); + ptep = hpte.ptep; + ret = false; pte =huge_ptep_get (ptep);
@@ -498,6 +506,14 @@ vm_fault_t handle_userfault(struct vm_fault *vmf, unsigned long reason) blocking_state =userfaultfd_get_blocking_state (vmf->flags); +if (is_vm_hugetlb_page (vmf->vma) &&hugetlb_hgm_enabled (vmf->vma)) +/* +* Lock the VMA lock so we can do a high-granularity walk in +* userfaultfd_huge_must_wait. We have to grab this lock before +* we set our state to blocking. +*/ +hugetlb_vma_lock_read (vmf->vma); +spin_lock_irq (&ctx->fault_pending_wqh.lock);/* * After the __add_wait_queue the uwq is visible to userland
@@ -513,12 +529,15 @@ vm_fault_t handle_userfault(struct vm_fault *vmf, unsigned long reason)spin_unlock_irq (&ctx->fault_pending_wqh.lock);if (!is_vm_hugetlb_page (vmf->vma)) - must_wait =userfaultfd_must_wait (ctx, vmf->address, vmf->flags, - reason); + must_wait =userfaultfd_must_wait (ctx, vmf->real_address, + vmf->flags, reason);else must_wait =userfaultfd_huge_must_wait (ctx, vmf->vma, - vmf->address, + vmf->real_address, vmf->flags, reason); + +if (is_vm_hugetlb_page (vmf->vma) &&hugetlb_hgm_enabled (vmf->vma)) +hugetlb_vma_unlock_read (vmf->vma);mmap_read_unlock (mm);if (likely (must_wait && !READ_ONCE (ctx->released))) {
@@ -1463,6 +1482,12 @@ static int userfaultfd_register(struct userfaultfd_ctx *ctx,mas_pause (&mas); } next: +if (is_vm_hugetlb_page (vma) && (ctx->features & +UFFD_FEATURE_MINOR_HUGETLBFS_HGM )) { + ret =enable_hugetlb_hgm (vma); +if (ret) +break ; + }/* * In the vma_merge() successful mprotect-like case 8: * the next vma was merged into the current one and
include/linux/hugetlb.h +13 -1
@@ -250,7 +250,8 @@ unsigned long hugetlb_total_pages(void);vm_fault_t hugetlb_fault (struct mm_struct *mm,struct vm_area_struct *vma,unsigned long address,unsigned int flags);#ifdef CONFIG_USERFAULTFD -int hugetlb_mcopy_atomic_pte(struct mm_struct *dst_mm, pte_t *dst_pte, +int hugetlb_mcopy_atomic_pte(struct mm_struct *dst_mm, + struct hugetlb_pte *dst_hpte, struct vm_area_struct *dst_vma, unsigned long dst_addr, unsigned long src_addr,
@@ -1272,6 +1273,9 @@ static inline enum hugetlb_level hpage_size_to_level(unsigned long sz)bool hugetlb_hgm_enabled (struct vm_area_struct *vma);bool hugetlb_hgm_eligible (struct vm_area_struct *vma);int enable_hugetlb_hgm (struct vm_area_struct *vma); +int hugetlb_alloc_largest_pte (struct hugetlb_pte *hpte,struct mm_struct *mm, +struct vm_area_struct *vma,unsigned long start, +unsigned long end);#else static inline bool hugetlb_hgm_enabled (struct vm_area_struct *vma) {
@@ -1285,6 +1289,14 @@ static inline int enable_hugetlb_hgm(struct vm_area_struct *vma) {return -EINVAL ; } + +static inline +int hugetlb_alloc_largest_pte (struct hugetlb_pte *hpte,struct mm_struct *mm, +struct vm_area_struct *vma,unsigned long start, +unsigned long end) +{ +return -EINVAL ; +}#endif static inline spinlock_t *huge_pte_lock(struct hstate *h,
mm/hugetlb.c +61 -15
159 lines not on this page · raw message
mm/userfaultfd.c +32 -14
115 lines not on this page · raw message
Signature · 1 line
2.38.0.135.g90850a2211-goog
Peter Xu
reviewerRe: [RFC PATCH v2 34/47] hugetlb: userfaultfd: add support for high-granularity UFFDIO_CONTINUE
Quoted 32 lines · James Houghton
Changes here are similar to the changes made for
hugetlb_no_page.Pass
vmf->real_addresstouserfaultfd_huge_must_waitbecausevmf->addressis rounded down to the hugepage size, and a high-granularity page table walk would look up the wrong PTE. Also change the call touserfaultfd_must_waitin the same way for consistency.This commit introduces
hugetlb_alloc_largest_ptewhich is used to find the appropriate PTE size to map pages withUFFDIO_CONTINUE.Signed-off-by James Houghton <[email protected]>--- fs/userfaultfd.c | 33 +++++++++++++++--- include/linux/hugetlb.h | 14 +++++++- mm/hugetlb.c | 76 +++++++++++++++++++++++++++++++++-------- mm/userfaultfd.c | 46 +++++++++++++++++-------- 4 files changed, 135 insertions(+), 34 deletions(-)fs/userfaultfd.c +3
-0@@ -245,14 +245,22 @@ static inline bool userfaultfd_huge_must_wait(struct userfaultfd_ctx *ctx,struct mm_struct *mm = ctx->mm;pte_t *ptep, pte;bool ret = true; +struct hugetlb_pte hpte; +unsigned long sz =vma_mmu_pagesize (vma); +unsigned int shift =huge_page_shift (hstate_vma (vma));mmap_assert_locked (mm);
fs/userfaultfd.c static inline bool userfaultfd_huge_must_wait(struct userfaultfd_ctx *ctx, · line 261
Quoted 10 lines
fs/userfaultfd.c +4
-1- ptep =huge_pte_offset (mm, address,vma_mmu_pagesize (vma)); + ptep =huge_pte_offset (mm, address, sz);if (!ptep) goto out; +hugetlb_pte_populate (&hpte, ptep, shift,hpage_size_to_level (sz)); +hugetlb_hgm_walk (mm, vma, &hpte, address,PAGE_SIZE , +/*stop_at_none=*/ true);
Side note: I had a feeling that we may want a helper function to walk the whole hugetlb pgtable, that may contain huge_pte_offset() and the hgm walk. Not really needed for this series, but maybe for the future.
Quoted 6 lines
fs/userfaultfd.c +2
-0+ ptep = hpte.ptep; + ret = false; pte =huge_ptep_get (ptep);
fs/userfaultfd.c vm_fault_t handle_userfault(struct vm_fault *vmf, unsigned long reason) · line 515
Quoted 10 lines
fs/userfaultfd.c +7
-0@@ -498,6 +506,14 @@ vm_fault_t handle_userfault(struct vm_fault *vmf, unsigned long reason) blocking_state =userfaultfd_get_blocking_state (vmf->flags); +if (is_vm_hugetlb_page (vmf->vma) &&hugetlb_hgm_enabled (vmf->vma)) +/* +* Lock the VMA lock so we can do a high-granularity walk in +* userfaultfd_huge_must_wait. We have to grab this lock before +* we set our state to blocking. +*/ +hugetlb_vma_lock_read (vmf->vma);
Yeah this will help with/without hgm, afaict. Maybe when I rework my other patchset I'll just take the vma lock unconditionally for this path.
Quoted 24 lines
fs/userfaultfd.c +7
-3+spin_lock_irq (&ctx->fault_pending_wqh.lock);/* * After the __add_wait_queue the uwq is visible to userland @@ -513,12 +529,15 @@ vm_fault_t handle_userfault(struct vm_fault *vmf, unsigned long reason)spin_unlock_irq (&ctx->fault_pending_wqh.lock);if (!is_vm_hugetlb_page (vmf->vma)) - must_wait =userfaultfd_must_wait (ctx, vmf->address, vmf->flags, - reason); + must_wait =userfaultfd_must_wait (ctx, vmf->real_address, + vmf->flags, reason);else must_wait =userfaultfd_huge_must_wait (ctx, vmf->vma, - vmf->address, + vmf->real_address, vmf->flags, reason); + +if (is_vm_hugetlb_page (vmf->vma) &&hugetlb_hgm_enabled (vmf->vma)) +hugetlb_vma_unlock_read (vmf->vma);mmap_read_unlock (mm);if (likely (must_wait && !READ_ONCE (ctx->released))) {
fs/userfaultfd.c static int userfaultfd_register(struct userfaultfd_ctx *ctx, · line 1489
Quoted 8 lines
fs/userfaultfd.c +5
-0@@ -1463,6 +1482,12 @@ static int userfaultfd_register(struct userfaultfd_ctx *ctx,mas_pause (&mas); } next: +if (is_vm_hugetlb_page (vma) && (ctx->features & +UFFD_FEATURE_MINOR_HUGETLBFS_HGM )) { + ret =enable_hugetlb_hgm (vma); +if (ret) +break ;
[1]
Quoted 177 lines
fs/userfaultfd.c +1
-0+ }/* * In the vma_merge() successful mprotect-like case 8: * the next vma was merged into the current one and include/linux/hugetlb.h +13
-1@@ -250,7 +250,8 @@ unsigned long hugetlb_total_pages(void);vm_fault_t hugetlb_fault (struct mm_struct *mm,struct vm_area_struct *vma,unsigned long address,unsigned int flags);#ifdef CONFIG_USERFAULTFD -int hugetlb_mcopy_atomic_pte(struct mm_struct *dst_mm, pte_t *dst_pte, +int hugetlb_mcopy_atomic_pte(struct mm_struct *dst_mm, + struct hugetlb_pte *dst_hpte, struct vm_area_struct *dst_vma, unsigned long dst_addr, unsigned long src_addr,@@ -1272,6 +1273,9 @@ static inline enum hugetlb_level hpage_size_to_level(unsigned long sz)bool hugetlb_hgm_enabled (struct vm_area_struct *vma);bool hugetlb_hgm_eligible (struct vm_area_struct *vma);int enable_hugetlb_hgm (struct vm_area_struct *vma); +int hugetlb_alloc_largest_pte (struct hugetlb_pte *hpte,struct mm_struct *mm, +struct vm_area_struct *vma,unsigned long start, +unsigned long end);#else static inline bool hugetlb_hgm_enabled (struct vm_area_struct *vma) {@@ -1285,6 +1289,14 @@ static inline int enable_hugetlb_hgm(struct vm_area_struct *vma) {return -EINVAL ; } + +static inline +int hugetlb_alloc_largest_pte (struct hugetlb_pte *hpte,struct mm_struct *mm, +struct vm_area_struct *vma,unsigned long start, +unsigned long end) +{ +return -EINVAL ; +}#endif static inline spinlock_t *huge_pte_lock(struct hstate *h,mm/hugetlb.c +25
-15@@ -6325,7 +6325,7 @@ vm_fault_t hugetlb_fault(struct mm_struct *mm, struct vm_area_struct *vma,* modifications for huge pages. */ int hugetlb_mcopy_atomic_pte(struct mm_struct *dst_mm, - pte_t *dst_pte, + struct hugetlb_pte *dst_hpte, struct vm_area_struct *dst_vma, unsigned long dst_addr, unsigned long src_addr,@@ -6336,13 +6336,14 @@ int hugetlb_mcopy_atomic_pte(struct mm_struct *dst_mm,bool is_continue = (mode ==MCOPY_ATOMIC_CONTINUE );struct hstate *h =hstate_vma (dst_vma);struct address_space *mapping = dst_vma->vm_file->f_mapping; -pgoff_t idx =vma_hugecache_offset (h, dst_vma, dst_addr); +unsigned long haddr = dst_addr &huge_page_mask (h); +pgoff_t idx =vma_hugecache_offset (h, dst_vma, haddr);unsigned long size;int vm_shared = dst_vma->vm_flags &VM_SHARED ;pte_t _dst_pte;spinlock_t *ptl;int ret = -ENOMEM ; -struct page *page; +struct page *page, *subpage;int writable;bool page_in_pagecache = false;@@ -6357,12 +6358,12 @@ int hugetlb_mcopy_atomic_pte(struct mm_struct *dst_mm,* a non-missing case. Return -EEXIST. */ if (vm_shared && -hugetlbfs_pagecache_present (h, dst_vma, dst_addr)) { +hugetlbfs_pagecache_present (h, dst_vma, haddr)) { ret = -EEXIST ; goto out; } - page =alloc_huge_page (dst_vma, dst_addr,0 ); + page =alloc_huge_page (dst_vma, haddr,0 );if (IS_ERR (page)) { ret = -ENOMEM ; goto out;@@ -6378,13 +6379,13 @@ int hugetlb_mcopy_atomic_pte(struct mm_struct *dst_mm,/* Free the allocated page which may have * consumed a reservation. */ -restore_reserve_on_error (h, dst_vma, dst_addr, page); +restore_reserve_on_error (h, dst_vma, haddr, page);put_page (page);/* Allocate a temporary page to hold the copied * contents. */ - page =alloc_huge_page_vma (h, dst_vma, dst_addr); + page =alloc_huge_page_vma (h, dst_vma, haddr);if (!page) { ret = -ENOMEM ; goto out;@@ -6398,14 +6399,14 @@ int hugetlb_mcopy_atomic_pte(struct mm_struct *dst_mm, } }else {if (vm_shared && -hugetlbfs_pagecache_present (h, dst_vma, dst_addr)) { +hugetlbfs_pagecache_present (h, dst_vma, haddr)) {put_page (*pagep); ret = -EEXIST ; *pagep =NULL ; goto out; } - page =alloc_huge_page (dst_vma, dst_addr,0 ); + page =alloc_huge_page (dst_vma, haddr,0 );if (IS_ERR (page)) {put_page (*pagep); ret = -ENOMEM ;@@ -6447,7 +6448,7 @@ int hugetlb_mcopy_atomic_pte(struct mm_struct *dst_mm, page_in_pagecache = true; } - ptl =huge_pte_lock (h, dst_mm, dst_pte); + ptl =hugetlb_pte_lock (dst_mm, dst_hpte); ret = -EIO ;if (PageHWPoison (page))@@ -6459,7 +6460,7 @@ int hugetlb_mcopy_atomic_pte(struct mm_struct *dst_mm,* page backing it, then access the page. */ ret = -EEXIST ; -if (!huge_pte_none_mostly (huge_ptep_get (dst_pte))) +if (!huge_pte_none_mostly (huge_ptep_get (dst_hpte->ptep))) goto out_release_unlock;if (page_in_pagecache) {@@ -6478,7 +6479,11 @@ int hugetlb_mcopy_atomic_pte(struct mm_struct *dst_mm,else writable = dst_vma->vm_flags &VM_WRITE ; - _dst_pte =make_huge_pte (dst_vma, page, writable); + subpage =hugetlb_find_subpage (h, page, dst_addr); +WARN_ON_ONCE (subpage != page && !hugetlb_hgm_enabled (dst_vma)); + + _dst_pte =make_huge_pte_with_shift (dst_vma, subpage, writable, + dst_hpte->shift);/* * Always mark UFFDIO_COPY page dirty; note that this may not be * extremely important for hugetlbfs for now since swapping is not @@ -6491,12 +6496,12 @@ int hugetlb_mcopy_atomic_pte(struct mm_struct *dst_mm,if (wp_copy) _dst_pte =huge_pte_mkuffd_wp (_dst_pte); -set_huge_pte_at (dst_mm, dst_addr, dst_pte, _dst_pte); +set_huge_pte_at (dst_mm, dst_addr, dst_hpte->ptep, _dst_pte); -hugetlb_count_add (pages_per_huge_page (h), dst_mm); +hugetlb_count_add (hugetlb_pte_size (dst_hpte) /PAGE_SIZE , dst_mm);/* No need to invalidate - it was non-present before */ -update_mmu_cache (dst_vma, dst_addr, dst_pte); +update_mmu_cache (dst_vma, dst_addr, dst_hpte->ptep);spin_unlock (ptl);if (!is_continue)@@ -7875,6 +7880,47 @@ static unsigned int __shift_for_hstate(struct hstate *h) (tmp_h) <= &hstates[hugetlb_max_hstate]; \ (tmp_h)++) +/* +* Allocate a HugeTLB PTE that maps as much of [start, end) as possible with a +* single page table entry. The allocated HugeTLB PTE is returned in @hpte. +*/ +int hugetlb_alloc_largest_pte(struct hugetlb_pte *hpte, struct mm_struct *mm,
mm/hugetlb.c static unsigned int __shift_for_hstate(struct hstate *h) · line 7897
Quoted 10 lines
mm/hugetlb.c +10
-0+ struct vm_area_struct *vma, unsigned long start, + unsigned long end) +{ + struct hstate *h = hstate_vma(vma), *tmp_h; +unsigned int shift; +unsigned long sz; +int ret; +pte_t *ptep; + +for_each_hgm_shift (h, tmp_h, shift) {
The fallback to PAGE_SIZE (in __shift_for_hstate()) is not obvious. Would it be clearer we handle PAGE_SIZE explicitly here?
Quoted 53 lines
mm/hugetlb.c +26
-0+ sz =1UL << shift; + +if (!IS_ALIGNED (start, sz) || start + sz > end) +continue ; + goto found; + } +return -EINVAL ; +found: + ptep =huge_pte_alloc (mm, vma, start,huge_page_size (h)); +if (!ptep) +return -ENOMEM ; + +hugetlb_pte_populate (hpte, ptep,huge_page_shift (h), +hpage_size_to_level (huge_page_size (h))); + + ret =hugetlb_hgm_walk (mm, vma, hpte, start,1L << shift, +/*stop_at_none=*/ false); +if (ret) +return ret; + +if (hpte->shift > shift) +return -EEXIST ; + +return 0 ; +} +#endif /* CONFIG_HUGETLB_HIGH_GRANULARITY_MAPPING */ /* mm/userfaultfd.c +4
-2@@ -315,14 +315,16 @@ static __always_inline ssize_t __mcopy_atomic_hugetlb(struct mm_struct *dst_mm, {int vm_shared = dst_vma->vm_flags &VM_SHARED ;ssize_t err; -pte_t *dst_pte;unsigned long src_addr, dst_addr;long copied;struct page *page; -unsigned long vma_hpagesize; +unsigned long vma_hpagesize, target_pagesize;pgoff_t idx;u32 hash;struct address_space *mapping; +bool use_hgm =uffd_ctx_has_hgm (&dst_vma->vm_userfaultfd_ctx) && + mode ==MCOPY_ATOMIC_CONTINUE ; +struct hstate *h =hstate_vma (dst_vma);/* * There is no default zero huge page for all huge page sizes as
mm/userfaultfd.c static __always_inline ssize_t __mcopy_atomic_hugetlb(struct mm_struct *dst_mm, · line 345
Quoted 4 lines
mm/userfaultfd.c +1
-0@@ -340,12 +342,13 @@ static __always_inline ssize_t __mcopy_atomic_hugetlb(struct mm_struct *dst_mm, copied =0 ; page =NULL ; vma_hpagesize =vma_kernel_pagesize (dst_vma); + target_pagesize = use_hgm ?PAGE_SIZE : vma_hpagesize;
Nit: "target_pagesize" is slightly misleading? Because hgm can do e.g. 2M on 1G too. I feel like what you want to check here is the minimum requirement, hence.. "min_pagesize"?
Quoted 12 lines
mm/userfaultfd.c +2
-2/* -* Validate alignment based on huge page size +* Validate alignment based on the targeted page size. */ err = -EINVAL ; -if (dst_start & (vma_hpagesize -1 ) || len & (vma_hpagesize -1 )) +if (dst_start & (target_pagesize -1 ) || len & (target_pagesize -1 )) goto out_unlock; retry:
mm/userfaultfd.c static __always_inline ssize_t __mcopy_atomic_hugetlb(struct mm_struct *dst_mm, · line 369
Quoted 5 lines
mm/userfaultfd.c +2
-0@@ -362,6 +365,8 @@ static __always_inline ssize_t __mcopy_atomic_hugetlb(struct mm_struct *dst_mm, err = -EINVAL ;if (vma_hpagesize !=vma_kernel_pagesize (dst_vma)) goto out_unlock; +if (use_hgm && !hugetlb_hgm_enabled (dst_vma)) + goto out_unlock;
Nit: this seems not needed, because enabling the hgm feature for uffd requires to enable it in hugetlb already when register uffd above [1].
Quoted 29 lines
mm/userfaultfd.c +6
-4vm_shared = dst_vma->vm_flags &VM_SHARED ; }@@ -376,13 +381,15 @@ static __always_inline ssize_t __mcopy_atomic_hugetlb(struct mm_struct *dst_mm, }while (src_addr < src_start + len) { +struct hugetlb_pte hpte; +pte_t *dst_pte;BUG_ON (dst_addr >= dst_start + len);/* * Serialize via vma_lock and hugetlb_fault_mutex. -* vma_lock ensures the dst_pte remains valid even -* in the case of shared pmds. fault mutex prevents -* races with other faulting threads. +* vma_lock ensures the hpte.ptep remains valid even +* in the case of shared pmds and page table collapsing. +* fault mutex prevents races with other faulting threads. */ idx =linear_page_index (dst_vma, dst_addr); mapping = dst_vma->vm_file->f_mapping;@@ -390,23 +397,33 @@ static __always_inline ssize_t __mcopy_atomic_hugetlb(struct mm_struct *dst_mm,mutex_lock (&hugetlb_fault_mutex_table[hash]);hugetlb_vma_lock_read (dst_vma); - err = -ENOMEM ; + err =0 ; dst_pte =huge_pte_alloc (dst_mm, dst_vma, dst_addr, vma_hpagesize);
mm/userfaultfd.c static __always_inline ssize_t __mcopy_atomic_hugetlb(struct mm_struct *dst_mm, · line 410
Quoted 10 lines
mm/userfaultfd.c +9
-1-if (!dst_pte) { +if (!dst_pte) + err = -ENOMEM ; +else { +hugetlb_pte_populate (&hpte, dst_pte,huge_page_shift (h), +hpage_size_to_level (huge_page_size (h))); +if (use_hgm) + err =hugetlb_alloc_largest_pte (&hpte, + dst_mm, dst_vma, dst_addr, + dst_start + len);
dst_addr, not dst_start?
Quoted 45 lines
mm/userfaultfd.c +8
-5+ } +if (err) {hugetlb_vma_unlock_read (dst_vma);mutex_unlock (&hugetlb_fault_mutex_table[hash]); goto out_unlock; }if (mode !=MCOPY_ATOMIC_CONTINUE && - !huge_pte_none_mostly (huge_ptep_get (dst_pte))) { + !huge_pte_none_mostly (huge_ptep_get (hpte.ptep))) { err = -EEXIST ;hugetlb_vma_unlock_read (dst_vma);mutex_unlock (&hugetlb_fault_mutex_table[hash]); goto out_unlock; } - err =hugetlb_mcopy_atomic_pte (dst_mm, dst_pte, dst_vma, + err =hugetlb_mcopy_atomic_pte (dst_mm, &hpte, dst_vma, dst_addr, src_addr, mode, &page, wp_copy);@@ -418,6 +435,7 @@ static __always_inline ssize_t __mcopy_atomic_hugetlb(struct mm_struct *dst_mm,if (unlikely (err == -ENOENT )) {mmap_read_unlock (dst_mm);BUG_ON (!page); +BUG_ON (hpte.shift !=huge_page_shift (h)); err = copy_huge_page_from_user(page, (const void __user *)src_addr,@@ -435,9 +453,9 @@ static __always_inline ssize_t __mcopy_atomic_hugetlb(struct mm_struct *dst_mm,BUG_ON (page);if (!err) { - dst_addr += vma_hpagesize; - src_addr += vma_hpagesize; - copied += vma_hpagesize; + dst_addr +=hugetlb_pte_size (&hpte); + src_addr +=hugetlb_pte_size (&hpte); + copied +=hugetlb_pte_size (&hpte);if (fatal_signal_pending (current)) err = -EINTR ;Signature · 1 line
2.38.0.135.g90850a2211-goog
Signature · 1 line
Peter Xu
Peter Xu
reviewerRe: [RFC PATCH v2 34/47] hugetlb: userfaultfd: add support for high-granularity UFFDIO_CONTINUE
In reply to James HoughtonJames,
mm/userfaultfd.c static __always_inline ssize_t __mcopy_atomic_hugetlb(struct mm_struct *dst_mm, · line 326
Quoted 2 lines · James Houghton
mm/userfaultfd.c +2
-0+bool use_hgm =uffd_ctx_has_hgm (&dst_vma->vm_userfaultfd_ctx) && + mode ==MCOPY_ATOMIC_CONTINUE ;
Do you think in your new version use_hgm can work even for MISSING by default?
I had a feeling that the major components are ready for that anyway. Then no matter how HGM is enabled (assuming it'll switch to MADV, or even one can just register with MISSING+MINOR and enable the uffd HGM feature), an existing MISSING only app can easily switch to HGM support if it's on huge pages.
Thanks,
Signature · 1 line
Peter Xu
Re: [RFC PATCH v2 34/47] hugetlb: userfaultfd: add support for high-granularity UFFDIO_CONTINUE
Quoted 9 lines · Peter Xu
James,
mm/userfaultfd.c static __always_inline ssize_t __mcopy_atomic_hugetlb(struct mm_struct *dst_mm, · line 326
Quoted 2 lines · James Houghton
mm/userfaultfd.c +2
-0+bool use_hgm =uffd_ctx_has_hgm (&dst_vma->vm_userfaultfd_ctx) && + mode ==MCOPY_ATOMIC_CONTINUE ;Do you think in your new version
use_hgmcan work even for MISSING by default?
I don't think so -- UFFDIO_COPY will allocate a hugepage, so I'm not sure if it makes sense to allow it at high-granularity. If UFFDIO_COPY didn't allocate a new page, then it could make sense (maybe we'd need a new ioctl or new UFFDIO_COPY mode?). I think it makes most sense to add this with another series.
Thanks,
- James
Peter Xu
reviewerRe: [RFC PATCH v2 34/47] hugetlb: userfaultfd: add support for high-granularity UFFDIO_CONTINUE
Quoted 8 lines · James Houghton
Quoted 7 lines · Peter Xu
James,
Quoted 2 lines · James Houghton
mm/userfaultfd.c +2
-0+bool use_hgm =uffd_ctx_has_hgm (&dst_vma->vm_userfaultfd_ctx) && + mode ==MCOPY_ATOMIC_CONTINUE ;
Quoted 8 lines
Quoted 2 lines
Do you think in your new version
use_hgmcan work even for MISSING by default?I don't think so --
UFFDIO_COPYwill allocate a hugepage, so I'm not sure if it makes sense to allow it at high-granularity. IfUFFDIO_COPYdidn't allocate a new page, then it could make sense (maybe we'd need a new ioctl or newUFFDIO_COPYmode?). I think it makes most sense to add this with another series.
I forgot again on how the page cache is managed for the split pages, sorry.. Yeah let's stick with minor mode for now.
Signature · 1 line
Peter Xu
RFC PATCH v2 35/47 userfaultfd: require UFFD_FEATURE_EXACT_ADDRESS when using HugeTLB HGM
In reply to James HoughtonTo avoid bugs in userspace, we require that userspace provide UFFD_FEATURE_EXACT_ADDRESS when using UFFD_FEATURE_MINOR_HUGETLBFS_HGM, otherwise UFFDIO_API will fail with EINVAL.
The potential confusion is this: without EXACT_ADDRESS, the address given in the userfaultfd message will be rounded down to the hugepage size. Userspace may think that, because they're using HGM, just UFFDIO_CONTINUE the interval [address, address+PAGE_SIZE), but for faults that didn't occur in the first base page of the hugepage, this won't resolve the fault. The only choice it has in this scenario is to UFFDIO_CONTINUE the interval [address, address+hugepage_size), which negates the purpose of using HGM in the first place.
By requiring userspace to provide UFFD_FEATURE_EXACT_ADDRESS, there is no rounding, and userspace now has the information it needs to appropriately resolve the fault.
Another potential solution here is to change the behavior when UFFD_FEATURE_EXACT_ADDRESS is not provided: when HGM is enabled, start rounding to PAGE_SIZE instead of to the hugepage size. I think requiring UFFD_FEATURE_EXACT_ADDRESS is cleaner.
--- fs/userfaultfd.c | 11 +++++++++++ 1 file changed, 11 insertions(+)
fs/userfaultfd.c +11 -0
@@ -1990,6 +1990,17 @@ static int userfaultfd_api(struct userfaultfd_ctx *ctx, ~(UFFD_FEATURE_MINOR_HUGETLBFS |UFFD_FEATURE_MINOR_SHMEM );#ifndef CONFIG_HUGETLB_HIGH_GRANULARITY_MAPPING uffdio_api.features &= ~UFFD_FEATURE_MINOR_HUGETLBFS_HGM ; +#else + + ret = -EINVAL ; +if ((uffdio_api.features &UFFD_FEATURE_MINOR_HUGETLBFS_HGM ) && + !(uffdio_api.features &UFFD_FEATURE_EXACT_ADDRESS )) +/* +* UFFD_FEATURE_MINOR_HUGETLBFS_HGM is mostly +* useless without UFFD_FEATURE_EXACT_ADDRESS, +* so require userspace to provide both. +*/ + goto err_out;#endif /* CONFIG_HUGETLB_HIGH_GRANULARITY_MAPPING */ #endif /* CONFIG_HAVE_ARCH_USERFAULTFD_MINOR */
Signature · 1 line
2.38.0.135.g90850a2211-goog
Peter Xu
reviewerRe: [RFC PATCH v2 35/47] userfaultfd: require UFFD_FEATURE_EXACT_ADDRESS when using HugeTLB HGM
fs/userfaultfd.c static int userfaultfd_api(struct userfaultfd_ctx *ctx, · line 1997
Quoted 9 lines · James Houghton
fs/userfaultfd.c +5
-0@@ -1990,6 +1990,17 @@ static int userfaultfd_api(struct userfaultfd_ctx *ctx, ~(UFFD_FEATURE_MINOR_HUGETLBFS |UFFD_FEATURE_MINOR_SHMEM );#ifndef CONFIG_HUGETLB_HIGH_GRANULARITY_MAPPING uffdio_api.features &= ~UFFD_FEATURE_MINOR_HUGETLBFS_HGM ; +#else + + ret = -EINVAL ; +if ((uffdio_api.features &UFFD_FEATURE_MINOR_HUGETLBFS_HGM ) && + !(uffdio_api.features &UFFD_FEATURE_EXACT_ADDRESS ))
This check needs to be done upon "features" or "ctx_features", rather than "uffdio_api.features". The latter is the one we'll report to the user only.
Quoted 13 lines
fs/userfaultfd.c +6
-0+/* +* UFFD_FEATURE_MINOR_HUGETLBFS_HGM is mostly +* useless without UFFD_FEATURE_EXACT_ADDRESS, +* so require userspace to provide both. +*/ + goto err_out;#endif /* CONFIG_HUGETLB_HIGH_GRANULARITY_MAPPING */ #endif /* CONFIG_HAVE_ARCH_USERFAULTFD_MINOR */ Signature · 1 line
2.38.0.135.g90850a2211-goog
Signature · 1 line
Peter Xu
Re: [RFC PATCH v2 35/47] userfaultfd: require UFFD_FEATURE_EXACT_ADDRESS when using HugeTLB HGM
Quoted 14 lines · Peter Xu
fs/userfaultfd.c static int userfaultfd_api(struct userfaultfd_ctx *ctx, · line 1997
Quoted 9 lines · James Houghton
fs/userfaultfd.c +5
-0@@ -1990,6 +1990,17 @@ static int userfaultfd_api(struct userfaultfd_ctx *ctx, ~(UFFD_FEATURE_MINOR_HUGETLBFS |UFFD_FEATURE_MINOR_SHMEM );#ifndef CONFIG_HUGETLB_HIGH_GRANULARITY_MAPPING uffdio_api.features &= ~UFFD_FEATURE_MINOR_HUGETLBFS_HGM ; +#else + + ret = -EINVAL ; +if ((uffdio_api.features &UFFD_FEATURE_MINOR_HUGETLBFS_HGM ) && + !(uffdio_api.features &UFFD_FEATURE_EXACT_ADDRESS ))This check needs to be done upon "features" or "
ctx_features", rather than "uffdio_api.features". The latter is the one we'll report to the user only.
Ack, thanks Peter. I'm going to drop this patch given the API change (switching to MADV_SPLIT).
- James
RFC PATCH v2 36/47 hugetlb: add MADV_COLLAPSE for hugetlb
In reply to James HoughtonThis is a necessary extension to the UFFDIO_CONTINUE changes. When userspace finishes mapping an entire hugepage with UFFDIO_CONTINUE, the kernel has no mechanism to automatically collapse the page table to map the whole hugepage normally. We require userspace to inform us that they would like the mapping to be collapsed; they do this with MADV_COLLAPSE.
If userspace has not mapped all of a hugepage with UFFDIO_CONTINUE, but only some, hugetlb_collapse will cause the requested range to be mapped as if it were UFFDIO_CONTINUE'd already. The effects of any UFFDIO_WRITEPROTECT calls may be undone by a call to MADV_COLLAPSE for intersecting address ranges.
This commit is co-opting the same madvise mode that has been introduced to synchronously collapse THPs. The function that does THP collapsing has been renamed to madvise_collapse_thp.
As with the rest of the high-granularity mapping support, MADV_COLLAPSE is only supported for shared VMAs right now.
--- include/linux/huge_mm.h | 12 ++-- include/linux/hugetlb.h | 8 +++ mm/hugetlb.c | 142 ++++++++++++++++++++++++++++++++++++++++ mm/khugepaged.c | 4 +- mm/madvise.c | 24 ++++++- 5 files changed, 181 insertions(+), 9 deletions(-)
include/linux/huge_mm.h +6 -6
@@ -226,9 +226,9 @@ void __split_huge_pud(struct vm_area_struct *vma, pud_t *pud,int hugepage_madvise (struct vm_area_struct *vma,unsigned long *vm_flags,int advice); -int madvise_collapse (struct vm_area_struct *vma, -struct vm_area_struct **prev, -unsigned long start,unsigned long end); +int madvise_collapse_thp (struct vm_area_struct *vma, +struct vm_area_struct **prev, +unsigned long start,unsigned long end);void vma_adjust_trans_huge (struct vm_area_struct *vma,unsigned long start,unsigned long end,long adjust_next);spinlock_t *__pmd_trans_huge_lock (pmd_t *pmd,struct vm_area_struct *vma);
@@ -373,9 +373,9 @@ static inline int hugepage_madvise(struct vm_area_struct *vma,return -EINVAL ; } -static inline int madvise_collapse (struct vm_area_struct *vma, -struct vm_area_struct **prev, -unsigned long start,unsigned long end) +static inline int madvise_collapse_thp (struct vm_area_struct *vma, +struct vm_area_struct **prev, +unsigned long start,unsigned long end) {return -EINVAL ; }
include/linux/hugetlb.h +8 -0
@@ -1276,6 +1276,8 @@ int enable_hugetlb_hgm(struct vm_area_struct *vma);int hugetlb_alloc_largest_pte (struct hugetlb_pte *hpte,struct mm_struct *mm,struct vm_area_struct *vma,unsigned long start,unsigned long end); +int hugetlb_collapse (struct mm_struct *mm,struct vm_area_struct *vma, +unsigned long start,unsigned long end);#else static inline bool hugetlb_hgm_enabled (struct vm_area_struct *vma) {
@@ -1297,6 +1299,12 @@ int hugetlb_alloc_largest_pte(struct hugetlb_pte *hpte, struct mm_struct *mm, {return -EINVAL ; } +static inline +int hugetlb_collapse (struct mm_struct *mm,struct vm_area_struct *vma, +unsigned long start,unsigned long end) +{ +return -EINVAL ; +}#endif static inline spinlock_t *huge_pte_lock(struct hstate *h,
mm/hugetlb.c +142 -0
148 lines not on this page · raw message
mm/khugepaged.c +2 -2
@@ -2555,8 +2555,8 @@ static int madvise_collapse_errno(enum scan_result r) } } -int madvise_collapse (struct vm_area_struct *vma,struct vm_area_struct **prev, -unsigned long start,unsigned long end) +int madvise_collapse_thp (struct vm_area_struct *vma,struct vm_area_struct **prev, +unsigned long start,unsigned long end) {struct collapse_control *cc;struct mm_struct *mm = vma->vm_mm;
mm/madvise.c +23 -1
@@ -986,6 +986,24 @@ static long madvise_remove(struct vm_area_struct *vma,return error; } +static int madvise_collapse (struct vm_area_struct *vma, +struct vm_area_struct **prev, +unsigned long start,unsigned long end) +{ +/* Only allow collapsing for HGM-enabled, shared mappings. */ +if (is_vm_hugetlb_page (vma)) { + *prev = vma; +if (!hugetlb_hgm_eligible (vma)) +return -EINVAL ; +if (!hugetlb_hgm_enabled (vma)) +return 0 ; +return hugetlb_collapse (vma->vm_mm, vma, start, end); + } + +return madvise_collapse_thp (vma, prev, start, end); + +} +/* * Apply an madvise behavior to a region of a vma. madvise_update_vma * will handle splitting a vm area into separate areas, each area with its own
@@ -1157,6 +1175,9 @@ madvise_behavior_valid(int behavior)#ifdef CONFIG_TRANSPARENT_HUGEPAGE case MADV_HUGEPAGE :case MADV_NOHUGEPAGE : +#endif +#if defined(CONFIG_HUGETLB_HIGH_GRANULARITY_MAPPING ) || \ + defined(CONFIG_TRANSPARENT_HUGEPAGE )case MADV_COLLAPSE :#endif case MADV_DONTDUMP :
@@ -1347,7 +1368,8 @@ int madvise_set_anon_name(struct mm_struct *mm, unsigned long start,* MADV_NOHUGEPAGE - mark the given range as not worth being backed by * transparent huge pages so the existing pages will not be * coalesced into THP and new pages will not be allocated as THP. -* MADV_COLLAPSE - synchronously coalesce pages into new THP. +* MADV_COLLAPSE - synchronously coalesce pages into new THP, or, for HugeTLB +* pages, collapse the mapping. * MADV_DONTDUMP - the application wants to prevent pages in the given range * from being included in its core dump. * MADV_DODUMP - cancel MADV_DONTDUMP: no longer exclude from core dump.
Signature · 1 line
2.38.0.135.g90850a2211-goog
RFC PATCH v2 37/47 hugetlb: remove huge_pte_lock and huge_pte_lockptr
In reply to James HoughtonThey are replaced with hugetlb_pte_lock{,ptr}. All callers that haven't already been replaced don't get called when using HGM, so we handle them by populating hugetlb_ptes with the standard, hstate-sized huge PTEs.
--- include/linux/hugetlb.h | 28 +++------------------------- mm/hugetlb.c | 15 ++++++++++----- 2 files changed, 13 insertions(+), 30 deletions(-)
include/linux/hugetlb.h +3 -25
@@ -1015,14 +1015,6 @@ static inline gfp_t htlb_modify_alloc_mask(struct hstate *h, gfp_t gfp_mask)return modified_mask; } -static inline spinlock_t *huge_pte_lockptr (unsigned int shift, -struct mm_struct *mm,pte_t *pte) -{ -if (shift ==PMD_SHIFT ) -return pmd_lockptr (mm, (pmd_t *) pte); -return &mm->page_table_lock; -} -#ifndef hugepages_supported/* * Some platform decide whether they support huge pages at boot
@@ -1226,12 +1218,6 @@ static inline gfp_t htlb_modify_alloc_mask(struct hstate *h, gfp_t gfp_mask)return 0 ; } -static inline spinlock_t *huge_pte_lockptr (unsigned int shift, -struct mm_struct *mm,pte_t *pte) -{ -return &mm->page_table_lock; -} -static inline void hugetlb_count_init (struct mm_struct *mm) { }
@@ -1307,16 +1293,6 @@ int hugetlb_collapse(struct mm_struct *mm, struct vm_area_struct *vma, }#endif -static inline spinlock_t *huge_pte_lock (struct hstate *h, -struct mm_struct *mm,pte_t *pte) -{ -spinlock_t *ptl; - - ptl =huge_pte_lockptr (huge_page_shift (h), mm, pte); -spin_lock (ptl); -return ptl; -} -static inline spinlock_t *hugetlb_pte_lockptr (struct mm_struct *mm,struct hugetlb_pte *hpte) {
@@ -1324,7 +1300,9 @@ spinlock_t *hugetlb_pte_lockptr(struct mm_struct *mm, struct hugetlb_pte *hpte)BUG_ON (!hpte->ptep);if (hpte->ptl)return hpte->ptl; -return huge_pte_lockptr (hugetlb_pte_shift (hpte), mm, hpte->ptep); +if (hugetlb_pte_level (hpte) ==HUGETLB_LEVEL_PMD ) +return pmd_lockptr (mm, (pmd_t *) hpte->ptep); +return &mm->page_table_lock; } static inline
mm/hugetlb.c +10 -5
@@ -5164,9 +5164,8 @@ int copy_hugetlb_page_range(struct mm_struct *dst, struct mm_struct *src,put_page (hpage);/* Install the new huge page if src pte stable */ - dst_ptl =huge_pte_lock (h, dst, dst_pte); - src_ptl =huge_pte_lockptr (huge_page_shift (h), - src, src_pte); + dst_ptl =hugetlb_pte_lock (dst, &dst_hpte); + src_ptl =hugetlb_pte_lockptr (src, &src_hpte);spin_lock_nested (src_ptl,SINGLE_DEPTH_NESTING ); entry =huge_ptep_get (src_pte);if (!pte_same (src_pte_old, entry)) {
@@ -7465,6 +7464,7 @@ pte_t *huge_pmd_share(struct mm_struct *mm, struct vm_area_struct *vma,pte_t *spte =NULL ;pte_t *pte;spinlock_t *ptl; +struct hugetlb_pte hpte;i_mmap_lock_read (mapping); vma_interval_tree_foreach(svma, &mapping->i_mmap, idx, idx) {
@@ -7485,7 +7485,8 @@ pte_t *huge_pmd_share(struct mm_struct *mm, struct vm_area_struct *vma,if (!spte) goto out; - ptl =huge_pte_lock (hstate_vma (vma), mm, spte); +hugetlb_pte_populate (&hpte, (pte_t *)pud,PUD_SHIFT ,HUGETLB_LEVEL_PUD ); + ptl =hugetlb_pte_lock (mm, &hpte);if (pud_none (*pud)) {pud_populate (mm, pud, (pmd_t *)((unsigned long )spte &PAGE_MASK ));
@@ -8179,6 +8180,7 @@ void hugetlb_unshare_all_pmds(struct vm_area_struct *vma)unsigned long address, start, end;spinlock_t *ptl;pte_t *ptep; +struct hugetlb_pte hpte;if (!(vma->vm_flags &VM_MAYSHARE ))return ;
@@ -8203,7 +8205,10 @@ void hugetlb_unshare_all_pmds(struct vm_area_struct *vma) ptep =huge_pte_offset (mm, address, sz);if (!ptep)continue ; - ptl =huge_pte_lock (h, mm, ptep); + +hugetlb_pte_populate (&hpte, ptep,huge_page_shift (h), +hpage_size_to_level (sz)); + ptl =hugetlb_pte_lock (mm, &hpte);huge_pmd_unshare (mm, vma, address, ptep);spin_unlock (ptl); }
Signature · 1 line
2.38.0.135.g90850a2211-goog
Peter Xu
reviewerRe: [RFC PATCH v2 37/47] hugetlb: remove huge_pte_lock and huge_pte_lockptr
Quoted 3 lines · James Houghton
They are replaced with hugetlb_pte_lock{,ptr}. All callers that haven't already been replaced don't get called when using HGM, so we handle them by populating
hugetlb_pteswith the standard, hstate-sized huge PTEs.
I didn't yet check the rational at all, but just noticed there's one more of it for ppc code:
*** arch/powerpc/mm/pgtable.c:
huge_ptep_set_access_flags[264] assert_spin_locked(huge_pte_lockptr(huge_page_shift(h),
Signature · 1 line
Peter Xu
RFC PATCH v2 38/47 hugetlb: replace make_huge_pte with make_huge_pte_with_shift
In reply to James HoughtonThis removes the old definition of make_huge_pte, where now we always require the shift to be explicitly given. All callsites are cleaned up.
--- mm/hugetlb.c | 31 ++++++++++++------------------- 1 file changed, 12 insertions(+), 19 deletions(-)
mm/hugetlb.c +12 -19
@@ -4908,9 +4908,9 @@ const struct vm_operations_struct hugetlb_vm_ops = { .pagesize = hugetlb_vm_op_pagesize, }; -static pte_t make_huge_pte_with_shift (struct vm_area_struct *vma, -struct page *page,int writable, -int shift) +static pte_t make_huge_pte (struct vm_area_struct *vma, +struct page *page,int writable, +int shift) {pte_t entry;
@@ -4926,14 +4926,6 @@ static pte_t make_huge_pte_with_shift(struct vm_area_struct *vma,return entry; } -static pte_t make_huge_pte (struct vm_area_struct *vma,struct page *page, -int writable) -{ -unsigned int shift =huge_page_shift (hstate_vma (vma)); - -return make_huge_pte_with_shift (vma, page, writable, shift); -} -static void set_huge_ptep_writable (struct vm_area_struct *vma,unsigned long address,pte_t *ptep) {
@@ -4974,10 +4966,12 @@ static void hugetlb_install_page(struct vm_area_struct *vma, pte_t *ptep, unsigned long addr, struct page *new_page) { + struct hstate *h = hstate_vma(vma);__SetPageUptodate (new_page);hugepage_add_new_anon_rmap (new_page, vma, addr); -set_huge_pte_at (vma->vm_mm, addr, ptep,make_huge_pte (vma, new_page,1 )); -hugetlb_count_add (pages_per_huge_page (hstate_vma (vma)), vma->vm_mm); +set_huge_pte_at (vma->vm_mm, addr, ptep,make_huge_pte (vma, new_page,1 , +huge_page_shift (h))); +hugetlb_count_add (pages_per_huge_page (h), vma->vm_mm);ClearHPageRestoreReserve (new_page);SetHPageMigratable (new_page); }
@@ -5737,7 +5731,8 @@ static vm_fault_t hugetlb_wp(struct mm_struct *mm, struct vm_area_struct *vma,page_remove_rmap (old_page, vma, true);hugepage_add_new_anon_rmap (new_page, vma, haddr);set_huge_pte_at (mm, haddr, ptep, -make_huge_pte (vma, new_page, !unshare)); +make_huge_pte (vma, new_page, !unshare, +huge_page_shift (h)));SetHPageMigratable (new_page);/* Make the old page be freed below */ new_page = old_page;
@@ -6033,7 +6028,7 @@ static vm_fault_t hugetlb_no_page(struct mm_struct *mm,page_dup_file_rmap (page, true); subpage =hugetlb_find_subpage (h, page, haddr_hgm); - new_pte =make_huge_pte_with_shift (vma, subpage, + new_pte =make_huge_pte (vma, subpage, ((vma->vm_flags &VM_WRITE ) && (vma->vm_flags &VM_SHARED )), hpte->shift);
@@ -6481,8 +6476,7 @@ int hugetlb_mcopy_atomic_pte(struct mm_struct *dst_mm, subpage =hugetlb_find_subpage (h, page, dst_addr);WARN_ON_ONCE (subpage != page && !hugetlb_hgm_enabled (dst_vma)); - _dst_pte =make_huge_pte_with_shift (dst_vma, subpage, writable, - dst_hpte->shift); + _dst_pte =make_huge_pte (dst_vma, subpage, writable, dst_hpte->shift);/* * Always mark UFFDIO_COPY page dirty; note that this may not be * extremely important for hugetlbfs for now since swapping is not
@@ -8044,8 +8038,7 @@ int hugetlb_collapse(struct mm_struct *mm, struct vm_area_struct *vma,page_dup_file_rmap (hpage, true); subpage =hugetlb_find_subpage (h, hpage, curr); - entry =make_huge_pte_with_shift (vma, subpage, - writable, hpte.shift); + entry =make_huge_pte (vma, subpage, writable, hpte.shift);set_huge_pte_at (mm, curr, hpte.ptep, entry); next_hpte: curr +=hugetlb_pte_size (&hpte);
Signature · 1 line
2.38.0.135.g90850a2211-goog
RFC PATCH v2 39/47 mm: smaps: add stats for HugeTLB mapping size
In reply to James HoughtonWhen the kernel is compiled with HUGETLB_HIGH_GRANULARITY_MAPPING, smaps may provide HugetlbPudMapped, HugetlbPmdMapped, and HugetlbPteMapped. Levels that are folded will not be outputted.
--- fs/proc/task_mmu.c | 101 +++++++++++++++++++++++++++++++++------------ 1 file changed, 75 insertions(+), 26 deletions(-)
fs/proc/task_mmu.c +75 -26
@@ -405,6 +405,15 @@ struct mem_size_stats {unsigned long swap;unsigned long shared_hugetlb;unsigned long private_hugetlb; +#ifdef CONFIG_HUGETLB_HIGH_GRANULARITY_MAPPING +#ifndef __PAGETABLE_PUD_FOLDED +unsigned long hugetlb_pud_mapped; +#endif +#ifndef __PAGETABLE_PMD_FOLDED +unsigned long hugetlb_pmd_mapped; +#endif +unsigned long hugetlb_pte_mapped; +#endif u64 pss;u64 pss_anon;u64 pss_file;
@@ -720,6 +729,35 @@ static void show_smap_vma_flags(struct seq_file *m, struct vm_area_struct *vma) }#ifdef CONFIG_HUGETLB_PAGE + +static void smaps_hugetlb_hgm_account (struct mem_size_stats *mss, +struct hugetlb_pte *hpte) +{ +#ifdef CONFIG_HUGETLB_HIGH_GRANULARITY_MAPPING +unsigned long size =hugetlb_pte_size (hpte); + +switch (hpte->level) { +#ifndef __PAGETABLE_PUD_FOLDED +case HUGETLB_LEVEL_PUD : + mss->hugetlb_pud_mapped += size; +break ; +#endif +#ifndef __PAGETABLE_PMD_FOLDED +case HUGETLB_LEVEL_PMD : + mss->hugetlb_pmd_mapped += size; +break ; +#endif +case HUGETLB_LEVEL_PTE : + mss->hugetlb_pte_mapped += size; +break ; +default : +break ; + } +#else +return ; +#endif /* CONFIG_HUGETLB_HIGH_GRANULARITY_MAPPING */ +} +static int smaps_hugetlb_range (struct hugetlb_pte *hpte,unsigned long addr,struct mm_walk *walk)
@@ -753,6 +791,8 @@ static int smaps_hugetlb_range(struct hugetlb_pte *hpte, mss->shared_hugetlb +=hugetlb_pte_size (hpte); else mss->private_hugetlb += hugetlb_pte_size(hpte); + +smaps_hugetlb_hgm_account (mss, hpte); }return 0 ; }
@@ -822,38 +862,47 @@ static void smap_gather_stats(struct vm_area_struct *vma,static void __show_smap (struct seq_file *m,const struct mem_size_stats *mss,bool rollup_mode) { -SEQ_PUT_DEC ("Rss: " , mss->resident); -SEQ_PUT_DEC (" kB\nPss: " , mss->pss >>PSS_SHIFT ); -SEQ_PUT_DEC (" kB\nPss_Dirty: " , mss->pss_dirty >>PSS_SHIFT ); +SEQ_PUT_DEC ("Rss: " , mss->resident); +SEQ_PUT_DEC (" kB\nPss: " , mss->pss >>PSS_SHIFT ); +SEQ_PUT_DEC (" kB\nPss_Dirty: " , mss->pss_dirty >>PSS_SHIFT );if (rollup_mode) {/* * These are meaningful only for smaps_rollup, otherwise two of * them are zero, and the other one is the same as Pss. */ -SEQ_PUT_DEC (" kB\nPss_Anon: " , +SEQ_PUT_DEC (" kB\nPss_Anon: " , mss->pss_anon >>PSS_SHIFT ); -SEQ_PUT_DEC (" kB\nPss_File: " , +SEQ_PUT_DEC (" kB\nPss_File: " , mss->pss_file >>PSS_SHIFT ); -SEQ_PUT_DEC (" kB\nPss_Shmem: " , +SEQ_PUT_DEC (" kB\nPss_Shmem: " , mss->pss_shmem >>PSS_SHIFT ); } -SEQ_PUT_DEC (" kB\nShared_Clean: " , mss->shared_clean); -SEQ_PUT_DEC (" kB\nShared_Dirty: " , mss->shared_dirty); -SEQ_PUT_DEC (" kB\nPrivate_Clean: " , mss->private_clean); -SEQ_PUT_DEC (" kB\nPrivate_Dirty: " , mss->private_dirty); -SEQ_PUT_DEC (" kB\nReferenced: " , mss->referenced); -SEQ_PUT_DEC (" kB\nAnonymous: " , mss->anonymous); -SEQ_PUT_DEC (" kB\nLazyFree: " , mss->lazyfree); -SEQ_PUT_DEC (" kB\nAnonHugePages: " , mss->anonymous_thp); -SEQ_PUT_DEC (" kB\nShmemPmdMapped: " , mss->shmem_thp); -SEQ_PUT_DEC (" kB\nFilePmdMapped: " , mss->file_thp); -SEQ_PUT_DEC (" kB\nShared_Hugetlb: " , mss->shared_hugetlb); -seq_put_decimal_ull_width (m," kB\nPrivate_Hugetlb: " , +SEQ_PUT_DEC (" kB\nShared_Clean: " , mss->shared_clean); +SEQ_PUT_DEC (" kB\nShared_Dirty: " , mss->shared_dirty); +SEQ_PUT_DEC (" kB\nPrivate_Clean: " , mss->private_clean); +SEQ_PUT_DEC (" kB\nPrivate_Dirty: " , mss->private_dirty); +SEQ_PUT_DEC (" kB\nReferenced: " , mss->referenced); +SEQ_PUT_DEC (" kB\nAnonymous: " , mss->anonymous); +SEQ_PUT_DEC (" kB\nLazyFree: " , mss->lazyfree); +SEQ_PUT_DEC (" kB\nAnonHugePages: " , mss->anonymous_thp); +SEQ_PUT_DEC (" kB\nShmemPmdMapped: " , mss->shmem_thp); +SEQ_PUT_DEC (" kB\nFilePmdMapped: " , mss->file_thp); +SEQ_PUT_DEC (" kB\nShared_Hugetlb: " , mss->shared_hugetlb); +seq_put_decimal_ull_width (m," kB\nPrivate_Hugetlb: " , mss->private_hugetlb >>10 ,7 ); -SEQ_PUT_DEC (" kB\nSwap: " , mss->swap); -SEQ_PUT_DEC (" kB\nSwapPss: " , +#ifdef CONFIG_HUGETLB_HIGH_GRANULARITY_MAPPING +#ifndef __PAGETABLE_PUD_FOLDED +SEQ_PUT_DEC (" kB\nHugetlbPudMapped: " , mss->hugetlb_pud_mapped); +#endif +#ifndef __PAGETABLE_PMD_FOLDED +SEQ_PUT_DEC (" kB\nHugetlbPmdMapped: " , mss->hugetlb_pmd_mapped); +#endif +SEQ_PUT_DEC (" kB\nHugetlbPteMapped: " , mss->hugetlb_pte_mapped); +#endif /* CONFIG_HUGETLB_HIGH_GRANULARITY_MAPPING */ +SEQ_PUT_DEC (" kB\nSwap: " , mss->swap); +SEQ_PUT_DEC (" kB\nSwapPss: " , mss->swap_pss >>PSS_SHIFT ); -SEQ_PUT_DEC (" kB\nLocked: " , +SEQ_PUT_DEC (" kB\nLocked: " , mss->pss_locked >>PSS_SHIFT );seq_puts (m," kB\n" ); }
@@ -869,18 +918,18 @@ static int show_smap(struct seq_file *m, void *v)show_map_vma (m, vma); -SEQ_PUT_DEC ("Size: " , vma->vm_end - vma->vm_start); -SEQ_PUT_DEC (" kB\nKernelPageSize: " ,vma_kernel_pagesize (vma)); -SEQ_PUT_DEC (" kB\nMMUPageSize: " ,vma_mmu_pagesize (vma)); +SEQ_PUT_DEC ("Size: " , vma->vm_end - vma->vm_start); +SEQ_PUT_DEC (" kB\nKernelPageSize: " ,vma_kernel_pagesize (vma)); +SEQ_PUT_DEC (" kB\nMMUPageSize: " ,vma_mmu_pagesize (vma));seq_puts (m," kB\n" );__show_smap (m, &mss, false); -seq_printf (m,"THPeligible: %d\n" , +seq_printf (m,"THPeligible: %d\n" ,hugepage_vma_check (vma, vma->vm_flags, true, false, true));if (arch_pkeys_enabled ()) -seq_printf (m,"ProtectionKey: %8u\n" ,vma_pkey (vma)); +seq_printf (m,"ProtectionKey: %8u\n" ,vma_pkey (vma));show_smap_vma_flags (m, vma);return 0 ;
Signature · 1 line
2.38.0.135.g90850a2211-goog
RFC PATCH v2 40/47 hugetlb: x86: enable high-granularity mapping
In reply to James HoughtonNow that HGM is fully supported for GENERAL_HUGETLB, x86 can enable it. The x86 KVM MMU already properly handles HugeTLB HGM pages (it does a page table walk to determine which size to use in the second-stage page table instead of, for example, checking vma_mmu_pagesize, like arm64 does).
We could also enable HugeTLB HGM for arm (32-bit) at this point, as it also uses GENERAL_HUGETLB and I don't see anything else that is needed for it. However, I haven't tested on arm at all, so I won't enable it.
--- arch/x86/Kconfig | 1 + 1 file changed, 1 insertion(+)
arch/x86/Kconfig +1 -0
@@ -124,6 +124,7 @@ config X86 select ARCH_WANT_GENERAL_HUGETLB select ARCH_WANT_HUGE_PMD_SHARE select ARCH_WANT_HUGETLB_PAGE_OPTIMIZE_VMEMMAP if X86_64 + select ARCH_WANT_HUGETLB_HIGH_GRANULARITY_MAPPING select ARCH_WANT_LD_ORPHAN_WARN select ARCH_WANTS_THP_SWAP if X86_64 select ARCH_HAS_PARANOID_L1D_FLUSH
Signature · 1 line
2.38.0.135.g90850a2211-goog
RFC PATCH v2 41/47 docs: hugetlb: update hugetlb and userfaultfd admin-guides with HGM info
In reply to James HoughtonThis includes information about how UFFD_FEATURE_MINOR_HUGETLBFS_HGM should be used and when MADV_COLLAPSE should be used with it.
--- Documentation/admin-guide/mm/hugetlbpage.rst | 4 ++++ Documentation/admin-guide/mm/userfaultfd.rst | 16 +++++++++++++++- 2 files changed, 19 insertions(+), 1 deletion(-)
Documentation/admin-guide/mm/hugetlbpage.rst +4 -0
@@ -454,6 +454,10 @@ errno set to EINVAL or exclude hugetlb pages that extend beyond the length if not hugepage aligned. For example, munmap(2) will fail if memory is backed by a hugetlb page and the length is smaller than the hugepage size. +It is possible for users to map HugeTLB pages at a higher granularity than +normal using HugeTLB high-granularity mapping (HGM). For example, when using 1G +pages on x86, a user could map that page with 4K PTEs, 2M PMDs, a combination of +the two. See Documentation/admin-guide/mm/userfaultfd.rst. Examples ========
Documentation/admin-guide/mm/userfaultfd.rst +15 -1
@@ -115,6 +115,14 @@ events, except page fault notifications, may be generated: areas. ``UFFD_FEATURE_MINOR_SHMEM`` is the analogous feature indicating support for shmem virtual memory areas. +- ``UFFD_FEATURE_MINOR_HUGETLBFS_HGM`` indicates that the kernel supports + small-page-aligned regions for ``UFFDIO_CONTINUE`` in HugeTLB-backed + virtual memory areas. ``UFFD_FEATURE_MINOR_HUGETLBFS_HGM`` and + ``UFFD_FEATURE_EXACT_ADDRESS`` must both be specified explicitly to enable + this behavior. If ``UFFD_FEATURE_MINOR_HUGETLBFS_HGM`` is specified but + ``UFFD_FEATURE_EXACT_ADDRESS`` is not, then ``UFFDIO_API`` will fail with + ``EINVAL``. + The userland application should set the feature flags it intends to use when invoking the ``UFFDIO_API`` ioctl, to request that those features be enabled if supported.
@@ -169,7 +177,13 @@ like to do to resolve it: the page cache). Userspace has the option of modifying the page's contents before resolving the fault. Once the contents are correct (modified or not), userspace asks the kernel to map the page and let the - faulting thread continue with ``UFFDIO_CONTINUE``. + faulting thread continue with ``UFFDIO_CONTINUE``. If this is done at the + base-page size in a transparent-hugepage-eligible VMA or in a HugeTLB VMA + (requires ``UFFD_FEATURE_MINOR_HUGETLBFS_HGM``), then userspace may want to + use ``MADV_COLLAPSE`` when a hugepage is fully populated to inform the kernel + that it may be able to collapse the mapping. ``MADV_COLLAPSE`` will may undo + the effect of any ``UFFDIO_WRITEPROTECT`` calls on the collapsed address + range. Notes:
Signature · 1 line
2.38.0.135.g90850a2211-goog
RFC PATCH v2 42/47 docs: proc: include information about HugeTLB HGM
In reply to James HoughtonThis includes the updates that have been made to smaps, specifically, the addition of Hugetlb[Pud,Pmd,Pte]Mapped.
--- Documentation/filesystems/proc.rst | 56 +++++++++++++++++------------- 1 file changed, 32 insertions(+), 24 deletions(-)
Documentation/filesystems/proc.rst +32 -24
@@ -444,29 +444,32 @@ Memory Area, or VMA) there is a series of lines such as the following:: 08048000-080bc000 r-xp 00000000 03:02 13130 /bin/bash - Size: 1084 kB - KernelPageSize: 4 kB - MMUPageSize: 4 kB - Rss: 892 kB - Pss: 374 kB - Pss_Dirty: 0 kB - Shared_Clean: 892 kB - Shared_Dirty: 0 kB - Private_Clean: 0 kB - Private_Dirty: 0 kB - Referenced: 892 kB - Anonymous: 0 kB - LazyFree: 0 kB - AnonHugePages: 0 kB - ShmemPmdMapped: 0 kB - Shared_Hugetlb: 0 kB - Private_Hugetlb: 0 kB - Swap: 0 kB - SwapPss: 0 kB - KernelPageSize: 4 kB - MMUPageSize: 4 kB - Locked: 0 kB - THPeligible: 0 + Size: 1084 kB + KernelPageSize: 4 kB + MMUPageSize: 4 kB + Rss: 892 kB + Pss: 374 kB + Pss_Dirty: 0 kB + Shared_Clean: 892 kB + Shared_Dirty: 0 kB + Private_Clean: 0 kB + Private_Dirty: 0 kB + Referenced: 892 kB + Anonymous: 0 kB + LazyFree: 0 kB + AnonHugePages: 0 kB + ShmemPmdMapped: 0 kB + Shared_Hugetlb: 0 kB + Private_Hugetlb: 0 kB + HugetlbPudMapped: 0 kB + HugetlbPmdMapped: 0 kB + HugetlbPteMapped: 0 kB + Swap: 0 kB + SwapPss: 0 kB + KernelPageSize: 4 kB + MMUPageSize: 4 kB + Locked: 0 kB + THPeligible: 0 VmFlags: rd ex mr mw me dw The first of these lines shows the same information as is displayed for the
@@ -507,10 +510,15 @@ implementation. If this is not desirable please file a bug report. "ShmemPmdMapped" shows the ammount of shared (shmem/tmpfs) memory backed by huge pages. -"Shared_Hugetlb" and "Private_Hugetlb" show the ammounts of memory backed by +"Shared_Hugetlb" and "Private_Hugetlb" show the amounts of memory backed by hugetlbfs page which is *not* counted in "RSS" or "PSS" field for historical reasons. And these are not included in {Shared,Private}_{Clean,Dirty} field. +If the kernel was compiled with ``CONFIG_HUGETLB_HIGH_GRANULARITY_MAPPING``, +"HugetlbPudMapped", "HugetlbPmdMapped", and "HugetlbPteMapped" will appear and +show the amount of HugeTLB memory mapped with PUDs, PMDs, and PTEs respectively. +See Documentation/admin-guide/mm/hugetlbpage.rst. + "Swap" shows how much would-be-anonymous memory is also used, but out on swap. For shmem mappings, "Swap" includes also the size of the mapped (and not
Signature · 1 line
2.38.0.135.g90850a2211-goog
RFC PATCH v2 43/47 selftests/vm: add HugeTLB HGM to userfaultfd selftest
In reply to James HoughtonThis test case behaves similarly to the regular shared HugeTLB configuration, except that it uses 4K instead of hugepages, and that we ignore the UFFDIO_COPY tests, as UFFDIO_CONTINUE is the only ioctl that supports PAGE_SIZE-aligned regions.
This doesn't test MADV_COLLAPSE. Other tests are added later to exercise MADV_COLLAPSE.
--- tools/testing/selftests/vm/userfaultfd.c | 90 +++++++++++++++++++----- 1 file changed, 74 insertions(+), 16 deletions(-)
tools/testing/selftests/vm/userfaultfd.c +74 -16
@@ -73,9 +73,10 @@ static unsigned long nr_cpus, nr_pages, nr_pages_per_cpu, page_size, hpage_size;#define BOUNCE_POLL (1<<3)static int bounces; -#define TEST_ANON 1 -#define TEST_HUGETLB 2 -#define TEST_SHMEM 3 +#define TEST_ANON 1 +#define TEST_HUGETLB 2 +#define TEST_HUGETLB_HGM 3 +#define TEST_SHMEM 4static int test_type;#define UFFD_FLAGS (O_CLOEXEC | O_NONBLOCK | UFFD_USER_MODE_ONLY)
@@ -93,6 +94,8 @@ static volatile bool test_uffdio_zeropage_eexist = true;static bool test_uffdio_wp = true;/* Whether to test uffd minor faults */ static bool test_uffdio_minor = false; +static bool test_uffdio_copy = true; +static bool map_shared;static int mem_fd;static unsigned long long *count_verify;
@@ -151,7 +154,7 @@ static void usage(void)fprintf (stderr,"\nUsage: ./userfaultfd <test type> <MiB> <bounces> " "[hugetlbfs_file]\n\n" );fprintf (stderr,"Supported <test type>: anon, hugetlb, " -"hugetlb_shared, shmem\n\n" ); +"hugetlb_shared, hugetlb_shared_hgm, shmem\n\n" );fprintf (stderr,"'Test mods' can be joined to the test type string with a ':'. " "Supported mods:\n" );fprintf (stderr,"\tsyscall - Use userfaultfd(2) (default)\n" );
@@ -167,6 +170,11 @@ static void usage(void)exit (1 ); } +static bool test_is_hugetlb (void ) +{ +return test_type ==TEST_HUGETLB || test_type ==TEST_HUGETLB_HGM ; +} +#define _err (fmt, ...) \ do { \ int ret = errno; \
@@ -381,8 +389,12 @@ static struct uffd_test_ops *uffd_test_ops;static inline uint64_t uffd_minor_feature (void ) { -if (test_type ==TEST_HUGETLB && map_shared) -return UFFD_FEATURE_MINOR_HUGETLBFS ; +if (test_is_hugetlb () && map_shared) +return UFFD_FEATURE_MINOR_HUGETLBFS | + (test_type ==TEST_HUGETLB_HGM + ? (UFFD_FEATURE_MINOR_HUGETLBFS_HGM | +UFFD_FEATURE_EXACT_ADDRESS ) + :0 );else if (test_type ==TEST_SHMEM )return UFFD_FEATURE_MINOR_SHMEM ;else
@@ -393,7 +405,7 @@ static uint64_t get_expected_ioctls(uint64_t mode) {uint64_t ioctls =UFFD_API_RANGE_IOCTLS ; -if (test_type ==TEST_HUGETLB ) +if (test_is_hugetlb ()) ioctls &= ~(1 << _UFFDIO_ZEROPAGE);if (!((mode &UFFDIO_REGISTER_MODE_WP ) && test_uffdio_wp))
@@ -500,13 +512,16 @@ static void uffd_test_ctx_clear(void)static void uffd_test_ctx_init (uint64_t features) {unsigned long nr, cpu; +uint64_t enabled_features = features;uffd_test_ctx_clear (); uffd_test_ops->allocate_area ((void **)&area_src, true); uffd_test_ops->allocate_area ((void **)&area_dst, false); -userfaultfd_open (&features); +userfaultfd_open (&enabled_features); +if ((enabled_features & features) != features) +err ("couldn't enable all features" ); count_verify =malloc (nr_pages *sizeof (unsigned long long ));if (!count_verify)
@@ -726,13 +741,21 @@ static void uffd_handle_page_fault(struct uffd_msg *msg, struct uffd_stats *stats) {unsigned long offset; +unsigned long address;if (msg->event !=UFFD_EVENT_PAGEFAULT )err ("unexpected msg event %u" , msg->event); +/* +* Round down address to nearest page_size. +* We do this manually because we specified UFFD_FEATURE_EXACT_ADDRESS +* to support UFFD_FEATURE_MINOR_HUGETLBFS_HGM. +*/ + address = msg->arg.pagefault.address & ~(page_size -1 ); +if (msg->arg.pagefault.flags &UFFD_PAGEFAULT_FLAG_WP ) {/* Write protect page faults */ -wp_range (uffd, msg->arg.pagefault.address, page_size, false); +wp_range (uffd, address, page_size, false); stats->wp_faults++; }else if (msg->arg.pagefault.flags &UFFD_PAGEFAULT_FLAG_MINOR ) {uint8_t *area;
@@ -751,11 +774,10 @@ static void uffd_handle_page_fault(struct uffd_msg *msg,*/ area = (uint8_t *)(area_dst + - ((char *)msg->arg.pagefault.address - - area_dst_alias)); + ((char *)address - area_dst_alias));for (b =0 ; b < page_size; ++b) area[b] = ~area[b]; -continue_range (uffd, msg->arg.pagefault.address, page_size); +continue_range (uffd, address, page_size); stats->minor_faults++; }else {/*
@@ -782,7 +804,7 @@ static void uffd_handle_page_fault(struct uffd_msg *msg,if (msg->arg.pagefault.flags &UFFD_PAGEFAULT_FLAG_WRITE )err ("unexpected write fault" ); - offset = (char *)(unsigned long )msg->arg.pagefault.address - area_dst; + offset = (char *)address - area_dst; offset &= ~(page_size-1 );if (copy_page (uffd, offset))
@@ -1192,6 +1214,12 @@ static int userfaultfd_events_test(void)char c;struct uffd_stats stats = {0 }; +if (!test_uffdio_copy) { +printf ("Skipping userfaultfd events test " +"(test_uffdio_copy=false)\n" ); +return 0 ; + } +printf ("testing events (fork, remap, remove): " );fflush (stdout);
@@ -1245,6 +1273,12 @@ static int userfaultfd_sig_test(void)char c;struct uffd_stats stats = {0 }; +if (!test_uffdio_copy) { +printf ("Skipping userfaultfd signal test " +"(test_uffdio_copy=false)\n" ); +return 0 ; + } +printf ("testing signal delivery: " );fflush (stdout);
@@ -1538,6 +1572,12 @@ static int userfaultfd_stress(void)pthread_attr_init (&attr);pthread_attr_setstacksize (&attr,16 *1024 *1024 ); +if (!test_uffdio_copy) { +printf ("Skipping userfaultfd stress test " +"(test_uffdio_copy=false)\n" ); + bounces =0 ; + } +while (bounces--) {printf ("bounces: %d, mode:" , bounces);if (bounces &BOUNCE_RANDOM )
@@ -1696,6 +1736,16 @@ static void set_test_type(const char *type) uffd_test_ops = &hugetlb_uffd_test_ops;/* Minor faults require shared hugetlb; only enable here. */ test_uffdio_minor = true; + }else if (!strcmp (type,"hugetlb_shared_hgm" )) { + map_shared = true; + test_type =TEST_HUGETLB_HGM ; + uffd_test_ops = &hugetlb_uffd_test_ops; +/* +* HugeTLB HGM only changes UFFDIO_CONTINUE, so don't test +* UFFDIO_COPY. +*/ + test_uffdio_minor = true; + test_uffdio_copy = false; }else if (!strcmp (type,"shmem" )) { map_shared = true; test_type =TEST_SHMEM ;
@@ -1731,6 +1781,7 @@ static void parse_test_type_arg(const char *raw_type)err ("Unsupported test: %s" , raw_type);if (test_type ==TEST_HUGETLB ) +/* TEST_HUGETLB_HGM gets small pages. */ page_size = hpage_size;else page_size =sysconf (_SC_PAGE_SIZE);
@@ -1813,22 +1864,29 @@ int main(int argc, char **argv) nr_cpus = x < y ? x : y; } nr_pages_per_cpu = bytes / page_size / nr_cpus; +if (test_type ==TEST_HUGETLB_HGM ) +/* +* `page_size` refers to the page_size we can use in +* UFFDIO_CONTINUE. We still need nr_pages to be appropriately +* aligned, so align it here. +*/ + nr_pages_per_cpu -= nr_pages_per_cpu % (hpage_size / page_size);if (!nr_pages_per_cpu) {_err ("invalid MiB" );usage (); } + nr_pages = nr_pages_per_cpu * nr_cpus; bounces =atoi (argv[3 ]);if (bounces <=0 ) {_err ("invalid bounces" );usage (); } - nr_pages = nr_pages_per_cpu * nr_cpus; -if (test_type ==TEST_SHMEM || test_type ==TEST_HUGETLB ) { +if (test_type ==TEST_SHMEM ||test_is_hugetlb ()) {unsigned int memfd_flags =0 ; -if (test_type ==TEST_HUGETLB ) +if (test_is_hugetlb ()) memfd_flags =MFD_HUGETLB ; mem_fd =memfd_create (argv[0 ], memfd_flags);if (mem_fd <0 )
Signature · 1 line
2.38.0.135.g90850a2211-goog
RFC PATCH v2 44/47 selftests/kvm: add HugeTLB HGM to KVM demand paging selftest
In reply to James HoughtonThis test exercises the GUP paths for HGM. MADV_COLLAPSE is not tested.
--- .../selftests/kvm/demand_paging_test.c | 20 ++++++++++++++++--- .../testing/selftests/kvm/include/test_util.h | 2 ++ tools/testing/selftests/kvm/lib/kvm_util.c | 2 +- tools/testing/selftests/kvm/lib/test_util.c | 14 +++++++++++++ 4 files changed, 34 insertions(+), 4 deletions(-)
tools/testing/selftests/kvm/demand_paging_test.c +17 -3
@@ -76,6 +76,12 @@ static int handle_uffd_page_request(int uffd_mode, int uffd, uint64_t addr)clock_gettime (CLOCK_MONOTONIC , &start); +/* +* We're using UFFD_FEATURE_EXACT_ADDRESS, so round down the address. +* This is needed to support HugeTLB high-granularity mapping. +*/ + addr &= ~(demand_paging_size -1 ); +if (uffd_mode ==UFFDIO_REGISTER_MODE_MISSING ) {struct uffdio_copy copy;
@@ -214,7 +220,8 @@ static void setup_demand_paging(struct kvm_vm *vm, pthread_t *uffd_handler_thread, int pipefd, int uffd_mode, useconds_t uffd_delay, struct uffd_handler_args *uffd_args, - void *hva, void *alias, uint64_t len) + void *hva, void *alias, uint64_t len, + enum vm_mem_backing_src_type src_type) { bool is_minor = (uffd_mode == UFFDIO_REGISTER_MODE_MINOR);int uffd;
@@ -244,9 +251,15 @@ static void setup_demand_paging(struct kvm_vm *vm,TEST_ASSERT (uffd >=0 ,__KVM_SYSCALL_ERROR ("userfaultfd()" , uffd)); uffdio_api.api =UFFD_API ; - uffdio_api.features =0 ; + uffdio_api.features = is_minor + ?UFFD_FEATURE_EXACT_ADDRESS |UFFD_FEATURE_MINOR_HUGETLBFS_HGM + :0 ; ret =ioctl (uffd,UFFDIO_API , &uffdio_api);TEST_ASSERT (ret !=-1 ,__KVM_SYSCALL_ERROR ("UFFDIO_API" , ret)); +if (src_type ==VM_MEM_SRC_SHARED_HUGETLB_HGM ) +TEST_ASSERT (uffdio_api.features & +UFFD_FEATURE_MINOR_HUGETLBFS_HGM , +"UFFD_FEATURE_MINOR_HUGETLBFS_HGM not present" ); uffdio_register.range.start = (uint64_t )hva; uffdio_register.range.len = len;
@@ -329,7 +342,8 @@ static void run_test(enum vm_guest_mode mode, void *arg) pipefds[i *2 ], p->uffd_mode, p->uffd_delay, &uffd_args[i], vcpu_hva, vcpu_alias, - vcpu_args->pages * perf_test_args.guest_page_size); + vcpu_args->pages * perf_test_args.guest_page_size, + p->src_type); } }
tools/testing/selftests/kvm/include/test_util.h +2 -0
@@ -96,6 +96,7 @@ enum vm_mem_backing_src_type {VM_MEM_SRC_ANONYMOUS_HUGETLB_16GB ,VM_MEM_SRC_SHMEM ,VM_MEM_SRC_SHARED_HUGETLB , +VM_MEM_SRC_SHARED_HUGETLB_HGM , NUM_SRC_TYPES, };
@@ -114,6 +115,7 @@ size_t get_def_hugetlb_pagesz(void);const struct vm_mem_backing_src_alias *vm_mem_backing_src_alias (uint32_t i);size_t get_backing_src_pagesz (uint32_t i);bool is_backing_src_hugetlb (uint32_t i); +bool is_backing_src_shared_hugetlb (enum vm_mem_backing_src_type src_type);void backing_src_help (const char *flag);enum vm_mem_backing_src_type parse_backing_src_type (const char *type_name);long get_run_delay (void );
tools/testing/selftests/kvm/lib/kvm_util.c +1 -1
@@ -896,7 +896,7 @@ void vm_userspace_mem_region_add(struct kvm_vm *vm, region->fd =-1 ;if (backing_src_is_shared (src_type)) region->fd =kvm_memfd_alloc (region->mmap_size, - src_type ==VM_MEM_SRC_SHARED_HUGETLB ); +is_backing_src_shared_hugetlb (src_type)); region->mmap_start = mmap(NULL, region->mmap_size, PROT_READ | PROT_WRITE,
tools/testing/selftests/kvm/lib/test_util.c +14 -0
@@ -254,6 +254,13 @@ const struct vm_mem_backing_src_alias *vm_mem_backing_src_alias(uint32_t i)*/ .flag = MAP_SHARED, }, + [VM_MEM_SRC_SHARED_HUGETLB_HGM] = { +/* +* Identical to shared_hugetlb except for the name. +*/ + .name = "shared_hugetlb_hgm", + .flag = MAP_SHARED, + }, };_Static_assert (ARRAY_SIZE (aliases) ==NUM_SRC_TYPES ,"Missing new backing src types?" );
@@ -272,6 +279,7 @@ size_t get_backing_src_pagesz(uint32_t i)switch (i) {case VM_MEM_SRC_ANONYMOUS :case VM_MEM_SRC_SHMEM : +case VM_MEM_SRC_SHARED_HUGETLB_HGM :return getpagesize ();case VM_MEM_SRC_ANONYMOUS_THP :return get_trans_hugepagesz ();
@@ -288,6 +296,12 @@ bool is_backing_src_hugetlb(uint32_t i)return !!(vm_mem_backing_src_alias (i)->flag &MAP_HUGETLB ); } +bool is_backing_src_shared_hugetlb (enum vm_mem_backing_src_type src_type) +{ +return src_type ==VM_MEM_SRC_SHARED_HUGETLB || + src_type ==VM_MEM_SRC_SHARED_HUGETLB_HGM ; +} +static void print_available_backing_src_types (const char *prefix) {int i;
Signature · 1 line
2.38.0.135.g90850a2211-goog
RFC PATCH v2 45/47 selftests/vm: add anon and shared hugetlb to migration test
In reply to James HoughtonShared HugeTLB mappings are migrated best-effort. Sometimes, due to being unable to grab the VMA lock for writing, migration may just randomly fail. To allow for that, we allow retries.
--- tools/testing/selftests/vm/migration.c | 83 ++++++++++++++++++++++++-- 1 file changed, 79 insertions(+), 4 deletions(-)
tools/testing/selftests/vm/migration.c +79 -4
@@ -13,6 +13,7 @@#include <sys/types.h> #include <signal.h> #include <time.h> +#include <sys/statfs.h> #define TWOMEG (2<<20)#define RUNTIME (60)
@@ -59,11 +60,12 @@ FIXTURE_TEARDOWN(migration)free (self->pids); } -int migrate (uint64_t *ptr,int n1,int n2) +int migrate (uint64_t *ptr,int n1,int n2,int retries) {int ret, tmp;int status =0 ;struct timespec ts1, ts2; +int failed =0 ;if (clock_gettime (CLOCK_MONOTONIC , &ts1))return -1 ;
@@ -78,6 +80,9 @@ int migrate(uint64_t *ptr, int n1, int n2) ret =move_pages (0 ,1 , (void **) &ptr, &n2, &status,MPOL_MF_MOVE_ALL );if (ret) { +if (++failed < retries) +continue ; +if (ret >0 )printf ("Didn't migrate %d pages\n" , ret);else
@@ -88,6 +93,7 @@ int migrate(uint64_t *ptr, int n1, int n2) tmp = n2; n2 = n1; n1 = tmp; + failed =0 ; }return 0 ;
@@ -128,7 +134,7 @@ TEST_F_TIMEOUT(migration, private_anon, 2*RUNTIME)if (pthread_create (&self->threads[i],NULL , access_mem, ptr))perror ("Couldn't create thread" ); -ASSERT_EQ (migrate (ptr, self->n1, self->n2),0 ); +ASSERT_EQ (migrate (ptr, self->n1, self->n2,1 ),0 );for (i =0 ; i < self->nthreads -1 ; i++)ASSERT_EQ (pthread_cancel (self->threads[i]),0 ); }
@@ -158,7 +164,7 @@ TEST_F_TIMEOUT(migration, shared_anon, 2*RUNTIME) self->pids[i] = pid; } -ASSERT_EQ (migrate (ptr, self->n1, self->n2),0 ); +ASSERT_EQ (migrate (ptr, self->n1, self->n2,1 ),0 );for (i =0 ; i < self->nthreads -1 ; i++)ASSERT_EQ (kill (self->pids[i],SIGTERM ),0 ); }
@@ -185,9 +191,78 @@ TEST_F_TIMEOUT(migration, private_anon_thp, 2*RUNTIME)if (pthread_create (&self->threads[i],NULL , access_mem, ptr))perror ("Couldn't create thread" ); -ASSERT_EQ (migrate (ptr, self->n1, self->n2),0 ); +ASSERT_EQ (migrate (ptr, self->n1, self->n2,1 ),0 ); +for (i =0 ; i < self->nthreads -1 ; i++) +ASSERT_EQ (pthread_cancel (self->threads[i]),0 ); +} + +/* +* Tests the anon hugetlb migration entry paths. +*/ +TEST_F_TIMEOUT(migration, private_anon_hugetlb, 2*RUNTIME) +{ +uint64_t *ptr; +int i; + +if (self->nthreads <2 || self->n1 <0 || self->n2 <0 ) +SKIP (return,"Not enough threads or NUMA nodes available" ); + + ptr =mmap (NULL ,TWOMEG ,PROT_READ |PROT_WRITE , +MAP_PRIVATE |MAP_ANONYMOUS |MAP_HUGETLB ,-1 ,0 ); +if (ptr ==MAP_FAILED ) +SKIP (return,"Could not allocate hugetlb pages" ); + +memset (ptr,0xde ,TWOMEG ); +for (i =0 ; i < self->nthreads -1 ; i++) +if (pthread_create (&self->threads[i],NULL , access_mem, ptr)) +perror ("Couldn't create thread" ); + +ASSERT_EQ (migrate (ptr, self->n1, self->n2,1 ),0 );for (i =0 ; i < self->nthreads -1 ; i++)ASSERT_EQ (pthread_cancel (self->threads[i]),0 ); } +/* +* Tests the shared hugetlb migration entry paths. +*/ +TEST_F_TIMEOUT(migration, shared_hugetlb, 2*RUNTIME) +{ +uint64_t *ptr; +int i; +int fd; +unsigned long sz; +struct statfs filestat; + +if (self->nthreads <2 || self->n1 <0 || self->n2 <0 ) +SKIP (return,"Not enough threads or NUMA nodes available" ); + + fd =memfd_create ("tmp_hugetlb" ,MFD_HUGETLB ); +if (fd <0 ) +SKIP (return,"Couldn't create hugetlb memfd" ); + +if (fstatfs (fd, &filestat) <0 ) +SKIP (return,"Couldn't fstatfs hugetlb file" ); + + sz = filestat.f_bsize; + +if (ftruncate (fd, sz)) +SKIP (return,"Couldn't allocate hugetlb pages" ); + ptr =mmap (NULL , sz,PROT_READ |PROT_WRITE ,MAP_SHARED , fd,0 ); +if (ptr ==MAP_FAILED ) +SKIP (return,"Could not map hugetlb pages" ); + +memset (ptr,0xde , sz); +for (i =0 ; i < self->nthreads -1 ; i++) +if (pthread_create (&self->threads[i],NULL , access_mem, ptr)) +perror ("Couldn't create thread" ); + +ASSERT_EQ (migrate (ptr, self->n1, self->n2,10 ),0 ); +for (i =0 ; i < self->nthreads -1 ; i++) { +ASSERT_EQ (pthread_cancel (self->threads[i]),0 ); +pthread_join (self->threads[i],NULL ); + } +ftruncate (fd,0 ); +close (fd); +} +TEST_HARNESS_MAIN
Signature · 1 line
2.38.0.135.g90850a2211-goog
RFC PATCH v2 46/47 selftests/vm: add hugetlb HGM test to migration selftest
In reply to James HoughtonThis is mostly the same as the shared HugeTLB case, but instead of mapping the page with a regular page fault, we map it with lots of UFFDIO_CONTINUE operations. We also verify that the contents haven't changed after the migration, which would be the case if the post-migration PTEs pointed to the wrong page.
--- tools/testing/selftests/vm/migration.c | 139 +++++++++++++++++++++++++ 1 file changed, 139 insertions(+)
tools/testing/selftests/vm/migration.c +139 -0
@@ -14,6 +14,11 @@#include <signal.h> #include <time.h> #include <sys/statfs.h> +#include <unistd.h> +#include <sys/ioctl.h> +#include <linux/userfaultfd.h> +#include <sys/syscall.h> +#include <fcntl.h> #define TWOMEG (2<<20)#define RUNTIME (60)
@@ -265,4 +270,138 @@ TEST_F_TIMEOUT(migration, shared_hugetlb, 2*RUNTIME)close (fd); } +#ifdef __NR_userfaultfd +static int map_at_high_granularity (char *mem,size_t length) +{ +int i; +int ret; +int uffd =syscall (__NR_userfaultfd,0 ); +struct uffdio_api api; +struct uffdio_register reg; +int pagesize =getpagesize (); + +if (uffd <0 ) { +perror ("couldn't create uffd" ); +return uffd; + } + + api.api =UFFD_API ; + api.features =UFFD_FEATURE_MISSING_HUGETLBFS + |UFFD_FEATURE_MINOR_HUGETLBFS + |UFFD_FEATURE_MINOR_HUGETLBFS_HGM ; + + ret =ioctl (uffd,UFFDIO_API , &api); +if (ret || api.api !=UFFD_API ) { +perror ("UFFDIO_API failed" ); + goto out; + } + + reg.range.start = (unsigned long )mem; + reg.range.len = length; + + reg.mode =UFFDIO_REGISTER_MODE_MISSING |UFFDIO_REGISTER_MODE_MINOR ; + + ret =ioctl (uffd,UFFDIO_REGISTER , ®); +if (ret) { +perror ("UFFDIO_REGISTER failed" ); + goto out; + } + +/* UFFDIO_CONTINUE each 4K segment of the 2M page. */ +for (i =0 ; i < length/pagesize; ++i) { +struct uffdio_continue cont; + + cont.range.start = (unsigned long long )mem + i * pagesize; + cont.range.len = pagesize; + cont.mode =0 ; + ret =ioctl (uffd,UFFDIO_CONTINUE , &cont); +if (ret) { +fprintf (stderr,"UFFDIO_CONTINUE failed " +"for %llx -> %llx: %d\n" , + cont.range.start, + cont.range.start + cont.range.len, + errno); + goto out; + } + } + ret =0 ; +out: +close (uffd); +return ret; +} +#else +static int map_at_high_granularity (char *mem,size_t length) +{ +fprintf (stderr,"Userfaultfd missing\n" ); +return -1 ; +} +#endif /* __NR_userfaultfd */ + +/* +* Tests the high-granularity hugetlb migration entry paths. +*/ +TEST_F_TIMEOUT(migration, shared_hugetlb_hgm, 2*RUNTIME) +{ +uint64_t *ptr; +int i; +int fd; +unsigned long sz; +struct statfs filestat; + +if (self->nthreads <2 || self->n1 <0 || self->n2 <0 ) +SKIP (return,"Not enough threads or NUMA nodes available" ); + + fd =memfd_create ("tmp_hugetlb" ,MFD_HUGETLB ); +if (fd <0 ) +SKIP (return,"Couldn't create hugetlb memfd" ); + +if (fstatfs (fd, &filestat) <0 ) +SKIP (return,"Couldn't fstatfs hugetlb file" ); + + sz = filestat.f_bsize; + +if (ftruncate (fd, sz)) +SKIP (return,"Couldn't allocate hugetlb pages" ); + +if (fallocate (fd,0 ,0 , sz) <0 ) { +perror ("fallocate failed" ); +SKIP (return,"fallocate failed" ); + } + + ptr =mmap (NULL , sz,PROT_READ |PROT_WRITE ,MAP_SHARED , fd,0 ); +if (ptr ==MAP_FAILED ) +SKIP (return,"Could not allocate hugetlb pages" ); + +/* +* We have to map_at_high_granularity before we memset, otherwise +* memset will map everything at the hugepage size. +*/ +if (map_at_high_granularity ((char *)ptr, sz) <0 ) +SKIP (return,"Could not map HugeTLB range at high granularity" ); + +/* Populate the page we're migrating. */ +for (i =0 ; i < sz/sizeof (*ptr); ++i) + ptr[i] = i; + +for (i =0 ; i < self->nthreads -1 ; i++) +if (pthread_create (&self->threads[i],NULL , access_mem, ptr)) +perror ("Couldn't create thread" ); + +ASSERT_EQ (migrate (ptr, self->n1, self->n2,10 ),0 ); +for (i =0 ; i < self->nthreads -1 ; i++) { +ASSERT_EQ (pthread_cancel (self->threads[i]),0 ); +pthread_join (self->threads[i],NULL ); + } + +/* Check that the contents didnt' change. */ +for (i =0 ; i < sz/sizeof (*ptr); ++i) { +ASSERT_EQ (ptr[i], i); +if (ptr[i] != i) +break ; + } + +ftruncate (fd,0 ); +close (fd); +} +TEST_HARNESS_MAIN
Signature · 1 line
2.38.0.135.g90850a2211-goog
RFC PATCH v2 47/47 selftests/vm: add HGM UFFDIO_CONTINUE and hwpoison tests
In reply to James HoughtonThis tests that high-granularity CONTINUEs at all sizes work (exercising contiguous PTE sizes for arm64, when support is added). This also tests that collapse works and hwpoison works correctly (although we aren't yet testing high-granularity poison).
--- tools/testing/selftests/vm/Makefile | 1 + tools/testing/selftests/vm/hugetlb-hgm.c | 326 +++++++++++++++++++++++ 2 files changed, 327 insertions(+) create mode 100644 tools/testing/selftests/vm/hugetlb-hgm.c
tools/testing/selftests/vm/Makefile +1 -0
@@ -32,6 +32,7 @@ TEST_GEN_FILES += compaction_test TEST_GEN_FILES += gup_test TEST_GEN_FILES += hmm-tests TEST_GEN_FILES += hugetlb-madvise +TEST_GEN_FILES += hugetlb-hgm TEST_GEN_FILES += hugepage-mmap TEST_GEN_FILES += hugepage-mremap TEST_GEN_FILES += hugepage-shm
tools/testing/selftests/vm/hugetlb-hgm.c new file +326 -0
326 lines not on this page · raw message
Signature · 1 line
2.38.0.135.g90850a2211-goog