mm: THP-agnostic refactor on huge mappings
Subsystems (18)
Peter Xu
reviewercover mm: THP-agnostic refactor on huge mappings
This is an RFC series, so not yet for merging. Please don't be scared by the code changes: most of them are code movements only.
This series is based on the dax mprotect fix series here (while that one is based on mm-unstable):
[PATCH v3 0/8] mm/mprotect: Fix dax puds
https://lore.kernel.org/r/[email protected]
Overview
========
This series doesn't provide any feature change. The only goal of this series is to start decoupling two ideas: "THP" and "huge mapping". We already started with having PGTABLE_HAS_HUGE_LEAVES config option, and this one extends that idea into the code.
The issue is that we have so many functions that only compile with CONFIG_THP=on, even though they're about huge mappings, and huge mapping is a pretty common concept, which can apply to many things besides THPs nowadays. The major THP file is mm/huge_memory.c as of now.
The first example of such huge mapping users will be hugetlb. We lived until now with no problem simply because Linux almost duplicated all the logics there in the "THP" files into hugetlb APIs. If we want to get rid of hugetlb specific APIs and paths, this _might_ be the first thing we want to do, because we want to be able to e.g., zapping a hugetlb pmd entry even if !CONFIG_THP.
Then consider other things like dax / pfnmaps. Dax can depend on THP, then it'll naturally be able to use pmd/pud helpers, that's okay. However is it a must? Do we also want to have every new pmd/pud mappings in the future to depend on THP (like PFNMAP)? My answer is no, but I'm open to opinions.
If anyone agrees with me that "huge mapping" (aka, PMD/PUD mappings that are larger than PAGE_SIZE) is a more generic concept than THP, then I think at some point we need to move the generic code out of THP code into a common code base.
This is what this series does as a start.
In general, this series tries to move many THP things (mostly resides in huge_memory.c right now) into two new files: huge_mapping_{pmd|pud}.c. When I move them out, I also put them separately into different files for different layers. Then if an arch supports e.g. only PMD, it can avoid compile the PUD helpers, with things like:
CONFIG_PGTABLE_HAS_PUD_LEAVES=n
obj-$(CONFIG_PGTABLE_HAS_PUD_LEAVES) += huge_mapping_pud.oNote that there're a few tree-wide changes into arch/, but that's not a lot, to make this not disturbing too much people, I only copied the open lists of each arch not yet the arch maintainers.
Tests
=====
My normal 19-archs cross-compilation tests pass with it, and smoke tested on x86_64 with a local config of mine.
Comments welcomed, thanks.
Peter Xu (6):
mm/treewide: Remove pgd_devmap()
mm: PGTABLE_HAS_P[MU]D_LEAVES config options mm/treewide: Make pgtable-generic.c THP agnostic
mm: Move huge mapping declarations from internal.h to huge_mm.h mm/huge_mapping: Create huge_mapping_pxx.c
mm: Convert "*_trans_huge() || *_devmap()" to use *_leaf()
arch/arm64/include/asm/pgtable.h | 11 +- arch/powerpc/include/asm/book3s/64/pgtable.h | 7 +- arch/powerpc/mm/book3s64/pgtable.c | 2 +- arch/riscv/include/asm/pgtable.h | 4 +- arch/s390/include/asm/pgtable.h | 2 +- arch/s390/mm/pgtable.c | 4 +- arch/sparc/mm/tlb.c | 2 +- arch/x86/include/asm/pgtable.h | 5 - arch/x86/mm/pgtable.c | 15 +- include/linux/huge_mm.h | 332 ++++-- include/linux/mm.h | 18 + include/linux/mm_types.h | 2 +- include/linux/pgtable.h | 61 +- include/trace/events/huge_mapping.h | 41 + include/trace/events/thp.h | 28 - mm/Kconfig | 6 + mm/Makefile | 2 + mm/gup.c | 2 - mm/hmm.c | 4 +- mm/huge_mapping_pmd.c | 976 +++++++++++++++ mm/huge_mapping_pud.c | 235 ++++ mm/huge_memory.c | 1125 +----------------- mm/internal.h | 33 - mm/mapping_dirty_helpers.c | 4 +- mm/memory.c | 16 +- mm/migrate_device.c | 2 +- mm/mprotect.c | 4 +- mm/mremap.c | 5 +- mm/page_vma_mapped.c | 5 +- mm/pgtable-generic.c | 37 +- 30 files changed, 1595 insertions(+), 1395 deletions(-) create mode 100644 include/trace/events/huge_mapping.h create mode 100644 mm/huge_mapping_pmd.c create mode 100644 mm/huge_mapping_pud.c
Signature · 1 line
2.45.0
Peter Xu
reviewerRFC PATCH 1/6 mm/treewide: Remove pgd_devmap()
It's always 0 for all archs, and there's no sign to even support p4d entry in the near future. Remove it until it's needed for real.
--- arch/arm64/include/asm/pgtable.h | 5 ----- arch/powerpc/include/asm/book3s/64/pgtable.h | 5 ----- arch/x86/include/asm/pgtable.h | 5 ----- include/linux/pgtable.h | 4 ---- mm/gup.c | 2 -- 5 files changed, 21 deletions(-)
arch/arm64/include/asm/pgtable.h +0 -5
@@ -1119,11 +1119,6 @@ static inline int pud_devmap(pud_t pud) {return 0 ; } - -static inline int pgd_devmap (pgd_t pgd) -{ -return 0 ; -}#endif #ifdef CONFIG_PAGE_TABLE_CHECK
arch/powerpc/include/asm/book3s/64/pgtable.h +0 -5
@@ -1431,11 +1431,6 @@ static inline int pud_devmap(pud_t pud) {return pte_devmap (pud_pte (pud)); } - -static inline int pgd_devmap (pgd_t pgd) -{ -return 0 ; -}#endif /* CONFIG_TRANSPARENT_HUGEPAGE */ #define __HAVE_ARCH_PTEP_MODIFY_PROT_TRANSACTION
arch/x86/include/asm/pgtable.h +0 -5
@@ -311,11 +311,6 @@ static inline int pud_devmap(pud_t pud)return 0 ; }#endif - -static inline int pgd_devmap (pgd_t pgd) -{ -return 0 ; -}#endif #endif /* CONFIG_TRANSPARENT_HUGEPAGE */
include/linux/pgtable.h +0 -4
@@ -1626,10 +1626,6 @@ static inline int pud_devmap(pud_t pud) {return 0 ; } -static inline int pgd_devmap (pgd_t pgd) -{ -return 0 ; -}#endif #if !defined(CONFIG_TRANSPARENT_HUGEPAGE) || \
mm/gup.c +0 -2
@@ -3149,8 +3149,6 @@ static int gup_fast_pgd_leaf(pgd_t orig, pgd_t *pgdp, unsigned long addr,if (!pgd_access_permitted (orig, flags &FOLL_WRITE ))return 0 ; -BUILD_BUG_ON (pgd_devmap (orig)); - page =pgd_page (orig); refs =record_subpages (page,PGDIR_SIZE , addr, end, pages + *nr);
Signature · 1 line
2.45.0
Peter Xu
reviewerRFC PATCH 2/6 mm: PGTABLE_HAS_P[MU]D_LEAVES config options
In reply to Peter XuIntroduce two more sub-options for PGTABLE_HAS_HUGE_LEAVES:
- PGTABLE_HAS_PMD_LEAVES: set when there can be PMD mappings
- PGTABLE_HAS_PUD_LEAVES: set when there can be PUD mappings
It will help to identify whether the current build may only want PMD helpers but not PUD ones, as these sub-options will also check against the arch support over HAVE_ARCH_TRANSPARENT_HUGEPAGE[_PUD].
Note that having them depend on HAVE_ARCH_TRANSPARENT_HUGEPAGE[_PUD] is still some intermediate step. The best way is to have an option say "whether arch XXX supports PMD/PUD mappings" and so on. However let's leave that for later as that's the easy part. So far, we use these options to stably detect per-arch huge mapping support.
--- include/linux/huge_mm.h | 10 +++++++--- mm/Kconfig | 6 ++++++ 2 files changed, 13 insertions(+), 3 deletions(-)
include/linux/huge_mm.h +7 -3
@@ -96,14 +96,18 @@ extern struct kobj_attribute thpsize_shmem_enabled_attr;#define thp_vma_allowable_order (vma, vm_flags, tva_flags, order) \ (!!thp_vma_allowable_orders(vma, vm_flags, tva_flags, BIT(order))) -#ifdef CONFIG_PGTABLE_HAS_HUGE_LEAVES -#define HPAGE_PMD_SHIFT PMD_SHIFT +#ifdef CONFIG_PGTABLE_HAS_PUD_LEAVES #define HPAGE_PUD_SHIFT PUD_SHIFT#else -#define HPAGE_PMD_SHIFT ({ BUILD_BUG(); 0; })#define HPAGE_PUD_SHIFT ({ BUILD_BUG(); 0; })#endif +#ifdef CONFIG_PGTABLE_HAS_PMD_LEAVES +#define HPAGE_PMD_SHIFT PMD_SHIFT +#else +#define HPAGE_PMD_SHIFT ({ BUILD_BUG(); 0; }) +#endif +#define HPAGE_PMD_ORDER (HPAGE_PMD_SHIFT-PAGE_SHIFT)#define HPAGE_PMD_NR (1<<HPAGE_PMD_ORDER)#define HPAGE_PMD_MASK (~(HPAGE_PMD_SIZE - 1))
mm/Kconfig +6 -0
@@ -860,6 +860,12 @@ endif # TRANSPARENT_HUGEPAGE config PGTABLE_HAS_HUGE_LEAVES def_bool TRANSPARENT_HUGEPAGE || HUGETLB_PAGE +config PGTABLE_HAS_PMD_LEAVES + def_bool HAVE_ARCH_TRANSPARENT_HUGEPAGE && PGTABLE_HAS_HUGE_LEAVES + +config PGTABLE_HAS_PUD_LEAVES + def_bool HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD && PGTABLE_HAS_HUGE_LEAVES + # # UP and nommu archs use km based percpu allocator #
Signature · 1 line
2.45.0
Re: [PATCH RFC 2/6] mm: PGTABLE_HAS_P[MU]D_LEAVES config options
Le 18/07/2024 à 00:02, Peter Xu a écrit :
Quoted 52 lines
Introduce two more sub-options for
PGTABLE_HAS_HUGE_LEAVES:-
PGTABLE_HAS_PMD_LEAVES: set when there can be PMD mappings
-PGTABLE_HAS_PUD_LEAVES: set when there can be PUD mappingsIt will help to identify whether the current build may only want PMD helpers but not PUD ones, as these sub-options will also check against the arch support over HAVE_ARCH_TRANSPARENT_HUGEPAGE[_PUD].
Note that having them depend on HAVE_ARCH_TRANSPARENT_HUGEPAGE[_PUD] is still some intermediate step. The best way is to have an option say "whether arch XXX supports PMD/PUD mappings" and so on. However let's leave that for later as that's the easy part. So far, we use these options to stably detect per-arch huge mapping support.
Signed-off-by Peter Xu <[email protected]>--- include/linux/huge_mm.h | 10 +++++++--- mm/Kconfig | 6 ++++++ 2 files changed, 13 insertions(+), 3 deletions(-)include/linux/huge_mm.h +7
-3@@ -96,14 +96,18 @@ extern struct kobj_attribute thpsize_shmem_enabled_attr;#define thp_vma_allowable_order (vma, vm_flags, tva_flags, order) \ (!!thp_vma_allowable_orders(vma, vm_flags, tva_flags, BIT(order))) -#ifdef CONFIG_PGTABLE_HAS_HUGE_LEAVES -#define HPAGE_PMD_SHIFT PMD_SHIFT +#ifdef CONFIG_PGTABLE_HAS_PUD_LEAVES #define HPAGE_PUD_SHIFT PUD_SHIFT#else -#define HPAGE_PMD_SHIFT ({ BUILD_BUG(); 0; })#define HPAGE_PUD_SHIFT ({ BUILD_BUG(); 0; })#endif +#ifdef CONFIG_PGTABLE_HAS_PMD_LEAVES +#define HPAGE_PMD_SHIFT PMD_SHIFT +#else +#define HPAGE_PMD_SHIFT ({ BUILD_BUG(); 0; }) +#endif +#define HPAGE_PMD_ORDER (HPAGE_PMD_SHIFT-PAGE_SHIFT)#define HPAGE_PMD_NR (1<<HPAGE_PMD_ORDER)#define HPAGE_PMD_MASK (~(HPAGE_PMD_SIZE - 1))mm/Kconfig +0
-0
mm/Kconfig endif # TRANSPARENT_HUGEPAGE · line 868
Quoted 9 lines
mm/Kconfig +6
-0@@ -860,6 +860,12 @@ endif # TRANSPARENT_HUGEPAGE config PGTABLE_HAS_HUGE_LEAVES def_bool TRANSPARENT_HUGEPAGE || HUGETLB_PAGE +config PGTABLE_HAS_PMD_LEAVES + def_bool HAVE_ARCH_TRANSPARENT_HUGEPAGE && PGTABLE_HAS_HUGE_LEAVES + +config PGTABLE_HAS_PUD_LEAVES + def_bool HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD && PGTABLE_HAS_HUGE_LEAVES +
What if an architecture has hugepages at PMD and/or PUD level and doesn't support THP ?
Christophe
Quoted 3 lines
mm/Kconfig +0
-0# # UP and nommu archs use km based percpu allocator #
Peter Xu
reviewerRe: [PATCH RFC 2/6] mm: PGTABLE_HAS_P[MU]D_LEAVES config options
Quoted 58 lines · LEROY Christophe
Le 18/07/2024 à 00:02, Peter Xu a écrit :
Quoted 55 lines
Introduce two more sub-options for
PGTABLE_HAS_HUGE_LEAVES:-
PGTABLE_HAS_PMD_LEAVES: set when there can be PMD mappings
-PGTABLE_HAS_PUD_LEAVES: set when there can be PUD mappingsIt will help to identify whether the current build may only want PMD helpers but not PUD ones, as these sub-options will also check against the arch support over HAVE_ARCH_TRANSPARENT_HUGEPAGE[_PUD].
Note that having them depend on HAVE_ARCH_TRANSPARENT_HUGEPAGE[_PUD] is still some intermediate step. The best way is to have an option say "whether arch XXX supports PMD/PUD mappings" and so on. However let's leave that for later as that's the easy part. So far, we use these options to stably detect per-arch huge mapping support.
Signed-off-by Peter Xu <[email protected]>--- include/linux/huge_mm.h | 10 +++++++--- mm/Kconfig | 6 ++++++ 2 files changed, 13 insertions(+), 3 deletions(-)include/linux/huge_mm.h +7
-3@@ -96,14 +96,18 @@ extern struct kobj_attribute thpsize_shmem_enabled_attr;#define thp_vma_allowable_order (vma, vm_flags, tva_flags, order) \ (!!thp_vma_allowable_orders(vma, vm_flags, tva_flags, BIT(order))) -#ifdef CONFIG_PGTABLE_HAS_HUGE_LEAVES -#define HPAGE_PMD_SHIFT PMD_SHIFT +#ifdef CONFIG_PGTABLE_HAS_PUD_LEAVES #define HPAGE_PUD_SHIFT PUD_SHIFT#else -#define HPAGE_PMD_SHIFT ({ BUILD_BUG(); 0; })#define HPAGE_PUD_SHIFT ({ BUILD_BUG(); 0; })#endif +#ifdef CONFIG_PGTABLE_HAS_PMD_LEAVES +#define HPAGE_PMD_SHIFT PMD_SHIFT +#else +#define HPAGE_PMD_SHIFT ({ BUILD_BUG(); 0; }) +#endif +#define HPAGE_PMD_ORDER (HPAGE_PMD_SHIFT-PAGE_SHIFT)#define HPAGE_PMD_NR (1<<HPAGE_PMD_ORDER)#define HPAGE_PMD_MASK (~(HPAGE_PMD_SIZE - 1))mm/Kconfig +0
-0@@ -860,6 +860,12 @@ endif # TRANSPARENT_HUGEPAGE config PGTABLE_HAS_HUGE_LEAVES def_bool TRANSPARENT_HUGEPAGE || HUGETLB_PAGE
Quoted 9 lines
mm/Kconfig endif # TRANSPARENT_HUGEPAGE · line 868
Quoted 6 lines
mm/Kconfig +6
-0+config PGTABLE_HAS_PMD_LEAVES + def_bool HAVE_ARCH_TRANSPARENT_HUGEPAGE && PGTABLE_HAS_HUGE_LEAVES + +config PGTABLE_HAS_PUD_LEAVES + def_bool HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD && PGTABLE_HAS_HUGE_LEAVES +What if an architecture has hugepages at PMD and/or PUD level and doesn't support THP ?
What's the arch to be discussed here?
The whole purpose of this series so far is trying to make some pmd/pud helpers that only defined with CONFIG_THP=on to be available even if not. It means this series alone (or any future plan) shouldn't affect any arch that has CONFIG_THP=off always.
But logically I think we should need some config option just to say "this arch supports pmd mappings" indeed, even if CONFIG_THP=off. When that's there, we should perhaps add that option into this equation so PGTABLE_HAS_*_LEAVES will also be selected in that case.
Signature · 1 line
Peter Xu
Re: [PATCH RFC 2/6] mm: PGTABLE_HAS_P[MU]D_LEAVES config options
Le 22/08/2024 à 21:16, Peter Xu a écrit :
Quoted 69 lines · LEROY Christophe
Quoted 58 lines
Le 18/07/2024 à 00:02, Peter Xu a écrit :
Quoted 55 lines
Introduce two more sub-options for
PGTABLE_HAS_HUGE_LEAVES:-
PGTABLE_HAS_PMD_LEAVES: set when there can be PMD mappings
-PGTABLE_HAS_PUD_LEAVES: set when there can be PUD mappingsIt will help to identify whether the current build may only want PMD helpers but not PUD ones, as these sub-options will also check against the arch support over HAVE_ARCH_TRANSPARENT_HUGEPAGE[_PUD].
Note that having them depend on HAVE_ARCH_TRANSPARENT_HUGEPAGE[_PUD] is still some intermediate step. The best way is to have an option say "whether arch XXX supports PMD/PUD mappings" and so on. However let's leave that for later as that's the easy part. So far, we use these options to stably detect per-arch huge mapping support.
Signed-off-by Peter Xu <[email protected]>--- include/linux/huge_mm.h | 10 +++++++--- mm/Kconfig | 6 ++++++ 2 files changed, 13 insertions(+), 3 deletions(-)include/linux/huge_mm.h +7
-3@@ -96,14 +96,18 @@ extern struct kobj_attribute thpsize_shmem_enabled_attr;#define thp_vma_allowable_order (vma, vm_flags, tva_flags, order) \ (!!thp_vma_allowable_orders(vma, vm_flags, tva_flags, BIT(order))) -#ifdef CONFIG_PGTABLE_HAS_HUGE_LEAVES -#define HPAGE_PMD_SHIFT PMD_SHIFT +#ifdef CONFIG_PGTABLE_HAS_PUD_LEAVES #define HPAGE_PUD_SHIFT PUD_SHIFT#else -#define HPAGE_PMD_SHIFT ({ BUILD_BUG(); 0; })#define HPAGE_PUD_SHIFT ({ BUILD_BUG(); 0; })#endif +#ifdef CONFIG_PGTABLE_HAS_PMD_LEAVES +#define HPAGE_PMD_SHIFT PMD_SHIFT +#else +#define HPAGE_PMD_SHIFT ({ BUILD_BUG(); 0; }) +#endif +#define HPAGE_PMD_ORDER (HPAGE_PMD_SHIFT-PAGE_SHIFT)#define HPAGE_PMD_NR (1<<HPAGE_PMD_ORDER)#define HPAGE_PMD_MASK (~(HPAGE_PMD_SIZE - 1))mm/Kconfig +0
-0@@ -860,6 +860,12 @@ endif # TRANSPARENT_HUGEPAGE config PGTABLE_HAS_HUGE_LEAVES def_bool TRANSPARENT_HUGEPAGE || HUGETLB_PAGEQuoted 9 lines
mm/Kconfig endif # TRANSPARENT_HUGEPAGE · line 868
Quoted 6 lines
mm/Kconfig +6
-0+config PGTABLE_HAS_PMD_LEAVES + def_bool HAVE_ARCH_TRANSPARENT_HUGEPAGE && PGTABLE_HAS_HUGE_LEAVES + +config PGTABLE_HAS_PUD_LEAVES + def_bool HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD && PGTABLE_HAS_HUGE_LEAVES +What if an architecture has hugepages at PMD and/or PUD level and doesn't support THP ?
Quoted 1 line
What's the arch to be discussed here?
It is LOONGARCH and MIPS, they provide pud_leaf() that can return true even when they have no PUD.
Quoted 11 lines
The whole purpose of this series so far is trying to make some pmd/pud helpers that only defined with
CONFIG_THP=onto be available even if not. It means this series alone (or any future plan) shouldn't affect any arch that hasCONFIG_THP=offalways.But logically I think we should need some config option just to say "this arch supports pmd mappings" indeed, even if
CONFIG_THP=off. When that's there, we should perhaps add that option into this equation so PGTABLE_HAS_*_LEAVES will also be selected in that case.
Why is an option needed for that ? If pmd_leaf() returns always false, it means the arch doesn't support pmd mappings and if properly used all related code should fold away without a config option, shouldn't it ?
Peter Xu
reviewerRe: [PATCH RFC 2/6] mm: PGTABLE_HAS_P[MU]D_LEAVES config options
Quoted 3 lines · LEROY Christophe
Why is an option needed for that ? If
pmd_leaf()returns always false, it means the arch doesn't support pmd mappings and if properly used all related code should fold away without a config option, shouldn't it ?
It's not always easy to leverage an "if" clause there, IIUC. Take the case of when a driver wants to inject a pmd pfnmap, we may want something like:
if (pmd_leaf_supported())
inject_pmd_leaf(&pmd);We don't have a pmd entry to reference at the point of pmd_leaf_supported() when making the decision.
Thanks,
Signature · 1 line
Peter Xu
Peter Xu
reviewerRFC PATCH 3/6 mm/treewide: Make pgtable-generic.c THP agnostic
In reply to Peter XuMake pmd/pud helpers to rely on the new PGTABLE_HAS_*_LEAVES option, rather than THP alone, as THP is only one form of huge mapping.
--- arch/arm64/include/asm/pgtable.h | 6 ++-- arch/powerpc/include/asm/book3s/64/pgtable.h | 2 +- arch/powerpc/mm/book3s64/pgtable.c | 2 +- arch/riscv/include/asm/pgtable.h | 4 +-- arch/s390/include/asm/pgtable.h | 2 +- arch/s390/mm/pgtable.c | 4 +-- arch/sparc/mm/tlb.c | 2 +- arch/x86/mm/pgtable.c | 15 ++++----- include/linux/mm_types.h | 2 +- include/linux/pgtable.h | 4 +-- mm/memory.c | 2 +- mm/pgtable-generic.c | 32 ++++++++++---------- 12 files changed, 40 insertions(+), 37 deletions(-)
arch/arm64/include/asm/pgtable.h +4 -2
@@ -1105,7 +1105,7 @@ extern int __ptep_set_access_flags(struct vm_area_struct *vma, unsigned long address, pte_t *ptep, pte_t entry, int dirty); -#ifdef CONFIG_TRANSPARENT_HUGEPAGE +#ifdef CONFIG_PGTABLE_HAS_PMD_LEAVES #define __HAVE_ARCH_PMDP_SET_ACCESS_FLAGS static inline int pmdp_set_access_flags(struct vm_area_struct *vma, unsigned long address, pmd_t *pmdp,
@@ -1114,7 +1114,9 @@ static inline int pmdp_set_access_flags(struct vm_area_struct *vma,return __ptep_set_access_flags (vma, address, (pte_t *)pmdp,pmd_pte (entry), dirty); } +#endif +#ifdef CONFIG_PGTABLE_HAS_PUD_LEAVES static inline int pud_devmap (pud_t pud) {return 0 ;
@@ -1178,7 +1180,7 @@ static inline int __ptep_clear_flush_young(struct vm_area_struct *vma,return young; } -#ifdef CONFIG_TRANSPARENT_HUGEPAGE +#ifdef CONFIG_PGTABLE_HAS_PMD_LEAVES #define __HAVE_ARCH_PMDP_TEST_AND_CLEAR_YOUNG static inline int pmdp_test_and_clear_young(struct vm_area_struct *vma, unsigned long address,
arch/powerpc/include/asm/book3s/64/pgtable.h +1 -1
@@ -1119,7 +1119,7 @@ static inline bool pmd_access_permitted(pmd_t pmd, bool write)return pte_access_permitted (pmd_pte (pmd), write); } -#ifdef CONFIG_TRANSPARENT_HUGEPAGE +#ifdef CONFIG_PGTABLE_HAS_PMD_LEAVES extern pmd_t pfn_pmd (unsigned long pfn,pgprot_t pgprot);extern pud_t pfn_pud (unsigned long pfn,pgprot_t pgprot);extern pmd_t mk_pmd (struct page *page,pgprot_t pgprot);
arch/powerpc/mm/book3s64/pgtable.c +1 -1
@@ -37,7 +37,7 @@ EXPORT_SYMBOL(__pmd_frag_nr);unsigned long __pmd_frag_size_shift;EXPORT_SYMBOL (__pmd_frag_size_shift); -#ifdef CONFIG_TRANSPARENT_HUGEPAGE +#ifdef CONFIG_PGTABLE_HAS_HUGE_LEAVES /* * This is called when relaxing access to a hugepage. It's also called in the page * fault path when we don't hit any of the major fault cases, ie, a minor
arch/riscv/include/asm/pgtable.h +2 -2
@@ -752,7 +752,7 @@ static inline bool pud_user_accessible_page(pud_t pud) }#endif -#ifdef CONFIG_TRANSPARENT_HUGEPAGE +#ifdef CONFIG_PGTABLE_HAS_PMD_LEAVES static inline int pmd_trans_huge (pmd_t pmd) {return pmd_leaf (pmd);
@@ -802,7 +802,7 @@ static inline pmd_t pmdp_establish(struct vm_area_struct *vma,#define pmdp_collapse_flush pmdp_collapse_flushextern pmd_t pmdp_collapse_flush (struct vm_area_struct *vma,unsigned long address,pmd_t *pmdp); -#endif /* CONFIG_TRANSPARENT_HUGEPAGE */ +#endif /* CONFIG_PGTABLE_HAS_PMD_LEAVES */ /* * Encode/decode swap entries and swap PTEs. Swap PTEs are all PTEs that
arch/s390/include/asm/pgtable.h +1 -1
@@ -1710,7 +1710,7 @@ pmd_t pmdp_xchg_direct(struct mm_struct *, unsigned long, pmd_t *, pmd_t);pmd_t pmdp_xchg_lazy (struct mm_struct *,unsigned long ,pmd_t *,pmd_t );pud_t pudp_xchg_direct (struct mm_struct *,unsigned long ,pud_t *,pud_t ); -#ifdef CONFIG_TRANSPARENT_HUGEPAGE +#ifdef CONFIG_PGTABLE_HAS_PMD_LEAVES #define __HAVE_ARCH_PGTABLE_DEPOSIT void pgtable_trans_huge_deposit(struct mm_struct *mm, pmd_t *pmdp,
arch/s390/mm/pgtable.c +2 -2
@@ -561,7 +561,7 @@ pud_t pudp_xchg_direct(struct mm_struct *mm, unsigned long addr, }EXPORT_SYMBOL (pudp_xchg_direct); -#ifdef CONFIG_TRANSPARENT_HUGEPAGE +#ifdef CONFIG_PGTABLE_HAS_PMD_LEAVES void pgtable_trans_huge_deposit (struct mm_struct *mm,pmd_t *pmdp,pgtable_t pgtable) {
@@ -600,7 +600,7 @@ pgtable_t pgtable_trans_huge_withdraw(struct mm_struct *mm, pmd_t *pmdp)set_pte (ptep,__pte (_PAGE_INVALID));return pgtable; } -#endif /* CONFIG_TRANSPARENT_HUGEPAGE */ +#endif /* CONFIG_PGTABLE_HAS_PMD_LEAVES */ #ifdef CONFIG_PGSTE void ptep_set_pte_at(struct mm_struct *mm, unsigned long addr,
arch/sparc/mm/tlb.c +1 -1
@@ -143,7 +143,7 @@ void tlb_batch_add(struct mm_struct *mm, unsigned long vaddr,tlb_batch_add_one (mm, vaddr,pte_exec (orig), hugepage_shift); } -#ifdef CONFIG_TRANSPARENT_HUGEPAGE +#ifdef CONFIG_PGTABLE_HAS_PMD_LEAVES static void tlb_batch_pmd_scan (struct mm_struct *mm,unsigned long vaddr,pmd_t pmd) {
arch/x86/mm/pgtable.c +8 -7
@@ -511,7 +511,7 @@ int ptep_set_access_flags(struct vm_area_struct *vma,return changed; } -#ifdef CONFIG_TRANSPARENT_HUGEPAGE +#ifdef CONFIG_PGTABLE_HAS_PMD_LEAVES int pmdp_set_access_flags (struct vm_area_struct *vma,unsigned long address,pmd_t *pmdp,pmd_t entry,int dirty)
@@ -532,7 +532,9 @@ int pmdp_set_access_flags(struct vm_area_struct *vma,return changed; } +#endif /* PGTABLE_HAS_PMD_LEAVES */ +#ifdef CONFIG_PGTABLE_HAS_PUD_LEAVES int pudp_set_access_flags (struct vm_area_struct *vma,unsigned long address,pud_t *pudp,pud_t entry,int dirty) {
@@ -552,7 +554,7 @@ int pudp_set_access_flags(struct vm_area_struct *vma, unsigned long address, return changed; } -#endif +#endif /* PGTABLE_HAS_PUD_LEAVES */ int ptep_test_and_clear_young (struct vm_area_struct *vma,unsigned long addr,pte_t *ptep)
@@ -566,7 +568,7 @@ int ptep_test_and_clear_young(struct vm_area_struct *vma,return ret; } -#if defined(CONFIG_TRANSPARENT_HUGEPAGE ) || defined(CONFIG_ARCH_HAS_NONLEAF_PMD_YOUNG ) +#if defined(CONFIG_PGTABLE_HAS_PMD_LEAVES ) || defined(CONFIG_ARCH_HAS_NONLEAF_PMD_YOUNG )int pmdp_test_and_clear_young (struct vm_area_struct *vma,unsigned long addr,pmd_t *pmdp) {
@@ -580,7 +582,7 @@ int pmdp_test_and_clear_young(struct vm_area_struct *vma, }#endif -#ifdef CONFIG_TRANSPARENT_HUGEPAGE +#ifdef CONFIG_PGTABLE_HAS_PUD_LEAVES int pudp_test_and_clear_young (struct vm_area_struct *vma,unsigned long addr,pud_t *pudp) {
@@ -613,7 +615,7 @@ int ptep_clear_flush_young(struct vm_area_struct *vma,return ptep_test_and_clear_young (vma, address, ptep); } -#ifdef CONFIG_TRANSPARENT_HUGEPAGE +#ifdef CONFIG_PGTABLE_HAS_PMD_LEAVES int pmdp_clear_flush_young (struct vm_area_struct *vma,unsigned long address,pmd_t *pmdp) {
@@ -641,8 +643,7 @@ pmd_t pmdp_invalidate_ad(struct vm_area_struct *vma, unsigned long address, }#endif -#if defined(CONFIG_TRANSPARENT_HUGEPAGE ) && \ - defined(CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD ) +#ifdef CONFIG_PGTABLE_HAS_PUD_LEAVES pud_t pudp_invalidate (struct vm_area_struct *vma,unsigned long address,pud_t *pudp) {
include/linux/mm_types.h +1 -1
@@ -942,7 +942,7 @@ struct mm_struct {#ifdef CONFIG_MMU_NOTIFIER struct mmu_notifier_subscriptions *notifier_subscriptions;#endif -#if defined(CONFIG_TRANSPARENT_HUGEPAGE ) && !USE_SPLIT_PMD_PTLOCKS +#if defined(CONFIG_PGTABLE_HAS_PMD_LEAVES ) && !USE_SPLIT_PMD_PTLOCKS pgtable_t pmd_huge_pte;/* protected by page_table_lock */ #endif #ifdef CONFIG_NUMA_BALANCING
include/linux/pgtable.h +2 -2
@@ -362,7 +362,7 @@ static inline int ptep_test_and_clear_young(struct vm_area_struct *vma,#endif #ifndef __HAVE_ARCH_PMDP_TEST_AND_CLEAR_YOUNG -#if defined(CONFIG_TRANSPARENT_HUGEPAGE ) || defined(CONFIG_ARCH_HAS_NONLEAF_PMD_YOUNG ) +#if defined(CONFIG_PGTABLE_HAS_PMD_LEAVES ) || defined(CONFIG_ARCH_HAS_NONLEAF_PMD_YOUNG )static inline int pmdp_test_and_clear_young (struct vm_area_struct *vma,unsigned long address,pmd_t *pmdp)
@@ -383,7 +383,7 @@ static inline int pmdp_test_and_clear_young(struct vm_area_struct *vma,BUILD_BUG ();return 0 ; } -#endif /* CONFIG_TRANSPARENT_HUGEPAGE || CONFIG_ARCH_HAS_NONLEAF_PMD_YOUNG */ +#endif /* CONFIG_PGTABLE_HAS_PMD_LEAVES || CONFIG_ARCH_HAS_NONLEAF_PMD_YOUNG */ #endif #ifndef __HAVE_ARCH_PTEP_CLEAR_YOUNG_FLUSH
mm/memory.c +1 -1
@@ -666,7 +666,7 @@ struct folio *vm_normal_folio(struct vm_area_struct *vma, unsigned long addr,return NULL ; } -#ifdef CONFIG_TRANSPARENT_HUGEPAGE +#ifdef CONFIG_PGTABLE_HAS_PMD_LEAVES struct page *vm_normal_page_pmd (struct vm_area_struct *vma,unsigned long addr,pmd_t pmd) {
mm/pgtable-generic.c +16 -16
@@ -103,7 +103,7 @@ pte_t ptep_clear_flush(struct vm_area_struct *vma, unsigned long address, }#endif -#ifdef CONFIG_TRANSPARENT_HUGEPAGE +#ifdef CONFIG_PGTABLE_HAS_PMD_LEAVES #ifndef __HAVE_ARCH_PMDP_SET_ACCESS_FLAGS int pmdp_set_access_flags(struct vm_area_struct *vma,
@@ -145,20 +145,6 @@ pmd_t pmdp_huge_clear_flush(struct vm_area_struct *vma, unsigned long address,flush_pmd_tlb_range (vma, address, address +HPAGE_PMD_SIZE ); return pmd; } - -#ifdef CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD -pud_t pudp_huge_clear_flush (struct vm_area_struct *vma,unsigned long address, -pud_t *pudp) -{ -pud_t pud; - -VM_BUG_ON (address & ~HPAGE_PUD_MASK ); -VM_BUG_ON (!pud_trans_huge (*pudp) && !pud_devmap (*pudp)); - pud =pudp_huge_get_and_clear (vma->vm_mm, address, pudp); -flush_pud_tlb_range (vma, address, address +HPAGE_PUD_SIZE ); -return pud; -} -#endif #endif #ifndef __HAVE_ARCH_PGTABLE_DEPOSIT
@@ -252,7 +238,21 @@ void pte_free_defer(struct mm_struct *mm, pgtable_t pgtable)call_rcu (&page->rcu_head, pte_free_now); }#endif /* pte_free_defer */ -#endif /* CONFIG_TRANSPARENT_HUGEPAGE */ +#endif /* CONFIG_PGTABLE_HAS_PMD_LEAVES */ + +#ifdef CONFIG_PGTABLE_HAS_PUD_LEAVES +pud_t pudp_huge_clear_flush (struct vm_area_struct *vma,unsigned long address, +pud_t *pudp) +{ +pud_t pud; + +VM_BUG_ON (address & ~HPAGE_PUD_MASK ); +VM_BUG_ON (!pud_trans_huge (*pudp) && !pud_devmap (*pudp)); + pud =pudp_huge_get_and_clear (vma->vm_mm, address, pudp); +flush_pud_tlb_range (vma, address, address +HPAGE_PUD_SIZE ); +return pud; +} +#endif /* CONFIG_PGTABLE_HAS_PUD_LEAVES */ #if defined(CONFIG_GUP_GET_PXX_LOW_HIGH ) && \ (defined(CONFIG_SMP ) || defined(CONFIG_PREEMPT_RCU ))
Signature · 1 line
2.45.0
Peter Xu
reviewerRFC PATCH 4/6 mm: Move huge mapping declarations from internal.h to huge_mm.h
In reply to Peter XuMost of the huge mapping relevant helpers are declared in huge_mm.h, not internal.h. Move the only few from internal.h into huge_mm.h.
Here to move pmd_needs_soft_dirty_wp() over, we'll also need to move vma_soft_dirty_enabled() into mm.h as it'll be needed in two headers later (internal.h, huge_mm.h).
--- include/linux/huge_mm.h | 10 ++++++++++ include/linux/mm.h | 18 ++++++++++++++++++ mm/internal.h | 33 --------------------------------- 3 files changed, 28 insertions(+), 33 deletions(-)
include/linux/huge_mm.h +10 -0
@@ -8,6 +8,11 @@#include <linux/fs.h> /* only for vma_is_dax() */ #include <linux/kobject.h> +void touch_pud (struct vm_area_struct *vma,unsigned long addr, +pud_t *pud,bool write); +void touch_pmd (struct vm_area_struct *vma,unsigned long addr, +pmd_t *pmd,bool write); +pmd_t maybe_pmd_mkwrite (pmd_t pmd,struct vm_area_struct *vma);vm_fault_t do_huge_pmd_anonymous_page (struct vm_fault *vmf); int copy_huge_pmd(struct mm_struct *dst_mm, struct mm_struct *src_mm, pmd_t *dst_pmd, pmd_t *src_pmd, unsigned long addr,
@@ -629,4 +634,9 @@ static inline int split_folio_to_order(struct folio *folio, int new_order)#define split_folio_to_list (f, l) split_folio_to_list_to_order(f, l, 0)#define split_folio (f) split_folio_to_order(f, 0) +static inline bool pmd_needs_soft_dirty_wp (struct vm_area_struct *vma,pmd_t pmd) +{ +return vma_soft_dirty_enabled (vma) && !pmd_soft_dirty (pmd); +} +#endif /* _LINUX_HUGE_MM_H */
include/linux/mm.h +18 -0
@@ -1117,6 +1117,24 @@ static inline unsigned int folio_order(struct folio *folio)return folio->_flags_1 &0xff ; } +static inline bool vma_soft_dirty_enabled (struct vm_area_struct *vma) +{ +/* +* NOTE: we must check this before VM_SOFTDIRTY on soft-dirty +* enablements, because when without soft-dirty being compiled in, +* VM_SOFTDIRTY is defined as 0x0, then !(vm_flags & VM_SOFTDIRTY) +* will be constantly true. +*/ +if (!IS_ENABLED (CONFIG_MEM_SOFT_DIRTY )) +return false; + +/* +* Soft-dirty is kind of special: its tracking is enabled when the +* vma flags not set. +*/ +return !(vma->vm_flags &VM_SOFTDIRTY ); +} +#include <linux/huge_mm.h> /*
mm/internal.h +0 -33
@@ -917,8 +917,6 @@ bool need_mlock_drain(int cpu);void mlock_drain_local (void );void mlock_drain_remote (int cpu); -extern pmd_t maybe_pmd_mkwrite (pmd_t pmd,struct vm_area_struct *vma); -/** * vma_address - Find the virtual address a page range is mapped at * @vma: The vma which maps this object.
@@ -1229,14 +1227,6 @@ int migrate_device_coherent_page(struct page *page);int __must_check try_grab_folio (struct folio *folio,int refs,unsigned int flags); -/* -* mm/huge_memory.c -*/ -void touch_pud (struct vm_area_struct *vma,unsigned long addr, -pud_t *pud,bool write); -void touch_pmd (struct vm_area_struct *vma,unsigned long addr, -pmd_t *pmd,bool write); -/* * mm/mmap.c */
@@ -1342,29 +1332,6 @@ static __always_inline void vma_set_range(struct vm_area_struct *vma, vma->vm_pgoff = pgoff; } -static inline bool vma_soft_dirty_enabled (struct vm_area_struct *vma) -{ -/* -* NOTE: we must check this before VM_SOFTDIRTY on soft-dirty -* enablements, because when without soft-dirty being compiled in, -* VM_SOFTDIRTY is defined as 0x0, then !(vm_flags & VM_SOFTDIRTY) -* will be constantly true. -*/ -if (!IS_ENABLED (CONFIG_MEM_SOFT_DIRTY )) -return false; - -/* -* Soft-dirty is kind of special: its tracking is enabled when the -* vma flags not set. -*/ -return !(vma->vm_flags &VM_SOFTDIRTY ); -} - -static inline bool pmd_needs_soft_dirty_wp (struct vm_area_struct *vma,pmd_t pmd) -{ -return vma_soft_dirty_enabled (vma) && !pmd_soft_dirty (pmd); -} -static inline bool pte_needs_soft_dirty_wp (struct vm_area_struct *vma,pte_t pte) {return vma_soft_dirty_enabled (vma) && !pte_soft_dirty (pte);
Signature · 1 line
2.45.0
Peter Xu
reviewerRFC PATCH 5/6 mm/huge_mapping: Create huge_mapping_pxx.c
In reply to Peter XuAt some point, we need to decouple "huge mapping" with THP, for any non-THP huge mappings in the future (hugetlb, pfnmap, etc..). This is the first step towards it.
Or say, we already started to do this when PGTABLE_HAS_HUGE_LEAVES option was introduced: that is the first thing Linux start to describe LEAVEs rather than THPs when it is about huge mappings. Before that, mostly any huge mapping will have THP involved, like devmap. Hugetlb is special only because we duplicated the whole world there, but we also have a demand to decouple that now.
Linux used to have huge_memory.c which only compiles with THP enabled, I wished it was called thp.c from the start. In reality, it contains more than processing THP: any huge mapping (even if not falling into THP category) will be able to leverage many of these helpers, but unfortunately this file is not compiled if !THP. These helpers are normally only about the pgtable operations, which may not be directly relevant to what type of huge folio (e.g. THP) underneath, or perhaps even if there's no vmemmap to back it.
It's better we move them out of THP world.
Create a new set of files huge_mapping_p[mu]d.c. This patch starts to move quite a few essential helpers from huge_memory.c into these new files, so that they'll start to work and compile rely on PGTABLE_HAS_PXX_LEAVES rather than THP. Split them into two files by nature so that e.g. archs that only supports PMD huge mapping can avoid compiling the whole -pud file, with the hope to reduce the size of object compiled and linked.
No functional change intended, but only code movement. Said that, there will be some "ifdef" machinery changes to pass all kinds of compilations.
--- include/linux/huge_mm.h | 318 +++++--- include/linux/pgtable.h | 23 +- include/trace/events/huge_mapping.h | 41 + include/trace/events/thp.h | 28 - mm/Makefile | 2 + mm/huge_mapping_pmd.c | 979 +++++++++++++++++++++++ mm/huge_mapping_pud.c | 235 ++++++ mm/huge_memory.c | 1125 +-------------------------- 8 files changed, 1472 insertions(+), 1279 deletions(-) create mode 100644 include/trace/events/huge_mapping.h create mode 100644 mm/huge_mapping_pmd.c create mode 100644 mm/huge_mapping_pud.c
include/linux/huge_mm.h +200 -118
@@ -8,43 +8,214 @@#include <linux/fs.h> /* only for vma_is_dax() */ #include <linux/kobject.h> +#ifdef CONFIG_PGTABLE_HAS_PUD_LEAVES +void huge_pud_set_accessed (struct vm_fault *vmf,pud_t orig_pud);void touch_pud (struct vm_area_struct *vma,unsigned long addr,pud_t *pud,bool write); -void touch_pmd (struct vm_area_struct *vma,unsigned long addr, -pmd_t *pmd,bool write); -pmd_t maybe_pmd_mkwrite (pmd_t pmd,struct vm_area_struct *vma); -vm_fault_t do_huge_pmd_anonymous_page (struct vm_fault *vmf); -int copy_huge_pmd (struct mm_struct *dst_mm,struct mm_struct *src_mm, -pmd_t *dst_pmd,pmd_t *src_pmd,unsigned long addr, -struct vm_area_struct *dst_vma,struct vm_area_struct *src_vma); -void huge_pmd_set_accessed (struct vm_fault *vmf);int copy_huge_pud (struct mm_struct *dst_mm,struct mm_struct *src_mm,pud_t *dst_pud,pud_t *src_pud,unsigned long addr,struct vm_area_struct *vma); +int zap_huge_pud (struct mmu_gather *tlb,struct vm_area_struct *vma,pud_t *pud, +unsigned long addr); +int change_huge_pud (struct mmu_gather *tlb,struct vm_area_struct *vma, +pud_t *pudp,unsigned long addr,pgprot_t newprot, +unsigned long cp_flags); +void __split_huge_pud (struct vm_area_struct *vma,pud_t *pud, +unsigned long address); +spinlock_t *__pud_trans_huge_lock (pud_t *pud,struct vm_area_struct *vma); -#ifdef CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD -void huge_pud_set_accessed (struct vm_fault *vmf,pud_t orig_pud); -#else -static inline void huge_pud_set_accessed (struct vm_fault *vmf,pud_t orig_pud) +static inline spinlock_t * +pud_trans_huge_lock (pud_t *pud,struct vm_area_struct *vma) { +if (pud_trans_huge (*pud) ||pud_devmap (*pud)) +return __pud_trans_huge_lock (pud, vma); +else +return NULL ; } -#endif +#define split_huge_pud (__vma, __pud, __address) \ + do { \ + pud_t *____pud = (__pud); \ + if (pud_trans_huge(*____pud) || pud_devmap(*____pud)) \ + __split_huge_pud(__vma, __pud, __address); \ + } while (0) +#else /* CONFIG_PGTABLE_HAS_PUD_LEAVES */ +static inline void +huge_pud_set_accessed (struct vm_fault *vmf,pud_t orig_pud) +{ +} + +static inline int +change_huge_pud (struct mmu_gather *tlb,struct vm_area_struct *vma, +pud_t *pudp,unsigned long addr,pgprot_t newprot, +unsigned long cp_flags) +{ +return 0 ; +} + +static inline spinlock_t * +pud_trans_huge_lock (pud_t *pud, +struct vm_area_struct *vma) +{ +return NULL ; +} + +static inline void +touch_pud (struct vm_area_struct *vma,unsigned long addr, +pud_t *pud,bool write) +{ +} + +static inline int +copy_huge_pud (struct mm_struct *dst_mm,struct mm_struct *src_mm, +pud_t *dst_pud,pud_t *src_pud,unsigned long addr, +struct vm_area_struct *vma) +{ +return 0 ; +} + +static inline int +zap_huge_pud (struct mmu_gather *tlb,struct vm_area_struct *vma,pud_t *pud, +unsigned long addr) +{ +return 0 ; +} + +static inline void +__split_huge_pud (struct vm_area_struct *vma,pud_t *pud,unsigned long address) +{ +} + +#define split_huge_pud (__vma, __pud, __address) do {} while (0) +#endif /* CONFIG_PGTABLE_HAS_PUD_LEAVES */ + +#ifdef CONFIG_PGTABLE_HAS_PMD_LEAVES +void touch_pmd (struct vm_area_struct *vma,unsigned long addr, +pmd_t *pmd,bool write); +pmd_t maybe_pmd_mkwrite (pmd_t pmd,struct vm_area_struct *vma); +int copy_huge_pmd (struct mm_struct *dst_mm,struct mm_struct *src_mm, +pmd_t *dst_pmd,pmd_t *src_pmd,unsigned long addr, +struct vm_area_struct *dst_vma,struct vm_area_struct *src_vma); +void huge_pmd_set_accessed (struct vm_fault *vmf);vm_fault_t do_huge_pmd_wp_page (struct vm_fault *vmf); -bool madvise_free_huge_pmd (struct mmu_gather *tlb,struct vm_area_struct *vma, -pmd_t *pmd,unsigned long addr,unsigned long next);int zap_huge_pmd (struct mmu_gather *tlb,struct vm_area_struct *vma,pmd_t *pmd,unsigned long addr); -int zap_huge_pud (struct mmu_gather *tlb,struct vm_area_struct *vma,pud_t *pud, -unsigned long addr);bool move_huge_pmd (struct vm_area_struct *vma,unsigned long old_addr,unsigned long new_addr,pmd_t *old_pmd,pmd_t *new_pmd);int change_huge_pmd (struct mmu_gather *tlb,struct vm_area_struct *vma,pmd_t *pmd,unsigned long addr,pgprot_t newprot,unsigned long cp_flags); +void __split_huge_pmd (struct vm_area_struct *vma,pmd_t *pmd, +unsigned long address,bool freeze,struct folio *folio); +void split_huge_pmd_locked (struct vm_area_struct *vma,unsigned long address, +pmd_t *pmd,bool freeze,struct folio *folio); +void split_huge_pmd_address (struct vm_area_struct *vma,unsigned long address, +bool freeze,struct folio *folio); +spinlock_t *__pmd_trans_huge_lock (pmd_t *pmd,struct vm_area_struct *vma); +bool can_change_pmd_writable (struct vm_area_struct *vma,unsigned long addr, +pmd_t pmd); +void zap_deposited_table (struct mm_struct *mm,pmd_t *pmd); + +static inline int is_swap_pmd (pmd_t pmd) +{ +return !pmd_none (pmd) && !pmd_present (pmd); +} + +/* mmap_lock must be held on entry */ +static inline spinlock_t * +pmd_trans_huge_lock (pmd_t *pmd,struct vm_area_struct *vma) +{ +if (is_swap_pmd (*pmd) ||pmd_trans_huge (*pmd) ||pmd_devmap (*pmd)) +return __pmd_trans_huge_lock (pmd, vma); +else +return NULL ; +} + +#define split_huge_pmd (__vma, __pmd, __address) \ + do { \ + pmd_t *____pmd = (__pmd); \ + if (is_swap_pmd(*____pmd) || pmd_is_leaf(*____pmd)) \ + __split_huge_pmd(__vma, __pmd, __address, \ + false, NULL); \ + } while (0) +#else /* CONFIG_PGTABLE_HAS_PMD_LEAVES */ +static inline spinlock_t * +pmd_trans_huge_lock (pmd_t *pmd,struct vm_area_struct *vma) +{ +return NULL ; +} + +static inline int is_swap_pmd (pmd_t pmd) +{ +return 0 ; +} +static inline void +__split_huge_pmd (struct vm_area_struct *vma,pmd_t *pmd, +unsigned long address,bool freeze,struct folio *folio) +{ +} +#define split_huge_pmd (__vma, __pmd, __address) do {} while (0) + +static inline int +copy_huge_pmd (struct mm_struct *dst_mm,struct mm_struct *src_mm, +pmd_t *dst_pmd,pmd_t *src_pmd,unsigned long addr, +struct vm_area_struct *dst_vma,struct vm_area_struct *src_vma) +{ +return 0 ; +} + +static inline int +zap_huge_pmd (struct mmu_gather *tlb,struct vm_area_struct *vma,pmd_t *pmd, +unsigned long addr) +{ +return 0 ; +} + +static inline vm_fault_t +do_huge_pmd_wp_page (struct vm_fault *vmf) +{ +return 0 ; +} + +static inline void +huge_pmd_set_accessed (struct vm_fault *vmf) +{ +} + +static inline int +change_huge_pmd (struct mmu_gather *tlb,struct vm_area_struct *vma, +pmd_t *pmd,unsigned long addr,pgprot_t newprot, +unsigned long cp_flags) +{ +return 0 ; +} + +static inline bool +move_huge_pmd (struct vm_area_struct *vma,unsigned long old_addr, +unsigned long new_addr,pmd_t *old_pmd,pmd_t *new_pmd) +{ +return false; +} + +static inline void +split_huge_pmd_locked (struct vm_area_struct *vma,unsigned long address, +pmd_t *pmd,bool freeze,struct folio *folio) +{ +} + +static inline void +split_huge_pmd_address (struct vm_area_struct *vma,unsigned long address, +bool freeze,struct folio *folio) +{ +} +#endif /* CONFIG_PGTABLE_HAS_PMD_LEAVES */ + +bool madvise_free_huge_pmd (struct mmu_gather *tlb,struct vm_area_struct *vma, +pmd_t *pmd,unsigned long addr,unsigned long next);vm_fault_t vmf_insert_pfn_pmd (struct vm_fault *vmf,pfn_t pfn,bool write);vm_fault_t vmf_insert_pfn_pud (struct vm_fault *vmf,pfn_t pfn,bool write); +struct folio *mm_get_huge_zero_folio (struct mm_struct *mm);enum transparent_hugepage_flag {TRANSPARENT_HUGEPAGE_UNSUPPORTED ,
@@ -130,6 +301,9 @@ extern unsigned long huge_anon_orders_always;extern unsigned long huge_anon_orders_madvise;extern unsigned long huge_anon_orders_inherit; +void __split_huge_zero_page_pmd (struct vm_area_struct *vma, +unsigned long haddr,pmd_t *pmd); +static inline bool hugepage_global_enabled (void ) { return transparent_hugepage_flags &
@@ -332,44 +506,6 @@ static inline int split_huge_page(struct page *page) }void deferred_split_folio (struct folio *folio); -void __split_huge_pmd (struct vm_area_struct *vma,pmd_t *pmd, -unsigned long address,bool freeze,struct folio *folio); - -#define split_huge_pmd (__vma, __pmd, __address) \ - do { \ - pmd_t *____pmd = (__pmd); \ - if (is_swap_pmd(*____pmd) || pmd_trans_huge(*____pmd) \ - || pmd_devmap(*____pmd)) \ - __split_huge_pmd(__vma, __pmd, __address, \ - false, NULL); \ - } while (0) - - -void split_huge_pmd_address (struct vm_area_struct *vma,unsigned long address, -bool freeze,struct folio *folio); - -void __split_huge_pud (struct vm_area_struct *vma,pud_t *pud, -unsigned long address); - -#ifdef CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD -int change_huge_pud (struct mmu_gather *tlb,struct vm_area_struct *vma, -pud_t *pudp,unsigned long addr,pgprot_t newprot, -unsigned long cp_flags); -#else -static inline int -change_huge_pud (struct mmu_gather *tlb,struct vm_area_struct *vma, -pud_t *pudp,unsigned long addr,pgprot_t newprot, -unsigned long cp_flags) {return 0 ; } -#endif - -#define split_huge_pud (__vma, __pud, __address) \ - do { \ - pud_t *____pud = (__pud); \ - if (pud_trans_huge(*____pud) \ - || pud_devmap(*____pud)) \ - __split_huge_pud(__vma, __pud, __address); \ - } while (0) -int hugepage_madvise (struct vm_area_struct *vma,unsigned long *vm_flags,int advice); int madvise_collapse(struct vm_area_struct *vma,
@@ -377,31 +513,6 @@ int madvise_collapse(struct vm_area_struct *vma, unsigned long start, unsigned long end);void vma_adjust_trans_huge (struct vm_area_struct *vma,unsigned long start,unsigned long end,long adjust_next); -spinlock_t *__pmd_trans_huge_lock (pmd_t *pmd,struct vm_area_struct *vma); -spinlock_t *__pud_trans_huge_lock (pud_t *pud,struct vm_area_struct *vma); - -static inline int is_swap_pmd (pmd_t pmd) -{ -return !pmd_none (pmd) && !pmd_present (pmd); -} - -/* mmap_lock must be held on entry */ -static inline spinlock_t *pmd_trans_huge_lock (pmd_t *pmd, -struct vm_area_struct *vma) -{ -if (is_swap_pmd (*pmd) ||pmd_trans_huge (*pmd) ||pmd_devmap (*pmd)) -return __pmd_trans_huge_lock (pmd, vma); -else -return NULL ; -} -static inline spinlock_t *pud_trans_huge_lock (pud_t *pud, -struct vm_area_struct *vma) -{ -if (pud_trans_huge (*pud) ||pud_devmap (*pud)) -return __pud_trans_huge_lock (pud, vma); -else -return NULL ; -}/** * folio_test_pmd_mappable - Can we map this folio with a PMD?
@@ -416,6 +527,7 @@ struct page *follow_devmap_pmd(struct vm_area_struct *vma, unsigned long addr, pmd_t *pmd, int flags, struct dev_pagemap **pgmap);vm_fault_t do_huge_pmd_numa_page (struct vm_fault *vmf); +vm_fault_t do_huge_pmd_anonymous_page (struct vm_fault *vmf);extern struct folio *huge_zero_folio;extern unsigned long huge_zero_pfn;
@@ -445,13 +557,17 @@ static inline bool thp_migration_supported(void) return IS_ENABLED(CONFIG_ARCH_ENABLE_THP_MIGRATION); } -void split_huge_pmd_locked (struct vm_area_struct *vma,unsigned long address, -pmd_t *pmd,bool freeze,struct folio *folio);bool unmap_huge_pmd_locked (struct vm_area_struct *vma,unsigned long addr,pmd_t *pmdp,struct folio *folio);#else /* CONFIG_TRANSPARENT_HUGEPAGE */ +static inline void +__split_huge_zero_page_pmd (struct vm_area_struct *vma, +unsigned long haddr,pmd_t *pmd) +{ +} +static inline bool folio_test_pmd_mappable (struct folio *folio) {return false;
@@ -505,16 +621,6 @@ static inline int split_huge_page(struct page *page)return 0 ; }static inline void deferred_split_folio (struct folio *folio) {} -#define split_huge_pmd (__vma, __pmd, __address) \ - do { } while (0) - -static inline void __split_huge_pmd (struct vm_area_struct *vma,pmd_t *pmd, -unsigned long address,bool freeze,struct folio *folio) {} -static inline void split_huge_pmd_address (struct vm_area_struct *vma, -unsigned long address,bool freeze,struct folio *folio) {} -static inline void split_huge_pmd_locked (struct vm_area_struct *vma, -unsigned long address,pmd_t *pmd, -bool freeze,struct folio *folio) {} static inline bool unmap_huge_pmd_locked(struct vm_area_struct *vma, unsigned long addr, pmd_t *pmdp,
@@ -523,9 +629,6 @@ static inline bool unmap_huge_pmd_locked(struct vm_area_struct *vma,return false; } -#define split_huge_pud (__vma, __pmd, __address) \ - do { } while (0) -static inline int hugepage_madvise (struct vm_area_struct *vma,unsigned long *vm_flags,int advice) {
@@ -545,20 +648,6 @@ static inline void vma_adjust_trans_huge(struct vm_area_struct *vma, long adjust_next) { } -static inline int is_swap_pmd (pmd_t pmd) -{ -return 0 ; -} -static inline spinlock_t *pmd_trans_huge_lock (pmd_t *pmd, -struct vm_area_struct *vma) -{ -return NULL ; -} -static inline spinlock_t *pud_trans_huge_lock (pud_t *pud, -struct vm_area_struct *vma) -{ -return NULL ; -}static inline vm_fault_t do_huge_pmd_numa_page (struct vm_fault *vmf) {
@@ -606,15 +695,8 @@ static inline int next_order(unsigned long *orders, int prev)return 0 ; } -static inline void __split_huge_pud (struct vm_area_struct *vma,pud_t *pud, -unsigned long address) -{ -} - -static inline int change_huge_pud (struct mmu_gather *tlb, -struct vm_area_struct *vma,pud_t *pudp, -unsigned long addr,pgprot_t newprot, -unsigned long cp_flags) +static inline vm_fault_t +do_huge_pmd_anonymous_page (struct vm_fault *vmf) {return 0 ; }
include/linux/pgtable.h +13 -10
@@ -628,8 +628,8 @@ static inline pud_t pudp_huge_get_and_clear(struct mm_struct *mm,#endif /* __HAVE_ARCH_PUDP_HUGE_GET_AND_CLEAR */ #endif /* CONFIG_TRANSPARENT_HUGEPAGE */ -#ifdef CONFIG_TRANSPARENT_HUGEPAGE -#ifndef __HAVE_ARCH_PMDP_HUGE_GET_AND_CLEAR_FULL +#if defined(CONFIG_PGTABLE_HAS_PMD_LEAVES ) && \ + !defined(__HAVE_ARCH_PMDP_HUGE_GET_AND_CLEAR_FULL)static inline pmd_t pmdp_huge_get_and_clear_full (struct vm_area_struct *vma,unsigned long address,pmd_t *pmdp,int full)
@@ -638,14 +638,14 @@ static inline pmd_t pmdp_huge_get_and_clear_full(struct vm_area_struct *vma, }#endif -#ifndef __HAVE_ARCH_PUDP_HUGE_GET_AND_CLEAR_FULL +#if defined(CONFIG_PGTABLE_HAS_PUD_LEAVES ) && \ + !defined(__HAVE_ARCH_PUDP_HUGE_GET_AND_CLEAR_FULL)static inline pud_t pudp_huge_get_and_clear_full (struct vm_area_struct *vma,unsigned long address,pud_t *pudp,int full) {return pudp_huge_get_and_clear (vma->vm_mm, address, pudp); } -#endif #endif /* CONFIG_TRANSPARENT_HUGEPAGE */ #ifndef __HAVE_ARCH_PTEP_GET_AND_CLEAR_FULL
@@ -894,9 +894,9 @@ static inline void pmdp_set_wrprotect(struct mm_struct *mm, }#endif /* CONFIG_TRANSPARENT_HUGEPAGE */ #endif + #ifndef __HAVE_ARCH_PUDP_SET_WRPROTECT -#ifdef CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD -#ifdef CONFIG_TRANSPARENT_HUGEPAGE +#ifdef CONFIG_PGTABLE_HAS_PUD_LEAVES static inline void pudp_set_wrprotect(struct mm_struct *mm, unsigned long address, pud_t *pudp) {
@@ -910,8 +910,7 @@ static inline void pudp_set_wrprotect(struct mm_struct *mm, {BUILD_BUG (); } -#endif /* CONFIG_TRANSPARENT_HUGEPAGE */ -#endif /* CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD */ +#endif /* CONFIG_PGTABLE_HAS_PUD_LEAVES */ #endif #ifndef pmdp_collapse_flush
@@ -1735,7 +1734,6 @@ static inline int pmd_free_pte_page(pmd_t *pmd, unsigned long addr)#endif /* CONFIG_HAVE_ARCH_HUGE_VMAP */ #ifndef __HAVE_ARCH_FLUSH_PMD_TLB_RANGE -#ifdef CONFIG_TRANSPARENT_HUGEPAGE /* * ARCHes with special requirements for evicting THP backing TLB entries can * implement this. Otherwise also, it can help optimize normal TLB flush in
@@ -1745,10 +1743,15 @@ static inline int pmd_free_pte_page(pmd_t *pmd, unsigned long addr)* invalidate the entire TLB which is not desirable. * e.g. see arch/arc: flush_pmd_tlb_range */ +#ifdef CONFIG_PGTABLE_HAS_PMD_LEAVES #define flush_pmd_tlb_range (vma, addr, end) flush_tlb_range(vma, addr, end) -#define flush_pud_tlb_range (vma, addr, end) flush_tlb_range(vma, addr, end)#else #define flush_pmd_tlb_range (vma, addr, end) BUILD_BUG() +#endif + +#ifdef CONFIG_PGTABLE_HAS_PUD_LEAVES +#define flush_pud_tlb_range (vma, addr, end) flush_tlb_range(vma, addr, end) +#else #define flush_pud_tlb_range (vma, addr, end) BUILD_BUG()#endif #endif
include/trace/events/huge_mapping.h new file +41 -0
41 lines not on this page · raw message
include/trace/events/thp.h +0 -28
@@ -66,34 +66,6 @@ DEFINE_EVENT(hugepage_update, hugepage_update_pud, TP_PROTO(unsigned long addr, unsigned long pud, unsigned long clr, unsigned long set), TP_ARGS(addr, pud, clr, set) ); - -DECLARE_EVENT_CLASS(migration_pmd, - - TP_PROTO(unsigned long addr, unsigned long pmd), - - TP_ARGS(addr, pmd), - - TP_STRUCT__entry( - __field(unsigned long, addr) - __field(unsigned long, pmd) - ), - - TP_fast_assign( - __entry->addr = addr; - __entry->pmd = pmd; - ), - TP_printk("addr=%lx, pmd=%lx", __entry->addr, __entry->pmd) -); - -DEFINE_EVENT(migration_pmd, set_migration_pmd, - TP_PROTO(unsigned long addr, unsigned long pmd), - TP_ARGS(addr, pmd) -); - -DEFINE_EVENT(migration_pmd, remove_migration_pmd, - TP_PROTO(unsigned long addr, unsigned long pmd), - TP_ARGS(addr, pmd) -);#endif /* _TRACE_THP_H */ /* This part must be outside protection */
mm/Makefile +2 -0
@@ -95,6 +95,8 @@ obj-$(CONFIG_MIGRATION) += migrate.o obj-$(CONFIG_NUMA) += memory-tiers.o obj-$(CONFIG_DEVICE_MIGRATION) += migrate_device.o obj-$(CONFIG_TRANSPARENT_HUGEPAGE) += huge_memory.o khugepaged.o +obj-$(CONFIG_PGTABLE_HAS_PMD_LEAVES) += huge_mapping_pmd.o +obj-$(CONFIG_PGTABLE_HAS_PUD_LEAVES) += huge_mapping_pud.o obj-$(CONFIG_PAGE_COUNTER) += page_counter.o obj-$(CONFIG_MEMCG_V1) += memcontrol-v1.o obj-$(CONFIG_MEMCG) += memcontrol.o vmpressure.o
mm/huge_mapping_pmd.c new file +979 -0
979 lines not on this page · raw message
mm/huge_mapping_pud.c new file +235 -0
235 lines not on this page · raw message
mm/huge_memory.c +2 -1123
1,164 lines not on this page · raw message
Signature · 1 line
2.45.0
Peter Xu
reviewerRFC PATCH 6/6 mm: Convert "*_trans_huge() || *_devmap()" to use *_leaf()
In reply to Peter XuThis patch converted all such checks into one *_leaf() check under common mm/, as "thp+devmap" should compose everything for a *_leaf() for now. I didn't yet touch arch code in other directories, as some arch may need some special attention, so I left those separately.
It should start to save some cycles on such check and pave way for the new leaf types. E.g., when a new type of leaf is introduced, it'll naturally go the same route to what we have now for thp+devmap.
Here one issue with pxx_leaf() API is that such API will be defined by arch but it doesn't consider kernel config. For example, below "if" branch cannot be automatically optimized:
if (pmd_leaf()) { ... }
Even if both THP && HUGETLB are not enabled (which means pmd_leaf() can never return true).
To provide a chance for compilers to optimize and omit code when possible, introduce a light wrapper for them and call them pxx_is_leaf(). That will take kernel config into account and properly allow omitting branches when the compiler knows it'll constantly returns false. This tries to mimic what we used to have with pxx_trans_huge() when !THP, so it now also applies to pxx_leaf() API.
--- include/linux/huge_mm.h | 6 +++--- include/linux/pgtable.h | 30 +++++++++++++++++++++++++++++- mm/hmm.c | 4 ++-- mm/huge_mapping_pmd.c | 9 +++------ mm/huge_mapping_pud.c | 6 +++--- mm/mapping_dirty_helpers.c | 4 ++-- mm/memory.c | 14 ++++++-------- mm/migrate_device.c | 2 +- mm/mprotect.c | 4 ++-- mm/mremap.c | 5 ++--- mm/page_vma_mapped.c | 5 ++--- mm/pgtable-generic.c | 7 +++---- 12 files changed, 58 insertions(+), 38 deletions(-)
include/linux/huge_mm.h +3 -3
@@ -27,7 +27,7 @@ spinlock_t *__pud_trans_huge_lock(pud_t *pud, struct vm_area_struct *vma);static inline spinlock_t *pud_trans_huge_lock (pud_t *pud,struct vm_area_struct *vma) { -if (pud_trans_huge (*pud) ||pud_devmap (*pud)) +if (pud_is_leaf (*pud))return __pud_trans_huge_lock (pud, vma);else return NULL ;
@@ -36,7 +36,7 @@ pud_trans_huge_lock(pud_t *pud, struct vm_area_struct *vma)#define split_huge_pud (__vma, __pud, __address) \ do { \ pud_t *____pud = (__pud); \ - if (pud_trans_huge(*____pud) || pud_devmap(*____pud)) \ + if (pud_is_leaf(*____pud)) \ __split_huge_pud(__vma, __pud, __address); \ } while (0)#else /* CONFIG_PGTABLE_HAS_PUD_LEAVES */
@@ -125,7 +125,7 @@ static inline int is_swap_pmd(pmd_t pmd)static inline spinlock_t *pmd_trans_huge_lock (pmd_t *pmd,struct vm_area_struct *vma) { -if (is_swap_pmd (*pmd) ||pmd_trans_huge (*pmd) ||pmd_devmap (*pmd)) +if (is_swap_pmd (*pmd) ||pmd_is_leaf (*pmd))return __pmd_trans_huge_lock (pmd, vma);else return NULL ;
include/linux/pgtable.h +29 -1
@@ -1641,7 +1641,7 @@ static inline int pud_trans_unstable(pud_t *pud)defined (CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD) pud_t pudval = READ_ONCE(*pud); -if (pud_none (pudval) ||pud_trans_huge (pudval) ||pud_devmap (pudval)) +if (pud_none (pudval) ||pud_leaf (pudval))return 1 ;if (unlikely (pud_bad (pudval))) {pud_clear_bad (pud);
@@ -1901,6 +1901,34 @@ typedef unsigned int pgtbl_mod_mask;#define pmd_leaf (x) false#endif +/* +* Wrapper of pxx_leaf() helpers. +* +* Comparing to pxx_leaf() API, the only difference is: using these macros +* can help code generation, so unnecessary code can be omitted when the +* specific level of leaf is not possible due to kernel config. It is +* needed because normally pxx_leaf() can be defined in arch code without +* knowing the kernel config. +* +* Currently we only need pmd/pud versions, because the largest leaf Linux +* supports so far is pud. +* +* Defining here also means that in arch's pgtable headers these macros +* cannot be used, pxx_leaf()s need to be used instead, because this file +* will not be included in arch's pgtable headers. +*/ +#ifdef CONFIG_PGTABLE_HAS_PMD_LEAVES +#define pmd_is_leaf (x) pmd_leaf(x) +#else +#define pmd_is_leaf (x) false +#endif + +#ifdef CONFIG_PGTABLE_HAS_PUD_LEAVES +#define pud_is_leaf (x) pud_leaf(x) +#else +#define pud_is_leaf (x) false +#endif +#ifndef pgd_leaf_size#define pgd_leaf_size (x) (1ULL << PGDIR_SHIFT)#endif
mm/hmm.c +2 -2
@@ -351,7 +351,7 @@ static int hmm_vma_walk_pmd(pmd_t *pmdp,return hmm_pfns_fill (start, end, range,HMM_PFN_ERROR ); } -if (pmd_devmap (pmd) ||pmd_trans_huge (pmd)) { +if (pmd_is_leaf (pmd)) {/* * No need to take pmd_lock here, even if some other thread * is splitting the huge pmd we will get that event through
@@ -362,7 +362,7 @@ static int hmm_vma_walk_pmd(pmd_t *pmdp,* values. */ pmd =pmdp_get_lockless (pmdp); -if (!pmd_devmap (pmd) && !pmd_trans_huge (pmd)) +if (!pmd_is_leaf (pmd)) goto again;return hmm_vma_handle_pmd (walk, addr, end, hmm_pfns, pmd);
mm/huge_mapping_pmd.c +3 -6
@@ -60,8 +60,7 @@ spinlock_t *__pmd_trans_huge_lock(pmd_t *pmd, struct vm_area_struct *vma)spinlock_t *ptl; ptl =pmd_lock (vma->vm_mm, pmd); -if (likely (is_swap_pmd (*pmd) ||pmd_trans_huge (*pmd) || -pmd_devmap (*pmd))) +if (likely (is_swap_pmd (*pmd) ||pmd_is_leaf (*pmd)))return ptl;spin_unlock (ptl);return NULL ;
@@ -627,8 +626,7 @@ static void __split_huge_pmd_locked(struct vm_area_struct *vma, pmd_t *pmd,VM_BUG_ON (haddr & ~HPAGE_PMD_MASK );VM_BUG_ON_VMA (vma->vm_start > haddr, vma);VM_BUG_ON_VMA (vma->vm_end < haddr +HPAGE_PMD_SIZE , vma); -VM_BUG_ON (!is_pmd_migration_entry (*pmd) && !pmd_trans_huge (*pmd) && - !pmd_devmap (*pmd)); +VM_BUG_ON (!is_pmd_migration_entry (*pmd) && !pmd_is_leaf (*pmd));#ifdef CONFIG_TRANSPARENT_HUGEPAGE count_vm_event (THP_SPLIT_PMD );
@@ -845,8 +843,7 @@ void split_huge_pmd_locked(struct vm_area_struct *vma, unsigned long address,* require a folio to check the PMD against. Otherwise, there * is a risk of replacing the wrong folio. */ -if (pmd_trans_huge (*pmd) ||pmd_devmap (*pmd) || -is_pmd_migration_entry (*pmd)) { +if (pmd_is_leaf (*pmd) ||is_pmd_migration_entry (*pmd)) {if (folio && folio !=pmd_folio (*pmd))return ;__split_huge_pmd_locked (vma, pmd, address, freeze);
mm/huge_mapping_pud.c +3 -3
@@ -57,7 +57,7 @@ spinlock_t *__pud_trans_huge_lock(pud_t *pud, struct vm_area_struct *vma)spinlock_t *ptl; ptl =pud_lock (vma->vm_mm, pud); -if (likely (pud_trans_huge (*pud) ||pud_devmap (*pud))) +if (likely (pud_is_leaf (*pud)))return ptl;spin_unlock (ptl);return NULL ;
@@ -90,7 +90,7 @@ int copy_huge_pud(struct mm_struct *dst_mm, struct mm_struct *src_mm, ret = -EAGAIN ; pud = *src_pud; -if (unlikely (!pud_trans_huge (pud) && !pud_devmap (pud))) +if (unlikely (!pud_leaf (pud))) goto out_unlock;/*
@@ -225,7 +225,7 @@ void __split_huge_pud(struct vm_area_struct *vma, pud_t *pud, (address & HPAGE_PUD_MASK) + HPAGE_PUD_SIZE);mmu_notifier_invalidate_range_start (&range); ptl =pud_lock (vma->vm_mm, pud); -if (unlikely (!pud_trans_huge (*pud) && !pud_devmap (*pud))) +if (unlikely (!pud_is_leaf (*pud))) goto out;__split_huge_pud_locked (vma, pud, range.start);
mm/mapping_dirty_helpers.c +2 -2
@@ -129,7 +129,7 @@ static int wp_clean_pmd_entry(pmd_t *pmd, unsigned long addr, unsigned long end,pmd_t pmdval =pmdp_get_lockless (pmd);/* Do not split a huge pmd, present or migrated */ -if (pmd_trans_huge (pmdval) ||pmd_devmap (pmdval)) { +if (pmd_is_leaf (pmdval)) {WARN_ON (pmd_write (pmdval) ||pmd_dirty (pmdval)); walk->action =ACTION_CONTINUE ; }
@@ -152,7 +152,7 @@ static int wp_clean_pud_entry(pud_t *pud, unsigned long addr, unsigned long end,pud_t pudval =READ_ONCE (*pud);/* Do not split a huge pud */ -if (pud_trans_huge (pudval) ||pud_devmap (pudval)) { +if (pud_is_leaf (pudval)) {WARN_ON (pud_write (pudval) ||pud_dirty (pudval)); walk->action =ACTION_CONTINUE ; }
mm/memory.c +6 -8
@@ -1235,8 +1235,7 @@ copy_pmd_range(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma, src_pmd =pmd_offset (src_pud, addr); do { next = pmd_addr_end(addr, end); - if (is_swap_pmd(*src_pmd) || pmd_trans_huge(*src_pmd) - || pmd_devmap(*src_pmd)) { + if (is_swap_pmd(*src_pmd) || pmd_is_leaf(*src_pmd)) { int err; VM_BUG_ON_VMA(next-addr != HPAGE_PMD_SIZE, src_vma); err = copy_huge_pmd(dst_mm, src_mm, dst_pmd, src_pmd,
@@ -1272,7 +1271,7 @@ copy_pud_range(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma, src_pud =pud_offset (src_p4d, addr); do { next = pud_addr_end(addr, end); - if (pud_trans_huge(*src_pud) || pud_devmap(*src_pud)) { + if (pud_is_leaf(*src_pud)) { int err; VM_BUG_ON_VMA(next-addr != HPAGE_PUD_SIZE, src_vma);
@@ -1710,7 +1709,7 @@ static inline unsigned long zap_pmd_range(struct mmu_gather *tlb, pmd =pmd_offset (pud, addr); do { next = pmd_addr_end(addr, end); - if (is_swap_pmd(*pmd) || pmd_trans_huge(*pmd) || pmd_devmap(*pmd)) { + if (is_swap_pmd(*pmd) || pmd_is_leaf(*pmd)) { if (next - addr != HPAGE_PMD_SIZE) __split_huge_pmd(vma, pmd, addr, false, NULL); else if (zap_huge_pmd(tlb, vma, pmd, addr)) {
@@ -1752,7 +1751,7 @@ static inline unsigned long zap_pud_range(struct mmu_gather *tlb, pud =pud_offset (p4d, addr); do { next = pud_addr_end(addr, end); - if (pud_trans_huge(*pud) || pud_devmap(*pud)) { + if (pud_is_leaf(*pud)) { if (next - addr != HPAGE_PUD_SIZE) { mmap_assert_locked(tlb->mm); split_huge_pud(vma, pud, addr);
@@ -5605,8 +5604,7 @@ static vm_fault_t __handle_mm_fault(struct vm_area_struct *vma,pud_t orig_pud = *vmf.pud;barrier (); -if (pud_trans_huge (orig_pud) ||pud_devmap (orig_pud)) { - +if (pud_is_leaf (orig_pud)) {/* * TODO once we support anonymous PUDs: NUMA case and * FAULT_FLAG_UNSHARE handling.
@@ -5646,7 +5644,7 @@ static vm_fault_t __handle_mm_fault(struct vm_area_struct *vma,pmd_migration_entry_wait (mm, vmf.pmd);return 0 ; } -if (pmd_trans_huge (vmf.orig_pmd) ||pmd_devmap (vmf.orig_pmd)) { +if (pmd_is_leaf (vmf.orig_pmd)) {if (pmd_protnone (vmf.orig_pmd) &&vma_is_accessible (vma))return do_huge_pmd_numa_page (&vmf);
mm/migrate_device.c +1 -1
@@ -596,7 +596,7 @@ static void migrate_vma_insert_page(struct migrate_vma *migrate, pmdp =pmd_alloc (mm, pudp, addr);if (!pmdp) goto abort; -if (pmd_trans_huge (*pmdp) ||pmd_devmap (*pmdp)) +if (pmd_leaf (*pmdp)) goto abort;if (pte_alloc (mm, pmdp)) goto abort;
mm/mprotect.c +2 -2
@@ -381,7 +381,7 @@ static inline long change_pmd_range(struct mmu_gather *tlb, goto next; _pmd =pmdp_get_lockless (pmd); -if (is_swap_pmd (_pmd) ||pmd_trans_huge (_pmd) ||pmd_devmap (_pmd)) { +if (is_swap_pmd (_pmd) ||pmd_is_leaf (_pmd)) {if ((next - addr !=HPAGE_PMD_SIZE ) ||pgtable_split_needed (vma, cp_flags)) {__split_huge_pmd (vma, pmd, addr, false,NULL );
@@ -452,7 +452,7 @@ static inline long change_pud_range(struct mmu_gather *tlb,mmu_notifier_invalidate_range_start (&range); } -if (pud_leaf (pud)) { +if (pud_is_leaf (pud)) {if ((next - addr !=PUD_SIZE ) ||pgtable_split_needed (vma, cp_flags)) {__split_huge_pud (vma, pudp, addr);
mm/mremap.c +2 -3
@@ -587,7 +587,7 @@ unsigned long move_page_tables(struct vm_area_struct *vma, new_pud =alloc_new_pud (vma->vm_mm, vma, new_addr);if (!new_pud)break ; -if (pud_trans_huge (*old_pud) ||pud_devmap (*old_pud)) { +if (pud_is_leaf (*old_pud)) {if (extent ==HPAGE_PUD_SIZE ) {move_pgt_entry (HPAGE_PUD , vma, old_addr, new_addr, old_pud, new_pud, need_rmap_locks);
@@ -609,8 +609,7 @@ unsigned long move_page_tables(struct vm_area_struct *vma,if (!new_pmd)break ; again: -if (is_swap_pmd (*old_pmd) ||pmd_trans_huge (*old_pmd) || -pmd_devmap (*old_pmd)) { +if (is_swap_pmd (*old_pmd) ||pmd_is_leaf (*old_pmd)) {if (extent ==HPAGE_PMD_SIZE &&move_pgt_entry (HPAGE_PMD , vma, old_addr, new_addr, old_pmd, new_pmd, need_rmap_locks))
mm/page_vma_mapped.c +2 -3
@@ -235,8 +235,7 @@ bool page_vma_mapped_walk(struct page_vma_mapped_walk *pvmw)*/ pmde =pmdp_get_lockless (pvmw->pmd); -if (pmd_trans_huge (pmde) ||is_pmd_migration_entry (pmde) || - (pmd_present (pmde) &&pmd_devmap (pmde))) { +if (pmd_is_leaf (pmde) ||is_pmd_migration_entry (pmde)) { pvmw->ptl =pmd_lock (mm, pvmw->pmd); pmde = *pvmw->pmd;if (!pmd_present (pmde)) {
@@ -251,7 +250,7 @@ bool page_vma_mapped_walk(struct page_vma_mapped_walk *pvmw)return not_found (pvmw);return true; } -if (likely (pmd_trans_huge (pmde) ||pmd_devmap (pmde))) { +if (likely (pmd_is_leaf (pmde))) {if (pvmw->flags &PVMW_MIGRATION )return not_found (pvmw);if (!check_pmd (pmd_pfn (pmde), pvmw))
mm/pgtable-generic.c +3 -4
@@ -139,8 +139,7 @@ pmd_t pmdp_huge_clear_flush(struct vm_area_struct *vma, unsigned long address, {pmd_t pmd;VM_BUG_ON (address & ~HPAGE_PMD_MASK ); -VM_BUG_ON (pmd_present (*pmdp) && !pmd_trans_huge (*pmdp) && - !pmd_devmap (*pmdp)); +VM_BUG_ON (pmd_present (*pmdp) && !pmd_leaf (*pmdp)); pmd =pmdp_huge_get_and_clear (vma->vm_mm, address, pmdp);flush_pmd_tlb_range (vma, address, address +HPAGE_PMD_SIZE );return pmd;
@@ -247,7 +246,7 @@ pud_t pudp_huge_clear_flush(struct vm_area_struct *vma, unsigned long address,pud_t pud;VM_BUG_ON (address & ~HPAGE_PUD_MASK ); -VM_BUG_ON (!pud_trans_huge (*pudp) && !pud_devmap (*pudp)); +VM_BUG_ON (!pud_leaf (*pudp)); pud =pudp_huge_get_and_clear (vma->vm_mm, address, pudp);flush_pud_tlb_range (vma, address, address +HPAGE_PUD_SIZE );return pud;
@@ -293,7 +292,7 @@ pte_t *__pte_offset_map(pmd_t *pmd, unsigned long addr, pmd_t *pmdvalp) *pmdvalp = pmdval;if (unlikely (pmd_none (pmdval) ||is_pmd_migration_entry (pmdval))) goto nomap; -if (unlikely (pmd_trans_huge (pmdval) ||pmd_devmap (pmdval))) +if (unlikely (pmd_leaf (pmdval))) goto nomap;if (unlikely (pmd_bad (pmdval))) {pmd_clear_bad (pmd);
Signature · 1 line
2.45.0
Quoted 34 lines · Peter Xu
This is an RFC series, so not yet for merging. Please don't be scared by the code changes: most of them are code movements only.
This series is based on the dax mprotect fix series here (while that one is based on mm-unstable):
[PATCH v3 0/8] mm/mprotect: Fix dax puds
https://lore.kernel.org/r/[email protected]Overview
========This series doesn't provide any feature change. The only goal of this series is to start decoupling two ideas: "THP" and "huge mapping". We already started with having
PGTABLE_HAS_HUGE_LEAVESconfig option, and this one extends that idea into the code.The issue is that we have so many functions that only compile with
CONFIG_THP=on, even though they're about huge mappings, and huge mapping is a pretty common concept, which can apply to many things besides THPs nowadays. The major THP file ismm/huge_memory.cas of now.The first example of such huge mapping users will be hugetlb. We lived until now with no problem simply because Linux almost duplicated all the logics there in the "THP" files into hugetlb APIs. If we want to get rid of hugetlb specific APIs and paths, this _might_ be the first thing we want to do, because we want to be able to e.g., zapping a hugetlb pmd entry even if !CONFIG_THP.
Then consider other things like dax / pfnmaps. Dax can depend on THP, then it'll naturally be able to use pmd/pud helpers, that's okay. However is it a must? Do we also want to have every new pmd/pud mappings in the future to depend on THP (like PFNMAP)? My answer is no, but I'm open to opinions.
Quoted 6 lines
If anyone agrees with me that "huge mapping" (aka, PMD/PUD mappings that are larger than
PAGE_SIZE) is a more generic concept than THP, then I think at some point we need to move the generic code out of THP code into a common code base.This is what this series does as a start.
Hi Peter!
From a quick glimpse, patch #1-#4 do make sense independent of patch #5.
I am not so sure about all of the code movement in patch #5. If large folios are the future, then likely huge_memory.c should simply be the home for all that logic.
Maybe the goal should better be to compile huge_memory.c not only for THP, but also for other use cases that require that logic, and fence off all THP specific stuff using #ifdef?
Not sure, though. But a lot of this code movements/churn might be avoidable.
Signature · 3 lines
Cheers, David / dhildenb
Peter Xu
reviewerQuoted 50 lines · David Hildenbrand
Quoted 34 lines · Peter Xu
This is an RFC series, so not yet for merging. Please don't be scared by the code changes: most of them are code movements only.
This series is based on the dax mprotect fix series here (while that one is based on mm-unstable):
[PATCH v3 0/8] mm/mprotect: Fix dax puds
https://lore.kernel.org/r/[email protected]Overview
========This series doesn't provide any feature change. The only goal of this series is to start decoupling two ideas: "THP" and "huge mapping". We already started with having
PGTABLE_HAS_HUGE_LEAVESconfig option, and this one extends that idea into the code.The issue is that we have so many functions that only compile with
CONFIG_THP=on, even though they're about huge mappings, and huge mapping is a pretty common concept, which can apply to many things besides THPs nowadays. The major THP file ismm/huge_memory.cas of now.The first example of such huge mapping users will be hugetlb. We lived until now with no problem simply because Linux almost duplicated all the logics there in the "THP" files into hugetlb APIs. If we want to get rid of hugetlb specific APIs and paths, this _might_ be the first thing we want to do, because we want to be able to e.g., zapping a hugetlb pmd entry even if !CONFIG_THP.
Then consider other things like dax / pfnmaps. Dax can depend on THP, then it'll naturally be able to use pmd/pud helpers, that's okay. However is it a must? Do we also want to have every new pmd/pud mappings in the future to depend on THP (like PFNMAP)? My answer is no, but I'm open to opinions.
Quoted 6 lines
If anyone agrees with me that "huge mapping" (aka, PMD/PUD mappings that are larger than
PAGE_SIZE) is a more generic concept than THP, then I think at some point we need to move the generic code out of THP code into a common code base.This is what this series does as a start.
Hi Peter!
From a quick glimpse, patch #1-#4 do make sense independent of patch #5.
I am not so sure about all of the code movement in patch #5. If large folios are the future, then likely
huge_memory.cshould simply be the home for all that logic.
Quoted 5 lines
Maybe the goal should better be to compile
huge_memory.cnot only for THP, but also for other use cases that require that logic, and fence off all THP specific stuff using #ifdef?Not sure, though. But a lot of this code movements/churn might be avoidable.
I'm fine using ifdefs in the current fine, but IMHO it's a matter of whether we want to keep huge_memory.c growing into even larger file, and keep all large folio logics only in that file. Currently it's ~4000 LOCs.
Nornally I don't see this as much of a "code churn" category, because it doesn't changes the code itself but only move things. I personally also prefer without code churns, but only in the case where there'll be tiny little functional changes here and there without real benefit.
It's pretty unavoidable to me when one file grows too large and we'll need to split, and in this case git doesn't have a good way to track such movement..
Irrelevant of this, just to mention I think there's still one option that I at least can make the huge pfnmap depends on THP again which shouldn't be a huge deal (I don't have any use case that needs huge pfnmap but disable THP, anyway..), so this series isn't an immediate concern to me for that route. But for a hugetlb rework this might be something we need to do, because we simplly can't make CONFIG_HUGETLB rely on CONFIG_THP..
Thanks,
Signature · 1 line
Peter Xu
Quoted 55 lines · Peter Xu
Quoted 54 lines · David Hildenbrand
Quoted 34 lines · Peter Xu
This is an RFC series, so not yet for merging. Please don't be scared by the code changes: most of them are code movements only.
This series is based on the dax mprotect fix series here (while that one is based on mm-unstable):
[PATCH v3 0/8] mm/mprotect: Fix dax puds
https://lore.kernel.org/r/[email protected]Overview
========This series doesn't provide any feature change. The only goal of this series is to start decoupling two ideas: "THP" and "huge mapping". We already started with having
PGTABLE_HAS_HUGE_LEAVESconfig option, and this one extends that idea into the code.The issue is that we have so many functions that only compile with
CONFIG_THP=on, even though they're about huge mappings, and huge mapping is a pretty common concept, which can apply to many things besides THPs nowadays. The major THP file ismm/huge_memory.cas of now.The first example of such huge mapping users will be hugetlb. We lived until now with no problem simply because Linux almost duplicated all the logics there in the "THP" files into hugetlb APIs. If we want to get rid of hugetlb specific APIs and paths, this _might_ be the first thing we want to do, because we want to be able to e.g., zapping a hugetlb pmd entry even if !CONFIG_THP.
Then consider other things like dax / pfnmaps. Dax can depend on THP, then it'll naturally be able to use pmd/pud helpers, that's okay. However is it a must? Do we also want to have every new pmd/pud mappings in the future to depend on THP (like PFNMAP)? My answer is no, but I'm open to opinions.
Quoted 6 lines
If anyone agrees with me that "huge mapping" (aka, PMD/PUD mappings that are larger than
PAGE_SIZE) is a more generic concept than THP, then I think at some point we need to move the generic code out of THP code into a common code base.This is what this series does as a start.
Hi Peter!
From a quick glimpse, patch #1-#4 do make sense independent of patch #5.
I am not so sure about all of the code movement in patch #5. If large folios are the future, then likely
huge_memory.cshould simply be the home for all that logic.Maybe the goal should better be to compile
huge_memory.cnot only for THP, but also for other use cases that require that logic, and fence off all THP specific stuff using #ifdef?
Quoted 5 lines
Quoted 1 line
Not sure, though. But a lot of this code movements/churn might be avoidable.
I'm fine using ifdefs in the current fine, but IMHO it's a matter of whether we want to keep
huge_memory.cgrowing into even larger file, and keep all large folio logics only in that file. Currently it's ~4000 LOCs.
Depends on "how much" for sure. huge_memory.c is currently on place 12 of the biggest files in mm/. So there might not be immediate cause for action ... just yet :) [guess which file is on #2 :) ]
Quoted 9 lines
Nornally I don't see this as much of a "code churn" category, because it doesn't changes the code itself but only move things. I personally also prefer without code churns, but only in the case where there'll be tiny little functional changes here and there without real benefit.
It's pretty unavoidable to me when one file grows too large and we'll need to split, and in this case git doesn't have a good way to track such movement..
Yes, that's what I mean.
I've been recently thinking if we should pursue a different direction:
Just as we recently relocated most follow_huge_* stuff into gup.c, likely we should rather look into moving copy_huge_pmd, change_huge_pmd, copy_huge_pmd ... into the files where they logically belong to.
In madvise.c, we've been doing that in some places already: For madvise_cold_or_pageout_pte_range() we inline the code, but not for madvise_free_huge_pmd().
pmd_trans_huge() would already compile to a NOP without CONFIG_TRANSPARENT_HUGEPAGE, but to make that code avoid most CONFIG_TRANSPARENT_HUGEPAGE, we'd need a couple more function stubs to make the compiler happy while still being able to compile that code out when not required.
The idea would be that e.g., pmd_leaf() would return "false" at compile time if no active configuration (THP, HUGETLB, ...) would be active. So we could just use pmd_leaf() similar to pmd_trans_huge() in relevant code and have the compiler optimize it all out without putting it into separate files.
That means, large folios and PMD/PUD mappings will become "more common" and better integrated, without the need to jump between files.
Just some thought about an alternative that would make sense to me.
Quoted 7 lines
Irrelevant of this, just to mention I think there's still one option that I at least can make the huge pfnmap depends on THP again which shouldn't be a huge deal (I don't have any use case that needs huge pfnmap but disable THP, anyway..), so this series isn't an immediate concern to me for that route. But for a hugetlb rework this might be something we need to do, because we simplly can't make
CONFIG_HUGETLBrely onCONFIG_THP..
Yes, likely. FSDAX went a similar direction and called that FSDAX thing a "THP" whereby it really doesn't have anything in common with a THP, besides being partially mappable -- IMHO.
Signature · 3 lines
Cheers, David / dhildenb
Peter Xu
reviewerQuoted 58 lines · David Hildenbrand
Quoted 57 lines · Peter Xu
Quoted 55 lines · David Hildenbrand
Quoted 34 lines · Peter Xu
This is an RFC series, so not yet for merging. Please don't be scared by the code changes: most of them are code movements only.
This series is based on the dax mprotect fix series here (while that one is based on mm-unstable):
[PATCH v3 0/8] mm/mprotect: Fix dax puds
https://lore.kernel.org/r/[email protected]Overview
========This series doesn't provide any feature change. The only goal of this series is to start decoupling two ideas: "THP" and "huge mapping". We already started with having
PGTABLE_HAS_HUGE_LEAVESconfig option, and this one extends that idea into the code.The issue is that we have so many functions that only compile with
CONFIG_THP=on, even though they're about huge mappings, and huge mapping is a pretty common concept, which can apply to many things besides THPs nowadays. The major THP file ismm/huge_memory.cas of now.The first example of such huge mapping users will be hugetlb. We lived until now with no problem simply because Linux almost duplicated all the logics there in the "THP" files into hugetlb APIs. If we want to get rid of hugetlb specific APIs and paths, this _might_ be the first thing we want to do, because we want to be able to e.g., zapping a hugetlb pmd entry even if !CONFIG_THP.
Then consider other things like dax / pfnmaps. Dax can depend on THP, then it'll naturally be able to use pmd/pud helpers, that's okay. However is it a must? Do we also want to have every new pmd/pud mappings in the future to depend on THP (like PFNMAP)? My answer is no, but I'm open to opinions.
Quoted 6 lines
If anyone agrees with me that "huge mapping" (aka, PMD/PUD mappings that are larger than
PAGE_SIZE) is a more generic concept than THP, then I think at some point we need to move the generic code out of THP code into a common code base.This is what this series does as a start.
Hi Peter!
From a quick glimpse, patch #1-#4 do make sense independent of patch #5.
I am not so sure about all of the code movement in patch #5. If large folios are the future, then likely
huge_memory.cshould simply be the home for all that logic.Maybe the goal should better be to compile
huge_memory.cnot only for THP, but also for other use cases that require that logic, and fence off all THP specific stuff using #ifdef?Not sure, though. But a lot of this code movements/churn might be avoidable.
Quoted 7 lines
Quoted 3 lines
I'm fine using ifdefs in the current fine, but IMHO it's a matter of whether we want to keep
huge_memory.cgrowing into even larger file, and keep all large folio logics only in that file. Currently it's ~4000 LOCs.Depends on "how much" for sure.
huge_memory.cis currently on place 12 of the biggest files inmm/. So there might not be immediate cause for action ... just yet :) [guess which file is on #2 :) ]
7821, hugetlb.c
7602, vmscan.c
7275, slub.c
7072, page_alloc.c
6673, memory.c
5402, memcontrol.c
5239, shmem.c
5155, vmalloc.c
4419, filemap.c
4060, mmap.c
3882, huge_memory.c
IMHO a split is normally better than keeping everything in one file, but yeah I'd confess THP file isn't that bad comparing to others.. And I'm definitely surprised it's even out of top ten.
Quoted 23 lines
Quoted 10 lines
Nornally I don't see this as much of a "code churn" category, because it doesn't changes the code itself but only move things. I personally also prefer without code churns, but only in the case where there'll be tiny little functional changes here and there without real benefit.
It's pretty unavoidable to me when one file grows too large and we'll need to split, and in this case git doesn't have a good way to track such movement..
Yes, that's what I mean.
I've been recently thinking if we should pursue a different direction:
Just as we recently relocated most follow_huge_* stuff into
gup.c, likely we should rather look into movingcopy_huge_pmd,change_huge_pmd,copy_huge_pmd... into the files where they logically belong to.In
madvise.c, we've been doing that in some places already: Formadvise_cold_or_pageout_pte_range()we inline the code, but not formadvise_free_huge_pmd().
Quoted 5 lines
pmd_trans_huge()would already compile to a NOP withoutCONFIG_TRANSPARENT_HUGEPAGE, but to make that code avoid mostCONFIG_TRANSPARENT_HUGEPAGE, we'd need a couple more function stubs to make the compiler happy while still being able to compile that code out when not required.
Right, I had a patch does exactly that, where it's called pmd_is_leaf(), for example, but taking CONFIG_* into account.
I remember I had some issue with that, e.g. I used to see pmd_trans_huge() (when !THP) can optimize some path but pmd_is_leaf() didn't do the same job even if all configs were off. But that's another story and I didn't yet dig deeper. Could be something small but overlooked.
Quoted 10 lines
The idea would be that e.g.,
pmd_leaf()would return "false" at compile time if no active configuration (THP, HUGETLB, ...) would be active. So we could just usepmd_leaf()similar topmd_trans_huge()in relevant code and have the compiler optimize it all out without putting it into separate files.That means, large folios and PMD/PUD mappings will become "more common" and better integrated, without the need to jump between files.
Just some thought about an alternative that would make sense to me.
Yeah comments are always welcomed, thanks.
So I suppose maybe it would be easier for now that I make the pfnmap branch depending on THP. It looks to me something like this may still take some time to consolidate. When it's light enough, maybe it can be a few initial patches on top of a hugetlb series that can start to use this. Maybe that'll at least make the patches easier to review.
Thanks,
Signature · 1 line
Peter Xu
Quoted 79 lines · Peter Xu
Quoted 58 lines · David Hildenbrand
Quoted 57 lines · Peter Xu
Quoted 55 lines · David Hildenbrand
Quoted 34 lines · Peter Xu
This is an RFC series, so not yet for merging. Please don't be scared by the code changes: most of them are code movements only.
This series is based on the dax mprotect fix series here (while that one is based on mm-unstable):
[PATCH v3 0/8] mm/mprotect: Fix dax puds
https://lore.kernel.org/r/[email protected]Overview
========This series doesn't provide any feature change. The only goal of this series is to start decoupling two ideas: "THP" and "huge mapping". We already started with having
PGTABLE_HAS_HUGE_LEAVESconfig option, and this one extends that idea into the code.The issue is that we have so many functions that only compile with
CONFIG_THP=on, even though they're about huge mappings, and huge mapping is a pretty common concept, which can apply to many things besides THPs nowadays. The major THP file ismm/huge_memory.cas of now.The first example of such huge mapping users will be hugetlb. We lived until now with no problem simply because Linux almost duplicated all the logics there in the "THP" files into hugetlb APIs. If we want to get rid of hugetlb specific APIs and paths, this _might_ be the first thing we want to do, because we want to be able to e.g., zapping a hugetlb pmd entry even if !CONFIG_THP.
Then consider other things like dax / pfnmaps. Dax can depend on THP, then it'll naturally be able to use pmd/pud helpers, that's okay. However is it a must? Do we also want to have every new pmd/pud mappings in the future to depend on THP (like PFNMAP)? My answer is no, but I'm open to opinions.
Quoted 6 lines
If anyone agrees with me that "huge mapping" (aka, PMD/PUD mappings that are larger than
PAGE_SIZE) is a more generic concept than THP, then I think at some point we need to move the generic code out of THP code into a common code base.This is what this series does as a start.
Hi Peter!
From a quick glimpse, patch #1-#4 do make sense independent of patch #5.
I am not so sure about all of the code movement in patch #5. If large folios are the future, then likely
huge_memory.cshould simply be the home for all that logic.Maybe the goal should better be to compile
huge_memory.cnot only for THP, but also for other use cases that require that logic, and fence off all THP specific stuff using #ifdef?Not sure, though. But a lot of this code movements/churn might be avoidable.
Quoted 7 lines
Quoted 3 lines
I'm fine using ifdefs in the current fine, but IMHO it's a matter of whether we want to keep
huge_memory.cgrowing into even larger file, and keep all large folio logics only in that file. Currently it's ~4000 LOCs.Depends on "how much" for sure.
huge_memory.cis currently on place 12 of the biggest files inmm/. So there might not be immediate cause for action ... just yet :) [guess which file is on #2 :) ]7821,
hugetlb.c
7602,vmscan.c
7275,slub.c
7072,page_alloc.c
6673,memory.c
5402,memcontrol.c
5239,shmem.c
5155,vmalloc.c
4419,filemap.c
4060,mmap.c
3882,huge_memory.c
Quoted 3 lines
IMHO a split is normally better than keeping everything in one file, but yeah I'd confess THP file isn't that bad comparing to others.. And I'm definitely surprised it's even out of top ten.
It's always interesting looking at the numbers here. For v6.10 we had:
8521 mm/memcontrol.c
7813 mm/hugetlb.c
7550 mm/vmscan.c
7266 mm/slub.c
7018 mm/page_alloc.c
6468 mm/memory.c
5154 mm/vmalloc.c
5002 mm/shmem.c
4419 mm/filemap.c
4019 mm/mmap.c
3954 mm/ksm.c
3740 mm/swapfile.c
3730 mm/huge_memory.c
3689 mm/gup.c
3542 mm/mempolicy.cI suspect memcontrol.c shrunk because of the v1 split-off, leaving hugetlb.c now at #1 :)
Signature · 3 lines
Cheers, David / dhildenb
Le 23/07/2024 à 23:04, Peter Xu a écrit :
Quoted 29 lines
Quoted 23 lines
Quoted 10 lines
Nornally I don't see this as much of a "code churn" category, because it doesn't changes the code itself but only move things. I personally also prefer without code churns, but only in the case where there'll be tiny little functional changes here and there without real benefit.
It's pretty unavoidable to me when one file grows too large and we'll need to split, and in this case git doesn't have a good way to track such movement..
Yes, that's what I mean.
I've been recently thinking if we should pursue a different direction:
Just as we recently relocated most follow_huge_* stuff into
gup.c, likely we should rather look into movingcopy_huge_pmd,change_huge_pmd,copy_huge_pmd... into the files where they logically belong to.In
madvise.c, we've been doing that in some places already: Formadvise_cold_or_pageout_pte_range()we inline the code, but not formadvise_free_huge_pmd().Quoted 5 lines
pmd_trans_huge()would already compile to a NOP withoutCONFIG_TRANSPARENT_HUGEPAGE, but to make that code avoid mostCONFIG_TRANSPARENT_HUGEPAGE, we'd need a couple more function stubs to make the compiler happy while still being able to compile that code out when not required.
Quoted 7 lines
Right, I had a patch does exactly that, where it's called
pmd_is_leaf(), for example, but taking CONFIG_* into account.I remember I had some issue with that, e.g. I used to see
pmd_trans_huge()(when !THP) can optimize some path butpmd_is_leaf()didn't do the same job even if all configs were off. But that's another story and I didn't yet dig deeper. Could be something small but overlooked.
When I prepared series
https://patchwork.kernel.org/project/linux-mm/list/?series=871008 , I detected that some architectures define some pXd_leaf() for levels they don't support, that's the reason why Andrew had to drop v2 at the last minute.
And that's maybe the reason why some of the code you expect to get folded-off remains.
Since then I sent v3 that fixes that. Don't know if Andrew is planning to take it.
Christophe