mm: thp: always enable mTHP support
Subsystems (14)
cover mm: thp: always enable mTHP support
Today, if an architecture implements has_transparent_hugepage() and the CPU lacks support for PMD-sized pages, the THP code disables all THP, including mTHP support. In addition, the kernel lacks a well defined API to check for PMD-sized page support. It currently relies on has_transparent_hugepage() and thp_disabled_by_hw(), but they are not well defined and are tied to THP support.
This series addresses both issues by introducing a new well defined API to query PMD-sized page support: pgtable_has_pmd_leaves(). Using this new helper, we ensure that mTHP remains enabled even when the architecture or CPU doesn't support PMD-sized pages.
An important detail is that we need to do the same refactoring for has_transparent_pud_hugepage(). I actually have patches for this one too, I'm not including them here because I want to get some initial feedback on the general approach first (and maybe it's better to do that in a separate series).
Thanks to David Hildenbrand for suggesting this improvement and for providing guidance (all bugs and misconcentpions are mine).
Luiz Capitulino (10):
docs: tmpfs: remove implementation detail reference
mm: introduce pgtable_has_pmd_leaves()
drivers: dax: use pgtable_has_pmd_leaves()
drivers: i915 selftest: use pgtable_has_pmd_leaves()
drivers: nvdimm: use pgtable_has_pmd_leaves()
mm: debug_vm_pgtable: use pgtable_has_pmd_leaves()
treewide: rename has_transparent_hugepage() to arch_has_pmd_leaves()
mm: replace thp_disabled_by_hw() with pgtable_has_pmd_leaves()
mm: thp: always enable mTHP support
mm: thp: x86: cleanup PSE feature bit usage
Documentation/filesystems/tmpfs.rst | 5 ++--- arch/mips/include/asm/pgtable.h | 4 ++-- arch/mips/mm/tlb-r4k.c | 4 ++-- arch/powerpc/include/asm/book3s/64/hash-4k.h | 2 +- arch/powerpc/include/asm/book3s/64/hash-64k.h | 2 +- arch/powerpc/include/asm/book3s/64/pgtable.h | 10 +++++----- arch/powerpc/include/asm/book3s/64/radix.h | 2 +- arch/powerpc/mm/book3s64/hash_pgtable.c | 4 ++-- arch/s390/include/asm/pgtable.h | 4 ++-- arch/x86/include/asm/pgtable.h | 6 ------ arch/x86/include/asm/pgtable_32.h | 6 ++++++ drivers/dax/dax-private.h | 2 +- .../gpu/drm/i915/gem/selftests/huge_pages.c | 2 +- drivers/nvdimm/pfn_devs.c | 4 ++-- include/linux/huge_mm.h | 7 ------- include/linux/pgtable.h | 14 +++++++++++-- mm/debug_vm_pgtable.c | 20 +++++++++---------- mm/huge_memory.c | 13 ++++++------ mm/memory.c | 12 ++++++++++- mm/shmem.c | 8 ++++---- 20 files changed, 72 insertions(+), 59 deletions(-)
Signature · 1 line
2.51.1
RFC 1/10 docs: tmpfs: remove implementation detail reference
The tmpfs.rst doc references the has_transparent_hugepage() helper, which is an implementation detail in the kernel and not relevant for users wishing to properly configure THP support for tmpfs. Remove it.
--- Documentation/filesystems/tmpfs.rst | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-)
Documentation/filesystems/tmpfs.rst +2 -3
@@ -109,9 +109,8 @@ noswap Disables swap. Remounts must respect the original settings. ====== =========================================================== tmpfs also supports Transparent Huge Pages which requires a kernel -configured with CONFIG_TRANSPARENT_HUGEPAGE and with huge supported for -your system (has_transparent_hugepage(), which is architecture specific). -The mount options for this are: +configured with CONFIG_TRANSPARENT_HUGEPAGE and with huge pages +supported for your system. The mount options for this are: ================ ============================================================== huge=never Do not allocate huge pages. This is the default.
Signature · 1 line
2.51.1
RFC 2/10 mm: introduce pgtable_has_pmd_leaves()
In reply to Luiz CapitulinoThis is a new helper which can be used to check if the architecture supports PMD-sized pages. It's intended to replace two existing helpers:
- has_transparent_hugepages(): it's used to check for PMD-sized pages (not exactly THP support), but it may perform a hardware check so it's not intended to be used in fast paths
- thp_disabled_by_hw(): also checks for PMD-sized pages support, but uses a cached value
pgtable_has_pmd_leaves() implementation is split in two parts:
1. init_arch_has_pmd_leaves(): runs at boottime as early_initcall as a wrapper to has_transparent_hugepages(). The result is cached
2. pgtable_has_pmd_leaves(): just returns the cached value
The next commits will convert users of both has_transparent_hugepages() and thp_disabled_by_hw() to pgtable_has_pmd_leaves().
--- include/linux/pgtable.h | 10 ++++++++++ mm/memory.c | 10 ++++++++++ 2 files changed, 20 insertions(+)
include/linux/pgtable.h +10 -0
@@ -2001,6 +2001,16 @@ static inline const char *pgtable_level_to_str(enum pgtable_level level) } } +/* +* IMPORTANT: pgtable_has_pmd_leaves() can only be called after +* early_initcall, since that's when __arch_has_pmd_leaves is set +*/ +extern bool __arch_has_pmd_leaves; +static inline bool pgtable_has_pmd_leaves (void ) +{ +return __arch_has_pmd_leaves; +} +#endif /* !__ASSEMBLY__ */ #if !defined(MAX_POSSIBLE_PHYSMEM_BITS ) && !defined(CONFIG_64BIT )
mm/memory.c +10 -0
@@ -176,6 +176,16 @@ static int __init init_zero_pfn(void) }early_initcall (init_zero_pfn); +bool __arch_has_pmd_leaves__read_mostly ; +EXPORT_SYMBOL (__arch_has_pmd_leaves); + +static int __init init_arch_has_pmd_leaves (void ) +{ + __arch_has_pmd_leaves =has_transparent_hugepage (); +return 0 ; +} +early_initcall (init_arch_has_pmd_leaves); +void mm_trace_rss_stat (struct mm_struct *mm,int member) {trace_rss_stat (mm, member);
Signature · 1 line
2.51.1
RFC 3/10 drivers: dax: use pgtable_has_pmd_leaves()
In reply to Luiz Capitulinodav_align_valid() uses has_transparent_hugepage() to check if PMD-sized pages are supported, use pgtable_has_pmd_leaves() instead.
--- drivers/dax/dax-private.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/dax/dax-private.h +1 -1
@@ -117,7 +117,7 @@ static inline bool dax_align_valid(unsigned long align) {if (align ==PUD_SIZE &&IS_ENABLED (CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD ))return true; -if (align ==PMD_SIZE &&has_transparent_hugepage ()) +if (align ==PMD_SIZE &&pgtable_has_pmd_leaves ())return true;if (align ==PAGE_SIZE )return true;
Signature · 1 line
2.51.1
RFC 4/10 drivers: i915 selftest: use pgtable_has_pmd_leaves()
In reply to Luiz Capitulinoigt_can_allocate_thp() uses has_transparente_hugepage() to check if PMD-sized pages are supported, use pgtable_has_pmd_leaves() instead.
--- drivers/gpu/drm/i915/gem/selftests/huge_pages.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/gpu/drm/i915/gem/selftests/huge_pages.c +1 -1
@@ -1316,7 +1316,7 @@ typedef struct drm_i915_gem_object *static inline bool igt_can_allocate_thp (struct drm_i915_private *i915) { -return i915->mm.gemfs &&has_transparent_hugepage (); +return i915->mm.gemfs &&pgtable_has_pmd_leaves (); } static struct drm_i915_gem_object *
Signature · 1 line
2.51.1
RFC 6/10 mm: debug_vm_pgtable: use pgtable_has_pmd_leaves()
In reply to Luiz Capitulinodebug_vm_pgtable calls has_transparent_hugepage() in multiple places to check if PMD-sized pages are supported, use pgtable_has_pmd_leaves() instead.
--- mm/debug_vm_pgtable.c | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-)
mm/debug_vm_pgtable.c +10 -10
@@ -170,7 +170,7 @@ static void __init pmd_basic_tests(struct pgtable_debug_args *args, int idx)unsigned long val = idx, *ptr = &val;pmd_t pmd; -if (!has_transparent_hugepage ()) +if (!pgtable_has_pmd_leaves ())return ;pr_debug ("Validating PMD basic (%pGv)\n" , ptr);
@@ -208,7 +208,7 @@ static void __init pmd_advanced_tests(struct pgtable_debug_args *args)pmd_t pmd;unsigned long vaddr = args->vaddr; -if (!has_transparent_hugepage ()) +if (!pgtable_has_pmd_leaves ())return ; page = (args->pmd_pfn !=ULONG_MAX ) ?pfn_to_page (args->pmd_pfn) :NULL ;
@@ -269,7 +269,7 @@ static void __init pmd_leaf_tests(struct pgtable_debug_args *args) {pmd_t pmd; -if (!has_transparent_hugepage ()) +if (!pgtable_has_pmd_leaves ())return ;pr_debug ("Validating PMD leaf\n" );
@@ -674,7 +674,7 @@ static void __init pmd_protnone_tests(struct pgtable_debug_args *args)if (!IS_ENABLED (CONFIG_NUMA_BALANCING ))return ; -if (!has_transparent_hugepage ()) +if (!pgtable_has_pmd_leaves ())return ;pr_debug ("Validating PMD protnone\n" );
@@ -721,7 +721,7 @@ static void __init pmd_soft_dirty_tests(struct pgtable_debug_args *args)if (!IS_ENABLED (CONFIG_MEM_SOFT_DIRTY ))return ; -if (!has_transparent_hugepage ()) +if (!pgtable_has_pmd_leaves ())return ;pr_debug ("Validating PMD soft dirty\n" );
@@ -738,7 +738,7 @@ static void __init pmd_swap_soft_dirty_tests(struct pgtable_debug_args *args) !IS_ENABLED(CONFIG_ARCH_ENABLE_THP_MIGRATION)) return; -if (!has_transparent_hugepage ()) +if (!pgtable_has_pmd_leaves ())return ;pr_debug ("Validating PMD swap soft dirty\n" );
@@ -801,7 +801,7 @@ static void __init pmd_swap_tests(struct pgtable_debug_args *args)swp_entry_t arch_entry;pmd_t pmd1, pmd2; -if (!has_transparent_hugepage ()) +if (!pgtable_has_pmd_leaves ())return ;pr_debug ("Validating PMD swap\n" );
@@ -881,7 +881,7 @@ static void __init pmd_thp_tests(struct pgtable_debug_args *args) {pmd_t pmd; -if (!has_transparent_hugepage ()) +if (!pgtable_has_pmd_leaves ())return ;pr_debug ("Validating PMD based THP\n" );
@@ -968,7 +968,7 @@ static void __init destroy_args(struct pgtable_debug_args *args) }if (IS_ENABLED (CONFIG_TRANSPARENT_HUGEPAGE ) && -has_transparent_hugepage () && +pgtable_has_pmd_leaves () && args->pmd_pfn !=ULONG_MAX ) {if (args->is_contiguous_page) {free_contig_range (args->pmd_pfn, (1 <<HPAGE_PMD_ORDER ));
@@ -1226,7 +1226,7 @@ static int __init init_args(struct pgtable_debug_args *args) }if (IS_ENABLED (CONFIG_TRANSPARENT_HUGEPAGE ) && -has_transparent_hugepage ()) { +pgtable_has_pmd_leaves ()) { page =debug_vm_pgtable_alloc_huge_page (args,HPAGE_PMD_ORDER );if (page) { args->pmd_pfn =page_to_pfn (page);
Signature · 1 line
2.51.1
RFC 7/10 treewide: rename has_transparent_hugepage() to arch_has_pmd_leaves()
In reply to Luiz CapitulinoNow that the majority of has_transparent_hugepage() callers have been converted to pgtable_has_pmd_leaves(), rename has_transparent_hugepage() to arch_has_pmd_leaves() since that's what the helper checks for.
arch_has_pmd_leaves() is supposed to be called only by init_arch_has_pmd_leaves(), except for two exeptions:
1. shmem: shmem code runs very early during boot so it can't use pgtable_has_pmd_leaves()
2. hugepage_init(): just a temporary exception, this function will be converted in a future commit
--- arch/mips/include/asm/pgtable.h | 4 ++-- arch/mips/mm/tlb-r4k.c | 4 ++-- arch/powerpc/include/asm/book3s/64/hash-4k.h | 2 +- arch/powerpc/include/asm/book3s/64/hash-64k.h | 2 +- arch/powerpc/include/asm/book3s/64/pgtable.h | 10 +++++----- arch/powerpc/include/asm/book3s/64/radix.h | 2 +- arch/powerpc/mm/book3s64/hash_pgtable.c | 4 ++-- arch/s390/include/asm/pgtable.h | 4 ++-- arch/x86/include/asm/pgtable.h | 4 ++-- include/linux/pgtable.h | 4 ++-- mm/huge_memory.c | 2 +- mm/memory.c | 2 +- mm/shmem.c | 6 +++--- 13 files changed, 25 insertions(+), 25 deletions(-)
arch/mips/include/asm/pgtable.h +2 -2
@@ -616,8 +616,8 @@ int io_remap_pfn_range(struct vm_area_struct *vma, unsigned long vaddr,/* We don't have hardware dirty/accessed bits, generic_pmdp_establish is fine.*/ #define pmdp_establish generic_pmdp_establish -#define has_transparent_hugepage has_transparent_hugepage -extern int has_transparent_hugepage (void ); +#define arch_has_pmd_leaves arch_has_pmd_leaves +extern int arch_has_pmd_leaves (void );static inline int pmd_trans_huge (pmd_t pmd) {
arch/mips/mm/tlb-r4k.c +2 -2
@@ -430,7 +430,7 @@ void add_wired_entry(unsigned long entrylo0, unsigned long entrylo1,#ifdef CONFIG_TRANSPARENT_HUGEPAGE -int has_transparent_hugepage (void ) +int arch_has_pmd_leaves (void ) {static unsigned int mask =-1 ;
@@ -446,7 +446,7 @@ int has_transparent_hugepage(void) }return mask ==PM_HUGE_MASK ; } -EXPORT_SYMBOL (has_transparent_hugepage); +EXPORT_SYMBOL (arch_has_pmd_leaves);#endif /* CONFIG_TRANSPARENT_HUGEPAGE */
arch/powerpc/include/asm/book3s/64/hash-4k.h +1 -1
@@ -165,7 +165,7 @@ extern void hash__pgtable_trans_huge_deposit(struct mm_struct *mm, pmd_t *pmdp,extern pgtable_t hash__pgtable_trans_huge_withdraw (struct mm_struct *mm,pmd_t *pmdp);extern pmd_t hash__pmdp_huge_get_and_clear (struct mm_struct *mm,unsigned long addr,pmd_t *pmdp); -extern int hash__has_transparent_hugepage (void ); +extern int hash__arch_has_pmd_leaves (void );#endif #endif /* !__ASSEMBLER__ */
arch/powerpc/include/asm/book3s/64/hash-64k.h +1 -1
@@ -278,7 +278,7 @@ extern void hash__pgtable_trans_huge_deposit(struct mm_struct *mm, pmd_t *pmdp,extern pgtable_t hash__pgtable_trans_huge_withdraw (struct mm_struct *mm,pmd_t *pmdp);extern pmd_t hash__pmdp_huge_get_and_clear (struct mm_struct *mm,unsigned long addr,pmd_t *pmdp); -extern int hash__has_transparent_hugepage (void ); +extern int hash__arch_has_pmd_leaves (void );#endif /* CONFIG_TRANSPARENT_HUGEPAGE */ #endif /* __ASSEMBLER__ */
arch/powerpc/include/asm/book3s/64/pgtable.h +5 -5
@@ -1094,14 +1094,14 @@ static inline void update_mmu_cache_pud(struct vm_area_struct *vma, { } -extern int hash__has_transparent_hugepage (void ); -static inline int has_transparent_hugepage (void ) +extern int hash__arch_has_pmd_leaves (void ); +static inline int arch_has_pmd_leaves (void ) {if (radix_enabled ()) -return radix__has_transparent_hugepage (); -return hash__has_transparent_hugepage (); +return radix__arch_has_pmd_leaves (); +return hash__arch_has_pmd_leaves (); } -#define has_transparent_hugepage has_transparent_hugepage +#define arch_has_pmd_leaves arch_has_pmd_leavesstatic inline int has_transparent_pud_hugepage (void ) {
arch/powerpc/include/asm/book3s/64/radix.h +1 -1
@@ -298,7 +298,7 @@ extern pmd_t radix__pmdp_huge_get_and_clear(struct mm_struct *mm,pud_t radix__pudp_huge_get_and_clear (struct mm_struct *mm,unsigned long addr,pud_t *pudp); -static inline int radix__has_transparent_hugepage (void ) +static inline int radix__arch_has_pmd_leaves (void ) {/* For radix 2M at PMD level means thp */ if (mmu_psize_defs[MMU_PAGE_2M ].shift ==PMD_SHIFT )
arch/powerpc/mm/book3s64/hash_pgtable.c +2 -2
@@ -366,7 +366,7 @@ pmd_t hash__pmdp_huge_get_and_clear(struct mm_struct *mm,return old_pmd; } -int hash__has_transparent_hugepage (void ) +int hash__arch_has_pmd_leaves (void ) {if (!mmu_has_feature (MMU_FTR_16M_PAGE ))
@@ -395,7 +395,7 @@ int hash__has_transparent_hugepage(void) return 1; } -EXPORT_SYMBOL_GPL (hash__has_transparent_hugepage); +EXPORT_SYMBOL_GPL (hash__arch_has_pmd_leaves);#endif /* CONFIG_TRANSPARENT_HUGEPAGE */
arch/s390/include/asm/pgtable.h +2 -2
@@ -1881,8 +1881,8 @@ static inline int pmd_trans_huge(pmd_t pmd)return pmd_leaf (pmd); } -#define has_transparent_hugepage has_transparent_hugepage -static inline int has_transparent_hugepage (void ) +#define arch_has_pmd_leaves arch_has_pmd_leaves +static inline int arch_has_pmd_leaves (void ) {return cpu_has_edat1 () ?1 :0 ; }
arch/x86/include/asm/pgtable.h +2 -2
@@ -313,8 +313,8 @@ static inline int pud_trans_huge(pud_t pud) }#endif -#define has_transparent_hugepage has_transparent_hugepage -static inline int has_transparent_hugepage (void ) +#define arch_has_pmd_leaves arch_has_pmd_leaves +static inline int arch_has_pmd_leaves (void ) {return boot_cpu_has (X86_FEATURE_PSE ); }
include/linux/pgtable.h +2 -2
@@ -2026,8 +2026,8 @@ static inline bool pgtable_has_pmd_leaves(void)#endif #endif -#ifndef has_transparent_hugepage -#define has_transparent_hugepage () IS_BUILTIN(CONFIG_TRANSPARENT_HUGEPAGE) +#ifndef arch_has_pmd_leaves +#define arch_has_pmd_leaves () IS_BUILTIN(CONFIG_TRANSPARENT_HUGEPAGE)#endif #ifndef has_transparent_pud_hugepage
mm/huge_memory.c +1 -1
@@ -904,7 +904,7 @@ static int __init hugepage_init(void)int err;struct kobject *hugepage_kobj; -if (!has_transparent_hugepage ()) { +if (!arch_has_pmd_leaves ()) { transparent_hugepage_flags =1 <<TRANSPARENT_HUGEPAGE_UNSUPPORTED ;return -EINVAL ; }
mm/memory.c +1 -1
@@ -181,7 +181,7 @@ EXPORT_SYMBOL(__arch_has_pmd_leaves);static int __init init_arch_has_pmd_leaves (void ) { - __arch_has_pmd_leaves =has_transparent_hugepage (); + __arch_has_pmd_leaves =arch_has_pmd_leaves ();return 0 ; }early_initcall (init_arch_has_pmd_leaves);
mm/shmem.c +3 -3
@@ -672,7 +672,7 @@ static int shmem_parse_huge(const char *str)else return -EINVAL ; -if (!has_transparent_hugepage () && +if (!arch_has_pmd_leaves () && huge !=SHMEM_HUGE_NEVER && huge !=SHMEM_HUGE_DENY )return -EINVAL ;
@@ -4646,7 +4646,7 @@ static int shmem_parse_one(struct fs_context *fc, struct fs_parameter *param) ctx->huge = result.uint_32;if (ctx->huge !=SHMEM_HUGE_NEVER && !(IS_ENABLED (CONFIG_TRANSPARENT_HUGEPAGE ) && -has_transparent_hugepage ())) +arch_has_pmd_leaves ())) goto unsupported_parameter; ctx->seen |=SHMEM_SEEN_HUGE ;break ;
@@ -5430,7 +5430,7 @@ void __init shmem_init(void)#endif #ifdef CONFIG_TRANSPARENT_HUGEPAGE -if (has_transparent_hugepage () && shmem_huge >SHMEM_HUGE_DENY ) +if (arch_has_pmd_leaves () && shmem_huge >SHMEM_HUGE_DENY )SHMEM_SB (shm_mnt->mnt_sb)->huge = shmem_huge;else shmem_huge =SHMEM_HUGE_NEVER ;/* just in case it was patched */
Signature · 1 line
2.51.1
RFC 9/10 mm: thp: always enable mTHP support
In reply to Luiz CapitulinoIf PMD-sized pages are not supported on an architecture (ie. the arch implements arch_has_pmd_leaves() and it returns false) then the current code disables all THP, including mTHP.
This commit fixes this by allowing mTHP to be always enabled for all archs. When PMD-sized pages are not supported, its sysfs entry won't be created and their mapping will be disallowed at page-fault time.
--- mm/huge_memory.c | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-)
mm/huge_memory.c +7 -4
@@ -115,6 +115,9 @@ unsigned long __thp_vma_allowable_orders(struct vm_area_struct *vma,else supported_orders =THP_ORDERS_ALL_FILE_DEFAULT ; +if (!pgtable_has_pmd_leaves ()) + supported_orders &= ~BIT (PMD_ORDER ); + orders &= supported_orders;if (!orders)return 0 ;
@@ -122,7 +125,7 @@ unsigned long __thp_vma_allowable_orders(struct vm_area_struct *vma,if (!vma->vm_mm)/* vdso */ return 0 ; -if (!pgtable_has_pmd_leaves () ||vma_thp_disabled (vma, vm_flags, forced_collapse)) +if (vma_thp_disabled (vma, vm_flags, forced_collapse))return 0 ;/* khugepaged doesn't collapse DAX vma, but page fault is fine. */
@@ -805,6 +808,9 @@ static int __init hugepage_init_sysfs(struct kobject **hugepage_kobj) } orders =THP_ORDERS_ALL_ANON |THP_ORDERS_ALL_FILE_DEFAULT ; +if (!pgtable_has_pmd_leaves ()) + orders &= ~BIT (PMD_ORDER ); + order =highest_order (orders);while (orders) { thpsize =thpsize_create (order, *hugepage_kobj);
@@ -904,9 +910,6 @@ static int __init hugepage_init(void)int err;struct kobject *hugepage_kobj; -if (!pgtable_has_pmd_leaves ()) -return -EINVAL ; -/* * hugepages can't be allocated by the buddy allocator */
Signature · 1 line
2.51.1
RFC 8/10 mm: replace thp_disabled_by_hw() with pgtable_has_pmd_leaves()
In reply to Luiz CapitulinoDespite its name, thp_disabled_by_hw() only checks whether the architecture supports PMD-sized pages. It returns true when TRANSPARENT_HUGEPAGE_UNSUPPORTED is set, which occurs if the architecture implements arch_has_pmd_leaves() and that function returns false.
Since pgtable_has_pmd_leaves() provide the same semantics, use it instead.
--- include/linux/huge_mm.h | 7 ------- mm/huge_memory.c | 6 ++---- mm/memory.c | 2 +- mm/shmem.c | 2 +- 4 files changed, 4 insertions(+), 13 deletions(-)
include/linux/huge_mm.h +0 -7
@@ -47,7 +47,6 @@ vm_fault_t vmf_insert_folio_pud(struct vm_fault *vmf, struct folio *folio, bool write);enum transparent_hugepage_flag { -TRANSPARENT_HUGEPAGE_UNSUPPORTED ,TRANSPARENT_HUGEPAGE_FLAG ,TRANSPARENT_HUGEPAGE_REQ_MADV_FLAG ,TRANSPARENT_HUGEPAGE_DEFRAG_DIRECT_FLAG ,
@@ -352,12 +351,6 @@ static inline bool vma_thp_disabled(struct vm_area_struct *vma,return mm_flags_test (MMF_DISABLE_THP_EXCEPT_ADVISED , vma->vm_mm); } -static inline bool thp_disabled_by_hw (void ) -{ -/* If the hardware/firmware marked hugepage support disabled. */ -return transparent_hugepage_flags & (1 <<TRANSPARENT_HUGEPAGE_UNSUPPORTED ); -} -unsigned long thp_get_unmapped_area (struct file *filp,unsigned long addr,unsigned long len,unsigned long pgoff,unsigned long flags); unsigned long thp_get_unmapped_area_vmflags(struct file *filp, unsigned long addr,
mm/huge_memory.c +2 -4
@@ -122,7 +122,7 @@ unsigned long __thp_vma_allowable_orders(struct vm_area_struct *vma,if (!vma->vm_mm)/* vdso */ return 0 ; -if (thp_disabled_by_hw () ||vma_thp_disabled (vma, vm_flags, forced_collapse)) +if (!pgtable_has_pmd_leaves () ||vma_thp_disabled (vma, vm_flags, forced_collapse))return 0 ;/* khugepaged doesn't collapse DAX vma, but page fault is fine. */
@@ -904,10 +904,8 @@ static int __init hugepage_init(void)int err;struct kobject *hugepage_kobj; -if (!arch_has_pmd_leaves ()) { - transparent_hugepage_flags =1 <<TRANSPARENT_HUGEPAGE_UNSUPPORTED ; +if (!pgtable_has_pmd_leaves ())return -EINVAL ; - }/* * hugepages can't be allocated by the buddy allocator
mm/memory.c +1 -1
@@ -5345,7 +5345,7 @@ vm_fault_t do_set_pmd(struct vm_fault *vmf, struct folio *folio, struct page *pa* PMD mappings if THPs are disabled. As we already have a THP, * behave as if we are forcing a collapse. */ -if (thp_disabled_by_hw () ||vma_thp_disabled (vma, vma->vm_flags, +if (!pgtable_has_pmd_leaves () ||vma_thp_disabled (vma, vma->vm_flags,/* forced_collapse=*/ true))return ret;
mm/shmem.c +1 -1
@@ -1780,7 +1780,7 @@ unsigned long shmem_allowable_huge_orders(struct inode *inode,vm_flags_t vm_flags = vma ? vma->vm_flags :0 ;unsigned int global_orders; -if (thp_disabled_by_hw () || (vma &&vma_thp_disabled (vma, vm_flags, shmem_huge_force))) +if (!pgtable_has_pmd_leaves () || (vma &&vma_thp_disabled (vma, vm_flags, shmem_huge_force)))return 0 ; global_orders = shmem_huge_global_enabled(inode, index, write_end,
Signature · 1 line
2.51.1
RFC 10/10 mm: thp: x86: cleanup PSE feature bit usage
In reply to Luiz CapitulinoHistorically, THP support on x86 checked the PSE feature bit to enable THP. On 64-bit, this check is redundant since PSE is always enabled by default for compatibility. On 32-bit, PSE enables 4MB page support and must be checked. To clean this up, this commit:
1. Drops arch_has_pmd_leaves() from common x86 code. For 64-bit, we assume PMD-sized pages are always supported
2. Checks for PSE only on 32-bit, using arch_has_pmd_leaves()
--- arch/x86/include/asm/pgtable.h | 6 ------ arch/x86/include/asm/pgtable_32.h | 6 ++++++ 2 files changed, 6 insertions(+), 6 deletions(-)
arch/x86/include/asm/pgtable.h +0 -6
@@ -313,12 +313,6 @@ static inline int pud_trans_huge(pud_t pud) }#endif -#define arch_has_pmd_leaves arch_has_pmd_leaves -static inline int arch_has_pmd_leaves (void ) -{ -return boot_cpu_has (X86_FEATURE_PSE ); -} -#ifdef CONFIG_ARCH_SUPPORTS_PMD_PFNMAP static inline bool pmd_special (pmd_t pmd) {
arch/x86/include/asm/pgtable_32.h +6 -0
@@ -45,6 +45,12 @@ do { \flush_tlb_one_kernel ((vaddr)); \ } while (0) +#define arch_has_pmd_leaves arch_has_pmd_leaves +static inline int arch_has_pmd_leaves (void ) +{ +return boot_cpu_has (X86_FEATURE_PSE ); +} +#endif /* !__ASSEMBLER__ */ /*
Signature · 1 line
2.51.1
RFC 5/10 drivers: nvdimm: use pgtable_has_pmd_leaves()
In reply to Luiz Capitulinond_pfn_supported_alignments() and nd_pfn_supported_alignments() use has_transparent_hugepage() to check if PMD-sized pages are supported, use pgtable_has_pmd_leaves() instead.
--- drivers/nvdimm/pfn_devs.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
drivers/nvdimm/pfn_devs.c +2 -2
@@ -94,7 +94,7 @@ static unsigned long *nd_pfn_supported_alignments(unsigned long *alignments) alignments[0 ] =PAGE_SIZE ; -if (has_transparent_hugepage ()) { +if (pgtable_has_pmd_leaves ()) { alignments[1 ] =HPAGE_PMD_SIZE ;if (has_transparent_pud_hugepage ()) alignments[2 ] =HPAGE_PUD_SIZE ;
@@ -109,7 +109,7 @@ static unsigned long *nd_pfn_supported_alignments(unsigned long *alignments)static unsigned long nd_pfn_default_alignment (void ) { -if (has_transparent_hugepage ()) +if (pgtable_has_pmd_leaves ())return HPAGE_PMD_SIZE ;return PAGE_SIZE ; }
Signature · 1 line
2.51.1
Re: [RFC 01/10] docs: tmpfs: remove implementation detail reference
In reply to Luiz CapitulinoQuoted 6 lines · Luiz Capitulino
The
tmpfs.rstdoc references thehas_transparent_hugepage()helper, which is an implementation detail in the kernel and not relevant for users wishing to properly configure THP support for tmpfs. Remove it.Signed-off-by Luiz Capitulino <[email protected]>---
Signature · 3 lines
Cheers David
Re: [RFC 03/10] drivers: dax: use pgtable_has_pmd_leaves()
In reply to Luiz CapitulinoQuoted 13 lines · Luiz Capitulino
dav_align_valid()useshas_transparent_hugepage()to check if PMD-sized pages are supported, usepgtable_has_pmd_leaves()instead.Signed-off-by Luiz Capitulino <[email protected]>--- drivers/dax/dax-private.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)drivers/dax/dax-private.h +0
-0
drivers/dax/dax-private.h static inline bool dax_align_valid(unsigned long align) · line 118
Quoted 2 lines
drivers/dax/dax-private.h +0
-0@@ -117,7 +117,7 @@ static inline bool dax_align_valid(unsigned long align) {if (align ==PUD_SIZE &&IS_ENABLED (CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD ))
As discussed offlist, it would be great to later also have this be
if (align == PUD_SIZE && pgtable_has_pud_leaves()
Signature · 3 lines
Cheers David
David Hildenbrand (Red Hat)
maintainerRe: [RFC 04/10] drivers: i915 selftest: use pgtable_has_pmd_leaves()
In reply to Luiz CapitulinoQuoted 3 lines · Luiz Capitulino
igt_can_allocate_thp()useshas_transparente_hugepage()to check if PMD-sized pages are supported, usepgtable_has_pmd_leaves()instead.
Right, I think we really only care about PMDs here for now.
Signature · 3 lines
Cheers David
David Hildenbrand (Red Hat)
maintainerRe: [RFC 04/10] drivers: i915 selftest: use pgtable_has_pmd_leaves()
In reply to Luiz CapitulinoQuoted 13 lines · Luiz Capitulino
igt_can_allocate_thp()useshas_transparente_hugepage()to check if PMD-sized pages are supported, usepgtable_has_pmd_leaves()instead.Signed-off-by Luiz Capitulino <[email protected]>--- drivers/gpu/drm/i915/gem/selftests/huge_pages.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)drivers/gpu/drm/i915/gem/selftests/huge_pages.c +0
-0
drivers/gpu/drm/i915/gem/selftests/huge_pages.c typedef struct drm_i915_gem_object * · line 1319
Quoted 5 lines
drivers/gpu/drm/i915/gem/selftests/huge_pages.c +1
-1@@ -1316,7 +1316,7 @@ typedef struct drm_i915_gem_object *static inline bool igt_can_allocate_thp (struct drm_i915_private *i915) { -return i915->mm.gemfs &&has_transparent_hugepage (); +return i915->mm.gemfs &&pgtable_has_pmd_leaves ();
On second thought, is it problematic that we might be losing the CONFIG_TRANSPARENT_HUGEPAGE check? Should we check for that separately?
Signature · 3 lines
Cheers David
David Hildenbrand (Red Hat)
maintainerRe: [RFC 05/10] drivers: nvdimm: use pgtable_has_pmd_leaves()
In reply to Luiz CapitulinoQuoted 14 lines · Luiz Capitulino
nd_pfn_supported_alignments()andnd_pfn_supported_alignments()usehas_transparent_hugepage()to check if PMD-sized pages are supported, usepgtable_has_pmd_leaves()instead.Signed-off-by Luiz Capitulino <[email protected]>--- drivers/nvdimm/pfn_devs.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)drivers/nvdimm/pfn_devs.c +0
-0
drivers/nvdimm/pfn_devs.c static unsigned long *nd_pfn_supported_alignments(unsigned long *alignments) · line 99
Quoted 7 lines
drivers/nvdimm/pfn_devs.c +1
-1@@ -94,7 +94,7 @@ static unsigned long *nd_pfn_supported_alignments(unsigned long *alignments) alignments[0 ] =PAGE_SIZE ; -if (has_transparent_hugepage ()) { +if (pgtable_has_pmd_leaves ()) { alignments[1 ] =HPAGE_PMD_SIZE ;if (has_transparent_pud_hugepage ())
Yeah, that should become pgtable_has_pud_leaves() later.
drivers/nvdimm/pfn_devs.c static unsigned long *nd_pfn_supported_alignments(unsigned long *alignments) · line 115
Quoted 10 lines
drivers/nvdimm/pfn_devs.c +1
-1alignments[2 ] =HPAGE_PUD_SIZE ;@@ -109,7 +109,7 @@ static unsigned long *nd_pfn_supported_alignments(unsigned long *alignments)static unsigned long nd_pfn_default_alignment (void ) { -if (has_transparent_hugepage ()) +if (pgtable_has_pmd_leaves ())return HPAGE_PMD_SIZE ;return PAGE_SIZE ; }
Similar question, what if the kernel would be built without CONFIG_TRANSPARENT_HUGEPAGE, would it matter here?
In that case, an additional eary "if (!IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE))" might do the trick.
Signature · 3 lines
Cheers David
David Hildenbrand (Red Hat)
maintainerRe: [RFC 06/10] mm: debug_vm_pgtable: use pgtable_has_pmd_leaves()
In reply to Luiz CapitulinoQuoted 2 lines · Luiz Capitulino
debug_vm_pgtablecallshas_transparent_hugepage()in multiple places to check if PMD-sized pages are supported, usepgtable_has_pmd_leaves()
Is that code even dealing with pages? Likely we really only want to check whether PMD leaves are supported, independent of any pages.
Quoted 12 lines
instead.
Signed-off-by Luiz Capitulino <[email protected]>--- mm/debug_vm_pgtable.c | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-)mm/debug_vm_pgtable.c +0
-0
mm/debug_vm_pgtable.c static void __init pmd_basic_tests(struct pgtable_debug_args *args, int idx) · line 174
Quoted 6 lines
mm/debug_vm_pgtable.c +1
-1@@ -170,7 +170,7 @@ static void __init pmd_basic_tests(struct pgtable_debug_args *args, int idx)unsigned long val = idx, *ptr = &val;pmd_t pmd; -if (!has_transparent_hugepage ()) +if (!pgtable_has_pmd_leaves ())return ;
All of these are currently protected by CONFIG_TRANSPARENT_HUGEPAGE, right?
Signature · 3 lines
Cheers David
David Hildenbrand (Red Hat)
maintainerRe: [RFC 07/10] treewide: rename has_transparent_hugepage() to arch_has_pmd_leaves()
In reply to Luiz CapitulinoQuoted 9 lines · Luiz Capitulino
Now that the majority of
has_transparent_hugepage()callers have been converted topgtable_has_pmd_leaves(), renamehas_transparent_hugepage()toarch_has_pmd_leaves()since that's what the helper checks for.
arch_has_pmd_leaves()is supposed to be called only byinit_arch_has_pmd_leaves(), except for two exeptions:1. shmem: shmem code runs very early during boot so it can't use
pgtable_has_pmd_leaves()
Can't we just initialize pgtable_has_pmd_leaves() earlier then?
Quoted 5 lines
2.
hugepage_init(): just a temporary exception, this function will be converted in a future commitSigned-off-by Luiz Capitulino <[email protected]>---
[...]
include/linux/pgtable.h static inline bool pgtable_has_pmd_leaves(void) · line 2031
Quoted 13 lines
include/linux/pgtable.h +2
-2@@ -2026,8 +2026,8 @@ static inline bool pgtable_has_pmd_leaves(void)#endif #endif -#ifndef has_transparent_hugepage -#define has_transparent_hugepage () IS_BUILTIN(CONFIG_TRANSPARENT_HUGEPAGE) +#ifndef arch_has_pmd_leaves +#define arch_has_pmd_leaves () IS_BUILTIN(CONFIG_TRANSPARENT_HUGEPAGE)#endif
Ah, so it stays for now only set with CONFIG_TRANSPARENT_HUGEPAGE. I guess that's something to sort out later :)
Signature · 3 lines
Cheers David
David Hildenbrand (Red Hat)
maintainerQuoted 18 lines · Luiz Capitulino
If PMD-sized pages are not supported on an architecture (ie. the arch implements
arch_has_pmd_leaves()and it returns false) then the current code disables all THP, including mTHP.This commit fixes this by allowing mTHP to be always enabled for all archs. When PMD-sized pages are not supported, its sysfs entry won't be created and their mapping will be disallowed at page-fault time.
Signed-off-by Luiz Capitulino <[email protected]>--- mm/huge_memory.c | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-)mm/huge_memory.c +0
-0
mm/huge_memory.c unsigned long __thp_vma_allowable_orders(struct vm_area_struct *vma, · line 120
Quoted 6 lines
mm/huge_memory.c +3
-0@@ -115,6 +115,9 @@ unsigned long __thp_vma_allowable_orders(struct vm_area_struct *vma,else supported_orders =THP_ORDERS_ALL_FILE_DEFAULT ; +if (!pgtable_has_pmd_leaves ()) + supported_orders &= ~BIT (PMD_ORDER ); +
Won't this leave PUD_ORDER set?
Ideally, later we'd also have a
if (!pgtable_has_pud_leaves())
supported_orders &= ~BIT(PUD_ORDER);
For now you could simply clear PUD_ORDER for now as well.
No PMD leaves -> No PUD leaves, so clear them in all cases to not leave PUD_ORDER set.
Signature · 3 lines
Cheers David
David Hildenbrand (Red Hat)
maintainerRe: [RFC 10/10] mm: thp: x86: cleanup PSE feature bit usage
In reply to Luiz CapitulinoQuoted 3 lines · Luiz Capitulino
Historically, THP support on x86 checked the PSE feature bit to enable THP. On 64-bit, this check is redundant since PSE is always enabled by default for compatibility. On 32-bit, PSE enables 4MB page support and
Or 2MB support with 4byte PTEs I guess?
Quoted 2 lines
must be checked. To clean this up, this commit:
Likely the subject should start with: "x86/mm:"
Quoted 35 lines
1. Drops
arch_has_pmd_leaves()from common x86 code. For 64-bit, we assume PMD-sized pages are always supported2. Checks for PSE only on 32-bit, using
arch_has_pmd_leaves()Signed-off-by Luiz Capitulino <[email protected]>--- arch/x86/include/asm/pgtable.h | 6 ------ arch/x86/include/asm/pgtable_32.h | 6 ++++++ 2 files changed, 6 insertions(+), 6 deletions(-)arch/x86/include/asm/pgtable.h +0
-6@@ -313,12 +313,6 @@ static inline int pud_trans_huge(pud_t pud) }#endif -#define arch_has_pmd_leaves arch_has_pmd_leaves -static inline int arch_has_pmd_leaves (void ) -{ -return boot_cpu_has (X86_FEATURE_PSE ); -} -#ifdef CONFIG_ARCH_SUPPORTS_PMD_PFNMAP static inline bool pmd_special (pmd_t pmd) {arch/x86/include/asm/pgtable_32.h +0
-0@@ -45,6 +45,12 @@ do { \flush_tlb_one_kernel ((vaddr)); \ }while (0 )
arch/x86/include/asm/pgtable_32.h do { \ · line 56
Quoted 10 lines
arch/x86/include/asm/pgtable_32.h +6
-0+#define arch_has_pmd_leaves arch_has_pmd_leaves +static inline int arch_has_pmd_leaves (void ) +{ +return boot_cpu_has (X86_FEATURE_PSE ); +} +#endif /* !__ASSEMBLER__ */ /*
Right, IIUC 64bit should neve rrequire this check.
Signature · 3 lines
Cheers David
Re: [RFC 04/10] drivers: i915 selftest: use pgtable_has_pmd_leaves()
In reply to David Hildenbrand (Red Hat)Quoted 9 lines · David Hildenbrand (Red Hat)
Quoted 8 lines · Luiz Capitulino
igt_can_allocate_thp()useshas_transparente_hugepage()to check if PMD-sized pages are supported, usepgtable_has_pmd_leaves()instead.Signed-off-by Luiz Capitulino <[email protected]>--- drivers/gpu/drm/i915/gem/selftests/huge_pages.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
Quoted 11 lines
drivers/gpu/drm/i915/gem/selftests/huge_pages.c typedef struct drm_i915_gem_object * · line 1318
Quoted 9 lines
drivers/gpu/drm/i915/gem/selftests/huge_pages.c +1
-1@@ -1316,7 +1316,7 @@ typedef struct drm_i915_gem_object *static inline bool igt_can_allocate_thp (struct drm_i915_private *i915) { -return i915->mm.gemfs &&has_transparent_hugepage (); +return i915->mm.gemfs &&pgtable_has_pmd_leaves ();On second thought, is it problematic that we might be losing the
CONFIG_TRANSPARENT_HUGEPAGEcheck? Should we check for that separately?
That's a good point.
In this RFC, pgtable_has_pmd_leaves() should be functionally equivalent to has_transparent_hugepage() so I think we're good. That beind said, I also think that we should disentangle pgtable_has_pmd_leaves() from THP now or in the future. When we do this the breakage you're spotting will happen.
What about adding thp_has_pmd_support() which does:
return IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE) && pgtable_has_pmd_leaves();
Then I can convert all the cases you spotted to thp_has_pmd_support().
Re: [RFC 07/10] treewide: rename has_transparent_hugepage() to arch_has_pmd_leaves()
In reply to David Hildenbrand (Red Hat)Quoted 5 lines · David Hildenbrand (Red Hat)
Quoted 4 lines · Luiz Capitulino
Now that the majority of
has_transparent_hugepage()callers have been converted topgtable_has_pmd_leaves(), renamehas_transparent_hugepage()toarch_has_pmd_leaves()since that's what the helper checks for.
Quoted 7 lines
Quoted 5 lines
arch_has_pmd_leaves()is supposed to be called only byinit_arch_has_pmd_leaves(), except for two exeptions:1. shmem: shmem code runs very early during boot so it can't use
pgtable_has_pmd_leaves()Can't we just initialize
pgtable_has_pmd_leaves()earlier then?
I can look into doing that. When I worked on this RFC I wondered if arch_has_pmd_leaves() (when implemented by the arch) could run so early given that some (all?) archs check feature bits so they must be available this early as well. But I'll check this, having pgtable_has_pmd_leaves() being available as early as possible is probably the right thing to do.
Quoted 9 lines
Quoted 5 lines
2.
hugepage_init(): just a temporary exception, this function will be converted in a future commitSigned-off-by Luiz Capitulino <[email protected]>---
[...]
Quoted 14 lines
include/linux/pgtable.h static inline bool pgtable_has_pmd_leaves(void) · line 2030
Quoted 12 lines
include/linux/pgtable.h +2
-2@@ -2026,8 +2026,8 @@ static inline bool pgtable_has_pmd_leaves(void)#endif #endif -#ifndef has_transparent_hugepage -#define has_transparent_hugepage () IS_BUILTIN(CONFIG_TRANSPARENT_HUGEPAGE) +#ifndef arch_has_pmd_leaves +#define arch_has_pmd_leaves () IS_BUILTIN(CONFIG_TRANSPARENT_HUGEPAGE)#endif Ah, so it stays for now only set with
CONFIG_TRANSPARENT_HUGEPAGE. I guess that's something to sort out later :)
I suggested something we could do in this series. Also, I skipped commenting on all the cases you spotted as I think they refer to the same issue (please, do point out if you think I'm wrong).
Quoted 29 lines · David Hildenbrand (Red Hat)
Quoted 18 lines · Luiz Capitulino
If PMD-sized pages are not supported on an architecture (ie. the arch implements
arch_has_pmd_leaves()and it returns false) then the current code disables all THP, including mTHP.This commit fixes this by allowing mTHP to be always enabled for all archs. When PMD-sized pages are not supported, its sysfs entry won't be created and their mapping will be disallowed at page-fault time.
Signed-off-by Luiz Capitulino <[email protected]>---
mm/huge_memory.c| 11 +++++++----1 file changed, 7 insertions(+), 4 deletions(-)mm/huge_memory.c +0
-0@@ -115,6 +115,9 @@ unsigned long __thp_vma_allowable_orders(struct vm_area_struct *vma,Quoted 5 lines
else
supported_orders=THP_ORDERS_ALL_FILE_DEFAULT; + if (!pgtable_has_pmd_leaves())
+supported_orders&= ~BIT(PMD_ORDER);
+Won't this leave
PUD_ORDERset?Ideally, later we'd also have a
Quoted 7 lines
if (!pgtable_has_pud_leaves())
supported_orders &= ~BIT(PUD_ORDER);For now you could simply clear
PUD_ORDERfor now as well.No PMD leaves -> No PUD leaves, so clear them in all cases to not leave
PUD_ORDERset.
OK, I'll do it.
Re: [RFC 10/10] mm: thp: x86: cleanup PSE feature bit usage
In reply to David Hildenbrand (Red Hat)Quoted 11 lines · David Hildenbrand (Red Hat)
Quoted 3 lines · Luiz Capitulino
Historically, THP support on x86 checked the PSE feature bit to enable THP. On 64-bit, this check is redundant since PSE is always enabled by default for compatibility. On 32-bit, PSE enables 4MB page support and
Or 2MB support with 4byte PTEs I guess?
Quoted 2 lines
must be checked. To clean this up, this commit:
Likely the subject should start with: "x86/mm:"
You're right on both points. I'll fix the changelog for the next version.
Quoted 47 lines
Quoted 34 lines
1. Drops
arch_has_pmd_leaves()from common x86 code. For 64-bit, we assume PMD-sized pages are always supported2. Checks for PSE only on 32-bit, using
arch_has_pmd_leaves()Signed-off-by Luiz Capitulino <[email protected]>--- arch/x86/include/asm/pgtable.h | 6 ------ arch/x86/include/asm/pgtable_32.h | 6 ++++++ 2 files changed, 6 insertions(+), 6 deletions(-)arch/x86/include/asm/pgtable.h +0
-6@@ -313,12 +313,6 @@ static inline int pud_trans_huge(pud_t pud) }#endif -#define arch_has_pmd_leaves arch_has_pmd_leaves -static inline int arch_has_pmd_leaves (void ) -{ -return boot_cpu_has (X86_FEATURE_PSE ); -} -#ifdef CONFIG_ARCH_SUPPORTS_PMD_PFNMAP static inline bool pmd_special (pmd_t pmd) {arch/x86/include/asm/pgtable_32.h +0
-0arch/x86/include/asm/pgtable_32.h do { \ · line 54
Quoted 10 lines
arch/x86/include/asm/pgtable_32.h +6
-0@@ -45,6 +45,12 @@ do { \flush_tlb_one_kernel ((vaddr)); \ } while (0) +#define arch_has_pmd_leaves arch_has_pmd_leaves +static inline int arch_has_pmd_leaves (void ) +{ +return boot_cpu_has (X86_FEATURE_PSE ); +} +#endif /* !__ASSEMBLER__ */ /* Right, IIUC 64bit should neve rrequire this check.
David Hildenbrand (Red Hat)
maintainerRe: [RFC 04/10] drivers: i915 selftest: use pgtable_has_pmd_leaves()
In reply to Luiz CapitulinoQuoted 30 lines · Luiz Capitulino
Quoted 9 lines · David Hildenbrand (Red Hat)
Quoted 8 lines · Luiz Capitulino
igt_can_allocate_thp()useshas_transparente_hugepage()to check if PMD-sized pages are supported, usepgtable_has_pmd_leaves()instead.Signed-off-by Luiz Capitulino <[email protected]>--- drivers/gpu/drm/i915/gem/selftests/huge_pages.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)Quoted 11 lines
drivers/gpu/drm/i915/gem/selftests/huge_pages.c typedef struct drm_i915_gem_object * · line 1318
Quoted 9 lines
drivers/gpu/drm/i915/gem/selftests/huge_pages.c +1
-1@@ -1316,7 +1316,7 @@ typedef struct drm_i915_gem_object *static inline bool igt_can_allocate_thp (struct drm_i915_private *i915) { -return i915->mm.gemfs &&has_transparent_hugepage (); +return i915->mm.gemfs &&pgtable_has_pmd_leaves ();On second thought, is it problematic that we might be losing the
CONFIG_TRANSPARENT_HUGEPAGEcheck? Should we check for that separately?That's a good point.
In this RFC,
pgtable_has_pmd_leaves()should be functionally equivalent tohas_transparent_hugepage()so I think we're good. That beind said, I also think that we should disentanglepgtable_has_pmd_leaves()from THP now or in the future. When we do this the breakage you're spotting will happen.
Quoted 5 lines
What about adding
thp_has_pmd_support()which does:return IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE) && pgtable_has_pmd_leaves();Then I can convert all the cases you spotted to
thp_has_pmd_support().
I hope we can avoid such a wrapper for the time being. Maybe we can just keep pgtable_has_pmd_leaves() glued to CONFIG_TRANSPARENT_HUGEPAGE for now, and leave untangling that for the next cleanup?
Signature · 3 lines
Cheers David
David Hildenbrand (Red Hat)
maintainerRe: [RFC 07/10] treewide: rename has_transparent_hugepage() to arch_has_pmd_leaves()
In reply to Luiz CapitulinoQuoted 45 lines · Luiz Capitulino
Quoted 5 lines · David Hildenbrand (Red Hat)
Quoted 4 lines · Luiz Capitulino
Now that the majority of
has_transparent_hugepage()callers have been converted topgtable_has_pmd_leaves(), renamehas_transparent_hugepage()toarch_has_pmd_leaves()since that's what the helper checks for.Quoted 7 lines
Quoted 5 lines
arch_has_pmd_leaves()is supposed to be called only byinit_arch_has_pmd_leaves(), except for two exeptions:1. shmem: shmem code runs very early during boot so it can't use
pgtable_has_pmd_leaves()Can't we just initialize
pgtable_has_pmd_leaves()earlier then?I can look into doing that. When I worked on this RFC I wondered if
arch_has_pmd_leaves()(when implemented by the arch) could run so early given that some (all?) archs check feature bits so they must be available this early as well. But I'll check this, havingpgtable_has_pmd_leaves()being available as early as possible is probably the right thing to do.Quoted 9 lines
Quoted 5 lines
2.
hugepage_init(): just a temporary exception, this function will be converted in a future commitSigned-off-by Luiz Capitulino <[email protected]>---
[...]
Quoted 14 lines
include/linux/pgtable.h static inline bool pgtable_has_pmd_leaves(void) · line 2030
Quoted 12 lines
include/linux/pgtable.h +2
-2@@ -2026,8 +2026,8 @@ static inline bool pgtable_has_pmd_leaves(void)#endif #endif -#ifndef has_transparent_hugepage -#define has_transparent_hugepage () IS_BUILTIN(CONFIG_TRANSPARENT_HUGEPAGE) +#ifndef arch_has_pmd_leaves +#define arch_has_pmd_leaves () IS_BUILTIN(CONFIG_TRANSPARENT_HUGEPAGE)#endif Ah, so it stays for now only set with
CONFIG_TRANSPARENT_HUGEPAGE. I guess that's something to sort out later :)
Quoted 3 lines
I suggested something we could do in this series. Also, I skipped commenting on all the cases you spotted as I think they refer to the same issue (please, do point out if you think I'm wrong).
Right. If we keep PMD-leave support glue to CONFIG_TRANSPARENT_HUGEPAGE in this series, could we also simplify patch #2, to have it reside in mm/huge_memory.c ?
Then, it's clearer that it is still glued to CONFIG_TRANSPARENT_HUGEPAGE.
Signature · 3 lines
Cheers David
Re: [RFC 04/10] drivers: i915 selftest: use pgtable_has_pmd_leaves()
In reply to David Hildenbrand (Red Hat)Quoted 31 lines · David Hildenbrand (Red Hat)
Quoted 30 lines · Luiz Capitulino
Quoted 9 lines · David Hildenbrand (Red Hat)
Quoted 8 lines · Luiz Capitulino
igt_can_allocate_thp()useshas_transparente_hugepage()to check if PMD-sized pages are supported, usepgtable_has_pmd_leaves()instead.Signed-off-by Luiz Capitulino <[email protected]>---
drivers/gpu/drm/i915/gem/selftests/huge_pages.c| 2 +-1 file changed, 1 insertion(+), 1 deletion(-)Quoted 11 lines
drivers/gpu/drm/i915/gem/selftests/huge_pages.c typedef struct drm_i915_gem_object * · line 1318
Quoted 9 lines
drivers/gpu/drm/i915/gem/selftests/huge_pages.c +1
-1@@ -1316,7 +1316,7 @@ typedef struct drm_i915_gem_object * static inline bool igt_can_allocate_thp(struct drm_i915_private *i915) { - return i915->mm.gemfs && has_transparent_hugepage(); + return i915->mm.gemfs && pgtable_has_pmd_leaves();On second thought, is it problematic that we might be losing the
CONFIG_TRANSPARENT_HUGEPAGEcheck? Should we check for that separately?That's a good point.
In this RFC,
pgtable_has_pmd_leaves()should be functionally equivalent tohas_transparent_hugepage()so I think we're good. That beind said, I also think that we should disentanglepgtable_has_pmd_leaves()from THP now or in the future. When we do this the breakage you're spotting will happen.
Quoted 7 lines
Quoted 5 lines
What about adding
thp_has_pmd_support()which does:return
IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE)&&pgtable_has_pmd_leaves();Then I can convert all the cases you spotted to
thp_has_pmd_support().I hope we can avoid such a wrapper for the time being. Maybe we can just keep
pgtable_has_pmd_leaves()glued toCONFIG_TRANSPARENT_HUGEPAGEfor now, and leave untangling that for the next cleanup?
OK, I can do that in the v1 posting.
Sorry I didn't say at the time - thanks, I do plan to look through this at some point :)
Cheers, Lorenzo
Quoted 58 lines · Luiz Capitulino
Today, if an architecture implements
has_transparent_hugepage()and the CPU lacks support for PMD-sized pages, the THP code disables all THP, including mTHP support. In addition, the kernel lacks a well defined API to check for PMD-sized page support. It currently relies onhas_transparent_hugepage()andthp_disabled_by_hw(), but they are not well defined and are tied to THP support.This series addresses both issues by introducing a new well defined API to query PMD-sized page support:
pgtable_has_pmd_leaves(). Using this new helper, we ensure that mTHP remains enabled even when the architecture or CPU doesn't support PMD-sized pages.An important detail is that we need to do the same refactoring for
has_transparent_pud_hugepage(). I actually have patches for this one too, I'm not including them here because I want to get some initial feedback on the general approach first (and maybe it's better to do that in a separate series).Thanks to David Hildenbrand for suggesting this improvement and for providing guidance (all bugs and misconcentpions are mine).
Luiz Capitulino (10):
docs: tmpfs: remove implementation detail reference
mm: introducepgtable_has_pmd_leaves()
drivers: dax: usepgtable_has_pmd_leaves()
drivers: i915 selftest: usepgtable_has_pmd_leaves()
drivers: nvdimm: usepgtable_has_pmd_leaves()
mm:debug_vm_pgtable: usepgtable_has_pmd_leaves()
treewide: renamehas_transparent_hugepage()toarch_has_pmd_leaves()
mm: replacethp_disabled_by_hw()withpgtable_has_pmd_leaves()
mm: thp: always enable mTHP support
mm: thp: x86: cleanup PSE feature bit usageDocumentation/filesystems/tmpfs.rst | 5 ++--- arch/mips/include/asm/pgtable.h | 4 ++-- arch/mips/mm/tlb-r4k.c | 4 ++-- arch/powerpc/include/asm/book3s/64/hash-4k.h | 2 +- arch/powerpc/include/asm/book3s/64/hash-64k.h | 2 +- arch/powerpc/include/asm/book3s/64/pgtable.h | 10 +++++----- arch/powerpc/include/asm/book3s/64/radix.h | 2 +- arch/powerpc/mm/book3s64/hash_pgtable.c | 4 ++-- arch/s390/include/asm/pgtable.h | 4 ++-- arch/x86/include/asm/pgtable.h | 6 ------ arch/x86/include/asm/pgtable_32.h | 6 ++++++ drivers/dax/dax-private.h | 2 +- .../gpu/drm/i915/gem/selftests/huge_pages.c | 2 +- drivers/nvdimm/pfn_devs.c | 4 ++-- include/linux/huge_mm.h | 7 ------- include/linux/pgtable.h | 14 +++++++++++-- mm/debug_vm_pgtable.c | 20 +++++++++---------- mm/huge_memory.c | 13 ++++++------ mm/memory.c | 12 ++++++++++- mm/shmem.c | 8 ++++---- 20 files changed, 72 insertions(+), 59 deletions(-)--
2.51.1
Quoted 2 lines · Lorenzo Stoakes
Sorry I didn't say at the time - thanks, I do plan to look through this at some point :)
Oh, thanks for letting me know. I appreciate that!
I assumed reviewers are waiting for v1, esp. after David's review.
PS: I hope I'll be able to send v1 in the next few weeks, but I can't guarantee it.
David Hildenbrand (Red Hat)
maintainerRe: [RFC 06/10] mm: debug_vm_pgtable: use pgtable_has_pmd_leaves()
In reply to David Hildenbrand (Red Hat)Quoted 6 lines · David Hildenbrand (Red Hat)
Quoted 2 lines · Luiz Capitulino
debug_vm_pgtablecallshas_transparent_hugepage()in multiple places to check if PMD-sized pages are supported, usepgtable_has_pmd_leaves()Is that code even dealing with pages? Likely we really only want to check whether PMD leaves are supported, independent of any pages.
I recall that some arch protect certain pmd_* helpers under CONFIG_TRANSPARENT_HUGEPAGE.
But it's certainly something to explore, to see if we can just get rid of that without causing compile-time or runtime issues.
Quoted 16 lines
Quoted 16 lines
instead.
Signed-off-by Luiz Capitulino <[email protected]>--- mm/debug_vm_pgtable.c | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-)mm/debug_vm_pgtable.c +0
-0@@ -170,7 +170,7 @@ static void __init pmd_basic_tests(struct pgtable_debug_args *args, int idx)unsigned long val = idx, *ptr = &val;pmd_t pmd;
Quoted 6 lines
mm/debug_vm_pgtable.c static void __init pmd_basic_tests(struct pgtable_debug_args *args, int idx) · line 174
Quoted 3 lines
mm/debug_vm_pgtable.c +1
-1-if (!has_transparent_hugepage ()) +if (!pgtable_has_pmd_leaves ())return ;All of these are currently protected by
CONFIG_TRANSPARENT_HUGEPAGE, right?
IIUC, yes.
Signature · 3 lines
Cheers David